RAG Pipeline

Overview

A complete retrieval-augmented generation (RAG) stack that runs on your own machine or server. You upload text, Markdown or PDF files, and ask questions about them over HTTP. Answers come with the passages they are based on.

The stack has three services: Qdrant stores the document chunks as vectors, Ollama runs the embedding model (nomic-embed-text by default) and the chat model (llama3.2:3b by default), and rag-api, a small FastAPI service, splits documents into overlapping chunks, stores them in Qdrant and answers questions from the best-matching chunks (POST /ingest, POST /query, GET /health).

Everything stays local: no external AI API is called. Qdrant requires an API key, rag-api can require one, and all ports are published on 127.0.0.1 only. An optional override runs Ollama on an NVIDIA GPU.

Requirements

• Docker Engine 25 or newer with the Docker Compose plugin
• 4 CPU cores and about 8 GB RAM for the default models (llama3.2:3b, nomic-embed-text); larger chat models need more
• About 15 GB free disk: images about 9 GB (mostly Ollama), default models about 2.3 GB, plus your documents
• Optional: NVIDIA GPU with the NVIDIA Container Toolkit for faster answers

Quick start

Quick start (Docker Compose)

1. Get the stack:

   git clone https://github.com/aisolutionscamp/stacks.git
   cd stacks/stacks/rag-pipeline

2. Create .env with two generated keys:

   cp .env.example .env
   sed -i "s/^QDRANT_API_KEY=.*/QDRANT_API_KEY=$(openssl rand -hex 32)/" .env
   sed -i "s/^RAG_API_KEY=.*/RAG_API_KEY=$(openssl rand -hex 32)/" .env

3. Start it. The first start downloads about 6 GB (images and models) and can take several minutes:

   docker compose up -d --build --wait

With an NVIDIA GPU: docker compose -f compose.yaml -f compose.gpu.yaml up -d --build --wait.

4. Add a document and ask a question:

   KEY=$(grep ^RAG_API_KEY= .env | cut -d= -f2)
   curl -s -X POST localhost:8000/ingest -H "X-API-Key: $KEY" -F files=@README.md
   curl -s -X POST localhost:8000/query -H "X-API-Key: $KEY" \
     -H 'Content-Type: application/json' -d '{"question":"Which services does the stack run?"}'

5. Optional: check the whole stack with ./smoke-test.sh. Stop it with docker compose down (add -v to delete documents and models).

Environment variables

• QDRANT_API_KEY (required) – API key for Qdrant, used by rag-api and the Qdrant dashboard. Generate with openssl rand -hex 32.
• RAG_API_KEY – API key clients send as X-API-Key or Bearer token. Empty disables authentication; set it for anything beyond local testing.
• EMBED_MODEL – Ollama embedding model. Changing it needs a new QDRANT_COLLECTION because the vector size changes.
• CHAT_MODEL – Ollama chat model that writes the answers.
• PULL_MODELS – Pull missing models into Ollama when rag-api starts.
• OLLAMA_KEEP_ALIVE – How long Ollama keeps a model loaded after a request.
• QDRANT_COLLECTION – Qdrant collection that stores the chunks.
• CHUNK_SIZE – Chunk size in characters.
• CHUNK_OVERLAP – Overlap between neighbouring chunks in characters.
• TOP_K – Number of chunks passed to the model per question.
• MAX_UPLOAD_MB – Maximum size of one uploaded file or JSON request in MB.
• LLM_TIMEOUT – Seconds to wait for Ollama; CPU inference can be slow.
• LLM_TEMPERATURE – Sampling temperature for answers.
• RAG_API_PORT – Host port for rag-api.
• RAG_API_BIND – Host address rag-api is published on. Use 0.0.0.0 only behind a firewall or TLS proxy.
• QDRANT_PORT – Host port for the Qdrant REST API and dashboard (127.0.0.1 only).
• OLLAMA_PORT – Host port for Ollama (127.0.0.1 only, no authentication).
• RAG_API_VERSION – Tag of ghcr.io/aisolutionscamp/rag-api, also used for the local build.

Documentation

RAG Pipeline

Ask questions about your own documents. You send text, Markdown or PDF files to an HTTP API; the stack splits them into chunks, stores them as vectors and answers questions with a local language model, citing the passages it used. No external AI service is involved.

Architecture

| Service | Image | Port (host) | What it does | |---|---|---|---| | rag-api | ghcr.io/aisolutionscamp/rag-api:1.0.0 (built from app/) | 127.0.0.1:8000 | POST /ingest, POST /query, DELETE /documents, GET /health | | qdrant | qdrant/qdrant:v1.20.0 | 127.0.0.1:6333 | Vector store; REST API and dashboard at /dashboard | | ollama | ollama/ollama:0.40.11 | 127.0.0.1:11434 | Runs the embedding model and the chat model |

How a request flows:

1. Ingest: rag-api extracts the text (PDF page by page), splits it into chunks of CHUNKSIZE characters with CHUNKOVERLAP overlap (breaking at paragraphs, lines or sentences where possible), embeds the chunks with EMBEDMODEL in Ollama and upserts them into the Qdrant collection QDRANTCOLLECTION. Each point stores source, chunk, page (PDF), text, metadata and ingested_at. Ingesting a source again replaces its earlier chunks.
2. Query: rag-api embeds the question, takes the TOPK most similar chunks from Qdrant and asks CHATMODEL to answer only from them. The response contains the answer and the sources with their scores.

On its first start rag-api pulls both models into Ollama (about 2.3 GB with the defaults) and creates the collection with the vector size of the embedding model. GET /health returns 503 until that is done, so docker compose up --wait returns once the stack can answer questions.

Data lives in the named volumes qdrantdata (vectors) and ollamadata (models).

API

All endpoints except /health and /health/live need the API key when RAGAPIKEY is set, as X-API-Key: or Authorization: Bearer . The OpenAPI description is at /docs.

KEY=$(grep ^RAG_API_KEY= .env | cut -d= -f2)

# Ingest files (.txt, .md, .pdf); the file name becomes the source
curl -s -X POST localhost:8000/ingest -H "X-API-Key: $KEY" \
  -F files=@handbook.pdf -F files=@notes.md

# Ingest text as JSON
curl -s -X POST localhost:8000/ingest -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"documents":[{"source":"faq","text":"Support is available Monday to Friday, 9 to 17.","metadata":{"team":"support"}}]}'

# Ask a question (optional: "top_k", and "source" to search one document only)
curl -s -X POST localhost:8000/query -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"question":"When is support available?"}'
# Response shape (answer text and score vary):
# {"answer":"...","model":"llama3.2:3b",
#  "sources":[{"n":1,"source":"faq","page":null,"chunk":0,"score":0.8,"text":"Support is ..."}]}

# Remove a document
curl -s -X DELETE "localhost:8000/documents?source=faq" -H "X-API-Key: $KEY"

Configuration

Copy .env.example to .env. Every setting is commented there; the main ones:

| Variable | Default | Notes | |---|---|---| | QDRANTAPIKEY | none, required | openssl rand -hex 32 | | RAGAPIKEY | empty | Set it. Empty means no authentication on rag-api. | | EMBEDMODEL | nomic-embed-text:v1.5 | Any Ollama embedding model, e.g. mxbai-embed-large, bge-m3 | | CHATMODEL | llama3.2:3b | Any Ollama chat model, e.g. qwen2.5:7b, llama3.1:8b (more RAM) | | CHUNKSIZE / CHUNKOVERLAP | 1000 / 200 | Characters | | TOPK | 4 | Chunks per question | | RAGAPIPORT, QDRANTPORT, OLLAMAPORT | 8000, 6333, 11434 | Published on 127.0.0.1 | | RAGAPI_BIND | 127.0.0.1 | Set 0.0.0.0 only behind a firewall or a TLS proxy |

Changing EMBEDMODEL changes the vector size. rag-api then reports the mismatch on /health and does not start serving; set a new QDRANTCOLLECTION and ingest your documents again.

GPU: with an NVIDIA GPU and the NVIDIA Container Toolkit, start with the override file:

docker compose -f compose.yaml -f compose.gpu.yaml up -d --build --wait

Published image instead of a local build: docker compose pull rag-api && docker compose up -d --no-build --wait.

Security notes

• Qdrant always requires QDRANTAPIKEY; compose refuses to start without it.
• Set RAGAPIKEY. Without it anyone who can reach port 8000 can read and change your documents.
• All ports are bound to 127.0.0.1. Ollama has no authentication, so never publish port 11434 to a network.
• rag-api runs as a non-root user with a read-only filesystem and no Linux capabilities.
• The API speaks plain HTTP. For access from other machines put a reverse proxy with TLS (Caddy, Traefik, nginx) in front, or use an SSH or SSM tunnel.
• Answers are generated by a language model and can be wrong. The returned sources let you check them.
• Telemetry: Qdrant telemetry is disabled (QDRANT_TELEMETRYDISABLED).

Deployment options

• Docker Compose: this directory, see quickstart/docker.md.
• Kubernetes (Helm): the rag-stack chart deploys the same three services (Qdrant and Ollama as subcharts), generates the API keys and adds NetworkPolicies. See helm/install.sh and helm/values.yaml.
• Terraform, Docker provider: terraform/local/main.tf creates the same containers on one Docker host.
• Terraform, AWS: terraform/aws/main.tf creates one EC2 instance (CPU, optional GPU type) that runs this compose stack, with SSM Session Manager instead of SSH and rag-api reachable only from allowed_cidr.

Upgrading

1. Read the release notes of the new Qdrant and Ollama versions.
2. Change the image tags in compose.yaml (or pull the new version of this repository) and run docker compose up -d --build --wait. Volumes are kept.
3. Back up Qdrant before major upgrades. Create a snapshot, then download it (snapshots are written inside the container, not to the volume):

   QK=$(grep ^QDRANT_API_KEY= .env | cut -d= -f2)
   curl -s -X POST -H "api-key: $QK" localhost:6333/collections/documents/snapshots   # prints the snapshot name
   curl -s -H "api-key: $QK" -o documents.snapshot localhost:6333/collections/documents/snapshots/

Model upgrades: change CHAT_MODEL and restart rag-api; the new model is pulled on start. Old models stay in the volume until you remove them with docker compose exec ollama ollama rm .

Troubleshooting

| Symptom | Check | |---|---| | up --wait takes long on first start | rag-api is pulling the models: docker compose logs -f rag-api | | /health returns 503 with vector size in detail | The embedding model changed; use a new QDRANTCOLLECTION | | /health detail waiting for dependencies | docker compose ps; Qdrant or Ollama is not healthy | | 401 invalid or missing API key | Send X-API-Key with the value of RAGAPIKEY | | 502 with Qdrant ... returned 401 | rag-api and Qdrant use different QDRANTAPIKEY values; recreate both containers | | Query times out | CPU inference is slow for large models; use a smaller CHATMODEL, raise LLM_TIMEOUT, or use the GPU override | | PDF gives no chunks | Scanned PDFs contain images, not text; run OCR first | | 415 unsupported file type | Only .txt, .md and .pdf are accepted |