Can I run a RAG pipeline without a GPU?

Yes. Ollama falls back to CPU, and embedding models like nomic-embed-text run acceptably on CPU. Generation with an 8B model will be slow on CPU, so use a smaller model or add a GPU for interactive use.

Which vector database should I use for self-hosted RAG?

Qdrant is a good default for a single-host Docker setup because it is one container with built-in API key auth, payload filtering and snapshots. If you already run PostgreSQL, pgvector avoids adding another service.

What chunk size should I use for RAG?

Start with roughly 300 to 800 tokens per chunk and 10 to 15 percent overlap, split on paragraph or heading boundaries. Then measure retrieval recall on your own questions and adjust one parameter at a time.

Why does Qdrant say the vector dimension is wrong?

The collection was created for a different embedding model. Each model produces a fixed vector size, for example 768 for nomic-embed-text and 1024 for bge-m3, so create a new collection and re-ingest when you change models.

Do I need LangChain or LlamaIndex for a self-hosted RAG pipeline?

No. A basic pipeline is chunking, an embedding call, a vector search and a chat call, which fits in a short Python service. Frameworks help later when you need many loaders, rerankers or agents.

How to Self-Host a RAG Pipeline with Docker Compose

Updated 2026-10-26 · 12 min read

A complete, working self-hosted RAG stack in one docker-compose.yml: Ollama for chat and embeddings, Qdrant for vectors, and a small Python service for ingestion and queries.

To self-host a RAG pipeline with Docker Compose you need three containers: Ollama to serve both the chat model and the embedding model, Qdrant to store and search vectors, and a small service that chunks documents, embeds them, retrieves the closest chunks for a question and asks the LLM to answer from them. This guide gives you a complete, working version of that self-hosted RAG stack: one compose.yaml with pinned image tags, healthchecks and volumes, an optional GPU override, and a Python service of about 150 lines you can read in one sitting.

Self-hosted RAG in short

A minimal self-hosted RAG pipeline on Docker Compose looks like this:

Ollama runs nomic-embed-text (768-dimensional embeddings) and llama3.1:8b (generation). A one-shot container pulls both models before anything else starts.
Qdrant stores one vector per chunk plus the chunk text and source as payload.
A Python service exposes /search and /query, and has an ingest command that reads a folder of .md, .txt and .pdf files.

Everything runs on one machine, nothing leaves your network, and only the RAG API is published, bound to localhost. If you would rather start from a packaged stack, the RAG pipeline stack in our catalog uses ChromaDB instead of Qdrant and follows the same pattern.

Architecture and why these components

| Component | Choice here | Why | Swap for | |---|---|---|---| | LLM server | Ollama | One binary serves chat and embedding models, simple model management, CPU fallback | vLLM for high concurrency on GPUs | | Embedding model | nomic-embed-text via Ollama | Small, runs fine on CPU, long input window | bge-m3, or a dedicated embedding service with TEI | | Vector DB | Qdrant | Single container, payload filters, snapshots, API key auth built in | pgvector if you already run Postgres | | Glue | FastAPI + qdrant-client + httpx | No framework lock-in, every step visible | LangChain or LlamaIndex once you know what you need |

Running embeddings through the same Ollama instance keeps the stack small. The trade-off is that ingestion and chat compete for the same CPU or GPU. For a few thousand documents that is fine. If you ingest continuously while serving users, move embeddings to their own container. The guides on choosing an embedding model and comparing vector databases go deeper on those two decisions.

Project layout

rag-stack/
├── compose.yaml
├── compose.gpu.yaml
├── .env
├── docs/              # your source documents, mounted read-only
└── rag/
    ├── Dockerfile
    ├── requirements.txt
    └── app.py

The docker-compose.yml for a self-hosted RAG pipeline

The file below is the whole stack. Read the comments; most of them explain a decision you would otherwise discover the hard way.

# compose.yaml
name: rag

services:
  ollama:
    # Pin the tag. "latest" changes model handling and defaults between releases,
    # and a silent upgrade is the last thing you want to debug in a RAG answer.
    image: ollama/ollama:0.40.11
    restart: unless-stopped
    environment:
      OLLAMA_KEEP_ALIVE: "30m"        # keep models loaded between requests (default is 5m)
      OLLAMA_CONTEXT_LENGTH: "8192"   # default is 4096, too small for 5 chunks plus a question
      OLLAMA_NUM_PARALLEL: "2"        # parallel requests per model
    volumes:
      - ollama:/root/.ollama          # model weights survive container rebuilds
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      timeout: 5s
      retries: 12
      start_period: 20s
    # No "ports": Ollama is only reachable from inside the compose network.

  ollama-pull:
    # One-shot job: pull the models, then exit 0. The rag service waits for it.
    image: ollama/ollama:0.40.11
    depends_on:
      ollama:
        condition: service_healthy
    environment:
      OLLAMA_HOST: http://ollama:11434
    entrypoint: ["/bin/sh", "-c", "ollama pull ${EMBED_MODEL} && ollama pull ${CHAT_MODEL}"]
    restart: "no"

  qdrant:
    image: qdrant/qdrant:v1.20.0
    restart: unless-stopped
    environment:
      QDRANT__SERVICE__API_KEY: ${QDRANT_API_KEY:?set QDRANT_API_KEY in .env}
    volumes:
      - qdrant:/qdrant/storage
    healthcheck:
      # The image has no curl, so speak HTTP to /readyz with bash.
      test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/6333 && printf 'GET /readyz HTTP/1.0\\r\\n\\r\\n' >&3 && grep -q ' 200 ' <&3"]
      interval: 10s
      timeout: 5s
      retries: 10

  rag:
    build: ./rag
    restart: unless-stopped
    depends_on:
      ollama-pull:
        condition: service_completed_successfully
      qdrant:
        condition: service_healthy
    environment:
      OLLAMA_URL: http://ollama:11434
      QDRANT_URL: http://qdrant:6333
      QDRANT_API_KEY: ${QDRANT_API_KEY}
      EMBED_MODEL: ${EMBED_MODEL}
      CHAT_MODEL: ${CHAT_MODEL}
      NUM_CTX: "8192"
    volumes:
      - ./docs:/docs:ro
    ports:
      - "127.0.0.1:8000:8000"         # localhost only; put a reverse proxy in front for remote access
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz')"]
      interval: 15s
      timeout: 5s
      retries: 5

volumes:
  ollama:
  qdrant:

And the .env next to it:

QDRANT_API_KEY=change-me-to-a-long-random-string
EMBED_MODEL=nomic-embed-text
CHAT_MODEL=llama3.1:8b

Generate the key with openssl rand -hex 32. Compose reads .env automatically and the :? syntax makes it refuse to start if the key is missing, which is better than running an open vector database by accident.

Why pinned tags matter here

A RAG pipeline has more moving parts than a typical web app: the model server, the vector database, the client library and the model weights all have versions. If Qdrant moves to a new minor version under you, the Python client may warn about incompatibility. If Ollama changes a default such as context length, answers change without any code change. Pin image tags, pin Python packages, and pin model tags (llama3.1:8b rather than llama3.1). Upgrade deliberately, then re-run your evaluation set. The RAG evaluation guide linked at the end shows how to build one.

Optional: GPU support

On a machine with an NVIDIA GPU and the NVIDIA Container Toolkit installed, add an override file instead of editing the main one:

# compose.gpu.yaml
services:
  ollama:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

docker compose -f compose.yaml -f compose.gpu.yaml up -d
docker compose exec ollama ollama ps

ollama ps shows a PROCESSOR column. It should say 100% GPU once a model is loaded. A split such as 30%/70% CPU/GPU means the model plus its context does not fit in VRAM, and generation will be much slower. Reduce OLLAMACONTEXTLENGTH, use a smaller quantization, or pick a smaller model. The VRAM guide linked at the end has the formula for estimating what fits.

The ingestion and query service

requirements.txt and Dockerfile

fastapi==0.115.12
uvicorn==0.34.0
httpx==0.28.1
qdrant-client==1.20.0
pypdf==5.4.0

Keep qdrant-client on the same minor version as the Qdrant server image. The other pins are known-good versions; bump them on purpose.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
RUN useradd --create-home app
USER app
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

app.py

import os
import sys
import uuid
from contextlib import asynccontextmanager
from pathlib import Path

import httpx
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from pypdf import PdfReader
from qdrant_client import QdrantClient
from qdrant_client.models import (Distance, FieldCondition, Filter, FilterSelector,
                                  MatchValue, PointStruct, VectorParams)

OLLAMA_URL = os.environ["OLLAMA_URL"]
EMBED_MODEL = os.environ.get("EMBED_MODEL", "nomic-embed-text")
CHAT_MODEL = os.environ.get("CHAT_MODEL", "llama3.1:8b")
COLLECTION = os.environ.get("COLLECTION", "docs")
CHUNK_CHARS = int(os.environ.get("CHUNK_CHARS", "1800"))
OVERLAP_CHARS = int(os.environ.get("OVERLAP_CHARS", "200"))
TOP_K = int(os.environ.get("TOP_K", "5"))
NUM_CTX = int(os.environ.get("NUM_CTX", "8192"))
# nomic-embed-text expects task prefixes; set both to "" for models that do not.
DOC_PREFIX = os.environ.get("DOC_PREFIX", "search_document: ")
QUERY_PREFIX = os.environ.get("QUERY_PREFIX", "search_query: ")

SYSTEM_PROMPT = (
    "Answer only from the numbered context passages. Cite passages like [1]. "
    "If the context does not contain the answer, say you do not know."
)

qdrant = QdrantClient(url=os.environ["QDRANT_URL"], api_key=os.environ.get("QDRANT_API_KEY") or None)
http = httpx.Client(timeout=300)


def embed(texts: list[str]) -> list[list[float]]:
    r = http.post(f"{OLLAMA_URL}/api/embed", json={"model": EMBED_MODEL, "input": texts})
    if r.status_code == 404:
        raise RuntimeError(f"Model {EMBED_MODEL} not found in Ollama. Run: ollama pull {EMBED_MODEL}")
    r.raise_for_status()
    return r.json()["embeddings"]


def ensure_collection() -> None:
    dim = len(embed(["dimension probe"])[0])
    if not qdrant.collection_exists(COLLECTION):
        qdrant.create_collection(COLLECTION, vectors_config=VectorParams(size=dim, distance=Distance.COSINE))
        return
    existing = qdrant.get_collection(COLLECTION).config.params.vectors.size
    if existing != dim:
        raise RuntimeError(
            f"Collection '{COLLECTION}' has dimension {existing}, but {EMBED_MODEL} returns {dim}. "
            "Use a new collection name or re-ingest after deleting the old one."
        )


def chunk(text: str) -> list[str]:
    """Pack paragraphs into chunks of up to CHUNK_CHARS, hard-splitting long paragraphs."""
    chunks, current = [], ""
    for para in (p.strip() for p in text.split("\n\n")):
        if not para:
            continue
        if len(current) + len(para) + 2 <= CHUNK_CHARS:
            current = f"{current}\n\n{para}" if current else para
            continue
        if current:
            chunks.append(current)
        while len(para) > CHUNK_CHARS:
            chunks.append(para[:CHUNK_CHARS])
            para = para[CHUNK_CHARS - OVERLAP_CHARS:]
        tail = chunks[-1][-OVERLAP_CHARS:] if chunks else ""
        current = f"{tail}\n\n{para}" if tail and len(tail) + len(para) + 2 <= CHUNK_CHARS e
…

A few deliberate choices in this code:

The collection dimension comes from the model, not from a hard-coded number. The service embeds a probe string at startup and refuses to run against a collection with a different size.
Point IDs are deterministic (uuid5 of source and chunk index), and each file's old chunks are deleted before upsert. Re-running ingestion is safe.
/query returns the retrieved contexts and the prompt token count. You need the first for evaluation and the second to detect truncation.

Chunking choices

Chunking has more influence on answer quality than most model swaps. The service above packs whole paragraphs into chunks of up to 1,800 characters with a 200-character overlap when a paragraph has to be split. For English prose, a rough rule of thumb is about four characters per token, so that is roughly 450 tokens per chunk.

| Strategy | Good for | Weakness | |---|---|---| | Fixed size with overlap | Uniform text, logs, transcripts | Cuts sentences and tables mid-way | | Paragraph packing (used here) | Docs, wikis, Markdown | Very long paragraphs still need a hard split | | Heading-aware (split on ##) | Structured manuals, API docs | Sections can be far too long or too short | | Semantic splitting | Mixed content | Needs an extra embedding pass, harder to debug |

As a starting point:

Chunk size: 300 to 800 tokens. Smaller chunks retrieve more precisely but lose surrounding context; larger ones dilute the embedding.
Overlap: 10 to 15 percent of the chunk size. Overlap mainly helps when facts span a boundary.
Budget the context window. With TOP_K=5 and 450-token chunks you send roughly 2,250 tokens of context plus the system prompt, question and answer. That is why the compose file raises the context length to 8192: the 4096 default leaves too little headroom once answers get long.
Keep the source in the payload and show it to users. Citations are what make RAG answers checkable.

Change one parameter at a time and measure retrieval quality before and after. Guessing does not work here.

Start the stack and ingest documents

docker compose up -d --build --wait
docker compose logs ollama-pull     # first run downloads several GB

--wait blocks until every service with a healthcheck is healthy. On the first run the pull job downloads llama3.1:8b (about 4.9 GB at the default quantization) and the embedding model, so give it time.

Put files into ./docs, then run the ingest command inside the rag container:

docker compose exec rag python app.py ingest /docs

Each file prints its chunk count. Check that the numbers look plausible; a 40-page PDF producing 0 or 1 chunk usually means the PDF is scanned images with no text layer, and you need OCR before ingestion.

Query the pipeline

Check retrieval on its own first:

curl -s http://127.0.0.1:8000/search \
  -H 'Content-Type: application/json' \
  -d '{"question": "How do I rotate the API key?", "top_k": 3}' | jq '.[] | {source, score}'

Then ask for an answer:

curl -s http://127.0.0.1:8000/query \
  -H 'Content-Type: application/json' \
  -d '{"question": "How do I rotate the API key?"}' | jq '{answer, sources, prompt_tokens}'

If the right document does not appear in /search, no prompt engineering will fix the answer. Debug retrieval first, generation second.

Common failure modes

Model not pulled

Symptom: /query returns a 500 and Ollama's log shows a 404 with a message like model "llama3.1:8b" not found, try pulling it first. Usually the pull job failed (disk full, network) or CHAT_MODEL in .env differs from what was pulled. Check with:

docker compose logs ollama-pull
docker compose exec ollama ollama list

Model tags must match exactly. llama3.1 and llama3.1:8b are different names to your code even when they resolve to the same weights.

Wrong embedding dimension

Symptom: Qdrant rejects upserts or searches with a vector dimension error, or the service above stops at startup with its own dimension message. This happens when you switch embedding models (768 dimensions for nomic-embed-text, 1024 for bge-m3) and keep the old collection. Vectors from different models are not comparable even when sizes match, so never mix them. Create a new collection (COLLECTION=docsbgem3), re-ingest, switch traffic, then delete the old one.

Context too long

Symptom: answers ignore the question or the first passages, or prompttokens in the response sits right at NUMCTX. When a prompt exceeds the context window, Ollama truncates it rather than failing, and the model silently loses part of the input. Fixes, in order of preference: retrieve fewer or smaller chunks, raise numctx if VRAM allows, and log prompttokens so you can alert when it approaches the limit. Remember that a larger context costs memory for the KV cache, so raising it can push a model from GPU to partial CPU.

Other things that bite

Healthy but slow first request: the model loads on first use. OLLAMAKEEPALIVE keeps it resident afterwards.
Out-of-memory kills on CPU-only hosts: an 8B model at 4-bit quantization needs roughly 5 to 6 GB of RAM plus the KV cache. Check docker stats.
Insecure-connection warning from qdrant-client: expected when using an API key over plain HTTP inside the compose network. Use TLS if Qdrant is reachable from other hosts.
Empty PDF text: scanned PDFs need OCR first.

Production hardening

The compose file above is a sound single-host setup. Before real users hit it, work through this list:

Network exposure. Only the rag API should be published, and preferably behind a reverse proxy (Caddy, Traefik, nginx) with TLS and authentication. Ollama has no authentication; never publish port 11434.
Secrets. Keep QDRANTAPIKEY in .env with chmod 600, or use Docker secrets. Do not commit .env.
Backups. Use Qdrant snapshots (POST /collections/{name}/snapshots) on a schedule and copy them off the host. Keep the original documents as well; re-ingesting from source is the most reliable recovery.
Resource limits. Set memory limits on rag and qdrant so a runaway ingestion job cannot starve the model server.
Upgrades. Change one pinned version at a time, run your evaluation set, then roll forward. Keep the previous tag noted for rollback.
Observability. Log question, retrieved sources, scores, prompt_tokens and latency per request. These logs become your evaluation data later.
Access control. If different users may see different documents, store an access label in the payload and filter on it in query_points. Filtering after retrieval leaks information through answers.
Concurrency. OLLAMANUMPARALLEL trades per-request speed for throughput. For many concurrent users on GPUs, compare dedicated serving engines such as vLLM.
Data protection. If documents contain personal data, plan logging, retention and deletion before go-live.

Next steps

Measure before you tune: build a small golden set and track recall@k and faithfulness with the RAG evaluation guide.
Browse ready-made configurations in the RAG stack category.
Size your hardware before buying it with the VRAM requirements guide.