Self-hosted RAG stacks

RAG stacks

• RAG Pipeline (rag) – Production-ready RAG pipeline with vector DB and LLM integration

About RAG stacks

A RAG stack answers questions from your own documents. Documents are split into chunks, turned into vectors by an embedding model and stored in a vector database. At question time the most similar chunks are retrieved and passed to the LLM as context, so answers are grounded in your data instead of the model's training set.

Running the whole pipeline yourself keeps documents, prompts and answers inside your network, and lets you swap each component independently.

What a production RAG stack needs:

• A vector database with persistence and backups (Chroma, Qdrant, pgvector and others)
• An embedding service whose model matches the index dimension
• An LLM server with an OpenAI-compatible API
• An ingestion job, plus a way to measure retrieval quality before you tune prompts

Frequently asked questions

What is a self-hosted RAG stack?

It is a retrieval-augmented generation pipeline (vector database, embedding model and LLM) that runs on servers you control instead of a hosted API, so your documents and prompts never leave your infrastructure.

Do I need a GPU to run RAG locally?

Not for small setups. Embedding models and small quantized LLMs run on CPU, just slower. A GPU becomes worthwhile once you need low latency, larger models or many concurrent users.

Which vector database should I start with?

If you already run PostgreSQL, pgvector is the simplest start. For a dedicated service with strong filtering, Qdrant is a common choice. Chroma is convenient for prototypes.

Guides

• How to Self-Host a RAG Pipeline with Docker Compose – A complete, working self-hosted RAG stack in one docker-compose.yml: Ollama for chat and embeddings, Qdrant for vectors, and a small Python service for ingestion and queries.
• Ollama vs vLLM vs llama.cpp: Choosing a Self-Hosted Inference Server – How the three most common self-hosted LLM inference servers differ in architecture, hardware support, concurrency and operations, and which one fits your workload.
• pgvector vs Qdrant vs Weaviate vs Milvus vs Chroma for RAG – A practical comparison of the five vector databases people actually self-host for RAG, with a decision guide by scenario instead of benchmark charts.
• How to Choose the Best Embedding Model for Self-Hosted RAG – A practical way to choose a self-hosted embedding model: what the leaderboard can and cannot tell you, what each dimension costs in storage, and how to serve, swap and rerank.
• How to Evaluate a RAG Pipeline: Metrics, Test Sets and CI – Measure retrieval and generation separately, build a small golden test set, use a self-hosted LLM judge carefully, and run RAG evaluations on every change.
• Self-Hosting LLMs for GDPR: A Practical Checklist – Running the model on your own servers removes one big data flow, but you are still the controller for every prompt, log and embedding. This checklist covers the rest.