How to Choose the Best Embedding Model for Self-Hosted RAG

Updated 2026-10-26 · 10 min read

A practical way to choose a self-hosted embedding model: what the leaderboard can and cannot tell you, what each dimension costs in storage, and how to serve, swap and rerank.

Choosing the best embedding model for RAG is mostly about fit, not leaderboard rank. For a self-hosted embedding model, the short answer is: start with a model in the 300M to 600M parameter range that matches your languages and has a permissive license, such as Qwen3-Embedding-0.6B, BGE-M3 or multilingual-e5-large for multilingual corpora, or bge-large-en-v1.5 or mxbai-embed-large-v1 for English only. Serve it with Hugging Face text-embeddings-inference, use the query prefix the model expects, and confirm the choice with an evaluation on your own questions before you index everything.

This guide covers how to read the MTEB leaderboard without being misled, what dimensions cost in storage, context length, multilingual needs, the model families worth knowing, prefixes, Matryoshka truncation, quantization, serving options, how to migrate to a new model, and where rerankers fit.

How to read the MTEB leaderboard

The MTEB leaderboard is the standard reference for embedding models. It is useful, as long as you know what it measures.

• Pick the right benchmark and column. The leaderboard offers several benchmark sets (English, multilingual, per-language, domain-specific such as code or legal). For RAG, look at the Retrieval score of the set closest to your data, not the overall mean. The overall average also includes classification, clustering, pair classification and STS tasks, which say little about whether the right chunk ends up in the top 5.
• Check zero-shot status. Many MTEB datasets are public, and some models were trained on their training splits or on closely related data. The leaderboard flags how much of a model's training data overlaps with the benchmark. A model that is zero-shot on the benchmark is a more honest signal than one ranked slightly higher with heavy overlap.
• Compare like with like. Small differences in average score between models are often within noise. A 7B model ranking above a 0.6B model tells you little if you cannot afford to run the 7B one at your ingestion volume.
• Read the model card. Max sequence length, required prefixes, embedding dimension, license and whether the score was reproduced or self-reported are all on the card, and all affect you more than a rank.
• Prefer held-out evaluations when available. The MTEB maintainers added retrieval benchmarks with private test sets to reduce overfitting to public data. If your target language or domain has one, weight it more.

Treat the leaderboard as a shortlist generator. The decision comes from running two or three shortlisted models on 50 to 200 real queries with known relevant chunks, and comparing recall@k. The RAG evaluation guide walks through building that test set.

Dimensions and storage cost per vector

Every dimension is stored for every chunk, so the embedding size multiplies directly into disk and memory. The formula is bytes per vector = dimensions x bytes per value:

| Dimensions | float32 (4 B) | float16 (2 B) | int8 (1 B) | binary (1 bit) | |---|---|---|---|---| | 384 | 1,536 B | 768 B | 384 B | 48 B | | 768 | 3,072 B | 1,536 B | 768 B | 96 B | | 1,024 | 4,096 B | 2,048 B | 1,024 B | 128 B | | 2,560 | 10,240 B | 5,120 B | 2,560 B | 320 B | | 4,096 | 16,384 B | 8,192 B | 4,096 B | 512 B |

Scale it to your corpus. Ten million chunks at 1,024 float32 dimensions is 10,000,000 x 4,096 B, about 41 GB of raw vectors. The same corpus with a 4,096-dimension model is about 164 GB. HNSW indexes add graph overhead on top, and most vector databases want the hot part of the index in RAM. That is why a model with four times the dimensions has to deliver a clearly measurable quality gain to be worth it.

Dimensions also interact with your database: pgvector indexes plain vector columns up to 2,000 dimensions and halfvec up to 4,000. The vector database comparison covers those limits per engine.

Context length and chunking

Max sequence length is the number of tokens the model reads; anything longer is silently truncated by most servers (text-embeddings-inference truncates by default with --auto-truncate, and Ollama's /api/embed has truncate set to true by default). A 512-token model embedding 1,500-token chunks only represents the first third of each chunk.

Long context (8K or 32K tokens) is convenient, but it does not mean you should embed whole documents. One vector for a long section averages many topics together and retrieves less precisely. A reasonable starting point for RAG is chunks of a few hundred tokens with some overlap. The practical rule: chunk size must stay below the model's limit, and long-context models give you headroom for tables, code and long paragraphs rather than an invitation to skip chunking.

Multilingual needs

If users ask questions in one language about documents in another, or your corpus mixes languages, you need a multilingual model trained for cross-lingual retrieval. English-only models (bge-large-en-v1.5, mxbai-embed-large-v1, gte-modernbert-base) can produce plausible-looking vectors for German or French text, but retrieval quality drops in ways that are hard to notice without an evaluation. When in doubt, pick a multilingual model and test with queries in each language your users actually write in.

Best embedding models for RAG: families worth knowing

All of the following are open-weight models on Hugging Face. Parameter counts, dimensions, context lengths and licenses come from the model cards and configs; verify again before you deploy, since cards get updated.

| Model | Params | Dims | Max tokens | Languages | License | |---|---|---|---|---|---| | Qwen/Qwen3-Embedding-0.6B | 0.6B | 1,024 (MRL, 32 to 1,024) | 32K | Multilingual | Apache 2.0 | | Qwen/Qwen3-Embedding-4B | 4B | 2,560 (MRL) | 32K | Multilingual | Apache 2.0 | | Qwen/Qwen3-Embedding-8B | 8B | 4,096 (MRL) | 32K | Multilingual | Apache 2.0 | | BAAI/bge-m3 | 568M | 1,024 | 8,192 | Multilingual | MIT | | BAAI/bge-large-en-v1.5 | 335M | 1,024 | 512 | English | MIT | | intfloat/multilingual-e5-large | 560M | 1,024 | 512 | Multilingual | MIT | | Alibaba-NLP/gte-modernbert-base | 149M | 768 | 8,192 | English | Apache 2.0 | | Alibaba-NLP/gte-multilingual-base | 305M | 768 | 8,192 | Multilingual | Apache 2.0 | | nomic-ai/nomic-embed-text-v1.5 | 137M | 768 (MRL) | 8,192 | English | Apache 2.0 | | nomic-ai/nomic-embed-text-v2-moe | 475M (MoE) | 768 (MRL) | 512 | Multilingual | Apache 2.0 | | mixedbread-ai/mxbai-embed-large-v1 | 335M | 1,024 (MRL) | 512 | English | Apache 2.0 | | Snowflake/snowflake-arctic-embed-l-v2.0 | 568M | 1,024 (MRL) | 8,192 | Multilingual | Apache 2.0 | | google/embeddinggemma-300m | 300M | 768 (MRL) | 2,048 | Multilingual | Gemma terms, gated |

Some practical notes on the families:

• Qwen3-Embedding is the strongest family here on public benchmarks and the 0.6B size is cheap enough to run on CPU for moderate volumes. The 4B and 8B sizes need a GPU and produce large vectors; use Matryoshka truncation if you go there.
• BGE-M3 produces dense, sparse (lexical weights) and multi-vector outputs from one model, which makes it attractive for hybrid search. Most servers expose only the dense output; check yours if you want the sparse one.
• E5 and arctic-embed are solid multilingual workhorses with permissive licenses.
• License traps: jina-embeddings-v3 is CC BY-NC 4.0, which rules out commercial use without a separate agreement, and EmbeddingGemma requires accepting Google's Gemma terms. Check the license tag on the card, not a blog post.

Query and passage prefixes

Many retrieval models were trained with different inputs for queries and documents. Leaving the prefix out does not cause an error, it just quietly lowers retrieval quality.

• bge-large-en-v1.5 and mxbai-embed-large-v1: prepend Represent this sentence for searching relevant passages: to queries only; documents get no prefix.
• E5 models: query: for queries and passage: for documents. The instruct variant uses a task instruction on the query instead.
• nomic-embed-text: searchquery: and searchdocument: .
• Qwen3-Embedding: an instruction on the query, Instruct: \nQuery:, and no prefix on documents. You can write a task description that matches your domain.

The catch is that not every model ships its prompts in its sentence-transformers config. Qwen3-Embedding and mxbai do, so encode_query applies them. bge-large-en-v1.5 and nomic-embed-text-v1.5 do not, so you add the prefix yourself. Put the prefix logic in one function in your code, and log it with the model name, so ingestion and query time can never drift apart.

Matryoshka truncation and quantization

Matryoshka Representation Learning (MRL) trains a model so that the first N dimensions of a vector are a usable embedding on their own. With an MRL model you can store, for example, 256 of 768 dimensions and cut storage by two thirds, usually with a modest quality loss that you should measure. Rules:

• Only truncate models trained for it (marked MRL in the table). Truncating other models works badly.
• Re-normalize the truncated vector if you use cosine or dot-product similarity.
• Truncate the same way at ingestion and query time.

Quantization goes further. int8 stores one byte per dimension; it needs calibration ranges, which your vector database or library computes from a sample. Binary quantization stores one bit per dimension and is typically used for a fast first pass, with rescoring on full-precision vectors. Many vector databases (Qdrant, Weaviate, Milvus, pgvector via binary_quantize) do this internally, which is usually simpler than quantizing in your application.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True, truncate_dim=256)

# nomic-embed-text-v1.5 ships no prompts config, so add the prefixes yourself
docs = model.encode(["search_document: TEI exposes /embed and /rerank endpoints."], normalize_embeddings=True)
query = model.encode(["search_query: which endpoint reranks results?"], normalize_embeddings=True)
print(docs.shape)  # (1, 256)

normalize_embeddings=True is applied after truncation, so the 256-dimension vectors come out unit length.

Serving a self-hosted embedding model: TEI vs Ollama vs sentence-transformers

| | text-embeddings-inference | Ollama | sentence-transformers | |---|---|---|---| | What it is | Rust inference server from Hugging Face | Local model runner, GGUF models | Python library | | Best for | Production embedding APIs | Teams already running Ollama for LLMs | Batch jobs, experiments, custom pipelines | | API | /embed, /rerank, /embed_sparse, OpenAI-compatible endpoint | /api/embed | Your own code | | Models | Safetensors from Hugging Face, many architectures | Ollama library (nomic-embed-text, mxbai-embed-large, bge-m3, qwen3-embedding, embeddinggemma, snowflake-arctic-embed2) | Almost anything on Hugging Face | | Batching | Dynamic, token-based | Basic | Manual | | Rerankers | Yes | No dedicated endpoint | Yes (CrossEncoder) |

text-embeddings-inference (TEI) is the default choice for a self-hosted embedding service. It batches requests dynamically, exposes Prometheus metrics, and has separate images per hardware target. Pin the image tag: cpu-1.9.4 is reproducible, while latest can change the server under you between deploys.

services:
  embeddings:
    image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.4
    command: ["--model-id", "Qwen/Qwen3-Embedding-0.6B", "--revision", "main"]
    ports:
      - "127.0.0.1:8080:80"
    volumes:
      - tei-data:/data

volumes:
  tei-data:

For NVIDIA GPUs use the image for your architecture, for example cuda-1.9.4 or 89-1.9.4 for Ada Lovelace cards, and add a GPU reservation. Replace main with a specific commit hash from the model repository once you have chosen a model, so a model update upstream cannot change your vectors. Query it with:

curl 127.0.0.1:8080/embed \
  -X POST -H 'Content-Type: application/json' \
  -d '{"inputs": ["Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:how do I rotate API keys?"]}'

Ollama is convenient if you already run it for chat models: ollama pull qwen3-embedding:0.6b and call /api/embed, which also accepts a dimensions parameter. Models are GGUF conversions, often quantized, so vectors are not identical to the original Hugging Face weights. Prefixes are your responsibility here too.

sentence-transformers is the library behind most model cards. Use it for offline batch embedding and for evaluation scripts, where you want direct control over prompts, truncation and precision.

The embedding service stack is a ready TEI setup with BAAI/bge-large-en-v1.5 that you can switch to another model by changing the model id.

Re-embedding: a migration plan

Vectors from different models are not comparable, so changing the model means re-embedding everything. Plan for it from day one:

1. Store the source text and metadata outside the vector index, or at least keep it retrievable. If the only copy of a chunk lives inside the vector store, a migration becomes a re-ingestion project.
2. Record the model id, revision, dimensions, prefix and truncation with each collection or table, for example in the collection name (chunksqwen306b_1024) or a metadata table.
3. Build the new index alongside the old one. Embed all chunks into a new collection or column while the old one keeps serving. Throughput is the constraint; estimate it from a timed sample batch on your hardware rather than from published numbers.
4. Evaluate both on the same query set before switching.
5. Switch reads atomically with an alias, a view, or a config flag. Keep writing new documents to both indexes until the switch, or replay them afterwards.
6. Delete the old index once you are confident you will not roll back.

Rerankers: the cheap quality upgrade

A reranker (cross-encoder) reads the query and each candidate chunk together and outputs a relevance score. The usual pattern is to retrieve the top 20 to 50 chunks with embeddings (and keywords, if you run hybrid search), rerank them, and pass the top 3 to 8 to the LLM. It often improves answer quality more than switching to a bigger embedding model, and it can be added without re-embedding anything.

Self-hostable options with permissive licenses include BAAI/bge-reranker-v2-m3 (Apache 2.0, multilingual), Alibaba-NLP/gte-reranker-modernbert-base, mixedbread-ai/mxbai-rerank-large-v2 and the Qwen3-Reranker family (Apache 2.0). TEI serves cross-encoder rerankers such as bge-reranker-v2-m3 on its /rerank endpoint; the LLM-based Qwen3 and mxbai v2 rerankers need a server that supports their architecture, so check before you plan around them. Reranking cost grows with the number of candidates and their length, so cap both.

Next steps

• Deploy a pinned embedding API with the embedding service stack, or browse all embedding stacks.
• Plug it into a full pipeline with the RAG pipeline stack and the guide to self-hosting a RAG pipeline with Docker Compose.
• Choose where the vectors go with the vector database comparison.