The embedding model is the one RAG decision you pay for twice if you get it wrong: once to embed the corpus, and again to re-embed all of it when you switch. Every vector in your database is a fingerprint of the exact model that produced it, so a model swap is a full re-index, not a config change. That is why the number most teams choose on, an MTEB leaderboard average, is close to the least reliable input available. This guide walks the tradeoffs a senior engineer actually weighs: retrieval quality you can trust, dimension against storage and latency, open versus hosted, context limits, language and domain fit, and cost once you are past a demo.
Key takeaways
- A model swap forces a full re-embed of the corpus and a rebuild of every index, so treat the choice as durable infrastructure rather than a tunable parameter.
- Matryoshka-trained models let you truncate dimensions after the fact; OpenAI's own docs show text-embedding-3-large at 256 dimensions still beating the older ada-002 at 1536.
- Vector dimension, not per-token price, is what scales your bill: storage and RAM grow linearly with the number of dimensions you keep.
- MTEB is saturated and partly contaminated; the multilingual rebuild MMTEB spans over 500 tasks across 250+ languages, and its top open model has only 560 million parameters, so rank order shifts on held-out domains.
- Context windows run from Cohere v3's 512 tokens to Embed v4's 128K and Voyage's 32K; match the window to your chunk size before optimizing anything else.
- Open models such as BGE-M3 (MIT, 100+ languages, 8192 tokens) and Qwen3-Embedding-8B (Apache 2.0, ranked first on the MTEB multilingual board at 70.58 in June 2025) close much of the gap when data cannot leave your infrastructure.
Why Is the MTEB Leaderboard the Wrong Place to Start?
Because the number that looks like an answer mostly measures how well a model fits the benchmark, not your data. The Massive Text Embedding Benchmark compresses dozens of tasks into one average, and the public board now lists hundreds of models separated by fractions of a point. Gaps that small at the top are noise, not signal.
Two problems compound. Many embedding models train on data that overlaps MTEB's own splits, so a leaderboard win can reflect memorization of the test shape rather than transfer to your documents. And the benchmark has saturated: when the community rebuilt it as the Massive Multilingual Text Embedding Benchmark (MMTEB) with over 500 tasks across 250+ languages, the strongest publicly available model was multilingual-e5-large-instruct at only 560 million parameters, well below the large proprietary systems that top the English board. Rank order is not stable across distributions.
The practical move is to treat MTEB as a shortlist filter, never a verdict. Pull three or four candidates that score well on the retrieval subset in a language and domain near yours, then measure them on a few hundred labeled query-document pairs drawn from your own corpus. A model that degrades gracefully on your held-out data is worth more than one that tops a public average and cliffs on your jargon. If you have not built that harness yet, a Ragas evaluation on your own corpus gives you recall and precision numbers you can defend in a review.
How Much Does Embedding Dimension Actually Cost You?
Every extra dimension is bytes on disk, bytes in RAM, and cycles per comparison, multiplied by every vector you store. The arithmetic is blunt: a 3072-dimensional float32 vector, which is OpenAI's text-embedding-3-large default output, occupies 3072 times 4 bytes, roughly 12 KB. Ten million chunks is about 120 GB before indexes, and most vector indexes want that resident in memory for low-latency search. The same corpus at 1024 dimensions is near 40 GB.
Matryoshka Representation Learning changes this calculus. Models trained with it pack the most important information into the leading dimensions, so you can truncate the vector afterward and keep most of the quality. OpenAI's documentation notes a text-embedding-3-large embedding shortened to 256 dimensions still outperforms the older ada-002 at 1536. Cohere's Embed v4 exposes 256, 512, 1024 and 1536 output sizes, and Voyage's voyage-3-large supports 2048, 1024, 512 and 256 dimensions with int8 and binary quantization layered on top.
Quantization is the second lever, and it stacks with truncation. Storing each dimension as an 8-bit integer instead of a 32-bit float cuts memory fourfold; binary quantization at one bit per dimension cuts it by 32x and turns cosine similarity into a Hamming distance you can compute on packed integers. A 1024-dimension binary vector is 128 bytes, so those ten million chunks drop from roughly 120 GB to under 1.3 GB. The trade is recall: aggressive quantization needs a rescoring pass over full-precision vectors to recover the true top results, which most vector databases now support natively.
Open Weights or a Hosted API: Which Should You Run?
The deciding factor is rarely quality anymore. It is where your data is allowed to live and how much operational load your team will carry.
| Model | Access | Max dims | Context (tokens) | Notable strength | License |
|---|---|---|---|---|---|
| OpenAI text-embedding-3-large | Hosted API | 3072 (truncatable) | 8192 | Ubiquitous, strong general retrieval | Proprietary |
| Cohere Embed v4 | Hosted API | 1536 (Matryoshka) | 128K | Native text and image in one payload | Proprietary |
| Voyage voyage-3-large | Hosted API | 2048 (down to 256) | 32K | Top retrieval scores, int8 and binary output | Proprietary |
| Qwen3-Embedding-8B | Open weights | 4096 (MRL) | 32K | Ranked first on MTEB multilingual (70.58) | Apache 2.0 |
| BGE-M3 | Open weights | 1024 | 8192 | Dense, sparse and multi-vector in one model | MIT |
A hosted API removes serving entirely: you send text, you get vectors, and the provider owns the GPUs. The cost is data egress and lock-in to a remote model you cannot inspect or freeze at a version. For regulated data, a customer-managed key or an on-prem deployment is often mandatory, which pushes the decision toward open weights.
Open models have closed most of the quality gap. Qwen's Qwen3-Embedding-8B topped the MTEB multilingual leaderboard at 70.58 as of June 2025 under an Apache 2.0 license, and BGE-M3 ships dense, sparse and multi-vector retrieval from a single MIT-licensed model. Running them is a solved problem now: serve the weights with Hugging Face Text Embeddings Inference and you get batched GPU inference behind an HTTP endpoint. The BGE-M3 combination of dense and lexical output pairs well with hybrid search that blends dense and sparse vectors, which recovers the exact-keyword matches pure semantic vectors miss.
Does the Model's Context Window Match How You Chunk?
A model that accepts 512 tokens forces small chunks, and small chunks strip the surrounding context that makes a passage answerable. The window range is wide: Voyage notes its voyage-3-large accepts 32K tokens against OpenAI's 8K and Cohere v3's 512, and Cohere's Embed v4 reaches 128K per its release notes. OpenAI's text-embedding-3 models sit at 8192.
Longer is not automatically better. Averaging a 4000-token document into one vector blurs distinct topics into a muddy centroid that matches everything weakly and nothing precisely. The window sets your ceiling; how you chunk documents for retrieval sets the quality. A large window earns its keep when your natural unit, a full contract clause, a support-ticket thread, or a function with its docstring, genuinely runs past a few hundred tokens and should stay intact.
One detail senior teams check: whether the model is symmetric or expects an instruction prefix. Instruction-tuned embedders such as the E5 and Qwen3 families want a short task prefix on the query side, and sometimes the document side. Skip it and recall quietly drops, which is exactly the kind of thing a leaderboard run does correctly and a rushed integration does not.
Will It Hold Up in Your Language and Domain?
General benchmarks average over exactly the specificity that decides whether retrieval works for you. If your corpus is German case law or Python with inline comments, the relevant question is performance on that slice, not the global mean.
Multilingual coverage is where open models pulled ahead. BGE-M3 supports more than 100 working languages in one 1024-dimensional model, and Qwen3-Embedding covers over 100 languages including code, per Qwen's release. The MMTEB result is the honest signal: the strongest publicly available multilingual model in that study was a 560-million-parameter e5 variant, which means a mid-sized open model can beat a large proprietary one once you leave English.
Domain fit is the other axis. Voyage publishes domain-specialized retrieval numbers across law, finance and code and reports voyage-3-large beating text-embedding-3-large by 9.74% on average across 100 datasets by NDCG@10, a vendor claim worth reproducing on your own data but a real sign that domain tuning moves retrieval. When no off-the-shelf model fits, two cheaper options usually beat training an embedder from scratch: prepend an instruction that names the retrieval task, or add contextual retrieval so each chunk carries document-level context before it is embedded.
Where Does the Money Actually Go at Scale?
Per-token embedding pricing is the number everyone compares and the one that matters least in production. Embedding is close to a one-time cost: a large corpus embedded once is cheap. OpenAI's published list runs about $0.02 per million tokens for text-embedding-3-small and near $0.13 for 3-large, while Voyage lists voyage-3-large around $0.18 and gives the first 200 million tokens free per its launch post. Query-time embedding is a rounding error unless you serve millions of searches a day.
The recurring cost is the vector database, and it scales with dimension and vector count, not tokens. A 3072-dimension index over tens of millions of chunks needs the memory computed earlier, and that RAM is a monthly line item whether or not anyone queries it. This inverts the usual instinct: choosing 3-large over 3-small doubles vector size and can double your storage bill for a quality gain you may never measure, while truncating to 1024 dimensions or quantizing to int8 cuts that bill directly.
The cost no price sheet lists is migration. Because vectors are model-specific, switching embedders means re-embedding the whole corpus, rebuilding every index, and re-validating retrieval quality end to end. On a large or continuously growing corpus that is days of compute and engineering, which is why the model you pick at the prototype stage tends to outlive the decision that chose it. Pick as if you will run it for two years, because you probably will.
How Should You Actually Choose?
Work the constraints in order, hardest first, because each one eliminates options the next step would otherwise waste time on.
- Data residency. If the corpus cannot leave your infrastructure, you are choosing among open weights such as BGE-M3, Qwen3-Embedding and the E5 family, and self-hosting is the baseline rather than an upgrade.
- Language and domain. Filter to models with genuine coverage of your languages and content type, then confirm on your slice instead of the global average.
- Context and chunking. Match the window to your natural document unit so you are not fragmenting coherent passages just to fit the model.
- Dimension and budget. Start at the smallest dimension that holds quality on your eval; Matryoshka truncation and int8 or binary quantization are how you buy scale back without changing models.
- Retrieval quality on your data. Run the two or three survivors against labeled pairs from your corpus and measure recall at the k you actually retrieve.
One caveat outlives all five steps: the embedder sets a ceiling on retrieval, but it need not be the whole system. A weaker, cheaper embedding model paired with a reranker reordering the top results often beats a stronger embedder alone, and it keeps the index small. Decide the embedding model as durable infrastructure, then spend the rest of your quality budget on the parts you can change without a re-index.