Embeddings in Production
Running embeddings in production means choosing a model, batching generation, and handling re-embedding when documents change.
Prerequisites
Overview
Generating one embedding is a single API call. Running embeddings in production means deciding which embedding model to standardize on, batching calls for cost and throughput, and re-embedding content whenever it changes or the model is upgraded.
Where It Fits
New, Changed, or Re-Embedded Content
Re-embedding is triggered by content changes or a model upgrade.
Batched Embedding Calls
Vector Store
Key Points
- Model consistency
- Every embedding in one vector index must come from the same model — mixing models makes similarity scores meaningless.
- Batching
- Embedding calls are typically batched to reduce per-request overhead and cost.
- Re-embedding cost
- Switching embedding models means re-processing the entire existing corpus, which can be a significant, planned cost.
Interview Question
What happens if you switch embedding models but don’t re-embed your existing documents?
Similarity search would compare vectors from two different models, which live in unrelated vector spaces — the distances between them are meaningless, so retrieval quality degrades silently. Every vector in one index needs to come from the same embedding model, which means switching models requires re-embedding the entire existing corpus.
Explain It in 30 Seconds
Running embeddings in production means standardizing on one model per index, batching generation for cost and throughput, and planning for full re-embedding whenever that model changes, since mixed-model vectors aren’t comparable.
Real-World Stack
Technologies commonly used to implement this in production.