Embeddings
Numerical vector representations of text, images or other data that capture meaning, so similar content ends up close together in vector space.
An embedding converts a piece of text, an image or other data into a fixed-length vector of floating-point numbers, typically a few hundred to a few thousand dimensions. Inputs with similar meaning produce vectors that sit close together; unrelated inputs end up far apart. This property makes embeddings the foundation of semantic search, retrieval-augmented generation (RAG), clustering, classification and recommendation systems. Many current embedding models are multimodal and place images, audio or video in the same vector space as text, so a text query can find a matching image or video clip.
How embedding models work
Embedding models are typically transformer encoders (like BERT) or models derived from LLMs, trained contrastively on pairs of similar and dissimilar examples. They map variable-length input to a fixed-length vector, usually by mean-pooling the token representations or taking a special token's output. Many recent models are trained with Matryoshka representation learning, so you can keep only the first part of each vector (for example 256 of 768 dimensions) to save storage, at some cost in quality.
Common embedding models
- OpenAI text-embedding-3-large: 3,072 dimensions, $0.13 per million tokens
- OpenAI text-embedding-3-small: 1,536 dimensions, $0.02 per million tokens, often sufficient for RAG. Both OpenAI models can be shortened with the
dimensionsparameter. - Google Gemini Embedding 2: Multimodal (text, images, video, audio and PDFs), 3,072 dimensions by default with 128 to 3,072 supported, 8,192-token input limit, $0.20 per million text tokens
- Cohere Embed 5: Pro and Fast tiers that share one embedding space, so a corpus indexed with one can be queried with the other; text and image input, up to 2,048 dimensions, 128K-token context
- Voyage AI voyage-4 series: voyage-4-large, voyage-4, voyage-4-lite and voyage-code-4, with a 32K-token context and 1,024 dimensions by default; voyage-4-nano has open weights
- EmbeddingGemma 2 (Google, October 2026): Open (Apache 2.0) 740M-parameter model that embeds text, images, audio and video into one 768-dimension space, with an 8K-token context
- BGE (BAAI) and Sentence-Transformers: Open-source options for self-hosting; the Sentence-Transformers library's all-MiniLM-L6-v2 is a small, fast baseline
Choosing an embedding model
The MTEB (Massive Text Embedding Benchmark) leaderboard on Hugging Face is the standard comparison; for RAG, look at the retrieval scores rather than the overall average. Match the model's input limit to your chunk size: older models with 512-token limits will cut off longer chunks. Weigh cost, latency and dimension count (which drives vector storage cost) against quality; smaller models like text-embedding-3-small hold up well in many production RAG systems. Queries and documents must be embedded with the same model, so switching models means re-embedding the whole corpus.
Embeddings vs LLM knowledge
Embeddings capture semantic similarity through learned vector representations. They lack the broad knowledge and reasoning abilities of large language models. RAG systems combine embeddings for fast, approximate retrieval with LLMs for comprehension and generation: the embedding search finds candidate chunks, and the LLM reads and reasons over them.
Related terms
Models relevant to Embeddings
Cohere Parse 5
Cost-effective document parsing at enterprise scale
View model →Claude Haiku 5.5
Anthropic's fastest, lowest-cost current model at $0.10 per million input tokens with a 1M context window
View model →GPT-6 Luna
OpenAI's cheapest GPT-6 model, and since October 2026 the default for free ChatGPT users
View model →