Picture asking a search engine way back in 2010 for “cheap mobile phone”, and it basically just looks for those exact words, with zero imagination. Then you ask the same again, now in 2026, and it not only grabs stuff that says “budget smartphone” or “affordable Android”, but also works to even find pages like “phone under $200” where those terms don’t show up anywhere! But you see, nobody explicitly trained it to treat those as “synonyms” or anything like that. It somehow figures out the similarity, sort of, from geometry and such.
Vector embeddings allow modern search engines to move far beyond keyword matching. This matters a whole lot more than people usually think, because it’s not confined to only search. The quiet plumbing under recommendations, chatbots that answer using your company documents, and most of what people have come to know as ‘AI understanding‘ these days is this.
What is an embedding?
Taking off the fancy words, an embedding is simply a list of numbers. Just like a neural network, it takes the input, whether it be a word, a sentence, a photo, or a whole document, and outputs a vector against it (usually somewhere around 256 up to 3,072 numbers, give or take). That vector is the position of your input in some math space.
The important part is how that space is learned. During training, the model gets better at placing “similar meaning” closer together, while “different meaning” ends up farther apart. I am not talking about similar spelling here; I mean similar meaning.
- “Dog” and “puppy” land near each other.
- “Dog” and “canine” also land near each other, even though they share absolutely no letters.
- Meanwhile, “Dog” and “stock market” are far apart, not close at all.
Complexity of Meaning
To decide what “close” even means, these systems use a few common measures, such as cosine similarity, which is the angle between vectors, dot product, or Euclidean distance. Cosine similarity is super common because it focuses on direction and not magnitude, which helps when comparing a short query vector with a long document vector.
The Three Jobs Vector Embeddings Do
(Same underlying idea but with different outcomes.)
- Semantic search: Instead of keyword matching, you embed the user query and embed every document in the index with it. Then you pull back the documents whose vectors are closest to the query’s vector. This way, search gets less brittle with typos, synonyms, and vague phrasing better than old-school keyword matching ever did.
- Recommendations: This is the same embedding process used for products, music, and articles. If your activity tends to land near certain points in the vector space, the system suggests other items that are nearby. Using the same geometry, the search gets shifted into the territory of “taste”.
- Retrieval-Augmented Generation (RAG): This is where embeddings get tangled up with LLMs. Instead of assuming the LLM will magically know everything (of course it doesn’t; it’s frozen at some cutoff, and it won’t automatically come to know of your internal docs), a RAG system:
- Breaks the documents into smaller pieces
- Embeds each chunk and stores the vectors in a vector database
- Embeds the user question the same way
- Grabs the most similar chunks
- Sends those chunks into the LLM as context right before it can answer
In other words, the LLM is not remembering your documents, because it is not capable of it. It is getting the most relevant fragments at runtime; like a lookup, right before it talks.
The Range of Reliability
Most guides skip this, but I’ll include this portion, as for real production systems, this is the big stuff.
Rule one (not optional): the model used to embed your documents has to be the exact same model used to embed the query. If you use different embedding models, you end up with vectors in completely incompatible spaces, giving out basically meaningless results.
And understand that this rule implies something painful. If you ever switch embedding models later, you can’t keep your old vector database. You need to re-embed everything, from the beginning. Everything.
After that, there are three common failure modes that explain most “why is my RAG hallucinating?” tickets.
- Bad chunking: If you split at a weird boundary, you can end up detaching a sentence from its context. You then get a chunk that may appear to be technically relevant, in our vector sense, but is useless in practice.
- Stale indexes: Your docs change. But vector databases don’t magically refresh themselves. So you can end up retrieving old facts. It can look fine until someone checks.
- Dimensionality mismatches: Some embedding models output variable-length vectors or different fixed sizes. If you compare a 1536-dimension vector against a 768-dimension vector, it just won’t line up. You can’t round it off either.
Also, here’s a heads-up! A bad embedding model often fails without an announced error. It retrieves plausible but wrong chunks, and then the LLM stitches together a wrong answer… which we are oh so familiar with as a hallucination, that gets blamed on the LLM, whereas the real culprit was the embedding layer.
Choosing an Embedding Model Without Regretting It Later
The embedding model ecosystem has definitely moved around recently. Open-source models now match or beat closed APIs on raw retrieval benchmarks like MTEB, and that shift wasn’t really happening even about a year and a half ago.
| Model | Notable Trait | Context Window / Dims |
| BGE-M3 | Self-hosted, multilingual production model | 8,192 tokens / 1,024-d |
| OpenAI text-embedding-3-large | Strong general performance, easiest to integrate | 8,191 tokens / 3,072-d |
| Gemini Embedding | Multimodal support across extended token ranges | Up to 32,768 tokens / 3,072-d |
| Qwen3-Embedding-8B | Leads MTEB retrieval benchmarks | 32,768 tokens / 4,096-d |
Most RAG setups find that 768 to 1024 dimensions land the sweet spot between precision and cost, and going bigger usually doesn’t pay off proportionally. There’s a trick though that can really shave the bill down.
The Trick That Cuts Storage Costs 4x
One recent thing people should actually know is Matryoshka Representation Learning, named after those nesting dolls (because the idea is you teach the model to stuff the most useful semantic info right up at the beginning), like in the first dimensions of the vector.
Which in practice means you can take a 1024-d embedding, and truncate it down to 256 dimensions, and might lose just 2–3% retrieval precision. Meanwhile, your storage and search costs drop by 4x! For teams doing vector search at real scale, this is much more than a tiny optimization. This is a legitimate infrastructure choice.
The Real Takeaway
“Here’s a list of 1024 numbers” will not look great on a product slide. So, embeddings are the part of an AI system nobody demos. But they’re doing a lot more of the actual thinking than most teams admit.
If you get the embedding layer right, make sure the model choice matches what you trained with, use sensible chunking, build a fresh index, then search, recommendations, and RAG start behaving as they should. Mess it up, and no amount of prompt tinkering on the LLM side can save you, because the system already got the wrong context in the first place.