Vector search · A beginner's guide

How machines learn to search by meaning

A connected walk from the problem with keyword search all the way to how production systems compress a billion vectors into memory — each idea building on the last.

Search is a solved problem — until you try to search for what something means rather than what it literally says. This post walks through why keyword search breaks, what embeddings are and why they work, how production systems find the right vector among a billion of them in milliseconds, and how a language model gets connected to all of it so it can reason over data it was never trained on. Each idea is motivated by the problem the one before it left unsolved — so by the end, the whole stack makes sense as a chain of decisions rather than a list of components to memorise.

Start with a broken search

Imagine you're building a recipe app. A user types: “something light for a hot day.” Your search engine returns nothing. Not because the recipes don't exist — you have gazpacho, salads, smoothies — but because none of them literally contain those words in their title.

This is keyword search's core failure. It matches strings, not intent. It finds what you typed, not what you meant. And for most real-world queries — natural language, synonyms, paraphrases, multilingual input — what you typed and what you meant are not the same thing.

Vector search asks a different question: what if, instead of comparing words, we compared meaning?

To compare meaning, you first need a way to represent it numerically — something a computer can actually work with. That's what an embedding is.

Meaning as a point in space

An embedding is a list of numbers — typically hundreds or thousands of them — that represents a piece of content. Not the words themselves, but the meaning behind them. These numbers come from a machine learning model trained on massive amounts of text. Through that training, the model learns to place similar things near each other in this numerical space.

The model doesn't store words — it stores positions in a high-dimensional space, and position encodes meaning. “The dog chased the ball” lands near “the puppy ran after the toy” even though they share zero words. Searching by meaning becomes searching by proximity.

Two families of vectors exist, and each does something the other can't.

Dense vectors capture meaning. But there's a whole class of queries where meaning isn't enough — where the exact word matters. That's where sparse vectors come in.

Two kinds of vectors, two kinds of knowledge

A dense vector has a value in every one of its dimensions — hundreds of numbers, all contributing to a holistic representation of meaning. When you embed “something light for a hot day,” every dimension shifts to encode the composite of lightness, warmth, and food context. This is what lets it find gazpacho.

A sparse vector works differently. It's mostly zeros. Only the dimensions that correspond to actual tokens in the text get non-zero values, weighted by how important each token is. For a sentence like “NullPointerException at line 42,” a sparse vector preserves the exact string. A dense vector might round it into a vague neighbourhood of “software errors” — losing the specificity you need.

Neither is strictly better. Dense wins on meaning; sparse wins on exactness. In production you use both. Running them simultaneously and merging results — called hybrid search — is the standard approach. The fusion algorithm most commonly used is RRF (Reciprocal Rank Fusion): each document's final score comes from how well it ranks in each list, so documents that do well in both float to the top naturally.

Now we have vectors that represent meaning. The next problem is scale. How do you find the most similar vector among a hundred million of them — fast?

The search problem at scale

Finding the closest vector to a query seems straightforward: compare the query to every stored vector, rank by similarity, return the top results. This works at ten thousand vectors. At ten million, it's too slow. At a billion, it's completely impractical.

The solution is an index — a data structure that lets you skip most comparisons and still find the right answer. The dominant algorithm in modern vector databases is HNSW: Hierarchical Navigable Small World.

The intuition: imagine a multi-floor building. The top floor has sparse, long-range connections — highways between distant neighbourhoods. Lower floors have denser, local connections. When a query comes in, you enter at the top floor and walk greedily toward the query vector, using the highways to cover distance quickly. Then you descend to lower floors and close in on the exact neighbourhood. The result is approximate — you might miss the absolute nearest neighbour — but the miss is controlled, and you do it in logarithmic time instead of linear.

This handles the speed problem. But speed alone isn't enough — real search always has conditions. Not just “find the most similar vector,” but “find the most similar vector that also matches these filters.”

Most vector databases handle this badly. They either filter first (scan the entire filter, then search — destroying recall when the filter is selective) or filter after (search broadly, then discard non-matching results — wasting the work). The better approach is to weave the filter into the HNSW traversal itself, evaluating it as you walk the graph. One pass, no recall penalty. This is the technical detail that separates production-grade vector databases from toy implementations.

The index handles millions of queries at low latency. But standard dense retrieval compresses each document into a single vector, and that compression loses detail. For high-stakes retrieval, you need something finer-grained.

Token-level precision: how ColBERT works

Standard dense search embeds an entire document into one vector. That single vector averages out the meaning of every sentence, paragraph, and phrase — the specific detail that makes a document relevant to a particular query often gets washed out.

ColBERT takes a different approach: every token in the document gets its own embedding. At query time, for each query token, you find its maximum similarity to any token in the document. You sum these per-token maxima across all query tokens. The score is called MaxSim.

The practical pattern: use HNSW to retrieve the top 100 candidates cheaply, then rerank them with ColBERT for precision. You get the speed of approximate search and the accuracy of token-level comparison.

At this point you have a complete retrieval engine: vectors that encode meaning, an index that finds them fast, filters that narrow by metadata, and reranking that sharpens precision. The natural question is: what do you do with all this retrieved content? That's where LLMs enter.

Giving a language model memory

Large language models are powerful, but they have a hard boundary: they only know what they were trained on, and their training has a cutoff date. They don't know your company's internal docs, your product catalogue, your customers' conversation history, or anything that happened last week.

RAG — Retrieval Augmented Generation — solves this by connecting a language model to a vector database. Before generating a response, you retrieve the relevant pieces of your data and include them in the context window. The model doesn't need to memorise your data; it just needs to reason over what you hand it.

There's a step in this pipeline that most tutorials skip but that determines retrieval quality more than any other: chunking. Before you can embed a document, you have to split it into pieces. Split too small and you lose context. Split too large and the embedding averages out too much detail. Split at the wrong boundaries and you'll cleave a key sentence in half — and the chunk will never retrieve correctly, no matter how good your embedding model is.

The right chunking strategy depends on your content: fixed-size chunks with overlap for dense prose, sentence-level splits for conversational content, header-based splits for structured documents. Chunking is the highest-leverage decision in a RAG system, and it gets the least attention.

The pipeline works. Now imagine it at production scale — a billion vectors. A 1536-dimensional float32 vector takes 6KB. A billion of them: 6TB of RAM. Something has to give. That's what quantization is for.

Compressing vectors without losing what matters

Quantization reduces the memory footprint of stored vectors by representing each value with fewer bits. The tradeoff is recall: you trade some precision for dramatically smaller indices that fit in memory and search faster.

Binary quantization deserves special mention. Rather than storing each dimension as a float, you store a single bit: 1 if positive, 0 if negative. Distance is computed using XOR and popcount operations, which modern CPUs execute extremely fast. The catch: you retrieve 5x more candidates than you need using binary search, then rescore the top candidates with full-precision vectors to recover recall. Two-stage retrieval, extreme memory savings.

TurboQuant handles the centering problem by rotating the vector space before compression. A Hadamard transform normalises the distribution regardless of how the original embedding model was trained. At 4-bit precision, it delivers recall close to scalar quantization at twice the compression.

The full stack, assembled

Each layer exists because of a specific limitation in the layer before it. This is the shape of a production semantic search or RAG system — not a list of components, but a chain of solutions.

You don't need all of this on day one. Start with dense retrieval. Add sparse when exact terms matter. Add reranking when precision matters more than latency. Add quantization when memory matters more than simplicity.

What looks like a complex system is really just a sequence of problems and the solutions they forced into existence. The keyword search failure demanded embeddings. The scale of embeddings demanded an index. The single-vector compression loss demanded token-level reranking. The memory cost of billions of vectors demanded quantization. Follow the chain of problems and you understand why every piece is there.

— shreyas