Embeddings and Retrieval

Embeddings: coordinates of meaning

Computers compare numbers easily and meanings not at all. An embedding bridges that gap: it is a vector (a fixed-length list of numbers) that a trained model produces for a piece of text, an image or another item, such that items with similar meaning get vectors that lie close together. "VPN drops at the hotel" and "can't stay connected on guest Wi-Fi" share almost no words, yet their embeddings are neighbors.

Think of the numbers as coordinates on a map of meaning. The figure squeezes that map into two dimensions; real embedding models use hundreds to a few thousand dimensions, which is what leaves room for many different kinds of similarity at once.

Nobody hand-codes these positions. An embedding model learns them from enormous amounts of text, by being trained so that texts that appear in similar contexts, or that people mark as matching, get nearby vectors. The idea goes back to word embeddings such as word2vec (Mikolov et al., 2013), which famously placed "king" and "queen" in related positions. Modern embedding models, often built on the same transformer architecture as chat models, embed whole sentences and passages.

Two practical rules. The individual numbers mean nothing on their own; only distances between vectors do. And vectors from different embedding models are not comparable, so a document collection and its questions must be embedded with the same model. Changing the model means re-embedding everything.

Cosine similarity

To find "close" vectors you need a measure. Cosine similarity is the cosine of the angle between two vectors: 1 when they point in exactly the same direction, 0 when they are unrelated (perpendicular), and −1 when they point in opposite directions. It looks only at direction, not length, which suits text: a long article and a short question about the same topic point the same way. The calculation is the dot product (multiply the vectors position by position and add up) divided by the product of the two lengths.

The example uses tiny, hand-made four-dimensional vectors for five knowledge-base articles. To make the numbers readable, we pretend the dimensions mean network, account, hardware and printing. It then ranks the articles for a query about a connection problem.

import math

# Hand-made 4-dimensional "embeddings". Pretend the dimensions mean
# (network, account, hardware, printing). Real embeddings have hundreds
# or thousands of dimensions that have no such readable meaning.
articles = {
    "KB-0217 VPN error 809 on guest networks": [0.9, 0.2, 0.1, 0.0],
    "KB-0301 Reset your password":             [0.1, 0.9, 0.0, 0.0],
    "KB-0388 Laptop does not charge":          [0.0, 0.1, 0.9, 0.1],
    "KB-0413 Printer shows offline":           [0.4, 0.0, 0.3, 0.9],
    "KB-0450 Wi-Fi drops in meeting rooms":    [0.9, 0.0, 0.2, 0.0],
}


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    length_a = math.sqrt(sum(x * x for x in a))
    length_b = math.sqrt(sum(y * y for y in b))
    return dot / (length_a * length_b)


# "I can't connect from the hotel" would land close to the network axis.
query = [0.8, 0.3, 0.1, 0.0]

ranked = sorted(articles.items(), key=lambda item: cosine(query, item[1]),
                reverse=True)
for title, vector in ranked:
    print(f"{cosine(query, vector):.3f}  {title}")

0.990  KB-0217 VPN error 809 on guest networks
0.933  KB-0450 Wi-Fi drops in meeting rooms
0.449  KB-0301 Reset your password
0.395  KB-0413 Printer shows offline
0.153  KB-0388 Laptop does not charge

The two network articles score far above the rest, even though the query vector does not match either exactly. That is the whole trick of semantic search. Many embedding models return vectors scaled to length 1; for those, cosine similarity equals the plain dot product, which is cheaper to compute. In real projects a library such as NumPy, or the vector database itself, does this arithmetic for millions of vectors at once.

Semantic search versus keyword search

Classic keyword search (also called lexical or full-text search) finds documents that contain the words of the query, ranked by how often and how distinctively those words appear (BM25 is the common ranking formula). Semantic search finds documents whose meaning is close to the meaning of the query, by embedding both and comparing the vectors.

Because their strengths complement each other, production systems often use hybrid search: run both, merge the result lists, and optionally let a second model (a re-ranker) reorder the top results by relevance. On a service desk, where questions mix error codes with everyday descriptions, hybrid search is usually the better default.

Vector databases

Comparing a query with every stored vector works for five articles, not for ten million chunks. A vector database stores embeddings together with the original text and metadata, and builds an index that finds the nearest vectors to a query quickly. Those indexes use approximate nearest neighbor (ANN) algorithms, such as HNSW (hierarchical navigable small world graphs), which trade a little accuracy (now and then a true neighbor is missed) for answers in milliseconds.

What a vector store adds besides nearest-neighbor search:

• Metadata filters: "only articles for product X", "only documents updated after January", "only Dutch".
• Access control: return only chunks the asking user is allowed to read. Without this, a RAG assistant happily quotes HR documents to anyone who asks the right question.
• Updates and deletes, so the index follows the source documents.

It does not have to be a separate product. Examples as of 2026: the pgvector extension adds vector columns and indexes to PostgreSQL; search engines such as Azure AI Search and OpenSearch support vector and hybrid search alongside keyword search; and dedicated vector databases such as Pinecone are offered as a managed service. If your data already lives in one of these, that is usually the simplest place to add vectors. For general database background, see the Databases & SQL topic.

Chunking

A 40-page manual embedded as one vector becomes a blurry average of everything in it, and it is too long to put into a prompt anyway. Chunking splits documents into smaller passages that are embedded, stored and retrieved separately. Getting it right matters more than most people expect:

• Size. Chunks of a few hundred tokens (roughly a few paragraphs) are a common starting point. Too small and a chunk loses its context ("Click Save." – save what?); too large and it mixes topics and wastes prompt space.
• Overlap. Letting consecutive chunks share some text, for example 10–20%, prevents an answer from being cut in half at a boundary.
• Structure. Split at headings, list items or table rows rather than at a fixed character count, and copy the document title and section heading into each chunk so it still makes sense on its own.
• Metadata. Store the source, section, date and permissions with every chunk, so answers can cite and filter.

There is no universally best setting. Test a few on your own documents and questions, and measure with the method in Evaluating retrieval.

Retrieval-augmented generation

A language model knows what was in its training data, up to a cutoff date, and nothing about your internal knowledge base. Retrieval-augmented generation (RAG) fixes that at question time: the system first retrieves the passages most relevant to the question from your own documents, puts them in the prompt, and asks the model to answer from those passages. The name comes from Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Facebook AI, NeurIPS 2020), one of the papers that established the approach.

RAG became the standard way to build "chat with your documents" systems for practical reasons. Knowledge stays in documents you already maintain, so an updated knowledge article is used the moment it is re-indexed, with no retraining. Answers can cite their sources, so people can check them. Access control can be applied at retrieval time. And it is far cheaper than fine-tuning a model, which teaches style and format well but is a poor way to add facts that change.

A mini RAG without a language model

Everything in RAG except the final step is ordinary programming. The example below retrieves the two best chunks for a question and assembles the exact prompt that would be sent to the model. To stay short it ranks by shared words (a crude keyword score); a real system would rank by cosine similarity of embeddings, as in the first example, or by both.

import re

chunks = {
    "KB-0217#1": "VPN error 809 means the network blocks VPN traffic. "
                 "Hotel and guest networks often do this.",
    "KB-0217#2": "Fix for error 809: connect through a phone hotspot, or ask "
                 "the hotel to allow VPN traffic.",
    "KB-0301#1": "To reset your password, open the self-service portal and "
                 "choose Forgot password.",
    "KB-0450#1": "Wi-Fi drops in meeting rooms are usually caused by too many "
                 "devices on one access point.",
}
STOP = {"the", "a", "an", "and", "or", "to", "of", "on", "in", "is", "are",
        "my", "your", "i", "it", "what", "do", "this", "through", "by"}


def words(text):
    return {w for w in re.findall(r"[a-z0-9-]+", text.lower()) if w not in STOP}


question = "My VPN gives error 809 in the hotel. What do I do?"
q = words(question)
scores = {cid: len(q & words(text)) for cid, text in chunks.items()}
top = sorted(scores, key=lambda cid: (-scores[cid], cid))[:2]

for cid in sorted(scores, key=lambda cid: (-scores[cid], cid)):
    print(f"{scores[cid]} shared words  {cid}")

sources = "\n".join(f"[{cid}] {chunks[cid]}" for cid in top)
prompt = (
    "Answer the question using only the sources below. Cite the source id "
    "in brackets. If the sources do not contain the answer, say so.\n\n"
    f"Sources:\n{sources}\n\nQuestion: {question}"
)
print()
print(prompt)

4 shared words  KB-0217#1
4 shared words  KB-0217#2
0 shared words  KB-0301#1
0 shared words  KB-0450#1

Answer the question using only the sources below. Cite the source id in brackets. If the sources do not contain the answer, say so.

Sources:
[KB-0217#1] VPN error 809 means the network blocks VPN traffic. Hotel and guest networks often do this.
[KB-0217#2] Fix for error 809: connect through a phone hotspot, or ask the hotel to allow VPN traffic.

Question: My VPN gives error 809 in the hotel. What do I do?

Note the three instructions in the prompt: answer only from the sources, cite them, and admit when the answer is not there. They are what turns "a model with some extra text" into grounded generation. Try the question "No internet in my hotel room": it shares only the word "hotel" with the chunks, which is exactly where embeddings beat word overlap.

Grounded generation and its limits

Grounded generation means producing an answer that is based on, and traceable to, specific supplied sources rather than the model's general training, ideally with a citation for each claim. RAG is the most common way to achieve it. Grounding is the main defense against hallucination, but it reduces the problem; it does not remove it. Things that still go wrong:

• Retrieval misses. The right chunk is not among the top results, and the model answers from the wrong ones or from its own general knowledge.
• The model ignores or misreads the sources. It blends them with what it "knows", merges two chunks into a claim neither makes, or cites a source that does not support the sentence.
• The sources are wrong or outdated. RAG faithfully repeats an obsolete knowledge article. Grounding moves the quality problem to your documents, which is why knowledge management matters more, not less (see Knowledge Management).
• The sources contain instructions. A retrieved document or email can hide text such as "ignore previous instructions". This is indirect prompt injection, covered in AI Security; treat retrieved text as data, never as commands.

Evaluating retrieval

A RAG system has two parts that fail in different ways, so test them separately. First the retrieval: did we fetch the right chunks? Build a small test set of real questions, each with the chunk or article that answers it (service desk analysts can write these in an afternoon). Then measure:

• Hit rate (recall at k): for what share of questions is a correct chunk among the top k results? If the answer is in the top 5 for only 60% of questions, the model cannot give a grounded, correct answer for more than about 60% of them.
• Rank: how high does the correct chunk appear? (Mean reciprocal rank, MRR, averages 1/position.)
• Precision of the context: how much of what was retrieved is actually relevant? Irrelevant chunks cost tokens and can distract the model.

Only then evaluate generation: is the answer correct, supported by the cited sources, and complete? That part usually needs human review of a sample, sometimes assisted by a second model as a judge. Rerun the whole test set whenever you change the chunking, the embedding model, the prompt or the documents; the general approach is the same as in Evaluating Models.