Retrieval Augmented Generation: Architecture und Design-Decisions

LLMs are incredibly useful tools that bring a lot of world knowledge with them. Knowledge that was learned during the training of the LLM. As a result, they not only master programming languages, but also bring along a lot of knowledge about frameworks and libraries. Benchmarks like Humanity's Last Exam demonstrate that this increasingly includes very specialized niche knowledge as well. When I have a problem such as how to clean plaster residue off my 3D print, I ask Claude and get an answer that works.

Why classical LLMs are not enough

But when it comes to questions about company processes and policies, or whether I am allowed to bring my dog into the office, they cannot provide any information on their own, since that data is usually not public and therefore not part of the training set. Here we have to give the LLM a way to look up knowledge and to improve the generated answers with this external knowledge. The technical term for this is Retrieval Augmented Generation, RAG for short.

Technically speaking, every incorporation of external information is covered by the term RAG -- be it a web search or the reading of skill markdown files. But usually RAG means searching through large amounts of data in order to answer a question. The classic case is a collection of documentation of internal processes. For example a Confluence dump or a SharePoint folder with a great many PDF files. Sometimes it is just a single document that is too long to fit into an LLM's context all at once, and from which one would like to extract relevant information.

Before we get into the details, we should get an overview of the overall process. A RAG system splits into two phases that run at separate times. One of them is the offline phase, which takes care of the indexing of new documents only when such documents are added. In the diagram, on the upper left. The other is the online phase, which is run through every time a question is asked, marked by green lines.

The steps that are carried out in each of the phases are what we will look at in the following sections, showing a diagram each time in which the step we are currently talking about is highlighted. Like the red dots on public maps with the "You are here" label.

The following sections work through exactly this path: first, offline, the indexing, consisting of chunking and embedding, that is, splitting documents into pieces and preparing them for storage in a vector database. Then, online, query construction, search in the vector database, reranking and generation. If you would like to see a compact implementation of this pipeline, you will find it in the series GenAI für Full Stack EntwicklerInnen starting with part 2 (in German), using TypeScript and Ollama.

Document preparation and chunking

In general, in this article we will concentrate on purely text-based RAG, without going into how we extract that text from PDF files, for example. PDFs in particular have their own challenges, which one could write a separate blog article about. That starts with reliable text extraction and does not end with embedded images by a long shot. And while we are on the subject of images -- there are interesting approaches for those as well, such as using direct image embeddings (for example with models like CLIP), which we will likewise not go into here.

Why chunking is necessary

There are two mutually independent reasons for splitting long documents into small pieces, the so-called chunks. One concerns retrieval: as we will see shortly, an embedding model translates a piece of text into one vector. If you squeeze a whole document into it, its diverse content turns into a single, washed-out average. Accordingly, this then also only matches a specific question moderately well. On top of that, embedding models have a maximum input length and simply have to discard everything beyond it. Smaller chunks therefore mean a finer granularity with which the search can hit exactly the right spot.

The other reason concerns the later generation: for the LLM it is best if exactly the required information is delivered, so that the context is not filled unnecessarily with irrelevant text. This irrelevant information is not only unnecessary tokens that cost money, but also a distraction for the LLM that can degrade the quality of the generated text.

Chunking strategies for different document types

How small these chunks should be is a trade-off: they should be long enough for the context to be clear, but short enough that not too much irrelevant information fills the LLM context. As a rule of thumb for prose, a chunk should cover about one paragraph. For concrete parameter suggestions I can recommend the article "Retrieval Augmented Generation" (in German) by my colleagues Daniel Töws and Daniel Ladischenski.

A related parameter is the overlap: instead of cutting the document cleanly at fixed boundaries, you let consecutive chunks overlap a little. This prevents a sentence that unluckily falls exactly on a chunk boundary from being torn apart and no longer being understandable in either of the two chunks. Overlaps of about 10 to 20 percent of the chunk size are common. The usual trade-off applies here as well: too much overlap bloats the index and produces redundant hits in the search. Too little risks context being lost at the seams.

Unfortunately, not every text is neatly structured into paragraphs. Often there are tables and lists. In some applications we want to enrich the LLM with knowledge from YAML or JSON files. And my personal final boss are Excel files, which contain very worthwhile information but can only be split into chunks understandable to the LLM with extreme difficulty in a generic way.

Here we will present a selection of chunking methods that are meant to convey a rough feeling for the topic and may perhaps serve as inspiration for your specific problem.

Recursive Character Splitter

This is (apart from "just split every X characters") one of the simplest splitters. The parameters for such a splitter are the maximum length of the resulting chunks in characters and a hierarchical list of characters at which the text is to be separated.

For the example of a markdown text, one could imagine that a simple list looks like this: ["---", "#", "\n\n"]. In markdown, --- produces a horizontal line that is often used to separate sections. # introduces headings (and to keep this example simple, we forgo distinguishing between different heading levels such as ## here). Last comes the weakest indicator, \n\n: a blank line that separates paragraphs.

The splitter takes the entire document as the first chunk candidate. It recognizes that this chunk is too long and splits it at every ---. Let us say that in our example we now have 3 chunk candidates. The first one is already below the character limit, so this chunk is finished. The other two are now split further recursively. First at headings, at #. If one of the remaining chunks is still above the character limit, it is now split recursively into paragraphs at \n\n.

The advantage for a format like markdown is obvious: we can split into semantically meaningful chunks and at the same time make sure that no chunk stays too large. But this kind of splitting reaches its limits very quickly with some formats. Imagine a table with column headers. You could now split at the line ends, but then only the first chunk would have the column headers and all the others would be very hard to impossible for the LLM to interpret on their own.

Format-specific splitters

For such cases we therefore need strategies that preserve the structure of the source document.

Here I would like to show a few possible strategies using JSON as an example. Let us look at this example JSON.

1{
2    "Ducks": [
3        {
4            "name": "Scrooge",
5            "money_bin": true
6        },
7        {
8            "name": "Donald"
9        }
10    ],
11    "Geese": [
12        {
13            "name": "Gladstone"
14        }
15    ]
16}

Obviously, a chunk "name": "Donald" } ], "Geese" would not only be useless, but directly harmful. So we have to make sure that the individual chunks do not discard any relevant information. We could therefore imagine splitting larger JSON objects into several individual objects:

1{
2    "Ducks": [
3        {
4            "name": "Scrooge",
5            "money_bin": true
6        },
7        {
8            "name": "Donald"
9        }
10    ]
11}

1{
2    "Geese": [
3        {
4            "name": "Gladstone"
5        }
6    ]
7}

Another approach is flattening. You convert the JSON structure directly into flat key-value pairs, some of which can then be combined into one chunk.

1Ducks[0].name: Scrooge
2Ducks[0].money_bin: true
3Ducks[1].name: Donald
4Geese[0].name: Gladstone

If you know the structure of the JSON beforehand, you can also replace the index entries directly with a relevant key here in order to have better context. As a rule of thumb: the better a human understands a chunk, the better both the embedding model and the LLM understand the chunk.

1Ducks.Scrooge.name: Scrooge
2Ducks.Scrooge.money_bin: true

And for particularly large and deep structures, a combination of both is conceivable: similar to the Recursive Character Splitter, we descend into the JSON tree and check at every point whether the subtree is small enough to fit into a chunk. As soon as that is the case, we write the whole subtree as the value of the flattened key:

1Ducks: [
2    { "name": "Scrooge", "money_bin": true },
3    { "name": "Donald" }
4]

1Geese: [
2    { "name": "Gladstone" }
3]

The problem with tables and Excel

However, you always have to be very careful that the chunks are appropriate for the use case. Suppose we have an Excel table with column headers. We can split it into meaningful chunks by always processing a few rows together with the column headers into one chunk. That way every single number can be interpreted well by the LLM.

But a typical question to an Excel table would be something like "What is the sum of column X?". Well, in that case the LLM only has a chance of giving a correct answer if all the chunks belonging to the table are delivered. For such cases, this form of text-based RAG is not the right solution. (Such advanced evaluations could perhaps be solved with a tool/MCP server that translates the user's request via an LLM into a Python script or an SQL statement and evaluates it against the data. That would also fall under the term RAG, but is not the topic of this article.)

LLM-assisted semantic chunking

For manageable amounts of data where a high answer quality is important, it can also make sense to have the chunks determined by an LLM (or manually). After all, an LLM can judge the context that meaningfully belongs together in a chunk better than the mere syntactic analysis of the methods mentioned before. This is especially the case when the syntactic markers do not exist, which can often be the case with texts extracted from PDF or via OCR.

Having every cut made by an LLM is, however, slow and expensive for large text collections. A middle ground is embedding-based semantic chunking. You embed the individual sentences with an embedding model (which we will look at in more detail in the next section) and always cut where the distance between consecutive sentence vectors becomes large -- that is, where the topic jumps. That is cheaper than an LLM at every cut and better than pure syntax for certain text collections, but not for others.

Retrieval: vector search and hybrid search

Now we have our chunks, which contain the information with meaningful context. But how do we give the LLM the right chunk when the user asks a question?

So if we have chunked our JSON from the example above and the user asks "Who is the richest duck?", how does our system know which chunk is relevant and should be delivered to the LLM?

Embeddings: text as a vector representation

The established standard tool is to convert the chunks into a vector representation using an embedding model. This vector then points to a place in a high-dimensional vector space that corresponds to a semantic meaning. That is very abstract. We can make it vivid with a simplified model. We imagine a diagram with 2 dimensions into which we sort our chunks. One dimension shows how much it is about a duck, and the other how much it is about a goose. Our embedding would therefore probably give our two example JSONs from above vectors close to $(1, 0)$ and $(0, 1)$: the first chunk contains the word "Ducks", so semantically it has more to do with ducks. The second one correspondingly more with geese. We remember these vectors together with the chunk itself.

Now if the LLM is asked "Who is the richest duck?", this question is embedded and should have a vector that points strongly in the duck direction. We take this vector and, out of all the vectors our chunks have been given, we look for the nearest one (or the $k$ nearest ones). So we find the vector that belongs to the first JSON snippet. So we deliver this snippet to the LLM, which will derive from the "money_bin" field that Scrooge is the richest duck and not Donald.

If we imagine that we have yet another dimension that indicates how much it is about "wealth", the chunk containing "money_bin": true would lie further in this wealth direction than the others. So that, even if we have more chunks that are about ducks, we still find the chunk that is about Scrooge with the money bin as the most relevant one. So we find the best-fitting vector across many dimensions.

Real embeddings do not live in two or three, but in hundreds to thousands of dimensions, and their axes unfortunately do not correspond to neatly nameable concepts like "duck" or "goose" -- the meaning is distributed over many dimensions and not directly interpretable for us.

The decisive property is that the embedding ensures that two texts with a similar meaning get vectors that are close to each other in the vector space. Synonyms therefore pose no problem, and even other languages work very well (with multilingual embedding models).

Incidentally, that also means that the chunks and the question have to be embedded with the same embedding model, otherwise they end up in incomparable vector spaces and the search returns nonsense. Anyone who changes the embedding model has to re-embed the entire stock. The choice of the model itself is therefore a central design decision: factors such as the dimensionality of the vectors (more dimensions cost memory and search time), multilingualism, the maximum input length and, not least, how well the model fits your own domain shape the quality of the entire system. A strategy for migrating the data stock to a different embedding model should therefore also be considered early on.

Semantic search with vector databases

Now that we have understood the concept of embeddings, the question arises of how we use them for a search.

Evidently we have to store a great many vectors (one for each chunk), and within these stored vectors we have to find the vectors that are most similar to a given vector (that of the question).

A vector database is the right solution for this. By now there are very many offerings, from extensions for Postgres through the offerings of the hyperscalers to highly specialized vector databases. A review of the options would go beyond the scope of this article. Instead, we will look at the fundamental concepts.

Similarity metrics

First the question arises of what similarity of two vectors actually means. Well, every chunk/vector represents a point $P_i$, and we have to find the points that lie closest to the point $P_Q$ that represents our question. The intuitive idea of simply using the Euclidean distance between the points is, however, not the best choice. As a rule, cosine similarity is used instead, which measures not the distance but the angle between two vectors -- it is 1 when the two vectors point in the same direction, 0 when there is a right angle between them, and -1 when they point in opposite directions.

But why is cosine similarity used? The main reason is that the embedding model providers recommend it, e.g. OpenAI for their embedding models. The popular open model all-MiniLM-L6-v2 states that it was trained specifically with cosine similarity.

The OpenAI embedding models are moreover normalized, i.e. all vectors have the same length. It follows that cosine similarity and Euclidean distance produce the same ordering anyway when you sort the vectors by distance. The great advantage of cosine similarity with normalized vectors is that it can be computed mathematically very efficiently as a dot product.

Approximate Nearest Neighbor (ANN)

We have to assume that there are very many vectors in our database. Therefore an exact nearest neighbor search, which has to compute the distance from every vector to our vector, is often too slow and approximations are used.

The most widespread algorithm today is HNSW (Hierarchical Navigable Small World). The basic idea: instead of blindly going through all vectors, you build the vectors into a navigable graph in which neighboring vectors are connected. In a search, you then "work your way" from a starting point onward to whichever neighbor lies closer to the question vector, until you cannot get any closer.

The hierarchical in the name stands for a trick that speeds up this working-your-way: the graph lies in several layers on top of each other. The topmost one contains exactly one entry node, which is connected to few, but widely distributed neighbors. Each of these neighbor nodes is part of the next layer, where the connections are somewhat less long-range. Down to the lowest layer, in which all connections are very short. The search starts at the top and descends layer by layer. This mixture of long and short connections means that every node can be reached in a few hops. That is the small-world property in the name. As you go through these layers, you focus your search on an ever smaller area, so that you do not have to compute the distance from all vectors to the question vector, but only from relatively few candidates that lie roughly in the right area.

That finds the nearest neighbors most of the time, but not with a guarantee. The algorithm and its parameters can usually be tuned and have effects on the recall (i.e. whether all nearest neighbors are actually found) as well as on search speed and memory requirements. So there is a classic trade-off here that you should keep in mind.

Hybrid search: keyword + vector

But there are also cases in which semantic search via vectors fails. A classic example are acronyms, names or jargon terms that are clearly defined in the internal context and well known to all users, but unknown to the embedding model. For these we do not need the strengths of our semantic search, but a simple keyword search. So we can run a keyword search in parallel and return the results of both searches to the LLM.

This way the search always finds our first JSON chunk when questions about Scrooge are asked, even if the embedding model has never seen the name "Scrooge" and therefore cannot place it meaningfully in the vector space.

While semantic search aims at meaning, keyword search is based on the literal match of terms. The established methods for this are BM25 and the classic TF-IDF, which evaluate how often the search terms occur in a chunk, weighted by how rare a term is in the overall stock. Not every vector database comes with such a keyword search out of the box. Sometimes one therefore combines the vector search with a separate full-text search engine like Elasticsearch or OpenSearch. The results of both worlds are then merged, for example via Reciprocal Rank Fusion (RRF): instead of directly comparing the differently scaled scores of the two methods, only the rank position of a hit in each of the two lists counts here. A document that is near the top in both lists also ends up near the top in the combined list.

Filters and metadata

Quickly back to the databases: together with the chunks and their corresponding vectors, we should also store important metadata.

• Which document and which page this chunk comes from.
• Possibly the creation date or the author is of interest.
• Which users are allowed to see the content of this document.

All of this metadata can be used for filtering during the search. The filter criteria can be specified by the system in the case of access control, set by the user in special applications (e.g. search only in this one file), or specified by the LLM (e.g. date ranges).

And all of this metadata can in the end be given to the LLM together with the chunk for the answer, which is particularly interesting for source references. This allows the LLM to say in its answer on which page of which document the relevant information is. And of course this mechanism can be expanded further, for example to highlight the used text directly in the original PDF.

Reranking: from similar to relevant hits

Now we have $k$ results from the hybrid search. We naturally tend to want to choose $k$ high so that no relevant results are lost, but we do not want to give the LLM too many pieces of information, and no irrelevant ones.

One solution to this dilemma is reranking. The fundamental idea of reranking is that we no longer have to consider the large set of documents (possibly millions of chunks), but can apply more computationally intensive methods to the $k$ results (typically $10$ to $100$).

Through reranking, the quality of the results is re-evaluated and only the relevant results are passed back to the LLM.

Typically one uses a cross-encoder, which is called with both the question and the chunk and then judges how relevant the two texts are to each other. Technically speaking, cross-encoders today are mostly transformers -- but all that matters is their function: to judge the relevance of two texts to each other.

This is exactly where the reason lies why reranking is a stage of its own and does not do the search right away. Our embeddings are so-called bi-encoders: question and chunk are each translated separately into a vector. That is the prerequisite for being able to precompute the chunk vectors already at indexing time. At search time, only the question still has to be embedded. A cross-encoder, in contrast, looks at question and chunk together and can thus capture fine interactions between the two texts that two isolated vectors cannot capture that way. That makes it considerably more accurate, but also not precomputable: for every question-chunk pair a separate run is necessary. With millions of chunks that is not feasible -- with the $k$ candidates from the first search, on the other hand, it is easily doable. Reranking is therefore the place where we can finally afford the expensive but precise method.

LLM-based reranking

For certain cases it can also make sense to have the relevance of a result assessed directly by an LLM. LLMs are usually considerably slower and more expensive than simpler cross-encoders, but also more generally applicable and more flexible. This way you can also give the reranker LLM specialized criteria for the reranking.

Suppose our RAG searches the documentation of a piece of software that has gone through many versions over the years. Then the instruction for the LLM could be to rank higher, out of two chunks that fit equally well in terms of content, the one that refers to a newer version, and to downgrade references to outdated functions or ones marked as deprecated. Unlike with a pure similarity measure, the LLM can here read unstructured signals from the chunk text itself -- a mentioned version number, a "since version 3.0" or an "outdated" -- and let them flow into the assessment.

Through the relevance scores you not only get a new ordering of the results, but can also define a cutoff for irrelevant results that should not be contained in the RAG system's answer to the requesting LLM. This reduces the set of results to only the best ones, which keeps the LLM's context clean and improves its answer quality.

Query construction: what are we actually searching for?

So far we have passed over rather superficially what the "question" actually is with which we start our semantic search. This query is the input parameter of our search and thus decisively determines the result of the search.

And as so often, the answer here is "it depends", and it depends on the nature of the data basis as well as on the effort (which makes itself felt in cost and latency) one is willing to spend.

Direct use of the user question

The simplest option is to simply use the prompt with which the user asks the question unchanged. This usually works passably for semantic search. However, it can happen that the similarity of the vectors decreases when the user asks the question colloquially, but the data stock is formulated rather formally.

Besides, this hides the context of the conversation from the search. Let us look at this example conversation:

User: "Where is our office in Frankfurt?"
LLM: "At Lise-Meitner-Straße 4."
User: "And am I allowed to bring my dog there?"

If we use only the unchanged prompt "And am I allowed to bring my dog there?", the query loses the context that it is about the office in Frankfurt, and possibly only delivers the dog policy of the offices in Berlin and Munich. That way the LLM then has no chance of giving the right answer.

Query rewriting and normalization

One way to prevent this is to have an LLM reformulate the question. In doing so, normalization should take place, i.e. colloquial language should be replaced by formal language where applicable, references resolved and irrelevant details removed.

This is exactly where we solve the dog problem: if the reformulating LLM is given the previous conversation, it can resolve the second question "And am I allowed to bring my dog there?" into a standalone query like "Are dogs allowed in the Frankfurt office?". The ambiguous "there" has disappeared, and the search again has a chance of finding the right chunk.

Hypothetical Document Embeddings (HyDE)

An interesting strategy for increasing the matching of the query vector with the documents in the database, and thus for finding the fitting information more accurately, is to formulate the query the way you expect it in the data stock: namely as an answer.

So instead of searching for the vector of the question "Who is the richest duck?", you first have an LLM hallucinate a hypothetical answer ("Peter Müller is the richest duck.") and embed that. The fact that this answer may be factually wrong does not matter -- even a made-up answer like "Peter Müller is the richest duck" lands closer in the vector space to the actually stored chunks, which are, after all, likewise declarative sentences and not questions. The actual, reliable answer is generated in the end by the LLM anyway (without it seeing the made-up answers) from the real chunks that were found.

In order not to be led astray by a single hallucination, one usually generates several hypothetical answers at once. What you then do with them is a design decision of its own. In the classic HyDE variant you average the individual vectors (often including the real question) into one search vector and run only one search with it. That is cheap and smooths out outliers. The alternative is to search with each vector individually and merge the hit lists. That covers several regions of the vector space at the same time, but costs several searches -- and is basically already the multi-query retrieval of the next section.

Multi-query retrieval

If the recall of the RAG is too poor, i.e. not all relevant documents are found, it can help to have several queries constructed by an LLM, which are then all searched for via the semantic search.

The idea is that with a simple prompt like

Generate 3 alternative formulations of this question: {query}

we obtain several vectors that each cover a slightly different part of the vector space. Where a single formulation may narrowly miss a relevant chunk, one of the variants catches it. The hits of all queries are then merged -- and with that we are back at reranking and cutoff, in order to filter the best chunks out of the now larger set of hits.

Generation: from chunks to an answer

Now to the actual purpose! In the end the LLM is supposed to generate an answer -- the "G" in RAG. The simplest variant is to build a prompt that brings together the chunks that were found and the question, something like this:

Answer the following question exclusively on the basis of the provided context.
If the context does not yield the answer, say that you do not know.

Context:
{chunks}

Question: {query}

As inconspicuous as this prompt looks, two important aspects that ensure the quality of the RAG system are contained in it.

The first one is grounding: the instruction to rely exclusively on the context is meant to keep the LLM from sprinkling in its -- possibly outdated or simply invented -- world knowledge. This becomes even more valuable when we annotate the chunks with their origin (document title, page number, URL) and ask the LLM to back up its statements with these sources. Then the user can check the answer instead of blindly believing it.

The second one is the permission to say "I don't know". RAG is by no means a reliable protection against hallucinations. If the search finds no matching chunk, a helpful LLM is only too happy to fill the gap with a plausible-sounding invention. Giving no answer is in this case the best answer the system can give. And if the RAG query has been provided to the LLM as a tool, the LLM can also start another, modified query.

For special cases one can also choose other methods. If the questions often ask for summaries of larger amounts of data, a map-reduce approach is conceivable. You would adjust the parameters so that many chunks are allowed as a result, have each of them summarized by an LLM with a view to the question, and then deliver only the summaries to the LLM that talks to the user. That the question is already known at this intermediate stage is important here: if you summarize blindly, the very detail that mattered may be dropped. We deliberately override the cutoff from the reranking with this. This is paid for with one LLM call per chunk instead of a single one -- and with the fact that cross-connections are lost: a question whose answer is spread over two chunks does not survive this intermediate stage.

Evaluation: how do you measure whether it is any good?

We have now got to know adjusting screws at every station -- chunk size, hybrid search, reranking, query rewriting. But how do we know whether a change makes the system better or worse? Gut feeling is a poor advisor here, because the effects are often subtle and a change in one place can break something in another. So we need numbers.

This topic, however, is so extensive that it needs a blog entry of its own -- or several:

• RAG Evaluation mit TypeScript (in German) -- Robin Schlenker builds faithfulness, answer relevance and LLM-as-a-judge by hand and thereby makes visible how changes to the system affect the answer quality.
• Reliable by Design: Building an Evaluation Harness -- Niklas Niggemann sets up a gold dataset along with regression detection. The example is a text-to-SQL system, but the setup carries over.
• Closing the Loop: A Prompt Iteration Workflow -- what you then do with the numbers: one change per run, let the suite run completely again, compare, keep or discard.

Building the RAG that fits

With that we have walked the whole path once: from the raw document via chunking, embedding and hybrid search through to reranking and the question of what we are actually searching for. At almost every station the answer is "it depends". The right chunk size, the fitting search strategy, whether an LLM reranker is worth it -- all of that depends on the data, the questions and the budget.

The good news is that you do not necessarily need everything at once. A RAG system can be built up incrementally quite well: first a simple Recursive Character Splitter and a pure vector search, and then you measure. Poor recall? Add multi-query. Too much noise in the context? Reranking with cutoff. Acronyms that nobody finds? Add keyword search. That way, step by step, a system grows out of a simple prototype that can reliably answer whether the dog is allowed to come into the office.