Unit 02 of 02
Retrieval Pipeline
Counting what a retrieved chunk does to the model and what each step of the pipeline costs on its own: comparing three runs, no retrieval, retrieval and retrieval with wrong chunks, and the run with wrong chunks falling below the baseline; what document preparation adds to retrieval recall and what it throws away; recall that rises and chunk and token counts that multiply as chunk size and overlap grow; the stale chunk rate that grows and the number of re-embedded chunks as the refresh interval lengthens; what one-step query transformation adds in calls and what it gains on a short query and loses on a long one; what reranking adds to the precision of the top k against the growing candidate count and latency class; the share of answers changed by the number of chunks that fit the context limit and by giving the same chunks in a different order; the four-cell contingency of retrieval and generation of the right chunk and the right answer and how a single number hides all four; what retrieval and fine-tuning add on the tasks of knowledge freshness, format compliance and domain language; what row-level and block-level chunking of a source carrying tables does to recall.
Published Updated
Lessons
- 01 What Is Retrieval-Augmented Generation Running the same forty queries in three setups over a fictional corpus of two hundred documents and printing retrieval and generation as two separate columns: the no-retrieval baseline scores 0.2000 on generation, while the pipeline built with an in-domain embedding raises retrieval recall to 0.9500 and generation to 0.9250, and the same pipeline built with an out-of-domain embedding leaves recall at 0.1750 and drops generation to 0.0500, below the baseline. The cost is written side by side: from 3033 tokens to 4848 tokens, from zero retrieval steps to twenty-seven thousand seven hundred twenty steps, from 379.1 tokens per correct answer to 131.0 tokens, and to 2457.5 tokens in the wrong-chunk run. The run of six seeds gives this workload a threshold of 0.0750, and one of the thirty-eight queries that receive the correct chunk is still answered wrongly. 14 min read
- 02 Document Preparation Measuring four preparation levels on the same forty queries, starting from the raw form of the same fictional corpus: the source left as one block per document pushes retrieval recall to the highest value, 1.0000, but every request overflows the context window and generation falls to 0.0000, so had a single column been printed the worst setup would have won. Crude line-based splitting stores 1805 chunks and gives 0.9250 on generation, the same as structure-aware preparation with 693 chunks, which reproduces exactly where the previous lesson started; what separates them is the stored vector count, not the hit rate. The repetition-based boilerplate filter drops 798 lines and 7021 tokens while destroying 96 of 568 concept-document links and erasing the evidence of the main concept in eight documents; when the threshold is lowered to two, the loss rises to 283 links and sixty documents. The retrieval and generation columns show none of this loss. 6 min read
- 03 Chunking Strategies Treating the chunk boundary as a decision and sweeping size and overlap on the same fictional corpus: eight-token chunks keep only 0.0765 of the source's 693 relation occurrences whole, eighty-token chunks keep 0.9971 of them but 0.15 of the requests exceed the context window and generation falls from 0.9250 to 0.5500. In a twenty-token chunk, raising overlap from zero to twelve lifts relation recall from 0.6392 to 1.0000 while the stored tokens climb from 8985 to 17206, that is 1.91 times; the first step gains nothing, and the cost per relation gained jumps from 23 to 138 after the eighth token, which is where the crossover point lies. Splitting along the source's own paragraph boundary gives the same 1.0000 relation recall with 693 chunks and 8985 tokens, roughly half the space of the overlap. Retrieval recall and generation do not move enough to cross the threshold in any row of the overlap sweep. 6 min read
- 04 Indexing and Updating Measuring that the index is a snapshot and goes stale as the source changes: in a fictional twelve-period change stream, when the refresh interval goes from one to twelve the stale chunk rate rises from 0.0000 to 0.2562 and the wrong answer rate from 0.0750 to 0.3875. The retrieval column shows none of this degradation: retrieval recall is 0.9500 and concept hit rate 0.9250 at all six intervals; the only column that falls is the share of answers whose copied validity record is up to date. The cost of refreshing reads differently under the two policies: full reindexing embeds 8316 chunks at interval one, incremental refresh embeds 881. Multiplying the interval by twelve lowers the re-embedded chunks per up-to-date answer under incremental refresh only from 1.98 to 1.77, a gain of eleven percent; the price of the same decision is 0.3125 in the up-to-date answer rate. The difference between the two runs is 0.0000 in every row. 14 min read
- 05 Query Transformation Translating the query the user typed into the document language with a single model call before it reaches the index, and counting what this step gains and what it loses separately: on a fictional set of forty queries the short phrasing has a baseline of 0.3250 recall and 0.1500 generation, while expansion raises the two to 0.7500 and 0.7000, which is where a query written in the index language leaves off. The same setup drops the long phrasing from a baseline of 0.5500 recall to 0.5000 with expansion and to 0.2750 with rewriting, because the share with which the transformer finds the correct concept falls from 0.9500 to 0.3500. The cost on both sets is one call per query and more tokens: from 4609 to 5804 on the short set and from 5827 to 8553 on the long set; the run-to-run difference was measured at 0.0500 in recall and 0.0500 in generation. 14 min read
- 06 Reranking Splitting retrieval into two stages and counting side by side what the second stage gains and what it makes you pay: in the single-stage pipeline the precision of the top three stays at 0.4167, because a single similarity score cannot search for the concept and the record type of the query at the same time. The second stage, which reads the candidates together with the query, raises precision to 0.8250 with twelve candidates and to 0.9000 with twenty-four, and takes generation from 0.7000 to 1.0000. With forty-eight candidates precision rises by only 0.0083, that is, it stays below the run-to-run difference of 0.0250; in return the recall of the top three falls from 1.0000 to 0.9500, the call count goes from forty to one thousand nine hundred sixty, and tokens go from 3654 to 71797. 14 min read
- 07 Context Construction Placing the retrieved chunks into the prompt and measuring that this placement is a mechanism in its own right: after the instruction and the query, 13.4 chunks fit into a window of two hundred forty tokens, yet at the eleventh chunk six of the forty queries, at the twelfth and the thirteenth fifteen of them, exceed the limit. When the chunk count is raised from one to ten, retrieval recall rises from 0.7750 to 0.9750 while generation falls from 0.7750 to 0.5750. When the context always holds one relevant chunk with four irrelevant chunks beside it, generation is 0.7750 with the relevant chunk at the head, 0.6250 at the tail and 0.2000 in the middle, and when the order of the same chunks changes, 0.7000 of the answers change in the middle layout; when the seed is changed in the same order, the share of change is 0.4250. 14 min read
- 08 Measuring Retrieval and Generation Quality Separately Showing that a single end-to-end accuracy rate stacks four separate cases on top of one another: on the forty-query fictional set the baseline without retrieval comes out at 0.3750, and the whole of this rate is chance, because the correct chunk is never retrieved. With a three-chunk context the single number is 0.9500, and with a five-chunk context it is also 0.9500; but with three chunks the two failures are retrieval faults, with five chunks the same two failures are generation faults, and the single number does not tell them apart. Retrieval recall is 0.9500 with three chunks and 1.0000 with five; the cost goes from 120 chunks and 1872 tokens to 200 chunks and 2932 tokens. The difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the three-chunk context. The class subgroup shows that both faults are collected in the same class. 12 min read
- 09 Measuring Retrieval and Fine-Tuning Side by Side Measuring three separate tasks on the same workload, once with retrieval and once with fine-tuning: on the forty-query fictional set, knowledge freshness comes out at 0.0000 without retrieval, 0.9500 with retrieval alone and 0.6750 with fine-tuning on forty-eight examples, and adding examples does not lift this ceiling, because stale knowledge is not a coverage problem. Format compliance stays at 0.0000 with retrieval and reaches 1.0000 with fine-tuning on twelve examples: this is the task retrieval does not win. Domain language is 0.9500 with retrieval and 1.0000 with fully covering fine-tuning. The cost is extra tokens per request and forty retrieval steps for retrieval, and forty-eight labeled examples for fine-tuning. The difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the other three setups. 12 min read
- 10 Chunking Sources Containing Tables and Images Counting what the chunking decision does to retrieval recall when the same fictional corpus is presented in table form: because the procedure name sits only in the header, row-level chunking drops recall to 0.0000; block-level chunking gives 0.9500 but carries 88.7 tokens per request; storing each row together with its header gives the same 0.9500 recall with 38.5 tokens per request and raises the tokens stored in the index from 5702 to 8634. When the table arrives as an image, the source without a description gives 0.0000, and the source with a one-line description gives 0.9500 recall but 0.0000 answerability: the source is found, and the value inside it cannot be retrieved. This lesson closes the Embeddings and Retrieval-Augmented Generation course. 14 min read