Skip to content
academia.sh
tr
Log in

Course Intermediate

Embeddings and Retrieval-Augmented Generation

Published Updated

By the end of this course

Start course

01

Embeddings and Vector Search

Measuring which chunk of the corpus a query retrieves and what each layer of the search scaffolding gains and loses: the retrieval precision and retrieval recall of keyword matching against embedding similarity on the same query set, the difference in retrieved chunks between an in-domain and an out-of-domain embedding at the same dimension, in how many queries cosine similarity and the dot product separate the first three results and in which one the length bias appears, the recall of an indexed search against the exhaustive search baseline read side by side with its comparison count, the neighborhood breaking down in a space that narrows as filter selectivity rises and the candidate count dropping, and the comparison of keyword-only, embedding-only and combined setups together with the query class where the combined setup stays worse than both.

  1. 01 What Is an Embedding Deciding which document in the corpus a query resembles by meaning rather than by words, and counting that decision with two separate measures. Forty queries are run on a fictional corpus of two hundred documents, 693 paragraphs and twenty concepts: keyword matching leaves a retrieval precision of 0.0800 and a retrieval recall of 0.0363, while embedding similarity gives 0.4700 and 0.2258; the first result's concept hit rate rises from 0.0250 to 0.5000 and its class hit rate from 0.1000 to 1.0000. The ceiling on recall for five chunks is 0.4904, and reading the class hit rate alone hides the half share in the concept hit rate. The cost is 8000 comparisons, 240 embedding calls and a stored value count of 6400.
  2. 02 Embedding Models Placing two of an embedding model's three settings side by side on the same workload: dimension and domain fit. At the same thirty-two dimensions, the same 6400 stored values and the same forty queries, the in-domain embedding gives a retrieval precision of 0.4700 and a concept hit rate of 0.5000, while the out-of-domain embedding falls to 0.0650 and 0.0500; on concept and class hit rate it rises only a step above the keyword search baseline, while on the retrieved set it drops below the baseline. With a second embedding model of the same family the difference stays at 0.0250 in concept hit rate while the loss from falling out of domain is 0.4500; the first is unmeasured, the second is measured. In the dimension sweep the two models do not agree between 64 and 8 on which dimension is good, and the cliff appears only between 8 and 4.
  3. 03 Similarity Metrics Ranking the same vectors with two separate measures and how much the choice changes the retrieved chunk. The dot product is the cosine similarity multiplied by the norms of the two vectors; for one query the query norm is constant, so the entire ranking difference comes from the norm of the document vector, and this norm is 4.6737 in the shortest quarter and 5.4708 in the longest. In thirty of the forty queries the first three results diverge; cosine similarity gives a retrieval precision of 0.4700 and a concept hit rate of 0.5000 while the dot product stays at 0.4100 and 0.4250, and the same direction appears in a second embedding model. The length bias is in the dot product: the mean of the retrieved set is 66.2 tokens, 57.7 under cosine similarity and 64.1 in the corpus; since the mean of the relevant documents is also 64.7, length carries no relevance.
  4. 04 Vector Databases Removing the assumption that a query is compared with the whole corpus and measuring the price of approximation. On two hundred fictional documents, exhaustive search needs 200 comparisons per query; the bucket-based index holds recall at 0.9500 with 16 buckets and 2 probes and brings the comparisons down to 47.2, and with 4 probes recall is 1.0000 and comparisons are 69.2. With 1 probe recall falls to 0.7200 and the concept precision of the first five falls from 0.4700 to 0.3800: the index damages not only the candidate set but also the answer. The neighborhood graph at degree 8 can reach only 27 documents from the entry node and its recall is 0.0800; at degree 48 it stores 9600 edges and, with 84.9 comparisons, stays at 0.7650, that is, both more expensive and worse than the bucket-based index. The difference between two index builds is 0.0100, and this is the threshold of this workload.
  5. 05 Metadata Filtering Combining vector search with a metadata constraint and measuring how the neighborhood breaks down as the filter narrows. Two hundred fictional documents are distributed over sixteen regions; the unfiltered bucket-based index gives a recall of 0.9500 with 47.2 comparisons. When the filter is applied after the search, recall falls to 0.4450 and to 0.0900, and forty of the forty queries return fewer than five results; retrieval raised to fifty pulls this up to 0.8900 and 0.4750 but cannot exceed the ceiling of in-index filtering. In in-index filtering, as the filter width goes from 16 down to 1, the candidate count falls from 31.2 to 2.8 and recall from 0.9500 to 0.4750; the comparisons fall from 47.2 to 18.8 while the exhaustive search of the filtered set falls from 200 to 18. At width 1 the index gives 0.4750 with 18.8 comparisons and exhaustive search gives 1.0000 with 18 comparisons. The difference between two index builds is 0.0100 without a filter and between 0.0850 and 0.1300 with one.
  6. 06 Hybrid Search Measuring keyword matching and embedding search separately and together. Three setups run side by side on three fictional query classes: in subscriber language keyword search is 0.0250 and embedding is 0.5000; in document language both are 0.5000; on the document code query keyword search is 1.0000 and embedding is 0.0000. Rank fusion gives 0.4000 in the document language class, that is, worse than both; this holds at constants of 5, 20 and 60. In the code class fusion drops the share of keeping the target document in the first five from 1.0000 to 0.0500. The reason is the number of documents shared by the first five of the two lists: 0.12 in subscriber language, 1.05 in document language. A document that stands second in both lists passes a document that is first in one and absent from the other. Fusion raises recall up to rank twenty from 0.7122 to 0.7214; the place it gains is depth, the place it loses is the first rank.

02

Retrieval Pipeline

Counting what a retrieved chunk does to the model and what each step of the pipeline costs on its own: comparing three runs, no retrieval, retrieval and retrieval with wrong chunks, and the run with wrong chunks falling below the baseline; what document preparation adds to retrieval recall and what it throws away; recall that rises and chunk and token counts that multiply as chunk size and overlap grow; the stale chunk rate that grows and the number of re-embedded chunks as the refresh interval lengthens; what one-step query transformation adds in calls and what it gains on a short query and loses on a long one; what reranking adds to the precision of the top k against the growing candidate count and latency class; the share of answers changed by the number of chunks that fit the context limit and by giving the same chunks in a different order; the four-cell contingency of retrieval and generation of the right chunk and the right answer and how a single number hides all four; what retrieval and fine-tuning add on the tasks of knowledge freshness, format compliance and domain language; what row-level and block-level chunking of a source carrying tables does to recall.

  1. 01 What Is Retrieval-Augmented Generation Running the same forty queries in three setups over a fictional corpus of two hundred documents and printing retrieval and generation as two separate columns: the no-retrieval baseline scores 0.2000 on generation, while the pipeline built with an in-domain embedding raises retrieval recall to 0.9500 and generation to 0.9250, and the same pipeline built with an out-of-domain embedding leaves recall at 0.1750 and drops generation to 0.0500, below the baseline. The cost is written side by side: from 3033 tokens to 4848 tokens, from zero retrieval steps to twenty-seven thousand seven hundred twenty steps, from 379.1 tokens per correct answer to 131.0 tokens, and to 2457.5 tokens in the wrong-chunk run. The run of six seeds gives this workload a threshold of 0.0750, and one of the thirty-eight queries that receive the correct chunk is still answered wrongly.
  2. 02 Document Preparation Measuring four preparation levels on the same forty queries, starting from the raw form of the same fictional corpus: the source left as one block per document pushes retrieval recall to the highest value, 1.0000, but every request overflows the context window and generation falls to 0.0000, so had a single column been printed the worst setup would have won. Crude line-based splitting stores 1805 chunks and gives 0.9250 on generation, the same as structure-aware preparation with 693 chunks, which reproduces exactly where the previous lesson started; what separates them is the stored vector count, not the hit rate. The repetition-based boilerplate filter drops 798 lines and 7021 tokens while destroying 96 of 568 concept-document links and erasing the evidence of the main concept in eight documents; when the threshold is lowered to two, the loss rises to 283 links and sixty documents. The retrieval and generation columns show none of this loss.
  3. 03 Chunking Strategies Treating the chunk boundary as a decision and sweeping size and overlap on the same fictional corpus: eight-token chunks keep only 0.0765 of the source's 693 relation occurrences whole, eighty-token chunks keep 0.9971 of them but 0.15 of the requests exceed the context window and generation falls from 0.9250 to 0.5500. In a twenty-token chunk, raising overlap from zero to twelve lifts relation recall from 0.6392 to 1.0000 while the stored tokens climb from 8985 to 17206, that is 1.91 times; the first step gains nothing, and the cost per relation gained jumps from 23 to 138 after the eighth token, which is where the crossover point lies. Splitting along the source's own paragraph boundary gives the same 1.0000 relation recall with 693 chunks and 8985 tokens, roughly half the space of the overlap. Retrieval recall and generation do not move enough to cross the threshold in any row of the overlap sweep.
  4. 04 Indexing and Updating Measuring that the index is a snapshot and goes stale as the source changes: in a fictional twelve-period change stream, when the refresh interval goes from one to twelve the stale chunk rate rises from 0.0000 to 0.2562 and the wrong answer rate from 0.0750 to 0.3875. The retrieval column shows none of this degradation: retrieval recall is 0.9500 and concept hit rate 0.9250 at all six intervals; the only column that falls is the share of answers whose copied validity record is up to date. The cost of refreshing reads differently under the two policies: full reindexing embeds 8316 chunks at interval one, incremental refresh embeds 881. Multiplying the interval by twelve lowers the re-embedded chunks per up-to-date answer under incremental refresh only from 1.98 to 1.77, a gain of eleven percent; the price of the same decision is 0.3125 in the up-to-date answer rate. The difference between the two runs is 0.0000 in every row.
  5. 05 Query Transformation Translating the query the user typed into the document language with a single model call before it reaches the index, and counting what this step gains and what it loses separately: on a fictional set of forty queries the short phrasing has a baseline of 0.3250 recall and 0.1500 generation, while expansion raises the two to 0.7500 and 0.7000, which is where a query written in the index language leaves off. The same setup drops the long phrasing from a baseline of 0.5500 recall to 0.5000 with expansion and to 0.2750 with rewriting, because the share with which the transformer finds the correct concept falls from 0.9500 to 0.3500. The cost on both sets is one call per query and more tokens: from 4609 to 5804 on the short set and from 5827 to 8553 on the long set; the run-to-run difference was measured at 0.0500 in recall and 0.0500 in generation.
  6. 06 Reranking Splitting retrieval into two stages and counting side by side what the second stage gains and what it makes you pay: in the single-stage pipeline the precision of the top three stays at 0.4167, because a single similarity score cannot search for the concept and the record type of the query at the same time. The second stage, which reads the candidates together with the query, raises precision to 0.8250 with twelve candidates and to 0.9000 with twenty-four, and takes generation from 0.7000 to 1.0000. With forty-eight candidates precision rises by only 0.0083, that is, it stays below the run-to-run difference of 0.0250; in return the recall of the top three falls from 1.0000 to 0.9500, the call count goes from forty to one thousand nine hundred sixty, and tokens go from 3654 to 71797.
  7. 07 Context Construction Placing the retrieved chunks into the prompt and measuring that this placement is a mechanism in its own right: after the instruction and the query, 13.4 chunks fit into a window of two hundred forty tokens, yet at the eleventh chunk six of the forty queries, at the twelfth and the thirteenth fifteen of them, exceed the limit. When the chunk count is raised from one to ten, retrieval recall rises from 0.7750 to 0.9750 while generation falls from 0.7750 to 0.5750. When the context always holds one relevant chunk with four irrelevant chunks beside it, generation is 0.7750 with the relevant chunk at the head, 0.6250 at the tail and 0.2000 in the middle, and when the order of the same chunks changes, 0.7000 of the answers change in the middle layout; when the seed is changed in the same order, the share of change is 0.4250.
  8. 08 Measuring Retrieval and Generation Quality Separately Showing that a single end-to-end accuracy rate stacks four separate cases on top of one another: on the forty-query fictional set the baseline without retrieval comes out at 0.3750, and the whole of this rate is chance, because the correct chunk is never retrieved. With a three-chunk context the single number is 0.9500, and with a five-chunk context it is also 0.9500; but with three chunks the two failures are retrieval faults, with five chunks the same two failures are generation faults, and the single number does not tell them apart. Retrieval recall is 0.9500 with three chunks and 1.0000 with five; the cost goes from 120 chunks and 1872 tokens to 200 chunks and 2932 tokens. The difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the three-chunk context. The class subgroup shows that both faults are collected in the same class.
  9. 09 Measuring Retrieval and Fine-Tuning Side by Side Measuring three separate tasks on the same workload, once with retrieval and once with fine-tuning: on the forty-query fictional set, knowledge freshness comes out at 0.0000 without retrieval, 0.9500 with retrieval alone and 0.6750 with fine-tuning on forty-eight examples, and adding examples does not lift this ceiling, because stale knowledge is not a coverage problem. Format compliance stays at 0.0000 with retrieval and reaches 1.0000 with fine-tuning on twelve examples: this is the task retrieval does not win. Domain language is 0.9500 with retrieval and 1.0000 with fully covering fine-tuning. The cost is extra tokens per request and forty retrieval steps for retrieval, and forty-eight labeled examples for fine-tuning. The difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the other three setups.
  10. 10 Chunking Sources Containing Tables and Images Counting what the chunking decision does to retrieval recall when the same fictional corpus is presented in table form: because the procedure name sits only in the header, row-level chunking drops recall to 0.0000; block-level chunking gives 0.9500 but carries 88.7 tokens per request; storing each row together with its header gives the same 0.9500 recall with 38.5 tokens per request and raises the tokens stored in the index from 5702 to 8634. When the table arrives as an image, the source without a description gives 0.0000, and the source with a one-line description gives 0.9500 recall but 0.0000 answerability: the source is found, and the value inside it cannot be retrieved. This lesson closes the Embeddings and Retrieval-Augmented Generation course.

Final exam

Embeddings and Retrieval-Augmented Generation — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close