Skip to content
academia.sh
tr
Log in

Course Advanced

Search Engines and Text Retrieval

By the end of this course

Start course

01

Indexing and Analysis

Text becoming searchable: which question separates a search engine from a relational database, the construction and cost of the inverted index mapping term to document, the analyzer chain of tokenization, normalization, and filters deciding the result, the distinction between text and keyword field types, the schema-explosion risk of dynamic mapping, and the mechanics of a document being added, updated, and deleted.

  1. 01 Search Engine vs. Database Reaching a document through text asking for a different structure than reaching one through a known key: the distinction between key-known access and word-known access, the same catalog question answered by a relational LIKE scan and by an inverted index mapping term to document, counting the records and entries each path scans, the document set each returns, the documents each misses, and the effect of case and inflection on the set.
  2. 02 Inverted Index Building the structure that maps term to document and its byte accounting: the distinction between the dictionary and the posting list, pricing the document id, term frequency, and position information in a posting entry separately, measuring the index's ratio to the source text across three formats, the effect of keeping posting lists in document id order on intersection cost, and the adjacency distinction that position information buys.
  3. 03 Analyzer Chain The rule for turning text into terms deciding the outcome: building the chain of tokenization, normalization, and filters in three configurations over the same corpus, comparing raw, lowercasing, and stemming-plus-stop-word chains on index size, returned set, and the top five results, and counting the gap created by Turkish case folding, the apostrophe, and inflectional suffixes.
  4. 04 Mappings and Field Types A field's type deciding the indexing rule: separating the text field from the keyword field, indexing the same field with both types and comparing term count, posting entries, and index bytes, the effect of exact matching versus partial matching on the returned set, the cross-match a multi-valued field produces, whether a sort key is defined, and how aggregation buckets diverge between the two types.
  5. 05 Dynamic and Explicit Mapping Who decides the mapping: comparing dynamic mapping, which adds every incoming field to the index, against explicit mapping, whose field list is fixed in advance, on the same catalog import; the field count growing with document diversity; counting the mapping metadata per field in bytes; and measuring the documents rejected when the same name arrives with two types, and their fall from the query set.
  6. 06 Document Lifecycle A document entering, changing, and leaving the index: an update happening as a delete plus an add, old posting entries staying in place, the room a delete marker takes per internal id, dead entries being filtered out at query time while adding to the scanned count, an updated document moving to the end of the result order, and how bloated the index gets before a cleanup round.

02

Querying and Relevance

Which documents come back in which order: the structure of leaf and compound queries, exact-match term queries versus order-sensitive phrase queries, must, must-not, and should context conditions, the distinction between a query that enters scoring and a filter that does not, term-frequency and inverse-document-frequency based relevance scoring, field weight and function-score tuning, metric and bucket aggregations, highlighting and the deep-pagination problem in result presentation, embedding-based semantic retrieval, and the hybrid approach.

  1. 01 Query Language Structure The same catalog question is asked in plain text and in structured form: the returned set ranges from 4,211 to 71 documents when how the conditions combine goes unsaid, the compound query is evaluated as a tree, and the tree's evaluation order lowers the entries scanned without changing the set.
  2. 02 Term and Phrase Queries The same two terms are asked first at document level, then at position level: a term query returns 757 documents, an adjacent phrase falls to 144, slop 3 raises that to 377, and the position information that makes this distinction possible grows the index by 51.7%.
  3. 03 Context Queries Required, excluding, and optional conditions are measured in the same compound query: an excluding condition removes 512 documents from the set, an optional condition rewrites nine of the top ten results without removing a single document, but with no required condition present that same optional condition turns into the condition that determines the set.
  4. 04 Filtering and Query Distinction The same condition is run first in query context, then in filter context: the returned set stays fixed at 1,138 documents while nine of the top ten results change, only four of 1,138 documents stay in place, score computation is cut in half, and the filter's result can be reused as a 750-byte bit set.
  5. 05 Relevance Scoring The scorer is built from three components — term frequency, inverse document frequency, and field length — and which component a document's position comes from is shown one by one; precision and recall are defined here, and the measurement shows ranking raising precision from 0.353 to 0.640 while hitting a ceiling on a measure the model cannot see.
  6. 06 Score Tuning Two adjustments that rewrite the same query's order: how many documents field weight changes in the top ten and how many ranks it moves a document, how field selection — unlike weight — narrows the matching set, at what weight function-based scoring lets recency and popularity override relevance, and the difference between the two combination modes.
  7. 07 Aggregation Queries Queries that produce a number instead of a document: measuring metric, bucket, and pipeline aggregations on the same loan record, the byte cost of bucket count, comparing an exact set against an estimate in high-cardinality distinct counting, and how running an aggregation over the whole corpus, the matching set, or the filtered set changes the result.
  8. 08 Highlighting and Pagination The result's face shown to the reader: highlighting's per-document re-scan cost against the permanent byte overhead of storing position in the index, the linear growth with page depth of the candidate count pulled from each shard in deep pagination, cursor-based navigation producing the same page at a fixed cost, and the repeats and skips offset pagination produces when the index changes.
  9. 09 Semantic and Vector Search An access method that compares the context a word appears in rather than the word itself: building embeddings from term counts, two synonymous terms coming out close without ever appearing together, the documents vector access gains and loses relative to term access, hybrid search's fusion of two lists, and the vector index's byte and scan cost.

03

Cluster Operations

The index living on a cluster: the roles of the master, data, and coordinating nodes, the partitioning and redundancy balance between shard and replica count, the split brain problem and the majority requirement, bulk indexing's effect on write throughput, segment merging's background cost, the index lifecycle through rollover, cooling, and deletion policies, snapshot and restore strategy, and user, role, and API key management.

  1. 01 Node Roles Moving search from a single process to a cluster: separating the data, coordinating, and master node roles, running the same three hundred queries under two role placements and counting the postings each node reads and the merge steps it performs, showing that as shard count grows the data work splits while the coordinating work does not, and measuring how placing the master role on a loaded data node delays cluster state and changes the returned document set and the order of the top five results.
  2. 02 Shards and Replicas Showing that shard count is a scoring decision: splitting the same corpus into one, two, four, eight, and sixteen shards and comparing each placement's top ten result with the single-shard index's top ten by shared documents and rank shift, measuring that raising candidate depth does not fix the drift, counting that a global document-frequency round restores the top ten exactly and costs one extra round, and separately measuring how replica count changes index bytes and the returned document set under node loss.
  3. 03 Split Brain Problem Splitting a five-replica index into a three-node and a two-node half: counting the indexing operations that are accepted in the minority half while the majority requirement is off and lost on repair, measuring that the same query returns a different document set and different top five in each half, comparing the cost of the minority rejecting writes and answering queries short — under the majority requirement on — in write messages and missing documents, and comparing the post-repair index with the index that would exist had the split never happened.
  4. 04 Bulk Indexing Optimization Indexing the same four-thousand-record corpus one at a time and in bulk: comparing them by request count, segments produced, total index bytes, and dictionary lookups per query, showing that batch size and refresh interval are two independent axes, and measuring the trade-off between visibility lag and query cost as the refresh interval changes, counted on a six-hundred-record donation batch by matched documents, missing documents, and the order of the top five results.
  5. 05 Segment Merging The cost every batch's leftover segments charge a query: how segment count affects dictionary lookups, scanned postings, and index size, clearing delete markers, the bytes a merge copies, and comparing two factor settings of the tiered merge policy.
  6. 06 Index Lifecycle The measured effect of rolling an endlessly growing index over by time: the rollover interval's effect on query scope and scanned postings, the cost gap between deleting documents in one large index and dropping an index, the data rollover granularity keeps outside the retention window, and the memory a hot, warm, and cold tier placement holds.
  7. 07 Snapshot and Restore A backup decision specific to the index: the bytes an incremental snapshot built on segment immutability copies, the background merge breaking that incrementality, the store holding more than the live index, and comparing restoring from a snapshot against reindexing from the source by copied bytes, documents processed, the set, and the order.
  8. 08 Access Control The search cluster's authorization surface: counting the indexes, documents, and fields the same query returns under a reader, staff, and unscoped key, measuring how applying document-level filtering before or after scoring affects the top ten result and the deep ranking, showing the information filtered documents leak through a counter, counting the index count an unscoped key reaches against the queries an expired key leaves unanswered, and gathering the course's twenty-three lessons into a set, order, and cost table.

Start typing to search.

↑↓ ↵ Esc navigate · open · close