Skip to content
academia.sh
tr
Log in

Course Intermediate

Scaling the Data Layer

By the end of this course

Start course

01

Data Distribution

Decisions about where data sits: tying the choice between relational and non-relational to the access pattern, leader–follower and multi-leader replication models, splitting the database by function, key selection with rebalancing and hot spot problems, range, hash, and lookup-table partitioning strategies, denormalization that shortens the read path, precomputed results, and the use cases of key–value, document, wide-column, and graph stores.

  1. 01 Data Store Selection Tying the store type to the access pattern: separating the request and record rates of three patterns, measuring the single-store and pattern-split layouts in a real engine by tables touched and records processed, drawing the decision axis from pattern and scale rather than the data model, and calculating what the split costs the introductory course's store load and stored data rows.
  2. 02 Replication What keeping the same data in more than one place costs at scale: splitting reads while writes repeat at every node in a leader–follower layout, the diminishing gain from adding replicas, stored data multiplying by the replica count, converting asynchronous lag into a stale tracking response, and counting two zones writing the same shipment in a multi-leader layout as permanent divergence.
  3. 03 Federation Splitting data by function rather than by key: separating the pricing and delivery operations contexts into their own stores, measuring in a real engine the stores touched and the keys carried between stores by a query that touches both contexts, calculating what duplicating shared fields adds to stored data, and taking the batch scan out of the online store.
  4. 04 Sharding Distributing the same context's data across independent nodes by a key: comparing three candidate shard keys by data distribution, hot spot ratio, and the node count each access pattern touches, measuring the data and requests the busiest node carries, converting the fraction of keys that move on rebalancing into bytes moved, and choosing the key by the most frequent pattern.
  5. 05 Partitioning Strategies Distributing the same key to partitions by range, hash, and lookup-table rules: measuring the hot spot ratio across three access patterns, the partition touched and the record count scanned by a periodic scan, and the store touches per access, plus the fraction of keys that move during rebalancing and the floor the largest unsplit group sets on the balance.
  6. 06 Denormalization Shortening the read path as a scaling decision: how the narrow and wide copy drive secondary-table records toward zero, why the number of tables touched is a misleading measure, measuring the bytes the copy adds to a shipment record in a real engine, and how far the scope the copy has to track the source over pushes the store's write/read ratio past 2.33.
  7. 07 Materialized Views Precomputing and storing a repeated sum: how the result's grain determines the record count scanned, incremental refresh's cost against full recompute, why the storage cost stays negligible, and the trade-off the refresh interval sets up between the staleness window and the store's write/read ratio.
  8. 08 Store Types Measuring the key-value, document, wide-column, and graph store against the same access patterns: the query count and record count each type requires for a pattern, which pattern the structure that cheapens one makes expensive, and the cost the choice charges to the introductory course's store request rate and write/read ratio.

02

Cache Architecture

Which layer a cache sits in with which strategy: the placement decision across the client, edge, web, application, and database layers, application-controlled cache-aside, write-through and write-behind where the write path is wired to the cache, refresh-ahead that renews an entry before its lifetime expires, the cases where a cache is not really needed, and settling on an acceptable staleness window.

  1. 01 Cache Placement Treating which layer a copy sits in as a design decision: separating the client, edge, web, application, and database layers by scope and capacity, measuring the per-layer hit ratio and miss penalty of a four-layer chain, turning the introductory course's 0.90 cache hit assumption into a result of the design, and showing that a single correctly sized cache delivers the same ratio with fewer touches.
  2. 02 Cache-Aside Treating an application-controlled cache as a scaling tool: turning the hit ratio into a function of the write/read ratio, showing the introductory course's 0.90 assumption fall to 0.7287 at its 0.2333 ratio, showing that every invalidation causes a store read, and measuring whether the application can go to the store on its own when the cache layer partly fails.
  3. 03 Write-Through and Write-Behind Two arrangements for wiring the write path to the cache: write-through removing the delete and raising the hit ratio from 0.7285 to 0.9170, the second write target added by denormalization stretching the path to two stores, write-behind lowering store writes per event from 2.00 to 1.20 through batching, and measuring the 200 events pending in memory in exchange as a durability window.
  4. 04 Refresh-Ahead Measuring an entry's refresh in the background before it expires: the lifetime dropping the hit ratio from 0.9170 to 0.7109, waiting requests falling and wasted refreshes rising as the threshold ratio grows, the store load at threshold 1 equaling the cacheless design's 513.89 req/s while the hit ratio still reads 0.8286, and showing that the real lever is lifetime, not threshold.
  5. 05 Cacheless Design Anti-Pattern Deciding whether a cache is needed from the access pattern: a cacheless design loading 513.89 req/s onto the store for the tracking query, the cache missing entirely on the period scan while still costing 2.00 extra touches per request, a 5000x repeat rate in hot writes dropping to a 0.3001 hit ratio, and passing the bulk scan through the same cache cutting the tracking hit ratio by 0.4263 while raising store access by 85,252.
  6. 06 Consistency and Stale Data Computing the staleness window from how fast the data changes: deriving an 8228.57-second peak-hour event interval from a 48-hour transit time and V4, the window saturating at 60 seconds to bring the hit ratio to 0.8968 and requests reaching the store to 140.23, a stale response always falling exactly one event behind, and replacing the earlier lessons' invalidation upper bound with the real 0.0010 share.

03

Read–Write Separation

Scaling the read and write paths separately: the decision to scale with distinct models, storing change as a record, deriving and rebuilding read models by folding, and eventual consistency showing up in the user interface as waiting and conflict.

  1. 01 Command and Query Separation Scaling the read and write paths with separate models: the tracking query touching eight records in the single model and one in the projection, the command climbing from three touches to five, comparing the two layouts' node counts under read and write multipliers, and the read store's inability to escape K01's write/read ratio.
  2. 02 Event-Sourced Design The scaling cost of storing change as a record: the append-only log removing the read-modify-write read from the write path, store operations per request dropping from 4.50 to 3.10, the source event's extra fields raising daily growth from 976 to 1,168 MB and stored data from 712.48 to 852.64 GB, and the periodic scan climbing from 10.80 to 35.04 GB.
  3. 03 Deriving Read Models Building the projection from the event log by folding, and rebuilding it from scratch: incremental maintenance producing the same rows as a rebuild, the projection adding 192 MB to daily growth and 140.16 GB to stored data, the open window reading 146 times fewer events than full retention, and a rebuild's 2.11-hour lag window against side-by-side rebuilding's 140.16 GB temporary space cost.
  4. 04 Eventual Consistency and the User Experience The lag window's counterpart in the interface: the stale views, wait rounds, conflict notifications, and silent losses four client strategies produce over the same event sequence, the optimistic display producing silent loss once the lag exceeds the view count, and read-your-writes raising the write store from 97.22 to 137.22 ops/s.

Start typing to search.

↑↓ ↵ Esc navigate · open · close