Skip to content
academia.sh
tr
Log in

Unit 02 of 02

Training and Deployment

Counting, step by step, the path by which a trained model reaches production: connecting data preparation, training and evaluation steps into a single pipeline and the rows each step drops, code, data and model gates firing separately in continuous integration and which gate stops which corruption, pinning the runtime and the interface contract in packaging and the caller breaking when no contract is written, the latency class, the cost per prediction and the freshness trade-off of batch, online and streaming inference, request batching and caching lowering the cost per prediction while adding to latency, and what shrinking a model to run at the edge takes from accuracy and on which task it takes it first.

Published Updated

Lessons

  1. 01 Training Pipelines Counting the training pipeline as the first step of a trained model's path to production: the rows dropped by each of six steps are measured separately, and while the consumption step loses 200 rows its own counter reads 0, the feature step leaves 200 rows history-less without dropping a single row, the split separates 1600 and 600 without dropping a single row; with a label distribution of 1752 to 448 and a base class ratio of 0.7964, the model says 0 to 600 of the 600 rows in the test set and its accuracy of 0.8067 is exactly the base class ratio; when the 40 late-arriving rows are backfilled, the only setup that raises the training rows from 1520 to 1600 is the pipeline that re-runs end to end, and all seven setups report the same 0.8067. 10 min read
  2. 02 Continuous Integration for Machine Learning Counting which corruption each of the code, data and model gates placed between the run and the release stops: of the six corruption classes the data gate stops four and the model gate stops three, the code gate stops none but catches a feature code change at the first step without looking at production data at all; the unit change stops only at the model gate, by dropping the training accuracy from 0.7931 to 0.5919; the late-arriving row stops at no gate because its deviation in training rows is exactly 0.0500 and the tolerance is 0.0500 with a strict comparison; two of the four runs the data gate stopped would have produced exactly the same model as the reference, and in eight of eight runs the test accuracy of 0.8067 cannot rise above the base class ratio. 4 min read
  3. 03 Model Packaging Turning a trained model into a single unit with its weights, threshold and interface declaration, and counting the callers that break when no contract is written: when a field name changes in the 1800 calls of three callers the dropped calls rise from 600 to 1200, when the threshold field falls out of the package the number of dropped calls does not change but all 1200 of the 1200 calls that pass change their answer; a contract that declares only the input fields catches two of the three versions while a contract that also declares the output and the parameter catches all three before release, and for that it reads 3 sample records instead of 1800 calls; the label distribution is 1752 to 448, the base class ratio is 0.7964, and the test accuracy of the model, 0.8067, is exactly the base ratio in the test set. 15 min read
  4. 04 Serving Patterns Running the same package in three placements, batch, online and streaming, and counting the latency class, the freshness and the cost per prediction of the three separately: all three patterns give the same answer to the same 600 requests and the accuracy is 0.8067 in all three, so an observation that looks only at the output cannot tell which pattern ran. The difference is in three columns: the work done at request time is a lookup in the batch and streaming patterns and a scan of 352 to 550 rows in the online pattern; the age of the answer is one period in the batch pattern and zero in the others; the rows read per prediction are 11.00, 463.87 and 3.67. The staleness sweep gives 0 changed predictions up to six periods, and the reason is not the pattern but that the decision score stays at least 7.6468 away from the threshold. The cost is a denominator decision: the batch and online patterns are equal at 4.74 predictions used, and which task rests on which pattern is read from that point. 14 min read
  5. 05 Inference Scaling Counting how request batching, caching and the choice of accelerator class are traded off between the cost per prediction and the latency of the waiting request: as the batch size goes from one to two, four, eight, sixteen, thirty-two, sixty-four, one hundred twenty-eight and two hundred over the same 600 requests, the cost per prediction falls from 463.87 to 22.00 while the worst wait rises from 0 to 199 arrivals and the average wait to 99.50; the answers are identical in all nine rows and the accuracy stays at 0.8067, so the batch size is not a quality setting. The last group of each period stays at eight requests at sizes sixteen, thirty-two and sixty-four and waits for a group that will never fill; this wait does not show in the worst wait column at all. The crossover point of the two accelerator classes is the ratio of the setup difference to the unit difference and is 80 requests, but in the table the switch shows at one hundred twenty-eight because what is measured is not the requested size but the realized group size. On a stream of 600 requests the cache, with a hit rate of 0.6783, lowers the rows read per request from 464.86 to 148.70 and together with batching gives 26.38; the two gains cannot be multiplied. 13 min read
  6. 06 Edge and On-Device Inference Counting what moving the model to the source of the request takes from accuracy and from which task it takes first: the number written to the device is five, what is dropped is the history of 341 to 539 rows that produces the region feature, and when that history is dropped the two-class task stays at 0.8067 while the three-tier task falls from 0.4833 to 0.4733. The quantization sweep gives the same order: the two-class task does not move a single prediction down to level 4, the three-tier task moves two predictions at level 16 and five at level 8, and at level 2 the two collapse to 0.3833 and 0.2250. The independence gained by running at the edge is measured in correct answers: when the network outage rate rises to 0.20, the correct answers of the central setup fall from 484 to 391 and from 290 to 234, while the edge stays fixed at 484 and 284; the outage rate at which the two setups are equal is 0.0000 for the two-class task and 0.0207 for the three-tier task. The lesson closes the course and gathers the numbers of the twelve lessons into a single table. 11 min read

Start typing to search.

↑↓ ↵ Esc navigate · open · close