Unit 04 of 04
Local Models and Adaptation
Counting the tradeoffs of running a model on your own hardware and of adapting it: the number of values that parameter count and context length hold in memory and the concurrent requests a budget affords, the memory that falls as the number of digits drops and the point where accuracy falls and which task it falls on first, the prompt-side mechanism and fine-tuning measured side by side on the same task and the case where fine-tuning does not win, the updated-parameter share of full update, layer freezing and low-rank update with its accuracy and the count of stored values, the gap a small model trained on a large model's output closes and the gap it leaves open, and the deployment options that four classes of usage conditions close in one scenario.
Published Updated
Lessons
- 01 Self-Hosted Models Running a model on your own hardware makes the unit memory instead of tokens: the weight memory of three fictional scales is 70.5 million, 367.5 million and 2.14 billion values, the intermediate state per request in a 960-token context is 11.8 million, 47.2 million and 157.3 million values, the concurrent requests a three-billion-value budget affords are 248, 55 and 5, and the label accuracy on the same 120 fictional tickets is 0.6333, 0.8167 and 0.8667. The product of concurrent requests and accuracy falls from 157.1 to 4.3, so the largest scale yields one thirty-sixth of the correct-response capacity. With the distribution temperature at zero both runs give the same 0.8667, and at 0.02 28 of the 120 tickets change from run to run. 5 min read
- 02 Quantization Measuring on the same 120 fictional tickets how the number of digits a weight is stored with affects memory and accuracy: at 16 digits weight memory is 34.3 billion digits and concurrent requests are 5, at 8 digits 17.2 billion and 24, at 4 digits 8.6 billion and 62. Five-class topic assignment is 0.8667 at 16 digits, stays at 0.8750 at 8 digits, and falls to 0.8000 at 4 digits, 0.6333 at 3 digits and 0.2083 at 2 digits; the two-class urgency distinction at the last four levels gives 0.9417, 0.9333, 0.8583 and 0.3583. Because the mean decision margin is 0.1597 in five classes and 0.2665 in two classes, the fine distinction goes first. The distinguishing difference of the 121 rows is erased completely in 24 rows at 4 digits and in 53 rows at 3 digits. 13 min read
- 03 Fine-Tuning Measuring side by side on the same 60 fictional test records what happens when the same labeled examples are put into the prompt and when they are written into the weights: the baseline without scaffolding is 0.6000, fine-tuning on four examples moves no weight and stays at 0.6000, at eight examples it reaches 0.7333 and passes the 0.6500 of prompt scaffolding, at sixteen examples 0.8833. At thirty-two examples the two paths overlap in the band 0.9000 to 0.9500, at sixty examples prompt scaffolding gives 0.9667 and fine-tuning stays at 0.9500: fine-tuning does not win. The cost is in two separate units; prompt scaffolding raises the context per request from 28.48 to 623.48 tokens, that is 21.9 times, while fine-tuning is a one-time 180 update steps and moves a 0.2992 share of the 605 weights. 13 min read
- 04 Adaptation Methods Measuring side by side, in the same field adaptation, the updated-parameter share and accuracy of full update, layer freezing and a low-rank update: on field spelling a fictional local model of 2292 numbers gives 0.2500 and 0.0833, full update 1.0000 and 0.9861, layer freezing that updates only the head 0.6944 and 0.7500, a low-rank update that updates 0.2094 of the parameters 0.9583 and 0.9306. Raising the rank from 3 to 4 makes 210 more numbers stored per adaptation and its mean gain is 0.0070, below the two-run gap. For adaptation to eight domains the stored count is 6132 against 18336, and as the hidden width grows the share of the low-rank update falls from 0.2094 to 0.0284. 13 min read
- 05 Distillation Measuring, on two tasks, what a small model trained on 240 fictional records labeled by the large model gains over the same model trained on only six human labels: on the topic task it rises from 0.5000 to 0.6333 and captures the whole teacher accuracy, but it closes only a third of the gap to the 0.9000 given by 240 human labels. On the separate topic-count task distillation moves backward, falling from 0.5833 to 0.3667 and 0.2417, because the teacher does not count. Of the distilled model's errors, 0.9545 are on records where the teacher also errs; for the model trained on six labels this share is 0.3000. 5 min read
- 06 Model Repositories and Terms of Use Counting how many distribution options four license condition classes close in a use scenario: of sixteen options built from four asset types and four channels, the usage limit alone closes 6, the redistribution right 3, the derivative-work condition 2 and the commercial-use record 2, a total of 13; together they close 10, because three options are closed by more than one condition. Four of the six options left open are in a single channel. Removing the usage limit opens three options, not six. When the monthly request volume triples, the open options fall from six to four, and at ten times they fall to one. This lesson is the close of the Building Applications with Model APIs course. 11 min read