--- title: Fine-Tuning source: 'https://academia.sh/en/courses/model-apis/fine-tuning' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:32+00:00' license: 'CC BY-SA 4.0' objectives: - 'Compares putting the same labeled example set into the prompt with writing it into the weights, on the same test set.' - 'Writes the cost of fine-tuning as example count, epoch count and the share of weights that move.' - 'Uses the difference between two draws to decide which gain counts as measured.' - 'Determines the example count at which fine-tuning does not win and computes the per-request cost of prompt scaffolding.' --- # Fine-Tuning Measuring side by side on the same 60 fictional test records what happens when the same labeled examples are put into the prompt and when they are written into the weights: the baseline without scaffolding is 0.6000, fine-tuning on four examples moves no weight and stays at 0.6000, at eight examples it reaches 0.7333 and passes the 0.6500 of prompt scaffolding, at sixteen examples 0.8833. At thirty-two examples the two paths overlap in the band 0.9000 to 0.9500, at sixty examples prompt scaffolding gives 0.9667 and fine-tuning stays at 0.9500: fine-tuning does not win. The cost is in two separate units; prompt scaffolding raises the context per request from 28.48 to 623.48 tokens, that is 21.9 times, while fine-tuning is a one-time 180 update steps and moves a 0.2992 share of the 605 weights. In the previous two lessons the weights were always taken as given: a scale was chosen, the number of digits was lowered, and memory and accuracy were written side by side. The weights themselves were never touched. Yet this is the real possibility of a local setup that a distant endpoint does not have; because the weights are in your hands, they can be changed. The table at the end of the second lesson imposed a choice: when the budget forces the small scale, accuracy on this task stays low. There are two ways to raise it, and both need **the same thing**, labeled examples. The first way is to **put the examples in the prompt**; with every request the model is shown a few solved tickets. The second way is to **write the examples into the weights**; the model is trained once and the prompt stays short. This lesson measures both at the same example count and shows the point where fine-tuning **does not win**. The two-stage structure of pretraining and fine-tuning was covered in the architectures topic of the Introduction to Machine Learning course and in the language models topic of the Introduction to AI Engineering course; that distinction is not repeated here, only the numbers of the two paths are compared. - **LA21.** The code is a **simulator**; the tickets are **fictional**, the seed is **20260218**. The scale is the **small** scale of the previous lessons, because the budget forced it. - **LA22.** The first 60 of the 120 fictional tickets are the **training pool**, the last 60 are the **test set**. No scaffolding touches the test set. - **LA23. Prompt scaffolding:** the chosen examples are put into the prompt. An example in the context gives its own pieces a **temporary** support for its own topic; the support is not carried from request to request. - **LA24. Fine-tuning:** an **error-driven** update on the same examples. Weights move only on examples the model knows wrongly, and the weights that move are **permanent**. - **LA25.** The number of epochs is how many times the training set is passed through from start to end; the number of update steps is the example count times the number of epochs. - **LA26.** The two paths use **the same examples**. The comparison is at equal labeled example counts; the only thing that changes is where the example is put. - **LA27.** The two runs are **two separate draws** of the examples (seed 20260218 and 20260219). At sixty examples the two draws give the same set, so the difference there is zero. - **LA28. Gain rule:** the superiority of a path does not count as measured unless it points the same way in **both** draws. - **LA29.** The cost is in two separate units: prompt scaffolding is **tokens per request**, fine-tuning is **a one-time example count and epoch count**. - **LA30.** The weight table is 121 rows times five topics, that is **605 weights**; the number of records fine-tuning moves is counted out of this. ## The Same Examples, Two Different Places The simulator is the same as in the previous lessons; the only change is that a **correction table** can be added on top of the score. If the correction comes from the prompt it is temporary and goes away with the context of that request; if it comes from training it is permanent and the prompt stays short. The input of both is the same labeled examples. ```python # SIMULATOR -- FICTIONAL. The simulator of the previous lessons, built in shortened form. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] TEMPLATES = {"invoice": ["bu ayki tutar gecen aya gore cok yuksek", "odedim ama sistemde borc gorunuyor", "ayrintili dokumu nereden gorebilirim"], "breakdown": ["evde su akmiyor bir sorun var", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "reading": ["okuma yanlis girilmis endeks tutmuyor", "cihaz degisimi icin randevu istiyorum", "endeksi ile fatura ayni degil"], "tariff": ["hangi basamaktayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmaliyim", "birim ucret abone tipine gore degisir mi"], "outage": ["mahallede elektrik yok ne zaman gelecek", "planli duyuru nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen hafta da aradim", "kayit numarasi verildi", "ekip bilgi verecekti", "degisiklik talebim de var"] TICKETS = [] # 120 FICTIONAL support tickets for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "topic": k, "text": m}) PIECE_TOPICS = {} for k, items in TEMPLATES.items(): for m in items: for t in set(tokenize(m)): PIECE_TOPICS.setdefault(t, {})[k] = 1 for e in EXTRAS: for t in set(tokenize(e)): PIECE_TOPICS.setdefault(t, {}) BASE, WEIGHTS = 0.50, {} for i, t in enumerate(sorted(PIECE_TOPICS)): r = rng(SEED + 7919 * (i + 1)) row = {k: BASE for k in TOPICS} if PIECE_TOPICS[t]: for k in PIECE_TOPICS[t]: row[k] = BASE + 0.05 + 0.40 * r() else: row[TOPICS[int(r() * 5)]] = BASE + 0.02 + 0.10 * r() WEIGHTS[t] = row ROWS = 12 # small scale: the scale the budget chose LEARNED = set(sorted(WEIGHTS, key=lambda t: (BASE - max(WEIGHTS[t].values()), t))[:ROWS]) TRAIN, TEST = TICKETS[:60], TICKETS[60:] INSTRUCTION = "Label this ticket as one of: invoice breakdown reading tariff outage . " def score(text, correction=None): """The base score is the highest value of the learned rows; the correction is added on top.""" pieces = tokenize(text) p = {k: BASE for k in TOPICS} for t in pieces: if t in LEARNED: for k in TOPICS: p[k] = max(p[k], WEIGHTS[t][k]) for t in pieces: for k in TOPICS: p[k] += (correction or {}).get((t, k), 0.0) return max(sorted(p), key=lambda a: p[a]) def measure(correction=None): return sum(score(x["text"], correction) == x["topic"] for x in TEST) / 60 print("training pool", len(TRAIN), "records ; test", len(TEST), "records") print("baseline accuracy (small scale without scaffolding):", f"{measure():.4f}") print("instruction tokens:", len(tokenize(INSTRUCTION)), "; mean ticket tokens:", round(sum(len(tokenize(x["text"])) for x in TEST) / 60, 2)) ``` ``` training pool 60 records ; test 60 records baseline accuracy (small scale without scaffolding): 0.6000 instruction tokens: 18 ; mean ticket tokens: 10.48 ``` The baseline is **0.6000**. This is the state, on this task, of the scale the budget imposed at the end of the second lesson, and it is the number both paths have to beat. An average ticket is 10.48 tokens and the instruction is 18 tokens; that is, a request without scaffolding is about 28 tokens. This number will be the yardstick when reading the cost of prompt scaffolding. ## As the Example Count Grows The code below puts the same example set in both places and measures only on the test set. The examples are chosen from the training pool with two separate draws; the second draw is there to decide whether a gain counts as measured. ```python EXAMPLE, STEP = 0.30, 0.05 # support from the context, and the fine-tuning step def pick_examples(n, seed): r, h = rng(seed), list(range(60)) for i in range(59, 0, -1): j = int(r() * (i + 1)) h[i], h[j] = h[j], h[i] return [TRAIN[i] for i in sorted(h[:n])] def from_context(examples): # examples put into the prompt: temporary support return {(t, o["topic"]): EXAMPLE for o in examples for t in tokenize(o["text"])} def train_model(examples, epochs): # error-driven update; the weights are permanent D = {} for _ in range(epochs): for o in examples: s = score(o["text"], D) if s != o["topic"]: for t in tokenize(o["text"]): D[(t, o["topic"])] = D.get((t, o["topic"]), 0.0) + STEP D[(t, s)] = D.get((t, s), 0.0) - STEP return D print(" n prompt A prompt B tune k=1 tune k=3 tuneB k=3 prompt tok updated") for n in (4, 8, 16, 32, 60): a, b = pick_examples(n, SEED), pick_examples(n, SEED + 1) ia, ib = measure(from_context(a)), measure(from_context(b)) y1, y3, yb = measure(train_model(a, 1)), measure(train_model(a, 3)), measure(train_model(b, 3)) tok = sum(len(tokenize(o["text"])) for o in a) print(f"{n:>2} {ia:>8.4f} {ib:>8.4f} {y1:>8.4f} {y3:>8.4f} {yb:>9.4f}" f" {tok:>10} {len(train_model(a, 3)):>7}") print("\nn=60 fixed , number of epochs varies:") print(" k accuracy update steps updated records") for k in (1, 2, 3, 6, 10): D = train_model(pick_examples(60, SEED), k) print(f" {k:>2} {measure(D):>8.4f} {60 * k:>12} {len(D):>15}") mean = sum(len(tokenize(x["text"])) for x in TEST) / 60 tok60 = sum(len(tokenize(o["text"])) for o in pick_examples(60, SEED)) with_scaffolding = len(tokenize(INSTRUCTION)) + tok60 + mean tuned = len(tokenize(INSTRUCTION)) + mean D60 = train_model(pick_examples(60, SEED), 3) print(f"\ncontext per request: prompt scaffolding {with_scaffolding:.2f} , fine-tuning {tuned:.2f} tokens" f" ({with_scaffolding / tuned:.1f} times)") print(f"cost of fine-tuning n=60 , k=3: {60 * 3} update steps , {len(D60)} records ," f" {len(D60) / 605:.4f} share of the 605 weights") ``` ``` n prompt A prompt B tune k=1 tune k=3 tuneB k=3 prompt tok updated 4 0.5667 0.6500 0.6000 0.6000 0.7333 38 0 8 0.6500 0.6667 0.7333 0.7333 0.8000 73 35 16 0.8333 0.7500 0.8833 0.8833 0.8667 138 49 32 0.9000 0.9500 0.8500 0.9167 0.9000 292 101 60 0.9667 0.9667 0.9333 0.9500 0.9500 595 181 n=60 fixed , number of epochs varies: k accuracy update steps updated records 1 0.9333 60 121 2 0.9333 120 162 3 0.9500 180 181 6 0.9500 360 181 10 0.9500 600 181 context per request: prompt scaffolding 623.48 , fine-tuning 28.48 tokens (21.9 times) cost of fine-tuning n=60 , k=3: 180 update steps , 181 records , 0.2992 share of the 605 weights ``` The top row is where neither path works. With four examples, fine-tuning **moves no weight**: the number of updated records is **0**, because the model already knew those four examples correctly and an error is needed to trigger an error-driven update. Accuracy stays at the baseline, at 0.6000. When the same four examples are put into the prompt, on the other hand, accuracy **falls** to **0.5667**; the four examples in the context pull a part of the test records toward their own topics. In the second draw the same two numbers are 0.6500 and 0.7333, that is, not even the direction is the same. With four examples nothing has been measured. At eight and sixteen examples fine-tuning is clearly ahead. At eight it is **0.7333** and **0.8000**, above 0.6667, the best of prompt scaffolding in the two draws; at sixteen it is **0.8833** and 0.8667, above 0.8333 and 0.7500 of prompt scaffolding. Because it points the same way in both draws, this superiority counts as measured. The reason is structural: an error-driven update uses an example **several times** and corrects only the places that are known wrongly, while an example put into the context is read once and also moves the places that were already known correctly. ## The Point Where Fine-Tuning Does Not Win At thirty-two examples the table turns. Fine-tuning with three epochs gives **0.9167** and **0.9000**, prompt scaffolding gives **0.9000** and **0.9500**. The bands overlap and the superiority does not point the same way in the two draws; according to LA28 there is no measured gain here. In the same row, fine-tuning with a single epoch is **0.8500**, that is, below both values of prompt scaffolding. Raising the epochs to three gains 0.0667, and this is a gain that comes not from adding examples but from **passing over the same examples again**. At sixty examples no uncertainty is left. Because both draws select the whole pool, the numbers are the same: prompt scaffolding **0.9667**, fine-tuning **0.9500**. The difference is **0.0167** and the run-to-run variation is **zero**. This is the point where fine-tuning **does not win**, and it is not chance. If all sixty examples sit in the context, the model reads every request together with a solved set; fine-tuning, on the other hand, has used from the same examples only the ones that **gave errors**, has moved 181 of the 605 weights and has left the rest untouched. Squeezing all the labeled data you have into a single training run is a loss on this task. More epochs cannot close this gap. When sixty examples are held fixed and the epochs are increased, accuracy is **0.9333** at one epoch, **0.9500** at three, and stays there; six and ten epochs give the same 0.9500. Because after the third epoch no errors are left to correct in the training set, the number of updated records also settles at **181**. The update steps go from 180 to 600 and nothing is received in return. The ceiling of fine-tuning on this task is 0.9500, and that ceiling rises not by adding epochs but only by another way. What decides is not the accuracy column but the last two rows. With sixty examples, prompt scaffolding raises the context of the request from 28.48 tokens to **623.48** tokens, that is, to **21.9 times**, and this cost is paid again on every request. According to the computation of the first lesson, context length is proportional to the intermediate state per request; twenty-one times the context means twenty-one times fewer concurrent requests in the same budget. The cost of fine-tuning, on the other hand, is **one-time**: sixty labeled examples, three epochs, 180 update steps and a 0.2992 share of weights that move. Every later request is still 28.48 tokens. The comparison therefore does not close with a single number. When labeled examples are few, fine-tuning is both ahead and cheaper. As examples multiply, prompt scaffolding catches up and passes in accuracy, but it does so by enlarging the context; if how many requests are processed at the same time matters, an accuracy difference of 0.0167 is not worth twenty-one times the capacity. What has to be measured is both, and these two numbers are not in the same unit. ## Summary - Both paths ask for the same input: labeled examples. The only thing that changes is whether the example is put into the prompt or into the weights; so the comparison is made at equal example counts. - With four examples fine-tuning moves no weight and the baseline stays at 0.6000; when the same four examples are put into the prompt, accuracy falls to 0.5667. The direction of the two draws is not even the same. - At eight and sixteen examples fine-tuning is ahead in both draws: 0.7333 and 0.8833 against 0.6500 and 0.8333. This superiority counts as measured. - At thirty-two examples the bands overlap (0.9000 and 0.9500 against 0.9000 and 0.9167); there is no measured gain, and single-epoch fine-tuning falls behind with 0.8500. - At sixty examples prompt scaffolding is 0.9667 and fine-tuning 0.9500; because the run-to-run variation is zero, the loss of fine-tuning is real. - The costs are not in the same unit: prompt scaffolding raises the context 21.9 times per request, fine-tuning moves a one-time 180 update steps and a 0.2992 share of the 605 weights. ## Next Step The 0.2992 share in the last row was passed over without being asked about. Fine-tuning did not **decide** which weights would move; the error-driven update touched whichever piece passed through an example known wrongly, and 181 records came out that way. This number is a by-product, not a design decision. Yet the number of weights that move is directly a cost: it is the thing that will be stored, carried, rolled back and kept separately for every task. Could the same accuracy be obtained by moving fewer weights, what would be lost if it were decided beforehand which weights to freeze, and how many numbers would be stored if the update itself were written onto the table as an **add-on**: none of these was tried. The next lesson measures side by side the full update, layer freezing and a low-rank correction added to the table, and prints two numbers: the updated-parameter share and accuracy.