Quantization

In the previous lesson memory was counted as a count of values, and how much room a value takes in memory was held fixed. That assumption is a choice, not a necessity. The same weight can be stored with fewer digits; then memory shrinks while the parameter count stays the same, more concurrent requests fit into the same budget, and the ceiling in the last table of the previous lesson rises.

The price is in the same place. What distinguished the weight rows of the previous lesson was not the shared base but the small difference added to the base for each topic. As the number of digits falls, the representable values grow sparse, and past a certain point those small differences are rounded to the same value. This lesson places three digit levels side by side and answers two questions separately: how much the memory falls, at what point accuracy falls and on which task first.

• LA11. The code is a simulator; the simulator of the previous lesson is rebuilt in shortened form. The tickets are fictional, the seed is 20260218.
• LA12. Quantization here is a single operation: the range from zero to two is divided into the equal step given by the number of digits, and each weight is rounded to the nearest step. The only quantity measured is the number of digits.
• LA13. Memory is counted in digits: weight memory is the parameter count times the number of digits. The intermediate state is held with the same number of digits.
• LA14. The budget is 48 billion digits; this is the 16-digit counterpart of the previous lesson’s three-billion-value budget. The scale is fixed and is the large scale of the previous lesson.
• LA15. The same response is read at three resolutions: five-class topic assignment, three-class team routing (billing, field, meter) and two-class urgency distinction. The mappings are fictional.
• LA16. Margin is the robustness of a decision: the distance between the score of the chosen topic and the score of the nearest rival. The rival is defined differently in the three tasks, so the margin is different too.
• LA17. The distribution temperature is zero; all numbers are independent of the run. In place of uncertainty, the margin distribution is measured.
• LA18. Ties are resolved alphabetically. When all weights are rounded to the same value, the model collapses to a single label.
• LA19. The number of digits affects only the weights; tokenization, the vocabulary and the task definitions do not change.
• LA20. The concurrent request count is the same as the previous lesson’s computation; the only input that changes is how many digits a value holds.

What the Number of Digits Frees in Memory

The memory side of quantization is linear and beyond dispute: if the digits are halved, weight memory is halved. The interesting part is the second effect. Because the intermediate state is held with the same number of digits, the budget is freed in two places at once, and the concurrent request count grows faster than the ratio of the digits as the number of digits falls: the weights are subtracted from the budget once, and all of the remainder goes to requests.

# SIMULATOR -- FICTIONAL. The simulator of the previous lesson, built in shortened form.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, a piece is at most five characters
    b = []
    for s in m.lower().split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"]
TEMPLATES = {"invoice": ["bu ayki tutar gecen aya gore cok yuksek",
                         "odedim ama sistemde borc gorunuyor",
                         "ayrintili dokumu nereden gorebilirim"],
             "breakdown": ["evde su akmiyor bir sorun var",
                           "baglanti bozuk oldu ve hic calismiyor",
                           "ariza kaydi actim ama ekip gelmedi"],
             "reading": ["okuma yanlis girilmis endeks tutmuyor",
                         "cihaz degisimi icin randevu istiyorum",
                         "endeksi ile fatura ayni degil"],
             "tariff": ["hangi basamaktayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmaliyim",
                        "birim ucret abone tipine gore degisir mi"],
             "outage": ["mahallede elektrik yok ne zaman gelecek",
                        "planli duyuru nereden takip edilir",
                        "aksamdan beri kesinti yasiyoruz"]}
EXTRAS = ["ayrica gecen hafta da aradim", "kayit numarasi verildi",
          "ekip bilgi verecekti", "degisiklik talebim de var"]
TICKETS = []                            # 120 FICTIONAL support tickets
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "topic": k, "text": m})

PIECE_TOPICS = {}
for k, items in TEMPLATES.items():
    for m in items:
        for t in set(tokenize(m)):
            PIECE_TOPICS.setdefault(t, {})[k] = 1
for e in EXTRAS:
    for t in set(tokenize(e)):
        PIECE_TOPICS.setdefault(t, {})

BASE, WEIGHTS = 0.50, {}
for i, t in enumerate(sorted(PIECE_TOPICS)):
    r = rng(SEED + 7919 * (i + 1))
    row = {k: BASE for k in TOPICS}
    if PIECE_TOPICS[t]:
        for k in PIECE_TOPICS[t]:
            row[k] = BASE + 0.05 + 0.40 * r()
    else:
        row[TOPICS[int(r() * 5)]] = BASE + 0.02 + 0.10 * r()
    WEIGHTS[t] = row

GROUP = {"invoice": "billing", "tariff": "billing", "breakdown": "field",   # team routing
         "outage": "field", "reading": "meter"}
URGENCY = {"invoice": "normal", "tariff": "normal", "reading": "normal",  # urgency distinction
           "breakdown": "urgent", "outage": "urgent"}


def quantize(x, bits):
    """The range from zero to two is divided into 2**bits - 1 equal steps; rounded to the nearest step."""
    a = 2.0 / (2 ** bits - 1)
    return round(x / a) * a


def score(text, bits):
    p = {k: quantize(BASE, bits) for k in TOPICS}
    for t in tokenize(text):
        if t in WEIGHTS:
            for k in TOPICS:
                p[k] = max(p[k], quantize(WEIGHTS[t][k], bits))
    return p


def choose(p):
    return max(sorted(p), key=lambda a: p[a])


PARAMETERS, STATE, BUDGET = 2144337920, 157286400, 48_000_000_000

print("digits    step  weights(digits)  concurrent  five-class  three-class  two-class")
for bits in (16, 12, 8, 6, 5, 4, 3, 2):
    a5 = a3 = a2 = 0
    for x in TICKETS:
        s = choose(score(x["text"], bits))
        a5 += s == x["topic"]
        a3 += GROUP[s] == GROUP[x["topic"]]
        a2 += URGENCY[s] == URGENCY[x["topic"]]
    e = max(0, (BUDGET // bits - PARAMETERS) // STATE)
    print(f"{bits:>6} {2.0 / (2 ** bits - 1):>7.5f} {PARAMETERS * bits:>16} {e:>11}"
          f" {a5 / 120:>11.4f} {a3 / 120:>12.4f} {a2 / 120:>10.4f}")

digits    step  weights(digits)  concurrent  five-class  three-class  two-class
    16 0.00003      34309406720           5      0.8667       0.8667     0.9417
    12 0.00049      25732055040          11      0.8667       0.8667     0.9417
     8 0.00784      17154703360          24      0.8750       0.8750     0.9417
     6 0.03175      12866027520          37      0.8500       0.8750     0.9417
     5 0.06452      10721689600          47      0.8417       0.8667     0.9333
     4 0.13333       8577351680          62      0.8000       0.9083     0.9333
     3 0.28571       6433013760          88      0.6333       0.7583     0.8583
     2 0.66667       4288675840         138      0.2083       0.3583     0.3583

The three main rows of the table are 16, 8 and 4 digits. Memory goes from 34.3 billion to 17.2 billion and from there to 8.6 billion digits; exactly a quarter. Concurrent requests, on the other hand, go from 5 to 24 and from there to 62: twelve times. This disproportion comes from LA13. The weights are subtracted from the budget once and take almost the whole budget at 16 digits; as the digits decrease, the share of the weights shrinks and all of the remainder goes to requests.

The accuracy columns do not move at the same speed. Five-class topic assignment is 0.8667 at 16 digits and 0.8750 at eight digits, that is, the first halving leaves no measurable trace in accuracy. At six and five digits it slips slightly, to 0.8500 and 0.8417. At four digits 0.8000, at three digits 0.6333, at two digits 0.2083. The last row is not an accuracy but a collapse: 0.2083 is the rate at which the simulator falls on the alphabetically first label.

The Margin Falling Below the Step

The reason for this fall is not something to guess but something to compute. The robustness of a decision is the margin: the distance between the score of the chosen topic and the score of its nearest rival. Rounding can move this distance only by about a step; if the margin is larger than the step the decision stays in place, if it is smaller it may vanish. Both measures below are independent of the run and can be looked at before accuracy is measured.

def margins(bits):
    f = {"five-class": [], "three-class": [], "two-class": []}
    for x in TICKETS:
        p = score(x["text"], bits)
        v = sorted(p.values(), reverse=True)
        s = choose(p)
        f["five-class"].append(v[0] - v[1])
        f["three-class"].append(p[s] - max(p[k] for k in TOPICS if GROUP[k] != GROUP[s]))
        f["two-class"].append(p[s] - max(p[k] for k in TOPICS if URGENCY[k] != URGENCY[s]))
    return f


print("task        mean margin   share with margin below the step (digits 8 / 4 / 3)")
for name, f in margins(16).items():
    o = [sum(1 for x in f if x < 2.0 / (2 ** b - 1)) / 120 for b in (8, 4, 3)]
    print(f"{name:<11} {sum(f) / 120:>11.4f}   " + "   ".join(f"{x:.4f}" for x in o))
print("\ndigits  distinct values  rows with distinction erased (of 121 rows)")
for b in (16, 8, 4, 3, 2):
    d = {quantize(w, b) for s in WEIGHTS.values() for w in s.values()}
    z = sum(1 for s in WEIGHTS.values() if len({quantize(w, b) for w in s.values()}) == 1)
    print(f"{b:>6} {len(d):>15} {z:>27}")
print("\nthe 'endek' row at the digit levels:")
for b in (16, 8, 4, 3):
    print(f"  {b:>2} " + " ".join(f"{k}={quantize(WEIGHTS['endek'][k], b):.4f}"
                                  for k in TOPICS))

task        mean margin   share with margin below the step (digits 8 / 4 / 3)
five-class       0.1597   0.0167   0.4167   0.7250
three-class      0.1995   0.0167   0.2750   0.6667
two-class        0.2665   0.0083   0.2000   0.4500

digits  distinct values  rows with distinction erased (of 121 rows)
    16             133                           0
     8              55                           0
     4               4                          24
     3               2                          53
     2               1                         121

the 'endek' row at the digit levels:
  16 invoice=0.5000 breakdown=0.5000 reading=0.7095 tariff=0.5000 outage=0.5000
   8 invoice=0.5020 breakdown=0.5020 reading=0.7059 tariff=0.5020 outage=0.5020
   4 invoice=0.5333 breakdown=0.5333 reading=0.6667 tariff=0.5333 outage=0.5333
   3 invoice=0.5714 breakdown=0.5714 reading=0.5714 tariff=0.5714 outage=0.5714

The first table explains the order of the accuracy table without looking at accuracy. The mean margin is 0.1597 in the five-class task, 0.1995 in the three-class task and 0.2665 in the two-class task. As the task gets coarser the rival moves away, because confusion between invoice and tariff does not break the team routing, and confusion between breakdown and outage does not break the urgency. The same response, read coarsely, is a sturdier decision.

The three columns on the right count this level by level. At eight digits the step is 0.00784 and the margin is below this step in only 0.0167 of the records; that accuracy moves so little at this level is therefore an expected result. At four digits the step rises to 0.13333 and 0.4167 of the records become fragile in the five-class task, and 0.2000 in the two-class task. At three digits the shares are 0.7250 and 0.4500. The order of the fragility share is the same as the order in which accuracy falls, and this order can be computed before the digits are lowered.

The second table shows the mechanism. The 605 weights of the 121 rows carry 133 distinct values at 16 digits; at eight digits this falls to 55 but no row loses its distinction. At four digits the number of representable values falls to 4, and the values that 24 rows hold for the five topics are made equal to one another; those rows no longer distinguish anything. At three digits this number is 53, at two digits 121, that is, the whole table. Quantization does not shrink the weights, it makes them indistinguishable from one another; what is erased is not magnitude but distinction.

The last block shows the same thing in a single row. The endek piece, which points to a meter reading, carries 0.7095 for the meter and 0.5000 for the other four topics at 16 digits; the distinction is 0.2095. At eight digits it is 0.7059 against 0.5020, a distinction of 0.2039, almost the same. At four digits the values settle at 0.6667 and 0.5333 and the distinction falls to 0.1334. At three digits all five values are 0.5714; the row no longer points to the meter. None of the weights was zeroed and none shrank meaningfully; only the distance between them became unrepresentable.

The Point Where Accuracy Falls

The place to read in the table is between four and three digits. Going from eight to four, five-class accuracy slips 0.0750 and memory is halved; going from four to three, accuracy slips 0.1667 and memory falls by only another quarter. The curve of cost and the curve of gain cross here: below four digits, each digit gained comes at a higher price than the one before.

The answer to which task falls first is also in the table. At four digits the five-class distinction falls from 0.8667 to 0.8000, while the three-class team routing stays inside the band between 0.8667 and 0.9083 and the two-class urgency distinction slips from 0.9417 to only 0.9333. Two digits lower, at three digits, the coarse tasks fall too, but far behind the five-class task: 0.7583 and 0.8583. The rule is this: the fine distinction goes first. The same quantized model has become unable to tell which topic a ticket belongs to while still knowing which team to send it to.

The counterpart in design is direct. Whether lowering the number of digits is acceptable in a setup depends not on the model but on the task. If two jobs run on the same weights, running the coarse one with four digits and running the fine one with four digits are not the same decision. What has to be measured is not a single accuracy but a separate accuracy for each task.

Summary

• Quantization here is the division of a range into equal steps; the only quantity measured is the number of digits, and memory is written as parameter count times digits.
• Weight memory is 34.3 billion at 16 digits, 17.2 billion at 8 digits, 8.6 billion at 4 digits; the concurrent requests in the same budget rise from 5 to 24 and to 62, because the weights are subtracted from the budget once.
• Five-class accuracy stays in the 0.8667 band down to 8 digits, falls to 0.8000 at 4 digits, 0.6333 at 3 digits and 0.2083 at 2 digits; the last row is an alphabetical collapse, not an accuracy.
• The mean decision margin is 0.1597 in five classes and 0.2665 in two classes; the share with the margin below the step is 0.4167 and 0.2000 at 4 digits, and this order gives the order in which accuracy falls before the digits are lowered.
• 605 weights fall to 4 distinct values at 4 digits and 24 rows lose all of their distinction; quantization erases distinction, not magnitude.
• The point where accuracy falls is between four and three digits, and the task that falls first is five-class topic assignment; the coarse tasks hold two digits longer.

Next Step

In these two lessons the weights were always taken as given. A scale was chosen, the number of digits was lowered, memory and accuracy were written side by side; but the weights themselves were never touched. Yet this is the real possibility of a local setup that a distant endpoint does not have: because the weights are in your hands, they can be changed. If training is continued on a few hundred examples of your own data, the model’s accuracy on this task may rise. Whether it rises, by how much, and whether the same gain could already be had by adding a few sentences to the prompt were not measured. The next lesson places the prompt-side mechanism and fine-tuning side by side on the same task, writes the cost in example count and run count, and shows a point where fine-tuning does not win.