Self-Hosted Models

In the previous twenty lessons the model was always an endpoint standing far away. Everything measured was read from the surface of that endpoint: how many calls went out, how many tokens were spent, how many requests were rejected, how many responses were cut short. All of those numbers shared one assumption: the model itself sits on someone else’s machine, and how many requests are processed there at the same time is not your problem.

When the model moves to your own hardware, that assumption falls and the unit changes. What is spent is no longer tokens but memory, and memory is not a growing bill but a fixed ceiling. This lesson counts that ceiling: how many values the weights of a scale hold in memory, how many values the intermediate state of one request adds, and how many concurrent requests a given budget affords. The memory accounting on the training side was measured in the Training Practice topic of the Deep Learning course; that procedure is not repeated here, only its counterpart on the inference side is built.

• LA1. The code is a simulator; no real model, runtime or hardware is called. The tickets and the scales are fictional, the seed is 20260218.
• LA2. Memory is counted as a count of values. How many digits a value is stored with is the subject of the next lesson and is held fixed here.
• LA3. A scale is given by three structural quantities: the number of layers, the hidden width and the vocabulary size. The parameter count is computed from them.
• LA4. The simulator’s own weight table is small and exists only to produce accuracy. The memory accounting is done from the structural description of the scales, not from the simulator’s table.
• LA5. Weight memory is independent of the requests and is a single copy. The intermediate state is separate for each request and is directly proportional to the context length.
• LA6. The intermediate state keeps two arrays of width size in every layer for every token in the context; that is, per request two times layers times width times context values.
• LA7. As the scale grows, the number of distinguishing rows of the simulator grows (12, 26, 121). This models capability growing with scale and carries no other claim.
• LA8. With the distribution temperature at zero the most likely choice is made and the output is the same from run to run; above zero, seed-dependent sampling is performed.
• LA9. The budget is three billion values; weights and intermediate state are paid for from the same budget.
• LA10. The units measured are the count of values and the number of concurrent requests; raw time is not written.

Three Scales, Three Capabilities

The simulator splits a text into pieces of five characters and keeps a weight row for each piece. The row has a base shared by all five topics; what distinguishes is the small difference added on top of the base for each topic. While reading a ticket, the model takes the highest value it sees for each topic and picks the topic with the highest score. Scale changes only one thing here: how many pieces have a distinguishing row.

# SIMULATOR -- FICTIONAL. There is no real model, runtime or hardware;
# memory is counted as a count of values.
import math

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, a piece is at most five characters
    b = []
    for s in m.lower().split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def text_hash(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"]
TEMPLATES = {"invoice": ["bu ayki tutar gecen aya gore cok yuksek",
                         "odedim ama sistemde borc gorunuyor",
                         "ayrintili dokumu nereden gorebilirim"],
             "breakdown": ["evde su akmiyor bir sorun var",
                           "baglanti bozuk oldu ve hic calismiyor",
                           "ariza kaydi actim ama ekip gelmedi"],
             "reading": ["okuma yanlis girilmis endeks tutmuyor",
                         "cihaz degisimi icin randevu istiyorum",
                         "endeksi ile fatura ayni degil"],
             "tariff": ["hangi basamaktayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmaliyim",
                        "birim ucret abone tipine gore degisir mi"],
             "outage": ["mahallede elektrik yok ne zaman gelecek",
                        "planli duyuru nereden takip edilir",
                        "aksamdan beri kesinti yasiyoruz"]}
EXTRAS = ["ayrica gecen hafta da aradim", "kayit numarasi verildi",
          "ekip bilgi verecekti", "degisiklik talebim de var"]
TICKETS = []                            # 120 FICTIONAL support tickets
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "topic": k, "text": m})

PIECE_TOPICS = {}                       # which topic's template each piece occurs in
for k, items in TEMPLATES.items():
    for m in items:
        for t in set(tokenize(m)):
            PIECE_TOPICS.setdefault(t, {})[k] = 1
for e in EXTRAS:
    for t in set(tokenize(e)):
        PIECE_TOPICS.setdefault(t, {})

BASE = 0.50                             # weight of a piece shared by all topics
WEIGHTS = {}
for i, t in enumerate(sorted(PIECE_TOPICS)):
    r = rng(SEED + 7919 * (i + 1))
    row = {k: BASE for k in TOPICS}
    if PIECE_TOPICS[t]:
        for k in PIECE_TOPICS[t]:
            row[k] = BASE + 0.05 + 0.40 * r()    # small distinguishing difference
    else:
        row[TOPICS[int(r() * 5)]] = BASE + 0.02 + 0.10 * r()   # misleading link
    WEIGHTS[t] = row
ORDER = sorted(WEIGHTS, key=lambda t: (BASE - max(WEIGHTS[t].values()), t))

SCALES = {"small": (12, 512, 12), "medium": (24, 1024, 26),
          "large": (40, 2048, len(WEIGHTS))}        # layers, width, distinguishing rows


def score(text, rows):
    learned = set(ORDER[:rows])
    p = {k: BASE for k in TOPICS}
    for t in tokenize(text):
        if t in learned:
            for k in TOPICS:
                p[k] = max(p[k], WEIGHTS[t][k])
    return p


def respond(text, scale, temperature=0.0, seed=SEED):
    p = score(text, SCALES[scale][2])
    if temperature <= 0.0:
        return max(sorted(p), key=lambda a: p[a])   # most likely choice
    u = [math.exp((p[k] - max(p.values())) / temperature) for k in TOPICS]
    return TOPICS[pick(rng(seed ^ text_hash(text))(), [x / sum(u) for x in u])]


print("vocabulary", len(WEIGHTS), "pieces ; sample ticket:", TICKETS[0]["text"])
print("scale   accuracy")
for name in SCALES:
    d = sum(respond(x["text"], name) == x["topic"] for x in TICKETS)
    print(f"{name:<7} {d / 120:.4f}")
print("\ntemperature  seed 20260218  seed 20260219")
for s in (0.0, 0.02):
    a, b = (sum(respond(x["text"], "large", s, t) == x["topic"]
                for x in TICKETS) / 120 for t in (SEED, SEED + 1))
    print(f"{s:>11.2f}  {a:>13.4f}  {b:>13.4f}")

vocabulary 121 pieces ; sample ticket: planli duyuru nereden takip edilir
scale   accuracy
small   0.6333
medium  0.8167
large   0.8667

temperature  seed 20260218  seed 20260219
       0.00         0.8667         0.8667
       0.02         0.8167         0.8583

The first table says what the scale buys. The small scale assigns 0.6333 of the hundred and twenty tickets to the right topic, the medium 0.8167, the large 0.8667. The difference between them is not capability itself but how many distinctions have been learned: the small scale knows which topic only twelve pieces point to, and the remaining one hundred and nine pieces are indistinguishable from one another for it.

The second table shows a side of self-hosting that an endpoint far away does not have. With the distribution temperature at zero, two different seeds give the same 0.8667; the response to no ticket changes. When the temperature rises to 0.02, the same two runs give 0.8167 and 0.8583. The sampling setting is not in the provider’s hands but in yours; determinism becomes not a request but a configuration. This is also the reason the numbers in the following sections stay fixed from run to run.

Memory Budget and Concurrent Requests

Memory consists of two separate items, and the two behave very differently. Weight memory holds the parameters of the model; it is a single copy no matter how many requests arrive, and it does not change. The intermediate state is kept for the context of each request; it grows with the number of requests and with the context length. The budget has to cover both, so the number of concurrent requests is determined by what remains of the budget after the weights are subtracted.

BUDGET, VOCAB = 3_000_000_000, 32000    # memory budget and FICTIONAL structural vocabulary
CONTEXTS = (240, 960, 3840)


def parameter_count(L, H):
    return 12 * L * H * H + 2 * VOCAB * H


def intermediate_state(L, H, context):
    return 2 * L * H * context


print("scale   weights(count)  state/request  concurrent  accuracy  capacity")
for name, (L, H, rows) in SCALES.items():
    a, d = parameter_count(L, H), intermediate_state(L, H, 960)
    i = sum(respond(x["text"], name) == x["topic"] for x in TICKETS) / 120
    e = max(0, (BUDGET - a) // d)
    print(f"{name:<7} {a:>14} {d:>14} {e:>11} {i:>9.4f} {e * i:>9.1f}")
print("\nscale   " + "  ".join(f"context {b:>4}" for b in CONTEXTS))
for name, (L, H, rows) in SCALES.items():
    a = parameter_count(L, H)
    e = [max(0, (BUDGET - a) // intermediate_state(L, H, b)) for b in CONTEXTS]
    print(f"{name:<7} " + "  ".join(f"{x:>12}" for x in e))
print("\nrounds for 120 requests (context 960):")
for name, (L, H, rows) in SCALES.items():
    e = max(0, (BUDGET - parameter_count(L, H)) // intermediate_state(L, H, 960))
    print(f"  {name:<7} concurrent {e:>4}  round {-(-120 // e):>3}")
print("if the budget drops to 2000000000 values, the large scale cannot be loaded:",
      parameter_count(40, 2048) > 2_000_000_000)

scale   weights(count)  state/request  concurrent  accuracy  capacity
small         70516736       11796480         248    0.6333     157.1
medium       367525888       47185920          55    0.8167      44.9
large       2144337920      157286400           5    0.8667       4.3

scale   context  240  context  960  context 3840
small            993           248            62
medium           223            55            13
large             21             5             1

rounds for 120 requests (context 960):
  small   concurrent  248  round   1
  medium  concurrent   55  round   3
  large   concurrent    5  round  24
if the budget drops to 2000000000 values, the large scale cannot be loaded: True

The first column gives the distance between the scales: weight memory is 70.5 million, 367.5 million and 2.14 billion values. When the width doubles, the layer parameters quadruple, because the parameter count of each layer grows with the square of the width. The effect of the number of layers, on the other hand, is linear. The difference between these two growth rates explains why the choice of scale is not gradual but jumps.

The second column is less intuitive. The intermediate state grows with the width itself, not its square, but it is directly proportional to the context length. In the large scale, a context of 960 tokens asks for 157.3 million values per request; that is one thirteenth of the weights of the same scale. Thirteen concurrent requests consume the same memory as loading a second model.

The Choice the Budget Imposes

The concurrent request column turns the table into a choice. The three-billion-value budget can carry 248 requests at the same time in the small scale, 55 in the medium scale and 5 in the large scale, with a context of 960 tokens. Moving to the large scale raises accuracy from 0.8167 to 0.8667; the gain is 0.0500. The cost of the same step is that concurrent requests drop to one tenth.

The last column reduces these two numbers to a single measure. The product of concurrent requests and accuracy is the correct-response capacity the budget can produce at the same time: 157.1 in the small scale, 44.9 in the medium, 4.3 in the large. The largest scale gives the highest accuracy and produces one thirty-sixth of the correct-response capacity. Which column wins depends on the workload: if the number of incoming tickets is below what the budget can carry, the accuracy column decides; if it is above, the capacity column decides. This is a question never asked at a distant endpoint, because there concurrency is paid for from the provider’s budget and the bill shows up only as tokens.

The second table shows the same tradeoff on the context axis. When the context goes from 240 to 3840, concurrent requests fall to one sixteenth in every scale; in the large scale to 1, that is, to queueing. Context length looks like a quality setting but in a local setup it is directly a capacity setting.

The last line writes the hard side of the ceiling. If the budget drops to two billion values, the large scale cannot be loaded; the number of concurrent requests does not fall, it becomes zero. At a distant endpoint, overload produces a rate limit response and is met with a retry; in a local setup the counterpart is never starting. The two failures do not have the same form, and the second has no retry.

Concurrency Is a Latency Class

The number of concurrent requests alone is a capacity measure; applied to a workload it turns into a latency class. When all one hundred and twenty fictional tickets are queued, the small scale serves them all in one round, the medium scale needs three, the large scale twenty-four rounds. The round count is not raw time and does not vary with hardware; it is a number, fixed from run to run, that comes out of the ratio of the budget to the intermediate state. This is exactly what is needed to measure how long the last ticket in line waits: a queue of twenty-four rounds is not in the same latency class as a queue of one round.

The question of which column a runtime choice touches in this table and which it cannot touch is answered from here as well. The weight column comes from the structural description of the scale and does not depend on which runtime is chosen. The only item a runtime can change is how the intermediate state is held: whether the part of the context shared between requests is shared, whether the place of a finished request is freed immediately, how many requests are processed together in a round. All of these move the second column, and therefore the number of concurrent requests and the round count. The measurable way to compare one runtime with another is also to put these two numbers side by side at the same budget and the same context length; the column is compared, not the name.

Summary

• In self-hosting the unit is the count of values, not tokens; memory is not a growing bill but a fixed ceiling.
• Weight memory is a single copy and independent of the requests; the intermediate state is separate for each request and grows in direct proportion to the context length.
• The weights of the three fictional scales are 70.5 million, 367.5 million and 2.14 billion values; when the width doubles, the layer parameters quadruple.
• A three-billion-value budget affords 248, 55 and 5 concurrent requests in a 960-token context; accuracy is 0.6333, 0.8167 and 0.8667.
• The product of concurrent requests and accuracy falls from 157.1 to 4.3: the scale with the highest accuracy yields one thirty-sixth of the correct-response capacity.
• With the distribution temperature at zero both runs give the same 0.8667; in a local setup determinism is a configuration decision.

Next Step

All of this lesson’s accounting rested on a single assumption: memory was counted as a count of values, and how much room a value takes in memory was assumed fixed. That assumption can be lifted. If a value is stored with fewer digits, the same parameter count takes less room, and a larger scale or more concurrent requests fit into the same budget. In return, the small differences the model rests on become coarser, and what distinguished the weight rows of this lesson was exactly those small differences. The next lesson lowers the number of digits at three levels and prints two numbers side by side: how much the memory falls, at what point accuracy falls, and on which task it falls first.