--- title: 'Self-Hosted Models' source: 'https://academia.sh/en/courses/model-apis/self-hosted-models' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:33+00:00' license: 'CC BY-SA 4.0' objectives: - 'Computes weight memory and the intermediate state per request separately as counts of values.' - 'Finds how many concurrent requests a memory budget affords given the parameter count and the context length.' - 'Writes the choice the budget imposes by placing the accuracy and the concurrent request count of the three scales side by side.' - 'Shows with two runs the determinism that self-hosting brings.' --- # Self-Hosted Models Running a model on your own hardware makes the unit memory instead of tokens: the weight memory of three fictional scales is 70.5 million, 367.5 million and 2.14 billion values, the intermediate state per request in a 960-token context is 11.8 million, 47.2 million and 157.3 million values, the concurrent requests a three-billion-value budget affords are 248, 55 and 5, and the label accuracy on the same 120 fictional tickets is 0.6333, 0.8167 and 0.8667. The product of concurrent requests and accuracy falls from 157.1 to 4.3, so the largest scale yields one thirty-sixth of the correct-response capacity. With the distribution temperature at zero both runs give the same 0.8667, and at 0.02 28 of the 120 tickets change from run to run. In the previous twenty lessons the model was always an endpoint standing **far away**. Everything measured was read from the surface of that endpoint: how many calls went out, how many tokens were spent, how many requests were rejected, how many responses were cut short. All of those numbers shared one assumption: the model itself sits on someone else's machine, and how many requests are processed there at the same time is **not your problem**. When the model moves to your own hardware, that assumption falls and the unit changes. What is spent is no longer tokens but **memory**, and memory is not a growing bill but a **fixed ceiling**. This lesson counts that ceiling: how many values the weights of a scale hold in memory, how many values the intermediate state of one request adds, and how many **concurrent requests** a given budget affords. The memory accounting on the training side was measured in the Training Practice topic of the Deep Learning course; that procedure is not repeated here, only its counterpart on the inference side is built. - **LA1.** The code is a **simulator**; no real model, runtime or hardware is called. The tickets and the scales are **fictional**, the seed is **20260218**. - **LA2.** Memory is counted as a **count of values**. How many digits a value is stored with is the subject of the next lesson and is held fixed here. - **LA3.** A scale is given by three structural quantities: the **number of layers**, the **hidden width** and the vocabulary size. The parameter count is computed from them. - **LA4.** The simulator's own weight table is small and exists only to produce **accuracy**. The memory accounting is done from the structural description of the scales, not from the simulator's table. - **LA5.** Weight memory is **independent** of the requests and is a single copy. The **intermediate state** is separate for each request and is directly proportional to the context length. - **LA6.** The intermediate state keeps two arrays of width size in every layer for every token in the context; that is, per request two times layers times width times context values. - **LA7.** As the scale grows, the number of **distinguishing rows** of the simulator grows (12, 26, 121). This models capability growing with scale and carries no other claim. - **LA8.** With the distribution temperature at zero the **most likely choice** is made and the output is the same from run to run; above zero, seed-dependent sampling is performed. - **LA9.** The budget is three billion values; weights and intermediate state are paid for from the **same** budget. - **LA10.** The units measured are the count of values and the number of concurrent requests; raw time is not written. ## Three Scales, Three Capabilities The simulator splits a text into pieces of five characters and keeps a **weight row** for each piece. The row has a base shared by all five topics; what distinguishes is the **small difference** added on top of the base for each topic. While reading a ticket, the model takes the highest value it sees for each topic and picks the topic with the highest score. Scale changes only one thing here: how many pieces have a distinguishing row. ```python # SIMULATOR -- FICTIONAL. There is no real model, runtime or hardware; # memory is counted as a count of values. import math SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def text_hash(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] TEMPLATES = {"invoice": ["bu ayki tutar gecen aya gore cok yuksek", "odedim ama sistemde borc gorunuyor", "ayrintili dokumu nereden gorebilirim"], "breakdown": ["evde su akmiyor bir sorun var", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "reading": ["okuma yanlis girilmis endeks tutmuyor", "cihaz degisimi icin randevu istiyorum", "endeksi ile fatura ayni degil"], "tariff": ["hangi basamaktayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmaliyim", "birim ucret abone tipine gore degisir mi"], "outage": ["mahallede elektrik yok ne zaman gelecek", "planli duyuru nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen hafta da aradim", "kayit numarasi verildi", "ekip bilgi verecekti", "degisiklik talebim de var"] TICKETS = [] # 120 FICTIONAL support tickets for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "topic": k, "text": m}) PIECE_TOPICS = {} # which topic's template each piece occurs in for k, items in TEMPLATES.items(): for m in items: for t in set(tokenize(m)): PIECE_TOPICS.setdefault(t, {})[k] = 1 for e in EXTRAS: for t in set(tokenize(e)): PIECE_TOPICS.setdefault(t, {}) BASE = 0.50 # weight of a piece shared by all topics WEIGHTS = {} for i, t in enumerate(sorted(PIECE_TOPICS)): r = rng(SEED + 7919 * (i + 1)) row = {k: BASE for k in TOPICS} if PIECE_TOPICS[t]: for k in PIECE_TOPICS[t]: row[k] = BASE + 0.05 + 0.40 * r() # small distinguishing difference else: row[TOPICS[int(r() * 5)]] = BASE + 0.02 + 0.10 * r() # misleading link WEIGHTS[t] = row ORDER = sorted(WEIGHTS, key=lambda t: (BASE - max(WEIGHTS[t].values()), t)) SCALES = {"small": (12, 512, 12), "medium": (24, 1024, 26), "large": (40, 2048, len(WEIGHTS))} # layers, width, distinguishing rows def score(text, rows): learned = set(ORDER[:rows]) p = {k: BASE for k in TOPICS} for t in tokenize(text): if t in learned: for k in TOPICS: p[k] = max(p[k], WEIGHTS[t][k]) return p def respond(text, scale, temperature=0.0, seed=SEED): p = score(text, SCALES[scale][2]) if temperature <= 0.0: return max(sorted(p), key=lambda a: p[a]) # most likely choice u = [math.exp((p[k] - max(p.values())) / temperature) for k in TOPICS] return TOPICS[pick(rng(seed ^ text_hash(text))(), [x / sum(u) for x in u])] print("vocabulary", len(WEIGHTS), "pieces ; sample ticket:", TICKETS[0]["text"]) print("scale accuracy") for name in SCALES: d = sum(respond(x["text"], name) == x["topic"] for x in TICKETS) print(f"{name:<7} {d / 120:.4f}") print("\ntemperature seed 20260218 seed 20260219") for s in (0.0, 0.02): a, b = (sum(respond(x["text"], "large", s, t) == x["topic"] for x in TICKETS) / 120 for t in (SEED, SEED + 1)) print(f"{s:>11.2f} {a:>13.4f} {b:>13.4f}") ``` ``` vocabulary 121 pieces ; sample ticket: planli duyuru nereden takip edilir scale accuracy small 0.6333 medium 0.8167 large 0.8667 temperature seed 20260218 seed 20260219 0.00 0.8667 0.8667 0.02 0.8167 0.8583 ``` The first table says what the scale buys. The small scale assigns **0.6333** of the hundred and twenty tickets to the right topic, the medium **0.8167**, the large **0.8667**. The difference between them is not capability itself but how many distinctions have been learned: the small scale knows which topic only twelve pieces point to, and the remaining one hundred and nine pieces are indistinguishable from one another for it. The second table shows a side of self-hosting that an endpoint far away does not have. With the distribution temperature at zero, two different seeds give the same **0.8667**; the response to no ticket changes. When the temperature rises to 0.02, the same two runs give **0.8167** and **0.8583**. The sampling setting is not in the provider's hands but in yours; determinism becomes not a request but a **configuration**. This is also the reason the numbers in the following sections stay fixed from run to run. ## Memory Budget and Concurrent Requests Memory consists of two separate items, and the two behave very differently. **Weight memory** holds the parameters of the model; it is a single copy no matter how many requests arrive, and it does not change. The **intermediate state** is kept for the context of each request; it grows with the number of requests and with the context length. The budget has to cover both, so the number of concurrent requests is determined by what remains of the budget after the weights are subtracted. ```python BUDGET, VOCAB = 3_000_000_000, 32000 # memory budget and FICTIONAL structural vocabulary CONTEXTS = (240, 960, 3840) def parameter_count(L, H): return 12 * L * H * H + 2 * VOCAB * H def intermediate_state(L, H, context): return 2 * L * H * context print("scale weights(count) state/request concurrent accuracy capacity") for name, (L, H, rows) in SCALES.items(): a, d = parameter_count(L, H), intermediate_state(L, H, 960) i = sum(respond(x["text"], name) == x["topic"] for x in TICKETS) / 120 e = max(0, (BUDGET - a) // d) print(f"{name:<7} {a:>14} {d:>14} {e:>11} {i:>9.4f} {e * i:>9.1f}") print("\nscale " + " ".join(f"context {b:>4}" for b in CONTEXTS)) for name, (L, H, rows) in SCALES.items(): a = parameter_count(L, H) e = [max(0, (BUDGET - a) // intermediate_state(L, H, b)) for b in CONTEXTS] print(f"{name:<7} " + " ".join(f"{x:>12}" for x in e)) print("\nrounds for 120 requests (context 960):") for name, (L, H, rows) in SCALES.items(): e = max(0, (BUDGET - parameter_count(L, H)) // intermediate_state(L, H, 960)) print(f" {name:<7} concurrent {e:>4} round {-(-120 // e):>3}") print("if the budget drops to 2000000000 values, the large scale cannot be loaded:", parameter_count(40, 2048) > 2_000_000_000) ``` ``` scale weights(count) state/request concurrent accuracy capacity small 70516736 11796480 248 0.6333 157.1 medium 367525888 47185920 55 0.8167 44.9 large 2144337920 157286400 5 0.8667 4.3 scale context 240 context 960 context 3840 small 993 248 62 medium 223 55 13 large 21 5 1 rounds for 120 requests (context 960): small concurrent 248 round 1 medium concurrent 55 round 3 large concurrent 5 round 24 if the budget drops to 2000000000 values, the large scale cannot be loaded: True ``` The first column gives the distance between the scales: weight memory is **70.5 million**, **367.5 million** and **2.14 billion** values. When the width doubles, the layer parameters quadruple, because the parameter count of each layer grows with the square of the width. The effect of the number of layers, on the other hand, is linear. The difference between these two growth rates explains why the choice of scale is not gradual but jumps. The second column is less intuitive. The intermediate state grows with the width itself, **not its square**, but it is directly proportional to the context length. In the large scale, a context of 960 tokens asks for **157.3 million** values per request; that is one thirteenth of the weights of the same scale. Thirteen concurrent requests consume the same memory as loading a second model. ## The Choice the Budget Imposes The concurrent request column turns the table into a choice. The three-billion-value budget can carry **248** requests at the same time in the small scale, **55** in the medium scale and **5** in the large scale, with a context of 960 tokens. Moving to the large scale raises accuracy from 0.8167 to 0.8667; the gain is **0.0500**. The cost of the same step is that concurrent requests drop to **one tenth**. The last column reduces these two numbers to a single measure. The product of concurrent requests and accuracy is the **correct-response capacity** the budget can produce at the same time: **157.1** in the small scale, **44.9** in the medium, **4.3** in the large. The largest scale gives the highest accuracy and produces **one thirty-sixth** of the correct-response capacity. Which column wins depends on the workload: if the number of incoming tickets is below what the budget can carry, the accuracy column decides; if it is above, the capacity column decides. This is a question never asked at a distant endpoint, because there concurrency is paid for from the provider's budget and the bill shows up only as tokens. The second table shows the same tradeoff on the context axis. When the context goes from 240 to 3840, concurrent requests fall to one sixteenth in every scale; in the large scale to **1**, that is, to queueing. Context length looks like a quality setting but in a local setup it is **directly a capacity setting**. The last line writes the hard side of the ceiling. If the budget drops to two billion values, the large scale cannot be loaded; the number of concurrent requests does not fall, it becomes **zero**. At a distant endpoint, overload produces a rate limit response and is met with a retry; in a local setup the counterpart is **never starting**. The two failures do not have the same form, and the second has no retry. ## Concurrency Is a Latency Class The number of concurrent requests alone is a capacity measure; applied to a workload it turns into a **latency class**. When all one hundred and twenty fictional tickets are queued, the small scale serves them all in **one round**, the medium scale needs **three**, the large scale **twenty-four** rounds. The round count is not raw time and does not vary with hardware; it is a number, fixed from run to run, that comes out of the ratio of the budget to the intermediate state. This is exactly what is needed to measure how long the last ticket in line waits: a queue of twenty-four rounds is not in the same latency class as a queue of one round. The question of which column a runtime choice touches in this table and which it cannot touch is answered from here as well. The weight column comes from the structural description of the scale and does not depend on which runtime is chosen. The only item a runtime can change is **how the intermediate state is held**: whether the part of the context shared between requests is shared, whether the place of a finished request is freed immediately, how many requests are processed together in a round. All of these move the second column, and therefore the number of concurrent requests and the round count. The measurable way to compare one runtime with another is also to put these two numbers side by side at the same budget and the same context length; the column is compared, not the name. ## Summary - In self-hosting the unit is the **count of values**, not tokens; memory is not a growing bill but a fixed ceiling. - Weight memory is a single copy and independent of the requests; the intermediate state is separate for each request and grows in direct proportion to the context length. - The weights of the three fictional scales are 70.5 million, 367.5 million and 2.14 billion values; when the width doubles, the layer parameters quadruple. - A three-billion-value budget affords 248, 55 and 5 concurrent requests in a 960-token context; accuracy is 0.6333, 0.8167 and 0.8667. - The product of concurrent requests and accuracy falls from 157.1 to 4.3: the scale with the highest accuracy yields one thirty-sixth of the correct-response capacity. - With the distribution temperature at zero both runs give the same 0.8667; in a local setup determinism is a configuration decision. ## Next Step All of this lesson's accounting rested on a single assumption: memory was counted as a **count of values**, and how much room a value takes in memory was assumed fixed. That assumption can be lifted. If a value is stored with fewer digits, the same parameter count takes less room, and a larger scale or more concurrent requests fit into the same budget. In return, the **small differences** the model rests on become coarser, and what distinguished the weight rows of this lesson was exactly those small differences. The next lesson lowers the number of digits at three levels and prints two numbers side by side: how much the memory falls, at what point accuracy falls, and **on which task it falls first**.