The KV cache is the new working set
If you want to understand the economics of serving language models, ignore the model for a minute and follow the memory. There is a data structure that grows with every token of every conversation, competes with the model itself for space, and quietly decides what your tokens cost. Old systems people will recognize the silhouette: it is a working set, and the discipline for managing it was invented in the 1960s.
Two phases, one bottleneck
Generation happens in two phases with opposite personalities. First the model reads your whole prompt in one parallel gulp, heavy compute, well fed. Then it writes the answer one token at a time, and each token requires streaming essentially all of the model's weights out of GPU memory to do a comparatively tiny amount of math. That second phase is bandwidth-bound: the arithmetic units idle while bytes arrive. Single conversations cannot fix this. Batches can: serve thirty two conversations at once and every fetched weight gets used thirty two times. Batch size is throughput, almost linearly.
Enter the tenant
But each conversation in the batch must bring its memory: the key and value vectors for every token seen so far, cached so the model does not recompute its entire past per token. On a seven billion parameter model that cache runs about half a megabyte per token, which means a two thousand token conversation carries roughly a gigabyte, growing every step. And it lives in the same GPU memory as the weights. The accounting follows: the memory left over after the model is the housing supply, conversations are tenants, and how many tenants fit is your batch, which is your throughput, which is your unit cost. Serving is property management.
The waste, and the old idea that fixed it
Early serving engines allocated each tenant one contiguous slab sized for the worst case, because nobody knows how long a conversation will run. Published measurements from the team behind the vLLM project found sixty to eighty percent of this memory wasted: reservations never used, plus gaps between slabs too small to fit anyone. The fix was a great systems déjà vu: virtual memory. Chop the cache into fixed-size blocks, let them live scattered, keep a per-conversation table mapping logical positions to physical blocks, and teach the attention computation to follow the table. Fragmentation vanishes, utilization goes to the mid nineties, and reported throughput multiplied by two to four on identical hardware. No new silicon, just better bookkeeping, which is the most repeatable trick in systems history.
What this explains about your bill
Prompt caching discounts are shared walls: identical prompt prefixes across requests can be mapped to the same physical blocks, copy-on-write when a conversation diverges, so providers price the shared part cheaply. Long contexts are expensive because they are large tenants. And latency wobbles under load are often evictions: someone's cache got reclaimed and must be recomputed. Once you see the housing market, provider pricing pages stop looking arbitrary.
The takeaway
The model is the building; conversations are tenants; throughput is occupancy. When serving costs or latency surprise you, ask the landlord questions: how big are the tenants, how full is the building, who is being evicted? The mental model is fifty years old, and it still pays rent.
One level down. The deep version of this lives in the private atlas: 006, the token factory. The same working set, taken apart: block tables, continuous batching, shared walls, preemption. Private atlas, not deployed with this site.