Not getting enough tokens per second

Your local model runs, but the words crawl. Before buying anything, know that speed has one law and a short list of levers — and one silent killer that is the usual culprit.

The law: every token reads the whole model. Generating a token requires streaming essentially all of the model's weights through the processor, so your ceiling is memory bandwidth divided by model size. A machine with twice the bandwidth is roughly twice as fast on the same model; a model half the size is roughly twice as fast on the same machine. Everything below is a way of changing one of those two numbers.

The silent killer: offloading. If the model doesn't quite fit in graphics memory, the tools helpfully spill the remainder into ordinary RAM — and speed falls off a cliff, often to a tenth. This failure is quiet: everything still works, just miserably. The fix is not a faster machine, it is fitting: a smaller quantization of the same model that lives entirely in VRAM beats a bigger quant that spills, almost regardless of the quality difference. Check your machine against the model's sizes in the fit checker.

The lever people forget: context. The working memory for a long conversation grows token by token, and past a point it both eats your fit headroom and slows every step. If a chat gets slower the longer it runs, that is not your imagination — start a fresh conversation, or trim what the tool resends each turn.

The honest lever: a smaller model. Model quality per parameter has improved fast; the mid-sized models of this year often match the large ones of last year at several times the speed. Check what a step down actually costs you on the catalogue's ratings before assuming you need the big one.

When it really is the hardware. If the model fits, the context is short, the quant is sensible and it is still too slow — that is the bandwidth law talking, and the hardware index shows exactly which machines move it. Bandwidth is the number to shop on, not compute; the buying guides go through it per platform.

Where next: Check your machine · VRAM and what fits · What is quantization