Running Local LLMs, Privacy Without the Cloud

Running a model locally is often framed as a compromised version of the hosted experience. It is more usefully understood as a different set of constraints, where two hardware properties decide almost everything and the processor is not either of them.

Memory capacity determines whether a model runs at all. Memory bandwidth determines how fast it produces tokens. Everything else is secondary.

The capacity arithmetic

A model's weights must be held in memory to be used, and the space required is the parameter count multiplied by the bytes used per parameter.

Weights are not the whole requirement. The attention cache grows with conversation length and must also be held, and working values during a forward pass need room, so sizing a deployment to exactly the weight figure produces a system that loads and then fails partway through a long conversation.

What quantisation costs

Storing each weight with fewer bits loses information, and the question is how much accuracy that costs.

The relationship is not linear. Reducing from sixteen bits to eight is close to free for most purposes. Four bits is where the trade becomes visible, with modest degradation that is acceptable for many tasks. Below that, quality falls away quickly and the saved memory stops being worth it.

A useful consequence, frequently missed, is that a larger model quantised aggressively usually outperforms a smaller model at full precision when both occupy the same memory. Parameter count carries more capability than per weight precision does, so given a fixed budget the better choice is generally the bigger model compressed harder.

Why bandwidth sets the speed

Generating each token requires reading the model's weights from memory. The arithmetic performed on them is small in comparison, which makes generation a memory bound operation rather than a compute bound one.

This gives a rough but reliable estimate. Divide the available memory bandwidth by the size of the model in memory and the result approximates the tokens per second achievable. It also explains why quantisation improves speed as well as capacity, since a smaller model means fewer bytes read per token.

It explains the hardware landscape too. A discrete graphics card offers very high bandwidth but limited memory, so it is fast on models that fit and cannot run those that do not. Systems with unified memory shared between processor and accelerator offer far more capacity at lower bandwidth, so they run large models at a moderate pace. General purpose system memory has an order of magnitude less bandwidth again, which is why processor only inference works and is slow, in a way no amount of processor performance corrects.

What local models handle well

Realistic expectations matter more here than in most comparisons, because the gap is uneven rather than uniform.

Models in the seven to fourteen billion parameter range handle summarising, extracting structured fields from text, classifying, rewriting, and completing common code patterns competently. These are the high volume, repetitive tasks that constitute the bulk of most workloads, and running them locally removes a per call cost entirely.

They are noticeably weaker at extended multi step reasoning, at tasks requiring broad factual recall, and at holding a long context coherently. Larger models run locally close much of that gap, at a hardware cost that has to be justified.

Context length is the other capacity limit

A model may advertise a long context window and still be unable to use it on your hardware, because the attention cache holding that conversation has to fit in memory alongside the weights.

The cache grows with the length of the conversation, so a machine comfortably running a model on short prompts can exhaust its memory partway through a long document. The failure arrives during use rather than at load time, which makes it confusing the first time it happens.

The practical consequence is to size for the longest conversation the system will realistically see rather than for the weights alone, and to be sceptical of advertised context lengths on constrained hardware. Serving software can help by limiting how many conversations are held simultaneously, which trades concurrency for length.

What is actually gained

Privacy is the obvious benefit and it is absolute rather than contractual. Content that never leaves the machine has no retention policy, no terms of service, no jurisdictional question, and no possibility of appearing in an incident somewhere else. For material that genuinely cannot leave, this is the only complete answer.

Three other benefits are less discussed and often decisive. There is no rate limit, so throughput is bounded by hardware rather than by a quota. There is no external dependency, so the system works during a provider outage or without connectivity. And the marginal cost of a request is electricity, which changes what is economically sensible at high volume.

What it costs

Hardware is capital rather than usage, and it is spent before any value is produced. Electricity is ongoing and non trivial when the hardware is busy.

The less visible cost is operational. Models must be selected, obtained, updated, and evaluated. The serving software changes quickly. Nothing improves on its own, and a locally deployed model is exactly as capable in a year as it is today, whereas a hosted one improves without any action.

For most organisations the sensible arrangement is neither one nor the other. Route sensitive and high volume work to local hardware, and send the genuinely difficult minority to a hosted model, which places each workload where its constraints actually point.

Note: before buying hardware, calculate the memory requirement for the model you intend to run at the quantisation you intend to use, then add headroom for the attention cache. Capacity decides what is possible and bandwidth decides whether it is pleasant.