Every prompt sent to a hosted model becomes a record on infrastructure you do not control, subject to that provider's retention, logging, and training policies. For a great deal of routine use that is an acceptable exchange, but for material that carries obligations, source code under a nondisclosure agreement, client records, unreleased work, the right question is whether the model needs to run off your machine at all, and increasingly the answer is that it does not. A local large language model runs entirely on your own hardware, the weights reside on your disk, inference executes on your CPU or GPU, and nothing you submit crosses the network. Tooling such as Ollama and llama.cpp has made this practical on ordinary machines, you retrieve a model once and run it offline from then on.
The constraint that governs what you can run locally is memory. A model's parameter count, quoted in billions, largely determines how much system memory or video memory it requires, and quantization, storing the weights at reduced precision such as four bits, is the technique that lets larger models fit on consumer hardware at a modest cost in quality. The practical tradeoff is worth knowing before choosing a model.
| MODEL SIZE | WHAT IT NEEDS |
|---|---|
| 3B to 7B | Runs on 8 to 16 GB of memory, workable on a modern laptop. |
| 8B to 13B | Wants 16 to 32 GB, comfortable on a desktop with a mid range card. |
| 30B to 70B | Needs a strong graphics card or 64 GB and up, slow on the processor alone. |
The tradeoff deserves an honest accounting rather than advocacy. A local model in the seven to thirteen billion range will not match the largest hosted frontier models on complex reasoning, long context, or breadth of knowledge, and it will generate more slowly on the same task. What it returns in exchange is meaningful, privacy that does not depend on a vendor's terms of service, availability with no network and no dependency on an external service that can change its pricing or withdraw a model, and a fixed cost rather than a bill that scales with every request. For summarization of private documents, drafting, and a large share of everyday coding assistance, the capability gap is narrower than the marketing around frontier models implies, and for sensitive work the privacy is not a feature to weigh against others, it is the requirement that decides the matter.
Running a model locally is not the right answer for every workload, and pretending otherwise ignores the real advantages of frontier systems on genuinely hard problems. The point is that the choice should be made on the merits of each task rather than by defaulting to the cloud out of habit, weighing the capability you need against the control you are willing to give up. For any work where the data itself is the thing that must be protected, the ability to run a capable model on hardware you own, with nothing leaving your control, is one of the more consequential shifts in how these tools can be deployed, and it moves privacy from something you must trust a provider to honor into something the architecture guarantees.