Run a language model on your own machine, with the network unplugged.

01 · THE CASE

Three good reasons, and the limits nobody mentions.

A local model is worse than a frontier hosted model at almost every task. It is also the only option when the data cannot leave the machine — and that trade is worth understanding precisely rather than emotionally.

What local genuinely gives you

• The data does not leave. No terms of service, no retention policy, no question about training. For client material, medical notes, legal drafts or anything under an NDA, this is the whole argument.
• It works with no connection. On a plane, in a lab, on a site with no signal, or during an outage.
• No per-token cost and no rate limit. Bulk classification, batch summarisation and repeated experiments become free after the electricity.
• It cannot be deprecated underneath you. The model you saved works the same in two years. Hosted models change behaviour without notice.

What it does not give you

• Frontier reasoning. A model that fits in consumer VRAM is not competitive with the largest hosted models on hard reasoning, long-horizon coding or obscure knowledge. Expect a capable assistant, not a replacement.
• Speed on long context. Processing a large document is where consumer hardware feels slowest, and it is exactly what people want local models for.
• Fewer confident errors. Smaller models hallucinate more, not less. Local does not mean careful.
• A free lunch on RAM. Whatever the model occupies is memory your other work does not get.

02 · THE BUDGET

Memory decides everything. Here is the arithmetic.

Model size in billions of parameters, multiplied by the bytes per parameter your quantisation uses, plus room for the context window. That is the entire calculation.

Add headroom on top of every figure. The key-value cache for the context window grows with how much text you feed in, and a long context on a large model can consume several more gigabytes. A model that just fits with an empty context will fail halfway through a long document.

Quantisation, in one paragraph

Weights are stored at reduced precision so the model fits in less memory. Going from 16-bit to 8-bit is nearly free in quality. 8-bit to 4-bit is the sweet spot most people run: a small, usually acceptable quality loss for roughly half the memory. Below 4-bit, degradation becomes obvious — instruction-following slips first, then coherence on long outputs. A larger model at 4-bit generally beats a smaller model at 8-bit, which is the single most useful heuristic in this whole area.

03 · THE TOOLS

Four ways to run the same model files.

All four run the same quantised model files, so the choice is about workflow rather than capability. A common arrangement is Ollama as the engine with a separate chat interface pointed at its API, which keeps the model server and the interface independent.

04 · THE WALKTHROUGH

From nothing to a working model in about ten minutes.

# Pull a model sized to your hardware and start chatting
ollama run llama3.1:8b

# List what you have downloaded, and how much space it uses
ollama list

# Remove one you no longer want
ollama rm llama3.1:8b

# Serve the local API for other tools to use
ollama serve

The API listens on localhost, which is what editor plugins, note apps and scripts connect to. Nothing leaves the machine.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Summarise this in three bullet points: ...",
  "stream": false
}'

Choosing a first model

• General assistant, 8 GB VRAM — a current 7B to 8B instruction-tuned model at 4-bit. This is the default answer and it is a good one.
• Code — a code-specialised model of the same size. Noticeably better at completion and refactoring than a general model of equal size.
• Small and fast — a 3B model for classification, extraction, tagging and other structured work where you do not need conversation.
• Best quality you can fit — go up in parameters before you go up in precision. Larger at 4-bit beats smaller at 8-bit.

05 · TUNING

Four settings that decide whether it feels usable.

• GPU layer offload. The most important setting on a machine where the model does not fully fit. Push as many layers onto the GPU as VRAM allows; the remainder runs on CPU. Going one layer too far causes an out-of-memory failure, so step down until it is stable.
• Context window. Larger contexts cost memory and slow down generation. Set it to what you actually need — 4K for chat, more only when you are genuinely feeding in documents.
• Batch size for prompt processing. Affects how quickly a long prompt is ingested, separately from how fast tokens come out. If pasting a large document feels slow but replies are fine, this is the setting.
• Quantisation level. If it does not fit, drop from 5-bit to 4-bit before you drop to a smaller model. Below 4-bit, prefer a smaller model instead.

Speed expectations

If generation is far slower than the table suggests, the model is almost certainly spilling out of VRAM into system memory. Check the layer offload before blaming the hardware — this one misconfiguration accounts for most disappointing first attempts.

06 · THE VERIFICATION

Prove the model runs without a network.

This is the step that separates a genuinely local setup from a local interface in front of a remote API. Do it once, deliberately.

Keeping it offline afterwards

• Block outbound network access for the runner in Windows Firewall once your models are downloaded. Update deliberately rather than automatically.
• Turn off telemetry and update checks in whichever tool you chose. Most have a switch; a few need a configuration file.
• Keep the model files backed up. They are large but not infinite, and a model that gets withdrawn or relicensed is unrecoverable if you did not keep a copy. The backup guide covers where they should live.
• Be careful with chat interfaces. Many desktop front-ends support both local and hosted providers, and switching is one dropdown away. Check which provider is selected before pasting anything sensitive.

07 · QUICK ANSWERS

Local models, briefly.