How to Use Local LLMs for Coding: Setup, Workflows, and Honest Limits

Running a language model on your own computer can make coding assistance available without sending each prompt to a hosted inference API. But “local AI coding” covers three very different jobs: suggesting the next line, answering a question about code you provide, and editing a repository through an agent. A setup that feels useful for the first may fail badly at the third.

This guide starts with a laptop or desktop, not a rack of GPUs. It gives you one concrete path to try, explains what to measure, and marks the point where a larger machine or a hosted model may be the more practical choice. The commands are documented integrations, not a benchmark we ran. Model behavior, speed, and memory use still need checking on your hardware and codebase.

First decide what “local” means

For this article, local inference means the model weights are loaded and the answer is generated on your laptop or PC. A model served from a GPU elsewhere in your office is self-hosted, but not running on your device. A desktop app that sends requests to a vendor's cloud is neither, even if its interface is installed locally.

That distinction matters for privacy and cost. A local model can remove the per-token inference bill, but not the cost of hardware, electricity, downloads, maintenance, or developer time. Nor does it prove that nothing leaves the machine: the editor, plugins, update checks, model provider, and any tools the assistant calls have their own network behavior. If confidentiality is the reason for going local, inspect the complete workflow and network policy—not just the model's name.

Match the workflow to the machine

As a concrete starting point, Ollama lists Qwen2.5-Coder 7B Instruct at a 4.7 GB model download with a 32K advertised context window . That figure is not a promise that a 5 GB GPU, or even a particular 8 GB laptop, will run it comfortably: runtime memory also includes the context cache and other overhead, and the operating system and editor need room. A smaller 3B variant may fit a constrained machine more easily, but you should expect to use it for narrower tasks.

For a machine with more memory, Ollama lists Qwen3-Coder 30B at a 19 GB download and 256K advertised context . It has about 3.3B active parameters per token, but 30B total parameters still need to be stored. Treat 32 GB of usable unified/system memory or VRAM as a *starting point to investigate*, not a guaranteed minimum: context length, other apps, offloading, and runtime overhead can change whether it fits and how fast it feels. Do not assume the maximum context is free to use. Ollama's context guide says larger allocations consume more memory and shows how to inspect whether a model is running on GPU or partly on CPU.

The rule is simple: choose the smallest model that completes your actual task acceptably. Published parameter counts and context limits are inputs to a trial, not a ranking of coding usefulness.

A reproducible local starter setup

One transparent route is Ollama for inference plus Aider for repository edits. Aider documents this integration and recommends its Ollama chat adapter. Install both tools using their official instructions, then start with a small, disposable Git repository or a clean branch. Aider normally makes Git commits when it edits files and can commit pre-existing dirty files, so read its Git behavior before pointing it at valuable uncommitted work.

For macOS or Linux, after installing Ollama and Aider, a first run can look like this:

ollama pull qwen2.5-coder:7b-instruct
ollama run qwen2.5-coder:7b-instruct

The second command is only a quick check that the model responds. Exit that chat, ensure Ollama is running, then from your test repository use:

export OLLAMA_API_BASE=http://127.0.0.1:11434
aider --model ollama_chat/qwen2.5-coder:7b-instruct --no-auto-commits --no-dirty-commits

The adapter and endpoint follow Aider's Ollama guide . The two --no-…-commits flags are our conservative choice for a first trial; inspect each diff and make your own commits. Windows users should follow the same guide's environment-variable syntax for their shell. If Ollama's app is already serving on the default port, do not start a second server. If you deliberately start it from a terminal, set a context length that fits your machine and task; the current Ollama context documentation shows the setting and how to check the allocated context with ollama ps.

Start with one file and one bounded request. For example:

Read the parser and its existing tests. Add one failing test for an empty
input, then make the smallest change that passes it. Do not modify other
files. Show the diff and the test command. Stop if the expected behavior is
unclear.

That prompt is an illustrative task, not a proven template. Add the relevant files to Aider's chat and run the project test suite yourself. Aider may return a plausible explanation yet fail to apply the change: its own documentation warns that some local models cannot reliably produce the edit format it needs. A failed edit is a real product limitation, not a prompt you should endlessly repair.

Why context and tools matter more than a giant number

The advertised context window is the maximum a model *might* accept under a supported configuration. It is not the amount your local runtime necessarily allocates, and it is not an instruction to paste an entire repository into every prompt. More context uses more memory; irrelevant files also distract from the change. Provide the smallest set of source files, tests, error output, and repository conventions needed for the task. Expand only when the model identifies a dependency it cannot inspect.

Coding agents have another constraint: they must repeatedly decide which file to read, how to express an edit, whether a tool result matters, and when to stop. A model that can explain a function may not reliably call tools or maintain a valid diff across many steps. Test that behavior separately from chat quality. Aider's edit-format documentation explains why writing good-looking code in a response is not the same as changing files correctly.

If the machine is short on memory, a smaller context and tighter file selection are usually better than forcing a large model to spill onto CPU. Use ollama ps to see where Ollama has loaded the model. If your regular tasks truly need long context and sustained tool use, trial a larger local machine or an on-prem GPU server rather than pretending a 7B laptop setup is a drop-in replacement for every hosted agent.

Evaluate it on your own repository

Do not choose a coding model from a single demo or a vendor's benchmark table. Prepare five to ten representative tasks with a known baseline: a small bug with a test, a narrow refactor, a documentation change, a multi-file change, and a task that should cause the assistant to ask for clarification. Run each from the same Git commit and record:

1. Outcome: did the expected test pass, and did the change actually meet the request?
2. Edit reliability: did the tool apply a correct diff without manual repair or unrelated changes?
3. Time: how long until a reviewable change, including retries and human fixes?
4. Resources: peak memory, whether inference stayed on GPU, and whether context was truncated or reduced.
5. Review burden: how many wrong assumptions, security issues, and unexplained edits did a human catch?

Compare the total cost of a usable change, not merely tokens per second. A fast answer that cannot be applied, tested, or trusted is not a fast coding workflow. Keep the test tasks and acceptance criteria stable when you compare a smaller local model, a larger local model, and any hosted alternative.

Privacy and safety are workflow properties

Keep the model endpoint bound to the local machine unless you intentionally need remote access. Audit editor extensions and agent tools for network calls; do not assume that selecting a local model disables every online feature. Aider says its optional analytics are opt-in and can be disabled , but that is only one component of a wider setup. Check the exact model license before using weights in a commercial product or redistributing a bundled model.

Run an editing agent on a clean branch with least-privilege access. Keep secrets out of prompts and test fixtures; review shell commands before execution; inspect the diff; run tests; and keep a reliable revert path. Local inference does not make a model's edits correct or its tool calls harmless. If you need strict isolation, use a sandbox or a separate development environment in addition to the local model.

When a GPU server is the right next step

A self-hosted server can centralize a larger model for a team and keep inference under your organization's control, but it adds authentication, network security, capacity planning, logging, and operational cost. It is not the same privacy boundary as an offline laptop. As one striking example, Aleph Alpha's new Kolibri-1 model card lists 78.1B total parameters, roughly 78 GB of FP8 weights, and minimum hardware including two A100 80 GB GPUs or one H200. Its 3.46B *active* parameters describe compute per token, not laptop memory demand. We cover that release separately in the news.

Bottom line: start with a narrow coding task and a model that fits comfortably, then verify the diff and tests. Move to a larger model or server only when your own task set shows that the extra memory and complexity buy you a reliably better result. Local coding works best when “runs here” is the beginning of the evaluation—not the conclusion.

Sources

• Ollama: Qwen2.5-Coder model tags
• Ollama: Qwen3-Coder model listing
• Ollama: context length and model placement
• Aider: Ollama integration , Git integration , and edit formats
• Aleph Alpha: Kolibri-1 model card