Why local inference changes the cost equation
A developer asks an AI assistant to investigate a bug, change the code and run the tests. The first attempt fails, so the assistant tries again. What began as a simple request becomes a long exchange of code, logs and instructions. Across an engineering team, that repeated work can make AI usage expensive enough that people start rationing it.
Running an open model on infrastructure you control offers a different way to manage that workload. You take responsibility for the service and its capacity, while gaining more control over where inference happens and how it is paid for. Revolut provides a useful example of what this can look like at scale.
At AI Engineer NYC in October 2026, Revolut’s David Abitbol described serving roughly 2,000 engineers with self-hosted GLM models. He reported inference spending around one tenth of the previous Anthropic and OpenAI invoice, alongside improvements of about 60% in coding time and 70% in lead time. These were internal observations, not a controlled study or a promise of equivalent savings elsewhere. His deployment used 56 NVIDIA B200 GPUs for GLM 5.3. Frontier services remained available for exceptions, with image understanding routed to models suited to that work. [1]
That is shared infrastructure at substantial scale. A local model can also run on a single workstation, while a company deployment might use an internal server or dedicated rented GPUs. Self-hosted does not necessarily mean on premises or offline. The useful question is where the model runs, which systems receive your data and who controls them.
Start with a measurable pilot
For a smaller team, we would begin with a few recurring tasks: understanding an unfamiliar service, fixing a bounded defect, extending an existing screen and writing tests for the change. Use real examples from your own repositories. Judge the result by correctness, review effort and elapsed time. A model that produces an impressive first draft can still be expensive if an engineer spends the afternoon repairing it.
A practical pilot could pair Ollama with an open coding model such as Qwen3-Coder 30B. The current Ollama Q4_K_M package is approximately 19 GB. That is the model download, not the complete memory requirement: the runtime and working context need additional space. Treat this as an accessible example to evaluate, rather than a claim that it is the strongest model available. [2]
Size the machine around the intended context and concurrent users. The context is the material the model can work with during a request, including instructions, code, conversation history and tool results. Longer contexts consume memory, and a model that fits at short context may become slow or fail at the setting your coding agent needs. Test on existing equipment or a short GPU rental before committing to hardware.
Configure Ollama and OpenCode correctly
Once Ollama and OpenCode are installed, configure the Ollama server for local use. Set OLLAMA_NO_CLOUD=1 and restart the service. Ollama’s current OpenCode integration guide calls for at least 64K context, so set OLLAMA_CONTEXT_LENGTH=65536 in that same service environment and check that the machine can sustain it. These settings belong to the running server; setting them only in an unrelated terminal will not reconfigure an existing service. [3, 4]
From a separate working branch in the project directory, download the model and launch the coding agent:
ollama pull qwen3-coder:30b
ollama launch opencodeTest the agent inside a real repository
Select the downloaded local model in the launcher. OpenCode also supports manual configuration through Ollama’s local API at http://localhost:11434/v1. Keep the pilot endpoint bound to the local machine. Before using confidential code, check that optional external providers, web tools and telemetry follow your intended data policy; disabling Ollama’s cloud features does not govern every component around it. [3, 4, 5]
Start with a task you can verify quickly. Ask the agent to explain a failing test, propose a small change and run the relevant checks. Confirm that it can call tools reliably, edit the correct files and stop when a command fails. Being able to answer a coding question is only one part of working effectively inside a repository.
Give the agent a short project guide covering the build commands, architectural conventions, important directories and tests. Let it retrieve relevant code and documentation as needed. For the first pilot, this is a sensible place to invest effort before considering fine-tuning. Keep document access tied to the user’s permissions, and treat instructions found inside tickets, source comments or retrieved pages as untrusted content.
Add a visual feedback loop for UI engineering
UI engineering needs an additional feedback loop. A text model can generate components and CSS, but assessing a rendered interface requires image understanding or other inspection tools. A local vision-language model such as Qwen3-VL is one candidate to evaluate for screenshot interpretation; its capabilities need testing against your own interfaces. [6]
Consider a customer portal with a new account settings screen. Give the agent the existing components, design tokens, validation rules and examples of nearby screens. Then run the application and inspect the result at desktop and mobile sizes. A workable sequence is:
- Implement the change using the project’s existing components and conventions.
- Run the application with synthetic data and exercise the relevant user journeys.
- Capture screenshots and inspect the DOM, console errors and network failures.
- Use a vision model to identify visual problems, then let the coding agent propose corrections.
- Run the checks again and have an engineer review the code and the interface.
Validate more than the screenshot
Playwright can capture screenshots and compare them with approved baselines. Keep the test environment consistent because rendering varies across operating systems and browsers. [7] Add checks for keyboard navigation, error states, long labels and narrow screens. A visual comparison will not establish that a form submits the correct data, and a model’s approval of its own output should never be the acceptance test.
If the team also uses a hosted vision model, make that route explicit. Screenshots can contain customer information or internal product details. A local coding model paired with an external screenshot service is a hybrid workflow, and its data handling should be assessed as such.
Turn the pilot into a reliable service
Once several developers depend on the pilot, assign an owner for the service. A shared deployment needs authenticated access, capacity management, monitoring and a recovery plan. vLLM is one serving option to evaluate; it exposes compatible APIs and supports deployment patterns that can grow beyond a single GPU. Verify the exact model and agent combination, including tool calls, streaming and image input where needed. API compatibility alone does not establish that the whole workflow works. [8]
Measure how long users wait before a response begins, how often requests queue and how long completed tasks take. Include the slowest typical requests, not only averages. Keep the context window appropriate to the workload and assess prefix caching, which can reuse computation when requests begin with the same material. [9] These settings deserve load tests with several active agents; a fast demonstration with one user tells you little about a busy team.
Protect the service through the surrounding infrastructure. Use an authenticated gateway and private network rules, and expose only the endpoints users need. vLLM’s documentation explicitly warns that its API key option alone does not secure every endpoint. Where clients or teams require strong isolation, use separate serving instances and separate repository and retrieval permissions. [10]
Run agents in restricted development environments with the permissions needed for their task. Keep production credentials out of their reach and retain normal review and release controls. Record enough information to investigate failures, while avoiding unnecessary retention of source code, secrets and personal data in logs. Pin the model, quantization and serving versions, rerun the evaluation set before upgrades and retain a known working configuration for rollback.
Measure accepted work and total cost
The commercial decision should include hardware rental or depreciation, power, hosting, maintenance, engineering support and any external model usage. Divide those costs by work that passes review. Counting generated tokens or proposed changes rewards activity, even when the output needs replacing.
Lower token costs also do not translate directly into lower project budgets. Requirements, integration, testing and approval still take time. During the pilot, track review effort and escaped defects alongside delivery time. If code generation becomes faster while the review queue grows, the team has moved the bottleneck rather than removed it.
A four-week evaluation plan
For an initial evaluation, we would plan four weeks, assuming suitable hardware and repository access are already available. Use the first week to establish baseline tasks and compare models. Connect a small group of engineers in the second. Add the UI inspection loop and test concurrent use in the third. Use the fourth to compare accepted work, total costs and developer feedback. A broader production rollout may need longer, particularly where access controls and infrastructure procurement are involved.
At TechHub, our recommendation would be to expand only where that evidence supports it. Keep the tasks the local model handles reliably, improve the weak parts of the workflow and retain an approved alternative for difficult cases. The investment earns its place when engineers can complete useful work with less waiting and rework, at a cost the business can explain.
Sources
Primary sources and official documentation checked on 4 November 2026. Revolut figures are attributed to the speaker. The proposed setup, evaluation method and four week pilot are recommendations for readers, not a description of Revolut’s internal implementation.