A Question That Looked Settled
For two years, the answer to "which model do we use?" was almost a reflex: the best one you can rent. OpenAI, Anthropic, Google – grab an API key and go. That wasn't laziness, it was good sense. The gap between the frontier models and everything below was so wide that the alternative – running a model yourself – smelled of ideology. Of "we do it ourselves on principle," rather than an economic decision.
That gap has shrunk. Llama, Mistral, Qwen, DeepSeek, Gemma and their relatives now reach, on a growing list of tasks, a quality that two years ago belonged only to the closed frontier models. And that reopens a question many had long since filed away: when is it worth running a model yourself instead of calling someone else's API?
The honest answer is neither "always" nor "never." It's a decision framework – and it pays to walk through it soberly rather than fall onto one side out of principle.
Rent or Own – Applied to Intelligence
At heart this isn't a new question, just a new object. Rent or own – every company makes that call constantly: office, servers, fleet. What's new is that it now applies to intelligence itself.
Using a frontier API means renting the best brain in the world on demand. You pay per use, you get the latest thing immediately, you worry about nothing behind it. Running an open-weight model yourself means owning a good-enough brain you fully control. It isn't the sharpest on the market, but it's yours, it runs where you want, it changes only when you want it to, and it costs essentially nothing extra per request once the hardware is standing.
As with any rent-or-buy question, there's no answer that's right in principle. There's only the question too few people ask: what exactly is the workload – and does it suit renting or owning?
Where Self-Hosting Wins
Several conditions tip the maths in favour of your own model – and they often show up together.
High, steady volume. The API's per-token economics are unbeatable at low volume: you only pay for what you use. They flip the moment you saturate a GPU. A request that costs a few pence over the API costs a fraction of that on a well-utilised card of your own – because the hardware's fixed cost spreads across millions of requests. Anyone classifying, extracting, summarising, or translating at constant high volume hits a break-even point beyond which every further request on your own hardware is all but free.
Data that can't leave the building. Some data you simply cannot send to someone else's API – not on principle, but on regulation: patient records, contracts, personal data under strict residency rules. When the data isn't allowed to leave your perimeter, a model running in your own data centre isn't the more expensive option – it's the only one.
Latency, offline, edge. Every API call is a network round trip to someone else's data centre. For an application that has to respond in milliseconds, that runs in a place with no reliable connection, or that lives directly on a device – on the factory floor, in a vehicle, at the point of sale – that's a problem no API contract solves. A local model answers even when the line is down.
Predictable cost. An API bill swings with usage, and the provider changes its prices whenever it likes. Your own GPU costs the same every month, no matter how often it computes. For planning, that's the difference that matters: a fixed cost instead of a variable someone else controls. No surprise invoice, no pricing email that puts a business model in question overnight.
Specialisation on a narrow task. This is the most underrated advantage. A generic frontier model is decent at everything. A small open model fine-tuned for exactly one task – your product classification, your tone of voice, your domain vocabulary – regularly beats the generic giant on that one task: faster, cheaper, often more accurate too. You trade breadth for depth in the thing that counts.
Independence from the vendor's schedule. Build on someone else's API and you build on their roadmap: models get deprecated, endpoints disappear, rate limits change, behaviour shifts with the next silent update. A model whose weights you own runs the same today as in three years – byte-for-byte identical, for as long as you like. For anything that has to be reproducible and auditable, that isn't a luxury, it's a prerequisite.
Where the API Still Wins
The other direction matters just as much – otherwise you build, on principle, an infrastructure you don't need.
When you need the absolute frontier. On the hardest tasks – multi-step reasoning, complex agents, the newest capabilities days after they ship – the closed frontier models still lead. The gap has shrunk, not vanished. When the task demands precisely that peak, renting isn't the easy choice, it's the right one.
Spiky or low volume. Self-hosting pays off through utilisation. A GPU sitting idle most of the time is pure cost – you pay for it in the twenty hours a day nobody needs it, too. With fluctuating, low, or still-uncertain load, the API wins: you pay only for the peaks, not the waiting in between.
When you don't want to run inference infra. Hosting a model means provisioning GPUs, planning capacity, patching, monitoring, owning uptime. That's an ongoing operation with on-call and accountability, not a one-off setup. If you don't want to run that business – for good reasons – you're better off renting.
Speed to first prototype. Nothing is faster than an API key and three lines of code. For the first weeks of a product, for validating an idea, for anything that might be scrapped again tomorrow, self-hosting is the wrong effort at the wrong time. Rent first, learn – and only own once the volume and the requirements have hardened.
A small team without MLOps muscle. Running your own model sensibly demands skill: inference stacks, evaluation, monitoring, knowing when a model is quietly getting worse. If that skill isn't in the team and can't be built or bought, the self-hosted solution isn't an advantage, it's a source of silent risk.
The Honest Cost Calculation
The most common miscalculation is to set GPU price against token price and stop at the card's hourly rate. It isn't that simple.
On the self-hosting side of the ledger belong not just the hardware or the GPU-hours in the cloud, but:
- the engineers who stand up the inference stack and keep it running
- the ongoing operations – monitoring, patching, on-call, uptime
- evaluation and quality assurance, so nobody finds out too late that the model has drifted or an update shifted the results
- the idle capacity – every hour the expensive card does nothing
- the opportunity cost – the same people could be working on the product instead of the infrastructure
Only with these line items is the comparison fair. And only then does the real point emerge: the break-even isn't a matter of gut feel, it's a calculation of volume and control. At high, steady volume, self-hosting amortises quickly, and every further million requests widens the gap. At low volume, you're paying for expensive hardware to wait. And the need for control – compliance, residency, reproducibility – can tip the calculation on its own, even when the volume wouldn't carry it: sometimes self-hosting is more expensive and still the only permissible option.
The Pattern Most Teams Should Actually Land On
In practice the answer is rarely "everything over the API" or "everything in-house." It's hybrid – and that's not a compromise born of indecision, it's usually the best architecture.
The principle: route by task. The large, uniform, sensitive bulk – classification, extraction, first drafts, standard translations, the cheap pre-sorting – runs on your own open-weight model. The rare, heavy, genuinely demanding cases – the complex reasoning, the edge case, the task at the frontier – go to the API. A classifier or a simple heuristic decides what goes where.
The rule of thumb behind it: own the boring 80%, rent the critical 20%. The volume that eats your costs and touches your data, you bring in-house. The peak capability you need rarely but then absolutely, you buy per request. That way you pay the API only for what it's irreplaceable at – and not as a tollbooth on every trivial request.
The Checklist
Before you build, it's worth a sober pass through five questions:
- Volume? High and steady enough to saturate a GPU? Then the economics favour self-hosting. Spiky or low? Then they favour the API.
- Data constraints? Do the data have to stay inside your perimeter for compliance, residency, or confidentiality? Then self-hosting is often not an option but an obligation.
- Capability ceiling? Does a good-enough model cover the task, or do you need the absolute frontier? The closer to the edge of what's possible, the more it points to the API.
- Your team's operational maturity? Is the skill there to run inference, to evaluate, to monitor – or can it be built? Without that maturity, your own model becomes a risk.
- How much cost predictability does the business need? Are fixed, plannable costs more important than minimal effort to get started? Then self-hosting counts for more than the raw token bill shows.
Mostly "self-host"? Then it's worth building – probably for a slice of the load. Mostly "API"? Then renting is the right call, not the lazy one. And in most cases the answer sits in between: some of it here, some of it there.
Conclusion
The fact that open-weight models have narrowed the gap to the frontier doesn't make self-hosting the new obligation – it makes it a serious option where it used to look like ideology. Running your own model is neither a creed nor a free lunch. It's an engineering commitment with an ongoing operation, one that pays off for specific, well-understood workloads – and, for others, simply doesn't.
The right question isn't "API or your own model" as an article of faith, but "which workload, and what suits it." High volume, hard data constraints, latency, predictable cost, a narrowly specialised task – that points to owning. The absolute frontier, spiky load, no MLOps muscle, speed to prototype – that points to renting. And most of the time both point at once, for different parts of the same system.
This is exactly the build-or-rent call we make with our clients at nh labs – without ideology, along the actual workload. The goal is never "self-host" or "always the API," but the right tool per workload. Often that's a hybrid: the boring 80% in-house, the critical 20% rented.