The question lands in our inbox several times a quarter: should we self-host an open-weight model instead of paying the API rates? Sometimes it comes from a CFO looking at a runaway bill. Sometimes from a CTO worried about data residency. Sometimes from an engineering lead who's read that open models have closed the gap and wants to ship with one.
The answer is rarely a yes-or-no. It's a math problem and a control problem. By the end of this post you should be able to do the calculation honestly for your own situation — and avoid the two failure modes that bracket the conversation: the team that overpays for an API forever because they never modeled the alternative, and the team that builds a self-hosted GPU fleet to save money and ends up spending more for worse outputs.
The two questions, separated
It helps to split the question into two cleanly separable concerns.
Cost. What does inference actually cost on each side, including all the parts people forget?
Control. What do you gain or lose in latency, privacy, customization, regulatory posture, and lock-in?
Most teams collapse these into "should we self-host?" and end up arguing across two different value systems. Pull them apart and the decision becomes tractable.
The honest cost model
Hosted-API cost is straightforward: a published per-token rate, multiplied by your traffic. The number is visible on every bill. Self-hosted cost is not.
A complete self-hosted cost model includes:
GPU compute. Cloud GPU hours or amortized capex on owned hardware. This is the big number, and it's almost always larger than people first estimate. A serving fleet sized for peak traffic is idle most of the time. A fleet sized for average traffic queues during peak. Real utilization tends to land at 30–60% under reasonable engineering, which means the effective per-hour cost is 1.5–3x the headline rate.
Engineering time. Whoever sets up vLLM, TGI, or Triton, configures quantization, tunes batching, monitors GPU utilization, and is paged when a node fails. This is a continuing cost, not a one-time setup.
Storage and bandwidth. Model weights are tens to hundreds of gigabytes. Multi-region serving multiplies this. Traffic egress is sometimes free in the same cloud, expensive across clouds.
Inference engineering. Speculative decoding, KV-cache optimization, batching strategy, dynamic context truncation. All of this is included for free in API pricing. On self-hosted, you either build it or accept worse throughput — sometimes 3–5x worse.
Eval and quality drift. When you self-host, you also own the responsibility for catching regressions. The open model that scored 87 on your eval set in March may score 84 on new traffic in November because your data distribution shifted. With an API, the provider absorbs some of this; with self-hosted, you don't.
Reliability. The hosted API has been through five-9s of operational scar tissue. Your self-hosted setup has not. Outage budgets are real costs.
The frontier-quality premium. If your application benefits from a frontier-tier model and the open ecosystem is one tier behind, the right comparison isn't "API of model X vs self-hosted of model X." It's "API of frontier model vs self-hosted of next-tier model." Quality differences are an economic input.
A reasonable break-even calculation for a typical 2026 production workload — high but not enormous traffic, mid-tier model class, moderate engineering team — usually puts self-hosting ahead at very high steady-state token volume (think hundreds of millions of tokens a day, sustained) and behind everywhere else.
The math changes meaningfully if you have spiky traffic (APIs win — you don't pay for idle), if you have very steady traffic (self-host can win — utilization stays high), if you can use a smaller open model where the frontier-class API would be wasteful (self-host wins more), or if you have specialized hardware access (research labs, on-prem GPU fleets) that's already paid for.
When control beats cost
Cost is sometimes the wrong frame entirely. There are real situations where you self-host even if it's more expensive.
Data residency and privacy. Some workloads cannot leave a tenant's infrastructure. Customer-managed keys help, but for the strictest cases (sovereign cloud, healthcare on-prem, government), self-hosting inside the customer's environment is the only option. The relevant cost comparison stops being self-hosted-vs-API and becomes self-hosted-vs-no-product.
Latency floor. Hosted APIs have an irreducible network hop. For sub-100ms-token-latency requirements (very rare in 2026 but real for some real-time experiences), local inference can be the only path.
Heavy fine-tuning or LoRA portfolios. If you're running dozens of fine-tuned variants for different customers or workflows, self-hosting with multi-LoRA serving can amortize cost across them in a way that hosted fine-tuning APIs can't match.
Lock-in management. Some companies make a deliberate strategic choice to avoid hosted-API dependency, treating model serving as a core capability. This is a legitimate posture; just be honest about the cost of that strategy.
Regulatory pre-approval requirements. In some sectors, getting a model approved for a use case requires the operator to control the entire stack including the weights. The API path is closed.
When the API is obviously right
For most teams, most of the time, the hosted API is the correct default. Specifically:
You're under steady-state hundreds-of-millions-of-tokens-a-day traffic. Below that volume, self-hosting almost never beats the API on real cost. The break-even is higher than people guess.
Your workload is spiky. APIs let you pay for what you use. Self-hosted means provisioning for peak and watching the meter run during troughs.
You don't have ML platform engineers in-house. Self-hosting LLMs at production reliability is a real engineering practice, not a weekend project. Without dedicated platform talent, the operational cost compounds quickly.
You need frontier model quality. The gap between frontier API models and the best open weights closes and re-opens with each generation. As of writing, for the most demanding tasks, the gap is real and worth paying for.
Your privacy requirements are satisfiable by tenant isolation, BAAs, or zero-retention modes. Most workloads are. Don't pay the self-hosting tax for control you don't actually need.
The middle ground people forget
The conversation usually frames itself as a binary, but in practice the strongest 2026 architectures are hybrid.
Frontier model via API for the hardest 20% of requests — the ones where quality matters most.
Smaller open-weight model self-hosted for the easy 80% — bulk classification, simple extraction, retrieval-grounded summarization.
Caching layer in front — repeated requests never hit either.
A router that decides which path each request takes, based on input characteristics or a cheap classifier.
This setup gets you most of the cost savings of self-hosting on the easy traffic, the quality of frontier models where it matters, and the operational simplicity of having only one cluster to maintain (the easy-tier one). The hard part is good routing — but routing is also the thing that pays for itself most directly.
How to actually decide
Here's the calculation we do with clients.
Measure your actual current usage. Tokens in and out per day, broken down by feature and complexity tier. If you don't have this data, you have a higher-leverage problem to solve before this one.
Estimate what fraction of traffic could be served by a smaller open model with comparable quality. Run a side-by-side eval. Most products find that 50–80% of traffic doesn't need frontier quality. If yours does, the conversation usually ends there in favor of the API.
Project forward 12 months. What's the expected growth? Self-hosting tends to look better at scale and worse at the start; API tends to look better at the start and worse at scale. Pick the regime you'll actually be in.
Quantify the operational tax. Add a candid number for engineering time on self-hosting. If your team is two ML engineers and a backend, this number is large. If your team is a fifty-person platform org with mature GPU infra, it's smaller.
Identify the control requirements. Are there workloads that cannot go to a hosted API? Carve those out and address them separately — possibly with a small dedicated self-host while the rest stays on the API.
Pick a posture, not a religion. "We'll start on the API and re-evaluate at 5x current volume" is a perfectly good answer. "We'll self-host the easy tier within six months" is also a good answer. "We'll self-host everything because the API feels expensive" is, almost always, the wrong answer.
The bottom line
The 2026 reality is that hosted APIs are exceptional value for most teams, that open-weight self-hosting has narrowed the gap to the point of being viable for a specific traffic profile, and that the most cost-effective architectures are usually hybrid rather than purely one or the other. Don't let the cost-vs-control debate be a prestige conversation. Measure, model, and pick the cheapest stack that meets your control requirements. Revisit yearly.
Trying to decide between hosted APIs and self-hosting for your AI workload? We'll run the math with you, including the parts your AWS bill won't show. Book a 30-minute call and bring your traffic numbers.
