The phrase "MLOps" used to evoke training pipelines, feature stores, and model registries. For most teams shipping AI in 2026, that's the wrong picture. The model isn't yours; it's a hosted API or a fine-tuned variant of one. The pipeline that matters runs at inference time, on every user request, and it has a fundamentally different shape from the classical ML stack.
This is what the LLM operations stack actually looks like at companies running real production traffic in 2026 — what to build, in what order, and how to know when each piece is paying off.
The three things that have to work
Strip an LLM operations stack down to its irreducible core and there are three concerns. Get them right and almost everything else falls into place. Skip any one and the system rots.
Observability. What did the model see? What did it return? How long did it take? What did it cost? Per request, per user, per feature, per prompt version. Without this, every other piece of the stack is guessing.
Evaluation. Is the system getting better or worse? Per change, per release, per upstream model update. Without this, you can't iterate; you can only pray.
Cost control. What is this thing costing right now, and what will it cost when traffic 3x's? Without this, the CFO ends a quarter unhappy and the program ends with it.
Everything else — prompt management, model routing, caching, fine-tuning pipelines, agent orchestration — is plumbing built on top of these three foundations.
Observability: instrument every request, not every endpoint
Classical APM tools (Datadog, New Relic, Honeycomb) give you the request and response, latency, and error rate. Those tools are necessary but radically insufficient for LLM products. The signals that matter aren't on the HTTP envelope; they're inside the model call.
A complete LLM observability record for a single request includes: the user-visible input, the resolved prompt (after templating, retrieval, tool injection), the model identifier and version, every parameter (temperature, max_tokens, response format, tool definitions), the full output, the input and output token counts, the latency broken down by retrieval / model / post-processing phases, the cost in dollars, the prompt version hash, the eval-set membership flag, and the user / session / tenant identifiers. Tool calls if the request was an agent. Retry attempts if there were any.
Capture all of this, indexed, queryable, with a short retention for the verbose payloads (7–30 days) and a long retention for the structured metadata (years). Now you can answer the questions that actually matter: which prompt versions are regressing, which tenants are running away with cost, which inputs are causing the most retries, which model is faster for which input length.
The market in 2026 has decent off-the-shelf options for this — LangSmith, Helicone, Arize, Phoenix, Langfuse, Honeycomb's LLM features. Pick one and integrate from day one. Building this in retrospect after the first cost incident is twice the work and ten times the stress.
Evaluation: the harness that decides what ships
The most underbuilt part of every LLM stack we audit. The default is "we look at outputs sometimes." This is not evaluation; it's wishful thinking with a feedback delay.
A real eval harness has three layers.
Offline evals on a curated set. A versioned dataset of inputs paired with reference outputs or assertions. Runs on every prompt change, every model swap, every architecture revision. Reports per-case pass/fail, aggregate metrics (exact match, semantic similarity, faithfulness, latency, cost), and a diff against the previous run. CI-gated: a regression past a threshold blocks the merge.
Online evals on production samples. A continuous stream of production requests, scored automatically (LLM-as-judge for subjective dimensions, deterministic checks for objective ones), with the scores logged as time series. Now you can detect drift — when an upstream model update silently breaks 6% of your traffic, the dashboard shows it within hours.
Human spot checks on flagged outputs. A queue populated by automated triggers (low judge score, high cost, long latency, user thumbs-down, schema validation failure). A human reviews the queue daily, labels what they see, and the labels feed back into the offline eval set. This is how the eval set stays alive over time instead of slowly going stale.
The single most important practice is gate every change on an eval. No prompt change ships without a re-run. No model swap ships without a side-by-side. No architecture revision ships without a regression check. This is not bureaucracy. It's the only way to compound progress instead of accidentally trading wins for losses.
Cost control: budget at the request, not the bill
The classical mistake is to treat the AI bill as a monthly fact you absorb. That works at small volume and explodes at scale.
The minimum cost-control discipline:
Per-request cost logging. Every model call logs its dollar cost, computed from prompt and completion tokens. Tag the cost with feature, tenant, prompt version, and user.
Per-tenant and per-feature budgets. Soft thresholds that trigger alerts. Hard thresholds that throttle or fail-open to a cheaper model.
Prompt-token budget per request. Count tokens before the model call. If the prompt exceeds budget — usually because retrieval pulled too much context — prune, don't pay.
Caching tiers. Identical request → cached response. Semantically similar request → cached partial. Repeated retrieval → cached embedding. Caching is the single highest-leverage cost lever; teams that skip it routinely pay 3–10x what they need to.
Model routing. Cheap model first; escalate to expensive only when the cheap model fails its eval or returns a low-confidence output. Done well, this cuts cost by 40–70% with negligible quality loss.
The instrumentation pays for itself the first time a runaway prompt change pushes spend up 5x in a day and the dashboard catches it within an hour.
The 2026 stack we'd build today
If we were starting an LLM product from zero this quarter, the operations stack would look approximately like this:
Tracing and observability: Langfuse or LangSmith for LLM-specific traces, alongside whatever APM the team already runs.
Eval harness: a homegrown harness on top of the chosen tracing tool's eval API, with the eval set checked into the application repo so it versions with the code.
Prompt management: prompts stored in versioned files in the application repo (not a SaaS prompt registry — the reproducibility loss isn't worth the convenience). Hot-reload supported via a config service for emergencies.
Cost dashboards: a Looker or Metabase view on top of the trace store, with breakdowns by feature, tenant, and prompt version. Slack alerts for budget thresholds.
Model routing layer: a thin router (LiteLLM-style) that abstracts providers, with policy hooks for fallback, retry, and tiered routing.
CI gates: prompt diffs trigger eval re-runs. Eval regressions block merge. Cost-per-request regressions trigger a review.
Human-in-the-loop labeling: a lightweight internal tool for the daily queue of flagged outputs, with labels flowing back to the eval set.
This is more infrastructure than most teams expect to build for "an AI feature." It is also the difference between a feature that grows up into a product and a feature that quietly costs more than it earns.
When to start each piece
Order matters. We sequence it like this:
Week 1. Tracing and per-request cost logging. Cheapest piece, highest leverage. You can't manage what you can't see.
Week 2–3. First offline eval set, CI gating. Even fifty hand-labeled examples is enough to start. The set will grow.
Week 4–6. Online evals and the flagged-output queue. Now drift becomes visible.
Month 2–3. Caching, model routing, per-tenant budgets. The cost levers, in the order they pay off.
Month 3–6. Fine-tuning pipelines (only if needed), prompt-management upgrades, multi-region routing.
Most teams get this sequence wrong by starting at Month 3 (multi-region routing, fine-tuning) and never doing Week 1 (tracing). They spend a year operating in the dark and then a quarter trying to catch up. The order above is the cheapest path to a stable, observable, controllable LLM product.
The cultural shift
The hardest part of LLM ops in 2026 isn't technical — it's cultural. ML and software engineering have different reflexes. Software engineers want determinism and tests; ML practitioners want experiments and metrics. LLM products need both: the deterministic discipline of software (versions, gates, rollbacks, observability) applied to the probabilistic substrate of models (evals, drift detection, judge calibration).
Teams that internalize this hybrid mindset ship AI products that get better over time. Teams that pick one or the other ship products that either move slowly or break unpredictably. The operations stack is how you make the hybrid mindset operational. It's where the discipline lives.
Need a sober look at your LLM operations stack? Diffco audits, designs, and builds production LLM ops for teams running real traffic. Book a 30-minute call and we'll tell you what you're missing.
