The failure that never trips an alarm
Classic software monitoring rests on one reassuring assumption: when something breaks, it breaks loudly. An exception is thrown, a status code jumps to 500, the latency curve spikes, a dashboard turns red. The entire toolkit of the last twenty years — error rates, p99 latency, traces spanning microservices — is built on the premise that failures have a visible symptom. You just have to measure the right symptom and set the alarm at the right threshold.
AI systems break that assumption. Their signature failure isn't the crash, it's the wrong-but-plausible answer — the summary that drops a crucial point, the classification that's confidently off, the extracted figure that's out by an order of magnitude. And that answer throws no exception. It carries a 200 status code. It arrives at normal latency. In the logs, it is indistinguishable from a perfect response. The model never flagged that it was unsure. It delivered, as self-assured as ever, only wrong.
This is exactly where classic observability fails. It was built to watch systems that either work or crash. AI systems have a third state the whole instrumentation stack is blind to: they work, but worse. And because that state trips no alarm, the person who notices it first is often the customer.
Why AI systems are uniquely opaque
To see why AI needs an observability discipline of its own, you have to take seriously four properties that deterministic software simply doesn't have.
Non-determinism. Send the same input twice and you get two different answers. That isn't a bug, it's the built-in character of probabilistic models. And it costs you the single most reliable debugging technique in the history of software: deterministic reproduction. You can no longer catch a fault by triggering it again — on the second attempt the system may behave impeccably. If you didn't capture exactly what happened the first time, the fault is gone for good.
Silent quality regression. Deterministic code that worked yesterday works today, unless someone changed it. With AI systems that guarantee is gone. Answer quality can drift without a single line of code being touched: because the model behind it changed, because the distribution of user queries shifted, because an upstream retrieval step got worse. The regression is real, but there's no commit to point at.
Prompt and version drift. An AI feature is rarely just "the model". It's a prompt template, a body of context, a set of tool definitions, a model version and a dozen parameters. Every one of those parts can change independently — someone tweaks the system prompt, a retrieval index gets rebuilt, the provider rolls out an update. Each of those changes can shift behaviour, and none of them looks like a classic deployment.
No stack trace for a "bad answer". When deterministic code falls over, you get a line number and a path through the call stack. When a model gives a bad answer, there is no stack trace. There is no line where "the fault happened". The answer emerged from billions of weights and the entire context you fed in. Debugging here doesn't mean "trace the fault back to its source", it means "reconstruct the exact conditions under which this answer was produced" — and that's only possible if you captured those conditions beforehand.
What you actually need to capture
If the failure leaves no stack trace, observability has to supply the context the stack trace would otherwise have given you. That means capturing considerably more than with classic software — and doing it deliberately. What an instrumented AI system should record per request:
- The full trace: prompt, context, response. Not just the user's input, but the entire assembled prompt — system prompt, injected context, retrieval results, few-shot examples — plus the raw model response. This is the one thing you cannot reconstruct if you didn't write it down. Without the exact prompt, you have no hope of understanding a bad answer from a non-deterministic system.
- Token usage and cost per request. Input tokens, output tokens, the resulting price — at the request level, not as an aggregated monthly total. There's a section on why below.
- The tool-call chain. For agentic systems: which tools were called, in what order, with what arguments, and what they returned. An agent that wanders off does so almost always via a traceable chain of poor tool decisions — but only if you recorded the chain.
- The latency breakdown. Not "the request took 4 seconds", but where the 4 seconds went: retrieval, first model call, three tool calls, second model call. In multi-stage AI pipelines total latency is a sum, and you need to see the addends individually.
- User feedback signals. The thumbs up/down, the "regenerate", the silent abandonment, the copying of the result. These are the only correctness signals that arrive in real time, in production. They're crude, but they're gold once you attach them to the trace that produced them.
- Eval scores over time. The link between observability and a real evaluation suite. Observability tells you that behaviour has changed; evals tell you whether it got worse. If you don't yet have an eval culture, our piece "Evals over gut feeling" is where to start — observability is the wiring that feeds those evals with production data on a continuous basis.
The toolkit for this is growing fast. LLM tracing platforms in the LangSmith mould specialise in laying out whole prompt-context-response traces and agent runs; in parallel, the OpenTelemetry conventions for LLMs are forming an open standard that folds AI spans into the same tracing world the rest of your system already lives in. Which tool you pick is secondary. The decision that matters is capturing the full trace at all — before you need it.
The silent regression: three ways it quietly gets worse
The most dangerous case is the one where nothing about your system changes and it gets worse anyway. Three patterns come up again and again.
Same model name, different behaviour. A provider updates the model behind a stable name. You keep calling the same endpoint, the same identifier, and suddenly get subtly different answers — more cautious, more verbose, formatted differently, stronger in one place and weaker in another. No deployment on your side, no code change, no alarm. Just a feature that ticks a little differently after the weekend. Without a trace history to compare before and after, this is simply invisible — you find out from the complaints.
The prompt template changes. Someone on the team improves the system prompt to fix an edge case. The fix works locally but shifts behaviour in another, untested area. That's not bad luck, it's the nature of prompts: they're globally coupled, a change in one place lands everywhere. Without versioned prompts written into every trace, nobody can later tell which wording was running on the day the quality tipped over.
Retrieval quality degrades. In RAG systems the model often isn't the problem at all — the context is. The vector index gets rebuilt and an embedding version moves, a document set grows and dilutes the hits, a chunking parameter changes. The model gets worse context and gives correspondingly worse answers — but it looks as though the model has degraded. Only if you instrument the retrieval step separately do you see that the cause sits one layer upstream.
All three patterns share one thing: they leave no trace in classic metrics. Error rate steady, latency steady, availability green. The only way to catch them is to watch the substance of the quality over time — and that presupposes you still have yesterday's traces and eval scores to compare against today's.
Cost as a first-class metric
In deterministic software, cost per request is usually a decimal place nobody bothers with. In AI systems it's a first-class operational metric — sometimes the one that catches fire first.
The reason is that AI cost can be explosive and quiet at the same time. Two patterns stand out.
The runaway agent. An agentic system falls into a loop: it calls a tool, isn't happy with the result, calls it again slightly differently, over and over. Every round burns tokens. Functionally it often doesn't even look like a fault — the agent "is working", after all. But a single request stuck in a loop can cost a hundred times a normal one, and when that happens under load, the bill at month-end is a shock nobody saw coming. Only a recorded tool-call chain with per-step token counts makes such a loop visible — ideally in real time, with a hard limit that throttles it.
Context bloat. It's tempting to give the model more context — more retrieval results, more history, more examples. Each addition maybe improves quality a little and increases the cost of every single request for certain. Without request-level cost transparency this bloat creeps in, a field here, a document there, until cost per request has doubled and nobody can name the moment it happened.
Instrumenting cost per request makes this dynamic visible while it's still cheap to fix. It also lets you ask the question every serious AI product has to face eventually: what does this feature cost us per user, and does it pay for itself? That question can't be answered retrospectively from an aggregated cloud bill. It needs the cost at the individual request, captured from the start.
The other side: don't drown in telemetry
Up to here you might draw the lesson: log everything, every prompt, every response, every intermediate step, forever. That would be a mistake — an expensive and risky one.
Recording every full prompt and response has three uncomfortable consequences. Privacy and PII: user inputs to an AI system often contain exactly the sensitive data you least want mirrored into a log store — names, emails, health or financial details. A complete prompt log is a second, more poorly guarded data lake, a compliance problem and an attack target. Cost: trace storage isn't free; retaining whole context windows — tens of thousands of tokens per request on modern models — indefinitely can become a cost line in its own right. Noise: a log that contains everything is a log in which you find nothing. Completeness without structure isn't observability, it's a haystack.
The answer isn't "log less", it's log more cleverly. Sampling: not every request needs the full trace; a representative share plus full capture of every failure case and every negative user signal yields most of the knowledge at a fraction of the volume. Redaction: PII can be stripped or masked out of prompts and responses before they're persisted — you keep the structure of the trace without hoarding the sensitive content. Instrument for the cases that genuinely hurt, not for all of them equally.
And the most important limit of all: observability does not replace evals. It tells you that something changed — the tone is different, latency rose, cost doubled. It does not tell you whether the answer is correct. A trace can look impeccable and still contain a wrong answer; every metric is green and the output is still nonsense. Judging correctness is the job of evals — a suite that checks against known expectations. Observability and evals are two halves of the same system: one detects the change, the other judges it. Build only one and you have either an early-warning system with no judgement or a judgement with no early warning.
Conclusion
Classic observability was built for software that either runs or falls over. AI systems have a third state — runs, but worse — and it's precisely that state the old instrumentation is blind to. A wrong-but-plausible answer carries a 200 status code, arrives at normal latency and is indistinguishable in the logs from a correct one. Measure only error rates and latency and you won't see the silent degradation coming — the customer sees it first.
The discipline that follows is new but learnable: capture the full trace before you need it; treat cost per request as a first-class metric; watch the substance of the quality over time rather than just the infrastructure; and balance the whole thing with sampling, redaction and a real eval suite, so you neither drown in telemetry nor mistake change for correctness.
That's exactly how we treat AI features at NH Labs: we instrument them like production systems — full traces, cost per request, eval trends over time — so that silent degradation surfaces fast, not once the complaints start piling up. You can't debug what you can't see. So we make sure we can see it.