Most AI agent demos work beautifully.

The model calls the right tools, chains reasoning steps together, recovers gracefully from errors, and delivers the result in a tidy little terminal session. You record it, post it, get the engagement. Then you try to put it in production and it falls apart in ways that feel impossible to reason about.

This isn’t a bug. It’s a fundamental architectural mismatch that the industry is only beginning to take seriously.

The Core Problem: You Built a Deterministic System Around a Probabilistic Component

Traditional software systems have a contract: same input, same output. This is so deeply assumed in how we build infrastructure — retries, idempotency keys, circuit breakers, health checks — that we’ve stopped noticing it. It’s just how software works.

LLMs violate this contract at the foundation.

Give the same prompt to the same model at the same temperature twice and you may get different tool calls, different reasoning chains, different decisions. The model isn’t broken. That’s how it works. Non-determinism isn’t a bug to be squashed — it’s a core property of the system.

The trap is treating it like a bug. Teams add retries. Retries make it worse. The agent calls a payment API twice, sends two emails, creates two tickets. Now you have an idempotency problem on top of a reliability problem.

// This looks fine. It isn't.
async function runAgent(task: string) {
  try {
    return await agent.run(task);
  } catch (e) {
    // "Just retry it" — famous last words
    return await agent.run(task); // may duplicate side effects
  }
}

The retry will re-run the entire reasoning chain. The model may take a completely different path. Your downstream systems aren’t prepared for that.

Non-Determinism Has a Blast Radius

The failure mode isn’t just occasional wrong answers. The blast radius of non-determinism scales with the permissions you grant the agent.

An agent that can only read files? Non-determinism means wrong answers. Annoying, fixable.

An agent that can read and write files? Non-determinism means corrupted state. Harder.

An agent that can call external APIs, provision infrastructure, send communications? Non-determinism means production incidents. You’re now in SRE territory, not demo territory.

This is why least-privilege isn’t just a security concern for AI agents — it’s an operational resilience concern. Every capability you add to an agent multiplies the blast radius of every wrong decision it makes. The principle should be: grant the minimum set of tools required to complete the task, evaluated per task, not per agent deployment.

Static tool grants — “this agent gets these 20 tools” — are lazy infrastructure. They’re the equivalent of running your web server as root because it’s easier than thinking about permissions.

The Observability Gap

Here’s something that surprises teams: your existing observability stack is nearly useless for debugging AI agents.

Distributed tracing tells you what happened. For agents, you need to know why the model decided to do that. Those are completely different questions.

A trace might show:

→ agent.run() [2.3s]
  → tool.readFile("/config/prod.yaml") [12ms]
  → tool.writeFile("/config/prod.yaml") [8ms]  ← PROBLEM

But it won’t tell you why the model decided to overwrite the config, what reasoning chain led there, what context it had at that decision point, or whether that decision would reproduce under the same inputs. You’re debugging a black box with a flashlight.

Effective agent observability requires capturing:

1. The full prompt context at each decision point — not just the final prompt, but the accumulated context window including prior tool outputs
2. Tool call rationale — the model’s stated reasoning before each action (chain-of-thought should be logged, not discarded)
3. Branching paths — when the model considered multiple actions and chose one, what were the alternatives?
4. Context window pressure — how full was the context at decision time? Models degrade in surprising ways as context fills

Most teams log input → output. That’s not sufficient. You need the full reasoning trace, and you need it correlated with outcomes so you can identify which reasoning patterns lead to failures.

The “Just Add Guardrails” Trap

The instinct when agents misbehave is to add output validation. If the model does something bad, check the output and reject it.

This is necessary but not sufficient, and it creates a false sense of security.

Output validation catches the bad action after the reasoning is already broken. By the time your guardrail fires, the model has already decided to do the wrong thing. You’ve stopped the action, but you haven’t fixed the reasoning. The model will find a different path to the same bad outcome — or it’ll give up and return a subtly wrong result that passes validation.

The better intervention point is at the tool invocation layer, not the output layer.

// Weak: validate after the fact
const result = await agent.run(task);
if (isSafeOutput(result)) return result;
else throw new Error("Unsafe output");

// Stronger: gate at the action boundary
const tools = createToolset({
  writeFile: withApproval(writeFileTool, {
    requireConfirmation: isProductionPath,
    auditLog: true,
    rateLimit: { calls: 5, window: '1m' }
  })
});
const result = await agent.run(task, { tools });

Every tool invocation is a decision boundary. Treat them like transactions: they should be auditable, rate-limited, idempotent where possible, and reversible where feasible.

State Management Is the Unsolved Problem

Demos avoid state. Production agents can’t.

A multi-step agent task is essentially a distributed transaction with a probabilistic participant. You need to handle partial completion, rollback, resumption after failure — all the hard problems of distributed systems — except your transaction coordinator is a language model that might change its mind.

Most agent frameworks don’t have a serious answer to this. They give you conversation history (a flat list of messages) and call it state management. That’s not enough.

Production-grade agent state needs to be:

• Persistent — survives crashes, restarts, and context window resets
• Inspectable — human-readable, queryable, diffable
• Reversible — checkpoints before destructive operations
• Scoped — task-local state vs. long-term memory vs. shared team context are different things and should not live in the same flat list

Until the frameworks take this seriously, you’ll need to build it yourself. That means explicit checkpointing before high-risk tool calls, structured state objects outside the context window, and recovery procedures that can resume mid-task from a known-good checkpoint.

What Good Agent Infrastructure Actually Looks Like

Stop thinking of the agent as your system. The agent is a component in your system. Design accordingly.

Gateway pattern: Route all agent tool calls through a single gateway service that handles auth, rate limiting, audit logging, and capability enforcement. The agent never calls external services directly. This is the “least-privilege agent gateway” pattern and it’s the most impactful structural improvement you can make.

Structured reasoning capture: Before each tool call, require the model to output structured reasoning (tool choice + rationale as a typed object). Store it. This forces the model to commit its reasoning before acting and gives you the debug data you need when things go wrong.

Idempotency by default: Design all agent-facing tool implementations to be idempotent. The agent will call them multiple times. That’s not a bug you’ll fix — design your tools to survive it.

Human escalation paths: Define the conditions under which the agent should stop and ask for confirmation rather than proceeding. Build these as first-class infrastructure, not afterthoughts. The signal to escalate should come from the task specification, not from post-hoc output checking.

The Shift That Needs to Happen

The industry is still treating AI agents as chatbots with tools. The mental model is “chat with capabilities” — same reliability bar, same operational model, just with function calling bolted on.

That mental model is wrong and it’s causing production failures at scale.

Agents are autonomous processes with access to external systems. The mental model should be closer to a microservice: needs its own observability, its own SLA, its own failure modes, its own deployment considerations. Except harder, because the business logic is stochastic and opaque.

The teams shipping reliable agents in production have internalized this. They’re not asking “how do we make the model more reliable?” — that’s a question you can’t fully answer. They’re asking “how do we build a system where non-determinism is contained, auditable, and survivable?”

That’s the right question. It’s an infrastructure problem, not an AI problem.

Start there.