Skip to content

Demos test reasoning. Production needs someone on the pager.

Teams treat a working demo as proof of readiness. In production the failures are expiring sessions, silent prompt regressions and loops, so give the agent one accountable owner.

In brief

  • A vendor-run 2026 survey by Inngest of 130 engineers found only 19% of AI teams very confident their stack could handle two to three times their current scale.
  • Much of the breakage sits in systems the agent touches, and product and platform teams disagree on who owns it.
  • Name one reliability owner who can pause the agent, gate prompt changes like code, and cap cost and steps per task.

An engineer at a 40-person SaaS company estimated two weeks for an internal finance agent that pulled vendor cost data from six portals. It took about four months, by their own account on r/LLMDevs.

A demo covers reasoning and tool choice. Production adds two-factor codes with a five-minute window, sessions that expire unpredictably and retry loops nobody notices. When something goes wrong, the platform team blames the prompt while the product team blames the harness (the code running the agent's loop and tools).

We would staff the agent like a service, not a feature. Give it one named reliability owner with authority to pause it, and make a shared evaluation suite, the agreed test cases and scores, the contract between product and platform. Gate prompt changes like code, detect drift with synthetic transactions (scripted test runs against live systems), cap cost and loops per task, and give the agent a designed way to say it is stuck.

The breakage sits at the edges of other people's systems

If the demo is the first 70% of the work, the last stretch took roughly eight times the original estimate for the whole job.

The extra months went into roughly 800 lines of edge-case glue. The failures were email two-factor codes, a CAPTCHA variant (a challenge meant to block bots) and a layout change that went unnoticed for three days.

Session timeouts ranged from 30 minutes to 24 hours, with no way to check session health. None of it was reasoning.

In the replies, one commenter described a token's expiry quietly shrinking from 24 hours to one hour, which burned credentials for three days before an alert fired.

The same commenter suggested budgeting roughly half to one engineer permanently for such an integration. Another said security sign-off for an agent holding production credentials became a multi-week blocker.

Some failures look like success. On r/LocalLLM, a developer described a one-line system prompt edit shipped with a deploy. By their own measure, quality fell from 84% to 52%, while HTTP responses, latency and error rates stayed green for 11 days until a customer complained.

One symptom was the model telling users refunds were valid for 60 days when the policy was 30.

Then there are loops. On r/AI_Agents, a practitioner described two agents with overlapping duties consulting each other for 60 rounds while cost climbed. Better prompts did not fix it. A middleware check that blocked repeated tool calls did.

Reliability falls between product and platform, so nobody owns it

A practitioner on r/AI_Agents who reported 50-plus conversations with engineering teams summarised the pattern: most agent failures were not model failures, prompt changes were tested more casually than code, and nobody agreed who owned reliability.

Survey data, thin and vendor-run, points the same way. Inngest, a workflow-orchestration vendor, surveyed 130 engineers running production workflows for a 2026 report. Only 19% of teams running AI were very confident their stack could handle two to three times their current scale. Among organisations with 500 or more engineers, a small subgroup of 16, that fell to zero.

Observability, seeing what a running system is doing, was the most-named unsolved problem. One in five AI teams said they spend up to half their engineering time on reliability, twice the rate of non-AI teams.

Gartner, as reported by ITPro in June 2025, predicted that 40% of agentic AI projects would be cancelled within two years over rising costs, weak returns and poor risk controls.

Better models will shrink some of this as they improve at long tasks, memory and recovery. But two-factor codes, expiring sessions, changed layouts and shortened tokens belong to the systems the agent touches, not the model, and no release will fix them. They are operating cost, and they need an owner like any production service.

A named owner with pause authority settles the product-platform standoff

What we put in front of teams is a one-page ownership sheet with a name, not a team, in every row.

Each reliability concern needs one named person and a specific signal that pages them.
ConcernOwnerWhat pages them
Definition of good output and the eval suiteProductEval scores below the ship threshold on any change
Runtime, retries, queues and costPlatformError-budget burn (failures beyond the agreed allowance), cost per task above cap, loop breaker trips
External dependencies (portals, APIs, credentials)Integration ownerSynthetic transaction failure; login or CAPTCHA challenge
Semantic quality in productionReliability ownerSampled quality score drops; override rate (how often people correct the agent) spikes
Pause, rollback and customer communicationReliability ownerAny of the above at severity one

Source: Wennovate framework.

The reliability owner is the person with authority to pause the agent and roll back a prompt, model or tool change. Product still owns what "good" means, and platform still owns the runtime. The sheet only works if one person can act when those two disagree at 2 a.m.

Prompt gates, task budgets and quality alerts catch what uptime misses

Treat prompts as code. Version them, test them against the eval suite, review them and keep a rollback. A model or provider change goes through the same gate.

Detect, do not just recover. One commenter on the demo-gap thread ran hourly synthetic transactions against each portal to catch drift early. Separate recoverable failures, like a stale selector (a page element the agent can no longer find), from account-risk signals, like a login challenge, which should halt the agent rather than retry.

Budget every task and design the stuck state. Set a cost ceiling and a maximum step count per task, and keep a ledger of tool calls so repeated calls are blocked. When the budget trips, or the agent cannot finish, it hands a resumable task, with its context, to a human queue.

Page on meaning, not only on uptime. Score a sample of production outputs continuously and alert on drops. The 11-day silent regression above had perfect infrastructure metrics.

Set objectives a customer would recognise, and a runbook to defend them. Uptime is not the promise. Two measures usually are: the share of tasks completed correctly and the time from a stuck task to a human picking it up. Before launch, write the runbook, the step-by-step incident procedure: how to pause, roll back a prompt, replay failed tasks and tell affected customers. Review incidents with the change log open, because the cause is often a prompt, model or provider change rather than code.

For Canadian banks and insurers, OSFI's E-21 turns the runbook into evidence

The Office of the Superintendent of Financial Institutions (OSFI), which oversees federally regulated banks and insurers, published its operational resilience Guideline E-21 in August 2024. It expected those institutions to have identified and mapped their critical operations and set tolerances for disruption by 1 September 2026. End-to-end scenario testing of all critical operations is expected by 1 September 2027, with critical third parties taking part on a best-effort basis. An institution whose critical operation includes an agent will need evidence that it stays within tolerance. Expect to be asked for your runbook, your incident history and your rollback time.

Put it to work: Name the person on the pager before launch

  1. Whose name is in the reliability row, and can that person pause the agent tonight without asking anyone?
  2. Did the last prompt or model change pass the same eval gate as a code change?
  3. When a task hits its cost or step cap, does it stop and reach a person, or retry?

If nobody accepts the pager, the agent is not ready

The most costly omission is an agent with no "I'm stuck" state and no human takeover path, so its fallbacks retry silently. Close behind is treating prompts as configuration anyone can edit, when one line can move quality as far as a code change.

Paging only on HTTP errors and latency misses the failures that matter. Assigning reliability to "the AI team" leaves it with no one; it belongs to a person. And some agents should have been workflows: Gartner's own advice is to keep agents for decisions and use automation for routine work.

Before the next agent goes live, put a name in every row of that sheet. If no one will accept the reliability row, the agent is not ready.

Running AI across your teams? Book a 30-minute discovery call with the Wennovate team. No pitch deck required. Start with a scoping call

The Studio Letter

One letter a month from the CEO: what moved, what stalled, what we learned.

Get the Studio Letter