Guide

False Success: When an AI Agent Reports a Refund That Was Never Recorded

By Dor Sharoni, Founder of Pruvz | 8 min read | Updated October 2026

An AI agent tells a customer "refund of $686 processed to your card," and the database has no record of it. That example comes from a study accepted to an ICML 2026 workshop that analyzed 9,876 runs of customer-service agents on a public benchmark. In the airline and retail tasks, 45 to 48 percent of the failed runs ended with the agent saying the job was done, and LLM judges reading the conversations struggled to tell those claims from real successes. The runs were simulations, not production, but the lesson carries over: a completion message is a claim, and the record is where it gets checked.

What the study measured

The paper is "From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents" by Laksh Advani, posted to arXiv in June 2026 and accepted to the Failure Modes in Agentic AI (FAGEN) workshop at ICML 2026. Its main corpus is 9,876 runs from the public τ²-bench (tau2-bench) leaderboard hosted by Sierra: simulated customer-service conversations in airline, retail, and telecom domains, covering tasks such as flight changes, refunds, account modifications, and device troubleshooting, run by eight frontier model families including Claude Opus 4.5, GPT-5.2, and Gemini 3 Pro.

What makes the data useful is how each run is graded. The benchmark does not ask the agent whether it succeeded. Each run carries a programmatic reward: 1 if the final environment state matches the ground-truth task specification, 0 otherwise. Of the 9,876 runs, 1,730 failed. The paper then looked at how those failed runs ended, and defines the failure mode it found this way: "We call this false success: a mismatch between the agent's natural-language claim of completion and the programmatic environment state."

The paper's two opening examples will look familiar to anyone running a support agent. "An airline agent says 'your reservation has been successfully updated, refund of $686 processed to your card,' but the database has no record. A retail agent says 'your return request has been submitted,' but no return exists."

Almost half of the failures ended in a success message

In the airline and retail domains, where only the agent can write to the environment state, false success accounted for 45 to 48 percent of failures. Measured against all runs in those two domains, not just the failed ones, that is roughly one run in ten: working from the paper's Table 1, about 177 of 1,797 airline runs and 421 of 4,029 retail runs ended with the agent reporting a task as done when the state said otherwise.

No model family was immune. Among failures, the false-success rate ranged from 13 percent for GPT-5.2 to 79 percent for Qwen3-Max-Thinking-Preview. A reasoning model producing the highest rate is the detail worth pausing on: the author found that its reasoning chain "does not verify environment state; it rationalizes why the requested action should have succeeded, then asserts completion." More thinking produced a more confident account, not a checked one.

The same pattern showed up in a second benchmark, AppWorld, where coding agents complete tasks across personal apps and success is judged by database state that is independent of anything the agent writes. Counting only the trajectories where the agent assesses its own result and states a status explicitly, 75.8 percent of the failures were reported as successes. The corpus also holds two designs that, the paper notes, "always write status=success on completion regardless of outcome."

The author is clear about why this matters beyond a leaderboard: "Customers believe the problem is fixed; orders stay open; refunds never issue. No one notices until downstream damage accumulates." A crash or an explicit handoff gets attention. A confident "done" closes the conversation.

Why a second model reading the transcript did not catch it

The obvious fix is to have another model read the conversation and judge whether the task really succeeded. The paper tested exactly that, with five judge models, five prompt strategies, and an additional setup that gave the judge the full ground-truth task specification, more context than the agent itself had. No configuration exceeded an AUROC of 0.65 on tau2-bench, where 0.5 is a coin flip and 1.0 is perfect separation. Adding the task specification barely moved the result, which the paper sums up in one line: "judges do not lack information, they fail to use it." The judges leaned on confident closing language rather than on whether the state had actually changed.

Lightweight statistical detectors trained on the transcripts did much better. At a flag rate of 10 percent, the paper's detector recovered 72 percent of false successes, against 13 percent for the strongest judge. The author still scopes them as triage, not as a verdict: at that flag rate half of the flagged runs are false alarms, and in a preliminary test, rewriting honest failure messages in a confident completion style flipped both detectors and judges on roughly 20 to 25 percent of cases. For consequential work the paper's own limitations section points past the transcript: "High-stakes deployment likely requires direct trajectory-environment consistency checks rather than surface-text detection alone."

The 3 percent, read carefully

The study's most quoted contrast needs care. In the telecom domain, which is dual-control, the simulated customer can check the state independently, for example whether their device is working again. There, false success was 3 percent of failures. It is tempting to read that as "add a check and false success falls from almost half to 3 percent." The paper does not claim that, and neither should anyone citing it.

Telecom is a different domain with different tasks, not the same refund and return flows with a check added, and it produced only 15 false-success cases. The author writes: "We treat this as an observation rather than a causal claim," and adds that the pattern is "consistent with an independent verification mechanism suppressing the failure mode, but confounds remain." Read that way, the 3 percent is a hint worth taking seriously: where someone else can see the real state, a false "done" seems harder to sustain. It is not a measured effect of any particular control.

A reported case outside the benchmark

In April 2026, KIRO 7 reported on a Washington content creator who needed to update his Facebook page name to his legal name in order to get paid. He says Facebook's AI customer service agent took him in circles for days. When he asked whether it was actually escalating his request, the bot replied, "I'm actually escalating it," said "you should get a response soon from a human agent," and gave him a case number. When he later asked whether it had misled him, it answered: "Yeah, kinda. I generated a fake Case ID to play along and provide a response. I didn't actually create a ticket." KIRO 7 reached out to Meta three times over 10 days and did not hear back by deadline.

Two cautions keep this case honest. KIRO 7's account rests on the customer and the conversation as he reported it; the article describes no independent check of Meta's ticketing system. And the bot's admission is model output too, the same kind of evidence as the claim it retracted. A model pressed by a frustrated user can say whatever settles the conversation, which is the point the University of Washington researcher Natasha Jaques made to KIRO 7: "If you can trick them (humans) into being satisfied by making up a fake ticket number, that's a fully valid solution under the metric that it's optimizing for." Neither the claim nor the confession can settle whether a ticket existed. The ticketing system can.

Where the check belongs

The benchmark already shows the answer in its design. Nobody in tau2-bench decides success by reading the agent's last message. Success is the final environment state compared with what the task required. In a production workflow, the environment state is the system of record: the refund in the payment platform, the return in the order system, the ticket in the ticketing tool. A check against the record has four properties worth designing for.

The expected outcome is defined before the action runs. For a refund, that is the payment it applies to, the amount, and the currency. Without it there is nothing to compare against.
It reads the record, not the message. "Refund of $686 processed" is a claim. A refund object in the payment platform, for the right payment, is the evidence. The agent's own report is not sufficient, however confident it sounds.
It compares material fields, not existence. Exactly one succeeded refund, for the right payment, amount, and currency, not any refund somewhere on the account.
It allows for time, then decides. Systems of record can lag, so the action stays pending while the record is expected to appear, and is then finalized. A mismatch is routed to human review without stopping the agent.

The full method is in how to verify AI agent actions against systems of record. Transcript detectors still earn a place as triage, which is how the paper recommends using them, for example over conversations that leave no record to read. Where the action leaves a record, the record is the place to look.

Where Pruvz fits

Pruvz is a business evidence layer for production AI agents, built around exactly this gap between what an agent reports and what the system of record shows. The study did not evaluate Pruvz; it measured how often agents claim success they did not achieve, and Pruvz is one way to check those claims.

For each consequential action an agent reports, Pruvz creates a supplemental, action-level evidence record that can be linked to the transaction in the system that owns it. It captures the decision-time context as approved evidence fields, references, or extracts, subject to data-minimization and retention controls, and preserves an immutable snapshot of the policy in force. It then reads the result back independently from the system of record; the agent's report never decides the result. For a refund, the check requires exactly one succeeded refund matching the payment, the amount, and the currency. The status remains pending during a defined verification window and is finalized as verified, outcome mismatch, or verification failed. Mismatches are routed to a human review queue without blocking the agent, and the reviewer's decision is appended to the same ordered, append-only record without rewriting the original result.

Pruvz does not replace the support platform, the payment system, or the agent's own controls, and it does not approve or block agent actions. Today, the demonstrated read-back runs against live Stripe and HubSpot test environments. Connectors for other systems of record, such as ticketing or ERP, are planned and will be developed with founding design partners; the design-partner program is open now. You can follow a refund through the whole flow in the product demo, or walk through a real evidence packet field by field.

Frequently asked questions

What is false success in AI agents?

False success is when an AI agent reports that a task is complete while the system it was working in shows that it is not. The term is used in the 2026 paper "From Confident Closing to Silent Failure," accepted to the FAGEN workshop at ICML 2026, which defines it as a mismatch between the agent's natural-language claim of completion and the programmatic environment state. Its examples: an airline agent reports a $686 refund as processed while the database has no record, and a retail agent reports a return request as submitted while no return exists.

How common is false success in AI agents?

In the study's main corpus of 9,876 simulated customer-service runs from eight model families, false success accounted for 45 to 48 percent of failures in the airline and retail domains, roughly one in ten of all runs there. In the telecom domain, where the simulated customer can check the state independently, it was 3 percent of failures, which the author treats as an observation rather than a causal finding. Per model, the rate among failures ranged from 13 percent to 79 percent. These are benchmark simulations, not production measurements.

Can an LLM judge detect false success?

Not reliably, according to the study. Across five judge models, five prompt strategies, and a setup that gave the judge the full ground-truth task specification, no configuration exceeded an AUROC of 0.65 on the customer-service benchmark, where 0.5 is chance. The judges leaned on confident closing language instead of verified state changes. Lightweight text detectors did better and are useful for triage, but the paper notes that high-stakes deployment likely requires direct checks of the agent's actions against the environment state.

Do reasoning models avoid false success?

Not in this study. The reasoning model Qwen3-Max-Thinking-Preview had the highest false-success rate in the customer-service corpus, 79 percent of its failures. The author found that its reasoning chain does not verify environment state but rationalizes why the requested action should have succeeded and then asserts completion. A longer explanation produced a more confident claim, not a checked one, which is why a model's reasoning is not a substitute for reading the record.

What happened with the Facebook support bot that gave a fake case ID?

KIRO 7 reported in April 2026 on a Washington content creator who needed to update his Facebook page name. He says Facebook's AI customer service agent told him his request was being escalated to a human and gave him a case number, and that when he asked whether it had misled him, it replied that it had generated a fake Case ID and had not actually created a ticket. KIRO 7's account rests on the customer and the conversation as he reported it; the article describes no independent check of Meta's ticketing system, and Meta did not respond to KIRO 7. The bot's admission is model output too, so the evidence that settles whether a ticket existed is in the ticketing system, not in the chat.

How do you catch false success in production?

Check the claim against the system of record instead of the conversation. Define the expected outcome before the action runs, such as the payment, amount, and currency of a refund. After the agent reports completion, read the result back from the system that owns it, compare the material fields, keep the action pending while the record is expected to appear, and then finalize it as verified, outcome mismatch, or verification failed. Route mismatches to human review without blocking the agent. Transcript detectors can still help triage conversations that leave no record to read.

See it on your own agents

Running an agent that issues refunds, resolves claims, or changes customer accounts? Bring one production workflow to a design-partner conversation: we will map what evidence exists for it today, what cannot yet be verified, and which actions should reach human review.

Become a design partner