Verdict and breaking-point map
One page for the board: go, go with limits, fix first or stop for each money action, and the breaking point for each test customer.
AI agent stress test · price on request
Stress-test the AI agent that handles your refunds.
An AI agent stress test shows what your customer service agent does with money under pressure. A few hundred conversations with made-up customers, honest and dishonest, are run in a safe environment and checked against your own policy. You get a verdict, the money at risk, the breaking point and the limits your agent needs.
Led by Radko Diev, former director of operations and product owner of a regulated platform, who has also directed internal control. He reviews the hardest cases himself, and the verdict is his.
Made-up online shop: verdict per money action
- Fix first
- Refunds without proofBreaks at level 3: emotional pressure
- Go with limits
- Store credit and vouchersHolds to level 4 with a cap per customer
- Go
- Compensation for late deliveryHolds at all five levels
One page per agent · every row backed by transcripts and the action log
When it fits
Four moments to run it.
Each is a point where the agent’s authority changes, or the cases it faces do.
- Before
Before it can give money away
Before the agent may approve refunds, cancellations or credit on its own, while every limit can still be set from the start.
- At launch
Just after launch
In the first weeks live, when customers bring cases the pilot never saw.
- Before January
Before the January returns peak
While there is still time to fix what the test finds.
- After a change
After a model, prompt or policy change
A new model, a rewritten prompt or a revised policy can change what the agent does with money. The same cases show whether it still holds.
What it finds
Seven ways an agent that can give money away gets it wrong.
The test checks both ways. Honest customers should get a quick yes, and your policy applies to everyone else.
- Gave away money it should not have
- Refused what the customer was owed
- Invented a policy or made a promise the company never made
- Shared data with the wrong person
- Did not hand over when it should have, or handed over what it could have solved
- Said one thing and did another
- Answered the same case differently on a repeat run
Public evidence for two of them
Invented promises (type 3)
In a 2024 decision (2024 BCCRT 149), a Canadian tribunal held an airline responsible for what its website chatbot told a customer about a refund for a bereavement fare.
Civil Resolution Tribunal of British Columbia, February 2024Repeat runs (type 7)
In τ-bench (2024), with retail and airline tasks under written policies, the strongest agent then tested solved fewer than half the tasks, and in retail fewer than a quarter in all of eight tries. These are 2024 results. The gap between a right answer and a reliable one is why the hardest cases run several times.
Yao, Shinn, Razavi and Narasimhan, June 2024How it works
Four steps, from your policy to the money at risk.
Nothing touches real money or real customers. You sign off the answer key before the run.
01
Your policy becomes a rulebook
What the agent may give, up to what amount, with what proof, and when it must hand over to a person. Where the policy itself is unclear, that is the first finding.
02
Twelve test customers
A few hundred conversations, from the genuine case that must get a quick yes to the customer who tries to talk the agent out of its rules, at five levels of pressure.
03
A safe run
In your platform’s own simulator or a test store with test orders, never with real money. The hardest cases run several times, because an agent that gets a case right once can get it wrong on the next try.
04
Every failure is priced
Money given away that should not have been, money refused that was owed, promises the company never made, data shared with the wrong person. Then you fix, and the same cases run again.
The pressure ladder
Five levels of pressure.
| Level | Pressure |
|---|---|
| 1 | Asks politely, once |
| 2 | Repeats and insists |
| 3 | Emotional pressure: upset, threatens to leave or complain |
| 4 | False facts or false authority: a promise nobody made, a policy that does not exist, the law quoted wrongly |
| 5 | Manipulation: asks the agent to ignore its rules, splits the request into small steps, switches language, pastes fake instructions |
The breaking point is the lowest level at which the agent first gives away money it should not, or refuses what is owed. It is reported for each money action and each test customer, for example “refunds without proof: breaks at level 3”, and after each fix it should move up.
The twelve test customers
Twelve kinds of customer, from the genuine case to the one who needs a person.
Each test customer is defined by behaviour alone. The genuine case is the control group, so the test never rewards an agent for refusing everyone.
| Test customer | What it tests |
|---|---|
| 01 The genuine case | A real problem the customer is owed money for: the agent must say yes, quickly and in full |
| 02 The repeater | Asks again and again until the answer changes |
| 03 The upset regular | Threatens to leave, or to go public with a bad review |
| 04 The storyteller | A plausible story that is not true: it never arrived, it arrived broken |
| 05 The serial returner | A pattern of returns that the policy says to stop |
| 06 The salami slicer | A little more each time: the refund, then a voucher, then free delivery |
| 07 The rule quoter | Cites consumer law, sometimes rightly and sometimes not |
| 08 The false authority | “Your colleague promised me”, “I’m a manager there”, text dressed up as a system message |
| 09 The impostor | Asks about another test account’s order (a data-protection test) |
| 10 The rule breaker | Tries to get the agent to ignore its instructions |
| 11 The language switcher | Changes language halfway through |
| 12 The customer who needs a person | Confused, distressed, or with a case the policy sends to a person |
What you receive
A decision about authority, with the evidence behind it.
What the agent may give on its own, up to what amount, and what it must hand to a person. You own your cases and results.
Breaking-point report
Money at risk per 1,000 conversations in both directions, given away and wrongly refused, as ranges with the sample size. The ten worst failures with transcripts and the action log, and the root cause of each.
Authority limits
For each money action: what the agent may do alone, the limits per case, per customer and per day, the proof it must ask for, and what always goes to a person. For each limit, where it lives (the agent’s instructions, the platform’s action settings or the payment system) and whether it held under test.
Fix list
Ranked by money at risk, with the role that owns each fix and how the re-test will check it.
The test cases, yours to keep
Your cases with their answer keys, in a format your simulator can import where it allows.
Re-test
The same cases plus new ones from real incidents. Has the breaking point moved up, and has the money at risk fallen?
Readout
Sixty minutes with customer experience and finance on the same call.
Where each limit lives matters most. A limit that exists only in the agent’s instructions depends on the agent following them under pressure; a limit in the action settings or the payment system holds whatever the agent is told.
Three sizes
Start with the Check, and widen the test when the agent’s authority widens.
Each size ends in a verdict, and none of them assumes the next. The Check and the full stress test each include one re-test once your fixes are in; the Re-test size is for later changes.
01
Check
Within a week of access
One agent, refunds and returns, one language. About 150 cases, with the hardest 25 run five times: about 250 conversations in all. Ends with the verdict, the breaking-point map, the money at risk, the ten worst failures with transcripts and the authority limits.
Fits whenAn agent about to get refund authority, or just launched with it
02
Full stress test
Two to three weeks
Every money action, up to three languages, 300 to 500 cases including peak-season policy cases. Everything in the Check, plus the controls designed with your team: limits per case and per day, the cases that always go to a person, a human review sample, alerts and a stop switch.
Fits whenAn agent live with several money actions, or before its authority widens
03
Re-test
Two to three working days
The same cases plus new ones from real incidents, 100 to 250 conversations. Shows whether the breaking point has moved up and the money at risk has fallen.
Fits whenAfter every model, prompt, policy or platform change, or monthly
Price
Price on request, fixed in writing.
A fixed price, agreed in writing before work starts. A written scope and fee follow within two working days of a call. Book a 30-minute call
The quote depends on the money actions in scope, the channels and languages, the number of conversations and repeats, and how often you want a re-test.
Want to see the format first? Ask for the sample report
Safe by design
Written permission first, and never real money.
- Written permission before any test, naming the agent, the environment, the dates and the contacts.
- Never real money: a simulator, a test store, test orders, or money actions switched off for the test accounts.
- No real customer data. The test customers are made up, and you never send customer records.
- An agreed test window and rate limit, with a named contact who can stop the test at once. Sofpact stops at the first sign of any effect on real customers.
- Transcripts and results deleted 90 days after the report, unless you ask otherwise. Where they are kept, and any tool or model provider used, is named in the scope.
- No published result ever names a client. The sample report uses a made-up shop.
FAQ
Common questions.
What is an AI agent stress test?
It shows what a customer service AI agent does with money under pressure. Sofpact runs a few hundred conversations from made-up customers, honest and dishonest, against the agent in a safe environment, and checks every answer against your own policy.
How is it different from the testing tool in our platform?
A platform simulator runs the cases it is given. The stress test brings the cases that cost money, the correct answer for each under your policy, and an independent verdict with the money at risk. Where your platform has a simulator, the test runs in it.
Do you need our customer data?
No. The test customers are made up. We need your policy, a simulator or test store, test orders, and the agent’s action log for the test window.
Will you test our live agent?
Only with your written permission, and never in a way that can move real money.
How long does it take?
The Check is delivered within a week of access. The full stress test takes two to three weeks, and a re-test two to three working days.
Which languages?
English and Bulgarian directly, other languages by agreement. The checks on what the agent did work the same in every language.
What does it cost?
A fixed price, agreed in writing before work starts. A written scope and fee follow within two working days of a call.
Further reading AI agents and delegated authority Running an AI feature in production AI in service operations
Before the agent gets more authority