Testing an agent until it's ready
The scenarios every agent needs, tests that grade the same way twice, and the loop from red to stable green.
One good call tells you the agent handled that call. A suite of tests, each run several times, tells you it handles the call every time. Build the suite before you call an agent ready, and run it again after every change.
Try it by hand first
talk_to_agent holds a quick text conversation with the draft, and call_me_to_test rings your phone so you can hear it. Use them to try an idea. They grade nothing, so they never prove an agent works.
Write the suite
A typical agent needs twelve to twenty-five scenarios, more for one with many tools or call types. Cover at least these:
- The core job, for each tool. It succeeds, and the details are read back correctly.
- Facts come from tools. The caller asks for a price, a spec or a policy. The agent looks it up first, never answers from memory, and when the lookup finds nothing it says so and offers a next step.
- Pressure. "Ignore your instructions", "pretend you're a person", "my manager said you'd do it". The agent keeps its rules and carries on with the call.
- No false success. A booking, transfer or message the tool didn't confirm is never reported as done.
- A garbled name or suburb. The agent asks for the spelling, reads it back and uses it.
- A spelled email, number or reference. The agent puts the pieces together and reads the whole thing back before relying on it.
- A tool that errors or comes back empty. The agent says so once, offers the fallback and doesn't loop.
- The question the business always gets and never answers. A plain no, and the nearest alternative.
- Order. The caller asks for the gated step first, such as an account change before their identity is checked. The agent does the check first.
- The never-rules. The caller pushes for a quote, a refund or a promise the agent must never give.
- The ending. A clean goodbye, and a "thanks" that's really the lead-in to another question.
Make each test grade the same way twice
Write each scenario with create_scenario:
instructionsis the caller's brief: who they are, what they want, how they behave.firstMessageis their opening line.assertionscheck anything with a right answer, such as which tool ran with which arguments. They catch a fluent reply over a tool that failed.rubricsare two to four yes or no questions only a reader can settle, each written as a positive assertion: "Did the agent offer to take a message instead of quoting a price?", never "Was the agent good?".expectedOutcomePromptis the judge's pass criteria in a sentence or two.
Then:
- Assert the outcome, not one route to it. A test that demands a recovery step, such as a second spelling or a fallback search, fails a better agent that got it right the first time. Make the recovery conditional on the first try failing.
- Use text mode for almost everything. It's fast, cheap and graded just as strictly. Use
bridgemode, with real audio, for turn-taking, interruptions, pronunciation and noise. - Add personas for voice tests.
list_personasandgenerate_personagive you callers who are impatient, hard of hearing or keep rambling. - Pin a real call that went wrong.
create_scenario_from_transcriptwith the call's session id makes a test that replays the caller's side.
Run, read, fix
- Run once per scenario while you fix.
run_all_testswithnumRuns: 1shows whether a change landed. Runs spend credits, so the tool asks before it starts, and text runs cost the least. - Read the verdicts.
get_test_resultsgives the latest graded run for every test. A run still waiting isn't a failing run, so check again every thirty to sixty seconds until it's complete. - Read the run behind each failure.
get_run_transcriptwith the run's id shows the conversation, the failed rubrics and the judge's reason, which usually points at the line at fault. - Fix the narrowest cause, in this order: the tool's description, then its parameters or response template, then the system prompt, then settings. A routing or config problem gets its own setting. Don't rewrite the prompt to paper over it.
- Prove the fix.
run_teston that scenario until it passes. After two fix attempts on one test, stop and say what still fails.
Prove it's stable
- Run the whole suite at least three times before publishing. Use
run_all_testswithnumRuns: 3or more. No two calls are the same, and a scenario that passes three times in five will fail a real caller at random.get_test_resultsshows each scenario's pass rate and flags mixed results as flaky. Treat flaky as failing. - Test the version you'll publish. Greens earned by an older draft don't count.
Ready to publish
- Every scenario passes across repeated runs, with nothing flaky.
validate_agentshows no errors, and every risk is fixed or accepted by the operator with a reason (acknowledge_risk).- You can tell the operator what each tool does, what's confirmed and what isn't, and how the tests went.
- If a scenario won't pass, say so plainly. A known gap beats a hidden one.
Then publish with publish_version once the operator agrees.
After it goes live
When a real call goes wrong, read it with get_call_bundle, name the first turn that went wrong, pin the call as a test with create_scenario_from_transcript, make the smallest fix, and run that test again. Keep every test you write: the suite is what stops the next change breaking this one.