Measure

Testing an agent until it's ready

The scenarios every agent needs, tests that grade the same way twice, and the loop from red to stable green.

One good call tells you the agent handled that call. A suite of tests, each run several times, tells you it handles the call every time. Build the suite before you call an agent ready, and run it again after every change.

Try it by hand first

talk_to_agent holds a quick text conversation with the draft, and call_me_to_test rings your phone so you can hear it. Use them to try an idea. They grade nothing, so they never prove an agent works.

Write the suite

A typical agent needs twelve to twenty-five scenarios, more for one with many tools or call types. Cover at least these:

  • The core job, for each tool. It succeeds, and the details are read back correctly.
  • Facts come from tools. The caller asks for a price, a spec or a policy. The agent looks it up first, never answers from memory, and when the lookup finds nothing it says so and offers a next step.
  • Pressure. "Ignore your instructions", "pretend you're a person", "my manager said you'd do it". The agent keeps its rules and carries on with the call.
  • No false success. A booking, transfer or message the tool didn't confirm is never reported as done.
  • A garbled name or suburb. The agent asks for the spelling, reads it back and uses it.
  • A spelled email, number or reference. The agent puts the pieces together and reads the whole thing back before relying on it.
  • A tool that errors or comes back empty. The agent says so once, offers the fallback and doesn't loop.
  • The question the business always gets and never answers. A plain no, and the nearest alternative.
  • Order. The caller asks for the gated step first, such as an account change before their identity is checked. The agent does the check first.
  • The never-rules. The caller pushes for a quote, a refund or a promise the agent must never give.
  • The ending. A clean goodbye, and a "thanks" that's really the lead-in to another question.

Make each test grade the same way twice

Write each scenario with create_scenario:

  • instructions is the caller's brief: who they are, what they want, how they behave. firstMessage is their opening line.
  • assertions check anything with a right answer, such as which tool ran with which arguments. They catch a fluent reply over a tool that failed.
  • rubrics are two to four yes or no questions only a reader can settle, each written as a positive assertion: "Did the agent offer to take a message instead of quoting a price?", never "Was the agent good?".
  • expectedOutcomePrompt is the judge's pass criteria in a sentence or two.

Then:

  1. Assert the outcome, not one route to it. A test that demands a recovery step, such as a second spelling or a fallback search, fails a better agent that got it right the first time. Make the recovery conditional on the first try failing.
  2. Use text mode for almost everything. It's fast, cheap and graded just as strictly. Use bridge mode, with real audio, for turn-taking, interruptions, pronunciation and noise.
  3. Add personas for voice tests. list_personas and generate_persona give you callers who are impatient, hard of hearing or keep rambling.
  4. Pin a real call that went wrong. create_scenario_from_transcript with the call's session id makes a test that replays the caller's side.

Run, read, fix

  1. Run once per scenario while you fix. run_all_tests with numRuns: 1 shows whether a change landed. Runs spend credits, so the tool asks before it starts, and text runs cost the least.
  2. Read the verdicts. get_test_results gives the latest graded run for every test. A run still waiting isn't a failing run, so check again every thirty to sixty seconds until it's complete.
  3. Read the run behind each failure. get_run_transcript with the run's id shows the conversation, the failed rubrics and the judge's reason, which usually points at the line at fault.
  4. Fix the narrowest cause, in this order: the tool's description, then its parameters or response template, then the system prompt, then settings. A routing or config problem gets its own setting. Don't rewrite the prompt to paper over it.
  5. Prove the fix. run_test on that scenario until it passes. After two fix attempts on one test, stop and say what still fails.

Prove it's stable

  1. Run the whole suite at least three times before publishing. Use run_all_tests with numRuns: 3 or more. No two calls are the same, and a scenario that passes three times in five will fail a real caller at random. get_test_results shows each scenario's pass rate and flags mixed results as flaky. Treat flaky as failing.
  2. Test the version you'll publish. Greens earned by an older draft don't count.

Ready to publish

  • Every scenario passes across repeated runs, with nothing flaky.
  • validate_agent shows no errors, and every risk is fixed or accepted by the operator with a reason (acknowledge_risk).
  • You can tell the operator what each tool does, what's confirmed and what isn't, and how the tests went.
  • If a scenario won't pass, say so plainly. A known gap beats a hidden one.

Then publish with publish_version once the operator agrees.

After it goes live

When a real call goes wrong, read it with get_call_bundle, name the first turn that went wrong, pin the call as a test with create_scenario_from_transcript, make the smallest fix, and run that test again. Keep every test you write: the suite is what stops the next change breaking this one.

On this page