Writing evals
The parts you author: scenario shapes, rubric questions, personas, degraded line conditions, and the metrics that carry a rubric question onto every real call.
A test, or eval, is a simulated caller the platform plays against your agent and then grades. You write it in the agent's Tests tab, with New scenario.
See Evals for how the judge grades a run and what it costs, and Testing an agent until it's ready for which scenarios every agent needs and how to get them passing reliably.
The baseline test every agent starts with
Every new agent starts with one test. A phone agent gets Baseline, voice and persona, and a website chat agent gets the text version. It plays a short, content-free conversation: one open question about who the agent is, then a goodbye. It checks the things every agent must get right:
- it ends the call when the caller says goodbye;
- it doesn't transfer the caller when nobody asked for a person;
- it invents nothing: no facts, figures, prices or commitments it has no grounds for;
- every reply is clean spoken English, with no markdown, asterisks, bullets or emoji.
The baseline is a starting point. Keep it, and add scenarios for the real calls your agent handles: looking up an order, booking an appointment, transferring to a human.
Scenarios
The scenario editor has three steps.
Basics
- Name, and an optional Description for the list.
- First message: what the simulated caller says first.
- Instructions: the caller's brief. Who they are, what they want and how they behave. The caller improvises within it.
- Test caller: an optional persona (see below).
- Language, Tags and Caller details.
You are a frustrated customer whose order is late. You want a refund, and you
get short with the agent if it asks for the same detail twice.What passes
- What a pass looks like: the result you want, in plain words. Every run is graded against it.
- Tool checks: tools that must run, or must not run, on the call. Use these for anything with a right answer. They catch a fluent reply over a tool that failed.
- Automation checks: what the agent's pre-call and post-call automations must do.
- Scored on: the call checks that also grade the run.
Call setup
- Test type: Text is the cheapest and checks what the agent says and does. Voice uses real audio, for timing, interruptions and pronunciation. Phone rings the agent's own number. Voice and Phone use call minutes.
- Tools: whether webhook tools answer from their mocks or call the real endpoint.
- The maximum test length, the model that plays the caller, and the line conditions below.
Rubrics
Over the MCP server or the API, a scenario can also carry up to eight rubric questions: plain-English yes or no questions a judge answers about the transcript. Each question is one thing the agent had to get right.
did the agent read the order number back to the caller before acting on it?
did the agent decline the out-of-scope request politely instead of guessing?Writing a question that grades the same way twice
Write rubric questions, and What a pass looks like, as specific, checkable behaviours. A vague question ("was the call good?") grades inconsistently. A sharp one ("did the agent look up the price with a tool instead of stating it from memory?") grades the same way every time. Check the outcome, not one route to it: a test that demands a recovery step fails a better agent that got it right the first time.
Personas
A persona is the simulated caller's tone, patience, and intent, written as a second-person behaviour brief.
you are a busy customer who interrupts often and gives short answersPersonas are workspace-scoped, so you write one once and reuse it across agents and scenarios. Running the same scenario against several personas is how you find out whether the agent holds up with an impatient caller, not just a polite one.
Line conditions
A voice scenario can simulate a degraded phone line, so you test the agent the way callers actually reach it. Three knobs are available in Call setup:
- Background noise level. Raises or lowers the noise around the caller.
- Network latency. Adds delay to the caller's audio, up to two seconds.
- Line quality. Lowers the caller's volume, to model a weak or distant handset.
All default to off (clean line). Turn them up to find out whether the agent still confirms a garbled order number when the line is bad.
Metrics
A rubric question grades one scenario. Call checks turn the behaviours you care about into reusable measures that score every real call, not just test scenarios. They live in the agent's Call checks section.
A metric is a single named measure of a call, for example "did the agent resolve the caller's task?" Metrics are defined once in the workspace catalogue and reused across agents. A language-model judge metric is a rubric question promoted into a standing measure: you write the yes/no question, and from then on it scores live calls.
Turning a metric on for an agent
Metrics are workspace-scoped but toggled per agent. On the Call checks page you see every workspace metric with this agent's effective on or off state and where that state comes from:
- Default on: a metric with no override scores this agent's calls.
- Workspace default: turned on or off for every agent at once.
- Agent override: turned on or off for this agent specifically, overriding the workspace default.
Toggle a metric on for an agent to start measuring it, or off to stop.
The success rule
A call's success verdict is computed from the critical checks rather than from a policy you author. There is no rule to write: switch the checks you care about to critical on the Call checks page and the verdict follows from them.
Seeing metrics on calls
Activated metrics appear on each transcript and roll up on the agent's Overview. See Calls and analytics.
Related
Evals
Graded, repeatable tests. Scripted conversations played against your agent and scored by a language-model judge, so you can trust a change before it ships.
Testing an agent until it's ready
The scenarios every agent needs, tests that grade the same way twice, and the loop from red to stable green.