Evals
Graded, repeatable tests. Scripted conversations played against your agent and scored by a language-model judge, so you can trust a change before it ships.
An eval does repeatably, and at scale, what you do by hand in the Test panel.
Evals give you three things you cannot get by retrying calls by hand:
- Repeatability. Run the same conversation again after a prompt or tool change and see whether the result held.
- Breadth. Cover the awkward cases on purpose: the caller who changes their mind, gives a wrong detail, talks over the agent, or asks something out of scope. You would never think to retry those by hand every time.
- Confidence across runs. Trust a result that holds across several runs of the same scenario, never a single pass. Five runs tell you far more than one green run.
Together they move you from "it worked when I tried it" to "it works every call".
Why a single green call proves nothing
The agent runs on a language model, so the same prompt can answer beautifully on one call and stumble on the next. The bar for a phone agent is reliability: it has to behave correctly on every call, not most calls.
One bad call in ten is too many when a caller is waiting on the line. The discipline that follows: write a failing scenario for the behaviour you want, then change the prompt or a tool until it passes and stays passing.
Evals run against whichever version you point them at: the published version, or the draft you are currently iterating on. So you can prove a change before you publish it, not after.
Live testing with the test panel
Before you write an eval, talk to the agent yourself. The Test panel holds a live conversation with your draft, one call at a time, exactly as a caller would: how it greets, how it handles a question, whether it reaches for the right tool.
It is the right tool for "does this even work", not for proving a change holds. See Testing before you ship.
What you author
You write each eval in the agent's Tests section: a scenario, its rubric, and the persona the simulated caller runs as. All three, plus line conditions and metrics, are on Writing evals.
The judge
Every scenario is graded by a language-model judge: a separate model that reads the transcript and answers each rubric question. It runs on a Balanced-tier model, deliberately stronger than the model the agent itself runs on, which is the standard practice for grading.
It answers each rubric question pass or fail with a one-line reason, and it is told to lean "fail" when genuinely uncertain, because a false pass hides a real defect.
did the agent read the order number back? pass
did the agent look up the price? fail
quoted a price without calling a toolVerdicts
- Passed: the agent satisfied every check.
- Failed: one or more rubric questions or tool checks failed, or a call check you marked critical failed. The judge's reason for each tells you what went wrong, and often points straight at the line of the prompt or the tool description at fault.
- Scoring: the run's own checks passed and its critical call checks are still being scored.
- Error: the run could not complete (a misconfigured tool, for example).
You can override a run's verdict by hand, and the override wins wherever the result is read.
When a scenario fails, read the per-question reasoning before changing anything. The judge usually pinpoints the exact behaviour that broke, which is a far better guide than re-running and hoping.
Structural checks
Alongside the judge, fast model-free structural checks run on the transcript too: catching the agent repeating itself on consecutive turns, stalling after a tool call, or speaking scaffolding it should never voice aloud.
These are the cheap first line that catches obvious regressions before any judge spend.
Running one scenario
Run one scenario on its own for quick feedback while you iterate on a prompt or a tool. The workflow: pick the scenario that covers the behaviour you are changing, run it, read the judge's reasoning, edit, run again. Stay on one scenario until it passes reliably, then run the full batch to check you did not break anything else.
Running a batch
To prove the agent overall, run a batch: the whole suite at once, or a configured sweep. A batch expands across three axes, so you can be deliberate about coverage:
- Scenarios: every eval, or a chosen subset.
- Runs per scenario: repeat each scenario several times. The default is three. Use one while you fix a single failure, and five or more for a stability run before you publish a big change, because that is what tells you a result holds rather than got lucky once.
- Personas: sweep a scenario across several simulated callers; each scenario runs once per persona.
A batch returns straight away with a batch id and a total run count rather than blocking for a verdict. Open its results to read per-scenario outcomes once it finishes.
Cancel a runaway batch if you launched too large a sweep: queued runs stop, and calls already in flight finish.
A batch expands to scenarios times runs-per-scenario times personas, and each scenario run is billed. A sweep of 10 scenarios, 5 runs each, across 3 personas is 150 runs. Check the expanded total before launching a large one.
Comparing batches
To find out whether one setup beats another, for example a faster speech-to-text model or a reworked prompt, run them as variants in one batch and compare. The comparison groups the runs by variant and reports, for each:
- the success rate;
- the rubric pass-rate, the granular score that separates variants when the all-or-nothing success rate is flat;
- the per-scenario breakdown and per-metric scores;
- per-stage latency for speech-to-text, the language model, and text-to-speech.
The winner is the variant with the highest rubric pass-rate.
What it costs
On a Vertical AI public plan, text test runs are included, and voice test runs are included up to the plan's monthly voice test allowance (20 on Starter, 100 on Growth, 400 on Scale). Past the allowance, and on any other offer, a run is billed on two legs, whatever the outcome:
- The call itself, charged per minute of audio at the same rate a real call on that agent's model and voice would cost.
- The judge, charged per judge metric the run scores. Each metric carries its own price, so the judge leg is the sum of the metrics active on that scenario. The deterministic structural checks cost nothing.
There is no flat per-run price, because a scenario scoring three judge metrics costs less than one scoring twenty.
The cost is per run, so it multiplies by the expanded total, which is why the run dialog shows an estimated total before you launch. Treat it as an estimate: actual billing follows the metered minutes and metrics.
On a Vertical AI plan, text tests are included and voice test calls count against the plan's monthly allowance. See Billing.
Where to go next
- Testing an agent until it's ready: the scenarios every agent needs, and the loop from red to a stable green.
- Writing evals: scenarios, rubrics, personas, line conditions, metrics.
- Agents: the build-and-publish loop evals gate.
- Writing the prompt: often the right fix when a rubric fails.
- Tools: when a failure is the agent reaching for the wrong tool, or none.
- Calls and analytics: where metrics land on real calls.
- Billing: what runs cost.
- Agent tests in CI: run the tests on every pull request, with the same verdicts, and fail the job when too few pass. The vai CLI runs them from a terminal.
Analytics dashboards
The Analytics page charts trends across many calls, with chart fields, starter templates and the source filter.
Writing evals
The parts you author: scenario shapes, rubric questions, personas, degraded line conditions, and the metrics that carry a rubric question onto every real call.