Agent tests in CI
Run an agent's test scenarios on every pull request with the verticalau/vai-agent-tests GitHub Action, and fail the job when too few pass.
The verticalau/vai-agent-tests GitHub Action runs an agent's
test scenarios and fails the job when too few pass. It
runs vai test, so a run passes or fails
by the same verdict the app shows: the scenario's own criteria, then any
critical call check that failed the run, then any verdict an operator set by
hand.
name: Agent tests
on: pull_request
jobs:
agent-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/setup-node@v4
with:
node-version: 20
- uses: verticalau/vai-agent-tests@v1
with:
vai_token: ${{ secrets.VAI_TOKEN }}
agent: acme/support
fail_under: "0.9"
max_spend: "10"The token
Make a personal access token under Settings > API tokens with Edit
access, pinned to the agent's workspace, and store it as the repository secret
VAI_TOKEN. Starting test runs needs Edit. The action hands the token to the
CLI in the environment, never on a command line.
Inputs
| Input | Default | Meaning |
|---|---|---|
vai_token | required | A personal access token (vai_pat_…) |
agent | required | workspace-slug/agent-slug |
only | Scenario names or ids, one per line or comma separated. Empty runs the whole suite | |
tags | Run the scenarios carrying any of these tags | |
folder | Run the scenarios in this folder | |
runs | Runs per scenario. Empty runs a selection once and the whole suite three times | |
fail_under | 1 | Fail when the pass rate over graded runs is below this, from 0 to 1 |
max_spend | The most a batch may cost, in Australian dollars | |
version | Test this version number instead of the latest | |
timeout_minutes | 60 | Stop waiting after this many minutes |
base_url | https://app.verticalai.com.au | The platform's address |
cli_version | 1 | The vai CLI version or range to run |
The runner needs Node 20 or later, which actions/setup-node provides.
Spend
Tests cost money, so the platform asks before it starts a batch with any voice
test or more than 20 runs. Set max_spend: a batch estimated at or under it
starts, and one over it, or one the platform could not price, starts nothing
and fails the job with the estimate. With max_spend empty, any batch the
platform asks about fails the job before it starts.
Outputs and the result
| Output | Meaning |
|---|---|
pass_rate | Passed runs over graded runs, from 0 to 1. Empty when nothing was graded |
passed, graded | The counts behind it |
batch_id | The batch, to read again with vai run get_test_results |
The step passes when the pass rate meets fail_under. It fails with exit code
1 when the pass rate is below it, and with exit code 2 when the tests could
not run: a bad token, an unknown scenario, a batch over max_spend, or a batch
still running at timeout_minutes. Runs the
test harness dropped are reported and not counted, and a batch that was cut off
fails. The job summary lists every scenario with its verdict.
Other CI systems
The action is a wrapper around the CLI. On GitLab, CircleCI or any other runner, run the same command yourself: the CLI's CI recipes push the agent's files to its draft, check the draft matches, and run the tests.
Related
The vai CLI
Log in from the browser, keep an agent's config in a repo, push edits to the draft, rerun its tests on every save, and gate CI on the same verdicts the app shows.
Campaigns API
Create, fill, launch and run outbound campaigns from your own servers over the REST API, authenticated with a personal access token.