Connect

Agent tests in CI

Run an agent's test scenarios on every pull request with the verticalau/vai-agent-tests GitHub Action, and fail the job when too few pass.

The verticalau/vai-agent-tests GitHub Action runs an agent's test scenarios and fails the job when too few pass. It runs vai test, so a run passes or fails by the same verdict the app shows: the scenario's own criteria, then any critical call check that failed the run, then any verdict an operator set by hand.

.github/workflows/agent-tests.yml
name: Agent tests
on: pull_request

jobs:
  agent-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - uses: verticalau/vai-agent-tests@v1
        with:
          vai_token: ${{ secrets.VAI_TOKEN }}
          agent: acme/support
          fail_under: "0.9"
          max_spend: "10"

The token

Make a personal access token under Settings > API tokens with Edit access, pinned to the agent's workspace, and store it as the repository secret VAI_TOKEN. Starting test runs needs Edit. The action hands the token to the CLI in the environment, never on a command line.

Inputs

InputDefaultMeaning
vai_tokenrequiredA personal access token (vai_pat_…)
agentrequiredworkspace-slug/agent-slug
onlyScenario names or ids, one per line or comma separated. Empty runs the whole suite
tagsRun the scenarios carrying any of these tags
folderRun the scenarios in this folder
runsRuns per scenario. Empty runs a selection once and the whole suite three times
fail_under1Fail when the pass rate over graded runs is below this, from 0 to 1
max_spendThe most a batch may cost, in Australian dollars
versionTest this version number instead of the latest
timeout_minutes60Stop waiting after this many minutes
base_urlhttps://app.verticalai.com.auThe platform's address
cli_version1The vai CLI version or range to run

The runner needs Node 20 or later, which actions/setup-node provides.

Spend

Tests cost money, so the platform asks before it starts a batch with any voice test or more than 20 runs. Set max_spend: a batch estimated at or under it starts, and one over it, or one the platform could not price, starts nothing and fails the job with the estimate. With max_spend empty, any batch the platform asks about fails the job before it starts.

Outputs and the result

OutputMeaning
pass_ratePassed runs over graded runs, from 0 to 1. Empty when nothing was graded
passed, gradedThe counts behind it
batch_idThe batch, to read again with vai run get_test_results

The step passes when the pass rate meets fail_under. It fails with exit code 1 when the pass rate is below it, and with exit code 2 when the tests could not run: a bad token, an unknown scenario, a batch over max_spend, or a batch still running at timeout_minutes. Runs the test harness dropped are reported and not counted, and a batch that was cut off fails. The job summary lists every scenario with its verdict.

Other CI systems

The action is a wrapper around the CLI. On GitLab, CircleCI or any other runner, run the same command yourself: the CLI's CI recipes push the agent's files to its draft, check the draft matches, and run the tests.

On this page