Connect

The vai CLI

Log in from the browser, keep an agent's config in a repo, push edits to the draft, rerun its tests on every save, and gate CI on the same verdicts the app shows.

vai is the Vertical AI command line. It is a client of the same MCP server your AI assistant connects to, so every tool the server has is reachable from the terminal, and the everyday jobs have their own commands.

npm i -g @verticalai.com.au/vai

It needs Node 20 or later. To run it once without installing, use npx @verticalai.com.au/vai.

Log in

vai login

vai login opens the browser on an approval page. It starts at read and write, which is what vai push and vai test need. Choose read only for a token that only reads, or allow Delete (a push that removes a tool, test or setting you deleted from the files) or Manage the workspace only if you need them. Pick one workspace or all of them, then press Authorize. The page hands the token back to vai on your machine directly, so it never appears in a link or your browser history. The token lasts 90 days and is saved to ~/.vai/config.json, which only you can read on systems that support file permissions.

InsteadUse
CI, or a machine with no browservai login --token vai_pat_…, or set VAI_TOKEN and skip login
Paste a token you made under Settings > API tokensvai login --paste
vai whoami          # who the token belongs to, its scopes and workspaces
vai doctor --json   # is the server reachable, is the token valid, what can it reach
vai logout          # revoke the saved token and delete it from this machine

VAI_TOKEN wins over the saved token. vai logout never revokes a token set in VAI_TOKEN, and says so when one is still set.

Read agents

vai agents list --workspace acme
vai agents get acme/support
vai agents get acme/support --section prompt --section tools --json

agents list shows the agents you can reach, each as published or draft only; --archived includes archived ones. agents get prints a summary of the latest version, or the whole snapshot with --json. --section picks parts of it (prompt, tools, variables, voice, behaviour, privacy, settings, model, workflow), and --version published or a version number reads another version.

Keep an agent's config in a repo

vai pull acme/support
vai diff acme/support
vai push acme/support

vai pull writes the agent's config into ./support, or the directory you name with --out. It reads the latest version, or the one you name with --version. A file lists only what differs from a new agent, so an empty field or a setting at its default is not in it.

FileHolds
prompt.mdThe instructions
greetings.jsonThe inbound, returning-caller and outbound greetings
agent.jsonName, description, channels (phone, website chat), the brief (call reasons and never-rules), the reply, fallback, speech-to-text and grading models, the call engine, behaviour and ambience, turn-taking and caller memory
voice.jsonThe voice and the fallback voice, how it speaks, keyterms, pronunciations and vocabulary
calls.jsonCall handling (voicemail, call screening, phone menu and abuse hang-ups, keypad input, silence, ring and length limits, the passphrase gate, what a blocked caller gets), the overflow number, the closed message, answering hours, whether it is closed early, and caller routing
safety.jsonGuardrails, and privacy and disclosure: recording and AI disclosure, recording, redaction, retention, telemetry, post-call analysis, the reprompt on silence, the timezone, and whether a phone-number match is enough for low-risk details
tools.jsonEvery tool in the agent's order, with its endpoint, headers, parameters, mock and follow-up
variables.jsonVariables. A secret is listed by name only; a variable filled per call lists its default
workflow.jsonPre-call, hang-up and post-call functions
analysis.jsonPost-call summary instructions and fields
knowledge.json, knowledge/Knowledge sources; each text source is its own file
tests/One file per test scenario
vai.jsonWhich agent and version the files match

What pull leaves out

Not pulledWhy
Secret valuesThey never leave the server. Name an environment variable for push to set one from (below)
Agent response guardrailsvai has no file for them yet, so push leaves them as they are. Set them in the app, or with vai run set_agent_response_guardrails
The content of uploaded knowledge filesknowledge.json lists each as {"name": …, "file": true}. No tool uploads a file, so upload it in the app
Connected apps, MCP servers and website endpointsA connection holds a sign-in or a key that cannot be written to a file. The tools they add are in tools.json
The look of the website chat, the share page and the details form, chat handoff, and share-page limitsThey change how those pages look and limit visitors, not how the agent answers. Set them in the Channels tab
Blocked callersThe list grows from live calls. Caller routing, which you write yourself, is in calls.json
Call checks, test callers, personas, config sets, alerts and dashboardsThey belong to the workspace and are shared between agents
The knowledge-search tool the first knowledge source addsThe server adds it and keeps it while the agent has knowledge. Once you edit it, it is an ordinary tool in tools.json
Scenarios the brief made from a never-ruleThe brief in agent.json owns them; change the rule and they follow
A sync schedule on a knowledge source, and a scenario's audio-degradation settingsThey are set on the source or scenario by its id. The source and the scenario themselves are pulled

Diff and push

vai diff prints what differs between the files and the agent, and exits 1 when anything does, so a CI job can catch an agent edited in the app but not in the repo. JSON files are compared by content, so moving a key is not a difference. A file with a field vai does not know stops vai diff exactly as it stops vai push: the same message, naming the field and the fields the file takes, and exit 2.

vai push makes the agent's draft match the files. It prints the diff and the plan, one line per change, and writes only what differs.

  • What you delete is cleared. A field you remove from a file goes back to its default, a setting removed from an object (a key in callHandling, say) goes back to its default, and a tool, variable, function, field, knowledge source or scenario you remove is deleted from the draft. The name, the instructions, the reply model, the voice and the channels cannot be removed, and the closed message can be replaced but not emptied.
  • A failed push changes nothing. Every step is checked before the first write, and deletions run last. If a step still fails, the steps before it are undone, so the draft is left as it was, and the error names the step that failed.
  • Push opens a draft when the latest version is published. It never publishes. Publish in the app when the tests pass.

Push cannot create a scripted scenario (one with caller lines and no instructions), because no tool takes a script; it says so, and you build one in the app. A scenario's caller lines are pulled so you can read them.

Some changes ask first, because they apply to every version and reach live calls the moment they are saved: keyterms, pronunciations, vocabulary, the fallback voice, the call engine, call handling, the overflow number, the closed message, answering hours, closing early, caller routing, privacy and disclosure, behaviour settings and the channels. A deletion cannot be undone. Push marks those lines with !. In a terminal, push asks before it applies any plan. Without one, it stops on a ! line unless you pass --yes. --dry-run prints the plan and changes nothing, and leaves nothing in the audit log.

The audit log records a change you confirmed with --yes or --max-spend as confirmed by the CLI's flag (confirmed_by: cli_flag), and one you confirmed by typing y at push's prompt as confirmed_by: cli_prompt, not by a model.

Undo

vai undo acme/support
vai undo acme/support --steps 3

vai undo puts the agent's draft back the way it was before your last change, and prints what it changed back and how many more changes you can undo. Before each change you make, from vai push, the app's Builder or assistant, the MCP or the REST API, the draft is saved for you; your newest ten are kept for each agent. --steps goes back up to ten changes at once. Each undo uses up the saves it goes back past, so running it again goes back one change further. It never publishes, and it refuses, changing nothing, when there is nothing of yours to undo or the draft was published since.

When putting the draft back would change live calls straight away or delete something, undo asks first, as push does: in a terminal it asks, and without one it stops with exit code 2 unless you pass --yes. Your pulled files still hold the undone change, so run vai pull afterwards to bring them up to date, or vai diff a copy pulled before the change to see the draft matches it again.

Secrets

Secret values never go into a file. variables.json lists a secret as {"name": "api_key", "secret": true}. To set one from CI, name an environment variable that holds it:

{ "name": "api_key", "secret": true, "from": "env:ORDERS_API_KEY" }

Push sets the secret from that variable only when the agent has no value for it yet, so it never replaces a key a live call is using.

Work on an agent with a fast loop

vai dev acme/support --only "Where is my order" --bail

vai dev watches the pulled files. When it starts, and then each time you save, it pushes the files to the draft and reruns the scenarios you named with --only or --tag, once each, and prints a pass or fail for each. Name none and it only pushes. Scenarios that failed last time run first, as their own batch, then the rest. With --bail, the runs not yet started are cancelled as soon as one fails, and while last time's failures still fail the rest are skipped. A change that reaches live calls at once, or deletes something, is held back and named, unless you started it with --yes. A batch the server asks about starts only within --max-spend, or whatever it costs with --yes. It waits 300 milliseconds after the last save, or as long as you set with --debounce. Press Ctrl+C to stop.

vai talk acme/support --watch --message "Where is order 42?"

vai talk --watch does the same for a conversation. It sends your message, then on each save pushes the change and sends your last message again in a new conversation, so you see the reply change as you edit. In a terminal, each message you type becomes the one it resends. If a push is held back, nothing is resent.

Without --watch, vai talk acme/support is a text conversation with the draft in your terminal; Ctrl+D or /exit leaves. --message sends one message, prints the reply and exits. --version published talks to the live version, and --conversation <id> carries on an earlier conversation.

Run the tests

vai test acme/support --fail-under 0.9

vai test starts the agent's tests and prints each run's verdict the moment that run finishes, with the first failed check of a failing run. It checks a second after starting and backs off to every 10 seconds. Once every run has settled it prints a verdict per scenario: passed, failed, flaky or inconclusive. A run passes or fails by the same verdict the app shows: the scenario's own criteria, a critical call check that failed the run, and any verdict an operator set by hand. vai test exits 0 when the pass rate meets --fail-under (default 1, every graded run passing) and 1 when it does not. Runs the test harness dropped are reported but not counted, and a batch that was cut off always fails.

OptionMeaning
--only <scenario>Run one scenario by name or id. Repeat it for more
--tag <tag>, --folder <folder>Run the scenarios with a tag, or in a folder
--runs <n>Runs per scenario. A selection runs once; the whole suite runs three times
--version <n>Test a specific version instead of the latest
--bailCancel the runs not yet started as soon as one fails
--max-spend <A$>Start a batch the server asks about when its estimate is at most this. Without it or --yes, such a batch starts nothing and exits 2
--yesStart it whatever the estimate
--timeout <minutes>Stop waiting after this long (default 60). The batch carries on, and vai prints how to read it later

For each failed run, vai test prints the vai calls get command that opens its call.

Read calls

vai calls list acme/support --since 6h
vai calls get acme/support <sessionId>

calls list shows the 20 most recent calls with their outcome and call checks. --limit fetches up to 50, --search searches the transcripts, and --include-tests adds the test calls made in the app. calls get prints the call as JSON: the transcript with every tool call, per-turn latency, call checks and links to its recordings. Pass --section full for the whole record.

Call any tool

vai tools                      # every tool and what it does
vai tools set_greeting_message # one tool's arguments
vai run set_greeting_message --arg agent=acme/support --arg greetingMessage="Hi, Acme here."

--arg key=value values are read as JSON when they can be (limit=5, flag=true), and as text otherwise. --json '{...}' passes the whole argument object. A tool that deletes something, publishes, changes what live callers get straight away, or spends money asks for confirmation first. vai shows the reason and any estimate; re-run with --yes to confirm. A change to a published version is refused with the code not_draft; call create_draft first, or use vai push, which opens one.

Scripts and CI

vai never prompts or opens a browser when it is not attached to a terminal or when CI is set. In CI, set VAI_TOKEN instead of running vai login, as a masked or secret variable.

Tests cost money, so the server asks before it starts a batch with any voice test or more than 20 runs. Give CI a limit with --max-spend: a batch estimated at or under it starts, and one over it, or one the server could not price, starts nothing and exits 2.

Each recipe below keeps the pulled files in agent/ in the repo. It pushes them to the draft, checks the draft now matches, and runs the tests, keeping their results as JSON Lines.

.github/workflows/agent-tests.yml
name: Agent tests
on: pull_request
jobs:
  agent-tests:
    runs-on: ubuntu-latest
    env:
      VAI_TOKEN: ${{ secrets.VAI_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - run: npx --yes @verticalai.com.au/vai@1 push acme/support agent --yes
      - run: npx --yes @verticalai.com.au/vai@1 diff acme/support agent
      - run: npx --yes @verticalai.com.au/vai@1 test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonl
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: vai-results
          path: vai-results.jsonl

Store VAI_TOKEN as a repository secret.

.gitlab-ci.yml
agent-tests:
  image: node:22
  script:
    - npx --yes @verticalai.com.au/vai@1 push acme/support agent --yes
    - npx --yes @verticalai.com.au/vai@1 diff acme/support agent
    - npx --yes @verticalai.com.au/vai@1 test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonl
  artifacts:
    when: always
    paths:
      - vai-results.jsonl

Set VAI_TOKEN under Settings > CI/CD > Variables, masked.

.circleci/config.yml
version: 2.1
jobs:
  agent-tests:
    docker:
      - image: cimg/node:22.11
    steps:
      - checkout
      - run:
          name: Push the files to the draft
          command: npx --yes @verticalai.com.au/vai@1 push acme/support agent --yes
      - run:
          name: Check the draft matches the files
          command: npx --yes @verticalai.com.au/vai@1 diff acme/support agent
      - run:
          name: Run the agent's tests
          command: npx --yes @verticalai.com.au/vai@1 test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonl
      - store_artifacts:
          path: vai-results.jsonl
workflows:
  agent:
    jobs:
      - agent-tests:
          context: vai

Put VAI_TOKEN in a context named vai.

For any other runner, a plain shell script:

ci.sh
set -euo pipefail
vai() { npx --yes @verticalai.com.au/vai@1 "$@"; }
vai push acme/support agent --yes
vai diff acme/support agent
vai test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonl

These recipes are run in the CLI's own tests against a mock server, so the commands on this page are the commands that pass. On GitHub, the agent tests action runs vai test for you and writes each scenario's verdict to the job summary.

  • --json works on every command except vai run, which always prints JSON and takes its arguments with --json. Commands that stream (vai test, vai dev and a piped or watching vai talk) print JSON Lines: one object per line, each with a type. vai test prints a run line as each run finishes (test, runId, verdict and the first failed check as reason) and ends with a result line holding the pass rate and every scenario's verdict.
  • Errors carry the server's own message, on stderr, or as an {"ok": false, "error": ...} object under --json.
Exit codeMeaning
0Success
1The command ran and the answer is no: tests below the threshold, files that differ, or a change the server refused
2The command could not run: a usage mistake, a missing or rejected token, a missing scope, a confirmation that needs --yes or --max-spend, a server it could not reach, or a test batch still running at --timeout

The base URL is --base-url, then VAI_BASE_URL, then the one you logged in to, then https://app.verticalai.com.au.

On this page