The vai CLI
Log in from the browser, keep an agent's config in a repo, push edits to the draft, rerun its tests on every save, and gate CI on the same verdicts the app shows.
vai is the Vertical AI command line. It is a client of the same MCP server
your AI assistant connects to, so every tool the server has is reachable from
the terminal, and the everyday jobs have their own commands.
npm i -g @verticalai.com.au/vaiIt needs Node 20 or later. To run it once without installing, use
npx @verticalai.com.au/vai.
Log in
vai loginvai login opens the browser on an approval page. It starts at read and write,
which is what vai push and vai test need. Choose read only for a token that
only reads, or allow Delete (a push that removes a tool, test or setting you
deleted from the files) or Manage the workspace only if you need them. Pick
one workspace or all of them, then press Authorize. The page
hands the token back to vai on your machine directly, so it never appears in
a link or your browser history. The token lasts 90 days and is saved to
~/.vai/config.json, which only you can read on systems that support file
permissions.
| Instead | Use |
|---|---|
| CI, or a machine with no browser | vai login --token vai_pat_…, or set VAI_TOKEN and skip login |
| Paste a token you made under Settings > API tokens | vai login --paste |
vai whoami # who the token belongs to, its scopes and workspaces
vai doctor --json # is the server reachable, is the token valid, what can it reach
vai logout # revoke the saved token and delete it from this machineVAI_TOKEN wins over the saved token. vai logout never revokes a token set
in VAI_TOKEN, and says so when one is still set.
Read agents
vai agents list --workspace acme
vai agents get acme/support
vai agents get acme/support --section prompt --section tools --jsonagents list shows the agents you can reach, each as published or draft only;
--archived includes archived ones. agents get prints a summary of the
latest version, or the whole snapshot with --json. --section picks parts of
it (prompt, tools, variables, voice, behaviour, privacy,
settings, model, workflow), and --version published or a version
number reads another version.
Keep an agent's config in a repo
vai pull acme/support
vai diff acme/support
vai push acme/supportvai pull writes the agent's config into ./support, or the directory you
name with --out. It reads the latest version, or the one you name with
--version. A file lists only what differs from a new agent, so an empty
field or a setting at its default is not in it.
| File | Holds |
|---|---|
prompt.md | The instructions |
greetings.json | The inbound, returning-caller and outbound greetings |
agent.json | Name, description, channels (phone, website chat), the brief (call reasons and never-rules), the reply, fallback, speech-to-text and grading models, the call engine, behaviour and ambience, turn-taking and caller memory |
voice.json | The voice and the fallback voice, how it speaks, keyterms, pronunciations and vocabulary |
calls.json | Call handling (voicemail, call screening, phone menu and abuse hang-ups, keypad input, silence, ring and length limits, the passphrase gate, what a blocked caller gets), the overflow number, the closed message, answering hours, whether it is closed early, and caller routing |
safety.json | Guardrails, and privacy and disclosure: recording and AI disclosure, recording, redaction, retention, telemetry, post-call analysis, the reprompt on silence, the timezone, and whether a phone-number match is enough for low-risk details |
tools.json | Every tool in the agent's order, with its endpoint, headers, parameters, mock and follow-up |
variables.json | Variables. A secret is listed by name only; a variable filled per call lists its default |
workflow.json | Pre-call, hang-up and post-call functions |
analysis.json | Post-call summary instructions and fields |
knowledge.json, knowledge/ | Knowledge sources; each text source is its own file |
tests/ | One file per test scenario |
vai.json | Which agent and version the files match |
What pull leaves out
| Not pulled | Why |
|---|---|
| Secret values | They never leave the server. Name an environment variable for push to set one from (below) |
| Agent response guardrails | vai has no file for them yet, so push leaves them as they are. Set them in the app, or with vai run set_agent_response_guardrails |
| The content of uploaded knowledge files | knowledge.json lists each as {"name": …, "file": true}. No tool uploads a file, so upload it in the app |
| Connected apps, MCP servers and website endpoints | A connection holds a sign-in or a key that cannot be written to a file. The tools they add are in tools.json |
| The look of the website chat, the share page and the details form, chat handoff, and share-page limits | They change how those pages look and limit visitors, not how the agent answers. Set them in the Channels tab |
| Blocked callers | The list grows from live calls. Caller routing, which you write yourself, is in calls.json |
| Call checks, test callers, personas, config sets, alerts and dashboards | They belong to the workspace and are shared between agents |
| The knowledge-search tool the first knowledge source adds | The server adds it and keeps it while the agent has knowledge. Once you edit it, it is an ordinary tool in tools.json |
| Scenarios the brief made from a never-rule | The brief in agent.json owns them; change the rule and they follow |
| A sync schedule on a knowledge source, and a scenario's audio-degradation settings | They are set on the source or scenario by its id. The source and the scenario themselves are pulled |
Diff and push
vai diff prints what differs between the files and the agent, and exits 1
when anything does, so a CI job can catch an agent edited in the app but not in
the repo. JSON files are compared by content, so moving a key is not a
difference. A file with a field vai does not know stops vai diff exactly as
it stops vai push: the same message, naming the field and the fields the file
takes, and exit 2.
vai push makes the agent's draft match the files. It prints the diff and the
plan, one line per change, and writes only what differs.
- What you delete is cleared. A field you remove from a file goes back to
its default, a setting removed from an object (a key in
callHandling, say) goes back to its default, and a tool, variable, function, field, knowledge source or scenario you remove is deleted from the draft. The name, the instructions, the reply model, the voice and the channels cannot be removed, and the closed message can be replaced but not emptied. - A failed push changes nothing. Every step is checked before the first write, and deletions run last. If a step still fails, the steps before it are undone, so the draft is left as it was, and the error names the step that failed.
- Push opens a draft when the latest version is published. It never publishes. Publish in the app when the tests pass.
Push cannot create a scripted scenario (one with caller lines and no instructions), because no tool takes a script; it says so, and you build one in the app. A scenario's caller lines are pulled so you can read them.
Some changes ask first, because they apply to every version and reach live
calls the moment they are saved: keyterms, pronunciations, vocabulary, the
fallback voice, the call engine, call handling, the overflow number, the closed
message, answering hours, closing early, caller routing, privacy and disclosure,
behaviour settings and the channels. A deletion cannot be undone. Push marks
those lines with !. In a terminal, push asks before it applies any plan.
Without one, it stops on a ! line unless you pass --yes. --dry-run prints the plan and changes
nothing, and leaves nothing in the audit log.
The audit log records a change you confirmed with --yes or --max-spend as
confirmed by the CLI's flag (confirmed_by: cli_flag), and one you confirmed by
typing y at push's prompt as confirmed_by: cli_prompt, not by a model.
Undo
vai undo acme/support
vai undo acme/support --steps 3vai undo puts the agent's draft back the way it was before your last change,
and prints what it changed back and how many more changes you can undo. Before
each change you make, from vai push, the app's Builder or assistant, the MCP
or the REST API, the draft is saved for you; your newest ten are kept for each
agent. --steps goes back up to ten changes at once. Each undo uses up the
saves it goes back past, so running it again goes back one change further.
It never publishes, and it refuses, changing nothing, when there is nothing of
yours to undo or the draft was published since.
When putting the draft back would change live calls straight away or delete
something, undo asks first, as push does: in a terminal it asks, and without
one it stops with exit code 2 unless you pass --yes. Your pulled files still
hold the undone change, so run vai pull afterwards to bring them up to date,
or vai diff a copy pulled before the change to see the draft matches it
again.
Secrets
Secret values never go into a file. variables.json lists a secret as
{"name": "api_key", "secret": true}. To set one from CI, name an environment
variable that holds it:
{ "name": "api_key", "secret": true, "from": "env:ORDERS_API_KEY" }Push sets the secret from that variable only when the agent has no value for it yet, so it never replaces a key a live call is using.
Work on an agent with a fast loop
vai dev acme/support --only "Where is my order" --bailvai dev watches the pulled files. When it starts, and then each time you
save, it pushes the files to the draft and reruns the scenarios you named with
--only or --tag, once each, and prints a pass or fail for each. Name none
and it only pushes.
Scenarios that failed last time run first, as their own batch, then the rest.
With --bail, the runs not yet started are cancelled as soon as one fails,
and while last time's failures still fail the rest are skipped. A change that
reaches live calls at once, or deletes something, is held back and named,
unless you started it with --yes. A batch the server asks about starts only
within --max-spend, or whatever it costs with --yes. It waits 300 milliseconds after the last save, or as
long as you set with --debounce. Press Ctrl+C to stop.
vai talk acme/support --watch --message "Where is order 42?"vai talk --watch does the same for a conversation. It sends your message,
then on each save pushes the change and sends your last message again in a new
conversation, so you see the reply change as you edit. In a terminal, each
message you type becomes the one it resends. If a push is held back, nothing is
resent.
Without --watch, vai talk acme/support is a text conversation with the
draft in your terminal; Ctrl+D or /exit leaves. --message sends one
message, prints the reply and exits. --version published talks to the live
version, and --conversation <id> carries on an earlier conversation.
Run the tests
vai test acme/support --fail-under 0.9vai test starts the agent's tests and prints each run's verdict the moment
that run finishes, with the first failed check of a failing run. It checks a
second after starting and backs off to every 10 seconds. Once every run has
settled it prints a verdict per scenario: passed, failed, flaky or
inconclusive. A run passes or fails by the same verdict the app shows: the scenario's own criteria, a
critical call check that failed the run, and any verdict an operator set by
hand. vai test exits 0 when the pass rate meets --fail-under (default
1, every graded run passing) and 1 when it does not. Runs the test harness
dropped are reported but not counted, and a batch that was cut off always
fails.
| Option | Meaning |
|---|---|
--only <scenario> | Run one scenario by name or id. Repeat it for more |
--tag <tag>, --folder <folder> | Run the scenarios with a tag, or in a folder |
--runs <n> | Runs per scenario. A selection runs once; the whole suite runs three times |
--version <n> | Test a specific version instead of the latest |
--bail | Cancel the runs not yet started as soon as one fails |
--max-spend <A$> | Start a batch the server asks about when its estimate is at most this. Without it or --yes, such a batch starts nothing and exits 2 |
--yes | Start it whatever the estimate |
--timeout <minutes> | Stop waiting after this long (default 60). The batch carries on, and vai prints how to read it later |
For each failed run, vai test prints the vai calls get command that opens
its call.
Read calls
vai calls list acme/support --since 6h
vai calls get acme/support <sessionId>calls list shows the 20 most recent calls with their outcome and call
checks. --limit fetches up to 50, --search searches the transcripts, and
--include-tests adds the test calls made in the app. calls get prints the call as JSON:
the transcript with every tool call, per-turn latency, call checks and links
to its recordings. Pass --section full for the whole record.
Call any tool
vai tools # every tool and what it does
vai tools set_greeting_message # one tool's arguments
vai run set_greeting_message --arg agent=acme/support --arg greetingMessage="Hi, Acme here."--arg key=value values are read as JSON when they can be (limit=5,
flag=true), and as text otherwise. --json '{...}' passes the whole argument
object. A tool that deletes something, publishes, changes what live callers get
straight away, or spends money asks for confirmation first. vai shows the
reason and any estimate; re-run with --yes to confirm. A change to a
published version is refused with the code not_draft; call create_draft
first, or use vai push, which opens one.
Scripts and CI
vai never prompts or opens a browser when it is not attached to a terminal or
when CI is set. In CI, set VAI_TOKEN instead of running vai login, as a
masked or secret variable.
Tests cost money, so the server asks before it starts a batch with any voice
test or more than 20 runs. Give CI a limit with --max-spend: a batch estimated at
or under it starts, and one over it, or one the server could not price, starts
nothing and exits 2.
Each recipe below keeps the pulled files in agent/ in the repo. It pushes
them to the draft, checks the draft now matches, and runs the tests, keeping
their results as JSON Lines.
name: Agent tests
on: pull_request
jobs:
agent-tests:
runs-on: ubuntu-latest
env:
VAI_TOKEN: ${{ secrets.VAI_TOKEN }}
steps:
- uses: actions/checkout@v4
- run: npx --yes @verticalai.com.au/vai@1 push acme/support agent --yes
- run: npx --yes @verticalai.com.au/vai@1 diff acme/support agent
- run: npx --yes @verticalai.com.au/vai@1 test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonl
- uses: actions/upload-artifact@v4
if: always()
with:
name: vai-results
path: vai-results.jsonlStore VAI_TOKEN as a repository secret.
agent-tests:
image: node:22
script:
- npx --yes @verticalai.com.au/vai@1 push acme/support agent --yes
- npx --yes @verticalai.com.au/vai@1 diff acme/support agent
- npx --yes @verticalai.com.au/vai@1 test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonl
artifacts:
when: always
paths:
- vai-results.jsonlSet VAI_TOKEN under Settings > CI/CD > Variables, masked.
version: 2.1
jobs:
agent-tests:
docker:
- image: cimg/node:22.11
steps:
- checkout
- run:
name: Push the files to the draft
command: npx --yes @verticalai.com.au/vai@1 push acme/support agent --yes
- run:
name: Check the draft matches the files
command: npx --yes @verticalai.com.au/vai@1 diff acme/support agent
- run:
name: Run the agent's tests
command: npx --yes @verticalai.com.au/vai@1 test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonl
- store_artifacts:
path: vai-results.jsonl
workflows:
agent:
jobs:
- agent-tests:
context: vaiPut VAI_TOKEN in a context named vai.
For any other runner, a plain shell script:
set -euo pipefail
vai() { npx --yes @verticalai.com.au/vai@1 "$@"; }
vai push acme/support agent --yes
vai diff acme/support agent
vai test acme/support --fail-under 0.9 --max-spend 10 --json > vai-results.jsonlThese recipes are run in the CLI's own tests against a mock server, so the
commands on this page are the commands that pass. On GitHub, the
agent tests action runs vai test for you and
writes each scenario's verdict to the job summary.
--jsonworks on every command exceptvai run, which always prints JSON and takes its arguments with--json. Commands that stream (vai test,vai devand a piped or watchingvai talk) print JSON Lines: one object per line, each with atype.vai testprints arunline as each run finishes (test,runId,verdictand the first failed check asreason) and ends with aresultline holding the pass rate and every scenario's verdict.- Errors carry the server's own message, on stderr, or as an
{"ok": false, "error": ...}object under--json.
| Exit code | Meaning |
|---|---|
0 | Success |
1 | The command ran and the answer is no: tests below the threshold, files that differ, or a change the server refused |
2 | The command could not run: a usage mistake, a missing or rejected token, a missing scope, a confirmation that needs --yes or --max-spend, a server it could not reach, or a test batch still running at --timeout |
The base URL is --base-url, then VAI_BASE_URL, then the one you logged in
to, then https://app.verticalai.com.au.
Related
Python SDK
Call the REST API from Python with a typed function for every route and operation, sync or async, generated from the API's OpenAPI specification.
Agent tests in CI
Run an agent's test scenarios on every pull request with the verticalau/vai-agent-tests GitHub Action, and fail the job when too few pass.