Test runs
Test runs

List recent test runs

List this agent's recent test batches, newest first, the way to FIND a batchId you no longer have. Every batch-scoped tool (get_test_results, cancel_test_batch, rescore_batch) takes a batchId, and otherwise the only source is the live return value of run_all_tests, which is gone the moment the message scrolls away or the session ends. Call this first whenever you need to read, cancel or re-score a batch you did not just launch. Each row carries `batchId`, `status`, `total` / `completed` / `failed` / `errored`, `startedAt` / `finishedAt`, and `cronJob`; `has_more` and `next_cursor` page back through older batches. READ THE STATUS: 'running' means it is still executing (poll get_test_results and check `complete` before trusting counts); 'cancelling' means an operator (or a supersede launch) requested cancellation, in-flight runs finish and queued runs will not start, so expect a terminal 'cancelled' within a couple of minutes and do not cancel it again; 'complete' means every requested run produced a verdict; 'truncated' means the batch was CUT OFF, the reaper abandoned runs that never ran, so its pass counts cover only a fraction of the suite and you must never publish on it, re-run instead; 'error' means the runner itself failed; 'cancelled' means an operator stopped it. `cronJob` names the schedule that fired the batch when one did, and null means it was launched by hand or over the API; pass that cron job's id to list_cron_batch_history for that schedule's full history. A pure read, it launches nothing and costs nothing.

GET
/workspaces/{workspace}/agents/{agent}/test-runs

Authorization

bearerAuth
AuthorizationBearer <token>

A personal access token from Settings > API tokens.

In: header

Path Parameters

workspace*string

Workspace slug (GET /me lists yours).

agent*string

Agent slug within the workspace.

Query Parameters

limit?integer

Items per page, 1 to 200. Default 50.

Range1 <= value <= 200
cursor?string

The next_cursor of the previous page.

Response Body

application/json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

curl -X GET "https://example.com/workspaces/string/agents/string/test-runs"
{
  "data": [
    {
      "batchId": "string",
      "status": "string",
      "total": 0,
      "startedAt": "string",
      "property1": null,
      "property2": null
    }
  ],
  "has_more": true,
  "next_cursor": "string"
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}

Get a test run's results GET

Read test (eval) verdicts. Pass the batchId from run_all_tests for that batch, or omit it for the latest GRADED run of every test; pass `test` (a name from list_tests, or an id) to read one test, the shape you want while iterating on one scenario. Lost the batch id? list_batches lists this agent's recent batches. Each row carries the test name, its `runId` (pass a failing one to get_run_transcript to read the exchange behind the verdict: that is the debugging ladder), its `sessionId` (pass it to get_call_bundle), the verdict (passed, failed, scoring, review_required, error, infra_error, running or pending: an operator's override first, then a failed critical metric turning a pass red, then the run's own status), the failed rubrics with their reasoning, and `rubricsNote`, which explains any row whose verdict or empty rubric list needs explaining. A red run with no failed rubric was vetoed by the critical metric that `vetoedByMetrics` names: fix what it measures rather than re-running. `scoring` means the scenario passed but critical metrics are still grading; it counts as in flight. CHECK `complete` BEFORE YOU TRUST THE COUNTS: run rows exist up front and a voice run takes 1-5 minutes, so poll every 30-60 seconds until `complete` is true. Faster polling buys nothing, a batch finishes on its own, and a quiet run is not a stuck one: compare `elapsedSeconds` with `typicalSeconds`, and never cancel a batch for it; `batchStatus: 'cancelling'` is already stopping. `infraFailed` runs and a `truncated` batch tested nothing about the agent: re-run them, and never publish on a truncated batch. A row whose `stale` is a sentence scored an earlier version, and publish accepts only greens earned on the version being shipped (`currentVersion`), so re-run stale tests before publishing. With a batchId, gate on `perTest` (passed, failed, flaky or inconclusive across repeats), not the flat counts. A `fixLadder` on a failing row means prompt-only edits have failed across two or more versions: stop editing the prompt and change the tool's description (set_tool_description), its output (set_tool_response_template) or its inputs (set_tool_params), as `fixLadder.nextRung` says. For the success-rate trend across many runs, use get_runs_overview.

Run the test suite POST

Run the agent's test suite as one server-side batch, every eval by default, or a configured sweep: scope with evalIds, repeat each scenario numRuns times (1-50), sweep caller personalities with personaIds (each scenario runs once per persona), cap parallelism with concurrentLimit, and pin a version to evaluate. The batch expands to evalIds × numRuns × personas, so a big sweep can burn significant credits, quote the expanded total to the operator before launching one. Returns immediately with the batch id + total; there is no inline verdict. Call get_test_results with the returned batchId to read per-test outcomes, and cancel_test_batch to stop a runaway batch (queued runs stop; in-flight calls finish). BATCHES SHARE ONE QUEUE at concurrency 3, so a new batch waits behind every outstanding run of this agent's earlier ones: the response reports `runsAhead` (and `batchesAhead`) so you can see the wait before you start polling. When you are re-running after an edit and the earlier batch is testing the version you just replaced, pass supersede: true, or the new runs sit behind work you have already invalidated. Supersede REQUESTS cancellation rather than stopping the earlier batches on the spot: their in-flight runs finish, their queued runs never start, and the queue slots free up as those runs drain, so `runsAhead` can still be non-zero for a minute or two after a supersede, and the superseded batches (returned as `supersededBatches`) read as 'cancelling' until their last run lands.