Test runs
Test runs

Run the test suite

Run the agent's test suite as one server-side batch, every eval by default, or a configured sweep: scope with evalIds, repeat each scenario numRuns times (1-50), sweep caller personalities with personaIds (each scenario runs once per persona), cap parallelism with concurrentLimit, and pin a version to evaluate. The batch expands to evalIds × numRuns × personas, so a big sweep can burn significant credits, quote the expanded total to the operator before launching one. Returns immediately with the batch id + total; there is no inline verdict. Call get_test_results with the returned batchId to read per-test outcomes, and cancel_test_batch to stop a runaway batch (queued runs stop; in-flight calls finish). BATCHES SHARE ONE QUEUE at concurrency 3, so a new batch waits behind every outstanding run of this agent's earlier ones: the response reports `runsAhead` (and `batchesAhead`) so you can see the wait before you start polling. When you are re-running after an edit and the earlier batch is testing the version you just replaced, pass supersede: true, or the new runs sit behind work you have already invalidated. Supersede REQUESTS cancellation rather than stopping the earlier batches on the spot: their in-flight runs finish, their queued runs never start, and the queue slots free up as those runs drain, so `runsAhead` can still be non-zero for a minute or two after a supersede, and the superseded batches (returned as `supersededBatches`) read as 'cancelling' until their last run lands.

POST
/workspaces/{workspace}/agents/{agent}/test-runs

Authorization

bearerAuth
AuthorizationBearer <token>

A personal access token from Settings > API tokens.

In: header

Path Parameters

workspace*string

Workspace slug (GET /me lists yours).

agent*string

Agent slug within the workspace.

Header Parameters

Idempotency-Key?string

A key you choose, such as a UUID, that makes this POST safe to retry. Within 24 hours the same key and request answers with the first successful answer and runs nothing, the same key with a different request answers 422 idempotency_key_reused, and a key whose request is still running answers 409 idempotency_key_in_use. A failed request keeps nothing, so the same key runs it again.

Length1 <= length <= 255

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

Response Body

application/json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

curl -X POST "https://example.com/workspaces/string/agents/string/test-runs" \  -H "Content-Type: application/json" \  -d '{}'
{
  "batchId": "string",
  "total": 0,
  "property1": null,
  "property2": null
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}
{
  "type": "http://example.com",
  "title": "string",
  "status": 0,
  "detail": "string",
  "code": "string",
  "errors": [
    {
      "path": "string",
      "message": "string"
    }
  ],
  "estimate": {},
  "retryAfterSeconds": 0
}

List recent test runs GET

List this agent's recent test batches, newest first, the way to FIND a batchId you no longer have. Every batch-scoped tool (get_test_results, cancel_test_batch, rescore_batch) takes a batchId, and otherwise the only source is the live return value of run_all_tests, which is gone the moment the message scrolls away or the session ends. Call this first whenever you need to read, cancel or re-score a batch you did not just launch. Each row carries `batchId`, `status`, `total` / `completed` / `failed` / `errored`, `startedAt` / `finishedAt`, and `cronJob`; `has_more` and `next_cursor` page back through older batches. READ THE STATUS: 'running' means it is still executing (poll get_test_results and check `complete` before trusting counts); 'cancelling' means an operator (or a supersede launch) requested cancellation, in-flight runs finish and queued runs will not start, so expect a terminal 'cancelled' within a couple of minutes and do not cancel it again; 'complete' means every requested run produced a verdict; 'truncated' means the batch was CUT OFF, the reaper abandoned runs that never ran, so its pass counts cover only a fraction of the suite and you must never publish on it, re-run instead; 'error' means the runner itself failed; 'cancelled' means an operator stopped it. `cronJob` names the schedule that fired the batch when one did, and null means it was launched by hand or over the API; pass that cron job's id to list_cron_batch_history for that schedule's full history. A pure read, it launches nothing and costs nothing.

Run one test scenario POST

Run a single test against the agent's current published or draft version. Returns `verdict` (passed / failed / scoring / review_required / error), read as Test analytics reads it: an operator override wins, a passed run whose critical metric failed is failed with `failedCritical` naming the metric, and scoring means the criteria passed but critical metrics are still scoring. Also returns the graded rubric count and (on failure) the judge's reasoning for each failed rubric. A voice test, and over MCP a text test still going after 45 seconds, returns a `verdict` of running with a `batchId` instead: read the verdict with get_test_results, and do not run the test again, which would start a second paid run. After a failure, read the reasoning carefully: the judge often pinpoints exactly which line of the system prompt or which tool description is at fault. Cap at 2 retry passes per test, if the second fix doesn't take, mark it as a known-failing test and move on rather than spinning forever.