Start an error analysis of a leaderboard row
Picks the failing samples of one model on one suite version (100 by default, sample_size up to 200), sends them (with their audio on speech boards) to the analyzer in two stages and stores a report of error categories, weak tasks, judge and benchmark issues and actionable items. The job runs in the background: poll the returned id until status is done or failed. A finished analysis of the same samples, metric and prompt version is returned as is unless force is true; so is an analysis that is still running. Owner org only; shared orgs read reports but cannot start one. Returns 400 for embedding and coding-agent boards, for a row with no failures, and when the analyzer is not configured on the cluster.
Authorizations
Send Authorization: Bearer , using an MKA1 API key for automation or an OAuth API access token for a signed-in application. Use a dynamic security callback to refresh OAuth access tokens. Browser session-only operations still require their declared cookie.
Headers
Optional external end-user identifier forwarded by the API gateway.
Path Parameters
Body
The leaderboard row to analyse: a model id exactly as it appears on the board.
1 - 255Suite version whose runs are analysed. Defaults to the suite's active version.
1 <= x <= 9007199254740991Headline metric that defines a failure (e.g. wer, scribe_v2_wer, score). Defaults to the board's default metric.
1 - 128Overrides the failure threshold: a sample fails when a lower-is-better metric is at or above it, or a higher-is-better metric is below it. Defaults: 0.1 and 0.5.
How many failures to analyse. Default 100.
10 <= x <= 200Start a new analysis even when a finished one exists for the same samples, metric and prompt version.
Analyzer model to use, one of GET /evals/error-analysis/analyzers. Defaults to the cluster's default analyzer.
1 - 128Response
OK
-9007199254740991 <= x <= 9007199254740991tts, asr, text queued, running, done, failed Counts behind the sample: failures in total, per task and per tier, the threshold, the board context.
-9007199254740991 <= x <= 9007199254740991The user id that started it; null for a reader in a shared org, which does not see the owner org's users.
-9007199254740991 <= x <= 9007199254740991-9007199254740991 <= x <= 9007199254740991-9007199254740991 <= x <= 9007199254740991The report rendered as Markdown; null until the analysis is done.