Results and artifacts

First run an eval and keep its run ID. Results describe the suite version, model, inputs, and graders used for that run.

Inspect samples

List samples when you need per-row debugging. You can filter by task_id, model, or status.

import { SDK } from '@meetkai/mka1';

const sdk = new SDK({ bearerAuth: 'Bearer <mka1-api-key>' });

const result = await sdk.llm.evals.listSamples({ runId: 'eval_run_abc123', limit: 10, taskId: 'repeat_exactly' });
console.log(result);

from meetkai_mka1 import SDK

sdk = SDK(bearer_auth="Bearer <mka1-api-key>")

result = sdk.llm.evals.list_samples(run_id="eval_run_abc123", limit=10, task_id="repeat_exactly")
print(result)

using MeetKai.MKA1;
using MeetKai.MKA1.Types.Requests;

var sdk = new SDK(bearerAuth: "Bearer <mka1-api-key>");

var result = await sdk.Llm.Evals.ListSamplesAsync(new ListEvalSamplesRequest { RunId = "eval_run_abc123", Limit = 10, TaskId = "repeat_exactly" });
Console.WriteLine(result);

mka1 llm evals list-samples \
  --run-id eval_run_abc123 \
  --limit 10 \
  --task-id repeat_exactly

curl 'https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123/samples?limit=10&task_id=repeat_exactly' \
  --header 'Authorization: Bearer <mka1-api-key>'

You can also:

Filter by a numeric score band using score_metric + score_min/score_max.
Skip large rows (like inlined audio) by setting include_dataset_row=false (samples will return dataset_row: null).
Fetch only specific sample indices using sample_index (comma-separated indices).

Each sample includes the source row, rendered prompt, target, stored Responses response_id, raw model output, extracted output, scores, judge details, and error details. It also carries the tokens and time the sample spent, described in Token and time fields.

{
  "object": "eval.sample",
  "task_id": "repeat_exactly",
  "model": "openai:gpt-4.1-mini",
  "status": "completed",
  "dataset_row": {
    "question": "Repeat exactly: MKA1_EVAL_SMOKE_OK",
    "answer": "MKA1_EVAL_SMOKE_OK"
  },
  "prompt": "Repeat exactly: MKA1_EVAL_SMOKE_OK",
  "target": "MKA1_EVAL_SMOKE_OK",
  "response_id": "resp_...",
  "output_text": "MKA1_EVAL_SMOKE_OK",
  "extracted_output": "MKA1_EVAL_SMOKE_OK",
  "scores": {
    "exact_match": 1
  },
  "judge": {
    "output": "MKA1_EVAL_SMOKE_OK",
    "target": "MKA1_EVAL_SMOKE_OK"
  },
  "error": null
}

Fetch sample audio (transcription tasks)

For transcription evals, sample lists may redact inline data: audio blobs in dataset_row. To fetch a single sample’s clip reference, call: GET /api/v1/llm/evals/runs/{run_id}/samples/{sample_index}/audio sample_index is unique per task (and model), not per run. If a run includes more than one transcription task and the same sample_index could match multiple tasks, pass task_id or the API returns 400. Optionally pass model to choose which model row to read the clip from (the clip is identical across models within a task, so this does not change the audio).

const result = await sdk.llm.evals.getSampleAudio({ runId: 'eval_run_abc123', sampleIndex: 0, taskId: 'Common_voice' });
console.log(result);

result = sdk.llm.evals.get_sample_audio(run_id="eval_run_abc123", sample_index=0, task_id="Common_voice")
print(result)

var result = await sdk.Llm.Evals.GetSampleAudioAsync(new GetEvalSampleAudioRequest { RunId = "eval_run_abc123", SampleIndex = 0, TaskId = "Common_voice" });
Console.WriteLine(result);

mka1 llm evals get-sample-audio \
  --run-id eval_run_abc123 \
  --sample-index 0 \
  --task-id Common_voice

curl 'https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123/samples/0/audio?task_id=Common_voice' \
  --header 'Authorization: Bearer <mka1-api-key>'

The response includes { object, audio, sample_index, task_id, model } where audio is a base64 data: URI or a URL.

Fetch artifacts

Completed runs create result files with purpose=evals. Use the artifacts endpoint to find the result and sample artifact file IDs.

const result = await sdk.llm.evals.getArtifacts({ runId: 'eval_run_abc123' });
console.log(result);

result = sdk.llm.evals.get_artifacts(run_id="eval_run_abc123")
print(result)

var result = await sdk.Llm.Evals.GetArtifactsAsync(runId: "eval_run_abc123");
Console.WriteLine(result);

mka1 llm evals get-artifacts \
  --run-id eval_run_abc123

curl https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123/artifacts \
  --header 'Authorization: Bearer <mka1-api-key>'

Then download the files through the Files API:

Bash

curl https://apigw.mka1.com/api/v1/llm/files/file_result123/content \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --output eval-result.json

The result artifact summarizes run metadata and final metrics. The samples artifact preserves per-sample details for offline analysis.

Compare models

Open the leaderboard

In the console, open Evaluations and select an evaluation. Its Leaderboard lists every model with a completed run on the active version, scored task by task, with Token Usage and Cost charts above the table. To compare two models answer by answer, select both on the leaderboard and click Compare selected. The compare page shows both models’ answers to each sample of a task, and can narrow them to the samples only one model got right. Through the API, GET /api/v1/llm/evals/suites/{suite_id}/leaderboard-runs returns the runs behind the leaderboard. Pass suite_version, because scores are only comparable within one version of a suite.

const result = await sdk.llm.evals.listSuiteLeaderboardRuns({ suiteId: 'eval_suite_abc123', suiteVersion: 1 });
console.log(result);

result = sdk.llm.evals.list_suite_leaderboard_runs(suite_id="eval_suite_abc123", suite_version=1)
print(result)

var result = await sdk.Llm.Evals.ListSuiteLeaderboardRunsAsync(new ListEvalSuiteLeaderboardRunsRequest { SuiteId = "eval_suite_abc123", SuiteVersion = 1 });
Console.WriteLine(result);

mka1 llm evals list-suite-leaderboard-runs \
  --suite-id eval_suite_abc123 \
  --suite-version 1

curl 'https://apigw.mka1.com/api/v1/llm/evals/suites/eval_suite_abc123/leaderboard-runs?suite_version=1' \
  --header 'Authorization: Bearer <mka1-api-key>'

data holds the full run objects, and groups places them on the board. Each group entry names a model, a task_id (null when the run covered every task), a rank (1 is the newest completed run), and the run_id. Build the board from groups, not from each run’s models. If truncated is true, request again with cursor set to next_cursor and combine the responses. The API reference lists the other parameters.

Token and time fields

Completed runs report what each model spent. Each metrics.by_task.<task>.<model> entry carries tokens and timing next to its scores when they were measured, and the run’s tokens field holds the run totals:

{
  "metrics": {
    "by_task": {
      "repeat_exactly": {
        "openai:gpt-4.1-mini": {
          "sample_count": 1,
          "failed_count": 0,
          "metrics": {
            "exact_match": 1
          },
          "tokens": {
            "input": 17,
            "output": 7,
            "reasoning": 0,
            "reasoning_estimated": 0
          },
          "timing": {
            "generation_seconds": 0.784,
            "timed_samples": 1
          }
        }
      }
    }
  },
  "tokens": {
    "input": 17,
    "output": 7,
    "cached": null,
    "reasoning": null
  }
}

Field	What it holds
`metrics.by_task.<task>.<model>.tokens`	Tokens one model spent on one task, failed samples included. `input` includes `cached`, and `output` includes `reasoning`. Where the provider reports no reasoning split, `reasoning` is estimated from the stored text, and `reasoning_estimated` counts the estimated samples.
`metrics.by_task.<task>.<model>.timing`	`generation_seconds` is the wall-clock time of the model calls, summed over `timed_samples` samples. Divide one by the other for the average time per answer.
`tokens`	The run totals over every sample: `input`, `output`, `cached`, and `reasoning`. `reasoning` counts only splits the provider reported, so it is `null` when none did.

Each sample carries its own figures in input_tokens, output_tokens, and phase_timings. The generation phase records when the model call started and finished:

{
  "object": "eval.sample",
  "task_id": "repeat_exactly",
  "model": "openai:gpt-4.1-mini",
  "input_tokens": 17,
  "output_tokens": 7,
  "phase_timings": {
    "generation": {
      "started_at": "2026-10-07T09:12:03.118Z",
      "finished_at": "2026-10-07T09:12:03.902Z"
    }
  }
}

A figure that was never measured is null or absent, not zero. When the provider reports no token counts, the tokens are counted from the stored text and the sample’s tokens_estimated is true. Use those counts to compare how verbose models are, not as billing figures.

Cost per answer

The console prices answers at your organization’s effective rates, which GET /api/v1/llm/models/pricing/effective returns (Check effective prices). It applies those rates to the per-task token totals and divides by the number of answers. A model without a price has no cost per answer, so the Cost charts leave it out instead of showing it as free. Cost stays empty while your models are unpriced.

Compare more than 20 models

A run takes up to 20 models. The leaderboard is not limited to one run: it combines every completed run of a suite version and uses each model’s most recent result. To compare more than 20 models, split them across several runs of the same suite version.

Import external results

To put a coding-agent run you executed outside MKA1 on the leaderboard, import it with POST /api/v1/llm/evals/imports/agent-run. The import records a completed run against the suite version you name, or the active version by default, without executing anything. Every task_id in its trials must already exist in that version’s manifest. Pass an idempotency_key so a retried upload does not import the run twice. In the console, an evaluation of coding agents shows Import run in place of Run evaluation. The API reference describes the request body.

Pagination and filtering

List endpoints use cursor pagination.

Endpoint	Filters
`GET /evals/suites`	`after`, `limit`
`GET /evals/suites/{suite_id}/versions`	`after`, `limit`
`GET /evals/runs`	`after`, `limit`, `suite_id`, `suite_version`, `status` (Note: `suite_version` must be paired with `suite_id`.)
`GET /evals/schedules`	`after`, `limit`, `suite_id`, `enabled` (string)
`GET /evals/schedules/{schedule_id}/runs`	`after`, `limit`, `suite_id`, `suite_version`, `status` (Note: `suite_version` may be used without `suite_id` since the schedule already pins the suite; supplying a different `suite_id` returns 400.)
`GET /evals/runs/{run_id}/samples`	`after`, `limit`, `task_id`, `model`, `status`, `score_metric`, `score_min`, `score_max`, `include_dataset_row`, `include_reasoning`, `sample_index`
`GET /files`	`after`, `limit`, `order`, `purpose`

Example:

const result = await sdk.llm.evals.listRuns({ status: 'completed', limit: 20 });
console.log(result);

result = sdk.llm.evals.list_runs(status="completed", limit=20)
print(result)

var result = await sdk.Llm.Evals.ListRunsAsync(new ListEvalRunsRequest { Status = ListEvalRunsStatus.Completed, Limit = 20 });
Console.WriteLine(result);

mka1 llm evals list-runs \
  --status completed \
  --limit 20

curl 'https://apigw.mka1.com/api/v1/llm/evals/runs?status=completed&limit=20' \
  --header 'Authorization: Bearer <mka1-api-key>'

Get started

SDKs and CLI

Responses

Standalone APIs

Agents

Evals

Compute

Authentication

Production

Results and artifacts

Inspect samples

Fetch sample audio (transcription tasks)

Fetch artifacts

Compare models

Open the leaderboard

Token and time fields

Cost per answer

Compare more than 20 models

Import external results

​Inspect samples

​Fetch sample audio (transcription tasks)

​Fetch artifacts

​Compare models

​Open the leaderboard

​Token and time fields

​Cost per answer

​Compare more than 20 models

​Import external results

​Pagination and filtering

Inspect samples

Fetch sample audio (transcription tasks)

Fetch artifacts

Compare models

Open the leaderboard

Token and time fields

Cost per answer

Compare more than 20 models

Import external results

Pagination and filtering