> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mka1.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Run an eval
> Create reusable eval suites, run them against MKA1-routed models, and inspect per-sample results, metrics, and artifacts.
Use evals when you need to measure model behavior against your own tasks, datasets, scoring code, and operational settings.
An eval has two layers:
| Layer | What it stores |
| - | - |
| Suite | A versioned manifest with tasks, datasets, prompt templates, preprocessors, graders, and metric definitions. |
| Run | A durable execution of a suite version against one or more models, with generation settings, judge model, embedding model, concurrency, and result artifacts. |
Eval runs use normal MKA1 routing.
Candidate generations go through `POST /api/v1/llm/responses`.
Model-backed Python graders call Responses and Embeddings through a gateway-owned bridge, so grader code never receives your API key.
## Before you start
You need:
| Requirement | Notes |
| - | - |
| API key scopes | Use a key with `write:evals` to create suites and runs, `read:evals` to read results, and file upload access for `/files`. |
| Candidate model access | The run creator chooses the candidate model IDs in `models`. |
| Optional judge model access | Required when Python graders call `ctx.responses_create(model="auto", ...)`. |
| Optional embedding model access | Required when Python graders call `ctx.embeddings_create(model="auto", ...)`. |
| Dataset | Upload JSONL/CSV with `purpose=evals`, or reference a supported Hugging Face dataset. |
| Python grader | Provide inline Python in the manifest, or upload a `.py` file with `purpose=evals`. |
Use `X-On-Behalf-Of` when the eval belongs to a specific end user context.
Suites, runs, uploaded eval files, and result artifacts are scoped to the authenticated team context.
## Workflow
The normal flow is:
1. Upload dataset and optional Python files through `/files`.
2. Create an eval suite with a manifest.
3. Start an eval run for one or more models.
4. Poll the run until it reaches a terminal status.
5. Inspect sample rows and download generated artifact files.
6. Create a new suite version when you edit the manifest.
Eval run statuses move through:
```text theme={null}
queued -> in_progress -> finalizing -> completed
\ \-> failed
\-> cancelling -> cancelled
```
Sample statuses are `queued`, `generating`, `ready_to_score`, `scoring`, `running`, `completed`, and `failed`.
## Step 1 - Upload a dataset
Upload JSONL or CSV files with `purpose=evals`.
JSONL preserves nested objects and arrays.
CSV values are parsed as strings.
```json eval-smoke.jsonl theme={null}
{"question":"Repeat exactly: MKA1_EVAL_SMOKE_OK","answer":"MKA1_EVAL_SMOKE_OK"}
```
```ts TypeScript SDK theme={null}
import { SDK } from '@meetkai/mka1';
import { readFileSync } from 'node:fs';
const sdk = new SDK({ bearerAuth: 'Bearer ' });
const file = { fileName: 'eval-smoke.jsonl', content: readFileSync('./eval-smoke.jsonl') };
const result = await sdk.llm.files.upload({
xOnBehalfOf: '', // optional — attribute the request to one of your end users
requestBody: { file, purpose: 'evals' },
});
```
```ts OpenAI SDK theme={null}
import OpenAI from 'openai';
import fs from 'fs';
const openai = new OpenAI({
apiKey: '',
baseURL: 'https://apigw.mka1.com/api/v1/llm/',
defaultHeaders: { 'X-On-Behalf-Of': '' },
});
const file = await openai.files.create({
file: fs.createReadStream('./eval-smoke.jsonl'),
purpose: 'evals',
});
```
```python Python SDK theme={null}
from meetkai_mka1 import SDK
sdk = SDK(bearer_auth="Bearer ")
result = sdk.llm.files.upload(
file={
"file_name": "eval-smoke.jsonl",
"content": open("./eval-smoke.jsonl", "rb"),
},
purpose="evals",
)
```
```csharp C# SDK theme={null}
using MeetKai.MKA1;
using MeetKai.MKA1.Types.Components;
using MeetKai.MKA1.Types.Requests;
var sdk = new SDK(bearerAuth: "Bearer ");
var res = await sdk.Llm.Files.UploadAsync(new UploadFileRequestBody()
{
File = new UploadFileFile()
{
FileName = "eval-smoke.jsonl",
Content = System.IO.File.ReadAllBytes("./eval-smoke.jsonl"),
},
Purpose = UploadFilePurpose.Evals,
});
```
```bash CLI theme={null}
mka1 llm files upload \
--file ./eval-smoke.jsonl \
--purpose evals \
-H 'X-On-Behalf-Of: '
```
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/files \
--request POST \
--header 'Authorization: Bearer ' \
--header 'X-On-Behalf-Of: ' \
--form 'file=@./eval-smoke.jsonl' \
--form 'purpose=evals'
```
Store the returned `file-...` ID.
## Step 2 - Upload a Python grader file
You can put grader source inline in the manifest.
For reusable graders, upload Python files with `purpose=evals`.
```python exact_match_grader.py theme={null}
def grade(sample, item):
output = (sample.get("extracted_output") or "").strip()
target = (item.get("target") or "").strip()
return {
"scores": {
"exact_match": 1.0 if output == target else 0.0
},
"judge": {
"output": output,
"target": target
}
}
```
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/files \
--request POST \
--header 'Authorization: Bearer ' \
--form 'purpose=evals' \
--form 'file=@./exact_match_grader.py;type=text/x-python'
```
Store the returned grader `file-...` ID.
## Step 3 - Create a suite
A suite manifest defines one or more tasks.
Each task renders a prompt from one dataset row, sends the prompt to each run model, extracts the model output, and grades the sample.
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/suites \
--request POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ' \
--data '{
"name": "Production smoke eval",
"description": "A minimal uploaded JSONL and Python grader eval.",
"manifest": {
"schema_version": "2026-05-27",
"tasks": [
{
"id": "repeat_exactly",
"type": "custom",
"dataset": {
"file_id": "file_dataset123",
"format": "jsonl"
},
"prompt_template": "{{question}}",
"target_template": "{{answer}}",
"output_extraction": {
"type": "none"
},
"metrics": [
{ "id": "exact_match" }
],
"grader": {
"type": "python",
"contract": "sample",
"file_id": "file_grader123",
"timeout_seconds": 120
}
}
]
},
"metadata": {
"owner": "eval-team"
}
}'
```
The response returns an `eval.suite` object.
Use the suite `id` when you start a run.
## Step 4 - Start a run
A run chooses the model or models to test.
It can also choose a task subset, judge model, embedding model, generation settings, concurrency, and sample cap.
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/runs \
--request POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ' \
--data '{
"suite_id": "eval_suite_abc123",
"models": [
"openai:gpt-4.1-mini"
],
"task_ids": [
"repeat_exactly"
],
"generation": {
"temperature": 0,
"max_output_tokens": 32,
"max_retries": 1,
"max_empty_retries": 1,
"timeout_seconds": 120
},
"generation_concurrency": 1,
"grader_concurrency": 1,
"max_samples_per_task": 1,
"metadata": {
"experiment": "smoke"
}
}'
```
Useful run fields:
| Field | Purpose |
| - | - |
| `suite_id` | Suite to run. |
| `suite_version` | Optional immutable version number. Defaults to the active suite version. |
| `models` | Candidate model IDs. Maximum 20 per run. Duplicate model IDs are rejected. |
| `task_ids` | Optional subset of task IDs. Omit it to run every task in the suite version. |
| `judge_model` | Model used when Python grader code calls `ctx.responses_create(model="auto", ...)`. |
| `embedding_model` | Model used when Python grader code calls `ctx.embeddings_create(model="auto", ...)`. |
| `generation` | Candidate model settings and eval execution controls. |
| `generation_concurrency` | Number of samples to generate concurrently. Range is 1 to 256. |
| `grader_concurrency` | Number of samples to grade concurrently. Range is 1 to 256. |
| `max_samples_per_task` | Optional cap for smoke tests or partial runs. |
## Step 5 - Poll the run
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123 \
--header 'Authorization: Bearer '
```
While a run is active, `metrics` is `null` or empty.
When it completes, metrics are grouped by model and by task:
```json theme={null}
{
"status": "completed",
"request_counts": {
"total": 3,
"completed": 3,
"failed": 0
},
"metrics": {
"by_model": {
"openai:gpt-4.1-mini": {
"sample_count": 3,
"failed_count": 0,
"metrics": {
"exact_match": 1
}
}
},
"by_task": {
"repeat_exactly": {
"openai:gpt-4.1-mini": {
"sample_count": 1,
"failed_count": 0,
"metrics": {
"exact_match": 1
}
}
}
}
}
}
```
`by_task` entries can also carry the tokens and time the model spent, described in [Token and time fields](/docs/eval-results#token-and-time-fields).
## Inspect results
Use [Results and artifacts](/docs/eval-results) to inspect samples, filter scores, and download the output.
To rank models across runs and compare their token use, time, and cost, see [Compare models](/docs/eval-results#compare-models).
## Edit a suite
Suites are versioned.
Create a new immutable version when you change a manifest.
Set `make_active` to `false` when you want to stage a draft version without making it the default for new runs.
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/suites/eval_suite_abc123/versions \
--request POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ' \
--data '{
"make_active": true,
"manifest": {
"schema_version": "2026-05-27",
"tasks": [
{
"id": "repeat_exactly",
"type": "custom",
"dataset": { "file_id": "file_dataset456", "format": "jsonl" },
"prompt_template": "{{question}}",
"target_template": "{{answer}}",
"metrics": [{ "id": "exact_match" }],
"grader": {
"type": "python",
"contract": "sample",
"file_id": "file_grader456"
}
}
]
},
"metadata": {
"change": "larger validation split"
}
}'
```
Runs keep the suite version they were created with.
Changing the active version does not mutate historical runs.
## Cancel a run
Cancel a run when it is queued, in progress, or finalizing.
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123/cancel \
--request POST \
--header 'Authorization: Bearer '
```
Cancellation is best effort.
Samples that are already running may finish before the workflow reaches `cancelled`.
## Delete suites, runs, schedules, and files
Evals and schedules support soft deletion.
Files are deleted from storage.
### Delete an eval run
Soft-deletes an eval run so it no longer appears in run lists, details, or score leaderboards.
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123 \
--request DELETE \
--header 'Authorization: Bearer '
```
### Delete an eval suite
Soft-deletes an eval suite and all of its eval runs so they no longer appear in user-facing reads. The response includes `deleted_runs` (how many runs were soft-deleted along with the suite).
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/suites/eval_suite_abc123 \
--request DELETE \
--header 'Authorization: Bearer '
```
### Delete an eval schedule
Soft-deletes an eval schedule and removes its Temporal schedule. Historical runs are preserved.
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123 \
--request DELETE \
--header 'Authorization: Bearer '
```
### Delete a file
Deletes a file from storage. The file is first removed from every vector store it is attached to (embeddings included), so it no longer appears in vector store search or `file_search` results.
```bash Bash theme={null}
curl https://apigw.mka1.com/api/v1/llm/files/file-abc123 \
--request DELETE \
--header 'Authorization: Bearer '
```
## What to read next
* [Design eval task suites](/docs/evals-task-suites) covers datasets, task manifests, templates, few-shot examples, output extraction, generation knobs, and metrics.
* [Write Python eval graders](/docs/evals-python-graders) covers sample, batch, and model-backed Python contracts.
* Use the endpoint paths in this guide with the generated request and response objects returned by the API.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.