Run an eval

This section has moved to Results and artifacts.

Use evals when you need to measure model behavior against your own tasks, datasets, scoring code, and operational settings. An eval has two layers:

Layer	What it stores
Suite	A versioned manifest with tasks, datasets, prompt templates, preprocessors, graders, and metric definitions.
Run	A durable execution of a suite version against one or more models, with generation settings, judge model, embedding model, concurrency, and result artifacts.

Eval runs use normal MKA1 routing. Candidate generations go through POST /api/v1/llm/responses. Model-backed Python graders call Responses and Embeddings through a gateway-owned bridge, so grader code never receives your API key.

Before you start

You need:

Requirement	Notes
API key scopes	Use a key with `write:evals` to create suites and runs, `read:evals` to read results, and file upload access for `/files`.
Candidate model access	The run creator chooses the candidate model IDs in `models`.
Optional judge model access	Required when Python graders call `ctx.responses_create(model="auto", ...)`.
Optional embedding model access	Required when Python graders call `ctx.embeddings_create(model="auto", ...)`.
Dataset	Upload JSONL/CSV with `purpose=evals`, or reference a supported Hugging Face dataset.
Python grader	Provide inline Python in the manifest, or upload a `.py` file with `purpose=evals`.

Use X-On-Behalf-Of when the eval belongs to a specific end user context. Suites, runs, uploaded eval files, and result artifacts are scoped to the authenticated team context.

Workflow

The normal flow is:

Upload dataset and optional Python files through /files.
Create an eval suite with a manifest.
Start an eval run for one or more models.
Poll the run until it reaches a terminal status.
Inspect sample rows and download generated artifact files.
Create a new suite version when you edit the manifest.

Eval run statuses move through:

queued -> in_progress -> finalizing -> completed
       \                         \-> failed
        \-> cancelling -> cancelled

Sample statuses are queued, generating, ready_to_score, scoring, running, completed, and failed.

Step 1 - Upload a dataset

Upload JSONL or CSV files with purpose=evals. JSONL preserves nested objects and arrays. CSV values are parsed as strings.

eval-smoke.jsonl

{"question":"Repeat exactly: MKA1_EVAL_SMOKE_OK","answer":"MKA1_EVAL_SMOKE_OK"}

import { SDK } from '@meetkai/mka1';
import { readFileSync } from 'node:fs';

const sdk = new SDK({ bearerAuth: 'Bearer <mka1-api-key>' });

const file = { fileName: 'eval-smoke.jsonl', content: readFileSync('./eval-smoke.jsonl') };
const result = await sdk.llm.files.upload({
  xOnBehalfOf: '<end-user-id>', // optional — attribute the request to one of your end users
  requestBody: { file, purpose: 'evals' },
});

import OpenAI from 'openai';
import fs from 'fs';

const openai = new OpenAI({
  apiKey: '<mka1-api-key>',
  baseURL: 'https://apigw.mka1.com/api/v1/llm/',
  defaultHeaders: { 'X-On-Behalf-Of': '<end-user-id>' },
});

const file = await openai.files.create({
  file: fs.createReadStream('./eval-smoke.jsonl'),
  purpose: 'evals',
});

from meetkai_mka1 import SDK

sdk = SDK(bearer_auth="Bearer <mka1-api-key>")

result = sdk.llm.files.upload(
    file={
        "file_name": "eval-smoke.jsonl",
        "content": open("./eval-smoke.jsonl", "rb"),
    },
    purpose="evals",
)

using MeetKai.MKA1;
using MeetKai.MKA1.Types.Components;
using MeetKai.MKA1.Types.Requests;

var sdk = new SDK(bearerAuth: "Bearer <mka1-api-key>");

var res = await sdk.Llm.Files.UploadAsync(new UploadFileRequestBody()
{
    File = new UploadFileFile()
    {
        FileName = "eval-smoke.jsonl",
        Content = System.IO.File.ReadAllBytes("./eval-smoke.jsonl"),
    },
    Purpose = UploadFilePurpose.Evals,
});

mka1 llm files upload \
  --file ./eval-smoke.jsonl \
  --purpose evals \
  -H 'X-On-Behalf-Of: <end-user-id>'

curl https://apigw.mka1.com/api/v1/llm/files \
  --request POST \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --header 'X-On-Behalf-Of: <end-user-id>' \
  --form 'file=@./eval-smoke.jsonl' \
  --form 'purpose=evals'

Store the returned file-... ID.

Step 2 - Upload a Python grader file

You can put grader source inline in the manifest. For reusable graders, upload Python files with purpose=evals.

exact_match_grader.py

def grade(sample, item):
    output = (sample.get("extracted_output") or "").strip()
    target = (item.get("target") or "").strip()
    return {
        "scores": {
            "exact_match": 1.0 if output == target else 0.0
        },
        "judge": {
            "output": output,
            "target": target
        }
    }

Bash

curl https://apigw.mka1.com/api/v1/llm/files \
  --request POST \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --form 'purpose=evals' \
  --form 'file=@./exact_match_grader.py;type=text/x-python'

Store the returned grader file-... ID.

Step 3 - Create a suite

A suite manifest defines one or more tasks. Each task renders a prompt from one dataset row, sends the prompt to each run model, extracts the model output, and grades the sample.

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/suites \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --data '{
    "name": "Production smoke eval",
    "description": "A minimal uploaded JSONL and Python grader eval.",
    "manifest": {
      "schema_version": "2026-05-27",
      "tasks": [
        {
          "id": "repeat_exactly",
          "type": "custom",
          "dataset": {
            "file_id": "file_dataset123",
            "format": "jsonl"
          },
          "prompt_template": "{{question}}",
          "target_template": "{{answer}}",
          "output_extraction": {
            "type": "none"
          },
          "metrics": [
            { "id": "exact_match" }
          ],
          "grader": {
            "type": "python",
            "contract": "sample",
            "file_id": "file_grader123",
            "timeout_seconds": 120
          }
        }
      ]
    },
    "metadata": {
      "owner": "eval-team"
    }
  }'

The response returns an eval.suite object. Use the suite id when you start a run.

Step 4 - Start a run

A run chooses the model or models to test. It can also choose a task subset, judge model, embedding model, generation settings, concurrency, and sample cap.

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/runs \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --data '{
    "suite_id": "eval_suite_abc123",
    "models": [
      "openai:gpt-4.1-mini"
    ],
    "task_ids": [
      "repeat_exactly"
    ],
    "generation": {
      "temperature": 0,
      "max_output_tokens": 32,
      "max_retries": 1,
      "max_empty_retries": 1,
      "timeout_seconds": 120
    },
    "generation_concurrency": 1,
    "grader_concurrency": 1,
    "max_samples_per_task": 1,
    "metadata": {
      "experiment": "smoke"
    }
  }'

Useful run fields:

Field	Purpose
`suite_id`	Suite to run.
`suite_version`	Optional immutable version number. Defaults to the active suite version.
`models`	Candidate model IDs. Maximum 20 per run. Duplicate model IDs are rejected.
`task_ids`	Optional subset of task IDs. Omit it to run every task in the suite version.
`judge_model`	Model used when Python grader code calls `ctx.responses_create(model="auto", ...)`.
`embedding_model`	Model used when Python grader code calls `ctx.embeddings_create(model="auto", ...)`.
`generation`	Candidate model settings and eval execution controls.
`generation_concurrency`	Number of samples to generate concurrently. Range is 1 to 256.
`grader_concurrency`	Number of samples to grade concurrently. Range is 1 to 256.
`max_samples_per_task`	Optional cap for smoke tests or partial runs.

Step 5 - Poll the run

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123 \
  --header 'Authorization: Bearer <mka1-api-key>'

While a run is active, metrics is null or empty. When it completes, metrics are grouped by model and by task:

{
  "status": "completed",
  "request_counts": {
    "total": 3,
    "completed": 3,
    "failed": 0
  },
  "metrics": {
    "by_model": {
      "openai:gpt-4.1-mini": {
        "sample_count": 3,
        "failed_count": 0,
        "metrics": {
          "exact_match": 1
        }
      }
    },
    "by_task": {
      "repeat_exactly": {
        "openai:gpt-4.1-mini": {
          "sample_count": 1,
          "failed_count": 0,
          "metrics": {
            "exact_match": 1
          }
        }
      }
    }
  }
}

by_task entries can also carry the tokens and time the model spent, described in Token and time fields.

Inspect results

Use Results and artifacts to inspect samples, filter scores, and download the output. To rank models across runs and compare their token use, time, and cost, see Compare models.

Edit a suite

Suites are versioned. Create a new immutable version when you change a manifest. Set make_active to false when you want to stage a draft version without making it the default for new runs.

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/suites/eval_suite_abc123/versions \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --data '{
    "make_active": true,
    "manifest": {
      "schema_version": "2026-05-27",
      "tasks": [
        {
          "id": "repeat_exactly",
          "type": "custom",
          "dataset": { "file_id": "file_dataset456", "format": "jsonl" },
          "prompt_template": "{{question}}",
          "target_template": "{{answer}}",
          "metrics": [{ "id": "exact_match" }],
          "grader": {
            "type": "python",
            "contract": "sample",
            "file_id": "file_grader456"
          }
        }
      ]
    },
    "metadata": {
      "change": "larger validation split"
    }
  }'

Runs keep the suite version they were created with. Changing the active version does not mutate historical runs.

Cancel a run

Cancel a run when it is queued, in progress, or finalizing.

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123/cancel \
  --request POST \
  --header 'Authorization: Bearer <mka1-api-key>'

Cancellation is best effort. Samples that are already running may finish before the workflow reaches cancelled.

Delete suites, runs, schedules, and files

Evals and schedules support soft deletion. Files are deleted from storage.

Delete an eval run

Soft-deletes an eval run so it no longer appears in run lists, details, or score leaderboards.

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/runs/eval_run_abc123 \
  --request DELETE \
  --header 'Authorization: Bearer <mka1-api-key>'

Delete an eval suite

Soft-deletes an eval suite and all of its eval runs so they no longer appear in user-facing reads. The response includes deleted_runs (how many runs were soft-deleted along with the suite).

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/suites/eval_suite_abc123 \
  --request DELETE \
  --header 'Authorization: Bearer <mka1-api-key>'

Delete an eval schedule

Soft-deletes an eval schedule and removes its Temporal schedule. Historical runs are preserved.

Bash

curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123 \
  --request DELETE \
  --header 'Authorization: Bearer <mka1-api-key>'

Delete a file

Deletes a file from storage. The file is first removed from every vector store it is attached to (embeddings included), so it no longer appears in vector store search or file_search results.

Bash

curl https://apigw.mka1.com/api/v1/llm/files/file-abc123 \
  --request DELETE \
  --header 'Authorization: Bearer <mka1-api-key>'

Get started

SDKs and CLI

Responses

Standalone APIs

Agents

Evals

Compute

Authentication

Production

Run an eval

Before you start

Workflow

Step 1 - Upload a dataset

Step 2 - Upload a Python grader file

Step 3 - Create a suite

Step 4 - Start a run

Step 5 - Poll the run

Inspect results

Edit a suite

Cancel a run

Delete suites, runs, schedules, and files

Delete an eval run

Delete an eval suite

Delete an eval schedule

Delete a file

What to read next

​Before you start

​Workflow

​Step 1 - Upload a dataset

​Step 2 - Upload a Python grader file

​Step 3 - Create a suite

​Step 4 - Start a run

​Step 5 - Poll the run

​Inspect results

​Edit a suite

​Cancel a run

​Delete suites, runs, schedules, and files

​Delete an eval run

​Delete an eval suite

​Delete an eval schedule

​Delete a file

​What to read next

Before you start

Workflow

Step 1 - Upload a dataset

Step 2 - Upload a Python grader file

Step 3 - Create a suite

Step 4 - Start a run

Step 5 - Poll the run

Inspect results

Edit a suite

Cancel a run

Delete suites, runs, schedules, and files

Delete an eval run

Delete an eval suite

Delete an eval schedule

Delete a file

What to read next