Schedule eval runs

First create an eval suite and keep its ID. A schedule stores a run configuration plus a cadence. Each time the cadence fires, the schedule starts a normal eval run, the same kind you start with POST /api/v1/llm/evals/runs. You read its samples and metrics the same way, through Results and artifacts. Creating, updating, triggering, and deleting schedules requires an API key with write:evals. Reading schedules and their runs requires read:evals.

Create a schedule

import { SDK } from '@meetkai/mka1';

const sdk = new SDK({ bearerAuth: 'Bearer <mka1-api-key>' });

const schedule = await sdk.llm.evals.createSchedule({
  createEvalScheduleRequest: {
    name: 'Nightly smoke eval',
    suiteId: 'eval_suite_abc123',
    cron: '0 3 * * *',
    timezone: 'America/Los_Angeles',
    run: {
      models: ['openai:gpt-4.1-mini'],
      generation: { temperature: 0, maxOutputTokens: 32 },
      maxSamplesPerTask: 50,
    },
    webhookUrl: 'https://example.com/hooks/evals',
    webhookSecret: '<webhook-signing-secret>',
    metadata: { owner: 'eval-team' },
  },
});

console.log(schedule.id, schedule.nextRunAt);

from meetkai_mka1 import SDK

sdk = SDK(bearer_auth="Bearer <mka1-api-key>")

schedule = sdk.llm.evals.create_schedule(
    name="Nightly smoke eval",
    suite_id="eval_suite_abc123",
    cron="0 3 * * *",
    timezone="America/Los_Angeles",
    run={
        "models": ["openai:gpt-4.1-mini"],
        "generation": {"temperature": 0, "max_output_tokens": 32},
        "max_samples_per_task": 50,
    },
    webhook_url="https://example.com/hooks/evals",
    webhook_secret="<webhook-signing-secret>",
    metadata={"owner": "eval-team"},
)

print(schedule.id, schedule.next_run_at)

using MeetKai.MKA1;
using MeetKai.MKA1.Types.Components;
using MeetKai.MKA1.Types.Requests;

var sdk = new SDK(bearerAuth: "Bearer <mka1-api-key>");

var created = await sdk.Llm.Evals.CreateScheduleAsync(new MeetKai.MKA1.Types.Components.CreateEvalScheduleRequest()
{
    Name = "Nightly smoke eval",
    SuiteId = "eval_suite_abc123",
    Cron = "0 3 * * *",
    Timezone = "America/Los_Angeles",
    Run = new EvalScheduleRunRequestConfig()
    {
        Models = new List<string> { "openai:gpt-4.1-mini" },
        Generation = new EvalGenerationRequestConfig() { Temperature = 0, MaxOutputTokens = 32 },
        MaxSamplesPerTask = 50,
    },
    WebhookUrl = "https://example.com/hooks/evals",
    WebhookSecret = "<webhook-signing-secret>",
    Metadata = new Dictionary<string, string> { { "owner", "eval-team" } },
});

var schedule = created.Object!;
Console.WriteLine($"{schedule.Id} {schedule.NextRunAt}");

mka1 llm evals create-schedule \
  --body '{
    "name": "Nightly smoke eval",
    "suite_id": "eval_suite_abc123",
    "cron": "0 3 * * *",
    "timezone": "America/Los_Angeles",
    "run": {
      "models": ["openai:gpt-4.1-mini"],
      "generation": { "temperature": 0, "max_output_tokens": 32 },
      "max_samples_per_task": 50
    },
    "webhook_url": "https://example.com/hooks/evals",
    "webhook_secret": "<webhook-signing-secret>",
    "metadata": { "owner": "eval-team" }
  }'

curl https://apigw.mka1.com/api/v1/llm/evals/schedules \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --data '{
    "name": "Nightly smoke eval",
    "suite_id": "eval_suite_abc123",
    "cron": "0 3 * * *",
    "timezone": "America/Los_Angeles",
    "run": {
      "models": ["openai:gpt-4.1-mini"],
      "generation": { "temperature": 0, "max_output_tokens": 32 },
      "max_samples_per_task": 50
    },
    "webhook_url": "https://example.com/hooks/evals",
    "webhook_secret": "<webhook-signing-secret>",
    "metadata": { "owner": "eval-team" }
  }'

The response is an eval.schedule object with an eval_sched_... ID. enabled is true unless you pass "enabled": false, which creates the schedule paused.

Choose a cadence

Provide exactly one of these fields:

Field	Fires
`cron`	On every match of a 5-field cron expression, evaluated in `timezone`. `timezone` is an IANA name and defaults to `UTC`. Daylight saving changes are handled for you.
`interval_seconds`	Every `interval_seconds` seconds. The minimum is 60. `timezone` does not apply.

An invalid cron expression is rejected with a 400 that explains the problem. Each fire starts up to 30 seconds after its scheduled time, so schedules that share a boundary do not all start at once. A schedule set to * * * * * and created at 18:31:33 starts its first run at about 18:32. If the scheduler is unavailable when a fire is due, that fire still starts when the scheduler recovers within 5 minutes. A fire missed by more than 5 minutes is skipped.

What each run uses

run is a frozen copy of the run fields, without suite_id and suite_version. It sets the models, optional task subset, judge and embedding models, generation settings, concurrency, and sample cap. models is required.
suite_version pins one suite version. Omit it to run the suite’s active version at each fire, so a new active version takes effect at the next fire.
metadata is copied onto every run the schedule starts.
Each run executes as the identity that created the schedule, or that most recently updated it. At each fire, that identity’s access is checked again, and its API key scopes are capped at the scopes it had when it last saved the schedule.

Check a schedule

Retrieve a schedule to see when it fires next and what happened the last time it fired.

const current = await sdk.llm.evals.getSchedule({ scheduleId: 'eval_sched_abc123' });
console.log(current.enabled, current.nextRunAt, current.lastRunId, current.lastRunStatus);

current = sdk.llm.evals.get_schedule(schedule_id="eval_sched_abc123")
print(current.enabled, current.next_run_at, current.last_run_id, current.last_run_status)

var current = (await sdk.Llm.Evals.GetScheduleAsync(scheduleId: "eval_sched_abc123")).Object!;
Console.WriteLine($"{current.Enabled} {current.NextRunAt} {current.LastRunId} {current.LastRunStatus}");

mka1 llm evals get-schedule \
  --schedule-id eval_sched_abc123

curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123 \
  --header 'Authorization: Bearer <mka1-api-key>'

Field	Meaning
`next_run_at`	Unix timestamp of the next scheduled fire, when it can be resolved.
`last_run_at`	When the schedule last fired.
`last_run_id`	The run that fire created. `null` when the fire was skipped.
`last_run_status`	The outcome recorded when the schedule fired. It is not the run’s live status.

last_run_status takes these values:

Value	Meaning
`queued`	The fire created a run. The value stays `queued` after the run finishes. To see the run’s current status, retrieve the run by `last_run_id` or list the schedule’s runs.
`failed`	The fire created a run, but the run failed to start.
`skipped_suite_deleted`	The target suite or pinned suite version was deleted. The schedule was paused.
`skipped_identity_revoked`	The identity that owns the schedule lost access. For example, its API key was revoked or its membership was removed. The schedule was paused.

List the runs a schedule created

const runs = await sdk.llm.evals.listScheduleRuns({ scheduleId: 'eval_sched_abc123', limit: 10 });
console.log(runs.data.map((run) => [run.id, run.status]));

runs = sdk.llm.evals.list_schedule_runs(schedule_id="eval_sched_abc123", limit=10)
print([(run.id, run.status) for run in runs.data])

var runs = await sdk.Llm.Evals.ListScheduleRunsAsync(new ListEvalScheduleRunsRequest { ScheduleId = "eval_sched_abc123", Limit = 10 });
foreach (var run in runs.Object!.Data) Console.WriteLine($"{run.Id} {run.Status}");

mka1 llm evals list-schedule-runs \
  --schedule-id eval_sched_abc123 \
  --limit 10

curl 'https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123/runs?limit=10' \
  --header 'Authorization: Bearer <mka1-api-key>'

Runs are listed newest first. Filter with status, suite_version, or suite_id. Passing a suite_id other than the schedule’s returns 400.

Trigger a run now

Trigger a schedule to fire it immediately, in addition to its normal cadence. The fire creates a run exactly as a scheduled fire would.

await sdk.llm.evals.triggerSchedule({ scheduleId: 'eval_sched_abc123' });

sdk.llm.evals.trigger_schedule(schedule_id="eval_sched_abc123")

await sdk.Llm.Evals.TriggerScheduleAsync(scheduleId: "eval_sched_abc123");

mka1 llm evals trigger-schedule \
  --schedule-id eval_sched_abc123

curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123/trigger \
  --request POST \
  --header 'Authorization: Bearer <mka1-api-key>'

A 200 response means the fire was accepted, not that a run was created. Confirm the run with last_run_id or the schedule’s runs list.

Under the default skip overlap policy, a trigger while one of the schedule’s runs is still active returns 200, but no run is created. See Overlapping runs.
Triggering a paused schedule returns 409. Resume it first.
A 400 with code schedule_trigger_unknown means the API could not confirm whether the fire started. Check the schedule’s runs before you trigger again. Do not retry this error automatically.

Overlapping runs

overlap_policy decides what happens when a fire arrives while an earlier run from the same schedule is still active.

Value	Behavior
`skip`	Default. The fire is dropped and no run is created. `last_run_*` keeps describing the earlier fire.
`allow_all`	Every fire starts a run, so runs can stack.
`buffer_one`	Behaves like `allow_all` for runs that take longer than a few seconds.

Use skip for schedules whose runs can outlast the interval between fires. There is no team-wide cap on concurrent scheduled runs, so with allow_all a short interval can start runs faster than they finish. Keep generation_concurrency and grader_concurrency in run low enough for that.

Pause, resume, and update

Update any field with a partial request. To pause a schedule, set enabled to false. To resume it, set enabled back to true.

await sdk.llm.evals.updateSchedule({
  scheduleId: 'eval_sched_abc123',
  updateEvalScheduleRequest: { enabled: false },
});

sdk.llm.evals.update_schedule(schedule_id="eval_sched_abc123", enabled=False)

await sdk.Llm.Evals.UpdateScheduleAsync(
    scheduleId: "eval_sched_abc123",
    body: new MeetKai.MKA1.Types.Components.UpdateEvalScheduleRequest() { Enabled = false }
);

mka1 llm evals update-schedule \
  --schedule-id eval_sched_abc123 \
  --body '{ "enabled": false }'

curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123 \
  --request PATCH \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <mka1-api-key>' \
  --data '{ "enabled": false }'

A paused schedule keeps its cadence and fires again when you resume it.
To switch cadence, set the new field, for example { "interval_seconds": 3600 }. The other cadence field is cleared. Sending both returns 400.
Replacing run replaces the whole run configuration.
Every successful update, even a rename, rebinds future runs to the identity making the update.
If an update returns a server error, retrieve the schedule. If enabled is now false, the update was saved but the schedule was paused for safety. Send the update again with "enabled": true to resume it.

The API also pauses a schedule on its own, setting enabled to false, when the target suite version is deleted or the owning identity loses access. last_run_status tells you which. Fix the cause, then set enabled to true. To stop a schedule permanently, delete it. Deleting a schedule keeps the runs it already created.

Get notified about runs

Set webhook_url to receive a POST each time a run the schedule created changes status. The URL must use HTTPS and resolve to public addresses. The webhook receives these events:

Event	Sent when
`eval.run.started`	The run starts.
`eval.run.completed`	The run completes.
`eval.run.failed`	The run fails, including a run that fails before it starts.
`eval.run.cancelled`	The run is cancelled.

Each request body has this shape:

{
  "event": "eval.run.failed",
  "resource_id": "eval_run_abc123",
  "created_at": 1793128895,
  "data": {
    "eval_schedule_id": "eval_sched_abc123",
    "run_id": "eval_run_abc123",
    "suite_id": "eval_suite_abc123",
    "status": "failed",
    "metrics": null,
    "error": { "code": "...", "message": "..." }
  }
}

When you set webhook_secret (16 to 256 characters), each request carries X-Webhook-Signature: sha256=<hmac>, an HMAC-SHA256 of the request body. The secret is never returned. webhook_configured tells you whether one is stored.
Delivery is at least once. Deduplicate on resource_id plus event.
Each event is attempted up to 3 times with backoff. Only a 2xx response counts as delivered. Events that still fail are not queued for later delivery.

Use eval.run.failed as your failure alert. A run whose samples all fail still completes, so also check request_counts.failed and metrics in eval.run.completed events.

Get started

SDKs and CLI

Responses

Standalone APIs

Agents

Evals

Compute

Authentication

Production

Schedule eval runs

Create a schedule

Choose a cadence

What each run uses

Check a schedule

List the runs a schedule created

Trigger a run now

Overlapping runs

Pause, resume, and update

Get notified about runs

What to read next

​Create a schedule

​Choose a cadence

​What each run uses

​Check a schedule

​List the runs a schedule created

​Trigger a run now

​Overlapping runs

​Pause, resume, and update

​Get notified about runs

​What to read next

Create a schedule

Choose a cadence

What each run uses

Check a schedule

List the runs a schedule created

Trigger a run now

Overlapping runs

Pause, resume, and update

Get notified about runs

What to read next