> ## Documentation Index > Fetch the complete documentation index at: https://docs.mka1.com/llms.txt > Use this file to discover all available pages before exploring further. # Schedule eval runs > Run an eval suite on a cron or interval cadence, trigger it on demand, and get notified when runs finish or fail. First [create an eval suite](https://docs.mka1.com/docs/evals.md#step-3---create-a-suite) and keep its ID. A schedule stores a run configuration plus a cadence. Each time the cadence fires, the schedule starts a normal eval run, the same kind you start with `POST /api/v1/llm/evals/runs`. You read its samples and metrics the same way, through [Results and artifacts](https://docs.mka1.com/docs/eval-results.md). Creating, updating, triggering, and deleting schedules requires an API key with `write:evals`. Reading schedules and their runs requires `read:evals`. ## Create a schedule ```ts TypeScript SDK theme={null} import { SDK } from '@meetkai/mka1'; const sdk = new SDK({ bearerAuth: 'Bearer ' }); const schedule = await sdk.llm.evals.createSchedule({ createEvalScheduleRequest: { name: 'Nightly smoke eval', suiteId: 'eval_suite_abc123', cron: '0 3 * * *', timezone: 'America/Los_Angeles', run: { models: ['openai:gpt-4.1-mini'], generation: { temperature: 0, maxOutputTokens: 32 }, maxSamplesPerTask: 50, }, webhookUrl: 'https://example.com/hooks/evals', webhookSecret: '', metadata: { owner: 'eval-team' }, }, }); console.log(schedule.id, schedule.nextRunAt); ``` ```python Python SDK theme={null} from meetkai_mka1 import SDK sdk = SDK(bearer_auth="Bearer ") schedule = sdk.llm.evals.create_schedule( name="Nightly smoke eval", suite_id="eval_suite_abc123", cron="0 3 * * *", timezone="America/Los_Angeles", run={ "models": ["openai:gpt-4.1-mini"], "generation": {"temperature": 0, "max_output_tokens": 32}, "max_samples_per_task": 50, }, webhook_url="https://example.com/hooks/evals", webhook_secret="", metadata={"owner": "eval-team"}, ) print(schedule.id, schedule.next_run_at) ``` ```csharp C# SDK theme={null} using MeetKai.MKA1; using MeetKai.MKA1.Types.Components; using MeetKai.MKA1.Types.Requests; var sdk = new SDK(bearerAuth: "Bearer "); var created = await sdk.Llm.Evals.CreateScheduleAsync(new MeetKai.MKA1.Types.Components.CreateEvalScheduleRequest() { Name = "Nightly smoke eval", SuiteId = "eval_suite_abc123", Cron = "0 3 * * *", Timezone = "America/Los_Angeles", Run = new EvalScheduleRunRequestConfig() { Models = new List { "openai:gpt-4.1-mini" }, Generation = new EvalGenerationRequestConfig() { Temperature = 0, MaxOutputTokens = 32 }, MaxSamplesPerTask = 50, }, WebhookUrl = "https://example.com/hooks/evals", WebhookSecret = "", Metadata = new Dictionary { { "owner", "eval-team" } }, }); var schedule = created.Object!; Console.WriteLine($"{schedule.Id} {schedule.NextRunAt}"); ``` ```bash CLI theme={null} mka1 llm evals create-schedule \ --body '{ "name": "Nightly smoke eval", "suite_id": "eval_suite_abc123", "cron": "0 3 * * *", "timezone": "America/Los_Angeles", "run": { "models": ["openai:gpt-4.1-mini"], "generation": { "temperature": 0, "max_output_tokens": 32 }, "max_samples_per_task": 50 }, "webhook_url": "https://example.com/hooks/evals", "webhook_secret": "", "metadata": { "owner": "eval-team" } }' ``` ```bash Bash theme={null} curl https://apigw.mka1.com/api/v1/llm/evals/schedules \ --request POST \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "name": "Nightly smoke eval", "suite_id": "eval_suite_abc123", "cron": "0 3 * * *", "timezone": "America/Los_Angeles", "run": { "models": ["openai:gpt-4.1-mini"], "generation": { "temperature": 0, "max_output_tokens": 32 }, "max_samples_per_task": 50 }, "webhook_url": "https://example.com/hooks/evals", "webhook_secret": "", "metadata": { "owner": "eval-team" } }' ``` The response is an `eval.schedule` object with an `eval_sched_...` ID. `enabled` is `true` unless you pass `"enabled": false`, which creates the schedule paused. ### Choose a cadence Provide exactly one of these fields: | Field | Fires | | - | - | | `cron` | On every match of a 5-field cron expression, evaluated in `timezone`. `timezone` is an IANA name and defaults to `UTC`. Daylight saving changes are handled for you. | | `interval_seconds` | Every `interval_seconds` seconds. The minimum is 60. `timezone` does not apply. | An invalid cron expression is rejected with a `400` that explains the problem. Each fire starts up to 30 seconds after its scheduled time, so schedules that share a boundary do not all start at once. A schedule set to `* * * * *` and created at 18:31:33 starts its first run at about 18:32. If the scheduler is unavailable when a fire is due, that fire still starts when the scheduler recovers within 5 minutes. A fire missed by more than 5 minutes is skipped. ### What each run uses * `run` is a frozen copy of the [run fields](https://docs.mka1.com/docs/evals.md#step-4---start-a-run), without `suite_id` and `suite_version`. It sets the models, optional task subset, judge and embedding models, generation settings, concurrency, and sample cap. `models` is required. * `suite_version` pins one suite version. Omit it to run the suite's active version at each fire, so a new active version takes effect at the next fire. * `metadata` is copied onto every run the schedule starts. * Each run executes as the identity that created the schedule, or that most recently updated it. At each fire, that identity's access is checked again, and its API key scopes are capped at the scopes it had when it last saved the schedule. ## Check a schedule Retrieve a schedule to see when it fires next and what happened the last time it fired. ```ts TypeScript SDK theme={null} const current = await sdk.llm.evals.getSchedule({ scheduleId: 'eval_sched_abc123' }); console.log(current.enabled, current.nextRunAt, current.lastRunId, current.lastRunStatus); ``` ```python Python SDK theme={null} current = sdk.llm.evals.get_schedule(schedule_id="eval_sched_abc123") print(current.enabled, current.next_run_at, current.last_run_id, current.last_run_status) ``` ```csharp C# SDK theme={null} var current = (await sdk.Llm.Evals.GetScheduleAsync(scheduleId: "eval_sched_abc123")).Object!; Console.WriteLine($"{current.Enabled} {current.NextRunAt} {current.LastRunId} {current.LastRunStatus}"); ``` ```bash CLI theme={null} mka1 llm evals get-schedule \ --schedule-id eval_sched_abc123 ``` ```bash Bash theme={null} curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123 \ --header 'Authorization: Bearer ' ``` | Field | Meaning | | - | - | | `next_run_at` | Unix timestamp of the next scheduled fire, when it can be resolved. | | `last_run_at` | When the schedule last fired. | | `last_run_id` | The run that fire created. `null` when the fire was skipped. | | `last_run_status` | The outcome recorded when the schedule fired. It is not the run's live status. | `last_run_status` takes these values: | Value | Meaning | | - | - | | `queued` | The fire created a run. The value stays `queued` after the run finishes. To see the run's current status, retrieve the run by `last_run_id` or list the schedule's runs. | | `failed` | The fire created a run, but the run failed to start. | | `skipped_suite_deleted` | The target suite or pinned suite version was deleted. The schedule was paused. | | `skipped_identity_revoked` | The identity that owns the schedule lost access. For example, its API key was revoked or its membership was removed. The schedule was paused. | ### List the runs a schedule created ```ts TypeScript SDK theme={null} const runs = await sdk.llm.evals.listScheduleRuns({ scheduleId: 'eval_sched_abc123', limit: 10 }); console.log(runs.data.map((run) => [run.id, run.status])); ``` ```python Python SDK theme={null} runs = sdk.llm.evals.list_schedule_runs(schedule_id="eval_sched_abc123", limit=10) print([(run.id, run.status) for run in runs.data]) ``` ```csharp C# SDK theme={null} var runs = await sdk.Llm.Evals.ListScheduleRunsAsync(new ListEvalScheduleRunsRequest { ScheduleId = "eval_sched_abc123", Limit = 10 }); foreach (var run in runs.Object!.Data) Console.WriteLine($"{run.Id} {run.Status}"); ``` ```bash CLI theme={null} mka1 llm evals list-schedule-runs \ --schedule-id eval_sched_abc123 \ --limit 10 ``` ```bash Bash theme={null} curl 'https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123/runs?limit=10' \ --header 'Authorization: Bearer ' ``` Runs are listed newest first. Filter with `status`, `suite_version`, or `suite_id`. Passing a `suite_id` other than the schedule's returns `400`. ## Trigger a run now Trigger a schedule to fire it immediately, in addition to its normal cadence. The fire creates a run exactly as a scheduled fire would. ```ts TypeScript SDK theme={null} await sdk.llm.evals.triggerSchedule({ scheduleId: 'eval_sched_abc123' }); ``` ```python Python SDK theme={null} sdk.llm.evals.trigger_schedule(schedule_id="eval_sched_abc123") ``` ```csharp C# SDK theme={null} await sdk.Llm.Evals.TriggerScheduleAsync(scheduleId: "eval_sched_abc123"); ``` ```bash CLI theme={null} mka1 llm evals trigger-schedule \ --schedule-id eval_sched_abc123 ``` ```bash Bash theme={null} curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123/trigger \ --request POST \ --header 'Authorization: Bearer ' ``` A `200` response means the fire was accepted, not that a run was created. Confirm the run with `last_run_id` or the schedule's runs list. * Under the default `skip` overlap policy, a trigger while one of the schedule's runs is still active returns `200`, but no run is created. See [Overlapping runs](#overlapping-runs). * Triggering a paused schedule returns `409`. Resume it first. * A `400` with code `schedule_trigger_unknown` means the API could not confirm whether the fire started. Check the schedule's runs before you trigger again. Do not retry this error automatically. ## Overlapping runs `overlap_policy` decides what happens when a fire arrives while an earlier run from the same schedule is still active. | Value | Behavior | | - | - | | `skip` | Default. The fire is dropped and no run is created. `last_run_*` keeps describing the earlier fire. | | `allow_all` | Every fire starts a run, so runs can stack. | | `buffer_one` | Behaves like `allow_all` for runs that take longer than a few seconds. | Use `skip` for schedules whose runs can outlast the interval between fires. There is no team-wide cap on concurrent scheduled runs, so with `allow_all` a short interval can start runs faster than they finish. Keep `generation_concurrency` and `grader_concurrency` in `run` low enough for that. ## Pause, resume, and update Update any field with a partial request. To pause a schedule, set `enabled` to `false`. To resume it, set `enabled` back to `true`. ```ts TypeScript SDK theme={null} await sdk.llm.evals.updateSchedule({ scheduleId: 'eval_sched_abc123', updateEvalScheduleRequest: { enabled: false }, }); ``` ```python Python SDK theme={null} sdk.llm.evals.update_schedule(schedule_id="eval_sched_abc123", enabled=False) ``` ```csharp C# SDK theme={null} await sdk.Llm.Evals.UpdateScheduleAsync( scheduleId: "eval_sched_abc123", body: new MeetKai.MKA1.Types.Components.UpdateEvalScheduleRequest() { Enabled = false } ); ``` ```bash CLI theme={null} mka1 llm evals update-schedule \ --schedule-id eval_sched_abc123 \ --body '{ "enabled": false }' ``` ```bash Bash theme={null} curl https://apigw.mka1.com/api/v1/llm/evals/schedules/eval_sched_abc123 \ --request PATCH \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "enabled": false }' ``` * A paused schedule keeps its cadence and fires again when you resume it. * To switch cadence, set the new field, for example `{ "interval_seconds": 3600 }`. The other cadence field is cleared. Sending both returns `400`. * Replacing `run` replaces the whole run configuration. * Every successful update, even a rename, rebinds future runs to the identity making the update. * If an update returns a server error, retrieve the schedule. If `enabled` is now `false`, the update was saved but the schedule was paused for safety. Send the update again with `"enabled": true` to resume it. The API also pauses a schedule on its own, setting `enabled` to `false`, when the target suite version is deleted or the owning identity loses access. `last_run_status` tells you which. Fix the cause, then set `enabled` to `true`. To stop a schedule permanently, [delete it](https://docs.mka1.com/docs/evals.md#delete-an-eval-schedule). Deleting a schedule keeps the runs it already created. ## Get notified about runs Set `webhook_url` to receive a `POST` each time a run the schedule created changes status. The URL must use HTTPS and resolve to public addresses. The webhook receives these events: | Event | Sent when | | - | - | | `eval.run.started` | The run starts. | | `eval.run.completed` | The run completes. | | `eval.run.failed` | The run fails, including a run that fails before it starts. | | `eval.run.cancelled` | The run is cancelled. | Each request body has this shape: ```json theme={null} { "event": "eval.run.failed", "resource_id": "eval_run_abc123", "created_at": 1793128895, "data": { "eval_schedule_id": "eval_sched_abc123", "run_id": "eval_run_abc123", "suite_id": "eval_suite_abc123", "status": "failed", "metrics": null, "error": { "code": "...", "message": "..." } } } ``` * When you set `webhook_secret` (16 to 256 characters), each request carries `X-Webhook-Signature: sha256=`, an HMAC-SHA256 of the request body. The secret is never returned. `webhook_configured` tells you whether one is stored. * Delivery is at least once. Deduplicate on `resource_id` plus `event`. * Each event is attempted up to 3 times with backoff. Only a `2xx` response counts as delivered. Events that still fail are not queued for later delivery. Use `eval.run.failed` as your failure alert. A run whose samples all fail still completes, so also check `request_counts.failed` and `metrics` in `eval.run.completed` events. ## What to read next * [Run an eval](https://docs.mka1.com/docs/evals.md) covers suites, one-off runs, and deletion. * [Results and artifacts](https://docs.mka1.com/docs/eval-results.md) covers samples, scores, and artifact downloads. * The [eval schedules API reference](https://docs.mka1.com/api-reference/evals/create-an-eval-schedule.md) lists every request and response field. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.