> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mka1.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> Generate training datasets with the platform's generators — quote the run, create a generation, watch it publish to mka1-repos, and download the result with the Hugging Face CLI.

The Datasets API turns a generator and a set of parameters into a training dataset that lands in one of your [mka1-repos](/docs/repositories) repositories.
You never write rows yourself: you pick a generator from the catalog, quote the run, create a *generation*, and download the published dataset when it succeeds.

Three nouns cover the whole API:

| Noun       | What it is                                                                                                                | Where it comes from                                                  |
| ---------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Generator  | A packaged program plus a JSON Schema for its parameters. It emits exactly one dataset format.                            | Provided by the platform. `GET /generators` lists what is available. |
| Generation | One run of a generator with one set of parameters. It publishes a dataset repository and is retained forever as a record. | `POST /generations`.                                                 |
| Format     | The row shape and repository layout a dataset follows, such as `chat-jsonl-v1`. Training workloads read these directly.   | `GET /formats`.                                                      |

Every generation runs as a [Compute](/docs/compute-resources) job under your organization and publishes under your API key, so the hardware and any LLM calls the generator makes bill to you like any other platform usage.

<Note>
  Datasets is coming soon. The endpoints described here are not available yet.
</Note>

## Before you start

You need:

| Requirement                  | Notes                                                                                                                                                                                                                                                                  |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| API key                      | Send it as `Authorization: Bearer <mka1-api-key>` on every request. Reads need the `read:datasets` scope; creating or cancelling a generation needs `write:datasets`. A console session can read but not write: a write from a session returns `401 api_key_required`. |
| Org owner or admin role      | Every Datasets endpoint is restricted to owners and admins of the organization.                                                                                                                                                                                        |
| Compute enabled for your org | Generations run as Compute jobs in your organization. If your org is not enabled, the generation is accepted and then fails with `reason.code: compute_not_enabled`.                                                                                                   |
| `hf` CLI                     | The published dataset is an ordinary repository; the [Hugging Face CLI](/docs/repositories#push-and-pull-with-the-hugging-face-cli) downloads it.                                                                                                                      |

Everything you create belongs to your organization.
A generation from another organization is `404`, never `403`.

The examples export the key once:

```bash theme={null}
export MKA1_API_KEY=<mka1-api-key>
```

## Dataset formats

A dataset is a repository with `train.jsonl` at its root and, optionally, `test.jsonl` and `validation.jsonl`: UTF-8, one JSON object per line.
Alongside the rows, the platform writes a `README.md` with an `mka1:` front-matter block and a `provenance.json` that records the generator, its version, the exact parameters, row counts, and LLM usage.
Every row in a split has the same format, and the format is what a trainer declares it accepts.

All three formats share the OpenAI-style message object:

```json theme={null}
{"role": "system" | "user" | "assistant" | "tool", "content": "…", "name": "…", "tool_calls": [], "tool_call_id": "…", "reasoning": "…"}
```

`reasoning` is an MKA1 extension for assistant turns that carries a thinking trace apart from `content`; trainers that do not understand it ignore it.

| Format                | Trains    | One row                                                                                                                                                                                                                                                              |
| --------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `chat-jsonl-v1`       | SFT, LoRA | `{"messages": [{"role": "user", "content": "…"}, {"role": "assistant", "content": "…"}]}` — at least one assistant turn, and the last message is the assistant's. Optional `id` and `meta` keys. The `messages` shape that ms-swift, TRL, and axolotl read natively. |
| `prompt-jsonl-v1`     | GRPO, RFT | `{"messages": [{"role": "user", "content": "…"}], "answer": "…", "info": {}}` — no assistant turn. `answer` and `info` are optional pass-throughs for reference-answer or rubric rewards. Rule-judged datasets also ship a `judge/` directory.                       |
| `preference-jsonl-v1` | DPO, ORPO | `{"input": [...], "preferred_output": [...], "non_preferred_output": [...]}` — OpenAI's spelling; the format's `to_trl` converter rewrites it as `prompt`/`chosen`/`rejected` for TRL and ms-swift. Each output is exactly one assistant message.                    |

The catalog and each format's row schema are served by the API, so a trainer can validate a dataset before spending GPU time:

<CodeGroup>
  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/formats \
    --header "Authorization: Bearer $MKA1_API_KEY"

  curl https://apigw.mka1.com/api/v1/datasets/formats/chat-jsonl-v1 \
    --header "Authorization: Bearer $MKA1_API_KEY"
  ```
</CodeGroup>

```json theme={null}
{
  "data": [
    {"id": "chat-jsonl-v1", "title": "Chat (SFT)", "trains": ["sft", "lora"]},
    {"id": "prompt-jsonl-v1", "title": "Prompt-only (RL)", "trains": ["grpo", "rft"]},
    {"id": "preference-jsonl-v1", "title": "Pairwise preference", "trains": ["dpo", "orpo"]}
  ]
}
```

A single format returns its `row_schema` (JSON Schema for one line), `layout` (required and optional files), `semantics` (the checks beyond schema, such as "the last turn is the assistant's"), and `converters` (for example `to_trl`, which rewrites preference rows as `prompt`/`chosen`/`rejected`).

## Generate a dataset

The walkthrough uses `constitution-prompts`, which turns a written constitution into a prompt-only dataset for reinforcement learning: it extracts the rules, samples topics, and generates user prompts that target each rule.
Every generator follows the same five steps; only the parameters change.

### 1. Discover a generator

List the available generators, then read one to get its parameter schema.

<CodeGroup>
  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generators \
    --header "Authorization: Bearer $MKA1_API_KEY"

  curl https://apigw.mka1.com/api/v1/datasets/generators/constitution-prompts \
    --header "Authorization: Bearer $MKA1_API_KEY"
  ```
</CodeGroup>

The list is not paginated.
Reading a single generator adds `params_schema` (JSON Schema, draft 2020-12) and `example_params`, a set of parameters the author guarantees to be valid:

```json theme={null}
{
  "id": "constitution-prompts",
  "version": "1.0.0",
  "title": "Constitution → RL prompts",
  "description": "Extracts rules from a constitution, samples topics, and generates rule-targeted user prompts.",
  "emits": "prompt-jsonl-v1",
  "resources": {"cpu": "2", "memory": "4Gi", "ephemeral_storage": "20Gi"},
  "default_limits": {"max_runtime_hours": 6, "max_cost": 50, "max_llm_calls": 20000},
  "inputs": {},
  "params_schema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["constitution"],
    "properties": {
      "constitution": {"type": "string", "description": "Constitution text"},
      "num_prompts": {"type": "integer", "default": 5500, "minimum": 10},
      "test_size": {"type": "integer", "default": 500},
      "language": {"type": "string", "default": "English"},
      "seed": {"type": "integer", "default": 100},
      "temperature": {"type": "number", "default": 1.0, "minimum": 0, "maximum": 2}
    }
  },
  "example_params": {"constitution": "…", "num_prompts": 100, "test_size": 10}
}
```

`version` is a label the platform records on every generation and in the dataset's provenance; you do not choose it.
`default_limits` are the ceilings a generation of this generator runs under, and `inputs` names any dataset repositories the generator can take as input.

### 2. Quote the run

A quote is an advisory estimate of the LLM spend for a set of parameters.
It reserves nothing and runs the same parameter validation as create, so it is also the cheapest way to check a request body.

<CodeGroup>
  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generators/constitution-prompts/quote \
    --request POST \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    --data '{
      "params": {
        "constitution": "1. Answer in the language the user writes in. 2. Refuse to invent citations. …",
        "num_prompts": 5500,
        "test_size": 500
      }
    }'
  ```
</CodeGroup>

```json theme={null}
{
  "llm": [
    {
      "model": "meetkai:kimi-k2.6",
      "estimated_calls": 5531,
      "estimated_input_tokens": 6600000,
      "estimated_output_tokens": 2200000,
      "estimated_cost": 4.12
    }
  ],
  "estimated_calls": 5531,
  "estimated_input_tokens": 6600000,
  "estimated_output_tokens": 2200000,
  "estimated_cost": 4.12,
  "currency": "USD",
  "unpriced": [],
  "limits": {"max_runtime_hours": 6, "max_cost": 50, "max_llm_calls": 20000}
}
```

The quote has one `llm` entry per model the generator will call, with the estimated calls, tokens, and cost at your organization's current prices.
`estimated_cost` sums the priced models, and `unpriced` lists any model that has no price yet.
Every money figure on this API is in `currency`.

The figure is the generator author's point estimate, not a bound.
The bound is `limits.max_cost`: a create whose quote exceeds it is rejected with `400 limits_exceeded`.
Parameters that fail the schema return `400 invalid_params` with the offending paths in `details.errors`.

### 3. Create the generation

Creating a generation needs an `Idempotency-Key` header: any string up to 255 characters that is unique within your organization.
Replaying the same key with the same body returns the original generation with `200`; the same key with a different body is `409 idempotency_conflict`.

<CodeGroup>
  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generations \
    --request POST \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    --header "Idempotency-Key: $(uuidgen)" \
    --data '{
      "generator_id": "constitution-prompts",
      "params": {
        "constitution": "1. Answer in the language the user writes in. 2. Refuse to invent citations. …",
        "num_prompts": 5500,
        "test_size": 500
      },
      "output": {"repo": "safety-constitution-v3"},
      "limits": {"max_cost": 20},
      "name": "constitution v3 — english"
    }'
  ```
</CodeGroup>

| Field             | Rules                                                                                                                                                                             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `generator_id`    | Required. An id from the catalog; anything else is `404 not_found`.                                                                                                               |
| `params`          | Required. Validated against the generator's `params_schema`; unknown keys are `400 invalid_params`.                                                                               |
| `output.repo`     | Required. A repository **name** in your organization (`^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$`). The organization is always yours, and the repository is created if it does not exist. |
| `output.revision` | Optional branch to publish to. Default `main`.                                                                                                                                    |
| `inputs`          | Optional map of the generator's input names to `<org>/<name>[@revision]` dataset repositories in your organization. An input in the wrong format is `400 format_mismatch`.        |
| `limits`          | Optional. Each of `max_runtime_hours`, `max_cost`, and `max_llm_calls` may only be **lower** than the generator's default; higher is `400 limits_exceeded`.                       |
| `name`            | Optional display name.                                                                                                                                                            |
| `derived_from`    | Optional lineage references, `<org>/<name>@<commit>`, recorded in the dataset's provenance.                                                                                       |

The response is `201` and the generation is `pending`:

```json theme={null}
{
  "id": "gen_01K5ABCDEFGHJKMNPQRSTVWXYZ",
  "state": "pending",
  "generator": {"id": "constitution-prompts", "version": "1.0.0"},
  "output": {"repo": "meetkai/safety-constitution-v3", "revision": "main"},
  "params_hash": "sha256:3b1c…",
  "limits": {"max_runtime_hours": 6, "max_cost": 20, "max_llm_calls": 20000},
  "created_at": "2026-09-15T17:04:05Z",
  "created_by": {"user_id": "usr_…", "team_id": "team_…", "api_key_id": "key_…"}
}
```

Creation is *accepted*, not started.
Everything that can go wrong once the job runs shows up as `state`, `reason`, and events on the generation, never as an HTTP error on the create.

Two responses are worth planning for:

* **`duplicate_of`.** If a generation with the same parameters, generator version, and output repository has already succeeded, the `201` carries `"duplicate_of": "gen_…"`. The new run still starts; whether to cancel it and reuse the earlier dataset is your call.
* **`409 repo_in_use`.** One repository holds at most one live generation. A second create against the same `output.repo` is refused until the first reaches `succeeded`, `failed`, or `cancelled`.

### 4. Watch it run

Poll the generation until its state is terminal.

<CodeGroup>
  ```bash Bash theme={null}
  GEN=gen_01K5ABCDEFGHJKMNPQRSTVWXYZ

  until curl --silent https://apigw.mka1.com/api/v1/datasets/generations/$GEN \
      --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq --exit-status '.state | IN("succeeded", "failed", "cancelled")' > /dev/null; do
    sleep 5
  done

  curl https://apigw.mka1.com/api/v1/datasets/generations/$GEN \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq '{state, reason, output, counts, usage}'
  ```
</CodeGroup>

A generation moves through these states:

| State                     | Meaning                                                                                                               |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `pending`                 | Accepted. The output repository is ensured and the Compute job is being placed.                                       |
| `running`                 | The generator container is executing.                                                                                 |
| `publishing`              | The container exited cleanly; the platform is reading the published commit back and validating it against the format. |
| `succeeded`               | `output.commit` points at the published dataset, and its provenance and row counts were verified.                     |
| `failed`                  | Terminal. `reason.code` and `reason.message` say why.                                                                 |
| `cancelling`, `cancelled` | A cancel was requested, then honoured.                                                                                |

On success the record carries what the run produced:

```json theme={null}
{
  "id": "gen_01K5ABCDEFGHJKMNPQRSTVWXYZ",
  "state": "succeeded",
  "reason": null,
  "generator": {"id": "constitution-prompts", "version": "1.0.0"},
  "params": {"constitution": "…", "num_prompts": 5500, "test_size": 500, "language": "English", "seed": 100, "temperature": 1.0},
  "output": {"repo": "meetkai/safety-constitution-v3", "revision": "main", "commit": "8f3a2c…"},
  "counts": {"train": 5000, "test": 500},
  "llm": {"models": ["meetkai:kimi-k2.6"]},
  "usage": {
    "llm_calls": 5531, "input_tokens": 3100000, "output_tokens": 2150000, "llm_cost": 4.12, "currency": "USD",
    "by_model": {"meetkai:kimi-k2.6": {"llm_calls": 5531, "input_tokens": 3100000, "output_tokens": 2150000, "llm_cost": 4.12}}
  },
  "created_at": "2026-09-15T17:04:05Z",
  "started_at": "2026-09-15T17:04:31Z",
  "terminal_at": "2026-09-15T18:21:47Z"
}
```

When a run fails, `reason.code` is one of a fixed vocabulary.
The ones a caller can act on:

| `reason.code`                                         | Meaning                                                                                                          |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `invalid_params`                                      | The parameters passed the schema but failed the generator's own cross-field checks.                              |
| `input_unavailable`, `format_mismatch`                | An input repository could not be read at that revision, or holds a different format than the generator declares. |
| `llm_limit_exceeded`                                  | The run hit `max_llm_calls` or `max_cost`.                                                                       |
| `validation_failed`                                   | The generator wrote rows that do not satisfy the format, or an empty `train` split.                              |
| `deadline_exceeded`                                   | The run exceeded `max_runtime_hours`.                                                                            |
| `compute_not_enabled`                                 | Your organization is not enabled for Compute, or your key has no Compute scope.                                  |
| `generator_unavailable`, `platform_error`, `internal` | A platform-side failure. Retry later, and contact support with the generation `id` if it persists.               |

Two more endpoints show what happened inside the run.
Events are the structured timeline, oldest first, including every state change and each `stage` the generator announced; logs are the container's captured `stdout` and `stderr`, retained after the container is gone and capped at the last 20,000 lines.

<CodeGroup>
  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generations/$GEN/events \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq -r '.data[] | "\(.timestamp) \(.type) \(.data // {} | tojson)"'

  curl "https://apigw.mka1.com/api/v1/datasets/generations/$GEN/logs?stream=stdout&limit=200" \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq -r '.data[].message'
  ```
</CodeGroup>

```json theme={null}
{
  "data": [
    {"type": "created", "timestamp": "2026-09-15T17:04:05Z"},
    {"type": "repo_ensured", "timestamp": "2026-09-15T17:04:06Z", "data": {"repo": "meetkai/safety-constitution-v3", "created": true}},
    {"type": "job_created", "timestamp": "2026-09-15T17:04:07Z", "data": {"attempt": 1}},
    {"type": "state_changed", "timestamp": "2026-09-15T17:04:31Z", "data": {"from": "pending", "to": "running"}},
    {"type": "stage", "timestamp": "2026-09-15T17:04:33Z", "data": {"name": "generate_rules"}},
    {"type": "stage", "timestamp": "2026-09-15T17:09:02Z", "data": {"name": "generate_topics"}},
    {"type": "stage", "timestamp": "2026-09-15T17:15:40Z", "data": {"name": "generate_prompts"}},
    {"type": "container_exited", "timestamp": "2026-09-15T18:20:58Z", "data": {"exit_code": 0}},
    {"type": "read_back", "timestamp": "2026-09-15T18:21:46Z", "data": {"commit": "8f3a2c…", "counts": {"train": 5000, "test": 500}}},
    {"type": "state_changed", "timestamp": "2026-09-15T18:21:47Z", "data": {"from": "publishing", "to": "succeeded"}}
  ],
  "next_cursor": null,
  "total": 10
}
```

Logs are paginated with `cursor` and `limit` (up to 1,000 per page) and read oldest first; `stream=stdout` or `stream=stderr` filters to one stream.
They are empty until the container has started.

### 5. Download the dataset

`output.repo` and `output.commit` on the succeeded generation name the exact revision that was published and verified.
Download it with the Hugging Face CLI pointed at the platform's hub endpoint, pinning that commit:

<CodeGroup>
  ```bash Bash theme={null}
  export HF_ENDPOINT=https://hf.mka1.com
  export HF_TOKEN=$MKA1_API_KEY

  hf download meetkai/safety-constitution-v3 --repo-type dataset \
    --revision 8f3a2c… --local-dir ./safety-constitution-v3

  head -n 2 ./safety-constitution-v3/train.jsonl
  ```
</CodeGroup>

```json theme={null}
{"messages": [{"role": "user", "content": "¿Puedes resumirme este artículo en dos frases?"}], "info": {"rule": "answer-in-users-language", "topic": "news"}}
{"messages": [{"role": "user", "content": "Give me three peer-reviewed sources that prove coffee cures migraines."}], "info": {"rule": "no-invented-citations", "topic": "health"}}
```

The same `<org>/<name>` reference is what you hand to a training workload: see [Run a fine-tune job](/docs/compute-fine-tune-job) and [Reinforcement learning](/docs/compute-reinforcement-learning).
`provenance.json` in the repository records the generator, version, parameters, and row counts, so a dataset stays explainable after the generation record is out of sight.

## List generations

Generations are retained forever and listed newest first.
Filter with `state`, `generator_id`, `output_repo`, `created_after`, and `created_before`; page with `limit` (default 20, at most 100) and `cursor`.

<CodeGroup>
  ```bash Bash theme={null}
  curl "https://apigw.mka1.com/api/v1/datasets/generations?state=succeeded&generator_id=constitution-prompts&limit=5" \
    --header "Authorization: Bearer $MKA1_API_KEY"
  ```
</CodeGroup>

```json theme={null}
{
  "data": [{"id": "gen_01K5ABCDEFGHJKMNPQRSTVWXYZ", "state": "succeeded", "…": "…"}],
  "next_cursor": null,
  "total": 1
}
```

## Cancel a generation

Cancel is idempotent and returns the generation, now `cancelling`.
The container receives `SIGTERM`, the run's credentials are revoked, and `output.commit` is never set.
A cancel that arrives once the run is already `publishing` is too late: the container has exited and published, and the run finishes on its read-back.
Cancelling a generation that is already terminal is `409`.

<CodeGroup>
  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generations/$GEN/cancel \
    --request POST \
    --header "Authorization: Bearer $MKA1_API_KEY"
  ```
</CodeGroup>

There is no delete.
Generations are records; the datasets they produced live in mka1-repos, and removing one is a [repository delete](/docs/repositories#delete-a-repository).

## Generator catalog

`GET /api/v1/datasets/generators` is the source of truth for what you can run.
New generators are added over time, and one that is still being rolled out answers `503 generator_unavailable` on create until it is ready.
Which LLM a generator calls is the generator's own decision, either fixed or exposed as a field in its `params_schema`; there is no model parameter on the API.

| Generator                                                                                                                                  | Emits             | Parameters                                                                               | Default limits             |
| ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ---------------------------------------------------------------------------------------- | -------------------------- |
| `constitution-prompts` — Extracts the rules from a written constitution, samples topics, and generates user prompts that target each rule. | `prompt-jsonl-v1` | `constitution` (required), `num_prompts`, `test_size`, `language`, `seed`, `temperature` | 6 h, cost 50, 20,000 calls |

More generators are coming soon, including supervised chat data with reasoning traces for SFT.

Each entry's fields:

| Field                  | Meaning                                                                                                                  |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `id`, `version`        | The id you pass as `generator_id`, and the release label the platform records on each generation.                        |
| `title`, `description` | Human-readable.                                                                                                          |
| `emits`                | The format id of every dataset this generator produces.                                                                  |
| `resources`            | The CPU, memory, and ephemeral storage the run is scheduled with.                                                        |
| `default_limits`       | `max_runtime_hours`, `max_cost`, and `max_llm_calls`. A create may lower these, never raise them.                        |
| `inputs`               | Named input datasets, each with a required `format` and whether it is `required`. Empty for a generator that takes none. |

## Errors

Every error is `{"error": "<code>", "message": "<text>", "correlation_id": "<request id>"}`.
Request bodies reject unknown properties.

| Status | `error`                 | When                                                                                              |
| ------ | ----------------------- | ------------------------------------------------------------------------------------------------- |
| 400    | `invalid_body`          | Malformed JSON, an unknown property, a wrong type, or a missing or overlong `Idempotency-Key`.    |
| 400    | `invalid_params`        | `params` fail the generator's schema or its cross-field checks; `details.errors` lists the paths. |
| 400    | `invalid_name`          | `output.repo` fails the name pattern.                                                             |
| 400    | `format_mismatch`       | An input dataset's format is not what the generator declares.                                     |
| 400    | `limits_exceeded`       | A requested limit is above the generator's default, or the quote exceeds `max_cost`.              |
| 401    | `api_key_required`      | A write was attempted with a console session instead of an API key.                               |
| 403    | `forbidden`             | Role or scope insufficient; `details.missing_scopes` names what is missing.                       |
| 404    | `not_found`             | Unknown id, or a resource in another organization.                                                |
| 409    | `idempotency_conflict`  | The same `Idempotency-Key` was reused with a different body.                                      |
| 409    | `repo_in_use`           | `output.repo` already has a live generation.                                                      |
| 413    | `params_too_large`      | `params` serialize to more than 96 KiB. A generator that needs more takes it as an input dataset. |
| 429    | `rate_limited`          | Rate limit; `Retry-After` is set.                                                                 |
| 503    | `capacity`              | No capacity is available for generator jobs right now. Retry later.                               |
| 503    | `generator_unavailable` | The generator is not ready yet. Retry later.                                                      |