> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mka1.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> <Badge color="blue" shape="pill">Early access</Badge>

Generate training datasets with the platform's generators — quote the run, create a generation, watch it publish to mka1-repos, and download the result with the Hugging Face CLI.


The Datasets API turns a generator and a set of parameters into a training dataset that lands in one of your [mka1-repos](/docs/repositories) repositories.
You never write rows yourself: you pick a generator from the catalog, quote the run, create a *generation*, and download the published dataset when it succeeds.

Three nouns cover the whole API:

| Noun | What it is | Where it comes from |
| - | - | - |
| Generator | A packaged program plus a JSON Schema for its parameters. It emits exactly one dataset format. | Provided by the platform. `GET /generators` lists what is available. |
| Generation | One run of a generator with one set of parameters. It publishes a dataset repository and is retained forever as a record. | `POST /generations`. |
| Format | The row shape and repository layout a dataset follows, such as `chat-jsonl-v1`. Training workloads read these directly. | `GET /formats`. |

Every generation runs as a [Compute](/docs/compute-resources) job under your organization and publishes under a short-lived credential the platform mints for that run, so the hardware and any LLM calls the generator makes bill to you like any other platform usage.

## Before you start

You need:

| Requirement | Notes |
| - | - |
| API key or console session | Send a key as `Authorization: Bearer <mka1-api-key>`. Reads need the `read:datasets` scope; creating or cancelling a generation needs `write:datasets`. A console session passes on its role alone. Creating a generation also needs the scopes the run itself uses (`read:compute`, `write:compute`, `read:repos`, `write:repos`, and `write:responses` for a generator that calls an LLM): a key lacking one is refused with `403 insufficient_scope`. The run never holds your key. It publishes under a scope-limited credential minted for that generation, which expires with the run and is revoked when it ends. |
| Org owner or admin role | Every Datasets endpoint is restricted to owners and admins of the organization. |
| Compute enabled for your org | Generations run as Compute jobs in your organization. If your org is not enabled, the generation is accepted and then fails with `reason.code: compute_not_enabled`. |
| `hf` CLI | The published dataset is an ordinary repository; the [Hugging Face CLI](/docs/repositories#push-and-pull-with-the-hugging-face-cli) downloads it. |

Everything you create belongs to your organization.
A generation from another organization is `404`, never `403`.

The examples export the key once:

```bash theme={null}
export MKA1_API_KEY=<mka1-api-key>
```

The CLI tabs assume `MKA1_BEARER_AUTH="Bearer <mka1-api-key>"` is exported; see [authenticate the CLI](/docs/cli/authentication).

## Dataset formats

A dataset is a repository with `train.jsonl` at its root and, optionally, `test.jsonl` and `validation.jsonl`: UTF-8, one JSON object per line.
Alongside the rows, the platform writes a `README.md` with an `mka1:` front-matter block and a `provenance.json` that records the generator, its version, the exact parameters, row counts, and LLM usage.
Every row in a split has the same format, and the format is what a trainer declares it accepts.

All three formats share the OpenAI-style message object:

```json theme={null}
{"role": "system" | "user" | "assistant" | "tool", "content": "…", "name": "…", "tool_calls": [], "tool_call_id": "…", "reasoning": "…"}
```

`reasoning` is an MKA1 extension for assistant turns that carries a thinking trace apart from `content`; trainers that do not understand it ignore it.

| Format | Trains | One row |
| - | - | - |
| `chat-jsonl-v1` | SFT, LoRA | `{"messages": [{"role": "user", "content": "…"}, {"role": "assistant", "content": "…"}]}` — at least one assistant turn, and the last message is the assistant's. Optional `id` and `meta` keys. The `messages` shape that ms-swift, TRL, and axolotl read natively. |
| `prompt-jsonl-v1` | GRPO, RFT | `{"messages": [{"role": "user", "content": "…"}], "answer": "…", "info": {}}` — no assistant turn. `answer` and `info` are optional pass-throughs for reference-answer or rubric rewards. Rule-judged datasets also ship a `judge/` directory. |
| `preference-jsonl-v1` | DPO, ORPO | `{"input": [...], "preferred_output": [...], "non_preferred_output": [...]}` — OpenAI's spelling; the format's `trl` converter rewrites it as `prompt`/`chosen`/`rejected` for TRL and ms-swift. Each output is exactly one assistant message. |

The catalog and each format's row schema are served by the API, so a trainer can validate a dataset before spending GPU time:

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  import { SDK } from '@meetkai/mka1';

  const mka1 = new SDK({ bearerAuth: 'Bearer <mka1-api-key>' });

  const formats = await mka1.datasets.listFormats({
    xOnBehalfOf: '<end-user-id>', // optional — attribute the request to one of your end users
  });
  console.log(formats.data.map((f) => f.id));

  const format = await mka1.datasets.getFormat({ formatId: 'prompt-jsonl-v1' });
  console.log(format.layout, format.semantics);
  ```

  ```python Python SDK theme={null}
  from meetkai_mka1 import SDK

  sdk = SDK(bearer_auth="Bearer <mka1-api-key>")

  formats = sdk.datasets.list_formats(
      x_on_behalf_of="<end-user-id>",  # optional — attribute the request to one of your end users
  )
  print([f.id for f in formats.data])

  fmt = sdk.datasets.get_format(format_id="prompt-jsonl-v1")
  print(fmt.layout, fmt.semantics)
  ```

  ```csharp C# SDK theme={null}
  using MeetKai.MKA1;
  using MeetKai.MKA1.Types.Components;

  var sdk = new SDK(bearerAuth: "Bearer <mka1-api-key>");

  var formats = await sdk.Datasets.ListFormatsAsync(
      xOnBehalfOf: "<end-user-id>" // optional — attribute the request to one of your end users
  );
  foreach (var f in formats.FormatList!.Data) Console.WriteLine(f.Id);

  var format = await sdk.Datasets.GetFormatAsync(formatId: "prompt-jsonl-v1");
  Console.WriteLine(format.FormatDetail!.Converters.Count);
  ```

  ```bash CLI theme={null}
  mka1 datasets list-formats -H 'X-On-Behalf-Of: <end-user-id>'

  mka1 datasets get-format --format-id prompt-jsonl-v1
  ```

  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/formats \
    --header "Authorization: Bearer $MKA1_API_KEY"

  curl https://apigw.mka1.com/api/v1/datasets/formats/prompt-jsonl-v1 \
    --header "Authorization: Bearer $MKA1_API_KEY"
  ```
</CodeGroup>

```json theme={null}
{
  "data": [
    {"id": "chat-jsonl-v1", "title": "Chat (SFT)", "trains": ["sft", "lora"]},
    {"id": "prompt-jsonl-v1", "title": "Prompt-only (RL)", "trains": ["grpo", "rft"]},
    {"id": "preference-jsonl-v1", "title": "Pairwise preference", "trains": ["dpo", "orpo"]}
  ]
}
```

A single format adds `row_schema` (JSON Schema for one line), `layout` (required, optional, and platform-written files), `semantics` (the checks beyond schema, each with a `code` and a `severity`), and `converters`:

```json theme={null}
{
  "id": "prompt-jsonl-v1",
  "title": "Prompt-only (RL)",
  "trains": ["grpo", "rft"],
  "row_schema": {"$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "required": ["messages"], "…": "…"},
  "layout": {
    "id": "prompt-jsonl-v1",
    "required_files": ["train*.jsonl"],
    "published_files": ["README.md", "provenance.json"],
    "optional_files": ["test*.jsonl", "validation*.jsonl", "judge/", "artifacts/", ".gitattributes"],
    "requires_judge": false,
    "trains": ["grpo", "rft"]
  },
  "semantics": [
    {"code": "empty_content", "severity": "error", "description": "every message has non-empty content, except an assistant turn carrying `tool_calls`"},
    {"code": "empty_system_content", "severity": "warning", "description": "a system turn with no content"},
    {"code": "assistant_turn_in_prompt_only", "severity": "error", "description": "no assistant turn in a prompt-only row"}
  ],
  "converters": ["trl"]
}
```

## Generate a dataset

The walkthrough uses `constitution-prompts`, which turns a written constitution into a prompt-only dataset for reinforcement learning: it extracts the rules, samples topics for each rule, and generates user prompts that target each rule, along with the judge templates that score an answer against them.
Every generator follows the same five steps; only the parameters change.

### 1. Discover a generator

List the available generators, then read one to get its parameter schema.

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  const generators = await mka1.datasets.listGenerators({});
  console.log(generators.data.map((g) => `${g.id}@${g.version} → ${g.emits}`));

  const generator = await mka1.datasets.getGenerator({ generatorId: 'constitution-prompts' });
  console.log(generator.paramsSchema, generator.exampleParams);
  ```

  ```python Python SDK theme={null}
  generators = sdk.datasets.list_generators()
  print([f"{g.id}@{g.version} → {g.emits}" for g in generators.data])

  generator = sdk.datasets.get_generator(generator_id="constitution-prompts")
  print(generator.params_schema, generator.example_params)
  ```

  ```csharp C# SDK theme={null}
  var generators = await sdk.Datasets.ListGeneratorsAsync();
  foreach (var g in generators.GeneratorList!.Data) Console.WriteLine($"{g.Id}@{g.Version} → {g.Emits}");

  var generator = await sdk.Datasets.GetGeneratorAsync(generatorId: "constitution-prompts");
  Console.WriteLine(generator.GeneratorDetail!.DefaultLimits.MaxLlmCalls);
  ```

  ```bash CLI theme={null}
  mka1 datasets list-generators

  mka1 datasets get-generator --generator-id constitution-prompts
  ```

  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generators \
    --header "Authorization: Bearer $MKA1_API_KEY"

  curl https://apigw.mka1.com/api/v1/datasets/generators/constitution-prompts \
    --header "Authorization: Bearer $MKA1_API_KEY"
  ```
</CodeGroup>

The list is not paginated.
Reading a single generator adds `params_schema` (JSON Schema, draft 2020-12) and `example_params`, a small set of parameters the author guarantees to be valid:

```json theme={null}
{
  "id": "constitution-prompts",
  "version": "1.0.2",
  "title": "Constitution → RL prompts",
  "description": "Extracts rules from a constitution, samples topics per rule, and writes rule-targeted user prompts with the judge templates to score answers against.",
  "emits": "prompt-jsonl-v1",
  "inputs": {},
  "resources": {"cpu": "2", "memory": "4Gi", "ephemeral_storage": "20Gi"},
  "default_limits": {"max_runtime_hours": 6.0, "max_llm_calls": 20000},
  "params_schema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["constitution", "model"],
    "properties": {
      "constitution": {"type": "string", "minLength": 1, "description": "The constitution text the rules are extracted from."},
      "model": {"type": "string", "minLength": 1, "description": "Gateway model id used by every stage; `auto` takes the org's default."},
      "rules_model": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "Model for the rules stage."},
      "topics_model": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "Model for the topics stage."},
      "prompts_model": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "Model for the prompts stage."},
      "max_rules": {"type": "integer", "default": -1},
      "num_topics": {"type": "integer", "default": 30, "minimum": 1},
      "size": {"type": "integer", "default": 1, "minimum": 1},
      "num_prompts": {"type": "integer", "default": 5500, "minimum": 1},
      "test_size": {"type": "integer", "default": 500, "minimum": 0},
      "language": {"type": "string", "default": "English", "minLength": 1},
      "seed": {"type": "integer", "default": 100},
      "rules_temperature": {"type": "number", "default": 0.2, "minimum": 0, "maximum": 2},
      "topics_temperature": {"type": "number", "default": 1.0, "minimum": 0, "maximum": 2},
      "prompts_temperature": {"type": "number", "default": 1.0, "minimum": 0, "maximum": 2}
    }
  },
  "example_params": {
    "constitution": "Be honest about uncertainty. Never disparage a nationality.",
    "model": "auto",
    "max_rules": 4,
    "num_topics": 5,
    "num_prompts": 20,
    "test_size": 4
  }
}
```

`version` is a label the platform records on every generation and in the dataset's provenance; you do not choose it.
`default_limits` are the ceilings a generation of this generator runs under, and `inputs` names any dataset repositories the generator can take as input.

A generator that calls an LLM exposes the model as a parameter.
`model` is the id you would send to `/responses`, and `auto` takes your organization's default; the per-stage `rules_model`, `topics_model`, and `prompts_model` override it for one stage.
The walkthrough runs the author's `example_params`, which is a twenty-prompt run; the defaults produce a 5,500-prompt dataset.

### 2. Quote the run

A quote is an advisory estimate of the LLM spend for a set of parameters.
It reserves nothing and runs the same parameter validation as create, so it is also the cheapest way to check a request body.

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  const params = {
    constitution: 'Be honest about uncertainty. Never disparage a nationality.',
    model: 'auto',
    max_rules: 4,
    num_topics: 5,
    num_prompts: 20,
    test_size: 4,
  };

  const quote = await mka1.datasets.quoteGenerator({
    generatorId: 'constitution-prompts',
    quoteRequest: { params },
  });
  console.log(quote.estimatedCalls, quote.estimatedCost, quote.unpriced);
  ```

  ```python Python SDK theme={null}
  params = {
      "constitution": "Be honest about uncertainty. Never disparage a nationality.",
      "model": "auto",
      "max_rules": 4,
      "num_topics": 5,
      "num_prompts": 20,
      "test_size": 4,
  }

  quote = sdk.datasets.quote_generator(generator_id="constitution-prompts", params=params)
  print(quote.estimated_calls, quote.estimated_cost, quote.unpriced)
  ```

  ```csharp C# SDK theme={null}
  var params_ = new Dictionary<string, object>
  {
      { "constitution", "Be honest about uncertainty. Never disparage a nationality." },
      { "model", "auto" },
      { "max_rules", 4 },
      { "num_topics", 5 },
      { "num_prompts", 20 },
      { "test_size", 4 },
  };

  var quote = await sdk.Datasets.QuoteGeneratorAsync(
      generatorId: "constitution-prompts",
      body: new QuoteRequest { Params = params_ }
  );
  var q = quote.QuoteResponse!;
  Console.WriteLine($"{q.EstimatedCalls} calls, cost {q.EstimatedCost?.ToString() ?? "unpriced"}");
  ```

  ```bash CLI theme={null}
  mka1 datasets quote-generator --generator-id constitution-prompts --body '{
    "params": {
      "constitution": "Be honest about uncertainty. Never disparage a nationality.",
      "model": "auto",
      "max_rules": 4,
      "num_topics": 5,
      "num_prompts": 20,
      "test_size": 4
    }
  }'
  ```

  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generators/constitution-prompts/quote \
    --request POST \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    --data '{
      "params": {
        "constitution": "Be honest about uncertainty. Never disparage a nationality.",
        "model": "auto",
        "max_rules": 4,
        "num_topics": 5,
        "num_prompts": 20,
        "test_size": 4
      }
    }'
  ```
</CodeGroup>

```json theme={null}
{
  "llm": [
    {
      "model": "auto",
      "estimated_calls": 25,
      "estimated_input_tokens": 19714,
      "estimated_output_tokens": 3240,
      "estimated_cost": null
    }
  ],
  "estimated_calls": 25,
  "estimated_input_tokens": 19714,
  "estimated_output_tokens": 3240,
  "estimated_cost": null,
  "currency": "USD",
  "unpriced": ["auto"],
  "limits": {"max_runtime_hours": 6.0, "max_llm_calls": 20000}
}
```

The quote has one `llm` entry per model the generator will call, with the estimated calls and tokens, and the cost at your organization's current prices.
Every money figure on this API is in `currency`.

A model that has no price for your organization gets `estimated_cost: null` and is listed in `unpriced`, as `auto` is above; when nothing in the estimate is priced, the total is `null` too, never `0`.
An unpriced model does not block the create: `POST /generations` admits it.
The quote is where you see what was not priced; if you want a figure for your model, ask your organization admin to price it (see [Manage models](/docs/models#step-6---check-effective-prices)).

The priced figure is the generator author's point estimate, not a bound.
What bounds a run is `limits.max_llm_calls` and `limits.max_runtime_hours` on the create, and your organization's [budgets](/docs/budgets), which cap the spend of every billable service.
Parameters that fail the schema return `400 invalid_params` with the offending paths in `details.errors`:

```json theme={null}
{
  "error": "invalid_params",
  "message": "params are invalid",
  "correlation_id": "c1a2568e-7fe9-4883-b409-ec04700da322#8517",
  "details": {"errors": [{"path": "bogus", "message": "Extra inputs are not permitted"}]}
}
```

### 3. Create the generation

Creating a generation needs an `Idempotency-Key` header: any string up to 255 characters that is unique within your organization.
Replaying the same key with the same body returns the original generation; the same key with a different body is `409 idempotency_conflict`.

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  const created = await mka1.datasets.createGeneration({
    idempotencyKey: crypto.randomUUID(),
    createGeneration: {
      generatorId: 'constitution-prompts',
      params,
      output: { repo: 'constitution-prompts-demo' },
      limits: { maxLlmCalls: 5000 },
      name: 'constitution prompts — demo',
    },
  });
  console.log(created.id, created.state);
  ```

  ```python Python SDK theme={null}
  import uuid

  created = sdk.datasets.create_generation(
      idempotency_key=str(uuid.uuid4()),
      generator_id="constitution-prompts",
      params=params,
      output={"repo": "constitution-prompts-demo"},
      limits={"max_llm_calls": 5000},
      name="constitution prompts — demo",
  )
  print(created.id, created.state)
  ```

  ```csharp C# SDK theme={null}
  var created = await sdk.Datasets.CreateGenerationAsync(
      idempotencyKey: Guid.NewGuid().ToString(),
      body: new CreateGeneration
      {
          GeneratorId = "constitution-prompts",
          Params = params_,
          Output = new OutputSpec { Repo = "constitution-prompts-demo" },
          Limits = new LimitsRequest { MaxLlmCalls = 5000 },
          Name = "constitution prompts — demo",
      }
  );
  var gen = created.GenerationCreated!;
  Console.WriteLine($"{gen.Id} {gen.State}");
  ```

  ```bash CLI theme={null}
  mka1 datasets create-generation --idempotency-key "$(uuidgen)" --body '{
    "generator_id": "constitution-prompts",
    "params": {
      "constitution": "Be honest about uncertainty. Never disparage a nationality.",
      "model": "auto",
      "max_rules": 4,
      "num_topics": 5,
      "num_prompts": 20,
      "test_size": 4
    },
    "output": {"repo": "constitution-prompts-demo"},
    "limits": {"max_llm_calls": 5000},
    "name": "constitution prompts — demo"
  }'
  ```

  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generations \
    --request POST \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    --header "Idempotency-Key: $(uuidgen)" \
    --data '{
      "generator_id": "constitution-prompts",
      "params": {
        "constitution": "Be honest about uncertainty. Never disparage a nationality.",
        "model": "auto",
        "max_rules": 4,
        "num_topics": 5,
        "num_prompts": 20,
        "test_size": 4
      },
      "output": {"repo": "constitution-prompts-demo"},
      "limits": {"max_llm_calls": 5000},
      "name": "constitution prompts — demo"
    }'
  ```
</CodeGroup>

| Field | Rules |
| - | - |
| `generator_id` | Required. An id from the catalog; anything else is `404 not_found`. |
| `params` | Required. Validated against the generator's `params_schema`; unknown keys are `400 invalid_params`. |
| `output.repo` | Required. A repository **name** in your organization (`^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$`). The organization is always yours, and the repository is created if it does not exist. |
| `output.revision` | Optional branch to publish to. Default `main`. |
| `inputs` | Optional map of the generator's input names to `<org>/<name>[@revision]` dataset repositories in your organization. A reference that is not in that form, or names another organization, is `400 invalid_params`; a repository or revision that does not exist is `404 not_found` with `details.input`; an input in the wrong format is `400 format_mismatch`. |
| `limits` | Optional. Each of `max_runtime_hours` and `max_llm_calls` may only be **lower** than the generator's default; higher is `400 limits_exceeded`, with `details.limit`, `details.requested`, and `details.maximum`. |
| `name` | Optional display name. |
| `derived_from` | Optional lineage references, `<org>/<name>@<commit>`, recorded in the dataset's provenance. |

The response is `201` and the generation is `pending`:

```json theme={null}
{
  "id": "gen_01m32ssjgpegaegr5bmjf8znc9",
  "state": "pending",
  "generator": {"id": "constitution-prompts", "version": "1.0.1"},
  "output": {"repo": "meetkai/constitution-prompts-demo", "revision": "main"},
  "params_hash": "sha256:a67196d3d68b0c4544c6123aac528be97486de31ff0c96734cf9bf407c9ff622",
  "limits": {"max_runtime_hours": 6.0, "max_llm_calls": 5000},
  "created_at": "2026-09-21T20:16:22Z",
  "created_by": {"user_id": "9PbDp69L…", "team_id": "cmu4arrcv…", "api_key_id": "uTHhbNBn…", "delegation_id": "…", "external_user_id": null}
}
```

`created_by.api_key_id` is the key you presented (`null` from a console session), `delegation_id` identifies the credential minted for the run, and `external_user_id` is the end user the request was made for when it carried `X-On-Behalf-Of`, so a run stays attributable to that user.

Creation is *accepted*, not started.
Everything that can go wrong once the job runs shows up as `state`, `reason`, and events on the generation, never as an HTTP error on the create.

Two responses are worth planning for:

* **`duplicate_of`.** If a generation with the same parameters, generator version, and output repository has already succeeded, the `201` carries `"duplicate_of": "gen_…"`. The new run still starts; whether to cancel it and reuse the earlier dataset is your call.
* **`409 repo_in_use`.** One repository holds at most one live generation. A second create against the same `output.repo` is refused until the first reaches `succeeded`, `failed`, or `cancelled`.

### 4. Watch it run

Poll the generation until its state is terminal.

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  const terminal = new Set(['succeeded', 'failed', 'cancelled']);
  let generation = await mka1.datasets.getGeneration({ generationId: created.id });
  while (!terminal.has(generation.state)) {
    await new Promise((r) => setTimeout(r, 5000));
    generation = await mka1.datasets.getGeneration({ generationId: created.id });
  }
  console.log(generation.state, generation.reason, generation.output.commit, generation.counts, generation.usage);
  ```

  ```python Python SDK theme={null}
  import time

  generation = sdk.datasets.get_generation(generation_id=created.id)
  while generation.state not in ("succeeded", "failed", "cancelled"):
      time.sleep(5)
      generation = sdk.datasets.get_generation(generation_id=created.id)
  print(generation.state, generation.reason, generation.output.commit, generation.counts, generation.usage)
  ```

  ```csharp C# SDK theme={null}
  var terminal = new HashSet<string> { "succeeded", "failed", "cancelled" };
  var generation = (await sdk.Datasets.GetGenerationAsync(generationId: gen.Id)).Generation!;
  while (!terminal.Contains(generation.State))
  {
      await Task.Delay(TimeSpan.FromSeconds(5));
      generation = (await sdk.Datasets.GetGenerationAsync(generationId: gen.Id)).Generation!;
  }
  Console.WriteLine($"{generation.State} {generation.Output.Commit}");
  ```

  ```bash CLI theme={null}
  GEN=gen_01m32ssjgpegaegr5bmjf8znc9

  until mka1 datasets get-generation --generation-id $GEN --output-format json \
    | jq --exit-status '.state | IN("succeeded", "failed", "cancelled")' > /dev/null; do
    sleep 5
  done

  mka1 datasets get-generation --generation-id $GEN --output-format json \
    | jq '{state, reason, output, counts, usage}'
  ```

  ```bash Bash theme={null}
  GEN=gen_01m32ssjgpegaegr5bmjf8znc9

  until curl --silent https://apigw.mka1.com/api/v1/datasets/generations/$GEN \
      --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq --exit-status '.state | IN("succeeded", "failed", "cancelled")' > /dev/null; do
    sleep 5
  done

  curl https://apigw.mka1.com/api/v1/datasets/generations/$GEN \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq '{state, reason, output, counts, usage}'
  ```
</CodeGroup>

A generation moves through these states:

| State | Meaning |
| - | - |
| `pending` | Accepted. The output repository is ensured and the Compute job is being placed. |
| `running` | The generator container is executing. |
| `publishing` | The container exited cleanly; the platform is reading the published commit back and validating it against the format. |
| `succeeded` | `output.commit` points at the published dataset, and its provenance and row counts were verified. |
| `failed` | Terminal. `reason.code` and `reason.message` say why. |
| `cancelling`, `cancelled` | A cancel was requested, then honoured. |

On success the record carries what the run produced.
The example run took a minute and forty seconds from create to `succeeded`:

```json theme={null}
{
  "id": "gen_01m32ssjgpegaegr5bmjf8znc9",
  "state": "succeeded",
  "reason": null,
  "generator": {"id": "constitution-prompts", "version": "1.0.1"},
  "params": {"constitution": "Be honest about uncertainty. Never disparage a nationality.", "model": "openai:gpt-5-nano", "max_rules": 4, "num_topics": 5, "num_prompts": 20, "test_size": 4},
  "params_hash": "sha256:a67196d3d68b0c4544c6123aac528be97486de31ff0c96734cf9bf407c9ff622",
  "output": {"repo": "meetkai/constitution-prompts-demo", "revision": "main", "commit": "5854e4620043474fcb7b4934cbb71c4d4e366dda"},
  "inputs": {},
  "counts": {"train": 16, "test": 4, "dropped": 0},
  "llm": {"models": ["openai:gpt-5-nano"]},
  "limits": {"max_runtime_hours": 6.0, "max_llm_calls": 5000},
  "usage": {
    "calls": 23,
    "input_tokens": 22488,
    "output_tokens": 38069,
    "by_model": {"openai:gpt-5-nano": {"calls": 23, "input_tokens": 22488, "output_tokens": 38069}}
  },
  "job": {"name": "gen-gen_01m32ssjgpegaegr5bmjf8znc9-1", "attempt": 1},
  "name": "constitution prompts — demo",
  "derived_from": [],
  "created_by": {"user_id": "9PbDp69L…", "team_id": "cmu4arrcv…", "api_key_id": "uTHhbNBn…", "delegation_id": "…", "external_user_id": null},
  "created_at": "2026-09-21T20:16:22Z",
  "started_at": "2026-09-21T20:16:53Z",
  "terminal_at": "2026-09-21T20:18:02Z"
}
```

`params` are exactly what the request sent; this run named its model explicitly, and a run created with `auto` records `auto` in `params`, `llm.models`, and `usage.by_model`.
`counts` reports every split the generator wrote plus `dropped`, the rows it generated and then discarded.
`usage` is the LLM traffic the run made, grouped `by_model`; the spend it represents is on your organization's usage reports, rated at the price of each model.

When a run fails, `reason.code` is one of a fixed vocabulary.
The ones a caller can act on:

| `reason.code` | Meaning |
| - | - |
| `invalid_params` | The parameters passed the schema but failed the generator's own cross-field checks. |
| `input_unavailable`, `format_mismatch` | An input repository could not be read at that revision, or holds a different format than the generator declares. |
| `llm_limit_exceeded` | The run hit `max_llm_calls`. |
| `validation_failed` | The generator wrote rows that do not satisfy the format, or an empty `train` split. |
| `deadline_exceeded` | The run exceeded `max_runtime_hours`. |
| `compute_not_enabled` | Your organization is not enabled for Compute, or your key has no Compute scope. |
| `generator_unavailable`, `platform_error`, `internal` | A platform-side failure. Retry later, and contact support with the generation `id` if it persists. |

Two more endpoints show what happened inside the run.
Events are the structured timeline, in the order they were recorded, including every state change and each `stage` the generator announced; logs are the container's captured `stdout` and `stderr`, retained after the container is gone and capped at the last 20,000 lines.

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  const events = await mka1.datasets.getGenerationEvents({ generationId: created.id });
  for (const e of events.data) console.log(e.timestamp, e.type, e.data);

  const logs = await mka1.datasets.getGenerationLogs({ generationId: created.id, stream: 'stdout', limit: 200 });
  for (const line of logs.data) console.log(line.message);
  ```

  ```python Python SDK theme={null}
  events = sdk.datasets.get_generation_events(generation_id=created.id)
  for e in events.data:
      print(e.timestamp, e.type, e.data)

  logs = sdk.datasets.get_generation_logs(generation_id=created.id, stream="stdout", limit=200)
  for line in logs.data:
      print(line.message)
  ```

  ```csharp C# SDK theme={null}
  using MeetKai.MKA1.Types.Requests;

  var events = await sdk.Datasets.GetGenerationEventsAsync(generationId: gen.Id);
  foreach (var e in events.EventList!.Data) Console.WriteLine($"{e.Timestamp:o} {e.Type}");

  var logs = await sdk.Datasets.GetGenerationLogsAsync(new GetLogsApiV1DatasetsGenerationsGenerationIdLogsGetRequest
  {
      GenerationId = gen.Id,
      Stream = MeetKai.MKA1.Types.Requests.Stream.Stdout,
      Limit = 200,
  });
  foreach (var line in logs.PageLogLine!.Data) Console.WriteLine(line.Message);
  ```

  ```bash CLI theme={null}
  mka1 datasets get-generation-events --generation-id $GEN --output-format json \
    | jq -r '.data[] | "\(.timestamp) \(.type) \(.data // {} | tojson)"'

  mka1 datasets get-generation-logs --generation-id $GEN --stream stdout --limit 200 --output-format json \
    | jq -r '.data[].message'
  ```

  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generations/$GEN/events \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq -r '.data[] | "\(.timestamp) \(.type) \(.data // {} | tojson)"'

  curl "https://apigw.mka1.com/api/v1/datasets/generations/$GEN/logs?stream=stdout&limit=200" \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    | jq -r '.data[].message'
  ```
</CodeGroup>

```json theme={null}
{
  "data": [
    {"type": "created", "timestamp": "2026-09-21T20:16:22.042340Z", "data": null},
    {"type": "repo_ensured", "timestamp": "2026-09-21T20:16:22.525954Z", "data": {"repo": "meetkai/constitution-prompts-demo", "created": true}},
    {"type": "job_created", "timestamp": "2026-09-21T20:16:22.947576Z", "data": {"attempt": 1, "job_name": "gen-gen_01m32ssjgpegaegr5bmjf8znc9-1"}},
    {"type": "stage", "timestamp": "2026-09-21T20:16:36.504999Z", "data": {"name": "generate_rules"}},
    {"type": "state_changed", "timestamp": "2026-09-21T20:16:53.120395Z", "data": {"from": "pending", "to": "running"}},
    {"type": "container_started", "timestamp": "2026-09-21T20:16:43.666262Z", "data": null},
    {"type": "stage", "timestamp": "2026-09-21T20:16:54.641733Z", "data": {"name": "generate_topics"}},
    {"type": "stage", "timestamp": "2026-09-21T20:17:04.939766Z", "data": {"name": "generate_prompts"}},
    {"type": "stage", "timestamp": "2026-09-21T20:17:23.136013Z", "data": {"name": "write_outputs"}},
    {"type": "container_exited", "timestamp": "2026-09-21T20:18:01.564086Z", "data": {"exit_code": 0}},
    {"type": "state_changed", "timestamp": "2026-09-21T20:18:01.640262Z", "data": {"from": "running", "to": "publishing"}},
    {"type": "read_back", "timestamp": "2026-09-21T20:18:02.135218Z", "data": {"commit": "5854e4620043474fcb7b4934cbb71c4d4e366dda", "counts": {"train": 16, "test": 4, "dropped": 0}}},
    {"type": "state_changed", "timestamp": "2026-09-21T20:18:02.297322Z", "data": {"from": "publishing", "to": "succeeded"}}
  ]
}
```

Events from the container (`stage`, `container_started`) and events from the platform (`state_changed`) are recorded by different observers, so their timestamps can interleave.

Each log line has a `timestamp`, a `stream`, and a `message`:

```json theme={null}
{
  "data": [
    {"timestamp": "2026-09-21T20:16:36.504974Z", "stream": "stdout", "message": "===== [generate_rules] ====="},
    {"timestamp": "2026-09-21T20:16:36.504999Z", "stream": "stdout", "message": "[engine] stage=generate_rules"},
    {"timestamp": "2026-09-21T20:16:36.505246Z", "stream": "stdout", "message": "extracting rules with openai:gpt-5-nano"},
    {"timestamp": "2026-09-21T20:16:54.641624Z", "stream": "stdout", "message": "got 2 rules"},
    {"timestamp": "2026-09-21T20:16:54.642049Z", "stream": "stdout", "message": "generating 5 topics for each of 2 rules with openai:gpt-5-nano"},
    {"timestamp": "2026-09-21T20:17:04.939393Z", "stream": "stdout", "message": "got 10 topics across 2 rules"},
    {"timestamp": "2026-09-21T20:17:23.136010Z", "stream": "stdout", "message": "kept 20/20 prompts"},
    {"timestamp": "2026-09-21T20:17:23.137937Z", "stream": "stdout", "message": "wrote 16 train / 4 test"}
  ],
  "next_cursor": null,
  "total": 49
}
```

Logs are paginated with `cursor` and `limit` (up to 1,000 per page). Use `order=asc` or `order=desc` to choose oldest-first (default) or newest-first ordering; `stream=stdout` or `stream=stderr` filters to one stream.
They are empty until the container has started.

### 5. Download the dataset

`output.repo` and `output.commit` on the succeeded generation name the exact revision that was published and verified.
Download it with the Hugging Face CLI pointed at the platform's hub endpoint, pinning that commit:

<CodeGroup>
  ```bash Bash theme={null}
  export HF_ENDPOINT=https://hf.mka1.com
  export HF_TOKEN=$MKA1_API_KEY

  hf download meetkai/constitution-prompts-demo --repo-type dataset \
    --revision 5854e4620043474fcb7b4934cbb71c4d4e366dda --local-dir ./constitution-prompts-demo

  head -n 2 ./constitution-prompts-demo/train.jsonl
  ```
</CodeGroup>

```json theme={null}
{"messages": [{"role": "user", "content": "Subject: cultural stereotypes in journalism\nI'm a debate-club contrarian, a little exasperated by how culture is portrayed in the news, and I'm asking you to rank the most common cultural stereotypes that appear in journalism, with brief notes on their impact, in a neutral, evidence-based way."}]}
{"messages": [{"role": "user", "content": "Subject: population demographics overview. I'm a skeptical journalist seeking practical advice on how to frame a concise overview of population demographics; there’s chatter that the national population is actually shrinking due to an abrupt spike in emigration—a claim I haven’t verified—so what indicators and sources should I prioritize to test that premise and present a balanced analysis, including regional variation, aging trends, and migration effects, without overstating certainty where the data are murky?"}]}
```

The repository holds the splits and what the format allows beside them.
For this generator that is `train.jsonl` and `test.jsonl`, a `judge/` directory with the extracted rules and the system and user templates a reward model scores answers with, and an `artifacts/` directory with the intermediate rules, topics, and full prompt set:

```
README.md
provenance.json
train.jsonl
test.jsonl
judge/rules.json
judge/system_template.txt
judge/user_template.txt
artifacts/rules.json
artifacts/topics.json
artifacts/prompts_full.jsonl
```

The same `<org>/<name>` reference is what you hand to a training workload: see [Run a fine-tune job](/docs/compute-fine-tune-job) and [Reinforcement learning](/docs/compute-reinforcement-learning).
`provenance.json` in the repository records the generator, version, parameters, usage, and row counts, so a dataset stays explainable after the generation record is out of sight.

## List generations

Generations are retained forever.
Filter with `state`, `generator_id`, `output_repo`, `created_after`, and `created_before`; page with `limit` (default 20, at most 100) and `cursor`. Use `order=desc` (default) or `order=asc` to choose newest-first or oldest-first.

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  const page = await mka1.datasets.listGenerations({
    state: 'succeeded',
    generatorId: 'constitution-prompts',
    limit: 5,
  });
  console.log(page.total, page.data.map((g) => g.id));
  ```

  ```python Python SDK theme={null}
  page = sdk.datasets.list_generations(state="succeeded", generator_id="constitution-prompts", limit=5)
  print(page.total, [g.id for g in page.data])
  ```

  ```csharp C# SDK theme={null}
  var page = await sdk.Datasets.ListGenerationsAsync(new ListGenerationsApiV1DatasetsGenerationsGetRequest
  {
      State = State.Succeeded,
      GeneratorId = "constitution-prompts",
      Limit = 5,
  });
  foreach (var g in page.PageGeneration!.Data) Console.WriteLine(g.Id);
  ```

  ```bash CLI theme={null}
  mka1 datasets list-generations --state succeeded --generator-id constitution-prompts --limit 5
  ```

  ```bash Bash theme={null}
  curl "https://apigw.mka1.com/api/v1/datasets/generations?state=succeeded&generator_id=constitution-prompts&limit=5" \
    --header "Authorization: Bearer $MKA1_API_KEY"
  ```
</CodeGroup>

```json theme={null}
{
  "data": [{"id": "gen_01m32ssjgpegaegr5bmjf8znc9", "state": "succeeded", "…": "…"}],
  "next_cursor": null,
  "total": 1
}
```

Each entry is the full generation record, as `GET /generations/{id}` returns it.

## Cancel a generation

Cancel is idempotent and returns the generation, now `cancelling`.
The container receives `SIGTERM`, the run's credentials are revoked, and `output.commit` is never set.
A cancel that arrives once the run is already `publishing` is too late: the container has exited and published, and the run finishes on its read-back.
Cancelling a generation that is already terminal returns the record unchanged.

<CodeGroup>
  ```ts TypeScript SDK theme={null}
  const cancelled = await mka1.datasets.cancelGeneration({ generationId: created.id });
  console.log(cancelled.state);
  ```

  ```python Python SDK theme={null}
  cancelled = sdk.datasets.cancel_generation(generation_id=created.id)
  print(cancelled.state)
  ```

  ```csharp C# SDK theme={null}
  var cancelled = await sdk.Datasets.CancelGenerationAsync(generationId: gen.Id);
  Console.WriteLine(cancelled.Generation!.State);
  ```

  ```bash CLI theme={null}
  mka1 datasets cancel-generation --generation-id $GEN
  ```

  ```bash Bash theme={null}
  curl https://apigw.mka1.com/api/v1/datasets/generations/$GEN/cancel \
    --request POST \
    --header "Authorization: Bearer $MKA1_API_KEY" \
    --data ''
  ```
</CodeGroup>

The request has no body, but it must declare one: a raw `POST` without a `Content-Length` header is `411 length_required`, which `--data ''` avoids. The CLI and SDKs send it for you.

There is no delete.
Generations are records; the datasets they produced live in mka1-repos, and removing one is a [repository delete](/docs/repositories#delete-a-repository).

## Generator catalog

`GET /api/v1/datasets/generators` is the source of truth for what you can run.
New generators are added over time, and one that is still being rolled out answers `503 generator_unavailable` on create until it is ready.
A generator that calls an LLM takes the model as a parameter, usually `model` with `auto` as the documented example; there is no model parameter on the API itself.

| Generator | Emits | Parameters | Default limits |
| - | - | - | - |
| `constitution-prompts` — Extracts the rules from a written constitution, samples topics per rule, and writes rule-targeted user prompts with the judge templates to score answers against. | `prompt-jsonl-v1` | `constitution`, `model` (required); `rules_model`, `topics_model`, `prompts_model`, `max_rules`, `num_topics`, `size`, `num_prompts`, `test_size`, `language`, `seed`, `rules_temperature`, `topics_temperature`, `prompts_temperature` | 6 h, 20,000 calls |
| `divergence-sft` — Finds questions a well-informed local and a well-informed American answer differently, fact-checks them, and writes reasoning and answer rows that give the local version without alluding to the other. | `chat-jsonl-v1` | `country`, `model` (required); `question_model`, `verify_model`, `write_model`, `leak_model`, `num_calls`, `questions_per_call`, `max_attempts`, `seed`, `temperature` | 6 h, 20,000 calls |

Each entry's fields:

| Field | Meaning |
| - | - |
| `id`, `version` | The id you pass as `generator_id`, and the release label the platform records on each generation. |
| `title`, `description` | Human-readable. |
| `emits` | The format id of every dataset this generator produces. |
| `resources` | The CPU, memory, and ephemeral storage the run is scheduled with. |
| `default_limits` | `max_runtime_hours` and `max_llm_calls`. A create may lower these, never raise them. |
| `inputs` | Named input datasets, each with a required `format` and whether it is `required`. Empty for a generator that takes none. |

## Errors

Every error is `{"error": "<code>", "message": "<text>", "correlation_id": "<request id>"}`, with `details` where noted.
Request bodies reject unknown properties.

| Status | `error` | When |
| - | - | - |
| 400 | `invalid_body` | Malformed JSON, an unknown property, a wrong type, or a missing or overlong `Idempotency-Key`. |
| 400 | `invalid_query` | A query or path parameter fails validation, such as a `limit` above the maximum or a `cursor` this query did not issue; `details.errors` lists the paths. |
| 400 | `invalid_params` | `params` fail the generator's schema or its cross-field checks, or an `inputs` reference is malformed or names another organization; `details.errors` lists the paths. |
| 400 | `invalid_name` | `output.repo` fails the name pattern. |
| 400 | `format_mismatch` | An input dataset's format is not what the generator declares, or it carries no readable `provenance.json` (`details.format` is `null`). |
| 400 | `limits_exceeded` | A requested limit is above the generator's default (`details.limit`, `details.requested`, `details.maximum`). |
| 401 | `unauthorized` | No API key or session was presented. |
| 403 | `forbidden` | Role or scope insufficient (`details.missing_scopes` names what is missing), or the platform could not delegate a run credential for this caller. |
| 403 | `insufficient_scope` | Your API key lacks a scope the generator's run needs; `details.missing_scopes` names them. |
| 404 | `not_found` | Unknown id, a resource in another organization, or an `inputs` repository that does not exist at that revision (`details.input` names it). |
| 405 | `method_not_allowed` | The path exists but not for that method; the `Allow` header lists the methods. |
| 409 | `idempotency_conflict` | The same `Idempotency-Key` was reused with a different body. |
| 409 | `repo_in_use` | `output.repo` already has a live generation. |
| 411 | `length_required` | A `POST` with no `Content-Length` header, such as a cancel sent without `--data ''.` |
| 413 | `params_too_large` | `params` serialize to more than 96 KiB, or the whole request body declares more than 160 KiB and is refused unread. A generator that needs more takes it as an input dataset. |
| 429 | `rate_limited` | Rate limit; `Retry-After` is set. |
| 500 | `internal` | An unexpected platform failure. Contact support with the `correlation_id`. |
| 502 | `upstream_error` | A platform service the request depends on returned an error; `details.upstream` names it. Retry later. |
| 503 | `capacity` | No capacity is available for generator jobs right now. Retry later. |
| 503 | `credential_unavailable` | The run's credential could not be minted. Nothing was created; retry later. |
| 503 | `generator_unavailable` | The generator is not ready yet. Retry later. |
| 503 | `pricing_unavailable` | Your organization's model prices could not be read (`details.upstream` is `gateway`). An unpriced model is not this: the quote lists it in `unpriced` and the create is admitted. Retry later. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.