Every generation runs as a Compute job under your organization and publishes under your API key, so the hardware and any LLM calls the generator makes bill to you like any other platform usage.
Datasets is coming soon. The endpoints described here are not available yet.
Before you start
You need:
Everything you create belongs to your organization.
A generation from another organization is
404, never 403.
The examples export the key once:
Dataset formats
A dataset is a repository withtrain.jsonl at its root and, optionally, test.jsonl and validation.jsonl: UTF-8, one JSON object per line.
Alongside the rows, the platform writes a README.md with an mka1: front-matter block and a provenance.json that records the generator, its version, the exact parameters, row counts, and LLM usage.
Every row in a split has the same format, and the format is what a trainer declares it accepts.
All three formats share the OpenAI-style message object:
reasoning is an MKA1 extension for assistant turns that carries a thinking trace apart from content; trainers that do not understand it ignore it.
The catalog and each format’s row schema are served by the API, so a trainer can validate a dataset before spending GPU time:
row_schema (JSON Schema for one line), layout (required and optional files), semantics (the checks beyond schema, such as “the last turn is the assistant’s”), and converters (for example to_trl, which rewrites preference rows as prompt/chosen/rejected).
Generate a dataset
The walkthrough usesconstitution-prompts, which turns a written constitution into a prompt-only dataset for reinforcement learning: it extracts the rules, samples topics, and generates user prompts that target each rule.
Every generator follows the same five steps; only the parameters change.
1. Discover a generator
List the available generators, then read one to get its parameter schema.params_schema (JSON Schema, draft 2020-12) and example_params, a set of parameters the author guarantees to be valid:
version is a label the platform records on every generation and in the dataset’s provenance; you do not choose it.
default_limits are the ceilings a generation of this generator runs under, and inputs names any dataset repositories the generator can take as input.
2. Quote the run
A quote is an advisory estimate of the LLM spend for a set of parameters. It reserves nothing and runs the same parameter validation as create, so it is also the cheapest way to check a request body.llm entry per model the generator will call, with the estimated calls, tokens, and cost at your organization’s current prices.
estimated_cost sums the priced models, and unpriced lists any model that has no price yet.
Every money figure on this API is in currency.
The figure is the generator author’s point estimate, not a bound.
The bound is limits.max_cost: a create whose quote exceeds it is rejected with 400 limits_exceeded.
Parameters that fail the schema return 400 invalid_params with the offending paths in details.errors.
3. Create the generation
Creating a generation needs anIdempotency-Key header: any string up to 255 characters that is unique within your organization.
Replaying the same key with the same body returns the original generation with 200; the same key with a different body is 409 idempotency_conflict.
The response is
201 and the generation is pending:
state, reason, and events on the generation, never as an HTTP error on the create.
Two responses are worth planning for:
duplicate_of. If a generation with the same parameters, generator version, and output repository has already succeeded, the201carries"duplicate_of": "gen_…". The new run still starts; whether to cancel it and reuse the earlier dataset is your call.409 repo_in_use. One repository holds at most one live generation. A second create against the sameoutput.repois refused until the first reachessucceeded,failed, orcancelled.
4. Watch it run
Poll the generation until its state is terminal.
On success the record carries what the run produced:
reason.code is one of a fixed vocabulary.
The ones a caller can act on:
Two more endpoints show what happened inside the run.
Events are the structured timeline, oldest first, including every state change and each
stage the generator announced; logs are the container’s captured stdout and stderr, retained after the container is gone and capped at the last 20,000 lines.
cursor and limit (up to 1,000 per page) and read oldest first; stream=stdout or stream=stderr filters to one stream.
They are empty until the container has started.
5. Download the dataset
output.repo and output.commit on the succeeded generation name the exact revision that was published and verified.
Download it with the Hugging Face CLI pointed at the platform’s hub endpoint, pinning that commit:
<org>/<name> reference is what you hand to a training workload: see Run a fine-tune job and Reinforcement learning.
provenance.json in the repository records the generator, version, parameters, and row counts, so a dataset stays explainable after the generation record is out of sight.
List generations
Generations are retained forever and listed newest first. Filter withstate, generator_id, output_repo, created_after, and created_before; page with limit (default 20, at most 100) and cursor.
Cancel a generation
Cancel is idempotent and returns the generation, nowcancelling.
The container receives SIGTERM, the run’s credentials are revoked, and output.commit is never set.
A cancel that arrives once the run is already publishing is too late: the container has exited and published, and the run finishes on its read-back.
Cancelling a generation that is already terminal is 409.
Generator catalog
GET /api/v1/datasets/generators is the source of truth for what you can run.
New generators are added over time, and one that is still being rolled out answers 503 generator_unavailable on create until it is ready.
Which LLM a generator calls is the generator’s own decision, either fixed or exposed as a field in its params_schema; there is no model parameter on the API.
More generators are coming soon, including supervised chat data with reasoning traces for SFT.
Each entry’s fields:
Errors
Every error is{"error": "<code>", "message": "<text>", "correlation_id": "<request id>"}.
Request bodies reject unknown properties.