> ## Documentation Index > Fetch the complete documentation index at: https://docs.mka1.com/llms.txt > Use this file to discover all available pages before exploring further. # Create an eval suite version > Creates an immutable suite version. Use this endpoint for edits to an existing eval suite. ## OpenAPI ````yaml https://apigw.mka1.com/speakeasy.json post /api/v1/llm/evals/suites/{suite_id}/versions openapi: 3.1.1 info: title: MKA1 API version: 1.1.0 description: >- The MKA1 API is a RESTful API that provides access to the MKA1 platform. Learn how to get started with the API and the TypeScript SDK [here](https://mka1.apidocumentation.com/guides/getting-started). license: name: Proprietary servers: - url: https://apigw.mka1.com description: MKA1 API Gateway - url: / description: Relative server URL (configurable via SDK constructor) security: [] tags: - name: Resource Authorization description: >- Manage permissions for LLM resources. Create resources, grant/revoke permissions, and delete resources. Only resource owners can grant, revoke, or delete permissions. x-displayName: Resource Authorization - name: Embeddings description: >- Text embedding API endpoints for generating vector representations of text. Create semantic embeddings for search, clustering, and similarity matching using various embedding models. x-displayName: Embeddings - name: Feedback description: >- User feedback API for rating and commenting on chat completions. Collect thumbs up/down ratings and detailed feedback to improve model responses and track user satisfaction. x-displayName: Feedback - name: Images description: >- Image generation API endpoints for creating images from text descriptions. Generate images with control over size, quality, and style. x-displayName: Images - name: MCP Vault description: >- MCP vault API for storing user-owned MCP server configurations and encrypted credentials. Agents reference vault IDs so secrets are resolved only at tool execution time. x-displayName: MCP Vault - name: Speech description: >- Speech API endpoints for audio processing. Convert text to natural-sounding speech (TTS) or transcribe speech to text (STT) in different languages. x-displayName: Speech - name: Usage description: >- Usage tracking and analytics API for monitoring token consumption, request counts, and cost analysis. View detailed statistics per user, model, and time period. x-displayName: Usage - name: Extract description: >- Structured data extraction API for extracting information from files. Define JSON schemas to extract structured data from images, PDFs, and documents. Supports reusable schema templates. x-displayName: Extract - name: Text Classification description: >- Text classification API for categorizing text into predefined labels. Use AI models to classify text content for sentiment analysis, topic categorization, and content moderation. x-displayName: Text Classification - name: Responses description: >- Agent-powered responses API for creating AI agents with autonomous tool usage. Build conversational assistants that can use web search, file operations, image generation, code execution, computer use simulation, and MCP integrations. Supports background processing, streaming, and real-time status tracking. x-displayName: Responses - name: Files description: >- File management API for uploading, storing, and managing files with automatic expiration and S3 integration. Upload files that can be used with Assistants, Vector Stores, and other features. Files are stored in S3 with metadata tracked in PostgreSQL. Supports automatic cleanup of expired files. x-displayName: Files - name: Vector Stores description: >- Vector store API for storing and searching documents using embeddings. Create vector stores, upload files with automatic chunking and embedding generation, and perform semantic search. Files are processed asynchronously using Temporal workflows for durability. Supports automatic cleanup of expired stores and LanceDB for efficient vector storage. x-displayName: Vector Stores - name: Conversations description: >- Conversation management API for storing and retrieving conversation state across Response API calls. Create conversations, add items (user messages, assistant messages, system messages), and maintain conversation history. Supports metadata tracking and multi-turn dialogue state management. x-displayName: Conversations - name: Guardrails description: >- AI safety guardrails API for configuring content moderation and security policies. Set up ban word lists, prompt injection detection, and system prompt leakage prevention. Guardrails apply to all requests from an account and can be tested before deployment. x-displayName: Guardrails - name: Models description: >- Model listing API for discovering available models. Returns model IDs, ownership, and metadata for all registered models in the gateway. x-displayName: Models - name: Skills description: >- Skills API for managing versioned bundles of instructions and files following the Agent Skills standard. Create, version, and download reusable skill packages that include SKILL.md manifests for agent environments. x-displayName: Skills - name: Chat Completions description: >- **Deprecated: Use the Responses API (`/api/v1/llm/responses`) instead.** Chat completion endpoints with support for streaming, tool calls, and multiple providers. x-deprecated: true x-displayName: Chat Completions - name: Batches x-displayName: Batches - name: Evals x-displayName: Evals - name: Fine-Tuning x-displayName: Fine-Tuning - name: Memory Stores x-displayName: Memory Stores - name: Prompts x-displayName: Prompts - name: API Key x-displayName: API Key - name: Organization x-displayName: Organization - name: Cluster Admin x-displayName: Cluster Admin - name: Sessions description: Create, inspect, access, and terminate sandbox sessions. x-displayName: Sessions - name: Browser description: >- Connect to browser sessions through the gateway port proxy. Browser sessions expose a Chrome DevTools Protocol endpoint on port 9222. x-displayName: Browser - name: Execution description: Run shell commands and code inside an existing sandbox session. x-displayName: Execution - name: Workspace description: >- Inspect the workspace manifest, transfer files or archives, and download generated artifacts. x-displayName: Workspace - name: Sandbox Usage description: >- Aggregate sandbox usage statistics across sessions, execution, and workspace operations. x-displayName: Sandbox Usage - name: Sandbox Pricing description: >- Cluster-admin management of the sandbox compute rate card used for budgeted spend. x-displayName: Sandbox Pricing - name: Agents description: Create and manage reusable agent definitions. x-displayName: Agents - name: Agent Versions description: Inspect an agent's configuration history and roll back to a prior version. x-displayName: Agent Versions - name: Agent Runs description: Execute saved agents and inspect persisted run results. x-displayName: Agent Runs - name: Agent Schedules description: Create and manage scheduled or recurring saved agent runs. x-displayName: Agent Schedules - name: schema-4_other x-displayName: other - name: Budgets x-displayName: Budgets - name: Settings x-displayName: Settings - name: Deployments description: Long-lived inference servers. x-displayName: Deployments - name: Fine-Tune Jobs description: Submit, monitor, and cancel fine-tune jobs. x-displayName: Fine-Tune Jobs - name: Container Images description: Custom container images for deployments and jobs. x-displayName: Container Images - name: Serving Models description: Models registered for deployment and fine-tuning. x-displayName: Serving Models - name: Volumes description: Persistent storage for weights and checkpoints. x-displayName: Volumes - name: Secrets description: Credentials injected into your workloads. x-displayName: Secrets - name: Accelerators description: Available accelerator types (GPU, NPU, TPU). x-displayName: Accelerators paths: /api/v1/llm/evals/suites/{suite_id}/versions: post: tags: - Evals summary: Create an eval suite version description: >- Creates an immutable suite version. Use this endpoint for edits to an existing eval suite. operationId: createEvalSuiteVersion parameters: - name: suite_id in: path required: true schema: type: string example: eval_suite_aa87e2b1112a455b8deabed784372198 - name: X-On-Behalf-Of in: header required: false schema: type: string description: Optional external end-user identifier forwarded by the API gateway. requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CreateEvalSuiteVersionRequest' example: manifest: schema_version: '2026-05-27' tasks: - id: spanish_qa type: custom dataset: source: huggingface path: IIC/AQuAS split: test num_fewshot: 1 prompt_template: |- Responde usando el contexto. Contexto: {{context}} Pregunta: {{question}} Respuesta: target_template: '{{answer}}' preprocess: type: python source: | def transform(row): return row grader: type: python contract: model_backed model_access: mka1 file_id: file_grader123 timeout_seconds: 120 make_active: true responses: '200': description: OK content: application/json: schema: type: object properties: id: type: string object: const: eval.suite.version suite_id: type: string version: type: integer minimum: -9007199254740991 maximum: 9007199254740991 manifest: type: object properties: schema_version: type: string default: '2026-05-27' tasks: type: array minItems: 1 maxItems: 100 items: anyOf: - description: One declarative eval task. - type: 'null' metadata: type: object propertyNames: type: string additionalProperties: {} default: {} required: - tasks dataset_file_ids: type: array items: type: string metadata: type: object propertyNames: type: string maxLength: 64 additionalProperties: type: string maxLength: 512 created_at: type: integer minimum: -9007199254740991 maximum: 9007199254740991 required: - id - object - suite_id - version - manifest - dataset_file_ids - metadata - created_at security: - bearerAuth: [] x-codeSamples: - lang: python label: Python (SDK) source: |- from meetkai_mka1 import SDK, models with SDK( bearer_auth="", ) as sdk: res = sdk.llm.evals.create_suite_version(suite_id="eval_suite_aa87e2b1112a455b8deabed784372198", manifest={ "tasks": [ models.EvalTask( id="spanish_qa", type="custom", dataset=models.EvalDataset( source="huggingface", path="IIC/AQuAS", split="test", ), prompt_template="Responde usando el contexto.\n\nContexto: {{context}}\n\nPregunta: {{question}}\n\nRespuesta:", target_template="{{answer}}", grader=models.EvalPythonGrader( contract="model_backed", model_access="mka1", file_id="file_grader123", ), preprocess=models.EvalPythonPreprocessor( source="def transform(row):\n return row\n", ), num_fewshot=1, ), ], }, make_active=True) # Handle response print(res) - lang: typescript label: Typescript (SDK) source: |- import { SDK } from "@meetkai/mka1"; const sdk = new SDK({ bearerAuth: "", }); async function run() { const result = await sdk.llm.evals.createSuiteVersion({ suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198", createEvalSuiteVersionRequest: { manifest: { tasks: [ { id: "spanish_qa", type: "custom", dataset: { source: "huggingface", path: "IIC/AQuAS", split: "test", }, promptTemplate: "Responde usando el contexto.\n\nContexto: {{context}}\n\nPregunta: {{question}}\n\nRespuesta:", targetTemplate: "{{answer}}", grader: { type: "python", contract: "model_backed", modelAccess: "mka1", fileId: "file_grader123", }, preprocess: { type: "python", source: "def transform(row):\n return row\n", }, numFewshot: 1, }, ], }, }, }); console.log(result); } run(); - lang: csharp label: CSharp (SDK) source: |- using MeetKai.MKA1; using MeetKai.MKA1.Types.Components; using System.Collections.Generic; var sdk = new SDK(bearerAuth: ""); var res = await sdk.Llm.Evals.CreateSuiteVersionAsync( suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198", body: new MeetKai.MKA1.Types.Components.CreateEvalSuiteVersionRequest() { Manifest = new EvalSuiteManifest() { Tasks = new List() { new EvalTask() { Id = "spanish_qa", Type = EvalTaskType.Custom, Dataset = new EvalDataset() { Source = EvalDatasetSource.Huggingface, Path = "IIC/AQuAS", Split = "test", }, PromptTemplate = @"Responde usando el contexto. Contexto: {{context}} Pregunta: {{question}} Respuesta:", TargetTemplate = "{{answer}}", Grader = new EvalPythonGrader() { Contract = EvalPythonGraderContract.ModelBacked, ModelAccess = EvalPythonGraderModelAccess.Mka1, FileId = "file_grader123", }, Preprocess = new EvalPythonPreprocessor() { Source = @"def transform(row): return row ", }, NumFewshot = 1, }, }, }, } ); // handle response components: schemas: CreateEvalSuiteVersionRequest: type: object properties: manifest: $ref: '#/components/schemas/EvalSuiteManifest' metadata: type: object propertyNames: type: string maxLength: 64 additionalProperties: type: string maxLength: 512 default: {} make_active: type: boolean default: true required: - manifest EvalSuiteManifest: type: object properties: schema_version: type: string default: '2026-05-27' tasks: type: array minItems: 1 maxItems: 100 items: $ref: '#/components/schemas/EvalTask' metadata: type: object propertyNames: type: string additionalProperties: {} default: {} required: - tasks EvalTask: type: object properties: id: type: string pattern: ^[A-Za-z0-9_.-]+$ name: type: string maxLength: 255 type: $ref: '#/components/schemas/EvalTaskType' dataset: $ref: '#/components/schemas/EvalDataset' prompt_template: type: string minLength: 1 target_template: type: string choices: type: array items: type: string output_extraction: $ref: '#/components/schemas/EvalOutputExtraction' default: type: none metrics: type: array items: $ref: '#/components/schemas/EvalMetric' grader: $ref: '#/components/schemas/EvalPythonGrader' preprocess: $ref: '#/components/schemas/EvalPythonPreprocessor' fewshot: $ref: '#/components/schemas/EvalFewshotConfig' num_fewshot: type: integer minimum: 0 maximum: 100 description: lm-eval style alias for fewshot.count. metadata: type: object propertyNames: type: string additionalProperties: {} default: {} required: - id - type - dataset - prompt_template - grader description: One declarative eval task. EvalTaskType: enum: - classification - multiple_choice - qa - summarization - semantic_similarity - llm_judge - numeric - math - custom type: string EvalDataset: type: object properties: source: enum: - file - huggingface type: string description: Dataset source. Inferred from file_id or path when omitted. file_id: type: string description: File ID for a JSONL or CSV dataset uploaded with purpose 'evals'. format: $ref: '#/components/schemas/EvalDatasetFormat' description: Dataset format. Inferred from the filename when omitted. path: type: string description: Hugging Face dataset path, for example facebook/belebele. name: type: string description: Hugging Face dataset config/name. split: type: string description: Hugging Face split to load, for example test, train, or validation. revision: type: string description: Optional Hugging Face dataset revision. data_files: anyOf: - type: string - type: array items: type: string - type: object propertyNames: type: string additionalProperties: anyOf: - type: string - type: array items: type: string description: >- Optional Hugging Face data_files value for json/csv/parquet style datasets. dataset_kwargs: type: object propertyNames: type: string additionalProperties: {} description: >- Additional dataset loading kwargs retained for import/parity metadata. max_rows: type: integer minimum: 1 maximum: 1000000 description: Optional safety cap for remote dataset loading. description: Dataset backing an eval task. EvalOutputExtraction: anyOf: - type: object properties: type: const: none required: - type - type: object properties: type: const: take_first lines: type: integer minimum: 1 maximum: 10 default: 1 required: - type - type: object properties: type: const: regex pattern: type: string group: anyOf: - type: integer minimum: 0 maximum: 9007199254740991 - type: string default: 1 flags: type: string required: - type - pattern - type: object properties: type: const: regex_last pattern: type: string group: anyOf: - type: integer minimum: 0 maximum: 9007199254740991 - type: string default: 1 flags: type: string required: - type - pattern - type: object properties: type: const: label_set labels: type: array minItems: 1 items: type: string case_sensitive: type: boolean default: false required: - type - labels - type: object properties: type: const: number required: - type discriminator: propertyName: type mapping: {} EvalMetric: type: object properties: id: type: string pattern: ^[A-Za-z0-9_.-]+$ aggregation: enum: - mean - none type: string default: mean higher_is_better: type: boolean default: true description: type: string maxLength: 1000 required: - id additionalProperties: {} EvalPythonGrader: type: object properties: type: const: python contract: $ref: '#/components/schemas/EvalPythonGraderContract' default: sample description: >- Python contract to execute: sample grade(sample, item), batch grade_batch(samples), or model_backed with ctx model helpers. execution: $ref: '#/components/schemas/EvalPythonGraderExecution' description: >- Whether the grader runs per sample or once at aggregate/finalization time. Defaults from contract. model_access: $ref: '#/components/schemas/EvalPythonGraderModelAccess' description: >- Whether the Python ctx may request MKA1 Responses or embeddings through the gateway-owned bridge. metric_id: type: string pattern: ^[A-Za-z0-9_.-]+$ default: score description: Default metric key used when grade returns a single float. source: type: string minLength: 1 description: Inline Python source that defines a supported eval grading contract. file_id: type: string description: File ID for a Python grader uploaded with purpose 'evals'. timeout_seconds: type: integer minimum: 1 maximum: 600 default: 120 max_model_calls: type: integer minimum: 0 maximum: 500 default: 64 required: - type EvalPythonPreprocessor: type: object properties: type: const: python contract: enum: - row - batch type: string default: row description: Run transform(row) per row or transform_batch(rows) once. source: type: string minLength: 1 description: >- Inline Python source that transforms dataset rows before prompt rendering. file_id: type: string description: File ID for a Python preprocessor uploaded with purpose 'evals'. timeout_seconds: type: integer minimum: 1 maximum: 600 default: 120 required: - type EvalFewshotConfig: type: object properties: count: type: integer minimum: 0 maximum: 100 default: 0 dataset: $ref: '#/components/schemas/EvalDataset' description: Optional separate dataset used for few-shot examples. prompt_template: type: string description: >- Optional prompt template for few-shot examples. Defaults to task prompt_template. target_template: type: string description: >- Optional target template for few-shot examples. Defaults to task target_template. example_template: type: string default: |- {{prompt}} {{target}} description: Template for one rendered few-shot example. separator: type: string default: |+ description: Separator between few-shot examples and the evaluated prompt. strategy: enum: - first - random type: string default: first seed: type: integer minimum: -9007199254740991 maximum: 9007199254740991 default: 0 EvalDatasetFormat: enum: - jsonl - csv type: string EvalPythonGraderContract: enum: - sample - batch - model_backed type: string EvalPythonGraderExecution: enum: - per_sample - aggregate type: string EvalPythonGraderModelAccess: enum: - none - mka1 type: string securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: API Key description: >- Gateway auth: send `Authorization: Bearer `. For multi-user server-side integrations, you can also send `X-On-Behalf-Of: `. ````