> ## Documentation Index > Fetch the complete documentation index at: https://docs.mka1.com/llms.txt > Use this file to discover all available pages before exploring further. # Get the coding-agent leaderboard for an eval suite > Ranks the suite's completed coding-agent runs, one row per (agent, model, effort). Only runs that recorded an `agent_name` participate — a QA or transcription run of the same suite has no agent to attribute a score to and is not a board row. `accuracy` is the mean of `metric` (default `reward`) over the row's scored trials, and `accuracy_ci` is the 95% Wald interval around it; both are derived per request rather than stored, so rerunning a failed trial moves them instead of leaving a stale copy behind. `agent_version` is reported but is not part of the row identity, so an agent upgrade does not split a row in two. Rows are ordered by accuracy descending, nulls last. Pass `suite_version` when the suite has more than one, otherwise the board mixes task sets. ## OpenAPI ````yaml https://apigw.mka1.com/speakeasy.json get /api/v1/llm/evals/suites/{suite_id}/leaderboard openapi: 3.1.1 info: title: MKA1 API version: 1.1.0 description: >- The MKA1 API is a RESTful API that provides access to the MKA1 platform. Learn how to get started with the API and the TypeScript SDK [here](https://mka1.apidocumentation.com/guides/getting-started). license: name: Proprietary servers: - url: https://apigw.mka1.com description: MKA1 API Gateway - url: / description: Relative server URL (configurable via SDK constructor) security: [] tags: - name: Resource Authorization description: >- Manage permissions for LLM resources. Create resources, grant/revoke permissions, and delete resources. Only resource owners can grant, revoke, or delete permissions. x-displayName: Resource Authorization - name: Embeddings description: >- Text embedding API endpoints for generating vector representations of text. Create semantic embeddings for search, clustering, and similarity matching using various embedding models. x-displayName: Embeddings - name: Feedback description: >- User feedback API for rating and commenting on chat completions. Collect thumbs up/down ratings and detailed feedback to improve model responses and track user satisfaction. x-displayName: Feedback - name: Images description: >- Image generation API endpoints for creating images from text descriptions. Generate images with control over size, quality, and style. x-displayName: Images - name: MCP Vault description: >- MCP vault API for storing user-owned MCP server configurations and encrypted credentials. Agents reference vault IDs so secrets are resolved only at tool execution time. x-displayName: MCP Vault - name: Speech description: >- Speech API endpoints for audio processing. Convert text to natural-sounding speech (TTS) or transcribe speech to text (STT) in different languages. x-displayName: Speech - name: Usage description: >- Usage tracking and analytics API for monitoring token consumption, request counts, and cost analysis. View detailed statistics per user, model, and time period. x-displayName: Usage - name: Extract description: >- Structured data extraction API for extracting information from files. Define JSON schemas to extract structured data from images, PDFs, and documents. Supports reusable schema templates. x-displayName: Extract - name: Text Classification description: >- Text classification API for categorizing text into predefined labels. Use AI models to classify text content for sentiment analysis, topic categorization, and content moderation. x-displayName: Text Classification - name: Responses description: >- Agent-powered responses API for creating AI agents with autonomous tool usage. Build conversational assistants that can use web search, file operations, image generation, code execution, computer use simulation, and MCP integrations. Supports background processing, streaming, and real-time status tracking. x-displayName: Responses - name: Files description: >- File management API for uploading, storing, and managing files with automatic expiration and S3 integration. Upload files that can be used with Assistants, Vector Stores, and other features. Files are stored in S3 with metadata tracked in PostgreSQL. Supports automatic cleanup of expired files. x-displayName: Files - name: Vector Stores description: >- Vector store API for storing and searching documents using embeddings. Create vector stores, upload files with automatic chunking and embedding generation, and perform semantic search. Files are processed asynchronously using Temporal workflows for durability. Supports automatic cleanup of expired stores and LanceDB for efficient vector storage. x-displayName: Vector Stores - name: Conversations description: >- Conversation management API for storing and retrieving conversation state across Response API calls. Create conversations, add items (user messages, assistant messages, system messages), and maintain conversation history. Supports metadata tracking and multi-turn dialogue state management. x-displayName: Conversations - name: Guardrails description: >- AI safety guardrails API for configuring content moderation and security policies. Set up ban word lists, prompt injection detection, and system prompt leakage prevention. Guardrails apply to all requests from an account and can be tested before deployment. x-displayName: Guardrails - name: Models description: >- Model listing API for discovering available models. Returns model IDs, ownership, and metadata for all registered models in the gateway. x-displayName: Models - name: Skills description: >- Skills API for managing versioned bundles of instructions and files following the Agent Skills standard. Create, version, and download reusable skill packages that include SKILL.md manifests for agent environments. x-displayName: Skills - name: Chat Completions description: >- **Deprecated: Use the Responses API (`/api/v1/llm/responses`) instead.** Chat completion endpoints with support for streaming, tool calls, and multiple providers. x-deprecated: true x-displayName: Chat Completions - name: Batches x-displayName: Batches - name: Evals x-displayName: Evals - name: Fine-Tuning x-displayName: Fine-Tuning - name: Memory Stores x-displayName: Memory Stores - name: Prompts x-displayName: Prompts - name: API Key x-displayName: API Key - name: Session x-displayName: Session - name: Organization x-displayName: Organization - name: Cluster Admin x-displayName: Cluster Admin - name: Sessions description: Create, inspect, access, and terminate sandbox sessions. x-displayName: Sessions - name: Browser description: >- Connect to browser sessions through the gateway port proxy. Browser sessions expose a Chrome DevTools Protocol endpoint on port 9222. x-displayName: Browser - name: Execution description: Run shell commands and code inside an existing sandbox session. x-displayName: Execution - name: Workspace description: >- Inspect the workspace manifest, transfer files or archives, and download generated artifacts. x-displayName: Workspace - name: Sandbox Usage description: >- Aggregate sandbox usage statistics across sessions, execution, and workspace operations. x-displayName: Sandbox Usage - name: Sandbox Pricing description: >- Cluster-admin management of the sandbox compute rate card used for budgeted spend. x-displayName: Sandbox Pricing - name: schema-3_other x-displayName: other - name: Agents description: Create and manage reusable agent definitions. x-displayName: Agents - name: Agent Versions description: Inspect an agent's configuration history and roll back to a prior version. x-displayName: Agent Versions - name: Agent Runs description: Execute saved agents and inspect persisted run results. x-displayName: Agent Runs - name: Agent Connectors description: >- Connect saved agents to external messaging channels such as Telegram, including text, photo, and supported document exchange. x-displayName: Agent Connectors - name: Agent Schedules description: Create and manage scheduled or recurring saved agent runs. x-displayName: Agent Schedules - name: schema-4_other x-displayName: other - name: Budgets x-displayName: Budgets - name: Settings x-displayName: Settings - name: Compute Jobs x-displayName: Compute Jobs - name: Compute Services x-displayName: Compute Services - name: Compute Catalog x-displayName: Compute Catalog - name: Compute Volumes x-displayName: Compute Volumes - name: Compute Pricing x-displayName: Compute Pricing - name: Compute Secrets x-displayName: Compute Secrets - name: Compute Usage x-displayName: Compute Usage - name: Compute Tenants x-displayName: Compute Tenants - description: Create, list, read, update, and delete repositories. name: Repositories x-displayName: Repositories paths: /api/v1/llm/evals/suites/{suite_id}/leaderboard: get: tags: - Evals summary: Get the coding-agent leaderboard for an eval suite description: >- Ranks the suite's completed coding-agent runs, one row per (agent, model, effort). Only runs that recorded an `agent_name` participate — a QA or transcription run of the same suite has no agent to attribute a score to and is not a board row. `accuracy` is the mean of `metric` (default `reward`) over the row's scored trials, and `accuracy_ci` is the 95% Wald interval around it; both are derived per request rather than stored, so rerunning a failed trial moves them instead of leaving a stale copy behind. `agent_version` is reported but is not part of the row identity, so an agent upgrade does not split a row in two. Rows are ordered by accuracy descending, nulls last. Pass `suite_version` when the suite has more than one, otherwise the board mixes task sets. operationId: getEvalSuiteLeaderboard parameters: - name: suite_id in: path required: true schema: type: string example: eval_suite_aa87e2b1112a455b8deabed784372198 - name: suite_version in: query schema: type: integer minimum: 1 maximum: 2147483647 description: >- Restrict the board to one suite version. Omit to aggregate every version, which mixes task sets — prefer pinning it when the suite has more than one. allowEmptyValue: true example: 1 - name: metric in: query schema: type: string maxLength: 128 pattern: ^[\w.-]+$ default: reward description: >- The `scores` key to average into `accuracy`. Defaults to `reward`, what the harness writes for a pass/fail task. Matched as a single top-level key, so `reward.avg` looks up the literal key. allowEmptyValue: true example: reward - name: agent in: query schema: type: string maxLength: 128 description: Restrict to one agent name. allowEmptyValue: true - name: model in: query schema: type: string maxLength: 255 description: Restrict to one model. allowEmptyValue: true - name: limit in: query schema: type: integer minimum: 1 maximum: 200 default: 100 description: Maximum rows returned. allowEmptyValue: true example: 100 - name: X-On-Behalf-Of in: header required: false schema: type: string description: Optional external end-user identifier forwarded by the API gateway. responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/EvalSuiteLeaderboard' example: object: eval.leaderboard suite_id: eval_suite_aa87e2b1112a455b8deabed784372198 suite_version: 1 metric: reward data: - agent_name: omp agent_version: 17.3.8 agent_effort: medium model: meetkai:functionary-urdu-mini-pak accuracy: 0.539 accuracy_ci: low: 0.435 high: 0.643 margin: 0.104 trials: 89 scored_trials: 89 failed_trials: 0 cost_usd: 3.77 run_count: 1 run_ids: - eval_run_aa87e2b1112a455b8deabed784372198 last_run_at: 1704067200 security: - bearerAuth: [] x-codeSamples: - lang: python label: Python (SDK) source: |- from meetkai_mka1 import SDK with SDK( bearer_auth="", ) as sdk: res = sdk.llm.evals.get_suite_leaderboard(suite_id="eval_suite_aa87e2b1112a455b8deabed784372198", suite_version=1, metric="reward", limit=100) # Handle response print(res) - lang: typescript label: Typescript (SDK) source: |- import { SDK } from "@meetkai/mka1"; const sdk = new SDK({ bearerAuth: "", }); async function run() { const result = await sdk.llm.evals.getSuiteLeaderboard({ suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198", suiteVersion: 1, }); console.log(result); } run(); - lang: csharp label: CSharp (SDK) source: >- using MeetKai.MKA1; using MeetKai.MKA1.Types.Components; using MeetKai.MKA1.Types.Requests; var sdk = new SDK(bearerAuth: ""); GetEvalSuiteLeaderboardRequest req = new GetEvalSuiteLeaderboardRequest() { SuiteId = "eval_suite_aa87e2b1112a455b8deabed784372198", SuiteVersion = 1, }; var res = await sdk.Llm.Evals.GetSuiteLeaderboardAsync(req); // handle response components: schemas: EvalSuiteLeaderboard: type: object properties: object: const: eval.leaderboard suite_id: type: string suite_version: anyOf: - type: integer minimum: -9007199254740991 maximum: 9007199254740991 - type: 'null' description: >- The version this board was restricted to, or null when it spans every version. metric: type: string description: >- The `scores` key `accuracy` averages. Echoed back so a board rendered from a non-default metric is self-describing. data: type: array items: $ref: '#/components/schemas/EvalLeaderboardRow' description: >- Rows ordered by `accuracy` descending, nulls last, then by agent, model, and effort. required: - object - suite_id - suite_version - metric - data EvalLeaderboardRow: type: object properties: agent_name: type: string agent_version: anyOf: - type: string - type: 'null' description: >- Version from the row's newest run. Not part of the row identity, so older runs in this row may have reported a different one. agent_effort: anyOf: - type: string - type: 'null' model: type: string accuracy: anyOf: - type: number - type: 'null' description: >- Mean of `metric` over the row's scored trials. Null when no trial carried a numeric value for that metric — distinct from 0, which means every trial scored zero. accuracy_ci: anyOf: - type: object properties: low: type: number high: type: number margin: type: number required: - low - high - margin - type: 'null' description: >- 95% Wald interval, p ± 1.96·sqrt(p(1-p)/n), clamped to [0, 1]. Derived per request rather than stored, so it cannot drift when a failed trial is rerun. It assumes `metric` is a 0/1 outcome (the harness `reward`); on a continuous metric the mean is still right but this interval is not, so read it only for pass/fail metrics. Null when `accuracy` is. trials: type: integer minimum: -9007199254740991 maximum: 9007199254740991 description: Every sample in the row's runs, whatever its status. scored_trials: type: integer minimum: -9007199254740991 maximum: 9007199254740991 description: >- Trials carrying a numeric `metric` — the denominator behind `accuracy` and `accuracy_ci`. failed_trials: type: integer minimum: -9007199254740991 maximum: 9007199254740991 description: >- Trials that ended in the `failed` status, i.e. produced no score at all. An agent timeout is NOT one of these: the harness still verifies and scores that trial. cost_usd: anyOf: - type: number - type: 'null' description: >- Summed trial spend, falling back to the runs' own totals when trials carry no cost. Null when neither reported any. run_count: type: integer minimum: -9007199254740991 maximum: 9007199254740991 run_ids: type: array items: type: string description: >- The runs behind this row, largest (most trials) first, ties newest first — element 0 is the run to open when the reader drills into this row, so a full benchmark job outranks a newer one-task smoke run. last_run_at: type: integer minimum: -9007199254740991 maximum: 9007199254740991 description: Creation time of the newest run in the row, unix seconds. required: - agent_name - agent_version - agent_effort - model - accuracy - accuracy_ci - trials - scored_trials - failed_trials - cost_usd - run_count - run_ids - last_run_at securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: API Key description: >- Gateway auth: send `Authorization: Bearer `. For multi-user server-side integrations, you can also send `X-On-Behalf-Of: `. ````