> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mka1.com/llms.txt
> Use this file to discover all available pages before exploring further.

# List eval samples

> Returns prompt, target, model output, extraction, score, and judge details for an eval run.



## OpenAPI

````yaml https://apigw.mka1.com/speakeasy.json get /api/v1/llm/evals/runs/{run_id}/samples
openapi: 3.1.1
info:
  title: MKA1 API
  version: 1.1.0
  description: >-
    The MKA1 API is a RESTful API that provides access to the MKA1 platform.
    Learn how to get started with the API and the TypeScript SDK
    [here](https://mka1.apidocumentation.com/guides/getting-started).
  license:
    name: Proprietary
servers:
  - url: https://apigw.mka1.com
    description: MKA1 API Gateway
  - url: /
    description: Relative server URL (configurable via SDK constructor)
security: []
tags:
  - name: Resource Authorization
    description: >-
      Manage permissions for LLM resources. Create resources, grant/revoke
      permissions, and delete resources. Only resource owners can grant, revoke,
      or delete permissions.
    x-displayName: Resource Authorization
  - name: Audit
    description: >-
      Audit API for org owners and admins. Trace one request end to end by its
      X-Request-ID: API gateway timing, the responses it created, and their
      model invocations and tool calls.
    x-displayName: Audit
  - name: Embeddings
    description: >-
      Text embedding API endpoints for generating vector representations of
      text. Create semantic embeddings for search, clustering, and similarity
      matching using various embedding models.
    x-displayName: Embeddings
  - name: Feedback
    description: >-
      User feedback API for rating and commenting on chat completions. Collect
      thumbs up/down ratings and detailed feedback to improve model responses
      and track user satisfaction.
    x-displayName: Feedback
  - name: Images
    description: >-
      Image generation API endpoints for creating images from text descriptions.
      Generate images with control over size, quality, and style.
    x-displayName: Images
  - name: MCP Vault
    description: >-
      MCP vault API for storing user-owned MCP server configurations and
      encrypted credentials. Agents reference vault IDs so secrets are resolved
      only at tool execution time.
    x-displayName: MCP Vault
  - name: Speech
    description: >-
      Speech API endpoints for audio processing. Convert text to
      natural-sounding speech (TTS) or transcribe speech to text (STT) in
      different languages.
    x-displayName: Speech
  - name: Usage
    description: >-
      Usage tracking and analytics API for monitoring token consumption, request
      counts, and cost analysis. View detailed statistics per user, model, and
      time period.
    x-displayName: Usage
  - name: Extract
    description: >-
      Structured data extraction API for extracting information from files.
      Define JSON schemas to extract structured data from images, PDFs, and
      documents. Supports reusable schema templates.
    x-displayName: Extract
  - name: Text Classification
    description: >-
      Text classification API for categorizing text into predefined labels. Use
      AI models to classify text content for sentiment analysis, topic
      categorization, and content moderation.
    x-displayName: Text Classification
  - name: Responses
    description: >-
      Agent-powered responses API for creating AI agents with autonomous tool
      usage. Build conversational assistants that can use web search, file
      operations, image generation, code execution, and MCP integrations.
      Supports background processing, streaming, and real-time status tracking.
    x-displayName: Responses
  - name: Files
    description: >-
      File management API for uploading, storing, and managing files with
      automatic expiration and S3 integration. Upload files that can be used
      with Assistants, Vector Stores, and other features. Files are stored in S3
      with metadata tracked in PostgreSQL. Supports automatic cleanup of expired
      files.
    x-displayName: Files
  - name: Vector Stores
    description: >-
      Vector store API for storing and searching documents using embeddings.
      Create vector stores, upload files with automatic chunking and embedding
      generation, and perform semantic search. Files are processed
      asynchronously using Temporal workflows for durability. Supports automatic
      cleanup of expired stores and LanceDB for efficient vector storage.
    x-displayName: Vector Stores
  - name: Conversations
    description: >-
      Conversation management API for storing and retrieving conversation state
      across Response API calls. Create conversations, add items (user messages,
      assistant messages, system messages), and maintain conversation history.
      Supports metadata tracking and multi-turn dialogue state management.
    x-displayName: Conversations
  - name: Guardrails
    description: >-
      AI safety guardrails API for configuring content moderation and security
      policies. Set up ban word lists, prompt injection detection, and system
      prompt leakage prevention. Guardrails apply to all requests from an
      account and can be tested before deployment.
    x-displayName: Guardrails
  - name: Models
    description: >-
      Model listing API for discovering available models. Returns model IDs,
      ownership, and metadata for all registered models in the gateway.
    x-displayName: Models
  - name: Skills
    description: >-
      Skills API for managing versioned bundles of instructions and files
      following the Agent Skills standard. Create, version, and download
      reusable skill packages that include SKILL.md manifests for agent
      environments.
    x-displayName: Skills
  - name: Chat Completions
    description: >-
      **Deprecated: Use the Responses API (`/api/v1/llm/responses`) instead.**
      Chat completion endpoints with support for streaming, tool calls, and
      multiple providers.
    x-deprecated: true
    x-displayName: Chat Completions
  - name: Batches
    x-displayName: Batches
  - name: Evals
    x-displayName: Evals
  - name: Fine-Tuning
    x-displayName: Fine-Tuning
  - name: Memory Stores
    x-displayName: Memory Stores
  - name: Prompts
    x-displayName: Prompts
  - name: API Key
    x-displayName: API Key
  - name: OAuth
    x-displayName: OAuth
  - name: Session
    x-displayName: Session
  - name: Organization
    x-displayName: Organization
  - name: Cluster Admin
    x-displayName: Cluster Admin
  - name: Sessions
    description: Create, inspect, access, and terminate sandbox sessions.
    x-displayName: Sessions
  - name: Browser
    description: >-
      Connect to browser sessions through the gateway port proxy. Browser
      sessions expose a Chrome DevTools Protocol endpoint on port 9222.
    x-displayName: Browser
  - name: Execution
    description: Run shell commands and code inside an existing sandbox session.
    x-displayName: Execution
  - name: Workspace
    description: >-
      Inspect the workspace manifest, transfer files or archives, and download
      generated artifacts.
    x-displayName: Workspace
  - name: Sandbox Usage
    description: >-
      Aggregate sandbox usage statistics across sessions, execution, and
      workspace operations.
    x-displayName: Sandbox Usage
  - name: Sandbox Pricing
    description: >-
      Cluster-admin management of the sandbox compute rate card used for
      budgeted spend.
    x-displayName: Sandbox Pricing
  - name: Sandbox GitHub
    description: >-
      Connect a GitHub account, clone its repositories into sandbox sessions,
      and push branches and open pull requests from a session. The GitHub token
      stays in the sandbox service and never enters a session.
    x-displayName: Sandbox GitHub
  - name: Sandbox Environments
    description: >-
      Save reusable session configuration (repositories, environment variables,
      network access, and a setup script) and run the setup script in a session.
    x-displayName: Sandbox Environments
  - name: Sandbox Connectors
    description: >-
      Connect remote MCP servers (Linear, Notion, Sentry and others) to your
      account so Coding Harness sessions started from an environment that lists
      them can call their tools. The sandbox service keeps each connector's
      credential and never puts it in a session.
    x-displayName: Sandbox Connectors
  - name: Agents
    description: Create and manage reusable agent definitions.
    x-displayName: Agents
  - name: Agent Versions
    description: Inspect an agent's configuration history and roll back to a prior version.
    x-displayName: Agent Versions
  - name: Agent Runs
    description: Execute saved agents and inspect persisted run results.
    x-displayName: Agent Runs
  - name: Agent Connectors
    description: >-
      Connect saved agents to external messaging channels such as Telegram,
      including text, photo, and supported document exchange.
    x-displayName: Agent Connectors
  - name: Agent Schedules
    description: Create and manage scheduled or recurring saved agent runs.
    x-displayName: Agent Schedules
  - name: schema-4_other
    x-displayName: other
  - name: Billing
    x-displayName: Billing
  - name: Budgets
    x-displayName: Budgets
  - name: Settings
    x-displayName: Settings
  - name: Compute Jobs
    x-displayName: Compute Jobs
  - name: Compute Services
    x-displayName: Compute Services
  - name: Compute Catalog
    x-displayName: Compute Catalog
  - name: Compute Volumes
    x-displayName: Compute Volumes
  - name: Compute Pricing
    x-displayName: Compute Pricing
  - name: Compute Secrets
    x-displayName: Compute Secrets
  - name: Compute Usage
    x-displayName: Compute Usage
  - name: Compute Tenants
    x-displayName: Compute Tenants
  - description: Create, list, read, update, and delete repositories.
    name: Repositories
    x-displayName: Repositories
  - name: Tracking Experiments
    description: Create and query tenant experiments.
    x-displayName: Tracking Experiments
  - name: Tracking Runs
    description: Record and retrieve run parameters, metrics, and tags.
    x-displayName: Tracking Runs
  - name: analytics.projects
    x-displayName: Projects
    description: Create, list, rename, and archive analytics projects.
  - name: analytics.keys
    x-displayName: Capture keys
    description: Create and revoke project keys for sending events.
  - name: analytics
    x-displayName: Events
    description: Send events and query event counts or unique visitors.
  - name: analytics.events
    x-displayName: Event search
    description: List individual events with their properties, newest first.
  - name: analytics.visitors
    x-displayName: Visitor search
    description: List visitors by their latest activity, with first and last seen times.
  - name: generators
    x-displayName: generators
  - name: generations
    x-displayName: generations
  - name: formats
    x-displayName: formats
  - description: >-
      Flags, environments, evaluation, and the audit log. Administrative
      operations require a cluster administrator; evaluation uses an
      environment-specific evaluation key.
    name: Feature Flags
    x-displayName: Feature Flags
  - name: Service Status
    description: >-
      Current status of each service and the incident history, for this
      deployment's region.
    x-displayName: Service Status
  - name: Support Cases
    description: >-
      Open and follow support cases. Organization owners and admins see all of
      the organization's cases; other members see the cases they opened.
    x-displayName: Support Cases
paths:
  /api/v1/llm/evals/runs/{run_id}/samples:
    get:
      tags:
        - Evals
      summary: List eval samples
      description: >-
        Returns prompt, target, model output, extraction, score, and judge
        details for an eval run.
      operationId: listEvalSamples
      parameters:
        - name: run_id
          in: path
          required: true
          schema:
            type: string
          example: eval_run_aa87e2b1112a455b8deabed784372198
        - name: after
          in: query
          schema:
            type: string
          allowEmptyValue: true
        - name: limit
          in: query
          schema:
            type: integer
            minimum: 1
            maximum: 500
            default: 100
          allowEmptyValue: true
        - name: task_id
          in: query
          schema:
            type: string
          allowEmptyValue: true
        - name: model
          in: query
          schema:
            type: string
          allowEmptyValue: true
        - name: status
          in: query
          schema:
            enum:
              - queued
              - generating
              - ready_to_score
              - scoring
              - running
              - completed
              - failed
            type: string
          allowEmptyValue: true
        - name: score_metric
          in: query
          schema:
            type: string
            maxLength: 128
            pattern: ^[\w.-]+$
            description: >-
              Numeric metric key inside the `scores` jsonb to filter on (e.g.
              `wer`, `cer`, `accuracy`). Required when `score_min`/`score_max`
              are set. Only samples whose metric is a number match — unscored
              samples (queued/failed, `scores = {}`) are excluded. The key is
              matched as a single top-level jsonb key, so `wer.avg` looks up the
              literal key `"wer.avg"` rather than a nested path.
          allowEmptyValue: true
        - name: score_min
          in: query
          schema:
            type: number
            minimum: -9007199254740991
            maximum: 9007199254740991
            description: >-
              Inclusive lower bound for `score_metric`. Requires `score_metric`.
              A one-sided bound relies on jsonb numeric ordering, so keep the
              metric numeric.
          allowEmptyValue: true
        - name: score_max
          in: query
          schema:
            type: number
            minimum: -9007199254740991
            maximum: 9007199254740991
            description: Inclusive upper bound for `score_metric`. Requires `score_metric`.
          allowEmptyValue: true
        - name: include_dataset_row
          in: query
          schema:
            type: boolean
            default: true
            description: >-
              Set to false to skip the (potentially large, e.g. inlined ASR
              audio) `dataset_row` column — the column is not read from Postgres
              at all, and each sample comes back with `dataset_row: null`
              (indistinguishable from a sample that genuinely has none). Callers
              that only need scores/output (model comparison) avoid the TOAST
              read + JSON parse. Defaults to true.
          allowEmptyValue: true
        - name: include_reasoning
          in: query
          schema:
            type: boolean
            default: false
            description: >-
              Set to true to include the model's `reasoning` column. Off by
              default: reasoning routinely runs several times longer than the
              answer, and each sample's is capped at 64 KB. Because of that,
              `limit` must be <= 100 when this is true — a larger page is
              rejected with 400 rather than allowed to reach tens of megabytes.
              When false the column is not read from Postgres at all and each
              sample comes back with `reasoning: null` — indistinguishable from
              a sample that genuinely has none, so do not treat a null here as
              evidence the model did not reason.
          allowEmptyValue: true
        - name: sample_index
          in: query
          schema:
            type: string
            maxLength: 6000
            pattern: ^\d+(,\d+)*$
            description: >-
              Comma-separated sample indices to restrict to (deduped, max 500).
              Lets a caller fetch just the rows currently on screen (e.g.
              hydrate audio for one page of comparison pairs). Note a single
              index can return multiple rows since indices are unique per (task,
              model), and 500 indices can span more than one page since `limit`
              also caps at 500.
          allowEmptyValue: true
        - name: X-On-Behalf-Of
          in: header
          required: false
          schema:
            type: string
          description: Optional external end-user identifier forwarded by the API gateway.
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/EvalSampleList'
      security:
        - bearerAuth: []
      x-codeSamples:
        - lang: csharp
          label: CSharp (SDK)
          source: |-
            using MeetKai.MKA1;
            using MeetKai.MKA1.Types.Components;
            using MeetKai.MKA1.Types.Requests;

            var sdk = new SDK(bearerAuth: "<YOUR_BEARER_TOKEN_HERE>");

            ListEvalSamplesRequest req = new ListEvalSamplesRequest() {
                RunId = "eval_run_aa87e2b1112a455b8deabed784372198",
            };

            var res = await sdk.Llm.Evals.ListSamplesAsync(req);

            // handle response
        - lang: python
          label: Python (SDK)
          source: |-
            from meetkai_mka1 import SDK


            with SDK(
                bearer_auth="<YOUR_BEARER_TOKEN_HERE>",
            ) as sdk:

                res = sdk.llm.evals.list_samples(run_id="eval_run_aa87e2b1112a455b8deabed784372198", limit=100, include_dataset_row=True, include_reasoning=False)

                # Handle response
                print(res)
        - lang: typescript
          label: Typescript (SDK)
          source: |-
            import { SDK } from "@meetkai/mka1";

            const sdk = new SDK({
              bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
            });

            async function run() {
              const result = await sdk.llm.evals.listSamples({
                runId: "eval_run_aa87e2b1112a455b8deabed784372198",
              });

              console.log(result);
            }

            run();
        - lang: go
          label: Go (SDK)
          source: "package main\n\nimport(\n\t\"context\"\n\tmka1 \"github.com/MeetKai/mka1-sdks/go\"\n\t\"github.com/MeetKai/mka1-sdks/go/models/operations\"\n\t\"log\"\n)\n\nfunc main() {\n    ctx := context.Background()\n\n    s := mka1.New(\n        mka1.WithSecurity(\"<YOUR_BEARER_TOKEN_HERE>\"),\n    )\n\n    res, err := s.Llm.Evals.ListSamples(ctx, operations.ListEvalSamplesRequest{\n        RunID: \"eval_run_aa87e2b1112a455b8deabed784372198\",\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n    if res.EvalSampleList != nil {\n        // handle response\n    }\n}"
components:
  schemas:
    EvalSampleList:
      type: object
      properties:
        object:
          const: list
        data:
          type: array
          items:
            $ref: '#/components/schemas/EvalSampleObject'
        first_id:
          type: string
        last_id:
          type: string
        has_more:
          type: boolean
      required:
        - object
        - data
        - has_more
    EvalSampleObject:
      type: object
      properties:
        id:
          type: string
        object:
          const: eval.sample
        run_id:
          type: string
        task_id:
          type: string
        model:
          type: string
        sample_index:
          type: integer
          minimum: -9007199254740991
          maximum: 9007199254740991
        status:
          $ref: '#/components/schemas/EvalSampleStatus'
        dataset_row:
          anyOf:
            - type: object
              propertyNames:
                type: string
              additionalProperties: {}
            - type: 'null'
        prompt:
          type: string
        target:
          anyOf:
            - type: string
            - type: 'null'
        response_id:
          anyOf:
            - type: string
            - type: 'null'
        output_text:
          anyOf:
            - type: string
            - type: 'null'
        reasoning:
          anyOf:
            - type: string
            - type: 'null'
          description: >-
            Model reasoning ('thinking') captured from the response's reasoning
            items, kept separate from `output_text` so graders and extraction
            never see it. Truncated at 64 KB of UTF-8 with a marker appended.
            Null when the model returned none, when the provider only sent
            encrypted reasoning, and for transcription tasks.
        extracted_output:
          anyOf:
            - type: string
            - type: 'null'
        scores:
          type: object
          propertyNames:
            type: string
          additionalProperties: {}
        judge:
          anyOf:
            - type: object
              propertyNames:
                type: string
              additionalProperties: {}
            - type: 'null'
        error:
          anyOf:
            - type: object
              propertyNames:
                type: string
              additionalProperties: {}
            - type: 'null'
          description: >-
            The exception that ended this sample, if any. For a harness trial
            this also carries an agent timeout — which is still a *scored*
            trial, so do not read a non-null `error` here as an unscored one;
            check `status` and `scores`.
        cost_usd:
          anyOf:
            - type: number
            - type: 'null'
          description: >-
            Spend for this one trial in USD. Null outside harness runs, and when
            the harness reported no cost.
        input_tokens:
          anyOf:
            - type: integer
              minimum: -9007199254740991
              maximum: 9007199254740991
            - type: 'null'
          description: Prompt tokens for the whole trial session. Includes `cached_tokens`.
        output_tokens:
          anyOf:
            - type: integer
              minimum: -9007199254740991
              maximum: 9007199254740991
            - type: 'null'
          description: Completion tokens for the whole trial session.
        cached_tokens:
          anyOf:
            - type: integer
              minimum: -9007199254740991
              maximum: 9007199254740991
            - type: 'null'
          description: >-
            The share of `input_tokens` served from the provider's prompt cache.
            Reported separately because it is what makes two agents at the same
            token count cost an order of magnitude apart.
        reasoning_tokens:
          anyOf:
            - type: integer
              minimum: -9007199254740991
              maximum: 9007199254740991
            - type: 'null'
          description: >-
            The share of `output_tokens` the provider attributed to reasoning.
            Null when it reported no split.
        tokens_estimated:
          type: boolean
          description: >-
            True when the token figures were counted from the stored text by the
            gateway (tiktoken o200k_base, one ruler for every model) because
            neither the provider nor the imported row reported usage. Comparable
            verbosity, not a billing count.
        phase_timings:
          anyOf:
            - type: object
              propertyNames:
                type: string
              additionalProperties: {}
            - type: 'null'
          description: >-
            Per-phase wall clock for a harness trial: `environment_setup`,
            `agent_setup`, `agent_execution`, `verifier`, each `{ started_at,
            finished_at }` as ISO 8601 timestamps exactly as the harness
            recorded them. A phase the harness did not time is absent rather
            than null. Drives the trial timing bar. Null outside harness runs.
        created_at:
          type: integer
          minimum: -9007199254740991
          maximum: 9007199254740991
        started_at:
          anyOf:
            - type: integer
              minimum: -9007199254740991
              maximum: 9007199254740991
            - type: 'null'
        completed_at:
          anyOf:
            - type: integer
              minimum: -9007199254740991
              maximum: 9007199254740991
            - type: 'null'
      required:
        - id
        - object
        - run_id
        - task_id
        - model
        - sample_index
        - status
        - dataset_row
        - prompt
        - target
        - response_id
        - output_text
        - reasoning
        - extracted_output
        - scores
        - judge
        - error
        - cost_usd
        - input_tokens
        - output_tokens
        - cached_tokens
        - reasoning_tokens
        - tokens_estimated
        - phase_timings
        - created_at
        - started_at
        - completed_at
    EvalSampleStatus:
      enum:
        - queued
        - generating
        - ready_to_score
        - scoring
        - running
        - completed
        - failed
      type: string
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: API Key
      description: >-
        Send Authorization: Bearer <credential>, using an MKA1 API key for
        automation or an OAuth API access token for a signed-in application. Use
        a dynamic security callback to refresh OAuth access tokens. Browser
        session-only operations still require their declared cookie.

````

This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.