model: "auto", resolves through your organization’s model registry. This guide covers the two layers behind that lookup and the operations that change them.
- The catalog is what your organization could activate: its own bring-your-own (BYO) definitions and the cluster models a cluster admin has granted it access to.
- The registry is what it has activated. A model name resolves only while the registry holds it.
GET /models omits it, a request that names it fails to resolve, and auto cannot land on it either.
Before you start
The console has the same views under Admin → Model Registry, on the Models, Auto models, Add model, and Raw JSON tabs. LLM → Models shows what your requests can use right now.
Step 1 - List the models your requests can use
GET /models returns the active entries of your registry, minus any whose definition sets hidden: true (see Troubleshooting). Read each item’s capabilities before you send sampling or reasoning parameters.
Use the SDKs or the CLI to manage the model catalog and registry.
GET /models/{model_id} returns 404 for a name that is not in the catalog, for one that is in the catalog but inactive, and for one whose definition is hidden. The gateway does not say which, so a response cannot reveal what other organizations have.
Step 2 - Read the catalog
GET /models/catalog lists every definition your organization could activate, with its source, activation state, provider health, and a price preview. Cluster entries appear only after a cluster admin grants access to them.
The same model id can appear more than once, once per source. Only one of them can hold the name at a time.
Provider health
health holds is_available, last_health_check (a Unix timestamp in milliseconds), and error. The definition’s healthCheck sets when the gateway checks the provider:
A check sends
method (GET by default) to baseUrl plus path with the entry’s credentials, and any 2xx response counts as available. A result is kept for cacheTtlSeconds, which defaults to 300 seconds for preflight and to intervalSeconds plus 60 seconds for periodic. Unless the mode is off, defining or updating a BYO entry checks its provider at once, and so does checkCatalogHealth. Entries that share a provider label share one result.
When the kept result says the provider is unavailable, Responses and Chat Completions requests skip it. They follow the entry’s error-fallback reroute rule if it has one, and fail otherwise. Health does not change what GET /models lists or which candidates auto routing considers.
Step 3 - Activate a catalog entry
Activation claims a model name for one source. Pass the id and the source you are activating from.- The entry exists for your organization from that source. A cluster id you have no access to, or a BYO id nobody defined, returns
422. - Nobody else holds the name. If the same source already holds it, the call returns the existing activation. If a different source holds it, the call returns
409. Neither case changes anything.
model_id, source, activated_by, and activated_at. The activation applies to the next request. In the console, the Activate button on Admin → Model Registry → Models makes the same call. The setup wizard’s first step activates every available cluster entry the same way, but only while nothing in the registry is active yet.
Step 4 - Review the registry and deactivate
GET /models/registry is the catalog filtered to active entries, so each row has the same shape as in Step 2. DELETE /models/registry?id=... releases a name.
org_override rate in Step 6). It does not touch the catalog definition, so you can activate the entry again later, from the same source or a different one. The call is idempotent: deactivated is false when nothing was active.
Requests already in flight finish. The next request that names the model fails to resolve, and if an auto-model override pointed at it, auto fails for that endpoint too (see Step 5).
Step 5 - Choose what auto resolves to
The gateway applies an auto policy per endpoint. With no organization override,
model: "auto" remains a direct alias to the cluster default. A direct
organization override aliases auto to one
exact active model. A balanced override asks a configured LLM judge to choose
from a configured candidate list.
The endpoint key is the model’s API format:
Bash
meetkai:functionary-pt, while other requests can route to
openai:gpt-4.1-mini. Use a fast, non-reasoning judge that supports
structured outputs, such as openai:gpt-4.1-nano.
Balanced policies are supported for responses, completions,
transcriptions, and tts. The judge must be an active text LLM. Each
candidate must be active and compatible with the endpoint when the policy is
written. At request time, a revoked candidate is excluded, but an unavailable
judge causes a transparent error. One eligible candidate is selected directly;
two or more invoke the judge.
PUT returns 422 when any configured model is the literal auto, is
not active in your registry, or has an incompatible API format. A direct
override stores the canonical id and later looks up that exact name. If you
deactivate it, the override’s resolvable flag turns false and auto
fails on that endpoint rather than moving to the cluster default. Clear the
override or activate the model again. DELETE is idempotent. Direct requests
use the body key model_id; judge-based requests use judge_model_id and
candidate_model_ids. Request and response keys are snake_case; the older
camelCase request keys are rejected with 400.
GET /models/auto reports mode: "balanced",
judge_resolvable, and resolvable_candidate_model_ids for a balanced
override. Auto routing explains what
the judge weighs, including modalities, file support, pricing, gateway
latency, comparable evals, and conversation continuity.
The console shows the same overrides on Admin → Model Registry → Auto models.
Step 6 - Check effective prices
GET /models/pricing/effective returns the rate your organization pays for every model it can call. Any bearer can read it, with no role or scope, because anyone whose requests are billed can see the rate they are billed at. Audit fields such as who set a price stay on the admin pricing routes.
source is org_override when a price override was set for your organization through PUT /models/pricing/org/{org_id}, cluster_default when a cluster entry uses the operator’s rate card, and unpriced (billed as 0) otherwise. An organization admin can set that override for their own organization’s active byo entries with setOrgPrice, read it back with listOrgPrices, and remove it with clearOrgPrice; an override on a cluster entry is cluster-admin only. currency is the cluster-wide ISO 4217 code, or null until the operator sets one.
Bring your own model
An organization admin can define a BYO entry in the catalog and then activate it withsource: "byo". Defining does not activate. This is the flow the Deploy a model server guide uses to register a vLLM endpoint. Only curl is shown; the SDKs’ addCatalogEntry takes the same fields.
Bash
id is the name your requests send as model; modelId is the name the gateway forwards to the provider at baseUrl. The response is a catalog entry with the auth value redacted and activation_state set to available, or available_name_blocked if another source already holds the name. Activate it as in Step 3 with source: "byo". provider is a free-form label, and entries that share one share a health result. apiProviderType is the wire protocol the gateway speaks to baseUrl; openai covers any OpenAI-compatible server. Add healthCheck to have the gateway check the provider before it sends requests (see Provider health), and rerouteRules to send some requests to a cluster model instead (see Reroute rules).
All four BYO operations need write:registry, the health check included.
Entries the operator manages in
model-registry.yaml cannot be changed through the API.
Reroute rules
rerouteRules on a BYO definition sends some of its requests to a cluster model. Each rule pairs a trigger with a targetModelId:
input-modalitywithmodality: "image"sends a request that contains an image to the target.error-fallbackretries a failed request on the target. An error from the provider, a timeout, and a failed health check all count.
base64-input, context-exceeded, and input-modality with text, audio, or video, but none of them reroutes a request. Only the first rule for each trigger is used.
The target must be a cluster catalog model, named by its canonical id rather than an alias. POST and PATCH return 422 for any other target. The target must also be active in your registry when the request runs; if it is not, an image request stays on your model and a failed request returns its original error.
Rerouting cannot loop: a request tries each model at most once. If an error-fallback target fails too, the gateway follows that target’s own error-fallback rule, set by the operator.
Rules apply to Responses and Chat Completions requests and the features built on them. When auto resolves through an LLM judge policy, the gateway ignores reroute rules, including the selected candidate’s; a direct auto override and an explicitly named model keep them. The request is billed at the price of the model that served it, which a Responses response names in resolved_model.
For cluster admins
Everything in this section needs a cluster admin bearer. The four access operations, the two reads included, also need that bearer bound to the cluster organization (
org_cluster); cluster prices and the currency need only the cluster-admin flag, from any organization. Organization admins get 403 here.available; it activates nothing. In the console this is Access → Organizations → Cluster, then the organization’s Available models tab, where you stage toggles, and Save sends the whole set through replaceCatalogOrgAccess.
The two reads require
read:registry and the two mutations write:registry, as at the organization level. Both mutations reject an unknown cluster catalog id with 422 Unknown cluster catalog model id(s): .... The cluster organization activates its own catalog with source: "byo"; "cluster" is for tenant organizations only, meaning every organization other than org_cluster.
Activation states
Model lifecycle
Catalog entries have no versions or retirement dates. Changes to the cluster catalog take effect when the operator deploys them. When a deploy removes a cluster model, every organization loses its access grant and activation for it, so requests that name it fail to resolve and anauto override that points at it reports resolvable: false.
A dated name such as openai:gpt-4.1-2025-04-14 is an alias of the undated entry, here openai:gpt-4.1. It follows the provider’s current release of that model rather than pinning the dated snapshot. The alias resolves only while the undated entry is active, GET /models does not list it, and GET /models/openai:gpt-4.1-2025-04-14 returns the entry with id: "openai:gpt-4.1".
Troubleshooting
API reference
For the full request and response schemas, open the Models group in the API Reference.See also
- Getting started - the setup wizard’s activation step.
- Deploy a model server - serve your own weights and register them as a BYO entry.
- Generate a response - call an activated model.