Skip to main content
Every request that names a model, including model: "auto", resolves through your organization’s model registry. This guide covers the two layers behind that lookup and the operations that change them.
  • The catalog is what your organization could activate: its own bring-your-own (BYO) definitions and the cluster models a cluster admin has granted it access to.
  • The registry is what it has activated. A model name resolves only while the registry holds it.
Access to a cluster model does nothing by itself. Until someone activates the entry, GET /models omits it, a request that names it fails to resolve, and auto cannot land on it either.

Before you start

The console has the same views under Admin → Model Registry, on the Models, Auto models, Add model, and Raw JSON tabs. LLM → Models shows what your requests can use right now.

Step 1 - List the models your requests can use

GET /models returns the active entries of your registry, minus any whose definition sets hidden: true (see Troubleshooting). Read each item’s capabilities before you send sampling or reasoning parameters. Use the SDKs or the CLI to manage the model catalog and registry.
A list item looks like this:
GET /models/{model_id} returns 404 for a name that is not in the catalog, for one that is in the catalog but inactive, and for one whose definition is hidden. The gateway does not say which, so a response cannot reveal what other organizations have.

Step 2 - Read the catalog

GET /models/catalog lists every definition your organization could activate, with its source, activation state, provider health, and a price preview. Cluster entries appear only after a cluster admin grants access to them.
Field notes: The same model id can appear more than once, once per source. Only one of them can hold the name at a time.

Provider health

health holds is_available, last_health_check (a Unix timestamp in milliseconds), and error. The definition’s healthCheck sets when the gateway checks the provider: A check sends method (GET by default) to baseUrl plus path with the entry’s credentials, and any 2xx response counts as available. A result is kept for cacheTtlSeconds, which defaults to 300 seconds for preflight and to intervalSeconds plus 60 seconds for periodic. Unless the mode is off, defining or updating a BYO entry checks its provider at once, and so does checkCatalogHealth. Entries that share a provider label share one result. When the kept result says the provider is unavailable, Responses and Chat Completions requests skip it. They follow the entry’s error-fallback reroute rule if it has one, and fail otherwise. Health does not change what GET /models lists or which candidates auto routing considers.

Step 3 - Activate a catalog entry

Activation claims a model name for one source. Pass the id and the source you are activating from.
The gateway checks two things, in order:
  1. The entry exists for your organization from that source. A cluster id you have no access to, or a BYO id nobody defined, returns 422.
  2. Nobody else holds the name. If the same source already holds it, the call returns the existing activation. If a different source holds it, the call returns 409. Neither case changes anything.
The response carries model_id, source, activated_by, and activated_at. The activation applies to the next request. In the console, the Activate button on Admin → Model Registry → Models makes the same call. The setup wizard’s first step activates every available cluster entry the same way, but only while nothing in the registry is active yet.

Step 4 - Review the registry and deactivate

GET /models/registry is the catalog filtered to active entries, so each row has the same shape as in Step 2. DELETE /models/registry?id=... releases a name.
Deactivation removes the name from the registry and closes any organization price override for it (the org_override rate in Step 6). It does not touch the catalog definition, so you can activate the entry again later, from the same source or a different one. The call is idempotent: deactivated is false when nothing was active. Requests already in flight finish. The next request that names the model fails to resolve, and if an auto-model override pointed at it, auto fails for that endpoint too (see Step 5).

Step 5 - Choose what auto resolves to

The gateway applies an auto policy per endpoint. With no organization override, model: "auto" remains a direct alias to the cluster default. A direct organization override aliases auto to one exact active model. A balanced override asks a configured LLM judge to choose from a configured candidate list. The endpoint key is the model’s API format:
The SDK examples above configure the original direct policy. The following request configures balanced routing for Responses:
Bash
The candidates illustrate how a Portuguese request can route to meetkai:functionary-pt, while other requests can route to openai:gpt-4.1-mini. Use a fast, non-reasoning judge that supports structured outputs, such as openai:gpt-4.1-nano. Balanced policies are supported for responses, completions, transcriptions, and tts. The judge must be an active text LLM. Each candidate must be active and compatible with the endpoint when the policy is written. At request time, a revoked candidate is excluded, but an unavailable judge causes a transparent error. One eligible candidate is selected directly; two or more invoke the judge. PUT returns 422 when any configured model is the literal auto, is not active in your registry, or has an incompatible API format. A direct override stores the canonical id and later looks up that exact name. If you deactivate it, the override’s resolvable flag turns false and auto fails on that endpoint rather than moving to the cluster default. Clear the override or activate the model again. DELETE is idempotent. Direct requests use the body key model_id; judge-based requests use judge_model_id and candidate_model_ids. Request and response keys are snake_case; the older camelCase request keys are rejected with 400. GET /models/auto reports mode: "balanced", judge_resolvable, and resolvable_candidate_model_ids for a balanced override. Auto routing explains what the judge weighs, including modalities, file support, pricing, gateway latency, comparable evals, and conversation continuity. The console shows the same overrides on Admin → Model Registry → Auto models.

Step 6 - Check effective prices

GET /models/pricing/effective returns the rate your organization pays for every model it can call. Any bearer can read it, with no role or scope, because anyone whose requests are billed can see the rate they are billed at. Audit fields such as who set a price stay on the admin pricing routes.
source is org_override when a price override was set for your organization through PUT /models/pricing/org/{org_id}, cluster_default when a cluster entry uses the operator’s rate card, and unpriced (billed as 0) otherwise. An organization admin can set that override for their own organization’s active byo entries with setOrgPrice, read it back with listOrgPrices, and remove it with clearOrgPrice; an override on a cluster entry is cluster-admin only. currency is the cluster-wide ISO 4217 code, or null until the operator sets one.

Bring your own model

An organization admin can define a BYO entry in the catalog and then activate it with source: "byo". Defining does not activate. This is the flow the Deploy a model server guide uses to register a vLLM endpoint. Only curl is shown; the SDKs’ addCatalogEntry takes the same fields.
Bash
id is the name your requests send as model; modelId is the name the gateway forwards to the provider at baseUrl. The response is a catalog entry with the auth value redacted and activation_state set to available, or available_name_blocked if another source already holds the name. Activate it as in Step 3 with source: "byo". provider is a free-form label, and entries that share one share a health result. apiProviderType is the wire protocol the gateway speaks to baseUrl; openai covers any OpenAI-compatible server. Add healthCheck to have the gateway check the provider before it sends requests (see Provider health), and rerouteRules to send some requests to a cluster model instead (see Reroute rules). All four BYO operations need write:registry, the health check included. Entries the operator manages in model-registry.yaml cannot be changed through the API.

Reroute rules

rerouteRules on a BYO definition sends some of its requests to a cluster model. Each rule pairs a trigger with a targetModelId:
Two triggers take effect:
  • input-modality with modality: "image" sends a request that contains an image to the target.
  • error-fallback retries a failed request on the target. An error from the provider, a timeout, and a failed health check all count.
The API also accepts base64-input, context-exceeded, and input-modality with text, audio, or video, but none of them reroutes a request. Only the first rule for each trigger is used. The target must be a cluster catalog model, named by its canonical id rather than an alias. POST and PATCH return 422 for any other target. The target must also be active in your registry when the request runs; if it is not, an image request stays on your model and a failed request returns its original error. Rerouting cannot loop: a request tries each model at most once. If an error-fallback target fails too, the gateway follows that target’s own error-fallback rule, set by the operator. Rules apply to Responses and Chat Completions requests and the features built on them. When auto resolves through an LLM judge policy, the gateway ignores reroute rules, including the selected candidate’s; a direct auto override and an explicitly named model keep them. The request is billed at the price of the model that served it, which a Responses response names in resolved_model.

For cluster admins

Everything in this section needs a cluster admin bearer. The four access operations, the two reads included, also need that bearer bound to the cluster organization (org_cluster); cluster prices and the currency need only the cluster-admin flag, from any organization. Organization admins get 403 here.
Cluster admins decide which cluster models each organization may activate. A grant adds an entry to the organization’s catalog as available; it activates nothing. In the console this is Access → Organizations → Cluster, then the organization’s Available models tab, where you stage toggles, and Save sends the whole set through replaceCatalogOrgAccess.
The operations, and what each one touches: The two reads require read:registry and the two mutations write:registry, as at the organization level. Both mutations reject an unknown cluster catalog id with 422 Unknown cluster catalog model id(s): .... The cluster organization activates its own catalog with source: "byo"; "cluster" is for tenant organizations only, meaning every organization other than org_cluster.

Activation states

Model lifecycle

Catalog entries have no versions or retirement dates. Changes to the cluster catalog take effect when the operator deploys them. When a deploy removes a cluster model, every organization loses its access grant and activation for it, so requests that name it fail to resolve and an auto override that points at it reports resolvable: false. A dated name such as openai:gpt-4.1-2025-04-14 is an alias of the undated entry, here openai:gpt-4.1. It follows the provider’s current release of that model rather than pinning the dated snapshot. The alias resolves only while the undated entry is active, GET /models does not list it, and GET /models/openai:gpt-4.1-2025-04-14 returns the entry with id: "openai:gpt-4.1".

Troubleshooting

API reference

For the full request and response schemas, open the Models group in the API Reference.

See also

For application-side discovery, see Models.