Classify the failure
Check status on Responses calls
A Responses call can fail after the gateway has accepted it. If the model’s provider rejects a non-streamed request, the gateway still returns HTTP 200. The body is a response object with status: "failed" and an error object that carries code and message. Check status before you read the output; a 200 alone does not mean the call succeeded.
error.code names the kind of failure. invalid_request_error means the provider rejected the request itself, so correct it before retrying. rate_limit_exceeded and server_error can be retried with the backoff described below.
A streamed call that fails this way ends with a response.failed event that carries the same object. A background response shows the same status and error when you retrieve it. Requests the gateway rejects before calling a model, such as one with an out-of-range field, still return a 4xx status.
Back off between attempts
For a retryable failure, wait progressively longer between attempts, add jitter so concurrent clients do not retry together, and bound both the attempt count and total time. HonorRetry-After when an endpoint returns it.
Check the selected SDK’s retry settings before adding another retry loop. Nested retry policies can multiply the number of requests.
Avoid duplicate writes
A timeout does not prove the server rejected the request. Before repeating a create operation, determine whether the resource was created. Where an endpoint supports an idempotency key, reuse the same key only for an identical retry. Generate a new key for a new operation. Do not assume that an idempotency header is supported by every API. For Compute, use the examples in Run a fine-tune job and Deploy a model server.Handle streaming and background work
A streaming connection can end after delivering partial output. Avoid treating the partial output as a completed result. Follow the status and retrieval mechanisms described in Background responses.Separate refusals from transport errors
A guardrail can return a refusal in a successful HTTP response. Inspect the output; retrying the same blocked request is not a transport-recovery strategy.Investigate a failure
- Confirm the gateway URL and caller identity.
- Record the request identifier, resource ID, status, and error body without logging credentials.
- Check Rate limits, Budgets, and Resource permissions as appropriate.
- Use CLI debugging to inspect request construction.
- Correlate the request with Usage and auditing.