Before you ship: timeouts, retries and tool calls

Once integration works and before you turn up the volume, this page collects the few things that tend to bite in production. The details live on their own pages; this one doesn't repeat them, it tells you how to decide.

Tool calls (function calling)

tools, tool_choice and friends are passed straight through. We don't rewrite them and we don't probe for capability.

So: whether they work, and how well, depends on the model you picked. The same code can stop working when you switch models — some models don't recognise these fields at all (they ignore them), others support them with differences in the details. Follow that model's own documentation, and re-test after every model change.

When a model ignores fields it doesn't know, you get a perfectly normal reply that simply has no tool_calls — no error is raised. So any tool-calling path needs a branch for "no tool_calls came back".

Choosing timeouts

Two cases:

• Non-streaming: the whole request waits for the model to finish writing. Longer output means longer waits, so size the timeout by the longest output you allow, not by "how fast it usually is".
• Streaming: watch time to first byte, not total duration. A long gap between chunks usually means the upstream is queueing.

The gateway also enforces a hard deadline of its own; exceeding it returns 504. Nothing was produced and you are not charged.

⚠️ Retries: deal with double billing first

The gateway does no cross-request de-duplication (matching OpenAI's behaviour on this endpoint) — re-sending the same logical call after a timeout is treated as a brand-new request and genuinely charged again, while the first attempt may already have completed upstream.

So, in order:

1. Turn the SDK's automatic retries off or down (the official openai SDK has them on by default);
2. When you do need retries, de-duplicate on your side (record against an order id whether this call was already sent);
3. On rate_limit_exceeded, back off by the Retry-After in the response rather than a fixed interval — a fixed interval makes you compete with yourself at peak;
4. Log the request_id of every call. When a double charge happens, it is the only thread that ties the two together.

Concurrency

Rate limits apply to your tenant and your key. When you hit one you get rate_limit_exceeded, and the message states which dimension was exceeded (a specific model, your plan, or the tenant as a whole). Read that sentence before changing anything — slowing down one model and reducing overall concurrency are different fixes.

Before scaling up, ramp: run an hour of real traffic, look at the latency distribution and failure rate in usage details, then increase.

Pre-launch checklist

• Keys live in a secret store — not in code, not in a front-end bundle, not in CI logs;
• The production key has an IP allowlist and a budget cap (being blocked by the allowlist returns 401 invalid_api_key, the same code as an invalid key — check the source IP first when debugging a 401);
• SDK automatic retries handled per point 1 above;
• request_id is written to your logs;
• There is a fallback path for "model unavailable" (switch models, or queue and retry);
• The balance is sufficient and someone is watching the low-balance notifications (at zero, new requests are refused — see Top-ups and refunds).