Stop AI retries from multiplying token usage
Prevent client and gateway retry amplification with logical request IDs, in-flight deduplication, retry budgets, and strict streaming boundaries.
A user may click Retry once, the desktop client may retry automatically, and the gateway may retry a transient upstream failure. Without one shared logical identity, a single visible action can produce several billable provider requests.
How retry amplification happens
If a client makes three attempts and a gateway independently allows two provider attempts for each one, the user's single action can reach the provider six times. Rate limits and timeouts become worse, while token usage appears disconnected from what the client UI shows.
3 client attempts × 2 gateway attempts = up to 6 provider requests
A retry budget must apply to the logical request across layers, not reset every time the request crosses a process boundary.
Assign one logical request identity
Prefer a client-supplied idempotency identifier. When one is absent, a gateway can derive a stable fingerprint from the authenticated key, route, and canonical request body. The value should never include the raw API key.
- Store the logical request ID with every provider attempt and usage record.
- Reuse an existing in-flight promise or distributed lock for identical work.
- Return the completed result when the protocol safely permits replay.
- Expire deduplication records after a bounded interval.
In a replicated gateway, process-local maps are insufficient. The in-flight claim must be coordinated through shared state such as Redis.
Never retry after visible streaming starts
Before the first response byte is delivered, a stalled connection may still be safely replaceable. After text or a tool-call fragment becomes visible, a retry can duplicate output and provider usage while the client sees one conversation.
Treat the first visible content event as a hard boundary: settle the partial request accurately if the stream later fails, but do not start another provider generation automatically.
Retry only narrow transient failures
Reasonable retry candidates are failures that happen before visible output and are likely to be temporary:
- connection establishment timeouts,
- selected rate-limit responses with a bounded
Retry-After, - temporary upstream 5xx responses,
- connection resets before response headers.
Authentication failures, invalid parameters, unsupported models, exhausted balances, and deterministic context-limit errors should not be retried. Add jitter, an overall deadline, and a small attempt cap.
Measure the logical request and every attempt
| Metric | Why it matters |
|---|---|
| Logical request ID | Joins client retries, gateway attempts, and usage settlement. |
| Client retry count | Shows whether the caller is amplifying failures. |
| Provider attempt count | Reveals internal retry cost. |
| Time to first token | Separates connection delay from generation time. |
| Visible-output flag | Proves whether another attempt was allowed safely. |
| Deduplication outcome | Shows whether duplicate in-flight work was prevented. |
CloudService caps gateway retry attempts, does not retry after visible streaming begins, and coordinates identical in-flight requests across API replicas. Client-side retries still need bounded settings because the client controls when it creates a new logical request.