Production cost review
LLM retry costs: the hidden multiplier on your AI bill
Every retry is a paid attempt. Count them per planned call, find which layer creates them, and price each source before you change a limit.
01 / Definition
Retry rate has two denominators, and they give different numbers.
The calculator on this site defines retry rate as average extra attempts per planned call. A 25% rate means 1.25 attempts per planned call. The Snapshot analyzer reports retries as a share of total attempts. The same traffic produces 20% under that definition, because 0.25 extra attempts out of 1.25 total is one in five.
Pick one definition before comparing numbers across teams or tools. Cost follows the first definition: total attempts equal planned calls times (1 + r), and each attempt is billed.
02 / Illustrative worked example
A 25% retry rate adds 25% to the line item.
Suppose a workload makes 100,000 planned calls in a month at an average of $0.004 per attempt, and each planned call averages 0.25 extra attempts.
These hypothetical figures illustrate the method. Retried attempts can cost more or less than the average attempt, so use measured cost per attempt from your own traces.
03 / Sources
Five places extra attempts come from.
A single retry limit rarely explains the total. Find the layer that creates each extra attempt, because each source has a different fix.
Nested retry layers
A client library retries a failed request, and your application wraps that call in its own retry loop. If the library allows 3 attempts and the application allows 3, one logical call can produce up to 9 provider attempts.
Whole-workflow restarts
A failure at the last step of a multi-call workflow restarts the workflow from step one. Every earlier call repeats even though it already succeeded.
Retries on permanent errors
An invalid request, an oversized prompt, or a schema the model cannot satisfy fails the same way every time. Retrying it adds cost and no chance of success.
Timeout retries after the provider finished
The client gives up before the response arrives, retries, and pays for both generations. Check how your provider bills requests that complete after the client disconnects.
Validation loops
The application rejects an output and asks the model again. Each pass resends the full prompt, and agent loops often resend a longer context each time.
04 / Scope of the retry
Retrying one step costs a fifth of restarting the workflow.
Take a workflow of 5 sequential calls where 10% of runs fail at the final step. Restarting the whole workflow repeats all 5 calls on those runs. Retrying only the failed step repeats one.
| Recovery policy | Extra calls per run | Added to 5 planned calls |
|---|---|---|
| Restart the whole workflow | 0.10 × 5 = 0.50 | 10% |
| Retry the failed step only | 0.10 × 1 = 0.10 | 2% |
Step-level recovery needs the intermediate results saved so the workflow can resume. Include that engineering and storage work when you compare the two policies.
05 / Measurement
Log the reason for every extra attempt.
- Tag each attempt.Record the planned-call ID, attempt number, retry layer, and error class so retries separate from intentional fan-out.
- Count attempts, not requests.Export total provider attempts, then compute extra attempts per planned call and retries as a share of attempts.
- Price retried attempts.Use the token counts and provider charge on the retried attempts, not the account average.
- Separate permanent from transient failures.Only transient failures justify a retry. Route permanent errors to a terminal path.
- Check the effect on success.A lower retry ceiling saves money and can reduce the success rate. Compare cost per successful outcome, not spend alone.
See retry share and unknown fields in your usage data.
The Snapshot analyzer runs in your browser and reports retry share of attempts when your file includes the fields.
06 / Common questions
Apply the numbers to common cases.
Are failed requests billed?
It depends on the provider and the failure type. Check the billing documentation and your own invoices for errors, timeouts, and cancelled requests before assuming they are free.
What retry rate is normal?
No public benchmark applies to every workload. Snapshot flags a retry share of 5% or more as worth investigating, which is a review threshold and not an industry standard.
Should I remove retries to save money?
Not without measuring success. Retries recover transient failures. Cap them, add backoff, and stop retrying permanent errors, then check the effect on cost per successful outcome.
How do I model retries in the calculator?
Enter average extra attempts per planned call. Keep intentional fan-out in calls per workflow so it does not count as a retry.
07 / Limitations
Averages hide bursts and uneven costs.
Retries cluster during provider incidents and traffic spikes, so a monthly average understates peak cost and rate-limit pressure. Retried attempts also differ in length from first attempts. Treat modeled savings as an estimate and verify the change against a normalized baseline over a stated observation window.