Production cost review
Production LLM cost-optimization checklist
Find the source of a production workload's cost, choose one bounded change, and verify the result against quality and operating requirements.
Before changing the system
Attribute workload costs before choosing a tactic.
Caching, batching, smaller models, shorter prompts, and routing can all reduce a line item. Choose the first change after you define the workload and outcome, establish a comparable baseline, and locate the source of the cost. Test one bounded alternative against the same operating constraints.
Attribute→Trace→Change→Evaluate→Verify
This checklist identifies evidence to gather and experiments to run. Use production measurements to calculate the impact of each change.
01 / Checklist
Baseline one production workload
- Look for
- Eligible workflows, model attempts, input and output token categories, provider charges, review labor, fixed operating cost, successful outcomes, latency, errors, and the observation period.
- Decision
- Choose the workload with a material bill or decision. Do not optimize a blended account total that hides products, customers, or task types.
- Evidence needed
- A period total tied to one workload and a defined cost-per-request, cost-per-workflow, or cost-per-successful-outcome denominator.
02 / Checklist
Map the complete call path
- Look for
- Direct SDK calls, shared wrappers, gateways, agent steps, tool loops, regenerations, evaluators, fallbacks, background jobs, and calls triggered by one user action.
- Decision
- Draw the path from business event to final outcome. Mark intentional fan-out separately from retries and accidental duplicate work.
- Evidence needed
- Source locations plus runtime traces or request identifiers that show how many model attempts one workflow creates.
03 / Checklist
Measure input and context growth
- Look for
- Repeated system prompts, tool definitions, retrieved passages, conversation history, examples, boilerplate, and context that grows on each agent step.
- Decision
- Remove content only after checking why it is present. Change retrieval count, history policy, tool exposure, or prompt structure one mechanism at a time.
- Evidence needed
- Input-token distributions by workload and call site, not one account-wide average. Keep cached input identified as a subset of total input.
04 / Checklist
Bound outputs and reasoning effort
- Look for
- Loose maximum-token settings, verbose formats, unused explanation, repeated structured fields, reasoning-token behavior, and generations discarded by downstream validation.
- Decision
- Set the shortest output contract that still satisfies the task. Prefer explicit schemas and stop conditions where the model and provider support them.
- Evidence needed
- Output and separately reported reasoning-token distributions, truncation rate, schema failures, task success, and downstream review time.
05 / Checklist
Control retries, fallbacks, and loops
- Look for
- Unbounded retries, retries for permanent errors, whole-workflow restarts, nested library retries, fallback cascades, agent iteration limits, and repeated calls after client timeouts.
- Decision
- Give transient failures bounded backoff and permanent failures a terminal path. Preserve the reason for each extra attempt so retry cost can be separated from intentional fan-out.
- Evidence needed
- Attempts per planned call, retry reasons, error classes, fallback transitions, terminal outcomes, latency, and cost per successful workflow.
06 / Checklist
Test caching against the traffic shape
- Look for
- Stable prompt prefixes eligible for provider prompt caching and exact or safely reusable responses eligible for application caching.
- Decision
- Model prompt caching and full-response caching separately. Include lookup, write, storage, invalidation, privacy, staleness, and miss costs before implementation.
- Evidence needed
- Eligible traffic, observed hit rate, token discount or avoided calls, added latency, invalidations, correctness checks, and total net operating change.
07 / Checklist
Batch or move work off the synchronous path
- Look for
- Offline classification, enrichment, evaluation, embeddings, summarization, or other work that does not require an immediate user response.
- Decision
- Batch only where waiting, failure recovery, ordering, and data-handling requirements permit it. Do not trade away an actual latency requirement for a nominal unit-price discount.
- Evidence needed
- Queue delay, completion time, batch utilization, failure recovery, throughput, provider charge, and the same task-level quality measure.
08 / Checklist
Route only with a representative evaluation
- Look for
- Task classes where a smaller model, constrained prompt, local model, or cascade might meet the existing quality and reliability floor.
- Decision
- Reserve expensive paths for requests that need them. Include router calls, misroutes, fallbacks, retries, operating labor, and infrastructure when calculating the alternative.
- Evidence needed
- A representative evaluation set, per-segment quality, latency, route share, fallback rate, full cost per successful outcome, and rollback criteria.
09 / Checklist
Verify savings against a normalized baseline
- Look for
- Changes in traffic volume, provider price, workload mix, task difficulty, quality, reliability, latency, review effort, and fixed commitments during the comparison window.
- Decision
- Report measured spending, modeled savings, implementation expense, cost avoidance, and verified savings as different quantities. Keep the current system when the evidence does not justify a change.
- Evidence needed
- A stated baseline and observation window, comparable workload segments, constraint results, implementation and recurring costs, and an owner for ongoing monitoring.
Review boundary
Limits of source review.
A repository can show configured models, retry branches, prompt assembly, cache calls, output limits, and instrumentation gaps. Production data establishes traffic, provider charges, cache hits, customer outcomes, quality, and savings. Treat source findings as implementation observations and test their production impact with runtime evidence.
Run the source review locally
Get the review skill Use the open Production AI Cost Review skill.
It maps call paths and evidence gaps, then lists the runtime data needed to calculate savings.