Evidence-first production review
Production LLM cost-optimization checklist
Find what makes a production workload expensive, choose one bounded change, and verify the result without treating lower token spend as proof of better economics.
Before changing the system
Optimization starts with attribution, not a tactic.
Caching, batching, smaller models, shorter prompts, and routing can all reduce a line item. None is automatically the right first change. The useful sequence is to define the workload and outcome, establish a comparable baseline, locate the mechanism creating cost, test one bounded alternative, and verify the result against the same operating constraints.
This checklist is a review method, not a savings estimate. A finding identifies evidence to gather or an experiment to run; it does not establish production impact by itself.
01 / Checklist
Baseline one production workload
- Look for
- Eligible workflows, model attempts, input and output token categories, provider charges, review labor, fixed operating cost, successful outcomes, latency, errors, and the observation period.
- Decision
- Choose the workload with a material bill or decision. Do not optimize a blended account total that hides products, customers, or task types.
- Evidence needed
- A period total tied to one workload and a defined cost-per-request, cost-per-workflow, or cost-per-successful-outcome denominator.
02 / Checklist
Map the complete call path
- Look for
- Direct SDK calls, shared wrappers, gateways, agent steps, tool loops, regenerations, evaluators, fallbacks, background jobs, and calls triggered by one user action.
- Decision
- Draw the path from business event to final outcome. Mark intentional fan-out separately from retries and accidental duplicate work.
- Evidence needed
- Source locations plus runtime traces or request identifiers that show how many model attempts one workflow actually creates.
03 / Checklist
Measure input and context growth
- Look for
- Repeated system prompts, tool definitions, retrieved passages, conversation history, examples, boilerplate, and context that grows on each agent step.
- Decision
- Remove content only after checking why it is present. Change retrieval count, history policy, tool exposure, or prompt structure one mechanism at a time.
- Evidence needed
- Input-token distributions by workload and call site, not one account-wide average. Keep cached input identified as a subset of total input.
04 / Checklist
Bound outputs and reasoning effort
- Look for
- Loose maximum-token settings, verbose formats, unused explanation, repeated structured fields, reasoning-token behavior, and generations discarded by downstream validation.
- Decision
- Set the shortest output contract that still satisfies the task. Prefer explicit schemas and stop conditions where the model and provider support them.
- Evidence needed
- Output and separately reported reasoning-token distributions, truncation rate, schema failures, task success, and downstream review time.
05 / Checklist
Control retries, fallbacks, and loops
- Look for
- Unbounded retries, retries for permanent errors, whole-workflow restarts, nested library retries, fallback cascades, agent iteration limits, and repeated calls after client timeouts.
- Decision
- Give transient failures bounded backoff and permanent failures a terminal path. Preserve the reason for each extra attempt so retry cost can be separated from intentional fan-out.
- Evidence needed
- Attempts per planned call, retry reasons, error classes, fallback transitions, terminal outcomes, latency, and cost per successful workflow.
06 / Checklist
Test caching against the traffic shape
- Look for
- Stable prompt prefixes eligible for provider prompt caching and exact or safely reusable responses eligible for application caching.
- Decision
- Model prompt caching and full-response caching separately. Include lookup, write, storage, invalidation, privacy, staleness, and miss costs before implementation.
- Evidence needed
- Eligible traffic, observed hit rate, token discount or avoided calls, added latency, invalidations, correctness checks, and total net operating change.
07 / Checklist
Batch or move work off the synchronous path
- Look for
- Offline classification, enrichment, evaluation, embeddings, summarization, or other work that does not require an immediate user response.
- Decision
- Batch only where waiting, failure recovery, ordering, and data-handling requirements permit it. Do not trade away an actual latency requirement for a nominal unit-price discount.
- Evidence needed
- Queue delay, completion time, batch utilization, failure recovery, throughput, provider charge, and the same task-level quality measure.
08 / Checklist
Route only with a representative evaluation
- Look for
- Task classes where a smaller model, constrained prompt, local model, or cascade might meet the existing quality and reliability floor.
- Decision
- Reserve expensive paths for requests that need them. Include router calls, misroutes, fallbacks, retries, operating labor, and infrastructure when calculating the alternative.
- Evidence needed
- A representative evaluation set, per-segment quality, latency, route share, fallback rate, full cost per successful outcome, and rollback criteria.
09 / Checklist
Verify savings against a normalized baseline
- Look for
- Changes in traffic volume, provider price, workload mix, task difficulty, quality, reliability, latency, review effort, and fixed commitments during the comparison window.
- Decision
- Report measured spending, modeled savings, implementation expense, cost avoidance, and verified savings as different quantities. Keep the current system when the evidence does not justify a change.
- Evidence needed
- A stated baseline and observation window, comparable workload segments, constraint results, implementation and recurring costs, and an owner for ongoing monitoring.
Review boundary
What source code cannot prove.
A repository can show configured models, retry branches, prompt assembly, cache calls, output limits, and instrumentation gaps. It cannot establish live traffic, actual provider charges, realized cache hits, customer outcomes, representative quality, or verified savings. Treat source findings as observations about implementation and as hypotheses about production impact until runtime evidence confirms them.
Use the open Production AI Cost Review skill.
It maps call paths and evidence gaps without claiming savings from code alone.