Inference architecture economics
API vs. self-hosted AI: compare total cost, not GPU price.
Put both options against the same workload, success rule, service level, and cost boundary. Then test where the answer changes.
01 / Decision boundary
Compare one production workload under equal constraints.
Start with a named workload, a defined month, and a successful business outcome. Give each option the same eligible traffic, task-quality floor, latency target, availability requirement, data controls, and recovery expectations. A cheaper system that fails a required constraint is not the lower-cost equivalent.
Do not begin with average tokens alone. Record request arrivals by time interval, concurrent demand, input and output distributions, cache eligibility, retry and fallback behavior, and the measured success rate. These inputs affect an API bill and the capacity needed to serve the same traffic yourself.
02 / Cost boundary
Build two complete monthly cost models.
| Cost category | API | Self-hosted |
|---|---|---|
| Inference | Input, cached input, output, batch, retries, fallbacks | Accelerator, CPU, memory, storage, and network runtime |
| Capacity | Committed spend, quotas, reserved throughput | Peak replicas, warm reserve, redundancy, and unused capacity |
| Platform | Gateway, observability, queues, and storage | Serving stack, orchestration, registry, observability, and licenses |
| People | Integration, evaluation, incident response, vendor management | ML/platform engineering, optimization, upgrades, security, on-call |
| Change | Provider migration and application changes | Evaluation, deployment, migration, hardening, and training |
| Exceptions | Review, correction, and escalation | Review, correction, and escalation |
Cloud accelerator price is one line, not self-hosted TCO. Likewise, the API invoice is not the whole API cost. Apply an agreed loaded labor rate to recurring work, and show one-time implementation cost separately or amortize it over an explicit decision horizon.
Purchasing terms change the risk allocation. For example, AWS documents On-Demand capacity without a long-term commitment and separate commitment, Spot, and reservation options with different economics and constraints. Model the purchasing option you can actually operate, not the lowest visible rate. See the official EC2 purchasing-options documentation and On-Demand documentation.
03 / Capacity and utilization
Size for the arrival pattern and service level.
Benchmark a representative model, quantization, input/output mix, batching policy, and serving stack. Find the sustainable per-replica throughput that still meets the latency and error targets. Then size the production floor:
Average GPU utilization cannot answer this alone. Queueing, request length, batching, cold starts, model loading, deployment headroom, and failure domains determine how much paid capacity can do useful work. NVIDIA's Triton documentation, for example, treats batching as a throughput-versus-latency configuration and exposes request, execution, queue, failure, and GPU metrics for measurement. See the official batching and metrics references.
Capacity arrives in steps.
A workload may fit on two production replicas until one traffic or latency threshold requires a third. The self-hosted cost curve jumps at that point. Calculate break-even within each tested capacity band and run low, expected, and peak-demand cases.
04 / Break-even
Use the equation only after the options are comparable.
The denominator must be positive. The resulting volume must also fit inside the capacity band used for the self-hosted estimate. If it crosses the next replica threshold, recalculate with the higher fixed cost. Report the volume range where each answer holds rather than one permanent crossover point.
API pricing can also be nonlinear. Cached input, asynchronous batch work, tiered commitments, or reserved capacity can change the marginal rate. OpenAI currently documents an asynchronous Batch API with a completion window and discounted inputs and outputs; it is relevant only when that completion behavior fits the workload. See the official Batch API documentation.
05 / Illustrative worked example
Lower infrastructure spend does not guarantee lower TCO.
Consider a hypothetical workload with two million planned requests per month, 2,000 input tokens and 500 output tokens per request. The figures below are assumptions for demonstrating the method, not current provider quotes, benchmark results, or a client result.
| Metric | API | Self-hosted |
|---|---|---|
| Usage or accelerator runtime | $17,600 | $6,570 |
| Supporting infrastructure | $1,000 | $1,500 |
| Platform and observability | Included above | $1,000 |
| Recurring operating labor | $2,000 | $10,000 |
| Implementation amortization | $0 | $2,000 |
| Total monthly cost | $20,600 | $21,070 |
| Successful outcomes | 1,880,000 | 1,800,000 |
| Cost per successful outcome | $0.0110 | $0.0117 |
The API usage assumption is $2 per million input tokens and $8 per million output tokens, plus 10% full-request retry usage. The API scenario succeeds on 94% of planned requests. The self-hosted scenario assumes three replicas running 730 hours at $3 per hour and a 90% success rate. Both success rates must come from representative task evaluation; neither should be inferred from infrastructure metrics.
Self-hosting cuts the illustrative compute line by $11,030 but does not win the full comparison. Labor, platform cost, implementation, and fewer successful outcomes reverse the apparent saving. At higher sustained demand, the same capacity might win; at the next replica step, the result could reverse again.
06 / Decision and validation
Choose the reversible next test.
- Define equivalence.Write the success, quality, latency, reliability, security, and data requirements that both options must meet.
- Measure demand.Use request and token distributions, concurrency, arrival peaks, retries, fallbacks, and growth scenarios—not one monthly average.
- Benchmark the candidate.Test representative inputs and outputs at rising concurrency. Record throughput, latency percentiles, queue time, errors, and task success.
- Price production capacity.Apply replica floors, redundancy, deployment headroom, purchasing terms, supporting infrastructure, and expected idle time.
- Add people and change.Estimate implementation, evaluation, security, upgrades, incident response, on-call work, and the opportunity cost of the team doing it.
- Run sensitivity cases.Vary demand, request length, success, labor, price, commitment, and capacity thresholds. Name the assumptions that change the answer.
- Pilot before migration.Use a bounded workload or traffic slice, preserve rollback, and compare both paths over the same observation window.
Export a workload-specific inference plan.
Model routing, cache, retry, quality, and latency assumptions before pricing the production infrastructure and labor around them.
07 / Sources and limits
Refresh prices; preserve the method.
This guide was reviewed against official AWS EC2 purchasing and On-Demand documentation, OpenAI Batch API documentation, and NVIDIA Triton batching and metrics documentation on September 10, 2026. Those sources support platform mechanics, not the illustrative rates. Obtain dated quotes for the providers, regions, hardware, commitments, support plans, and software you would actually use.
A TCO model cannot prove task quality, production throughput, or operational readiness. Benchmark representative work and validate a bounded production change. Regulatory, residency, licensing, model availability, and internal security requirements may decide the architecture before economic break-even.