The Economics of Self-Hosting Open-Source LLMs vs. API Calls
The pitch for self-hosting is usually 'it's cheaper at scale.' We wanted actual numbers, so across three client workloads we compared the fully loaded cost of a self-hosted open-weight model against equivalent API usage, including GPU rental, engineering time for deployment and monitoring, and the utilization rate we could realistically achieve rather than theoretical peak throughput.
For a moderate-volume workload — around 500,000 tokens a day — API calls to a hosted provider were cheaper than self-hosting once we accounted for the GPU sitting mostly idle outside business hours. A single always-on A100 instance cost more per month than the API bill, because our utilization was under 15% across a 24-hour period. Self-hosting only won when we could keep utilization consistently above roughly 40%, which meant batching or serving multiple internal use cases off the same instance.
The economics flipped for a high-volume client processing north of 8 million tokens a day with steady traffic. There, self-hosting a fine-tuned 8B parameter model on reserved GPU capacity cut costs by about 65% compared to equivalent API usage, and latency improved too since we weren't paying the network round trip to an external provider. The upfront cost was real, though — roughly six weeks of engineering time to get deployment, autoscaling, and monitoring production-ready.
The hidden cost we underweighted at first was model maintenance. Frontier API providers ship quality improvements for free; self-hosted models need us to actively track new open-weight releases, re-evaluate them against our task, and manage the migration. We now budget for a recurring evaluation cycle every quarter rather than treating the self-hosted model as a one-time deployment decision.
Our rule of thumb now: below roughly 1-2 million tokens a day of steady traffic, API calls almost always win once engineering time is priced in. Above that, with consistent (not spiky) usage, self-hosting is worth a serious cost model. Spiky or unpredictable traffic favors APIs regardless of volume, since idle GPU capacity erodes the savings fast.
