Evaluating Open-Source Models vs Frontier APIs for Client Work
A client came to us wanting to self-host an open-source model specifically to avoid per-token API costs at scale, which is a reasonable instinct once volume is high enough — but the comparison has to include GPU infrastructure, ongoing maintenance, and the engineering time to keep a self-hosted model competitive with a frontier API's steady stream of improvements, not just the sticker price per million tokens.
On raw task accuracy for their specific extraction task, the open-source model we evaluated came within about 4% of the frontier API's performance after fine-tuning on the client's data, a gap small enough that self-hosting was clearly viable from a quality standpoint. What tipped the decision was volume: at their projected request rate, self-hosted infrastructure broke even against API costs in under five months.
For a different client with lower, spikier volume — a seasonal retail application — the math went the other way entirely. Provisioning GPU capacity for peak season while it sat mostly idle the rest of the year made the API's pay-per-use pricing meaningfully cheaper even before accounting for the engineering time saved by not managing inference infrastructure at all.
Frontier APIs still won decisively on tasks requiring strong general reasoning or handling genuinely novel, unstructured inputs — the open-source models we tested needed much more careful prompting and still showed more variance on edge cases outside their fine-tuning distribution. For narrow, well-defined tasks with enough training data, that gap closed considerably; for open-ended ones, it didn't.
Our actual recommendation to clients now depends less on ideology around open versus closed models and more on three concrete questions: what's the request volume, how narrow is the task, and does the team have the operational capacity to run inference infrastructure. Answering those honestly settles the decision faster than any benchmark comparison we could hand them.
