Fine-Tuning a Small Model to Beat a Bigger One on a Narrow Task
A logistics client needed to extract structured fields — carrier, tracking number, delivery status, exception reason — from freeform shipping emails. Our first pass used a large general-purpose model with a detailed extraction prompt, and it worked, but at roughly 900 milliseconds and a meaningful per-call cost across the volume they processed, it wasn't going to scale to their target of 40,000 emails a day.
We fine-tuned a much smaller open-weight model on about 6,000 labeled examples pulled from their historical email archive, using LoRA adapters to keep training cheap and iteration fast. On their held-out test set, the fine-tuned model matched the large model's field-extraction accuracy at 96.4% and beat it on the harder exception-reason field, where the narrow training data captured carrier-specific phrasing the general model had never been tuned on.
Latency dropped to about 80 milliseconds self-hosted on a single GPU, and cost per call fell by more than 90% compared to the frontier API. The trade-off was real, though: the fine-tuned model was noticeably worse the moment the input drifted outside its training distribution, like a new carrier format we hadn't seen. We added a confidence threshold that routes low-confidence extractions to the larger model as a fallback, which kept the cost savings while catching the edge cases.
The training data mattered more than the model architecture. Our first fine-tuning attempt with 1,500 examples underperformed badly on rare exception types simply because they were underrepresented. We went back and oversampled the rare classes and added synthetic variations of real emails, which closed most of the gap without needing more raw data than we could reasonably label by hand.
The general takeaway: a big general-purpose model is the right starting point for prototyping and for tasks with wide, unpredictable input variety. Once a task is narrow, well-defined, and high-volume, a small fine-tuned model plus a fallback path is usually the better production architecture — it's just a bigger upfront investment in data and evaluation before you get there.
