Prompt Caching and Its Real Impact on Latency and Cost
Prompt caching cuts cost and latency by letting the API skip reprocessing a prompt prefix it's seen recently, but the first time we enabled it on a client's production endpoint, cache hit rate sat at around 12% — far lower than expected — because the system prompt included a timestamp that changed on every request, invalidating the cache before it could ever be reused.
Once we pulled the timestamp and other per-request variables out of the cached prefix and into a separate suffix, hit rate jumped to over 85% on that same endpoint, and time-to-first-token on cached requests dropped by roughly 60%. The lesson generalized to nearly every project since: anything that needs to vary per request has to live after the stable, cacheable portion of the prompt, never inside it.
The cost savings compound with scale in a way that changes architectural decisions. For a high-volume classification endpoint with a long, detailed system prompt and few-shot examples, caching turned what had been a genuinely expensive per-request system prompt into something close to free on cache hits, which made it viable to keep a much richer, more detailed prompt than we'd have used otherwise.
Cache lifetime matters for bursty traffic. One client's usage pattern had long gaps between requests during off-hours, long enough that the cache expired between calls, and hit rate looked great in load testing but disappointing in production until we understood the actual request cadence and adjusted expectations rather than chasing a caching bug that didn't exist.
We now treat prompt structure — what's stable versus what's dynamic — as a first-class design decision from the start of a project rather than an optimization applied after the fact, because retrofitting caching onto a prompt that interleaves stable and dynamic content throughout is far more disruptive than designing for it up front.
