On-Device vs Cloud Inference for Mobile AI Features
For an iOS client building a journaling app with on-device sentiment tagging, running inference locally with Core ML was an easy call — the model was small, the task was narrow, and users cared a lot about their entries never leaving the device. That combination of small model, narrow task, and strong privacy requirement is where on-device inference earns its complexity.
The complexity is real, though. On-device models need to be quantized and often distilled down from a larger teacher model to fit reasonable app size and run within acceptable battery and thermal budgets, and that compression costs accuracy. On that journaling app, the on-device model scored about 8% lower on our sentiment eval set than the cloud equivalent, a trade-off the client accepted explicitly for the privacy guarantee.
Device fragmentation is the part people underestimate. A model that runs smoothly on a recent iPhone can be unusably slow on a three-year-old device still in active use among a meaningful share of the user base, so we build a device-tier detection layer that routes older or lower-memory devices to a cloud fallback rather than forcing a uniformly degraded on-device experience across the board.
For most features we've shipped, cloud inference still wins on pure product velocity — you can update a cloud model overnight, and you're not shipping app updates and waiting on App Store review every time you improve the model. We reserve on-device for the specific cases where latency has to be near-instant or the data genuinely can't leave the device for trust or regulatory reasons.
The hybrid pattern we default to now: cloud inference as the primary path with a lightweight on-device model as an offline fallback, keeping basic functionality available without connectivity while the richer, more capable experience runs server-side whenever a connection is available. It's more engineering up front but avoids a hard commitment to either extreme.
