Debugging Intermittent Mobile Bugs That Only Happen in Production
A client's app was seeing a crash affecting about 0.4% of sessions, concentrated on a specific set of Samsung devices running Android 13, that we couldn't reproduce on any device in our own test lab. The first real progress came from stopping the reproduction attempt and instead treating the crash reporter, Firebase Crashlytics, as the primary data source — filtering by device model, OS version, and app version to find the exact intersection where the crash rate spiked, rather than trying to guess at a root cause first.
Breadcrumbs made the difference between a stack trace and an actual understanding of what happened. We instrument custom logging — non-fatal breadcrumb events for key state transitions like screen appeared, network request started, cache read attempted — that get attached to crash reports, which turned an opaque null-pointer crash in a caching layer into a clear picture: it only happened when a background app-refresh cache write raced against a foreground read on devices with slower flash storage.
Remote configuration let us test a fix hypothesis without a full release cycle. We used a feature flag, LaunchDarkly in this case, to gate the suspected fix — adding a mutex around the cache access — to 5% of production traffic on the affected device profile, watched the crash rate for that cohort over 48 hours, confirmed it dropped to zero, then rolled it out to 100% with actual release confidence.
Device farms filled the reproduction gap when we needed hands-on debugging rather than just crash report data. BrowserStack's real-device cloud let us get an actual session on the specific Samsung model and OS combination, which surfaced a manufacturer-specific quirk in how that device's OEM Android skin handled low-memory background app termination — a case standard Android emulators don't replicate.
The broader practice we've built from repeated encounters with this class of bug is to treat production telemetry as a first-class debugging tool, not just a monitoring dashboard checked after the fact. Session-replay-style breadcrumbs, feature-flag-gated rollouts for risky fixes, and access to a real-device cloud for the long tail of device-specific quirks together cut our average time-to-fix for production-only bugs from what used to be one to two weeks of frustrated guessing down to two or three days on recent projects.
