Building a Camera-Heavy Feature: Performance Lessons
A fintech client's ID and document-scanning feature — live camera preview with real-time edge detection to guide users to a good scan — was dropping frames noticeably on mid-range Android devices and causing visible device heating within about three minutes of continuous use on both platforms. The initial implementation ran edge detection on every single camera frame at full preview resolution, which was far more processing than the feature actually needed.
The first fix was resolution mismatch — we were running detection on frames at the camera's full preview resolution (often 1920x1080 or higher) when detection accuracy for guiding a scan didn't need anywhere near that. Downscaling frames to roughly 640x480 before running edge detection cut processing time per frame by more than 70% with no perceptible loss in guide accuracy, since the guide overlay itself is drawn at full resolution separately from the detection input.
We also moved from processing every frame to a throttled cadence — detection running at roughly 10 frames per second rather than the full 30 or 60fps camera feed, since human perception of the guide overlay updating doesn't need to match raw camera frame rate to feel responsive. This was implemented differently per platform: on iOS using AVCaptureVideoDataOutput with a delegate queue that drops frames when processing falls behind rather than queuing them, and on Android using CameraX's ImageAnalysis use case configured with STRATEGY_KEEP_ONLY_LATEST so a backed-up analyzer doesn't create a growing backlog.
On-device ML for edge detection (we used a lightweight custom model rather than a general-purpose vision framework, since accuracy requirements were narrow — rectangular document edges against typical backgrounds) ran on the CPU initially and moved to GPU delegation once we set up TensorFlow Lite's GPU delegate on Android and Core ML's automatic compute unit selection on iOS, cutting per-frame inference time roughly in half and meaningfully reducing the sustained CPU load causing the heating.
Memory management around the camera buffers themselves needed explicit attention too — early versions were retaining references to CMSampleBuffer objects on iOS longer than necessary within the detection pipeline, causing memory pressure during long scanning sessions. Making sure buffers were released promptly after each detection pass, rather than relying on ARC to clean them up eventually, fixed a slow memory climb that had been causing occasional crashes on longer scanning sessions with multiple retries.
After these changes, sustained frame rate held steady at the target 10fps detection cadence with no visible preview stutter, and device heating during a typical 90-second scanning session became imperceptible to testers. The general lesson: camera-heavy features fail on performance not because of one big mistake but because it's easy to default to 'process everything at full quality' when the feature only actually needs a fraction of that fidelity.
