Feature Flags and a Soft Launch for a Risky Data Migration
A retail client needed to replace their homegrown tax calculation logic with a third-party service, because sales tax rules had grown too complex for a hand-maintained rate table to keep up with across the states they'd expanded into. The risk was obvious: a bug in tax calculation means either overcharging customers or undercharging and eating the difference, and we had no way to be fully confident in the new integration without live traffic.
We built the new tax service integration behind a feature flag using LaunchDarkly, with the flag evaluated per-order rather than per-user, targeting a configurable percentage of traffic. We started at 2% of orders, chosen specifically because it was large enough to be statistically meaningful within a day but small enough that a serious bug would affect a small, boundable number of customers rather than the whole business.
For every order in the 2% cohort, we ran both the old and new tax calculations and logged both results without acting on the new one initially — a shadow mode before the flag actually controlled behavior. This surfaced three real discrepancies in the first week: a rounding difference on orders with multiple line items, a missed exemption for one product category, and a mismatch specific to orders shipping to Colorado's home-rule cities, which have their own local tax rules the new service handled differently than expected.
We fixed each discrepancy category before increasing the rollout percentage, moving from 2% to 10% to 25% to 100% over three weeks, watching the shadow-mode discrepancy log at each step rather than a fixed calendar schedule — the criterion for advancing was zero new discrepancy categories for 48 hours, not a target date. This meant the rollout took longer than the original two-week estimate, which we'd flagged as a real possibility to the client upfront rather than something that could be scoped tightly in advance.
The flag stayed in the codebase for another month after reaching 100%, disabled but ready, as a fast rollback path rather than something we removed immediately — the cost of an extra conditional in the pricing code was trivial next to the value of a one-click revert if something surfaced under a traffic pattern the shadow-mode period hadn't hit. We only removed the flag and the old tax logic once we were confident the new path had been fully proven out.
