Designing Audit Logging That's Actually Useful During an Incident
A client's existing audit log recorded events like 'order updated' with a timestamp and a user ID — technically an audit trail, but useless during an actual incident because it didn't capture what changed. When a support escalation came in about a price that had silently changed on an order, we had no way to answer 'changed from what, to what, by whom, and through which code path' without digging through application logs that weren't designed for this.
We rebuilt it to capture a before-and-after diff for every mutation on audited entities, not just the fact that a mutation occurred — storing the changed fields as a JSON diff rather than the full row, which kept storage reasonable while still answering the actual question people ask during an incident. Alongside the diff, every entry records the actor (user or system process), the request's correlation ID, the IP address, and whether the change came from the API, an admin panel action, or an internal script.
Distinguishing system-initiated changes from user-initiated ones mattered more than we expected. A batch job correcting a data inconsistency and a support agent manually editing a record both mutate the same field, but they mean very different things during an incident review, and our original logging conflated them under a generic 'system' actor that made both look the same.
We made audit log writes happen in the same database transaction as the mutation they're recording, rather than as an async afterthought, because an audit log that can silently fail to write is worse than useless — it creates false confidence that a trail exists. The cost is a small amount of extra write latency, which we've found acceptable given what it prevents.
The real test came during an actual incident: a data correction script had a bug and modified more rows than intended. Because every change was diffed and attributed, we could query exactly which rows the script touched, what they looked like before, and revert precisely those rows — a process that took about twenty minutes instead of the multi-hour manual reconciliation the old logging would have required.
