Building Event-Driven Architecture Without Losing Events
We moved a subscription billing platform to an event-driven model so that order creation, inventory updates, and email notifications could happen independently instead of as one long synchronous chain. The first version published events directly from application code to SNS at the point of the database write — two separate operations with no transactional link between them.
The failure mode was quiet and expensive. During deploys, a request could succeed in writing to the database moments before the process received a SIGTERM, killing it before the SNS publish fired. We estimated afterward that this had silently dropped roughly 0.3% of order-created events over four months — not catastrophic, but enough that a customer support team had been manually reconciling missing confirmation emails without knowing why.
We fixed this with the transactional outbox pattern: instead of publishing directly, the event gets written to an outbox table in the same database transaction as the business data, guaranteeing atomicity. A separate relay process (we used Debezium reading Postgres's write-ahead log) picks up outbox rows and publishes them to SNS, then marks them delivered. If the relay crashes mid-publish, it resumes from the last unmarked row rather than losing anything.
This introduced at-least-once delivery instead of at-most-once, which meant every consumer had to become idempotent — a lesson we learned the hard way when a brief relay restart caused the inventory service to double-decrement stock for a handful of orders. We added idempotency keys derived from the event ID to every consumer, checked against a small dedup table before processing.
Event-driven architecture bought us real benefits: the email service can be down for ten minutes without affecting checkout at all, and adding a new consumer (a Slack notification for large orders) took an afternoon with zero changes to existing services. But none of that was safe until we treated 'don't lose events' as a correctness requirement enforced by the data model, not a hope enforced by careful code.
