Building Webhook Delivery That's Actually Reliable
The first version of our webhook system for a marketplace client did a synchronous HTTP call at the point an event occurred, inline with the request that triggered it. A slow or down partner endpoint meant the triggering request itself hung waiting on a POST that had nothing to do with the user's actual action. We moved webhook delivery entirely off the request path into a background job queue, so a partner's downtime never touches our own response times.
Retries with exponential backoff were the next requirement — partner endpoints go down for maintenance, deploy, or just have bad days, and a single failed attempt shouldn't mean a lost event. We retry at 1 minute, 5 minutes, 30 minutes, 2 hours, and 12 hours, then mark the delivery as failed and surface it in a partner-facing dashboard so they can see what didn't land rather than silently losing events.
Ordering turned out to matter more than we initially assumed. A partner receiving an 'order.shipped' webhook before 'order.created' because of retry timing caused real confusion on their end. We now include a monotonically increasing sequence number per tenant in the payload and, where the partner's system supports it, deliver events for a given resource strictly in order by holding back later events until earlier ones for that resource succeed.
Every webhook payload is signed with an HMAC using a per-partner secret, included as a header, so partners can verify the request actually came from us and wasn't replayed or forged. We document the verification steps clearly because we've seen partners skip this and just trust the payload, which becomes a real vulnerability if their endpoint URL ever leaks.
The feature partners have asked for most since launch is a replay tool — the ability to manually re-trigger a specific webhook delivery from our dashboard, for cases where they fixed a bug on their end after missing several days of events. Building that meant we had to store the full payload history for a longer retention window than we originally planned (30 days instead of 7), which was a small storage cost for a large support-burden reduction.
