Your webhooks will fail. Design for that instead of hoping it doesn't happen

Webhooks look simple: an event happens somewhere, an HTTP request gets sent to your endpoint, you process it. In practice they are one of the more failure-prone parts of any integration, because they combine network unreliability, out-of-order delivery, and — unlike an API you call yourself — a source you don't fully control the retry behavior of.

Verify before you trust

An endpoint with a guessable or unauthenticated URL is an open door. Every serious webhook provider signs its payloads with an HMAC signature and a shared secret; verify that signature before processing anything, and reject requests that fail verification rather than logging and continuing. Timestamp the signature check too, to reject replayed requests outside a short validity window.

Design for duplicate delivery

Most providers guarantee at-least-once delivery, not exactly-once — meaning the same event can legitimately arrive twice. An endpoint that isn't idempotent will double-charge a customer or double-fulfil an order on a retry. Store the event ID and check it before processing; if you've seen it, acknowledge and skip.

• Use the provider's event ID as an idempotency key, stored with a short TTL matching the provider's retry window
• Make the actual side effect idempotent too, not just the logging — 'has this order already been marked paid' beats 'have I seen this webhook ID'
• Respond with a 2xx only after the event is durably queued or processed, never before
• Respond fast — acknowledge within a few seconds and do slow processing asynchronously, since most providers time out and retry a slow response as a failure

Handle out-of-order arrival

Network retries and provider-side queueing mean events don't always arrive in the order they were generated. A 'subscription cancelled' event arriving before 'subscription created' due to a retry will corrupt state if the handler assumes strict ordering. Include a sequence number or timestamp in the payload and check it against the last known state before applying an update.

When your endpoint is down

• Confirm what the provider's retry policy actually is — exponential backoff over hours, or a handful of retries over minutes — and don't assume it covers an extended outage
• For anything that can't tolerate silent loss, poll the provider's API on a schedule as a backstop, independent of webhook delivery
• Route processing failures to a dead-letter queue for manual review rather than silently dropping them after retries are exhausted
• Alert on a sustained drop in webhook volume, not just on errors — a silent integration break often looks like zero traffic, not failed requests

The mindset shift

Treat every webhook handler as if the network will duplicate, delay, reorder or drop the request, because eventually it will. The endpoints that hold up in production are boring: verify, dedupe, ack fast, process async, and keep a manual review path for anything that falls through.

Sources