Webhook Replay Quarantine for Self-Hosted SaaS Integrations

# Webhook Replay Quarantine for Self-Hosted SaaS Integrations

Webhook retries are useful until the second delivery repeats a side effect. A payment event gets processed twice. A customer receives two emails. A CRM note is appended again. A usage counter increments after the first run already succeeded.

For mid-level developers running practical self-hosted stacks, the fix is not an expensive integration platform. The fix is a replay quarantine.

A replay quarantine is a small queue between inbound webhook delivery and side-effect execution. It catches events that look risky before they can mutate state.

What To Check

Start with four checks.

1. Idempotency key. 2. Payload hash. 3. Previous side-effect marker. 4. Source timestamp window.

The idempotency key is the main guard. If the provider sends one, store it. If not, derive one from provider, event type, object id, and timestamp bucket.

The payload hash catches a harder case: the same event id arrives with different content. That should not proceed automatically. Put it in quarantine and review.

The side-effect marker records what your system already did. For example: email sent, invoice marked paid, account upgraded, usage counter written, or support note created.

Queue Shape

A simple implementation needs three queues.

The processor reads inbound events, validates the signature, checks idempotency, and decides whether to execute, quarantine, or dead-letter.

This works with Redis, Postgres, RabbitMQ, or any queue already in the stack. The important part is the contract, not the product.

Minimal Table

Use one table to track processed events.

Fields:

When a retry arrives, compare the derived key and payload hash. If the event already produced a side effect, do not run the side effect again.

Manual Release

Quarantine is only useful if someone can release safely. The review screen should show the raw payload, previous payload hash, previous side effect, and a release action.

Release should create a new audit row. Do not silently move the event back to inbound without recording who approved it and why.

What Not To Do

Do not rely only on provider retry headers. They help, but they are not your source of truth. Do not use timestamps alone. Clock skew and delayed delivery will hurt you. Do not skip signature verification because the endpoint sits behind a private network.

Practical APAC SaaS Pattern

For India, Singapore, and Australia SaaS teams, this is a low-cost reliability improvement. It fits a small Docker stack and does not assume a paid-tool budget. Start with Postgres for the event table and the queue you already operate.

CTA

TechSaaS helps teams build self-hosted SaaS infrastructure, queues, and integration reliability without unnecessary platform spend. Service CTA: https://techsaas.cloud/services