Webhook Security: Signature Verification, Replay Protection, And Idempotency

Webhooks look simple. Expose an endpoint, receive JSON, do something useful.

In practice, they’re one of the easiest ways to accidentally punch a hole straight through your backend. Webhook Security: Signature Verification, Replay Protection, And Idempotency

I’ve seen webhook systems fail not because teams were careless,

but because webhooks combine three different problems that are easy to conflate:

Most implementations handle maybe one of these well. Rarely all three.

Signature verification tutorials focus on crypto details but skip the part where JSON parsing subtly breaks everything. Replay protection gets bolted on without understanding retry behavior. Idempotency is treated as “just store an ID somewhere” until a retry storm takes production down.

This post isn’t a spec rewrite. It’s the mental model I wish more teams had before shipping webhooks.

I’ll explain:

• How signature verification actually works in production systems

• Why replay attacks are boring but very real

• The simplest idempotency design I’ve seen hold up under retries, crashes, and bugs

No theory for theory’s sake. Just what breaks, why it breaks, and how to avoid learning it at 2am.

Threat model in plain English

Let’s be clear about what you’re defending against.

Attackers are not usually doing anything fancy.

They:

• Capture a legitimate webhook request (from logs, proxies, leaked traffic, or shared environments)

• Replay it later to trigger the same side effects again

• Or flood your endpoint with garbage to see what falls over

Meanwhile, legitimate providers will:

• Retry requests aggressively on network errors

• Retry after timeouts even if your code eventually succeeded

• Deliver events out of order

• Deliver the same event multiple times on purpose

Retries and attacks often look identical at the HTTP level. Same headers. Same body. Same signature. Different intent.

You should assume:

• Your webhook URL will leak eventually

• Every request can be replayed byte-for-byte

• Attackers can send requests as fast as your infrastructure allows

• Providers will not coordinate retries with your internal state

What attackers cannot usually do:

• Forge signatures without your secret

• Modify payloads without breaking signatures (if you verify correctly)

• Guess future valid event IDs

This leads to a key mindset shift:

Signature verification proves authenticity, not safety.

It answers “who sent this,” not “should I process this again.”

Everything else  freshness, dedupe, side effects  is your responsibility.

Signature verification done correctly

This is where most implementations are almost right  and still broken.

Raw body vs parsed JSON

If you remember one thing from this article, make it this:

You must verify the signature against the raw request body, exactly as received.

Not:

• Parsed JSON

• Re-serialized JSON

• Pretty-printed JSON

• A subset of fields

I’ve seen production outages caused by:

• Frameworks automatically parsing JSON before middleware runs

• Whitespace or key ordering differences changing the signed payload

• Unicode normalization differences between languages

If the provider signs bytes, you must verify bytes. Full stop.

If your framework doesn’t give you access to the raw body, fix that before shipping. Don’t “work around” it.

How signing usually works

Most providers do some variation of:

They then send:

• The timestamp

• One or more signatures

• Sometimes multiple secrets (for rotation)

Your job is to:

1. Reconstruct the exact signed string

2. Compute the HMAC with your secret

3. Compare it to the provided signature

Sounds easy. It’s not.

Constant-time comparison matters

Do not use normal string equality for signatures.

Timing attacks on webhook endpoints are rare, but:

• The fix is trivial

• The cost of not fixing it is unbounded

Use a constant-time comparison function. Always. No exceptions.

If your language’s standard library doesn’t have one, you’re probably using the wrong function.

Multi-signature headers and key rotation

Many providers include multiple signatures in a single header. This is not optional complexity  it’s how key rotation works.

Common mistake:

• Only checking the first signature

• Breaking all webhooks during secret rotation

• Or worse, silently accepting invalid signatures

Correct behavior:

• Parse all signatures

• Verify against all active secrets

• Accept if any valid combination matches

Yes, it’s extra work. It’s also how you avoid emergency rotations breaking production.

Real mistakes I’ve seen

Some greatest hits:

• Verifying the signature after parsing JSON (already broken)

• Using the wrong character encoding

• Logging the raw body after modifying it

• Forgetting to include the timestamp in the signed payload

• Trusting the timestamp without validating its freshness

Signature verification is binary. It either matches or it doesn’t. If it’s flaky, your implementation is wrong.

Preventing replay attacks

Here’s the uncomfortable truth:

A valid signature can be replayed forever unless you stop it

Signatures do not expire on their own.

Why signatures alone are not enough

If an attacker captures a single legitimate webhook request, they can:

• Replay it tomorrow

• Replay it next week

• Replay it a thousand times in a row

Every replay will have:

• A valid signature

• A valid payload

• Perfect authenticity

Without replay protection, your system cannot tell the difference between:

• A provider retry

• A malicious replay

• A developer accidentally re-sending a request

Timestamp freshness windows

Most providers include a timestamp for a reason. Use it.

Typical windows I’ve seen work well:

• 5 minutes

  for internet-facing systems

• 10 minutes

  if provider retries are slow or inconsistent

Shorter than that and you’ll reject legitimate retries during outages. Longer than that and replay attacks get easier.

Important nuance:

• Validate freshness before heavy processing

• But after signature verification (otherwise attackers can spam invalid timestamps)

Event IDs / nonces

Timestamps stop old replays. They don’t stop rapid replays.

That’s where event IDs come in.

Most providers include a unique event ID. If they don’t, generate your own nonce by hashing the payload.

Store the ID when you first see it. Reject or short-circuit duplicates.

Redis vs database for replay tracking

Redis:

• Fast

• Simple TTL-based expiry

• Great for high-volume webhooks

Database:

• Durable

• Easier to reason about long-term state

• Can double as idempotency storage

In practice:

• Redis for freshness + short-term replay protection

• Database for long-term idempotency

If you only pick one, pick the database. Redis outages during webhook storms are not fun.

What actually happens during an attack or retry storm

Without replay protection:

• The same event triggers side effects repeatedly

• Downstream systems get hammered

• You can’t distinguish bugs from attacks

With basic replay protection:

• Attacks collapse into harmless duplicates

• Retries become cheap

• Your on-call stops sweating

The simplest idempotency design that works

This is where theory meets reality.

You do not need a perfect system. You need one that fails safely.

Dedupe

Your idempotency key should be:

• Provider name

• Event ID

Nothing else.

Do not hash payloads. Do not include timestamps. Do not overthink this.

If the provider doesn’t give you an ID, create one deterministically from the raw body.

Processing state model

Store a row like:

• event_id

• provider

• timestamps

• optional error info

The flow:

1. Insert row as

2. Do the work

If the insert fails due to a unique constraint, you’ve seen this event before.

How to return 2xx safely

This is where many people panic.

If you see a duplicate event:

• Return 2xx

• Do not re-run side effects

• Do not throw errors

Providers interpret non-2xx as “retry forever.”

Your job is to be boring and predictable.

Making side effects idempotent

Your webhook handler should not:

• Charge a credit card directly

• Send irreversible emails inline

• Trigger one-way external calls without safeguards

Instead:

• Enqueue jobs with idempotency keys

• Use database constraints

• Make side effects conditional on state transitions

Yes, this takes discipline. It’s also why mature systems survive retries.

How this design fails and why it’s still good

It can fail if:

• You crash after marking processing

• You forget to handle stuck rows

• Your database is unavailable

That’s okay.

You can:

• Retry failed events manually

• Add cleanup jobs for stuck processing

• Rebuild state from logs if needed

The key point: failures are visible and bounded, not silent and catastrophic.

The “golden path” webhook handler flow

This is the flow I aim for in production:

1. Enforce method, content-type, and size limits

2. Read and store the raw request body

3. Verify signature using raw body

4. Validate timestamp freshness

5. Parse JSON

6. Extract provider + event_id

7. Attempt idempotency insert

   • If duplicate → return 2xx

8. Acknowledge quickly (2xx)

9. Process asynchronously

10. Update processing state

Short-circuit early. Fail fast. Be boring.

Synchronous processing is fine for trivial side effects. For anything non-trivial, async buys you retries, isolation, and sleep.

Production hardening checklist

Things that actually matter:

• Rate limiting

  Per IP and globally. Not for security theater  for survival.

• Payload size limits

  If you don’t expect 5MB payloads, reject them.

• Safe logging

  Log event IDs, provider, and status.
  Do not log raw bodies by default. Ever.

• Metrics

  • Signature verification failures

  • Duplicate event rate

  • Processing latency

  • Retry counts

• Alerts

  • Sudden spike in signature failures

  • Sustained retry storms

  • Growing number of stuck  events

If you can’t answer “are we dropping or duplicating events right now?”, you’re flying blind.

---

You Might Be Interested In

• Passkeys Rollout Plan For Saas Products: Migration Without Support Tickets
• Deepfakes Types And Warning Signs
• Which Cybersecurity Certification Is Best?
• Secrets In Identity Systems: How Api Tokens Leak And What To Do About It
• Csrf Vs Xss: Practical Differences And Real Fixes

---

Conclusion

Webhook security isn’t about crypto tricks. It’s about controlling repetition.

Authenticity tells you who sent the request.
Replay protection tells you when.
Idempotency tells you whether to act.

Get those three right and webhooks become boring. That’s the goal.

Pragmatic checklist:

• Verify signatures on raw bytes

• Enforce timestamp freshness

• Deduplicate by provider + event ID

• Return 2xx on duplicates

• Make side effects idempotent

• Monitor retries, not just errors

If your webhook handler is boring at 2am, you did it right.

FAQs

What’s the difference between webhook retries and replay attacks?

Webhook retries are a normal, expected behavior from legitimate providers. They happen when the provider doesn’t receive a clear success signal from your endpoint  maybe your server timed out, crashed after doing the work, or returned a transient 5xx. From the provider’s perspective, retrying is the safest option. These retries often arrive minutes later, sometimes hours later, and can arrive multiple times even if the original request actually succeeded on your side.

Replay attacks, on the other hand, are intentional. An attacker captures a real, signed webhook request and resends it to trigger the same behavior again. Technically, the requests can look identical. Same payload, same headers, same signature. That’s why systems that only rely on signature verification get burned. In practice, your system should treat retries and replay attacks the same way: safely dedupe them and make repeated deliveries harmless.

Should I verify the signature before parsing JSON?

Yes. Always. This is non-negotiable if you want signature verification to actually mean anything.

Most webhook providers sign the raw byte stream of the HTTP request body, not the parsed JSON structure. The moment you parse JSON, you risk changing key ordering, whitespace, number formatting, or Unicode normalization. Even if the parsed object looks identical, the underlying bytes may no longer match what was signed. I’ve seen teams spend days debugging “intermittent” signature failures that were caused entirely by JSON parsing happening too early in the request lifecycle. Verify first, parse second, every time.

What timestamp window should I use?

In real systems, a ±5 minute window is a good default. It’s short enough to make large-scale replay attacks impractical, but long enough to tolerate network delays, queueing, and provider retries during brief outages. If you’re working with a provider known for slow retries or if your infrastructure occasionally pauses under load, extending this to ±10 minutes is reasonable.

The mistake I see most often is choosing an aggressively small window because it “feels more secure.” In practice, that just causes legitimate events to get rejected during incidents  exactly when retries matter most. Security that breaks reliability will get disabled. A slightly larger window that actually stays enabled is almost always the better tradeoff.

What’s the simplest idempotency key?

The simplest idempotency key that consistently works is the combination of the provider name and the provider’s event ID. That’s it. This maps directly to how webhook systems behave in the real world: providers retry the same logical event, not slightly modified versions of it.

If a provider doesn’t supply an event ID, you can derive one deterministically from the raw request body, for example by hashing it. What you want is a stable identifier that will be identical across retries but different for genuinely distinct events. Anything more complicated than that usually introduces edge cases without adding meaningful protection.

Redis vs database for dedupe which should I pick?

If you have to choose one, pick your database. Databases give you durability, observability, and a clear source of truth when something goes wrong. When an incident happens, being able to inspect which events were processed, which are stuck, and which were deduped is invaluable. That visibility is often worth more than raw speed.

Redis can be a great complement when you’re dealing with very high throughput or want cheap, short-lived replay protection using TTLs. But Redis outages during webhook storms are painful, and losing your dedupe state at the wrong moment can cause a cascade of repeated side effects. In my experience, starting with the database and adding Redis later  only if you actually need it  leads to fewer 2am surprises.