Webhook delivery is idempotent by construction

Applies to The Fabrica v0.1.2. Documentation source: 1ee1bc9cb661, including post-release documentation corrections.

Every webhook the factory receives is idempotent: replay the same event 10 times → exactly one side effect. The two-table pattern (processed_webhook_events + webhook_events) is the load-bearing invariant; both Paddle and Clerk webhooks ride it.

The competitor pattern


# competitor/billing.py
@app.post("/webhooks/paddle")
async def paddle_webhook(payload: dict):
    if payload["event_type"] == "subscription.created":
        await grant_credits(payload)
    return {"ok": True}

No dedup. No archive. Trust the payload.

Where it breaks

Paddle’s webhook retry policy: up to 25 retries over 3 days on any non-2xx response (and “non-2xx” includes “TCP RST mid-response”). Clerk uses Svix → same 5xx retry semantics.

Real failure modes the competitor pattern hits:

TCP reset post-2xx. Server processes the event, commits the DB transaction, writes the 200 OK response → and the TCP ACK packet gets dropped (NAT timeout, load balancer keepalive expiry, mobile network blip). Paddle marks the delivery as failed + retries 15 min later. The handler runs again. Credits granted twice.
Worker timeout mid-process. The handler is 30s into a heavy subscription.created flow when the LB times out the request. The DB transaction is mid-commit; the response is 504. Paddle retries. Handler runs again. State half-applied + replayed.
Deploy-mid-webhook. You deploy. Mid-deploy a worker that’s been processing a webhook for 10s gets SIGTERM. The handler is killed before commit. Paddle retries. New deploy’s handler picks it up fresh. Looks fine — but if you don’t have idempotency, a stuck-retry that suddenly succeeds 12 hours later DOES grant credits twice if the first run actually completed its DB commit but failed to ACK.

The cost is real — the failure modes this guards against: duplicate credit grants, duplicate welcome emails, and duplicate subscription.cancelled handlers (downgrading users twice → a support nightmare).

What we ship

Two tables. One job each.

processed_webhook_events (the dedup):


CREATE TABLE processed_webhook_events (
    id UUID PRIMARY KEY,
    provider VARCHAR(32) NOT NULL,
    event_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(128) NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    CONSTRAINT uq_processed_webhook_events_provider_event_id
        UNIQUE (provider, event_id)
);

The UNIQUE constraint is the dedup. On every webhook delivery, we INSERT first. Postgres enforces the uniqueness atomically. If the INSERT trips IntegrityError, the event was already processed — we rollback the dedup attempt, mark the archive row SKIPPED, return 200 with idempotent: true. Zero downstream side effects.

webhook_events (the archive):


CREATE TABLE webhook_events (
    id UUID PRIMARY KEY,
    provider VARCHAR(32) NOT NULL,
    event_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(128) NOT NULL,
    payload JSONB NOT NULL,
    signature TEXT,
    processing_status VARCHAR(32) DEFAULT 'pending',
    error_message TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Every payload + signature stored unconditionally. The processing_status column tracks PENDING / PROCESSED / SKIPPED / FAILED so a buyer doing forensics can SELECT * WHERE processing_status='FAILED' AND created_at > NOW() - INTERVAL '1 day' and see what’s broken.

The handler shape:


@router.post("/webhook")
async def paddle_webhook(request, db, paddle_signature):
    raw_body = await request.body()
    payload = verify_paddle_signature(raw_body, paddle_signature, secret)
 
    archive = WebhookEvent(provider="paddle", event_id=..., payload=payload, ...)
    db.add(archive)
    await db.flush()
 
    idem = ProcessedWebhookEvent(provider="paddle", event_id=event_id, ...)
    db.add(idem)
    try:
        await db.flush() # ⬅ UNIQUE constraint fires here on replay
    except IntegrityError:
        await db.rollback()
        archive.processing_status = "SKIPPED"
        return {"event_id": event_id, "idempotent": True}
 
    # First-delivery path: process the event, mark archive PROCESSED
    await _handle_event(db, payload)
    archive.processing_status = "PROCESSED"
    return {"event_id": event_id}

Why two tables, not one

The dedup table is small + hot (one row per event, indexed). The archive is large + cold (full payload + signature + error messages, queried for forensics maybe once a week).

Splitting:

Dedup table fits in shared_buffers → INSERT-on-conflict is ~µs.
Archive can grow to GBs without hurting dedup latency.
Buyers retention-purge the archive after 90 days; the dedup keeps forever (each row is ~80 bytes).

One table would force a trade-off: either you keep the archive forever (storage cost), or you purge it + lose dedup for replays that arrive after the purge window.

Replay protection vs idempotency

These are two layers:

Replay protection (in services/paddle.py): rejects requests with timestamps > 5 minutes old via constant-time HMAC compare. Stops attackers who captured a webhook payload + replay it from a different IP days later.
Idempotency (this recipe): handles legitimate replays (Paddle retries within the 3-day window) that pass signature + timestamp but were already processed.

Both layers are needed. Signature-only catches forgeries; idempotency catches the dups.

What about Stripe / Clerk?

Same pattern works for any provider that:

Emits a stable event_id per delivery (Stripe: evt_*, Clerk: svix-id header, Paddle: event_id in the body).
Retries on non-2xx.

The factory ships handlers for Paddle + Clerk. Buyers adding a third (Twilio, SendGrid, GitHub webhook) drop into the same two-table pattern by reusing WebhookEvent + ProcessedWebhookEvent with a new provider enum value.

Receipts

Idempotency model: backend/src/models/processed_webhook_event.py
Archive model: backend/src/models/webhook_event.py
Paddle handler: backend/src/api/routers/billing.py:paddle_webhook
Clerk handler: backend/src/api/routers/account.py:clerk_webhook
Replay protection: backend/src/services/paddle.py:verify_paddle_signature

Provenance. Documentation source: 1ee1bc9cb6619c19f57766731e7884b37f515dc9 (main).

Extracted from docs/recipes/why-webhook-idempotency.md.

That commit is later than the release these pages describe: it carries documentation corrections made after thefabrica-v0.1.2 went out. Nothing that changes how the product behaves landed between the two, which is what makes naming that version honest.