Background work

Applies to The Fabrica v0.1.2. Documentation source: 1ee1bc9cb661, including post-release documentation corrections.

Worker hardening: the config bundle

backend/src/workers/celery_app.py ships a non-default Celery config. Each setting addresses a specific failure mode. The pattern is “secure defaults” — flipping any of these to Celery’s default is a downgrade.

Why each one

task_serializer="json" + accept_content=["json"]

Celery used pickle as the default serializer until 4.x. A malicious broker — or any attacker who can write to Redis — can ship a payload that the worker unpickles into arbitrary code execution. JSON serialization makes the broker payload data-only.

Cost: you can’t ship custom Python objects as task args. Hard rule worth keeping; if a buyer needs to pass a complex object they serialize it themselves (e.g. UUID → str at the call site).

task_track_started=True

Default is False — Celery doesn’t write a “STARTED” state to the result backend. With this on, monitoring tools (Flower, Sentry’s Celery integration) can show “task is alive, working” vs “task is queued” — the difference between a worker being stuck and a worker being idle.

worker_prefetch_multiplier=1

Default is 4. Celery prefetches N×concurrency tasks from the broker per worker so the worker has work queued up locally. Default is fine for short tasks; ruins long tasks because:

1. Prefetched tasks aren’t visible to other workers.
2. If worker A grabs 4 tasks and one takes 30 minutes, the other 3 sit waiting — even if worker B is idle.

The factory’s tasks (reconciler, expiry, deletion) are short, but buyer-added tasks might not be. =1 is the safe default; buyers who add lots of small tasks can raise it.

worker_max_tasks_per_child=10

Recycle the worker process every 10 tasks. Defeats memory leaks in async loops + third-party libraries (looking at you, every SDK that caches DNS lookups forever). 10 is the safe default; buyers can raise it if task startup cost is high.

worker_cancel_long_running_tasks_on_connection_loss=True

If the broker connection dies mid-task, the broker won’t see the ACK when the task completes — so on broker reconnect, the task gets redelivered. With this setting, the worker gives up on the orphaned task instead of completing it (and having it re-delivered → double work). Pair with task_acks_late=True for at-least-once semantics that won’t double-bill the user.

worker_hijack_root_logger=False

Celery’s default replaces the root logger config — which means your careful FastAPI logging setup gets clobbered the moment a worker boots. Off means our src/config/logging.py keeps owning the logger hierarchy.

task_acks_late=True + task_reject_on_worker_lost=True

Default is “ack the task on receipt” — if the worker crashes mid-task, the broker considers it done. With acks_late, the worker only ACKs after task_completed. If the worker dies, the broker redelivers. Pair with the orphan-deduct reconciler (why-credit-ledger.md) and you have at-least-once semantics that don’t burn user credits when the worker dies.

Operational notes

• Flower / Celery beat: same config applies — both processes inherit app.conf. No special setup.
• CI tests: tests don’t talk to the broker directly; tasks are invoked via _task_func.delay(...) or directly via the async helpers exposed by each task module (so we never need a live Celery worker in CI).
• Replacing the broker: if a buyer swaps Redis for RabbitMQ, the task_serializer / accept_content / acks_late settings are broker-agnostic and survive. The TLS hardening in get_broker_url is Redis-specific; for RabbitMQ replace with the equivalent AMQP TLS config.

Receipts

• backend/src/workers/celery_app.py

Bounded reconciler sweeps

reconcile_orphan_credits (Celery beat task, every 60 seconds) caps its per-pass work at DEFAULT_RECONCILE_LIMIT = 200 orphan candidates. Two reasons.

Why bound at all

A pristine system has zero orphans. The reconciler scans the DB, finds nothing, returns. Fast.

A failing system — broker outage, Celery worker crash loop, ill-timed deploy — can leave hundreds of orphan deducts in one beat interval. Without a limit:

1. The reconciler’s list_orphan_deducts_older_than(...) query returns thousands of rows.
2. The Python loop iterates each, issuing a refund per orphan.
3. Each refund is a SQL INSERT — fast individually, but the loop holds the session open the whole time.
4. The beat scheduler doesn’t fire the next reconciler tick because the previous one is still running.
5. Real-time correctness: a user whose orphan deduct landed AFTER the sweep began waits until the next sweep — which is delayed.

With limit=200:

• Worst-case 200 refunds × ~5ms each = ~1 second per pass.
• Next pass fires on schedule, picks up the next 200.
• A backlog of 10k orphans clears in ~50 passes / ~50 minutes.

Trade-off: backlogged orphans take longer to refund. Mitigation: the lazy-reconcile path on GET /billing/credits (planned — flag in audit) reconciles a single user on-demand, so user-visible lag is bounded by HTTP request time, not the beat cadence.

Why structured result

Old API:

@shared_task
def reconcile_orphan_credits() -> int:
    return await _reconcile_async()  # returned the refund count

New API:

@dataclass(frozen=True, slots=True)
class ReconcileResult:
    scanned: int = 0
    refunded: int = 0
    skipped_lifecycle: int = 0
 
async def _reconcile_async(...) -> ReconcileResult: ...

The Celery task entrypoint still returns the int (for compatibility with monitoring tools that surface task return values), but the async helper returns the rich shape. Why it matters:

• Tests: assert result.scanned == 5 and result.refunded == 3 is more informative than assert result == 3. When the test fails, you see which orphans got skipped.
• Observability: the beat task logs extra={"scanned": ..., "refunded": ..., "skipped_lifecycle": ...} so you can grep “scans found 200 candidates but refunded 0” — a signal that something’s writing task_started after the orphan threshold (i.e. workers are running very slowly).
• Buyer extension: a buyer who wires the reconciler into a Sentry / DataDog metric wants the scanned / refunded split, not just the total.

Reading the skip count

skipped_lifecycle means: “we found N deducts that LOOKED orphan in the initial scan, but by the time we checked again a task_started or task_completed row had landed”. This is the race condition the reconciler is built around — workers writing task_started slightly after the deduct’s age crosses the orphan threshold. Healthy values:

• skipped_lifecycle = 0 — normal. Worker latency is well under the 5-min threshold.
• skipped_lifecycle > 0, refunded > 0 — borderline. Workers occasionally slow. Investigate p99 task pickup latency.
• skipped_lifecycle >> refunded — workers consistently slow. Either raise the orphan threshold (ORPHAN_THRESHOLD_S in credit_ledger.py) or scale the worker pool. Otherwise the reconciler is mostly doing pointless work.

Receipts

• backend/src/workers/credit_reconciler.py
• Repository method: backend/src/services/repositories_async/credit_transaction.py:list_orphan_deducts_older_than

Provenance. Documentation source: 1ee1bc9cb6619c19f57766731e7884b37f515dc9 (main).

Extracted from docs/recipes/celery-hardening.md, docs/recipes/why-reconciler-limit.md.

That commit is later than the release these pages describe: it carries documentation corrections made after thefabrica-v0.1.2 went out. Nothing that changes how the product behaves landed between the two, which is what makes naming that version honest.