Payment and customer events into reliable multi-system automations. Signature verification, idempotent ingestion, queue-backed execution, per-step retries, encrypted credentials, and an operator console.
When a payment, refund, failed charge or high-value lead occurs, multiple systems need to react in lock-step — CRM lifecycle flips, finance gets a Slack ping, ops gets an audit log, the customer gets an email. The cost of these systems falling out of sync is real (refunded customers still being billed, leads that never get routed, finance dashboards that don't match Stripe). Off-the-shelf tools solve a slice but rarely give you the visibility and reliability that production revenue ops actually requires. This is the reference implementation.
Raw request body is read once, HMAC verified, then persisted with (tenant_id, provider, idempotency_key) as a UNIQUE constraint. Duplicate webhooks return 200 duplicate=true instead of double-processing.
Workflow definitions are YAML; each version is content-hashed (SHA-256) so reruns are reproducible against the exact version that originally executed. Workflow changes never silently alter historical runs.
Each step is a job. Retries happen inside the engine (step-level boundary), not inside RQ (job-level). Exponential backoff base 2s → max 30s, 4 attempts by default — configurable per step.
Failed runs land in a Postgres dead_letter table (not Redis) — operators need days of visibility, not seconds. HTMX operator console for replay, claim, and inspect with no JS framework dependency.
Connector credentials are Fernet-encrypted at rest; keys are environment-injected, never persisted. Per-connector scope means a leaked Slack token can't read CRM data.
Raw body is read once, HMAC verified, event persisted. The IntegrityError on the UNIQUE (tenant_id, provider, idempotency_key) constraint becomes "200 duplicate=true" — duplicates can never cause downstream side effects.
Step-level retry boundary means a step that fails after 3 attempts marks the run as needs-attention without burning RQ job-retry budget. Operators see exactly which step failed and why.
Need to be careful about retrying non-idempotent steps (sending emails). Step config has explicit idempotent=true|false flag.
Operators need days of visibility, audit-grade history, and SQL search over failures. Redis lists evict and have no query story.
A bit more write load on Postgres. Negligible in practice — failures are rare; the DLQ is small relative to the event log.
Shared ORM between FastAPI threadpool and RQ workers without the async/sync boundary. Easier reasoning, no sync_to_async dance, no surprise event-loop blocking.