Lambda architecture over Kafka — speed layer for sub-second sessionisation and bot scoring, batch layer for nightly CLV and funnel rollups, both over the same source-of-truth event log.
E-commerce platforms need both real-time decisions (live session counters, inventory state, bot rejection) and batch analytics (CLV modelling, cohort retention, conversion funnels) over the same event stream. Most teams end up with two separate pipelines that drift apart. This is the Lambda-architecture pattern done cleanly — single event log, split serving layer, deterministic merge.
Clickstream and orders land in two topics, partitioned by session_id and order_id respectively. Producers write once; both pipelines downstream consume the same bytes.
A Python Kafka consumer maintains in-memory session state (30-min inactivity timeout) and runs a lightweight bot heuristic on each event. Live counters go to Redis HSETs with a 24-hour TTL for the dashboard.
Reads the same source events from S3 (Kafka Connect sink), recomputes CLV via BG/NBD (lifetimes package with a heuristic fallback for CI), generates funnel rollups, and writes to Postgres marts.
Reads combine the speed layer (last 24h from Redis) with the batch layer (everything before that from Postgres). Determinism comes from event_id idempotency — re-processing produces the same result.
Streaming for the things that need to be live (bot rejection, live counters). Batch for the things that need to be right (CLV, retention). Two simple pipelines, one event log; cheaper to operate than a single streaming pipeline that has to be both.
Two code paths to maintain. Acceptable because the boundary is clean — the speed path knows nothing about CLV, the batch path knows nothing about live state.
Manual offset commit on the consumer keeps semantics at-least-once. Redis HSET writes are idempotent (same key = same value), so duplicate events are absorbed without special handling.
A duplicate event will briefly inflate session event-count metrics. Stable within seconds; not used for billing-grade aggregates.
Single-process simplicity for the demo. Multi-instance production swaps the state dict for a Redis hash — same interface, no code-shape change.
Avro/Protobuf migration is isolated to a single deserialisation function. The pattern is set up to upgrade, without forcing the team to do it day one.