Plugin-driven data movement across AWS, Azure, GCP, Snowflake, Postgres and Oracle. Each connector implements the same five-method interface; jobs configured via UI wizard or YAML.
Most data teams accumulate a long tail of one-off scripts that copy data between systems — Postgres-to-Snowflake replication, S3-to-Azure DR mirroring, BigQuery-to-Oracle bridges for legacy reports, ad-hoc migrations. This consolidates that work into a single application a non-engineer can configure and operate, with a real connector interface underneath instead of script soup.
AWS S3, Azure Blob, GCP BigQuery, Snowflake, PostgreSQL and Oracle each implement the same five methods — connect, close, read, write, validate. New connectors are 200 lines, not a rewrite.
Operators define source, target, schedule and CDC watermark through a Flask wizard; engineers edit the YAML directly. Both produce the same canonical job record.
Jobs run on APScheduler in the same Python process — no Airflow dependency, no separate scheduler service. Default batch size 10,000 rows; pluggable for memory-constrained sources.
Each run streams progress, error events, and reconciliation deltas to a browser via SSE — no polling, no WebSocket complexity. Operators see the same view in real time that the job is writing to its log.
For 50 jobs and one operator team, Airflow is over-architecture. APScheduler runs in the same process as Flask, deploys as one container, and is sufficient up to roughly 200 jobs.
No backfill UI; backfills are a manual job-trigger. Acceptable at this scale; escape hatch is documented (swap APScheduler for an Airflow operator that calls the same Connector interface).
Timestamp watermarks work for most operational tables; full-table hash works for small dimension tables without a reliable timestamp.
Hash-CDC reads the full source each run. Flagged in the README; documented limit at ~10M rows.
Most teams deploy this behind their SSO proxy (oauth2-proxy, Cloudflare Access). Re-implementing auth inside the app is a maintenance tax with no marginal security benefit.
Catches the bugs that matter (counts mismatch, columns lost) without doubling the data volume. Column-distribution checks are on the roadmap behind an opt-in flag.