ClickHouse's WalShadow replicates Postgres to ClickHouse in 200 ms from physical WAL
WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL
ClickHouse has open-sourced WalShadow, an engine that replicates Postgres data to ClickHouse directly from physical WAL. In benchmarks, committed transactions became visible in ClickHouse in about 200 ms, and WalShadow sustained 289K rows/sec, matching the source Postgres instance. Unlike traditional CDC systems, it avoids logical replication slots, decodes WAL outside the source, and writes ClickHouse-native blocks. It supports initial load, continuous replication, schema evolution, restart recovery, and switchovers. A managed version is in private preview for ClickHouse Managed Postgres.
The source only needs to ship physical WAL, resulting in a load profile similar to a physical standby while allowing changes to reach ClickHouse within a second.
- vmsp
They're not using `wal_level = logical`, which has been the "friendly" way of doing CDC on Postgres since ever, but are going straight to `wal_level = replica` which, afaik, has never really been used to build something atop of except Postgres' own replication.
This is very interesting. I'd never have guessed that it'd make such a difference. I also bet this is the sort of thing that would have never end up being implemented without access to coding agents. Having to figure out these protocol-level details is no longer the huge time sink it was
- rgbrgb
would this work with a postgres hosted on supabase? for our company brain mcp thing we mirror postgres data (along with server logs, banking data, slack, etc) in clickhouse but took a very inefficient polling approach since it works with any read-only db connection string. always looking to improve that and get rid of latency though.