Hacker news

  • Top
  • New
  • Past
  • Ask
  • Show
  • Jobs

Show HN: pgwire-replication - pure rust client for Postgres CDC (https://github.com)

45 points by sacs0ni 7 days ago | 8 comments | View on ycombinator

sacs0ni 7 days ago |

I built this to have a dedicated wire-protocol client for postgres logical replication. General-purpose SQL clients either don't implement the replication protocol at all, or bury it behind abstractions designed for other use cases. Replication has a bit different mechanics - it's a stateful binary stream requiring LSN tracking, standby heartbeats, and feedback to prevent WAL bloat. Bolting that onto a query-focused client has its own challenges.

This is just the transport - raw XLogData frames and LSNs. Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure.

What it does:

- Explicit LSN control - start/stop at specific WAL positions for deterministic recovery

- Automatic standby feedback - no more forgotten heartbeats filling your disk with WAL

- Bounded channels - backpressure propagates to Postgres naturally

- Pure Rust, no libpq

What it doesn't do: pgoutput decoding (intentionally). That belongs in a higher layer. Simplest way of using this:

while let Some(event) = client.recv().await? { match event { ReplicationEvent::XLogData { wal_end, data, .. } => { process(&data); client.update_applied_lsn(wal_end); } _ => {} } }

gunnarmorling 1 day ago |

Nice one, great to see this addition to the Rust ecosystem!

Reading through the README, this piqued my curiosity:

> Small or fast transactions may share the same WAL position.

I don't think that's true; each data change and each commit (whether explicit or not) has its own dedicated LSN.

> LSNs should be treated as monotonic but not dense.

That's not correct; commit LSNs are monotonically increasing, and within a transaction, event LSNs are monotonically increasing. I.e. the tuple commit-LSN/event-LSN is monotonically increasing, but not LSNs per se. You can run multiple concurrent transactions to observe this.