Metis

PeerDB wants to help you quickly export data from PostgreSQL

Itay Braun's photo
Itay Braun
·

1 min read

There are many great ETL tools, such as airbyte. PeerDB is a new ETL tool, dedicated to PostgreSQL only.

The ETL engine uses built-in Change Data Capture (CDC) to detect changes and forward them to your destination. Here you can see the supported connectors (the source is always PG).

The implementation looks pretty straightforward and simple.

CREATE PEER postgres_peer
FROM postgres (...);

postgres connection validated and created peer

CREATE PEER snowflake_peer
FROM snowflake (...);

snowflake connection validated and created peer

CREATE MIRROR real_time_cdc
FROM postgres_peer
TO snowflake_peer
WITH TABLE MAPPING (transactions:transactions, users:users);

transactions and users table replicated in realtime from postgres to snowflake.

Total time: 2 sec

My Take

  • Implementing an ETL which uses a CDC might be a complicated task. So I like their approach of simplifying things.

  • Postgres become more and more popular, so it makes perfect sense to me to create a product dedicated to this DB engine.

  • I'm still curious why their engine is much faster than AirByte. I give them the credit that the performance tests are accurate. However, I usually take all benchmarks with a grain of salt.

  • Their license is Elastic v2 (ELv2). Probably worth another post about recent changes in the business model of Open Source projects.

pg toolscdc