debezium

distributed platform for change data capture

TLDR

Start Debezium server

$ debezium-server

Run Debezium Server with Docker

$ docker run -it --name debezium -p 8080:8080 -v $PWD/conf:/debezium/conf debezium/server

Run MySQL connector in Kafka Connect

$ curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d @[mysql-connector.json]

Check connector status

$ curl http://localhost:8083/connectors/[connector-name]/status

List running connectors

$ curl http://localhost:8083/connectors

SYNOPSIS

debezium-server (standalone)Kafka Connect REST API at http://localhost:8083/connectors

Debezium is an open-source distributed platform for change data capture (CDC). It monitors databases and produces events for every row-level change, enabling real-time data streaming and synchronization.Debezium connectors read database transaction logs (MySQL binlog, PostgreSQL WAL, etc.) and convert changes to events. These events can be sent to Apache Kafka, Amazon Kinesis, Google Pub/Sub, or other sinks for processing by downstream applications.The platform runs either as Kafka Connect connectors (distributed, scalable) or as Debezium Server (standalone, simpler deployment). It captures inserts, updates, and deletes with before/after values and metadata.

CONFIGURATION

application.properties (Debezium Server):

$ debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.database.hostname=localhost
debezium.source.database.port=3306
debezium.source.database.user=debezium
debezium.source.database.password=dbz
debezium.source.database.server.id=1
debezium.source.database.server.name=mydb
debezium.sink.type=kafka
debezium.sink.kafka.producer.bootstrap.servers=localhost:9092

CONNECTORS

MySQL: Reads MySQL/MariaDB binlogPostgreSQL: Uses logical replicationMongoDB: Reads oplogSQL Server: Uses CDC tablesOracle: Uses LogMiner or XStreamCassandra: Reads commit log

CAVEATS

Requires database configuration changes to enable log reading. Initial snapshot of large databases can take significant time. Kafka Connect mode requires Kafka infrastructure. Schema changes require careful handling. Database privileges needed for log access vary by platform.

HISTORY

Debezium was created at Red Hat by Randall Hauch and first released in 2016. It was developed to provide a reliable, open-source CDC solution for microservices architectures. The Apache-licensed project attracted a large and diverse community of contributors. In December 2024, Debezium joined the Commonhaus Foundation to establish vendor-neutral governance while maintaining its established community and processes.