LinuxCommandLibrary

pg_recvlogical

Stream PostgreSQL logical replication data

TLDR

Create a new logical replication slot

$ pg_recvlogical [[-d|--dbname]] [dbname] [[-S|--slot]] [slot_name] --create-slot
copy

Start streaming changes from a logical replication slot to a file
$ pg_recvlogical [[-d|--dbname]] [dbname] [[-S|--slot]] [slot_name] --start [[-f|--file]] [filename]
copy

Drop a logical replication slot
$ pg_recvlogical [[-d|--dbname]] [dbname] [[-S|--slot]] [slot_name] --drop-slot
copy

Create a slot with two-phase commit enabled
$ pg_recvlogical [[-d|--dbname]] [dbname] [[-S|--slot]] [slot_name] --create-slot [[-t|--enable-two-phase]]
copy

Display help
$ pg_recvlogical [[-?|--help]]
copy

SYNOPSIS

pg_recvlogical [OPTION]...

PARAMETERS

-d, --dbname=DBNAME
    Connect to the database specified by DBNAME.

-f, --file=FILE
    Write the streamed output to FILE instead of standard output.

-h, --host=HOSTNAME
    Database server host name or directory for Unix-domain socket.

-o, --option=LSOPTION
    Pass an option LSOPTION to the logical decoding plugin. Can be used multiple times.

-p, --port=PORT
    Database server port number.

-P, --plugin=PLUGIN
    Use the specified logical decoding PLUGIN (e.g., test_decoding).

-s, --startpos=LSN
    Start streaming at the specified Log Sequence Number (LSN).

-S, --slot=SLOTNAME
    Use the replication slot named SLOTNAME. Creates it if it doesn't exist (unless --if-not-exists is used).

-U, --username=USERNAME
    Connect as the specified database user USERNAME.

-v, --verbose
    Enable verbose output, showing more detail on operations.

-W, --password
    Force pg_recvlogical to prompt for a password before connecting.

-X, --setting=SETTING
    Set a run-time parameter SETTING for this session.

-Z, --compress=COMPRESSION_METHOD
    Compress the replication stream using the specified method (e.g., gzip, zstd).

-w, --no-wait
    Do not wait for connection to be established; fail immediately if it can't connect.

--endpos=LSN
    Stop streaming once the specified LSN (Log Sequence Number) is reached.

--if-not-exists
    Do not error if the replication slot already exists when attempting to create it.

--no-loop
    Exit after receiving one complete file of output; do not stream continuously.

--no-sync
    Do not sync output files to disk (faster but less crash-safe).

--status-interval=INTERVAL
    Send status updates to the server every INTERVAL seconds (default 10s).

--synchronous
    Request synchronous replication from the server.

--version
    Show pg_recvlogical version information, then exit.

--help
    Show help message, then exit.

DESCRIPTION

pg_recvlogical is a powerful PostgreSQL utility designed to stream real-time database changes using the logical replication protocol.
It works by connecting to a PostgreSQL server and consuming the Write-Ahead Log (WAL) through a specified replication slot.
The WAL content is then transformed into a readable format using a server-side logical decoding plugin (e.g., test_decoding, wal2json).
This allows external applications to receive continuous, structured data about transactions, enabling various use cases like Change Data Capture (CDC), data integration, auditing, and building custom data pipelines.
It can create, manage, and utilize replication slots, streaming the decoded output to standard output or a file.

CAVEATS

Using pg_recvlogical requires the PostgreSQL server's wal_level to be set to logical.
A replication slot is critical; if not properly managed (e.g., consumer fails), it can lead to excessive WAL retention and disk space issues on the server.
The output format is entirely determined by the logical decoding plugin used, which must be installed and loaded on the PostgreSQL server.
Network connectivity and appropriate database permissions are necessary for the client to connect and stream data.

PREREQUISITES

To effectively use pg_recvlogical, the PostgreSQL server must have its wal_level parameter set to logical in the postgresql.conf file. Additionally, a specific logical decoding plugin (e.g., test_decoding, wal2json) must be installed on the server and loaded into the database that the replication slot is associated with.

REPLICATION SLOTS

Replication slots are server-side objects that ensure the PostgreSQL server retains all necessary WAL segments until they have been successfully consumed by all connected consumers.
This mechanism is vital for preventing data loss and ensuring continuous logical replication. However, if a consumer stops reading from a slot, the server will continue to accumulate WAL, potentially leading to significant disk space consumption. It is crucial to monitor slot activity and drop unused slots promptly.

HISTORY

pg_recvlogical was introduced with PostgreSQL 9.4, accompanying the native support for logical decoding.
This release marked a significant advancement in PostgreSQL's capabilities for Change Data Capture (CDC) and flexible data integration, providing a standardized and robust mechanism to extract and transform WAL data, forming the basis for many modern data pipeline solutions.

SEE ALSO

Copied to clipboard