kcat

Consume messages starting with the newest offset

$ kcat -C -t [topic] -b [brokers]

$ kcat -C -t [topic] -b [brokers] -o beginning -e

$ kcat -G [group_id] [topic] -b [brokers]

$ echo [message] | kcat -P -t [topic] -b [brokers]

$ kcat -P -t [topic] -b [brokers] [path/to/file]

$ kcat -L -b [brokers]

$ kcat -L -t [topic] -b [brokers]

$ kcat -Q -t [topic]:[partition]:[unix_timestamp] -b [brokers]

kcat [-b broker-list] [-t topic] [options]
kcat -P [-b broker-list] -t topic [producer-options]
kcat -C [-b broker-list] -t topic [consumer-options]
kcat -L [-b broker-list] [metadata-options]

-b broker-list
    Comma-separated list of Kafka brokers (e.g., "localhost:9092").

-t topic
    The Kafka topic to produce to or consume from.

-P
    Operate in producer mode, reading messages from stdin and sending to Kafka.

-C
    Operate in consumer mode, reading messages from Kafka and printing to stdout.

-L
    Operate in metadata list mode, displaying cluster, topic, and partition information.

-G group-id
    Consumer group ID for consumer mode.

-o offset
    Start consuming from a specific offset (e.g., beginning, end, stored, ).

-c count
    Exit after consuming 'count' messages.

-Z
    Enable Kafka protocol's built-in gzip/snappy/lz4/zstd compression for producers.

-K key-delimiter
    Key delimiter for producer (input) or consumer (output). E.g., ':' for 'key:value'.

-X property=value
    Pass librdkafka configuration properties. Can be used multiple times.

-F format
    Format string for consumed messages (e.g., %T for topic, %K for key, %V for value).

-e
    Exit consumer on EOF (end of file) or last message (if -c is used).

-a
    All topics and partitions. Automatically subscribe to all topics in consumer mode or list all topics in metadata mode.

kcat, formerly known as kafkacat, is a powerful and lightweight command-line utility designed for interacting with Apache Kafka brokers. Built upon the librdkafka C library, it provides a simple yet versatile interface for producing messages to Kafka topics, consuming messages from them, and inspecting cluster metadata such as broker information, topic configurations, and partition assignments. It serves as an invaluable tool for developers, system administrators, and DevOps engineers for rapid prototyping, debugging Kafka applications, scripting common Kafka operations, and performing ad-hoc data manipulation directly from the terminal. Its ability to handle various data formats and integrate seamlessly with standard Unix pipes makes it exceptionally flexible for a wide range of Kafka-related tasks, from simple message testing to complex data stream processing.

kcat is primarily designed for development, debugging, and ad-hoc scripting, not as a robust production client. It relies on the librdkafka library, inheriting its capabilities and occasional limitations. While powerful for quick interactions, configuring advanced Kafka features like SSL, SASL authentication, or Kerberos can require a deep understanding of librdkafka properties passed via the -X option, which might be less straightforward than using high-level Kafka client libraries.

kcat is extensively used for quick Kafka interactions, including:
Producing messages: Piping log files or other data directly into a Kafka topic (e.g., cat file.log | kcat -P -t mytopic).
Consuming messages: Monitoring real-time data streams for debugging or inspection (e.g., kcat -C -t mytopic -o end -q).
Metadata inspection: Quickly checking broker availability, topic partition layouts, or consumer group offsets (e.g., kcat -L -b localhost:9092 or kcat -L -G mygroup).
Scripting: Integrating Kafka operations into shell scripts for automation purposes.

The -X option is highly powerful, allowing users to pass any valid librdkafka configuration property. This includes network settings, security protocols (SSL, SASL), client IDs, debug options, and more. For a complete list of properties, refer to the librdkafka documentation or the kcat man page (usually man kcat). Examples include -X security.protocol=SASL_SSL or -X ssl.ca.location=/path/to/ca.pem.

Developed by Magnus Edenhill, the lead developer of the librdkafka C library, kcat was originally known as kafkacat. It quickly gained popularity as a versatile and fast command-line tool for Kafka due to its direct integration with the high-performance librdkafka. In 2021, the project was officially renamed to kcat to mitigate potential trademark concerns related to the Apache Kafka brand, while maintaining its core functionality and user base.

kafka-console-producer.sh(1), kafka-console-consumer.sh(1), kafka-topics.sh(1)