aws-kinesis

Show all streams in the account

$ aws kinesis list-streams

$ aws kinesis put-record --stream-name [name] --partition-key [key] --data [base64_encoded_message]

$ aws kinesis put-record --stream-name [name] --partition-key [key] --data "$( echo "[my raw message]" | base64 )"

$ aws kinesis list-shards --stream-name [name]

$ aws kinesis get-shard-iterator --shard-iterator-type TRIM_HORIZON --stream-name [name] --shard-id [id]

$ aws kinesis get-records --shard-iterator [iterator]

aws kinesis <subcommand> [<parameters>] [<global-options>]

Examples:
aws kinesis create-stream --stream-name my-data-stream --shard-count 1
aws kinesis put-record --stream-name my-data-stream --partition-key user1 --data SGVsbG8gS2luZXNpcyE=

--region <region-name>
    Specifies the AWS region to send the request to (e.g., us-east-1). This is a global AWS CLI option.

--output <output-format>
    Specifies the output format (e.g., json, text, table). This is a global AWS CLI option.

--profile <profile-name>
    Uses a specific profile from the AWS credential file. This is a global AWS CLI option.

--endpoint-url <url>
    Specifies an override URL for the service endpoint. This is a global AWS CLI option.

--stream-name <string>
    The name of the Kinesis data stream. Required for most stream-related operations.

--shard-count <integer>
    The number of shards in the stream. Used specifically with the create-stream subcommand.

--partition-key <string>
    A hash value used to determine which shard a record belongs to. Required for put-record.

--data <blob>
    The data blob to put into the record, typically Base64 encoded. Required for put-record.

--shard-id <string>
    The unique identifier of the shard. Used with operations like get-shard-iterator or get-records.

--shard-iterator-type <string>
    Determines where to start reading records within a shard (e.g., AT_SEQUENCE_NUMBER, LATEST). Used with get-shard-iterator.

--limit <integer>
    The maximum number of records to return. Used with get-records.

The aws kinesis command is a crucial component of the AWS Command Line Interface (CLI), providing a powerful interface to interact with various AWS Kinesis services directly from your terminal. Kinesis is a scalable, real-time data streaming service designed to process large streams of data. The aws kinesis command allows users to perform operations such as creating and deleting data streams, putting records into streams, retrieving records from shards, and managing stream consumers.

It supports Kinesis Data Streams for real-time data ingestion, Kinesis Firehose for data delivery to destinations like S3 or Redshift, and Kinesis Video Streams for video data. This CLI tool is essential for automation, scripting, and quick administrative tasks related to Kinesis resources, offering flexibility that complements the AWS Management Console and SDKs. It streamlines the process of managing streaming data workflows, enabling developers and administrators to integrate Kinesis into their existing CI/CD pipelines or operational scripts.

IAM Permissions: Users must have appropriate AWS Identity and Access Management (IAM) permissions to perform Kinesis operations. Lack of permissions will result in 'Access Denied' errors.

Region Specificity: Kinesis streams are region-specific. Operations must be targeted to the correct AWS region using the '--region' option or environment variables.

Data Encoding: When putting records, the '--data' parameter expects Base64 encoded input. The CLI handles this for simple strings, but for binary data, explicit encoding is required.

Throughput Limits: Kinesis streams have throughput limits per shard. Exceeding these limits can lead to 'ProvisionedThroughputExceededException'.

Asynchronous Operations: Many Kinesis operations (like stream creation or deletion) are asynchronous, meaning the command returns quickly but the resource change takes time to propagate.

The AWS CLI supports '--query' and '--output' options, allowing users to filter and reformat the JSON output using JMESPath expressions, which is extremely powerful for scripting and automation.

For commands that return large sets of data (e.g., 'list-streams', 'get-records'), the CLI automatically handles pagination. Users can also control pagination using '--starting-token', '--max-items', and '--page-size' options.

AWS Kinesis was launched in 2013 as a fully managed service for real-time processing of streaming data, initially focusing on Kinesis Data Streams. AWS has since expanded the Kinesis family to include Kinesis Firehose (2015), Kinesis Data Analytics (2016), and Kinesis Video Streams (2017). The aws CLI integration for Kinesis has evolved alongside the service, providing comprehensive command-line access to these new capabilities, becoming an indispensable tool for developers and operators managing streaming data workloads on AWS.

aws(1), curl(1), jq(1), grep(1), kafka-console-producer(1) (from Apache Kafka)