Beam Kinesis Consume

Beam Kinesis Consume

Description

The Beam Kinesis Consume transform consumes (reads) messages from an Amazon Kinesis stream using the Beam pipeline execution engine.

Hop Engine	X
Spark	✓
Flink	✓
Dataflow	✓

Limitations

The main limitation of the Kinesis consumer is that it currently only supports reading String messages. Please create a request in GitHub if you want to see support for Avro messages and other formats.

Options

Option	Description
Transform name	Name of the transform, this name has to be unique in a single pipeline.
AWS access key	The Amazon Web Services access key to authenticate with.
AWS secret key	The Amazon Web Services secret key to authenticate with.
Stream name	The name of the Kinesis stream to consume from.
The name of the unique ID output field	This is the name of the field in the output of the transform which will contain a unique ID for the data (message) received.
The name of the data output field	This is the name of the field that will contain the actual data (message)
The type of data retrieved	Only String data (messages) are supported for now.
Partition key field name	Optional: the name of the field in the output which will contain the partition key
Sequence number field name	Optional: the name of the field in the output which will contain the sequence number of the message
Sub-sequence number field name	Optional: the name of the field in the output which will contain the sub-sequence number of the message
Shard ID field name	Optional: the name of the field in the output which will contain the Shard ID of the message
Stream name field name	Optional: the name of the field in the output which will contain the name of the stream being read
Maximum number of records	Optional: the maximum number of records this transform will read before stopping
Maximum read time (ms)	Optional: the maximum time in ms this transform will read before stopping
Request record limit	Optional: the number of records which will be read at once in every request (micro-batch)
Arrival time watermark policy?	Watermark messages at the time of arrival in the Beam pipeline.
Arrival watermark idle time (ms)	Denotes the duration for which the watermark can be idle
Processing time watermark?	Watermark message at the time of processing in the Beam pipeline.
Fixed delay rate limit policy?	Specifies a fixed delay rate limit policy with the default delay of 1s
Fixed delay rate limit policy delay (ms)	Specifies a fixed delay rate limit policy with the given delay.
Maximum capacity per shard	Specifies the maximum number of messages per one shard.