Beam Kinesis Produce

Beam Kinesis Produce

Description

The Beam Kinesis Produce transform sends data (messages) to an Amazon Kinesis stream using a Beam pipeline execution engine.

Hop Engine	X
Spark	✓
Flink	✓
Dataflow	✓

Limitations

The main limitation of the Kinesis produce transform is that it currently only supports sending String data. Please create a request in GitHub if you want to see support for Avro messages and other formats.

Options

Option	Description
Transform name	Name of the transform, this name has to be unique in a single pipeline.
AWS access key	The Amazon Web Services access key to authenticate with.
AWS secret key	The Amazon Web Services secret key to authenticate with.
Stream name	The name of the Kinesis stream to write to.
The name of the data field to send	This is the name of the field that contains the actual data (message)
The type of data to send	Only String data (messages) are supported for now.
Partition key	The partition key to use
Configuration options	A list of configuration options. See here for a list of examples.