# Beam Kinesis Consume

## <img src="/files/BUTbQCJwaS9LcYgjhOYZ" alt="" data-size="line"> Beam Kinesis Consume

### Description <a href="#description" id="description"></a>

The Beam [Kinesis](https://aws.amazon.com/kinesis/) Consume transform consumes (reads) messages from an Amazon Kinesis stream using the Beam pipeline execution engine.

| Hop Engine | X            |
| ---------- | ------------ |
| Spark      | <sup>✓</sup> |
| Flink      | <sup>✓</sup> |
| Dataflow   | <sup>✓</sup> |

### Limitations

The main limitation of the Kinesis consumer is that it currently only supports reading String messages. Please create a request in GitHub if you want to see support for Avro messages and other formats.

### Options

| Option                                   | Description                                                                                                                  |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| Transform name                           | Name of the transform, this name has to be unique in a single pipeline.                                                      |
| AWS access key                           | The Amazon Web Services access key to authenticate with.                                                                     |
| AWS secret key                           | The Amazon Web Services secret key to authenticate with.                                                                     |
| Stream name                              | The name of the Kinesis stream to consume from.                                                                              |
| The name of the unique ID output field   | This is the name of the field in the output of the transform which will contain a unique ID for the data (message) received. |
| The name of the data output field        | This is the name of the field that will contain the actual data (message)                                                    |
| The type of data retrieved               | Only String data (messages) are supported for now.                                                                           |
| Partition key field name                 | Optional: the name of the field in the output which will contain the partition key                                           |
| Sequence number field name               | Optional: the name of the field in the output which will contain the sequence number of the message                          |
| Sub-sequence number field name           | Optional: the name of the field in the output which will contain the sub-sequence number of the message                      |
| Shard ID field name                      | Optional: the name of the field in the output which will contain the Shard ID of the message                                 |
| Stream name field name                   | Optional: the name of the field in the output which will contain the name of the stream being read                           |
| Maximum number of records                | Optional: the maximum number of records this transform will read before stopping                                             |
| Maximum read time (ms)                   | Optional: the maximum time in ms this transform will read before stopping                                                    |
| Request record limit                     | Optional: the number of records which will be read at once in every request (micro-batch)                                    |
| Arrival time watermark policy?           | Watermark messages at the time of arrival in the Beam pipeline.                                                              |
| Arrival watermark idle time (ms)         | Denotes the duration for which the watermark can be idle                                                                     |
| Processing time watermark?               | Watermark message at the time of processing in the Beam pipeline.                                                            |
| Fixed delay rate limit policy?           | Specifies a fixed delay rate limit policy with the default delay of 1s                                                       |
| Fixed delay rate limit policy delay (ms) | Specifies a fixed delay rate limit policy with the given delay.                                                              |
| Maximum capacity per shard               | Specifies the maximum number of messages per one shard.                                                                      |


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.primeur.com/data-shaper-1.21/knowing-the-data-shaper-designer/pipelines/transforms/beamkinesisconsume.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.