Avro File Output
Avro File Output
Avro File OutputDescription
The Avro File Output transform can write Apache Avro messages to a file or field.
Hop Engine
✓
Spark
✓
Flink
✓
Dataflow
✓
Options
Transform name
Name of the transform. Note: This name has to be unique in a single pipeline.
Output Type
The type of output you want, this can be binary file, binary message or a JSON message. When selecting message it will be sent to the Output Field.
Filename
The filepath where to save the avro data file.
Output Field
When not writing to a file this is the fieldname used to output the message to.
Automatically create schema
You can either select an existing schema or create a schema based on the fields in the fields tab.
Write schema to file
Select this to write the schema to a separate file.
Avro namespace
Namespace used in the Avro file.
Avro record name
Name used for the Avro record.
Avro documentation
Schema filename
Filepath to write or read the schema from.
Create parent folder
When creating a file, select this if you want to create the parent folder.
Compression codec
Compression used for the avro messages when writing to a file.
Include transform nr in filename
Add the Transform instance number to the filename.
Include partition nr in filename
Add the Partition number to the filename.
Include date in filename
Include the date when the file was created in the filename
Include time in filename
Include the time when the file was created in the filename
Specify date format
Specify a custom date and format to attach to the filename
Date time format
Date time specification used for the formatting
Add filenames to result
Add the resulting filenames that are created to the filename results
Fields tab:
Name
The source field in the stream.
Avro Schema Path
The destination field, when an existing schema is specified this is populated with the fields from the schema to map the input fields to.
Avro Type
The Avro type of the field in the Avro destination file.
Nullable
Is the field nullable?
Metadata Injection Support
To inject the Avro field type use following codes
AVRO_TYPE_NONE = 0 AVRO_TYPE_BOOLEAN = 1 AVRO_TYPE_DOUBLE = 2 AVRO_TYPE_FLOAT = 3 AVRO_TYPE_INT = 4 AVRO_TYPE_LONG = 5 AVRO_TYPE_STRING = 6 AVRO_TYPE_ENUM = 7
Last updated