Common Properties of Transformers
Transformers have both input and output ports. They can:
- put together more data flows with the same metadata (Concatenate, SimpleGather and Merge);
- remove duplicate records (Dedup);
- filter data records (Filter);
- create samples from input records (DataSampler ), sort data records (ExtSort, FastSort and SortWithinGroups);
- multiply existing data flow (SimpleCopy);
- split one data flow into more data flows (Partition at all, but optionally also Dedup , Filter and Map);
- intersect two data flows (even with different metadata on inputs) (DataIntersection), aggregate data information (Aggregate);
- and perform much more complicated transformations of data flows (Map , Denormalizer , Normalizer , Rollup and XSLTransformer).
Metadata can be propagated through some of these transformers, whereas the same is not possible in such components that transform data flows in a more complicated manner. You must have the output metadata defined prior to configuring these components.
Some of these transformers use transformations that have been described above. For detailed information about how transformation should be defined, see Defining Transformations.
- Some Transformers can have a transformation attribute defined, it may be optional or required. For information about transformation templates for transformations written in CTL, see CTL Templates for Transformers.
- Some Transformers can have a transformation attribute defined, it may be optional or required. For information about transformation interfaces that must be implemented in transformations written in Java, see: Java Interfaces for Transformers.
Below is an overview of all Transformers:
| COMPONENT | SAME INPUT METADATA | SORTED INPUTS | INPUTS | OUTPUTS | JAVA | CTL | AUTO-PROPAGATED METADATA |
|---|---|---|---|---|---|---|---|
| Aggregate | - | x | 1 | 1 | x | x | x |
| Concatenate | ✓ | x | 1-n | 1 | x | x | ✓ |
| DataIntersection | x | ✓ | 2 | 3 | ✓ | ✓ | ✓ |
| DataSampler | - | x | 1 | n | x | x | ✓ |
| Dedup | - | ✓ | 1 | 1-2 | x | x | ✓ |
| Denormalizer | - | x | 1 | 1 | ✓ | ✓ | x |
| ExtSort | - | x | 1 | 1-n | x | x | ✓ |
| FastSort | - | x | 1 | 1-n | x | x | ✓ |
| Filter | - | x | 1 | 1-2 | x | x | ✓ |
| Merge | ✓ | ✓ | 2-n | 1 | x | x | ✓ |
| MetaPivot | - | x | 1 | 1 | x | x | ✓ |
| Normalizer | - | x | 1 | 1 | ✓ | ✓ | x |
| Partition | - | x | 1 | 1-n | [1] | [1] | ✓ |
| Pivot | - | x | 1 | 1 | ✓ | ✓ | x |
| Map | - | x | 1 | 1-n | ✓ | ✓ | ✓ |
| Rollup | - | x | 1 | 1-n | ✓ | ✓ | x |
| SimpleCopy | - | x | 1 | 1-n | x | x | ✓ |
| SimpleGather | ✓ | x | 1-n | 1 | x | x | ✓ |
| SortWithinGroups | - | ✓ | 1 | 1-n | x | x | ✓ |
| XSLTransformer | - | x | 0-1 | 0-2 | x | x | x |
[1] Partition can use either a transformation or two other attributes (Ranges or Partition key). The transformation must be defined unless one of these is specified.
CTL Templates for Transformers
- Partition requires a transformation (which can be written in both CTL and Java) unless Partition key or Ranges are defined.
Remember that this component sends each record through the connected output port whose number is equal to the value returned by the transformation (Return Values of Transformations). Mapping does not need to be done, records are mapped automatically. - DataIntersection requires a transformation which can be written in both CTL and Java.
For more information about the transformation template, see CTL Templates for DataIntersection. - Map requires a transformation which can be written in both CTL and Java.
For more information about the transformation template, see CTL Templates for Map.
Remember that this component sends each record through the connected output port whose number is equal to the value returned by the transformation (Return Values of Transformations). Mapping must be defined for this port. - Denormalizer requires a transformation which can be written in both CTL and Java.
For more information about the transformation template, see CTL Templates. - Normalizer requires a transformation which can be written in both CTL and Java.
For more information about the transformation template, see CTL Templates for Normalizer. - Rollup requires a transformation which can be written in both CTL and Java.
For more information about the transformation template, see CTL Templates for Rollup.
Remember that this component sends each record through the connected output port whose number is equal to the value returned by the transformation (Return Values of Transformations). Mapping must be defined for this port.
Java Interfaces for Transformers
- Partition requires a transformation (which can be written in both CTL and Java) unless Partition key or Ranges are defined.
Remember that this component sends each record through the connected output port whose number is equal to the value returned by the transformation (Return Values of Transformations). Mapping does not need to be done, records are mapped automatically. - DataIntersection requires a transformation which can be written in both CTL and Java.
For more information about the interface, see Java Interfaces for DataIntersection. - Map requires a transformation which can be written in both CTL and Java.
For more information about the interface, see Java Interfaces for Map.
Remember that this component sends each record through the connected output port whose number is equal to the value returned by the transformation (Return Values of Transformations). Mapping must be defined for such port. - Denormalizer requires a transformation which can be written in both CTL and Java.
For more information about the interface, see Java Interface. - Normalizer requires a transformation which can be written in both CTL and Java.
For more information about the interface, see see Java Interface. - Rollup requires a transformation which can be written in both CTL and Java.
For more information about the interface, see see Java Interface.
Remember that this component sends each record through the connected output port whose number is equal to the value returned by the transformation (Return Values of Transformations). Mapping must be defined for such port.
Updated 11 months ago