Data Partitioning (Parallel Running)
Last updated
Last updated
This chapter describes way to speed up graph runs with help of data partitioning.
Data partitioning runs parts of graph in parallel. A component that is a bottleneck of a graph is run in multiple instances and each instance processes one part of the original data stream.
The processing can be further scaled to Cluster without modification to the graph.
Data partitioning is convenient to speed up processing when:
some components are significantly slower than other components in a graph.
The way to speed up the run is to partition the data and run the slow component in parallel.
In Designer and Server, you can speed up processing with copying the slow component and running it in parallel.
There is a better solution that avoids copying components and is scalable.
Set allocation to the components positioned between the Cluster components: right click the component and choose Set Allocation.
In Component Allocation dialog choose By number of workers and enter the number of parallel workers.
Components in your graph will contain text denoting the allocation.
Data partitioning runs part of a graph in parallel. The number of parallel workers is configured without copying the components. Data-partitioned graphs can take advantage of Data Shaper Cluster without modification.
Clean design, no duplication. Avoid copying parts of graph to speed-up the processing. Set the number of parallel workers with a single option.
Scales to Cluster. You can use the same graph on a multi-node Cluster without any additional modifications.
Maximize use of available hardware. Take advantage of parallel processing on multi-core processors.
When you run some component in parallel, you should be aware of limits of hardware and other systems.
If you run parallel a component that connects to a web service, you may reach the limit of parallel connections to the service or run out of the quota on number of requests.
Consider other jobs running on server. Too many jobs running in parallel may slow down run of other graphs.
Some tasks cannot be easily parallelized.
In , you can partition files with temporary data to multiple Cluster nodes using Partitioned sandboxes. A file stored in a partitioned sandbox is split into several parts. Each part of the file is on a different Cluster node. This way, you can partition both: processing and data. It reduces amount of data being transferred between Cluster nodes.
components communicate over the network with high latency (, );
If you run parallel a component that does many I/O operations (e.g ), you may be limited by speed of hard drive.
If you run parallel a component that opens many files (e.g ), you may reach limit on number of opened files.
