Dataflow Modeler

In the Dataflow Modeler page you can classify flows and associate them to a model to transform them into Dataflows. In this way, you create Dataflow Models.
A Dataflow Model is the abstract description of a type of flow, with specific features, and it defines the transfer of expected information between the monitored systems involved or between the monitored systems and other external sources or destination systems.

Head over to the Design section of the Data Watcher documentation for the detailed steps you must follow to create a Dataflow Model. Here is the link.

When you define a Dataflow, you create a model that will be used to recognize flows and transform them into Dataflows.
Once a flow is recognized through the Dataflow Model - in particular through the defined matching criteria - you can take advantage of additional services, such as defining attributes, associating them to the flow and setting Cut-offs.

Matching criteria

Matching criteria are groups of rules that are applied to monitored flows to transform them into dataflows.
Multiple matching criteria can be defined and each group of criteria can be applied to a specific node.
It is up to you to decide whether each group must be searched in AND (all rules must be matched to recognize the model) or in OR (only one rule is enough to recognize the model).
The set of rules inside each group of matching criteria is always searched in AND.
Matching criteria are defined in the Dataflow Properties > Matching Criteria pane. Follow this link to learn how to set them.

Cut-offs

Cut-offs are timing rules applied at business level to control the duration of a flow or of an activity or to monitor if a flow or an activity is completed in a specific time frame.

Cut-offs can be defined:

• at flow-level and in this case they are called Dataflow Cut-offs
• at node-level and they are called Node Cut-offs (a node is a system that is monitored by Data Watcher)

In both cases, cut-offs can:

• control the duration of a flow or of an activity
• monitor if a flow or an activity is completed in a specific time frame

The former are called Duration cut-offs and the latter Planned cut-offs.

Dataflow cut-offs

A Dataflow cut-off is defined at flow level. With this cut-off, you can set the duration and timing requirements of the transfers expected on the flow, without going down to the node level.

Duration Dataflow Cut-off
Use this cut-off to monitor the duration of a Dataflow instance . An alert is sent if the instance runs for too long.

Planned Dataflow Cut-off
Use this cut-off to define an expected number of Dataflow instances to complete within a specific recurring time frame, ensuring that Dataflows are meeting their performance goals. You can add multiple time windows with recurrence for each node and specify the number of expected Dataflow Instances that must be observed within a given time frame or within a set of time frames:

• 1 dataflow instance every working day before 7pm with a max advance of 2 hours
• 3 dataflow instances every 6 hours, every day
• 1 dataflow instance from 5am to 6am and 1 dataflow instance from 2pm to 4pm, every day

Node Cut-offs

A Node cut-off is defined at node level and is used to control the duration or the timing requirements on the transfers that are expected on the monitored system.
You can also set the Activity of the cut-off, that is if the transfer is in input to a node or in output from a node.
Multiple Cut-offs can be defined on each node.

Duration Node Cut-off
Use this cut-off to monitor the duration of the transfers expected on the node. An alert is sent if the instance runs for too long.

Planned Node Cut-off
Use this cut-off to define an expected number of transfers that will complete within a specific recurring time frame.

Overlap

Planned Dataflow and Node Cut-offs can have different intervals defined and can be made recurring with a Cron Expression.
While setting the slots when the checks will occur, it may happen that time frames overlap. This can become an issue because the system will no longer be able to understand if a Dataflow instance satisfies a cut-off or another.
To make an example, imagine a cut-off planned from 3pm to 7pm and another from 6pm to 8pm. The system would not be able to understand which of the 2 cut-offs a dataflow instance running at 6.30pm should satisfy.

To avoid this overlap, the time frame defined for Planned Cut-offs will be checked as soon as the model is promoted to Master. Up to when the Model is a Draft, anything can be defined and no control will occur. As soon as the model is promoted to Master, possible overlapping cut-offs will be identified. No worries, a warning message will appear and you will be able to fix the time frame.

Other restrictions verified when promoting a model to Master version:

• Only one Duration cut-off can be created.
• Duration cut-offs and planned cut-offs can be created for the same instance.
• If a Dataflow cut-off exists, a Node cut-off cannot be added and vice versa.
• Multiple cut-offs can be set at node level.

Time is expressed in minutes.
The time-zone of the Browser installed is loaded.

Want to know how to set cut-offs? Then the Setting Cut-offs page is what you need!