Cut-offs 🚀

Dataflow monitoring can be enabled by defining cut-off rules ensuring that data is in the right place at the right time. The cut-off function monitors if flows are completed within a pre-defined time frame. It is available exclusively for modeled flows. Cut-offs can be set both at flow and node level. Head over to this page for details.

You can set Cut-offs either at flow level or at node level, but not both! Be aware that you can save Cut-offs at both levels at the same time in DRAFT version dataflows. However, the DRAFTS will not be promoted to MASTER with these settings.

Within Data Watcher, 2 different cut-offs can be configured:

• Duration cut-off: the file must reach a specific status by the end of the time interval. For example, within 2 hours after the beginning of the flow.

• Planned cut-off: the file must reach a specific status within the scheduled time frame. For example, by 10 am.

For each cut-off, a warning message or error message notification can be configured to inform the configured list of Recipients that the flow is lasting longer than expected. Learn how to set Notifications here.

To view the details of an already created Cut-off, select the clock icon in the Data Modeler → Dataflow Properties panel or in the Node Cut-off list.

Create a Cut-off

There are two panels in the Dataflow Modeler page where Cut-offs can be created:

• Dataflow Properties → Dataflow cut-off: this panel appears when a dataflow is designed in the Dataflow Modeler pane.

• Node Properties → Node Cut-off list: this panel appears when a node is selected in the Dataflow Modeler pane.

You can refer to the Dataflow Properties panel and Node Properties panel pages for details.

Follow these steps to create a Cut-off in the Dataflow cut-off panel:

The Cut-off creation process is the same both in the Dataflow cut-off panel described below and in the Node Cut-off list panel!

1. In the Dataflow Properties panel, click on the NEW button.

2. In the New Dataflow cut-off panel, the Enable toggle is on by default. Click on it if you want to disable the cut-off.

3. Enter a Name for your cut-off.

4. Choose the Category: Duration or Planned according to your needs - see the sections below.

5. Click the SAVE button. Once you have saved a Cut-off, it will appear in the Dataflow Cut-off list below. Click on the 3-dots icon to Disable, Edit, or Delete the Cut-off.

You can only add one Duration Cut-off per flow!

Duration type cut-off

• Max duration time: enter minutes of the time interval as needed.

• Activity Type (only for Node Cut-offs):

  • Any: any event occurring in the system, e.g. the anti-virus scan, an incoming event, an outgoing event.

  • Incoming: the timer will stop when the file enters the flow.

  • Outgoing: the timer will stop when the file exits the flow.

• Warning:

  • No (default).

  • Select how many minutes before an event the warning will be sent. Values go from 1 minute to 360 minutes (6 hours).

• Notification: configure notifications for your cut-offs. Learn more here.

Planned type cut-off

• Activity Type (only for Node Cut-offs): - Any: any event occurring in the system, e.g. the anti-virus scan, an incoming event, an outgoing event. - Incoming: the timer will stop when the file enters the flow. - Outgoing: the timer will stop when the file exits the flow.

Time settings panel

• Calendar:

  • Every day

  • Every working day

  • Administrators can configure additional calendars, see Cut-off Calendars.
• Time zone: select the time zone.

• Time: set at what time the file must have reached the required status.

• Cron: either set a cron expression manually or click on the Generate button.

• Max advance: it can be modified as described in Define a time window for early cut-offs here below.

• Warning: - No (default). - Select how many minutes before an event the warning will be sent. Values go from 1 minute to 360 minutes (6 hours).

Instances panel

The Instances panel is a Dataflow Cut-off option only!

• Instances configuration:

  • Single Instance Dataflow: select this option when one dataflow instance is expected for the time range.

  • Multiple Instance Dataflow: in the Number of Instances field, enter the number of expected Dataflow instances for each time range set.

  • Ignore exceeding completed instances: this option is enabled by default, and the cut-off status is not affected by several completed instances higher than expected. When the option is disabled, if at least one exceeding instance is completed, the cut-off evaluation status will turn into "Error" and instances will be marked as "Overflow".

• Notification: configure notifications for your cut-offs. Learn more here.

Define a time window for early cut-offs

If a cut-off notice arrives hours after the expected time, an optimist will see it as an early notice and a pessimist as a late notice. Both of them may be right. To solve this ambiguity, the Max Advance field can be used to define a time window within which an early cut-off is accepted. Isn't this smart?

For example, imagine that a daily cut-off is defined every day at noon. If a file arrives at 8 am, it might be a file exceeding the cut-off of the day before (therefore late) or a file arriving earlier than expected (therefore in advance). The Max Advance field solves this ambiguity. In our example, setting the Max Advance field to 6 hours, the file arriving at 8 am would be in advance and therefore correct.

This field is updated automatically according to the value entered in the Time field. However, it can be modified as needed, selecting from 1 to 24 hours.

When a cut-off is set on the node, the clock symbol appears in the node box.

🚀 Managing cut-off notifications

Cut-off notifications can be switched on or off by clicking the 3 dots icon on the Dataflow Models page and selecting Enable Cut-off notifications or Disable Cut-off notifications.

If you select multiple models, you can set multiple cut-offs in one go by going to the Select drop-down menu and choosing the Enable Cut-Off Notifications or Disable Cut-Off Notifications entry.