Primeur Online Docs
Data Mover 1.20
Data Mover 1.20
  • 🚀GETTING STARTED
    • What is Primeur Data Mover
    • Main features of Primeur Data Mover
    • Primeur Data Mover deployment
    • Navigate through Primeur Data Mover
  • 📚INFRASTRUCTURE
    • STENG, clusters and servers
      • Adding a cluster and a STENG
      • Deleting a STENG
    • DMZ Gateways
      • Blocking users automatically at the DMZ level 🚀
    • DMZ Clusters
      • Load balancing across active DMZ clusters for outbound file transfers 🚀
  • 👥Actors
    • Who are the actors
    • Create your first actor
    • Configure an actor 🚀
      • Users Tab
      • Groups Tab
      • VFS Tab
      • File Resource Tab
      • Connection Contract Tab
      • Client Connections Tab
    • Search files by actor
    • Actor Lineage 🚀
      • Aggregation of flows by protocol 🚀
      • Lineage with connection contracts 🚀
      • Lineage with input, mediation and output contracts 🚀
      • Lineage with any contract type 🚀
  • 🗄️VIRTUAL FILE SYSTEMS
    • Virtual File Systems (VFS) 🚀
      • Creating a VFS 🚀
      • Configuring a VFS
      • Adding Virtual Paths
      • Modifying and Deleting a VFS
    • Searching files in all VFS
    • Storage Classes 🚀
      • Storage Class: SMB v3 or later versions 🚀
      • Storage Class: Azure Blob Storage 🚀
      • Storage Class: Amazon S3 🚀
      • Storage Class: Google Storage 🚀
      • Storage Class: Local File System 🚀
    • Retention Classes
  • 📝Contracts
    • What is a contract
    • Create your first contract
      • Create an Input Contract
        • Define the contract info
        • Associate the contract with the actor
        • Define the contract actions
        • Set the contract variables
      • Create a Mediation Contract
      • Create an Output Contract
      • Create a Connection Contract
        • Create a contract clause
        • Associate the VFS with file processing rules
        • File Processing Rules
    • Managing contracts 🚀
    • File Resources
      • Creating File Resources
      • Navigating File Resources
      • How to use File Resources
  • 🧱Workflows
    • What is a workflow
    • Create your first workflow template
    • Trigger types
      • Trigger types for input contracts
      • Trigger types for mediation and output contracts
    • Service tasks
      • Standard service tasks
      • Triggerable service tasks 🚀
      • Spazio selectors and filebox metadata management
      • Error management
    • Variables
      • Variables in workflows and contracts
      • Handling process variables
    • Workflow templates
      • System workflow templates
        • Workflow templates for input contracts
        • Workflow templates for mediation contracts
        • Workflow templates for output contracts
      • Custom workflow templates
        • Workflow template toolbar
        • Workflow template Shape repository panel
        • Workflow template working area
        • Workflow template BPMN-diagram panel
      • Error workflow templates
    • Editing workflow templates
    • DataFlow Instance Context (DFIC) 🚀
  • 🛸TRANSPORT PROTOCOLS AND CONNECTORS
    • Data Mover client and server roles
    • Client Connections
      • Client Connection: FTP
      • Client Connection: FTPS
      • Client Connection: SFTP
      • Client Connection: HTTP
      • Client Connection: HTTPS
      • Client Connection: PESIT
      • Client Connection: SMB v3 or later versions
      • Client Connection: POP3 or IMAP
      • Client Connection: SMTP
      • Client Connection: PR4/PR4S
      • Client Connection: PR5
      • Client Connection: PR5S
      • Client Connection: HDFS
      • Client Connection: HDFSS
      • Client Connection: Amazon S3 🚀
      • Client Connection: Google Cloud Storage
        • Credentials
      • Client Connection: Azure Blob Storage
      • Client Connection: IBM Sterling Connect:Direct
      • Appendix
    • Server Connections 🚀
      • Server Connection: FTP
      • Server Connection: FTPS
      • Server Connection: SFTP
      • Server Connection: HTTP
      • Server Connection: HTTPS
      • Server Connection: PeSIT
      • Server Connection: PR4
      • Server Connection: PR5
      • Server Connection: PR5S 🚀
      • Server Connection: IBM Sterling Connect:Direct
    • Stopping all servers in one go
  • 💻API
    • HTTP MFT Rest API
    • Job Manager APIs 🚀
    • SFTP Server sessions APIs 🚀
    • Audit Logs APIs 🚀
  • 🔓Security
    • Identity and Access Management
    • Users & Groups
      • Setting the password policy
      • Creating Internal Users 🚀
      • Creating Internal Groups
      • Creating External Users
      • Creating External Groups
    • Key Stores and Trust Stores
      • Key Store 🚀
        • Creating a Key 🚀
        • Creating a Certificate 🚀
        • Importing a Key or a Certificate
        • Creating a Symmetric key
        • Examples
      • Trust Store 🚀
        • Importing Keys 🚀
        • Importing Certificates
      • Untrusted Cache 🚀
      • Trusting Keys and Certificates
      • PGP Key Store and PGP Trust Store
        • PGP Key Store
        • Importing keys into the PGP Trust Store
    • ICAP
      • Configuring ICAP
      • Defining an ICAP rule
  • 🎧FILE EVENT LISTENER
    • What is the File Event Listener
    • Configuring File Event Listeners
      • Setting the File Event Listener Engine
      • Defining a contract for the File Event Listener
      • Setting events to be monitored 🚀
    • RegEx Rules 🚀
    • Monitoring File Event Listeners
  • 👑FILE MANAGER
    • What is the File Manager
    • Logging into File Manager
    • Managing the File Manager 🚀
      • The list of results
      • Creating new folders
      • Uploading files
      • Downloading files 🚀
      • Searching for files and folders
      • Deleting files 🚀
      • Bulk actions 🚀
    • File Manager and VFS
    • Customizing File Manager externals
      • The configuration-wui.json file 🚀
      • How to customize the Login window and the logo
      • How to customize the footer
      • How to configure the Upload with Metadata option
      • How to customize bulk actions 🚀
  • 🧑‍⚖️FILE ROUTING
    • What is File Routing 🚀
    • Routing Rules
      • The Rules tab
      • The Categories tab
      • The Output tab
    • How to create a rule 🚀
      • Add metadata 🚀
      • Select ACTIONS
      • Select OUTPUTS
      • Policy for the selection of metadata rules
    • Configuration of the environment in Data One
      • Set up Storage Classes
      • Set up Retention Classes
      • Configure the Actor
      • Set up File Resources
    • Associate the Routing Rule with a Contract
    • Example
  • 📩NOTIFICATION CHANNELS
    • What are Notification Channels
    • Configuring the default Email Notification Channel
    • Configuring a new Email Notification Channel
    • Trusting Certificates
    • Managing Templates
      • Data Watcher Macros
      • Contract Macros
      • ICAP Macros
      • Central Log Macros
      • Email Templates
      • Editing default templates
      • Loading a new template
  • 💬LOGS & AUDIT
    • Logs 🚀
      • Logs options 🚀
      • Troubleshooting error analysis in Logs
    • Audit Options 🚀
      • Export audit logs 🚀
      • List of Audit entity types 🚀
      • Audit message codes 🚀
    • Log Notifiers 🚀
      • FEL message codes
  • 🕒MONITORING
    • Jobs
      • Details about Jobs 🚀
      • jobman.sh CLI
    • Job Manager
    • Job Queues
      • Managing Job Queues
    • File Transfers
      • Ongoing
      • Finished
      • Reports
    • File Transfers Rules
      • Configuring Rules
  • 🧐HOW TO...
    • ... use different DNS names - NEW! 🚀
    • ... configure a Cron Expression
    • ... configure an application
    • ... customize a header
    • ... run searches in Data Watcher 🚀
    • ... use Data Shaper graphs in Data Mover contracts
    • ... modify DMCFG and deploy it
    • ... tune Data One data retention
    • ... fine tune Data Mover
  • 🗒️RELEASE NOTES
    • Data One 1.20.10
    • Data One 1.20.9
    • Data One 1.20.8
    • Data One 1.20.7
      • Data One 1.20.7.1
    • Data One 1.20.6
    • Data One 1.20.5
    • Data One 1.20.4
    • Data One 1.20.3
    • Data One 1.20.2
    • Data One 1.20.1
    • Data One 1.20.0
Powered by GitBook
On this page
  • System templates
  • Trigger type: On Demand
  • 1. SpExec operation
  • 2. Send email notification
  • Trigger type: New File
  • 1. Push file to monitored Virtual Path and optionally send a confirmation
  • 2. Push file to a remote location
  • 3. Get file
  • 🚀 4. Push new file as JMS message
  • Trigger type: Virtual Box Bind
  • 1. Push file to a remote directory
  • 2. Push file to a remote location and optionally send a confirmation and/or specify a flag file
  • 🚀 3. Push vBox bind file as JMS message
  • Dataflow templates
  • Trigger type: Virtual Box Bind
  • 1. Save file on filesystem (spget)
  • 2. Push file
  1. Workflows
  2. Workflow templates
  3. System workflow templates

Workflow templates for output contracts 🚀 HIDDEN

Pagina per JMS con system_newFile_JMS-push, system_vBind_JMS-push

Last updated 2 days ago

System templates

The following properties are common to all templates. They are also optional and therefore, can be skipped if not needed.

Common properties:

  • Error handling The Enable workflow error handling option can be set to ON to use PRIMEUR system workflows to manage errors. Enabling the toggle switch, the Select error handling template drop-down list will appear with 2 templates: one to log all variables and one to notify errors. You can find details about the ehwft_logAllVariables and ehwft_logAllVariables templates in the section.

  • systemEnableLogDebugToFile Set this toggle button to ON to activate logs

Trigger type: On Demand

The On Demand trigger does not require any mandatory variable. The On Demand triggered workflow templates are triggered by a direct API invocation via REST. This API will accept a map of variables that will be passed to the instance as process variables. Any variable explicitly declared on the template to be requested when creating the Contract is added to the set of variables coming from the REST invocation. If a variable name is used when creating the Contract and is used in the map passed via REST as well, this second one wins over the Contract one.

1. SpExec operation

system_onDemand_exec

Goal: System template that will perform a remote spExec operation when onDemand triggered.

Variables:

  • Cluster Steng cluster where the spExec operation is performed

  • executable Executable command to be run on the remote steng shell

  • arguments Arguments that will be passed to the selected executable command

2. Send email notification

system_onDemand_sendTemplatedEmail

Goal: System template OnDemand triggered sending a notification email.

Variables:

  • Cluster Cluster that performs the transport

  • Select remote server Select the remote server

  • mailSubjectTemplate Subject of the email

  • templateId ID of the Template

  • mailRecipients Email addresses of the recipients

  • mailFrom Sender of the email

  • templateDefaultReplacer Template default variable re-placer

Trigger type: New File

A New File trigger requires a VirtualPath.

FileSet is written by the FileBox component in the VirtualPath configured in the Contract. The variable added to the execution is file of type FileDetail. Its attributes must be set according to the event that starts this trigger.

1. Push file to monitored Virtual Path and optionally send a confirmation

system_newFile_flagFile-triggerFile-outTransport

Goal: System Template triggered by a New file that remotely pushes a data file. If appropriate options are specified, it checks that a flag file exists before pushing and/or sending a confirmation file after pushing. Optionally, it can notify that the pull operation is successful using the notification channel options.

Operation:

  1. If the name of a flag file is specified, before proceeding Data Mover checks that a flag file exists. If the flag file is specified but not found, the workflow exits. If the flag file is not specified or if the flag file is both specified and found in the appropriate folder, Data Mover sends the file that triggered the workflow through Virtual Box bind to the desired destination.

  2. If the name of a confirmation file is specified, a dummy confirmation file is created and sent to the specified folder.

  3. If the Notification Channel option is specified, a notification is sent to the appropriate recipients through the selected Notification Channel.

Variables:

  • Select Virtual Path

    • When the Select Virtual Path toggle is ON: Actor, VFS, VFS Path Destination Virtual path where data files will be pulled.

    • When the Select Virtual Path toggle is OFF: Enter VFSNAME:/VPATH The * wildcard character is accepted.

  • remoteFolder Remote folder where the file will be written.

  • Select remote server Transfer profile (or Client Connection) to use to push the data file to destination.

  • flagFileName Flag file filename If set, before trying to pull files, the workflow will check the presence of a Flag file with this filename and extension in the selected folder.

  • remotePathFlagFile Flag file folder Applicable only if the Flag file name is set. Remote folder where the Flag file will be verified

  • removeFlagFile Applicable only if the Flag file name is set. If enabled, the Flag file will be removed after successfully pulling the file.

  • confirmationFileName Trigger file filename If set, once the file is successfully pushed, a confirmation file with this filename (and extension) will be pushed

  • remotePathConfirmationFile Trigger file folder Applicable only if trigger file filename is set. Remote path where the trigger file will be pushed

  • Actor, VFS, VFS Path Actor, VFS and Virtual Path where the file will be placed. Applicable only if trigger file filename is set. Virtual Path for trigger file.

  • Notification Channel If set, the success of the push operation will be notified using the selected notification channel

    • Embedded Broker Notification Channel: this is a notification channel that transforms input notification events into output JSON JMS messages, publishing them on a user-defined queue on Data Mover embedded Artemis broker. EBNC is not and end-user feature, but rather an enabler feature that can be flexibly exploited in integration projects involving custom applications that need to interact with Data Mover.

  • newFileFilter Enter the string to filter new files. For example, you may want the flow to be triggered only for files beginning with a specific filename or extension.

2. Push file to a remote location

system_newFile_push

Goal: Push the file triggering the instance to a remote location.

Notes: It is possible to rename the file while pushing it.

Variables:

  • Select Virtual Path

    • When the Select Virtual Path toggle is ON: Actor, VFS, VFS Path Destination Virtual path where data files will be pulled.

    • When the Select Virtual Path toggle is OFF: Enter VFSNAME:/VPATH The * wildcard character is accepted.

  • Cluster Cluster where the operation will be executed.

  • spPush_remotePath Remote Path where the file triggering this instance will be pushed to.

  • Select remote server Transfer profile used for pushing the file onto the remote path.

  • File resource Resource profile to apply during the Push operation.

  • spPush_renameFile If set, its value will be the new "file and extension" for the file written with the Push operation.

  • newFileFilter Enter the string to filter new files. For example, you may want the flow to be triggered only for files beginning with a specific filename or extension.

3. Get file

system_newFile_getFile

Goal: System template newFile triggered that copies the file triggering the instance to a physical folder.

Notes: The file can be renamed

Variables:

  • Select Virtual Path

    • When the Select Virtual Path toggle is ON: Actor, VFS, VFS Path Destination Virtual path where data files will be pulled.

    • When the Select Virtual Path toggle is OFF: Enter VFSNAME:/VPATH The * wildcard character is accepted.

  • destination_path Path where the file triggering this instance will be saved to.

  • destination_filename If set, its value will be the new ”file and extension” for the file written with the Get operation.

  • File resource Resource profile to apply during the file Get operation.

  • newFileFilter Enter the string to filter new files. For example, you may want the flow to be triggered only for files beginning with a specific filename or extension.

🚀 4. Push new file as JMS message

system_newFile_JMS-push

Goal: New File triggered system template that performs a JMS push action.

Variables:

  • Select Virtual Path

    • When the Select Virtual Path toggle is ON: > Actor, VFS, VFS Path Destination Virtual path where data files will be pulled. Virtual path storing the file that will be split in one or more messages.

    • When the Select Virtual Path toggle is OFF: > Enter VFSNAME:/VPATH The * wildcard character is accepted.

  • Cluster Cluster where the push operation will be executed.

  • JMSDestination Name of the JMS destination.

  • JMSDestinationType Enter QUEUE or TOPIC.

  • Priority You can set a priority to the messages that are created when splitting the file.

  • resourceProfile (Optional) This is the resource profile to apply during the file Push operation. In the output contract, this variable is in the FileResource field.

  • DeliveryMode Method of delivery of JMS messages. Select one of the options available:

    • Persistent: the message is saved and its delivery is ensured.

    • Non_persistent: the message must not be saved and could be lost (for example in case of problems on the server).

  • TimeToLive (Optional) You can specify how many seconds messages will live. A number higher than 0 is accepted.

  • JMS TransferProfile Enter the Client Connection that must be used to push the file to the remote path. In the output contract, this variable is in the Select remote server field.

  • newFileFilter Enter the string to filter new files. For example, you may want the flow to be triggered only for files beginning with a specific filename or extension.

Trigger type: Virtual Box Bind

A Virtual Box Unbind requires a VirtualBox. The instance starts when a VirtualBoxEvent is emitted by FileBox when the FileSet and the Virtual Box defined in the Contract are no longer associated. The variable added to the execution is file of type FileDetail. Its attributes must be set according to the event that fires this trigger.

1. Push file to a remote directory

system_vbBind_outTransport

Goal: Virtual Box Bind triggered System template that will remotely push a file with an optional resource profile and optionally backs it up on VFS.

Variables:

  • sourceVirtualBox Enter the source virtual box or create a new one. In the output contract this variable is in the Select VirtualBox field.

  • virtualBoxFilter If you want the flow to be triggered only for specific virtual boxes, enter the string to filter virtual boxes.

  • Cluster Cluster that performs the transport.

  • remoteFolder Remote folder where the file will be written.

  • Select remote server Transfer profile to use for the transport.

  • Actor, VFS, VFS Path Actor, VFS and Virtual Path where the file will be placed. If specified, the file will be backed up to the desired destination.

  • File resource If specified a resource profile will be applied.

2. Push file to a remote location and optionally send a confirmation and/or specify a flag file

system_vbBind_flagFile-triggerFile-outTransport

Goal: System Template, triggered by a Virtual Box bind, pushing a file remotely. If appropriate options are specified, the presence of a flag file can be verified before pushing and/or sending a confirmation file after pushing. Optionally, it can notify that the pull operation is successful using the notification channel options.

Description:

  1. If the name of a flag file is specified, before proceeding Data Mover verifies that a flag file exists. If the flag file is specified but not found, the workflow exits. If the flag file is not specified or if the flag file is both specified and found in the appropriate folder, Data Mover sends the file that triggered the workflow through Virtual Box bind to the desired destination.

  2. If the name of a confirmation file is specified, a dummy confirmation file is created and sent to the specified folder.

  3. If the Notification Channel option is specified, a notification is sent to appropriate recipients through the selected Notification Channel.

Variables:

  • sourceVirtualBox Enter the source virtual box or create a new one. In the output contract this variable is in the Select VirtualBox field.

  • virtualBoxFilter Enter the string to filter virtual boxes. For example, you may want the flow triggered only for specific virtual boxes.

  • remoteFolder Remote folder where the file will be written.

  • Select remote server Transfer profile (or Client Connection) to use to push the data file to destination.

  • flagFileName Optionally, a flag file can be specified so that the data file is pulled only when the flag file is in the specified folder. The pulled data file can optionally be archived and/or deleted at the source.

  • remotePathFlagFile (Required only if Flag file name is set) Remote folder where Flag file presence will be checked.

  • removeFlagFile (Required only if Flag file name is set) If enabled, the flag file will be removed after successful data file pull.

  • confirmationFileName Trigger file filename If set, once the file is successfully pushed, a confirmation file with this filename (and extension) will be pushed.

  • remotePathConfirmationFile Trigger file folder Applicable only if trigger file filename is set. Remote path where the trigger file will be pushed.

  • Actor, VFS, VFS Path Actor, VFS and Virtual Path where the file will be placed.

  • Notification Channel If set, the success of the push operation will be notified using the selected notification channel.

    • Embedded Broker Notification Channel: this is a notification channel that transforms input notification events into output JSON JMS messages, publishing them on a user-defined queue on Data Mover embedded Artemis broker. EBNC is not and end-user feature, but rather an enabler feature that can be flexibly exploited in integration projects involving custom applications that need to interact with Data Mover.

  • confirmationFilePath Local path for the creation of a trigger file.

🚀 3. Push vBox bind file as JMS message

system_vBind_JMS-push

Goal: System template triggered by a Virtual Box bind. It performs a JMS push action.

Variables:

  • sourceVirtualBox Enter the source virtual box or create a new one. In the output contract this variable is in the Select VirtualBox field.

  • virtualBoxFilter Enter the string to filter virtual boxes. For example, you may want the flow triggered only for specific virtual boxes.

  • DeliveryMode Method of delivery of JMS messages. Select one of the options available:

    • Persistent: the message is saved and its delivery is ensured.

    • Non_persistent: the message must not be saved and could be lost (for example in case of problems on the server).

  • TimeToLive (Optional) You can specify how many seconds messages will live. A number higher than 0 is accepted.

  • JMSTransferProfile Enter the Client Connection that must be used to push the file to the remote path. In the output contract this variable corresponds to the Select remote server field.

  • Cluster Cluster where the file will be pushed.

  • JMSDestination Name of the JMS destination.

  • JMSDestinationType Only QUEUE is currently supported.

  • Priority You can set a priority to the messages that are created when splitting the file.

  • resourceProfile (Optional) This is the resource profile that will be applied when pushing the file. In the output contract this variable corresponds to the File Resource field.

Dataflow templates

These templates can be used with dataflows.

Common variables:

  • systemEnableLogDebugToFile Set this toggle button to ON to activate logs.

Trigger type: Virtual Box Bind

A Virtual Box Unbind requires a VirtualBox. The instance starts when a VirtualBoxEvent is emitted by FileBox when the FileSet and the Virtual Box defined in the Contract are no longer associated. The variable added to the execution is file of type FileDetail. Its attributes must be set according to the event that fires this trigger.

1. Save file on filesystem (spget)

df_vbBind_outFilesystem

Goal: Virtual Box Bind triggered DataFlow template that saves a file on filesystem (spget), and optionally backs it up on a VFS.

Variables:

  • sourceVirtualBox Enter the source virtual box or create a new one. In the output contract this variable is in the Select VirtualBox field.

  • virtualBoxFilter If you want the flow to be triggered only for specific virtual boxes, enter the string to filter virtual boxes.

  • jobManagerPriorityQueueName Default Queue.

  • Cluster Cluster where the operation will be executed.

  • destinationFolder Destination folder where the file will be written on the filesystem.

  • Actor, VFS, VFS Path Actor, VFS and Virtual Path where the file will be placed.

2. Push file

df_vbBind_outTransport

Goal: Virtual Box Bind triggered DataFlow template that remotely pushes a file and optionally backs it up on a VFS. Eligible for DataFlow.

Variables:

  • sourceVirtualBox Enter the source virtual box or create a new one. In the output contract this variable is in the Select VirtualBox field.

  • virtualBoxFilter If you want the flow to be triggered only for specific virtual boxes, enter the string to filter virtual boxes.

  • remoteFolder Remote folder where the file will be written.

  • Select remote server Select the remote server.

  • Actor, VFS, VFS Path Actor, VFS and Virtual Path where the file will be placed.

System Email Notification Channel: see the page for details. In the Add email field, enter the email address and click the + button to add it.

The JMS connector's documentation can be found in the section.

MessageProcessingRuleF2M Select the splitting Message Processing Rule to be applied. It must be configured in Setup > .

PropagatesDataOneMetadata (Optional) Enable it to propagate system metadata to JMS Properties. See the page for details.

System Email Notification Channel: see the page for details. In the Add email field, enter the email address and click the + button to add it.

The JMS connector's documentation can be found in the section.

MessageProcessingRuleF2M Select the splitting Message Processing Rule to be applied. It must be configured in Setup → .

PropagatesDataOneMetadata (Optional) Enable it to propagate system metadata to JMS Properties. See the page for details.

Error handling The Enable workflow error handling option can be set to ON to use PRIMEUR system workflows to manage errors. Enabling the toggle switch, the Select error handling template drop-down list will appear with 2 templates: one to log all variables and one to notify errors. You can find details about the ehwft_logAllVariables and ehwft_logAllVariables templates in the section.

🧱
Error workflow templates
Configuring the default Email Notification Channel
JMS Connector
Message processing rule
Metadata mapping for split file content
Configuring the default Email Notification Channel
JMS Connector
Message processing rule
Metadata mapping for split file content
Error Workflow Templates