File Event Listener

The File Event Listener Engine is one of the components of Primeur Data One™ whose goal is to monitor changes to files residing on shared file systems, SMB3 shares and SFTP Servers accessible from STENG peers of a STENG cluster, in an efficient way.

Monitored locations are not directly managed by Data One, but simply use a source for customer-provided input files to be processed by Data One.

The File Event Listener Engine detects all changes in the monitored directories, i.e. new files appearing on those directories as well as existing files being updated.

The only exception to this behavior is when a previously detected file is deleted by the user or Data One and then, for whatever reason, is copied again to the same monitored location with identical file name, file size and last modification timestamp. If this occurs within 24 hours of the initial detection, the file is ignored and considered an accidental duplicate to be skipped. If this happens after 24 hours, the file is processed again.

Once the File Event Listener detects a change, an on-demand FEL input contract is triggered to process the file.

Processed files can remain in their original location and will never be processed again, since FEL keeps track of the status of the monitored location.

The File Event Listener Engine operates in a parallel fashion, with one distinct thread to the monitoring of a group of directories. These worker threads are called File Event Listeners.

The File Event Listener Engine runtime behavior is active/passive: peers coordinate to elect the leader and ensure that at any given time just one peer holds the leader role and polls directories, while all other peers will stand-by, waiting to take over the role of leader in case of crash or shutdown of the current leader.

Irrespective of monitored locations types, the File Event Listener persists internal status information on a dedicated shared file system, which requires an NFS 4.1 and these mandatory mount settings:

nfsvers=4.1,sec=sys,sync,intr,noac,hard,lookupcache=none,timeo=50,retrans=2