Configuring DWA Emitters
INTRODUCTION
Each DWA Emitter has its own configuration section.
The following sections provide details on DWA Emitter configuration and are structured as follows:
- DWA Emitter common parameters
- DWA Emitter for XFERLOG
- DWA Emitter for OpenSSH SFTP Server
- DWA Emitter for C:D for distributed platforms
Each DWA Emitter has very FT/MFT middleware specific configuration requirements. The following sections provide details on each DWA Emitter will be discussed in more detail.
DWA EMITTER COMMON PARAMETERS
DWA Emitter common parameters -- Prefix emitter.<type>. [<label>].*
This section contains parameters that are shared by all DWA Emitters.
The actual property name contains a qualifier indicating the communicator type and its instance label, e.g. emitter.xferlog[MYXFERLOG1].active=false.
Several instances of the same type of emitter, identified by different labels, can be included in the same dwagent.properties file.
| Properties | Description |
|---|---|
| active = true | false | Switch that enables this DWA Emitter. NOTE: Several DWA Emitters can be active at runtime at any time. |
| logPollingTimeoutSeconds | Time to wait between each polling to check if new events have been produced. |
| logPollingMaxQuantity | Max number of log records to be read for each poll. |
| staleSessionCleanUpHours | Number of hours after which an idle session is marked as stale and removed from the emitter local cache. Sessions are persistent objects that emitters use to maintain state while scanning an event source, the persisted state information is normally very short lived. |
| localeLanguage localeCountry localeVariant | Combining these three values, the agent can understand and decode the Date format found in log files. It defaults to current user settings. The current values accepted for these three parameters are specified at this link: https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html |
DWA EMITTER FOR XFERLOG
Prerequisites
This emitter supports XFERLOG format, which is a very common log format supported by many FTP and FTP/S servers like WU-FTPD and vsFTPd.
XFERLOG is a text file where each line contains a transfer event made up of space-separated event attributes.
DWA Emitter for XFERLOG log format expects the following fields in the input XFERLOG formatted files:
- Current date and time
- Transfer time in seconds
- Remote host
- File size
- Filename and path
- Transfer type (a for ASCII, b for binary)
- Special action flag
- Direction (i for incoming, o for outgoing)
- Access mode
- Userid
- Service name
- Authentication method
- Authenticated userid
- Completion status (c for complete, i for incomplete)
Depending on the actual FTP Server type and version, XFERLOG logging might be turned off by default. Please refer to your FTP Server specific documentation for details on how to turn on XFERLOG logging.
DWA Emitter for XFERLOG allows for limited customization of the pattern matching configuration driving XFERLOG parsing, in normal circumstances the provided defaults will just work, but if the monitored FTP Server logs to XFERLOG in non-standard way they can be adjusted to some extent.
On the other hand, major changes in the log structure cannot be covered by simply adjusting pattern matching configuration.
DWA Emitter for XFERLOG handles rolling logs, i.e. log dynamically renamed by the monitored FTP Server according to some configurable policy (for instance, log file size threshold and/or log age threshold) giving rise to current log and archived logs.
Configuration reference
DWA Emitter for XFERLOG -- Prefix emitter.xferlog[<label>].*
All pattern properties included in the configuration conform to Java Message Format syntax as defined in:
https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
| Properties | Description |
|---|---|
| protocolType = ftp | ftps | Hint about FTP vs FTP/S protocol given that no detection is possible by reading XFERLOG contents. |
| logFilePath | Name of the current log file to monitor, complete with absolute path. The named file, plus any other file having that name plus a suffix will be processed. For example, when this parameter is set to: /usr/log/xferlog.log all files like the following will be processed (if available) in the right order and in accordance with dynamic file rotation policies: /usr/log/xferlog.log /usr/log/xferlog.log.1 /usr/log/xferlog.log.2 /usr/log/xferlog.log.3 … |
| logPattern | Pattern used to parse each xferlog log file line. The provided default value: {0,date,E MMM d HH:mm:ss yyyy} {1,number,integer} {2} {3,number,integer} {4} {5} {6} {7} {8} {9} {10} {11} {12} {13} successfully matches this sample log line: 'Wed Oct 24 16:34:27 2021 3 10.50.61.200 19123080 /export/home/user01/tmp/test.txt b _ i r user01 ftp 0 * c Where: {0} = current time {1} = transfer time in seconds {2} = remote host {3} = transferred file size {4} = transferred filename, file path is absolute {5} = transfer type (a for ascii, b for binary) {6} = special action flag {7} = direction (i for incoming, o for outgoing) {8} = access mode {9} = username {10} = service name {11} = authentication method {12} = authenticated user id {13} = completion status (c for complete, i for incomplete) |
DWA EMITTER FOR OPENSSH SFTP SERVER
Prerequisites
This emitter supports monitoring of Linux OpenSSH 8.0 SFTP Server configured with verbose logging.
Verbose logging can be turned on by configuring sftp service in /etc/ssh/sshd_config as shown below (root authority required):
Subsystem sftp /usr/libexec/openssh/sftp-server -l VERBOSE
The logging usually happens on /var/log/messages file, make sure that this file has been made accessible to DWAgent runtime user by your administrator (by default it is not).
Moreover, consider the option of asking your administrator to route just sftp-server messages to a dedicated log file that can be accessed by DWAgent at runtime, for better isolation and performance.
Example
Here is an example of a configuration with a dedicated /var/log/sftp.log for DWAgent containing just OpenSSH SFTP messages.
/etc/rsyslog.conf contents
$umask 0000 local3.* action(type="omfile" fileCreateMode="0644" file="/var/log/sftp.log")
/etc/ssh/sshd_config contents
Subsystem sftp /usr/libexec/openssh/sftp-server -l VERBOSE -f LOCAL3
DWA Emitter for OpenSSH SFTP Server allows for limited customization of the pattern matching configuration driving log parsing, in normal circumstances the provided defaults will just work, but if the monitored OpenSSH SFTP Server version logs in a slightly modified way, the default settings can be adjusted to some extent.
On the other hand, major changes in the log structure cannot be covered by simply adjusting pattern matching configuration.
DWA Emitter for OpenSSH SFTP Server handles rolling logs, i.e. log dynamically renamed by the system according to some configurable policy (for instance, log file size threshold and/or log age threshold) giving rise to current log and archived logs.
Configuration reference
DWA Emitter for OpenSSH SFTP Server -- Prefix emitter.osshsftp[<label>].*
All pattern properties included in the configuration conform to Java Message Format syntax as defined in:
https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
| Properties | Description |
|---|---|
| logFilePath | Name of the primary log file to monitor, complete with absolute path. |
| logFileFilterRegEx | Regular expression identifying all archived log files to monitor, located in the same directory configured for logFilePath primary log. Secondary log files are those logs that have been renamed by the system as part of some dynamic log rotation policy. |
| filterSftpdPattern | Pattern used to filter only SFTP lines related to current OpenSSH SFTP daemon. The provided default value is: {0,date,MMM d HH:mm:ss} {1} sftp-server[{2,number,integer}]: {3} Where: {0,date,MMM d HH:mm:ss} = timestamp according to the specified locale {1} = hostname {2} = session {3} = remainder of the line |
| sessionOpenedPattern | Pattern used to match user session opening. The provided default value is: session opened for local user {0} from [{1}] Where: {0} = user name {1} = client IP |
| sessionClosedPattern | Pattern used to match user session closing. The provided default value is: session closed for local user {0} from [{1}] Where: {0} = user name {1} = client IP |
| openPattern | Pattern used to match file opening. The provided default value is: open "{0}" flags {1} mode {2} Where: {0} = file name {1} = file opening flag {2} = file opening mode bitmask |
| closePattern | Pattern used to match file closing. The provided default value is: close "{0}" bytes read {1,number,integer} written {2,number,integer} Where: {0} = file name {1} = total read bytes {2} = total written bytes |
| sendStatusPattern | Pattern used to match send errors. The provided default value is: sent status {0} Where: {0} = error message |
| forcedClosePattern | Pattern used to match session errors/session abort. The provided default value is: forced close "{0}" bytes read {1,number,integer} written {2,number,integer} Where: {0} = file name {1} = total read bytes {2} = total written bytes |
DWA EMITTER FOR IBM STERLING CONNECT:DIRECT
Prerequisites
This emitter supports monitoring of IBM Sterling Connect:Direct 6.0.0 on distributed platforms (non-z/OS).
Actual detection of Connect:Direct (C:D) file transfer events is performed by accessing C:D Server statistics at runtime using C:D Java API.
C:D Java API jar must be provisioned by the customer on the machine where DWAgent is installed, this jar (usually named CDJAI.jar) is part of IBM Sterling Connect:Direct product installation and it is not distributed with DWAgent.
To include C:D Java API jar into DWAgent CLASSPATH you must create this environment variable DWAGENT_EXT_CP and set its value to the jar file name including its absolute path, then you must export it / make it in the environment where DWAgent is started.
IBM Sterling Connect:Direct versions different from 6.0.0 may be supported as well, depending on C:D Java API compatibility guarantees provided by IBM Sterling across versions.
Given that C:D Java API supports remote connections, DWAgent can also be installed on a machine different from the one where the monitored C:D Server is running.
Configuration reference
DWA Emitter for C:D for distributed platforms -- Prefix emitter.cdmini[<label>].*
| Properties | Description |
|---|---|
| nodeName port userId password | Hostname (nodeName), port and credentials to be used when connecting to a C:D instance to retrieve statistical records. |
| connectTimeoutSeconds | Connection timeout to be assumed when connecting to a C:D instance to retrieve statistical records. |
| startStatisticsFromDate | Optional start date and time in YYYYMMDD-HHMMSS format starting from which old statistical records will be processed by DWAgent the first time it is started. When this parameter is blank, the assumed default timestamp is the date and time DWAgent is started up for the first time. |
| additionalRecordIDs=IFED,RJED,RTED | Used to emit event also for non-file transfer operations in this list: if statement (IFED), Run Job (RJED) and Run Task (RTED) operations. |
Updated 6 months ago