Pipeline Logging
Pipeline Logging
Pipeline LoggingDescription
The pipeline logging transform provides detailed logging information about a pipeline and its transforms. This transform receives input from a Pipeline Log metadata item.
The transform requires very little configuration, but provides a lot of information.
Hop Engine
✓
Spark
?
Flink
?
Dataflow
?
Related Plugins
Options
Transform name
Name of this transform
Also log transform details
checked: the transform outputs both pipeline and transform logging and metrics. In this case, the logging will contain one line per transform. For each transform, the pipeline logging and metrics information is repeated.
unchecked: the transform only outputs pipeline logging and metrics.
Output Fields
Pipeline Logging Fields
loggingDate
Date
the date and time this pipeline was executed
loggingPhase
String
the phase this pipeline is in at the time of logging (e.g. 'end')
pipelineName
String
the logged pipeline name
pipelineFilename
String
the logged pipeline file name
pipelineStart
Date
execution start date for the logged pipeline
pipelineEnd
Date
execution end date for the logged pipeline
pipelineLogChannelId
String
logging channel id for the logged pipeline
parentLogChannelId
String
parent logging channel id for the logged pipeline (e.g. the channel id of the parent workflow)
pipelineLogging
String
full logging text for the logged pipeline
pipelineErrorCount
Integer
number of errors that occurred in the logged pipeline execution
pipelineStatusDescription
String
status description for the logged pipeline (e.g. 'Finished')
Transform Logging Fields
transformName
String
the name of the transform in the logged pipeline
transformCopyNr
Integer
the copy number of the current transform
transformStatusDescription
String
status of the current transform (e.g. 'Finished')
transformLogChannelId
String
log channel id for the current transform
transformLoggingText
String
logging text for the current transform
transformLinesRead
Integer
lines read for the current transform
transformLinesWritten
Integer
lines read for the current transform
transformLinesInput
Integer
input lines for the current transform
transformLinesOutput
Integer
output lines for the current transform
transformLinesUpdated
Integer
updated lines for the current transform
transformLinesRejected
Integer
rejected lines for the current transform
transformErrors
Integer
number of errors for the current transform
transformStart
Date
start date and time for the current transform
transformEnd
Date
end date and time for the current transform
transformDuration
Integer
total duration for the current transform
Last updated