Pipeline Log
Pipeline Log
Description
Allows to log the activity of a pipeline with another pipeline.
A pipeline log streams logging information from a running pipeline to another pipeline.
The only requirement for the receiving pipeline is that it starts with a Pipeline Logging transform. Other than that, the logging pipeline is "just another pipeline". In this logging pipeline, you can process the logging information and e.g. write to a relational or NoSQL database, a Kafka topic etc
Samples
The samples project comes with an example pipeline log.
Check the metadata perspective for the pipeline log pipeline-log-example. This pipeline log is configured to send logging information for the pipeline ${PROJECT_HOME}/reflection/generate-fake-books.hpl to the logging pipeline ${PROJECT_HOME}/reflection/pipeline-log-example.hpl.
Related Plugins
Options
Name
The name to be used for this pipeline log
Enabled?
true
Logging parent pipelines only
false
If you enable this, only the parent pipelines, executed by Hop Run, GUI, Server or API will be logged. When disabled, every pipeline execution will be logged.
Pipeline executed to capture logging
the pipeline to process the logging information for this pipeline log
Execute at the start of the pipeline?
true
Should this pipeline log be executed at the start of a pipeline run
Execute at the end of the pipeline?
false
Should this pipeline log be executed at the end of a pipeline run
Execute periodically during execution?
true
Should this pipeline log be executed periodically during a pipeline run
Interval in seconds
30
if executed periodically, indicates the interval at which the pipeline log is executed
Last updated