... tune Data One data retention

This recap topic provides details and pointers to other documentation pages discussing Data One own data retention for managed entities and files.

Each production or non-production environment has its own unique data retention requirements, driven by regulatory compliance rules and/or company-specific best practices and conventions, so fine tuning of data retention parameters is entirely expected.

Central log retention tuning

Data One product components logs entries are written into a database table known as Central Log, or CLOG (see also Logs). Log entries are retained for a configurable period of time.

The property that regulates this retention can be tuned via Setup → Advanced Settings by selecting: ghibli-messages module, db-cleaner section and delete-messages-older-than property.

The value is expressed in days, using ISO-8601 syntax, for example "P14D" means fourteen days.

The default retention period is set to 7 days (i.e. "P7D").

Executed contract instance retention tuning

Contract execution generates two types of data for each run: a Job Manager job instance and a BPMN workflow instance. The retention of each of these instances can be independently tuned, but for consistency reasons it is recommended to keep the two retention settings close to one another, enabling optimal two-way cross-navigation between job instances and workflow instances in the WUI.

Executed contract job instance retention tuning

For each executed contract, the Job Manager retains executed jobs details for a configurable period of time.

The property that regulates this retention can be tuned via Setup → Advanced Settings by selecting: job-manager module, config section and archive.retentionDays property.

The default retention period is set to 5 days.

Executed contract workflow instance retention tuning

Data One executed workflow instances associated to input contracts, mediation contracts and output contracts are retained for a configurable period of time.

The property that regulates this retention is named BPMN_HISTORY_CLEANER_RETENTION_DAYS and must be injected into the Extended Configuration section of the DMCFG.

The default retention period is set to 7 days.

This example sets retention to 14 days:

ext_cfg:
  BPMN_HISTORY_CLEANER_RETENTION_DAYS: 14

Please refer to this topic for additional information on the DMCFG lifecyle: How do I modify DMCFG and deploy it?

Executed file transfer job instance retention tuning

For each executed file transfer, the Job Manager retains executed job instance details for a configurable period of time.

The property that regulates this retention can be tuned via Setup → Advanced Settings by selecting: ghibli-speng-centralnode module, transportMonitor-config section and job.archive.retentionDays property.

The default retention period is set to 15 days.

Data Watcher managed event and dataflow instance retention tuning

Data Watcher incoming events are persisted to the MongoDB database for a period of time decided by the user.

Data One ships a mongodb-cleaner.sh CLI that must be independently scheduled at regular intervals (e.g. twice a day) using operating system level schedulers, such as cron, to remove all events older than a given number of days.

No automatic scheduling of the cleaner CLI is provided, nor a default value for the retention days parameter. Explicit user action is required.

For more information, please refer to this page: Clean up the Data Watcher MongoDB.

Storage class managed file retention tuning

Files managed by the platform are organized in virtual paths belonging to a virtual file system. Each virtual path is underpinned by some physical storage whose retention is regulated by a storage class descriptor. The descriptor determines distinct retention periods for processed files (expiry after get parameter) and unprocessed files (expiry after put parameter).

No default retention period is provided, each retention class can be tuned independently from the others.

For more information, please refer to Retention Classes.