Engine configuration

Suggest Edits

Data Shaper internal settings (defaults) are stored in the defaultProperties file located in the Data Shaper engine. This source file contains various parameters that are loaded at run-time and used during the transformation execution. We do not recommend changing values in this file.

In the Designer, the path to the file is plugins/com.cloveretl.gui/lib/lib/cloveretl.engine.jar. In the Server Core, the path to the file is WEB-INF/lib/cloveretl.engine.jar.

If you need to change the default setting, create a local file with only those properties you need to override and place the file in the project directory. To instruct Data Shaper to retrieve the properties from this local file, go to Window > Preferences > Data Shaper > Data Shaper Runtime and either define the path to the file in the Data Shaper Engine Properties field or put the following parameter in the VM parameters field:

-Dclover.engine.config.file=/full/path/to/file.properties

Note: engine properties have to be set for each workspace individually.

Content of defaultProperties file

Here we present some of the properties and their values as they are presented in the defaultProperties file:

  • Record.RECORD_LIMIT_SIZE = 268435456
    Limits the maximum size of a record. Theoretically, the limit can be set very high, but you should keep it as low as possible for an easier error detection. For more details on memory demands, see Edge Memory Allocation.

  • Record.RECORD_INITIAL_SIZE = 65536
    Sets the initial amount of memory allocated to each record. The memory can grow dynamically up to Record.RECORD_LIMIT_SIZE, depending on how memory-greedy an edge is. See Edge Memory Allocation.

  • Record.FIELD_LIMIT_SIZE = 268435456
    Limits the maximum size of one field within a record. For more details on memory demands, see Edge Memory Allocation.

  • Record.FIELD_INITIAL_SIZE = 65536
    Sets the initial amount of memory allocated to each field within a record. The memory can grow dynamically up to Record.FIELD_LIMIT_SIZE, depending on how memory-greedy an edge is. See Edge Memory Allocation.

  • Record.DEFAULT_COMPRESSION_LEVEL = 5
    This sets the compression level for compressed data fields (cbyte).

  • DEFAULT_INTERNAL_IO_BUFFER_SIZE = 32768
    Determines the internal buffer size the components allocate for I/O operations. Increasing this value affects performance negligibly.

  • USE_DIRECT_MEMORY = false
    The Data Shaper engine can use direct memory for data records manipulation. For example, the underlying memory of the CloverBuffer (container for serialized data records) uses direct memory (if the usage is enabled). This attribute is by default false.
    Using direct memory can slightly improve performance in some cases. However, direct memory is out of control of a Java Virtual Machine, as the direct memory is allocated outside of the Java heap space in direct memory. If the OutOfMemory exception occurs and usage of direct memory is enabled, try to turn it off.

  • DEFAULT_DATE_FORMAT = yyyy-MM-dd

  • DEFAULT_TIME_FORMAT = HH:mm:ss

  • DEFAULT_DATETIME_FORMAT = yyyy-MM-dd HH:mm:ss

  • DEFAULT_REGEXP_TRUE_STRING = true|T|TRUE|YES|Y|t|1|yes|y

  • DEFAULT_REGEXP_FALSE_STRING = false|F|FALSE|NO|N|f|0|no|n

  • DataParser.DEFAULT_CHARSET_DECODER = UTF-8

  • DataFormatter.DEFAULT_CHARSET_ENCODER = UTF-8

  • Lookup.LOOKUP_INITIAL_CAPACITY = 512
    The initial capacity of a lookup table when created without specifying the size.

  • DataFieldMetadata.DECIMAL_LENGTH = 12
    Determines the default maximum precision of decimal data field metadata. Precision is the number of digits in a number, e.g. the number 123.45 has a precision of 5.

  • DataFieldMetadata.DECIMAL_SCALE = 2
    Determines the default scale of decimal data field metadata. Scale is the number of digits to the right of the decimal point in a number, e.g. the number 123.45 has a scale of 2.

  • Record.MAX_RECORD_SIZE = 33554432

    Warning!

    This is a deprecated property. Nowadays, you should use Record.RECORD_LIMIT_SIZE.


Limits the maximum size of a record. Theoretically, the limit is tens of MBs, but you should keep it as low as possible for easier error detection.

Capt. Eddie to ground control:

"You can define locale that should be used as the default one. The setting is the following: # DEFAULT_LOCALE = en.US By default, system locale is used by Data Shaper. If you un-comment this row you can set the DEFAULT_LOCALE property to any locale supported by Data Shaper, see the List of all Locale). Similarly, the default time zone can be overridden by un-commenting the following entry: # DEFAULT_TIME_ZONE = 'java:America/Chicago';'joda:America/Chicago' For more information about time zones, see the Time Zone section."


Updated 4 months ago