Primeur Online Docs
Data Shaper
Data Shaper
  • 🚀GETTING STARTED
    • What is Primeur Data Shaper
      • What is the Data Shaper Designer
      • What is the Data Shaper Server
      • What is the Data Shaper Cluster
    • How does the Data Shaper Designer work
      • Designer Views and Graphs
      • Data Shaper Graphs
      • Designer Projects and Sandboxes
      • Data Shaper Designer Reference
    • How do the Data Shaper Server and Cluster work
      • Data Shaper Server and Cluster
      • Data Shaper Server Reference
    • VFS Graph Components
      • DataOneFileDescriptor (DOFD) metadata
      • Passing parameters from Data One Contract to Data Shaper graph
      • Inheriting Data One context attributes in Data Shaper graph
  • DATA SHAPER DESIGNER
    • Configuration
      • Runtime configuration
        • Logging
        • Master Password
        • User Classpath
      • Data Shaper Server Integration
      • Execution monitoring
      • Java configuration
      • Engine configuration
      • Refresh Operation
    • Designer User Interface
      • Graph Editor with Palette of Components
      • Project Explorer Pane
      • Outline Pane
      • Tabs Pane
      • Execution Tab
      • Keyboard Shortcuts
    • Projects
      • Creating Data Shaper projects
      • Converting Data Shaper projects
      • Structure of Data Shaper projects
      • Versioning of server project content
      • Working with Data Shaper Server Projects
      • Project configuration
    • Graphs
      • Creating an empty graph
      • Creating a simple graph
        • Placing Components
        • Placing Components from Palette
        • Connecting Components with Edges
    • Execution
      • Successful Graph Execution
      • Run configuration
      • Connecting to a running job
      • Graph states
    • Common dialogs
      • URL file dialog
      • Edit Value dialog
      • Open Type dialog
    • Import
      • Import Data Shaper projects
      • Import from Data Shaper server sandbox
      • Import graphs
      • Import metadata
    • Export
      • Export graphs to HTML
      • Export to Data Shaper Server sandbox
      • Export image
      • Export Project as Library
    • Graph tracking
      • Changing record count font size
    • Search functionality
    • Working with Data Shaper server
      • Data Shaper server project basic principles
      • Connecting via HTTP
      • Connecting via HTTPS
      • Connecting via Proxy Server
    • Graph components
      • Adding components
      • Finding components
      • Edit component dialog box
      • Enable/disable component
      • Passing data through disabled component
      • Common properties of components
      • Specific attribute types
      • Metadata templates
    • Edges
      • Connecting Components with Edges
      • Types of Edges
      • Assigning Metadata to Edges
      • Colors of Edges
      • Debugging Edges
      • Edge Memory Allocation
    • Metadata
      • Records and Fields
        • Record Types
        • Data Types in Metadata
        • Data Formats
        • Locale and Locale Sensitivity
        • Time Zone
        • Autofilling Functions
      • Metadata Types
        • Internal Metadata
        • External (Shared) Metadata
        • SQL Query Metadata
        • Reading Metadata from Special Sources
      • Auto-propagated Metadata
        • Sources of Auto-Propagated Metadata
        • Explicitly Propagated Metadata
        • Priorities of Metadata
        • Propagation of SQL Query Metadata
      • Creating Metadata
        • Extracting Metadata from a Flat File
        • Extracting Metadata from an XLS(X) File
        • Extracting Metadata from a Database
        • Extracting Metadata from a DBase File
        • Extracting Metadata from Salesforce
        • SQL Query Metadata
        • User Defined Metadata
      • Merging Existing Metadata
      • Creating Database Table from Metadata and Database Connection
      • Metadata Editor
        • Opening Metadata Editor
        • Basics of Metadata Editor
        • Record Pane
        • Field Name vs. Label vs. Description
        • Details Pane
      • Changing and Defining Delimiters
      • Editing Metadata in the Source Code
      • Multi-value Fields
        • Lists and Maps Support in Components
        • Joining on multivalue fields (Comparison Rules)
    • Connections
      • Database Connections
        • Internal Database Connections
        • External (Shared) Database Connections
        • Database Connections Properties
        • Encryption of Access Password
        • Browsing Database and Extracting Metadata from Database Tables
        • Windows Authentication on Microsoft SQL Server
        • Snowflake Connection
        • Hive Connection
        • Troubleshooting
      • JMS Connections
      • QuickBase Connections
      • Hadoop Connections
      • Kafka Connections
      • OAuth2 Connections
      • MongoDB Connections
      • Salesforce Connections
    • Lookup Tables
      • Lookup Tables in Cluster Environment
      • Internal Lookup Tables
      • External (Shared) Lookup Tables
      • Types of Lookup Tables
    • Sequences
      • Persistent Sequences
      • Non Persistent Sequences
      • Internal Sequences
      • External (Shared) Sequences
      • Editing a Sequence
      • Sequences in Cluster Environment
    • Parameters
      • Internal Parameters
      • External (Shared) Parameters
      • Secure Graph Parameters
      • Graph Parameter Editor
      • Secure Graph Parameters
      • Parameters with CTL2 Expressions (Dynamic Parameters)
      • Environment Variables
      • Canonicalizing File Paths
      • Using Parameters
    • Internal/External Graph Elements
    • Dictionary
      • Creating a Dictionary
      • Using a Dictionary in Graphs
    • Execution Properties
    • Notes in Graphs
      • Placing Notes into Graph
      • Resizing Notes
      • Editing Notes
      • Formatted Text
      • Links from Notes
      • Folding Notes
      • Notes Properties
    • Transformations
      • Defining Transformations
      • Transform Editor
      • Common Java Interfaces
    • Data Partitioning (Parallel Running)
    • Data Partitioning in Cluster
      • High Availability
      • Scalability
      • Graph Allocation Examples
      • Example of Distributed Execution
      • Remote Edges
    • Readers
      • Common Properties of Readers
      • ComplexDataReader
      • DatabaseReader
      • DataGenerator
      • DataOneVFSReader
      • EDIFACTReader
      • FlatFileReader
      • JSONExtract
      • JSONReader
      • LDAPReader
      • MultiLevelReader
      • SpreadsheetDataReader
      • UniversalDataReader
      • X12Reader
      • XMLExtract
      • XMLReader
      • XMLXPathReader
    • Writers
      • Common Properties of Writers
      • DatabaseWriter
      • DataOneVFSWriter
      • EDIFACTWriter
      • FlatFileWriter
      • JSONWriter
      • LDAPWriter
      • SpreadsheetDataWriter
      • HIDDEN StructuredDataWriter
      • HIDDEN TableauWriter
      • Trash
      • UniversalDataWriter
      • X12Writer
      • XMLWriter
    • Transformers
      • Common Properties of Transformers
      • Aggregate
      • Concatenate
      • DataIntersection
      • DataSampler
      • Dedup
      • Denormalizer
      • ExtSort
      • FastSort
      • Filter
      • Map
      • Merge
      • MetaPivot
      • Normalizer
      • Partition
      • Pivot
      • Rollup
      • SimpleCopy
      • SimpleGather
      • SortWithinGroups
      • XSLTransformer
    • Joiners
      • Common Properties of Joiners
      • Combine
      • CrossJoin
      • DBJoin
      • ExtHashJoin
      • ExtMergeJoin
      • LookupJoin
      • RelationalJoin
    • Others
      • Common Properties of Others
      • CheckForeignKey
      • DBExecute
      • HTTPConnector
      • LookupTableReaderWriter
      • WebServiceClient
    • CTL2 - Data Shaper Transformation Language
    • Language Reference
      • Program Structure
      • Comments
      • Import
      • Data Types in CTL2
      • Literals
      • Variables
      • Dictionary in CTL2
      • Operators
      • Simple Statement and Block of Statements
      • Control Statements
      • Error Handling
      • Functions
      • Conditional Fail Expression
      • Accessing Data Records and Fields
      • Mapping
      • Parameters
      • Regular Expressions
    • CTL Debugging
      • Debug Perspective
      • Importing and Exporting Breakpoints
      • Inspecting Variables and Expressions
      • Examples
    • Functions Reference
      • Conversion Functions
      • Date Functions
      • Mathematical Functions
      • String Functions
      • Mapping Functions
      • Container Functions
      • Record Functions (Dynamic Field Access)
      • Miscellaneous Functions
      • Lookup Table Functions
      • Sequence Functions
      • Data Service HTTP Library Functions
      • Custom CTL Functions
      • CTL2 Appendix - List of National-specific Characters
      • HIDDEN Subgraph Functions
    • Tutorial
      • Creating a Transformation Graph
      • Filtering the records
      • Sorting the Records
      • Processing Speed-up with Parallelization
      • Debugging the Java Transformation
  • DATA SHAPER SERVER
    • Introduction
    • Administration
      • Monitoring
    • Using Graphs
      • Job Queue
      • Execution History
      • Job Inspector
    • Cluster
      • Sandboxes in Cluster
      • Troubleshooting
  • Install Data Shaper
    • Install Data Shaper
      • Introduction to Data Shaper installation process
      • Planning Data Shaper installation
      • Data Shaper System Requirements
      • Data Shaper Domain Master Configuration reference
      • Performing Data Shaper initial installation and master configuration
        • Creating database objects for PostgreSQL
        • Creating database objects for Oracle
        • Executing Data Shaper installer
        • Configuring additional firewall rules for Data Shaper
Powered by GitBook
On this page
  • boolean
  • byte
  • cbyte
  • date
  • decimal
  • integer
  • long
  • number (double)
  • string
  • list
  • Assignments
  • map
  • variant
  • record
  1. DATA SHAPER DESIGNER
  2. Language Reference

Data Types in CTL2

PreviousImportNextLiterals

Last updated 1 month ago

For basic information about data types used in metadata, see . In any program, you can use some variables. Data types in CTL are the following:

  • boolean

  • byte

  • cbyte

  • date

  • decimal

  • integer

  • long

  • number (double)

  • string

  • list

  • map

  • variant

  • record

boolean

The boolean data type contains values of logical expressions.

The default value is false.

It can be either true or false.

Its declaration looks like this: boolean identifier;.

Example 54. Declaration of boolean variable

boolean b;        // declaration
boolean b = true; // declaration with assignment

byte

This data type stores binary data of a length that can be up to Integer.MAX_VALUE as a maximum. The default value is null. Its declaration looks like this: byte identifier;.

Example 55. Declaration of byte variable

byte b;
// declaration of variable with assignment
byte b = hex2byte("414243");

cbyte

This data type is a compressed representation of byte data type to reduce runtime memory footprint. Compressed size of the data can be up to Integer.MAX_VALUE as a maximum.

The default value is null.

Its declaration looks like this: cbyte identifier;.

Example 56. Declaration of cbyte variable

cbyte c1;
cbyte c2 = hex2byte("61"); // declaration with assignment

date

The date data type contains date and time.

The default value is 1970-01-01 00:00:00 GMT.

Its declaration looks like this: date identifier;

Example 57. Declaration of date variable

// declaration of variable
date d;
// declaration of variable with assignment from function
date d = str2date("1600-01-31", "yyyy-MM-dd");

Note: If you work with date, you should be aware of time zone of the data.

decimal

The decimal data type serves to store decimal numbers.

Calculations with the decimal data type are performed in fixed point arithmetic. It makes decimal data type suitable for calculations with money.

The default value is 0.

Its declaration looks like this: decimal identifier;.

By default, any decimal may have up to 32 significant digits. If you want to have different **Length **or Scale, you need to set these properties of decimal field in metadata.

Example 58. Usage of decimal data type in CTL2 If you assign 100.0 / 3 to a decimal variable, its value might for example be 33.333333333333335701809119200333. As 100.0 is double and 3 is integer, the both operands were firstly converted to double, then the value has been calculated and finally the result value has been converted to decimal. Assigning it to a decimal field (with default Length and Scale, which are 12 and 2, respectively), it will be converted to 33.33D.

You can cast any float number to the decimal data type by appending the d letter to its end.

Any numeric data type (integer, long, number/double) can be converted to decimal.

Example 59. Declaration of decimal variable

decimal d;
decimal d2 = 4.56D; // declaration of variable with assignment

integer

The integer data type can contain integral values.

CTL2 integer can store values from -2147483648 to 2147483647.

The integer data type can overflow (i.e. adding 1 to the maximum value returns -2147483648; similarly, subtracting 1 from the minimum value returns 2147483647) which may lead to errors and/or incorrect results.

The default value is 0.

Its declaration looks like this: integer identifier;

Note: The value -2147483648 can be stored in CTL2 variable but cannot be stored in an integer field of record metadata (value of the field would be null). If the value -2147483648 is expected to arise, consider usage of data type with wider range of values in metadata; e.g. long.

If you append the L letter to the end of any integer number, you can cast it to the long data type.

Integercan be converted to long, double or decimal using automatic conversions.

Example 60. Declaration of integer variable

integeri1;
integer i2 = 1241;

long

long is an integral data type allowing to store greater values than the integer data type.

CTL2 long can store values from -9223372036854775808 to 9223372036854775807.

The long data type can overflow (i.e. adding 1 to the maximum value returns -92233720368547758088; similarly, subtracting 1 from the minimum value returns 9223372036854775807) which may lead to errors and/or incorrect results.

The default value is 0.

Its declaration looks like this: long identifier;

Note: The value -9223372036854775808 can be stored in CTL2 variable but the value is used in long field in record metadata for null value. If the value -9223372036854775808 is expected to arise, consider usage of data type with wider range of values in metadata; e.g. decimal.

Any integer number can be cast to long data type by appending the l letter to its end.

Long data type can be converted to number/double or decimal without explicit casting. Example 61. Declaration of long variable

long myLong;
long myLong2 = 2141L;

number (double)

The number data type is used for floating point number.

The default value is 0.0.

Its declaration looks like this: number identifier;

If you need a data type for money amount, we advise using decimal instead of number (double).

The integer and long data types can be converted to double using automatic conversions. If long is being converted to number (double), lost of precision may occur.

Number(double) can be converted to decimal without explicit casting.

Example 62. Declaration of number (double) variable

double d;
double d2 = 1.5e2;

string

This data type serves to store sequences of characters.

The default value is empty string.

The declaration looks like this:string identifier;

Example 63. Declaration of string variable

string s;
string s2 = "Hello world!";

list

The type of elements of a list may be any other data type, including nested lists or maps.

The elements of a list are indexed by integers starting from 0.

Its declaration can look like this: string[] identifier;

For nested lists or maps, use the following syntax instead: listtype of elements>] identifier;

The default list is an empty list.

Example 64. List

integer[] myIntegerList;
myIntegerList[5] = 123;

// Customer is metadata record name
Customer JohnSmith;
Customer PeterBrown;
Customer[] CompanyCustomers;
CompanyCustomers[0] = JohnSmith;
CompanyCustomers[1] = PeterBrown;

// Nested lists and maps:
list[list[string]] listOfLists;
list[map[string, integer]] listOfMaps;

Assignments

  • myStringList[3] = "abc"; The string "abc" is put to the fourth position in the string list. The preceding items are filled with null as follows: myStringList is [null,null,null,"abc"]

  • myList1 = myList2; Assigns a copy of myList2 to myList1. It means that both lists will contain the same elements.

  • myList1 = myList1 + myList2; Adds all elements of myList2 to the end of myList1. Both lists must be based on the same primitive data type.

  • myList = []; Assigns an empty list to myList.

  • myList = ["a", "b", "c"]; Assigns a list containing three strings to myList.

  • myList = null; Discards the previous value of myList.

map

This data type is a container of pairs of a key and a value. Its declaration looks like this: maptype of key>, ]identifier; The Value can be any of the other data types, including records, nested lists or other maps, but the Keycan only be a primitive data type: boolean, date, decimal, integer, long, number or string. The default map is an empty map.

Example 65. Map

map[string, boolean] map1;
map1["abc"] = true;

// Customer is the name of record
Customer JohnSmith;
Customer PeterBrown;

map[integer, Customer] CompanyCustomersMap;
CompanyCustomersMap[JohnSmith.ID] = JohnSmith;
CompanyCustomersMap[PeterBrown.ID] = PeterBrown;

// Nested maps and lists:
map[string, map[string, integer]] mapOfMaps;
map[integer, list[string]] mapOfLists;
  • myMap["abc"] = 7; Puts the value 7 into myMap under the key "abc".

  • myMap2 = myMap1; Assigns a copy of myMap1 to myMap2.

  • myMap = {}; Assigns an empty map to myMap.

  • myMap = { "a" -> 20, "b" -> 10, "c" -> 30 }; Assigns a map containing three key-value pairs to myMap.

  • myMap = null; Discards the previous value of myMap.

variant

Variables of this type can be assigned values of any other type - no type checking is performed. In particular, variant can contain nested lists and maps, so it can be used for tree-like data with unknown structure, such as JSON. Its declaration looks like this: variant identifier;

Note: variant data type is supported in data records too. $out.0.myVariantField = myVariantVariable;

Variant can be used like lists and maps, allowing to access inner values using square brackets [ ]. The operation will fail unless the variable contains a list or a map at runtime.

The default value is null, so the variable must be initialized to an empty list or map before inserting inner values.

Example 66. Variant

variant myVariant = {};
myVariant["one"] = 1;
myVariant["string"] = "not a number";
myVariant["two"] = 2;

// working with unknown structures:
integer sum = 0;
if ( myVariant typeof list ) {
    for  ( integer index = 0; index < length(myVariant); index++) { // iterate through the list
        variant element = myVariant[index]; // get list element by the index
        string type = getType(element); // get the type of the element
        printLog(info, "List element " + index + ": " + element + " " + type);
        if (element typeof integer) { // test the type of the element
            sum += cast(element, integer); // cast to integer and add to the sum
        }
    }
} else if ( myVariant typeof map ) {
    variant keys = getKeys(myVariant); // returns the keys as a list
    for ( integer i = 0; i < length(keys); i++) { // iterate through the list of keys
        variant key = keys[i]; // get the key by the index
        variant value = myVariant[key]; // get the value by the key
        string type = getType(value); // get the type of the value
        printLog(info, "Map entry " + key + " = " + value + " " + type);
        if (value typeof integer) { // test the type of the value
            sum += cast(value, integer); // cast to integer and add to the sum
        }
    }
}
printLog(info, "Sum: " + sum);

The assignments are similar to those valid for a list or a map:

  • variant varMap = {}; Assigns an empty map to varMap.

  • varMap["abc"] = 7; If varMap contains a map, puts the value 7 into varMapunder the key "abc". Throws an exception otherwise.

  • variant varList = []; Assigns an empty list to varList.

  • var2 = var1; Assigns a copy of var1 to var2.

  • varMap = { "name" -> "John Doe", "weight" -> 75.3, "valid" -> true }; Assigns a JSON-like map containing three key-value pairs to varMap. Note that the values are of mixed types: string, number and boolean, respectively.

  • varMap = null; Discards the previous value of varMap.

record

Record is a container that can contain different primitive data types.

The structure of record is based on metadata. Any metadata item represents a data type.

Declaration of a record looks like this: identifier;

Metadata names must be unique in a graph. Different metadata must have different names.

Record does not have a default value.

It can be indexed by both integer numbers and strings (field names). If indexed by numbers, fields are indexed starting from 0.

Functions with arguments of type variant can be passed any value. However, they may throw runtime exceptions if the value is not valid for the function. For example, "" can be passed any value as the first argument, but it will throw an exception unless the value really is a list.

The type supports only a few basic operations (, , etc.). In order to perform type-specific operations, the values must be explicitly type-cast to a more specific type. See and and functions.

varList[5] = "abc" If varList contains a list with at least 6 elements, sets the list element at index 5 to "abc". Unlike with List data type, variantis not expanded automatically, so if varList contains fewer than 6 elements, the assignment fails. Use to expand the list. If varList is actually a map, puts "abc" into the map under the key 5. Otherwise, throws an exception.

For more detailed information about possible expressions and records usage, see .

Data Types in Metadata
append
Accessing Data Records and Fields
cast
getType
append(variant list, variant element)
toString
== and != comparison
typeof operator