Primeur Online Docs
Data Shaper
Data Shaper
  • 🚀GETTING STARTED
    • What is Primeur Data Shaper
      • What is the Data Shaper Designer
      • What is the Data Shaper Server
      • What is the Data Shaper Cluster
    • How does the Data Shaper Designer work
      • Designer Views and Graphs
      • Data Shaper Graphs
      • Designer Projects and Sandboxes
      • Data Shaper Designer Reference
    • How do the Data Shaper Server and Cluster work
      • Data Shaper Server and Cluster
      • Data Shaper Server Reference
    • VFS Graph Components
      • DataOneFileDescriptor (DOFD) metadata
      • Passing parameters from Data One Contract to Data Shaper graph
      • Inheriting Data One context attributes in Data Shaper graph
  • DATA SHAPER DESIGNER
    • Configuration
      • Runtime configuration
        • Logging
        • Master Password
        • User Classpath
      • Data Shaper Server Integration
      • Execution monitoring
      • Java configuration
      • Engine configuration
      • Refresh Operation
    • Designer User Interface
      • Graph Editor with Palette of Components
      • Project Explorer Pane
      • Outline Pane
      • Tabs Pane
      • Execution Tab
      • Keyboard Shortcuts
    • Projects
      • Creating Data Shaper projects
      • Converting Data Shaper projects
      • Structure of Data Shaper projects
      • Versioning of server project content
      • Working with Data Shaper Server Projects
      • Project configuration
    • Graphs
      • Creating an empty graph
      • Creating a simple graph
        • Placing Components
        • Placing Components from Palette
        • Connecting Components with Edges
    • Execution
      • Successful Graph Execution
      • Run configuration
      • Connecting to a running job
      • Graph states
    • Common dialogs
      • URL file dialog
      • Edit Value dialog
      • Open Type dialog
    • Import
      • Import Data Shaper projects
      • Import from Data Shaper server sandbox
      • Import graphs
      • Import metadata
    • Export
      • Export graphs to HTML
      • Export to Data Shaper Server sandbox
      • Export image
      • Export Project as Library
    • Graph tracking
      • Changing record count font size
    • Search functionality
    • Working with Data Shaper server
      • Data Shaper server project basic principles
      • Connecting via HTTP
      • Connecting via HTTPS
      • Connecting via Proxy Server
    • Graph components
      • Adding components
      • Finding components
      • Edit component dialog box
      • Enable/disable component
      • Passing data through disabled component
      • Common properties of components
      • Specific attribute types
      • Metadata templates
    • Edges
      • Connecting Components with Edges
      • Types of Edges
      • Assigning Metadata to Edges
      • Colors of Edges
      • Debugging Edges
      • Edge Memory Allocation
    • Metadata
      • Records and Fields
        • Record Types
        • Data Types in Metadata
        • Data Formats
        • Locale and Locale Sensitivity
        • Time Zone
        • Autofilling Functions
      • Metadata Types
        • Internal Metadata
        • External (Shared) Metadata
        • SQL Query Metadata
        • Reading Metadata from Special Sources
      • Auto-propagated Metadata
        • Sources of Auto-Propagated Metadata
        • Explicitly Propagated Metadata
        • Priorities of Metadata
        • Propagation of SQL Query Metadata
      • Creating Metadata
        • Extracting Metadata from a Flat File
        • Extracting Metadata from an XLS(X) File
        • Extracting Metadata from a Database
        • Extracting Metadata from a DBase File
        • Extracting Metadata from Salesforce
        • SQL Query Metadata
        • User Defined Metadata
      • Merging Existing Metadata
      • Creating Database Table from Metadata and Database Connection
      • Metadata Editor
        • Opening Metadata Editor
        • Basics of Metadata Editor
        • Record Pane
        • Field Name vs. Label vs. Description
        • Details Pane
      • Changing and Defining Delimiters
      • Editing Metadata in the Source Code
      • Multi-value Fields
        • Lists and Maps Support in Components
        • Joining on multivalue fields (Comparison Rules)
    • Connections
      • Database Connections
        • Internal Database Connections
        • External (Shared) Database Connections
        • Database Connections Properties
        • Encryption of Access Password
        • Browsing Database and Extracting Metadata from Database Tables
        • Windows Authentication on Microsoft SQL Server
        • Snowflake Connection
        • Hive Connection
        • Troubleshooting
      • JMS Connections
      • QuickBase Connections
      • Hadoop Connections
      • Kafka Connections
      • OAuth2 Connections
      • MongoDB Connections
      • Salesforce Connections
    • Lookup Tables
      • Lookup Tables in Cluster Environment
      • Internal Lookup Tables
      • External (Shared) Lookup Tables
      • Types of Lookup Tables
    • Sequences
      • Persistent Sequences
      • Non Persistent Sequences
      • Internal Sequences
      • External (Shared) Sequences
      • Editing a Sequence
      • Sequences in Cluster Environment
    • Parameters
      • Internal Parameters
      • External (Shared) Parameters
      • Secure Graph Parameters
      • Graph Parameter Editor
      • Secure Graph Parameters
      • Parameters with CTL2 Expressions (Dynamic Parameters)
      • Environment Variables
      • Canonicalizing File Paths
      • Using Parameters
    • Internal/External Graph Elements
    • Dictionary
      • Creating a Dictionary
      • Using a Dictionary in Graphs
    • Execution Properties
    • Notes in Graphs
      • Placing Notes into Graph
      • Resizing Notes
      • Editing Notes
      • Formatted Text
      • Links from Notes
      • Folding Notes
      • Notes Properties
    • Transformations
      • Defining Transformations
      • Transform Editor
      • Common Java Interfaces
    • Data Partitioning (Parallel Running)
    • Data Partitioning in Cluster
      • High Availability
      • Scalability
      • Graph Allocation Examples
      • Example of Distributed Execution
      • Remote Edges
    • Readers
      • Common Properties of Readers
      • ComplexDataReader
      • DatabaseReader
      • DataGenerator
      • DataOneVFSReader
      • EDIFACTReader
      • FlatFileReader
      • JSONExtract
      • JSONReader
      • LDAPReader
      • MultiLevelReader
      • SpreadsheetDataReader
      • UniversalDataReader
      • X12Reader
      • XMLExtract
      • XMLReader
      • XMLXPathReader
    • Writers
      • Common Properties of Writers
      • DatabaseWriter
      • DataOneVFSWriter
      • EDIFACTWriter
      • FlatFileWriter
      • JSONWriter
      • LDAPWriter
      • SpreadsheetDataWriter
      • HIDDEN StructuredDataWriter
      • HIDDEN TableauWriter
      • Trash
      • UniversalDataWriter
      • X12Writer
      • XMLWriter
    • Transformers
      • Common Properties of Transformers
      • Aggregate
      • Concatenate
      • DataIntersection
      • DataSampler
      • Dedup
      • Denormalizer
      • ExtSort
      • FastSort
      • Filter
      • Map
      • Merge
      • MetaPivot
      • Normalizer
      • Partition
      • Pivot
      • Rollup
      • SimpleCopy
      • SimpleGather
      • SortWithinGroups
      • XSLTransformer
    • Joiners
      • Common Properties of Joiners
      • Combine
      • CrossJoin
      • DBJoin
      • ExtHashJoin
      • ExtMergeJoin
      • LookupJoin
      • RelationalJoin
    • Others
      • Common Properties of Others
      • CheckForeignKey
      • DBExecute
      • HTTPConnector
      • LookupTableReaderWriter
      • WebServiceClient
    • CTL2 - Data Shaper Transformation Language
    • Language Reference
      • Program Structure
      • Comments
      • Import
      • Data Types in CTL2
      • Literals
      • Variables
      • Dictionary in CTL2
      • Operators
      • Simple Statement and Block of Statements
      • Control Statements
      • Error Handling
      • Functions
      • Conditional Fail Expression
      • Accessing Data Records and Fields
      • Mapping
      • Parameters
      • Regular Expressions
    • CTL Debugging
      • Debug Perspective
      • Importing and Exporting Breakpoints
      • Inspecting Variables and Expressions
      • Examples
    • Functions Reference
      • Conversion Functions
      • Date Functions
      • Mathematical Functions
      • String Functions
      • Mapping Functions
      • Container Functions
      • Record Functions (Dynamic Field Access)
      • Miscellaneous Functions
      • Lookup Table Functions
      • Sequence Functions
      • Data Service HTTP Library Functions
      • Custom CTL Functions
      • CTL2 Appendix - List of National-specific Characters
      • HIDDEN Subgraph Functions
    • Tutorial
      • Creating a Transformation Graph
      • Filtering the records
      • Sorting the Records
      • Processing Speed-up with Parallelization
      • Debugging the Java Transformation
  • DATA SHAPER SERVER
    • Introduction
    • Administration
      • Monitoring
    • Using Graphs
      • Job Queue
      • Execution History
      • Job Inspector
    • Cluster
      • Sandboxes in Cluster
      • Troubleshooting
  • Install Data Shaper
    • Install Data Shaper
      • Introduction to Data Shaper installation process
      • Planning Data Shaper installation
      • Data Shaper System Requirements
      • Data Shaper Domain Master Configuration reference
      • Performing Data Shaper initial installation and master configuration
        • Creating database objects for PostgreSQL
        • Creating database objects for Oracle
        • Executing Data Shaper installer
        • Configuring additional firewall rules for Data Shaper
Powered by GitBook
On this page
  • Record Details
  • Basic
  • Advanced
  • Custom
  • Field Details
  • Basic
  • Advanced
  1. DATA SHAPER DESIGNER
  2. Metadata
  3. Metadata Editor

Details Pane

PreviousField Name vs. Label vs. DescriptionNextChanging and Defining Delimiters

The contents of the Details pane changes in accordance with the row selected in the Record pane.

  • If you select the first row, details about the whole record are displayed. See Record Details below.

  • If you select another row, details about the selected field are displayed. See Field Details below.

Record Details

When the Details pane presents information about the record as a whole, its properties are displayed . The record details are split up into basic, advanced and custom.

Basic

  • Name Name is the name of the record. The name can be seen above a selected edge or in the Outline. Only limited set of characters is allowed here: letters of English alphabet, numbers and underscore.

  • Label Contrary to the Name, the Label can contain diacritic and space characters. See .

  • Type One of the following three can be selected: delimited, fixed, mixed. For more information, see .

  • Record delimiter Record delimiter is a delimiter following the last field meaning the end of the record. If the delimiter in the last row of the Record pane in its Delimiter column is displayed grayish, it is this record delimiter. If it is black, it is another, non-default delimiter defined for the last field which follows it and precedes the record delimiter. For more detailed information, see .

  • Record Size Record size is the length of the record counted in number of characters. It can be changed there. The record size is displayed for fixed or mixed record type only.

  • Default Delimiter Default delimiter is a delimiter which, by default, follows each field of the record except the last one. This delimiter is displayed in each other row (except the last one) of the Record pane in its Delimiter column if it is grayish. If it is black, it is another, non-default delimiter defined for such a field which overrides the default one and is used instead. The Default delimiter is displayed for delimited or mixed records type only. For more detailed information, see .

  • Skip Source Rows Skip source rows defines the number of records that will be skipped for each input file. If an edge with this attribute is connected to a Reader, this value overrides the default value of the Number of skipped records per source attribute, which is 0. If the Number of skipped records per source attribute is not specified, this number of records are skipped from each input file. If the attribute in the Reader is set to any value, it overrides this property value. Remember that these two values are not summed.

  • Description The description stores user notes concerning the record. The description can contain several paragraphs.

Advanced

  • Quoted strings Fields containing a special character (comma, newline, or double quote) have to be enclosed in quotes. Only single/double quote is accepted as the quote character. If Quoted strings is true, special characters are not treated as delimiters and are:

    • removed - when reading the input by a Reader;

  • Quote Character Quote character specifies which kind of quotes will be used in Quoted strings. If a component has this attribute (e.g. ComplexDataReader, FlatFileReader, FlatFileWriter), its value is set according to the settings of Quote character in metadata on input/output port. However, the value in a component has a higher priority than the one in metadata - you can override it.

  • Null Value This property is set for the whole record. It is used to specify what values of fields should be processed as null. By default, empty field or empty string ("") are processed as null. You can set this property value to any string of characters that should be interpreted as null. All of the other string values remain unchanged. If you set this property to any non-empty string, empty string or empty field value will remain to be empty string (""). Multiple null values can be specified using \| delimiter. For example, if you would like to recognize both strings NULL and N/A as a null value, just use NULL\|N/A. It can be overridden by the value of Null value property of individual field.

  • Preview Attachment This is the file URL of the file attached to the metadata. It can be changed there or located using the Browse…​ button.

  • Preview Charset This is the charset of the file attached to the metadata. It can be changed there or by selecting from the combobox.

  • Preview Attachment Metadata Row This is the number of the row of the attached file where record field names are located.

  • Preview Attachment Sample Data Row This is the number of the row of the attached file from where field data types are guessed.

  • Key Fields The Key fields field contains all field names of fields marked using Print field value into log with token status button (from the record pane).

  • EOF as Delimiter If EOF as delimiter is true, the end of file is considered as a record delimiter. If EOF as delimiter is set up on a record level and on a field level, the record level has higher priority.

Custom

Custom properties can be defined by clicking the Plus sign button. For example, these properties can be the following:

  • charset This is the charset of the record. For example, when metadata are extracted from dBase files, these properties may be displayed.

  • dataOffset dataOffset is displayed for fixed or mixed record type only.

Field Details

When the Details pane presents information about a field, there are displayed its properties. Field details are basic and advanced.

Basic

  • Label Label is similar to the Name, but the arbitrary characters can be used.

  • Size This is the same size as in the Record pane.

  • Nullable This can be true or false. The default value is true. In such a case, the field value can be null. Otherwise, null values are prohibited and graph fails if null is met.

  • Description Description is user defined long text concerning the particular field. The field can be several paragraphs long.

Advanced

Advanced properties are the following:

  • Null Value This property can be set up to specify what values of fields should be processed as null. By default, empty field or empty string ("") are processed as null. You can set this property value to any string of characters that should be interpreted as null. All of the other string values remain unchanged. If you set this property to any non-empty string, empty string or empty field value will remain to be empty string (""). It overrides the value of Null Value property of the whole record. Multiple values can be used as a null value. See Null Value as a record property.

  • Trim If true, the leading and trailing white space characters are trimmed. It is performed on data in readers.

  • Shift This is the gap between the end of one field and the start of the next one, when the fields are part of fixed or mixed record and their sizes are set to some value.

  • EOF as Delimiter This can be set to true or false according to whether EOF character is used as delimiter. It can be useful when your file does not end with any other delimiter. If you did not set this property to true, run of the graph with such data file would fail (by default it is false). Displayed in delimited or mixed data records only. The EOF as delimiter can be set up on the record level. If the values differ, the value on the record level has higher priority.

written out - output fields will be enclosed in Quoted strings (see ).

If a component has this attribute (e.g. , , ), its value is set according to the settings of Quoted strings in metadata on input/output port. However, the true/false value in a component has a higher priority than the one in metadata - you can override it.

Locale This is the locale that is used for the whole record. This property can be useful for date formats or for decimal separator, for example. It can be overridden by the Locale specified for individual field. For detailed information, see .

Locale Sensitivity Applied for the whole record. It can be overridden by the Locale sensitivity specified for individual field (of string data type). For detailed information, see .

Time Zone Applied for the whole record. It can be overridden by the Time zone specified for individual field (of date data type). See for detailed information.

Name This is the same field name as in the Record pane. See .

Type This is the same data type as in the Record pane. For more detailed information, see .

Container Type Container type determines whether a field can store multiple values (of the same type). There are two options: list and map. Switching back to single makes it a common single-value field again. For more information, see .

Delimiter This is the non-default field delimiter as in the Record pane. If it is empty, the default delimiter is used instead. The delimiter is on the right side of the corresponding field. For more detailed information, see .

Default This is the default value of the field. It is used if you set the Autofilling property to default_value. For more detailed information, see .

Length Displayed for decimal data type only. For decimal data types you can optionally define its length. It is the maximum number of digits in this number. The default value is 12. For more detailed information, see .

Scale Displayed for decimal data type only. For decimal data types you can optionally define scale. It is the maximum number of digits following the decimal dot. The default value is 2. For more detailed information, see .

Format Format defines the parsing and/or the formatting of a boolean, date, decimal, integer, long, number, and string data field. For more information, see .

Locale This property can be useful for date formats or for decimal separator, for example. It overrides the Locale specified for the whole record. For detailed information, see .

Locale Sensitivity Displayed for string data type only. Is applied only if Locale is specified for the field or the whole record. It overrides the Locale sensitivity specified for the whole record. For detailed information, see .

Time Zone Displayed for date data type only. It overrides the Time zone specified for the whole record. For detailed information, see .

Autofilling If defined, field marked as autofilling is filled with a value by one of the functions listed in the section.

Field Name vs. Label vs. Description
Record Types
Changing and Defining Delimiters
Changing and Defining Delimiters
ComplexDataReader
FlatFileReader
FlatFileWriter
Locale
Locale Sensitivity
Time Zone
Field Name vs. Label vs. Description
Data Types in Metadata
Multivalue Fields
Changing and Defining Delimiters
Autofilling Functions
Data Types in Metadata
Data Types in Metadata
Data Formats
Locale
Locale Sensitivity
Time Zone
Autofilling Functions
FlatFileWriter Attributes