Jump to Content
Home
Guides
π¬ Release Notes
v1.20
Log In
Log In
v1.20
Home
Guides
π¬ Release Notes
Loadingβ¦
Search
3, 2, 1... IGNITION
What is Data Integration?
What is Data One?
Data Integration vs Application Integration
... AND LIFT-OFF!
The legend
What's next?
The glossary
INSTALL BASE PRODUCT
Introduction to the installation process
Planning initial installation and master configuration
System requirements
Domain Master Configuration Reference
Performing initial installation and master configuration
Creating database objects for PostgreSQL
Creating database objects for Oracle
Executing the installer
Configuring load balancer rules for Data One WUI access on CEMAN
Configuring firewall rules
Running the system for the first time
Basic system control: starting, stopping and checking status
Launching Data One WUI
Wiring your STENG cluster together
Install Data shaper
Introduction to Data Shaper installation process
Planning Data Shaper installation
Data Shaper System Requirements
Data Shaper Domain Master Configuration reference
Performing Data Shaper initial installation and master configuration
Creating database objects for PostgreSQL
Creating database objects for Oracle
Executing Data Shaper installer
Configuring additional firewall rules for Data Shaper
DATA MOVER
Getting started
Login
Navigation tips
MANAGE
STENG, Clusters and Servers
Adding a cluster and a STENG
Adding a server to the STENG
ICAP Engines
Configuring an ICAP Engine
DMZ Gateways
DMZ Clusters
Company Profile
Actors - NEW! π
Creating an Actor
Configuring an Actor
Users Tab
Groups Tab
VFS Tab
File Resource Tab
Connection Contracts Tab
Client Connection Tab - NEW! π
Client Connection: FTP
Client Connection: FTPS
Client Connection: SFTP
Client Connection: HTTP
Client Connection: HTTPS
Client Connection: PESIT
Client Connection: SMB v3 or later versions
Client Connection: POP3 IMAP
Client Connection: SMTP
Client Connection: PR4/PR4S
Client Connection: PR5
Client Connection: PR5S
Client Connection: HDFS
Client Connection: HDFSS
Client Connection: Amazon S3 - NEW! π
Client Connection: Google Cloud Storage
Client Connection: Azure Blob Storage
Client Connection: IBM Sterling Connect:Direct
Virtual File Systems (VFS) - NEW! π
Creating a VFS - NEW! π
Configuring a VFS
Adding Virtual Paths
Modifying and Deleting a VFS
File Resources
Managing File Resources
Creating File Resources
How to use File Resources
Storage Classes - NEW! π
Storage Class: File System - NEW! π
Storage Class: SMB v3 or later versions - NEW! π
Storage Class: Amazon S3 - NEW! π
Storage Class: Google Cloud Storage - NEW! π
Storage Class: Azure Blob Storage - NEW! π
Retention Classes
Users & Groups
Setting the password policy
Creating Internal Users
Creating Internal Groups
Creating External Users
Creating External Groups
Credentials
Creating a Credential
File Event Listener
Configuring File Event Listeners
Setting the File Event Listener Engine
Defining a contract for the File Event Listener
Setting events to be monitored
RegEx Rules
Monitoring File Event Listeners
Log Notifiers
FEL message codes
Notification Channels
Configuring the Email Notification Channel
Trusting Certificates
Managing Templates
Data Watcher Macros
Contract Macros
ICAP Macros
Central Log Macros
Email Templates
Editing default templates
Loading a new template
Dataflow Attributes
Advanced Settings
DESIGN
Contracts
Creating a Contract
Managing Contracts
Workflows
Managing Workflows
Creating Workflow Templates
Trigger types
Designing Workflow Templates
Modeling Elements
Service Tasks
Configuring Service Tasks
Configuring Triggerable Service Tasks
Handling Process Variable
Spazio Selectors and Filebox Metadata management
Error Management
Workflows and contracts interaction
Editing Workflow Templates
Workflows hacks you may need
System Workflow Templates
Input templates
Mediation templates
Output templates
Workflow Error Templates
Dataflow Modeler
DataFlow Instance Context (DFIC) π
MONITOR
Jobs
Details about Jobs
Job Manager
Job Queues
Managing Job Queues
Logs
What to do in case of error
File Transfers
Ongoing
Finished
Reports
File Transfers Rules
Configuring Rules
Defining an ICAP rule
Searching files in all VFS
Searching files by Actor
Cut-off Board
Cut-off Calendars
Dataflows
Dataflow Board
Dataflow Inquiry
Data Shaper
Monitoring
Execution History
Sandboxes
SECURITY
Identity and Access Management
Key Store
Managing the Key Store
Creating a Key
Creating a Certificate
Importing a Key or a Certificate
Creating a Symmetric key
Examples
Trust Store
Importing Keys
Importing Certificates
Untrusted Cache
Trusting an element
When do I use the Keys tab?
When do I use the Certificates tab?
PGP Key Store / PGP Trust Store
Configuring the PGP Key Store
Importing keys into the PGP Trust Store
DATA SHAPER
Getting started
What is the Data Shaper Designer
What is the Data Shaper Server
What is the Data Shaper Cluster
How does the Data Shaper Designer work
Designer Views and Graphs
Data Shaper Graphs
Designer Projects and Sandboxes
Data Shaper Designer Reference
How do the Data Shaper Server and Cluster work
Data Shaper Server and Cluster
Data Shaper Server Reference
VFS Graph Components
DataOneFileDescriptor (DOFD) metadata
Passing parameters from Data One Contract to Data Shaper graph
Inheriting Data One context attributes in Data Shaper graph
DATA WATCHER
Getting started
Manage
Data Watcher Users
Cut-off Calendars
Dataflow Attributes
Saved Queries
Back up the Data Watcher MongoDB
Restore the Data Watcher MongoDB
Clean up the Data Watcher MongoDB
Design
Modeling Dataflows
Setting Matching Criteria
Setting Cut-offs
Setting Notifications
Monitor - NEW! π
Dataflow Board
Cut-off Board
Dataflow Inquiry - NEW! π
Security
Authorizing Users or Groups to access models
Troubleshooting
DWAGENT 2.0.0
Getting started
Preparing to install
Installing
After the Installation
Configuring
Configuring DWA Communicator for Artemis
Configuring DWA Emitters
Configuring DWA Proxies
System Administration
Troubleshooting
FILE MANAGER
Getting started
Logging into File Manager
Managing the File Manager
The list of results
Creating new folders
Uploading files
Downloading files
Searching for files and folders
Deleting files
File Manager and VFS
Customizing File Manager externals
The configuration-wui.json file
How to customize the Login window and the logo
How to customize the footer
How to configure the Upload with Metadata option
API
HTTP MFT Rest API
DATA SHAPER DESIGNER
Configuration
Runtime configuration
Logging
Master Password
User Classpath
Data Shaper Server Integration
Execution monitoring
Java configuration
Engine configuration
Refresh Operation
Designer User Interface
Graph Editor with Palette of Components
Project Explorer Pane
Outline Pane
Tabs Pane
Execution Tab
Keyboard Shortcuts
Projects
Creating Data Shaper projects
Converting Data Shaper projects
Structure of Data Shaper projects
Versioning of server project content
Working with Data Shaper Server Projects
Project configuration
Graphs
Creating an empty graph
Creating a simple graph
Placing Components
Placing Components from Palette
Connecting Components with Edges
Execution
Successful Graph Execution
Run configuration
Connecting to a running job
Graph states
Common dialogs
URL file dialog
Edit Value dialog
Open Type dialog
Import
Import Data Shaper projects
Import from Data Shaper server sandbox
Import graphs
Import metadata
Export
Export graphs to HTML
Export to Data Shaper Server sandbox
Export image
Export Project as Library
Graph tracking
Changing record count font size
Search functionality
Working with Data Shaper server
Data Shaper server project basic principles
Connecting via HTTP
Connecting via HTTPS
Connecting via Proxy Server
Graph components
Adding components
Finding components
Edit component dialog box
Enable/disable component
Passing data through disabled component
Common properties of components
Specific attribute types
Metadata templates
Edges
Connecting Components with Edges
Types of Edges
Assigning Metadata to Edges
Colors of Edges
Debugging Edges
Edge Memory Allocation
Metadata
Records and Fields
Record Types
Data Types in Metadata
Data Formats
Locale and Locale Sensitivity
Time Zone
Autofilling Functions
Metadata Types
Auto-propagated Metadata
Creating Metadata
Merging Existing Metadata
Creating Database Table from Metadata and Database Connection
Metadata Editor
Changing and Defining Delimiters
Editing Metadata in the Source Code
Multi-value Fields
Connections
Database Connections
Internal Database Connections
External (Shared) Database Connections
Database Connections Properties
Encryption of Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Snowflake Connection
Hive Connection
Troubleshooting
JMS Connections
QuickBase Connections
Hadoop Connections
Kafka Connections
OAuth2 Connections
MongoDB Connections
Salesforce Connections
Lookup Tables
Lookup Tables in Cluster Environment
Internal Lookup Tables
External (Shared) Lookup Tables
Types of Lookup Tables
Sequences
Persistent Sequences
Non Persistent Sequences
Internal Sequences
External (Shared) Sequences
Editing a Sequence
Sequences in Cluster Environment
Parameters
Internal Parameters
External (Shared) Parameters
Graph Parameter Editor
Secure Graph Parameters
Parameters with CTL2 Expressions (Dynamic Parameters)
Environment Variables
Canonicalizing File Paths
Using Parameters
Internal/External Graph Elements
Dictionary
Creating a Dictionary
Using a Dictionary in Graphs
Execution Properties
Notes in Graphs
Placing Notes into Graph
Resizing Notes
Editing Notes
Formatted Text
Links from Notes
Folding Notes
Notes Properties
Transformations
Defining Transformations
Transform Editor
Common Java Interfaces
Data Partitioning (Parallel Running)
Data Partitioning in Cluster
High Availability
Scalability
Graph Allocation Examples
Example of Distributed Execution
Remote Edges
Readers
Common Properties of Readers
ComplexDataReader
DatabaseReader
DataGenerator
DataOneVFSReader
EDIFACTReader
FlatFileReader
JSONExtract
JSONReader
LDAPReader
MultiLevelReader
SpreadsheetDataReader
UniversalDataReader
X12Reader
XMLExtract
XMLReader
XMLXPathReader
Writers
Common Properties of Writers
DatabaseWriter
DataOneVFSWriter
EDIFACTWriter
FlatFileWriter
JSONWriter
LDAPWriter
SpreadsheetDataWriter
Trash
UniversalDataWriter
X12Writer
XMLWriter
Transformers
Common Properties of Transformers
Aggregate
Concatenate
DataIntersection
DataSampler
Dedup
Denormalizer
ExtSort
FastSort
Filter
Merge
MetaPivot
Normalizer
Partition
Pivot
Map
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
Joiners
Common Properties of Joiners
Combine
CrossJoin
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
Others
Common Properties of Others
CheckForeignKey
DBExecute
HTTPConnector
LookupTableReaderWriter
WebServiceClient
CTL2 - Data Shaper Transformation Language
Language Reference
Program Structure
Comments
Import
Data Types in CTL2
Literals
Variables
Dictionary in CTL2
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Regular Expressions
CTL Debugging
Debug Perspective
Importing and Exporting Breakpoints
Inspecting Variables and Expressions
Examples
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Mapping Functions
Container Functions
Record Functions (Dynamic Field Access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Data Service HTTP Library Functions
Custom CTL Functions
CTL2 Appendix - List of National-specific Characters
Tutorial
Creating a Transformation Graph
Filtering the records
Sorting the Records
Processing Speed-up with Parallelization
Debugging the Java Transformation
DATA SHAPER SERVER
Introduction
Administration
Monitoring
Using Graphs
Job Queue
Viewing Job Runs - Execution History
Job Inspector
Cluster
Sandboxes in Cluster
Troubleshooting
CLI and Configuration files
jobman.sh reference
dataone.sh reference
doim.sh reference
HOW TO...
... configure a Cron Expression
... create a Mediation Contract
... create an Output Contract
... configure an Application
... customize a header
... run searches in Data Watcher
... use Data Shaper graphs in Data One contracts
... modify DMCFG and deploy it
... tune Data One data retention
RELEASE NOTES
Data One 1.20.4
Data One 1.20.3
Data One 1.20.2
Data One 1.20.1
Data One 1.20.0
Records and Fields
Suggest Edits
Updated 3 months ago