Jump to Content
Home
Guides
v1.2.4-Panorama
v1.20
Log In
Guides
Log In
LONDON
Home
Guides
Search
3, 2, 1... IGNITION
What is Data Integration?
What is Data One?
Data Integration vs Application Integration
... AND LIFT-OFF!
The legend
What's next?
The glossary
INSTALL BASE PRODUCT
Introduction to the installation process
Planning initial installation and master configuration
System requirements - NEW! 🚀
Domain Master Configuration Reference - NEW! 🚀
Performing initial installation and master configuration
Creating database objects for PostgreSQL
Creating database objects for Oracle
Executing the installer
Configuring load balancer rules for Data One WUI access on CEMAN
Configuring firewall rules
Running the system for the first time
Basic system control: starting, stopping and checking status
Launching Data One WUI
Wiring your STENG cluster together
Install Data shaper
Introduction to Data Shaper installation process
Planning Data Shaper installation
Data Shaper System Requirements
Data Shaper Domain Master Configuration reference
Performing Data Shaper initial installation and master configuration
Creating database objects for PostgreSQL
Creating database objects for Oracle
Executing Data Shaper installer
Configuring additional firewall rules for Data Shaper
DATA MOVER
Getting started
Login
Navigation tips
MANAGE
STENG, Clusters and Servers
Adding a cluster and a STENG
Adding a server to the STENG - NEW! 🚀
Stopping all servers in one go - NEW! 🚀
Deleting a STENG
ICAP Engines
Configuring an ICAP Engine
DMZ Gateways
DMZ Clusters
Company Profile
Actors - NEW! 🚀
Creating an Actor
Configuring an Actor
Users Tab
Groups Tab
VFS Tab
File Resource Tab
Connection Contracts Tab
Client Connection Tab - NEW! 🚀
Client Connection: FTP
Client Connection: FTPS
Client Connection: SFTP
Client Connection: HTTP
Client Connection: HTTPS
Client Connection: PESIT
Client Connection: SMB v3 or later versions
Client Connection: POP3 IMAP
Client Connection: SMTP
Client Connection: PR4/PR4S
Client Connection: PR5
Client Connection: PR5S
Client Connection: HDFS
Client Connection: HDFSS
Client Connection: Amazon S3 - NEW! 🚀
Client Connection: Google Cloud Storage
Client Connection: Azure Blob Storage
Client Connection: IBM Sterling Connect:Direct
Actor Lineage - NEW! 🚀
Aggregation of flows by protocol - NEW! 🚀
Lineage with connection contracts - NEW! 🚀
Lineage with input, mediation and output contracts - NEW! 🚀
Lineage with any contract type - NEW! 🚀
Virtual File Systems (VFS) - NEW! 🚀
Creating a VFS - NEW! 🚀
Configuring a VFS
Adding Virtual Paths
Modifying and Deleting a VFS
File Resources
Managing File Resources
Creating File Resources
How to use File Resources
Storage Classes - NEW! 🚀
Storage Class: File System - NEW! 🚀
Storage Class: SMB v3 or later versions - NEW! 🚀
Storage Class: Amazon S3 - NEW! 🚀
Storage Class: Google Cloud Storage - NEW! 🚀
Storage Class: Azure Blob Storage - NEW! 🚀
Retention Classes
Users & Groups
Setting the password policy
Creating Internal Users - NEW! 🚀
Creating Internal Groups
Creating External Users
Creating External Groups
Credentials
Creating a Credential
File Event Listener
Configuring File Event Listeners
Setting the File Event Listener Engine
Defining a contract for the File Event Listener
Setting events to be monitored
RegEx Rules
Monitoring File Event Listeners
Log Notifiers - NEW! 🚀
FEL message codes
Notification Channels
Configuring the default Email Notification Channel
Configuring a new Email Notification Channel
Trusting Certificates
Managing Templates
Data Watcher Macros
Contract Macros
ICAP Macros
Central Log Macros
Email Templates
Editing default templates
Loading a new template
Dataflow attributes - NEW 🚀
Advanced Settings
DESIGN
Contracts
Creating a Contract
Managing Contracts - NEW! 🚀
File Routing - NEW! 🚀
Main page
Navigation bar
The Rules tab
The Categories tab
The Output tab
How to create a rule - NEW! 🚀
Add metadata - NEW! 🚀
Select ACTIONS
Select OUTPUTS
Policy for the selection of metadata rules
Configuration of the environment in Data One
Set up Storage Classes
Set up Retention Classes
Configure the Actor
Set up File Resources
Associate the Routing Rule to a Contract
Example
Workflows
Managing Workflows
Creating Workflow Templates
Trigger types - NEW! 🚀
Designing Workflow Templates
Modeling Elements
Service Tasks
Configuring Service Tasks
Configuring Triggerable Service Tasks - NEW! 🚀
Handling Process Variable
Spazio Selectors and Filebox Metadata management
Error Management
Workflows and contracts interaction
Editing Workflow Templates
Workflows hacks you may need
System Workflow Templates
Input templates
Mediation templates
Output templates
Workflow Error Templates
Dataflow Modeler
DataFlow Instance Context (DFIC) 🚀
MONITOR
Job
Details about Jobs - NEW! 🚀
Job Manager
Job Queues
Managing Job Queues
Logs and Audit - NEW! 🚀
Logs - NEW! 🚀
Troubleshooting error analysis in Logs
Audit - NEW! 🚀
Export audit logs
List of Audit entity types
Audit message codes
File Transfers
Ongoing
Finished
Reports
File Transfers Rules
Configuring Rules
Defining an ICAP rule
Searching files in all VFS
Searching files by Actor
Cut-off Board
Cut-off Calendars
Dataflow Inquiry
Data Shaper
Monitoring
Execution History
Sandboxes
SECURITY
Identity and Access Management
Key Store - NEW! 🚀
Creating a Key - NEW! 🚀
Creating a Certificate - NEW! 🚀
Importing a Key or a Certificate
Creating a Symmetric key
Examples
Trust Store - NEW! 🚀
Importing Keys - NEW! 🚀
Importing Certificates
Untrusted Cache - NEW! 🚀
Trusting an element
When do I use the Keys tab?
When do I use the Certificates tab?
PGP Key Store / PGP Trust Store
Configuring the PGP Key Store
Importing keys into the PGP Trust Store
DATA SHAPER
Getting started
What is the Data Shaper Designer
What is the Data Shaper Server
What is the Data Shaper Cluster
How does the Data Shaper Designer work
Designer Views and Graphs
Data Shaper Graphs
Designer Projects and Sandboxes
Data Shaper Designer Reference
How do the Data Shaper Server and Cluster work
Data Shaper Server and Cluster
Data Shaper Server Reference
VFS Graph Components
DataOneFileDescriptor (DOFD) metadata
Passing parameters from Data One Contract to Data Shaper graph
Inheriting Data One context attributes in Data Shaper graph
DATA WATCHER
Getting started
Manage
Data Watcher users - NEW! 🚀
Cut-off Calendars
Attributes - NEW 🚀
Saved Queries
MongoDB
Back up the Data Watcher MongoDB
Restore the Data Watcher MongoDB
Clean up the Data Watcher MongoDB
Design
Modeling Dataflows - NEW! 🚀
Dataflow Model lifecycle - NEW! 🚀
Creating a dataflow model from scratch
Node Properties panel
Setting Matching Criteria
Setting Cut-offs - NEW! 🚀
Setting Notifications - NEW! 🚀
Monitor - NEW! 🚀
Cut-off Board - NEW! 🚀
Dataflow Inquiry - NEW! 🚀
Security
Authorizing Users or Groups to access models
Troubleshooting
DWAGENT 2.0.0
Getting started
Preparing to install
Installing
After the Installation
Configuring
Configuring DWA Communicator for Artemis
Configuring DWA Emitters
Configuring DWA Proxies
System Administration
Troubleshooting
FILE MANAGER
Getting started
Logging into File Manager
Managing the File Manager - NEW 🚀
The list of results
Creating new folders
Uploading files
Downloading files - NEW! 🚀
Searching for files and folders
Deleting files - NEW! 🚀
Bulk actions - NEW! 🚀
File Manager and VFS
Customizing File Manager externals
The configuration-wui.json file - NEW! 🚀
How to customize the Login window and the logo
How to customize the footer
How to configure the Upload with Metadata option
How to customize bulk actions - NEW! 🚀
API
HTTP MFT Rest API
Job Manager APIs - NEW 🚀
SFTP Server sessions APIs - NEW 🚀
DATA SHAPER DESIGNER
Configuration
Runtime configuration
Logging
Master Password
User Classpath
Data Shaper Server Integration
Execution monitoring
Java configuration
Engine configuration
Refresh Operation
Designer User Interface
Graph Editor with Palette of Components
Project Explorer Pane
Outline Pane
Tabs Pane
Execution Tab
Keyboard Shortcuts
Projects
Creating Data Shaper projects
Converting Data Shaper projects
Structure of Data Shaper projects
Versioning of server project content
Working with Data Shaper Server Projects
Project configuration
Graphs
Creating an empty graph
Creating a simple graph
Placing Components
Placing Components from Palette
Connecting Components with Edges
Execution
Successful Graph Execution
Run configuration
Connecting to a running job
Graph states
Common dialogs
URL file dialog
Edit Value dialog
Open Type dialog
Import
Import Data Shaper projects
Import from Data Shaper server sandbox
Import graphs
Import metadata
Export
Export graphs to HTML
Export to Data Shaper Server sandbox
Export image
Graph tracking
Changing record count font size
Search functionality
Working with Data Shaper server
Data Shaper server project basic principles
Connecting via HTTP
Connecting via HTTPS
Connecting via Proxy Server
Graph components
Adding components
Finding components
Edit component dialog box
Enable/disable component
Passing data through disabled component
Common properties of components
Specific attribute types
Metadata templates
Edges
Connecting Components with Edges
Types of Edges
Assigning Metadata to Edges
Colors of Edges
Debugging Edges
Edge Memory Allocation
Metadata
Records and Fields
Record Types
Data Types in Metadata
Data Formats
Locale and Locale Sensitivity
Time Zone
Autofilling Functions
Metadata Types
Internal Metadata
External (Shared) Metadata
SQL Query Metadata
Reading Metadata from Special Sources
Auto-propagated Metadata
Sources of Auto-Propagated Metadata
Explicitly Propagated Metadata
Priorities of Metadata
Propagation of SQL Query Metadata
Creating Metadata
Extracting Metadata from a Flat File
Extracting Metadata from an XLS(X) File
Extracting Metadata from a Database
Extracting Metadata from a DBase File
Extracting Metadata from Salesforce
SQL Query Metadata
User Defined Metadata
Merging Existing Metadata
Creating Database Table from Metadata and Database Connection
Metadata Editor
Opening Metadata Editor
Basics of Metadata Editor
Record Pane
Field Name vs. Label vs. Description
Details Pane
Changing and Defining Delimiters
Editing Metadata in the Source Code
Multi-value Fields
Lists and Maps Support in Components
Joining on multivalue fields (Comparison Rules)
Connections
Database Connections
Internal Database Connections
External (Shared) Database Connections
Database Connections Properties
Encryption of Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Snowflake Connection
Hive Connection
Troubleshooting
JMS Connections
QuickBase Connections
Hadoop Connections
Kafka Connections
OAuth2 Connections
MongoDB Connections
Salesforce Connections
Lookup Tables
Lookup Tables in Cluster Environment
Internal Lookup Tables
External (Shared) Lookup Tables
Types of Lookup Tables
Sequences
Persistent Sequences
Non Persistent Sequences
Internal Sequences
External (Shared) Sequences
Editing a Sequence
Sequences in Cluster Environment
Parameters
Internal Parameters
External (Shared) Parameters
Graph Parameter Editor
Secure Graph Parameters
Parameters with CTL2 Expressions (Dynamic Parameters)
Environment Variables
Canonicalizing File Paths
Using Parameters
Internal/External Graph Elements
Dictionary
Creating a Dictionary
Using a Dictionary in Graphs
Execution Properties
Notes in Graphs
Placing Notes into Graph
Resizing Notes
Editing Notes
Formatted Text
Links from Notes
Folding Notes
Notes Properties
Transformations
Defining Transformations
Transform Editor
Common Java Interfaces
Data Partitioning (Parallel Running)
Data Partitioning in Cluster
High Availability
Scalability
Graph Allocation Examples
Example of Distributed Execution
Remote Edges
Readers
Common Properties of Readers
ComplexDataReader
DatabaseReader
DataGenerator
DataOneVFSReader
EDIFACTReader
FlatFileReader
JSONExtract
JSONReader
LDAPReader
MultiLevelReader
SpreadsheetDataReader
UniversalDataReader
X12Reader
XMLExtract
XMLReader
XMLXPathReader
Writers
Common Properties of Writers
DatabaseWriter
DataOneVFSWriter
EDIFACTWriter
FlatFileWriter
JSONWriter
LDAPWriter
SpreadsheetDataWriter
Trash
UniversalDataWriter
X12Writer
XMLWriter
Transformers
Common Properties of Transformers
Aggregate
Concatenate
DataIntersection
DataSampler
Dedup
Denormalizer
ExtSort
FastSort
Filter
Merge
MetaPivot
Normalizer
Partition
Pivot
Map
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
Joiners
Common Properties of Joiners
Combine
CrossJoin
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
Others
Common Properties of Others
CheckForeignKey
DBExecute
HTTPConnector
LookupTableReaderWriter
WebServiceClient
CTL2 - Data Shaper Transformation Language
Language Reference
Program Structure
Comments
Import
Data Types in CTL2
Literals
Variables
Dictionary in CTL2
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Regular Expressions
CTL Debugging
Debug Perspective
Importing and Exporting Breakpoints
Inspecting Variables and Expressions
Examples
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Mapping Functions
Container Functions
Record Functions (Dynamic Field Access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Data Service HTTP Library Functions
Custom CTL Functions
CTL2 Appendix - List of National-specific Characters
Tutorial
Creating a Transformation Graph
Filtering the records
Sorting the Records
Processing Speed-up with Parallelization
Debugging the Java Transformation
DATA SHAPER SERVER
Introduction
Administration
Monitoring
Using Graphs
Job Queue
Viewing Job Runs - Execution History
Job Inspector
Cluster
Sandboxes in Cluster
Troubleshooting
CLI and Configuration files
jobman.sh reference
dataone.sh reference
doim.sh reference
dataonedbmgr.sh reference - NEW! 🚀
dataonedb.cfg reference - NEW 🚀
HOW TO...
... configure a Cron Expression
... create a Mediation Contract
... create an Output Contract
... configure an Application
... customize a header
... run searches in Data Watcher - NEW! 🚀
... use Data Shaper graphs in Data One contracts
... modify DMCFG and deploy it
... tune Data One data retention
... use different DNS names - NEW! 🚀
RELEASE NOTES
Data One 1.20.9
Data One 1.20.8
Data One 1.20.7
Data One 1.20.7.1
Data One 1.20.6
Data One 1.20.5
Data One 1.20.4
Data One 1.20.3
Data One 1.20.2
Data One 1.20.1
Data One 1.20.0
Suggest
