Primeur Online Docs
Data Shaper
Data Shaper
  • 🚀GETTING STARTED
    • What is Primeur Data Shaper
      • What is the Data Shaper Designer
      • What is the Data Shaper Server
      • What is the Data Shaper Cluster
    • How does the Data Shaper Designer work
      • Designer Views and Graphs
      • Data Shaper Graphs
      • Designer Projects and Sandboxes
      • Data Shaper Designer Reference
    • How do the Data Shaper Server and Cluster work
      • Data Shaper Server and Cluster
      • Data Shaper Server Reference
    • VFS Graph Components
      • DataOneFileDescriptor (DOFD) metadata
      • Passing parameters from Data One Contract to Data Shaper graph
      • Inheriting Data One context attributes in Data Shaper graph
  • DATA SHAPER DESIGNER
    • Configuration
      • Runtime configuration
        • Logging
        • Master Password
        • User Classpath
      • Data Shaper Server Integration
      • Execution monitoring
      • Java configuration
      • Engine configuration
      • Refresh Operation
    • Designer User Interface
      • Graph Editor with Palette of Components
      • Project Explorer Pane
      • Outline Pane
      • Tabs Pane
      • Execution Tab
      • Keyboard Shortcuts
    • Projects
      • Creating Data Shaper projects
      • Converting Data Shaper projects
      • Structure of Data Shaper projects
      • Versioning of server project content
      • Working with Data Shaper Server Projects
      • Project configuration
    • Graphs
      • Creating an empty graph
      • Creating a simple graph
        • Placing Components
        • Placing Components from Palette
        • Connecting Components with Edges
    • Execution
      • Successful Graph Execution
      • Run configuration
      • Connecting to a running job
      • Graph states
    • Common dialogs
      • URL file dialog
      • Edit Value dialog
      • Open Type dialog
    • Import
      • Import Data Shaper projects
      • Import from Data Shaper server sandbox
      • Import graphs
      • Import metadata
    • Export
      • Export graphs to HTML
      • Export to Data Shaper Server sandbox
      • Export image
      • Export Project as Library
    • Graph tracking
      • Changing record count font size
    • Search functionality
    • Working with Data Shaper server
      • Data Shaper server project basic principles
      • Connecting via HTTP
      • Connecting via HTTPS
      • Connecting via Proxy Server
    • Graph components
      • Adding components
      • Finding components
      • Edit component dialog box
      • Enable/disable component
      • Passing data through disabled component
      • Common properties of components
      • Specific attribute types
      • Metadata templates
    • Edges
      • Connecting Components with Edges
      • Types of Edges
      • Assigning Metadata to Edges
      • Colors of Edges
      • Debugging Edges
      • Edge Memory Allocation
    • Metadata
      • Records and Fields
        • Record Types
        • Data Types in Metadata
        • Data Formats
        • Locale and Locale Sensitivity
        • Time Zone
        • Autofilling Functions
      • Metadata Types
        • Internal Metadata
        • External (Shared) Metadata
        • SQL Query Metadata
        • Reading Metadata from Special Sources
      • Auto-propagated Metadata
        • Sources of Auto-Propagated Metadata
        • Explicitly Propagated Metadata
        • Priorities of Metadata
        • Propagation of SQL Query Metadata
      • Creating Metadata
        • Extracting Metadata from a Flat File
        • Extracting Metadata from an XLS(X) File
        • Extracting Metadata from a Database
        • Extracting Metadata from a DBase File
        • Extracting Metadata from Salesforce
        • SQL Query Metadata
        • User Defined Metadata
      • Merging Existing Metadata
      • Creating Database Table from Metadata and Database Connection
      • Metadata Editor
        • Opening Metadata Editor
        • Basics of Metadata Editor
        • Record Pane
        • Field Name vs. Label vs. Description
        • Details Pane
      • Changing and Defining Delimiters
      • Editing Metadata in the Source Code
      • Multi-value Fields
        • Lists and Maps Support in Components
        • Joining on multivalue fields (Comparison Rules)
    • Connections
      • Database Connections
        • Internal Database Connections
        • External (Shared) Database Connections
        • Database Connections Properties
        • Encryption of Access Password
        • Browsing Database and Extracting Metadata from Database Tables
        • Windows Authentication on Microsoft SQL Server
        • Snowflake Connection
        • Hive Connection
        • Troubleshooting
      • JMS Connections
      • QuickBase Connections
      • Hadoop Connections
      • Kafka Connections
      • OAuth2 Connections
      • MongoDB Connections
      • Salesforce Connections
    • Lookup Tables
      • Lookup Tables in Cluster Environment
      • Internal Lookup Tables
      • External (Shared) Lookup Tables
      • Types of Lookup Tables
    • Sequences
      • Persistent Sequences
      • Non Persistent Sequences
      • Internal Sequences
      • External (Shared) Sequences
      • Editing a Sequence
      • Sequences in Cluster Environment
    • Parameters
      • Internal Parameters
      • External (Shared) Parameters
      • Secure Graph Parameters
      • Graph Parameter Editor
      • Secure Graph Parameters
      • Parameters with CTL2 Expressions (Dynamic Parameters)
      • Environment Variables
      • Canonicalizing File Paths
      • Using Parameters
    • Internal/External Graph Elements
    • Dictionary
      • Creating a Dictionary
      • Using a Dictionary in Graphs
    • Execution Properties
    • Notes in Graphs
      • Placing Notes into Graph
      • Resizing Notes
      • Editing Notes
      • Formatted Text
      • Links from Notes
      • Folding Notes
      • Notes Properties
    • Transformations
      • Defining Transformations
      • Transform Editor
      • Common Java Interfaces
    • Data Partitioning (Parallel Running)
    • Data Partitioning in Cluster
      • High Availability
      • Scalability
      • Graph Allocation Examples
      • Example of Distributed Execution
      • Remote Edges
    • Readers
      • Common Properties of Readers
      • ComplexDataReader
      • DatabaseReader
      • DataGenerator
      • DataOneVFSReader
      • EDIFACTReader
      • FlatFileReader
      • JSONExtract
      • JSONReader
      • LDAPReader
      • MultiLevelReader
      • SpreadsheetDataReader
      • UniversalDataReader
      • X12Reader
      • XMLExtract
      • XMLReader
      • XMLXPathReader
    • Writers
      • Common Properties of Writers
      • DatabaseWriter
      • DataOneVFSWriter
      • EDIFACTWriter
      • FlatFileWriter
      • JSONWriter
      • LDAPWriter
      • SpreadsheetDataWriter
      • HIDDEN StructuredDataWriter
      • HIDDEN TableauWriter
      • Trash
      • UniversalDataWriter
      • X12Writer
      • XMLWriter
    • Transformers
      • Common Properties of Transformers
      • Aggregate
      • Concatenate
      • DataIntersection
      • DataSampler
      • Dedup
      • Denormalizer
      • ExtSort
      • FastSort
      • Filter
      • Map
      • Merge
      • MetaPivot
      • Normalizer
      • Partition
      • Pivot
      • Rollup
      • SimpleCopy
      • SimpleGather
      • SortWithinGroups
      • XSLTransformer
    • Joiners
      • Common Properties of Joiners
      • Combine
      • CrossJoin
      • DBJoin
      • ExtHashJoin
      • ExtMergeJoin
      • LookupJoin
      • RelationalJoin
    • Others
      • Common Properties of Others
      • CheckForeignKey
      • DBExecute
      • HTTPConnector
      • LookupTableReaderWriter
      • WebServiceClient
    • CTL2 - Data Shaper Transformation Language
    • Language Reference
      • Program Structure
      • Comments
      • Import
      • Data Types in CTL2
      • Literals
      • Variables
      • Dictionary in CTL2
      • Operators
      • Simple Statement and Block of Statements
      • Control Statements
      • Error Handling
      • Functions
      • Conditional Fail Expression
      • Accessing Data Records and Fields
      • Mapping
      • Parameters
      • Regular Expressions
    • CTL Debugging
      • Debug Perspective
      • Importing and Exporting Breakpoints
      • Inspecting Variables and Expressions
      • Examples
    • Functions Reference
      • Conversion Functions
      • Date Functions
      • Mathematical Functions
      • String Functions
      • Mapping Functions
      • Container Functions
      • Record Functions (Dynamic Field Access)
      • Miscellaneous Functions
      • Lookup Table Functions
      • Sequence Functions
      • Data Service HTTP Library Functions
      • Custom CTL Functions
      • CTL2 Appendix - List of National-specific Characters
      • HIDDEN Subgraph Functions
    • Tutorial
      • Creating a Transformation Graph
      • Filtering the records
      • Sorting the Records
      • Processing Speed-up with Parallelization
      • Debugging the Java Transformation
  • DATA SHAPER SERVER
    • Introduction
    • Administration
      • Monitoring
    • Using Graphs
      • Job Queue
      • Execution History
      • Job Inspector
    • Cluster
      • Sandboxes in Cluster
      • Troubleshooting
  • Install Data Shaper
    • Install Data Shaper
      • Introduction to Data Shaper installation process
      • Planning Data Shaper installation
      • Data Shaper System Requirements
      • Data Shaper Domain Master Configuration reference
      • Performing Data Shaper initial installation and master configuration
        • Creating database objects for PostgreSQL
        • Creating database objects for Oracle
        • Executing Data Shaper installer
        • Configuring additional firewall rules for Data Shaper
Powered by GitBook
On this page
  • Short Description
  • Ports
  • Metadata
  • WebServiceClient Attributes
  • Details
  • Authentication
  • Special Characters in Request Body
  • Examples
  • Getting Echo from Data Shaper Server
  • See also
  1. DATA SHAPER DESIGNER
  2. Others

WebServiceClient

Short Description

WebServiceClient calls a web-service. It sends incoming data record to a web-service and passes the response to the output ports if they are connected.

WebServiceClient supports document/literal styles only.

WebServiceClient supports only SOAP (version 1.1 and 1.2) messaging protocol with document style binding and literal use (document/literal binding).

COMPONENT
SAME INPUT METADATA
SORTED INPUTS
INPUTS
OUTPUTS
EACH TO ALL OUTPUTS
JAVA
CTL
AUTO-PROPAGATED METADATA

WebServiceClient

-

x

0-1

0-n

x

x

x

x

Ports

PORT TYPE
NUMBER
REQUIRED
DESCRIPTION
METADATA

Input

0

x

For request

Any1(In0)

Output

0-N

x:pa

For response mapped to these ports

Any2(Out#)

[1] Response does not need to be sent to output if output ports are not connected.

Metadata

WebServiceClient does not propagate metadata.

WebServiceClient has no metadata template.

WebServiceClient Attributes

ATTRIBUTE

REQ

DESCRIPTION

POSSIBLE VALUES

Basic

WSDL URL

yes

A URL of the WSD server to which the component will connect. Connecting via a proxy server is available, too, e.g.: http:(proxy://proxyHost:proxyPort)//

Operation name

yes

The name of the operation to be performed. See Details below.

Request Body structure

yes

A structure of the request that is received from the input port or written directly to the graph. For more information about request generation, see Details below.

Request Header structure

An optional attribute to Request Body structure. If not specified, automatic generation is disabled. For more information about request generation, see Details below.

Response mapping

Fault mapping

Namespace bindings

A set of name-value assignments defining custom namespaces. The namespace binding attribute is used only for Response Mapping and Fault Mapping.

e.g. weather = http://ws.cdyne.com/WeatherWS/

Use nested nodes

When true, all elements with the same name are mapped, no matter their depth in the tree. See example in Details below.

true (default) | false

Security

Username

k:p

A username to be used when connecting to the server.

Password

k:p

A password to be used when connecting to the server.

Auth Domain

k:p

An authentication domain. If not set, the NTLM authentication scheme will be disabled. Does not affect Digest and Basic authentication methods.

Auth Realm

k:p

An authentication realm to which the specified credentials apply. If left empty, the credentials will be used for any realm. Does not affect NTLM authentication scheme.

Key store

k:p

Path to the key store which contains the key pair for client certificate authentication. Leave empty to use the JVM default key store.

Key store password

k:p

The password for the Key store.

Key alias

k:p

Selects a key from the Key store. If not set, the first key will be used.

Key password

k:p

The password for the selected client key. If not set, Key store password is used.

Trust store

k:p

Path to the trust store which contains certificates of trusted servers and certification authorities. Leave empty to use the JVM default trust store.

Trust store password

k:p

The password for the Trust store.

Disable SSL Certificate Validation

If true, the component ignores certificate validation problems for SSL connection.

true | false (default)

Advanced

Request HTTP Headers

HTTP Header(s) and their values to be added to HTTP request headers.

Response HTTP Headers

HTTP header(s), that will be read from the HTTP response. You can map the headers in Response mapping to output metadata fields.

Timeout (ms)

Override Server URL

Specifies a URL that should be used for the requests instead of the one specified in WSDL definition.

Override Server URL from field

Specifies a field containing a URL that should be used for the requests instead of the one specified in WSDL definition.

[1] See the Authentication section below for more details.

[2] Available since release 5.7.

Details

After sending your request to the server, WebServiceClient waits up to 10 minutes for a response. If there is none, the component fails on error.

Operation name opens a dialog, depicted in the figure below, in which you can select a WS operation by double clicking on one of them. Operations not supporting the document style of the input message are displayed with a red error icon.

Request Body structure and Request Header structure - open a dialog showing the request structure. The Generate button generates the request sample based on a schema defined for the chosen operation. The Customized generation…​ option in the button’s drop-down menu opens a dialog which helps to customize the generated request sample by allowing to select only suitable elements or to choose a subtype for an element.

Example 48. Use nested nodes example

Mapping

<?xml version="1.0" encoding="UTF-8"?>
<Mappings>
    <Mapping element="request">
        <Mapping element="message" outPort="0" />
    </Mapping>
</Mappings>

applied to

<?xml version="1.0" encoding="UTF-8"?>
<request>
    <message>msg1</message>
    <operation>
        <message>msg2</message>
    </operation>
</request>

produces:

  • msg1 and msg2 with Use nested nodes switched on (default behavior)

  • msg1 with Use nested nodes switched off. In order to extract msg2, you would need to create an explicit <Mapping> tag (one for every nested element).

Authentication

If authentication is required by the web service server, the Username, Password and, in the case of NTLM authentication, Auth Domain component properties need to be set.

There are currently three authentication schemes supported: NTLM, Digest and Basic. NTLM is the most secure while Basic is the least secure of these methods. The server advertises which authentication methods it supports and WebServiceClient automatically selects the most secure one.

Auth Realm can be used to restrict the specified credential only to a desired realm, in case Basic or Digest authentication schema is selected.

Note

Auth Domain is required by the NTLM authentication. If it is not set, only Digest and Basic authentication schemes will be enabled.

In case server requires NTLM authentication, but Auth Domain is left empty, you will get errors like these in graph execution log:

  • ERROR [Axis2 Task] - Credentials cannot be used for NTLM authentication: org.apache.commons.httpclient.UsernamePasswordCredentials

  • ERROR [WatchDog] - Node WEB_SERVICE_CLIENT0 finished with status: ERROR caused by: org.apache.axis2.AxisFault: Transport error: 401 Error: Unauthorized

Also note that it is not possible to specify the domain as part of the Username in form of domain\username as is sometimes customary. The domain name has to be specified separately in the Auth Domain component property.

Special Characters in Request Body

Special characters, e.g. < that appear in a request body should be typed as entities (>). Otherwise < would be interpreted as a beginning of a new xml element.

Examples

Getting Echo from Data Shaper Server

Data Shaper Server provides a webservice API to get status or perform operations. Send echo request to Data Shaper Server and get the response. The server listens on 127.0.0.1 on TCP port 38080.

Solution

Use the WSDL URL, Operation name, Request Body structure and Response mapping attributes.

In Operation name choose the echo operation.

In Request Body structure, use Generate in the upper left corner and then fill in the string to be echoed.

In Response mapping, map the out element to any output metadata field. The output port has to be attached, otherwise the output cannot be mapped.

See also

PreviousLookupTableReaderWriterNextCTL2 - Data Shaper Transformation Language

e.g.

A mapping of a successful response to output ports. The same mapping as in XMLExtract. For more information, see .

A mapping of a fault response to output ports. The same mapping as in XMLExtract. For more information, see .

Timeout for the request; by default in milliseconds, but other may be used.

If you switch to DEBUG, you can examine the full SOAP request and response in a log. This is useful for development and issue investigation purposes.

Set WSDL URL to .

log level
http://127.0.0.1:38080/clover/webservice?wsdl
HTTPConnector
Common properties of components
Specific attribute types
http://wsf.cdyne.com/WeatherWS/Weather.asmx?wsdl
XMLExtract Mapping Definition
time units
XMLExtract Mapping Definition