HTTPConnector

Short Description

HTTPConnector sends HTTP requests to a web server and receives responses. The request is written in a file or in the graph itself or it is received through a single input port. The response can be sent to an output port, stored to a specified file or stored to a temporary file. The path to the file can then be sent to a specified output port.

Ports

Metadata

HTTPConnector does not propagate metadata.

HTTPConnector has metadata templates on its ports available.

You do not have to use metadata templates on input and output edges.

See general details on metadata templates.

Input

Output

HTTPConnector_Response

HTTPConnector_Error

HTTPConnector Attributes

[1] A URL must be specified by setting one of the URL or URL from field attributes or mapping it in the Input mapping.
[2] The response can be stored either in a file specified in Output file URL or in a temporary file (when Store response file URL to output field is set to true) - it is not possible to use both options.

Details

Input Mapping

Editing the Input mapping attribute opens the Transform Editor where you can decide which component attributes should be set using the input record.

The dialog provides you with all the power and features known from Transform Editor and CTL2 - Data Shaper Transformation Language.

All kinds of CTL functions are available to modify the input field value to be used.

Multipart entities

You can set up multipart entities in the transform editor. Input mapping now offers new fields derived from the value of the Multipart entities attribute. For example, field1;field2 as the value of multipart entities generates the following fields.

The generated fields can be used to control multipart entities.

If you deal with Multipart entities, you have to use the POST method.

Possible ways of configuration of multipart entities

List of input fields

Compatible with previous versions. The Multipart entities attribute contains a semicolon separated list of fields from the input record. Each field is a multipart entity. The name is same as the field name, the field value is used as a content.

Map content of multipart entity

Use input mapping to set a content of multipart. The multipart name will be same as the fieldname and the content will be specified by a mapping.

Map content and filename

The multipart content will be used by the mapping, but there will be an additional multipart header in the request using the filename as mapped.

Example 47. CTL Mapping and multipart entities

The CTL mapping

function integer transform() {
    $out.4.field1_EntityContent="My custom content";
    $out.4.field1_EntityFileNameAttribute="MyFilename";
    returnALL;
}

produces following multipart content.

­­CB5PZVJDq5RyTWoZqxvtjlbVM0CrMa3Mt
Content­Disposition: form­data; name="field1"; filename="MyFilename"
Content­Type: text/plain; charset=UTF­8
Content­Transfer­Encoding: 8bit

My custom content
­­CB5PZVJDq5RyTWoZqxvtjlbVM0CrMa3Mt

Use file as multipart entity

To use files as multipart entities, map only the *_File field. Do not map the _Content field.

$out.3.field3_EntitySourceFile = "${PROJECT}/workspace.prm";

This will upload the file workspace.prm as a multipart entity.

­­3xEKe3wUSOl2cRnjwh1UsPVnDOoL7D
Content­Disposition: form­data; name="field3"; filename="workspace.prm"
Content­Type: application/octet­stream
Content­Transfer­Encoding: binary

... [here is content of file]

­­3xEKe3wUSOl2cRnjwh1UsPVnDOoL7D­­

The file can be specified by a URL similar to the fileURL attribute in readers. But it cannot use the port reading or dictionary reading.

Output Mapping

Editing the attribute opens the Transform Editor where you can decide what should be sent to an output port.

The dialog provides you with all the power and features known from Transform Editor and CTL2 - Data Shaper Transformation Language.

To do the mapping in a few basic steps:

1. Provided you already have some output metadata, just left-click an item in the left-hand pane and drag it onto an output field. This will send the result data to the output.

2. If you do not have any output metadata:
   a. Drag a Field from the left pane and drop it into the right pane (an empty space).
   b.This produces a new field in the output metadata.

You can map various data to the output port:

• Values of fields from input metadata - you can send values from input fields to the output port. This is mainly useful, when you are using some kind of a session identifier for HTTP requests.

• Result - provides result data. These includes:

  • content - the content of the HTTP response as a string. This field will be null if the response is written to a file.
  • contentByte - the raw content of the HTTP response as an array of bytes. This field will be null if the response is written to a file.
  • outputFilePath - the path to a file, where the response has been written. Will be null if the response is not written to a file.
  • statusCode - the HTTP status code of the response.
  • header - the map representing HTTP header properties from the response.
  • rawHeaders - headers of the response.
  • errorMessage - the error message in case that the error output is redirected to a standard output port.

• Attributes - provides values of the component attributes:

  • URL - the URL where the request has been sent.
  • requestMethod - the method that was used for the request.
  • requestContent - the content of the request, that has been sent (if specified as a string).
  • inputFileUrl - a URL of the file containing the request content.

Output mapping uses CTL (you can switch to the Source tab). All kinds of functions are available to modify the value to be stored in the output field.
$out.0.prices = find($in.1.content, "price: .*? USD")
finds all occurrences of the form price: [some text] USD in the response content.

If you let output mapping empty, the default output mapping is used:

$out.0.* = $in.0.*;
$out.0.* = $in.1.*;

The default mapping has been introduced in version 4.1.0.

Error mapping

Editing the Error mapping attribute opens the Transform Editor where you can map error details to an output port. The behavior is very similar to the Output mapping above.

If you let error mapping empty, the default error mapping is used:

$out.1.* = $in.0.*;
$out.1.* = $in.1.*;

The default mapping has been introduced in version 4.1.0.

Notes

When the graph’s log level is set to DEBUG, the HTTPConnector prints the HTTP request and response to graph log.

Examples

Downloading a Web Page

Download the content using HTTPConnector. Save the result to the file for further processing.

Solution

Use the URL and Output file URL attributes. The downloaded page will be saved into the result.html file in the ${DATAOUT_DIR} directory.

Downloading Document Requiring HTTP Authentication

Download a document from https://protected.example.org/document.html. The site requires HTTP basic authentication.

Solution

Set up the URL, Output file URL, Username and Password attributes. We suggest to use secure parameters to store your password.

An alternative solution is to connect an edge to the first output port instead of filling the Output file URL attribute. The result will be send to the edge. No output mapping is necessary.

Connecting via HTTP Proxy without Password

Download the content of the page http://. The page is accessible via proxy on 10.0.3.5 listening on TCP port 3128.

Solution

Use the URL attribute. You can use Output file URL to write a result to a file, or connect an output edge.

Note: The proxy may introduce some limitations. For example, it may deny you to connect via HTTPS, etc.

Connecting via HTTP Proxy using Password

The problem to be solved is similar to the previous example. The difference is that proxy requires a username (test) and password (securePassword).

Solution

Using OAuth in HTTPConnector

Connect to Twitter API and get some tweets about Java.

Solution

Use the URL, OAuth Consumer key, OAuth Consumer secret, OAuth Access Token and OAuth Access Token secret attributes.

Connect an edge to the first output port to pass results by the edge or fill in the Output file URL attribute to write down results to a file.

Note: The credentials in this example are not valid, you have to use your own credentials.

Upload a File using Multipart Entities

Send a file using multipart entities. The file content is available in field1 field.

Solution

Use the URL, Request method, Multipart entities and Input mapping attributes.

function integer transform() {
    $out.4.field1_EntityContent = $in.0.field1;

    return ALL;
}

Map multipart entities in the Input mapping dialog.

Using Connection Timeout and Retry Count

Connect to www.my-sometimes-responding-server.com which sometimes fails to respond. The response has to be returned within 20 seconds, otherwise connection should be considered as nonresponding. Make at most 5 attempts in total.

Solution

Use Timeout to set up time limit on connection to avoid waiting if server does not reply. If server responds sometimes only, use Retry count to ask several times.

Timeout is in milliseconds. If you need to set it in seconds, minutes, hours, etc., add the unit. See Time Intervals. Retry count set to 4 causes up to 4 additional retries (if necessary). At most five requests are performed in total.

Best Practices

We recommend users to explicitly specify Request/response charset.

See also
WebServiceClient
Common properties of components
Specific attribute types