Guides

Client Connection: Amazon S3 - NEW! 🚀

Suggest Edits

Fields with the asterisk * are mandatory.

PropertiesDescription
NAME *Name of the Amazon S3 connection you are creating. The maximum number of characters is 128. Blanks and these special characters: ' " | are not supported.
DESCRIPTIONDescription of the Amazon S3 connection you are creating. The maximum number of characters is 2048.
STATUSEnable or disable the client connection. Possible values:

  • Enabled (default value)
  • Disabled
    • CONNECTION TIMEOUTNumber of seconds without network activity to wait before closing a session due to inactivity.
    Default value: 60.
    AUTHENTICATION TYPE Select if authentication occurs with Access Key (BASIC) or with Security Token Service (STS):
  • BASIC (default)

  • STS (Security Token Service)
    The STS authentication mode gives the user temporary access to use S3 api based on permissions given by credentials of another user. For example, if user A has permissions to upload and download a file, user B that connects using STS based on credentials of user A will inherit the same privileges.
    To learn more about STS authentication, please refer to Understanding and managing Amazon S3 STS authentication below.
    • ACCESS KEY ID * This field appears when BASIC is selected.
    Enter the AWS access key ID.
    SECRET ACCESS KEY * This field appears when BASIC is selected.
    Enter the AWS secret access key.
    ROLE ARN * This field appears when STS is selected.
    Enter the role’s Amazon Resource Name (ARN).
    Format: arn:partition:service:region:namespace:relative-id
    ROLE SESSION NAME *This field appears when STS is selected.
    Enter the unique identifier role session name.
    BUCKET NAME * Enter the name of the S3 bucket. The minimum number of characters is 3 and the maximum is 63. Blanks are not supported.
    REGION *Select the Region of the S3 bucket specified in the BUCKET NAME field.
    ENDPOINT URL Enter the URL serving as the endpoint for the S3-compatible endpoint.
    When populated with a valid URL, this endpoint allows connections to S3-compatible storage services not provided by AWS.
    It is recommended to leave this field blank for standard AWS S3 connections. If you leave this parameter blank, the default AWS S3 endpoint for the above specified Region will be automatically determined and used. Otherwise, a valid Amazon AWS S3 URL or the URL of a non-AWS S3-compatible service provider must be provided.
    If you are connecting to an HTTPS endpoint, your S3 service provider's certificates must be imported into Data One Trust Store going to Setup → Trust Store → Certificates.
    🚀 In case you are using AWS over HTTPS, the certificates to be imported can be download from this page: https://www.amazontrust.com/repository/.
    UPLOAD PART SIZE *This is the size of each part in a multipart upload. By default, it is set to 5 MB. 🚀 The maximum value is 5000 MB.
    Multipart upload allows uploading an object (file) as a set of parts. Once uploaded, the object will be available as a unique part. This facilitates the upload, especially in case of pause and resume.
    MAX RETRIESMaximum number of retries. If this field is empty or set to 0, no retries will be executed and the job will be set to Failed if the first attempt is not successful. A high value corresponds to a high number of retrying attempts.
    RETRY INTERVAL This field will appear only if you have set the maximum numbers of retries to 1 or greater.
    Time (in seconds) waited before retrying executing a job. The lower the value, the more frequent the retrying attempts. If this field is empty or set to 0, no retries will be executed and the job will be set to Failed if the first attempt is not successful.
    CONFIGURATION TEST Use this section to test that the Cluster and the STENG are connected correctly. Select a Cluster and a STENG you want to test and click the TEST button. Data One automatically selects a Cluster and a STENG to test the connection. Note that a Cluster will be selected only if it is the only one available.
    A message with a green check will confirm a successful connection. A message with a red exclamation mark will warn about an unsuccessful connection. The message also details the reason why the connection fails.
    The connection might not be successful for different reasons, each of them depending on the transport type and the configuration.
    The first time you try to test the connection, if trusting certificates are missing, an error message detailing the unsuccessful connection will appear. In this case, go to Setup → Untrusted Cache → Certificates tab and trust the relevant certificates.
    Note that trusting certificates in the Untrusted Cache page may not be sufficient to establish a successful connection. Refer to the Untrusted Cache section of this guide for all the details.

    Understanding and managing Amazon S3 STS authentication STS authentication

    STS (Security Token Service) is an authentication mode that gives the user temporary access to use S3 api based on permissions given by another user’s credentials.

    For example: user Anne has permissions to upload and download a file. If user Bob connects using STS based on Anne’s credentials, Bob will have the same permissions as Anne.

    Let’s see the steps to follow to authenticate with STS:

    1. Using the AWS console, add a new role or edit an existing one:

    1. Trust entities that can assume the role and the access conditions for the role. To do so, use the Edit Trust Relationship and configure the role:

    1. Update the existing json with a custom one. Here is an example:
    {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::account_id:user/user_name"
},
"Action": "sts:AssumeRole"
}
]
}

    Where:

    • account_id is the id of the user account in the aws console
    • user_name is the login of the user in the aws console
    1. In PRIMEUR Data One, update the global configuration adding basic user credential properties on the existing ghibli-speng-centralnode module and create a new s3 section.

    Role name and role session name parameters are also needed.
    The role name is the Amazon Resource Name (ARN) of the role to assume. This information is available in the AWS Console:

    The format used is arn:aws:iam::account_id:role/role_name.
    Where:

    • account_id is the id of the user account in the aws console
    • role_name is the name of the assumed role

    The role session name is an identifier for the assumed role session.

    Updated 26 days ago