Connect AWS S3 Bucket

You can use your own S3 bucket with Illumina Connected Analytics (ICA) for data storage. This section describes how to configure your AWS account to allow ICA to connect to an S3 bucket.
These instructions utilize the AWS CLI. Follow the AWS CLI documentation for instructions to download and install.
Key points for connected AWS S3 buckets to ICA:
The AWS S3 bucket must exist in the same AWS region as the ICA project. Refer to the table below for a mapping of ICA project regions to AWS regions:
ICA Project Region
AWS Region
United States
us-east-1
Singapore
ap-southeast-1
Australia
ap-southeast-2
Japan
ap-northeast-1
South Korea*
ap-northeast-2
India
ap-south-1
UK
eu-west-2
Canada
ca-central-1
Germany
eu-central-1
*Note: BSSH is not deployed currently on the South Korea instance, therefore there will be limited functionality in this region with regard to sequencer integration.
You can enable SSE using an Amazon S3-managed key (SSE-S3). Instructions for using KMS-managed (SSE-KMS) keys are found here.
Because of how Amazon S3 handles folders and does not send events for S3 folders, the following restrictions must be taken into account for ICA project data stored in S3.
When creating an empty folder in S3, it will not be visible in ICA.
When moving folders in S3, the original, but empty, folder will remain visible in ICA and must be manually deleted there.
When deleting a folder and its contents in S3, the empty folder will remain visible in ICA and must be manually deleted there.
Projects cannot be created with ./ as prefix since S3 does not allow uploading files with this key prefix.
When configuring a new project in ICA to use a preconfigured S3 bucket, you cannot use the root folder of the S3 bucket. This is done to prevent locking issues.
❗️ For Bring Your Own Storage buckets, only buckets without object versioning are supported. If users connect buckets with object versioning, the data in objectstore and in ICA can get out of sync. For Bring Your Own Storage buckets with versioning enabled, when an object is deleted, a "Delete marker" is created to indicate that the object has been deleted, instead of immediately deleting the object. The user should manually delete both the object and its corresponding delete marker or set up a lifecycle cleanup to remove unwanted versions.
Configure Bucket CORS Permission
ICA requires cross-origin resource sharing (CORS) permissions to write to the S3 bucket for uploads via the browser. Refer to the Configuring cross-origin resource sharing (CORS) (expand the "Using the S3 console" section) documentation for instructions on enabling CORS via the AWS Management Console. Use the following configuration during the process:
In the cross-origin resource sharing (CORS) section, enter the following content.
[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "HEAD",
            "GET",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "https://ica.illumina.com"
        ],
        "ExposeHeaders": [
            "ETag",
            "x-amz-meta-custom-header"
        ]
    }
]
Create AWS IAM Policy
ICA requires specific permissions to access data in an AWS S3 bucket. These permissions are contained in an AWS IAM Policy.
Refer to the Creating policies on the JSON tab documentation for instructions on creating an AWS IAM Policy via the AWS Management Console. Use the following configuration during the process:
Paste the JSON policy document below. Replace BUCKET_NAME with the actual name of your bucket. Note the example below provides access to all objects prefixes in the bucket.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutBucketNotification",
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:RestoreObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetFederationToken"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
(Optional) Set policy name to "illumina-ica-admin-policy"
To create the IAM Policy via the AWS CLI, create a local file named illumina-ica-admin-policy.json containing the policy content above and run the following command. Be sure the path to the policy document (--policy-document) leads to the path where you saved the file:
aws iam create-policy --policy-name illumina-ica-admin-policy --policy-document file://illumina-ica-admin-policy.json
Create AWS IAM User
An AWS IAM User is needed to create an Access Key for ICA to connect to the AWS S3 Bucket. The policy will be attached to the IAM user to grant the user the necessary permissions.
Refer to the Creating IAM users (console) documentation for instructions on creating an AWS IAM User via the AWS Management Console. Use the following configuration during the process:
(optional) Set user name to "illumina_ica_admin"
Select the Programmatic access option for the type of access
Select Attach existing policies directly when setting the permissions, and choose the policy created in Create AWS IAM Policy​
(Optional) Retrieve the Access Key ID and Secret Access Key by choosing to Download .csv
To create the IAM user and attach the policy via the AWS CLI, enter the following command (AWS IAM users are global resources and do not require a region to be specified). This command creates an IAM user illumina_ica_admin, retrieves your AWS account number, and then attaches the policy to the user.
aws iam create-user --user-name illumina_ica_admin
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
aws iam attach-user-policy --policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/illumina-ica-admin-policy --user-name illumina_ica_admin
Create AWS Access Key
If the Access Key information was retrieved during the IAM user creation, skip this step.
Refer to the Managing access keys (console) AWS documentation for instructions on creating an AWS Access Key via the AWS Console. See the "To create, modify, or delete another IAM user's access keys (console)" sub-section.
Use the below command to create the Access Key for the illumina_ica_admin IAM user. Note the SecretAccessKey is sensitive and should be stored securely. The access key is only displayed when this command is executed and cannot be recovered. A new access key must be created if it is lost.
aws iam create-access-key --user-name illumina_ica_admin
​
    "AccessKey": {
        "UserName": "illumina_ica_admin",
        "AccessKeyId": "<access key id>",
        "Status": "Active",
        "SecretAccessKey": "<secret access key>",
        "CreateDate": "2020-10-22 09:42:24+00:00"
    }
The AccessKeyId and SecretAccessKey values will be provided to ICA in the next step.
Create ICA Secret Credential
To connect your S3 account to ICA, you need to add a secret credential in ICA containing the Access Key ID and Secret Access Key created in the previous step. From the ICA home screen, navigate to System Settings > Storage > Credentials and click the +New button to create a new secret credential.
Provide a name for the secret credentials, ensure the type is set to "AWS user" and provide the Access Key ID and Secret Access Key.
With the secret credentials created, a storage configuration can be created using the secret credential. Refer to the instructions to Create a Storage Configuration for details.
ICA Project Region	AWS Region
United States	us-east-1
Singapore	ap-southeast-1
Australia	ap-southeast-2
Japan	ap-northeast-1
South Korea*	ap-northeast-2
India	ap-south-1
UK	eu-west-2
Canada	ca-central-1
Germany	eu-central-1
Last modified 1mo ago