LogoLogo
Illumina Connected Software
  • Introduction
  • Get Started
    • About the Platform
    • Get Started
  • Home
    • Projects
    • Bundles
    • Event Log
    • Metadata Models
    • Docker Repository
    • Tool Repository
    • Storage
      • Connect AWS S3 Bucket
        • SSE-KMS Encryption
  • Project
    • Data
      • Data Integrity
    • Samples
    • Activity
    • Flow
      • Reference Data
      • Pipelines
        • Nextflow
        • CWL
        • XML Input Form
        • 🆕JSON-Based input forms
          • InputForm.json Syntax
          • JSON Scatter Gather Pipeline
        • Tips and Tricks
      • Analyses
    • Base
      • Tables
        • Data Catalogue
      • Query
      • Schedule
      • Snowflake
    • Bench
      • Workspaces
      • JupyterLab
      • 🆕Bring Your Own Bench Image
      • 🆕Bench Command Line Interface
      • 🆕Pipeline Development in Bench (Experimental)
        • Creating a Pipeline from Scratch
        • nf-core Pipelines
        • Updating an Existing Flow Pipeline
      • 🆕Containers in Bench
      • FUSE Driver
    • Cohorts
      • Create a Cohort
      • Import New Samples
      • Prepare Metadata Sheets
      • Precomputed GWAS and PheWAS
      • Cohort Analysis
      • Compare Cohorts
      • Cohorts Data in ICA Base
      • Oncology Walk-through
      • Rare Genetic Disorders Walk-through
      • Public Data Sets
    • Details
    • Team
    • Connectivity
      • Service Connector
      • Project Connector
    • Notifications
  • Command-Line Interface
    • Installation
    • Authentication
    • Data Transfer
    • Config Settings
    • Output Format
    • Command Index
    • Releases
  • Sequencer Integration
    • Cloud Analysis Auto-launch
  • Tutorials
    • Nextflow Pipeline
      • Nextflow DRAGEN Pipeline
      • Nextflow: Scatter-gather Method
      • Nextflow: Pipeline Lift
        • Nextflow: Pipeline Lift: RNASeq
      • Nextflow CLI Workflow
    • CWL CLI Workflow
      • CWL Graphical Pipeline
      • CWL DRAGEN Pipeline
      • CWL: Scatter-gather Method
    • Base Basics
      • Base: SnowSQL
      • Base: Access Tables via Python
    • Bench ICA Python Library
    • API Beginner Guide
    • Launch Pipelines on CLI
      • Mount projectdata using CLI
    • Data Transfer Options
    • Pipeline Chaining on AWS
    • End-to-End User Flow: DRAGEN Analysis
  • Reference
    • Software Release Notes
      • 2025
      • 2024
      • 2023
      • 2022
      • 2021
    • Document Revision History
      • 2025
      • 2024
      • 2023
      • 2022
    • Known Issues
    • API
    • Pricing
    • Security and Compliance
    • Network Settings
    • ICA Terminology
    • Resources
    • Data Formats
    • FAQ
Powered by GitBook
On this page
  • Prerequisites
  • Configuration
  • 1 - Configure Bucket CORS Permission
  • 2 - Create Data Access Permission - AWS IAM Policy
  • 3 - Create AWS IAM User
  • 4. -Create AWS Access Key
  • 5 - S3 Bucket Policy
  • 6 - Create ICA Storage Credential
  • 7 - Enabling Cross Account Access for Copy and Move Operations
  • Troubleshooting
  • Common Issues
  • Conflicting bucket notifications
  • GetTemporaryUploadCredentialsAsync failure

Was this helpful?

Export as PDF
  1. Home
  2. Storage

Connect AWS S3 Bucket

PreviousStorageNextSSE-KMS Encryption

Last updated 3 days ago

Was this helpful?

You can use your own S3 bucket with Illumina Connected Analytics (ICA) for data storage. This section describes how to configure your AWS account to allow ICA to connect to an S3 bucket.

These instructions utilize the AWS CLI. Follow the for instructions to download and install.

When configuring a new project in ICA to use a preconfigured S3 bucket, create a folder on your S3 bucket in the AWS console. This folder will be connected to ICA as a prefix.

Failure to create a folder will result in the root folder of your S3 bucket being assigned which will block your S3 bucket from being used for other ICA projects with the error "Conflict while updating file/folder. Please try again later."

  • When creating an empty folder in S3, it will not be visible in ICA.

  • When moving folders in S3, the original, but empty, folder will remain visible in ICA and must be manually deleted there.

  • When deleting a folder and its contents in S3, the empty folder will remain visible in ICA and must be manually deleted there.

  • Projects cannot be created with ./ as prefix since S3 does not allow uploading files with this key prefix.

Prerequisites

The AWS S3 bucket must exist in the same AWS region as the ICA project. Refer to the table below for a mapping of ICA project regions to AWS regions:

ICA Project Region
AWS Region

Australia

ap-southeast-2

Canada

ca-central-1

Germany

eu-central-1

India

ap-south-1

Indonesia

ap-southeast-3

Israel

il-central-1

Japan

ap-northeast-1

Singapore

ap-southeast-1

South Korea*

ap-northeast-2

UK

eu-west-2

United Arab Emirates

me-central-1

United States

us-east-1

(*) BSSH is not currently deployed on the South Korea instance, resulting in limited functionality in this region with regard to sequencer integration.

You can use unversioned, versioned and suspended buckets as own S3 storage.

If you connect buckets with object versioning, the data in ICA will be automatically synced with the data in objectstore. When an object is deleted without specifying a particular version, a Delete marker is created on the objectstore to indicate that the object has been deleted. ICA will reflect the object state by deleting the record from the database. No further action on your side is needed to sync.

Configuration

1 - Configure Bucket CORS Permission

In the cross-origin resource sharing (CORS) section, enter the following content.

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "HEAD",
            "GET",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "https://ica.illumina.com"
        ],
        "ExposeHeaders": [
            "ETag",
            "x-amz-meta-custom-header"
        ]
    }
]

2 - Create Data Access Permission - AWS IAM Policy

ICA requires specific permissions to access data in an AWS S3 bucket. These permissions are contained in an AWS IAM Policy.

Permissions

paste the JSON policy document below. Note the example below provides access to all objects prefixes in the bucket.

Replace YOUR_BUCKET_NAME with the name of the S3 bucket you created for ICA. Replace YOUR_FOLDER_NAME with the name of the folder in your S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutBucketNotification",
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:RestoreObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/YOUR_FOLDER_NAME/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetFederationToken"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

On Versioned OR Suspended buckets, paste the JSON policy document below. Note the example below provides access to all objects prefixes in the bucket.

Replace YOUR_BUCKET_NAME with the name of the S3 bucket you created for ICA. Replace YOUR_FOLDER_NAME with the name of the folder in your S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutBucketNotification",
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "s3:GetBucketLocation",
                "s3:ListBucketVersions",
                "s3:GetBucketVersioning"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:RestoreObject",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/YOUR_FOLDER_NAME/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetFederationToken"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

(Optional) Set policy name to "illumina-ica-admin-policy"

To create the IAM Policy via the AWS CLI, create a local file named illumina-ica-admin-policy.json containing the policy content above and run the following command. Be sure the path to the policy document (--policy-document) leads to the path where you saved the file:

aws iam create-policy --policy-name illumina-ica-admin-policy --policy-document file://illumina-ica-admin-policy.json

3 - Create AWS IAM User

An AWS IAM User is needed to create an Access Key for ICA to connect to the AWS S3 Bucket. The policy will be attached to the IAM user to grant the user the necessary permissions.

  • (optional) Set user name to "illumina_ica_admin"

  • Select the Programmatic access option for the type of access

  • (Optional) Retrieve the Access Key ID and Secret Access Key by choosing to Download .csv

To create the IAM user and attach the policy via the AWS CLI, enter the following command (AWS IAM users are global resources and do not require a region to be specified). This command creates an IAM user illumina_ica_admin, retrieves your AWS account number, and then attaches the policy to the user.

aws iam create-user --user-name illumina_ica_admin
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
aws iam attach-user-policy --policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/illumina-ica-admin-policy --user-name illumina_ica_admin

4. -Create AWS Access Key

If the Access Key information was retrieved during the IAM user creation, skip this step.

Use the command below to create the Access Key for the illumina_ica_admin IAM user. Note the SecretAccessKey is sensitive and should be stored securely. The access key is only displayed when this command is executed and cannot be recovered. A new access key must be created if it is lost.

aws iam create-access-key --user-name illumina_ica_admin

    "AccessKey": {
        "UserName": "illumina_ica_admin",
        "AccessKeyId": "<access key id>",
        "Status": "Active",
        "SecretAccessKey": "<secret access key>",
        "CreateDate": "2020-10-22 09:42:24+00:00"
    }

The AccessKeyId and SecretAccessKey values will be provided to ICA in the next step.

5 - S3 Bucket Policy

Connecting your S3 bucket to ICA does not require any additional bucket policies.

What if you need a bucket policy for use cases beyond ICA?

The bucket policy must then support the essential permissions needed by ICA without inadvertently restricting its functionality.

Be sure to replace the following fields:

  • YOUR_BUCKET_NAME: Replace this field with the name of the S3 bucket you created for ICA.

  • YOUR_ACCOUNT_ID: Replace this field with your account ID number.

  • YOUR_IAM_USER: Replace this field with the name of your IAM user created for ICA.

{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Deny",
             "Principal": {
                 "AWS": "*"
             },
             "Action": [
                 "s3:PutObject",
                 "s3:GetObject",
                 "s3:RestoreObject",
                 "s3:DeleteObject",
                 "s3:DeleteObjectVersion",
                 "s3:GetObjectVersion"
             ],
             "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*",
             "Condition": {
                 "ArnNotLike": {
                     "aws:PrincipalArn": [
                         "arn:aws:iam::YOUR_ACCOUNT_ID:user/YOUR_IAM_USER",
                         "arn:aws:sts::YOUR_ACCOUNT_ID:federated-user/*"
                     ]
                 }
             }
         }
     ]
 }

In this example, restriction is enabled on the bucket policy to prevent any kind of access to the bucket. However, there is an exception rule added for the IAM user that ICA is using to connect to the S3 bucket. The exception rule is allowing ICA to perform the above S3 action permissions necessary for ICA functionalities.

Additionally, the exception rule is applied to the STS federated user session principal associated with ICA. Since ICA leverages the AWS STS to provide temporary credentials that allow users to perform actions on the S3 bucket, it is crucial to include these STS federated user session principals in your policy's whitelist. Failing to do so could result in 403 Forbidden errors when users attempt to interact with the bucket's objects using the provided temporary credentials.

6 - Create ICA Storage Credential

To connect your S3 account to ICA, you need to add a storage credential in ICA containing the Access Key ID and Access Key created in the previous step. From the ICA home screen, navigate to System Settings > Credentials and click the Create button to create a new storage credential.

Provide a name for the storage credentials, ensure the type is set to "AWS user" and provide the Access Key ID and Secret Access Key.

7 - Enabling Cross Account Access for Copy and Move Operations

ICA uses AssumeRole to copy and move objects from a bucket in an AWS account to another bucket in another AWS account. To allow cross account access to a bucket, the following policy statements must be added in the bucket policy:

Be sure to replace the following fields:

  • ASSUME_ROLE_ARN: Replace this field with the ARN of the cross account role you want to give permission to. Refer to the table below to determine which region-specific Role ARN should be used.

  • YOUR_BUCKET_NAME: Replace this field with the name of the S3 bucket you created for ICA.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Allow cross account access",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "ASSUME_ROLE_ARN"
                },
                "Action": [
                    "s3:PutObject",
                    "s3:DeleteObject",
                    "s3:ListMultipartUploadParts",
                    "s3:AbortMultipartUpload",
                    "s3:GetObject"
                ],
                "Resource": [
                    "arn:aws:s3:::YOUR_BUCKET_NAME",
                    "arn:aws:s3:::YOUR_BUCKET_NAME/*"
                ]
            }
        ]
    }

The ARN of the cross account role you want to give permission to is specified in the Principal. Refer to the table below to determine which region-specific Role ARN should be used.

Region
Role ARN

Australia (AU)

arn:aws:iam::079623148045:role/ica_aps2_crossacct

Canada (CA)

arn:aws:iam::079623148045:role/ica_cac1_crossacct

Germany (EU)

arn:aws:iam::079623148045:role/ica_euc1_crossacct

India (IN)

arn:aws:iam::079623148045:role/ica_aps3_crossacct

Indonesia (ID)

arn:aws:iam::079623148045:role/ica_aps4_crossacct

Israel (IL)

arn:aws:iam::079623148045:role/ica_ilc1_crossacct

Japan (JP)

arn:aws:iam::079623148045:role/ica_apn1_crossacct

Singapore (SG)

arn:aws:iam::079623148045:role/ica_aps1_crossacct

South Korea (KR)

arn:aws:iam::079623148045:role/ica_apn2_crossacct

UK (GB)

arn:aws:iam::079623148045:role/ica_euw2_crossacct

United Arab Emirates (AE)

arn:aws:iam::079623148045:role/ica_mec1_crossacct

United States (US)

arn:aws:iam::079623148045:role/ica_use1_crossacct


Troubleshooting

Common Issues

The following are common issues encountered when connecting an AWS S3 bucket through a storage configuration

Error Type
Error Message
Description/Fix

Access Forbidden

Access forbidden: {message}

Mostly occurs because of lack of permission. Fix: Review IAM policy, Bucket policy, ACLs for required permissions

Conflict

System topic is not in a valid state

Conflict

Found conflicting storage container notifications with overlapping prefixes

Conflict

Found conflicting storage container notifications for {prefix}{eventTypeMsg}

Conflict

Found conflicting storage container notifications with overlapping prefixes{prefixMsg}{eventTypeMsg}

Customer Container Notification Exists

Volume Configuration cannot be provisioned: storage container is already set up for customer's own notification

Invalid Access Key ID

Failed to update bucket policy: The AWS Access Key Id you provided does not exist in our records.

Check the status of the AWS Access Key ID in the console. If not active, activate it. If missing, create it.

Invalid Paramater

Missing credentials for storage container

Invalid Parameter

Missing bucket name for storage container

Invalid Parameter

The storage container name has invalid characters

Invalid Parameter

Storage Container '{storageContainer}' does not exist

Invalid Parameter

Invalid parameters for volume configuration: {message}

Invalid Storage Container Location

Storage container must be located in the {region} region

Invalid Storage Container Location

Storage container must be located in one of the following regions: {regions}

Missing Configuration

Missing queue name for storage container notification

Missing Configuration

Missing system topic name for storage container notification

Missing Configuration

Missing lambda ARN for storage container notification

Missing Configuration

Missing subscription name for storage container notification

Missing Storage Account Settings

The storage account '{storageAccountName}' needs HNS (Hierarchical Namespace) enabled.

Missing Storage Container Settings

Missing settings for storage container

Conflicting bucket notifications

  • Volume Configuration cannot be provisioned: storage container is already set up for customer's own notification

  • Invalid parameters for volume configuration: found conflicting storage container notifications with overlapping prefixes

  • Failed to update bucket policy: Configurations overlap. Configurations on the same bucket cannot share a common event type

Solution:

  1. In the Amazon S3 Console, review your current S3 bucket's notification configuration and look for prefixes that overlaps with your Storage Configuration's key prefix

  2. Delete the existing notification that overlaps with your Storage Configuration's key prefix

  3. ICA will perform a series of steps in the background to re-verify the connection to your bucket.

GetTemporaryUploadCredentialsAsync failure

This error can occur when recreating a recently deleted storage configuration. To fix the issue, you have to delete the bucket notifications:

  1. Choose properties

  2. Navigate to the Event Notifications section and choose the check box for the event notifications with name gds:objectcreated, gds:objectremoved and gds:objectrestore and click Delete.

  3. Wait 15 minutes for the storage to become available in ICA

If you do not want to wait 15 minutes, you can delete the current storage configuration, delete the bucket notifications in the bucket and create a new storage configuration.

Because of how and does not send events for S3 folders, the following restrictions must be taken into account for ICA project data stored in S3.

You can enable SSE using an Amazon S3-managed key (SSE-S3). Instructions for using KMS-managed (SSE-KMS) keys are found .

ICA requires cross-origin resource sharing (CORS) permissions to write to the S3 bucket for uploads via the browser. Refer to the (expand the "Using the S3 console" section) documentation for instructions on enabling CORS via the AWS Management Console.

Refer to the documentation for instructions on creating an AWS IAM Policy via the AWS Management Console. Use the following configuration during the process:

Refer to the documentation for instructions on creating an AWS IAM User via the AWS Management Console. Use the following configuration during the process:

Select Attach existing policies directly when setting the permissions, and choose the policy created in

Refer to the AWS documentation for instructions on creating an AWS Access Key via the AWS Console. See the "To create, modify, or delete another IAM user's access keys (console)" sub-section.

With the secret credentials created, a storage configuration can be created using the secret credential. Refer to the instructions to for details.

See

See

See

See

This error occurs when an existing bucket notification's event information overlaps with the notifications ICA is trying to add. only allows overlapping events with non-overlapping prefix. Depending on the conflicts on the notifications, the error can be presented in any of the following:

In the select the bucket for which you need to delete the notifications from the list.

Amazon S3 handles folders
here
Configuring cross-origin resource sharing (CORS)
Creating policies on the JSON tab
Creating IAM users (console)
Managing access keys (console)
Amazon S3 event notification
Amazon S3 Console
Create AWS IAM Policy
Conflicting bucket notifications
Conflicting bucket notifications
Conflicting bucket notifications
Conflicting bucket notifications
AWS CLI documentation
Connect AWS S3 Bucket to ICA Project
Create a Storage Configuration