Data Transfer

The ICA CLI can be used for uploading, downloading and viewing information about data stored within ICA projects. If not already authenticated, please see the Authentication section. Once the CLI has been authenticated with your account, use the command below to list all projects:

icav2`` projects list

The first column of the output (in default table format) will show the ID. This is the project-id and will be used in the examples below.

Upload Data

In this example, we will upload a file called Sample-1_S1_L001_R1_001.fastq.gz to the project. Copy your project-id obtained above and use the following command:

icav2`` projectdata upload ``Sample-1_S1_L001_R1_001.fastq.gz --project-id <project-id>

To check if the file has uploaded, run the following command to get a list of all files stored within the specified project:

icav2`` projectdata list ``--project-id <project-id>

This will show a file ID starting with fil. which can be used to get more information about the file and attributes.

icav2`` projectdata get ``<file-id> --project-id <project-id>

We have to use --project-id in the examples above because we have not entered into a specific project context. To enter a project context use the following command.

icav2`` projects enter ``<project-name or project-id>

This will infer the project-id, so that it does not need to be entered for each command.

Uploading Multiple Files

You can only upload individual files with the projectdata upload command, wildcards are not supported.

If you want to upload multiple files in a folder, based on a common name, you can use the following method from the folder where the files are located ls VAL-0* | xargs -I {} icav2 projectdata upload "{}" /my_upload/my_files/ --project-id <project-id>

• ls VAL-0* lists all files in the current directory whose names start with VAL-O, for example VAL-001.txt, VAL-002.bin,...

• | The pipe symbol takes the output of the ls command and passes is as input to the next command

• xargs -I {} take the list of files and execute the next command while replacing the curly brackets for every individual file.

• icav2 projectdata upload "{}" /my_upload/my_files/ This command gets executed for each file and uploads that file to the /my_upload/my_files/ folder

• --project-id <project-id> The id of the project in which you want to upload the files (only needed if you have not entered a project context)

Filenames beginning with / are not allowed, so be careful when entering full path names as those will result in the file being stored on S3 but not being visible in ICA. Likewise, folders containing a / in their individual folder name and folders named '.' are not supported

Download Data

The ICA CLI can also be used to download files. This can be especially useful if the download destination is a remote server or HPC cluster into which you are logged in. To download data into the current folder, run the following command:

icav2`` projectdata download ``<file-id> ./ --project-id <project-id>

Temporary Credentials

To fetch temporary AWS credentials for given project data, use the command icav2`` projectdata temporarycredentials ``[path or data Id] [flags]. If the path is provided, the project id from the flag --project-id is used. If the --project-id flag is not present, then the project id is taken from the context. The returned AWS credentials for file or folder upload expire after 36 hours.

Data Transfer Options

For information on options such as using the ICA API and AWS CLI to transfer data, visit the Data Transfer Options tutorial.