Links

Data

The Data section shows files and folders stored in the project.

File/Folder Naming

ICA supports UTF-8 characters in file and folder names for data. Following recommended practices, users are encouraged to follow additional guidelines detailed below. (For more information about recommended approaches to file naming that can be applicable across platforms, please refer to this AWS S3 documentation.)
Characters generally considered "safe":
  • Alphanumeric characters
    • 0-9
    • a-z
    • A-Z
  • Special characters
    • Exclamation point !
    • Hyphen -
    • Underscore _
    • Period .
    • Asterisk *
    • Single quote '
    • Open parenthesis (
    • Closed parenthesis )
Length of file name (minus prefixes and delimiters) generally should be limited to 32 characters.

Data Formats

The ICA environment currently supports the following formats for data files:
Format
File Extension (if applicable)
AMBIG
.ambig
AVRO
.avro, .AVRO
BAI
.bai
BAM
.bam
BCL
.bcl
BED
.bed
BedGraph
.BedGraph, .bedGraph
BGEN
.bgen
BIM
.bim
BMP
.bmp, .BMP
COV
.cov
CSV
.csv
DEL
.del
DICT
.dict
DOCKER
FAI
.fa.fai, .fasta.fai
FASTA
.fasta, .fa
FASTQ
.fastq, .fastq.gz, .fq, .fq.gz
FASTQC
FLAGSTAT
.flagstat
GEM
.gem
GFF
.gff
GIF
.gif, .GIF
GTF
.gtf
GVCF
.gvcf, .gvcf.gz, .hard-filterd.gvcf.gz
GZ
.gz
HTML
.htm, .html
INSER
.inser
JPG
.jpg, .JPG, .jpeg, .JPEG
JSON
.json
JSON_NEWLINE
.ndjson
KINSHIP
.kinship
LEN
.len
LOOM
.loom
MATRIX
.matrix
PARQUET
.parquet, .PARQUET
PDF
.pdf, .PDF
PGEN
.pgen
PNG
.png, .PNG
PSAM
.psam
PVAR
.pvar
RDATA
.Rdata, .RDATA, .RData, .rdata
SAM
.sam
SAMPLEMETADATA
SNP
.snp
SVG
.svg, .SVG
TAR
.tar.gz, .tar, .tgz, .v8.tar
TBI
.tbi
TSV
.tsv
TXT
.txt, .TXT, .out, .err, .log, .Rlog
UNKNOWN
VCF
.vcf.gz, .vcf, .hard-filtered.vcf.gz, .cnv.vcf.gz
VCF.IDX
.vcf.idx
XLSX
.xls, .xlsx
XML
.xml, .XML
YPILEUP
.ypileup
ZIP
.zip

Data Privacy

Users should be mindful of their data's particular access permissions and conditions when linking data files in ICA, either through Bring Your Own Bucket (BYOB) or by uploading. With BYOB, in particular, exposing one's own bucket through ICA may provide access to those files to a different set of users who would not have access to the original storage bucket outside of ICA. In general, users should ensure the path to their source data is correct and does not include unintended files when performing uploads in order to avoid unintentional privacy breaches. More guidance can be found in the ICA Security and Compliance section.

View Data

On the Data page, you can view information on or preview files.
To view file data, do as follows.
  1. 1.
    Select a file to view.
  2. 2.
    Select one of the following viewing options:
    • Data Details — View information on the data.
    • View — Preview the file. You can also preview the file content by selecting the file on the Data page, and then selecting View

Upload Data

Uploading data to the platform makes it available for consumption by analysis workflows and tools. There are multiple methods to upload data.

Upload Data via UI

Uploads via the UI are limited to 5GB
Use the following instructions to upload data manually via the drag-and-drop interface in the platform UI.
  1. 1.
    From the projects menu, select Data.
  2. 2.
    To add data, use one of the following methods. Make sure the Illumina Connected Analytics tab is open in the browser while data uploads.
    • Drag a file from your system into Choose a file or drag it here box.
    • Select the Choose a file or drag it here box, and then choose a file. Select Open to upload the file.
Your file or files are added to the Data page when upload completes.
data-0

Upload Data via CLI

For instructions on uploading/downloading data via CLI, see CLI Data Transfer.

Download Data

Some small files can be downloaded directly from within the UI. Things like .txt and .csv files can be viewed by clicking on the filename when in the project Data section. On the View tab, the file can be viewed directly (larger files may take some time to load) and the Download button will allow you to download the file directly from the UI.
download-ui-1

Schedule for Download

You can trigger an asynchronous download via service connector using the Schedule for Download button with one or more files selected.
  1. 1.
    Select a file or files to download.
  2. 2.
    Select Schedule for Download.
  3. 3.
    Select a connector, and then select Schedule for Download.
You can view the progress of the download or stop the download on the Activity page for the project.

Export Project Data Information

The data records contained in a project can be exported in CSV, JSON, and excel format.
  1. 1.
    Select one or more files to export.
  2. 2.
    Select Export.
  3. 3.
    Select the following export options:
    • To export only the selected file, select the Selected rows as the Rows to export option. To export all files on the page, select Current page.
    • To export only the columns present for the file, select the Visible columns as the Columns to export option.
  4. 4.
    Select the export file type

Data Lifecycle Management

Uploaded files are automatically added with the standard storage tier. You can use files in the standard tier in your analysis.
To manually archive or delete files, do as follows.
  1. 1.
    Select the checkbox next to the file or files to delete or archive.
  2. 2.
    Select Manage, and then select one of the following options:
    • Archive — Move the file or files to long-term storage.
    • Unarchive — Remove the file or files from long-term storage. Unarchived files can be used in analysis.
    • Delete — Remove the file completely.
You can perform analysis on data from other projects by linking data from that project.
  1. 1.
    Select Manage, and then select Link Data.
  2. 2.
    To view data by project, select Add filter, and then select Projects.
  3. 3.
    Select the checkbox next to the file or files to add.
  4. 4.
    Select Select Data.
Your files are added to the Data page. To view the linked data file, select Add filter, and then select Links.