Instrument Run Upload

Sample data produced by Illumina sequencing instruments can be made available in ICA. A V2 Sample Sheet is required to be used for the instrument run. The following instruments are currently supported:
NextSeq 1000/2000
NovaSeq 6000
There are several software components involved in the end to end data flow for the instrument run upload:
Software
Purpose
BSSH
Plan and monitor the instrument run, store raw run output files (BCLs)
Instrument
Login and launch the planned instrument run
BCLConvert App
Demultiplex the raw instrument run outputs and store the resulting FASTQ files in ICA
ICA
Store the demultiplexed run data
BaseSpace Sequence Hub (BSSH) is used to plan and monitor the instrument run upload. Raw BCL run data is streamed to BSSH from the instrument. The FASTQ generation process launched after the instrument run upload completes places the demultiplexed FASTQs into an ICA project where the data is accessible through an externally-mananged project. The FASTQ data can then be linked to other projects to be used as input for analyses.
BSSH: Instrument Run Setup
The BSSH Instrument Run Setup provides an interface for planning an instrument run based on a V2 Sample Sheet.
Navigate to BSSH and change to your desired workgroup or personal context. This may require you login to your domain first.
instrument-run-upload-9
After selecting the appropriate user/workgroup context, navigate to the Runs tab in BSSH. Then click the New Run button and select Instrument Run Setup.
instrument-run-upload-1
Complete the instrument run setup form to create a planned run with the following configuration.
Set the Analysis Location to "BaseSpace"
Set the "Type of Analysis" to the "Illumina BCL Convert for ICA" app (recommended to use latest version)
The Illumina BCL Convert for ICA analysis app currently requires an Illumina operator to make the app available for a workgroup.
instrument-run-upload-8
Manual SampleSheet Configuration
An alternative to using the BSSH Instrument Run Setup interface is to provide a samplesheet directly on the Instrument. Use the following settings in the SampleSheet when configuring manually:
v2 SampleSheet is required
In the [Cloud_Settings] section, the value of BsshApp (appslug) must be set to illumina-inc.bcl-convert-for-ica.1.0.0
Instrument: Launch Instrument Run
On the instrument, login using the desired user/workgroup context. Find the planned run and launch the run.
BSSH: Monitor Instrument Run
After an instrument run is started, BSSH is used for monitoring the run through completion. Use the BSSH Runs view to track the run progress.
instrument-run-upload-6
ICA: Access Demulitplexed Run Data
In order to create resources including sample data in ICA, an externally-managed project is created by BSSH. Operations in BSSH, including planning and monitoring instrument runs, are performed in a given user or workgroup context. To preserve permission boundaries between applications, an ICA project is created for each user/workgroup context per region. The ICA project will be named using the convention BSSH {context name}, where {context-name} is the user or workgroup name operating in BSSH. For workgroups, the project will be created on behalf of the workgroup owner.
In the example below, the project is created for a workgroup with name "ICA_BSSH_Demo_WG" where the workgroup owner is "[email protected]".
instrument-run-upload-0
The project Team includes an entry for the workgroup to grant all workgroup users read-only access to the project. While Team members may be assigned the Contributor role in the externally-managed project, they will not have write access (ie, create/delete data, run pipelines) in the project.
instrument-run-upload-4
The project contains the following tags:
Technical Tags: bssh.project.user:{user/workgroup ID}
instrument-run-upload-3
The demultiplexed FASTQ data is uploaded to the BSSH externally-managed project. Sample and data entities in ICA are tagged with key BSSH identifiers for tracability to the associated upstream BSSH resources.
The image below shows details of a sample in the externally-managed project, including tags.
instrument-run-upload-5
The image below shows details of FASTQ data in the externally-managed project, selected from the sample details view.
instrument-run-upload-7
See the table below for the tags added to Samples and Data.
BSSH Property
Tag
Sample
Data
Run ID
bssh.run.id
​
x
Run Name
bssh.run.name
​
x
Run URL
bssh.run.url
​
x
Run API URL
bssh.run.apiurl
​
x
Biosample ID
bssh.biosample.id
x
​
Biosample Name
bssh.biosample.name
x
​
Biosample URL
bssh.biosample.url
x
​
Biosample API URL
bssh.biosample.apiurl
x
​
BSSH Project ID
bssh.project.id
x
x
BSSH Project Name
bssh.project.name
x
x
BSSH Project URL
bssh.project.url
x
x
BSSH Project API URL
bssh.project.apiurl
x
x
Dataset ID
bssh.dataset.id
​
x
Dataset ID Token
bssh.dataset.idtoken
​
x
Dataset Name
bssh.dataset.name
​
x
Dataset API URL
bssh.dataset.apiurl
​
x
The results from the FASTQ Generation (bclConvert), including FASTQs and logs files, are stored in the ICA project data. The FASTQ files are linked to the associated Sample resource. The data organziation is important to preserve BSSH functionality as BSSH expects sepcific file/folder paths. The data is read-only for users accessing the data directly through ICA to prevent corrupting BSSH.
Walking through the folder structure, at the top level there is the datasets/ folder. This serves as a parent container. Under this folder are folders named with a BSSH App Session, representing each execution of the FASTQ Generation app. Inside the app session folder are log files and outputs of the app. The output is stored in the output/ folder. Inside this folder is a fastqs/ folder, containing the outputs from the FASTQ Generation task. The Samples/ folder contains nested folders for the Lanes and Sample IDs which contain the FASTQ files. See the below path for a complete example path of a FASTQ file.
/datasets/app.27849123/output/fastqs/Samples/Lane_1/ICA_SC_NexteraFlex-NA12878-UDP0002/ICA_SC_NexteraFlex-NA12878-UDP0002_S2_L001_R2_001.fastq.gz
Link Sample Data to Other Projects
With sample data available in the BSSH externally-managed project, the sample data can be linked to other ICA projects to serve as inputs for running pipelines, etc. Users with access to the BSSH externally-managed project can create links from the destination project Samples and Data views. See the Link Project Data and Link Project Sample documentation for details on linking sample data.
​Project Connectors provide a mechanism for automatically linking the sample data to a project. Use the BSSH externally-managed project as the "Source" project when configuring the connector.
BSSH: Requeue
A "requeue" is a re-execution of the FASTQ generation (bclConvert) often performed due to an error in the samplesheet provided for the initial execution.
A requeue is performed by manually relaunching the FASTQ Generation app with a corrected samplesheet. Find the original samplesheet by navigating to the Runs tab, click the desired Run to be requeued, then click the Files tab. In the list of files, find the SampleSheet.csv file. Click to open the file, and click the Download button to download locally. After downloading make any desired modifications to the samplesheet to prepare for the requeue.
Once the modified samplesheet is prepared, you'll need to upload the file to an existing BSSH project. Navigate to the "Projects" tab and choose a project to upload the modified samplesheet. Navigate to the "Other Datasets" tab, click the File icon, select Upload > Files.
instrument-run-upload-12
Select "Other" from the list of file types, and use the browser upload UI to select the modified samplesheet.
Find the Illumina BCL Convert for ICA app in the BSSH Apps. Navigate to the Apps tab.
instrument-run-upload-10
In the list of BSSH Apps, select the "BCL Convert for ICA (Beta)" app.
instrument-run-upload-11
Click to Launch the application. Complete the launch form by selecting the instrument run to be requeued and choosing the modified samplesheet in the Override Sample Sheet form field. When ready, click the "Launch Application" button to start the app.