Launch Pipelines on CLI
Prerequisite - Launch a CWL or Nextflow pipeline to completion using the ICA GUI with the intended set of parameters.
Configure and Authenticate ICA command line interface (CLI) Enter the project context using the CLI
icav2 projects list
ID NAME OWNER
a5690b16-a739-4bd7-a62a-dc4dc5c5de6c Project1 670fd8ea-2ddb-377d-bd8b-587e7781f2b5
ccb0667b-5949-489a-8902-692ef2f31827 Project2 f1aa8430-7058-4f6c-a726-b75ddf6252eb
No of items : 2
Use the ID of the project in the next step to enter the project context
icav2 projects enter a5690b16-a739-4bd7-a62a-dc4dc5c5de6c
Find the pipeline you want to start from the CLI
icav2 projectpipelines list
ID CODE DESCRIPTION
fbd6f3c3-cb70-4b35-8f57-372dce2aaf98 DRAGEN Somatic 3.9.5 The DRAGEN Somatic tool identifies somatic variants
b4dc6b91-5283-41f6-8095-62a5320ed092 DRAGEN Somatic Enrichment 3-10-4 The DRAGEN Somatic Enrichment pipeline identifies somatic variants which can exist at low allele frequencies in the tumor sample.
No of items : 2
Find ID associated with pipeline of interest, move on to next step
To find the input files parameter you can use a previously launched projectanalyses with the input command
Find the previous analyses launched using the GUI
icav2 projectanalyses list
ID REFERENCE CODE STATUS
3539d676-ae99-4e5f-b7e4-0835f207e425 kyle-test-somatic-2-DRAGEN Somatic 3_9_5 DRAGEN Somatic 3.9.5 SUCCEEDED
f11e248e-9944-4cde-9061-c41e70172f20 kyle-test-somatic-1-DRAGEN Somatic 3_9_5 DRAGEN Somatic 3.9.5 FAILED
No of items : 2
List the analyses inputs
icav2 projectanalyses input 3539d676-ae99-4e5f-b7e4-0835f207e425
CODE NAMES DATA ID
BED
CNV_B_Allele_VCF
CNV_Population_B_Allele_VCF
HLA_Allele_Frequency_File
HLA_BED
HLA_reference_file_(protein_FASTA)
Microsatellites_File
Normal_BAM_File
Normal_FASTQ_Files
Panel_of_Normals
Panel_of_Normals_TAR
Reference hg38_altaware_nohla-cnv-anchored.v8.tar fil.35e27101fdec404fb37d08d9adf63307
Systematic_Noise_BED
Tumor_BAM_File
Tumor_FASTQ_Files HCC1187C_S1_L001_R1_001.fastq.gz,HCC1187C_S1_L001_R2_001.fastq.gz fil.e1ec77f2647f45804fe508d9aecb19c4,fil.d89018f0c7784fc4b76708d9adf63307
This will return the Input File Codes, as well as the file names and data IDs of the associated data used to previously launch the pipeline
Currently this step for CWL requires the use of the ICA API to access the configuration settings of a project analyses that ran successfully, it is optional for Nextflow since the XML configuration file can be accessed in the ICA GUI.
Click the previous GUI run, and select the pipeline that was run. On the pipeline page, select the XML Configuration Tab to view the configuration settings.
In the "steps" section of the XML file, you will find various steps labeled with
<pd:tool code="map_align">
and subsequent labels of parameters with a similar structure
<pd:parameter code="enable_map_align" minValues="0" maxValues="1" classification="USER">
<pd:label>Enable Map/Align</pd:label>
<pd:description></pd:description>
<pd:optionsType>
<pd:option>true</pd:option>
<pd:option>false</pd:option>
</pd:optionsType>
<pd:value></pd:value>
The code should be used to generate the later command line parameters e.g.
--parameters enable_map_align:true
- Generate JWT Token from API Key or Basic login credentials
- Instructions on how to get an API Key https://illumina.gitbook.io/ica/account-management/am-iam#api-keys
- If your user has access to multiple domains, you will need to need to add a "?tenant=($domain)" to the request
curl -X 'POST' \
'https://ica.illumina.com/ica/rest/api/tokens' \
-H 'accept: application/vnd.illumina.v3+json' \
-H 'X-API-Key: <APIKEY>' \
-d ''
echo -ne '[email protected]:testpassword' | base64
<BASE64UN+PW>
curl -X 'POST' \
'https://ica.illumina.com/ica/rest/api/tokens' \
-H 'accept: application/vnd.illumina.v3+json' \
-H 'Authorization: Basic <BASE64UN+PW>' \
-d ''
- Response to this request will provide a JWT token {"token":($token)}, use the value of the token in further requests
- Using the API endpoint
/api/projects/{projectID}/analyses/{analysisId}/configurations
to find the configuration file listing out all of required and optional parameters
curl -X 'GET' \
'https://ica.illumina.com/ica/rest/api/projects/e501a0d5-f5e7-458c-a590-586c79bb87e0/analyses/3539d676-ae99-4e5f-b7e4-0835f207e425/configurations' \
-H 'accept: application/vnd.illumina.v3+json' \
-H 'Authorization: Bearer <Token>' \
-H 'X-API-Key: <APIKEY>'
The response JSON to this API will have configuration items listed as
{
"items": [{
"name": "DRAGEN_Somatic__enable_variant_caller",
"multiValue": false,
"values": [
"true"
]
}]
}
Structure of the final command
icav2 projectpipelines start cwl $(pipelineID) --user-reference
Plus input optionsInput Options - For CLI, the entire input can be broken down as individual command line arguments
To launch the same analysis as using the GUI, use the same file ID and parameters, if using new data you can use the CLI command
ica projectdata list
to find new file IDs to launch a new instance of the pipeline Required information in Input - Input Data and Parameters- This option requires the use of
--type input STRUCTURED
along with--input
and--parameters
icav2 projectpipelines start cwl fbd6f3c3-cb70-4b35-8f57-372dce2aaf98 \
--user-reference kyle-test-somatic-9 \
--storage-size small \
--type-input STRUCTURED \
--input Reference:fil.35e27101fdec404fb37d08d9adf63307 \
--input Tumor_FASTQ_Files:fil.e1ec77f2647f45804fe508d9aecb19c4,fil.d89018f0c7784fc4b76708d9adf63307 \
--parameters DRAGEN_Somatic__enable_variant_caller:true \
--parameters DRAGEN_Somatic__enable_hrd:false \
--parameters DRAGEN_Somatic__enable_sv:true \
--parameters DRAGEN_Somatic__output_file_prefix:tumor \
--parameters DRAGEN_Somatic__enable_map_align:true \
--parameters DRAGEN_Somatic__cnv_use_somatic_vc_baf:false \
--parameters DRAGEN_Somatic__enable_cnv:false \
--parameters DRAGEN_Somatic__output_format:BAM \
--parameters DRAGEN_Somatic__vc_emit_ref_confidence:BP_RESOLUTION \
--parameters DRAGEN_Somatic__enable_hla:false \
--parameters DRAGEN_Somatic__enable_map_align_output:true
Successful Response
analysisStorage.description 1.2 TB
analysisStorage.id 6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
analysisStorage.name Small
analysisStorage.ownerId 8ec463f6-1acb-341b-b321-043c39d8716a
analysisStorage.tenantId f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
analysisStorage.tenantName ica-cp-admin
analysisStorage.timeCreated 2021-11-05T10:28:20Z
analysisStorage.timeModified 2021-11-05T10:28:20Z
id 51abe34a-2506-4ab5-adef-22df621d95d5
ownerId 47793c21-75a6-3aa8-8147-81b354d0af4d
pipeline.analysisStorage.description 1.2 TB
pipeline.analysisStorage.id 6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
pipeline.analysisStorage.name Small
pipeline.analysisStorage.ownerId 8ec463f6-1acb-341b-b321-043c39d8716a
pipeline.analysisStorage.tenantId f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
pipeline.analysisStorage.tenantName ica-cp-admin
pipeline.analysisStorage.timeCreated 2021-11-05T10:28:20Z
pipeline.analysisStorage.timeModified 2021-11-05T10:28:20Z
pipeline.code DRAGEN Somatic 3.9.5
pipeline.description The DRAGEN Somatic tool identifies somatic variants which can exist at low allele frequencies in the tumor sample. The pipeline can analyze tumor/normal pairs and tumor-only sequencing data. The normal sample, if present, is used to avoid calls at sites with germline variants or systematic sequencing artifacts. Unlike germline analysis, the somatic platform makes no ploidy assumptions about the tumor sample, allowing sensitive detection of low-frequency alleles.
pipeline.id fbd6f3c3-cb70-4b35-8f57-372dce2aaf98
pipeline.language CWL
pipeline.ownerId e9dd2ff5-c9ba-3293-857e-6546c5503d76
pipeline.tenantId 55cb0a54-efab-4584-85da-dc6a0197d4c4
pipeline.tenantName ilmn-dragen
pipeline.timeCreated 2021-11-23T22:55:49Z
pipeline.timeModified 2021-12-09T16:42:14Z
reference kyle-test-somatic-9-DRAGEN Somatic 3_9_5-bc56d4b1-f90e-4039-b3a4-b11d29263e4e
status REQUESTED
summary
tenantId b5b750a6-49d4-49de-9f18-75f4f6a81112
tenantName ilmn-cci
timeCreated 2022-03-16T22:48:31Z
timeModified 2022-03-16T22:48:31Z
userReference kyle-test-somatic-9
Unsuccessful Response Pipeline ID not formatted correctly
400 Bad Request : ICA_API_004 : com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `java.util.UUID` from String "8f57-372dce2aaf98": UUID has to be represented by standard 36-char representation
at [Source: (io.undertow.servlet.spec.ServletInputStreamImpl); line: 1, column: 983] (through reference chain: com.bluebee.rest.v3.publicapi.dto.analysis.SearchMatchingActivationCodesForCwlAnalysisDto["pipelineId"]) (ref. c9cd9090-4ddb-482a-91b5-8471bff0be58)
- Check that the pipeline ID is correct based on
icav2 projectpipelines list
File ID not found
404 Not Found : ICA_GNRC_001 : Could not find data with ID [fil.35dec404fb37d08d9adf63307] (ref. 91b70c3c-378c-4de2-acc9-794bf18258ec)
- Check that the file ID is correct based on
icav2 projectdata list
Parameter not found
400 Bad Request : ICA_EXEC_007 : The specified variableName [DRAGEN] does not exist. Make sure to use an existing variableName (ref. ab296d4e-9060-412c-a4c9-562c63450022)
When using nextflow to start runs, the input-type parameter is not used, but the
--project-id
is requiredStructure of the file command
icav2 projectpipelines start nextflow $(pipelineID) --user-reference
Plus input optionsicav2 projectpipelines start nextflow b4dc6b91-5283-41f6-8095-62a5320ed092 \
--user-reference "somatic-3-10-test5" \
--project-id e501a0d5-f5e7-458c-a590-586c79bb87e0 \
--storage-size Small \
--input ref_tar:fil.35e27101fdec404fb37d08d9adf63307 \
--input tumor_fastqs:fil.e1ec77f2647f45804fe508d9aecb19c4,fil.d89018f0c7784fc4b76708d9adf63307 \
--parameters enable_map_align:true \
--parameters enable_map_align_output:true \
--parameters output_format:BAM \
--parameters enable_variant_caller:true \
--parameters vc_emit_ref_confidence:BP_RESOLUTION \
--parameters enable_cnv:false \
--parameters enable_sv:true \
--parameters repeat_genotype_enable:true \
--parameters enable_hla:false \
--parameters enable_variant_annotation:false \
--parameters output_file_prefix:Tumor
- Response status can be used to determine if the pipeline was submitted successfully
- status options: REQUESTED,SUCCEEDED,FAILED,ABORTED
analysisStorage.description 1.2 TB
analysisStorage.id 6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
analysisStorage.name Small
analysisStorage.ownerId 8ec463f6-1acb-341b-b321-043c39d8716a
analysisStorage.tenantId f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
analysisStorage.tenantName ica-cp-admin
analysisStorage.timeCreated 2021-11-05T10:28:20Z
analysisStorage.timeModified 2021-11-05T10:28:20Z
id 9b8f9e84-2e7f-4adb-92e5-738b032c2328
ownerId 47793c21-75a6-3aa8-8147-81b354d0af4d
pipeline.analysisStorage.description 1.2 TB
pipeline.analysisStorage.id 6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
pipeline.analysisStorage.name Small
pipeline.analysisStorage.ownerId 8ec463f6-1acb-341b-b321-043c39d8716a
pipeline.analysisStorage.tenantId f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
pipeline.analysisStorage.tenantName ica-cp-admin
pipeline.analysisStorage.timeCreated 2021-11-05T10:28:20Z
pipeline.analysisStorage.timeModified 2021-11-05T10:28:20Z
pipeline.code DRAGEN Somatic Enrichment 3-10-4
pipeline.description The DRAGEN Somatic Enrichment pipeline identifies somatic variants which can exist at low allele frequencies in the tumor sample.
pipeline.id b4dc6b91-5283-41f6-8095-62a5320ed092
pipeline.language NEXTFLOW
pipeline.ownerId e9dd2ff5-c9ba-3293-857e-6546c5503d76
pipeline.tenantId 55cb0a54-efab-4584-85da-dc6a0197d4c4
pipeline.tenantName ilmn-dragen
pipeline.timeCreated 2022-03-07T18:09:38Z
pipeline.timeModified 2022-03-08T17:47:20Z
reference somatic-3-10-test5-DRAGEN Somatic Enrichment 3-10-4-3df131a2-2187-489a-b9f8-140e3ec5efb0
status REQUESTED
tenantId b5b750a6-49d4-49de-9f18-75f4f6a81112
tenantName ilmn-cci
timeCreated 2022-07-13T22:44:47Z
timeModified 2022-07-13T22:44:47Z
userReference somatic-3-10-test5
Last modified 1yr ago