Launch Pipelines on CLI

Launch a Prepackaged DRAGEN Pipeline

Prerequisite - Launch a CWL or Nextflow pipeline to completion using the ICA GUI with the intended set of parameters.

Configure CLI and Identify Pipeline ID

Configure and Authenticate ICA command line interface (CLI).

Obtain a list of your projects with their associated IDs:

icav2 projects list

ID                                      NAME         OWNER
a5690b16-a739-4bd7-a62a-dc4dc5c5de6c    Project1     670fd8ea-2ddb-377d-bd8b-587e7781f2b5
ccb0667b-5949-489a-8902-692ef2f31827    Project2     f1aa8430-7058-4f6c-a726-b75ddf6252eb
No of items :  2

Use the ID of the project from the previous step to enter the project context:

icav2 projects enter a5690b16-a739-4bd7-a62a-dc4dc5c5de6c

Find the pipeline you want to start from the CLI by obtaining a list of pipelines associated with your project:

icav2 projectpipelines list

ID                                      CODE					DESCRIPTION      
fbd6f3c3-cb70-4b35-8f57-372dce2aaf98    DRAGEN Somatic 3.9.5			The DRAGEN Somatic tool identifies somatic variants
b4dc6b91-5283-41f6-8095-62a5320ed092    DRAGEN Somatic Enrichment 3-10-4	The DRAGEN Somatic Enrichment pipeline identifies somatic variants which can exist at low allele frequencies in the tumor sample.
No of items :  2

Find the ID associated with your pipeline of interest.

Identify Input File Parameters

To find the input files parameter, you can use a previously launched projectanalyses with the input command.

Find the previous analyses launched along with their associated IDs:

icav2 projectanalyses list

ID                                      REFERENCE                                       CODE                    STATUS
3539d676-ae99-4e5f-b7e4-0835f207e425    kyle-test-somatic-2-DRAGEN Somatic 3_9_5        DRAGEN Somatic 3.9.5    SUCCEEDED
f11e248e-9944-4cde-9061-c41e70172f20    kyle-test-somatic-1-DRAGEN Somatic 3_9_5        DRAGEN Somatic 3.9.5    FAILED
No of items :  2

List the analyses inputs by using the ID found in the previous step:

icav2 projectanalyses input 3539d676-ae99-4e5f-b7e4-0835f207e425

CODE                                    NAMES                                                                   DATA ID
BED
CNV_B_Allele_VCF
CNV_Population_B_Allele_VCF
HLA_Allele_Frequency_File
HLA_BED
HLA_reference_file_(protein_FASTA)
Microsatellites_File
Normal_BAM_File
Normal_FASTQ_Files
Panel_of_Normals
Panel_of_Normals_TAR
Reference                               hg38_altaware_nohla-cnv-anchored.v8.tar                                 fil.35e27101fdec404fb37d08d9adf63307
Systematic_Noise_BED
Tumor_BAM_File
Tumor_FASTQ_Files                       HCC1187C_S1_L001_R1_001.fastq.gz,HCC1187C_S1_L001_R2_001.fastq.gz       fil.e1ec77f2647f45804fe508d9aecb19c4,fil.d89018f0c7784fc4b76708d9adf63307

This will return the Input File Codes, as well as the file names and data IDs of the associated data used to previously launch the pipeline

Identify Configuration Settings

Currently, this step for CWL requires the use of the ICA API to access the configuration settings of a project analyses that ran successfully. It is optional for Nextflow since the XML configuration file can be accessed in the ICA GUI.

Nextflow XML file parameters

Click the previous GUI run, and select the pipeline that was run. On the pipeline page, select the XML Configuration Tab to view the configuration settings.

In the "steps" section of the XML file, you will find various steps labeled with

<pd:tool code="map_align">

and subsequent labels of parameters with a similar structure

<pd:parameter code="enable_map_align" minValues="0" maxValues="1" classification="USER">
                    <pd:label>Enable Map/Align</pd:label>
                    <pd:description></pd:description>
                    <pd:optionsType>
                        <pd:option>true</pd:option>
                        <pd:option>false</pd:option>
                    </pd:optionsType>
                    <pd:value></pd:value>

The code should be used to generate the later command line parameters e.g.

--parameters enable_map_align:true

API based Configuration Settings (CWL or Nextflow)

Generate JWT Token from API Key or Basic login credentials
Instructions on how to get an API Key https://illumina.gitbook.io/ica/account-management/am-iam#api-keys
If your user has access to multiple domains, you will need to need to add a "?tenant=($domain)" to the request

curl -X 'POST' \
  'https://ica.illumina.com/ica/rest/api/tokens' \
  -H 'accept: application/vnd.illumina.v3+json' \
  -H 'X-API-Key: <APIKEY>' \
  -d ''

echo -ne 'testemail@testdomain.com:testpassword' | base64
    <BASE64UN+PW>

curl -X 'POST' \
  'https://ica.illumina.com/ica/rest/api/tokens' \
  -H 'accept: application/vnd.illumina.v3+json' \
  -H 'Authorization: Basic <BASE64UN+PW>' \
  -d ''

Response to this request will provide a JWT token {"token":($token)}, use the value of the token in further requests
Using the API endpoint /api/projects/{projectID}/analyses/{analysisId}/configurations to find the configuration file listing out all of required and optional parameters

curl -X 'GET' \
  'https://ica.illumina.com/ica/rest/api/projects/e501a0d5-f5e7-458c-a590-586c79bb87e0/analyses/3539d676-ae99-4e5f-b7e4-0835f207e425/configurations' \
  -H 'accept: application/vnd.illumina.v3+json' \
  -H 'Authorization: Bearer <Token>' \
  -H 'X-API-Key: <APIKEY>'

The response JSON to this API will have configuration items listed as

{
	"items": [{
		"name": "DRAGEN_Somatic__enable_variant_caller",
		"multiValue": false,
		"values": [
			"true"
		]
	}]
}

Create Launch Command CWL

Structure of the final command

icav2 projectpipelines start cwl $(pipelineID) --user-reference Plus input options

Input Options - For CLI, the entire input can be broken down as individual command line arguments

To launch the same analysis as using the GUI, use the same file ID and parameters, if using new data you can use the CLI command icav2 projectdata list to find new file IDs to launch a new instance of the pipeline Required information in Input - Input Data and Parameters

Command Line Arguments

This option requires the use of --type input STRUCTURED along with --input and --parameters

icav2 projectpipelines start cwl fbd6f3c3-cb70-4b35-8f57-372dce2aaf98 \
--user-reference kyle-test-somatic-9 \
--storage-size small \
--type-input STRUCTURED \
--input Reference:fil.35e27101fdec404fb37d08d9adf63307 \
--input Tumor_FASTQ_Files:fil.e1ec77f2647f45804fe508d9aecb19c4,fil.d89018f0c7784fc4b76708d9adf63307 \
--parameters DRAGEN_Somatic__enable_variant_caller:true \
--parameters DRAGEN_Somatic__enable_hrd:false \
--parameters DRAGEN_Somatic__enable_sv:true \
--parameters DRAGEN_Somatic__output_file_prefix:tumor \
--parameters DRAGEN_Somatic__enable_map_align:true \
--parameters DRAGEN_Somatic__cnv_use_somatic_vc_baf:false \
--parameters DRAGEN_Somatic__enable_cnv:false \
--parameters DRAGEN_Somatic__output_format:BAM \
--parameters DRAGEN_Somatic__vc_emit_ref_confidence:BP_RESOLUTION \
--parameters DRAGEN_Somatic__enable_hla:false \
--parameters DRAGEN_Somatic__enable_map_align_output:true

Responses:

Successful Response

analysisStorage.description           1.2 TB
analysisStorage.id                    6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
analysisStorage.name                  Small
analysisStorage.ownerId               8ec463f6-1acb-341b-b321-043c39d8716a
analysisStorage.tenantId              f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
analysisStorage.tenantName            ica-cp-admin
analysisStorage.timeCreated           2021-11-05T10:28:20Z
analysisStorage.timeModified          2021-11-05T10:28:20Z
id                                    51abe34a-2506-4ab5-adef-22df621d95d5
ownerId                               47793c21-75a6-3aa8-8147-81b354d0af4d
pipeline.analysisStorage.description  1.2 TB
pipeline.analysisStorage.id           6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
pipeline.analysisStorage.name         Small
pipeline.analysisStorage.ownerId      8ec463f6-1acb-341b-b321-043c39d8716a
pipeline.analysisStorage.tenantId     f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
pipeline.analysisStorage.tenantName   ica-cp-admin
pipeline.analysisStorage.timeCreated  2021-11-05T10:28:20Z
pipeline.analysisStorage.timeModified 2021-11-05T10:28:20Z
pipeline.code                         DRAGEN Somatic 3.9.5
pipeline.description                  The DRAGEN Somatic tool identifies somatic variants which can exist at low allele frequencies in the tumor sample. The pipeline can analyze tumor/normal pairs and tumor-only sequencing data. The normal sample, if present, is used to avoid calls at sites with germline variants or systematic sequencing artifacts. Unlike germline analysis, the somatic platform makes no ploidy assumptions about the tumor sample, allowing sensitive detection of low-frequency alleles.
pipeline.id                           fbd6f3c3-cb70-4b35-8f57-372dce2aaf98
pipeline.language                     CWL
pipeline.ownerId                      e9dd2ff5-c9ba-3293-857e-6546c5503d76
pipeline.tenantId                     55cb0a54-efab-4584-85da-dc6a0197d4c4
pipeline.tenantName                   ilmn-dragen
pipeline.timeCreated                  2021-11-23T22:55:49Z
pipeline.timeModified                 2021-12-09T16:42:14Z
reference                             kyle-test-somatic-9-DRAGEN Somatic 3_9_5-bc56d4b1-f90e-4039-b3a4-b11d29263e4e
status                                REQUESTED
summary
tenantId                              b5b750a6-49d4-49de-9f18-75f4f6a81112
tenantName                            ilmn-cci
timeCreated                           2022-03-16T22:48:31Z
timeModified                          2022-03-16T22:48:31Z
userReference                         kyle-test-somatic-9

Unsuccessful Response Pipeline ID not formatted correctly

400 Bad Request : ICA_API_004 : com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `java.util.UUID` from String "8f57-372dce2aaf98": UUID has to be represented by standard 36-char representation
 at [Source: (io.undertow.servlet.spec.ServletInputStreamImpl); line: 1, column: 983] (through reference chain: com.bluebee.rest.v3.publicapi.dto.analysis.SearchMatchingActivationCodesForCwlAnalysisDto["pipelineId"]) (ref. c9cd9090-4ddb-482a-91b5-8471bff0be58)

Check that the pipeline ID is correct based on icav2 projectpipelines list

File ID not found

404 Not Found : ICA_GNRC_001 : Could not find data with ID [fil.35dec404fb37d08d9adf63307] (ref. 91b70c3c-378c-4de2-acc9-794bf18258ec)

Check that the file ID is correct based on icav2 projectdata list

Parameter not found

400 Bad Request : ICA_EXEC_007 : The specified variableName [DRAGEN] does not exist. Make sure to use an existing variableName (ref. ab296d4e-9060-412c-a4c9-562c63450022)

Create Launch Command Nextflow

When using nextflow to start runs, the input-type parameter is not used, but the --project-id is required

Structure of the file command icav2 projectpipelines start nextflow $(pipelineID) --user-reference Plus input options

icav2 projectpipelines start nextflow b4dc6b91-5283-41f6-8095-62a5320ed092 \
--user-reference "somatic-3-10-test5" \
--project-id e501a0d5-f5e7-458c-a590-586c79bb87e0 \
--storage-size Small \
--input ref_tar:fil.35e27101fdec404fb37d08d9adf63307 \
--input tumor_fastqs:fil.e1ec77f2647f45804fe508d9aecb19c4,fil.d89018f0c7784fc4b76708d9adf63307 \
--parameters enable_map_align:true \
--parameters enable_map_align_output:true \
--parameters output_format:BAM \
--parameters enable_variant_caller:true \
--parameters vc_emit_ref_confidence:BP_RESOLUTION \
--parameters enable_cnv:false \
--parameters enable_sv:true \
--parameters repeat_genotype_enable:true \
--parameters enable_hla:false \
--parameters enable_variant_annotation:false \
--parameters output_file_prefix:Tumor

Response status can be used to determine if the pipeline was submitted successfully
status options: REQUESTED,SUCCEEDED,FAILED,ABORTED

analysisStorage.description           1.2 TB
analysisStorage.id                    6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
analysisStorage.name                  Small
analysisStorage.ownerId               8ec463f6-1acb-341b-b321-043c39d8716a
analysisStorage.tenantId              f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
analysisStorage.tenantName            ica-cp-admin
analysisStorage.timeCreated           2021-11-05T10:28:20Z
analysisStorage.timeModified          2021-11-05T10:28:20Z
id                                    9b8f9e84-2e7f-4adb-92e5-738b032c2328
ownerId                               47793c21-75a6-3aa8-8147-81b354d0af4d
pipeline.analysisStorage.description  1.2 TB
pipeline.analysisStorage.id           6e1b6c8f-f913-48b2-9bd0-7fc13eda0fd0
pipeline.analysisStorage.name         Small
pipeline.analysisStorage.ownerId      8ec463f6-1acb-341b-b321-043c39d8716a
pipeline.analysisStorage.tenantId     f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3
pipeline.analysisStorage.tenantName   ica-cp-admin
pipeline.analysisStorage.timeCreated  2021-11-05T10:28:20Z
pipeline.analysisStorage.timeModified 2021-11-05T10:28:20Z
pipeline.code                         DRAGEN Somatic Enrichment 3-10-4
pipeline.description                  The DRAGEN Somatic Enrichment pipeline identifies somatic variants which can exist at low allele frequencies in the tumor sample.
pipeline.id                           b4dc6b91-5283-41f6-8095-62a5320ed092
pipeline.language                     NEXTFLOW
pipeline.ownerId                      e9dd2ff5-c9ba-3293-857e-6546c5503d76
pipeline.tenantId                     55cb0a54-efab-4584-85da-dc6a0197d4c4
pipeline.tenantName                   ilmn-dragen
pipeline.timeCreated                  2022-03-07T18:09:38Z
pipeline.timeModified                 2022-03-08T17:47:20Z
reference                             somatic-3-10-test5-DRAGEN Somatic Enrichment 3-10-4-3df131a2-2187-489a-b9f8-140e3ec5efb0
status                                REQUESTED
tenantId                              b5b750a6-49d4-49de-9f18-75f4f6a81112
tenantName                            ilmn-cci
timeCreated                           2022-07-13T22:44:47Z
timeModified                          2022-07-13T22:44:47Z
userReference                         somatic-3-10-test5

PreviousAPI Beginner Guide NextMount projectdata using CLI

Last updated 8 months ago