Nextflow Pipeline

In this tutorial, we will show how to create and launch a pipeline using the Nextflow language in ICA.

This tutorial references the Basic pipeline example in the Nextflow documentation.

Create the pipeline

The first step in creating a pipeline is to create a Project. In the example below, the project is named Getting Started.

Pipeline

After creating your project,

1. Open the project at Projects > your_project.

2. Navigate to the Flow > Pipelines view in the left navigation pane.

3. From the Pipelines view, click +Create > Nextflow > XML based to start creating the Nextflow pipeline.

In the Nextflow pipeline creation view, the Description field is used to add information about the pipeline. Add values for the required Code (unique pipeline name), description and size fields.

Nextflow Files

Next a Nextflow pipeline definition must be created. The pipeline in this example is a modified version of the Basic pipeline example from the Nextflow documentation.

The description of the pipeline from the linked Nextflow docs:

This example shows a pipeline that is made of two processes. The first process receives a FASTA formatted file and splits it into file chunks whose names start with the prefix seq_.

The process that follows, receives these files and it simply reverses their content by using the rev command line tool.

Some modifications are made to the Nextflow pipeline, you do not need to copy these modification by hand. Copyable code is provided below.

• Adding the container directive to each process with the desired ubuntu image. If no Docker image is specified, public.ecr.aws/lts/ubuntu:22.04_stable is used as default. If you want to use the latest image, use container 'public.ecr.aws/lts/ubuntu:latest'

• Adding the publishDir directive with value 'out' to the reverse process.

• Modifying the reverse process to write the output to a file test.txt instead of stdout.

• Creating a channel with the input file.

Setting Process Resources: For each process, you can use the memory directive and cpus directive to set the Compute Types. ICA will then determine the best matching compute type based on those settings. Suppose you set memory '10240 GB' and cpus 6, then ICA will determine you need standard-large ICA Compute Type.

Syntax example:

process iwantstandardsmallresources {
    cpus 2
    memory '8 GB'
    ...

Navigate to the Nextflow files > main.nf tab to add the definition to the pipeline. Since this is a single file pipeline, we don't need to add any additional definition files. Paste the following definition into the text editor:

#!/usr/bin/env nextflow
params.in = "$HOME/sample.fa"

// -----------------------------
// Processes
// -----------------------------

// Split the file
process splitSequences {
    container 'public.ecr.aws/lts/ubuntu:latest'

    input:
    path 'input_fa'

    output:
    path "seq_*"

    script:
    """
   awk '/^>/{f="seq_"++d} {print > f}' < ${input_fa}
    """
}

// Reverse the Sequence
process reverse {
    container 'public.ecr.aws/lts/ubuntu:latest'
    publishDir 'out'

    input:
    path x

    output:
    path "test.txt"

    script:
    """
    cat ${x} | rev > test.txt
    """
}

// -----------------------------
// Workflow block
// -----------------------------

workflow {
//     Create a channel with your input file
    sequences = Channel.fromPath(params.in)
    splitSequences(sequences) | reverse | view
}

Input Form

Next create the input form used when launching the pipeline. This is done in the XML Configuration tab. Since the pipeline takes in a single FASTA file as input, the input form includes a single file input.

Paste the below XML input form into the XML CONFIGURATION text editor

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<pd:pipeline xmlns:pd="xsd://www.illumina.com/ica/cp/pipelinedefinition">
    <pd:dataInputs>
        <pd:dataInput code="in" format="FASTA" type="FILE" required="true" multiValue="false">
            <pd:label>in</pd:label>
            <pd:description>fasta file input</pd:description>
        </pd:dataInput>
    </pd:dataInputs>
    <pd:steps/>
</pd:pipeline>

On the left, you see the XML code, on the right, you can see the input form simulation which appears when you use the simulate button at the bottom.

Once the definition has been added and the input form has been defined, the pipeline is complete.

On the Documentation tab, you can add additional information about your pipeline. This information will be presented under the Documentation tab whenever a user starts a new analysis on the pipeline.

Click the Save button at the top right. The pipeline will now be visible from the Projects > your_project > Pipelines view within the project.

Launch the pipeline

Before launching the pipeline, upload a FASTA file to use as input. For this tutorial, use a public FASTA file from the UCSC Genome Browser. Download chr1_GL383518v1_alt.fa.gz and unzip yjr FASTA file to decompress it.

To upload the FASTA file to the project, navigate to Projects > your_project > Data. In the Data view, drag and drop the FASTA file from your local machine in the input section (2) in the browser. Once the file upload completes, the file record will show in the Data explorer. The file format should be auto-detected and be FASTA. If this is not the case, you can set it by hand by selecting the file and changing the format from the manage menu item.

Now that the input data is uploaded, we can proceed to launch the pipeline. Navigate to Projects > your_project > Flow > Analyses click on Start. Next, select your pipeline from the list.

Alternatively you can start your pipeline from Projects > your_project > Flow > Pipelines > your_pipeline > Start analysis.

In the Launch Pipeline view, the input form fields are shown along with some required information to create the analysis.

With the required information set, click Start Analysis.

Monitoring Analysis

After launching the pipeline, navigate to Projects > your_project > Flow > Analysis.

The analysis record will be visible from the Analyses view. The Status will transition through the analysis states as the pipeline progresses. It may take some time (depending on resource availability) for the environment to initialize and the analysis to move to the In Progress status. Once the pipeline succeeds, the analysis record will show Succeeded as status.

This may take considerable time if it is your first analysis due to the required resource management.

Once the analysis has succeeded, click the analysis details tab for more information.

From the analysis details view, the logs produced by each process within the pipeline are accessible via the Steps tab.

View Results

Analysis outputs are written to an output folder in the project with the naming convention {Analysis User Reference}-{Pipeline Code}-{GUID}. (1)

Inside of the analysis output folder are the files generated by the analysis processes written to the out folder. In this tutorial, the file test.txt (2) is written to by the reverse process. Navigating to the analysis output folder, opening the test.txt file details, and selecting the VIEW tab (3) shows the output file contents.

Use the download button (4) if you want to download the data to the local machine.