In this tutorial, we will show how to create and launch a pipeline using the Nextflow language in Platform Core.

This tutorial references the Basic pipeline example in the Nextflow documentation.

Create the pipeline

The first step in creating a pipeline is to create a Project. In the example below, the project is named Getting Started.

Pipeline

After creating your project,

1. Open the project at Projects > your_project.

2. Navigate to the Flow > Pipelines view in the left navigation pane.

3. From the Pipelines view, click +Create > Nextflow > XML based to start creating the Nextflow pipeline.

In the Nextflow pipeline creation view, the Description field is used to add information about the pipeline. Add values for the required Code (unique pipeline name), description and size fields.

Nextflow Files

Next a Nextflow pipeline definition must be created. The pipeline in this example is a modified version of the Basic pipeline example from the Nextflow documentation.

The description of the pipeline from the linked Nextflow docs:

Some modifications are made to the Nextflow pipeline, you do not need to copy these modification by hand. Copyable code is provided below.

• Adding the container directive to each process with the desired ubuntu image. If no Docker image is specified, public.ecr.aws/lts/ubuntu:22.04_stable is used as default. If you want to use the latest image, use container 'public.ecr.aws/lts/ubuntu:latest'

• Adding the publishDir directive with value 'out' to the reverse process.

• Modifying the reverse process to write the output to a file test.txt instead of stdout.

• Creating a channel with the input file.

Setting Process Resources: For each process, you can use the memory directive and cpus directive to set the Compute Types. Platform Core will then determine the best matching compute type based on those settings. Suppose you set memory '10240 GB' and cpus 6, then Platform Core will determine you need the standard-large compute type.

Syntax example:

Navigate to the Nextflow files > main.nf tab to add the definition to the pipeline. Since this is a single file pipeline, we don't need to add any additional definition files. Paste the following definition into the text editor:

Input Form

Next create the input form used when launching the pipeline. This is done in the XML Configuration tab. Since the pipeline takes in a single FASTA file as input, the input form includes a single file input.

Paste the below XML input form into the XML CONFIGURATION text editor

On the left, you see the XML code, on the right, you can see the input form simulation which appears when you use the simulate button at the bottom.

Once the definition has been added and the input form has been defined, the pipeline is complete.

Click the Save button at the top right. The pipeline will now be visible from the Projects > your_project > Pipelines view within the project.

Launch the pipeline

Before launching the pipeline, upload a FASTA file to use as input. For this tutorial, use a public FASTA file from the UCSC Genome Browser. Download chr1_GL383518v1_alt.fa.gz and unzip yjr FASTA file to decompress it.

To upload the FASTA file to the project, navigate to Projects > your_project > Data. In the Data view, drag and drop the FASTA file from your local machine in the input section (2) in the browser. Once the file upload completes, the file record will show in the Data explorer. The file format should be auto-detected and be FASTA. If this is not the case, you can set it by hand by selecting the file and changing the format from the manage menu item.

Now that the input data is uploaded, we can proceed to launch the pipeline. Navigate to Projects > your_project > Flow > Analyses click on Start. Next, select your pipeline from the list.

In the Launch Pipeline view, the input form fields are shown along with some required information to create the analysis.

With the required information set, click Start Analysis.

Monitoring Analysis

After launching the pipeline, navigate to Projects > your_project > Flow > Analysis.

The analysis record will be visible from the Analyses view. The Status will transition through the analysis states as the pipeline progresses. It may take some time (depending on resource availability) for the environment to initialize and the analysis to move to the In Progress status. Once the pipeline succeeds, the analysis record will show Succeeded as status.

Once the analysis has succeeded, click the analysis details tab for more information.

From the analysis details view, the logs produced by each process within the pipeline are accessible via the Steps tab.

View Results

Analysis outputs are written to an output folder in the project with the naming convention {Analysis User Reference}-{Pipeline Code}-{GUID}. (1)

Inside of the analysis output folder are the files generated by the analysis processes written to the out folder. In this tutorial, the file test.txt (2) is written to by the reverse process. Navigating to the analysis output folder, opening the test.txt file details, and selecting the VIEW tab (3) shows the output file contents.

Use the download button (4) if you want to download the data to the local machine.

In this tutorial, we will demonstrate how to create and launch a simple DRAGEN pipeline using the Nextflow language in the Platform Core UI. More information about Nextflow on Platform Core can be found here. For this example, we will implement the alignment and variant calling example from this DRAGEN support page for Paired-End FASTQ Inputs.

Linking a DRAGEN bundle

You need a project in which the pipeline will reside. You can choose an existing project or create a new one. See the Projects page for information on how to create a project. For this tutorial, we will use a project called Getting Started.

Once you have selected or created your project, you need to link a DRAGEN bundle to it give the project access to the DRAGEN docker image. Open your project and navigate to Projects > your_project > Project settings > Details > Edit. From here, select the + symbol next to linked bundles and select a DRAGEN Demo Tool bundle to add to the project. For this tutorial, link DRAGEN Demo Bundle 4.0.3.

Once the bundle has been linked to your project, you can access the docker image by navigating to the main level and opening System Settings > Docker Repository. There, click the docker image dragen-ica-4.0.3. At the bottom of the screen, you will see the regions where this bundle is available.

Creating the pipeline

Select Projects > your_project > Flow > Pipelines. From the Pipelines view, click +Create > Nextflow > JSON based to start creating a Nextflow pipeline.

Details

In the Nextflow pipeline creation view, use the Details tab to add information about the pipeline. Add values for the required Code (pipeline name) and Description fields. Nextflow Version and Storage size defaults to preassigned values.

Main.nf

Next, add the Nextflow pipeline definition by navigating to the Nextflow files > main files > main.nf. You will see a text editor. Copy and paste the following definition into the text editor. Modify the container directive by replacing the current URL with the URL found in the docker image dragen-ica-4.0.3. (System Settings > Docker Repository > your_docker_image > Regions).

This pipeline performs the following actions:

1. Accepts one paired FASTQ, one compressed reference file and a sample name

2. Schedules a FPGA‑backed Kubernetes pod on Platform Core

3. Unpacks the reference to local scratch

4. Runs DRAGEN with variant calling

5. Uploads all outputs to Platform Core cloud storage

Refer to the Nextflow page for details on Platform Core-specific attributes within the Nextflow definition.

• To specify a compute type for a Nextflow process, use the pod directive within each process.

• Outputs for Nextflow pipelines are uploaded from the out folder in the attached shared filesystem. The publishDir directive specifies the output folder for a given process. Only data moved to the out folder using the publishDir directive will be uploaded to the Platform Core project after the pipeline finishes executing.

Input Form

Next, we create the input form used for the pipeline. This is done on the Inputform files tab. More information on the specifications for the input form can be found in Input Form page.

This pipeline takes two FASTQ files, one reference file and one sample_id parameter as input.

Paste the following JSON input form into the inputForm.json text editor.

Click the Simulate button (bottom left) to preview the launch form fields.

Click the Save button (top right) to save the changes.

Running the pipeline

Go to the projects > your_project > flow > pipelines > your_pipeline and click Start Analysis.

Fill in the required fields and click on Start Analysis button.

Results

You can monitor the run from the Projects > your_project > Flow > analysis page. Once the Status changes to Succeeded, you can click on the run to access the results.

Useful Links

• Illumina DRAGEN Documentation

• Official Nextflow documentation

Nextflow supports scatter-gather patterns natively through Channels. The initial uses this pattern by splitting the FASTA file into chunks to channel records in the task splitSequences, then by processing these chunks in the task reverse.

In this tutorial, we will create a pipeline which will split a TSV file into chunks, sort them, and merge them together.

Select Projects > your_project > Flow > Pipelines. From the Pipelines view, click the +Create > Nextflow > XML based button to start creating a Nextflow pipeline.

In the Details tab, add values for the required Code (unique pipeline name) and Description fields. Nextflow Version and

Nextflow CLI

In this tutorial, we will demonstrate how to create and launch a Nextflow pipeline using the Platform Core command line interface (CLI).

Installation

Please refer to these instructions for installing the Platform Core CLI. To authenticate, please follow the steps in the Authentication page.

Tutorial project

In this tutorial, we will create the Simple RNA-Seq pipeline in Platform Core, which includes four processes:

• index creation

• quantification

• FastQC

• MultiQC

We will also upload a Docker container to the Platform Core Docker repository for use within the pipeline.

The 'main.nf' file defines the pipeline that orchestrates various RNASeq analysis processes.

The script uses the following tools:

• Salmon: Software tool for quantification of transcript abundance from RNA-seq data.

• FastQC: QC tool for sequencing data

• MultiQC: Tool to aggregate and summarize QC reports

We need a Docker container containing these tools. For the sake of this tutorial, we will use the container from the original tutorial. You can refer to the section to build your own docker image with the required tools.

With in your computer, download the image required for this project using the following command.

docker pull nextflow/rnaseq-nf

Create a tarball of the image to upload to Platform Core.

Following are lists of commands that you can use to upload the tarball to your project.

Add the image to the Platform Core Docker repository

The uploaded image can be added to the Platform Core Docker repository from the graphical user interface.

Change the format for the image tarball to DOCKER:

1. Navigate to Projects > your_project > Data.

2. Check the checkbox for the uploaded tarball.

3. Click on Manage > Change format.

To add this image to the Platform Core Docker repository, first click on Projects to go back to the home page.

1. From the Platform Core home page, click on System Settings > Docker Repository > Create > Image.

2. This will open a new window that lets you select the region (US, EU, CA) in which your your project is and the docker image from the bottom pane.

3. Edit the Name field to rename it. For this tutorial, we will change the name to "rnaseq". Select the region, and give it a version number, and description. Click on "Save".

After creating a new docker image, you can click on the image to get the container URL (under Regions) for the nextflow configuration file.

Create a configuration file called "nextflow.config" in the same folder as the main.nf file above. Use the URL copied above to add the process.container line in the config file.

You can add a pod directive within a process or in the config file to specify a compute type. The following is an example of a configuration file with the 'standard-small' compute type for all processes. Please refer to the page for a list of available compute types.

The parameters file defines the pipeline input parameters. Refer to the or input for detailed information for creating correctly formatted parameters files.

An empty form looks as follows:

The input files are specified within a single dataInputs node with individual input file specified in a separate dataInput node. Settings (as opposed to files) are specified within the steps node. Settings represent any non-file input to the pipeline, including but not limited to, strings, booleans, integers, etc..

For this tutorial, we do not have any settings parameters but it requires multiple file inputs. The parameters.xml file looks as follows:

Use the following commands to create the pipeline with the above contents in your project.

If not already in the project context, enter it by using the following command:

icav2 enter <PROJECT NAME or ID>

Create pipeline using icav2 project pipelines create nextflow Example:

If you prefer to organize the processes in different folders/files, you can use --other parameter to upload the different processes as additional files. Example:

You can refer to page to explore options to automate this process.

Refer to for details on running the pipeline from CLI.

Example command to run the pipeline from CLI:

You can get the pipeline id under "ID" column by running the following command:

You can get the file ids under "ID" column by running the following commands:

Please refer to command help (icav2 [command] --help) to determine available flags to filter output of above commands if necessary. You can also refer to page for available flags for the icav2 commands.

For more help on uploading data to Platform Core, please refer to the page.