Nextflow
In order to run Nextflow pipelines, there are some process-level attributes within the Nextflow definition that must be considered.
Info | Details |
|---|---|
Nextflow version | 20.10.0 (default), 22.04.3 |
Executor | Kubernetes |
A user can select the Nextflow version while building a pipeline using Graphical User Interface (GUI) or API.
In the GUI, a user can choose the Nextflow version from a dropdown menu in the "Create new Nextflow pipeline" Information page under "Nextflow Version" field.
In the API, a user can select the Nextflow version by passing the version through the optional field "pipelineLanguageVersionId". When this value is not passed, a default Nextflow version will be used for the pipeline.
For each compute type, the
standard (default) or economy tiers can be selected, which corresponds to AWS on-demand or spot instance types, respectively. Annotation scheduler.illumina.com/lifecycle: standard and scheduler.illumina.com/lifecycle: economy.To specify a compute type for a Nextflow process, use the
pod directive within each process. Set the annotation to scheduler.illumina.com/presetSize and the value to the desired compute type. A list of available compute types can be found here. The default compute type, when this directive is not specified, is standard-small (2 CPUs and 8 GB of memory).pod annotation: 'scheduler.illumina.com/presetSize', value: 'fpga-small'
Oftentimes, there is a need to select the compute size for a process dynamically either based on user input or other factors. Given that the Kubernetes executor used on ICA does not use the
cpu and memory directives, it cannot be that way. Fortunately, with Nextflow flexibility in allowing any directives to be dynamic, we can also dynamically set the pod directive, as mentioned here. e.g.process foo {
// Assuming that params.compute_size is set to a valid size such as 'standard-small', 'standard-medium', etc.
pod annotation: 'scheduler.illumina.com/presetSize', value: "${params.compute_size}"
}
// Set the default pod
pod = [
annotation: 'scheduler.illumina.com/presetSize',
value : 'standard-small'
]
withName: 'big_memory_process' {
pod = [
annotation: 'scheduler.illumina.com/presetSize',
value : 'himem-large'
]
}
// Use an FPGA instance for dragen processes
withLabel: 'dragen' {
pod = [
annotation: 'scheduler.illumina.com/presetSize',
value : 'fpga-medium'
]
}
Inputs are specified via the input form XML. The specified
code in the XML will correspond to the field in the params object that is available in the workflow. Refer to the tutorial for an example.Outputs for Nextflow pipelines are uploaded from the
out directory in the attached shared filesystem. The publishDir directive can be used to symlink (recommended), copy or move data to the correct folder. Data will be uploaded to the ICA project after the pipeline execution completes.publishDir 'out', mode: 'symlink'
During execution, the Nextflow pipeline runner determines environment settings based on values passed via command-line or via a configuration file (see the Nextflow Configuration documentation). When creating a Nextflow pipeline, use the nextflow.config tab in the UI (also available via API) to specify a nextflow configuration file to be used when launching the pipeline.
nextflowconfig-0
If no Docker image is specified, Ubuntu will be used as default.
The following configuration settings will be ignored if provided as they are overridden by the system:
executor.name
executor.queueSize
k8s.namespace
k8s.serviceAccount
k8s.launchDir
k8s.projectDir
k8s.workDir
k8s.storageClaimName
k8s.storageMountPath
trace.enabled
trace.file
trace.fields
timeline.enabled
timeline.file
report.enabled
report.file
dag.enabled
dag.file
Last modified 3d ago