Git-Sourced Pipelines

Introduction

Git can be used as source-control system for your pipelines. This offers portability, versioning and easy integration of existing public pipelines.

To use Git-sourced pipelines:

1. If you want to use a private repository, Create Git credentials with your personal access tokens.

2. Configure in ICA where on GitHub the pipeline is located, which tag or commit-id of that pipeline to use and select the credentials if it is a private repository or to prevent rate limiting.

Using Private Repositories

Git Credentials

To access private pipelines stored on Git, you need to enter the Git credentials in ICA. This is done at System Settings > Credentials > Create > Git Credential. If you want to use a public repository, you do not need credentials or an access token. However, there is a limit to how many anonymous calls can be made to a GitHub repository, so if that limit is exceeded, you will encounter repo lookup rate limit reached and will not be able to import the pipeline. For this reason, it is best practice to use Git credentials even for public repositories.

Fill out the following fields:

• Name—Provide a name to easily identify your Git credentials

• Git url—This is fixed to https://github.com

• Personal Access Token —the personal access token. Either an existing one, or you can generate one on Git. When you save and reopen your credentials, the personal access token is hidden. If you want to make changes to the credentials, you will need to re-enter the personal access token. This is to prevent the access token from being copied.

• Git username — This field appears after entering the personal access token and is extracted from your token.

A link is provided to create your personal access token on github.com in case you do not have a token.

The link to the personal access token on GitHub.com will open a pop-up window, so if it does not open, you might have a pop-up blocker active.

You can share the credentials you own with other users of your tenant. To do so, select your credentials at System Settings > Credentials and choose Share. Those users will be able to use your credentials, but they will not be able to see the actual access token.

Personal Access Token

Personal access tokens work in the same way as OAuth access tokens. You can create a new personal access token at https://github.com/settings/tokens/.

• The personal access token must have scope:repo (meaning all elements in the repo category must be selected).

• If you set an expiration date, all pipelines which use this personal access token will stop working on that date. For this reason, it is advised not to set an expiration date unless absolutely necessary.

Select Generate token at the bottom of the screen to get your personal access token. Copy the value over to a secure location after generating it.

Creating the pipeline in ICA

Regardless of using a public or a private repository, you need to create the connection in ICA to pick up pipeline the from Git.

1. Create a new pipeline at Projects > your_project > flow > Pipelines > Create > Nextflow > from Git.

2. Fill out the fields to access the pipeline. During configuration, a test will be performed to see if the repository can be reached with the provided credentials. Even if the tests fail, you can still save the entered values, this is to prevent you from being blocked from creating a configuration if the repository is unavailable. If you enter the URL for a private repository before you have entered the credentials, ICA will try to access it as if it were a public GitHub URL until you have entered the credentials.

Repository url (max 255 chars)	The url where the pipeline is located. For example https://github.com/nf-core/demo Do not use a / as last character of your path as this will result in a failed pipeline with Remote resource not found as error.
Pipeline name (max 255 chars)	Name of your pipeline. This is automatically filled out when the repository URL has been validated, but you can change the name.
Version number	The version of your pipeline. This is not the same as the tag or commit-id, but used to identify the version of your pipeline to people using it. This field can hold 25 characters.
Git credential	Only needed for private repositories. This is your previously configured Git credential. If you need Git credentials, but you have not configured them in the previous step, you can use the create button to configure them now.
Main file path (max 255 chars)	For NextFlow, If the main.nf is not in the root folder, edit the main file path to the main.nf file, otherwise keep main.nf.
Config file path (max 255 chars)	When configuring a nextflow pipeline, this is usually nextflow.config and located in the root folder. Edit this field to match your equivalent file and folder.
Schema file path (max 255 chars)	When configuring a nextflow pipeline, this is the file which describes the different parameters used by the workflow. This is usually nextflow_schema.json.
Version	Use the long commit-id or the tag to identify the version. ICA will automatically convert the tag to fill out the commit-id.

1. Select Create to start importing the pipeline. During import, the inputForm.json file is generated from the nextflow_schema.json file.

   • While the pipeline is being imported, you can still choose Abort import by opening the pipeline details if you decide not to import the pipeline.

   • If the import fails, you can investigate the reason by opening the pipeline and looking at the import summary on the details tab.

   • After import, the pipeline will be in draft status.

2. The details of your pipeline are now visible when opening the pipeline details tab. Here you can change the name and version number, provide a description, update the Nextflow version or edit other details as needed.

3. Proceed to the InputForm files tab to edit the generated inputForm.json file to match the inputs as needed by your pipeline. While you are editing the input form, you can use the simulate button to preview the input form.

4. Once all details are updated, save your pipeline. Analyses with the pipeline can now be executed in the same way as with regular pipelines.

To minimise the risk of triggering API-rate limiting on GitHub, the validations above are not performed when configuring pipelines via the API.

Example

Basic Git-Sourced Pipeline Example

Restrictions

The following restrictions apply to Git-sourced pipelines.

• Only JSON-based NextFlow are supported, no CWL pipelines or XML-based pipelines.

• Because translation from the generic nextflow pipeline to the ica-specific input forms takes place during import, you cannot use input forms created in ICA in your GitHub repository as the system will try to reconvert them.

• Only Github.com is supported as repository location.

• GitHub enterprise is not supported.

• Git-branches are not supported, only tag or commit-id for version control.

• On GitHub, repository names are limited to 100 characters