# Git-Sourced Pipelines

## Introduction

**Git** can be used as source-control system for your pipelines. This offers portability, versioning and easy integration of existing public pipelines.

To use Git-sourced pipelines:

1. If you want to use a private repository, Create [Git credentials ](#git-credentials)with your [personal access tokens](#personal-access-token).
2. [Configure in ICA](#creating-the-pipeline-in-ica) **where on GitHub the pipeline is located**, which **tag or commit-id** of that pipeline to use and select the **credentials** if it is a private repository or to prevent rate limiting.

## Using Private Repositories

### Git Credentials

To access private pipelines stored on Git, you need to enter the Git credentials in ICA. This is done at **System Settings > Credentials > Create > Git Credential**. If you want to use a public repository, you do not need credentials or an access token. However, there is a limit to how many anonymous calls can be made to a GitHub repository, so if that limit is exceeded, you will encounter *repo lookup rate limit reached* and will not be able to import the pipeline. For this reason, **it is best practice to use Git credentials even for public repositories.**

Fill out the following fields:

* **Name**—Provide a name to easily identify your Git credentials
* **Git url**—This is fixed to <https://github.com>
* **Personal Access Token** —the personal access token. Either an existing one, or you can [generate](#personal-access-token) one on Git. When you save and reopen your credentials, the personal access token is hidden. If you want to make changes to the credentials, you will need to re-enter the personal access token. This is to prevent the access token from being copied.
* **Git username** — This field **appears after entering the personal access token** and is extracted from your token.

<figure><img src="https://3193631692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MWUqIqZhOK_i4HqCUpT%2Fuploads%2Fgit-blob-f3b5f0173d1ac4dd603f582b8b11ce523f8c483f%2Fimage%20(127).png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

A link is provided to create your [personal access token](#personal-access-token) on [github.com](https://github.com/settings/tokens/new?description=Illumina%20Connected%20Analytics\&scopes=repo) in case you do not have a token.

{% hint style="info" %}
The link to the personal access token on GitHub.com will open a pop-up window, so if it does not open, you might have a pop-up blocker active.
{% endhint %}

You can **share the credentials** you own with other users of your tenant. To do so, select your credentials at **System Settings > Credentials** and choose **Share**. Those users will be able to use your credentials, but they will not be able to see the actual access token.

### Personal Access Token

Personal access tokens work in the same way as OAuth access tokens. You can create a new personal access token at <https://github.com/settings/tokens/>.

* The personal access token must have **scope:repo** (meaning all elements in the repo category must be selected).
* If you set an expiration date, all pipelines which use this personal access token will stop working on that date. For this reason, it is advised **not to set an expiration date unless absolutely necessary**.

Select **Generate token** at the bottom of the screen to get your personal access token. **Copy the value over to a secure location** after generating it.

## Creating the pipeline in ICA

Regardless of using a public or a private repository, you need to create the connection in ICA to pick up pipeline the from Git.

1. Create a new pipeline at **Projects > your\_project > flow > Pipelines > Create > Nextflow > from Git**.
2. Fill out the fields to access the pipeline. During configuration, a test will be performed to see if the repository can be reached with the provided credentials. *Even if the tests fail, you can still save the entered values, this is to prevent you from being blocked from creating a configuration if the repository is unavailable.*\
   If you enter the URL for a private repository before you have entered the credentials, ICA will try to access it as if it were a public GitHub URL until you have entered the credentials.

<figure><img src="https://3193631692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MWUqIqZhOK_i4HqCUpT%2Fuploads%2Fgit-blob-386c8a026c404fa7a22e801372bec7e5f12ef8ff%2Fimage%20(126).png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

<table data-header-hidden><thead><tr><th width="151"></th><th></th></tr></thead><tbody><tr><td>Repository url (max 255 chars)</td><td>The url where the pipeline is located.<br>For example https://github.com/nf-core/demo<br>Do not use a / as last character of your path as this will result in a failed pipeline with <em>Remote resource not found</em> as error.</td></tr><tr><td>Pipeline name</td><td>Name of your pipeline. This is automatically filled out when the repository URL has been validated, but you can change the name.</td></tr><tr><td>Version number</td><td>The version of your pipeline. This is not the same as the tag or commit-id, but used to identify the version of your pipeline to people using it. This field can hold 25 characters.</td></tr><tr><td>Git credential</td><td><strong>Only needed for private repositories</strong>. This is your previously configured <a href="#git-credentials-private-repository">Git credential</a>. If you need Git credentials, but you have not configured them in the previous step, you can use the create button to configure them now.</td></tr><tr><td>Main file path<br>(max 255 chars)</td><td>For NextFlow, If the <strong>main.nf</strong> is not in the root folder, edit the main file path to the main.nf file, otherwise keep main.nf.</td></tr><tr><td>Config file path<br>(max 255 chars)</td><td>When configuring a nextflow pipeline, this is usually <em>nextflow.config</em> and located in the root folder. Edit this field to match your equivalent file and folder.</td></tr><tr><td>Schema file path<br>(max 255 chars)</td><td>When configuring a nextflow pipeline, this is the file which describes the different parameters used by the workflow. This is usually <strong>nextflow_schema.json</strong>.</td></tr><tr><td>Version</td><td>Use the long <strong>commit-id</strong> or the tag to identify the version. ICA will automatically convert the tag to fill out the commit-id.</td></tr></tbody></table>

3. Select **Create** to start importing the pipeline. During import, the **inputForm.json** file is generated from the **nextflow\_schema.json** file.
   * While the pipeline is being imported, you can still choose **Abort import** by opening the pipeline details if you decide not to import the pipeline.
   * If the **import fails**, you can investigate the reason by opening the pipeline and looking at the **import summary on the details tab**.
   * After import, the pipeline will be in **draft** [status](https://help.ica.illumina.com/project/p-flow/f-pipelines/..#pipeline-statuses).
4. The details of your pipeline are now visible when opening the pipeline details tab. Here you can change the **name** and **version number**, provide a **description**, update the **Nextflow version** or edit other details as needed.
5. Proceed to the **InputForm files** **tab** to **edit the generated inputForm.json** file to match the inputs as needed by your pipeline. While you are editing the input form, you can use the **simulate** button to preview the input form.
6. Once all details are updated, **save your pipeline**. [Analyses](https://help.ica.illumina.com/project/p-flow/f-analyses) with the pipeline can now be executed in the same way as with regular pipelines.

{% hint style="info" %}
To minimise the risk of triggering API-rate limiting on GitHub, the validations above are not performed when configuring pipelines via the API.
{% endhint %}

## Example

{% content-ref url="git-sourced-pipelines-experimental/basic-git-sourced-pipeline-example" %}
[basic-git-sourced-pipeline-example](https://help.ica.illumina.com/project/p-flow/f-pipelines/git-sourced-pipelines-experimental/basic-git-sourced-pipeline-example)
{% endcontent-ref %}

## Restrictions

The following restrictions apply to Git-sourced pipelines.

* Only **JSON-based NextFlow** are supported, no CWL pipelines or XML-based pipelines.
* Because translation from the generic nextflow pipeline to the ica-specific input forms takes place during import, **you cannot use input forms created in ICA in your GitHub repository** as the system will try to reconvert them.
* **Only** [**Github.com**](http://github.com) is supported as repository location.
* GitHub enterprise is not supported.
* **Git-branches are not supported**, only tag or commit-id for version control.