Service Connector
ICA provides a Service Connector, which is a small program that runs on your local machine to sync data between the platform's cloud-hosted data store and your local computer or server. The Service Connector securely uploads data or downloads results using TLS 1.2. In order to do this, the Connector makes 2 connections:
A control connection, which the Connector uses to get configuration information from the platform, and to update the platform about its activities
A connection towards the storage node, used to transfer the actual data between your local or network storage and your cloud-based ICA storage.
This Connector runs in the background, and configuration is done in the Illumina Connected Analytics (ICA) platform, where you can add upload and download rules to meet the requirements of the current project and any new projects you may create.
The Service Connector looks at any new files and checks their size. As long as the file size is changing, it knows data is still being added to the file and it is not ready for transfer. Only when the file size is stable and does not change anymore will it consider the file to be complete and initiate transfer. Despite this, it is still best practice to not connect the Service Connector to active folders which are used as streaming output for other processes as this can result in incomplete files being transferred when the active processes have extended compute periods in which the file size remains unchanged.
The service connector will handle integrity checking during file transfer, which requires the calculation of hashes on the data. In addition, Transmission speed depends on the available data transfer bandwidth and connection stability. For these reasons, uploading large amounts of data can take considerable time. This can in turn result in temporarily seeing empty folders at the destination location since these are created at the beginning of the transfer process.
Creating a New Connector
Select Projects > your_project.
From the projects menu, select Project Settings > Connectivity.
Select + Request New.
Fill out the fields in the New Connector configuration page.
Unique name (required) — Enter the name of the connector.
Description (optional) — Enter the connector description.
Mode (required) — Specify if the connector can upload data, download data, both or neither.
Operating system (required) — Select your server or computer operating system.
Debug Information Accessible by Illumina (optional) - Illumina support can request connector debugging information to help diagnose issues. For security reasons, support can only collect this data if the option Debug Information Accessible by Illumina is active. You can choose to either proactively enable this when encountering issues to speed up diagnosis or to only activate it when support requests access. You can at any time revoke access again by deselecting the option.
Add any upload or download rules. See Connector Rules below.
Select Install and wait for the installer to download. An initialization key will be displayed in the platform now. Copy this value as it will be needed during installation.
Launch the installer after the download completes and follow the on-screen prompts to complete the installation, including entering the initialization key copied in the previous step. Do not install the connector in the upload folder as this will result in the connector attempting to upload itself and the associated log files.
Windows
Run the downloaded .exe file. During the installation, the installer will ask for the initialization key. Fill out the initialization key you see in the platform. \
The installer will create an Illumina Service Connector, register it as a Windows service, and start the service. That means, if you wait for about 60 seconds, and then refresh the screen in the Platform by using the refresh button in the right top corner of the page, the connector should display as connected. \
You can only install 1 connector on Windows. If for some reason, you need to install a new one, first uninstall the old one. You only need to do this when there is a problem with your existing connector. Upgrading a connector is also possible. To do this, you don’t need to uninstall the old one first. \
macOS
Double click the downloaded .dmg file. Double click Illumina Service Connector in the window that opens to start the installer. Run through the installer, and fill out the initialization key when asked for it. \
To start the connector once installed or after a reboot, open the app. You can find the app on the location where you installed it. The connector icon will appear in your dock when the app is running. \
In the platform on the Connectivity page, you can check whether your local connector has been connected with the platform. This can take 60 seconds after you started your connector locally, and you may need to refresh the Connectivity page using the refresh button in the top right corner of the page to see the latest status of your connector. \
The connector app needs to be closed to shut down your computer. You can do this from within your dock. \
Linux
Installations require Java SE 8 or 11. You can check this with ‘java –version’ from a command line terminal. With Java installed, you can run the installer from the command line using the command
bash illumina_unix_develop.sh
. \Depending on whether you have an X server running or not, it will display a UI, or follow a command line installation procedure. You can force a command line installation by adding a –c flag:
bash illumina_unix_develop.sh -c
. \The connector can be started by running
./illuminaserviceconnector start
from the directory in which the connector was installed.
Connector Rules
In the upload and download rules, you define different properties when setting up a connector. A connector can be used by multiple projects and a connector can have multiple upload and download rules. Configuration can be changed anytime. Changes to the configuration will be applied approximately 60 seconds after changes are made in ICA if the connector is already connected. If the connector is not already started when configuration changes are made in ICA, it will take about 60 seconds after the connector is started for the configuration changes to be propagated to the connector. The following are the different properties you can configure when setting up a connector. After adding a rule and installing the connector, you can use the Active checkbox to disable rules.
Below is an example of a new connector setup with an Upload Rule to find all files ending with .tar
or .tar.gz
located within the local folder C:\Users\username\data\docker-images
.
Upload Rules
An upload rule tells the connector which folder on your local disk it needs to watch for new files to upload. The connector contacts the platform every minute to pick up changes to upload rules. To configure upload rules for different projects, first switch into the desired project and select Connectivity. Choose the connector from the list and select Click to add a new upload rule and define the rule. The project field will be automatically filled with the project you are currently within.
Download Rules
When you schedule downloads in the platform, you can choose which connector needs to download the files. That connector needs some way to know how and where it needs to download your files. That’s what a download rule is for. The connector contacts the platform every minute to pick up changes to download rules. The following are the different download rule settings.
Advanced Rules
Select the chevron at the middle-bottom of the rule configuration to expand the following optional advanced download rule settings.
Shared Drives
When you set up your connector for the first time, and your sample files are located on a shared drive, it’s best to create a folder on your local disk, put one of the sample files in there, and do the connector setup with that folder. When this works, try to configure the shared drive.
Transfer to and from a shared drive may be quite slow. That means it can take up to 30 minutes after you configured a shared drive before uploads start. This is due to the integrity check the connector does for each file before it starts uploading. The connector can upload from or download to a shared drive, but there are a few conditions:
The drive needs to be mounted locally.
X:\illuminaupload
or/Volumes/shareddrive/illuminaupload
will work,\\shareddrive\illuminaupload
orsmb://shareddrive/illuminaupload
will not.The user running the connector must have access to the shared drive without a password being requested.
The user who runs the Illumina Service Connector process on the Linux machine needs to have read, write and execute permissions on the installation folder.
Update connector to newer version
Illumina might release new versions of the Service Connector, with improvements and/or bug fixes. You can easily download a new version of the Connector with the Download button on the Connectivity screen in the platform. After you downloaded the new installer, run it and choose the option ‘Yes, update the existing installation’.
Uninstall a connector
To uninstall the connector, perform one of the following:
Windows and Linux: Run the uninstaller located in the directory the connector was installed.
Mac: Move the Illumina Service Connector to your Trash folder.
Log files
The Connector has a log file containing technical information about what’s happening. When something doesn’t work, it often contains clues to why it doesn’t work. Interpreting this log file is not always easy, but it can help the support team to give a fast answer on what is wrong, so it is suggested to attach it to your email when you have upload or download problems. You can find this log file at the following location:
Common issues
Last updated