Release Notes

2024 March 7 - ICA v2.23.1

Fixed Issues

  • Fixed an issue where automated analysis after sequencing run in non-US regions may fail for certain analysis configurations

2024 February 28 - ICA v2.23.0

Features and Enhancements

  • Data Management

    • The --exclude-source-path flag has been added to the ‘project data download’ command so that subfolders can be downloaded to the target path without including the parent path

    • The system automatically re-validates storage credentials updated in the graphical UI

    • Added a new API endpoint to validate storage configurations after credentials are changed: /api/storageConfigurations/{storageConfigurationId}:validate

  • Notifications

    • Added support for multi-version notification event payloads corresponding to versioned API response models

  • Flow

    • (API) Improved the analysis-dto by adding a new POST search endpoint as a replacement for the search analysis GET endpoint. The GET endpoint will keep working but we advise using the new POST endpoint.

    • Improved analysis statuses to reflect the actual status more accurately

    • Parallelized analysis input data downloads and output data uploads to reduce overall analysis time

    • No scratch size is allocated if tmpdirMin is not specified

  • Cohorts

    • Performance improvements of the ingestion pipeline

    • Performance improvements to subject list retrieval

    • Increased the character limit of ingestion log messages to the user

Fixed Issues

  • Data Access & Management

    • Fixed an issue where the target user cannot see analysis outputs after a successful transfer of analysis ownership in BaseSpace Sequence Hub

    • Update the API Swagger documentation to include paging information for: /projects/{projectId}/samples/{sampleId}/data

    • Fixed an issue resulting in errors when creating a new bundle version

    • Fixed an issue where the GET API call with the ‘Sort’ parameter returns an error when multiple values are separated by commas followed by a space

    • Fixed an issue where adding the –eligible-link flag to the ‘projectdata list’ API endpoint caused other flags to not work correctly

    • Added cursor-based pagination for the ‘projectdata list’ API endpoint

    • Fixed an issue with the entitled bundles cards view where the region is cut off when the Status is not present

    • Fixed an issue where bundle filtering on categories did not work as expected

    • Fixed an issue where file copy across tenants did not work as expected

    • Added a cross-account permission check so that file copy jobs fail when the cross-account set up is missing instead of being retried indefinitely

    • Fixed an issue where ‘Get Projects’ API endpoint returns an error when too many projects are in the tenant

    • Fixed an issue where the UpdateProject API call (PUT /api/projects/{projectId}) returns an error when technical tags are removed from the request

    • Fixed an issue where users need to confirm they want to cancel an action multiple times when clicking the back button in the graphical UI

    • Fixed an issue where clicking into a new version of a bundle from the details view does not open the new version, and instead directs to the bundle card view

  • Flow

    • Fixed an issue where the analysis logs are returned in the analysis screen “outputs” section and included in the getAnalysisOutputs API response. The log output is no longer considered as part of the analysis outputs

    • Analysis history screen has been removed

    • Fixed an issue resulting in inability to retrieve pipeline files via the API when the pipeline is shared cross-tenant

    • Fixed an issue where the API endpoint to retrieve files for a project pipeline would not return all files for pipelines created via CLI or API

    • Fixed an issue where the API does not check the proprietary flag of a pipeline before retrieving or downloading the pipeline files

  • Base

    • The ‘Download’ button is available to download Base activity data locally (and replaces the non-functional ‘Export’ button for restricted bundles)

    • Fixed an issue resulting in missing ICA reference fields in table records if the file was loaded into the table with no metadata

    • Improved consistency of the references included in the scheduler

  • Bench

    • Users are now logged out from a terminal window opened in a workspace after a period of inactivity

    • Fixed an issue where permissions could not be enabled after a workspaces has been created

    • Fixed an issue where a Contributor could not start/stop a workspace

  • Cohorts

    • Fixed an issue where large genes with subjects with large sets of diagnostic concepts cause a 503 error

    • Fixed an issue where the value in tumor_or_normal field in the phenotype table in the database is not set properly for germline and somatic mutation data

    • Resolved a discrepancy between the number of samples reported when hovering over the needle plot and the variant list

Documentation

  • General updates and improvements to reflect changes in graphical user interface

  • Added information on how to add scratch space to all processes in a pipeline

  • Updated the list of addresses which need to be whitelisted for the service connector

  • Added the 500GB limit per file for the FUSE driver

  • Batch jobs which have a final state such as Failed or Succeeded are removed from the activity list after 7 days

  • Updated the list of Notification Subscriptions to include all possible notifications

  • For Nextflow version 20.10.10, using the ‘copy’ method in thepublishDir directive for uploading output files that consume large amounts of storage may cause workflow runs to complete with missing files. Added information on workarounds to prevent this from occurring.

  • Updated the CLI Command index to reflect addition of --exclude-source-path flag for ‘project data download’ command

  • Added a video tutorial to demonstrate the end-to-end user flow for planning a NovaSeq X run with BaseSpace Run Planner and performing the run with data storage and automated analysis in ICA

  • Updated the list of auto-launch analysis workflows in the Sequencer Reference table

2024 January 31 - ICA v2.22.0

Features and Enhancements

  • Users can now access the system via https://ica.illumina.com or https://ica.illumina.com/ica

  • Data Management

    • Users are now able to revalidate storage configurations in an Error state

    • Improved existing endpoints and added new endpoints to link and unlink data to a bundle or a project in batch:

      • POST /api/projects/{projectId}/dataUnlinkingBatch

      • GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}

      • GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items

      • GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items/{itemId}

  • Flow

    • Analyses started via the API can now leverage data stored in BaseSpace Sequence Hub as input

    • ICA now supports auto-launching analysis pipelines upon sequencing run completion with run data stored in BaseSpace Sequence Hub (instead of ICA)

    • Updated the API for creating pipelines to include "proprietary" setting, which hides pipeline scripts and details from users who do not belong to the tenant which owns the pipeline and prevents pipeline cloning.

  • Cohorts

    • Added support for partial matches against a single input string to the “Search subjects” flexible filtering logic

    • Users can now view an overview page for a gene when they search for it or click on a gene in the marker frequency charts

    • ICA Cohorts includes access to both pathogenic and benign variants, which are plotted in the “Pathogenic variants” track underneath the needle plot

    • Ingestion: UI notifications and/or errors will be displayed in the event of partially completed ingestions

    • Users can share cohort comparisons with any other users with access to the same project

Fixed Issues

  • General

    • Improved the project card view in the UI

    • Fixed an issue with user administration where changing the permissions of multiple users at the same time would result in users receiving Invalid OAuth access token messages

  • Data Access & Management:

    • Improved the error message when downloading project data if the storage configuration is not ready for use

    • Fixed an issue causing Folder Copy jobs to time out and restart, resulting in delays in copy operations

    • Fixed an issue where only the Docker image of the first restricted bundle that was added could be selected

    • Improved the performance of folder linking with "api/projects/{ProjectID}/dataLinkingBatch"

    • The URL for links for "post/api/bundles" endpoint can be up to 2048 characters long

    • Improved the error response when using offset-based paging on API responses which contain too much data and require cursor-based paging

    • Fixed an issue resulting in failures downloading data from CLI using a path

    • The correct error message is displayed if the user does not have a valid subscription when creating a new project

    • Fixed an issue where changing ownership of a project does not change previous owner access for Base tables

  • Flow:

    • Input parameters of pipelines are now displayed in the "label (code)" format unless there is no label available or the label equals the code, in which case only the code is shown

    • Fixed an issue where multiple folders were created upon starting new analyses

    • Fixed an issue preventing analyses from using inputs with BaseSpace v1pre3 APIs

    • Fixed an issue causing analyses with a specified output path to incorrectly return an error stating that the data does not exist

    • The following endpoint "/api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs" now supports using external data as input

    • Any value other than "economy" or "standard" for submitted analysis jobs will default to "standard" and use "standard"

    • The parameter to pass an activationcode is now optional for start-analysis API endpoints

  • Base:

    • Improved the display of errors in the activity jobs screen if a Meta Data schedule fails

    • If an error occurs when processing metadata a failed job entry will be added in the Base Activity screen

    • Fixed an issue where records ingested via schedules from the same file could be duplicated

    • Fixed an issue where exporting the view shared via bundle would show an error 'Could not find data with ID (fol. ....)'

    • Resolved a NullPointerException error when clicking on Format and Status filters in the details screen of a Schedule in the Results tab

    • Fixed an issue where a schedule download would fail when performed by different user than the initial user

  • Bench:

    • Fixed an issue when trying to query a Base table with a high limit within a workspace

    • Fixed an apt-get error when building images due to an outdated repository

    • Fixed an issue where a stopped workspace would display "Workspace paused" instead of "Workspace stopped"

    • Fixed an issue where large files (e.g., 150GB+) could not be downloaded to a fuse-driver location from a Workspace, and set the new limit to 500GB

  • Cohorts:

    • Fixed an issue where split Nirvana JSON files are not recognized during ingestion

    • Fixed an issue causing the UI hangs on large genes and returns a 502 error

    • Fixed an issue where OMOP files are not correctly converted to CAM data model, preventing OMOP data ingestions

    • Fixed an issue where OMOP large drug ingestions led to memory issues and preventing further drug data ingestion

    • Fixed an issue where users from a different tenant accessing a shared project could not ingest data

Documentation

  • General updates and improvements

  • Clarified that bundle assets can only be linked once to the same data set

  • Made it more explicit that the custom bucket name needs to be entered in the IAM configuration

  • Added information on how to define scratch space in CWL workflows.

  • Added explanation of how to query arrays in table data

  • Described analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes

  • Removed outdated information about creating new tabs from workspace description.

  • Added information about creating an OAuth access token

  • Added information about stopping Workspaces and removed information about pausing Workspaces, as pausing is no longer an option.

  • Added preliminary beginner's tutorial for ICA API usage

  • Added information about how to use data stored in Basespace as an input when starting an analysis via the API

  • Clarified that upload and download rights act independent of user roles

  • Updated the table which shows the regions ICA is hosted on to include all the available regions in alphabetical order

  • Updated instructions for creating an AWS IAM user policy to remove a permission which is not correct

  • Added language specifying that folders with the name '.' are not supported

  • Added information about how to query views from Data Catalogues using the Snowflake FLATTEN function

  • Updated the CLI release history

  • Updated the Nextflow DRAGEN tutorial to account for no default scratch space

2023 December 05 - ICA v2.21.0

Features and Enhancements

  • Flow:

    • Analysis logs (task stdout/stderr files) are now written to a folder named ‘ica_logs’ within the analysis output folder

    • Default scratch disk size attached to analysis steps reduced from 2TB to 0B to improve cost and performance of analyses. Pipelines created before ICA v2.21.0 will not be impacted

  • Notifications: Notifications can now be updated and deleted in externally managed Projects

  • API: Clarified on the Swagger page which sorting options apply to which paging strategy (cursor-based versus offset-based). Changed the default sorting behavior so that:

    • When no paging strategy is specified and no sort is requested, then cursor-based paging is default

    • When no paging strategy is specified and sort is requested, then offset-based paging is default

  • Cohorts

    • Procedure Search Box: Users can now access additional UI functionalities for Procedures

      • Users can now access Procedure codes from OMOP

      • Improved handling of drug codes across all reports, excluding Survival comparison

    • Ingestion

      • Users now have enhanced job warning log and API status improvements

      • Users now require download permissions to facilitate the data ingestion process

    • Fetch Molecular Files: Improved import – Users can now input a directory path and select sample files individually

    • Variant Type Summary: Users can now access a new variants tab that summarizes Variant type statistics per gene

    • Added sorting and filtering capabilities to report tables, such as variants observed in genes

    • Users can now view sample barcodes, replacing internal auto-increment sample IDs in the Structural Variants table within the Genes tab

    • “Search subjects” functionality improved with flexible filtering logic that now supports partial matches against a single input string

Fixed Issues

  • Data Management:

    • Fixed an issue with data copy via the CLI where the file was being copied to a subfolder of the intended location instead of the specified folder

    • Resolved an issue where browser upload hangs intermittently when creating data

    • Fixed an issue where the delete popup does not always disappear when deleting data

    • Fixed an issue where GetFolder API call returns 404 error if the Create and Get operations are performed 100ms apart

    • Fixed an issue where file copy would fail if the file was located at the root level of User’s S3 storage bucket

    • Fixed an issue causing data linked from externally managed projects to be incorrectly excluded from the list project data API response

    • Fixed an issue where User cannot use data URNs to identify the destination folder when interacting with copy data API endpoints

    • Bundles: Fixed an issue where clicking the back button before saving a new bundle leads to inconsistencies

  • Flow:

    • Fixed an issue where pipeline documentation is not scrollable when launching pipeline

    • Fixed an issue with logfiles of a task not being available for streaming while the task is still running

    • Fixed an issue where using the 're-run' button from the analysis page reverts the storage size selection to default

    • Fixed an inconsistency where the following two endpoints would show different analysis statuses:

      • GET /api/projects/{projectId}/analyses

      • GET /api/projects/{projectId}/analyses/{analysisId}

    • Improved performance issues with UI loading data records when selecting inputs for analysis

    • Fixed a caching issue which resulted in delays when running pipelines

    • Fixed an issue where back button for analysis or pipeline details does not always direct Users back to analysis or pipelines view, respectively

    • Fixed an issue where system performance is degraded when large batches (e.g., 1,000) of data are added as input to Analyses via the graphical UI. It is recommended to start Analyses with large numbers of input files via API

  • Base

    • Fixed an issue where enabling Base from a Base view other than Base Tables returned a warning message

    • Fixed an issue where Base access was not enabled when a bundle with tables is added to a project without Base (Base is automatically enabled so users can see the bundle's tables). However, access to the bundle's tables is revoked upon the deletion of Base, and was not granted again once Base was re-enabled

    • Fixed an issue where a Base job to load data into a table never finished because the file was deleted after the job started and before it finished. Now the job will end up in a Failed state

  • Cohorts

    • Fixed an issue where needle plot filtered out data points reappear when zooming in the exon when a filter is in place

    • Fixed an issue where users from a different tenant who accept a project share may encounter a failure at the final step of the data ingestion process

    • Fixed an issue where users can encounter intermittent errors when browsing and typing for a gene

    • Fixed an issue where the UI hangs on large genes and returns a 502 error

Documentation

  • General updates and improvements

  • Updated organization of Table of Contents

  • Data Management: Added explanation of the impact of billing mode when users bring their own S3 storage

  • Flow: Added guidance about the ica_logs subfolder in the analysis output folder

  • Flow: Updated default scratch space reference documentation

  • API: Documented how to archive or delete files programmatically using API endpoints

  • API: Added information about how the sort parameter is used to choose the default pagination

  • Cohorts: Updated documentation to reflect OMOP ingestion and UI search for Procedures. The documentation changes also reflect instruction on the simplified ingestion and comparison sharing

2023 November 9 - ICA v2.20.1

Fixed Issues

  • Data Management:

    • Fixed an issue where multiple folder copy jobs with the same destination may get stuck In Progress

    • Fixed an intermittent issue where tags on the target folder for a batch data update call are not set, but are set for all child data

  • Flow:

    • Fixed an issue causing intermittent pipeline failures due to an infrastructure error

2023 October 31 - ICA v2.20.0

Features and Enhancements

  • Navigation: If multiple regions are enabled for the same tenant, the region will be indicated in the waffle menu

  • Logging: Data transfers of BaseSpace Sequence Hub projects with data stored in ICA will be traced in ICA logs

  • Cohorts

    • Disease Search Box: Added support for specifying subjects by age of onset of disease(s)

    • Drug Search Box: Added a new query builder box for Drugs

      • Ingestion: Support for Drug, drug route, etc. attached to subjects

      • Cohorts building: Users can build cohorts by specifying drugs, drug route, etc.

    • Ingestion:

      • Combine different variant types during ingestion (small variants, cnv, sv)

      • Cohorts supports Illumina Pisces variant caller for hg19 VCFs

Fixed Issues

  • General:

    • Fixed an issue where the graphical UI hands with ha spinning wheel when saving or executing a command

    • Fixed an issue where rich text editor for Documentation tab on Pipelines, Tools, Projects and Bundles does not populate with correct styles in edit mode

  • Data Management:

    • Fixed an issue where multiple clicks on create data in Project API endpoint resulted in multiple requests

    • Fixed an issue where the secondary data selection screen could not be resized

    • A spinning wheel icon with ‘copying’ status is displayed at the folder level in the target Project when a folder is being copied. This applies to the actual folder itself and not for folders higher up in the hierarchy

    • Fixed an issue where API to retrieve a project data update batch is failing with 500 error when either the Technical or the User tags are updated during the batch update request

    • Fixed an issue where linking jobs fail to complete if other linking jobs are running

    • Improved performance for data transfer to support BaseSpace Sequence Hub Run transfers

    • Fixed an issue causing some folder copy jobs to remain in "Partially Succeeded" status despite being completed successfullyBundles: Fixed an issue where the URL and Region where a Docker image is available is not displayed for a Docker image Tool shared via an entitled Bundle

    • Fixed an issue where the folder copy job was getting stuck copying large amounts of big files

    • Fixed an issue where the folder counts were not matching expected counts after Data linking

    • Fixed an issue where delete data popup would occasionally not disappear after deleting data.

    • Fixed an issue with data copy where referencing data from another region would not result in immediate failure

    • Fixed issue where uploading a folder using the CLI was not working

    • Fixed an issue where a Docker image shared via an entitled Bundle can be added to another region

  • Workflows: Fixed an issue where workflow does not fail if BCL Convert fails for a BCL Convert-only run

  • Flow:

    • Improved performance when batches of data up to 1000 are added as input to an Analysis

    • Nextflow engine will return exit code 55 if the pipeline runner task is preempted

    • Fixed an issue where log files cannot be opened for any steps in an analysis while the analysis is in progress

    • Fixed an issue with concurrent updates on analysis

    • Fixed an issue where unknown data inputs in the XML of an analysis are not being ignored

    • The warning, close, and machine profile icons for Tools can now be seen in the graphical CWL pipeline editor

    • Fixed an issue where user cannot expand analysis output folder if user permissions change after starting analysis. Now, if a user has the correct permissions to start an analysis, that analysis should be able to finish correctly no matter the permissions at the time it succeeds

  • Base:

  • Fixed an issue switching back from template to Empty Table did not clear the fields

  • Data linked from an externally managed project can be added to Base Tables

  • Fixed an issue in the graphical UI where schema definition does not scroll correctly when many columns are defined

Documentation

  • General updates and improvements

  • Sequencer Integration: Added information that auto-launch from instrument is not currently compatible with MiSeq systems

  • Data Management

    • Documented an alternate method of adding files to samples and explained how to remove files from samples

    • Provided more guidance on data transfer options

    • Updated reference documentation for Data Copy functionality

    • Added a message to inform users that folder copy can be impacted in cases where a folder with the same name already exists in the target project

  • Flow

  • Described analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes

  • Updated ‘CWL: Scatter-Gather’ tutorial

  • Added documentation for analysis timeout

  • Base:

  • Added a new tutorial for creating a Python dashboard to access Base metrics via API

  • Added information about Snowflake user/role setup

  • Included sample TSV file in Base Basics tutorial

  • Bench: Updated Fuse Driver troubleshooting

  • Notifications: Updated Filtering section of Notifications page

  • CLI: Updated CLI release history

  • Reference

  • Updated Illumina Connected Software registration guide

  • Added reference documentation for data integrity recommended practices

2023 October 3 - ICA v2.19.0

Features and Enhancements

  • Data Management/API: Added a new endpoint available to change project owner

    • POST /api/projects/{projectId}:changeOwner { “newOwnerId”:”}

  • Data Management/API: Added a new endpoint to copy data from one project to another:

    • /api/projects/{projectId}/projectDataCopyBatch

  • Data Management/CLI: Added the ability to copy files and folders between projects in the UI and CLI. This includes support for copying data from projects with ICA-managed storage (default) to projects with S3-configured storage.

  • Flow/API: When starting an analysis via the API, you can specify the input files based on HTTP(s). When your analysis is done, you will see the URL corresponding to the inputs in the UI, but you will not be able to start an analysis from the UI using this URL

  • Flow: Removed the ability to start analyses from data and sample views in the UI where a single input is selected to start analyses in bulk

  • Flow/CLI: Duplicate analyses submitted via the CLI will be avoided

  • Flow/API: Added two new endpoints for workflow sessions:

    • Get /api/projects/{projectId}/workflowSessions

    • Get /api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs

  • Flow/API: Added a new endpoint to retrieve configurations from a workflow session

  • Flow/Autolaunch ICA Workflow Session and Orchestrated Analyses (launched by the workflow session) now saves outputs in an organized folder structure: /ilmn-analysis/<name_used_to_create_sequencer_run_output_folder>

Base: The Base module has a new feature called ‘Data Catalogue’. This allows you to add usage data from your tenant/project if that data is available for you.

  • Data Catalogue views will be available and can be used in Base to query on.

  • You will be able to preview and query Data Catalogue views through Base Tables and Query screens

  • The Data Catalogue will always be up to date with the available views for your tenant/project

  • Data Catalogue views cannot be shared through a Bundle

  • Data Catalogue views will also be available to team members that were added after the view was added

  • Data Catalogue views can be removed from the Base tables and corresponding project

  • By removing Base from a project, the Data Catalogue will also be removed from that project

Cohorts: Disease Search box

  • Cohorts now includes a disease search box to search for disease concepts. This replaces the disease concept tree explorer

  • Disease search box located under a Disease tab in main Query builder

  • Search box allows for a copy/paste action of codes to be processed as separate query elements. Currently, the feature is limited to a complete valid list

  • Each disease entered into the search box is displayed as a separate query item and can be set to include or exclude.

  • Diseases in search box can be used with boolean logic in cohort creation

  • Search box allows for an auto-complete of diagnosis concepts and identifiers

  • The disease filter is included in the cohort query summary on cohort page

Fixed Issues

  • Data Management: Data copy between ICA-managed projects and S3 storage configured projects is supported

  • Data Management: Fixed an issue where storage configurations matching ICA-managed buckets would cause volume records to get associated with the wrong storage configuration in the system

  • Data Management/API: The endpoint GET/api/projects/{ProjectID}/samples/{SampleID} correctly returns all the own samples and linked samples

  • Data Management/API: Improved handling of bulk update via API when concurrent deletion of file has occurred

  • Data Management/CLI: Fixed an issue where projectdata update tags would not update the tags

  • Flow: Fixed an issue resulting in failure to send a notification resulting in a failed workflow

  • Flow: Fixed an issue where one workflow session may override another when both are executed at the same time

  • Base: Fixed an issue where query download in JSON format returns an error

  • Base: Added a message in the UI when a query takes longer than 30 seconds to inform the user that the query is ongoing and can be monitored in the Activity view

  • Base: Added a section describing the Data Catalogue functionality

  • Bench: Fixed an issue where resizing the workspace to current size would prevent users from resizing for the next 6 hours

  • Cohorts: Fixed an issue where Gene Expression table does not display with TCGA data or for tenants with a hyphen (e.g., ‘genome-group’)

  • Cohorts: Fixed an issue where user had no way to delete a cohort comparison from a deleted cohort

  • Cohorts: Fixed an issue in the UI where multi-cohort needle plot tracks are overlapping

  • Cohorts: Fixed an issue causing failures during annotation step with ‘CNV’ data type when selection ‘GB=hg19’ and ‘CNV data’ for liftover; also observed with ‘SM data’ and ‘hg38’ without liftover (in APS1 and CAC1 regions) due to a ‘404 Not Found’ error.

  • CLI: Fixed an issue to support adding the server-url as a parameter instead of having the config set

Documentation

  • General updates and improvement

  • General: Added link to Illumina Knowledge resource

  • Data Management: Add documentation around storage configuration synchronization behavior

  • Data Management: Documented the new Data Copy functionality

  • Data Management: Provided more guidance on data transfer options

  • Data Management: Added a section describing Data Catalogue functionality

  • Data Management: Added guidance about not using file names beginning with '/'

  • Service Connector: Added troubleshooting guidance for service connector

  • Sequencer Integration: Updated analysis autolaunch reference docs

  • Sequencer Integration: Updated the NovaSeq 6000 analysis auto-launch tutorial

  • Flow: Improved the tutorial for Scatter-gather method in Nextflow

  • Flow: Improved tutorial for Nextflow: RNASeq Lift

  • Flow: Added a tutorial for converting open-source nf-core pipelines to ICA Nextflow pipelines

  • Flow: Updated reference docs for starting analyses

  • Flow: Improved tutorial for scatter-gather in CWL

  • Base: Improved descriptions and scenarios of Base tables and queries

  • Cohorts: Documented Cohorts Diagnosis Concept attribute search box functionality

  • CLI: Updated the CLI release history page with all required build files

  • API: Added tip for users experiencing API rate limiting

  • API: Added troubleshooting guide for GetTemporaryUploadCredentialsAsync failure error scenario

  • SDK: Included new reference to Illumina's ICA Python SDK resource available on GitHub.com

2023 September 14 - ICA v2.18.4

Fixed Issue

  • Fixed an issue uploading folders via the CLI

2023 September 8 - ICA v2.18.3

Fixed Issue

  • Fixed an issue causing CWL pipelines using Docker images that do not contain bash shell executable to fail.

2023 September 7 - ICA v2.18.2

Fixed Issue

  • Fixed an issue leading to intermittent system instability.

2023 September 6 - ICA v2.18.1

Fixed Issue

  • Cohorts: Issue fixed where GTEx plot is not available for tenants with a hyphen (e.g. ilmn-demo).

2023 August 31 - ICA v2.18.0

Features and Enhancements

  • Data Management: New API endpoints are available for:

    • Creation of a data update in bulk: POST/api/projects/{projectId}/dataUpdateBatch

    • A list of data updates for a certain project: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}

    • A list of items from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items

    • A specific item from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items/{itemId} Note: Batch updates include tags, format, date to be archived and date to be deleted

  • Data Management/API: The sequencing run information can be retrieved through its Id by using the API endpoint GET/api/sequencingRuns/{sequencingRunId}

  • Flow: Auto launch now supports BCL Convert v3.10.9 pipeline and both TruSight Oncology 500 v2 pipelines (from FASTQs)

  • Flow/API: getPipelineParameters API returns parameter type in response

  • Flow/API: Added endpoints to retrieve and update a project pipeline definition

  • Flow/API/CLI: Include "mountPaths" field in response for API and CLI command to retrieve analysis inputs

  • Flow: Removed "fpga-small" from available compute types. Pipelines using "fpga-small" will use the "fpga-medium"-equivalent compute specifications instead

  • Flow/API: New API endpoint available to request the analyses in which a sample is being used

  • Flow/API: When leaving activationCodeDetailId empty when starting an analysis, the best match activation code will be used

  • Flow: Analyses launched/tracked by BaseSpace Sequence Hub contain relevant BaseSpace information in analysis details view

  • Bench: A new Bench permission is being introduced: Administrator. This permission allows users to manages existing workspaces and create new workspaces

  • Bench: The Bench Administrator role allows you to create new Bench workspaces with any permissions even if you as a Bench administrator do not have these permissions. In that case, you can create and modify the workspace, but you cannot enter that workspace. Modifying is only possible when the workspace is stopped

  • Bench: As a Bench Contributor you are not allowed anymore to delete a Bench Workspace, you need the Bench Administrator role.

  • Cohorts: Users can now ingest raw DRAGEN bulk RNAseq results for genes and transcripts (TPM), with the option to precompute differential expression during ingestion

  • Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements

  • Cohorts: In multiple needle plot view, individual needle plots can now be collapsed and expanded

  • Cohorts: Pop-outs for needle plot variants now contain additional links to external resources, such as UCSC

  • Cohorts: For a given cohort, display a distribution of raw expression values (TPM per gene) for a selected attributes

  • Cohorts: Use of the Cohorts maintains session between core ICA and Cohorts iFrame to prevent unwanted timeouts

  • Cohorts: Cohorts displays structural variants that include or overlap with a gene of interest

  • API: Two new API endpoints added to accept Terms and Conditions on a bundle:

    • GET /api/bundles/{bundleId}/termsOfUse/userAcceptance/currentUser Returns you the time of acceptance when you, the current user, accepted the Terms & Conditions.

    • POST /api/bundles/{bundleId}/termOfUse:accept

  • API: Add temporary credentials duration to API documentation

  • Notifications: List of events to which you can subscribe contains new ICA notification containing analyses updates

  • Versioning: The ICA version can now be found under your user when you select "About"

  • Versioning/API: It is possible to retrieve system information about ICA, such as the current version through GET/api/systeminfo

  • Logging: When an action is initiated by another application, such as BaseSpace Sequence Hub, it will be traced as well in the ICA logs

Fixed Issues

  • Data Management/API: Improved error handling for API endpoint: DELETE/api/bundles/{bundleId}samples{sampleId}

  • Data Management: CreatorId is now present on all data, including subfolders

  • Data Management: Improved external linking to data inside ICA using deep linking

  • Data Management: Improved error handling when creating folders with invalid characters.

  • Data Management: Fixed an inconsistency for URN formats on output files from Analyses. This fix will apply only for analyses that are completed starting from ICAv2.18.0

  • Data Management: Improved resilience in situations of concurrent linking and unlinking of files and folders from projects

  • Data Management/API: Fixed an issue where the API endpoint GET /api/samples erroneously returned a 500

  • Data Management/API: API endpoint GET/api/projects/{projectId}/analyses now returns the correct list when filtering on UserTags whereas it previously returned too many

  • Data Management: It is only possible to delete a storage configuration if all projects that are using this storage configuration have been hidden and are not active projects anymore

  • Data Management: Improved accuracy of the displayed project data size. Prior cost calculations were accurate, but the project data size visualization included technical background data

  • Data Management/API: Improved retry mechanism for API endpoint to create folderuploadsession

  • Data Management/CLI: When an upload of a folder/file is done through the CLI, it returns the information and ID of the folder/file

  • Data Management: Fixed an issue where there is a discrepancy in number of configurations between Storage->Configurations and Configurations-> Genomics.Byob.Storage Configuration view

  • Collaboration: Fixed an issue where a user is presented with a blank screen when responding to a project invitation

  • Flow/API: Improved error handling when invalid project-id is used in API endpoint GET /api/projects/{projectId}/pipelines

  • Flow: When a spot instance is configured, but revoked by AWS, the pipeline will fail and exit code 55 is returned

  • Flow/API: Fixed an issue when an Analysis completed with error "incomplete folder session", the outputs of the Analysis are not always completely listed in the data listing APIs

  • Flow/API: Updated ICA Swagger Project > createProject to correctly state that the analysis priority must be in uppercase

  • Flow: Fix to return meaningful error message when instrument run ID is missing from Run Completion event during an auto launched analysis

  • Flow: Improved parallel processing of the same analysis multiple times

  • Base: Improved error handling when creating queries which use two or more fields with the same name. The error message now reads "Query contains duplicate column names. Please use column alias in the query"

  • Base: Fixed an issue where queries on tables with many entries fail with NullPointerException

  • Bench: Clarified that changes to Bench workspace size only take effect after a restart

  • Cohorts: Fixed issue where counts of subjects are hidden behind attribute names

  • Cohorts: Fixed issue where the state of checked files are not retained when selecting molecular files that are in multiple nested folders

  • Cohorts: Fixed issue where projects that contain files from linked bundles cause a time out, resulting in users not being able to select files for ingestion

  • Cohorts: Fixed an issue where the 'Import Jobs' page loaded within the Data Sets frame, depending on where the import was initiated

  • Cohorts: Fixed an issue in the Correlation plat where x-axis counts were hidden under attribute names

  • Fixed an issue where users were previously incorrectly signed out of their active sessions

Documentation

  • General updates and improvements

  • General: Added ICA Reference Terminology page

  • Data Management: Document how to change Project owner

  • Data Management: Updated Storage Size table for Compute on Pricing page to include XL, 2XL, 3XL

  • Data Management: Improved explanation of hiding Projects when connecting to your own S3 bucket for data storage

  • Data Management: Added default and shared explanation to storage documentation

  • Data Management: Updated documentation to reflect the temp credentials returned for AWS datastore is for 36 hours

  • Flow: Documented event for Analysis status transition

  • Flow: Added Nextflow best practices to reduce Analysis failures

  • Flow: Updated Pipeline activation code behavior explanation

  • Flow: Added documentation for specifying external docker images (i.e. Docker Hub)

  • Base/CLI: Added a tutorial for using Base via the CLI

  • Base: Updated documentation to mention that Schema does not contain type ‘variant’

  • Base: Documented Base query which joins to tables can result in duplicate columns

  • Base: Documented SnowSQL show view vs show grants

  • Bench: Documented Bench permission changes

  • Cohorts: Documentation updates:

    • RNASeq ingestion

    • Applicable user-supplied RNASeq visualizations – Raw Expression Values in GTEx-like table

    • NeedlePlot

    • Structure Variant list

  • CLI: Added documentation for CLI config settings

  • Logging: Documented event logging transfer of ownership

2023 August 3 - ICA v2.17.1

Fixed Issues

  • Fixed an issue causing analyses requesting FPGA compute resources to experience long wait times (>24h) or not be scheduled

2023 June 27 - ICA v2.17.0

Features and Enhancements

  • Data Management: Performance improvements for data link and unlink operations – Larger and more complex folders can now be linked in the graphical UI, and progress can be monitored with a new visual indication under Activity > Batch Jobs

  • Flow: Increased the allowed Docker image size from 10GB to >20GB

  • Flow/CWL: Added support for javascript expressions “ResourceRequirements” fields (i.e., type, size, tier, etc.) in CWL Pipeline definitions

  • Flow/API: Added support for using Pipeline APIs to query Pipelines included in Entitled Bundles (i.e., to retrieve input parameters)

  • Flow/API: Added support for providing S3 URLs as Pipeline data inputs when launching via the API (using storage credentials)

  • Flow/API: Added support for specifying multi-value input parameters in a Pipeline launch command

  • Bench: Project and Tenant Administrators are now allowed to stop running Workspaces

  • Notifications: Notifications are now available for batch job changes

  • Cohorts: Enhanced ingestion workflow to ingest RNAseq raw data from DRAGEN output into backend Snowflake database

  • Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements

  • Cohorts: Multi-Cohort Marker Frequency - Added Multi-Cohort Marker Frequency tab allowing users to compare expression data across up to four Cohorts at the gene level

  • Cohorts: Multi-Cohort Marker Frequency includes a pairwise p-value heat map

  • Cohorts: Multi-Cohort Marker Frequency - Includes frequencies for Somatic and Copy Number Variants

  • Cohorts: Tab added for a multi-cohort marker frequency analysis in cohort comparisons

  • Cohorts: Multi-Cohort Needle Plot - Added new tab in the Comparison view with vertically aligned needle plots per cohort for a specified gene, allowing collapsible and expandable individual needle plots

  • Cohorts: Additional filter logic added to multi-cohort needle plot

  • Cohorts: Improved DRAGEN data type determination during ingestion allowing for multiple variant type ingestion

  • Cohorts: Enhanced list of observed variants with grouped phenotypes and individual counts, including a column for total sample count; tooltips/pop-outs provide extended information

  • Cohorts: Updates to needle plot link outs

  • Cohorts: Improved the Comparison feature by optimizing API calls to handle subjects with multiple attributes, ensuring successful loading of the page and enabling API invocation only when the user selects or expands a section

  • Cohorts: Removed unused columns (genotype, mrna_feature_id, allele1, allele2, ref_allele, start_pos, stop_pos, snp_id) from annotated_somatic_mutations table in backend database

  • Cohorts: Refactored shared functionality for picking consequence type to reduce code duplication in PheWAS-plot and GWAS-plot components

  • Cohorts: Invalid comparisons on the Comparisons page are now grayed out and disabled This improvement prevents the selection of invalid options

  • Cohorts: Automatic retry of import jobs when there are failures accessing data from ICA API

Fixed Issues

  • Data Management: The content of hidden Projects can now be displayed

  • Data Management: Fixed the TimeModified timestamp on files

  • Bundles: Resolved issues when linking a large number of files within a folder to a Bundle

  • Flow: Single values are now passed as a list when starting an Analysis

  • Flow: Pipelines will succeed if the input and output formats specified on the pipeline level match at the Tool level

  • Flow: Fixed an issue causing Analysis failures due to intermittent AWS S3 network errors when downloading input data

  • Flow/CWL: Improved performance on output processing after a CWL Pipeline Analysis completes

  • Flow/UI: Mount path details for Analysis input files are now visible

  • Flow/UI: Improved usability when starting an Analysis by filtering entitlement options based o inputs selected and available entitlements

  • Flow/API: List of Analyses can now be retrieved via the API based on filters for UserReference and UserTags

  • Base: Fixed an issue where the Scheduler continues to retry uploading files which cannot be loaded

  • Bench: Resolved an issue when attempting to access Workspaces with multi-factor authentication (MFA) enabled at the Tenant-level

  • API: Improved error messaging for POST /api/projects/{projectId}/data/{dataId}:scheduleDownload

  • Navigation: Removed breadcrumb indication in the graphical UI

  • Cohorts: Fixed issue where Correlation bubble plot not showing for any projects intermittently

  • Cohorts: Fixed issue where importing Germline/hg19 test file did not load variants for a specific gene in the Needle plot due to missing entries in the Snowflake table

  • Cohorts: Fixed a bug causing an HTTP 400 error while loading the Cohort for the second time due to the UI passing "undefined" as variantGroup, which failed to convert to the VariantGroup Enum type

  • Cohorts: Fixed issue where scale (y-axis) of needle plot is changed even if value of sample count gnomAD frequency is not accepted

  • Cohorts: Fixed an issue where no data was generated in the Base Tables after a successful import job in Canada - Central Region (CAC1)

  • Cohorts: Fixed issue where long chart axis labels overlap with tick marks on graph

Documentation

  • General improvements and fixes

  • User Management: Added details about how users can manage an overview of their Projects

  • Data Management: Clarified the visibility of Batch jobs for different users in a Tenant

  • Bundles: Provided more details about Bundle linking and unlinking with data in folders

  • Flow: Described Analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes

  • Flow: Added an explanation for how external S3 files can be used as inputs to Pipelines

  • Bench: Added an explanation that Project/Tenant Administrators can stop running Workspaces

  • Bench: Added guidance for downloading data when using JupyterLab Notebooks

  • Cohorts: Added instructions for RNA-Seq data ingestion into Cohorts

  • Cohorts: Added reference documentation for multi-cohort marker frequency and needle plot comparisons

  • CLI: Updated the CLI command index

  • CLI: Updated the CLI release history page with all required build files

  • CLI: Added link to legacy ICA v1 CLI release history page

  • SDK: Included a new reference to Illumina’s ICA Python SDK resource available on Github

  • Network Settings: Updated the list of whitelisted URLs on https://help.ica.illumina.com/reference/r-networksettings

2023 May 31 - ICA v2.16.0

Features and Enhancements

  • Data Management: You can now select 'CYTOBAND' as format after file upload

  • Data Management: Added support for selecting the root folder (of the S3 bucket) for Projects with user-managed

  • Data Management: Added support for creating an AWS Storage Configuration with an S3 bucket with Versioning enabled

  • Auto-launch: Added technical tags for upstream BaseSpace Run information to auto-launched analyses

  • Auto-launch: Added support for multiple versions of BCL Convert for auto-launched analyses

  • Flow: Added support for '/' as separator in CWL ResourceRequirements when specifying Compute Type

  • Flow/API: The API to retrieve analysis steps now includes exit code for completed steps

  • Bench: Restricted Workspaces (Open or Restricted) always allow for access to Project Data within the Workspace

  • Bench: Restricted Bench workspaces have limited access through whitelisted URLs that are checked before entry

  • Bench: Restricted Bench Workspaces allow for Open or Restricted workspaces. Restricted workspaces do not have access to the internet except for user-entered whitelist URLs

Fixed Issues

  • Data Management: Upload for files names including spaces is now consistent for connector and browser upload. We do still advise not to use spaces in file names in general

  • Data Management: Fixed search functionality in Activity > Data Transfers screen

  • Data Management: Improved performance on opening samples

  • Data Management: Fixed an issue where reference data in download tab initiates an unexpected download

  • Data Management: Fixed intermittent issue where the Storage configuration within a Project can go into Error status and can block users from creating records such as folders and files

  • Data Management/API: Improved error handling for API endpoints: Delete/api/projects/ {projectId}/bundles/{bundleId} and POST/api/projects/{projectId}/bundles/{bundleId}

  • Data Management/API: Improved error handling for POST/api/projects/{projectId}/base:ConnectionDetails

  • Bundles: Fixed an issue where the Table view in Bundles is not available when linking to a new Bundle version

  • Bundles/API: Improved error handling for DELETE/api/bundles/{bundleId}/tools/{toolId} and POST/api/bundles/{bundleId}/tools/{toolId}

  • Bundles/API: Improved error message for POST/api/bundles/{bundleId}/samples/{sampleId}

  • Bundles: Fixed an issue where linking/unlinking a Bundle with Base Tables could result in errors

  • Service Connector: Improved error message for DELETE/api/connectors/{connectorId}/downloadRules/{downloadRuleId}

  • Notifications/API: Custom subscriptions with empty filter expressions will not fail when retrieving them via the API

  • Notifications/API: Improved error handling for POST/api/projects/{projectId}/notificationSubscriptions

  • Notifications/API: Improved notification for Pipeline success events

  • Flow: When the input for a pipeline is too large, ICA will fail the Analysis and will not retry

  • Flow: Fixed issue where analysis list does not search-filter by ID correctly

  • Flow: Improved error handling when issues occur with provisioning resources

  • Flow: When retry succeeds in a Nextflow pipeline, exit code is now '0' instead of '143'

  • Flow/API: Fixed an issue causing API error when attempting to launch an Analysis with 50,000 input files

  • Flow/API: Improved pipeline error code for GET/api/projects/{projectId}/pipelines/{pipelineId} when already unlinked pipeline Id is used for API call

  • Flow/API: Fixed an issue where Analyses could not be retrieved via API when the Pipeline contained reference data and originated from a different tenant

  • Flow/API: Fixed filtering analyses on analysisId. Filtering happens via exact match, so part of the Id won't work

  • Bench/CLI: Fixed issue where the latest CLI version was not available in Bench workspace images

  • Navigation: Updated URLs for Correlation Engine and Emedgene in the waffle menu

  • Authentication: Using POST /api/tokens:refresh for refreshing the JWT is not possible if it has been created using an API-key.

  • Authentication: Improved error handling when there is an issue reaching the authentication server

  • Authentication: Improved usability of "Create OAuth access token" screen

  • Cohorts: Issue fixed where CNV data converted from hg19 to hg38 do not show up in Base table views.

  • Cohorts: Issue fixed accounting for multiple methods of referring to the alternate allele in a deletion from Nirvana data.

  • Cohorts: Fixed intermittent issue where GWAS ingestions not working after Base enabled in a project.

Documentation

  • User Management: Improved explanation of Team functionality

  • Data Management: Added guidance on versioning Bring Your Own storage buckets

  • Data Management: Improved explanation on linking to data from other projects

  • Data Management: Described functionality of Secondary Data

  • Data Management: Updated guidance for file upload size

  • Data Management: Updated configuration guidance for Bring Your Own Storage bucket using S3

  • Data Management: Updated list of supported Data formats

  • Data Management/CLI: Added a tutorial for mounting project data using the CLI

  • Flow: Added a tutorial for building and running a CWL graphical pipeline

  • Flow: Included best practices for minimizing errors in Nextflow pipelines by adding retry logic

  • Base: Updated the Base schedule descriptions to include information about Metadata and Administrative schedules, as well about frequency of scheduled ingestions

  • Bench: Updated restrictions of available features when in the FUSE mount in a Workspace

  • Bench: Updated descriptions about Bench restrictions, including details about whitelisted URLs

  • CLI: Added CLI authentication best practices for setting up, validating, and resetting the configuration of a user's profile

  • API: Added information to launch analysis API spec on output mappings overwrite behavior when files already exist on the target project during analysis output

  • Network Settings: Updated Network Settings regional S3 URLs list

  • General updates and improvements

2023 May 2 - ICA v2.15.1

Fixed Issue

  • Fixed an issue causing incorrect empty storage configuration dropdown during Project creation when using the “I want to manage my own storage” option for users with access to a single region

2023 April 25 - ICA v2.15.0

Features and Enhancements

  • General availability of sequencer integration for Illumina sequencing systems and analysis auto launch

  • General usability improvements in the graphical interface, including improved navigation structure and ability to switch between applications via the waffle menu in the header

  • Storage Bundle field will be auto-filled based on the Project location that is being chosen if multiple regions are available

  • Event Log entries will be paged in the UI and will contain a maximum of 1,000 entries. Exports are limited to the maximum number of entries displayed on the page.

  • Read-only temporary credentials will be returned when you are not allowed to modify the contents of a file

  • Creating Project notifications for BaseSpace externally managed projects is now supported

  • The ICA UI will only allow selection of storage bundles belonging to ICA during Project creation, and the API will only return storage bundles for ICA

  • Flow: Allow attached storage for Pipeline steps to be set to 0 to disable provisioning attached storage and improve performance

  • Cohorts: GRCh37/hg19-aligned molecular data will get converted to GRCh38/hg38 coordinates to facilitate cross-project analyses and incorporating publicly available data sets.

  • API: Project list API now contains a parameter to filter on (a) specific workgroup(s)

  • API: Two new API endpoints are added to retrieve regular parameters from a pipeline within or without a Project context

Fixed Issues

  • Optimized price calculations resulting in less overhead and logging

  • Improved error handling:

    • during Project creation

    • of own storage Project creation failures.

    • to indicate connection issue with credentials

    • for graphical CWL draft Pipelines being updated during an Analysis.

  • Fixed an issue causing data records to remain incorrectly in Unarchiving status when an unarchive operation is requested in the US and Germany regions

  • Improved error messaging in cases where the AWS path contains (a) special character(s)

  • Fixed an issue causing errors when navigating via deep link to the Analysis Details view

  • CLI: Unarchival of folders is supported when archive or unarchive actions are not in progress for the folder

  • CLI: Improved error message to indicate connection issue with credentials

  • Flow: Fixed an issue causing incorrect naming of Analysis tasks generated from CWL ExpressionTools

  • Flow: Fixed an issue when cloning Pipelines linked from Entitled Bundles to preserve the original Tenant as the Owning Tenant of the cloned Pipeline instead of the cloning user’s Tenant

  • Flow: Fixed an issue causing outputs from CWL Pipelines to not show in the Analysis Details despite being uploaded to the Project Data Analysis output folder when an output folder is empty

  • Flow: When a Contributor starts an Analysis, but is removed afterwards, the Analysis still runs as expected

  • Flow: Fixed an issue where Analyses fail where Nextflow is run a second time

  • Flow: Fixed an issue causing API error when attempting to launch an Analysis with up to 50,000 input files

  • Flow: Fixed an issue causing degraded performance in APIs to retrieve Analysis steps in Pipelines with many steps

  • Flow: Fixed an issue causing Analysis failure during output upload with error “use of closed network connection”

  • Flow: Fixed an issue causing disk capacity alter log to not show when an Analysis fails due to disk capacity and added error message

  • Flow: Fixed an issue preventing cross-tenant users from being able to open a shared CWL pipeline

  • Base: Improved target Table selection for schedulers to be limited to your own Tables

  • Bench: Fixed an issue causing Workspaces to hang in the Starting or Stopping statuses

  • Cohorts: Now handles large VCFs/gVCFs correctly by splitting them into smaller files for subsequent annotation by Nirvana

  • API: Fixed returning list of unlinked data in a sample that was linked before in GET/api/projects/{projectId}/data

  • API: Fixed error for getSampleCreationBatch when using status filter

Documentation

  • Updated guidance for selecting Project Location and availability of Storage Bundles when configuring a new Project

  • Included information about receiving and using Entitled Bundles

  • Added guidance that a sample must be complete to be linked to a Bundle

  • Updated details about paging in the Event Log

  • Refined and expanded guidance for sequencer integration and analysis auto-launch, including adding a new tutorial for NovaSeq 6000 and expanding the sequencer reference table

  • Added tutorial for using scatter/gather for Nextflow pipelines

  • Refined Pipelines documentation around clone function

  • Added information that Base Tables must not contain reserved Snowflake terms in their definition

  • Added description for Copy Number Variants and Structural Variants tables now viewable in ICA Base for Cohorts

  • Added description for hg19 support as an alternative during ingestion for Cohorts

  • Updated CLI command index

  • Updated CLI binary and checksum files

  • Updated information about data archive pricing

  • Updated software registration guide

  • General updates and improvements

2023 March 28 - ICA v2.14.0

Features and Enhancements

  • Added a limit to Event Log and Audit UI screens to show 10,000 records

  • API: Parent output folder can be specified in URN format when launching a Workflow session via the API

  • Flow: Reduced Analysis delays when system is experiencing heavy load

  • Flow: Improved formatting of Pipeline error text shown in Analysis Details view

  • Flow: Users can now start Analyses from the Analysis Overview screen

  • Flow: Superfluous “Namespace check-0” step was removed to reduce Analysis failures

  • Flow: Number of input files for an Analysis is limited to 50,000

  • Flow: Auto launched Workflow sessions will fail if duplicate sample IDs are detected under Analysis Settings in the Sample Sheet

  • Base: Activity screen now contains the size of the query

Cohorts

  • Detect and Lift Genome Build: Cohorts documentation provides set-up instructions to convert hg19/GRCh37 VCFs to GRCh38 before import into Cohorts.

  • Attribute Queries: Improved the user experience choosing a range of values for numerical attributes when defining a cohort

  • Export Cohort to ICA Project Data: Improved the user experience exporting list of subjects that match cohort definition criteria to their ICA project for further analysis

  • Ingest Structural Variants into database

    • The Cohorts ingestion pipeline supports structural variant VCFs and will deposit all such variants into an ICA Base table if Base is enabled for the given project

    • Structural variants can be ingested and viewed in base tables

  • Needle Plot Enhancements

    • Users can input a numerical value in the Needle Plot legend to display variants with a specific gnomAD frequency percentage or sample count

    • The needle plot combines variants that are observed among subjects in the current project as well as shared and public projects into a single needle, using an additional shape to indicate these occurrences

    • Needle Plot legend color changes for Variant severity; pathogenic color coding is the same as the color coding in the visualization; differentiating hue between proteins and variants; and other color coding changes.

    • Needle plot tool tips that display additional information on variants and mutations are now larger and modal

    • The needle plot now allows to filter by gnomAD allele frequency and sample count in the selected cohort. Variants include links to view a list of all subjects carrying that variant and export that list.

  • Remove Samples Individually from Cohorts

    • Exclude individual subjects from a cohort and save the refined list

    • The subjects view allows users to exclude individual subjects from subsequent analyses and plots and save these changes

    • Subject exclusions are reset when editing a cohort

  • Subject Selection in Analysis Visualization: Users can follow the link for subject counts in the needle plot to view a list of subjects carrying the selected variant or mutation.

  • UI/UX: Start and End time points are available as a date or age with a condition attribute in the subject data summary screen.

Fixed Issues

  • Improved resilience against misconfiguration of the team page when there is an issue with Workgroup availability

  • Improved handling of multi-file upload when system is experiencing heavy loads

  • Fixed an issue to allow upload of zero-byte files via the UI

  • Fixed issue where other Bundles would not be visible after editing and saving your Bundle

  • Removed ‘IGV (beta)’ button from ‘View’ drop down when selecting Project Data in UI

  • API: Improved error handling for API endpoint: POST /api/projects/{projectId}/analysisCreationBatch

  • API: Improved performance of API endpoint: getbestmatchingfornextflow

  • Flow: Fixed an issue causing Analysis output mapping to incorrectly use source path as target path

  • Flow: Fixed an issue where the UI may display incorrect or invalid parameters for DRAGEN workflows which do not accurately show the true parameters passed. Settings can be confirmed by looking at the DRAGEN analysis log files.

  • Base: “Allow jagged rows” setting in the Scheduler has been replaced with “Ignore unknown values” to handle files containing records with more fields than there are Table columns

  • Base: Improved Base Activity view loading time

  • Base: Fixed an error message when using the API to load data into a Base Table that has been deleted

  • Bench: Fixed an issue resulting in incorrect Bench compute pricing calculations

  • Bench: Fixed an issue preventing building Docker images from Workspaces in UK, Australia, and India regions

  • Bench: Fixed an issue where /tmp path is not writeable in a Workspace

Cohorts:

  • Fixed issue where the bubble plot sometimes failed to display results even though the corresponding scatter plot showed data correctly.

  • The order of messages and warnings for ingestion jobs was not consistent between the UI and an error report sent out via e-mail.

  • The UI now displays any open cohort view tabs using shortened (“…”) names where appropriate

  • Issue fixed where ingestions with multiple errors caused halting to the ingestion queue.

  • The needle plot sometimes showed only one source for a given variant as opposed to all projects in which the variant had been observed in.

  • Issue fixed with unhandled genotype index format in annotation file to base database table conversion

  • Status updates via e-mail sometimes contained individual error messages or warnings without a text.

  • Fixed issue where items show in needle plot with incorrect numbering on the y-axis.

  • Fixed performance issue with subject count.

  • Widget bar-chart counts are intermittently cut off over four digits.

  • Fixed slowness when switching between tabs in query builder

Documentation

  • Refined and expanded reference documents for sequencer integration and analysis autolaunch, including new pages for sequencer reference and troubleshooting.

  • Added tutorial for accessing Base databases and tables via snowSQL.

  • Added tutorial for setting up a notification service and Lambda function in AWS for chaining pipelines.

  • Included guidance for setting override behavior for file output of advanced output mappings.

  • Added instructions to remove specific subjects from a cohort.

  • Documented cohort needle plot enhancements: List of subjects and specific frequency data entry.

  • Updated cohorts ingestion instruction including metadata documents, GWAS ingestion, OMOP ingestion, Structural Variant ingestion, and hg19 lift-over information.

  • Added reference documentation for using Cohorts attributes metadata template.

  • Added instruction for cohorts attribute query enhancement on numeric data.

  • Described selection of entitlements when creating a Project.

  • Described options for starting analyses from Analyses page, from Data page, and from a pipeline.

  • Updated FAQ with guidance on analysis run times.

  • Updated documentation about network settings for pipelines.

  • Described limitation on amount of input files for analyses.

  • Explained how to unlink a folder from a project.

  • Updated explanation for adding Terms of Use to a Bundle.

  • Included details for the Event Log.

  • Refined and expanded description of Base Activity screen.

  • Expanded description of Bench Activity screen.

  • Clarified details about S3 bucket behavior and Bring Your Own Storage bucket (BYOB) compatibility.

  • Provided guidance about deleting old versions of Bring Your Own Storage buckets.

  • Explained multi-factor authentication compatibility with Bench and provide workaround suggestion.

  • Updated CLI command index.

  • Fixed a bug in the CLI release history files and updated integrity check instructions.

  • General updates and improvements.

2023 March 23 - ICA v2.13.2

Fixed Issue

  • Fixed issue with BaseSpace Free Trial and Professional users storing data in ICA

2023 March 9 - ICA v2.13.1

Fixed Issue

  • Fixed an issue resulting in analysis failures caused by a Kubernetes 404 timeout error

2023 February 28 - ICA v2.13.0

Features and Enhancements

  • API: POST /api/projects/{projectId}/data:createDownloadUrls now supports a list of paths (in addition to a list of IDs)

  • Each tenant supports a maximum of 30,000 Projects

  • .MAF files are now recognized as .TSV files instead of UNKNOWN

  • Added VCF.IDX as a recognized file format

  • General scalability optimizations and performance improvements

Fixed Issues

  • After creating a new Bundle the user will be taken to the Bundle Overview page

  • Fixed an issue preventing the ‘Owning Project’ column from being used outside of Project

  • Fixed an issue which prevented changing the format of a file back to UNKNOWN

  • Fixed an issue causing inaccurate upload progress to be displayed for UI uploads. The Service Connector or CLI are recommended for large file uploads.

  • Fixed an issue showing an incorrect status for data linking batch jobs when data is deleted during the linking job

  • Fixed an issue allowing the region of a Project to be changed. Changing the region of a resource is not supported.

  • Strengthened data separation and improved resilience against cross-Project metadata contamination

  • Service Connector: Fixed an issue allowing download of a Service Connector when no operating system is set

  • Service Connector: Cleaned up information available on Service Connectors by removing empty address information fields

  • API: Fixed date formatting for GET /api/eventLog (yyyy-MM-dd’T’HH:mm:ss.SSS’Z’)

  • API: Fixed an issue where the GET users API was not case sensitive on email address

  • API: Fixed an issue causing the metadata model to be returned twice in PSOT /api/projects/{projectId}/samples:search

  • API: Fixed the listProjects API 500 response when using the pageoffset query parameter

  • API: The searchProjectSamples API returns Sample metadata for Samples shared via a Bundle

  • API: Fixed an issue causing createProjectDataDownloadUrls API 400 and 502 errors when server is under load

  • Flow: Fixed analysis failures caused by kubernetes 404 timeout error

  • Flow: Fixed an issue where Workflwos would prematurely report completion of an Analysis

  • Flow: Improved Pipeline retry logic to reduce startup delays

  • Flow: Fixed an issue where Nextflow pipelines were created with empty files (Nextflow config is allowed to be empty)

  • Flow: Removed the 1,000 input file limitation when starting an Analysis

  • Flow: Improved the performance of status update messages for pipelines with many parallel steps

  • Flow: Fixed an issue with overlapping fields on the Analysis Details screen

  • Flow: Deactivated the Abort button for Succeeded analyses

  • Base: Fixed an issue where Pipeline metadata was not captured in the metadata Table generated by the metadata schedule

  • Base: Error logging and notification enhancements

  • Bench: Fixed an issue where Workspaces could be started twice

  • Bench: Fixed an issue where the system checkpoint folder was incorrectly created in Project data when opening a file in a Workspace

Documentation

  • Added reference documentation for deep links to a folder.

  • Added descriptions of cursor and offset pagination when using the API.

  • Added guidance for Base JSON export.

  • Updated supported formats for Base table/query export.

  • Expanded explanation of default settings for the CLI.

  • Updated description of Base Activity screen.

  • Provided documentation for available Bench compute types.

  • Updated Nextflow DRAGEN tutorial to include details about how to obtain access to the docker image on ICA.

  • Improved documentation about pipeline activation codes.

  • Created new page for and updated list of available data file formats.

  • Updated guidance for requirements for adding assets to a bundle.

  • Explained limit of number of projects and bundles allowed per tenant.

  • Updated service connector reference docs.

  • Added guidance for Base schedule's optional parameters.

  • Explained progress details for service connector.

  • Added information about archiving or unarchiving a file.

2023 February 13 - ICA v2.12.1

Features and Enhancements

  • Analysis system infrastructure updates

Documentation

  • Added documentation for Analysis auto launch

2023 January 31 - ICA v2.12.0

Features and Enhancements

  • Added ability to refresh Batch Jobs updates without needing to leave the Details screen.

  • Projects will receive a job queuing priority which can be adjusted by an Administrator.

  • The text "Only showing the first 100 projects. Use the search criteria to find your projects or switch to Table view." when performing queries is now displayed both on the top and bottom of the page for more clarity.

  • API: Added a new endpoint to retrieve download URLs for data: POST/api/projects/{projectId}/data:createDownloadUrls

  • API: Added support for paging of the Project Data/getProjectDataChildren endpoint to handle large amounts of data.

  • API: Added anew endpoint to deprecate a bundle (POST /api/bundles/{bundleId}:deprecate)

  • API: If the API client provides request header "Accept-Encoding: gzip", then the API applies GZIP compression on the JSON response. This way the size of the response is significantly smaller which improves the download time of the response, resulting in faster end-to-end API calls. In case of compression the API also provides header "Content-Encoding: gzip" in the response, indicating that compression was effectively applied.

  • Flow: Optimized Analysis storage billing, resulting in reduced pipeline charges.

  • Flow: Internal details of a (non-graphical) pipeline marked ‘Proprietary’ will not be shared with users from a different tenant.

  • Flow: A new grid layout is used to display Logs for Analyses with more than 50 steps. The classic view is retained for analyses with 50 steps or less, though you can choose to also use the grid layout by means of a grid button on the top right on the Analysis Log tab.

  • CLI: Command to launch a CWL and Nextlfow Pipeline now contains the mount path as a parameter.

  • CLI: Version command now contains the build number.

  • CLI: Added support for providing the nextflow.config file when creating a Nextflow pipeline via CLI.

  • API: HTML documentation for aPipeline can now be returned with the following requests:

    • GET /api/pipelines/{pipelineId}/documentation/HTML

    • GET /api/projects/{projectId}/pipelines/{pipelineId}/documentation/HTML

  • API: Added a new endpoint for creating and starting multiple analyses in batch: POST /api/projects/{projectId}/analysisCreationBatch

  • Flow: Linking to individual Analyses and Workflow sessions is now supported by /ica/link/project//analysis/ and /ica/link/project//workflowSession/

  • Cohorts: Users can now export subject lists to the ICA Project Data as a file.

  • Cohorts: Users can query their ingested data through ICA Base. For users who already have ingested private data into ICA Cohorts, another ingestion will need to happen prior to seeing available database shares. Customers can contact support to have previously ingested data sets available in Base.

  • Cohorts: Correlation bubble plot counts now link to a subject/sample list.

Fixed Issues

  • Tooltip in the Project Team page provides information about the status of an invite

  • ‘Resend invite’ button in the Project Team page will become available only when the invite is expired instead of from the moment the invite is being send out

  • Folders, subfolders and files all contain information about which user created the data

  • Files and folders with UTF-8 character are not supported. Please look at the documentation on how to recover from it in case you already have used them.

  • Improved performance for creating or hiding a Project in a tenant with many Projects

  • Service Connector: Updated information in the Service Connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format"

  • Service Connector: Folders within a Bundle can be downloaded via the Service Connector

  • Service Connector: Upload rules can only be modified in the Project where they apply

  • Service Connector: A message describes when a file is skipped during upload because it already exists in the Project

  • Service Connector: Fixed an issue where opening the Connectivity tab occasionally results in a null pointer error

  • Service Connector: Fixed an issue causing excessive logging when downloading files with long file paths

  • Service Connector: Fixed an issue where the Service Connector log may contain spurious errors which do not impact data transfers

  • Existing storage configurations are displayed and accessible via API and UI

  • Newly added storage configurations do no longer remain in ‘Initializing’ state

  • Fixed error when creating a volume configuration with more than 63 characters

  • Clicking on a Data folder in flat mode will now open the details of the folder

  • Only Tools in Released state can be added to a Bundle

  • Fixed issue preventing new Bundle versions to be created from Restricted Bundles

  • Deprecated Bundles are displayed upon request in card and table view

  • Bundles view limited to 100 Bundles

  • API: Fixed the API spec for ProjectDataTransfer.getDataTransfers

  • API: Fixed an issue with the projectData getChildren endpoint which returned incorrect object and pagination

  • API: Fixed an issue where multiple clicks on Create sample batch API endpoint resulted in multiple requests

  • API: POST /api/projects/{projectId}/data/{dataId}:scheduleDownload can now also perform folder downloads

  • API: Improved information on the Swagger page for GET /api/pipelines, GET/api/projects/{projectId}/pipelines, and GET/api/projects/{projectId}/pipelines/{pipelineId}

  • API: Fixed and issue when a user provides the same input multiple times to a multi-value input on an analysis run, that input is only passed to the pipeline once instead of multiple times: POST /api/projects/{projectId}/analysis:nextflow

  • CLI: Copying files in the CLI from a local directory on MacOS to your Project can result in both the desired file and the metadata file (beginning with ‘./’) being uploaded. The metadata file can safely be deleted from the Project

  • CLI: Hardened protection against accidental file overwriting

  • CLI: Improved handling for FUSE when connection to ICA is lost

  • CLI: icav2 projectdata mount –list shows updated list of mounted Projects

  • CLI: Paging improvements made for project list, projectanalyses list, and projectsdata list

  • CLI: When there is no config or session file the user will not be asked to create one for icav2 config reset and icav2 config get

  • CLI: Fixed an issue where Bundle data could not be seen through FUSE in Bench

  • CLI: Fixed an error message when missing config file upon entering the Project context

  • CLI: The unmount is possible without a path and will work via the stored Project ID or with a directory path resulting in an unmount of that path

  • CLI: Fixed an error when creating a Pipeline using URN for Project identifier

  • CLI: Attempting to delete a file from an externally-managed project returns an error indicating this not allowed

  • CLI: Fix to delete session file when config file is not detected

  • CLI: Paging option added to projectsamples list data

  • CLI: Fixed “Error finding children for data” error in CLI when downloading a folder

  • CLI: projectdata list now returns the correct page-size results

  • Flow: Fixed handling of special characters in CWL pipeline file names

  • Flow: Fixed an issue where task names exceeding 25 characters cause analysis failure in CWL pipelines

  • Flow: Fixed an issue which prevented requests for economy tier compute

  • Flow: Fixed an issue limiting CWL workflow concurrency to two running tasks

  • Flow: Fixed an issue where analysis file inputs specified in the input.json with ‘location’set to an external URL cause to CWL pipelines to fail

  • Flow: Fixed an issue resulting in out of sync Pipeline statuses

  • Flow: Improved Nextflow engine resiliency, including occurrences where Nextflow pipelines fail with ‘pod 404 not found’ error

  • Flow: Fix issue with intermittent system runtime failures incorrectly causing analysis failures

  • Flow: Fixed an issue where links to Analysis Details returned errors

  • Flow: Enabled scrolling for Pipeline documentation

  • Flow: Improved performance for handling analyses with large numbers of inputs

  • Flow: Improved handling of hanging Analyses

  • Flow: Improved error messages for failed Pipelines

  • Flow: Added documentation on how to use XML configuration files for CWL Pipelines

  • Flow: Duplicate values for multi-value parameters are no longer automatically removed

  • Flow: Correct exit code 0 is shown for successful Pipeline steps

  • Base: Fixed an issue so that only users with correct permissions are allowed to retrieve a list of Base tables

  • Base: Fixed an issue with metadata scheduler resulting in a null pointer

  • Base: An empty Table description will not return an error when requesting to list all Tables in a Project

  • Base: Jobs failed with an error containing 'has locked table' are not shown on the Base Job activity list. They can be displayed by selecting the 'Show transient failures' checkbox at Projects > Activity > Base Jobs.

  • Base: Users can see Schedulers and their results for the entire tenant if created by a tenant administrator in their project, but not create, edit or run them

  • Base: Fixed an issue preventing data format change in a schedule

  • Base: Fixed an issue preventing exporting data to Excel format

  • Bench: Improved handling to prevent multiple users in a single running Workspace

  • Bench: Fixed an issue causing Workspaces to be stuck in "Starting" state

  • Bench: Fixed an issue where usage does not showing up on usage CSV-based report

  • Bench: Fixed an issue where Bundle data could not be seen via the Fuse driver

  • Bench: Users can now consistently exit Workspaces with a single click on the ‘Back’ button.

  • Bench: After leaving a Workspace by clicking on the ‘Back’ button, the Workspace will remain in a ‘Running’ state and become available for a new user to access

  • Bench: Workspaces in a ‘Stuck’ state can be manually changed to ‘Error’ state, allowing users to restart or delete them

  • Cohorts: Fixed issue where file system cleanup not occurring after delete.

  • Cohorts: Fixed sign in and authentication issues in APN1 region.

  • Cohorts: Fixed issue where gene filter missing when editing a cohort and removing the edited filter and cancelling. The filter was preserved and should not have been.

  • Cohorts: Fixed issue where users see an application tile in the Illumina application dashboard selection screen called "Cohort Analysis Module".

  • Cohorts: Correlation: Fixed issue, Data type selections shows half when loading the search result

  • Cohorts: Fixed issue, Users will see an application tile on the Connected Platform home page screen called “Cohort Analysis Module” if the Cohorts module is added to the domain. Users should not enter the ICA Cohorts through this page. They should enter through ICA."

Documentation

  • Added guidance on using Cohorts data in ICA Base.

  • Included video overview of Cohorts Multiomics Cancer workflow.

  • Updated steps for Cohorts ingestion.

  • Updated steps for Cohorts analysis.

  • Provided a command index for the ICA v2 CLI.

  • Expanded explanation on default settings for the CLI.

  • Clarified details about hidden metadata files behavior during CLI copy for MacOS users.

  • Added information about standard and economy tier selection for compute types.

  • Updated information in the service connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format".

  • Expanded guidance for XML-based input form required of pipelines defined using the "Code" mode.

  • Expanded guidance on importing external docker images in ICA when running a pipeline.

  • Included information about tile/grid view for analysis logs.

  • Added guidance about balancing analysis queuing priorities when setting up a Project.

  • Updated reference docs for workflow session events.

  • Provided details about price-per-sample and zero-cost pipelines.

  • Added details about which data formats are compatible with UI Preview.

  • Updated information about Batch Jobs.

  • Added details about syntax for linking to individual analyses and workflow sessions.

  • Described pipeline Proprietary flag available.

  • Added guidance about overriding CWL workflow requirements.

  • Explained how to add assets to a bundle.

2022 December 13 - ICA v2.11.0

Features and Enhancements

  • New notifications added for data upload, data created, and analyses with ‘aborted’ state

  • Flow: Progress on batch jobs can now be monitored in the Activity screen under the ‘Batch Jobs’ tab

  • Base: Shared database activity is available in the Base Activity Logging Screen

Cohorts:

  • Enhanced Attribute Queries: Filter logic now allows users to include or exclude filter criteria.

  • Ingestion Wizard: Human readable project name added to email notification of ingestion status

  • Ingestion Wizard: Notification email now includes the project name and job failure details

  • Ingestion Wizard: File lists are now sortable and filterable by type

  • Ingestion Wizard: UI enhancements on ingestion wizard selections

  • Ingestion Wizard: Based on data type chosen, the list of files for ingestion will be filtered for easier navigation of files to ingest

  • Share Cohorts with Collaborators - Cohort definitions are now sharable in a project

  • Statistical significance on Cohort Analysis - A p-value calculation is added to the Survival Comparison data points

Fixed Issues

  • Resolved an error when creating a Bucket Name with more than 63 characters

  • Fixed an issue causing failure when attempting to link "complex" folders (i.e., contains 100 sub-folders each with sub-folders) via GUI throws a “…transaction is not active…" error

  • Fixed an issue causing intermittent failure of "complex" folder (i.e., contains 100 sub-folders each with sub-folders) downloads

  • Fixed an issue so that the API to create data download URL correctly returns a 503 instead of a 404 response when the underlying system is experiencing network-related issues

  • Fixed an issue where the user is not redirected to the Project overview view upon clicking ‘Hide a Project’

  • Fixed an issue which prevented linking and unlinking multiple Bundles to and from a Project in a single operation via the GUI

  • Fixed an issue to show records in Bundle Data Transfers when data from a Bundle is accessed via View Tab and when downloaded

  • Fixed an issue showing stack trace when attempting to add Secondary Data to Data when not allowed

  • Fixed issue where the File Upload selection disappears after navigating back to Data after navigating to the Data History Tab

  • Fixed an issue where create folder session and get upload URL APIs are allowed for data accessible via an entitled Bundle

  • Fixed an issue where OpenAPI spec for delete project bundle API does not include proper "accept" header when using "Try It" functionality

  • Fixed an issue where write-access can be acquired for data accessible via an entitled Bundle

  • Fixed an issue where Project owner of an externally managed Project could not open Analysis Overview

  • Fixed an issue where multiple OS versions of the Service Connector are incorrectly downloaded at once when a single version is selected

  • Fixed an issue when selecting the "empty" option in the Operating System drop-down when downloading a connector

  • Fixed an issue where users with Contributor role and no data upload access can add data to samples without error

  • Added an event log entry for Service Connector log uploads initiated by Illumina Support

  • Flow: Fixed an issue to support Aborting an Analysis immediately after starting it

  • Flow: Fixed an issue when publishDir in Nextflow pipelines contains symlinks to data, those data are not included when uploading analysis outputs to a Project

  • Flow: Fixed an issue preventing closure of tabs in the Logs section for technical steps in the Analysis Details view

  • Flow: Cancel button added to the define CWL Pipeline selection dialogue window, and the order of buttons has been changed

  • Fixed an issue causing pipelines to stall for hours after all steps are completed

  • Flow: Fixed an issue where Nextflow pipelines fail with ‘pod 404 not found’ error

  • Base: Linked files from a Bundle can be inserted in a Base table

  • Base: Fixed an error on previewing Base tables that have been shared from a Bundle

  • Base: Fixed an issue where Base API to retrieve a list of Base jobs is not paging correctly

  • Bench: Fixed an issue where Bench Workspaces could not be opened with a double-click

  • Bench: After the “Save and Start” of a new Bench Workspace, the breadcrumb shows the actual name of the Workspace

  • Bench: Clarified error messages when creating new Workspaces with file permissions

  • Cohorts: Issue fixed where cohort is created during filtering process for RNA Expression on Apply

  • Cohorts: Issue fixed where order of items in the Survival Curve comparison legend are shuffled when reloaded

  • Cohorts: Issue fixed where long comparison names caused by a string concatenation between cohorts were causing the comparison feature to malfunction

  • Cohorts: Issue fixed where comparison name updates not reflected in the UI

  • Cohorts: Issue fixed where option still remains checked after Molecular filter removed from the right panel while on the project tab

  • Cohorts: Issue fixed where TCGA data not showing Molecular breakdown for selected genes

  • Cohorts: Issue fixed where data selections in the correlation feature where the second option visually shows half of the result in the UI

  • Cohorts: Issue fixed where "missense_and_ptv_all" missing from PHWAS/GWAS variant types

Documentation

  • Added guidance about modifying existing Tools already added to a Pipeline

  • Clarified expected behavior and recommended order of steps when connecting input to Tools and editing Input Details when building graphical CWL pipelines

  • Added information about details for monitoring Batch Jobs in Project Activity Screen

  • Updated tutorials for building a DRAGEN pipeline using Nextflow

  • Added Pipeline “tips and tricks”

  • Added guidance on expected behavior when configuring user-managed AWS Buckets with Projects

  • Updated list of system event notifications

2022 November 8 - ICA v2.10.0

Features and Enhancements

  • Added format auto-detection for VCF (.hard-filtered.vcf.gz, .cnv.vcf.gz) and TAR (.v8.tar, .tar.gz) files

  • Pipeline documentation will be available if present when starting a new Analysis

  • Added Workflow Session event topics

  • Base: Shared databases are visible in Base tables. Shared databases are read-only and queryable tables originating from ICA modules such as Cohorts

  • Reduced pipeline spin up time delays due to shared files system provisioning

  • Added support for Illumina applications to add custom attributes to data, samples, workflow sessions, and analyses

  • API: "creatorId" for Project data records returned in the API

  • API: Added instructions for paging parameters on list endpoints in API specification

  • API: The edit and delete API endpoint for Terms of Use are no longer available

  • Updated ICA CLI to version 2.10.0

Fixed Issues

  • Fixed an issue allowing unauthorized users to access external-managed Project contents via a direct link

  • Fixed an issue in the Docker Repository view causing region text and check boxes to be misaligned

  • Fixed an issue causing storage configuration status to incorrectly report an error

  • Fixed an error preventing the creation of an AWS S3 storage configuration in a Project

  • Fixed an issue where the Service Connector status incorrectly showing ‘Currently Connected’ when the associated Connector is not connected

  • Fixed an intermittent error when clicking to expand a Connector rule details in the Connectivity View

  • Removed red exclamation mark icon next to "Unlinked Files" tab in the Samples view; improperly indicates an error

  • Fixed an issue where Bundle Terms of Use acceptance flag unset upon saving bundle version

  • Fixed an issue preventing users from viewing details of an Entitled Bundle

  • Fixed an issue resulting in an error message when creating Bundle Terms of Use with an existing version

  • API: Fixed an issue where the API endpoint to link data to a Bundle to responds with errors when data is part of a Reference Data

  • API: Fixed an issue causing 500 API error response when retrieving sample details

  • Fixed an issue causing incomplete list of users when adding users within the same tenant to the Project Team

  • API: Updated the API spec description for create upload URL and create inline view URL for Project Data

  • Fix to prevent editing of an existing active terms of use

  • Improved the Terms of Use UI flow for forcing re-acceptance upon update

  • API: Added a warning response for API to create custom events when event code exceeds 50 characters

  • API: Fixed the create data download URL API to correctly return a 503 instead of a 404 response when the underlying system is experiencing network-related issues

  • API: Added API endpoint to hide a Project

  • Fixed an issue with Boolean inputs incorrectly displayed as false on Analysis Details view (correct value is actually passed in)

  • Fixed an issue where CWL parameters containing only a white space are passed incorrectly to underlying pipeline runner

  • API: Fixed an issue where launch Nextflow analysis API returns a 500 error response when including duplicate parameters

  • Fixed an issue causing analysis failures due to docker image authorization check timeout

  • Fixed a shared storage issue causing analyses to remain In Progress until timeout

  • Fixed an intermittent issue causing Analyses to be stuck in progress with inability to view logs in the UI

  • Fixed an issue where Analysis output data is not immediately shown in the Analysis Details view upon completion

  • Fixed an issue causing Analysis failures showing “…Connection timed out (Connection timed out)…” in stdout logs

  • Fixed an issue causing CWL pipeline runner restarts during an Analysis which may lead to unexpected results

  • Fixed an issue resulting in stuck analyses when invalid Docker image is specified for a task; timeout reduced to 15 minutes

  • Fixed the name of Pipeline running logs tab from "workflow_runner" to "pipeline_runner" in Analysis Details view in the UI

  • Fixed missing start time on Analysis tasks and incorrectly reported costs for such tasks

  • Fixed an issue resulting in an error message when attempting to create a graphical CWL pipeline with a Tool with a Docker image that is not available in the Project's region

  • Fixed an issue causing errors when saving a new Bench Workspace

  • Fixed an issue causing out of sync Bench table and card views when a Workspace is created or deleted

  • Fixed an issue causing an error when starting Bench Workspace when enabling service account on the Workspace from tenant receiving the Workspace via cross-tenant share

Documentation

  • Added tutorial for using scatter/gather for CWL graphical pipelines

  • Added information about Documentation tab in Nextflow pipelines

  • Included guidance on creating custom events for projects via the API

  • Provided list of ICA-supported data file formats

  • Updated links and fixed typos

2022 October 17 - ICA v2.9.0

Features and Enhancements

  • ICA Cohorts is released in production and added to the ICA user documentation

  • Clicking on a folder in the Data view will open the folder. Once a Folder is opened, the Folder details are available on the top right of the screen.

  • Added XML Configuration documentation and examples

  • General ICA robustness enhancements

Fixed Issues

  • Limit the Projects Overview view to 100 Projects with an indication that filtering must be used to see excluded Projects to improve UI performance

  • Fixed an issue preventing removal of permissions on externally managed projects

  • API – Fixed a 500 error response from Search Project Samples API when using a page token parameter

  • Fixed an issue allowing Bundles to be saved without mandatory Release Version when back-navigating from the Create New Bundle screen in the graphical UI

  • Fix to prevent data added to Bundle via secondary data upload on Data Details view

  • API – Fixed an error message when attempting to link data to a Bundle from a Project with Data Sharing disabled

  • API – Fixed an issue with tag filtering on getBundleSamples API

  • Fixed an error when filtering on size in Docker Repository where size filter is ignored

  • Fixed an issue where Analysis stuck in Aborting status when abort request sent during final stages of Analysis (such as Output, Upload)

  • Fixed an issue where intermittent Analysis failures are caused by system scaling errors under high load

  • Reduced repetitive Nextflow runner log entries from internal system logs to improve readability

  • API – Fixed an error in the Launch Nextflow Pipeline API when invalid parameter names are requested

  • API – Correctly support multi-value input parameters in launch Analysis APIs

  • Deprecated Tools are only visible when ticking the box "Show deprecated Tools" box in both table and card views

  • Fixed an issue preventing files linked to a Project from a Bundle as available to load into a Base Table

  • When exporting a query result from Base the file receives the name specified by the user and contains the header from the table

  • Removed unsupported AVRO and PARQUET formats from Available Formats to load data into Base tables

  • Fixed an issue where Bench service account API key is not revoked when a Workspace is stopped

Documentation

  • Added ICA Cohorts Module documentation, including summary of features and example walk-throughs

  • Added tutorial for building DRAGEN pipeline using Nextflow on ICA

  • Added tutorial for building DRAGEN pipeline using CWL on ICA

  • Added XML configuration documentation and examples

  • Added metadata model documentation

  • Added instructions for setting up ICA v2 Bring Your Own Bucket (BYOB) with SSE-KMS key

  • Updated supported ICA regions

  • Added data privacy recommendations regarding uploading data files

  • Updated bucket name character length constraints

  • Added guidance on supported characters in file/folder names

  • Updated fields and descriptions under Tool Settings

  • Added documentation for linking bundles to a project

2022 September 12 - ICA v2.8.0 and v2.8.1

Features and Enhancements

  • Added support for providing a nextflow.config file with a Nextflow Pipeline to improve pipeline building experience (see Nextflow Configuration docs)

  • Each tenant is limited to 100 in-progress Analyses at any given time to prevent resource exhaustion; additional analyses will be queued. Note: For assistance with planning batch jobs which require greater than 100 concurrent analyses, please contact Illumina Tech Support.

  • Added support for deep link URLs for direct navigation to child views in UI

  • API – Added an API to link Data to Projects in batch

  • API - Limit number of tags on data to 100; Excess tags in request will be ignored

  • API - Added "slug" attribute to Pipeline, Project, and Data asset types

  • API - Added support for the use of Project slug in place of ID in API request bodies

  • API - Added “Uniform Resource Name” (URN) attribute to Pipelines and Project object types

  • API - Added Project ID to notification payload for Pipeline Succeeded/Failed events

  • API - Added owning Project ID for data records in the getAnalysisOutput API response

  • API - Indicate originating Workflow/Workflow Session on Analyses returned via API, when applicable

  • Added support to use bundled Pipelines and data to create Analyses via API without first linking to the Project

  • Added support to automatically mount project data in Bench Workspace with ICA CLI

Fixed Issues

  • Renamed “Delete” to “Manage” in the Samples page dropdown menu

  • Removed the IGV button under View when selecting a VCF file

  • API – Fixed an issue causing the List Files API to incorrectly respond with an error when passing a folder or file path parameter value containing a leading or training space

  • Fixed an issue where files linked through a Project Connector will appear on root level as folder linking is not supported yet in the Project Connector

  • Fixed an issue when deleting files from S3 along with records from ICA via either volume delete (with purgeobjectstore=true) or folder delete API calls, if there are more than 1,000 items to delete the system deleted only the first 1,000 items, however, all the data records got deleted from ICA. In this case the data in the user-managed S3 bucket and ICA was out of sync.

  • Fixed an incorrect count displayed in UI when scheduling a download for a folder containing sub-folders

  • Fixed an issue where scheduling a download of a folder containing sub-folders creates duplicate data transfer operations

  • Disabled support for "fileNameExpression" on download rules; if provided when creating a download rule, the value will be ignored

  • Fixed an issue allowing public domain users to incorrectly access ICA

  • API – Fixed an issue where API to retrieve folder upload session intermittently responding incorrectly with error

  • API – Fixed a 500 response from the API to add a pipeline to a bundle when pipeline has already been added

  • Fixed an issue where user with "Viewer" permission for Flow in a project cannot link Samples from that Project

  • Return the correct error response when creating a new Nextflow or CWL pipeline through the API or CLI when an existing pipeline with the same name already exists

  • Improved UI messaging when setting up a Storage Configuration with permission error

  • Fixed an issue when attempting to edit a saved incomplete Tool (i.e., missing Docker image)

  • API - Fixed API to create notification channel to properly respond with error when attempting to create a duplicate notification channel

  • Fixed an issue where boolean Analysis input settings are incorrectly set to false when passed to the Pipeline runner

  • Fixed an issue when launching a Nextflow pipeline via API with multi-value inputs the order of the listed multi-value input is not preserved when passed to the pipeline runner

  • Fixed an issue where "Re-run" analysis button is broken for Nextflow pipeline analyses

  • When different versions of DRAGEN are run on the same host, reset the DRAGEN state so that the next task can correctly configure the FPGA compute instance

  • Fixed an issue causing intermittent analysis failures due to system error when evaluating permission to pull Docker image

  • Fixed empty start time attribute for Analysis tasks with a short duration (within a few seconds)

  • Fixed an issue where analysis start time is set when in fact the analysis is still queued; analysis start time will be set when compute resources (i.e., shared file system) have been requested

  • Fixed an issue where log file content for the pipeline runner is overwritten if system error occurs during an Analysis causing a pod restart; multiple log files may be created for the pipeline runner task if such an error occurs

  • Fixed an issue causing UI log stream view for Analysis tasks to incorrectly show errors or blank views

  • Fixed an issue causing incorrect Analysis task log to be displayed on the UI Logs view

  • Fixed an issue where analyses get stuck in Running status if input file paths contain leading or trailing spaces in path for analysis stuck in running status if input files contain leading or trailing spaces

  • Fixed an issue causing Analysis to be incorrectly marked as Success due to incorrect evaluation of exit code of an analysis task

  • Fixed an issue preventing the same file to be passed to multiple input fields when launching an Analysis

  • Workflow failures due to unavailable compute resources will fail quickly with a more descriptive error

  • Fixed an issue where analysis output data is not immediately shown in the Analysis Details view after the analysis completes

  • Removed the “Print” and “Save” buttons in the Analysis Report

  • Fixed an issue preventing Pipeline creation and Tool editing when an invalid Tool is saved in the Tool Repository

  • API – Fixed a 500 response from API to list Projects to properly return a 400 response when providing a search filter value with invalid characters

  • API - Fixed a 500 response from create download URL API to properly return a 400 response when data is not in "Available" status (ie, Partial/Archived)

  • Fixed an issue with loading data into a Base Table created directly in Snowflake with fields containing quotes

  • Fixed an issue where data was added to Base Table twice when multiple Scheduled Jobs run simultaneously

  • Fixed an issue with previewing Base Tables created directly via Snowflake

  • Fixed an issue where status is stuck at "In Progress" for Base Copy Table operations

  • Fixed an issue where Base Tables linked to a project from a bundle are shown as Views (read-only)

  • Fixed an intermittent issue causing duplicate data in Base Tables

  • API - Fixed a 500 response when sending a duplicate requests to load data for a Base Table to properly return a 400 response

  • API - Fixed a 500 response from API to load data into Base Table to correctly return a 400 response when attempting to load data with a type mismatch

  • API - Corrected Swagger UI responses

  • Fix error "Unable to save workspace" when attempting to start a Bench Workspace

Documentation

  • Added documentation for analysis output mappings

  • Added documentation for Nextflow configuration

  • Added documentation for reference data

  • Added documentation for metadata models

  • Added Quick Registration Guide

  • Added tutorial for creating/running CWL pipeline on CLI

  • Added end-to-end tutorial highlighting DRAGEN Germline pipeline

2022 July 27 - ICA v2.7.1

Fixed Issues

  • Fixed an intermittent issue where analyses fail due to "Connection timed out" system error (seen in analysis error logs)

  • Added retry logic to decrease rate of pipeline failures due to internal system error (e.g. errors in pipeline runner log such as connection timeout, 404 pod not found, 500 errors)

  • Fixed an intermittent issue where "Cannot access folder" error is shown in pipeline error logs (most common in long-running analyses)

  • Fixed an issue resulting in analysis failures due to system error under high load

  • Fixed a compute instance configuration issue impacting DRAGEN analyses

2022 June 29 - ICA v2.7.0 and CLI v2.4.0

Features and Enhancements

  • API - Added method to retrieve an analysis JSON input data

  • Added Analysis Details views in the UI

  • Added Analysis ID to the project analyses

  • Support deep links for direct navigation to Analysis Details view

  • Added support for multiple Nextflow versions when creating Nextflow pipeline

  • API - Added method to create multiple samples in batches

  • API - Improved response time performance of API that launches analysis with many data inputs

Fixed Issues

  • Repaired issue where Pipeline-generated Sample Metadata is not picked up by platform

  • Restored Time to delete function; now executes file deletes at the specified time

  • Bench example code now lists project data and pagination correctly

  • Restored ability to link secondary data to primary data

  • Fixed issue where batch sample creation jobs get stuck in PARTIALLY_SUCCEEDED status

  • Resolved issue where batch sample creation is not completed if data items could not be linked

  • API - Fixed issue where mountPaths were ignored when launching analyses via API

  • Restored function so that bundle contributors can link assets to their bundle

  • Restored data linking to a sample in an ICA project

  • Resolved issue where child workflow configurations prevent termination of parent workflow

  • Removed incorrect error reported in cases where previous failure occurred

  • Restored pipeline linking for bundle contributors

  • Improved UI messaging for Storage Configuration permission errors

  • API - Updated GET /api/projects/{projectId}/data to returns correct file details

2022 June 07 - ICA v2.6.0 and CLI v2.3.0

Features and Enhancements

  • Re-labeled "Run" and "Runs" to “Analysis” and “Analyses” in the UI

  • Allow storage configuration to be disconnected from a hidden Project

  • Added support for a read-only view of Entitled Bundles and Bundles shared from another Tenant

  • Changed analysis timeouts for setting up environment, downloading inputs, pipeline runner (inclusive of all child steps), and finalizing output steps to 96 hours

  • API – Add idempotency key header to createAnalysis API

  • API – Analysis logs are available via getAnalysisSteps API; includes streamable via websocket endpoint during execution and downloadable S3 URL upon completion

  • JSON data with array field can be added in a Base table

  • Cohorts – Additional submenu items now available

  • CLI – Added support for setting the current Project ID to the value set in the ICA_PROJECT_UUID environment variable upon starting the CLI

  • CLI – Added support for mounting Project data to local operating system (Read-Only)

  • CLI – Added support for the outputParentFolderId flag for the ‘projectpripeline start’ command

  • CLI – Added support for customizing the filename/path of additional Pipeline files with the pipeline create commands

  • CLI – Added owning Project name/id attributes to the project data list output

  • CLI – Added owning Project name attribute to output for projectdata list command

  • CLI – Added commands to create and refresh API tokens (JWT)

  • CLI – Added support for CLI use without writing to disk by enabling API Key and Server flags for all commands

  • CLI – Added support for linking data to a Sample with projectdata upload command

Fixed Issues

  • Fixed an issue with UI stability leading to slow response times

  • Fixed an issue causing data uploads via browser to hang without error

  • Fixed an issue with UI automatically refreshing upon load when many Projects are present in the Projects view

  • Fixed an issue where Accept Share pop-up remains on screen after pressing to Accept the share

  • Fixed an issue with uploading nested folders using the Windows Connector

  • Fixed an issue where path separators are misinterpreted in upload rules for Windows Connectors.

  • Fixed an issue where modifying a local file uploaded (tracked) using the Connector causes a Connector failure

  • A User with Contributor Role for Flow and a Viewer Role for a Project can start an Analysis using input data from a Sample

  • API – Fixed an issue where sampleDeepDelete API fails with 500 error response when Sample contains a linked folder in archived status

  • Fixed an issue where ICA_STARTING_USER_TENANT_NAME is not set to the correct value on initial start

  • API – Fixed an issue where upload rules are not deleted via the deleteUploadRule API despite a success response

  • Sensitive fields in metadata models are now shown in italics

  • Fixed an issue where filtering on the Data view gives incorrect results

  • All Contributors in a Bundle are allowed to link assets to the Bundle

  • Content of a Bundle shared via an entitlement in another Tenant is now visible

  • Fixed an externally managed Project Team to only allow adding members with "Viewer" and "Allow Download" permissions

  • API – Fixed an issue with project sample APIs returning 500 error response when sample is not linked to the project specified in the request

  • API – Fixed issues with cursor-based paging on listData and listBundleData APIs returning expected results

  • Fixed an issue where a User can create empty folders inside an externally managed Project with Viewer permission

  • Fix auto-detection of AVRO and PARQUET data formats

  • API – Fixed an issue where the API to link Samples to a Bundle incorrectly returns a 204 response when the Sample is not in Available status and the Sample is not actually linked

  • API - Fixed an issue where changing the Project owner causes unexpected error responses from the listProjects, listProjectPermissions, and getProjectPermission APIs

  • API – Fixed an issue where deleteDownloadRule API returns 204 responses but does not delete the specified download rule

  • Fixed an issue causing delays in operations performed on data

  • Fixed an intermittent permission error when writing to /scratch from a Pipeline

  • Fix to make tags optional (not required) with Pipeline launch requests

  • Fixed an issue causing Analyses to be incorrectly set to SUCCEEDED status when failure occurs

  • API – Possible to create a Pipeline with a metadata model via API

  • Fixed an issue where logs are not streamed during Analysis

  • Fixed an issue when a User tries to start an Analysis with missing required parameters, the analysis is stuck in "awaiting input"

  • Fixed an issue preventing renaming a pipeline after attempting to save and receiving error that a pipeline exists with the same name

  • API – Fixed an issue when a Pipeline is launched via API with no input value provided for a nonmandatory input parameter the analysis status remains in "Awaiting Input" indefinitely

  • API – Fixed an issue causing unpredictable error responses from createAnalysis API when many Analyses are launched in rapid succession

  • API – Added operationID to API spec for the listAnalysisStorages and getAnalysisRawOutput APIs

  • Fixed an issue when Analyses are queued and Aborted before they are Running, the status is not correctly updated in the API/UI, and they appear stuck in “Aborting”

  • API – Fixed an issue with createAnalysis API causing error responses when launching many Analyses in rapid succession

  • Possible to load JSON in Base with a repeated field (array)

  • Tables created by the meta and admin data Schedules are updated after a sync

  • Fixed an issue causing Base Table field names to be changed from lowercase to UPPERCASE during a manual sync

  • Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when schedules are run

  • Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when scheduled

  • Fixed an issue where the User receives an obscure error message when creating a Tool from a Bench Workspace with an image name that already exists

  • Fixed an error message upon attempting to save Bench Workspace when input values are invalid

  • Fixed an issue where the Bench Workspace is temporarily inaccessible and stuck in Starting status

  • CLI – Fixed an issue with list samples command output limited to 20 unique records

2022 April - ICA v2.5.0

Features and Enhancements

  • The value of the "Show Hidden Projects" filter on the Project Overview page will be saved for each user

  • Change analysis task log names to be more human-readable

  • Analyses will queue when system load is high

  • Add owning project tenant name (ICA_PROJECT_TENANT_NAME) and the tenant name of the user which last started the Workspace (ICA_STARTING_USER_TENANT_NAME) environment variables to Bench Workspaces

  • Add Cohorts URL environment variable to Bench Workspaces

  • API - Add support for Cross Origin Resource Sharing (CORS) to the API

Fixed Issues

  • Fixed an error message in UI when attempting to link a Sample that cannot be linked

  • API - Fix issue where API to update Sample metadata fields does not update the fields

  • Fixed error message when creating a Bundle with a name conflict

  • Fixed error message when a user attempts to view Bundle details without permission

  • API - Fix Project Bundle APIs to properly link, retrieve, and unlink entitled Bundles

  • Fixed to block ability to link data with Contributor role and no upload permission on the target project

  • Fixed an issue where the Owner of an externally managed Project cannot add Team members with Viewer or Download permission

  • Fixed field for providing an e-mail address when adding a Team member to a Project to be required

  • Removed temporary directory creation by service connector

  • Fix for configuring service connector to use proxy settings on Linux clients

  • Fixed an issue where data downloaded through Service Connector is incomplete

  • Fixed an issue causing errors in Service Connector when a directory is deleted in the UI but still present on the local machine. Data added to the folder on the local machine will not be uploaded if the corresponding folder is deleted in the UI.

  • Fix to display an error message when saving a Pipeline with invalid fields

  • Fix issue where CWL file/directory inputs are not mounted when provided within nested structures (ie, CommandInputRecordSchema, CommandInputArraySchema)

  • Fix to show error when attempting to run a graphical CWL pipeline with an invalid definition

  • Fix error when attempting to save a graphical CWL Pipeline with a Tool with an output folder and no specified format

  • API - Fix to ignore input JSON field when launching graphical CWL pipelines via API

  • API - Fix ignored default compute type (machine profile) defined in the Pipeline definition when launching a graphical CWL Pipeline via API

  • Fixed Base Query result export file format set to UNKNOWN

  • Fixed Base activity log to show Table associated with log event

  • API - Ignore unknown attributes in request bodies

  • API - Fixed API error responses formatted as HTML instead of JSON in error scenarios

  • API - Fix list analyses API not returning the "items[].analysisStorage" field

2022 April - ICA v2.4.0, v2.4.1, v2.4.2

Features and Enhancements

  • Samples may only be linked when status is Available

  • Added Data ID to grid view in UI

  • Added ability to select and copy tag values to the clipboard

  • Added an indication of source domain in list of reference data sets when adding new data to Reference Data

  • Updated the service connector client icon

  • Added ability to filter projects by workgroup, including projects where user is a member of a workgroup added to the project

  • Added ability to specify output directory within a project when launching a pipeline

  • Added ability to Favorite Projects and filter the Projects view by Favorites

  • Remove obsolete commented text from auto-generated CWL Tool definitions

  • Scheduler displays the number of files to be loaded into a Base Table

  • Added an indication that Base Table previews display up to 50 columns and 100 rows

  • API - New endpoint to manage SecondaryData on Data objects

  • API - Added support for "mounts" parameter when launching a pipeline to provide input mount paths for file and folder inputs

  • API - Added fields to API error response schema: HTTP Method, User ID, Tenant Name, Tenant ID

  • API – Added new endpoints for retrieving Entitled Bundles and to manage Project Bundles

Fixed Issues

  • Docker images available via entitled bundles are listed in green text in the Docker Repository view

  • Fixed an issue in OpenAPI spec causing invalid cURL requests in 'Try it Out' functionality for the create upload URL API

  • Fixed an issue that resulted in empty Workgroup listing via the API and when adding Workgroups to a Project Team

  • Fixed an intermittent error when navigating to Bundles view

  • Fixed an issue where Tools available via Bundles are not displayed in the Tool Repository

  • Fixed an issue where example JSON generation for root Metadata Model fails when the root Metadata Model is modified

  • Fixed an issue where incorrect reporting of compute start times resulted in incorrect compute billing

  • Fixed an issue where Nextflow analyses get stuck "In Progress" when output folder is deleted during an ongoing pipeline run

  • Fixed an issue where incorrect casing on the scatter method 'dotproduct' in CWL auto-generated from graphical pipelines results in failures in pipelines with scatter-gather steps

  • CLI - Fixed an issue with startAnalysis APIs no longer accepting the 'Accept' header with value "application/problem+json,application/vnd.illumina.v3+json”

  • API/CLI - Fixed issue where graphical CWL pipelines launched via API/CLI do not use machine profiles (compute types) specified in the pipeline definition

  • Fixed an issue where pipelines within shared bundles could not execute across regions

  • Fixed an issue where exported Base Tables with null values populate a CSV with 'N' values instead of empty values

  • Fixed an error when loading data into Base Tables with an invalid number of columns

  • Fixed an issue causing View Results in Base Query History to return an expiration error

  • Fixed an issue that caused JSON import into Base Tables to fail when sub-field (record type) is used in the Table schema

  • Updated the error message when invalid SQL is used in a Base Query

  • Fixed an issue with JSON import to Base Tables failing when reference data fields are enabled

  • Fixed an issue where updated Base Table fields in the UI do not update the underlying Snowflake Table

  • Fixed an issue resulting in "Unable to create file error" message when creating a Tool in a Bench Workspace

  • Fixed an issue where Bench Workspaces cannot be deleted

  • Fixed an intermittent issue where Bench Workspaces unexpectedly fail in an error state

  • Updated messaging for users without access to Variant Viewer and Cohorts

  • API - Fixed an issue where the API did not accept multiple 'Accept' header values

  • API - Fixed an issue where the create folder upload sessions API returns date and time with incorrect formatting

  • API - Fixed the 500 response on APIs when the user has not yet accessed the UI

  • API - Fixed missing version header in OpenAPI spec for APIs to link a pipeline to a Project or Bundle

  • API - Fixed the 500 response from create project API when the 'storageConfigurationSubfolder' value is invalid

  • API - Fixed a 500 response from the API to link data to a bundle when linking a Folder with child data records

  • API - Fixed an issue where the API to create File data record creates the data record at the root of the project data when a trailing forward slash ('/') is missing after the folder name

  • API - Fixed an issue where API to retrieve pipeline reference sets does not display linked reference data

  • API - Fixed an issue where APIs to initiate analysis perform validation of required properties in request body

2022 March - ICA v2.3.1

Fixed Issues

  • Fix issue where pipeline launch is blocked when a single file is used as the input to multiple pipeline inputs

2022 March - ICA v2.3.0

Features and Enhancements

  • A Base query is now possible with null marker 'nan'

  • A user can search on Secondary Data tab

  • A user is allowed to run a pipeline from a shared project if the Billing mode is set as 'Tenant'

  • Add "application" field to Project API model to indicate whether a project is externally managed.

  • An error is shown when a user tries to build a workspace that is not running.

  • An error is thrown when the user uses the same name of an existing project connector rule in the same project

  • Base query can be exported as a view

  • Bug fix for allowing multiple files with same name but different paths to be passed as input; causes conflict when downloading inputs to pipeline filesystem, one file will be overwritten (random) causing unpredictable results

  • Data filters and columns now also apply to technical tags

  • Externally managed projects are now supported via API

  • Meta data schedule can be created on tenant level

  • New ICA notification for Data/Sample linked

  • Preview of the content of a Base table is available with null marker nan

  • Project connector permissions are aligned with manual linking permissions

  • Remove obsolete "BlueVantage" option from the Source drop down selections during Base schedule creation

  • Revert cwltool upgrade

  • Samples linked via a project connector are visible in the 'receiving' project

  • The 'Type of Files' field in the configuration of Upload Rules in a Connector is changed to 'Assign Format' so that its purpose is clear

  • The project connector rules are available across tenants when the user has Contributor rights

  • The user is able to use the filters 'Type' and 'Status' and sort the files

  • The yaks-scheduler used to check the pod-to-node fit, but will now defer to the kubernetes default-scheduler as the pod-placement authority. The assumption is that when the pod is not scheduled, there are no nodes available, and a node will be provisioned.

  • When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.

  • [Base] When selecting 'Include reference', you will see the record first and when you unfold the record, you will see the fields belonging to the record

  • When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.

  • [API] Add ability to filter by region for list projects API

  • [API] Add psToken authorization type to enable creating JWT from psToken via the create token API

  • [Bench] Add ICA_PROJECT_UUID environment variable to the Jupyter Notebook Bench image.

Fixed Issues

  • Fix 400 error response when launching an analysis with a pipeline from a bundle

  • Fix Base Query results file export set to format UNKNOWN

  • Fix Bench pricing calculation accuracy

  • Fix for Base table export unexpected failure

  • Fix for auto-linking pipeline outputs files to a sample when input files provided are linked to the sample

  • Fix for failed Base table import when imported data contains "nan" (null marker) for a column of type integer

  • Fix for linking Base Query templates to Bundles

  • Fix for modifying Bench Workspace image unless Workspace is in Stopped state

  • Fix for more informative API error response when EULA has not been accepted

  • Fix for preserving sample data links for samples linked from shared bundles

  • Fix for retrieving download URL for data in an externally managed project.

  • Fix for silent error when running Base schedule on Administrative Data or Metadata

  • Fix intermittent UI issue where checkbox for file selection during docker image add is partially visible

  • Fix issue where Base schedule for Metadata does not work when source is set to "Tenant"

  • Fix issue where Base table schema definition changes from record to string after sync

  • Fix issue where Project Connectors cannot link samples where the source project is owned by a different tenant

  • Fix issue where an analysis cannot be started when a metadata model is defined for the pipeline

  • Fix issue where data cannot be removed from a Sample using the "Remove Data" button from the sample details view.

  • Fix issue where pipelines from linked bundles do not show as available selections in the "Generated by Pipelines" drop down when creating a Base schedule

  • Fix issue where samples cannot be updated after configuring a Project Connector rule with the sample.

  • Fix issue where scroll bar is not shown in the metadata models list view

  • Fix issue where secondary data is not shown in the reference data view

  • Fix issue where team members added to eternally managed projects cannot be granted Contributor permissions, and cannot link data to other projects

  • Fix issue where user cannot use sample filtering when samples and files are manually linked to their project and the files do not appear associated with their sample in that project

  • Fix issue where user is unable to see their input files when using sample filtering if they are linked from another project without selecting the flat view option

  • Fix issue where workgroups cannot be added to project team of externally managed projects with Contributor permission

  • Handle K8s api timeouts by retrying query.

  • Improve pipeline analysis startup times to approximately 10 mins maximum (times may vary)

  • [API] Fix 500 response for getProjectDataLinkedProjects API

  • [API] Fix error response when invalid API Key is provided for authorization

  • [API] Fix get bundles API error response when user has access to a project with a linked bundle that the user does not have access to

  • [API] Fix getProjectData API to return correct results when using the "id" query param

  • [API] Fix incorrect error response for Project Data API to create folder upload session

2022 February - ICA v2.2.0

Features and Enhancements

  • the cwl workflow engine logs now also show debug logs, these includes the javascript expressions evaluated etc. these logs are similar to running cwltool --debug locally

  • Show "username" of pipeline creator in Pipelines overview view

  • Removed excessive logging from the "Audit Trail" logging.

  • Various performance improvements - reuse of connections to Base instances

  • A particular storage configuration should only be available during project creation in case it is your own storage configuration, or when the owner (or a tenant administrator) gives explicit access to it for other users in their tenant.

  • Bench - when trying to get an access token through the API, access was denied in case Base was not enabled within the project.

  • Base - 'You are not allowed to modify data' was shown to the user when clicking the Sync button as a Contributor

  • Base - importing data with format JSON in a table was not possible.

  • Files within a sample will only show up in case the data is also explicitly linked to the project/bundle.

  • Bundles - a user with Contributor access did not have the necessary access rights for managing all resource of the bundle.

  • Project billing modes can be 'project' or 'tenant'. In case 'tenant', the users from another tenant than the owner of the project was not able to upload files.

  • Within the Activation Codes view, activation codes for access to the Variants/Cohorts modules were not listed.

  • It was not possible to add a bundle at project creation. The project had to be saved before the bundle could be added.

  • The roles explanation in the Team page on a permission record did not show the entire information.

  • Various stability improvements - In case of a database outage, ICA was not able to automatically reconnect.

  • When managing docker images, a partial docker image was not allowed to be deleted nor updated resulting in a stuck image.

  • When exporting the contents of a grid from the UI, the exported file contained tenant ids instead of tenant names.

  • Base - the export of query results to a json format resulted in a failure in case the query ended with ';'

  • A non-tenant-admin user was not able to create a metadata model.

  • Base - exporting query results, specifying a custom delimiter did not work.

  • When a user clicks the bundle card in the "Bundles" view, and the user has no access to manage the bundle because it is shared with the user by another tenant, the following notification will be displayed: "Details for a bundle from another tenant are not accessible" instead of "Bundle details unavailable"

  • When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.

  • Various performance improvements - linking a folder with a lot files to another project or bundle timed out.

  • Base - loading metadata into a table using a schedule resulted in the same data being added over and over again.

  • When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.

  • Limit usable character set for the user reference of an analysis run

  • When an analysis was aborted, the output files which were generated so far were not visible in the details of the analysis, but only in the data view.

  • Base - schedule metadata population in a table did not work.

  • Added support for Docker Images to be added to Bundles

  • Added Project Connectors in the project connectivity section to enable creation of automatic sample and data linking rules between projects.

  • Added Queue Date to analysis Logs view to indicate when the specified job was submitted to the underlying infrastructure.

  • Added support for externally-managed projects for Illumina software applications built on top of ICA to manage resources inside ICA projects while limiting tenant users' direct access to the project to read-only capacity.

  • Improved performance of API calls when using API Key for authentication

  • Added tenantName to API resource DTOs

  • Add API to abort analysis run

  • Added APIs to share a storageConfiguration, share a storageCredential, update a storageCredential secret

  • Added mime-type field to the DataFormat DTO for the data APIs

  • Added "objectEtag" field to DataDto for Data APIs to expose e-tag from the cloud provider object store

  • Added Base APIs to retrieve Jobs, retrieve Tables, and load data into Table

  • API - POST‚/api/projects/{projectId}/samples:search was missing request parameters to browse through samples in a paged manner.

  • API - POST /api/projects/{projectId}/analysis:cwl - tags were not persisted when starting a CWL analysis run

  • The PUT /project/{projectid}/data/{dataid} gives the user the impression that all fields can be updated on the data, but it is a different set for files as for folders. This distinction is not properly documented.

  • API - GET /api/projects/{projectId}/analyses - when the pipeline did not belong to the tenant, but the user still has access to it (eg: through a bundle), the analysis details could not be retrieved through the API.

  • API - GET /api/projects/{projectId}/analyses/{analysisId} returned the details of an analysis, even if the projectid did not match the actual project the analysis ran in.

  • API - GET /projects/{projectId}/samples/{sampleId} did not expose metadata information

  • API - GET /projects/{projectsId}/data presented the user with an unclear error message in case providing invalid request parameters.

  • API - An unclear error was presented in the problem response in case an invalid json was provided as request body.

  • API - An unlcear error was presented in the problem response when creating a project using invalid uuids in the request body.

  • API - Removed deprecated "sample matching strategy" endpoint

  • CLI - Added command to update project data

  • CLI - Added command to retrieve a list of project samples

  • CLI - Added command to create a project sample

  • CLI - Added command to list available data formats

  • CLI - API Key encryption modified to use local value on machine where API Key is configured to prevent hijacking config.

  • CLI - Added command to get details of a project sample

  • CLI - Added command to list data linked to a project sample

  • CLI - Added command to update a project analysis

  • CLI - Added command to update a project sample

  • CLI - Added command to unlink data from a sample

  • CLI - Added commands to delete a project sample

  • CLI - Added command to link data to a sample

  • CLI - Added command to complete a sample

  • CLI - Removed unnecessary flags on the config command.

  • CLI - For starting CWL and Nextflow pipelines, the flag 'storage-size' was ignored by CLI, and the default storage size of the pipeline was taken.

Fixed Issues

  • [bugfix] when CWL workflows failed or produced null as one of the outputs, all the outputs were incorrectly processed. in this case, the user may not be able to see the details of the analysis output in UI/API.

  • [Base] Fix error message when viewing a query from query history that no longer returns results

  • [bugfix] in some cases after aborting an analysis, it may remain stuck in aborting state indefinitely. Now more resources are explicitly cleaned to avoid this issue from happening.

  • Fixed bug when showing logfiles of steps in an analysis.

  • [bugfix] cwl workflow sometimes fails in spite of all steps in the workflows have succeeded. Made the workflow engine robust to detect and retry in this case

  • When linking a folder containing +2000 files to another project, only 2000 files were actually linked.

  • When a user did not choose a country when filling out the user account details, login to ICA could fail.

  • Fixed bug when handling custom output objects in CWL.

  • Fixes issue with compute type selection when launching CWL graphical pipelines

  • The Project > Team permissions showed a little pencil in front of the row indicating a modification. With this fix, the pencil will only show when there were actual modifications done to the row in particular.

  • API - All dates in API responses were formatted in 12 hours notation without AM/PM notation. This way a time in PM is presented as AM. With this fix, all time notations are in 24 hours.

  • When starting an analysis through the sample page, the start window is not closed after pressing start. Users would click it again, which would start the analysis multiple times. With this fix, the start window is closed after pressing start.

  • When starting an analysis fails because the backend service returns an error, the status of the analysis is "Awaiting Input". With this fix, the analysis goes to "Failed".

  • Base - in case the export of query results failed, an empty file was created and never cleaned up. With this fix, there are no leftovers in case of export failure.

  • When a sample was linked from another project to the current project, the details view of the sample shows all data linked to it, even of the current project does not have access to it. With this fix, only the data which is also linked to the current project will be shown on the details of the sample.

  • API - PUT /project/{projectid}/data/{dataid} update sometimes had to be done twice for it to persist. With this fix, all editable attributes can be updated at once.

2021 Decemeber - ICA v2.1.0

Features and Enhancements

  • [CLI] Storage size can be entered by name

  • [CLI] Details of a project analysis can be retrieved by entering the analysis Id

  • [CLI] A list of metadata models can be retrieved

  • [CLI] A list of regions can be retrieved

  • [CLI] A list of storage configurations can be retrieved

  • [CLI] A pipeline can be linked to a project by using the name/id of the pipeline as an argument

  • [CLI] A pipeline can be unlinked from a project by defining the name/id as an argument

  • [CLI] A list of storage bundles can be retrieved

  • [API] Usability improvements for the list project data API:

    • Modify filePath query param to behave as prefix instead of exact match by default

    • Add filePathMatchMode query param

    • Add parentFolderPath query param

  • [API] Add API endpoint to refresh a JWT (POST /api/tokens:refresh)

  • [CLI] Add CLI command to create projects

  • [API] Support search by tags on samples search API

  • Renew macOS connector certificate

  • When a filter is applied and you select a folder, you can see the content from that folder without th* Connector rules for a project can only be changed when you are active in that project

  • It is possible to link more than 2000 files to a bundle or another project

  • Data coming from a project in another tenant can be linked when you are invite for collaboration

  • The error from a failed pipeline analysis is visible on the details page

  • Species and type are unique within a tenant, not across tenants

  • Sqs and sns test messages are aligned with the email test message

  • A collaborating user from another tenant is able to see the details view of a pipeline analysis run.

  • Bundle data can be used via API

  • When executing api/projects/{projectId}/data and a file already exist, there is an indication that thresponse.

  • Connector cancel button is renamed to delete to make its functionality more clearly

  • [CLI] Modify pipeline start command to use pipeline name or ID as an argument instead of a flag parameter

  • [API] Removed endpoint to POST on a sample metadata model

Bugs Fixed

  • [Base] Fix error message when viewing a query from query history that no longer returns results

  • [CLI] Fix for CLI upload command when uploading a folder with 0 bytes files

  • [API] Fix create bundle API to return error when bundle with existing name is attempted to be created

  • [API] Fix for non-functional If-Match header on PUT requests

  • [CLI] Fix folder upload "panic: close of closed channel" error

  • [API] Fix API for project creation returning success with invalid storage configuration provided

  • [Bench] Fix for docker build logs not showing when build is interrupted

  • Fix population of sample metadata from multiple analyses on the same sample

  • Fix error message for missing docker image on tool creation

  • [CLI] Fix pipeline create command to make pipeline code an argument instead of a flag parameter

  • Fix UI element for tags to remove active button appearance

  • Fix error when attempting to link pipeline with no pipeline selected

  • Fix error message when attempting to add field to Base table schema with invalid characters

  • Fix issue with analysis outputs files and logs not showing after analysis completion

  • Fix UI issue with pipeline selection in the Generated by Pipelines field of the view to create a Base s

  • Fix issue with a duplicate workspace being created while editing a workspace

  • Fix issue for table creation using schedule for metadata

  • Fix Base viewer role when applied to workgroups

  • Fixed Issue: optional parameters for Nextflow pipeline cause error when value not provided

  • Fixed issue when selecting data records to export from the Data overview page on a Bundle

  • Fix for Base saved queries not visible in shared project

  • Fix minor price calculation error (rounding) for Base storage

2021 November - ICA v2.0.0

Features and Enhancements

  • API cursor-based and offset-set paging

  • Authorize calls using Illumina API key

  • Multiple reference data elements are possible when starting an analysis

  • APIs added for:

    • Account, User, Workgroup

    • Connector, Download Rule, Upload Rule

    • Data, Data Format, Data Transfer

    • Region

    • Download

    • Entitlement

    • Event Log

    • Project, Project Analysis, Project Data, Project Permission, Project Sample, Project Pipeline

    • Metadata model

    • Pipeline

    • Bundle

    • Notifications, Event Codes

    • Base

  • OpenAPI v3 API Spec published at https://ica.illumina.com/ica/api/swagger/index.html

  • Added sort parameter to several APIs

  • Custom event payload is defined as a json

  • Separate call to retrieve metadata model fields available

  • Add rclone temporary credentials support in the API to retrieve folder upload session details

  • Tokens API POST also contains basic authentication

  • Activation code endpoints are aligned with start analysis endpoints

  • Base templates cannot be deleted

  • Information can be written from Bench to Base

  • Viewer/Contributor permissions are introduced in Snowflake

  • Saved Template details can be edited

  • Access via Illumina authentication is available

  • Snowflake access token can be created

  • Added new Bundles feature for sharing data with other users

  • ICA allows the user to connect an S3 bucket as storage instead of an Illumina managed storage

  • An overview of all storage configurations is available

  • Secrets can be managed for the storage configuration

  • A new storage configuration can be added

  • The health status of a configuration can be monitored

  • A storage configuration can be deleted when not used in a project

  • Storage type and health are displayed in the overview

  • When selecting own storage ICA will return you a list of all BYOBs in the chosen region

  • A default bucket can be selected for every region

  • Data is automatically archived/deleted when the grace period (if present) is over

  • Data is archived/deleted automatically by a housekeeping mechanism

  • An admin user can change the date to be archived/deleted at any time

  • Data has a status

  • Data format is automatically detected at creation

  • Files can be grouped in a sample

  • Data can have additional linked projects

  • The data screen shows project data and data linked via a bundle

  • 3 new columns are present in the data grid: "Owning Project", "Linked Projects" and "Linked Bundles"

  • The record is displayed in the data grid depending on the data being linked or not

  • Filter on Status with checkboxes on status values is possible

  • Total Data Size is available

  • Data can be linked/unlinked to/from a project

  • Only root parent files and folders are shown when opening the data grid

  • Deleted status does not exist anymore

  • Data from projects with Data Sharing disabled cannot be linked

  • Project is hidden instead of (soft-)delete

  • Metadata models can be created for linking to a project

  • Add support for Nextflow pipeline language

  • A pipeline input can have multiple formats

  • Output JSON for CWL analysis is shown

  • Ongoing logfiles are available for the user

  • You have two options to define a CWL pipeline: Graphical editor or code

  • Test connection button is available for SQS

  • Checkbox is added to toggle view between all rules or only your own rules

  • Notification channels can be validated

  • Subscription can be edited

  • Custom subscriptions are possible

  • Notifications can be received via SQS, SNS, Mail, and Webhook

  • All users have an overview of all the notifications used in their tenant

  • Notification channels are only visible for tenant admins

  • Notifications can have a filter expression to only receive specific events

  • A project administrator can grant permission to a single user on a project

  • A project administrator can invite somebody unknown to ICA to his project

  • A contributor can manage reference data

  • A project administrator can define who pays for using my project

  • A project administrator can invite a workgroup to a project

  • When multiple project roles are given to a user/workgroup the correct project role is applied

  • Team-page in projects and bundles are aligned

  • Reference data can exist in multiple regions

  • Species and type are being populated per tenant

  • Data can become reference data

  • Data tree shows all the regions in which the reference data is available

Last updated