2023

2023 December 05 - ICA v2.21.0

Features and Enhancements

• Flow

  • Analysis logs (task stdout/stderr files) are now written to a folder named ‘ica_logs’ within the analysis output folder

  • Default scratch disk size attached to analysis steps reduced from 2TB to 0B to improve cost and performance of analyses. Pipelines created before ICA v2.21.0 will not be impacted

• Notifications

  • Notifications can now be updated and deleted in externally managed Projects

• API

  • Clarified on the Swagger page which sorting options apply to which paging strategy (cursor-based versus offset-based). Changed the default sorting behavior so that:

    • When no paging strategy is specified and no sort is requested, then cursor-based paging is default

    • When no paging strategy is specified and sort is requested, then offset-based paging is default

• Cohorts

  • Procedure Search Box: Users can now access additional UI functionalities for Procedures

    • Users can now access Procedure codes from OMOP

    • Improved handling of drug codes across all reports, excluding Survival comparison

  • Ingestion

    • Users now have enhanced job warning log and API status improvements

    • Users now require download permissions to facilitate the data ingestion process

  • Fetch Molecular Files: Improved import – Users can now input a folder path and select sample files individually

  • Variant Type Summary: Users can now access a new variants tab that summarizes Variant type statistics per gene

  • Added sorting and filtering capabilities to report tables, such as variants observed in genes

  • Users can now view sample barcodes, replacing internal auto-increment sample IDs in the Structural Variants table within the Genes tab

  • “Search subjects” functionality improved with flexible filtering logic that now supports partial matches against a single input string

Fixed Issues

• Data Management

  • Fixed an issue with data copy via the CLI where the file was being copied to a subfolder of the intended location instead of the specified folder

  • Resolved an issue where browser upload hangs intermittently when creating data

  • Fixed an issue where the delete popup does not always disappear when deleting data

  • Fixed an issue where GetFolder API call returns 404 error if the Create and Get operations are performed 100ms apart

  • Fixed an issue where file copy would fail if the file was located at the root level of User’s S3 storage bucket

  • Fixed an issue causing data linked from externally managed projects to be incorrectly excluded from the list project data API response

  • Fixed an issue where User cannot use data URNs to identify the destination folder when interacting with copy data API endpoints

  • Bundles: Fixed an issue where clicking the back button before saving a new bundle leads to inconsistencies

• Flow

  • Fixed an issue where pipeline documentation is not scrollable when launching pipeline

  • Fixed an issue with logfiles of a task not being available for streaming while the task is still running

  • Fixed an issue where using the 're-run' button from the analysis page reverts the storage size selection to default

  • Fixed an inconsistency where the following two endpoints would show different analysis statuses:

    • GET /api/projects/{projectId}/analyses

    • GET /api/projects/{projectId}/analyses/{analysisId}

  • Improved performance issues with UI loading data records when selecting inputs for analysis

  • Fixed a caching issue which resulted in delays when running pipelines

  • Fixed an issue where back button for analysis or pipeline details does not always direct Users back to analysis or pipelines view, respectively

  • Fixed an issue where system performance is degraded when large batches (e.g., 1,000) of data are added as input to Analyses via the graphical UI. It is recommended to start Analyses with large numbers of input files via API

• Base

  • Fixed an issue where enabling Base from a Base view other than Base Tables returned a warning message

  • Fixed an issue where Base access was not enabled when a bundle with tables is added to a project without Base (Base is automatically enabled so users can see the bundle's tables). However, access to the bundle's tables is revoked upon the deletion of Base, and was not granted again once Base was re-enabled

  • Fixed an issue where a Base job to load data into a table never finished because the file was deleted after the job started and before it finished. Now the job will end up in a Failed state

• Cohorts

  • Fixed an issue where needle plot filtered out data points reappear when zooming in the exon when a filter is in place

  • Fixed an issue where users from a different tenant who accept a project share may encounter a failure at the final step of the data ingestion process

  • Fixed an issue where users can encounter intermittent errors when browsing and typing for a gene

  • Fixed an issue where the UI hangs on large genes and returns a 502 error

2023 November 9 - ICA v2.20.1

Fixed Issues

• Data Management

  • Fixed an issue where multiple folder copy jobs with the same destination may get stuck In Progress

  • Fixed an intermittent issue where tags on the target folder for a batch data update call are not set, but are set for all child data

• Flow

  • Fixed an issue causing intermittent pipeline failures due to an infrastructure error

2023 October 31 - ICA v2.20.0

Features and Enhancements

• General

  • Navigation: If multiple regions are enabled for the same tenant, the region will be indicated in the waffle menu

  • Logging: Data transfers of BaseSpace Sequence Hub projects with data stored in ICA will be traced in ICA logs

• Cohorts

  • Disease Search Box: Added support for specifying subjects by age of onset of disease(s)

  • Drug Search Box: Added a new query builder box for Drugs

    • Ingestion: Support for Drug, drug route, etc. attached to subjects

    • Cohorts building: Users can build cohorts by specifying drugs, drug route, etc.

  • Ingestion

    • Combine different variant types during ingestion (small variants, cnv, sv)

    • Cohorts supports Illumina Pisces variant caller for hg19 VCFs

Fixed Issues

• General

  • Fixed an issue where the graphical UI hands with ha spinning wheel when saving or executing a command

  • Fixed an issue where rich text editor for Documentation tab on Pipelines, Tools, Projects and Bundles does not populate with correct styles in edit mode

• Data Management

  • Fixed an issue where multiple clicks on create data in Project API endpoint resulted in multiple requests

  • Fixed an issue where the secondary data selection screen could not be resized

  • A spinning wheel icon with ‘copying’ status is displayed at the folder level in the target Project when a folder is being copied. This applies to the actual folder itself and not for folders higher up in the hierarchy

  • Fixed an issue where API to retrieve a project data update batch is failing with 500 error when either the Technical or the User tags are updated during the batch update request

  • Fixed an issue where linking jobs fail to complete if other linking jobs are running

  • Improved performance for data transfer to support BaseSpace Sequence Hub Run transfers

  • Fixed an issue causing some folder copy jobs to remain in "Partially Succeeded" status despite being completed successfullyBundles: Fixed an issue where the URL and Region where a Docker image is available is not displayed for a Docker image Tool shared via an entitled Bundle

  • Fixed an issue where the folder copy job was getting stuck copying large amounts of big files

  • Fixed an issue where the folder counts were not matching expected counts after Data linking

  • Fixed an issue where delete data popup would occasionally not disappear after deleting data.

  • Fixed an issue with data copy where referencing data from another region would not result in immediate failure

  • Fixed issue where uploading a folder using the CLI was not working

  • Fixed an issue where a Docker image shared via an entitled Bundle can be added to another region

• Workflows

  • Fixed an issue where workflow does not fail if BCL Convert fails for a BCL Convert-only run

• Flow

  • Improved performance when batches of data up to 1000 are added as input to an Analysis

  • Nextflow engine will return exit code 55 if the pipeline runner task is preempted

  • Fixed an issue where log files cannot be opened for any steps in an analysis while the analysis is in progress

  • Fixed an issue with concurrent updates on analysis

  • Fixed an issue where unknown data inputs in the XML of an analysis are not being ignored

  • The warning, close, and machine profile icons for Tools can now be seen in the graphical CWL pipeline editor

  • Fixed an issue where user cannot expand analysis output folder if user permissions change after starting analysis. Now, if a user has the correct permissions to start an analysis, that analysis should be able to finish correctly no matter the permissions at the time it succeeds

• Base

  • Fixed an issue switching back from template to Empty Table did not clear the fields

  • Data linked from an externally managed project can be added to Base Tables

  • Fixed an issue in the graphical UI where schema definition does not scroll correctly when many columns are defined

2023 October 3 - ICA v2.19.0

Features and Enhancements

• Data Management/API

  • Added a new endpoint available to change project owner

    • POST /api/projects/{projectId}:changeOwner { “newOwnerId”:”}

  • Added a new endpoint to copy data from one project to another:

    • /api/projects/{projectId}/projectDataCopyBatch

• Data Management/CLI

  • Added the ability to copy files and folders between projects in the UI and CLI. This includes support for copying data from projects with ICA-managed storage (default) to projects with S3-configured storage.

• Flow/API

  • When starting an analysis via the API, you can specify the input files based on HTTP(s). When your analysis is done, you will see the URL corresponding to the inputs in the UI, but you will not be able to start an analysis from the UI using this URL

  • Added two new endpoints for workflow sessions:

    • Get /api/projects/{projectId}/workflowSessions

    • Get /api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs

  • Added a new endpoint to retrieve configurations from a workflow session

• Flow/CLI

  • Duplicate analyses submitted via the CLI will be avoided

• Flow

  • Removed the ability to start analyses from data and sample views in the UI where a single input is selected to start analyses in bulk

  • Flow/Autolaunch ICA Workflow Session and Orchestrated Analyses (launched by the workflow session) now saves outputs in an organized folder structure: /ilmn-analysis/<name_used_to_create_sequencer_run_output_folder>

• Base

  • The Base module has a new feature called ‘Data Catalogue’. This allows you to add usage data from your tenant/project if that data is available for you.

    • Data Catalogue views will be available and can be used in Base to query on

    • You will be able to preview and query Data Catalogue views through Base Tables and Query screens

    • The Data Catalogue will always be up to date with the available views for your tenant/project

    • Data Catalogue views cannot be shared through a Bundle

    • Data Catalogue views will also be available to team members that were added after the view was added

    • Data Catalogue views can be removed from the Base tables and corresponding project

    • By removing Base from a project, the Data Catalogue will also be removed from that project

• Cohorts: Disease Search box

  • Cohorts now includes a disease search box to search for disease concepts. This replaces the disease concept tree explorer

  • Disease search box located under a Disease tab in main Query builder

  • Search box allows for a copy/paste action of codes to be processed as separate query elements. Currently, the feature is limited to a complete valid list

  • Each disease entered into the search box is displayed as a separate query item and can be set to include or exclude.

  • Diseases in search box can be used with boolean logic in cohort creation

  • Search box allows for an auto-complete of diagnosis concepts and identifiers

  • The disease filter is included in the cohort query summary on cohort page

Fixed Issues

• Data Management

  • Data copy between ICA-managed projects and S3 storage configured projects is supported

  • Fixed an issue where storage configurations matching ICA-managed buckets would cause volume records to get associated with the wrong storage configuration in the system

• API

  • The endpoint GET/api/projects/{ProjectID}/samples/{SampleID} correctly returns all the own samples and linked samples

  • Improved handling of bulk update via API when concurrent deletion of file has occurred

• CLI

  • Fixed an issue where projectdata update tags would not update the tags

  • Fixed an issue to support adding the server-url as a parameter instead of having the config set

• Flow

  • Fixed an issue resulting in failure to send a notification resulting in a failed workflow

  • Fixed an issue where one workflow session may override another when both are executed at the same time

• Base

  • Fixed an issue where query download in JSON format returns an error

  • Added a message in the UI when a query takes longer than 30 seconds to inform the user that the query is ongoing and can be monitored in the Activity view

  • Added a section describing the Data Catalogue functionality

• Bench

  • Fixed an issue where resizing the workspace to current size would prevent users from resizing for the next 6 hours

• Cohorts

  • Fixed an issue where Gene Expression table does not display with TCGA data or for tenants with a hyphen (e.g., ‘genome-group’)

  • Fixed an issue where user had no way to delete a cohort comparison from a deleted cohort

  • Fixed an issue in the UI where multi-cohort needle plot tracks are overlapping

  • Fixed an issue causing failures during annotation step with ‘CNV’ data type when selection ‘GB=hg19’ and ‘CNV data’ for liftover; also observed with ‘SM data’ and ‘hg38’ without liftover (in APS1 and CAC1 regions) due to a ‘404 Not Found’ error.

2023 September 14 - ICA v2.18.4

Fixed Issue

• Fixed an issue uploading folders via the CLI

2023 September 8 - ICA v2.18.3

Fixed Issue

• Fixed an issue causing CWL pipelines using Docker images that do not contain bash shell executable to fail.

2023 September 7 - ICA v2.18.2

Fixed Issue

• Fixed an issue leading to intermittent system instability.

2023 September 6 - ICA v2.18.1

Fixed Issue

• Cohorts

  • Issue fixed where GTEx plot is not available for tenants with a hyphen (e.g. ilmn-demo).

2023 August 31 - ICA v2.18.0

Features and Enhancements

• General

  • Versioning: The ICA version can now be found under your user when you select "About"

  • Versioning/API: It is possible to retrieve system information about ICA, such as the current version through GET/api/systeminfo

  • Logging: When an action is initiated by another application, such as BaseSpace Sequence Hub, it will be traced as well in the ICA logs

• Data Management

  • New API endpoints are available for:

    • Creation of a data update in bulk: POST/api/projects/{projectId}/dataUpdateBatch

      • A list of data updates for a certain project: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}

      • A list of items from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items

      • A specific item from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items/{itemId} Note: Batch updates include tags, format, date to be archived and date to be deleted

• Data Management/API

  • The sequencing run information can be retrieved through its Id by using the API endpoint GET/api/sequencingRuns/{sequencingRunId}

• Flow:

  • Auto launch now supports BCL Convert v3.10.9 pipeline and both TruSight Oncology 500 v2 pipelines (from FASTQs)

  • Removed "fpga-small" from available compute types. Pipelines using "fpga-small" will use the "fpga-medium"-equivalent compute specifications instead

  • Analyses launched/tracked by BaseSpace Sequence Hub contain relevant BaseSpace information in analysis details view

• Flow/API

  • getPipelineParameters API returns parameter type in response

  • Added endpoints to retrieve and update a project pipeline definition

  • New API endpoint available to request the analyses in which a sample is being used

  • When leaving activationCodeDetailId empty when starting an analysis, the best match activation code will be used

• Flow/API/CLI

  • Include "mountPaths" field in response for API and CLI command to retrieve analysis inputs

• API

  • Two new API endpoints added to accept Terms and Conditions on a bundle:

    • GET /api/bundles/{bundleId}/termsOfUse/userAcceptance/currentUser Returns you the time of acceptance when you, the current user, accepted the Terms & Conditions.

    • POST /api/bundles/{bundleId}/termOfUse:accept

  • Add temporary credentials duration to API documentation

• Notifications

  • List of events to which you can subscribe contains new ICA notification containing analyses updates

• Bench

  • A new Bench permission is being introduced: Administrator. This permission allows users to manages existing workspaces and create new workspaces

  • The Bench Administrator role allows you to create new Bench workspaces with any permissions even if you as a Bench administrator do not have these permissions. In that case, you can create and modify the workspace, but you cannot enter that workspace. Modifying is only possible when the workspace is stopped

  • As a Bench Contributor you are not allowed anymore to delete a Bench Workspace, you need the Bench Administrator role.

• Cohorts

  • Users can now ingest raw DRAGEN bulk RNAseq results for genes and transcripts (TPM), with the option to precompute differential expression during ingestion

  • Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements

  • In multiple needle plot view, individual needle plots can now be collapsed and expanded

  • Pop-outs for needle plot variants now contain additional links to external resources, such as UCSC

  • For a given cohort, display a distribution of raw expression values (TPM per gene) for a selected attributes

  • Use of the Cohorts maintains session between core ICA and Cohorts iFrame to prevent unwanted timeouts

  • Cohorts displays structural variants that include or overlap with a gene of interest

Fixed Issues

• General

  • Collaboration: Fixed an issue where a user is presented with a blank screen when responding to a project invitation

• Data Management/API

  • Improved error handling for API endpoint: DELETE/api/bundles/{bundleId}samples{sampleId}

  • Fixed an issue where the API endpoint GET /api/samples erroneously returned a 500

  • API endpoint GET/api/projects/{projectId}/analyses now returns the correct list when filtering on UserTags whereas it previously returned too many

  • Improved retry mechanism for API endpoint to create folderuploadsession

• Data Management/CLI

  • When an upload of a folder/file is done through the CLI, it returns the information and ID of the folder/file

• Data Management

  • CreatorId is now present on all data, including subfolders

  • Improved external linking to data inside ICA using deep linking

  • Improved error handling when creating folders with invalid characters.

  • Fixed an inconsistency for URN formats on output files from Analyses. This fix will apply only for analyses that are completed starting from ICAv2.18.0

  • Improved resilience in situations of concurrent linking and unlinking of files and folders from projects

  • It is only possible to delete a storage configuration if all projects that are using this storage configuration have been hidden and are not active projects anymore

  • Improved accuracy of the displayed project data size. Prior cost calculations were accurate, but the project data size visualization included technical background data

  • Fixed an issue where there is a discrepancy in number of configurations between Storage->Configurations and Configurations-> Genomics.Byob.Storage Configuration view

• Flow/API

  • Improved error handling when invalid project-id is used in API endpoint GET /api/projects/{projectId}/pipelines

  • Fixed an issue when an Analysis completed with error "incomplete folder session", the outputs of the Analysis are not always completely listed in the data listing APIs

  • Updated ICA Swagger Project > createProject to correctly state that the analysis priority must be in uppercase

• Flow

  • When a spot instance is configured, but revoked by AWS, the pipeline will fail and exit code 55 is returned

  • Fix to return meaningful error message when instrument run ID is missing from Run Completion event during an auto launched analysis

  • Improved parallel processing of the same analysis multiple times

• Base

  • Improved error handling when creating queries which use two or more fields with the same name. The error message now reads "Query contains duplicate column names. Please use column alias in the query"

  • Fixed an issue where queries on tables with many entries fail with NullPointerException

• Bench

  • Clarified that changes to Bench workspace size only take effect after a restart

• Cohorts

  • Fixed issue where counts of subjects are hidden behind attribute names

  • Fixed issue where the state of checked files are not retained when selecting molecular files that are in multiple nested folders

  • Fixed issue where projects that contain files from linked bundles cause a time out, resulting in users not being able to select files for ingestion

  • Fixed an issue where the 'Import Jobs' page loaded within the Data Sets frame, depending on where the import was initiated

  • Fixed an issue in the Correlation plat where x-axis counts were hidden under attribute names

  • Fixed an issue where users were previously incorrectly signed out of their active sessions

2023 August 3 - ICA v2.17.1

Fixed Issues

• Fixed an issue causing analyses requesting FPGA compute resources to experience long wait times (>24h) or not be scheduled

2023 June 27 - ICA v2.17.0

Features and Enhancements

• Data Management

  • Performance improvements for data link and unlink operations – Larger and more complex folders can now be linked in the graphical UI, and progress can be monitored with a new visual indication under Activity > Batch Jobs

• Notifications

  • Notifications are now available for batch job changes

• Flow

  • Increased the allowed Docker image size from 10GB to >20GB

  • CWL: Added support for javascript expressions “ResourceRequirements” fields (i.e., type, size, tier, etc.) in CWL Pipeline definitions

• Flow/API

  • Added support for using Pipeline APIs to query Pipelines included in Entitled Bundles (i.e., to retrieve input parameters)

  • Added support for providing S3 URLs as Pipeline data inputs when launching via the API (using storage credentials)

  • Added support for specifying multi-value input parameters in a Pipeline launch command

• Bench

  • Project and Tenant Administrators are now allowed to stop running Workspaces

• Cohorts

  • Enhanced ingestion workflow to ingest RNAseq raw data from DRAGEN output into backend Snowflake database

  • Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements

  • Multi-Cohort Marker Frequency - Added Multi-Cohort Marker Frequency tab allowing users to compare expression data across up to four Cohorts at the gene level

  • Multi-Cohort Marker Frequency includes a pairwise p-value heat map

  • Multi-Cohort Marker Frequency - Includes frequencies for Somatic and Copy Number Variants

  • Tab added for a multi-cohort marker frequency analysis in cohort comparisons

  • Multi-Cohort Needle Plot - Added new tab in the Comparison view with vertically aligned needle plots per cohort for a specified gene, allowing collapsible and expandable individual needle plots

  • Additional filter logic added to multi-cohort needle plot

  • Improved DRAGEN data type determination during ingestion allowing for multiple variant type ingestion

  • Enhanced list of observed variants with grouped phenotypes and individual counts, including a column for total sample count; tooltips/pop-outs provide extended information

  • Updates to needle plot link outs

  • Improved the Comparison feature by optimizing API calls to handle subjects with multiple attributes, ensuring successful loading of the page and enabling API invocation only when the user selects or expands a section

  • Removed unused columns (genotype, mrna_feature_id, allele1, allele2, ref_allele, start_pos, stop_pos, snp_id) from annotated_somatic_mutations table in backend database

  • Refactored shared functionality for picking consequence type to reduce code duplication in PheWAS-plot and GWAS-plot components

  • Invalid comparisons on the Comparisons page are now grayed out and disabled This improvement prevents the selection of invalid options

  • Automatic retry of import jobs when there are failures accessing data from ICA API

Fixed Issues

• General

  • Navigation: Removed breadcrumb indication in the graphical UI

• Data Management

  • The content of hidden Projects can now be displayed

  • Fixed the TimeModified timestamp on files

  • Bundles: Resolved issues when linking a large number of files within a folder to a Bundle

• Flow

  • Single values are now passed as a list when starting an Analysis

  • Pipelines will succeed if the input and output formats specified on the pipeline level match at the Tool level

  • Fixed an issue causing Analysis failures due to intermittent AWS S3 network errors when downloading input data

  • CWL: Improved performance on output processing after a CWL Pipeline Analysis completes

  • Flow/UI: Mount path details for Analysis input files are now visible

  • Flow/UI: Improved usability when starting an Analysis by filtering entitlement options based o inputs selected and available entitlements

• Flow/API

  • List of Analyses can now be retrieved via the API based on filters for UserReference and UserTags

• Base

  • Fixed an issue where the Scheduler continues to retry uploading files which cannot be loaded

• Bench

  • Resolved an issue when attempting to access Workspaces with multi-factor authentication (MFA) enabled at the Tenant-level

• API

  • Improved error messaging for POST /api/projects/{projectId}/data/{dataId}:scheduleDownload

• Cohorts

  • Fixed issue where Correlation bubble plot not showing for any projects intermittently

  • Fixed issue where importing Germline/hg19 test file did not load variants for a specific gene in the Needle plot due to missing entries in the Snowflake table

  • Fixed a bug causing an HTTP 400 error while loading the Cohort for the second time due to the UI passing "undefined" as variantGroup, which failed to convert to the VariantGroup Enum type

  • Fixed issue where scale (y-axis) of needle plot is changed even if value of sample count gnomAD frequency is not accepted

  • Fixed an issue where no data was generated in the Base Tables after a successful import job in Canada - Central Region (CAC1)

  • Fixed issue where long chart axis labels overlap with tick marks on graph

2023 May 31 - ICA v2.16.0

Features and Enhancements

• General

  • Navigation: Updated URLs for Correlation Engine and Emedgene in the waffle menu

  • Authentication: Using POST /api/tokens:refresh for refreshing the JWT is not possible if it has been created using an API-key.

  • Authentication: Improved error handling when there is an issue reaching the authentication server

  • Authentication: Improved usability of "Create OAuth access token" screen

• Data Management

  • You can now select 'CYTOBAND' as format after file upload

  • Added support for selecting the root folder (of the S3 bucket) for Projects with user-managed

  • Added support for creating an AWS Storage Configuration with an S3 bucket with Versioning enabled

• Auto-launch

  • Added technical tags for upstream BaseSpace Run information to auto-launched analyses

  • Added support for multiple versions of BCL Convert for auto-launched analyses

• Flow

  • Added support for '/' as separator in CWL ResourceRequirements when specifying Compute Type

• Flow/API

  • The API to retrieve analysis steps now includes exit code for completed steps

• Bench

  • Restricted Workspaces (Open or Restricted) always allow for access to Project Data within the Workspace

  • Restricted Bench workspaces have limited access through whitelisted URLs that are checked before entry

  • Restricted Bench Workspaces allow for Open or Restricted workspaces. Restricted workspaces do not have access to the internet except for user-entered whitelist URLs

Fixed Issues

• Data Management

  • Upload for files names including spaces is now consistent for connector and browser upload. We do still advise not to use spaces in file names in general

  • Fixed search functionality in Activity > Data Transfers screen

  • Improved performance on opening samples

  • Fixed an issue where reference data in download tab initiates an unexpected download

  • Fixed intermittent issue where the Storage configuration within a Project can go into Error status and can block users from creating records such as folders and files

  • Service Connector: Improved error message for DELETE/api/connectors/{connectorId}/downloadRules/{downloadRuleId}

• Data Management/API

  • Improved error handling for API endpoints: Delete/api/projects/ {projectId}/bundles/{bundleId} and POST/api/projects/{projectId}/bundles/{bundleId}

  • Improved error handling for POST/api/projects/{projectId}/base:ConnectionDetails

• Bundles

  • Fixed an issue where the Table view in Bundles is not available when linking to a new Bundle version

  • Fixed an issue where linking/unlinking a Bundle with Base Tables could result in errors

• Bundles/API

  • Improved error handling for DELETE/api/bundles/{bundleId}/tools/{toolId} and POST/api/bundles/{bundleId}/tools/{toolId}

  • Improved error message for POST/api/bundles/{bundleId}/samples/{sampleId}

• Notifications/API

  • Custom subscriptions with empty filter expressions will not fail when retrieving them via the API

  • Improved error handling for POST/api/projects/{projectId}/notificationSubscriptions

  • Improved notification for Pipeline success events

• Flow

  • When the input for a pipeline is too large, ICA will fail the Analysis and will not retry

  • Fixed issue where analysis list does not search-filter by ID correctly

  • Improved error handling when issues occur with provisioning resources

  • When retry succeeds in a Nextflow pipeline, exit code is now '0' instead of '143'

• Flow/API

  • Fixed an issue causing API error when attempting to launch an Analysis with 50,000 input files

  • Improved pipeline error code for GET/api/projects/{projectId}/pipelines/{pipelineId} when already unlinked pipeline Id is used for API call

  • Fixed an issue where Analyses could not be retrieved via API when the Pipeline contained reference data and originated from a different tenant

  • Fixed filtering analyses on analysisId. Filtering happens via exact match, so part of the Id won't work

• Bench/CLI

  • Fixed issue where the latest CLI version was not available in Bench workspace images

• Cohorts

  • Fixed an issue where CNV data converted from hg19 to hg38 do not show up in Base table views

  • Fixed an issue accounting for multiple methods of referring to the alternate allele in a deletion from Nirvana data

  • Fixed intermittent issue where GWAS ingestions not working after Base enabled in a project.

2023 May 2 - ICA v2.15.1

Fixed Issue

• Fixed an issue causing incorrect empty storage configuration dropdown during Project creation when using the “I want to manage my own storage” option for users with access to a single region

2023 April 25 - ICA v2.15.0

Features and Enhancements

• General

  • General availability of sequencer integration for Illumina sequencing systems and analysis auto launch

  • General usability improvements in the graphical interface, including improved navigation structure and ability to switch between applications via the waffle menu in the header

  • Storage Bundle field will be auto-filled based on the Project location that is being chosen if multiple regions are available

  • Event Log entries will be paged in the UI and will contain a maximum of 1,000 entries. Exports are limited to the maximum number of entries displayed on the page.

  • Read-only temporary credentials will be returned when you are not allowed to modify the contents of a file

  • The ICA UI will only allow selection of storage bundles belonging to ICA during Project creation, and the API will only return storage bundles for ICA

• Notifications

  • Creating Project notifications for BaseSpace externally managed projects is now supported

• Flow

  • Allow attached storage for Pipeline steps to be set to 0 to disable provisioning attached storage and improve performance

• Cohorts

  • GRCh37/hg19-aligned molecular data will get converted to GRCh38/hg38 coordinates to facilitate cross-project analyses and incorporating publicly available data sets.

• API

  • Project list API now contains a parameter to filter on (a) specific workgroup(s)

  • Two new API endpoints are added to retrieve regular parameters from a pipeline within or without a Project context

Fixed Issues

• General

  • Optimized price calculations resulting in less overhead and logging

  • Improved error handling:

    • during Project creation

    • of own storage Project creation failures.

    • to indicate connection issue with credential

    • for graphical CWL draft Pipelines being updated during an Analysis

  • Improved error messaging in cases where the AWS path contains (a) special character(s)

  • Fixed an issue causing errors when navigating via deep link to the Analysis Details view

• Data Management

  • Fixed an issue causing data records to remain incorrectly in Unarchiving status when an unarchive operation is requested in the US and Germany regions

• API

  • Fixed returning list of unlinked data in a sample that was linked before in GET/api/projects/{projectId}/data

  • Fixed error for getSampleCreationBatch when using status filter

• CLI

  • Unarchive of folders is supported when archive or unarchive actions are not in progress for the folder

  • Improved error message to indicate connection issue with credentials

• Flow

  • Fixed an issue causing incorrect naming of Analysis tasks generated from CWL Expression Tools

  • Fixed an issue when cloning Pipelines linked from Entitled Bundles to preserve the original Tenant as the Owning Tenant of the cloned Pipeline instead of the cloning user’s Tenant

  • Fixed an issue causing outputs from CWL Pipelines to not show in the Analysis Details despite being uploaded to the Project Data Analysis output folder when an output folder is empty

  • When a Contributor starts an Analysis, but is removed afterwards, the Analysis still runs as expected

  • Fixed an issue where Analyses fail where Nextflow is run a second time

  • Fixed an issue causing API error when attempting to launch an Analysis with up to 50,000 input files

  • Fixed an issue causing degraded performance in APIs to retrieve Analysis steps in Pipelines with many steps

  • Fixed an issue causing Analysis failure during output upload with error “use of closed network connection”

  • Fixed an issue causing disk capacity alter log to not show when an Analysis fails due to disk capacity and added error message

  • Fixed an issue preventing cross-tenant users from being able to open a shared CWL pipeline

• Base

  • Improved target Table selection for schedulers to be limited to your own Tables

• Bench

  • Fixed an issue causing Workspaces to hang in the Starting or Stopping statuses

• Cohorts

  • Now handles large VCFs/gVCFs correctly by splitting them into smaller files for subsequent annotation by Nirvana

2023 March 28 - ICA v2.14.0

Features and Enhancements

• General

  • Added a limit to Event Log and Audit UI screens to show 10,000 records

• API

  • Parent output folder can be specified in URN format when launching a Workflow session via the API

• Flow

  • Reduced Analysis delays when system is experiencing heavy load

  • Improved formatting of Pipeline error text shown in Analysis Details view

  • Users can now start Analyses from the Analysis Overview screen

  • Superfluous “Namespace check-0” step was removed to reduce Analysis failures

  • Number of input files for an Analysis is limited to 50,000

  • Auto launched Workflow sessions will fail if duplicate sample IDs are detected under Analysis Settings in the Sample Sheet

• Base

  • Activity screen now contains the size of the query

• Cohorts

  • Detect and Lift Genome Build: Cohorts documentation provides set-up instructions to convert hg19/GRCh37 VCFs to GRCh38 before import into Cohorts.

  • Attribute Queries: Improved the user experience choosing a range of values for numerical attributes when defining a cohort

  • Export Cohort to ICA Project Data: Improved the user experience exporting list of subjects that match cohort definition criteria to their ICA project for further analysis

  • Ingest Structural Variants into database

    • The Cohorts ingestion pipeline supports structural variant VCFs and will deposit all such variants into an ICA Base table if Base is enabled for the given project

    • Structural variants can be ingested and viewed in base tables

  • Needle Plot Enhancements

    • Users can input a numerical value in the Needle Plot legend to display variants with a specific gnomAD frequency percentage or sample count

    • The needle plot combines variants that are observed among subjects in the current project as well as shared and public projects into a single needle, using an additional shape to indicate these occurrences

    • Needle Plot legend color changes for Variant severity; pathogenic color coding is the same as the color coding in the visualization; differentiating hue between proteins and variants; and other color coding changes.

    • Needle plot tool tips that display additional information on variants and mutations are now larger and modal

    • The needle plot now allows to filter by gnomAD allele frequency and sample count in the selected cohort. Variants include links to view a list of all subjects carrying that variant and export that list.

  • Remove Samples Individually from Cohorts

    • Exclude individual subjects from a cohort and save the refined list

    • The subjects view allows users to exclude individual subjects from subsequent analyses and plots and save these changes Subject exclusions are reset when editing a cohort

  • Subject Selection in Analysis Visualization: Users can follow the link for subject counts in the needle plot to view a list of subjects carrying the selected variant or mutation.

  • UI/UX: Start and End time points are available as a date or age with a condition attribute in the subject data summary screen.

Fixed Issues

• General

  • Improved resilience against misconfiguration of the team page when there is an issue with Workgroup availability

  • Removed ‘IGV (beta)’ button from ‘View’ drop down when selecting Project Data in UI

• Data Management

  • Improved handling of multi-file upload when system is experiencing heavy loads

  • Fixed an issue to allow upload of zero-byte files via the UI

  • Fixed issue where other Bundles would not be visible after editing and saving your Bundle

• API:

  • Improved error handling for API endpoint: POST /api/projects/{projectId}/analysisCreationBatch

  • Improved performance of API endpoint: getbestmatchingfornextflow

• Flow

  • Fixed an issue causing Analysis output mapping to incorrectly use source path as target path

  • Fixed an issue where the UI may display incorrect or invalid parameters for DRAGEN workflows which do not accurately show the true parameters passed. Settings can be confirmed by looking at the DRAGEN analysis log files.

• Base

  • “Allow jagged rows” setting in the Scheduler has been replaced with “Ignore unknown values” to handle files containing records with more fields than there are Table columns

  • Improved Base Activity view loading time

  • Fixed an error message when using the API to load data into a Base Table that has been deleted

• Bench

  • Fixed an issue resulting in incorrect Bench compute pricing calculations

  • Fixed an issue preventing building Docker images from Workspaces in UK, Australia, and India regions

  • Fixed an issue where /tmp path is not writeable in a Workspace

• Cohorts

  • Fixed issue where the bubble plot sometimes failed to display results even though the corresponding scatter plot showed data correctly.

  • The order of messages and warnings for ingestion jobs was not consistent between the UI and an error report sent out via e-mail.

  • The UI now displays any open cohort view tabs using shortened (“…”) names where appropriate

  • Issue fixed where ingestions with multiple errors caused halting to the ingestion queue.

  • The needle plot sometimes showed only one source for a given variant as opposed to all projects in which the variant had been observed in.

  • Issue fixed with unhandled genotype index format in annotation file to base database table conversion

  • Status updates via e-mail sometimes contained individual error messages or warnings without a text.

  • Fixed issue where items show in needle plot with incorrect numbering on the y-axis.

  • Fixed performance issue with subject count.

  • Widget bar-chart counts are intermittently cut off over four digits.

  • Fixed slowness when switching between tabs in query builder

2023 March 23 - ICA v2.13.2

Fixed Issue

• Fixed issue with BaseSpace Free Trial and Professional users storing data in ICA

2023 March 9 - ICA v2.13.1

Fixed Issue

• Fixed an issue resulting in analysis failures caused by a Kubernetes 404 timeout error

2023 February 28 - ICA v2.13.0

Features and Enhancements

• General *

  • Each tenant supports a maximum of 30,000 Projects

  • .MAF files are now recognized as .TSV files instead of UNKNOWN

  • Added VCF.IDX as a recognized file format

  • General scalability optimizations and performance improvements

• API

  • POST /api/projects/{projectId}/data:createDownloadUrls now supports a list of paths (in addition to a list of IDs)

Fixed Issues

• General

  • Fixed an issue preventing the ‘Owning Project’ column from being used outside of Project

  • Fixed an issue allowing the region of a Project to be changed. Changing the region of a resource is not supported

  • Strengthened data separation and improved resilience against cross-Project metadata contamination

• Bundles

  • After creating a new Bundle the user will be taken to the Bundle Overview page

• Data Management

  • Fixed an issue which prevented changing the format of a file back to UNKNOWN

  • Fixed an issue causing inaccurate upload progress to be displayed for UI uploads. The Service Connector or CLI are recommended for large file uploads.

  • Fixed an issue showing an incorrect status for data linking batch jobs when data is deleted during the linking job

  • Service Connector: Fixed an issue allowing download of a Service Connector when no operating system is set

  • Service Connector: Cleaned up information available on Service Connectors by removing empty address information fields

• API

  • Fixed date formatting for GET /api/eventLog (yyyy-MM-dd’T’HH:mm:ss.SSS’Z’)

  • Fixed an issue where the GET users API was not case sensitive on email address

  • Fixed an issue causing the metadata model to be returned twice in PSOT /api/projects/{projectId}/samples:search

  • Fixed the listProjects API 500 response when using the pageoffset query parameter

  • The searchProjectSamples API returns Sample metadata for Samples shared via a Bundle

  • Fixed an issue causing createProjectDataDownloadUrls API 400 and 502 errors when server is under load

• Flow

  • Fixed analysis failures caused by kubernetes 404 timeout error

  • Fixed an issue where Workflwos would prematurely report completion of an Analysis

  • Improved Pipeline retry logic to reduce startup delays

  • Fixed an issue where Nextflow pipelines were created with empty files (Nextflow config is allowed to be empty)

  • Removed the 1,000 input file limitation when starting an Analysis

  • Improved the performance of status update messages for pipelines with many parallel steps

  • Fixed an issue with overlapping fields on the Analysis Details screen

  • Deactivated the Abort button for Succeeded analyses

• Base

  • Fixed an issue where Pipeline metadata was not captured in the metadata Table generated by the metadata schedule

  • Error logging and notification enhancements

• Bench

  • Fixed an issue where Workspaces could be started twice

  • Fixed an issue where the system checkpoint folder was incorrectly created in Project data when opening a file in a Workspace

2023 February 13 - ICA v2.12.1

Features and Enhancements

• Analysis system infrastructure updates

2023 January 31 - ICA v2.12.0

Features and Enhancements

• Added ability to refresh Batch Jobs updates without needing to leave the Details screen.

• Projects will receive a job queuing priority which can be adjusted by an Administrator.

• The text "Only showing the first 100 projects. Use the search criteria to find your projects or switch to Table view." when performing queries is now displayed both on the top and bottom of the page for more clarity.

• API: Added a new endpoint to retrieve download URLs for data: POST/api/projects/{projectId}/data:createDownloadUrls

• API: Added support for paging of the Project Data/getProjectDataChildren endpoint to handle large amounts of data.

• API: Added anew endpoint to deprecate a bundle (POST /api/bundles/{bundleId}:deprecate)

• API: If the API client provides request header "Accept-Encoding: gzip", then the API applies GZIP compression on the JSON response. This way the size of the response is significantly smaller which improves the download time of the response, resulting in faster end-to-end API calls. In case of compression the API also provides header "Content-Encoding: gzip" in the response, indicating that compression was effectively applied.

• Flow: Optimized Analysis storage billing, resulting in reduced pipeline charges.

• Flow: Internal details of a (non-graphical) pipeline marked ‘Proprietary’ will not be shared with users from a different tenant.

• Flow: A new grid layout is used to display Logs for Analyses with more than 50 steps. The classic view is retained for analyses with 50 steps or less, though you can choose to also use the grid layout by means of a grid button on the top right on the Analysis Log tab.

• CLI: Command to launch a CWL and Nextlfow Pipeline now contains the mount path as a parameter.

• CLI: Version command now contains the build number.

• CLI: Added support for providing the nextflow.config file when creating a Nextflow pipeline via CLI.

• API: HTML documentation for aPipeline can now be returned with the following requests:

  • GET /api/pipelines/{pipelineId}/documentation/HTML

  • GET /api/projects/{projectId}/pipelines/{pipelineId}/documentation/HTML

• API: Added a new endpoint for creating and starting multiple analyses in batch: POST /api/projects/{projectId}/analysisCreationBatch

• Flow: Linking to individual Analyses and Workflow sessions is now supported by /ica/link/project//analysis/ and /ica/link/project//workflowSession/

• Cohorts: Users can now export subject lists to the ICA Project Data as a file.

• Cohorts: Users can query their ingested data through ICA Base. For users who already have ingested private data into ICA Cohorts, another ingestion will need to happen prior to seeing available database shares. Customers can contact support to have previously ingested data sets available in Base.

• Cohorts: Correlation bubble plot counts now link to a subject/sample list.

Fixed Issues

• Tooltip in the Project Team page provides information about the status of an invite

• ‘Resend invite’ button in the Project Team page will become available only when the invite is expired instead of from the moment the invite is being send out

• Folders, subfolders and files all contain information about which user created the data

• Files and folders with UTF-8 character are not supported. Please look at the documentation on how to recover from it in case you already have used them.

• Improved performance for creating or hiding a Project in a tenant with many Projects

• Service Connector: Updated information in the Service Connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format"

• Service Connector: Folders within a Bundle can be downloaded via the Service Connector

• Service Connector: Upload rules can only be modified in the Project where they apply

• Service Connector: A message describes when a file is skipped during upload because it already exists in the Project

• Service Connector: Fixed an issue where opening the Connectivity tab occasionally results in a null pointer error

• Service Connector: Fixed an issue causing excessive logging when downloading files with long file paths

• Service Connector: Fixed an issue where the Service Connector log may contain spurious errors which do not impact data transfers

• Existing storage configurations are displayed and accessible via API and UI

• Newly added storage configurations do no longer remain in ‘Initializing’ state

• Fixed error when creating a storage configuration with more than 63 characters

• Clicking on a Data folder in flat mode will now open the details of the folder

• Only Tools in Released state can be added to a Bundle

• Fixed issue preventing new Bundle versions to be created from Restricted Bundles

• Deprecated Bundles are displayed upon request in card and table view

• Bundles view limited to 100 Bundles

• API: Fixed the API spec for ProjectDataTransfer.getDataTransfers

• API: Fixed an issue with the projectData getChildren endpoint which returned incorrect object and pagination

• API: Fixed an issue where multiple clicks on Create sample batch API endpoint resulted in multiple requests

• API: POST /api/projects/{projectId}/data/{dataId}:scheduleDownload can now also perform folder downloads

• API: Improved information on the Swagger page for GET /api/pipelines, GET/api/projects/{projectId}/pipelines, and GET/api/projects/{projectId}/pipelines/{pipelineId}

• API: Fixed and issue when a user provides the same input multiple times to a multi-value input on an analysis run, that input is only passed to the pipeline once instead of multiple times: POST /api/projects/{projectId}/analysis:nextflow

• CLI: Copying files in the CLI from a local folder on MacOS to your Project can result in both the desired file and the metadata file (beginning with ‘./’) being uploaded. The metadata file can safely be deleted from the Project

• CLI: Hardened protection against accidental file overwriting

• CLI: Improved handling for FUSE when connection to ICA is lost

• CLI: icav2 projectdata mount –list shows updated list of mounted Projects

• CLI: Paging improvements made for project list, projectanalyses list, and projectsdata list

• CLI: When there is no config or session file the user will not be asked to create one for icav2 config reset and icav2 config get

• CLI: Fixed an issue where Bundle data could not be seen through FUSE in Bench

• CLI: Fixed an error message when missing config file upon entering the Project context

• CLI: The unmount is possible without a path and will work via the stored Project ID or with a folder path resulting in an unmount of that path

• CLI: Fixed an error when creating a Pipeline using URN for Project identifier

• CLI: Attempting to delete a file from an externally-managed project returns an error indicating this not allowed

• CLI: Fix to delete session file when config file is not detected

• CLI: Paging option added to projectsamples list data

• CLI: Fixed “Error finding children for data” error in CLI when downloading a folder

• CLI: projectdata list now returns the correct page-size results

• Flow: Fixed handling of special characters in CWL pipeline file names

• Flow: Fixed an issue where task names exceeding 25 characters cause analysis failure in CWL pipelines

• Flow: Fixed an issue which prevented requests for economy tier compute

• Flow: Fixed an issue limiting CWL workflow concurrency to two running tasks

• Flow: Fixed an issue where analysis file inputs specified in the input.json with ‘location’set to an external URL cause to CWL pipelines to fail

• Flow: Fixed an issue resulting in out of sync Pipeline statuses

• Flow: Improved Nextflow engine resiliency, including occurrences where Nextflow pipelines fail with ‘pod 404 not found’ error

• Flow: Fix issue with intermittent system runtime failures incorrectly causing analysis failures

• Flow: Fixed an issue where links to Analysis Details returned errors

• Flow: Enabled scrolling for Pipeline documentation

• Flow: Improved performance for handling analyses with large numbers of inputs

• Flow: Improved handling of hanging Analyses

• Flow: Improved error messages for failed Pipelines

• Flow: Added documentation on how to use XML configuration files for CWL Pipelines

• Flow: Duplicate values for multi-value parameters are no longer automatically removed

• Flow: Correct exit code 0 is shown for successful Pipeline steps

• Base: Fixed an issue so that only users with correct permissions are allowed to retrieve a list of Base tables

• Base: Fixed an issue with metadata scheduler resulting in a null pointer

• Base: An empty Table description will not return an error when requesting to list all Tables in a Project

• Base: Jobs failed with an error containing 'has locked table' are not shown on the Base Job activity list. They can be displayed by selecting the 'Show transient failures' checkbox at Projects > Activity > Base Jobs.

• Base: Users can see Schedulers and their results for the entire tenant if created by a tenant administrator in their project, but not create, edit or run them

• Base: Fixed an issue preventing data format change in a schedule

• Base: Fixed an issue preventing exporting data to Excel format

• Bench: Improved handling to prevent multiple users in a single running Workspace

• Bench: Fixed an issue causing Workspaces to be stuck in "Starting" state

• Bench: Fixed an issue where usage does not showing up on usage CSV-based report

• Bench: Fixed an issue where Bundle data could not be seen via the Fuse driver

• Bench: Users can now consistently exit Workspaces with a single click on the ‘Back’ button.

• Bench: After leaving a Workspace by clicking on the ‘Back’ button, the Workspace will remain in a ‘Running’ state and become available for a new user to access

• Bench: Workspaces in a ‘Stuck’ state can be manually changed to ‘Error’ state, allowing users to restart or delete them

• Cohorts: Fixed issue where file system cleanup not occurring after delete.

• Cohorts: Fixed sign in and authentication issues in APN1 region.

• Cohorts: Fixed issue where gene filter missing when editing a cohort and removing the edited filter and cancelling. The filter was preserved and should not have been.

• Cohorts: Fixed issue where users see an application tile in the Illumina application dashboard selection screen called "Cohort Analysis Module".

• Cohorts: Correlation: Fixed issue, Data type selections shows half when loading the search result

• Cohorts: Fixed issue, Users will see an application tile on the Connected Platform home page screen called “Cohort Analysis Module” if the Cohorts module is added to the domain. Users should not enter the ICA Cohorts through this page. They should enter through ICA."