Release Notes
2024 March 7 - ICA v2.23.1
Fixed Issues
Fixed an issue where automated analysis after sequencing run in non-US regions may fail for certain analysis configurations
2024 February 28 - ICA v2.23.0
Features and Enhancements
Data Management
The --exclude-source-path flag has been added to the ‘project data download’ command so that subfolders can be downloaded to the target path without including the parent path
The system automatically re-validates storage credentials updated in the graphical UI
Added a new API endpoint to validate storage configurations after credentials are changed: /api/storageConfigurations/{storageConfigurationId}:validate
Notifications
Added support for multi-version notification event payloads corresponding to versioned API response models
Flow
(API) Improved the analysis-dto by adding a new POST search endpoint as a replacement for the search analysis GET endpoint. The GET endpoint will keep working but we advise using the new POST endpoint.
Improved analysis statuses to reflect the actual status more accurately
Parallelized analysis input data downloads and output data uploads to reduce overall analysis time
No scratch size is allocated if tmpdirMin is not specified
Cohorts
Performance improvements of the ingestion pipeline
Performance improvements to subject list retrieval
Increased the character limit of ingestion log messages to the user
Fixed Issues
Data Access & Management
Fixed an issue where the target user cannot see analysis outputs after a successful transfer of analysis ownership in BaseSpace Sequence Hub
Update the API Swagger documentation to include paging information for: /projects/{projectId}/samples/{sampleId}/data
Fixed an issue resulting in errors when creating a new bundle version
Fixed an issue where the GET API call with the ‘Sort’ parameter returns an error when multiple values are separated by commas followed by a space
Fixed an issue where adding the –eligible-link flag to the ‘projectdata list’ API endpoint caused other flags to not work correctly
Added cursor-based pagination for the ‘projectdata list’ API endpoint
Fixed an issue with the entitled bundles cards view where the region is cut off when the Status is not present
Fixed an issue where bundle filtering on categories did not work as expected
Fixed an issue where file copy across tenants did not work as expected
Added a cross-account permission check so that file copy jobs fail when the cross-account set up is missing instead of being retried indefinitely
Fixed an issue where ‘Get Projects’ API endpoint returns an error when too many projects are in the tenant
Fixed an issue where the UpdateProject API call (PUT /api/projects/{projectId}) returns an error when technical tags are removed from the request
Fixed an issue where users need to confirm they want to cancel an action multiple times when clicking the back button in the graphical UI
Fixed an issue where clicking into a new version of a bundle from the details view does not open the new version, and instead directs to the bundle card view
Flow
Fixed an issue where the analysis logs are returned in the analysis screen “outputs” section and included in the getAnalysisOutputs API response. The log output is no longer considered as part of the analysis outputs
Analysis history screen has been removed
Fixed an issue resulting in inability to retrieve pipeline files via the API when the pipeline is shared cross-tenant
Fixed an issue where the API endpoint to retrieve files for a project pipeline would not return all files for pipelines created via CLI or API
Fixed an issue where the API does not check the proprietary flag of a pipeline before retrieving or downloading the pipeline files
Base
The ‘Download’ button is available to download Base activity data locally (and replaces the non-functional ‘Export’ button for restricted bundles)
Fixed an issue resulting in missing ICA reference fields in table records if the file was loaded into the table with no metadata
Improved consistency of the references included in the scheduler
Bench
Users are now logged out from a terminal window opened in a workspace after a period of inactivity
Fixed an issue where permissions could not be enabled after a workspaces has been created
Fixed an issue where a Contributor could not start/stop a workspace
Cohorts
Fixed an issue where large genes with subjects with large sets of diagnostic concepts cause a 503 error
Fixed an issue where the value in tumor_or_normal field in the phenotype table in the database is not set properly for germline and somatic mutation data
Resolved a discrepancy between the number of samples reported when hovering over the needle plot and the variant list
Documentation
General updates and improvements to reflect changes in graphical user interface
Added information on how to add scratch space to all processes in a pipeline
Updated the list of addresses which need to be whitelisted for the service connector
Added the 500GB limit per file for the FUSE driver
Batch jobs which have a final state such as Failed or Succeeded are removed from the activity list after 7 days
Updated the list of Notification Subscriptions to include all possible notifications
For Nextflow version 20.10.10, using the ‘copy’ method in thepublishDir directive for uploading output files that consume large amounts of storage may cause workflow runs to complete with missing files. Added information on workarounds to prevent this from occurring.
Updated the CLI Command index to reflect addition of --exclude-source-path flag for ‘project data download’ command
Added a video tutorial to demonstrate the end-to-end user flow for planning a NovaSeq X run with BaseSpace Run Planner and performing the run with data storage and automated analysis in ICA
Updated the list of auto-launch analysis workflows in the Sequencer Reference table
2024 January 31 - ICA v2.22.0
Features and Enhancements
Users can now access the system via https://ica.illumina.com or https://ica.illumina.com/ica
Data Management
Users are now able to revalidate storage configurations in an Error state
Improved existing endpoints and added new endpoints to link and unlink data to a bundle or a project in batch:
POST /api/projects/{projectId}/dataUnlinkingBatch
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items/{itemId}
Flow
Analyses started via the API can now leverage data stored in BaseSpace Sequence Hub as input
ICA now supports auto-launching analysis pipelines upon sequencing run completion with run data stored in BaseSpace Sequence Hub (instead of ICA)
Updated the API for creating pipelines to include "proprietary" setting, which hides pipeline scripts and details from users who do not belong to the tenant which owns the pipeline and prevents pipeline cloning.
Cohorts
Added support for partial matches against a single input string to the “Search subjects” flexible filtering logic
Users can now view an overview page for a gene when they search for it or click on a gene in the marker frequency charts
ICA Cohorts includes access to both pathogenic and benign variants, which are plotted in the “Pathogenic variants” track underneath the needle plot
Ingestion: UI notifications and/or errors will be displayed in the event of partially completed ingestions
Users can share cohort comparisons with any other users with access to the same project
Fixed Issues
General
Improved the project card view in the UI
Fixed an issue with user administration where changing the permissions of multiple users at the same time would result in users receiving Invalid OAuth access token messages
Data Access & Management:
Improved the error message when downloading project data if the storage configuration is not ready for use
Fixed an issue causing Folder Copy jobs to time out and restart, resulting in delays in copy operations
Fixed an issue where only the Docker image of the first restricted bundle that was added could be selected
Improved the performance of folder linking with "api/projects/{ProjectID}/dataLinkingBatch"
The URL for links for "post/api/bundles" endpoint can be up to 2048 characters long
Improved the error response when using offset-based paging on API responses which contain too much data and require cursor-based paging
Fixed an issue resulting in failures downloading data from CLI using a path
The correct error message is displayed if the user does not have a valid subscription when creating a new project
Fixed an issue where changing ownership of a project does not change previous owner access for Base tables
Flow:
Input parameters of pipelines are now displayed in the "label (code)" format unless there is no label available or the label equals the code, in which case only the code is shown
Fixed an issue where multiple folders were created upon starting new analyses
Fixed an issue preventing analyses from using inputs with BaseSpace v1pre3 APIs
Fixed an issue causing analyses with a specified output path to incorrectly return an error stating that the data does not exist
The following endpoint "/api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs" now supports using external data as input
Any value other than "economy" or "standard" for submitted analysis jobs will default to "standard" and use "standard"
The parameter to pass an activationcode is now optional for start-analysis API endpoints
Base:
Improved the display of errors in the activity jobs screen if a Meta Data schedule fails
If an error occurs when processing metadata a failed job entry will be added in the Base Activity screen
Fixed an issue where records ingested via schedules from the same file could be duplicated
Fixed an issue where exporting the view shared via bundle would show an error 'Could not find data with ID (fol. ....)'
Resolved a NullPointerException error when clicking on Format and Status filters in the details screen of a Schedule in the Results tab
Fixed an issue where a schedule download would fail when performed by different user than the initial user
Bench:
Fixed an issue when trying to query a Base table with a high limit within a workspace
Fixed an apt-get error when building images due to an outdated repository
Fixed an issue where a stopped workspace would display "Workspace paused" instead of "Workspace stopped"
Fixed an issue where large files (e.g., 150GB+) could not be downloaded to a fuse-driver location from a Workspace, and set the new limit to 500GB
Cohorts:
Fixed an issue where split Nirvana JSON files are not recognized during ingestion
Fixed an issue causing the UI hangs on large genes and returns a 502 error
Fixed an issue where OMOP files are not correctly converted to CAM data model, preventing OMOP data ingestions
Fixed an issue where OMOP large drug ingestions led to memory issues and preventing further drug data ingestion
Fixed an issue where users from a different tenant accessing a shared project could not ingest data
Documentation
General updates and improvements
Clarified that bundle assets can only be linked once to the same data set
Made it more explicit that the custom bucket name needs to be entered in the IAM configuration
Added information on how to define scratch space in CWL workflows.
Added explanation of how to query arrays in table data
Described analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes
Removed outdated information about creating new tabs from workspace description.
Added information about creating an OAuth access token
Added information about stopping Workspaces and removed information about pausing Workspaces, as pausing is no longer an option.
Added preliminary beginner's tutorial for ICA API usage
Added information about how to use data stored in Basespace as an input when starting an analysis via the API
Clarified that upload and download rights act independent of user roles
Updated the table which shows the regions ICA is hosted on to include all the available regions in alphabetical order
Updated instructions for creating an AWS IAM user policy to remove a permission which is not correct
Added language specifying that folders with the name '.' are not supported
Added information about how to query views from Data Catalogues using the Snowflake FLATTEN function
Updated the CLI release history
Updated the Nextflow DRAGEN tutorial to account for no default scratch space
2023 December 05 - ICA v2.21.0
Features and Enhancements
Flow:
Analysis logs (task stdout/stderr files) are now written to a folder named ‘ica_logs’ within the analysis output folder
Default scratch disk size attached to analysis steps reduced from 2TB to 0B to improve cost and performance of analyses. Pipelines created before ICA v2.21.0 will not be impacted
Notifications: Notifications can now be updated and deleted in externally managed Projects
API: Clarified on the Swagger page which sorting options apply to which paging strategy (cursor-based versus offset-based). Changed the default sorting behavior so that:
When no paging strategy is specified and no sort is requested, then cursor-based paging is default
When no paging strategy is specified and sort is requested, then offset-based paging is default
Cohorts
Procedure Search Box: Users can now access additional UI functionalities for Procedures
Users can now access Procedure codes from OMOP
Improved handling of drug codes across all reports, excluding Survival comparison
Ingestion
Users now have enhanced job warning log and API status improvements
Users now require download permissions to facilitate the data ingestion process
Fetch Molecular Files: Improved import – Users can now input a directory path and select sample files individually
Variant Type Summary: Users can now access a new variants tab that summarizes Variant type statistics per gene
Added sorting and filtering capabilities to report tables, such as variants observed in genes
Users can now view sample barcodes, replacing internal auto-increment sample IDs in the Structural Variants table within the Genes tab
“Search subjects” functionality improved with flexible filtering logic that now supports partial matches against a single input string
Fixed Issues
Data Management:
Fixed an issue with data copy via the CLI where the file was being copied to a subfolder of the intended location instead of the specified folder
Resolved an issue where browser upload hangs intermittently when creating data
Fixed an issue where the delete popup does not always disappear when deleting data
Fixed an issue where GetFolder API call returns 404 error if the Create and Get operations are performed 100ms apart
Fixed an issue where file copy would fail if the file was located at the root level of User’s S3 storage bucket
Fixed an issue causing data linked from externally managed projects to be incorrectly excluded from the list project data API response
Fixed an issue where User cannot use data URNs to identify the destination folder when interacting with copy data API endpoints
Bundles: Fixed an issue where clicking the back button before saving a new bundle leads to inconsistencies
Flow:
Fixed an issue where pipeline documentation is not scrollable when launching pipeline
Fixed an issue with logfiles of a task not being available for streaming while the task is still running
Fixed an issue where using the 're-run' button from the analysis page reverts the storage size selection to default
Fixed an inconsistency where the following two endpoints would show different analysis statuses:
GET /api/projects/{projectId}/analyses
GET /api/projects/{projectId}/analyses/{analysisId}
Improved performance issues with UI loading data records when selecting inputs for analysis
Fixed a caching issue which resulted in delays when running pipelines
Fixed an issue where back button for analysis or pipeline details does not always direct Users back to analysis or pipelines view, respectively
Fixed an issue where system performance is degraded when large batches (e.g., 1,000) of data are added as input to Analyses via the graphical UI. It is recommended to start Analyses with large numbers of input files via API
Base
Fixed an issue where enabling Base from a Base view other than Base Tables returned a warning message
Fixed an issue where Base access was not enabled when a bundle with tables is added to a project without Base (Base is automatically enabled so users can see the bundle's tables). However, access to the bundle's tables is revoked upon the deletion of Base, and was not granted again once Base was re-enabled
Fixed an issue where a Base job to load data into a table never finished because the file was deleted after the job started and before it finished. Now the job will end up in a Failed state
Cohorts
Fixed an issue where needle plot filtered out data points reappear when zooming in the exon when a filter is in place
Fixed an issue where users from a different tenant who accept a project share may encounter a failure at the final step of the data ingestion process
Fixed an issue where users can encounter intermittent errors when browsing and typing for a gene
Fixed an issue where the UI hangs on large genes and returns a 502 error
Documentation
General updates and improvements
Updated organization of Table of Contents
Data Management: Added explanation of the impact of billing mode when users bring their own S3 storage
Flow: Added guidance about the ica_logs subfolder in the analysis output folder
Flow: Updated default scratch space reference documentation
API: Documented how to archive or delete files programmatically using API endpoints
API: Added information about how the sort parameter is used to choose the default pagination
Cohorts: Updated documentation to reflect OMOP ingestion and UI search for Procedures. The documentation changes also reflect instruction on the simplified ingestion and comparison sharing
2023 November 9 - ICA v2.20.1
Fixed Issues
Data Management:
Fixed an issue where multiple folder copy jobs with the same destination may get stuck In Progress
Fixed an intermittent issue where tags on the target folder for a batch data update call are not set, but are set for all child data
Flow:
Fixed an issue causing intermittent pipeline failures due to an infrastructure error
2023 October 31 - ICA v2.20.0
Features and Enhancements
Navigation: If multiple regions are enabled for the same tenant, the region will be indicated in the waffle menu
Logging: Data transfers of BaseSpace Sequence Hub projects with data stored in ICA will be traced in ICA logs
Cohorts
Disease Search Box: Added support for specifying subjects by age of onset of disease(s)
Drug Search Box: Added a new query builder box for Drugs
Ingestion: Support for Drug, drug route, etc. attached to subjects
Cohorts building: Users can build cohorts by specifying drugs, drug route, etc.
Ingestion:
Combine different variant types during ingestion (small variants, cnv, sv)
Cohorts supports Illumina Pisces variant caller for hg19 VCFs
Fixed Issues
General:
Fixed an issue where the graphical UI hands with ha spinning wheel when saving or executing a command
Fixed an issue where rich text editor for Documentation tab on Pipelines, Tools, Projects and Bundles does not populate with correct styles in edit mode
Data Management:
Fixed an issue where multiple clicks on create data in Project API endpoint resulted in multiple requests
Fixed an issue where the secondary data selection screen could not be resized
A spinning wheel icon with ‘copying’ status is displayed at the folder level in the target Project when a folder is being copied. This applies to the actual folder itself and not for folders higher up in the hierarchy
Fixed an issue where API to retrieve a project data update batch is failing with 500 error when either the Technical or the User tags are updated during the batch update request
Fixed an issue where linking jobs fail to complete if other linking jobs are running
Improved performance for data transfer to support BaseSpace Sequence Hub Run transfers
Fixed an issue causing some folder copy jobs to remain in "Partially Succeeded" status despite being completed successfullyBundles: Fixed an issue where the URL and Region where a Docker image is available is not displayed for a Docker image Tool shared via an entitled Bundle
Fixed an issue where the folder copy job was getting stuck copying large amounts of big files
Fixed an issue where the folder counts were not matching expected counts after Data linking
Fixed an issue where delete data popup would occasionally not disappear after deleting data.
Fixed an issue with data copy where referencing data from another region would not result in immediate failure
Fixed issue where uploading a folder using the CLI was not working
Fixed an issue where a Docker image shared via an entitled Bundle can be added to another region
Workflows: Fixed an issue where workflow does not fail if BCL Convert fails for a BCL Convert-only run
Flow:
Improved performance when batches of data up to 1000 are added as input to an Analysis
Nextflow engine will return exit code 55 if the pipeline runner task is preempted
Fixed an issue where log files cannot be opened for any steps in an analysis while the analysis is in progress
Fixed an issue with concurrent updates on analysis
Fixed an issue where unknown data inputs in the XML of an analysis are not being ignored
The warning, close, and machine profile icons for Tools can now be seen in the graphical CWL pipeline editor
Fixed an issue where user cannot expand analysis output folder if user permissions change after starting analysis. Now, if a user has the correct permissions to start an analysis, that analysis should be able to finish correctly no matter the permissions at the time it succeeds
Base:
Fixed an issue switching back from template to Empty Table did not clear the fields
Data linked from an externally managed project can be added to Base Tables
Fixed an issue in the graphical UI where schema definition does not scroll correctly when many columns are defined
Documentation
General updates and improvements
Sequencer Integration: Added information that auto-launch from instrument is not currently compatible with MiSeq systems
Data Management
Documented an alternate method of adding files to samples and explained how to remove files from samples
Provided more guidance on data transfer options
Updated reference documentation for Data Copy functionality
Added a message to inform users that folder copy can be impacted in cases where a folder with the same name already exists in the target project
Flow
Described analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes
Updated ‘CWL: Scatter-Gather’ tutorial
Added documentation for analysis timeout
Base:
Added a new tutorial for creating a Python dashboard to access Base metrics via API
Added information about Snowflake user/role setup
Included sample TSV file in Base Basics tutorial
Bench: Updated Fuse Driver troubleshooting
Notifications: Updated Filtering section of Notifications page
CLI: Updated CLI release history
Reference
Updated Illumina Connected Software registration guide
Added reference documentation for data integrity recommended practices
2023 October 3 - ICA v2.19.0
Features and Enhancements
Data Management/API: Added a new endpoint available to change project owner
POST /api/projects/{projectId}:changeOwner { “newOwnerId”:”}
Data Management/API: Added a new endpoint to copy data from one project to another:
/api/projects/{projectId}/projectDataCopyBatch
Data Management/CLI: Added the ability to copy files and folders between projects in the UI and CLI. This includes support for copying data from projects with ICA-managed storage (default) to projects with S3-configured storage.
Flow/API: When starting an analysis via the API, you can specify the input files based on HTTP(s). When your analysis is done, you will see the URL corresponding to the inputs in the UI, but you will not be able to start an analysis from the UI using this URL
Flow: Removed the ability to start analyses from data and sample views in the UI where a single input is selected to start analyses in bulk
Flow/CLI: Duplicate analyses submitted via the CLI will be avoided
Flow/API: Added two new endpoints for workflow sessions:
Get /api/projects/{projectId}/workflowSessions
Get /api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs
Flow/API: Added a new endpoint to retrieve configurations from a workflow session
Flow/Autolaunch ICA Workflow Session and Orchestrated Analyses (launched by the workflow session) now saves outputs in an organized folder structure: /ilmn-analysis/<name_used_to_create_sequencer_run_output_folder>
Base: The Base module has a new feature called ‘Data Catalogue’. This allows you to add usage data from your tenant/project if that data is available for you.
Data Catalogue views will be available and can be used in Base to query on.
You will be able to preview and query Data Catalogue views through Base Tables and Query screens
The Data Catalogue will always be up to date with the available views for your tenant/project
Data Catalogue views cannot be shared through a Bundle
Data Catalogue views will also be available to team members that were added after the view was added
Data Catalogue views can be removed from the Base tables and corresponding project
By removing Base from a project, the Data Catalogue will also be removed from that project
Cohorts: Disease Search box
Cohorts now includes a disease search box to search for disease concepts. This replaces the disease concept tree explorer
Disease search box located under a Disease tab in main Query builder
Search box allows for a copy/paste action of codes to be processed as separate query elements. Currently, the feature is limited to a complete valid list
Each disease entered into the search box is displayed as a separate query item and can be set to include or exclude.
Diseases in search box can be used with boolean logic in cohort creation
Search box allows for an auto-complete of diagnosis concepts and identifiers
The disease filter is included in the cohort query summary on cohort page
Fixed Issues
Data Management: Data copy between ICA-managed projects and S3 storage configured projects is supported
Data Management: Fixed an issue where storage configurations matching ICA-managed buckets would cause volume records to get associated with the wrong storage configuration in the system
Data Management/API: The endpoint GET/api/projects/{ProjectID}/samples/{SampleID} correctly returns all the own samples and linked samples
Data Management/API: Improved handling of bulk update via API when concurrent deletion of file has occurred
Data Management/CLI: Fixed an issue where projectdata update tags would not update the tags
Flow: Fixed an issue resulting in failure to send a notification resulting in a failed workflow
Flow: Fixed an issue where one workflow session may override another when both are executed at the same time
Base: Fixed an issue where query download in JSON format returns an error
Base: Added a message in the UI when a query takes longer than 30 seconds to inform the user that the query is ongoing and can be monitored in the Activity view
Base: Added a section describing the Data Catalogue functionality
Bench: Fixed an issue where resizing the workspace to current size would prevent users from resizing for the next 6 hours
Cohorts: Fixed an issue where Gene Expression table does not display with TCGA data or for tenants with a hyphen (e.g., ‘genome-group’)
Cohorts: Fixed an issue where user had no way to delete a cohort comparison from a deleted cohort
Cohorts: Fixed an issue in the UI where multi-cohort needle plot tracks are overlapping
Cohorts: Fixed an issue causing failures during annotation step with ‘CNV’ data type when selection ‘GB=hg19’ and ‘CNV data’ for liftover; also observed with ‘SM data’ and ‘hg38’ without liftover (in APS1 and CAC1 regions) due to a ‘404 Not Found’ error.
CLI: Fixed an issue to support adding the server-url as a parameter instead of having the config set
Documentation
General updates and improvement
General: Added link to Illumina Knowledge resource
Data Management: Add documentation around storage configuration synchronization behavior
Data Management: Documented the new Data Copy functionality
Data Management: Provided more guidance on data transfer options
Data Management: Added a section describing Data Catalogue functionality
Data Management: Added guidance about not using file names beginning with '/'
Service Connector: Added troubleshooting guidance for service connector
Sequencer Integration: Updated analysis autolaunch reference docs
Sequencer Integration: Updated the NovaSeq 6000 analysis auto-launch tutorial
Flow: Improved the tutorial for Scatter-gather method in Nextflow
Flow: Improved tutorial for Nextflow: RNASeq Lift
Flow: Added a tutorial for converting open-source nf-core pipelines to ICA Nextflow pipelines
Flow: Updated reference docs for starting analyses
Flow: Improved tutorial for scatter-gather in CWL
Base: Improved descriptions and scenarios of Base tables and queries
Cohorts: Documented Cohorts Diagnosis Concept attribute search box functionality
CLI: Updated the CLI release history page with all required build files
API: Added tip for users experiencing API rate limiting
API: Added troubleshooting guide for GetTemporaryUploadCredentialsAsync failure error scenario
SDK: Included new reference to Illumina's ICA Python SDK resource available on GitHub.com
2023 September 14 - ICA v2.18.4
Fixed Issue
Fixed an issue uploading folders via the CLI
2023 September 8 - ICA v2.18.3
Fixed Issue
Fixed an issue causing CWL pipelines using Docker images that do not contain bash shell executable to fail.
2023 September 7 - ICA v2.18.2
Fixed Issue
Fixed an issue leading to intermittent system instability.
2023 September 6 - ICA v2.18.1
Fixed Issue
Cohorts: Issue fixed where GTEx plot is not available for tenants with a hyphen (e.g. ilmn-demo).
2023 August 31 - ICA v2.18.0
Features and Enhancements
Data Management: New API endpoints are available for:
Creation of a data update in bulk: POST/api/projects/{projectId}/dataUpdateBatch
A list of data updates for a certain project: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}
A list of items from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items
A specific item from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items/{itemId} Note: Batch updates include tags, format, date to be archived and date to be deleted
Data Management/API: The sequencing run information can be retrieved through its Id by using the API endpoint GET/api/sequencingRuns/{sequencingRunId}
Flow: Auto launch now supports BCL Convert v3.10.9 pipeline and both TruSight Oncology 500 v2 pipelines (from FASTQs)
Flow/API: getPipelineParameters API returns parameter type in response
Flow/API: Added endpoints to retrieve and update a project pipeline definition
Flow/API/CLI: Include "mountPaths" field in response for API and CLI command to retrieve analysis inputs
Flow: Removed "fpga-small" from available compute types. Pipelines using "fpga-small" will use the "fpga-medium"-equivalent compute specifications instead
Flow/API: New API endpoint available to request the analyses in which a sample is being used
Flow/API: When leaving activationCodeDetailId empty when starting an analysis, the best match activation code will be used
Flow: Analyses launched/tracked by BaseSpace Sequence Hub contain relevant BaseSpace information in analysis details view
Bench: A new Bench permission is being introduced: Administrator. This permission allows users to manages existing workspaces and create new workspaces
Bench: The Bench Administrator role allows you to create new Bench workspaces with any permissions even if you as a Bench administrator do not have these permissions. In that case, you can create and modify the workspace, but you cannot enter that workspace. Modifying is only possible when the workspace is stopped
Bench: As a Bench Contributor you are not allowed anymore to delete a Bench Workspace, you need the Bench Administrator role.
Cohorts: Users can now ingest raw DRAGEN bulk RNAseq results for genes and transcripts (TPM), with the option to precompute differential expression during ingestion
Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
Cohorts: In multiple needle plot view, individual needle plots can now be collapsed and expanded
Cohorts: Pop-outs for needle plot variants now contain additional links to external resources, such as UCSC
Cohorts: For a given cohort, display a distribution of raw expression values (TPM per gene) for a selected attributes
Cohorts: Use of the Cohorts maintains session between core ICA and Cohorts iFrame to prevent unwanted timeouts
Cohorts: Cohorts displays structural variants that include or overlap with a gene of interest
API: Two new API endpoints added to accept Terms and Conditions on a bundle:
GET /api/bundles/{bundleId}/termsOfUse/userAcceptance/currentUser Returns you the time of acceptance when you, the current user, accepted the Terms & Conditions.
POST /api/bundles/{bundleId}/termOfUse:accept
API: Add temporary credentials duration to API documentation
Notifications: List of events to which you can subscribe contains new ICA notification containing analyses updates
Versioning: The ICA version can now be found under your user when you select "About"
Versioning/API: It is possible to retrieve system information about ICA, such as the current version through GET/api/systeminfo
Logging: When an action is initiated by another application, such as BaseSpace Sequence Hub, it will be traced as well in the ICA logs
Fixed Issues
Data Management/API: Improved error handling for API endpoint: DELETE/api/bundles/{bundleId}samples{sampleId}
Data Management: CreatorId is now present on all data, including subfolders
Data Management: Improved external linking to data inside ICA using deep linking
Data Management: Improved error handling when creating folders with invalid characters.
Data Management: Fixed an inconsistency for URN formats on output files from Analyses. This fix will apply only for analyses that are completed starting from ICAv2.18.0
Data Management: Improved resilience in situations of concurrent linking and unlinking of files and folders from projects
Data Management/API: Fixed an issue where the API endpoint GET /api/samples erroneously returned a 500
Data Management/API: API endpoint GET/api/projects/{projectId}/analyses now returns the correct list when filtering on UserTags whereas it previously returned too many
Data Management: It is only possible to delete a storage configuration if all projects that are using this storage configuration have been hidden and are not active projects anymore
Data Management: Improved accuracy of the displayed project data size. Prior cost calculations were accurate, but the project data size visualization included technical background data
Data Management/API: Improved retry mechanism for API endpoint to create folderuploadsession
Data Management/CLI: When an upload of a folder/file is done through the CLI, it returns the information and ID of the folder/file
Data Management: Fixed an issue where there is a discrepancy in number of configurations between Storage->Configurations and Configurations-> Genomics.Byob.Storage Configuration view
Collaboration: Fixed an issue where a user is presented with a blank screen when responding to a project invitation
Flow/API: Improved error handling when invalid project-id is used in API endpoint GET /api/projects/{projectId}/pipelines
Flow: When a spot instance is configured, but revoked by AWS, the pipeline will fail and exit code 55 is returned
Flow/API: Fixed an issue when an Analysis completed with error "incomplete folder session", the outputs of the Analysis are not always completely listed in the data listing APIs
Flow/API: Updated ICA Swagger Project > createProject to correctly state that the analysis priority must be in uppercase
Flow: Fix to return meaningful error message when instrument run ID is missing from Run Completion event during an auto launched analysis
Flow: Improved parallel processing of the same analysis multiple times
Base: Improved error handling when creating queries which use two or more fields with the same name. The error message now reads "Query contains duplicate column names. Please use column alias in the query"
Base: Fixed an issue where queries on tables with many entries fail with NullPointerException
Bench: Clarified that changes to Bench workspace size only take effect after a restart
Cohorts: Fixed issue where counts of subjects are hidden behind attribute names
Cohorts: Fixed issue where the state of checked files are not retained when selecting molecular files that are in multiple nested folders
Cohorts: Fixed issue where projects that contain files from linked bundles cause a time out, resulting in users not being able to select files for ingestion
Cohorts: Fixed an issue where the 'Import Jobs' page loaded within the Data Sets frame, depending on where the import was initiated
Cohorts: Fixed an issue in the Correlation plat where x-axis counts were hidden under attribute names
Fixed an issue where users were previously incorrectly signed out of their active sessions
Documentation
General updates and improvements
General: Added ICA Reference Terminology page
Data Management: Document how to change Project owner
Data Management: Updated Storage Size table for Compute on Pricing page to include XL, 2XL, 3XL
Data Management: Improved explanation of hiding Projects when connecting to your own S3 bucket for data storage
Data Management: Added default and shared explanation to storage documentation
Data Management: Updated documentation to reflect the temp credentials returned for AWS datastore is for 36 hours
Flow: Documented event for Analysis status transition
Flow: Added Nextflow best practices to reduce Analysis failures
Flow: Updated Pipeline activation code behavior explanation
Flow: Added documentation for specifying external docker images (i.e. Docker Hub)
Base/CLI: Added a tutorial for using Base via the CLI
Base: Updated documentation to mention that Schema does not contain type ‘variant’
Base: Documented Base query which joins to tables can result in duplicate columns
Base: Documented SnowSQL show view vs show grants
Bench: Documented Bench permission changes
Cohorts: Documentation updates:
RNASeq ingestion
Applicable user-supplied RNASeq visualizations – Raw Expression Values in GTEx-like table
NeedlePlot
Structure Variant list
CLI: Added documentation for CLI config settings
Logging: Documented event logging transfer of ownership
2023 August 3 - ICA v2.17.1
Fixed Issues
Fixed an issue causing analyses requesting FPGA compute resources to experience long wait times (>24h) or not be scheduled
2023 June 27 - ICA v2.17.0
Features and Enhancements
Data Management: Performance improvements for data link and unlink operations – Larger and more complex folders can now be linked in the graphical UI, and progress can be monitored with a new visual indication under Activity > Batch Jobs
Flow: Increased the allowed Docker image size from 10GB to >20GB
Flow/CWL: Added support for javascript expressions “ResourceRequirements” fields (i.e., type, size, tier, etc.) in CWL Pipeline definitions
Flow/API: Added support for using Pipeline APIs to query Pipelines included in Entitled Bundles (i.e., to retrieve input parameters)
Flow/API: Added support for providing S3 URLs as Pipeline data inputs when launching via the API (using storage credentials)
Flow/API: Added support for specifying multi-value input parameters in a Pipeline launch command
Bench: Project and Tenant Administrators are now allowed to stop running Workspaces
Notifications: Notifications are now available for batch job changes
Cohorts: Enhanced ingestion workflow to ingest RNAseq raw data from DRAGEN output into backend Snowflake database
Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
Cohorts: Multi-Cohort Marker Frequency - Added Multi-Cohort Marker Frequency tab allowing users to compare expression data across up to four Cohorts at the gene level
Cohorts: Multi-Cohort Marker Frequency includes a pairwise p-value heat map
Cohorts: Multi-Cohort Marker Frequency - Includes frequencies for Somatic and Copy Number Variants
Cohorts: Tab added for a multi-cohort marker frequency analysis in cohort comparisons
Cohorts: Multi-Cohort Needle Plot - Added new tab in the Comparison view with vertically aligned needle plots per cohort for a specified gene, allowing collapsible and expandable individual needle plots
Cohorts: Additional filter logic added to multi-cohort needle plot
Cohorts: Improved DRAGEN data type determination during ingestion allowing for multiple variant type ingestion
Cohorts: Enhanced list of observed variants with grouped phenotypes and individual counts, including a column for total sample count; tooltips/pop-outs provide extended information
Cohorts: Updates to needle plot link outs
Cohorts: Improved the Comparison feature by optimizing API calls to handle subjects with multiple attributes, ensuring successful loading of the page and enabling API invocation only when the user selects or expands a section
Cohorts: Removed unused columns (genotype, mrna_feature_id, allele1, allele2, ref_allele, start_pos, stop_pos, snp_id) from annotated_somatic_mutations table in backend database
Cohorts: Refactored shared functionality for picking consequence type to reduce code duplication in PheWAS-plot and GWAS-plot components
Cohorts: Invalid comparisons on the Comparisons page are now grayed out and disabled This improvement prevents the selection of invalid options
Cohorts: Automatic retry of import jobs when there are failures accessing data from ICA API
Fixed Issues
Data Management: The content of hidden Projects can now be displayed
Data Management: Fixed the TimeModified timestamp on files
Bundles: Resolved issues when linking a large number of files within a folder to a Bundle
Flow: Single values are now passed as a list when starting an Analysis
Flow: Pipelines will succeed if the input and output formats specified on the pipeline level match at the Tool level
Flow: Fixed an issue causing Analysis failures due to intermittent AWS S3 network errors when downloading input data
Flow/CWL: Improved performance on output processing after a CWL Pipeline Analysis completes
Flow/UI: Mount path details for Analysis input files are now visible
Flow/UI: Improved usability when starting an Analysis by filtering entitlement options based o inputs selected and available entitlements
Flow/API: List of Analyses can now be retrieved via the API based on filters for UserReference and UserTags
Base: Fixed an issue where the Scheduler continues to retry uploading files which cannot be loaded
Bench: Resolved an issue when attempting to access Workspaces with multi-factor authentication (MFA) enabled at the Tenant-level
API: Improved error messaging for POST /api/projects/{projectId}/data/{dataId}:scheduleDownload
Navigation: Removed breadcrumb indication in the graphical UI
Cohorts: Fixed issue where Correlation bubble plot not showing for any projects intermittently
Cohorts: Fixed issue where importing Germline/hg19 test file did not load variants for a specific gene in the Needle plot due to missing entries in the Snowflake table
Cohorts: Fixed a bug causing an HTTP 400 error while loading the Cohort for the second time due to the UI passing "undefined" as variantGroup, which failed to convert to the VariantGroup Enum type
Cohorts: Fixed issue where scale (y-axis) of needle plot is changed even if value of sample count gnomAD frequency is not accepted
Cohorts: Fixed an issue where no data was generated in the Base Tables after a successful import job in Canada - Central Region (CAC1)
Cohorts: Fixed issue where long chart axis labels overlap with tick marks on graph
Documentation
General improvements and fixes
User Management: Added details about how users can manage an overview of their Projects
Data Management: Clarified the visibility of Batch jobs for different users in a Tenant
Bundles: Provided more details about Bundle linking and unlinking with data in folders
Flow: Described Analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes
Flow: Added an explanation for how external S3 files can be used as inputs to Pipelines
Bench: Added an explanation that Project/Tenant Administrators can stop running Workspaces
Bench: Added guidance for downloading data when using JupyterLab Notebooks
Cohorts: Added instructions for RNA-Seq data ingestion into Cohorts
Cohorts: Added reference documentation for multi-cohort marker frequency and needle plot comparisons
CLI: Updated the CLI command index
CLI: Updated the CLI release history page with all required build files
CLI: Added link to legacy ICA v1 CLI release history page
SDK: Included a new reference to Illumina’s ICA Python SDK resource available on Github
Network Settings: Updated the list of whitelisted URLs on https://help.ica.illumina.com/reference/r-networksettings
2023 May 31 - ICA v2.16.0
Features and Enhancements
Data Management: You can now select 'CYTOBAND' as format after file upload
Data Management: Added support for selecting the root folder (of the S3 bucket) for Projects with user-managed
Data Management: Added support for creating an AWS Storage Configuration with an S3 bucket with Versioning enabled
Auto-launch: Added technical tags for upstream BaseSpace Run information to auto-launched analyses
Auto-launch: Added support for multiple versions of BCL Convert for auto-launched analyses
Flow: Added support for '/' as separator in CWL ResourceRequirements when specifying Compute Type
Flow/API: The API to retrieve analysis steps now includes exit code for completed steps
Bench: Restricted Workspaces (Open or Restricted) always allow for access to Project Data within the Workspace
Bench: Restricted Bench workspaces have limited access through whitelisted URLs that are checked before entry
Bench: Restricted Bench Workspaces allow for Open or Restricted workspaces. Restricted workspaces do not have access to the internet except for user-entered whitelist URLs
Fixed Issues
Data Management: Upload for files names including spaces is now consistent for connector and browser upload. We do still advise not to use spaces in file names in general
Data Management: Fixed search functionality in Activity > Data Transfers screen
Data Management: Improved performance on opening samples
Data Management: Fixed an issue where reference data in download tab initiates an unexpected download
Data Management: Fixed intermittent issue where the Storage configuration within a Project can go into Error status and can block users from creating records such as folders and files
Data Management/API: Improved error handling for API endpoints: Delete/api/projects/ {projectId}/bundles/{bundleId} and POST/api/projects/{projectId}/bundles/{bundleId}
Data Management/API: Improved error handling for POST/api/projects/{projectId}/base:ConnectionDetails
Bundles: Fixed an issue where the Table view in Bundles is not available when linking to a new Bundle version
Bundles/API: Improved error handling for DELETE/api/bundles/{bundleId}/tools/{toolId} and POST/api/bundles/{bundleId}/tools/{toolId}
Bundles/API: Improved error message for POST/api/bundles/{bundleId}/samples/{sampleId}
Bundles: Fixed an issue where linking/unlinking a Bundle with Base Tables could result in errors
Service Connector: Improved error message for DELETE/api/connectors/{connectorId}/downloadRules/{downloadRuleId}
Notifications/API: Custom subscriptions with empty filter expressions will not fail when retrieving them via the API
Notifications/API: Improved error handling for POST/api/projects/{projectId}/notificationSubscriptions
Notifications/API: Improved notification for Pipeline success events
Flow: When the input for a pipeline is too large, ICA will fail the Analysis and will not retry
Flow: Fixed issue where analysis list does not search-filter by ID correctly
Flow: Improved error handling when issues occur with provisioning resources
Flow: When retry succeeds in a Nextflow pipeline, exit code is now '0' instead of '143'
Flow/API: Fixed an issue causing API error when attempting to launch an Analysis with 50,000 input files
Flow/API: Improved pipeline error code for GET/api/projects/{projectId}/pipelines/{pipelineId} when already unlinked pipeline Id is used for API call
Flow/API: Fixed an issue where Analyses could not be retrieved via API when the Pipeline contained reference data and originated from a different tenant
Flow/API: Fixed filtering analyses on analysisId. Filtering happens via exact match, so part of the Id won't work
Bench/CLI: Fixed issue where the latest CLI version was not available in Bench workspace images
Navigation: Updated URLs for Correlation Engine and Emedgene in the waffle menu
Authentication: Using POST /api/tokens:refresh for refreshing the JWT is not possible if it has been created using an API-key.
Authentication: Improved error handling when there is an issue reaching the authentication server
Authentication: Improved usability of "Create OAuth access token" screen
Cohorts: Issue fixed where CNV data converted from hg19 to hg38 do not show up in Base table views.
Cohorts: Issue fixed accounting for multiple methods of referring to the alternate allele in a deletion from Nirvana data.
Cohorts: Fixed intermittent issue where GWAS ingestions not working after Base enabled in a project.
Documentation
User Management: Improved explanation of Team functionality
Data Management: Added guidance on versioning Bring Your Own storage buckets
Data Management: Improved explanation on linking to data from other projects
Data Management: Described functionality of Secondary Data
Data Management: Updated guidance for file upload size
Data Management: Updated configuration guidance for Bring Your Own Storage bucket using S3
Data Management: Updated list of supported Data formats
Data Management/CLI: Added a tutorial for mounting project data using the CLI
Flow: Added a tutorial for building and running a CWL graphical pipeline
Flow: Included best practices for minimizing errors in Nextflow pipelines by adding retry logic
Base: Updated the Base schedule descriptions to include information about Metadata and Administrative schedules, as well about frequency of scheduled ingestions
Bench: Updated restrictions of available features when in the FUSE mount in a Workspace
Bench: Updated descriptions about Bench restrictions, including details about whitelisted URLs
CLI: Added CLI authentication best practices for setting up, validating, and resetting the configuration of a user's profile
API: Added information to launch analysis API spec on output mappings overwrite behavior when files already exist on the target project during analysis output
Network Settings: Updated Network Settings regional S3 URLs list
General updates and improvements
2023 May 2 - ICA v2.15.1
Fixed Issue
Fixed an issue causing incorrect empty storage configuration dropdown during Project creation when using the “I want to manage my own storage” option for users with access to a single region
2023 April 25 - ICA v2.15.0
Features and Enhancements
General availability of sequencer integration for Illumina sequencing systems and analysis auto launch
General usability improvements in the graphical interface, including improved navigation structure and ability to switch between applications via the waffle menu in the header
Storage Bundle field will be auto-filled based on the Project location that is being chosen if multiple regions are available
Event Log entries will be paged in the UI and will contain a maximum of 1,000 entries. Exports are limited to the maximum number of entries displayed on the page.
Read-only temporary credentials will be returned when you are not allowed to modify the contents of a file
Creating Project notifications for BaseSpace externally managed projects is now supported
The ICA UI will only allow selection of storage bundles belonging to ICA during Project creation, and the API will only return storage bundles for ICA
Flow: Allow attached storage for Pipeline steps to be set to 0 to disable provisioning attached storage and improve performance
Cohorts: GRCh37/hg19-aligned molecular data will get converted to GRCh38/hg38 coordinates to facilitate cross-project analyses and incorporating publicly available data sets.
API: Project list API now contains a parameter to filter on (a) specific workgroup(s)
API: Two new API endpoints are added to retrieve regular parameters from a pipeline within or without a Project context
Fixed Issues
Optimized price calculations resulting in less overhead and logging
Improved error handling:
during Project creation
of own storage Project creation failures.
to indicate connection issue with credentials
for graphical CWL draft Pipelines being updated during an Analysis.
Fixed an issue causing data records to remain incorrectly in Unarchiving status when an unarchive operation is requested in the US and Germany regions
Improved error messaging in cases where the AWS path contains (a) special character(s)
Fixed an issue causing errors when navigating via deep link to the Analysis Details view
CLI: Unarchival of folders is supported when archive or unarchive actions are not in progress for the folder
CLI: Improved error message to indicate connection issue with credentials
Flow: Fixed an issue causing incorrect naming of Analysis tasks generated from CWL ExpressionTools
Flow: Fixed an issue when cloning Pipelines linked from Entitled Bundles to preserve the original Tenant as the Owning Tenant of the cloned Pipeline instead of the cloning user’s Tenant
Flow: Fixed an issue causing outputs from CWL Pipelines to not show in the Analysis Details despite being uploaded to the Project Data Analysis output folder when an output folder is empty
Flow: When a Contributor starts an Analysis, but is removed afterwards, the Analysis still runs as expected
Flow: Fixed an issue where Analyses fail where Nextflow is run a second time
Flow: Fixed an issue causing API error when attempting to launch an Analysis with up to 50,000 input files
Flow: Fixed an issue causing degraded performance in APIs to retrieve Analysis steps in Pipelines with many steps
Flow: Fixed an issue causing Analysis failure during output upload with error “use of closed network connection”
Flow: Fixed an issue causing disk capacity alter log to not show when an Analysis fails due to disk capacity and added error message
Flow: Fixed an issue preventing cross-tenant users from being able to open a shared CWL pipeline
Base: Improved target Table selection for schedulers to be limited to your own Tables
Bench: Fixed an issue causing Workspaces to hang in the Starting or Stopping statuses
Cohorts: Now handles large VCFs/gVCFs correctly by splitting them into smaller files for subsequent annotation by Nirvana
API: Fixed returning list of unlinked data in a sample that was linked before in GET/api/projects/{projectId}/data
API: Fixed error for getSampleCreationBatch when using status filter
Documentation
Updated guidance for selecting Project Location and availability of Storage Bundles when configuring a new Project
Included information about receiving and using Entitled Bundles
Added guidance that a sample must be complete to be linked to a Bundle
Updated details about paging in the Event Log
Refined and expanded guidance for sequencer integration and analysis auto-launch, including adding a new tutorial for NovaSeq 6000 and expanding the sequencer reference table
Added tutorial for using scatter/gather for Nextflow pipelines
Refined Pipelines documentation around clone function
Added information that Base Tables must not contain reserved Snowflake terms in their definition
Added description for Copy Number Variants and Structural Variants tables now viewable in ICA Base for Cohorts
Added description for hg19 support as an alternative during ingestion for Cohorts
Updated CLI command index
Updated CLI binary and checksum files
Updated information about data archive pricing
Updated software registration guide
General updates and improvements
2023 March 28 - ICA v2.14.0
Features and Enhancements
Added a limit to Event Log and Audit UI screens to show 10,000 records
API: Parent output folder can be specified in URN format when launching a Workflow session via the API
Flow: Reduced Analysis delays when system is experiencing heavy load
Flow: Improved formatting of Pipeline error text shown in Analysis Details view
Flow: Users can now start Analyses from the Analysis Overview screen
Flow: Superfluous “Namespace check-0” step was removed to reduce Analysis failures
Flow: Number of input files for an Analysis is limited to 50,000
Flow: Auto launched Workflow sessions will fail if duplicate sample IDs are detected under Analysis Settings in the Sample Sheet
Base: Activity screen now contains the size of the query
Cohorts
Detect and Lift Genome Build: Cohorts documentation provides set-up instructions to convert hg19/GRCh37 VCFs to GRCh38 before import into Cohorts.
Attribute Queries: Improved the user experience choosing a range of values for numerical attributes when defining a cohort
Export Cohort to ICA Project Data: Improved the user experience exporting list of subjects that match cohort definition criteria to their ICA project for further analysis
Ingest Structural Variants into database
The Cohorts ingestion pipeline supports structural variant VCFs and will deposit all such variants into an ICA Base table if Base is enabled for the given project
Structural variants can be ingested and viewed in base tables
Needle Plot Enhancements
Users can input a numerical value in the Needle Plot legend to display variants with a specific gnomAD frequency percentage or sample count
The needle plot combines variants that are observed among subjects in the current project as well as shared and public projects into a single needle, using an additional shape to indicate these occurrences
Needle Plot legend color changes for Variant severity; pathogenic color coding is the same as the color coding in the visualization; differentiating hue between proteins and variants; and other color coding changes.
Needle plot tool tips that display additional information on variants and mutations are now larger and modal
The needle plot now allows to filter by gnomAD allele frequency and sample count in the selected cohort. Variants include links to view a list of all subjects carrying that variant and export that list.
Remove Samples Individually from Cohorts
Exclude individual subjects from a cohort and save the refined list
The subjects view allows users to exclude individual subjects from subsequent analyses and plots and save these changes
Subject exclusions are reset when editing a cohort
Subject Selection in Analysis Visualization: Users can follow the link for subject counts in the needle plot to view a list of subjects carrying the selected variant or mutation.
UI/UX: Start and End time points are available as a date or age with a condition attribute in the subject data summary screen.
Fixed Issues
Improved resilience against misconfiguration of the team page when there is an issue with Workgroup availability
Improved handling of multi-file upload when system is experiencing heavy loads
Fixed an issue to allow upload of zero-byte files via the UI
Fixed issue where other Bundles would not be visible after editing and saving your Bundle
Removed ‘IGV (beta)’ button from ‘View’ drop down when selecting Project Data in UI
API: Improved error handling for API endpoint: POST /api/projects/{projectId}/analysisCreationBatch
API: Improved performance of API endpoint: getbestmatchingfornextflow
Flow: Fixed an issue causing Analysis output mapping to incorrectly use source path as target path
Flow: Fixed an issue where the UI may display incorrect or invalid parameters for DRAGEN workflows which do not accurately show the true parameters passed. Settings can be confirmed by looking at the DRAGEN analysis log files.
Base: “Allow jagged rows” setting in the Scheduler has been replaced with “Ignore unknown values” to handle files containing records with more fields than there are Table columns
Base: Improved Base Activity view loading time
Base: Fixed an error message when using the API to load data into a Base Table that has been deleted
Bench: Fixed an issue resulting in incorrect Bench compute pricing calculations
Bench: Fixed an issue preventing building Docker images from Workspaces in UK, Australia, and India regions
Bench: Fixed an issue where /tmp path is not writeable in a Workspace
Cohorts:
Fixed issue where the bubble plot sometimes failed to display results even though the corresponding scatter plot showed data correctly.
The order of messages and warnings for ingestion jobs was not consistent between the UI and an error report sent out via e-mail.
The UI now displays any open cohort view tabs using shortened (“…”) names where appropriate
Issue fixed where ingestions with multiple errors caused halting to the ingestion queue.
The needle plot sometimes showed only one source for a given variant as opposed to all projects in which the variant had been observed in.
Issue fixed with unhandled genotype index format in annotation file to base database table conversion
Status updates via e-mail sometimes contained individual error messages or warnings without a text.
Fixed issue where items show in needle plot with incorrect numbering on the y-axis.
Fixed performance issue with subject count.
Widget bar-chart counts are intermittently cut off over four digits.
Fixed slowness when switching between tabs in query builder
Documentation
Refined and expanded reference documents for sequencer integration and analysis autolaunch, including new pages for sequencer reference and troubleshooting.
Added tutorial for accessing Base databases and tables via snowSQL.
Added tutorial for setting up a notification service and Lambda function in AWS for chaining pipelines.
Included guidance for setting override behavior for file output of advanced output mappings.
Added instructions to remove specific subjects from a cohort.
Documented cohort needle plot enhancements: List of subjects and specific frequency data entry.
Updated cohorts ingestion instruction including metadata documents, GWAS ingestion, OMOP ingestion, Structural Variant ingestion, and hg19 lift-over information.
Added reference documentation for using Cohorts attributes metadata template.
Added instruction for cohorts attribute query enhancement on numeric data.
Described selection of entitlements when creating a Project.
Described options for starting analyses from Analyses page, from Data page, and from a pipeline.
Updated FAQ with guidance on analysis run times.
Updated documentation about network settings for pipelines.
Described limitation on amount of input files for analyses.
Explained how to unlink a folder from a project.
Updated explanation for adding Terms of Use to a Bundle.
Included details for the Event Log.
Refined and expanded description of Base Activity screen.
Expanded description of Bench Activity screen.
Clarified details about S3 bucket behavior and Bring Your Own Storage bucket (BYOB) compatibility.
Provided guidance about deleting old versions of Bring Your Own Storage buckets.
Explained multi-factor authentication compatibility with Bench and provide workaround suggestion.
Updated CLI command index.
Fixed a bug in the CLI release history files and updated integrity check instructions.
General updates and improvements.
2023 March 23 - ICA v2.13.2
Fixed Issue
Fixed issue with BaseSpace Free Trial and Professional users storing data in ICA
2023 March 9 - ICA v2.13.1
Fixed Issue
Fixed an issue resulting in analysis failures caused by a Kubernetes 404 timeout error
2023 February 28 - ICA v2.13.0
Features and Enhancements
API: POST /api/projects/{projectId}/data:createDownloadUrls now supports a list of paths (in addition to a list of IDs)
Each tenant supports a maximum of 30,000 Projects
.MAF files are now recognized as .TSV files instead of UNKNOWN
Added VCF.IDX as a recognized file format
General scalability optimizations and performance improvements
Fixed Issues
After creating a new Bundle the user will be taken to the Bundle Overview page
Fixed an issue preventing the ‘Owning Project’ column from being used outside of Project
Fixed an issue which prevented changing the format of a file back to UNKNOWN
Fixed an issue causing inaccurate upload progress to be displayed for UI uploads. The Service Connector or CLI are recommended for large file uploads.
Fixed an issue showing an incorrect status for data linking batch jobs when data is deleted during the linking job
Fixed an issue allowing the region of a Project to be changed. Changing the region of a resource is not supported.
Strengthened data separation and improved resilience against cross-Project metadata contamination
Service Connector: Fixed an issue allowing download of a Service Connector when no operating system is set
Service Connector: Cleaned up information available on Service Connectors by removing empty address information fields
API: Fixed date formatting for GET /api/eventLog (yyyy-MM-dd’T’HH:mm:ss.SSS’Z’)
API: Fixed an issue where the GET users API was not case sensitive on email address
API: Fixed an issue causing the metadata model to be returned twice in PSOT /api/projects/{projectId}/samples:search
API: Fixed the listProjects API 500 response when using the pageoffset query parameter
API: The searchProjectSamples API returns Sample metadata for Samples shared via a Bundle
API: Fixed an issue causing createProjectDataDownloadUrls API 400 and 502 errors when server is under load
Flow: Fixed analysis failures caused by kubernetes 404 timeout error
Flow: Fixed an issue where Workflwos would prematurely report completion of an Analysis
Flow: Improved Pipeline retry logic to reduce startup delays
Flow: Fixed an issue where Nextflow pipelines were created with empty files (Nextflow config is allowed to be empty)
Flow: Removed the 1,000 input file limitation when starting an Analysis
Flow: Improved the performance of status update messages for pipelines with many parallel steps
Flow: Fixed an issue with overlapping fields on the Analysis Details screen
Flow: Deactivated the Abort button for Succeeded analyses
Base: Fixed an issue where Pipeline metadata was not captured in the metadata Table generated by the metadata schedule
Base: Error logging and notification enhancements
Bench: Fixed an issue where Workspaces could be started twice
Bench: Fixed an issue where the system checkpoint folder was incorrectly created in Project data when opening a file in a Workspace
Documentation
Added reference documentation for deep links to a folder.
Added descriptions of cursor and offset pagination when using the API.
Added guidance for Base JSON export.
Updated supported formats for Base table/query export.
Expanded explanation of default settings for the CLI.
Updated description of Base Activity screen.
Provided documentation for available Bench compute types.
Updated Nextflow DRAGEN tutorial to include details about how to obtain access to the docker image on ICA.
Improved documentation about pipeline activation codes.
Created new page for and updated list of available data file formats.
Updated guidance for requirements for adding assets to a bundle.
Explained limit of number of projects and bundles allowed per tenant.
Updated service connector reference docs.
Added guidance for Base schedule's optional parameters.
Explained progress details for service connector.
Added information about archiving or unarchiving a file.
2023 February 13 - ICA v2.12.1
Features and Enhancements
Analysis system infrastructure updates
Documentation
Added documentation for Analysis auto launch
2023 January 31 - ICA v2.12.0
Features and Enhancements
Added ability to refresh Batch Jobs updates without needing to leave the Details screen.
Projects will receive a job queuing priority which can be adjusted by an Administrator.
The text "Only showing the first 100 projects. Use the search criteria to find your projects or switch to Table view." when performing queries is now displayed both on the top and bottom of the page for more clarity.
API: Added a new endpoint to retrieve download URLs for data: POST/api/projects/{projectId}/data:createDownloadUrls
API: Added support for paging of the Project Data/getProjectDataChildren endpoint to handle large amounts of data.
API: Added anew endpoint to deprecate a bundle (POST /api/bundles/{bundleId}:deprecate)
API: If the API client provides request header "Accept-Encoding: gzip", then the API applies GZIP compression on the JSON response. This way the size of the response is significantly smaller which improves the download time of the response, resulting in faster end-to-end API calls. In case of compression the API also provides header "Content-Encoding: gzip" in the response, indicating that compression was effectively applied.
Flow: Optimized Analysis storage billing, resulting in reduced pipeline charges.
Flow: Internal details of a (non-graphical) pipeline marked ‘Proprietary’ will not be shared with users from a different tenant.
Flow: A new grid layout is used to display Logs for Analyses with more than 50 steps. The classic view is retained for analyses with 50 steps or less, though you can choose to also use the grid layout by means of a grid button on the top right on the Analysis Log tab.
CLI: Command to launch a CWL and Nextlfow Pipeline now contains the mount path as a parameter.
CLI: Version command now contains the build number.
CLI: Added support for providing the nextflow.config file when creating a Nextflow pipeline via CLI.
API: HTML documentation for aPipeline can now be returned with the following requests:
GET /api/pipelines/{pipelineId}/documentation/HTML
GET /api/projects/{projectId}/pipelines/{pipelineId}/documentation/HTML
API: Added a new endpoint for creating and starting multiple analyses in batch: POST /api/projects/{projectId}/analysisCreationBatch
Flow: Linking to individual Analyses and Workflow sessions is now supported by /ica/link/project//analysis/ and /ica/link/project//workflowSession/
Cohorts: Users can now export subject lists to the ICA Project Data as a file.
Cohorts: Users can query their ingested data through ICA Base. For users who already have ingested private data into ICA Cohorts, another ingestion will need to happen prior to seeing available database shares. Customers can contact support to have previously ingested data sets available in Base.
Cohorts: Correlation bubble plot counts now link to a subject/sample list.
Fixed Issues
Tooltip in the Project Team page provides information about the status of an invite
‘Resend invite’ button in the Project Team page will become available only when the invite is expired instead of from the moment the invite is being send out
Folders, subfolders and files all contain information about which user created the data
Files and folders with UTF-8 character are not supported. Please look at the documentation on how to recover from it in case you already have used them.
Improved performance for creating or hiding a Project in a tenant with many Projects
Service Connector: Updated information in the Service Connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format"
Service Connector: Folders within a Bundle can be downloaded via the Service Connector
Service Connector: Upload rules can only be modified in the Project where they apply
Service Connector: A message describes when a file is skipped during upload because it already exists in the Project
Service Connector: Fixed an issue where opening the Connectivity tab occasionally results in a null pointer error
Service Connector: Fixed an issue causing excessive logging when downloading files with long file paths
Service Connector: Fixed an issue where the Service Connector log may contain spurious errors which do not impact data transfers
Existing storage configurations are displayed and accessible via API and UI
Newly added storage configurations do no longer remain in ‘Initializing’ state
Fixed error when creating a volume configuration with more than 63 characters
Clicking on a Data folder in flat mode will now open the details of the folder
Only Tools in Released state can be added to a Bundle
Fixed issue preventing new Bundle versions to be created from Restricted Bundles
Deprecated Bundles are displayed upon request in card and table view
Bundles view limited to 100 Bundles
API: Fixed the API spec for ProjectDataTransfer.getDataTransfers
API: Fixed an issue with the projectData getChildren endpoint which returned incorrect object and pagination
API: Fixed an issue where multiple clicks on Create sample batch API endpoint resulted in multiple requests
API: POST /api/projects/{projectId}/data/{dataId}:scheduleDownload can now also perform folder downloads
API: Improved information on the Swagger page for GET /api/pipelines, GET/api/projects/{projectId}/pipelines, and GET/api/projects/{projectId}/pipelines/{pipelineId}
API: Fixed and issue when a user provides the same input multiple times to a multi-value input on an analysis run, that input is only passed to the pipeline once instead of multiple times: POST /api/projects/{projectId}/analysis:nextflow
CLI: Copying files in the CLI from a local directory on MacOS to your Project can result in both the desired file and the metadata file (beginning with ‘./’) being uploaded. The metadata file can safely be deleted from the Project
CLI: Hardened protection against accidental file overwriting
CLI: Improved handling for FUSE when connection to ICA is lost
CLI: icav2 projectdata mount –list shows updated list of mounted Projects
CLI: Paging improvements made for project list, projectanalyses list, and projectsdata list
CLI: When there is no config or session file the user will not be asked to create one for icav2 config reset and icav2 config get
CLI: Fixed an issue where Bundle data could not be seen through FUSE in Bench
CLI: Fixed an error message when missing config file upon entering the Project context
CLI: The unmount is possible without a path and will work via the stored Project ID or with a directory path resulting in an unmount of that path
CLI: Fixed an error when creating a Pipeline using URN for Project identifier
CLI: Attempting to delete a file from an externally-managed project returns an error indicating this not allowed
CLI: Fix to delete session file when config file is not detected
CLI: Paging option added to projectsamples list data
CLI: Fixed “Error finding children for data” error in CLI when downloading a folder
CLI: projectdata list now returns the correct page-size results
Flow: Fixed handling of special characters in CWL pipeline file names
Flow: Fixed an issue where task names exceeding 25 characters cause analysis failure in CWL pipelines
Flow: Fixed an issue which prevented requests for economy tier compute
Flow: Fixed an issue limiting CWL workflow concurrency to two running tasks
Flow: Fixed an issue where analysis file inputs specified in the input.json with ‘location’set to an external URL cause to CWL pipelines to fail
Flow: Fixed an issue resulting in out of sync Pipeline statuses
Flow: Improved Nextflow engine resiliency, including occurrences where Nextflow pipelines fail with ‘pod 404 not found’ error
Flow: Fix issue with intermittent system runtime failures incorrectly causing analysis failures
Flow: Fixed an issue where links to Analysis Details returned errors
Flow: Enabled scrolling for Pipeline documentation
Flow: Improved performance for handling analyses with large numbers of inputs
Flow: Improved handling of hanging Analyses
Flow: Improved error messages for failed Pipelines
Flow: Added documentation on how to use XML configuration files for CWL Pipelines
Flow: Duplicate values for multi-value parameters are no longer automatically removed
Flow: Correct exit code 0 is shown for successful Pipeline steps
Base: Fixed an issue so that only users with correct permissions are allowed to retrieve a list of Base tables
Base: Fixed an issue with metadata scheduler resulting in a null pointer
Base: An empty Table description will not return an error when requesting to list all Tables in a Project
Base: Jobs failed with an error containing 'has locked table' are not shown on the Base Job activity list. They can be displayed by selecting the 'Show transient failures' checkbox at Projects > Activity > Base Jobs.
Base: Users can see Schedulers and their results for the entire tenant if created by a tenant administrator in their project, but not create, edit or run them
Base: Fixed an issue preventing data format change in a schedule
Base: Fixed an issue preventing exporting data to Excel format
Bench: Improved handling to prevent multiple users in a single running Workspace
Bench: Fixed an issue causing Workspaces to be stuck in "Starting" state
Bench: Fixed an issue where usage does not showing up on usage CSV-based report
Bench: Fixed an issue where Bundle data could not be seen via the Fuse driver
Bench: Users can now consistently exit Workspaces with a single click on the ‘Back’ button.
Bench: After leaving a Workspace by clicking on the ‘Back’ button, the Workspace will remain in a ‘Running’ state and become available for a new user to access
Bench: Workspaces in a ‘Stuck’ state can be manually changed to ‘Error’ state, allowing users to restart or delete them
Cohorts: Fixed issue where file system cleanup not occurring after delete.
Cohorts: Fixed sign in and authentication issues in APN1 region.
Cohorts: Fixed issue where gene filter missing when editing a cohort and removing the edited filter and cancelling. The filter was preserved and should not have been.
Cohorts: Fixed issue where users see an application tile in the Illumina application dashboard selection screen called "Cohort Analysis Module".
Cohorts: Correlation: Fixed issue, Data type selections shows half when loading the search result
Cohorts: Fixed issue, Users will see an application tile on the Connected Platform home page screen called “Cohort Analysis Module” if the Cohorts module is added to the domain. Users should not enter the ICA Cohorts through this page. They should enter through ICA."
Documentation
Added guidance on using Cohorts data in ICA Base.
Included video overview of Cohorts Multiomics Cancer workflow.
Updated steps for Cohorts ingestion.
Updated steps for Cohorts analysis.
Provided a command index for the ICA v2 CLI.
Expanded explanation on default settings for the CLI.
Clarified details about hidden metadata files behavior during CLI copy for MacOS users.
Added information about standard and economy tier selection for compute types.
Updated information in the service connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format".
Expanded guidance for XML-based input form required of pipelines defined using the "Code" mode.
Expanded guidance on importing external docker images in ICA when running a pipeline.
Included information about tile/grid view for analysis logs.
Added guidance about balancing analysis queuing priorities when setting up a Project.
Updated reference docs for workflow session events.
Provided details about price-per-sample and zero-cost pipelines.
Added details about which data formats are compatible with UI Preview.
Updated information about Batch Jobs.
Added details about syntax for linking to individual analyses and workflow sessions.
Described pipeline Proprietary flag available.
Added guidance about overriding CWL workflow requirements.
Explained how to add assets to a bundle.
2022 December 13 - ICA v2.11.0
Features and Enhancements
New notifications added for data upload, data created, and analyses with ‘aborted’ state
Flow: Progress on batch jobs can now be monitored in the Activity screen under the ‘Batch Jobs’ tab
Base: Shared database activity is available in the Base Activity Logging Screen
Cohorts:
Enhanced Attribute Queries: Filter logic now allows users to include or exclude filter criteria.
Ingestion Wizard: Human readable project name added to email notification of ingestion status
Ingestion Wizard: Notification email now includes the project name and job failure details
Ingestion Wizard: File lists are now sortable and filterable by type
Ingestion Wizard: UI enhancements on ingestion wizard selections
Ingestion Wizard: Based on data type chosen, the list of files for ingestion will be filtered for easier navigation of files to ingest
Share Cohorts with Collaborators - Cohort definitions are now sharable in a project
Statistical significance on Cohort Analysis - A p-value calculation is added to the Survival Comparison data points
Fixed Issues
Resolved an error when creating a Bucket Name with more than 63 characters
Fixed an issue causing failure when attempting to link "complex" folders (i.e., contains 100 sub-folders each with sub-folders) via GUI throws a “…transaction is not active…" error
Fixed an issue causing intermittent failure of "complex" folder (i.e., contains 100 sub-folders each with sub-folders) downloads
Fixed an issue so that the API to create data download URL correctly returns a 503 instead of a 404 response when the underlying system is experiencing network-related issues
Fixed an issue where the user is not redirected to the Project overview view upon clicking ‘Hide a Project’
Fixed an issue which prevented linking and unlinking multiple Bundles to and from a Project in a single operation via the GUI
Fixed an issue to show records in Bundle Data Transfers when data from a Bundle is accessed via View Tab and when downloaded
Fixed an issue showing stack trace when attempting to add Secondary Data to Data when not allowed
Fixed issue where the File Upload selection disappears after navigating back to Data after navigating to the Data History Tab
Fixed an issue where create folder session and get upload URL APIs are allowed for data accessible via an entitled Bundle
Fixed an issue where OpenAPI spec for delete project bundle API does not include proper "accept" header when using "Try It" functionality
Fixed an issue where write-access can be acquired for data accessible via an entitled Bundle
Fixed an issue where Project owner of an externally managed Project could not open Analysis Overview
Fixed an issue where multiple OS versions of the Service Connector are incorrectly downloaded at once when a single version is selected
Fixed an issue when selecting the "empty" option in the Operating System drop-down when downloading a connector
Fixed an issue where users with Contributor role and no data upload access can add data to samples without error
Added an event log entry for Service Connector log uploads initiated by Illumina Support
Flow: Fixed an issue to support Aborting an Analysis immediately after starting it
Flow: Fixed an issue when publishDir in Nextflow pipelines contains symlinks to data, those data are not included when uploading analysis outputs to a Project
Flow: Fixed an issue preventing closure of tabs in the Logs section for technical steps in the Analysis Details view
Flow: Cancel button added to the define CWL Pipeline selection dialogue window, and the order of buttons has been changed
Fixed an issue causing pipelines to stall for hours after all steps are completed
Flow: Fixed an issue where Nextflow pipelines fail with ‘pod 404 not found’ error
Base: Linked files from a Bundle can be inserted in a Base table
Base: Fixed an error on previewing Base tables that have been shared from a Bundle
Base: Fixed an issue where Base API to retrieve a list of Base jobs is not paging correctly
Bench: Fixed an issue where Bench Workspaces could not be opened with a double-click
Bench: After the “Save and Start” of a new Bench Workspace, the breadcrumb shows the actual name of the Workspace
Bench: Clarified error messages when creating new Workspaces with file permissions
Cohorts: Issue fixed where cohort is created during filtering process for RNA Expression on Apply
Cohorts: Issue fixed where order of items in the Survival Curve comparison legend are shuffled when reloaded
Cohorts: Issue fixed where long comparison names caused by a string concatenation between cohorts were causing the comparison feature to malfunction
Cohorts: Issue fixed where comparison name updates not reflected in the UI
Cohorts: Issue fixed where option still remains checked after Molecular filter removed from the right panel while on the project tab
Cohorts: Issue fixed where TCGA data not showing Molecular breakdown for selected genes
Cohorts: Issue fixed where data selections in the correlation feature where the second option visually shows half of the result in the UI
Cohorts: Issue fixed where "missense_and_ptv_all" missing from PHWAS/GWAS variant types
Documentation
Added guidance about modifying existing Tools already added to a Pipeline
Clarified expected behavior and recommended order of steps when connecting input to Tools and editing Input Details when building graphical CWL pipelines
Added information about details for monitoring Batch Jobs in Project Activity Screen
Updated tutorials for building a DRAGEN pipeline using Nextflow
Added Pipeline “tips and tricks”
Added guidance on expected behavior when configuring user-managed AWS Buckets with Projects
Updated list of system event notifications
2022 November 8 - ICA v2.10.0
Features and Enhancements
Added format auto-detection for VCF (.hard-filtered.vcf.gz, .cnv.vcf.gz) and TAR (.v8.tar, .tar.gz) files
Pipeline documentation will be available if present when starting a new Analysis
Added Workflow Session event topics
Base: Shared databases are visible in Base tables. Shared databases are read-only and queryable tables originating from ICA modules such as Cohorts
Reduced pipeline spin up time delays due to shared files system provisioning
Added support for Illumina applications to add custom attributes to data, samples, workflow sessions, and analyses
API: "creatorId" for Project data records returned in the API
API: Added instructions for paging parameters on list endpoints in API specification
API: The edit and delete API endpoint for Terms of Use are no longer available
Updated ICA CLI to version 2.10.0
Fixed Issues
Fixed an issue allowing unauthorized users to access external-managed Project contents via a direct link
Fixed an issue in the Docker Repository view causing region text and check boxes to be misaligned
Fixed an issue causing storage configuration status to incorrectly report an error
Fixed an error preventing the creation of an AWS S3 storage configuration in a Project
Fixed an issue where the Service Connector status incorrectly showing ‘Currently Connected’ when the associated Connector is not connected
Fixed an intermittent error when clicking to expand a Connector rule details in the Connectivity View
Removed red exclamation mark icon next to "Unlinked Files" tab in the Samples view; improperly indicates an error
Fixed an issue where Bundle Terms of Use acceptance flag unset upon saving bundle version
Fixed an issue preventing users from viewing details of an Entitled Bundle
Fixed an issue resulting in an error message when creating Bundle Terms of Use with an existing version
API: Fixed an issue where the API endpoint to link data to a Bundle to responds with errors when data is part of a Reference Data
API: Fixed an issue causing 500 API error response when retrieving sample details
Fixed an issue causing incomplete list of users when adding users within the same tenant to the Project Team
API: Updated the API spec description for create upload URL and create inline view URL for Project Data
Fix to prevent editing of an existing active terms of use
Improved the Terms of Use UI flow for forcing re-acceptance upon update
API: Added a warning response for API to create custom events when event code exceeds 50 characters
API: Fixed the create data download URL API to correctly return a 503 instead of a 404 response when the underlying system is experiencing network-related issues
API: Added API endpoint to hide a Project
Fixed an issue with Boolean inputs incorrectly displayed as false on Analysis Details view (correct value is actually passed in)
Fixed an issue where CWL parameters containing only a white space are passed incorrectly to underlying pipeline runner
API: Fixed an issue where launch Nextflow analysis API returns a 500 error response when including duplicate parameters
Fixed an issue causing analysis failures due to docker image authorization check timeout
Fixed a shared storage issue causing analyses to remain In Progress until timeout
Fixed an intermittent issue causing Analyses to be stuck in progress with inability to view logs in the UI
Fixed an issue where Analysis output data is not immediately shown in the Analysis Details view upon completion
Fixed an issue causing Analysis failures showing “…Connection timed out (Connection timed out)…” in stdout logs
Fixed an issue causing CWL pipeline runner restarts during an Analysis which may lead to unexpected results
Fixed an issue resulting in stuck analyses when invalid Docker image is specified for a task; timeout reduced to 15 minutes
Fixed the name of Pipeline running logs tab from "workflow_runner" to "pipeline_runner" in Analysis Details view in the UI
Fixed missing start time on Analysis tasks and incorrectly reported costs for such tasks
Fixed an issue resulting in an error message when attempting to create a graphical CWL pipeline with a Tool with a Docker image that is not available in the Project's region
Fixed an issue causing errors when saving a new Bench Workspace
Fixed an issue causing out of sync Bench table and card views when a Workspace is created or deleted
Fixed an issue causing an error when starting Bench Workspace when enabling service account on the Workspace from tenant receiving the Workspace via cross-tenant share
Documentation
Added tutorial for using scatter/gather for CWL graphical pipelines
Added information about Documentation tab in Nextflow pipelines
Included guidance on creating custom events for projects via the API
Provided list of ICA-supported data file formats
Updated links and fixed typos
2022 October 17 - ICA v2.9.0
Features and Enhancements
ICA Cohorts is released in production and added to the ICA user documentation
Clicking on a folder in the Data view will open the folder. Once a Folder is opened, the Folder details are available on the top right of the screen.
Added XML Configuration documentation and examples
General ICA robustness enhancements
Fixed Issues
Limit the Projects Overview view to 100 Projects with an indication that filtering must be used to see excluded Projects to improve UI performance
Fixed an issue preventing removal of permissions on externally managed projects
API – Fixed a 500 error response from Search Project Samples API when using a page token parameter
Fixed an issue allowing Bundles to be saved without mandatory Release Version when back-navigating from the Create New Bundle screen in the graphical UI
Fix to prevent data added to Bundle via secondary data upload on Data Details view
API – Fixed an error message when attempting to link data to a Bundle from a Project with Data Sharing disabled
API – Fixed an issue with tag filtering on getBundleSamples API
Fixed an error when filtering on size in Docker Repository where size filter is ignored
Fixed an issue where Analysis stuck in Aborting status when abort request sent during final stages of Analysis (such as Output, Upload)
Fixed an issue where intermittent Analysis failures are caused by system scaling errors under high load
Reduced repetitive Nextflow runner log entries from internal system logs to improve readability
API – Fixed an error in the Launch Nextflow Pipeline API when invalid parameter names are requested
API – Correctly support multi-value input parameters in launch Analysis APIs
Deprecated Tools are only visible when ticking the box "Show deprecated Tools" box in both table and card views
Fixed an issue preventing files linked to a Project from a Bundle as available to load into a Base Table
When exporting a query result from Base the file receives the name specified by the user and contains the header from the table
Removed unsupported AVRO and PARQUET formats from Available Formats to load data into Base tables
Fixed an issue where Bench service account API key is not revoked when a Workspace is stopped
Documentation
Added ICA Cohorts Module documentation, including summary of features and example walk-throughs
Added tutorial for building DRAGEN pipeline using Nextflow on ICA
Added tutorial for building DRAGEN pipeline using CWL on ICA
Added XML configuration documentation and examples
Added metadata model documentation
Added instructions for setting up ICA v2 Bring Your Own Bucket (BYOB) with SSE-KMS key
Updated supported ICA regions
Added data privacy recommendations regarding uploading data files
Updated bucket name character length constraints
Added guidance on supported characters in file/folder names
Updated fields and descriptions under Tool Settings
Added documentation for linking bundles to a project
2022 September 12 - ICA v2.8.0 and v2.8.1
Features and Enhancements
Added support for providing a nextflow.config file with a Nextflow Pipeline to improve pipeline building experience (see Nextflow Configuration docs)
Each tenant is limited to 100 in-progress Analyses at any given time to prevent resource exhaustion; additional analyses will be queued. Note: For assistance with planning batch jobs which require greater than 100 concurrent analyses, please contact Illumina Tech Support.
Added support for deep link URLs for direct navigation to child views in UI
API – Added an API to link Data to Projects in batch
API - Limit number of tags on data to 100; Excess tags in request will be ignored
API - Added "slug" attribute to Pipeline, Project, and Data asset types
API - Added support for the use of Project slug in place of ID in API request bodies
API - Added “Uniform Resource Name” (URN) attribute to Pipelines and Project object types
API - Added Project ID to notification payload for Pipeline Succeeded/Failed events
API - Added owning Project ID for data records in the getAnalysisOutput API response
API - Indicate originating Workflow/Workflow Session on Analyses returned via API, when applicable
Added support to use bundled Pipelines and data to create Analyses via API without first linking to the Project
Added support to automatically mount project data in Bench Workspace with ICA CLI
Fixed Issues
Renamed “Delete” to “Manage” in the Samples page dropdown menu
Removed the IGV button under View when selecting a VCF file
API – Fixed an issue causing the List Files API to incorrectly respond with an error when passing a folder or file path parameter value containing a leading or training space
Fixed an issue where files linked through a Project Connector will appear on root level as folder linking is not supported yet in the Project Connector
Fixed an issue when deleting files from S3 along with records from ICA via either volume delete (with purgeobjectstore=true) or folder delete API calls, if there are more than 1,000 items to delete the system deleted only the first 1,000 items, however, all the data records got deleted from ICA. In this case the data in the user-managed S3 bucket and ICA was out of sync.
Fixed an incorrect count displayed in UI when scheduling a download for a folder containing sub-folders
Fixed an issue where scheduling a download of a folder containing sub-folders creates duplicate data transfer operations
Disabled support for "fileNameExpression" on download rules; if provided when creating a download rule, the value will be ignored
Fixed an issue allowing public domain users to incorrectly access ICA
API – Fixed an issue where API to retrieve folder upload session intermittently responding incorrectly with error
API – Fixed a 500 response from the API to add a pipeline to a bundle when pipeline has already been added
Fixed an issue where user with "Viewer" permission for Flow in a project cannot link Samples from that Project
Return the correct error response when creating a new Nextflow or CWL pipeline through the API or CLI when an existing pipeline with the same name already exists
Improved UI messaging when setting up a Storage Configuration with permission error
Fixed an issue when attempting to edit a saved incomplete Tool (i.e., missing Docker image)
API - Fixed API to create notification channel to properly respond with error when attempting to create a duplicate notification channel
Fixed an issue where boolean Analysis input settings are incorrectly set to false when passed to the Pipeline runner
Fixed an issue when launching a Nextflow pipeline via API with multi-value inputs the order of the listed multi-value input is not preserved when passed to the pipeline runner
Fixed an issue where "Re-run" analysis button is broken for Nextflow pipeline analyses
When different versions of DRAGEN are run on the same host, reset the DRAGEN state so that the next task can correctly configure the FPGA compute instance
Fixed an issue causing intermittent analysis failures due to system error when evaluating permission to pull Docker image
Fixed empty start time attribute for Analysis tasks with a short duration (within a few seconds)
Fixed an issue where analysis start time is set when in fact the analysis is still queued; analysis start time will be set when compute resources (i.e., shared file system) have been requested
Fixed an issue where log file content for the pipeline runner is overwritten if system error occurs during an Analysis causing a pod restart; multiple log files may be created for the pipeline runner task if such an error occurs
Fixed an issue causing UI log stream view for Analysis tasks to incorrectly show errors or blank views
Fixed an issue causing incorrect Analysis task log to be displayed on the UI Logs view
Fixed an issue where analyses get stuck in Running status if input file paths contain leading or trailing spaces in path for analysis stuck in running status if input files contain leading or trailing spaces
Fixed an issue causing Analysis to be incorrectly marked as Success due to incorrect evaluation of exit code of an analysis task
Fixed an issue preventing the same file to be passed to multiple input fields when launching an Analysis
Workflow failures due to unavailable compute resources will fail quickly with a more descriptive error
Fixed an issue where analysis output data is not immediately shown in the Analysis Details view after the analysis completes
Removed the “Print” and “Save” buttons in the Analysis Report
Fixed an issue preventing Pipeline creation and Tool editing when an invalid Tool is saved in the Tool Repository
API – Fixed a 500 response from API to list Projects to properly return a 400 response when providing a search filter value with invalid characters
API - Fixed a 500 response from create download URL API to properly return a 400 response when data is not in "Available" status (ie, Partial/Archived)
Fixed an issue with loading data into a Base Table created directly in Snowflake with fields containing quotes
Fixed an issue where data was added to Base Table twice when multiple Scheduled Jobs run simultaneously
Fixed an issue with previewing Base Tables created directly via Snowflake
Fixed an issue where status is stuck at "In Progress" for Base Copy Table operations
Fixed an issue where Base Tables linked to a project from a bundle are shown as Views (read-only)
Fixed an intermittent issue causing duplicate data in Base Tables
API - Fixed a 500 response when sending a duplicate requests to load data for a Base Table to properly return a 400 response
API - Fixed a 500 response from API to load data into Base Table to correctly return a 400 response when attempting to load data with a type mismatch
API - Corrected Swagger UI responses
Fix error "Unable to save workspace" when attempting to start a Bench Workspace
Documentation
Added documentation for analysis output mappings
Added documentation for Nextflow configuration
Added documentation for reference data
Added documentation for metadata models
Added Quick Registration Guide
Added tutorial for creating/running CWL pipeline on CLI
Added end-to-end tutorial highlighting DRAGEN Germline pipeline
2022 July 27 - ICA v2.7.1
Fixed Issues
Fixed an intermittent issue where analyses fail due to "Connection timed out" system error (seen in analysis error logs)
Added retry logic to decrease rate of pipeline failures due to internal system error (e.g. errors in pipeline runner log such as connection timeout, 404 pod not found, 500 errors)
Fixed an intermittent issue where "Cannot access folder" error is shown in pipeline error logs (most common in long-running analyses)
Fixed an issue resulting in analysis failures due to system error under high load
Fixed a compute instance configuration issue impacting DRAGEN analyses
2022 June 29 - ICA v2.7.0 and CLI v2.4.0
Features and Enhancements
API - Added method to retrieve an analysis JSON input data
Added Analysis Details views in the UI
Added Analysis ID to the project analyses
Support deep links for direct navigation to Analysis Details view
Added support for multiple Nextflow versions when creating Nextflow pipeline
API - Added method to create multiple samples in batches
API - Improved response time performance of API that launches analysis with many data inputs
Fixed Issues
Repaired issue where Pipeline-generated Sample Metadata is not picked up by platform
Restored Time to delete function; now executes file deletes at the specified time
Bench example code now lists project data and pagination correctly
Restored ability to link secondary data to primary data
Fixed issue where batch sample creation jobs get stuck in PARTIALLY_SUCCEEDED status
Resolved issue where batch sample creation is not completed if data items could not be linked
API - Fixed issue where mountPaths were ignored when launching analyses via API
Restored function so that bundle contributors can link assets to their bundle
Restored data linking to a sample in an ICA project
Resolved issue where child workflow configurations prevent termination of parent workflow
Removed incorrect error reported in cases where previous failure occurred
Restored pipeline linking for bundle contributors
Improved UI messaging for Storage Configuration permission errors
API - Updated GET /api/projects/{projectId}/data to returns correct file details
2022 June 07 - ICA v2.6.0 and CLI v2.3.0
Features and Enhancements
Re-labeled "Run" and "Runs" to “Analysis” and “Analyses” in the UI
Allow storage configuration to be disconnected from a hidden Project
Added support for a read-only view of Entitled Bundles and Bundles shared from another Tenant
Changed analysis timeouts for setting up environment, downloading inputs, pipeline runner (inclusive of all child steps), and finalizing output steps to 96 hours
API – Add idempotency key header to createAnalysis API
API – Analysis logs are available via getAnalysisSteps API; includes streamable via websocket endpoint during execution and downloadable S3 URL upon completion
JSON data with array field can be added in a Base table
Cohorts – Additional submenu items now available
CLI – Added support for setting the current Project ID to the value set in the ICA_PROJECT_UUID environment variable upon starting the CLI
CLI – Added support for mounting Project data to local operating system (Read-Only)
CLI – Added support for the outputParentFolderId flag for the ‘projectpripeline start’ command
CLI – Added support for customizing the filename/path of additional Pipeline files with the pipeline create commands
CLI – Added owning Project name/id attributes to the project data list output
CLI – Added owning Project name attribute to output for projectdata list command
CLI – Added commands to create and refresh API tokens (JWT)
CLI – Added support for CLI use without writing to disk by enabling API Key and Server flags for all commands
CLI – Added support for linking data to a Sample with projectdata upload command
Fixed Issues
Fixed an issue with UI stability leading to slow response times
Fixed an issue causing data uploads via browser to hang without error
Fixed an issue with UI automatically refreshing upon load when many Projects are present in the Projects view
Fixed an issue where Accept Share pop-up remains on screen after pressing to Accept the share
Fixed an issue with uploading nested folders using the Windows Connector
Fixed an issue where path separators are misinterpreted in upload rules for Windows Connectors.
Fixed an issue where modifying a local file uploaded (tracked) using the Connector causes a Connector failure
A User with Contributor Role for Flow and a Viewer Role for a Project can start an Analysis using input data from a Sample
API – Fixed an issue where sampleDeepDelete API fails with 500 error response when Sample contains a linked folder in archived status
Fixed an issue where ICA_STARTING_USER_TENANT_NAME is not set to the correct value on initial start
API – Fixed an issue where upload rules are not deleted via the deleteUploadRule API despite a success response
Sensitive fields in metadata models are now shown in italics
Fixed an issue where filtering on the Data view gives incorrect results
All Contributors in a Bundle are allowed to link assets to the Bundle
Content of a Bundle shared via an entitlement in another Tenant is now visible
Fixed an externally managed Project Team to only allow adding members with "Viewer" and "Allow Download" permissions
API – Fixed an issue with project sample APIs returning 500 error response when sample is not linked to the project specified in the request
API – Fixed issues with cursor-based paging on listData and listBundleData APIs returning expected results
Fixed an issue where a User can create empty folders inside an externally managed Project with Viewer permission
Fix auto-detection of AVRO and PARQUET data formats
API – Fixed an issue where the API to link Samples to a Bundle incorrectly returns a 204 response when the Sample is not in Available status and the Sample is not actually linked
API - Fixed an issue where changing the Project owner causes unexpected error responses from the listProjects, listProjectPermissions, and getProjectPermission APIs
API – Fixed an issue where deleteDownloadRule API returns 204 responses but does not delete the specified download rule
Fixed an issue causing delays in operations performed on data
Fixed an intermittent permission error when writing to /scratch from a Pipeline
Fix to make tags optional (not required) with Pipeline launch requests
Fixed an issue causing Analyses to be incorrectly set to SUCCEEDED status when failure occurs
API – Possible to create a Pipeline with a metadata model via API
Fixed an issue where logs are not streamed during Analysis
Fixed an issue when a User tries to start an Analysis with missing required parameters, the analysis is stuck in "awaiting input"
Fixed an issue preventing renaming a pipeline after attempting to save and receiving error that a pipeline exists with the same name
API – Fixed an issue when a Pipeline is launched via API with no input value provided for a nonmandatory input parameter the analysis status remains in "Awaiting Input" indefinitely
API – Fixed an issue causing unpredictable error responses from createAnalysis API when many Analyses are launched in rapid succession
API – Added operationID to API spec for the listAnalysisStorages and getAnalysisRawOutput APIs
Fixed an issue when Analyses are queued and Aborted before they are Running, the status is not correctly updated in the API/UI, and they appear stuck in “Aborting”
API – Fixed an issue with createAnalysis API causing error responses when launching many Analyses in rapid succession
Possible to load JSON in Base with a repeated field (array)
Tables created by the meta and admin data Schedules are updated after a sync
Fixed an issue causing Base Table field names to be changed from lowercase to UPPERCASE during a manual sync
Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when schedules are run
Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when scheduled
Fixed an issue where the User receives an obscure error message when creating a Tool from a Bench Workspace with an image name that already exists
Fixed an error message upon attempting to save Bench Workspace when input values are invalid
Fixed an issue where the Bench Workspace is temporarily inaccessible and stuck in Starting status
CLI – Fixed an issue with list samples command output limited to 20 unique records
2022 April - ICA v2.5.0
Features and Enhancements
The value of the "Show Hidden Projects" filter on the Project Overview page will be saved for each user
Change analysis task log names to be more human-readable
Analyses will queue when system load is high
Add owning project tenant name (ICA_PROJECT_TENANT_NAME) and the tenant name of the user which last started the Workspace (ICA_STARTING_USER_TENANT_NAME) environment variables to Bench Workspaces
Add Cohorts URL environment variable to Bench Workspaces
API - Add support for Cross Origin Resource Sharing (CORS) to the API
Fixed Issues
Fixed an error message in UI when attempting to link a Sample that cannot be linked
API - Fix issue where API to update Sample metadata fields does not update the fields
Fixed error message when creating a Bundle with a name conflict
Fixed error message when a user attempts to view Bundle details without permission
API - Fix Project Bundle APIs to properly link, retrieve, and unlink entitled Bundles
Fixed to block ability to link data with Contributor role and no upload permission on the target project
Fixed an issue where the Owner of an externally managed Project cannot add Team members with Viewer or Download permission
Fixed field for providing an e-mail address when adding a Team member to a Project to be required
Removed temporary directory creation by service connector
Fix for configuring service connector to use proxy settings on Linux clients
Fixed an issue where data downloaded through Service Connector is incomplete
Fixed an issue causing errors in Service Connector when a directory is deleted in the UI but still present on the local machine. Data added to the folder on the local machine will not be uploaded if the corresponding folder is deleted in the UI.
Fix to display an error message when saving a Pipeline with invalid fields
Fix issue where CWL file/directory inputs are not mounted when provided within nested structures (ie, CommandInputRecordSchema, CommandInputArraySchema)
Fix to show error when attempting to run a graphical CWL pipeline with an invalid definition
Fix error when attempting to save a graphical CWL Pipeline with a Tool with an output folder and no specified format
API - Fix to ignore input JSON field when launching graphical CWL pipelines via API
API - Fix ignored default compute type (machine profile) defined in the Pipeline definition when launching a graphical CWL Pipeline via API
Fixed Base Query result export file format set to UNKNOWN
Fixed Base activity log to show Table associated with log event
API - Ignore unknown attributes in request bodies
API - Fixed API error responses formatted as HTML instead of JSON in error scenarios
API - Fix list analyses API not returning the "items[].analysisStorage" field
2022 April - ICA v2.4.0, v2.4.1, v2.4.2
Features and Enhancements
Samples may only be linked when status is Available
Added Data ID to grid view in UI
Added ability to select and copy tag values to the clipboard
Added an indication of source domain in list of reference data sets when adding new data to Reference Data
Updated the service connector client icon
Added ability to filter projects by workgroup, including projects where user is a member of a workgroup added to the project
Added ability to specify output directory within a project when launching a pipeline
Added ability to Favorite Projects and filter the Projects view by Favorites
Remove obsolete commented text from auto-generated CWL Tool definitions
Scheduler displays the number of files to be loaded into a Base Table
Added an indication that Base Table previews display up to 50 columns and 100 rows
API - New endpoint to manage SecondaryData on Data objects
API - Added support for "mounts" parameter when launching a pipeline to provide input mount paths for file and folder inputs
API - Added fields to API error response schema: HTTP Method, User ID, Tenant Name, Tenant ID
API – Added new endpoints for retrieving Entitled Bundles and to manage Project Bundles
Fixed Issues
Docker images available via entitled bundles are listed in green text in the Docker Repository view
Fixed an issue in OpenAPI spec causing invalid cURL requests in 'Try it Out' functionality for the create upload URL API
Fixed an issue that resulted in empty Workgroup listing via the API and when adding Workgroups to a Project Team
Fixed an intermittent error when navigating to Bundles view
Fixed an issue where Tools available via Bundles are not displayed in the Tool Repository
Fixed an issue where example JSON generation for root Metadata Model fails when the root Metadata Model is modified
Fixed an issue where incorrect reporting of compute start times resulted in incorrect compute billing
Fixed an issue where Nextflow analyses get stuck "In Progress" when output folder is deleted during an ongoing pipeline run
Fixed an issue where incorrect casing on the scatter method 'dotproduct' in CWL auto-generated from graphical pipelines results in failures in pipelines with scatter-gather steps
CLI - Fixed an issue with startAnalysis APIs no longer accepting the 'Accept' header with value "application/problem+json,application/vnd.illumina.v3+json”
API/CLI - Fixed issue where graphical CWL pipelines launched via API/CLI do not use machine profiles (compute types) specified in the pipeline definition
Fixed an issue where pipelines within shared bundles could not execute across regions
Fixed an issue where exported Base Tables with null values populate a CSV with 'N' values instead of empty values
Fixed an error when loading data into Base Tables with an invalid number of columns
Fixed an issue causing View Results in Base Query History to return an expiration error
Fixed an issue that caused JSON import into Base Tables to fail when sub-field (record type) is used in the Table schema
Updated the error message when invalid SQL is used in a Base Query
Fixed an issue with JSON import to Base Tables failing when reference data fields are enabled
Fixed an issue where updated Base Table fields in the UI do not update the underlying Snowflake Table
Fixed an issue resulting in "Unable to create file error" message when creating a Tool in a Bench Workspace
Fixed an issue where Bench Workspaces cannot be deleted
Fixed an intermittent issue where Bench Workspaces unexpectedly fail in an error state
Updated messaging for users without access to Variant Viewer and Cohorts
API - Fixed an issue where the API did not accept multiple 'Accept' header values
API - Fixed an issue where the create folder upload sessions API returns date and time with incorrect formatting
API - Fixed the 500 response on APIs when the user has not yet accessed the UI
API - Fixed missing version header in OpenAPI spec for APIs to link a pipeline to a Project or Bundle
API - Fixed the 500 response from create project API when the 'storageConfigurationSubfolder' value is invalid
API - Fixed a 500 response from the API to link data to a bundle when linking a Folder with child data records
API - Fixed an issue where the API to create File data record creates the data record at the root of the project data when a trailing forward slash ('/') is missing after the folder name
API - Fixed an issue where API to retrieve pipeline reference sets does not display linked reference data
API - Fixed an issue where APIs to initiate analysis perform validation of required properties in request body
2022 March - ICA v2.3.1
Fixed Issues
Fix issue where pipeline launch is blocked when a single file is used as the input to multiple pipeline inputs
2022 March - ICA v2.3.0
Features and Enhancements
A Base query is now possible with null marker 'nan'
A user can search on Secondary Data tab
A user is allowed to run a pipeline from a shared project if the Billing mode is set as 'Tenant'
Add "application" field to Project API model to indicate whether a project is externally managed.
An error is shown when a user tries to build a workspace that is not running.
An error is thrown when the user uses the same name of an existing project connector rule in the same project
Base query can be exported as a view
Bug fix for allowing multiple files with same name but different paths to be passed as input; causes conflict when downloading inputs to pipeline filesystem, one file will be overwritten (random) causing unpredictable results
Data filters and columns now also apply to technical tags
Externally managed projects are now supported via API
Meta data schedule can be created on tenant level
New ICA notification for Data/Sample linked
Preview of the content of a Base table is available with null marker nan
Project connector permissions are aligned with manual linking permissions
Remove obsolete "BlueVantage" option from the Source drop down selections during Base schedule creation
Revert cwltool upgrade
Samples linked via a project connector are visible in the 'receiving' project
The 'Type of Files' field in the configuration of Upload Rules in a Connector is changed to 'Assign Format' so that its purpose is clear
The project connector rules are available across tenants when the user has Contributor rights
The user is able to use the filters 'Type' and 'Status' and sort the files
The yaks-scheduler used to check the pod-to-node fit, but will now defer to the kubernetes default-scheduler as the pod-placement authority. The assumption is that when the pod is not scheduled, there are no nodes available, and a node will be provisioned.
When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.
[Base] When selecting 'Include reference', you will see the record first and when you unfold the record, you will see the fields belonging to the record
When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.
[API] Add ability to filter by region for list projects API
[API] Add psToken authorization type to enable creating JWT from psToken via the create token API
[Bench] Add ICA_PROJECT_UUID environment variable to the Jupyter Notebook Bench image.
Fixed Issues
Fix 400 error response when launching an analysis with a pipeline from a bundle
Fix Base Query results file export set to format UNKNOWN
Fix Bench pricing calculation accuracy
Fix for Base table export unexpected failure
Fix for auto-linking pipeline outputs files to a sample when input files provided are linked to the sample
Fix for failed Base table import when imported data contains "nan" (null marker) for a column of type integer
Fix for linking Base Query templates to Bundles
Fix for modifying Bench Workspace image unless Workspace is in Stopped state
Fix for more informative API error response when EULA has not been accepted
Fix for preserving sample data links for samples linked from shared bundles
Fix for retrieving download URL for data in an externally managed project.
Fix for silent error when running Base schedule on Administrative Data or Metadata
Fix intermittent UI issue where checkbox for file selection during docker image add is partially visible
Fix issue where Base schedule for Metadata does not work when source is set to "Tenant"
Fix issue where Base table schema definition changes from record to string after sync
Fix issue where Project Connectors cannot link samples where the source project is owned by a different tenant
Fix issue where an analysis cannot be started when a metadata model is defined for the pipeline
Fix issue where data cannot be removed from a Sample using the "Remove Data" button from the sample details view.
Fix issue where pipelines from linked bundles do not show as available selections in the "Generated by Pipelines" drop down when creating a Base schedule
Fix issue where samples cannot be updated after configuring a Project Connector rule with the sample.
Fix issue where scroll bar is not shown in the metadata models list view
Fix issue where secondary data is not shown in the reference data view
Fix issue where team members added to eternally managed projects cannot be granted Contributor permissions, and cannot link data to other projects
Fix issue where user cannot use sample filtering when samples and files are manually linked to their project and the files do not appear associated with their sample in that project
Fix issue where user is unable to see their input files when using sample filtering if they are linked from another project without selecting the flat view option
Fix issue where workgroups cannot be added to project team of externally managed projects with Contributor permission
Handle K8s api timeouts by retrying query.
Improve pipeline analysis startup times to approximately 10 mins maximum (times may vary)
[API] Fix 500 response for getProjectDataLinkedProjects API
[API] Fix error response when invalid API Key is provided for authorization
[API] Fix get bundles API error response when user has access to a project with a linked bundle that the user does not have access to
[API] Fix getProjectData API to return correct results when using the "id" query param
[API] Fix incorrect error response for Project Data API to create folder upload session
2022 February - ICA v2.2.0
Features and Enhancements
the cwl workflow engine logs now also show debug logs, these includes the javascript expressions evaluated etc. these logs are similar to running
cwltool --debug
locallyShow "username" of pipeline creator in Pipelines overview view
Removed excessive logging from the "Audit Trail" logging.
Various performance improvements - reuse of connections to Base instances
A particular storage configuration should only be available during project creation in case it is your own storage configuration, or when the owner (or a tenant administrator) gives explicit access to it for other users in their tenant.
Bench - when trying to get an access token through the API, access was denied in case Base was not enabled within the project.
Base - 'You are not allowed to modify data' was shown to the user when clicking the Sync button as a Contributor
Base - importing data with format JSON in a table was not possible.
Files within a sample will only show up in case the data is also explicitly linked to the project/bundle.
Bundles - a user with Contributor access did not have the necessary access rights for managing all resource of the bundle.
Project billing modes can be 'project' or 'tenant'. In case 'tenant', the users from another tenant than the owner of the project was not able to upload files.
Within the Activation Codes view, activation codes for access to the Variants/Cohorts modules were not listed.
It was not possible to add a bundle at project creation. The project had to be saved before the bundle could be added.
The roles explanation in the Team page on a permission record did not show the entire information.
Various stability improvements - In case of a database outage, ICA was not able to automatically reconnect.
When managing docker images, a partial docker image was not allowed to be deleted nor updated resulting in a stuck image.
When exporting the contents of a grid from the UI, the exported file contained tenant ids instead of tenant names.
Base - the export of query results to a json format resulted in a failure in case the query ended with ';'
A non-tenant-admin user was not able to create a metadata model.
Base - exporting query results, specifying a custom delimiter did not work.
When a user clicks the bundle card in the "Bundles" view, and the user has no access to manage the bundle because it is shared with the user by another tenant, the following notification will be displayed: "Details for a bundle from another tenant are not accessible" instead of "Bundle details unavailable"
When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.
Various performance improvements - linking a folder with a lot files to another project or bundle timed out.
Base - loading metadata into a table using a schedule resulted in the same data being added over and over again.
When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.
Limit usable character set for the user reference of an analysis run
When an analysis was aborted, the output files which were generated so far were not visible in the details of the analysis, but only in the data view.
Base - schedule metadata population in a table did not work.
Added support for Docker Images to be added to Bundles
Added Project Connectors in the project connectivity section to enable creation of automatic sample and data linking rules between projects.
Added Queue Date to analysis Logs view to indicate when the specified job was submitted to the underlying infrastructure.
Added support for externally-managed projects for Illumina software applications built on top of ICA to manage resources inside ICA projects while limiting tenant users' direct access to the project to read-only capacity.
Improved performance of API calls when using API Key for authentication
Added tenantName to API resource DTOs
Add API to abort analysis run
Added APIs to share a storageConfiguration, share a storageCredential, update a storageCredential secret
Added mime-type field to the DataFormat DTO for the data APIs
Added "objectEtag" field to DataDto for Data APIs to expose e-tag from the cloud provider object store
Added Base APIs to retrieve Jobs, retrieve Tables, and load data into Table
API - POST‚/api/projects/{projectId}/samples:search was missing request parameters to browse through samples in a paged manner.
API - POST /api/projects/{projectId}/analysis:cwl - tags were not persisted when starting a CWL analysis run
The PUT /project/{projectid}/data/{dataid} gives the user the impression that all fields can be updated on the data, but it is a different set for files as for folders. This distinction is not properly documented.
API - GET /api/projects/{projectId}/analyses - when the pipeline did not belong to the tenant, but the user still has access to it (eg: through a bundle), the analysis details could not be retrieved through the API.
API - GET /api/projects/{projectId}/analyses/{analysisId} returned the details of an analysis, even if the projectid did not match the actual project the analysis ran in.
API - GET /projects/{projectId}/samples/{sampleId} did not expose metadata information
API - GET /projects/{projectsId}/data presented the user with an unclear error message in case providing invalid request parameters.
API - An unclear error was presented in the problem response in case an invalid json was provided as request body.
API - An unlcear error was presented in the problem response when creating a project using invalid uuids in the request body.
API - Removed deprecated "sample matching strategy" endpoint
CLI - Added command to update project data
CLI - Added command to retrieve a list of project samples
CLI - Added command to create a project sample
CLI - Added command to list available data formats
CLI - API Key encryption modified to use local value on machine where API Key is configured to prevent hijacking config.
CLI - Added command to get details of a project sample
CLI - Added command to list data linked to a project sample
CLI - Added command to update a project analysis
CLI - Added command to update a project sample
CLI - Added command to unlink data from a sample
CLI - Added commands to delete a project sample
CLI - Added command to link data to a sample
CLI - Added command to complete a sample
CLI - Removed unnecessary flags on the config command.
CLI - For starting CWL and Nextflow pipelines, the flag 'storage-size' was ignored by CLI, and the default storage size of the pipeline was taken.
Fixed Issues
[bugfix] when CWL workflows failed or produced null as one of the outputs, all the outputs were incorrectly processed. in this case, the user may not be able to see the details of the analysis output in UI/API.
[Base] Fix error message when viewing a query from query history that no longer returns results
[bugfix] in some cases after aborting an analysis, it may remain stuck in aborting state indefinitely. Now more resources are explicitly cleaned to avoid this issue from happening.
Fixed bug when showing logfiles of steps in an analysis.
[bugfix] cwl workflow sometimes fails in spite of all steps in the workflows have succeeded. Made the workflow engine robust to detect and retry in this case
When linking a folder containing +2000 files to another project, only 2000 files were actually linked.
When a user did not choose a country when filling out the user account details, login to ICA could fail.
Fixed bug when handling custom output objects in CWL.
Fixes issue with compute type selection when launching CWL graphical pipelines
The Project > Team permissions showed a little pencil in front of the row indicating a modification. With this fix, the pencil will only show when there were actual modifications done to the row in particular.
API - All dates in API responses were formatted in 12 hours notation without AM/PM notation. This way a time in PM is presented as AM. With this fix, all time notations are in 24 hours.
When starting an analysis through the sample page, the start window is not closed after pressing start. Users would click it again, which would start the analysis multiple times. With this fix, the start window is closed after pressing start.
When starting an analysis fails because the backend service returns an error, the status of the analysis is "Awaiting Input". With this fix, the analysis goes to "Failed".
Base - in case the export of query results failed, an empty file was created and never cleaned up. With this fix, there are no leftovers in case of export failure.
When a sample was linked from another project to the current project, the details view of the sample shows all data linked to it, even of the current project does not have access to it. With this fix, only the data which is also linked to the current project will be shown on the details of the sample.
API - PUT /project/{projectid}/data/{dataid} update sometimes had to be done twice for it to persist. With this fix, all editable attributes can be updated at once.
2021 Decemeber - ICA v2.1.0
Features and Enhancements
[CLI] Storage size can be entered by name
[CLI] Details of a project analysis can be retrieved by entering the analysis Id
[CLI] A list of metadata models can be retrieved
[CLI] A list of regions can be retrieved
[CLI] A list of storage configurations can be retrieved
[CLI] A pipeline can be linked to a project by using the name/id of the pipeline as an argument
[CLI] A pipeline can be unlinked from a project by defining the name/id as an argument
[CLI] A list of storage bundles can be retrieved
[API] Usability improvements for the list project data API:
Modify filePath query param to behave as prefix instead of exact match by default
Add filePathMatchMode query param
Add parentFolderPath query param
[API] Add API endpoint to refresh a JWT (POST /api/tokens:refresh)
[CLI] Add CLI command to create projects
[API] Support search by tags on samples search API
Renew macOS connector certificate
When a filter is applied and you select a folder, you can see the content from that folder without th* Connector rules for a project can only be changed when you are active in that project
It is possible to link more than 2000 files to a bundle or another project
Data coming from a project in another tenant can be linked when you are invite for collaboration
The error from a failed pipeline analysis is visible on the details page
Species and type are unique within a tenant, not across tenants
Sqs and sns test messages are aligned with the email test message
A collaborating user from another tenant is able to see the details view of a pipeline analysis run.
Bundle data can be used via API
When executing api/projects/{projectId}/data and a file already exist, there is an indication that thresponse.
Connector cancel button is renamed to delete to make its functionality more clearly
[CLI] Modify pipeline start command to use pipeline name or ID as an argument instead of a flag parameter
[API] Removed endpoint to POST on a sample metadata model
Bugs Fixed
[Base] Fix error message when viewing a query from query history that no longer returns results
[CLI] Fix for CLI upload command when uploading a folder with 0 bytes files
[API] Fix create bundle API to return error when bundle with existing name is attempted to be created
[API] Fix for non-functional If-Match header on PUT requests
[CLI] Fix folder upload "panic: close of closed channel" error
[API] Fix API for project creation returning success with invalid storage configuration provided
[Bench] Fix for docker build logs not showing when build is interrupted
Fix population of sample metadata from multiple analyses on the same sample
Fix error message for missing docker image on tool creation
[CLI] Fix pipeline create command to make pipeline code an argument instead of a flag parameter
Fix UI element for tags to remove active button appearance
Fix error when attempting to link pipeline with no pipeline selected
Fix error message when attempting to add field to Base table schema with invalid characters
Fix issue with analysis outputs files and logs not showing after analysis completion
Fix UI issue with pipeline selection in the Generated by Pipelines field of the view to create a Base s
Fix issue with a duplicate workspace being created while editing a workspace
Fix issue for table creation using schedule for metadata
Fix Base viewer role when applied to workgroups
Fixed Issue: optional parameters for Nextflow pipeline cause error when value not provided
Fixed issue when selecting data records to export from the Data overview page on a Bundle
Fix for Base saved queries not visible in shared project
Fix minor price calculation error (rounding) for Base storage
2021 November - ICA v2.0.0
Features and Enhancements
API cursor-based and offset-set paging
Authorize calls using Illumina API key
Multiple reference data elements are possible when starting an analysis
APIs added for:
Account, User, Workgroup
Connector, Download Rule, Upload Rule
Data, Data Format, Data Transfer
Region
Download
Entitlement
Event Log
Project, Project Analysis, Project Data, Project Permission, Project Sample, Project Pipeline
Metadata model
Pipeline
Bundle
Notifications, Event Codes
Base
OpenAPI v3 API Spec published at https://ica.illumina.com/ica/api/swagger/index.html
Added sort parameter to several APIs
Custom event payload is defined as a json
Separate call to retrieve metadata model fields available
Add rclone temporary credentials support in the API to retrieve folder upload session details
Tokens API POST also contains basic authentication
Activation code endpoints are aligned with start analysis endpoints
Base templates cannot be deleted
Information can be written from Bench to Base
Viewer/Contributor permissions are introduced in Snowflake
Saved Template details can be edited
Access via Illumina authentication is available
Snowflake access token can be created
Added new Bundles feature for sharing data with other users
ICA allows the user to connect an S3 bucket as storage instead of an Illumina managed storage
An overview of all storage configurations is available
Secrets can be managed for the storage configuration
A new storage configuration can be added
The health status of a configuration can be monitored
A storage configuration can be deleted when not used in a project
Storage type and health are displayed in the overview
When selecting own storage ICA will return you a list of all BYOBs in the chosen region
A default bucket can be selected for every region
Data is automatically archived/deleted when the grace period (if present) is over
Data is archived/deleted automatically by a housekeeping mechanism
An admin user can change the date to be archived/deleted at any time
Data has a status
Data format is automatically detected at creation
Files can be grouped in a sample
Data can have additional linked projects
The data screen shows project data and data linked via a bundle
3 new columns are present in the data grid: "Owning Project", "Linked Projects" and "Linked Bundles"
The record is displayed in the data grid depending on the data being linked or not
Filter on Status with checkboxes on status values is possible
Total Data Size is available
Data can be linked/unlinked to/from a project
Only root parent files and folders are shown when opening the data grid
Deleted status does not exist anymore
Data from projects with Data Sharing disabled cannot be linked
Project is hidden instead of (soft-)delete
Metadata models can be created for linking to a project
Add support for Nextflow pipeline language
A pipeline input can have multiple formats
Output JSON for CWL analysis is shown
Ongoing logfiles are available for the user
You have two options to define a CWL pipeline: Graphical editor or code
Test connection button is available for SQS
Checkbox is added to toggle view between all rules or only your own rules
Notification channels can be validated
Subscription can be edited
Custom subscriptions are possible
Notifications can be received via SQS, SNS, Mail, and Webhook
All users have an overview of all the notifications used in their tenant
Notification channels are only visible for tenant admins
Notifications can have a filter expression to only receive specific events
A project administrator can grant permission to a single user on a project
A project administrator can invite somebody unknown to ICA to his project
A contributor can manage reference data
A project administrator can define who pays for using my project
A project administrator can invite a workgroup to a project
When multiple project roles are given to a user/workgroup the correct project role is applied
Team-page in projects and bundles are aligned
Reference data can exist in multiple regions
Species and type are being populated per tenant
Data can become reference data
Data tree shows all the regions in which the reference data is available
Last updated