Comment on page
Release Notes
- Data Management:
- Fixed an issue where multiple folder copy jobs with the same destination may get stuck In Progress
- Fixed an intermittent issue where tags on the target folder for a batch data update call are not set, but are set for all child data
- Flow:
- Fixed an issue causing intermittent pipeline failures due to an infrastructure error
- Navigation: If multiple regions are enabled for the same tenant, the region will be indicated in the waffle menu
- Logging: Data transfers of BaseSpace Sequence Hub projects with data stored in ICA will be traced in ICA logs
Cohorts
- Disease Search Box: Added support for specifying subjects by age of onset of disease(s)
- Drug Search Box: Added a new query builder box for Drugs
- Ingestion: Support for Drug, drug route, etc. attached to subjects
- Cohorts building: Users can build cohorts by specifying drugs, drug route, etc.
- Ingestion:
- Combine different variant types during ingestion (small variants, cnv, sv)
- Cohorts supports Illumina Pisces variant caller for hg19 VCFs
- General:
- Fixed an issue where the graphical UI hands with ha spinning wheel when saving or executing a command
- Fixed an issue where rich text editor for Documentation tab on Pipelines, Tools, Projects and Bundles does not populate with correct styles in edit mode
- Data Management:
- Fixed an issue where multiple clicks on create data in Project API endpoint resulted in multiple requests
- Fixed an issue where the secondary data selection screen could not be resized
- A spinning wheel icon with ‘copying’ status is displayed at the folder level in the target Project when a folder is being copied. This applies to the actual folder itself and not for folders higher up in the hierarchy
- Fixed an issue where API to retrieve a project data update batch is failing with 500 error when either the Technical or the User tags are updated during the batch update request
- Fixed an issue where linking jobs fail to complete if other linking jobs are running
- Improved performance for data transfer to support BaseSpace Sequence Hub Run transfers
- Fixed an issue causing some folder copy jobs to remain in "Partially Succeeded" status despite being completed successfullyBundles: Fixed an issue where the URL and Region where a Docker image is available is not displayed for a Docker image Tool shared via an entitled Bundle
- Fixed an issue where the folder copy job was getting stuck copying large amounts of big files
- Fixed an issue where the folder counts were not matching expected counts after Data linking
- Fixed an issue where delete data popup would occasionally not disappear after deleting data.
- Fixed an issue with data copy where referencing data from another region would not result in immediate failure
- Fixed issue where uploading a folder using the CLI was not working
- Fixed an issue where a Docker image shared via an entitled Bundle can be added to another region
- Workflows: Fixed an issue where workflow does not fail if BCL Convert fails for a BCL Convert-only run
- Flow:
- Improved performance when batches of data up to 1000 are added as input to an Analysis
- Nextflow engine will return exit code 55 if the pipeline runner task is preempted
- Fixed an issue where log files cannot be opened for any steps in an analysis while the analysis is in progress
- Fixed an issue with concurrent updates on analysis
- Fixed an issue where unknown data inputs in the XML of an analysis are not being ignored
- The warning, close, and machine profile icons for Tools can now be seen in the graphical CWL pipeline editor
- Fixed an issue where user cannot expand analysis output folder if user permissions change after starting analysis. Now, if a user has the correct permissions to start an analysis, that analysis should be able to finish correctly no matter the permissions at the time it succeeds
- Base:
- Fixed an issue switching back from template to Empty Table did not clear the fields
- Data linked from an externally managed project can be added to Base Tables
- Fixed an issue in the graphical UI where schema definition does not scroll correctly when many columns are defined
- General updates and improvements
- Sequencer Integration: Added information that auto-launch from instrument is not currently compatible with MiSeq systems
- Data Management
- Documented an alternate method of adding files to samples and explained how to remove files from samples
- Provided more guidance on data transfer options
- Updated reference documentation for Data Copy functionality
- Added a message to inform users that folder copy can be impacted in cases where a folder with the same name already exists in the target project
- Flow
- Described analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes
- Updated ‘CWL: Scatter-Gather’ tutorial
- Added documentation for analysis timeout
- Base:
- Added a new tutorial for creating a Python dashboard to access Base metrics via API
- Added information about Snowflake user/role setup
- Included sample TSV file in Base Basics tutorial
- Bench: Updated Fuse Driver troubleshooting
- Notifications: Updated Filtering section of Notifications page
- CLI: Updated CLI release history
- Reference
- Updated Illumina Connected Software registration guide
- Added reference documentation for data integrity recommended practices
- Data Management/API: Added a new endpoint available to change project owner
- POST /api/projects/{projectId}:changeOwner { “newOwnerId”:”}
- Data Management/API: Added a new endpoint to copy data from one project to another:
- /api/projects/{projectId}/projectDataCopyBatch
- Data Management/CLI: Added the ability to copy files and folders between projects in the UI and CLI. This includes support for copying data from projects with ICA-managed storage (default) to projects with S3-configured storage.
- Flow/API: When starting an analysis via the API, you can specify the input files based on HTTP(s). When your analysis is done, you will see the URL corresponding to the inputs in the UI, but you will not be able to start an analysis from the UI using this URL
- Flow: Removed the ability to start analyses from data and sample views in the UI where a single input is selected to start analyses in bulk
- Flow/CLI: Duplicate analyses submitted via the CLI will be avoided
- Flow/API: Added two new endpoints for workflow sessions:
- Get /api/projects/{projectId}/workflowSessions
- Get /api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs
- Flow/API: Added a new endpoint to retrieve configurations from a workflow session
- Flow/Autolaunch ICA Workflow Session and Orchestrated Analyses (launched by the workflow session) now saves outputs in an organized folder structure: /ilmn-analysis/<name_used_to_create_sequencer_run_output_folder>
Base: The Base module has a new feature called ‘Data Catalogue’. This allows you to add usage data from your tenant/project if that data is available for you.
- Data Catalogue views will be available and can be used in Base to query on.
- You will be able to preview and query Data Catalogue views through Base Tables and Query screens
- The Data Catalogue will always be up to date with the available views for your tenant/project
- Data Catalogue views cannot be shared through a Bundle
- Data Catalogue views will also be available to team members that were added after the view was added
- Data Catalogue views can be removed from the Base tables and corresponding project
- By removing Base from a project, the Data Catalogue will also be removed from that project
Cohorts: Disease Search box
- Cohorts now includes a disease search box to search for disease concepts. This replaces the disease concept tree explorer
- Disease search box located under a Disease tab in main Query builder
- Search box allows for a copy/paste action of codes to be processed as separate query elements. Currently, the feature is limited to a complete valid list
- Each disease entered into the search box is displayed as a separate query item and can be set to include or exclude.
- Diseases in search box can be used with boolean logic in cohort creation
- Search box allows for an auto-complete of diagnosis concepts and identifiers
- The disease filter is included in the cohort query summary on cohort page
- Data Management: Data copy between ICA-managed projects and S3 storage configured projects is supported
- Data Management: Fixed an issue where storage configurations matching ICA-managed buckets would cause volume records to get associated with the wrong storage configuration in the system
- Data Management/API: The endpoint GET/api/projects/{ProjectID}/samples/{SampleID} correctly returns all the own samples and linked samples
- Data Management/API: Improved handling of bulk update via API when concurrent deletion of file has occurred
- Data Management/CLI: Fixed an issue where projectdata update tags would not update the tags
- Flow: Fixed an issue resulting in failure to send a notification resulting in a failed workflow
- Flow: Fixed an issue where one workflow session may override another when both are executed at the same time
- Base: Fixed an issue where query download in JSON format returns an error
- Base: Added a message in the UI when a query takes longer than 30 seconds to inform the user that the query is ongoing and can be monitored in the Activity view
- Base: Added a section describing the Data Catalogue functionality
- Bench: Fixed an issue where resizing the workspace to current size would prevent users from resizing for the next 6 hours
- Cohorts: Fixed an issue where Gene Expression table does not display with TCGA data or for tenants with a hyphen (e.g., ‘genome-group’)
- Cohorts: Fixed an issue where user had no way to delete a cohort comparison from a deleted cohort
- Cohorts: Fixed an issue in the UI where multi-cohort needle plot tracks are overlapping
- Cohorts: Fixed an issue causing failures during annotation step with ‘CNV’ data type when selection ‘GB=hg19’ and ‘CNV data’ for liftover; also observed with ‘SM data’ and ‘hg38’ without liftover (in APS1 and CAC1 regions) due to a ‘404 Not Found’ error.
- CLI: Fixed an issue to support adding the server-url as a parameter instead of having the config set
- General updates and improvement
- General: Added link to Illumina Knowledge resource
- Data Management: Add documentation around storage configuration synchronization behavior
- Data Management: Documented the new Data Copy functionality
- Data Management: Provided more guidance on data transfer options
- Data Management: Added a section describing Data Catalogue functionality
- Data Management: Added guidance about not using file names beginning with '/'
- Service Connector: Added troubleshooting guidance for service connector
- Sequencer Integration: Updated analysis autolaunch reference docs
- Sequencer Integration: Updated the NovaSeq 6000 analysis auto-launch tutorial
- Flow: Improved the tutorial for Scatter-gather method in Nextflow
- Flow: Improved tutorial for Nextflow: RNASeq Lift
- Flow: Added a tutorial for converting open-source nf-core pipelines to ICA Nextflow pipelines
- Flow: Updated reference docs for starting analyses
- Flow: Improved tutorial for scatter-gather in CWL
- Base: Improved descriptions and scenarios of Base tables and queries
- Cohorts: Documented Cohorts Diagnosis Concept attribute search box functionality
- CLI: Updated the CLI release history page with all required build files
- API: Added tip for users experiencing API rate limiting
- API: Added troubleshooting guide for GetTemporaryUploadCredentialsAsync failure error scenario
- SDK: Included new reference to Illumina's ICA Python SDK resource available on GitHub.com
- Fixed an issue uploading folders via the CLI
- Fixed an issue causing CWL pipelines using Docker images that do not contain bash shell executable to fail.
- Fixed an issue leading to intermittent system instability.
- Cohorts: Issue fixed where GTEx plot is not available for tenants with a hyphen (e.g. ilmn-demo).
- Data Management: New API endpoints are available for:
- Creation of a data update in bulk: POST/api/projects/{projectId}/dataUpdateBatch
- A list of data updates for a certain project: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}
- A list of items from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items
- A specific item from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items/{itemId} Note: Batch updates include tags, format, date to be archived and date to be deleted
- Data Management/API: The sequencing run information can be retrieved through its Id by using the API endpoint GET/api/sequencingRuns/{sequencingRunId}
- Flow: Auto launch now supports BCL Convert v3.10.9 pipeline and both TruSight Oncology 500 v2 pipelines (from FASTQs)
- Flow/API: getPipelineParameters API returns parameter type in response
- Flow/API: Added endpoints to retrieve and update a project pipeline definition
- Flow/API/CLI: Include "mountPaths" field in response for API and CLI command to retrieve analysis inputs
- Flow: Removed "fpga-small" from available compute types. Pipelines using "fpga-small" will use the "fpga-medium"-equivalent compute specifications instead
- Flow/API: New API endpoint available to request the analyses in which a sample is being used
- Flow/API: When leaving activationCodeDetailId empty when starting an analysis, the best match activation code will be used
- Flow: Analyses launched/tracked by BaseSpace Sequence Hub contain relevant BaseSpace information in analysis details view
- Bench: A new Bench permission is being introduced: Administrator. This permission allows users to manages existing workspaces and create new workspaces
- Bench: The Bench Administrator role allows you to create new Bench workspaces with any permissions even if you as a Bench administrator do not have these permissions. In that case, you can create and modify the workspace, but you cannot enter that workspace. Modifying is only possible when the workspace is stopped
- Bench: As a Bench Contributor you are not allowed anymore to delete a Bench Workspace, you need the Bench Administrator role.
- Cohorts: Users can now ingest raw DRAGEN bulk RNAseq results for genes and transcripts (TPM), with the option to precompute differential expression during ingestion
- Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
- Cohorts: In multiple needle plot view, individual needle plots can now be collapsed and expanded
- Cohorts: Pop-outs for needle plot variants now contain additional links to external resources, such as UCSC
- Cohorts: For a given cohort, display a distribution of raw expression values (TPM per gene) for a selected attributes
- Cohorts: Use of the Cohorts maintains session between core ICA and Cohorts iFrame to prevent unwanted timeouts
- Cohorts: Cohorts displays structural variants that include or overlap with a gene of interest
- API: Two new API endpoints added to accept Terms and Conditions on a bundle:
- GET /api/bundles/{bundleId}/termsOfUse/userAcceptance/currentUser Returns you the time of acceptance when you, the current user, accepted the Terms & Conditions.
- POST /api/bundles/{bundleId}/termOfUse:accept
- API: Add temporary credentials duration to API documentation
- Notifications: List of events to which you can subscribe contains new ICA notification containing analyses updates
- Versioning: The ICA version can now be found under your user when you select "About"
- Versioning/API: It is possible to retrieve system information about ICA, such as the current version through GET/api/systeminfo
- Logging: When an action is initiated by another application, such as BaseSpace Sequence Hub, it will be traced as well in the ICA logs
- Data Management/API: Improved error handling for API endpoint: DELETE/api/bundles/{bundleId}samples{sampleId}
- Data Management: CreatorId is now present on all data, including subfolders
- Data Management: Improved external linking to data inside ICA using deep linking
- Data Management: Improved error handling when creating folders with invalid characters.
- Data Management: Fixed an inconsistency for URN formats on output files from Analyses. This fix will apply only for analyses that are completed starting from ICAv2.18.0
- Data Management: Improved resilience in situations of concurrent linking and unlinking of files and folders from projects
- Data Management/API: Fixed an issue where the API endpoint GET /api/samples erroneously returned a 500
- Data Management/API: API endpoint GET/api/projects/{projectId}/analyses now returns the correct list when filtering on UserTags whereas it previously returned too many
- Data Management: It is only possible to delete a storage configuration if all projects that are using this storage configuration have been hidden and are not active projects anymore
- Data Management: Improved accuracy of the displayed project data size. Prior cost calculations were accurate, but the project data size visualization included technical background data
- Data Management/API: Improved retry mechanism for API endpoint to create folderuploadsession
- Data Management/CLI: When an upload of a folder/file is done through the CLI, it returns the information and ID of the folder/file
- Data Management: Fixed an issue where there is a discrepancy in number of configurations between Storage->Configurations and Configurations-> Genomics.Byob.Storage Configuration view
- Collaboration: Fixed an issue where a user is presented with a blank screen when responding to a project invitation
- Flow/API: Improved error handling when invalid project-id is used in API endpoint GET /api/projects/{projectId}/pipelines
- Flow: When a spot instance is configured, but revoked by AWS, the pipeline will fail and exit code 55 is returned
- Flow/API: Fixed an issue when an Analysis completed with error "incomplete folder session", the outputs of the Analysis are not always completely listed in the data listing APIs
- Flow/API: Updated ICA Swagger Project > createProject to correctly state that the analysis priority must be in uppercase
- Flow: Fix to return meaningful error message when instrument run ID is missing from Run Completion event during an auto launched analysis
- Flow: Improved parallel processing of the same analysis multiple times
- Base: Improved error handling when creating queries which use two or more fields with the same name. The error message now reads "Query contains duplicate column names. Please use column alias in the query"
- Base: Fixed an issue where queries on tables with many entries fail with NullPointerException
- Bench: Clarified that changes to Bench workspace size only take effect after a restart
- Cohorts: Fixed issue where counts of subjects are hidden behind attribute names
- Cohorts: Fixed issue where the state of checked files are not retained when selecting molecular files that are in multiple nested folders
- Cohorts: Fixed issue where projects that contain files from linked bundles cause a time out, resulting in users not being able to select files for ingestion
- Cohorts: Fixed an issue where the 'Import Jobs' page loaded within the Data Sets frame, depending on where the import was initiated
- Cohorts: Fixed an issue in the Correlation plat where x-axis counts were hidden under attribute names
- Fixed an issue where users were previously incorrectly signed out of their active sessions
- General updates and improvements
- General: Added ICA Reference Terminology page
- Data Management: Document how to change Project owner
- Data Management: Updated Storage Size table for Compute on Pricing page to include XL, 2XL, 3XL
- Data Management: Improved explanation of hiding Projects when connecting to your own S3 bucket for data storage
- Data Management: Added default and shared explanation to storage documentation
- Data Management: Updated documentation to reflect the temp credentials returned for AWS datastore is for 36 hours
- Flow: Documented event for Analysis status transition
- Flow: Added Nextflow best practices to reduce Analysis failures
- Flow: Updated Pipeline activation code behavior explanation
- Flow: Added documentation for specifying external docker images (i.e. Docker Hub)
- Base/CLI: Added a tutorial for using Base via the CLI
- Base: Updated documentation to mention that Schema does not contain type ‘variant’
- Base: Documented Base query which joins to tables can result in duplicate columns
- Base: Documented SnowSQL show view vs show grants
- Bench: Documented Bench permission changes
- Cohorts: Documentation updates:
- RNASeq ingestion
- Applicable user-supplied RNASeq visualizations – Raw Expression Values in GTEx-like table
- NeedlePlot
- Structure Variant list
- CLI: Added documentation for CLI config settings
- Logging: Documented event logging transfer of ownership
- Fixed an issue causing analyses requesting FPGA compute resources to experience long wait times (>24h) or not be scheduled
- Data Management: Performance improvements for data link and unlink operations – Larger and more complex folders can now be linked in the graphical UI, and progress can be monitored with a new visual indication under Activity > Batch Jobs
- Flow: Increased the allowed Docker image size from 10GB to >20GB
- Flow/CWL: Added support for javascript expressions “ResourceRequirements” fields (i.e., type, size, tier, etc.) in CWL Pipeline definitions
- Flow/API: Added support for using Pipeline APIs to query Pipelines included in Entitled Bundles (i.e., to retrieve input parameters)
- Flow/API: Added support for providing S3 URLs as Pipeline data inputs when launching via the API (using storage credentials)
- Flow/API: Added support for specifying multi-value input parameters in a Pipeline launch command
- Bench: Project and Tenant Administrators are now allowed to stop running Workspaces
- Notifications: Notifications are now available for batch job changes
- Cohorts: Enhanced ingestion workflow to ingest RNAseq raw data from DRAGEN output into backend Snowflake database
- Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
- Cohorts: Multi-Cohort Marker Frequency - Added Multi-Cohort Marker Frequency tab allowing users to compare expression data across up to four Cohorts at the gene level
- Cohorts: Multi-Cohort Marker Frequency includes a pairwise p-value heat map
- Cohorts: Multi-Cohort Marker Frequency - Includes frequencies for Somatic and Copy Number Variants
- Cohorts: Tab added for a multi-cohort marker frequency analysis in cohort comparisons
- Cohorts: Multi-Cohort Needle Plot - Added new tab in the Comparison view with vertically aligned needle plots per cohort for a specified gene, allowing collapsible and expandable individual needle plots
- Cohorts: Additional filter logic added to multi-cohort needle plot
- Cohorts: Improved DRAGEN data type determination during ingestion allowing for multiple variant type ingestion
- Cohorts: Enhanced list of observed variants with grouped phenotypes and individual counts, including a column for total sample count; tooltips/pop-outs provide extended information
- Cohorts: Updates to needle plot link outs
- Cohorts: Improved the Comparison feature by optimizing API calls to handle subjects with multiple attributes, ensuring successful loading of the page and enabling API invocation only when the user selects or expands a section
- Cohorts: Removed unused columns (genotype, mrna_feature_id, allele1, allele2, ref_allele, start_pos, stop_pos, snp_id) from annotated_somatic_mutations table in backend database
- Cohorts: Refactored shared functionality for picking consequence type to reduce code duplication in PheWAS-plot and GWAS-plot components
- Cohorts: Invalid comparisons on the Comparisons page are now grayed out and disabled This improvement prevents the selection of invalid options
- Cohorts: Automatic retry of import jobs when there are failures accessing data from ICA API
- Data Management: The content of hidden Projects can now be displayed
- Data Management: Fixed the TimeModified timestamp on files
- Bundles: Resolved issues when linking a large number of files within a folder to a Bundle
- Flow: Single values are now passed as a list when starting an Analysis
- Flow: Pipelines will succeed if the input and output formats specified on the pipeline level match at the Tool level
- Flow: Fixed an issue causing Analysis failures due to intermittent AWS S3 network errors when downloading input data
- Flow/CWL: Improved performance on output processing after a CWL Pipeline Analysis completes
- Flow/UI: Mount path details for Analysis input files are now visible
- Flow/UI: Improved usability when starting an Analysis by filtering entitlement options based o inputs selected and available entitlements
- Flow/API: List of Analyses can now be retrieved via the API based on filters for UserReference and UserTags
- Base: Fixed an issue where the Scheduler continues to retry uploading files which cannot be loaded
- Bench: Resolved an issue when attempting to access Workspaces with multi-factor authentication (MFA) enabled at the Tenant-level
- API: Improved error messaging for POST /api/projects/{projectId}/data/{dataId}:scheduleDownload
- Navigation: Removed breadcrumb indication in the graphical UI
- Cohorts: Fixed issue where Correlation bubble plot not showing for any projects intermittently
- Cohorts: Fixed issue where importing Germline/hg19 test file did not load variants for a specific gene in the Needle plot due to missing entries in the Snowflake table
- Cohorts: Fixed a bug causing an HTTP 400 error while loading the Cohort for the second time due to the UI passing "undefined" as variantGroup, which failed to convert to the VariantGroup Enum type
- Cohorts: Fixed issue where scale (y-axis) of needle plot is changed even if value of sample count gnomAD frequency is not accepted
- Cohorts: Fixed an issue where no data was generated in the Base Tables after a successful import job in Canada - Central Region (CAC1)
- Cohorts: Fixed issue where long chart axis labels overlap with tick marks on graph
- General improvements and fixes
- User Management: Added details about how users can manage an overview of their Projects
- Data Management: Clarified the visibility of Batch jobs for different users in a Tenant
- Bundles: Provided more details about Bundle linking and unlinking with data in folders
- Flow: Described Analysis runtime environment used to execute user-defined pipeline tasks, steps, and processes
- Flow: Added an explanation for how external S3 files can be used as inputs to Pipelines
- Bench: Added an explanation that Project/Tenant Administrators can stop running Workspaces
- Bench: Added guidance for downloading data when using JupyterLab Notebooks
- Cohorts: Added instructions for RNA-Seq data ingestion into Cohorts
- Cohorts: Added reference documentation for multi-cohort marker frequency and needle plot comparisons
- CLI: Updated the CLI command index
- CLI: Updated the CLI release history page with all required build files
- CLI: Added link to legacy ICA v1 CLI release history page
- SDK: Included a new reference to Illumina’s ICA Python SDK resource available on Github
- Network Settings: Updated the list of whitelisted URLs on https://help.ica.illumina.com/reference/r-networksettings
- Data Management: You can now select 'CYTOBAND' as format after file upload
- Data Management: Added support for selecting the root folder (of the S3 bucket) for Projects with user-managed
- Data Management: Added support for creating an AWS Storage Configuration with an S3 bucket with Versioning enabled
- Auto-launch: Added technical tags for upstream BaseSpace Run information to auto-launched analyses
- Auto-launch: Added support for multiple versions of BCL Convert for auto-launched analyses
- Flow: Added support for '/' as separator in CWL ResourceRequirements when specifying Compute Type
- Flow/API: The API to retrieve analysis steps now includes exit code for completed steps
- Bench: Restricted Workspaces (Open or Restricted) always allow for access to Project Data within the Workspace
- Bench: Restricted Bench workspaces have limited access through whitelisted URLs that are checked before entry
- Bench: Restricted Bench Workspaces allow for Open or Restricted workspaces. Restricted workspaces do not have access to the internet except for user-entered whitelist URLs
- Data Management: Upload for files names including spaces is now consistent for connector and browser upload. We do still advise not to use spaces in file names in general
- Data Management: Fixed search functionality in Activity > Data Transfers screen
- Data Management: Improved performance on opening samples
- Data Management: Fixed an issue where reference data in download tab initiates an unexpected download
- Data Management: Fixed intermittent issue where the Storage configuration within a Project can go into Error status and can block users from creating records such as folders and files
- Data Management/API: Improved error handling for API endpoints: Delete/api/projects/ {projectId}/bundles/{bundleId} and POST/api/projects/{projectId}/bundles/{bundleId}
- Data Management/API: Improved error handling for POST/api/projects/{projectId}/base:ConnectionDetails
- Bundles: Fixed an issue where the Table view in Bundles is not available when linking to a new Bundle version
- Bundles/API: Improved error handling for DELETE/api/bundles/{bundleId}/tools/{toolId} and POST/api/bundles/{bundleId}/tools/{toolId}
- Bundles/API: Improved error message for POST/api/bundles/{bundleId}/samples/{sampleId}
- Bundles: Fixed an issue where linking/unlinking a Bundle with Base Tables could result in errors
- Service Connector: Improved error message for DELETE/api/connectors/{connectorId}/downloadRules/{downloadRuleId}
- Notifications/API: Custom subscriptions with empty filter expressions will not fail when retrieving them via the API
- Notifications/API: Improved error handling for POST/api/projects/{projectId}/notificationSubscriptions
- Notifications/API: Improved notification for Pipeline success events
- Flow: When the input for a pipeline is too large, ICA will fail the Analysis and will not retry
- Flow: Fixed issue where analysis list does not search-filter by ID correctly
- Flow: Improved error handling when issues occur with provisioning resources
- Flow: When retry succeeds in a Nextflow pipeline, exit code is now '0' instead of '143'
- Flow/API: Fixed an issue causing API error when attempting to launch an Analysis with 50,000 input files
- Flow/API: Improved pipeline error code for GET/api/projects/{projectId}/pipelines/{pipelineId} when already unlinked pipeline Id is used for API call
- Flow/API: Fixed an issue where Analyses could not be retrieved via API when the Pipeline contained reference data and originated from a different tenant
- Flow/API: Fixed filtering analyses on analysisId. Filtering happens via exact match, so part of the Id won't work
- Bench/CLI: Fixed issue where the latest CLI version was not available in Bench workspace images
- Navigation: Updated URLs for Correlation Engine and Emedgene in the waffle menu
- Authentication: Using POST /api/tokens:refresh for refreshing the JWT is not possible if it has been created using an API-key.
- Authentication: Improved error handling when there is an issue reaching the authentication server
- Authentication: Improved usability of "Create OAuth access token" screen
- Cohorts: Issue fixed where CNV data converted from hg19 to hg38 do not show up in Base table views.
- Cohorts: Issue fixed accounting for multiple methods of referring to the alternate allele in a deletion from Nirvana data.
- Cohorts: Fixed intermittent issue where GWAS ingestions not working after Base enabled in a project.
- User Management: Improved explanation of Team functionality
- Data Management: Added guidance on versioning Bring Your Own storage buckets
- Data Management: Improved explanation on linking to data from other projects
- Data Management: Described functionality of Secondary Data
- Data Management: Updated guidance for file upload size
- Data Management: Updated configuration guidance for Bring Your Own Storage bucket using S3
- Data Management: Updated list of supported Data formats
- Data Management/CLI: Added a tutorial for mounting project data using the CLI
- Flow: Added a tutorial for building and running a CWL graphical pipeline
- Flow: Included best practices for minimizing errors in Nextflow pipelines by adding retry logic
- Base: Updated the Base schedule descriptions to include information about Metadata and Administrative schedules, as well about frequency of scheduled ingestions
- Bench: Updated restrictions of available features when in the FUSE mount in a Workspace
- Bench: Updated descriptions about Bench restrictions, including details about whitelisted URLs
- CLI: Added CLI authentication best practices for setting up, validating, and resetting the configuration of a user's profile
- API: Added information to launch analysis API spec on output mappings overwrite behavior when files already exist on the target project during analysis output
- Network Settings: Updated Network Settings regional S3 URLs list
- General updates and improvements
- Fixed an issue causing incorrect empty storage configuration dropdown during Project creation when using the “I want to manage my own storage” option for users with access to a single region
- General availability of sequencer integration for Illumina sequencing systems and analysis auto launch
- General usability improvements in the graphical interface, including improved navigation structure and ability to switch between applications via the waffle menu in the header
- Storage Bundle field will be auto-filled based on the Project location that is being chosen if multiple regions are available
- Event Log entries will be paged in the UI and will contain a maximum of 1,000 entries. Exports are limited to the maximum number of entries displayed on the page.
- Read-only temporary credentials will be returned when you are not allowed to modify the contents of a file
- Creating Project notifications for BaseSpace externally managed projects is now supported
- The ICA UI will only allow selection of storage bundles belonging to ICA during Project creation, and the API will only return storage bundles for ICA
- Flow: Allow attached storage for Pipeline steps to be set to 0 to disable provisioning attached storage and improve performance
- Cohorts: GRCh37/hg19-aligned molecular data will get converted to GRCh38/hg38 coordinates to facilitate cross-project analyses and incorporating publicly available data sets.
- API: Project list API now contains a parameter to filter on (a) specific workgroup(s)
- API: Two new API endpoints are added to retrieve regular parameters from a pipeline within or without a Project context
- Optimized price calculations resulting in less overhead and logging
- Improved error handling:
- during Project creation
- of own storage Project creation failures.
- to indicate connection issue with credentials
- for graphical CWL draft Pipelines being updated during an Analysis.
- Fixed an issue causing data records to remain incorrectly in Unarchiving status when an unarchive operation is requested in the US and Germany regions
- Improved error messaging in cases where the AWS path contains (a) special character(s)
- Fixed an issue causing errors when navigating via deep link to the Analysis Details view
- CLI: Unarchival of folders is supported when archive or unarchive actions are not in progress for the folder
- CLI: Improved error message to indicate connection issue with credentials
- Flow: Fixed an issue causing incorrect naming of Analysis tasks generated from CWL ExpressionTools
- Flow: Fixed an issue when cloning Pipelines linked from Entitled Bundles to preserve the original Tenant as the Owning Tenant of the cloned Pipeline instead of the cloning user’s Tenant
- Flow: Fixed an issue causing outputs from CWL Pipelines to not show in the Analysis Details despite being uploaded to the Project Data Analysis output folder when an output folder is empty
- Flow: When a Contributor starts an Analysis, but is removed afterwards, the Analysis still runs as expected
- Flow: Fixed an issue where Analyses fail where Nextflow is run a second time
- Flow: Fixed an issue causing API error when attempting to launch an Analysis with up to 50,000 input files
- Flow: Fixed an issue causing degraded performance in APIs to retrieve Analysis steps in Pipelines with many steps
- Flow: Fixed an issue causing Analysis failure during output upload with error “use of closed network connection”
- Flow: Fixed an issue causing disk capacity alter log to not show when an Analysis fails due to disk capacity and added error message
- Flow: Fixed an issue preventing cross-tenant users from being able to open a shared CWL pipeline
- Base: Improved target Table selection for schedulers to be limited to your own Tables
- Bench: Fixed an issue causing Workspaces to hang in the Starting or Stopping statuses
- Cohorts: Now handles large VCFs/gVCFs correctly by splitting them into smaller files for subsequent annotation by Nirvana
- API: Fixed returning list of unlinked data in a sample that was linked before in GET/api/projects/{projectId}/data
- API: Fixed error for getSampleCreationBatch when using status filter
- Updated guidance for selecting Project Location and availability of Storage Bundles when configuring a new Project
- Included information about receiving and using Entitled Bundles
- Added guidance that a sample must be complete to be linked to a Bundle
- Updated details about paging in the Event Log
- Refined and expanded guidance for sequencer integration and analysis auto-launch, including adding a new tutorial for NovaSeq 6000 and expanding the sequencer reference table
- Added tutorial for using scatter/gather for Nextflow pipelines
- Refined Pipelines documentation around clone function
- Added information that Base Tables must not contain reserved Snowflake terms in their definition
- Added description for Copy Number Variants and Structural Variants tables now viewable in ICA Base for Cohorts
- Added description for hg19 support as an alternative during ingestion for Cohorts
- Updated CLI command index
- Updated CLI binary and checksum files
- Updated information about data archive pricing
- Updated software registration guide
- General updates and improvements
- Added a limit to Event Log and Audit UI screens to show 10,000 records
- API: Parent output folder can be specified in URN format when launching a Workflow session via the API
- Flow: Reduced Analysis delays when system is experiencing heavy load
- Flow: Improved formatting of Pipeline error text shown in Analysis Details view
- Flow: Users can now start Analyses from the Analysis Overview screen
- Flow: Superfluous “Namespace check-0” step was removed to reduce Analysis failures
- Flow: Number of input files for an Analysis is limited to 50,000
- Flow: Auto launched Workflow sessions will fail if duplicate sample IDs are detected under Analysis Settings in the Sample Sheet
- Base: Activity screen now contains the size of the query
Cohorts
- Detect and Lift Genome Build: Cohorts documentation provides set-up instructions to convert hg19/GRCh37 VCFs to GRCh38 before import into Cohorts.
- Attribute Queries: Improved the user experience choosing a range of values for numerical attributes when defining a cohort
- Export Cohort to ICA Project Data: Improved the user experience exporting list of subjects that match cohort definition criteria to their ICA project for further analysis
- Ingest Structural Variants into database
- The Cohorts ingestion pipeline supports structural variant VCFs and will deposit all such variants into an ICA Base table if Base is enabled for the given project
- Structural variants can be ingested and viewed in base tables
- Needle Plot Enhancements
- Users can input a numerical value in the Needle Plot legend to display variants with a specific gnomAD frequency percentage or sample count
- The needle plot combines variants that are observed among subjects in the current project as well as shared and public projects into a single needle, using an additional shape to indicate these occurrences
- Needle Plot legend color changes for Variant severity; pathogenic color coding is the same as the color coding in the visualization; differentiating hue between proteins and variants; and other color coding changes.
- Needle plot tool tips that display additional information on variants and mutations are now larger and modal
- The needle plot now allows to filter by gnomAD allele frequency and sample count in the selected cohort. Variants include links to view a list of all subjects carrying that variant and export that list.
- Remove Samples Individually from Cohorts
- Exclude individual subjects from a cohort and save the refined list
- The subjects view allows users to exclude individual subjects from subsequent analyses and plots and save these changes
- Subject exclusions are reset when editing a cohort
- Subject Selection in Analysis Visualization: Users can follow the link for subject counts in the needle plot to view a list of subjects carrying the selected variant or mutation.
- UI/UX: Start and End time points are available as a date or age with a condition attribute in the subject data summary screen.
- Improved resilience against misconfiguration of the team page when there is an issue with Workgroup availability
- Improved handling of multi-file upload when system is experiencing heavy loads
- Fixed an issue to allow upload of zero-byte files via the UI
- Fixed issue where other Bundles would not be visible after editing and saving your Bundle
- Removed ‘IGV (beta)’ button from ‘View’ drop down when selecting Project Data in UI
- API: Improved error handling for API endpoint: POST /api/projects/{projectId}/analysisCreationBatch
- API: Improved performance of API endpoint: getbestmatchingfornextflow
- Flow: Fixed an issue causing Analysis output mapping to incorrectly use source path as target path
- Flow: Fixed an issue where the UI may display incorrect or invalid parameters for DRAGEN workflows which do not accurately show the true parameters passed. Settings can be confirmed by looking at the DRAGEN analysis log files.
- Base: “Allow jagged rows” setting in the Scheduler has been replaced with “Ignore unknown values” to handle files containing records with more fields than there are Table columns
- Base: Improved Base Activity view loading time
- Base: Fixed an error message when using the API to load data into a Base Table that has been deleted
- Bench: Fixed an issue resulting in incorrect Bench compute pricing calculations
- Bench: Fixed an issue preventing building Docker images from Workspaces in UK, Australia, and India regions
- Bench: Fixed an issue where /tmp path is not writeable in a Workspace
Cohorts:
- Fixed issue where the bubble plot sometimes failed to display results even though the corresponding scatter plot showed data correctly.
- The order of messages and warnings for ingestion jobs was not consistent between the UI and an error report sent out via e-mail.
- The UI now displays any open cohort view tabs using shortened (“…”) names where appropriate
- Issue fixed where ingestions with multiple errors caused halting to the ingestion queue.
- The needle plot sometimes showed only one source for a given variant as opposed to all projects in which the variant had been observed in.
- Issue fixed with unhandled genotype index format in annotation file to base database table conversion
- Status updates via e-mail sometimes contained individual error messages or warnings without a text.
- Fixed issue where items show in needle plot with incorrect numbering on the y-axis.
- Fixed performance issue with subject count.
- Widget bar-chart counts are intermittently cut off over four digits.
- Fixed slowness when switching between tabs in query builder
- Refined and expanded reference documents for sequencer integration and analysis autolaunch, including new pages for sequencer reference and troubleshooting.
- Added tutorial for accessing Base databases and tables via snowSQL.
- Added tutorial for setting up a notification service and Lambda function in AWS for chaining pipelines.
- Included guidance for setting override behavior for file output of advanced output mappings.
- Added instructions to remove specific subjects from a cohort.
- Documented cohort needle plot enhancements: List of subjects and specific frequency data entry.
- Updated cohorts ingestion instruction including metadata documents, GWAS ingestion, OMOP ingestion, Structural Variant ingestion, and hg19 lift-over information.
- Added reference documentation for using Cohorts attributes metadata template.
- Added instruction for cohorts attribute query enhancement on numeric data.
- Described selection of entitlements when creating a Project.
- Described options for starting analyses from Analyses page, from Data page, and from a pipeline.
- Updated FAQ with guidance on analysis run times.
- Updated documentation about network settings for pipelines.
- Described limitation on amount of input files for analyses.
- Explained how to unlink a folder from a project.
- Updated explanation for adding Terms of Use to a Bundle.
- Included details for the Event Log.
- Refined and expanded description of Base Activity screen.
- Expanded description of Bench Activity screen.
- Clarified details about S3 bucket behavior and Bring Your Own Storage bucket (BYOB) compatibility.
- Provided guidance about deleting old versions of Bring Your Own Storage buckets.
- Explained multi-factor authentication compatibility with Bench and provide workaround suggestion.
- Updated CLI command index.
- Fixed a bug in the CLI release history files and updated integrity check instructions.
- General updates and improvements.
- Fixed issue with BaseSpace Free Trial and Professional users storing data in ICA
- Fixed an issue resulting in analysis failures caused by a Kubernetes 404 timeout error
- API: POST /api/projects/{projectId}/data:createDownloadUrls now supports a list of paths (in addition to a list of IDs)
- Each tenant supports a maximum of 30,000 Projects
- .MAF files are now recognized as .TSV files instead of UNKNOWN
- Added VCF.IDX as a recognized file format
- General scalability optimizations and performance improvements
- After creating a new Bundle the user will be taken to the Bundle Overview page
- Fixed an issue preventing the ‘Owning Project’ column from being used outside of Project
- Fixed an issue which prevented changing the format of a file back to UNKNOWN
- Fixed an issue causing inaccurate upload progress to be displayed for UI uploads. The Service Connector or CLI are recommended for large file uploads.
- Fixed an issue showing an incorrect status for data linking batch jobs when data is deleted during the linking job
- Fixed an issue allowing the region of a Project to be changed. Changing the region of a resource is not supported.
- Strengthened data separation and improved resilience against cross-Project metadata contamination
- Service Connector: Fixed an issue allowing download of a Service Connector when no operating system is set
- Service Connector: Cleaned up information available on Service Connectors by removing empty address information fields
- API: Fixed date formatting for GET /api/eventLog (yyyy-MM-dd’T’HH:mm:ss.SSS’Z’)
- API: Fixed an issue where the GET users API was not case sensitive on email address
- API: Fixed an issue causing the metadata model to be returned twice in PSOT /api/projects/{projectId}/samples:search
- API: Fixed the listProjects API 500 response when using the pageoffset query parameter
- API: The searchProjectSamples API returns Sample metadata for Samples shared via a Bundle
- API: Fixed an issue causing createProjectDataDownloadUrls API 400 and 502 errors when server is under load
- Flow: Fixed analysis failures caused by kubernetes 404 timeout error
- Flow: Fixed an issue where Workflwos would prematurely report completion of an Analysis
- Flow: Improved Pipeline retry logic to reduce startup delays
- Flow: Fixed an issue where Nextflow pipelines were created with empty files (Nextflow config is allowed to be empty)
- Flow: Removed the 1,000 input file limitation when starting an Analysis
- Flow: Improved the performance of status update messages for pipelines with many parallel steps
- Flow: Fixed an issue with overlapping fields on the Analysis Details screen
- Flow: Deactivated the Abort button for Succeeded analyses
- Base: Fixed an issue where Pipeline metadata was not captured in the metadata Table generated by the metadata schedule
- Base: Error logging and notification enhancements