Data
The Data inventory provides access to the files and folders stored in the project or linked to the project. Here, you can perform searches and data management operations such as moving, copying, deleting and (un)archiving.
Recommended Practices
File/Folder Naming
Platform Core supports UTF-8 characters in file and folder names for data. Please follow the guidelines detailed below. (For more information about recommended approaches to file naming that can be applicable across platforms, please refer to the AWS S3 documentation.)
Data Formats
See the list of supported Data Formats
Data Privacy
When adding data to Platform Core, prioritize data privacy. Whether you're using storage configurations like AWS S3 or performing Platform Core data uploads, careful management of data access needs to be considered. When setting up cloud storage, confirm that configuration settings prevent unauthorized access. Always verify that uploads are free of unintended data to avoid privacy breaches. For more detailed information, refer to the Platform Core Security and Compliance section.
Data Integrity
See Data Integrity
Viewing Data
On the Projects > your_project > Data page, you can view file information and preview files.
Folder view (structured) vs flat view (list)
You can switch between folder view and flat view with the icons at the left.
Folder view shows the navigation structure and only the files and folders in the current folder. Searches in folder view will be performed on the current folder and all subfolders of the current folder.
Flat view shows a list of all files and folders within the current project. When you perform searches in flat view, all data of your project will be considered.
Tree view shows the navigation structure and only the files and folders in the current folder. Searches in tree view will be performed on the current folder and all subfolders of the current folder.
List view shows all files and folders within the current project. When you perform searches in list view, all data of your project will be considered.
Files
To view file details click on the filename to see the file details.
Run input tags identifies the last 100 pipelines which used this file as input.
Run output tags identifies the pipeline which created the file.
Connector tags show if the file was added via browser upload or connector.
Clicking on a folder will open the folder itself, to see the folder details, use the folder details link at the top right of the screen.
To view file contents, select the checkbox at the beginning of the line and then select View from the top menu. Alternatively, you can first click on the filename to see the details and then click the view tab to preview the file.
If your data is the result of an analysis, you can find the analysis which created it at Projects > your_project > Data > your_data > view > Data details tab > Source analysis. Clicking the link here will open the analysis.
Filtering
To add filters, select the funnel/filter symbol at the top right, next to the search field.
Filters are reset when you exit the current screen.
Sorting
To sort data, select the three vertical dots in the column header on which you want to sort and chose ascending or descending.
Sorting is retained when you exit the current screen.
Displaying Columns
To change which columns are displayed, select the three columns symbol and select which columns should be shown.
In externally-managed projects, you can see which files are externally controlled and which are ICA-managed by means of the “managed by” column.
In flat view, you can jump to the folder in which files are located with the "Path" column.
The displayed columns are retained when you exit the current screen.
To see the ongoing actions (copying and moving) on data in the data overview (Projects > your_project > Data), add the ongoing actions column from the column list if it is not present yet. You can also consult the data detail view for ongoing actions by clicking on the data in the overview. When clicking on an ongoing action itself, the data job details of the most recent created data job are shown.
Folders
If you open a folder by clicking it, you can see the folder details link at the top right. This will open the details screen where you can consult the folder size and number of files in that folder, the owning project, ongoing actions and folder id. You can also download the folder and all contents here with the download button.
To help navigate between folders in flat view, you can use the "path' column in the data view which will open the folder containing the selected file in tree view. If you want to go further up the folder path, you can use the folder structure above the file view. If the path is not visible, you can add it with the three-columns symbol next to the filter symbol.
Searching for Data
To quickly find data, use the search dialog at the top right. Search is performed with automatic wildcards before and after the search text. You can use * as additional wildcard, for example b*n will match bunny as it is interpreted as *b*n*.
You can search on the file name, the path (/folder/subfolder) and tags.
Secondary Data
When Secondary Data is added to a data record, those secondary data records are mounted in the same parent folder path as the primary data file when the primary data file is provided as an input to a pipeline. Secondary data is intended to work with the CWL secondaryFiles feature. This is commonly used with genomic data such as BAM files with companion BAM index files.
Hyperlinking to Data
You can create hyperlinks to data to quickly share it with the following syntax:
ServerURL
See browser address bar.
projectID
At YourProject > Details > URN > urn:ilmn:ica:project:ProjectID#MyProject
FolderID
At YourProject > Data > folder > folder details > ID
AnalysisID
At YourProject > Flow > Analyses > YourAnalysis > ID
Exporting the Data List
You can export the list of data which you see in the overview as a CSV, JSON, or excel file.
Select one or more files to export at Projects > your_project > Data.
Select Export at the bottom of the screen.
Choose between the following export options:
To export only the list of selected files, select the Selected rows as the Rows to export option. To export the list of all files on the page, select Current page.
To export only the columns which are currently shown in your view, select Visible columns as the Columns to export option, otherwise, choose All columns.
Select the export format. (CSV/JSON/Excel)
Data Management
Downloading Data
Single files can be downloaded directly from within the UI.
Select the checkbox next to the file which you want to download, followed by Download > Browser Download > Download.
You can also download files from their details screen. Click on the file name and select Download at the bottom of the screen. Depending on the size of your file, it may take some time to load the file contents.
Schedule for Download
You can trigger an asynchronous download via service connector using the Schedule for Download button with one or more files selected.
Select a file or files to download.
Select Download > Schedule download (for files or folders). This will display a list of all available connectors.
Select a connector and optionally, enter your email address if you want to be notified of download completion, and then select Download.
You can view the progress of the download or abort the scheduled download on the Activity page for the project.
Uploading Data
Uploading data to the platform makes it available to analysis workflows and tools.
UI Upload
To upload data manually via the drag-and-drop interface in the platform UI, go to Projects > your_project > Data and either
Drag a file from your system into the Choose a file or drag it here box.
Select the Choose a file or drag it here box, and then choose a file. Select Open to upload the file.
Your files are added to the Data page with status partial during upload and become available when upload completes.
Upload Data via CLI
For instructions on uploading/downloading data via CLI, see CLI Data Transfer.
Copying Data
You can copy data from your project to a different folder within the same project or you can copy data from another project to your current project, provided you have the necessary access rights.
You can copy data from a subfolder to a higher-level folder to move data up one or more levels (folder/destination/source). You can not copy data from the source folder onto itself or onto a subfolder of the source folder as this would result in a loop.
Required Rights
The person copying the data must have the following rights:
Within a project
Contributor rights
Upload and Download rights
Contributor rights
Upload and Download rights
Between different projects
Download rights
Viewer rights
Upload rights
Contributor rights
Restrictions
The following restrictions apply when copying data:
Within a project
No linked data
No partial data
No archived data
No Linked data
Between different projects
Data sharing enabled
No partial data
No archived data
Within the same region
No linked data
Within the same region
Data in the "Partial" or "Archived" state will be skipped during a copy job.
Copying Data
Go to the destination project for your data copy and proceed to Projects > your_project > Data > Manage > Copy From.
Optionally, use the filters or search with the search box for the desired data.
Select the data (individual files or folders with data) you want to copy.
Select any meta data which you want to keep with the copied data (user tags, technical system tags or instrument information).
Select which action to take if the data already exists (overwrite existing data, don't copy or keep both the original and the new copy by appending a version number to the copied data).
Select Copy to copy the data to your project. You can see the progress in Projects > your_project > Activity > Batch Jobs and if your browser permits it, a pop-up message will be displayed when the copy process completes.
Replace
Overwrites the existing data. Folders will copy their data in an existing folder with existing files. Existing files will be replaced when a file with the same name is copied and new files will be added. The remaining files in the target folder will remain unchanged.
Don't copy
The original files are kept. If you selected a folder, files that do not yet exist in the destination folder are added to it. Files that already exist at the destination are not copied over and the originals are kept.
Keep both
Files have a number appended to them if they already exist. If you copy folders, the folders are merged, with new files added to the destination folder and original files kept. New files with the same name get copied over into the folder with a number appended.
Copy Status
INITIALIZED
WAITING_FOR_RESOURCES
RUNNING
STOPPED - When choosing to stop the batch job.
SUCCEEDED - All files and folders are copied.
PARTIALLY_SUCCEEDED - Some files and folders could be copied, but not all. Partially succeeded will typically occur when files were being modified or unavailable while the copy process was running.
FAILED - None of the files and folders could be copied.
To see the ongoing actions on data in the data overview (Projects > your_project > Data), you can add the ongoing actions column from the column list with the three column symbol at the top right, next to the filter funnel. You can also consult the data detail view for ongoing actions by clicking on the data in the overview.
Moving Data
You can move data within a project or between different projects to which you have access. If your browser allows notifications, a pop-up will appear when the move is completed.
Move From is used when you are in the destination location.
Move To is used when you are in the source location.
Before moving the data, pre-checks are performed to verify that the data can be moved and no currently running operations are being performed on the folder. Conflicting jobs and missing permissions will be reported.
Once the move has started, no other operation must be performed on the data being moved to avoid potential data loss or duplication. When modifying data at the source or destination during a move process, incomplete data transfers may occur with duplicate folders and files with different identifiers.. You can manually transfer any remaining data and delete duplicate files and folders afterward.
Move jobs will fail if any data being moved is in the Partial or Archived state.
Required Rights
There are a number of rights and restrictions related to data move as this will delete the data in the source location.
Within a project
Contributor rights
Contributor rights
Between different projects
Download rights
Contributor rights
Upload rights
Viewer rights
Restrictions
Within a project
No linked data
No partial data
No archived data
No Linked data
Between different projects
Data sharing enabled
Data owned by user's tenant
No linked data
No partial data
No archived data
No externally managed projects
Within the same region
No linked data
Within same region
Moving Data Constraints
1000 Maximum Items: Up to 1000 items per move. Items include files and folders. Folders with subfolders and subfiles still count as one item.
Naming Conflicts: Cannot move to a destination with existing files/folders of the same name.
Linked Data Restrictions: Cannot move linked data move data to linked data.
Self Move: Folders cannot be moved to themselves.
In-Transit Data: Cannot move data that is being moved.
Region Restrictions: No cross-region moves allowed.
Project Constraints: No moves from externally-managed projects or externally-managed data.
Status Requirement: Data must be in status available.
Ownership: Data must be owned by the user's tenant for cross-project moves.
Destination Default: If no target folder is selected, data moves to the root folder of the target project.
Move Data From
Move Data From is used when you are in the destination location.
Navigate to Projects > your_project > Data > your_destination_location > Manage > Move From.
Select the files and folders which you want to move.
Select the Move button.
Move Data To
Move Data To is used when you are in the source location. You will need to select the data you want to move from to current location and the destination to move it to.
Navigate to Projects > your_project > Data > your_source_location.
Select the files and folders which you want to move.
Select to Projects > your_project > Data > your_source_location > Manage > Move To.
Select your target project and location. You can create a new folder to move data to by filling in the "New folder name (optional)" field. This does NOT rename an existing folder. To rename a folder, you will need to create a new folder with the desired name, move the contents from the original folder into the new one, and then delete the original folder.
Select the Move button.
Move Status
INITIALIZED
WAITING_FOR_RESOURCES
RUNNING
STOPPED - When choosing to stop the batch job.
SUCCEEDED - All files and folders are moved.
PARTIALLY_SUCCEEDED - Some files and folders could be moved, but not all. Partially succeeded will typically occur when files were being modified or unavailable while the move process was running.
FAILED - None of the files and folders could be moved.
To see the ongoing actions on data in the data overview (Projects > your_project > Data), add the ongoing actions column from the column list with the three column symbol at the top right, next to the filter funnel. You can also consult the data detail view for ongoing actions by clicking on the data in the overview.
Deleting, Archiving and Unarchiving
To manually archive or delete files:
Select the checkbox next to the file or files to delete or archive.
Select Manage, and then select one of the following options:
Archive — Move the file or files to long-term storage (event code ICA_DATA_110).
Unarchive — Return the file or files from long-term storage. Unarchiving can take up to 48 hours, regardless of file size. Unarchived files can be used in analysis (event code ICA_DATA_114).
Delete — Remove the file completely (event code ICA_DATA_106).
When attempting concurrent archiving or unarchiving of the same file, a message will inform you to wait for the currently running (un)archiving to finish first.
To archive or delete files programmatically, you can use Platform Core's API endpoints:
GET the file's information.
Modify the dates of the file to be deleted/archived.
PUT the updated information back in Platform Core.
Python Example
The Python snippet below exemplifies the approach: it sets (or updates if set already) the time to be archived for a specific file:
To delete a file at specific timepoint, the key 'willBeDeletedAt' should be added or changed using the API call. If running in the terminal, a successful run will finish with the message ‘200’. In the Platform Core UI, you can check the details of the file to see the updated values for ‘Time To Be Archived’ (willBeArchivedAt) or ‘Time To Be Deleted’ (willBeDeletedAt), as shown in the screenshot.
Linking and Unlinking
Data linking creates a dynamic read-only view to the source data. You can use data linking to get access to data without running the risk of modifying the source material and to share data between projects. Linking ensures changes to the source data are immediately visible and no additional storage is required. You can recognise linked data by the green color and see the owning project as part of the details.
Since this is read-only access, you cannot perform actions such as deleting, adding, moving or (un)archiving on linked data as these actions require write access.
Linking data from another project.
Select Projects > your_project > Data > Manage, and then select Link.
To view data by project, select the funnel symbol, and then select Owning Project. If you know to which project the data is linked to, you can choose to filter on linked projects. If you click on a folder, the folder will open so you can access the files, if you click on a file, the file details will be opened.
Select the checkbox next to the file or files to add.
Select Link.
Your files will be added and visible in the Data page.
Display Owning Project
if you have selected multiple owning projects, you can add the owning project column to see which project owns the data.
At the top of the screen, next to the filer icon, select the three columns.
The Add/remove columns tab will appear.
Choose Owning Project (or Linked Projects)
Owning Project Filter
Linking Folders
If you link a folder instead of individual files, a warning is displayed indicating that, depending on the size of the folder, linking may take considerable time. The linking process will run in the background and the progress can be monitored on the Projects > your_project > activity > Batch Jobs screen. From here you can see more details such as how many files have already been linked, by clicking the batch job.
Unlinking Project Data
To unlink the data, go to the root level of your project and select the linked folder or, if you have linked individual files separately, then you can select those linked files (limited to 100 at a time) and select Manage > Unlink. The progress can be monitored at Projects > your_project > Activity > Batch Jobs.
Last updated
Was this helpful?