Notifications

Notifications (Projects > your_project > Project Settings > Notifications ) are events to which you can subscribe. When they are triggered, they deliver a message to an external target system such as emails, Amazon SQS or SNS systems or HTTP post requests. The following table describes available system events to subscribe to:

Description	Code	Details	Payload
Analysis failure	ICA_EXEC_001	Emitted when an analysis fails	Analysis
Analysis success	ICA_EXEC_002	Emitted when an analysis succeeds	Analysis
Analysis aborted	ICA_EXEC_027	Emitted when an analysis is aborted either by the system or the user	Analysis
Analysis status change	ICA_EXEC_028	Emitted when an state transition on an analysis occurs	Analysis
Base Job failure	ICA_BASE_001	Emitted when a Base job fails	BaseJob
Base Job success	ICA_BASE_002	Emitted when a Base job succeeds	BaseJob
Data transfer success	ICA_DATA_002	Emitted when a data transfer is marked as Succeeded	DataTransfer
Data transfer stalled	ICA_DATA_025	Emitted when data transfer hasn't progressed in the past 2 minutes	DataTransfer
Data <action>	ICA_DATA_100	Subscribing to this serves as a wildcard for all project data status changes and covers those changes that have no separate code. This does not include DataTransfer events or changes that trigger no data status changes such as adding tags to data.	ProjectData
Data linked to project	ICA_DATA_104	Emitted when a file is linked to a project	ProjectData
Data deleted	ICA_DATA_106	Emitted when data is deleted	ProjectData
Data created	ICA_DATA_107	Emitted when data is created	ProjectData
Data uploaded	ICA_DATA_108	Emitted when data is uploaded	ProjectData
Data updated	ICA_DATA_109	Emitted when data is updated	ProjectData
Data archived	ICA_DATA_110	Emitted when data is archived	ProjectData
Job staus changed	ICA_JOB_001	Emitted when a job changes status (INITIALIZED, WAITING_FOR_RESOURCES, RUNNING, STOPPED, SUCCEEDED, PARTIALLY_SUCCEEDED, FAILED)	JobId
Sample completed	ICA_SMP_002	Emitted when a sample is marked as completed	ProjectSample
Sample linked to a project	ICA_SMP_003	Emitted when a sample is linked to a project	ProjectSample
Workflow session start	ICA_WFS_001	Emitted when workflow is started	WorkflowSession
Workflow session failure	ICA_WFS_002	Emitted when workflow fails	WorkflowSession
Workflow session success	ICA_WFS_003	Emitted when workflow succeeds	WorkflowSession

Description

Code

Details

Payload

Analysis failure

ICA_EXEC_001

Emitted when an analysis fails

Analysis

Analysis success

ICA_EXEC_002

Emitted when an analysis succeeds

Analysis

Analysis aborted

ICA_EXEC_027

Emitted when an analysis is aborted either by the system or the user

Analysis

Analysis status change

ICA_EXEC_028

Emitted when an state transition on an analysis occurs

Analysis

Base Job failure

ICA_BASE_001

Emitted when a Base job fails

BaseJob

Base Job success

ICA_BASE_002

Emitted when a Base job succeeds

BaseJob

Data transfer success

ICA_DATA_002

Emitted when a data transfer is marked as Succeeded

DataTransfer

Data transfer stalled

ICA_DATA_025

Emitted when data transfer hasn't progressed in the past 2 minutes

DataTransfer

Data <action>

ICA_DATA_100

Subscribing to this serves as a wildcard for all project data status changes and covers those changes that have no separate code. This does not include DataTransfer events or changes that trigger no data status changes such as adding tags to data.

ProjectData

Data linked to project

ICA_DATA_104

Emitted when a file is linked to a project

ProjectData

Data deleted

ICA_DATA_106

Emitted when data is deleted

ProjectData

Data created

ICA_DATA_107

Emitted when data is created

ProjectData

Data uploaded

ICA_DATA_108

Emitted when data is uploaded

ProjectData

Data updated

ICA_DATA_109

Emitted when data is updated

ProjectData

Data archived

ICA_DATA_110

Emitted when data is archived

ProjectData

Job staus changed

ICA_JOB_001

Emitted when a job changes status (INITIALIZED, WAITING_FOR_RESOURCES, RUNNING, STOPPED, SUCCEEDED, PARTIALLY_SUCCEEDED, FAILED)

JobId

Sample completed

ICA_SMP_002

Emitted when a sample is marked as completed

ProjectSample

Sample linked to a project

ICA_SMP_003

Emitted when a sample is linked to a project

ProjectSample

Workflow session start

ICA_WFS_001

Emitted when workflow is started

WorkflowSession

Workflow session failure

ICA_WFS_002

Emitted when workflow fails

WorkflowSession

Workflow session success

ICA_WFS_003

Emitted when workflow succeeds

WorkflowSession

When you subscribe to overlapping event codes such as ICA_EXEC_002 (analysis success) and ICA_EXEC_028 (analysis status change) you will get both notifications when analysis success occurs.

Delivery Targets

Event notifications can be delivered to the following delivery targets:

Delivery Target	Description	Value
Mail	E-mail delivery	E-mail Address
Sqs	AWS SQS Queue	AWS SQS Queue URL
Sns	AWS SNS Topic	AWS SNS Topic ARN
Http	Webhook (POST request)	URL

Delivery Target

Description

Value

Mail

E-mail delivery

E-mail Address

Sqs

AWS SQS Queue

AWS SQS Queue URL

Sns

AWS SNS Topic

AWS SNS Topic ARN

Http

Webhook (POST request)

URL

Amazon Resource Policy Settings

In order to allow the platform to deliver events to Amazon SQS or SNS delivery targets, a cross-account policy needs to be added to the target Amazon service.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::<platform_aws_account>:root"
         },
         "Action":"<action>",
         "Resource": "<arn>"
      }
   ]
}

Substitute the variables in the example above according to the table below.

Variable Description

Variable	Description
platform_aws_account	The platform AWS account ID: `079623148045`
action	For SNS use `SNS:Publish`. For SQS, use `SQS:SendMessage`
arn	The Amazon Resource Name (ARN) of the target SNS topic or SQS queue

platform_aws_account

The platform AWS account ID: 079623148045

action

For SNS use SNS:Publish. For SQS, use SQS:SendMessage

arn

The Amazon Resource Name (ARN) of the target SNS topic or SQS queue

See examples for setting policies in Amazon SQS and Amazon SNS

To create a subscription to deliver events to an Amazon SNS topic, one can use either GUI or API endpoints.

To create a subscription via GUI select Projects > your_project > Project Settings > Notifications and then select New ICA subscription. Then select an event from the dropdown menu, insert optional filter, select the channel type (SNS), and then insert the ARN from the target SNS topic.

To create a subscription via API use the endpoint /api/notificationChannel to create a channel and then /api/projects/{projectId}/notificationSubscriptions to create a notification subscription.

Amazon SQS Queue

To create a subscription to deliver events to an Amazon SQS queue, one can use either GUI or API endpoints.

To create a subscription via GUI, select Notifications menu within a project and then select New ICA subscription. Next, select an event from the dropdown menu, insert optional filter, select the channel type (SQS) and API version, and then insert the ARN from the target SQS topic. Not all payload versions are applicable for all events and targets, so the system will filter the options out for you.

To create a subscription via API use the endpoint /api/notificationChannel to create a channel and then /api/projects/{projectId}/notificationSubscriptions to create a notification subscription.

Messages delivered to AWS SQS contain the following event body attributes:

Attribute	Description
correlationId	GUID used to identify the event
timestamp	Date when the event was sent
eventCode	Event code of the event
description	Description of the event
payload	Event payload

Attribute

Description

correlationId

GUID used to identify the event

timestamp

Date when the event was sent

eventCode

Event code of the event

description

Description of the event

payload

Event payload

The following example is a Data Updated event payload sent to an AWS SQS delivery target (condensed for readability):

{
    "correlationId": "2471d3e2-f3b9-434c-ae83-c7c7d3dcb4e0",
    "timestamp": "2022-10-06T07:51:09.128Z",
    "eventCode": "ICA_DATA_100",
    "description": "Data updates",
    "payload": {
        "id": "fil.8f6f9511d70e4036c60908daa70ea21c",
        ...
    }
}

Filtering

Notification subscriptions will trigger for all events matching the configured event type. A filter may be configured on a subscription to limit the matching strategy to only those event payloads which match the filter.

The filter expressions leverage the JsonPath library for describing the matching pattern to be applied to event payloads. The filter must be in the format [?(<expression>)].

Examples

The Analysis Success event delivers a JSON event payload matching the Analysis data model (as output from the API to retrieve a project analysis).

{
  "id": "0c2ed19d-9452-4258-809b-0d676EXAMPLE",
  "timeCreated": "2021-09-20T12:23:18Z",
  "timeModified": "2021-09-20T12:43:02Z",
  "ownerId": "15d51d71-b8a1-4b38-9e3d-74cdfEXAMPLE",
  "tenantId": "022c9367-8fde-48fe-b129-741a4EXAMPLE",
  "reference": "210920-1-CopyToolDev-9d78096d-35f4-47c9-b9b6-e0cbcEXAMPLE",
  "userReference": "210920-1",
  "pipeline": {
    "id": "20261676-59ac-4ea0-97bd-8a684EXAMPLE",
    "timeCreated": "2021-08-25T01:49:41Z",
    "timeModified": "2021-08-25T01:49:41Z",
    "ownerId": "15d51d71-b8a1-4b38-9e3d-74cdfEXAMPLE",
    "tenantId": "022c9367-8fde-48fe-b129-741a4EXAMPLE",
    "code": "CopyToolDev",
    "description": "CopyToolDev",
    "language": "CWL",
    "pipelineTags": {
      "technicalTags": ["Demo"]
    }
  },
  "status": "SUCCEEDED",
  "startDate": "2021-09-20T12:23:21Z",
  "endDate": "2021-09-20T12:43:00Z",
  "summary": "",
  "finishedSteps": 0,
  "totalSteps": 1,
  "tags": {
    "technicalTags": [],
    "userTags": [],
    "referenceTags": []
  }
}

The below examples demonstrate various filters operating on the above event payload:

Filter on a pipeline, with a code that starts with ‘Copy’. You’ll need a regex expression for this:
[?($.pipeline.code =~ /Copy.*/)]
Filter on status (note that the Analysis success event is only emitted when the analysis is successful):
[?($.status == 'SUCCEEDED')]
Both payload Version V3 and V4 guarantee the presence of the final state (SUCCEEDED, FAILED, FAILED_FINAL, ABORTED) but depending on the flow (so not every intermediate state is guaranteed):
- V3 can have REQUESTED - IN_PROGRESS - SUCCEEDED
- V4 can have the status REQUESTED - QUEUED - INITIALIZING - PREPARING_INPUTS - IN_PROGRESS - GENERATING_OUTPUTS - SUCCEEDED
Filter on pipeline, having a technical tag “Demo":
[?('Demo' in $.pipeline.pipelineTags.technicalTags)]
Combination of multiple expressions using &&. It's best practice to surround each individual expression with parentheses:
[?(($.pipeline.code =~ /Copy.*/) && $.status == 'SUCCEEDED')]

Examples for other events

Filtering ICA_DATA_104 on owning project name. The top level keys on which you can filter are under the payload key, so payload is not included in this filter expression.
[?($.details.owningProjectName == 'my_project_name')]

Custom Events

Custom events enable triggering notification subscriptions using event types beyond the system-defined event types. When creating a custom subscription, a custom event code may be specified to use within the project. Events may then be sent to the specified event code using a POST API with the request body specifying the event payload.

Currently custom events can only be defined using the API. In order to create a custom event for your project please follow the steps below:

Create a new custom event POST {ICA_URL}/ica/rest/api/projects/{projectId}/customEvents a. Your custom event code must be 1-20 characters long, e.g. 'ICA_CUSTOM_123'. b. That event code will be used to reference that custom event type.
Create a new notification channel POST {ICA_URL}/ica/rest/api/notificationChannels a. If there is already a notification channel created with the desired configuration within the same project, it is also possible to get the existing channel ID using the call GET {ICA_URL}/ica/rest/api/notificationChannels.
Create a notification subscription POST {ICA_URL}/ica/rest/api/projects/{projectId}/customNotificationSubscriptions. a. Use the event code created in step 1. b. Use the channel ID from step 2.

Once the steps above have been completed successfully, the call from the first step POST {ICA_URL}/ica/rest/api/projects/{projectId}/customEvents could be reused with the same event code to continue sending events through the same channel and subscription.

Following is a sample Python function used inside an ICA pipeline to post custom events for each failed metric:

def post_custom_event(metric_name: str, metric_value: str, threshold: str, sample_name: str):
    api_url = f"{ICA_HOST}/api/projects/{PROJECT_ID}/customEvents"
    headers = {
        "Content-Type": "application/vnd.illumina.v3+json",
        "accept": "application/vnd.illumina.v3+json",
        "X-API-Key": f"{ICA_API_KEY}"
    }
    content = {\"code\": \"ICA_CUSTOM_123\", \"content\": { \"metric_name\": metric_name, \"metric_value\": metric_value,\"threshold\": threshold, \"sample_name\": sample_name}}
    json_data = json.dumps(content)
    response = requests.post(api_url, data=json_data, headers=headers)

    if response.status_code != 204:
        print(f"[EVENT-ERROR] Could not post metric failure event for the metric {metric_name} (sample {sample_name}).")

PreviousProject Connector NextInstallation

Last updated 28 days ago

Delivery Targets

Amazon Resource Policy Settings

Amazon SNS Topic

Amazon SQS Queue

Filtering

Examples

Custom Events