Notifications
Notifications are used to subscribe to events in the platform and trigger the delivery of a message to an external delivery target. The following table describes availabe system events to subscribe to:
Description | Code | Details | Payload |
|---|---|---|---|
Analysis failure | ICA_EXEC_001 | Emitted when an analysis fails | Analysis |
Analysis success | ICA_EXEC_002 | Emitted when an analysis succeeds | Analysis |
Base Job failure | ICA_BASE_001 | Emitted when a Base job fails | BaseJob |
Base Job success | ICA_BASE_002 | Emitted when a Base job succeeeds | BaseJob |
Data transfer success | ICA_DATA_002 | Emmitted when a data transfer is marked as Succeeded | DataTransfer |
Data transfer stalled | ICA_DATA_025 | Emitted when data transfer hasn't progressed in the past 2 minutes | DataTransfer |
Data updates | ICA_DATA_100 | Emitted when data transfer status changes | ProjectData |
Data linked to project | ICA_DATA_104 | Emitted when a file is linked to a project | ProjectData |
Data deleted | ICA_DATA_106 | Emitted when data is deleted | ProjectData |
Sample completed | ICA_SMP_002 | Emitted when a sample is marked as completed | ProjectSample |
Sample linked to a project | ICA_SMP_003 | Emitted when a sample is linked to a project | ProjectSample |
Event notifications can be delivered to the following delivery targets:
Delivery Target | Description | Value |
|---|---|---|
Mail | E-mail delivery | E-mail Address |
Sqs | AWS SQS Queue | AWS SQS Queue URL |
Sns | AWS SNS Topic | AWS SNS Topic ARN |
Http | Webhook (POST request) | URL |
In order to allow the platform to deliver events to Amazon SQS or SNS delivery targets, a cross-account policy needs to be added to the target Amazon service.
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"AWS":"arn:aws:iam::<platform_aws_account>:root"
},
"Action":"<action>",
"Resource": "<arn>"
}
]
}
Substitute the variables in the example above according to the table below.
Variable | Description |
|---|---|
platform_aws_account | The platform AWS account ID: 079623148045 |
action | For SNS use SNS:Publish. For SQS, use SQS:SendMessage |
arn | The Amazon Resource Name (ARN) of the target SNS topic or SQS queue |
To create a subscription to deliver events to an Amazon SNS topic, use the
--aws-sns-topic flag from the CLI.$ ica subscriptions create --name aws-sns-example --type tes.runs --actions updated --aws-sns-topic arn:aws:sns:us-east-1:<aws-account-id>:<sns-topic>
actions.0 updated
deliveryTarget.awsSnsTopic.topicArn arn:aws:sns:us-east-1:<aws-account-id>:<sns-topic>
id sub.EXAMPLE
matchIdentities.0 cid:EXAMPLE
name aws-sns-example
type tes.runs
...
To create a subscription to deliver events to an Amazon SQS queue, use the
--aws-sqs-queue flag from the CLI.$ ica subscriptions create --name aws-sqs-example --type tes.runs --actions updated --aws-sqs-queue https://sqs.us-east-1.amazonaws.com/<account>/EXAMPLE
actions.0 updated
deliveryTarget.awsSqsQueue.queueUrl https://sqs.us-east-1.amazonaws.com/<account>/EXAMPLE
id sub.EXAMPLE
matchIdentities.0 cid:EXAMPLE
name aws-sqs-example
type tes.runs
...
Messages delivered to AWS SQS contain the following event body attributes:
Attribute | Description |
|---|---|
correlationId | GUID used to identify the event |
timestamp | Date when the event was sent |
eventCode | Event code of the event |
description | Description of the event |
payload | Event payload |
The following example is a Data Updated event payload sent to an AWS SQS delivery target (condensed for readability):
{
"correlationId": "2471d3e2-f3b9-434c-ae83-c7c7d3dcb4e0",
"timestamp": "2022-10-06T07:51:09.128Z",
"eventCode": "ICA_DATA_100",
"description": "Data updates",
"payload": {
"id": "fil.8f6f9511d70e4036c60908daa70ea21c",
...
}
}
Notification subscriptions will trigger for all events matching the configured event type. A filter may be configured on a subscription to limit the matching strategy to only those event payloads which match the filter.
The filter expressions leverage the JsonPath library for describing the matching pattern to be applied to event payloads. The filter must be in the format
[?(<expression>)].The
Analysis Success event delivers a JSON event payload matching the Analysis data model (as output from the API to retrieve a project analysis).{
"id": "0c2ed19d-9452-4258-809b-0d676EXAMPLE",
"timeCreated": "2021-09-20T12:23:18Z",
"timeModified": "2021-09-20T12:43:02Z",
"ownerId": "15d51d71-b8a1-4b38-9e3d-74cdfEXAMPLE",
"tenantId": "022c9367-8fde-48fe-b129-741a4EXAMPLE",
"reference": "210920-1-CopyToolDev-9d78096d-35f4-47c9-b9b6-e0cbcEXAMPLE",
"userReference": "210920-1",
"pipeline": {
"id": "20261676-59ac-4ea0-97bd-8a684EXAMPLE",
"timeCreated": "2021-08-25T01:49:41Z",
"timeModified": "2021-08-25T01:49:41Z",
"ownerId": "15d51d71-b8a1-4b38-9e3d-74cdfEXAMPLE",
"tenantId": "022c9367-8fde-48fe-b129-741a4EXAMPLE",
"code": "CopyToolDev",
"description": "CopyToolDev",
"language": "CWL",
"pipelineTags": {
"technicalTags": ["Demo"]
}
},
"status": "SUCCEEDED",
"startDate": "2021-09-20T12:23:21Z",
"endDate": "2021-09-20T12:43:00Z",
"summary": "",
"finishedSteps": 0,
"totalSteps": 1,
"tags": {
"technicalTags": [],
"userTags": [],
"referenceTags": []
}
}
The below examples demonstrate various filters operating on the above event payload:
- Filter on a pipeline, with a code that starts with ‘Copy’. You’ll need a regex expression for this:
[?($.pipeline.code =~ /Copy.*/)] - Filter on status (note that the
Analysis successevent is only emitted when the analysis is successful):[?($.status == 'SUCCEEDED')] - Filter on pipeline, having a technical tag “Demo":
[?('Demo' in $.pipeline.pipelineTags.technicalTags)] - Combination of multiple expressions using
&&. It's best practice to surround each individual expression with paranthees:[?(($.pipeline.code =~ /Copy.*/) && $.status == 'SUCCEEDED')]
Custom events enable triggering notification subscriptions using event types beyond the system-defined event types. When creating a custom subscription, a custom event code may be specified to use within the project. Events may then be sent to the specified event code using a POST API with the request body specifying the event payload.
Currently custom events can only be defined using the API. In order to create a custom event for your project please follow the steps below:
- 1.Create a new custom event
POST {ICA_URL}/ica/rest/api/projects/{projectId}/customEventsa. Your custom event code must be 1-20 characters long, e.g. 'ICA_CUSTOM_123'. b. That event code will be used to reference that custom event type. - 2.Create a new notification channel
POST {ICA_URL}/ica/rest/api/notificationChannelsa. If there is already a notification channel created with the desired configuration within the same project, it is also possible to get the existing channel ID using the callGET {ICA_URL}/ica/rest/api/notificationChannels. - 3.Create a notification subscription
POST {ICA_URL}/ica/rest/api/projects/{projectId}/customNotificationSubscriptions. a. Use the event code created in step 1. b. Use the channel ID from step 2.
Once the steps above have been completed successfully, the call from the first step
POST {ICA_URL}/ica/rest/api/projects/{projectId}/customEvents could be reused with the same event code to continue sending events through the same channel and subscription.Following is a sample Python function used inside an ICA pipeline to post custom events for each failed metric:
def post_custom_event(metric_name: str, metric_value: str, threshold: str, sample_name: str):
api_url = f"{ICA_HOST}/api/projects/{PROJECT_ID}/customEvents"
headers = {
"Content-Type": "application/vnd.illumina.v3+json",
"accept": "application/vnd.illumina.v3+json",
"X-API-Key": f"{ICA_API_KEY}"
}
content = {\"code\": \"ICA_CUSTOM_123\", \"content\": { \"metric_name\": metric_name, \"metric_value\": metric_value,\"threshold\": threshold, \"sample_name\": sample_name}}
json_data = json.dumps(content)
response = requests.post(api_url, data=json_data, headers=headers)
if response.status_code != 204:
print(f"[EVENT-ERROR] Could not post metric failure event for the metric {metric_name} (sample {sample_name}).")
Last modified 20d ago