Documentation homepage Last update: Sep 22, 2021

Notification System Overview

A Notification Template is an instance of a notification type (Email, Slack, Webhook, etc.) with a name, description, and a defined configuration. A few examples include:

At a high level, the typical notification task flow is:

Templated notification messages

When creating a notification template, the user can optionally provide their own custom messages for each notification event (start, success, error). If a message is not provided, the default message generated by AWX will be used.

The notification message can include templated fields written using Jinja templates. The templates may reference a set of white-listed fields found in the associated job's serialization.

HTTP POST /api/v2/notification_templates/

{
    "name": "E-mail notification",
    "description": "Custom e-mail notification",
    "organization": 1,
    "notification_type": "email",
    "notification_configuration": {
      ..
    },
    "messages": {
        "started": {
            "message": "{{ job.name }} completed successfully in {{ job.elapsed }} seconds using instance group {{ job.summary_fields.instance_group.name }}",
            "body": null        # default body will be used
        },
        "success": {
            "message": null,    # default message will be used
            "body": null        # default body will be used
        },
        "error": {
            "message": "{{ job.name }} was unsuccessful ({{ job.status }})",
            "body": "{{ job.job_explanation }}"
        }
    }
}

Notification Hierarchy

Notification templates assigned at certain levels will inherit notifications defined on parent objects as such:

Workflow

When a job starts, succeeds or fails, the running, success or error handler, respectively, will pull a list of relevant notifications using the procedure defined above. It then creates a Notification Object for each one containing relevant details about the job and then sends it to the destination (email addresses, Slack channel(s), SMS numbers, etc.). These Notification objects are available as related resources on job types (Jobs, Inventory Updates, Project Updates), and also at /api/v2/notifications. You may also see what notifications have been sent by examining its related resources.

Notifications can succeed or fail but that will not cause its associated job to succeed or fail. The status of the notification can be viewed at its detail endpoint: /api/v2/notifications/<n>

Testing Notifications Before Using Them

Once a Notification Template has been created, its configuration can be tested by utilizing the endpoint at /api/v2/notification_templates/<n>/test. This will emit a test notification given the configuration defined by the notification. These test notifications will also appear in the notifications list at /api/v2/notifications

Notification Types

The currently-defined Notification Types are:

Each of these have their own configuration and behavioral semantics and testing them may need to be approached in different ways. The following sections will give as much detail as possible.

Email

The email notification type supports a wide variety of SMTP servers and has support for SSL/TLS connections and timeouts.

Testing Considerations

The following should be performed for good acceptance:

Test Service

Set up a local SMTP mail service. Some options are listed below:

Slack

Slack is simple to configure; it requires a token, which you can get from creating a bot in the integrations settings for the Slack team.

Testing Considerations

The following should be performed for good acceptance:

Mattermost

The Mattermost notification integration uses Incoming Webhooks. A password is not required because the webhook URL itself is the secret. Webhooks must be enabled in the System Console of Mattermost. If the user wishes to allow AWX notifications to modify the Icon URL and username of the notification, then they must enable these options as well.

In order to enable these settings in Mattermost:

  1. Go to System Console > Integrations > Custom Integrations. Check "Enable Incoming Webhooks".
  2. Optionally, go to System Console > Integrations > Custom Integrations. Check "Enable integrations to override usernames" and Check "Enable integrations to override profile picture icons".
  3. Go to Main Menu > Integrations > Incoming Webhook. Click "Add Incoming Webhook".
  4. Choose a "Display Name", "Description", and Channel. This channel will be overridden if the notification uses the channel option.

Testing Considerations

Test Service

Rocket.Chat

The Rocket.Chat notification integration uses Incoming Webhooks. A password is not required because the webhook URL itself is the secret. An integration must be created in the Administration section of the Rocket.Chat settings.

The following fields are available for the Rocket.Chat notification type:

Testing Considerations

Test Service

Pagerduty

Pagerduty is a fairly straightforward integration. The user will create an API Key in the Pagerduty system (this will be the token that is given to AWX) and then create a "Service" which will provide an "Integration Key" that will also be given to AWX. The other options of note are:

Testing considerations

Test Service

Pagerduty allows you to sign up for a free trial with the service.

Twilio

Twilio is a Voice and SMS automation service. Once you are signed in, you'll need to create a phone number from which the message will be sent. You'll then define a "Messaging Service" under Programmable SMS and associate the number (the one you created for this purpose) with it. Note that you may need to verify this number or some other information before you are allowed to use it to send to any numbers. The Messaging Service does not need a status callback URL nor does it need the ability to process inbound messages.

Under your individual (or sub) account settings, you will have API credentials. The Account SID and AuthToken are what will be given to AWX. There are a couple of other important fields:

Testing Considerations

Test Service

Twilio is fairly straightforward to sign up for but there may not be a free trial offered; a credit card will be needed to sign up for it though the charges are fairly minimal per message.

IRC

The AWX IRC notification takes the form of an IRC bot that will connect, deliver its messages to channel(s) or individual user(s), and then disconnect. The AWX notification bot also supports SSL authentication. The AWX bot does not currently support Nickserv identification. If a channel or user does not exist or is not online, then the Notification will not fail; the failure scenario is reserved specifically for connectivity.

Connectivity information is straightforward:

Test Considerations

Test Service

There are a few modern IRC servers to choose from. InspIRCd is recommended because it is actively maintained and pretty straightforward to configure.

Webhook

The webhook notification type in AWX provides a simple interface for sending POSTs to a predefined web service. AWX will POST to this address using application/json content type with the data payload containing all relevant details in JSON format.

The parameters are fairly straightforward:

Test Considerations

Test Service

A very basic test can be performed by using netcat:

netcat -l 8099

...and then sending the request to: http://\<host>:8099

Note that this won't respond correctly to the notification, so it will yield an error. Using a very basic Flask application for verifying the POST request is recommended; you can see an example here:

https://gist.github.com/matburt/73bfbf85c2443f39d272

The link below shows how to define an endpoint and parse headers and JSON content. It doesn't show how to configure Flask for HTTPS, but is fairly straightforward: http://flask.pocoo.org/snippets/111/

You can also link an httpbin service to the development environment for testing webhooks using:

docker run --network="tools_default" --name httpbin -p 8204:80 kennethreitz/httpbin

This will create an httpbin service reachable from the AWX container at http://httpbin/post, http://httpbin/put, etc. Outside of the container, you can reach the service at http://localhost:8204.

Grafana

The Grafana notification type allows you to create Grafana annotations. Details about this feature of Grafana are available at http://docs.grafana.org/reference/annotations/. In order to allow AWX to add annotations, an API Key needs to be created in Grafana. Note that the created annotations are region events with start and endtime of the associated AWX Job. The annotation description is also provided by the subject of the associated AWX Job, for example:

Job #1 'Ping Macbook' succeeded: https://towerhost/#/jobs/playbook/1

The configurable options of the Grafana notification type are:

Test Considerations

Test Service