Incident channels: PagerDuty, JSM/Opsgenie, ServiceNow and the rest
Every channel on this page sits on the same architecture notifications already documents: a channel is one delivery destination, a rule decides which event reaches it, and delivery is fail-open so a channel problem can never block a backup. Nothing here changes that. What this page adds is two new channel kinds built on it (Jira Service Management/Opsgenie and ServiceNow Event Management), one wire that was missing before (a recovered backup now tells PagerDuty, and the two new channels, to close what it opened), and honest mapping guidance for three more platforms that do not speak our shape natively.
The auto-resolve wire: what changed
Four conditions the engine already alerted on can also clear: a failing or stale downpipe returning to health, and a downpipe’s replication falling below its configured copy count then recovering, or a run that was about to age out of history while a replica still lacked it no longer being at risk. Until now, nothing told a channel when one of those four cleared; a PagerDuty incident that opened on a failure stayed open until a human closed it, even after the very next run succeeded.
The engine now detects the falling edge of each of those four conditions, the same cooldown state it already tracked to avoid re-paging on every tick, and emits a second, matching event the instant the condition clears: the identical event name and downpipe as the original alert, marked as a recovery, with the same severity the original alert used (so the recovery clears the same rule threshold the trigger did). That one field, and the stable pairing of an event with its downpipe, is the entire mechanism. A channel that has a concept of closing something it opened reads that pairing and closes it; a channel that does not just delivers another line.
| Channel | Recognises a recovery? | What happens |
|---|---|---|
| PagerDuty | Yes | Sends event_action: "resolve" against the same dedup_key the trigger used, closing the matching incident. |
| Jira Service Management / Opsgenie | Yes | Closes the alert by the same alias the trigger’s create used. |
| ServiceNow Event Management | Yes, at the Alert layer | Sends severity 0 (Clear) against the same message_key; see the honest scope caveat below for what that does and does not guarantee at the Incident layer. |
| Generic webhook, Slack, Microsoft Teams, email | No distinct signal | Delivers the recovery as another message. The detail text says the condition cleared, in plain English; there is no boolean field a receiver can key on. See the mapping recipes below for how far a generic webhook can be pushed toward auto-resolve regardless. |
PagerDuty: trigger and resolve
Adding a PagerDuty channel is unchanged from notifications: a routing key, nothing else. What is new is what a recovery does. Every emission carries a dedup_key derived from the downpipe id and the event name (or the event name alone for an account-level event like a canary), and a recovery emission reuses the identical key with event_action: "resolve" instead of "trigger". PagerDuty’s own Events API v2 correlates the two, so the incident a failure opened is the exact incident its later recovery closes; there is nothing to configure on the PagerDuty side beyond the integration you already have.
Jira Service Management and Opsgenie
JSM absorbed Opsgenie’s alerting engine, and both accept the same Alert API shape today. Opsgenie itself is sunsetting in April 2027, so this channel is built against the JSM endpoint contract, which both providers honour identically now and for the life of that migration. One channel client covers either provider; which one you are pointed at is simply whichever alert-create URL you configure.
Unlike PagerDuty’s single trigger-or-resolve envelope, the Alert API is a create, then a separate close-by-alias: a new or continuing condition posts a create (a re-post with the same alias de-duplicates onto the still-open alert rather than opening a second one), and a recovery posts to that alias’s close endpoint. The alias is the same downpipe-plus-event pairing PagerDuty’s dedup_key uses, so an operator running both channels sees identical correlation behaviour across them.
| Our severity | JSM/Opsgenie priority |
|---|---|
| critical | P1 |
| warning | P3 |
| info | P5 |
P2 and P4 are left as headroom for your own hand-raised alerts to sit above or below ours without collision.
Add the channel on the Notifications screen, the same Channels tab notifications covers. Pick the kind “Jira Service Management / Opsgenie (Alert API)”, give it a name, the HTTPS alert-create URL (for Opsgenie, https://api.opsgenie.com/v2/alerts; for a JSM Cloud tenant, its own alert-api base), and the GenieKey API token. The token field is write-only: it is sent once over the authenticated channel, then sealed by your engine and never re-displayed, so on a later edit you leave it blank to keep the current token and paste a new one only to rotate it. Once saved, the channel is selectable in a rule exactly like any other, on the Rules tab.
Or add it against the API
An Owner with notify.config can create the same channel directly, the route the console calls:
POST /admin/notify/channels HTTP/1.1
Host: console.example.com
Content-Type: application/json
{
"kind": "jsm",
"name": "Opsgenie: platform team",
"url": "https://api.opsgenie.com/v2/alerts",
"apiKey": "<your GenieKey API token>",
"enabled": true
}The GenieKey token rides only in the Authorization header at send time, sealed at rest under its own domain-separated key, never in the alert body.
ServiceNow Event Management
ServiceNow’s Event Management module ingests a POST to its em_event table API, and its correlation engine groups events sharing a message_key into one Alert. A later event on the same key at a lower severity updates that Alert, and severity 0 clears it. There is no separate close endpoint the way PagerDuty and JSM have one: closing is posting another event with the same key at severity 0, so create and recover share the identical shape here, varying only severity and description.
| Our severity or state | ServiceNow severity |
|---|---|
| critical | 1 |
| warning | 4 |
| info | 5 |
| recovered (any original severity) | 0 (Clear) |
Severities 2 (Major) and 3 (Minor) are left unused, reserved for a human to hand-adjust an Alert’s severity inside ServiceNow itself. Authentication is HTTP Basic: a username (not a secret, always resupplied) and a password, sealed at rest the same way the JSM token is, under its own distinct key so one channel’s credential can never be resolved through the other’s wrapping.
Severity 0 clears the Alert. Whether the Incident auto-closes is yours to configure
Reaching severity 0 clears the ServiceNow Alert this channel creates. Whether that Alert’s associated Incident (if Event Management is configured to open one) also auto-closes depends entirely on your own ITOM or Event Management business rules, the “auto-close incident on alert-clear” property that lives in your ServiceNow instance. This is a genuine customer-side configuration choice, not a guarantee this engine makes or controls. If you want a recovered backup to close an Incident, not only the Alert beneath it, set that up on the ServiceNow side; this channel’s part of the contract ends at clearing the Alert.
Add it the same way as JSM, on the Notifications Channels tab, choosing the kind “ServiceNow Event Management”. It carries the em_event table URL, the Basic-auth username (not a secret, so it is prefilled on an edit and always resupplied), and the password. The password is write-only like the JSM token: sent once, then sealed, and left blank on a later edit to keep the current one. Because the password is kept only while the URL is unchanged, repointing the URL means resupplying it.
Or add it against the API
The equivalent direct call:
POST /admin/notify/channels HTTP/1.1
Host: console.example.com
Content-Type: application/json
{
"kind": "servicenow",
"name": "ServiceNow: platform events",
"url": "https://yourinstance.service-now.com/api/now/table/em_event",
"username": "downpipes-integration",
"apiKey": "<the account's password>",
"enabled": true
}Generic webhook: mapping recipes for three more platforms
A generic webhook channel posts a small, versioned JSON body to your own URL on every matching event (the full security model is described here):
{
"kind": "downpipe-event-v1",
"at": "2026-07-05T02:14:08.221Z",
"event": "backup-failure",
"severity": "critical",
"downpipe": { "id": "dp_8f2a1", "name": "Prod KV" },
"detail": "Prod KV: the last run failed"
}Two things are true of this body and matter for mapping it into another platform. First, event plus downpipe.id is a stable pairing that recurs unchanged across a trigger and its later recovery, exactly the correlation key PagerDuty’s dedup_key, JSM’s alias and ServiceNow’s message_key are all built from; a platform that lets you choose your own grouping or deduplication key from an inbound payload should be pointed at that pair. Second, there is no boolean field marking a recovery. A recovery arrives as another POST of the identical event and downpipe.id, distinguished only by its detail wording, and that wording is not one fixed keyword across every event:
| Event | Recovery detail reads |
|---|---|
backup-failure |
“{name}: the last run succeeded; the previous failure has cleared” |
backup-stale |
“{name}: a recent run succeeded; the staleness has cleared” |
replication-degraded |
“{name}: replication recovered; N of M copies proven, every destination reachable” |
run-at-risk-eviction |
“{name}: the at-risk run is no longer exposed to eviction; N of M copies proven, every destination reachable” |
This is genuinely today’s wording, read from the engine source rather than guessed, but it is prose, not a versioned field, so a rule built against it should be treated as a convenience you verify occasionally, not a permanent contract.
incident.io
Its custom HTTP alert source is the cleanest of the three to wire up, because its transform expression runs directly against whatever JSON you send it, no relay required. Point the source at this webhook’s URL, and in the transform (plain JavaScript, the incoming body available as $) compute both the correlation key and a firing-or-resolved status from the fields above:
return {
title: $.detail,
status: /has cleared|replication recovered|no longer exposed to eviction/.test($.detail) ? "resolved" : "firing",
description: $.detail,
metadata: { event: $.event, downpipe_id: $.downpipe ? $.downpipe.id : null },
};Set the deduplication key to a value unique per event-and-downpipe pair (a metadata field you compute above, or downpipe.id alone if you run one alert source per event type). incident.io resolves an alert when a later event with the same deduplication key reports status: "resolved", which is exactly what the expression above produces from our recovery wording.
Grafana OnCall / IRM
Its Formatted Webhook integration recognises a fixed field set: alert_uid (the grouping key), title, state (ok or alerting, the field that drives auto-resolution), and message. Our webhook body does not use those field names, so reaching this path means an intermediary you control (a small relay, for example a Cloudflare Worker, that receives our downpipe-event-v1 body and re-posts Grafana’s shape), mapping alert_uid from event plus downpipe.id, state from the same recovery-wording check as above, and message from detail. Grafana’s raw “Webhook” integration type accepts arbitrary JSON directly with no relay, but whether its Jinja2 templating can compute a resolve signal from an arbitrary field on that path is not confirmed here; check Grafana’s own current documentation before relying on it, or use the Formatted Webhook path above, which is.
Splunk On-Call
Its REST endpoint integration expects its own fixed fields too: message_type (one of CRITICAL, WARNING, ACKNOWLEDGEMENT, INFO or RECOVERY), entity_id (the field that must stay identical across an incident’s whole lifecycle), entity_display_name and state_message. As with Grafana’s Formatted Webhook, our body’s field names do not match, so reaching auto-resolve here needs the same kind of relay: map entity_id from event plus downpipe.id, message_type from severity (critical maps to CRITICAL, warning to WARNING, info to INFO) or to RECOVERY when the detail-wording check above matches, entity_display_name from downpipe.name, and state_message from detail. A RECOVERY message against the same entity_id closes the incident Splunk On-Call opened for it.
Where a relay is the honest answer above, it is a small piece of code you own and can inspect, not a dependency this product adds. The alternative, simpler answer for any of these three platforms is to run the webhook fire-only, the same as Slack and Teams below, and let a human close what it opens.
Slack and Teams: notification fan-out, not an incident system
Slack and Microsoft Teams stay exactly what notifications already describes: a message to a channel or an incoming-webhook connector, built from the same redaction-safe detail line. Neither carries any notion of an open or closed incident, so a recovery reaches them as a second message, worded as a recovery, and nothing closes on either platform’s side. Treat both as where a human reads that something happened, never as the system of record for whether it is still happening. Where you want an actual incident that a recovery can close, wire one of the channels above instead, or fan out to Slack or Teams alongside it for visibility.
Where this fits
For adding any of the five built-in channel kinds, writing the rules that route events to them, and the delivery-history view, read notifications. For the canonical, honest catalogue of which events actually fire, including the four this page’s auto-resolve wire covers, read alert events. For the egress screening every URL-bearing channel here inherits, read securing notification webhooks. For the two-audience framing this page sits under, read the monitoring overview.
Last updated .