Skip to content
downpipes docs

Send downpipes alerts to Splunk On-Call

Splunk On-Call (formerly VictorOps, now under Splunk and Cisco) receives downpipes alerts through a REST endpoint integration. There is no dedicated Splunk On-Call channel here; instead you add a generic webhook channel and map downpipes’ event JSON to Splunk On-Call’s fields with a documented recipe.

What you need

  • A Splunk On-Call account where you can add a REST endpoint integration and copy its notify URL.
  • Access to the downpipes console with permission to configure notifications (Operator, Approver or Owner) to add and test the channel.

Set it up

  1. In Splunk On-Call, add a REST endpoint integration and copy the notify URL it gives you.
  2. In the downpipes console, open Notifications, choose Add a channel and pick Webhook (generic HTTPS POST). Adding a channel needs permission to configure notifications, so an Operator, Approver or Owner can add one.
  3. Paste the notify URL into the HTTPS URL field. It must be https and carry no embedded credentials. Name the channel and save it, then a rule routes events to it.
  4. In Splunk On-Call, set up the mapping that turns downpipes’ event JSON into Splunk On-Call’s fields. downpipes posts a downpipe-event-v1 body with the event name, the severity, the downpipe id and name, and a one-line detail.
  5. Set entity_id from dedupKey so every event for one downpipe and event lands on the same incident. An account-level event has one key for each event name, so repeated alerts of one account-level event group onto one open alert, and no recovery closes it. Map message_type to RECOVERY when the body carries recovered: true, and from the severity in all other cases (Splunk On-Call reads CRITICAL, WARNING, ACKNOWLEDGEMENT, INFO and RECOVERY). Read the note below for engines before 0.3.6.
  6. Choose Test on the channel to confirm delivery without sending real alert content, then check the incident arrived in Splunk On-Call.

Good to know

  • entity_id is what ties an incident together. Splunk On-Call resolves an incident when it receives a RECOVERY on the same entity_id, so keep entity_id constant across the incident’s life. From engine 0.3.6, dedupKey is the same on a trigger and on the recovery that clears it, so map entity_id from it. Before engine 0.3.6, derive it from the downpipe id.
  • recovered tells your recipe to emit RECOVERY. From engine 0.3.6, a recovery carries recovered: true. Do not key RECOVERY on event or severity: a recovery reuses the same event name and the same severity as the trigger it clears, so a recipe keyed on those fields reads it as another firing and keeps the incident open. Before engine 0.3.6, the body has no recovered field, and only the detail wording tells a recovery from a trigger. The canary stream names its recovery in the event field as canary-recovered on every version. That event carries no recovered field, and its dedupKey is account:canary-recovered, not the key of the canary-dead alert. So this mapping does not close a canary-dead alert. Close it by hand. Confirm the current field names in the Splunk On-Call console too, because the mapping is theirs to define. When you do not want to maintain a mapping, route to the PagerDuty, Jira Service Management / Opsgenie or ServiceNow channel alongside Splunk On-Call: each of those sends a real resolve, close-by-alias or severity-0 clear on recovery with no mapping from you.

Last updated .