Opening a war room from a Grafana alert rule

This page is about opening incidents automatically the moment a Grafana Unified Alerting rule fires. It is not about reading telemetry. If you want the investigator agent to query through Grafana once an incident already exists, see Grafana instead. The two are independent, and most teams eventually set up both.

How it works

Grafana's own Unified Alerting has a contact point type it labels "Alertmanager." Despite the name and despite Grafana's UI grouping it alongside webhook shaped integrations, the real wire behavior is that Grafana pushes fired alerts directly into Alertmanager's own alert ingestion API (POST /api/v2/alerts), the identical entry point Prometheus and Loki's ruler already use. It is not a webhook call to a URL Landfall controls, and it is not a new trigger path into Landfall. Setting up that Alertmanager and issuing the shared secret its own webhook uses toward Landfall is common to every rule engine, so it lives on its own page: see the shared Alertmanager webhook. What's specific to Grafana is the contact point below, which points at that same Alertmanager, not at Landfall directly.

Because Grafana pushes into Alertmanager rather than calling Landfall, a Grafana sourced incident carries the exact same Alertmanager provenance badge as a Prometheus or Loki sourced one. The rule's own service and summary labels are what distinguish "this came from Grafana" from the other two engines, not a separate provenance value.

This also means a Grafana alert rule can watch any datasource Grafana has configured, not only Prometheus or Loki, and still open a Landfall incident through this same unchanged pipeline.

What you'll need

• The shared Alertmanager webhook already set up, per the page linked above.
• Grafana's Unified Alerting enabled, with at least one alert rule.

Configure the contact point and a rule

Add a contact point of Grafana's own prometheus-alertmanager type, pointed at the same Alertmanager the shared setup page configured:

apiVersion: 1
contactPoints:
  - orgId: 1
    name: landfall-alertmanager
    receivers:
      - uid: landfall-alertmanager-1
        type: prometheus-alertmanager
        settings:
          url: http://alertmanager:9093

Grafana's own default root notification policy groups by grafana_folder and alertname. A provisioned policy replaces that default for the org, so keep both unless you have a stated reason to narrow them:

apiVersion: 1
policies:
  - orgId: 1
    receiver: landfall-alertmanager
    group_by: ['grafana_folder', 'alertname']

Then an alert rule, with service and severity labels set explicitly and a title distinct from any existing Prometheus or Loki rule:

apiVersion: 1
groups:
  - orgId: 1
    name: landfall-application
    folder: Landfall
    interval: 1m
    rules:
      - title: GrafanaCoreApiElevatedErrorRate
        condition: C
        # data: a query against a Grafana datasource, reduced to a single value,
        # then compared against a threshold
        for: 5m
        labels:
          service: core-api
          severity: critical
        annotations:
          summary: 'core-api condition exceeded threshold over 5 minutes (via Grafana Unified Alerting)'

A real result

This is the real result from Landfall's own dev environment. A Grafana Unified Alerting rule watching a log pattern condition fired for real, and Alertmanager delivered the notification exactly as it does for Prometheus and Loki:

Incidents

Re-firing the same rule attached to the already open incident rather than opening a duplicate, the same attach behavior every other rule engine gets from the shared groupKey mechanism.

Reference