Opening a war room from a Prometheus alerting rule

This page is about opening incidents automatically the moment a Prometheus alerting rule fires. It is not about reading telemetry. If you want the investigator agent to query Prometheus metrics once an incident already exists, see Prometheus instead. The two are independent, and most teams eventually set up both.

How it works

Prometheus itself never calls Landfall. Alertmanager does, Prometheus's own alerting companion, through a webhook receiver pointed at Landfall's own trigger endpoint. Setting up that receiver and issuing the shared secret it authenticates with is common to every rule engine that delivers through Alertmanager, so it lives on its own page: see the shared Alertmanager webhook and come back here once it's done.

What you'll need

• The shared Alertmanager webhook already set up, per the page linked above.
• Prometheus and Alertmanager already running, with at least one alerting rule defined and loaded via rule_files:.

Define what should page you

Landfall's own endpoint only ever reacts to what Alertmanager sends it. The alerting rules themselves live in Prometheus's own rule_files:. At minimum, a pipeline health rule (so a failing collector or scrape target is itself visible) and a representative application level rule (elevated error rate or latency) are worth having from day one:

groups:
  - name: landfall-observability-pipeline
    rules:
      - alert: OtelCollectorDown
        expr: up{job="otel-collector"} == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: 'OpenTelemetry Collector is not being scraped successfully'

  - name: landfall-application
    rules:
      - alert: CoreApiElevatedErrorRate
        expr: |
          (
            sum(rate(traces_spanmetrics_calls_total{http_response_status_code=~"5.."}[5m]))
            / sum(rate(traces_spanmetrics_calls_total[5m]))
          ) > 0.05
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: 'core-api 5xx rate exceeds 5% over 5 minutes'

The severity label you set on a rule maps directly to the incident's proposed severity. critical maps to sev1, page to sev2, and anything else (including no label) to sev3.

Test it

The fastest check is posting a realistic Alertmanager notification body directly, without waiting for a real rule to fire. This is close to what Alertmanager itself sends:

curl -i -X POST https://api.landfalls.ai/triggers/alertmanager \
  -H "X-Landfall-Alertmanager-Secret: <your secret>" \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4",
    "status": "firing",
    "groupKey": "{}:{alertname=\"OtelCollectorDown\"}",
    "receiver": "landfall",
    "groupLabels": { "alertname": "OtelCollectorDown" },
    "commonLabels": { "alertname": "OtelCollectorDown", "severity": "critical", "job": "otel-collector" },
    "commonAnnotations": { "summary": "OpenTelemetry Collector is not being scraped successfully" },
    "externalURL": "http://prometheus:9090",
    "alerts": [{
      "status": "firing",
      "labels": { "alertname": "OtelCollectorDown", "severity": "critical" },
      "annotations": { "summary": "OpenTelemetry Collector is not being scraped successfully" },
      "startsAt": "2026-08-24T00:00:00Z",
      "fingerprint": "abc123"
    }]
  }'

Opens an incident titled OtelCollectorDown at sev1 (from severity=critical), with service:otel-collector as its entity (from the job label). POST it again unchanged and it attaches to the same incident rather than opening a second one. The same groupKey is the attach identity.

This is the real result from Landfall's own dev environment. This exact rule fired for real, the collector was deliberately stopped to test the path end to end, and Alertmanager delivered the notification above. The incidents list shows it with an Alertmanager provenance badge, the otel-collector entity, and the rule's own summary text. Nothing here is a mock-up:

Incidents

Reference