Opening a war room from a Prometheus alert

This page is about opening incidents automatically the moment a Prometheus alerting rule fires and Alertmanager notifies. It is not about reading telemetry; if you want the investigator agent to query Prometheus metrics once an incident already exists, see OpenTelemetry Collector & Prometheus instead. The two are independent, and most teams eventually set up both.

How it works

Prometheus itself never calls Landfall — Alertmanager does, Prometheus's own alerting companion. Alertmanager has its own dedicated endpoint, POST /triggers/alertmanager, authenticated by a per-organization shared secret presented in a custom header — the same shape Opsgenie's own action webhook uses (see Integrations architecture for why: Alertmanager's own webhook receiver can set a static header but cannot compute an HMAC signature the way PagerDuty's or GitHub's webhooks do, so a shared secret is the applicable model, not a signature).

Every notification group gets its own attach identity. Alertmanager's own groupKey — stable across every repeat notification for a still-firing group (Alertmanager's repeat_interval) — is what Landfall uses to recognize "this is the same alert group, not a new one." A repeat notification attaches to the already-open incident and writes nothing new; the resolved batch Alertmanager sends when a group stops firing opens nothing and changes no incident's status (a resolution signal, if ever surfaced, would be a distinct timeline entry — not implemented today).

What you'll need

• An owner/admin seat on your Landfall organization, to issue the webhook secret below.
• Prometheus and Alertmanager already running, with at least one alerting rule defined and loaded via rule_files:.
• Network reachability from wherever Alertmanager runs to Landfall's public API — the same reachability any real webhook caller needs.

1. Issue a webhook secret

This step doesn't have a button in the app yet: an owner/admin issues it via the API, using their normal Landfall session:

SESSION=$(curl -s -X POST https://api.landfalls.ai/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"<you>","password":"<your password>","orgSlug":"<your-org>"}' \
  | jq -r .accessToken)

curl -X POST https://api.landfalls.ai/o/<your-org>/triggers/alertmanager-secret \
  -H "Authorization: Bearer $SESSION"

The response is { "secret": "<64 hex characters>" } — shown exactly once. Store it now: you'll paste it into Alertmanager's own config next, and Landfall never stores or returns the raw value again (only its SHA-256 hash, to verify future deliveries against). Re-issuing overwrites the stored hash immediately, with no dual-valid window — the previous secret stops working the moment a new one is issued.

2. Point Alertmanager at Landfall

Add a webhook_configs receiver to Alertmanager's own alertmanager.yml, targeting Landfall's trigger endpoint with the secret from step 1 as a custom header. Alertmanager's http_config.headers field is what lets a receiver attach an arbitrary header (distinct from authorization:, which only ever sets the Authorization header itself):

route:
  receiver: landfall
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: landfall
    webhook_configs:
      - url: https://api.landfalls.ai/triggers/alertmanager
        send_resolved: true
        http_config:
          headers:
            X-Landfall-Alertmanager-Secret:
              secrets:
                - <the secret from step 1>

3. Define what should page you

This endpoint only ever reacts to what Alertmanager sends it — the alerting rules themselves live in Prometheus's own rule_files:. At minimum, a pipeline-health rule (so a failing Collector/Prometheus is itself visible) and a representative application-level rule (elevated error rate or latency) are worth having from day one:

groups:
  - name: landfall-observability-pipeline
    rules:
      - alert: OtelCollectorDown
        expr: up{job="otel-collector"} == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: 'OpenTelemetry Collector is not being scraped successfully'

  - name: landfall-application
    rules:
      - alert: CoreApiElevatedErrorRate
        expr: |
          (
            sum(rate(traces_spanmetrics_calls_total{http_response_status_code=~"5.."}[5m]))
            / sum(rate(traces_spanmetrics_calls_total[5m]))
          ) > 0.05
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: 'core-api 5xx rate exceeds 5% over 5 minutes'

The severity label you set on a rule maps directly to the incident's proposed severity: critical → sev1, page → sev2, anything else (including no label) → sev3.

4. Test it

The fastest check is posting a realistic Alertmanager notification body directly, without waiting for a real rule to fire — this is close to what Alertmanager itself sends:

curl -i -X POST https://api.landfalls.ai/triggers/alertmanager \
  -H "X-Landfall-Alertmanager-Secret: <your secret>" \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4",
    "status": "firing",
    "groupKey": "{}:{alertname=\"OtelCollectorDown\"}",
    "receiver": "landfall",
    "groupLabels": { "alertname": "OtelCollectorDown" },
    "commonLabels": { "alertname": "OtelCollectorDown", "severity": "critical", "job": "otel-collector" },
    "commonAnnotations": { "summary": "OpenTelemetry Collector is not being scraped successfully" },
    "externalURL": "http://prometheus:9090",
    "alerts": [{
      "status": "firing",
      "labels": { "alertname": "OtelCollectorDown", "severity": "critical" },
      "annotations": { "summary": "OpenTelemetry Collector is not being scraped successfully" },
      "startsAt": "2026-08-24T00:00:00Z",
      "fingerprint": "abc123"
    }]
  }'

Opens an incident titled OtelCollectorDown at sev1 (from severity=critical), with service:otel-collector as its entity (from the job label). POST it again unchanged and it attaches to the same incident rather than opening a second one — the same groupKey is the attach identity.

This is the real result from Landfall's own dev environment — this exact rule fired for real (the Collector was deliberately stopped to test the path end to end), and Alertmanager delivered the notification above. The incidents list shows it with an Alertmanager provenance badge, the otel-collector entity, and the rule's own summary text — nothing here is a mock-up:

Incidents

What Alertmanager's fields become

Troubleshooting

Reference