Opening a war room from a Loki ruler alert

This page is about opening incidents automatically the moment a log pattern condition fires in Loki's own bundled ruler. It is not about reading telemetry. If you want the investigator agent to query your logs once an incident already exists, see Loki instead. The two are independent, and most teams eventually set up both.

How it works

Loki's own bundled ruler evaluates LogQL based alerting rules (a log volume or log pattern condition, for example an elevated rate of level="error" lines) and notifies Alertmanager using the identical Prometheus compatible alerting API Prometheus itself uses. It is not a new trigger path into Landfall, only a second caller into the same webhook a Prometheus alerting rule already uses. Setting up that webhook and issuing the shared secret it authenticates with is common to every rule engine, so it lives on its own page: see the shared Alertmanager webhook and come back here once it's done.

A log pattern sourced incident carries the exact same Alertmanager provenance badge as a Prometheus rule sourced one, because Alertmanager is the actual delivering system in both cases. The rule's own service and summary labels are what distinguish "this came from a log pattern" from "this came from a metric threshold," not a separate provenance value.

What you'll need

• The shared Alertmanager webhook already set up, per the page linked above.
• Loki running with its own bundled ruler pointed at that same Alertmanager, including enable_alertmanager_v2: true. Alertmanager removed its v1 API in v0.27.0; left at the ruler's own default, notifications silently fail to reach a v2-only Alertmanager.

Define what should page you

Loki's ruler config points at Alertmanager directly, no per-rule webhook config needed:

ruler:
  alertmanager_url: http://alertmanager:9093
  enable_alertmanager_v2: true
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /tmp/loki-rules-wal
  enable_api: true

Then a rule group evaluating a LogQL condition, set explicit service and severity labels so the resulting incident carries a deterministic entity hint rather than one that depends on whatever label shape the underlying stream happens to have:

groups:
  - name: landfall-log-alerts
    rules:
      - alert: CoreApiElevatedErrorLogRate
        expr: |
          sum(rate({container="landfall-core-api-1"} | json | level="error" [5m])) > 0.5
        for: 5m
        labels:
          severity: critical
          service: core-api
        annotations:
          summary: 'core-api error-level log rate exceeds 0.5/s over 5 minutes'

A real result

This is the real result from Landfall's own dev environment. This exact LogQL rule fired for real, and Alertmanager delivered the notification. The incident carries the same Alertmanager badge as a Prometheus sourced incident. The core-api entity tag and summary text are what actually say "this came from a log rule," not the badge:

Incidents

Reference