Alertmanager alerts, next to the deploys that caused them

Alertmanager tells you something is wrong. It cannot tell you that a deploy went out four minutes earlier, because it has never heard of your deploys. Kollaber puts both on one timeline, which turns the usual question — what changed just before this fired? — into something you can read off the screen.

What lands on the timeline

Service
Resolved from the service, job, or app label, falling back to alertname.
Severity
The severity label, kept with the full label set.
Summary
The summary and description annotations.
Source
The generator URL back to the firing rule.

Setup

  1. 1Add a receiver

    Point a webhook receiver at Kollaber with your environment UUID, and send the shared secret as a bearer token.

    receivers:
      - name: kollaber
        webhook_configs:
          - url: https://kollaber.io/webhooks/alertmanager?environment_id=<uuid>
            send_resolved: true
            http_config:
              authorization:
                type: Bearer
                credentials: <WEBHOOK_SECRET>
  2. 2Route alerts to it

    Add the receiver to your route tree. Alerts can go to Kollaber as well as to your existing paging receiver.

    route:
      receiver: pagerduty
      routes:
        - receiver: kollaber
          continue: true
  3. 3Wait for something to fire

    Each entry in the delivery becomes one alert event, timestamped when the alert started rather than when the webhook arrived.

Good to know

  • Keep send_resolved: true. A firing alert is recorded as a failure and its resolution as a success, so without it the timeline shows problems that never end.
  • Alertmanager re-delivers firing alerts every repeat_interval. Deliveries whose fingerprint already sits at the same status are skipped, so a long-running alert does not bury the timeline. A firing-to-resolved transition differs, so it still lands.
  • Events are timestamped from startsAt, and from endsAt once resolved. Grouping and repeat_interval can delay a delivery by minutes, which would otherwise scramble the ordering against deploys.
  • Alertmanager cannot compute an HMAC over the body, so the bearer token is the strongest authentication it can offer. Kollaber also accepts the secret in an X-Kollaber-Secret header.

Questions

Does this replace PagerDuty or Opsgenie?

No. Kollaber does not page anyone and evaluates no alert rules. Route alerts to it alongside your pager so the alert is recorded next to the changes around it.

Will a noisy alert flood the timeline?

No. Repeat deliveries are deduplicated by fingerprint, so an alert that fires for six hours is one event, not one event per repeat_interval.

Which label becomes the service name?

The first of service, job, or app that is set, falling back to alertname. Setting a service label on your rules gives the cleanest grouping.

Other integrations

Start recording Prometheus Alertmanager changes

The free plan covers two environments and five members. The full reference for this integration lives in the docs.