Notifications, Alerts and Webhooks

notification-controller runs in both directions: outbound Providers and Alerts push reconciliation events to Slack, PagerDuty and Git commit status, while inbound Receivers accept webhooks that cut git-to-cluster lag from minutes to seconds.

Overview

The part of Flux that talks to humans

Everything else in Flux is a closed loop between a Git repository and a cluster. notification-controller lets that loop talk to humans and lets humans poke it. It is installed by default in every bootstrap and owns three CRDs that split cleanly into two directions.

Outbound — Provider — notification.toolkit.fluxcd.io/v1beta3 — Destination events are dispatched to
Outbound — Alert — notification.toolkit.fluxcd.io/v1beta3 — Routing rule: which events to care about
Inbound — Receiver — notification.toolkit.fluxcd.io/v1 — Authenticated endpoint that requests reconciliation

Outbound: event bus plus routing table

Every Flux controller emits structured events to an internal endpoint that notification-controller serves: a Kustomization failing health checks, a HelmRelease upgrade succeeding, a GitRepository failing to authenticate. Events carry involvedObject, severity, reason, message and a metadata map.

Provider types

The type list tells you which integrations you never have to build.

Chat and paging — slack, discord, msteams, googlechat, telegram, webex, rocket, matrix, lark, zulip
Incident and observability — opsgenie, pagerduty, alertmanager, sentry, datadog, grafana
Transport and fan-out — generic, generic-hmac, azureeventhub, googlepubsub, nats, otel, githubdispatch
Git commit status — github, gitlab, gitea, bitbucket, bitbucketserver, azuredevops, plus pull request comment variants

Commit status closes the feedback loop

CI tells you the build passed. Nothing normally tells the pull request whether the deploy actually reconciled. Point a github Provider at your repository and alert on Kustomization events, and every commit carries a status check for that revision in the cluster.

> [!TIP] Reviewers get the deploy outcome next to the test outcome.

Inbound: the Receiver

Flux is pull-based by design, which is what makes it safe: nothing outside the cluster needs cluster credentials. The cost is latency, bounded by spec.interval on your sources.

A Receiver exposes an authenticated HTTP endpoint that does exactly one thing: request reconciliation of a named list of resources. Because it cannot do anything else, exposing it is a much smaller risk than exposing the Kubernetes API.

What interviewers actually probe

It is rarely the YAML. It is whether you have run alerting that people actually read.

> [!WARNING] The failure mode is not missing alerts. It is a channel full of successful-reconciliation noise that everyone mutes, after which the one real failure goes unseen.

eventSeverity — Drops info noise
exclusionList — Drops known-transient messages
matchLabels — Routes by owning team
eventMetadata — Adds cluster and runbook context
Per-team Alert objects — Gives every event an owner

Key concepts

Provider: notification.toolkit.fluxcd.io/v1beta3. The outbound destination. spec.type selects the integration, spec.address the endpoint, spec.channel and spec.username shape the message, spec.secretRef points at a Secret that may carry token, address, username, password and headers. spec.certSecretRef supplies mTLS or a custom CA, spec.proxySecretRef a proxy, spec.timeout the HTTP deadline.
Alert: notification.toolkit.fluxcd.io/v1beta3. The routing rule. spec.providerRef names a Provider in the same namespace. spec.eventSources lists kind and name pairs (name may be the wildcard) optionally scoped by namespace and matchLabels. spec.eventSeverity is info or error. spec.inclusionList and spec.exclusionList are message regexes, exclusion taking precedence. spec.eventMetadata attaches context to every dispatch.
Receiver: notification.toolkit.fluxcd.io/v1. The inbound endpoint. spec.type selects payload validation, spec.events filters webhook event types, spec.resources lists the Flux objects to reconcile by apiVersion, kind and name, spec.secretRef supplies the shared token. The generated path lands in status.webhookPath.
Commit status Provider: The github, gitlab, gitea, bitbucket, bitbucketserver and azuredevops types do not send chat messages. They write a status check onto the Git commit whose revision the event refers to, so the repository shows whether that revision reconciled in the cluster. spec.address is the repository URL, not an API endpoint.
generic-hmac: Available as both a Provider type and a Receiver type. As a Provider it signs the outbound POST with an X-Signature header of the form hash-function equals hash, defaulting to sha256, using the token from the referenced Secret. As a Receiver it verifies the same header on inbound requests. Use it whenever the peer is your own service rather than a known SaaS.
Webhook path: The unguessable URL suffix a Receiver serves at, published to status.webhookPath as /hook/ followed by a hex digest. It is sha256 over the Secret token concatenated with the Receiver name and namespace, so rotating the token rotates the URL. For generic-oidc, which has no Secret, the digest covers name and namespace only.
Event rate limiting: notification-controller deduplicates dispatches keyed on involvedObject name, namespace and kind together with the message and metadata. The default suppression window is 5m, set by the --rate-limit-interval flag. Suppressed dispatches surface as HTTP 429 and are exported as Prometheus metrics.
resourceFilter: A CEL expression on a Receiver, evaluated per candidate resource with res bound to the resource metadata, req to the incoming webhook payload, and claims to OIDC token claims for generic-oidc. Used to reconcile only the subset of resources a monorepo push actually affects.

When to use it

Routing Flux failures to the team that owns the workload rather than to one firehose channel, using per-namespace Alert objects with matchLabels
Paging on-call for production reconciliation failures via an opsgenie or pagerduty Provider while chat gets the informational stream
Writing deploy outcome back onto Git commits and pull requests so reviewers see whether a revision actually reconciled, not just whether CI passed
Cutting git-to-cluster latency from the source interval down to seconds with a Receiver, then raising the interval to reduce API calls against your Git host
Triggering an immediate ImageRepository scan from a registry push webhook (harbor, dockerhub, quay, acr, gcr) instead of waiting out the scan interval
Fanning Flux events into an existing pipeline — googlepubsub, azureeventhub, nats, or a generic-hmac webhook into your own service

Interview questions

Walk through the outbound path: a Kustomization fails and the on-call engineer gets paged. What happens at each step?
How do you make Flux report deploy status back onto the Git commit, and why does that matter?
Explain how webhook receivers work and set one up end to end. Why is this worth doing when Flux already polls?
Your Slack channel is full of Flux noise and the team has muted it. How do you fix the alerting design?
What is generic-hmac, where does it appear, and when would you choose it?