Event-Driven Automation and StackStorm

If-this-then-that for infrastructure. StackStorm sensors, triggers, rules, actions and Orquesta workflows, how the model compares to Event-Driven Ansible and to a Kubernetes operator, and the safety machinery that stops an automation storm from amplifying the incident it was meant to fix.

Overview

Event-driven automation, built on purpose

Every platform team eventually builds this by accident. An alert fires, someone reads a runbook, runs three commands and closes the incident; fifty times later somebody writes a script and wires it to the alert. This topic is about the machinery: the objects, the plumbing, and what separates automation that shortens outages from automation that causes them.

The StackStorm object model

StackStorm is the canonical general-purpose implementation and the one interviewers name.

Sensor — Watches the outside world and emits events
Trigger — Typed representation of those events
Rule — Matches trigger payloads against criteria and decides what runs
Action — What runs: Python, shell, HTTP or a workflow
Pack — Bundles it all into a versioned, installable unit
Orquesta — Workflow engine: branching, parallelism, explicit failure paths
Datastore — Key-value store with encryption and TTLs for shared state

Operator versus event-driven automation

Trigger model — Level-triggered — Edge-triggered
Style — Declarative — Imperative
Missed event — Next reconcile fixes it — The event is gone
Failure halfway — Reconciler finishes it — Nothing finishes it
Fits — In-cluster lifecycles — Tickets, edge load balancers, vendors, switches, paging humans

Other engines

Event-Driven Ansible — Rulebooks with sources, rules, conditions and actions; a Drools rules engine makes multi-event correlation over a window first-class; run_playbook and run_job_template reuse an Ansible estate
Rundeck — Job- and scheduler-centric, strong access control, self-service execution; a governed runbook runner more than a rules engine

Safety is the hard part

An automation with production credentials will, at some point, receive a burst of correlated alerts during a real incident. Unconstrained, it runs hundreds of remediations against a degraded system and becomes the incident.

Idempotent actions — A duplicate execution is harmless
Rate limits and concurrency caps — A storm cannot amplify
Circuit breaker — Trips a rule off after too many firings in a window
Dry run and approval gates — Required for anything destructive
Kill switch — One command a stressed on-call can run without reading docs

The audit trail

Automated remediation removes the human whose memory was the record. What replaces it must be better: which event, which rule matched on what criteria, which workflow and parameters, which identity authorized it, what each step returned and the final state.

> [!IMPORTANT] A system that changes production without a defensible record of who changed what and why is not a platform capability; it is an unbounded liability.

Key concepts

Sensor: The inbound integration. A Python plugin that either polls on an interval or holds a long-lived connection and dispatches trigger instances when it observes something. Sensors are the piece with no natural health signal, so a dead sensor presents as automation that quietly stopped working. Monitor sensor liveness explicitly rather than inferring it from rule activity.
Trigger and trigger instance: A trigger is the event type with a reference and a payload shape; a trigger instance is one occurrence carrying a concrete payload. Generic triggers cover webhooks and timers (IntervalTimer with unit and delta, CronTimer, DateTimer); integration triggers come from packs. st2 trigger-instance list and re-emit are the first two debugging commands you reach for.
Rule and criteria: YAML binding a trigger type to an action, with criteria that all must match. Operators include equals, nequals, lessthan, greaterthan, regex, iregex, matchwildcard, the case-insensitive iequals family, contains and inside, exists and nexists, search for arrays, and timediff_lt and timediff_gt. Jinja references such as {{ trigger.labels.namespace }} map payload fields onto action parameters. There is no OR at rule level.
Action and runner: The outbound integration, described by metadata that declares a runner type and typed parameters. Runners include local-shell-cmd, local-shell-script, remote-shell-cmd over SSH, python-script, http-request, and orquesta. Typed parameters are a real control surface: an enum or a regex on a parameter stops a malformed payload from becoming a malformed command.
Orquesta workflow: The workflow DSL. Top level is version, input, vars, tasks, and output. A task names an action, its input, and next transitions; each transition has a when expression, an optional publish into the context, and a do naming successor tasks. join: all is the barrier for parallel branches, with drives iteration, and retry and delay handle transient failure. Failure paths are written explicitly with when failed().
Pack: The versioned deployment unit — a directory of actions, sensors, rules, aliases, and config, installed from StackStorm Exchange or a Git remote. Packs are what make automation reviewable and reproducible instead of a directory of scripts on a jump host, and they are the natural boundary for ownership and RBAC.
Datastore with TTL: The key-value store behind st2 key set, get, list, and delete, with --scope for system versus user, --encrypt for secrets read back through the decrypt_kv Jinja filter, and --ttl for automatic expiry. The TTL is the standard throttle primitive: write a key named for the target before remediating, refuse to remediate while it exists, and let expiry define the cooldown.
Action alias: The ChatOps binding. YAML with name, pack, action_ref, a formats list of chat phrases that invoke it, and ack and result blocks controlling the confirmation and the outcome message. This is what puts a remediation and its result into the channel where the incident is already being discussed, which is half of why the audit story works at all.

When to use it

Alert-to-remediation paths where the runbook is well understood and mechanical: restart a stuck consumer, expand a volume, rotate a leaking credential, fail over a replica
Procedures that cross system boundaries a Kubernetes controller has no API for — ticketing, cloud accounts, network devices, vendor portals, paging
Enforcement and compliance sweeps triggered by cloud audit events: a public bucket appears, an untagged instance launches, a security group opens a port
Onboarding, offboarding, and other multi-system business workflows that need branching, human approval, and an audit record
Whenever the same three commands have been run from the same runbook more than a handful of times — see the toil-reduction topic for the decision framework

Interview questions

Walk me through everything that happens from an external alert to a completed automated remediation.
A monitoring change causes a burst of two hundred correlated alerts. How do you make sure your automation does not turn that into a bigger outage?
When would you build this as a Kubernetes operator instead, and when is an operator the wrong tool?
Compare StackStorm, Event-Driven Ansible, and Rundeck. How do you pick?
Design an auto-remediation for a disk-filling alert, end to end, including what you would refuse to automate.
What has to be in the audit trail for automated remediation, and why does an interviewer care?