Toil Reduction & Automation Strategy

Identifying, measuring, and eliminating toil — manual, repetitive, automatable operations work — to reclaim engineering time for lasting improvements.

Overview

What counts as toil

Toil is a specific kind of work defined by Google SRE. Not all operational work is toil: responding thoughtfully to a novel incident is not; running the same manual DB backup script every Friday is.

Manual — A human does it by hand
Repetitive — The same task again and again
Automatable — A machine could do it
Tactical — Not strategic
No enduring value — Nothing is better afterward
O(n) with service growth — Grows as the service grows

The 50% cap

The SRE team commitment model caps toil at 50% of engineer time. The other 50% goes to engineering work that reduces future toil: automation, reliability and scalability.

> [!WARNING] Above 50% the team is in a reactive death spiral. Measuring the current toil percentage is the first step out.

Finding toil: the toil diary

Each team member tracks their tasks for two weeks and classifies each as toil or engineering work. It is the most reliable way to identify toil.

Manual deployments
Certificate renewals
Access provisioning requests
Recurring data dumps
Manual scaling actions
Weekly report generation
Recurring alert investigation that always resolves the same way

Prioritizing: the automation ROI matrix

Frequency — Daily >> weekly >> monthly
Time cost per occurrence — Larger
Error risk — High human-error risk
Automation difficulty — Trivial scripts first

> [!TIP] The best target is frequent, time-consuming, high-error-risk work that is technically straightforward to automate.

Automation patterns

Runbook automation — PagerDuty Process Automation, Rundeck — Manual runbooks become push-button or auto-triggered workflows
Self-service portals — Backstage scaffolder, internal Slack bots — Access provisioning and environment creation requests
GitOps — ArgoCD, Flux — Manual deployment steps
Auto-remediation — Triggered by alerts — Pager wakeups for known failure patterns

> [!IMPORTANT] The goal is not zero toil (impossible) but sustainable toil below 30%, leaving 70% of time for strategic work.

Key concepts

Toil (SRE definition): Work that is manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with service growth. Distinct from operational overhead or project work.
50% toil cap: Google SRE organizational commitment: toil must not exceed 50% of any team's time. When exceeded, leadership intervention is required to reduce service load or add capacity.
Toil diary: Two-week time-tracking exercise where engineers classify each task as toil or engineering work. The most reliable method for measuring actual toil percentage before designing automation strategy.
Automation ROI matrix: Prioritization framework: score tasks by frequency × time cost × error risk ÷ automation difficulty. High-frequency, high-cost, high-risk, low-difficulty tasks are automated first.
Runbook automation: Converting manual step-by-step runbooks into push-button or trigger-based automated workflows using tools like PagerDuty Process Automation, Rundeck, or custom scripts. The lowest-risk form of auto-remediation.
Self-service portal: Developer-facing interface (Backstage, Slack bot, internal web app) that handles access provisioning, environment creation, and routine ops requests without SRE involvement. Eliminates the largest source of interrupt-driven toil.

When to use it

Interview questions about how SRE teams prioritize work and balance operational vs engineering time
Designing an automation strategy for an ops team drowning in manual work
Building a toil measurement framework for engineering leadership reporting
Evaluating which manual tasks to automate first using ROI prioritization
Explaining the difference between toil and other types of operational work

Interview questions

You join an SRE team where engineers spend 70% of their time on manual operations. What is your plan?