Tackling Alert Fatigue with SLOs, Automation, and Machine Learning
Alert Fatigue is real, and it impacts your team's ability to make decisions about the information they are seeing about your environment. Too many alerts, false positives, alerts that have been over-prioritized, all accumulate and overwhelm already busy teams. Improving your alerts and employing better methods for dealing with real-time information will spare your team and improve customer experience.
SLOs allow teams to prioritize the more impactful or important aspects of their services. Metrics that center the user experience gives teams focus and goals. Highlighting the metrics that matter most gives teams space to disable alerts that contribute nothing to overall customer happiness.
Automation gives us more hands to deal with more issues without taking time away from more interesting work. Machine Learning tools group alerts together and help add context when things are noisy and distracting.
These tools combined help teams tackle a common incident response problem - alert fatigue. Full Service Ownership teams looking to improve the quality of their services can experiment with their SLIs and SLOs to find the work that will benefit their systems the most while also preserving their own sanity. Prevent the stress and anxiety that can arise from unexpected system failures by setting clear expectations and allowing for planned responses to potential issues based on pre-agreed norms.
Kat spent years leading technical support teams on the front lines of incident response before moving to the stage to talk about it. As Senior Manager of Developer Relations at PagerDuty, she leverages over 12 years of experience in the field to help teams communicate clearly when everything is on fire. She speaks and writes on the topics of sustainable incident management practices, cross-functional collaboration, and career development. When she's not wrangling incidents, she's wrangling two rabbits (Lupin and Ginny), a dog named Violet, and an ambitious collection of houseplants.