Description

Karpenter is a newer node autoscaler that promises faster scheduling and better cost efficiency. But this kind of migration touches everything: scheduling latency, scale-down behavior, and disruption semantics across every workload. Just one of our production clusters has thousands of nodes and hundreds of services, this meant unknown unknowns were inevitable.

We needed two things: a way to collapse months of unknowns into days, and a rollout strategy where stopping felt as safe as continuing.

The approach:

First, explicit go/no-go criteria defined before starting, not discovered mid-rollout. The talk covers what those criteria looked like for an autoscaler migration, and how we got stakeholder alignment on "bail is a valid outcome."

Second, staged learning: synthetic tests for behaviors we knew were problematic, staging to remove doubts, then a canary production cluster where all the real iteration happened. Within that cluster, we used a hand-picked batch of services chosen to surface different classes of problems fast. The talk walks through how we selected that batch, and what each category was designed to catch.

Third, tiered rollout with explicit gates: expansion by service criticality, percentage-based phases, and dedicated alarms tied to our go/no-go signals. When thresholds crossed, on-call was notified and the decision to revert was obvious.

What we found:

Karpenter was supposed to be faster, but we found consolidation scenarios where it was actually slower than Cluster Autoscaler. We found workloads silently assuming pods would live "long enough." These weren't edge cases; they would have broken at scale. The selective batch surfaced these issues early, and we fixed them before the broader rollout even started. The talk covers these discoveries, how the batch selection strategy caught them, and what we did about them.

The result: double-digit percentage reductions in idle costs, and a pattern we've since reused for other high-blast-radius migrations.

Attendees Will Take Away

• Go/no-go criteria: How to define must-haves and negotiables before starting—and get stakeholder alignment that "bail" is a valid outcome
• Canary batch selection: How to pick services that surface different classes of problems fast (non-consequential baseline, critical services with aligned teams, intentional stressors)
• Gating signals: Which metrics to watch (scheduling latency P99, sustained pending pods, service disruptions) and how to tie them to batch-specific alarms—so on-call isn't guessing, just responding