2024-09-28 –, Cloud Native
We will explore how to horizontally scale Kubernetes workloads based on metrics other than CPU and Memory. We start by exploring how HPA works in K8S and how we can scale based on metrics like Kafka Lag or traffic requests etc. We would explore two options for achieving this - Prometheus Adapter and KEDA .
We will start with quick intro of how Horizontal Pod Autoscaler works in K8S, to set context. Post this we go over metrics API in Kubernetes and discuss about how to use Prometheus Adapter to use Kafka Lag metrics (or any metric for that matter) to scale our workloads.
We conclude by going over features of KEDA (Kubernetes Event-driven Autoscaling) and how to achieve Kafka Lag based autoscaling in a declarative way using CR/CRDs in KEDA.
Software Engineer with 12+ years of experience, working on Infrastructure, distributed systems and databases. Leading infrastructure team at Rapido handling a scale (150K rps at edge), multiple self managed databases (mongodb/redis/kafka/cassandra) with 100s of TB of data. Prior to this was working as a Software Consultant at ThoughtWorks.