2026-09-10 –, Ballroom All times in US/Pacific
Our analytics team wanted to show how we're adopting AI tools, but faced a measurement problem: standard AI metrics focus on lines of code deployed to production which is not as meaningful for SQL pipelines and data work. We built a custom AI toolkit using IBM Bob and GitHub, and more importantly, created a tracking system that measured what actually matters for analytics teams. Six months later: 40-60% time savings, 50+ documented use cases, and a framework for measuring AI impact beyond code generation with the added bonus of sharing knowledge across the team as we all ramp and iterate our AI tool skills in real time.
This talk shares how an analytics engineering team adopted AI when standard metrics didn't apply. Most AI adoption tracking focuses on lines of code deployed to production but we're not shipping features, we're translating business requirements into metrics, building pipelines, dbt models, and optimizing SQL. We built a custom toolkit and obsessively tracked metrics that actually mattered: time to complete tasks, onboarding speed, documentation quality, and knowledge transfer, not lines of code generated. The result? Tools that genuinely improved our work and a measurement framework we could actually defend. Learn what metrics prove AI value for non-product teams, what failed spectacularly, and how to track AI adoption when you're not counting commits.
Solutions focused data nerd currently working on Product Analytics Engineering at IBM Hashicorp.