Automated Evals for LLM Agents

In this live demo, I’ll show how we built an automated evaluation system (“evals”) to test AI agents powered by LLMs in production. We’ll quickly walk through our setup—how we structured test datasets, automated stability checks, handled edge cases, and implemented reproducible quality benchmarks. I’ll run a live eval demo from the terminal, showcasing how we detect regressions and ensure consistent performance in our AI agents.