Building with Agents

Evals and Benchmarks

About the track

In a world where models are released daily, how do we know when the right time is to swap models? External benchmarks give us hints, but internal, use-case-specific ones help teams make decisions on accuracy, cost and latency. This track is for ML/AI engineers looking to grow their intermediate-to-advanced eval skills.

Talks cover industry-leading benchmarks, how to build internal benchmarks, when evals fail to lead to business outcomes, and auditing benchmarks. Expect practical eval advice that will help teams move more quickly, build better products and save on costs. You’ll walk out with a blueprint for building an effective eval.

Track host

Kathryn Hume

VP AI Engineering, Vector Institute

Kathryn Hume leads AI Engineering at the Vector Institute, where she builds tools and platforms that further industry adoption of frontier AI. Prior to joining Vector, she held multiple technology leadership roles in financial services and AI, including Chief Technology Officer for Tangerine Bank, Vice President of Technology for RBC’s mobile, online banking, and direct investing businesses, and Interim Head of Borealis AI, RBC’s AI research institute. Before joining RBC, she held leadership roles at AI startups integrate.ai and Fast Forward Labs (acquired by Cloudera). Kathryn is a recognized author and public speaker, with work appearing at TED, the Harvard Business Review, and the Globe & Mail. She has served as a guest lecturer and adjunct professor — teaching courses on applied AI and responsible AI — at the Harvard Business School, Stanford, the University of Toronto, and the University of Calgary. She is a mother of two young boys, a board member at CanadaHelps active in the charity sector, and an advocate for women in technology and educational reform in the age of generative AI.

Sessions in this track

Friday, November 6th | 2:40 PM – 3:10 PM

The Illustrated Code Agent: How They Work and How to Know When They Don't

Jay Alammar

Author,

Friday, November 6th | 2:40 PM – 3:10 PM

Detecting and Classifying Contextually Relevant Personally Identifiable Information: Challenges and Big Opportunities

David Emerson

ML Scientist,

Vector Institute

Friday, November 6th | 2:05 PM – 2:35 PM

Beyond Public Benchmarks: Building Enterprise AI Evaluations That Actually Predict Production Performance

Anton Shesternev, PhD

Lead Applied AI Research Scientist,

CIBC

Friday, November 6th | 11:45 AM – 12:30 PM

Responsible AI Beyond Accuracy: Fairness, Safety, and Sustainability

Dr. Shaina Raza

Applied ML Scientist Responsible AI,

Vector institute

Friday, November 6th | 10:55 AM – 11:40 AM

From Experimentation to Continuous AI Assurance

Mohammad Danesh

Head of Data and AI,

Tangerine

Friday, November 6th | 10:55 AM – 11:40 AM

From Experimentation to Continuous AI Assurance

Kevin Yan

Senior AI Engineer,

Tangerine

Friday, November 6th | 10:55 AM – 11:40 AM

From Experimentation to Continuous AI Assurance

Moamen Abdelrazek

Principal AI Engineer,

Tangerine

Thursday, November 5th | 2:55 PM – 3:25 PM

Evaluating the Adversarial Robustness of Production AI Agents

Erfan Pirmorad

Principal AI Scientist,

Vanguard

Thursday, November 5th | 3:55 PM – 4:25 PM

Beyond the Benchmark: Evaluating AI Agents for Longitudinal Care Journeys

Aiden Arnold

VP, AI Models & Head of League Labs,

League

Friday, November 6th | 1:30 PM – 2:00 PM

Grading the Agent: How We Built Evals for Disco's MCP Server

Wendy Foster

Principal Data Scientist,

Disco

Summit (2 days).

Toronto Summit · Presented by TMLS · Nov 5–6, 2026 · RBC WaterPark Place, TorontoThe people building agents, and the people building with them – one room, on the Toronto waterfront. Two days of practitioner-curated talks on what actually ships: the stack underneath agentic systems, and the products, workflows, and agents teams are running in production right now.

Share