- MLOps North
- Toronto Summit
- Presented by TMLS
Evals and Benchmarks
About the track
In a world where models are released daily, how do we know when the right time is to swap models? External benchmarks give us hints, but internal, use-case-specific ones help teams make decisions on accuracy, cost and latency. This track is for ML/AI engineers looking to grow their intermediate-to-advanced eval skills.
Talks cover industry-leading benchmarks, how to build internal benchmarks, when evals fail to lead to business outcomes, and auditing benchmarks. Expect practical eval advice that will help teams move more quickly, build better products and save on costs. You’ll walk out with a blueprint for building an effective eval.
Track host
VP AI Engineering, Vector Institute
See Track Lead Profile
Kathryn Hume leads AI Engineering at the Vector Institute, where she builds tools and platforms that further industry adoption of frontier AI. Prior to joining Vector, she held multiple technology leadership roles in financial services and AI, including Chief Technology Officer for Tangerine Bank, Vice President of Technology for RBC’s mobile, online banking, and direct investing businesses, and Interim Head of Borealis AI, RBC’s AI research institute. Before joining RBC, she held leadership roles at AI startups integrate.ai and Fast Forward Labs (acquired by Cloudera). Kathryn is a recognized author and public speaker, with work appearing at TED, the Harvard Business Review, and the Globe & Mail. She has served as a guest lecturer and adjunct professor — teaching courses on applied AI and responsible AI — at the Harvard Business School, Stanford, the University of Toronto, and the University of Calgary. She is a mother of two young boys, a board member at CanadaHelps active in the charity sector, and an advocate for women in technology and educational reform in the age of generative AI.
Sessions in this track
Friday, November 6th | 2:40 PM – 3:10 PM
The Illustrated Code Agent: How They Work and How to Know When They Don't
Author,
Friday, November 6th | 2:40 PM – 3:10 PM
Detecting and Classifying Contextually Relevant Personally Identifiable Information: Challenges and Big Opportunities
ML Scientist,
Vector Institute
Friday, November 6th | 2:05 PM – 2:35 PM
Beyond Public Benchmarks: Building Enterprise AI Evaluations That Actually Predict Production Performance
Lead Applied AI Research Scientist,
CIBC
Friday, November 6th | 11:45 AM – 12:30 PM
Responsible AI Beyond Accuracy: Fairness, Safety, and Sustainability
Applied ML Scientist Responsible AI,
Vector institute
Friday, November 6th | 10:55 AM – 11:40 AM
From Experimentation to Continuous AI Assurance
Head of Data and AI,
Tangerine
Friday, November 6th | 10:55 AM – 11:40 AM
From Experimentation to Continuous AI Assurance
Senior AI Engineer,
Tangerine
Friday, November 6th | 10:55 AM – 11:40 AM
From Experimentation to Continuous AI Assurance
Principal AI Engineer,
Tangerine
Thursday, November 5th | 2:55 PM – 3:25 PM
Evaluating the Adversarial Robustness of Production AI Agents
Principal AI Scientist,
Vanguard
Thursday, November 5th | 3:55 PM – 4:25 PM
Beyond the Benchmark: Evaluating AI Agents for Longitudinal Care Journeys
VP, AI Models & Head of League Labs,
League
Friday, November 6th | 1:30 PM – 2:00 PM
Grading the Agent: How We Built Evals for Disco's MCP Server
Principal Data Scientist,
Disco
Toronto Summit · Presented by TMLS · Nov 5–6, 2026 · RBC WaterPark Place, TorontoThe people building agents, and the people building with them – one room, on the Toronto waterfront. Two days of practitioner-curated talks on what actually ships: the stack underneath agentic systems, and the products, workflows, and agents teams are running in production right now.