TMLS is Canada’s flagship summit for applied ML, AI infrastructure, and enterprise adoption.

AI agents are moving rapidly from experimental prototypes to production systems embedded in critical business workflows. In regulated environments such as capital markets, deploying agents requires more than model performance. It requires governance, reliability, human oversight, and a clear path to measurable value.

Feature importance estimation is crucial for model interpretability, but traditional permutation-based methods break down when features exhibit dependencies. Standard permutation importance shuffles features independently, creating out-of-distribution samples that don’t reflect realistic data relationships—leading to unreliable and often misleading importance scores. As warned by Hooker et al. (2021), “unrestricted permutation forces extrapolation.”

This talk introduces a conditional subgroup approach for computing model-agnostic feature importance that respects feature dependencies through row and column blocking strategies. The method combines two complementary Model-X techniques that model the joint feature distribution:

1. Conditional Imputation: Using Gaussian Copula and other statistical models to replace masked features while preserving the joint feature distribution, avoiding impossible feature combinations.
2. Restricted Permutations: Partitioning samples into blocks using Random Trees Embedding, then permuting features only within similar samples to maintain local feature dependencies.

The approach uses Fraction of Variance Unexplained (FVU) as a variance-based sensitivity measure with well-defined bounds [0,1], making it comparable across problems. Unlike SHAP or standard permutation importance, this method correctly handles multicollinear features without requiring model retraining or manual feature dropping.

Foundation models have achieved remarkable success in natural language and vision, but applying the same paradigm to structured, sequential event data — transactions, interactions, behavioural signals — introduces a distinct set of technical challenges that existing literature largely overlooks.

In this talk, we share what we learned building and productionizing a domain-specific foundation model trained on millions of heterogeneous event sequences. We dig into:

• Tokenization for non-language sequences: Event data mixes categorical fields, continuous values, and irregular timestamps. We explore representations from naive text serialization to structured entity encodings, and the surprising impact tokenization strategy has on downstream performance
• Architecture trade-offs: Head-to-head comparisons across three approaches — off-the-shelf LLM embeddings, a custom set-aware transformer, and a hybrid fine-tuned LLM — and when each breaks down
• Multi-objective training: Combining next-event prediction, next-token prediction, and contrastive learning with Matryoshka representation learning for flexible embedding dimensionality without retraining
• Temporal encoding for irregular time series: Encoding events spaced across seconds to months, unlike NLP’s uniform token positions
• Inference at scale: From a 10-day batch pipeline to sub-300ms real-time inference for millions of sequences via model distillation, context window management, and SageMaker serving
• Evaluation beyond perplexity: Why standard LM metrics fail for event prediction, and the framework we built to measure both predictive accuracy and embedding quality

Foundation models have achieved remarkable success in natural language and vision, but applying the same paradigm to structured, sequential event data — transactions, interactions, behavioural signals — introduces a distinct set of technical challenges that existing literature largely overlooks.

In this talk, we share what we learned building and productionizing a domain-specific foundation model trained on millions of heterogeneous event sequences. We dig into:

• Tokenization for non-language sequences: Event data mixes categorical fields, continuous values, and irregular timestamps. We explore representations from naive text serialization to structured entity encodings, and the surprising impact tokenization strategy has on downstream performance
• Architecture trade-offs: Head-to-head comparisons across three approaches — off-the-shelf LLM embeddings, a custom set-aware transformer, and a hybrid fine-tuned LLM — and when each breaks down
• Multi-objective training: Combining next-event prediction, next-token prediction, and contrastive learning with Matryoshka representation learning for flexible embedding dimensionality without retraining
• Temporal encoding for irregular time series: Encoding events spaced across seconds to months, unlike NLP’s uniform token positions
• Inference at scale: From a 10-day batch pipeline to sub-300ms real-time inference for millions of sequences via model distillation, context window management, and SageMaker serving
• Evaluation beyond perplexity: Why standard LM metrics fail for event prediction, and the framework we built to measure both predictive accuracy and embedding quality

Most enterprise AI architectures put the “intelligence” in a thick agent layer: elaborate tool graphs, custom planners, and hand‑rolled orchestration logic. That feels robust—until the next model, framework, or interaction pattern ships and you’re stuck rewriting the whole thing. In this talk, Zeke Miller shares how Workday’s Agent Factory team flipped that mindset: treating agents as a thin, rewritable veneer on top of a thick foundation of systems of record, systems of action, and governance. Using real examples from conversational analytics and HR/finance workflows, he’ll show how a single well‑designed connector plus strong evals outperformed months of bespoke agent engineering, how they use evals as an executable contract between users and systems, and how they bake security and auditability in from day one. Attendees leave with a concrete mental model and patterns for building agentic systems that can change quickly without breaking the parts of the business that can’t.

While recent advances in LLMs emphasize improved model capabilities, many systems fail to scale in real-world production settings. Beyond a certain point, adding GPUs or data yields diminishing returns: training stops scaling efficiently, hardware remains underutilized, and inference latency is dominated by system constraints rather than compute. These failures are often silent, poorly documented, and difficult to diagnose in distributed environments.

In this talk, we share lessons from building enterprise-scale domain LLM systems, focusing on the system-level bottlenecks that limit scaling in practice. We examine failure modes across distributed training and inference—including communication overhead, pipeline imbalance, numerical instability during training as well as memory-bound decoding, KV cache growth, and throughput–latency tradeoffs at inference—and show how they manifest in production systems.

Rather than introducing new modeling techniques, this session presents a practical, symptom-driven approach to debugging: identifying failure patterns, tracing their root causes, and applying targeted mitigations. The key takeaway is that scaling LLMs is fundamentally a systems problem, and attendees will leave with a concrete framework to diagnose bottlenecks and make better design decisions when moving from prototype to production.

Sprout is a Toronto nonprofit operating as an AI- and data-first organization that works alongside urban Canadian communities and grassroots organizations, providing data tools, local insights, and storytelling support that reflects community resilience building work already happening on the ground. With a core team of three, we built and run the Multilingual Climate Chatbot (MLCC) — a production RAG system serving 200+ languages to communities that don’t speak English or French as their first language across Toronto. We did it on a near-zero budget, under real infrastructure constraints, with no option to optimize later. And we did it without compromising on our values: every model and infrastructure choice was evaluated not just on cost and quality, but on environmental footprint. This talk covers four things: team design as architecture, responsible model selection under constraint, what broke in production, and a retrieval architecture built from failure. Every design decision traces back to a constraint the team couldn’t ignore: budget, latency, language quality, team size, or environmental responsibility. Infrastructure cost: $26/month. Languages served: 200+. Team size: 3.

Retrieval-augmented generation (RAG) enables generative AI models to extract accurate facts from external unstructured data sources. For structured data, RAG can be further enhanced with function (tool) calls to query databases. This paper presents an industrial case study of implementing RAG in the call center of a large financial institution. The study outlines the architecture and practical lessons learned from building a scalable RAG deployment. It also introduces enhancements for retrieving facts from structured data sources using data embeddings, achieving both low latency and high reliability. Our optimized production application demonstrates an average response time of only 7.33 seconds. Additionally, the paper compares various open‑source and closed‑source models for answer generation in an industrial environment. This paper is published in the prestigious NAACL (North American Chapter of the Association for Computational Linguistics) conference: https://aclanthology.org/2025.naacl-industry.48/

In the end, we are going to discuss how Skills based agentic application design impacts the agentic architecture for Retrieval augmented generation technique.

The session explores the evolving landscape of human AI coordination, focusing on the Progressive Trust Framework, a structured approach to building, calibrating, and scaling trust between humans and AI systems. As AI transforms business operations, organizations must navigate the inherent uncertainty in AI outcomes, characterized by aleatoric and epistemic factors, and ensure confidence scores are properly calibrated.

Three stages of human-AI interaction are detailed: Human-in-the-Loop (HITL), Human-on-the-Loop (HOTL), and Agentic AI, each with distinct roles, controls, and risk profiles. Technical guardrails, performance monitoring, and feedback loops enable safe progression toward greater AI autonomy,

The session bridges classic and modern AI paradigms, offering actionable strategies for technical leaders to advance trustworthy automation in complex enterprise environments.

The session explores the evolving landscape of human AI coordination, focusing on the Progressive Trust Framework, a structured approach to building, calibrating, and scaling trust between humans and AI systems. As AI transforms business operations, organizations must navigate the inherent uncertainty in AI outcomes, characterized by aleatoric and epistemic factors, and ensure confidence scores are properly calibrated.

Three stages of human-AI interaction are detailed: Human-in-the-Loop (HITL), Human-on-the-Loop (HOTL), and Agentic AI, each with distinct roles, controls, and risk profiles. Technical guardrails, performance monitoring, and feedback loops enable safe progression toward greater AI autonomy,

The session bridges classic and modern AI paradigms, offering actionable strategies for technical leaders to advance trustworthy automation in complex enterprise environments.

This virtual talk will introduce the 01 Quantum AI Marketplace concept and demonstrate an example of an FHE-encrypted decision tree. The demo will show a client logging in with private data, encrypting that data using fully homomorphic encryption, sending it to a model owner, and receiving encrypted inference results.

Decision trees are a useful starting point for encrypted inference because they are deterministic and interpretable: for a given input, the model follows a clear path to a prediction. In an FHE setting, however, that path cannot be followed with normal plaintext branching. Instead, the tree logic is transformed into encrypted comparisons, path scoring, and leaf selection, allowing the model owner to evaluate the decision tree without accessing the client’s raw data.

The client then decrypts the result and obtains the final prediction.
The second part of the talk will explain why FHE development is fundamentally different from regular AI software development. Unlike traditional applications, FHE systems cannot rely on normal branching over private data. Comparisons are difficult and often approximate, algorithms must be redesigned as mathematical circuits, and performance depends heavily on packing, rotations, multiplicative depth, and parameter selection.

Attendees will leave with a practical understanding of how encrypted inference works, why it matters for sensitive-data AI use cases, and what engineering challenges must be solved to make FHE-based AI systems production-ready. This session is intended for AI leaders, applied ML engineers, privacy engineers, and technical decision-makers exploring secure AI deployment models.

This virtual talk will introduce the 01 Quantum AI Marketplace concept and demonstrate an example of an FHE-encrypted decision tree. The demo will show a client logging in with private data, encrypting that data using fully homomorphic encryption, sending it to a model owner, and receiving encrypted inference results.

Decision trees are a useful starting point for encrypted inference because they are deterministic and interpretable: for a given input, the model follows a clear path to a prediction. In an FHE setting, however, that path cannot be followed with normal plaintext branching. Instead, the tree logic is transformed into encrypted comparisons, path scoring, and leaf selection, allowing the model owner to evaluate the decision tree without accessing the client’s raw data.

The client then decrypts the result and obtains the final prediction.
The second part of the talk will explain why FHE development is fundamentally different from regular AI software development. Unlike traditional applications, FHE systems cannot rely on normal branching over private data. Comparisons are difficult and often approximate, algorithms must be redesigned as mathematical circuits, and performance depends heavily on packing, rotations, multiplicative depth, and parameter selection.

Attendees will leave with a practical understanding of how encrypted inference works, why it matters for sensitive-data AI use cases, and what engineering challenges must be solved to make FHE-based AI systems production-ready. This session is intended for AI leaders, applied ML engineers, privacy engineers, and technical decision-makers exploring secure AI deployment models.