TMLS is Canada’s flagship summit for applied ML, AI infrastructure, and enterprise adoption.

When integrating structured data into a RAG system, engineers often default to embedding raw JSON into a vector database. The reality, however, is that this intuitive approach leads to dramatically poor retrieval performance. Modern embeddings leverage BERT architectures optimized for natural language, which struggle with the high frequency of non-alphanumeric characters found in JSON syntax.
In this session, I will break down the exact failures of embedding structured data—from tokenization and attention mechanism disruption to the mathematical liability of Mean Pooling on syntax tokens. I will then demonstrate a practical, production-ready solution: implementing a simple preprocessing step to convert structured JSON into natural language templates. Backed by empirical testing on the Amazon ESCI dataset, I will show how this straightforward architectural shift natively boosts Recall@10 by over 19% and MRR by 27%.
Note: This article was recently featured as a top weekly article on Towards Data Science, reaching over 9,000 views.

Most AI agent evaluation frameworks are built to measure capability, not adversarial robustness. The result: agents that ace benchmarks but collapse under real-world attack conditions — prompt injection, goal hijacking, tool misuse, and output trust exploitation that standard evals never surface.

Drawing on research developing game-theoretic prompt injection frameworks and zero-knowledge ML systems for high-stakes financial infrastructure at Ripple, this session presents a concrete methodology for adversarial agent evaluation that practitioners can apply to their own pipelines.

You’ll leave with a working model for mapping your agent’s attack surface across orchestration logic, tool boundaries, and downstream trust chains — and a set of design patterns for building agents that are auditable and adversarially robust by architecture, not by accident. Whether you’re building coding agents, deploying LLM workflows in production, or responsible for governing AI systems at scale, this session reframes how you think about evaluation before your next deployment.

Walmart holds the largest share of the U.S. e-commerce grocery market, where food and beverage categories generate some of the highest search traffic and, consequently, drive a substantial portion of sponsored search revenue. At this scale, even small mismatches between user intent and retrieved products can lead to significant losses in both user engagement and monetization. Yet, understanding user intent in grocery search is inherently challenging. Queries are typically short, ambiguous, and highly diverse, often underspecifying critical preferences.

For example, a query like schar white bread implicitly encodes a gluten-free preference through brand association, while queries such as chickpea pasta or oatmilk reflect underlying dietary preferences like gluten-free, plant based, or lactose-free alternatives. Failing to capture these signals results in retrieving products that might be semantically similar but misaligned with the user’s true needs.

From the advertiser’s perspective, many products are explicitly designed to target specific intents—such as dietary preferences or size variants—and must be surfaced at the right moment to be effective. For example, a brand like Quest Nutrition, which sells high-protein, low-sugar snacks, wants its products to appear for queries like protein bars, low carb snacks, or keto snacks, even when these attributes might not be explicitly stated in the product title text. When retrieval systems fail to capture these intent signals, relevant products are not shown to the right users at the right time. From an advertiser’s perspective, this means their products are missing high-intent opportunities where conversion is most likely. Over time, this leads to lower returns on ad spend, reduced trust in the platform, and potential advertiser attrition. Losing advertisers directly translates to a loss in advertising revenue and weakens the overall sponsored search ecosystem. This challenge is further amplified in sponsored search, where only a limited number of ad slots are available, making precise relevance essential. Thus, we propose INSPIRE (Intent-aware Neural Sponsored Product Retrieval for E-commerce), an intent aware retrieval framework for sponsored search that leverages structured intent signals to better align user queries with relevant food and beverage products. INSPIRE represents intent as a set of structured, multi-dimensional attributes derived from both user queries and product content, capturing explicit signals (e.g., brand, flavor) as well as implicit preferences (e.g., dietary constraints, cuisine types) that are often not directly expressed in queries.

We develop a weakly supervised intent learning pipeline, where a large language model serves as a teacher to generate structured intent annotations from product titles and descriptions. We then distill these annotations by using them to finetune a lightweight student LLM model through LoRA based supervised finetuning (LoRA-SFT) that predicts intent attributes—such as brand, flavor, dietary preference, ingredient, product subtype, and cuisine type—at Walmart catalog scale. We then introduce an intent-augmented dense retrieval framework, where predicted intents are incorporated into query and product representations within a bi-encoder, enabling more precise matching between queries and sponsored products. To support real-world usage, we deploy the system as a scalable inference service. The distilled student model is served via a high-throughput API powered by vLLM, enabling efficient intent prediction over large product catalogs with low latency. This design ensures that
intent-aware retrieval can be applied in production settings while maintaining efficiency and scalability.

This session walks through a practical, end-to-end framework for evaluating AI applications in production. We start with foundations: what evals are, when to start, and why they’re a team sport across engineering, product, and domain experts. From there, we cover the common eval process — gathering improvement signals from production logs, user feedback, and human review; defining success criteria with primary metrics, tracking metrics, and guardrails; and building scorers (code-based, LLM-as-a-judge, and human review) matched to the right use case. The second half focuses on agentic AI evaluation: how to assess task success, tool accuracy, and cost for single agents, then layer on orchestration, routing, and conversation quality metrics for multi-agent and multi-turn systems. We close with remote evals — testing real agents against real dependencies without mocking — and the mindset shift that eval is not a gate but a continuous improvement loop.

To address the critical gap in privacy risk assessment, we introduce PANORAMA (Profile-based Assemblage for Naturalistic Online Representation and Attribute Memorization Analysis). PANORAMA is a large-scale, fully synthetic text corpus containing 384,789 samples derived from 9,674 internally consistent synthetic human profiles. Generated using constrained selection and reasoning LLMs, the dataset spans six distinct online modalities, including social media posts, forum discussions, reviews, and marketplace listings. This session will explore how PANORAMA accurately emulates the naturalistic distribution and variety of sensitive data, enabling researchers to systematically study PII memorization, conduct rigorous model auditing, and benchmark privacy-preserving techniques without exposing real user data.

Many coding agents fail in production not because of model limitations, but because the surrounding harness is poorly engineered. The harness, which governs how agents retrieve context, manage state, and interact with tools, often introduces failure modes such as irrelevant context, inconsistent memory, and brittle execution.

This session addresses the practical gap between model capability and system reliability. Practitioners frequently encounter agents that perform well in demos but degrade in real workflows due to issues like context overload, uncontrolled state growth, and unstructured tool use. These problems lead to higher costs, hallucinations, and unpredictable behavior.

We focus on the core technical challenge of harness design: how to structure retrieval, memory, and execution loops so that agents remain grounded, efficient, and reliable over longer tasks. By making these design choices explicit, this session helps practitioners move from prototype agents to production-grade systems that behave consistently under real-world constraints.

Agentic AI systems are rapidly evolving from experimental prototypes to enterprise-critical applications, yet most organizations struggle to scale them reliably in production. The challenge is not just about increasing compute, but managing orchestration complexity, controlling costs, and ensuring resilience across distributed cloud environments.

This session explores how MCP (Multi-Cloud Practices) can be leveraged to address these challenges and enable scalable, production-grade agentic AI systems. We will dive into architectural patterns, intelligent orchestration strategies, and cost optimization techniques that go beyond brute-force scaling.

Attendees will gain practical insights into designing resilient, efficient, and enterprise-ready AI systems, along with real-world learnings on how to scale agentic workflows across cloud environments while maintaining performance and cost efficiency.

Stateless autonomous agents in production typically stall at a 44% task success rate due to repeated API failures, a pattern of relying on ephemeral context windows instead of persistent learning.

To close this gap, we present Agentic Memory, a reflection-episodic memory architecture that combines vector storage, automated reflection loops, and heuristic extraction to enable continuous agent learning without model fine-tuning. Across diverse simulated enterprise workflows including IT incident response and data pipeline orchestration, Agentic Memory achieves task completion rates ranging from 85% to 95%, with peak performance (93.3%) in complex, multi-step scenarios.

The method outperforms five standard stateless and naive-RAG agent baselines across all evaluation scenarios. Our background “Critic” process extracts and indexes failure heuristics with near-zero latency overhead (latency penalty ≤ 0.05s) while significantly reducing the API error rate compared to baseline approaches. The framework integrates episodic vector stores, actor-critic reflection patterns, and shared experience banks to address the amnesia and reliability gap in autonomous agent operations.

All organizations hold valuable information that originates as or can be translated into text. AI models extract semantic meaning and store the results in large-scale vector databases, which LLMs can then leverage to derive actionable insight. The desired goal is to scale with large enterprise datasets without compromising analytical integrity. This session presents a workflow that automatically segments text data, identifies patterns, and generates insight to drive business outcomes. Two case studies will be discussed: Database Query Optimization and Automated Topic Detection — two very different problems solved with similar analytical techniques operating at massive scale.

As AI adoption accelerates across sectors, many organizations are discovering a persistent gap: technically strong models often fail to translate into measurable real-world value. In practice, AI initiatives frequently stall not because the models underperform, but because economic impact, operational fit, and long-term sustainability were never rigorously evaluated.

This executive-focused talk introduces a practical framework for embedding economic evaluation throughout the AI lifecycle, from business problem definition and data acquisition through deployment, monitoring, and scale decisions. Moving beyond traditional model metrics, the session outlines how organizations can systematically assess costs, benefits, risks, and time horizons to determine whether an AI system is truly worth building and sustaining.

Drawing on applied experience supporting AI deployments across healthcare, finance, and public sector contexts, the talk presents:

• A structured definition of economic evaluation for AI
• The “evaluation stack” for multi-dimensional value assessment
• Lifecycle checkpoints where value is created, eroded, or misestimated
• Practical tools including ROI, cost-benefit, cost-effectiveness, and budget impact analysis
• A real-world triage AI case illustrating how value must be validated continuously—not assumed

Designed for executive and senior technical leaders, this session provides a clear, actionable lens to move AI initiatives beyond experimentation toward accountable, economically defensible deployment. Attendees will leave with concrete strategies to forecast value earlier, avoid common failure patterns, and make more confident scale-or-retire decisions for AI investments.

Every agent deployment has a postmortem — or it should. Across 120+ production workflows in healthcare, financial services, and global telecommunications, the pattern behind both the failures and the survivors is consistent: the most expensive problems weren’t technical, they were organizational.

This talk examines the hidden variable that determines whether production AI systems live or die in regulated environments: the organizational readiness to close the meaning gap between what an agent outputs and what a human actually needs to act responsibly — and to build enough trust to keep that system online after the first anomaly.

LLMs optimize for statistical similarity; humans require meaning. “”””Top 10,”””” “”””Best 10,”””” and “”””Highest 10″””” can cluster identically in vector space, yet imply completely different decisions for the person on the other end. Multiply that LLM Similarity Trap across every handoff between agent output and human judgment, and you get the most expensive unsolved problem in enterprise AI — one no model update will fix.

This is not a framework talk. It is a what-actually-happened talk grounded in real incidents: an executive who shut down eleven weeks of production gains after a single anomaly because no one gave her a vocabulary for “”””91% accuracy””””; a frontline team that quietly routed around a bot for six months because it never fit how they actually worked; and a clinical team that became the loudest internal champions for an AI system because they were involved before a single line of production code was written.

From these cases and my post-go-live experiences implementing GenAI workflows and agents, the talk distills a six-part operational playbook: the 4 Modes of Human-Agent Collaboration, the Agent Seniority Ladder, the Dignity Clause, the 3-Tier AI Literacy Model, the Aikido Framework for organizational resistance, and the Protocol of Interaction. Each emerged from a production failure — not a lab or a slide deck. The playbook is grounded in a four-layer architecture that connects technical human-in-the-loop design to organizational accountability — because confidence thresholds and escalation routing without named human ownership above them are just infrastructure with no one responsible for what comes out.

The audience leaves with one question: in your current agent deployment, who owns the meaning gap — and how, specifically, are you closing it?

Most teams can get an LLM workflow to look good in a demo. Far fewer have a reliable answer for what happens after that.

This talk focuses on the day 2 to day 10 problems of real-world LLM systems: how to keep evals useful as prompts, workflows, and failure modes change over time. I’ll share a practical framework for operationalizing evals through automated dataset curation, failure-mode detection, and uncertainty-aware decisioning. I’ll also cover how batching and flexible compute can make evaluation fast enough to support real developer workflows instead of becoming a bottleneck.

The session bridges familiar ML evaluation discipline with the realities of LLM systems. I’ll show how to move beyond static benchmarks, how to use failure signals to keep eval sets relevant, and how to design eval feedback loops that help teams move faster with more confidence. The goal is to give practitioners a concrete path from ad hoc prompt checks to a durable evaluation system for production LLMs.

Tabular data is ubiquitous worldwide, driving solutions for generic business problems, applied time series forecasting, and beyond. This inherent heterogeneity had hindered Tabular Foundation Models (TFMs) from rapidly generalizing to unseen datasets. In-Context Learning (ICL) offers a promising path for TFMs, enabling dynamic task adaptation without fine-tuning. Moving beyond re-purposed language models, we propose combining ICL-based retrieval with self-supervised learning to train dedicated TFMs. We evaluate real versus synthetic pre-training data, demonstrating that real data captures complex signals critical for improving downstream generalization. Incorporating this real data yields significantly faster training and superior adaptability across diverse contexts. Our resulting model, TabDPT, achieves strong performance across varied classification and regression benchmarks. Importantly, our pre-training procedure demonstrates that scaling model and data size drives consistent, power-law performance improvements. This echoes foundational scaling laws, confirming that robust, large-scale, and equitable TFMs are highly achievable. We have open-sourced our complete training and inference pipeline.

In the legal domain, moving from a successful prototype to a production-grade system is rarely a straight line. Scaling trustworthy AI and Information Retrieval solutions requires a transition from experimental algorithms, RAG, and agentic prototypes to production-grade systems with a robust delivery strategy. The challenge extends beyond technical implementation to the intersection of research uncertainty, engineering and operational constraints, team dynamics, and product alignment.

This session outlines a framework for a Technical Lead, guiding teams through this complexity, drawn from the experience of delivering high-stakes regulated AI solutions.

We will show how establishing clear metrics and progressive target values defines what ‘Good’ means to stakeholders. These targets become the shared objectives between technical teams and Product stakeholders that build trust through an iterative discovery and delivery process. Each sprint then demonstrates measurable progress beyond experimental “”””vibes.”””” These shared metrics also inform capacity-effort planning, enabling honest reprioritization when resources are constrained. This method prevents falling into the ‘Hero Trap’ of unsustainable delivery, a pressure familiar to Tech Leads.

We will reframe technical debt not as a drawback, but as a calculated liability—treating it as a high-ROI instrument for speed-to-market and as a strategic onboarding tool for new team members. Furthermore, we will address risk management through actively challenging our thinking; seeking uncomfortable opposing views, constructing honest pros/cons analyses, and stress-testing assumptions before they become costly commitments. This practice reduces cognitive biases, surfaces hidden risks early, and fosters more inclusive and psychologically safe team environments where dissent is treated as a resource rather than resistance.

Finally, the session will turn to the human side of delivery. We will cover team practices that sustain velocity: integrating SME feedback loops to iterate quickly and learn early, shielding researchers from agile ceremony fatigue, celebrating every win and learning from setbacks, and staying adaptable through ambiguity and changes. We will also address the often-invisible “glue work” that sustains production excellence, presenting original survey data from Engineering, Science, and Product teams to quantify its impact and offer methods to identify common blind spots.

Target Audience:

• Technical Leaders and Lead AI/ML Engineers
• Product Managers and Executives overseeing AI portfolios
• Engineering Managers in trustworthy or regulated domains

Long-context capabilities are becoming essential for modern LLM applications, from document understanding and agent workflows to code, RAG, and multi-step reasoning. But supporting long sequences efficiently is still one of the hardest practical challenges in large-scale model training and inference, especially when memory, bandwidth, and latency become the real bottlenecks rather than raw compute alone. In this talk, I will discuss the key systems and modeling considerations for long-context training and inference on AMD GPUs. I will cover the main sources of cost in long-sequence workloads, including KV-cache growth, attention complexity, memory movement, parallelism strategies, and kernel efficiency. I will also discuss practical approaches to make long-context workloads feasible in production and research settings, including efficient attention variants, context extension strategies, distributed training design, cache optimization, precision choices, and inference-time serving trade-offs. The talk is grounded in a practitioner-focused perspective: what actually matters when moving from promising ideas to stable, scalable implementations on AMD hardware. The goal is to provide a clear view of the design space and a practical roadmap for building efficient long-context systems on modern AMD GPU platforms.

Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process sequential audio-video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting. We introduce SONIC-O1, a comprehensive, fully human-verified benchmark spanning 13 real-world conversational domains with 4,958 annotations and demographic metadata. SONIC-O1 evaluates MLLMs on key tasks, including open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Experiments on closed- and open-source models reveal limitations. While the performance gap in MCQ accuracy between two model families is relatively small, we observe a substantial 22.6% performance difference in temporal localization between the best performing closed-source and open-source models. Performance further degrades across demographic groups, indicating persistent disparities in model behavior. Overall, SONIC-O1 provides an open evaluation suite for temporally grounded and socially robust multimodal understanding.

Modern institutional sales is low-frequency and high-stakes: a small coverage team needs early, defensible signals on which allocators (pensions/endowments/insurers) are likely to be “in market” for a new mandate before an RFP appears. This session walks through a production-ready prospect ranking system that handles missing/noisy CRM data, learns intent using proxy targets under delayed ground truth, and augments structured models with controlled LLM-assisted research from unstructured sources (e.g., mandate news, personnel changes, policy updates). I’ll cover evaluation choices for top-K ranking (precision@K, stability, and leakage traps), operational handoff to human coverage, and the feedback/monitoring loop that keeps recommendations actionable over time.

Most developers use LLM APIs like a black box: pick a model, send requests, and live with whatever performance they get. This talk argues that this is the wrong way to think about performance. Modern inference stacks implement optimizations such as prompt caching, speculative decoding, disaggregated inference, and more. But for those optimization gains to be maximized, the application needs to use the API in the right way. I will explain the key optimizations that matter, how they work at a high level, and what API users can do to fully benefit from them in practice. The focus is not on theory or provider internals. It is on helping practitioners get better real-world performance from the same LLM APIs.

Financial institutions often collect events from various touch points such as call transcripts, chatbots, branch visits, transactions etc., and often struggle to describe the ‘customer task’ or ‘paths leading to interesting outcomes’ from those. Knowing this, allows the businesses to understand intent behind user behavior and provide better services/offers and customer retention. While these discrete event sequences are not exactly NLP, they do have a vocabulary of their own along with timestamps. This talk describes how to build various white box and deep learning transformers/generative models and addresses tradeoffs across range of accuracy/ explainabiity/ inferencing complexity. This way businesses can pick and choose the models depending on regulatory and non-regulatory use cases and achieve the same objective.

Foundation models have achieved remarkable success in natural language and vision, but applying the same paradigm to structured, sequential event data — transactions, interactions, behavioural signals — introduces a distinct set of technical challenges that existing literature largely overlooks.

In this talk, we share what we learned building and productionizing a domain-specific foundation model trained on millions of heterogeneous event sequences. We dig into:

• Tokenization for non-language sequences: Event data mixes categorical fields, continuous values, and irregular timestamps. We explore representations from naive text serialization to structured entity encodings, and the surprising impact tokenization strategy has on downstream performance
• Architecture trade-offs: Head-to-head comparisons across three approaches — off-the-shelf LLM embeddings, a custom set-aware transformer, and a hybrid fine-tuned LLM — and when each breaks down
• Multi-objective training: Combining next-event prediction, next-token prediction, and contrastive learning with Matryoshka representation learning for flexible embedding dimensionality without retraining
• Temporal encoding for irregular time series: Encoding events spaced across seconds to months, unlike NLP’s uniform token positions
• Inference at scale: From a 10-day batch pipeline to sub-300ms real-time inference for millions of sequences via model distillation, context window management, and SageMaker serving
• Evaluation beyond perplexity: Why standard LM metrics fail for event prediction, and the framework we built to measure both predictive accuracy and embedding quality

Feature importance estimation is crucial for model interpretability, but traditional permutation-based methods break down when features exhibit dependencies. Standard permutation importance shuffles features independently, creating out-of-distribution samples that don’t reflect realistic data relationships—leading to unreliable and often misleading importance scores. As warned by Hooker et al. (2021), “unrestricted permutation forces extrapolation.”

This talk introduces a conditional subgroup approach for computing model-agnostic feature importance that respects feature dependencies through row and column blocking strategies. The method combines two complementary Model-X techniques that model the joint feature distribution:

1. Conditional Imputation: Using Gaussian Copula and other statistical models to replace masked features while preserving the joint feature distribution, avoiding impossible feature combinations.
2. Restricted Permutations: Partitioning samples into blocks using Random Trees Embedding, then permuting features only within similar samples to maintain local feature dependencies.

The approach uses Fraction of Variance Unexplained (FVU) as a variance-based sensitivity measure with well-defined bounds [0,1], making it comparable across problems. Unlike SHAP or standard permutation importance, this method correctly handles multicollinear features without requiring model retraining or manual feature dropping.

Everyone agrees that moving from conversational tools to autonomous, multi-agent systems is the future. But the demos lie. In production, the “Agentic Shift” is hitting a massive wall: evaluation is fundamentally broken. When engineering, legal, and product teams can’t even agree on the definition of “good,” processes fail, user trust evaporates, and compliance teams hit the brakes.

Drawing from international efforts to standardize AI system quality and conformity assessment, alongside hard lessons learned deploying autonomous agents in highly regulated industries (banking, insurance, healthcare) this session bypasses the hype to look at why AI evaluations actually fail. We will dissect the gap between human-machine teaming in theory versus reality, why logging traces isn’t enough, and how to build scenario-based evaluations that satisfy both the engineers building the agents and the governance teams protecting the enterprise.