TMLS is Canada’s flagship summit for applied ML, AI infrastructure, and enterprise adoption.

When integrating structured data into a RAG system, engineers often default to embedding raw JSON into a vector database. The reality, however, is that this intuitive approach leads to dramatically poor retrieval performance. Modern embeddings leverage BERT architectures optimized for natural language, which struggle with the high frequency of non-alphanumeric characters found in JSON syntax.
In this session, I will break down the exact failures of embedding structured data—from tokenization and attention mechanism disruption to the mathematical liability of Mean Pooling on syntax tokens. I will then demonstrate a practical, production-ready solution: implementing a simple preprocessing step to convert structured JSON into natural language templates. Backed by empirical testing on the Amazon ESCI dataset, I will show how this straightforward architectural shift natively boosts Recall@10 by over 19% and MRR by 27%.
Note: This article was recently featured as a top weekly article on Towards Data Science, reaching over 9,000 views.

Most AI agent evaluation frameworks are built to measure capability, not adversarial robustness. The result: agents that ace benchmarks but collapse under real-world attack conditions — prompt injection, goal hijacking, tool misuse, and output trust exploitation that standard evals never surface.

Drawing on research developing game-theoretic prompt injection frameworks and zero-knowledge ML systems for high-stakes financial infrastructure at Ripple, this session presents a concrete methodology for adversarial agent evaluation that practitioners can apply to their own pipelines.

You’ll leave with a working model for mapping your agent’s attack surface across orchestration logic, tool boundaries, and downstream trust chains — and a set of design patterns for building agents that are auditable and adversarially robust by architecture, not by accident. Whether you’re building coding agents, deploying LLM workflows in production, or responsible for governing AI systems at scale, this session reframes how you think about evaluation before your next deployment.

Walmart holds the largest share of the U.S. e-commerce grocery market, where food and beverage categories generate some of the highest search traffic and, consequently, drive a substantial portion of sponsored search revenue. At this scale, even small mismatches between user intent and retrieved products can lead to significant losses in both user engagement and monetization. Yet, understanding user intent in grocery search is inherently challenging. Queries are typically short, ambiguous, and highly diverse, often underspecifying critical preferences.

For example, a query like schar white bread implicitly encodes a gluten-free preference through brand association, while queries such as chickpea pasta or oatmilk reflect underlying dietary preferences like gluten-free, plant based, or lactose-free alternatives. Failing to capture these signals results in retrieving products that might be semantically similar but misaligned with the user’s true needs.

From the advertiser’s perspective, many products are explicitly designed to target specific intents—such as dietary preferences or size variants—and must be surfaced at the right moment to be effective. For example, a brand like Quest Nutrition, which sells high-protein, low-sugar snacks, wants its products to appear for queries like protein bars, low carb snacks, or keto snacks, even when these attributes might not be explicitly stated in the product title text. When retrieval systems fail to capture these intent signals, relevant products are not shown to the right users at the right time. From an advertiser’s perspective, this means their products are missing high-intent opportunities where conversion is most likely. Over time, this leads to lower returns on ad spend, reduced trust in the platform, and potential advertiser attrition. Losing advertisers directly translates to a loss in advertising revenue and weakens the overall sponsored search ecosystem. This challenge is further amplified in sponsored search, where only a limited number of ad slots are available, making precise relevance essential. Thus, we propose INSPIRE (Intent-aware Neural Sponsored Product Retrieval for E-commerce), an intent aware retrieval framework for sponsored search that leverages structured intent signals to better align user queries with relevant food and beverage products. INSPIRE represents intent as a set of structured, multi-dimensional attributes derived from both user queries and product content, capturing explicit signals (e.g., brand, flavor) as well as implicit preferences (e.g., dietary constraints, cuisine types) that are often not directly expressed in queries.

We develop a weakly supervised intent learning pipeline, where a large language model serves as a teacher to generate structured intent annotations from product titles and descriptions. We then distill these annotations by using them to finetune a lightweight student LLM model through LoRA based supervised finetuning (LoRA-SFT) that predicts intent attributes—such as brand, flavor, dietary preference, ingredient, product subtype, and cuisine type—at Walmart catalog scale. We then introduce an intent-augmented dense retrieval framework, where predicted intents are incorporated into query and product representations within a bi-encoder, enabling more precise matching between queries and sponsored products. To support real-world usage, we deploy the system as a scalable inference service. The distilled student model is served via a high-throughput API powered by vLLM, enabling efficient intent prediction over large product catalogs with low latency. This design ensures that
intent-aware retrieval can be applied in production settings while maintaining efficiency and scalability.

This session walks through a practical, end-to-end framework for evaluating AI applications in production. We start with foundations: what evals are, when to start, and why they’re a team sport across engineering, product, and domain experts. From there, we cover the common eval process — gathering improvement signals from production logs, user feedback, and human review; defining success criteria with primary metrics, tracking metrics, and guardrails; and building scorers (code-based, LLM-as-a-judge, and human review) matched to the right use case. The second half focuses on agentic AI evaluation: how to assess task success, tool accuracy, and cost for single agents, then layer on orchestration, routing, and conversation quality metrics for multi-agent and multi-turn systems. We close with remote evals — testing real agents against real dependencies without mocking — and the mindset shift that eval is not a gate but a continuous improvement loop.

To address the critical gap in privacy risk assessment, we introduce PANORAMA (Profile-based Assemblage for Naturalistic Online Representation and Attribute Memorization Analysis). PANORAMA is a large-scale, fully synthetic text corpus containing 384,789 samples derived from 9,674 internally consistent synthetic human profiles. Generated using constrained selection and reasoning LLMs, the dataset spans six distinct online modalities, including social media posts, forum discussions, reviews, and marketplace listings. This session will explore how PANORAMA accurately emulates the naturalistic distribution and variety of sensitive data, enabling researchers to systematically study PII memorization, conduct rigorous model auditing, and benchmark privacy-preserving techniques without exposing real user data.

Stateless autonomous agents in production typically stall at a 44% task success rate due to repeated API failures, a pattern of relying on ephemeral context windows instead of persistent learning.

To close this gap, we present Agentic Memory, a reflection-episodic memory architecture that combines vector storage, automated reflection loops, and heuristic extraction to enable continuous agent learning without model fine-tuning. Across diverse simulated enterprise workflows including IT incident response and data pipeline orchestration, Agentic Memory achieves task completion rates ranging from 85% to 95%, with peak performance (93.3%) in complex, multi-step scenarios.

The method outperforms five standard stateless and naive-RAG agent baselines across all evaluation scenarios. Our background “Critic” process extracts and indexes failure heuristics with near-zero latency overhead (latency penalty ≤ 0.05s) while significantly reducing the API error rate compared to baseline approaches. The framework integrates episodic vector stores, actor-critic reflection patterns, and shared experience banks to address the amnesia and reliability gap in autonomous agent operations.

Most enterprise AI architectures put the “intelligence” in a thick agent layer: elaborate tool graphs, custom planners, and hand‑rolled orchestration logic. That feels robust—until the next model, framework, or interaction pattern ships and you’re stuck rewriting the whole thing. In this talk, Zeke Miller shares how Workday’s Agent Factory team flipped that mindset: treating agents as a thin, rewritable veneer on top of a thick foundation of systems of record, systems of action, and governance. Using real examples from conversational analytics and HR/finance workflows, he’ll show how a single well‑designed connector plus strong evals outperformed months of bespoke agent engineering, how they use evals as an executable contract between users and systems, and how they bake security and auditability in from day one. Attendees leave with a concrete mental model and patterns for building agentic systems that can change quickly without breaking the parts of the business that can’t.

Detecting anomalies across thousands of high-dimensional time-series streams is hard; explaining them to the engineer who has to act is harder. In this session I’ll walk through the system we built and deployed at TELUS to close the gap end-to-end: a hybrid multi-head LSTM that trains a dedicated encoder per KPI, paired with an adaptive threshold that scales by hour-of-day to suppress the false positives static thresholds produce during predictable daily cycles.

Detection is half the work. The session’s second half is the explainability and resolution layer: we isolate the sector counters and individual cells driving each anomaly, an LLM turns those signals into a diagnosed cause and a ranked list of recommended actions from a YAML-defined playbook, and an outcome store records what the engineer actually did and whether the KPI recovered. Those outcomes feed back as Wilson-smoothed action win-rates that re-rank future recommendations — closing the learning loop.

The architecture generalizes to any domain where rare events hide in correlated time-series — fraud, observability, IoT.

Getting LLMs to reliably call tools in production is not just a prompting problem but also a training problem but most practitioners lack a principled way to measure progress. This talk uses tau2-bench, a rigorous tool-calling benchmark, as the backbone for a complete fine-tuning workflow: generating training scenarios, running supervised fine-tuning, and applying reinforcement learning to push past the ceiling of imitation. The result is a model measurably better at domain-specific tool use with concrete before/after numbers. Practitioners leave with a reusable approach: use a structured benchmark to drive your fine-tuning loop, not just to evaluate at the end.

Multi-hop question answering over financial disclosures is often constrained more by evidence retrieval than by reasoning capability. Relevant facts are dispersed across filings, fiscal periods, and peer firms, and noisy long-context inputs degrade reliability even for strong reasoning models. We introduce FinReflectKG–MultiHop, a large-scale benchmark grounded in a temporally indexed financial knowledge graph derived from SEC 10-K filings, containing multi-hop QA pairs spanning intra-document, inter-year, and cross-company reasoning regimes. Questions are generated from statistically informed 2–3 hop typed motifs and paired with provenance-linked evidence to enable controlled evaluation of evidence structure. Across multiple open-weight reasoning LLMs and six structured evidence protocols, KG-linked provenance consistently improves correctness while reducing token usage by over 70% relative to text-window and semantic retrieval baselines. Our results demonstrate that reasoning reliability in finance is fundamentally governed by evidence composition and structure, and that model scaling alone cannot compensate for poorly organized retrieval contexts.

Most RAG discussions start and end with vector embeddings. That makes sense because vector search is approachable, fast to prototype, and widely supported. But semantic similarity is not the same thing as answer retrieval. When teams rely on embeddings as the default for every use case, they often end up with systems that sound convincing while returning weak, incomplete, or confidently incorrect answers. This talk reframes retrieval as the real design decision in RAG, not a backend detail.

We will walk through the major retrieval options at a high level, including vector, graph, and BM25 approaches, and explain where each one fits. Then we will show why hybrid designs, such as Vector + Graph and Vector + BM25, often produce stronger results by combining semantic context with stronger grounding and greater precision. The goal is to give AI engineers a practical mental model for choosing a retrieval approach based on the shape of their data and the kinds of answers they need, rather than defaulting to embeddings because everyone else did.

Large Language Models (LLMs) and Large Reasoning Models (LRMs) hold transformative potential for high-stakes domains such as finance and law, but their tendency to hallucinate poses a critical reliability risk. This talk explores detection strategies, including uncertainty estimation, reasoning consistency checks, and factual validation, paired with mitigation approaches such as knowledge grounding, prompt engineering, and confidence calibration. What sets this discussion apart is the emphasis on root cause awareness: by categorizing hallucination sources into model, data, and context-related factors, detection and mitigation strategies can be precisely matched to the underlying cause rather than applied as generic fixes.

Most large AI organizations have a data problem that isn’t what they think it is. The problem isn’t missing data — it’s data that exists, was built deliberately, is maintained by real people, and still isn’t being reused. It sits in a pipeline nobody else can find.

This talk examines what happens when a large-scale AI organization shifts from a project-centric to a product-centric operating model — and discovers that building data products doesn’t automatically mean anyone benefits from them. Across a portfolio of more than 140 data products supporting five distinct AI strategies, the patterns are consistent: data scientists rebuild ingestion pipelines for data that already exists, reusable frameworks go undiscovered by the teams who need them most, and the inventory grows while the acceleration effect of reuse does not.

This is not a clean success story. We’ll walk through what the product-centric model solved, what it didn’t, and what the discovery and trust gap actually costs — in terms practitioners recognize: the project that started from scratch because no one knew a shared datamart existed; the parser framework that sat idle while three teams built their own. We’ll then cover what it takes to close the gap: building a searchable, unified context layer that makes data products findable, evaluable, and reusable without requiring every team to know what every other team built.

Practitioners will leave with a diagnostic framework: how to distinguish a discovery problem from a data quality problem, the leading indicators that your organization has an invisible asset layer, and the ordering of interventions that helps — starting with discoverability, then trust signals, then governance.