This session walks through a practical, end-to-end framework for evaluating AI applications in production. We start with foundations: what evals are, when to start, and why they’re a team sport across engineering, product, and domain experts. From there, we cover the common eval process — gathering improvement signals from production logs, user feedback, and human review; defining success criteria with primary metrics, tracking metrics, and guardrails; and building scorers (code-based, LLM-as-a-judge, and human review) matched to the right use case. The second half focuses on agentic AI evaluation: how to assess task success, tool accuracy, and cost for single agents, then layer on orchestration, routing, and conversation quality metrics for multi-agent and multi-turn systems. We close with remote evals — testing real agents against real dependencies without mocking — and the mindset shift that eval is not a gate but a continuous improvement loop.