Foundation models have achieved remarkable success in natural language and vision, but applying the same paradigm to structured, sequential event data — transactions, interactions, behavioural signals — introduces a distinct set of technical challenges that existing literature largely overlooks.

In this talk, we share what we learned building and productionizing a domain-specific foundation model trained on millions of heterogeneous event sequences. We dig into:

• Tokenization for non-language sequences: Event data mixes categorical fields, continuous values, and irregular timestamps. We explore representations from naive text serialization to structured entity encodings, and the surprising impact tokenization strategy has on downstream performance
• Architecture trade-offs: Head-to-head comparisons across three approaches — off-the-shelf LLM embeddings, a custom set-aware transformer, and a hybrid fine-tuned LLM — and when each breaks down
• Multi-objective training: Combining next-event prediction, next-token prediction, and contrastive learning with Matryoshka representation learning for flexible embedding dimensionality without retraining
• Temporal encoding for irregular time series: Encoding events spaced across seconds to months, unlike NLP’s uniform token positions
• Inference at scale: From a 10-day batch pipeline to sub-300ms real-time inference for millions of sequences via model distillation, context window management, and SageMaker serving
• Evaluation beyond perplexity: Why standard LM metrics fail for event prediction, and the framework we built to measure both predictive accuracy and embedding quality