Video summary

Son nouveau modèle IA va faire oublier les LLMs

Main summary

Key takeaways

Technology

The “New Era” of AI for Tabular Data

The episode argues that tabular foundation models may be as disruptive for structured datasets as LLMs have been for text.

The core problem (why this matters)

  • Traditional predictive modeling on tables has relied on older algorithmic paradigms, especially:
    • Gradient boosting / boosted trees (popular since the early 2000s and heavily optimized for tabular tasks).
  • The episode claims that LLMs don’t naturally solve tabular prediction when you simply prompt them with tables, because they are trained primarily for next-token/next-word prediction, not statistical estimation.
  • Researchers are trying to address tabular challenges such as:
    • Missing values
    • Uncertainty / probability calibration
    • Small datasets
    • Reducing engineering overhead for deployment

Marine Morvent’s Work (Inria / Nia)

Marine Morvent is a researcher at Inria who focuses on machine learning for tabular foundation models.

Key topics discussed

1. Tabular foundation models

  • Models pre-trained across many tables, intended for reuse on new tabular datasets.

2. Missing data in tables

  • Missing cells (“holes”) can bias models if encoded as zeros, and can break models that can’t handle missingness.
  • Classic linear models fail when the feature set differs between training and inference (e.g., missing variables).
  • Within the foundation-model setup, she says there still isn’t a definitive “best” strategy for handling missingness.

3. Uncertainty quantification / calibration

  • Instead of predicting only a class (e.g., “cancer / no cancer”), models should output probabilities with meaningful confidence (e.g., 0.8 vs 0.2).
  • Calibration goal example:
    • Among patients predicted with 90% risk, about 90% should truly have the condition.

Why LLMs Aren’t Enough for Tabular Data (As-Is)

  • You can feed an entire table to an LLM and ask for predictions, but:
    • LLMs aren’t trained for statistical reasoning on structured data
    • performance is worse than tabular-specialized models on benchmarks
    • context size and attention across the whole table can be problematic
  • Benchmarks compare LLMs against classical models across many table tasks, and the episode concludes that LLMs underperform overall in state-of-the-art tabular settings unless further specialized pretraining is applied.

“Before”: Boosted Trees Dominate Tabular Benchmarks

The episode explains why gradient boosting remains extremely strong:

  • Decision trees ask sequential questions until reaching a leaf prediction.
  • Random forests train many independent trees and average/vote outputs (“wisdom of crowds”).
  • Gradient boosting trains trees sequentially, where each new tree tries to correct the previous model’s errors (residuals), with the practical question being when to stop.

Key claims from the discussion:

  • Boosted trees (e.g., XGBoost) have remained top performers for about a decade on many tabular benchmarks and Kaggle-style competitions.
  • Neural networks, under standard supervised settings, haven’t clearly beaten boosted trees on classic tabular tasks.

The “Foundation Model” Shift for Tables

Definition given: foundation model

  • A model pre-trained on large amounts of data and then reused for downstream tasks, often with minimal or no fine-tuning.

What changes for tables

The paradigm shifts from:

  • Before: train on one dataset split (train/test) → refit from scratch for each new dataset
  • Now: pretrain across many tables → reuse the pretrained model for a new tabular prediction task with minimal fitting

A starting point paper mentioned:

  • TPFN (2023): introduced pre-trained context-based learning for tables (described as early/prototype at first).

Learning-in-Context for Tables vs Supervised Learning

She contrasts two “signatures” of models:

  1. Supervised tabular model

    • Training: X_train → predict Y_train
    • Inference: X_test → predict Y_test

  2. In-context / foundation tabular model

    • During inference/prediction, the model consumes as context: X_train, Y_train, X_test

    • It then outputs Y_test, with the idea that internal mechanisms infer a prediction algorithm from examples.

Clarification: she pushes back on calling it true “learning in context,” because weights aren’t updated. It’s better described as deduction / pattern extraction rather than parameter learning.

Additional notes:

  • The mechanism is associated with Transformer-based flexibility.
  • She notes earlier supervised deep learning attempts with flexible architectures didn’t work well—possibly due to data regime limitations.

What Changes Compared to XGBoost (Beyond Data Scale)

She says “a lot changes,” emphasizing:

  • Training paradigm: pretraining + reuse across tasks
  • Transformer-style architectures: tabular foundation models leverage transformers
  • Better handling of small-data regimes compared to heavy supervised tuning

Synthetic Data for Pretraining (When Real Data Is Private)

A major practical barrier is that many real tabular datasets are private.

The foundation-model approach uses synthetic tabular datasets, such as:

  • synthetic tables with controlled column distributions, covariances, and relationships to outputs
  • learning statistical mappings without “semantic understanding” of what columns mean (e.g., “age” vs “eye color”)

The episode claims this can work well even on real-data benchmarks, arguing it learns robust X→Y statistical relationships rather than semantics.

Concrete Model(s) Mentioned: Tabical / “Tab ICL”

She describes lab work that evolved from studying missing data within a TPFN-like framework into a model named Tabical:

  • Tab = tabular
  • ICL = in-context learning

Reported milestones:

  • Tabical v1 published in early 2025
  • Tabical v2 published in Nature, framed as state-of-the-art and scientifically notable

Open aspect:

  • Not only model weights: an open-source library and open components, including the prior (the synthetic data generator).

Adoption and Industry Use Cases (Examples)

She claims strong traction because models can be easier to deploy:

  • Compared to boosted trees requiring extensive hyperparameter tuning and expertise, foundation models reduce engineering burden: “give it data and see what comes out.”

Examples of domains mentioned:

  • Healthcare: predicting risks like septic shock after trauma
  • Physics: predicting whether CERN collision outcomes produce certain particles
  • Banking: repayment propensity / credit risk
  • Fraud detection: transaction monitoring
  • Business forecasting: turnover/orders prediction from sales/marketing databases

A recurring point:

  • Advantage in limited data regimes—foundation tabular models reportedly work well with small to moderate row counts where gradient boosting tuning may be harder.

Limitations Discussed

  • Inference cost is higher than classic boosted trees (though still described as “lighter than LLMs”).
  • Rough runtime estimates mentioned:
    • ~10,000 samples with not many columns: 1–2 seconds on GPU
    • ~1 million samples with hundreds of columns: minutes (roughly “7–8 minutes” mentioned)
  • Still early and open research issues, including:
    • best practices for missing data within the foundation-model framework
    • integrating external knowledge

Ecosystem / Competition / Collaboration

  • She differentiates her work from other tabular foundation-model labs (not framed as direct “racing”).
  • Mentions rapid movement in the space: startups and investments.
  • A startup based on TPFN is mentioned as being acquired (she says SAP).
  • Fundraising mention: “Fundamental” raised $200M.

Main Speakers / Sources

  • Mathieu Sanchez — host; introduces the episode in a “Tech Face / Tech Head” style format
  • Marine Morvent — researcher at Inria / Nia, discussing:
    • tabular foundation models
    • Tabical
    • missing data
    • uncertainty calibration
    • the TPFN / Tabical landscape

Original video