TabPFN and TabICL Beat Tuned XGBoost on All 14 Tables Without Training
TabPFN and TabICL vs. tuned XGBoost: the model that doesn't train won 14/14
A hands-on benchmark pits two pretrained tabular foundation models, TabPFN and TabICL, against tuned XGBoost across fourteen datasets. The models that never see the training data during fitting win on all fourteen, often by small but consistent margins, and answer in about a second. Tuning XGBoost took up to fifty seconds per dataset, suggesting hyperparameter search may become optional for many tabular tasks.
A tabular foundation model predicts on a table without ever having trained on it and still beats tuned boosting.
- rdedev
Tabular foundation models are one of those things where when you first look into it, it does not make sense as to why they would work so well but it does.
In drug property prediction domain, tabular foundation models coupled with another foundation model for molecules are pretty close to being the state of art.
Btw the article makes heavy use of AI or is written in that way A lot of unnecessary dramatic flair that gets very tiring
- icfly2
Nice little test, but man this AI writing is a pain. I understand that you want to churn out blog posts, but please don't write in this breathless style. Tell your LLM that it is writing a lab report.
- opensandwich
Perhaps a more interesting exploration of when it does well, and when it doesn't against xgboost and also tabm
https://engineering.block.xyz/blog/blocktabbench-evaluating-...