---
title: "Why Large Language Models Fail at Tabular Prediction"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.02412v1"
permalink: "https://feed.astralor.com/items/itm_6c5481fdddcae466b2803110"
published: "2026-08-03T15:52:51.000Z"
score: 78
channel: "ai"
tags: ["arXiv","Large Language Models","Curse of Dimensionality","Tabular Foundation Model","Tabular Prediction"]
---

# Why Large Language Models Fail at Tabular Prediction

## AI commentary

> This paper is a must-read for anyone working on tabular learning—it rigorously shows that LLM failure on tables is not about effort but about dimensionality. For tabular foundation model researchers, it's a signal to attack; for practitioners, it suggests pivoting rather than prompt-engineering further.

## At a glance

This paper systematically investigates why large language models underperform on tabular data prediction. Controlled experiments falsify hypotheses about noise, CSV formatting, numeric tokenization, and batch prediction. Dimensionality is decisive: as dimensionality grows, LLM accuracy decreases while classical baselines stay flat or improve. Behavioral comparison shows the LLM behaves like a local distance-based method in 2D, but in higher dimensions no classical model—even with tuned dimension-dependent noise—reproduces its predictions. This explains why LLMs lose to 50-year-old baselines on tables, though the internal mechanism remains open.

## Key points

- Controlled experiments falsify hypotheses on noise, CSV formatting, tokenization, and batch prediction
- Dimensionality is decisive: LLM accuracy degrades with dimension while classical models remain stable
- In 2D the LLM aligns with local distance-based methods; in higher dimensions no classical model reproduces it
- The internal mechanism remains open, calling for deeper interpretability analysis

## Full text

Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)-(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_6c5481fdddcae466b2803110