We build two very different tools for tabular prediction, so we put them head-to-head on the same data.
ModelForge is our autonomous AutoML agent: give it a dataset and it analyzes the columns, researches approaches, writes training code, and trains and tunes a suite of real models — XGBoost, RandomForest, gradient boosting, SVMs, linear models — then keeps the best. PredictLM is our tabular foundation model: you send a small table of examples and the rows you want predicted, and it answers in a single API call, with no training at all.
One spends minutes building a bespoke model. The other answers in a fraction of a second. We wanted an honest read on what that difference actually costs you in accuracy.
How we ran it
We kept the comparison boringly fair:
- Four standard public datasets — breast-cancer (426 train / 143 test), wine (124 / 54), diabetes (331 / 111), and California housing (1,000 / 393).
- One train/test split per dataset, fixed seed, fed identically to both sides. Both engines train on exactly the same rows (context capped at ≤1,000 rows), and neither sees the test targets.
- ModelForge built and tuned a model from the training split, then predicted the held-out test features.
- PredictLM used the training split as in-context examples and predicted the same test features in a single
/v1/predictcall. Features were standardized first — a two-line client step; ModelForge does its own preprocessing internally. - Both sets of predictions were scored with the same code on the held-out truth: accuracy for classification, R² for regression.
The results
| Dataset | Metric | PredictLM | ModelForge | Gap | PredictLM time | ModelForge time | Speed-up |
|---|---|---|---|---|---|---|---|
| breast-cancer | accuracy | 0.965 | 0.965 | tie | 0.45 s | 6.6 min | 888× |
| wine | accuracy | 0.963 | 0.982 | MF +0.02 | 0.13 s | 7.2 min | 3,333× |
| diabetes | R² | 0.512 | 0.487 | PLM +0.02 | 0.21 s | 18.6 min | 5,334× |
| california | R² | 0.715 | 0.776 | MF +0.06 | 0.72 s | 6.9 min | 574× |
Read that as a directional result, not a scoreboard. These are single splits with one seed, no confidence intervals, and small test sets (54–393 rows) — the wine "loss" comes down to a single test row (53/54 vs 52/54), and the diabetes "win" is comfortably inside sampling noise. What the numbers do support is the shape: PredictLM landed within about two points on three of the four datasets, and about six R² points behind on the hardest one (California housing).
The totals are where the contrast is unambiguous. PredictLM answered every dataset in 1.5 seconds combined, with zero per-run tokens and one API call each. ModelForge took 39 minutes and roughly 2 million tokens — inherent to an LLM agent that writes and runs its own training code, across two-to-three dozen tool calls per build.
And PredictLM's answers aren't bare numbers: the API also returns an uncertainty estimate, ranked feature attributions, and a plain-language rationale per prediction — the kind of thing you'd otherwise bolt on afterwards. (We didn't measure calibration quality in this run; that's a separate study.)
What this means
The honest headline is not "PredictLM beats AutoML." It's that a single zero-training API call landed in the same ballpark as a fully-tuned model on these datasets — within a couple of points on three of four, and about six R² points back on the fourth — while being hundreds to thousands of times faster than ModelForge's autonomous build-and-tune loop. To be precise about what that multiplier measures: the winning models themselves (XGBoost, SVMs, linear models) fit on ≤1,000 rows in well under a second too — the gap is the agent's research, code generation, and model search, not raw training time.
Neither tool is better in the abstract, and that's the point:
- ModelForge is what you reach for when you need the last points of accuracy on a harder problem — here it took wine and California housing — and want to own a bespoke, tuned model you can deploy and version yourself.
- PredictLM is what you reach for when you want a trustworthy answer now — inside a spreadsheet, an internal tool, or an agent using prediction as a tool call — with zero setup, and uncertainty and explanations attached.
For many small, well-behaved tabular problems where you want an answer now, "one call, sub-second, explained, at near-parity accuracy" is often the trade you want to make. Larger, messier, high-cardinality or imbalanced tables are a different regime we haven't tested here — that's future work.
Try it yourself — it's free while in research preview: predictlm.zerooneresearch.ai. The model weights (predictlm-base-26m) are Apache-2.0 on Hugging Face, so you can run the same kind of comparison on your own data.