Benchmarks · Science & research
MLE-bench
Can an AI agent do the work of a machine-learning engineer end to end — preparing data, training and tuning models, and iterating — well enough to place on a real Kaggle leaderboard?
OpenAIReleased 9 October 2024Live
MLE-bench measures a different skill from most coding benchmarks: not whether a model can write a correct function, but whether an agent can carry out the whole, messy job of a machine-learning engineer — reading a problem, preparing the data, choosing and training a model, and iterating under a deadline. OpenAI built it from 75 real Kaggle competitions, covering image, text, tabular and audio tasks, and scored each agent’s submission against the competition’s own public leaderboard using Kaggle’s bronze/silver/gold medal thresholds — the same bar real competitors are judged against.
At launch the results were sobering: the best combination tested, o1-preview running the open-source AIDE scaffold, reached at least a bronze-medal-equivalent finish in 16.9% of competitions, against 8.7% for GPT-4o with the same scaffold. That gap between “occasionally competitive” and “reliably good” was the point — MLE-bench was designed to stay hard rather than to flatter current models, and the paper also probed how far more compute and more attempts per task could close it, alongside checks for whether competition data had leaked into pre-training.
The benchmark now runs an open leaderboard on GitHub, where later agent scaffolds have posted considerably higher medal rates on the lower-compute “Lite” subset of 22 easier competitions, though OpenAI has paused new submissions while it reworks the evaluation process, and the harder full 75-competition set remains the more demanding test. MLE-bench sits alongside PaperBench and RE-Bench as part of a cluster of OpenAI and METR evaluations built specifically to measure whether AI agents can do the practical work of machine-learning research, not just answer questions about it.
The set
75 real Kaggle machine-learning competitions spanning image classification, text classification, tabular prediction and audio tasks, each run inside a sandboxed environment with a fixed time and compute budget. An agent's submission is scored against the competition's actual public leaderboard, using Kaggle's own medal thresholds (bronze/silver/gold) as the success bar. A lower-compute 'Lite' subset of 22 competitions is also offered.
Example
From the included "OpenVaccine: COVID-19 mRNA Vaccine Degradation Prediction" Kaggle competition, the task description an agent receives: "mRNA vaccines have taken the lead as the fastest vaccine candidates for COVID-19, but currently, they face key potential limitations... Researchers have observed that RNA molecules have the tendency to spontaneously degrade. This is a serious limitation--a single cut can render the mRNA vaccine useless." The agent must train on labelled RNA sequences and predict degradation rates, scored by Mean Columnwise RMSE.github.com
Where it stands
At launch, OpenAI's o1-preview with the open-source AIDE scaffold reached at least a bronze-medal result in 16.9% of competitions, roughly double GPT-4o's 8.7%; the GitHub leaderboard has since recorded much higher scores from newer agent scaffolds on the easier 'Low complexity' subset, though OpenAI paused new submissions while it revises the evaluation process.
In the timeline · 2 entries
OpenAI publishes SimpleQA, a benchmark for factuality
A short-form factuality benchmark designed to be more challenging and less saturated than prior QA benchmarks.
Benchmarks & progress
OpenAI publishes MLE-bench for evaluating agents on ML engineering
A benchmark of Kaggle-style machine-learning engineering competitions for measuring AI agents' research and engineering skill.
Benchmarks & progress