OpenAI publishes MLE-bench for evaluating agents on ML engineering

OpenAI published MLE-bench, a benchmark that measured AI agents against 75 real Kaggle machine-learning competitions rather than static question sets, testing whether an agent could prepare data, train models, run experiments and iterate the way a human ML engineer would.

Agents were scored against the same leaderboards human competitors were judged on, with a Kaggle bronze-medal finish or better used as the success threshold. The best result reported, OpenAI’s o1-preview using an open-source scaffold called AIDE, achieved at least a bronze-medal-equivalent result in 16.9% of competitions — nearly double GPT-4o’s 8.7% with the same scaffold. The paper also examined how performance scaled with more compute and attempts per task, and checked for contamination from competition data appearing in pre-training.

The gap between a Kaggle medal in one in six attempts and reliable machine-learning engineering was the point: MLE-bench was designed to be hard for current models rather than to showcase near-human competence, joining a wave of late-2024 benchmarks aimed at measuring agentic, multi-step task competence rather than single-turn question answering.