OpenAI releases BrowseComp benchmark
On 1,266 hard-to-find-online questions, GPT-4o answered under 2% correctly even with browsing, while OpenAI's Deep Research agent solved roughly half — more than humans given up to two hours per question.
- Benchmarks & progress
- Minor
OpenAI released BrowseComp, a benchmark of 1,266 questions designed to be deliberately hard to answer by web search, each constructed so the answer exists online but is scattered across sources that require persistent, multi-step navigation to connect. Questions were built by first picking a hard answer, then writing a question inverting the usual search direction, so that no single query or page reliably surfaced it.
The results showed a sharp split between models with and without dedicated browsing agent architecture. GPT-4o without browsing solved under 1% of questions; GPT-4o with basic web browsing reached under 2%; OpenAI’s o1 reasoning model, without browsing at all, scored under 10% on reasoning alone. Deep Research, OpenAI’s dedicated browsing agent, solved roughly half — a result the authors contrasted with human testers, who were allowed up to two hours per question and solved under a third, illustrating that the task rewarded persistence and tool use over raw knowledge.
The benchmark’s point was that browsing capability, not underlying model knowledge, was the binding constraint on this class of task: giving a model access to a browser tool did little unless it was also built to search iteratively and synthesise across many pages. BrowseComp became a reference evaluation for agentic web-research systems in the months that followed, cited alongside similar benchmarks as labs raced to build and compare “deep research” style agents.