FRAMES

Most retrieval-augmented generation demos are tested on questions answerable from a single passage, which understates how hard real research questions are. Google DeepMind and Harvard researchers published FRAMES in September 2024 — the name stands for Factuality, Retrieval, And reasoning MEasurement Set, though the paper itself carries the title “Fact, Fetch, and Reason” — with 824 questions, each requiring facts drawn from two to 15 separate Wikipedia articles. A typical item asks something like which person shares a first name with one historical figure’s mother and a surname with another’s, forcing a system to chain several lookups together rather than retrieve once and answer.

The paper’s own results illustrated why multi-hop retrieval is the hard part. A naive single-step retrieval pipeline using Gemini 1.5 Pro answered 40.8% of questions correctly; an iterative pipeline that re-planned its search across multiple rounds reached 66%; and an oracle version, handed every necessary article directly with no retrieval required, reached 72.9%. The size of that last gap — more than 25 points even with perfect retrieval — was itself informative: some of the failure sat in the model’s reasoning, not only in finding the right documents.

FRAMES is young enough that it has not built up a long history of leaderboard turnover, but it has been adopted as a standard component in evaluating agentic and RAG-based systems specifically, distinct from benchmarks like SimpleQA that test what a model already knows without looking anything up. Its distinguishing contribution is separating retrieval quality from reasoning quality within a single evaluation, rather than measuring only the combined result.