[
  {
    "name": "MMLU-Pro (and MMLU)",
    "measures": "Multiple-choice questions across dozens of academic and professional subjects, testing broad knowledge and reasoning. MMLU-Pro expands the original MMLU to ten answer options and adds harder, reasoning-focused questions to better separate strong models.",
    "caveats": "The original 2020 MMLU is effectively saturated and contains documented label errors and ambiguous items (prompting fixes such as MMLU-Redux). MMLU-Pro reduces but does not eliminate contamination, and frontier models are now approaching roughly 90%, so its remaining discriminative headroom is shrinking.",
    "links": {
      "paper": "https://arxiv.org/abs/2406.01574"
    }
  },
  {
    "name": "GPQA (Diamond)",
    "measures": "448 graduate-level multiple-choice questions in biology, physics, and chemistry written by domain experts; the 'Diamond' subset is the hardest ~198 items. Designed to be 'Google-proof' so skilled non-experts with web access score low (~34%) while PhD-level experts score much higher.",
    "caveats": "The small size makes scores noisy and sensitive to a handful of items. It is a public set with canary strings but not immune to contamination or to models pattern-matching similar training data, and top reasoning models now score in the 80-90%+ range, approaching saturation.",
    "links": {
      "paper": "https://arxiv.org/abs/2311.12022"
    }
  },
  {
    "name": "SWE-bench Verified",
    "measures": "Real GitHub issues from popular Python repositories where a model must produce a code patch that passes the project's hidden tests. Original SWE-bench has 2,294 instances; SWE-bench Verified is a 500-instance human-filtered subset (built with OpenAI) confirmed to be solvable and clearly specified.",
    "caveats": "Verified is a curated subset, not a random sample, so it is easier than full SWE-bench and skews toward well-specified fixes. Public repositories raise contamination risk, results depend heavily on the agent scaffold wrapped around the model, and it covers only Python.",
    "links": {
      "site": "https://www.swebench.com/verified.html",
      "paper": "https://arxiv.org/abs/2310.06770"
    }
  },
  {
    "name": "HumanEval",
    "measures": "164 hand-written Python programming problems where the model generates a function from a docstring and is graded by unit tests (pass@k). It was an early standard for functional code generation.",
    "caveats": "Largely legacy and saturated: frontier models cluster around 96-98% pass@1, so it no longer separates leading systems, and its problems are widely believed to be present in training data (contamination). It is now used mainly as a minimum-capability bar, with SWE-bench and live coding benchmarks preferred for frontier comparison.",
    "links": {
      "paper": "https://arxiv.org/abs/2107.03374"
    }
  },
  {
    "name": "MATH and AIME (competition math)",
    "measures": "MATH is 12,500 competition mathematics problems with step-by-step solutions across difficulty levels. Reasoning models are increasingly evaluated on AIME (American Invitational Mathematics Examination) sets such as AIME 2024 and 2025 - 15 integer-answer problems per exam - as a harder, contamination-fresh test.",
    "caveats": "MATH is largely saturated for frontier reasoning models. AIME sets are tiny (about 30 problems for a given year), so differences of a few percentage points are within noise, and once a year's problems become public they can leak into training data, which is why the evaluation year matters.",
    "links": {
      "paper": "https://arxiv.org/abs/2103.03874"
    }
  },
  {
    "name": "ARC-AGI and ARC-AGI-2",
    "measures": "Abstract visual reasoning puzzles: from a few input-output colored-grid examples the model must infer a transformation rule and apply it to a new grid. ARC-AGI-2 (released 2025) is a harder redesign emphasizing symbolic interpretation, compositional reasoning, and contextual rule application, calibrated against human test panels.",
    "caveats": "Deliberately resists memorization, so it is hard to contaminate, but that also makes it niche and unrepresentative of typical language or coding work. Scores are usually reported under a compute/cost budget, and different rule sets (public vs. semi-private eval sets, with or without tools) make cross-report comparisons difficult.",
    "links": {
      "site": "https://arcprize.org/arc-agi/2",
      "paper": "https://arxiv.org/abs/2505.11831"
    }
  },
  {
    "name": "Chatbot Arena (LMArena / Arena)",
    "measures": "Crowdsourced human-preference ranking: users chat with two anonymous models side by side and vote for the better response, aggregated into an Elo-style leaderboard. Run by the team formerly known as LMSYS Chatbot Arena / LMArena, now operating as Arena (lmarena.ai redirects to arena.ai).",
    "caveats": "It measures human preference, not correctness, and has a documented style/verbosity bias toward longer, well-formatted answers (a separate Style Control ranking partially corrects this). Votes come from non-expert, self-selected users, and rankings can be affected by which models are served and how prompts are distributed.",
    "links": {
      "site": "https://arena.ai/",
      "paper": "https://arxiv.org/abs/2403.04132"
    }
  },
  {
    "name": "Humanity's Last Exam (HLE)",
    "measures": "About 2,500 expert-written questions across dozens of subjects, in multiple-choice and short-answer form, each with an unambiguous verifiable answer that cannot be found by quick web retrieval. Built by over 1,000 contributors to probe the frontier of academic knowledge.",
    "caveats": "Intended to be extremely hard; as of 2026 frontier models still score well below the roughly 90% expert baseline, and reported numbers vary widely with model, reasoning effort, and tool access. Models tend to be overconfident and poorly calibrated on it, and as a public set it faces the usual future-contamination risk.",
    "links": {
      "paper": "https://arxiv.org/abs/2501.14249"
    }
  },
  {
    "name": "OSWorld",
    "measures": "369 real computer tasks executed in a live operating-system environment (files, office apps, browsers, multi-app workflows), graded by execution-based checks on whether the resulting state is correct. A standard test of GUI/computer-use agents; WebArena is a companion benchmark focused specifically on browser tasks.",
    "caveats": "Execution-based grading can be brittle (environment setup, app versions, flaky UI). At release the best agents solved about 12% versus about 72% for humans; scores have since risen sharply, but tasks are mostly short and single-app, so high numbers overstate long-horizon reliability, which an updated OSWorld 2.0 targets.",
    "links": {
      "site": "https://os-world.github.io/",
      "paper": "https://arxiv.org/abs/2404.07972"
    }
  },
  {
    "name": "RULER (long context)",
    "measures": "Synthetic long-context benchmark that extends needle-in-a-haystack with 13 tasks in four categories (multi-key/multi-value retrieval, multi-hop variable tracing, aggregation, and question answering) at controlled lengths up to 128K+ tokens, measuring effective rather than advertised context length.",
    "caveats": "Because it is synthetic, it may not reflect messy real-world long-document reasoning, and the exact tasks and lengths are configurable, so setups differ between reports. Its central finding - that models degrade well before their claimed context window - continues to hold, which is the reason to use it over simple needle tests.",
    "links": {
      "paper": "https://arxiv.org/abs/2404.06654"
    }
  },
  {
    "name": "MMMU (multimodal)",
    "measures": "About 11,500 college-level multimodal multiple-choice questions across six disciplines and 30 subjects, pairing text with charts, diagrams, tables, chemical structures, medical images and more, testing perception plus subject-specific reasoning in vision-language models.",
    "caveats": "The multiple-choice format allows some guessing and text-only shortcutting; a harder MMMU-Pro variant was created to address this. It is a public set (contamination risk), and its answers come from academic materials, so it rewards textbook knowledge more than genuinely novel visual reasoning.",
    "links": {
      "site": "https://mmmu-benchmark.github.io/",
      "paper": "https://arxiv.org/abs/2311.16502"
    }
  },
  {
    "name": "tau-bench / tau2-bench (tool-agent-user)",
    "measures": "Simulates multi-turn customer-service conversations in which an agent must use domain APIs (e.g., retail, airline) while following written policies and interacting with a simulated user. It reports a pass^k reliability metric over repeated attempts; tau2-bench extends it to dual-control settings where the user can also take actions.",
    "caveats": "It is domain-specific (a handful of business domains), so it does not capture general tool use, and the simulated user is itself an LLM, which can introduce its own errors. Even strong function-calling models score under about 50% with low run-to-run consistency, making it a demanding but narrow slice of agentic ability.",
    "links": {
      "site": "https://github.com/sierra-research/tau2-bench",
      "paper": "https://arxiv.org/abs/2406.12045"
    }
  }
]