screamingface
Leaderboard

BIG-Bench Extra Hard

4,519 reasoning problems across BBEH's 23 tasks (about 200 each: boardgame QA, boolean expressions, buggy tables, causal understanding, dyck languages, multistep arithmetic, shuffled objects, spatial reasoning, time arithmetic, web of lies, word sorting, zebra puzzles and eleven more), each a harder successor to a BIG-Bench Hard task; the eval drops one duplicated problem from the published 4,520. Every prompt carries the paper's instruction to think step by step and end with "The answer is:" and a bare answer: free text, a number, a listed option's letter like (a), or yes/no. Imported from inspect_evals by Task replay: the Cases are fetched the way inspect fetches them and sealed by a Case Digest. Graded by the eval's own rule-based matcher (the paper's evaluate.py): it reads the text after the prefix and accepts the published answer, a letter with or without its brackets, an equal number, or quote and bracket variants, so no judge tokens are spent. Benchmark score = plain accuracy (micro average) over the cases run, the figure the paper reports for BBEH Mini; the paper's headline BBEH score is the harmonic mean of the 23 per-task accuracies (each plus 0.01), which the board does not compute; each Case keeps its task name, so the per-task accuracies can be regrouped from a full run. Answer format: end the reply with 'The answer is: 42', the final answer alone after that prefix, as the question asks. The rest of that line is compared with the key, ignoring case, bold marks and a final period, so 'The answer is: **42**.' counts, while 'The answer is: 42 apples' and 'ANSWER: 42' score 0.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub