screamingface
Leaderboard

MGSM (English)

250 grade-school math word problems (the English set of MGSM, the multilingual GSM8K translation), imported from inspect_evals by Task replay: the Cases are fetched from the TSV inspect downloads and checks against inspect_evals' sha256, then sealed by a Case Digest. Each prompt carries the eval's own instruction to reason first and give the final answer on the last line. Graded by inspect's numeric match, so no judge tokens are spent. Benchmark score = plain accuracy over the cases run; the eval also reports a standard error clustered by question, which the board does not. Answer format: end the reply with a line like 'Answer: 18', the bare integer. The last number in the whole reply is graded, with dollar signs and commas ignored, so 'Answer: $18' also counts, but any number after the answer is read instead: 'Answer: 18 (9 eggs at 2 each)' is read as 2 and scores 0.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub