screamingface
Leaderboard

AIME 2026

All 30 problems of the 2026 American Invitational Mathematics Examination (AIME I and II), imported from inspect_evals. Every answer is an integer from 0 to 999; the model solves step by step and commits its final answer on a closing 'ANSWER:' line. Grading is the eval's own scorer β€” a numeric match of the reply's final line against the answer key β€” so no judge tokens are spent. A reply the token limit cut off before its 'ANSWER:' line is wrong: it never answered, though v0.20.0's scorer would read whatever number ended its last line. This is inspect_evals' proposed fix (issue #2701), adopted here. Cases are served in the dataset's own order, as the eval serves them. Benchmark score = plain accuracy over the cases run. Answer format: end the reply with a line like 'ANSWER: 104'. Only the last line is read and its last number is the answer ('\boxed{104}' also counts), so a note after the number, as in 'ANSWER: 104 (mod 1000)', is read as 1000 and scores 0, as does a reply the token limit cut off before its ANSWER: line.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub