screamingface
Leaderboard

ChemBench

2,785 chemistry questions from ChemBench, across topics from general and organic chemistry to toxicity and safety, imported from inspect_evals. Most are multiple choice, some with several correct options the model must list together (for example A,D); the rest ask for a number. A multiple-choice answer is right when it names exactly the key's letters, in any order and case, separated by commas or spaces ('A,D', 'A, D', 'D A' and 'd,a' all count), as the ChemBench authors grade and as the eval's prompt invites; a number must fall within 1% of the answer, as the eval grades. No judge tokens are spent. The letter rule is inspect_evals' proposed fix (PR #2591), adopted here: the pinned v0.20.0 scorer compares the letters as text, so on the 394 Cases with several correct letters a right answer written 'A, D' or 'D,A' scored 0. Cases are served in a fixed seeded shuffle (the eval shuffles per run). Benchmark score = plain accuracy over the cases run; the eval's per-topic breakdown is not reproduced. Answer format: put the answer inside answer tags, as in '[ANSWER]A,C[/ANSWER]'. On a multiple-choice question, name every right letter and no others, in any order, so '[ANSWER]C, A[/ANSWER]' also counts, while '[ANSWER]A[/ANSWER]' scores 0 when A and C are right. Without the tags, as in 'ANSWER: A, C', the reply scores 0. A numeric question takes one number inside the tags.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub