screamingface
Leaderboard

LAB-Bench SuppQA

82 multiple-choice questions about the supplementary materials of named biology papers (the LAB-Bench SuppQA subset β€” each prompt carries the paper title and DOI), imported from inspect_evals. Every question offers an explicit 'insufficient information' opt-out; grading is the eval's own precision-aware choice scorer (opting out counts as no answer, not wrong), so no judge tokens are spent. Serving order and choice order are pinned by policy seeds. Benchmark score = plain accuracy over the cases run. Answer format: end the reply with a line like 'ANSWER: B', the letter of one option. A period after it, as in 'ANSWER: B.', still counts, and so does a wrapper around the letter, as in 'ANSWER: $B' or 'ANSWER: (B)'. Words after the letter, as in 'ANSWER: B because it is red', are read as no answer and score 0. Picking the option Insufficient information to answer the question also scores 0.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub