screamingface
Leaderboard

LAB-Bench LitQA2

199 multiple-choice questions answerable only from specific recent biology papers (the LAB-Bench LitQA2 subset), imported from inspect_evals. Every question offers the distractors plus an explicit 'insufficient information' opt-out; grading is the eval's own precision-aware choice scorer (opting out counts as no answer, not wrong), so no judge tokens are spent. Serving order and choice order are pinned by policy seeds (inspect_evals shuffles both per run). Benchmark score = plain accuracy over the cases run. Answer format: end the reply with a line like 'ANSWER: B', the letter of one option. A period after it, as in 'ANSWER: B.', still counts, and so does a wrapper around the letter, as in 'ANSWER: $B' or 'ANSWER: (B)'. Words after the letter, as in 'ANSWER: B because it is red', are read as no answer and score 0. Picking the option Insufficient information to answer the question also scores 0.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub