screamingface
Leaderboard

LAB-Bench TableQA

244 multiple-choice questions about one to three tables from a biology research paper (the LAB-Bench TableQA subset), imported from inspect_evals. Each table comes as a picture, so the Candidate must read images; the pictures are saved inside the Case at import and nothing is fetched at run time. Every question offers the distractors plus an explicit 'insufficient information' opt-out; grading is the eval's own precision-aware choice scorer (opting out counts as no answer, not wrong), so no judge tokens are spent. Serving order and choice order are pinned by policy seeds (inspect_evals shuffles both per run). Benchmark score = plain accuracy over the cases run; the eval's precision and coverage are not reproduced. Answer format: end the reply with a line like 'ANSWER: B', the letter of one option. A period after it, as in 'ANSWER: B.', still counts, and so does a wrapper around the letter, as in 'ANSWER: $B' or 'ANSWER: (B)'. Words after the letter, as in 'ANSWER: B because it is red', are read as no answer and score 0. Picking the option Insufficient information to answer the question also scores 0.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub