screamingface
Leaderboard

DROP

9,535 questions over Wikipedia paragraphs from DROP's validation split, each needing discrete reasoning over the text: counting, sorting, adding or comparing numbers and dates, imported from inspect_evals. The model sees three worked examples from the training split (a fixed seeded draw, as the eval sends them), then the passage and question, and ends its reply with an 'Answer:' line. Grading is inspect's own F1, which reads that line through the eval's answer extractor and takes the best token overlap with any accepted answer, so no judge tokens are spent. inspect's F1 is not DROP's official scorer, which also matches numbers span by span, so numbers are close to, not identical with, the paper's. Benchmark score = mean F1 over the cases run. Answer format: end the reply with a line like 'Answer: 4'. Only the text after the first Answer: is read (the whole reply when there is none), scored by word overlap with the accepted answers, so extra words earn partial credit. Write numbers as digits: 'Answer: four' shares no word with 4 and scores 0.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub