screamingface
Leaderboard

MuSiQue-Ans

2,417 multi-hop reading questions from the MuSiQue-Ans dev split (the test split's answers are withheld). Each question chains 2 to 4 facts, each fact sits in a different paragraph, and the model is given 17 to 20 numbered paragraphs, most of them decoys chosen to look relevant. It may reason first, then must end its reply with two lines: "Supporting paragraphs:" (the numbers of the paragraphs it used) and "Answer:" (the answer in as few words as possible). Grading is the paper's own scoring code, copied verbatim, so there is no Judge and no grading tokens. Three Named Scores per run: answer F1, the headline (token F1 against the answer and its accepted aliases), exact match, and support F1 (F1 of the cited paragraph numbers against the gold ones). A reply missing either line is still graded. The Frontier Score, 0.692 answer F1, is a fine-tuned retrieval pipeline's result on the test split, not a prompted model on dev, so our runs sit beside it rather than on the same scale. The dev set has been public since 2022 and may be in a model's training data. Answer format: end the reply with a Supporting paragraphs: line, then a line like 'Answer: Paris'. The text after the last Answer: is scored by word overlap with the answer and its aliases, so extra words earn partial credit, and an answer sharing no word with them, such as 'Answer: paragraph 3', scores 0. The cited paragraphs are scored apart, shown but not ranked.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub