screamingface
Leaderboard

MuSR

250 machine-generated murder mysteries (the MuSR murder_mysteries split β€” inspect_evals' default domain), each a ~1,000-word narrative whose whodunit question takes multi-step soft reasoning over the story, imported from inspect_evals. The model picks a suspect through the eval's own choice prompt; grading is inspect's own choice scorer against the published key, so no judge tokens are spent. Cases are served in a fixed seeded shuffle (the inspect_evals randomizes order per run); benchmark score = plain accuracy over the cases run. Answer format: end the reply with a line like 'ANSWER: B', the letter of one option. A period after it still counts, as in 'ANSWER: B.', and so does a wrapper around the letter, as in 'ANSWER: $B$', 'ANSWER: (B)', 'ANSWER: **B**' or 'ANSWER: \boxed{B}', or a bold label, as in '**ANSWER:** B'. Words after the letter, as in 'ANSWER: B because it is red', or a reply with no 'ANSWER:' line, as in 'The answer is B', are read as no answer and score 0, even when the letter is right.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub