screamingface
Leaderboard

IFEval

The canonical 541-prompt instruction-following benchmark (https://arxiv.org/abs/2311.07911), graded by deterministic strict and loose verification with the official checker. Each Case invokes the Candidate exactly once. The headline is prompt-level strict accuracy; instruction-level accuracy is pooled over the run's instructions. Case ids are positions 1..541 in the pinned dataset; one row (official key 2785) is patched to the official harness prompt, whose text matches its graded constraints. Offers Draft Feedback: a Corrective Loop recipe can ask mid-run whether a draft passes, and hears the fraction of instructions met and the checker's words for each failed one, at no judge cost. Answer format: no answer line; the whole reply is checked by the official IFEval checker against every instruction in the prompt, and the headline scores 1 only when all are followed strictly, else 0 (instructions partly followed count in the instruction-level scores beside it). Told to use no commas, 'Paris is in France' scores 1 and 'Paris, in France' scores 0.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub