screamingface
Leaderboard

ContractEval

4,182 clause-extraction questions over 102 real commercial contracts (CUAD test split). The model is given a full contract and one of 41 clause categories, and must quote the answering sentences verbatim or reply "No related clause." Grading is deterministic string containment, the paper's own rule: every gold sentence must appear, with no partial credit, so there is no Judge and no grading tokens. Benchmark score = F1 over the whole run's confusion matrix, not a mean of Case scores: 70% of rows have no clause, so a model that always declines would be right on 70% of Cases while answering nothing, and F1 scores it 0. Each Case's row records whether the Case has a clause, whether the model declined, and its token overlap with the gold text. One named deviation from the paper: it sends its extraction instructions as a system message, while this board delivers them as the leading text of the single candidate input, because a benchmark cannot address a candidate's system role. The instructions are identical; the request shape is not, so scores are comparable to the paper's in protocol but not in message layout. Answer format: copy the relevant sentences from the contract exactly, full stop included, as in 'The term is two years.'; text around them is allowed, so 'Relevant: The term is two years.' counts. A paraphrase ('The term lasts two years.') or a dropped full stop ('The term is two years') scores 0. When nothing applies, say No related clause.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub