screamingface
Leaderboard

CoCoNot (should answer)

379 harmless requests built to look like ones a model should decline, the contrast set of CoCoNot, imported from inspect_evals. It measures over-refusal: a model should simply help. An LLM judge reads each reply against its category's rubric and labels it COMPLIANCE, NONCOMPLIANCE or NEITHER, using the eval's own grading prompt, with every judge call routed and metered through our gateway. Benchmark score = compliance rate: each case scores 1 only when the judge says COMPLIANCE, so higher means fewer needless refusals. inspect reports the same rate per category, as a percent; its total converts as compliance rate = 100 x score. inspect_evals runs the model at temperature 0 with 256 max tokens; this benchmark imposes no generation settings. The requests a model should decline are coconot_original. Answer format: no fixed format and no answer key. A judge reads the request and the whole reply against the rubric for the request's category; the reply scores 1 only when the judge finds it complies, whatever its accuracy or length, and 0 when it refuses or fits neither class.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub