screamingface
Leaderboard

SimpleQA

4,326 short fact-seeking questions, each with one indisputable answer, from OpenAI's SimpleQA, imported from inspect_evals. The model answers in plain text and may decline. An LLM judge grades each reply against the gold answer with the paper's own grading prompt, as correct, incorrect or not attempted, and every judge call is routed and metered through our gateway. Each Case publishes those three as Named Scores (1 for the judge's grade, 0 for the other two). Benchmark score = the share of cases answered correctly. The paper's headline, the F-score (the harmonic mean of that share and the share correct among those attempted, so guessing costs and declining does not), is published beside it, with the share attempted. The paper graded with GPT-4o; this Benchmark pins Gemini 3.1 Pro Preview, so its numbers are not directly comparable to the paper's. Answer format: plain text, and declining is allowed. A judge grades the whole reply against the gold answer as correct, incorrect or not attempted; only correct scores 1. Correct means it contains the gold answer and contradicts nothing; hedging is fine, and a number must match to the gold answer's last significant figure.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub