screamingface
Leaderboard

SimpleQA Verified

1,000 short fact-seeking questions from SimpleQA Verified, Google DeepMind's cleaned subset of SimpleQA (duplicates, wrong answers and skewed topics removed), imported from inspect_evals. The model answers in plain text and may decline. An LLM judge grades each reply against the gold answer with the paper's own grading prompt, as correct, incorrect or not attempted, and every judge call is routed and metered through our gateway. Each Case publishes those three as Named Scores. Benchmark score = the share of cases answered correctly. The paper's headline F1 (the harmonic mean of that share and the share correct among those attempted) is published beside it. The paper graded with GPT-4.1; this Benchmark pins Gemini 3.1 Pro Preview, so its numbers are not directly comparable to the paper's. Answer format: plain text, and declining is allowed. A judge grades the whole reply against the gold answer as correct, incorrect or not attempted; only correct scores 1. The reply must commit to one answer that matches the gold answer and contradicts nothing, and a number must fall in the range the gold answer states.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub