screamingface
Leaderboard

FutureX-Past (study scoring)

FutureX-Past as the crowd-wisdom study (behavior-aware LLM crowds for future prediction) scored it, to replicate that study's numbers: the 1,118 questions of the 2026-08-02 release, which holds all 100 of the study's questions with the study's exact answer keys. Same prompts as futurex_past, word for word. No Judge: the boxed answer is split on commas, case-folded and compared with the key as text by the study's own code (level 1 exact, level 2 set F1, levels 3-4 0.8 of the overlap unless the order is exact). As in the study, a reply with no usable answer (none in the box, a hedge, every option) leaves the score, and the run reports how many did as the run metric excluded; the headline weights the level means 10/20/30/40. Names and numbers must match as text, so this scores below futurex_past on levels 3-4 by design; use futurex_past for a score on the FutureX authors' scale. Pass the study's 100 Case positions as cases=[...] (README) to rerun its runs. All 27 of the study's models reproduce exactly; its best committee's stored answers score 0.306 here against the study's 0.302, because the study counts a committee's abstain as 0 and here an empty answer leaves the score. Answer format: put the answer in \boxed{}, as each question asks; only the last box is read and compared with the key as text, ignoring case and extra spaces, with no judge. '\boxed{Yes}' and 'Probably \boxed{yes}' count; 'Yes' with no box and 'First \boxed{Yes}, then \boxed{No}' score 0. A reply with no usable answer is left out of the headline score, as in the study.

about The leaderboard's top row is the current state of the art. Every submission carries its url4 expression, a complete recipe for the run. Copy it with the Copy button in the Run Locally column to rerun, verify or build on any score here.
Loading…
😱 screamingface GitHub