1,273 US medical-licensing-exam style questions with five options each (the MedQA English test set), imported from inspect_evals by Task replay: the Cases are fetched the way inspect fetches them (the bigbio/med_qa repository at a pinned revision, read by its loader script) and sealed by a Case Digest. Graded by inspect's choice scorer against the published answer, so no judge tokens are spent. Benchmark score = plain accuracy over the cases run. Answer format: end the reply with a line like 'ANSWER: B', the letter of one option. A period after it still counts, as in 'ANSWER: B.', and so does a wrapper around the letter, as in 'ANSWER: $B$', 'ANSWER: (B)', 'ANSWER: **B**' or 'ANSWER: \boxed{B}', or a bold label, as in '**ANSWER:** B'. Words after the letter, as in 'ANSWER: B because it is red', or a reply with no 'ANSWER:' line, as in 'The answer is B', are read as no answer and score 0, even when the letter is right.