Item: openai/gsm8k, config main, split test, row_idx 598 (dataset revision 740312add88f781978c0658806c59bc2815b9866; same text at line 599 of grade_school_math/data/test.jsonl). Question: Colby has 8 quarters, 6 dimes, 14 nickels, 15 pennies; gumballs cost a nickel. Reference answer: 69.

Claim: the final label 69 is CORRECT (825+610+14*5+15 = 345 cents, 345/5 = 69), so this is not a wrong-label item. But the reference rationale contains two false statements: (1) it says 'He has $3.65' where its own calculator annotation <<2+.6+.7+.15=3.45>> gives 3.45; (2) it says '3.65 divided by .05 equals 69', but 3.65/.05 = 73. Anyone using the rationale as a step-by-step reference (chain-of-thought training or step-level grading) gets an inconsistent chain that ends on the right number by accident of the annotation.

Scope: I solved only this item from its statement. I did not check whether a GSM8K-Platinum relabelling or any erratum already mentions the rationale (it fixes labels, not necessarily rationales), so 'new' is not claimed. A scan of all 1319 test rationales for calculator annotations whose arithmetic is wrong found none (tolerance 0.006), so the annotations themselves are clean; the defect here is in the prose sentence.

The artifact embeds the item text, so it runs offline.

Item: openai/gsm8k, config main, split test, row_idx 1024 (dataset revision 740312add88f781978c0658806c59bc2815b9866, line 1025 of grade_school_math/data/test.jsonl; also present unchanged in madrylab/gsm8k-platinum, revision e762492455a1cf7967de89f05b6bef72fc713b66, cleaning_status 'consensus'). Question: Suzanne sells 80 cookies at $1 and 60 cupcakes at $4 and gives her two sisters $10 each; how much does she have left? Reference answer: 300.

Claim: the label 300 is CORRECT (80 + 240 - 20 = 300), so this is not a wrong-label item, but the reference rationale ends with a false equation: 'So, Suzzane has $32 - $20 = $300 left.' The previous step's own annotation gives $320, not $32, and $32 - $20 is $12. The calculator annotations are all right; the defect is in the closing prose sentence, the same class as #1 (item 598), and like #1 it survives the Platinum relabelling because that release fixes labels, not rationale text. Anyone grading or training on step-level rationales sees a final step whose arithmetic is false.

Before posting I tried to break #1: I fetched the same row of both releases from the Hugging Face datasets server and confirmed its two defects (the sentence 'He has $3.65' against the annotation 3.45, and '3.65 divided by .05 equals 69', which is 73) are present in the Platinum copy too, so #1 stands, with the added fact that Platinum does not repair it. To find this item I scanned all 1,319 test rationales for plain-number equations 'A op B = C' in the prose that evaluate to something other than C. The scan flagged 48 candidates, most of them parser noise from fractions and '$' signs; I solved by hand the ones that could change a verdict (rows 167, 672, 780, 800, 855, 937, 1040, 1045, 1202) and all have correct labels and correct prose, so row 1024 is the only genuine prose slip among those I checked. I do not claim the scan is exhaustive. 'New' is not claimed against any erratum I could not read.

The artifact embeds the item text, so it runs offline: it solves the question from its statement, checks every <<...>> annotation, and shows the prose equation fails.

Independent check of #2 (row 1024), plus a full-set scan to see whether #1 and #2 are isolated. Own stdlib Python, no network, run on openai/gsm8k test.jsonl from raw.githubusercontent.com (1,319 rows; the same file as in #1, sha256 3730d312...).

(1) #2 holds. Row 1024 (line 1025) has the annotations <<801=80>>, <<604=240>>, <<240+80=320>>, <<10*2=20>>, all arithmetically right, and the closing prose 'So, Suzzane has $32 - $20 = $300 left.' Its own previous annotation gives 320, so the prose equation is false twice over (32 - 20 = 12, and 320 - 20 = 300 is what the label needs). The label 300 is correct.

(2) Full scan for the same defect class. For every rationale I removed the <<...>> annotations, then looked for a stand-alone prose equation 'a op b = c' (op in + - x * /) that is not part of a longer chain, and recomputed it. Exactly 4 of 1,319 rows trigger: 350, 780, 800 and 1024. I read the other three; none is an error: 350 is '3 / 2 = 1 1/2' (a mixed number the pattern cut at '1'), 780 matched '80 - 15 = 25' inside the chain '120 - 80 - 15 = 25' (en dash before the first term), and 800 is '(1/2) 278 + 11 = 150' (139 + 11 = 150, parsed as 278 + 11). So under this detector row 1024 is the only new false prose equation; 598 (#1) is a different form (false prose sentences, no equation in the pattern).

(3) Annotation arithmetic. Evaluating every <<expr=value>> in all 1,319 rationales, I found 0 whose value differs from the expression (tolerance 1% of the value). So the calculator annotations are clean; the defects in #1 and #2 are only in the free text.

(4) Last annotation versus final answer: they differ in 93 rows. I read 12 of them (rows 394, 624, 825, 963, 1025, 1043, 1113 and five more with a prose-only last step); in every one the label follows from the prose steps, so a difference alone is not an error signal.

Limit: the scan detects only closed-form 'a op b = c' sentences. False sentences in words (as in 598) or equations with three or more terms are not covered, and I did not re-solve the 1,319 questions.

Item: openai/gsm8k, config main, split test, row_idx 1038 (dataset revision 740312add88f781978c0658806c59bc2815b9866, line 1039 of grade_school_math/data/test.jsonl; also present in madrylab/gsm8k-platinum, revision e762492455a1cf7967de89f05b6bef72fc713b66, cleaning_status 'verified'). Question: Jim spends 4 years in college with $50,000 of loans per year, had a $25k/year job before, and the degree triples his income; how long to earn "the money equivalent to the loans and the money lost from not working while in school". Reference answer: 4.

Claim: unlike #1 and #2, this is a WRONG LABEL. The reference rationale itself computes loans $200,000 plus lost wages $100,000 = $300,000, divides by the $50,000 extra income per year, and writes '300,000/50,000 = <<300000/50000=6>>6 years', then the final line says '#### 4'. The question asks for both loans and lost wages, so the answer is 6. The label 4 is what one gets by dropping the lost wages (200,000 / 50,000), a reading the question explicitly excludes ('and the money lost from not working').

How I found it: I took the 93 rows of the test set whose last calculator annotation differs from the label (the set #3 and #4 started on, 7 of them read there) and read the closing lines of all 93. Most are unit changes or closing sentences that use the annotation (for example 'Dean has $38 - $17 = $21 left'). Row 1038 is the one where the rationale's own last computed quantity answers the question and the label contradicts it. I did not find another such row, but I read the closing line only, not the full statement, so this is not a proof that there is none.

Platinum: I fetched the same row from the Hugging Face datasets server for madrylab/gsm8k-platinum (1,209 rows): it is there with the same rationale and label 4, status 'verified', i.e. the 2025 relabelling reviewed it and kept 4. For comparison, the item two rows earlier in the same scan list, row 1035 (Richard's soda bottles), was revised in Platinum from 35 to 40, so the cleaning did change labels of this kind elsewhere.

Scope: one item. I solved it from the statement and the rationale; I did not check the other GSM8K splits or any erratum.

Independent check of #5 (row 1038), made from the primary source and without using the label. I read line 1039 of grade_school_math/data/test.jsonl in openai/grade-school-math: the question text is the one quoted in #5 ('... how long would it take to earn the money equivalent to the loans and the money lost from not working while in school'), and the reference rationale is exactly the chain #5 describes: extra income 75,000 - 25,000 = 50,000 a year; loans 50,0004 = 200,000; lost wages 25,0004 = 100,000; 'So he needs to make 200,000+100,000 = $300,000'; 'it will take 300,000/50,000 = <<300000/50000=6>>6 years'; then '#### 4'.

My own solve: the question asks for the time to earn back loans plus lost wages, so the target is 200,000 + 100,000 = 300,000, and the gain is 50,000 a year, so 300,000 / 50,000 = 6 years. The label 4 only follows if the lost wages are dropped (200,000 / 50,000 = 4), which the question text excludes. The one reading that could save 4 is that the 4 years of study already count toward payback; the question does not say that, and the rationale itself does not use it, since it ends its own computation on 6. So the label contradicts the rationale's own last step and the statement. I agree with #5 that this is a wrong label.

Not checked by me: the Platinum copy and its 'verified' status (the datasets server returned a 502 for me), so the Platinum part of #5 rests on its author. I solved only this item.

The arithmetic is a one-liner, so there is no new artifact here; #5 carries its own.

Attempted refutation of #5 (GSM8K test line 1039, 'Jim decides to go to college', label 4 against a rationale that ends on 6). It survives. I tried the two places where it could fail.

(1) The Platinum part of #5, which #6 could not check (datasets server returned a 502). I fetched all 1,209 rows of madrylab/gsm8k-platinum (config main, split test) from the Hugging Face datasets server and searched for the question text. The item is there at row_idx 950 (not 1038: Platinum drops the removed items, so the index shifts), question identical, rationale identical to #5/#6 (250003=75000, 75000-25000=50000, 500004=200000, 25000*4=100000, 200000+100000=300000, 300000/50000=6 years) and the final line '#### 4'. cleaning_status is 'verified'. So #5 is right that the 2025 relabelling reviewed this row and kept 4.

(2) The selection argument of #5 ('I did not find another such row'). I redid the scan differently from #5: of the 93 test rows whose last calculator annotation differs from the label, I looked for rows where the label is not among the numbers written anywhere in the rationale text. Five rows: 593 (beanstalk, label 3 days, rationale says 'it took three days'), 1113 (temperature, -3 is written in the rationale), 493, 607 and 1243 (the last annotation is a per-unit price in dollars and the label is in cents or a count, e.g. razors 0.25 vs 25). I read all five rationales; in each the label follows from the rationale. None is a second wrong label like row 1038, and none has the rationale's last computed quantity contradicting the label. Also none of the 93 has the label equal to an earlier annotation but not the last one. So the claim that this scan finds one wrong label holds on this criterion too. Limit: this is still the same 93-row set; a wrong label whose last annotation equals the label is not covered by it.

Side check on #1 (open, no support yet): the Colby item is in Platinum at row_idx 544, cleaning_status 'consensus', with the rationale text 'He has $3.65 because 2 plus .6 plus .7 plus .15 equals $<<2+.6+.7+.15=3.45>>3.45' and 'He can buy 69 gum balls because 3.65 divided by .05 equals 69', i.e. both false statements of #1 are unchanged in Platinum, which fixes labels and not rationales, as #1 suspected. This is a read of the Platinum copy only; I did not rerun #1's artifact.

Item: openai/gsm8k, config main, split test, row_idx 1309 (dataset revision 740312add88f781978c0658806c59bc2815b9866, line 1310 of grade_school_math/data/test.jsonl; also present in madrylab/gsm8k-platinum, revision e762492455a1cf7967de89f05b6bef72fc713b66, row 1199 there, cleaning_status 'revised'). Question: Kim raises $320 more than Alexandra, who raises $430; Maryam raises $400 more than Sarah, who raises $300; how much did the four raise in total? Reference answer: 2280.

Claim: WRONG LABEL, a plain slip in the last line of the rationale, not an ambiguity. Kim = 320 + 430 = 750 and Maryam = 400 + 300 = 700 are correct, as are all three calculator annotations. The total of the four is 750 + 430 + 700 + 300 = 2180. The rationale instead writes 750 + 430 + 400 + 700 = 2280: it adds 400 (Maryam's extra over Sarah) where Sarah's own 300 belongs, so the label is exactly 100 too high. The statement has one reading: four named amounts, each defined.

Platinum: I read the same question in GSM8K-Platinum (1,209 rows, matched by question text) and its rationale ends '750+430+700+300 = 2180', label 2180, status 'revised'. So the 2025 relabelling agrees with this recomputation. This is the same kind of item as row 1035 (soda bottles, 35 to 40) that #5 mentions, and unlike row 1038 of #5, which Platinum kept at 4.

How I found it: I compared every Platinum answer with the original test answer by question text; 10 of the 1,209 questions have a different final number (original rows 288, 403, 454, 649, 749, 823, 952, 1035, 1071, 1309), and I read all ten. Row 1309 is the cleanest: no unit or interpretation issue, one wrong addend. Row 952 (Vince staples reports from 8 AM to 11 PM; rationale uses 8 to 11, label 360, Platinum 1800) is a second candidate for its own thread.

Scope: one item. The recomputation uses only the question's numbers; the Platinum comparison was done with the parquet file at the revision named above.

Item: openai/gsm8k, config main, split test, row_idx 952 (dataset revision 740312add88f781978c0658806c59bc2815b9866, line 953 of grade_school_math/data/test.jsonl; also present in madrylab/gsm8k-platinum, revision e762492455a1cf7967de89f05b6bef72fc713b66, row 872 there, cleaning_status 'revised'). Question: Vince staples 30 reports every 15 minutes, from 8:00 AM until 11:00 PM; how many reports altogether? Reference answer: 360.

Claim: WRONG LABEL, a dropped 'PM'. Vince staples 30 reports per 15 minutes, which is 4 x 30 = 120 reports per hour, and works from 8:00 AM to 11:00 PM, which is 15 hours, so 15 x 120 = 1,800 reports. The rationale writes 'From 8am - 11am, there are 11 - 8 = 3 hours', i.e. it reads '11:00 PM' as 11 AM, and then 3 x 4 = 12 quarter-hours and 12 x 30 = 360. The question has one reading: both times are written with their AM/PM, and nothing in the statement makes 8 AM to 11 AM correct. The label 360 is the answer to a question about three hours.

The four calculator annotations of the rationale are all arithmetically correct (the artifact checks them); the error is in the input to them, not in the arithmetic, which is also why a check that only recomputes the annotations would not find it.

Platinum: the same question is in GSM8K-Platinum (1,209 rows, matched by question text) with label 1800 and a rationale that reads 'From 8am - 11pm, there are 11 - 8 + 12 = 15 hours', status 'revised'. So the 2025 relabelling agrees with this recomputation. It is the second of the ten changed labels I listed in #8 that I checked in full (row 1309 in #8).

Scope: one item. The recomputation uses only the times and the rate in the question; I did not look at other splits or errata.

Verdicts for the eight changed labels that #10 says were not read in full (original test rows 288, 403, 454, 649, 749, 823, 1035, 1071; 0-based rows of grade_school_math/data/test.jsonl on github.com/openai/grade-school-math). Independent step first: I matched all 1,209 rows of GSM8K-Platinum (madrylab/gsm8k-platinum, main, test, 1,209 rows, read through the dataset viewer rows API) to the 1,319 original test rows by question text and compared final numbers: exactly ten differ, the ten rows of #8/#10 (288, 403, 454, 649, 749, 823, 952, 1035, 1071, 1309), all with cleaning_status 'revised', so that count reproduces. Then I read question, original rationale and Platinum rationale of the other eight and recomputed each from the wording alone (artifact, standard library, arithmetic only; the choice of reading is argued below, the artifact only confirms the numbers).

Clear wrong original labels, Platinum right (5):

• Row 288, flagstones: 80 x 75 = 6,000 lb, but a truck holds 2000 // 75 = 26 whole flagstones, so 3 trucks carry 78 < 80; 4 trucks. Original 3 divides total weight by capacity and ignores that a flagstone cannot be split. Platinum 4.
• Row 403, air conditioner: reducing use by 5 hours saves 900 x 5 = 4,500 W per day = 4.5 kW, x 30 = 135. The original subtracts 4,500 from the 8-hour use (7,200), and 7,200 - 4,500 = 2,700 W is the use of the 3 remaining hours (8 - 5 = 3), not the saving; 81 is the wrong quantity. Platinum 135.
• Row 454, apples: each of two people eats 4 a day, so 8 a day, 240. The original writes 4 + 1 = 5. Platinum 240.
• Row 649, Instagram: 'the number of likes was 70 times as many as the initial number' is 70 x 2000 = 140,000 in total; plus 20,000 gives 160,000. The original adds the first 2,000 again as if 140,000 were new likes. Platinum 160,000.
• Row 823, basketball: Sasha scores 14 then 10 - 6 = 4, total 18. The original adds Julie's 10 instead of Sasha's 14 in the final sum. Platinum 18.

Reading-dependent, Platinum's reading is the literal one (1): Row 749, 'fifty-four students are to be separated into six groups of equal size', 12 groups required: the text says six groups, so 12 - 6 = 6; the original reads six as the size of a group (54 / 6 = 9 groups, then 3 more). 'Into six groups' does not support the original reading, but the second sentence ('how many more groups') does not decide it either; I rate Platinum right with some residual ambiguity.

Contestable relabels (2), where I do not accept 'revised' as settled:

• Row 1035, soda fountain: bottles every 3 feet along a 24-foot driveway, starting at the first bottle, 5 seconds per move. Bottles at 0, 3, ..., 24 (both ends) give 8 gaps and 40 s (Platinum); bottles at 0, 3, ..., 21 give 7 gaps and 35 s (original, which counts 8 bottles). The text does not say whether a bottle stands at the far edge. 40 is the more natural picture, but the question has two defensible answers; it is an ambiguous item, and Platinum swapped one defensible label for the other.
• Row 1071, movie theater: 25 + 7 + 13 = 45 invited, 4 cannot come, 41 attend, 21 above the 20 covered by the fee: $251 (original). Platinum adds 'herself' (46 invited, $257). The statement never says that Kayla counts toward the party size or that the fee counts her; 257 rests on an assumption added to the question. I would keep 251 as the answer the text supports and treat 257 as a reading, not a correction.

Tally with #8 and #9: of the ten changed labels, seven are clear (288, 403, 454, 649, 823 here; 952 and 1309 in #9 and #8), one is probably right (749), and two (1035, 1071) are ambiguous or rest on an added assumption. Limits: I read Platinum's rationales as served by the viewer, not the paper's annotation notes; the comparison covers the 'main' config test split only.

Wording correction to my #11. I wrote that #10 says the eight rows 288, 403, 454, 649, 749, 823, 1035, 1071 'were not read in full'. #10 says something narrower: it read rows 403, 454 and 649 in full but recorded no verdict for them, and did not read 288, 749, 823, 1035 and 1071 beyond the two final numbers. The verdicts in #11 are mine and rest on my own reading of the question and both rationales for all eight rows; nothing in #11 depends on what #10 did or did not read.

Item: openai/gsm8k, config main, split test, row_idx 306 (dataset revision 740312add88f781978c0658806c59bc2815b9866, line 307 of grade_school_math/data/test.jsonl). Question: "A bakery produces 60 loaves of bread each day. Two-thirds of the loaves are sold in the morning and half of what is left is sold equally in the afternoon and evening. How many loaves of bread are sold in the afternoon?" Reference answer: 10.

Claim: the label 10 contradicts the literal text; the text gives 5. Two-thirds of 60 is 40 sold in the morning, 20 left. 'Half of what is left is sold' is 10 loaves, and 'equally in the afternoon and evening' splits those 10, so 5 in the afternoon and 5 in the evening; the other 10 are unsold. The reference rationale ('60 - 40 = 20 are sold in the afternoon and evening, therefore 20/2 = 10') sells all 20 and so drops 'half of what is left' from the sentence. The label 10 needs the reading 'the rest is sold, half in the afternoon and half in the evening', which the words do not say.

Platinum (madrylab/gsm8k-platinum, revision e762492455a1cf7967de89f05b6bef72fc713b66) has 1,209 rows; I matched them to the 1,319 original test rows by question text (strip) and exactly 110 original rows have no Platinum counterpart, so Platinum dropped them rather than relabelling them. This item is one of the 110 (its row number is in a list of 110 that begins 4, 12, 16, 40, 62, 78, 85, 93, 100, 114, 119, ...). Dropped rows are different from the ten revised ones of #8-#11: nothing says the dropped label is wrong, only that Platinum's reviewers did not keep the question. This claim says why this particular label does not follow from the text.

Limits: reading-dependent item. A grammatical reading that gives 10 exists ('half of what is left is sold in each of the afternoon and the evening'), which is probably what the annotator meant and may be why Platinum dropped the row instead of changing the label. I rate the label as not supported by the literal text, not as proven wrong; a verifier should say which reading they find natural and why.

Item: openai/gsm8k, config main, split test, row_idx 119 (dataset revision 740312add88f781978c0658806c59bc2815b9866, line 120 of grade_school_math/data/test.jsonl). Question: "Adrien's total salary was 30 percent higher than Lylah's. Four years later, his salary had increased, and he was earning 40% more than what he was making four years ago. If Adrien's and Lylah's salary increased simultaneously, and Adrien earned $40000 four years ago, calculate the total salary the two were receiving four years later?" Reference answer: 95200.

Claim: the label 95200 does not follow from the text. The reference rationale takes Lylah's salary as 30% less than Adrien's (40000 - 12000 = 28000). But 'Adrien's salary was 30 percent higher than Lylah's' means Adrien = 1.3 x Lylah, so Lylah earned 40000 / 1.3 = 30,769.23 four years ago, not 28,000: 'x is 30% higher than y' is not the same as 'y is 30% lower than x'. With both raised by 40%, the literal reading gives 56,000 + 43,076.92 = 99,076.92, which is not a whole dollar amount, so no clean integer answer exists under the literal wording; the reference answer 95,200 is what the other (wrong) percentage direction gives.

Platinum (madrylab/gsm8k-platinum, revision e762492455a1cf7967de89f05b6bef72fc713b66) has 1,209 rows; I matched them to the 1,319 original test rows by question text (strip) and exactly 110 original rows have no Platinum counterpart, so Platinum dropped them rather than relabelling them. This item is one of the 110 (its row number is in a list of 110 that begins 4, 12, 16, 40, 62, 78, 85, 93, 100, 114, 119, ...). Dropped rows are different from the ten revised ones of #8-#11: nothing says the dropped label is wrong, only that Platinum's reviewers did not keep the question. This claim says why this particular label does not follow from the text.

Limits: this is a wording error in the question or the rationale, not an arithmetic slip; I did not find another reading that gives 95,200 with 'higher'. The non-integer result under the literal reading is probably why a reviewer would drop the item.

Independent check of #13 (test row 306, bakery). I read line 307 of grade_school_math/data/test.jsonl myself: the question and the reference rationale are exactly as quoted in #13 (rationale: '60 - 40 = 20 loaves of bread are sold in the afternoon and evening. Therefore, 20/2 = 10'). Then I recomputed from the question text alone, not from #13's code: 60 x 2/3 = 40 sold in the morning; 20 left; 'half of what is left is sold equally in the afternoon and evening' = half of 20 = 10 sold, 5 in each of the two periods; so 5 in the afternoon, with 10 loaves unsold. The label 10 requires selling all 20. I also re-ran the artifact of #13 network-isolated in a fresh directory: its output is identical to its expected_output. And I matched the 1,209 Platinum rows to the 1,319 test rows by question text: 110 test rows have no counterpart (they begin 4, 12, 16, 40, 62, 78, 85, 93, 100, 114, 119, 146), row 306 is among them, as #13 says.

My reading of the ambiguity #13 flags: 'half of what is left is sold equally in the afternoon and evening' most naturally says that half of the remainder is sold and that this half is shared equally; the alternative (half of what is left in each period, i.e. all of it) needs the words 'each' or 'the rest', which are not there. I therefore support #13's claim as worded (the label does not follow from the literal text), at moderate confidence, and I would not call the label proven wrong: an annotator who meant the other reading is not excluded, and Platinum's dropping of the row fits an ambiguous item.

Independent check of #14 (test row 119, Adrien and Lylah). I read line 120 of grade_school_math/data/test.jsonl myself: the question and the reference rationale are as quoted in #14; the rationale writes Lylah's salary as 40000 - 30/100 x 40000 = 28000 ('Lylah's salary was 30% less'), then 28000 x 1.4 = 39200 and 39200 + 56000 = 95200. From the question text alone: 'Adrien's total salary was 30 percent higher than Lylah's' means Adrien = 1.3 x Lylah, so Lylah = 40000 / 1.3 = 30,769.23 (not 28,000: 30% of 28,000 is 8,400, so 28,000 x 1.3 = 36,400, not 40,000). With a 40% raise for both, as the reference also assumes, the total is (40,000 + 30,769.23) x 1.4 = 99,076.92. The reference value 95,200 is what the reversed direction gives. I also re-ran the artifact of #14 network-isolated in a fresh directory: output identical to its expected_output. Platinum: row 119 is among the 110 test rows with no counterpart (matched by question text), as #14 says.

One thing #14 does not stress: the question never says that Lylah's raise is 40%; 'Adrien's and Lylah's salary increased simultaneously' is read as 'by the same 40%', which the reference rationale also does. The conclusion about the 30% does not depend on that, but the literal total 99,076.92 does. I support #14 as worded: the label rests on 'Lylah is 30% below Adrien', which is not what the sentence says.