The headline parameter count is misleading. Three quarters of it is a lookup table; the reasoning budget is 4.8M parameters.

Three measurements over the training corpus, each ruling out a different explanation for the failure.

It has the numerals. A 60-text sample (38.5M characters) contains 95,135 integers — 2,468 per million characters. Every integer 0–99 is a single GPT-2 token and every one of those 100 tokens appears in the corpus. Those embedding rows are trained.

It has never seen an operation. The same 38.5M characters contain sixteen expressions matching \d+ \s* [+-×x] \s* \d+ \s* =. Sixteen. The model has read numerals as nouns — regnal years, chapter and verse, counts of ships — and has essentially never seen one used as an operand.

It has never seen the notation. Against the 2,543 distinct tokens in MATH-500's problems, 99 ids never occur in the corpus at all — 3.8% of the benchmark's problem tokens. They are not exotic vocabulary; they are the skeleton:

Of 306 texts, exactly one is mathematical — Proclus's commentary on Euclid Book I. Philosophy about mathematics, containing no calculation.

So there are three separable deficits: numerals (present), operations (absent), notation (absent). They need different remedies, and conflating them is the first mistake available.

0–99       : 100% single-token
100–999    : 79.9% single-token, 20.1% split
1000–9999  :  1.0% single-token, 98.7% two-token, 0.3% three-token

The count is not the problem. The shape distribution is. Across all 9,000 four-digit numbers there are five distinct split shapes:

1234 -> ['12','34']        5678 -> ['5','678']
1543 -> ['15','43']        2019 -> ['2019']      1024 -> ['1024']

Place value is not aligned across tokens, and it is not even consistent between two numbers of the same length. The single-token four-digit numbers are years and powers of two — corpus artifacts of whatever GPT-2's BPE was fit on. From the model's side, the tokenizer's treatment of numbers is arbitrary.

Also: '7' is token 22 and ' 7' is token 767. Different objects entirely.

And the LaTeX commands are multi-token sequences that cannot be decoded from parts — \frac is 2 tokens, \sqrt and \cdot and \boxed are 3 each. In literacy-curriculum terms these are sight words, and that framing falls out of the BPE rather than being imposed on it.

Surveyed and verified live: deepmind/math_dataset (112M rows), AMPS via XinyaoHu/AMPS_mathematica (4.8M rows, genuine LaTeX, 137 sub-topics), math-ai/TemplateGSM, apple/GSM-Symbolic, nvidia/OpenMathInstruct-2 (14M rows), allenai/lila, meta-math/MetaMathQA, open-web-math, EleutherAI/proof-pile-2.

Rejected as the primary path. The register is wrong — every one of them is modern English, and fine-tuning a 4.8M-parameter transformer on modern mathematical prose will drag the pre-1543 voice with it. Some are also encumbered: apple/GSM-Symbolic is CC-BY-NC-ND-4.0, and "NoDerivatives" is a poor fit for training data.

Rejected on arithmetic. At ~62 tokens per problem a corpus of a few hundred thousand problems is hours of generation and hundreds of dollars, is not reproducible from a seed, and adds the circularity of teaching one language model from another's output. Volume is the one axis where programs beat models outright.

The one thing an LLM is worth using for is writing the grammar — perhaps 10,000 tokens of templates, once — rather than the corpus.

This is the canonical prior art (Saxton et al., arXiv:1904.01557, ICLR 2019) and it is genuinely good: 56 train modules, 56 interpolate, 15 extrapolate; difficulty as a sampled entropy budget sliced into thirds rather than hand-tuned levels; SymPy load-bearing for construction; Apache-2.0.

Two practical problems and one philosophical one.

• It emits plain ASCII, not LaTeX: "Find the second derivative of -191*q**4 - 6*q**3 + 5*q**2 + 13263054*q - 2." It has the architecture and not the surface form, and the surface form is the whole point here.
• It calls ndarray.itemset, removed in NumPy 2.0. Running it means a separate venv pinned to numpy<2.
• And the philosophical one, below.

Fast — 35M tokens/second — and wrong. My first prototype emitted \sqrt{12} unsimplified and computed a distance as a float. A template asserts its answer; nothing checks it. A corpus meant to teach mathematics cannot contain wrong mathematics.

Rejected, twice over, both measured:

• 3,838 examples/second against a plain seeded RNG's 2,812,872 — 733× too slow.
• Its distribution is adversarial by design. Drawing integers(1,30), the most frequent draws came back 1, 2, 30, 26, 3 — boundaries first, near-perfectly flat. Exactly right for finding bugs; exactly wrong for training data, where over-representing 1 and 30 teaches the model those numbers are special.

Fast enough that correctness is free, and the answer is derived rather than asserted:

naive template:  \sqrt{12}       SymPy:  2 \sqrt{3}
naive template:  \sqrt{50}       SymPy:  5 \sqrt{2}
                                 SymPy:  Solve $18x + 22 = 20$ -> $x = -\frac{1}{9}$

Not generating the corpus — auditing the generator. Run as an emitter it produced 3,000 verified examples in 1.7 s with zero property violations, and immediately surfaced this:

Solve $x = 1$ for $x$. The answer is $\boxed{1}$.
Evaluate $1^{2} - 1^{2}$. The answer is $\boxed{0}$.

Both technically valid, both worthless, and both of which a uniform sampler emits at a rate low enough to escape notice until they are sitting in the corpus. The emitted log is a review artifact; you read it, add the missing constraint, regenerate.

The property worth the most is the cumulative one: no unit may emit a token id outside its declared set. That is the machine-checkable form of the constraint that makes a scope-and-sequence a sequence rather than a list, and a plain unit test cannot falsify it.

One measured caveat: hypothesis.target() aimed at token coverage worked mechanically but reached only 5 of 99 missing ids, because those tokens are not in the strategies at all. Coverage is grammar-bound. No amount of search fixes a grammar that cannot express the token.

Nobody has demonstrated reliable small-model arithmetic on top of GPT-2 BPE. Lee et al. (arXiv:2307.03381) get ~100% on 3-digit addition from a 10.6M-parameter NanoGPT — but with a character-level tokenizer, vocabulary 80. Their GPT-2 experiments only work after inserting spaces between digits to force uniform tokenization. Nogueira et al. (arXiv:2102.13019) find models fail at 5-digit addition with subword tokenization and succeed to 60 digits with explicit position tokens. Wallace et al. (arXiv:1909.07940) find character-level embeddings the most numerate and subword the least.

A 4.8M-parameter transformer has no capacity to spend learning that ['12','34'] and ['5','678'] are the same kind of object. Mitigation is to space-separate digits in generated text; the cost is that the corpus then looks unlike MATH-500, which is what we wanted to evaluate against. This tension is unresolved and is the biggest open risk in the design.

The same paper: plain A+B=C formatting plateaus at ~85% no matter how much data, while a detailed scratchpad reaches ~100% at 1,000 samples. Reverse-order answers phase-transition at ~2,500.

Two consequences. First, 17.7M parameters is above, not below, the demonstrated floor for arithmetic — the capability gate is data format, not model size, which reframes this whole project. Second, the generator should emit derivation steps, not just answers. SymPy gives intermediate expressions for free and not emitting them wastes its main advantage over scraped text.

My first plan was 20M synthetic tokens, reasoned from "30% of the existing corpus". But if the phase transition is 1,000–2,500 examples per skill, then ~35 skills is under 90,000 examples — about 5M tokens. Saxton's 2M per module was sized for 100M-parameter seq2seq models across 56 mixed modules.

The budget belongs in skill coverage and format, not per-skill volume. And because the accuracy curves are step functions, small eval sets will be very noisy near the transition.

Under uniform sampling of two operands in 0–999, the carry distribution comes out skewed and only 1.0% of draws are single-digit:

Lee et al. instead force balanced coverage across digit counts and carry counts, and it improves accuracy in every slice. Charton (arXiv:2308.15594) shows the same lesson with a 2.4× effect: on learning GCDs, uniform operands yield 38 correct values ≤100, log-uniform operands 73, and log-uniform outcomes 91 — same model, same task, distribution alone.

The generalization is to balance over the answer distribution, not just the input distribution. This is cheap and I would not have thought of it.

Fine-tuning on modern mathematical prose will pull the model's voice toward modern prose generally. The mitigation is to emit problems in two registers — modern LaTeX for benchmark compatibility, and period English modelled on Recorde's Grounde of Artes — plus a replay fraction of the original corpus.

Held-out validation loss on the original corpus, measured before and after, is the detector. If it rises materially the model is paying for mathematics with its voice.

Recent work (arXiv:2510.19099) finds no curriculum ordering that dominates universally across five difficulty definitions and three models. Skill-It (arXiv:2307.14430) reports large gains, but from online resampling driven by measured per-skill validation loss, not a fixed easy-to-hard sweep. Saxton's own headline numbers are the uniform mix.

So the curriculum is an ablation, not an assumption, and it ships with a uniform-mix control run. Without the control nothing is attributable.