algorism: when the source text is the test suite
Process supervision without a judge, because the medieval text already wrote down its own intermediates
Table of Contents
Status: written ahead of release. The dataset is private at the time of writing; this note goes live with it.
1. The problem with grading a translation
Benchmarks for autoformalization have an oracle problem. ProofNet grades by whether a prover closes the goal, which needs a prover. MMA grades by model judgement, which needs a judge and inherits its failures. Neither can cheaply decide whether this formal rendering means what that informal statement meant.
Medieval arithmetic solves it by accident. These are teaching texts, and they show their work.
2. Mediation, and the state the text writes down
From The Crafte of Nombrynge (c. 1300, sole witness BL Egerton 2622), halving 245:
lo an Ensampull. 245. the first figure here is betokenynge odde nombre, þe quych is 5, for 5 is odde; þerefore do away þat þat is odde, þe quych is 1. þen leues
- þen medye 4 & þen leues 2. do away 4. & sette þere 2, & make such a merke ʷ
upon his hede … as þus. 242.ʷ
The answer is 245/2. A Scheme rendering reading (/ 245 2) reproduces it and
captures nothing of the method.
But the passage does not only state the answer. It states an intermediate:
242.ʷ – the units figure already halved, the half-mark set, the other figures
untouched. (/ 245 2) cannot produce that state at any point in its execution,
because it never has one.
So each record carries two tiers of assertion, and we use the vocabulary of Lightman et al. (Lightman et al. 2023) rather than inventing our own:
| tier | asserts | kills |
|---|---|---|
| outcome | the final value the text states | arithmetic that is simply wrong |
| process | intermediate states the text names | a right answer by a method the passage does not describe |
The distinction is theirs; what is different here is the source of the process labels. Their process supervision required human annotators rating each step of a solution – the paper's contribution is partly the expense of collecting 800,000 such labels. Ours were written down in the fourteenth century by someone teaching a boy to halve numbers on a slate, and they are checkable by running the code rather than by asking a model.
That is the whole idea. The verbosity that makes medieval arithmetic tedious to read is exactly what makes method fidelity mechanical.
3. The obvious objection
EvalPlus is the adversarial case. Its thesis is that HumanEval's tests were too weak and that the remedy is more inputs – it grows the test set roughly eightyfold. Why not simply assert more input/output pairs?
Because digit-wise halving and (/ n 2) are extensionally equivalent. They
agree on every input in the domain. No quantity of I/O pairs separates them,
because there is no input on which they differ. Only intermediate state does.
That is the precise claim and it is the only thing justifying the second tier. Where two methods are extensionally distinguishable – one overflows, one is defined on a narrower domain – more inputs are the correct and cheaper tool, and process assertions are not warranted.
An honest limit: process assertions pin the states the text happens to name. A translation can still deviate between two named states. Records naming no intermediate carry an empty process list and may not be counted toward any process figure. Reporting one number over both populations would claim method fidelity for records where none was checked.
4. Two findings that were not the point
4.1. OCR destroys the prose and spares the numerals
The binding constraint on building this from historical sources is not
copyright. Measured across 599 KB of Middle English in archive.org's
_djvu.txt of Rara Mathematica (1839), against a hand transcription of the
same tradition:
| signal | OCR edition | hand transcription |
|---|---|---|
thorn þ |
0 | 2825 |
yogh ȝ |
0 | 81 |
J: / )7 / f^ damage |
52 | 0 |
Zero thorns in a Middle English text is not an editorial choice. The OCR destroyed precisely the characters that make it Middle English – and left the numerals intact.
That is the wrong way round. The arithmetic reads correctly while the passage is corrupt, so a numeric extractor harvests valid worked examples wrapped in unusable prose, and nothing flags it. A pipeline that checked only "did we get the numbers" would report success.
The dataset therefore requires a transcription field whose enumeration offers
hand-transcribed, TEI-encoded and OCR-corrected. Raw OCR is deliberately not
offered: it cannot support a byte-identical verbatim claim however good its
numerals look.
4.2. The licence risk is in the edition, not the work
The works are public domain by centuries. The editions are not automatically, and the trap is specific.
The Oxford Text Archive serves about 69,800 CC0 texts alongside a 335-item "Academic Use" set holding Chaucer, Langland, the York cycle and the Helsinki Corpus – the best Middle English TEI anywhere, and the single most tempting material in the field. Its agreement permits duplication only for your own research or teaching and states copies "may not be offered, whether for sale or otherwise to anyone who is not an authorised user". Publishing to an open hub offers them to non-authorised users regardless of price, so nothing from that set can enter a published dataset – not even under a NonCommercial licence.
Two things make this tractable rather than fatal. The restriction attaches to
the Oxford files, not to the works: Chaucer has been public domain for six
centuries, and the same text from a public-domain source carries none of it.
And the rule is enforceable rather than remembered – each record's license
field is restricted by JSON Schema to an enumeration of redistributable values,
so a NonCommercial edition is rejected by the validator rather than by
somebody's memory.
5. Standard eval formats cannot express this
Worth recording because it is a general result, not a local inconvenience.
inspect-ai supports loading a benchmark straight from a dataset repository via
an eval.yaml config – the convention MATH-500 uses, itself drawn from the
Lightman paper's problem subset. It is the natural distribution mechanism.
It cannot express this benchmark. The schema restricts the scorer to a closed
set of ten built-ins – includes, match, pattern, answer,
exact, f1, model_graded_qa, model_graded_fact, choice, math – and forbids unknown keys.
Every one of them either compares strings or asks a model. None executes
anything.
That is a reasonable design for benchmarks graded by matching or by judgement,
which is nearly all of them. It means execution-graded benchmarks, and process
grading in particular, fall outside what a declarative eval config can say. So
the dataset ships both: an eval.yaml expressing the answer-only tier, with a
warning in its first paragraph that a number from it is not the benchmark score,
and a Python task carrying the real scorers.
The gap between the two numbers is itself a measurement – how often a model gets the value right by a method the text does not describe.
6. What it is not
Small by construction. The usual way to grow an informal-to-formal corpus is back-translation from formal statements; that is unavailable here, because the informal side must be genuine period English and a model writing plausible Middle English produces exactly the artefact this is a control against.
The nearest existing work is nuprl/MultiPL-E, whose Racket configuration is
the only other execution-graded natural-language-to-Lisp benchmark. It is
larger, mechanically derived from HumanEval, and covers 47 languages. This
differs on two axes only: historical prose rather than modern docstrings, and
process-level rather than outcome-only grading.