presci-6L256d: a language model that stops at 1543
What Chinchilla-optimal looks like at 17.7M parameters, and what a date boundary in the training data actually buys

presci-6L256d is a 17,665,792-parameter GPT – the architecture from Raschka's Build a Large Language Model (From Scratch) (Raschka 2024) – pretrained from scratch on 306 English texts, every one composed before 1543. No pretrained weights were used. It has never seen modern text.

1543 is De revolutionibus, taken as the cut for the scientific revolution. The model's world is pre-Copernican as a matter of training data rather than of prompting, which is a different and more interesting thing than a system prompt telling a model to pretend.

The selection rule is executable rather than conventional: entries dated at or after the cut are rejected, as are straddling works with no explicit dating note, duplicate identifiers, and malformed registers. A model whose training corpus has a machine-checked boundary is a controlled instrument. One whose boundary is a described intention is an anecdote.

The model card written after the first run contained a claim about the second, made before it existed:

This model is trained at 2.0 tokens per parameter against a Chinchilla-optimal ~20: it is undertrained by an order of magnitude, and the fix is another 18 hours rather than another machine. Corpus size is not the binding constraint at this scale; patience is.

That is falsifiable, so it was worth running.

Chinchilla-optimal at this parameter count (Hoffmann et al. 2022) is 20 tokens per parameter, or 353,315,840 tokens. The second run landed at 19.94. That was the target, not a coincidence.

What the prediction got right. The card measured 4,790 tokens/second on the laptop and derived 20.5 hours for the Chinchilla budget. The run actually happened on an M4 Mac mini, which sustained 5,895 tokens/second – 1.23× faster. The same token budget at the measured speed predicts 16.6 hours. The run took 16.6 hours.

What that does and does not establish. It establishes that throughput is stable across a run roughly ten times longer, and that a token-budget-to-wall- clock extrapolation holds at fixed model size. It establishes nothing whatever about the 1/N scaling across model sizes that every later row of the table below depends on, because both runs used the same N. The interesting rows remain untested, and saying otherwise would be the exact error this project exists to catch.

What ten times the compute bought. 9.8× the tokens moved validation loss from 3.7455 to 3.4034 – 0.342 nats, perplexity 42.3 to 30.1, a factor of 1.41. That is the shape the scaling literature predicts (Kaplan et al. 2020) and it is worth sitting with: an order of magnitude of compute, for a 1.4× improvement in a metric that is itself only loosely coupled to whether the output is any good.

The constraint is the lesson, not an apology for one. Extrapolating at 6ND from measured throughput:

The first two rows are measured. Everything below them is arithmetic, and optimistic arithmetic at that: throughput does not scale as 1/N, attention is quadratic in sequence length, memory bandwidth binds before arithmetic does at larger widths, and none of it accounts for the batch sizes those runs used. Every figure past row two is wrong in the direction that makes the point weaker, which is the only direction it is safe to be wrong in here.

The smallest model in the book's own lineage is two years away on this hardware. That is the edge this sits against – a scale at which a complete pipeline can be built, run, and verified end to end, which is precisely the property that does not survive being scaled up.

It removes post-1543 concepts from the source works. It does not remove them from the artifact.

Roughly a quarter of a sampled batch of raw corpus texts flags for post-cut vocabulary – almost all of it from 19th-century translators' prefaces and editorial apparatus rather than from the works. Cleaning removes most of it. Some survives.

So the honest statement is: this model has never read a modern physics textbook. It has quite possibly read a Victorian scholar explaining one.

That gap between corpus and artifact is the recurring difficulty in building a period corpus from modern editions, and it generalises past this project. A single page of a 1922 edition of a c.1300 English arithmetic can carry three texts of three dates: the c.1220 Latin verse, the Middle English commentary, and the editor's sidenote glossing it in modern English. Only one of those is the thing you meant to collect.

The tokenizer is GPT-2 byte-level BPE, 50,257 tokens, unchanged. At emb_dim 256 that vocabulary is enormous relative to the model: about 73% of the parameters are the embedding table. The model is mostly a lookup table with a transformer attached.

It is also fitted to modern web text. A GPT-2 tokenizer spends 3.88 tokens per word on Old English against 1.37 on modernised prose – so extending the corpus toward genuinely early material buys roughly a third of the content per token of budget, the vocabulary going on morphology rather than meaning.

The consequence is an ordering claim: at this scale, settle the tokenizer before the corpus. A corpus decision that looks like a gain in authenticity is a 2.8× loss in effective context.

It is a base model – next-token prediction only, no instruction tuning, no chat template. It continues text; it does not answer questions.

Context is 256 tokens, and shorter than that in words for the reason above.

Perplexity 30 is not fluency. It produces grammatical sentences with convincing period cadence that do not sustain an argument across a paragraph:

> And it came to passe in those dayes
  that was in the place of the temple, and the city of the temple of God came
  into the house of Israel, and they pitched upon the throne of the sanctuary

Cadence, register and vocabulary are right. Reference is not: the temple is in the place of the temple. This is what a 17.7M-parameter model at 20 tokens per parameter sounds like, and publishing the sample rather than describing it is the point.

It will also assert a geocentric cosmos and humoral medicine – correctly reflecting its training data and incorrectly reflecting the world.

This note and the weights go public together. Before that:

• [ ] Confirm the repository is private until the moment of release. It was briefly public during upload: hf upload auto-creates a missing repository and the auto-created repository is public, with no --private flag on the upload subcommand. Create first, then upload, then verify – none of those three steps tells you what you got.
• [X]

  Corpus licence audit – done, and made executable. The weights licence is now computed from the corpus rather than asserted: a gate maps each source host to a rights class, derives the most permissive licence those sources can support, and refuses to package if the declared licence exceeds it. An unrecognised host fails rather than warns. All 306 sources are Project Gutenberg, so the ceiling is unrestricted and the declared licence is CC BY 4.0 – changed from Apache-2.0, which is a software licence whose patent grant does not map onto a tensor file.

  The distinction that motivated it: "public domain" is a property of a work, licences attach to editions, and Gutenberg texts are public domain in the United States specifically. A model licence cannot grant more than its corpus permits, and nothing checks that by default.

• [ ] State in the card that no loss figure was measured on the exact tensors shipped. Evaluation runs on a fixed interval and the final checkpoint is written after the loop exits, so the released weights are 250 steps past the nearest measurement.
• [ ] Decide whether 12k is published alongside 172k or discarded. The pair is more useful than either alone, since the pair is what makes the compute claim checkable.

The released model card lives with the weights. Its structure, for reference: architecture and a three-way parameter reconciliation (raw state_dict 30,924,800; minus the tied output head 18,059,008; minus causal-mask buffers 17,665,792 – one number is ambiguous when weights are tied and buffers are stored); training data and the cut; the two versions above; usage, which requires the shipped modeling_presci.py because this is not a transformers architecture; limitations; and safety.

On safety, briefly: the corpus predates 1543, so the model has no vocabulary for modern harms – it cannot describe technologies that did not exist. It reproduces the attitudes of its sources, which include period religious polemic and the social assumptions of pre-modern Europe. That is a property of the corpus and is not mitigated.