bac and P-99: an external corpus for a clean-room reconstruction
Table of Contents
1. What is under test
bac reconstructs Clojure from observed behaviour, in Python, without reading
Clojure's implementation source. The deliverable is a paper rather than a
language: a rung that stalls counts as a result provided the stall is written
down.
Two oracles answer different questions, and keeping them apart is most of the method.
| oracle | what it is | what it answers |
|---|---|---|
| sequence | --filter=blob:none --no-checkout clone of clojure/clojure — commits, trees, tags, no file contents |
what to build next |
| behavioural | a Clojure jar with every *.clj removed by zip -d, run as a subprocess |
whether an answer is correct |
The second is the interesting one. Clojure boots from AOT-compiled classes, so
the stripped jar still runs: (doc map) returns a docstring, (source map)
returns Source not found. A zip operation enforces the clean-room boundary,
which is cheaper and harder to violate than a rule about what you may read.
2. Thirty primitives, and the test that admits them
The kernel is what the Python host must expose before a core.clj analogue can
define the remainder. The count is a claim, so admission is adversarial: could this
be written in Clojure using only what is already in the kernel? If yes, it
stays out.
+ is in and inc is out, because (def inc (fn* [n] (+ n 1))) needs nothing
from the host. prn is in because printing is I/O. count and first are in
because they read the host's representation of a collection.
| group | primitives | n |
|---|---|---|
| arithmetic | + - * / quot rem |
6 |
| equality and order | = compare < > <= >= |
6 |
| collections | count seq first rest cons conj get nth assoc vector list |
11 |
| printing | prn println pr-str str |
4 |
| host reflection | type class Exception |
3 |
Ten special forms complete it: quote if do def let* fn* loop* recur throw try.
There is no macro system, so defn, let, cond, when, and, or and
-> do not exist. All seven are macros in Clojure — though let is a trap
for this framing, since its var carries :special-form true and clojure.org
documents it under Special Forms, even while it expands to the let* special
form.
A loop* with no recur binds locals and stands in for let. That holds only
under the stated qualifier: with a recur present the two diverge, because
loop* steals the recur target and the let* version compiles where the
loop* version does not.
3. The library hypothesis
The claim under test is that what separates a reconstruction from ordinary list
programming is a library, not the bridge to the host. core/prelude.clj
defines 34 functions using only the 30 primitives and the 10 special forms, and
the kernel did not change while it was written.
;; each name means what clojure.core means by it
(def map
(fn* [f xs]
(loop* [in (seq xs) out (list)]
(if (= in nil)
(reverse out)
(recur (next* in) (cons (f (first in)) out))))))
P-99 calls 15 of the 34, directly or transitively. It never reaches the remaining 19 — a fact that turns out to set the ceiling on what mutation can detect.
4. Why the corpus is external
P-99 is Werner Hett's Ninety-Nine Prolog Problems, written for a Prolog course with no knowledge of this project. That provenance is the whole reason to use it.
An earlier rung built its corpus alongside the implementation. It reported 28 agreements out of 28, then detected 6 of 10 planted defects. One author writing both the implementation and its tests applies the same assumptions twice, and the agreement rate says nothing about either.
The grading is deliberately asymmetric. bac loads the prelude and then the
solutions; the oracle loads only the solutions, on top of the real
clojure.core. So the oracle grades prelude and solutions together, and a
defect in our filter surfaces as a divergence in every solution that calls it.
Loading the prelude into both sides would compare the solutions against
themselves.
No translation step exists. The same source text runs on both, because the solutions stay inside the subset the two languages share.
5. Results
39 expressions over 35 solutions, all returning what the sealed jar returns.
| problems | content | result |
|---|---|---|
| P01–P22 | list ops: last, k'th, reverse, palindrome, flatten, compress, pack, run-length encode and decode, duplicate, drop, split, slice, rotate, remove, insert, range | 24/24 |
| P26, P28 | combinations, sort by length | 3/3 |
| P31–P41 | primality, gcd, coprimality, Euler totient two ways, prime factorisation, Goldbach | 12/12 |
(count (combinations 3 (range 1 13))) 220 (prime-factors 315) (3 3 5 7) (prime-factors-mult 315) ((3 2) (5 1) (7 1)) (encode-modified [:a :a :a :a :b :c :c]) ((4 :a) :b (2 :c)) (goldbach 28) (5 23) (goldbach-list 9 20) ((3 7) (5 7) (3 11) (3 13) (5 13) (3 17))
Three exclusions, each following from the problem rather than from bac.
P23–P25 select at random, so two correct implementations disagree on every run
and a value oracle reports a divergence where no defect exists. P27 returns
groupings whose order the problem leaves unspecified, and the oracle provides no
canonical form.
The third exclusion is stated wrongly at source. The writeup gives P54 and above
as needing map and set literals, which the reader does not parse. Checked
against the problems: P-99 contains no associative or set literal anywhere.
P54 onward are binary trees written as Prolog terms t(X,L,R) with nil, or as
plain lists (X L R) in the Lisp variant, and the graph section explicitly
simulates sets with sorted lists — "the lists are kept sorted, they are really
sets, without duplicated elements". A reconstruction needs cons, car, cdr,
nil and symbol equality, nothing more. If those sections are genuinely out of
reach, the reason is nested tagged structures, not literals.
6. What the corpus found
It was not green when written. Four defects and one priced deviation.
6.1. catch did not catch host refusals
An earlier rung added a HostRefusal exception so (+ 1 :kw) would refuse
cleanly rather than raise a Python TypeError, and did not touch _try, which
caught only Clojure-level throw. On the oracle all three of these return
:caught, because Clojure raises ClassCastException and
IllegalArgumentException and both descend from Exception. bac propagated
all three.
(try (+ 1 :kw) (catch Exception e :caught)) (try (seq :a) (catch Exception e :caught)) (try (count :a) (catch Exception e :caught))
6.2. concat treated () as nil
The loop guard read ( in nil)=. next* normalises an exhausted sequence to
nil, so the guard is right for every value the loop computes — but the
loop's initial value is (reverse a), which is () for empty input, and
() is not nil. An empty left argument therefore prepended (first ()),
which is nil.
Sixteen spot-checks written beside the prelude passed, because every one of them
concatenated two non-empty lists. (reduce concat (list) xs) starts from the
empty list. P26 counted 232 combinations where the answer is 220 — reported,
not reconstructed. The excess of 12 is exactly the input length, which is
suggestive, but sixty-four modelled variants of a buggy combinations recurrence
produce 232 in none of them. Take the figure as observed output rather than as
an explained mechanism.
That is the shape of the whole argument in one defect: the tests written next to the code shared the code's blind spot, and an external corpus did not.
6.3. type returns the host's type names
(type (list 1)) is tuple in bac and clojure.lang.PersistentList on the
oracle. The deviation is deliberate and documented — returning
java.lang.Long for a Python object would state something false that a reader
would repeat.
The corpus did not find the decision. It found the cost, which nothing
recorded: source that dispatches on type cannot run on both. P07 flatten
must tell an atom from a list, so the solutions use try=/=catch instead —
which is what made the catch defect blocking rather than cosmetic.
6.4. The harness graded programs on their first form
read-string reads one form, and the harness passed whole programs to it. A
program shaped (def f ...) (f 10) was graded on the def, with the expected
value coming back as #'user/fact. Uncorrected, the harness would have marked
bac incorrect for returning the right answer.
7. Planted defects
39 of 39 establishes nothing until the corpus is shown capable of failing.
Fourteen defects were planted in the prelude, one at a time. The prelude is the
right target: a defect in filter reaches thirty problems, one in slice
reaches one.
| outcome | plants | n |
|---|---|---|
| corpus detected | even-is-odd, map-drops-first, filter-inverted, reduce-wrong-order, range-off-by-one, take-one-extra, drop-one-short | 7 |
| timeout detected | reverse-drops-last, empty-never-true, concat-loses-empty-left | 3 |
| escaped | inc-off-by-one, last-returns-first, every-vacuous, some-never-finds | 4 |
Ten plants produced a failing run, but three of those failed because the mutated prelude did not terminate and the timeout ended it — no values were compared. So the corpus detected 7 of 14, and the honest headline is that number rather than the ten.
The four escapes are not mysterious. They modify inc, last, every? and
some, and P-99 calls none of them. They live in the 19 definitions the corpus
never reaches. Coverage, not sensitivity, is the binding constraint.
7.1. The plants exposed a harness defect
reverse-drops-last replaces a loop's exit test with one that is never true, so
reverse ran forever. Stopping it sent SIGKILL, the finally block never
ran, and the planted defect stayed in core/prelude.clj. The next command
loaded the planted prelude and also ran forever.
Non-termination is a common outcome of mutating a loop*=/=recur prelude,
because breaking an exit test is among the simplest available mutations. A
mutation harness without a timeout has no outcome for those plants at all. The
harness now applies a 120-second timeout, restores the file from atexit and
signal handlers, and refuses to start when core/ holds uncommitted changes.
8. Where the limit is
The library hypothesis held. Thirty primitives sufficed to define map,
filter, reduce, range, concat, take and drop in the language, and
the list and arithmetic sections run on those definitions with no kernel change.
Three limits remain, and the bridge is none of them.
- No map or set literals.
{:a 1}is a reader error. Idiomatic Clojure uses maps throughout and P-99's tree and graph sections require them. The largest gap. - No macro system. Mostly syntactic, and measurable in one place:
somecalls its predicate twice on a match, because calling it once needs a local binding in theiftest position andloop*cannot bind there. - Non-tail recursion stops at 141 frames, each
bacframe costing about seven Python frames against CPython's default limit of 1000 — adjustable withsetrecursionlimit, not a hard bound.loop*=/=recurhas no such limit; a run of 1,000,000 iterations completed. The JVM figure of roughly 10,000 frames measured as 11,578–12,340 on a default main thread, and it moves with stack size: 8,406 on a 512 KB thread, 172,246 on an 8 MB one. A quantitative difference between hosts, and one that depends on how you ask.
9. Reading map
- Type Systems for Software Developers: One Tool, Four Ways — the same question about what a discipline can express, asked of four languages rather than one harness
- Shipping the Model — why a model that cannot disagree with its implementation is uninformative, which is the self-written-corpus problem in another domain
- P-99, Werner Hett