bac and P-99: an external corpus for a clean-room reconstruction

Table of Contents

1. What is under test

bac reconstructs Clojure from observed behaviour, in Python, without reading Clojure's implementation source. The deliverable is a paper rather than a language: a rung that stalls counts as a result provided the stall is written down.

Two oracles answer different questions, and keeping them apart is most of the method.

oracle what it is what it answers
sequence --filter=blob:none --no-checkout clone of clojure/clojure — commits, trees, tags, no file contents what to build next
behavioural a Clojure jar with every *.clj removed by zip -d, run as a subprocess whether an answer is correct

The second is the interesting one. Clojure boots from AOT-compiled classes, so the stripped jar still runs: (doc map) returns a docstring, (source map) returns Source not found. A zip operation enforces the clean-room boundary, which is cheaper and harder to violate than a rule about what you may read.

2. Thirty primitives, and the test that admits them

The kernel is what the Python host must expose before a core.clj analogue can define the remainder. The count is a claim, so admission is adversarial: could this be written in Clojure using only what is already in the kernel? If yes, it stays out.

+ is in and inc is out, because (def inc (fn* [n] (+ n 1))) needs nothing from the host. prn is in because printing is I/O. count and first are in because they read the host's representation of a collection.

group primitives n
arithmetic + - * / quot rem 6
equality and order = compare < > <= >= 6
collections count seq first rest cons conj get nth assoc vector list 11
printing prn println pr-str str 4
host reflection type class Exception 3

Ten special forms complete it: quote if do def let* fn* loop* recur throw try.

There is no macro system, so defn, let, cond, when, and, or and -> do not exist. All seven are macros in Clojure — though let is a trap for this framing, since its var carries :special-form true and clojure.org documents it under Special Forms, even while it expands to the let* special form.

A loop* with no recur binds locals and stands in for let. That holds only under the stated qualifier: with a recur present the two diverge, because loop* steals the recur target and the let* version compiles where the loop* version does not.

3. The library hypothesis

The claim under test is that what separates a reconstruction from ordinary list programming is a library, not the bridge to the host. core/prelude.clj defines 34 functions using only the 30 primitives and the 10 special forms, and the kernel did not change while it was written.

;; each name means what clojure.core means by it
(def map
  (fn* [f xs]
    (loop* [in (seq xs) out (list)]
      (if (= in nil)
          (reverse out)
          (recur (next* in) (cons (f (first in)) out))))))

P-99 calls 15 of the 34, directly or transitively. It never reaches the remaining 19 — a fact that turns out to set the ceiling on what mutation can detect.

4. Why the corpus is external

P-99 is Werner Hett's Ninety-Nine Prolog Problems, written for a Prolog course with no knowledge of this project. That provenance is the whole reason to use it.

An earlier rung built its corpus alongside the implementation. It reported 28 agreements out of 28, then detected 6 of 10 planted defects. One author writing both the implementation and its tests applies the same assumptions twice, and the agreement rate says nothing about either.

The grading is deliberately asymmetric. bac loads the prelude and then the solutions; the oracle loads only the solutions, on top of the real clojure.core. So the oracle grades prelude and solutions together, and a defect in our filter surfaces as a divergence in every solution that calls it. Loading the prelude into both sides would compare the solutions against themselves.

No translation step exists. The same source text runs on both, because the solutions stay inside the subset the two languages share.

5. Results

39 expressions over 35 solutions, all returning what the sealed jar returns.

problems content result
P01–P22 list ops: last, k'th, reverse, palindrome, flatten, compress, pack, run-length encode and decode, duplicate, drop, split, slice, rotate, remove, insert, range 24/24
P26, P28 combinations, sort by length 3/3
P31–P41 primality, gcd, coprimality, Euler totient two ways, prime factorisation, Goldbach 12/12
(count (combinations 3 (range 1 13)))     220
(prime-factors 315)                       (3 3 5 7)
(prime-factors-mult 315)                  ((3 2) (5 1) (7 1))
(encode-modified [:a :a :a :a :b :c :c])  ((4 :a) :b (2 :c))
(goldbach 28)                             (5 23)
(goldbach-list 9 20)                      ((3 7) (5 7) (3 11) (3 13) (5 13) (3 17))

Three exclusions, each following from the problem rather than from bac. P23–P25 select at random, so two correct implementations disagree on every run and a value oracle reports a divergence where no defect exists. P27 returns groupings whose order the problem leaves unspecified, and the oracle provides no canonical form.

The third exclusion is stated wrongly at source. The writeup gives P54 and above as needing map and set literals, which the reader does not parse. Checked against the problems: P-99 contains no associative or set literal anywhere. P54 onward are binary trees written as Prolog terms t(X,L,R) with nil, or as plain lists (X L R) in the Lisp variant, and the graph section explicitly simulates sets with sorted lists — "the lists are kept sorted, they are really sets, without duplicated elements". A reconstruction needs cons, car, cdr, nil and symbol equality, nothing more. If those sections are genuinely out of reach, the reason is nested tagged structures, not literals.

6. What the corpus found

It was not green when written. Four defects and one priced deviation.

6.1. catch did not catch host refusals

An earlier rung added a HostRefusal exception so (+ 1 :kw) would refuse cleanly rather than raise a Python TypeError, and did not touch _try, which caught only Clojure-level throw. On the oracle all three of these return :caught, because Clojure raises ClassCastException and IllegalArgumentException and both descend from Exception. bac propagated all three.

(try (+ 1 :kw)   (catch Exception e :caught))
(try (seq :a)    (catch Exception e :caught))
(try (count :a)  (catch Exception e :caught))

6.2. concat treated () as nil

The loop guard read ( in nil)=. next* normalises an exhausted sequence to nil, so the guard is right for every value the loop computes — but the loop's initial value is (reverse a), which is () for empty input, and () is not nil. An empty left argument therefore prepended (first ()), which is nil.

Sixteen spot-checks written beside the prelude passed, because every one of them concatenated two non-empty lists. (reduce concat (list) xs) starts from the empty list. P26 counted 232 combinations where the answer is 220 — reported, not reconstructed. The excess of 12 is exactly the input length, which is suggestive, but sixty-four modelled variants of a buggy combinations recurrence produce 232 in none of them. Take the figure as observed output rather than as an explained mechanism.

That is the shape of the whole argument in one defect: the tests written next to the code shared the code's blind spot, and an external corpus did not.

6.3. type returns the host's type names

(type (list 1)) is tuple in bac and clojure.lang.PersistentList on the oracle. The deviation is deliberate and documented — returning java.lang.Long for a Python object would state something false that a reader would repeat.

The corpus did not find the decision. It found the cost, which nothing recorded: source that dispatches on type cannot run on both. P07 flatten must tell an atom from a list, so the solutions use try=/=catch instead — which is what made the catch defect blocking rather than cosmetic.

6.4. The harness graded programs on their first form

read-string reads one form, and the harness passed whole programs to it. A program shaped (def f ...) (f 10) was graded on the def, with the expected value coming back as #'user/fact. Uncorrected, the harness would have marked bac incorrect for returning the right answer.

7. Planted defects

39 of 39 establishes nothing until the corpus is shown capable of failing. Fourteen defects were planted in the prelude, one at a time. The prelude is the right target: a defect in filter reaches thirty problems, one in slice reaches one.

outcome plants n
corpus detected even-is-odd, map-drops-first, filter-inverted, reduce-wrong-order, range-off-by-one, take-one-extra, drop-one-short 7
timeout detected reverse-drops-last, empty-never-true, concat-loses-empty-left 3
escaped inc-off-by-one, last-returns-first, every-vacuous, some-never-finds 4

Ten plants produced a failing run, but three of those failed because the mutated prelude did not terminate and the timeout ended it — no values were compared. So the corpus detected 7 of 14, and the honest headline is that number rather than the ten.

The four escapes are not mysterious. They modify inc, last, every? and some, and P-99 calls none of them. They live in the 19 definitions the corpus never reaches. Coverage, not sensitivity, is the binding constraint.

7.1. The plants exposed a harness defect

reverse-drops-last replaces a loop's exit test with one that is never true, so reverse ran forever. Stopping it sent SIGKILL, the finally block never ran, and the planted defect stayed in core/prelude.clj. The next command loaded the planted prelude and also ran forever.

Non-termination is a common outcome of mutating a loop*=/=recur prelude, because breaking an exit test is among the simplest available mutations. A mutation harness without a timeout has no outcome for those plants at all. The harness now applies a 120-second timeout, restores the file from atexit and signal handlers, and refuses to start when core/ holds uncommitted changes.

8. Where the limit is

The library hypothesis held. Thirty primitives sufficed to define map, filter, reduce, range, concat, take and drop in the language, and the list and arithmetic sections run on those definitions with no kernel change.

Three limits remain, and the bridge is none of them.

  • No map or set literals. {:a 1} is a reader error. Idiomatic Clojure uses maps throughout and P-99's tree and graph sections require them. The largest gap.
  • No macro system. Mostly syntactic, and measurable in one place: some calls its predicate twice on a match, because calling it once needs a local binding in the if test position and loop* cannot bind there.
  • Non-tail recursion stops at 141 frames, each bac frame costing about seven Python frames against CPython's default limit of 1000 — adjustable with setrecursionlimit, not a hard bound. loop*=/=recur has no such limit; a run of 1,000,000 iterations completed. The JVM figure of roughly 10,000 frames measured as 11,578–12,340 on a default main thread, and it moves with stack size: 8,406 on a 512 KB thread, 172,246 on an 8 MB one. A quantitative difference between hosts, and one that depends on how you ask.

9. Reading map