Annotation Systems for Evolving Documents

The simplest anchor: character position 4,273. Dies on the first edit upstream of the annotation. Not used in practice except in binary formats.

XPath (//section[3]/p[1]), CSS selectors, heading paths (Introduction > Background > Prior Work). Survive whitespace changes. Break on any structural reorganization. The W3C Web Annotation Data Model (Sanderson, Ciccarese, and Young 2017) uses a layered selector chain: a RangeSelector wraps a XPathSelector which wraps a TextPositionSelector, so at least one layer may still resolve when others fail.

Store the exact text (exact) plus a prefix and suffix for context. On re-anchoring, search for the exact string; if missing, fuzzy-match against candidates weighted by context overlap. The Hypothesis annotation client uses this approach. It survives minor rewrites. It does not survive paragraph deletion or heavy paraphrase.

Hash the paragraph's canonical text (strip trailing whitespace, normalize Unicode). Store sha256:a3f7.... An annotation indexed this way survives file moves and heading renames, as long as the paragraph text itself is stable. Dies on any edit to the target paragraph. Useful for locking a "needs citation" flag to a specific claim.

Reference a stable heading ID (:CUSTOM_ID: background-prior-work) rather than the heading text. In org-mode, the :CUSTOM_ID: property is set by the author and survives heading renames. It does not survive heading deletion, but deletion is detectable: a script can scan for orphaned annotation references.

The trade-off table:

No anchor survives deletion. The best you can do is detect it.

#+begin_comment
REVIEW 2026-06-07 jwalsh: Claims about XPath brittleness need a citation.
Hypothesis client source code would work. Or the W3C spec itself.
#+end_comment

Invisible on HTML/PDF export. Inline with the text they annotate. Version controlled. The weakness: they live inside the section they comment on, so heading renames do not break them, but section deletion takes them with it. Good for author-to-author notes that belong to a specific passage.

* Background
:PROPERTIES:
:CUSTOM_ID: background
:REVIEW:   2026-06-07 jwalsh: Needs citation for Kalir & Garcia claim
:STATUS:   needs-citation
:END:

Every heading can carry arbitrary key-value metadata. :REVIEW: and :STATUS: are author-defined. These survive heading renames — they sit below the heading text. They survive file reorganization. They do not survive heading deletion.

A useful pattern: combine :CUSTOM_ID: with :STATUS: needs-citation so that a search across the repo can find all outstanding annotation tasks:

grep -rn ':STATUS:.*needs-citation' site/research/

VERIFIED_AT: 2026-06-07T22:50Z
VERDICT: corrected
FINDING: no package called org-annotate exists on MELPA; correct name is org-annotate-file (EmacsWiki) or annotate.el (bastibe/annotate.el, MELPA)

org-annotate-file (EmacsWiki, not MELPA) maintains a separate annotations file. Each entry records the file path and heading text of its target. annotate.el (bastibe/annotate.el, MELPA) is a more actively maintained alternative that stores annotations keyed to file and line number.

Decoupled: the annotation file lives in a private overlay repo, separate from the document source. But both key on heading text or line position, not :CUSTOM_ID:. A heading rename breaks the link silently.

VERIFIED_AT: 2026-06-07T22:50Z
VERDICT: correct
FINDING: exists on MELPA (org-noter/org-noter fork), description accurate

org-noter is designed for annotating PDFs and EPUBs via a parallel note buffer synchronized by position. It works for org-mode documents too, with the document in one window and the notes in another. Position sync is by heading for org files.

Works for reading sessions: open document, open notes buffer, write as you read. Does not persist review flags across editing sessions.

The anchor problem has three sub-problems[fn:: Position, survival, and
discoverability. See also the W3C selector chain design.]: position encoding,
survival under edits, and discoverability.

Inline footnotes ([fn:: ...]) are rendered as footnotes on export and are searchable in the source. They sit immediately adjacent to the text they annotate. Not suitable for structured metadata, but useful for brief clarifications that belong in the published document.

Assign :CUSTOM_ID: properties to every heading at creation time. Use kebab-case slugs derived from the heading text, but do not update them when the heading text changes. The ID is the stable handle; the heading text is the display label.

* The Anchor Problem
:PROPERTIES:
:CUSTOM_ID: anchor-problem
:END:

An annotation referencing anchor-problem survives renaming "The Anchor Problem" to "Anchoring Strategies" because the :CUSTOM_ID: does not change.

Keep annotations out of the document source when they are transient. A companion annotations.org file in the same directory can hold review notes keyed by :CUSTOM_ID::

* Annotation Index
** anchor-problem
:PROPERTIES:
:TARGET_ID: anchor-problem
:DATE:      2026-06-07
:AUTHOR:    jwalsh
:STATUS:    open
:END:

The table on anchor survival rates should note that fuzzy matching degrades
gracefully rather than failing hard. Add a note about Levenshtein distance
thresholds in the Hypothesis implementation.

** scholarly-annotation-tradition
:PROPERTIES:
:TARGET_ID: scholarly-annotation-tradition
:DATE:      2026-06-07
:AUTHOR:    jwalsh
:STATUS:    closed
:CLOSED:    2026-06-08
:END:

Cite Kalir & Garcia 2021. Done.

When a heading is deleted, annotations referencing its :CUSTOM_ID: become orphans. A short script detects them:

# Extract all CUSTOM_IDs from a document
IDS=$(grep ':CUSTOM_ID:' site/research/annotation-systems/index.org \
      | awk '{print $2}')

# Extract all TARGET_IDs from the annotations file
TARGETS=$(grep ':TARGET_ID:' site/research/annotation-systems/annotations.org \
          | awk '{print $2}')

# Report targets with no matching ID
for t in $TARGETS; do
  if ! echo "$IDS" | grep -qx "$t"; then
    echo "ORPHAN: $t"
  fi
done

Run this as a pre-publish hook or as part of gmake lint.

Git blame surfaces the commit that last touched each line. Combined with a :CUSTOM_ID:-keyed annotation, you can ask: was this annotation created before or after the commit that rewrote the target paragraph?

git log --follow --oneline -- site/research/annotation-systems/index.org | head -5

This is not automatic in any current tool. It requires correlating annotation :DATE: fields with commit timestamps. The tooling gap is real: no org-mode annotation system currently integrates with git blame output. That is the research gap worth closing.

The zero-infrastructure option. An agent adds :VERIFIED_AT:, :VERIFIED_BY:, and :VERDICT: properties directly to the heading it reviewed:

* Janus: A Reversible Programming Language
:PROPERTIES:
:CUSTOM_ID: janus-lang
:VERIFIED_AT: 2026-06-07T22:00Z
:VERIFIED_BY: skeptic-agent
:VERDICT: corrected
:FINDING: date was 1982, corrected to 1986
:END:

Properties are invisible on HTML export, greppable, and tied to the heading. They survive renames because :CUSTOM_ID: is the stable anchor. Multi-line values use the + append syntax: :FINDING+: second observation.

The concern is clutter — a heading with eight review properties is visually heavy in the raw source. Emacs folds drawers by default, so it is invisible in the editor. In git diffs it is visible and can feel noisy.

The project already uses bd for issue tracking. A verification bead references a heading by :CUSTOM_ID: and carries the verdict:

bd create --title="verify: Janus date" \
  --description="Heading janus-lang. Original: 1982. Correct: 1986." \
  --type=task --priority=1
bd close <id> --reason="corrected inline, commit abc1234"

Clean-check beads record what was checked and found correct — the thing git blame cannot tell you:

bd create --title="verify: Toffoli gate semantics" \
  --description="Heading toffoli-gates. Verified correct." \
  --type=task --priority=3
bd close <id> --reason="verified correct, no changes"

The hybrid approach: property drawers for per-heading verdicts, beads for session-level scope ("7 agents, 4 corrections, 12 claims in scope").

The Hypothesis annotation platform (hypothesis/h, Python, BSD-2) implements the W3C Web Annotation spec with a REST API. An LLM agent can POST annotations programmatically:

POST https://hypothes.is/api/annotations
Authorization: Bearer <token>

{
  "uri": "https://wal.sh/research/2026-reversible-pipeline-transforms/",
  "text": "Verified: reverse is anti-homomorphism, not homomorphism.",
  "tags": ["skeptic-review", "verdict:corrected", "agent:claude-opus-4-6"],
  "target": [{
    "selector": [{
      "type": "TextQuoteSelector",
      "exact": "reverse is a monoid homomorphism",
      "prefix": "Examples: upper, lower, rot13, ",
      "suffix": ". Non-homomorphism"
    }]
  }]
}

This stores the verification trail outside the document, queryable via GET /api/search?uri=<url>&tag=skeptic-review. No self-hosting required — use the public service with a private group.

The gap: the conversation transcript itself. What the agent was told, what searches it ran, what it rejected — none of this is captured by any annotation system. PaperTrail (Martin-Boyle et al. 2026) decomposes claims and maps verdicts at claim granularity, but found that showing provenance reduced trust because it revealed what was NOT checked.

• ProvBook (Samuel and König-Ries 2018): Jupyter extension embedding W3C PROV-O provenance in cell metadata. Direct analogy to org property drawers.
• GenProve: LLMs generating sentence-level provenance triples during text generation, not post-hoc.
• PaperTrail (Martin-Boyle et al. 2026): claim-level verification with three verdicts (supported, unsupported, omitted). Published CHI 2026.
• ai-blame: extends git blame with AI session provenance for code files.
• Racket contracts (Findler and Felleisen 2002): Findler and Felleisen's blame-tracking contracts enforce pre/post conditions at module boundaries. A contract on a codec's decode function checks the round-trip property on every call — the contract IS the verification annotation, enforced at runtime rather than recorded after the fact. The Racket implementation of the transform tool uses involution/c and bijection/c contract combinators to catch the same bugs that skeptic agents found post-hoc.