Org-Publish at Scale: Managing a Static Site with Emacs and TRAMP
Table of Contents
A 660-file org-mode site published to a remote VPS via Emacs TRAMP. What works, what breaks, and where the time goes.
1. The system
site/ ← 660 .org files, 388 images
├── index.org ← homepage
├── research/ ← ~550 research notes
├── events/ ← conference notes
├── current/ ← weekly summaries
├── static/ ← CSS, JS, images
└── .well-known/ ← agent discovery docs
│
▼
publish.el ← org-publish configuration
│
▼ (emacs --batch)
TRAMP / scp ──────────► VPS (remote host)
Emacs runs in batch mode. org-publish exports each .org file to HTML,
then copies it over SSH via TRAMP to the remote host. Static assets
(images, CSS, JS) are copied as attachments.
2. Publish components
publish.el had 6 components when this note was scaffolded (commit d81f08e2, 2026-05-25) -- main, markdown, static, well-known, research-images, sitemap -- so "5" undercounted even then (markdown/GFM export was missing from the table). Live publish.el on this machine today defines 9: those six plus tools-images, events-images, current-images (added by commits through 2026-09-11). Table below updated to the current 9.
The site has 9 publish components, each with its own base directory, file extension filter, and publishing function:
| Component | Files | Function | What it does |
|---|---|---|---|
main |
.org |
org-html-publish-to-html |
Export org to HTML |
markdown |
.org |
org-gfm-publish-to-gfm |
Export org to GFM Markdown |
static |
css,js,json,png,... |
org-publish-attachment |
Copy static assets |
well-known |
json,txt,wsdl,xsd |
org-publish-attachment |
Copy agent discovery |
research-images |
png,jpg,svg,pdf,pptx |
org-publish-attachment |
Copy research images |
tools-images |
png,jpg,svg,pdf,pptx |
org-publish-attachment |
Copy tool images |
events-images |
png,jpg,gif,svg |
org-publish-attachment |
Copy event images |
current-images |
png,jpg,gif,svg |
org-publish-attachment |
Copy daily-brief images |
sitemap |
.org |
custom | Generate sitemap.xml |
3. Timing
FreeBSD 15.0 / Emacs 30.1 check out for this note's write date (2026-05-25) -- git log shows the FreeBSD 15.1 upgrade landed 2026-06-21 (commit d450f8c7), a month later. Live `uname -a` / `emacs --version` on this machine today report FreeBSD 15.1-RELEASE and GNU Emacs 30.2 -- expected drift since writing, not an error in the note.
Measured on FreeBSD 15.0, Emacs 30.1, publishing to a remote VPS over SSH (TRAMP) with multiplexed connections:
| Scenario | Files touched | Wall clock |
|---|---|---|
| Single file edit | 1-3 | ~30s |
| Multiple content edits | 10-20 | 2-5 min |
| Config property change | ~all HTML re-exported | ~35 min |
| Full cache-cleared rebuild | ~2085 total | 25-35 min |
4. Where the time goes
4.1. HTML export (~30% of wall clock)
Emacs org-mode exports each .org file: parse the org AST, resolve
#+INCLUDE directives, evaluate #+begin_src blocks (if :exports
results), generate HTML, write to the TRAMP target. Batch mode avoids
interactive prompts.
htmlize is required for syntax highlighting in code blocks. Without
it, src blocks export as plain <pre> tags.
4.2. TRAMP file transfer (~60% of wall clock)
TRAMP copies each file individually over SSH. For 360 static assets,
this means 360 separate scp operations over the multiplexed
connection. Each operation has overhead: open the file descriptor on the
remote, write, close, confirm.
rsync would batch unchanged files in a single pass, but org-publish's
org-publish-attachment function uses TRAMP's copy-file, not rsync.
4.2.1. Optimization: custom attachment publisher
Replace org-publish-attachment with a function that collects files into
a manifest, then runs a single rsync at the end:
(defun wal-sh/rsync-publish (plist filename pub-dir)
"Collect files for batch rsync instead of per-file TRAMP copy."
(let ((src (expand-file-name filename (plist-get plist :base-directory)))
(dst (expand-file-name filename pub-dir)))
;; Accumulate in a buffer; rsync at the end
(push (cons src dst) wal-sh/rsync-manifest)))
(defun wal-sh/rsync-flush ()
"Flush accumulated files via rsync."
(when wal-sh/rsync-manifest
(let ((tmpfile (make-temp-file "rsync-manifest")))
(with-temp-file tmpfile
(dolist (pair wal-sh/rsync-manifest)
(insert (car pair) "\n")))
(shell-command
(format "rsync -avz --files-from=%s / %s"
tmpfile wal-sh/remote-base))
(setq wal-sh/rsync-manifest nil))))
Estimated improvement: 360 individual scp calls to 1 rsync invocation. For unchanged files, rsync's checksum skip eliminates the transfer entirely.
4.3. Cache check (~10% of wall clock)
~/.org-timestamps/wal.sh-main.cache is not a Lisp alist -- it is generated Elisp that rebuilds a hash table via `(puthash ...)` forms into `org-publish-cache`. Loading it in a batch Emacs on this machine took 106ms for today's 1093 tracked files / 3281 hash-table entries, matching the body's ~100ms order of magnitude for the 660-file site at write time. Body corrected to say hash table instead of alist.
Org-publish maintains a cache at ~/.org-timestamps/<project>.cache
that records file modification times. On each publish, it compares the
cached mtime against the current mtime and skips unchanged files.
The cache is an Elisp hash table serialized to disk as generated code. For 660 files, reading and comparing is fast (~100ms). An incremental publish with 1 changed file skips 659 exports.
5. Cache management
| Action | When to do it |
|---|---|
| Preserve cache | Content-only changes (editing .org files) |
| Clear cache | After changing publish.el (preamble, postamble, publish properties) |
| Force single file | gmake publish-file FILE=site/research/<slug>/index.org |
Clearing the cache when not needed causes a full rebuild (35 min).
Changing a publish property (e.g. ~:with-sub-superscript) effectively
invalidates the cache even without deleting it, because org-publish
detects the config change and re-exports all HTML files.
6. The #+INCLUDE pattern
(1) includes/ under a research dir were published standalone; the ^includes/ exclude matched only site/includes/ (fixed in #207). (2) Body also claimed image paths from an include resolve relative to index.org's directory -- backwards: confirmed via site/research/2026-repl-driven-flight-tracking/includes/case-002-storm.org, whose [[file:storm-message-volume.png]] resolves as a sibling inside includes/ itself (file lives at includes/storm-message-volume.png, not the parent dir); body corrected.
Large research notes split into modular files under an includes/
subdirectory:
site/research/2026-topic/
├── index.org ← preamble + #+INCLUDE directives
└── includes/
├── data-sources.org ← * Data Sources
├── architecture.org ← * Architecture
└── resources.org ← * Resources
Each include file starts with its * heading at the original level.
Image references resolve relative to the includes/ directory, not to
index.org's directory: the image lives as a sibling of the include
file, so [[file:diagram.png]] in includes/architecture.org points
to includes/diagram.png.
Benefits:
- Agents can edit a section without loading the full file
- Merge conflicts are scoped to the section, not the whole document
- The composed document exports as one page: numbering,
CUSTOM_IDanchors and[[*Heading]]cross-references between includes all resolve inindex.html
Earlier text here said org-publish treats the composed document as one
file with no extra cache entries. That was wrong. org-publish walks the
tree and matches each path against :exclude. The pattern was
^includes/, which matches only site/includes/, so every
research/<topic>/includes/*.org was also published on its own, as
both .html and .md. Those pages have an empty <title>, and any
[[*Heading]] that points into a sibling include becomes a
BROKEN LINK marker. On 2026-10-08,
/research/2026-code-graph-mcp-survey/includes/gitnexus-clojure
returned 200.
The sitemap (sitemap_gen.cljc, (^|/)includes/) and the search
indexer (pocket_es/indexer.clj skips the includes directory) already
excluded these files, so they were reachable only by direct URL or by
links. The fix makes the publish exclude use the sitemap's rule:
\\(^\\|/\\)includes/. Images under includes/ are unaffected, since
they ship through the separate wal.sh-research-images attachment
project.
Experiment: three layouts published to HTML, GFM and org with the
site's own :exclude and with-broken-links mark:
| layout | page output | side effect, old regex | side effect, new regex |
|---|---|---|---|
index.org + includes/NN-x.org |
inlined, numbered, linked | one page per fragment | none |
index.org + flat NN-x.org siblings |
inlined | one page per fragment | one page per fragment |
source.org assembles, index.org includes it |
inlined | fragments plus source page |
source page |
A separate source document adds nothing: index.org with
#+INCLUDE: already is the source, and org-to-org publish
(org-org-publish-to-org) gives the assembled file if one is wanted.
Pages already on the server stay until removed on nexus; org-publish
only uploads.
7. FreeBSD-specific
| Issue | Workaround |
|---|---|
make is BSD make |
Use gmake (GNU Make) |
| Emacs not in PATH | poetry run emacs |
md5sum unavailable |
Use md5 |
| ripgrep may be missing | grep fallback |
.org-timestamps permissions |
Emacs batch must run as the same user |
8. Automation: the /publish skill
A Claude Code skill encodes the publish workflow so the agent doesn't skip verification steps:
git pull --rebase(start from HEAD)- Detect what changed (content vs config)
gmake publish(incremental or full)- Verify: SSH to VPS, check the deployed page
- Report timing and file count
The skill exists because gmake publish alone is necessary but not
sufficient – verification was being skipped, leading to broken rendering
on production.
9. Optimizations not yet implemented
| Optimization | Expected improvement | Complexity |
|---|---|---|
| rsync batch for attachments | 360 scp → 1 rsync | Medium (custom publisher fn) |
| Parallel HTML export | N cores × faster | High (Emacs is single-threaded) |
| Local export + rsync all | Separate export from upload | Medium |
| Selective component publish | Only publish changed component | Low (already partially works) |
| TRAMP async | Non-blocking file copy | High (experimental in Emacs 30) |
The highest-value change is rsync batching for attachments. The export step is CPU-bound in Emacs and cannot be parallelized without significant architectural changes (multiple Emacs processes, each handling a subset of files).
10. Related
- REPL-Driven Compliance – multi-language development on the same site
- REPL-Driven Flight Tracking – the literate programming methodology