wal.sh
home > research > 2026-org-publish-experience · May 25, 2026
org-modeemacstrampstatic-sitepublishingperformance

Org-Publish at Scale: Managing a Static Site with Emacs and TRAMP

Table of Contents

A 660-file org-mode site published to a remote VPS via Emacs TRAMP. What works, what breaks, and where the time goes.

1. The system

site/                          ← 660 .org files, 388 images
  ├── index.org                ← homepage
  ├── research/                ← ~550 research notes
  ├── events/                  ← conference notes
  ├── current/                 ← weekly summaries
  ├── static/                  ← CSS, JS, images
  └── .well-known/             ← agent discovery docs
          │
          ▼
    project-config.el          ← org-publish configuration
          │
          ▼  (emacs --batch)
    TRAMP / scp ──────────►   VPS (remote host)

Emacs runs in batch mode. org-publish exports each .org file to HTML, then copies it over SSH via TRAMP to the remote host. Static assets (images, CSS, JS) are copied as attachments.

2. Publish components

The site has 5 publish components, each with its own base directory, file extension filter, and publishing function:

Component Files Function What it does
main .org org-html-publish-to-html Export org to HTML
static css,js,json,png,... org-publish-attachment Copy static assets
well-known json,txt,wsdl org-publish-attachment Copy agent discovery
research-images png,jpg,svg,pdf org-publish-attachment Copy research images
sitemap .org custom Generate sitemap.xml

3. Timing

Measured on FreeBSD 15.0, Emacs 30.1, publishing to a DreamHost VPS over SSH with multiplexed connections:

Scenario Files touched Wall clock
Single file edit 1-3 ~30s
Multiple content edits 10-20 2-5 min
Config property change ~all HTML re-exported ~35 min
Full cache-cleared rebuild ~2085 total 25-35 min

4. Where the time goes

4.1. HTML export (~30% of wall clock)

Emacs org-mode exports each .org file: parse the org AST, resolve #+INCLUDE directives, evaluate #+begin_src blocks (if :exports results), generate HTML, write to the TRAMP target. Batch mode avoids interactive prompts.

htmlize is required for syntax highlighting in code blocks. Without it, src blocks export as plain <pre> tags.

4.2. TRAMP file transfer (~60% of wall clock)

TRAMP copies each file individually over SSH. For 360 static assets, this means 360 separate scp operations over the multiplexed connection. Each operation has overhead: open the file descriptor on the remote, write, close, confirm.

This is the dominant bottleneck. rsync would batch unchanged files in a single pass, but org-publish's org-publish-attachment function uses TRAMP's copy-file, not rsync.

4.2.1. Optimization: custom attachment publisher

Replace org-publish-attachment with a function that collects files into a manifest, then runs a single rsync at the end: 

(defun wal-sh/rsync-publish (plist filename pub-dir)
  "Collect files for batch rsync instead of per-file TRAMP copy."
  (let ((src (expand-file-name filename (plist-get plist :base-directory)))
        (dst (expand-file-name filename pub-dir)))
    ;; Accumulate in a buffer; rsync at the end
    (push (cons src dst) wal-sh/rsync-manifest)))

(defun wal-sh/rsync-flush ()
  "Flush accumulated files via rsync."
  (when wal-sh/rsync-manifest
    (let ((tmpfile (make-temp-file "rsync-manifest")))
      (with-temp-file tmpfile
        (dolist (pair wal-sh/rsync-manifest)
          (insert (car pair) "\n")))
      (shell-command
        (format "rsync -avz --files-from=%s / %s"
                tmpfile wal-sh/remote-base))
      (setq wal-sh/rsync-manifest nil))))

Estimated improvement: 360 individual scp calls → 1 rsync invocation. For unchanged files, rsync's checksum skip eliminates the transfer entirely.

4.3. Cache check (~10% of wall clock)

Org-publish maintains a cache at ~/.org-timestamps/<project>.cache that records file modification times. On each publish, it compares the cached mtime against the current mtime and skips unchanged files.

The cache is a Lisp alist serialized to disk. For 660 files, reading and comparing is fast (~100ms). The savings are large: an incremental publish with 1 changed file skips 659 exports.

5. Cache management

Action When to do it
Preserve cache Content-only changes (editing .org files)
Clear cache After changing project-config.el (preamble, postamble, publish properties)
Force single file gmake publish-file FILE=site/research/<slug>/index.org

Clearing the cache when not needed causes a full rebuild (35 min). Changing a publish property (e.g. ~:with-sub-superscript) effectively invalidates the cache even without deleting it, because org-publish detects the config change and re-exports all HTML files.

6. The #+INCLUDE pattern

Large research notes split into modular files under an includes/ subdirectory: 

site/research/2026-topic/
  ├── index.org                ← preamble + #+INCLUDE directives
  └── includes/
      ├── data-sources.org     ← * Data Sources
      ├── architecture.org     ← * Architecture
      └── resources.org        ← * Resources

Each include file starts with its * heading at the original level. Image references are relative to the index.org directory (includes are resolved from the parent), so [[file:diagram.png]] works unchanged.

Benefits:

  • Agents can edit a section without loading the full file
  • Merge conflicts are scoped to the section, not the whole document
  • org-publish treats the composed document as one file (no extra cache entries)

7. FreeBSD-specific

Issue Workaround
make is BSD make Use gmake (GNU Make)
Emacs not in PATH poetry run emacs
md5sum unavailable Use md5
ripgrep may be missing grep fallback
.org-timestamps permissions Emacs batch must run as the same user

8. Automation: the /publish skill

A Claude Code skill encodes the publish workflow so the agent doesn't skip verification steps:

  1. git pull --rebase (start from HEAD)
  2. Detect what changed (content vs config)
  3. gmake publish (incremental or full)
  4. Verify: SSH to VPS, check the deployed page
  5. Report timing and file count

The skill exists because gmake publish alone is necessary but not sufficient — verification was being skipped, leading to broken rendering on production.

9. Optimizations not yet implemented

Optimization Expected improvement Complexity
rsync batch for attachments 360 scp → 1 rsync Medium (custom publisher fn)
Parallel HTML export N cores × faster High (Emacs is single-threaded)
Local export + rsync all Separate export from upload Medium
Selective component publish Only publish changed component Low (already partially works)
TRAMP async Non-blocking file copy High (experimental in Emacs 30)

The highest-value change is rsync batching for attachments. The export step is CPU-bound in Emacs and cannot be parallelized without significant architectural changes (multiple Emacs processes, each handling a subset of files).

10. Related