CI/CD and Local Development Setup Guide for wal.sh Website

Introduction
Prerequisites
Local Development Setup
Makefile Targets
Static File Extensions
Docker Setup
- Build and Run Docker Image
Deployment Key Setup
GitHub Actions Setup
- Create GitHub Actions Workflow
- Add GitHub Secrets
Environment Variables
Replit
CSS Customization
Verifying the Setup
Troubleshooting
Maintenance
Conclusion

Introduction

This document provides a comprehensive guide for setting up both the local development environment and the CI/CD pipeline for the wal.sh website project.

Prerequisites

Git
Docker
Emacs (version 27.2 or later)
Python 3
Access to the wal.sh server

Local Development Setup

Clone the Repository

git clone https://github.com/your-username/wal-sh-website.git
cd wal-sh-website

Set Up Python Environment

make env

Install Emacs Packages

make ensure-packages

Test Local Publishing

make publish

Makefile Targets

The project includes a Makefile with various targets for different tasks:

make publish: Publish the website
make publish-force: Force publish the website
make publish-local: Publish the website locally
make publish-local-force: Force publish the website locally (excludes requirements.txt)
make clean: Clean the output directory
make deploy: Deploy to the server (needs configuration)
make check-links: Check links in the published files (needs implementation)
make test: Run tests (needs implementation)
make generate-deploy-key: Generate a deployment key
make add-deploy-key: Add deployment key to GitHub
make add-deploy-key-server: Add deployment key to the server
make deploy-key: Generate and add deploy key to both GitHub and server
make docker: Build and run Docker image
make server: Start a local server for preview
make sitemap: Generate sitemap

Static File Extensions

The Makefile uses a STATIC_EXTENSIONS variable to determine which file types should be copied during the publishing process. The current extensions are:

js png jpeg jpg gif css xml json txt tmpl mmd dot

To modify this list, update the STATIC_EXTENSIONS variable in the Makefile.

Docker Setup

Build and Run Docker Image

make docker

This command builds the Docker image and runs it, mounting your local SSH keys.

Deployment Key Setup

Generate Deployment Key

make generate-deploy-key

Add Deployment Key to GitHub

Go to your GitHub repository
Navigate to Settings > Deploy keys
Click "Add deploy key"
Paste the contents of ~/.ssh/wal_sh_deploy_key.pub
Give it a title (e.g., "WAL.SH Deploy Key")
Check "Allow write access" if needed
Click "Add key"

Add Deployment Key to Server

make add-deploy-key-server

GitHub Actions Setup

Create GitHub Actions Workflow

Create a file named .github/workflows/publish.yml in your repository
Copy the following content into the file:

name: Publish wal.sh Website

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Set up Emacs
      uses: purcell/setup-emacs@master
      with:
        version: 28.2

    - name: Set up SSH key
      uses: webfactory/ssh-agent@v0.7.0
      with:
        ssh-private-key: ${{ secrets.WAL_SH_DEPLOY_KEY }}

    - name: Add server to known hosts
      run: ssh-keyscan ${{ secrets.SERVER_HOSTNAME }} >> ~/.ssh/known_hosts

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y make python3 python3-pip python3-venv

    - name: Publish website
      env:
        WAL_SH_BASE: ${{ github.workspace }}
        WAL_SH_PUBLISHING: ${{ secrets.WAL_SH_PUBLISHING }}
      run: |
        make publish
        make publish-local

Add GitHub Secrets

Go to your GitHub repository
Navigate to Settings > Secrets > Actions
Add the following secrets:
- Name: WAL_SH_DEPLOY_KEY Value: [Paste the contents of ~/.ssh/wal_sh_deploy_key]
- Name: SERVER_HOSTNAME Value: [Your server's hostname]
- Name: WAL_SH_PUBLISHING Value: ssh:user@example.com:~/public_html
Click "Add secret" for each

Environment Variables

Create a .env file in your project root with the following content:

# Base directory for wal.sh project
export WAL_SH_BASE=/path/to/your/project
# Publishing directory for wal.sh project
export WAL_SH_PUBLISHING=/ssh:user@example.com:~/public_html/

Replace the paths and SSH details with your actual project setup.

Replit

[ -s ~/.ssh/replit.pub ] || ssh-keygen -t ed25519 -f ~/.ssh/replit -q -N "" && cat ~/.ssh/replit.pub

CSS Customization

The project includes a custom CSS file (static/default.css) for styling the website. Key features include:

Responsive design
Custom header and navigation styles
Breadcrumb styling
Page header customization

Verifying the Setup

Local Verification

Make changes to your Org files
Run make publish-local to test local publishing
Check the output in the LOCAL_OUTPUT directory
Use make server to start a local server and preview the site

Docker Verification

Run make docker
Verify that the Docker container can publish the website

GitHub Actions Verification

Push changes to the main branch
Go to the "Actions" tab in your GitHub repository
Verify that the workflow runs successfully and publishes the website

Troubleshooting

If Emacs packages fail to install, try removing the /.emacs.d/straight directory and running ~make ensure-packages again
If the GitHub Action fails due to SSH issues, double-check that the deploy key is correctly added to both GitHub and the server
For any other issues, refer to the error messages in the GitHub Actions logs or local output

Maintenance

Regularly update the Emacs packages and Docker image
Rotate the deploy key periodically for security
Keep this document updated with any changes to the CI/CD process
Update the CSS file as needed to maintain the desired look and feel of the website
Periodically review and update the STATIC_EXTENSIONS list in the Makefile to ensure all necessary file types are included

Conclusion

This setup provides a comprehensive environment for both local development and CI/CD for the wal.sh website. It allows for efficient development, testing, and automated publishing, ensuring consistency across different environments.