The Shell Script From Hell
IT’S TAKEN ME 60 HOURS TO SAVE 2 MINUTES OF TYPING#
Git hooks are scripts of some kind that run when you do something in Git. Real developers (not me) use them to kick off a bunch of crap like QA tests, validation tools, sending messages to people, pretty much anything you can imagine. They do all kinds of things, but I am going to do the thing that I always do and barely scratch the surface of its potential. AND I am using it to run Bash scripts. If you are a proper dev, or DevOps type, this is your opportunity to look away.
Building modern websites reminds me of my days in Seattle working for a small software startup. There, the QA team did tons of automation to run tons of tests on the product codebase. Mostly I wrote specifications, and helped the ex-Microsoft people (the whole department) do testing on Linux and Solaris. Back then, Real Devs™ wrote code that had to be compiled, like Java or C. Scripting languages and web pages were for weaklings. Nowadays, I guess proper devs just do REST API calls all day.
Today, I am using Git like it was CVS, to check in what are essentially text files and kick off Bash shell scripts to produce a static website. It feels like using precision aircraft tools to open beer bottles.
The precision tool I am focused on today is the Git post-receive hook. Post-receive happens on the remote repo when you push an update. I have combed through Git documentation that was written Dev-ese and I have managed to kludge together a deployment script from other people’s code. I cannot begin to express how far out of my depth I am.
The Staging Server#
I am using a Linux container on my Proxmox server to push my MD files to. It runs Git, to serve the remote repo for my workstations, Hugo to build the static site, and Caddy to serve the test site. I don’t use any special features in Caddy, I just want to see what the site will look like before it gets deployed to my VPS where it’s visible to the Internet. I think of it like clicking “Print Preview” before clicking “Print”.
The Caddy server config is incredibly simple, since the staging server is only accessible from my home lab, I don’t need to worry about HTTPS, or domain names, or anything really:
Caddyfile
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
:80 {
root * /var/www
file_server
}
The /var/www directory is chowned to caddy:caddy and my username is added to the caddy group (usermod -a -G caddy me). This way I can write to www as a regular user. I think this is fine for an internal testing site. This is not the case on my production VPS. When I push files to the VPS they land in the remote user’s home directory (/home/me/www), and I have a cron job that sync’s the files as root. A better solution would be to figure out how to lock down sudo.
In the future, when I have proper DNS for my overlay network, and a bunch of other things, I will try to get clever with the Caddy config, but first I have to get this damn thing working.
The file Structure#
In my home directory on the staging server, I have some directories to hold the different files during the staging process. I wish I could just pipe everything right into Hugo, or maybe hard linking the preview directory to /var/www/example.com, but that is beyond my skill to heal. So for now, I use subdirectories. If this sounds overly complicated for a deploy script, that’s because it is. The file structure keeps reusing “example.com” because it can be used for multiple websites. Each website gets its own repo (foo.com.git, bar.com.git) and this same script can be run when new MD files are pushed. The tooling doesn’t care about the little idiosyncrasies of Hugo templates. For that, I have a dedicated build directory for each site to host the Hugo build environment.
/home/me <-- home directory on the staging server, ssh://me@staging/home/me
-- auto <-- automation directory. Put the shell scripts here.
stage.sh <-- uses git, Hugo, and rsync to deploy to the staging web server
deploy.sh <-- uses Hugo and rsyn to deploy to the production VPS
-- build <-- the hugo site folder, required by Hugo
-- example.com <-- created with "hugo new site example.com"
-- archetypes
-- assets
-- content <-- this is where the markdown files go
-- data
hugo.toml <-- the config file for my site, use it to set up the template
-- i18n
-- layouts
-- static <-- anything that isn't markdown is supposed to go here, IDK if it always works.
-- themes <-- git clone the hugo template to here, no submodules.
-- git
-- example.com.git <-- this is the bare repo for git remote created by 'git init --bare'
HEAD
-- branches
config
description
-- hooks
post-receive <-- create this file to call stage.sh
-- info
-- objects
-- refs
-- markdown <-- the git repo gets cloned to here
-- preview <-- the "print preview" site gets put here by Hugo
-- deploy <-- the "ready for primetime" static site goes here
There are separate “git” and “markdown” directories because the git repo is bare. I guess that means the markdown files are kept in the git database somewhere (no clue where that is) and are not accessible to the file system. If I want to see the MD files on the local file system, I need to clone the Git repo. Git tanks when you clone a repo into an existing folder, so the “markdown/example.com folder” gets deleted when stage.sh runs.
There are separate “markdown” and “build” directories because Hugo requires the MD files to sit in the “content” subdirectory, and Git won’t want the “build/example.com/content” to be pre-existing. Right now the script just uses rsync to copy the MD files from “markdown/example.com” to “build/example.com/content”. There is probably something I can do with hard links to avoid copying the files, or I could move the files instead of syncing, or I could be brave and delete “build/example.com/content” and clone the repo to there. But right now rsync and redundancy is my safety net.
There are separate “build/example.com/public” and “preview/example.com” directories so that I can rsync the generated static site to /var/www on the staging server to verify that the site looks ok, and then rsync a different version of the site to the VPS. Again, there is a more elegant way, but during the development process, I am being redundant. I can watch these scripts fail only so many times before I lose my mind.
I will post about the actual staging script in my next post.