In my opinion, the difference between a novice and an expert is that an expert has broken it before.

60% of the Time, It Works Every Time#

My self-hosting and homelab experience is slowly shifting from installing and configuring servers to writing my own configs and scripts to build DIY “infrastructure”. I use the term lightly because right now it’s [not well] managed chaos. It works most of the time. The workflow is: hack, crash, patch, repeat.

I have a term for something that works, but not the right way: clunk. Like a car that drives but it makes a clunking sound; it’ll get you home, but probably not in style. I used to call these things “hacks” in the pejorative sense, but a hack is generally a singular fix for a singular problem. A clunk is a suite of hacks that solve a bigger problem, usually not having the right tool, enough time, or knowledge. A clunk is what happens when a hack becomes part of the infrastructure.

Imagination is more important than knowledge#

“Hack” is a term borrowed from programming, but “clunk” is more inter-disciplinary. It’s using knowledge in one field to bypass your limitations in another. Clunking is using knowledge of Ethernet and radio propagation to extend a WiFi network with antennas and wiring, rather than buying an expensive WiFi mesh system, or taking on the burden of building a WiFi mesh.

A while back, I set up a DNS server on my Synology NAS to resolve local hostnames, but I set it as a secondary server so that queries for local hostnames failed on the primary DNS server and were then picked up and resolved by the secondary. That’s a hack. I replaced that with a script to periodically update the /etc/hosts file on all my lab machines with SCP. That’s a clunk. The proper solution is knowing how to set up a proper authoritative DNS, but ain’t nobody got time for that.

When I worked at a little startup software company, I participated in a bunch of distributed system architecture meetings. Despite not being a dev, I was able to help the department find a couple of elegant solutions by explaining how Unix did something similar. In a room full of ex-Microsoft devs, the Unix philosophy of “do one thing, but do it well” was a novel approach.

Clunking isn’t just recycling hardware, it’s recycling knowledge.

Worse is Better#

Long ago in a galaxy far away, there were two schools of thought when it came to software design. In university AI labs, you had “The Right Thing” and at Bell Laboratories, you had “Worse is Better”. The university way was endless tooling on a purpose-built LISP Machine, and the Bell way was Unix. The moral of the story is that Unix won.

Over the years, I have come to respect the beautiful savagery of clunks. Clunking is the embodiment of the German saying:

“Nichts hält länger als ein Provisorium.” “Nothing lasts longer than a temporary fix.”

Clunking is more than just ignoring “best practices” with consumer hardware. It’s also the realization that enterprise systems have their own janky bits and silly dependencies—they’re just hidden under all the branding. Clunking is kind of the opposite of best practice. Best practice can sometimes be a form of gatekeeping. I am not saying that best practices are wrong. I am saying that best practice is the “approved practice”, approved by people who have access to budgets and teams of experts. “Approving” means deciding who gets taken seriously.

In hackerspace culture, the saying is “Done is better than good”, which means that when choosing between doing something or waiting for the ideal solution, doing is always better than waiting. If you have to choose between a crappy working tool and a theoretical perfect one, go with the crappy working system.

Radical Infrastructure: Punk Rock with Computers#

If there was a musical theme for clunking, it would be Sid Vicious’ version of “My Way”. There certainly better versions of the song, but Sid’s is gnarly. My homelab uses little Linux VMs to imitate firewall/routers because my girlfriend (the infosec professional) doesn’t want me ‘jailbreaking’ her router. She wanted my dangerous hacker stuff isolated from her network, and did it my way.

There is a point in the process where an ugly fix becomes awesome in the way that some hacks are so good that they are billiant. I think if the aesthetic like when a beat up car crosses the threshold to resemble something from Mad Max. The ugliness becomes a form of rebellion. Mad Max is about surviving in a post-apocalypse. When I think about The Dark Ages, they were basically the post-apocalypse of the fall of the Roman Empire. Big Tech monopolies and their spineless vassal politicians are a kind of “Roman Empire” vibe. Just like the VMWare price-hike is a form of Information Technology apocalypse, Big tech “enshittification” is headed for some form of change. Maybe it will reform, maybe it will rupture, but whatever happens, people need a place to go. Like the sanctuaries for civilized people in the wasteland of Mad Max, people need shelter from raiders.