4.14Break-fix
- 1Read
A web server is broken and you have a root shell on it. Each scenario starts with the page someone sent you; you diagnose and fix it from a terminal, the way you would at 3am. There are 4 scenarios, and a different one leads the list each day.
- 2Note
Nothing here is a real machine. Each scenario is a script: the commands an engineer would reach for, with output captured from Ubuntu 24.04 and state that changes as you fix things. Anything outside the script says it is not available rather than pretending. Type help to see the tools on the box. Hints cost points, and a destructive mistake ends the scenario.
KeysEnter runs a command · up and down arrows step through history · Ctrl+L clears · type hint for a hint
Pick a broken server
4 scenarios, each a different way a web server falls over at 3am. The day’s pick is first.
The health check that can't write
The load balancer has pulled web-01: its health check returns 500 since about 03:00.
Somebody else on port 80
The shop's home page has turned into a bare directory listing with a database backup in it.
Five restarts and out
502 Bad Gateway since a config change at 02:10. nginx is up; the app behind it is not.
Uploads restored, uploads broken
Customers can't upload product photos, and the health check returns 500 since a 02:10 maintenance job.
4.14.1A first pass on any broken box
- 1Read
Read the error before you touch anything
The status code, the last line of the traceback, the line systemd printed when it gave up. Most incidents say what is wrong in the first screen of output; the work is reading it slowly instead of reaching for a restart.
- 2Check
Check the boring things first
Is the disk full (df -h)? Is the service running (systemctl status)? Is something listening where it should (ss -ltnp)? Who owns the file (ls -l)? These four answer a surprising share of 3am pages.
- 3Find
Find who, not just what
A full disk has a file that filled it and a process writing that file. A taken port has a process ID (PID). A denied write has a user and an owner. Get the name, because the fix is about that name.
- 4Change
Change one thing, the smallest that works
Empty one file, not a directory. Give one tree to one user, not the whole box to everyone. Small changes are easy to explain in the write-up and easy to undo if you were wrong.
- 5Prove
Prove it from the outside
A command that printed nothing has not proved anything: systemctl restart is silent even when the service dies a second later. Finish with the check the users see, usually a curl against the site.
Inspired by SadServers, which hands you a real broken Linux server to fix (its Show HN post on Hacker News). A static page can’t run a server, so these are scripted simulations of the same habits, written for this site.