Recovery Runbook

What to do when something dies. Concrete commands, no theory. Pair with Backups for the data side.

Quick check when "the site is down"

ping discourse.<TAILNET>.ts.net          # is the host up at all?
sudo systemctl status cloudflared        # tunnel up?
docker ps                                # all containers running?
docker ps -a | grep -v 'Up '             # any in Exited/Restarting?
ss -tlnp | sort -k4                      # everything listening on the right ports?

Almost always: cloudflared died, a container crashed, or the box is rebooting.

Service-specific recovery

Discourse won't start

cd /var/discourse
sudo ./launcher logs app                   # what's the error?
sudo ./launcher rebuild app                # most fixes — recreates the container, keeps data

If rebuild fails:

• Free disk? df -h /var — Discourse rebuilds need several GB.
• Image broken? sudo ./launcher cleanup then rebuild.
• app.yml syntax error? git diff it (assuming you committed) — revert.

If data is corrupted:

# Restore from a Discourse backup tarball
cd /var/discourse
sudo ./launcher rebuild app                # ensure container exists
# Web UI: Admin → Backups → Upload → Restore
# Or CLI:
sudo cp <backup>.tar.gz /var/discourse/shared/standalone/backups/default/
sudo ./launcher enter app
discourse enable_restore
discourse restore <backup>.tar.gz

Forgejo won't start

cd /opt/forgejo
docker compose logs -f forgejo
docker compose down && docker compose up -d

Common causes:

• Permissions drift on data/ after a host change → sudo chown -R 1000:1000 /opt/forgejo/data
• sqlite WAL lock → stop containers, delete data/gitea/gitea.db-wal and -shm, restart (only if you're sure no process holds it).
• Disk full → df -h /opt; clear /srv/isos if needed.

Restore from backup:

cd /opt/forgejo
docker compose down
sudo rm -rf data/                          # only after confirming the backup
sudo tar xzf /backups/forgejo-<date>.tar.gz -C /opt/forgejo/
sudo chown -R 1000:1000 data/
docker compose up -d

Filedrop / tusd not accepting uploads

cd /opt/filedrop
docker compose ps
docker compose logs -f tusd
docker compose logs -f nginx

If uploads land in tusdata/ but never move to /uploads:

• Check the post-finish hook: cat /opt/filedrop/hooks/post-finish — must be executable (chmod +x).
• Mount issue: docker exec filedrop-tusd-1 ls /uploads — should show the SFTP bucket contents.
• Manually drain stuck files:

# Move any orphaned tusdata files into /uploads (use sparingly — destination filename will be the tus id, not the original)
cd /opt/filedrop/tusdata
for f in *.bin; do sudo mv "$f" /var/sftp/shared/uploads/recovered_$f; done
sudo rm -f *.info

Rotate the drop URL if it's been leaked — see Filedrop.

SFTP user can't log in

# Reproduce on the box
sudo journalctl -u ssh -f &
sftp guest@localhost                       # in another terminal

# Common causes:
sudo sshd -t                               # syntax check sshd config
id <user>                                  # is sftpusers in their groups?
ls -ld /var/sftp /var/sftp/shared          # both must be root:root, mode 755 (not 775!)
sudo grep <user> /etc/passwd               # shell must be /sbin/nologin

If ChrootDirectory permission is wrong, sshd silently disconnects after auth — the journal will say bad ownership or modes for chroot directory.

Cloudflare Tunnel down

sudo systemctl status cloudflared
sudo journalctl -u cloudflared -n 100 --no-pager

# Common errors:
# "context deadline exceeded" → outbound 7844/tcp blocked or Cloudflare incident
# "404 looking up tunnel" → wrong UUID or tunnel deleted
# "credentials file not found" → /root/.cloudflared/<UUID>.json missing

Bring back up:

sudo systemctl restart cloudflared

Tunnel 2 (cloudflared-thelinuxcast) is a Docker container:

cd /opt/forgejo
docker compose restart cloudflared
docker compose logs -f cloudflared

If the token is rotated/revoked: get the new token from the Cloudflare dashboard, paste into docker-compose.yml, docker compose up -d cloudflared.

Tailscale Funnel down

tailscale status
sudo systemctl status tailscaled
tailscale funnel status

# Re-enable if missing:
sudo tailscale funnel --bg --https=443 http://127.0.0.1:8081

If the node isn't logged in: sudo tailscale up and follow the auth URL.

Filebrowser admin locked out

docker exec filebrowser /filebrowser users update admin --password <new>
docker compose restart filebrowser

Full host loss — rebuild from scratch

Order matters because some services depend on others (cloudflared serves them all, so it goes up last).

1. Install Debian 13 (trixie) on the new box. Hostname discourse, user drew (uid 1000), key-only SSH.
2. Install Docker + compose plugin: apt install docker.io docker-compose-plugin.
3. Install Tailscale: curl -fsSL https://tailscale.com/install.sh | sh, then sudo tailscale up.
4. Restore configs tarball: untar configs-<date>.tar.gz to /.
5. Install cloudflared: apt install cloudflared (or download .deb from CF). Credentials are already in place from step 4.
6. Restore data tarballs:
   • tar xzf forgejo-<date>.tar.gz -C /opt/forgejo/
   • tar xzf filebrowser-<date>.tar.gz -C /home/drew/docker/filebrowser/
   • tar xzf uploads-<date>.tar.gz -C /var/sftp/shared/
7. Recreate SFTP users:
   • groupadd -g 1001 sftpusers
   • For each user, useradd -m -d /var/sftp/shared -s /sbin/nologin -G sftpusers <name> and passwd <name>.
8. Bring stacks up:

   cd /opt/forgejo && docker compose up -d
   cd /opt/filedrop && docker compose up -d
   cd /var/landing && docker compose up -d
   cd /home/drew/docker/filebrowser && docker compose up -d
   

9. Restore Discourse: clone /var/discourse (git clone https://github.com/discourse/discourse_docker.git /var/discourse), restore app.yml from configs tarball, ./launcher rebuild app, then restore via Admin UI from a Discourse backup tarball.
10. Start cloudflared: systemctl enable --now cloudflared.
11. Re-enable Tailscale Funnel: sudo tailscale funnel --bg --https=443 http://127.0.0.1:8081.
12. Verify every public URL: forum, git, files, both landing sites, filedrop.

With backups in hand, this takes a couple of hours. Most of it is Discourse's first rebuild.

When something breaks again

If you fix the same thing twice, write it down here so the third time is fast.