watchtower-observability-stack/docs/runbook.md

Runbook

Day-to-day operations

Start / stop services

systemctl start mimir loki tempo grafana
systemctl stop  mimir loki tempo grafana
systemctl status mimir
journalctl -u mimir -f

Reload config

After editing a config template:

ansible-playbook ansible/playbooks/deploy.yml --ask-vault-pass

The handler restarts only services whose config changed.

View logs

journalctl -u loki --since '1 hour ago'
podman logs -f loki

WireGuard

Add a peer

1. Peer generates a key pair: wg genkey | tee priv | wg pubkey > pub.
2. Add to ansible/group_vars/all/vault.yml (vault-encrypted):

   vault_wireguard_peers:
     - name: alice-laptop
       public_key: <CONTENTS_OF_PUB>
       allowed_ips: 10.8.0.11/32
   

3. ansible-playbook playbooks/site.yml --tags wireguard --ask-vault-pass
4. Send peer their config (including server public key + endpoint).

Revoke a peer

1. Remove the peer block from vault.
2. Re-run the wireguard role.
3. Live revocation: wg set wg0 peer <PUB> remove.

GDPR right to erasure

ansible-playbook ansible/playbooks/gdpr-purge.yml \
  -e tenant=team-alpha \
  -e 'selector={user_id="abc-123"}' \
  -e start=2025-01-01T00:00:00Z \
  --ask-vault-pass

The Loki compactor processes deletes after retention_delete_delay = 2h. Confirm completion via:

curl -H 'X-Scope-OrgID: team-alpha' \
     'http://10.8.0.1:3100/loki/api/v1/delete'

Audit log: /var/log/observability/gdpr-audit.log.

Breach response

1. Detect — alert from Grafana audit log or Hetzner abuse notice.
2. Contain — block the affected WireGuard peer, rotate Hetzner Object Storage credentials (Cloud Console → Security → Generate new keys), update vault.yml, redeploy.
3. Assess — what tenant, what data, what window. Use Grafana audit log and journald for forensics.
4. Notify — within 72 hours notify BayLDA (https://www.lda.bayern.de/). If high risk to data subjects, notify them directly per Art. 34.
5. Document — append to /var/log/observability/breach-log.md.

Recovery / DR

• All telemetry is reproducible from clients — no need to back up local WAL.
• Grafana DB (dashboards, users, orgs): nightly export via scripts/backup-grafana.sh to S3 (watchtower-grafana-backup).
• OpenTofu state: versioned in Cloudflare R2 with object versioning enabled.
• Server replacement procedure:
  1. tofu apply -replace=hcloud_server.watchtower
  2. ansible-playbook playbooks/site.yml
  3. Restore Grafana DB from backup.
  4. Buckets are unchanged → telemetry resumes ingesting.

Common issues

Symptom	Likely cause	Fix
Service won't start, cannot assign requested address	wg0 not up	systemctl status wg-quick@wg0; restart
Mimir 403 from S3	bad keys in s3.env	regenerate Hetzner keys, redeploy
Loki "no org id"	client missing X-Scope-OrgID header	check Alloy config
LUKS volume not mounted after reboot	keyfile missing	check /root/observability.luks.key exists, re-run luks role
Grafana login redirects loop	cookie_secure=true behind plain HTTP	leave cookie_secure=false (VPN provides encryption)

Host firewall (nftables)

Inspect ruleset

# Watchtower's own table:
nft list table inet watchtower

# Everything (includes fail2ban's table inet f2b-table and Podman's tables):
nft list ruleset

Reload after editing

The ruleset is rendered to /etc/nftables/watchtower.nft and loaded via /etc/sysconfig/nftables.conf.

# Validate before applying:
nft --check --file /etc/nftables/watchtower.nft

# Apply:
systemctl reload nftables   # or: systemctl restart nftables

Drop counters / log

Drops are logged at info level (rate-limited 5/min) with prefix nft-drop-input:.

journalctl -k --since '1 hour ago' | grep nft-drop-input
nft list table inet watchtower | grep counter

Adjust admin allow lists

Edit ansible/group_vars/all/main.yml (admin_allow_ipv4, admin_allow_ipv6) and re-run ansible-playbook ansible/playbooks/site.yml --tags firewall (or just rerun the whole site playbook).

fail2ban

Status

fail2ban-client status              # list of active jails
fail2ban-client status sshd         # per-jail: banned IPs, totals
fail2ban-client status recidive

Unban an IP

fail2ban-client set sshd unbanip 203.0.113.42
fail2ban-client set recidive unbanip 203.0.113.42

Inspect bans at the kernel level

fail2ban uses its own nftables table — bans are sets inside inet f2b-table:

nft list table inet f2b-table

Logs

tail -f /var/log/fail2ban.log
journalctl -u fail2ban -f

Tuning

Defaults live in ansible/roles/fail2ban/defaults/main.yml. Override in group_vars/all/main.yml — e.g. shorten fail2ban_findtime, raise fail2ban_maxretry, or extend fail2ban_recidive_bantime.

Automatic security updates

Configured via dnf-automatic (roles/base):

• Scope: upgrade_type = security only — feature/major version bumps are not auto-applied.
• Apply + reboot: apply_updates = yes, reboot = when-needed (kernel/glibc/systemd triggers shutdown -r +5).
• Schedule: dnf-automatic-install.timer, pinned via drop-in to 01:00 America/Chicago (follows CST/CDT automatically) with up to 30m random delay.
• Excluded packages (operator applies these manually): podman*, conmon, crun, containers-common, netavark, aardvark-dns, container-selinux — upgrading these mid-day would restart the observability stack.

Inspect

systemctl list-timers dnf-automatic-install.timer
systemctl status dnf-automatic-install.service
journalctl -u dnf-automatic-install.service --since '2 days ago'
dnf updateinfo list security        # what's pending
dnf needs-restarting -r             # is a reboot needed?

Run on demand

systemctl start dnf-automatic-install.service

Update the excluded container packages (manual)

Schedule a maintenance window, then:

dnf -y upgrade podman conmon crun containers-common netavark aardvark-dns container-selinux
systemctl restart mimir loki tempo grafana

Disable temporarily

systemctl disable --now dnf-automatic-install.timer