initial config
This commit is contained in:
Jason Ross
+213
@@ -0,0 +1,213 @@
|
||||
# Runbook
|
||||
|
||||
## Day-to-day operations
|
||||
|
||||
### Start / stop services
|
||||
|
||||
```bash
|
||||
systemctl start mimir loki tempo grafana
|
||||
systemctl stop mimir loki tempo grafana
|
||||
systemctl status mimir
|
||||
journalctl -u mimir -f
|
||||
```
|
||||
|
||||
### Reload config
|
||||
|
||||
After editing a config template:
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/playbooks/deploy.yml --ask-vault-pass
|
||||
```
|
||||
|
||||
The handler restarts only services whose config changed.
|
||||
|
||||
### View logs
|
||||
|
||||
```bash
|
||||
journalctl -u loki --since '1 hour ago'
|
||||
podman logs -f loki
|
||||
```
|
||||
|
||||
## WireGuard
|
||||
|
||||
### Add a peer
|
||||
|
||||
1. Peer generates a key pair: `wg genkey | tee priv | wg pubkey > pub`.
|
||||
2. Add to `ansible/group_vars/all/vault.yml` (vault-encrypted):
|
||||
```yaml
|
||||
vault_wireguard_peers:
|
||||
- name: alice-laptop
|
||||
public_key: <CONTENTS_OF_PUB>
|
||||
allowed_ips: 10.8.0.11/32
|
||||
```
|
||||
3. `ansible-playbook playbooks/site.yml --tags wireguard --ask-vault-pass`
|
||||
4. Send peer their config (including server public key + endpoint).
|
||||
|
||||
### Revoke a peer
|
||||
|
||||
1. Remove the peer block from vault.
|
||||
2. Re-run the wireguard role.
|
||||
3. Live revocation: `wg set wg0 peer <PUB> remove`.
|
||||
|
||||
## GDPR right to erasure
|
||||
|
||||
```bash
|
||||
ansible-playbook ansible/playbooks/gdpr-purge.yml \
|
||||
-e tenant=team-alpha \
|
||||
-e 'selector={user_id="abc-123"}' \
|
||||
-e start=2025-01-01T00:00:00Z \
|
||||
--ask-vault-pass
|
||||
```
|
||||
|
||||
The Loki compactor processes deletes after `retention_delete_delay = 2h`.
|
||||
Confirm completion via:
|
||||
|
||||
```bash
|
||||
curl -H 'X-Scope-OrgID: team-alpha' \
|
||||
'http://10.8.0.1:3100/loki/api/v1/delete'
|
||||
```
|
||||
|
||||
Audit log: `/var/log/observability/gdpr-audit.log`.
|
||||
|
||||
## Breach response
|
||||
|
||||
1. **Detect** — alert from Grafana audit log or Hetzner abuse notice.
|
||||
2. **Contain** — block the affected WireGuard peer, rotate Hetzner Object
|
||||
Storage credentials (Cloud Console → Security → Generate new keys), update
|
||||
`vault.yml`, redeploy.
|
||||
3. **Assess** — what tenant, what data, what window. Use Grafana audit log
|
||||
and journald for forensics.
|
||||
4. **Notify** — within 72 hours notify **BayLDA**
|
||||
(https://www.lda.bayern.de/). If high risk to data subjects, notify them
|
||||
directly per Art. 34.
|
||||
5. **Document** — append to `/var/log/observability/breach-log.md`.
|
||||
|
||||
## Recovery / DR
|
||||
|
||||
- All telemetry is reproducible from clients — no need to back up local WAL.
|
||||
- Grafana DB (dashboards, users, orgs): nightly export via
|
||||
`scripts/backup-grafana.sh` to S3 (`watchtower-grafana-backup`).
|
||||
- OpenTofu state: versioned in Cloudflare R2 with object versioning enabled.
|
||||
- Server replacement procedure:
|
||||
1. `tofu apply -replace=hcloud_server.watchtower`
|
||||
2. `ansible-playbook playbooks/site.yml`
|
||||
3. Restore Grafana DB from backup.
|
||||
4. Buckets are unchanged → telemetry resumes ingesting.
|
||||
|
||||
## Common issues
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---------|--------------|-----|
|
||||
| Service won't start, `cannot assign requested address` | wg0 not up | `systemctl status wg-quick@wg0`; restart |
|
||||
| Mimir 403 from S3 | bad keys in `s3.env` | regenerate Hetzner keys, redeploy |
|
||||
| Loki "no org id" | client missing X-Scope-OrgID header | check Alloy config |
|
||||
| LUKS volume not mounted after reboot | keyfile missing | check `/root/observability.luks.key` exists, re-run luks role |
|
||||
| Grafana login redirects loop | cookie_secure=true behind plain HTTP | leave cookie_secure=false (VPN provides encryption) |
|
||||
|
||||
## Host firewall (nftables)
|
||||
|
||||
### Inspect ruleset
|
||||
```bash
|
||||
# Watchtower's own table:
|
||||
nft list table inet watchtower
|
||||
|
||||
# Everything (includes fail2ban's table inet f2b-table and Podman's tables):
|
||||
nft list ruleset
|
||||
```
|
||||
|
||||
### Reload after editing
|
||||
The ruleset is rendered to `/etc/nftables/watchtower.nft` and loaded via
|
||||
`/etc/sysconfig/nftables.conf`.
|
||||
|
||||
```bash
|
||||
# Validate before applying:
|
||||
nft --check --file /etc/nftables/watchtower.nft
|
||||
|
||||
# Apply:
|
||||
systemctl reload nftables # or: systemctl restart nftables
|
||||
```
|
||||
|
||||
### Drop counters / log
|
||||
Drops are logged at `info` level (rate-limited 5/min) with prefix `nft-drop-input:`.
|
||||
|
||||
```bash
|
||||
journalctl -k --since '1 hour ago' | grep nft-drop-input
|
||||
nft list table inet watchtower | grep counter
|
||||
```
|
||||
|
||||
### Adjust admin allow lists
|
||||
Edit `ansible/group_vars/all/main.yml` (`admin_allow_ipv4`, `admin_allow_ipv6`) and
|
||||
re-run `ansible-playbook ansible/playbooks/site.yml --tags firewall` (or just rerun
|
||||
the whole site playbook).
|
||||
|
||||
## fail2ban
|
||||
|
||||
### Status
|
||||
```bash
|
||||
fail2ban-client status # list of active jails
|
||||
fail2ban-client status sshd # per-jail: banned IPs, totals
|
||||
fail2ban-client status recidive
|
||||
```
|
||||
|
||||
### Unban an IP
|
||||
```bash
|
||||
fail2ban-client set sshd unbanip 203.0.113.42
|
||||
fail2ban-client set recidive unbanip 203.0.113.42
|
||||
```
|
||||
|
||||
### Inspect bans at the kernel level
|
||||
fail2ban uses its own nftables table — bans are sets inside `inet f2b-table`:
|
||||
|
||||
```bash
|
||||
nft list table inet f2b-table
|
||||
```
|
||||
|
||||
### Logs
|
||||
```bash
|
||||
tail -f /var/log/fail2ban.log
|
||||
journalctl -u fail2ban -f
|
||||
```
|
||||
|
||||
### Tuning
|
||||
Defaults live in `ansible/roles/fail2ban/defaults/main.yml`. Override in
|
||||
`group_vars/all/main.yml` — e.g. shorten `fail2ban_findtime`, raise
|
||||
`fail2ban_maxretry`, or extend `fail2ban_recidive_bantime`.
|
||||
|
||||
## Automatic security updates
|
||||
|
||||
Configured via `dnf-automatic` (`roles/base`):
|
||||
|
||||
- **Scope**: `upgrade_type = security` only — feature/major version bumps are not auto-applied.
|
||||
- **Apply + reboot**: `apply_updates = yes`, `reboot = when-needed` (kernel/glibc/systemd
|
||||
triggers `shutdown -r +5`).
|
||||
- **Schedule**: `dnf-automatic-install.timer`, pinned via drop-in to `01:00 America/Chicago` (follows CST/CDT automatically)
|
||||
with up to 30m random delay.
|
||||
- **Excluded packages** (operator applies these manually): `podman*`, `conmon`,
|
||||
`crun`, `containers-common`, `netavark`, `aardvark-dns`, `container-selinux` —
|
||||
upgrading these mid-day would restart the observability stack.
|
||||
|
||||
### Inspect
|
||||
```bash
|
||||
systemctl list-timers dnf-automatic-install.timer
|
||||
systemctl status dnf-automatic-install.service
|
||||
journalctl -u dnf-automatic-install.service --since '2 days ago'
|
||||
dnf updateinfo list security # what's pending
|
||||
dnf needs-restarting -r # is a reboot needed?
|
||||
```
|
||||
|
||||
### Run on demand
|
||||
```bash
|
||||
systemctl start dnf-automatic-install.service
|
||||
```
|
||||
|
||||
### Update the excluded container packages (manual)
|
||||
Schedule a maintenance window, then:
|
||||
```bash
|
||||
dnf -y upgrade podman conmon crun containers-common netavark aardvark-dns container-selinux
|
||||
systemctl restart mimir loki tempo grafana
|
||||
```
|
||||
|
||||
### Disable temporarily
|
||||
```bash
|
||||
systemctl disable --now dnf-automatic-install.timer
|
||||
```
|
||||
Reference in New Issue
Block a user