JMR-dev/watchtower-observability-stack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial config

Browse Source
This commit is contained in:
Jason Ross
2026-05-09 20:08:13 -05:00
commit 267b6fc39b
54 changed files with 2611 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+82
View File
@@ -0,0 +1,82 @@
# Architecture
## Overview
Watchtower is a single-node observability stack running on a Hetzner CPX42 in
Nuremberg (nbg1). It hosts Mimir (metrics), Loki (logs), Tempo (traces), and
Grafana (UI). Clients ship telemetry over a WireGuard VPN; nothing is exposed
to the public internet.
## Diagram
```
Internet
│
┌───────────┴───────────┐
│ Hetzner Cloud Firewall│ allow: 22/tcp, 51820/udp
└───────────┬───────────┘
│
┌──────┴──────┐
│ AlmaLinux 10│
│ CPX42 │
│ │
│ nftables │ default-drop input + fail2ban
│ ┌──────────┴──────────┐
│ │ wg0 (10.8.0.1/24) │ ◄── WireGuard peers
│ └──┬──────────────────┘
│ │ all service ports bind to 10.8.0.1
│ ┌──┴──────────────────────┐
│ │ Podman observability │
│ │ network (172.20.0.0/24) │
│ │ mimir :8080 :9095 │
│ │ loki :3100 │
│ │ tempo :3200/4317/4318│
│ │ grafana :3000 │
│ └──┬──────────────────────┘
│ │
│ /var/lib/observability ◄── LUKS2 (xfs)
└─────┘
│ S3 API (HTTPS)
▼
Hetzner Object Storage (nbg1)
- watchtower-mimir
- watchtower-loki
- watchtower-tempo
```
## Data flow
1. **Clients** run Grafana Alloy with PII scrubbing pipelines (see
`config/alloy/config.alloy.example`).
2. Alloy ships **logs → Loki**, **traces → Tempo OTLP**, **metrics → Mimir
remote_write**, all over WireGuard with `X-Scope-OrgID` headers.
3. Mimir/Loki/Tempo write blocks/chunks to Hetzner Object Storage. WALs and
compactor scratch live on the LUKS-encrypted local volume.
4. Grafana queries Mimir/Loki/Tempo via the `observability` Podman bridge
(container DNS), with tenant header injected per Org's datasource config.
## Components and ports
| Service | Container | Bind | Public? |
|----------|-----------|------------------|---------|
| Mimir | mimir | 10.8.0.1:8080,9095| no |
| Loki | loki | 10.8.0.1:3100 | no |
| Tempo | tempo | 10.8.0.1:3200, OTLP 4317/4318 | no |
| Grafana | grafana | 10.8.0.1:3000 | no |
## Storage
| Layer | Backend | Encryption |
|-------------|----------------|-----------------------------|
| Long-term | Hetzner S3 | Server-side (Hetzner SSE) |
| Local WAL | LUKS2 + xfs | At-rest (LUKS2 on /dev/sdX or loop) |
| Transit | WireGuard | ChaCha20-Poly1305 |
## Lifecycle
- **OpenTofu** provisions the server, firewall, and S3 buckets (state in
Cloudflare R2).
- **Ansible** configures AlmaLinux, LUKS, nftables + fail2ban, WireGuard, and deploys
the Podman Quadlets.
- **systemd Quadlet generator** turns `*.container` files into
`mimir.service`, `loki.service`, `tempo.service`, `grafana.service`.
docs/gdpr-compliance.md
+137
View File
@@ -0,0 +1,137 @@
# GDPR Compliance
This stack processes telemetry that is **likely to contain personal data**
(IPs, user IDs, request bodies, emails). The deployment is hosted by Hetzner
in Nuremberg, Germany — all data stays within the EU.
## Roles
- **Data Controller:** the operator deploying this stack.
- **Data Processor (infrastructure):** Hetzner Online GmbH (DE).
- **Sub-processors:** none by default.
## Legal basis (Art. 6)
Document the basis for each tenant under
`docs/gdpr-compliance.md` (per-tenant section to be filled in). Typical bases:
- **Art. 6(1)(f)** legitimate interest — operational logging.
- **Art. 6(1)(b)** contract performance — telemetry necessary to deliver service.
## Required documents
| Document | Status | Notes |
|----------|--------|-------|
| Data Processing Agreement with Hetzner | sign in Cloud Console | required Art. 28 |
| Record of Processing Activities (RoPA) | template below | required Art. 30 |
| Data Protection Impact Assessment (DPIA) | required | Art. 35 — high-risk processing |
| Breach response runbook | `docs/runbook.md` | Art. 33/34 |
## Data residency
All telemetry buckets live in `nbg1` (Hetzner, Nuremberg). The OpenTofu
state lives in Cloudflare R2 — operators must configure R2 to use only EU
jurisdictions (`Jurisdiction: EU`) when creating the state bucket, otherwise
state metadata may be replicated outside the EU.
## Retention register
| Signal | Application Retention | Bucket Lifecycle | Effective Erasure Window |
|---------|----------------------|------------------|--------------------------|
| Logs | 30 days (Loki) | 90 days (S3 lifecycle) | 30 days (compactor) |
| Metrics | 90 days (Mimir) | 365 days (S3 lifecycle)| 90 days (compactor) |
| Traces | 14 days (Tempo) | 30 days (S3 lifecycle) | 14 days (compactor) |
Bucket-level lifecycle is a **safety net** — application-level retention is
the primary erasure mechanism. The wider lifecycle window allows
application-level rollback in case of a misconfiguration.
## PII minimization
### Collector-side scrubbing
Clients run Grafana Alloy with the redaction pipeline at
`config/alloy/config.alloy.example`. Patterns redacted:
- IPv4 / IPv6 addresses → `[REDACTED-IP*]`
- Email addresses → `[REDACTED-EMAIL]`
- JWT tokens → `[REDACTED-JWT]`
- Bearer tokens in `Authorization` headers → `[REDACTED-TOKEN]`
- Card numbers (PAN, 13–19 digits) → `[REDACTED-PAN]`
- UUIDs → `[REDACTED-UUID]`
- Lines containing `password=`, `ssn=`, `cvv=`, `cvc=` → **dropped**
### OTLP attribute scrubbing
The `otelcol.processor.attributes` block hashes/deletes these attributes
before traces reach Tempo:
| Attribute | Action |
|-----------|--------|
| `enduser.id` | hashed (correlatable but not reversible) |
| `user.email` | deleted |
| `client.address` | deleted |
| `http.request.header.authorization` | deleted |
| `http.request.header.cookie` | deleted |
| `http.request.body` | deleted |
### Label / cardinality discipline
Mimir and Tempo do **not** support per-record deletion. To stay compliant,
**no PII may ever appear in metric labels or trace resource attributes**.
Tenants must use opaque identifiers (UUIDs hashed at the collector, opaque
session IDs) — never emails, names, or phone numbers as label values.
## Right to erasure (Art. 17)
| Signal | Mechanism | RTO |
|---------|-----------|-----|
| Logs | Loki Delete API via `playbooks/gdpr-purge.yml` | hours |
| Metrics | Retention only — short window + label discipline | ≤90 days |
| Traces | Retention only — 14-day window | ≤14 days |
Document each erasure in `/var/log/observability/gdpr-audit.log` (the playbook
appends automatically). Notify the data subject within one month per Art. 12(3).
## Encryption
| Layer | Mechanism |
|-------|-----------|
| In transit (client → server) | WireGuard (ChaCha20-Poly1305) |
| In transit (server → S3) | TLS 1.2+ |
| At rest (local WAL/Grafana DB) | LUKS2 |
| At rest (S3) | Hetzner SSE |
## Access control & audit
- WireGuard: each peer has its own key; revoke by removing from `wg0.conf`.
- Host firewall: nftables with default-drop input policy. Service ports are
bound to `10.8.0.1` (wg0) and unreachable on the public interface. SSH and
WireGuard are restricted to administrator CIDRs (`admin_allow_ipv4` /
`admin_allow_ipv6`) and rate-limited at the kernel.
- fail2ban: brute-force defense for SSH (jail `sshd`) plus a `recidive` jail
that long-bans repeat offenders. Bans are enforced via fail2ban's own
nftables table (`inet f2b-table`).
- Grafana: `[auditing]` enabled — all dashboard / API actions logged to
`/var/log/grafana/audit.log` with rotation.
- SSH: ed25519 keys only, root password auth disabled.
- Tenant isolation: enforced by `X-Scope-OrgID` at Mimir/Loki/Tempo plus
Grafana Org RBAC.
## Breach response (Art. 33/34)
72-hour notification to the **Bayerisches Landesamt für Datenschutzaufsicht
(BayLDA)** — Bavaria's supervisory authority. Procedure documented in
`docs/runbook.md` § Breach Response.
## RoPA template
For each tenant, fill in:
- Name & contact of controller
- Categories of data subjects
- Categories of personal data processed in telemetry
- Recipients (none — internal only)
- Transfers outside EEA (none)
- Retention periods (per signal — see above)
- Technical & organisational measures (this document)
docs/multi-tenancy.md
+60
View File
@@ -0,0 +1,60 @@
# Multi-tenancy
This stack is multi-tenant: multiple teams or projects share the same Mimir,
Loki, Tempo, and Grafana instances. Tenant isolation is enforced at two layers:
1. **Storage layer** — Mimir/Loki/Tempo route reads and writes by the
`X-Scope-OrgID` HTTP header. Each tenant has a unique slug.
2. **Grafana layer** — one Org per tenant. Each Org has its own datasources
that inject the tenant's `X-Scope-OrgID` automatically.
## Tenant slug rules
- Lowercase ASCII letters, digits, and dashes only.
- Max 40 characters.
- Must be unique across the deployment.
- Examples: `team-alpha`, `web-prod`, `customer-acme`.
## Onboarding
```bash
ansible-playbook ansible/playbooks/onboard-tenant.yml \
-e tenant_slug=team-alpha \
-e tenant_name='Team Alpha' \
-e admin_email=alpha-admin@example.com
```
This creates a Grafana Org named "Team Alpha", switches into it, and
provisions Mimir/Loki/Tempo datasources whose secure HTTP header
`X-Scope-OrgID = team-alpha` is set.
## Client configuration
Clients use Grafana Alloy with the tenant slug embedded:
```alloy
loki.write "default" {
endpoint {
url = "http://10.8.0.1:3100/loki/api/v1/push"
headers = { "X-Scope-OrgID" = "team-alpha" }
}
}
```
See `config/alloy/config.alloy.example`.
## Offboarding
1. Submit Loki delete requests for the tenant's logs:
`ansible-playbook playbooks/gdpr-purge.yml -e tenant=team-alpha -e selector='{job=~".+"}' -e start=2020-01-01T00:00:00Z`
2. Stop ingesting (revoke the tenant's WireGuard peer + remove their Alloy
instance).
3. Delete the Grafana Org via the Grafana API.
4. Wait for retention to lapse on Mimir/Tempo (90/14 days respectively) — they
have no per-tenant deletion API on this version.
## Per-tenant overrides
Mimir limits, Loki retention, etc. can be customized per tenant via
`runtime.yaml` (Mimir) or `limits_config` overrides (Loki). Add per-tenant
sections under `overrides:` in each config — see upstream docs for the schema.
docs/runbook.md
+213
View File
@@ -0,0 +1,213 @@
# Runbook
## Day-to-day operations
### Start / stop services
```bash
systemctl start mimir loki tempo grafana
systemctl stop mimir loki tempo grafana
systemctl status mimir
journalctl -u mimir -f
```
### Reload config
After editing a config template:
```bash
ansible-playbook ansible/playbooks/deploy.yml --ask-vault-pass
```
The handler restarts only services whose config changed.
### View logs
```bash
journalctl -u loki --since '1 hour ago'
podman logs -f loki
```
## WireGuard
### Add a peer
1. Peer generates a key pair: `wg genkey | tee priv | wg pubkey > pub`.
2. Add to `ansible/group_vars/all/vault.yml` (vault-encrypted):
```yaml
vault_wireguard_peers:
- name: alice-laptop
public_key: <CONTENTS_OF_PUB>
allowed_ips: 10.8.0.11/32
```
3. `ansible-playbook playbooks/site.yml --tags wireguard --ask-vault-pass`
4. Send peer their config (including server public key + endpoint).
### Revoke a peer
1. Remove the peer block from vault.
2. Re-run the wireguard role.
3. Live revocation: `wg set wg0 peer <PUB> remove`.
## GDPR right to erasure
```bash
ansible-playbook ansible/playbooks/gdpr-purge.yml \
-e tenant=team-alpha \
-e 'selector={user_id="abc-123"}' \
-e start=2025-01-01T00:00:00Z \
--ask-vault-pass
```
The Loki compactor processes deletes after `retention_delete_delay = 2h`.
Confirm completion via:
```bash
curl -H 'X-Scope-OrgID: team-alpha' \
'http://10.8.0.1:3100/loki/api/v1/delete'
```
Audit log: `/var/log/observability/gdpr-audit.log`.
## Breach response
1. **Detect** — alert from Grafana audit log or Hetzner abuse notice.
2. **Contain** — block the affected WireGuard peer, rotate Hetzner Object
Storage credentials (Cloud Console → Security → Generate new keys), update
`vault.yml`, redeploy.
3. **Assess** — what tenant, what data, what window. Use Grafana audit log
and journald for forensics.
4. **Notify** — within 72 hours notify **BayLDA**
(https://www.lda.bayern.de/). If high risk to data subjects, notify them
directly per Art. 34.
5. **Document** — append to `/var/log/observability/breach-log.md`.
## Recovery / DR
- All telemetry is reproducible from clients — no need to back up local WAL.
- Grafana DB (dashboards, users, orgs): nightly export via
`scripts/backup-grafana.sh` to S3 (`watchtower-grafana-backup`).
- OpenTofu state: versioned in Cloudflare R2 with object versioning enabled.
- Server replacement procedure:
1. `tofu apply -replace=hcloud_server.watchtower`
2. `ansible-playbook playbooks/site.yml`
3. Restore Grafana DB from backup.
4. Buckets are unchanged → telemetry resumes ingesting.
## Common issues
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Service won't start, `cannot assign requested address` | wg0 not up | `systemctl status wg-quick@wg0`; restart |
| Mimir 403 from S3 | bad keys in `s3.env` | regenerate Hetzner keys, redeploy |
| Loki "no org id" | client missing X-Scope-OrgID header | check Alloy config |
| LUKS volume not mounted after reboot | keyfile missing | check `/root/observability.luks.key` exists, re-run luks role |
| Grafana login redirects loop | cookie_secure=true behind plain HTTP | leave cookie_secure=false (VPN provides encryption) |
## Host firewall (nftables)
### Inspect ruleset
```bash
# Watchtower's own table:
nft list table inet watchtower
# Everything (includes fail2ban's table inet f2b-table and Podman's tables):
nft list ruleset
```
### Reload after editing
The ruleset is rendered to `/etc/nftables/watchtower.nft` and loaded via
`/etc/sysconfig/nftables.conf`.
```bash
# Validate before applying:
nft --check --file /etc/nftables/watchtower.nft
# Apply:
systemctl reload nftables # or: systemctl restart nftables
```
### Drop counters / log
Drops are logged at `info` level (rate-limited 5/min) with prefix `nft-drop-input:`.
```bash
journalctl -k --since '1 hour ago' | grep nft-drop-input
nft list table inet watchtower | grep counter
```
### Adjust admin allow lists
Edit `ansible/group_vars/all/main.yml` (`admin_allow_ipv4`, `admin_allow_ipv6`) and
re-run `ansible-playbook ansible/playbooks/site.yml --tags firewall` (or just rerun
the whole site playbook).
## fail2ban
### Status
```bash
fail2ban-client status # list of active jails
fail2ban-client status sshd # per-jail: banned IPs, totals
fail2ban-client status recidive
```
### Unban an IP
```bash
fail2ban-client set sshd unbanip 203.0.113.42
fail2ban-client set recidive unbanip 203.0.113.42
```
### Inspect bans at the kernel level
fail2ban uses its own nftables table — bans are sets inside `inet f2b-table`:
```bash
nft list table inet f2b-table
```
### Logs
```bash
tail -f /var/log/fail2ban.log
journalctl -u fail2ban -f
```
### Tuning
Defaults live in `ansible/roles/fail2ban/defaults/main.yml`. Override in
`group_vars/all/main.yml` — e.g. shorten `fail2ban_findtime`, raise
`fail2ban_maxretry`, or extend `fail2ban_recidive_bantime`.
## Automatic security updates
Configured via `dnf-automatic` (`roles/base`):
- **Scope**: `upgrade_type = security` only — feature/major version bumps are not auto-applied.
- **Apply + reboot**: `apply_updates = yes`, `reboot = when-needed` (kernel/glibc/systemd
triggers `shutdown -r +5`).
- **Schedule**: `dnf-automatic-install.timer`, pinned via drop-in to `01:00 America/Chicago` (follows CST/CDT automatically)
with up to 30m random delay.
- **Excluded packages** (operator applies these manually): `podman*`, `conmon`,
`crun`, `containers-common`, `netavark`, `aardvark-dns`, `container-selinux` —
upgrading these mid-day would restart the observability stack.
### Inspect
```bash
systemctl list-timers dnf-automatic-install.timer
systemctl status dnf-automatic-install.service
journalctl -u dnf-automatic-install.service --since '2 days ago'
dnf updateinfo list security # what's pending
dnf needs-restarting -r # is a reboot needed?
```
### Run on demand
```bash
systemctl start dnf-automatic-install.service
```
### Update the excluded container packages (manual)
Schedule a maintenance window, then:
```bash
dnf -y upgrade podman conmon crun containers-common netavark aardvark-dns container-selinux
systemctl restart mimir loki tempo grafana
```
### Disable temporarily
```bash
systemctl disable --now dnf-automatic-install.timer
```