Gitea/README.md

# Gitea on GCE — podman quadlets, Pulumi, Cloud Build

A self-hosted [Gitea](https://gitea.com) instance on a single Google Compute
Engine VM, served at `gitea.jasonmross.dev`.

| Layer | Choice |
|---|---|
| Host | AlmaLinux 10 (`almalinux-cloud/almalinux-10`), Shielded VM |
| Size | `e2-small` (2 shared vCPU, 2 GB RAM) in `us-east1`, 20 GB boot + 30 GB pd-balanced data |
| Runtime | Podman **quadlets** — systemd `.container`/`.network`/`.volume` units, not compose |
| Images | Gitea and Caddy, both built on **Debian 13 (trixie)** |
| TLS | Caddy with ACME **DNS-01** via Google Cloud DNS (custom `xcaddy` build) |
| WAF | **Coraza** + OWASP CRS compiled into Caddy, feeding fail2ban → nftables |
| Database | SQLite in WAL mode on a dedicated persistent disk |
| Firewall | nftables + fail2ban on the host, VPC firewall outside it |
| IaC | Pulumi, Go, self-managed GCS state backend |
| CI/CD | Cloud Build — push triggers plus a weekly rebuild |

## Layout

```
infra/        Pulumi program (Go). One package per slice of infrastructure.
image/        Dockerfiles for the Gitea and Caddy images, plus pinned versions.
vm/           Everything that lands on the VM: quadlets, systemd units,
              nftables, fail2ban, config templates, and bootstrap.sh.
cloudbuild/   The two build pipelines.
scripts/      One-time project bootstrap.
docs/         Runbook, WAF tuning guide, migration-to-Gitea-SCM plan.
```

`vm/` is uploaded to a GCS bucket by Pulumi and pulled down by the instance, so
changing a quadlet is a normal pull request.

## First-time setup

Pulumi cannot create the bucket holding its own state, the identity that runs
it, or Gitea's signing secrets — so there is one manual step first.

```bash
# 1. Project bootstrap: APIs, state bucket, Pulumi runner SA, Gitea secrets.
scripts/bootstrap.sh <project-id> us-east1

# 2. Install the Cloud Build GitHub App on this repository, note the
#    installation id, and store a PAT (repo + read:user scope):
printf %s '<token>' | gcloud secrets versions add github-pat --data-file=- --project <project-id>

# 3. Confirm the Cloud DNS zone is authoritative. DNS-01 cannot work otherwise.
dig NS gitea.jasonmross.dev

# 4. The ACME contact address. Kept in Secret Manager, not stack config, so it
#    stays out of this public repo; the VM reads it when rendering the Caddyfile.
printf %s 'you@example.com' | gcloud secrets versions add gitea-acme-email --data-file=- --project <project-id>

# 5. Configure and apply. Pulumi's GCS backend and Google provider use
#    Application Default Credentials, not your gcloud login.
gcloud auth application-default login
# Use the passphrase bootstrap.sh generated. Letting `stack init` prompt for a
# new one encrypts the stack with a key Cloud Build's infra trigger never sees.
export PULUMI_CONFIG_PASSPHRASE=$(gcloud secrets versions access latest \
    --secret=pulumi-config-passphrase --project <project-id>)
cd infra
pulumi login gs://<project-id>-pulumi-state
pulumi stack init prod
pulumi config set gcp:project <project-id>
pulumi config set gitea:domain gitea.jasonmross.dev
pulumi config set gitea:dnsZone <cloud-dns-managed-zone-name>   # gcloud dns managed-zones list
pulumi config set gitea:githubOwner <owner>
pulumi config set gitea:githubAppInstallationId <id>
pulumi config set gitea:infraBuildServiceAccount cb-infra@<project-id>.iam.gserviceaccount.com
# WAF starts in DetectionOnly. Tune, then switch to On -- see docs/waf.md.
pulumi config set gitea:wafMode DetectionOnly
pulumi up

# 6. First image build. Until this runs, the :prod images do not exist.
cd .. && make build

# 7. Create the admin user.
make ssh
sudo podman exec -u 1000 gitea gitea admin user create \
    -c /etc/gitea/app.ini --admin --username <you> --email <you@example.com> --random-password
```

### Expected on the first run, not a bug

Between step 5 and step 6 the `:prod` images do not exist yet, so `gitea.service`
and `caddy.service` crash-loop. That is intentional: the units carry
`Restart=always` with `StartLimitIntervalSec=0`, so they recover on their own
within 30 seconds of the first successful push. Likewise, `app.ini` is not
rendered until Gitea's secrets are readable — `bootstrap.sh` skips rendering
rather than writing a config with empty signing keys.

## Day-to-day

```bash
make status     # services, containers, timers
make logs       # tail gitea + caddy
make rollout    # pull the latest :prod images now
make sync       # re-render VM config after a vm/ change
make backup     # on-demand gitea dump to GCS
make ssh        # shell via IAP
```

The targets read the project and zone from the Pulumi stack, which needs
Application Default Credentials (`gcloud auth application-default login`).
Without them `pulumi config get` fails quietly and gcloud runs with an empty
`--project=`; pass `PROJECT=<project-id>` to skip the lookup.

A push to `main` under `image/**` builds, pushes, rolls out, and gates on
`/api/healthz`. A push under `infra/**` or `vm/**` runs `pulumi up` and then
re-syncs the VM configuration. Anything else does nothing.

## How updates happen

Three independent layers, because no single one covers everything:

1. **OS packages** — `dnf5-automatic` applies updates nightly. It never reboots;
   `gitea-reboot-window.timer` does that weekly, and only when
   `needs-restarting -r` says a reboot is genuinely required.
2. **Container images** — `podman-auto-update.timer` polls the `:prod` tag daily.
   `Notify=healthy` on the quadlets means systemd withholds "started" until the
   healthcheck passes, which is what arms podman's automatic rollback.
3. **Image contents** — a Cloud Scheduler job re-runs the image build every
   Sunday, rebuilding from a floating `debian:13-slim` so base-OS and Go
   security fixes reach the running containers. Without this, `:prod` never
   changes and layer 2 has nothing to pull.

Bumping the Gitea or Caddy version itself stays a deliberate change to
`image/gitea.version` / `image/caddy.version`.

> **Fire the weekly rebuild once by hand after the first deploy.** A broken
> scheduler request fails silently at 04:00 on a Sunday and stops layer 3
> from feeding layer 2. See *The weekly rebuild* in
> [docs/runbook.md](docs/runbook.md).

## Things worth knowing before you change something

- **The `:prod` tag is load-bearing.** `AutoUpdate=registry` compares digests
  *for a tag*. Pinning a digest in the quadlet silently disables auto-updates.
- **`app.ini` is fully managed** and `INSTALL_LOCK=true`. Gitea settings changed
  in the web UI that map to `app.ini` will not survive a config sync. Edit
  `vm/config/app.ini.tmpl` instead.
- **The podman subnet is pinned** (`10.89.10.0/24`). It is what
  `REVERSE_PROXY_TRUSTED_PROXIES` names; an unpinned subnet would silently make
  fail2ban ban Caddy instead of the attacker.
- **Never put `flush ruleset` in the nftables config.** It would wipe netavark's
  rules and break all container networking on reload.
- **The DNS zone, the backup bucket, and the Gitea secrets are not Pulumi-owned**
  by design, so `pulumi destroy` cannot take them with it.
- **Git and LFS deliberately bypass the WAF.** Remove that bypass and `git push`
  returns 403 — verified, not theoretical. See [docs/waf.md](docs/waf.md).
- **`image/caddy.version` and `image/coraza.version` are coupled.** coraza-caddy
  pins a minimum Caddy version; bump them together or the build fails.
- **`us-east1` has no `-a` zone** (it is b/c/d). The stack pins `us-east1-b`.
- **2 GB of RAM is the real constraint**, not disk or CPU. A 2 GB swap file is
  provisioned as ballast; sustained swap use means move to `e2-medium`. Measured
  numbers are in [docs/runbook.md](docs/runbook.md).

See [docs/runbook.md](docs/runbook.md) for verification drills, restores, and
rollbacks, and [docs/waf.md](docs/waf.md) for WAF tuning and the
`DetectionOnly` → `On` rollout.