With DNS resolution fixed, issuance failed at the challenge:
presenting for challenge: adding temporary record for zone
"gitea.jasonmross.dev.": googleapi: Error 403: Forbidden
Testing with the VM service account's own token: managedZones/main and
its rrsets return 200, but managedZones (list) returns 403. The
googleclouddns plugin resolves the domain to a zone by listing the
project's managed zones, and listing is a project-level permission that
the zone-scoped dns.admin binding cannot grant.
Grant roles/dns.reader on the project. It adds read access only, in a
project that holds this single zone; every write stays zone-scoped. A
custom role with just dns.managedZones.list was the alternative, but
managing it would need iam.roleAdmin on the cb-infra Pulumi runner,
which widens a far more powerful identity to narrow a read-only one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy could not obtain a certificate on first boot: every request to
acme-v02.api.letsencrypt.org timed out. Containers could reach
1.1.1.1:443 by address but resolved no names at all.
Container DNS goes to aardvark-dns on the bridge gateway (10.89.10.1:53).
That traffic terminates on the host, so it takes the input hook, not
forward. Netavark accepts it in its own table, but gitea_filter's input
chain has policy drop, and a packet must be accepted by every base chain
on the hook. Our drop won.
gitea_filter now accepts tcp/udp 53 from the podman subnet to its
gateway. Both come from instance metadata, so the ruleset is rendered
with envsubst, as the fail2ban jail already is. It is not interface-based
because netavark's bridge name (podman1) is not pinned.
setup_nftables also validates the rendered file before installing it.
Previously it installed first and validated second, so a ruleset that
failed to parse stayed in /etc/sysconfig and would fail nftables.service
on the next boot, leaving the host with no gitea_filter table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gitea:acmeEmail sat in Pulumi.prod.yaml, which this public repository
publishes, and was then copied into instance metadata. It now lives in a
gitea-acme-email secret instead, read by the VM when it renders the
Caddyfile.
- scripts/bootstrap.sh creates the secret empty and prints how to set
it, the same as github-pat: the address is chosen, not generated.
- Pulumi grants the VM secretAccessor on it and nothing more. It is kept
out of secrets.Names, whose members also get secretVersionAdder and are
mapped to `gitea generate secret` by vm/bootstrap.sh.
- The gitea:acmeEmail config key and the acme-email metadata entry are
gone.
- vm/bootstrap.sh renders the whole `email` directive. If the secret is
unreadable it renders a comment instead and warns: Caddy still issues
certificates under an account with no contact address, whereas an
empty `email` would fail to parse and leave nothing serving TLS. Same
directive-or-comment pattern as CADDY_PUBLISH_PORTS and CADDY_SYSCTL.
README setup gains the secret step, plus two that were missing: ADC
login (Pulumi's GCS backend and provider do not use the gcloud login),
and exporting PULUMI_CONFIG_PASSPHRASE from Secret Manager before
`stack init`. Without the latter, init prompts for a new passphrase and
the stack is encrypted with a key Cloud Build's infra trigger never sees.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Self-hosted Gitea on a single e2-small AlmaLinux 10 VM in us-east1,
serving gitea.jasonmross.dev.
Runtime is podman quadlets (systemd .container/.network/.volume units).
Both images are built on Debian 13: Gitea from a GPG-verified release
binary, and Caddy from an xcaddy build carrying the Google Cloud DNS
provider (ACME DNS-01) and the Coraza WAF with the OWASP CRS embedded.
Infrastructure is a Pulumi program in Go against a GCS state backend.
Cloud Build handles CI: a push trigger for images, one for infra, and a
weekly scheduled rebuild. Everything Cloud Build touches is 2nd gen.
Notable design decisions, each documented where it lives:
- Quadlets track a floating :prod tag. AutoUpdate=registry compares
digests for a tag, so a digest-pinned image silently disables
auto-updates.
- Git transport and LFS bypass the WAF. With the bypass removed, a plain
git push returns 403 -- packfiles trip CRS reliably.
- gitea:wafMode drives both SecRuleEngine and whether the fail2ban jail
acting on WAF verdicts exists. Banning on detections that were never
blocks would turn a tuning false positive into an nftables ban.
- fail2ban bans at the nftables prerouting hook. Published container
ports are DNAT'd and never traverse INPUT, where the stock actions
install their rules.
- The DNS zone, backup bucket, and Gitea signing secrets are not
Pulumi-owned, so pulumi destroy cannot take them with it.
- The podman subnet is pinned because it is what Gitea's
REVERSE_PROXY_TRUSTED_PROXIES names.
Three update layers: dnf5-automatic for the OS, podman-auto-update with
health-gated rollback for containers, and a weekly image rebuild that
gives the second layer something to pull.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JMR-devandClaude Opus 5.5