Gitea/docs/runners.md

Gitea Actions runners on GKE

Gitea Actions jobs run on a small GKE cluster in the same project, not on gitea-vm. The VM has no memory to spare (see Memory on a 2 GB instance in runbook.md), and a runner on the host it deploys is exactly the loop migrate-to-gitea-scm.md warns about.

Everything here is behind one flag, gitea:runnersEnabled, which is off by default.

Shape

Piece	Choice	Where
Cluster	GKE Standard, zonal (gitea:zone), named gitea-runners, used for nothing else	infra/pkg/gke
Nodes	One autoscaled pool, e2-standard-2 spot by default, private (no external IPs), COS	infra/pkg/gke
Node identity	gitea-gke-nodes@, holding only roles/container.defaultNodeServiceAccount	infra/pkg/gke
Egress	Cloud NAT on a reserved static IP (runnerEgressIp stack output)	infra/pkg/network
Control plane	DNS endpoint only, authorised by IAM; IP endpoints are disabled	infra/pkg/gke
Runner	StatefulSet of docker.io/gitea/runner:<tag>-dind-rootless, one pod per node	infra/pkg/runner
Registration	Preset token in Secret Manager, shared by Gitea and the runners	scripts/bootstrap.sh
WAF	Runner protocol and artifact routes bypass Coraza	vm/config/Caddyfile.tmpl

Why Standard and not Autopilot: workflows need a Docker daemon (docker build, Docker actions), which means a privileged Docker-in-Docker pod, and Autopilot refuses privileged pods. The runner's newer Kubernetes backend, which runs each job as its own pod, needs no privileges, but it cannot run Docker actions or docker build.

Why zonal: GKE's free tier credits the management fee of one zonal or Autopilot cluster per billing account. If this billing account already has a cluster, this one adds about $73/month.

Cost

Roughly $25–35/month for one replica, provided the free-tier credit covers the cluster fee:

	approx. / month
spot e2-standard-2 node	$15
static NAT IP (in use)	$3.65
Cloud NAT gateway + data processing	$1 + $0.045/GB
boot disk 30 GB + PVCs 10 + 20 GB, pd-balanced	$6

Each extra replica adds a node, its boot disk, and two PVCs.

Threat model

Read this before pointing a repository you do not fully trust at these runners.

• Jobs can act as the runner. Jobs run inside the runner pod's own dockerd and get its socket, which is what makes docker build work. Through that socket a job can read anything the daemon can see, including the runner's /data/.runner credentials and the mounted registration token. A malicious workflow could register a rogue runner, or impersonate this one and pick up other repositories' jobs and secrets. That is acceptable here only because this is a single-tenant instance with DISABLE_REGISTRATION = true: every workflow author is already trusted with every secret.
• Jobs cannot reach Google Cloud. This part is enforced, not assumed:
  • Job pods get no Google identity. The runner's Kubernetes service account holds exactly one permission, read access to the registration token secret.
  • The metadata server is blocked. Job containers' traffic leaves from the runner pod's network namespace, so the GKE metadata server would issue them that KSA's token. The NetworkPolicy blocks 169.254.0.0/16 for that reason. It is load-bearing: do not "simplify" it away.
  • Escaping to the node gains little. dockerd runs as uid 1000 (rootless), so a job that breaks out of its own container lands as an unprivileged user. Root on the node would only yield the node SA, which holds logging and monitoring writes and nothing else.
• The cluster is not a path to the VM's private side. The NetworkPolicy also blocks all RFC1918 ranges. Jobs reach Gitea the way the rest of the internet does: through its public address, Caddy, and the WAF (except the bypassed runner routes below).

The WAF bypass

vm/config/Caddyfile.tmpl routes these prefixes to Gitea without Coraza:

/api/actions/*                                         runner protocol (Connect RPC)
/api/actions_pipeline/*                                artifacts v3, job summaries
/twirp/github.actions.results.api.v1.ArtifactService/* artifacts v4

This is not optional. The runner speaks Connect with binary protobuf bodies (Content-Type: application/proto), and artifact uploads are PUT/PATCH with application/octet-stream. CRS rejects both outright in On mode: the content type is not in tx.allowed_request_content_type, and the methods are not in tx.allowed_methods. Log uploads would trip SQLi/XSS rules on build output even if they got that far. Every bypassed route authenticates with a runner or task token. The reasoning is the same as for the git bypass in waf.md.

The matcher uses prefixes, not a list of RPC names, so a Gitea upgrade that adds an RPC does not silently start returning 403. /api/v1/ is a different prefix and stays behind the WAF. Caddy cleans the path before matching, so /api/actions/../v1/... is inspected like any other /api/v1 request.

fail2ban and the shared egress IP

Every runner and every job shares the Cloud NAT address. With the gitea jail at maxretry = 5, a single job retrying a bad token would ban all of CI for an hour. It would also take the runners offline, since they poll through the same IP. So the VM adds that address to ignoreip, using instance metadata runner-egress-ip. The trade-off is that hostile traffic from a job is never banned, which is acceptable under the threat model above.

Turning it on

1. Re-run scripts/bootstrap.sh <project-id>. It is idempotent, and on an existing project it adds three things:
   • the container.googleapis.com API
   • roles/container.admin for cb-infra@
   • the gitea-runner-registration-token secret, as 64 random hex characters
2. Set the flag and push:

   (cd infra && pulumi config set gitea:runnersEnabled true)
   make preview        # local: the infra trigger only fires on main, so a branch gets no preview
   git commit infra/Pulumi.prod.yaml -m 'Enable GKE runners' && git push
   

   The first apply takes 15–25 minutes: cluster, node pool, then the StatefulSet. If the StatefulSet wait times out (IAM or CSI propagation on a brand-new cluster), run pulumi up again. Everything already created stays.
3. The infra pipeline's config-sync step restarts Gitea with the token. Meanwhile the runner retries registration every 5 seconds. If the runner logs keep saying the token is invalid after the pipeline finishes, the VM most likely read the secret before its IAM grant had propagated. Run make sync.
4. Check:

   make kube && make runners    # pod Running, log shows it registered and declared
   

   make kube needs gke-gcloud-auth-plugin: gcloud components install gke-gcloud-auth-plugin, or google-cloud-cli-gke-gcloud-auth-plugin from the same package repository as a distro-packaged gcloud. Site Administration → Actions → Runners should list gitea-runner-0 as Idle, with labels ubuntu-latest, ubuntu-24.04 and ubuntu-22.04.
5. Run the smoke workflow below once.

Smoke workflow

Put this in a scratch repository as .gitea/workflows/smoke.yaml:

on: [push]
jobs:
  smoke:
    runs-on: ubuntu-latest      # required: runner 5.x fails jobs without runs-on
    steps:
      - uses: actions/checkout@v4
      - name: docker build
        run: |
          printf 'FROM alpine:3\nRUN echo built\n' > Dockerfile
          docker build -t smoke . && docker run --rm smoke echo ran
          docker info | grep -A1 'Registry Mirrors'    # expect mirror.gcr.io
      - uses: actions/cache@v4
        with: { path: cached, key: smoke-${{ github.run_id }}, restore-keys: smoke- }
      - run: mkdir -p cached && date >> cached/stamp && cat cached/stamp
      - uses: actions/upload-artifact@v4
        with: { name: smoke, path: cached }
      - name: metadata server must be unreachable
        run: |
          ! curl -sf -m 5 -H 'Metadata-Flavor: Google' \
            http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
      - name: egress IP
        run: curl -s https://api.ipify.org   # must equal `pulumi stack output runnerEgressIp`

Push twice. The second run's cached/stamp should have two lines. While it runs, make logs should show no Coraza audit entries for /api/actions, /api/actions_pipeline or /twirp/.

Day-to-day

Task	How
More parallel jobs on one node	pulumi config set gitea:runnerCapacity 2
More nodes	gitea:runnerReplicas (≤ gitea:runnerMaxNodes, max 10)
On-demand instead of spot	gitea:runnerSpot false. The pool is replaced create-first, so the runners reschedule.
Bigger nodes	gitea:runnerMachineType e2-standard-4
Upgrade the runner	bump gitea:runnerImageTag; it must stay a -dind-rootless tag
Shell into the runner	kubectl --context=gke_<p>_<zone>_gitea-runners -n gitea-runner exec -it gitea-runner-0 -- sh

Scaling follows gitea:runnerReplicas, not the job queue. Pod anti-affinity puts one runner pod on each node. The node pool's minimum and initial size are max(1, runnerReplicas): at least 1 because the runner must keep polling Gitea, and sized up front so that a first apply with several replicas does not hang waiting on node provisioning. After that, raising runnerReplicas triggers a normal autoscaler scale-up, one node per new pod, taking a few minutes each. The first time you go above 1, watch it with make runners.

Spot preemption gives a pod about 15 seconds, so a job running at the time fails. Re-run it. On on-demand nodes, a drain (node upgrade or scale-down) lets running jobs finish for up to 55 minutes. Node upgrades normally happen in the Sunday 05:00–09:00 UTC window, alongside the VM's reboot window. GKE can still push urgent security patches and auto-repairs outside it, and those interrupt running jobs without warning, just as preemption does.

Runner 5.x is tested upstream against current Gitea, not 1.27. If registration or job fetching misbehaves after a runner upgrade, roll gitea:runnerImageTag back to the previous tag before debugging anything else.

Rotating the registration token

Registered runners authenticate with their own .runner credentials, so rotation never affects them; it only governs new registrations.

openssl rand -hex 32 | tr -d '\n' \
  | gcloud secrets versions add gitea-runner-registration-token --project <p> --data-file=-
make sync     # renders the new token file and restarts Gitea

Two Gitea behaviours make this a one-way street:

• Never re-add a value the secret has held before. When Gitea starts with a new preset token, it deactivates every older instance-level token, and it never reactivates one. Rotating A→B→A leaves A dead: Gitea only logs a warning and every new registration fails.
• "Reset registration token" in the admin UI also kills the preset token, permanently, across restarts. Use it only to revoke, then add a fresh secret version as above.

To revoke without replacing, click Reset in the admin UI. Removing the secret does not revoke anything, because Gitea keeps the token active in its database.

Turning it off

Two applies, in this order:

cd infra
pulumi config set gitea:runnerReplicas 0 && pulumi up     # 1. deletes the pods AND their PVC disks
pulumi config set gitea:runnersEnabled false && pulumi up # 2. deletes the cluster, NAT, subnet, IP

Step 1 matters because deleting a cluster does not delete PVC-backed disks. Skipping it leaves orphaned PDs that keep billing. To find them:

gcloud compute disks list --filter='name~^pvc- AND -users:*'

The Kubernetes objects carry DeletedWith(cluster), so step 2 does not try to delete them one by one through an API whose token expired long ago.

Then remove stale runners from Site Administration → Actions → Runners. The token file on the VM is left in place on purpose (see Rotating above). It is harmless.

Troubleshooting

The first apply fails creating the Namespace or the SecretProviderClass. A brand-new cluster can report RUNNING a little before its API server, or the Secret Manager add-on's CRD, is ready. Run pulumi up again; nothing is lost.

pulumi refresh or destroy fails with 401 against the cluster. The Kubernetes provider's token in state has expired. Add --run-program, which re-runs the program and fetches a fresh one. As a last resort for a cluster that is already gone, set PULUMI_K8S_DELETE_UNREACHABLE=true for one run.

Pod crashloops; the log mentions user namespaces. Rootless dockerd needs /proc/sys/user/max_user_namespaces > 0 on the node. COS_CONTAINERD on GKE 1.36 has it; check if a node image change ever breaks this:

kubectl --context=… debug node/<node> -it --image=busybox -- cat /proc/sys/user/max_user_namespaces

If it is 0, the rootless image cannot work on that node image. The fallback is the rootful docker:dind native-sidecar pattern from the upstream examples/kubernetes/statefulset-dind.yaml. A job escape then means root on the node: the node SA is minimal and the metadata block still applies, but update the threat model above to match.

docker build in a job fails to reach the daemon. Check whether the runner log says "jobs get the daemon socket directly". The runner probes whether the inner dockerd can see its temp directory, and falls back to mounting the daemon socket directly if it cannot. If neither path works, it is the same fallback as above.

actions/cache never restores. Jobs reach the runner's cache server at the pod IP on port 8088. If Dataplane V2 turns out to evaluate that traffic, the NetworkPolicy already allows it between runner pods. If it still fails, set cache.host explicitly in runnerConfig (infra/pkg/runner) from a downward-API POD_IP environment variable; the config expands ${NAME}.

The runner cannot reach Gitea at all (cannot ping the Gitea instance server: deadline_exceeded). Check DNS first:

kubectl --context=… -n gitea-runner exec gitea-runner-0 -- nslookup gitea.jasonmross.dev

A timeout here means the NetworkPolicy is not letting queries through. GKE enables NodeLocal DNSCache on new clusters, so the kube-dns Service IP is answered by the node-local-dns pod on the node. The DNS egress rule must allow k8s-app=node-local-dns as well as k8s-app=kube-dns. If DNS works, note that Cloud NAT does hairpin to the VM's external address (verified on the first deploy). Look at the VPC firewall and fail2ban on the VM next.

dockerd exits with chmod /home/rootless/.local/share/docker: operation not permitted. dockerd was given the PVC mount point itself as its data root. That directory is owned by host root, which is unmapped inside rootlesskit's user namespace. The data root must be a subdirectory that dockerd creates itself: data-root in daemon.json, dockerDataRoot in infra/pkg/runner.