dev_blog/docs/networking.md

Caddy on host networking + loopback-published app

This doc captures why the production deployment puts Caddy in the host network namespace and reaches the app over loopback, and exactly what that requires in the Quadlet units and the Caddyfile.

Note: this file lives under docs/ at the repo root. It is not part of the Astro site (src/pages/ + the blog content collection are what get built and served). Do not move it under src/.

Why

With Podman's default port-publish path (PublishPort=80:80 on a user-defined bridge), inbound packets are NAT'd by netavark before they hit the container. Caddy then sees the bridge gateway as the source address instead of the real client. That breaks two things we care about:

• fail2ban / nftables bans — the nftables-multiport action would populate the f2b set with the gateway's address, banning nothing useful.
• Coraza's audit log — every "blocked" entry would record the same internal address, so manual triage is impossible.

Switching Caddy to the host network namespace removes that NAT hop: Caddy binds 80/443 directly on the host's interfaces and sees the real client IP for both v4 and v6.

What changed

1. quadlet/dev-blog-caddy.container

Caddy now lives in the host network namespace.

-Network=dev-blog.network
-NetworkAlias=caddy
-
-PublishPort=80:80
-PublishPort=443:443
-PublishPort=443:443/udp
-PublishPort=[::]:80:80
-PublishPort=[::]:443:443
-PublishPort=[::]:443:443/udp
+Network=host

Network=host is mutually exclusive with both Network=dev-blog.network and any PublishPort= directives — Caddy binds 80/443 directly on the host's interfaces, so the user-defined network and port-forwarding are gone for this container. CAP_NET_BIND_SERVICE is still kept because the container's non-root user still has to bind privileged ports.

2. quadlet/dev-blog-app.container

App stays on the user-defined network and publishes on host loopback only.

-# Not exposed publicly – Caddy reverse-proxies in over the internal network.
-# PublishPort=4321:4321
+# Caddy now runs on the host network namespace, so it cannot reach the app
+# via podman DNS (`app:4321`). Publish the app port on the host's *loopback*
+# only -- it is reachable to Caddy on the host but not from outside.
+PublishPort=127.0.0.1:4321:4321
+PublishPort=[::1]:4321:4321

The app keeps Network=dev-blog.network (so it could still talk to future sidecars on that bridge), but now also surfaces on 127.0.0.1:4321 and [::1]:4321. nftables already accepts iif "lo" accept, so loopback traffic is unfiltered.

3. caddy/Caddyfile

Reverse-proxy target updated.

-reverse_proxy app:4321 {
+reverse_proxy 127.0.0.1:4321 {
   header_up X-Real-IP        {remote_host}
   header_up X-Forwarded-Proto {scheme}
 }

{remote_host} is now the real client IP (v4 or v6) because Caddy sees the connection unmodified, instead of the bridge gateway. That value flows into the JSON access log ("remote_ip") and Coraza's serial audit log, which is exactly what the fail2ban filter keys on — so the nftables-multiport action populates the right addresses in f2b-table and bans actually work.

Topology, before vs. after

Before

Internet ─┐
         (host nft) ─► podman PublishPort NAT ─► dev-blog (bridge, dual-stack)
                                                  ├─ caddy   (real IP lost)
                                                  └─ app     (Caddy → app:4321 via podman DNS)

After

Internet ─┐
         (host nft) ─► caddy in host netns (real client v4/v6 preserved)
                          │  reverse_proxy
                          ▼
                       127.0.0.1:4321 / [::1]:4321 (loopback publish)
                          │
                       dev-blog (bridge, dual-stack)
                          └─ app

Things that did not need to change

• dev-blog.network — still a dual-stack bridge; the app still uses it (and any future sidecar can join it).
• nftables.conf.j2 — already accepted tcp dport { 22, 80, 443 } and udp dport { 443 } in the inet filter table, plus iif "lo" accept, so both Caddy's public listeners and the loopback hop to the app are covered.
• fail2ban filter regexes, Coraza config, Ansible roles, Quadlet [Install] lines.

Operational heads-up

• compose.yaml (local-dev convenience) was deliberately left alone — it still wires Caddy through the bridge with PublishPort, which is fine for local iteration where preserving real client IPs doesn't matter.
• If you ever add another container that Caddy needs to reach, follow the same pattern: have that container publish to 127.0.0.1:<port> (and [::1]:<port> for v6) and reference it from the Caddyfile via loopback.
• Because the app's loopback publish is bound to 127.0.0.1 / [::1] only, it is not reachable from any other host on the network even if nftables were ever flushed — the kernel itself drops non-loopback traffic destined to those addresses.