JMR-devandClaude Opus 4.8 0d66969a97 #16 Test suite: end-to-end pipeline integration tests
Add internal/integration: a build-tag-free, host-only test package that
wires the Worker's REAL components together and exercises the full flow,
catching regressions in how the stages compose that per-ticket unit tests
miss. CI (#3) runs it automatically via `go test ./...` — no ci.yml change.

Scenarios (real http.Handler over loopback, real scrub->encrypt->store
Sink + host AES-256-GCM keyring, real lifecycle.Manager over a shared
MemoryStore, real publish.Publisher driving the real GitHub client against
a mock GitHub REST server, real schedule.Run gate):

- ingest->scrub->encrypt->store: POST /v1/reports with PII is 202; the
  object at rest is AES-256-GCM ciphertext leaking neither the PII nor the
  placeholder text, and decrypts to the fully scrubbed body.
- pending listed then removed via the authed admin API (#11); fail-closed
  401 without a token; removed report drops from pending.
- Friday-17:00-Central cron publishes one labeled issue per pending report
  with PII scrubbed from the issue body, marks each published, and a second
  run creates no new issues (cross-run de-dup); a later ingest publishes
  exactly once.
- a removed report is never published (admin removal x publish, #11 x #14).
- endpoint status-code contract on the assembled handler (ingest + admin).

Deterministic and fast: virtual clock + zero GitHub request spacing, no
real sleeps or network. No production code, wrangler.jsonc, or ci.yml
changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 17:00:42 -05:00

LibreMail Bug Report Ingest

Server-side infrastructure for LibreMail's debug bug-report pipeline. This repo is intentionally separate from the Android app repo — it owns the Cloudflare Worker and infrastructure-as-code, not the client.

What this is

Per JMR-dev/LibreMail#11:

  1. The LibreMail app lets a user opt in to submitting a debug report (LibreMail#33).
  2. A Cloudflare Worker in this repo receives the report over HTTPS, best-effort scrubs PII, and stores it encrypted in a Cloudflare R2 bucket (#34).
  3. Every Friday at 17:00 (Central Time, DST-aware), a scheduled job publishes any report not manually removed as a GitHub issue on the LibreMail repo (#35).

Stack

  • Worker: Go, compiled to WebAssembly with TinyGo and served through the syumai/workers runtime adapter
  • Infrastructure as code: Pulumi (Go)
  • Deployment: GitHub Actions
  • Secrets/key custody: Cloudflare Secret Manager
  • DNS: Google Cloud DNS

Build & run locally

The request-handling logic lives in internal/handler as plain, build-tag-free Go, so it is unit-tested and run locally with the standard Go toolchain — no TinyGo needed. Only the actual Wasm Worker build requires TinyGo.

Layout:

  • internal/handler/ — the core http.Handler (health/hello endpoints). No build tags; all request logic and its tests live here.
  • cmd/devserver/ — a plain net/http server that mounts the core handler for local dev without TinyGo.
  • worker/ — the Cloudflare Workers (Wasm) entrypoint, guarded by //go:build js && wasm, wiring the same core handler into the Workers runtime. Excluded from host builds and tests.

Test

go vet ./...
go test ./...

Run locally (no TinyGo)

go run ./cmd/devserver   # listens on :8787; override with ADDR, e.g. ADDR=:9000

Then, from another shell:

$ curl -s localhost:8787/
{"service":"libremail-bug-report-ingest","status":"ok","message":"hello from the LibreMail bug-report ingest Worker"}
$ curl -s localhost:8787/healthz
{"status":"ok"}

This runs the exact handler the deployed Worker uses, minus the Workers runtime.

Build & run the real Worker (requires TinyGo)

Node tooling is managed with pnpm; wrangler is a dev dependency. The Wasm build uses TinyGo 0.41.1 on the Go 1.26 toolchain.

Temporary toolchain patch. TinyGo 0.41.1 and earlier vendor a net/http js/wasm overlay (tinygo-org/net@e54965e) that fails to compile against Go 1.25+/1.26 with t.roundTrip undefined (see tinygo-org/tinygo#5467). CI applies the exact upstream fix (tinygo-org/net@1026408a, checked in as .ci/tinygo-net-roundtrip.patch) to the installed TinyGo before building. Building locally on Go 1.26 needs the same one-file patch until a TinyGo release later than 0.41.1 ships it, at which point the patch and the CI step are removed (tracked in #26).

pnpm install                 # install wrangler
pnpm run build               # workers-assets-gen + TinyGo -> ./build/app.wasm + ./build/worker.mjs
pnpm exec wrangler dev       # serve the Wasm Worker locally on :8787
pnpm exec wrangler deploy    # deploy (CI only)

pnpm run build runs, verbatim:

go run github.com/syumai/workers/cmd/workers-assets-gen && tinygo build -o ./build/app.wasm -target wasm -no-debug ./worker

TinyGo is not required for tests or the dev server; it is needed only for the Wasm build above and is installed in CI. The generated ./build/ output is git-ignored.

Why TinyGo + syumai/workers

Cloudflare Workers execute WebAssembly, not native binaries, so Go must be compiled to Wasm. Of the two options — the standard compiler's GOOS=js GOARCH=wasm output or TinyGo — TinyGo emits far smaller modules that sit comfortably inside the Worker size limit, which is why it is the standard path for Go on Workers. The syumai/workers package adapts Go's net/http handler model to the Workers fetch event, so a single http.Handler runs unchanged on the dev server and in the deployed Worker.

Status

Early bootstrap. See the project board and open issues for the current breakdown of work.

License

GNU AGPL v3.0.

S
Description
Server-side ingest pipeline (Cloudflare Worker + IaC) for LibreMail's opt-in bug-report feature. See JMR-dev/LibreMail#11.
Readme AGPL-3.0
467 KiB
Languages
Go 100%