#9 Encrypted-at-rest R2 storage for scrubbed reports #34

Merged
JMR-dev merged 1 commits from ticket-9-encrypted-r2-storage into main 2026-07-02 20:10:43 +00:00
JMR-dev commented 2026-07-02 19:58:40 +00:00 (Migrated from github.com)

Closes #9.

Implements the storage path behind #7's POST /v1/reports: for each accepted report scrub (#8) → encrypt (ADR #5) → put into R2, wired in as the real ingest Sink replacing NopSink. No new HTTP endpoint.

Crypto (matches ADR #5)

New internal/crypto package. AES-256-GCM authenticated encryption producing the exact ADR wire format:

magic "LMB1" (4) || format_version 0x01 (1) || key_id uint16 BE (2) || nonce (12) || ciphertext (N) || auth_tag (16)
  • The 7-byte header (magic || version || key_id) is passed to GCM as AAD (authenticated, stored in the clear) so key_id / format can't be flipped, downgraded, or transplanted.
  • 12-byte random nonce per object from crypto/rand; 128-bit tag.
  • Versioned keyring keyed by key_id; key_id-based rotation (old objects keep decrypting while their version is retained). ParseKeyring reads the Secrets Store JSON secret {active, keys{ver: base64-32B}}.

Host-testable without TinyGo (mirrors the existing build-tag split): the frame/keyring logic carries no build tags; only the raw AEAD differs by provider:

  • gcm_host.go (//go:build !(js && wasm)) — Go crypto/aes + crypto/cipher. Backs go test + cmd/devserver.
  • gcm_wasm.go (//go:build js && wasm) — Workers SubtleCrypto via syscall/js (crypto.subtle.encrypt/decrypt, {name:"AES-GCM", iv, additionalData, tagLength:128}), per the ADR's TinyGo note.

AES-256-GCM is deterministic for a given key+nonce+plaintext+AAD, so both providers emit byte-identical frames — the wire format is the contract. A known-answer vector + a "matches stdlib GCM" test lock this; the host impl is fully unit-tested, and CI compiles the Wasm impl.

Storage + R2 wiring

New internal/storage package:

  • ObjectStore interface Put(ctx, key, data) / Get(ctx, key); in-memory fake (tests + devserver) and a Wasm R2Store over the syumai/workers R2 binding (//go:build js && wasm).
  • Sink (ingest.Sink) ties scrub → crypto.Seal(active key) → Put under an unguessable key reports/<utc-ts>-<80-bit-rand>. The store only ever sees ciphertext.
  • WorkerSink (Wasm) loads the keyring from Secrets Store inside the request (await env.BUGREPORT_ENC_KEYRING.get()), cached for the isolate lifetime, then delegates to Sink.
  • handler.New now takes an injectable ingest.Sink: the Worker uses WorkerSink; cmd/devserver uses memory store + a throwaway per-run key; a nil sink still defaults to NopSink.

Bindings added (wrangler.jsonc)

  • r2_buckets: REPORTS_BUCKET → libremail-bug-reports (matches infra defaultR2BucketName).
  • secrets_store_secrets: BUGREPORT_ENC_KEYRING → bugreport-enc-keyring (ADR name). store_id is a documented <store-id> placeholder filled at deploy; not needed for pnpm run build or the devserver, so CI is unaffected.

workers-assets-gen + tinygo build do not read wrangler.jsonc, so the CI Wasm build is unaffected by the placeholder.

Test evidence

go vet ./... — clean.

$ go test ./...
?   github.com/JMR-dev/LibreMail-Bug-Report-Ingest/cmd/devserver	[no test files]
ok  github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/crypto	0.585s
ok  github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/handler	0.806s
ok  github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/ingest	0.881s
ok  github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/scrub	0.514s
ok  github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/storage	0.760s

Wasm entrypoint compiles (CI builds it with the TinyGo patch):

$ GOOS=js GOARCH=wasm go build ./...   # clean

Coverage highlights:

  • crypto: roundtrip; ciphertext ≠ plaintext; wrong key fails; tamper (ciphertext / tag / nonce / header-AAD / relabeled key_id) fails; exact wire layout; known-answer vector; key_id rotation with retained keys; keyring parse/validation.
  • storage: MemoryStore roundtrip; full sink path — raw report with PII → scrubbed (PII gone, verified against #8) → encrypted → stored; readback requires the correct key (wrong key fails) and yields the scrubbed (not raw) content; Put-error propagation (drives 503); distinct keys.
  • ingest/handler: existing behavior preserved (202 on valid POST); added a test that handler.New(sink) routes /v1/reports to the injected sink.

Also verified go run ./cmd/devserver end-to-end (matching the api-tests/ Bruno assertions): valid → 202, malformed → 400, wrong content-type → 415, wrong method → 405 (+Allow: POST), oversized → 413.

Notes / out of scope

  • The SubtleCrypto Wasm path compiles under GOOS=js GOARCH=wasm and CI's TinyGo build, but is not executed in these host tests (it needs a live Workers runtime). Its output is guaranteed byte-identical to the host impl by the AES-256-GCM standard, which the known-answer / stdlib-parity tests pin.
  • Bindings are declared in wrangler.jsonc per the ticket. If the deploy path ends up uploading the Worker via Pulumi (infra/) rather than wrangler deploy, the same two bindings would also need wiring on the WorkersScript resource — left for the deploy ticket.

🤖 Generated with Claude Code

Closes #9. Implements the storage path behind #7's `POST /v1/reports`: for each accepted report **scrub (#8) → encrypt (ADR #5) → put into R2**, wired in as the real ingest `Sink` replacing `NopSink`. No new HTTP endpoint. ## Crypto (matches ADR #5) New `internal/crypto` package. AES-256-GCM authenticated encryption producing the exact ADR wire format: ``` magic "LMB1" (4) || format_version 0x01 (1) || key_id uint16 BE (2) || nonce (12) || ciphertext (N) || auth_tag (16) ``` - The 7-byte header (`magic || version || key_id`) is passed to GCM as **AAD** (authenticated, stored in the clear) so key_id / format can't be flipped, downgraded, or transplanted. - 12-byte random nonce per object from `crypto/rand`; 128-bit tag. - **Versioned keyring** keyed by `key_id`; `key_id`-based rotation (old objects keep decrypting while their version is retained). `ParseKeyring` reads the Secrets Store JSON secret `{active, keys{ver: base64-32B}}`. **Host-testable without TinyGo (mirrors the existing build-tag split):** the frame/keyring logic carries no build tags; only the raw AEAD differs by provider: - `gcm_host.go` (`//go:build !(js && wasm)`) — Go `crypto/aes` + `crypto/cipher`. Backs `go test` + `cmd/devserver`. - `gcm_wasm.go` (`//go:build js && wasm`) — Workers **SubtleCrypto** via `syscall/js` (`crypto.subtle.encrypt/decrypt`, `{name:"AES-GCM", iv, additionalData, tagLength:128}`), per the ADR's TinyGo note. AES-256-GCM is deterministic for a given key+nonce+plaintext+AAD, so both providers emit **byte-identical frames** — the wire format is the contract. A known-answer vector + a "matches stdlib GCM" test lock this; the host impl is fully unit-tested, and CI compiles the Wasm impl. ## Storage + R2 wiring New `internal/storage` package: - `ObjectStore` interface `Put(ctx, key, data)` / `Get(ctx, key)`; in-memory fake (tests + devserver) and a Wasm `R2Store` over the `syumai/workers` R2 binding (`//go:build js && wasm`). - `Sink` (`ingest.Sink`) ties **scrub → `crypto.Seal(active key)` → `Put`** under an unguessable key `reports/<utc-ts>-<80-bit-rand>`. The store only ever sees ciphertext. - `WorkerSink` (Wasm) loads the keyring from Secrets Store inside the request (`await env.BUGREPORT_ENC_KEYRING.get()`), cached for the isolate lifetime, then delegates to `Sink`. - `handler.New` now takes an injectable `ingest.Sink`: the Worker uses `WorkerSink`; `cmd/devserver` uses memory store + a throwaway per-run key; a nil sink still defaults to `NopSink`. ## Bindings added (`wrangler.jsonc`) - `r2_buckets`: `REPORTS_BUCKET` → `libremail-bug-reports` (matches infra `defaultR2BucketName`). - `secrets_store_secrets`: `BUGREPORT_ENC_KEYRING` → `bugreport-enc-keyring` (ADR name). `store_id` is a documented `<store-id>` placeholder filled at deploy; not needed for `pnpm run build` or the devserver, so CI is unaffected. `workers-assets-gen` + `tinygo build` do not read `wrangler.jsonc`, so the CI Wasm build is unaffected by the placeholder. ## Test evidence `go vet ./...` — clean. ``` $ go test ./... ? github.com/JMR-dev/LibreMail-Bug-Report-Ingest/cmd/devserver [no test files] ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/crypto 0.585s ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/handler 0.806s ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/ingest 0.881s ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/scrub 0.514s ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/storage 0.760s ``` Wasm entrypoint compiles (CI builds it with the TinyGo patch): ``` $ GOOS=js GOARCH=wasm go build ./... # clean ``` Coverage highlights: - **crypto:** roundtrip; ciphertext ≠ plaintext; wrong key fails; tamper (ciphertext / tag / nonce / header-AAD / relabeled key_id) fails; exact wire layout; known-answer vector; `key_id` rotation with retained keys; keyring parse/validation. - **storage:** MemoryStore roundtrip; **full sink path** — raw report with PII → scrubbed (PII gone, verified against #8) → encrypted → stored; readback requires the correct key (wrong key fails) and yields the **scrubbed** (not raw) content; Put-error propagation (drives 503); distinct keys. - **ingest/handler:** existing behavior preserved (202 on valid POST); added a test that `handler.New(sink)` routes `/v1/reports` to the injected sink. Also verified `go run ./cmd/devserver` end-to-end (matching the `api-tests/` Bruno assertions): valid → 202, malformed → 400, wrong content-type → 415, wrong method → 405 (+`Allow: POST`), oversized → 413. ## Notes / out of scope - The **SubtleCrypto Wasm path** compiles under `GOOS=js GOARCH=wasm` and CI's TinyGo build, but is not executed in these host tests (it needs a live Workers runtime). Its output is guaranteed byte-identical to the host impl by the AES-256-GCM standard, which the known-answer / stdlib-parity tests pin. - Bindings are declared in `wrangler.jsonc` per the ticket. If the deploy path ends up uploading the Worker via Pulumi (`infra/`) rather than `wrangler deploy`, the same two bindings would also need wiring on the `WorkersScript` resource — left for the deploy ticket. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitguardian[bot] commented 2026-07-02 19:58:43 +00:00 (Migrated from github.com)

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

#### ️✅ There are no secrets present in this pull request anymore. If these secrets were true positive and are still valid, we highly recommend you to revoke them. While these secrets were previously flagged, we no longer have a reference to the specific commits where they were detected. Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately. Find [here](https://docs.gitguardian.com/platform/remediate/remediate-incidents) more information about risks. --- <sup><sub>🦉 [GitGuardian](https://dashboard.gitguardian.com/auth/login/?utm_medium=checkruns&amp;utm_source=github&amp;utm_campaign=cr1) detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.<br/></sup></sub>
Sign in to join this conversation.