#7 Ingest HTTPS endpoint: accept report POST, size limit #29

Merged
JMR-dev merged 4 commits from ticket-7-ingest-endpoint into main 2026-07-02 19:30:42 +00:00
JMR-dev commented 2026-07-02 18:53:57 +00:00 (Migrated from github.com)

What

Implements the ingest HTTPS endpoint POST /v1/reports (GitHub issue #7). New package internal/ingest/ holds the endpoint logic and is wired into the core, build-tag-free http.Handler (internal/handler/handler.go), so the exact same route serves on cmd/devserver and on the Cloudflare Worker (worker/, //go:build js && wasm).

Storage is decoupled behind a tiny Sink interface so PII scrubbing (#8) and encrypted R2 storage (#9) can slot in later without touching the HTTP contract.

Route

POST /v1/reports — accepts a user-initiated, opt-in debug bug-report as a JSON body.

v1 report schema (the contract LibreMail#33 targets)

JSON object. Required: appVersion, platform, report. Everything else is optional. Validation is deliberately loose (reject only clearly-invalid payloads); unknown fields are ignored, not rejected, so newer app versions can add fields without breaking ingest.

Field Type Required Notes
appVersion string yes App version, e.g. "1.4.2 (142)". Non-empty, ≤ 256 chars.
platform string yes e.g. "android". Non-empty, ≤ 64 chars.
report string yes Free-text report: user description, logs, stack traces. Non-empty.
osVersion string no e.g. "Android 14". ≤ 128 chars.
device string no e.g. "Pixel 7". ≤ 256 chars.
clientTimestamp string no Client capture time. If present, must be RFC 3339.

Example:

{
  "appVersion": "1.4.2 (142)",
  "platform": "android",
  "osVersion": "Android 14",
  "device": "Pixel 7",
  "clientTimestamp": "2026-07-02T12:34:56Z",
  "report": "NullPointerException in SyncService\n at line 42\n<attached logs>"
}

Response contract (per ADR #6 §2.4)

Status When Notes
202 Accepted Valid JSON within the size cap, stored via the Sink Body {"status":"accepted"}
400 Bad Request Malformed JSON or failed schema validation Generic {"error":"..."}; never echoes request content
413 Payload Too Large Body exceeds 256 KiB (262,144 bytes) Content-Length fast path and a http.MaxBytesReader hard cap on the stream, so a missing/chunked/lying Content-Length cannot bypass it
415 Unsupported Media Type Content-Type is not application/json (params like ; charset=utf-8 are tolerated)
405 Method Not Allowed Any method other than POST Sends Allow: POST
503 Service Unavailable The storage Sink returns an error "storage unavailable" row of the ADR

Rate limiting is out of scope for the Worker (by design)

Per ADR #6 §2.3, 429 (rate limit) and volumetric 503 shedding are enforced at the Cloudflare edge via Pulumi-provisioned Rate Limiting rules — ticket #2 — before the Worker ever runs. They are intentionally not implemented in this PR. The Worker owns only what an edge rule cannot express: the size cap and schema validation.

Storage decoupling (for #9)

type Sink interface {
    Store(ctx context.Context, raw []byte) error
}

The endpoint calls Store once with the raw, validated body after it decides to accept. Provided stubs: NopSink (default wiring — discards, so the full HTTP contract is exercisable today) and MemorySink (tests). No scrubbing/encryption/R2 here — that is #8/#9.

Tests — both kinds

1. Go unit tests (net/http/httptest)

Cover every response code, including 413 via both Content-Length and an oversized streamed body (forced ContentLength = -1), plus exact-limit boundary, storage-failure 503, nil-sink default, and a "never echoes request content" check.

$ go vet ./... && go test ./...
?   	github.com/JMR-dev/LibreMail-Bug-Report-Ingest/cmd/devserver	[no test files]
ok  	github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/handler	0.944s
ok  	github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/ingest	0.902s

Verbose internal/ingest run:

=== RUN   TestAccepted202
--- PASS: TestAccepted202 (0.00s)
=== RUN   TestAcceptedContentTypeWithCharset
--- PASS: TestAcceptedContentTypeWithCharset (0.00s)
=== RUN   TestAcceptedAtExactLimit
--- PASS: TestAcceptedAtExactLimit (0.00s)
=== RUN   TestOversizedViaContentLength413
--- PASS: TestOversizedViaContentLength413 (0.00s)
=== RUN   TestOversizedViaStreamedBody413
--- PASS: TestOversizedViaStreamedBody413 (0.00s)
=== RUN   TestWrongContentType415
--- PASS: TestWrongContentType415 (0.00s)
=== RUN   TestMalformedJSON400  (5 subtests: truncated/not-json/empty/trailing/array)
--- PASS: TestMalformedJSON400 (0.00s)
=== RUN   TestSchemaValidation400  (6 subtests incl. bad RFC3339 clientTimestamp)
--- PASS: TestSchemaValidation400 (0.00s)
=== RUN   TestWrongMethod405
--- PASS: TestWrongMethod405 (0.00s)
=== RUN   TestStorageFailure503
--- PASS: TestStorageFailure503 (0.00s)
=== RUN   TestErrorBodiesDoNotEchoRequest
--- PASS: TestErrorBodiesDoNotEchoRequest (0.00s)
=== RUN   TestNilSinkDefaultsToNop
--- PASS: TestNilSinkDefaultsToNop (0.00s)
PASS
ok  	github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/ingest

2. Bruno API tests — OpenCollection YAML format (api-tests/)

@usebruno/cli@3.5.0 added as a pnpm devDependency. The collection is authored in OpenCollection YAML (not .bru): the presence of api-tests/opencollection.yml selects the CLI's yml format, and each request is a *.yml file with info / http / runtime blocks (assertions via runtime.scripts type: tests). Requests assert the full contract: 202 valid, 413 oversized (payload generated at run time in a before-request script so no 256 KiB fixture is committed), 415 wrong type, 400 malformed, 405 wrong method + Allow: POST.

OpenCollection YAML status: WORKS. @usebruno/cli@3.5.0 executes the OpenCollection YAML collection natively — no fallback to .bru was needed.

Run through the pnpm wrapper against the local dev server (go run ./cmd/devserver on :8787, serving the same handler) — both pnpm exec bru run --env local (from api-tests/) and pnpm run test:api (from repo root) pass, exit 0:

$ pnpm exec bru run --env local
valid-report (202 Accepted) - 10 ms
Tests
   ✓ valid JSON report within the size limit returns 202
   ✓ 202 body reports accepted status
oversized-report (413 Request Entity Too Large) - 2 ms
Tests
   ✓ body larger than 256 KiB returns 413
wrong-content-type (415 Unsupported Media Type) - 1 ms
Tests
   ✓ Content-Type other than application/json returns 415
malformed-json (400 Bad Request) - 1 ms
Tests
   ✓ malformed JSON returns 400
   ✓ 400 body carries a generic error string
wrong-method (405 Method Not Allowed) - 1 ms
Tests
   ✓ a non-POST method returns 405
   ✓ 405 advertises the allowed method via the Allow header

📊 Execution Summary
┌───────────────┬──────────────┐
│ Status        │    ✓ PASS    │
│ Requests      │ 5 (5 Passed) │
│ Tests         │     8/8      │
└───────────────┴──────────────┘

pnpm build-script approval (pnpm-workspace.yaml)

@usebruno/cli pulls in protobufjs, whose postinstall build script pnpm 11 leaves un-approved by default. Left as-is, pnpm install exits non-zero (ERR_PNPM_IGNORED_BUILDS), which also aborts pnpm exec / pnpm run (their verify-deps-before-run precheck runs pnpm install) and would break CI's pnpm install once this lands on main. Fixed here by adding protobufjs to the existing allowBuilds allowlist in pnpm-workspace.yaml (same mechanism already used for esbuild/sharp/workerd). pnpm install now exits 0 and the pnpm-wrapped Bruno run above is green.

CI note: the ci "Build Wasm Worker" step may still be red until the #26 toolchain fix (PR #25) lands on main — that is unrelated to this change.

Files

  • internal/ingest/{ingest,report,sink}.go + ingest_test.go
  • internal/handler/handler.go — route registration only
  • api-tests/** — Bruno OpenCollection YAML collection
  • package.json (@usebruno/cli devDep + test:api script) + pnpm-lock.yaml
  • pnpm-workspace.yaml — approve the protobufjs build script (keeps pnpm install green)

Out of scope and untouched: scrubbing/encryption/R2 (#8/#9), edge rate limiting / Pulumi (#2), README, workflows, infra/.

Closes #7

## What Implements the ingest HTTPS endpoint **`POST /v1/reports`** (GitHub issue #7). New package `internal/ingest/` holds the endpoint logic and is wired into the core, build-tag-free `http.Handler` (`internal/handler/handler.go`), so the exact same route serves on `cmd/devserver` and on the Cloudflare Worker (`worker/`, `//go:build js && wasm`). Storage is decoupled behind a tiny `Sink` interface so PII scrubbing (#8) and encrypted R2 storage (#9) can slot in later without touching the HTTP contract. ## Route `POST /v1/reports` — accepts a user-initiated, opt-in debug bug-report as a JSON body. ## v1 report schema (the contract LibreMail#33 targets) JSON object. **Required:** `appVersion`, `platform`, `report`. Everything else is optional. Validation is deliberately *loose* (reject only clearly-invalid payloads); **unknown fields are ignored**, not rejected, so newer app versions can add fields without breaking ingest. | Field | Type | Required | Notes | | --- | --- | --- | --- | | `appVersion` | string | yes | App version, e.g. `"1.4.2 (142)"`. Non-empty, ≤ 256 chars. | | `platform` | string | yes | e.g. `"android"`. Non-empty, ≤ 64 chars. | | `report` | string | yes | Free-text report: user description, logs, stack traces. Non-empty. | | `osVersion` | string | no | e.g. `"Android 14"`. ≤ 128 chars. | | `device` | string | no | e.g. `"Pixel 7"`. ≤ 256 chars. | | `clientTimestamp` | string | no | Client capture time. If present, must be **RFC 3339**. | Example: ```json { "appVersion": "1.4.2 (142)", "platform": "android", "osVersion": "Android 14", "device": "Pixel 7", "clientTimestamp": "2026-07-02T12:34:56Z", "report": "NullPointerException in SyncService\n at line 42\n<attached logs>" } ``` ## Response contract (per ADR #6 §2.4) | Status | When | Notes | | --- | --- | --- | | **202 Accepted** | Valid JSON within the size cap, stored via the Sink | Body `{"status":"accepted"}` | | **400 Bad Request** | Malformed JSON or failed schema validation | Generic `{"error":"..."}`; never echoes request content | | **413 Payload Too Large** | Body exceeds **256 KiB** (262,144 bytes) | `Content-Length` fast path **and** a `http.MaxBytesReader` hard cap on the stream, so a missing/chunked/lying `Content-Length` cannot bypass it | | **415 Unsupported Media Type** | `Content-Type` is not `application/json` (params like `; charset=utf-8` are tolerated) | | | **405 Method Not Allowed** | Any method other than POST | Sends `Allow: POST` | | **503 Service Unavailable** | The storage `Sink` returns an error | "storage unavailable" row of the ADR | ### Rate limiting is out of scope for the Worker (by design) Per ADR #6 §2.3, **429 (rate limit) and volumetric 503 shedding are enforced at the Cloudflare edge via Pulumi-provisioned Rate Limiting rules — ticket #2 — before the Worker ever runs.** They are intentionally **not** implemented in this PR. The Worker owns only what an edge rule cannot express: the size cap and schema validation. ## Storage decoupling (for #9) ```go type Sink interface { Store(ctx context.Context, raw []byte) error } ``` The endpoint calls `Store` once with the raw, validated body after it decides to accept. Provided stubs: `NopSink` (default wiring — discards, so the full HTTP contract is exercisable today) and `MemorySink` (tests). No scrubbing/encryption/R2 here — that is #8/#9. ## Tests — both kinds ### 1. Go unit tests (`net/http/httptest`) Cover every response code, including 413 via **both** `Content-Length` and an oversized **streamed** body (forced `ContentLength = -1`), plus exact-limit boundary, storage-failure 503, nil-sink default, and a "never echoes request content" check. ``` $ go vet ./... && go test ./... ? github.com/JMR-dev/LibreMail-Bug-Report-Ingest/cmd/devserver [no test files] ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/handler 0.944s ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/ingest 0.902s ``` Verbose `internal/ingest` run: ``` === RUN TestAccepted202 --- PASS: TestAccepted202 (0.00s) === RUN TestAcceptedContentTypeWithCharset --- PASS: TestAcceptedContentTypeWithCharset (0.00s) === RUN TestAcceptedAtExactLimit --- PASS: TestAcceptedAtExactLimit (0.00s) === RUN TestOversizedViaContentLength413 --- PASS: TestOversizedViaContentLength413 (0.00s) === RUN TestOversizedViaStreamedBody413 --- PASS: TestOversizedViaStreamedBody413 (0.00s) === RUN TestWrongContentType415 --- PASS: TestWrongContentType415 (0.00s) === RUN TestMalformedJSON400 (5 subtests: truncated/not-json/empty/trailing/array) --- PASS: TestMalformedJSON400 (0.00s) === RUN TestSchemaValidation400 (6 subtests incl. bad RFC3339 clientTimestamp) --- PASS: TestSchemaValidation400 (0.00s) === RUN TestWrongMethod405 --- PASS: TestWrongMethod405 (0.00s) === RUN TestStorageFailure503 --- PASS: TestStorageFailure503 (0.00s) === RUN TestErrorBodiesDoNotEchoRequest --- PASS: TestErrorBodiesDoNotEchoRequest (0.00s) === RUN TestNilSinkDefaultsToNop --- PASS: TestNilSinkDefaultsToNop (0.00s) PASS ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/ingest ``` ### 2. Bruno API tests — OpenCollection YAML format (`api-tests/`) `@usebruno/cli@3.5.0` added as a pnpm devDependency. The collection is authored in **OpenCollection YAML** (not `.bru`): the presence of `api-tests/opencollection.yml` selects the CLI's `yml` format, and each request is a `*.yml` file with `info` / `http` / `runtime` blocks (assertions via `runtime.scripts` `type: tests`). Requests assert the full contract: 202 valid, 413 oversized (payload generated at run time in a `before-request` script so no 256 KiB fixture is committed), 415 wrong type, 400 malformed, 405 wrong method + `Allow: POST`. **OpenCollection YAML status: WORKS.** `@usebruno/cli@3.5.0` executes the OpenCollection YAML collection natively — no fallback to `.bru` was needed. Run through the **pnpm wrapper** against the local dev server (`go run ./cmd/devserver` on `:8787`, serving the same handler) — both `pnpm exec bru run --env local` (from `api-tests/`) and `pnpm run test:api` (from repo root) pass, exit 0: ``` $ pnpm exec bru run --env local valid-report (202 Accepted) - 10 ms Tests ✓ valid JSON report within the size limit returns 202 ✓ 202 body reports accepted status oversized-report (413 Request Entity Too Large) - 2 ms Tests ✓ body larger than 256 KiB returns 413 wrong-content-type (415 Unsupported Media Type) - 1 ms Tests ✓ Content-Type other than application/json returns 415 malformed-json (400 Bad Request) - 1 ms Tests ✓ malformed JSON returns 400 ✓ 400 body carries a generic error string wrong-method (405 Method Not Allowed) - 1 ms Tests ✓ a non-POST method returns 405 ✓ 405 advertises the allowed method via the Allow header 📊 Execution Summary ┌───────────────┬──────────────┐ │ Status │ ✓ PASS │ │ Requests │ 5 (5 Passed) │ │ Tests │ 8/8 │ └───────────────┴──────────────┘ ``` #### pnpm build-script approval (`pnpm-workspace.yaml`) `@usebruno/cli` pulls in `protobufjs`, whose postinstall build script pnpm 11 leaves un-approved by default. Left as-is, `pnpm install` exits non-zero (`ERR_PNPM_IGNORED_BUILDS`), which also aborts `pnpm exec` / `pnpm run` (their `verify-deps-before-run` precheck runs `pnpm install`) and would break CI's `pnpm install` once this lands on main. **Fixed here** by adding `protobufjs` to the existing `allowBuilds` allowlist in `pnpm-workspace.yaml` (same mechanism already used for `esbuild`/`sharp`/`workerd`). `pnpm install` now exits 0 and the pnpm-wrapped Bruno run above is green. > CI note: the `ci` "Build Wasm Worker" step may still be red until the #26 toolchain fix (PR #25) lands on main — that is unrelated to this change. ## Files - `internal/ingest/{ingest,report,sink}.go` + `ingest_test.go` - `internal/handler/handler.go` — route registration only - `api-tests/**` — Bruno OpenCollection YAML collection - `package.json` (`@usebruno/cli` devDep + `test:api` script) + `pnpm-lock.yaml` - `pnpm-workspace.yaml` — approve the `protobufjs` build script (keeps `pnpm install` green) Out of scope and untouched: scrubbing/encryption/R2 (#8/#9), edge rate limiting / Pulumi (#2), README, workflows, `infra/`. Closes #7
gitguardian[bot] commented 2026-07-02 19:29:07 +00:00 (Migrated from github.com)

⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
34486411 Triggered JSON Web Token cd59c79521 internal/scrub/scrub_test.go View secret
34486413 Triggered Bearer Token cd59c79521 internal/scrub/scrub_test.go View secret
34486414 Triggered GitHub Personal Access Token cd59c79521 internal/scrub/scrub_test.go View secret
34486412 Triggered Google API Key cd59c79521 internal/scrub/scrub_test.go View secret
34486414 Triggered GitHub Personal Access Token cd59c79521 internal/scrub/scrub_test.go View secret
34486412 Triggered Google API Key cd59c79521 internal/scrub/scrub_test.go View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

#### ⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request. Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components. <details> <summary>🔎 Detected hardcoded secrets in your pull request</summary> <br> | GitGuardian id | GitGuardian status | Secret | Commit | Filename | | | -------------- | ------------------ | ------------------------------ | ---------------- | --------------- | -------------------- | | [34486411](https://dashboard.gitguardian.com/workspace/616578/incidents/34486411?occurrence=277220194) | Triggered | JSON Web Token | cd59c7952120b1b10d06784c149a93b75777df38 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/cd59c7952120b1b10d06784c149a93b75777df38#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R78) | | [34486413](https://dashboard.gitguardian.com/workspace/616578/incidents/34486413?occurrence=277220195) | Triggered | Bearer Token | cd59c7952120b1b10d06784c149a93b75777df38 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/cd59c7952120b1b10d06784c149a93b75777df38#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R92) | | [34486414](https://dashboard.gitguardian.com/workspace/616578/incidents/34486414?occurrence=277220196) | Triggered | GitHub Personal Access Token | cd59c7952120b1b10d06784c149a93b75777df38 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/cd59c7952120b1b10d06784c149a93b75777df38#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R100) | | [34486412](https://dashboard.gitguardian.com/workspace/616578/incidents/34486412?occurrence=277220197) | Triggered | Google API Key | cd59c7952120b1b10d06784c149a93b75777df38 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/cd59c7952120b1b10d06784c149a93b75777df38#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R106) | | [34486414](https://dashboard.gitguardian.com/workspace/616578/incidents/34486414?occurrence=277220198) | Triggered | GitHub Personal Access Token | cd59c7952120b1b10d06784c149a93b75777df38 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/cd59c7952120b1b10d06784c149a93b75777df38#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R100) | | [34486412](https://dashboard.gitguardian.com/workspace/616578/incidents/34486412?occurrence=277220199) | Triggered | Google API Key | cd59c7952120b1b10d06784c149a93b75777df38 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/cd59c7952120b1b10d06784c149a93b75777df38#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R106) | </details> <details> <summary>🛠 Guidelines to remediate hardcoded secrets</summary> <br> 1. Understand the implications of revoking this secret by investigating where it is used in your code. 2. Replace and store your secrets safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&amp;utm_medium=GitHub_checks&amp;utm_campaign=check_run_comment) the best practices. 3. Revoke and [rotate these secrets](https://docs.gitguardian.com/secrets-detection/secrets-detection-engine/detectors/generics/json_web_token#revoke-the-secret?utm_source=product&amp;utm_medium=GitHub_checks&amp;utm_campaign=check_run_comment). 4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&amp;utm_medium=GitHub_checks&amp;utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data. To avoid such incidents in the future consider - following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&amp;utm_medium=GitHub_checks&amp;utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials - install [secret detection on pre-commit](https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&amp;utm_medium=GitHub_checks&amp;utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation. </details> --- <sup>🦉 [GitGuardian](https://dashboard.gitguardian.com/auth/login/?utm_medium=checkruns&amp;utm_source=github&amp;utm_campaign=cr1) detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.<br/></sup>
Sign in to join this conversation.