Adds internal/integration/ — a build-tag-free, host-only test package that wires the Worker's real components together and exercises the full pipeline end to end, catching regressions in how the stages compose that the per-ticket unit suites (which test each stage in isolation) would miss. No production code, wrangler.jsonc, or ci.yml changes.
Because the Worker's production backends are Wasm-only (R2, SubtleCrypto), the tests run against the host seams the code already provides, mirroring the exact wiring in cmd/devserver and worker/buildPublish:
real http.Handler (handler.New: ingest + admin) over a real loopback httptest.Server, driven by a real net/http client;
real scrub → encrypt → store Sink (storage.NewSink) over a shared storage.MemoryStore, with a real host AES-256-GCM keyring (internal/crypto);
real lifecycle.Manager over that same store;
real internal/publish Publisher driving the real GitHub *Client against a mock GitHub REST API (an httptest server implementing POST /labels + POST /issues) — reusing #14's "point the client at an httptest server" seam;
real internal/schedule.Run Friday-17:00-Central gate.
Integration scenarios covered
TestIngestScrubsEncryptsAndStores — POST /v1/reports with PII → 202; the object at rest is AES-256-GCM ciphertext (LMB1 magic, active key id) that leaks neither the PII nor even the scrubbed placeholder text; decrypting it yields the fully scrubbed body (PII gone, placeholders present, non-sensitive stack trace preserved). End-to-end proof that ingest → scrub → encrypt → store works as one unit.
TestPendingListedThenRemovedViaAdmin — the ingested report is pending; unauthenticated admin list is 401 (fail-closed, WWW-Authenticate: Bearer); authed GET /v1/admin/reports lists exactly it; admin remove (#11) transitions it to removed and it drops from pending (and moves under the removed prefix).
TestCronPublishesPendingAndDedups — schedule.Run at Friday-17:00-Central lists pending, publishes one labeled issue per report via the mock GitHub with PII scrubbed out of the issue body, and marks each published; a second run creates no new issues (cross-run de-dup end-to-end); a later ingest publishes exactly once (de-dup is selective).
TestRemovedReportIsNotPublished — a report a maintainer removes before the run never becomes a GitHub issue, while pending reports do (admin removal #11 × publish #14).
TestEndpointContractsOverWiredHandler — ingest + admin status-code contract (202/400/415/405, admin 401/200/404/405) on the fully assembled handler over real HTTP.
Deterministic and fast (~1.7s): a virtual clock + zero GitHub request spacing means no real sleeps or network.
Unit-test gap analysis
Per the ticket, I only add genuinely missing coverage rather than duplicating. The existing per-ticket suites are already comprehensive: scrub (#8) covers every redaction category (redact + over-redaction "keep" + exact-output + integration + idempotency + immutability + nil/empty); lifecycle (#10) covers list-exactness, both transitions, idempotent retry, interrupted-state convergence, and unknown-id errors; de-dup (#14/#15) covers transition + cross-run de-dup + partial-failure retry + idempotency + the mark-failure edge case. I found no genuine unit-level gaps, so the added value is the end-to-end composition above (which also re-exercises scrub, lifecycle, and de-dup together).
.github/workflows/ci.yml already runs go vet ./... and go test ./... on every PR to main, so these tests run automatically on every PR — no ci.yml change needed (confirmed).
Test evidence
go vet ./... # clean
go build ./... # exit 0 (host)
GOOS=js GOARCH=wasm go build ./... # exit 0 (Wasm Worker still compiles)
go test ./... # all packages ok, incl. internal/integration
go test -count=5 ./internal/integration/ # deterministic across 5 runs
go 1.26.2. The test-only package is handled correctly by ./... on both host and Wasm targets (it only errors if built by explicit name, which nothing does).
Notes
No new HTTP endpoint → no new Bruno needed; existing api-tests/ already cover the wire contract. Not extended (optional).
Constructed all components via their current public APIs, so there is no file conflict with the parallel OTEL ticket (#17) modifying ingest/publish/schedule production files.
## Summary
Adds `internal/integration/` — a build-tag-free, **host-only** test package that wires the Worker's **real** components together and exercises the **full pipeline end to end**, catching regressions in how the stages compose that the per-ticket unit suites (which test each stage in isolation) would miss. No production code, `wrangler.jsonc`, or `ci.yml` changes.
Because the Worker's production backends are Wasm-only (R2, SubtleCrypto), the tests run against the **host seams the code already provides**, mirroring the exact wiring in `cmd/devserver` and `worker/buildPublish`:
- real `http.Handler` (`handler.New`: ingest + admin) over a real loopback `httptest.Server`, driven by a real `net/http` client;
- real `scrub → encrypt → store` Sink (`storage.NewSink`) over a shared `storage.MemoryStore`, with a real host AES-256-GCM keyring (`internal/crypto`);
- real `lifecycle.Manager` over that same store;
- real `internal/publish` Publisher driving the **real GitHub `*Client`** against a **mock GitHub REST API** (an `httptest` server implementing `POST /labels` + `POST /issues`) — reusing #14's "point the client at an httptest server" seam;
- real `internal/schedule.Run` Friday-17:00-Central gate.
## Integration scenarios covered
1. **`TestIngestScrubsEncryptsAndStores`** — `POST /v1/reports` with PII → `202`; the object **at rest** is AES-256-GCM ciphertext (`LMB1` magic, active key id) that leaks **neither the PII nor even the scrubbed placeholder text**; decrypting it yields the fully scrubbed body (PII gone, placeholders present, non-sensitive stack trace preserved). End-to-end proof that ingest → scrub → encrypt → store works as one unit.
2. **`TestPendingListedThenRemovedViaAdmin`** — the ingested report is `pending`; unauthenticated admin list is `401` (fail-closed, `WWW-Authenticate: Bearer`); authed `GET /v1/admin/reports` lists exactly it; admin remove (#11) transitions it to `removed` and it drops from pending (and moves under the `removed` prefix).
3. **`TestCronPublishesPendingAndDedups`** — `schedule.Run` at Friday-17:00-Central lists pending, publishes **one labeled issue per report** via the mock GitHub with **PII scrubbed out of the issue body**, and marks each `published`; a **second run creates no new issues** (cross-run de-dup end-to-end); a later ingest publishes **exactly once** (de-dup is selective).
4. **`TestRemovedReportIsNotPublished`** — a report a maintainer removes before the run **never becomes a GitHub issue**, while pending reports do (admin removal #11 × publish #14).
5. **`TestEndpointContractsOverWiredHandler`** — ingest + admin status-code contract (`202/400/415/405`, admin `401/200/404/405`) on the fully assembled handler over real HTTP.
Deterministic and fast (~1.7s): a virtual clock + zero GitHub request spacing means no real sleeps or network.
## Unit-test gap analysis
Per the ticket, I only add genuinely missing coverage rather than duplicating. The existing per-ticket suites are already comprehensive: scrub (#8) covers every redaction category (redact + over-redaction "keep" + exact-output + integration + idempotency + immutability + nil/empty); lifecycle (#10) covers list-exactness, both transitions, idempotent retry, interrupted-state convergence, and unknown-id errors; de-dup (#14/#15) covers transition + cross-run de-dup + partial-failure retry + idempotency + the mark-failure edge case. I found **no genuine unit-level gaps**, so the added value is the end-to-end composition above (which also re-exercises scrub, lifecycle, and de-dup together).
## CI (#3)
`.github/workflows/ci.yml` already runs `go vet ./...` and `go test ./...` on every PR to `main`, so these tests run automatically on every PR — **no `ci.yml` change needed** (confirmed).
## Test evidence
```
go vet ./... # clean
go build ./... # exit 0 (host)
GOOS=js GOARCH=wasm go build ./... # exit 0 (Wasm Worker still compiles)
go test ./... # all packages ok, incl. internal/integration
go test -count=5 ./internal/integration/ # deterministic across 5 runs
```
`go` 1.26.2. The test-only package is handled correctly by `./...` on both host and Wasm targets (it only errors if built by explicit name, which nothing does).
## Notes
- No new HTTP endpoint → no new Bruno needed; existing `api-tests/` already cover the wire contract. Not extended (optional).
- Constructed all components via their **current public APIs**, so there is no file conflict with the parallel OTEL ticket (#17) modifying ingest/publish/schedule production files.
Closes #16
⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.
Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
following these best practices for managing and storing secrets including API keys and other credentials
🦉GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
#### ⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.
Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.
<details>
<summary>🔎 Detected hardcoded secret in your pull request</summary>
<br>
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
| -------------- | ------------------ | ------------------------------ | ---------------- | --------------- | -------------------- |
| [34489199](https://dashboard.gitguardian.com/workspace/616578/incidents/34489199?occurrence=277261755) | Triggered | JSON Web Token | 0d66969a972a70dd9251989d2b1694562542e2bf | internal/integration/pipeline_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/0d66969a972a70dd9251989d2b1694562542e2bf#diff-2bcebc37e70dab10e94af849eda1a221e2a15ca5caec4007eae0a10b1bc12a5aR65) |
</details>
<details>
<summary>🛠 Guidelines to remediate hardcoded secrets</summary>
<br>
1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) the best practices.
3. Revoke and [rotate this secret](https://docs.gitguardian.com/secrets-detection/secrets-detection-engine/detectors/generics/json_web_token#revoke-the-secret?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment).
4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials
- install [secret detection on pre-commit](https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation.
</details>
---
<sup>🦉 [GitGuardian](https://dashboard.gitguardian.com/auth/login/?utm_medium=checkruns&utm_source=github&utm_campaign=cr1) detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.<br/></sup>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Adds
internal/integration/— a build-tag-free, host-only test package that wires the Worker's real components together and exercises the full pipeline end to end, catching regressions in how the stages compose that the per-ticket unit suites (which test each stage in isolation) would miss. No production code,wrangler.jsonc, orci.ymlchanges.Because the Worker's production backends are Wasm-only (R2, SubtleCrypto), the tests run against the host seams the code already provides, mirroring the exact wiring in
cmd/devserverandworker/buildPublish:http.Handler(handler.New: ingest + admin) over a real loopbackhttptest.Server, driven by a realnet/httpclient;scrub → encrypt → storeSink (storage.NewSink) over a sharedstorage.MemoryStore, with a real host AES-256-GCM keyring (internal/crypto);lifecycle.Managerover that same store;internal/publishPublisher driving the real GitHub*Clientagainst a mock GitHub REST API (anhttptestserver implementingPOST /labels+POST /issues) — reusing #14's "point the client at an httptest server" seam;internal/schedule.RunFriday-17:00-Central gate.Integration scenarios covered
TestIngestScrubsEncryptsAndStores—POST /v1/reportswith PII →202; the object at rest is AES-256-GCM ciphertext (LMB1magic, active key id) that leaks neither the PII nor even the scrubbed placeholder text; decrypting it yields the fully scrubbed body (PII gone, placeholders present, non-sensitive stack trace preserved). End-to-end proof that ingest → scrub → encrypt → store works as one unit.TestPendingListedThenRemovedViaAdmin— the ingested report ispending; unauthenticated admin list is401(fail-closed,WWW-Authenticate: Bearer); authedGET /v1/admin/reportslists exactly it; admin remove (#11) transitions it toremovedand it drops from pending (and moves under theremovedprefix).TestCronPublishesPendingAndDedups—schedule.Runat Friday-17:00-Central lists pending, publishes one labeled issue per report via the mock GitHub with PII scrubbed out of the issue body, and marks eachpublished; a second run creates no new issues (cross-run de-dup end-to-end); a later ingest publishes exactly once (de-dup is selective).TestRemovedReportIsNotPublished— a report a maintainer removes before the run never becomes a GitHub issue, while pending reports do (admin removal #11 × publish #14).TestEndpointContractsOverWiredHandler— ingest + admin status-code contract (202/400/415/405, admin401/200/404/405) on the fully assembled handler over real HTTP.Deterministic and fast (~1.7s): a virtual clock + zero GitHub request spacing means no real sleeps or network.
Unit-test gap analysis
Per the ticket, I only add genuinely missing coverage rather than duplicating. The existing per-ticket suites are already comprehensive: scrub (#8) covers every redaction category (redact + over-redaction "keep" + exact-output + integration + idempotency + immutability + nil/empty); lifecycle (#10) covers list-exactness, both transitions, idempotent retry, interrupted-state convergence, and unknown-id errors; de-dup (#14/#15) covers transition + cross-run de-dup + partial-failure retry + idempotency + the mark-failure edge case. I found no genuine unit-level gaps, so the added value is the end-to-end composition above (which also re-exercises scrub, lifecycle, and de-dup together).
CI (#3)
.github/workflows/ci.ymlalready runsgo vet ./...andgo test ./...on every PR tomain, so these tests run automatically on every PR — noci.ymlchange needed (confirmed).Test evidence
go1.26.2. The test-only package is handled correctly by./...on both host and Wasm targets (it only errors if built by explicit name, which nothing does).Notes
api-tests/already cover the wire contract. Not extended (optional).Closes #16
⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.
Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.
🔎 Detected hardcoded secret in your pull request
0d66969a97🛠 Guidelines to remediate hardcoded secrets
To avoid such incidents in the future consider
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.