main
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  JMR-devandClaude Opus 4.8
|
45ac7236ee |
#17 Observability: OpenTelemetry logging + tracing + alerting
Instrument the Worker with OpenTelemetry traces + structured logs over OTLP, plus alertable signals, via a minimal hand-rolled OTLP/HTTP exporter that fits the TinyGo/Wasm Worker build. New internal/telemetry package (build-tag-free, host-tested): - Span/log shim: Telemetry provider, Span (attrs/status/events/end), Log (Info/Warn/Error), trace/span-id correlation, W3C-style ids from crypto/rand. - Exporter seam: MemoryExporter (in-memory, for tests) and OTLPHTTPExporter (OTLP/HTTP JSON over net/http). No go.opentelemetry.io/otel/sdk dependency: the full OTEL-Go SDK + OTLP exporters pull in a large, reflection-heavy tree (protobuf, grpc) that bloats the Wasm binary and is unreliable under TinyGo. The shim uses only stdlib already proven under this project's js/wasm target (net/http per #26, encoding/json, crypto/rand). OTLP is the wire format, so any OTLP backend can ingest it. - Behaviour-preserving by construction: instrumentation is threaded through context. Instrumented code pulls an optional *Telemetry from ctx; absent (or nil exporter) => every method is a no-op. No public signatures change (NewHandler, handler.New, publish.New/Publish, schedule.Run are untouched), so parallel work built on the current APIs keeps compiling. Instrumentation: - ingest: an "ingest.request" server span + correlated log per request, classifying accepted / rejected / rate_limited / error. Observe-only (wraps the response writer to read the status); the HTTP contract is unchanged. A 5xx (e.g. 503 storage-unavailable) sets the span to Error and emits the alertable ingest.error signal; 4xx client rejections are INFO, not alerts. - publish: a "publish.run" span with per-report "publish.report" child spans and a log per report (published/failed). A failed report/run sets Error and emits alert.type=publish.run_failed. The per-run cap-hit (folding in the #14 follow-up) is now emitted as a structured, alertable OTEL signal (alert.type=publish.cap_hit + counts), not merely a log line. - schedule: a "schedule.run" span parenting the publish run; a list/publish failure emits alert.type=schedule.run_failed. Config (OTLP endpoint TBD, issue #17): - OTEL_EXPORTER_OTLP_ENDPOINT (plain var) - base OTLP/HTTP URL; empty => telemetry disabled (Worker behaves as before). /v1/traces and /v1/logs are appended. - OTEL_EXPORTER_OTLP_HEADERS (Secrets Store secret) - auth header(s), never committed. OTEL_SERVICE_NAME (plain var) - service.name override. - worker/telemetry_wasm.go builds the exporter lazily per run and injects the provider into the request/scheduled context; wrangler.jsonc gains only these OTEL keys. Alerting: run-failure, cap-hit, and elevated-ingest-error are emitted as span status=Error and structured log records carrying alert=true + a specific alert.type, so a backend alert rule can key on them once the OTLP endpoint is chosen. Tests: host unit tests with the in-memory exporter assert the ingest spans+logs for accepted/rejected/error, the publish run span + per-report spans + the cap-hit and run-failed signals, the schedule run span + list-error alert, and the OTLP/JSON encoding + HTTP round trip (httptest, no real backend). No-op default verified. go vet ./... and go test ./... green; GOOS=js GOARCH=wasm go build ./... compiles. Closes #17 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
JMR-devandClaude Opus 4.8
|
288f7e7fea |
#15 Mark reports processed after publishing
Wire the publisher's onPublished hook to lifecycle.MarkPublished so each confirmed-201 publish immediately transitions that report pending->published, completing cross-run de-duplication. - internal/publish: add the narrow Marker seam (write half of lifecycle.Manager) and WithMarkPublished(m) option. It sets onPublished to call m.MarkPublished on each confirmed create; on a mark failure it logs loudly (naming the report and the duplicate-next-run risk) and surfaces the error so the run is recorded failed. Mirrors the existing PendingGetter read-half seam, so publish stays host-testable and free of the Wasm-only storage backends. - worker/scheduled_wasm.go: buildPublish now passes WithMarkPublished(manager); the one Manager instance is both pending getter and marker. worker/main.go untouched. Partial-failure guarantee falls out of #14's seam: the hook runs only on a 201 and per-report failures are isolated, so successes leave the pending set and a failed report stays pending and is retried next run without duplicating the already-published ones. Tests (host, real lifecycle.Manager over MemoryStore + mock issue creator): all-succeed run marks all published and a second run creates no new issues; partial failure retries only the failed report next run without duplicating the rest; MarkPublished is idempotent on an already-published report; and the mark-failure edge case is surfaced, logged, and (honestly) re-publishes once. Closes #15 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
JMR-devandClaude Opus 4.8
|
f9ec7978b8 |
Add internal/publish: the real schedule.Publisher that turns pending
encrypted reports into labeled GitHub issues. - GitHub REST client on net/http (host-testable via httptest; works under TinyGo js/wasm per #26). Encodes ADR #6 §3.2: serial mutations spaced >=1s, honour Retry-After, wait until x-ratelimit-reset, >=60s floor for secondary-limit 403s, full-jitter exponential backoff (base 1s, cap 60s, <=5 attempts). Ensures the three ADR #6 labels (create-or-ignore). - Publisher: GetPending -> crypto.Open -> format -> CreateIssue per id, with the ADR #6 per-run cap (50) and 65,536-char body cap (truncate). Per-report failures are isolated and surfaced, never abort the batch. - onPublished(ctx, id) seam, called only after a confirmed 201, default no-op: #15 wires it to lifecycle.MarkPublished to complete cross-run de-dup. #14 does not implement the mark-published transition. - Issue body wraps report free-text in a length-adaptive code fence and metadata in inline code, neutralising Markdown/@mention injection. - Worker: swap schedule.LogPublisher for the real publisher in scheduled_wasm.go; read GITHUB_TOKEN (Secrets Store) + GITHUB_REPO (var); pre-gate so the sibling cron fire does no secret I/O. worker/main.go untouched. Export storage.GetSecret for the token read. - wrangler.jsonc: add GITHUB_TOKEN secret + GITHUB_REPO var (my keys only). Tests (host, httptest mock, virtual clock): N reports -> N labeled issues + onPublished per success; >65,536-char body truncated; transient 5xx and Retry-After retried per policy; persistent failure isolated (no onPublished); permission 403 not retried; per-run cap; decrypt failure isolated. go vet + go test ./... green; GOOS=js GOARCH=wasm build compiles. Closes #14 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
JMR-devandClaude Opus 4.8
|
047391d01c |
#11 Manual review/removal path for maintainers
Add an authenticated admin API to the ingest Worker so the single maintainer
can review the pending queue and pull a report before Friday's publish run.
Endpoints (on the existing handler):
GET /v1/admin/reports list pending report ids
POST /v1/admin/reports/{id}/remove mark a report removed
DELETE /v1/admin/reports/{id} remove alias
Remove calls lifecycle.MarkRemoved (#10), transitioning pending -> removed so
#13's ListPending excludes it from the next publish. Codes: 200 list/remove,
404 unknown id, 401 missing/bad/unset-secret token, 405 wrong method.
Auth: shared-secret Bearer token compared with crypto/subtle.ConstantTimeCompare,
fail-closed when the secret is unset. Injected via handler.New's new AdminBackend
arg: the dev server and tests wire a memory-backed lifecycle.Manager + ADMIN_TOKEN
env; the Worker reads ADMIN_TOKEN from Secrets Store and builds an R2-backed
Manager per request. Choice documented in docs/decisions/admin-auth.md.
Tests: Go httptest unit tests (list, remove+exclusion, 404, 401 incl. fail-closed,
405) and a Bruno api-tests flow (seed, authed list/remove, exclusion, no/bad
token 401). wrangler.jsonc gains only the ADMIN_TOKEN secret binding; worker
triggers untouched (owned by #13).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
JMR-devandClaude Opus 4.8
|
e650e5f161 |
#13 Cloudflare Cron Trigger: Friday 17:00 America/Chicago, DST-correct
Add a weekly Cron Trigger that fires at 17:00 America/Chicago (Central) every Friday year-round, correct across the CST/CDT DST transition, and on fire lists the pending reports (#10 Manager.ListPending) and hands their ids to the publish step. Cloudflare crons are UTC-only, and 17:00 Central is 22:00 UTC under CDT (summer) and 23:00 UTC under CST (winter), so no single UTC cron expresses it. Register BOTH Friday UTC hours in wrangler.jsonc (`0 22 * * 5` and `0 23 * * 5`) and gate each fire: only the fire that is actually 17:00 Central does the work, so publishing runs exactly once per Friday. TinyGo/Wasm may lack the IANA tz database, so the gate does not call time.LoadLocation. Instead internal/schedule computes the US Central DST rule from first principles (CDT from the 2nd Sunday of March 02:00 to the 1st Sunday of November 02:00, else CST) behind a pure func IsFriday1700Central(time.Time), host-testable without TinyGo and cross-checked against the real America/Chicago zone (via a test-only time/tzdata import) over a 20-year sweep. - internal/schedule: pure DST gate + Run orchestrator; Publisher/PendingLister seams; LogPublisher no-op default (the seam #14 replaces). - worker/scheduled_wasm.go: js/wasm-only adapter registering the scheduled task via init()+cron.ScheduleTaskNonBlock, wiring the R2-backed lifecycle Manager to schedule.Run. worker/main.go is untouched. - wrangler.jsonc: add triggers.crons (only the triggers section changed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
JMR-devandClaude Opus 4.8
|
bd7fe21d97 |
#9 Encrypted-at-rest R2 storage for scrubbed reports
Implement the storage path: for each accepted report, scrub PII (#8), encrypt with AES-256-GCM (ADR #5), and write only ciphertext to R2, wired in as the real ingest Sink replacing NopSink. - internal/crypto: AES-256-GCM in the exact ADR #5 wire format (magic "LMB1" || version || key_id BE16 || nonce(12) || ct || tag(16); the 7-byte header is the GCM AAD). Provider-independent framing shared by a host crypto/aes+crypto/cipher impl (tests, devserver) and a Wasm SubtleCrypto impl (syscall/js, //go:build js && wasm) per the TinyGo constraint; both produce byte-identical frames. Versioned keyring with key_id rotation; ParseKeyring reads the Secrets Store JSON secret. - internal/storage: ObjectStore interface with an in-memory fake (tests, devserver) and a Wasm R2Store (syumai/workers R2 binding). Sink ties scrub -> Seal -> Put under a unique reports/<ts>-<rand> key. WorkerSink loads the keyring from Secrets Store (BUGREPORT_ENC_KEYRING), cached for the isolate lifetime. - handler.New now takes an injectable ingest.Sink; the Worker uses the real R2/Secrets-Store sink, the devserver a memory + throwaway-key sink. - wrangler.jsonc: add REPORTS_BUCKET (R2) and BUGREPORT_ENC_KEYRING (Secrets Store) bindings. Tests (host, no TinyGo): encrypt/decrypt roundtrip; ciphertext != plaintext; wrong key + tamper (ct/tag/nonce/header-AAD) fail; exact wire layout plus a known-answer vector; key_id rotation with retained keys; full sink path (PII scrubbed then encrypted, readback requires the key and yields the scrubbed content). Existing ingest/handler behavior preserved (202 on valid POST). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Jason RossandClaude Opus 4.8
|
499bf7f655 |
Bootstrap Go module + Cloudflare Worker build tooling (#21)
Initialize the Go module and the Go -> Cloudflare Workers (TinyGo/Wasm) build path, structured so `go test` and a local dev server run on plain Go without TinyGo, while the real Wasm entrypoint is isolated behind build tags. - go.mod/go.sum: module github.com/JMR-dev/LibreMail-Bug-Report-Ingest (Go 1.26), requiring github.com/syumai/workers. - internal/handler: build-tag-free core http.Handler (GET / and GET /healthz, JSON responses, 404/405 handling) with net/http/httptest unit tests. - cmd/devserver: plain net/http server mounting the core handler for local dev without TinyGo (listens on :8787, override with ADDR). - worker/main.go: Cloudflare Workers (Wasm) entrypoint behind //go:build js && wasm, wiring the same handler via github.com/syumai/workers; excluded from host builds/tests. - package.json + pnpm-lock.yaml + pnpm-workspace.yaml: wrangler dev dependency managed with pnpm, with toolchain build scripts approved. - wrangler.jsonc: name=libremail-bug-report-ingest, main=./build/worker.mjs, build via `pnpm run build` (TinyGo). - README: "Build & run locally" section with exact commands and the rationale for the TinyGo + syumai/workers path. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |