JMR-dev/LibreMail-Bug-Report-Ingest
1
0
Fork 0
Code Issues 1 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
7e644be4f9de4e5c1611598f5039210a6774d8eb
LibreMail-Bug-Report-Ingest/infra/README.md
T
JMR-devandClaude Opus 4.8 985fb684c5 #4 GitHub Actions CD: deploy via workflow_dispatch 
Add a manual, workflow_dispatch-only CD workflow and close the infra
binding-wiring gap (#9) so the deployed Worker is actually functional.

Deliverable 1 - .github/workflows/deploy.yml:
- workflow_dispatch only, with a `stack` choice input (default prod).
- Gated to the `production` GitHub Actions environment and to the main
  branch (guard step fails otherwise); no push/PR trigger.
- Reuses ci.yml's Go 1.26 + TinyGo + pnpm setup and the TinyGo net/http
  patch, builds the Wasm Worker (pnpm run build), then runs pulumi up over
  infra/ via pulumi/actions, injecting the built ../build/worker.mjs as the
  workerScriptPath config. Provider/backend creds come from environment
  secrets (nothing committed). All actions pinned by commit SHA; actionlint
  clean.

Deliverable 2 - infra/deploy.go binding wiring (#9):
- WorkersScript now carries the R2 bucket binding (REPORTS_BUCKET), the four
  Secrets Store bindings (BUGREPORT_ENC_KEYRING, ADMIN_TOKEN, GITHUB_TOKEN,
  OTEL_EXPORTER_OTLP_HEADERS), and the plain vars (GITHUB_REPO, OTEL_*),
  matching wrangler.jsonc and the Worker runtime contract.
- New WorkersCronTrigger resource registers the two Friday UTC crons (#13),
  bound to the Worker.
- Real built artifact wired via ContentFile + computed ContentSha256 when
  the workerScriptPath config is set; documented placeholder otherwise
  (keeps the program testable without the artifact, as #2 did).
- secretsStoreId is a new required config; git rate-limit ruleset insertion
  point (#6/#7) kept reserved.
- Extended the WithMocks tests to assert the R2 + Secrets Store + var
  bindings, the crons, and the artifact ContentFile/ContentSha256 path.
- Updated Pulumi.<stack>.yaml and infra/README.md with the full
  secret/config list and how a maintainer triggers the deploy.

Verified in infra/: go build, go vet, go test all green; actionlint clean
on deploy.yml. No deploy/preview/provision was run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 17:40:08 -05:00

12 KiB
Raw Blame History

Infrastructure (Pulumi, Go)

Infrastructure-as-code for the LibreMail bug-report ingest pipeline. This is its own Go module (github.com/JMR-dev/LibreMail-Bug-Report-Ingest/infra) so the Worker module at the repo root stays lean.

It provisions, with the Pulumi Go SDK and the pulumi-cloudflare and pulumi-gcp providers:

Resource Type Purpose
ingest-worker cloudflare.WorkersScript The ingest Worker (libremail-bug-report-ingest, built in #1), wired with its full binding set (R2 + Secrets Store + vars, see Worker bindings).
reports-bucket cloudflare.R2Bucket Encrypted bug-report storage (libremail-bug-reports). Per ADR 0001 only ciphertext is written.
ingest-cron-triggers cloudflare.WorkersCronTrigger The two Friday UTC Cron Triggers (#13) that drive the weekly publish job; bound to the ingest Worker.
ingest-dns-record gcp.dns.RecordSet Google Cloud DNS record pointing the ingest hostname at the Worker's route/custom domain.

DNS authority is Google Cloud DNS. The managed zone is referenced by name (it already exists / is managed elsewhere), and this stack only adds a record to it.

Worker content: placeholder by default, real artifact at deploy. The deployed Worker is Go compiled by TinyGo to Wasm plus the syumai/workers ES-module shim, emitted by pnpm run build into ../build/ (git-ignored, produced by CI). Set the workerScriptPath config to the built main module (../build/worker.mjs) and the program uploads it via ContentFile + ContentSha256 (computed from the file). When workerScriptPath is unset, a documented placeholder module body is uploaded instead, so the resource is fully described and unit-testable without the artifact. The CD workflow (.github/workflows/deploy.yml) builds the Worker and injects workerScriptPath=../build/worker.mjs at deploy time, so a local pulumi preview (no build) still works off the placeholder.

Note: a TinyGo Worker is worker.mjs (main module) that imports app.wasm. ContentFile uploads the main module; if a functional deploy needs the wasm as a separate module part, add it alongside worker.mjs in the build output the pipeline points at. The bindings/crons below are provider-verified by the mock tests regardless of which content source is used.

Worker bindings

The Worker resource carries the bindings the runtime code reads, matching wrangler.jsonc and the Worker source (internal/storage.*Binding, worker/*_wasm.go). This is the wiring #4 added (the gap #9 flagged); the mock tests (deploy_test.go) assert every one of them:

Binding (JS var) Type Points at
REPORTS_BUCKET r2_bucket r2BucketName (libremail-bug-reports). Encrypted report objects (ADR 0001).
BUGREPORT_ENC_KEYRING secrets_store_secret secretsStoreId / secret bugreport-enc-keyring — AES-256 keyring (ADR 0001).
ADMIN_TOKEN secrets_store_secret secretsStoreId / secret bugreport-admin-token — admin API token (ADR 0003).
GITHUB_TOKEN secrets_store_secret secretsStoreId / secret github-token — publish PAT (#14).
OTEL_EXPORTER_OTLP_HEADERS secrets_store_secret secretsStoreId / secret otel-exporter-otlp-headers — OTLP auth headers (#17).
GITHUB_REPO plain_text githubRepo (JMR-dev/LibreMail) — publish target (#14).
OTEL_EXPORTER_OTLP_ENDPOINT plain_text otelExporterOtlpEndpoint (empty ⇒ telemetry disabled, #17).
OTEL_SERVICE_NAME plain_text otelServiceName (libremail-bug-report-ingest, #17).

The Cron Triggers resource registers both Friday UTC crons (0 22 * * 5 and 0 23 * * 5) from wrangler.jsonc (#13); the Worker's scheduled handler gates on the real America/Chicago rule so exactly one publishes each Friday.

The Secrets Store secret values themselves (keyring, admin token, GitHub token, OTLP headers) are not created by this stack — the maintainer stores them in Cloudflare Secrets Store under secretsStoreId. This stack only binds them to the Worker by name.

Prerequisites

  • Go (matching the repo toolchain) — go build ./... / go test ./... work without the Pulumi CLI.
  • The Pulumi CLI — required only to pulumi preview / pulumi up. It is not installed in CI yet, so a real pulumi preview is a follow-up (see "Deploying").

Configuration

Config is per-stack (Pulumi.<stack>.yaml). Program keys are namespaced by the project name, libremail-bug-report-ingest-infra. Provider keys use the cloudflare: / gcp: namespaces.

Program config (namespace libremail-bug-report-ingest-infra)

Key Required Default Description
cloudflareAccountId yes — Cloudflare account that owns the Worker + R2 bucket.
secretsStoreId yes — Cloudflare Secrets Store id holding the Worker's secrets (keyring, admin token, GitHub token, OTLP headers). Account-specific; not itself secret.
dnsManagedZone yes — Name of the existing Google Cloud DNS managed zone.
dnsRecordName yes — Ingest hostname as an FQDN with a trailing dot (e.g. bugreport.libremail.example.).
dnsRecordTarget yes — CNAME target = the Worker route/custom domain (FQDN, trailing dot).
workerName no libremail-bug-report-ingest Worker script name (matches wrangler.jsonc).
workerCompatibilityDate no 2025-06-01 Worker runtime compatibility date.
workerScriptPath no — Path to the built main module (../build/worker.mjs). When set, upload it via ContentFile + computed ContentSha256. Set by the CD workflow at deploy.
workerScriptContent no placeholder Inline Worker module body used when workerScriptPath is unset.
githubRepo no JMR-dev/LibreMail GITHUB_REPO var: owner/repo the weekly publish job files issues on (#14).
otelExporterOtlpEndpoint no "" OTEL_EXPORTER_OTLP_ENDPOINT var: OTLP base URL; empty disables telemetry (#17).
otelServiceName no libremail-bug-report-ingest OTEL_SERVICE_NAME var: reported service.name (#17).
r2BucketName no libremail-bug-reports R2 bucket name (also the REPORTS_BUCKET binding target).
r2BucketLocation no (provider default) R2 location hint: apac, eeur, enam, weur, wnam, oc.
dnsRecordType no CNAME DNS record type.
dnsTtlSeconds no 300 DNS record TTL (seconds).
gcpProject no (from gcp:project) Project the record belongs to, if different from the provider project.
cloudflareZoneId no — Reserved for #6/#7 (rate-limit ruleset) and Worker routes/custom domain. Unused today.

Set a non-secret value with, e.g.:

pulumi config set libremail-bug-report-ingest-infra:cloudflareAccountId <account-id>

Secrets and credentials — DO NOT COMMIT

Set these as encrypted Pulumi secrets (--secret) or via the provider's environment variables. Never place plaintext secrets in Pulumi.<stack>.yaml.

What How Notes
Cloudflare API token pulumi config set --secret cloudflare:apiToken <token> (or env CLOUDFLARE_API_TOKEN) Scope it to Workers Scripts + R2 admin for the account.
GCP project pulumi config set gcp:project <project-id> Not secret; identifies the GCP project.
GCP credentials pulumi config set --secret gcp:credentials "$(cat key.json)" (or env GOOGLE_CREDENTIALS / Application Default Credentials) Service account with Cloud DNS admin on the zone.

The bug-report encryption keyring (BUGREPORT_ENC_KEYRING, see ADR 0001) lives in Cloudflare Secrets Store, bound to the Worker — it is managed there, not committed here.

Deploying

Via GitHub Actions (recommended)

.github/workflows/deploy.yml is a manual, workflow_dispatch-only CD workflow. A maintainer runs it from the Actions tab, on the main branch (a guard step fails otherwise), and picks a stack (default prod). It reuses ci.yml's Go/TinyGo/pnpm setup + the TinyGo net/http patch, runs pnpm run build, then pulumi up --stack <stack> over this program (injecting the built ../build/worker.mjs as workerScriptPath). It is gated to the production GitHub Actions environment, so that environment's secrets and any required-reviewer / protection rules apply.

One-time setup before the first deploy — the maintainer configures:

  1. production environment secrets (repo Settings → Environments → production):

    Secret Purpose
    PULUMI_ACCESS_TOKEN Pulumi Cloud token (state backend). Self-managed backend? Use the action's cloud-url input + a PULUMI_CONFIG_PASSPHRASE secret instead.
    CLOUDFLARE_API_TOKEN Cloudflare token scoped to Workers Scripts + R2 (+ Cron Triggers) on the account.
    CLOUDFLARE_ACCOUNT_ID Cloudflare account id.
    GOOGLE_CREDENTIALS GCP service-account JSON with Cloud DNS admin on the managed zone.

  2. Stack config in Pulumi.<stack>.yaml — replace every REPLACE_ME_* (cloudflareAccountId, secretsStoreId, dnsManagedZone, dnsRecordName, dnsRecordTarget, gcp:project).

  3. Cloudflare Secrets Store — under secretsStoreId, store the four secret values the Worker binds: bugreport-enc-keyring, bugreport-admin-token, github-token, otel-exporter-otlp-headers.

  4. Pulumi stack — create it and select it once (pulumi stack init <stack>); the workflow runs with upsert: false and expects it to exist.

Trigger: Actions → CD → Run workflow → branch main, stack prod.

Via the Pulumi CLI (manual)

pulumi stack select prod           # or: pulumi stack init prod
# set the REPLACE_ME_* config values + provider secrets above, then build + deploy:
pnpm run build                     # produces ../build/worker.mjs (needed if workerScriptPath is set)
pulumi preview
pulumi up

No real deploy has been run from this repo yet: the mechanism (this program + the CD workflow) is compile-, test-, and actionlint-verified only. The maintainer supplies the credentials/config above and triggers the first deploy.

Testing (no Pulumi CLI required)

The program is exercised with the Pulumi Go SDK's mocking (pulumi.RunErr + pulumi.WithMocks), which registers resources against an in-memory monitor — no cloud calls, no CLI. The tests assert that the expected resources are registered with the expected inputs: Worker name/account, its full binding set (R2 REPORTS_BUCKET, the four Secrets Store secrets, the three plain vars) and the Cron Triggers, the ContentFile/ContentSha256 artifact path, R2 bucket name/location, and DNS type/name/target/ttl.

go vet ./...
go build ./...
go test ./...

Adding rate limiting later (#6 ADR / #7 impl)

The abuse/rate-limit ADR (#6) chose to implement ingest rate limiting as Cloudflare Rate Limiting rules via Pulumi (cloudflare.NewRuleset, phase http_ratelimit, scoped to a zone). That is out of scope for this ticket. The cloudflareZoneId config key and the forward note in deploy.go leave a clean insertion point; #7 adds the ruleset resource there.

Reference in New Issue View Git Blame Copy Permalink