Adds package internal/scrub: a best-effort, schema-agnostic PII
redaction pass that runs over raw bug-report payloads before they are
persisted (storage is #9). Because the payload schema is not finalised (#7),
it operates on raw text/bytes via regex + heuristics, so it works regardless
of whether the payload is JSON, form-encoded, a log excerpt, or free text.
Matches are masked, not deleted — each is replaced with a bracketed
placeholder (e.g. [REDACTED_EMAIL]) so the surrounding payload structure is
preserved for triage. The pass is non-mutating and idempotent, and is build-tag-free so it compiles for the host toolchain and the Wasm Worker
target alike.
This is explicitly best-effort, not a guarantee. The package docs state
in several places that it WILL miss data and MAY over-redact, and that a
scrubbed payload must not be treated as certified PII-free. This aligns with
the forthcoming privacy doc (#12). The root README is intentionally untouched
to keep this PR disjoint from the parallel CI (#3) / Pulumi (#2) PRs.
API
funcScrub(payload[]byte)[]byte// primary entry point for the storage path (#9)funcScrubString(sstring)string// string-typed equivalent// Composable per-category passes (a caller can pick only what it wants):funcRedactEmails(sstring)stringfuncRedactTokens(sstring)stringfuncRedactIPs(sstring)stringfuncRedactNames(sstring)string// Exported placeholders so downstream code / tests need no string literals:constPlaceholderEmail/PlaceholderToken/PlaceholderAuth/PlaceholderIP/PlaceholderName
ScrubString applies the categories in a fixed order (tokens → emails → IPs →
names) chosen so the rules don't clobber each other's output.
Redaction categories & their tests
Each category is tested with positive cases and negative / over-redaction
guard cases (things that must NOT be redacted). 89 passing checks total.
Preserves: "bearer of bad news", the sk-loading-spinner CSS class, short AWS-like strings, www.example.com / dotted Java package names, the word "secret" in prose, semver.
IP addresses —
IPv4 with per-octet 0–255 validation; comprehensive IPv6 (full, compressed ::, loopback ::1, IPv4-mapped ::ffff:…), alternation ordered for correct unanchored extraction.
Preserves: clock times 12:34:56, MAC addresses, semver, out-of-range octets (999.1.1.1, 256.…), key:value.
Names (best-effort, deliberately weak) —
Key-directed only: masks the value after obvious keys (name, first_name, last_name, full_name, display_name, username, user, nickname), \b-anchored.
Documented limitation covered by a test: a human name in free-form prose ("my name is Robert…") is not detected.
Plus integration, exact-output (structure-preservation), idempotency, bytes-API,
nil/empty, and immutability tests.
go test ./...
? github.com/JMR-dev/LibreMail-Bug-Report-Ingest/cmd/devserver [no test files]
ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/handler 0.821s
ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/scrub 0.586s
Also verified: gofmt clean, go vet ./... clean, and the package builds for GOOS=js/GOARCH=wasm and GOOS=wasip1/GOARCH=wasm (Wasm Worker target).
Notes / limitations worth capturing for the privacy doc (#12)
Name redaction is the weakest link: key-directed only. It misses names in
prose and over-redacts non-personal values under name-like keys (e.g. name: my-service in a k8s manifest). Do not rely on it.
A genuine 4-part version like 1.2.3.4 and dotted-decimal identifiers (e.g.
SNMP OIDs) are indistinguishable from IPv4 by regex and will be masked.
Non-standard JWTs that do not start with eyJ are only caught if they match
another token rule.
Cookie / Set-Cookie values are out of scope here; could be added later.
## Summary
Adds package `internal/scrub`: a **best-effort**, schema-agnostic PII
redaction pass that runs over raw bug-report payloads **before** they are
persisted (storage is #9). Because the payload schema is not finalised (#7),
it operates on raw text/bytes via regex + heuristics, so it works regardless
of whether the payload is JSON, form-encoded, a log excerpt, or free text.
Matches are **masked, not deleted** — each is replaced with a bracketed
placeholder (e.g. `[REDACTED_EMAIL]`) so the surrounding payload structure is
preserved for triage. The pass is **non-mutating** and **idempotent**, and is
**build-tag-free** so it compiles for the host toolchain and the Wasm Worker
target alike.
> This is explicitly **best-effort, not a guarantee**. The package docs state
> in several places that it WILL miss data and MAY over-redact, and that a
> scrubbed payload must not be treated as certified PII-free. This aligns with
> the forthcoming privacy doc (#12). The root README is intentionally untouched
> to keep this PR disjoint from the parallel CI (#3) / Pulumi (#2) PRs.
## API
```go
func Scrub(payload []byte) []byte // primary entry point for the storage path (#9)
func ScrubString(s string) string // string-typed equivalent
// Composable per-category passes (a caller can pick only what it wants):
func RedactEmails(s string) string
func RedactTokens(s string) string
func RedactIPs(s string) string
func RedactNames(s string) string
// Exported placeholders so downstream code / tests need no string literals:
const PlaceholderEmail/PlaceholderToken/PlaceholderAuth/PlaceholderIP/PlaceholderName
```
`ScrubString` applies the categories in a fixed order (tokens → emails → IPs →
names) chosen so the rules don't clobber each other's output.
## Redaction categories & their tests
Each category is tested with **positive** cases and **negative / over-redaction
guard** cases (things that must NOT be redacted). 89 passing checks total.
- **Email addresses** — robust local@domain.tld regex.
- Redacts: plain, `+tag`, sub-domains, multi-label TLDs, uppercase, in JSON, in `<...>`.
- Preserves: `@handle` mentions, `@channel`, `meet @ 3pm`, `@Override`, `user@localhost` (no TLD), `5@each`.
- **Auth tokens / secrets** —
- `Authorization:` / `Proxy-Authorization:` header values (header form and JSON form, Bearer/Basic/…).
- Standalone `Bearer <token>` (length-gated so the prose word "bearer" is safe).
- JWTs, anchored on the `eyJ` header prefix (near-zero false positives).
- Well-known provider key formats: GitHub, GitHub fine-grained PAT, GitLab, Slack, Stripe, OpenAI, Google, AWS access-key IDs.
- Values under secret-named keys: `password`, `api_key`, `token`, `access_token`, `client_secret`, `private_key`, `X-Auth-Token`, …
- Preserves: "bearer of bad news", the `sk-loading-spinner` CSS class, short AWS-like strings, `www.example.com` / dotted Java package names, the word "secret" in prose, semver.
- **IP addresses** —
- IPv4 with per-octet 0–255 validation; comprehensive IPv6 (full, compressed `::`, loopback `::1`, IPv4-mapped `::ffff:…`), alternation ordered for correct unanchored extraction.
- Preserves: clock times `12:34:56`, MAC addresses, semver, out-of-range octets (`999.1.1.1`, `256.…`), `key:value`.
- **Names (best-effort, deliberately weak)** —
- Key-directed only: masks the value after obvious keys (`name`, `first_name`, `last_name`, `full_name`, `display_name`, `username`, `user`, `nickname`), `\b`-anchored.
- Preserves: suffix collisions (`filename:`, `hostname:`, `codename:`, `pathname:`), `user-agent:`, `user_id:`.
- Documented limitation covered by a test: a human name in free-form prose ("my name is Robert…") is **not** detected.
Plus integration, exact-output (structure-preservation), idempotency, bytes-API,
nil/empty, and immutability tests.
## `go test ./...`
```
? github.com/JMR-dev/LibreMail-Bug-Report-Ingest/cmd/devserver [no test files]
ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/handler 0.821s
ok github.com/JMR-dev/LibreMail-Bug-Report-Ingest/internal/scrub 0.586s
```
Also verified: `gofmt` clean, `go vet ./...` clean, and the package builds for
`GOOS=js/GOARCH=wasm` and `GOOS=wasip1/GOARCH=wasm` (Wasm Worker target).
## Notes / limitations worth capturing for the privacy doc (#12)
- Name redaction is the weakest link: key-directed only. It **misses** names in
prose and **over-redacts** non-personal values under name-like keys (e.g.
`name: my-service` in a k8s manifest). Do not rely on it.
- A genuine 4-part version like `1.2.3.4` and dotted-decimal identifiers (e.g.
SNMP OIDs) are indistinguishable from IPv4 by regex and will be masked.
- Non-standard JWTs that do not start with `eyJ` are only caught if they match
another token rule.
- Cookie / `Set-Cookie` values are out of scope here; could be added later.
Closes #8
⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request.
Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
following these best practices for managing and storing secrets including API keys and other credentials
🦉GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
#### ⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request.
Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.
<details>
<summary>🔎 Detected hardcoded secrets in your pull request</summary>
<br>
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
| -------------- | ------------------ | ------------------------------ | ---------------- | --------------- | -------------------- |
| [34486412](https://dashboard.gitguardian.com/workspace/616578/incidents/34486412?occurrence=277197859) | Triggered | Google API Key | 6508679d8657aa73127b83549f3915eb5269d832 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/6508679d8657aa73127b83549f3915eb5269d832#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R106) |
| [34486414](https://dashboard.gitguardian.com/workspace/616578/incidents/34486414?occurrence=277197860) | Triggered | GitHub Personal Access Token | 6508679d8657aa73127b83549f3915eb5269d832 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/6508679d8657aa73127b83549f3915eb5269d832#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R100) |
| [34486411](https://dashboard.gitguardian.com/workspace/616578/incidents/34486411?occurrence=277197861) | Triggered | JSON Web Token | 6508679d8657aa73127b83549f3915eb5269d832 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/6508679d8657aa73127b83549f3915eb5269d832#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R78) |
| [34486413](https://dashboard.gitguardian.com/workspace/616578/incidents/34486413?occurrence=277197862) | Triggered | Bearer Token | 6508679d8657aa73127b83549f3915eb5269d832 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/6508679d8657aa73127b83549f3915eb5269d832#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R92) |
| [34486412](https://dashboard.gitguardian.com/workspace/616578/incidents/34486412?occurrence=277197863) | Triggered | Google API Key | 6508679d8657aa73127b83549f3915eb5269d832 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/6508679d8657aa73127b83549f3915eb5269d832#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R106) |
| [34486414](https://dashboard.gitguardian.com/workspace/616578/incidents/34486414?occurrence=277197864) | Triggered | GitHub Personal Access Token | 6508679d8657aa73127b83549f3915eb5269d832 | internal/scrub/scrub_test.go | [View secret](https://github.com/JMR-dev/LibreMail-Bug-Report-Ingest/commit/6508679d8657aa73127b83549f3915eb5269d832#diff-2932eff712531fa44ddcd1da4509da244b7f5941341d6725be74ceb5a8a6ca26R100) |
</details>
<details>
<summary>🛠 Guidelines to remediate hardcoded secrets</summary>
<br>
1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) the best practices.
3. Revoke and [rotate these secrets](https://docs.gitguardian.com/secrets-detection/secrets-detection-engine/detectors/specifics/googleaiza#revoke-the-secret?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment).
4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials
- install [secret detection on pre-commit](https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation.
</details>
---
<sup>🦉 [GitGuardian](https://dashboard.gitguardian.com/auth/login/?utm_medium=checkruns&utm_source=github&utm_campaign=cr1) detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.<br/></sup>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Adds package
internal/scrub: a best-effort, schema-agnostic PIIredaction pass that runs over raw bug-report payloads before they are
persisted (storage is #9). Because the payload schema is not finalised (#7),
it operates on raw text/bytes via regex + heuristics, so it works regardless
of whether the payload is JSON, form-encoded, a log excerpt, or free text.
Matches are masked, not deleted — each is replaced with a bracketed
placeholder (e.g.
[REDACTED_EMAIL]) so the surrounding payload structure ispreserved for triage. The pass is non-mutating and idempotent, and is
build-tag-free so it compiles for the host toolchain and the Wasm Worker
target alike.
API
ScrubStringapplies the categories in a fixed order (tokens → emails → IPs →names) chosen so the rules don't clobber each other's output.
Redaction categories & their tests
Each category is tested with positive cases and negative / over-redaction
guard cases (things that must NOT be redacted). 89 passing checks total.
+tag, sub-domains, multi-label TLDs, uppercase, in JSON, in<...>.@handlementions,@channel,meet @ 3pm,@Override,user@localhost(no TLD),5@each.Authorization:/Proxy-Authorization:header values (header form and JSON form, Bearer/Basic/…).Bearer <token>(length-gated so the prose word "bearer" is safe).eyJheader prefix (near-zero false positives).password,api_key,token,access_token,client_secret,private_key,X-Auth-Token, …sk-loading-spinnerCSS class, short AWS-like strings,www.example.com/ dotted Java package names, the word "secret" in prose, semver.::, loopback::1, IPv4-mapped::ffff:…), alternation ordered for correct unanchored extraction.12:34:56, MAC addresses, semver, out-of-range octets (999.1.1.1,256.…),key:value.name,first_name,last_name,full_name,display_name,username,user,nickname),\b-anchored.filename:,hostname:,codename:,pathname:),user-agent:,user_id:.Plus integration, exact-output (structure-preservation), idempotency, bytes-API,
nil/empty, and immutability tests.
go test ./...Also verified:
gofmtclean,go vet ./...clean, and the package builds forGOOS=js/GOARCH=wasmandGOOS=wasip1/GOARCH=wasm(Wasm Worker target).Notes / limitations worth capturing for the privacy doc (#12)
prose and over-redacts non-personal values under name-like keys (e.g.
name: my-servicein a k8s manifest). Do not rely on it.1.2.3.4and dotted-decimal identifiers (e.g.SNMP OIDs) are indistinguishable from IPv4 by regex and will be masked.
eyJare only caught if they matchanother token rule.
Set-Cookievalues are out of scope here; could be added later.Closes #8
⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request.
Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.
🔎 Detected hardcoded secrets in your pull request
6508679d866508679d866508679d866508679d866508679d866508679d86🛠 Guidelines to remediate hardcoded secrets
To avoid such incidents in the future consider
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.