Author SHA1 Message Date
Jason Ross 8597a0d4d4 Merge branch 'main' into spike-359-sqlcipher-ondevice 2026-07-07 16:03:10 -05:00
Jason Ross 6be3b41dd1 Merge pull request #424 from JMR-dev/feat-369-encrypt-reports-at-rest
feat(security): encrypt persisted reports at rest when cache encryption is on (#369)
2026-07-07 16:02:03 -05:00
JMR-dev 099474d32a feat(security): encrypt persisted reports at rest when cache encryption is on (#369)
When the opt-in cache encryption (encryptCache) is ON, persist crash and
"Report a problem" reports encrypted at rest, decrypting them on read; when
OFF they stay plaintext exactly as before.

- ReportStore gains a ReportEncryption collaborator (default None = plaintext,
  so existing call sites are unchanged). On write it seals the storage JSON with
  AES-256-GCM and tags it with a marker prefix; on read it sniffs the prefix, so
  pre-toggle plaintext and post-toggle sealed reports coexist. Writes FAIL
  CLOSED: a sealing failure drops the report rather than leaving plaintext on
  disk. Decrypt failures are logged (PII-free) and skipped.
- KeystoreReportEncryption reuses the vetted KeystoreCrypto (non-auth master
  key, so a crash while the app is locked can still seal), and mirrors the
  encryptCache setting into a crash-safe in-memory flag warmed at startup (no
  DataStore read on the crashing thread).
- PII-free AppLog logging at the enable/disable transition and both fallback
  paths; never logs report contents.

Tests: JVM unit tests for the ReportStore branching (seal-on-write, plaintext
when off, crash persistence, fail-closed, mixed files, decrypt-failure skip,
markSurfaced re-seal) and for KeystoreReportEncryption; an instrumented test
proves real Keystore ciphertext on disk + round-trip on device.
2026-07-07 14:55:28 -05:00
JMR-dev c04825a637 spike(security): characterize #359 SQLCipher SDK-37/16KB-page failure on-device
Add SqlCipherOpenSpikeTest (androidTest), a focused on-device harness for #359
that exercises the encrypted-cache open path in three isolated stages so a
failure pinpoints the break site: Stage A loads libsqlcipher.so
(System.loadLibrary), Stage B reaches SQLiteConnection.nativeOpen via a keyed
open, Stage C opens the full Room encrypted cache through the production
SupportOpenHelperFactory. Each stage records the in-process page size
(Os.sysconf _SC_PAGESIZE) and re-raises the full UnsatisfiedLinkError (which
.so, cause chain, stacktrace) on failure. Investigation only; no app/src/main
crypto change.

On-device A/B finding (Pixel 8 Pro, husky, real SDK 37 / Android 17):
- 4 KB pages (PAGE_SIZE=4096): Stages A, B, C ALL PASS.
- 16 KB pages: NOT tested on-device — the Pixel "Boot with 16 KB page size"
  toggle is gated behind an unlocked bootloader ("All user data and settings
  will be wiped when activating 16 KB mode"), i.e. destructive + out of scope.

Static ELF proof (refutes the #359 root-cause hypothesis): every bundled
native library is already 16 KB-aligned (all PT_LOAD p_align = 0x4000),
including arm64-v8a libsqlcipher.so from sqlcipher-android 4.16.0 (unchanged
since the original encrypted-cache commit, so the crashing build shipped the
same aligned lib) plus libandroidx.graphics.path.so and
libdatastore_shared_counter.so. So the "unaligned .so" theory does not hold;
the nativeOpen UnsatisfiedLinkError needs a different root cause (library-load
ordering / a nativeOpen reached without a loaded lib, or an APK-delivery /
device-specific issue).
2026-07-07 14:52:47 -05:00
Jason Ross 41dca1840d Merge pull request #406 from JMR-dev/ci-404-wedge-diagnostics
ci: capture thread-dump + service state on an E2E wedge to prove the root cause (#404)
2026-07-07 11:55:11 -05:00
JMR-dev 90dfb189e6 fix(ci): drop matrix E2E wedge-capture that hangs all 8 legs
The `timeout -k 30s` wrapper + `capture_wedge()` added in 2f32657 for the
matrix `e2e` job (API 29-36) reproducibly wedges every leg, while the
manually-provisioned API 37 preview shard running the identical capture
logic passes. Revert the two matrix "Run E2E tests" steps' `script:`
blocks to main's plain script (just the backgrounded logcat stream +
`./gradlew connectedDebugAndroidTest`) and drop the now-dead "Upload wedge
diagnostics" step from the matrix job.

Kept untouched: the job-level `timeout-minutes: 50` backstop added in
27ede55, and the entire `e2e-preview` job (its own capture_wedge/watchdog
and wedge-diagnostics-api37-preview-shard* upload are unaffected).
2026-07-07 11:29:41 -05:00
JMR-dev 27ede55dbd ci(e2e): add job-level timeout to matrix E2E job (#404)
The matrix E2E job (api-level 29-36) had no timeout-minutes, so a wedge
hangs until GitHub's 6-hour default instead of being force-killed. The
sibling e2e-preview job already sets timeout-minutes: 35. A normal
matrix run is ~15-20 min and a retry-inclusive run ~40 min, so set
timeout-minutes: 50 to give headroom above the in-step wedge-capture
timeout (1200s) while still bounding worst-case runtime.
2026-07-07 07:46:33 -05:00
JMR-devandClaude Opus 4.8 2f32657aff ci: capture thread-dump + service state on an E2E wedge to prove the root cause (#404)
E2E legs intermittently WEDGE (hang) with no fast-fail until the job force-kill,
and GitHub's post-force-kill step behavior is unreliable, so #388's diagnostics
don't reliably capture the wedge — and don't capture wedge-specific state anyway.

Wrap the `connectedDebugAndroidTest` run (both the `e2e` matrix first-attempt +
retry, and each `e2e-preview` shard) in an explicit `timeout -k 30s 1200`
(20 min) — comfortably above a normal run (~13-15 min), well below the hard cap —
so a wedge trips the wrapper (exit 124), NOT the force-kill, GUARANTEEING the
capture runs while the emulator is still alive. On 124, capture_wedge grabs the
smoking gun into a `wedge-diagnostics-api<level>` artifact: the running/last test
(logcat TestRunner), SIGQUIT (kill -3) thread dumps of the app + instrumentation
processes (ART -> logcat + /data/anr), dumpsys activity/window, `service list` +
`service check input/window/activity` (the boot-race crux), sys.boot_completed +
init.svc.* state, the snapshot cache-hit note, and accel/kvm/mem/disk. Then it
exits with the real status so #388's diagnostics + the existing retry still fire;
a normal run finishes before the wrapper and is unaffected.

EVIDENCE ONLY — no boot-readiness guard/fix (maintainer: prove the cause first).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 07:46:33 -05:00
Jason Ross 5656b3cd99 Merge pull request #414 from JMR-dev/mergify/configuration-deprecated-update
ci(mergify): upgrade configuration to current format
2026-07-07 07:41:55 -05:00
Jason Ross d3264db920 Merge branch 'main' into mergify/configuration-deprecated-update 2026-07-07 07:20:22 -05:00
Jason Ross 404c107aef Merge pull request #419 from JMR-dev/refactor-405-harness-coldfetch
refactor(scripts): fold cold-fetch pause-hook A/B into device-testing harness (#405)
2026-07-07 05:29:48 -05:00
JMR-devandClaude Opus 4.8 55f1f59e3d refactor(scripts): fold cold-fetch pause-hook A/B into device-testing harness (#405)
Bakes the proven 2026-07-06 cold-vs-warm pause-hook flow into
scripts/device-testing/ as a first-class, reproducible `cold-fetch-ab`
scenario, upstreaming the scratchpad driver.

- fetchgate.py: FETCH_GATE pause/resume/query helpers through the guarded
  adb wrapper, with ordered-broadcast read-back parsing (paused=[...]).
- scenarios.cold_fetch_ab: pre-arm halt -> detect sign-in (sync all
  breadcrumb) -> confirm halt (prefetch skipped) -> wait for header sync ->
  measure cold opens -> resume -> measure warm opens. ALWAYS resumes on exit
  (finally), even on error -- never leaves fetch paused.
- report.render_cold_fetch_ab: gate summary, cold/warm tables, cold-vs-warm
  delta, connect=0ms reuse proof, throttle signature.
- Portability (subsumes #392): file-based uiautomator dump (not /dev/tty),
  UTF-8 adb decode + PYTHONUTF8/console I/O, openMessage-breadcrumb readiness,
  row-selection hardening (skip non-message rows).

The pause hook is debug-build-only (#393/#395), so the scenario needs a debug
APK. Automated validation: mocked unittest coverage (adb/breadcrumbs/gate) for
the helpers and the A/B scenario incl. restore-on-error, plus a --dry-run path
exercised end-to-end through perf_harness.main. A full on-device run is a
follow-up. Dev-tooling only; no app/src changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 05:07:11 -05:00
Jason Ross 582d1f3077 Merge pull request #418 from JMR-dev/test-386-ratchet-floor
test(compose): re-ratchet JaCoCo line-coverage floor to 0.84 (#386)
2026-07-07 03:33:12 -05:00
JMR-dev 4464e3f5e4 test(compose): re-ratchet JaCoCo line-coverage floor to 0.84 (#386)
Final step of the Robolectric Compose epic (#373): now that batches
#376-384 have all landed and proven stable, measure the new whole-app
JVM line-coverage baseline and raise the no-regression floor to match.

Measured 87.89% line (7994/9095), up from 80.21% (4838/6032) when the
floor was last set. Floor moves 0.79 -> 0.84, a deliberately wider
~3.9% headroom (vs. the usual ~0.5-1%) for this first post-epic
measurement; the maintainer can tighten it further in a follow-up PR.
Docs (CLAUDE.md, preflight SKILL.md) updated to match.
2026-07-07 03:14:01 -05:00
Jason Ross 4f09efaf84 Merge pull request #395 from JMR-dev/feat-393-debug-fetch-gate
feat(debug): dev-only pause/halt mail-fetch hook for test harnesses (#393)
2026-07-07 02:46:14 -05:00
Jason Ross 85009ee88a Merge branch 'main' into feat-393-debug-fetch-gate 2026-07-07 02:26:28 -05:00
Jason Ross 2887516faa Merge pull request #417 from JMR-dev/test-384-robolectric-app-shell
test(compose): Robolectric JVM tests — app shell & lock gate host (#384)
2026-07-07 01:38:33 -05:00
JMR-devandClaude Opus 4.8 9f7ebdafb9 test(compose): Robolectric JVM tests — app shell & lock gate host (#384)
Batch 9/9 (final) of the Robolectric Compose JVM-test epic (#373).

- AppLockGateHostJvmTest: drives the app-lock gate host on the JVM via the
  v2 createComposeRule under Robolectric, covering the Unlocked / Checking /
  Locked render branches, the "content stays composed after re-lock" latch,
  and the no-FragmentActivity auth-error path. AppLockViewModel is mocked.
  Drops **/AppLockGateHost* from jacocoNonJvmTestableSurface.
- LibreMailAppJvmTest: covers the JVM-tractable parts of LibreMailApp.kt —
  LibreMailBottomBar, StartupCrashPrompt (+ its dialog buttons), and
  LibreMailApp's cold-start "hold until known" guards.
- LibreMailApp itself KEPT excluded (the acceptable exception noted in #384):
  its NavHost start destinations call hiltViewModel() and the graph needs
  owners a plain JVM compose rule can't surface, so graph-level nav stays on
  the instrumented OnboardingFlowTest. Documented in the jacoco list.

Instrumented LibreMailBottomBarTest / StartupCrashPromptTest stay as the
on-device E2E. JaCoCo floor unchanged (0.79); scoped line coverage 0.8426.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 01:15:07 -05:00
Jason Ross 97a51ed17d Merge pull request #416 from JMR-dev/test-383-robolectric-mailbox
test(compose): Robolectric JVM tests — mailbox screen + folder drawer (#383)
2026-07-07 01:11:42 -05:00
Jason Ross a1cf982a84 Merge branch 'main' into test-383-robolectric-mailbox 2026-07-07 00:53:06 -05:00
Jason Ross 5f69082ba3 Merge pull request #415 from JMR-dev/test-382-robolectric-compose-editor
test(compose): Robolectric JVM tests — compose editor screen (#382)
2026-07-07 00:43:33 -05:00
Jason Ross 4b6f206f2d Merge branch 'main' into test-382-robolectric-compose-editor 2026-07-07 00:23:13 -05:00
Jason Ross 190343bd84 Merge pull request #401 from JMR-dev/test-380-robolectric-settings
test(compose): Robolectric JVM tests for settings screens (#380)
2026-07-07 00:19:42 -05:00
JMR-devandClaude Opus 4.8 cd25544e07 test(compose): Robolectric JVM tests — mailbox screen + folder drawer (#383)
Convert the Paging 3 mailbox list + folder drawer to Robolectric JVM Compose
tests (batch 8/9 of umbrella #373) and drop them from jacocoNonJvmTestableSurface.

- MailboxScreenJvmTest drives the real MailboxScreen + MailboxViewModel over
  mocked repositories, feeding Paging via static PagingData.from flows (no real
  Room/Paging source, mirroring MailboxViewModelTest). Covers the no-accounts
  welcome fallback, populated list (sender/subject/snippet, offline badge,
  unified per-account labels + filter chips, drafts/outbox entries), the
  empty/loading/no-results states, search open/close, and the multi-select
  contextual action bar (overflow, archive/spam/delete confirms, move picker,
  archive-hidden-in-archive, disambiguated app-bar title).
- FolderDrawerJvmTest drives the callback-driven FolderDrawer: friendly role
  names, duplicate-name provider disambiguation + account-switch gap, folder
  taps, the multi-account switcher/dropdown, and the unread badge (incl. 99+ cap).
- Remove **/MailboxScreen* and **/FolderDrawer* from jacocoNonJvmTestableSurface;
  overall JVM line coverage 84.69% (floor unchanged at 0.79).

The instrumented MailboxScreenTest/FolderDrawerTest stay as the on-device E2E.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 00:08:04 -05:00
JMR-devandClaude Opus 4.8 c42f9a7e01 test(compose): Robolectric JVM tests for settings screens (#380)
Convert the settings screens/components to Robolectric JVM Compose tests
(umbrella #373, batch 5/9) and drop their globs from
`jacocoNonJvmTestableSurface`, so they count toward JaCoCo's JVM-testable
surface without an emulator.

New `src/test` Robolectric Compose tests (v2 createComposeRule, @Config sdk=36,
NATIVE graphics), mocking each ViewModel where needed:
- SettingsComponentsJvmTest (SectionHeader/SwitchRow/ClickRow/RadioRow/RetentionSection)
- SettingsScreenJvmTest (+ stateless ContactAutocompleteRow)
- AccountSettingsScreenJvmTest
- SignaturesScreenJvmTest
- SignatureEditScreenJvmTest

Line coverage of the newly-included files: SettingsComponents 100%,
SignatureEditScreen 100%, SignaturesScreen 97%, SettingsScreen 95%,
AccountSettingsScreen 84%. Overall scoped line coverage 86.2%. The instrumented
androidTest E2E stay; the JaCoCo floor is unchanged (re-ratchet is #386).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:57:05 -05:00
JMR-devandClaude Opus 4.8 b9cadde5bd test(compose): Robolectric JVM tests — compose editor screen (#382)
Port the instrumented ComposeScreenTest to a Robolectric JVM Compose
test (batch 7/9 of umbrella #373) so ComposeScreen's render + interaction
code counts toward JaCoCo's JVM-testable surface, and drop
`**/ComposeScreen*` from `jacocoNonJvmTestableSurface`.

ComposeViewModel is large (7 collaborators, several Context/Room-backed),
so it is mocked — mirroring AccountPickerScreenJvmTest / ManualSetup
ScreenJvmTest — with its state/accounts/finished flows stubbed so every
render/state branch is injectable. A RESUMED lifecycle owner (also the
back-press dispatcher owner) and a no-op ActivityResultRegistryOwner let
`collectAsStateWithLifecycle`, the BackHandler, and the attachment/inline
-image launchers compose on the JVM. The embedded RichTextBodyField renders
live; its toolbar accessibility labels and body-change plumbing are covered.

The instrumented ComposeScreenTest stays as the on-device E2E. JaCoCo floor
unchanged (0.79); overall line coverage 0.86.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:54:29 -05:00
Jason Ross 74a2cde7b1 Merge pull request #400 from JMR-dev/test-381-robolectric-reader
test(compose): Robolectric JVM tests for the reader screen (#381)
2026-07-06 23:51:44 -05:00
JMR-devandClaude Opus 4.8 15f4af864f test(compose): Robolectric JVM tests for the reader screen (#381)
Port the reader screen's chrome to a Robolectric JVM Compose test (umbrella

ReaderScreenJvmTest drives the real ReaderViewModel over a mocked
MailRepository/SettingsRepository via the v2 createComposeRule() — no emulator —
covering the top bar, star/delete/reply/reply-all/forward actions, the
attachment accordion + downloaded indicator, the attachment download-failure
snackbar, and the loading/plain-text/empty/error/remote-images-banner branches.

WebView caveat: the HTML body renders through HtmlBody, a hardened WebView that
Robolectric can only present as a non-rendering shadow, so the banner branch is
driven via an HTML message with a blank body (no HtmlBody call) and no
WebView-rendered HTML is asserted. HtmlBody.kt stays in scope, covered by its
existing HtmlBodyTest/InlineImageResolverTest. The instrumented ReaderScreenTest
stays as the on-device E2E. ReaderScreenKt lands at 94.3% line coverage; the
bundle rises to 82.7%, above the unchanged 0.79 floor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:27:34 -05:00
Jason Ross 9259591dc8 Merge pull request #397 from JMR-dev/test-379-robolectric-mail-lists
test(compose): Robolectric JVM tests — mail list screens drafts/outbox/reports (batch 4/9)
2026-07-06 23:15:26 -05:00
mergify[bot] 33a7ca1b7d ci(mergify): upgrade configuration to current format 2026-07-07 04:05:32 +00:00
Jason Ross 37008d622e Merge branch 'main' into test-379-robolectric-mail-lists 2026-07-06 22:57:06 -05:00
Jason Ross b4453f6991 Merge pull request #412 from JMR-dev/ci-409-mergify-phase1
ci(mergify): implement Phase 1 — serial merge queue (require-up-to-date KEPT ON)
2026-07-06 22:50:20 -05:00
JMR-devandClaude Opus 4.8 40b14d51cc ci(mergify): add Phase 1 serial merge queue (.mergify.yml)
Implements issue #409: a serial Mergify merge queue that supersedes the
hand-rolled poor-man's queue (autoupdate.yml + ci-trigger.yml +
traffic-control.yml, all already disabled_manually).

- queue_rules "default": batch_size 1 (serial, no batching), merge_method
  merge (merge commits, never squash/rebase), merge_conditions gate on
  check-success = "CI passed" + -draft + -conflict + label != broken.
- merge_queue.max_parallel_checks 1 (true serial; unambiguously
  require-up-to-date-compatible).
- priority_rules map P0..P9 labels (P0 = 10000 highest .. P9 = 1000).
- pull_request_rules queue action triggers auto-queueing (queue_conditions
  alone do NOT auto-queue per Mergify lifecycle docs).

require-up-to-date STAYS ON (Phase 1 is the only trilemma combo that keeps
the checkbox literally enabled AND preserves merge commits). No batching
(that is Phase 2 / #410). Single required gate stays "CI passed".

Validated: YAML parses and conforms to Mergify's published JSON schema
(negative-control confirmed). Merging this activates Mergify, so NO
auto-merge — must be reviewed first.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 22:28:14 -05:00
Jason Ross 594c6167d3 Merge pull request #408 from JMR-dev/ci-407-mergify-spec
docs(ci): propose Mergify merge-queue integration spec (#407)
2026-07-06 22:08:34 -05:00
Jason Ross 6a5e86bb11 Merge branch 'main' into ci-407-mergify-spec 2026-07-06 22:08:23 -05:00
Jason Ross e4db3cadf6 Merge branch 'main' into test-379-robolectric-mail-lists 2026-07-06 21:59:00 -05:00
JMR-devandClaude Opus 4.8 0307df88a1 docs(ci): propose Mergify merge-queue integration spec (#407)
Investigate Mergify (free-for-OSS merge queue + batching + speculative checks)
as the right-way replacement for the hand-rolled traffic-controller
(autoupdate.yml + ci-trigger.yml + mothballed traffic-control.yml) and the
manual serial-bump grind, now that GitHub's native merge queue is org-only and
unavailable to a user account.

Proposal only — NO live .mergify.yml, nothing activates:
- docs/ci/mergify-integration-spec.md: how the queue coexists with the single
  `CI passed` gate; the require-up-to-date x merge-commits x batching trilemma
  and its resolution (Phase 1 serial keeps the rule literally; Phase 2 merge-batch
  moves the up-to-date GUARANTEE into the queue); P0-P9 -> priority_rules mapping;
  what it replaces; interaction with path-filter/sharding/wedge-diag; risks;
  phased adopt recommendation.
- docs/ci/mergify.yml.proposed: annotated, NOT-active proposed config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 21:55:51 -05:00
Jason Ross 41aed01544 Merge pull request #402 from JMR-dev/ci-399-e2e-path-filter
ci: skip the E2E matrix for test-only/docs PRs via a paths-filter + skip-tolerant gate (#399)
2026-07-06 21:54:25 -05:00
JMR-devandClaude Opus 4.8 98b90a19c3 test(compose): Robolectric JVM tests for mail list screens (#379)
Convert the VM-driven mail list screens (DraftsScreen, OutboxScreen,
ProblemReportsScreen) to Robolectric JVM Compose tests in the `test`
source set, driving each real ViewModel over a mocked MailRepository /
ReportStore + DiagnosticsCollector via the v2 createComposeRule() — no
emulator. Each test covers the empty/populated render states, item
rendering (subject/recipient/body, queued-vs-failed status, crash/manual
kind labels), and the interactions (open, delete, cancel, retry, create).

Drop the three now-JVM-covered globs from jacocoNonJvmTestableSurface so
the screens count toward the JaCoCo denominator; measured coverage is
DraftsScreen 100%, OutboxScreen 100%, ProblemReportsScreen 97%, and the
bundle line ratio rises to ~82.7% (floor 0.79 unchanged). The instrumented
androidTest E2Es (DraftsScreenTest / OutboxScreenTest /
ProblemReportsScreenTest) stay as the on-device tests.

Part of the Robolectric Compose umbrella (#373); mirrors the #375/#376
pattern (AddAnotherAccountScreenJvmTest, format-control JVM tests).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 21:40:27 -05:00
Jason Ross 7f1fb3ea43 Merge branch 'main' into ci-399-e2e-path-filter 2026-07-06 21:35:38 -05:00
Jason Ross b8e3b97377 Merge pull request #396 from JMR-dev/test-377-robolectric-onboarding-lock
test(compose): Robolectric JVM tests for onboarding & lock screens (#377)
2026-07-06 21:25:50 -05:00
Jason Ross c4607c2df1 Merge branch 'main' into test-377-robolectric-onboarding-lock 2026-07-06 21:05:23 -05:00
JMR-devandClaude Opus 4.8 e668330151 ci: skip the E2E matrix for test-only/docs PRs via a paths-filter + skip-tolerant gate (#399)
Add a cheap `changes` job (dorny/paths-filter v4, pinned SHA) that sets
e2e_needed=false only when EVERY changed file is in a safe allow-list
(app/src/test/**, **/*.md, docs/**, scripts/**, .claude/**); anything
else -- or any non-pull_request event -- defaults to true (conservative,
"err toward running E2E").

Gate `e2e` and `e2e-preview` on needs.changes.outputs.e2e_needed so the
whole matrix runs or skips together, and rewrite the `ci-passed` gate:
it now BLOCKS on changes!=success, any of traffic-control-tests /
static-analysis / debug-build / unit-tests !=success, or e2e/e2e-preview
==failure|cancelled -- while TOLERATING an intentional e2e/e2e-preview
'skipped'. So test-only/docs PRs go green on the fast gate, a real E2E
failure/cancel still blocks, and a broken filter (changes!=success)
still blocks. Branch protection ("CI passed") context is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:55:58 -05:00
Jason Ross d922b9a6ee Merge pull request #398 from JMR-dev/test-378-robolectric-account-setup
test(compose): Robolectric JVM tests for account setup screens (#378)
2026-07-06 20:54:48 -05:00
JMR-devandClaude Opus 4.8 3ea00a1ed9 test(compose): Robolectric JVM tests for account setup screens (#378)
Add Robolectric JVM Compose tests (umbrella #373, batch 3/9) for the
account-setup screens and drop them from `jacocoNonJvmTestableSurface` so
their render/interaction code counts toward the JVM-testable coverage surface:

- AccountPickerScreen (98.9% line)
- AppPasswordSetupScreen (98.7% line)
- ManualSetupScreen (98.5% line)

Each test drives the real screen via the v2 `createComposeRule()` under
RobolectricTestRunner with a mocked ViewModel (their own logic stays covered by
the ViewModel unit tests), a RESUMED LifecycleOwner for
`collectAsStateWithLifecycle`, a no-op ActivityResultRegistry for the Outlook
launcher, and a recording UriHandler for the app-password help links — covering
render, per-provider chrome, field/submit wiring, and the enabled/busy/error/
done branches. The instrumented androidTest E2Es stay as the on-device coverage.

The JaCoCo floor (0.79) is unchanged — the re-ratchet is the final #373 step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:34:21 -05:00
JMR-devandClaude Opus 4.8 799669d6a6 test(compose): Robolectric JVM tests for onboarding & lock screens (#377)
Add Robolectric JVM Compose tests (umbrella #373, batch 2/9) for the
stateless onboarding + lock screens, and drop each from
jacocoNonJvmTestableSurface so its render/interaction code now counts as
JVM-testable surface:

- LockScreen: locked title/body, optional error, unlock callback.
- WelcomeContent + OnboardingWelcomeScreen: render + add-account; the
  wrapper's NotificationPermissionEffect launcher is wired to a no-op
  ActivityResultRegistry so no system dialog is surfaced on the JVM.
- LicenseScreen: real bundled GPL text renders, Agree gated on
  scroll-to-end, Decline.
- ContactsAccessContent (skip/grant/request/rationale) plus the
  ContactsAccessScreen wrapper, driven by a mocked OnboardingViewModel.
- BatteryOptimizationScreen: offered vs. done states; Take me there marks
  the prompt handled and resolves the settings intent; Not now finishes.

Contacts/Battery use a tall @Config qualifier so their centered,
non-scrolling columns fit without the lower controls clipping. The
instrumented androidTest tests are kept (and remain the coverage for the
system back press, which the JVM compose rule cannot drive). JaCoCo floor
unchanged at 0.79 (the re-ratchet is the final #373 step).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:13:49 -05:00
JMR-devandClaude Opus 4.8 8f8608a430 feat(debug): dev-only pause/halt mail-fetch hook for test harnesses (#393)
The on-device perf harness cannot force a genuinely uncached body fetch: proactive
backfill (#12) and post-sync body prefetch (#88/#89) warm the cache before a test can
open a message. Add a debug-only, adb-reachable hook to pause proactive fetch so a real
uncached open can be measured.

Components:
- DebugFetchGate (src/main): thread-safe in-memory holder of paused FetchScopes
  (BACKFILL, PREFETCH; `all` alias). Defaults to not-paused; HEADER_SYNC and on-demand
  OPEN are never gateable.
- FetchGateReceiver (src/debug only): BroadcastReceiver registered in the debug manifest,
  driven by `adb shell am broadcast -a org.libremail.debug.FETCH_GATE -n .../FetchGateReceiver
  --es action <pause|resume|query> --es scope <backfill,prefetch|all>`. Returns the state as
  ordered-broadcast result data (paused=[...]) for a synchronous read-back.

Enforcement (each read guarded by BuildConfig.DEBUG so R8 strips it from release):
- BackfillWorker.doWork() entry -> skip-and-reschedule when BACKFILL is paused, mirroring
  the existing cache-lock deferral (covers periodic + backfillNow()).
- MailSyncer/MailBackfiller.prefetchIfEnabled -> early-return when PREFETCH is paused.
  openMessage / fetchBodyMarkingSeen / fetchAttachment are deliberately NOT gated.

Debug-only: receiver + <receiver> live wholly in src/debug; every gate read in main is
behind BuildConfig.DEBUG. Verified on assembleRelease that R8 strips DebugFetchGate /
FetchScope / FetchGateReceiver and the log strings from the release APK, and the merged
release manifest has no FETCH_GATE receiver.

PII-free AppLog breadcrumbs on pause/resume/query and on each gate-triggered defer/skip
(scope names only).

Tests: DebugFetchGateTest, BackfillWorkerTest / MailSyncerTest / MailBackfillerTest
enforcement cases, and FetchGateReceiverInstrumentedTest (ordered-broadcast -> gate ->
read-back; gated worker defers while an un-gated path runs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:13:22 -05:00
Jason Ross ae10a5ff3b Merge pull request #394 from JMR-dev/test-376-robolectric-richtext
test(compose): Robolectric JVM tests for rich-text format controls (#376)
2026-07-06 20:09:40 -05:00
Jason Ross c2ab0e3141 Merge branch 'main' into test-376-robolectric-richtext 2026-07-06 19:54:03 -05:00
JMR-devandClaude Opus 4.8 9bbfa2108a test(compose): Robolectric JVM tests for rich-text format controls (#376)
Port the instrumented ColorSwatchRow / FontPicker / FontSizePicker /
ParagraphAlignmentControl tests to Robolectric JVM Compose tests (v2
createComposeRule, @GraphicsMode NATIVE, @Config sdk=36) in the `test`
source set, and drop their four globs from `jacocoNonJvmTestableSurface`
so they count toward the JVM coverage metric. The instrumented tests stay.

Also fix a latent gap in the #375 infra: the JaCoCo agent skips classes
with no code-source location, which is exactly how Robolectric loads the
classes-under-test through its sandbox classloader — so Robolectric-only
Compose coverage recorded as zero (the PoC AddAnotherAccountScreen
included). `isIncludeNoLocationClasses = true` on the Test tasks makes
that coverage register; scoped bundle line coverage rises ~0.80 -> ~0.82.
Floor left at 0.79 (#386 re-ratchets).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 19:49:59 -05:00
Jason Ross cfd433af6f Merge pull request #367 from JMR-dev/fix-359-sqlcipher-16kb
fix(security): fail closed on cache-encryption load failure + StrongBox-back keys (#359)
2026-07-06 19:41:35 -05:00
Jason Ross 8689885964 Merge branch 'main' into fix-359-sqlcipher-16kb 2026-07-06 19:23:24 -05:00
Jason Ross 6213ee8901 Merge pull request #391 from JMR-dev/ci-389-sdk-setup-hardening
ci: retry + cache Android SDK/emulator setup to survive corrupt-zip sdkmanager failures (#389)
2026-07-06 19:22:02 -05:00
Jason Ross 8ac1d28dc8 Merge branch 'main' into ci-389-sdk-setup-hardening 2026-07-06 19:03:26 -05:00
JMR-devandClaude Opus 4.8 aa628831be ci: retry + cache Android SDK/emulator setup to survive corrupt-zip sdkmanager failures (#389)
The dominant merge-blocking flake was the "Set up Android SDK" step
(android-actions/setup-android v4.0.1) dying BEFORE the emulator starts:

    Wrong version in preinstalled sdkmanager
    Warning: ... preparing SDK package Android Emulator: Error reading Zip
    content from a SeekableByteChannel.
    Error: The process '.../sdkmanager' failed with exit code 1

Root cause: the action's default cmdline-tools version (20.0) rarely matches the
runner image's preinstalled one, so it logs "Wrong version in preinstalled
sdkmanager" and re-fetches cmdline-tools with NO checksum; it then runs its
default `sdkmanager tools platform-tools` install. Any of those downloads can be
a corrupt/truncated zip, which sdkmanager turns into an un-retried exit 1. v4.0.1
is the latest release, so this is fixed by configuration + hardening, not a bump.

Harden with verify -> reject -> retry, never trusting sdkmanager's exit code
alone, via a new stdlib-only helper .github/scripts/setup_android_sdk.py:

- bootstrap: download the pinned cmdline-tools zip, verify size + SHA-256
  (authoritative pin, cross-checked against Google's published SHA-1), and
  install it to $ANDROID_SDK_ROOT/cmdline-tools/20.0 -- the exact path
  setup-android probes first, so the action reuses the verified tree and never
  does its own unverified "Wrong version" re-download. A mismatch (corrupt OR
  wrong version) deletes the bad zip + any half-extracted dir and re-downloads.
- install: sdkmanager --install with retry + backoff; on a corrupt package zip it
  purges the partial/corrupt package dir (and sdkmanager's temp dirs) before
  retrying, forcing a fresh download instead of a re-read.
- setup-android now runs with packages: "" (no flaky tools/platform-tools
  install) and cmdline-tools-version: "14742923" (reuse the verified bootstrap).
- actions/cache restore + success-gated save so only a verified SDK is ever
  cached (integrity gates the cache); shrinks the re-download/corruption surface.

Applied to every SDK-setup job (debug-build, unit-tests, static-analysis, e2e
matrix, e2e-preview). Emulator BOOT logic, #372 API-37 sharding, and #388
diagnostics are untouched. Pure-logic helpers are unit-tested
(test_setup_android_sdk.py, run by the traffic-control-tests job).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:59:05 -05:00
Jason Ross 1053fa80f0 Merge pull request #375 from JMR-dev/robolectric-compose-scope
test(compose): Robolectric JVM Compose testing infra + PoC (#373)
2026-07-06 18:36:36 -05:00
Jason Ross 35b6b35d16 Merge branch 'main' into robolectric-compose-scope 2026-07-06 17:44:37 -05:00
Jason Ross 5a8c4e386e Merge branch 'main' into fix-359-sqlcipher-16kb 2026-07-06 16:56:27 -05:00
Jason Ross c04198a158 Merge pull request #388 from JMR-dev/ci-387-e2e-diagnostics
ci(e2e): capture logcat + emulator/system diagnostics across the E2E matrix (#387)
2026-07-06 16:55:52 -05:00
Jason Ross 0b1fb05a90 Merge branch 'main' into ci-387-e2e-diagnostics 2026-07-06 16:42:18 -05:00
Jason Ross e062331e09 Merge pull request #374 from JMR-dev/fix-signatures-test-teardown-race
test(settings): cancel viewModelScope before db.close in SignaturesScreenTest to fix a Room teardown race
2026-07-06 16:21:56 -05:00
JMR-devandClaude Opus 4.8 187a8effb0 ci(e2e): capture logcat + emulator/system diagnostics across the E2E matrix (#387)
The API 29-36 `e2e` matrix uploaded only its test report, so an emulator
flake or a red leg (e.g. `E2E (31)` dying on a bare `sdkmanager` exit 1)
left nothing to diagnose. Bring the #334 API-37 diagnostics to the matrix,
inline (no changes to `e2e-preview`, which PR #372 is restructuring):

- Stream `adb logcat -v time` to `$RUNNER_TEMP/logcat-api<level>.txt` at the
  top of both the "Run E2E tests" and retry reactivecircus steps (emulator is
  booted there); backgrounded so gradle stays the exit-status-bearing command.
- New `if: failure()` step dumps device + runner state (adb devices, logcat
  tail, emulator -accel-check, /dev/kvm, free -h, df -h) to the step log and a
  diagnostics file; every probe guarded with `|| true`.
- New `if: always()` upload-artifact (same pinned v7 SHA) `e2e-diagnostics-api<level>`
  carries the logcat + diagnostics files, `if-no-files-found: warn`.
- Make "Install SDK platform and build-tools" diagnosable: bounded 3x retry with
  backoff for a transient sdkmanager failure, and print `--list_installed` on a
  hard failure instead of a bare exit 1.

Keeps reactivecircus/android-emulator-runner and the existing boot-race retry.
Additive/diagnostic only; no boot-affecting flags change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 15:53:54 -05:00
Jason Ross 0186fa5ed9 Merge branch 'main' into robolectric-compose-scope 2026-07-06 15:53:48 -05:00
JMR-devandClaude Opus 4.8 324f7c2c51 fix(test): resolve Robolectric android-all via Gradle so the JVM Compose test runs in CI (#373)
Robolectric resolved its android-all-instrumented runtime jar lazily at test
time via its own MavenDependencyResolver/MavenArtifactFetcher, and that
download is unreliable on CI runners: AddAnotherAccountScreenJvmTest failed
with `AssertionError at MavenArtifactFetcher ... IOException` ("Failed to
fetch maven artifact"), though it passed locally where ~/.m2 was warm.

Resolve the jar through Gradle instead (reliable, cached, persisted by the CI
Gradle cache) and hand it to Robolectric in offline mode so it never hits the
network at test time:
- Pin org.robolectric:android-all-instrumented:16-robolectric-13921718-i7
  (exactly what Robolectric 4.16.1 DefaultSdkProvider maps @Config(sdk=36) to)
  in the version catalog.
- Add it to a dedicated resolvable configuration (NOT testImplementation/
  testRuntimeOnly, which would flatten the ~200MB instrumented framework onto
  the JVM test classpath and collide with the stub android.jar).
- syncRobolectricAndroidAll stages the jar under its Maven filename, and
  robolectric.offline + robolectric.dependency.dir point Robolectric's
  LocalDependencyResolver at it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 15:49:07 -05:00
Jason Ross 62797bc8c3 Merge branch 'main' into fix-359-sqlcipher-16kb 2026-07-06 15:46:14 -05:00
Jason Ross c8290ee545 Merge branch 'main' into fix-signatures-test-teardown-race 2026-07-06 15:45:09 -05:00
Jason Ross 9aa83a458c Merge pull request #372 from JMR-dev/ci-api37-e2e-sharding-spike
ci: shard the API 37 preview E2E into 2 parallel shards + retry parity
2026-07-06 15:41:02 -05:00
JMR-devandClaude Opus 4.8 cf1a8f6b83 test(compose): Robolectric JVM Compose testing infra + PoC (#373)
Enables unit-testing Jetpack Compose UI on the JVM via Robolectric, so
render-only screens can leave the jacocoNonJvmTestableSurface exclusion
list and be counted by JaCoCo without an emulator.

- add Robolectric 4.16.1 (test scope) + Compose ui-test-junit4/-manifest
- testOptions.unitTests.isIncludeAndroidResources = true so resources
  (strings, Material3 theme) resolve on the JVM
- src/test/resources/robolectric.properties pins sdk=36 (targetSdk 37 is
  a preview level Robolectric 4.16 has no sandbox for)
- PoC: AddAnotherAccountScreenJvmTest drives the screen with the v2
  createComposeRule under RobolectricTestRunner (3 tests, green on the JVM)
- drop AddAnotherAccountScreen from jacocoNonJvmTestableSurface (now
  JVM-covered); floor stays 0.79 — re-ratchet deferred to end of #373

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 15:37:23 -05:00
Jason Ross bb412e263e Merge branch 'main' into fix-signatures-test-teardown-race 2026-07-06 15:29:59 -05:00
Jason Ross b02ed87313 Merge branch 'main' into ci-api37-e2e-sharding-spike 2026-07-06 15:23:07 -05:00
Jason Ross 32199ef037 Merge branch 'main' into fix-359-sqlcipher-16kb 2026-07-06 15:22:45 -05:00
Jason Ross 1423395454 Merge pull request #370 from JMR-dev/feat-device-testing-harness
feat(scripts): device-testing perf harness
2026-07-06 15:22:20 -05:00
Jason Ross 1a3393da53 Merge branch 'main' into ci-api37-e2e-sharding-spike 2026-07-06 15:12:47 -05:00
JMR-devandClaude Opus 4.8 08ee9e7abc ci: productionize API 37 preview E2E sharding (adopt N=2)
The spike commits already implemented the N=2 shard matrix (numShards/
shardIndex via -Pandroid.testInstrumentationRunnerArguments.*), per-shard
test-retry parity, adb start-server before the boot loop, and shard-suffixed
artifact names. This drops the SPIKE / DRAFT "do not merge as-is" framing from
the ci.yml comments and reframes docs/perf/api37-e2e-sharding-spike.md from a
feasibility spike into the adopted design, so the change is mergeable as-is.

Also fixes the doc's section 3a example, which showed 1-based shardIndex values
[1, 2]; shardIndex is 0-based (0..numShards-1) and the implementation correctly
uses matrix.shard: [0, 1] -- [1, 2] would run an empty bucket and silently drop
half the suite.

Fan-in unchanged and verified: ci-passed still lists e2e-preview once; GHA
matrix aggregation makes its result `failure` if either shard fails, so both
shards must pass for the gate to go green. Branch protection requires the
"CI passed" context (not the per-leg "E2E (API 37 preview) (N)" check names),
so no branch-protection change is needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:52:48 -05:00
Jason Ross 5f77eb7d8b Merge branch 'main' into fix-359-sqlcipher-16kb 2026-07-06 14:51:12 -05:00
Jason Ross d6c694c7e0 Merge branch 'main' into feat-device-testing-harness 2026-07-06 14:47:16 -05:00
JMR-devandClaude Opus 4.8 ca6d90b602 test(settings): cancel viewModelScope before db.close in SignaturesScreenTest to fix a Room teardown race (flaky on API-37 CI)
SignaturesScreenTest built a real SignaturesViewModel by hand but tore down
with a bare `db.close()` that never cancelled viewModelScope. The ViewModel's
`signatures` StateFlow is a Room InvalidationTracker Flow kept alive by
stateIn(WhileSubscribed(5_000)), so the collector could stay live up to 5s
after the UI detached — a re-query then landed on the just-closed in-memory DB
and threw SQLITE_MISUSE ("connection is closed"). Timing-dependent, hence the
intermittent API-37 CI failure in tappingRadioOnNonDefault_makesItTheDefault.

Fix: hold the ViewModel in an androidx.lifecycle.ViewModelStore and, in @After,
call store.clear() (→ ViewModel.onCleared() → cancels viewModelScope) BEFORE
db.close(), so the collector is gone before the DB closes. Behaviour and
assertions are unchanged; the fix removes the race by construction.

Audited the androidTest tree for the same hazard and fixed two siblings the
same way:
- AccountSettingsScreenTest: had the same live-Room-Flow-vs-close race,
  previously worked around by never closing the in-memory DB at all; now
  clears the ViewModel then closes the DB.
- ComposeScreenTest: ComposeViewModel's init launches a viewModelScope
  coroutine that reads the real accountSettings/signature Room repos; clear
  the store before db.close() to avoid the same in-flight-read-vs-close race.

Verified locally: connectedDebugAndroidTest green for all three classes
(12/12) on a cold-booted emulator, plus the JVM fast gate (assembleDebug,
testDebugUnitTest, jacocoTestCoverageVerification, compileDebugAndroidTestKotlin,
lintDebug, ktlintCheck, detekt).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:46:40 -05:00
JMR-devandClaude Opus 4.8 0e9f54e686 ci(spike): retry parity + adb start-server for API 37 shard PoC
Folds the #370 root-cause finding into the spike. The stable `e2e` matrix
retries its test run once; `e2e-preview` runs connectedDebugAndroidTest exactly
once, so a flaky test self-heals on API 29-36 but wedges the required gate on
API 37 (e.g. #370's SignaturesScreenTest teardown race).

Doc: adds risk item 9 (retry-parity gap + its sharding interaction — per-test
flake is NOT amplified by sharding unlike boot flake, and a per-shard retry
costs only B + T/N; framed mitigation-not-fix) and two §6 recommendations
(retry parity, mirrored into api37_e2e.py; adb start-server before the boot
loop).

PoC (ci.yml): per-shard single test retry (::warning:: on retried-but-passed)
+ adb start-server before the boot loop. The api37_e2e.py retry mirror stays a
documented recommendation (local path needs a real-emulator validation this
spike did not boot). Still DRAFT, not auto-merged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:40:02 -05:00
Jason Ross a84b042af5 Merge pull request #371 from JMR-dev/chore-preflight-coverage-gate
chore(preflight): run jacocoTestCoverageVerification in the fast gate
2026-07-06 14:36:55 -05:00
JMR-devandClaude Opus 4.8 66da643249 ci(spike): PoC shard API 37 preview E2E + feasibility doc
Feasibility spike for sharding the e2e-preview job (the hand-provisioned
API 37 / google_apis_ps16k 16 KB-page emulator), CI's longest leg
(~16.4-17.6 min). docs/perf/api37-e2e-sharding-spike.md breaks the leg into
fixed overhead B ~8.3 min (setup + boot + Gradle daemon/config/compile/install)
vs parallelizable test execution T ~8.8 min, models B + T/N for N=2/3/4, and
recommends N=2 (~17.1 -> ~12.7 min, ~28% off the critical path) capped by the
API 30 matrix wall (~12.0 min) beyond N=3.

DRAFT PoC (do NOT merge as-is): converts e2e-preview to a strategy.matrix.shard
[0, 1] fan-out passing AndroidJUnitRunner numShards/shardIndex through the
existing -Pandroid.testInstrumentationRunnerArguments.* channel (no GMD, no
orchestrator, no Gradle change). Artifact names gain a shard suffix;
ci-passed still lists e2e-preview once (matrix fan-in keeps the single gate).
Local preflight stays single-emulator. Relates to #258.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:31:40 -05:00
JMR-devandClaude Opus 4.8 042b50116c build(jacoco): scope the fail-closed encryption UI out of the JVM coverage surface (#359)
CacheEncryptionGate.kt (the gate composable, blank cover, error screen, and ephemeral
report-review screen added for #359) is pure Compose render code, structurally
unreachable from a JVM unit test the same way every other Screen file in
jacocoNonJvmTestableSurface is. Left in scope, it dragged the whole-app line ratio to
0.78, just under the 0.79 no-regression floor.

Excluded it via "**/CacheEncryptionGateKt*" rather than the usual bare
"**/CacheEncryptionGate*" pattern this list otherwise uses, because
CacheEncryptionGateViewModel is named with "CacheEncryptionGate" as a literal
prefix - the bare wildcard would also have swallowed the already JVM-tested,
94%-covered ViewModel and its sealed CacheEncryptionGateState. CacheEncryptionGateViewModel
and CacheEncryptionUnavailableException stay in scope unchanged.

Verified locally: testDebugUnitTest + jacocoTestCoverageVerification now pass, with
the line ratio recovered to about 0.807 (5,044 covered / 6,249 total lines) - the
same 5,044 covered lines as before, just a smaller, honestly-JVM-testable denominator.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:19:22 -05:00
JMR-devandClaude Opus 4.8 b0ca5421b6 chore(preflight): run jacocoTestCoverageVerification in the fast gate
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:18:40 -05:00
JMR-devandClaude Opus 4.8 bfe5d46654 fix(security): fail closed on cache-encryption load failure with an error gate (#359)
Reworks #367. When SQLCipher native library loading fails while the opt-in
encrypted cache is enabled, the app previously degraded to a plaintext cache
(a silent fail-open that defeats the feature). Now it FAILS CLOSED.

DatabaseProvisioner raises a distinct CacheEncryptionUnavailableException
instead of degrading: it does NOT open plaintext, NOT wipe the on-disk
ciphertext, and NOT write the encryptCache setting. The throw is not
memoized, so a later launch re-attempts and recovers automatically if the
library loads.

A new CacheEncryptionGate wraps the app inside AppLockGateHost (so the
passphrase is already unlocked), probes prepareCache() before any DB-backed
screen composes, and on failure shows CacheEncryptionErrorScreen with the
exact message "Error - decryption could not proceed. Native decryption
library load failure." plus a "Report a problem" action. That action
generates an EPHEMERAL PII-free report via the existing DiagnosticsCollector
(never written to ReportStore, since encryption is unavailable in that
moment) for on-screen review and explicit Copy/Save; the copy says so.

The plaintext AccountDatabase tolerates the exception so accounts stay
readable for the error gate and the report. The encryptCache setting is now
written by exactly one caller: the user Settings toggle.

Tests: fail-closed raises the signal with no plaintext open / no wipe / no
setting write / not memoized; the gate VM resolves Ready vs Unavailable and
builds the ephemeral report; an instrumented error-screen UI test and an
AccountDatabase-resilience instrumented test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 13:48:35 -05:00
JMR-devandClaude Opus 4.8 551a2df66f feat(security): back cache-key Keystore keys with StrongBox, fall back to TEE (#359)
Bind the non-exportable AES-256-GCM keys that seal the SQLCipher cache
passphrase to the hardware StrongBox secure element when the device has
one. Applied in the single shared place, AesGcmKeystoreCipher, so it
covers both the master (KeystoreCrypto) and auth-bound (DatabaseKeyCipher)
keys.

Devices without StrongBox throw StrongBoxUnavailableException at
KeyGenerator.generateKey(); a new generate-with-fallback path catches it
and regenerates a TEE-backed key so key creation still succeeds
everywhere. Guarded on API 28+ (minSdk is 29). Framing, seal/unseal, and
the missing-key policies are unchanged; the passphrase is still never
plaintext at rest and never logged.

Adds a JVM regression test for the StrongBox->TEE fallback via the
existing test seams (existingKey / a new generateKey seam).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 13:46:03 -05:00
JMR-devandClaude Opus 4.8 e436503eaf feat(scripts): device-testing perf harness
Add a cross-platform, standard-library-only Python package under
scripts/device-testing/ that replicates LibreMail's on-device performance-test
scenarios and logging capture, codifying the methodology run by hand on
2026-07-05 (Pixel 10 Pro XL).

Modules:
- breadcrumbs.py: a pure, unit-tested parser for the ImapPerf / MailReader /
  Reader / MailBackfiller breadcrumbs, plus open-correlation that reproduces the
  manual timing-tables.md figures exactly.
- adb.py: a safety-guarded adb wrapper -- an allow-list of adb subcommands and a
  deny-list + assertions on shell commands. The only sanctioned app-state
  mutation is clearing LibreMail's own cache/ (exact-match); no pm clear /
  uninstall / data wipe, and no touching databases/ files/ shared_prefs/
  datastore/ can be constructed.
- uidump.py: uiautomator XML parser + screen recognition (mailbox rows with the
  cached "Available offline" flag, reader, and the keyguard / foreign-app guards).
- scenarios.py: cold-open, message-open (uncached), back-nav, prefetch A/B
  (fetch-policy toggle) and cross-provider, each keyguard-guarded and driven
  through the guarded wrapper.
- report.py + perf_harness.py: aggregates, a timing-tables.md renderer mirroring
  the manual write-up, and the CLI (timestamped run dir with the raw logcat, a
  filtered breadcrumb extract, and the tables). --dry-run prints the exact
  command plan without touching device state.

Tests (stdlib unittest, 68 cases) validate the parser, guardrails, UI
recognition and report against the manual run's real captures (fixtures include
a verbatim perf-extract slice and the reader/lockscreen/alarm dumps). Track the
*.log fixture past the gitignore *.log rule via a scoped negation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 13:21:34 -05:00
Jason Ross 5c715e1676 Merge branch 'main' into fix-359-sqlcipher-16kb 2026-07-06 12:50:29 -05:00
Jason Ross b8d67557a0 Merge pull request #368 from JMR-dev/feat-125-imap-connection-reuse
perf(mail): enable IMAP connection reuse by default with a hardened cache
2026-07-05 22:18:11 -05:00
JMR-dev e36afc8ade changed conditional style to easier to read/maintain when (like switch) statement 2026-07-05 21:18:49 -05:00
JMR-devandClaude Opus 4.8 81a3b7ea34 refactor(mail): early-return guard in ImapConnectionCache (#357 review)
Restructure isConnectionDrop as leading guard clauses (definite-drop
types, then a not-MessagingException early return) instead of a when
expression, per maintainer review feedback on PR #368. Behavior is
unchanged; verified by the existing ImapConnectionCacheTest suite
(all 8 cases still pass), including the FolderClosedException /
StoreClosedException cases that depend on the check running before
the MessagingException .cause guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 20:36:38 -05:00
JMR-devandClaude Opus 4.8 060b7b1a71 Merge origin/main into feat-125-imap-connection-reuse
Resolve the IdleService.kt conflict as a union of both intents:
- #354 (already on main): foreground-service lifecycle rework —
  onStartCommand delegates to the IdleForegroundStarter seam
  (START_NOT_STICKY), cap-window skip/degrade.
- #357 Part 2 / #368: reused-connection idle-eviction sweep and
  low-battery teardown of reused connections.

In startWatchingIfNeeded(), reconcileWatchers() stays inside the
cache-lock-guarded launch and evictIdleReuseConnectionsLoop() launches as
a sibling coroutine that runs while the service lives (its original #368
placement, independent of the cache-lock guard). No behavior change to
either side.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 20:15:53 -05:00
JMR-devandClaude Opus 4.8 cc067408b8 perf(mail): enable IMAP connection reuse by default with a hardened cache
An on-device drilldown proved Gmail server-side throttles LibreMail's
connect-per-operation IMAP: every op was a fresh CONNECT+TLS+LOGIN, and
full-history backfill's body+attachment prefetch generated ~601 connections in
~22 min, tripping (and sustaining) Gmail's per-account rate/bandwidth clamp
(body download collapsed to ~4 KB/s). The `live` gauge peaked at only 5 (Gmail
allows ~15), so it is connection *volume*, not count. Outlook IMAP on the same
device opened in 2-3 s. Reusing one warm socket per account (~601 -> ~1) removes
the throttle's trigger. This wires the reuse path the #125 spike built and left
OFF (issue #357 Part 2 — connection reuse only; prefetch is a separate PR).

How it is enabled (with a safety switch):
- New `BuildConfig.IMAP_CONNECTION_REUSE` (default true) drives the production
  `ImapClient` no-arg `@Inject` constructor. To disable if a server misbehaves,
  flip it to "false" in app/build.gradle.kts — a build-config change, no Kotlin
  edit. The internal `ImapClient(reuseConnections, reuseIdleTimeoutMillis)`
  constructor stays the test/harness seam.
- Universal: applies to all providers (incl. Outlook). No per-provider caps or
  throttling here — that is a separate effort (#356/#360-#364).

Hardening `ImapConnectionCache` for production (was a spike):
- Transparent stale recovery: broadened drop detection to Angus's own
  `iap.ConnectionException` (and a MessagingException caused by one) — the real
  signal `folder.open()` throws on a server-dropped idle socket, which the
  IOException-only check missed, so the reconnect now actually fires. A dropped
  reused socket is rebuilt once and the op retried, so callers see no spurious
  error; a genuine app error (e.g. message-not-found) is never retried.
- Idle eviction: `evictIdle()` closes a connection unused past the reuse idle
  timeout (default 5 min), swept every 2 min by `IdleService`; skips any
  in-use connection.
- Teardown: `IdleService` also tears down reused connections on the low-battery
  push-teardown path (#88/#89/#90), mirroring the IDLE connection teardown.
- Concurrency: one connection per account behind a per-account mutex; the
  eviction sweep takes the lock non-blockingly so it never stalls or interrupts
  an in-flight op. Coexists with IMAP IDLE (its own separate connection).
- PII-free AppLog on the lifecycle (open / reuse-hit / reconnect-stale / evict /
  teardown) keyed by an opaque per-cache ordinal, plus the #358 ImapPerf
  breadcrumb (connect~=0ms on a reuse hit).

Tests (all via the fast gate, no emulator):
- ImapConnectionCacheTest: reuse, retry-once stale recovery, narrow drop
  detection, deterministic idle eviction (injected clock), teardown.
- ImapFolderOpenLatencyTest (GreenMail + counting proxy): N ops share one
  connection/LOGIN; a force-dropped socket is transparently reconnected; an app
  error does not reconnect; idle eviction LOGS-OUT and the next op reconnects.
- Correctness suites (ImapClientTest/ImapClientBackfillTest/MailBackfillerTest)
  pinned to reuse-off to keep their connect-per-op assertions unchanged.

Fast gate green: assembleDebug, testDebugUnitTest, compileDebugAndroidTestKotlin,
lintDebug, ktlintCheck, detekt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 19:47:01 -05:00
Jason Ross a069188b65 Merge pull request #366 from JMR-dev/fix-354-idleservice-fgs
fix(push): stop IdleService dataSync FGS crash-loop on exhausted 24h cap (#354)
2026-07-05 19:36:27 -05:00
JMR-devandClaude Opus 4.8 921681812c fix(data): degrade encrypted cache to plaintext on SQLCipher native-load failure
On Android 15+ / SDK 37 devices with 16 KB memory pages (e.g. Pixel 10 Pro XL,
and the API-37 `google_apis_ps16k` emulator image), a native `.so` not aligned
for 16 KB pages fails to load with `UnsatisfiedLinkError` at
`SQLiteConnection.nativeOpen`. With the opt-in SQLCipher encrypted cache on, this
crashed the app on every cold start (issue #359, x4 on-device) instead of
degrading, and encryption silently never applied.

Fix: DatabaseProvisioner's encryption gate now catches `LinkageError`
(UnsatisfiedLinkError and related native-link failures) when opening/converting
the encrypted cache and degrades cleanly instead of propagating the crash — it
turns `encryptCache` off (so the next start does not re-attempt and re-wipe),
clears any on-disk ciphertext the plaintext framework opener cannot parse
(resetting its now-useless seals), and opens the cache unencrypted. The cache is
a re-syncable copy of server mail, so clearing it loses nothing that cannot be
re-fetched. PII-free AppLog.w breadcrumb on the degrade path.

Dependency: no bump needed or available. The repo already pins the newest
SQLCipher it references, `net.zetetic:sqlcipher-android:4.16.0`, which
docs/play-compliance.md certifies (ELF p_align = 0x4000) as 16 KB-aligned on
every ABI; SQLCipher has shipped 16 KB-aligned binaries since well before it, and
the other two bundled `.so` files (Compose graphics-path, DataStore
shared-counter) are already 16 KB-aligned per that doc. The graceful-degrade
catch is therefore the actionable fix.

Tests:
- Unit (DatabaseProvisionerTest): a simulated native-load failure degrades to a
  plaintext open without crashing, turns encryptCache off, and wipes + reseals an
  already-encrypted cache.
- Instrumented (DatabaseProvisionerInstrumentedTest): a fresh encrypt-on start
  loads the real SQLCipher native library and opens the keyed cache — CI's API-37
  `google_apis_ps16k` 16 KB job exercises the actual `.so` load, catching any
  future 16 KB-alignment regression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 19:26:05 -05:00
JMR-devandClaude Opus 4.8 52503c000c fix(push): stop IdleService dataSync FGS crash-loop on exhausted 24h cap
Root cause: after #302's runtime-cap fallBackToPeriodicSync() stops the
dataSync foreground service, IdleService was restarted (START_STICKY
null-intent redelivery + explicit startForegroundService) and onStartCommand
unconditionally called startForeground(DATA_SYNC) while the rolling-24h budget
was still exhausted. The platform rejected the start with
ForegroundServiceStartNotAllowedException; it was uncaught, the process
crashed, and START_STICKY restarted straight back into the same rejection -- a
crash loop until the 24h window freed budget (#354).

Fix (IdleService.kt):
- onStartCommand now returns START_NOT_STICKY. Push is app-managed
  (LibreMailApplication.ensurePushStarted deterministically restarts it), so the
  sticky null-intent auto-restart was redundant and fired exactly when a dataSync
  FGS start is illegal.
- Guard the foreground start via a new JVM-testable IdleForegroundStarter seam:
  a ForegroundServiceStartNotAllowedException (caught via its IllegalStateException
  supertype, so no minSdk-29 class load) degrades like the cap handler --
  schedulePeriodicSync(), keep the degraded POLLING notification, stopSelf()
  promptly (avoids the "did not call startForeground in time" ANR) -- instead of
  propagating.
- Record the cap event (elapsedRealtime); while still inside the cap window,
  onStartCommand skips the now-guaranteed-illegal foreground start entirely.
- onTimeout stop path kept fast so ForegroundServiceDidNotStopInTimeException
  stays mitigated.

PII-free AppLog.w/i on the degrade paths.

Tests:
- Unit (IdleForegroundStarterTest): onStartCommand returns START_NOT_STICKY; a
  rejected start is caught and routed to degrade without propagating; the cap
  window skips the attempt; a non-ISE propagates.
- Instrumented (IdleServiceForegroundStartInstrumentedTest): the degrade path on
  a real Context -- rejection caught, periodic-sync fallback scheduled, degraded
  "instant delivery paused" notification built, watching skipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 19:17:19 -05:00
Jason Ross 6cb179bd8b Merge pull request #365 from JMR-dev/feat-358-reader-perf-logging
feat(reporting): PII-free latency breadcrumbs on the message-open path (#358)
2026-07-05 18:54:32 -05:00
JMR-devandClaude Opus 4.8 35b869944e feat(reporting): PII-free latency breadcrumbs on the message-open path (#358)
Adds AppLog breadcrumbs to the message-open path so a debug report can show
where the reader's spinner time goes:

- ImapClient.withStore: per-op connect vs. work timing plus a live
  connect-per-op connection gauge (issue #125's provider-ceiling context).
- fetchBodyMarkingSeen: select/body/flag phase timings plus PII-free size
  counts (RFC822 size, body chars, attachment count).
- MailRepositoryImpl.openMessage: end-to-end open latency plus the
  cached-vs-fetched branch, keyed by accountLogRef and logSafeFolderLabel.
- ReaderViewModel: spinner-to-ready latency, split success vs. failure.

All breadcrumbs are PII-free: accounts are logged via the existing
accountLogRef hash, folders via the existing logSafeFolderLabel allowlist,
and everything else is sizes/durations/booleans only.

Fixes the 4 unit-test classes that exercise this code without mocking
android.util.Log (a throwing stub under plain JVM tests): mockkStatic(Log)
is now installed in MailRepositoryImplCoverageTest, ImapClientBackfillTest,
ImapFolderOpenLatencyTest, and ReaderViewModelActionsTest, following the
existing MailBackfillerTest/ImapClientTest conventions. detekt.yml gains two
more ForbiddenImport excludes for the newly Log-importing test files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 17:46:48 -05:00
Jason Ross f93e2dc2f0 Merge pull request #353 from JMR-dev/ci-extract-traffic-control
ci: extract traffic-control into its own workflow file
2026-07-05 16:17:10 -05:00
JMR-devandClaude Opus 4.8 939906986b ci: extract traffic-control into its own workflow file
Move the traffic-control (runner-priority orchestration) job verbatim out of
.github/workflows/ci.yml into a new standalone workflow,
.github/workflows/traffic-control.yml, so the heavy CI jobs no longer depend
on it. The job's YAML (name, runs-on, timeout-minutes, permissions, env,
steps) and its documentation comment move unchanged; the decision core
.github/scripts/traffic_control.py is untouched and still unit-tested by the
traffic-control-tests job in ci.yml.

In ci.yml: removed the traffic-control job, dropped needs: traffic-control
from the five heavy jobs (static-analysis, debug-build, unit-tests, e2e,
e2e-preview) and from traffic-control-tests (its only needs, which would
otherwise dangle at a now-deleted job), and updated the now-stale header and
ci-passed comments to point at the extracted workflow.

The new workflow will be disabled pending a rebuild as a published GitHub
Action.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 14:48:03 -05:00
Jason Ross 6118b6ded0 Merge pull request #285 from JMR-dev/perf-187-covering-index
perf(db): covering index for unified-inbox summary scans
2026-07-05 14:31:22 -05:00
github-actions[bot] 1cd0d66fa4 Merge main into perf-187-covering-index 2026-07-05 18:48:15 +00:00
Jason Ross c3933a3612 Merge pull request #352 from JMR-dev/ci-351-pat-triggering
ci: trigger CI with the PAT so dispatches don't need manual approval (#351)
2026-07-05 13:47:39 -05:00
JMR-devandClaude Opus 4.8 2fcee291ce ci: trigger CI with the PAT so dispatches don't need manual approval (#351)
#350 made ci-trigger.yml dispatch ci.yml with the built-in GITHUB_TOKEN, on the
claim that a workflow_dispatch is anti-recursion-exempt so no PAT is needed. In
practice a GITHUB_TOKEN-triggered run is held in `action_required` awaiting manual
approval and never runs un-attended, so auto-updated PRs' CI never ran (stalled
#285). The original #349 design was right: dispatch with a PAT so the run executes
as the authorized owner with no approval gate.

- ci-trigger.yml: the trigger step's GH_TOKEN is now
  `${{ secrets.AUTOUPDATE_TOKEN || github.token }}` (was `${{ github.token }}`).
  AUTOUPDATE_TOKEN (the PAT) is REQUIRED for the scheduler; the `|| github.token`
  fallback stays fail-open but only starts CI if repo settings don't gate
  GITHUB_TOKEN-triggered runs.
- autoupdate.yml: branch update stays on GITHUB_TOKEN (must NOT retrigger CI --
  that would re-introduce the cascade). Clarified that AUTOUPDATE_TOKEN is still
  required by the repo (by ci-trigger.yml) so the secret isn't deleted.
- Corrected the now-wrong "no PAT needed / workflow_dispatch anti-recursion-exempt"
  comments in ci-trigger.yml and the traffic_control.py docstrings.

updates = GITHUB_TOKEN, triggering = PAT.

Validation: all three workflow YAMLs parse clean; traffic-control unit tests still
pass (59 tests) -- the change is workflow-env only, script logic unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 13:28:10 -05:00
github-actions[bot] 33217c6cb2 Merge main into perf-187-covering-index 2026-07-05 18:10:50 +00:00
Jason Ross d8f6a66856 Merge pull request #317 from JMR-dev/fix-306-outlook-redundant-token
fix(auth): drop redundant second Outlook token request on sign-in
2026-07-05 13:10:19 -05:00
JMR-devandClaude Opus 4.8 46bc0e2258 Merge main into perf-187-covering-index
Resolve DatabaseModule conflict from #320: main replaced the explicit .addMigrations(...) chain with .addMigrations(*ALL_MIGRATIONS) plus an introspectable ALL_MIGRATIONS list guarded by databaseModuleRegistersEveryDeclaredMigration (registered == declared). Add MIGRATION_19_20 to ALL_MIGRATIONS so the unified-inbox covering-index migration (cache schema v19->v20) is both registered on the Room builder and satisfies that safety-net test. Schema 20.json, the v20 @Database version, and DatabaseEncryptionTest's schema-version assertion (20) are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 13:05:07 -05:00
github-actions[bot] 44d6d1edb6 Merge main into fix-306-outlook-redundant-token 2026-07-05 17:52:20 +00:00
Jason Ross 6f4275ee1e Merge pull request #350 from JMR-dev/ci-349-traffic-owns-triggering
ci: traffic-controller owns CI triggering (GITHUB_TOKEN updates, priority-ordered dispatch)
2026-07-05 12:51:48 -05:00
JMR-devandClaude Opus 4.8 05d06eb45b ci: traffic-controller owns CI triggering (GITHUB_TOKEN updates, priority-ordered dispatch)
End the merge cascade and give the traffic-controller ownership of CI *triggering*.

- autoupdate.yml updates PR branches with the built-in GITHUB_TOKEN instead of a PAT,
  so an update push no longer auto-retriggers CI (GitHub's anti-recursion rule) — the
  cascade (every merge re-runs every PR, cancel-in-progress thrashing them) is gone.
- New scheduler ci-trigger.yml -> traffic_control.py --mode trigger (re-)triggers CI
  for the highest-priority PR(s) whose head SHA has absent/stale checks, a few at a
  time (inflight cap), in the existing P0-P9 / broken-draft priority order — a
  poor-man's merge queue reusing the priority core. It runs after autoupdate finishes
  (workflow_run, race-free) plus a cron backstop plus manual dispatch.
- Triggering uses workflow_dispatch, which is EXEMPT from anti-recursion, so the
  built-in GITHUB_TOKEN (actions: write) starts the run — NO PAT / secret change needed.
- ci.yml gains a workflow_dispatch trigger (pr/head_sha/reason inputs) and a per-PR
  concurrency group unifying pull_request and dispatch runs; its on: pull_request path
  is kept so brand-new PRs, human pushes, and fork PRs always get CI (fail-open).

Pure select_triggers / classify_sha_runs decision core added to traffic_control.py with
24 new unit tests (priority order, oldest-first fairness, inflight cap, fork skip, P0
bypass+preempt, head-SHA needy classification, and a liveness/anti-starvation simulation).

Closes #349

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 01:27:40 -05:00
Jason Ross 4bf6fa5535 Merge main into fix-306-outlook-redundant-token 2026-07-05 00:55:24 -05:00
Jason Ross 833dfc030a Merge pull request #339 from JMR-dev/chore-claudemd-dod-logging
chore(docs): require app logging in the definition of done
2026-07-05 00:54:55 -05:00
Jason Ross 11e0a85475 Merge main into fix-306-outlook-redundant-token 2026-07-05 00:37:36 -05:00
Jason Ross d795639a32 Merge main into chore-claudemd-dod-logging 2026-07-05 00:37:35 -05:00
Jason Ross 94a7562256 Merge pull request #344 from JMR-dev/docs-343-workflows-readme
docs: plain-English README for the CI traffic-controller
2026-07-05 00:37:08 -05:00
Jason Ross 7d60c7ac60 Merge main into docs-343-workflows-readme 2026-07-05 00:05:37 -05:00
Jason Ross 88b8f6cb07 Merge main into fix-306-outlook-redundant-token 2026-07-05 00:05:35 -05:00
Jason Ross dd9f6bb8d9 Merge main into chore-claudemd-dod-logging 2026-07-05 00:05:34 -05:00
Jason Ross 348d7adc28 Merge pull request #348 from JMR-dev/feat-331-detekt-log-guard
feat(logging): detekt guard banning android.util.Log outside AppLog (#331)
2026-07-05 00:05:06 -05:00
Jason Ross 45df6c7709 Merge main into chore-claudemd-dod-logging 2026-07-04 23:16:54 -05:00
Jason Ross cefd92864c Merge main into fix-306-outlook-redundant-token 2026-07-04 23:16:53 -05:00
Jason Ross b3ec24d9f3 Merge main into docs-343-workflows-readme 2026-07-04 23:16:52 -05:00
Jason Ross 5d50b60f68 Merge main into feat-331-detekt-log-guard 2026-07-04 23:16:52 -05:00
Jason Ross 5047e3eb2a Merge pull request #340 from JMR-dev/feat-326-logging-authlock
feat(logging): auth/lock -> AppLog + breadcrumbs (#326)
2026-07-04 23:16:23 -05:00
Jason Ross 494649b7d8 Merge main into feat-331-detekt-log-guard 2026-07-04 23:05:20 -05:00
JMR-devandClaude Opus 4.8 faab0e3260 feat(logging): detekt guard banning android.util.Log outside AppLog (#331)
Add a detekt style>ForbiddenImport rule that forbids `import android.util.Log`
so all logging flows through org.libremail.reporting.AppLog, which mirrors each
line into the debug-report RingLogBuffer. A raw android.util.Log import writes to
Logcat only and never reaches a user-reviewed DebugReport (epic #324, strangler
final step).

Excludes the AppLog facade itself (the one sanctioned wrapper) and the unit tests
that mockkStatic(Log) to verify forwarding — AppLog forwards to Log, a throwing
stub under plain JVM unit tests, so those tests must mock it; they do not bypass
the facade.

Closes #331
Part of #324

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 23:02:13 -05:00
Jason Ross a9f0c220da Merge main into docs-343-workflows-readme 2026-07-04 22:59:33 -05:00
Jason Ross a44f568f9c Merge main into feat-326-logging-authlock 2026-07-04 22:59:32 -05:00
Jason Ross c65fb26ad0 Merge main into chore-claudemd-dod-logging 2026-07-04 22:59:31 -05:00
Jason Ross 17d40119ca Merge main into fix-306-outlook-redundant-token 2026-07-04 22:59:31 -05:00
Jason Ross d0ec1949a2 Merge pull request #347 from JMR-dev/ci-346-traffic-control-tests
ci: run traffic-controller unit tests as a gate job
2026-07-04 22:59:01 -05:00
Jason Ross b2c017b52d Merge main into fix-306-outlook-redundant-token 2026-07-04 22:38:21 -05:00
Jason Ross 02153b5287 Merge main into chore-claudemd-dod-logging 2026-07-04 22:38:20 -05:00
Jason Ross ef16b1ef2c Merge main into feat-326-logging-authlock 2026-07-04 22:38:19 -05:00
Jason Ross 76caaddd33 Merge main into docs-343-workflows-readme 2026-07-04 22:38:18 -05:00
JMR-devandClaude Opus 4.8 8b89219734 ci: run traffic-controller unit tests as a gate job
Adds a fast traffic-control-tests job (ubuntu, actions/checkout +
actions/setup-python, no emulator/Gradle) that runs the 37 pure-stdlib
unit tests for .github/scripts/traffic_control.py on every PR, and
wires it into ci-passed's needs so a regression blocks merge instead
of only being caught locally.

Closes #346

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 22:35:39 -05:00
Jason Ross dc21411aed Merge pull request #345 from JMR-dev/ci-342-traffic-control-python
ci: extract traffic-controller into a testable Python module (#342)
2026-07-04 22:33:37 -05:00
Jason Ross a13bc9eb07 Merge main into docs-343-workflows-readme 2026-07-04 22:29:35 -05:00
Jason Ross 37bdaae304 Merge main into feat-326-logging-authlock 2026-07-04 22:29:34 -05:00
Jason Ross 527e31fb5c Merge main into chore-claudemd-dod-logging 2026-07-04 22:29:33 -05:00
Jason Ross 28745827c2 Merge main into fix-306-outlook-redundant-token 2026-07-04 22:29:32 -05:00
Jason Ross 647490c1a8 Merge main into ci-342-traffic-control-python 2026-07-04 22:29:32 -05:00
Jason Ross 785c6aa6c6 Merge pull request #338 from JMR-dev/feat-328-logging-connsend
feat(logging): connectivity/send → AppLog + breadcrumbs, scrub email PII (#328)
2026-07-04 22:29:04 -05:00
JMR-devandClaude Opus 4.8 a34db54b97 fix(ci): let any strictly-higher PR reclaim a broken/draft run (#342)
Restore (and extend to drafts) the old bash's broken-reclaim behaviour that the
initial Python refactor had dropped. runs_to_cancel now cancels an OTHER PR's
active/queued runs when EITHER:
  (a) THIS PR is P0 and that PR is strictly-lower (reclaim every lower runner); OR
  (b) that PR is broken/draft (effective priority 10) and THIS PR is strictly-higher
      (effective priority < 10) — a wasted run any ready PR may reclaim.

P1-P9 still never bump a *normal* (non-broken/draft) lower run; a broken/draft PR
(P10) preempts nothing (nothing is strictly-lower than the bottom, and the
equal-or-higher invariant means a P10 never cancels another P10). Self / main-push /
equal-or-higher invariants unchanged.

Updates the module docstring + ci.yml comments (the "only P0 preempts" wording
becomes: P0 preempts everything strictly-lower; additionally, any strictly-higher PR
preempts a broken/draft run) and the job step/permission/needs comments. Adds unit
tests: P3 reclaims a broken P10 run and a draft P10 run; P3 does not bump a normal P5
run; a P10 self preempts nothing; plus an end-to-end P5-reclaims-draft-then-waits
scenario. 37 unit tests pass; ci.yml parses clean; --dry-run shows a P3 cancelling a
draft (and broken) run while still yielding to a higher P1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 22:19:01 -05:00
Jason Ross 14eebcb6c1 Merge main into fix-306-outlook-redundant-token 2026-07-04 22:09:43 -05:00
Jason Ross cbf159160f Merge main into feat-328-logging-connsend 2026-07-04 22:09:42 -05:00
Jason Ross d51778379a Merge main into chore-claudemd-dod-logging 2026-07-04 22:09:41 -05:00
Jason Ross 30026c1b29 Merge main into feat-326-logging-authlock 2026-07-04 22:09:40 -05:00
Jason Ross 6055fdc44d Merge main into docs-343-workflows-readme 2026-07-04 22:09:39 -05:00
Jason Ross ab41d8f24b Merge main into ci-342-traffic-control-python 2026-07-04 22:09:15 -05:00
Jason Ross c0dc5c5e75 Merge pull request #336 from JMR-dev/feat-330-logging-stragglers
feat(logging): stragglers -> AppLog (#330)
2026-07-04 22:09:07 -05:00
JMR-devandClaude Opus 4.8 8b7f895d6a ci: extract traffic-controller into a testable Python module (#342)
The priority-based runner orchestration ("traffic-control") lived as a large
inline-bash step in ci.yml — a two-pass preemption + hold-back script that was
effectively untestable in YAML. Move it into .github/scripts/traffic_control.py,
structured as a pure decision CORE + a thin gh-I/O SHELL:

* Pure functions (no network/clock/subprocess), unit-testable in isolation:
  - effective_priority(pr): lowest-numbered P0-P9, default P5; broken OR draft => 10.
  - runs_to_cancel(this_pr, all_prs, self_run_id): PASS 1 — run ids to cancel,
    empty unless THIS PR is P0; only strictly-lower running/queued runs; never self
    (by number or run id), never equal-or-higher.
  - wait_blockers(this_pr, all_prs): PASS 2 — yield to any strictly-higher PR with
    an active/queued run, and to same-level peers ordered ahead (running-first,
    then oldest createdAt). Empty => proceed.
* Shell (run_live): gathers the snapshot via gh, applies cancels, runs the bounded
  hold-back poll loop; always exits 0. --dry-run feeds the core a snapshot JSON and
  prints decisions with zero network.
* No jq/bash dependency (cross-platform, per the repo's Python-stdlib convention).

ci.yml's traffic-control job now checks out the repo and runs the module. Job
permissions gain `contents: read` (for checkout) alongside the existing
`actions: write` / `pull-requests: read`; env and downstream `needs:` wiring
unchanged; step stays `continue-on-error`.

33 stdlib unittest cases cover priority resolution, P0-only preemption, the
self/main/equal-or-higher invariants, and the same-level running-first/oldest
ordering.

Behaviour is preserved except the ticket's refinements: (1) drafts now count as
P10 (bottom); (2) an explicit same-level running-first-then-oldest tiebreaker; and
(3) per the ticket's order-of-operations, ONLY P0 preempts — the old bash also let
any higher-priority PR cancel a `broken` target's run, which no longer happens
(a broken/draft run is only cancelled by a P0, via the same strictly-lower rule).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 22:08:13 -05:00
JMR-devandClaude Opus 4.8 a24435a4fd docs(ci): add plain-English README for the CI traffic-controller
Explains the traffic-control job in ci.yml (priority labels, preemption
vs. bounded hold-back, safety invariants, and the known FIFO-runner
limitation) for developers new to the repo. Notes that #342 will refactor
this logic into a tested Python module, at which point this doc gets
updated.

Closes #343

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 22:06:09 -05:00
Jason Ross fc38255be5 Merge main into feat-326-logging-authlock 2026-07-04 21:52:01 -05:00
Jason Ross 6f20a23ae9 Merge main into chore-claudemd-dod-logging 2026-07-04 21:51:59 -05:00
Jason Ross 7e0b5c2b1c Merge main into feat-328-logging-connsend 2026-07-04 21:51:58 -05:00
Jason Ross 793346ab46 Merge main into feat-330-logging-stragglers 2026-07-04 21:51:57 -05:00
Jason Ross fff8afb453 Merge main into fix-306-outlook-redundant-token 2026-07-04 21:51:56 -05:00
Jason Ross b912af0ee2 Merge pull request #341 from JMR-dev/feat-329-logging-sync
feat(logging): sync-engine breadcrumbs via AppLog (#329)
2026-07-04 21:51:26 -05:00
Jason Ross 03179cda15 Merge main into fix-306-outlook-redundant-token 2026-07-04 21:34:12 -05:00
Jason Ross 8cc792c06e Merge main into feat-330-logging-stragglers 2026-07-04 21:34:11 -05:00
Jason Ross 59f86016cb Merge main into feat-328-logging-connsend 2026-07-04 21:34:10 -05:00
Jason Ross a891be4b14 Merge main into chore-claudemd-dod-logging 2026-07-04 21:34:09 -05:00
Jason Ross ca4c310a66 Merge main into feat-326-logging-authlock 2026-07-04 21:34:08 -05:00
Jason Ross 3d4dde6bc5 Merge main into feat-329-logging-sync 2026-07-04 21:34:07 -05:00
Jason Ross ff49c6c410 Merge pull request #337 from JMR-dev/feat-327-logging-dbkeystore
feat(logging): DB/keystore -> AppLog + breadcrumbs (#327)
2026-07-04 21:33:41 -05:00
Jason Ross 209371c9a5 Merge main into feat-329-logging-sync 2026-07-04 21:32:36 -05:00
JMR-devandClaude Opus 4.8 59b4252ce8 feat(logging): sync-engine breadcrumbs via AppLog (#329)
The sync engine (MailSyncer, MailBackfiller, MailPruner, and their WorkManager
workers) was completely silent, so a submitted debug report showed nothing
about whether sync ran, how much it fetched, or why it was skipped. Add
net-new AppLog breadcrumbs at each class's lifecycle points per the #324
strangler-migration plan: sync start/done/failed and per-folder fetch counts,
backfill slice start/done and per-folder page counts, prune's removed count,
and each worker's cache-locked deferral and success/retry outcome (the retry
path now also carries the scrubbed failure throwable via AppLog's #325
overloads).

Every breadcrumb is PII-safe by construction: accounts are identified only via
accountLogRef(account.id) (never the id or email directly), and a new
logSafeFolderLabel() helper logs a folder's name only when it matches a fixed
allowlist of known system folders (INBOX, Sent, Drafts, Trash, Spam/Junk,
Archive, and their common provider variants) — every other folder, however
nested or named, logs as a fixed placeholder.

Adding logging to these previously-silent classes meant every existing test
exercising them now hits android.util.Log (a throwing stub under plain JVM
unit tests), so each affected suite gains the same static Log mock already
established by AppLogTest/SendWorkerTest/ImapClientTest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 21:31:46 -05:00
Jason Ross d6e886bde0 Merge main into feat-326-logging-authlock 2026-07-04 21:28:28 -05:00
JMR-devandClaude Opus 4.8 0236494ecb chore(docs): require app logging in the definition of done
Codify the maintainer rule that app source-code changes must add
appropriate, PII-free logging via the AppLog facade at key points
(lifecycle transitions, error/fallback paths, state changes) so
behaviour is diagnosable from a user's debug report.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 21:27:39 -05:00
JMR-devandClaude Opus 4.8 6e0fe14b06 feat(logging): auth/lock -> AppLog + breadcrumbs (#326)
Migrate AppLockViewModel (9 sites) and AccountSetupViewModel (1 site) off
raw android.util.Log onto the AppLog seam (#325), so their diagnostic
lines land in the RingLogBuffer and reach a submitted DebugReport instead
of only Logcat. Adds three new breadcrumbs that were previously silent:
auth-seal unlock success, the clear-cache-and-restart recovery trigger
(with the disableAppLock flag), and every onForeground LockAction
decision. AccountSetupViewModel's success path also now logs "Outlook
account added" (no email). None of these call sites carry PII; where a
throwable is attached, AppLog's StackTraceScrubber redacts it before it
reaches the buffer.

Both ViewModel test suites now install a real RingLogBuffer and assert
against it instead of `verify { Log... }`, including dedicated no-PII
assertions (a known test email never appears in a recorded line). A
minimal `mockkStatic(Log::class)` stub stays in both test files' shared
setUp — AppLog still forwards to the real android.util.Log internally,
which throws "not mocked" in JVM unit tests when uninvoked; the not-yet
-landed guard-rule ticket (#331) will need to reconcile that with a
repo-wide "no raw Log outside AppLog.kt" rule.

Closes #326
Part of #324

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 21:27:30 -05:00
Jason Ross c149411a9d Merge main into feat-328-logging-connsend 2026-07-04 21:23:29 -05:00
JMR-devandClaude Opus 4.8 21c74df794 feat(logging): connectivity/send -> AppLog + breadcrumbs, scrub email PII
Migrate ImapClient/SendWorker/IdleService off raw android.util.Log to AppLog,
per the debug-logging strangler epic (#324), so their diagnostics reach the
RingLogBuffer (and a submitted DebugReport) instead of logcat-only:

- ImapClient: IDLE connect + IDLE push (message count) breadcrumbs.
- SendWorker: outbox-drain count on entry, per-message sent/failed result,
  and the existing Graph->SMTP fallback warning.
- IdleService: IDLE watch start, cache-locked defer, and the existing
  IDLE-dropped/retrying warning.

Also closes #297: SendWorker and IdleService logged the raw account.email via
Log.w on the Graph->SMTP fallback and IDLE-drop paths. Both now log
accountLogRef(account.id) instead -- a short, stable, non-reversible
per-account reference -- so the account's email never reaches Logcat or a
report.

Rewrites ImapClientTest/SendWorkerTest to install a real RingLogBuffer via
AppLog.install(...) and assert on its contents (migrated calls + new
breadcrumbs), instead of verifying a mocked Log; every assertion also checks
no line carries the test account's email, regression-covering #297. Adds a
SendWorkerTest case that drives a real SmtpSender against an in-process
GreenMail SMTP server end to end. android.util.Log is still stubbed (by
fully-qualified name, without importing it) where AppLog's Logcat passthrough
would otherwise crash the unmocked Android stub in a JVM test.

Closes #328
Closes #297
Part of #324

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 21:22:37 -05:00
Jason Ross f642f2c2b5 Merge main into fix-306-outlook-redundant-token 2026-07-04 21:15:54 -05:00
Jason Ross 73c636704f Merge main into feat-330-logging-stragglers 2026-07-04 21:15:53 -05:00
Jason Ross eee1475a61 Merge main into feat-327-logging-dbkeystore 2026-07-04 21:15:53 -05:00
Jason Ross a2f96b930d Merge pull request #335 from JMR-dev/ci-334-api37-boot-diagnostics
ci(e2e): boot diagnostics + verbose/debug logging on the API 37 preview job
2026-07-04 21:15:24 -05:00
JMR-devandClaude Opus 4.8 0cb9bb2906 refactor(data): route DB/keystore logging through AppLog (#327)
Migrates the DB/keystore area's raw android.util.Log calls to AppLog so
key-invalidation and DB-conversion breadcrumbs land in the process
RingLogBuffer (and thus a user-reviewed debug report) even in release
builds, where Log.d is otherwise stripped from Logcat only.

- DatabaseKeyCipher: 4 auth-bound-key decision points (encrypt retry,
  isInvalidated's three branches) now log via AppLog.d(tag, msg, e).
- DatabaseEncryption.migrate: adds an AppLog.i "converting local cache
  database (targetEncrypted=...)" breadcrumb at the start, alongside the
  existing "converted" completion line now routed through AppLog.d.
- AccountDataMigrator: the "moved account tables into the account
  database: $present" breadcrumb (table names only) now routed through
  AppLog.d.

No PII or key material is logged; table-name sets and boolean flags only.

Adds instrumented tests (DatabaseKeyCipher is device-only and
behavior-preserving, so no new test there) asserting the breadcrumbs
land in a RingLogBuffer and never contain the seeded email, secret, or
passphrase.

Part of #324.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 21:12:59 -05:00
JMR-devandClaude Opus 4.8 9e771a9590 ci(e2e): boot diagnostics + verbose/debug logging on the API 37 preview job
The API 37 preview E2E boot has flaked twice (#285, #333) with only
"did not boot within 300s" and no root-cause signal. Add rich boot
diagnostics by default, kept in parity between CI and the local
hand-provisioning script (api37_e2e.py):

- Launch the emulator with `-verbose -debug init,avd_config,kernel`
  (diagnostics only; no boot-affecting flag changed), still redirecting
  to $EMU_LOG.
- Stream `adb logcat -v time` to a file from the moment the device
  registers (via `adb wait-for-device logcat`, backgrounded).
- On a boot timeout, dump accel-check, /dev/kvm presence, GPU mode,
  free mem/disk, the AVD config.ini and the emulator.log tail; CI writes
  these to a boot-diagnostics file, the local script prints them.
- CI uploads emulator.log + logcat.txt + boot-diagnostics.txt as an
  artifact with `if: always()` so they survive a timeout/cancel, and
  prints a concise summary (accel/KVM status + last 50 lines of
  emulator.log) to the step log.

The existing 2-attempt boot retry + boot-completed wait loop are
unchanged; the diagnostics are additive.

Closes #334

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 21:09:28 -05:00
JMR-devandClaude Opus 4.8 d92d4c1a14 feat(logging): stragglers -> AppLog (#330)
Migrate RestartActivity's one raw Log.w site to AppLog, clearing the
final raw android.util.Log site outside the auth/lock, DB/keystore,
connectivity/send, and sync-engine migration areas so the codebase is
ready for the detekt android.util.Log guard (#331).

RestartActivity runs in the separate :restart trampoline process,
where LibreMailApplication.onCreate returns early and never calls
AppLog.install, so this breadcrumb reaches Logcat only, never a
DebugReport. The migration is guard-compliance + Logcat-consistency
only; behavior is unchanged since AppLog forwards to Logcat.

RestartActivity is DEVICE-ONLY (multi-process kill/relaunch), so a
JVM buffer-capture test doesn't apply here. Added
RestartActivityLoggingTest, which instead pins the null-buffer shape
this call runs under in the trampoline process: it forwards to
Logcat and no-ops the buffer cleanly.

Closes #330
Part of #324

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 21:09:22 -05:00
Jason Ross 6893649f0d Merge main into fix-306-outlook-redundant-token 2026-07-04 21:02:12 -05:00
Jason Ross 41015a5c40 Merge pull request #333 from JMR-dev/feat-325-applog-seam
feat(logging): AppLog seam — record scrubbed throwables + accountLogRef (#325)
2026-07-04 21:01:39 -05:00
Jason Ross 9ea339436d Merge main into feat-325-applog-seam 2026-07-04 20:24:29 -05:00
JMR-devandClaude Opus 4.8 1a1fbf8d7f feat(logging): AppLog seam — record scrubbed throwables + accountLogRef (#325)
Add throwable-recording overloads to AppLog.d/w and make AppLog.e record the
throwable it is given: the throwable's stack trace is scrubbed via the existing
StackTraceScrubber (exception class names + frames kept; host/email-bearing
exception messages stripped) and appended to the buffered log line, so a
throwable can reach a user-reviewed DebugReport without leaking PII. The
existing no-throwable overloads are unchanged.

Add accountLogRef(accountId): a short, stable, non-reversible reference
(scheme prefix + truncated SHA-256 of the id) so downstream logging can
identify an account without logging the raw Account.id, which embeds the email.

Foundation for the #324 debug-logging strangler epic; consumed by #326–#330.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 20:23:38 -05:00
Jason Ross 0d9761dda2 Merge main into fix-306-outlook-redundant-token 2026-07-04 20:17:19 -05:00
Jason Ross b352e3838c Merge pull request #332 from JMR-dev/feat-281-python-dev-scripts
test-infra: port local_instrumented.sh to Python + rewire preflight off GMD (#281)
2026-07-04 20:16:51 -05:00
JMR-devandClaude Opus 4.8 f54e9c67fa test-infra: port local_instrumented.sh to Python + rewire preflight off GMD (#281)
Port the last bash dev-script (local_instrumented.sh) to a cross-platform,
stdlib-only Python 3 script (local_instrumented.py), matching api37_e2e.py's
style, and rewire the /preflight skill's local E2E off the GMD
apiXXDebugAndroidTest tasks (which fail locally under AEHD 2.2) onto it.

- local_instrumented.py preserves the .sh's behavior exactly: comma-separated
  test-class CLI arg, pre-boot orphan-kill, manual cold-boot of the dev36 AVD
  (no GMD, no snapshot), targeted connectedDebugAndroidTest, and the EXIT-trap
  teardown (now try/finally + atexit + SIGINT/SIGTERM handlers, idempotent).
  Exit codes 0/2/3/4 preserved.
- Cross-platform process kill abstracted per-OS: taskkill /F /IM on Windows,
  pkill -f qemu-system on *nix; process listing via tasklist / ps ax.
- Teardown hardened vs the .sh: it now also reaps the emulator *launcher*
  image, not just qemu -- the Windows -no-window emulator spawns a sibling
  emulator.exe that briefly outlives the qemu VM, which a qemu-only sweep left
  as an orphan on return (caught by the smoke run).
- SKILL.md + CLAUDE.md: replace the local api35/api36 GMD E2E steps with
  local_instrumented.py; CI's own multi-API matrix is untouched. CLAUDE.md
  documents the Python-first dev-script convention.

Validated: py_compile, argparse (--help / no-arg exit 2), and a guarded
emulator smoke run of org.libremail.data.local.DatabaseEncryptionTest -- boots,
passes, and tears down clean (no qemu/emulator orphan on return).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 19:58:14 -05:00
Jason Ross be2afd42c4 Merge main into fix-306-outlook-redundant-token 2026-07-04 11:46:19 -05:00
Jason Ross 47aaebccf5 Merge pull request #323 from JMR-dev/feat-251-292-coverage-noregression-gate
feat(ci): no-regression JVM coverage gate scoped to the testable surface
2026-07-04 11:45:47 -05:00
JMR-devandClaude Opus 4.8 2e0eaef4e9 feat(ci): no-regression JVM coverage gate scoped to the testable surface
Scope :app:jacocoTestReport's denominator to the JVM-testable surface and
add a :app:jacocoTestCoverageVerification no-regression gate that shares the
same classDirectories/executionData/sourceDirectories, wired into both the
`check` lifecycle task and CI's unit-test job (part of the `CI passed` gate).

Excluded from the denominator (structurally unreachable from a JVM unit
test): Compose screen/component render code, Android framework entry points
(*Activity/*Service/Application/*BackupAgent), Hilt DI (**/di/**), and the
src/debug cold-open probe. Kept in scope: ViewModels, repositories, mappers,
DAOs, utils, richtext, mail, reporting logic, and the six WorkManager Workers.

Corrects PR #292, which excluded **/*Worker*: SyncWorker, BackfillWorker,
PruneWorker, SendWorker, ReportPurgeWorker and ReportUploadWorker are all
directly unit-tested, so they stay counted in both numerator and denominator
(only their Hilt wiring, WorkManagerModule, is excluded, via **/di/**).

Baseline: 80.21% line (4838/6032). Floor: 0.79 (~1.2% headroom) so ordinary
noise doesn't red-flag it while a real drop fails. Manual ratchet for now:
bump the floor up in the same PR when coverage rises materially.

Closes #251
Closes #292

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 11:27:17 -05:00
Jason Ross cc1f4442b6 Merge main into fix-306-outlook-redundant-token 2026-07-04 04:00:45 -05:00
Jason Ross 0e451fc80e Merge pull request #316 from JMR-dev/fix-reporting-pii-mainthread
fix(reporting): scrub PII from crash stack traces + move ReportStore scan off the main thread
2026-07-04 04:00:17 -05:00
JMR-devandClaude Opus 4.8 e4457cfec1 test(db): expect schema v20 in encryption round-trip after v19->v20 bump
MIGRATION_19_20 (issue #187) bumped the Room cache schema to version 20,
but DatabaseEncryptionTest.schemaVersionIsCarriedOntoTheEncryptedFile still
asserted the plaintext -> encrypted conversion carried version 19, so it
failed across all E2E levels after the rebase onto main.

DatabaseEncryption.migrate() carries PRAGMA user_version dynamically
(userVersion = source.version -> target.version = userVersion), and a fresh
Room open now stamps 20, so v20 genuinely survives the conversion. Update the
expected constant to 20; the assertion's intent (the version survives the
round-trip) is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 03:53:46 -05:00
Jason Ross 2e04934e19 Merge main into fix-306-outlook-redundant-token 2026-07-04 03:43:43 -05:00
Jason Ross 7bd909882f Merge main into fix-reporting-pii-mainthread 2026-07-04 03:43:43 -05:00
Jason Ross 9110915629 Merge pull request #321 from JMR-dev/fix-303-304-ui-correctness
fix(ui): reader Reply quotes original + guard one-shot actions against double-tap
2026-07-04 03:43:14 -05:00
Jason Ross def89c4e2a Merge main into fix-reporting-pii-mainthread 2026-07-04 03:21:30 -05:00
Jason Ross 8cb57cba50 Merge main into fix-306-outlook-redundant-token 2026-07-04 03:21:29 -05:00
Jason Ross 1649154ac4 Merge main into fix-303-304-ui-correctness 2026-07-04 03:21:27 -05:00
Jason Ross 5125a6674c Merge pull request #320 from JMR-dev/fix-310-311-312-dao
perf/fix(data): batch sync updates, paging tiebreaker, migration-registration test
2026-07-04 03:20:59 -05:00
Jason Ross c3d335a753 Merge main into fix-303-304-ui-correctness 2026-07-04 03:01:24 -05:00
Jason Ross 42b8bcc91e Merge main into fix-310-311-312-dao 2026-07-04 03:01:22 -05:00
Jason Ross 2289f402e5 Merge main into perf-187-covering-index 2026-07-04 03:01:21 -05:00
Jason Ross 97b782ecb2 Merge main into fix-306-outlook-redundant-token 2026-07-04 03:01:19 -05:00
Jason Ross 61dc35e290 Merge main into fix-reporting-pii-mainthread 2026-07-04 03:01:17 -05:00
Jason Ross cbf284b5f5 Merge pull request #315 from JMR-dev/fix-account-lifecycle-integrity
fix(data): account-lifecycle data integrity (non-destructive upsert, id normalization, deleteAccount cleanup)
2026-07-04 03:00:44 -05:00
Jason Ross c645e0e1fe Merge main into fix-reporting-pii-mainthread 2026-07-04 02:45:04 -05:00
Jason Ross b33f73273d Merge main into fix-306-outlook-redundant-token 2026-07-04 02:45:03 -05:00
Jason Ross 61437e8670 Merge main into fix-account-lifecycle-integrity 2026-07-04 02:45:01 -05:00
Jason Ross 1d4bd6346c Merge main into perf-187-covering-index 2026-07-04 02:44:59 -05:00
Jason Ross 2b616df3e0 Merge main into fix-310-311-312-dao 2026-07-04 02:44:58 -05:00
Jason Ross f821e0274e Merge main into fix-303-304-ui-correctness 2026-07-04 02:44:57 -05:00
Jason Ross 3062a3a3d9 Merge pull request #318 from JMR-dev/fix-295-targeted-batch-expunge
fix(mail): targeted + batch expunge (stop deleting unrelated \Deleted mail)
2026-07-04 02:44:21 -05:00
Jason Ross 205f1f6de1 Merge main into fix-303-304-ui-correctness 2026-07-04 02:37:40 -05:00
JMR-devandClaude Opus 4.8 6245533368 fix(ui): reader Reply quotes original + guard one-shot actions against double-tap
#303: the reader's Reply now routes through MailRepository.buildReplyDraft
(quotes the original into a <blockquote>, bakes the signature, prefixes Re:/Fwd:
without double-prefixing) and opens compose on the built draft via
ReaderEvent.OpenCompose — the same high-fidelity path the mailbox uses — instead
of a bare compose prefill with an empty body. Adds Reply-All and Forward via an
app-bar overflow menu.

#304: SignatureEditViewModel.save, ReportReviewViewModel.submit,
AccountSettingsViewModel.removeAccount, and ProblemReportsViewModel.createManualReport
now flip a busy/saving flag synchronously before the first suspension and gate
their buttons, so a rapid double-tap can't create duplicate signatures/reports,
enqueue two uploads, or over-pop the back stack.

Closes #303
Closes #304

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:36:15 -05:00
Jason Ross c98eac6c5b Merge main into fix-310-311-312-dao 2026-07-04 02:32:20 -05:00
JMR-devandClaude Opus 4.8 edbb1eb839 perf/fix(data): batch sync updates, paging tiebreaker, migration-registration test
#310: add a single-transaction MessageDao.updateHeaderContents(List<MessageEntity>)
and route MailSyncer's per-message updateHeaderContent loop through it, so a folder's
recent-window refresh commits once instead of once per message (fsync/journal write
per message, amplified on the encrypted cache).

#311: append the `id` primary key as a tiebreaker to the four paged MessageDao
`ORDER BY timestampMillis DESC` queries for a total order, so rows sharing a second
(bulk mail) can't duplicate or skip across a LIMIT/OFFSET page boundary. Pure query-text
change: the exported Room schema (identityHash) is derived from table/index structure,
not @Query SQL, so no schema re-export or version bump; id is already in the projection,
so no new index.

#312: expose the registered migration list as DatabaseModule.ALL_MIGRATIONS (spread into
addMigrations) and assert in MigrationTest that it equals the reflectively-discovered set
of every Migration val, so a migration forgotten in addMigrations fails a test instead of
crash-looping all upgrading users at DB open (there is deliberately no destructive fallback).

Tests: new MessageDaoTest cases for the batch update and the id tiebreaker (all four
pagers), and the MigrationTest registration assertion; wired updateHeaderContents into
MailSyncConcurrencyTest's fake DAO. Instrumented MessageDaoTest + MigrationTest (31 tests)
green on a local emulator; unit tests + androidTest compile + ktlint + detekt green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:30:50 -05:00
Jason Ross df57dcd18d Merge main into perf-187-covering-index 2026-07-04 02:26:52 -05:00
Jason Ross 6810cc5390 Merge main into fix-account-lifecycle-integrity 2026-07-04 02:26:51 -05:00
Jason Ross 5e2d65d91a Merge main into fix-reporting-pii-mainthread 2026-07-04 02:26:50 -05:00
Jason Ross e80f8333f4 Merge main into fix-306-outlook-redundant-token 2026-07-04 02:26:49 -05:00
Jason Ross 6de885d53e Merge main into fix-295-targeted-batch-expunge 2026-07-04 02:26:48 -05:00
Jason Ross b95ae2e7eb Merge pull request #314 from JMR-dev/fix-302-idleservice-fgs-timeout
fix(push): handle Service.onTimeout to survive the dataSync FGS runtime cap
2026-07-04 02:26:16 -05:00
JMR-devandClaude Opus 4.8 f72c66d291 fix(mail): targeted + batch expunge (stop deleting unrelated \Deleted mail)
ImapClient.deleteMessage and moveMessages flagged the target \Deleted then
called the untargeted Folder.expunge(), which permanently removes EVERY
\Deleted-flagged message in the folder — not just the intended UIDs. That is a
data-loss window whenever a second client, Gmail, or a partial earlier move has
left other messages flagged \Deleted. The repository's batch delete/expunge and
trash-fallback paths also looped single-UID deleteMessage, paying N logins + N
expunges for an N-message selection.

Add a batch deleteMessages(uids) that opens the folder once, flags the matched
messages \Deleted, and issues a single targeted UID EXPUNGE (RFC 4315) via
IMAPFolder.expunge(Message[]) through a shared expungeTargeted() helper. Route
moveMessages through the same helper, delegate single-UID deleteMessage to
deleteMessages, and route MailRepositoryImpl.expunge and the moveByRole trash
fallback through the batch method. moveToFolder already batches via moveMessages,
so it inherits the targeted expunge.

On a server without UIDPLUS, Angus raises "UID EXPUNGE not supported" rather than
silently falling back to the unrelated-mail-destroying untargeted expunge — a
loud failure is the safe outcome. Gmail, Outlook, and GreenMail all advertise
UIDPLUS.

Tests (GreenMail, no emulator): deleteMessages/moveMessages expunge only the
given UIDs and spare other \Deleted-flagged mail; a batch delete of three
messages opens exactly one connection and pays one LOGIN.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:24:40 -05:00
JMR-devandClaude Opus 4.8 da2963aa86 fix(auth): drop redundant second Outlook token request on sign-in
exchangeToken's authorization-code exchange already requests
`openid email offline_access $OUTLOOK_SCOPE`, so the returned access token
is an outlook.office.com token usable for IMAP verification and the AuthState
already carries the refresh token and expiry. The immediate follow-up
refreshForScope(authState, OUTLOOK_SCOPE) was a second round-trip for the
same resource that only rotated the just-issued refresh token and added a
needless onboarding failure point (a transient network error there failed
sign-in after consent + code-exchange had already succeeded).

Build OAuthResult directly from the code-exchange tokenResponse
(accessToken + authState.jsonSerializeString()), dropping the extra refresh.
The durable AuthState is still serialized and persisted for later token
refresh; the Graph token remains a distinct resource minted on demand via
freshGraphToken.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:22:00 -05:00
Jason Ross 64f4c64936 Merge main into fix-reporting-pii-mainthread 2026-07-04 02:19:16 -05:00
JMR-devandClaude Opus 4.8 005f8a3aca fix(reporting): scrub PII from crash stack traces + move ReportStore scan off the main thread
A crash report captured throwable.stackTraceToString() verbatim, so mail/network
exceptions (Jakarta Mail, java.net) could embed server host:port tokens and account
emails/usernames in the report's stackTrace field — violating the PII-free-reports
constraint. Add StackTraceScrubber, applied in DiagnosticsCollector before the trace
enters toSubmissionPayload()/toStorageJson(): it keeps the non-PII value (exception
class names + every frame's class/method/file/line) and drops each header line's
free-text message (where hostnames/usernames live), then redacts any residual email
or host:port left on a wrapped continuation line. Frame lines are untouched, so a
frame's File.kt:42 is never mistaken for a host:port.

ReportStore did MutableStateFlow(scan()) in its constructor — a dir list + read +
JSON-parse of every stored report. As an eager @Singleton dep of CrashReporter, whose
install() runs on the MAIN thread in Application.onCreate(), this was main-thread disk
I/O that grows with the 30-day retention. Seed the flow empty and dispatch the initial
scan to an injectable scope (Dispatchers.IO by default); reactive consumers update when
it lands, and writes still re-scan synchronously so a crash-time save is never lost.

Tests: StackTraceScrubberTest (host/ip/port/email dropped from a ConnectException +
auth-failure trace while classes/frames survive; regex redaction of a continuation
line; null-message trace preserved verbatim); DiagnosticsCollector end-to-end scrub
test; ReportStore empty-seed + off-thread populate via a StandardTestDispatcher. Store
constructions in existing tests use an Unconfined scope to keep their synchronous
reopen semantics.

Closes #294
Closes #296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:18:10 -05:00
Jason Ross 659f6e1d74 Merge main into fix-account-lifecycle-integrity 2026-07-04 02:14:30 -05:00
JMR-devandClaude Opus 4.8 cae11233f3 fix(data): account-lifecycle data integrity (non-destructive upsert, id normalization, deleteAccount cleanup)
#309: AccountDao no longer uses @Insert(REPLACE). New insertIfAbsent (IGNORE)
+ @Update back a non-destructive upsert, and insertAtEnd updates an existing id
in place (preserving its sortOrder) instead of REPLACE. Re-adding an existing
account id (e.g. re-authing an Outlook account, whose id is the deterministic
outlook:<email>) therefore no longer cascade-deletes its account_settings +
signatures.

#305: normalizeEmailForAccountId always lowercases the domain (mail domains are
case-insensitive), and the whole address for the consumer providers (Gmail,
Yahoo, iCloud, AOL, Outlook). Applied at every id-derivation site
(MailProvider.createAccount, Account.outlook, ManualSetupViewModel) so
differently-cased addresses can't spawn duplicate accounts. The displayed email
keeps the user's casing.

#299: deleteAccount collects the account's message ids and draft attachment URIs
while the rows still exist, deletes the rows (now including the account's
drafts), then deleteRecursively()'s each message's on-disk attachment cache dir
and releases the drafts' now-unreferenced persistable URI grants.

Adds unit tests for id normalization + deleteAccount cleanup and DAO-level
instrumented tests for the non-destructive create/update.

Closes #309
Closes #305
Closes #299

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:13:21 -05:00
Jason Ross ce9e54f704 Merge main into fix-302-idleservice-fgs-timeout 2026-07-04 02:10:04 -05:00
JMR-devandClaude Opus 4.8 8904351a96 fix(push): handle Service.onTimeout to survive the dataSync FGS runtime cap
IdleService runs continuously as a FOREGROUND_SERVICE_TYPE_DATA_SYNC
foreground service (push is on by default). With targetSdk 37, Android 14+'s
dataSync FGS runtime cap (~6h per rolling 24h) calls Service.onTimeout(...)
and then force-stops the service — throwing a system FGS-timeout exception —
if it doesn't stop itself. IdleService overrode onStartCommand/onDestroy/onBind
but not onTimeout, so after ~6 cumulative hours push silently died and the app
hit the exception; on API 35+ the budget is cumulative and a restart can't
recover it until the next 24h window.

Override both onTimeout(startId) (deprecated, API 34) and
onTimeout(startId, fgsType) (API 35+); both route to a clean shutdown that
re-asserts the already-scheduled 15-minute periodic sync, swaps the persistent
notification to a degraded "paused" text and DETACHes it so it survives, then
stopForeground(DETACH) + stopSelf so we never leave a dataSync FGS running past
its cap (the exact condition the platform kills on). This mirrors the existing
low-battery PushMode.POLLING fallback.

The push-status text choice is pulled into a pure PushStatusNotification.statusTextRes
seam and unit-tested on the JVM; the built notification's new timed-out text is
covered by PushStatusNotificationInstrumentedTest.

Closes #302

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:09:01 -05:00
Jason Ross 8230dd0341 Merge main into perf-187-covering-index 2026-07-04 01:56:44 -05:00
Jason Ross b64c7f8dda Merge pull request #293 from JMR-dev/test-289-branch-coverage
test(coverage): high-value branch-coverage additions
2026-07-04 01:56:15 -05:00
Jason Ross 0e42358d27 Merge main into test-289-branch-coverage 2026-07-04 01:38:09 -05:00
JMR-devandClaude Opus 4.8 b165f72e3a test(coverage): high-value branch-coverage additions
Add focused JVM unit tests that close real (non-coroutine) branch gaps in
pure logic already >=95% line-covered (issue #289):

- richtext/RichTextEditing: removeLink, applyLink/styleAt/isStyled edges,
  quote/ordered marker detection+removal, remap* null branches.
- richtext/RichTextHtmlParser: new suite driving parseCssColor/parseFontSizePt/
  parseInlineStyles/parseBaseStyle/parseTextAlign/extractHref/unescape and the
  parser's malformed/stray/unclosed-tag edges.
- ui/compose/ComposeViewModel + ui/mailbox/MailboxViewModel: nav-arg blanks,
  signature-swap rebuild, autosave content detection, refresh/selection edges.
- data/repository/MailRepositoryImpl: non-selectable role folder, cancelOutboxMessage.
- data/repository/AccountRepositoryImpl: reorderAccounts.
- mail/HtmlToText, data/SignatureBlock, reporting/AppLog, reporting/DiagnosticsCollector.

Test-only; no production changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 01:36:31 -05:00
Jason Ross 5c34b01bf0 Merge main into perf-187-covering-index 2026-07-04 01:22:27 -05:00
Jason Ross 7966efde2f Merge pull request #291 from JMR-dev/test-288-coverage-gaps
test(coverage): close clean JVM-testable coverage gaps
2026-07-04 01:21:57 -05:00
Jason Ross 1af6ad3d52 Merge main into test-288-coverage-gaps 2026-07-04 01:05:26 -05:00
JMR-devandClaude Opus 4.8 a6c24f3cea test(coverage): close clean JVM-testable coverage gaps
Close the cleanly JVM-testable coverage gaps from the Phase-2 JaCoCo
audit (#288), bringing each targeted class to 100% line coverage:

- AccountSettingsRepository: observe Flow + the sibling setters
  (signature/notifications enabled, retention count/months incl. clamp).
- SignatureRepository: observeForAccount/get/getDefault/update + toDomain.
- CredentialStore (new): save/load/delete with a mocked KeystoreCrypto.
- EncryptedCacheGuard (new): the isCacheLocked() truth table.
- AttachmentUriGrants: releaseUnreferenced/referencedUris wiring.
- SyncScheduler: schedulePeriodicReportPurge.
- AppLockViewModel: nonce, onBackground cover branch, unwrap cancellation
  rethrow, awaitSyncEnqueue execution/interrupt branches.
- SmtpSender: send error paths (SSL/STARTTLS) + cc + inline+regular body.
- StartupReportViewModel: the @Inject real-clock constructor.

All test-only. Unit tests + jacocoTestReport + ktlintCheck + detekt green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 01:04:24 -05:00
Jason Ross f4a95a6051 Merge main into perf-187-covering-index 2026-07-04 00:59:28 -05:00
Jason Ross a8399228f6 Merge pull request #287 from JMR-dev/test-infra-284-helper-cwd
test-infra: make local_instrumented.sh CWD-independent
2026-07-04 00:58:57 -05:00
Jason Ross 991f9b77e4 Merge main into perf-187-covering-index 2026-07-04 00:38:29 -05:00
Jason Ross 6ff43c1035 Merge main into test-infra-284-helper-cwd 2026-07-04 00:38:28 -05:00
Jason Ross fdb45131de Merge pull request #286 from JMR-dev/chore-237-localclipboard-migration
chore(ui): migrate LocalClipboardManager to LocalClipboard
2026-07-04 00:37:57 -05:00
JMR-devandClaude Opus 4.8 0b67cb952a fix(test-infra): make local_instrumented.sh CWD-independent
The wrapper jar path was resolved from the script's own location, but
gradlew picks the *project* to build from the process's current
directory, not from its own script location. Invoking the helper from
a CWD outside its tree (e.g. another worktree) silently built the
wrong repo's :app, once observed as a ClassNotFoundException for a
test class that only existed in the intended worktree.

cd to the already-resolved repo/worktree root before invoking gradlew
so connectedDebugAndroidTest always targets the correct tree
regardless of the caller's CWD. Update the README's usage note to
match.

Closes #284

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 00:22:24 -05:00
Jason Ross bdb49f996f Merge main into perf-187-covering-index 2026-07-04 00:17:05 -05:00
Jason Ross b78921cd0b Merge main into chore-237-localclipboard-migration 2026-07-04 00:17:04 -05:00
Jason Ross 15f53633fa Merge pull request #283 from JMR-dev/test-257-reportupload-seams-e2e
test(reporting): ReportUpload testability seams + push IdleService E2E
2026-07-04 00:16:33 -05:00
JMR-devandClaude Opus 4.8 191e802eac chore(ui): migrate LocalClipboardManager to LocalClipboard
LocalClipboardManager/ClipboardManager are deprecated in Compose in favor
of LocalClipboard's suspend Clipboard API. Migrates the one call site,
ReportReviewScreen's "Copy report" action: LocalClipboardManager.current
becomes LocalClipboard.current, and the synchronous
clipboard.setText(AnnotatedString(...)) becomes a suspend
clipboard.setClipEntry(ClipEntry(ClipData.newPlainText(...))) run inside
the existing rememberCoroutineScope(). The clipboard interaction is
pulled into a small internal suspend function, copyReportPayloadToClipboard,
so it's unit-testable against a mocked Clipboard without an emulator.

Closes #237.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 00:16:32 -05:00
JMR-devandClaude Opus 4.8 5a3669f017 perf(db): covering index for unified-inbox summary scans
The paged "All inboxes" query (MessageDao.pagingUnifiedFolderSummaries:
WHERE folder = ? AND inInbox = 1 ORDER BY timestampMillis DESC) had no
folder-leading index, so it SCANned the whole messages table via
index_messages_timestampMillis and filtered folder/inInbox per row.

Add a (folder, inInbox, timestampMillis) index so the two equality
predicates become an index seek and the ORDER BY is supplied by the
index. EXPLAIN QUERY PLAN for the query goes from
  SCAN messages USING INDEX index_messages_timestampMillis
to
  SEARCH messages USING INDEX index_messages_folder_inInbox_timestampMillis (folder=? AND inInbox=?)
with no temp B-tree sort.

Pure additive index (no column/table change): bump the Room DB to v20
with MIGRATION_19_20 (CREATE INDEX IF NOT EXISTS), register it in
DatabaseModule, export 20.json, and add a MigrationTest that runs the
migration and asserts the index shape plus the SEARCH plan on real
Android SQLite.

Closes #187

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 00:14:39 -05:00
Jason Ross eda5103da2 Merge main into test-257-reportupload-seams-e2e 2026-07-03 23:58:56 -05:00
JMR-devandClaude Opus 4.8 0b44276909 test(reporting): ReportUpload testability seams + push IdleService E2E
Coverage lane 4 (#249) flagged reporting/push classes as unreachable by tests.
Add minimal, behaviour-preserving seams and the tests they unblock (issue #257):

- ReportUploadScheduler: inject Provider<WorkManager> (mirroring SyncScheduler)
  instead of calling the WorkManager.getInstance static that MockK can't stub on
  the abstract WorkManager (AbstractMethodError). New ReportUploadSchedulerTest
  pins the per-report unique-work name + REPLACE policy.
- ReportUploadWorker: take the ingest endpoint via a new @DebugReportEndpoint
  qualifier (provided from BuildConfig.DEBUG_REPORT_ENDPOINT in ReportingModule)
  rather than reading the BuildConfig static inline. New ReportUploadWorkerHttpTest
  drives the transmit path against an in-process JDK HttpServer on loopback and
  covers 2xx success + delete, 4xx failure, 5xx retry/attempt-cap, and network
  error. Production value is unchanged (empty by default).
- IdleService: extract the foreground-notification channel + push-mode-to-text
  logic into PushStatusNotification. New PushStatusNotificationInstrumentedTest
  asserts channel importance and the IDLE/POLLING notification text with a real
  application Context (never a mocked Context).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 23:58:00 -05:00
Jason Ross e587ba0d82 Merge pull request #282 from JMR-dev/test-221-cold-open-encrypted-cache
test(db): instrumented cold-open of a pre-encrypted cache
2026-07-03 23:47:44 -05:00
Jason Ross 24b9572179 Merge main into test-221-cold-open-encrypted-cache 2026-07-03 23:31:47 -05:00
JMR-devandClaude Opus 4.8 17d7135f59 test(db): instrumented cold-open of a pre-encrypted cache
Guards the SQLCipher cold-start crash fixed in 592a797 (bug #210): a cold
process opening an already-encrypted cache with nothing to convert reached
Room's keyed nativeOpen with the native .so unloaded and crash-looped with
UnsatisfiedLinkError. Every existing on-device test (DatabaseEncryptionTest,
DatabaseProvisionerInstrumentedTest, DatabaseModuleInstrumentedTest,
AccountDataMigratorTest) runs a conversion first, which loads the process-global
library in-process, masking the bug exactly as production did.

System.loadLibrary is process-global, so the instrumentation process can no
longer observe a cold open once it has minted the encrypted fixture. This adds
ColdOpenCacheProbe -- a debug-only ContentProvider declared with
android:process=":coldopen" -- to host the open in a separate, pristine app
process. The test mints the encrypted fixture in the instrumentation process
(a file created by a prior encrypted DB instance) and drives the cold open in
the :coldopen process via ContentResolver.call, mirroring DatabaseProvisioner's
encrypted branch + DatabaseModule's open lambda against the real DatabaseEncryption,
DeferredOpenHelperFactory and SupportOpenHelperFactory. A cold probe (a keyed open
with no preceding load, asserted to throw UnsatisfiedLinkError) makes the isolation
self-verifying: the test fails rather than passing hollow if the library was
already loaded in the harness process.

Verified locally on an API 36 emulator (connectedDebugAndroidTest): 1 test,
0 failures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 23:30:49 -05:00
Jason Ross 165699d950 Merge pull request #280 from JMR-dev/test-infra-269-local-instrumented-helper
test-infra: reliable local instrumented-test helper (connectedDebugAndroidTest + emulator hygiene)
2026-07-03 23:29:06 -05:00
Jason Ross fc78d81886 Merge main into test-infra-269-local-instrumented-helper 2026-07-03 23:10:33 -05:00
Jason Ross 6151fea4d1 Merge pull request #279 from JMR-dev/test-275-lane5-followup-screens
test(coverage): lane 5 follow-up — UI tests for remaining screens
2026-07-03 23:10:03 -05:00
Jason Ross 633b3a94e2 Merge main into test-infra-269-local-instrumented-helper 2026-07-03 22:56:08 -05:00
JMR-devandClaude Opus 4.8 e006600424 test-infra: reliable local instrumented-test helper (connectedDebugAndroidTest + emulator hygiene)
Local Gradle Managed Device tasks (apiXXDebugAndroidTest) fail on this machine:
GMD's AVD snapshot step times out under AEHD 2.2
(AvdSnapshotHandler$EmulatorSnapshotCannotCreatedException), though the emulator
itself boots fine. CI is unaffected (it uses connectedDebugAndroidTest, not GMD).

Add .claude/skills/preflight/local_instrumented.sh, which cold-boots ONE emulator
by hand (-no-snapshot, no GMD) and runs :app:connectedDebugAndroidTest filtered to
a targeted set of test classes -- the same technique CI and api37_e2e.py already use.

The helper is deliberately targeted (the full ~114-test suite tends to wedge mid-run
on this box) and enforces emulator hygiene: it force-kills stray qemu/emulator
processes before booting, tears the emulator down afterward, and exits non-zero if an
orphaned qemu-system-x86_64-headless.exe survives -- accumulated orphans have frozen
this machine. Ships with a documented header and a short sibling README.

Closes #269

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 22:54:37 -05:00
Jason Ross 6bde35507b Merge main into test-275-lane5-followup-screens 2026-07-03 22:53:16 -05:00
Jason Ross e57ba70fb1 Merge pull request #278 from JMR-dev/test-220-native-lib-before-keyed-open
test(db): pin native-lib-before-keyed-open at the DatabaseModule factory site
2026-07-03 22:52:46 -05:00
Jason Ross 46c4110b19 Merge main into test-275-lane5-followup-screens 2026-07-03 22:39:38 -05:00
JMR-devandClaude Opus 4.8 65ff45ec31 test(coverage): lane 5 follow-up — UI tests for remaining screens
Adds instrumented Compose UI tests for the screens #250/#274 left uncovered,
so lane 6's ui-package coverage ratchet (#251, >=95%) can pass:

- ColorSwatchRow (compose/format): none entry + swatch rendering, selection
  callbacks, and selected-state semantics.
- LockScreen: locked title/body, optional error text, unlock callback.
- AddAnotherAccountScreen: confirmation + both onboarding choices.
- SignatureEditScreen: real ViewModel over an in-memory Room-backed
  SignatureRepository — new-vs-edit title, create/update round-trips.
- ReportReviewScreen: real ViewModel over a file-backed ReportStore (submitter
  stubbed disabled) — disclaimer/fields render, Submit gated on comment length
  + email validity, discard deletes and leaves.
- AppPasswordSetupScreen: real ViewModel over FakeAccountRepository — provider
  chrome + credential add, and the app-password help link asserted via
  Espresso-Intents (mirrors AccountPickerScreenTest) so no real browser opens.

All 23 tests pass locally on an API 36 emulator.

Closes #275

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 22:38:47 -05:00
Jason Ross e4a27944bd Merge main into test-220-native-lib-before-keyed-open 2026-07-03 22:37:29 -05:00
Jason Ross 11490b82fe Merge pull request #273 from JMR-dev/fix-219-empty-state-flash
fix(mailbox): keep the inbox pager warm on reader return so it doesn't flash empty
2026-07-03 22:36:51 -05:00
JMR-devandClaude Opus 4.8 9ab7c22c48 fix(test): assert only the empty-state gate in MailboxScreenTest loading test
The compose FAB does not render reliably under a never-completing refresh==Loading pager (flaked as not-displayed, not-found, then waitForText-timeout across CI runs). Drop the positive FAB anchor; assert only mailbox_empty.assertDoesNotExist() — the actual #219 gate behavior, which is stable and idle-completes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 22:21:32 -05:00
JMR-devandClaude Opus 4.8 44c662a0fd fix(test): make MailboxScreenTest empty-state gate robust under perpetual Loading
Poll for the compose FAB via waitForText instead of a one-shot assert: under refresh==Loading the LazyPagingItems presenter settles non-deterministically, and the FAB flaked as both not-displayed and not-found across CI runs. Keeps the stable mailbox_empty assertDoesNotExist gate check (#219).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 21:53:35 -05:00
Jason Ross c38db3b213 Merge main into test-220-native-lib-before-keyed-open 2026-07-03 21:49:03 -05:00
Jason Ross ddc8c30ade Merge main into fix-219-empty-state-flash 2026-07-03 21:49:03 -05:00
Jason Ross 1e2554e86b Merge pull request #277 from JMR-dev/test-276-outlook-browser-launch
test(onboarding): assert the Outlook onboarding button launches the browser
2026-07-03 21:48:36 -05:00
Jason Ross fc6a79ddb0 Merge main into test-220-native-lib-before-keyed-open 2026-07-03 21:37:29 -05:00
JMR-devandClaude Opus 4.8 1ff9d32b80 test(db): pin native-lib-before-keyed-open at the DatabaseModule factory site
DatabaseProvisionerTest (mocked) and DatabaseProvisionerInstrumentedTest
(real SQLCipher) both pin that prepareCache() loads SQLCipher's native
library for the encrypted branch, but neither exercises
DatabaseModule.provideDatabase itself — the instrumented one opens
through a hand-rolled SupportOpenHelperFactory, bypassing the branch
that actually maps CacheOpenMode to a real factory. A regression that
breaks that wiring would slip through both existing guards.

Adds DatabaseModuleInstrumentedTest, calling provideDatabase directly
and driving the first real open through its own
DeferredOpenHelperFactory lambda: the encrypted branch loads the
native lib and opens a genuinely-encrypted file, the plaintext branch
never touches the native lib, and a fault-injected load failure
proves the keyed open is causally gated on the load rather than just
usually preceded by it.

Closes #220

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 21:36:34 -05:00
JMR-devandClaude Opus 4.8 36cbbe7b5d test(onboarding): assert the Outlook onboarding button launches the browser
Extends #274's AccountPickerScreenTest with an Espresso-Intents check that
tapping Outlook fires AppAuth's authorization intent. AppAuth always routes
through its own AuthorizationManagementActivity before it ever reaches a
real browser, so that component name is the one characteristic of the
launch that's both guaranteed and installed-browser-independent; matching
it also lets the test stub a canceled result so no real browser opens.
Verified against the real OutlookAuthManager + AppAuth 0.11.1 on a
google_apis API 29 emulator (the same image CI's managed devices use).
Redirect handling, token exchange, and account creation stay out of scope
per #276.

Closes #276

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 21:30:00 -05:00
JMR-dev 264cad83f8 Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts:
#	app/src/androidTest/kotlin/org/libremail/ui/Fakes.kt
2026-07-03 21:18:02 -05:00
Jason Ross a3b49737b9 Merge pull request #274 from JMR-dev/test-250-compose-screens-e2e
test(coverage): lane 5 — Compose screens E2E coverage
2026-07-03 21:04:40 -05:00
JMR-devandClaude Opus 4.8 bc45c9f3c9 fix(test): assert a robust node in MailboxScreenTest empty-state-gate test
emptyState_isHidden_whileTheInboxPagerIsStillLoading asserted the compose
FAB with assertIsDisplayed(), but MailboxScreen renders no loading
affordance in this exact scenario (isSyncingFolder only flips true from
selectFolder(), which this test never calls), so there is nothing else
guaranteed visible while refresh == Loading. The FAB is unconditionally
composed in Scaffold's floatingActionButton slot regardless of loading
state, so its role here is only to prove the screen composed rather than
crashing or rendering blank. Swap to assertExists(), which checks presence
in the semantics tree without requiring on-screen visibility, and keep the
core assertion (mailbox_empty assertDoesNotExist()) that verifies the
actual issue #219 behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 20:59:36 -05:00
Jason Ross 2498725f83 Merge main into fix-219-empty-state-flash 2026-07-03 20:49:15 -05:00
Jason Ross 148780735f Merge main into test-250-compose-screens-e2e 2026-07-03 20:49:13 -05:00
Jason Ross 1d84dd437f Merge pull request #240 from JMR-dev/feat-164-reorder-accounts
feat(settings): reorder accounts by drag in settings
2026-07-03 20:48:44 -05:00
JMR-devandClaude Opus 4.8 f98425667d fix(test): pass accountRepository to DiagnosticsCollector in ProblemReportsScreenTest
Rebase collision with the accountRepository param added to DiagnosticsCollector's
constructor broke :app:compileDebugAndroidTestKotlin. Mirrors the #245
CrashReporterInstallTest fix: mock AccountRepository.observeAccounts() to an empty flow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 20:47:16 -05:00
Jason Ross 52636fd459 Merge main into feat-164-reorder-accounts 2026-07-03 20:31:00 -05:00
Jason Ross f537717c23 Merge main into fix-219-empty-state-flash 2026-07-03 20:30:59 -05:00
Jason Ross 049c11f5f0 Merge main into test-250-compose-screens-e2e 2026-07-03 20:30:58 -05:00
Jason Ross cf31f99ae1 Merge pull request #272 from JMR-dev/test-226-worker-cachelock-deferral
test(sync): instrumented cache-lock deferral for PruneWorker/BackfillWorker
2026-07-03 20:30:29 -05:00
Jason Ross 8af6882234 Merge main into test-250-compose-screens-e2e 2026-07-03 20:27:37 -05:00
Jason Ross 5e581d4b52 Merge main into fix-219-empty-state-flash 2026-07-03 20:26:56 -05:00
JMR-devandClaude Opus 4.8 dea6804619 test(coverage): lane 5 — Compose screens E2E coverage
Add instrumented Compose UI/E2E tests for five previously-untested screens,
raising the ui/** view-layer coverage (issue #250):

- OutboxScreen: empty state, queued/failed rows, retry + cancel actions
- DraftsScreen: empty state, subject/recipient/body render + blank fallbacks,
  open + delete
- ProblemReportsScreen: empty state, crash/manual rows, open, create-and-review
  flow (real ReportStore + DiagnosticsCollector)
- AccountPickerScreen: all setup choices listed, app-password + manual routing
- SignaturesScreen: empty state, default badge, make-default (DB round-trip),
  delete (real in-memory Room + SignatureRepository)

Extend the shared FakeMailRepository to back observeOutbox()/observeDrafts()
with mutable state and record cancel/retry so the screens' actions are exercised
end to end.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 20:26:41 -05:00
JMR-devandClaude Opus 4.8 e190dc3191 fix(mailbox): keep the inbox pager warm on reader return so it doesn't flash empty
Returning from the reader/message screen to the inbox briefly showed the
"No messages" empty state and reloaded: the inbox's only Paging presenter
(collectAsLazyPagingItems) is torn down while a message is open, so the
cachedIn pager loses its downstream collector. Opening an unread message
writes setRead, invalidating the Room PagingSource; with nothing collecting,
the fresh generation only cold-loaded once the inbox re-entered composition —
a multi-second stall plus a one-frame empty-state flash. (The empty-state
gate itself already landed with #214/#223.)

Add an always-on, invisible PagingDataPresenter in MailboxViewModel that stays
subscribed to the cached paged flow across the reader visit (collectLatest
hands each new generation to collectFrom), so the post-setRead generation
loads in the background and the return replays a full window with no empty
frame.

Tests:
- MailboxViewModelTest: a real, invalidatable Pager proves the pager loads its
  initial window and reloads after invalidation with no UI collector attached.
- MailboxScreenTest: the empty state is held back while refresh is Loading and
  shown only once the pager settles genuinely empty, driven via PagingData.from
  with explicit LoadStates through a new FakeMailRepository paged override.

Closes #219

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 20:26:04 -05:00
JMR-devandClaude Opus 4.8 64d89d1e2a fix(accounts): break AccountDao sortOrder ties by email
AccountDaoTest (added by #270's coverage lane) asserts email ordering,
but AccountDao.observeAll()/getAll() now order by the user-defined
sortOrder (#164) with no tiebreaker. Two accounts inserted via plain
upsert() both land on the default sortOrder (0), so they came back in
rowid/insertion order instead — failing the test deterministically on
CI (API 30 & 31): expected [ada, zed], got [zed, ada].

Add `email` as a secondary ORDER BY key. Real accounts always get
distinct sortOrders via insertAtEnd()/reorder(), so drag order is
untouched; only equal-sortOrder rows now fall back to a stable,
deterministic email order. This also matches the "rank by email"
convention already used to seed sortOrder in ACCOUNT_MIGRATION_1_2 and
AccountDataMigrator.copyAccountTables, so the existing coverage test
passes unchanged. No schema/migration change is needed since this
only edits a @Query string, not the entity.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 20:24:25 -05:00
JMR-devandClaude Opus 4.8 153f8784a7 test(sync): instrumented cache-lock deferral for PruneWorker/BackfillWorker
Adds an on-device test proving PruneWorker/BackfillWorker defer (Result.retry())
while the encrypted cache is locked, using the REAL EncryptedCacheGuard instead of
the mocked guard the JVM PruneWorkerTest/BackfillWorkerTest use (issue #225). The
locked state is reproduced with no device auth by mocking SettingsRepository (the
same pattern DatabaseProvisionerInstrumentedTest already uses) and leaving a real
PassphraseSession never-unlocked. Adds androidx.work:work-testing so the workers
can be driven via TestListenableWorkerBuilder with a custom WorkerFactory (their
extra Hilt-assisted constructor args aren't supported by the default factory).

Closes #226

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 20:15:19 -05:00
Jason Ross fa0c7c48c8 Merge main into feat-164-reorder-accounts 2026-07-03 20:01:32 -05:00
Jason Ross ea470e065c Merge pull request #245 from JMR-dev/feat-235-debug-report-accounts
feat(reporting): add PII-free account summary to debug reports
2026-07-03 20:00:57 -05:00
Jason Ross 1ed94f9ea0 Merge main into feat-164-reorder-accounts 2026-07-03 19:47:41 -05:00
Jason Ross d29f146bd2 Merge main into feat-235-debug-report-accounts 2026-07-03 19:47:40 -05:00
Jason Ross 817f0dd642 Merge pull request #270 from JMR-dev/test-248-coverage-persistence-daos-migrations
test(coverage): lane 3 — persistence, DAOs & migrations to >=95%
2026-07-03 19:47:09 -05:00
JMR-devandClaude Opus 4.8 f16e1a19be chore: drop stray emulator log accidentally committed
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:32:58 -05:00
JMR-devandClaude Opus 4.8 d8a37d280c test(local): use ContextWrapper not mockk<Context> in provisioner test
DatabaseProvisionerInstrumentedTest crashed in @Before setUp() on API 31/32 with ArrayIndexOutOfBoundsException (length=0; index=0), passing on API 29. The stack shows the throw is entirely in the test harness: mockk<Context>() -> MockK JvmMockFactoryHelper.isKotlinInline -> kotlin-reflect ReflectJavaMember.getValueParameters, which indexes parameterAnnotations[0] on an empty array. Mocking android.content.Context makes MockK walk the whole framework class with kotlin-reflect, and on Android 12/12L ART returns a parameter-annotation array shorter than the parameter-type array for some Context method, so kotlin-reflect throws. Production DatabaseProvisioner/DatabaseEncryption code never runs. Replace the mockk<Context> with a real ContextWrapper(appContext) that overrides getDatabasePath to route the cache file to the test DB and delegates everything else, sidestepping the framework-class reflection walk. Verified 3/3 pass on the dev36 GMD emulator; API 31/32 left to CI (images not installed locally).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:31:03 -05:00
JMR-devandClaude Opus 4.8 be89bcc6d9 test(coverage): align lane-3 data/local tests with #234 casefold search + schema v19
PR #234 (issue #232, merged into this branch) bumped the Room schema 18->19 and made the MessageDao search queries match Unicode-casefolded *Fold columns. Two lane-3 tests were stale against it:

- DatabaseEncryptionTest.schemaVersionIsCarriedOntoTheEncryptedFile hardcoded the pre-#234 schema version 18; bump to 19 (matches LibreMailDatabase version = 19).
- MessageDaoTest's search-summary tests inserted MessageEntity fixtures without populating the new senderFold/senderEmailFold/subjectFold/snippetFold columns, so the casefolded LIKE matched nothing ([]). Populate them in the message() helper via lowercase(), mirroring production (Mappers.toEntity + MessageDao.updateHeaderContent/updateBody).

MigrationTest already covers 18->19 (migrate18To19_addsAndBackfillsCasefoldSearchColumns plus the auto-discovered full-chain replays), so no change there. Verified: targeted connectedDebugAndroidTest of MessageDaoTest+DatabaseEncryptionTest+MigrationTest on the API 36 emulator = 32 tests, 0 failures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:02:57 -05:00
JMR-devandClaude Opus 4.8 664d79a427 fix(test): pass accountRepository to DiagnosticsCollector in CrashReporterInstallTest
DiagnosticsCollector gained a 4th constructor param (accountRepository) for the
PII-free account summary, but CrashReporterInstallTest still constructed it with
3 args — a compile error that broke the Unit tests and Static analysis gates.
Add the AccountRepository mock with observeAccounts() stubbed to an empty flow
(matching DiagnosticsCollectorTest) and pass it to both call sites.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 18:48:25 -05:00
JMR-devandClaude Opus 4.8 53f600369b fix(test): stub the new insertAtEnd DAO call in AccountRepositoryImplTest
The #164 reorder-accounts feature switched addImapAccount/addOutlookAccount from accountDao.upsert(...) to accountDao.insertAtEnd(...) (a @Transaction default method that stamps sortOrder before delegating to upsert), but the test's mocks/verifies still targeted upsert directly. Since MockK doesn't invoke a mocked interface's default method body, the unstubbed insertAtEnd call threw MockKException. Updated the stub/verify pairs in both add-account happy-path tests and the exactly-0 verifies in both failure-path tests to reference insertAtEnd.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 18:45:45 -05:00
Jason Ross ac061d3138 Merge main into test-248-coverage-persistence-daos-migrations 2026-07-03 18:40:01 -05:00
JMR-devandClaude Opus 4.8 0d3b0e2a73 test(coverage): lane 3 — persistence, DAOs & migrations instrumented tests
Instrumented (androidTest) coverage for data/local: Room DAO queries/mutations,
every exported-schema migration, and DatabaseProvisioner/DatabaseEncryption
(SQLCipher) provisioning branches, incl. a regression guard for the SQLCipher
System.loadLibrary cold-start crash (592a797).

Validated locally: 114/181 instrumented tests passed, 0 failed, via
connectedDebugAndroidTest on a manually-provisioned api36 emulator. The local
GMD emulator wedges mid-suite (~112) on this machine (see #269); CI validates
the full 181 on its own runners.

Closes #248

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 18:32:36 -05:00
Jason Ross 0ceaf97754 Merge main into feat-235-debug-report-accounts 2026-07-03 18:29:37 -05:00
Jason Ross 5705e0e7f8 Merge main into feat-164-reorder-accounts 2026-07-03 18:29:36 -05:00
Jason Ross a297e38304 Merge pull request #234 from JMR-dev/feat-unicode-search-casefold
feat(search): Unicode-aware case-insensitive search via casefold columns
2026-07-03 18:29:03 -05:00
Jason Ross 0fd3954f79 Merge main into feat-unicode-search-casefold 2026-07-03 17:38:56 -05:00
Jason Ross 579d4864b9 Merge main into feat-164-reorder-accounts 2026-07-03 17:38:55 -05:00
Jason Ross 0205bff8a0 Merge main into feat-235-debug-report-accounts 2026-07-03 17:38:54 -05:00
Jason Ross f714352a08 Merge pull request #242 from JMR-dev/feat-239-purge-old-reports
feat(reporting): purge crash/problem reports older than a month while charging
2026-07-03 17:38:27 -05:00
Jason Ross 6f3daae4b2 Merge main into feat-235-debug-report-accounts 2026-07-03 17:25:01 -05:00
Jason Ross 5b5e48054d Merge main into feat-239-purge-old-reports 2026-07-03 17:25:00 -05:00
Jason Ross 358dcc788e Merge main into feat-164-reorder-accounts 2026-07-03 17:24:58 -05:00
Jason Ross 0a462fda0c Merge main into feat-unicode-search-casefold 2026-07-03 17:24:57 -05:00
Jason Ross ba2dc7fcf6 Merge pull request #266 from JMR-dev/ci-preflight-api35-api37
ci(preflight): add API 35 and API 37 emulator E2E to preflight
2026-07-03 17:24:27 -05:00
Jason Ross fb802a8556 Merge main into ci-preflight-api35-api37 2026-07-03 17:11:13 -05:00
Jason Ross 7543350330 Merge main into feat-unicode-search-casefold 2026-07-03 17:11:12 -05:00
Jason Ross dad3345ce0 Merge main into feat-164-reorder-accounts 2026-07-03 17:11:11 -05:00
Jason Ross 0948d1406e Merge main into feat-235-debug-report-accounts 2026-07-03 17:11:10 -05:00
Jason Ross 9d8581a551 Merge main into feat-239-purge-old-reports 2026-07-03 17:11:10 -05:00
Jason Ross b6e9c12dab Merge pull request #268 from JMR-dev/ci-p0-only-preemption
ci(runners): P0 & broken-target preemption; P1-P9 yield without bumping in-progress
2026-07-03 17:10:43 -05:00
Jason Ross 7485fecfea Merge main into ci-p0-only-preemption 2026-07-03 17:09:21 -05:00
JMR-devandClaude Opus 4.8 249381b249 ci(runners): P0 & broken-target preemption; P1-P9 yield without bumping
Only P0 preempts in-progress runs (emergency reservation). P1-P9 no longer
cancel lower-priority runs; instead traffic-control holds back (bounded poll,
kept under timeout-minutes) while strictly-higher-priority PRs still have
active/queued CI runs, so their heavy jobs reach the runner queue first.

New `broken` label forces effective priority below P9 (sentinel 10): a broken
PR never preempts (even if also labelled P0 -- broken wins) and always yields,
and because its run is wasted, ANY higher-priority PR (not just P0) may cancel
its in-progress run to reclaim the runner. Net rule: a strictly-lower run is
cancelled iff (self is P0) OR (target is broken); otherwise yield.

All existing safety preserved: never main/push runs, never our own run, never
an equal-or-higher-priority PR; PR-controlled strings via env/jq only;
continue-on-error + set +e + always exit 0; traffic-control stays a
non-required best-effort job and ci-passed is unchanged.

Validated with actionlint and a mocked-gh + fake-clock logic harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 17:08:37 -05:00
Jason Ross fa64c7b334 Merge main into feat-239-purge-old-reports 2026-07-03 16:49:27 -05:00
Jason Ross e7158f344d Merge main into feat-164-reorder-accounts 2026-07-03 16:49:26 -05:00
Jason Ross 50db69b060 Merge main into feat-unicode-search-casefold 2026-07-03 16:49:25 -05:00
Jason Ross 93ca14bc18 Merge main into feat-235-debug-report-accounts 2026-07-03 16:49:24 -05:00
Jason Ross 5217c5b7ad Merge main into ci-preflight-api35-api37 2026-07-03 16:49:23 -05:00
Jason Ross d903ded2d4 Merge pull request #267 from JMR-dev/build-gmd-serial-emulators
build(gmd): cap managed-device emulators to 1 concurrent (serial) to avoid local VT-x contention
2026-07-03 16:48:57 -05:00
JMR-devandClaude Opus 4.8 05b43f4510 build(gmd): cap managed-device emulators to 1 concurrent (serial) to avoid local VT-x contention
The e2e Gradle Managed Device group spans api29-36 and org.gradle.parallel=true
is set, so a local e2eGroupDebugAndroidTest (or preflight's api36DebugAndroidTest)
can launch several emulators at once. They contend for the same VT-x/HAXM
virtualization slot on a single machine and hang at 0% CPU with "another
emulator instance is running". Set
android.experimental.testOptions.managedDevices.maxConcurrentDevices=1 in
gradle.properties to force GMD emulator runs serial locally.

CI is unaffected: its e2e matrix boots one emulator per API level on separate
GitHub Actions runners via reactivecircus/android-emulator-runner and
connectedDebugAndroidTest, not these Gradle Managed Device tasks, so the cap
doesn't apply there regardless.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 16:45:24 -05:00
JMR-devandClaude Opus 4.8 ac5ed162ae ci(preflight): use host-GPU auto-no-window for local api37 emulator
Change the api37_e2e.py emulator launch from `-gpu swiftshader_indirect`
to `-gpu auto-no-window`. For a LOCAL run the host GPU is faster and
auto-no-window is the mode that boots cleanly on this machine; CI's
e2e-preview keeps swiftshader_indirect for headless-runner determinism.
This is now the single deliberate divergence from e2e-preview; the image
string, provisioning, boot sequence, and every other emulator flag stay
in lockstep. Updated the script comments/docstring, SKILL.md, CLAUDE.md,
and the build.gradle.kts managed-devices comment to document it.

Syntax-only change (python -m py_compile clean); emulator not run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 16:31:41 -05:00
JMR-dev b5a29a0450 Merge remote-tracking branch 'origin/ci-preflight-api35-api37' into ci-preflight-api35-api37 2026-07-03 16:23:33 -05:00
JMR-devandClaude Opus 4.8 5ecc2401e3 ci(preflight): hand-provision API 37 preview E2E locally
Per the repo owner's decision, preflight now runs the API 37 preview
emulator locally instead of leaving it to CI. Since there is no Gradle
Managed Device DSL path to the nonstandard android-37.0 /
google_apis_ps16k image, add a stdlib-only, cross-platform Python 3
helper (.claude/skills/preflight/api37_e2e.py) that mirrors CI's
e2e-preview job EXACTLY: same system image string
(system-images;android-37.0;google_apis_ps16k;x86_64), same emulator
flags, same provisioning/boot sequence. It installs the image via
sdkmanager, creates the AVD via avdmanager, cold-boots headless, waits
for sys.boot_completed, runs :app:connectedDebugAndroidTest, then tears
the emulator + AVD down. Cross-platform: per-OS tool discovery/suffixes
and cmd /c wrapping for Windows .bat launchers.

Update SKILL.md + CLAUDE.md so preflight runs api35 + api36 (GMDs) +
api37 (this script), and the app/build.gradle.kts managed-devices
comment now points at the script. Add a caveat that emulators need a
free hardware hypervisor (VT-x/WHPX) — shut down VirtualBox/other VMs
first or the AVD hangs at 0% CPU.

Validated syntactically only (python -m py_compile + ast.parse +
argparse --help); no emulator was booted and no build was run, to avoid
contending with an in-progress api36 run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 16:22:31 -05:00
Jason Ross 8e7b82f86a Merge main into feat-235-debug-report-accounts 2026-07-03 16:17:08 -05:00
Jason Ross a5b26632ac Merge main into ci-preflight-api35-api37 2026-07-03 16:17:08 -05:00
Jason Ross de04bc2cae Merge main into feat-unicode-search-casefold 2026-07-03 16:17:07 -05:00
Jason Ross 91e267bb44 Merge main into feat-164-reorder-accounts 2026-07-03 16:17:06 -05:00
Jason Ross 0b05df6bef Merge main into feat-239-purge-old-reports 2026-07-03 16:17:05 -05:00
Jason Ross f33b111256 Merge pull request #265 from JMR-dev/ci-priority-runner-orchestration
ci(runners): priority-based runner orchestration via P0–P9 labels
2026-07-03 16:16:33 -05:00
Jason Ross bef197b628 Merge main into feat-239-purge-old-reports 2026-07-03 16:09:18 -05:00
Jason Ross 3fedc854de Merge main into feat-164-reorder-accounts 2026-07-03 16:09:16 -05:00
Jason Ross 40067a069f Merge main into feat-unicode-search-casefold 2026-07-03 16:09:15 -05:00
Jason Ross b384647a9f Merge main into feat-235-debug-report-accounts 2026-07-03 16:09:14 -05:00
Jason Ross 29e22042cb Merge main into ci-priority-runner-orchestration 2026-07-03 16:09:13 -05:00
Jason Ross 596b387697 Merge main into ci-preflight-api35-api37 2026-07-03 16:09:12 -05:00
Jason Ross a8784d2864 Merge pull request #253 from JMR-dev/fix-193-age-retention-sync-window
fix(sync): bound the foreground fetch window by the age cutoff in age retention
2026-07-03 16:08:42 -05:00
JMR-devandClaude Opus 4.8 1774a33158 ci(preflight): add API 35 and API 37 emulator E2E to preflight
Extend the local preflight gate from api36 (the sole latest-API GMD
run) to api35 + api36, the top two stable levels in the E2E matrix.
Both Gradle Managed Devices already existed in app/build.gradle.kts
(the api29..36 loop) — confirmed via `:app:tasks --group verification`,
no emulator run needed.

API 37 (preview) was investigated but NOT added as a GMD: its only
published system image is the nonstandard "android-37.0" /
google_apis_ps16k pairing that ci.yml's e2e-preview job installs by
hand via sdkmanager. ManagedVirtualDevice's apiLevel (Int) builds
"android-<N>" and apiPreview (codename) builds "android-<Codename>" —
neither produces "android-37.0", the same gap ci.yml documents as why
reactivecircus/android-emulator-runner can't provision it either.
docs/perf/issue-124-unified-inbox-paging.md independently corroborates
this: its API 37 measurements used a physical Pixel, not an AVD. There
is no api37DebugAndroidTest task to run, so it stays CI-only
(e2e-preview) until a managed-device-compatible image ships; the
comment above testOptions.managedDevices in app/build.gradle.kts now
documents this in detail for the next person who looks.

.claude/skills/preflight/SKILL.md and CLAUDE.md are updated to run
both api35DebugAndroidTest and api36DebugAndroidTest as part of the
required gate, with the API 37 gap called out inline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 16:05:12 -05:00
Jason Ross 4fb9db2a51 Merge main into ci-priority-runner-orchestration 2026-07-03 16:04:33 -05:00
JMR-devandClaude Opus 4.8 0e3ffd58ef ci(runners): priority-based runner orchestration via P0–P9 labels
Add a lightweight `traffic-control` job that runs first (the heavy
build/E2E jobs `needs:` it) and preempts contended runners by PR
priority. It reads the triggering PR's P0–P9 label (P0 = highest,
P9 = lowest; default P5 when unlabeled) and cancels the in-progress /
queued CI runs of strictly-lower-priority OTHER open PRs, freeing their
runners for the higher-priority PR.

Safety: never cancels main/push runs, the PR's own run, or an
equal-or-higher-priority PR — only strictly-lower-priority OTHER open
PRs' active CI runs. The job is best-effort (every gh call guarded,
always exits 0, step is continue-on-error) and is NOT part of the
`CI passed` merge gate. `ci-passed` now also treats a `skipped` heavy
job as a gate failure, so a (should-never-happen) traffic-control
failure blocks the merge fail-safe rather than passing it untested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 16:02:45 -05:00
Jason Ross fecd4d67dc Merge branch 'main' into feat-235-debug-report-accounts 2026-07-03 15:55:42 -05:00
Jason Ross 4cead4d520 Merge branch 'main' into fix-193-age-retention-sync-window 2026-07-03 15:55:13 -05:00
Jason Ross 8779b439f7 Merge pull request #264 from JMR-dev/ci-262-autoupdate-tighten
ci(autoupdate): drop synchronize trigger, keep draft PRs updated
2026-07-03 15:51:30 -05:00
JMR-devandClaude Opus 4.8 3da0c3a580 ci(autoupdate): drop synchronize trigger, keep draft PRs updated
Narrow the pull_request trigger to opened/reopened/ready_for_review so
per-commit pushes to open PRs no longer storm the runners via a
rebase-of-all-PRs (PR_FILTER: all) on every synchronize event. Pin
PR_READY_STATE to "all" so draft PRs remain in scope for updates
triggered by push (main advancing) and opened.

Closes #262

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 15:49:37 -05:00
Jason Ross f7285a0152 Merge branch 'main' into fix-193-age-retention-sync-window 2026-07-03 15:47:48 -05:00
Jason Ross 402e19a707 Merge pull request #263 from JMR-dev/fix-255-crash-prompt-gating
fix(reporting): auto-prompt to submit a crash only on first re-open, for a legitimate <24h crash
2026-07-03 15:47:05 -05:00
Jason Ross 944bd45a1b Merge branch 'main' into fix-255-crash-prompt-gating 2026-07-03 15:31:03 -05:00
JMR-devandClaude Opus 4.8 6333dd4511 fix(reporting): gate startup crash prompt to a legitimate <24h crash, first re-open only
The auto-submit crash prompt over-triggered: it re-surfaced the newest saved
crash report on every launch, with no age bound, so a pre-update crash kept
popping "LibreMail crashed" long after the crash was fixed (#255).

Gate StartupReportViewModel.pendingCrash so a crash is auto-offered:
- first re-open only — dismiss() now persists a "surfaced" marker instead of an
  in-memory-only hide, so a report is offered at most once across launches; it
  stays in the store (still listed in Problem Reports) and only discard() deletes.
- < 24h only — inject a clock provider and filter to createdAtMillis within 24h.
- legitimate crash only — reports come solely from CrashReporter's uncaught-
  exception handler, so update / force-stop / user-close create none; made
  explicit and covered by a test.

The marker is a minimal additive `surfaced` flag on DebugReport (persisted in
storage JSON, kept out of the submission payload; a missing flag = not surfaced)
plus ReportStore.markSurfaced(id). Extracted StartupCrashPrompt from LibreMailApp
so the real dialog + gating is E2E-testable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 15:17:22 -05:00
Jason Ross cb59417f59 Merge pull request #261 from JMR-dev/test-247-coverage-sync-workers-transport-auth
test(coverage): lane 2 — sync, workers, transport & auth to >=95%
2026-07-03 15:17:01 -05:00
Jason Ross 5a114517dc Merge main into test-247-coverage-sync-workers-transport-auth 2026-07-03 14:58:28 -05:00
JMR-devandClaude Opus 4.8 be0e699fcc test(coverage): lane 2 — sync, workers, transport & auth to >=95%
Test-only (zero production changes). Raises JVM unit-test LINE coverage
for the sync/worker, IMAP/SMTP/Graph transport, and OAuth packages:
data/sync 98.6%, mail 96.7%, auth 100.0% LINE.

New/extended cover:
- SendWorker outbox drain (SMTP/Graph, may-have-sent, SMTP fallback, staged
  attachments), MailConnectionFactory token cache/refresh, MailSyncer.syncAll,
  SendScheduler, MailBackfiller pre-existing-row refresh.
- ImapConnectionCache reuse + drop-retry, ImapClient fetchAttachment/setFlag/
  deleteMessage/idle + edge cases, GraphSender.send transport.
- OutlookAuthManager token exchange/refresh + failure branches, OAuth models.

Instruction/branch coverage stays lower (coroutine suspend-state synthetics
under synchronous mocks) — a known JaCoCo x coroutines limitation, not
untested logic; JaCoCo config is untouched (owned by the capstone lane).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 14:54:12 -05:00
Jason Ross 99be90ef48 Merge main into feat-239-purge-old-reports 2026-07-03 14:53:31 -05:00
Jason Ross cc77611d43 Merge main into feat-164-reorder-accounts 2026-07-03 14:53:30 -05:00
Jason Ross 056ac69185 Merge main into feat-235-debug-report-accounts 2026-07-03 14:53:29 -05:00
Jason Ross 853b40f419 Merge main into feat-unicode-search-casefold 2026-07-03 14:53:28 -05:00
Jason Ross 8ab9c7ec29 Merge main into fix-193-age-retention-sync-window 2026-07-03 14:53:27 -05:00
Jason Ross 1e3cd7cae5 Merge pull request #260 from JMR-dev/ci-259-autoupdate-pr-trigger
ci(autoupdate): also trigger on pull_request so newly-opened PRs update immediately
2026-07-03 14:52:23 -05:00
JMR-devandClaude Opus 4.8 ef77a7559f ci(autoupdate): simplify pull_request trigger to any PR to main
Per repo-owner preference, drop the explicit types list and use the
default pull_request event set, keeping only the base-branch filter
(branches: [main]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 14:49:39 -05:00
Jason Ross 78b7ebf777 Merge main into fix-193-age-retention-sync-window 2026-07-03 14:48:15 -05:00
Jason Ross 4c852a1083 Merge main into feat-unicode-search-casefold 2026-07-03 14:48:14 -05:00
Jason Ross 339b4017ef Merge main into feat-235-debug-report-accounts 2026-07-03 14:48:13 -05:00
Jason Ross 02a0082c7f Merge main into feat-164-reorder-accounts 2026-07-03 14:48:12 -05:00
Jason Ross 8eac5807bc Merge main into feat-239-purge-old-reports 2026-07-03 14:48:10 -05:00
Jason Ross e03788da6d Merge main into ci-259-autoupdate-pr-trigger 2026-07-03 14:48:09 -05:00
Jason Ross f01ec66e85 Merge pull request #256 from JMR-dev/test-249-coverage-viewmodels-nonui
test(coverage): lane 4 — ViewModels & non-UI modules to >=95%
2026-07-03 14:47:40 -05:00
JMR-devandClaude Opus 4.8 9a3e3aa42d ci(autoupdate): also trigger on pull_request so newly-opened PRs update immediately
The workflow only fired on push to main, so a PR opened during a quiet
period (no subsequent merge to main) sat behind main until manually
updated. Add an opened/reopened/ready_for_review pull_request trigger;
synchronize is intentionally excluded to avoid re-running on every push,
including the autoupdate action's own branch updates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 14:44:14 -05:00
Jason Ross 50f8cf2e2b Merge branch 'main' into test-249-coverage-viewmodels-nonui 2026-07-03 14:34:32 -05:00
JMR-devandClaude Opus 4.8 b5997f75fe test(coverage): lane 4 — ViewModels & non-UI modules to >=95%
Add JVM unit tests (test-only; no production changes) covering the in-scope
ViewModels + UI state holders and the reporting/push/power/contacts modules
for issue #249.

New ViewModel coverage: Drafts, Outbox, Signatures, SignatureEdit,
AccountSettings, AccountSetup, ManualSetup, ProblemReports, StartupReport,
plus gap-filling for Compose, Mailbox, Reader, Settings, ReportReview and
AppPassword (contacts autocomplete, inline images, send/refresh failure
branches, drawer/search hooks, state-holder value semantics).

New module coverage: AppLog, AppVersionProvider, ReportSubmitter,
ReportUploadWorker (reachable paths), CrashReporter.install, LogEntry,
IntentComposeParser, ContactsRepository, ContactsPermissionManager,
IdlePushManager, BatteryOptimizationManager (Context methods) and
AndroidBatteryStatusProvider.

Android-framework-bound classes with no JVM seam are deliberately left to the
instrumented suite: IdleService (foreground Service), ReportUploadScheduler
(WorkManager.getInstance is not statically mockable), the HTTP transmit path in
ReportUploadWorker (unreachable while BuildConfig.DEBUG_REPORT_ENDPOINT is
empty), and CrashReporter.terminate (calls exitProcess).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 14:26:41 -05:00
Jason Ross c707fd29e5 Merge main into feat-239-purge-old-reports 2026-07-03 14:10:10 -05:00
Jason Ross d3f8136f12 Merge main into feat-164-reorder-accounts 2026-07-03 14:10:09 -05:00
Jason Ross 1ea50ea316 Merge main into feat-unicode-search-casefold 2026-07-03 14:10:08 -05:00
Jason Ross 81a362c30a Merge main into feat-235-debug-report-accounts 2026-07-03 14:10:07 -05:00
Jason Ross 873f45ff33 Merge main into fix-193-age-retention-sync-window 2026-07-03 14:10:06 -05:00
Jason Ross 77731e06e4 Merge pull request #254 from JMR-dev/test-246-coverage-repo-mappers-domain
test(coverage): lane 1 — repository, mappers & domain logic to >=95%
2026-07-03 14:09:34 -05:00
JMR-devandClaude Opus 4.8 e2606b7751 test(coverage): lane 1 — repository, mappers & domain logic to >=95%
Add JVM-only unit tests (74 across 4 new, purely-additive files) covering
the data/repository, data-mapper, and pure domain packages. No production
code is changed.

- AccountRepositoryImplTest: first tests for AccountRepositoryImpl — add/
  test/delete/observe + reset-backfill, success and rejected-LIST failure
  paths (class now 100% instruction & line).
- MailRepositoryImplCoverageTest: the MailRepositoryImpl methods/edges the
  existing suite skipped — observe-* flows, getMessage/getDraft, setStarred,
  deleteMessage, sendMessage + copyAttachments (incl. unreadable-URI skip),
  searchServer (all-accounts vs. filtered), and the account/row-gone
  fall-throughs.
- MappersTest: entity<->domain mappers not otherwise pinned, incl. the
  unknown-enum fallbacks and FetchedMessage id/uid rules.
- DomainModelCoverageTest: AccountSettings.signatureBlock branches,
  Signature.plainText, default-arg constructors, and display-name fallbacks
  (domain/model now 100% instruction & line).

Closes #246

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 13:55:51 -05:00
JMR-devandClaude Opus 4.8 e823f9b17e fix(sync): bound the foreground fetch window by the age cutoff in age retention
In age-based retention, MailSyncer fetched the newest-N headers but only capped that window by the
retention COUNT, not the age cutoff. On a low-traffic mailbox whose newest-N span older than the
cutoff, each sync re-inserted messages the age pruner had just deleted, and the next prune deleted
them again — a churn loop of wasted DB writes + prune deletes (issue #193).

Sync now drops fetched messages older than policy.ageCutoffMillis before persisting (the same cutoff
the pruner uses), so sync and prune keep exactly the same set in both retention modes. Count/unlimited
modes have a null cutoff and are unchanged. The empty-folder wipe is keyed on the raw fetch (server
truth), so a folder holding only past-cutoff mail is left to the pruner rather than wiped.

MailPruner's KDoc now documents the sync alignment for both modes.

Closes #193

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 13:17:24 -05:00
Jason Ross f2cbabb59a Merge main into feat-235-debug-report-accounts 2026-07-03 13:15:30 -05:00
Jason Ross c86e8acf2f Merge main into feat-unicode-search-casefold 2026-07-03 13:15:29 -05:00
Jason Ross 4a3a360471 Merge main into feat-164-reorder-accounts 2026-07-03 13:15:27 -05:00
Jason Ross 1e738a6670 Merge main into feat-239-purge-old-reports 2026-07-03 13:15:26 -05:00
Jason Ross e600122cc0 Merge pull request #252 from JMR-dev/docs-definition-of-done
docs(claude): require unit + latest-API E2E in the definition of done
2026-07-03 13:14:56 -05:00
JMR-devandClaude Opus 4.8 3903a4b4b1 docs(claude): run latest-API emulator E2E in preflight and require it for done
Make the latest-API-level emulator E2E (api36DebugAndroidTest, the
highest level in the E2E matrix and its Gradle Managed Device task) an
actually-run, required step:

- CLAUDE.md: preflight now runs api36DebugAndroidTest, and a change is
  not done until that E2E runs and passes locally (not merely compiles).
  The full multi-API matrix and the API 37 preview job stay CI's job.
- preflight skill: add the api36 E2E as the final step, note the
  emulator/managed-device precondition, and replace the old
  "don't run E2E locally" guidance so the two files agree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 13:12:24 -05:00
JMR-devandClaude Opus 4.8 321d90432b docs(claude): drop local-emulator carve-out from definition of done
State plainly that a change isn't complete without passing unit tests
and E2E/instrumented tests covering it, with no softening about
running the emulator matrix locally being optional.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 13:01:47 -05:00
JMR-devandClaude Opus 4.8 d52cd2b4bf docs(claude): require unit + E2E tests in the definition of done
Codify that a task/PR isn't complete without both passing unit tests
and E2E/instrumented tests covering the change. Writing and committing
the E2E/instrumented test is required; only running it against a
booted emulator locally stays optional, since CI's E2E matrix covers
that.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 13:00:09 -05:00
Jason Ross d39617e5d8 Merge main into feat-239-purge-old-reports 2026-07-03 12:55:52 -05:00
Jason Ross f10129c4c6 Merge main into feat-164-reorder-accounts 2026-07-03 12:55:51 -05:00
Jason Ross 78fa555d87 Merge main into feat-unicode-search-casefold 2026-07-03 12:55:50 -05:00
Jason Ross 92a85c1f9e Merge main into feat-235-debug-report-accounts 2026-07-03 12:55:49 -05:00
Jason Ross b3e3ddd59d Merge pull request #241 from JMR-dev/build-192-jacoco
build: wire up JaCoCo code-coverage reporting
2026-07-03 12:55:16 -05:00
JMR-devandClaude Opus 4.8 4260a304bd feat(reporting): add PII-free account summary to debug reports
DiagnosticsCollector now includes one "<provider> (<authType>)" entry per account (the count is the
list size) in DebugReport.accounts, alongside the existing settings + recent-log capture. The provider
is a coarse bucket derived from the IMAP host (Gmail/Yahoo/iCloud/Outlook/AOL/Other) — never the raw
host or email — so no PII leaks; a custom domain buckets to "Other". Accounts are cached like settings
so crash reports (built on the crashing thread) include the last-known snapshot; accounts live in the
non-auth AccountDatabase, so reading them never blocks on the encrypted cache.

Recent device/app logs were already captured (RingLogBuffer) and serialize as the report's "logs".

Closes #235

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 12:53:58 -05:00
Jason Ross 276e08840a Merge main into feat-unicode-search-casefold 2026-07-03 12:43:07 -05:00
Jason Ross c5dd42b94b Merge main into feat-164-reorder-accounts 2026-07-03 12:43:06 -05:00
Jason Ross 9cd136f81b Merge main into build-192-jacoco 2026-07-03 12:43:05 -05:00
Jason Ross a7ab6275ac Merge main into feat-239-purge-old-reports 2026-07-03 12:43:04 -05:00
Jason Ross 7b52a9682d Merge pull request #244 from JMR-dev/ci-autoupdate-all-prs
ci(autoupdate): keep all open PRs up to date (drop the auto_merge filter)
2026-07-03 12:42:36 -05:00
JMR-devandClaude Opus 4.8 aad9d99ffc ci(autoupdate): keep all open PRs up to date (drop the auto_merge filter)
Closes #243

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 12:39:24 -05:00
JMR-devandClaude Opus 4.8 7f623167e7 feat(reporting): purge crash/problem reports older than a month while charging
Adds ReportPurgeWorker, deleting locally-stored crash/problem reports older than
30 days via new ReportStore.purgeOlderThan(cutoffMillis). Scheduled as a periodic
WorkManager job with a charging constraint (SyncScheduler.schedulePeriodicReportPurge,
enqueued at startup alongside sync/backfill/prune) so it never costs battery. Reports
are file-backed (no DB), so the worker needs no cache-lock gate.

Also discloses the auto-deletion: the problem-reports list and the submission review
screen state reports are deleted from the device after 1 month.

Tests: ReportStore.purgeOlderThan cutoff (boundary kept); ReportPurgeWorker computes a
~30-day cutoff and retries on failure.

Closes #239

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 12:27:24 -05:00
JMR-devandClaude Opus 4.8 535cbcb49a build(coverage): finish wiring JaCoCo unit-test coverage reporting
Completes the crash-interrupted #192 WIP (app/build.gradle.kts already had a
jacocoTestReport task and toolVersion pin recovered onto build-192-jacoco):

- Move the JaCoCo tool version into gradle/libs.versions.toml instead of a
  hardcoded string in app/build.gradle.kts, matching how every other plugin
  version in this repo is sourced.
- Fix the generated-code exclusion list against the real compileDebugKotlin
  output (verified by inspecting the compiled class tree): Room's
  KSP-generated `_Impl` DAOs/database and the Compose compiler's per-file
  ComposableSingletons holders are the only generated code that actually
  lands in classDirectories, since Hilt/Dagger's generated Java and AGP's
  BuildConfig/R/Manifest are compiled by a separate javac task this report
  never reads. Drop the blanket `**/*$$*` exclude the WIP had — it was
  silently discarding ~200 real classes' worth of coverage on Kotlin's own
  `$$inlined$` synthetic classes (e.g. Flow.map { ... } transforms in the
  repositories), which is hand-written logic, not generated boilerplate.
- Add Hilt_*/Dagger* prefix patterns so the (currently inert,
  belt-and-suspenders) Hilt exclusions are actually correct if the
  classDirectories scope ever changes.
- Add a minimal CI step to the existing unit-tests job that runs
  jacocoTestReport and uploads the XML+HTML report as a build artifact.
  No coverage threshold gate yet (a jacocoTestCoverageVerification rule is
  a natural follow-up once there's a baseline).
- Document the new :app:jacocoTestReport task in CLAUDE.md.

Verified on JDK 21: fast gate (assembleDebug, testDebugUnitTest,
compileDebugAndroidTestKotlin, lintDebug, ktlintCheck, detekt) plus
jacocoTestReport all pass, from both a warm and a `clean` build. The
report shows real signal (30% instruction / 38% line coverage) with no
generated classes leaking in.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 12:23:40 -05:00
JMR-devandClaude Opus 4.8 9a69d175a8 feat(settings): reorder accounts by drag in settings
Give accounts a user-controlled order (issue #164). Every surface that
lists accounts -- the Settings list, the drawer account switcher, and the
compose account picker -- reads the same `ORDER BY sortOrder` query, so a
reorder in Settings is honored app-wide. In Settings a row can be
long-pressed and dragged to a new position; the order persists and
survives restart.

Data layer:
- AccountEntity gains `sortOrder` (@ColumnInfo defaultValue "0"); AccountDao
  orders by it and adds insertAtEnd / reorder / nextSortOrder / setSortOrder,
  the mutations wrapped in transactions.
- New accounts are appended (current max + 1) via insertAtEnd.

Migration (AccountDatabase v1 -> v2):
- ACCOUNT_MIGRATION_1_2 adds the column and backfills existing accounts by
  their previous alphabetical (email) rank, so the already-shown order does
  not shuffle on upgrade. Registered in AccountDatabaseModule; 2.json is
  exported and AccountMigrationTest replays and validates it.
- AccountDataMigrator (the pre-#111 cache->account-db move) creates the
  v2-shaped table and applies the same email-rank backfill, since
  ACCOUNT_MIGRATION_1_2 does not run for that path.

UI:
- AccountReorderList drives long-press drag over a plain Column (no nested
  lazy list inside the scrolling settings column, no extra dependency); the
  pure reorder-index maths (commitDrag) is unit-tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 12:22:16 -05:00
JMR-dev 05dde9399c Merge remote-tracking branch 'origin/main' into continue-192 2026-07-03 12:11:26 -05:00
Jason Ross df6cbd4a76 Merge pull request #238 from JMR-dev/chore-hiltviewmodel-package
chore(ui): migrate hiltViewModel to its new androidx.hilt.lifecycle.viewmodel.compose package
2026-07-03 12:09:43 -05:00
JMR-devandClaude Opus 4.8 cbc9fc62ef wip: recover work from interrupted session (issue #192)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 12:04:36 -05:00
JMR-devandClaude Opus 4.8 7e69fcf185 chore(ui): migrate hiltViewModel to its new androidx.hilt.lifecycle.viewmodel.compose package
Clears the "hiltViewModel is deprecated; moved to package
androidx.hilt.lifecycle.viewmodel.compose" compile warnings across the 16 ui/**
files. Pure import swap: the old androidx.hilt.navigation.compose.hiltViewModel
inline-delegates to the new symbol, which is already transitively on the compile
classpath via the pinned hilt-navigation-compose:1.3.0 -- no dependency change,
identical signatures, behaviour byte-for-byte identical.

Closes #236

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 11:57:10 -05:00
JMR-devandClaude Opus 4.8 aae4f2218b feat(search): Unicode-aware case-insensitive search via casefold columns
Restores Unicode-aware case-insensitive substring search (approach A of #227).
Paging (#223) moved search to a SQL LIKE scan whose case-folding is ASCII-only,
so non-ASCII terms stopped matching case-insensitively.

Adds per-field casefold columns to `messages` (senderFold/senderEmailFold/
subjectFold/snippetFold), each = Kotlin lowercase() of its source (Unicode-aware).
Per-field (not one concatenated column) because the fields are maintained by
partial UPDATEs that don't carry all four: toEntity sets all folds, updateBody
keeps snippetFold in sync, updateHeaderContent keeps the header folds -- via thin
DAO default-method wrappers so the five call sites are unchanged. Search matches
the fold columns with a pattern built from the lowercased query.

Schema v18->v19 (additive; ASCII lower() backfill, non-ASCII rows re-fold on next
write). Adds a MigrationTest v18->v19 case and a repo test asserting the query is
casefolded.

Closes #232

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 11:41:53 -05:00
Jason Ross 2a2db6d0eb Merge pull request #231 from JMR-dev/test-cache-lock-gate-coverage
test(sync): cover the cache-lock gate on SyncWorker/SendWorker + the provisioner await
2026-07-03 11:41:19 -05:00
Jason Ross 34de875601 Merge main into test-cache-lock-gate-coverage 2026-07-03 11:28:04 -05:00
Jason Ross 3168bfa59e Merge pull request #230 from JMR-dev/build-release-arm-only
build(release): build the release APK for ARM only (arm64-v8a + armeabi-v7a)
2026-07-03 11:27:35 -05:00
Jason Ross b76447b8cf Merge main into build-release-arm-only 2026-07-03 11:15:18 -05:00
Jason Ross 2a07b2423e Merge main into test-cache-lock-gate-coverage 2026-07-03 11:15:17 -05:00
Jason Ross 48fe8d2fb0 Merge pull request #229 from JMR-dev/fix-prune-backfill-cache-lock-gate
fix(sync): gate PruneWorker & BackfillWorker on the encrypted-cache lock
2026-07-03 11:14:48 -05:00
JMR-devandClaude Opus 4.8 6d75bf5c6d test(sync): cover the cache-lock gate on SyncWorker/SendWorker + the provisioner await
Locks in the "pre-auth DB entry point defers while the encrypted cache is locked"
invariant that had zero coverage (which is how the PruneWorker/BackfillWorker gap
in #224 slipped in). Adds SyncWorkerTest and SendWorkerTest (locked -> retry with
the Lazy DB deps never resolved; unlocked -> runs), and a DatabaseProvisionerTest
case proving prepareCache() suspends on an auth-bound resolvePassphrase until it
resolves.

IdleService shares the same guard but is an Android Service (its start path needs
startForeground/Context), so its gate is covered by the instrumented test (#226)
rather than a JVM unit test.

Closes #225

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 11:04:34 -05:00
Jason Ross e2da97a2ff Merge main into fix-prune-backfill-cache-lock-gate 2026-07-03 11:01:14 -05:00
Jason Ross 82ed7dad1a Merge pull request #223 from JMR-dev/perf-mailbox-page-folder-search
perf(mailbox): page the per-account folder view and search
2026-07-03 11:00:45 -05:00
JMR-devandClaude Opus 4.8 a62d5edfdd build(release): build the release APK for ARM only (arm64-v8a + armeabi-v7a)
Adds ndk.abiFilters = ["arm64-v8a", "armeabi-v7a"] to the release build
type only, so the release APK/AAB ship the two ARM ABIs (64-bit + 32-bit)
instead of a universal build. x86 and x86_64 are intentionally dropped.
defaultConfig and the debug type are left untouched: CI's E2E matrix runs
the debug build on x86_64 emulators and needs the x86_64 native libs
(incl. libsqlcipher.so).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:57:30 -05:00
JMR-devandClaude Opus 4.8 7b5819021a fix(sync): gate PruneWorker & BackfillWorker on the encrypted-cache lock
PruneWorker and BackfillWorker were the only two pre-auth background DB entry
points that opened the database without first checking
EncryptedCacheGuard.isCacheLocked(). With encryptCache + appLock both on and a
headless cold start where the user hasn't authenticated (WorkManager after
reboot, or the periodic backfill/prune window while locked), the first DAO call
runs prepareCache() -> resolvePassphrase() -> session.await(), parking the
worker thread until unlock and serializing other DB openers behind the held
prepareCache mutex. Self-heals on unlock, but wastes wakelock/battery and makes
zero progress while locked.

Switch both workers' MailPruner/MailBackfiller injection to dagger.Lazy and add
the isCacheLocked() guard before resolving it, mirroring SyncWorker/SendWorker.
Add JVM regression tests (locked -> retry with the Lazy dep never resolved;
unlocked -> runs; failure -> retry).

Closes #224

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:54:48 -05:00
Jason Ross 6eb1a74a5b Merge branch 'main' into perf-mailbox-page-folder-search 2026-07-03 10:48:36 -05:00
Jason Ross cbeac0f26f Merge pull request #222 from JMR-dev/ci-e2e-boot-retry
ci(e2e): retry the API-29 emulator boot to absorb the android-emulator-runner keyevent race
2026-07-03 10:35:58 -05:00
JMR-devandClaude Opus 4.8 541f4809cb perf(mailbox): page the per-account folder view and search
Issue #124 paged only the unified "All inboxes" browse list. The
per-account folder view and every search path still loaded the whole
folder / entire unified inbox into memory and re-materialized it on each
cache write. Extend Paging 3 to them, mirroring the unified pager.

- MessageDao: add Room PagingSources for the per-account browse list
  (`pagingFolderSummaries`, `inInbox = 1`) and for paged search
  (`pagingUnifiedFolderSearchSummaries` / `pagingFolderSearchSummaries`).
  The search queries LIKE-match the same columns the old in-memory
  `matchesSearch` filter scanned (sender, sender address, subject,
  snippet) and leave `inInbox` unfiltered so transient server-search hits
  still surface. Read-only @Query methods — no schema change, no migration.
- MailRepository: add `pagedFolderMessages` and the two paged-search
  flows via a shared `mailboxPager` whose PagingConfig adds
  `maxSize = MAILBOX_PAGE_SIZE * 5` so a long scroll drops far-offscreen
  pages. A `likePattern` helper escapes the LIKE metacharacters (\ % _)
  so a query containing them still matches literally. The unified pager
  (`pagedUnifiedFolderMessages`) is left untouched for its sibling PR.
- MailboxViewModel: collapse the old `messages` list flow and the
  unified-only paged flow into one `pagedMessages` that dispatches each
  (account, folder, query) to the matching pager; `cachedIn` is kept so
  "select all" reads the loaded snapshot. Removes the now-dead
  observe*FolderMessages / matchesSearch paths.
- MailboxScreen: render every mode from the single paged list. Gate the
  empty state on `itemCount == 0 && refresh is NotLoading &&
  append.endOfPaginationReached` so it no longer flashes on an
  empty→loaded transition, preserving the search "no results", the
  syncing-folder spinner (#149), and browse "no messages" states.

Behaviour is preserved: search filtering columns/scope, multi-select and
"select all", unread counts, and the browse-vs-search state machine
(server search + debounce + open/close) are unchanged — only the local
list materialization moved from Kotlin into paged SQL.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:32:49 -05:00
JMR-devandClaude Opus 4.8 68a225ed9c ci(e2e): retry the API-29 emulator boot to absorb the android-emulator-runner keyevent race
The matrix E2E (29) job intermittently fails (~2%, API-29 only) in
reactivecircus/android-emulator-runner's un-guarded, fatal post-boot
`adb shell input keyevent 82`: on snapshot resume sys.boot_completed=1 is
restored before system_server republishes the `input` binder service, so
the job aborts before Gradle runs with "No service published for: input"
(fast-fail ~1m43s). Proven on run 28667366203.

Make the "Run E2E tests" step non-fatal (id + continue-on-error) and add a
guarded second attempt (if steps.e2e.outcome == 'failure'). Two independent
boots drop the race to ~0.04%; a genuine failure on both attempts still
fails the job (outcome, not conclusion). Definitive manual-boot fix: #218.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:23:36 -05:00
Jason Ross 440d1e9c62 Merge pull request #216 from JMR-dev/perf-mailbox-list
perf(mailbox): bound paging window (maxSize) and memoize per-row timestamp
2026-07-03 10:11:52 -05:00
JMR-devandClaude Opus 4.8 5cfd1b4065 perf(mailbox): bound paging window (maxSize) and memoize per-row timestamp
Two fixes from the Paging 3 perf audit of the mailbox list (#212, #213):

- Bound the unified-inbox paging window (#212): the only PagingConfig left
  maxSize at Int.MAX_VALUE, so pages were never evicted and a deep scroll
  accumulated the whole inbox in memory. Set maxSize = MAILBOX_PAGE_SIZE * 5
  (200), satisfying maxSize >= pageSize + 2*prefetchDistance (120). Safe with
  enablePlaceholders = false (the UI null-guards evicted positions).

- Memoize the per-row relative timestamp (#213): MessageRow formatted it via
  DateUtils.getRelativeTimeSpanString un-remembered, allocating a String on
  every recomposition. Wrapped in remember(timestampMillis).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 09:56:08 -05:00
Jason Ross 0e2be05be7 Merge pull request #209 from JMR-dev/chore-preflight-compile-androidtest
chore(preflight): compile androidTest source set in the fast gate
2026-07-03 09:54:15 -05:00
Jason Ross ae977c7539 Merge main into chore-preflight-compile-androidtest 2026-07-03 09:39:04 -05:00
Jason Ross 78c0fded29 Merge pull request #208 from JMR-dev/fix-sqlcipher-native-lib-load
fix(cache): load SQLCipher native lib before every keyed open
2026-07-03 09:38:35 -05:00
JMR-devandClaude Opus 4.8 3728efb55e chore(preflight): compile androidTest source set in the fast gate
assembleDebug, testDebugUnitTest, and lintDebug never compile the
androidTest source set, so a change that breaks it (e.g. an
instrumented test calling a UI API that just changed signature) passed
preflight locally yet only failed once CI ran. Add
:app:compileDebugAndroidTestKotlin to the fast gate to catch that
class of breakage before pushing, and update CLAUDE.md's summary of
the gate to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 09:27:12 -05:00
Jason Ross c1b80c3668 Merge branch 'main' into fix-sqlcipher-native-lib-load 2026-07-03 09:25:38 -05:00
JMR-devandClaude Opus 4.8 592a797dd0 fix(cache): load SQLCipher native lib before every keyed open
The opt-in encrypted cache crash-looped on launch (UnsatisfiedLinkError:
No implementation found for SQLiteConnection.nativeOpen) on any cold start
after encryption was enabled — reported after an app upgrade.

System.loadLibrary("sqlcipher") was only invoked as a side effect of an
actual plaintext<->encrypted conversion (DatabaseEncryption.migrate) or the
one-time #111 account migration. On a steady-state start the cache is
already encrypted and the account migration is already done, so both no-op
and nothing loads the native library before Room opens the keyed database
via SupportOpenHelperFactory -> nativeOpen. The previous process survived
only because an earlier conversion had loaded the .so in-memory; the next
cold start (e.g. an upgrade) crashes.

Load the library explicitly in DatabaseProvisioner whenever it commits to an
encrypted open (idempotent; no-ops when already loaded). Add regression
assertions: the encrypted path must load it, the plaintext path must not.

Verified on a Pixel 10 Pro XL — an in-place update preserving the already-
encrypted cache now launches to the mailbox instead of crash-looping.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 09:22:03 -05:00
Jason Ross 7784dbf3ba Merge pull request #207 from JMR-dev/fix-204-contentid-validation
chore(security): validate Content-ID before MIME/Graph use
2026-07-03 08:57:53 -05:00
Jason Ross 364fe32aaa Merge main into fix-204-contentid-validation 2026-07-03 07:23:23 -05:00
Jason Ross 1a0d140c03 Merge pull request #206 from JMR-dev/fix-205-fontfamily-whitelist
chore(security): whitelist font-family on HTML emit
2026-07-03 07:22:53 -05:00
JMR-devandClaude Opus 4.8 69cce168ef chore(security): validate Content-ID before MIME/Graph use
SmtpSender.inlinePart built the MIME header as `<${attachment.contentId}>`
and GraphSender put contentId straight into the Graph JSON — neither
stripped CR/LF or other ISO control characters. Not exploitable today
(contentId is always an app-generated `img-<uuid>@libremail`), but if an
external value ever reached contentId the SMTP path would be a MIME
header-injection vector.

New shared sanitizeContentId() strips ISO control chars (incl. CR/LF),
applied at both sinks: the SMTP Content-ID header and the Graph JSON
field. No behavior change for the app-generated ids in use today.

Tests: SmtpSenderTest sends an inline image whose contentId contains
`\r\nX-Injected: evil` and asserts (via GreenMail) no injected header
appears on any MIME part and the Content-ID stays a single line;
GraphSenderTest asserts the control chars are stripped from the payload.

Closes #204

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 07:11:00 -05:00
JMR-devandClaude Opus 4.8 7d7fef7b90 chore(security): whitelist font-family on HTML emit
RichTextHtml.inlineCss and baseCss interpolated the font-family CSS value into
the emitted style attribute raw. The whole attribute is escaped (escapeAttr), so
a value can't break out of style="…" or inject a tag, and it isn't reachable
today (the picker only offers the fixed FontRegistry stacks; reply/forward
flattens sender HTML to plaintext first) — but a non-registry value would let
`;`/`:` inject a sibling CSS declaration inside the attribute.

Constrain the emitted font-family to a safe charset (the characters a real font
stack uses — letters, digits, spaces, commas, quotes, hyphens, periods,
underscores), dropping anything else instead of emitting it raw. All seven
bundled FontRegistry stacks are within this set, so the built-in fonts are
unaffected. RichTextHtml stays a pure module (no dependency on the UI-layer
FontRegistry), so the charset restriction is the layer-clean form of the
whitelist.

Test: a RichStyle.FontFamily("Arial; color:red") no longer appears raw in the
emitted HTML (inline and base-style), while a registry stack with quotes/commas
still emits.

Closes #205

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 07:09:33 -05:00
Jason Ross 54c941d771 Merge pull request #203 from JMR-dev/fix-release-uri-grants
fix(compose): release persistable URI permissions for attachments/inline images
2026-07-03 06:48:09 -05:00
JMR-devandClaude Opus 4.8 1a8d6e1f7b fix(compose): release persistable URI permissions for attachments/inline images
The compose attachment picker and inline-image picker call
takePersistableUriPermission on every pick, but nothing ever released
those grants (releasePersistableUriPermission was absent repo-wide). The
app accumulated indefinite read access to every file/photo ever attached
and could hit the per-app persisted-grant cap — after which the take
(swallowed by runCatching) silently fails and a later draft's image
won't reload. (Post-batch security review, Low.)

The picked bytes are copied into the app cache at enqueue
(copyAttachments), so a grant is only truly needed to reload an image
when a *draft* is reopened. New AttachmentUriGrants releases a URI's
grant once no remaining draft or outbox row references it; callers invoke
it after the referencing row is gone:
- MailRepositoryImpl.deleteDraft (a deleted draft can't reopen)
- MailRepositoryImpl.cancelOutboxMessage
- SendWorker after a send succeeds, and when a queued message is dropped
  because its account was removed
A URI still referenced by another live draft/outbox row is kept; a
release of a grant not actually held throws and is swallowed.

Also hardens attachment filenames: sanitizeAttachmentName strips path
separators and ISO control chars (incl. CR/LF) from picked/received
display names before they become on-disk or MIME filenames, so a crafted
name can't traverse directories or inject header lines. Applied in the
compose picker (queryFileName) and outbox/incoming staging.

Tests: pure release-decision (unreferencedUris), the filename sanitizer,
and the repository wiring (deleteDraft/cancelOutboxMessage call the
releaser with the removed row's URIs, after deleting the row).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 03:29:43 -05:00
316 changed files with 36293 additions and 899 deletions
+75 -5
View File
@@ -1,6 +1,6 @@
---
name: preflight
description: Run LibreMail's fast CI gate locally (assembleDebug + testDebugUnitTest + lintDebug + ktlintCheck + detekt) before pushing or opening a PR. Mirrors the merge gate; does NOT run emulator E2E. Use before treating a change as done.
description: Run LibreMail's fast CI gate locally (assembleDebug + testDebugUnitTest + jacocoTestCoverageVerification + compileDebugAndroidTestKotlin + lintDebug + ktlintCheck + detekt) plus the local emulator E2E — the instrumented test class(es) you changed via local_instrumented.py (cold-boot, no Gradle Managed Devices), then the API 37 preview via api37_e2e.py — before pushing or opening a PR. Mirrors the merge gate; CI runs the full multi-API matrix. Use before treating a change as done.
---
# /preflight
@@ -13,6 +13,24 @@ Run the same fast checks CI enforces on every PR, in order, and report the outco
with a JDK/AGP version mismatch, check `java -version` / `JAVA_HOME` and point it at a 17–21
JDK (e.g. Android Studio's bundled JBR) before retrying.
- PowerShell: invoke the wrapper as `.\gradlew`. Git Bash / the Bash tool: `./gradlew`.
- The final two E2E steps each boot an emulator, so the host needs a **free hardware
hypervisor** (Intel VT-x / AMD-V, exposed as WHPX on Windows, KVM on Linux, HVF on macOS).
Shut down VirtualBox, Hyper-V-based VMs, WSL2, Docker Desktop, or any other emulator first — a
VM holding the hypervisor starves the AVD, and it hangs at 0% CPU and never reaches
`sys.boot_completed`. Both steps are hand-provisioned by cross-platform Python scripts (**not**
Gradle Managed Devices, which fail locally on this box — see Steps): `local_instrumented.py`
cold-boots one existing AVD and runs the instrumented class(es) you changed; `api37_e2e.py`
installs + boots the API 37 preview image. The first API 37 run is slow while its system image
downloads. If the host has no accelerated emulator and a device cannot boot, report the E2E
step as not run rather than treating the gate as green.
- Both E2E steps are stdlib-only, cross-platform **Python 3** scripts. `local_instrumented.py`
needs `python3` plus the Android SDK `emulator` + `adb` on `PATH` and an existing AVD (any local
`apiXXDebugAndroidTest` run creates one; override with `LOCAL_INSTRUMENTED_AVD` /
`ANDROID_AVD_HOME`), and it pins `JAVA_HOME` to a JDK 17–21 itself (override with
`LOCAL_INSTRUMENTED_JDK`). `api37_e2e.py` additionally needs the Android SDK command-line tools
(`sdkmanager`/`avdmanager`), located via `ANDROID_SDK_ROOT`/`ANDROID_HOME` (or the per-OS
default: `%LOCALAPPDATA%\Android\Sdk` on Windows, `~/Library/Android/sdk` on macOS,
`~/Android/Sdk` on Linux); it installs the preview system image itself on first run.
## Steps
@@ -20,24 +38,76 @@ Run these, stopping at the first failure:
```bash
./gradlew :app:assembleDebug
./gradlew :app:testDebugUnitTest
./gradlew :app:testDebugUnitTest :app:jacocoTestCoverageVerification
./gradlew :app:compileDebugAndroidTestKotlin
./gradlew :app:lintDebug
./gradlew :app:ktlintCheck :app:detekt
python3 .claude/skills/preflight/local_instrumented.py <your.Changed.TestClass>[,<Class2>,...] # local instrumented/E2E, cold-boot (no GMD)
python3 .claude/skills/preflight/api37_e2e.py # api37 preview E2E (hand-provisioned; on Windows: py or python)
```
`jacocoTestCoverageVerification` runs right after `testDebugUnitTest` because it reads that
task's JVM exec data — it enforces the whole-app **no-regression line-coverage floor (currently
0.84)**, so a coverage regression is caught locally instead of only in CI (the exact class of
failure that reached CI on #367).
`compileDebugAndroidTestKotlin` compiles the `androidTest` source set — the E2E/instrumented
tests — without needing an emulator. `assembleDebug`, `testDebugUnitTest`, and `lintDebug` never
compile that source set, so a change that breaks it (e.g. an instrumented test calling a UI API
that just changed signature) passes the rest of this gate locally yet fails CI.
`ktlintCheck` + `detekt` are exactly what CI's **Static analysis** job runs — they cover the
`test`/`androidTest` source sets that `lintDebug` does not, so a style violation there fails the
merge gate even when the build and lint are green. Add `--continue` to any command (e.g.
`:app:ktlintCheck :app:detekt --continue`) to collect every failure in one pass instead of
stopping at the first.
The two E2E steps run last because they are the slowest, and they run through **cross-platform
Python scripts, not Gradle Managed Devices (GMD)**. GMD's `apiXXDebugAndroidTest` tasks fail
locally on this box — GMD's AVD-snapshot step times out under the AEHD 2.2 hypervisor
(`AvdSnapshotHandler$EmulatorSnapshotCannotCreatedException`), cycling for hours — so preflight
does **not** call them (issue #269/#281). `local_instrumented.py` instead cold-boots one existing
AVD by hand with `-no-snapshot` (the exact `connectedDebugAndroidTest` technique CI and
`api37_e2e.py` use), runs `:app:connectedDebugAndroidTest` filtered to the instrumented class(es)
you pass, then tears the emulator down and verifies no orphaned `qemu` process is left behind
(exit 3 if one survives). Pass the instrumented/E2E class(es) you actually changed
(comma-separated, no spaces) — the full ~114-test suite tends to wedge mid-run locally, so
targeted runs are deliberate; the whole suite across every API level is CI's job.
`api37_e2e.py` then runs the instrumented/E2E suite on the **API 37 preview** emulator. API 37 has no Gradle
Managed Device — its only published system image is the nonstandard `android-37.0` /
`google_apis_ps16k` pairing, which neither `ManagedVirtualDevice`'s `apiLevel` (Int) nor
`apiPreview` (codename) DSL resolves (see the comment above `testOptions.managedDevices` in
`app/build.gradle.kts`), so there is no `api37DebugAndroidTest` task. The script hand-provisions
it essentially the way CI's `e2e-preview` job does — installs the image with `sdkmanager`,
creates the AVD with `avdmanager`, cold-boots it headless, waits for `sys.boot_completed`, runs
`:app:connectedDebugAndroidTest`, then kills the emulator and deletes the AVD. The emulator flags
match `e2e-preview` with one deliberate local exception: the **GPU mode**. CI uses
`-gpu swiftshader_indirect` (software rendering, deterministic on a headless CI runner); the
local run uses `-gpu auto-no-window`, which renders on the host GPU — faster, and the mode that
boots cleanly on a dev machine.
Both E2E steps are the E2E that preflight runs locally and both must pass; CI then fans the full
instrumented/E2E suite out across the whole matrix (API 29–36 in `e2e`, plus API 37 in
`e2e-preview`). Keep `local_instrumented.py` pointed at the instrumented class(es) you changed,
and keep `api37_e2e.py` in lockstep with the `e2e-preview` job in `.github/workflows/ci.yml`
(same image string + emulator flags, apart from the intentional GPU-mode difference noted above).
The local gate no longer runs the GMD `apiXXDebugAndroidTest` tasks (they are unusable locally —
see above); full multi-API coverage stays CI's job.
## Reporting
- If everything passes, say so plainly (e.g. "preflight green: build, unit tests, lint, ktlint, detekt").
- If everything passes, say so plainly (e.g. "preflight green: build, unit tests, lint, ktlint, detekt, local + api37 E2E").
- On failure, surface the actual Gradle error and point at the relevant report:
- unit tests → `app/build/reports/tests/testDebugUnitTest/`
- lint → `app/build/reports/lint-results-debug.html`
- ktlint → `app/build/reports/ktlint/` (per source set, e.g. `ktlintTestSourceSetCheck/`)
- detekt → `app/build/reports/detekt/`
- Do **not** run emulator/E2E (`connectedDebugAndroidTest`) here — that's CI's job unless the
user explicitly asks.
- local + api37 E2E → `app/build/reports/androidTests/connected/` (the `connectedDebugAndroidTest`
report both scripts drive — the later run overwrites the earlier); each emulator's own boot log
is at the temp path the script prints (`local_instrumented.py` also exits **3** if it leaves an
orphaned `qemu`, **4** if the emulator never booted).
- Run the instrumented class(es) you changed via `local_instrumented.py`, plus the **API 37
preview** via `api37_e2e.py`; the full multi-API matrix (API 29–37) stays CI's job. If the host
has no accelerated emulator and a device cannot boot (see the hypervisor note in Preconditions),
report that E2E could not run rather than treating the gate as green.
+399
View File
@@ -0,0 +1,399 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-3.0-or-later
"""Hand-provision an API 37 (Android 17, preview) emulator, run LibreMail's instrumented/E2E
suite against it, then tear the emulator and AVD down. Invoked by the /preflight skill as the
third (api37) E2E step, alongside the api35/api36 Gradle Managed Devices.
WHY THIS SCRIPT EXISTS
----------------------
API 37's only published system image is the nonstandard "android-37.0" / google_apis_ps16k
(16 KB page size) pairing. AGP's Gradle Managed Device DSL can only build an
"android-<apiLevel:Int>" package id (apiLevel = 37 -> "android-37") or an
"android-<apiPreview:codename>" one -- neither resolves to "android-37.0" -- so there is NO
api37DebugAndroidTest task to run. This script custom-provisions the emulator with
sdkmanager / avdmanager / emulator directly, closely mirroring CI's `e2e-preview` job in
.github/workflows/ci.yml (same system image string, same provisioning/boot sequence, same
emulator flags EXCEPT the GPU mode -- CI uses `-gpu swiftshader_indirect` for headless
determinism, while this local run uses `-gpu auto-no-window` to render on the host GPU, which
is faster; see start_emulator). Keep the two in lockstep on everything but that GPU flag: when a
stable, GMD-compatible API 37 image ships, delete this script, fold api37 into
testOptions.managedDevices, and fold 37 into CI's `e2e` matrix (dropping the `e2e-preview` job).
Pure standard library, cross-platform (Windows / Linux / macOS): tool paths and executable
suffixes are resolved per-OS, and `.bat` launchers are wrapped through `cmd /c` on Windows.
HYPERVISOR REQUIREMENT
----------------------
The emulator boots with `-accel on`, so it needs a FREE hardware hypervisor (Intel VT-x / AMD-V,
exposed as WHPX on Windows, KVM on Linux, HVF on macOS). If VirtualBox, Hyper-V, WSL2, Docker
Desktop, or another emulator is holding it, `-accel on` fails or the AVD hangs at 0% CPU and never
reaches sys.boot_completed. Shut those down before running preflight.
JDK: the final Gradle step needs a JDK 17-21 daemon (AGP 9.2 fails on JDK 25+), same as the rest
of preflight -- point JAVA_HOME at a 17-21 JDK before invoking.
"""
from __future__ import annotations
import argparse
import os
import platform
import shutil
import subprocess
import sys
import tempfile
import time
from pathlib import Path
# Defaults mirror .github/workflows/ci.yml e2e-preview (env.API37_IMAGE, env.ANDROID_PLATFORM,
# env.ANDROID_BUILD_TOOLS) and its avdmanager invocation. Do not diverge without updating ci.yml.
API37_IMAGE = "system-images;android-37.0;google_apis_ps16k;x86_64"
PLATFORM_PKG = "platforms;android-37.0"
BUILD_TOOLS = "build-tools;37.0.0"
AVD_NAME = "api37"
DEVICE_PROFILE = "pixel_2"
BOOT_TIMEOUT = 300
# GPU mode: the ONE deliberate divergence from CI's e2e-preview (which uses `swiftshader_indirect`
# for headless determinism). Locally we render on the host GPU -- faster, and the mode that boots
# cleanly on a dev machine. See start_emulator. Kept as a constant so start_emulator and the
# boot-failure diagnostics dump report the same value.
GPU_MODE = "auto-no-window"
IS_WINDOWS = os.name == "nt"
BAT = ".bat" if IS_WINDOWS else ""
EXE = ".exe" if IS_WINDOWS else ""
def cmd(tool: str, *args: str) -> list[str]:
"""Build an argv list, wrapping Windows `.bat`/`.cmd` launchers through `cmd /c`."""
if IS_WINDOWS and tool.lower().endswith((".bat", ".cmd")):
return ["cmd", "/c", tool, *args]
return [tool, *args]
def find_sdk_root() -> str:
candidates = [os.environ.get("ANDROID_SDK_ROOT"), os.environ.get("ANDROID_HOME")]
system = platform.system()
if system == "Windows":
local = os.environ.get("LOCALAPPDATA")
if local:
candidates.append(os.path.join(local, "Android", "Sdk"))
elif system == "Darwin":
candidates.append(os.path.expanduser("~/Library/Android/sdk"))
else:
candidates.append(os.path.expanduser("~/Android/Sdk"))
for candidate in candidates:
if candidate and os.path.isdir(candidate):
return os.path.abspath(candidate)
raise RuntimeError(
"Android SDK not found. Set ANDROID_SDK_ROOT (or ANDROID_HOME) to your SDK location."
)
def resolve_tool(sdk_root: str, rel_paths: list[list[str]], name: str) -> str:
for rel in rel_paths:
path = os.path.join(sdk_root, *rel)
if os.path.isfile(path):
return path
raise RuntimeError(
f"Could not find {name} under {sdk_root}. "
"Install the Android SDK command-line tools + emulator."
)
def run_sdkmanager(sdkmanager: str, args: list[str]) -> None:
# Feed a stream of "y" so any unaccepted (incl. preview) license prompt is auto-accepted; this
# is the non-interactive equivalent of the CI runner having licenses pre-accepted.
result = subprocess.run(cmd(sdkmanager, *args), input="y\n" * 50, text=True, check=False)
if result.returncode != 0:
raise RuntimeError(f"sdkmanager failed (exit {result.returncode}) for: {' '.join(args)}")
def create_avd(avdmanager: str, emulator: str) -> None:
print(f"Creating AVD '{AVD_NAME}' from {API37_IMAGE} (device: {DEVICE_PROFILE})...")
# "no" answers avdmanager's "create a custom hardware profile?" prompt, mirroring CI.
result = subprocess.run(
cmd(avdmanager, "create", "avd", "-n", AVD_NAME, "-k", API37_IMAGE,
"-d", DEVICE_PROFILE, "--force"),
input="no\n", text=True, check=False,
)
if result.returncode != 0:
raise RuntimeError(f"avdmanager create avd failed (exit {result.returncode}).")
print("AVDs visible to the emulator:")
subprocess.run(cmd(emulator, "-list-avds"), check=False)
def start_emulator(emulator: str, emu_log: Path, attempt: int) -> subprocess.Popen:
print(f"Starting API 37 emulator (attempt {attempt})...")
# Flags mirror .github/workflows/ci.yml e2e-preview (cold headless boot, hardware accel
# required, no cameras), with ONE deliberate LOCAL exception -- the GPU mode (GPU_MODE above:
# CI uses `-gpu swiftshader_indirect`, deterministic on a headless CI runner; locally we render
# on the host GPU -- faster, and the mode that boots cleanly on a dev machine). `-verbose -debug
# init,avd_config,kernel` turns emulator boot logging on by default (mirrors CI) so a boot flake
# is diagnosable from $EMU_LOG; it is DIAGNOSTICS ONLY and does not change any boot-affecting
# flag. Keep everything except the GPU mode in lockstep with that job.
flags = [
"-avd", AVD_NAME,
"-no-window", "-no-audio", "-no-boot-anim", "-no-snapshot", "-accel", "on",
"-gpu", GPU_MODE, "-camera-back", "none", "-camera-front", "none",
"-verbose", "-debug", "init,avd_config,kernel",
]
log = open(emu_log, "wb") # noqa: SIM115 - handed to the child; closed in the parent below
try:
proc = subprocess.Popen(cmd(emulator, *flags), stdout=log, stderr=subprocess.STDOUT)
finally:
log.close() # the child has inherited its own fd; the parent's copy is no longer needed
return proc
def wait_for_boot(adb: str, proc: subprocess.Popen, timeout: int) -> bool:
subprocess.run(cmd(adb, "start-server"), check=False)
deadline = time.monotonic() + timeout
# Phase 1 -- the literal `adb wait-for-device`, bounded so a dead emulator can't hang the run
# (returns as soon as the emulator registers). Mirrors CI's `adb wait-for-device`.
try:
subprocess.run(cmd(adb, "wait-for-device"), timeout=max(1, deadline - time.monotonic()),
check=False)
except subprocess.TimeoutExpired:
return False
if proc.poll() is not None:
return False
# Phase 2 -- poll sys.boot_completed until it flips to 1 (mirrors CI's getprop loop).
while time.monotonic() < deadline:
if proc.poll() is not None:
print(f"Emulator process exited during boot (code {proc.returncode}).",
file=sys.stderr)
return False
out = subprocess.run(cmd(adb, "shell", "getprop", "sys.boot_completed"),
capture_output=True, text=True, check=False)
if out.stdout.strip() == "1":
return True
time.sleep(2)
return False
def stop_emulator(adb: str | None, proc: subprocess.Popen | None) -> None:
if adb:
subprocess.run(cmd(adb, "emu", "kill"), check=False,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
time.sleep(2)
if proc and proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
def tail(path: Path, lines: int = 80) -> None:
try:
with open(path, "r", errors="replace") as handle:
content = handle.readlines()[-lines:]
print("--- emulator.log (tail) ---")
print("".join(content))
except OSError:
pass
def _accel_check(emulator: str) -> str:
"""`emulator -accel-check` output -- the accelerator status (WHPX / KVM / HVF availability)."""
try:
out = subprocess.run(cmd(emulator, "-accel-check"), capture_output=True, text=True,
check=False)
return (out.stdout + out.stderr).strip() or f"(no output; exit {out.returncode})"
except OSError as exc:
return f"(accel-check failed: {exc})"
def _kvm_status() -> str:
"""/dev/kvm presence (Linux). Off-Linux the accelerator is WHPX/HVF -- see -accel-check."""
if os.path.exists("/dev/kvm"):
return "/dev/kvm present"
return f"/dev/kvm absent (expected off-Linux; platform={platform.system()})"
def _mem_info() -> str:
"""Free/total memory. Reads /proc/meminfo on Linux (where CI runs); best-effort elsewhere."""
try:
meminfo = Path("/proc/meminfo")
if meminfo.exists():
wanted = {"MemTotal", "MemFree", "MemAvailable"}
lines = [line.strip() for line in meminfo.read_text().splitlines()
if line.split(":", 1)[0] in wanted]
if lines:
return "; ".join(lines)
except OSError:
pass
return f"(memory stats unavailable on {platform.system()})"
def _disk_info(path: Path) -> str:
"""Free/total disk for the filesystem holding `path` (cross-platform via shutil.disk_usage)."""
try:
usage = shutil.disk_usage(path)
gib = 1024 ** 3
return f"total={usage.total / gib:.1f}GiB free={usage.free / gib:.1f}GiB ({path})"
except OSError as exc:
return f"(disk stats unavailable: {exc})"
def start_logcat(adb: str, logcat_log: Path, attempt: int) -> subprocess.Popen | None:
"""Background `adb wait-for-device logcat -v time` to a file. wait-for-device blocks until the
device registers, so streaming starts the moment the emulator appears and captures the whole
boot. Mirrors CI's e2e-preview logcat capture; appended (with a header) per boot attempt."""
try:
with open(logcat_log, "a") as marker:
marker.write(f"===== logcat (attempt {attempt}) =====\n")
log = open(logcat_log, "ab") # noqa: SIM115 - child inherits fd; parent copy closed below
try:
return subprocess.Popen(cmd(adb, "wait-for-device", "logcat", "-v", "time"),
stdout=log, stderr=subprocess.STDOUT)
finally:
log.close() # the child has inherited its own fd; the parent's copy is no longer needed
except OSError as exc:
print(f"WARNING: could not start logcat capture: {exc}", file=sys.stderr)
return None
def stop_logcat(proc: subprocess.Popen | None) -> None:
if proc and proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
def dump_diagnostics(adb: str, emulator: str, emu_log: Path, avd_home: Path, attempt: int) -> None:
"""Print boot diagnostics + a concise failure summary to the console -- the local mirror of CI's
e2e-preview boot-timeout dump (accel/KVM/GPU/mem/disk/AVD config + emulator.log tail). Local
runs PRINT these; CI uploads the same set as an artifact and prints only the concise summary."""
accel = _accel_check(emulator)
kvm = _kvm_status()
config_ini = avd_home / f"{AVD_NAME}.avd" / "config.ini"
print(f"===== API 37 boot diagnostics (attempt {attempt}) =====")
print("--- adb devices ---")
subprocess.run(cmd(adb, "devices"), check=False)
print(f"--- emulator -accel-check ---\n{accel}")
print(f"--- KVM/hypervisor ---\n{kvm}")
print(f"--- GPU mode ---\n{GPU_MODE}")
print(f"--- free memory ---\n{_mem_info()}")
print(f"--- free disk ---\n{_disk_info(Path(tempfile.gettempdir()))}")
print("--- AVD config.ini ---")
try:
print(config_ini.read_text(errors="replace"))
except OSError as exc:
print(f"(could not read {config_ini}: {exc})")
# Concise failure summary (mirrors CI): accel/KVM status + the last 50 lines of emulator.log.
print(f"----- BOOT FAILURE SUMMARY (attempt {attempt}) -----")
print(f"accel-check: {accel}")
print(f"kvm: {kvm}")
tail(emu_log, 50)
def main() -> int:
parser = argparse.ArgumentParser(
description="Hand-provision + run the API 37 preview E2E suite.")
parser.add_argument("--boot-timeout", type=int, default=BOOT_TIMEOUT,
help="Seconds to wait for the emulator to reach sys.boot_completed.")
args = parser.parse_args()
sdk_root = find_sdk_root()
print(f"Using Android SDK at: {sdk_root}")
sdkmanager = resolve_tool(sdk_root, [
["cmdline-tools", "latest", "bin", "sdkmanager" + BAT],
["cmdline-tools", "bin", "sdkmanager" + BAT],
["tools", "bin", "sdkmanager" + BAT],
], "sdkmanager")
avdmanager = resolve_tool(sdk_root, [
["cmdline-tools", "latest", "bin", "avdmanager" + BAT],
["cmdline-tools", "bin", "avdmanager" + BAT],
["tools", "bin", "avdmanager" + BAT],
], "avdmanager")
# Pin ANDROID_AVD_HOME so avdmanager (writes it) and the emulator (reads it) agree on the AVD
# dir -- the same fix CI's e2e-preview applies to avoid "Unknown AVD name [api37]".
avd_home = Path.home() / ".android" / "avd"
avd_home.mkdir(parents=True, exist_ok=True)
os.environ["ANDROID_AVD_HOME"] = str(avd_home)
emu_log = Path(tempfile.gettempdir()) / "libremail-api37-emulator.log"
logcat_log = Path(tempfile.gettempdir()) / "libremail-api37-logcat.txt"
try:
logcat_log.unlink() # start fresh; start_logcat appends (with a header) per attempt
except OSError:
pass
adb: str | None = None
proc: subprocess.Popen | None = None
logcat_proc: subprocess.Popen | None = None
test_exit = 1
try:
# 1. Install the SDK platform, build-tools, platform-tools, emulator, and preview image.
print(f"Installing SDK packages + API 37 preview system image ({API37_IMAGE})...")
run_sdkmanager(sdkmanager, ["--licenses"])
run_sdkmanager(sdkmanager, [PLATFORM_PKG, BUILD_TOOLS, "platform-tools", "emulator",
API37_IMAGE])
# adb + emulator are only guaranteed present after the install above.
adb = resolve_tool(sdk_root, [["platform-tools", "adb" + EXE]], "adb")
emulator = resolve_tool(sdk_root, [["emulator", "emulator" + EXE]], "emulator")
# 2. Create the AVD, mirroring CI.
create_avd(avdmanager, emulator)
# 3. Cold-boot headless, retrying once (mirrors CI's two-attempt boot loop). Diagnostics
# (logcat capture + a boot-timeout dump) are ADDITIVE -- the retry/boot-wait is unchanged.
booted = False
for attempt in (1, 2):
proc = start_emulator(emulator, emu_log, attempt)
# Capture logcat from device registration onward (mirrors CI); killed on failure.
logcat_proc = start_logcat(adb, logcat_log, attempt)
if wait_for_boot(adb, proc, args.boot_timeout):
booted = True
break
print(f"API 37 emulator did not boot within {args.boot_timeout}s (attempt {attempt}).",
file=sys.stderr)
dump_diagnostics(adb, emulator, emu_log, avd_home, attempt)
stop_logcat(logcat_proc)
logcat_proc = None
stop_emulator(adb, proc)
proc = None
time.sleep(5)
if not booted:
raise RuntimeError("API 37 preview emulator failed to boot after 2 attempts.")
# 4. Dismiss the keyguard, then run the instrumented/E2E suite against the booted emulator.
subprocess.run(cmd(adb, "shell", "input", "keyevent", "82"), check=False)
repo_root = Path(__file__).resolve().parents[3]
gradlew = repo_root / ("gradlew.bat" if IS_WINDOWS else "gradlew")
print(f"Running :app:connectedDebugAndroidTest against {AVD_NAME}...")
test_exit = subprocess.run(
cmd(str(gradlew), ":app:connectedDebugAndroidTest", "--stacktrace"),
cwd=str(repo_root), check=False,
).returncode
except Exception as exc: # noqa: BLE001 - top-level guard so teardown always runs
print(f"ERROR: {exc}", file=sys.stderr)
test_exit = 1
finally:
# 5. Always tear the emulator down and delete the AVD, even on failure.
print("Tearing down API 37 emulator and AVD...")
stop_logcat(logcat_proc)
stop_emulator(adb, proc)
subprocess.run(cmd(avdmanager, "delete", "avd", "-n", AVD_NAME), check=False,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(f"(emulator boot log: {emu_log})")
print(f"(logcat: {logcat_log})")
if test_exit != 0:
print(f"api37 connectedDebugAndroidTest failed (exit {test_exit}).", file=sys.stderr)
return test_exit
print("api37 E2E passed.")
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,62 @@
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
# `local_instrumented.py` — reliable local instrumented/E2E runs
A helper for running LibreMail's instrumented / E2E tests **locally** without Gradle
Managed Devices (GMD). Cross-platform, pure Python 3 standard library (Windows / Linux /
macOS). Companion to `api37_e2e.py`; born from issue #269, ported from bash to Python in
issue #281 so it runs the same on the Windows primary dev box and on \*nix — no Git Bash,
no `jq`, no `taskkill`-vs-`kill` gaps.
## Usage
```bash
# from anywhere — invoke the script by path (Windows: use `py` or `python`):
python .claude/skills/preflight/local_instrumented.py org.libremail.ui.compose.ComposeScreenE2ETest
# multiple classes (comma-separated, no spaces):
python .claude/skills/preflight/local_instrumented.py org.libremail.a.FooTest,org.libremail.b.BarTest
```
The script is CWD-independent: it resolves its own repo/worktree root from its script
location (three directories up from `.claude/skills/preflight`) and runs gradlew there, so
it always builds *that* tree's `:app` — never whatever tree your shell happens to be
sitting in. This matters most when you have several worktrees checked out side by side; run
the copy of this script that lives inside the worktree you want to test, regardless of your
current directory (issue #284).
It cold-boots **one** emulator (`-no-snapshot`, no GMD), waits for `sys.boot_completed`,
runs `:app:connectedDebugAndroidTest` filtered to the class(es) you pass, then tears the
emulator down and verifies no orphaned `qemu` process is left behind (exit **3** if one is).
## Why (short version)
- **GMD is broken locally on this box.** `apiXXDebugAndroidTest` fails in GMD's snapshot
step — `AvdSnapshotHandler$EmulatorSnapshotCannotCreatedException: Snapshot creation
timed out` (AEHD 2.2 can't save/load the snapshot). The emulator itself boots fine; only
GMD's snapshot machinery is broken. CI is unaffected (it uses `connectedDebugAndroidTest`,
not GMD). See issue #269.
- **Keep runs targeted.** The full ~114-test suite tends to wedge mid-run on this machine;
small, targeted class sets do not. That's why the script requires an explicit class list —
run only what you changed. The full matrix is CI's job.
- **Emulator hygiene is mandatory.** A hung `adb emu kill` leaves a detached qemu VM
(`qemu-system-x86_64-headless.exe` on Windows, a `qemu-system-*` process on \*nix);
accumulated orphans have frozen this machine. The script force-kills stragglers before
booting and after tearing down, and fails loudly (exit 3) if a zombie survives. The
orphan-kill is abstracted per-OS (`taskkill /F /IM …` on Windows, `pkill -f qemu-system`
on \*nix), and teardown always runs — even on Ctrl-C / error / SIGTERM (try/finally +
atexit + SIGINT/SIGTERM handlers).
Exit codes: **0** pass · **2** usage/precondition failure · **3** a qemu zombie survived
teardown · **4** emulator never booted · any other non-zero = `connectedDebugAndroidTest`'s
own test-failure exit code.
See the module docstring at the top of `local_instrumented.py` for the full rationale,
requirements, and the `LOCAL_INSTRUMENTED_*` environment overrides (AVD name, JDK home,
boot timeout, …).
## Requirements
`python3` (Windows: `py`/`python`); Android SDK `emulator` + `adb` on `PATH`; a JDK
**17–21** (AGP 9.2 fails on 25+ — the script pins `JAVA_HOME` to a known JDK 21, overridable
via `LOCAL_INSTRUMENTED_JDK`); and a free hardware hypervisor (shut down VirtualBox / other
VMs first).
@@ -0,0 +1,451 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-3.0-or-later
"""local_instrumented.py -- reliable LOCAL instrumented / E2E test runner for LibreMail.
Usage: local_instrumented.py <fully.qualified.TestClass>[,<Class2>,...]
Example:
python .claude/skills/preflight/local_instrumented.py \
org.libremail.ui.compose.ComposeScreenE2ETest
python .claude/skills/preflight/local_instrumented.py \
org.libremail.ui.compose.ComposeScreenE2ETest,org.libremail.ui.compose.RecipientChipTest
Cross-platform (Windows / Linux / macOS), pure standard library. Companion to
``api37_e2e.py``; ported from the original ``local_instrumented.sh`` (issue #281) so the
helper runs the same on the Windows primary dev box and on *nix -- no Git Bash, no ``jq``,
no ``taskkill`` vs ``kill`` portability gaps.
WHY THIS SCRIPT EXISTS (issue #269)
------------------------------------
On this machine (Windows + the AEHD 2.2 hypervisor) the Gradle Managed Device (GMD)
instrumented tasks -- ``apiXXDebugAndroidTest`` -- FAIL during setup. GMD tries to
save/load an AVD *snapshot* and AEHD 2.2 cannot complete it:
AvdSnapshotHandler$EmulatorSnapshotCannotCreatedException: Snapshot creation timed out
GMD retries the snapshot ~5x, rebooting the AVD each time -- that endless reboot is the
"cycling" that eats hours. The emulator ITSELF is healthy (8 GB RAM, sys.boot_completed=1,
shell-responsive); only GMD's snapshot step is broken. So every LOCAL GMD task is affected:
the coverage lanes and the /preflight api35/api36 steps. CI is unaffected -- it uses
reactivecircus/android-emulator-runner + ``connectedDebugAndroidTest``, never GMD.
THE RELIABLE LOCAL PATH (this script):
Cold-boot ONE emulator by hand with ``-no-snapshot`` (no GMD, no snapshot machinery),
then run ``:app:connectedDebugAndroidTest`` -- the exact technique CI and ``api37_e2e.py``
already use. We reuse a GMD-provisioned AVD by name so we don't re-download a system
image; GMD re-provisions its own copy on its next run, so the ``-wipe-data`` cold boot
here does not disturb it.
KEEP RUNS TARGETED -- THE ~114-TEST MID-SUITE WEDGE
---------------------------------------------------
Running the WHOLE instrumented suite (~114 tests) via ``connectedDebugAndroidTest`` on this
box tends to wedge partway through -- the emulator stops making progress mid-run. Small,
targeted class sets do NOT hit that wedge. That is why this helper takes an explicit
``<fully.qualified.TestClass>[,...]`` argument and filters the run with
``-Pandroid.testInstrumentationRunnerArguments.class=...`` instead of running everything.
Run the class(es) you actually changed; do not use this to run the full suite (that is
CI's / preflight's job across the API matrix).
FREEZE / HYGIENE RATIONALE -- WHY THE ORPHAN-KILL + TEARDOWN VERIFY ARE MANDATORY
--------------------------------------------------------------------------------
A hung ``adb emu kill`` (or an interrupted run) leaves a detached qemu VM process behind
(``qemu-system-x86_64-headless.exe`` on Windows; a ``qemu-system-*`` process on *nix).
These orphans do not show up in ``adb devices``, they keep holding the hypervisor + RAM,
and accumulated orphans have FROZEN this machine outright. So this script:
* PREAMBLE -- force-kills any pre-existing qemu/emulator processes and resets the adb
server BEFORE booting, so we always start from a clean slate.
* TEARDOWN -- ``adb emu kill``, kill the launcher we spawned, then re-check for ANY
surviving emulator/qemu process and force-kill it (the ``-no-window`` emulator
can leave a sibling ``emulator.exe`` that briefly outlives the qemu VM). Teardown
runs even on Ctrl-C / error / SIGTERM (try/finally + atexit + SIGINT/SIGTERM
handlers) and is idempotent.
* VERIFY -- if a qemu process is STILL alive after the force-kill, the script exits
non-zero (code 3) so the leak is never silently ignored.
Never leave an emulator running after this script; if it exits 3, hunt the zombie down by
hand (Windows: ``tasklist | findstr qemu`` then ``taskkill /F /IM
qemu-system-x86_64-headless.exe``; *nix: ``pgrep -fa qemu-system`` then ``pkill -f
qemu-system``).
CROSS-PLATFORM PROCESS KILL
---------------------------
Listing and force-killing the emulator/qemu processes is abstracted per-OS (see
``list_procs`` / ``force_kill``): Windows uses ``tasklist`` + ``taskkill /F /IM <image>``;
*nix uses ``ps ax`` + ``pkill -f qemu-system`` (alongside the graceful ``adb emu kill``).
The qemu VM is the freeze-causing orphan on every platform.
EXIT CODES (preserved from local_instrumented.sh)
0 tests passed
2 usage / precondition failure
3 a qemu zombie survived teardown -- clean it up by hand before the next run
4 emulator never reached sys.boot_completed
<n> connectedDebugAndroidTest's own non-zero exit code (test failures)
REQUIREMENTS
* Android SDK ``emulator`` + ``adb`` on PATH.
* A JDK 17-21 for the Gradle daemon -- AGP 9.2 fails on JDK 25+. This script pins
JAVA_HOME to a known JDK 21 (override with LOCAL_INSTRUMENTED_JDK) because the ambient
JAVA_HOME on the primary box points at JDK 25.
* A free hardware hypervisor (VT-x/WHPX/AEHD/KVM/HVF). Shut down VirtualBox / other VMs
first or the AVD hangs at 0% CPU and never reaches sys.boot_completed.
Overridable via environment (defaults target the primary Windows dev box):
LOCAL_INSTRUMENTED_AVD AVD name to boot (dev36_google_apis_x86_64_Pixel_2)
ANDROID_AVD_HOME AVD home dir (C:/Users/jasonross/.android/avd/gradle-managed)
LOCAL_INSTRUMENTED_JDK JDK 17-21 home (Eclipse Adoptium jdk-21.0.11.10-hotspot)
LOCAL_INSTRUMENTED_SERIAL adb serial (emulator-5554)
LOCAL_INSTRUMENTED_BOOT_TIMEOUT boot wait seconds (300)
"""
from __future__ import annotations
import argparse
import atexit
import os
import shutil
import signal
import subprocess
import sys
import tempfile
import time
from pathlib import Path
IS_WINDOWS = os.name == "nt"
# ---- configuration (env-overridable; defaults are correct for the primary dev box) ------
AVD_NAME = os.environ.get("LOCAL_INSTRUMENTED_AVD", "dev36_google_apis_x86_64_Pixel_2")
AVD_HOME = os.environ.get("ANDROID_AVD_HOME", "C:/Users/jasonross/.android/avd/gradle-managed")
JDK_HOME = os.environ.get(
"LOCAL_INSTRUMENTED_JDK", "C:/Program Files/Eclipse Adoptium/jdk-21.0.11.10-hotspot"
)
SERIAL = os.environ.get("LOCAL_INSTRUMENTED_SERIAL", "emulator-5554")
BOOT_TIMEOUT = int(os.environ.get("LOCAL_INSTRUMENTED_BOOT_TIMEOUT", "300"))
# Windows qemu/emulator image names (see FREEZE / HYGIENE above). The ``-headless`` variant is
# what a ``-no-window`` emulator launches; the plain qemu name is swept too, belt-and-suspenders.
QEMU_IMAGE = "qemu-system-x86_64-headless.exe"
QEMU_IMAGE_ALT = "qemu-system-x86_64.exe"
EMULATOR_IMAGE = "emulator.exe"
REPO_ROOT = Path(__file__).resolve().parents[3] # .claude/skills/preflight -> repo root
GRADLEW = REPO_ROOT / ("gradlew.bat" if IS_WINDOWS else "gradlew")
EMU_LOG = Path(tempfile.gettempdir()) / "libremail-local-instrumented-emulator.log"
class _RunState:
"""Mutable run state shared by main(), teardown(), the atexit hook and the signal
handlers -- mirrors the bash globals EMU_PID / TEST_EXIT / ZOMBIE / TEARDOWN_DONE."""
def __init__(self) -> None:
self.proc: subprocess.Popen | None = None
self.test_exit = 1
self.zombie = False
self.teardown_done = False
_STATE = _RunState()
def log(msg: str) -> None:
print(f"\n=== {msg} ===")
def warn(msg: str) -> None:
print(f"WARNING: {msg}", file=sys.stderr)
def die(msg: str) -> None:
"""Print an error and exit 2 (usage / precondition failure). Called before the teardown
backstops are armed, so nothing has booted and there is nothing to tear down."""
print(f"ERROR: {msg}", file=sys.stderr)
sys.exit(2)
def cmd(tool: str, *args: str) -> list[str]:
"""Build an argv list, wrapping Windows ``.bat``/``.cmd`` launchers (e.g. gradlew.bat)
through ``cmd /c`` -- matching api37_e2e.py. ``.exe`` tools pass through unchanged."""
if IS_WINDOWS and tool.lower().endswith((".bat", ".cmd")):
return ["cmd", "/c", tool, *args]
return [tool, *args]
def _run_quiet(argv: list[str]) -> None:
"""Run a command, discarding output and swallowing any error -- teardown/kill helpers
must always make progress."""
try:
subprocess.run(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)
except (OSError, subprocess.SubprocessError):
pass
def _run_capture(argv: list[str]) -> str:
try:
return subprocess.run(argv, capture_output=True, text=True, check=False).stdout or ""
except (OSError, subprocess.SubprocessError):
return ""
def list_procs(*needles: str) -> str:
"""Return the lines of currently-running processes whose name/command line contains any
of ``needles`` (case-insensitive); empty string if none. Cross-platform stand-in for the
.sh's ``tasklist | grep``: ``tasklist`` on Windows, ``ps ax`` on *nix."""
out = _run_capture(["tasklist"] if IS_WINDOWS else ["ps", "ax"])
lowered = [n.lower() for n in needles]
return "\n".join(ln for ln in out.splitlines() if any(n in ln.lower() for n in lowered))
def list_qemu() -> str:
return list_procs("qemu")
def list_emu_procs() -> str:
return list_procs("qemu", "emulator")
def force_kill(win_images: list[str], nix_patterns: list[str]) -> None:
"""Best-effort force-kill. Windows: ``taskkill /F /IM <image> ...``. *nix: ``pkill -f
<pattern>`` per pattern. Never raises -- teardown must always make progress."""
if IS_WINDOWS:
argv = ["taskkill", "/F"]
for image in win_images:
argv += ["/IM", image]
_run_quiet(argv)
else:
for pattern in nix_patterns:
_run_quiet(["pkill", "-f", pattern])
def tail(path: Path, lines: int = 40) -> None:
try:
with open(path, "r", errors="replace") as handle:
content = handle.readlines()[-lines:]
print("".join(content), file=sys.stderr, end="")
except OSError:
pass
def teardown() -> None:
"""Kill the emulator and verify no orphaned emulator/qemu process remains. Idempotent --
safe to call from the finally block, the atexit hook and the signal handlers (mirrors the
.sh TEARDOWN_DONE guard). Sets _STATE.zombie if a *qemu* process survives the force-kill --
the machine-freezing case (exit 3)."""
if _STATE.teardown_done:
return
_STATE.teardown_done = True
log("Teardown: killing emulator and verifying no orphaned emulator/qemu remains")
adb = shutil.which("adb")
if adb:
_run_quiet(cmd(adb, "-s", SERIAL, "emu", "kill"))
time.sleep(2)
# Belt-and-suspenders: kill the emulator launcher process we started, if still alive.
proc = _STATE.proc
if proc is not None and proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=1)
except subprocess.TimeoutExpired:
proc.kill()
# Reap any lingering emulator/qemu process, then verify. Unlike the original .sh -- which
# swept qemu ONLY -- we also force-kill the emulator *launcher* image: on Windows the
# ``-no-window`` emulator spawns a sibling ``emulator.exe`` that is NOT the Popen child we
# tracked and outlives both it and the qemu VM by a few seconds, so a qemu-only sweep
# returns while it is still shutting down -- an orphan the freeze-safety rule forbids. So we
# trigger on any emulator-or-qemu survivor and taskkill the launcher too.
if list_emu_procs():
warn("emulator/qemu still present after 'adb emu kill'; force-killing:")
print(list_emu_procs(), file=sys.stderr)
force_kill([QEMU_IMAGE, QEMU_IMAGE_ALT, EMULATOR_IMAGE], ["qemu-system"])
time.sleep(2)
# A surviving QEMU is the machine-freezing zombie (exit 3); a stray launcher is not.
remaining = list_qemu()
if remaining:
warn("qemu ZOMBIE survived teardown -- kill it by hand or the machine may freeze:")
print(remaining, file=sys.stderr)
_STATE.zombie = True
if adb:
_run_quiet(cmd(adb, "kill-server"))
def orphan_kill_preamble(adb: str) -> None:
"""Force-kill any pre-existing qemu/emulator processes and reset the adb server, so we
always cold-boot from a clean slate."""
log("Orphan-kill preamble: ensuring a clean slate before boot")
existing = list_emu_procs()
if existing:
warn("Pre-existing emulator/qemu processes found -- force-killing them first:")
print(existing, file=sys.stderr)
force_kill([QEMU_IMAGE, EMULATOR_IMAGE], ["qemu-system"])
time.sleep(2)
else:
print("No pre-existing qemu/emulator processes.")
_run_quiet(cmd(adb, "kill-server"))
_run_quiet(cmd(adb, "start-server"))
def start_emulator(emulator: str) -> subprocess.Popen:
"""Cold-boot ONE emulator by hand (no GMD, no snapshot), logging to EMU_LOG. Records the
launcher process in _STATE so teardown can reap it even if we are interrupted next."""
log(f"Cold-booting @{AVD_NAME} (no GMD, no snapshot); log -> {EMU_LOG}")
flags = [
f"@{AVD_NAME}",
"-no-window", "-no-snapshot", "-no-boot-anim", "-no-audio",
"-gpu", "auto-no-window", "-cores", "8", "-wipe-data",
]
logf = open(EMU_LOG, "wb") # noqa: SIM115 - handed to the child; parent copy closed below
try:
proc = subprocess.Popen(cmd(emulator, *flags), stdout=logf, stderr=subprocess.STDOUT)
finally:
logf.close() # the child inherited its own fd; the parent's copy is no longer needed
_STATE.proc = proc
print(f"emulator launcher pid={proc.pid}")
return proc
def wait_for_boot(adb: str, proc: subprocess.Popen, timeout: int) -> bool:
"""Poll ``adb get-state`` + ``getprop sys.boot_completed`` until the emulator is up, or the
launcher dies, or ``timeout`` seconds elapse. Mirrors the .sh boot loop."""
print(f"Waiting up to {timeout}s for sys.boot_completed on {SERIAL}...")
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if proc.poll() is not None:
warn("emulator process exited during boot; last log lines:")
tail(EMU_LOG, 40)
return False
state = _run_capture(cmd(adb, "-s", SERIAL, "get-state")).strip()
if state == "device":
booted = _run_capture(
cmd(adb, "-s", SERIAL, "shell", "getprop", "sys.boot_completed")
).strip()
if booted == "1":
return True
time.sleep(3)
return False
def dismiss_keyguard(adb: str) -> None:
# Dismiss the keyguard (mirrors CI + api37_e2e.py). Best-effort: a cold -wipe-data boot
# rarely needs it, and the input service can lose a race right after boot.
_run_quiet(cmd(adb, "-s", SERIAL, "shell", "input", "keyevent", "82"))
def run_tests(test_classes: str) -> int:
"""Run :app:connectedDebugAndroidTest filtered to ``test_classes`` from the repo root
(JAVA_HOME / ANDROID_AVD_HOME are already in the environment)."""
log(f"Running :app:connectedDebugAndroidTest for: {test_classes}")
print(f"JAVA_HOME={os.environ.get('JAVA_HOME', '')}")
return subprocess.run(
cmd(
str(GRADLEW),
":app:connectedDebugAndroidTest",
f"-Pandroid.testInstrumentationRunnerArguments.class={test_classes}",
"--stacktrace",
),
cwd=str(REPO_ROOT),
check=False,
).returncode
def main() -> int:
parser = argparse.ArgumentParser(
prog="local_instrumented.py",
formatter_class=argparse.RawDescriptionHelpFormatter,
description=(
"Cold-boot ONE emulator (no GMD, no snapshot) and run "
":app:connectedDebugAndroidTest filtered to the given instrumented test class(es)."
),
epilog=(
"Keep the class set small and targeted -- the full ~114-test suite tends to wedge\n"
"mid-run on this box (see the module docstring). The full matrix is CI's job.\n"
"Example:\n"
" python .claude/skills/preflight/local_instrumented.py \\\n"
" org.libremail.ui.compose.ComposeScreenE2ETest,org.libremail.ui.compose.RecipientChipTest"
),
)
parser.add_argument(
"test_classes",
metavar="TEST_CLASSES",
help=(
"Comma-separated fully-qualified instrumented test class(es), no spaces "
"(e.g. org.libremail.a.FooTest,org.libremail.b.BarTest)."
),
)
args = parser.parse_args()
# ---- preconditions (before arming teardown; nothing has booted yet) ------------------
emulator = shutil.which("emulator")
adb = shutil.which("adb")
if not emulator:
die("emulator not on PATH (install Android SDK emulator).")
if not adb:
die("adb not on PATH (install Android SDK platform-tools).")
kill_tool = "taskkill" if IS_WINDOWS else "pkill"
if not shutil.which(kill_tool):
die(f"{kill_tool} not found -- required to force-kill orphaned emulator/qemu processes.")
if not GRADLEW.is_file():
die(f"gradlew not found at {GRADLEW}.")
if not os.path.isdir(JDK_HOME):
die(f"JDK 17-21 not found at '{JDK_HOME}'. Set LOCAL_INSTRUMENTED_JDK.")
if not os.path.isfile(os.path.join(AVD_HOME, AVD_NAME + ".ini")):
die(
f"AVD '{AVD_NAME}' not found under '{AVD_HOME}'. "
"Set LOCAL_INSTRUMENTED_AVD / ANDROID_AVD_HOME. "
"(GMD AVDs are created by any local apiXXDebugAndroidTest run.)"
)
# Move gradlew's working dir to this tree's repo root (below) and pin JAVA_HOME/AVD home,
# exactly like the .sh -- the ambient JAVA_HOME on this box points at JDK 25 (AGP-incompatible).
os.environ["JAVA_HOME"] = JDK_HOME
os.environ["ANDROID_AVD_HOME"] = AVD_HOME
# ---- arm teardown backstops BEFORE touching the emulator -----------------------------
# try/finally is the primary path; atexit covers sys.exit()/unhandled-exception exits; the
# signal handlers make SIGINT/SIGTERM tear down too (Python does not raise on SIGTERM by
# default). teardown() is idempotent, so firing from several paths is safe (mirrors the
# .sh's ``trap teardown EXIT INT TERM`` + TEARDOWN_DONE guard).
atexit.register(teardown)
def _signal_teardown(signum: int, _frame: object) -> None:
teardown()
sys.exit(128 + signum)
signal.signal(signal.SIGINT, _signal_teardown)
if hasattr(signal, "SIGTERM"):
signal.signal(signal.SIGTERM, _signal_teardown)
boot_failed = False
try:
orphan_kill_preamble(adb)
proc = start_emulator(emulator)
if wait_for_boot(adb, proc, BOOT_TIMEOUT):
print("Emulator booted.")
dismiss_keyguard(adb)
_STATE.test_exit = run_tests(args.test_classes)
else:
warn(f"Emulator did not reach sys.boot_completed within {BOOT_TIMEOUT}s.")
tail(EMU_LOG, 40)
boot_failed = True
finally:
teardown()
if boot_failed:
return 4
if _STATE.zombie:
warn(
"Exiting 3: a qemu zombie was left behind (see above) -- "
"clean it up before the next run."
)
return 3
if _STATE.test_exit != 0:
warn(
f"connectedDebugAndroidTest failed (exit {_STATE.test_exit}). "
"Report: app/build/reports/androidTests/connected/"
)
return _STATE.test_exit
log(f"PASS -- instrumented tests green for: {args.test_classes}")
return 0
if __name__ == "__main__":
sys.exit(main())
+306
View File
@@ -0,0 +1,306 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-3.0-or-later
"""Hardened Android SDK setup for CI (issue #389).
The dominant merge-blocking flake was the **Set up Android SDK** step
(`android-actions/setup-android`) dying *before* the emulator ever starts:
Wrong version in preinstalled sdkmanager
Warning: ... preparing SDK package Android Emulator: Error reading Zip
content from a SeekableByteChannel.
Error: The process '.../sdkmanager' failed with exit code 1
Two root causes, both a corrupt/truncated download that a bare `sdkmanager`
turns into an un-retried exit 1:
* the action's own **unverified** cmdline-tools re-download (its default
cmdline-tools version rarely matches the runner image's preinstalled one, so
it logs "Wrong version in preinstalled sdkmanager" and re-fetches with *no*
checksum), and
* the action's default ``packages: tools platform-tools`` install (the "SDK
Tools" corrupt zip seen on a #388 preview shard) plus the emulator/platform
package installs.
This module hardens both with **verify -> reject -> retry**, never trusting
sdkmanager's exit code alone:
``bootstrap`` Download the *pinned* Android command-line tools zip, verify it
against a pinned size + SHA-256, and install it to
``$ANDROID_SDK_ROOT/cmdline-tools/<rev>`` -- the exact path
setup-android probes first, so the action reuses our verified
tree and never does its own unverified "Wrong version"
re-download. A size/hash mismatch (corrupt OR wrong version)
=> delete the bad zip + any half-extracted dir => re-download
clean. Only a verified tree is ever left in place, so the
success-gated cache can never bake in a corrupt SDK.
``install`` Run ``sdkmanager --install <packages>`` with retry + backoff.
"Error reading Zip content from a SeekableByteChannel" is a
corrupt package zip, so on failure each requested package's dir
(and sdkmanager's temp/intermediate dirs) is PURGED before the
retry -- forcing a fresh re-download instead of a re-read of the
corrupt file.
stdlib only (urllib/hashlib/zipfile/...), cross-platform, per the repo's "prefer
Python for dev/CI-helper scripts" rule. The pure helpers are unit-tested in
``test_setup_android_sdk.py`` (run by the ``traffic-control-tests`` job); the
full download/install path is validated by CI itself.
"""
from __future__ import annotations
import argparse
import hashlib
import os
import shutil
import subprocess
import sys
import tempfile
import time
import urllib.request
import zipfile
# --- Pinned Android command-line tools (revision 20.0) --------------------
# android-actions/setup-android v4.0.1 defaults to this same build (its
# getVersionShort() maps "14742923" -> "20.0"). We provision it OURSELVES,
# integrity-checked, into the path the action looks for first
# ($ANDROID_SDK_ROOT/cmdline-tools/20.0), so the action finds it, skips its own
# unverified download, and never prints "Wrong version in preinstalled
# sdkmanager".
#
# CLT_SIZE + the SHA-1 are Google's published values for this immutable,
# build-numbered zip (repository2-3.xml). CLT_SHA256 was computed locally from
# bytes that matched BOTH of Google's published values, so it is an authoritative
# integrity pin. A build-numbered URL is immutable, so these never drift; bumping
# the tools means bumping all four constants together.
CLT_VERSION_LONG = "14742923"
CLT_VERSION_SHORT = "20.0"
CLT_URL = (
"https://dl.google.com/android/repository/"
f"commandlinetools-linux-{CLT_VERSION_LONG}_latest.zip"
)
CLT_SIZE = 172789259
CLT_SHA256 = "04453066b540409d975c676d781da1477479dde3761310f1a7eb92a1dfb15af7"
# Total tries (1 initial + retries). Backoff is linear: 10s, 20s, 30s ...
MAX_ATTEMPTS = 4
def log(msg: str) -> None:
print(msg, flush=True)
def warn(msg: str) -> None:
print(f"::warning::{msg}", flush=True)
def error(msg: str) -> None:
print(f"::error::{msg}", flush=True)
def backoff_seconds(attempt: int) -> int:
"""Linear backoff before the next attempt: 10s after attempt 1, 20s after 2..."""
return 10 * attempt
def sdk_root() -> str:
"""The Android SDK root. GitHub-hosted runners preset ANDROID_SDK_ROOT /
ANDROID_HOME to /usr/local/lib/android/sdk; fall back to the SDK's default."""
root = os.environ.get("ANDROID_SDK_ROOT") or os.environ.get("ANDROID_HOME")
if not root:
root = os.path.join(os.path.expanduser("~"), ".android", "sdk")
return root
def sha256_of(path: str) -> str:
h = hashlib.sha256()
with open(path, "rb") as fh:
for chunk in iter(lambda: fh.read(1024 * 1024), b""):
h.update(chunk)
return h.hexdigest()
def verify_download(path, expected_size, expected_sha256):
"""(ok, detail) for a downloaded file: size first (cheap), then SHA-256.
A mismatch means a corrupt/truncated download OR the wrong version -- both
must be rejected and re-fetched."""
if not os.path.exists(path):
return False, "download missing"
actual_size = os.path.getsize(path)
if actual_size != expected_size:
return False, f"size {actual_size} != expected {expected_size}"
actual_sha = sha256_of(path)
if actual_sha != expected_sha256:
return False, f"sha256 {actual_sha} != expected {expected_sha256}"
return True, "ok"
def package_dir(root: str, package: str) -> str:
"""On-disk dir for an sdkmanager package id. sdkmanager lays packages out by
turning the ';' separators into path separators, e.g.
'platforms;android-37.0' -> <root>/platforms/android-37.0, so this is exactly
the tree to purge to force a corrupt package to re-download."""
return os.path.join(root, *package.split(";"))
def _rm(path: str) -> None:
"""Best-effort recursive delete of a file or dir (reject a bad download)."""
if os.path.islink(path) or os.path.isfile(path):
try:
os.remove(path)
except FileNotFoundError:
pass
elif os.path.isdir(path):
shutil.rmtree(path, ignore_errors=True)
def _extract_preserving_perms(zip_path: str, target_dir: str) -> None:
"""Extract a zip, restoring the unix permission bits stored in each entry's
external attributes. ZipFile.extractall drops the executable bit, which would
leave bin/sdkmanager non-executable and break the action's `sdkmanager
--licenses`; Google's zip is unix-built, so external_attr carries the +x."""
with zipfile.ZipFile(zip_path) as zf:
for info in zf.infolist():
extracted = zf.extract(info, target_dir)
mode = (info.external_attr >> 16) & 0o7777
if mode:
os.chmod(extracted, mode)
class _RejectAndRetry(Exception):
"""Internal signal: discard this attempt's download and retry from scratch."""
def bootstrap() -> int:
"""Ensure $ANDROID_SDK_ROOT/cmdline-tools/<rev> is a verified install."""
root = sdk_root()
dest = os.path.join(root, "cmdline-tools", CLT_VERSION_SHORT)
sdkmanager = os.path.join(dest, "bin", "sdkmanager")
if os.path.exists(sdkmanager):
# Cache hit (or already provisioned): the cache is populated only after a
# passing integrity check, so a present tree is trusted -> no re-download.
log(f"cmdline-tools {CLT_VERSION_SHORT} already present at {dest} "
"(cache hit) -- skipping verified download")
return 0
tools_parent = os.path.join(root, "cmdline-tools")
os.makedirs(tools_parent, exist_ok=True)
for attempt in range(1, MAX_ATTEMPTS + 1):
log(f"::group::Download + verify cmdline-tools {CLT_VERSION_SHORT} "
f"(attempt {attempt}/{MAX_ATTEMPTS})")
tmp_zip = os.path.join(tempfile.gettempdir(), f"clt-{CLT_VERSION_LONG}.zip")
# Extract on the SAME filesystem as `dest` so the final move is an atomic
# rename that preserves the restored +x bit on bin/sdkmanager.
tmp_extract = tempfile.mkdtemp(prefix=".clt-extract-", dir=tools_parent)
_rm(tmp_zip)
try:
log(f"Downloading {CLT_URL}")
urllib.request.urlretrieve(CLT_URL, tmp_zip) # noqa: S310 (pinned https)
ok, detail = verify_download(tmp_zip, CLT_SIZE, CLT_SHA256)
if not ok:
warn(f"cmdline-tools integrity check failed: {detail} -- "
"rejecting the bad download and retrying clean")
raise _RejectAndRetry()
log(f"Integrity OK (size {CLT_SIZE}, sha256 {CLT_SHA256})")
_extract_preserving_perms(tmp_zip, tmp_extract)
unpacked = os.path.join(tmp_extract, "cmdline-tools")
if not os.path.isdir(unpacked):
warn("extracted zip has no top-level cmdline-tools/ dir -- retrying")
raise _RejectAndRetry()
_rm(dest) # drop any half-extracted leftover before moving the good tree
shutil.move(unpacked, dest)
# Mirror the action: touch repositories.cfg so sdkmanager is happy.
open(os.path.join(root, "repositories.cfg"), "a", encoding="utf-8").close()
if os.path.exists(sdkmanager):
log(f"Installed verified cmdline-tools to {dest}")
return 0
warn("sdkmanager missing after extract -- retrying")
except _RejectAndRetry:
pass
except Exception as exc: # noqa: BLE001 - any transient error is retryable
warn(f"cmdline-tools bootstrap attempt {attempt} failed: {exc}")
finally:
_rm(tmp_zip)
_rm(tmp_extract)
log("::endgroup::")
if attempt < MAX_ATTEMPTS:
time.sleep(backoff_seconds(attempt))
error(f"Failed to provision verified cmdline-tools after {MAX_ATTEMPTS} attempts")
return 1
def find_sdkmanager(root: str):
"""Locate sdkmanager: our pinned rev first, then the action's `latest`, then PATH."""
candidates = [
os.path.join(root, "cmdline-tools", CLT_VERSION_SHORT, "bin", "sdkmanager"),
os.path.join(root, "cmdline-tools", "latest", "bin", "sdkmanager"),
]
for candidate in candidates:
if os.path.exists(candidate):
return candidate
return shutil.which("sdkmanager")
def install(packages) -> int:
"""`sdkmanager --install <packages>` with retry + purge-on-corrupt-zip."""
root = sdk_root()
sdkmanager = find_sdkmanager(root)
if not sdkmanager:
error("sdkmanager not found -- run the cmdline-tools bootstrap step first")
return 1
# Feed 'y' repeatedly in case any license needs accepting (setup-android's
# --licenses runs first, but this keeps the step self-contained).
accept = ("y\n" * 32).encode()
for attempt in range(1, MAX_ATTEMPTS + 1):
log(f"::group::sdkmanager --install {' '.join(packages)} "
f"(attempt {attempt}/{MAX_ATTEMPTS})")
result = subprocess.run([sdkmanager, "--install", *packages], input=accept)
log("::endgroup::")
if result.returncode == 0:
log(f"Installed SDK packages: {' '.join(packages)}")
return 0
warn(f"sdkmanager attempt {attempt} failed (exit {result.returncode}) -- "
"purging partial/corrupt packages before retry")
# REJECT: a corrupt package zip must be re-downloaded, not re-read. Purge
# each requested package's dir + sdkmanager's temp/intermediate dirs so
# the retry starts clean.
for pkg in packages:
_rm(package_dir(root, pkg))
_rm(os.path.join(root, ".temp"))
_rm(os.path.join(root, ".downloadIntermediates"))
if attempt < MAX_ATTEMPTS:
time.sleep(backoff_seconds(attempt))
error(f"sdkmanager failed to install {list(packages)} after {MAX_ATTEMPTS} attempts")
log("--- sdkmanager --list_installed ---")
subprocess.run([sdkmanager, "--list_installed"])
return 1
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Hardened Android SDK setup for CI (issue #389)."
)
sub = parser.add_subparsers(dest="command", required=True)
sub.add_parser(
"bootstrap",
help="Download + SHA-256-verify the pinned Android command-line tools.",
)
installer = sub.add_parser(
"install",
help="sdkmanager --install with retry + purge-on-corrupt-zip.",
)
installer.add_argument("packages", nargs="+", help="sdkmanager package ids")
return parser
def main(argv) -> int:
args = build_parser().parse_args(argv)
if args.command == "bootstrap":
return bootstrap()
if args.command == "install":
return install(args.packages)
return 2 # pragma: no cover - argparse requires a subcommand
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+178
View File
@@ -0,0 +1,178 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-3.0-or-later
"""Unit tests for the pure helpers of setup_android_sdk.py (no network, no SDK).
Covers the bits whose correctness is load-bearing for the hardening in #389:
the package-id -> purge-path mapping (a wrong mapping would purge the wrong dir),
the size/SHA-256 integrity gate (verify -> reject), the pinned-constant
self-consistency, the backoff schedule, sdkmanager discovery, and that extraction
restores the executable bit that sdkmanager needs. The full download/install path
is exercised by CI itself."""
from __future__ import annotations
import hashlib
import os
import stat
import tempfile
import unittest
import zipfile
import setup_android_sdk as sdk
class PackageDirTests(unittest.TestCase):
def test_semicolon_ids_map_to_nested_dirs(self):
root = os.path.join("opt", "sdk")
self.assertEqual(
sdk.package_dir(root, "platforms;android-37.0"),
os.path.join(root, "platforms", "android-37.0"),
)
self.assertEqual(
sdk.package_dir(root, "build-tools;37.0.0"),
os.path.join(root, "build-tools", "37.0.0"),
)
self.assertEqual(
sdk.package_dir(root, "system-images;android-37.0;google_apis_ps16k;x86_64"),
os.path.join(root, "system-images", "android-37.0", "google_apis_ps16k", "x86_64"),
)
def test_flat_ids_map_to_single_dir(self):
root = os.path.join("opt", "sdk")
self.assertEqual(sdk.package_dir(root, "emulator"), os.path.join(root, "emulator"))
self.assertEqual(
sdk.package_dir(root, "platform-tools"), os.path.join(root, "platform-tools")
)
def test_purge_target_stays_under_root(self):
# The purge path must never escape the SDK root (no absolute/`..` package ids).
root = os.path.abspath(os.path.join("opt", "sdk"))
target = os.path.abspath(sdk.package_dir(root, "platforms;android-37.0"))
self.assertTrue(target.startswith(root + os.sep))
class VerifyDownloadTests(unittest.TestCase):
def _write(self, data: bytes) -> str:
fd, path = tempfile.mkstemp()
with os.fdopen(fd, "wb") as fh:
fh.write(data)
self.addCleanup(lambda: os.path.exists(path) and os.remove(path))
return path
def test_accepts_matching_size_and_hash(self):
data = b"correct-cmdline-tools-bytes"
path = self._write(data)
ok, detail = sdk.verify_download(path, len(data), hashlib.sha256(data).hexdigest())
self.assertTrue(ok, detail)
self.assertEqual(detail, "ok")
def test_rejects_wrong_size_before_hashing(self):
data = b"truncated"
path = self._write(data)
ok, detail = sdk.verify_download(path, len(data) + 1, hashlib.sha256(data).hexdigest())
self.assertFalse(ok)
self.assertIn("size", detail)
def test_rejects_corrupt_bytes_with_right_size(self):
good = b"aaaaaaaa"
corrupt = b"aaaaaaab" # same length, different content (silent corruption)
path = self._write(corrupt)
ok, detail = sdk.verify_download(path, len(good), hashlib.sha256(good).hexdigest())
self.assertFalse(ok)
self.assertIn("sha256", detail)
def test_rejects_missing_file(self):
ok, detail = sdk.verify_download(
os.path.join(tempfile.gettempdir(), "does-not-exist-clt.zip"), 1, "0" * 64
)
self.assertFalse(ok)
class PinnedConstantsTests(unittest.TestCase):
def test_url_embeds_the_pinned_build_number(self):
self.assertIn(sdk.CLT_VERSION_LONG, sdk.CLT_URL)
self.assertTrue(sdk.CLT_URL.startswith("https://"))
self.assertTrue(sdk.CLT_URL.endswith("_latest.zip"))
def test_sha256_is_a_full_hex_digest(self):
self.assertEqual(len(sdk.CLT_SHA256), 64)
int(sdk.CLT_SHA256, 16) # raises if not hex
self.assertEqual(sdk.CLT_SHA256, sdk.CLT_SHA256.lower())
def test_size_is_positive(self):
self.assertGreater(sdk.CLT_SIZE, 0)
class BackoffTests(unittest.TestCase):
def test_backoff_is_linear_and_increasing(self):
seq = [sdk.backoff_seconds(a) for a in range(1, sdk.MAX_ATTEMPTS + 1)]
self.assertEqual(seq, [10, 20, 30, 40][: sdk.MAX_ATTEMPTS])
self.assertEqual(seq, sorted(seq))
class SdkRootTests(unittest.TestCase):
def test_prefers_android_sdk_root_over_home(self):
with _env(ANDROID_SDK_ROOT="/a/sdk-root", ANDROID_HOME="/b/home"):
self.assertEqual(sdk.sdk_root(), "/a/sdk-root")
def test_falls_back_to_android_home(self):
with _env(ANDROID_SDK_ROOT=None, ANDROID_HOME="/b/home"):
self.assertEqual(sdk.sdk_root(), "/b/home")
class FindSdkManagerTests(unittest.TestCase):
def test_prefers_pinned_revision_dir(self):
with tempfile.TemporaryDirectory() as root:
pinned = os.path.join(root, "cmdline-tools", sdk.CLT_VERSION_SHORT, "bin")
latest = os.path.join(root, "cmdline-tools", "latest", "bin")
for d in (pinned, latest):
os.makedirs(d)
open(os.path.join(d, "sdkmanager"), "w").close()
self.assertEqual(
sdk.find_sdkmanager(root),
os.path.join(pinned, "sdkmanager"),
)
class ExtractPermsTests(unittest.TestCase):
@unittest.skipUnless(os.name == "posix", "unix exec bit only meaningful on POSIX")
def test_executable_bit_is_restored(self):
with tempfile.TemporaryDirectory() as work:
zip_path = os.path.join(work, "clt.zip")
with zipfile.ZipFile(zip_path, "w") as zf:
info = zipfile.ZipInfo("cmdline-tools/bin/sdkmanager")
info.external_attr = 0o755 << 16 # -rwxr-xr-x, as Google's zip stores it
zf.writestr(info, "#!/bin/sh\n")
out = os.path.join(work, "out")
sdk._extract_preserving_perms(zip_path, out)
mode = os.stat(os.path.join(out, "cmdline-tools", "bin", "sdkmanager")).st_mode
self.assertTrue(mode & stat.S_IXUSR, "sdkmanager must be executable after extract")
class _env:
"""Context manager to set/clear env vars for a test, restoring them after."""
def __init__(self, **values):
self._values = values
self._saved = {}
def __enter__(self):
for key, value in self._values.items():
self._saved[key] = os.environ.get(key)
if value is None:
os.environ.pop(key, None)
else:
os.environ[key] = value
return self
def __exit__(self, *exc):
for key, previous in self._saved.items():
if previous is None:
os.environ.pop(key, None)
else:
os.environ[key] = previous
return False
if __name__ == "__main__":
unittest.main()
+509
View File
@@ -0,0 +1,509 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-3.0-or-later
"""Unit tests for the pure decision core of traffic_control.py (no network).
Covers: priority resolution (P-label / broken / draft / default P5), PASS 1
preemption (P0 reclaims all strictly-lower; ANY higher PR reclaims a broken/draft
lower run; P1-P9 never bump a *normal* lower run; self / main / equal-or-higher
never cancelled), PASS 2 hold-back (yield to strictly-higher with an active run;
same-level running-first then oldest-first), and a few end-to-end decision
scenarios.
Also covers the --mode trigger scheduler core (issue #349): head-SHA run
classification (absent/cancelled => needy; success/failure => not needy),
select_triggers (priority order, oldest-first fairness, inflight cap, fork skip,
P0 bypasses-cap-and-preempts), and a liveness/anti-starvation simulation proving
every eligible PR is triggered within a bounded number of passes."""
from __future__ import annotations
import json
import unittest
import traffic_control as tc
from traffic_control import PullRequest
def pr(number, labels=(), *, draft=False, created_at="", status=tc.NONE, run_ids=()):
"""Terse PullRequest builder for tests."""
return PullRequest(
number=number,
labels=tuple(labels),
is_draft=draft,
created_at=created_at,
run_status=status,
run_ids=tuple(run_ids),
)
class EffectivePriorityTests(unittest.TestCase):
def test_no_labels_defaults_to_p5(self):
self.assertEqual(tc.effective_priority(pr(1)), 5)
def test_non_priority_labels_ignored_default_p5(self):
self.assertEqual(tc.effective_priority(pr(1, ["bug", "enhancement"])), 5)
def test_single_p_label(self):
self.assertEqual(tc.effective_priority(pr(1, ["P3"])), 3)
self.assertEqual(tc.effective_priority(pr(1, ["P0"])), 0)
def test_lowest_numbered_p_label_wins(self):
self.assertEqual(tc.effective_priority(pr(1, ["P4", "P1", "P7"])), 1)
def test_broken_is_bottom_p10_overriding_p0(self):
self.assertEqual(tc.effective_priority(pr(1, ["broken", "P0"])), 10)
def test_draft_is_bottom_p10_overriding_p0(self):
self.assertEqual(tc.effective_priority(pr(1, ["P0"], draft=True)), 10)
def test_draft_and_broken_still_p10(self):
self.assertEqual(tc.effective_priority(pr(1, ["broken"], draft=True)), 10)
def test_double_digit_pseudo_label_is_not_a_priority(self):
# Only P0-P9 count (regex ^P[0-9]$); "P10" is not a valid priority label.
self.assertEqual(tc.effective_priority(pr(1, ["P10"])), 5)
def test_priority_label_text(self):
self.assertEqual(tc.priority_label(0), "P0")
self.assertEqual(tc.priority_label(5), "P5")
self.assertIn("bottom", tc.priority_label(10))
class RunsToCancelTests(unittest.TestCase):
def test_non_p0_self_does_not_bump_normal_lower_run(self):
# P1-P9 never preempt a *normal* strictly-lower run — they yield instead.
me = pr(1, ["P1"])
others = [pr(2, ["P5"], status=tc.RUNNING, run_ids=[200])]
self.assertEqual(tc.runs_to_cancel(me, [me, *others]), [])
def test_non_p0_self_reclaims_broken_lower_run(self):
# Any higher-priority PR (not just P0) may reclaim a broken target's runner.
me = pr(1, ["P3"])
broken = pr(2, ["broken"], status=tc.RUNNING, run_ids=[200])
self.assertEqual(tc.runs_to_cancel(me, [me, broken]), [200])
def test_non_p0_self_reclaims_draft_lower_run(self):
# A draft is not merge-ready — its run is likewise reclaimable by any higher PR.
me = pr(1, ["P3"])
draft = pr(2, [], draft=True, status=tc.QUEUED, run_ids=[200])
self.assertEqual(tc.runs_to_cancel(me, [me, draft]), [200])
def test_broken_self_does_not_cancel_equal_broken(self):
# Both effective P10 — the equal-or-higher invariant still forbids cancelling.
me = pr(1, ["broken"])
peer = pr(2, ["broken"], status=tc.RUNNING, run_ids=[200])
self.assertEqual(tc.runs_to_cancel(me, [me, peer]), [])
def test_bottom_self_preempts_nothing(self):
# A broken/draft PR (P10) is the bottom: nothing is strictly-lower, so it
# cancels neither a higher (P5) nor an equal (P10) run.
me = pr(1, [], draft=True) # P10
prs = [
me,
pr(2, ["P5"], status=tc.RUNNING, run_ids=[200]), # higher
pr(3, ["broken"], status=tc.RUNNING, run_ids=[300]), # equal P10
]
self.assertEqual(tc.runs_to_cancel(me, prs), [])
def test_p0_cancels_strictly_lower_active_runs(self):
me = pr(1, ["P0"])
low = pr(2, ["P5"], status=tc.RUNNING, run_ids=[200])
queued = pr(3, ["P9"], status=tc.QUEUED, run_ids=[300])
self.assertEqual(
sorted(tc.runs_to_cancel(me, [me, low, queued])), [200, 300])
def test_p0_cancels_broken_and_draft_lower_runs(self):
me = pr(1, ["P0"])
broken = pr(2, ["broken"], status=tc.RUNNING, run_ids=[200])
draft = pr(3, ["P2"], draft=True, status=tc.RUNNING, run_ids=[300])
self.assertEqual(
sorted(tc.runs_to_cancel(me, [me, broken, draft])), [200, 300])
def test_p0_never_cancels_equal_priority_p0(self):
me = pr(1, ["P0"])
peer = pr(2, ["P0"], status=tc.RUNNING, run_ids=[200])
self.assertEqual(tc.runs_to_cancel(me, [me, peer]), [])
def test_p0_never_cancels_self(self):
me = pr(1, ["P0"], status=tc.RUNNING, run_ids=[100])
self.assertEqual(tc.runs_to_cancel(me, [me]), [])
def test_p0_excludes_own_run_id_defensively(self):
me = pr(1, ["P0"], status=tc.RUNNING, run_ids=[100])
# A lower PR that somehow reports our own run id must not be cancelled.
low = pr(2, ["P5"], status=tc.RUNNING, run_ids=[100, 200])
self.assertEqual(
tc.runs_to_cancel(me, [me, low], self_run_id=100), [200])
def test_p0_skips_lower_with_no_active_run(self):
me = pr(1, ["P0"])
idle = pr(2, ["P5"], status=tc.NONE, run_ids=[])
self.assertEqual(tc.runs_to_cancel(me, [me, idle]), [])
class WaitBlockersTests(unittest.TestCase):
def test_p0_never_waits(self):
me = pr(1, ["P0"])
higher = pr(2, ["P0"], status=tc.RUNNING) # nothing outranks P0 anyway
self.assertEqual(tc.wait_blockers(me, [me, higher]), [])
def test_yields_to_strictly_higher_with_active_run(self):
me = pr(2, ["P5"], status=tc.RUNNING)
higher = pr(1, ["P2"], status=tc.RUNNING)
blockers = tc.wait_blockers(me, [me, higher])
self.assertEqual([b.number for b in blockers], [1])
self.assertEqual(blockers[0].kind, "higher-priority")
def test_does_not_yield_to_higher_without_active_run(self):
me = pr(2, ["P5"], status=tc.RUNNING)
higher_idle = pr(1, ["P2"], status=tc.NONE)
self.assertEqual(tc.wait_blockers(me, [me, higher_idle]), [])
def test_does_not_yield_to_lower_priority(self):
me = pr(1, ["P2"], status=tc.RUNNING)
lower = pr(2, ["P5"], status=tc.RUNNING)
self.assertEqual(tc.wait_blockers(me, [me, lower]), [])
def test_same_level_oldest_running_proceeds(self):
me = pr(1, ["P5"], created_at="2026-07-01T00:00:00Z", status=tc.RUNNING)
newer = pr(2, ["P5"], created_at="2026-07-02T00:00:00Z", status=tc.RUNNING)
self.assertEqual(tc.wait_blockers(me, [me, newer]), [])
def test_same_level_newer_running_yields_to_older(self):
# Coordinator clarification: within a level, older createdAt goes first.
older = pr(1, ["P5"], created_at="2026-07-01T00:00:00Z", status=tc.RUNNING)
me = pr(2, ["P5"], created_at="2026-07-02T00:00:00Z", status=tc.RUNNING)
blockers = tc.wait_blockers(me, [me, older])
self.assertEqual([b.number for b in blockers], [1])
self.assertEqual(blockers[0].kind, "same-level-ahead")
def test_same_level_running_first_beats_older_waiting(self):
# An in-flight peer keeps its place; a not-yet-running OLDER peer does not
# jump ahead of us while we are the one already running.
me = pr(2, ["P5"], created_at="2026-07-02T00:00:00Z", status=tc.RUNNING)
older_waiting = pr(1, ["P5"], created_at="2026-07-01T00:00:00Z", status=tc.NONE)
self.assertEqual(tc.wait_blockers(me, [me, older_waiting]), [])
def test_same_level_waiting_orders_oldest_before_newer(self):
# Neither running: strictly oldest-first among the waiting bucket.
oldest = pr(1, ["P5"], created_at="2026-07-01T00:00:00Z", status=tc.NONE)
middle = pr(2, ["P5"], created_at="2026-07-02T00:00:00Z", status=tc.NONE)
me = pr(3, ["P5"], created_at="2026-07-03T00:00:00Z", status=tc.NONE)
blockers = tc.wait_blockers(me, [oldest, middle, me])
self.assertEqual([b.number for b in blockers], [1, 2])
def test_same_level_queued_counts_as_waiting_ordered_by_age(self):
# A queued peer is "waiting to start", not in-flight: ordered purely by age.
me = pr(1, ["P5"], created_at="2026-07-01T00:00:00Z", status=tc.NONE)
newer_queued = pr(2, ["P5"], created_at="2026-07-02T00:00:00Z", status=tc.QUEUED)
self.assertEqual(tc.wait_blockers(me, [me, newer_queued]), [])
def test_broken_self_yields_to_everyone_active(self):
me = pr(1, ["broken"], status=tc.RUNNING) # effective P10
normal = pr(2, ["P5"], status=tc.RUNNING)
blockers = tc.wait_blockers(me, [me, normal])
self.assertEqual([b.number for b in blockers], [2])
self.assertEqual(blockers[0].kind, "higher-priority")
class EndToEndDecisionTests(unittest.TestCase):
def test_p0_emergency_cancels_lower_and_proceeds(self):
me = pr(10, ["P0"], status=tc.RUNNING, run_ids=[1000])
prs = [
me,
pr(11, ["P2"], status=tc.RUNNING, run_ids=[1100]),
pr(12, ["P5"], status=tc.QUEUED, run_ids=[1200]),
pr(13, ["P0"], status=tc.RUNNING, run_ids=[1300]), # equal — spared
]
dec = tc.decide(me, prs, self_run_id=1000)
self.assertEqual(sorted(dec.cancel_run_ids), [1100, 1200])
self.assertTrue(dec.proceed)
def test_p5_waits_behind_running_higher(self):
me = pr(20, ["P5"], status=tc.RUNNING, run_ids=[2000])
higher = pr(21, ["P2"], status=tc.RUNNING, run_ids=[2100])
dec = tc.decide(me, [me, higher])
self.assertEqual(dec.cancel_run_ids, ()) # not P0 — cancels nothing
self.assertFalse(dec.proceed)
self.assertEqual([b.number for b in dec.blockers], [21])
def test_lone_p5_proceeds(self):
me = pr(30, ["P5"], status=tc.RUNNING, run_ids=[3000])
dec = tc.decide(me, [me])
self.assertEqual(dec.cancel_run_ids, ())
self.assertTrue(dec.proceed)
def test_p5_reclaims_draft_then_waits_behind_higher(self):
# A non-P0 PR can BOTH reclaim a broken/draft lower run (PASS 1) AND still
# yield to a strictly-higher PR (PASS 2) in the same evaluation.
me = pr(40, ["P5"], status=tc.RUNNING, run_ids=[4000])
prs = [
me,
pr(41, ["P2"], status=tc.RUNNING, run_ids=[4100]), # higher — blocks
pr(42, [], draft=True, status=tc.RUNNING, run_ids=[4200]), # draft — reclaimed
]
dec = tc.decide(me, prs)
self.assertEqual(list(dec.cancel_run_ids), [4200])
self.assertFalse(dec.proceed)
self.assertEqual([b.number for b in dec.blockers], [41])
class SnapshotParsingTests(unittest.TestCase):
def test_from_json_label_objects_and_fields(self):
obj = {
"number": 7,
"labels": [{"name": "P3"}, {"name": "bug"}],
"isDraft": True,
"createdAt": "2026-07-01T00:00:00Z",
"runStatus": "running",
"runIds": [42, 43],
}
p = PullRequest.from_json(obj)
self.assertEqual(p.number, 7)
self.assertEqual(p.labels, ("P3", "bug"))
self.assertTrue(p.is_draft)
self.assertEqual(p.run_status, tc.RUNNING)
self.assertEqual(p.run_ids, (42, 43))
self.assertEqual(tc.effective_priority(p), 10) # draft => bottom
def test_from_json_plain_string_labels_and_unknown_status(self):
p = PullRequest.from_json(
{"number": 8, "labels": ["P1"], "runStatus": "bogus"})
self.assertEqual(p.labels, ("P1",))
self.assertEqual(p.run_status, tc.NONE) # unknown -> none
def test_load_snapshot_roundtrip(self):
text = json.dumps({
"self": 2,
"self_run_id": 222,
"prs": [
{"number": 1, "labels": ["P2"], "runStatus": "running",
"runIds": [111]},
{"number": 2, "labels": ["P5"], "runStatus": "running",
"runIds": [222]},
],
})
this_pr, all_prs, self_run_id = tc._load_snapshot(text)
self.assertEqual(this_pr.number, 2)
self.assertEqual(len(all_prs), 2)
self.assertEqual(self_run_id, 222)
# ── --mode trigger scheduler core (issue #349) ───────────────────────────────
def npr(number, labels=("P5",), *, draft=False, created_at="", status=tc.NONE, run_ids=()):
"""Terse builder defaulting to a P5 PR (for the trigger tests)."""
return pr(number, labels, draft=draft, created_at=created_at, status=status,
run_ids=run_ids)
class ClassifyShaRunsTests(unittest.TestCase):
def test_no_runs_is_needy(self):
status, ids, needy = tc.classify_sha_runs([])
self.assertEqual(status, tc.NONE)
self.assertEqual(ids, ())
self.assertTrue(needy) # absent checks => must be triggered
def test_success_verdict_not_needy(self):
_, ids, needy = tc.classify_sha_runs(
[{"status": "completed", "conclusion": "success", "databaseId": 1}])
self.assertEqual(ids, ())
self.assertFalse(needy)
def test_failure_verdict_not_needy(self):
# A real failure is the author's to fix — never auto-retriggered (no fail loop).
_, _, needy = tc.classify_sha_runs(
[{"status": "completed", "conclusion": "failure", "databaseId": 1}])
self.assertFalse(needy)
def test_only_cancelled_is_needy(self):
# Cancelled leaves no verdict → re-trigger so the PR can reach a mergeable state.
_, ids, needy = tc.classify_sha_runs(
[{"status": "completed", "conclusion": "cancelled", "databaseId": 1}])
self.assertEqual(ids, ())
self.assertTrue(needy)
def test_in_progress_is_active_not_needy(self):
status, ids, needy = tc.classify_sha_runs(
[{"status": "in_progress", "conclusion": None, "databaseId": 9}])
self.assertEqual(status, tc.RUNNING)
self.assertEqual(ids, (9,))
self.assertFalse(needy)
def test_queued_is_active_not_needy(self):
status, ids, needy = tc.classify_sha_runs(
[{"status": "queued", "conclusion": None, "databaseId": 8}])
self.assertEqual(status, tc.QUEUED)
self.assertEqual(ids, (8,))
self.assertFalse(needy)
def test_running_beats_queued_in_status(self):
status, ids, _ = tc.classify_sha_runs([
{"status": "queued", "databaseId": 1},
{"status": "in_progress", "databaseId": 2},
])
self.assertEqual(status, tc.RUNNING)
self.assertEqual(sorted(ids), [1, 2])
def test_cancelled_plus_active_not_needy(self):
# An active run already covers the SHA — cancelled siblings don't make it needy.
_, ids, needy = tc.classify_sha_runs([
{"status": "completed", "conclusion": "cancelled", "databaseId": 1},
{"status": "in_progress", "databaseId": 2},
])
self.assertEqual(ids, (2,))
self.assertFalse(needy)
class SelectTriggersTests(unittest.TestCase):
def test_empty_needy_triggers_nothing(self):
dec = tc.select_triggers([npr(1), npr(2)], set(), max_inflight=3)
self.assertEqual(dec.trigger_numbers, ())
def test_single_needy_triggered(self):
dec = tc.select_triggers([npr(1)], {1}, max_inflight=3)
self.assertEqual(dec.trigger_numbers, (1,))
self.assertEqual(dec.cancel_run_ids, ())
def test_priority_order(self):
prs = [npr(1, ["P5"]), npr(2, ["P2"]), npr(3, ["P8"])]
dec = tc.select_triggers(prs, {1, 2, 3}, max_inflight=3)
self.assertEqual(dec.trigger_numbers, (2, 1, 3)) # P2, P5, P8
def test_same_level_oldest_first(self):
prs = [
npr(1, ["P5"], created_at="2026-07-03T00:00:00Z"),
npr(2, ["P5"], created_at="2026-07-01T00:00:00Z"),
npr(3, ["P5"], created_at="2026-07-02T00:00:00Z"),
]
dec = tc.select_triggers(prs, {1, 2, 3}, max_inflight=3)
self.assertEqual(dec.trigger_numbers, (2, 3, 1)) # oldest createdAt first
def test_cap_limits_triggers(self):
prs = [npr(1, ["P2"]), npr(2, ["P3"]), npr(3, ["P4"])]
dec = tc.select_triggers(prs, {1, 2, 3}, max_inflight=2)
self.assertEqual(dec.trigger_numbers, (1, 2)) # only 2 free slots
self.assertEqual(dec.slots, 2)
def test_inflight_consumes_slots(self):
prs = [
npr(1, ["P5"], status=tc.RUNNING, run_ids=[100]), # inflight — occupies a slot
npr(2, ["P2"]),
npr(3, ["P3"]),
]
dec = tc.select_triggers(prs, {2, 3}, max_inflight=2)
self.assertEqual(dec.slots, 1) # 2 cap - 1 inflight
self.assertEqual(dec.trigger_numbers, (2,)) # highest-priority needy only
self.assertEqual(dec.inflight_numbers, (1,))
def test_running_needy_is_not_retriggered(self):
# Defensive: a PR flagged needy but already running is never a candidate.
prs = [npr(1, ["P5"], status=tc.RUNNING, run_ids=[100])]
dec = tc.select_triggers(prs, {1}, max_inflight=3)
self.assertEqual(dec.trigger_numbers, ())
def test_fork_pr_skipped(self):
dec = tc.select_triggers([npr(1, ["P2"]), npr(2, ["P1"])],
{1, 2}, max_inflight=3, forks={2})
self.assertEqual(dec.trigger_numbers, (1,)) # fork #2 not token-triggerable
self.assertEqual(dec.skipped_fork_numbers, (2,))
def test_p0_bypasses_cap_and_preempts_lower(self):
# Cap full (a P5 running), but a needy P0 still triggers AND preempts the strictly-
# lower running run to free a runner immediately.
prs = [
npr(1, ["P5"], status=tc.RUNNING, run_ids=[500]), # inflight, strictly-lower
npr(2, ["P0"]), # needy emergency
]
dec = tc.select_triggers(prs, {2}, max_inflight=1)
self.assertEqual(dec.slots, 0) # cap is full
self.assertEqual(dec.trigger_numbers, (2,)) # P0 bypasses the cap
self.assertEqual(dec.cancel_run_ids, (500,)) # preempts the lower run
def test_p0_does_not_preempt_equal_priority(self):
prs = [
npr(1, ["P0"], status=tc.RUNNING, run_ids=[500]), # equal P0 — spared
npr(2, ["P0"]), # needy emergency
]
dec = tc.select_triggers(prs, {2}, max_inflight=1)
self.assertEqual(dec.trigger_numbers, (2,))
self.assertEqual(dec.cancel_run_ids, ()) # never preempts an equal P0
def test_needy_numbers_reports_all_candidates_in_order(self):
dec = tc.select_triggers([npr(1, ["P5"]), npr(2, ["P2"])], {1, 2}, max_inflight=1)
self.assertEqual(dec.needy_numbers, (2, 1)) # priority order, cap-independent
self.assertEqual(dec.trigger_numbers, (2,)) # but only 1 slot triggered
def test_zero_cap_is_clamped_to_one(self):
# A misconfigured cap must never stall everything: clamp to >= 1 so at least the
# top-priority needy PR still gets a slot (fail-safe forward progress).
dec = tc.select_triggers([npr(1, ["P5"])], {1}, max_inflight=0)
self.assertEqual(dec.max_inflight, 1)
self.assertEqual(dec.trigger_numbers, (1,))
class TriggerStarvationTests(unittest.TestCase):
def test_every_needy_pr_is_triggered_within_bounded_passes(self):
# Liveness / anti-starvation: with a fixed needy set and cap=2, simulate scheduler
# passes where a triggered PR gains a run (leaves the needy set) and its run finishes
# one pass later (freeing its slot). Mixed priorities prove lower-priority PRs are
# served too — never starved — while higher-priority PRs still go first.
labels = {1: ["P1"], 2: ["P1"], 3: ["P5"], 4: ["P5"],
5: ["P8"], 6: ["P8"], 7: ["P5"]}
created = {n: f"2026-07-{n:02d}T00:00:00Z" for n in labels}
needy = set(labels)
running: dict[int, int] = {} # number -> passes left running
triggered_ever: set[int] = set()
first_pass: dict[int, int] = {}
cap, max_passes = 2, 12
for pass_no in range(1, max_passes + 1):
snap = [
pr(n, labels[n], created_at=created[n],
status=tc.RUNNING if n in running else tc.NONE,
run_ids=[1000 + n] if n in running else [])
for n in labels
]
dec = tc.select_triggers(snap, needy, max_inflight=cap)
for n in dec.trigger_numbers:
triggered_ever.add(n)
first_pass.setdefault(n, pass_no)
needy.discard(n) # gained a run -> no longer needy
running[n] = 1 # occupies a slot for one pass
for n in list(running): # running PRs finish after one pass
running[n] -= 1
if running[n] <= 0:
del running[n]
if not needy and not running:
break
self.assertEqual(triggered_ever, set(labels),
"a PR was starved (never triggered)")
# Priority respected: the two P1s go in the very first pass; everything else later.
self.assertTrue(all(first_pass[n] == 1 for n in (1, 2)))
self.assertTrue(all(first_pass[n] >= 2 for n in (3, 4, 5, 6, 7)))
class TriggerSnapshotParsingTests(unittest.TestCase):
def test_load_trigger_snapshot_roundtrip(self):
text = json.dumps({
"max_inflight": 2,
"needy": [1, 3],
"forks": [3],
"prs": [
{"number": 1, "labels": ["P2"], "createdAt": "2026-07-01T00:00:00Z"},
{"number": 2, "labels": ["P5"], "runStatus": "running", "runIds": [22]},
{"number": 3, "labels": ["P1"], "createdAt": "2026-07-02T00:00:00Z"},
],
})
all_prs, needy, forks, cap = tc._load_trigger_snapshot(text)
self.assertEqual(len(all_prs), 3)
self.assertEqual(needy, {1, 3})
self.assertEqual(forks, {3})
self.assertEqual(cap, 2)
dec = tc.select_triggers(all_prs, needy, max_inflight=cap, forks=forks)
# #2 is inflight (uses a slot); #3 is a fork (skipped); only #1 fits the 1 free slot.
self.assertEqual(dec.inflight_numbers, (2,))
self.assertEqual(dec.skipped_fork_numbers, (3,))
self.assertEqual(dec.trigger_numbers, (1,))
if __name__ == "__main__":
unittest.main()
+805
View File
@@ -0,0 +1,805 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-3.0-or-later
"""CI traffic-controller: priority-based runner orchestration for LibreMail's
`ci.yml`. Extracted out of the old inline-bash `traffic-control` step into a
Python module so the decision logic is developer-legible and, above all, unit-
testable (see `test_traffic_control.py`).
DESIGN: pure decision CORE + thin gh-I/O SHELL
----------------------------------------------
The decisions ("who do we cancel?", "do we proceed or wait?") are pure functions
over plain `PullRequest` snapshots — no network, no clock, no subprocess — so they
can be exercised exhaustively in unit tests. The SHELL (`run_live`) is the only part
that touches `gh`: it gathers the snapshot, applies the cancellations, and runs the
bounded hold-back poll loop. Feed the core a snapshot JSON (`--dry-run`) to see its
decisions with zero network.
TWO MODES
---------
* ``--mode orchestrate`` (default; unchanged behaviour): the in-run `traffic-control`
job of `ci.yml`. Orders runner ACCESS for the PR whose run is already executing —
PASS 1 preemption + PASS 2 hold-back (below). This is `run_live`.
* ``--mode trigger`` (issue #349): the *scheduler* (companion `ci-trigger.yml`, run
after each auto-update and on a cron backstop). It OWNS CI *triggering*: it
(re-)triggers CI for the highest-priority PR(s) whose head SHA has absent/stale
required checks — a few at a time (an inflight cap), in the SAME priority order —
via a `workflow_dispatch`. This is `run_trigger` / the pure `select_triggers`.
WHY --mode trigger EXISTS (issue #349): `autoupdate.yml` now updates PR branches with the
built-in GITHUB_TOKEN instead of a PAT, so an update push no longer auto-retriggers CI
(GitHub's anti-recursion rule) — killing the merge-cascade that cancelled every open PR's
run on every merge. The cost is that a freshly-updated PR's required checks go stale/absent
on its NEW head SHA, so this scheduler deliberately (re-)triggers them in priority order (a
poor-man's merge queue). The dispatch uses the AUTOUPDATE_TOKEN PAT, NOT the built-in
GITHUB_TOKEN: a GITHUB_TOKEN-triggered run is held for MANUAL approval (`action_required`) and
never runs un-attended, whereas a PAT dispatch runs as the authorized owner with no approval gate
(#350's "no PAT needed" claim was wrong — see ci-trigger.yml + issue #351). FAIL-OPEN,
structurally: `ci.yml` KEEPS its `on: pull_request`
trigger, so any human push — and a brand-new PR — always gets CI regardless of this
scheduler; the scheduler only fills the gap left by GITHUB_TOKEN auto-updates and can never
leave a PR un-triggerable. Fork PRs (no token/secret access) are skipped by the scheduler and
left to `on: pull_request`, so they are never wedged either.
ORDER OF OPERATIONS (issue #342)
--------------------------------
1. Effective priority orders everything: the lowest-numbered `P0`-`P9` label present
(P0 = highest), default `P5` if none. A `broken` OR `draft` PR is effectively P10
(bottom, below P9), overriding any P0-P9 label.
2. PASS 1 - preemption: a strictly-lower OTHER PR's in-progress / queued run is
cancelled iff THIS PR is P0 (an emergency reclaims ALL lower runners) OR the target
is broken/draft (a wasted run any higher-priority PR may reclaim). P1-P9 never bump
a *normal* lower run mid-flight — only a P0 does that.
3. PASS 2 - bounded hold-back: a non-P0 PR yields (cancels nothing) to any strictly-
higher-priority OTHER PR that has an active/queued run, and — among its OWN
priority level — to any PR ordered ahead of it (running-first, then oldest by
`createdAt`). It proceeds the moment it is at the front, or when the wait budget
elapses (a PR never blocks itself).
SAFETY INVARIANTS (preserved from the original step)
----------------------------------------------------
* never cancel a run on `main` / a push event — the shell's `gh run list` query
filters `--event pull_request` and drops `headBranch == main`, so only PR-event
runs ever reach the core;
* never cancel THIS PR's own run — skipped by PR number AND by run id;
* never cancel an equal-or-higher-priority PR — only strictly-lower (prio > self).
This is deliberately NOT a merge gate: every gh call is guarded, the shell always
exits 0, and the ci.yml step stays `continue-on-error`, so a hiccup (API error,
missing permission, fork PR) can never fail CI.
Pure standard library, cross-platform (the primary dev box is Windows, where the old
bash + `jq` pipeline had no clean equivalent).
"""
from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
import time
from dataclasses import dataclass, field
# ── Priority model ───────────────────────────────────────────────────────────
BROKEN_LABEL = "broken"
DEFAULT_PRIORITY = 5 # a PR with no P0-P9 label
BOTTOM_PRIORITY = 10 # broken OR draft — below P9
TOP_PRIORITY = 0 # P0, the only priority that preempts
_P_LABEL = re.compile(r"^P([0-9])$") # single digit only, matching the old jq `^P[0-9]$`
# ── Run-status model (normalised from gh's raw run statuses) ─────────────────
RUNNING = "running" # gh status in_progress
QUEUED = "queued" # gh status queued / waiting / requested / pending
NONE = "none" # no active run (completed or absent)
ACTIVE = frozenset({RUNNING, QUEUED})
# For aggregating a branch's overall status from its runs: running beats queued
# beats none (most-active wins). NB: the same-level ORDER (see _ordering_key) is a
# coarser two-bucket split — in-flight (running) vs everything-else-by-age.
_STATUS_RANK = {RUNNING: 0, QUEUED: 1, NONE: 2}
# createdAt sentinel so a PR with an unknown timestamp sorts LAST (never wrongly
# "oldest"/front, so it yields rather than preempts another PR's front slot).
_FAR_FUTURE = "9999-12-31T23:59:59Z"
# Run conclusions that count as a FINAL VERDICT on a head SHA (issue #349, --mode trigger).
# A SHA with one of these is NOT re-triggered: success = green, failure/timeout/etc. = the
# author's to fix — auto-retriggering a real failure would waste runners and could loop.
# Everything else a completed run can report (cancelled / skipped / stale / startup_failure /
# null) is treated as "no verdict", so a SHA whose only runs are those — or that has no run at
# all (absent checks after a GITHUB_TOKEN auto-update) — is NEEDY and gets (re-)triggered.
VERDICT_CONCLUSIONS = frozenset(
{"success", "failure", "timed_out", "action_required", "neutral"}
)
@dataclass(frozen=True)
class PullRequest:
"""A snapshot of one open PR. The pure decision core consumes only these — no
network. `run_ids` are the PR's active (non-completed) CI run ids, already
filtered to pull_request events on a non-main head by the shell that built them."""
number: int
labels: tuple[str, ...] = ()
is_draft: bool = False
created_at: str = ""
run_status: str = NONE
run_ids: tuple[int, ...] = ()
@classmethod
def from_json(cls, obj: dict) -> "PullRequest":
"""Build from a snapshot dict. `labels` may be a list of names or of gh's
label objects (`{"name": ...}`)."""
raw_labels = obj.get("labels") or []
names: list[str] = []
for lab in raw_labels:
if isinstance(lab, dict):
name = lab.get("name")
else:
name = lab
if name:
names.append(str(name))
status = (obj.get("runStatus") or obj.get("run_status") or NONE).lower()
if status not in (RUNNING, QUEUED, NONE):
status = NONE
raw_ids = obj.get("runIds") or obj.get("run_ids") or ()
return cls(
number=int(obj["number"]),
labels=tuple(names),
is_draft=bool(obj.get("isDraft") or obj.get("is_draft")
or obj.get("draft") or False),
created_at=str(obj.get("createdAt") or obj.get("created_at") or ""),
run_status=status,
run_ids=tuple(int(r) for r in raw_ids),
)
@dataclass(frozen=True)
class Blocker:
"""A PR that THIS PR must yield to in PASS 2 (purely informational for logging)."""
number: int
priority: int
kind: str # "higher-priority" | "same-level-ahead"
@dataclass(frozen=True)
class Decision:
"""The full point-in-time decision for THIS PR (used by --dry-run and tests)."""
self_number: int
self_priority: int
cancel_run_ids: tuple[int, ...] = ()
blockers: tuple[Blocker, ...] = ()
proceed: bool = True
# ── Pure decision core (no network / clock / subprocess) ─────────────────────
def effective_priority(pr: PullRequest) -> int:
"""Effective priority: `broken` OR `draft` => 10 (bottom, overriding any P0-P9);
else the lowest-numbered P0-P9 label present; else the default P5."""
if pr.is_draft or BROKEN_LABEL in pr.labels:
return BOTTOM_PRIORITY
nums = [int(m.group(1)) for name in pr.labels if (m := _P_LABEL.match(name))]
return min(nums) if nums else DEFAULT_PRIORITY
def priority_label(prio: int) -> str:
"""Human-readable priority for logs."""
if prio >= BOTTOM_PRIORITY:
return f"P{BOTTOM_PRIORITY} (broken/draft — bottom, below P9)"
return f"P{prio}"
def _ordering_key(pr: PullRequest) -> tuple[int, str, int]:
"""Same-level ordering (issue #342 rule 3): an in-flight (RUNNING) run keeps its
place at the front — a same-level peer never reorders it — then, among the PRs
still waiting to start (QUEUED or no run yet), OLDEST createdAt first (ascending),
then PR number as a stable final tiebreak so the order is fully deterministic."""
in_flight = 0 if pr.run_status == RUNNING else 1
return (in_flight, pr.created_at or _FAR_FUTURE, pr.number)
def runs_to_cancel(
this_pr: PullRequest,
all_prs: list[PullRequest],
*,
self_run_id: int | None = None,
) -> list[int]:
"""PASS 1. Run ids to cancel. A strictly-lower OTHER PR's active (running/queued)
run is cancelled iff keeping it running is wasteful, i.e. EITHER:
* THIS PR is P0 — an emergency reclaims every strictly-lower runner now; OR
* the target is broken/draft (effective priority 10) — its run can't merge /
isn't merge-ready, so ANY higher-priority PR may reclaim its runner.
P1-P9 never cancel a *normal* strictly-lower run — they yield in PASS 2 instead.
Invariants: never cancel self (by number or run id), never cancel an
equal-or-higher-priority PR (only strictly-lower, prio > self)."""
self_prio = effective_priority(this_pr)
to_cancel: list[int] = []
seen: set[int] = set()
for pr in all_prs:
if pr.number == this_pr.number:
continue # never cancel self
target_prio = effective_priority(pr)
if target_prio <= self_prio:
continue # only strictly-lower (skip equal-or-higher)
if pr.run_status not in ACTIVE:
continue # nothing running/queued to cancel
# Strictly lower: preemptible iff we're P0 OR the target is broken/draft
# (a bottom, priority-10, wasted run that any higher PR may reclaim).
if self_prio != TOP_PRIORITY and target_prio < BOTTOM_PRIORITY:
continue # P1-P9 don't bump a *normal* lower run
for rid in pr.run_ids:
if self_run_id is not None and rid == self_run_id:
continue # never cancel our own run
if rid in seen:
continue
seen.add(rid)
to_cancel.append(rid)
return to_cancel
def wait_blockers(this_pr: PullRequest, all_prs: list[PullRequest]) -> list[Blocker]:
"""PASS 2. The PRs THIS PR must yield to right now (empty => proceed). P0 never
yields. Otherwise yield to (a) any strictly-higher-priority OTHER PR with an
active/queued run, and (b) any SAME-priority PR ordered ahead of THIS PR
(running-first, then oldest createdAt)."""
self_prio = effective_priority(this_pr)
if self_prio == TOP_PRIORITY:
return [] # P0 outranks everything — never wait
others = [pr for pr in all_prs if pr.number != this_pr.number]
blockers: list[Blocker] = []
# (a) strictly-higher-priority PRs that actually have an active/queued run.
for pr in others:
p = effective_priority(pr)
if p < self_prio and pr.run_status in ACTIVE:
blockers.append(Blocker(pr.number, p, "higher-priority"))
# (b) same-level ordering: THIS PR proceeds only when it is at the front.
same_level = [pr for pr in others if effective_priority(pr) == self_prio]
same_level.append(this_pr) # this_pr appears exactly once
for pr in sorted(same_level, key=_ordering_key):
if pr.number == this_pr.number:
break # reached self => nobody ahead remains
blockers.append(Blocker(pr.number, self_prio, "same-level-ahead"))
return blockers
def decide(
this_pr: PullRequest,
all_prs: list[PullRequest],
*,
self_run_id: int | None = None,
) -> Decision:
"""Convenience: the full point-in-time decision (both passes) for THIS PR."""
cancels = runs_to_cancel(this_pr, all_prs, self_run_id=self_run_id)
blockers = wait_blockers(this_pr, all_prs)
return Decision(
self_number=this_pr.number,
self_priority=effective_priority(this_pr),
cancel_run_ids=tuple(cancels),
blockers=tuple(blockers),
proceed=not blockers,
)
# ── Pure TRIGGER-decision core (issue #349, --mode trigger) ──────────────────
def classify_sha_runs(runs: list[dict]) -> tuple[str, tuple[int, ...], bool]:
"""PURE. Summarise the CI runs on ONE head SHA. Returns (run_status, active_run_ids,
needy):
* run_status: RUNNING if any run is in progress, else QUEUED if any is queued/pending,
else NONE;
* active_run_ids: databaseIds of the non-completed (running/queued) runs;
* needy: True iff the SHA has NO active run AND NO run with a final VERDICT — i.e. its
required checks are absent/stale (a fresh SHA after a GITHUB_TOKEN auto-update) or
only cancelled/infra-aborted, so the PR cannot merge until CI is (re-)triggered on
that SHA. A success/failure/timeout verdict is NOT needy (green, or the author's to
fix — never auto-retried)."""
status = NONE
active_ids: list[int] = []
has_verdict = False
for r in runs:
raw = (r.get("status") or "").lower()
if raw == "completed":
if (r.get("conclusion") or "").lower() in VERDICT_CONCLUSIONS:
has_verdict = True
continue
norm = _normalise_status(raw) # in_progress -> running; else queued
if _STATUS_RANK[norm] < _STATUS_RANK[status]:
status = norm
rid = r.get("databaseId")
if rid is not None:
active_ids.append(int(rid))
needy = not active_ids and not has_verdict
return status, tuple(active_ids), needy
def _trigger_order_key(pr: PullRequest) -> tuple[int, str, int]:
"""Trigger ordering: highest priority first (lowest effective-priority number), then
OLDEST createdAt first (the longest-waiting PR at a level goes first — the same-level
fairness / anti-starvation rule), then PR number as a stable final tiebreak."""
return (effective_priority(pr), pr.created_at or _FAR_FUTURE, pr.number)
@dataclass(frozen=True)
class TriggerDecision:
"""PURE output of `select_triggers`: which PR(s) the scheduler should (re-)trigger CI
for right now, in order, plus any strictly-lower runs a P0 emergency preempts to free a
runner. Exercised by `--mode trigger --dry-run` and the unit tests."""
trigger_numbers: tuple[int, ...] = ()
cancel_run_ids: tuple[int, ...] = ()
inflight_numbers: tuple[int, ...] = ()
needy_numbers: tuple[int, ...] = ()
skipped_fork_numbers: tuple[int, ...] = ()
slots: int = 0
max_inflight: int = 0
def select_triggers(
all_prs: list[PullRequest],
needy: "set[int] | frozenset[int]",
*,
max_inflight: int,
forks: "set[int] | frozenset[int]" = frozenset(),
) -> TriggerDecision:
"""PURE. Choose the PR(s) to (re-)trigger CI for now — a poor-man's merge queue over the
existing priority model. No network / clock / subprocess, so it is exhaustively unit-
tested (see TestSelectTriggers / TestTriggerStarvation).
* inflight = PRs already running/queued on their head SHA — they occupy the cap.
* candidates = NEEDY PRs (absent/stale checks on their head SHA) that are not already
running and are not forks (forks have no token/secret access — see `run_trigger`).
* order = effective priority, then oldest createdAt, then number (`_trigger_order_key`).
* P0 = EMERGENCY: always triggered, BYPASSING the cap, and it PREEMPTS its strictly-lower
OTHER runs (reusing `runs_to_cancel`) so a runner frees for it immediately.
* P1–P10 fill only the remaining ``slots = max_inflight - len(inflight)``; the rest wait
for a later pass.
STARVATION is bounded, not by aging but structurally: triggering a PR gives its head SHA
a run, so it LEAVES the needy set; between merges the needy set only shrinks, and the
scheduler re-runs on every auto-update plus a cron backstop, so every eligible PR is
triggered within a bounded number of passes (proved by TestTriggerStarvation). Ordering
is still by priority, so higher-priority PRs are simply served first, never exclusively
forever (a served PR stops being needy until its next push/auto-update)."""
cap = max(1, max_inflight)
inflight = [p for p in all_prs if p.run_status in ACTIVE]
candidates = [
p for p in all_prs
if p.number in needy and p.run_status not in ACTIVE and p.number not in forks
]
ordered = sorted(candidates, key=_trigger_order_key)
emergencies = [p for p in ordered if effective_priority(p) == TOP_PRIORITY]
normal = [p for p in ordered if effective_priority(p) != TOP_PRIORITY]
slots = max(0, cap - len(inflight))
chosen = emergencies + normal[:slots] # P0 bypasses the cap; P1–P10 fill free slots
cancel_ids: list[int] = []
seen: set[int] = set()
for emergency in emergencies: # P0 preempts its strictly-lower active runs
for rid in runs_to_cancel(emergency, all_prs):
if rid not in seen:
seen.add(rid)
cancel_ids.append(rid)
return TriggerDecision(
trigger_numbers=tuple(p.number for p in chosen),
cancel_run_ids=tuple(cancel_ids),
inflight_numbers=tuple(sorted(p.number for p in inflight)),
needy_numbers=tuple(p.number for p in ordered),
skipped_fork_numbers=tuple(sorted(n for n in needy if n in forks)),
slots=slots,
max_inflight=cap,
)
# ── gh I/O shell (the only part that touches the network) ────────────────────
def _log(msg: str) -> None:
print(msg, flush=True)
def _gh_json(args: list[str]) -> list | dict | None:
"""Run `gh <args> --json ...` and parse stdout as JSON. Returns None (never
raises) on any failure — the caller fails open."""
try:
proc = subprocess.run(
["gh", *args],
capture_output=True,
text=True,
check=False,
)
except (OSError, ValueError) as exc:
_log(f"::warning::gh invocation failed ({' '.join(args[:2])}): {exc}")
return None
if proc.returncode != 0:
_log(f"::warning::gh exited {proc.returncode} ({' '.join(args[:2])}): "
f"{proc.stderr.strip()}")
return None
try:
return json.loads(proc.stdout or "null")
except json.JSONDecodeError as exc:
_log(f"::warning::could not parse gh JSON ({' '.join(args[:2])}): {exc}")
return None
def _normalise_status(raw: str) -> str:
"""Map a gh run status onto our RUNNING / QUEUED / NONE model."""
if raw == "in_progress":
return RUNNING
if raw == "completed":
return NONE
return QUEUED # queued / waiting / requested / pending
def _runs_by_head(limit: int = 300) -> dict[str, dict]:
"""One bulk `gh run list` -> {headBranch: {"status", "ids"}} for active PR-event
runs. Enforces the 'never cancel main/push' invariant at the source: only
`event == pull_request`, non-`main`, non-completed runs are kept. Active runs are
the most recent, so `limit` most-recent runs comfortably covers them."""
rows = _gh_json([
"run", "list", "--workflow", "ci.yml", "--event", "pull_request",
"--limit", str(limit),
"--json", "databaseId,status,headBranch,event",
])
by_head: dict[str, dict] = {}
for row in rows or []:
if row.get("event") != "pull_request":
continue
head = row.get("headBranch")
if not head or head == "main":
continue
if row.get("status") == "completed":
continue
entry = by_head.setdefault(head, {"status": NONE, "ids": []})
entry["ids"].append(int(row["databaseId"]))
status = _normalise_status(row.get("status", ""))
# running beats queued beats none for the branch's aggregate status.
if _STATUS_RANK[status] < _STATUS_RANK[entry["status"]]:
entry["status"] = status
return by_head
def gather_snapshot(self_pr_number: int) -> tuple[PullRequest | None, list[PullRequest]]:
"""Build (this_pr, all_prs) from live gh data. this_pr is forced to RUNNING —
by definition our own run is in progress while this job executes."""
prs = _gh_json([
"pr", "list", "--state", "open", "--limit", "300",
"--json", "number,headRefName,labels,isDraft,createdAt",
])
if prs is None:
return None, []
runs = _runs_by_head()
all_prs: list[PullRequest] = []
this_pr: PullRequest | None = None
for obj in prs:
head = obj.get("headRefName") or ""
run_info = runs.get(head, {"status": NONE, "ids": []})
number = int(obj["number"])
is_self = number == self_pr_number
pr = PullRequest.from_json({
**obj,
# self is definitionally running (this job is in progress).
"runStatus": RUNNING if is_self else run_info["status"],
"runIds": run_info["ids"],
})
all_prs.append(pr)
if is_self:
this_pr = pr
return this_pr, all_prs
def _cancel_run(run_id: int) -> bool:
try:
proc = subprocess.run(
["gh", "run", "cancel", str(run_id)],
capture_output=True, text=True, check=False,
)
except (OSError, ValueError) as exc:
_log(f"::warning::could not cancel run {run_id}: {exc}")
return False
if proc.returncode == 0:
return True
_log(f"::warning::could not cancel run {run_id} — likely already finished. "
f"{proc.stderr.strip()}")
return False
def _positive_int(env_name: str, default: int) -> int:
raw = os.environ.get(env_name, "")
return int(raw) if raw.isdigit() and int(raw) > 0 else default
def run_live() -> int:
"""The gh-driven shell: gather, PASS 1 (cancel), PASS 2 (bounded hold-back). Always
returns 0 — the traffic-controller must never fail CI."""
event = os.environ.get("GITHUB_EVENT_NAME", "")
self_raw = os.environ.get("SELF_PR", "")
if event != "pull_request" or not self_raw.isdigit():
_log("Not a pull_request event (or no PR number) — nothing to do.")
return 0
self_number = int(self_raw)
self_run_id = int(os.environ["GITHUB_RUN_ID"]) if os.environ.get(
"GITHUB_RUN_ID", "").isdigit() else None
this_pr, all_prs = gather_snapshot(self_number)
if this_pr is None:
_log("::warning::Could not resolve THIS PR from the open-PR list — skipping.")
return 0
self_prio = effective_priority(this_pr)
_log(f"This PR #{self_number} effective priority: {priority_label(self_prio)} "
"(P0 = highest/emergency, P9 = lowest, broken/draft = bottom).")
# ── PASS 1: PREEMPTION (P0 reclaims all lower; anyone reclaims broken/draft) ──
to_cancel = runs_to_cancel(this_pr, all_prs, self_run_id=self_run_id)
if not to_cancel:
if self_prio == TOP_PRIORITY:
_log("P0 emergency — no strictly-lower active runs to cancel.")
else:
_log("No preemptible runs (P1-P9 only reclaim broken/draft lower runs; "
"none active).")
else:
cancelled = 0
for rid in to_cancel:
if _cancel_run(rid):
_log(f" cancelled run {rid} (freed its runner).")
cancelled += 1
_log(f"P0 preemption complete — cancelled {cancelled}/{len(to_cancel)} run(s).")
# ── PASS 2: BOUNDED HOLD-BACK (yield to higher / same-level-ahead) ────────
if self_prio == TOP_PRIORITY:
_log("P0 emergency — not yielding; proceeding immediately.")
return 0
budget = _positive_int("HOLD_BACK_BUDGET_SECONDS", 180)
poll = _positive_int("HOLD_BACK_POLL_SECONDS", 15)
deadline = time.monotonic() + budget
_log(f"{priority_label(self_prio)} — holding back up to {budget}s for higher / "
"earlier same-level PRs (no cancellation).")
while True:
remaining = deadline - time.monotonic()
if remaining <= 0:
_log("Hold-back budget elapsed — proceeding; higher-priority PRs got their "
"head start.")
break
# Refresh OTHER PRs so newly-opened higher-priority PRs are seen mid-wait;
# THIS PR's own identity/priority stays fixed (matching the original).
_, fresh = gather_snapshot(self_number)
if not fresh:
_log("::warning::Could not refresh open PRs — proceeding.")
break
blockers = wait_blockers(this_pr, fresh)
if not blockers:
_log("No higher-priority or earlier same-level PR is ahead — proceeding.")
break
tags = " ".join(f"#{b.number}(P{b.priority},{b.kind})" for b in blockers)
sleep_s = min(poll, int(remaining)) if remaining >= 1 else 0
_log(f"Yielding to: {tags} — re-checking in {sleep_s}s "
f"({int(remaining)}s budget left).")
if sleep_s > 0:
time.sleep(sleep_s)
_log("Hold-back complete — this PR's heavy jobs may now start.")
return 0
# ── --dry-run: feed the pure core a snapshot JSON, print its decisions ───────
def _load_snapshot(text: str) -> tuple[PullRequest, list[PullRequest], int | None]:
data = json.loads(text)
all_prs = [PullRequest.from_json(o) for o in data.get("prs", [])]
self_number = int(data["self"])
self_run_id = data.get("self_run_id")
self_run_id = int(self_run_id) if self_run_id is not None else None
this_pr = next((p for p in all_prs if p.number == self_number), None)
if this_pr is None:
raise ValueError(f"self #{self_number} not present in prs[]")
return this_pr, all_prs, self_run_id
def run_dry(text: str) -> int:
this_pr, all_prs, self_run_id = _load_snapshot(text)
dec = decide(this_pr, all_prs, self_run_id=self_run_id)
_log(f"This PR #{dec.self_number} effective priority: "
f"{priority_label(dec.self_priority)}")
if dec.cancel_run_ids:
why = ("P0 emergency (reclaims all strictly-lower)"
if dec.self_priority == TOP_PRIORITY
else "reclaiming broken/draft lower runs")
_log(f"PASS 1 (preemption): {why} — cancel run ids: "
f"{list(dec.cancel_run_ids)}")
else:
_log("PASS 1 (preemption): nothing to cancel.")
if dec.proceed:
_log("PASS 2 (hold-back): PROCEED — no blockers.")
else:
tags = ", ".join(f"#{b.number}(P{b.priority}, {b.kind})" for b in dec.blockers)
_log(f"PASS 2 (hold-back): WAIT — yielding to: {tags}")
return 0
# ── --mode trigger: the PAT-free scheduler shell (issue #349) ────────────────
def _gh_ok(args: list[str]) -> bool:
"""Run `gh <args>` for its side effect (no JSON parse). Returns True on exit 0; never
raises — the scheduler fails open on any I/O error."""
try:
proc = subprocess.run(["gh", *args], capture_output=True, text=True, check=False)
except (OSError, ValueError) as exc:
_log(f"::warning::gh invocation failed ({' '.join(args[:2])}): {exc}")
return False
if proc.returncode != 0:
_log(f"::warning::gh exited {proc.returncode} ({' '.join(args[:3])}): "
f"{proc.stderr.strip()}")
return False
return True
def _ci_workflow_file() -> str:
return os.environ.get("CI_WORKFLOW_FILE", "ci.yml")
def gather_trigger_snapshot() -> tuple[list[PullRequest], set[int], set[int], dict[int, dict]]:
"""Build (all_prs, needy, forks, meta) from live gh data for the trigger scheduler.
* all_prs: PullRequest snapshots whose run_status / run_ids reflect the runs on each
PR's CURRENT head SHA (so 'inflight' means a live run on the mergeable SHA, never a
stale one on a superseded SHA);
* needy: PR numbers whose head SHA has absent/stale checks (must be (re-)triggered);
* forks: cross-repository PR numbers — no token/secret access, so NOT token-triggerable;
* meta: number -> {headRefName, headRefOid} for the dispatch I/O.
Returns empty structures (never raises) if gh can't be reached — the caller fails open."""
prs = _gh_json([
"pr", "list", "--state", "open", "--limit", "300",
"--json", "number,headRefName,headRefOid,isCrossRepository,labels,isDraft,createdAt",
])
if prs is None:
return [], set(), set(), {}
runs = _gh_json([
"run", "list", "--workflow", _ci_workflow_file(), "--limit", "300",
"--json", "databaseId,status,conclusion,headSha,headBranch,event",
]) or []
by_sha: dict[str, list[dict]] = {}
for row in runs:
sha = row.get("headSha")
if sha:
by_sha.setdefault(sha, []).append(row)
all_prs: list[PullRequest] = []
needy: set[int] = set()
forks: set[int] = set()
meta: dict[int, dict] = {}
for obj in prs:
number = int(obj["number"])
sha = obj.get("headRefOid") or ""
status, run_ids, is_needy = classify_sha_runs(by_sha.get(sha, []))
all_prs.append(PullRequest.from_json(
{**obj, "runStatus": status, "runIds": list(run_ids)}))
meta[number] = {
"headRefName": obj.get("headRefName") or "",
"headRefOid": sha,
}
if obj.get("isCrossRepository"):
forks.add(number)
if is_needy:
needy.add(number)
return all_prs, needy, forks, meta
def _dispatch_ci(pr_number: int, head_ref: str, head_sha: str) -> bool:
"""Trigger `ci.yml` for one PR via a `workflow_dispatch` on the PR's head branch. The
dispatch runs as GH_TOKEN, which ci-trigger.yml sets to the AUTOUPDATE_TOKEN PAT: a run
triggered by the built-in GITHUB_TOKEN is held for MANUAL approval (`action_required`) and
never runs un-attended, so the PAT (authorized owner) is what actually starts the run with no
approval gate (see issue #351). Running on the head branch puts the run's checks on the PR
head SHA, so they satisfy branch protection's required checks."""
if not head_ref:
_log(f"::warning::PR #{pr_number} has no head branch — cannot dispatch; skipping.")
return False
ok = _gh_ok([
"workflow", "run", _ci_workflow_file(), "--ref", head_ref,
"-f", f"pr={pr_number}",
"-f", f"head_sha={head_sha}",
"-f", "reason=traffic-controller",
])
if ok:
short = head_sha[:8] if head_sha else "?"
_log(f" triggered CI for #{pr_number} on {head_ref} (head {short}).")
return ok
def run_trigger() -> int:
"""The scheduler shell (companion `ci-trigger.yml`): pick the highest-priority needy
PR(s) within the inflight cap and (re-)trigger their CI via workflow_dispatch; a P0
emergency additionally preempts its strictly-lower runs. ALWAYS returns 0 — the scheduler
must never wedge CI, and structurally it cannot: `ci.yml` keeps `on: pull_request`, so any
human push (and a brand-new PR) still gets CI independently of this scheduler."""
max_inflight = _positive_int("MAX_INFLIGHT_RUNS", 2)
all_prs, needy, forks, meta = gather_trigger_snapshot()
if not all_prs:
_log("No open PRs (or could not list them) — nothing to trigger.")
return 0
dec = select_triggers(all_prs, needy, max_inflight=max_inflight, forks=forks)
_log(f"Open PRs: {len(all_prs)} | needy (absent/stale checks): {list(dec.needy_numbers)} "
f"| inflight: {list(dec.inflight_numbers)} | cap {dec.max_inflight}, "
f"free slots {dec.slots}.")
if dec.skipped_fork_numbers:
_log(f"Fork PR(s) needing CI left to `on: pull_request` (no token access — not "
f"wedged): {list(dec.skipped_fork_numbers)}.")
for rid in dec.cancel_run_ids: # P0 emergency preemption
if _cancel_run(rid):
_log(f" P0 preemption: cancelled lower run {rid} (freed its runner).")
if not dec.trigger_numbers:
_log("Nothing to trigger this pass (no needy PR fits a free slot).")
return 0
triggered = 0
for number in dec.trigger_numbers:
info = meta.get(number, {})
if _dispatch_ci(number, info.get("headRefName", ""), info.get("headRefOid", "")):
triggered += 1
_log(f"Trigger pass complete — dispatched {triggered}/{len(dec.trigger_numbers)} "
"run(s) in priority order.")
return 0
def _load_trigger_snapshot(
text: str,
) -> tuple[list[PullRequest], set[int], set[int], int]:
data = json.loads(text)
all_prs = [PullRequest.from_json(o) for o in data.get("prs", [])]
needy = {int(n) for n in data.get("needy", [])}
forks = {int(n) for n in data.get("forks", [])}
max_inflight = int(data.get("max_inflight", 2))
return all_prs, needy, forks, max_inflight
def run_trigger_dry(text: str) -> int:
all_prs, needy, forks, max_inflight = _load_trigger_snapshot(text)
dec = select_triggers(all_prs, needy, max_inflight=max_inflight, forks=forks)
_log(f"Trigger decision (cap {dec.max_inflight}, free slots {dec.slots}):")
_log(f" inflight (occupying the cap): {list(dec.inflight_numbers)}")
_log(f" needy candidates (priority order): {list(dec.needy_numbers)}")
if dec.skipped_fork_numbers:
_log(f" skipped forks (no token access): {list(dec.skipped_fork_numbers)}")
if dec.cancel_run_ids:
_log(f" P0 preemption — cancel run ids: {list(dec.cancel_run_ids)}")
_log(f" => TRIGGER (in priority order): {list(dec.trigger_numbers)}")
return 0
def main(argv: list[str] | None = None) -> int:
# Emit UTF-8 regardless of the host console so the log typography is stable on
# the UTF-8 CI runners (and never raises on a legacy Windows code page).
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
except (AttributeError, ValueError):
pass
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--mode", choices=("orchestrate", "trigger"), default="orchestrate",
help="orchestrate (default): in-run runner-priority for the executing PR "
"(unchanged). trigger: the scheduler that (re-)triggers CI by priority "
"(issue #349).")
parser.add_argument(
"--dry-run", action="store_true",
help="read a snapshot JSON (from --input or stdin), print decisions, no network.")
parser.add_argument(
"--input", help="snapshot JSON file for --dry-run (default: stdin).")
args = parser.parse_args(argv)
if args.dry_run:
text = (open(args.input, encoding="utf-8").read() if args.input
else sys.stdin.read())
return run_trigger_dry(text) if args.mode == "trigger" else run_dry(text)
return run_trigger() if args.mode == "trigger" else run_live()
if __name__ == "__main__":
try:
sys.exit(main())
except Exception as exc: # never let the controller fail CI
_log(f"::warning::traffic_control crashed, proceeding fail-open: {exc}")
sys.exit(0)
+153
View File
@@ -0,0 +1,153 @@
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
# GitHub Actions workflows
This directory holds the repo's workflows:
- **`ci.yml`** — the pull-request gate: build, unit tests, static analysis (ktlint /
detekt), and the E2E/instrumented-test matrix, aggregated into one `CI passed` check
that branch protection requires. It also runs the `traffic-control` job described
below.
- **`autoupdate.yml`** — rebases every open PR onto `main` whenever `main` advances, so
the "branches up to date" branch rule never needs a manual update.
- **`release.yml`** — turns a pushed version tag into signed, published release
artifacts; see [`docs/release.md`](../../docs/release.md).
The rest of this README is about **`traffic-control`** — the job (in the Checks tab it
shows up as **"Traffic control (runner priority)"**) that decides whose CI gets to run
first when several PRs are queued at once.
## Why this job exists
GitHub Actions has no concept of "run this PR's checks before that one" — every PR's
workflow run joins the same pool of runners and is served roughly first-come,
first-served. That's fine most of the time, but with several PRs open at once it means
an urgent one-line hotfix queues up as an equal to a routine refactor, and can end up
stuck waiting behind CI runs for changes that aren't in any hurry.
`traffic-control` addresses that by reading a **priority label** on the current PR,
comparing it against every other open PR, and then either freeing up a runner by
cancelling a lower-priority PR's run (**preemption**), or briefly waiting before this
PR's own heavy jobs start so a higher-priority PR's jobs get a head start
(**hold-back**). It runs first in every PR's CI: every other job in `ci.yml`
(`debug-build`, `unit-tests`, `static-analysis`, `e2e`, `e2e-preview`) declares
`needs: traffic-control`, so it always goes first —
```
PR's CI run starts
│
▼
traffic-control
│ 1. compute this PR's effective priority (see table below)
│ 2. PASS 1 — preemption: cancel strictly-lower-priority OTHER PRs' active
│ runs, but only if we're P0, or the target PR is `broken`
│ 3. PASS 2 — hold-back: if we're not P0, wait (up to 180s) while any
│ strictly-higher-priority OTHER PR still has an active run, then
│ proceed regardless
▼
debug-build · unit-tests · static-analysis · e2e · e2e-preview
```
## Effective priority
Priority comes from a label on the PR:
| Label | Effective priority | Meaning |
| --- | --- | --- |
| `P0` | 0 (highest) | **Emergency only** — production is broken, or an emergency security fix. |
| `P1` – `P9` | 1 – 9 | Higher number = lower priority. |
| *(no `P` label)* | 5 (default) | Normal priority — most PRs. |
| `broken` | 10 (lowest) | A stuck/failing PR, deprioritised below even `P9`. Overrides any `P0`–`P9` label also present. |
Apply at most one `P0`–`P9` label; if more than one is somehow present, the numerically
lowest (most urgent) one wins. The `broken` label is meant to be applied by a maintainer
to a PR whose CI is stuck or failing, as a "let everyone else go first while this gets
fixed" signal — not something a PR author sets on their own work. Removing it restores
whatever `P0`–`P9` priority (or the `P5` default) the PR would otherwise have.
## Preemption vs. holding back
### P0 preempts everyone lower
If *this* PR is `P0`, it's treated as an emergency: the job immediately cancels the
in-progress or queued CI runs of **every other open PR at a strictly lower priority**
(that is, anything that isn't also `P0`), freeing up their runners right away. A PR
that gets cancelled this way isn't harmed long-term — it simply reruns on its next push,
or the next time `autoupdate.yml` rebases it onto `main`. Because nothing outranks an
emergency, a `P0` PR also never does the hold-back wait described below.
### A `broken` PR can be preempted by anyone
A PR labelled `broken` can't merge while it's broken, so its CI run occupying a runner
is wasted capacity. Any PR that isn't itself `broken` — in other words, any PR with a
real `P0`–`P9` priority — outranks it and may cancel its active run to reclaim the
runner, not just a `P0` PR. `broken` is also the only priority level that yields to
*everything*: since it sits below every other level, it always waits for other PRs'
runs rather than the other way around.
### P1–P9 yield, but never cancel
Every other level (`P1`–`P9`, including the `P5` default) is cooperative rather than
aggressive: it never cancels a run that's already going, no matter how much lower that
run's priority is. Instead, before letting its own heavy jobs start, it checks whether
any **strictly higher**-priority PR currently has an active or queued CI run. If so, it
waits — polling every 15 seconds and re-checking the full list of open PRs each time, so
a newly opened higher-priority PR is picked up mid-wait too — giving that PR's jobs a
chance to reach the runner queue first. The wait is capped at **180 seconds**
(comfortably inside the job's 6-minute hard timeout); once the budget runs out, this PR
proceeds regardless. A PR should never be able to block itself indefinitely.
## Safety invariants
Whatever the priority math says, a few things are hard-coded to never happen:
- **Never touches `main` / push-triggered runs.** The job only acts on `pull_request`
events, and every run it's even allowed to consider cancelling is filtered down to
`event == pull_request` with `headBranch != main`.
- **Never cancels this PR's own run.** The current PR is excluded from the "other PRs"
list up front by PR number, and the currently-executing run ID is skipped too, just in
case.
- **Never cancels an equal-or-higher-priority run.** Only strictly-lower-priority PRs
(a numerically larger, i.e. worse, priority) are ever candidates for cancellation.
## Honest limitation
This is a **best-effort head start, not a real priority queue.** GitHub Actions has no
API for "give this run's jobs priority over that run's jobs" — runners are handed out
roughly FIFO no matter what this job does. Hold-back approximates priority by making
lower-priority PRs wait a little before their jobs even enter that FIFO queue, but under
sustained contention (many PRs queuing at once) the bounded wait can run out before a
higher-priority PR's jobs have actually made it through the runner pool. The waiting job
itself is cheap and short-lived, which is exactly why the wait is capped rather than
open-ended — occasionally under-prioritizing is preferable to a job that ties up a
runner indefinitely just to wait.
## Not a merge gate
`traffic-control` is an optimizer, not a check your PR needs to pass. It's deliberately
left out of `ci-passed`'s `needs:` list, every GitHub API call it makes is guarded
against failure, the script always exits `0`, and the step itself runs with
`continue-on-error: true`. A hiccup here — a transient API error, a missing permission,
a fork PR without write access — can never fail or block your PR.
That said, the heavy jobs still order themselves after it via `needs: traffic-control`,
so if this job were ever skipped or failed outright, GitHub would mark those jobs
`skipped` — and `ci-passed` treats a required job coming back `skipped` as a gate
failure. So the worst case is fail-safe: it blocks the merge rather than letting an
untested PR through.
It also needs very little to run: no checkout step (it only calls the `gh` CLI), and
just two permissions (`actions: write` to cancel runs, `pull-requests: read` to read
labels). Values that come from outside the repo — labels, branch names — are only ever
read through `gh`'s JSON output into shell variables, never interpolated as shell code.
## Where this is heading
**#342** is rewriting this logic as a tested Python module
(`.github/scripts/traffic_control.py`), with a couple of small behavior refinements:
draft PRs will also sink to the bottom (like `broken`), and PRs at the exact same
priority level get an explicit order (whichever run is already in flight finishes
first; among the rest, whoever has been waiting longest goes next). This README
describes the shell-script version currently in `ci.yml` — see the comment block above
the `traffic-control` job there for the byte-for-byte spec — and will be updated once
#342 lands.
+28 -15
View File
@@ -1,22 +1,32 @@
# SPDX-License-Identifier: GPL-3.0-or-later
name: Auto-update PR branches
# When main advances, rebase any auto-merge-armed PR that has fallen behind, so the
# "require branches up to date" branch rule doesn't need manual branch updates. Only PRs
# with GitHub auto-merge enabled are touched (PR_FILTER: auto_merge) — held/draft PRs are
# left alone.
# When main advances, rebase every open PR that has fallen behind, so the "require
# branches up to date" branch rule doesn't need manual branch updates. All open PRs are
# touched (PR_FILTER: all) — this is no longer limited to PRs with GitHub auto-merge
# enabled.
#
# IMPORTANT: for the branch update to RE-TRIGGER the PR's CI (so it can pass and merge),
# this must run with a PAT, not the default GITHUB_TOKEN — pushes made by GITHUB_TOKEN do
# not start new workflow runs (GitHub's anti-recursion rule), so the updated PR would sit
# with stale checks. Create a fine-grained PAT scoped to this repo with
# contents:read/write + pull-requests:read/write and add it as the AUTOUPDATE_TOKEN secret.
# Without it this falls back to GITHUB_TOKEN, which updates the branch but will NOT re-run
# the PR's checks.
# IMPORTANT (issue #349): the branch update runs with the default GITHUB_TOKEN — ON PURPOSE.
# A GITHUB_TOKEN push does NOT start new workflow runs (GitHub's anti-recursion rule), so
# updating every behind PR here NO LONGER re-triggers every PR's CI. That deliberately breaks
# the old merge-cascade (every merge -> autoupdate rebases all PRs with a PAT -> all re-run ->
# ci.yml's cancel-in-progress kills each in-flight run -> PRs thrash and can't converge).
# Branches still go up to date (satisfying "require branches up to date"); they just don't
# auto-run CI on the new head SHA. Re-triggering that SHA's CI is now OWNED by the traffic-
# controller scheduler (`.github/workflows/ci-trigger.yml` -> `traffic_control.py --mode
# trigger`), which triggers the updated PRs deliberately, in priority order, a few at a time.
# So this workflow must NOT use the PAT for the update push (that would re-introduce the
# cascade). This workflow itself doesn't need AUTOUPDATE_TOKEN — but the secret is still REQUIRED
# by the repo: the ci-trigger.yml scheduler dispatches CI with it (a GITHUB_TOKEN dispatch would
# be held for manual approval and never run un-attended). Don't delete the secret. See
# ci-trigger.yml + issue #351.
on:
push:
branches: [main]
pull_request:
types: [opened, reopened, ready_for_review]
branches: [main]
permissions:
contents: write
@@ -28,13 +38,16 @@ concurrency:
jobs:
autoupdate:
name: Auto-update armed PRs
name: Auto-update open PRs
runs-on: ubuntu-latest
environment: CI_CD
steps:
- name: Update behind PRs that have auto-merge enabled
- name: Update all behind PRs
uses: chinthakagodawita/autoupdate@0707656cd062a3b0cf8fa9b2cda1d1404d74437e # v1.7.0
env:
GITHUB_TOKEN: ${{ secrets.AUTOUPDATE_TOKEN || secrets.GITHUB_TOKEN }}
PR_FILTER: "auto_merge"
# Default GITHUB_TOKEN — NOT a PAT — so this update push does not auto-retrigger CI
# (anti-recursion). See the header: re-triggering is owned by ci-trigger.yml.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_FILTER: "all"
PR_READY_STATE: "all"
MERGE_CONFLICT_ACTION: "ignore"
+100
View File
@@ -0,0 +1,100 @@
# SPDX-License-Identifier: GPL-3.0-or-later
name: CI trigger (traffic-controller)
# The traffic-controller SCHEDULER (issue #349). It OWNS CI *triggering*. After main advances,
# autoupdate.yml updates every behind PR's branch with the built-in GITHUB_TOKEN which, by
# GitHub's anti-recursion rule, does NOT start CI — so those PRs sit with absent/stale required
# checks on their new head SHA and cannot merge. This workflow then (re-)triggers CI for the
# highest-priority such PR(s), a few at a time (an inflight cap), in the existing P0–P9 /
# broken-draft priority order — a poor-man's merge queue that replaces the old "every merge
# re-runs every PR" thundering herd (the cascade; see the ci-merge-cascade note + issue #349).
#
# HOW IT TRIGGERS: `traffic_control.py --mode trigger` runs `gh workflow run ci.yml --ref
# <pr-head-branch>`, dispatching with the AUTOUPDATE_TOKEN PAT — NOT the built-in GITHUB_TOKEN.
# A workflow run triggered by GITHUB_TOKEN is held in the `action_required` state waiting on
# MANUAL approval and never runs un-attended (confirmed empirically on #285 / #350: it sits
# `action_required`, while the same dispatch by an authorized user runs immediately) — which
# would defeat the whole scheduler. A PAT dispatch runs AS the authorized token owner, so the
# run starts immediately with no approval gate (this is the original #349 design; #350's "no
# PAT needed / workflow_dispatch is anti-recursion-exempt" claim was WRONG — see #351).
# AUTOUPDATE_TOKEN is therefore REQUIRED for this scheduler. The dispatched run executes on the
# PR's head branch, so its checks land on the PR head SHA and satisfy branch protection.
#
# WHEN IT RUNS:
# • workflow_run, after "Auto-update PR branches" completes — the race-free moment: autoupdate
# has finished moving branches to their new (checkless) head SHAs, so this pass sees exactly
# the PRs that now need a run. (A bare `push: main` trigger would race autoupdate and often
# read the pre-update SHAs, missing them until the next pass.)
# • schedule (cron) — a backstop so no PR is ever permanently un-triggered even if a
# workflow_run is missed/skipped (part of the fail-open guarantee), and so a brand-new PR
# whose first `on: pull_request` run got cancelled is still picked up.
# • workflow_dispatch — manual kick.
#
# FAIL-OPEN: the script guards every gh call and always exits 0; and structurally, ci.yml keeps
# its `on: pull_request` trigger, so a human push (and a brand-new PR) always triggers CI
# regardless of this scheduler — CI can never become permanently un-triggerable. Fork PRs (no
# token/secret access) are skipped here and left to `on: pull_request`, so they are never wedged.
on:
workflow_run:
workflows: ["Auto-update PR branches"]
types: [completed]
schedule:
# Backstop cadence (UTC). GitHub may delay scheduled runs under load; that is fine — this
# is only a safety net behind the immediate workflow_run trigger above.
- cron: "*/15 * * * *"
workflow_dispatch:
# Trigger-only; this workflow never gates a merge. The gh calls run as GH_TOKEN, which is
# normally the AUTOUPDATE_TOKEN PAT (see the step below). These permissions govern the built-in
# GITHUB_TOKEN, used only on the fail-open fallback path when AUTOUPDATE_TOKEN is absent:
# `actions: write` lets it dispatch ci.yml (workflow_dispatch) and cancel strictly-lower runs
# when a P0 emergency preempts; `pull-requests: read` + `contents: read` cover the PR/label
# enumeration. (A GITHUB_TOKEN dispatch needs manual approval, so that fallback only actually
# starts CI if repo settings don't gate GITHUB_TOKEN-triggered runs — the PAT is the real path.)
permissions:
contents: read
pull-requests: read
actions: write
# One trigger pass at a time. Do NOT cancel an in-flight pass (cancel-in-progress: false):
# a half-finished pass could leave some needy PRs un-triggered until the next pass.
concurrency:
group: ci-trigger
cancel-in-progress: false
jobs:
trigger:
name: Trigger CI by priority
runs-on: ubuntu-latest
timeout-minutes: 10 # generous backstop; the script only enumerates + dispatches, no waits
steps:
- name: Check out source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.x"
# gh is auto-configured from GH_TOKEN / GH_REPO. The script guards every gh call and
# always exits 0, so a hiccup (API error, missing permission, fork PR) can never wedge CI
# — and even a total failure here leaves ci.yml's `on: pull_request` path intact.
- name: Trigger CI for the highest-priority PR(s) needing a run
env:
# AUTOUPDATE_TOKEN (a PAT) is REQUIRED here: a CI run dispatched by the built-in
# GITHUB_TOKEN is held for MANUAL approval (`action_required`) and never runs
# un-attended, so the scheduler must dispatch AS the PAT's authorized owner to start
# runs with no approval gate. `|| github.token` keeps this fail-open when the secret is
# absent, but that GITHUB_TOKEN fallback only actually starts CI if repo settings don't
# gate GITHUB_TOKEN-triggered runs — the PAT is the intended path (see #351).
GH_TOKEN: ${{ secrets.AUTOUPDATE_TOKEN || github.token }}
GH_REPO: ${{ github.repository }}
# Poor-man's merge-queue width: at most this many PRs run CI concurrently under the
# scheduler (a P0 emergency bypasses this cap). Kept conservative because each PR
# fans out to the whole E2E matrix (~8 API levels + preview); this is the main knob
# to raise for throughput vs runner budget. The coordinator drives runner allocation.
MAX_INFLIGHT_RUNS: "2"
# The workflow file the scheduler enumerates runs for and dispatches.
CI_WORKFLOW_FILE: "ci.yml"
run: python3 .github/scripts/traffic_control.py --mode trigger
+652 -22
View File
@@ -4,10 +4,33 @@ name: CI
on:
pull_request:
branches: [main]
# The traffic-controller SCHEDULER (ci-trigger.yml, issue #349) (re-)triggers CI for a
# specific PR via this workflow_dispatch after a GITHUB_TOKEN auto-update has left the PR's
# head SHA with absent/stale checks. Dispatched on the PR's head BRANCH, so the run's checks
# land on the PR head SHA and satisfy branch protection. `on: pull_request` above is KEPT so
# brand-new PRs, human pushes, and fork PRs still get CI directly — this is the fail-open
# guarantee: CI is always triggerable even if the scheduler is broken or absent.
workflow_dispatch:
inputs:
pr:
description: "PR number this run is for (set by the traffic-controller scheduler)."
required: false
type: string
head_sha:
description: "Expected head SHA (informational, for traceability in the run log)."
required: false
type: string
reason:
description: "Why this run was dispatched (informational)."
required: false
type: string
# A new push to a PR cancels any in-flight run for that PR.
# A new trigger for a PR cancels that PR's own in-flight run (a newer head SHA supersedes).
# The group is keyed to the PR NUMBER so a `pull_request` run and a scheduler
# `workflow_dispatch` run for the SAME PR share one concurrency group (either supersedes a
# stale run of the other); it falls back to the ref when no PR number is in context.
concurrency:
group: ci-${{ github.ref }}
group: ci-pr-${{ github.event.pull_request.number || inputs.pr || github.ref }}
cancel-in-progress: true
permissions:
@@ -20,6 +43,84 @@ env:
ANDROID_BUILD_TOOLS: "build-tools;37.0.0"
jobs:
# Path-filter gate (issue #399): a cheap first job that decides whether the heavy E2E matrix
# (`e2e` + `e2e-preview`, ~10 emulator jobs) needs to run for this change. Test-only / docs /
# dev-script PRs then skip E2E and merge on the fast gate (unit + static) instead of queuing
# behind the emulator matrix. The `ci-passed` gate below is rewritten to treat an INTENTIONAL
# E2E skip as a pass while still blocking a real E2E failure/cancel or a broken filter.
changes:
name: Detect changed paths
runs-on: ubuntu-latest
# dorny/paths-filter lists a pull request's changed files via the GitHub API (no checkout),
# which needs pull-requests: read on top of the workflow-default contents: read.
permissions:
contents: read
pull-requests: read
outputs:
# 'true' -> run the E2E matrix; 'false' -> skip it (only for test-only/docs/script PRs).
e2e_needed: ${{ steps.decide.outputs.e2e_needed }}
steps:
- name: Filter changed paths (pull requests only)
id: filter
if: github.event_name == 'pull_request'
uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2
with:
# predicate-quantifier: 'every' makes the `skippable` filter true ONLY when EVERY changed
# file matches one of these safe patterns. We invert it below (e2e_needed = NOT skippable),
# so ANY file outside this small allow-list — app/src/main, app/src/androidTest,
# app/build.gradle.kts, root build.gradle*/settings.gradle*, gradle/** (incl. the version
# catalog & wrapper), gradle.properties, app/schemas, app/proguard-rules.pro, another
# workflow, .github/scripts, … — forces the E2E matrix to run. That is the conservative
# "err toward running E2E / default to true if unsure" rule: the skip list is an explicit
# allow-list of things that provably cannot affect app runtime or instrumented tests,
# never a guess about what is unsafe.
predicate-quantifier: 'every'
filters: |
skippable:
- 'app/src/test/**'
- '**/*.md'
- 'docs/**'
- 'scripts/**'
- '.claude/**'
- name: Decide whether the E2E matrix is needed
id: decide
run: |
if [ "${{ github.event_name }}" = "pull_request" ] && [ "${{ steps.filter.outputs.skippable }}" = "true" ]; then
echo "e2e_needed=false" >> "$GITHUB_OUTPUT"
echo "E2E matrix SKIPPED: every changed file is under a test-only / docs / script / .claude path."
else
echo "e2e_needed=true" >> "$GITHUB_OUTPUT"
echo "E2E matrix NEEDED: build/runtime/instrumented paths changed, or this is not a pull_request (conservative default)."
fi
# Runner-priority orchestration (traffic-control) has been EXTRACTED from this file.
# The `traffic-control` job that used to run here first (ordering the heavy jobs below, which
# each declared it as a `needs:` dependency) now lives VERBATIM in its own workflow at
# .github/workflows/traffic-control.yml, and is being mothballed: it will be disabled pending
# a rebuild as a published GitHub Action, so the heavy jobs below no longer depend on it. Its
# pure-Python decision core (.github/scripts/traffic_control.py) is unchanged and is still
# unit-tested by the `traffic-control-tests` job below.
# Fast, pure-stdlib-Python unit tests for the traffic-control decision core
# (.github/scripts/traffic_control.py / test_traffic_control.py — see the
# extracted `traffic-control` workflow). No emulator, no Gradle: this runs in seconds,
# independently of the Android jobs below, so a regression in the runner-priority
# logic fails fast and blocks merge via `ci-passed`.
traffic-control-tests:
name: Traffic-control unit tests
runs-on: ubuntu-latest
steps:
- name: Check out source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.x"
- name: Run traffic-controller unit tests
run: python -m unittest discover -s .github/scripts -p 'test_*.py' -v
debug-build:
name: Debug build
# x86_64: Linux-arm64 runners can't set up this SDK — android-actions/setup-android's sdkmanager
@@ -37,11 +138,57 @@ jobs:
distribution: temurin
java-version: "21"
- name: Restore Android SDK cache
id: android-sdk-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
# Cache the SHA-256-verified command-line tools + platform/build-tools so a
# green run doesn't re-download (and risk re-corrupting) them. Restore-only
# here + the success-gated save below == "integrity gates the cache": a
# corrupt/failed SDK is never saved (#389). Shared key (identical contents).
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
# Bump the prefix if ANDROID_PLATFORM/ANDROID_BUILD_TOOLS or the pinned
# cmdline-tools build (14742923) change.
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
# Provision the SHA-256-verified command-line tools into the exact path
# setup-android probes first, so the action reuses it and never does its own
# unverified "Wrong version in preinstalled sdkmanager" re-download (#389).
- name: Bootstrap verified Android command-line tools
run: python3 .github/scripts/setup_android_sdk.py bootstrap
- name: Set up Android SDK
uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699 # v4.0.1
with:
# Reuse the verified cmdline-tools bootstrapped above (matching version =>
# no unverified re-download) and pass '' packages so the action does NOT run
# the flaky `sdkmanager tools platform-tools` install — the corrupt-zip
# surface that failed the "Set up Android SDK" step (#389).
cmdline-tools-version: "14742923"
packages: ""
# verify -> reject -> retry: a corrupt package zip ("Error reading Zip
# content ...") is purged and re-downloaded instead of failing on sdkmanager's
# bare exit 1 (#389; supersedes the inline retry loop from #387/#388).
- name: Install SDK platform and build-tools
run: sdkmanager "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
run: python3 .github/scripts/setup_android_sdk.py install "platform-tools" "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
# Save the verified SDK only on success (never cache a corrupt SDK) and only on
# a miss (avoid redundant re-saves).
- name: Save Android SDK cache
if: success() && steps.android-sdk-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
- name: Set up Gradle
uses: gradle/actions/setup-gradle@3f131e8634966bd73d06cc69884922b02e6faf92 # v6.2.0
@@ -69,11 +216,57 @@ jobs:
distribution: temurin
java-version: "21"
- name: Restore Android SDK cache
id: android-sdk-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
# Cache the SHA-256-verified command-line tools + platform/build-tools so a
# green run doesn't re-download (and risk re-corrupting) them. Restore-only
# here + the success-gated save below == "integrity gates the cache": a
# corrupt/failed SDK is never saved (#389). Shared key (identical contents).
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
# Bump the prefix if ANDROID_PLATFORM/ANDROID_BUILD_TOOLS or the pinned
# cmdline-tools build (14742923) change.
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
# Provision the SHA-256-verified command-line tools into the exact path
# setup-android probes first, so the action reuses it and never does its own
# unverified "Wrong version in preinstalled sdkmanager" re-download (#389).
- name: Bootstrap verified Android command-line tools
run: python3 .github/scripts/setup_android_sdk.py bootstrap
- name: Set up Android SDK
uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699 # v4.0.1
with:
# Reuse the verified cmdline-tools bootstrapped above (matching version =>
# no unverified re-download) and pass '' packages so the action does NOT run
# the flaky `sdkmanager tools platform-tools` install — the corrupt-zip
# surface that failed the "Set up Android SDK" step (#389).
cmdline-tools-version: "14742923"
packages: ""
# verify -> reject -> retry: a corrupt package zip ("Error reading Zip
# content ...") is purged and re-downloaded instead of failing on sdkmanager's
# bare exit 1 (#389; supersedes the inline retry loop from #387/#388).
- name: Install SDK platform and build-tools
run: sdkmanager "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
run: python3 .github/scripts/setup_android_sdk.py install "platform-tools" "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
# Save the verified SDK only on success (never cache a corrupt SDK) and only on
# a miss (avoid redundant re-saves).
- name: Save Android SDK cache
if: success() && steps.android-sdk-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
- name: Set up Gradle
uses: gradle/actions/setup-gradle@3f131e8634966bd73d06cc69884922b02e6faf92 # v6.2.0
@@ -89,6 +282,29 @@ jobs:
path: app/build/reports/tests/testDebugUnitTest/
if-no-files-found: warn
# JaCoCo XML + HTML coverage for the JVM unit tests (issue #192), scoped to the JVM-testable
# surface (issues #290/#292). The report is generated and uploaded first, then a no-regression
# gate (issue #251) fails the job if overall LINE coverage drops below the floor pinned in
# app/build.gradle.kts. Kept in this unit-test job so it is part of the `CI passed` gate.
- name: Generate JaCoCo coverage report
if: ${{ !cancelled() }}
run: ./gradlew :app:jacocoTestReport --stacktrace
- name: Upload coverage report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: jacoco-coverage-report
path: app/build/reports/jacoco/jacocoTestReport/
if-no-files-found: warn
# No-regression coverage gate (issue #251): fails CI if scoped LINE coverage regresses below the
# floor pinned in app/build.gradle.kts. Runs after the upload so the HTML/XML report is always
# archived for triage even when this step goes red.
- name: Verify JaCoCo coverage (no-regression floor)
if: ${{ !cancelled() }}
run: ./gradlew :app:jacocoTestCoverageVerification --stacktrace
static-analysis:
name: Static analysis
runs-on: ubuntu-latest
@@ -103,11 +319,57 @@ jobs:
java-version: "21"
# AGP configuration needs the SDK even for ktlint/detekt (they run on the :app module).
- name: Restore Android SDK cache
id: android-sdk-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
# Cache the SHA-256-verified command-line tools + platform/build-tools so a
# green run doesn't re-download (and risk re-corrupting) them. Restore-only
# here + the success-gated save below == "integrity gates the cache": a
# corrupt/failed SDK is never saved (#389). Shared key (identical contents).
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
# Bump the prefix if ANDROID_PLATFORM/ANDROID_BUILD_TOOLS or the pinned
# cmdline-tools build (14742923) change.
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
# Provision the SHA-256-verified command-line tools into the exact path
# setup-android probes first, so the action reuses it and never does its own
# unverified "Wrong version in preinstalled sdkmanager" re-download (#389).
- name: Bootstrap verified Android command-line tools
run: python3 .github/scripts/setup_android_sdk.py bootstrap
- name: Set up Android SDK
uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699 # v4.0.1
with:
# Reuse the verified cmdline-tools bootstrapped above (matching version =>
# no unverified re-download) and pass '' packages so the action does NOT run
# the flaky `sdkmanager tools platform-tools` install — the corrupt-zip
# surface that failed the "Set up Android SDK" step (#389).
cmdline-tools-version: "14742923"
packages: ""
# verify -> reject -> retry: a corrupt package zip ("Error reading Zip
# content ...") is purged and re-downloaded instead of failing on sdkmanager's
# bare exit 1 (#389; supersedes the inline retry loop from #387/#388).
- name: Install SDK platform and build-tools
run: sdkmanager "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
run: python3 .github/scripts/setup_android_sdk.py install "platform-tools" "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
# Save the verified SDK only on success (never cache a corrupt SDK) and only on
# a miss (avoid redundant re-saves).
- name: Save Android SDK cache
if: success() && steps.android-sdk-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
- name: Set up Gradle
uses: gradle/actions/setup-gradle@3f131e8634966bd73d06cc69884922b02e6faf92 # v6.2.0
@@ -128,7 +390,14 @@ jobs:
e2e:
name: E2E
# Path-filter gate (issue #399): gating the whole job means every matrix leg runs — or is
# skipped — together. On an intentional skip the `ci-passed` gate treats e2e's 'skipped'
# result as OK (a real failure/cancel still blocks). `needs: changes` waits only on the
# seconds-long filter job.
needs: changes
if: needs.changes.outputs.e2e_needed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 50
strategy:
fail-fast: false
matrix:
@@ -151,11 +420,61 @@ jobs:
distribution: temurin
java-version: "21"
- name: Restore Android SDK cache
id: android-sdk-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
# Cache the SHA-256-verified command-line tools + platform/build-tools so a
# green run doesn't re-download (and risk re-corrupting) them. Restore-only
# here + the success-gated save below == "integrity gates the cache": a
# corrupt/failed SDK is never saved (#389). Shared key (identical contents).
# The emulator + system image stay with android-emulator-runner (boot logic
# is out of scope for #389).
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
# Bump the prefix if ANDROID_PLATFORM/ANDROID_BUILD_TOOLS or the pinned
# cmdline-tools build (14742923) change.
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
# Provision the SHA-256-verified command-line tools into the exact path
# setup-android probes first, so the action reuses it and never does its own
# unverified "Wrong version in preinstalled sdkmanager" re-download (#389).
- name: Bootstrap verified Android command-line tools
run: python3 .github/scripts/setup_android_sdk.py bootstrap
- name: Set up Android SDK
uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699 # v4.0.1
with:
# Reuse the verified cmdline-tools bootstrapped above (matching version =>
# no unverified re-download) and pass '' packages so the action does NOT run
# the flaky `sdkmanager tools platform-tools` install — the corrupt-zip
# surface that failed the "Set up Android SDK" step (#389).
cmdline-tools-version: "14742923"
packages: ""
# verify -> reject -> retry (#389, supersedes the #387/#388 inline loop): sdkmanager
# can exit 1 on a corrupt/truncated package zip ("Error reading Zip content ...") — an
# `E2E (31)` leg died this way. The helper purges each partial/corrupt package and
# re-downloads it clean, and on a hard failure prints the installed-package list so the
# cause is visible in the step log instead of a bare exit 1.
- name: Install SDK platform and build-tools
run: sdkmanager "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
run: python3 .github/scripts/setup_android_sdk.py install "platform-tools" "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS"
# Save the verified SDK only on success (never cache a corrupt SDK) and only on
# a miss (avoid redundant re-saves).
- name: Save Android SDK cache
if: success() && steps.android-sdk-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
- name: Set up Gradle
uses: gradle/actions/setup-gradle@3f131e8634966bd73d06cc69884922b02e6faf92 # v6.2.0
@@ -190,7 +509,15 @@ jobs:
disable-animations: false
script: echo "Generated AVD snapshot for caching."
# reactivecircus/android-emulator-runner runs an un-guarded, fatal `adb shell input keyevent 82`
# after boot. On snapshot resume that can race system_server (sys.boot_completed=1 before the
# `input` binder service is republished), aborting the job before Gradle runs with
# "No service published for: input" — an ~2%, API-29-only infra flake, not a test failure. Make
# the step non-fatal and retry once: two independent boots drop the race to ~0.04%. The definitive
# fix (adopt the e2e-preview job's manual-boot + `keyevent 82 || true`) is tracked separately.
- name: Run E2E tests
id: e2e
continue-on-error: true
uses: reactivecircus/android-emulator-runner@e89f39f1abbbd05b1113a29cf4db69e7540cae5a # v2.37.0
with:
api-level: ${{ matrix.api-level }}
@@ -199,7 +526,49 @@ jobs:
force-avd-creation: false
emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
disable-animations: true
script: ./gradlew connectedDebugAndroidTest --stacktrace
# Stream logcat to a per-api-level file (the emulator is booted here) before the tests,
# so a test failure or emulator flake is diagnosable from the uploaded artifact.
# Backgrounded; gradle stays the last foreground command so the step's exit status is
# still the test result (a real failure still trips continue-on-error -> the retry).
script: |
adb logcat -v time > "$RUNNER_TEMP/logcat-api${{ matrix.api-level }}.txt" 2>&1 &
./gradlew connectedDebugAndroidTest --stacktrace
- name: Run E2E tests (retry after emulator boot race)
if: steps.e2e.outcome == 'failure'
uses: reactivecircus/android-emulator-runner@e89f39f1abbbd05b1113a29cf4db69e7540cae5a # v2.37.0
with:
api-level: ${{ matrix.api-level }}
target: google_apis
arch: x86_64
force-avd-creation: false
emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
disable-animations: true
# Retry runs a fresh emulator boot; stream its logcat the same way. `>` overwrites
# attempt 1's file so the artifact holds the FINAL attempt's logs, matching the
# failure-time dump below (which reflects this last attempt's state).
script: |
adb logcat -v time > "$RUNNER_TEMP/logcat-api${{ matrix.api-level }}.txt" 2>&1 &
./gradlew connectedDebugAndroidTest --stacktrace
# On any E2E failure (both boot attempts failed, a hung emulator, or an earlier setup/SDK
# step), snapshot device + runner state to the step log AND a file for the artifact upload —
# the `E2E (31)` sdkmanager death (#387) left no trail. Each probe is guarded (|| true) so a
# missing tool / offline device can't abort the step; accel-check, /dev/kvm, free -h and
# df -h characterise the runner even when the emulator never booted.
- name: Dump emulator + system diagnostics on failure
if: failure()
run: |
DIAG="${RUNNER_TEMP:-/tmp}/diagnostics-api${{ matrix.api-level }}.txt"
{
echo "===== E2E API ${{ matrix.api-level }} failure diagnostics ====="
echo "--- adb devices ---"; adb devices 2>&1 || true
echo "--- adb logcat -d (tail 200) ---"; adb logcat -d 2>&1 | tail -200 || true
echo "--- emulator -accel-check ---"; "$ANDROID_SDK_ROOT/emulator/emulator" -accel-check 2>&1 || true
echo "--- /dev/kvm ---"; ls -l /dev/kvm 2>&1 || true
echo "--- free memory ---"; free -h 2>&1 || true
echo "--- free disk ---"; df -h 2>&1 || true
} 2>&1 | tee "$DIAG"
- name: Upload E2E test report
if: ${{ !cancelled() }}
@@ -209,6 +578,20 @@ jobs:
path: app/build/reports/androidTests/connected/
if-no-files-found: warn
# Always upload the streamed logcat + (on failure) the system-state dump so an emulator flake
# or a red matrix leg is diagnosable from artifacts without a re-run — parity with the
# e2e-preview boot-diagnostics artifact. Per-api-level name (upload-artifact@v7 rejects
# duplicate artifact names).
- name: Upload E2E diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-diagnostics-api${{ matrix.api-level }}
path: |
${{ runner.temp }}/logcat-api${{ matrix.api-level }}.txt
${{ runner.temp }}/diagnostics-api${{ matrix.api-level }}.txt
if-no-files-found: warn
# API 37 (Android 17, preview) E2E. Its only system image is the nonstandard
# android-37.0 / google_apis_ps16k (16 KB page size), which reactivecircus/android-emulator-runner
# can't provision (it builds android-37 / google_apis, neither of which exists), so this job
@@ -218,10 +601,30 @@ jobs:
# 37 into the main `e2e` matrix and delete this job.
e2e-preview:
name: E2E (API 37 preview)
# Path-filter gate (issue #399): runs or skips together with the `e2e` matrix. On an
# intentional skip the `ci-passed` gate treats e2e-preview's 'skipped' result as OK; a real
# failure/cancel still blocks. Both shard legs fan in under this one gated job.
needs: changes
if: needs.changes.outputs.e2e_needed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
# Sharded N=2 (docs/perf/api37-e2e-sharding-spike.md). The instrumented suite is split across
# 2 parallel API 37 emulators — each a separate matrix leg that hand-provisions its OWN emulator
# and runs one shard via AndroidJUnitRunner's numShards/shardIndex. This leg is the pipeline's
# critical path, so sharding cuts it ~17.1 min -> ~12.7 min (~28% off total CI) for +1 emulator
# job. FAN-IN: `ci-passed` lists `e2e-preview` once and GHA rolls BOTH shard legs under that one
# entry — a matrix job's aggregate result is `failure` if ANY leg fails — so both shards must
# pass, and branch protection (which requires the "CI passed" context, not the per-leg
# "E2E (API 37 preview) (N)" names) needs no change. `fail-fast: false` lets one shard's failure
# NOT cancel the other, so both reports always upload. Keep API37_NUM_SHARDS in lockstep with the
# length of matrix.shard.
strategy:
fail-fast: false
matrix:
shard: [0, 1] # 0-based shardIndex values; length must equal API37_NUM_SHARDS below
env:
API37_IMAGE: "system-images;android-37.0;google_apis_ps16k;x86_64"
API37_NUM_SHARDS: "2"
steps:
- name: Check out source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -232,8 +635,39 @@ jobs:
distribution: temurin
java-version: "21"
- name: Restore Android SDK cache
id: android-sdk-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
# Cache the SHA-256-verified command-line tools + platform/build-tools so a
# green run doesn't re-download (and risk re-corrupting) them. Restore-only
# here + the success-gated save below == "integrity gates the cache": a
# corrupt/failed SDK is never saved (#389). The ~1 GB preview system image
# keeps its own cache below.
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
# Bump the prefix if ANDROID_PLATFORM/ANDROID_BUILD_TOOLS or the pinned
# cmdline-tools build (14742923) change.
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
# Provision the SHA-256-verified command-line tools into the exact path
# setup-android probes first, so the action reuses it and never does its own
# unverified "Wrong version in preinstalled sdkmanager" re-download (#389).
- name: Bootstrap verified Android command-line tools
run: python3 .github/scripts/setup_android_sdk.py bootstrap
- name: Set up Android SDK
uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699 # v4.0.1
with:
# Reuse the verified cmdline-tools bootstrapped above (matching version =>
# no unverified re-download) and pass '' packages so the action does NOT run
# the flaky `sdkmanager tools platform-tools` install — the corrupt-zip
# surface that failed the "Set up Android SDK" step (#389).
cmdline-tools-version: "14742923"
packages: ""
- name: Set up Gradle
uses: gradle/actions/setup-gradle@3f131e8634966bd73d06cc69884922b02e6faf92 # v6.2.0
@@ -255,8 +689,25 @@ jobs:
path: /usr/local/lib/android/sdk/system-images/android-37.0
key: sysimg-android-37.0-google_apis_ps16k-x86_64
# verify -> reject -> retry (#389): the preview emulator + 16 KB-page system image
# download here; a corrupt/truncated package zip ("Error reading Zip content ...") is
# purged and re-downloaded clean instead of failing on sdkmanager's bare exit 1.
- name: Install SDK packages + preview system image
run: sdkmanager "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS" "platform-tools" "emulator" "$API37_IMAGE"
run: python3 .github/scripts/setup_android_sdk.py install "platform-tools" "emulator" "$ANDROID_PLATFORM" "$ANDROID_BUILD_TOOLS" "$API37_IMAGE"
# Save the verified command-line tools + platform/build-tools only on success and only
# on a miss. The ~1 GB system image keeps its own cache above; the emulator re-downloads
# (self-healing via the retry) to keep this shared key small.
- name: Save Android SDK cache
if: success() && steps.android-sdk-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
/usr/local/lib/android/sdk/cmdline-tools/20.0
/usr/local/lib/android/sdk/platforms
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/platform-tools
key: android-sdk-v1-${{ runner.os }}-clt14742923-plat37.0-bt37.0.0
- name: Create API 37 AVD
run: |
@@ -274,13 +725,53 @@ jobs:
run: |
set -euo pipefail
EMU_LOG="${RUNNER_TEMP:-/tmp}/emulator.log"
LOGCAT_LOG="${RUNNER_TEMP:-/tmp}/logcat.txt"
DIAG_LOG="${RUNNER_TEMP:-/tmp}/boot-diagnostics.txt"
# Wedge (hang) smoking-gun capture (#404) — see capture_wedge / run_shard below.
WEDGE_LOG="${RUNNER_TEMP:-/tmp}/wedge-diagnostics-api37-shard${{ matrix.shard }}.txt"
WEDGE_TIMEOUT=1200
GPU_MODE="swiftshader_indirect"
# On a boot timeout, capture the full system state (accel/KVM/GPU/mem/disk/AVD config +
# emulator.log tail) into $DIAG_LOG for the artifact upload, then print a CONCISE summary
# (accel/KVM status + last 50 lines of emulator.log) to the step log so the cause is
# visible in the run output without downloading artifacts. Every probe is guarded (|| true)
# so a missing tool can't abort the retry under `set -e`.
dump_diagnostics() {
local attempt="$1" accel kvm
accel=$("$ANDROID_SDK_ROOT/emulator/emulator" -accel-check 2>&1) || true
kvm=$(ls -l /dev/kvm 2>&1) || true
{
echo "===== API 37 boot diagnostics (attempt $attempt) ====="
echo "--- adb devices ---"; adb devices 2>&1 || true
echo "--- emulator -accel-check ---"; echo "$accel"
echo "--- /dev/kvm ---"; echo "$kvm"
echo "--- GPU mode ---"; echo "$GPU_MODE"
echo "--- free memory ---"; free -h 2>&1 || true
echo "--- free disk ---"; df -h 2>&1 || true
echo "--- AVD config.ini ---"; cat "${ANDROID_AVD_HOME:-$HOME/.android/avd}/api37.avd/config.ini" 2>&1 || true
echo "--- emulator.log (tail 200) ---"; tail -200 "$EMU_LOG" 2>&1 || true
} >> "$DIAG_LOG" 2>&1 || true
echo "----- BOOT FAILURE SUMMARY (attempt $attempt) -----"
echo "accel-check: $accel"
echo "/dev/kvm: $kvm"
echo "--- emulator.log (tail 50) ---"; tail -50 "$EMU_LOG" 2>&1 || true
}
boot_emulator() {
echo "::group::Start API 37 emulator (attempt $1)"
# Capture the emulator's own output — without this a boot failure is invisible.
# -verbose -debug init,avd_config,kernel turns boot logging on by default so a boot flake
# is diagnosable from $EMU_LOG; DIAGNOSTICS ONLY — no boot-affecting flag is changed.
"$ANDROID_SDK_ROOT/emulator/emulator" -avd api37 \
-no-window -no-audio -no-boot-anim -no-snapshot -accel on \
-gpu swiftshader_indirect -camera-back none -camera-front none > "$EMU_LOG" 2>&1 &
-gpu "$GPU_MODE" -camera-back none -camera-front none \
-verbose -debug init,avd_config,kernel > "$EMU_LOG" 2>&1 &
# Stream logcat from the moment the device registers (wait-for-device blocks until then)
# into a file that survives to the artifact upload. Appended (with a header) per attempt.
echo "===== logcat (attempt $1) =====" >> "$LOGCAT_LOG"
adb wait-for-device logcat -v time >> "$LOGCAT_LOG" 2>&1 &
logcat_pid=$!
# ONE bounded wait covering both device registration and full boot, so a stuck emulator
# fails fast instead of hanging the whole job until the 35-min cap (the original bug).
if timeout 300 adb wait-for-device shell \
@@ -289,19 +780,100 @@ jobs:
fi
echo "::endgroup::"
echo "::warning::API 37 emulator did not boot within 300s (attempt $1)"
adb devices || true
echo "--- emulator.log (tail) ---"; tail -120 "$EMU_LOG" || true
dump_diagnostics "$1"
kill "$logcat_pid" 2>/dev/null || true
adb emu kill 2>/dev/null || true
sleep 5
return 1
}
# Start the adb daemon up-front (mirrors api37_e2e.py wait_for_boot) so attempt 1 can't
# lose the adb-server "Address already in use" bind race that flaked #370's boot.
adb start-server || true
booted=0
for attempt in 1 2; do boot_emulator "$attempt" && { booted=1; break; }; done
[ "$booted" = "1" ] || { echo "::error::API 37 preview emulator failed to boot after 2 attempts"; exit 1; }
adb shell input keyevent 82 || true
./gradlew connectedDebugAndroidTest --stacktrace
# WEDGE (hang) smoking-gun capture (#404). On the wrapper `timeout` below (exit 124), grab
# the smoking gun WHILE this hand-provisioned emulator is still alive (it stays up until the
# "Shut down emulator" step): which test was running, SIGQUIT (kill -3) thread dumps of the
# app + instrumentation processes (ART -> logcat + /data/anr), activity/window state, and —
# the boot-race crux — whether the binder services are published. Every probe guarded so a
# missing tool / dead device can't abort it; appended so both attempts survive. `|| true`
# keeps it from tripping this step's `set -e`.
capture_wedge() {
{
echo "==================================================================="
echo "===== E2E WEDGE — API 37 preview shard ${{ matrix.shard }} — $1 ====="
echo "===== $(date -u +%FT%TZ) — after ${WEDGE_TIMEOUT}s wrapper timeout ====="
echo "==================================================================="
echo "--- snapshot: N/A — preview cold-boots (-no-snapshot); no AVD snapshot cache ---"
echo "--- running/last instrumented test (logcat TestRunner) ---"
grep -a TestRunner "$LOGCAT_LOG" 2>/dev/null | tail -25 || true
echo "--- getprop sys.boot_completed ---"
adb shell getprop sys.boot_completed 2>&1 || true
echo "--- getprop init.svc.* (per-service init state) ---"
adb shell getprop 2>&1 | grep -a init.svc || true
echo "--- service list (are binder services published?) ---"
adb shell service list 2>&1 || true
for svc in input window activity; do
echo "--- service check $svc ---"
adb shell service check "$svc" 2>&1 || true
done
echo "--- pids ---"
APP_PID="$(adb shell pidof org.libremail.app 2>/dev/null | tr -d '\r')" || true
TEST_PID="$(adb shell pidof org.libremail.app.test 2>/dev/null | tr -d '\r')" || true
echo "app pid: ${APP_PID:-<none>}"
echo "test pid: ${TEST_PID:-<none>}"
echo "--- SIGQUIT (kill -3) thread dumps -> ART writes to logcat + /data/anr ---"
for pid in $APP_PID $TEST_PID; do
[ -n "$pid" ] && adb shell kill -3 "$pid" 2>&1 || true
done
sleep 5
echo "--- /data/anr/* (SIGQUIT + ANR traces) ---"
adb shell 'cat /data/anr/* 2>/dev/null' 2>&1 || true
echo "--- dumpsys activity ---"
adb shell dumpsys activity 2>&1 || true
echo "--- dumpsys window ---"
adb shell dumpsys window 2>&1 || true
echo "--- logcat -d (tail 400 — includes the SIGQUIT thread dump) ---"
adb logcat -d 2>&1 | tail -400 || true
echo "--- emulator accel / kvm / mem / disk ---"
"$ANDROID_SDK_ROOT/emulator/emulator" -accel-check 2>&1 || true
ls -l /dev/kvm 2>&1 || true
free -h 2>&1 || true
df -h 2>&1 || true
} >> "$WEDGE_LOG" 2>&1 || true
echo "::warning::E2E API 37 preview shard ${{ matrix.shard }} WEDGED ($1) — see the wedge-diagnostics-api37-preview-shard${{ matrix.shard }} artifact"
}
# Run only this matrix leg's shard. AndroidJUnitRunner hashes each test name into one of
# numShards buckets and runs only shardIndex's bucket. The args flow through AGP's
# -Pandroid.testInstrumentationRunnerArguments.* channel (the same one local_instrumented.py
# uses for .class=…) — no GMD, no orchestrator, no Gradle-side change. The gradle run is
# wrapped in the #404 wrapper `timeout`: a wedge trips it (exit 124) -> capture_wedge runs,
# then the shard returns 124 so the retry / gate still see a failure. `|| status=$?` makes
# the exit code survive this step's `set -e`; -k 30s SIGKILLs a gradle client that ignores
# SIGTERM.
run_shard() {
local status=0
timeout -k 30s "$WEDGE_TIMEOUT" ./gradlew connectedDebugAndroidTest --stacktrace \
"-Pandroid.testInstrumentationRunnerArguments.numShards=${API37_NUM_SHARDS}" \
"-Pandroid.testInstrumentationRunnerArguments.shardIndex=${{ matrix.shard }}" || status=$?
if [ "$status" -eq 124 ]; then capture_wedge "$1"; fi
return "$status"
}
# Retry this shard's test run ONCE on failure — retry parity with the stable `e2e` matrix
# (which retries once, so a flaky test self-heals on API 29–36 but would otherwise wedge
# the required gate on API 37, e.g. #370). Per-shard, so the retry re-runs only this shard's
# ~T/N tests, not the whole suite. MITIGATION, NOT A FIX: a blanket retry also masks genuine
# regressions, so the retried-but-passed case is flagged as a ::warning:: and the real fix
# stays test-level. See docs/perf/api37-e2e-sharding-spike.md. A wedge on EITHER attempt is
# captured (capture_wedge runs inside run_shard on exit 124).
run_shard "attempt 1" || { echo "::warning::API 37 shard ${{ matrix.shard }} test run failed — retrying once"; run_shard "attempt 2 (retry)"; }
- name: Dump emulator log on failure
if: failure()
@@ -309,6 +881,31 @@ jobs:
echo "--- emulator.log ---"; tail -200 "${RUNNER_TEMP:-/tmp}/emulator.log" 2>/dev/null || echo "(none)"
echo "--- logcat ---"; adb logcat -d 2>/dev/null | tail -120 || echo "(device unavailable)"
# Always upload the emulator log + captured logcat + boot-diagnostics dump so a boot flake
# (which can time out or be cancelled) is diagnosable from artifacts without a re-run.
- name: Upload emulator boot diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
# Shard-unique name — upload-artifact@v7 errors on duplicate artifact names.
name: e2e-api37-boot-diagnostics-shard${{ matrix.shard }}
path: |
${{ runner.temp }}/emulator.log
${{ runner.temp }}/logcat.txt
${{ runner.temp }}/boot-diagnostics.txt
if-no-files-found: warn
# Wedge-specific smoking gun (#404): only present when the wrapper `timeout` tripped (a hang)
# on either shard attempt — separate from the boot-diagnostics artifact above. `if-no-files-
# found: ignore` keeps healthy runs quiet (no wedge => no file). Shard-unique name.
- name: Upload wedge diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: wedge-diagnostics-api37-preview-shard${{ matrix.shard }}
path: ${{ runner.temp }}/wedge-diagnostics-api37-shard${{ matrix.shard }}.txt
if-no-files-found: ignore
- name: Shut down emulator
if: always()
run: adb emu kill || true
@@ -317,7 +914,8 @@ jobs:
if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-test-report-api37-preview
# Shard-unique name — upload-artifact@v7 errors on duplicate artifact names.
name: e2e-test-report-api37-preview-shard${{ matrix.shard }}
path: app/build/reports/androidTests/connected/
if-no-files-found: warn
@@ -328,16 +926,48 @@ jobs:
ci-passed:
name: CI passed
if: always()
needs: [static-analysis, debug-build, unit-tests, e2e, e2e-preview]
# `traffic-control` is intentionally NOT listed here — it has been extracted to its own
# (mothballed/disabled) workflow, .github/workflows/traffic-control.yml, and the heavy jobs
# below no longer depend on it, so it plays no part in this gate. The `traffic-control-tests`
# job (its pure-Python decision-core unit tests) IS a required input below.
#
# E2E path-filter (issue #399): `e2e` / `e2e-preview` are SKIPPED for test-only/docs/script
# PRs (see the `changes` job). A skip there is INTENTIONAL and must PASS, so — unlike the
# naive `contains(needs.*.result, 'skipped')` gate this replaces — 'skipped' is TOLERATED for
# those two jobs only. The gate stays fail-safe by BLOCKING on:
# * changes != success (a broken/failed filter never waves a PR through)
# * traffic-control-tests / static-analysis / debug-build / unit-tests != success
# * e2e == failure OR cancelled (a real E2E failure/cancel still blocks)
# * e2e-preview == failure OR cancelled
# i.e. e2e/e2e-preview may be ONLY 'success' or 'skipped'; every other required job must be
# 'success'. (If `changes` itself fails, e2e/e2e-preview skip — but `changes != success`
# blocks the merge anyway, so a broken filter is never waved through.)
needs: [changes, traffic-control-tests, static-analysis, debug-build, unit-tests, e2e, e2e-preview]
runs-on: ubuntu-latest
steps:
- name: Verify every required job succeeded
if: ${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}
# Always echo every required job's result (and the filter decision) for debuggability,
# before the gate step decides pass/fail.
- name: Echo required job results
run: |
echo "Required CI jobs did not all succeed:"
echo " static-analysis: ${{ needs.static-analysis.result }}"
echo " debug-build: ${{ needs.debug-build.result }}"
echo " unit-tests: ${{ needs.unit-tests.result }}"
echo " e2e: ${{ needs.e2e.result }}"
echo " e2e-preview: ${{ needs.e2e-preview.result }}"
echo "Required-job results (an E2E 'skipped' is allowed ONLY via the #399 path-filter):"
echo " changes: ${{ needs.changes.result }} (e2e_needed=${{ needs.changes.outputs.e2e_needed }})"
echo " traffic-control-tests: ${{ needs.traffic-control-tests.result }}"
echo " static-analysis: ${{ needs.static-analysis.result }}"
echo " debug-build: ${{ needs.debug-build.result }}"
echo " unit-tests: ${{ needs.unit-tests.result }}"
echo " e2e: ${{ needs.e2e.result }}"
echo " e2e-preview: ${{ needs.e2e-preview.result }}"
- name: Fail unless every required job passed (an intentionally path-filtered E2E skip is OK)
if: >-
needs.changes.result != 'success' ||
needs.traffic-control-tests.result != 'success' ||
needs.static-analysis.result != 'success' ||
needs.debug-build.result != 'success' ||
needs.unit-tests.result != 'success' ||
needs.e2e.result == 'failure' || needs.e2e.result == 'cancelled' ||
needs.e2e-preview.result == 'failure' || needs.e2e-preview.result == 'cancelled'
run: |
echo "::error::Required CI jobs did not all succeed (see the results above)."
echo "A path-filtered E2E 'skipped' is allowed; an E2E 'failure'/'cancelled', or any"
echo "non-success in changes/traffic-control-tests/static-analysis/debug-build/unit-tests, is not."
exit 1
+100
View File
@@ -0,0 +1,100 @@
# SPDX-License-Identifier: GPL-3.0-or-later
# Extracted verbatim from .github/workflows/ci.yml (where it used to run first and gate the
# heavy jobs via `needs: traffic-control`). It is being mothballed pending a rebuild as a
# published GitHub Action and will be disabled after this merges; the heavy CI jobs no longer
# depend on it. The decision core it drives (.github/scripts/traffic_control.py) is unchanged
# and still unit-tested by the `traffic-control-tests` job in ci.yml.
name: Traffic control (runner priority)
# The job reads github.event.pull_request.number, so it needs PR context.
on: pull_request
jobs:
# ── Priority-based runner orchestration ─────────────────────────────
# Runs FIRST (the heavy jobs below all `needs: traffic-control`). It reads THIS
# PR's P0–P9 label, `broken` label, and draft state to order runner access. The
# decision logic lives in .github/scripts/traffic_control.py — a pure, unit-tested
# core (see .github/scripts/test_traffic_control.py) plus a thin gh-I/O shell; this
# step just checks out the repo and runs it.
#
# Effective priority: a `broken` OR `draft` PR => 10 (BOTTOM, below P9), overriding
# any P0–P9; else the lowest-numbered P0–P9 label present (P0 = highest); else P5.
#
# • P0 = EMERGENCY ONLY (app broken in production / emergency security update).
# P0 PREEMPTS: it cancels the in-progress / queued CI runs of ALL strictly-
# LOWER-priority OTHER open PRs to grab their runners immediately. A preempted
# PR simply re-runs on its next push / autoupdate rebase. P0 is the ONLY
# priority that preempts a *normal* lower run — P1–P9 never bump those (a
# higher PR may still reclaim a broken/draft lower run — see below).
#
# • P1–P9 = YIELD WITHOUT BUMPING a *normal* lower run. They do NOT cancel a
# normal lower-priority run already going — a higher-priority PR does not evict
# it, it just takes the next free slot (it MAY still reclaim a broken/draft
# lower run — see below). Mechanism: a bounded hold-back. This job defers (up to
# HOLD_BACK_BUDGET_SECONDS, kept well under timeout-minutes) while any strictly-
# higher-priority OTHER open PR still has an active/queued CI run, and — within
# its OWN priority level — while any peer is ordered ahead of it (an in-flight
# run keeps its place; then oldest createdAt first). It proceeds the moment it
# is at the front, or when the budget elapses (a PR never blocks itself).
#
# • `broken` / `draft` = BOTTOM (effective P10). Always yields, never preempts —
# and because its run is wasted (a broken PR can't merge; a draft isn't merge-
# ready), ANY higher-priority PR (not just P0) MAY cancel that run to reclaim
# its runner (still the strictly-lower rule: broken/draft is the bottom, so any
# ready PR outranks it). A maintainer marks a stuck/failing PR `broken` to drop
# it below everything so others aren't blocked behind it AND may reclaim its
# runner; a draft behaves the same until it is marked ready for review.
#
# Hard safety invariants, enforced in the script:
# • never cancels a run on main / a push event (the gh query filters
# --event pull_request and drops headBranch == main);
# • never cancels THIS PR's own run (skips self by PR number + run id);
# • never cancels an equal-or-higher-priority PR (only strictly-lower, prio > self);
# • P1–P9 never bump a *normal* lower run (they only reclaim broken/draft) —
# otherwise they just wait (bounded), then proceed.
#
# Honest limitation: GitHub Actions has no native priority queue and assigns
# runners roughly FIFO, so the hold-back is a BEST-EFFORT head-start, not a hard
# guarantee — under sustained contention the bounded wait can expire before a
# higher-priority PR drains. The waiting job also occupies a (cheap, short-lived)
# runner meanwhile, which is exactly why the wait is kept bounded.
#
# It is deliberately NOT a merge-gate check: it is absent from `ci-passed`'s
# needs, every API call is guarded, the script always exits 0, and the step is
# `continue-on-error` — so a hiccup (API error, missing permission, fork PR) can
# never fail or block CI. The heavy jobs only *order* after it via `needs`; if it
# were ever skipped/failed they'd be skipped, which `ci-passed` treats as a gate
# failure (fail-safe: blocks merge, never spuriously passes).
traffic-control:
name: Traffic control (runner priority)
runs-on: ubuntu-latest
timeout-minutes: 6 # hard backstop; the P1–P9 hold-back budget stays well under this
permissions:
contents: read # check out .github/scripts/traffic_control.py
actions: write # cancel lower-priority runs (P0 emergencies + broken/draft reclaim)
pull-requests: read # read PR P0–P9 labels + draft state
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
# On a `pull_request` run this is the PR number and the script does its full in-run
# runner-priority orchestration. On a scheduler `workflow_dispatch` run (issue #349)
# the event is not `pull_request`, so the script no-ops here (`--mode orchestrate`
# only acts on pull_request events) — priority was ALREADY applied at trigger time by
# ci-trigger.yml, so re-doing the in-run hold-back would just waste runner time. The
# `|| inputs.pr` keeps the number in the log for a dispatched run.
SELF_PR: ${{ github.event.pull_request.number || inputs.pr }}
# P1–P9 bounded hold-back knobs, read by traffic_control.py. BUDGET must stay
# comfortably below timeout-minutes so the poll loop always exits 0 before the
# hard job timeout fires — a timed-out job would skip the heavy jobs and fail
# `ci-passed`.
HOLD_BACK_BUDGET_SECONDS: "180"
HOLD_BACK_POLL_SECONDS: "15"
steps:
- name: Check out source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
# gh is auto-configured from GH_TOKEN / GH_REPO; python3 is preinstalled on the
# runner. The script guards every API call and always exits 0 (belt-and-braces
# with continue-on-error), so it can never fail or block CI.
- name: Apply runner priority (P0/broken/draft preempt; P1–P9 hold back)
continue-on-error: true
run: python3 .github/scripts/traffic_control.py
+6
View File
@@ -30,6 +30,8 @@ secrets.properties
# Log Files
*.log
# ...but keep the device-testing parser fixtures (verbatim logcat slices used as test inputs)
!scripts/device-testing/tests/fixtures/*.log
# Android Studio / IntelliJ
.idea/
@@ -42,5 +44,9 @@ captures/
# Kotlin
.kotlin/
# Python (dev/CI helper scripts under .github/scripts, .claude/…)
__pycache__/
*.pyc
# Claude Code — personal settings (the shared settings.json IS committed)
.claude/settings.local.json
+146
View File
@@ -0,0 +1,146 @@
# SPDX-License-Identifier: GPL-3.0-or-later
#
# ============================================================================
# Mergify configuration — PHASE 1: serial merge queue (issue #409).
#
# Spec: docs/ci/mergify-integration-spec.md + docs/ci/mergify.yml.proposed
# (issues #407 / #408).
# Schema: https://docs.mergify.com/configuration/file-format/
# Verified against the LIVE Mergify docs on 2026-07-06 (queue rules,
# the queue action, priority rules, parallel checks, batches, setup and
# lifecycle pages) — the config format evolves, so this is not from memory.
# ============================================================================
#
# WHAT THIS DOES
# A SERIAL merge queue that ends the manual serial-bump grind and supersedes the
# hand-rolled "poor-man's merge queue" (autoupdate.yml + ci-trigger.yml +
# traffic-control.yml — all already `disabled_manually`). Mergify updates each
# queued PR onto the latest `main`, re-runs CI, and merges it with a MERGE COMMIT
# when the single required gate — the "CI passed" check — is green. One PR at a
# time, in P0–P9 priority order.
#
# HARD INVARIANTS (do NOT relax without the trilemma decision recorded in the spec):
# * require-up-to-date STAYS ON. This is Phase 1 = batch_size 1 + merge_method:
# merge — the ONLY trilemma combination that keeps GitHub's "Require branches to
# be up to date before merging" LITERALLY enabled AND preserves the merge-commit
# policy. Mergify honours it by updating each PR onto the latest `main` and
# re-running CI before merging ("Updates PRs against the latest main before
# merging" — docs.mergify.com/merge-queue/setup). NO batching: batching would
# require turning that checkbox OFF (docs.mergify.com/merge-queue/batches) and is
# the blocked Phase 2 / issue #410 — explicitly OUT OF SCOPE here.
# * The single required status check stays "CI passed" — the exact `name:` of the
# `ci-passed` job in .github/workflows/ci.yml. NOT "ci-passed". A wrong name means
# PRs queue but never merge.
# ---------------------------------------------------------------------------
# queue_rules — how a queued PR is validated and merged.
# ---------------------------------------------------------------------------
queue_rules:
- name: default
# Final merge gate. Merge ONLY when the single required context is green (the exact
# same check branch protection requires), the PR is not a draft, has no merge
# conflicts, and is not flagged `broken`. NOTE: branch protection requires 0
# approvals here (the active repository ruleset sets required_approving_review_count
# = 0), so there is deliberately NO `#approved-reviews-by` condition — adding one
# would wedge the solo-maintainer flow, where nobody can approve their own PR.
merge_conditions:
- check-success = CI passed
- -draft
- -conflict
- label != broken
# SERIAL: exactly one PR per merge. No batching (Phase 2 / #410). One merge commit
# per PR, which is what lets require-up-to-date stay literally ON.
batch_size: 1
# Merge commit — never squash / rebase / fast-forward (repo policy: merges use
# merge commits, never squash).
merge_method: merge
# ---------------------------------------------------------------------------
# merge_queue — queue-wide options.
# ---------------------------------------------------------------------------
merge_queue:
# Validate ONE PR at a time — true serial, no speculative parallel checks. This is the
# strictest, unambiguously require-up-to-date-compatible setting: Mergify updates the
# REAL PR branch onto the latest `main`, runs CI on that branch, and merges on the real
# green "CI passed" — with no speculative temp-branch/real-branch check mismatch to
# reason about. It also caps the expensive, wedge-prone ~15-min E2E matrix at a single
# concurrent run. Raising this (speculative parallelism) is a throughput optimisation to
# weigh alongside the Phase 2 / #410 batching decision — not part of serial Phase 1.
max_parallel_checks: 1
# ---------------------------------------------------------------------------
# priority_rules — map the repo's P0–P9 labels onto queue priority.
# Higher number merges first (Mergify keywords: low=1000 / medium=2000 / high=3000;
# numeric range 1–10000). P0 is emergency-only and outranks everything. PRs with no P-label
# fall to Mergify's default `medium` (2000).
# ---------------------------------------------------------------------------
priority_rules:
- name: p0-emergency
conditions:
- label = P0
priority: 10000
allow_checks_interruption: true
- name: p1
conditions:
- label = P1
priority: 9000
allow_checks_interruption: true
- name: p2
conditions:
- label = P2
priority: 8000
allow_checks_interruption: true
- name: p3
conditions:
- label = P3
priority: 7000
allow_checks_interruption: true
- name: p4
conditions:
- label = P4
priority: 6000
allow_checks_interruption: true
- name: p5
conditions:
- label = P5
priority: 5000
allow_checks_interruption: true
- name: p6
conditions:
- label = P6
priority: 4000
allow_checks_interruption: true
- name: p7
conditions:
- label = P7
priority: 3000
allow_checks_interruption: true
- name: p8
conditions:
- label = P8
priority: 2000
- name: p9
conditions:
- label = P9
priority: 1000
# ---------------------------------------------------------------------------
# pull_request_rules — WHICH PRs enter the queue.
# The `queue` action is what actually ADDS a PR to the merge queue: per
# docs.mergify.com/merge-queue/lifecycle, queue_conditions alone do NOT auto-queue a PR —
# a queue action (or an `@mergifyio queue` command / auto_merge) is required, otherwise the
# "Mergify Merge Queue" check sits permanently pending. A PR is queued as soon as it is green
# on "CI passed", targets `main`, is not a draft, has no conflicts, and is not flagged
# `broken`. `broken` / `draft` PRs are never queued.
# ---------------------------------------------------------------------------
pull_request_rules:
- name: Queue green, non-draft, non-conflicting PRs targeting main
conditions:
- base = main
- -draft
- -conflict
- label != broken
- check-success = CI passed
actions:
queue:
name: default
+53 -4
View File
@@ -16,6 +16,7 @@ Use a **JDK 17–21** for the Gradle daemon. AGP 9.2 does **not** support JDK 25
./gradlew :app:testDebugUnitTest # JVM unit tests
./gradlew :app:lintDebug # Android lint
./gradlew :app:ktlintCheck :app:detekt # static analysis (CI's "Static analysis" gate)
./gradlew :app:jacocoTestReport # JVM unit-test coverage (XML+HTML under app/build/reports/jacoco/)
# single unit test:
./gradlew :app:testDebugUnitTest --tests "org.libremail.data.SomeClassTest"
```
@@ -26,10 +27,37 @@ or via Gradle Managed Devices `./gradlew e2eGroupDebugAndroidTest` (whole matrix
`app/build.gradle.kts` must stay in lockstep with the E2E matrix in `.github/workflows/ci.yml`.
**Before treating a change as done**, run the fast CI gate: `assembleDebug` +
`testDebugUnitTest` + `lintDebug` + `ktlintCheck` + `detekt` (the `/preflight` skill does
this). `ktlintCheck`/`detekt` cover the `test`/`androidTest` source sets that `lintDebug`
skips, so they catch style violations that would otherwise fail CI's Static analysis gate.
Emulator E2E is left to CI unless asked.
`testDebugUnitTest` + `jacocoTestCoverageVerification` + `compileDebugAndroidTestKotlin` +
`lintDebug` + `ktlintCheck` + `detekt` + the local emulator E2E — the instrumented test class(es)
you changed via `python3 .claude/skills/preflight/local_instrumented.py <classes>` + the API 37
preview E2E via `python3 .claude/skills/preflight/api37_e2e.py` (the `/preflight` skill does all
of this). `jacocoTestCoverageVerification` runs right after `testDebugUnitTest` (it reads that
task's JVM exec data) and enforces the whole-app **no-regression line-coverage floor (currently
0.84)**, catching coverage regressions locally instead of only in CI (the exact class of failure
that reached CI on #367). `compileDebugAndroidTestKotlin` compiles the `androidTest` source set
that the static part of the gate skips, catching E2E/instrumented-test compile errors before
they surface only in CI. `ktlintCheck`/`detekt` cover the `test`/`androidTest` source sets that
`lintDebug` skips, so they catch style violations that would otherwise fail CI's Static analysis
gate. The local E2E does **not** use Gradle Managed Devices (`apiXXDebugAndroidTest`): GMD's
snapshot step fails locally under the AEHD 2.2 hypervisor. Instead `local_instrumented.py`
cold-boots one existing AVD by hand (no GMD, no snapshot) and runs `connectedDebugAndroidTest`
filtered to the class(es) you pass — run the ones you changed; the full ~114-test suite wedges
mid-run locally. API 37 (preview) has no Gradle Managed Device — its
only image is the nonstandard `android-37.0` / `google_apis_ps16k` pairing (see the comment above
`testOptions.managedDevices` in `app/build.gradle.kts`) — so `api37_e2e.py` hand-provisions it,
mirroring CI's `e2e-preview` job (same image + emulator flags, except it uses host-GPU
`-gpu auto-no-window` locally vs CI's headless `-gpu swiftshader_indirect`), boots it headless,
runs `connectedDebugAndroidTest`, and tears it down. Both local E2E steps are required; the
full multi-API matrix (API 29–37) stays CI's job. Emulators need a free hardware
hypervisor (VT-x/WHPX), so shut down VirtualBox/other VMs first or the AVD hangs at 0% CPU.
**Dev scripts: prefer Python (stdlib).** Auxiliary dev / CI-helper scripts — like the preflight
E2E runners (`.claude/skills/preflight/local_instrumented.py`, `api37_e2e.py`) and
`.claude/hooks/check-spdx.py` — are written in **Python 3, standard library only**, for
cross-platform portability. The primary dev box is Windows, where bash-only helpers need Git Bash
and hit gaps (`jq` missing, `taskkill` vs `kill`, path/quoting). **Do not add new bash-only
(`.sh`) or PowerShell-only dev scripts**; write new helpers in Python (or extend the existing
ones). Scope is auxiliary tooling only — product code stays Kotlin and Gradle stays Kotlin DSL.
## Build-config gotchas
@@ -58,6 +86,27 @@ JVM unit tests use JUnit4 + `kotlin.test`, **Turbine** for `Flow`, **MockK** for
**GreenMail** for a real in-process IMAP/SMTP server, and coroutines-test. `org.json` is
pulled in as a real dependency for unit tests because `android.jar`'s version is a no-op stub.
## Definition of done
A change is not done until it ships with passing **unit tests** and **E2E/instrumented tests**
that exercise the new or changed behaviour. Writing and committing that E2E/instrumented test
is a required part of every task — and the test must actually **run and pass**, not merely
compile: preflight runs the changed instrumented test class(es) on a locally cold-booted
emulator via `local_instrumented.py` (no Gradle Managed Devices — they fail locally) plus the
API 37 preview via `api37_e2e.py` (hand-provisioned, mirroring CI's `e2e-preview` job) — and both
must be green before the change is done. CI then runs the full multi-API matrix plus the API 37
preview job.
No **app source-code** change is complete without **appropriate logging** added at its key
points — lifecycle transitions, error/fallback paths, significant state changes — so behaviour is
diagnosable from a user's debug report. Log through the `AppLog` facade
(`org.libremail.reporting.AppLog`), which mirrors to Logcat **and** the in-memory
`RingLogBuffer` that feeds a `DebugReport` — never raw `android.util.Log` (a detekt guard forbids
it). Logging must be **PII-free**: never log emails, server hosts, message content, or
credentials — use `accountLogRef(account.id)` for account references; throwables passed to
`AppLog` are auto-scrubbed. This applies to app source changes; pure test/config/doc changes
don't need new logging.
## Repo etiquette
- Branch off `main`; branch names like `feat-…` / `fix-…`. PRs target `main` and must pass
+348 -2
View File
@@ -1,4 +1,7 @@
// SPDX-License-Identifier: GPL-3.0-or-later
import org.gradle.testing.jacoco.plugins.JacocoTaskExtension
import org.gradle.testing.jacoco.tasks.JacocoCoverageVerification
import org.gradle.testing.jacoco.tasks.JacocoReport
import java.util.Properties
plugins {
@@ -9,6 +12,11 @@ plugins {
id("org.jetbrains.kotlin.plugin.compose")
id("com.google.devtools.ksp")
id("com.google.dagger.hilt.android")
// JaCoCo (Gradle built-in) — unit-test code-coverage reporting (issue #192). The base `jacoco`
// plugin auto-instruments the JVM `testDebugUnitTest` task; the jacocoTestReport task below turns
// its exec data into XML + HTML. AGP-9-safe: it does NOT apply org.jetbrains.kotlin.android (which
// ClassCastExceptions against AGP 9's built-in-Kotlin DSL — see CLAUDE.md) and touches no variant DSL.
jacoco
// Lint/format — resolved from the Gradle Plugin Portal (not the buildscript classpath).
alias(libs.plugins.ktlint)
alias(libs.plugins.detekt)
@@ -58,6 +66,12 @@ android {
buildConfigField("String", "OUTLOOK_OAUTH_CLIENT_ID", "\"$outlookOAuthClientId\"")
buildConfigField("String", "OUTLOOK_OAUTH_REDIRECT_URI", "\"$outlookRedirectScheme://oauth2redirect\"")
buildConfigField("String", "DEBUG_REPORT_ENDPOINT", "\"$debugReportEndpoint\"")
// IMAP connection reuse (issue #357 Part 2, wiring the #125 spike): keep one authenticated
// IMAP connection warm per account instead of paying a cold CONNECT+TLS+LOGIN on every
// operation — the fix for Gmail throttling LibreMail's connect-per-operation traffic. ON by
// default; this is the safety switch: flip to "false" here (a build-config change, no Kotlin
// edit) to fall back to connect-per-operation if a server misbehaves with a kept-alive socket.
buildConfigField("Boolean", "IMAP_CONNECTION_REUSE", "true")
// AppAuth's bundled manifest requires this placeholder; it registers the redirect scheme on
// RedirectUriReceiverActivity so the Outlook sign-in redirect returns to the app.
manifestPlaceholders["appAuthRedirectScheme"] = outlookRedirectScheme
@@ -88,6 +102,14 @@ android {
} else {
signingConfigs.getByName("debug")
}
// Ship ARM only — arm64-v8a (64-bit) + armeabi-v7a (32-bit); x86/x86_64 are
// intentionally dropped. Scoped to this build type ONLY — do NOT move to defaultConfig
// or the debug type: CI's E2E matrix runs the debug build on x86_64 emulators
// (.github/workflows/ci.yml) and needs the x86_64 native libs (incl. libsqlcipher.so).
// See docs/play-compliance.md and docs/fdroid-compliance.md for the coverage tradeoff.
ndk {
abiFilters += listOf("arm64-v8a", "armeabi-v7a")
}
}
}
@@ -126,14 +148,35 @@ android {
}
testOptions {
// Robolectric-backed Compose UI unit tests (issue #373) need the merged Android resources
// (drawables, strings, the compiled resource table) on the JVM unit-test classpath so
// `stringResource(...)` and Material3 theming resolve without an emulator. Off by default in
// AGP; JVM tests that don't touch resources are unaffected.
unitTests.isIncludeAndroidResources = true
// Gradle Managed Devices define the per-API E2E matrix as config-as-code: one virtual
// device per supported Android version (a rolling ~7-year window, API 29 → latest stable).
// Run the whole matrix with `./gradlew e2eGroupDebugAndroidTest`, or one level with e.g.
// `./gradlew api29DebugAndroidTest`. Gradle provisions/boots/tears down the emulators and
// downloads the system images on first use. Keep this list in lockstep with the CI matrix in
// .github/workflows/ci.yml; when a new Android ships, add it and drop the oldest level that
// has fallen outside ~7 years. API 37 (preview) is exercised on the dev emulator until a
// stable managed-device image is published, so it is intentionally not listed here.
// has fallen outside ~7 years.
//
// API 37 (preview) is intentionally NOT listed here (re-confirmed 2026-07, see PR that added
// API 35/37 to preflight): its only published system image is the nonstandard
// "android-37.0" / google_apis_ps16k pairing that the e2e-preview job in
// .github/workflows/ci.yml installs directly via sdkmanager. ManagedVirtualDevice only knows
// how to build an "android-<apiLevel:Int>" package id (e.g. `apiLevel = 37` → "android-37")
// or an "android-<apiPreview:codename>" one — neither produces "android-37.0" — so there is
// no DSL path to this image today, the same root cause documented on e2e-preview for why
// reactivecircus/android-emulator-runner can't provision it either. issue #124's perf doc
// (docs/perf/issue-124-unified-inbox-paging.md) independently corroborates this: its API 37
// cross-check used a physical Pixel, not an emulator/AVD. Locally, preflight covers API 37
// by hand-provisioning it with .claude/skills/preflight/api37_e2e.py, which mirrors the
// e2e-preview job (same image + emulator flags, except it renders on the host GPU via
// `-gpu auto-no-window` locally instead of CI's headless `-gpu swiftshader_indirect`). Once
// a managed-device-compatible image is published, add `api37` here (and to the CI matrix),
// delete that script, and drop the e2e-preview job.
managedDevices {
localDevices {
listOf(29, 30, 31, 32, 33, 34, 35, 36).forEach { api ->
@@ -164,6 +207,285 @@ detekt {
config.setFrom(rootProject.file("config/detekt/detekt.yml"))
}
// Pin a modern JaCoCo (version catalog) so the coverage agent understands Kotlin 2.4.0 bytecode
// on JDK 21.
jacoco {
toolVersion = libs.versions.jacoco.get()
}
// The Robolectric-backed JVM Compose UI tests (#373) load the classes-under-test through
// Robolectric's sandbox classloader, which presents them to the JaCoCo agent WITHOUT a code-source
// location. JaCoCo skips no-location classes by default, so on-the-fly coverage for every composable
// exercised only by a Robolectric test would silently record as zero — the file would be removed
// from `jacocoNonJvmTestableSurface` yet contribute nothing but missed lines, dragging the bundle
// ratio DOWN instead of up. `isIncludeNoLocationClasses = true` makes the agent keep that coverage;
// `jdk.internal.*` is excluded because instrumenting those JDK classes breaks under JDK 17+.
tasks.withType<Test>().configureEach {
configure<JacocoTaskExtension> {
isIncludeNoLocationClasses = true
excludes = listOf("jdk.internal.*")
}
}
// Unit-test coverage (issue #192). Two tasks share ONE scoping so they can never measure different
// surfaces: `jacocoTestReport` (XML+HTML under build/reports/jacoco/jacocoTestReport/) and
// `jacocoTestCoverageVerification` (the no-regression gate, further down). Both read the exec data
// the base `jacoco` plugin records for the JVM `testDebugUnitTest` task, mapped against the debug
// variant's compiled Kotlin classes and the hand-written main sources. Instrumented/E2E coverage is
// out of scope (issue #192).
// Strip generated code from the denominator so the % reflects hand-written Kotlin. Verified
// against an actual compileDebugKotlin output tree: Room's KSP-generated `_Impl` DAOs/database
// and the Compose compiler's per-file ComposableSingletons holders are the only generated code
// that actually lands in classDirectories below (Room's KSP output is added as an extra Kotlin
// source root on the *same* compile task, so it comes out the same door as hand-written code).
// Hilt/Dagger's generated Java (Hilt_*, Dagger*_HiltComponents*, *_GeneratedInjector, *_Factory,
// *_MembersInjector, hilt_aggregated_deps) and AGP's BuildConfig/R/Manifest are compiled by a
// separate javac task (hiltJavaCompileDebug / compileDebugJavaWithJavac) into a directory this
// report never reads, so those patterns are conventional belt-and-suspenders in case that ever
// changes. DataBinding isn't enabled in this module (no buildFeatures.dataBinding/viewBinding),
// so there's nothing generated for it to exclude; if it's turned on later, add "**/BR.class",
// "**/DataBinderMapperImpl*.class" and "**/*Binding.class".
//
// Deliberately NOT excluded: Kotlin's own `$$inlined$` synthetic classes (e.g. for
// `Flow.map { ... }` in the repositories) — those hold real hand-written transform logic, not
// generated boilerplate, so stripping them would silently shrink the measured surface.
val jacocoGeneratedExcludes = listOf(
"**/R.class",
"**/R\$*.class",
"**/BuildConfig.*",
"**/Manifest*.*",
"**/Hilt_*.class",
"**/Dagger*.class",
"**/*_Hilt*",
"**/*_GeneratedInjector.class",
"**/hilt_aggregated_deps/**",
"**/dagger/**",
"**/*_Factory*",
"**/*_MembersInjector*",
"**/*_Provide*",
"**/*_Impl*",
"**/ComposableSingletons*",
)
// Scope the denominator to the JVM-testable surface (issues #290/#292, following the Phase-2 coverage
// audit): unlike `jacocoGeneratedExcludes` above, none of this is generated code — it is hand-written
// but structurally unreachable from a JVM unit test, so counting it against the metric just measures
// how much Compose/framework glue exists rather than how well the logic is tested. Four buckets:
// 1. Compose screen/component render code. Historically only exercisable via an emulator, so it was
// excluded here. Issue #373 changes that: Robolectric runs the Android framework on the JVM, so a
// `createComposeRule()` test in the `test` source set now gives these files real JVM coverage
// without an emulator. This bucket therefore SHRINKS one screen at a time — each glob is deleted
// in the same PR that adds that screen's Robolectric JVM Compose test. AddAnotherAccountScreen was
// the first (see AddAnotherAccountScreenJvmTest) and has been removed below; the rest are tracked
// as per-area conversion tickets under #373. The coverage-floor re-ratchet is deferred until the
// whole conversion is done and stable (#373) — do NOT raise it in a conversion PR.
// 2. Android framework entry points the OS instantiates directly (Activity/Service/Application/
// BackupAgent) rather than the app's own code constructing them.
// 3. Hilt DI modules — `@Provides`/`@Binds` one-liners with no branching logic.
// 4. The `src/debug` cold-open probe (issue #221), a `ContentProvider` that only runs in a forked
// instrumented process (see its kdoc) and is never packaged in a release build anyway.
//
// KEPT IN SCOPE — this corrects #292, which excluded `**/*Worker*`: the six WorkManager workers
// (SyncWorker, BackfillWorker, PruneWorker, SendWorker, ReportPurgeWorker, ReportUploadWorker) are
// all directly unit-tested today (construct-the-worker-and-call-doWork(), e.g. SyncWorkerTest,
// SendWorkerTest), so they carry real tested logic and belong in BOTH the numerator and denominator.
// Only their Hilt wiring (WorkManagerModule) is excluded, and that falls under `**/di/**` below — so
// there is intentionally no `**/*Worker*` glob in the list.
//
// Also deliberately NOT excluded, even though each sits in a package/pattern above and renders UI:
// files that carry plain, unit-tested logic alongside their `@Composable` functions. JaCoCo has no
// finer granularity than a class file, and Kotlin compiles every top-level function in a .kt file —
// `@Composable` or not — into the SAME facade class (`<File>Kt.class`); excluding that class would
// silently zero out the tested function's coverage too, not just the render code's. Confirmed
// against these files' own dedicated tests before leaving them out of the list below:
// - ui/compose/RichTextEditor.kt (RichTextEditorTest) — the AnnotatedString<->RichTextContent
// editor-op functions (applyStyle/applyBlock/applyLink/toRichContent/toAnnotatedString/...).
// - ui/settings/AccountReorderList.kt (AccountReorderListTest) — commitDrag's reorder maths.
// - ui/reader/HtmlBody.kt (HtmlBodyTest, InlineImageResolverTest) — cidKey/resolveInlineImage/
// wrapHtml/toCssHex.
// - ui/reporting/ReportReviewScreen.kt (ReportReviewClipboardTest) — copyReportPayloadToClipboard.
// (ui/compose/format/FontRegistry.kt and ui/mailbox/FolderLabels.kt are plain logic files with no
// `@Composable` at all — never at risk — but sit right next to excluded files below.) For the same
// reason this list names each Screen/component file individually rather than a package-wide
// "**/ui/**": a blanket pattern can't carve the four files above back out, and would also reach
// every `*ViewModel*`.
val jacocoNonJvmTestableSurface = listOf(
// --- Compose UI render code: one glob per screen/component file (see the exceptions above) ---
// LibreMailApp KEPT excluded (#384, the acceptable exception): the composable is a real NavHost whose
// non-onboarding start destinations call hiltViewModel(), and standing the graph up needs owners a
// plain JVM compose rule can't surface — so graph-level nav stays on the instrumented OnboardingFlowTest.
// Its JVM-tractable parts (LibreMailBottomBar, StartupCrashPrompt, the cold-start hold guards) ARE
// exercised by LibreMailAppJvmTest, but the file's compiled facade (LibreMailAppKt) stays excluded.
"**/LibreMailApp*",
// AccountPickerScreen, AppPasswordSetupScreen & ManualSetupScreen converted to Robolectric JVM
// Compose tests (#378) — now JVM-covered.
// ComposeScreen (the email editor) converted to a Robolectric JVM Compose test (#382) — now
// JVM-covered.
// ColorSwatch(Row), FontPicker, FontSizePicker & ParagraphAlignmentControl converted to
// Robolectric JVM Compose tests (#376) — now JVM-covered.
// DraftsScreen, OutboxScreen & ProblemReportsScreen converted to Robolectric JVM Compose tests
// (#379) — now JVM-covered.
// LockScreen converted to a Robolectric JVM Compose test (#377) — now JVM-covered.
// AppLockGateHost converted to a Robolectric JVM Compose test (#384) — now JVM-covered.
// FolderDrawer & MailboxScreen (the Paging 3 mailbox list + folder drawer) converted to
// Robolectric JVM Compose tests (#383) — now JVM-covered.
// AddAnotherAccountScreen (#373) plus the onboarding welcome/license and contacts/battery steps
// (#377) converted to Robolectric JVM Compose tests — now JVM-covered.
// ReaderScreen converted to a Robolectric JVM Compose test (#381) — now JVM-covered. Its HTML body
// renders through HtmlBody, a hardened WebView that Robolectric can only present as a non-rendering
// shadow, so ReaderScreenJvmTest asserts the chrome (top bar, star/delete/reply actions, attachment
// accordion) and the loading/plain-text/empty/error/remote-images-banner branches — never the
// WebView's rendered HTML. HtmlBody.kt stays in scope covered by HtmlBodyTest/InlineImageResolverTest.
// SettingsScreen (+ ContactAutocompleteRow), AccountSettingsScreen, SettingsComponents (SwitchRow/
// ClickRow/RadioRow/RetentionSection), SignaturesScreen & SignatureEditScreen converted to
// Robolectric JVM Compose tests (#380) — now JVM-covered.
// CacheEncryptionGate.kt (issue #359/#367 fail-closed encryption gate) is pure render: the gate
// composable, its blank cover, the error screen, and the ephemeral report-review screen — no plain
// top-level logic. Spelled out to "...GateKt*" (the file's compiled facade class), NOT the bare
// "**/CacheEncryptionGate*" this list otherwise uses, because unlike every Screen/ViewModel pair
// above, CacheEncryptionGateViewModel's name literally starts with "CacheEncryptionGate" — a bare
// wildcard would also swallow the (94%-covered, dedicated-tested) ViewModel and its sealed
// CacheEncryptionGateState. CacheEncryptionGateViewModel and CacheEncryptionUnavailableException
// stay in scope (both have JVM tests: CacheEncryptionGateViewModelTest, DatabaseProvisionerTest).
"**/CacheEncryptionGateKt*",
// --- Android framework entry points (OS-instantiated). NB: Workers are intentionally NOT here
// --- (they are unit-tested — see the KEPT IN SCOPE note above).
"**/*Activity*",
"**/*Service*",
"**/LibreMailApplication*",
"**/*BackupAgent*",
// --- Hilt DI wiring (includes WorkManagerModule) ---
"**/di/**",
// --- src/debug cold-open probe (issue #221) ---
"**/data/local/coldopen/**",
// --- src/debug fetch-gate receiver (issue #393): a BroadcastReceiver that only runs on-device
// --- (adb-driven), covered by an instrumented test, never packaged into a release build. Its
// --- pure collaborators DebugFetchGate/FetchScope stay IN scope (unit-tested by DebugFetchGateTest).
"**/debug/FetchGateReceiver*",
)
// Classes = the debug variant's compiled Kotlin (AGP 9 built-in Kotlin output), with the generated
// code and the non-JVM-testable surface above stripped out. All hand-written code here is Kotlin, so
// the javac output (purely Hilt/Dagger/BuildConfig generated) is omitted. Hoisted to a shared val so
// the report and the verification gate always run against the identical denominator.
val jacocoDebugKotlinClasses = layout.buildDirectory.dir(
"intermediates/built_in_kotlinc/debug/compileDebugKotlin/classes",
)
val jacocoClassDirectories = fileTree(jacocoDebugKotlinClasses) {
exclude(jacocoGeneratedExcludes + jacocoNonJvmTestableSurface)
}
// Sources = hand-written main Kotlin.
val jacocoSourceDirectories = files("src/main/kotlin")
// Exec data written by the instrumented testDebugUnitTest task. Accept the base `jacoco` plugin's
// default location and AGP's enableUnitTestCoverage location so the wiring is robust either way.
val jacocoExecutionData = fileTree(layout.buildDirectory) {
include(
"jacoco/testDebugUnitTest.exec",
"outputs/unit_test_code_coverage/debugUnitTest/testDebugUnitTest.exec",
)
}
tasks.register<JacocoReport>("jacocoTestReport") {
// Ensure the unit tests (and thus their coverage exec data) have run first.
dependsOn("testDebugUnitTest")
group = "verification"
description = "Generates JaCoCo XML + HTML coverage for the debug JVM unit tests."
reports {
xml.required.set(true)
html.required.set(true)
}
classDirectories.setFrom(jacocoClassDirectories)
sourceDirectories.setFrom(jacocoSourceDirectories)
executionData.setFrom(jacocoExecutionData)
}
// No-regression coverage gate (closes #251; scoping from #290/#292). Fails `check` / CI when the
// overall LINE coverage of the scoped surface above drops below `jacocoLineCoverageFloor`. This is a
// FLOOR, not an absolute 95% target — the maintainer chose a ratchet over a fixed goal. The floor is
// normally set a hair (~0.5–1%) below the measured baseline so ordinary run-to-run noise doesn't
// red-flag it, while a real regression still fails the build. Manual ratchet FOR NOW: when coverage
// rises materially, bump this number up in the SAME PR so the floor tracks reality (there is no
// auto-ratchet yet).
//
// Re-ratcheted for #386 (final step of the Robolectric Compose epic #373, once infra/PoC #375 and
// conversion batches #376-384 had all landed and proven stable): new baseline 87.89% line
// (7994/9095), floor 0.84 — a wider ~3.9% headroom than the usual ~0.5-1%, chosen deliberately
// conservative for this first post-epic measurement; the maintainer can tighten it further in a
// follow-up PR. Prior baseline: 80.21% line (4838/6032), floor 0.79 (~1.2% headroom).
val jacocoLineCoverageFloor = "0.84"
tasks.register<JacocoCoverageVerification>("jacocoTestCoverageVerification") {
// Same inputs as jacocoTestReport (shared vals above) so the gate enforces exactly what the
// report shows. Depend on the unit tests so the exec data exists before verifying.
dependsOn("testDebugUnitTest")
group = "verification"
description = "Fails the build if scoped JVM unit-test LINE coverage regresses below the floor."
classDirectories.setFrom(jacocoClassDirectories)
sourceDirectories.setFrom(jacocoSourceDirectories)
executionData.setFrom(jacocoExecutionData)
violationRules {
rule {
element = "BUNDLE"
limit {
counter = "LINE"
value = "COVEREDRATIO"
minimum = jacocoLineCoverageFloor.toBigDecimal()
}
}
}
}
// Make the aggregate `check` lifecycle task enforce the no-regression floor locally too, so a
// coverage regression is caught by `./gradlew check` and not only in CI.
tasks.named("check") {
dependsOn("jacocoTestCoverageVerification")
}
// --- Robolectric android-all offline resolution (issue #373) ------------------------------------
// Robolectric runs the real Android framework on the JVM from a large `android-all-instrumented`
// jar. By default it resolves that jar LAZILY AT TEST TIME by downloading it from Maven Central
// (org.robolectric.internal.dependency.MavenDependencyResolver -> MavenArtifactFetcher). That
// runtime download is unreliable on CI runners and failed the JVM Compose PoC in CI with
// `java.lang.AssertionError at MavenArtifactFetcher ... Caused by: java.io.IOException` ("Failed to
// fetch maven artifact"). Fix: resolve the jar through Gradle instead — reliable, cached, and
// persisted by the CI Gradle cache, using the same repositories as every other dependency — then
// hand it to Robolectric in OFFLINE mode so it never touches the network at test time.
//
// A DEDICATED resolvable configuration (deliberately NOT testImplementation/testRuntimeOnly) keeps
// the ~200 MB instrumented framework jar OFF the JVM unit-test classpath: it must be loaded only by
// Robolectric's sandbox classloader, never flattened onto the app's test classpath where it would
// collide with the stub `android.jar`. `syncRobolectricAndroidAll` stages the resolved jar under
// its Maven filename (android-all-instrumented-<version>.jar) — exactly what Robolectric's
// LocalDependencyResolver looks up as <artifactId>-<version>.jar — and the two system properties
// below switch Robolectric onto that offline directory (see LegacyDependencyResolver). Every
// Robolectric test pins @Config(sdk = 36) (app/src/test/resources/robolectric.properties), so the
// single sdk=36 jar covers them all; a test on a different SDK must add that android-all version to
// this configuration too. The offline properties are inert for non-Robolectric JVM tests.
val robolectricAndroidAll: Configuration = configurations.create("robolectricAndroidAll") {
isCanBeConsumed = false
isCanBeResolved = true
}
val robolectricDepsDir = layout.buildDirectory.dir("robolectric-android-all")
val syncRobolectricAndroidAll = tasks.register<Sync>("syncRobolectricAndroidAll") {
description = "Stages Robolectric's android-all-instrumented jar for offline resolution (issue #373)."
from(robolectricAndroidAll)
into(robolectricDepsDir)
}
tasks.withType<Test>().configureEach {
dependsOn(syncRobolectricAndroidAll)
systemProperty("robolectric.offline", "true")
systemProperty("robolectric.dependency.dir", robolectricDepsDir.get().asFile.absolutePath)
}
dependencies {
implementation(libs.androidx.core.ktx)
implementation(libs.androidx.lifecycle.runtime.ktx)
@@ -224,6 +546,24 @@ dependencies {
// The real org.json for unit tests (android.jar ships a stubbed, no-op version).
testImplementation("org.json:json:20231013")
// Robolectric-backed JVM Compose UI tests (issue #373): Robolectric runs the Android framework
// on the JVM so `createComposeRule()` can drive composables without an emulator, bringing screen
// render code into the JaCoCo JVM-testable surface. The Compose test artifacts come from the same
// BOM as the app (aligned versions) and reuse the ui-test-junit4 / ui-test-manifest aliases the
// androidTest source set already declares — here in `test` (JVM), not `androidTest`. Robolectric
// sources Android's real org.json from its sandbox, so it does not clash with the stub-replacing
// org.json above (that is for the plain, non-Robolectric JVM tests).
testImplementation(libs.robolectric)
// The android-all-instrumented framework jar Robolectric loads into its sandbox — resolved via
// Gradle and staged for offline use by syncRobolectricAndroidAll above so no flaky test-time
// download happens in CI (issue #373). On its own dedicated configuration, NOT the test
// classpath — see that block for why. The artifact has no transitive dependencies (verified from
// its POM), so it resolves to exactly the one staged jar.
"robolectricAndroidAll"(libs.robolectric.android.all.instrumented)
testImplementation(platform(libs.androidx.compose.bom))
testImplementation(libs.androidx.compose.ui.test.junit4)
testImplementation(libs.androidx.compose.ui.test.manifest)
androidTestImplementation(libs.androidx.junit)
androidTestImplementation(libs.androidx.espresso.core)
androidTestImplementation(libs.androidx.espresso.intents)
@@ -231,4 +571,10 @@ dependencies {
androidTestImplementation(platform(libs.androidx.compose.bom))
androidTestImplementation(libs.androidx.compose.ui.test.junit4)
androidTestImplementation(libs.androidx.room.testing)
// Instrumented DatabaseProvisioner test: fakes the security/settings collaborators and spies the
// DatabaseEncryption object to regression-guard the SQLCipher native-lib load before a keyed open.
androidTestImplementation(libs.mockk.android)
// TestListenableWorkerBuilder for the instrumented PruneWorker/BackfillWorker cache-lock-deferral
// test (issue #226): builds a CoroutineWorker with its real (non-Hilt) constructor args on-device.
androidTestImplementation(libs.androidx.work.testing)
}
@@ -0,0 +1,241 @@
{
"formatVersion": 1,
"database": {
"version": 2,
"identityHash": "6705bfa56d02b55f47c5c66ba10143b0",
"entities": [
{
"tableName": "accounts",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `email` TEXT NOT NULL, `displayName` TEXT NOT NULL, `authType` TEXT NOT NULL, `sortOrder` INTEGER NOT NULL DEFAULT 0, `imap_host` TEXT NOT NULL, `imap_port` INTEGER NOT NULL, `imap_security` TEXT NOT NULL, `smtp_host` TEXT NOT NULL, `smtp_port` INTEGER NOT NULL, `smtp_security` TEXT NOT NULL, PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "email",
"columnName": "email",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "displayName",
"columnName": "displayName",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "authType",
"columnName": "authType",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "sortOrder",
"columnName": "sortOrder",
"affinity": "INTEGER",
"notNull": true,
"defaultValue": "0"
},
{
"fieldPath": "imap.host",
"columnName": "imap_host",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "imap.port",
"columnName": "imap_port",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "imap.security",
"columnName": "imap_security",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "smtp.host",
"columnName": "smtp_host",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "smtp.port",
"columnName": "smtp_port",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "smtp.security",
"columnName": "smtp_security",
"affinity": "TEXT",
"notNull": true
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
}
},
{
"tableName": "credentials",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`accountId` TEXT NOT NULL, `encryptedSecret` TEXT NOT NULL, PRIMARY KEY(`accountId`))",
"fields": [
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "encryptedSecret",
"columnName": "encryptedSecret",
"affinity": "TEXT",
"notNull": true
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"accountId"
]
}
},
{
"tableName": "account_settings",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`accountId` TEXT NOT NULL, `signature` TEXT NOT NULL, `signatureEnabled` INTEGER NOT NULL, `notificationsEnabled` INTEGER NOT NULL, `retentionCount` INTEGER, `retentionMonths` INTEGER, PRIMARY KEY(`accountId`), FOREIGN KEY(`accountId`) REFERENCES `accounts`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )",
"fields": [
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "signature",
"columnName": "signature",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "signatureEnabled",
"columnName": "signatureEnabled",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "notificationsEnabled",
"columnName": "notificationsEnabled",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "retentionCount",
"columnName": "retentionCount",
"affinity": "INTEGER"
},
{
"fieldPath": "retentionMonths",
"columnName": "retentionMonths",
"affinity": "INTEGER"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"accountId"
]
},
"foreignKeys": [
{
"table": "accounts",
"onDelete": "CASCADE",
"onUpdate": "NO ACTION",
"columns": [
"accountId"
],
"referencedColumns": [
"id"
]
}
]
},
{
"tableName": "signatures",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `accountId` TEXT NOT NULL, `name` TEXT NOT NULL, `contentHtml` TEXT NOT NULL, `isDefault` INTEGER NOT NULL, PRIMARY KEY(`id`), FOREIGN KEY(`accountId`) REFERENCES `accounts`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "name",
"columnName": "name",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "contentHtml",
"columnName": "contentHtml",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "isDefault",
"columnName": "isDefault",
"affinity": "INTEGER",
"notNull": true
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
},
"indices": [
{
"name": "index_signatures_accountId",
"unique": false,
"columnNames": [
"accountId"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_signatures_accountId` ON `${TABLE_NAME}` (`accountId`)"
}
],
"foreignKeys": [
{
"table": "accounts",
"onDelete": "CASCADE",
"onUpdate": "NO ACTION",
"columns": [
"accountId"
],
"referencedColumns": [
"id"
]
}
]
}
],
"setupQueries": [
"CREATE TABLE IF NOT EXISTS room_master_table (id INTEGER PRIMARY KEY,identity_hash TEXT)",
"INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, '6705bfa56d02b55f47c5c66ba10143b0')"
]
}
}
@@ -0,0 +1,495 @@
{
"formatVersion": 1,
"database": {
"version": 19,
"identityHash": "eb8510d29c6bd5fdb450ab835b6a770e",
"entities": [
{
"tableName": "messages",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `accountId` TEXT NOT NULL, `sender` TEXT NOT NULL, `senderEmail` TEXT NOT NULL, `subject` TEXT NOT NULL, `snippet` TEXT NOT NULL, `body` TEXT NOT NULL, `isHtml` INTEGER NOT NULL, `timestampMillis` INTEGER NOT NULL, `isRead` INTEGER NOT NULL, `isStarred` INTEGER NOT NULL, `folder` TEXT NOT NULL DEFAULT 'INBOX', `inInbox` INTEGER NOT NULL, `bodyFetched` INTEGER NOT NULL, `uid` INTEGER NOT NULL DEFAULT 0, `senderFold` TEXT NOT NULL DEFAULT '', `senderEmailFold` TEXT NOT NULL DEFAULT '', `subjectFold` TEXT NOT NULL DEFAULT '', `snippetFold` TEXT NOT NULL DEFAULT '', PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "sender",
"columnName": "sender",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "senderEmail",
"columnName": "senderEmail",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "subject",
"columnName": "subject",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "snippet",
"columnName": "snippet",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "body",
"columnName": "body",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "isHtml",
"columnName": "isHtml",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "timestampMillis",
"columnName": "timestampMillis",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "isRead",
"columnName": "isRead",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "isStarred",
"columnName": "isStarred",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "folder",
"columnName": "folder",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "'INBOX'"
},
{
"fieldPath": "inInbox",
"columnName": "inInbox",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "bodyFetched",
"columnName": "bodyFetched",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "uid",
"columnName": "uid",
"affinity": "INTEGER",
"notNull": true,
"defaultValue": "0"
},
{
"fieldPath": "senderFold",
"columnName": "senderFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "senderEmailFold",
"columnName": "senderEmailFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "subjectFold",
"columnName": "subjectFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "snippetFold",
"columnName": "snippetFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
},
"indices": [
{
"name": "index_messages_accountId",
"unique": false,
"columnNames": [
"accountId"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_messages_accountId` ON `${TABLE_NAME}` (`accountId`)"
},
{
"name": "index_messages_timestampMillis",
"unique": false,
"columnNames": [
"timestampMillis"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_messages_timestampMillis` ON `${TABLE_NAME}` (`timestampMillis`)"
},
{
"name": "index_messages_accountId_folder_uid",
"unique": false,
"columnNames": [
"accountId",
"folder",
"uid"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_messages_accountId_folder_uid` ON `${TABLE_NAME}` (`accountId`, `folder`, `uid`)"
}
]
},
{
"tableName": "attachments",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`messageId` TEXT NOT NULL, `partIndex` INTEGER NOT NULL, `filename` TEXT NOT NULL, `mimeType` TEXT NOT NULL, `sizeBytes` INTEGER NOT NULL, `contentId` TEXT, PRIMARY KEY(`messageId`, `partIndex`), FOREIGN KEY(`messageId`) REFERENCES `messages`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )",
"fields": [
{
"fieldPath": "messageId",
"columnName": "messageId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "partIndex",
"columnName": "partIndex",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "filename",
"columnName": "filename",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "mimeType",
"columnName": "mimeType",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "sizeBytes",
"columnName": "sizeBytes",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "contentId",
"columnName": "contentId",
"affinity": "TEXT"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"messageId",
"partIndex"
]
},
"indices": [
{
"name": "index_attachments_messageId",
"unique": false,
"columnNames": [
"messageId"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_attachments_messageId` ON `${TABLE_NAME}` (`messageId`)"
}
],
"foreignKeys": [
{
"table": "messages",
"onDelete": "CASCADE",
"onUpdate": "NO ACTION",
"columns": [
"messageId"
],
"referencedColumns": [
"id"
]
}
]
},
{
"tableName": "outbox",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `accountId` TEXT NOT NULL, `toAddresses` TEXT NOT NULL, `ccAddresses` TEXT NOT NULL, `bccAddresses` TEXT NOT NULL DEFAULT '', `subject` TEXT NOT NULL, `body` TEXT NOT NULL, `createdAt` INTEGER NOT NULL, `lastError` TEXT, `bodyHtml` TEXT, `attachments` TEXT NOT NULL DEFAULT '', PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "toAddresses",
"columnName": "toAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "ccAddresses",
"columnName": "ccAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "bccAddresses",
"columnName": "bccAddresses",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "subject",
"columnName": "subject",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "body",
"columnName": "body",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "createdAt",
"columnName": "createdAt",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "lastError",
"columnName": "lastError",
"affinity": "TEXT"
},
{
"fieldPath": "bodyHtml",
"columnName": "bodyHtml",
"affinity": "TEXT"
},
{
"fieldPath": "attachments",
"columnName": "attachments",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
}
},
{
"tableName": "drafts",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `accountId` TEXT, `toAddresses` TEXT NOT NULL, `ccAddresses` TEXT NOT NULL, `bccAddresses` TEXT NOT NULL DEFAULT '', `subject` TEXT NOT NULL, `body` TEXT NOT NULL, `updatedAt` INTEGER NOT NULL, `attachments` TEXT NOT NULL, `bodyHtml` TEXT, PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT"
},
{
"fieldPath": "toAddresses",
"columnName": "toAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "ccAddresses",
"columnName": "ccAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "bccAddresses",
"columnName": "bccAddresses",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "subject",
"columnName": "subject",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "body",
"columnName": "body",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "updatedAt",
"columnName": "updatedAt",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "attachments",
"columnName": "attachments",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "bodyHtml",
"columnName": "bodyHtml",
"affinity": "TEXT"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
}
},
{
"tableName": "folders",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`accountId` TEXT NOT NULL, `fullName` TEXT NOT NULL, `displayName` TEXT NOT NULL, `role` TEXT NOT NULL, `selectable` INTEGER NOT NULL, `sortOrder` INTEGER NOT NULL, `specialUse` INTEGER NOT NULL DEFAULT 0, `hierarchyDelimiter` TEXT, PRIMARY KEY(`accountId`, `fullName`))",
"fields": [
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "fullName",
"columnName": "fullName",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "displayName",
"columnName": "displayName",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "role",
"columnName": "role",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "selectable",
"columnName": "selectable",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "sortOrder",
"columnName": "sortOrder",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "specialUse",
"columnName": "specialUse",
"affinity": "INTEGER",
"notNull": true,
"defaultValue": "0"
},
{
"fieldPath": "hierarchyDelimiter",
"columnName": "hierarchyDelimiter",
"affinity": "TEXT"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"accountId",
"fullName"
]
}
},
{
"tableName": "backfill_progress",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`accountId` TEXT NOT NULL, `folder` TEXT NOT NULL, `nextBeforeUid` INTEGER NOT NULL, `complete` INTEGER NOT NULL, PRIMARY KEY(`accountId`, `folder`))",
"fields": [
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "folder",
"columnName": "folder",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "nextBeforeUid",
"columnName": "nextBeforeUid",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "complete",
"columnName": "complete",
"affinity": "INTEGER",
"notNull": true
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"accountId",
"folder"
]
}
}
],
"setupQueries": [
"CREATE TABLE IF NOT EXISTS room_master_table (id INTEGER PRIMARY KEY,identity_hash TEXT)",
"INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, 'eb8510d29c6bd5fdb450ab835b6a770e')"
]
}
}
@@ -0,0 +1,506 @@
{
"formatVersion": 1,
"database": {
"version": 20,
"identityHash": "8264768635364869a347064a0864df9c",
"entities": [
{
"tableName": "messages",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `accountId` TEXT NOT NULL, `sender` TEXT NOT NULL, `senderEmail` TEXT NOT NULL, `subject` TEXT NOT NULL, `snippet` TEXT NOT NULL, `body` TEXT NOT NULL, `isHtml` INTEGER NOT NULL, `timestampMillis` INTEGER NOT NULL, `isRead` INTEGER NOT NULL, `isStarred` INTEGER NOT NULL, `folder` TEXT NOT NULL DEFAULT 'INBOX', `inInbox` INTEGER NOT NULL, `bodyFetched` INTEGER NOT NULL, `uid` INTEGER NOT NULL DEFAULT 0, `senderFold` TEXT NOT NULL DEFAULT '', `senderEmailFold` TEXT NOT NULL DEFAULT '', `subjectFold` TEXT NOT NULL DEFAULT '', `snippetFold` TEXT NOT NULL DEFAULT '', PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "sender",
"columnName": "sender",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "senderEmail",
"columnName": "senderEmail",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "subject",
"columnName": "subject",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "snippet",
"columnName": "snippet",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "body",
"columnName": "body",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "isHtml",
"columnName": "isHtml",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "timestampMillis",
"columnName": "timestampMillis",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "isRead",
"columnName": "isRead",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "isStarred",
"columnName": "isStarred",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "folder",
"columnName": "folder",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "'INBOX'"
},
{
"fieldPath": "inInbox",
"columnName": "inInbox",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "bodyFetched",
"columnName": "bodyFetched",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "uid",
"columnName": "uid",
"affinity": "INTEGER",
"notNull": true,
"defaultValue": "0"
},
{
"fieldPath": "senderFold",
"columnName": "senderFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "senderEmailFold",
"columnName": "senderEmailFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "subjectFold",
"columnName": "subjectFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "snippetFold",
"columnName": "snippetFold",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
},
"indices": [
{
"name": "index_messages_accountId",
"unique": false,
"columnNames": [
"accountId"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_messages_accountId` ON `${TABLE_NAME}` (`accountId`)"
},
{
"name": "index_messages_timestampMillis",
"unique": false,
"columnNames": [
"timestampMillis"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_messages_timestampMillis` ON `${TABLE_NAME}` (`timestampMillis`)"
},
{
"name": "index_messages_accountId_folder_uid",
"unique": false,
"columnNames": [
"accountId",
"folder",
"uid"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_messages_accountId_folder_uid` ON `${TABLE_NAME}` (`accountId`, `folder`, `uid`)"
},
{
"name": "index_messages_folder_inInbox_timestampMillis",
"unique": false,
"columnNames": [
"folder",
"inInbox",
"timestampMillis"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_messages_folder_inInbox_timestampMillis` ON `${TABLE_NAME}` (`folder`, `inInbox`, `timestampMillis`)"
}
]
},
{
"tableName": "attachments",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`messageId` TEXT NOT NULL, `partIndex` INTEGER NOT NULL, `filename` TEXT NOT NULL, `mimeType` TEXT NOT NULL, `sizeBytes` INTEGER NOT NULL, `contentId` TEXT, PRIMARY KEY(`messageId`, `partIndex`), FOREIGN KEY(`messageId`) REFERENCES `messages`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE )",
"fields": [
{
"fieldPath": "messageId",
"columnName": "messageId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "partIndex",
"columnName": "partIndex",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "filename",
"columnName": "filename",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "mimeType",
"columnName": "mimeType",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "sizeBytes",
"columnName": "sizeBytes",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "contentId",
"columnName": "contentId",
"affinity": "TEXT"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"messageId",
"partIndex"
]
},
"indices": [
{
"name": "index_attachments_messageId",
"unique": false,
"columnNames": [
"messageId"
],
"orders": [],
"createSql": "CREATE INDEX IF NOT EXISTS `index_attachments_messageId` ON `${TABLE_NAME}` (`messageId`)"
}
],
"foreignKeys": [
{
"table": "messages",
"onDelete": "CASCADE",
"onUpdate": "NO ACTION",
"columns": [
"messageId"
],
"referencedColumns": [
"id"
]
}
]
},
{
"tableName": "outbox",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `accountId` TEXT NOT NULL, `toAddresses` TEXT NOT NULL, `ccAddresses` TEXT NOT NULL, `bccAddresses` TEXT NOT NULL DEFAULT '', `subject` TEXT NOT NULL, `body` TEXT NOT NULL, `createdAt` INTEGER NOT NULL, `lastError` TEXT, `bodyHtml` TEXT, `attachments` TEXT NOT NULL DEFAULT '', PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "toAddresses",
"columnName": "toAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "ccAddresses",
"columnName": "ccAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "bccAddresses",
"columnName": "bccAddresses",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "subject",
"columnName": "subject",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "body",
"columnName": "body",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "createdAt",
"columnName": "createdAt",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "lastError",
"columnName": "lastError",
"affinity": "TEXT"
},
{
"fieldPath": "bodyHtml",
"columnName": "bodyHtml",
"affinity": "TEXT"
},
{
"fieldPath": "attachments",
"columnName": "attachments",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
}
},
{
"tableName": "drafts",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`id` TEXT NOT NULL, `accountId` TEXT, `toAddresses` TEXT NOT NULL, `ccAddresses` TEXT NOT NULL, `bccAddresses` TEXT NOT NULL DEFAULT '', `subject` TEXT NOT NULL, `body` TEXT NOT NULL, `updatedAt` INTEGER NOT NULL, `attachments` TEXT NOT NULL, `bodyHtml` TEXT, PRIMARY KEY(`id`))",
"fields": [
{
"fieldPath": "id",
"columnName": "id",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT"
},
{
"fieldPath": "toAddresses",
"columnName": "toAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "ccAddresses",
"columnName": "ccAddresses",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "bccAddresses",
"columnName": "bccAddresses",
"affinity": "TEXT",
"notNull": true,
"defaultValue": "''"
},
{
"fieldPath": "subject",
"columnName": "subject",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "body",
"columnName": "body",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "updatedAt",
"columnName": "updatedAt",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "attachments",
"columnName": "attachments",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "bodyHtml",
"columnName": "bodyHtml",
"affinity": "TEXT"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"id"
]
}
},
{
"tableName": "folders",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`accountId` TEXT NOT NULL, `fullName` TEXT NOT NULL, `displayName` TEXT NOT NULL, `role` TEXT NOT NULL, `selectable` INTEGER NOT NULL, `sortOrder` INTEGER NOT NULL, `specialUse` INTEGER NOT NULL DEFAULT 0, `hierarchyDelimiter` TEXT, PRIMARY KEY(`accountId`, `fullName`))",
"fields": [
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "fullName",
"columnName": "fullName",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "displayName",
"columnName": "displayName",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "role",
"columnName": "role",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "selectable",
"columnName": "selectable",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "sortOrder",
"columnName": "sortOrder",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "specialUse",
"columnName": "specialUse",
"affinity": "INTEGER",
"notNull": true,
"defaultValue": "0"
},
{
"fieldPath": "hierarchyDelimiter",
"columnName": "hierarchyDelimiter",
"affinity": "TEXT"
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"accountId",
"fullName"
]
}
},
{
"tableName": "backfill_progress",
"createSql": "CREATE TABLE IF NOT EXISTS `${TABLE_NAME}` (`accountId` TEXT NOT NULL, `folder` TEXT NOT NULL, `nextBeforeUid` INTEGER NOT NULL, `complete` INTEGER NOT NULL, PRIMARY KEY(`accountId`, `folder`))",
"fields": [
{
"fieldPath": "accountId",
"columnName": "accountId",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "folder",
"columnName": "folder",
"affinity": "TEXT",
"notNull": true
},
{
"fieldPath": "nextBeforeUid",
"columnName": "nextBeforeUid",
"affinity": "INTEGER",
"notNull": true
},
{
"fieldPath": "complete",
"columnName": "complete",
"affinity": "INTEGER",
"notNull": true
}
],
"primaryKey": {
"autoGenerate": false,
"columnNames": [
"accountId",
"folder"
]
}
}
],
"setupQueries": [
"CREATE TABLE IF NOT EXISTS room_master_table (id INTEGER PRIMARY KEY,identity_hash TEXT)",
"INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, '8264768635364869a347064a0864df9c')"
]
}
}
@@ -0,0 +1,89 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.entity.AccountEntity
import org.libremail.data.local.entity.ServerConfigEmbedded
/**
* Real-SQLite behavior of [AccountDao] in the non-auth [AccountDatabase]: the email-ordered
* list/observe reads, point lookup, upsert-on-conflict, and deletion.
*/
@RunWith(AndroidJUnit4::class)
class AccountDaoTest {
private lateinit var db: AccountDatabase
private lateinit var dao: AccountDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
dao = db.accountDao()
}
@After
fun tearDown() = db.close()
private fun account(id: String, email: String, displayName: String = "Name") = AccountEntity(
id = id,
email = email,
displayName = displayName,
authType = "PASSWORD_IMAP",
imap = ServerConfigEmbedded("imap.example.org", 993, "SSL_TLS"),
smtp = ServerConfigEmbedded("smtp.example.org", 465, "SSL_TLS"),
)
@Test
fun observeAllAndGetAllReturnAccountsOrderedByEmail() = runBlocking {
dao.upsert(account("2", "zed@example.org"))
dao.upsert(account("1", "ada@example.org"))
assertEquals(listOf("ada@example.org", "zed@example.org"), dao.observeAll().first().map { it.email })
assertEquals(listOf("ada@example.org", "zed@example.org"), dao.getAll().map { it.email })
}
@Test
fun getByIdReturnsTheAccountOrNullAndEmbedsServerConfig() = runBlocking {
dao.upsert(account("acct", "ada@example.org"))
val stored = dao.getById("acct")
assertEquals("ada@example.org", stored?.email)
assertEquals(993, stored?.imap?.port)
assertEquals("smtp.example.org", stored?.smtp?.host)
assertNull(dao.getById("absent"))
}
@Test
fun upsertOnAConflictingIdUpdatesTheRowInPlace() = runBlocking {
// Non-destructive by design (issue #309): a second upsert of the same id refreshes the row
// rather than delete-then-reinserting it (which would cascade-delete settings/signatures).
dao.upsert(account("acct", "ada@example.org", displayName = "Ada"))
dao.upsert(account("acct", "ada@example.org", displayName = "Ada Lovelace"))
assertEquals("Ada Lovelace", dao.getById("acct")?.displayName)
assertEquals(1, dao.getAll().size)
}
@Test
fun deleteByIdRemovesTheAccount() = runBlocking {
dao.upsert(account("acct", "ada@example.org"))
dao.deleteById("acct")
assertNull(dao.getById("acct"))
}
}
@@ -13,6 +13,7 @@ import kotlinx.coroutines.runBlocking
import org.json.JSONObject
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
@@ -21,6 +22,8 @@ import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.entity.CredentialEntity
import org.libremail.reporting.AppLog
import org.libremail.reporting.RingLogBuffer
/**
* The one-time move performed by [AccountDataMigrator] (issue #111): copying accounts / credentials /
@@ -142,6 +145,26 @@ class AccountDataMigratorTest {
}
}
@Test
fun copyEmitsANonPiiAppLogBreadcrumbNamingOnlyTheMovedTables() = runBlocking<Unit> {
seedVersion14Cache()
val buffer = RingLogBuffer()
AppLog.install(buffer)
AccountDataMigrator.copyAccountTables(cacheFile, cachePassphrase = "", accountsFile = accountsFile)
val entry = buffer.snapshot()
.single { it.message.startsWith("moved account tables into the account database") }
assertEquals("the migration breadcrumb is a debug line", 'D', entry.level)
listOf("accounts", "credentials", "account_settings", "signatures").forEach { table ->
assertTrue("breadcrumb must name the moved table $table", entry.message.contains(table))
}
// The breadcrumb carries only table names — never the seeded email, secret, or passphrase.
assertFalse(entry.message.contains("ada@example.org"))
assertFalse(entry.message.contains("sealed-secret"))
assertFalse(entry.message.contains(passphrase))
}
@Test
fun reRunningTheCopyIsIdempotentAndKeepsLaterEdits() = runBlocking<Unit> {
seedVersion14Cache()
@@ -221,11 +244,38 @@ class AccountDataMigratorTest {
}
}
@Test
fun copiedAccountsGetAStableAlphabeticalInitialSortOrder() = runBlocking<Unit> {
// Insert three accounts in non-alphabetical order so the assertion proves the initial order is
// by email (the pre-#111 listing) rather than the copy's insertion order (issue #164). sortOrder
// is a destination-only column the v14 cache never had, so it is assigned entirely by the copy.
helper.createDatabase(cacheName, 14).apply {
listOf("c" to "carol@example.org", "a" to "ada@example.org", "b" to "bob@example.org")
.forEach { (id, email) ->
execSQL(
"INSERT INTO accounts (id, email, displayName, authType, imap_host, imap_port, " +
"imap_security, smtp_host, smtp_port, smtp_security) VALUES " +
"('$id', '$email', '$id', 'PASSWORD_IMAP', 'imap.example.org', 993, 'SSL_TLS', " +
"'smtp.example.org', 465, 'SSL_TLS')",
)
}
close()
}
AccountDataMigrator.copyAccountTables(cacheFile, cachePassphrase = "", accountsFile = accountsFile)
openAccountsDb().apply {
// getAll orders by sortOrder; the copy ranked by email, so ada(0) < bob(1) < carol(2).
assertEquals(listOf("a", "b", "c"), accountDao().getAll().map { it.id })
close()
}
}
@Test
fun migratorDdlMatchesExportedAccountDatabaseSchema() {
val schema = JSONObject(
InstrumentationRegistry.getInstrumentation().context.assets
.open("org.libremail.data.local.AccountDatabase/1.json")
.open("org.libremail.data.local.AccountDatabase/2.json")
.bufferedReader().use { it.readText() },
).getJSONObject("database")
val entities = schema.getJSONArray("entities")
@@ -39,9 +39,9 @@ class AccountDatabaseTest {
@After
fun tearDown() = db.close()
private fun account(id: String = "acct") = AccountEntity(
private fun account(id: String = "acct", email: String = "$id@example.org") = AccountEntity(
id = id,
email = "a@example.org",
email = email,
displayName = "A",
authType = "PASSWORD_IMAP",
imap = ServerConfigEmbedded("imap.example.org", 993, "SSL_TLS"),
@@ -69,6 +69,34 @@ class AccountDatabaseTest {
assertNull("account_settings must cascade-delete with its account", db.accountSettingsDao().get("acct"))
}
@Test
fun insertAtEndAppendsAccountsInIncreasingSortOrder() = runBlocking<Unit> {
db.accountDao().insertAtEnd(account("a"))
db.accountDao().insertAtEnd(account("b"))
db.accountDao().insertAtEnd(account("c"))
// Each new account lands at the end: current max + 1, starting from 0 (issue #164).
assertEquals(0, db.accountDao().getById("a")?.sortOrder)
assertEquals(1, db.accountDao().getById("b")?.sortOrder)
assertEquals(2, db.accountDao().getById("c")?.sortOrder)
// observeAll now orders by sortOrder, i.e. the append (insertion) order.
assertEquals(listOf("a", "b", "c"), db.accountDao().observeAll().first().map { it.id })
}
@Test
fun reorderPersistsAndBothQueriesReflectTheNewOrder() = runBlocking<Unit> {
listOf("a", "b", "c").forEach { db.accountDao().insertAtEnd(account(it)) }
db.accountDao().reorder(listOf("c", "a", "b"))
assertEquals(listOf("c", "a", "b"), db.accountDao().observeAll().first().map { it.id })
assertEquals(listOf("c", "a", "b"), db.accountDao().getAll().map { it.id })
// sortOrder is renumbered to the new positions, so the order survives an app restart.
assertEquals(0, db.accountDao().getById("c")?.sortOrder)
assertEquals(1, db.accountDao().getById("a")?.sortOrder)
assertEquals(2, db.accountDao().getById("b")?.sortOrder)
}
@Test
fun signaturesCascadeWithTheirAccount() = runBlocking<Unit> {
db.accountDao().upsert(account())
@@ -82,4 +110,40 @@ class AccountDatabaseTest {
db.signatureDao().observeForAccount("acct").first().isEmpty(),
)
}
@Test
fun upsertOnAnExistingIdUpdatesInPlaceWithoutCascadingSettingsOrSignatures() = runBlocking<Unit> {
// Regression for issue #309: an @Insert(REPLACE) upsert deletes-then-reinserts on an id
// conflict, firing the ON DELETE CASCADE that wipes the account's settings + signatures.
db.accountDao().upsert(account("acct"))
db.accountSettingsDao().upsert(AccountSettingsEntity("acct", signature = "Keep me"))
db.signatureDao().upsert(SignatureEntity("sig-1", "acct", "Work", "<p>Regards</p>", isDefault = true))
db.accountDao().upsert(account("acct").copy(displayName = "Updated"))
assertEquals("Updated", db.accountDao().getById("acct")?.displayName)
assertEquals("Keep me", db.accountSettingsDao().get("acct")?.signature)
assertEquals(1, db.signatureDao().observeForAccount("acct").first().size)
assertEquals(1, db.accountDao().getAll().size)
}
@Test
fun insertAtEndReAddingAnExistingAccountKeepsItsSettingsSignaturesAndPosition() = runBlocking<Unit> {
// The production re-add path (addOutlookAccount -> insertAtEnd -> upsert). Re-adding a
// deterministic id must not cascade-delete its settings/signatures nor move it to the end.
db.accountDao().insertAtEnd(account("a"))
db.accountDao().insertAtEnd(account("b"))
db.accountSettingsDao().upsert(AccountSettingsEntity("b", signature = "Sig B", signatureEnabled = false))
db.signatureDao().upsert(SignatureEntity("sig-b", "b", "Work", "<p>Regards</p>", isDefault = true))
db.accountDao().insertAtEnd(account("b").copy(displayName = "Renamed"))
// Updated in place, position preserved (still 1, not appended after), children intact.
assertEquals("Renamed", db.accountDao().getById("b")?.displayName)
assertEquals(1, db.accountDao().getById("b")?.sortOrder)
assertEquals("Sig B", db.accountSettingsDao().get("b")?.signature)
assertEquals(false, db.accountSettingsDao().get("b")?.signatureEnabled)
assertEquals(1, db.signatureDao().observeForAccount("b").first().size)
assertEquals(2, db.accountDao().getAll().size)
}
}
@@ -0,0 +1,77 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import androidx.room.testing.MigrationTestHelper
import androidx.sqlite.db.SupportSQLiteDatabase
import androidx.sqlite.db.framework.FrameworkSQLiteOpenHelperFactory
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
/**
* Replays [AccountDatabase]'s migrations against the schema JSONs exported to `app/schemas` (shipped
* to the test APK as assets). AccountDatabase is the non-auth account store split out of the cache in
* issue #111; like the cache database it registers no destructive fallback, so a drifted migration
* would crash upgrading users at first open — this test makes such drift fail in CI instead. Kept
* separate from [MigrationTest] because each database has its own version line and MigrationTestHelper
* binds to a single [androidx.room.RoomDatabase] class.
*/
@RunWith(AndroidJUnit4::class)
class AccountMigrationTest {
@get:Rule
val helper = MigrationTestHelper(
InstrumentationRegistry.getInstrumentation(),
AccountDatabase::class.java,
emptyList(),
FrameworkSQLiteOpenHelperFactory(),
)
/**
* v1 -> v2 (issue #164): `accounts.sortOrder` appears, and existing accounts are backfilled with a
* stable initial order matching the previous alphabetical (`ORDER BY email`) listing. The rows are
* inserted out of alphabetical order to prove the rank is by email, not insertion order.
*/
@Test
fun migrate1To2_addsSortOrderBackfilledByEmailRank() {
helper.createDatabase(TEST_DB, 1).apply {
insertAccount("c", "carol@example.org")
insertAccount("a", "ada@example.org")
insertAccount("b", "bob@example.org")
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 2, true, ACCOUNT_MIGRATION_1_2)
db.query("SELECT id, sortOrder FROM accounts ORDER BY sortOrder").use { c ->
assertTrue(c.moveToFirst())
assertEquals("ada ranks first alphabetically, so sortOrder 0", "a", c.getString(0))
assertEquals(0, c.getInt(1))
assertTrue(c.moveToNext())
assertEquals("b", c.getString(0))
assertEquals(1, c.getInt(1))
assertTrue(c.moveToNext())
assertEquals("c", c.getString(0))
assertEquals(2, c.getInt(1))
assertFalse("all three pre-upgrade accounts must survive, and nothing else", c.moveToNext())
}
db.close()
}
private fun SupportSQLiteDatabase.insertAccount(id: String, email: String) {
execSQL(
"INSERT INTO accounts (id, email, displayName, authType, imap_host, imap_port, imap_security, " +
"smtp_host, smtp_port, smtp_security) VALUES ('$id', '$email', '$id', 'PASSWORD_IMAP', " +
"'imap.example.org', 993, 'SSL_TLS', 'smtp.example.org', 465, 'SSL_TLS')",
)
}
private companion object {
const val TEST_DB = "account-migration-test.db"
}
}
@@ -0,0 +1,91 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.dao.AccountSettingsDao
import org.libremail.data.local.entity.AccountEntity
import org.libremail.data.local.entity.AccountSettingsEntity
import org.libremail.data.local.entity.ServerConfigEmbedded
/**
* Real-SQLite behavior of [AccountSettingsDao] in [AccountDatabase]: the live observer (which emits
* null before a row exists), the one-shot read, upsert-on-conflict, and the nullable retention
* overrides round-tripping. A parent account row is inserted first because the settings table
* foreign-keys to `accounts`.
*/
@RunWith(AndroidJUnit4::class)
class AccountSettingsDaoTest {
private lateinit var db: AccountDatabase
private lateinit var dao: AccountSettingsDao
private lateinit var accountDao: AccountDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
dao = db.accountSettingsDao()
accountDao = db.accountDao()
}
@After
fun tearDown() = db.close()
private suspend fun insertAccount(id: String = "acct") = accountDao.upsert(
AccountEntity(
id = id,
email = "$id@example.org",
displayName = "Name",
authType = "PASSWORD_IMAP",
imap = ServerConfigEmbedded("imap.example.org", 993, "SSL_TLS"),
smtp = ServerConfigEmbedded("smtp.example.org", 465, "SSL_TLS"),
),
)
@Test
fun observeEmitsNullBeforeARowExistsThenTheRow() = runBlocking {
insertAccount()
assertNull("no settings row yet -> the observer emits null", dao.observe("acct").first())
dao.upsert(AccountSettingsEntity("acct", signature = "Cheers"))
assertEquals("Cheers", dao.observe("acct").first()?.signature)
}
@Test
fun getReturnsTheStoredRowOrNull() = runBlocking {
insertAccount()
dao.upsert(AccountSettingsEntity("acct", signatureEnabled = false, notificationsEnabled = false))
val stored = dao.get("acct")
assertEquals(false, stored?.signatureEnabled)
assertEquals(false, stored?.notificationsEnabled)
assertNull(dao.get("absent"))
}
@Test
fun upsertReplacesTheSettingsAndRoundTripsNullableRetentionOverrides() = runBlocking {
insertAccount()
dao.upsert(AccountSettingsEntity("acct", retentionCount = null, retentionMonths = null))
assertNull(dao.get("acct")?.retentionCount)
assertNull(dao.get("acct")?.retentionMonths)
dao.upsert(AccountSettingsEntity("acct", retentionCount = 500, retentionMonths = 6))
val stored = dao.get("acct")
assertEquals(500, stored?.retentionCount)
assertEquals(6, stored?.retentionMonths)
}
}
@@ -0,0 +1,124 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.AttachmentDao
import org.libremail.data.local.dao.MessageDao
import org.libremail.data.local.entity.AttachmentEntity
import org.libremail.data.local.entity.MessageEntity
/**
* Real-SQLite behavior of [AttachmentDao]: the `partIndex` ordering, the inline-image filter shared
* with [LibreMailDatabaseTest], REPLACE-on-conflict inserts, and the delete/replace transaction. A
* parent message row is inserted first because the attachments table foreign-keys to `messages`
* (Room enables `PRAGMA foreign_keys = ON`).
*/
@RunWith(AndroidJUnit4::class)
class AttachmentDaoTest {
private lateinit var db: LibreMailDatabase
private lateinit var dao: AttachmentDao
private lateinit var messageDao: MessageDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, LibreMailDatabase::class.java).build()
dao = db.attachmentDao()
messageDao = db.messageDao()
}
@After
fun tearDown() = db.close()
private suspend fun insertParent(id: String) = messageDao.insertNew(
listOf(
MessageEntity(
id = id,
accountId = "acct",
sender = "Ada",
senderEmail = "ada@example.org",
subject = "Hi",
snippet = "",
body = "",
timestampMillis = 1_000L,
isRead = false,
isStarred = false,
),
),
)
@Test
fun getForMessageReturnsEveryPartOrderedByPartIndex() = runBlocking {
insertParent("m1")
dao.insert(
listOf(
AttachmentEntity("m1", 2, "third.pdf", "application/pdf", 30),
AttachmentEntity("m1", 0, "first.pdf", "application/pdf", 10),
AttachmentEntity("m1", 1, "second.png", "image/png", 20, contentId = "cid1"),
),
)
// getForMessage keeps inline (cid) parts and orders by partIndex ascending.
assertEquals(
listOf("first.pdf", "second.png", "third.pdf"),
dao.getForMessage("m1").map { it.filename },
)
}
@Test
fun insertReplacesAPartWithTheSamePrimaryKey() = runBlocking {
insertParent("m1")
dao.insert(listOf(AttachmentEntity("m1", 0, "old.pdf", "application/pdf", 10)))
// Same (messageId, partIndex) -> REPLACE overwrites the earlier row.
dao.insert(listOf(AttachmentEntity("m1", 0, "new.pdf", "application/pdf", 99)))
val parts = dao.getForMessage("m1")
assertEquals(1, parts.size)
assertEquals("new.pdf", parts.single().filename)
assertEquals(99L, parts.single().sizeBytes)
}
@Test
fun deleteForMessageRemovesOnlyThatMessagesParts() = runBlocking {
insertParent("m1")
insertParent("m2")
dao.insert(listOf(AttachmentEntity("m1", 0, "a.pdf", "application/pdf", 1)))
dao.insert(listOf(AttachmentEntity("m2", 0, "b.pdf", "application/pdf", 1)))
dao.deleteForMessage("m1")
assertTrue(dao.getForMessage("m1").isEmpty())
assertEquals(listOf("b.pdf"), dao.getForMessage("m2").map { it.filename })
}
@Test
fun replaceForMessageSwapsTheWholeAttachmentSetInOneTransaction() = runBlocking {
insertParent("m1")
dao.insert(
listOf(
AttachmentEntity("m1", 0, "old-a.pdf", "application/pdf", 1),
AttachmentEntity("m1", 1, "old-b.pdf", "application/pdf", 2),
),
)
dao.replaceForMessage(
"m1",
listOf(AttachmentEntity("m1", 0, "fresh.pdf", "application/pdf", 3)),
)
assertEquals(listOf("fresh.pdf"), dao.observeForMessage("m1").first().map { it.filename })
}
}
@@ -0,0 +1,84 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.BackfillProgressDao
import org.libremail.data.local.entity.BackfillProgressEntity
/**
* Real-SQLite behavior of [BackfillProgressDao] — the per-(account, folder) paging boundary the
* full-history backfill persists so it resumes after process death: point read, upsert-on-conflict,
* account-scoped clear, and the global reset used when the retention default changes.
*/
@RunWith(AndroidJUnit4::class)
class BackfillProgressDaoTest {
private lateinit var db: LibreMailDatabase
private lateinit var dao: BackfillProgressDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, LibreMailDatabase::class.java).build()
dao = db.backfillProgressDao()
}
@After
fun tearDown() = db.close()
@Test
fun getReturnsTheStoredBoundaryOrNull() = runBlocking {
dao.upsert(BackfillProgressEntity("acct", "INBOX", nextBeforeUid = 42, complete = false))
val stored = dao.get("acct", "INBOX")
assertEquals(42L, stored?.nextBeforeUid)
assertEquals(false, stored?.complete)
assertNull(dao.get("acct", "Archive"))
}
@Test
fun upsertReplacesTheBoundaryForTheSameAccountAndFolder() = runBlocking {
dao.upsert(BackfillProgressEntity("acct", "INBOX", nextBeforeUid = 100, complete = false))
// A later page lowers the boundary and can mark the folder complete.
dao.upsert(BackfillProgressEntity("acct", "INBOX", nextBeforeUid = 10, complete = true))
val stored = dao.get("acct", "INBOX")
assertEquals(10L, stored?.nextBeforeUid)
assertEquals(true, stored?.complete)
}
@Test
fun deleteForAccountClearsOnlyThatAccountsProgress() = runBlocking {
dao.upsert(BackfillProgressEntity("acct", "INBOX", nextBeforeUid = 1))
dao.upsert(BackfillProgressEntity("acct", "Archive", nextBeforeUid = 2))
dao.upsert(BackfillProgressEntity("acct2", "INBOX", nextBeforeUid = 3))
dao.deleteForAccount("acct")
assertNull(dao.get("acct", "INBOX"))
assertNull(dao.get("acct", "Archive"))
assertEquals(3L, dao.get("acct2", "INBOX")?.nextBeforeUid)
}
@Test
fun deleteAllClearsEveryAccountsProgress() = runBlocking {
dao.upsert(BackfillProgressEntity("acct", "INBOX", nextBeforeUid = 1))
dao.upsert(BackfillProgressEntity("acct2", "INBOX", nextBeforeUid = 2))
dao.deleteAll()
assertNull(dao.get("acct", "INBOX"))
assertNull(dao.get("acct2", "INBOX"))
}
}
@@ -0,0 +1,127 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import android.net.Uri
import android.os.Bundle
import androidx.core.os.bundleOf
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.coldopen.ColdOpenCacheProbe
import org.libremail.data.local.entity.MessageEntity
import java.io.File
/**
* Issue #221: a **process-isolated** cold open of a pre-encrypted cache — the SQLCipher regression fixed
* in 592a797. The crash only surfaced on a cold process opening an already-encrypted cache with nothing
* to convert, where the keyed open reached `SQLiteConnection.nativeOpen` with the native `.so` unloaded
* and threw `UnsatisfiedLinkError`. Every existing on-device test ([DatabaseEncryptionTest],
* [DatabaseProvisionerInstrumentedTest], `DatabaseModuleInstrumentedTest`, `AccountDataMigratorTest`)
* runs a conversion first, which loads the process-global library in-process — masking the bug exactly
* as production did before the fix.
*
* Process isolation is unavoidable here: `System.loadLibrary` is process-global, so once THIS
* (instrumentation) process mints the encrypted fixture, it can no longer observe a cold open. The
* fixture is therefore opened in a separate `:coldopen` app process hosted by [ColdOpenCacheProbe] (a
* debug-only [android.content.ContentProvider]); this test mints the fixture here — "a file created by a
* prior encrypted DB instance" — and drives the cold open there via `ContentResolver.call`, which spins
* that pristine process up on demand. The harness [ColdOpenCacheProbe.KEY_COLD_PROBE] check makes the
* isolation self-verifying: if the library were already loaded in the harness process, this test fails
* rather than passing a hollow assertion.
*/
@RunWith(AndroidJUnit4::class)
class ColdOpenEncryptedCacheTest {
private val appContext = ApplicationProvider.getApplicationContext<Context>()
private val dbName = "coldopen_encrypted_cache_test.db"
private val dbFile: File get() = appContext.getDatabasePath(dbName)
// 64 hex chars == a 32-byte SQLCipher passphrase, matching DatabaseKeyStore's format.
private val passphrase = "0123456789abcdef".repeat(4)
@Before
@After
fun clean() {
appContext.deleteDatabase(dbName)
dbFile.parentFile?.listFiles { f -> f.name.startsWith(dbName) }?.forEach { it.delete() }
}
@Test
fun coldProcessOpensAPreEncryptedCacheWithoutUnsatisfiedLinkError() {
// Mint a real SQLCipher-encrypted cache with one row HERE, in the instrumentation process. This
// loads the process-global .so in THIS process — precisely why the cold open must be observed in a
// different, pristine process.
seedEncryptedFixture()
assertTrue("precondition: the fixture is genuinely encrypted", DatabaseEncryption.isEncrypted(dbFile))
val result = callColdOpenProcess()
assertNotNull("the :coldopen process returned no result", result)
val probe = result?.getString(ColdOpenCacheProbe.KEY_COLD_PROBE)
val open = result?.getString(ColdOpenCacheProbe.KEY_OPEN)
// The harness proves its own process is genuinely cold: a keyed open with no preceding
// System.loadLibrary must fail at nativeOpen. Anything else means the .so was already loaded there
// and the "cold" open would be meaningless — so fail loudly instead of passing a hollow assertion.
assertEquals(
"the :coldopen process was not actually cold (SQLCipher already loaded); isolation broke — open=$open",
ColdOpenCacheProbe.PROBE_UNSATISFIED_LINK,
probe,
)
// The headline #221 guard: opening the pre-encrypted cache through the production wiring on a cold
// process succeeds (no UnsatisfiedLinkError) and reads the seeded row back.
assertEquals(
"cold open of the pre-encrypted cache failed (probe=$probe)",
ColdOpenCacheProbe.OPEN_OK,
open,
)
}
private fun callColdOpenProcess(): Bundle? {
val authority = appContext.packageName + ColdOpenCacheProbe.AUTHORITY_SUFFIX
return appContext.contentResolver.call(
Uri.parse("content://$authority"),
ColdOpenCacheProbe.METHOD_COLD_OPEN,
null,
bundleOf(
ColdOpenCacheProbe.KEY_DB_NAME to dbName,
ColdOpenCacheProbe.KEY_PASSPHRASE to passphrase,
),
)
}
/**
* Builds a plaintext cache with one row and converts it to real SQLCipher ciphertext — the steady-
* state shape an already-encrypted install presents at the next cold start, where `ensureEncrypted`
* has nothing left to convert (592a797's exact failure mode).
*/
private fun seedEncryptedFixture() {
Room.databaseBuilder(appContext, LibreMailDatabase::class.java, dbName).build().apply {
runBlocking { messageDao().insertNew(listOf(message(ColdOpenCacheProbe.EXPECTED_ROW_ID))) }
close()
}
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
}
private fun message(id: String) = MessageEntity(
id = id,
accountId = "acct",
sender = "Ada",
senderEmail = "ada@example.org",
subject = "Hi",
snippet = "",
body = "",
timestampMillis = 1_000L,
isRead = false,
isStarred = false,
)
}
@@ -0,0 +1,64 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.CredentialDao
import org.libremail.data.local.entity.CredentialEntity
/**
* Real-SQLite behavior of [CredentialDao] in [AccountDatabase]: the point read, upsert-on-conflict
* (a rotated secret overwrites the old one), and deletion. The credentials table is not foreign-keyed
* to `accounts`, so these rows stand alone.
*/
@RunWith(AndroidJUnit4::class)
class CredentialDaoTest {
private lateinit var db: AccountDatabase
private lateinit var dao: CredentialDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
dao = db.credentialDao()
}
@After
fun tearDown() = db.close()
@Test
fun getByIdReturnsTheSecretOrNull() = runBlocking {
dao.upsert(CredentialEntity("acct", "sealed-secret"))
assertEquals("sealed-secret", dao.getById("acct")?.encryptedSecret)
assertNull(dao.getById("absent"))
}
@Test
fun upsertReplacesTheSecretForTheSameAccount() = runBlocking {
dao.upsert(CredentialEntity("acct", "old"))
dao.upsert(CredentialEntity("acct", "rotated"))
assertEquals("rotated", dao.getById("acct")?.encryptedSecret)
}
@Test
fun deleteByIdRemovesTheCredential() = runBlocking {
dao.upsert(CredentialEntity("acct", "sealed-secret"))
dao.deleteById("acct")
assertNull(dao.getById("acct"))
}
}
@@ -7,6 +7,7 @@ import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import net.zetetic.database.sqlcipher.SQLiteDatabase
import net.zetetic.database.sqlcipher.SupportOpenHelperFactory
import org.junit.After
import org.junit.Assert.assertEquals
@@ -16,6 +17,8 @@ import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.entity.MessageEntity
import org.libremail.reporting.AppLog
import org.libremail.reporting.RingLogBuffer
import java.io.File
/**
@@ -66,6 +69,125 @@ class DatabaseEncryptionTest {
}
}
@Test
fun isEncryptedIsFalseForMissingEmptyAndPlaintextFiles() = runBlocking<Unit> {
val missing = File(dbFile.parentFile, "$dbName.missing")
assertFalse("a non-existent file is not encrypted", DatabaseEncryption.isEncrypted(missing))
val empty = File(dbFile.parentFile, "$dbName.empty")
empty.delete()
empty.createNewFile()
assertFalse("a zero-length file is not encrypted", DatabaseEncryption.isEncrypted(empty))
openPlaintext().apply {
messageDao().insertNew(listOf(message("acct:1")))
close()
}
assertFalse("a plaintext SQLite file is not encrypted", DatabaseEncryption.isEncrypted(dbFile))
}
@Test
fun ensureEncryptedNoOpsOnMissingEmptyOrAlreadyEncryptedFiles() = runBlocking<Unit> {
// Missing / empty: nothing to convert (the factory creates a fresh DB encrypted).
val empty = File(dbFile.parentFile, "$dbName.empty")
empty.createNewFile()
DatabaseEncryption.ensureEncrypted(File(dbFile.parentFile, "$dbName.missing"), passphrase)
DatabaseEncryption.ensureEncrypted(empty, passphrase)
openPlaintext().apply {
messageDao().insertNew(listOf(message("acct:1")))
close()
}
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
assertTrue(DatabaseEncryption.isEncrypted(dbFile))
// A second call on an already-encrypted file is an idempotent no-op; the data stays readable.
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
assertTrue("the file stays encrypted", DatabaseEncryption.isEncrypted(dbFile))
openEncrypted().apply {
assertEquals(listOf("acct:1"), messageDao().observeSummaries().first().map { it.id })
close()
}
}
@Test
fun ensurePlaintextNoOpsOnAnAlreadyPlaintextFile() = runBlocking<Unit> {
openPlaintext().apply {
messageDao().insertNew(listOf(message("acct:1")))
close()
}
// Already plaintext: decrypt must be a no-op (and must not corrupt the file).
DatabaseEncryption.ensurePlaintext(dbFile, passphrase)
assertFalse(DatabaseEncryption.isEncrypted(dbFile))
openPlaintext().apply {
assertEquals(listOf("acct:1"), messageDao().observeSummaries().first().map { it.id })
close()
}
}
@Test
fun schemaVersionIsCarriedOntoTheEncryptedFile() = runBlocking<Unit> {
// A first Room open stamps PRAGMA user_version to the schema version; the conversion must carry
// it across (sqlcipher_export copies tables but not that pragma), or Room would attempt a bogus
// migration on the re-keyed file.
openPlaintext().apply {
messageDao().insertNew(listOf(message("acct:1")))
close()
}
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
val encrypted = SQLiteDatabase.openOrCreateDatabase(
dbFile.absolutePath,
passphrase.toByteArray(Charsets.US_ASCII),
null,
null,
)
val version = try {
encrypted.version
} finally {
encrypted.close()
}
assertEquals("Room's schema version must survive the plaintext -> encrypted conversion", 20, version)
}
@Test
fun conversionEmitsNonPiiAppLogBreadcrumbs() = runBlocking<Unit> {
val buffer = RingLogBuffer()
AppLog.install(buffer)
// The seeded row carries an email address so the PII assertions below are meaningful.
openPlaintext().apply {
messageDao().insertNew(listOf(message("acct:1")))
close()
}
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
val afterEncrypt = buffer.snapshot()
val converting = afterEncrypt.single { it.message.startsWith("converting local cache database") }
assertEquals("the start breadcrumb is informational", 'I', converting.level)
assertEquals("converting local cache database (targetEncrypted=true)", converting.message)
val convertedAfterEncrypt = afterEncrypt.single { it.message == "local cache database converted" }
assertEquals('D', convertedAfterEncrypt.level)
// Converting back to plaintext logs the same pair with the flag flipped.
buffer.clear()
DatabaseEncryption.ensurePlaintext(dbFile, passphrase)
val afterDecrypt = buffer.snapshot()
assertTrue(
afterDecrypt.any { it.message == "converting local cache database (targetEncrypted=false)" },
)
assertTrue(afterDecrypt.any { it.message == "local cache database converted" })
// Neither conversion's breadcrumbs may leak the passphrase, the on-disk path, or account PII.
(afterEncrypt + afterDecrypt).forEach { entry ->
assertFalse("must not leak the passphrase", entry.message.contains(passphrase))
assertFalse("must not leak the db file path", entry.message.contains(dbFile.absolutePath))
assertFalse("must not leak the seeded email", entry.message.contains("ada@example.org"))
}
}
private fun openPlaintext(): LibreMailDatabase =
Room.databaseBuilder(context, LibreMailDatabase::class.java, dbName).build()
@@ -0,0 +1,205 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import android.content.ContextWrapper
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import io.mockk.Runs
import io.mockk.coEvery
import io.mockk.every
import io.mockk.just
import io.mockk.mockk
import io.mockk.mockkObject
import io.mockk.unmockkAll
import io.mockk.unmockkObject
import io.mockk.verify
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.runBlocking
import net.zetetic.database.sqlcipher.SupportOpenHelperFactory
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.entity.MessageEntity
import org.libremail.data.security.DatabaseKeyStore
import org.libremail.data.settings.AppSettings
import org.libremail.data.settings.SettingsRepository
import java.io.File
/**
* On-device behavior of [DatabaseProvisioner.prepareCache] against REAL SQLCipher (the JVM
* `DatabaseProvisionerTest` mocks [DatabaseEncryption], so it can't exercise a real keyed open). The
* security/settings collaborators are faked; [DatabaseEncryption] is a spy so its real conversions run
* while the native-lib load can still be verified.
*
* The headline is the steady-state regression guard for the crash fixed in 592a797: with the cache
* already encrypted (nothing to convert) `ensureEncrypted` no-ops, so the provisioner itself must load
* SQLCipher's native library before the keyed open — otherwise Room's `nativeOpen` throws
* `UnsatisfiedLinkError` on every cold start. The `verify(exactly = 1) { ensureNativeLibraryLoaded() }`
* fails if that explicit load is ever removed.
*/
@RunWith(AndroidJUnit4::class)
class DatabaseProvisionerInstrumentedTest {
private val appContext = ApplicationProvider.getApplicationContext<Context>()
private val dbName = "provisioner_instrumented_test.db"
private val dbFile: File get() = appContext.getDatabasePath(dbName)
// 64 hex chars == a 32-byte SQLCipher passphrase.
private val passphrase = "0123456789abcdef".repeat(4)
private val keyStore = mockk<DatabaseKeyStore>()
private val settingsRepository = mockk<SettingsRepository>()
private val migrator = mockk<AccountDataMigrator>()
// A real ContextWrapper, NOT a mockk<Context>: mocking android.content.Context makes MockK walk the
// whole framework class with kotlin-reflect (isKotlinInline), which trips an ART parameter-annotation
// length mismatch and throws ArrayIndexOutOfBoundsException on API 31/32 (it passes on API 29). The
// wrapper routes the provisioner's cache lookup to the test DB and delegates everything else.
private val context: Context = object : ContextWrapper(appContext) {
override fun getDatabasePath(name: String): File =
if (name == DatabaseFiles.NAME) dbFile else super.getDatabasePath(name)
}
@Before
fun setUp() {
clean()
coEvery { keyStore.isClearPending() } returns false
coEvery { keyStore.resolvePassphrase(any()) } returns passphrase
coEvery { migrator.migrateIfNeeded() } just Runs
}
@After
fun tearDown() {
unmockkAll()
clean()
}
private fun clean() {
appContext.deleteDatabase(dbName)
dbFile.parentFile?.listFiles { f -> f.name.startsWith(dbName) }?.forEach { it.delete() }
}
private fun provisioner() = DatabaseProvisioner(context, keyStore, settingsRepository, migrator, Dispatchers.IO)
private fun seedPlaintextRow() {
Room.databaseBuilder(appContext, LibreMailDatabase::class.java, dbName).build().apply {
runBlocking { messageDao().insertNew(listOf(message("acct:1"))) }
close()
}
}
private fun openEncrypted(): LibreMailDatabase =
Room.databaseBuilder(appContext, LibreMailDatabase::class.java, dbName)
.openHelperFactory(SupportOpenHelperFactory(passphrase.toByteArray(Charsets.US_ASCII), null, false))
.build()
private fun openPlaintext(): LibreMailDatabase =
Room.databaseBuilder(appContext, LibreMailDatabase::class.java, dbName).build()
private fun message(id: String) = MessageEntity(
id = id,
accountId = "acct",
sender = "Ada",
senderEmail = "ada@example.org",
subject = "Hi",
snippet = "",
body = "",
timestampMillis = 1_000L,
isRead = false,
isStarred = false,
)
@Test
fun steadyStateEncryptedStartLoadsTheNativeLibAndOpensKeyedWithoutCrashing() = runBlocking<Unit> {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = true, appLock = false))
// Build a genuinely-encrypted, steady-state cache. Encrypt BEFORE spying so the conversion is a
// real one; by the time prepareCache runs, ensureEncrypted has nothing left to do.
seedPlaintextRow()
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
assertTrue("precondition: the cache is already encrypted", DatabaseEncryption.isEncrypted(dbFile))
mockkObject(DatabaseEncryption) // spy: real implementations still run
val mode = provisioner().prepareCache()
assertEquals(CacheOpenMode.Encrypted(passphrase), mode)
// Regression guard (592a797): a steady-state encrypted start converts nothing, so the provisioner
// MUST load the native library itself before the keyed open below. Fails if that load is removed.
verify(exactly = 1) { DatabaseEncryption.ensureNativeLibraryLoaded() }
unmockkObject(DatabaseEncryption)
// The keyed open the provisioner reported must actually succeed on real SQLCipher (no crash).
openEncrypted().apply {
assertEquals(listOf("acct:1"), messageDao().observeSummaries().first().map { it.id })
close()
}
}
/**
* Issue #359: with `encryptCache` on and NO cache yet (a fresh install enabling encryption), the
* provisioner must load SQLCipher's native library, report [CacheOpenMode.Encrypted], and a real keyed
* open must then create and read the encrypted cache — i.e. `libsqlcipher.so` actually loads and runs.
*
* On a 16 KB memory-page device/image (Android 15+, and the CI API-37 preview `google_apis_ps16k`
* E2E image) an `.so` not aligned for 16 KB pages fails exactly here with `UnsatisfiedLinkError` at
* `SQLiteConnection.nativeOpen`. Running this on that image makes the 16 KB native-lib load a tested
* invariant, so a dependency bump that regressed alignment is caught in CI rather than on-device.
*/
@Test
fun freshEncryptOnStartLoadsThe16KbNativeLibAndOpensKeyedWithoutCrashing() = runBlocking<Unit> {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = true, appLock = false))
assertFalse("precondition: no cache file exists yet", dbFile.exists())
val mode = provisioner().prepareCache()
assertEquals(CacheOpenMode.Encrypted(passphrase), mode)
// The keyed open must actually succeed on real SQLCipher — loading and using libsqlcipher.so on
// whatever ABI / page size this device or emulator image uses.
openEncrypted().apply {
messageDao().insertNew(listOf(message("acct:1")))
assertEquals(listOf("acct:1"), messageDao().observeSummaries().first().map { it.id })
close()
}
assertTrue("the fresh cache was created in SQLCipher (encrypted) form", DatabaseEncryption.isEncrypted(dbFile))
}
@Test
fun encryptionTurnedOffDecryptsAnEncryptedCacheToPlaintext() = runBlocking<Unit> {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = false, appLock = false))
seedPlaintextRow()
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
assertTrue("precondition: the cache starts encrypted", DatabaseEncryption.isEncrypted(dbFile))
val mode = provisioner().prepareCache()
assertEquals(CacheOpenMode.Plaintext, mode)
assertFalse("the cache must be decrypted so the unkeyed open works", DatabaseEncryption.isEncrypted(dbFile))
openPlaintext().apply {
assertEquals(listOf("acct:1"), messageDao().observeSummaries().first().map { it.id })
close()
}
}
@Test
fun plaintextStartLeavesThePlaintextCacheUntouched() = runBlocking<Unit> {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = false, appLock = false))
seedPlaintextRow()
assertFalse("precondition: the cache is plaintext", DatabaseEncryption.isEncrypted(dbFile))
val mode = provisioner().prepareCache()
assertEquals(CacheOpenMode.Plaintext, mode)
assertFalse("a plaintext-with-encryption-off start converts nothing", DatabaseEncryption.isEncrypted(dbFile))
openPlaintext().apply {
assertEquals(listOf("acct:1"), messageDao().observeSummaries().first().map { it.id })
close()
}
}
}
@@ -0,0 +1,94 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.sqlite.db.SupportSQLiteDatabase
import androidx.sqlite.db.SupportSQLiteOpenHelper
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import io.mockk.mockk
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertSame
import org.junit.Test
import org.junit.runner.RunWith
/**
* Behavior of [DeferredOpenHelperFactory] (issue #93): Room touches `create` and
* `setWriteAheadLoggingEnabled` on the injection path (possibly the main thread), so the REAL open
* helper — and the blocking startup gate that builds it — must be deferred until the database is first
* actually opened. These tests pin that laziness, the WAL memoization, and the "never open just to
* close" contract using a recording fake delegate (no real database needed).
*/
@RunWith(AndroidJUnit4::class)
class DeferredOpenHelperFactoryTest {
private val context = ApplicationProvider.getApplicationContext<Context>()
private class FakeOpenHelper(private val db: SupportSQLiteDatabase) : SupportSQLiteOpenHelper {
var walEnabled: Boolean? = null
var closed = false
override val databaseName: String = "fake"
override fun setWriteAheadLoggingEnabled(enabled: Boolean) {
walEnabled = enabled
}
override val writableDatabase: SupportSQLiteDatabase get() = db
override val readableDatabase: SupportSQLiteDatabase get() = db
override fun close() {
closed = true
}
}
private fun configuration(name: String): SupportSQLiteOpenHelper.Configuration =
SupportSQLiteOpenHelper.Configuration.builder(context)
.name(name)
.callback(object : SupportSQLiteOpenHelper.Callback(1) {
override fun onCreate(db: SupportSQLiteDatabase) = Unit
override fun onUpgrade(db: SupportSQLiteDatabase, oldVersion: Int, newVersion: Int) = Unit
})
.build()
@Test
fun theDelegateIsBuiltLazilyOnFirstOpenAndThenMemoized() {
var builds = 0
val db = mockk<SupportSQLiteDatabase>(relaxed = true)
val fake = FakeOpenHelper(db)
val helper = DeferredOpenHelperFactory {
builds++
fake
}.create(configuration("deferred-lazy"))
// create() must not build the real delegate (it runs on the possibly-main injection thread).
assertEquals("create() must not build the delegate", 0, builds)
assertEquals("deferred-lazy", helper.databaseName)
// WAL can be set before the first open; it must be remembered, not force an early build.
helper.setWriteAheadLoggingEnabled(true)
assertEquals("setWriteAheadLoggingEnabled must not build the delegate", 0, builds)
// The first open builds the delegate and applies the remembered WAL setting.
assertSame(db, helper.writableDatabase)
assertEquals(1, builds)
assertEquals("the remembered WAL flag is applied when the delegate is built", true, fake.walEnabled)
// Subsequent opens reuse the same delegate.
assertSame(db, helper.readableDatabase)
assertEquals("the delegate is memoized after the first open", 1, builds)
}
@Test
fun closeBeforeAnyOpenNeverBuildsTheDelegate() {
var builds = 0
val fake = FakeOpenHelper(mockk(relaxed = true))
val helper = DeferredOpenHelperFactory {
builds++
fake
}.create(configuration("deferred-close"))
helper.close()
assertEquals("close() must not build a delegate just to close it", 0, builds)
assertFalse("a never-built delegate is never closed", fake.closed)
}
}
@@ -0,0 +1,89 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.DraftDao
import org.libremail.data.local.entity.DraftEntity
/**
* Real-SQLite behavior of [DraftDao] — the saved-but-unsent store the compose autosave writes:
* newest-first observation by `updatedAt`, the live count, point/bulk reads, upsert-on-conflict, and
* deletion.
*/
@RunWith(AndroidJUnit4::class)
class DraftDaoTest {
private lateinit var db: LibreMailDatabase
private lateinit var dao: DraftDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, LibreMailDatabase::class.java).build()
dao = db.draftDao()
}
@After
fun tearDown() = db.close()
private fun draft(id: String, updatedAt: Long = 1_000L, subject: String = "Draft $id") = DraftEntity(
id = id,
accountId = "acct",
toAddresses = "bob@example.org",
ccAddresses = "",
subject = subject,
body = "Body",
updatedAt = updatedAt,
)
@Test
fun observeAllReturnsDraftsNewestFirstAndObserveCountTracksThem() = runBlocking {
dao.upsert(draft("old", updatedAt = 100))
dao.upsert(draft("new", updatedAt = 300))
dao.upsert(draft("mid", updatedAt = 200))
assertEquals(listOf("new", "mid", "old"), dao.observeAll().first().map { it.id })
assertEquals(3, dao.observeCount().first())
}
@Test
fun getByIdAndGetAllReadStoredDrafts() = runBlocking {
dao.upsert(draft("d1"))
dao.upsert(draft("d2"))
assertEquals("Draft d1", dao.getById("d1")?.subject)
assertNull(dao.getById("absent"))
assertEquals(setOf("d1", "d2"), dao.getAll().map { it.id }.toSet())
}
@Test
fun upsertReplacesADraftWithTheSameId() = runBlocking {
dao.upsert(draft("d1", subject = "First"))
dao.upsert(draft("d1", subject = "Edited"))
assertEquals("Edited", dao.getById("d1")?.subject)
assertEquals(1, dao.observeCount().first())
}
@Test
fun deleteRemovesTheDraft() = runBlocking {
dao.upsert(draft("d1"))
dao.upsert(draft("d2"))
dao.delete("d1")
assertEquals(listOf("d2"), dao.getAll().map { it.id })
}
}
@@ -0,0 +1,78 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.FolderDao
import org.libremail.data.local.entity.FolderEntity
/**
* Real-SQLite behavior of [FolderDao] not already covered by [LibreMailDatabaseTest] (which pins
* `observeForAccount` ordering + `replaceForAccount`): the one-shot [FolderDao.getForAccountOnce]
* read, REPLACE-on-conflict for a re-listed folder, and the account scoping of a delete.
*/
@RunWith(AndroidJUnit4::class)
class FolderDaoTest {
private lateinit var db: LibreMailDatabase
private lateinit var dao: FolderDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, LibreMailDatabase::class.java).build()
dao = db.folderDao()
}
@After
fun tearDown() = db.close()
@Test
fun getForAccountOnceReturnsFoldersOrderedBySortOrder() = runBlocking {
dao.insertAll(
listOf(
FolderEntity("acct", "Archive", "Archive", "ARCHIVE", selectable = true, sortOrder = 2),
FolderEntity("acct", "INBOX", "INBOX", "INBOX", selectable = true, sortOrder = 0),
FolderEntity("acct", "Sent", "Sent", "SENT", selectable = true, sortOrder = 1),
),
)
assertEquals(listOf("INBOX", "Sent", "Archive"), dao.getForAccountOnce("acct").map { it.fullName })
}
@Test
fun insertAllReplacesAFolderWithTheSamePrimaryKey() = runBlocking {
dao.insertAll(
listOf(FolderEntity("acct", "INBOX", "Old label", "INBOX", selectable = true, sortOrder = 0)),
)
// A re-list of INBOX (same accountId + fullName) replaces the cached row.
dao.insertAll(
listOf(FolderEntity("acct", "INBOX", "New label", "INBOX", selectable = true, sortOrder = 0)),
)
val folders = dao.getForAccountOnce("acct")
assertEquals(1, folders.size)
assertEquals("New label", folders.single().displayName)
}
@Test
fun deleteForAccountRemovesOnlyThatAccountsFolders() = runBlocking {
dao.insertAll(listOf(FolderEntity("acct", "INBOX", "INBOX", "INBOX", selectable = true, sortOrder = 0)))
dao.insertAll(listOf(FolderEntity("acct2", "INBOX", "INBOX", "INBOX", selectable = true, sortOrder = 0)))
dao.deleteForAccount("acct")
assertTrue(dao.getForAccountOnce("acct").isEmpty())
assertEquals(listOf("INBOX"), dao.getForAccountOnce("acct2").map { it.fullName })
}
}
@@ -0,0 +1,405 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.paging.PagingSource
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.MessageDao
import org.libremail.data.local.entity.MessageEntity
import org.libremail.data.local.entity.MessageSummary
/**
* Real-SQLite behavior of the [MessageDao] query and mutation surface not already pinned by
* [MessageDaoRetentionTest] (the retention/backfill boundary probes) or [MessageDaoRoutingTest] (the
* body-less routing projection): the Paging 3 browse/search sources, the flag/body/header mutators,
* and the scoped deletes. Exercised against a real in-memory database so the generated SQL — and its
* `inInbox`/folder/account scoping and ordering — runs for real.
*/
@RunWith(AndroidJUnit4::class)
class MessageDaoTest {
private lateinit var db: LibreMailDatabase
private lateinit var dao: MessageDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, LibreMailDatabase::class.java).build()
dao = db.messageDao()
}
@After
fun tearDown() = db.close()
@Suppress("LongParameterList")
private fun message(
id: String,
accountId: String = "acct",
folder: String = "INBOX",
subject: String = "Hi",
sender: String = "Ada",
senderEmail: String = "ada@example.org",
snippet: String = "",
body: String = "",
timestampMillis: Long = 1_000L,
isRead: Boolean = false,
isStarred: Boolean = false,
inInbox: Boolean = true,
bodyFetched: Boolean = false,
isHtml: Boolean = false,
uid: Long = 0L,
) = MessageEntity(
id = id,
accountId = accountId,
sender = sender,
senderEmail = senderEmail,
subject = subject,
snippet = snippet,
body = body,
isHtml = isHtml,
timestampMillis = timestampMillis,
isRead = isRead,
isStarred = isStarred,
folder = folder,
inInbox = inInbox,
bodyFetched = bodyFetched,
uid = uid,
// Mirror production's Unicode-casefold population (issue #232): Mappers.toEntity + MessageDao's
// updateHeaderContent/updateBody write `lowercase()` copies of the searchable fields, and the
// *SearchSummaries queries match against these `*Fold` columns — so fixtures must set them too.
senderFold = sender.lowercase(),
senderEmailFold = senderEmail.lowercase(),
subjectFold = subject.lowercase(),
snippetFold = snippet.lowercase(),
)
/** Refreshes a [PagingSource] and returns the first loaded page's ids in order. */
private suspend fun PagingSource<Int, MessageSummary>.refreshIds(loadSize: Int = 20): List<String> {
val result = load(PagingSource.LoadParams.Refresh(key = null, loadSize = loadSize, placeholdersEnabled = false))
return (result as PagingSource.LoadResult.Page).data.map { it.id }
}
@Test
fun pagingUnifiedFolderSummariesReturnsSyncedRowsNewestFirstAcrossAccounts() = runBlocking {
dao.insertNew(
listOf(
message("acct:INBOX:1", timestampMillis = 100),
message("acct:INBOX:2", timestampMillis = 300),
message("acct2:INBOX:3", accountId = "acct2", timestampMillis = 200),
message("acct:INBOX:search", timestampMillis = 999, inInbox = false), // search-only excluded
message("acct:Archive:1", folder = "Archive", timestampMillis = 500), // other folder excluded
),
)
// Newest-first by timestamp, both accounts' INBOX rows, no search-only row, no other folder.
assertEquals(
listOf("acct:INBOX:2", "acct2:INBOX:3", "acct:INBOX:1"),
dao.pagingUnifiedFolderSummaries("INBOX").refreshIds(),
)
}
@Test
fun pagingFolderSummariesIsScopedToOneAccountAndFolder() = runBlocking {
dao.insertNew(
listOf(
message("acct:INBOX:1", timestampMillis = 100),
message("acct:INBOX:2", timestampMillis = 200),
message("acct2:INBOX:3", accountId = "acct2", timestampMillis = 300), // other account
message("acct:Archive:1", folder = "Archive", timestampMillis = 400), // other folder
message("acct:INBOX:s", timestampMillis = 999, inInbox = false), // search-only
),
)
assertEquals(
listOf("acct:INBOX:2", "acct:INBOX:1"),
dao.pagingFolderSummaries("acct", "INBOX").refreshIds(),
)
}
@Test
fun pagingUnifiedFolderSearchSummariesMatchesEveryScannedColumnAndSurfacesSearchRows() = runBlocking {
dao.insertNew(
listOf(
message("bySubject", subject = "Quarterly report", timestampMillis = 100),
message("bySender", sender = "Reporter", subject = "x", timestampMillis = 200),
message("bySenderEmail", senderEmail = "report@x.org", subject = "x", timestampMillis = 300),
message("bySnippet", snippet = "see the report", subject = "x", timestampMillis = 400),
message("searchHit", subject = "report", timestampMillis = 500, inInbox = false), // surfaced
message("noMatch", subject = "unrelated", timestampMillis = 600),
message("otherFolder", subject = "report", folder = "Archive", timestampMillis = 700),
),
)
// Unified search matches sender/senderEmail/subject/snippet, includes transient inInbox=0 hits,
// and is folder-scoped. Newest-first.
assertEquals(
listOf("searchHit", "bySnippet", "bySenderEmail", "bySender", "bySubject"),
dao.pagingUnifiedFolderSearchSummaries("INBOX", "%report%").refreshIds(),
)
}
@Test
fun pagingFolderSearchSummariesIsAccountScopedAndSurfacesSearchRows() = runBlocking {
dao.insertNew(
listOf(
message("mine", subject = "the report", timestampMillis = 100),
message("mineSearch", subject = "report", timestampMillis = 200, inInbox = false),
message("theirs", subject = "report", accountId = "acct2", timestampMillis = 300),
),
)
assertEquals(
listOf("mineSearch", "mine"),
dao.pagingFolderSearchSummaries("acct", "INBOX", "%report%").refreshIds(),
)
}
@Test
fun browsePagingBreaksTimestampTiesByAscendingIdForATotalPageOrder() = runBlocking {
// Bulk mail can share a timestamp (issue #311): without a unique tiebreaker, rows tied at a
// LIMIT/OFFSET page boundary can duplicate or skip. Insertion order is scrambled so the ORDER BY
// — not the storage order — must produce the result.
dao.insertNew(
listOf(
message("tie-c", timestampMillis = 1_000),
message("tie-a", timestampMillis = 1_000),
message("tie-b", timestampMillis = 1_000),
message("newer", timestampMillis = 2_000),
),
)
// Newest timestamp first, then ties broken by ascending id — a deterministic total order.
val expected = listOf("newer", "tie-a", "tie-b", "tie-c")
assertEquals(expected, dao.pagingUnifiedFolderSummaries("INBOX").refreshIds())
assertEquals(expected, dao.pagingFolderSummaries("acct", "INBOX").refreshIds())
}
@Test
fun searchPagingBreaksTimestampTiesByAscendingIdForATotalPageOrder() = runBlocking {
dao.insertNew(
listOf(
message("hit-c", subject = "report", timestampMillis = 1_000),
message("hit-a", subject = "report", timestampMillis = 1_000),
message("hit-b", subject = "report", timestampMillis = 1_000),
),
)
val expected = listOf("hit-a", "hit-b", "hit-c")
assertEquals(expected, dao.pagingUnifiedFolderSearchSummaries("INBOX", "%report%").refreshIds())
assertEquals(expected, dao.pagingFolderSearchSummaries("acct", "INBOX", "%report%").refreshIds())
}
@Test
fun getUnfetchedIdsReturnsOnlySyncedRowsMissingABody() = runBlocking {
dao.insertNew(
listOf(
message("unfetched", bodyFetched = false),
message("fetched", bodyFetched = true),
message("searchUnfetched", bodyFetched = false, inInbox = false), // not synced
message("otherFolder", folder = "Archive", bodyFetched = false), // different folder
),
)
assertEquals(listOf("unfetched"), dao.getUnfetchedIds("acct", "INBOX"))
}
@Test
fun insertNewIgnoresConflictsAndLeavesExistingRowsIntact() = runBlocking {
dao.insertNew(listOf(message("acct:1", subject = "Original", isRead = true)))
// A re-insert of the same id (e.g. the next sync re-listing it) must NOT clobber the cached row.
dao.insertNew(listOf(message("acct:1", subject = "Replaced", isRead = false)))
val row = dao.getById("acct:1")
assertEquals("Original", row?.subject)
assertEquals(true, row?.isRead)
}
@Test
fun existingIdsReturnsOnlyThoseAlreadyStored() = runBlocking {
dao.insertNew(listOf(message("acct:1"), message("acct:2")))
assertEquals(
setOf("acct:1", "acct:2"),
dao.existingIds(listOf("acct:1", "acct:2", "acct:absent")).toSet(),
)
}
@Test
fun updateHeaderContentRefreshesDisplayFieldsAndUidOnly() = runBlocking {
dao.insertNew(
listOf(
message("acct:1", isRead = true, isStarred = true, inInbox = true, body = "cached", uid = 0),
),
)
dao.updateHeaderContent(
id = "acct:1",
sender = "Charles",
senderEmail = "charles@example.org",
subject = "Refreshed",
timestampMillis = 5_000L,
uid = 42L,
)
val row = requireNotNull(dao.getById("acct:1"))
assertEquals("Charles", row.sender)
assertEquals("charles@example.org", row.senderEmail)
assertEquals("Refreshed", row.subject)
assertEquals(5_000L, row.timestampMillis)
assertEquals(42L, row.uid)
// Local flags, the cached body, and inbox membership are deliberately left untouched.
assertEquals(true, row.isRead)
assertEquals(true, row.isStarred)
assertEquals("cached", row.body)
assertEquals(true, row.inInbox)
}
@Test
fun updateHeaderContentsRefreshesEveryRowInTheBatchAndLeavesFlagsAndBodiesUntouched() = runBlocking {
dao.insertNew(
listOf(
message("acct:1", isRead = true, isStarred = false, body = "cached-1", uid = 0),
message("acct:2", isRead = false, isStarred = true, body = "cached-2", uid = 0),
),
)
// The sync path (issue #310) refreshes a whole recent window at once via this single-transaction
// batch. Only the six header fields of each passed entity are applied; the rest are ignored.
dao.updateHeaderContents(
listOf(
message(
"acct:1",
sender = "Charles",
senderEmail = "charles@example.org",
subject = "One",
timestampMillis = 5_000L,
uid = 42L,
),
message(
"acct:2",
sender = "Grace",
senderEmail = "grace@example.org",
subject = "Two",
timestampMillis = 6_000L,
uid = 43L,
),
),
)
val one = requireNotNull(dao.getById("acct:1"))
assertEquals("Charles", one.sender)
assertEquals("charles@example.org", one.senderEmail)
assertEquals("One", one.subject)
assertEquals(5_000L, one.timestampMillis)
assertEquals(42L, one.uid)
// The casefold search columns track the refreshed headers (issue #232).
assertEquals("charles", one.senderFold)
assertEquals("one", one.subjectFold)
// Flags and the cached body are deliberately left untouched.
assertTrue(one.isRead)
assertEquals("cached-1", one.body)
val two = requireNotNull(dao.getById("acct:2"))
assertEquals("Grace", two.sender)
assertEquals("Two", two.subject)
assertEquals(43L, two.uid)
assertTrue(two.isStarred)
assertEquals("cached-2", two.body)
}
@Test
fun markSyncedPromotesSearchOnlyRowsIntoTheFolder() = runBlocking {
dao.insertNew(
listOf(
message("promote", inInbox = false),
message("leaveAlone", inInbox = false),
),
)
dao.markSynced(listOf("promote"))
assertEquals(listOf("promote"), dao.getSyncedIds("acct", "INBOX"))
assertFalse(dao.getById("leaveAlone")!!.inInbox)
}
@Test
fun updateBodyStoresBodyHtmlSnippetAndMarksFetched() = runBlocking {
dao.insertNew(listOf(message("acct:1", bodyFetched = false)))
dao.updateBody("acct:1", body = "<p>Hello</p>", isHtml = true, snippet = "Hello")
val row = requireNotNull(dao.getById("acct:1"))
assertEquals("<p>Hello</p>", row.body)
assertTrue(row.isHtml)
assertEquals("Hello", row.snippet)
assertTrue(row.bodyFetched)
}
@Test
fun setReadAndSetStarredToggleOnlyTheirFlag() = runBlocking {
dao.insertNew(listOf(message("acct:1", isRead = false, isStarred = false)))
dao.setRead("acct:1", true)
dao.setStarred("acct:1", true)
val row = requireNotNull(dao.getById("acct:1"))
assertTrue(row.isRead)
assertTrue(row.isStarred)
}
@Test
fun deleteByIdsRemovesExactlyTheGivenRows() = runBlocking {
dao.insertNew(listOf(message("a"), message("b"), message("c")))
dao.deleteByIds(listOf("a", "c"))
assertEquals(listOf("b"), dao.observeSummaries().first().map { it.id })
}
@Test
fun deleteByAccountRemovesEveryRowOfThatAccountOnly() = runBlocking {
dao.insertNew(
listOf(
message("acct:1"),
message("acct:Archive:1", folder = "Archive"),
message("acct2:1", accountId = "acct2"),
),
)
dao.deleteByAccount("acct")
assertEquals(listOf("acct2:1"), dao.observeSummaries().first().map { it.id })
}
@Test
fun deleteSyncedByAccountFolderSparesSearchRowsAndOtherFolders() = runBlocking {
dao.insertNew(
listOf(
message("synced", inInbox = true),
message("searchOnly", inInbox = false),
message("otherFolder", folder = "Archive"),
message("otherAccount", accountId = "acct2"),
),
)
dao.deleteSyncedByAccountFolder("acct", "INBOX")
assertNull("the synced INBOX row is deleted", dao.getById("synced"))
assertTrue("a transient search-only row is spared", dao.getById("searchOnly") != null)
assertTrue("another folder is untouched", dao.getById("otherFolder") != null)
assertTrue("another account is untouched", dao.getById("otherAccount") != null)
}
}
@@ -13,6 +13,7 @@ import org.junit.Assert.assertTrue
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.di.DatabaseModule
import java.lang.reflect.Modifier
/**
@@ -37,6 +38,82 @@ class MigrationTest {
FrameworkSQLiteOpenHelperFactory(),
)
/** v18 -> v19 (issue #232): the four casefold search columns appear, backfilled from their source. */
@Test
fun migrate18To19_addsAndBackfillsCasefoldSearchColumns() {
helper.createDatabase(TEST_DB, 18).apply {
execSQL(
"INSERT INTO messages (id, accountId, sender, senderEmail, subject, snippet, body, isHtml, " +
"timestampMillis, isRead, isStarred, folder, inInbox, bodyFetched, uid) VALUES " +
"('acct:INBOX:1', 'acct', 'Ada LOVELACE', 'Ada@Example.ORG', 'HELLO There', 'World Preview', " +
"'', 0, 1000, 0, 0, 'INBOX', 1, 1, 1)",
)
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 19, true, MIGRATION_18_19)
db.query(
"SELECT senderFold, senderEmailFold, subjectFold, snippetFold FROM messages WHERE id = 'acct:INBOX:1'",
).use { c ->
assertTrue(c.moveToFirst())
assertEquals("ada lovelace", c.getString(0))
assertEquals("ada@example.org", c.getString(1))
assertEquals("hello there", c.getString(2))
assertEquals("world preview", c.getString(3))
}
}
/**
* v19 -> v20 (issue #187): the unified-inbox covering index appears over exactly
* `(folder, inInbox, timestampMillis)`, cached rows survive, and the paged "All inboxes" query now
* plans as a bounded `SEARCH` on that index with no temp B-tree sort (instead of a whole-table
* `SCAN`). Asserting the query plan on the real Android SQLite proves the index is genuinely
* covering the filter+order, not merely present.
*/
@Test
fun migrate19To20_addsUnifiedInboxCoveringIndexUsedByTheSummaryScan() {
helper.createDatabase(TEST_DB, 19).apply {
// A representative spread: two synced INBOX rows, plus a transient search hit (inInbox = 0)
// — all must survive the pure additive index migration. Fold columns default to ''.
execSQL(
"INSERT INTO messages (id, accountId, sender, senderEmail, subject, snippet, body, isHtml, " +
"timestampMillis, isRead, isStarred, folder, inInbox, bodyFetched, uid) VALUES " +
"('a:INBOX:2', 'a', 'Ada', 'ada@example.org', 'Hi', '', '', 0, 2000, 0, 0, 'INBOX', 1, 1, 2), " +
"('b:INBOX:1', 'b', 'Bob', 'bob@example.org', 'Yo', '', '', 0, 1000, 0, 0, 'INBOX', 1, 1, 1), " +
"('a:INBOX:9', 'a', 'Cy', 'cy@example.org', 'Q', '', '', 0, 3000, 0, 0, 'INBOX', 0, 0, 9)",
)
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 20, true, MIGRATION_19_20)
// The index exists over exactly (folder, inInbox, timestampMillis), in that order.
assertEquals(
"19->20 must create the (folder, inInbox, timestampMillis) unified-inbox covering index",
listOf("folder", "inInbox", "timestampMillis"),
db.indexColumns("index_messages_folder_inInbox_timestampMillis"),
)
// The cached rows are untouched by the additive migration.
assertEquals("19->20 must not touch the mail cache", 3, db.count("messages"))
// The production pagingUnifiedFolderSummaries query now SEARCHes the new index and drops the
// temp B-tree sort (before this index it SCANned index_messages_timestampMillis whole-table).
val plan = db.queryPlan(
"SELECT id, accountId, sender, senderEmail, subject, snippet, timestampMillis, isRead, " +
"isStarred, folder, inInbox, bodyFetched FROM messages " +
"WHERE folder = 'INBOX' AND inInbox = 1 ORDER BY timestampMillis DESC",
)
assertTrue(
"the unified-inbox summary query must SEARCH the covering index, not SCAN; plan was $plan",
plan.any { it.contains("SEARCH") && it.contains("index_messages_folder_inInbox_timestampMillis") },
)
assertTrue(
"the covering index must supply the ordering (no temp B-tree sort); plan was $plan",
plan.none { it.contains("TEMP B-TREE") },
)
db.close()
}
/** v11 -> v12 (PR #54): `folders.specialUse` appears defaulting to 0 and existing data survives. */
@Test
fun migrate11To12_defaultsExistingFoldersToNotSpecialUse() {
@@ -73,6 +150,170 @@ class MigrationTest {
db.close()
}
/** v7 -> v8: pre-upgrade messages are filed under INBOX and the folders table appears. */
@Test
fun migrate7To8_filesExistingMessagesUnderInboxAndAddsFoldersTable() {
helper.createDatabase(TEST_DB, 7).apply {
insertAccount()
execSQL(
"INSERT INTO messages (id, accountId, sender, senderEmail, subject, snippet, body, isHtml, " +
"timestampMillis, isRead, isStarred, inInbox, bodyFetched) VALUES " +
"('acct:1', 'acct', 'Ada', 'ada@example.org', 'Hi', '', '', 0, 1000, 0, 0, 1, 1)",
)
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 8, true, MIGRATION_7_8)
db.query("SELECT folder FROM messages WHERE id = 'acct:1'").use { c ->
assertTrue("the pre-upgrade message must survive", c.moveToFirst())
assertEquals("7->8 files pre-upgrade rows under INBOX", "INBOX", c.getString(0))
}
// The new folders table exists and accepts a row.
db.execSQL(
"INSERT INTO folders (accountId, fullName, displayName, role, selectable, sortOrder) " +
"VALUES ('acct', 'INBOX', 'INBOX', 'INBOX', 1, 0)",
)
assertEquals(1, db.count("folders"))
db.close()
}
/** v8 -> v9: a default per-account settings row is backfilled for every existing account. */
@Test
fun migrate8To9_backfillsADefaultSettingsRowPerAccount() {
helper.createDatabase(TEST_DB, 8).apply {
insertAccount()
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 9, true, MIGRATION_8_9)
db.query("SELECT signature, signatureEnabled, notificationsEnabled FROM account_settings").use { c ->
assertTrue("8->9 must backfill a settings row for the existing account", c.moveToFirst())
assertEquals("", c.getString(0))
assertEquals(1, c.getInt(1))
assertEquals(1, c.getInt(2))
assertFalse("exactly one settings row per account", c.moveToNext())
}
db.close()
}
/** v9 -> v10: `outbox`/`drafts` gain an empty `bccAddresses` and queued/saved rows survive. */
@Test
fun migrate9To10_addsEmptyBccToOutboxAndDrafts() {
helper.createDatabase(TEST_DB, 9).apply {
execSQL(
"INSERT INTO outbox (id, accountId, toAddresses, ccAddresses, subject, body, createdAt, " +
"lastError) VALUES ('out-1', 'acct', 'bob@example.org', '', 'Queued', 'Body', 3000, NULL)",
)
execSQL(
"INSERT INTO drafts (id, accountId, toAddresses, ccAddresses, subject, body, updatedAt, " +
"attachments) VALUES ('draft-1', 'acct', 'bob@example.org', '', 'Draft', 'Text', 4000, '')",
)
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 10, true, MIGRATION_9_10)
db.query("SELECT bccAddresses FROM outbox WHERE id = 'out-1'").use { c ->
assertTrue("the queued outbox row must survive", c.moveToFirst())
assertEquals("existing outbox rows read an empty bcc", "", c.getString(0))
}
db.query("SELECT bccAddresses FROM drafts WHERE id = 'draft-1'").use { c ->
assertTrue("the saved draft must survive", c.moveToFirst())
assertEquals("existing draft rows read an empty bcc", "", c.getString(0))
}
db.close()
}
/** v10 -> v11: nullable `bodyHtml` appears and a legacy per-account signature is carried across. */
@Test
fun migrate10To11_addsNullableBodyHtmlAndBackfillsTheDefaultSignature() {
helper.createDatabase(TEST_DB, 10).apply {
insertAccount()
execSQL(
"INSERT INTO account_settings (accountId, signature, signatureEnabled, notificationsEnabled) " +
"VALUES ('acct', 'Cheers,' || char(10) || 'Ada', 1, 1)",
)
execSQL(
"INSERT INTO outbox (id, accountId, toAddresses, ccAddresses, bccAddresses, subject, body, " +
"createdAt, lastError) VALUES ('out-1', 'acct', 'bob@example.org', '', '', 'Q', 'B', 1, NULL)",
)
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 11, true, MIGRATION_10_11)
// The v9 signature becomes the account's default rich-text signature (newlines -> <br>).
db.query("SELECT name, contentHtml, isDefault FROM signatures WHERE accountId = 'acct'").use { c ->
assertTrue("10->11 must backfill the legacy per-account signature", c.moveToFirst())
assertEquals("Signature", c.getString(0))
assertEquals("Cheers,<br>Ada", c.getString(1))
assertEquals(1, c.getInt(2))
assertFalse("exactly one signature row must be backfilled", c.moveToNext())
}
// bodyHtml is added nullable and reads back null for a message composed before formatting.
db.query("SELECT bodyHtml FROM outbox WHERE id = 'out-1'").use { c ->
assertTrue(c.moveToFirst())
assertTrue("bodyHtml must default to null (plaintext-only)", c.isNull(0))
}
db.close()
}
/** v13 -> v14: snippets are re-derived per `isHtml`; only fetched-body rows are touched. */
@Test
fun migrate13To14_reDerivesSnippetsRespectingIsHtmlAndSkipsUnfetchedRows() {
helper.createDatabase(TEST_DB, 13).apply {
insertV13Message(
id = "html",
isHtml = 1,
body = "<style>p{color:red}</style><p>Hello world</p>",
snippet = "STALE",
bodyFetched = 1,
)
insertV13Message(
id = "plain",
isHtml = 0,
body = "keep <not a tag> literal",
snippet = "STALE",
bodyFetched = 1,
)
insertV13Message(id = "unfetched", isHtml = 0, body = "", snippet = "UNTOUCHED", bodyFetched = 0)
close()
}
val db = helper.runMigrationsAndValidate(TEST_DB, 14, true, MIGRATION_13_14)
val htmlSnippet = snippetOf(db, "html")
assertTrue("HTML snippet keeps visible text", htmlSnippet.contains("Hello world"))
assertFalse("HTML snippet drops leaked <style> content", htmlSnippet.contains("color"))
assertFalse("the stale snippet is replaced", htmlSnippet.contains("STALE"))
// A plain-text body keeps its literal angle brackets (no markup handling).
assertTrue("plain snippet keeps literal markup", snippetOf(db, "plain").contains("<not a tag>"))
// Rows whose body was never fetched keep their existing snippet.
assertEquals("UNTOUCHED", snippetOf(db, "unfetched"))
db.close()
}
private fun snippetOf(db: SupportSQLiteDatabase, id: String): String =
db.query("SELECT snippet FROM messages WHERE id = ?", arrayOf<Any>(id)).use { c ->
assertTrue("row $id must exist", c.moveToFirst())
c.getString(0)
}
private fun SupportSQLiteDatabase.insertV13Message(
id: String,
isHtml: Int,
body: String,
snippet: String,
bodyFetched: Int,
) = execSQL(
"INSERT INTO messages (id, accountId, sender, senderEmail, subject, snippet, body, isHtml, " +
"timestampMillis, isRead, isStarred, folder, inInbox, bodyFetched, uid) " +
"VALUES (?, 'acct', 'Ada', 'ada@example.org', 'Hi', ?, ?, ?, 1000, 0, 0, 'INBOX', 1, ?, 1)",
arrayOf<Any>(id, snippet, body, isHtml, bodyFetched),
)
/** v14 -> v15 (issue #66): `folders.hierarchyDelimiter` appears defaulting to NULL; data survives. */
@Test
fun migrate14To15_addsNullHierarchyDelimiterToFolders() {
@@ -116,6 +357,23 @@ class MigrationTest {
)
}
/**
* The replay tests discover migrations reflectively, but nothing there checks that [DatabaseModule]
* actually *registers* them. With no destructive fallback, a migration authored, schema-committed,
* and replay-tested but omitted from `addMigrations` still crash-loops every upgrading user at first
* database open (issue #312). Assert the builder's registered set ([DatabaseModule.ALL_MIGRATIONS])
* is exactly the reflectively-discovered set, so such an omission fails here instead.
*/
@Test
fun databaseModuleRegistersEveryDeclaredMigration() {
assertEquals(
"DatabaseModule.ALL_MIGRATIONS must register every Migration in Migrations.kt " +
"(no destructive fallback, so a forgotten step crash-loops upgrades)",
allAppMigrations.map { it.startVersion to it.endVersion },
DatabaseModule.ALL_MIGRATIONS.sortedBy { it.startVersion }.map { it.startVersion to it.endVersion },
)
}
/**
* Creates a database at v7 (the oldest exported schema), fills it like a used install, then
* replays every migration one step at a time — `runMigrationsAndValidate` diffs the migrated
@@ -368,6 +626,22 @@ class MigrationTest {
c.getInt(0)
}
/** Column names of [index], in index (seqno) order — empty if the index does not exist. */
private fun SupportSQLiteDatabase.indexColumns(index: String): List<String> =
query("PRAGMA index_info(`$index`)").use { c ->
buildList {
// PRAGMA index_info rows are (seqno, cid, name); the cursor yields them in seqno order.
while (c.moveToNext()) add(c.getString(2))
}
}
/** The human-readable `detail` step of each `EXPLAIN QUERY PLAN [sql]` row (the last column). */
private fun SupportSQLiteDatabase.queryPlan(sql: String): List<String> = query("EXPLAIN QUERY PLAN $sql").use { c ->
buildList {
while (c.moveToNext()) add(c.getString(c.columnCount - 1))
}
}
private companion object {
const val TEST_DB = "migration-test.db"
@@ -0,0 +1,97 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.OutboxDao
import org.libremail.data.local.entity.OutboxEntity
/**
* Real-SQLite behavior of [OutboxDao] — the queue the send worker drains: FIFO ordering by
* `createdAt`, point lookups, the live list/count observers, the last-error mutator, and row
* deletion on successful send.
*/
@RunWith(AndroidJUnit4::class)
class OutboxDaoTest {
private lateinit var db: LibreMailDatabase
private lateinit var dao: OutboxDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, LibreMailDatabase::class.java).build()
dao = db.outboxDao()
}
@After
fun tearDown() = db.close()
private fun outbox(id: String, createdAt: Long = 1_000L) = OutboxEntity(
id = id,
accountId = "acct",
toAddresses = "bob@example.org",
ccAddresses = "",
subject = "Subject $id",
body = "Body",
createdAt = createdAt,
)
@Test
fun getAllReturnsQueuedMessagesOldestFirst() = runBlocking {
dao.insert(outbox("late", createdAt = 300))
dao.insert(outbox("early", createdAt = 100))
dao.insert(outbox("middle", createdAt = 200))
assertEquals(listOf("early", "middle", "late"), dao.getAll().map { it.id })
}
@Test
fun getByIdReturnsTheRowOrNull() = runBlocking {
dao.insert(outbox("out-1"))
assertEquals("Subject out-1", dao.getById("out-1")?.subject)
assertNull(dao.getById("absent"))
}
@Test
fun observeAllAndObserveCountReflectTheQueue() = runBlocking {
dao.insert(outbox("out-1", createdAt = 100))
dao.insert(outbox("out-2", createdAt = 200))
assertEquals(listOf("out-1", "out-2"), dao.observeAll().first().map { it.id })
assertEquals(2, dao.observeCount().first())
}
@Test
fun setErrorRecordsAndThenClearsTheLastError() = runBlocking {
dao.insert(outbox("out-1"))
dao.setError("out-1", "SMTP 550")
assertEquals("SMTP 550", dao.getById("out-1")?.lastError)
dao.setError("out-1", null)
assertNull("a successful retry clears the error", dao.getById("out-1")?.lastError)
}
@Test
fun deleteRemovesTheSentRow() = runBlocking {
dao.insert(outbox("out-1"))
dao.insert(outbox("out-2"))
dao.delete("out-1")
assertEquals(listOf("out-2"), dao.getAll().map { it.id })
assertEquals(1, dao.observeCount().first())
}
}
@@ -0,0 +1,153 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.dao.SignatureDao
import org.libremail.data.local.entity.AccountEntity
import org.libremail.data.local.entity.ServerConfigEmbedded
import org.libremail.data.local.entity.SignatureEntity
/**
* Real-SQLite behavior of [SignatureDao] in [AccountDatabase]: the default-first / name-ordered
* observation, the default/first/count reads, and the "exactly one default per account" transaction
* ([SignatureDao.setDefault]). A parent account row is inserted first because signatures foreign-key
* to `accounts`.
*/
@RunWith(AndroidJUnit4::class)
class SignatureDaoTest {
private lateinit var db: AccountDatabase
private lateinit var dao: SignatureDao
private lateinit var accountDao: AccountDao
@Before
fun setUp() {
val context = ApplicationProvider.getApplicationContext<Context>()
db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
dao = db.signatureDao()
accountDao = db.accountDao()
}
@After
fun tearDown() = db.close()
private suspend fun insertAccount(id: String = "acct") = accountDao.upsert(
AccountEntity(
id = id,
email = "$id@example.org",
displayName = "Name",
authType = "PASSWORD_IMAP",
imap = ServerConfigEmbedded("imap.example.org", 993, "SSL_TLS"),
smtp = ServerConfigEmbedded("smtp.example.org", 465, "SSL_TLS"),
),
)
private fun signature(id: String, name: String, isDefault: Boolean = false, accountId: String = "acct") =
SignatureEntity(
id = id,
accountId = accountId,
name = name,
contentHtml = "<p>$name</p>",
isDefault = isDefault,
)
@Test
fun observeForAccountOrdersDefaultFirstThenByNameCaseInsensitive() = runBlocking {
insertAccount()
dao.upsert(signature("s-zeta", "Zeta", isDefault = true))
dao.upsert(signature("s-alpha", "alpha"))
dao.upsert(signature("s-beta", "Beta"))
// isDefault DESC puts the default first; the rest sort by name COLLATE NOCASE (alpha < Beta).
assertEquals(
listOf("Zeta", "alpha", "Beta"),
dao.observeForAccount("acct").first().map { it.name },
)
}
@Test
fun getByIdGetDefaultFirstForAccountAndCountReadTheExpectedRows() = runBlocking {
insertAccount()
dao.upsert(signature("s-work", "Work", isDefault = true))
dao.upsert(signature("s-personal", "aPersonal"))
assertEquals("Work", dao.getById("s-work")?.name)
assertNull(dao.getById("absent"))
assertEquals("Work", dao.getDefault("acct")?.name)
// firstForAccount ignores isDefault and takes the name-first row (aPersonal < Work).
assertEquals("aPersonal", dao.firstForAccount("acct")?.name)
assertEquals(2, dao.countForAccount("acct"))
}
@Test
fun getDefaultIsNullWhenNoSignatureIsMarkedDefault() = runBlocking {
insertAccount()
dao.upsert(signature("s-1", "One"))
assertNull(dao.getDefault("acct"))
assertEquals(0, dao.countForAccount("absent"))
}
@Test
fun upsertReplacesASignatureAndDeleteRemovesIt() = runBlocking {
insertAccount()
dao.upsert(signature("s-1", "Original"))
dao.upsert(signature("s-1", "Edited"))
assertEquals("Edited", dao.getById("s-1")?.name)
dao.delete("s-1")
assertNull(dao.getById("s-1"))
}
@Test
fun clearDefaultAndMarkDefaultToggleTheFlag() = runBlocking {
insertAccount()
dao.upsert(signature("s-1", "One", isDefault = true))
dao.clearDefault("acct")
assertNull("clearDefault drops the account's default", dao.getDefault("acct"))
dao.markDefault("s-1")
assertEquals("s-1", dao.getDefault("acct")?.id)
}
@Test
fun setDefaultMakesExactlyOneSignatureTheAccountsDefault() = runBlocking {
insertAccount()
dao.upsert(signature("s-1", "One", isDefault = true))
dao.upsert(signature("s-2", "Two"))
dao.setDefault("acct", "s-2")
// The transaction clears every other default first, so only s-2 remains default.
assertEquals("s-2", dao.getDefault("acct")?.id)
assertEquals(false, dao.getById("s-1")?.isDefault)
assertEquals(true, dao.getById("s-2")?.isDefault)
}
@Test
fun setDefaultIsScopedToTheAccount() = runBlocking {
insertAccount("acct")
insertAccount("acct2")
dao.upsert(signature("a1", "A1", isDefault = true, accountId = "acct"))
dao.upsert(signature("b1", "B1", isDefault = true, accountId = "acct2"))
dao.setDefault("acct", "a1")
// clearDefault in setDefault only touches the target account; acct2's default is untouched.
assertEquals("b1", dao.getDefault("acct2")?.id)
}
}
@@ -0,0 +1,205 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.content.Context
import android.os.Build
import android.system.Os
import android.system.OsConstants
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import net.zetetic.database.sqlcipher.SQLiteDatabase
import net.zetetic.database.sqlcipher.SupportOpenHelperFactory
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.entity.MessageEntity
import org.libremail.reporting.AppLog
import java.io.File
import java.io.PrintWriter
import java.io.StringWriter
/**
* SPIKE for issue #359 — characterizes, on a real device, whether opening the opt-in SQLCipher-encrypted
* cache throws `UnsatisfiedLinkError` because the bundled `libsqlcipher.so` is not compatible with the
* device's memory **page size**. Android 15+/SDK-37 devices may run **16 KB pages**; a native `.so` not
* built/aligned for 16 KB pages fails to load (`dlopen`) or to bind its JNI methods, surfacing as
* `UnsatisfiedLinkError` at `System.loadLibrary("sqlcipher")` or at `SQLiteConnection.nativeOpen`.
*
* This is investigation-only: it does not change production behaviour. It runs the exact #359 open path in
* three independent stages so a failing run tells us **which** stage breaks, and it records the device page
* size ([Os.sysconf] `_SC_PAGESIZE`) so a pass/fail can be tied to 4 KB vs 16 KB pages. It is meant to be
* run twice on the SAME device — once in 4 KB mode, once in 16 KB mode (Pixel Developer Options toggle) —
* to give a definitive A/B: if every stage passes at 4 KB and fails at 16 KB, 16 KB pages are the cause.
*
* On success each stage asserts the encrypted DB genuinely opens and round-trips a row. On failure each
* stage re-raises the full throwable — class, message (which `.so`), whether a [LinkageError] is in the
* cause chain, the page size, and the complete stack trace — so the A/B report captures the real cause
* rather than a bare assertion. All data is synthetic; nothing logged or asserted is PII.
*/
@RunWith(AndroidJUnit4::class)
class SqlCipherOpenSpikeTest {
private val context = ApplicationProvider.getApplicationContext<Context>()
private val dbName = "sqlcipher_spike_test.db"
private val dbFile: File get() = context.getDatabasePath(dbName)
private val probeDbName = "sqlcipher_spike_probe.db"
private val probeDbFile: File get() = context.getDatabasePath(probeDbName)
// 64 hex chars == a 32-byte SQLCipher passphrase, matching DatabaseKeyStore's format.
private val passphrase = "0123456789abcdef".repeat(4)
@Before
@After
fun clean() {
listOf(dbName, probeDbName).forEach { name ->
context.deleteDatabase(name)
context.getDatabasePath(name).parentFile
?.listFiles { f -> f.name.startsWith(name) }
?.forEach { it.delete() }
}
}
/**
* Stage A — load the SQLCipher native library the way production does
* ([DatabaseEncryption.ensureNativeLibraryLoaded] -> `System.loadLibrary("sqlcipher")`). This is the
* first place a 16 KB-incompatible `.so` can fail (`dlopen` rejects an unaligned library).
*/
@Test
fun stageA_sqlCipherNativeLibraryLoads() {
AppLog.i(TAG, "stageA start: $environment")
try {
DatabaseEncryption.ensureNativeLibraryLoaded()
} catch (t: Throwable) {
surface("A/loadLibrary(\"sqlcipher\")", t)
}
AppLog.i(TAG, "stageA PASS: SQLCipher native library loaded; $environment")
}
/**
* Stage B — reach `SQLiteConnection.nativeOpen`: after loading the library (as production does), open a
* keyed SQLCipher database and round-trip a row through the cipher. This is the exact call site named in
* the #359 crash (`UnsatisfiedLinkError … SQLiteConnection.nativeOpen`).
*/
@Test
fun stageB_keyedNativeOpenSucceeds() {
AppLog.i(TAG, "stageB start: $environment")
try {
DatabaseEncryption.ensureNativeLibraryLoaded()
val db = SQLiteDatabase.openOrCreateDatabase(
probeDbFile.absolutePath,
passphrase.toByteArray(Charsets.US_ASCII),
null, // no CursorFactory
null, // no DatabaseErrorHandler
)
try {
db.execSQL("CREATE TABLE IF NOT EXISTS spike(x INTEGER)")
db.execSQL("INSERT INTO spike(x) VALUES (42)")
db.rawQuery("SELECT x FROM spike LIMIT 1", null).use { cursor ->
assertTrue("keyed DB returned no row", cursor.moveToFirst())
assertEquals("keyed DB round-trip mismatch", 42, cursor.getInt(0))
}
} finally {
db.close()
}
} catch (t: Throwable) {
surface("B/SQLiteConnection.nativeOpen (keyed open)", t)
}
AppLog.i(TAG, "stageB PASS: keyed nativeOpen + round-trip OK; $environment")
}
/**
* Stage C — the full #359 production path: create a plaintext Room cache with one row, convert it to
* SQLCipher ciphertext ([DatabaseEncryption.ensureEncrypted]), load the library, then reopen the cache
* through Room's [SupportOpenHelperFactory] (exactly [org.libremail.di.DatabaseModule]'s encrypted open
* lambda) and read the seeded row back.
*/
@Test
fun stageC_encryptedRoomCacheOpensThroughProductionFactory() {
AppLog.i(TAG, "stageC start: $environment")
try {
Room.databaseBuilder(context, LibreMailDatabase::class.java, dbName).build().apply {
runBlocking { messageDao().insertNew(listOf(message("acct:1"))) }
close()
}
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
assertTrue("precondition: fixture must be genuinely encrypted", DatabaseEncryption.isEncrypted(dbFile))
DatabaseEncryption.ensureNativeLibraryLoaded()
val database = Room.databaseBuilder(context, LibreMailDatabase::class.java, dbName)
.openHelperFactory(SupportOpenHelperFactory(passphrase.toByteArray(Charsets.US_ASCII), null, false))
.build()
try {
val ids = runBlocking { database.messageDao().observeSummaries().first().map { it.id } }
assertEquals("encrypted cache did not read the seeded row back", listOf("acct:1"), ids)
} finally {
database.close()
}
} catch (t: Throwable) {
surface("C/Room encrypted cache open (SupportOpenHelperFactory)", t)
}
AppLog.i(TAG, "stageC PASS: encrypted Room cache opened through production factory; $environment")
}
/** A one-line, PII-free description of the device + page size every stage stamps into its log/report. */
private val environment: String
get() = "PAGE_SIZE=${pageSizeBytes()} bytes (16384 => 16 KB pages), SDK=${Build.VERSION.SDK_INT}, " +
"release=${Build.VERSION.RELEASE}, abis=${Build.SUPPORTED_ABIS.joinToString(",")}"
private fun pageSizeBytes(): Long = Os.sysconf(OsConstants._SC_PAGESIZE)
/**
* Fails the stage while surfacing the complete cause so the on-device A/B report captures the real
* `UnsatisfiedLinkError` (which `.so`, full stack trace, page size) instead of a bare assertion.
*/
private fun surface(stage: String, t: Throwable): Nothing {
val stack = StringWriter().also { t.printStackTrace(PrintWriter(it)) }.toString()
val chain = buildString {
var current: Throwable? = t
while (current != null) {
append("\n - ").append(current.javaClass.name).append(": ").append(current.message)
current = current.cause
}
}
val diagnostic = buildString {
append("SQLCipher spike stage '").append(stage).append("' FAILED on this device.")
append("\n ").append(environment)
append("\n LinkageError in cause chain = ").append(hasLinkageError(t))
append("\n cause chain:").append(chain)
append("\n full stack trace:\n").append(stack)
}
AppLog.e(TAG, "SQLCipher spike stage '$stage' FAILED; $environment", t)
throw AssertionError(diagnostic, t)
}
private fun hasLinkageError(throwable: Throwable): Boolean {
var current: Throwable? = throwable
while (current != null) {
if (current is LinkageError) return true
current = current.cause
}
return false
}
private fun message(id: String) = MessageEntity(
id = id,
accountId = "acct",
sender = "Ada",
senderEmail = "ada@example.org",
subject = "Hi",
snippet = "",
body = "",
timestampMillis = 1_000L,
isRead = false,
isStarred = false,
)
private companion object {
const val TAG = "SqlCipherOpenSpike"
}
}
@@ -0,0 +1,157 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.sync
import android.content.Context
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.work.ListenableWorker.Result
import androidx.work.WorkerFactory
import androidx.work.WorkerParameters
import androidx.work.testing.TestListenableWorkerBuilder
import dagger.Lazy
import io.mockk.every
import io.mockk.mockk
import io.mockk.unmockkAll
import io.mockk.verify
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.security.EncryptedCacheGuard
import org.libremail.data.security.PassphraseSession
import org.libremail.data.settings.AppSettings
import org.libremail.data.settings.SettingsRepository
/**
* On-device proof that [PruneWorker] and [BackfillWorker] defer (`Result.retry()`) while the encrypted
* cache is locked, driving them with the REAL [EncryptedCacheGuard] rather than the mocked guard the JVM
* `PruneWorkerTest`/`BackfillWorkerTest` use (issue #225). [EncryptedCacheGuard] depends only on
* [SettingsRepository] and [PassphraseSession] — never the Keystore or `BiometricPrompt` — so the locked
* state is reproduced here with no device auth: a fixed [SettingsRepository] (mocked the same way
* `DatabaseProvisionerInstrumentedTest` fakes its security/settings collaborators) plus a real,
* never-unlocked [PassphraseSession].
*
* Caveat (see issue #226): the true "WorkManager cold-starts the process with no UI" can't be reproduced
* in-process. The locked-[PassphraseSession] seam is the faithful stand-in; the auth-bound Keystore path
* stays device-only (see `DatabaseKeyCipher`).
*
* "No `libremail.db` connection is opened" is proven by construction rather than by inspecting the
* on-disk file: [PruneWorker]/[BackfillWorker] can only reach the database through the `Lazy`
* [MailPruner]/[MailBackfiller] passed into their constructor (mirroring `DatabaseModule`'s real Hilt
* wiring), and both gate on [EncryptedCacheGuard.isCacheLocked] BEFORE ever resolving that `Lazy`.
* Asserting the `Lazy` is never resolved is therefore a direct, deterministic proof that no DAO method —
* and so no Room/SQLCipher open — ran, without coupling the test to whatever else the shared
* instrumentation process (or the real `libremail.db` file) happens to be doing.
*/
@RunWith(AndroidJUnit4::class)
class WorkerCacheLockDeferralInstrumentedTest {
private val context: Context = ApplicationProvider.getApplicationContext()
// A fresh, real instance per test (JUnit4 builds a new test-class instance per method) — never
// unlocked here, so isUnlocked() stays false exactly like a cold process start before the user has
// authenticated.
private val session = PassphraseSession()
@After
fun tearDown() {
unmockkAll()
}
@Test
fun pruneWorkerDefersWithoutResolvingThePrunerWhileTheRealGuardReportsLocked() = runBlocking<Unit> {
val cacheGuard = guardFor(appLock = true, encryptCache = true)
assertTrue("precondition: the real guard must report locked", cacheGuard.isCacheLocked())
val lazyPruner = mockk<Lazy<MailPruner>>()
val worker = TestListenableWorkerBuilder<PruneWorker>(context)
.setWorkerFactory(pruneWorkerFactory(lazyPruner, cacheGuard))
.build()
val result = withTimeout(TIMEOUT_MS) { worker.doWork() }
assertEquals(Result.retry(), result)
// The invariant under test: a locked cache must never even resolve the DB-backed collaborator.
verify(exactly = 0) { lazyPruner.get() }
}
@Test
fun backfillWorkerDefersWithoutResolvingTheBackfillerWhileTheRealGuardReportsLocked() = runBlocking<Unit> {
val cacheGuard = guardFor(appLock = true, encryptCache = true)
assertTrue("precondition: the real guard must report locked", cacheGuard.isCacheLocked())
val lazyBackfiller = mockk<Lazy<MailBackfiller>>()
val worker = TestListenableWorkerBuilder<BackfillWorker>(context)
.setWorkerFactory(backfillWorkerFactory(lazyBackfiller, cacheGuard))
.build()
val result = withTimeout(TIMEOUT_MS) { worker.doWork() }
assertEquals(Result.retry(), result)
verify(exactly = 0) { lazyBackfiller.get() }
}
/**
* Contrast case so the two tests above can't be vacuously true: the same real guard, driven by the
* same collaborator types, reports UNLOCKED once app-lock is off. [EncryptedCacheGuard] otherwise has
* no test of its own anywhere in the suite (only callers mocking the whole guard), so this is also
* this class's only direct coverage of its boolean logic.
*/
@Test
fun theRealGuardReportsUnlockedWhenAppLockIsOff() = runBlocking<Unit> {
val cacheGuard = guardFor(appLock = false, encryptCache = true)
assertFalse(cacheGuard.isCacheLocked())
}
/** Second branch of the same contrast: an authenticated session unlocks the cache even with app-lock on. */
@Test
fun theRealGuardReportsUnlockedOnceTheSessionIsUnlocked() = runBlocking<Unit> {
val cacheGuard = guardFor(appLock = true, encryptCache = true)
assertTrue("precondition: locked before authentication", cacheGuard.isCacheLocked())
session.unlock(FAKE_PASSPHRASE)
assertFalse(cacheGuard.isCacheLocked())
}
/** A real [EncryptedCacheGuard] over a fixed (mocked) [SettingsRepository] and the real [session]. */
private fun guardFor(appLock: Boolean, encryptCache: Boolean): EncryptedCacheGuard {
val settingsRepository = mockk<SettingsRepository>()
val settings = AppSettings(appLock = appLock, encryptCache = encryptCache)
every { settingsRepository.settings } returns flowOf(settings)
return EncryptedCacheGuard(settingsRepository, session)
}
private fun pruneWorkerFactory(lazyPruner: Lazy<MailPruner>, cacheGuard: EncryptedCacheGuard) =
object : WorkerFactory() {
override fun createWorker(
appContext: Context,
workerClassName: String,
workerParameters: WorkerParameters,
) = PruneWorker(appContext, workerParameters, lazyPruner, cacheGuard)
}
private fun backfillWorkerFactory(lazyBackfiller: Lazy<MailBackfiller>, cacheGuard: EncryptedCacheGuard) =
object : WorkerFactory() {
override fun createWorker(
appContext: Context,
workerClassName: String,
workerParameters: WorkerParameters,
) = BackfillWorker(appContext, workerParameters, lazyBackfiller, cacheGuard)
}
private companion object {
// Generous bound: a locked run must fail fast (no passphrase await), so this is only ever
// approached by a real regression — a passing run returns almost immediately.
const val TIMEOUT_MS = 5_000L
// 64 hex chars, matching DatabaseKeyStore's passphrase format. Never used to open a real database.
const val FAKE_PASSPHRASE = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
}
}
@@ -0,0 +1,182 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.debug
import android.content.BroadcastReceiver
import android.content.ComponentName
import android.content.Context
import android.content.Intent
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.work.ListenableWorker.Result
import androidx.work.WorkerFactory
import androidx.work.WorkerParameters
import androidx.work.testing.TestListenableWorkerBuilder
import dagger.Lazy
import io.mockk.coEvery
import io.mockk.every
import io.mockk.mockk
import io.mockk.unmockkAll
import io.mockk.verify
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.security.EncryptedCacheGuard
import org.libremail.data.security.PassphraseSession
import org.libremail.data.settings.AppSettings
import org.libremail.data.settings.SettingsRepository
import org.libremail.data.sync.BackfillWorker
import org.libremail.data.sync.DebugFetchGate
import org.libremail.data.sync.FetchScope
import org.libremail.data.sync.MailBackfiller
import java.util.concurrent.CountDownLatch
import java.util.concurrent.TimeUnit
/**
* On-device proof of the debug-only fetch gate (issue #393): the adb-reachable [FetchGateReceiver]
* updates [DebugFetchGate] and returns the resulting state as ordered-broadcast result data (exactly
* what `adb shell am broadcast ... FETCH_GATE` prints back to the harness), and a gated proactive path
* ([BackfillWorker]) genuinely defers while an un-gated path keeps running. The broadcast is sent
* ordered — the same delivery mode `am broadcast` uses — so [BroadcastReceiver.getResultData] on the
* final receiver reads back what the gate set, with no logcat race.
*
* The worker-deferral cases reuse `WorkerCacheLockDeferralInstrumentedTest`'s approach: build a
* [BackfillWorker] with a real, never-unlocked-or-off [EncryptedCacheGuard] and a `Lazy` [MailBackfiller]
* whose resolution is observable, so "the gate deferred before touching the DB" is proven by the `Lazy`
* never being resolved.
*/
@RunWith(AndroidJUnit4::class)
class FetchGateReceiverInstrumentedTest {
private val context: Context = ApplicationProvider.getApplicationContext()
// A fresh, real, never-unlocked session per test — so the real guard reports UNLOCKED only because
// app-lock is off (see [unlockedGuard]), never because of leftover auth state.
private val session = PassphraseSession()
@Before
@After
fun resetGate() {
DebugFetchGate.reset()
unmockkAll()
}
@Test
fun pauseUpdatesTheGateAndReturnsTheReadBack() {
val data = sendGateBroadcast(FetchGateReceiver.ACTION_PAUSE, "backfill,prefetch")
assertEquals("paused=[backfill,prefetch]", data)
assertTrue(DebugFetchGate.isPaused(FetchScope.BACKFILL))
assertTrue(DebugFetchGate.isPaused(FetchScope.PREFETCH))
}
@Test
fun resumeAllClearsTheGateAndReturnsAnEmptyReadBack() {
sendGateBroadcast(FetchGateReceiver.ACTION_PAUSE, "all")
val data = sendGateBroadcast(FetchGateReceiver.ACTION_RESUME, "all")
assertEquals("paused=[]", data)
assertFalse(DebugFetchGate.isPaused(FetchScope.BACKFILL))
assertFalse(DebugFetchGate.isPaused(FetchScope.PREFETCH))
}
@Test
fun queryReadsBackTheStateWithoutMutatingIt() {
sendGateBroadcast(FetchGateReceiver.ACTION_PAUSE, "backfill")
val data = sendGateBroadcast(FetchGateReceiver.ACTION_QUERY, scope = null)
assertEquals("paused=[backfill]", data)
assertTrue(DebugFetchGate.isPaused(FetchScope.BACKFILL))
assertFalse(DebugFetchGate.isPaused(FetchScope.PREFETCH))
}
@Test
fun aBackfillPausedGateDefersTheBackfillWorkerWithoutResolvingTheBackfiller() = runBlocking<Unit> {
sendGateBroadcast(FetchGateReceiver.ACTION_PAUSE, "backfill")
val lazyBackfiller = mockk<Lazy<MailBackfiller>>()
val worker = TestListenableWorkerBuilder<BackfillWorker>(context)
.setWorkerFactory(backfillWorkerFactory(lazyBackfiller, unlockedGuard()))
.build()
val result = withTimeout(TIMEOUT_MS) { worker.doWork() }
assertEquals(Result.retry(), result)
// The gate deferred BEFORE any DB-backed work — the Lazy was never resolved.
verify(exactly = 0) { lazyBackfiller.get() }
}
@Test
fun aPrefetchOnlyPauseLeavesTheBackfillWorkerRunning() = runBlocking<Unit> {
// The worker gate honours BACKFILL only; pausing PREFETCH must NOT defer history paging — the
// on-device analogue of "on-demand open and header sync stay live while prefetch is paused".
sendGateBroadcast(FetchGateReceiver.ACTION_PAUSE, "prefetch")
val backfiller = mockk<MailBackfiller> { coEvery { runBackfill(any()) } returns false }
val lazyBackfiller = mockk<Lazy<MailBackfiller>> { every { get() } returns backfiller }
val worker = TestListenableWorkerBuilder<BackfillWorker>(context)
.setWorkerFactory(backfillWorkerFactory(lazyBackfiller, unlockedGuard()))
.build()
val result = withTimeout(TIMEOUT_MS) { worker.doWork() }
assertEquals(Result.success(), result)
verify { lazyBackfiller.get() }
}
/**
* Sends the [FetchGateReceiver.ACTION] broadcast to the receiver by explicit component (mirroring
* `am broadcast -n`), ordered, and returns the result data the receiver set (the harness read-back).
*/
private fun sendGateBroadcast(action: String, scope: String?): String {
val latch = CountDownLatch(1)
val readBack = arrayOfNulls<String>(1)
val intent = Intent(FetchGateReceiver.ACTION).apply {
component = ComponentName(context, FetchGateReceiver::class.java)
putExtra(FetchGateReceiver.EXTRA_ACTION, action)
if (scope != null) putExtra(FetchGateReceiver.EXTRA_SCOPE, scope)
}
context.sendOrderedBroadcast(
intent,
null,
object : BroadcastReceiver() {
override fun onReceive(c: Context, i: Intent) {
readBack[0] = resultData
latch.countDown()
}
},
null,
0,
null,
null,
)
assertTrue("gate broadcast timed out", latch.await(TIMEOUT_MS, TimeUnit.MILLISECONDS))
return requireNotNull(readBack[0]) { "receiver set no result data" }
}
/** A real [EncryptedCacheGuard] reporting UNLOCKED (app-lock off) — so only the gate can defer. */
private fun unlockedGuard(): EncryptedCacheGuard {
val settingsRepository = mockk<SettingsRepository>()
every { settingsRepository.settings } returns flowOf(AppSettings(appLock = false, encryptCache = true))
return EncryptedCacheGuard(settingsRepository, session)
}
private fun backfillWorkerFactory(lazyBackfiller: Lazy<MailBackfiller>, cacheGuard: EncryptedCacheGuard) =
object : WorkerFactory() {
override fun createWorker(
appContext: Context,
workerClassName: String,
workerParameters: WorkerParameters,
) = BackfillWorker(appContext, workerParameters, lazyBackfiller, cacheGuard)
}
private companion object {
const val TIMEOUT_MS = 5_000L
}
}
@@ -0,0 +1,109 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.di
import android.content.Context
import android.content.ContextWrapper
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import io.mockk.Runs
import io.mockk.coEvery
import io.mockk.every
import io.mockk.just
import io.mockk.mockk
import io.mockk.mockkObject
import io.mockk.unmockkAll
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.AccountDataMigrator
import org.libremail.data.local.DatabaseEncryption
import org.libremail.data.local.DatabaseFiles
import org.libremail.data.local.DatabaseProvisioner
import org.libremail.data.security.DatabaseKeyStore
import org.libremail.data.settings.AppSettings
import org.libremail.data.settings.SettingsRepository
import java.io.File
/**
* Pins the fail-closed contract's ONE resilience exception (issue #359): the plaintext account store is
* never encrypted and never uses SQLCipher, so a cache-encryption native-load failure — which the
* provisioner surfaces as `CacheEncryptionUnavailableException` — must NOT brick it. If it did, the app
* couldn't read accounts to render the encryption error gate or assemble the PII-free problem report.
*
* Mirrors [DatabaseModuleInstrumentedTest]'s style: MockK collaborators, a real [ContextWrapper] (never
* `mockk<Context>()`, which trips an ART parameter-annotation mismatch on API 31/32), and a real
* [DatabaseProvisioner] whose encryption gate is forced to fail via a spied [DatabaseEncryption].
*/
@RunWith(AndroidJUnit4::class)
class AccountDatabaseModuleInstrumentedTest {
private val appContext = ApplicationProvider.getApplicationContext<Context>()
private val cacheDbName = "accountmodule_cache_test.db"
private val accountsDbName = "accountmodule_accounts_test.db"
private val cacheFile: File get() = appContext.getDatabasePath(cacheDbName)
private val accountsFile: File get() = appContext.getDatabasePath(accountsDbName)
// 64 hex chars == a 32-byte SQLCipher passphrase.
private val passphrase = "0123456789abcdef".repeat(4)
private val keyStore = mockk<DatabaseKeyStore>()
private val settingsRepository = mockk<SettingsRepository>()
private val migrator = mockk<AccountDataMigrator>()
// Route the provisioner's cache lookup and Room's account-store lookup to this test's private files,
// never the app's real databases.
private val context: Context = object : ContextWrapper(appContext) {
override fun getDatabasePath(name: String): File = when (name) {
DatabaseFiles.NAME -> cacheFile
DatabaseFiles.ACCOUNTS_NAME -> accountsFile
else -> super.getDatabasePath(name)
}
}
@Before
fun setUp() {
clean()
coEvery { keyStore.isClearPending() } returns false
coEvery { keyStore.resolvePassphrase(any()) } returns passphrase
coEvery { migrator.migrateIfNeeded() } just Runs
}
@After
fun tearDown() {
unmockkAll()
clean()
}
private fun clean() {
listOf(cacheDbName, accountsDbName).forEach { name ->
appContext.deleteDatabase(name)
appContext.getDatabasePath(name).parentFile?.listFiles { f -> f.name.startsWith(name) }
?.forEach { it.delete() }
}
}
private fun provisioner() = DatabaseProvisioner(context, keyStore, settingsRepository, migrator, Dispatchers.IO)
@Test
fun accountStoreStillOpensWhenTheCacheEncryptionLibraryFailsToLoad() = runBlocking<Unit> {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = true, appLock = false))
mockkObject(DatabaseEncryption) // spy: real impls run except the forced failure below
// Fault injection: the encrypted-cache gate can't load SQLCipher, so prepareCache() fails closed
// with CacheEncryptionUnavailableException — the exact condition provideAccountDatabase tolerates.
every { DatabaseEncryption.ensureNativeLibraryLoaded() } throws
UnsatisfiedLinkError("dlopen failed: libsqlcipher.so is not loadable")
val database = AccountDatabaseModule.provideAccountDatabase(context, provisioner())
try {
// The plaintext account store opens and a query succeeds despite the cache-encryption failure.
assertEquals(emptyList<Any>(), database.accountDao().getAll())
} finally {
database.close()
}
}
}
@@ -0,0 +1,191 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.di
import android.content.Context
import android.content.ContextWrapper
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import io.mockk.Runs
import io.mockk.coEvery
import io.mockk.every
import io.mockk.just
import io.mockk.mockk
import io.mockk.mockkObject
import io.mockk.unmockkAll
import io.mockk.verify
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertThrows
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.local.AccountDataMigrator
import org.libremail.data.local.DatabaseEncryption
import org.libremail.data.local.DatabaseFiles
import org.libremail.data.local.DatabaseProvisioner
import org.libremail.data.local.LibreMailDatabase
import org.libremail.data.local.entity.MessageEntity
import org.libremail.data.security.DatabaseKeyStore
import org.libremail.data.settings.AppSettings
import org.libremail.data.settings.SettingsRepository
import java.io.File
/**
* Pins the "native lib loaded before keyed open" invariant (592a797) at the OPEN site — issue #220's
* follow-up to #208/#210. `DatabaseProvisionerTest` (JVM, mocked) and `DatabaseProvisionerInstrumentedTest`
* (instrumented, real SQLCipher) both pin that [DatabaseProvisioner.prepareCache] calls
* `DatabaseEncryption.ensureNativeLibraryLoaded()` for the encrypted branch, but neither exercises
* [DatabaseModule.provideDatabase] itself: the instrumented one opens through a hand-rolled
* `SupportOpenHelperFactory`, bypassing the branch that actually picks between it and
* `FrameworkSQLiteOpenHelperFactory` based on what [DatabaseProvisioner] reports. A regression that
* breaks THAT wiring — swaps the branches, or stops gating the open on
* [DatabaseProvisioner.prepareCache] at all — would slip through both existing guards.
*
* These tests call [DatabaseModule.provideDatabase] directly (a plain function on the `object`, no Hilt
* graph needed) and drive the first real open through its own
* [org.libremail.data.local.DeferredOpenHelperFactory] lambda, mirroring the lane-3 instrumented style:
* MockK-mocked collaborators, a real [ContextWrapper] (never `mockk<Context>()`, which trips an ART
* parameter-annotation mismatch on API 31/32), and real SQLCipher for the encrypted path.
*/
@RunWith(AndroidJUnit4::class)
class DatabaseModuleInstrumentedTest {
private val appContext = ApplicationProvider.getApplicationContext<Context>()
private val dbName = "database_module_instrumented_test.db"
private val dbFile: File get() = appContext.getDatabasePath(dbName)
// 64 hex chars == a 32-byte SQLCipher passphrase, matching DatabaseKeyStore's format.
private val passphrase = "0123456789abcdef".repeat(4)
private val keyStore = mockk<DatabaseKeyStore>()
private val settingsRepository = mockk<SettingsRepository>()
private val migrator = mockk<AccountDataMigrator>()
// provideDatabase hardcodes DatabaseFiles.NAME as the Room db name, exactly like DatabaseProvisioner
// does internally when it resolves the cache file — so redirecting getDatabasePath for that one name
// routes BOTH the provisioner's file checks and Room's own open through this test's private file,
// never the app's real cache.
private val context: Context = object : ContextWrapper(appContext) {
override fun getDatabasePath(name: String): File =
if (name == DatabaseFiles.NAME) dbFile else super.getDatabasePath(name)
}
@Before
fun setUp() {
clean()
coEvery { keyStore.isClearPending() } returns false
coEvery { keyStore.resolvePassphrase(any()) } returns passphrase
coEvery { migrator.migrateIfNeeded() } just Runs
}
@After
fun tearDown() {
unmockkAll()
clean()
}
private fun clean() {
appContext.deleteDatabase(dbName)
dbFile.parentFile?.listFiles { f -> f.name.startsWith(dbName) }?.forEach { it.delete() }
}
private fun provisioner() = DatabaseProvisioner(context, keyStore, settingsRepository, migrator, Dispatchers.IO)
/**
* Builds a genuinely plaintext cache with one row, then converts it to real SQLCipher ciphertext —
* the steady-state shape an already-encrypted install has at the next cold start, where
* `ensureEncrypted` has nothing left to convert (592a797's exact failure mode: the load can't ride
* on a conversion that no-ops).
*/
private fun seedEncryptedCacheWithOneRow() {
Room.databaseBuilder(appContext, LibreMailDatabase::class.java, dbName).build().apply {
runBlocking { messageDao().insertNew(listOf(message("acct:1"))) }
close()
}
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
}
private fun message(id: String) = MessageEntity(
id = id,
accountId = "acct",
sender = "Ada",
senderEmail = "ada@example.org",
subject = "Hi",
snippet = "",
body = "",
timestampMillis = 1_000L,
isRead = false,
isStarred = false,
)
@Test
fun encryptedBranchLoadsNativeLibraryBeforeTheKeyedOpenSucceeds() = runBlocking<Unit> {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = true, appLock = false))
seedEncryptedCacheWithOneRow()
assertTrue("precondition: the cache is already encrypted", DatabaseEncryption.isEncrypted(dbFile))
mockkObject(DatabaseEncryption) // spy: real implementations still run
val database = DatabaseModule.provideDatabase(context, provisioner())
try {
// The first real open, driven by provideDatabase's own DeferredOpenHelperFactory lambda and
// CacheOpenMode branch — NOT a hand-rolled SupportOpenHelperFactory bypass. If that branch
// ever opened this genuinely-encrypted file with the plaintext framework helper instead of
// SQLCipher's, this would throw (a plaintext driver can't parse SQLCipher ciphertext) rather
// than return the seeded row.
assertEquals(listOf("acct:1"), database.messageDao().observeSummaries().first().map { it.id })
} finally {
database.close()
}
// Regression guard (592a797), pinned at the open site: a steady-state encrypted start converts
// nothing in ensureEncrypted, so provideDatabase's encrypted branch must itself have loaded
// SQLCipher's native library before the keyed open above could succeed.
verify(exactly = 1) { DatabaseEncryption.ensureNativeLibraryLoaded() }
}
@Test
fun plaintextBranchNeverTouchesTheNativeLibrary() = runBlocking<Unit> {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = false, appLock = false))
mockkObject(DatabaseEncryption)
val database = DatabaseModule.provideDatabase(context, provisioner())
try {
database.messageDao().insertNew(listOf(message("acct:1")))
assertEquals(listOf("acct:1"), database.messageDao().observeSummaries().first().map { it.id })
} finally {
database.close()
}
// The counterpart to the guard above: the plaintext branch must never construct or load
// SQLCipher, so a regression that swapped the branch mapping would show up here too.
verify(exactly = 0) { DatabaseEncryption.ensureNativeLibraryLoaded() }
}
@Test
fun encryptedOpenNeverSucceedsIfTheNativeLibraryLoadFails() {
every { settingsRepository.settings } returns flowOf(AppSettings(encryptCache = true, appLock = false))
seedEncryptedCacheWithOneRow()
mockkObject(DatabaseEncryption)
// Fault injection: if provideDatabase's encrypted branch ever stopped gating the keyed open on
// this load call, the open below would still succeed despite the injected failure. Failing here
// instead pins that the open is causally downstream of the load, not just usually preceded by it
// — the "wired without a preceding load" regression issue #220 calls out.
every { DatabaseEncryption.ensureNativeLibraryLoaded() } throws IllegalStateException("boom")
val database = DatabaseModule.provideDatabase(context, provisioner())
try {
assertThrows(Throwable::class.java) {
runBlocking { database.messageDao().observeSummaries().first() }
}
} finally {
runCatching { database.close() }
}
}
}
@@ -0,0 +1,87 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.push
import android.app.Notification
import android.app.Service
import android.content.Context
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertSame
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.data.sync.PushMode
/**
* On-device coverage of the #354 dataSync-FGS degrade path that [IdleService.onStartCommand] routes
* through [IdleForegroundStarter]. When a foreground start is rejected — the runtime-cap
* `ForegroundServiceStartNotAllowedException`, surfaced as its [IllegalStateException] supertype — the
* seam must catch it, skip IDLE watching, and degrade to periodic sync plus the degraded
* ("instant delivery paused") notification, never propagating. This drives the same decision seam the
* service uses and builds the real degraded notification with a real application `Context` (a
* `ContextWrapper`, never a mocked `Context`), mirroring `PushStatusNotificationInstrumentedTest`; it
* stands up no foreground service, Hilt graph, or network, so it is deterministic — and unlike a JVM
* unit test it exercises the real `Notification` build (the unit-test `android.jar`'s
* `NotificationCompat` is a no-op stub).
*/
@RunWith(AndroidJUnit4::class)
class IdleServiceForegroundStartInstrumentedTest {
private val context = ApplicationProvider.getApplicationContext<Context>()
@Test
fun rejectedForegroundStart_degradesToPeriodicSyncWithPausedNotification_andSkipsWatching() {
val rejection = IllegalStateException(
"Time limit already exhausted for foreground service type dataSync",
)
var watchingStarted = false
var periodicSyncScheduled = false
var degradedNotification: Notification? = null
val result = IdleForegroundStarter.startForegroundOrDegrade(
capActive = false,
enterForeground = { throw rejection },
onStarted = { watchingStarted = true },
onDegraded = { cause ->
assertSame("the runtime-cap rejection must reach the degrade path", rejection, cause)
// Mirror IdleService.degradeToPeriodicSync on a real Context: (re)assert periodic sync and
// build the degraded status notification the service would post.
periodicSyncScheduled = true
PushStatusNotification.ensureChannel(context)
degradedNotification = PushStatusNotification.build(context, PushMode.POLLING, timedOut = true)
},
)
assertEquals(Service.START_NOT_STICKY, result)
assertFalse("a rejected dataSync FGS start must not begin IDLE watching", watchingStarted)
assertTrue("the degrade path must (re)assert the 15-minute periodic sync fallback", periodicSyncScheduled)
val notification = requireNotNull(degradedNotification) { "the degrade path must build a status notification" }
assertEquals(
"the degraded notification must show the instant-delivery-paused text",
context.getString(R.string.notif_push_status_text_timed_out),
notification.extras.getCharSequence(Notification.EXTRA_TEXT).toString(),
)
}
@Test
fun activeCapWindow_skipsForegroundStartAttempt_andStillDegrades() {
var enterForegroundAttempted = false
var watchingStarted = false
var degraded = false
val result = IdleForegroundStarter.startForegroundOrDegrade(
capActive = true,
enterForeground = { enterForegroundAttempted = true },
onStarted = { watchingStarted = true },
onDegraded = { degraded = true },
)
assertEquals(Service.START_NOT_STICKY, result)
assertFalse("must not attempt a dataSync FGS start while still inside the cap window", enterForegroundAttempted)
assertFalse(watchingStarted)
assertTrue("must fall back to periodic sync while capped", degraded)
}
}
@@ -0,0 +1,81 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.push
import android.app.Notification
import android.app.NotificationManager
import android.content.Context
import androidx.core.app.NotificationManagerCompat
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.data.sync.PushMode
/**
* On-device coverage of [PushStatusNotification] — the foreground-notification logic [IdleService]
* delegates to (issue #257). `NotificationChannel`/`NotificationCompat` are no-op stubs in the
* unit-test `android.jar`, so this is the only place the real channel importance and the push-mode →
* text branch can be asserted. It uses the real application `Context` (a `ContextWrapper`, never a
* mocked `Context`), mirroring `BatteryOptimizationManagerIntentTest`, and touches no service
* lifecycle, Hilt graph, or network — so it is deterministic and side-effect-free beyond registering a
* low-importance notification channel.
*/
@RunWith(AndroidJUnit4::class)
class PushStatusNotificationInstrumentedTest {
private val context = ApplicationProvider.getApplicationContext<Context>()
@Test
fun ensureChannel_registersALowImportancePushChannel() {
PushStatusNotification.ensureChannel(context)
val channel = requireNotNull(
NotificationManagerCompat.from(context).getNotificationChannel(PushStatusNotification.CHANNEL_ID),
) { "push status channel must be registered" }
assertEquals(NotificationManager.IMPORTANCE_LOW, channel.importance)
}
@Test
fun build_inIdleMode_saysConnectedForInstantDeliveryAndIsAnOngoingServiceNotification() {
val notification = PushStatusNotification.build(context, PushMode.IDLE)
assertEquals(
context.getString(R.string.notif_push_status_title),
notification.extras.getCharSequence(Notification.EXTRA_TITLE).toString(),
)
assertEquals(
context.getString(R.string.notif_push_status_text),
notification.extras.getCharSequence(Notification.EXTRA_TEXT).toString(),
)
assertTrue(
"status notification must be ongoing",
(notification.flags and Notification.FLAG_ONGOING_EVENT) != 0,
)
assertEquals(Notification.CATEGORY_SERVICE, notification.category)
}
@Test
fun build_inPollingMode_saysLowBatteryFallback() {
val notification = PushStatusNotification.build(context, PushMode.POLLING)
assertEquals(
context.getString(R.string.notif_push_status_text_low_battery),
notification.extras.getCharSequence(Notification.EXTRA_TEXT).toString(),
)
}
@Test
fun build_afterDataSyncFgsTimeout_saysInstantDeliveryPaused() {
// The dataSync FGS runtime-cap fallback (#302): the timed-out text takes precedence over the
// mode the service was in when the platform fired onTimeout (still IDLE at that point).
val notification = PushStatusNotification.build(context, PushMode.IDLE, timedOut = true)
assertEquals(
context.getString(R.string.notif_push_status_text_timed_out),
notification.extras.getCharSequence(Notification.EXTRA_TEXT).toString(),
)
}
}
@@ -0,0 +1,95 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.reporting
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.data.security.KeystoreCrypto
import java.io.File
/**
* On-device proof for issue #369: with at-rest encryption ON, [ReportStore] persists a report as real
* Android Keystore ciphertext — no report content in plaintext on disk — and reads it back intact;
* with it OFF the file stays plaintext JSON. This closes the gap between the JVM-tested ReportStore
* branching (which fakes the cipher) and the device-only [KeystoreCrypto] the branching drives in
* production, using the same non-auth master key that lets a crash-while-locked report still be sealed.
*/
@RunWith(AndroidJUnit4::class)
class ReportStoreEncryptionInstrumentedTest {
private val context =
InstrumentationRegistry.getInstrumentation().targetContext.applicationContext
private val dir = File(context.cacheDir, "report-encryption-test")
@Before
fun setUp() {
dir.deleteRecursively()
}
@After
fun tearDown() {
dir.deleteRecursively()
}
@Test
fun encryptedReportIsCiphertextOnDiskAndReadsBack() {
val store = store(enabled = true)
store.save(report("enc"))
val raw = File(dir, "enc.json").readText()
// The distinctive plaintext token must NOT be on disk — the report is Keystore-sealed at rest.
assertFalse("report content must not be persisted in plaintext", raw.contains(SENTINEL))
assertFalse("a sealed report is not plaintext JSON", raw.startsWith("{"))
// A fresh store over the same directory (same master key) unseals and reads it back intact.
assertEquals(SENTINEL, store(enabled = true).find("enc")?.logs?.single())
}
@Test
fun plaintextReportWhenEncryptionOff() {
val store = store(enabled = false)
store.save(report("plain"))
val raw = File(dir, "plain.json").readText()
assertTrue("with encryption off the report stays plaintext JSON", raw.contains(SENTINEL))
assertTrue(raw.startsWith("{"))
}
private fun store(enabled: Boolean): ReportStore {
val crypto = KeystoreCrypto()
val encryption = object : ReportEncryption {
override fun enabled(): Boolean = enabled
override fun encrypt(plaintext: String): String = crypto.encrypt(plaintext)
override fun decrypt(encoded: String): String = crypto.decrypt(encoded)
}
return ReportStore(dir, CoroutineScope(Dispatchers.Unconfined), encryption)
}
private fun report(id: String) = DebugReport(
id = id,
createdAtMillis = 1_000L,
kind = ReportKind.CRASH,
appVersionName = "1.0",
appVersionCode = 1L,
androidRelease = "14",
androidSdkInt = 34,
deviceManufacturer = "Test",
deviceModel = "Model",
stackTrace = SENTINEL,
settings = emptyMap(),
logs = listOf(SENTINEL),
)
private companion object {
const val SENTINEL = "SENTINEL-PLAINTEXT-TOKEN-369"
}
}
@@ -62,6 +62,11 @@ class FakeAccountRepository(
accountsFlow.value = accountsFlow.value.filterNot { it.id == id }
}
override suspend fun reorderAccounts(orderedIds: List<String>) {
val byId = accountsFlow.value.associateBy { it.id }
accountsFlow.value = orderedIds.mapNotNull { byId[it] }
}
override suspend fun resetBackfillProgress(accountId: String?) = Unit
}
@@ -76,6 +81,12 @@ class FakeMailRepository(
private val attachments: List<Attachment> = emptyList(),
private val downloadedParts: Set<Int> = emptySet(),
private val unreadCounts: List<UnreadCount> = emptyList(),
drafts: List<Draft> = emptyList(),
outbox: List<OutboxMessage> = emptyList(),
// When set, every paged query returns this instead of a static page over [messages]. Lets a UI test
// drive an explicit LoadState (e.g. refresh = Loading) through collectAsLazyPagingItems to exercise
// the empty-state gate (issue #219).
private val pagedOverride: Flow<PagingData<Message>>? = null,
) : MailRepository {
val sentMessages = mutableListOf<OutgoingMessage>()
@@ -87,15 +98,35 @@ class FakeMailRepository(
val expungedIds = mutableListOf<List<String>>()
val movedToFolder = mutableListOf<Pair<List<String>, String>>()
val replyDrafts = mutableListOf<Pair<String, ReplyMode>>()
val canceledOutboxIds = mutableListOf<String>()
var retryOutboxCount = 0
private set
override fun observeFolderMessages(accountId: String, folder: String): Flow<List<Message>> =
flowOf(messages.filter { it.accountId == accountId && it.folder == folder })
override fun observeUnifiedFolderMessages(folder: String): Flow<List<Message>> =
flowOf(messages.filter { it.folder == folder })
// Backed by mutable state so a delete/cancel is reflected in the observed list, letting UI tests
// assert the row actually disappears (mirroring the real DB-backed repository's reactivity).
private val draftsFlow = MutableStateFlow(drafts)
private val outboxFlow = MutableStateFlow(outbox)
override fun pagedUnifiedFolderMessages(folder: String): Flow<PagingData<Message>> =
flowOf(PagingData.from(messages.filter { it.folder == folder && it.inInbox }))
pagedOverride ?: flowOf(PagingData.from(messages.filter { it.folder == folder && it.inInbox }))
override fun pagedFolderMessages(accountId: String, folder: String): Flow<PagingData<Message>> =
pagedOverride ?: flowOf(
PagingData.from(messages.filter { it.accountId == accountId && it.folder == folder && it.inInbox }),
)
override fun pagedUnifiedSearchMessages(folder: String, query: String): Flow<PagingData<Message>> =
pagedOverride ?: flowOf(PagingData.from(messages.filter { it.folder == folder && it.matchesSearch(query) }))
override fun pagedFolderSearchMessages(
accountId: String,
folder: String,
query: String,
): Flow<PagingData<Message>> = pagedOverride ?: flowOf(
PagingData.from(
messages.filter { it.accountId == accountId && it.folder == folder && it.matchesSearch(query) },
),
)
override fun observeFolders(accountId: String): Flow<List<Folder>> = flowOf(
folders.filter {
@@ -169,9 +200,9 @@ class FakeMailRepository(
return sendResult
}
override fun observeDrafts(): Flow<List<Draft>> = flowOf(emptyList())
override fun observeDrafts(): Flow<List<Draft>> = draftsFlow
override suspend fun getDraft(id: String): Draft? = null
override suspend fun getDraft(id: String): Draft? = draftsFlow.value.firstOrNull { it.id == id }
override suspend fun saveDraft(draft: Draft) {
savedDrafts += draft
@@ -179,13 +210,19 @@ class FakeMailRepository(
override suspend fun deleteDraft(id: String) {
deletedDraftIds += id
draftsFlow.value = draftsFlow.value.filterNot { it.id == id }
}
override fun observeOutbox(): Flow<List<OutboxMessage>> = flowOf(emptyList())
override fun observeOutbox(): Flow<List<OutboxMessage>> = outboxFlow
override suspend fun cancelOutboxMessage(id: String) {}
override suspend fun cancelOutboxMessage(id: String) {
canceledOutboxIds += id
outboxFlow.value = outboxFlow.value.filterNot { it.id == id }
}
override suspend fun retryOutbox() {}
override suspend fun retryOutbox() {
retryOutboxCount++
}
override suspend fun searchServer(query: String, accountId: String?, folder: String) {}
@@ -198,3 +235,9 @@ class FakeMailSyncer : Syncer {
override suspend fun syncAccount(accountId: String): Result<Int> = Result.success(0)
override suspend fun syncFolder(accountId: String, folder: String): Result<Int> = Result.success(0)
}
/** Mirrors the paged-search DAO queries' columns so the fake's search pagers filter like production. */
private fun Message.matchesSearch(query: String): Boolean = sender.contains(query, ignoreCase = true) ||
senderEmail.contains(query, ignoreCase = true) ||
subject.contains(query, ignoreCase = true) ||
snippet.contains(query, ignoreCase = true)
@@ -0,0 +1,122 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.accountsetup
import android.app.Activity
import android.app.Instrumentation
import android.content.Context
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.compose.ui.test.performScrollTo
import androidx.test.espresso.intent.Intents
import androidx.test.espresso.intent.matcher.IntentMatchers.hasComponent
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import net.openid.appauth.AuthorizationManagementActivity
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.auth.OutlookAuthManager
import org.libremail.domain.model.MailProvider
import org.libremail.ui.FakeAccountRepository
import org.libremail.ui.theme.LibreMailTheme
/**
* End-to-end UI test for the account-vendor picker used by onboarding and "Add account". Drives the
* real [AccountPickerScreen] + [AccountSetupViewModel]: every setup choice is listed, tapping an
* app-password vendor routes to [onPickProvider] with that [MailProvider], and tapping "Other"
* routes to [onManualSetup]. Tapping Outlook is covered separately, below, guarded by
* Espresso-Intents so no real browser ever opens.
*/
@RunWith(AndroidJUnit4::class)
class AccountPickerScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val context: Context =
InstrumentationRegistry.getInstrumentation().targetContext.applicationContext
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun setContent(onPickProvider: (MailProvider) -> Unit = {}, onManualSetup: () -> Unit = {}) {
val viewModel = AccountSetupViewModel(OutlookAuthManager(context), FakeAccountRepository())
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
AccountPickerScreen(
onBack = {},
onAccountAdded = {},
onPickProvider = onPickProvider,
onManualSetup = onManualSetup,
viewModel = viewModel,
)
}
}
}
@Test
fun allSetupChoices_areListed() {
setContent()
composeTestRule.onNodeWithText(string(R.string.account_setup_outlook)).performScrollTo().assertIsDisplayed()
MailProvider.entries.forEach { provider ->
composeTestRule.onNodeWithText(provider.displayName).performScrollTo().assertIsDisplayed()
}
composeTestRule.onNodeWithText(string(R.string.account_setup_other)).performScrollTo().assertIsDisplayed()
}
@Test
fun tappingAppPasswordProvider_invokesOnPickProvider() {
var picked: MailProvider? = null
setContent(onPickProvider = { picked = it })
composeTestRule.onNodeWithText(MailProvider.GMAIL.displayName).performScrollTo().performClick()
composeTestRule.waitUntil(5_000) { picked == MailProvider.GMAIL }
}
@Test
fun tappingOther_invokesOnManualSetup() {
var manualRequested = false
setContent(onManualSetup = { manualRequested = true })
composeTestRule.onNodeWithText(string(R.string.account_setup_other)).performScrollTo().performClick()
composeTestRule.waitUntil(5_000) { manualRequested }
}
/**
* Tapping Outlook must fire the browser-launch intent for Microsoft sign-in (#276).
* [OutlookAuthManager.createAuthIntent] delegates to AppAuth, whose
* `AuthorizationService.getAuthorizationRequestIntent()` never returns a bare browser intent: it
* always wraps it in an intent targeting AppAuth's own [AuthorizationManagementActivity], which
* only starts the actual browser/Custom Tab once it resumes. That component name is therefore the
* one characteristic of the launch that's both guaranteed (every Outlook tap goes through it) and
* stable (it doesn't depend on which browser, if any, is installed on the test device) — matching
* it both confirms the tap started the AppAuth authorization flow and, by stubbing a canceled
* result, stops [AuthorizationManagementActivity] from ever resuming and opening a real browser.
* The OAuth redirect, token exchange, and account creation a real sign-in would trigger next are
* deliberately out of scope here (see #276).
*/
@Test
fun tappingOutlook_launchesTheAppAuthBrowserIntent() {
setContent()
Intents.init()
try {
Intents.intending(hasComponent(AuthorizationManagementActivity::class.java.name))
.respondWith(Instrumentation.ActivityResult(Activity.RESULT_CANCELED, null))
composeTestRule.onNodeWithText(string(R.string.account_setup_outlook))
.performScrollTo()
.performClick()
Intents.intended(hasComponent(AuthorizationManagementActivity::class.java.name))
} finally {
Intents.release()
}
}
}
@@ -0,0 +1,111 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.accountsetup
import android.app.Activity
import android.app.Instrumentation
import android.content.Intent
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.compose.ui.test.performScrollTo
import androidx.compose.ui.test.performTextInput
import androidx.lifecycle.SavedStateHandle
import androidx.test.espresso.intent.Intents
import androidx.test.espresso.intent.matcher.IntentMatchers.hasAction
import androidx.test.espresso.intent.matcher.IntentMatchers.hasData
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.hamcrest.CoreMatchers.allOf
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.domain.model.MailProvider
import org.libremail.ui.FakeAccountRepository
import org.libremail.ui.navigation.Routes
import org.libremail.ui.theme.LibreMailTheme
/**
* End-to-end UI test for the guided app-password setup screen (#29). Drives the real
* [AppPasswordSetupScreen] + [AppPasswordViewModel] over a [FakeAccountRepository] for the preset
* Gmail vendor: the provider-specific chrome renders, entering an email + app password and tapping
* "Test & add" persists through the repository and reports the new account id, and tapping the
* "create an app password" help link fires the browser intent. That launch is asserted with
* Espresso-Intents (mirroring `AccountPickerScreenTest`'s Outlook test), so no real browser opens.
*/
@RunWith(AndroidJUnit4::class)
class AppPasswordSetupScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val provider = MailProvider.GMAIL
private fun string(resId: Int, vararg args: Any) = composeTestRule.activity.getString(resId, *args)
private fun setContent(
repository: FakeAccountRepository = FakeAccountRepository(),
onAccountAdded: (String) -> Unit = {},
) {
val viewModel = AppPasswordViewModel(
SavedStateHandle(mapOf(Routes.APP_PASSWORD_ARG_PROVIDER to provider.key)),
repository,
)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
AppPasswordSetupScreen(onBack = {}, onAccountAdded = onAccountAdded, viewModel = viewModel)
}
}
}
@Test
fun rendersProviderTitle_andCredentialFields() {
setContent()
composeTestRule.onNodeWithText(string(R.string.app_password_title, provider.displayName)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.app_password_email)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.app_password_field)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.app_password_test_and_add)).performScrollTo().assertIsDisplayed()
}
@Test
fun enteringCredentials_andTappingTestAndAdd_addsAccountAndReportsId() {
val repository = FakeAccountRepository()
var addedId: String? = null
setContent(repository = repository, onAccountAdded = { addedId = it })
composeTestRule.onNodeWithText(string(R.string.app_password_email)).performTextInput("me@gmail.com")
composeTestRule.onNodeWithText(string(R.string.app_password_field)).performTextInput("app-pw-1234")
composeTestRule.onNodeWithText(string(R.string.app_password_test_and_add)).performScrollTo().performClick()
// A successful add persists via the repository and hands the new account id to the caller.
composeTestRule.waitUntil(5_000) {
repository.addedAccount?.email == "me@gmail.com" && addedId == "imap:me@gmail.com"
}
}
/**
* Tapping the "create an app password" link opens the provider's help page via
* [androidx.compose.ui.platform.UriHandler], which starts an `ACTION_VIEW` intent. Stubbing that
* intent both proves the tap launched it and stops a real browser from opening on the device.
*/
@Test
fun tappingCreateAppPasswordPage_launchesBrowserIntentToHelpUrl() {
setContent()
Intents.init()
try {
Intents.intending(hasAction(Intent.ACTION_VIEW))
.respondWith(Instrumentation.ActivityResult(Activity.RESULT_CANCELED, null))
composeTestRule.onNodeWithText(string(R.string.app_password_open_page, provider.displayName))
.performScrollTo()
.performClick()
Intents.intended(allOf(hasAction(Intent.ACTION_VIEW), hasData(provider.appPasswordHelpUrl)))
} finally {
Intents.release()
}
}
}
@@ -15,6 +15,7 @@ import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.compose.ui.test.performTextInput
import androidx.lifecycle.SavedStateHandle
import androidx.lifecycle.ViewModelStore
import androidx.room.Room
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
@@ -60,10 +61,20 @@ class ComposeScreenTest {
private var db: AccountDatabase? = null
// Holds the real ComposeViewModel built by hand in setContent() below, so closeDb() can clear()
// it (triggering ViewModel.onCleared()) before closing the DB.
private val viewModelStore = ViewModelStore()
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
@After
fun closeDb() {
// Clear the store (→ ViewModel.onCleared() → cancels viewModelScope) BEFORE closing the DB.
// ComposeViewModel's init block launches a viewModelScope coroutine that reads the real
// accountSettings/signature Room repositories (applySignature()); without this, that read can
// still be in flight when the DB closes, racing a SQLITE_MISUSE ("connection is closed") —
// the same class of teardown race fixed in SignaturesScreenTest/AccountSettingsScreenTest.
viewModelStore.clear()
db?.close()
}
@@ -92,6 +103,7 @@ class ComposeScreenTest {
signatureRepository = SignatureRepository(database.signatureDao()),
settingsRepository = SettingsRepository(context),
)
viewModelStore.put("compose", viewModel)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
ComposeScreen(onBack = onBack, viewModel = viewModel)
@@ -0,0 +1,90 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.compose.format
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.assertIsNotSelected
import androidx.compose.ui.test.assertIsSelected
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onNodeWithContentDescription
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Assert.assertEquals
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.ui.theme.LibreMailTheme
/**
* UI tests for the shared font-color / highlight swatch row (#78). [ColorSwatchRow] is
* presentational, so it is driven in isolation - independent of the compose formatting toolbar that
* hosts it - mirroring how `ParagraphAlignmentControlTest` exercises its control. Every swatch is a
* TalkBack-labeled, selectable button, so nodes are addressed by their content description.
*/
@RunWith(AndroidJUnit4::class)
class ColorSwatchRowTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val red = ColorSwatch(argb = 0xFFD32F2F.toInt(), label = "Red")
private val blue = ColorSwatch(argb = 0xFF1976D2.toInt(), label = "Blue")
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun setContent(selectedArgb: Int?, onSelect: (Int?) -> Unit = {}) {
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
ColorSwatchRow(swatches = listOf(red, blue), selectedArgb = selectedArgb, onSelect = onSelect)
}
}
}
@Test
fun showsNoneEntry_andEverySwatch() {
setContent(selectedArgb = null)
composeTestRule.onNodeWithContentDescription(string(R.string.format_color_none)).assertIsDisplayed()
composeTestRule.onNodeWithContentDescription(red.label).assertIsDisplayed()
composeTestRule.onNodeWithContentDescription(blue.label).assertIsDisplayed()
}
@Test
fun tappingASwatch_reportsItsArgb() {
var picked: Int? = -1
setContent(selectedArgb = null) { picked = it }
composeTestRule.onNodeWithContentDescription(blue.label).performClick()
assertEquals(blue.argb, picked)
}
@Test
fun tappingNoneEntry_reportsNull() {
var picked: Int? = red.argb
setContent(selectedArgb = red.argb) { picked = it }
composeTestRule.onNodeWithContentDescription(string(R.string.format_color_none)).performClick()
assertEquals(null, picked)
}
@Test
fun selectedSwatch_isMarkedSelected_andOthersAreNot() {
setContent(selectedArgb = red.argb)
composeTestRule.onNodeWithContentDescription(red.label).assertIsSelected()
composeTestRule.onNodeWithContentDescription(blue.label).assertIsNotSelected()
// With a color selected, the leading "no color" entry is not the selected one.
composeTestRule.onNodeWithContentDescription(string(R.string.format_color_none)).assertIsNotSelected()
}
@Test
fun noSelection_marksTheNoneEntrySelected() {
setContent(selectedArgb = null)
composeTestRule.onNodeWithContentDescription(string(R.string.format_color_none)).assertIsSelected()
composeTestRule.onNodeWithContentDescription(red.label).assertIsNotSelected()
}
}
@@ -0,0 +1,121 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.drafts
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onAllNodesWithContentDescription
import androidx.compose.ui.test.onAllNodesWithText
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Assert.assertEquals
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.domain.model.Draft
import org.libremail.ui.FakeMailRepository
import org.libremail.ui.theme.LibreMailTheme
/**
* End-to-end UI test for [DraftsScreen] + [DraftsViewModel] over an in-memory [FakeMailRepository]:
* the empty state, the subject/recipient/body rendering (with the blank-field fallbacks), opening a
* draft, and deleting one (which removes the row and records the deletion in the repository).
*/
@RunWith(AndroidJUnit4::class)
class DraftsScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun draft(id: String, to: String, subject: String, body: String) = Draft(
id = id,
accountId = "imap:a",
to = to,
cc = "",
subject = subject,
body = body,
updatedAt = 1_000L,
)
private fun setContent(repo: FakeMailRepository, onOpenDraft: (String) -> Unit = {}) {
val viewModel = DraftsViewModel(repo)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
DraftsScreen(onBack = {}, onOpenDraft = onOpenDraft, viewModel = viewModel)
}
}
}
private fun waitForText(text: String) = composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText(text).fetchSemanticsNodes().isNotEmpty()
}
@Test
fun emptyDrafts_showsEmptyState() {
setContent(FakeMailRepository())
composeTestRule.onNodeWithText(string(R.string.drafts_empty)).assertIsDisplayed()
}
@Test
fun populatedDraft_showsSubjectRecipientAndBody() {
setContent(
FakeMailRepository(
drafts = listOf(draft("d1", "alice@example.org", "Lunch plans", "See you at noon")),
),
)
waitForText("Lunch plans")
composeTestRule.onNodeWithText("Lunch plans").assertIsDisplayed()
composeTestRule.onNodeWithText("alice@example.org").assertIsDisplayed()
composeTestRule.onNodeWithText("See you at noon").assertIsDisplayed()
}
@Test
fun blankDraft_usesNoSubjectAndNoRecipientFallbacks() {
setContent(FakeMailRepository(drafts = listOf(draft("d1", to = "", subject = "", body = ""))))
waitForText(string(R.string.draft_no_subject))
composeTestRule.onNodeWithText(string(R.string.draft_no_subject)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.draft_no_recipient)).assertIsDisplayed()
}
@Test
fun tappingRow_invokesOnOpenDraft() {
var opened: String? = null
setContent(
FakeMailRepository(drafts = listOf(draft("d1", "alice@example.org", "Lunch plans", "body"))),
onOpenDraft = { opened = it },
)
waitForText("Lunch plans")
composeTestRule.onNodeWithText("Lunch plans").performClick()
composeTestRule.waitUntil(5_000) { opened == "d1" }
}
@Test
fun tappingDelete_removesRow_andRecordsDeletion() {
val repo = FakeMailRepository(
drafts = listOf(
draft("d1", "alice@example.org", "Lunch plans", "body"),
draft("d2", "bob@example.org", "Second draft", "body"),
),
)
setContent(repo)
waitForText("Lunch plans")
// Delete the first draft: the fake removes it from the observed list, so its row disappears.
composeTestRule.onAllNodesWithContentDescription(string(R.string.draft_delete))[0].performClick()
composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText("Lunch plans").fetchSemanticsNodes().isEmpty()
}
assertEquals(listOf("d1"), repo.deletedDraftIds)
composeTestRule.onNodeWithText("Second draft").assertIsDisplayed()
}
}
@@ -0,0 +1,69 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.lock
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.ui.theme.LibreMailTheme
/**
* UI tests for the app-lock gate. [LockScreen] is presentational (its biometric prompt is driven by
* the caller via [onUnlock]), so it is exercised in isolation: the locked title/body always show, an
* optional error string appears only when non-null, and the unlock button reports back.
*/
@RunWith(AndroidJUnit4::class)
class LockScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun setContent(error: String? = null, onUnlock: () -> Unit = {}) {
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
LockScreen(error = error, onUnlock = onUnlock)
}
}
}
@Test
fun showsLockedTitleBody_andUnlockButton() {
setContent()
composeTestRule.onNodeWithText(string(R.string.app_lock_title)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.app_lock_locked_body)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.app_lock_unlock)).assertIsDisplayed()
}
@Test
fun noError_hidesErrorText() {
setContent(error = null)
composeTestRule.onNodeWithText(string(R.string.app_lock_unlock_failed)).assertDoesNotExist()
}
@Test
fun error_isDisplayed() {
setContent(error = string(R.string.app_lock_unlock_failed))
composeTestRule.onNodeWithText(string(R.string.app_lock_unlock_failed)).assertIsDisplayed()
}
@Test
fun tappingUnlock_invokesOnUnlock() {
var unlocked = false
setContent(onUnlock = { unlocked = true })
composeTestRule.onNodeWithText(string(R.string.app_lock_unlock)).performClick()
composeTestRule.waitUntil(5_000) { unlocked }
}
}
@@ -14,7 +14,11 @@ import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.compose.ui.test.performTouchInput
import androidx.lifecycle.SavedStateHandle
import androidx.paging.LoadState
import androidx.paging.LoadStates
import androidx.paging.PagingData
import androidx.test.ext.junit.runners.AndroidJUnit4
import kotlinx.coroutines.flow.flowOf
import org.junit.Assert.assertEquals
import org.junit.Rule
import org.junit.Test
@@ -284,4 +288,53 @@ class MailboxScreenTest {
composeTestRule.onNodeWithContentDescription(string(R.string.message_available_offline)).assertIsDisplayed()
}
// Issue #219: on return from the reader the inbox's LazyPagingItems is cold — it presents
// itemCount == 0 with refresh == Loading before the window repopulates. The empty-state gate must
// hold "No messages yet" back through that window, so the empty state never flashes.
@Test
fun emptyState_isHidden_whileTheInboxPagerIsStillLoading() {
val loadingEmpty = flowOf(
PagingData.from(
emptyList<Message>(),
LoadStates(
refresh = LoadState.Loading,
prepend = LoadState.NotLoading(endOfPaginationReached = false),
append = LoadState.NotLoading(endOfPaginationReached = false),
),
),
)
setContent(FakeMailRepository(pagedOverride = loadingEmpty))
// Assert only the gate — no positive "screen composed" anchor. Under a never-completing
// refresh == Loading pager the compose tree never settles deterministically, so no node (not even
// the always-present FAB) is reliable to assert *present* here: it flaked as not-displayed
// (assertIsDisplayed), not-found (assertExists), and waitForText-timeout across CI runs.
// The gate under test (issue #219): the empty state stays hidden while refresh == Loading, so
// "No messages yet" never flashes on return from the reader. This is the stable, meaningful
// assertion — NoMessagesState is composed only once the pager is "settled" (refresh NotLoading
// AND append end-of-pagination reached), which this frozen Loading state never reaches, so the
// string can never appear regardless of when the tree happens to settle.
composeTestRule.onNodeWithText(string(R.string.mailbox_empty)).assertDoesNotExist()
}
// The flip side of the gate: once the pager settles (refresh done, no further page) with no rows,
// the inbox really is empty and "No messages yet" must show.
@Test
fun emptyState_isShown_onceTheInboxPagerSettlesEmpty() {
val settledEmpty = flowOf(
PagingData.from(
emptyList<Message>(),
LoadStates(
refresh = LoadState.NotLoading(endOfPaginationReached = false),
prepend = LoadState.NotLoading(endOfPaginationReached = true),
append = LoadState.NotLoading(endOfPaginationReached = true),
),
),
)
setContent(FakeMailRepository(pagedOverride = settledEmpty))
waitForText(string(R.string.mailbox_empty))
composeTestRule.onNodeWithText(string(R.string.mailbox_empty)).assertIsDisplayed()
}
}
@@ -0,0 +1,66 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.onboarding
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.ui.theme.LibreMailTheme
/**
* UI tests for the post-add onboarding prompt. [AddAnotherAccountScreen] is presentational — it just
* confirms the add and routes the two choices back to the caller — so it is driven in isolation:
* "Add another" returns to the vendor picker and "No" finishes onboarding.
*/
@RunWith(AndroidJUnit4::class)
class AddAnotherAccountScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun setContent(onAddAnother: () -> Unit = {}, onFinish: () -> Unit = {}) {
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
AddAnotherAccountScreen(onAddAnother = onAddAnother, onFinish = onFinish)
}
}
}
@Test
fun showsConfirmation_andBothChoices() {
setContent()
composeTestRule.onNodeWithText(string(R.string.onboarding_account_added_title)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.onboarding_add_another_prompt)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.onboarding_add_another_yes)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.onboarding_add_another_no)).assertIsDisplayed()
}
@Test
fun tappingAddAnother_invokesOnAddAnother() {
var addAnother = false
setContent(onAddAnother = { addAnother = true })
composeTestRule.onNodeWithText(string(R.string.onboarding_add_another_yes)).performClick()
composeTestRule.waitUntil(5_000) { addAnother }
}
@Test
fun tappingNo_invokesOnFinish() {
var finished = false
setContent(onFinish = { finished = true })
composeTestRule.onNodeWithText(string(R.string.onboarding_add_another_no)).performClick()
composeTestRule.waitUntil(5_000) { finished }
}
}
@@ -0,0 +1,113 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.outbox
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onAllNodesWithContentDescription
import androidx.compose.ui.test.onAllNodesWithText
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Assert.assertEquals
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.domain.model.OutboxMessage
import org.libremail.ui.FakeMailRepository
import org.libremail.ui.theme.LibreMailTheme
/**
* End-to-end UI test for [OutboxScreen] + [OutboxViewModel] over an in-memory [FakeMailRepository]:
* the empty state, queued-vs-failed row rendering, and the cancel/retry actions round-tripping
* through the repository (cancel removes the row; retry calls back into the repo).
*/
@RunWith(AndroidJUnit4::class)
class OutboxScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun queued(id: String, subject: String, to: String) =
OutboxMessage(id = id, to = to, subject = subject, body = "body", createdAt = 1_000L, lastError = null)
private fun failed(id: String, subject: String, to: String) =
OutboxMessage(id = id, to = to, subject = subject, body = "body", createdAt = 2_000L, lastError = "SMTP 550")
private fun setContent(repo: FakeMailRepository) {
val viewModel = OutboxViewModel(repo)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
OutboxScreen(onBack = {}, viewModel = viewModel)
}
}
}
private fun waitForText(text: String) = composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText(text).fetchSemanticsNodes().isNotEmpty()
}
@Test
fun emptyOutbox_showsEmptyState_andNoRetryAction() {
setContent(FakeMailRepository())
composeTestRule.onNodeWithText(string(R.string.outbox_empty)).assertIsDisplayed()
// Retry only appears when there is something to retry.
composeTestRule.onNodeWithText(string(R.string.outbox_retry)).assertDoesNotExist()
}
@Test
fun populatedOutbox_showsQueuedAndFailedRows_andRetryAction() {
setContent(
FakeMailRepository(
outbox = listOf(
queued("q", "Queued mail", "alice@example.org"),
failed("f", "Failed mail", "bob@example.org"),
),
),
)
waitForText("Queued mail")
composeTestRule.onNodeWithText("Queued mail").assertIsDisplayed()
composeTestRule.onNodeWithText("Failed mail").assertIsDisplayed()
// The queued row shows the "queued" status; the failed row shows the "failed" status.
composeTestRule.onNodeWithText(string(R.string.outbox_status_queued)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.outbox_status_failed)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.outbox_retry)).assertIsDisplayed()
}
@Test
fun tappingRetry_callsRepositoryRetry() {
val repo = FakeMailRepository(outbox = listOf(failed("f", "Failed mail", "bob@example.org")))
setContent(repo)
waitForText("Failed mail")
composeTestRule.onNodeWithText(string(R.string.outbox_retry)).performClick()
composeTestRule.waitUntil(5_000) { repo.retryOutboxCount == 1 }
}
@Test
fun tappingCancel_removesRow_andRecordsCancellation() {
val repo = FakeMailRepository(
outbox = listOf(
queued("q", "Queued mail", "alice@example.org"),
queued("q2", "Second mail", "carol@example.org"),
),
)
setContent(repo)
waitForText("Queued mail")
// Cancel the first row: the fake removes it from the observed outbox, so the row disappears.
composeTestRule.onAllNodesWithContentDescription(string(R.string.outbox_cancel))[0].performClick()
composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText("Queued mail").fetchSemanticsNodes().isEmpty()
}
assertEquals(listOf("q"), repo.canceledOutboxIds)
composeTestRule.onNodeWithText("Second mail").assertIsDisplayed()
}
}
@@ -11,6 +11,7 @@ import androidx.compose.ui.test.performClick
import androidx.lifecycle.SavedStateHandle
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import org.junit.Assert.assertEquals
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
@@ -18,6 +19,7 @@ import org.libremail.R
import org.libremail.data.settings.SettingsRepository
import org.libremail.domain.model.Attachment
import org.libremail.domain.model.Message
import org.libremail.domain.model.ReplyMode
import org.libremail.ui.FakeMailRepository
import org.libremail.ui.navigation.Routes
import org.libremail.ui.theme.LibreMailTheme
@@ -44,8 +46,17 @@ class ReaderScreenTest {
private fun attachment(partIndex: Int, filename: String) =
Attachment(messageId, partIndex, filename, "application/pdf", 1_000L)
/** Renders [ReaderScreen] for the fixed [message] with the given [attachments] and awaits load. */
private fun renderReader(attachments: List<Attachment>, downloadedParts: Set<Int> = emptySet()) {
/** The draft id the reader last asked to open compose on (via OpenCompose), or null. */
private var openedDraftId: String? = null
/**
* Renders [ReaderScreen] for the fixed [message] with the given [attachments], awaits load, and
* returns the backing [FakeMailRepository] so a test can assert which reply drafts were built.
*/
private fun renderReader(
attachments: List<Attachment> = emptyList(),
downloadedParts: Set<Int> = emptySet(),
): FakeMailRepository {
val context = InstrumentationRegistry.getInstrumentation().targetContext.applicationContext
val repo = FakeMailRepository(
messages = listOf(message),
@@ -59,12 +70,14 @@ class ReaderScreenTest {
)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
ReaderScreen(onBack = {}, onReply = { _, _, _ -> }, viewModel = viewModel)
ReaderScreen(onBack = {}, onOpenCompose = { openedDraftId = it }, viewModel = viewModel)
}
}
// The Reply action appears once the message loads, regardless of whether it has attachments.
composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText(attachments.first().filename).fetchSemanticsNodes().isNotEmpty()
composeTestRule.onAllNodesWithText(string(R.string.reader_reply)).fetchSemanticsNodes().isNotEmpty()
}
return repo
}
@Test
@@ -111,4 +124,39 @@ class ReaderScreenTest {
composeTestRule.onNodeWithText(seeMore(1)).assertIsDisplayed()
composeTestRule.onNodeWithText("b.pdf").assertDoesNotExist()
}
@Test
fun reader_reply_buildsQuotedReplyDraftAndOpensCompose() {
val repo = renderReader()
composeTestRule.onNodeWithText(string(R.string.reader_reply)).performClick()
composeTestRule.waitUntil(5_000) { openedDraftId != null }
// The reader routes through buildReplyDraft (quotes the original + bakes the signature),
// not a bare compose prefill (#303), and opens compose on the resulting draft.
assertEquals(listOf(messageId to ReplyMode.REPLY), repo.replyDrafts)
assertEquals("draft-$messageId", openedDraftId)
}
@Test
fun reader_forward_viaOverflow_buildsForwardDraft() {
val repo = renderReader()
composeTestRule.onNodeWithContentDescription(string(R.string.action_more)).performClick()
composeTestRule.onNodeWithText(string(R.string.action_forward)).performClick()
composeTestRule.waitUntil(5_000) { openedDraftId != null }
assertEquals(listOf(messageId to ReplyMode.FORWARD), repo.replyDrafts)
}
@Test
fun reader_replyAll_viaOverflow_buildsReplyAllDraft() {
val repo = renderReader()
composeTestRule.onNodeWithContentDescription(string(R.string.action_more)).performClick()
composeTestRule.onNodeWithText(string(R.string.action_reply_all)).performClick()
composeTestRule.waitUntil(5_000) { openedDraftId != null }
assertEquals(listOf(messageId to ReplyMode.REPLY_ALL), repo.replyDrafts)
}
}
@@ -0,0 +1,137 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.reporting
import android.content.Context
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onAllNodesWithText
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import io.mockk.every
import io.mockk.mockk
import kotlinx.coroutines.flow.flowOf
import org.junit.After
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.data.settings.SettingsRepository
import org.libremail.domain.repository.AccountRepository
import org.libremail.reporting.AppVersionProvider
import org.libremail.reporting.DebugReport
import org.libremail.reporting.DiagnosticsCollector
import org.libremail.reporting.ReportKind
import org.libremail.reporting.ReportStore
import org.libremail.reporting.RingLogBuffer
import org.libremail.ui.theme.LibreMailTheme
import java.io.File
/**
* End-to-end UI test for [ProblemReportsScreen] + [ProblemReportsViewModel] backed by a real
* file-backed [ReportStore] (temp dir) and a real [DiagnosticsCollector]: the always-present create
* button and auto-delete notice, the empty state, the crash/manual row rendering, opening a report,
* and the create-report flow which saves a manual report and immediately opens it for review.
*/
@RunWith(AndroidJUnit4::class)
class ProblemReportsScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val context: Context =
InstrumentationRegistry.getInstrumentation().targetContext.applicationContext
private val storeDir = File(context.cacheDir, "problem-reports-test")
@Before
fun setUp() {
storeDir.deleteRecursively()
}
@After
fun tearDown() {
storeDir.deleteRecursively()
}
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun report(id: String, kind: ReportKind) = DebugReport(
id = id,
createdAtMillis = 1_000L,
kind = kind,
appVersionName = "1.0",
appVersionCode = 1L,
androidRelease = "14",
androidSdkInt = 34,
deviceManufacturer = "Test",
deviceModel = "Model",
stackTrace = if (kind == ReportKind.CRASH) "boom" else null,
settings = emptyMap(),
logs = emptyList(),
)
private fun setContent(reports: List<DebugReport>, onOpenReport: (String) -> Unit = {}) {
val store = ReportStore(storeDir)
reports.forEach(store::save)
val collector = DiagnosticsCollector(
AppVersionProvider(context),
SettingsRepository(context),
mockk<AccountRepository> { every { observeAccounts() } returns flowOf(emptyList()) },
RingLogBuffer(),
)
val viewModel = ProblemReportsViewModel(store, collector)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
ProblemReportsScreen(onBack = {}, onOpenReport = onOpenReport, viewModel = viewModel)
}
}
}
private fun waitForText(text: String) = composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText(text).fetchSemanticsNodes().isNotEmpty()
}
@Test
fun noReports_showsEmptyState_withCreateButtonAndAutoDeleteNotice() {
setContent(emptyList())
composeTestRule.onNodeWithText(string(R.string.reports_empty)).assertIsDisplayed()
// The create control and the retention notice are always visible, even with no reports.
composeTestRule.onNodeWithText(string(R.string.reports_create)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.report_auto_delete_notice)).assertIsDisplayed()
}
@Test
fun storedReports_showCrashAndManualKindLabels() {
setContent(listOf(report("r-crash", ReportKind.CRASH), report("r-manual", ReportKind.MANUAL)))
waitForText(string(R.string.report_kind_crash))
composeTestRule.onNodeWithText(string(R.string.report_kind_crash)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.report_kind_manual)).assertIsDisplayed()
}
@Test
fun tappingReportRow_invokesOnOpenReport() {
var opened: String? = null
setContent(listOf(report("r-crash", ReportKind.CRASH)), onOpenReport = { opened = it })
waitForText(string(R.string.report_kind_crash))
composeTestRule.onNodeWithText(string(R.string.report_kind_crash)).performClick()
composeTestRule.waitUntil(5_000) { opened == "r-crash" }
}
@Test
fun tappingCreate_savesManualReport_andOpensItForReview() {
var opened: String? = null
setContent(emptyList(), onOpenReport = { opened = it })
composeTestRule.onNodeWithText(string(R.string.reports_create)).performClick()
// Creating a manual report emits its id so the screen opens it straight into review.
composeTestRule.waitUntil(5_000) { opened != null }
}
}
@@ -0,0 +1,156 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.reporting
import android.content.ClipboardManager
import android.content.Context
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.assertIsEnabled
import androidx.compose.ui.test.assertIsNotEnabled
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onAllNodesWithText
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.compose.ui.test.performScrollTo
import androidx.compose.ui.test.performTextInput
import androidx.lifecycle.SavedStateHandle
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import io.mockk.every
import io.mockk.mockk
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.reporting.DebugReport
import org.libremail.reporting.ReportKind
import org.libremail.reporting.ReportStore
import org.libremail.reporting.ReportSubmitter
import org.libremail.ui.navigation.Routes
import org.libremail.ui.theme.LibreMailTheme
import java.io.File
/**
* End-to-end UI test for [ReportReviewScreen] + [ReportReviewViewModel] over a real file-backed
* [ReportStore] (temp dir) and a [ReportSubmitter] stubbed as disabled (no ingest endpoint, this
* repo's default): the disclaimer + comment/email fields render, Submit stays disabled until the
* comment reaches the minimum length and the email is valid, and Discard deletes the report and
* leaves the screen. The submit-enqueue/upload path (WorkManager) is deliberately out of scope.
*/
@RunWith(AndroidJUnit4::class)
class ReportReviewScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val context: Context =
InstrumentationRegistry.getInstrumentation().targetContext.applicationContext
private val storeDir = File(context.cacheDir, "report-review-test")
private val reportId = "r-1"
@Before
fun setUp() {
storeDir.deleteRecursively()
}
@After
fun tearDown() {
storeDir.deleteRecursively()
}
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun report(id: String) = DebugReport(
id = id,
createdAtMillis = 1_000L,
kind = ReportKind.MANUAL,
appVersionName = "1.0",
appVersionCode = 1L,
androidRelease = "14",
androidSdkInt = 34,
deviceManufacturer = "Test",
deviceModel = "Model",
stackTrace = null,
settings = emptyMap(),
logs = emptyList(),
)
private fun setContent(onDone: () -> Unit = {}): ReportStore {
val store = ReportStore(storeDir)
store.save(report(reportId))
val submitter = mockk<ReportSubmitter> { every { isEnabled } returns false }
val viewModel = ReportReviewViewModel(
SavedStateHandle(mapOf(Routes.REPORT_REVIEW_ARG_ID to reportId)),
store,
submitter,
)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
ReportReviewScreen(onDone = onDone, viewModel = viewModel)
}
}
return store
}
@Test
fun rendersDisclaimer_fieldsAndSubmit() {
setContent()
composeTestRule.onNodeWithText(string(R.string.report_pii_disclaimer_title)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.report_comment_label)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.report_email_label)).performScrollTo().assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.report_submit)).performScrollTo().assertIsDisplayed()
}
@Test
fun submit_isDisabled_whenCommentTooShort_orEmailInvalid() {
setContent()
composeTestRule.onNodeWithText(string(R.string.report_submit)).performScrollTo().assertIsNotEnabled()
}
@Test
fun submit_isEnabled_afterValidCommentAndEmail() {
setContent()
composeTestRule.onNodeWithText(string(R.string.report_comment_label))
.performScrollTo()
.performTextInput("x".repeat(ReportSubmissionRules.MIN_COMMENT_LENGTH))
composeTestRule.onNodeWithText(string(R.string.report_email_label))
.performScrollTo()
.performTextInput("me@example.com")
composeTestRule.onNodeWithText(string(R.string.report_submit)).performScrollTo().assertIsEnabled()
}
@Test
fun tappingCopy_putsThePayloadOnTheSystemClipboard_andShowsAConfirmation() {
// Exercises the #237 migration off LocalClipboardManager/ClipboardManager end-to-end: the
// real system clipboard (not a fake) must contain the exact payload after the suspend
// LocalClipboard/Clipboard call completes.
val store = setContent()
composeTestRule.onNodeWithText(string(R.string.report_copy)).performScrollTo().performClick()
composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText(string(R.string.report_copied)).fetchSemanticsNodes().isNotEmpty()
}
val clipboardManager = context.getSystemService(ClipboardManager::class.java)
val clipText = clipboardManager?.primaryClip?.getItemAt(0)?.text?.toString()
assertEquals(store.find(reportId)!!.toSubmissionPayload(), clipText)
}
@Test
fun tappingDiscard_deletesReport_andLeavesTheScreen() {
var done = false
val store = setContent(onDone = { done = true })
composeTestRule.onNodeWithText(string(R.string.report_discard)).performScrollTo().performClick()
// Discard removes the row; the screen then auto-navigates away once it observes it is gone.
composeTestRule.waitUntil(5_000) { store.find(reportId) == null && done }
}
}
@@ -0,0 +1,149 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.reporting
import androidx.activity.ComponentActivity
import androidx.compose.runtime.MutableState
import androidx.compose.runtime.mutableStateOf
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onAllNodesWithText
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import org.junit.After
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNull
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.reporting.DebugReport
import org.libremail.reporting.ReportKind
import org.libremail.reporting.ReportStore
import org.libremail.ui.StartupCrashPrompt
import org.libremail.ui.theme.LibreMailTheme
import java.io.File
/**
* E2E for the #255 startup-crash-prompt gating, driving the real [StartupCrashPrompt] composable over a
* real file-backed [ReportStore]: a legitimate recent crash pops the dialog exactly once (and never
* again after a simulated relaunch reads the persisted `surfaced` flag), while a stale (> 24h) crash
* never pops it. A fixed clock keeps the age gate independent of the device wall clock.
*/
@RunWith(AndroidJUnit4::class)
class StartupCrashPromptTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val now = 1_000_000_000_000L
private val dayMs = 24L * 60 * 60 * 1000
private lateinit var dir: File
@Before
fun setUp() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
dir = File(context.cacheDir, "startup_crash_prompt_test_${System.nanoTime()}")
dir.deleteRecursively()
dir.mkdirs()
}
@After
fun tearDown() {
dir.deleteRecursively()
}
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
// Unconfined scope runs ReportStore's initial scan inline so a reopened store (simulating a
// relaunch) is readable synchronously, as before the scan moved off-thread (#296).
private fun store() = ReportStore(dir, CoroutineScope(Dispatchers.Unconfined))
private fun crash(id: String, createdAt: Long) = DebugReport(
id = id,
createdAtMillis = createdAt,
kind = ReportKind.CRASH,
appVersionName = "0.1.0",
appVersionCode = 1,
androidRelease = "14",
androidSdkInt = 34,
deviceManufacturer = "Google",
deviceModel = "Pixel",
stackTrace = null,
settings = emptyMap(),
logs = emptyList(),
)
private fun viewModel(store: ReportStore) = StartupReportViewModel(store, now = { now })
/** Renders the prompt against [vmState]; swapping its value simulates a fresh process on relaunch. */
private fun render(vmState: MutableState<StartupReportViewModel>) {
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
StartupCrashPrompt(viewModel = vmState.value, onReview = {})
}
}
}
private fun awaitDialogShown() = composeTestRule.waitUntil(WAIT_MS) {
composeTestRule.onAllNodesWithText(string(R.string.crash_prompt_title)).fetchSemanticsNodes().isNotEmpty()
}
private fun awaitDialogGone() = composeTestRule.waitUntil(WAIT_MS) {
composeTestRule.onAllNodesWithText(string(R.string.crash_prompt_title)).fetchSemanticsNodes().isEmpty()
}
@Test
fun recentCrash_popsDialogOnce_andNotAgainOnRelaunch() {
val store = store()
store.save(crash("c", createdAt = now - 60_000L))
val vmState = mutableStateOf(viewModel(store))
render(vmState)
// First re-open after the crash: the prompt is offered.
awaitDialogShown()
composeTestRule.onNodeWithText(string(R.string.crash_prompt_title)).assertIsDisplayed()
// "Not now" hides it and persistently marks it surfaced (the report itself stays saved).
composeTestRule.onNodeWithText(string(R.string.crash_prompt_later)).performClick()
awaitDialogGone()
assertNotNull(store().find("c"))
// Relaunch: a fresh store + VM over the same dir reads the persisted flag → no re-nag.
composeTestRule.runOnUiThread { vmState.value = viewModel(store()) }
composeTestRule.waitForIdle()
composeTestRule.onNodeWithText(string(R.string.crash_prompt_title)).assertDoesNotExist()
}
@Test
fun staleCrash_doesNotPopDialog() {
val store = store()
store.save(crash("old", createdAt = now - dayMs - 60_000L))
render(mutableStateOf(viewModel(store)))
composeTestRule.waitForIdle()
composeTestRule.onNodeWithText(string(R.string.crash_prompt_title)).assertDoesNotExist()
}
@Test
fun discard_deletesTheReport() {
val store = store()
store.save(crash("c", createdAt = now - 60_000L))
render(mutableStateOf(viewModel(store)))
awaitDialogShown()
composeTestRule.onNodeWithText(string(R.string.crash_prompt_title)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.crash_prompt_discard)).performClick()
awaitDialogGone()
assertNull(store().find("c"))
}
private companion object {
const val WAIT_MS = 5_000L
}
}
@@ -0,0 +1,55 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.security
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.ui.theme.LibreMailTheme
/**
* UI coverage for the fail-closed encrypted-cache error screen (issue #359). [CacheEncryptionErrorScreen]
* is presentational (its report action is wired by the caller), so it is exercised in isolation: the
* verbatim error message shows, and tapping "Report a problem" reports back.
*/
@RunWith(AndroidJUnit4::class)
class CacheEncryptionErrorScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun setContent(onReportProblem: () -> Unit = {}) {
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
CacheEncryptionErrorScreen(onReportProblem = onReportProblem)
}
}
}
@Test
fun showsTheVerbatimErrorMessageAndReportAction() {
setContent()
// The exact maintainer-specified message must render, unchanged.
composeTestRule.onNodeWithText(string(R.string.cache_encryption_error_message)).assertIsDisplayed()
composeTestRule.onNodeWithText(string(R.string.cache_encryption_report_action)).assertIsDisplayed()
}
@Test
fun tappingReportProblem_invokesCallback() {
var reported = false
setContent(onReportProblem = { reported = true })
composeTestRule.onNodeWithText(string(R.string.cache_encryption_report_action)).performClick()
composeTestRule.waitUntil(5_000) { reported }
}
}
@@ -7,11 +7,13 @@ import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.lifecycle.SavedStateHandle
import androidx.lifecycle.ViewModelStore
import androidx.room.Room
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.work.WorkManager
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
@@ -54,13 +56,27 @@ class AccountSettingsScreenTest {
private var manageSignaturesClicked = false
private lateinit var db: AccountDatabase
// Holds the real AccountSettingsViewModel built by hand in setContent() below, so tearDown() can
// clear() it (triggering ViewModel.onCleared()) before closing the DB.
private val viewModelStore = ViewModelStore()
@After
fun tearDown() {
// Clear the store (→ ViewModel.onCleared() → cancels viewModelScope) BEFORE closing the DB.
// The ViewModel's `settings`/`signatureCount`/`defaultSignatureName`/`account` Room
// InvalidationTracker Flows are kept alive by stateIn(WhileSubscribed(5_000)): without this,
// a collector can still be live up to 5s after the UI detaches, so a re-query lands on the
// just-closed in-memory DB and throws SQLITE_MISUSE ("connection is closed") — an intermittent
// teardown race, not a real bug. (Previously worked around by never closing the DB at all.)
viewModelStore.clear()
db.close()
}
private fun setContent(): AccountSettingsRepository {
val context = ApplicationProvider.getApplicationContext<Context>()
// Intentionally not closed in an @After: the ViewModel's `settings` Room Flow (kept alive by
// stateIn/WhileSubscribed) keeps querying after the test body, so closing the in-memory DB out
// from under it races and crashes ("connection pool has been closed"). The DB is reclaimed with
// the test process.
val db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
val repository = AccountSettingsRepository(db.accountSettingsDao())
runBlocking {
db.accountDao().upsert(account.toEntity()) // FK parent for the account_settings row
@@ -74,6 +90,7 @@ class AccountSettingsScreenTest {
syncScheduler = SyncScheduler(Provider { WorkManager.getInstance(context) }),
settingsRepository = SettingsRepository(context),
)
viewModelStore.put("account-settings", viewModel)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
AccountSettingsScreen(
@@ -0,0 +1,135 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.settings
import android.content.Context
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onAllNodesWithText
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.compose.ui.test.performTextInput
import androidx.compose.ui.test.performTextReplacement
import androidx.lifecycle.SavedStateHandle
import androidx.room.Room
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.data.local.AccountDatabase
import org.libremail.data.local.entity.AccountEntity
import org.libremail.data.local.entity.ServerConfigEmbedded
import org.libremail.data.settings.SignatureRepository
import org.libremail.ui.navigation.Routes
import org.libremail.ui.theme.LibreMailTheme
/**
* End-to-end UI test for [SignatureEditScreen] + [SignatureEditViewModel] over a real in-memory Room
* DB and [SignatureRepository]: the new-vs-edit title, saving a freshly entered name (round-tripping
* through the repository, which makes the account's first signature its default), loading an existing
* signature's name into the field, and saving an edit back to the same row.
*/
@RunWith(AndroidJUnit4::class)
class SignatureEditScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val context: Context =
InstrumentationRegistry.getInstrumentation().targetContext.applicationContext
private val accountId = "imap:a"
private lateinit var db: AccountDatabase
private lateinit var repository: SignatureRepository
@Before
fun setUp() {
db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
repository = SignatureRepository(db.signatureDao())
// Signatures foreign-key to an account row, so the parent account must exist first.
runBlocking {
db.accountDao().upsert(
AccountEntity(
id = accountId,
email = "a@example.org",
displayName = "A",
authType = "PASSWORD_IMAP",
imap = ServerConfigEmbedded("imap.example.org", 993, "SSL_TLS"),
smtp = ServerConfigEmbedded("smtp.example.org", 465, "SSL_TLS"),
),
)
}
}
@After
fun tearDown() = db.close()
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
private fun setContent(signatureId: String? = null, onBack: () -> Unit = {}) {
val args = buildMap {
put(Routes.SIGNATURE_EDIT_ARG_ACCOUNT, accountId)
if (signatureId != null) put(Routes.SIGNATURE_EDIT_ARG_ID, signatureId)
}
val viewModel = SignatureEditViewModel(SavedStateHandle(args), repository)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
SignatureEditScreen(onBack = onBack, viewModel = viewModel)
}
}
}
private fun waitForText(text: String) = composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText(text).fetchSemanticsNodes().isNotEmpty()
}
@Test
fun newSignature_showsNewTitle() {
setContent(signatureId = null)
composeTestRule.onNodeWithText(string(R.string.signature_new_title)).assertIsDisplayed()
}
@Test
fun newSignature_savingEnteredName_persistsViaRepository() {
var backInvoked = false
setContent(signatureId = null, onBack = { backInvoked = true })
composeTestRule.onNodeWithText(string(R.string.signature_name)).performTextInput("Work")
composeTestRule.onNodeWithText(string(R.string.signature_save)).performClick()
// The save round-trips to the DB: the first signature created becomes the account's default.
composeTestRule.waitUntil(5_000) {
runBlocking { repository.getDefault(accountId)?.name } == "Work" && backInvoked
}
}
@Test
fun existingSignature_showsEditTitle_andPrefillsName() {
val id = runBlocking { repository.create(accountId, "Personal", "<p>hi</p>") }
setContent(signatureId = id)
waitForText("Personal")
composeTestRule.onNodeWithText(string(R.string.signature_edit_title)).assertIsDisplayed()
composeTestRule.onNodeWithText("Personal").assertIsDisplayed()
}
@Test
fun existingSignature_savingRenamedName_updatesTheSameRow() {
val id = runBlocking { repository.create(accountId, "Personal", "<p>hi</p>") }
setContent(signatureId = id)
waitForText("Personal")
composeTestRule.onNodeWithText("Personal").performTextReplacement("Renamed")
composeTestRule.onNodeWithText(string(R.string.signature_save)).performClick()
composeTestRule.waitUntil(5_000) {
runBlocking { repository.get(id)?.name } == "Renamed"
}
}
}
@@ -0,0 +1,155 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.ui.settings
import android.content.Context
import androidx.activity.ComponentActivity
import androidx.compose.ui.test.assertCountEquals
import androidx.compose.ui.test.assertIsDisplayed
import androidx.compose.ui.test.isNotSelected
import androidx.compose.ui.test.isSelectable
import androidx.compose.ui.test.isSelected
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.compose.ui.test.onAllNodesWithContentDescription
import androidx.compose.ui.test.onAllNodesWithText
import androidx.compose.ui.test.onNodeWithText
import androidx.compose.ui.test.performClick
import androidx.lifecycle.SavedStateHandle
import androidx.lifecycle.ViewModelStore
import androidx.room.Room
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import kotlinx.coroutines.runBlocking
import org.junit.After
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.libremail.R
import org.libremail.data.local.AccountDatabase
import org.libremail.data.local.entity.AccountEntity
import org.libremail.data.local.entity.ServerConfigEmbedded
import org.libremail.data.settings.SignatureRepository
import org.libremail.ui.navigation.Routes
import org.libremail.ui.theme.LibreMailTheme
/**
* End-to-end UI test for [SignaturesScreen] + [SignaturesViewModel] over a real in-memory Room DB
* and [SignatureRepository]: the empty state, name + default-badge rendering, making another
* signature the default (round-tripping through the DB via the radio), and deleting a signature
* (its row disappears as the repository re-emits the account's list).
*/
@RunWith(AndroidJUnit4::class)
class SignaturesScreenTest {
@get:Rule
val composeTestRule = createAndroidComposeRule<ComponentActivity>()
private val context: Context =
InstrumentationRegistry.getInstrumentation().targetContext.applicationContext
private val accountId = "imap:a"
private lateinit var db: AccountDatabase
private lateinit var repository: SignatureRepository
// Holds the real SignaturesViewModel built by hand in setContent() below, so tearDown() can
// clear() it (triggering ViewModel.onCleared()) before closing the DB.
private val viewModelStore = ViewModelStore()
@Before
fun setUp() {
db = Room.inMemoryDatabaseBuilder(context, AccountDatabase::class.java).build()
repository = SignatureRepository(db.signatureDao())
// Signatures foreign-key to an account row, so the parent account must exist first.
runBlocking {
db.accountDao().upsert(
AccountEntity(
id = accountId,
email = "a@example.org",
displayName = "A",
authType = "PASSWORD_IMAP",
imap = ServerConfigEmbedded("imap.example.org", 993, "SSL_TLS"),
smtp = ServerConfigEmbedded("smtp.example.org", 465, "SSL_TLS"),
),
)
}
}
@After
fun tearDown() {
// Clear the store (→ ViewModel.onCleared() → cancels viewModelScope) BEFORE closing the DB.
// SignaturesViewModel.signatures is a Room InvalidationTracker Flow kept alive by
// stateIn(WhileSubscribed(5_000)): without this, the collector can still be live up to 5s
// after the UI detaches, so a re-query lands on the just-closed in-memory DB and throws
// SQLITE_MISUSE ("connection is closed") — an intermittent teardown race, not a real bug.
viewModelStore.clear()
db.close()
}
private fun string(resId: Int) = composeTestRule.activity.getString(resId)
/** Seeds signatures (the first becomes the account's default) then renders the screen. */
private fun setContent(vararg names: String) {
runBlocking { names.forEach { repository.create(accountId, it, "<p>$it body</p>") } }
val viewModel = SignaturesViewModel(
SavedStateHandle(mapOf(Routes.SIGNATURES_ARG_ACCOUNT to accountId)),
repository,
)
viewModelStore.put("signatures", viewModel)
composeTestRule.setContent {
LibreMailTheme(darkTheme = false, dynamicColor = false) {
SignaturesScreen(onBack = {}, onEdit = {}, onAdd = {}, viewModel = viewModel)
}
}
}
private fun waitForText(text: String) = composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText(text).fetchSemanticsNodes().isNotEmpty()
}
@Test
fun noSignatures_showsEmptyState() {
setContent()
composeTestRule.onNodeWithText(string(R.string.signatures_empty)).assertIsDisplayed()
}
@Test
fun signatures_renderNames_withExactlyOneDefaultBadge() {
setContent("Work", "Personal")
waitForText("Work")
composeTestRule.onNodeWithText("Work").assertIsDisplayed()
composeTestRule.onNodeWithText("Personal").assertIsDisplayed()
// The first signature created is the account's sole default, so exactly one badge shows.
composeTestRule.onAllNodesWithText(string(R.string.signature_default_badge)).assertCountEquals(1)
}
@Test
fun tappingRadioOnNonDefault_makesItTheDefault() {
setContent("Work", "Personal")
waitForText("Work")
// "Work" (created first) is the default, so its radio is selected and "Personal"'s is not.
composeTestRule.onAllNodes(isSelectable() and isSelected()).assertCountEquals(1)
composeTestRule.onNode(isSelectable() and isNotSelected()).performClick()
// The tap round-trips through the repository/DB: "Personal" is now the account's default.
composeTestRule.waitUntil(5_000) {
runBlocking { repository.getDefault(accountId)?.name } == "Personal"
}
}
@Test
fun tappingDelete_removesTheRow() {
setContent("Work", "Personal")
waitForText("Work")
// Delete the first (default) signature; the repository re-emits, dropping its row.
composeTestRule.onAllNodesWithContentDescription(string(R.string.signature_delete))[0].performClick()
composeTestRule.waitUntil(5_000) {
composeTestRule.onAllNodesWithText("Work").fetchSemanticsNodes().isEmpty()
}
composeTestRule.onNodeWithText("Personal").assertIsDisplayed()
}
}
+37
View File
@@ -0,0 +1,37 @@
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools">
<!--
Debug-only test harness for issue #221 (never merged into a release APK — this manifest belongs to
the debug source set). Hosts ColdOpenCacheProbe in a dedicated ":coldopen" process so an
instrumented test can observe a GENUINE cold open of a pre-encrypted cache: a process where nothing
has yet loaded SQLCipher's process-global native library. The instrumentation process can't offer
that — minting the encrypted fixture (or any earlier test) loads the .so there, masking the 592a797
nativeOpen crash. The provider is exported="false" and inert unless ContentResolver.call() targets
it, so it has no effect on ordinary debug runs.
-->
<application>
<provider
android:name="org.libremail.data.local.coldopen.ColdOpenCacheProbe"
android:authorities="${applicationId}.coldopen"
android:exported="false"
android:process=":coldopen" />
<!--
Debug-only fetch-gate receiver (issue #393; also never merged into a release APK — this
manifest belongs to the debug source set). Lets the on-device perf harness pause proactive
fetch (backfill + body prefetch) via `adb shell am broadcast` so a genuine uncached
message-open can be measured. Must be exported="true" so the adb `shell` UID can reach it by
explicit component (`-n`); it targets the debug BuildConfig.DEBUG-guarded DebugFetchGate only,
carries no PII, and — being debug-only — can never ship. tools:ignore suppresses the
exported-without-permission lint note: a signature permission would (by design) also lock out
the shell UID this hook exists to serve.
-->
<receiver
android:name="org.libremail.debug.FetchGateReceiver"
android:exported="true"
tools:ignore="ExportedReceiver" />
</application>
</manifest>
@@ -0,0 +1,187 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local.coldopen
import android.content.ContentProvider
import android.content.ContentValues
import android.content.Context
import android.database.Cursor
import android.net.Uri
import android.os.Bundle
import androidx.room.Room
import androidx.sqlite.db.SupportSQLiteDatabase
import androidx.sqlite.db.SupportSQLiteOpenHelper
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import net.zetetic.database.sqlcipher.SupportOpenHelperFactory
import org.libremail.data.local.DatabaseEncryption
import org.libremail.data.local.DeferredOpenHelperFactory
import org.libremail.data.local.LibreMailDatabase
/**
* Test-only harness for issue #221. Lives in the **debug** source set (never in a release APK) and is
* declared in the debug manifest with `android:process=":coldopen"`, so its work runs in a SEPARATE app
* process from the instrumentation process.
*
* SQLCipher's `System.loadLibrary("sqlcipher")` is process-global: once the instrumentation process mints
* the encrypted fixture (or any earlier test opens a keyed database), the `.so` is loaded there and a
* later "cold open" can no longer be observed in that process. That is the exact blind spot behind the
* 592a797 crash — a cold start opening an already-encrypted cache with nothing to convert, where the
* keyed open reaches `SQLiteConnection.nativeOpen` with the library unloaded and throws
* `UnsatisfiedLinkError`. This provider gives the test a pristine process to reproduce it.
*
* [call] runs in the `:coldopen` process and reports two things back in a [Bundle]:
* 1. [KEY_COLD_PROBE] — proof the process is genuinely cold: a keyed open with NO preceding
* `System.loadLibrary` must fail at `nativeOpen`. Anything else means the `.so` was already loaded
* here, so the isolation premise broke and the "cold" open below would be meaningless.
* 2. [KEY_OPEN] — the result of opening the pre-encrypted cache exactly the way production does on a
* steady-state encrypted start ([DatabaseProvisioner]'s encrypted branch + [DatabaseModule]'s open
* lambda): [DatabaseEncryption.ensureEncrypted] (a no-op on an already-encrypted file),
* [DatabaseEncryption.ensureNativeLibraryLoaded] (the 592a797 fix), then a Room open through a
* [DeferredOpenHelperFactory] wrapping [SupportOpenHelperFactory]. Reports whether the seeded row
* reads back.
*
* The `DatabaseProvisioner`/`DatabaseModule` objects themselves are not invoked here because they require
* MockK-substituted collaborators (`DatabaseKeyStore`/`SettingsRepository`/`AccountDataMigrator` are
* final and read the real Keystore/DataStore), and the test APK — hence MockK — is not on a forked app
* process's classloader. The provisioner's encrypted-branch decision is instead mirrored line-for-line,
* against the real `DatabaseEncryption`, `DeferredOpenHelperFactory` and `SupportOpenHelperFactory`.
*/
class ColdOpenCacheProbe : ContentProvider() {
override fun onCreate(): Boolean = true
override fun call(method: String, arg: String?, extras: Bundle?): Bundle {
val out = Bundle()
if (method != METHOD_COLD_OPEN) {
out.putString(KEY_ERROR, "unknown method: $method")
return out
}
val appContext = requireNotNull(context).applicationContext
val bundle = requireNotNull(extras) { "cold-open call requires extras" }
val dbName = requireNotNull(bundle.getString(KEY_DB_NAME)) { "missing $KEY_DB_NAME" }
val passphrase = requireNotNull(bundle.getString(KEY_PASSPHRASE)) { "missing $KEY_PASSPHRASE" }
out.putString(KEY_COLD_PROBE, probeKeyedOpenWithoutLoad(appContext))
out.putString(KEY_OPEN, openThroughProductionWiring(appContext, dbName, passphrase))
return out
}
/**
* A keyed open with NO preceding `System.loadLibrary` — the exact call the pre-592a797 provisioner
* reached with the `.so` unloaded. In a genuinely cold process this hits `nativeOpen` and throws
* [UnsatisfiedLinkError]. Runs against a THROWAWAY file so a partially-created database can never
* disturb the real fixture opened afterwards.
*/
private fun probeKeyedOpenWithoutLoad(appContext: Context): String {
val probeName = "coldopen_probe_$PROBE_NONCE.db"
return try {
val helper = SupportOpenHelperFactory(PROBE_KEY.toByteArray(Charsets.US_ASCII), null, false)
.create(noOpConfiguration(appContext, probeName))
try {
helper.writableDatabase // forces the keyed nativeOpen — the 592a797 crash surface
PROBE_OPENED_UNEXPECTEDLY
} finally {
runCatching { helper.close() }
}
} catch (t: Throwable) {
if (hasUnsatisfiedLink(t)) PROBE_UNSATISFIED_LINK else "$PROBE_OTHER${t.javaClass.name}"
} finally {
appContext.deleteDatabase(probeName)
}
}
/**
* Opens the pre-encrypted fixture the same way production does on a steady-state encrypted start,
* against the real production classes. Returns [OPEN_OK] iff the keyed open succeeds cold and the
* seeded row reads back.
*/
private fun openThroughProductionWiring(appContext: Context, dbName: String, passphrase: String): String {
val dbFile = appContext.getDatabasePath(dbName)
return try {
// Mirror DatabaseProvisioner's encrypted branch: the conversion is a no-op on an already-
// encrypted file, so the explicit native-library load is what lets the keyed open succeed.
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
DatabaseEncryption.ensureNativeLibraryLoaded()
val keyBytes = passphrase.toByteArray(Charsets.US_ASCII)
val database = Room.databaseBuilder(appContext, LibreMailDatabase::class.java, dbName)
// Mirror DatabaseModule.provideDatabase's encrypted open lambda exactly.
.openHelperFactory(
DeferredOpenHelperFactory { configuration ->
SupportOpenHelperFactory(keyBytes, null, false).create(configuration)
},
)
.build()
try {
val ids = runBlocking { database.messageDao().observeSummaries().first().map { it.id } }
if (ids == listOf(EXPECTED_ROW_ID)) OPEN_OK else "$OPEN_ROWS$ids"
} finally {
database.close()
}
} catch (t: Throwable) {
"$OPEN_FAIL${t.javaClass.name}: ${t.message}"
}
}
private fun noOpConfiguration(appContext: Context, name: String): SupportSQLiteOpenHelper.Configuration =
SupportSQLiteOpenHelper.Configuration.builder(appContext)
.name(name)
.callback(object : SupportSQLiteOpenHelper.Callback(CALLBACK_VERSION) {
override fun onCreate(db: SupportSQLiteDatabase) = Unit
override fun onUpgrade(db: SupportSQLiteDatabase, oldVersion: Int, newVersion: Int) = Unit
override fun onDowngrade(db: SupportSQLiteDatabase, oldVersion: Int, newVersion: Int) = Unit
})
.build()
// Unused ContentProvider surface — this provider exists only for its process-isolated call().
override fun query(
uri: Uri,
projection: Array<out String>?,
selection: String?,
selectionArgs: Array<out String>?,
sortOrder: String?,
): Cursor? = null
override fun getType(uri: Uri): String? = null
override fun insert(uri: Uri, values: ContentValues?): Uri? = null
override fun delete(uri: Uri, selection: String?, selectionArgs: Array<out String>?): Int = 0
override fun update(uri: Uri, values: ContentValues?, selection: String?, selectionArgs: Array<out String>?): Int =
0
companion object {
/** Appended to the app's `applicationId` to form the provider authority (see the debug manifest). */
const val AUTHORITY_SUFFIX = ".coldopen"
const val METHOD_COLD_OPEN = "coldOpen"
const val KEY_DB_NAME = "dbName"
const val KEY_PASSPHRASE = "passphrase"
const val KEY_COLD_PROBE = "coldProbe"
const val KEY_OPEN = "open"
const val KEY_ERROR = "error"
/** The id of the single row the fixture is seeded with; the cold open must read it back. */
const val EXPECTED_ROW_ID = "acct:1"
const val PROBE_UNSATISFIED_LINK = "UNSATISFIED_LINK"
const val PROBE_OPENED_UNEXPECTEDLY = "OPENED_UNEXPECTEDLY"
const val PROBE_OTHER = "OTHER:"
const val OPEN_OK = "OK"
const val OPEN_FAIL = "FAIL:"
const val OPEN_ROWS = "ROWS:"
private const val PROBE_KEY = "coldopenprobe"
private const val CALLBACK_VERSION = 1
private val PROBE_NONCE = System.nanoTime()
private fun hasUnsatisfiedLink(throwable: Throwable): Boolean {
var current: Throwable? = throwable
while (current != null) {
if (current is UnsatisfiedLinkError) return true
current = current.cause
}
return false
}
}
}
@@ -0,0 +1,71 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.debug
import android.content.BroadcastReceiver
import android.content.Context
import android.content.Intent
import org.libremail.data.sync.DebugFetchGate
import org.libremail.data.sync.FetchScope
import org.libremail.reporting.AppLog
/**
* Debug-only [BroadcastReceiver] (issue #393) that lets an adb-driven perf harness pause/resume the
* proactive-fetch activities tracked by [DebugFetchGate], so a genuinely uncached message-open can be
* measured (add an account, let headers sync, then open a message that must hit the network instead of a
* warmed cache). Declared **only** in `app/src/debug/AndroidManifest.xml`, so it is physically absent
* from every release APK — the same source-set guarantee `ColdOpenCacheProbe` (#221) relies on.
*
* Driven by (component targeted with `-n`, so it needs no `<intent-filter>`):
* ```
* adb shell am broadcast -a org.libremail.debug.FETCH_GATE \
* -n org.libremail.app/org.libremail.debug.FetchGateReceiver \
* --es action <pause|resume|query> --es scope <backfill,prefetch|all>
* ```
* `am broadcast` delivers this **ordered**, so the receiver returns the resulting state as result data
* (e.g. `data=paused=[backfill,prefetch]`) which `am` prints — a synchronous, race-free read-back for
* the harness. A `query` reports the current state without changing it. The new state is logged via the
* PII-free [AppLog] (scope names only — never an email, host, or message content).
*/
class FetchGateReceiver : BroadcastReceiver() {
override fun onReceive(context: Context, intent: Intent) {
val action = intent.getStringExtra(EXTRA_ACTION)?.trim()?.lowercase()
val scopes = FetchScope.parse(intent.getStringExtra(EXTRA_SCOPE))
when (action) {
ACTION_PAUSE -> {
DebugFetchGate.pause(scopes)
AppLog.i(TAG, "fetch gate pause -> ${DebugFetchGate.pausedResult()}")
}
ACTION_RESUME -> {
DebugFetchGate.resume(scopes)
AppLog.i(TAG, "fetch gate resume -> ${DebugFetchGate.pausedResult()}")
}
ACTION_QUERY -> AppLog.i(TAG, "fetch gate query -> ${DebugFetchGate.pausedResult()}")
else -> AppLog.w(TAG, "fetch gate: unknown action")
}
// Return the gate state as ordered-broadcast result data for a synchronous read-back. Guarded so
// a non-ordered send (which has no result receiver) can't crash the receiver.
if (isOrderedBroadcast) {
resultCode = RESULT_CODE
resultData = DebugFetchGate.pausedResult()
}
}
companion object {
/** The broadcast action the harness sends (kept for parity with the adb command; delivery is by `-n`). */
const val ACTION = "org.libremail.debug.FETCH_GATE"
/** `--es action <pause|resume|query>`. */
const val EXTRA_ACTION = "action"
/** `--es scope <comma-list|all>` (see [FetchScope.parse]). */
const val EXTRA_SCOPE = "scope"
const val ACTION_PAUSE = "pause"
const val ACTION_RESUME = "resume"
const val ACTION_QUERY = "query"
private const val TAG = "FetchGateReceiver"
private const val RESULT_CODE = 0
}
}
@@ -13,6 +13,7 @@ import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.flow.distinctUntilChanged
import kotlinx.coroutines.flow.map
import kotlinx.coroutines.launch
import org.libremail.data.security.KeystoreReportEncryption
import org.libremail.data.settings.SettingsRepository
import org.libremail.data.sync.SyncScheduler
import org.libremail.domain.repository.AccountRepository
@@ -48,6 +49,8 @@ class LibreMailApplication :
@Inject lateinit var diagnosticsCollector: DiagnosticsCollector
@Inject lateinit var reportEncryption: KeystoreReportEncryption
private val appScope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
/** Whether the IDLE push service should currently be running (push enabled AND an account exists). */
@@ -74,11 +77,17 @@ class LibreMailApplication :
// Warm the settings cache so a later crash report can include non-PII settings without
// touching DataStore on the crashing thread.
appScope.launch { runCatching { diagnosticsCollector.warmSettingsCache() } }
// Mirror the encryptCache setting so a crash-time report save (synchronous, on the crashing
// thread) can seal the report at rest without touching DataStore (#369). Collects for the
// process lifetime, so a mid-session toggle takes effect on the next report write.
appScope.launch { runCatching { reportEncryption.observeEncryptCacheSetting() } }
syncScheduler.schedulePeriodicSync()
// Full-history backfill (#12) and device-only retention pruning (#13) run as their own bounded,
// resumable background jobs so they never block foreground sync / pull-to-refresh.
syncScheduler.schedulePeriodicBackfill()
syncScheduler.schedulePeriodicPrune()
// Delete local crash/problem reports older than a month, only while charging (issue #239).
syncScheduler.schedulePeriodicReportPurge()
// Run the IMAP IDLE push service only while it has something to do: the push setting is on
// AND at least one account exists. This starts it when the first account is added and stops
// it when the last is removed, reactively.
@@ -21,6 +21,7 @@ import org.libremail.ui.LibreMailApp
import org.libremail.ui.compose.ComposePrefill
import org.libremail.ui.compose.IntentComposeParser
import org.libremail.ui.lock.AppLockGateHost
import org.libremail.ui.security.CacheEncryptionGate
import org.libremail.ui.theme.LibreMailTheme
import javax.inject.Inject
@@ -73,12 +74,18 @@ class MainActivity : FragmentActivity() {
// Gate the whole app behind the screen-lock when app-lock is enabled. When it is off
// the gate resolves straight to the content, so this is a no-op for most users.
AppLockGateHost {
LibreMailApp(
pendingCompose = pendingCompose.value,
onComposeHandled = { pendingCompose.value = null },
pendingOpenMessageId = pendingOpenMessageId.value,
onOpenMessageHandled = { pendingOpenMessageId.value = null },
)
// Inside the app-lock gate (so the auth-bound passphrase is already unlocked): fail
// closed if the encrypted cache's SQLCipher library won't load (#359), showing the
// error gate instead of ever opening the cache unencrypted. Resolves straight to the
// content when the cache is openable, so it is a no-op for most users.
CacheEncryptionGate {
LibreMailApp(
pendingCompose = pendingCompose.value,
onComposeHandled = { pendingCompose.value = null },
pendingOpenMessageId = pendingOpenMessageId.value,
onOpenMessageHandled = { pendingOpenMessageId.value = null },
)
}
}
}
}
@@ -106,12 +106,16 @@ class OutlookAuthManager @Inject constructor(@ApplicationContext private val con
val authState = AuthState(response, exception).apply { update(tokenResponse, null) }
val email = emailFromIdToken(tokenResponse.idToken)
?: throw IllegalStateException("Could not read the account email from the token")
// Mint an Exchange Online token so the caller can verify the account over IMAP.
val outlook = refreshForScope(authState, OUTLOOK_SCOPE)
// The code exchange above already named the Exchange Online resource ($OUTLOOK_SCOPE), so
// this access token is an outlook.office.com token the caller can verify over IMAP directly.
// Don't re-refresh for the same scope: that second round-trip only rotates the just-issued
// refresh token and adds a needless onboarding failure point. The Graph token is a different
// resource and is minted on demand later (freshGraphToken); the durable AuthState — refresh
// token plus this token's expiry — is serialized here for those later refreshes.
return OAuthResult(
email = email,
accessToken = outlook.accessToken,
authStateJson = outlook.authStateJson,
accessToken = tokenResponse.accessToken.orEmpty(),
authStateJson = authState.jsonSerializeString(),
)
} finally {
service.dispose()
@@ -0,0 +1,68 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.attachment
import android.content.Context
import android.content.Intent
import android.net.Uri
import dagger.hilt.android.qualifiers.ApplicationContext
import org.libremail.data.local.dao.DraftDao
import org.libremail.data.local.dao.OutboxDao
import org.libremail.data.local.toOutgoingAttachments
import javax.inject.Inject
import javax.inject.Singleton
/**
* Releases the persistable read grants the compose pickers take on attachment / inline-image URIs.
* `ComposeScreen` calls `takePersistableUriPermission` on every pick, but nothing released them, so
* the app accumulated indefinite read access to every file/photo ever attached and could hit the
* per-app persisted-grant cap — after which a silently-swallowed take fails a later draft's image
* reload (post-batch security review, Low).
*
* The picked bytes are copied into the app cache when a message is enqueued
* (`MailRepositoryImpl.copyAttachments`), so a grant is only truly needed to reload an image when a
* *draft* is reopened. Callers therefore invoke [releaseUnreferenced] once the referencing row is
* gone — a draft is deleted, or an outbox message is sent or cancelled — passing that row's URIs.
* A URI still referenced by another live draft or outbox row is kept; releasing a grant we do not
* actually hold is expected and swallowed.
*/
@Singleton
class AttachmentUriGrants @Inject constructor(
@ApplicationContext private val context: Context,
private val draftDao: DraftDao,
private val outboxDao: OutboxDao,
) {
/**
* Releases the persistable grant of each URI in [uris] that no *remaining* draft or outbox row
* still references. Call it after deleting the row that referenced them, so that row no longer
* counts toward the "still referenced" check.
*/
suspend fun releaseUnreferenced(uris: Collection<String>) {
if (uris.isEmpty()) return
unreferencedUris(uris, referencedUris()).forEach(::release)
}
/** Every attachment / inline-image URI still referenced by a draft or a queued outbox message. */
private suspend fun referencedUris(): Set<String> {
val fromDrafts = draftDao.getAll().flatMap { it.attachments.toOutgoingAttachments() }
val fromOutbox = outboxDao.getAll().flatMap { it.attachments.toOutgoingAttachments() }
return (fromDrafts + fromOutbox).mapTo(HashSet()) { it.uri }
}
private fun release(uri: String) {
// Only a grant we actually hold can be released; a URI never persisted (or already released)
// throws SecurityException, which is expected here and deliberately ignored.
runCatching {
context.contentResolver.releasePersistableUriPermission(
Uri.parse(uri),
Intent.FLAG_GRANT_READ_URI_PERMISSION,
)
}
}
}
/**
* The distinct URIs in [candidates] not present in [referenced] — i.e. the grants safe to release.
* Kept as a pure top-level function so the release decision is unit-testable without Android types.
*/
internal fun unreferencedUris(candidates: Collection<String>, referenced: Set<String>): List<String> =
candidates.distinct().filterNot { it in referenced }
@@ -2,7 +2,6 @@
package org.libremail.data.local
import android.content.Context
import android.util.Log
import androidx.datastore.core.DataStore
import androidx.datastore.preferences.core.Preferences
import androidx.datastore.preferences.core.booleanPreferencesKey
@@ -15,6 +14,7 @@ import kotlinx.coroutines.withContext
import net.zetetic.database.sqlcipher.SQLiteDatabase
import org.libremail.data.security.DatabaseKeyStore
import org.libremail.data.settings.SettingsRepository
import org.libremail.reporting.AppLog
import java.io.File
import javax.inject.Inject
import javax.inject.Singleton
@@ -108,7 +108,8 @@ class AccountDataMigrator @Inject constructor(
internal val CREATE_TABLE_SQL = mapOf(
"accounts" to
"CREATE TABLE IF NOT EXISTS `accounts` (`id` TEXT NOT NULL, `email` TEXT NOT NULL, " +
"`displayName` TEXT NOT NULL, `authType` TEXT NOT NULL, `imap_host` TEXT NOT NULL, " +
"`displayName` TEXT NOT NULL, `authType` TEXT NOT NULL, " +
"`sortOrder` INTEGER NOT NULL DEFAULT 0, `imap_host` TEXT NOT NULL, " +
"`imap_port` INTEGER NOT NULL, `imap_security` TEXT NOT NULL, `smtp_host` TEXT NOT NULL, " +
"`smtp_port` INTEGER NOT NULL, `smtp_security` TEXT NOT NULL, PRIMARY KEY(`id`))",
"credentials" to
@@ -168,7 +169,19 @@ class AccountDataMigrator @Inject constructor(
val cols = sharedColumns(db, table)
db.rawExecSQL("INSERT OR IGNORE INTO `$table` ($cols) SELECT $cols FROM cache.`$table`")
}
Log.d(TAG, "moved account tables into the account database: $present")
// sortOrder (issue #164) is a destination-only column the pre-#111 cache never had, so
// the copy above leaves every account at its DEFAULT 0. Give them the same stable
// alphabetical initial order ACCOUNT_MIGRATION_1_2 assigns (rank by email), so a
// pre-#111 upgrade lands in the order it already showed rather than an undefined
// tie-break. Post-#111 installs migrate via ACCOUNT_MIGRATION_1_2 instead and never
// reach this copy; either way the user can then drag to reorder.
if ("accounts" in present) {
db.rawExecSQL(
"UPDATE `accounts` SET `sortOrder` = " +
"(SELECT COUNT(*) FROM `accounts` AS ranked WHERE ranked.`email` < `accounts`.`email`)",
)
}
AppLog.d(TAG, "moved account tables into the account database: $present")
} finally {
db.rawExecSQL("DETACH DATABASE cache;")
}
@@ -40,7 +40,7 @@ import org.libremail.data.local.entity.SignatureEntity
AccountSettingsEntity::class,
SignatureEntity::class,
],
version = 1,
version = 2,
exportSchema = true,
)
abstract class AccountDatabase : RoomDatabase() {
@@ -0,0 +1,33 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import androidx.room.migration.Migration
import androidx.sqlite.db.SupportSQLiteDatabase
/**
* Migrations for [AccountDatabase] — the non-auth account store split out of the cache database in
* issue #111. Kept in a SEPARATE file from the cache database's `MIGRATION_x_y` chain in
* `Migrations.kt` on purpose: both databases number their own schema versions from 1, and
* `MigrationTest` reflectively pulls every `Migration` val out of the `Migrations.kt` file facade to
* assert the cache chain is unbroken — an AccountDatabase migration declared alongside them would
* pollute that assertion with a second `1 -> 2`. A distinct file (and the `ACCOUNT_` prefix) keeps
* the two chains from colliding at the package level.
*/
/**
* AccountDatabase v1 -> v2 (issue #164): add the user-controlled [AccountEntity.sortOrder]. Existing
* accounts are given a stable initial order matching the previous alphabetical (`ORDER BY email`)
* listing — each account's rank among the others by email — so the order they were already shown in
* doesn't shuffle on upgrade. `DEFAULT 0` matches the entity's `@ColumnInfo(defaultValue = "0")` so a
* fresh install validates identically to a migrated one (the folders `specialUse` MIGRATION_11_12
* pattern); the backfill then overwrites that default with the real rank.
*/
val ACCOUNT_MIGRATION_1_2 = object : Migration(1, 2) {
override fun migrate(db: SupportSQLiteDatabase) {
db.execSQL("ALTER TABLE `accounts` ADD COLUMN `sortOrder` INTEGER NOT NULL DEFAULT 0")
db.execSQL(
"UPDATE `accounts` SET `sortOrder` = " +
"(SELECT COUNT(*) FROM `accounts` AS ranked WHERE ranked.`email` < `accounts`.`email`)",
)
}
}
@@ -0,0 +1,24 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
/**
* Raised by [DatabaseProvisioner] when the opt-in encrypted cache cannot be opened because SQLCipher's
* native library failed to load or link on this device (issue #359 — e.g. an `.so` the platform
* rejects, surfacing as an `UnsatisfiedLinkError`/`LinkageError` at `SQLiteConnection.nativeOpen` or
* from [DatabaseEncryption.ensureNativeLibraryLoaded]).
*
* The app must **fail closed**: it must NOT fall back to an unencrypted cache (that would silently
* defeat the user's opt-in encryption), NOT wipe the on-disk ciphertext, and NOT touch the
* `encryptCache` setting. Instead this distinct, expected signal is surfaced so the startup UI
* (`CacheEncryptionGate`) can show the encryption error gate — not the mailbox, and not a crash.
*
* Deliberately a dedicated type (not a bare [LinkageError]) so only this precise condition is treated
* as "encryption unavailable"; any other failure still propagates. The provisioner never memoizes it,
* so a later launch — where the library may load, e.g. after an app update — re-attempts and recovers
* automatically.
*/
class CacheEncryptionUnavailableException(cause: Throwable) :
Exception(
"Encrypted cache unavailable: the SQLCipher native library failed to load on this device",
cause,
)
@@ -1,8 +1,8 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local
import android.util.Log
import net.zetetic.database.sqlcipher.SQLiteDatabase
import org.libremail.reporting.AppLog
import java.io.File
/**
@@ -40,6 +40,7 @@ object DatabaseEncryption {
* tables but not that pragma, and a reset version would make Room attempt a bogus migration.
*/
private fun migrate(dbFile: File, sourcePassphrase: String, targetPassphrase: String) {
AppLog.i(TAG, "converting local cache database (targetEncrypted=${targetPassphrase.isNotEmpty()})")
ensureNativeLibraryLoaded()
val dir = dbFile.parentFile ?: error("database file has no parent directory")
val tmp = File(dir, dbFile.name + ".migrate").apply { delete() }
@@ -88,7 +89,7 @@ object DatabaseEncryption {
tmp.copyTo(dbFile, overwrite = true)
tmp.delete()
}
Log.d(TAG, "local cache database converted")
AppLog.d(TAG, "local cache database converted")
}
private fun startsWithSqliteHeader(dbFile: File): Boolean {
@@ -10,7 +10,10 @@ import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import org.libremail.data.security.DatabaseKeyStore
import org.libremail.data.settings.AppSettings
import org.libremail.data.settings.SettingsRepository
import org.libremail.reporting.AppLog
import java.io.File
import javax.inject.Inject
import javax.inject.Singleton
@@ -115,11 +118,47 @@ class DatabaseProvisioner internal constructor(
// resolvePassphrase waits on PassphraseSession until the user authenticates — which is why this
// must never run on the main thread while the cache is locked (issue #93).
val settings = settingsRepository.settings.first()
return try {
resolveOpenMode(settings, dbFile)
} catch (nativeLoadFailure: LinkageError) {
// FAIL CLOSED (issue #359, security rework of #367). SQLCipher's native library could not be
// loaded/linked (e.g. UnsatisfiedLinkError at SQLiteConnection.nativeOpen or from
// ensureNativeLibraryLoaded), so the encrypted cache cannot be opened OR converted. We must
// NOT silently degrade to an unencrypted cache (that would defeat the user's opt-in
// encryption), so we deliberately do NOT: open plaintext, wipe the on-disk ciphertext, or
// write the encryptCache setting. Instead raise a distinct signal the startup UI catches to
// show the encryption error gate. This throw is NOT memoized (it skips prepareCache's
// `.also { prepared = it }`), so the next launch re-attempts and recovers automatically if
// the library later loads.
AppLog.w(
TAG,
"SQLCipher native library failed to load; failing closed (encrypted cache unavailable)",
nativeLoadFailure,
)
throw CacheEncryptionUnavailableException(nativeLoadFailure)
}
}
/**
* The encryption gate (step 3 of [runStartupSequence]): convert the on-disk cache to the form the
* `encryptCache` setting asks for and report how Room must open it. Split out so a native-library
* load failure on either the encrypt or the decrypt-on-disable path (both need SQLCipher's `.so`) is
* caught in one place — see [runStartupSequence]'s handler, which fails closed by raising
* [CacheEncryptionUnavailableException] rather than degrading to an unencrypted cache.
*/
private suspend fun resolveOpenMode(settings: AppSettings, dbFile: File): CacheOpenMode {
val appLock = settings.appLock
return when {
settings.encryptCache -> {
val passphrase = keyStore.resolvePassphrase(appLock)
DatabaseEncryption.ensureEncrypted(dbFile, passphrase)
// Room is about to open the cache with SQLCipher (SupportOpenHelperFactory), so its
// native library must already be loaded. ensureEncrypted() above loads it only as a
// side effect of an actual plaintext -> encrypted conversion; on a steady-state start
// (cache already encrypted, nothing to convert) that no-ops, so without this explicit
// load the keyed open reaches SQLiteConnection.nativeOpen with no library loaded and
// crashes with UnsatisfiedLinkError on every cold start once encryption is enabled.
DatabaseEncryption.ensureNativeLibraryLoaded()
CacheOpenMode.Encrypted(passphrase)
}
@@ -133,4 +172,8 @@ class DatabaseProvisioner internal constructor(
else -> CacheOpenMode.Plaintext
}
}
private companion object {
const val TAG = "DatabaseProvisioner"
}
}
@@ -35,7 +35,7 @@ import org.libremail.data.local.entity.OutboxEntity
FolderEntity::class,
BackfillProgressEntity::class,
],
version = 18,
version = 20,
exportSchema = true,
)
abstract class LibreMailDatabase : RoomDatabase() {
@@ -149,6 +149,10 @@ internal fun FetchedMessage.toEntity(accountId: String, folder: String, inInbox:
inInbox = inInbox,
bodyFetched = false,
uid = uid.toLongOrNull() ?: 0L,
senderFold = sender.lowercase(),
senderEmailFold = senderEmail.lowercase(),
subjectFold = subject.lowercase(),
snippetFold = "",
)
internal fun FolderEntity.toDomain(): Folder = Folder(
@@ -359,3 +359,46 @@ val MIGRATION_17_18 = object : Migration(17, 18) {
db.execSQL("ALTER TABLE `outbox` ADD COLUMN `attachments` TEXT NOT NULL DEFAULT ''")
}
}
/**
* v18 -> v19: Unicode-aware case-insensitive search (issue #232; preserves existing data). SQLite's
* `LIKE` folds case only for ASCII, so search now matches against `lowercase()` copies of the four
* searchable fields. Adds the fold columns and backfills them. The backfill's SQL `lower()` is
* ASCII-only, so existing rows are casefolded for ASCII here and fully re-folded (via the Unicode-aware
* Kotlin `lowercase()` in the mapper / DAO) on their next write or sync.
*/
val MIGRATION_18_19 = object : Migration(18, 19) {
override fun migrate(db: SupportSQLiteDatabase) {
db.execSQL("ALTER TABLE `messages` ADD COLUMN `senderFold` TEXT NOT NULL DEFAULT ''")
db.execSQL("ALTER TABLE `messages` ADD COLUMN `senderEmailFold` TEXT NOT NULL DEFAULT ''")
db.execSQL("ALTER TABLE `messages` ADD COLUMN `subjectFold` TEXT NOT NULL DEFAULT ''")
db.execSQL("ALTER TABLE `messages` ADD COLUMN `snippetFold` TEXT NOT NULL DEFAULT ''")
db.execSQL(
"UPDATE `messages` SET `senderFold` = lower(`sender`), " +
"`senderEmailFold` = lower(`senderEmail`), `subjectFold` = lower(`subject`), " +
"`snippetFold` = lower(`snippet`)",
)
}
}
/**
* v19 -> v20: covering index for the unified-inbox summary scan (issue #187; preserves existing data
* — a pure additive index, no column/table change or data transformation). The paged "All inboxes"
* query [org.libremail.data.local.dao.MessageDao.pagingUnifiedFolderSummaries] filters
* `folder = ? AND inInbox = 1 ORDER BY timestampMillis DESC`, but no index led with `folder`, so the
* planner walked the whole table via `index_messages_timestampMillis` in timestamp order and filtered
* `folder`/`inInbox` per row (a full `SCAN`, verified via `EXPLAIN QUERY PLAN`). The
* `(folder, inInbox, timestampMillis)` index turns the two equality predicates into an index seek and
* supplies the `timestampMillis` ordering, so the scan becomes a bounded `SEARCH … USING INDEX
* index_messages_folder_inInbox_timestampMillis (folder=? AND inInbox=?)` with no temp B-tree sort.
* `CREATE INDEX IF NOT EXISTS` is idempotent, and the name/columns match the Room `@Index` on
* [org.libremail.data.local.entity.MessageEntity] so the migrated schema validates against 20.json.
*/
val MIGRATION_19_20 = object : Migration(19, 20) {
override fun migrate(db: SupportSQLiteDatabase) {
db.execSQL(
"CREATE INDEX IF NOT EXISTS `index_messages_folder_inInbox_timestampMillis` " +
"ON `messages` (`folder`, `inInbox`, `timestampMillis`)",
)
}
}
@@ -5,22 +5,86 @@ import androidx.room.Dao
import androidx.room.Insert
import androidx.room.OnConflictStrategy
import androidx.room.Query
import androidx.room.Transaction
import androidx.room.Update
import kotlinx.coroutines.flow.Flow
import org.libremail.data.local.entity.AccountEntity
@Dao
interface AccountDao {
@Query("SELECT * FROM accounts ORDER BY email")
/**
* Every listing surface (Settings, the drawer switcher, the unified-inbox filter chips) follows
* the user-defined [AccountEntity.sortOrder] (issue #164); `email` is only a tiebreaker for rows
* that share a sortOrder (e.g. inserted via plain [upsert] rather than [insertAtEnd]/[reorder]),
* so equal-sortOrder accounts still list in a stable, deterministic order.
*/
@Query("SELECT * FROM accounts ORDER BY sortOrder, email")
fun observeAll(): Flow<List<AccountEntity>>
@Query("SELECT * FROM accounts ORDER BY email")
/** See [observeAll] — same ordering: sortOrder, then email as a tiebreaker. */
@Query("SELECT * FROM accounts ORDER BY sortOrder, email")
suspend fun getAll(): List<AccountEntity>
@Query("SELECT * FROM accounts WHERE id = :id LIMIT 1")
suspend fun getById(id: String): AccountEntity?
@Insert(onConflict = OnConflictStrategy.REPLACE)
suspend fun upsert(account: AccountEntity)
/**
* Insert [account] only if its id is absent; a conflicting id is left untouched (returns -1).
* Non-destructive by design: an `@Insert(REPLACE)` would delete-then-reinsert the row on an id
* conflict, firing the `ON DELETE CASCADE` that permanently drops the account's `account_settings`
* + `signatures` (issue #309). Pair it with [update] to refresh an existing row in place instead.
*/
@Insert(onConflict = OnConflictStrategy.IGNORE)
suspend fun insertIfAbsent(account: AccountEntity): Long
/** Refreshes an existing account's columns in place (matched by primary key); no cascade. */
@Update
suspend fun update(account: AccountEntity)
/**
* Insert [account], or refresh the existing row **in place** when its id is already present —
* never the delete-then-reinsert an `@Insert(REPLACE)` does, so it does NOT cascade-delete the
* account's `account_settings` + `signatures` (issue #309). Assigns no list position: new accounts
* are appended via [insertAtEnd]; a plain upsert leaves [AccountEntity.sortOrder] as supplied.
*/
@Transaction
suspend fun upsert(account: AccountEntity) {
// -1 = the IGNORE insert was skipped because the id already exists → update the row instead.
if (insertIfAbsent(account) == -1L) update(account)
}
/** The sortOrder that appends a new account to the end of the list (0 when there are none yet). */
@Query("SELECT COALESCE(MAX(sortOrder), -1) + 1 FROM accounts")
suspend fun nextSortOrder(): Int
/**
* Insert [account] at the end of the user-defined order (issue #164), stamping it with the current
* max + 1. Re-adding an already-present id (e.g. re-authing an Outlook account, whose id is the
* deterministic `outlook:<email>`) instead refreshes the row in place, keeping its list position
* and — crucially — its settings + signatures, which a REPLACE would cascade-delete (issue #309).
* Runs in one transaction so the read + write can't interleave with a concurrent add.
*/
@Transaction
suspend fun insertAtEnd(account: AccountEntity) {
val existing = getById(account.id)
if (existing == null) {
insertIfAbsent(account.copy(sortOrder = nextSortOrder()))
} else {
update(account.copy(sortOrder = existing.sortOrder))
}
}
@Query("UPDATE accounts SET sortOrder = :sortOrder WHERE id = :id")
suspend fun setSortOrder(id: String, sortOrder: Int)
/**
* Persist a user-chosen ordering (issue #164): each account is stamped with its index in
* [orderedIds]. Wrapped in a transaction so the list is never observed half-renumbered.
*/
@Transaction
suspend fun reorder(orderedIds: List<String>) {
orderedIds.forEachIndexed { index, id -> setSortOrder(id, index) }
}
@Query("DELETE FROM accounts WHERE id = :id")
suspend fun deleteById(id: String)
@@ -19,9 +19,21 @@ interface DraftDao {
@Query("SELECT * FROM drafts WHERE id = :id LIMIT 1")
suspend fun getById(id: String): DraftEntity?
/** All drafts, used to check whether an attachment URI is still referenced before releasing its grant. */
@Query("SELECT * FROM drafts")
suspend fun getAll(): List<DraftEntity>
/** The drafts composed under [accountId] — enumerated to release their URI grants on account delete (#299). */
@Query("SELECT * FROM drafts WHERE accountId = :accountId")
suspend fun getByAccount(accountId: String): List<DraftEntity>
@Insert(onConflict = OnConflictStrategy.REPLACE)
suspend fun upsert(draft: DraftEntity)
@Query("DELETE FROM drafts WHERE id = :id")
suspend fun delete(id: String)
/** Removes every draft composed under [accountId] (the account is being deleted, #299). */
@Query("DELETE FROM drafts WHERE accountId = :accountId")
suspend fun deleteByAccount(accountId: String)
}
@@ -6,6 +6,7 @@ import androidx.room.Dao
import androidx.room.Insert
import androidx.room.OnConflictStrategy
import androidx.room.Query
import androidx.room.Transaction
import kotlinx.coroutines.flow.Flow
import org.libremail.data.local.entity.FolderUnreadCount
import org.libremail.data.local.entity.MessageEntity
@@ -26,54 +27,83 @@ interface MessageDao {
)
fun observeSummaries(): Flow<List<MessageSummary>>
/**
* Mailbox-list projection scoped in SQL to one account's [folder], newest-first. Unlike
* [observeSummaries] this pulls only that folder's rows and re-emits only when they actually
* change, so the mailbox list's cost scales with what's shown, not the whole cache (issue #86).
* `inInbox` is *not* filtered here so the one query serves both the normal list (caller keeps
* `inInbox = 1` rows) and search (caller keeps rows matching the query, including transient
* `inInbox = 0` server-search hits). Served by the existing `(accountId, folder, uid)` index (its
* `(accountId, folder)` prefix), so no new index — and no schema migration — is needed.
*/
@Query(
"SELECT id, accountId, sender, senderEmail, subject, snippet, timestampMillis, " +
"isRead, isStarred, folder, inInbox, bodyFetched FROM messages " +
"WHERE accountId = :accountId AND folder = :folder ORDER BY timestampMillis DESC",
)
fun observeFolderSummaries(accountId: String, folder: String): Flow<List<MessageSummary>>
/**
* Unified-inbox projection: [folder] across every account, newest-first. Scoped in SQL like
* [observeFolderSummaries] but without an account predicate (issue #86). No existing index leads
* with `folder`, so this still scans in timestamp order — far cheaper than [observeSummaries] (it
* materializes only this folder's rows, not the whole cache) but not O(1); a large multi-account
* unified inbox is a candidate for a `(folder, inInbox, timestampMillis)` index + paging.
*/
@Query(
"SELECT id, accountId, sender, senderEmail, subject, snippet, timestampMillis, " +
"isRead, isStarred, folder, inInbox, bodyFetched FROM messages " +
"WHERE folder = :folder ORDER BY timestampMillis DESC",
)
fun observeUnifiedFolderSummaries(folder: String): Flow<List<MessageSummary>>
/**
* Paged unified-inbox projection: folder-synced rows of [folder] across every account,
* newest-first, as a Paging 3 [PagingSource] (issue #124). Unlike [observeUnifiedFolderSummaries]
* — which materializes the *entire* unified inbox (~thousands of rows) on every emission — Room
* loads only the requested window (LIMIT/OFFSET), so the mailbox list's query, mapping, and
* recomposition cost scale with what's on screen, not the whole cache. Filters `inInbox = 1`
* because the paged browse list shows only synced rows; unified *search* (which must also surface
* transient `inInbox = 0` hits) stays on [observeUnifiedFolderSummaries]. Profiling (see
* newest-first, as a Paging 3 [PagingSource] (issue #124). Room loads only the requested window
* (LIMIT/OFFSET), so the mailbox list's query, mapping, and recomposition cost scale with what's on
* screen, not the whole cache — instead of materializing the *entire* unified inbox (~thousands of
* rows) on every emission. Filters `inInbox = 1` because the paged browse list shows only synced
* rows; unified *search* (which must also surface transient `inInbox = 0` hits) is paged separately
* by [pagingUnifiedFolderSearchSummaries] (issue #214). Profiling (see
* `docs/perf/issue-124-unified-inbox-paging.md`) showed the first page loads flat regardless of
* total cache size on the existing indices, so no `(folder, …)` index / schema migration is added.
* Breaks ties on the `id` primary key (issue #311): `timestampMillis` isn't unique — bulk mail
* shares a second — so without a unique tiebreaker two rows tied at a LIMIT/OFFSET page boundary
* could duplicate or skip across pages. `id` is in the projection, so the tiebreaker adds no index.
*/
@Query(
"SELECT id, accountId, sender, senderEmail, subject, snippet, timestampMillis, " +
"isRead, isStarred, folder, inInbox, bodyFetched FROM messages " +
"WHERE folder = :folder AND inInbox = 1 ORDER BY timestampMillis DESC",
"WHERE folder = :folder AND inInbox = 1 ORDER BY timestampMillis DESC, id",
)
fun pagingUnifiedFolderSummaries(folder: String): PagingSource<Int, MessageSummary>
/**
* Paged per-account folder projection: [accountId]'s folder-synced rows of [folder], newest-first,
* as a Paging 3 [PagingSource] (issue #214). The account-scoped counterpart of
* [pagingUnifiedFolderSummaries] — brings #124's window-at-a-time loading to the per-account browse
* list so opening a large account folder no longer materializes the whole folder into memory.
* Filters `inInbox = 1` (synced browse rows only; per-account *search* uses
* [pagingFolderSearchSummaries], which also surfaces transient `inInbox = 0` hits). Served by the
* `(accountId, folder, uid)` index's `(accountId, folder)` prefix, so no new index / migration.
*/
@Query(
"SELECT id, accountId, sender, senderEmail, subject, snippet, timestampMillis, " +
"isRead, isStarred, folder, inInbox, bodyFetched FROM messages " +
"WHERE accountId = :accountId AND folder = :folder AND inInbox = 1 ORDER BY timestampMillis DESC, id",
)
fun pagingFolderSummaries(accountId: String, folder: String): PagingSource<Int, MessageSummary>
/**
* Paged unified search: rows of [folder] across every account whose sender, sender address,
* subject, or snippet match [pattern] — a pre-built SQL `LIKE` pattern (`%term%`, with the LIKE
* metacharacters `\ % _` escaped by `\`) — newest-first, as a Paging 3 [PagingSource] (issue #214).
* Scans the same columns the old in-memory search filter (`matchesSearch`) did, but in SQL so a
* search over a large folder loads only the visible window. Unlike the browse pagers this does
* *not* filter `inInbox`, so it surfaces both synced rows and the transient `inInbox = 0`
* server-search hits `MailRepository.searchServer` inserts — exactly what the old filter saw.
* Matches the Unicode-casefolded `*Fold` columns (issue #232) with a pattern built from the
* lowercased query, so search is case-insensitive beyond ASCII — unlike the old ASCII-only `LIKE`.
* Breaks ties on the `id` primary key for a total page order, like the browse pagers (issue #311).
*/
@Query(
"SELECT id, accountId, sender, senderEmail, subject, snippet, timestampMillis, " +
"isRead, isStarred, folder, inInbox, bodyFetched FROM messages " +
"WHERE folder = :folder AND (senderFold LIKE :pattern ESCAPE '\\' OR " +
"senderEmailFold LIKE :pattern ESCAPE '\\' OR subjectFold LIKE :pattern ESCAPE '\\' OR " +
"snippetFold LIKE :pattern ESCAPE '\\') ORDER BY timestampMillis DESC, id",
)
fun pagingUnifiedFolderSearchSummaries(folder: String, pattern: String): PagingSource<Int, MessageSummary>
/**
* Paged per-account search: [accountId]'s rows of [folder] matching [pattern] (see
* [pagingUnifiedFolderSearchSummaries] for the pattern/column contract), newest-first, as a
* Paging 3 [PagingSource] (issue #214). Like the unified variant it leaves `inInbox` unfiltered so
* per-account search still surfaces transient server-search hits.
*/
@Query(
"SELECT id, accountId, sender, senderEmail, subject, snippet, timestampMillis, " +
"isRead, isStarred, folder, inInbox, bodyFetched FROM messages " +
"WHERE accountId = :accountId AND folder = :folder AND (senderFold LIKE :pattern ESCAPE '\\' OR " +
"senderEmailFold LIKE :pattern ESCAPE '\\' OR subjectFold LIKE :pattern ESCAPE '\\' OR " +
"snippetFold LIKE :pattern ESCAPE '\\') ORDER BY timestampMillis DESC, id",
)
fun pagingFolderSearchSummaries(
accountId: String,
folder: String,
pattern: String,
): PagingSource<Int, MessageSummary>
/**
* Live per-(account, folder) unread counts for the drawer's folder badges and the bold styling of
* accounts with unread mail. Counts only folder-synced rows (`inInbox = 1`), so transient
@@ -133,12 +163,8 @@ interface MessageDao {
* Refreshes the display fields (and the materialized [MessageEntity.uid], keeping it fresh for
* rows migrated before the column existed) from the server without touching the cached body, the
* local read/star flags (which may hold an optimistic change the server hasn't reflected yet), or
* the inbox membership.
* the inbox membership. Keeps the header casefold search columns (issue #232) in sync.
*/
@Query(
"UPDATE messages SET sender = :sender, senderEmail = :senderEmail, subject = :subject, " +
"timestampMillis = :timestampMillis, uid = :uid WHERE id = :id",
)
suspend fun updateHeaderContent(
id: String,
sender: String,
@@ -146,14 +172,64 @@ interface MessageDao {
subject: String,
timestampMillis: Long,
uid: Long,
) = updateHeaderContentInternal(
id, sender, senderEmail, subject,
sender.lowercase(), senderEmail.lowercase(), subject.lowercase(),
timestampMillis, uid,
)
@Query(
"UPDATE messages SET sender = :sender, senderEmail = :senderEmail, subject = :subject, " +
"senderFold = :senderFold, senderEmailFold = :senderEmailFold, subjectFold = :subjectFold, " +
"timestampMillis = :timestampMillis, uid = :uid WHERE id = :id",
)
suspend fun updateHeaderContentInternal(
id: String,
sender: String,
senderEmail: String,
subject: String,
senderFold: String,
senderEmailFold: String,
subjectFold: String,
timestampMillis: Long,
uid: Long,
)
/**
* Applies [updateHeaderContent] to every [messages] row in a single transaction (issue #310). A
* foreground sync / pull-to-refresh refreshes a whole recent window at once; running each row's
* UPDATE in its own implicit transaction fsyncs the journal once per message — N commits per folder
* per sync, amplified on the encrypted cache — so this collapses them into one commit. Refreshes the
* same display fields (plus the materialized [MessageEntity.uid] and the casefold search columns) as
* the single-row overload, still leaving cached bodies and optimistic read/star flags untouched.
*/
@Transaction
suspend fun updateHeaderContents(messages: List<MessageEntity>) {
for (message in messages) {
updateHeaderContent(
id = message.id,
sender = message.sender,
senderEmail = message.senderEmail,
subject = message.subject,
timestampMillis = message.timestampMillis,
uid = message.uid,
)
}
}
/** Marks rows as folder-synced (e.g. a former search-only row that the sync now returns). */
@Query("UPDATE messages SET inInbox = 1 WHERE id IN (:ids)")
suspend fun markSynced(ids: List<String>)
@Query("UPDATE messages SET body = :body, isHtml = :isHtml, snippet = :snippet, bodyFetched = 1 WHERE id = :id")
suspend fun updateBody(id: String, body: String, isHtml: Boolean, snippet: String)
/** Sets the fetched body + its derived snippet, keeping [MessageEntity.snippetFold] (search) in sync. */
suspend fun updateBody(id: String, body: String, isHtml: Boolean, snippet: String) =
updateBodyInternal(id, body, isHtml, snippet, snippet.lowercase())
@Query(
"UPDATE messages SET body = :body, isHtml = :isHtml, snippet = :snippet, " +
"snippetFold = :snippetFold, bodyFetched = 1 WHERE id = :id",
)
suspend fun updateBodyInternal(id: String, body: String, isHtml: Boolean, snippet: String, snippetFold: String)
@Query("UPDATE messages SET isRead = :isRead WHERE id = :id")
suspend fun setRead(id: String, isRead: Boolean)
@@ -168,6 +244,15 @@ interface MessageDao {
@Query("DELETE FROM messages WHERE id IN (:ids)")
suspend fun deleteByIds(ids: List<String>)
/**
* Ids of every row belonging to [accountId] (all folders; synced rows plus transient search-only
* hits). Lets [org.libremail.data.repository.AccountRepositoryImpl.deleteAccount] enumerate a
* deleted account's messages to purge their on-disk attachment cache before [deleteByAccount]
* removes the rows that name them (issue #299).
*/
@Query("SELECT id FROM messages WHERE accountId = :accountId")
suspend fun getIdsForAccount(accountId: String): List<String>
@Query("DELETE FROM messages WHERE accountId = :accountId")
suspend fun deleteByAccount(accountId: String)
@@ -15,6 +15,9 @@ interface OutboxDao {
@Query("SELECT * FROM outbox ORDER BY createdAt")
suspend fun getAll(): List<OutboxEntity>
@Query("SELECT * FROM outbox WHERE id = :id LIMIT 1")
suspend fun getById(id: String): OutboxEntity?
@Query("SELECT * FROM outbox ORDER BY createdAt")
fun observeAll(): Flow<List<OutboxEntity>>
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.local.entity
import androidx.room.ColumnInfo
import androidx.room.Embedded
import androidx.room.Entity
import androidx.room.PrimaryKey
@@ -14,6 +15,14 @@ data class AccountEntity(
val authType: String,
@Embedded(prefix = "imap_") val imap: ServerConfigEmbedded,
@Embedded(prefix = "smtp_") val smtp: ServerConfigEmbedded,
/**
* The account's position in the user-defined ordering shown across the app (issue #164): the
* Settings list, the drawer's account switcher, and the unified-inbox filter chips all list
* accounts `ORDER BY sortOrder`. New accounts are appended (current max + 1); the user can drag to
* reorder. `DEFAULT 0` matches [ACCOUNT_MIGRATION_1_2] so a fresh install validates identically to
* a migrated one (the folders `specialUse` pattern).
*/
@ColumnInfo(defaultValue = "0") val sortOrder: Int = 0,
)
/** Embedded host/port/security columns (prefixed per server in [AccountEntity]). */
@@ -11,7 +11,19 @@ import androidx.room.PrimaryKey
// The (accountId, folder, uid) index serves the folder-scoped UID probes the backfill/reconcile
// hot paths run on every page/sync: MIN(uid) (lowestSyncedUid) and the uid >= window bound
// (deleteSyncedInWindowNotIn / syncedIdsBeyondCountInFolder).
indices = [Index("accountId"), Index("timestampMillis"), Index("accountId", "folder", "uid")],
//
// The (folder, inInbox, timestampMillis) index serves the unified-inbox summary scan (issue #187):
// MessageDao.pagingUnifiedFolderSummaries filters `folder = ? AND inInbox = 1 ORDER BY
// timestampMillis DESC` with no folder-leading index, so it SCANned the whole table via
// index_messages_timestampMillis and filtered per row. This index makes the two equalities an
// index seek and supplies the timestampMillis ordering, turning the SCAN into a bounded SEARCH
// with no temp B-tree sort (verified via EXPLAIN QUERY PLAN).
indices = [
Index("accountId"),
Index("timestampMillis"),
Index("accountId", "folder", "uid"),
Index("folder", "inInbox", "timestampMillis"),
],
)
data class MessageEntity(
@PrimaryKey val id: String,
@@ -38,4 +50,12 @@ data class MessageEntity(
* (refreshed to the real UID on the next sync).
*/
@ColumnInfo(defaultValue = "0") val uid: Long = 0L,
// Unicode-casefolded copies of the searchable fields (issue #232). SQLite's LIKE folds case only for
// ASCII, so search matches against these `lowercase()` copies (Kotlin's lowercase is Unicode-aware)
// for case-insensitive Unicode search. Kept in sync wherever their source is written — toEntity,
// MessageDao.updateBody (snippet), and MessageDao.updateHeaderContent (headers).
@ColumnInfo(defaultValue = "") val senderFold: String = "",
@ColumnInfo(defaultValue = "") val senderEmailFold: String = "",
@ColumnInfo(defaultValue = "") val subjectFold: String = "",
@ColumnInfo(defaultValue = "") val snippetFold: String = "",
)
@@ -1,15 +1,21 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.repository
import android.content.Context
import dagger.hilt.android.qualifiers.ApplicationContext
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.map
import org.libremail.data.attachment.AttachmentUriGrants
import org.libremail.data.attachmentCacheDir
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.dao.BackfillProgressDao
import org.libremail.data.local.dao.DraftDao
import org.libremail.data.local.dao.FolderDao
import org.libremail.data.local.dao.MessageDao
import org.libremail.data.local.toDomain
import org.libremail.data.local.toEntity
import org.libremail.data.local.toImapParams
import org.libremail.data.local.toOutgoingAttachments
import org.libremail.data.security.CredentialStore
import org.libremail.data.settings.AccountSettingsRepository
import org.libremail.data.sync.SyncScheduler
@@ -23,15 +29,18 @@ import javax.inject.Singleton
@Singleton
class AccountRepositoryImpl @Inject constructor(
@ApplicationContext private val context: Context,
private val accountDao: AccountDao,
private val messageDao: MessageDao,
private val folderDao: FolderDao,
private val backfillProgressDao: BackfillProgressDao,
private val draftDao: DraftDao,
private val credentialStore: CredentialStore,
private val imapClient: ImapClient,
private val syncScheduler: SyncScheduler,
private val accountSettingsRepository: AccountSettingsRepository,
private val mailNotifier: MailNotifier,
private val attachmentUriGrants: AttachmentUriGrants,
) : AccountRepository {
override fun observeAccounts(): Flow<List<Account>> = accountDao.observeAll().map { rows ->
@@ -44,7 +53,7 @@ class AccountRepositoryImpl @Inject constructor(
override suspend fun addImapAccount(account: Account, password: String): Result<List<String>> = runCatching {
val folders = imapClient.listFolders(account.toImapParams(secret = password, useXoauth2 = false))
accountDao.upsert(account.toEntity())
accountDao.insertAtEnd(account.toEntity())
accountSettingsRepository.ensureDefaults(account.id)
credentialStore.saveSecret(account.id, password)
mailNotifier.ensureAccountChannel(account)
@@ -60,7 +69,7 @@ class AccountRepositoryImpl @Inject constructor(
): Result<List<String>> = runCatching {
val account = Account.outlook(email)
val folders = imapClient.listFolders(account.toImapParams(secret = accessToken, useXoauth2 = true))
accountDao.upsert(account.toEntity())
accountDao.insertAtEnd(account.toEntity())
accountSettingsRepository.ensureDefaults(account.id)
credentialStore.saveSecret(account.id, authStateJson)
mailNotifier.ensureAccountChannel(account)
@@ -69,15 +78,35 @@ class AccountRepositoryImpl @Inject constructor(
folders.map { it.fullName }
}
override suspend fun reorderAccounts(orderedIds: List<String>) = accountDao.reorder(orderedIds)
override suspend fun deleteAccount(id: String) {
// Enumerate the on-disk artifacts to clean up WHILE the rows that name them still exist: once
// the message + draft rows are gone, nothing can recover those message ids / draft URIs again,
// so the files/grants would leak forever (issue #299).
val messageIds = messageDao.getIdsForAccount(id)
val draftUris = draftDao.getByAccount(id)
.flatMap { it.attachments.toOutgoingAttachments() }
.map { it.uri }
accountDao.deleteById(id)
credentialStore.delete(id)
mailNotifier.deleteAccountChannel(id)
// Remove the account's cached mail (attachment rows cascade via the foreign key), folders, and
// backfill progress. The account_settings row is removed automatically by its cascading FK.
// Remove the account's cached mail (attachment rows cascade via the foreign key), folders,
// backfill progress, and drafts. The account_settings + signatures rows are removed
// automatically by their cascading FK when the account row above is deleted.
messageDao.deleteByAccount(id)
folderDao.deleteForAccount(id)
backfillProgressDao.deleteForAccount(id)
draftDao.deleteByAccount(id)
// Now the rows are gone: delete each message's on-disk attachment cache (keyed by message id,
// the same path MailRepositoryImpl writes), and release any persistable draft-URI grant no
// remaining draft/outbox row still needs (issue #299).
messageIds.forEach { messageId ->
runCatching { attachmentCacheDir(context.cacheDir, messageId).deleteRecursively() }
}
attachmentUriGrants.releaseUnreferenced(draftUris)
}
override suspend fun resetBackfillProgress(accountId: String?) {
@@ -6,6 +6,7 @@ import android.net.Uri
import androidx.paging.Pager
import androidx.paging.PagingConfig
import androidx.paging.PagingData
import androidx.paging.PagingSource
import androidx.paging.map
import dagger.hilt.android.qualifiers.ApplicationContext
import jakarta.mail.Flags
@@ -20,6 +21,7 @@ import kotlinx.coroutines.withContext
import org.libremail.data.ReplyBuilder
import org.libremail.data.SignatureBlock
import org.libremail.data.Snippet
import org.libremail.data.attachment.AttachmentUriGrants
import org.libremail.data.attachmentCacheDir
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.dao.AttachmentDao
@@ -29,14 +31,17 @@ import org.libremail.data.local.dao.MessageDao
import org.libremail.data.local.dao.OutboxDao
import org.libremail.data.local.entity.FolderEntity
import org.libremail.data.local.entity.MessageRouting
import org.libremail.data.local.entity.MessageSummary
import org.libremail.data.local.entity.OutboxEntity
import org.libremail.data.local.toDomain
import org.libremail.data.local.toEntity
import org.libremail.data.local.toOutgoingAttachments
import org.libremail.data.local.toOutgoingAttachmentsJson
import org.libremail.data.settings.AccountSettingsRepository
import org.libremail.data.settings.SignatureRepository
import org.libremail.data.sync.MailConnectionFactory
import org.libremail.data.sync.SendScheduler
import org.libremail.data.sync.logSafeFolderLabel
import org.libremail.domain.model.Attachment
import org.libremail.domain.model.Draft
import org.libremail.domain.model.Folder
@@ -49,8 +54,11 @@ import org.libremail.domain.model.OutgoingAttachment
import org.libremail.domain.model.OutgoingMessage
import org.libremail.domain.model.ReplyMode
import org.libremail.domain.model.UnreadCount
import org.libremail.domain.model.sanitizeAttachmentName
import org.libremail.domain.repository.MailRepository
import org.libremail.mail.ImapClient
import org.libremail.reporting.AppLog
import org.libremail.reporting.accountLogRef
import java.io.File
import java.util.UUID
import javax.inject.Inject
@@ -70,6 +78,7 @@ class MailRepositoryImpl @Inject constructor(
private val sendScheduler: SendScheduler,
private val accountSettingsRepository: AccountSettingsRepository,
private val signatureRepository: SignatureRepository,
private val attachmentUriGrants: AttachmentUriGrants,
) : MailRepository {
// Application-lifetime scope for fire-and-forget server pushes that must outlive the caller — e.g.
@@ -78,12 +87,6 @@ class MailRepositoryImpl @Inject constructor(
// Hilt's SingletonComponent), so the scope's lifetime is the process's, not any one caller's coroutine.
private val backgroundScope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
override fun observeFolderMessages(accountId: String, folder: String): Flow<List<Message>> =
messageDao.observeFolderSummaries(accountId, folder).map { rows -> rows.map { it.toDomain() } }
override fun observeUnifiedFolderMessages(folder: String): Flow<List<Message>> =
messageDao.observeUnifiedFolderSummaries(folder).map { rows -> rows.map { it.toDomain() } }
override fun pagedUnifiedFolderMessages(folder: String): Flow<PagingData<Message>> = Pager(
config = PagingConfig(
// A page comfortably exceeds a screenful so scrolling rarely waits on a load; loading
@@ -93,10 +96,45 @@ class MailRepositoryImpl @Inject constructor(
pageSize = MAILBOX_PAGE_SIZE,
initialLoadSize = MAILBOX_PAGE_SIZE * 3,
enablePlaceholders = false,
// Bound the in-memory window so a deep scroll can't accumulate the whole (potentially
// thousands-of-rows) inbox: keep ~5 pages resident and evict the rest. Must be
// >= pageSize + 2*prefetchDistance (40 + 2*40 = 120); with placeholders off, evicted leading
// positions drop from the loaded window and the list rendering already null-guards them.
maxSize = MAILBOX_PAGE_SIZE * 5,
),
pagingSourceFactory = { messageDao.pagingUnifiedFolderSummaries(folder) },
).flow.map { page -> page.map { it.toDomain() } }
override fun pagedFolderMessages(accountId: String, folder: String): Flow<PagingData<Message>> =
mailboxPager { messageDao.pagingFolderSummaries(accountId, folder) }
override fun pagedUnifiedSearchMessages(folder: String, query: String): Flow<PagingData<Message>> =
mailboxPager { messageDao.pagingUnifiedFolderSearchSummaries(folder, likePattern(query.lowercase())) }
override fun pagedFolderSearchMessages(
accountId: String,
folder: String,
query: String,
): Flow<PagingData<Message>> =
mailboxPager { messageDao.pagingFolderSearchSummaries(accountId, folder, likePattern(query.lowercase())) }
/**
* Shared [Pager] for the per-account and search mailbox lists (issue #214). Same window sizing as
* the unified browse pager, plus a bounded `maxSize` so scrolling a long list drops far-offscreen
* pages instead of retaining the whole scrolled-through range in memory. Placeholders stay off (the
* row height varies, and the list never sizes a scrollbar to the full uncounted result).
*/
private fun mailboxPager(pagingSourceFactory: () -> PagingSource<Int, MessageSummary>): Flow<PagingData<Message>> =
Pager(
config = PagingConfig(
pageSize = MAILBOX_PAGE_SIZE,
initialLoadSize = MAILBOX_PAGE_SIZE * 3,
enablePlaceholders = false,
maxSize = MAILBOX_PAGE_SIZE * 5,
),
pagingSourceFactory = pagingSourceFactory,
).flow.map { page -> page.map { it.toDomain() } }
override fun observeFolders(accountId: String): Flow<List<Folder>> =
folderDao.observeForAccount(accountId).map { rows ->
rows.map { it.toDomain() }
@@ -118,11 +156,15 @@ class MailRepositoryImpl @Inject constructor(
override suspend fun openMessage(id: String): Result<Message> = withContext(Dispatchers.IO) {
runCatching {
// Time the whole open so a debug report shows what the reader's spinner is waiting on — a
// cached open is a local read; a first open blocks on the IMAP body fetch below (issue #358).
val startNanos = System.nanoTime()
// Route on the body-less projection: a cached, already-read message needs no account, no
// credentials, and no network, so it skips the Keystore decrypt + DataStore read that
// resolving connection params costs (issue #186). Only the fetch / SEEN-push branches below
// pull the account and resolve params, and each does so lazily right where it is needed.
val routing = messageDao.getRouting(id) ?: error("Message not found")
val fetchedBody = !routing.bodyFetched
if (!routing.bodyFetched || !routing.isRead) {
val account = accountDao.getById(routing.accountId)?.toDomain()
if (account != null && !routing.bodyFetched) {
@@ -142,7 +184,14 @@ class MailRepositoryImpl @Inject constructor(
}
}
// The single full-body read, reserved for the value the reader actually renders (issue #186).
messageDao.getById(id)?.toDomain() ?: error("Message not found")
val message = messageDao.getById(id)?.toDomain() ?: error("Message not found")
// PII-free: hashed account ref, system-folder label only, plus the branch taken and elapsed ms.
AppLog.i(
READER_TAG,
"openMessage ${accountLogRef(routing.accountId)} folder=${logSafeFolderLabel(routing.folder)} " +
"fetchedBody=$fetchedBody took=${(System.nanoTime() - startNanos) / NANOS_PER_MS}ms",
)
message
}
}
@@ -291,7 +340,7 @@ class MailRepositoryImpl @Inject constructor(
val routings = messageDao.getRoutingByIds(ids)
messageDao.deleteByIds(ids) // optimistic
forEachAccountFolder(routings) { params, folder, group ->
group.forEach { imapClient.deleteMessage(params, folder, uidOf(it.id)) }
imapClient.deleteMessages(params, folder, group.map { uidOf(it.id) })
}
}
@@ -352,7 +401,7 @@ class MailRepositoryImpl @Inject constructor(
when (val dest = destByAccount[group.first().accountId]) {
null ->
if (fallbackExpunge) {
group.forEach { imapClient.deleteMessage(params, folder, uidOf(it.id)) }
imapClient.deleteMessages(params, folder, group.map { uidOf(it.id) })
} else {
error("No ${role.name.lowercase()} folder for this account")
}
@@ -421,7 +470,7 @@ class MailRepositoryImpl @Inject constructor(
private fun copyAttachments(outboxId: String, attachments: List<OutgoingAttachment>) {
if (attachments.isEmpty()) return
attachments.forEachIndexed { index, attachment ->
val safeName = attachment.name.substringAfterLast('/').substringAfterLast('\\').ifBlank { "attachment" }
val safeName = sanitizeAttachmentName(attachment.name)
val dir = File(context.cacheDir, "outbox/$outboxId/$index").apply { mkdirs() }
runCatching {
context.contentResolver.openInputStream(Uri.parse(attachment.uri))?.use { input ->
@@ -437,15 +486,24 @@ class MailRepositoryImpl @Inject constructor(
override suspend fun saveDraft(draft: Draft) = draftDao.upsert(draft.toEntity())
override suspend fun deleteDraft(id: String) = draftDao.delete(id)
override suspend fun deleteDraft(id: String) {
// Capture the draft's attachment URIs before the row is gone, then release any persistable grant
// no other live draft/outbox row still needs (security review): a deleted draft can never reopen.
val uris = draftDao.getById(id)?.attachments?.toOutgoingAttachments()?.map { it.uri }.orEmpty()
draftDao.delete(id)
attachmentUriGrants.releaseUnreferenced(uris)
}
override fun observeOutbox(): Flow<List<OutboxMessage>> = outboxDao.observeAll().map { rows ->
rows.map { it.toDomain() }
}
override suspend fun cancelOutboxMessage(id: String) {
val uris = outboxDao.getById(id)?.attachments?.toOutgoingAttachments()?.map { it.uri }.orEmpty()
outboxDao.delete(id)
File(context.cacheDir, "outbox/$id").deleteRecursively()
// The queued copy is gone; release any picked-URI grant no other live draft/outbox row needs.
attachmentUriGrants.releaseUnreferenced(uris)
}
override suspend fun retryOutbox() = sendScheduler.sendNow()
@@ -484,13 +542,17 @@ class MailRepositoryImpl @Inject constructor(
* and avoids filename collisions between messages.
*/
private fun attachmentFile(messageId: String, partIndex: Int, filename: String): File {
val safeName = filename.substringAfterLast('/').substringAfterLast('\\').ifBlank { "attachment" }
val safeName = sanitizeAttachmentName(filename)
return File(attachmentCacheDir(context.cacheDir, messageId), "$partIndex/$safeName")
}
}
private const val SEARCH_LIMIT = 50
/** Perf-breadcrumb tag and ns→ms divisor for the reader-open timing (issue #358). */
private const val READER_TAG = "MailReader"
private const val NANOS_PER_MS = 1_000_000L
/** Rows per page for the unified inbox (issue #124) — a page is a few screenfuls of message rows. */
private const val MAILBOX_PAGE_SIZE = 40
@@ -502,3 +564,13 @@ private const val SEEN_FLAG_RETRY_BACKOFF_MS = 2_000L
/** Message id is "<accountId>:<uid>"; the uid is the trailing segment. */
private fun uidOf(id: String): String = id.substringAfterLast(':')
/**
* Builds the SQL `LIKE` pattern the paged-search DAO queries take (issue #214), preserving the old
* `matchesSearch` literal-substring semantics: escape the LIKE metacharacters (`\ % _`) — the `\`
* first, so the escapes just added aren't themselves re-escaped — then wrap the term in wildcards.
*/
private fun likePattern(query: String): String {
val escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
return "%$escaped%"
}
@@ -1,9 +1,12 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.security
import android.os.Build
import android.security.keystore.KeyGenParameterSpec
import android.security.keystore.KeyProperties
import android.security.keystore.StrongBoxUnavailableException
import android.util.Base64
import org.libremail.reporting.AppLog
import java.security.GeneralSecurityException
import java.security.KeyStore
import javax.crypto.AEADBadTagException
@@ -115,23 +118,52 @@ abstract class AesGcmKeystoreCipher(private val alias: String, private val gener
// Synchronized so two concurrent first-run encrypts can't both generate a key under the same
// alias — the second would overwrite the first, leaving the first secret undecryptable.
protected open fun getOrCreateKey(): SecretKey = synchronized(keyLock) {
existingKey()?.let { return it }
existingKey() ?: generateKeyWithStrongBoxFallback()
}
/**
* Mint the key, preferring the hardware **StrongBox** secure element (a dedicated tamper-resistant
* chip) so the non-exportable key is bound to the strongest keystore available. Devices without
* StrongBox report [StrongBoxUnavailableException] at generation time; we then regenerate a
* TEE-backed key so key creation still succeeds on every device. Both the master ([KeystoreCrypto])
* and auth-bound ([DatabaseKeyCipher]) keys inherit this through the shared base.
*/
private fun generateKeyWithStrongBoxFallback(): SecretKey = try {
generateKey(strongBox = true)
} catch (e: StrongBoxUnavailableException) {
// Expected on devices with no StrongBox — not an error. PII-free (a device-capability fact).
AppLog.i(TAG, "StrongBox unavailable for Keystore alias '$alias'; using a TEE-backed key: ${e.message}")
generateKey(strongBox = false)
}
/**
* Test seam over the raw Android Keystore key generation for a given [strongBox] preference. The
* real [KeyGenerator] is device-only, so JVM unit tests override this to exercise the StrongBox
* fallback in [generateKeyWithStrongBoxFallback] without a Keystore.
*/
protected open fun generateKey(strongBox: Boolean): SecretKey {
val generator = KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, ANDROID_KEYSTORE)
generator.init(keySpec())
generator.generateKey()
generator.init(keySpec(strongBox))
return generator.generateKey()
}
/** The alias-bound [KeyGenParameterSpec] for this key; subclasses extend [keySpecBuilder]. */
protected abstract fun keySpec(): KeyGenParameterSpec
protected abstract fun keySpec(strongBox: Boolean): KeyGenParameterSpec
/** The common AES-256-GCM builder (encrypt + decrypt, GCM, no padding, 256-bit) to extend. */
protected fun keySpecBuilder(): KeyGenParameterSpec.Builder = KeyGenParameterSpec.Builder(
protected fun keySpecBuilder(strongBox: Boolean): KeyGenParameterSpec.Builder = KeyGenParameterSpec.Builder(
alias,
KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
)
.setBlockModes(KeyProperties.BLOCK_MODE_GCM)
.setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
.setKeySize(AES_KEY_SIZE_BITS)
.apply {
// Bind the key to the StrongBox secure element when requested and supported (API 28+; minSdk
// is 29, so the guard is defensive). If the device has no StrongBox, generateKey() catches
// StrongBoxUnavailableException and retries with strongBox = false for a TEE-backed key.
if (strongBox && Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) setIsStrongBoxBacked(true)
}
private companion object {
const val ANDROID_KEYSTORE = "AndroidKeyStore"
@@ -139,5 +171,6 @@ abstract class AesGcmKeystoreCipher(private val alias: String, private val gener
const val IV_LENGTH = 12
const val TAG_BITS = 128
const val AES_KEY_SIZE_BITS = 256
const val TAG = "AesGcmKeystoreCipher"
}
}
@@ -5,7 +5,7 @@ import android.os.Build
import android.security.keystore.KeyGenParameterSpec
import android.security.keystore.KeyPermanentlyInvalidatedException
import android.security.keystore.UserNotAuthenticatedException
import android.util.Log
import org.libremail.reporting.AppLog
import javax.inject.Inject
import javax.inject.Singleton
@@ -43,7 +43,7 @@ class DatabaseKeyCipher @Inject constructor() :
override fun encrypt(plaintext: String): String = try {
super.encrypt(plaintext)
} catch (e: KeyPermanentlyInvalidatedException) {
Log.d(TAG, "replacing invalidated auth-bound key before sealing", e)
AppLog.d(TAG, "replacing invalidated auth-bound key before sealing", e)
deleteKey()
super.encrypt(plaintext)
}
@@ -60,16 +60,16 @@ class DatabaseKeyCipher @Inject constructor() :
initEncryptCipher(key)
false
} catch (e: KeyPermanentlyInvalidatedException) {
Log.d(TAG, "auth-bound database key invalidated", e)
AppLog.d(TAG, "auth-bound database key invalidated", e)
true
} catch (e: UserNotAuthenticatedException) {
// Valid key, just outside its time-bound auth window — not invalidated.
Log.d(TAG, "auth-bound key outside its auth window; not invalidated", e)
AppLog.d(TAG, "auth-bound key outside its auth window; not invalidated", e)
false
} catch (e: Exception) {
// Never let a validity probe crash the foreground pass; a real decrypt later surfaces any
// genuine problem. Treat an unknown probe failure as "not invalidated" (don't wipe).
Log.d(TAG, "auth-bound key validity probe failed; treating as valid", e)
AppLog.d(TAG, "auth-bound key validity probe failed; treating as valid", e)
false
}
}
@@ -82,8 +82,8 @@ class DatabaseKeyCipher @Inject constructor() :
/** A missing auth-bound key means it was invalidated; surface that instead of regenerating. */
override fun onMissingDecryptionKey(): Nothing = error("auth-bound database key is missing")
override fun keySpec(): KeyGenParameterSpec {
val builder = keySpecBuilder()
override fun keySpec(strongBox: Boolean): KeyGenParameterSpec {
val builder = keySpecBuilder(strongBox)
.setUserAuthenticationRequired(true)
.setInvalidatedByBiometricEnrollment(true)
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
@@ -18,7 +18,7 @@ import javax.inject.Singleton
@Singleton
class KeystoreCrypto @Inject constructor() : AesGcmKeystoreCipher(alias = KEY_ALIAS, generateKeyOnDecrypt = true) {
override fun keySpec(): KeyGenParameterSpec = keySpecBuilder().build()
override fun keySpec(strongBox: Boolean): KeyGenParameterSpec = keySpecBuilder(strongBox).build()
private companion object {
const val KEY_ALIAS = "libremail.master.key"
@@ -0,0 +1,60 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.security
import kotlinx.coroutines.flow.distinctUntilChanged
import kotlinx.coroutines.flow.map
import org.libremail.data.settings.SettingsRepository
import org.libremail.reporting.AppLog
import org.libremail.reporting.ReportEncryption
import javax.inject.Inject
import javax.inject.Singleton
/**
* On-device [ReportEncryption]: seals a persisted report's JSON with the non-auth Keystore master key
* ([KeystoreCrypto]) so at-rest report storage honours the opt-in `encryptCache` setting (issue #369).
* Reuses the vetted AES-256-GCM crypto rather than rolling new; encryption is `Base64(iv || ciphertext)`.
*
* The **master** key (not the auth-bound cache key) is deliberate: it is usable without a user-presence
* prompt, so a crash that occurs while the app is locked can still seal and persist its report — the
* ticket requires crash reports to survive, encrypted, even then.
*
* [enabled] is answered from an in-memory mirror of the `encryptCache` setting, never a live DataStore
* read: a crash-time [org.libremail.reporting.ReportStore.save] runs synchronously on the crashing
* thread and must not touch DataStore (#296). [observeEncryptCacheSetting], launched once at startup,
* keeps that mirror current so a mid-session toggle takes effect on the next report write. The mirror
* defaults to `false` (plaintext) until the first settings value lands — the same brief unwarmed
* startup window [org.libremail.reporting.DiagnosticsCollector] accepts for a crash report's settings.
*/
@Singleton
class KeystoreReportEncryption @Inject constructor(
private val crypto: KeystoreCrypto,
private val settingsRepository: SettingsRepository,
) : ReportEncryption {
@Volatile
private var encryptionEnabled: Boolean = false
override fun enabled(): Boolean = encryptionEnabled
override fun encrypt(plaintext: String): String = crypto.encrypt(plaintext)
override fun decrypt(encoded: String): String = crypto.decrypt(encoded)
/**
* Mirrors the `encryptCache` setting into [encryptionEnabled] for the process lifetime. Collects
* forever, so launch it once from application startup. PII-free — only the on/off state is logged.
*/
suspend fun observeEncryptCacheSetting() {
settingsRepository.settings
.map { it.encryptCache }
.distinctUntilChanged()
.collect { enabled ->
encryptionEnabled = enabled
AppLog.i(TAG, "Report at-rest encryption is now ${if (enabled) "ON" else "OFF"}")
}
}
private companion object {
const val TAG = "ReportEncryption"
}
}
@@ -5,9 +5,13 @@ import android.content.Context
import androidx.hilt.work.HiltWorker
import androidx.work.CoroutineWorker
import androidx.work.WorkerParameters
import dagger.Lazy
import dagger.assisted.Assisted
import dagger.assisted.AssistedInject
import kotlinx.coroutines.CancellationException
import org.libremail.BuildConfig
import org.libremail.data.security.EncryptedCacheGuard
import org.libremail.reporting.AppLog
/**
* Runs one bounded slice of the full-history backfill (issue #12). Cancellable (WorkManager stops it
@@ -19,16 +23,50 @@ import kotlinx.coroutines.CancellationException
class BackfillWorker @AssistedInject constructor(
@Assisted appContext: Context,
@Assisted workerParams: WorkerParameters,
private val backfiller: MailBackfiller,
// Lazy: resolving MailBackfiller builds the Room DB graph, whose first query blocks while the
// encrypted cache is locked. Resolve it only after the cache-lock check passes, so a locked run
// fails fast instead of parking this thread on an unsatisfiable passphrase await (mirrors SyncWorker).
private val backfiller: Lazy<MailBackfiller>,
private val cacheGuard: EncryptedCacheGuard,
) : CoroutineWorker(appContext, workerParams) {
override suspend fun doWork(): Result = runCatching {
// Chain bounded slices back-to-back while history remains, so a large mailbox isn't limited to
// one slice per periodic run. runBackfill() returns true while any folder still has pages left;
// isStopped lets WorkManager end a long run gracefully (the periodic schedule resumes it).
while (backfiller.runBackfill() && !isStopped) { /* page the next slice */ }
}.fold(
onSuccess = { Result.success() },
onFailure = { error -> if (error is CancellationException) throw error else Result.retry() },
)
override suspend fun doWork(): Result {
// Debug-only fetch gate (issue #393): a test harness can pause backfill via an adb broadcast so a
// genuinely uncached message-open can be measured (proactive backfill would otherwise warm the
// cache first). Skip-and-reschedule exactly like the cache-lock deferral below; WorkManager
// retries and picks up from the persisted per-folder boundary once the gate resumes. The whole
// branch is compiled out of release: BuildConfig.DEBUG is a compile-time `false` there, so R8
// strips it (and DebugFetchGate with it).
if (BuildConfig.DEBUG && DebugFetchGate.isPaused(FetchScope.BACKFILL)) {
AppLog.i(TAG, "backfill deferred: fetch-gate paused")
return Result.retry()
}
// Can't open the encrypted DB without the user present — retry later rather than parking a
// WorkManager thread (which also wedges the shared serial executor) on an unsatisfiable await.
if (cacheGuard.isCacheLocked()) {
AppLog.i(TAG, "backfill deferred: cache locked")
return Result.retry()
}
return runCatching {
// Chain bounded slices back-to-back while history remains, so a large mailbox isn't limited
// to one slice per periodic run. runBackfill() returns true while any folder still has pages
// left; isStopped lets WorkManager end a long run gracefully (the periodic schedule resumes).
val mailBackfiller = backfiller.get()
while (mailBackfiller.runBackfill() && !isStopped) { /* page the next slice */ }
}.fold(
onSuccess = {
AppLog.i(TAG, "backfill worker: success")
Result.success()
},
onFailure = { error ->
if (error is CancellationException) throw error
AppLog.w(TAG, "backfill worker: retry", error)
Result.retry()
},
)
}
private companion object {
const val TAG = "BackfillWorker"
}
}
@@ -0,0 +1,97 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.sync
/**
* The proactive-fetch activities a debug harness can pause via [DebugFetchGate] (issue #393). Only the
* two *proactive* activities are gateable: full-history [BACKFILL] paging ([BackfillWorker]) and the
* post-sync body [PREFETCH] ([MailSyncer]/[MailBackfiller] `prefetchIfEnabled`). Header sync and the
* on-demand message open are deliberately absent — they are **never** gated, so a paused gate can defer
* background caching without ever blocking new mail arriving or a user-triggered (uncached) open. The
* `all` wire alias ([ALL_ALIAS]) expands to every entry here.
*/
enum class FetchScope(val wireName: String) {
BACKFILL("backfill"),
PREFETCH("prefetch"),
;
companion object {
/** The `all` scope alias accepted on the adb wire — expands to every [FetchScope]. */
const val ALL_ALIAS = "all"
/**
* Parses the comma-separated `scope` extra of the debug broadcast (e.g. `"backfill,prefetch"`
* or `"all"`) into the set of scopes it names. Case- and whitespace-insensitive; the [ALL_ALIAS]
* expands to every scope; unrecognised or blank tokens are ignored; a null/blank input yields
* the empty set. Declaration order is preserved so the read-back string is stable.
*/
fun parse(raw: String?): Set<FetchScope> {
if (raw.isNullOrBlank()) return emptySet()
val tokens = raw.split(',').map { it.trim().lowercase() }.filter { it.isNotEmpty() }
if (tokens.contains(ALL_ALIAS)) return entries.toSet()
return entries.filterTo(LinkedHashSet()) { it.wireName in tokens }
}
}
}
/**
* In-memory, thread-safe holder of the currently-paused proactive-fetch [FetchScope]s — a **debug-only**
* test hook (issue #393) that lets an adb-driven perf harness pause background body caching so a genuine
* uncached message-open can be measured (the harness adds an account, lets headers sync, then opens a
* message that must hit the network rather than a warmed cache).
*
* Lives in `src/main` so the workers/syncer/backfiller can reference it, but **every read is wrapped in
* `if (BuildConfig.DEBUG && ...)`**. `BuildConfig.DEBUG` is a compile-time `false` in release, so R8
* dead-code-eliminates each such branch, leaving this object unreferenced and stripping it (and
* [FetchScope]) from the release APK entirely — verified by issue #393's release-exclusion check. The
* writer, `FetchGateReceiver`, lives wholly in `src/debug` and is never packaged into release either;
* this is the same source-set guarantee `ColdOpenCacheProbe` (#221) relies on.
*
* Defaults to **nothing paused**, so the gate is inert until a debug broadcast pauses a scope. Reads are
* lock-free (a `@Volatile` snapshot of an immutable set, cheap enough for the fetch hot path); the rare
* writes swap the reference under a lock.
*/
object DebugFetchGate {
private val writeLock = Any()
@Volatile
private var paused: Set<FetchScope> = emptySet()
/** Whether [scope]'s proactive fetch is currently paused. */
fun isPaused(scope: FetchScope): Boolean = scope in paused
/** The scopes currently paused, in [FetchScope] declaration order. */
fun pausedScopes(): Set<FetchScope> {
val snapshot = paused
return FetchScope.entries.filterTo(LinkedHashSet()) { it in snapshot }
}
/** Pauses [scopes] (union with whatever is already paused). No-op for an empty set. */
fun pause(scopes: Set<FetchScope>) {
if (scopes.isEmpty()) return
synchronized(writeLock) { paused = paused + scopes }
}
/** Resumes [scopes] (removes them from the paused set). No-op for an empty set. */
fun resume(scopes: Set<FetchScope>) {
if (scopes.isEmpty()) return
synchronized(writeLock) { paused = paused - scopes }
}
/** Clears every pause, restoring the default not-paused state. Used to isolate tests. */
fun reset() {
synchronized(writeLock) { paused = emptySet() }
}
/**
* The synchronous read-back string the debug receiver returns as ordered-broadcast result data —
* e.g. `"paused=[backfill,prefetch]"` (declaration order) or `"paused=[]"` when nothing is paused.
*/
fun pausedResult(): String {
val snapshot = paused
return FetchScope.entries.filter { it in snapshot }.joinToString(
separator = ",",
prefix = "paused=[",
postfix = "]",
) { it.wireName }
}
}
@@ -9,6 +9,7 @@ import kotlinx.coroutines.delay
import kotlinx.coroutines.ensureActive
import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import org.libremail.BuildConfig
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.dao.BackfillProgressDao
import org.libremail.data.local.dao.MessageDao
@@ -25,6 +26,8 @@ import org.libremail.domain.model.ImapConnectionParams
import org.libremail.domain.repository.MailRepository
import org.libremail.mail.ImapClient
import org.libremail.power.BatteryStatusProvider
import org.libremail.reporting.AppLog
import org.libremail.reporting.accountLogRef
import javax.inject.Inject
import javax.inject.Singleton
@@ -65,13 +68,19 @@ class MailBackfiller @Inject constructor(
* count as more work — it is retried on a future scheduled run instead of spun on back-to-back.
*/
suspend fun runBackfill(maxBatches: Int = DEFAULT_MAX_BATCHES): Boolean = maintenanceGate.mutex.withLock {
AppLog.i(TAG, "backfill slice: maxBatches=$maxBatches")
var remaining = maxBatches
var moreWork = false
for (account in accountDao.getAll().map { it.toDomain() }) {
accounts@ for (account in accountDao.getAll().map { it.toDomain() }) {
val params = runCatching { connectionFactory.imapParamsFor(account) }.getOrNull() ?: continue
val policy = accountSettingsRepository.effectiveRetention(settingsRepository, account.id)
for (folder in messageDao.syncedFolders(account.id)) {
if (remaining <= 0) return@withLock true
if (remaining <= 0) {
// The budget ran out before every folder was visited, so there is very likely more
// work left even though nothing here reported it directly.
moreWork = true
break@accounts
}
// Per-folder failures (e.g. a transient server error) must not abort the whole slice.
val result = runCatching { backfillFolder(account, params, folder, policy, remaining) }
.getOrElse { FolderResult(batches = 0, moreWork = true) }
@@ -79,6 +88,7 @@ class MailBackfiller @Inject constructor(
if (result.moreWork) moreWork = true
}
}
AppLog.i(TAG, "backfill slice done: moreWork=$moreWork")
moreWork
}
@@ -153,6 +163,8 @@ class MailBackfiller @Inject constructor(
delay(BACKFILL_BATCH_DELAY_MS)
}
if (complete) markComplete(account.id, folder, beforeUid)
val folderLabel = logSafeFolderLabel(folder)
AppLog.d(TAG, "backfill ${accountLogRef(account.id)} folder=$folderLabel pages=$batches complete=$complete")
return FolderResult(batches, moreWork = !complete && !stalled)
}
@@ -213,6 +225,14 @@ class MailBackfiller @Inject constructor(
* fetched is filled in lazily when the message is opened.
*/
private suspend fun prefetchIfEnabled(ids: List<String>) {
// Debug-only fetch gate (issue #393): pause proactive body prefetch so a later open is a genuine
// uncached fetch. Header paging above is untouched (its own gate is the BackfillWorker entry), so
// history still lands; a skipped body is filled in lazily on open. Compiled out of release
// (BuildConfig.DEBUG is a compile-time false, so R8 drops the branch).
if (BuildConfig.DEBUG && DebugFetchGate.isPaused(FetchScope.PREFETCH)) {
AppLog.i(TAG, "prefetch skipped: fetch-gate paused")
return
}
val shouldPrefetch = SyncResourcePolicy.shouldPrefetchContent(
policy = settingsRepository.fetchPolicy(),
unmetered = { context.isActiveNetworkUnmetered() },
@@ -226,6 +246,8 @@ class MailBackfiller @Inject constructor(
}
private companion object {
const val TAG = "MailBackfiller"
/** Headers fetched per server page. */
const val BACKFILL_BATCH_SIZE = 50
@@ -13,6 +13,7 @@ import org.libremail.data.settings.AccountSettingsRepository
import org.libremail.data.settings.RetentionPolicy
import org.libremail.data.settings.SettingsRepository
import org.libremail.data.settings.effectiveRetention
import org.libremail.reporting.AppLog
import javax.inject.Inject
import javax.inject.Singleton
@@ -25,6 +26,11 @@ import javax.inject.Singleton
* Precedence with the #12 backfill is guaranteed two ways: backfill stops paging at the same
* retention floor this pruner deletes below (their working sets are disjoint), and both jobs share
* [MailMaintenanceGate] so they never run at once.
*
* Foreground sync ([MailSyncer]) is aligned the same way in BOTH retention modes so it never
* re-inserts what this pruner deletes: it caps its fetch window to the retention count AND drops
* anything older than the age cutoff before persisting (#193). The two limits are independent, so
* both bounds apply together.
*/
@Singleton
class MailPruner @Inject constructor(
@@ -44,6 +50,7 @@ class MailPruner @Inject constructor(
if (policy.isUnlimited) continue
removed += pruneAccount(account.id, policy, nowMillis)
}
AppLog.i(TAG, "prune done: removed=$removed")
removed
}
@@ -77,6 +84,8 @@ class MailPruner @Inject constructor(
}
private companion object {
const val TAG = "MailPruner"
/** Ids per DELETE, kept under SQLite's 999-host-parameter limit on older Android. */
const val DELETE_CHUNK = 500
}
@@ -9,6 +9,7 @@ import kotlinx.coroutines.ensureActive
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import org.libremail.BuildConfig
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.dao.MessageDao
import org.libremail.data.local.toDomain
@@ -21,6 +22,8 @@ import org.libremail.domain.repository.MailRepository
import org.libremail.mail.ImapClient
import org.libremail.notifications.MailNotifier
import org.libremail.power.BatteryStatusProvider
import org.libremail.reporting.AppLog
import org.libremail.reporting.accountLogRef
import javax.inject.Inject
import javax.inject.Singleton
@@ -47,6 +50,7 @@ class MailSyncer @Inject constructor(
/** Syncs every account's inbox. Succeeds if at least one account synced (or there are none). */
override suspend fun syncAll(): Result<Int> {
val accounts = accountDao.getAll().map { it.toDomain() }
AppLog.i(TAG, "sync all: ${accounts.size} accounts")
if (accounts.isEmpty()) return Result.success(0)
val result = syncMutex.withLock {
@@ -64,6 +68,8 @@ class MailSyncer @Inject constructor(
}
if (anySuccess || firstError == null) Result.success(total) else Result.failure(firstError)
}
result.onSuccess { total -> AppLog.i(TAG, "sync all done: fetched=$total") }
.onFailure { error -> AppLog.w(TAG, "sync all failed", error) }
if (result.isSuccess) accounts.forEach { prefetchIfEnabled(it, INBOX) }
return result
}
@@ -87,11 +93,19 @@ class MailSyncer @Inject constructor(
private suspend fun syncFolderHeaders(account: Account, folder: String, notify: Boolean): Result<Int> =
runCatching {
val params = connectionFactory.imapParamsFor(account)
val policy = accountSettingsRepository.effectiveRetention(settingsRepository, account.id)
// Never fetch more of the recent window than device-only retention (#13) would keep. Without
// this, a count limit BELOW the window would make foreground sync re-download the same rows
// the pruner just trimmed, on every sync — an endless re-download/re-prune fight.
val fetched = imapClient.fetchRecent(params, folder, recentWindowFor(account)) // cancellable network I/O
val window = policy.countLimit?.let { minOf(FETCH_LIMIT, it) } ?: FETCH_LIMIT
val fetched = imapClient.fetchRecent(params, folder, window) // cancellable network I/O
// Age-based retention (#193): drop anything older than the age cutoff before persisting. On a
// low-traffic mailbox the newest-N can extend PAST the cutoff, so without this a sync re-inserts
// rows the age pruner just deleted and the next prune deletes them again — a churn loop. Count/
// unlimited modes have a null cutoff and keep the full window, so their behavior is unchanged.
val cutoff = policy.ageCutoffMillis(System.currentTimeMillis())
val entities = fetched.map { it.toEntity(account.id, folder) }
.let { mapped -> if (cutoff == null) mapped else mapped.filter { it.timestampMillis >= cutoff } }
// Persist and notify atomically with respect to cancellation: an IDLE renewal that cancels
// mid-sync must not drop a notification (the rows would then look "already seen" next time).
@@ -104,26 +118,21 @@ class MailSyncer @Inject constructor(
entities.filter { it.id !in existingIds && !it.isRead }
}
if (entities.isEmpty()) {
if (fetched.isEmpty()) {
// An empty recent window means the server folder itself is empty, so nothing (not
// even backfilled history) should remain cached for it.
// even backfilled history) should remain cached for it. Keyed on the raw fetch, not the
// age-filtered set: a folder holding only mail older than the age cutoff is NOT empty on
// the server, so its stale local rows are left to the pruner rather than wiped here.
messageDao.deleteSyncedByAccountFolder(account.id, folder)
} else {
val ids = entities.map { it.id }
messageDao.insertNew(entities)
// Mark every fetched message as synced (upgrades any former search-only row) and refresh
// its display fields — without touching cached bodies or optimistic read/star flags.
// its display fields — without touching cached bodies or optimistic read/star flags. The
// per-row refreshes run in a single transaction (issue #310) so a whole recent window
// costs one commit instead of one fsync per message (amplified on the encrypted cache).
messageDao.markSynced(ids)
entities.forEach {
messageDao.updateHeaderContent(
id = it.id,
sender = it.sender,
senderEmail = it.senderEmail,
subject = it.subject,
timestampMillis = it.timestampMillis,
uid = it.uid,
)
}
messageDao.updateHeaderContents(entities)
// Reconcile server-side deletions ONLY within the fetched recent-UID window, so older
// history paged in by the background backfill (issue #12) survives each foreground sync
// instead of being wiped by a whole-folder "not in the recent 50" delete. Bound the
@@ -144,19 +153,11 @@ class MailSyncer @Inject constructor(
notifier.notifyNewMail(account, newMessages.sortedByDescending { it.timestampMillis })
}
}
val folderLabel = logSafeFolderLabel(folder)
AppLog.d(TAG, "sync ${accountLogRef(account.id)} folder=$folderLabel fetched=${fetched.size}")
fetched.size
}
/**
* The number of recent headers to fetch: the standard [FETCH_LIMIT], but capped by the account's
* effective device-only retention count so foreground sync never re-downloads rows the pruner
* would immediately trim. Age-only or unlimited retention leaves the full window in place.
*/
private suspend fun recentWindowFor(account: Account): Int {
val policy = accountSettingsRepository.effectiveRetention(settingsRepository, account.id)
return policy.countLimit?.let { minOf(FETCH_LIMIT, it) } ?: FETCH_LIMIT
}
/**
* Aggressively pre-caches each not-yet-fetched message's full content (body + attachments) per the
* user's fetch policy, pausing at low battery regardless of policy — see
@@ -166,6 +167,14 @@ class MailSyncer @Inject constructor(
* and is cancellable between messages so an IDLE renewal stops it promptly.
*/
private suspend fun prefetchIfEnabled(account: Account, folder: String) {
// Debug-only fetch gate (issue #393): a test harness pauses proactive body prefetch so a later
// open does a genuine uncached fetch. Header sync above already ran, so mail still arrives; the
// skipped prefetch is filled in lazily on open, exactly as the low-battery pause behaves. Compiled
// out of release (BuildConfig.DEBUG is a compile-time false, so R8 drops the branch).
if (BuildConfig.DEBUG && DebugFetchGate.isPaused(FetchScope.PREFETCH)) {
AppLog.i(TAG, "prefetch skipped: fetch-gate paused")
return
}
val shouldPrefetch = SyncResourcePolicy.shouldPrefetchContent(
policy = settingsRepository.fetchPolicy(),
unmetered = { context.isActiveNetworkUnmetered() },
@@ -179,6 +188,7 @@ class MailSyncer @Inject constructor(
}
private companion object {
const val TAG = "MailSyncer"
const val INBOX = "INBOX"
/**
@@ -5,8 +5,11 @@ import android.content.Context
import androidx.hilt.work.HiltWorker
import androidx.work.CoroutineWorker
import androidx.work.WorkerParameters
import dagger.Lazy
import dagger.assisted.Assisted
import dagger.assisted.AssistedInject
import org.libremail.data.security.EncryptedCacheGuard
import org.libremail.reporting.AppLog
/**
* Enforces device-only retention (issue #13) by running [MailPruner]. Purely local — it never
@@ -16,11 +19,33 @@ import dagger.assisted.AssistedInject
class PruneWorker @AssistedInject constructor(
@Assisted appContext: Context,
@Assisted workerParams: WorkerParameters,
private val pruner: MailPruner,
// Lazy: resolving MailPruner builds the Room DB graph, whose first query blocks while the encrypted
// cache is locked. Resolve it only after the cache-lock check passes, so a locked run fails fast
// instead of parking this thread on an unsatisfiable passphrase await (mirrors SyncWorker/SendWorker).
private val pruner: Lazy<MailPruner>,
private val cacheGuard: EncryptedCacheGuard,
) : CoroutineWorker(appContext, workerParams) {
override suspend fun doWork(): Result = runCatching { pruner.prune() }.fold(
onSuccess = { Result.success() },
onFailure = { Result.retry() },
)
override suspend fun doWork(): Result {
// Can't open the encrypted DB without the user present — retry later rather than parking a
// WorkManager thread (which also wedges the shared serial executor) on an unsatisfiable await.
if (cacheGuard.isCacheLocked()) {
AppLog.i(TAG, "prune deferred: cache locked")
return Result.retry()
}
return runCatching { pruner.get().prune() }.fold(
onSuccess = {
AppLog.i(TAG, "prune worker: success")
Result.success()
},
onFailure = { error ->
AppLog.w(TAG, "prune worker: retry", error)
Result.retry()
},
)
}
private companion object {
const val TAG = "PruneWorker"
}
}
@@ -2,15 +2,16 @@
package org.libremail.data.sync
import android.content.Context
import android.util.Log
import androidx.hilt.work.HiltWorker
import androidx.work.CoroutineWorker
import androidx.work.WorkerParameters
import dagger.Lazy
import dagger.assisted.Assisted
import dagger.assisted.AssistedInject
import org.libremail.data.attachment.AttachmentUriGrants
import org.libremail.data.local.dao.AccountDao
import org.libremail.data.local.dao.OutboxDao
import org.libremail.data.local.entity.OutboxEntity
import org.libremail.data.local.toDomain
import org.libremail.data.local.toOutgoingAttachments
import org.libremail.data.security.EncryptedCacheGuard
@@ -22,6 +23,8 @@ import org.libremail.mail.GraphSendException
import org.libremail.mail.GraphSender
import org.libremail.mail.SendableAttachment
import org.libremail.mail.SmtpSender
import org.libremail.reporting.AppLog
import org.libremail.reporting.accountLogRef
import java.io.File
import kotlin.coroutines.cancellation.CancellationException
@@ -39,6 +42,8 @@ class SendWorker @AssistedInject constructor(
private val graphSender: GraphSender,
private val connectionFactory: Lazy<MailConnectionFactory>,
private val cacheGuard: EncryptedCacheGuard,
// Lazy for the same reason as the DAOs above: resolving it touches the Room DB.
private val attachmentUriGrants: Lazy<AttachmentUriGrants>,
) : CoroutineWorker(appContext, workerParams) {
private companion object {
@@ -50,64 +55,92 @@ class SendWorker @AssistedInject constructor(
val outboxDao = this.outboxDao.get()
val accountDao = this.accountDao.get()
val connectionFactory = this.connectionFactory.get()
val attachmentUriGrants = this.attachmentUriGrants.get()
val pending = outboxDao.getAll()
if (pending.isEmpty()) return Result.success()
AppLog.i(TAG, "outbox drain: ${pending.size} queued")
var anyFailed = false
for (entity in pending) {
val attachmentDir = File(applicationContext.cacheDir, "outbox/${entity.id}")
val account = accountDao.getById(entity.accountId)?.toDomain()
if (account == null) {
outboxDao.delete(entity.id) // account removed — drop the queued message
attachmentDir.deleteRecursively()
continue
}
runCatching {
val message = OutgoingMessage(
accountId = entity.accountId,
to = entity.toAddresses,
cc = entity.ccAddresses,
bcc = entity.bccAddresses,
subject = entity.subject,
body = entity.body,
bodyHtml = entity.bodyHtml,
)
val attachments = stagedAttachments(attachmentDir, entity.attachments.toOutgoingAttachments())
if (account.authType == AuthType.OAUTH_OUTLOOK) {
sendOutlook(connectionFactory, account, message, attachments)
} else {
smtpSender.send(
connectionFactory.smtpParamsFor(account),
from = account.email,
message = message,
attachments = attachments,
)
}
}.fold(
onSuccess = {
outboxDao.delete(entity.id)
attachmentDir.deleteRecursively()
},
onFailure = { e ->
if (e is GraphSendException && e.mayHaveSent) {
// Graph may already have delivered this; auto-retrying (or any other send)
// would duplicate it, so leave it queued with a clear status and let the
// user decide. Not counted as a failure, so WorkManager won't auto-retry.
outboxDao.setError(
entity.id,
"Send status unknown — check your Sent folder, then retry or cancel",
)
} else {
outboxDao.setError(entity.id, e.message)
anyFailed = true
}
},
)
val failed = sendQueued(entity, outboxDao, accountDao, connectionFactory, attachmentUriGrants)
if (failed) anyFailed = true
}
// Retry (with WorkManager backoff) so failed sends are reattempted when conditions improve.
return if (anyFailed) Result.retry() else Result.success()
}
/**
* Sends one queued [entity] — or drops it if its account was removed — updating the outbox row
* and releasing its staged attachment grant. Returns true if the send genuinely failed (should
* count toward a WorkManager retry); the ambiguous "may have sent" Graph case returns false, since
* it is deliberately left queued rather than retried (see [sendOutlook]).
*/
private suspend fun sendQueued(
entity: OutboxEntity,
outboxDao: OutboxDao,
accountDao: AccountDao,
connectionFactory: MailConnectionFactory,
attachmentUriGrants: AttachmentUriGrants,
): Boolean {
val attachmentDir = File(applicationContext.cacheDir, "outbox/${entity.id}")
val account = accountDao.getById(entity.accountId)?.toDomain()
if (account == null) {
outboxDao.delete(entity.id) // account removed — drop the queued message
attachmentDir.deleteRecursively()
attachmentUriGrants.releaseUnreferenced(entity.attachmentUris())
return false
}
var failed = false
runCatching {
val message = OutgoingMessage(
accountId = entity.accountId,
to = entity.toAddresses,
cc = entity.ccAddresses,
bcc = entity.bccAddresses,
subject = entity.subject,
body = entity.body,
bodyHtml = entity.bodyHtml,
)
val attachments = stagedAttachments(attachmentDir, entity.attachments.toOutgoingAttachments())
if (account.authType == AuthType.OAUTH_OUTLOOK) {
sendOutlook(connectionFactory, account, message, attachments)
} else {
smtpSender.send(
connectionFactory.smtpParamsFor(account),
from = account.email,
message = message,
attachments = attachments,
)
}
}.fold(
onSuccess = {
outboxDao.delete(entity.id)
attachmentDir.deleteRecursively()
// The picked bytes were staged at enqueue; with the row sent, drop the persistable
// grant unless a live draft/outbox row still references the same URI (security review).
attachmentUriGrants.releaseUnreferenced(entity.attachmentUris())
val via = if (account.authType == AuthType.OAUTH_OUTLOOK) "Graph" else "SMTP"
AppLog.i(TAG, "sent ${accountLogRef(account.id)} via $via")
},
onFailure = { e ->
if (e is GraphSendException && e.mayHaveSent) {
// Graph may already have delivered this; auto-retrying (or any other send) would
// duplicate it, so leave it queued with a clear status and let the user decide.
// Not counted as a failure, so WorkManager won't auto-retry.
outboxDao.setError(
entity.id,
"Send status unknown — check your Sent folder, then retry or cancel",
)
} else {
outboxDao.setError(entity.id, e.message)
failed = true
AppLog.w(TAG, "send failed for ${accountLogRef(account.id)}; will retry")
}
},
)
return failed
}
/**
* Outlook prefers Microsoft Graph. Fall back to SMTP only when Graph definitely did NOT send
* (a rejection, a pre-send/transport error, or a token failure); never fall back when the Graph
@@ -134,7 +167,7 @@ class SendWorker @AssistedInject constructor(
throw e
} catch (e: Exception) {
// Graph was never reached (e.g. token refresh failed) — SMTP cannot duplicate it.
Log.w(TAG, "Graph send failed for ${account.email}; falling back to SMTP", e)
AppLog.w(TAG, "Graph send failed for ${accountLogRef(account.id)}; falling back to SMTP", e)
smtpSender.send(
connectionFactory.smtpParamsFor(account),
from = account.email,
@@ -167,4 +200,7 @@ class SendWorker @AssistedInject constructor(
?.sortedBy { it.name.toIntOrNull() ?: Int.MAX_VALUE }
?.mapNotNull { it.listFiles()?.firstOrNull() }
.orEmpty()
/** The picked content-URIs this queued message was built from, for releasing their persistable grants. */
private fun OutboxEntity.attachmentUris(): List<String> = attachments.toOutgoingAttachments().map { it.uri }
}
@@ -0,0 +1,45 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package org.libremail.data.sync
/**
* The folder name safe to write to an [org.libremail.reporting.AppLog] breadcrumb (and so, in turn, a
* submitted [org.libremail.reporting.DebugReport]): the leaf name itself for a known **system** folder
* (INBOX, Sent, Drafts, Trash, Spam/Junk, Archive — including the alternate names real IMAP servers use
* for them), or a fixed placeholder for anything else. A user-created folder or label (e.g. a client
* name or project) can be PII-ish, so only this fixed, closed set of well-known names is ever logged
* verbatim; every other folder logs as the placeholder, regardless of nesting or the server's hierarchy
* delimiter. Matching is name-only — no server SPECIAL-USE attributes are available down here at the
* sync layer — so it is necessarily best-effort in the same way
* [org.libremail.domain.model.FolderRole.roleOf]'s display-name fallback is. That is the safe direction:
* a false negative just logs the placeholder, never a leaked name.
*/
internal fun logSafeFolderLabel(folder: String): String {
val leaf = folder.substringAfterLast('/').substringAfterLast('.').trim()
return if (leaf.lowercase() in SYSTEM_FOLDER_NAMES) leaf else FOLDER_PLACEHOLDER
}
private const val FOLDER_PLACEHOLDER = "<folder>"
/** Case-insensitive leaf names recognized as provider-supplied system folders, never user-created. */
private val SYSTEM_FOLDER_NAMES = setOf(
"inbox",
"sent",
"sent mail",
"sent items",
"sent messages",
"drafts",
"draft",
"junk",
"spam",
"junk e-mail",
"junk email",
"bulk mail",
"trash",
"deleted",
"deleted items",
"deleted messages",
"bin",
"archive",
"archives",
"all mail",
)

Some files were not shown because too many files have changed in this diff Show More