From e436503eaf3d7b124a6f72cc5a39a81978af7baf Mon Sep 17 00:00:00 2001 From: Jason Ross Date: Mon, 6 Jul 2026 13:21:34 -0500 Subject: [PATCH] feat(scripts): device-testing perf harness Add a cross-platform, standard-library-only Python package under scripts/device-testing/ that replicates LibreMail's on-device performance-test scenarios and logging capture, codifying the methodology run by hand on 2026-07-05 (Pixel 10 Pro XL). Modules: - breadcrumbs.py: a pure, unit-tested parser for the ImapPerf / MailReader / Reader / MailBackfiller breadcrumbs, plus open-correlation that reproduces the manual timing-tables.md figures exactly. - adb.py: a safety-guarded adb wrapper -- an allow-list of adb subcommands and a deny-list + assertions on shell commands. The only sanctioned app-state mutation is clearing LibreMail's own cache/ (exact-match); no pm clear / uninstall / data wipe, and no touching databases/ files/ shared_prefs/ datastore/ can be constructed. - uidump.py: uiautomator XML parser + screen recognition (mailbox rows with the cached "Available offline" flag, reader, and the keyguard / foreign-app guards). - scenarios.py: cold-open, message-open (uncached), back-nav, prefetch A/B (fetch-policy toggle) and cross-provider, each keyguard-guarded and driven through the guarded wrapper. - report.py + perf_harness.py: aggregates, a timing-tables.md renderer mirroring the manual write-up, and the CLI (timestamped run dir with the raw logcat, a filtered breadcrumb extract, and the tables). --dry-run prints the exact command plan without touching device state. Tests (stdlib unittest, 68 cases) validate the parser, guardrails, UI recognition and report against the manual run's real captures (fixtures include a verbatim perf-extract slice and the reader/lockscreen/alarm dumps). Track the *.log fixture past the gitignore *.log rule via a scoped negation. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 2 + scripts/device-testing/README.md | 159 +++++++ scripts/device-testing/adb.py | 370 +++++++++++++++ scripts/device-testing/breadcrumbs.py | 422 +++++++++++++++++ scripts/device-testing/perf_harness.py | 258 ++++++++++ scripts/device-testing/report.py | 253 ++++++++++ scripts/device-testing/scenarios.py | 445 ++++++++++++++++++ scripts/device-testing/tests/__init__.py | 2 + .../tests/fixtures/perf-extract-sample.log | 17 + .../tests/fixtures/ui_alarm.xml | 1 + .../tests/fixtures/ui_lockscreen.xml | 1 + .../tests/fixtures/ui_mailbox.xml | 1 + .../tests/fixtures/ui_reader.xml | 1 + .../device-testing/tests/test_adb_safety.py | 160 +++++++ .../device-testing/tests/test_breadcrumbs.py | 222 +++++++++ scripts/device-testing/tests/test_report.py | 109 +++++ scripts/device-testing/tests/test_uidump.py | 108 +++++ scripts/device-testing/uidump.py | 230 +++++++++ 18 files changed, 2761 insertions(+) create mode 100644 scripts/device-testing/README.md create mode 100644 scripts/device-testing/adb.py create mode 100644 scripts/device-testing/breadcrumbs.py create mode 100644 scripts/device-testing/perf_harness.py create mode 100644 scripts/device-testing/report.py create mode 100644 scripts/device-testing/scenarios.py create mode 100644 scripts/device-testing/tests/__init__.py create mode 100644 scripts/device-testing/tests/fixtures/perf-extract-sample.log create mode 100644 scripts/device-testing/tests/fixtures/ui_alarm.xml create mode 100644 scripts/device-testing/tests/fixtures/ui_lockscreen.xml create mode 100644 scripts/device-testing/tests/fixtures/ui_mailbox.xml create mode 100644 scripts/device-testing/tests/fixtures/ui_reader.xml create mode 100644 scripts/device-testing/tests/test_adb_safety.py create mode 100644 scripts/device-testing/tests/test_breadcrumbs.py create mode 100644 scripts/device-testing/tests/test_report.py create mode 100644 scripts/device-testing/tests/test_uidump.py create mode 100644 scripts/device-testing/uidump.py diff --git a/.gitignore b/.gitignore index 4bb314e..0a16c85 100644 --- a/.gitignore +++ b/.gitignore @@ -30,6 +30,8 @@ secrets.properties # Log Files *.log +# ...but keep the device-testing parser fixtures (verbatim logcat slices used as test inputs) +!scripts/device-testing/tests/fixtures/*.log # Android Studio / IntelliJ .idea/ diff --git a/scripts/device-testing/README.md b/scripts/device-testing/README.md new file mode 100644 index 0000000..506ec6c --- /dev/null +++ b/scripts/device-testing/README.md @@ -0,0 +1,159 @@ + +# LibreMail device-testing perf harness + +A cross-platform, **standard-library-only** Python tool that replicates LibreMail's +on-device performance-test scenarios and logging capture. It codifies the methodology that +was first run by hand (2026-07-05, Pixel 10 Pro XL) and written up in +`scratchpad/perf/perf_summary.md`, `causation-report.md`, and `ab-run/timing-tables.md`. + +Everything runs against **LibreMail only** plus read-only system-log/settings/dumpsys +collection, behind hard device-safety guardrails (see [Device safety](#device-safety)). + +## Requirements + +- **Python 3.8+** (standard library only — no `pip install`, no third-party deps). +- **`adb`** on `PATH` (or pass `--adb /path/to/adb`). +- A connected device with: + - **LibreMail installed** as a **debuggable** build (the cache-clear uses `run-as`, which + only works on debuggable APKs), and + - **at least one account signed in** with some **not-yet-cached** messages in the inbox + (message bodies are fetched on first open), and + - the screen **unlocked** (the harness keeps it awake during a run and guards every step + against the keyguard, but it cannot get you *past* a secure lock screen). + +No build step. Run it straight from the repo. + +## Usage + +```bash +python scripts/device-testing/perf_harness.py [options] +``` + +Scenarios (each independently selectable): + +| Scenario | What it does | +|------------------|--------------| +| `cold-open` | Force-stop LibreMail, clear **only** its `cache/`, `am start -W` ×N, parse `TotalTime`/`WaitTime`. | +| `message-open` | Open N distinct **uncached** messages one at a time; time spinner→content from the breadcrumbs. | +| `back-nav` | Time reader→mailbox back transitions ×N (dump-latency-bound; see caveat in the report). | +| `prefetch-ab` | Run `message-open` under **Fetch all on Wi-Fi** (prefetch ON) vs **Always on-demand** (prefetch OFF), cache cleared between conditions. | +| `cross-provider` | Open N messages from the (unified) inbox and tabulate per provider (`imap:…` vs `outlook:…`) from the breadcrumb account refs. | + +Common options: + +| Option | Default | Meaning | +|--------|---------|---------| +| `--serial ` | auto (if exactly one device) | choose the device | +| `-n, --count ` | per-scenario | samples / runs (per condition for `prefetch-ab`) | +| `--out ` | `./device-perf-runs` | output root; a timestamped subdir is created per run | +| `--package ` | `org.libremail.app` | target package | +| `--component ` | `org.libremail.app/org.libremail.MainActivity` | launcher component | +| `--adb ` | `adb` | path to the adb executable | +| `--dry-run` | off | **print the exact command plan without changing device state** | + +**Always start with `--dry-run`** to review the command plan a scenario will issue: + +```bash +python scripts/device-testing/perf_harness.py prefetch-ab --dry-run +python scripts/device-testing/perf_harness.py cold-open -n 5 --serial 5C310DLCQ000G3 +``` + +## What each scenario measures + +Timing comes **primarily from the on-device breadcrumbs** (PII-free), with uiautomator used +only as a "content is ready" signal so the driver knows when to move on: + +- `MailReader: openMessage folder=