diff --git a/.gitignore b/.gitignore index 4bb314e..0a16c85 100644 --- a/.gitignore +++ b/.gitignore @@ -30,6 +30,8 @@ secrets.properties # Log Files *.log +# ...but keep the device-testing parser fixtures (verbatim logcat slices used as test inputs) +!scripts/device-testing/tests/fixtures/*.log # Android Studio / IntelliJ .idea/ diff --git a/scripts/device-testing/README.md b/scripts/device-testing/README.md new file mode 100644 index 0000000..506ec6c --- /dev/null +++ b/scripts/device-testing/README.md @@ -0,0 +1,159 @@ + +# LibreMail device-testing perf harness + +A cross-platform, **standard-library-only** Python tool that replicates LibreMail's +on-device performance-test scenarios and logging capture. It codifies the methodology that +was first run by hand (2026-07-05, Pixel 10 Pro XL) and written up in +`scratchpad/perf/perf_summary.md`, `causation-report.md`, and `ab-run/timing-tables.md`. + +Everything runs against **LibreMail only** plus read-only system-log/settings/dumpsys +collection, behind hard device-safety guardrails (see [Device safety](#device-safety)). + +## Requirements + +- **Python 3.8+** (standard library only — no `pip install`, no third-party deps). +- **`adb`** on `PATH` (or pass `--adb /path/to/adb`). +- A connected device with: + - **LibreMail installed** as a **debuggable** build (the cache-clear uses `run-as`, which + only works on debuggable APKs), and + - **at least one account signed in** with some **not-yet-cached** messages in the inbox + (message bodies are fetched on first open), and + - the screen **unlocked** (the harness keeps it awake during a run and guards every step + against the keyguard, but it cannot get you *past* a secure lock screen). + +No build step. Run it straight from the repo. + +## Usage + +```bash +python scripts/device-testing/perf_harness.py [options] +``` + +Scenarios (each independently selectable): + +| Scenario | What it does | +|------------------|--------------| +| `cold-open` | Force-stop LibreMail, clear **only** its `cache/`, `am start -W` ×N, parse `TotalTime`/`WaitTime`. | +| `message-open` | Open N distinct **uncached** messages one at a time; time spinner→content from the breadcrumbs. | +| `back-nav` | Time reader→mailbox back transitions ×N (dump-latency-bound; see caveat in the report). | +| `prefetch-ab` | Run `message-open` under **Fetch all on Wi-Fi** (prefetch ON) vs **Always on-demand** (prefetch OFF), cache cleared between conditions. | +| `cross-provider` | Open N messages from the (unified) inbox and tabulate per provider (`imap:…` vs `outlook:…`) from the breadcrumb account refs. | + +Common options: + +| Option | Default | Meaning | +|--------|---------|---------| +| `--serial ` | auto (if exactly one device) | choose the device | +| `-n, --count ` | per-scenario | samples / runs (per condition for `prefetch-ab`) | +| `--out ` | `./device-perf-runs` | output root; a timestamped subdir is created per run | +| `--package ` | `org.libremail.app` | target package | +| `--component ` | `org.libremail.app/org.libremail.MainActivity` | launcher component | +| `--adb ` | `adb` | path to the adb executable | +| `--dry-run` | off | **print the exact command plan without changing device state** | + +**Always start with `--dry-run`** to review the command plan a scenario will issue: + +```bash +python scripts/device-testing/perf_harness.py prefetch-ab --dry-run +python scripts/device-testing/perf_harness.py cold-open -n 5 --serial 5C310DLCQ000G3 +``` + +## What each scenario measures + +Timing comes **primarily from the on-device breadcrumbs** (PII-free), with uiautomator used +only as a "content is ready" signal so the driver knows when to move on: + +- `MailReader: openMessage folder=