Merge pull request #73 from JMR-dev/docs/instrumented-tests-correction

Say where instrumented tests run, instead of where they used to not run
This commit was merged in pull request #73.
This commit is contained in:
Jason Ross
2026-08-24 17:17:18 -05:00
committed by GitHub
+23 -11
View File
@@ -64,16 +64,28 @@ of the first one that fails.
on this machine (below), so without it an androidTest compile error is not discovered until CI.
ktlint and detekt also cover the `test`/`androidTest` source sets that `lintDebug` skips.
## Instrumented tests do not run locally
## Instrumented tests: where they actually run
Two independent reasons, so do not spend time on either:
This section said the opposite until 2026-08-24, and both of its claims had been false for two
days. Read it as the current answer, and see the git history if you need the old one.
- **Emulators segfault on this host.** qemu dies on every AVD. Instrumented tests run on CI or on
the physical Pixel, never in a local emulator.
- **The API 37 image is broken.** `android-37.0` crash-loops surfaceflinger inside its own gralloc
mapper, so every test fails there regardless of this app. `docs/api-37-emulator-crash.md` records
the evidence and the ruled-out fixes; CI's matrix therefore stops at API 36 even though targetSdk
is 37. **API 37 needs a manual check on the Pixel 10 Pro XL before each release.**
- **Local emulators work, for API 33-36.** `tools/local-emulator/run-e2e.sh` runs them on this
host. The segfault that made this look impossible was not a broken machine: SwiftShader's Reactor
JIT writes generated shader code onto the heap and executes it, Fedora's SELinux policy denies
`execheap`, and qemu dies. Choosing a different renderer avoids it entirely — `-gpu host`,
`angle_indirect` and `swangle_indirect` all boot, while `auto`, `off`, `guest` and
`swiftshader_indirect` do not. `docs/local-emulator.md` has the evidence and the per-API renderer
table.
- **CI runs API 37, and it gates.** The matrix is 33/34/35/36/37. Two Media3 hardware-transcode
tests fail inside the emulator's own `c2.goldfish.h264.decoder` rather than on anything this app
does; they carry `@FailsOnEmulatorApi37` and run in a separate `continue-on-error` job,
`E2E API 37 Media3 hardware transcode (advisory)`. The gating leg runs the other 55.
**That advisory job is red on every PR, by design** — do not read it as your change breaking
something, and do not read a green run as evidence those two tests pass.
`docs/api-37-emulator-crash.md` has the measurements.
Still true, and the reason the advisory job is not simply deleted: **API 37 needs a manual check on
the Pixel 10 Pro XL before each release.** The advisory pair is the one thing CI cannot answer for.
On a device or emulator, build only the ABI it can execute:
@@ -112,9 +124,9 @@ install for code that can never run — and on API 37 the full APK does not fit
documents the reasoning — turns "needs a device" into "a pure function plus a thin edge".
Robolectric is in the JVM source set, `compose-ui-test-junit4` with it, so Compose screens are
unit testable too. Reach for the seam before concluding something cannot be unit tested.
- **E2E is runnable locally now.** `tools/local-emulator/run-e2e.sh` runs API 33-36 on this
machine; see `docs/local-emulator.md`. That was believed impossible until the SELinux/renderer
cause was found, and it is what makes the e2e half of this norm enforceable.
- **E2E is runnable locally**, API 33-36, via `tools/local-emulator/run-e2e.sh` — see
"Instrumented tests: where they actually run" above. That was believed impossible until the
SELinux/renderer cause was found, and it is what makes the e2e half of this norm enforceable.
- **A test has to bite.** Revert the line it covers, confirm it goes red, restore. A review of
this codebase ran 46 mutations against a 257-test suite and **9 were vacuous** — five of them
passing the whole suite over a completely unguarded code path. Green is not evidence.