From 961cfa72a27de73b2edadf27d1cb5acc0316f09f Mon Sep 17 00:00:00 2001 From: Jason Ross Date: Sat, 22 Aug 2026 23:09:30 -0500 Subject: [PATCH] Derive the suite size instead of writing it down in two documents R4 / #13 and R20 / #29 are one defect: an absolute test total in an unregenerated document, written the same day it went stale. This branch was cut at 22c7914, where app/src/androidTest held 49 @Test methods; main is 57 (ReattachOnLaunchTest added eight in ec969c4). So the release instruction "expect 49 / 0 / 0 / 2, and if you get 40 you are on an old checkout" becomes false the moment this branch merges -- on the one check that has no CI backstop -- and docs/local-emulator.md's headline promises a 49-test local baseline main no longer produces. Re-derived rather than renumbered, because a third total would go stale the same way: - The total is the size of app/src/androidTest on the checkout that ran, and the reported total has equalled that checkout's @Test count everywhere it has been checked: 40 at edd6385 (the Pixel run), 49 at 22c7914 (the four local levels and API 37), 57 at 18c53a3 (counted, not run). The new "Reading these totals" section states that, gives the one-line grep, and makes the *mismatch* the signal: a total that disagrees with your own checkout's count means an old checkout, a stale build or tests that never ran. The pre-release Pixel instruction now reads "that many tests, 0 failures, 0 errors, 2 skipped" -- the invariant, not the total. - Measurements are kept verbatim and anchored to 22c7914 (the sweep table, the API 35 control, the tests="49" XML quote, the 51-on-screen console block). Only the claims built on top of them were rewritten. Two claims went with the number, both of which a rebase would have preserved: - "47 of 49" is not a defensible ratio when two of the 49 are skips. 49 = 45 passed + 2 failed + 2 skipped, and that is what it now says. - "against the Pixel's 49 of 49" and "matches the physical Pixel 10 Pro XL baseline of 49 / 0 / 0 / 2 exactly" describe a run that never happened: the Pixel measured 40 / 0 / 0 / 2 at edd6385, nine tests earlier, as the same file says a hundred lines further down. Both documents projected the local total onto the phone and called it a match. What compares between them is 0 failures and the same two skips. Also re-derived in the CLAUDE.md wording docs/local-emulator.md proposes, since that text is meant to be pasted out of the branch and would have carried "49 tests / 2 failures / 2 skipped" with it. CLAUDE.md itself is still untouched. Counts re-checked with git grep at each of the three commits; nothing here needed a device, and none was used. Co-Authored-By: Claude Opus 5 (1M context) --- docs/api-37-emulator-crash.md | 67 ++++++++++++++++++++++++++--------- docs/local-emulator.md | 24 +++++++++---- 2 files changed, 67 insertions(+), 24 deletions(-) diff --git a/docs/api-37-emulator-crash.md b/docs/api-37-emulator-crash.md index 321a959..cacc409 100644 --- a/docs/api-37-emulator-crash.md +++ b/docs/api-37-emulator-crash.md @@ -3,9 +3,12 @@ **Status:** the bug is real and still open upstream, but the previous diagnosis in this file was wrong about its most important detail. **The renderer decides whether API 37 boots**, and once it boots, disabling SystemUI collapses the crash rate far enough to run a suite — -`tools/local-emulator/run-e2e.sh 37` reports **49 tests, 2 failures, 0 errors, 2 skipped**. The -crashes do not stop outright, and the two failures are real; both are quantified below. CI's -matrix should still stop at 36 — see [So should CI take API 37?](#so-should-ci-take-api-37). +`tools/local-emulator/run-e2e.sh 37` gets through the whole instrumented suite and comes back with +**2 failures, 0 errors and the two by-design skips** (measured 49 / 2 / 0 / 2 at `22c7914`, where +the suite was 49 tests — [Reading these totals](#reading-these-totals) before comparing any total +with another). The crashes do not stop outright, and the two failures are real; both are quantified +below. CI's matrix should still stop at 36 — see +[So should CI take API 37?](#so-should-ci-take-api-37). **Last verified:** 2026-08-22, emulator `37.1.11.0` (build 15917651), Fedora 44, against system images `android-37.0` rev 6 **and** `android-37.1` rev 8. @@ -223,17 +226,25 @@ booted (`emulator_alive=yes`). The host emulator is fine; the guest is not. ## Can the suite run on it? -**Almost.** `tools/local-emulator/run-e2e.sh 37` now runs the whole suite locally and reports -**49 tests, 2 failures, 0 errors, 2 skipped**. That is 47 of 49 against the Pixel's 49 of 49, and -it costs two deviations from how every other level is run. Both are worth understanding before -trusting the leg. +**Almost.** `tools/local-emulator/run-e2e.sh 37` now runs the whole suite locally, and all of it +passes except two tests. Measured at `22c7914`: **49 tests, 2 failures, 0 errors, 2 skipped** — 45 +passed, the two `Media3EngineTest` failures dissected below, and the two `assumeTrue` skips every +level has. It costs two deviations from how every other level is run, and both are worth +understanding before trusting the leg. + +Two things about that total before it is compared with anything. It is the size of the suite on +the checkout that ran, not a property of API 37 — `app/src/androidTest` held 49 `@Test` methods at +`22c7914`, and a newer checkout reports its own count; see +[Reading these totals](#reading-these-totals). And **the Pixel has never run 49**: its green run +was 40 / 0 / 0 / 2 at `edd6385`, the same suite nine tests earlier. What compares across the two +is two failures against none, and the same two skips — not the totals. The same numbers and the same two test names came back twice, which is real corroboration — but by two different routes, and only one of them is the harness. The first was driven by hand (`pm disable-user`, then several minutes of incidental framework restarts, then `e2e-run.sh` directly); the second went through `disable_region_sampling`'s `stop; start`. **The harness path -itself has one green measurement.** Treat a second consecutive 49/2/0/2 from -`run-e2e.sh 37` as the thing that would make this routine. +itself has one green measurement.** What would make this routine is a second consecutive +`run-e2e.sh 37` whose only failures are the same two. ### Booting is not the same as being usable @@ -324,8 +335,9 @@ Three measurements say this is the emulator image and not this app, and not the renderer: - **Control at API 35 under the identical renderer.** `GPU_MODE=swangle_indirect - tools/local-emulator/run-e2e.sh 35` → **49 / 0 / 0 / 2**, green. `c2.goldfish.h264.decoder` is - perfectly happy under ANGLE one API level down, so the renderer is not what breaks it. + tools/local-emulator/run-e2e.sh 35` → **49 / 0 / 0 / 2** at `22c7914`, green. + `c2.goldfish.h264.decoder` is perfectly happy under ANGLE one API level down, so the renderer is + not what breaks it. - **Real API 37 hardware passes**, see below. There is no `c2.goldfish.*` codec on a Pixel. - The failing call is `dequeueOutputBuffer` on the *goldfish* decoder — the emulator's own codec, which like `RegionSamplingThread` gets its frames out of a host-side colour buffer. Same @@ -346,7 +358,7 @@ suite depends on those decoders existing. 2. The working configuration needs SystemUI disabled and a framework restart mid-job. That is a lot of bespoke device surgery to put behind a merge gate, and it silently weakens what the leg proves. -3. Even at its best it is 47 of 49, so the leg would be permanently red or permanently +3. Even at its best two tests fail, so the leg would be permanently red or permanently allow-listed. Neither is a gate worth having. What has changed is the *local* story: API 37 is no longer a level nobody can look at. A @@ -367,11 +379,32 @@ API: 37 (Android 17, codename REL -- a release build, not a preview) 40 tests, 0 failures, 0 errors, 2 skipped BUILD SUCCESSFUL ``` -**That "40" is not a baseline to compare against today, and it is not a contradiction of the 49 -in [`docs/local-emulator.md`](local-emulator.md).** It was accurate for the tree it ran on: at -`edd6385`, the commit that recorded it, `app/src/androidTest` contained exactly 40 `@Test` -methods. Nine have been added since and the suite is now 49. If you re-run on the Pixel, expect -49 / 0 / 0 / 2, and if you get 40, you are on an old checkout. +**That "40" is a measurement of the tree it ran on, not a baseline for today**, and it is not a +contradiction of the totals in [`docs/local-emulator.md`](local-emulator.md) either. + +### Reading these totals + +Every total in this file and in [`docs/local-emulator.md`](local-emulator.md) is the size of +`app/src/androidTest` on the checkout that produced it, and nothing else. The reported total has +equalled that checkout's `@Test` count everywhere it has been checked: + +| checkout | `@Test` methods | total the run reported | +|---|---|---| +| `edd6385` | 40 | 40 — the Pixel run above | +| `22c7914` | 49 | 49 — the four local levels, and API 37 | +| `18c53a3` | 57 | not run | + +So the number to expect is not written down here. It is derived from the checkout in front of +you, which is the only thing that cannot go stale: + +```bash +grep -rho '@Test' app/src/androidTest | wc -l +``` + +**Before each release, run the suite on the Pixel 10 Pro XL and expect that many tests, 0 +failures, 0 errors, 2 skipped.** The failure, error and skip counts are the invariant; the total +is not. A total that disagrees with your own checkout's count is the signal — an old checkout, a +stale build, or tests that never ran — and it is worth stopping on either way. The two skips are `RealMediaBenchmark.hardwareVersusSoftwareOnRealVideo` and `av1InputRoutesAccordingToDeviceDecodeSupport`, which `assumeTrue` their sample files are present diff --git a/docs/local-emulator.md b/docs/local-emulator.md index 1db38aa..e2eac49 100644 --- a/docs/local-emulator.md +++ b/docs/local-emulator.md @@ -1,8 +1,10 @@ # Emulators do run on this host: the segfault is SwiftShader's JIT against SELinux -**Status:** solved. Local instrumented runs work with `-gpu host`, and the suite is green -on API 33–36 — 49 tests, 0 failures, 0 errors, 2 skipped on every level. See -[The sweep, run](#the-sweep-run). +**Status:** solved. Local instrumented runs work with `-gpu host`, and the whole suite is green +on API 33–36 — 0 failures, 0 errors and the two by-design skips on every level, measured as +49 / 0 / 0 / 2 at `22c7914`, where the suite was 49 tests. See [The sweep, run](#the-sweep-run), +and [Reading these totals](api-37-emulator-crash.md#reading-these-totals) before comparing any +total with another checkout's. **Last verified:** 2026-08-22, emulator `37.1.11.0` (build 15917651), Fedora 44, kernel `7.1.8-200.fc44`, `selinux-policy-44.6-1.fc44` @@ -243,8 +245,9 @@ after an AGP upgrade. ## The sweep, run `tools/local-emulator/run-e2e.sh`, one invocation per level so each got a freshly created -AVD, `-gpu host` throughout, 2026-08-22 19:42–19:56. Every level matches the physical -Pixel 10 Pro XL (API 37) baseline of 49 / 0 / 0 / 2 exactly: +AVD, `-gpu host` throughout, 2026-08-22 19:42–19:56, on `22c7914`. All four levels agree exactly, +and 49 is that checkout's whole suite — every `@Test` in `app/src/androidTest`, two of which skip +by design everywhere: | API | Android | AVD | Boot | `connectedDebugAndroidTest` | Tests | Failures | Errors | Skipped | |---|---|---|---|---|---|---|---|---| @@ -253,6 +256,12 @@ Pixel 10 Pro XL (API 37) baseline of 49 / 0 / 0 / 2 exactly: | 35 | 15 | `lmc_e2e_api35` | 40 s | 3 m 46 s | 49 | 0 | 0 | 2 | | 36 | 16 | `lmc_e2e_api36` | 90 s | 2 m 18 s | 49 | 0 | 0 | 2 | +The physical Pixel has never reported 49, and an earlier version of this paragraph said the +sweep matched it exactly. Its green API 37 run was 40 / 0 / 0 / 2, at `edd6385` — the same suite +nine tests earlier. What matches is 0 failures, 0 errors and the same two skips; totals only ever +match between runs of one checkout, which +[`api-37-emulator-crash.md`](api-37-emulator-crash.md#reading-these-totals) sets out. + Thirteen and a half minutes for the four levels, AVD creation and cold boots included; fifteen with the pre-warm build in front of them. Nothing needed a retry, and no level produced a `diagnostics-api*.txt` — `e2e-run.sh` writes that only on the failure path, so @@ -449,8 +458,9 @@ Proposed replacement for the section, offered for review rather than applied her > - **API 37 needs the opposite renderer, and SystemUI turned off.** Both `android-37.0` and > `android-37.1` abort surfaceflinger inside their own gralloc mapper, and init SIGKILLs > zygote each time. Under `-gpu host` they never boot; under `-gpu swangle_indirect` they -> boot, and disabling SystemUI removes the trigger. `run-e2e.sh` does all of that per level; -> the local result is 49 tests / 2 failures / 2 skipped. CI's matrix still stops at 36. +> boot, and disabling SystemUI removes the trigger. `run-e2e.sh` does all of that per level, +> and the local API 37 result is two failures and the two usual skips, not a clean run. CI's +> matrix still stops at 36. > `docs/api-37-emulator-crash.md` has the matrix and the reasoning. **API 37 needs a manual > check on the Pixel 10 Pro XL before each release.**