JMR-dev/LibreMediaConverter
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

Derive the suite size instead of writing it down in two documents

Browse Source 
R4 / #13 and R20 / #29 are one defect: an absolute test total in an unregenerated
document, written the same day it went stale. This branch was cut at 22c7914, where
app/src/androidTest held 49 @Test methods; main is 57 (ReattachOnLaunchTest added eight
in ec969c4). So the release instruction "expect 49 / 0 / 0 / 2, and if you get 40 you are
on an old checkout" becomes false the moment this branch merges -- on the one check that
has no CI backstop -- and docs/local-emulator.md's headline promises a 49-test local
baseline main no longer produces.

Re-derived rather than renumbered, because a third total would go stale the same way:

- The total is the size of app/src/androidTest on the checkout that ran, and the reported
  total has equalled that checkout's @Test count everywhere it has been checked: 40 at
  edd6385 (the Pixel run), 49 at 22c7914 (the four local levels and API 37), 57 at
  18c53a3 (counted, not run). The new "Reading these totals" section states that, gives
  the one-line grep, and makes the *mismatch* the signal: a total that disagrees with
  your own checkout's count means an old checkout, a stale build or tests that never ran.
  The pre-release Pixel instruction now reads "that many tests, 0 failures, 0 errors, 2
  skipped" -- the invariant, not the total.
- Measurements are kept verbatim and anchored to 22c7914 (the sweep table, the API 35
  control, the tests="49" XML quote, the 51-on-screen console block). Only the claims
  built on top of them were rewritten.

Two claims went with the number, both of which a rebase would have preserved:

- "47 of 49" is not a defensible ratio when two of the 49 are skips. 49 = 45 passed + 2
  failed + 2 skipped, and that is what it now says.
- "against the Pixel's 49 of 49" and "matches the physical Pixel 10 Pro XL baseline of
  49 / 0 / 0 / 2 exactly" describe a run that never happened: the Pixel measured
  40 / 0 / 0 / 2 at edd6385, nine tests earlier, as the same file says a hundred lines
  further down. Both documents projected the local total onto the phone and called it a
  match. What compares between them is 0 failures and the same two skips.

Also re-derived in the CLAUDE.md wording docs/local-emulator.md proposes, since that text
is meant to be pasted out of the branch and would have carried "49 tests / 2 failures /
2 skipped" with it. CLAUDE.md itself is still untouched.

Counts re-checked with git grep at each of the three commits; nothing here needed a
device, and none was used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
JMR-dev
2026-08-22 23:09:30 -05:00
co-authored by Claude Opus 5
parent da6f2807e9
commit 961cfa72a2
2 changed files with 67 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api-37-emulator-crash.md
+50 -17
View File
@@ -3,9 +3,12 @@
**Status:** the bug is real and still open upstream, but the previous diagnosis in this file was
wrong about its most important detail. **The renderer decides whether API 37 boots**, and once it
boots, disabling SystemUI collapses the crash rate far enough to run a suite —
`tools/local-emulator/run-e2e.sh 37` reports **49 tests, 2 failures, 0 errors, 2 skipped**. The
crashes do not stop outright, and the two failures are real; both are quantified below. CI's
matrix should still stop at 36 — see [So should CI take API 37?](#so-should-ci-take-api-37).
`tools/local-emulator/run-e2e.sh 37` gets through the whole instrumented suite and comes back with
**2 failures, 0 errors and the two by-design skips** (measured 49 / 2 / 0 / 2 at `22c7914`, where
the suite was 49 tests — [Reading these totals](#reading-these-totals) before comparing any total
with another). The crashes do not stop outright, and the two failures are real; both are quantified
below. CI's matrix should still stop at 36 — see
[So should CI take API 37?](#so-should-ci-take-api-37).
**Last verified:** 2026-08-22, emulator `37.1.11.0` (build 15917651), Fedora 44,
against system images `android-37.0` rev 6 **and** `android-37.1` rev 8.
@@ -223,17 +226,25 @@ booted (`emulator_alive=yes`). The host emulator is fine; the guest is not.
## Can the suite run on it?
**Almost.** `tools/local-emulator/run-e2e.sh 37` now runs the whole suite locally and reports
**49 tests, 2 failures, 0 errors, 2 skipped**. That is 47 of 49 against the Pixel's 49 of 49, and
it costs two deviations from how every other level is run. Both are worth understanding before
trusting the leg.
**Almost.** `tools/local-emulator/run-e2e.sh 37` now runs the whole suite locally, and all of it
passes except two tests. Measured at `22c7914`: **49 tests, 2 failures, 0 errors, 2 skipped** — 45
passed, the two `Media3EngineTest` failures dissected below, and the two `assumeTrue` skips every
level has. It costs two deviations from how every other level is run, and both are worth
understanding before trusting the leg.
Two things about that total before it is compared with anything. It is the size of the suite on
the checkout that ran, not a property of API 37 — `app/src/androidTest` held 49 `@Test` methods at
`22c7914`, and a newer checkout reports its own count; see
[Reading these totals](#reading-these-totals). And **the Pixel has never run 49**: its green run
was 40 / 0 / 0 / 2 at `edd6385`, the same suite nine tests earlier. What compares across the two
is two failures against none, and the same two skips — not the totals.
The same numbers and the same two test names came back twice, which is real corroboration — but
by two different routes, and only one of them is the harness. The first was driven by hand
(`pm disable-user`, then several minutes of incidental framework restarts, then `e2e-run.sh`
directly); the second went through `disable_region_sampling`'s `stop; start`. **The harness path
itself has one green measurement.** Treat a second consecutive 49/2/0/2 from
`run-e2e.sh 37` as the thing that would make this routine.
itself has one green measurement.** What would make this routine is a second consecutive
`run-e2e.sh 37` whose only failures are the same two.
### Booting is not the same as being usable
@@ -324,8 +335,9 @@ Three measurements say this is the emulator image and not this app, and not the
renderer:
- **Control at API 35 under the identical renderer.** `GPU_MODE=swangle_indirect
tools/local-emulator/run-e2e.sh 35` → **49 / 0 / 0 / 2**, green. `c2.goldfish.h264.decoder` is
perfectly happy under ANGLE one API level down, so the renderer is not what breaks it.
tools/local-emulator/run-e2e.sh 35` → **49 / 0 / 0 / 2** at `22c7914`, green.
`c2.goldfish.h264.decoder` is perfectly happy under ANGLE one API level down, so the renderer is
not what breaks it.
- **Real API 37 hardware passes**, see below. There is no `c2.goldfish.*` codec on a Pixel.
- The failing call is `dequeueOutputBuffer` on the *goldfish* decoder — the emulator's own codec,
which like `RegionSamplingThread` gets its frames out of a host-side colour buffer. Same
@@ -346,7 +358,7 @@ suite depends on those decoders existing.
2. The working configuration needs SystemUI disabled and a framework restart mid-job. That is a
lot of bespoke device surgery to put behind a merge gate, and it silently weakens what the leg
proves.
3. Even at its best it is 47 of 49, so the leg would be permanently red or permanently
3. Even at its best two tests fail, so the leg would be permanently red or permanently
allow-listed. Neither is a gate worth having.
What has changed is the *local* story: API 37 is no longer a level nobody can look at. A
@@ -367,11 +379,32 @@ API: 37 (Android 17, codename REL -- a release build, not a preview)
40 tests, 0 failures, 0 errors, 2 skipped BUILD SUCCESSFUL
```
**That "40" is not a baseline to compare against today, and it is not a contradiction of the 49
in [`docs/local-emulator.md`](local-emulator.md).** It was accurate for the tree it ran on: at
`edd6385`, the commit that recorded it, `app/src/androidTest` contained exactly 40 `@Test`
methods. Nine have been added since and the suite is now 49. If you re-run on the Pixel, expect
49 / 0 / 0 / 2, and if you get 40, you are on an old checkout.
**That "40" is a measurement of the tree it ran on, not a baseline for today**, and it is not a
contradiction of the totals in [`docs/local-emulator.md`](local-emulator.md) either.
### Reading these totals
Every total in this file and in [`docs/local-emulator.md`](local-emulator.md) is the size of
`app/src/androidTest` on the checkout that produced it, and nothing else. The reported total has
equalled that checkout's `@Test` count everywhere it has been checked:
| checkout | `@Test` methods | total the run reported |
|---|---|---|
| `edd6385` | 40 | 40 — the Pixel run above |
| `22c7914` | 49 | 49 — the four local levels, and API 37 |
| `18c53a3` | 57 | not run |
So the number to expect is not written down here. It is derived from the checkout in front of
you, which is the only thing that cannot go stale:
```bash
grep -rho '@Test' app/src/androidTest | wc -l
```
**Before each release, run the suite on the Pixel 10 Pro XL and expect that many tests, 0
failures, 0 errors, 2 skipped.** The failure, error and skip counts are the invariant; the total
is not. A total that disagrees with your own checkout's count is the signal — an old checkout, a
stale build, or tests that never ran — and it is worth stopping on either way.
The two skips are `RealMediaBenchmark.hardwareVersusSoftwareOnRealVideo` and
`av1InputRoutesAccordingToDeviceDecodeSupport`, which `assumeTrue` their sample files are present
docs/local-emulator.md
+17 -7
View File
@@ -1,8 +1,10 @@
# Emulators do run on this host: the segfault is SwiftShader's JIT against SELinux
**Status:** solved. Local instrumented runs work with `-gpu host`, and the suite is green
on API 33–36 — 49 tests, 0 failures, 0 errors, 2 skipped on every level. See
[The sweep, run](#the-sweep-run).
**Status:** solved. Local instrumented runs work with `-gpu host`, and the whole suite is green
on API 33–36 — 0 failures, 0 errors and the two by-design skips on every level, measured as
49 / 0 / 0 / 2 at `22c7914`, where the suite was 49 tests. See [The sweep, run](#the-sweep-run),
and [Reading these totals](api-37-emulator-crash.md#reading-these-totals) before comparing any
total with another checkout's.
**Last verified:** 2026-08-22, emulator `37.1.11.0` (build 15917651), Fedora 44,
kernel `7.1.8-200.fc44`, `selinux-policy-44.6-1.fc44`
@@ -243,8 +245,9 @@ after an AGP upgrade.
## The sweep, run
`tools/local-emulator/run-e2e.sh`, one invocation per level so each got a freshly created
AVD, `-gpu host` throughout, 2026-08-22 19:42–19:56. Every level matches the physical
Pixel 10 Pro XL (API 37) baseline of 49 / 0 / 0 / 2 exactly:
AVD, `-gpu host` throughout, 2026-08-22 19:42–19:56, on `22c7914`. All four levels agree exactly,
and 49 is that checkout's whole suite — every `@Test` in `app/src/androidTest`, two of which skip
by design everywhere:
| API | Android | AVD | Boot | `connectedDebugAndroidTest` | Tests | Failures | Errors | Skipped |
|---|---|---|---|---|---|---|---|---|
@@ -253,6 +256,12 @@ Pixel 10 Pro XL (API 37) baseline of 49 / 0 / 0 / 2 exactly:
| 35 | 15 | `lmc_e2e_api35` | 40 s | 3 m 46 s | 49 | 0 | 0 | 2 |
| 36 | 16 | `lmc_e2e_api36` | 90 s | 2 m 18 s | 49 | 0 | 0 | 2 |
The physical Pixel has never reported 49, and an earlier version of this paragraph said the
sweep matched it exactly. Its green API 37 run was 40 / 0 / 0 / 2, at `edd6385` — the same suite
nine tests earlier. What matches is 0 failures, 0 errors and the same two skips; totals only ever
match between runs of one checkout, which
[`api-37-emulator-crash.md`](api-37-emulator-crash.md#reading-these-totals) sets out.
Thirteen and a half minutes for the four levels, AVD creation and cold boots included;
fifteen with the pre-warm build in front of them. Nothing needed a retry, and no level
produced a `diagnostics-api*.txt` — `e2e-run.sh` writes that only on the failure path, so
@@ -449,8 +458,9 @@ Proposed replacement for the section, offered for review rather than applied her
> - **API 37 needs the opposite renderer, and SystemUI turned off.** Both `android-37.0` and
> `android-37.1` abort surfaceflinger inside their own gralloc mapper, and init SIGKILLs
> zygote each time. Under `-gpu host` they never boot; under `-gpu swangle_indirect` they
> boot, and disabling SystemUI removes the trigger. `run-e2e.sh` does all of that per level;
> the local result is 49 tests / 2 failures / 2 skipped. CI's matrix still stops at 36.
> boot, and disabling SystemUI removes the trigger. `run-e2e.sh` does all of that per level,
> and the local API 37 result is two failures and the two usual skips, not a clean run. CI's
> matrix still stops at 36.
> `docs/api-37-emulator-crash.md` has the matrix and the reasoning. **API 37 needs a manual
> check on the Pixel 10 Pro XL before each release.**