Measure the fourth claim rather than asserting it, and fix three slips

Review of the previous commit found three things of exactly the kind it corrects.

"Both of the advisory runs that exist" asserted exhaustiveness that had not been
checked -- two jobs were read, and the #248 branch had three status_check runs.
All four advisory runs at baseline 6 are now read: 34041156680, 34041593697,
34042397320 and 34043502322 each report expected: 6, received: 4, failed: 4,
and the only SAF test reporting in any of them is the rotation one. The claim
was right; the wording claimed more than the evidence.

CONVERSION_TIMEOUT_MS's KDoc said the bound is "two orders of magnitude" clear
of the real cost. 120 s against a measured 11.8 s is one.

status_check.yml dated the save test's marker to 2026-09-05. fa10d94 is dated
2026-09-06.

Also reworded that comment's summary: "two abort the framework and two never
report" double-counts the picker test, so a reader summing gets seven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-06 11:20:05 -05:00
co-authored by Claude Opus 5
parent 54167c052f
commit e81403c5f3
5 changed files with 19 additions and 15 deletions
+4 -3
View File
@@ -277,7 +277,7 @@ jobs:
# leg; four gating logcats read on 2026-09-05 show it aborting system_server
# from the task-snapshot path on every single run, pass or fail, which is what
# had been failing unrelated PRs (#108). All THREE of that class's tests now
# carry the marker -- the save through the picker (#226) joined on 2026-09-05
# carry the marker -- the save through the picker (#226) joined on 2026-09-06
# by inheritance rather than measurement, since it opens the same picker.
# docs/api-37-emulator-crash.md has the timings and the correction, and
# FailsOnEmulatorApi37.kt has why the third one cannot be measured here.
@@ -293,8 +293,9 @@ jobs:
# notAnnotation removes the six tests that cannot be RUN on this image; they
# run in the advisory job below, off the same marker so they cannot end up
# in both or neither. "Cannot be run" rather than "do not pass" is deliberate:
# two of the six abort the framework rather than failing, and two never report
# at all. docs/api-37-emulator-crash.md has the measurements.
# four fail outright, one of those aborts the framework on its way down, and on
# the advisory leg the two picker tests behind it never report at all.
# docs/api-37-emulator-crash.md has the measurements.
- label: "37"
api-level: "37.0"
disable-system-ui: "1"
+5 -4
View File
@@ -83,10 +83,11 @@ days. Read it as the current answer, and see the git history if you need the old
`system_server` from the task-snapshot path whether it passes or not. The sixth, that class's
save through the picker (#226), carries the marker because it opens the same picker and a second
DocumentsUI dialog on top of it — **not** because it has ever been observed here. It cannot be:
the rotation test runs first and takes the framework down, so both of the advisory runs that
exist since it landed report `expected: 6, received: 4` and the four are the three Media3 tests
plus the rotation — runs 34042397320 and 34043502322. **Neither picker test has ever reported on
the advisory leg**, which is a correction to what the marker's own KDoc says. All six carry
the rotation test runs first and takes the framework down, so **all four** advisory runs at this
baseline report `expected: 6, received: 4, failed: 4`, and the four are the three Media3 tests
plus the rotation — runs 34041156680, 34041593697, 34042397320 and 34043502322. **Neither picker
test has ever reported on the advisory leg**, which is a correction to what the marker's own KDoc
says. All six carry
`@FailsOnEmulatorApi37` and run in a separate `continue-on-error` job; the gating leg runs the
other 64 — **the same 64 as before**, which is exactly how this paragraph went stale unnoticed.
@@ -79,9 +79,10 @@ annotation class FailsOnEmulatorApi37
* framework having died, which is this job's normal.
*
* **That is no longer what happens, and the difference is that neither picker test reports at
* all.** With six carriers the rotation test truncates the run before them: both advisory runs
* since #226 landed — 34042397320 and 34043502322 — report `expected: 6, received: 4`, and the
* four are the three Media3 tests plus the rotation. So the advisory leg currently answers for
* all.** With six carriers the rotation test truncates the run before them: **all four** advisory
* runs at this baseline — 34041156680, 34041593697, 34042397320 and 34043502322 — report
* `expected: 6, received: 4, failed: 4`, and the four are the three Media3 tests plus the
* rotation. So the advisory leg currently answers for
* four of its six, and the comparison below is unaffected only because `failed` is not compared
* on a truncated run. Read it as **unmeasured**, not as passing or failing.
*
@@ -534,8 +534,8 @@ class SafPickerRoundTripTest {
* [pickingAFileThroughTheSystemPickerFillsInTheFileCard], which was marked for aborting
* `system_server` from the task-snapshot path (#108), and then a second DocumentsUI dialog on
* top of it. It has never been observed at API 37 either way: the rotation test truncates the
* advisory run first, so both advisory runs since this landed report `expected: 6, received: 4`
* without reaching either picker test. Marking it was the conservative choice and it is
* advisory run first, so all four advisory runs at this baseline report
* `expected: 6, received: 4` without reaching either picker test. Marking it was the conservative choice and it is
* recorded as unmeasured in `FailsOnEmulatorApi37.kt` rather than dressed up as a measurement.
*/
@Test
@@ -1080,7 +1080,7 @@ class SafPickerRoundTripTest {
const val PERMISSION_DIALOG_MS = 5_000L
/**
* Only bounds a hang, and it is two orders of magnitude clear of the real cost: the whole
* Only bounds a hang, and it is an order of magnitude clear of the real cost: the whole
* test — pick, convert, save — takes **11.8 s** on the API 34 CI leg (run 34043502322).
* Deliberately generous because the engine is not fixed: the default `MP4_H265` at `FAST`
* lands on FFmpeg on an emulator and on Media3 on real hardware, which is faster rather
+3 -2
View File
@@ -423,8 +423,9 @@ green on `main`.
found it** — a passing test with a wrong explanation.
- **Neither picker test has ever reported on the advisory leg.** The marker's KDoc said the picker
test *fails* there behind the rotation test; with six carriers the rotation test truncates the run
first, and both advisory runs since #226 (`34042397320`, `34043502322`) report
`expected: 6, received: 4` — the three Media3 tests plus the rotation. The save test is therefore
first, and all four advisory runs at this baseline (`34041156680`, `34041593697`,
`34042397320`, `34043502322`) report `expected: 6, received: 4, failed: 4` — the three Media3
tests plus the rotation. The save test is therefore
marked by **inheritance, not measurement**, which is now what both KDocs say.
- **One substantive gap, filed as #250.** `FixtureDocumentsProvider.deletedDocumentIds()` has no
callers. #226 proved D4's *premise* — SAF hands back a document of exactly zero bytes — but drove