The advisory job's baseline check has said the tree carries 4 tests marked @FailsOnEmulatorApi37 but the baseline says 3 on every PR since #113. There are three,
and FAILS_ON_EMULATOR_API37_BASELINE says three — the fourth is a KDoc in Media3EngineTest saying the opposite. The old matcher looked for the string anywhere
on any line, so the comment explaining why a test does not carry the marker counted as
one that does.
The matcher
Re-measured here rather than taken on trust; the ticket's table reproduces exactly.
The middle row is the trap. "Only the annotation on a line of its own" is the obvious
repair and it silently stops counting @FailsOnEmulatorApi37 @Test, which is legal
Kotlin. Undercounting is the dangerous direction: it hides a genuine new marker, which is
the one thing this check exists to catch. Requiring whitespace-or-end-of-line after the
name keeps that case while still rejecting the KDoc (^\s*@ does not match * Deliberately not `@...) and the commented-out form (^\s*@ does not match // @...).
grep -v import goes with it — ^[[:space:]]*@ cannot match an import line, so leaving
it would imply a filter is still doing work.
The real tree now counts 3, which is what the baseline already says, so the baseline is
not touched. Setting it to 4 would have silenced the notice by breaking the check.
The fixture, and why it cannot move the real count
.github/scripts/testdata/marker-shapes/MarkerShapes.kt carries every shape at once —
three that count (own line, sharing a line with @Test, indented inside a nested class)
and three that must not (KDoc mention, commented out, the import).
It is under .github/, so Gradle compiles only app/src/**, ktlint and detekt are
applied to :app only, and the report's own count reads app/src/androidTest. Verified
rather than assumed — the branch script run against the real repo root with the fixture
committed:
baseline: matches (3 expected, 3 failed)
.github/scripts/e2e-report-shape-test.sh runs the real report against it, inside a
throwaway repo root assembled in mktemp -d. The script finds its own root from BASH_SOURCE, so no knob had to be added to production code — which matters twice over:
reverting the matcher reddens the test rather than a testing-only path beside it, and no
environment variable exists that could point the live count somewhere else, which is the
failure mode #83 built the baseline check to prevent. It also means XML_DIR resolves
inside the throwaway root, so a stale app/build/outputs cannot leak in.
Twelve checks, green:
ok fixture: the import
ok fixture: annotation own line
ok fixture: annotation with @Test on one line
ok fixture: annotation nested and indented
ok fixture: KDoc mention (this is #120)
ok fixture: commented-out annotation
ok 3 real markers, baseline 3: reports a match
ok 3 real markers, baseline 3: says nothing about the tree
ok 3 real markers, baseline 3: summary agrees
ok a 4th real marker: the deviation fires, and counts 4
ok a 4th real marker: the summary carries it too
ok same-line annotation removed: counts 2, so it was worth 1
Mutations — each confirmed to fail
1. Revert the matcher to the old one. The false deviation comes straight back on the
real tree:
baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
::notice::E2E api37: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
and the test goes red, 6 of 12 checks failing — including, against the fixture, the tree carries 5 tests marked ....
2. Add a fourth REAL annotation to the fixture tree. The count moves and the deviation
fires correctly — the half that proves precision was not bought by destroying the check:
baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
3a. Add a same-line @FailsOnEmulatorApi37 @Test fun .... It is counted — 3 becomes 4:
baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
3b. Delete the same-line entry from the fixture — the sharper direction, since it
proves the entry is load-bearing rather than decorative. The count drops to 2 and the
shape assertion fires by name:
FAIL fixture: annotation with @Test on one line
baseline DEVIATION: the tree carries 2 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
Byte-identical everywhere else
Per #121's method, and with both script versions run from the same worktree so the
only variable is the script text (git show origin/main:... into .github/scripts/,
deleted afterwards). Seven legs, both streams:
The two that differ are the two advisory legs, and they differ by exactly the false line
going away:
- baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
-::notice::E2E api37: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
+ baseline: matches (3 expected, 3 failed)
On advisory-wedged the other deviation is left untouched, which is the point:
baseline DEVIATION: the runner started 57 tests, the baseline is 3
- baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — ...
The harness was proven able to fail, on both streams separately — #121's first attempt
reddened only stdout. A stray echo added inside the $GITHUB_STEP_SUMMARY block
reddens all seven summary diffs and leaves the five non-advisory stdout diffs green; a
stray echo on stdout reddens all seven stdout diffs and leaves five summary diffs green.
Nothing about any job changes
The advisory leg is still continue-on-error and still red by design; deviations are
still ::notice::, never ::error::. No workflow file is touched.
Named exemptions
The test is not wired into CI, deliberately. Adding a step to Static analysis would
add a new way for a gating job to go red, and #120 was explicit that nothing here may
change any job's status or the pass/fail rules. It runs locally
(.github/scripts/e2e-report-shape-test.sh) and shellcheck still covers the file, since
that step reads git ls-files '*.sh'. Wiring it in is a one-step follow-up if wanted.
Still a regex over source text, not a parser. An annotation inside a multi-line
string, or inside a /* */ block that opened mid-line, would still be counted. Neither
exists in this tree; if one ever does, this check wants a different tool rather than a
longer regex. Written into the script's comment, not just here.
The same-line case remains hypothetical in this repo. grep -rnE '^\s*@[A-Za-z]+\s+@[A-Za-z]' finds no instance in androidTest or test. It
is legal Kotlin and cheap to keep working, not something currently relied on.
--include='*.kt' is not itself tested. A marker in a .java or .txt file under androidTest is uncovered; the clause is unchanged from before this PR.
CLAUDE.md is unchanged. The reasoning lives in the two script headers, and the
existing CLAUDE.md paragraph about the advisory job stays true as written.
Gate
assembleDebug, testDebugUnitTest, compileDebugAndroidTestKotlin, ktlintCheck, detekt, lintDebug --continue → BUILD SUCCESSFUL, 71 tasks. Pinned shellcheck 0.11.0
over all six tracked scripts → clean (five findings in the new test script were fixed, not
disabled: SC1007, and SC2016 answered by writing the expected strings with escaped
backticks in double quotes, the same way the script under test writes them). Pinned
actionlint → clean.
Run 32930571300, job 98061906295 — the notice the ticket was filed about is gone, from a
real API 37 leg that aborted the way it always does:
----- RUN SHAPE (api37-media3-transcode) -----
expected: 3
received: 3
failed: 3
completed cleanly: no
baseline: matches (3 expected, 3 failed)
No ::notice:: at all. The job is still red — three tests failed, as designed.
Closes #120.
The advisory job's baseline check has said `the tree carries 4 tests marked
@FailsOnEmulatorApi37 but the baseline says 3` on every PR since #113. There are three,
and `FAILS_ON_EMULATOR_API37_BASELINE` says three — the fourth is a KDoc in
`Media3EngineTest` saying the *opposite*. The old matcher looked for the string anywhere
on any line, so the comment explaining why a test does *not* carry the marker counted as
one that does.
## The matcher
Re-measured here rather than taken on trust; the ticket's table reproduces exactly.
| matcher | fixture (want 3) | real tree (want 3) |
|---|---|---|
| old: `grep -v import \| grep -c FailsOn` | 5 | 4 |
| `^[[:space:]]*@FailsOnEmulatorApi37[[:space:]]*$` | 2 | 3 |
| **shipped: `^[[:space:]]*@FailsOnEmulatorApi37([[:space:]]\|$)`** | **3** | **3** |
The middle row is the trap. "Only the annotation on a line of its own" is the obvious
repair and it silently stops counting `@FailsOnEmulatorApi37 @Test`, which is legal
Kotlin. Undercounting is the dangerous direction: it hides a genuine new marker, which is
the one thing this check exists to catch. Requiring whitespace-or-end-of-line *after* the
name keeps that case while still rejecting the KDoc (`^\s*@` does not match
`` * Deliberately not `@... ``) and the commented-out form (`^\s*@` does not match `// @...`).
`grep -v import` goes with it — `^[[:space:]]*@` cannot match an import line, so leaving
it would imply a filter is still doing work.
The real tree now counts 3, which is what the baseline already says, so **the baseline is
not touched**. Setting it to 4 would have silenced the notice by breaking the check.
## The fixture, and why it cannot move the real count
`.github/scripts/testdata/marker-shapes/MarkerShapes.kt` carries every shape at once —
three that count (own line, sharing a line with `@Test`, indented inside a nested class)
and three that must not (KDoc mention, commented out, the import).
It is under `.github/`, so Gradle compiles only `app/src/**`, ktlint and detekt are
applied to `:app` only, and the report's own count reads `app/src/androidTest`. Verified
rather than assumed — the branch script run against the real repo root with the fixture
committed:
```
baseline: matches (3 expected, 3 failed)
```
`.github/scripts/e2e-report-shape-test.sh` runs the **real** report against it, inside a
throwaway repo root assembled in `mktemp -d`. The script finds its own root from
`BASH_SOURCE`, so no knob had to be added to production code — which matters twice over:
reverting the matcher reddens the test rather than a testing-only path beside it, and no
environment variable exists that could point the *live* count somewhere else, which is the
failure mode #83 built the baseline check to prevent. It also means `XML_DIR` resolves
inside the throwaway root, so a stale `app/build/outputs` cannot leak in.
Twelve checks, green:
```
ok fixture: the import
ok fixture: annotation own line
ok fixture: annotation with @Test on one line
ok fixture: annotation nested and indented
ok fixture: KDoc mention (this is #120)
ok fixture: commented-out annotation
ok 3 real markers, baseline 3: reports a match
ok 3 real markers, baseline 3: says nothing about the tree
ok 3 real markers, baseline 3: summary agrees
ok a 4th real marker: the deviation fires, and counts 4
ok a 4th real marker: the summary carries it too
ok same-line annotation removed: counts 2, so it was worth 1
```
## Mutations — each confirmed to fail
**1. Revert the matcher to the old one.** The false deviation comes straight back on the
real tree:
```
baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
::notice::E2E api37: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
```
and the test goes red, 6 of 12 checks failing — including, against the fixture,
`the tree carries 5 tests marked ...`.
**2. Add a fourth REAL annotation to the fixture tree.** The count moves and the deviation
fires *correctly* — the half that proves precision was not bought by destroying the check:
```
baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
```
**3a. Add a same-line `@FailsOnEmulatorApi37 @Test fun ...`.** It is counted — 3 becomes 4:
```
baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
```
**3b. Delete the same-line entry from the fixture** — the sharper direction, since it
proves the entry is load-bearing rather than decorative. The count drops to 2 and the
shape assertion fires by name:
```
FAIL fixture: annotation with @Test on one line
baseline DEVIATION: the tree carries 2 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
```
## Byte-identical everywhere else
Per #121's method, and with both script versions run from the **same** worktree so the
only variable is the script text (`git show origin/main:...` into `.github/scripts/`,
deleted afterwards). Seven legs, both streams:
```
IDENTICAL pass-gating stdout IDENTICAL pass-gating summary
IDENTICAL fail-gating stdout IDENTICAL fail-gating summary
IDENTICAL wedged-gating stdout IDENTICAL wedged-gating summary
DIFFERS advisory-truncated stdout DIFFERS advisory-truncated summary
IDENTICAL advisory-unreadable stdout IDENTICAL advisory-unreadable summary
IDENTICAL norun-gating stdout IDENTICAL norun-gating summary
DIFFERS advisory-wedged stdout DIFFERS advisory-wedged summary
```
The two that differ are the two advisory legs, and they differ by exactly the false line
going away:
```
- baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
-::notice::E2E api37: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — update FAILS_ON_EMULATOR_API37_BASELINE
+ baseline: matches (3 expected, 3 failed)
```
On `advisory-wedged` the *other* deviation is left untouched, which is the point:
```
baseline DEVIATION: the runner started 57 tests, the baseline is 3
- baseline DEVIATION: the tree carries 4 tests marked `@FailsOnEmulatorApi37` but the baseline says 3 — ...
```
**The harness was proven able to fail, on both streams separately** — #121's first attempt
reddened only stdout. A stray `echo` added *inside* the `$GITHUB_STEP_SUMMARY` block
reddens all seven summary diffs and leaves the five non-advisory stdout diffs green; a
stray `echo` on stdout reddens all seven stdout diffs and leaves five summary diffs green.
## Nothing about any job changes
The advisory leg is still `continue-on-error` and still red by design; deviations are
still `::notice::`, never `::error::`. No workflow file is touched.
## Named exemptions
- **The test is not wired into CI, deliberately.** Adding a step to Static analysis would
add a new way for a gating job to go red, and #120 was explicit that nothing here may
change any job's status or the pass/fail rules. It runs locally
(`.github/scripts/e2e-report-shape-test.sh`) and shellcheck still covers the file, since
that step reads `git ls-files '*.sh'`. Wiring it in is a one-step follow-up if wanted.
- **Still a regex over source text, not a parser.** An annotation inside a multi-line
string, or inside a `/* */` block that opened mid-line, would still be counted. Neither
exists in this tree; if one ever does, this check wants a different tool rather than a
longer regex. Written into the script's comment, not just here.
- **The same-line case remains hypothetical in this repo.**
`grep -rnE '^\s*@[A-Za-z]+\s+@[A-Za-z]'` finds no instance in `androidTest` or `test`. It
is legal Kotlin and cheap to keep working, not something currently relied on.
- **`--include='*.kt'` is not itself tested.** A marker in a `.java` or `.txt` file under
`androidTest` is uncovered; the clause is unchanged from before this PR.
- **CLAUDE.md is unchanged.** The reasoning lives in the two script headers, and the
existing CLAUDE.md paragraph about the advisory job stays true as written.
## Gate
`assembleDebug`, `testDebugUnitTest`, `compileDebugAndroidTestKotlin`, `ktlintCheck`,
`detekt`, `lintDebug --continue` → `BUILD SUCCESSFUL`, 71 tasks. Pinned shellcheck 0.11.0
over all six tracked scripts → clean (five findings in the new test script were fixed, not
disabled: `SC1007`, and `SC2016` answered by writing the expected strings with escaped
backticks in double quotes, the same way the script under test writes them). Pinned
actionlint → clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Confirmed on this PR's own advisory run
Run 32930571300, job 98061906295 — the notice the ticket was filed about is gone, from a
real API 37 leg that aborted the way it always does:
```
----- RUN SHAPE (api37-media3-transcode) -----
expected: 3
received: 3
failed: 3
completed cleanly: no
baseline: matches (3 expected, 3 failed)
```
No `::notice::` at all. The job is still red — three tests failed, as designed.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #120.
The advisory job's baseline check has said
the tree carries 4 tests marked @FailsOnEmulatorApi37 but the baseline says 3on every PR since #113. There are three,and
FAILS_ON_EMULATOR_API37_BASELINEsays three — the fourth is a KDoc inMedia3EngineTestsaying the opposite. The old matcher looked for the string anywhereon any line, so the comment explaining why a test does not carry the marker counted as
one that does.
The matcher
Re-measured here rather than taken on trust; the ticket's table reproduces exactly.
grep -v import | grep -c FailsOn^[[:space:]]*@FailsOnEmulatorApi37[[:space:]]*$^[[:space:]]*@FailsOnEmulatorApi37([[:space:]]|$)The middle row is the trap. "Only the annotation on a line of its own" is the obvious
repair and it silently stops counting
@FailsOnEmulatorApi37 @Test, which is legalKotlin. Undercounting is the dangerous direction: it hides a genuine new marker, which is
the one thing this check exists to catch. Requiring whitespace-or-end-of-line after the
name keeps that case while still rejecting the KDoc (
^\s*@does not match* Deliberately not `@...) and the commented-out form (^\s*@does not match// @...).grep -v importgoes with it —^[[:space:]]*@cannot match an import line, so leavingit would imply a filter is still doing work.
The real tree now counts 3, which is what the baseline already says, so the baseline is
not touched. Setting it to 4 would have silenced the notice by breaking the check.
The fixture, and why it cannot move the real count
.github/scripts/testdata/marker-shapes/MarkerShapes.ktcarries every shape at once —three that count (own line, sharing a line with
@Test, indented inside a nested class)and three that must not (KDoc mention, commented out, the import).
It is under
.github/, so Gradle compiles onlyapp/src/**, ktlint and detekt areapplied to
:apponly, and the report's own count readsapp/src/androidTest. Verifiedrather than assumed — the branch script run against the real repo root with the fixture
committed:
.github/scripts/e2e-report-shape-test.shruns the real report against it, inside athrowaway repo root assembled in
mktemp -d. The script finds its own root fromBASH_SOURCE, so no knob had to be added to production code — which matters twice over:reverting the matcher reddens the test rather than a testing-only path beside it, and no
environment variable exists that could point the live count somewhere else, which is the
failure mode #83 built the baseline check to prevent. It also means
XML_DIRresolvesinside the throwaway root, so a stale
app/build/outputscannot leak in.Twelve checks, green:
Mutations — each confirmed to fail
1. Revert the matcher to the old one. The false deviation comes straight back on the
real tree:
and the test goes red, 6 of 12 checks failing — including, against the fixture,
the tree carries 5 tests marked ....2. Add a fourth REAL annotation to the fixture tree. The count moves and the deviation
fires correctly — the half that proves precision was not bought by destroying the check:
3a. Add a same-line
@FailsOnEmulatorApi37 @Test fun .... It is counted — 3 becomes 4:3b. Delete the same-line entry from the fixture — the sharper direction, since it
proves the entry is load-bearing rather than decorative. The count drops to 2 and the
shape assertion fires by name:
Byte-identical everywhere else
Per #121's method, and with both script versions run from the same worktree so the
only variable is the script text (
git show origin/main:...into.github/scripts/,deleted afterwards). Seven legs, both streams:
The two that differ are the two advisory legs, and they differ by exactly the false line
going away:
On
advisory-wedgedthe other deviation is left untouched, which is the point:The harness was proven able to fail, on both streams separately — #121's first attempt
reddened only stdout. A stray
echoadded inside the$GITHUB_STEP_SUMMARYblockreddens all seven summary diffs and leaves the five non-advisory stdout diffs green; a
stray
echoon stdout reddens all seven stdout diffs and leaves five summary diffs green.Nothing about any job changes
The advisory leg is still
continue-on-errorand still red by design; deviations arestill
::notice::, never::error::. No workflow file is touched.Named exemptions
add a new way for a gating job to go red, and #120 was explicit that nothing here may
change any job's status or the pass/fail rules. It runs locally
(
.github/scripts/e2e-report-shape-test.sh) and shellcheck still covers the file, sincethat step reads
git ls-files '*.sh'. Wiring it in is a one-step follow-up if wanted.string, or inside a
/* */block that opened mid-line, would still be counted. Neitherexists in this tree; if one ever does, this check wants a different tool rather than a
longer regex. Written into the script's comment, not just here.
grep -rnE '^\s*@[A-Za-z]+\s+@[A-Za-z]'finds no instance inandroidTestortest. Itis legal Kotlin and cheap to keep working, not something currently relied on.
--include='*.kt'is not itself tested. A marker in a.javaor.txtfile underandroidTestis uncovered; the clause is unchanged from before this PR.existing CLAUDE.md paragraph about the advisory job stays true as written.
Gate
assembleDebug,testDebugUnitTest,compileDebugAndroidTestKotlin,ktlintCheck,detekt,lintDebug --continue→BUILD SUCCESSFUL, 71 tasks. Pinned shellcheck 0.11.0over all six tracked scripts → clean (five findings in the new test script were fixed, not
disabled:
SC1007, andSC2016answered by writing the expected strings with escapedbackticks in double quotes, the same way the script under test writes them). Pinned
actionlint → clean.
🤖 Generated with Claude Code
Confirmed on this PR's own advisory run
Run 32930571300, job 98061906295 — the notice the ticket was filed about is gone, from a
real API 37 leg that aborted the way it always does:
No
::notice::at all. The job is still red — three tests failed, as designed.