Say what the advisory API 37 job actually found, so a new failure is not invisible
That job is continue-on-error and red on every PR by design, which CLAUDE.md states plainly -- and that instruction is exactly why nobody reads it. Nothing in a red X separates "the known three" from "the known three plus yours". A bare failure count would not have fixed it, and this is measured rather than assumed. The run is usually truncated: seven of eight advisory runs read on 2026-08-25 ended in `Test run failed to complete. Expected 3 tests, received 2.` with INSTRUMENTATION_ABORTED, and one did not. A count taken from a truncated run misleads in both directions -- a fourth marked test can still yield the same number if the abort lands earlier, and the known set getting worse can lower it. The test XML does not rescue it either, which was the thing worth checking before building on it: it IS written for an aborted run, and it reports a tidy tests="3" failures="3" for a run the runner had just described as truncated. So the XML is the authority on how many results landed, the runner's own output is the only authority on whether the run finished, and the report reads both and says which number came from where. The baseline is one number beside the marker, because the marker means "cannot pass on this image": the count is both how many tests the advisory leg runs and how many should fail. A smaller failure count is the interesting direction -- it means one now passes, which is the documented trigger for deleting the annotation. Nothing about the job's status changes. It stays continue-on-error, stays red, stays out of the required contexts; a deviation is a ::notice::, never an ::error::. The report is a separate script so it can be run against a real log saved from a real CI run, which is how the comparison was shown to fire. The gating legs get the shape without the comparison: they run the whole suite, so comparing there would announce a deviation five times a run -- but a truncated run reporting fewer results than it ran is what #108 looks like, and "completed cleanly" is the field that would show it. Closes #83
This commit is contained in:
@@ -377,6 +377,7 @@ jobs:
|
||||
path: |
|
||||
${{ runner.temp }}/logcat-api${{ matrix.label }}.txt
|
||||
${{ runner.temp }}/diagnostics-api${{ matrix.label }}.txt
|
||||
${{ runner.temp }}/gradle-api${{ matrix.label }}.txt
|
||||
if-no-files-found: warn
|
||||
|
||||
# Only exists when the wrapper timeout tripped, so `ignore` keeps healthy runs quiet
|
||||
@@ -467,6 +468,12 @@ jobs:
|
||||
# The complement of the gating row's notAnnotation, off the same marker,
|
||||
# so a test can never be excluded from both jobs or run in both.
|
||||
E2E_EXTRA_GRADLE_ARGS: "-Pandroid.testInstrumentationRunnerArguments.annotation=org.libremediaconverter.FailsOnEmulatorApi37"
|
||||
# Turns on the baseline comparison in the run-shape report, and only here. Every leg
|
||||
# prints the shape; this is the one that also says whether it matches
|
||||
# FAILS_ON_EMULATOR_API37_BASELINE, because this is the one whose test list is the
|
||||
# marker. A deviation is a `::notice::` -- this job stays continue-on-error and stays
|
||||
# out of the required contexts, so nothing the report finds can change a conclusion.
|
||||
E2E_ADVISORY: "1"
|
||||
with:
|
||||
# Every device pin below matches the gating row exactly, so a difference
|
||||
# between the two jobs is the test selection and nothing else.
|
||||
@@ -499,6 +506,7 @@ jobs:
|
||||
path: |
|
||||
${{ runner.temp }}/logcat-api${{ env.E2E_LABEL }}.txt
|
||||
${{ runner.temp }}/diagnostics-api${{ env.E2E_LABEL }}.txt
|
||||
${{ runner.temp }}/gradle-api${{ env.E2E_LABEL }}.txt
|
||||
if-no-files-found: warn
|
||||
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
|
||||
Reference in New Issue
Block a user