JMR-dev/LibreMediaConverter
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare the advisory failure count only on a run that finished

Browse Source 
The verification dispatch of the reworked harness (34011072884) came back
4 expected / 3 received / 3 failed, where the one before it (34008889182) had
been 4/4/4 on the identical configuration. Nothing about the test list changed
between them: the abort landed one test earlier and the picker test never
started.

The baseline check would have called that "one now passes", which is the wrong
reading and the kind of notice #120 is about -- a deviation that is wrong often
enough to teach everyone to skim past deviation notices. So `failed` is compared
only when `completed cleanly` is yes, and `expected` is compared always, because
`Starting N tests` is printed before anything can abort and is what actually
answers "is the marked set the size the baseline says".

Two cases in e2e-report-shape-test.sh, as a pair: a truncated run short by one is
not a deviation, and a CLEAN run short by one still is -- so the first cannot have
bought its quiet by disabling the check.

This is a consequence of adding the fourth marker rather than a pre-existing bug
worth its own ticket: with three, the advisory leg had been completing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
JMR-dev
2026-09-05 23:26:56 -05:00
co-authored by Claude Opus 5
parent 97558c259f
commit 557b3edab4
3 changed files with 63 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/scripts/e2e-report-shape.sh
+15 -1
View File
@@ -252,8 +252,22 @@ if [ -n "$baseline" ]; then
if [ "$expected" != "unknown" ] && [ "$expected" != "$baseline" ]; then
deviations+=("the runner started $expected tests, the baseline is $baseline")
fi
# `expected` is compared on every run and `failed` only on a run that finished, and the
# difference is the truncation this file already records rather than compares. `expected`
# comes from `Starting N tests`, which is printed before anything can abort, so it answers
# "is the marked set the size the baseline says" whatever happens afterwards. `failed` is a
# tally of what actually ran: on a truncated run the tests after the abort never start, so
# comparing it to the baseline announces a deviation about the framework dying rather than
# about the test list. Measured on 2026-09-05, two api37-debug dispatches of the same four
# marked tests: 4/4/4 and then 4/3/3, the second having lost the last test to the abort.
# Announcing that as "one now passes" is exactly the wrong reading, and #120 is the standing
# lesson about a notice that is wrong often enough to be skimmed past.
if [ "$failed" != "unknown" ] && [ "$failed" != "$baseline" ]; then
deviations+=("$failed tests failed, the baseline is $baseline — every test carrying the marker is expected to fail on this image, so fewer means one now passes and more means a new one joined")
if [ "$completed" = "**no**" ]; then
echo "::debug::$failed of $baseline marked tests failed, on a run the abort truncated — not compared"
else
deviations+=("$failed tests failed, the baseline is $baseline — every test carrying the marker is expected to fail on this image, so fewer means one now passes and more means a new one joined")
fi
fi
fi
if [ -n "$marked" ] && [ "$marked" != "$baseline" ]; then