diff --git a/.github/scripts/e2e-report-shape.sh b/.github/scripts/e2e-report-shape.sh new file mode 100755 index 0000000..865bef7 --- /dev/null +++ b/.github/scripts/e2e-report-shape.sh @@ -0,0 +1,270 @@ +#!/usr/bin/env bash +# +# Reports the SHAPE of an instrumented run -- how many tests were expected, how many +# reported, how many failed, and whether the run completed at all -- to the step log and to +# the job summary. In advisory mode it also compares that shape against a committed baseline +# and says plainly whether it matches. +# +# WHY THIS EXISTS (#83): the advisory API 37 leg is red on every PR by design, so a NEW failure +# joining the known ones is invisible -- nothing in a red X distinguishes "the known ones" from +# "the known ones plus yours". CLAUDE.md tells everyone not to read that job's red as their +# change breaking something, which is correct, and which also means nobody looks. +# +# WHY NOT A BARE FAILURE COUNT, measured rather than assumed. On this image the run is usually +# truncated: `Test run failed to complete. Expected 3 tests, received 2.` with +# `INSTRUMENTATION_ABORTED: System has crashed.` A count taken from a truncated run misleads in +# both directions -- a fourth marked test can still yield the same number if the abort lands +# earlier, and the known set getting worse can LOWER it. So all four fields are recorded, and +# the one saying the run was truncated is recorded with them. +# +# WHY IT IS A SEPARATE SCRIPT rather than a function inside e2e-run.sh: it is a pure seam. It +# reads a captured log plus the test XML and writes a report, so it can be run against a REAL +# log saved from a REAL CI run -- which is how the baseline comparison was shown to fire +# without waiting on an emulator. `git ls-files '*.sh'` also picks it up for shellcheck for +# free. +# +# THIS SCRIPT NEVER FAILS A RUN. It is a diagnostic, and e2e-run.sh's header explains why that +# rule is absolute here. Every field defaults to `unknown` and every comparison is guarded, +# because an unset variable under `set -u`, or a `[ "" -eq 3 ]`, is exactly how a diagnostic +# becomes the thing that turns a leg red. It exits 0 unconditionally. +# +# Usage: +# e2e-report-shape.sh