Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d94906ef42 | ||
|
|
959bd8be13 | ||
|
|
b93ef79931 | ||
|
|
d739b425c0 | ||
|
|
cd77aceff4 | ||
|
|
3d8b89bfab | ||
|
|
3f731d8ea7 | ||
|
|
25aac95db9 | ||
|
|
dce516224c |
@@ -30,16 +30,31 @@
|
||||
#
|
||||
# Usage:
|
||||
# e2e-report-shape.sh <label> <gradle-log> [<baseline-file>]
|
||||
# E2E_WEDGED_AFTER=<seconds> the wrapper timeout killed gradle after that many seconds
|
||||
#
|
||||
# With a third argument the run is compared against the baseline in that file (advisory mode)
|
||||
# and a `::notice::` is emitted per deviation. NEVER `::error::`: the advisory job is
|
||||
# `continue-on-error: true` and stays that way, and an error annotation would be a new way for
|
||||
# a diagnostic to change a conclusion.
|
||||
#
|
||||
# WHY THE WEDGE ARRIVES AS AN ENV VAR (#118) rather than being read out of the log like every
|
||||
# other field: there is nothing in the log to read. A wedge is gradle never returning, so gradle
|
||||
# never printed a verdict, never printed a truncation line, and never aborted instrumentation --
|
||||
# the log of a wedged leg is the log of a run that simply stops. Measured on job 98035980326:
|
||||
# `expected: 59`, `received: 59`, `completed cleanly: yes`, six seconds before the wedge warning,
|
||||
# for a leg that the timeout had killed 22 minutes in. Only e2e-run.sh knows, because only it
|
||||
# saw `timeout` exit 124, so it says so. Guessing it from a log that ends abruptly would call
|
||||
# every cancelled run a wedge.
|
||||
#
|
||||
# It is read as a STRING and only ever interpolated into one. `[ -n ... ]`, never `-gt`: it
|
||||
# crosses a process boundary from a shell that deliberately sets it EMPTY on every non-wedge
|
||||
# path, and an arithmetic test on an empty string is the header's rule four paragraphs up.
|
||||
set -uo pipefail
|
||||
|
||||
LABEL="${1:-unknown}"
|
||||
LOG="${2:-}"
|
||||
BASELINE_FILE="${3:-}"
|
||||
WEDGED_AFTER="${E2E_WEDGED_AFTER:-}"
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
|
||||
@@ -115,6 +130,13 @@ elif [ -n "$abort_received" ]; then
|
||||
elif [ -n "$expected" ] && [ -z "$abort_line" ]; then
|
||||
received="$expected"
|
||||
received_src="the run was not truncated, so every expected test reported"
|
||||
# ... unless it was killed, in which case "not truncated" is only "gradle never got as far as
|
||||
# saying so". This is the branch the wedged leg in #118 took -- with no XML written yet, the
|
||||
# number is what the runner was TOLD to run, and the source line said the opposite in the same
|
||||
# table that called the leg clean. The number is deliberately left alone: it is still the best
|
||||
# available answer, and only the claim about where it came from was wrong.
|
||||
[ -n "$WEDGED_AFTER" ] \
|
||||
&& received_src="no test XML was written and gradle never printed a truncation line — but the leg was killed mid-run, so this is what it was told to run, not what reported"
|
||||
fi
|
||||
|
||||
failed="unknown"
|
||||
@@ -152,6 +174,12 @@ elif [ "$no_run" = "nothing" ]; then
|
||||
# "cleanly" would be a lie about a run that left no evidence it happened.
|
||||
completed="unknown"
|
||||
completed_src="no runner output to read"
|
||||
elif [ -n "$WEDGED_AFTER" ]; then
|
||||
# The wedge is checked LAST of the four, so it only ever overrides the `yes`. The two "no"s
|
||||
# above are already right and name the abort, which the wedge row does not; `unknown` is
|
||||
# already right too. A wedge on top of an abort is both facts, and both get printed.
|
||||
completed="**no**"
|
||||
completed_src="the wrapper timeout killed gradle after ${WEDGED_AFTER}s — instrumentation itself was never aborted, which is why nothing in the log says so"
|
||||
else
|
||||
completed="yes"
|
||||
completed_src="no truncation line and no \`INSTRUMENTATION_ABORTED\`"
|
||||
@@ -217,6 +245,11 @@ echo "----- RUN SHAPE (api${LABEL}) -----"
|
||||
echo " expected: $expected"
|
||||
echo " received: $received"
|
||||
echo " failed: $failed"
|
||||
# Above `completed cleanly`, because it is the line that says what happened to the leg and the
|
||||
# other one only qualifies it. A reader who stops after three rows still sees it.
|
||||
if [ -n "$WEDGED_AFTER" ]; then
|
||||
echo " wedged: yes -- gradle was killed after ${WEDGED_AFTER}s and never returned"
|
||||
fi
|
||||
echo " completed cleanly: ${completed//\*/}"
|
||||
if [ -n "$abort_received" ]; then
|
||||
echo " received before the abort: $abort_received"
|
||||
@@ -249,6 +282,9 @@ if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
|
||||
echo "| expected | $expected | $expected_src |"
|
||||
echo "| received | $received | $received_src |"
|
||||
echo "| failed | $failed | $failed_src |"
|
||||
if [ -n "$WEDGED_AFTER" ]; then
|
||||
echo "| wedged | **yes** | \`timeout\` fired after ${WEDGED_AFTER}s and killed gradle (exit 124), which is what e2e-run.sh then captured the wedge diagnostics for |"
|
||||
fi
|
||||
echo "| completed cleanly | $completed | $completed_src |"
|
||||
if [ -n "$abort_received" ]; then
|
||||
echo "| received before the abort | $abort_received | the same line — the XML above counts the truncated test as a failure, this number does not |"
|
||||
|
||||
@@ -239,20 +239,34 @@ timeout -k 30s "$WEDGE_TIMEOUT" \
|
||||
${E2E_EXTRA_GRADLE_ARGS:-} 2>&1 | tee "$GRADLE_LOG" || status=$?
|
||||
echo "::endgroup::"
|
||||
|
||||
# Whether the wrapper timeout fired, decided ONCE. 124 is `timeout` saying it killed the
|
||||
# command, and two places downstream need that fact: capture_wedge below, and the report, which
|
||||
# otherwise calls a killed leg `completed cleanly: yes` (#118). Deriving it twice is how those
|
||||
# two would drift apart -- the report would keep printing after someone changed what a wedge
|
||||
# means here. It stays a string: empty on every other path, so those legs pass an empty
|
||||
# E2E_WEDGED_AFTER and the report behaves exactly as before.
|
||||
wedged=""
|
||||
[ "$status" -eq 124 ] && wedged="$WEDGE_TIMEOUT"
|
||||
|
||||
# The run-shape report: expected/received/failed and whether the run finished, every time,
|
||||
# green or red. It never changes `status` -- it is a diagnostic, and the header's rule about
|
||||
# diagnostics applies to it as much as to every probe below.
|
||||
#
|
||||
# E2E_WEDGED_AFTER is the wedge, told to the report rather than left for it to infer. It cannot
|
||||
# be inferred: a wedge is gradle never returning, so gradle printed no verdict at all, and the
|
||||
# log the report reads looks like a run that simply stopped. Only this script knows the
|
||||
# difference, because only this script saw the exit status.
|
||||
#
|
||||
# The baseline argument, and only it, turns on the comparison, and only the advisory API 37 job
|
||||
# passes E2E_ADVISORY=1. Comparing on the gating legs would announce a deviation on all five of
|
||||
# them every run, since they run the whole suite rather than the marked three. They still get
|
||||
# the report: a truncated run reporting fewer results than it ran is what #108 looks like, and
|
||||
# `completed cleanly` is the field that shows it.
|
||||
if [ "${E2E_ADVISORY:-}" = "1" ]; then
|
||||
bash "$SCRIPT_DIR/e2e-report-shape.sh" "$LABEL" "$GRADLE_LOG" \
|
||||
E2E_WEDGED_AFTER="$wedged" bash "$SCRIPT_DIR/e2e-report-shape.sh" "$LABEL" "$GRADLE_LOG" \
|
||||
"$REPO_ROOT/app/src/androidTest/java/org/libremediaconverter/FailsOnEmulatorApi37.kt" || true
|
||||
else
|
||||
bash "$SCRIPT_DIR/e2e-report-shape.sh" "$LABEL" "$GRADLE_LOG" || true
|
||||
E2E_WEDGED_AFTER="$wedged" bash "$SCRIPT_DIR/e2e-report-shape.sh" "$LABEL" "$GRADLE_LOG" || true
|
||||
fi
|
||||
|
||||
if [ "$status" -eq 0 ]; then
|
||||
@@ -260,7 +274,7 @@ if [ "$status" -eq 0 ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$status" -eq 124 ]; then
|
||||
if [ -n "$wedged" ]; then
|
||||
capture_wedge "api${LABEL}"
|
||||
else
|
||||
echo "::error::E2E api${LABEL} failed (exit $status)"
|
||||
|
||||
@@ -98,6 +98,13 @@ days. Read it as the current answer, and see the git history if you need the old
|
||||
is written anyway and says nothing about it — `.github/scripts/e2e-report-shape.sh` is where that
|
||||
is measured and explained.
|
||||
|
||||
Every leg prints that table, advisory or not, and **on the wedge path it also carries a `wedged:`
|
||||
row** (#118). `completed cleanly` only ever meant "instrumentation was not aborted", which stays
|
||||
true of a leg the `WEDGE_TIMEOUT` killed 22 minutes in — so without that row the table read
|
||||
`received: 59, completed cleanly: yes` for a leg that had just died. The wedge is passed to the
|
||||
report as `E2E_WEDGED_AFTER` by `e2e-run.sh`, which is the only thing that can know it: a wedge
|
||||
is gradle never returning, so the log it left says nothing about it.
|
||||
|
||||
Still true, and the reason the advisory job is not simply deleted: **API 37 needs a manual check on
|
||||
the Pixel 10 Pro XL before each release.** Those three tests are the one thing CI cannot answer
|
||||
for.
|
||||
|
||||
@@ -178,7 +178,12 @@ object ContainerCapabilities {
|
||||
if (!probe.hasVideo) {
|
||||
return Validation.Invalid(
|
||||
"This file has no video track to copy.",
|
||||
listOf(spec.copy(videoCodec = VideoCodec.NONE)),
|
||||
// Dropping the video is the right shape of answer, but it is only half of one:
|
||||
// `spec.copy(videoCodec = NONE)` is valid exactly when the audio axis already
|
||||
// happened to be fine, and refused otherwise — a Vorbis or PCM source into MP4,
|
||||
// an MP3 into WebM. Handing it to the shared path repairs both axes and drops
|
||||
// anything that still fails, so the chip cannot lead to a second error.
|
||||
suggestions(spec.copy(videoCodec = VideoCodec.NONE), probe, exclude = spec),
|
||||
)
|
||||
}
|
||||
val source = CodecNames.videoFromName(probe.videoCodec)
|
||||
|
||||
@@ -11,6 +11,11 @@ import org.junit.Test
|
||||
* `OutputFormat` used to be twelve hand-picked triples, and its KDoc defended that on the grounds
|
||||
* that a closed set was what made routing decidable. Opening it up moves that burden here, so this
|
||||
* is where decidability now has to be proven.
|
||||
*
|
||||
* That includes what a refusal offers instead. `Validation.Invalid` promises every suggestion is
|
||||
* itself valid and names this class as the proof, so a branch that assembles its own suggestion
|
||||
* list rather than going through `suggestions()` is only checked here if some row happens to reach
|
||||
* it — which is how a dead-end chip survived two widenings of that table.
|
||||
*/
|
||||
class ContainerCapabilitiesTest {
|
||||
|
||||
@@ -35,6 +40,29 @@ class ContainerCapabilitiesTest {
|
||||
container = Container.MP3,
|
||||
)
|
||||
|
||||
/**
|
||||
* An audio-only input carrying a codec MP4 has no place for at all.
|
||||
*
|
||||
* Vorbis lives in Ogg and Matroska; MP4 carries AAC, MP3, Opus and FLAC. That gap is what turns
|
||||
* a suggestion which merely drops the video track into a second refusal.
|
||||
*/
|
||||
private val vorbisSource = InputProbe(
|
||||
videoCodec = null,
|
||||
audioCodec = "vorbis",
|
||||
hasVideo = false,
|
||||
kind = InputKind.AUDIO_ONLY,
|
||||
container = Container.OGG,
|
||||
)
|
||||
|
||||
/** The same shape, for the other codec MP4 refuses. One case is a coincidence; two is the rule. */
|
||||
private val pcmSource = InputProbe(
|
||||
videoCodec = null,
|
||||
audioCodec = "pcm_s16le",
|
||||
hasVideo = false,
|
||||
kind = InputKind.AUDIO_ONLY,
|
||||
container = Container.WAV,
|
||||
)
|
||||
|
||||
// --- copy and encode are different questions ----------------------------
|
||||
|
||||
/**
|
||||
@@ -97,7 +125,17 @@ class ContainerCapabilitiesTest {
|
||||
}
|
||||
}
|
||||
|
||||
/** A suggestion that is itself invalid is worse than no suggestion. */
|
||||
/**
|
||||
* A suggestion that is itself invalid is worse than no suggestion.
|
||||
*
|
||||
* Only a branch that assembles its own suggestion list can break that promise: [suggestions]
|
||||
* ends by filtering on `validate(...).isValid`, so everything routed through it is valid by
|
||||
* construction. Those branches are what this table has to cover — the image output, and copy
|
||||
* the video from a file that has none, which built its list by hand and came back refused for
|
||||
* a Vorbis or PCM source into MP4 and an MP3 into WebM. The Advanced picker showed a one-tap
|
||||
* fix that led straight to a second error, through two widenings of this table that never
|
||||
* reached the branch.
|
||||
*/
|
||||
@Test
|
||||
fun `every suggestion is itself valid`() {
|
||||
val cases = listOf(
|
||||
@@ -109,15 +147,31 @@ class ContainerCapabilitiesTest {
|
||||
OutputSpec(Container.MP4, VideoCodec.COPY, AudioCodec.NONE) to mp3Source,
|
||||
OutputSpec(Container.MP4, VideoCodec.NONE, AudioCodec.NONE) to mp3Source,
|
||||
OutputSpec(Container.MP4, VideoCodec.COPY, AudioCodec.AAC) to mp3Source,
|
||||
// Copy-the-video-from-a-file-with-no-video, the last branch that built its offer by
|
||||
// hand. It escaped the five rows above because `spec.copy(videoCodec = NONE)` is valid
|
||||
// exactly when the audio axis happens to be fine — true for the AAC and MP3 sources
|
||||
// used there, false for any audio the target container cannot carry.
|
||||
OutputSpec(Container.MP4, VideoCodec.COPY, AudioCodec.COPY) to vorbisSource,
|
||||
OutputSpec(Container.MP4, VideoCodec.COPY, AudioCodec.COPY) to pcmSource,
|
||||
OutputSpec(Container.WEBM, VideoCodec.COPY, AudioCodec.COPY) to mp3Source,
|
||||
// The same branch with audio the container *can* hold, which is the half that already
|
||||
// worked and must keep working: the repair here is a copy, so the offer is the very
|
||||
// spec the caller handed to `suggestions`. It survives only because the exclusion is
|
||||
// against what the user asked for rather than against the repair.
|
||||
OutputSpec(Container.MP4, VideoCodec.COPY, AudioCodec.COPY) to mp3Source,
|
||||
// The one branch that still builds its list by hand, so that it is asserted rather
|
||||
// than merely reasoned about: an image container takes `None + None` and nothing else,
|
||||
// which makes its single offer valid by construction.
|
||||
OutputSpec(Container.GIF, VideoCodec.H264, AudioCodec.AAC) to h264Source,
|
||||
)
|
||||
|
||||
cases.forEach { (spec, probe) ->
|
||||
val invalid = ContainerCapabilities.validate(spec, probe) as? Validation.Invalid
|
||||
?: throw AssertionError("expected $spec to be rejected")
|
||||
assertTrue("no alternatives offered for $spec", invalid.suggestions.isNotEmpty())
|
||||
assertTrue("no alternatives offered for $spec on $probe", invalid.suggestions.isNotEmpty())
|
||||
invalid.suggestions.forEach { suggestion ->
|
||||
assertTrue(
|
||||
"suggested $suggestion for $spec is itself invalid",
|
||||
"suggested $suggestion for $spec on $probe is itself invalid",
|
||||
ContainerCapabilities.validate(suggestion, probe).isValid,
|
||||
)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user