Make the fallback test say when it cannot test the fallback (#223)

HardwareFallbackTest is the only automated check of the hardware->software fallback against a real codec failure, and it passed on every CI leg without ever attempting the hardware path. Measured on run 34004304566: the API 33, 34, 35 and 37 legs each log Routing sample_h264_444.mp4 -> ... via FFMPEG (NO_HARDWARE_ENCODER) Emulators expose no hardware encoder, so the router never chooses Media3 and runMedia3OrFallBack's catch is never entered. The test's assertions -- succeeded, output non-empty -- are true of that conversion too. It finished in 448 ms, which is not long enough to fail an export and then software-encode a three-second clip. Deleting the catch reddened nothing. The ticket offered two fixes and left the choice open. Trying the first one answered it, and not the way the ticket expected. Pinning deviceCodecs to PERMISSIVE, as ForcedFailureTest does, makes the router choose Media3 -- and the export then SUCCEEDS. On a local API 34 emulator, MediaCodecInfo logs NoSupport [codec.profileLevel, avc1.F4000C, video/avc] for both c2.goldfish.h264.decoder and c2.android.avc.decoder, and ExoPlayer allocates the goldfish decoder anyway, which decodes the High 4:4:4 fixture regardless of the profile it declares. c2.android.hevc.encoder then encodes it and the job reports MEDIA3. So the class KDoc's "Media3 fails partway through the export on every device" is not true of the emulator images, and no routing pressure makes this fixture force a fallback there. Pinning would also swap in software codecs, which is not the path a real device takes -- it is what made the forced run succeed. That leaves assumeTrue on the production premise as the honest answer, now with a measurement behind it rather than a coin flip. The test skips where it cannot mean anything and runs on the Pixel, where it always could. When it does run the assertion is a pair, because KEY_ENGINE_USED is FFMPEG whether the fallback fired or the router went straight there: the router chose MEDIA3 for this request on this device, AND the worker reported FFMPEG. Together, and only together, that is the fallback. Verified on a local API 34 emulator: the test reports SKIPPED and the level reports skipped=3. ForcedFailureTest still covers the fallback wiring on every leg with a double; what needs a real encoder is two real engines disagreeing about a real file. The third permanent skip is recorded in docs/local-emulator.md and beside SafPickerRoundTripTest's run-shape note. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-05 22:56:18 -05:00
parent bffcff92c7
commit ba16f5a89b
3 changed files with 91 additions and 0 deletions
@@ -12,11 +12,17 @@ import kotlinx.coroutines.withTimeout
 import org.junit.After
 import org.junit.Assert.assertEquals
 import org.junit.Assert.assertTrue
+import org.junit.Assume.assumeTrue
 import org.junit.Before
 import org.junit.Test
 import org.junit.runner.RunWith
+import org.libremediaconverter.codec.AndroidDeviceCodecs
+import org.libremediaconverter.model.ConversionRequest
+import org.libremediaconverter.model.ConversionRouter
+import org.libremediaconverter.model.Engine
 import org.libremediaconverter.model.OutputFormat
 import org.libremediaconverter.model.QualityTier
+import org.libremediaconverter.model.VideoCodec
 import org.libremediaconverter.work.ConversionWorker
 import java.io.File

@@ -34,6 +40,47 @@ import java.io.File
 * hand — a regression test that silently skips is worse than no test, because the count
 * still reads as coverage.
 *
+ * ## Why this skips on emulators, and why that is the honest answer (#223)
+ *
+ * **This test used to pass everywhere while proving nothing.** Two independent facts stop the
+ * fallback happening on an emulator, and both were measured rather than reasoned:
+ *
+ *  1. **The router never sends the job to Media3.** A Fast MP4/H.265 job goes to the hardware path
+ *     only when `device.canEncode(H265)`, and emulators expose no hardware encoder — every leg of
+ *     run `34004304566` logged
+ *     `Routing sample_h264_444.mp4 -> ... via FFMPEG (NO_HARDWARE_ENCODER)`. The whole test
+ *     finished in 448 ms, which is not long enough to fail an export and then re-encode.
+ *  2. **Forcing it to Media3 does not help either, which is the part that settles it.** Pinning
+ *     `ConversionDependencies.deviceCodecs` to [DeviceCodecs.PERMISSIVE] — the trick
+ *     [ForcedFailureTest] uses — makes the router choose Media3, and the export then *succeeds*.
+ *     Measured on a local API 34 emulator: `MediaCodecInfo` logs
+ *     `NoSupport [codec.profileLevel, avc1.F4000C, video/avc]` for **both**
+ *     `c2.goldfish.h264.decoder` and `c2.android.avc.decoder`, and ExoPlayer allocates the
+ *     goldfish decoder anyway, which decodes the file regardless of the profile it declares.
+ *     `c2.android.hevc.encoder` then encodes the result and the job reports `MEDIA3`.
+ *
+ * So the class KDoc above — "Media3 fails partway through the export on every device" — **is not
+ * true of the emulator images**, and no amount of routing pressure makes this fixture force a
+ * fallback there. The emulator cannot answer this question, so the test says so out loud instead
+ * of passing.
+ *
+ * That is why the gate is [assumeTrue] on the *production* premise (`canEncode(H265)`) rather than
+ * a pinned profile: pinning would also swap in software codecs, which is not the path a real
+ * device takes and is what made the forced run succeed. **This is now the third permanent skip**;
+ * the other two are [org.libremediaconverter.bench.RealMediaBenchmark]'s.
+ *
+ * `ForcedFailureTest.hardwareFailureFallsBackToSoftware` still covers the fallback *wiring* on
+ * every leg, with an `ExplodingHardware` double. What only a device with a real hardware encoder
+ * can show is two real engines disagreeing about a real file, and that is what this is for.
+ *
+ * ## Why the assertion is a pair
+ *
+ * `KEY_ENGINE_USED` is `FFMPEG` whether the fallback fired **or** the router went straight there,
+ * so asserting it alone would not have caught any of the above. The premise is asserted
+ * separately: [ConversionRouter.route] chooses `MEDIA3` for this request on this device. Static
+ * routing wanted hardware, the runtime result was software — together, and only together, that is
+ * the fallback.
+ *
 * The fixture was produced with x264, which the host toolchain cannot do (Fedora's
 * ffmpeg ships openh264, which is Constrained Baseline only):
 *
@@ -66,6 +113,15 @@ class HardwareFallbackTest {

    @Test
    fun aFileMedia3CannotDecodeStillConvertsViaFfmpeg(): Unit = runBlocking {
+        // See "Why this skips on emulators" on the class. Without a real hardware encoder the
+        // router never chooses Media3, and forcing it makes the export succeed instead of fail --
+        // so there is no fallback to observe and a green run would mean nothing.
+        assumeTrue(
+            "no hardware HEVC encoder, so the router cannot choose Media3 and there is no " +
+                "fallback to exercise",
+            AndroidDeviceCodecs.get().canEncode(VideoCodec.H265),
+        )
+
        val request = ConversionWorker.request(
            inputUri = Uri.fromFile(input),
            displayName = SAMPLE,
@@ -75,6 +131,19 @@ class HardwareFallbackTest {
            // the tier where the fallback has to rescue the conversion.
            quality = QualityTier.FAST,
        )
+        // The premise, asserted rather than assumed: this request is one the router wants to send
+        // to hardware on this device. Without it the test is green whether the fallback fired or
+        // the job never went near Media3, which is exactly how #223 stayed invisible.
+        val decision = ConversionRouter.route(
+            ConversionRequest(OutputFormat.MP4_H265.spec, quality = QualityTier.FAST),
+            AndroidDeviceCodecs.get(),
+        )
+        assertEquals(
+            "this test only means something if the router sends this job to Media3",
+            Engine.MEDIA3,
+            decision.engine,
+        )
+
        workManager.enqueue(request).result.get()

        val terminal = withTimeout(TIMEOUT_MS) {
@@ -88,6 +157,14 @@ class HardwareFallbackTest {
            terminal?.state,
        )

+        // The outcome. Paired with the routing assertion above this is the fallback and nothing
+        // else: hardware was chosen, software is what ran.
+        assertEquals(
+            "the router chose Media3, so a successful job must have fallen back to FFmpeg",
+            Engine.FFMPEG.name,
+            terminal?.outputData?.getString(ConversionWorker.KEY_ENGINE_USED),
+        )
+
        val out = File(terminal!!.outputData.getString(ConversionWorker.KEY_OUTPUT_PATH)!!)
        assertTrue("no output produced", out.exists() && out.length() > 0)
        out.delete()
@@ -207,6 +207,8 @@ import java.util.concurrent.atomic.AtomicInteger
 * driven there at all. That is why this gap survived as long as it did.
 * `tools/local-emulator/run-e2e.sh` runs API 33-36 on the development host, and both tests pass
 * there: **59 / 0 / 0 / 2 at API 33 and again at API 36**, whole suite, 2026-08-24.
+ * (Since #223 the skip column reads 3 on an emulator — `HardwareFallbackTest` now announces
+ * that it cannot run without a hardware HEVC encoder rather than passing vacuously.)
 *
 * ### Why only the rotation test carries [FailsOnEmulatorApi37]
 *
@@ -312,6 +312,18 @@ on sample media that is deliberately not committed. Its third test,
 `reportDeviceEncoderCapabilities`, has no such guard and runs. A level reporting 0 skipped
 would mean someone had staged sample files, not that something improved.

+**Since #223 there is a third, and it is the interesting one.**
+`HardwareFallbackTest.aFileMedia3CannotDecodeStillConvertsViaFfmpeg` is `assumeTrue`-guarded on
+`AndroidDeviceCodecs.get().canEncode(H265)`, which is false on every emulator image — so it now
+skips here and runs only on the Pixel. It used to *pass* on emulators without ever attempting the
+hardware path, which is worse. **Expect `skipped="3"` locally**, and note the guard is a property
+of the machine rather than of staged files: a level reporting 2 would mean an emulator image had
+gained a hardware HEVC encoder, which is worth knowing.
+
+That test's KDoc carries the measurement, including the part that decides it: forcing the route to
+Media3 anyway does *not* produce a fallback, because the goldfish decoder decodes the High 4:4:4
+fixture despite declaring `NoSupport` for its profile.
+
 ### What the sweep adds, and what it does not

 **The renderer rule held four more times.** No boot log contains the string