JMR-dev/LibreMediaConverter
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

HardwareFallbackTest passes on every emulator leg without ever attempting the hardware path #223

New Issue
Closed
opened 2026-09-06 02:52:56 +00:00 by JMR-dev · 0 comments
JMR-dev commented 2026-09-06 02:52:56 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Filed from the 2026-09-05 e2e read of the instrumented suite on main @ 4b02294. The findings that a test would not fix are E1-E6 in docs/e2e-read-findings.md.

HardwareFallbackTest.aFileMedia3CannotDecodeStillConvertsViaFfmpeg is the only automated test of the hardware→software fallback against a real codec failure. It passes on every CI leg without ever attempting the hardware path.

Measured, not inferred

Run 34004304566 (all legs green), from each leg's own e2e-diagnostics-api* logcat:

I/AndroidDeviceCodecs: Hardware video encoders: []
I/RealMediaBenchmark: BENCH can-encode: COPY=true, H264=false, H265=false, VP9=false, VP8=false, AV1=false
I/ConversionWorker: Routing sample_h264_444.mp4 -> OutputSpec(container=MP4, videoCodec=H265,
                    audioCodec=AAC) via FFMPEG (NO_HARDWARE_ENCODER)

Identical on API 33, 34, 35 and 37. API 36's logcat artifact on that run is truncated to 838 KB with no test output, so it is unread rather than different.

The test ran in 448 ms — a failed HEVC hardware export followed by a full software re-encode of a 3 s clip cannot happen in that time.

Why it passes anyway

assertEquals(WorkInfo.State.SUCCEEDED, terminal?.state)
assertTrue("no output produced", out.exists() && out.length() > 0)

Both are true of a conversion that went straight to FFmpeg. ConversionRouter.kt:153-155 routes to Media3 only when device.canEncode(H265); on an emulator that is false, so runMedia3OrFallBack's catch (ConversionWorker.kt:212-217) is never entered. The test has no hardware precondition and asserts nothing about the path.

Deleting that catch entirely reddens nothing, anywhere, on any leg. That is the mutation.

The repository already knew this

ForcedFailureTest.hardwareFailureFallsBackToSoftware, in the same package, pins the device profile and says why:

most emulators expose no hardware video encoder at all -- so the router would legitimately send the job straight to FFmpeg and the hardware path would never be attempted. Without this the test passes on a Pixel and fails on every emulator, which says nothing about the code under test.

ConversionWorkerTest.routesAFastMp4JobByDeviceCapability:142-148 records the same fact a third time.

The asymmetry is why nobody noticed. ForcedFailureTest asserts hardware.called, so without its pin it would fail on emulators — loudly. HardwareFallbackTest asserts only that an output appeared, so it passes. Same hazard, opposite symptom.

State the gap precisely

The fallback wiring is covered on every leg by ForcedFailureTest, with fakes (ExplodingHardware + RecordingSoftware). This ticket is not "the fallback is untested".

What has never run on any emulator is a fallback triggered by a real mid-export codec failure — the case this test exists for, and the only reason sample_h264_444.mp4 is committed. That fixture was generated with x264 specifically because Fedora's ffmpeg ships openh264 and cannot produce High 4:4:4 (HardwareFallbackTest.kt:38-43), and it does nothing on any CI leg today.

Its own KDoc states the standard it fails:

a regression test that silently skips is worse than no test, because the count still reads as coverage

It does not even skip.

What a fix has to decide

Not assertEquals(MEDIA3, KEY_ENGINE_USED). After a fallback the engine used is FFmpeg — that is the point — and it is FFmpeg whether the fallback fired or the router went straight there. Asserting it changes nothing.

The vacuity guard is two facts together: the router chose MEDIA3 for this request on this device, and the worker reported FFMPEG. Together they say static routing wanted hardware and the runtime result was software, which is the fallback and nothing else.

Reaching the first fact is the decision, and there is no third source of truth on a device — MediaCodecList is what AndroidDeviceCodecs reads, so any oracle built from it is the same oracle (E6):

  • assumeTrue(AndroidDeviceCodecs.get().canEncode(H265)) — a visible skip on emulators. The suite's "2 skipped" becomes 3, which is a number docs/local-emulator.md:305 and every leg's report already track. Honest, and the test still runs on the Pixel.
  • Assert the routing decision — red on emulators, announcing that the leg cannot test what the class claims. Correct, and it turns a permanently-red leg into noise of the kind #190 and #108 already cost an hour each.

Both are defensible and the ticket deliberately does not choose — same shape as #178 and #204. Whichever wins, say so in the class KDoc, because the current one claims the opposite of what runs.

Done means

The mutation above (delete the catch) goes red somewhere, or it is written down that it cannot on emulator hardware and the test announces that fact on every run. A green leg must stop being readable as evidence that the fallback works.

Related

  • E6 in docs/e2e-read-findings.md — why there is no independent oracle
  • #194 — AndroidDeviceCodecs' runCatching fallback returns empty sets, which would make this vacuous on the Pixel too
  • The Pixel 10 Pro XL pre-release check is currently the only thing that exercises this path
_Filed from the 2026-09-05 e2e read of the instrumented suite on `main` @ `4b02294`. The findings that a test would **not** fix are `E1`-`E6` in `docs/e2e-read-findings.md`._ `HardwareFallbackTest.aFileMedia3CannotDecodeStillConvertsViaFfmpeg` is the only automated test of the hardware→software fallback **against a real codec failure**. It passes on every CI leg without ever attempting the hardware path. ## Measured, not inferred Run **`34004304566`** (all legs green), from each leg's own `e2e-diagnostics-api*` logcat: ``` I/AndroidDeviceCodecs: Hardware video encoders: [] I/RealMediaBenchmark: BENCH can-encode: COPY=true, H264=false, H265=false, VP9=false, VP8=false, AV1=false I/ConversionWorker: Routing sample_h264_444.mp4 -> OutputSpec(container=MP4, videoCodec=H265, audioCodec=AAC) via FFMPEG (NO_HARDWARE_ENCODER) ``` Identical on **API 33, 34, 35 and 37**. API 36's logcat artifact on that run is truncated to 838 KB with no test output, so it is unread rather than different. The test ran in **448 ms** — a failed HEVC hardware export followed by a full software re-encode of a 3 s clip cannot happen in that time. ## Why it passes anyway ```kotlin assertEquals(WorkInfo.State.SUCCEEDED, terminal?.state) assertTrue("no output produced", out.exists() && out.length() > 0) ``` Both are true of a conversion that went straight to FFmpeg. `ConversionRouter.kt:153-155` routes to Media3 only when `device.canEncode(H265)`; on an emulator that is false, so `runMedia3OrFallBack`'s `catch` (`ConversionWorker.kt:212-217`) is never entered. The test has no hardware precondition and asserts nothing about the path. **Deleting that `catch` entirely reddens nothing, anywhere, on any leg.** That is the mutation. ## The repository already knew this `ForcedFailureTest.hardwareFailureFallsBackToSoftware`, in the same package, pins the device profile and says why: > most emulators expose no hardware video encoder at all -- so the router would legitimately send the job straight to FFmpeg and the hardware path would never be attempted. Without this the test passes on a Pixel and fails on every emulator, which says nothing about the code under test. `ConversionWorkerTest.routesAFastMp4JobByDeviceCapability:142-148` records the same fact a third time. **The asymmetry is why nobody noticed.** `ForcedFailureTest` asserts `hardware.called`, so without its pin it would *fail* on emulators — loudly. `HardwareFallbackTest` asserts only that an output appeared, so it *passes*. Same hazard, opposite symptom. ## State the gap precisely The fallback **wiring** is covered on every leg by `ForcedFailureTest`, with fakes (`ExplodingHardware` + `RecordingSoftware`). This ticket is **not** "the fallback is untested". What has never run on any emulator is a fallback triggered by a **real** mid-export codec failure — the case this test exists for, and the only reason `sample_h264_444.mp4` is committed. That fixture was generated with x264 specifically because Fedora's ffmpeg ships openh264 and cannot produce High 4:4:4 (`HardwareFallbackTest.kt:38-43`), and it does nothing on any CI leg today. Its own KDoc states the standard it fails: > a regression test that silently skips is worse than no test, because the count still reads as coverage It does not even skip. ## What a fix has to decide **Not** `assertEquals(MEDIA3, KEY_ENGINE_USED)`. After a fallback the engine used *is* FFmpeg — that is the point — and it is FFmpeg whether the fallback fired or the router went straight there. Asserting it changes nothing. The vacuity guard is **two facts together**: the router chose `MEDIA3` for this request on this device, **and** the worker reported `FFMPEG`. Together they say static routing wanted hardware and the runtime result was software, which is the fallback and nothing else. Reaching the first fact is the decision, and there is no third source of truth on a device — `MediaCodecList` is what `AndroidDeviceCodecs` reads, so any oracle built from it is the same oracle (`E6`): - **`assumeTrue(AndroidDeviceCodecs.get().canEncode(H265))`** — a visible skip on emulators. The suite's "2 skipped" becomes 3, which is a number `docs/local-emulator.md:305` and every leg's report already track. Honest, and the test still runs on the Pixel. - **Assert the routing decision** — red on emulators, announcing that the leg cannot test what the class claims. Correct, and it turns a permanently-red leg into noise of the kind #190 and #108 already cost an hour each. **Both are defensible and the ticket deliberately does not choose** — same shape as #178 and #204. Whichever wins, say so in the class KDoc, because the current one claims the opposite of what runs. ## Done means The mutation above (delete the `catch`) goes red somewhere, or it is written down that it cannot on emulator hardware and the test announces that fact on every run. A green leg must stop being readable as evidence that the fallback works. ## Related - `E6` in `docs/e2e-read-findings.md` — why there is no independent oracle - #194 — `AndroidDeviceCodecs`' `runCatching` fallback returns empty sets, which would make this vacuous on the Pixel too - The Pixel 10 Pro XL pre-release check is currently the only thing that exercises this path
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
above-cut
Worked autonomously overnight: local, JVM-verifiable, no product decision
 accessibility
Barrier affecting people with disabilities
 backlog
Held for manual review: product/UX call, CI/workflow, hardware, or unverifiable here
 bug
Something isn't working
 confirmed
Reviewer demonstrated the defect
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 plausible
Reviewer could not fully demonstrate it; treat as unproven
 question
Further information is requested
 sev:high
High severity
 sev:low
Low severity
 sev:medium
Medium severity
 wontfix
This will not be worked on
No labels bug confirmed sev:medium
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
JMR-dev
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: JMR-dev/LibreMediaConverter#223