JMR-dev/LibreMediaConverter
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

RealMediaBenchmark documents a populate path its own code explains cannot work #101

New Issue
Closed
opened 2026-08-25 14:19:48 +00:00 by JMR-dev · 1 comment
JMR-dev commented 2026-08-25 14:19:48 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Two findings in app/src/androidTest/java/org/libremediaconverter/bench/RealMediaBenchmark.kt. The first is certain; the second is observed but not explained, and is written that way.

1. The file tells you to populate it in the one way it later explains cannot work

The class KDoc:

Skips itself when the sample files are absent, so it is harmless in CI. Populate with:
adb push <file>.mp4 /sdcard/Android/data/org.libremediaconverter/files/

The samples property KDoc, twelve lines below, on private val samples: File get() = context.filesDir:

Internal storage, not the external files dir. Files placed in the external dir by adb push
or adb shell cp stay owned by the shell user, and the app then gets EACCES trying to read
them — which presents as an unparseable input rather than a permission problem. Piping through
run-as writes as the app's own uid, so ownership is unambiguous.

context.filesDir is /data/user/0/org.libremediaconverter/files/. The push target is
/sdcard/Android/data/org.libremediaconverter/files/, which is getExternalFilesDir(null). They
are different directories, and the property KDoc exists specifically to explain why the class
KDoc's instruction fails — including that it fails confusingly, as an unparseable input rather
than a permission error.

So anyone following the documented procedure stages files the benchmark cannot read, and gets a
skip they will read as "not staged yet". The fix is to make the class KDoc name the run-as route
its own property KDoc already describes.

Same defect family as R14, R15, R20, R25 and #46 — a claim that was true when written, next to the
correction, with nothing making them agree.

2. Observed, unexplained: the benchmark ran and failed on CI

The same KDoc says "harmless in CI". It was not, twice today, on gating legs:

PR leg signature
#95 E2E API 36 Abort: no output sample written in the last 25000 milliseconds / ExportException: Muxer error
#97 E2E API 34 same signature; failing tests named as RealMediaBenchmark > hardwareVersusSoftwareOnRealVideo and one more

What is not established: why. hardwareVersusSoftwareOnRealVideo has assumeTrue(input != null)
on line 87, and an unmet assumption reports as skipped, not failed — so on the face of it the
sample was present in context.filesDir on a CI emulator, which should not happen. I have not
confirmed that, and I am not asserting it.

Worth checking in this order:

  1. Does the run actually enter the body — is the sample really there, and if so what wrote it?
  2. Or is the failure arriving from somewhere else in the same leg and the benchmark names simply
    appearing alongside it?
  3. no output sample written in the last 25000 milliseconds is a starvation signature. #93's
    root cause was the launcher ANRing on a loaded runner; a runner loaded enough for that would also
    starve a hardware transcode. These may be one problem, in which case #96 fixed the accessibility
    symptom and this is the same illness elsewhere.

Why it matters beyond tidiness

A benchmark in a gating suite is timing-sensitive by construction, and this one's own KDoc says
it is "Not a correctness test — the assertions are deliberately loose" and "not part of the
automated suite". If it can run on CI at all, the gating legs inherit the flakiness of a
measurement rather than of a check.

Done means

  • The class KDoc names the run-as route, so the two halves of the file agree. Acceptance: a
    reader following the class KDoc alone stages files the test can read.
  • The CI question is answered with a measurement rather than a theory — either the benchmark cannot
    run on CI (and it is shown), or it can (and it is excluded from the gating legs, or the KDoc stops
    claiming otherwise).
_Two findings in `app/src/androidTest/java/org/libremediaconverter/bench/RealMediaBenchmark.kt`. The first is certain; the second is observed but not explained, and is written that way._ ### 1. The file tells you to populate it in the one way it later explains cannot work The **class** KDoc: > Skips itself when the sample files are absent, so it is harmless in CI. Populate with: > `adb push <file>.mp4 /sdcard/Android/data/org.libremediaconverter/files/` The **`samples` property** KDoc, twelve lines below, on `private val samples: File get() = context.filesDir`: > **Internal storage, not the external files dir.** Files placed in the external dir by `adb push` > or `adb shell cp` stay owned by the shell user, and the app then gets EACCES trying to read > them — which presents as an unparseable input rather than a permission problem. Piping through > `run-as` writes as the app's own uid, so ownership is unambiguous. `context.filesDir` is `/data/user/0/org.libremediaconverter/files/`. The push target is `/sdcard/Android/data/org.libremediaconverter/files/`, which is `getExternalFilesDir(null)`. **They are different directories**, and the property KDoc exists specifically to explain why the class KDoc's instruction fails — including that it fails *confusingly*, as an unparseable input rather than a permission error. So anyone following the documented procedure stages files the benchmark cannot read, and gets a skip they will read as "not staged yet". The fix is to make the class KDoc name the `run-as` route its own property KDoc already describes. Same defect family as R14, R15, R20, R25 and #46 — a claim that was true when written, next to the correction, with nothing making them agree. ### 2. Observed, unexplained: the benchmark ran and failed on CI The same KDoc says *"harmless in CI"*. It was not, twice today, on **gating** legs: | PR | leg | signature | |---|---|---| | #95 | E2E API 36 | `Abort: no output sample written in the last 25000 milliseconds` / `ExportException: Muxer error` | | #97 | E2E API 34 | same signature; failing tests named as `RealMediaBenchmark > hardwareVersusSoftwareOnRealVideo` and one more | **What is not established:** why. `hardwareVersusSoftwareOnRealVideo` has `assumeTrue(input != null)` on line 87, and an unmet assumption reports as *skipped*, not failed — so on the face of it the sample was present in `context.filesDir` on a CI emulator, which should not happen. I have not confirmed that, and I am not asserting it. Worth checking in this order: 1. Does the run actually enter the body — is the sample really there, and if so what wrote it? 2. Or is the failure arriving from somewhere else in the same leg and the benchmark names simply appearing alongside it? 3. `no output sample written in the last 25000 milliseconds` is a **starvation** signature. #93's root cause was the launcher ANRing on a loaded runner; a runner loaded enough for that would also starve a hardware transcode. These may be one problem, in which case #96 fixed the accessibility symptom and this is the same illness elsewhere. ### Why it matters beyond tidiness A **benchmark** in a gating suite is timing-sensitive by construction, and this one's own KDoc says it is *"Not a correctness test — the assertions are deliberately loose"* and *"not part of the automated suite"*. If it can run on CI at all, the gating legs inherit the flakiness of a measurement rather than of a check. ### Done means - The class KDoc names the `run-as` route, so the two halves of the file agree. **Acceptance:** a reader following the class KDoc alone stages files the test can read. - The CI question is answered with a measurement rather than a theory — either the benchmark cannot run on CI (and it is shown), or it can (and it is excluded from the gating legs, or the KDoc stops claiming otherwise).
JMR-dev commented 2026-08-25 14:25:01 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Correcting half of this ticket. Section 2 was wrong, and I filed it wrong.

I wrote that RealMediaBenchmark "ran and failed on CI" and that "harmless in CI" was false. It is true. Measured on the same run I cited (32857067112, PR #97, E2E API 34):

RealMediaBenchmark > av1InputRoutesAccordingToDeviceDecodeSupport[test(AVD) - 14]  SKIPPED
RealMediaBenchmark > hardwareVersusSoftwareOnRealVideo[test(AVD) - 14]             SKIPPED

Tests 53/59 completed. (2 skipped) (1 failed)

The two sample-dependent benchmarks skip, exactly as their assumeTrue guards intend, and the
(2 skipped) in every green leg all session has been these two. The guard works and the KDoc's
claim is accurate.

How I got it wrong: I grepped the failed-run log for test names and read RealMediaBenchmark > hardwareVersusSoftwareOnRealVideo as a failure. It was a SKIPPED line. Section 2 even listed this
as possibility (2) — "or is the failure arriving from somewhere else in the same leg and the
benchmark names simply appearing alongside it" — and that is what happened. I should have measured
before filing rather than after.

What actually failed on that gating leg is a different test:

Media3EngineTest > transcodesH264ToH265AndReportsProgress[test(AVD) - 14]  FAILED
  java.lang.IllegalStateException: androidx.media3.transformer.ExportException: Muxer error

That is now filed separately. [AVD - 17] in the same run is the advisory API 37 job failing its
three marked tests by design, which is the other thing my grep conflated.

What stands

Section 1 is unaffected and still worth fixing. The class KDoc tells you to adb push to
/sdcard/Android/data/.../files/, while the samples property twelve lines below reads
context.filesDir and its own KDoc explains that pushing to the external dir yields EACCES
presenting as an unparseable input. Those are different directories and the file contradicts itself.
That is verifiable by reading, needed no CI evidence, and is the reason this ticket should stay open.

Retitling accordingly. The "not harmless in CI" half is withdrawn.

**Correcting half of this ticket. Section 2 was wrong, and I filed it wrong.** I wrote that `RealMediaBenchmark` "ran and failed on CI" and that *"harmless in CI"* was false. **It is true.** Measured on the same run I cited (32857067112, PR #97, E2E API 34): ``` RealMediaBenchmark > av1InputRoutesAccordingToDeviceDecodeSupport[test(AVD) - 14] SKIPPED RealMediaBenchmark > hardwareVersusSoftwareOnRealVideo[test(AVD) - 14] SKIPPED Tests 53/59 completed. (2 skipped) (1 failed) ``` The two sample-dependent benchmarks skip, exactly as their `assumeTrue` guards intend, and the `(2 skipped)` in every green leg all session has been these two. **The guard works and the KDoc's claim is accurate.** **How I got it wrong:** I grepped the failed-run log for test names and read `RealMediaBenchmark > hardwareVersusSoftwareOnRealVideo` as a failure. It was a `SKIPPED` line. Section 2 even listed this as possibility (2) — *"or is the failure arriving from somewhere else in the same leg and the benchmark names simply appearing alongside it"* — and that is what happened. I should have measured before filing rather than after. **What actually failed** on that gating leg is a different test: ``` Media3EngineTest > transcodesH264ToH265AndReportsProgress[test(AVD) - 14] FAILED java.lang.IllegalStateException: androidx.media3.transformer.ExportException: Muxer error ``` That is now filed separately. `[AVD - 17]` in the same run is the advisory API 37 job failing its three marked tests by design, which is the other thing my grep conflated. ### What stands **Section 1 is unaffected and still worth fixing.** The class KDoc tells you to `adb push` to `/sdcard/Android/data/.../files/`, while the `samples` property twelve lines below reads `context.filesDir` and its own KDoc explains that pushing to the external dir yields EACCES presenting as an unparseable input. Those are different directories and the file contradicts itself. That is verifiable by reading, needed no CI evidence, and is the reason this ticket should stay open. **Retitling accordingly.** The "not harmless in CI" half is withdrawn.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
above-cut
Worked autonomously overnight: local, JVM-verifiable, no product decision
 accessibility
Barrier affecting people with disabilities
 backlog
Held for manual review: product/UX call, CI/workflow, hardware, or unverifiable here
 bug
Something isn't working
 confirmed
Reviewer demonstrated the defect
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 plausible
Reviewer could not fully demonstrate it; treat as unproven
 question
Further information is requested
 sev:high
High severity
 sev:low
Low severity
 sev:medium
Medium severity
 wontfix
This will not be worked on
No labels bug confirmed sev:low
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
JMR-dev
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: JMR-dev/LibreMediaConverter#101