JMR-dev/LibreMediaConverter
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

JaCoCo counts no Robolectric test in this repo; real line coverage is 69%, not 30% #75

New Issue
Closed
opened 2026-08-24 22:21:19 +00:00 by JMR-dev · 0 comments
JMR-dev commented 2026-08-24 22:21:19 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Found while measuring what the #52 test PRs actually moved. The answer was "nothing", and that turned out to be the finding.

JaCoCo has never counted a single Robolectric test in this repo

Robolectric loads classes through its own sandbox classloader, which gives them no source
location. JaCoCo skips no-location classes unless told otherwise, and this build never tells it:
there is no isIncludeNoLocationClasses anywhere in app/build.gradle.kts.

So every Robolectric test contributes zero coverage — and Robolectric is what tests the parts of
this codebase that were hardest to reach.

Measured, on e06b082, by adding the one block and re-running

tasks.withType<Test>().configureEach {
    extensions.configure<JacocoTaskExtension> {
        isIncludeNoLocationClasses = true
        excludes = listOf("jdk.internal.*")   // required; JaCoCo crashes on these without it
    }
}
before after
LINE 29.7% (652/2194) 69.2% (1519/2194)
BRANCH 29.8% (425/1424) 53.2% (758/1424)
OutputPublisher 0.0% 97.5%
ConversionViewModel 0.0% 85.4%
MainActivityKt 6.8% 86.4%
ConverterScreenKt 6.6% 62.8%
JoinScreenKt 0.0% 9.2%

Same commit, same 335 tests, same 0 failures. Only the measurement changed.

The discriminator, so this is not a guess

Within a single file, ConverterScreenKt: describe — the one non-composable, exercised by a
plain JVM test — reported 8/8 covered. Every @Composable in the same class reported 0,
including ones whose mutations demonstrably failed the build when reverted. Across files, the split
is exactly Robolectric-vs-not: StagingSweep (pure test) 100%; OutputPublisher,
ConversionViewModel, FailureOutcome (all Robolectric) 0%.

What this invalidates

  • CLAUDE.md's coverage paragraph. "29.8% of lines (629/2113), 28.7% of branches" is an
    artifact. Its explanation is wrong too: it blames "framework-edge code the JVM cannot reach"
    (setForeground/SAF/coroutine dispatch). The JVM reaches that code fine — JaCoCo was not
    recording it.
  • The claim that coverage FELL as the suite grew from 11 to 43 test files. Of course it did: the
    new tests were disproportionately Robolectric, so each one added denominator and no numerator.
  • #52's premise. "ConverterScreenKt 0 covered / 293 missed" was never evidence the screens were
    untested — it is what JaCoCo reports for a Robolectric-tested Compose file no matter what.
  • The remaining #52 children's projections (#61-#63). The real number is already 69.2%.

Done means

  • The block above is in app/build.gradle.kts, with a comment saying why (a future reader will
    otherwise "clean up" a setting that looks like noise).
  • CLAUDE.md's coverage bullet is re-measured and its explanation corrected.
  • A comment on #52 recording that its headline measurement was an artifact, so nobody re-derives
    the same wrong conclusion from the archived issue.
  • Mutation: remove the block, re-run :app:jacocoTestReport, and confirm the total collapses
    back toward 30%. A measurement fix that cannot be shown to move the measurement is not a fix.

Note on severity

Nothing ships differently because of this. It is filed high because it has already produced false
statements in CLAUDE.md, a false premise in #52, and a wrong projection in the #52 decomposition
plan — and because a coverage floor set against 29.7% would have been set against a number that was
never real.

_Found while measuring what the #52 test PRs actually moved. The answer was "nothing", and that turned out to be the finding._ ### JaCoCo has never counted a single Robolectric test in this repo Robolectric loads classes through its own sandbox classloader, which gives them **no source location**. JaCoCo skips no-location classes unless told otherwise, and this build never tells it: there is no `isIncludeNoLocationClasses` anywhere in `app/build.gradle.kts`. So every Robolectric test contributes **zero** coverage — and Robolectric is what tests the parts of this codebase that were hardest to reach. ### Measured, on `e06b082`, by adding the one block and re-running ```kotlin tasks.withType<Test>().configureEach { extensions.configure<JacocoTaskExtension> { isIncludeNoLocationClasses = true excludes = listOf("jdk.internal.*") // required; JaCoCo crashes on these without it } } ``` | | before | after | |---|---|---| | **LINE** | 29.7% (652/2194) | **69.2% (1519/2194)** | | **BRANCH** | 29.8% (425/1424) | **53.2% (758/1424)** | | `OutputPublisher` | 0.0% | **97.5%** | | `ConversionViewModel` | 0.0% | **85.4%** | | `MainActivityKt` | 6.8% | **86.4%** | | `ConverterScreenKt` | 6.6% | **62.8%** | | `JoinScreenKt` | 0.0% | 9.2% | Same commit, same 335 tests, same 0 failures. **Only the measurement changed.** ### The discriminator, so this is not a guess Within a single file, `ConverterScreenKt`: `describe` — the one **non**-composable, exercised by a plain JVM test — reported **8/8 covered**. Every `@Composable` in the same class reported **0**, including ones whose mutations demonstrably failed the build when reverted. Across files, the split is exactly Robolectric-vs-not: `StagingSweep` (pure test) 100%; `OutputPublisher`, `ConversionViewModel`, `FailureOutcome` (all Robolectric) 0%. ### What this invalidates - **`CLAUDE.md`'s coverage paragraph.** "29.8% of lines (629/2113), 28.7% of branches" is an artifact. Its *explanation* is wrong too: it blames "framework-edge code the JVM cannot reach" (`setForeground`/SAF/coroutine dispatch). The JVM reaches that code fine — JaCoCo was not recording it. - **The claim that coverage FELL as the suite grew** from 11 to 43 test files. Of course it did: the new tests were disproportionately Robolectric, so each one added denominator and no numerator. - **#52's premise.** "ConverterScreenKt 0 covered / 293 missed" was never evidence the screens were untested — it is what JaCoCo reports for a Robolectric-tested Compose file no matter what. - **The remaining #52 children's projections** (#61-#63). The real number is already 69.2%. ### Done means - The block above is in `app/build.gradle.kts`, with a comment saying why (a future reader will otherwise "clean up" a setting that looks like noise). - `CLAUDE.md`'s coverage bullet is re-measured and its explanation corrected. - A comment on #52 recording that its headline measurement was an artifact, so nobody re-derives the same wrong conclusion from the archived issue. - **Mutation:** remove the block, re-run `:app:jacocoTestReport`, and confirm the total collapses back toward 30%. A measurement fix that cannot be shown to move the measurement is not a fix. ### Note on severity Nothing ships differently because of this. It is filed high because it has already produced false statements in `CLAUDE.md`, a false premise in #52, and a wrong projection in the #52 decomposition plan — and because a coverage floor set against 29.7% would have been set against a number that was never real.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
above-cut
Worked autonomously overnight: local, JVM-verifiable, no product decision
 accessibility
Barrier affecting people with disabilities
 backlog
Held for manual review: product/UX call, CI/workflow, hardware, or unverifiable here
 bug
Something isn't working
 confirmed
Reviewer demonstrated the defect
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 plausible
Reviewer could not fully demonstrate it; treat as unproven
 question
Further information is requested
 sev:high
High severity
 sev:low
Low severity
 sev:medium
Medium severity
 wontfix
This will not be worked on
No labels bug confirmed sev:high
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
JMR-dev
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: JMR-dev/LibreMediaConverter#75