The rotation test hangs on API 33 and 34, wedging the leg for 20 minutes #122
Closed
opened 2026-08-26 03:26:45 +00:00 by JMR-dev
·
7 comments
No Branch/Tag Specified
main
fix/102-picker-back-press-overshoot
fix/268-saf-picker-determinism
feat/ogg-vorbis-libvorbis
feat/expedited-conversion-work
fix/gate-cache-in-worktrees
chore/gate-runs-shellcheck
test/publish-delete-arm-real-provider
docs/e8-instrumented-coverage
docs/api37-carrier-count-drift
docs/e7-second-constraint
test/publish-to-a-real-saf-destination
fix/api37-report-match-line
fix/api37-task-snapshot-crash
test/join-failure-message-on-device
docs/e2e-read-findings-e7
test/cancelling-a-running-export
test/reattach-to-a-running-job
test/content-uri-reaches-ffmpeg
fix/launcher-wiring-waits-for-the-pick
fix/cancel-tests-need-a-slower-encode
test/cancelling-a-running-join
test/cancelling-a-running-session
test/notification-cancel-action
test/ffmpeg-progress-is-observed
test/fallback-asserts-the-path
test/flac-and-opus-assert-their-format
docs/e2e-read-findings
docs/wave4-coverage-numbers
fix/injectable-startup-sweep-scope
test/session-outcome-seam
test/launcher-callback-identity
test/theme-follows-system-dark
test/audio-drop-arm
fix/rotation-waits-for-recreation
fix/convert-guards-on-ready
test/retry-save-mime
test/hardware-progress-reaches-workmanager
test/ffprobe-mapping-seam
test/device-codec-enumeration-seam
test/unknown-container-row
test/null-message-fallbacks
test/cancel-reaches-workmanager
docs/coverage-wave3-recovery
test/concat-engine-seam
docs/coverage-wave3
test/mediaprobe-merge-seam
test/adaptive-shell-wiring
test/aac-audio-args
test/notification-progress-text
test/media3-muxer-guard
test/hardware-fallback-and-cancellation
test/unprobeable-join-clip
test/one-branch-outcomes
test/foreground-type-regimes
fix/bound-wedge-diagnostics
docs/coverage-wave2
test/screen-wiring
test/viewmodel-setters
test/join-state-mapping
test/conversion-state-mapping
test/dedupe-user-messages
fix/restore-stack-merges
test/refused-jobs
test/concatworker-failure-arms
test/container-capabilities-audio
test/readspec-enum-fallbacks
test/outputpublisher-seams
test/mediaprobe-track-seam
test/outputpublisher-partial-branches
test/fake-provider-scaffolding
docs/coverage-read-findings
chore/gitignore-kotlin
test/bound-the-hangs
docs/coverage-remeasure
ci/baseline-counter-precision
fix/invalid-suggestion-chip
ci/wedged-leg-report
fix/reattachment-overwrites-pick
test/theme-live-branches
fix/failed-save-retry
fix/empty-composition-crash
ci/advisory-failure-report
docs/seven-run-counts
test/release-permission-guard
ci/build-workflow-permissions
docs/api37-point-release
docs/benchmark-populate-path
fix/dead-assertion-probe-test
ci/actionlint
test/device-codecs-encode-consequence
fix/sdkmanager-pipefail
docs/readme-restart-claim
fix/saf-picker-root-discovery
fix/probe-dispatcher-seam
test/media3engine-mime-tables
test/mediaprobe-pure-helpers
fix/codec-vocabulary-drift
docs/robolectric-rationale-correction
docs/api37-advisory-counts
test/r38-8-saf-e2e
test/r38-7-join-states
test/r38-6-conversion-states
test/r38-5-state-seam
fix/jacoco-robolectric-coverage
docs/instrumented-tests-correction
test/r38-2-filecard
test/r38-4-advanced-picker
test/r38-3-pickers
tools/file-issue-script
tools/api-37-emulator
fix/review-app-gaps
docs/review-corrections
No results found.
Labels
Clear labels
above-cut
accessibility
backlog
bug
confirmed
documentation
duplicate
enhancement
good first issue
help wanted
invalid
plausible
question
sev:high
sev:low
sev:medium
wontfix
Worked autonomously overnight: local, JVM-verifiable, no product decision
Barrier affecting people with disabilities
Held for manual review: product/UX call, CI/workflow, hardware, or unverifiable here
Something isn't working
Reviewer demonstrated the defect
Improvements or additions to documentation
This issue or pull request already exists
New feature or request
Good for newcomers
Extra attention is needed
This doesn't seem right
Reviewer could not fully demonstrate it; treat as unproven
Further information is requested
High severity
Low severity
Medium severity
This will not be worked on
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: JMR-dev/LibreMediaConverter#122
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
SafPickerRoundTripTest.thePickedInputSurvivesARealRotationcan hang forever onAPI 33 and 34. The leg then dies to
WEDGE_TIMEOUTafter 1200 s and the PR goesred for something it did not cause.
Two occurrences, both on PRs that cannot plausibly be responsible, and both
end at exactly the same place in the TestRunner log:
3292154786102:13:50.515 started: thePickedInputSurvivesARealRotation3292476672003:07:07.917 started: thePickedInputSurvivesARealRotationgrep -c "finished: thePickedInputSurvivesARealRotation"is 0 in both wedgediagnostics artifacts. It is the last test started in both, and nothing follows.
The framework is not down. In both artifacts every binder service the wedge
probe checks is present:
So this is not #108 and not the surfaceflinger/gralloc family — nothing has
crashed. The test simply never returns. The two differ in whether the
instrumentation process survives: API 34 shows
pids -- app: 6121 test: 6616,API 33 shows
test: <none>, and on API 33 the SIGQUIT dump could not be taken(
kill: 6473: Operation not permitted), so there is no thread dump for thehang yet.
Why this test specifically. It is the only test in the suite that rotates the
display, and it already carries
@FailsOnEmulatorApi37because a rotation"rebuilds every surface on screen at once, which the mapper does not survive" on
the 37 images. That marker covers a crash on 37. What is happening on 33 and 34
is a hang, which no marker covers and which costs 20 minutes a time.
Do not fix this by marking it @FailsOnEmulatorApi37 for more API levels. That
annotation means "this image cannot run this test", and the whole point of the
rotation test is that API 33-36 answer the question API 37 cannot. Silencing it
across the matrix would leave nothing asserting that a picked file survives a real
rotation.
Suggested first step: get a thread dump. The wedge probe already attempts
SIGQUIT and was refused on API 33; making that work (or using
am dumpheap/debuggerdagainst the instrumentation pid) would say where it is blocked —almost certainly a
UiAutomatorwait or an idling-resource wait that has notimeout. A per-test timeout would convert a 20-minute wedge into a normal failure
with a stack, which is strictly better even before the root cause is known.
Related but distinct:
completed cleanly: yes.Two measurements that constrain the fix
1. Normal duration: 9.572 s. From the JUnit XML of a passing API 33 leg (run
32925531744):So the wedge runs 1200 s against a 9.6 s test — roughly 125x. A per-test timeout anywhere in the
30-90 s range would keep enormous headroom over the normal case while turning a 20-minute anonymous
wedge into a fast failure with a stack. That is worth doing before the root cause is known: it
costs nothing when the test is healthy and it is the thing that would let the next occurrence
diagnose itself.
Pick the number against the slowest observed pass, not against 9.572 — this is the slowest test in
the suite and runner load varies. But there is a very wide safe band here.
2. Local reproduction is blocked right now, and not by the emulator. A VirtualBox VM is running
on this host (
VirtualBoxVM --comment Gentoo --startvm 83280c91-…, pid 709807). VirtualBox takesVT-x exclusively on Intel, so every AVD dies with
KVM: entry failed, hardware error 0x0even though/dev/kvmexists and is writable andkvm_intelis loaded. An agent hit this earlier today andindependently confirmed it on a second AVD.
That matters for whoever picks this up:
tools/local-emulator/run-e2e.shcovers API 33-36 and wouldnormally be the fastest way to reproduce a 33/34 hang, but not while that VM is up. It is the
maintainer's VM and should not be shut down to make room for a test run — either wait for it, or
iterate on CI, where the mode has appeared twice in roughly 500 gating leg-attempts. That is a slow
loop, which is the other reason the timeout-and-stack change is worth landing first.
Still fires after #127, on a diff that cannot have caused it.
Run
33033036857, PR #131 — a docs-only change adding one markdown file, no source, no build, no workflow:23m08s wall clock. Diagnostics are in the
e2e-wedge-api33artifact (49 KB).Frequency, from the last eight
status_checkruns (cancelled runs excluded):So it is intermittent, and on this sample it is API 33 only — this ticket's title names 34 as well, which nothing in this window reproduces.
The part worth keeping separate.
received: 60means all sixty tests reported before gradle hung, so the wedge is not costing test execution — it is costing the verdict, sincefailed:readsunknown. That distinction is only visible because of thewedged:row #118 added; without it this table would have readreceived: 60, completed cleanly: nowith no way to tell a hang from an abort.That has a consequence beyond this ticket, which is how I got here: #88 closed on the reasoning that the CI matrix exercises all three
ConversionForegroundTypeAPI regimes. That reasoning is intact — the tests do run — but the API 33 leg cannot be relied on to report a failure while this is live. Recorded indocs/coverage-read-findings.md(PR #131), which also notes that a@Config(sdk = 33)/@Config(sdk = 34)JVM test would pin those arms deterministically for about three lines, independent of emulator health.Retried the same leg on the same commit: it passed. That settles "intermittent" properly — the earlier comment inferred it from a cross-run sample, which cannot rule out something about the diff. This controls for the diff entirely.
Commit
2599286, jobE2E API 33, run33033036857:received: 60, failed: unknown, wedged: yes — gradle killed after 1200sexpected: 60, received: 60, failed: 0, completed cleanly: yesSame SHA, same runner image, same workflow. So the wedge is a race, not a property of any particular diff — and
received: 60on the wedged attempt says it lands after the last test reports, while gradle is shutting down.Unrelated trap found while doing this, worth recording here since this is the ticket people will land on when a leg wedges.
status_check.ymlsetsconcurrency: group: status-check-${{ github.ref }}withcancel-in-progress: true. Re-running a job on an older run for the same ref re-enters that group and cancels the newer in-flight run — in this case a full run that had just started for the branch head.gh pr checksthen reports every cancelled job asfail, which reads exactly like a build break: nine jobs "failing" in 42 seconds on a docs-only diff.So: retry a wedged leg only when nothing newer is queued for that ref, or re-run the newest run rather than the one that wedged.
API 34 wedges too. Correcting my earlier comment, which said "on this sample it is API 33 only" — that was a five-run window and it does not hold.
Run
33036060139, PR #144,E2E API 34:Identical shape to the API 33 wedge two hours earlier: all sixty tests report, then gradle does not return. So this ticket's title was right to name both levels, and the API-33-only reading was an artefact of sample size, not a narrowing of the bug.
Running tally across the PRs opened today (#131, #144–#149), counting only legs that completed or wedged:
Two wedges across roughly forty leg-runs, on two different API levels, on diffs that are docs-only or JVM-test-only. Both retried clean.
That last point is the useful one for anyone triaging: a wedged leg has always passed on a plain re-run so far, and
received: Non the wedged run says the tests themselves completed. What the wedge costs is the verdict (failed: unknown), not the coverage.The wedge guard works. What happens after it does not — and that is what turns a wedge into a cancelled job rather than a failed one.
Timeline from #151's
E2E API 37, run33071570911, job98515285726:> Task :app:connectedDebugAndroidTestwedged: yes -- gradle was killed after 1200s and never returned::warning::E2E api37 WEDGED (api37)##[error]The operation was canceled.— the job'stimeout-minutes: 60capTerminate orphan process: qemu-system-x86_64-headless,adb, twojavaSo
WEDGE_TIMEOUT=1200fired exactly as designed, the report printed, the warning printed — and then the step sat for another 36 minutes.Where it stops
.github/scripts/e2e-run.sh, last five lines:dump_diagnosticsprobes the device —adb logcat -d -b crash,adb shell, and friends. Every one of them is|| true-guarded, and the file header states the rule they exist for:|| trueguards against a non-zero exit. It does nothing about a probe that never exits at all — and these probes are aimed at the emulator that has just finished proving it stopped answering. That is the gap: the discipline is enforced for exit codes and not for time.Why it costs more than the 36 minutes
cancelled, not failed with the wedge's own status.gh pr checksrenders that as a failure with no cause, and the carefulwedged:row is buried above half an hour of silence — so the one mechanism built to explain a wedge is the thing least likely to be read.kill "$LOGCAT_PID"never runs, which is why the runner reports orphanadbandqemuat cleanup.Cheap fix
Bound the probes the same way the gradle client already is —
timeoutarounddump_diagnostics, or per probe. The value of the diagnostics is entirely in the first few seconds of output; nothing here is worth 36 minutes. That would also let lines 39-40 run, so a wedge would exit with its own status and reap its logcat.Not fixed here — filing rather than changing CI in the middle of merging a stack. Happy to do it as its own PR.
A third occurrence, and it is a different symptom of the same test — which points at a cause
PR #217, run
33698846104, API 33 gating leg. This one did not wedge:So the test ran to completion and asserted, rather than hanging. Same test, same API level as row 2
of the table above, opposite run shape.
What that rules in
The failure is on the second guard, which means the first one passed:
device.displayRotationwas already not
NATURAL_ROTATION. The display had rotated. What had not happened is the Activitybeing rebuilt —
System.identityHashCode(composeRule.activity)was unchanged.That combination is not "the emulator ignored the rotation". It is the test returning from its
barrier before the recreation it is waiting for has happened, because the barrier does not wait for
that:
waitForIdle()waits for the compose hierarchy to go idle. Immediately after a rotation request thatthe window manager has accepted but not yet delivered as a configuration change, the old Activity's
composition is idle — so
waitForIdle()returns,composeRule.activitystill resolves to the oldinstance, and the identity hash matches. Nothing in the test waits for
MainActivityto be destroyedand rebuilt.
This is a test bug, not only an image bug, and it is the more likely reading of the wedges too:
on API 33/34 the rotation is slow enough for the gap between "display rotated" and "Activity
recreated" to be observable. Land on the early side and you get this assertion; land while the
composition is being torn down and
waitForIdle()has nothing coherent to settle on, and you get the20-minute wedge in the table above. One race, two exits.
Candidate fix
Replace
waitForIdle()with a barrier that waits for the thing being asserted — poll untilSystem.identityHashCode(composeRule.activity)differs frombefore, bounded, or register anActivityLifecycleCallbacks/ActivityMonitorfor the recreation and wait on that. ThenwaitForIdle()for the composition. The two guards stay as they are; they are what caught this.Worth doing on this ticket rather than under the emulator-infra umbrella (#190): unlike the
hasReadColorBufferDmaaborts, nothing here needs the image to change.Filed while shepherding the wave-4 PRs (#206-#217). Ten of the twelve are green on every gating
leg; this is one of the two that are not.
The wedge recurred during the wave-4 merge train — on a PR that does not contain the fix
PR #213 (
test/theme-follows-system-dark, whose diff is one Robolectric test file), run34001741668, API 34:#213 sits below #219 in the merge order, so it carries the old
waitForIdle()barrier. This is thefourth occurrence of the wedge on this ticket and the first one observed while knowing what to look
for: same API level band (33/34), same 1200 s
WEDGE_TIMEOUT, no attribution in the run shapebeyond the
wedged:row that #118 added.Worth stating plainly because it is the argument for #219 rather than a restatement of the problem:
the leg cost 20 minutes and still cannot say which test hung. With the bounded barrier the same
condition fails in 15 s naming itself — measured on the local API 33 emulator in #219, where
neutering the recreation counter produced
with the run still reporting
received 60/60, completed cleanly: yes.