ci(e2e): harden android-emulator-runner emulator/system-image install against corrupt-zip (extend #389) #443

Closed
opened 2026-07-08 15:17:13 +00:00 by JMR-dev · 0 comments
JMR-dev commented 2026-07-08 15:17:13 +00:00 (Migrated from github.com)

Problem (observed 2026-07-08): the E2E (33) matrix leg deadlocked the Mergify merge queue for ~2 hours. Root cause: the reactivecircus android-emulator-runner "Create AVD and generate snapshot" step failed with Error on ZipFile unknown archive while "preparing SDK package Android Emulator" — a corrupt emulator SDK package. Because that step fails, no avd-33 snapshot cache is ever saved, so every subsequent run re-installs the emulator from scratch and hits the same corrupt zip again (deterministic, not a flake). API 29/30/32/34/35/36 restore their cached AVDs and skip the install, so only 33 dies — and it takes the whole queue with it.

Gap: #389 added corrupt-zip verify → reject → re-download hardening, but ONLY for the Install SDK platform and build-tools step (setup_android_sdk.py). The emulator + system-image install performed by android-emulator-runner is unguarded, so a corrupt emulator/system-image zip fails hard.

Fix options:

  1. Pre-install emulator + system-images;android-<api>;google_apis;x86_64 via the #389-hardened setup_android_sdk.py before the android-emulator-runner step (so the corrupt-zip verify/retry covers them), and point the runner at the already-installed SDK.
  2. Or wrap the emulator/system-image install with the same SHA-verify → purge → re-download loop.
  3. Add an AVD-create failure fallback that purges ~/.android + the emulator package and retries once (mirrors the boot-race retry).

Immediate unblock when it recurs: delete the android-sdk-v1-* (and any stale avd-*) Actions caches so the next run re-downloads clean.

Priority: it can deadlock the merge queue, so P2.

**Problem (observed 2026-07-08):** the `E2E (33)` matrix leg deadlocked the Mergify merge queue for ~2 hours. Root cause: the reactivecircus `android-emulator-runner` "Create AVD and generate snapshot" step failed with `Error on ZipFile unknown archive` while *"preparing SDK package Android Emulator"* — a **corrupt emulator SDK package**. Because that step fails, no `avd-33` snapshot cache is ever saved, so every subsequent run re-installs the emulator from scratch and hits the same corrupt zip again (deterministic, not a flake). API 29/30/32/34/35/36 restore their cached AVDs and skip the install, so only 33 dies — and it takes the whole queue with it. **Gap:** #389 added corrupt-zip verify → reject → re-download hardening, but ONLY for the `Install SDK platform and build-tools` step (`setup_android_sdk.py`). The **emulator + system-image install performed by `android-emulator-runner` is unguarded**, so a corrupt emulator/system-image zip fails hard. **Fix options:** 1. Pre-install `emulator` + `system-images;android-<api>;google_apis;x86_64` via the #389-hardened `setup_android_sdk.py` *before* the `android-emulator-runner` step (so the corrupt-zip verify/retry covers them), and point the runner at the already-installed SDK. 2. Or wrap the emulator/system-image install with the same SHA-verify → purge → re-download loop. 3. Add an AVD-create failure fallback that purges `~/.android` + the emulator package and retries once (mirrors the boot-race retry). **Immediate unblock when it recurs:** delete the `android-sdk-v1-*` (and any stale `avd-*`) Actions caches so the next run re-downloads clean. Priority: it can **deadlock the merge queue**, so P2.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: JMR-dev/LibreMail#443