#!/usr/bin/env python3 # SPDX-License-Identifier: GPL-3.0-or-later """local_instrumented.py -- reliable LOCAL instrumented / E2E test runner for LibreMail. Usage: local_instrumented.py [,,...] Example: python .claude/skills/preflight/local_instrumented.py \ org.libremail.ui.compose.ComposeScreenE2ETest python .claude/skills/preflight/local_instrumented.py \ org.libremail.ui.compose.ComposeScreenE2ETest,org.libremail.ui.compose.RecipientChipTest Cross-platform (Windows / Linux / macOS), pure standard library. Companion to ``api37_e2e.py``; ported from the original ``local_instrumented.sh`` (issue #281) so the helper runs the same on the Windows primary dev box and on *nix -- no Git Bash, no ``jq``, no ``taskkill`` vs ``kill`` portability gaps. WHY THIS SCRIPT EXISTS (issue #269) ------------------------------------ On this machine (Windows + the AEHD 2.2 hypervisor) the Gradle Managed Device (GMD) instrumented tasks -- ``apiXXDebugAndroidTest`` -- FAIL during setup. GMD tries to save/load an AVD *snapshot* and AEHD 2.2 cannot complete it: AvdSnapshotHandler$EmulatorSnapshotCannotCreatedException: Snapshot creation timed out GMD retries the snapshot ~5x, rebooting the AVD each time -- that endless reboot is the "cycling" that eats hours. The emulator ITSELF is healthy (8 GB RAM, sys.boot_completed=1, shell-responsive); only GMD's snapshot step is broken. So every LOCAL GMD task is affected: the coverage lanes and the /preflight api35/api36 steps. CI is unaffected -- it uses reactivecircus/android-emulator-runner + ``connectedDebugAndroidTest``, never GMD. THE RELIABLE LOCAL PATH (this script): Cold-boot ONE emulator by hand with ``-no-snapshot`` (no GMD, no snapshot machinery), then run ``:app:connectedDebugAndroidTest`` -- the exact technique CI and ``api37_e2e.py`` already use. We reuse a GMD-provisioned AVD by name so we don't re-download a system image; GMD re-provisions its own copy on its next run, so the ``-wipe-data`` cold boot here does not disturb it. KEEP RUNS TARGETED -- THE ~114-TEST MID-SUITE WEDGE --------------------------------------------------- Running the WHOLE instrumented suite (~114 tests) via ``connectedDebugAndroidTest`` on this box tends to wedge partway through -- the emulator stops making progress mid-run. Small, targeted class sets do NOT hit that wedge. That is why this helper takes an explicit ``[,...]`` argument and filters the run with ``-Pandroid.testInstrumentationRunnerArguments.class=...`` instead of running everything. Run the class(es) you actually changed; do not use this to run the full suite (that is CI's / preflight's job across the API matrix). FREEZE / HYGIENE RATIONALE -- WHY THE ORPHAN-KILL + TEARDOWN VERIFY ARE MANDATORY -------------------------------------------------------------------------------- A hung ``adb emu kill`` (or an interrupted run) leaves a detached qemu VM process behind (``qemu-system-x86_64-headless.exe`` on Windows; a ``qemu-system-*`` process on *nix). These orphans do not show up in ``adb devices``, they keep holding the hypervisor + RAM, and accumulated orphans have FROZEN this machine outright. So this script: * PREAMBLE -- force-kills any pre-existing qemu/emulator processes and resets the adb server BEFORE booting, so we always start from a clean slate. * TEARDOWN -- ``adb emu kill``, kill the launcher we spawned, then re-check for ANY surviving emulator/qemu process and force-kill it (the ``-no-window`` emulator can leave a sibling ``emulator.exe`` that briefly outlives the qemu VM). Teardown runs even on Ctrl-C / error / SIGTERM (try/finally + atexit + SIGINT/SIGTERM handlers) and is idempotent. * VERIFY -- if a qemu process is STILL alive after the force-kill, the script exits non-zero (code 3) so the leak is never silently ignored. Never leave an emulator running after this script; if it exits 3, hunt the zombie down by hand (Windows: ``tasklist | findstr qemu`` then ``taskkill /F /IM qemu-system-x86_64-headless.exe``; *nix: ``pgrep -fa qemu-system`` then ``pkill -f qemu-system``). CROSS-PLATFORM PROCESS KILL --------------------------- Listing and force-killing the emulator/qemu processes is abstracted per-OS (see ``list_procs`` / ``force_kill``): Windows uses ``tasklist`` + ``taskkill /F /IM ``; *nix uses ``ps ax`` + ``pkill -f qemu-system`` (alongside the graceful ``adb emu kill``). The qemu VM is the freeze-causing orphan on every platform. EXIT CODES (preserved from local_instrumented.sh) 0 tests passed 2 usage / precondition failure 3 a qemu zombie survived teardown -- clean it up by hand before the next run 4 emulator never reached sys.boot_completed connectedDebugAndroidTest's own non-zero exit code (test failures) REQUIREMENTS * Android SDK ``emulator`` + ``adb`` on PATH. * A JDK 17-21 for the Gradle daemon -- AGP 9.2 fails on JDK 25+. This script pins JAVA_HOME to a known JDK 21 (override with LOCAL_INSTRUMENTED_JDK) because the ambient JAVA_HOME on the primary box points at JDK 25. * A free hardware hypervisor (VT-x/WHPX/AEHD/KVM/HVF). Shut down VirtualBox / other VMs first or the AVD hangs at 0% CPU and never reaches sys.boot_completed. Overridable via environment (defaults target the primary Windows dev box): LOCAL_INSTRUMENTED_AVD AVD name to boot (dev36_google_apis_x86_64_Pixel_2) ANDROID_AVD_HOME AVD home dir (C:/Users/jasonross/.android/avd/gradle-managed) LOCAL_INSTRUMENTED_JDK JDK 17-21 home (Eclipse Adoptium jdk-21.0.11.10-hotspot) LOCAL_INSTRUMENTED_SERIAL adb serial (emulator-5554) LOCAL_INSTRUMENTED_BOOT_TIMEOUT boot wait seconds (300) """ from __future__ import annotations import argparse import atexit import os import shutil import signal import subprocess import sys import tempfile import time from pathlib import Path IS_WINDOWS = os.name == "nt" # ---- configuration (env-overridable; defaults are correct for the primary dev box) ------ AVD_NAME = os.environ.get("LOCAL_INSTRUMENTED_AVD", "dev36_google_apis_x86_64_Pixel_2") AVD_HOME = os.environ.get("ANDROID_AVD_HOME", "C:/Users/jasonross/.android/avd/gradle-managed") JDK_HOME = os.environ.get( "LOCAL_INSTRUMENTED_JDK", "C:/Program Files/Eclipse Adoptium/jdk-21.0.11.10-hotspot" ) SERIAL = os.environ.get("LOCAL_INSTRUMENTED_SERIAL", "emulator-5554") BOOT_TIMEOUT = int(os.environ.get("LOCAL_INSTRUMENTED_BOOT_TIMEOUT", "300")) # Windows qemu/emulator image names (see FREEZE / HYGIENE above). The ``-headless`` variant is # what a ``-no-window`` emulator launches; the plain qemu name is swept too, belt-and-suspenders. QEMU_IMAGE = "qemu-system-x86_64-headless.exe" QEMU_IMAGE_ALT = "qemu-system-x86_64.exe" EMULATOR_IMAGE = "emulator.exe" REPO_ROOT = Path(__file__).resolve().parents[3] # .claude/skills/preflight -> repo root GRADLEW = REPO_ROOT / ("gradlew.bat" if IS_WINDOWS else "gradlew") EMU_LOG = Path(tempfile.gettempdir()) / "libremail-local-instrumented-emulator.log" class _RunState: """Mutable run state shared by main(), teardown(), the atexit hook and the signal handlers -- mirrors the bash globals EMU_PID / TEST_EXIT / ZOMBIE / TEARDOWN_DONE.""" def __init__(self) -> None: self.proc: subprocess.Popen | None = None self.test_exit = 1 self.zombie = False self.teardown_done = False _STATE = _RunState() def log(msg: str) -> None: print(f"\n=== {msg} ===") def warn(msg: str) -> None: print(f"WARNING: {msg}", file=sys.stderr) def die(msg: str) -> None: """Print an error and exit 2 (usage / precondition failure). Called before the teardown backstops are armed, so nothing has booted and there is nothing to tear down.""" print(f"ERROR: {msg}", file=sys.stderr) sys.exit(2) def cmd(tool: str, *args: str) -> list[str]: """Build an argv list, wrapping Windows ``.bat``/``.cmd`` launchers (e.g. gradlew.bat) through ``cmd /c`` -- matching api37_e2e.py. ``.exe`` tools pass through unchanged.""" if IS_WINDOWS and tool.lower().endswith((".bat", ".cmd")): return ["cmd", "/c", tool, *args] return [tool, *args] def _run_quiet(argv: list[str]) -> None: """Run a command, discarding output and swallowing any error -- teardown/kill helpers must always make progress.""" try: subprocess.run(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False) except (OSError, subprocess.SubprocessError): pass def _run_capture(argv: list[str]) -> str: try: return subprocess.run(argv, capture_output=True, text=True, check=False).stdout or "" except (OSError, subprocess.SubprocessError): return "" def list_procs(*needles: str) -> str: """Return the lines of currently-running processes whose name/command line contains any of ``needles`` (case-insensitive); empty string if none. Cross-platform stand-in for the .sh's ``tasklist | grep``: ``tasklist`` on Windows, ``ps ax`` on *nix.""" out = _run_capture(["tasklist"] if IS_WINDOWS else ["ps", "ax"]) lowered = [n.lower() for n in needles] return "\n".join(ln for ln in out.splitlines() if any(n in ln.lower() for n in lowered)) def list_qemu() -> str: return list_procs("qemu") def list_emu_procs() -> str: return list_procs("qemu", "emulator") def force_kill(win_images: list[str], nix_patterns: list[str]) -> None: """Best-effort force-kill. Windows: ``taskkill /F /IM ...``. *nix: ``pkill -f `` per pattern. Never raises -- teardown must always make progress.""" if IS_WINDOWS: argv = ["taskkill", "/F"] for image in win_images: argv += ["/IM", image] _run_quiet(argv) else: for pattern in nix_patterns: _run_quiet(["pkill", "-f", pattern]) def tail(path: Path, lines: int = 40) -> None: try: with open(path, "r", errors="replace") as handle: content = handle.readlines()[-lines:] print("".join(content), file=sys.stderr, end="") except OSError: pass def teardown() -> None: """Kill the emulator and verify no orphaned emulator/qemu process remains. Idempotent -- safe to call from the finally block, the atexit hook and the signal handlers (mirrors the .sh TEARDOWN_DONE guard). Sets _STATE.zombie if a *qemu* process survives the force-kill -- the machine-freezing case (exit 3).""" if _STATE.teardown_done: return _STATE.teardown_done = True log("Teardown: killing emulator and verifying no orphaned emulator/qemu remains") adb = shutil.which("adb") if adb: _run_quiet(cmd(adb, "-s", SERIAL, "emu", "kill")) time.sleep(2) # Belt-and-suspenders: kill the emulator launcher process we started, if still alive. proc = _STATE.proc if proc is not None and proc.poll() is None: proc.terminate() try: proc.wait(timeout=1) except subprocess.TimeoutExpired: proc.kill() # Reap any lingering emulator/qemu process, then verify. Unlike the original .sh -- which # swept qemu ONLY -- we also force-kill the emulator *launcher* image: on Windows the # ``-no-window`` emulator spawns a sibling ``emulator.exe`` that is NOT the Popen child we # tracked and outlives both it and the qemu VM by a few seconds, so a qemu-only sweep # returns while it is still shutting down -- an orphan the freeze-safety rule forbids. So we # trigger on any emulator-or-qemu survivor and taskkill the launcher too. if list_emu_procs(): warn("emulator/qemu still present after 'adb emu kill'; force-killing:") print(list_emu_procs(), file=sys.stderr) force_kill([QEMU_IMAGE, QEMU_IMAGE_ALT, EMULATOR_IMAGE], ["qemu-system"]) time.sleep(2) # A surviving QEMU is the machine-freezing zombie (exit 3); a stray launcher is not. remaining = list_qemu() if remaining: warn("qemu ZOMBIE survived teardown -- kill it by hand or the machine may freeze:") print(remaining, file=sys.stderr) _STATE.zombie = True if adb: _run_quiet(cmd(adb, "kill-server")) def orphan_kill_preamble(adb: str) -> None: """Force-kill any pre-existing qemu/emulator processes and reset the adb server, so we always cold-boot from a clean slate.""" log("Orphan-kill preamble: ensuring a clean slate before boot") existing = list_emu_procs() if existing: warn("Pre-existing emulator/qemu processes found -- force-killing them first:") print(existing, file=sys.stderr) force_kill([QEMU_IMAGE, EMULATOR_IMAGE], ["qemu-system"]) time.sleep(2) else: print("No pre-existing qemu/emulator processes.") _run_quiet(cmd(adb, "kill-server")) _run_quiet(cmd(adb, "start-server")) def start_emulator(emulator: str) -> subprocess.Popen: """Cold-boot ONE emulator by hand (no GMD, no snapshot), logging to EMU_LOG. Records the launcher process in _STATE so teardown can reap it even if we are interrupted next.""" log(f"Cold-booting @{AVD_NAME} (no GMD, no snapshot); log -> {EMU_LOG}") flags = [ f"@{AVD_NAME}", "-no-window", "-no-snapshot", "-no-boot-anim", "-no-audio", "-gpu", "auto-no-window", "-cores", "8", "-wipe-data", ] logf = open(EMU_LOG, "wb") # noqa: SIM115 - handed to the child; parent copy closed below try: proc = subprocess.Popen(cmd(emulator, *flags), stdout=logf, stderr=subprocess.STDOUT) finally: logf.close() # the child inherited its own fd; the parent's copy is no longer needed _STATE.proc = proc print(f"emulator launcher pid={proc.pid}") return proc def wait_for_boot(adb: str, proc: subprocess.Popen, timeout: int) -> bool: """Poll ``adb get-state`` + ``getprop sys.boot_completed`` until the emulator is up, or the launcher dies, or ``timeout`` seconds elapse. Mirrors the .sh boot loop.""" print(f"Waiting up to {timeout}s for sys.boot_completed on {SERIAL}...") deadline = time.monotonic() + timeout while time.monotonic() < deadline: if proc.poll() is not None: warn("emulator process exited during boot; last log lines:") tail(EMU_LOG, 40) return False state = _run_capture(cmd(adb, "-s", SERIAL, "get-state")).strip() if state == "device": booted = _run_capture( cmd(adb, "-s", SERIAL, "shell", "getprop", "sys.boot_completed") ).strip() if booted == "1": return True time.sleep(3) return False def prepare_focus(adb: str) -> None: # Force the emulator to grant the app window focus BEFORE the suite, then gate on it -- the # SAME shared helper CI's e2e / e2e-preview jobs invoke (issue #468), so local preflight # exercises the identical fix. It wakes the display, dismisses + disables the keyguard, keeps # the screen on, disables animations, and waits for a focused window. Best-effort: fall back # to the legacy `input keyevent 82` nudge if the shared helper is somehow missing. gate = REPO_ROOT / ".github" / "scripts" / "emulator_focus_gate.py" if gate.is_file(): subprocess.run([sys.executable, str(gate), "--adb", adb, "--serial", SERIAL], check=False) else: _run_quiet(cmd(adb, "-s", SERIAL, "shell", "input", "keyevent", "82")) def run_tests(test_classes: str) -> int: """Run :app:connectedDebugAndroidTest filtered to ``test_classes`` from the repo root (JAVA_HOME / ANDROID_AVD_HOME are already in the environment).""" log(f"Running :app:connectedDebugAndroidTest for: {test_classes}") print(f"JAVA_HOME={os.environ.get('JAVA_HOME', '')}") return subprocess.run( cmd( str(GRADLEW), ":app:connectedDebugAndroidTest", f"-Pandroid.testInstrumentationRunnerArguments.class={test_classes}", "--stacktrace", ), cwd=str(REPO_ROOT), check=False, ).returncode def main() -> int: parser = argparse.ArgumentParser( prog="local_instrumented.py", formatter_class=argparse.RawDescriptionHelpFormatter, description=( "Cold-boot ONE emulator (no GMD, no snapshot) and run " ":app:connectedDebugAndroidTest filtered to the given instrumented test class(es)." ), epilog=( "Keep the class set small and targeted -- the full ~114-test suite tends to wedge\n" "mid-run on this box (see the module docstring). The full matrix is CI's job.\n" "Example:\n" " python .claude/skills/preflight/local_instrumented.py \\\n" " org.libremail.ui.compose.ComposeScreenE2ETest,org.libremail.ui.compose.RecipientChipTest" ), ) parser.add_argument( "test_classes", metavar="TEST_CLASSES", help=( "Comma-separated fully-qualified instrumented test class(es), no spaces " "(e.g. org.libremail.a.FooTest,org.libremail.b.BarTest)." ), ) args = parser.parse_args() # ---- preconditions (before arming teardown; nothing has booted yet) ------------------ emulator = shutil.which("emulator") adb = shutil.which("adb") if not emulator: die("emulator not on PATH (install Android SDK emulator).") if not adb: die("adb not on PATH (install Android SDK platform-tools).") kill_tool = "taskkill" if IS_WINDOWS else "pkill" if not shutil.which(kill_tool): die(f"{kill_tool} not found -- required to force-kill orphaned emulator/qemu processes.") if not GRADLEW.is_file(): die(f"gradlew not found at {GRADLEW}.") if not os.path.isdir(JDK_HOME): die(f"JDK 17-21 not found at '{JDK_HOME}'. Set LOCAL_INSTRUMENTED_JDK.") if not os.path.isfile(os.path.join(AVD_HOME, AVD_NAME + ".ini")): die( f"AVD '{AVD_NAME}' not found under '{AVD_HOME}'. " "Set LOCAL_INSTRUMENTED_AVD / ANDROID_AVD_HOME. " "(GMD AVDs are created by any local apiXXDebugAndroidTest run.)" ) # Move gradlew's working dir to this tree's repo root (below) and pin JAVA_HOME/AVD home, # exactly like the .sh -- the ambient JAVA_HOME on this box points at JDK 25 (AGP-incompatible). os.environ["JAVA_HOME"] = JDK_HOME os.environ["ANDROID_AVD_HOME"] = AVD_HOME # ---- arm teardown backstops BEFORE touching the emulator ----------------------------- # try/finally is the primary path; atexit covers sys.exit()/unhandled-exception exits; the # signal handlers make SIGINT/SIGTERM tear down too (Python does not raise on SIGTERM by # default). teardown() is idempotent, so firing from several paths is safe (mirrors the # .sh's ``trap teardown EXIT INT TERM`` + TEARDOWN_DONE guard). atexit.register(teardown) def _signal_teardown(signum: int, _frame: object) -> None: teardown() sys.exit(128 + signum) signal.signal(signal.SIGINT, _signal_teardown) if hasattr(signal, "SIGTERM"): signal.signal(signal.SIGTERM, _signal_teardown) boot_failed = False try: orphan_kill_preamble(adb) proc = start_emulator(emulator) if wait_for_boot(adb, proc, BOOT_TIMEOUT): print("Emulator booted.") prepare_focus(adb) _STATE.test_exit = run_tests(args.test_classes) else: warn(f"Emulator did not reach sys.boot_completed within {BOOT_TIMEOUT}s.") tail(EMU_LOG, 40) boot_failed = True finally: teardown() if boot_failed: return 4 if _STATE.zombie: warn( "Exiting 3: a qemu zombie was left behind (see above) -- " "clean it up before the next run." ) return 3 if _STATE.test_exit != 0: warn( f"connectedDebugAndroidTest failed (exit {_STATE.test_exit}). " "Report: app/build/reports/androidTests/connected/" ) return _STATE.test_exit log(f"PASS -- instrumented tests green for: {args.test_classes}") return 0 if __name__ == "__main__": sys.exit(main())