covenant-setup/docs/vagrant-smoke-debugging-notes.md

Vagrant Smoke Test Debugging Notes

This document records the work done while replacing the PowerShell UI with a C# presentation layer, adding guest-side diagnostics, and stabilizing the Windows Vagrant smoke test. It is intended as a reference for future installer hangs where the VM console is not visible.

Scope

The work covered these areas:

• Removed the PowerShell-hosted UI path.
• Added a C# WinForms UI process.
• Connected Rust business logic to the C# UI over Windows named pipes.
• Added guest trace collection so hangs can be diagnosed without watching the VM console.
• Rebuilt and tested the packaged installer in the Hyper-V Vagrant guest.
• Extended the smoke test to install, verify installed state, uninstall, and verify removed state.

Current Architecture

The installer is still driven by Rust. The C# process is presentation only.

• Rust business logic lives primarily in src/main.rs.
• Rust C# UI IPC lives in src/ui.rs.
• C# WinForms UI lives in ui/Covenant.Setup.Ui/Program.cs.
• build.rs publishes the C# UI as a self-contained win-x64 single-file executable.
• The Rust binary embeds the published C# UI executable with include_bytes!.
• At runtime, Rust extracts the C# UI executable to %TEMP%\covenant-setup-ui, starts it, and connects to a named pipe.
• The C# UI owns the pipe server and reads newline-delimited JSON messages.
• Rust sends messages such as init, progress, log, finish, prompt, and close.
• The C# UI writes prompt responses back as JSON.

Trace Outputs

The guest trace directory is:

C:\Users\vagrant\AppData\Local\Temp\covenant-setup-smoke\trace

The host harness pulls this into:

dist\vagrant-self-test\trace

Important trace files:

• guest-events.jsonl: host/guest harness events, scheduled task status, verification phases.
• installer-heartbeat-<pid>.jsonl: Rust process heartbeat and installer phases.
• csharp-ui-pipe-<pid>.jsonl: C# UI process and pipe receive/send events.
• interactive-context.json: context captured by the interactive wrapper.
• interactive-processes.json: relevant guest processes during wrapper diagnostics.
• interactive-windows.json: visible windows and process window titles.
• interactive-application-events.json: recent Application event log entries.
• abort-*.json: snapshots created by the abort collector.

The host harness always tries to pull the trace bundle in finally, even when the smoke test fails.

Errors Encountered

1. Host Sandbox and Vagrant Permissions

Running Vagrant and Hyper-V actions from the coding sandbox required escalation. This affected commands such as:

.\scripts\run-windows-vm-smoke.ps1 -SkipViewer -HaltAfter
vagrant status
vagrant winrm ...
vagrant upload ...

This was expected: Vagrant controls an external VM, uses WinRM, and interacts with Hyper-V.

2. Pre-main Guest Failure: Missing VCRUNTIME140.dll

The first meaningful guest diagnostics showed a Windows system error dialog:

covenant-setup-installer.exe - System Error
The code execution cannot proceed because VCRUNTIME140.dll was not found.

Evidence:

• interactive-windows.json showed a covenant-setup-installer.exe - System Error window.
• system-events.json had an Application Popup event for the missing DLL.
• There were no installer-heartbeat-*.jsonl files.
• There were no csharp-ui-pipe-*.jsonl files.

Conclusion:

The executable failed before Rust main() ran. The heartbeat and pipe logs were absent because neither Rust nor the C# UI started.

Fix:

Added .cargo/config.toml:

[target.x86_64-pc-windows-msvc]
rustflags = ["-C", "target-feature=+crt-static"]

This statically links the MSVC C runtime into the Rust executable, removing the guest dependency on VCRUNTIME140.dll.

3. Packaging Looked Successful but Produced an Unbundled EXE

Manual PowerShell invocations of the Windows-subsystem Rust executable were misleading. A direct command such as:

target\release\covenant-setup.exe --json package vm\self-test\install.toml --output dist\vagrant-self-test

could return quickly with EXIT=0 while the output file still matched the base executable size.

Reason:

The Rust binary is built as a Windows GUI subsystem executable. Direct invocation from PowerShell does not behave like a normal console command in all cases.

Fixes:

• The smoke harness invokes the packager with Start-Process -Wait -PassThru.
• The harness now validates that the packaged installer ends with the bundle magic marker COVENANT_SETUP_BUNDLE_V1.
• The embedded bundle format was changed from JSON byte arrays to an appended raw bundle with a JSON index plus raw file data. This avoids large JSON expansion of payload bytes.

4. Journal Written Beside the Smoke Installer

After the VCRUNTIME fix, the installer succeeded but the smoke verifier failed with:

Journal missing: C:\Users\vagrant\AppData\Local\CovenantSetupSelfTest\journal.json

Evidence:

The Rust heartbeat showed:

{"phase":"install_journal_written","detail":{"journal":"C:\\Users\\vagrant\\AppData\\Local\\Temp\\covenant-setup-smoke\\journal.json"}}

Cause:

The packaged install path was passing an explicit journal path next to the packaged installer. The expected product behavior is to infer the install root and write journal.json there.

Fix:

run_bundled_installer now calls:

install(&manifest_path, None, true, ui_mode, logger)

This lets build_install_runtime infer the journal path from the install root.

5. Scheduled Task Re-ran During Diagnostics

The guest harness originally registered a scheduled task with a trigger one minute in the future and also started it manually.

Failure mode:

• The manual task run completed.
• If verification or diagnostics took long enough, the scheduled trigger fired and launched a second copy.

Fix:

The trigger is now set far in the future:

New-ScheduledTaskTrigger -Once -At (Get-Date).AddDays(1)

The harness still starts the task manually with Start-ScheduledTask.

6. WinRM Error 1726 During Success Diagnostics

After the installer succeeded, the host sometimes saw:

WSMAN ERROR CODE: 1726
The WSMan provider host process did not return a proper response.

The trace showed the installer succeeded and verification reached the success diagnostics phase, but the WinRM command failed while returning.

Fix:

• The success path now writes guest-result.json immediately after verification.
• Heavy diagnostics are retained for failure paths.
• Diagnostic file writes now emit diagnostic_file_start, diagnostic_file_finish, and diagnostic_file_error markers so future diagnostic hangs show the exact capture that blocked.

7. Install-plus-uninstall Harness Hung

When uninstall testing was added, the install scheduled task exited with LastTaskResult=1. The parent loop waited until timeout because no result file was written.

Evidence:

• guest-events.jsonl showed the install scheduled task was registered and started.
• The task quickly moved to Ready with LastTaskResult=1.
• There was no interactive_installer_start.
• There was no Rust heartbeat.
• There was no C# pipe log.

This meant the PowerShell wrapper failed before starting the installer.

Reproduction:

A harmless wrapper test failed:

Invoke-InteractiveInstaller.ps1 `
  -InstallerPath C:\Windows\System32\cmd.exe `
  -InstallerArguments "/c" "exit 0"

Error:

A positional parameter cannot be found that accepts argument 'exit 0'.

Cause:

Under powershell.exe -File, passing multiple values to a script [string[]] parameter was not binding as intended.

Fix:

The task wrapper now passes child process arguments as base64-encoded JSON:

$argumentsJson = ConvertTo-Json -InputObject $Arguments -Compress
$argumentsBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($argumentsJson))

The interactive wrapper decodes that back to a real argument array:

$argumentsJson = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($InstallerArgumentsBase64))
$decodedArguments = ConvertFrom-Json -InputObject $argumentsJson
$InstallerArguments = @()
foreach ($argument in $decodedArguments) {
    $InstallerArguments += [string]$argument
}

The harmless wrapper test then produced:

"installerArgs": ["/c", "exit 0"],
"exitCode": 0

Uninstall Test Flow

The smoke harness now does this inside the guest:

1. Schedules an interactive install task.

2. Runs the packaged installer with:

   --headed --automation
   

3. Verifies installed state:

   • %LOCALAPPDATA%\CovenantSetupSelfTest\bin\covenant-setup.exe
   • %LOCALAPPDATA%\CovenantSetupSelfTest\journal.json
   • %LOCALAPPDATA%\CovenantSetupSelfTest\covenant-setup-uninstall.exe
   • HKCU:\Software\CovenantSetupSelfTest
   • Desktop\CovenantSetupSelfTest.lnk

4. Schedules an interactive uninstall task.

5. Runs the installed uninstaller with:

   --headed --automation uninstall <journal path>
   

6. Waits for cleanup helper completion.

7. Verifies removed state:

   • install root removed
   • payload removed
   • journal removed
   • installed uninstaller removed
   • desktop shortcut removed
   • application registry key removed
   • Installed Apps uninstall registration removed

Automation Changes for Uninstall

The uninstall path can spawn a cleanup helper to delete the running uninstaller executable after the main uninstall process exits. The cleanup helper previously could still show a GUI success or reboot prompt.

Fix:

• uninstall now receives the automation flag.
• cleanup now receives the automation flag.
• spawn_cleanup_helper propagates --automation.
• When the parent UI mode is GUI, spawn_cleanup_helper also passes --headed.
• GUI success/reboot prompts are skipped in automation mode.

This keeps the C# progress UI visible while preventing blocking prompts during automated tests.

Abort Collector

Added:

scripts/windows-vm/Abort-SmokeDiagnostics.ps1

Purpose:

• Write an explicit abort_requested event.
• Capture processes, visible windows, scheduled tasks, task info, and recent event logs.
• Stop installer, uninstaller, C# UI, and smoke scheduled tasks.
• Unregister smoke scheduled tasks.
• Zip and return the trace bundle as base64.

This is useful when the host-side test command is interrupted and the normal finally block does not complete.

Final Verified Result

The final smoke test command was:

.\scripts\run-windows-vm-smoke.ps1 -SkipViewer -HaltAfter

It passed with:

{
  "success": true,
  "exitCode": 0,
  "installExitCode": 0,
  "uninstallExitCode": 0,
  "uninstallVerified": true
}

The final trace showed both scheduled tasks completing:

• CovenantSetupSelfInstall-Install-...
• CovenantSetupSelfInstall-Uninstall-...

It also showed:

• install Rust heartbeat
• install C# pipe log
• uninstall Rust heartbeat
• uninstall C# pipe log
• cleanup helper heartbeat
• uninstall_cleanup_observed with every checked path/key absent

Relevant Code Changes

Build and Packaging

• .cargo/config.toml
  • Enables static MSVC runtime linking for the Rust executable.
• build.rs
  • Publishes the C# WinForms UI as self-contained win-x64.
  • Sets COVENANT_SETUP_UI_EXE for Rust embedding.
• src/main.rs
  • Adds trace events.
  • Uses raw embedded bundle format.
  • Uses C# UI IPC instead of PowerShell UI.
  • Writes packaged install journal to the inferred install root.
  • Propagates automation through uninstall cleanup.

C# UI

• src/ui.rs
  • Extracts embedded C# UI executable.
  • Connects to a named pipe.
  • Sends JSON UI messages.
  • Logs Rust-side pipe events.
• ui/Covenant.Setup.Ui/Program.cs
  • Hosts the named pipe server.
  • Displays progress, logs, and prompts.
  • Logs C# pipe events to csharp-ui-pipe-<pid>.jsonl.

Vagrant Harness

• scripts/run-windows-vm-smoke.ps1
  • Requires dotnet.
  • Packages with Start-Process -Wait.
  • Validates embedded bundle marker.
  • Uploads guest scripts.
  • Pulls trace bundle in finally.
  • Reports install and uninstall status.
• scripts/windows-vm/Start-InteractiveSelfInstall.ps1
  • Runs install and uninstall as separate interactive scheduled tasks.
  • Verifies installed state before uninstall.
  • Verifies removed state after uninstall.
  • Writes detailed guest trace events.
• scripts/windows-vm/Invoke-InteractiveInstaller.ps1
  • Starts a target executable with decoded argument list.
  • Polls process state instead of relying only on WaitForExit.
  • Writes per-operation diagnostics.
• scripts/windows-vm/Abort-SmokeDiagnostics.ps1
  • Captures and aborts an in-progress smoke run.

Removed PowerShell UI

• scripts/windows-vm/Approve-InstallerDialogs.ps1
  • Removed because the automation path no longer clicks PowerShell UI dialogs.
• src/win.rs
  • PowerShell/TaskDialog UI helpers were removed from the primary UI flow.

Troubleshooting Guide for the Next Hang

1. Check whether the host command is still running:

   Get-Process | Where-Object { $_.ProcessName -match 'vagrant|ruby|covenant' }
   

2. If the host-side Vagrant process is stuck and the run should be aborted, stop only the Vagrant/Ruby processes for that run.

3. Pull guest diagnostics:

   vagrant upload scripts\windows-vm\Abort-SmokeDiagnostics.ps1 C:\Users\vagrant\AppData\Local\Temp\covenant-setup-smoke\scripts\Abort-SmokeDiagnostics.ps1
   vagrant winrm -s powershell -c "& 'C:\Users\vagrant\AppData\Local\Temp\covenant-setup-smoke\scripts\Abort-SmokeDiagnostics.ps1'"
   

4. Inspect dist\vagrant-self-test\trace\guest-events.jsonl.

5. Interpret missing logs:

   • No interactive_installer_start: scheduled task or PowerShell wrapper failed before launching the installer.
   • interactive_installer_start exists, but no installer-heartbeat-*.jsonl: executable failed before Rust main(), usually loader/dependency/signing/OS error.
   • Rust heartbeat exists, but no csharp-ui-pipe-*.jsonl: C# UI failed to start or pipe connection failed.
   • Both heartbeat and pipe logs exist: inspect the last Rust phase and last C# pipe phase to find the blocked operation.

6. Check interactive-windows.json for modal system dialogs.

7. Check abort-processes.json and abort-scheduled-task-info.json for orphaned tasks or running installers.