The real-user-flow job passed end to end (run 31492613931, 10m57s):
website Hermes-Setup.exe installed headed (AHK Install+Launch, real app
window), then TWO GUI updates driven by real Settings -> About ->
'Update now' clicks, each carried through the detached hand-off to a
relaunched desktop on the target commit. Every assertion green on both
legs (marker cleanup, checkout on target sha, working hermes, relaunch).
Two finishing touches:
* Foreground the relaunched Hermes window before the 99-relaunched proof
screenshot — the full-desktop grab is z-order dependent and one run
caught VS Code on top. The relaunch ASSERT already passed on the
process signal; this is purely to make the proof image show Hermes.
* Remove the temporary branch push trigger used for pre-merge validation;
back to main + nightly + release tags + manual dispatch only.
Attempt 10 ran 1h40m and the diagnosis is precise: the GUI update hung on
a bare input() in hermes update's _sync_with_upstream_if_needed. Our
serve.git origin is a file:// URL, so _is_fork() is true and the updater
asks 'Add official repo as upstream? [Y/n]' via raw input(). When the
Desktop spawns the hand-off through 'cmd start /min' that child has a real
but EMPTY console, so input() blocks forever (no EOF, no keystroke). The
contract job spawns the hand-off with inherited non-interactive stdin, so
input() hits EOF and defaults immediately -- which is why it never hung.
The proof chain confirmed everything else worked: backend exited, venv
unlocked, git pull found the commit and applied it; the process then just
sat in input(). update.log was never created because the hang is BEFORE
the desktop-build step.
Fix: create HERMES_HOME/.skip_upstream_prompt after install -- the
product's own 'don't ask about upstream' marker (_should_skip_upstream_
prompt). Real GUI users install from the official github origin where
_is_fork() is false and this prompt never fires, so this only neutralizes
a staging artifact of the file:// serve repo, not real behavior.
(Noted for a separate product follow-up: hermes update --gateway should
route this input() through _gateway_prompt like its other prompts, so a
fork-origin GUI update can't hang even without the marker.)
Attempt 9 drove the full real update through the hand-off: marker
detected, desktop exited, hermes update fetched from serve.git, found the
commit, pulled, restored -- all correct. It then timed out because the
updater legitimately runs LONG here: the website release we install
(v0.20.0) is weeks of main behind CURRENT, so the update pulls a large
diff AND does a full Electron desktop rebuild (vite + electron-builder)
plus uv sync. The contract job's BASE->CURRENT is a 1-commit tests-only
diff that skips the rebuild, which is why it finishes in ~1 min; the GUI
job's release->CURRENT does not.
* wait window 40 -> 90 min per leg; job timeout 180 -> 240 min
* tail logs/update.log during the wait so the desktop-rebuild phase is
visible in CI output instead of tens of minutes of silence (the rebuild
streams there, not to the handoff log)
The CURRENT->NEXT leg stays fast (NEXT is a same-tree child of CURRENT,
no rebuild), so total stays well within 240 min.
Attempt 8 drove the ENTIRE GUI update click-path successfully: onboarding
dismissed, Settings opened, About opened, Update now clicked, updating
overlay shown. The hand-off log proves the real update then ran: desktop
(pid 8880) exited, venv unlocked, 'hermes update --yes --gateway --force
--branch main' fetched from serve.git, found 1 new commit, pulled, and
restored. Everything worked.
The only failure was the driver waiting on Playwright's app 'close'
event, which doesn't fire reliably when the Electron app self-quits for
the hand-off. Switch to the authoritative signal: poll for the
HERMES_HOME/.hermes-update-in-progress marker (or the result JSON, or a
genuine window-gone), which the hand-off writes ~4s after the click. The
PowerShell driver still owns asserting the OUTCOME (target sha, marker
cleanup, working hermes, relaunched app) after the driver returns.
Attempt 7 got the whole way into the GUI update leg: the installed
Electron app launched under Playwright, booted, composer attached, first
screenshot captured. It then couldn't find the settings gear -- the
ERROR screenshot showed why: a fresh install with no CONFIGURED provider
(the seeded .env key isn't read as model.provider) shows the onboarding
card ('Let''s get you setup with Hermes Agent'), which covers the shell
and its settings gear.
The update path needs no provider, so the driver now clicks 'I'll choose
a provider later' (with skip fallbacks) to dismiss onboarding and reach
the shell before looking for the gear. Harmless no-op when onboarding
isn't shown. Gear (aria-label 'Open settings') and About nav ('About')
selectors already match the real components.
Install + GUI update leg now reached (attempt 6): full install passes,
first update leg begins. It tripped a preflight assert checking
apps/desktop/node_modules/@playwright/test — but the root npm ci HOISTS
workspace devDependencies to the REPO-ROOT node_modules, so that path is
empty by design. Node's own resolution walks up from apps/desktop and
finds it (which is exactly how the copied-in drive-update.cjs will load
it), so assert via 'node -e require.resolve(...)' from apps/desktop
instead of a hardcoded nested path.
Windows PowerShell 5.1 reads .ps1 without a BOM under the legacy OEM
codepage, mis-decoding UTF-8 bytes. Em-dashes/box-drawing survived in
comments through attempts 2-4, but the previous commit added an em-dash
INSIDE a double-quoted Write-Host string — the misdecode there ate the
quote boundary and cascaded into a whole-file parse failure at the Stage
step ('Unexpected token', 'string is missing the terminator').
scripts/install.ps1 documents this exact constraint ('pure ASCII for PS
5.1 parser compatibility'). Strip all non-ASCII from the .ps1 and .ahk
files (em-dash->--, arrows->->, box-drawing->-). drive-update.cjs keeps
UTF-8 (Node decodes it natively). Both PowerShell files parse clean.
The full GUI install flow now works end-to-end (attempt 4 proof: Install
clicked, bootstrap complete, Launch clicked, real Hermes.exe window
appeared 1024x720, installer exited, 5 Hermes processes running). The
only failure was an over-strict staging assertion.
The website Hermes-Setup.exe pins a main release commit. On a real
push-to-main run CURRENT is main's tip, so that pin is its ancestor and
the check holds. On a diverged feature branch CURRENT is a branch commit
the release pin is not an ancestor of — a legitimate topology, not a bug.
The update leg resets the checkout to serve.git's main ref (= CURRENT)
regardless of ancestry and asserts it lands there, which is the actual
forward-update proof. Downgrade the ancestor check to an informational
note so branch validation can exercise the update legs.
Attempt 3's proof frames showed the install SUCCEEDED end-to-end
(bootstrap complete, installer self-copied to HERMES_HOME) and the
window advanced to 'HERMES IS READY' with a [ LAUNCH ] button at the
same centered CTA spot the [ INSTALL ] button occupied — screen (511,454)
inside window x=64 y=34 w=896 h=659.
Two Launch-step bugs, both fixed from that evidence:
* launch-button.png was the stale #68183 template and never matched the
restyled '[ LAUNCH ]' button. Re-captured from the live frame.
* the window-relative fallback used fy=0.59, clicking y=422 — above the
real button. Correct fraction is (454-34)/659 = 0.637. With the
template now matching, the fallback is belt-and-braces anyway.
Install click, completion detection, and the app-window wait were all
already correct in attempt 3; only the Launch click missed.
Attempt 2's proof frames showed two bugs, both now fixed from the live
evidence:
1. The #68183 install-button.png predated the installer UI restyle to the
'[ INSTALL ]' bracket look, so the template never matched and we fell
through to the position fallback. Re-captured install-button.png from a
real CI desktop frame (the actual rendered button).
2. The fallback then clicked the WRONG spot: ahk_exe's first WinGetPos
matched a hidden 16x16 helper window ('Window found at w=16 h=16' in
ahk.log), and BTN_FY=0.87 aimed below the real button anyway. The
button center measured at ~(0.50, 0.59) of the ~full-screen window.
Rewrite:
* WaitForRealWindow() skips phantom/hidden matches (requires w>400,h>300)
and returns the true rect; the installer window is then activated before
any click.
* Install-finished is now driven primarily by the authoritative
'bootstrap complete' line in bootstrap-installer.log (matches
BootstrapEvent::Complete), with the Launch template as a secondary
signal and a window-relative fallback click.
* Fallback clicks use the corrected (0.50, 0.59) window fraction.
Frame-0005 of the proof capture showed the exact failure: 'Unhandled
error: (6) The handle is invalid' rendered over the installer within
seconds of launch. AutoHotkey started via Start-Process has no console,
so FileAppend to '*' (stdout) throws — and the throw fired inside Log(),
killing the script before it clicked anything. The installer then sat
untouched at the INSTALL screen for 50 minutes.
* Log() now try-wraps the stdout write (file log is the real record)
* Install/Launch clicks fall back to the button's relative window
position when the #68183-era PNG templates don't match the restyled
UI ('[ INSTALL ]' bracket style visible in the same frame)
* install-finished has a second signal: 'bootstrap complete' in
bootstrap-installer.log (read with write-sharing), so a template miss
can't strand the wait
* driver passes the bootstrap log path as arg 3
Second job on the Windows E2E workflow covering the surfaces a user
actually touches, per Teknium's requirement:
* INSTALL: downloads the production Hermes-Setup.exe from
hermes-assets.nousresearch.com, launches it HEADED, and AutoHotkey
clicks Install -> waits -> clicks Launch (button templates + ImageSearch
approach from @ethernet8023's #68183, retargeted by process name and
extended to exercise the Launch hand-off). The real Electron Hermes.exe
window must appear.
* UPDATE x2: the installed Hermes.exe is launched under Playwright's
Electron driver and the test CLICKS Settings -> About -> Update now.
The production hand-off chain runs untouched: app quits, detached
updater (repo script or staged binary) runs hermes update, rebuilds
the desktop, relaunches Hermes.exe. Asserts: target sha, marker
cleanup, result JSON when the script path wrote one, working hermes,
and the RELAUNCHED app window. Leg 1 -> CURRENT, leg 2 -> synthetic
NEXT.
Proof artifacts: per-step renderer screenshots (booted app, settings,
About panel, update-available, updating overlay), full-desktop frames
every 3s across the whole run, ahk.log, bootstrap-installer.log,
desktop-update-handoff.log — uploaded on success AND failure.
The website exe runs exactly as shipped (its own pinned install.ps1,
its baked release-pin commit); the only environmental deltas are the
serve.git URL redirect, uploadpack.allowAnySHA1InWant for the commit
pin fetch, and a placeholder provider key so the update legs meet the
app shell instead of onboarding.
The contract job from the previous commits is unchanged and independent
— it remains the rollback position if the GUI job proves flaky.
First CI run's install leg cloned real GitHub main instead of the staged
BASE (caught by the HEAD-at-BASE assert): install.ps1 sets
GIT_CONFIG_COUNT=1 / windows.appendAtomically itself, silently clobbering
the driver's env-config insteadOf rewrites. A driver-owned gitconfig file
selected via GIT_CONFIG_GLOBAL survives that (and install.ps1's own
--global writes land harmlessly in the same file). Verified locally by
cloning with the clobber vars set: clone lands on staged BASE.
Every commit on main now proves, on a real Windows machine, that:
1. the PRIOR commit (HEAD~1) installs from scratch through its own
scripts/install.ps1 (-IncludeDesktop: uv, managed Python, Node,
venv, packaged Electron Hermes.exe),
2. that install updates TO this commit through the real Desktop GUI
update path (scripts/desktop-update.ps1, the exact hand-off the
Update button spawns -- fail-closed gates, marker lifecycle,
hermes update, result JSON), and
3. this commit updates FORWARD to a synthetic next commit, proving
the updater code shipping in this commit is not the one that
strands users when the next commit lands.
Staging: the driver bare-clones the checkout into serve.git and
redirects the canonical GitHub URLs at it with git insteadOf env
config, then advances the served main ref BASE -> CURRENT -> NEXT
between legs. Installer and updater run byte-for-byte unmodified.
Supersedes the AutoHotkey pixel-driving approach (#68183): the GUI
Update button's entire effect is spawning desktop-update.ps1 with
documented flags, so driving that contract directly tests the same
production code deterministically.
Nothing covered the update path, which is the worst thing to break: a broken
updater strands users on the version that cannot fix itself. `hermes update`
alone is ~2000 lines (hermes_cli/update_cmd.py) and had no end-to-end test.
tests/install/install-update-e2e.sh installs a genuine earlier Hermes through
the real one-liner (curl -fsSL https://…/install.sh | bash, served by
dev-sandbox's MITM proxy at the canonical URL, cloning "github.com" through the
upload-pack shim), which really installs uv, a managed Python, Node and the
venv. It then applies ONE update route and requires the checkout to land on this
commit with `hermes --version` still working -- so a pass means the venv and
entry point survived, not merely that git moved.
One route per run, each on a sandbox built from scratch. Sharing one install
across routes -- or rewinding with `git reset --hard` between them -- leaves the
second route running against a tree the first already updated (same venv, same
console script, same __pycache__), which is not the state any real user is in: a
route could pass only because its predecessor did the work, and a failure in the
first left the second exercising something undefined.
--install-ref chooses what to install first, so this covers "update from an
older release", not just from the tip. Installer flags are probed against the
target rather than assumed, because releases from months back predate flags
current Hermes takes for granted: --skip-browser is read out of that ref's own
install.sh, and `--yes` is asked of the installed `hermes update --help` (the
update subcommand has lived in main.py, subcommands/update.py and update_cmd.py
across the tags we sample, so a static parse rots silently -- and did). Without
those probes, old releases die on "Unknown option: --skip-browser" and
"unrecognized arguments: --yes" before doing any work.
Installer output is streamed through tee rather than captured: a real install of
uv, Python, Node and the venv IS the substance of this test, so it belongs in
the job log, not only in an artifact. pipefail keeps the installer's exit status
rather than tee's, so a failed install cannot look like a pass. The sandbox's
own proxy log is printed in full on failure, since a rejected TLS handshake
explains a failure that otherwise reads as a bare `curl: (35)`.
Deliberately reuses dev-sandbox rather than adding a second harness. An earlier
draft rewrote install.sh's hardcoded URLs with insteadOf and ran it against the
host; that tested the installer LESS faithfully (bash install.sh instead of the
real one-liner, host libs instead of a clean machine, ssh disabled to keep a
failed rewrite from reaching real GitHub) while duplicating a fake Internet we
already have.
Shell, not pytest, so scripts/run_tests.sh and run_tests_parallel.py stay
untouched: a pytest version needed an entry in the former's `env -i` credential
allowlist and a _SKIP_PARTS exclusion in the latter, and every meaningful line
was a command run inside the sandbox anyway.
Two guards, both earned during bring-up. It prefers the `sandbox` wrapper and
falls back to the raw script only when bwrap is on PATH (under Nix the wrapper
supplies the PATH and DEV_SANDBOX_* vars, so the bare script exits 127). And it
refuses to run on a dirty worktree: every dev-sandbox invocation re-derives fake
main from the working copy, so uncommitted changes move the update target
between the call that installs and the call that verifies -- a failure that
looks like a broken updater but is a moving reference.
Teknium
ethernet