The real-user-flow job passed end to end (run 31492613931, 10m57s):
website Hermes-Setup.exe installed headed (AHK Install+Launch, real app
window), then TWO GUI updates driven by real Settings -> About ->
'Update now' clicks, each carried through the detached hand-off to a
relaunched desktop on the target commit. Every assertion green on both
legs (marker cleanup, checkout on target sha, working hermes, relaunch).
Two finishing touches:
* Foreground the relaunched Hermes window before the 99-relaunched proof
screenshot — the full-desktop grab is z-order dependent and one run
caught VS Code on top. The relaunch ASSERT already passed on the
process signal; this is purely to make the proof image show Hermes.
* Remove the temporary branch push trigger used for pre-merge validation;
back to main + nightly + release tags + manual dispatch only.
Attempt 10 ran 1h40m and the diagnosis is precise: the GUI update hung on
a bare input() in hermes update's _sync_with_upstream_if_needed. Our
serve.git origin is a file:// URL, so _is_fork() is true and the updater
asks 'Add official repo as upstream? [Y/n]' via raw input(). When the
Desktop spawns the hand-off through 'cmd start /min' that child has a real
but EMPTY console, so input() blocks forever (no EOF, no keystroke). The
contract job spawns the hand-off with inherited non-interactive stdin, so
input() hits EOF and defaults immediately -- which is why it never hung.
The proof chain confirmed everything else worked: backend exited, venv
unlocked, git pull found the commit and applied it; the process then just
sat in input(). update.log was never created because the hang is BEFORE
the desktop-build step.
Fix: create HERMES_HOME/.skip_upstream_prompt after install -- the
product's own 'don't ask about upstream' marker (_should_skip_upstream_
prompt). Real GUI users install from the official github origin where
_is_fork() is false and this prompt never fires, so this only neutralizes
a staging artifact of the file:// serve repo, not real behavior.
(Noted for a separate product follow-up: hermes update --gateway should
route this input() through _gateway_prompt like its other prompts, so a
fork-origin GUI update can't hang even without the marker.)
Attempt 9 drove the full real update through the hand-off: marker
detected, desktop exited, hermes update fetched from serve.git, found the
commit, pulled, restored -- all correct. It then timed out because the
updater legitimately runs LONG here: the website release we install
(v0.20.0) is weeks of main behind CURRENT, so the update pulls a large
diff AND does a full Electron desktop rebuild (vite + electron-builder)
plus uv sync. The contract job's BASE->CURRENT is a 1-commit tests-only
diff that skips the rebuild, which is why it finishes in ~1 min; the GUI
job's release->CURRENT does not.
* wait window 40 -> 90 min per leg; job timeout 180 -> 240 min
* tail logs/update.log during the wait so the desktop-rebuild phase is
visible in CI output instead of tens of minutes of silence (the rebuild
streams there, not to the handoff log)
The CURRENT->NEXT leg stays fast (NEXT is a same-tree child of CURRENT,
no rebuild), so total stays well within 240 min.
Attempt 8 drove the ENTIRE GUI update click-path successfully: onboarding
dismissed, Settings opened, About opened, Update now clicked, updating
overlay shown. The hand-off log proves the real update then ran: desktop
(pid 8880) exited, venv unlocked, 'hermes update --yes --gateway --force
--branch main' fetched from serve.git, found 1 new commit, pulled, and
restored. Everything worked.
The only failure was the driver waiting on Playwright's app 'close'
event, which doesn't fire reliably when the Electron app self-quits for
the hand-off. Switch to the authoritative signal: poll for the
HERMES_HOME/.hermes-update-in-progress marker (or the result JSON, or a
genuine window-gone), which the hand-off writes ~4s after the click. The
PowerShell driver still owns asserting the OUTCOME (target sha, marker
cleanup, working hermes, relaunched app) after the driver returns.
Attempt 7 got the whole way into the GUI update leg: the installed
Electron app launched under Playwright, booted, composer attached, first
screenshot captured. It then couldn't find the settings gear -- the
ERROR screenshot showed why: a fresh install with no CONFIGURED provider
(the seeded .env key isn't read as model.provider) shows the onboarding
card ('Let''s get you setup with Hermes Agent'), which covers the shell
and its settings gear.
The update path needs no provider, so the driver now clicks 'I'll choose
a provider later' (with skip fallbacks) to dismiss onboarding and reach
the shell before looking for the gear. Harmless no-op when onboarding
isn't shown. Gear (aria-label 'Open settings') and About nav ('About')
selectors already match the real components.
Install + GUI update leg now reached (attempt 6): full install passes,
first update leg begins. It tripped a preflight assert checking
apps/desktop/node_modules/@playwright/test — but the root npm ci HOISTS
workspace devDependencies to the REPO-ROOT node_modules, so that path is
empty by design. Node's own resolution walks up from apps/desktop and
finds it (which is exactly how the copied-in drive-update.cjs will load
it), so assert via 'node -e require.resolve(...)' from apps/desktop
instead of a hardcoded nested path.
Windows PowerShell 5.1 reads .ps1 without a BOM under the legacy OEM
codepage, mis-decoding UTF-8 bytes. Em-dashes/box-drawing survived in
comments through attempts 2-4, but the previous commit added an em-dash
INSIDE a double-quoted Write-Host string — the misdecode there ate the
quote boundary and cascaded into a whole-file parse failure at the Stage
step ('Unexpected token', 'string is missing the terminator').
scripts/install.ps1 documents this exact constraint ('pure ASCII for PS
5.1 parser compatibility'). Strip all non-ASCII from the .ps1 and .ahk
files (em-dash->--, arrows->->, box-drawing->-). drive-update.cjs keeps
UTF-8 (Node decodes it natively). Both PowerShell files parse clean.
The full GUI install flow now works end-to-end (attempt 4 proof: Install
clicked, bootstrap complete, Launch clicked, real Hermes.exe window
appeared 1024x720, installer exited, 5 Hermes processes running). The
only failure was an over-strict staging assertion.
The website Hermes-Setup.exe pins a main release commit. On a real
push-to-main run CURRENT is main's tip, so that pin is its ancestor and
the check holds. On a diverged feature branch CURRENT is a branch commit
the release pin is not an ancestor of — a legitimate topology, not a bug.
The update leg resets the checkout to serve.git's main ref (= CURRENT)
regardless of ancestry and asserts it lands there, which is the actual
forward-update proof. Downgrade the ancestor check to an informational
note so branch validation can exercise the update legs.
Attempt 3's proof frames showed the install SUCCEEDED end-to-end
(bootstrap complete, installer self-copied to HERMES_HOME) and the
window advanced to 'HERMES IS READY' with a [ LAUNCH ] button at the
same centered CTA spot the [ INSTALL ] button occupied — screen (511,454)
inside window x=64 y=34 w=896 h=659.
Two Launch-step bugs, both fixed from that evidence:
* launch-button.png was the stale #68183 template and never matched the
restyled '[ LAUNCH ]' button. Re-captured from the live frame.
* the window-relative fallback used fy=0.59, clicking y=422 — above the
real button. Correct fraction is (454-34)/659 = 0.637. With the
template now matching, the fallback is belt-and-braces anyway.
Install click, completion detection, and the app-window wait were all
already correct in attempt 3; only the Launch click missed.
Attempt 2's proof frames showed two bugs, both now fixed from the live
evidence:
1. The #68183 install-button.png predated the installer UI restyle to the
'[ INSTALL ]' bracket look, so the template never matched and we fell
through to the position fallback. Re-captured install-button.png from a
real CI desktop frame (the actual rendered button).
2. The fallback then clicked the WRONG spot: ahk_exe's first WinGetPos
matched a hidden 16x16 helper window ('Window found at w=16 h=16' in
ahk.log), and BTN_FY=0.87 aimed below the real button anyway. The
button center measured at ~(0.50, 0.59) of the ~full-screen window.
Rewrite:
* WaitForRealWindow() skips phantom/hidden matches (requires w>400,h>300)
and returns the true rect; the installer window is then activated before
any click.
* Install-finished is now driven primarily by the authoritative
'bootstrap complete' line in bootstrap-installer.log (matches
BootstrapEvent::Complete), with the Launch template as a secondary
signal and a window-relative fallback click.
* Fallback clicks use the corrected (0.50, 0.59) window fraction.
Frame-0005 of the proof capture showed the exact failure: 'Unhandled
error: (6) The handle is invalid' rendered over the installer within
seconds of launch. AutoHotkey started via Start-Process has no console,
so FileAppend to '*' (stdout) throws — and the throw fired inside Log(),
killing the script before it clicked anything. The installer then sat
untouched at the INSTALL screen for 50 minutes.
* Log() now try-wraps the stdout write (file log is the real record)
* Install/Launch clicks fall back to the button's relative window
position when the #68183-era PNG templates don't match the restyled
UI ('[ INSTALL ]' bracket style visible in the same frame)
* install-finished has a second signal: 'bootstrap complete' in
bootstrap-installer.log (read with write-sharing), so a template miss
can't strand the wait
* driver passes the bootstrap log path as arg 3
Second job on the Windows E2E workflow covering the surfaces a user
actually touches, per Teknium's requirement:
* INSTALL: downloads the production Hermes-Setup.exe from
hermes-assets.nousresearch.com, launches it HEADED, and AutoHotkey
clicks Install -> waits -> clicks Launch (button templates + ImageSearch
approach from @ethernet8023's #68183, retargeted by process name and
extended to exercise the Launch hand-off). The real Electron Hermes.exe
window must appear.
* UPDATE x2: the installed Hermes.exe is launched under Playwright's
Electron driver and the test CLICKS Settings -> About -> Update now.
The production hand-off chain runs untouched: app quits, detached
updater (repo script or staged binary) runs hermes update, rebuilds
the desktop, relaunches Hermes.exe. Asserts: target sha, marker
cleanup, result JSON when the script path wrote one, working hermes,
and the RELAUNCHED app window. Leg 1 -> CURRENT, leg 2 -> synthetic
NEXT.
Proof artifacts: per-step renderer screenshots (booted app, settings,
About panel, update-available, updating overlay), full-desktop frames
every 3s across the whole run, ahk.log, bootstrap-installer.log,
desktop-update-handoff.log — uploaded on success AND failure.
The website exe runs exactly as shipped (its own pinned install.ps1,
its baked release-pin commit); the only environmental deltas are the
serve.git URL redirect, uploadpack.allowAnySHA1InWant for the commit
pin fetch, and a placeholder provider key so the update legs meet the
app shell instead of onboarding.
The contract job from the previous commits is unchanged and independent
— it remains the rollback position if the GUI job proves flaky.
The Windows E2E ran end-to-end green on this branch (run 31462244593):
install at BASE, update BASE->CURRENT, update CURRENT->NEXT, all asserts
passing. Back to main/nightly/tags/dispatch triggers only.
First CI run's install leg cloned real GitHub main instead of the staged
BASE (caught by the HEAD-at-BASE assert): install.ps1 sets
GIT_CONFIG_COUNT=1 / windows.appendAtomically itself, silently clobbering
the driver's env-config insteadOf rewrites. A driver-owned gitconfig file
selected via GIT_CONFIG_GLOBAL survives that (and install.ps1's own
--global writes land harmlessly in the same file). Verified locally by
cloning with the clobber vars set: clone lands on staged BASE.
The push-triggered validation run failed with zero jobs ('workflow file
issue'): job-level env only allows github/inputs/matrix/needs/secrets/
strategy/vars. Use a sibling of github.workspace for the E2E workroot
instead. actionlint now passes clean.
Every commit on main now proves, on a real Windows machine, that:
1. the PRIOR commit (HEAD~1) installs from scratch through its own
scripts/install.ps1 (-IncludeDesktop: uv, managed Python, Node,
venv, packaged Electron Hermes.exe),
2. that install updates TO this commit through the real Desktop GUI
update path (scripts/desktop-update.ps1, the exact hand-off the
Update button spawns -- fail-closed gates, marker lifecycle,
hermes update, result JSON), and
3. this commit updates FORWARD to a synthetic next commit, proving
the updater code shipping in this commit is not the one that
strands users when the next commit lands.
Staging: the driver bare-clones the checkout into serve.git and
redirects the canonical GitHub URLs at it with git insteadOf env
config, then advances the served main ref BASE -> CURRENT -> NEXT
between legs. Installer and updater run byte-for-byte unmodified.
Supersedes the AutoHotkey pixel-driving approach (#68183): the GUI
Update button's entire effect is spawning desktop-update.ps1 with
documented flags, so driving that contract directly tests the same
production code deterministically.
The first-run provider picker showed Fireworks AI alongside Nous Portal
before the user opened the 'Other providers' disclosure. Only Nous Portal
should be visible up front; Fireworks now lives inside the expanded list
but keeps its #1 position there (Nous -> Fireworks ordering preserved).
get_env_value/load_config read through the shared os.environ mirror that
save_env_value writes, so a reader-based assertion cannot prove which
profile's store actually received the write. Read the two profiles'
config.yaml and .env directly instead, and cover the credential path.
The custom-endpoint REST handlers ran bare load_config/save_config, so
every add/activate/delete landed in the process-level default profile
regardless of which profile the desktop settings UI was targeting. A
provider added under a non-default profile silently went to default:
visible only in default-bound sessions, absent everywhere else, and
un-addable to another profile without hand-editing its config.yaml.
Scope all four handlers (list/upsert/activate/delete) to the requested
profile via _config_profile_scope, matching /api/config, and spread the
active profile into the four hermes.ts wrappers alongside their existing
validateCustomEndpoint sibling.
Reconcile the salvaged #81533 lifecycle helper with the renderer-log
console pipeline that landed in #83535 (the two PRs raced):
- window-renderer-lifecycle.ts no longer handles console-message —
renderer-log.ts is the single owner (per-window labels, boundary
reports). One owner means no double-logged errors on windows wearing
both, and OAuth/portal windows (lifecycle-wired for process events)
cannot spill third-party page console output into desktop.log.
- wake indicator window gets attachRendererConsoleCapture, keeping the
console coverage it previously got from the helper.
- HUD window (added after the PR branched) gets log-only lifecycle
coverage — it was the one renderer window the PR couldn't have known
about.
- Tests updated: lifecycle helper asserts it attaches NO console-message
listener; parser tests live in renderer-log.test.ts.
@spfcraze's triage review noted the PR description claimed "every
BrowserWindow" but the OAuth and portal sign-in windows were not wired:
a crashed sign-in renderer leaves the window's promise path never
settling, with no trace in desktop.log.
Wire both with the same log-only lifecycle diagnostics as the overlay
and quick windows — `kind: 'oauth'` and `kind: 'portal'` respectively.
Neither window gets crash-reload treatment (a sign-in window that
reloads itself mid-auth would be surprising); the lifecycle helper's
log-only callback is the exact contract needed here.
window-renderer-lifecycle.test.ts: 17/17 pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An unreadable profile dir made (entry / '.env').exists() raise
PermissionError out of the sidebar fallback, 500ing /api/profiles.
Found by hostile fixture during live E2E of the scandir conversion.
Two-part fix for the dashboard fd exhaustion reported in #81547:
1. Raise RLIMIT_NOFILE soft limit on startup (before uvicorn binds).
macOS defaults to 256 for LaunchAgent processes — too tight for the
dashboard which opens 3 fds (db+wal+shm) per SessionDB per request
across all profiles. After days of polling the soft limit exhausts
and every os.listdir/open raises OSError [Errno 24]. The helper raises
to the hard limit (or minimum 4096), matching the reporter's ulimit
workaround. No-op on Windows (no resource module).
2. Replace bare Path.iterdir() with context-managed os.scandir() in four
dashboard hot paths: _fallback_profile_dicts, file manager list,
checkpoint listing, and plugin discovery. iterdir() returns a
generator that holds an open directory fd until fully consumed; if
an exception interrupts iteration the fd leaks. os.scandir() is an
explicit context manager that guarantees close on exit, following
the same idiom already used in /api/fs/list.
Tests: 6 passed, 3 skipped (resource-module tests skip on Windows).
_spawn_gateway_restart() now calls _reap_unsupervised_gateway_orphans()
before spawning a new `hermes gateway restart` child. On desktop-app
restart the old serve exits but its gateway child gets reparented to
launchd (PPID=1) and keeps its platform connection alive. The new
serve then spawns a fresh gateway, resulting in two live gateways
racing the same connection.
The reap was already implemented for the CLI restart path (#75936) but
the dashboard's _spawn_gateway_restart path was not covered.
Fixes#77276
Three-part class closure for the React #310 / lost-renderer-crash family
(#79428, follow-up to #80560 / #82763):
1. Diagnosability (#79428 defect B): error-boundary catches now persist to
desktop.log with their component stack via a new fire-and-forget
hermes:logs:renderer-error IPC (synchronous flush — the window may be
dying). Every renderer-content window (main, secondary session, instance,
HUD, quick entry, pet overlay) gets the error-level console capture that
previously only the main window had, labeled per window. 'Open logs' on
the crash dialog now reveals a file that actually contains the crash.
2. Recurrence guard: eslint no-restricted-syntax rule banning inline
render() calls in JSX — the mechanism behind #80560. The rule
immediately caught two live sites #82763's audit missed (floating
panes, narrow-overlay reveal), both hosting plugin panes.
3. Fix those two missed sites with the same ContribRender mount.
extracted console-capture/report formatting to electron/renderer-log.ts
with unit tests; renderer console lines now carry the window label.
os.kill(pid, 0) sends CTRL_C_EVENT on Windows (bpo-14484). The reap path
is POSIX-only, but the blocking lint rejects the pattern repo-wide and
psutil is a core dependency.
On Desktop serve startup, reap orphan gateway processes (PPID=1) left
behind by a previous serve session that exited abnormally. This prevents
the old and new gateways from racing for the same QQ WebSocket
credential, which splits messages across parallel session trees (#77276).
Production incident: the orphan reap killed a legitimate SSH remote backend
started by another client machine. Its process sat at ppid 1 with the same
cmdline shape as a genuine orphan, and the exclusion list only covered THIS
app instance's children — ownership by OTHER clients was invisible.
The reap now treats every backend.lock.json under ~/.hermes/desktop-ssh/*/
as an ownership claim: lock payloads are schema-validated (mirroring
remote-lifecycle.ts) and their PIDs are excluded both before the scan and
re-checked after it (defense in depth against a lock written mid-scan).
Regression tests cover the exact incident shape: a lock-owned PID and a
genuine orphan with identical process shapes — only the orphan is reaped.
Also: fold the new single-field `runtime` config category into `agent`
(_CATEGORY_MERGE) and fix an env leak in the serve-startup test
(HERMES_SERVE_HEADLESS restored via monkeypatch) so the combined suites
run green in any order.
launchd starts children with soft nofile=256; hermes gateway start rewrites the plist and previously stripped any manually-added SoftResourceLimits, silently reintroducing EMFILE crashes under load. The plist generator now embeds the configured runtime.nofile_soft_limit so the persisted service definition and the in-process floor share one knob.
When Desktop exits uncleanly, leftover `hermes serve --host 127.0.0.1 --port 0`
processes can be reparented to pid 1 and keep full MCP trees alive. The next
boot then stacks another backend on top of the corpses until EMFILE kills
sidebar/session APIs and tabs disappear.
- Detect Desktop-local serve shape (loopback + ephemeral port 0)
- Only reap processes whose ppid is 0/1 (true orphans)
- Spare fixed-port remote serves (e.g. --port 9119) and HERMES_DESKTOP_CHILD_PID
- Run at Desktop backend start (HERMES_DESKTOP=1) before parent-death watchdog
Complements parent-death watchdog (prevents future orphans) and configurable
nofile soft limit (capacity floor). Together these stop the multi-backend
pile-up cascade observed on macOS Desktop SSH/local installs.
An unclean desktop exit (crash / SIGKILL / update handoff) stranded every
`hermes serve` profile backend as an orphan (ppid=1) still serving, each
holding its MCP child subtree — 31 orphans / ~1.3 GiB RSS on one install.
Root causes + fixes:
- serve had no parent-death watchdog: add _start_parent_death_watchdog() in
web_server.py (mirrors slash_worker.py), gated on HERMES_PARENT_PID; os._exit
cascades to MCP watchdogs. No-op for standalone `hermes serve`.
- desktop passes HERMES_PARENT_PID in both serve spawn env blocks (main.ts).
- POSIX teardown now group-kills (process.kill(-pid, ...)) so MCP grandchildren
die too (backend-child.ts + waitForBackendExit SIGKILL fallback).
Windows path unchanged (forceKillProcessTree). Tests updated + passing.
Review of this PR was right that maxsize=8 bounds the wrong thing. The
LifoQueue caps how many connections are RETURNED; _checkout_read_conn opened
unconditionally on a miss, so N readers arriving on a cold pool all missed, all
opened, and peaked at N. The surplus was closed on release, so nothing
accumulated forever -- but EMFILE is a peak-instant condition and the burst
that empties the pool is exactly the burst that exhausts the fd table, so the
original wedge was still reachable. Measured on the previous commit: 64
concurrent readers held 64 live connections at once.
A connection now holds a permit for its whole lifetime -- acquired in
_get_read_conn() before the open, released in _close_read_conn() after the
close -- so open+checked-out is bounded together. A pool hit costs no permit
because the connection it hands back already holds one, which leaves
_get_read_conn() as the only place that can open. The acquire is non-blocking:
past the ceiling readers fall back to the locked writer connection rather than
queueing, since blocking would convert descriptor exhaustion into a stall,
which is the same outage with a different stack trace. Same burst now peaks at
8. BoundedSemaphore rather than Semaphore so an unpaired release raises instead
of silently widening the ceiling.
Two latent leaks in the same function, found while doing this:
- a CJK extension load that failed after a successful open returned None
without closing the connection, leaking a descriptor the tracking registry
still counted -- the same leak shape one level down;
- any non-sqlite3.Error between open and return stranded a permit
permanently, which would ratchet the ceiling down to zero and silently
demote every later read to the writer lock.
On the test: the existing one joins every worker before counting, so it
measures the pool at rest and structurally cannot observe peak -- which is why
this got through. The new one uses a barrier so all 64 workers hold their
connections until every worker has checked out, making the count taken at that
moment the actual simultaneous peak. Verified it fails against the previous
commit (64 checked out, 65 live) and passes at 8/9. Also covers the
writer-connection fallback, permit recovery after a failed open, and that
close() releases exactly the permits it drained.
Teknium
brooklyn!
Math
hermes-seaeye[bot]
Chen Jin
Michael Gannotti
a-espinoza
Quark Assistant
RelaxJonh
cadezhou
Leon Phull
XiaoZAZA
Eva
Yishova