Review follow-ups on the salvage:
- The helper script's success check accepted any "PID" line, including
the "PID" = -1 a recently-crashed job reports — while the in-process
path's _parse_launchd_pid_from_list_output rejects non-positive PIDs.
Both bash sites now require a positive PID (grep -qE '"PID" = [0-9]+;')
so the two paths enforce the same supervised-PID standard.
- _graceful_restart_via_sigusr1's drain-wait tail was a duplicate of the
new _wait_for_pid_exit — now delegates to it.
- Stale comments: the ancestry-detection framing at the top of the reload
block, and the exhaustion log's '(refresh ran outside gateway process
tree)' which is false on the new helper-spawn-failure fallback path
(now '(in-process fallback path)').
The reload retry loop treated `launchctl list <label>` exit 0 as success,
but exit 0 also covers a registered-but-not-running definition (macOS 26+
`state = not running`) — the same trap _probe_launchd_service_running
already guards against. Require a PID so success means launchd is
supervising a live process, in both the Python loop and the shell helper.
Verified against live launchd: a RunAtLoad=false job reports exit 0 with
no PID, which the old check accepted and the new one rejects.
Note this is NOT what distinguishes a draining instance — measured, the
label deregisters within ~1s of bootout while the old process drains on.
Waiting for the old PID to exit is what covers that.
Reload chose the in-process bootout/bootstrap path based on POSIX
ancestry, but bootout tears down the job's process coalition, and
coalition membership is inherited at spawn and survives reparenting.
A gateway-spawned process reparented to PID 1 is no longer an ancestor
yet still dies with the coalition, so the retry loop was killed
mid-bootstrap and nothing re-registered the label (KeepAlive can't
revive a job launchd no longer knows about).
- always prefer the detached transient-job helper; it's also correct
when genuinely outside the coalition, just asynchronous
- wait for the old gateway PID to exit before bootstrapping; bootout
only sends SIGTERM and every bootstrap during the drain fails EIO
- fall through to the in-process path when the helper can't spawn
instead of leaving the plist rewritten but never reloaded
request_restart was calling stop() immediately, so the requesting turn stayed
in the drain wait set and got force-killed at restart_drain_timeout. Wait for
active work to reach zero first, then stop against an idle gateway.
- Pass extra_exclude={pid} to _reap_unsupervised_gateway_orphans so the
killed PID isn't double-killed during the sweep (#75936).
- Add extra_exclude param to _reap_unsupervised_gateway_orphans signature.
- Replace bare except:pass with logger.debug for diagnosability.
- Fix existing test (mock _reap_unsupervised_gateway_orphans so it
doesn't scan real processes and trigger conftest live-system guard).
- Add regression test asserting the killed PID is excluded from the sweep.
stop_profile_gateway() only killed the single PID recorded in the pid
file. On repeated restarts, the new process overwrites the pid file
before the old one exits, making older gateway instances invisible
to subsequent stops. Each restart stacked another orphan — observed
as N threads and N replies per Discord message.
After killing the recorded PID, also call _reap_unsupervised_gateway_orphans()
to sweep any remaining gateway processes for this profile. The reap function
already handles the no-systemd guard and SIGTERM+SIGKILL escalation.
Fixes#75936
`hermes update` aborted its managed-Python runtime repair with an error
that reads like a contradiction:
⚠ Managed Python runtime repair skipped: cannot import name
'venv_python_path' from 'hermes_constants'
(/home/teknium/.hermes/hermes-agent/hermes_constants.py)
The named file does contain the symbol. The module in memory does not.
main.py imports hermes_constants from the OLD checkout, `git pull` then
replaces that file on disk, and the freshly-pulled managed_uv runs its
lazy `from hermes_constants import venv_python_path` against the module
object Python already cached in sys.modules — the pre-upgrade one. The
ImportError reports the path of the new file, so it looks like the symbol
is missing from a file that plainly has it.
Same update-boundary class already documented on `_UvResult` for the
ensure_uv() arity skew. It fires on the first update from any release
older than 83314ca38, which introduced the symbol.
Both lazy call sites now reload the module from disk on ImportError:
- hermes_cli/managed_uv.py::_venv_python — the reported path
- hermes_cli/gateway.py::get_python_path — same flaw, same fix; a gateway
restarted mid-update hits it identically
Reload rather than a local fallback on purpose. Recomputing the layout
inline would hand-roll `Scripts`/`bin` a second time — exactly what #76105
deduped into venv_bin_dir()/venv_python_path(), and what
test_no_open_coded_venv_layout_remains_in_hermes_cli bans. Reloading fixes
the actual problem (a stale module) and keeps one owner for the layout.
hermes_cli/update_cmd.py imports the symbol at module scope, which is a
different failure mode (the module fails to import at all) and is already
covered by the installer's retry-once for the update-boundary crash.
Tests reproduce the stale-module state by deleting the attribute from the
imported hermes_constants: recovery resolves through the reloaded shared
helper (asserted via a sentinel, so an open-coded copy cannot pass), and
the normal path never reloads. Against origin/main's managed_uv they fail
with the exact reported ImportError.
`_append_node_dir_for_service()` had two bugs, both caught by
`test_system_unit_uses_target_user_home_not_calling_user`:
1. It crashed. `iter_hermes_node_dirs()` defaults to the *calling* user's
Hermes home, so under sudo it stats `/root/.hermes/node/bin` — which
raises `PermissionError` for a non-root caller rather than returning
False. An unreadable candidate dir means "skip this rung", not "kill
the generator", so the probe now swallows OSError.
2. Worse than the crash: had the stat succeeded, a `--system` unit
targeting alice would have baked *root's* managed Node into alice's
PATH. The generator now skips the managed-Node rung on the system
path and re-runs it after `_hermes_home_for_target_user()` resolves,
passing that home explicitly. Entries are prepended so the managed
Node still outranks the remapped shell-PATH entries, matching the
user-unit ordering.
The launchd generator is unaffected — it has no target-user remapping,
so the default home is already correct there.
Hermes installs runtimes for itself — `uv` at `$HERMES_HOME/bin/uv`, Node
at `$HERMES_HOME/node` — and neither directory is on an arbitrary
process's PATH. Every `shutil.which("node"/"npm"/"npx"/"uv")` in Hermes's
own code therefore has two failure modes: the managed runtime is invisible,
so the caller reports "not installed" or degrades to a slower tier on a
machine that has exactly what it needed; and when a system copy also
exists, the one Hermes does not own wins.
Routed the Hermes-owned call sites through managed-aware resolvers:
- `agent/lsp/install.py`, `hermes_cli/dep_ensure.py`, `hermes_cli/main.py`
(`_make_tui_argv`), `hermes_cli/tools_config.py` (`_run_post_setup`) now
use `find_node_executable()`.
- `hermes_cli/tools_config.py::_pip_install` and `hermes_cli/setup.py`'s
vercel install use `ensure_uv()` (installing uv is in scope during setup,
and the Windows installer's `uv venv` does not seed pip, so the fallback
tier is "No module named pip"). `tools/lazy_deps.py` uses `resolve_uv()`
— a lookup, not a bootstrap, because it runs mid-turn for an optional
dependency and downloading a runtime as a side effect exceeds what the
caller asked for.
- `hermes_cli/gateway.py`: extracted `_append_node_dir_for_service()`,
shared by the systemd unit and launchd plist generators, which appends
the managed dirs before the PATH-resolved one. A service definition is
written once and survives reboots, so resolving a system Node that
happens to lead the installing shell's PATH bakes the wrong interpreter
in permanently. Managed dirs are profile-scoped, so each profile's unit
still names its own Node; the existing symlink-parent rule (don't
`.resolve()`) is preserved verbatim.
- `tools/environments/local.py`: the terminal tool's subshell PATH gains
the managed dirs, appended alongside the sane entries rather than
prepended — a tool the user deliberately put on their own PATH still
wins, and the managed one only fills a gap. This is also what makes the
bare `which("uv")` in `tools/env_probe.py` correct: that probe reports
the environment the *model* sees, and the model can only run what is on
that subshell's PATH.
`scripts/install.ps1`: the persisted User PATH update becomes
`Set-ManagedNodeFirstOnUserPath`, a move-to-front rather than an
add-if-missing. Installs made by an older install.ps1 already have the
managed dir in User PATH — at the tail, behind a system Node — and an
add-if-missing check sees it present and leaves that ordering in place
forever, so the users the bug hurt would never be repaired. Unrelated
entries keep their relative order (empty segments included; a trailing
`;` is legal and the installer's other PATH code preserves them),
duplicates collapse, and it writes only when the string actually changes.
Tests:
- `tests/test_managed_runtime_resolution.py` — AST guard that fails any
new bare `which()` for a managed runtime, with a short justified
allow-list and a companion test that fails when an allow-list entry goes
stale. Reading source is banned by AGENTS.md and this is the documented
exception: the property is "no call site anywhere spells it this way",
which no runtime seam can observe.
- `scripts/ci/test_install_ps1_path_migration.ps1` — behavioral, not a
source regex: it lifts the real `Set-ManagedNodeFirstOnUserPath` out of
install.ps1's AST and rewrites only the two registry calls into an
in-memory store, so the shipped split/dedupe/prepend/change-detection
logic executes for real. Not in the default lane (Linux runners have no
PowerShell host); runs under `pwsh`. 13/13 assertions pass.
CI slice 8/8 red:
test_verify_core_dependencies.py::test_uses_virtual_env_from_environment
AssertionError: assert None == PosixPath('.../newvenv/Scripts/python.exe')
The Phase 2 reviewer flagged this exact risk (W4) and I under-weighted it as
"latent, not broken". It was neither — it was already failing.
The suite exercises Windows-only paths on Linux CI by patching predicates
(`hermes_cli.main._is_windows`, `is_windows`, `platform.system`). Routing
those call sites through a helper that reads `sys.platform` unconditionally
meant the patches no longer reached the path derivation: the test built
`Scripts/python.exe` while the code looked for `bin/python`.
venv_bin_dir/venv_python_path now take an optional `windows=` verdict,
defaulting to the host. Every converted site passes its own predicate, so
the patched-predicate coverage is restored — the dedup keeps the layout in
one place without hijacking the platform decision.
Verified by causation: dropping `windows=` reproduces the CI failure exactly;
restoring it goes green. Added two regression tests, including one asserting
a patched `_is_windows` still reaches the derivation.
Closes#76104, closes#76105.
#76104 — `_atomic_replace_dir` (#49145) made each individual directory swap
safe, but `_update_via_zip` replaced ~70 top-level entries in a loop with no
atomicity across iterations. `agent/` lands at os.listdir index 13 and
`tools/` at 66, so an interruption between them left the new
`agent/context_compressor.py` (module-level `from tools.todo_tool import
TODO_INJECTION_HEADER`) beside a stale `tools/todo_tool.py` — every file
valid Python, the tree unbootable. That is the mechanism behind the
ImportError fixed in #76091, and the "partial update" field report in #63717.
Split into stage-all-then-swap-all:
- `_stage_replacement` copies each dir to a sibling staging path, touching
nothing live, so a failure during the long copy phase is a no-op.
- `_commit_staged_replacements` performs the renames and, if any fails,
restores every entry already swapped — the tree lands wholly new or
wholly old, never mixed.
This shrinks the failure window from a full tree copy to N renames and makes
what remains recoverable. Added an up-front free-space check, since staging
needs a second copy of the tree; a clear error beats running out mid-swap.
#76105 — venv interpreter resolution was open-coded in 7 places across 4
files using 3 different Windows predicates. #76091 added the seventh because
the correct behaviour lived 2400 lines away. Hoisted `venv_bin_dir()` /
`venv_python_path()` into hermes_constants (import-safe, no new imports) and
routed every site through them; `managed_uv._venv_python` now delegates so
its 6 callers are untouched.
`_atomic_replace_dir` is retained — it is re-exported from main.py and has
its own #49145 regression test; removing it is out of scope here.
Tests: 10 new (rollback-on-mid-swap-failure is mutation-verified — it fails
when the rollback loop is removed), plus a guard that fails if a new call
site hand-rolls Scripts/bin again. E2E-verified against the real staging +
commit helpers with a live tree.
Salvaged from #57016 by @lEWFkRAD:
- cli.py: handle file:///C:/... drive-letter URIs on nt (strip the
leading slash urlparse leaves); join Termux example paths with literal
forward slashes so hints stay POSIX on Windows.
- gateway/status.py + hermes_cli/gateway.py: normalize backslashes to
forward slashes before the HERMES_HOME substring match so separator
style cannot defeat profile ownership detection.
- hermes_cli/banner.py: cprint degrades to plain print when
prompt_toolkit has no console (NoConsoleScreenBufferError on
redirected/absent Windows stdout).
- hermes_cli/browser_connect.py: posixpath.join for WSL /mnt/c/... bases
(os.path.join would emit backslashes on nt).
- Test hardening: symlink skip-guards, USERPROFILE alongside HOME for
ntpath.expanduser, SIGKILL absence skipif fixed via monkeypatch,
drive-letter URI / separator-normalization / banner-fallback coverage.
Dropped from the original PR: tests/cli/conftest.py fixture and the
AppSession _output monkeypatch — main's merged tests/cli/conftest.py
already handles that prompt_toolkit pollution.
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
Connecting Discord asked five questions when the platform needs one. The card
listed a home channel ID you need Developer Mode to copy, an allow-all-users
security toggle, a reply-threading preference, and a home channel display name
— all with working defaults, none discoverable from the form.
Drops them from the setup surfaces entirely: the dashboard/Desktop channel
cards and the `hermes setup gateway` wizard. Discord is now bot token +
allowlist. Matrix drops from 11 fields to 7, Mattermost from 6 to 3.
Suffix-matched (`*_HOME_CHANNEL*`, `*_ALLOW_ALL_USERS`, `*_REPLY_TO_MODE`,
`*_REQUIRE_MENTION`, `*_AUTO_THREAD`, `*_FREE_RESPONSE_*`, `*_PROXY`) so plugin
platforms nobody enumerated get the same treatment. Allowlists deliberately
stay — the gateway denies everyone until one is set, so that IS the decision a
new user has to make. Required credentials are never hidden.
Nothing is removed from the product. The vars still work through
`hermes config set`, .env, and config.yaml, the gateway reads them unchanged,
and dropping them from the cards hands them back to the Keys page rather than
orphaning them (Keys hides only what a Channels card owns).
AST-driven pass over every Path.read_text()/write_text() without an
explicit encoding= across non-test code: 71 sites in 34 files
(skills_hub, hermes_cli/main+profiles+service_manager+container_boot,
mem0/hindsight/honcho plugins, achievements dashboard, release/CI
scripts, productivity+comfyui skill helpers, agent/*). Verified zero
positional-encoding collisions before insertion; per-file compile()
check after.
Adds a check-windows-footguns rule flagging bare single-line
read_text/write_text (multi-line forms stay covered by the AST guard
test from #38985). Together with the salvaged contributor commits this
retires the ~169-site bare file-I/O class (#37423's long tail).
Path.read_text() and Path.write_text() without explicit encoding
default to the system locale encoding. On Windows this is typically
cp1252, which causes UnicodeDecodeError for UTF-8 content (JSON
configs, user data, service scripts).
Add encoding="utf-8" to all read_text() and write_text() calls
across 8 CLI files, matching the pattern established in PR #50534
(security_audit_startup.py) and ruff rule PLW1514.
Fixed files:
- main.py: 4 read_text calls
- auth.py: 3 read_text calls
- banner.py: 1 read_text + 1 write_text
- service_manager.py: 1 read_text + 4 write_text
- container_boot.py: 1 read_text + 4 write_text
- doctor.py: 3 read_text calls
- uninstall.py: 2 read_text calls
- gateway.py: 1 write_text call
Follow-up on the #69500 salvage: 'launchctl submit' jobs remain
registered with launchd after they exit, so every plist reload leaked
one dead '<label>.reload.<pid>.<ts>' label. The helper script now ends
with 'launchctl remove' of its own transient label, and the recovery
test asserts the self-removal is present.
The deferred launchd reload helper used start_new_session=True to detach
from the gateway's process group. However, setsid(2) alone does NOT move
the child outside the launchd job's process coalition — when launchctl
bootout fires on the gateway label, launchd terminates ALL processes in
that coalition, including the setsid-detached helper, leaving the service
permanently unloaded.
Fix by spawning the helper via launchctl submit, which creates a
transient launchd one-shot job that is wholly independent of the
gateway's coalition. This ensures the helper survives bootout and can
complete the bootstrap+verify cycle.
Also writes a durable pre-bootout marker to the reload log so the
distinction between 'helper never started' and 'helper ran but
bootout/bootstrap failed' can be diagnosed.
Fixes#69098
- Strip the salvaged commit's inline encoding kwargs where main had since
gained its own (process_registry, local env, cua doctor, gateway,
commands, gateway_windows — the latter keeps its locale-aware
_schtasks_encoding() from #38186)
- Revert encoding kwargs mistakenly applied to non-subprocess APIs
(exa get_contents, tempfile.mkstemp in webhook.py)
- Guard the ddgs worker Popen (new on main since #55339)
- Update two kwarg-snapshot test assertions for the new kwargs
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.
Fixes#53428 (master tracker for Windows GBK locale crash).
Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
Extends the desktop backend's root-cause fix (aa2ae36c3f) to all remaining
console-less parent launch paths. The Windows console-flash class
(#54220/#56747) is governed by the PARENT's console: a DETACHED_PROCESS or
pythonw.exe daemon has no console, so every console-subsystem descendant
(git, gh, cmd, node, wmic, powershell) allocates its own visible conhost —
one flash per spawn, unreachable by any per-call-site CREATE_NO_WINDOW
sweep. Worse, MSDN specifies CREATE_NO_WINDOW is IGNORED when combined
with DETACHED_PROCESS, so the hide bit in the old detach bundle was dead.
Changes:
- _subprocess_compat: drop DETACHED_PROCESS from windows_detach_flags()
and windows_detach_flags_without_breakaway(); the daemon now owns a
single hidden console (CREATE_NO_WINDOW) that all descendants inherit.
- gateway_windows: _resolve_detached_python() returns the venv console
python.exe (no pythonw/base-interpreter detour — the uv-shim flash
premise only held while DETACHED_PROCESS was masking the hide bit);
UAC handoff launches console python under SW_HIDE; cmd/vbs launchers
render console python (vbs runs it window-style 0).
- gateway/run.py: restart watcher keeps sys.executable instead of
swapping in GUI-subsystem pythonw.
- web_server: dashboard actions spawn sys.executable (already carries
windows_detach_flags()).
Tests updated to pin the new invariants, including an explicit
DETACHED_PROCESS-must-stay-out regression guard.
The KeyboardInterrupt handler in run_gateway() was the only exit path
that still used bare 'return' instead of _hard_exit_after_gateway_teardown().
While less common than service-managed restarts, a console Ctrl+C still
leaves the process vulnerable to the same Python finalization hang on
non-daemon worker threads (cron ThreadPoolExecutor jobs). Route it through
the same backstop, with a 'return' guard for test stubs that don't raise
on code 0 (production os._exit never returns).
Verified: applies cleanly and the patched module compiles. Tests are
described in the PR body (not bundled in this commit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace REST-based Discord liveness probe with local WebSocket/heartbeat
state detection. REST success doesn't prove Gateway event delivery — a
half-closed WebSocket can leave Bot.start() alive while REST returns 200.
Now samples ready/open/ACK state and heartbeat latency; consecutive
unhealthy samples emit one retryable fatal code so GatewayRunner rebuilds
the adapter through the existing reconnect path.
Also fixes three lifecycle gaps in the recovery path:
1. asyncio.wait_for() can remain blocked if adapter cleanup swallows
cancellation — now uses bounded asyncio.wait() with task detachment.
2. Multiplexed secondary-profile adapters had no profile-scoped reconnect
owner — now uses one runner-owned reconnect slot per profile.
3. An in-flight turn could send its final text through the disconnected
adapter after a replacement was registered — now resolves the live
same-profile replacement for unsent final responses only (message IDs
never migrate, edits/deletes stay on the old transport).
Adds an opt-in Linux/systemd event-loop watchdog (gateway.systemd_watchdog_seconds,
default 0) for the failure mode where the whole asyncio loop stops making
progress and no in-process liveness task can run. stdlib-only sd_notify,
Type=notify/WatchdogSec generation, READY/STOPPING lifecycle.
Co-authored-by: 王鑫 <wx.xw@bytedance.com>
Follow-up to PR #66479 salvage. The three new env vars
(HERMES_LOCAL_STREAM_STALE_TIMEOUT, HERMES_GATEWAY_MAX_STARTS,
HERMES_GATEWAY_START_WINDOW_S) were introduced as bare env-var reads with no
config.yaml surface or documentation — violating the .env-is-for-secrets-only
policy (behavioral settings must live in config.yaml, bridged to env internally).
- config.yaml: add gateway.respawn_storm {max_starts, window_seconds} to
DEFAULT_CONFIG, mirroring the existing restart_loop_guard pattern.
- gateway.py: read config.yaml first, env vars override as escape-hatch.
- environment-variables.md: document all three new env vars, noting the
config.yaml alternative for the gateway ones.
- chat_completion_helpers.py: cross-reference the env-var docs from the
local stale timeout comment.
Re-applied against v0.18.2. Still unconditional KeepAlive <true/> with no
ThrottleInterval, and upstream's new restart_loop_guard only skips session
auto-resume (never throttles the boot), so the launchd respawn storm is
unguarded.
- launchd plist: add ThrottleInterval=30 + ExitTimeOut=25.
- status: record_start_and_check_storm — portable breaker (atomic gateway-starts.log,
backs off when too many starts land in a window); run_gateway sleeps it before
asyncio.run. Env HERMES_GATEWAY_MAX_STARTS (<=0 disables) / _START_WINDOW_S.
Separate file from upstream's restart_loop.json — no collision.
- tests: breaker (threshold/prune/atomic) + plist throttle keys.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to HexLab98's fix. The sync-before-regenerate invariant was
enforced by convention across 6 callsites (~3 idempotent unit-file reads
per command). Consolidate it into the one function every compare/regenerate
path funnels through — systemd_unit_is_current — and drop the now-redundant
callsite pre-syncs in refresh_systemd_unit_if_needed / systemd_start /
systemd_restart / systemd_status.
Kept the systemd_install pre-sync: the --force path bypasses the
is_current gate and calls generate_systemd_unit() directly, so it needs
its own sync to avoid baking /root/.hermes under sudo.
Reworked the two callsite-ordering tests into a chokepoint-invariant guard
(test_is_current_syncs_before_reading_unit) + a delegation test proving
start/restart no longer pre-sync. Both fail if the chokepoint sync is
removed; the pre-existing behavior test still passes.
Under sudo, start/restart refreshed the unit from /root/.hermes before
adopting the unit's pinned home, so TimeoutStopSec and env drifted and
status stayed stuck on "service definition is outdated".
The connector now depends on the single multiplexed gateway for per-profile
relay routing, so hosted deployments need to FORCE multiplexing on regardless
of the image's config.yaml. gateway.multiplex_profiles was config.yaml-only,
which a user could leave unset or flip off.
Add GATEWAY_MULTIPLEX_PROFILES as a standard operator override on top of the
existing config key — the same 'config.yaml is canonical, env is the operator
override' pattern the Telegram/Signal require_mention bridges use:
env (recognized token) > config.yaml (top-level or nested gateway.*) > False
- gateway/config.py: _env_multiplex_profiles_override() resolves the env var
tri-state — recognized truthy/falsy token → bool; unset/blank/unrecognized
→ None (fall through to config). Blank is deliberately None, not False, so a
provisioned-but-unpopulated Fly secret ('') can't shadow a config.yaml opt-in
(the empty-secret trap). Wired into GatewayConfig.from_dict so every consumer
(run.py, session.py via self.config) sees the resolved value.
- hermes_cli/gateway.py: the named-profile-start guard
(_guard_named_profile_under_multiplexer) reads config.yaml directly, so it
gets the SAME env precedence — otherwise env-forced multiplex would leave the
guard blind and someone could start a conflicting per-profile gateway that
double-binds a bot token. Env-forced-on trips the guard even with no
config.yaml key; env-forced-off disables it over a config opt-in.
Tests: full 3-tier precedence in test_config.py (incl. the discriminating
env-overrides-config cases + the empty/whitespace/unrecognized fall-through
trap + resolver tri-state), mutation-verified (flipping precedence fails
exactly the two env-wins tests); guard env cases in test_multiplex_lifecycle.py.
Force-on is safe on a single-profile instance: session keys stay byte-identical
(agent:main) and the _run_agent wrapper installs the per-turn secret scope, so
the fail-closed get_secret() path is satisfied.
ruff check --fix --select F541 . on current main. Pure prefix removals;
adjacent-string concatenations keep the f only on interpolating fragments.
No string content or live placeholder altered.
UV's bundled Python ships a minimal PATH that excludes /bin and /usr/bin,
causing launchctl/systemctl subprocess calls to fail with FileNotFoundError.
Fixes#3849
generate_systemd_unit() and generate_launchd_plist() used
Path(shutil.which('node')).resolve().parent to find the node bin dir.
When ~/.local/bin/node is a symlink into a specific profile's node
install (e.g. ~/.hermes/profiles/<p>/node/bin/node), .resolve() chases
it and bakes that one profile's path into EVERY profile's service
definition.
This breaks profile isolation and makes systemd_unit_is_current()
perpetually False: each gateway rewrites its unit + daemon-reload on
every boot, destabilizing multi-profile setups into a ~5-minute restart
loop (observed NRestarts ~1600 across two gateways).
Fix: use Path(resolved_node).parent — the directory where node is found
on PATH — instead of chasing the symlink to its resolved target. This
keeps generated service definitions profile-agnostic.
Affects both the systemd (Linux) and launchd (macOS) unit generators.
Reworks @valenteff's #53277 fix per review (Teknium's 3 findings):
- Route refresh_launchd_plist_if_needed's bootstrap through the existing
_launchctl_bootstrap() EIO-recovery helper (canonical since #56256),
wrapped in a wall-clock retry loop, instead of an ad-hoc 5x2s loop.
- Window sized to agent.restart_drain_timeout (default 180s), not a fixed
~10s: the failure happens while the old gateway is still draining (finding 1).
- Retry on subprocess.TimeoutExpired too, not just CalledProcessError — a
bootstrap timeout after bootout otherwise escapes and leaves the service
unloaded (finding 2).
- Confirm success with launchctl list, not a bare bootstrap exit 0 (finding 3);
mirror verify+drain-window in the detached-helper bash path.
- Shared helpers _launchd_reload_log_path / _append_launchd_reload_log /
_launchctl_label_registered / _retry_launchctl_bootstrap_until_registered.
3 new tests cover retry-until-listed, TimeoutExpired-retried, deadline-exhaust.
E2E: real reload log + mocked launchctl — retries CalledProcessError+TimeoutExpired,
verifies via launchctl list, logs failures.
refresh_launchd_plist_if_needed ran `launchctl bootout` then
`launchctl bootstrap` with errors silenced (`2>/dev/null` in the
detached helper, `check=False` in the direct subprocess path).
Under high load or a launchd race, the bootout succeeds — removing
the service from launchd — but the follow-up bootstrap fails
silently. The service stays unregistered; KeepAlive can't revive
a service launchd no longer knows about, so the gateway stays dark
until a manual `launchctl bootstrap`.
Observed incident (2026-06-26): `/restart` in chat triggered a
planned drain; during the drain a separate call re-triggered the
plist refresh, which bootout'd the live service. Under loadavg
9.48 the bootstrap failed silently — 2h35min offline until manual
recovery.
Fix: retry the bootstrap up to 5 times with 2s back-off, verify
with `launchctl list <label>` afterwards, and log failures to
~/.hermes/logs/launchd-reload.log so the health watchdog can
detect a persistent orphan. Mirrors the contract across both
the detached helper (refresh inside gateway tree) and the direct
subprocess path (refresh from external CLI).
Existing tests pass:
- test_refresh_defers_reload_when_running_inside_gateway_tree
- test_refresh_uses_direct_reload_when_not_inside_gateway_tree
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On macOS, `launchctl bootstrap` of a label still registered in the domain
fails with 5: Input/output error (EIO). That is the *already loaded* case — a
stale registration from an interrupted restart or a bootout that didn't settle
— recoverable by booting the leftover out and bootstrapping again, and distinct
from the domain being genuinely unmanageable.
launchd_install and launchd_start (both bootstrap paths) treated exit 5 as
'launchd cannot manage this macOS version' and silently degraded to a detached
process, losing auto-start at login and crash-restart. Centralize bootstrap in
_launchctl_bootstrap(), which on EIO boots the stale label out and retries once;
only if the retry also fails does the error propagate so callers apply their
existing _launchctl_domain_unsupported fallback for a genuinely broken domain.
launchd_restart already boots out before bootstrapping (its drained job is
almost always still registered, so a plain bootstrap would hit EIO on the common
path), so it keeps its explicit pre-bootout rather than routing through the
bootstrap-first helper. Corrected the stale exit-5 comment that claimed it
always meant an unmanageable domain.
Adds TestLaunchctlBootstrapEioRetry covering clean bootstrap (no bootout),
EIO -> bootout -> retry success, persistent EIO re-raise, and non-EIO re-raise
without a spurious bootout.
The Windows desktop GUI runs its backend headless via pythonw.exe. Several
auxiliary subprocess sites that run inside that windowless backend spawned
console-subsystem children (git, gh, wmic, powershell, bash, rg, taskkill)
WITHOUT CREATE_NO_WINDOW, so Windows allocated a fresh conhost per call and
flashed a black window on screen — sometimes continuously (the dashboard
Projects-tree git probe alone fired ~118 spawns in 60s on startup).
The terminal tool, cron, browser, code_execution, and gateway-spawn paths
already carry windows_hide_flags(); these auxiliary probe/scan/launcher legs
were missed. Wire the existing helper into them:
- tui_gateway/git_probe.py: run_git (+ encoding=utf-8/errors=replace, fixes the
cp950 UnicodeDecodeError on CJK paths from the same site)
- agent/coding_context.py: _git (per-turn git status/log/diff)
- agent/context_references.py: _run_git + _rg_files (@file/@ref resolution)
- hermes_cli/copilot_auth.py: gh auth token probe (auxiliary provider:auto)
- hermes_cli/gateway.py: wmic + PowerShell Get-CimInstance PID scan
- hermes_cli/main.py: wmic stale-dashboard PID scan
- gateway/status.py: taskkill /T /F force-kill
windows_hide_flags() returns 0 on POSIX, so every changed call is a no-op on
Linux/macOS (verified: real git/rg probes still work; Windows-simulated calls
all pass creationflags=CREATE_NO_WINDOW).
Scoped to the windowless-backend paths that cause the reported flashing. The
Electron updater-handoff leg (main.cjs windowsHide:false) and the
interactive-CLI banner probes (cli.py) are intentionally NOT touched here —
the former needs a Windows-tested change of its own, the latter runs in a
visible console anyway.
Tracking: #54220
Refs: #53178#53631#53781#53957#49602#52982#53424#53053#53016
launchd restart can leave the gateway job stopped but still registered after
update-time drain logic, so a direct bootstrap hits exit 5 and falls back to a
detached process. Booting the stale registration out before bootstrap keeps the
launchd-managed restart path intact and locks it with a regression test.
Constraint: Keep upstream-facing conventional commit style while preserving local decision context
Rejected: Treat bootstrap exit 5 as expected | Leaves macOS launchd restart outside launchd supervision after update
Confidence: high
Scope-risk: narrow
Directive: Keep launchd start/restart recovery flows aligned when changing launchctl handling
Tested: pytest -q tests/hermes_cli/test_gateway_service.py -k "launchd_restart_boots_out_stale_registration_before_bootstrap or launchd_restart_falls_back_to_detached_on_error_5 or launchd_restart_drains_running_gateway_before_kickstart or launchd_restart_self_requests_graceful_restart_without_kickstart"
Tested: pytest -q tests/hermes_cli/test_gateway_service.py -k launchd
Not-tested: Manual macOS launchctl restart after hermes update
Folds in PR #42124 (kyssta-exe): systemd_install gained a non_interactive
flag so the 'Remove the legacy unit(s)?' prompt — the second hidden prompt
not guarded by --start-now/--start-on-login — is also skipped in headless
contexts. Updates systemd_install test mocks to accept the new kwarg and
adds coverage for the legacy-unit-skip path.
When running `hermes gateway install` on Linux/systemd, the command
unconditionally prompts with two `prompt_yes_no` questions, breaking
headless installs (SSH, CI, provisioning scripts) and ignoring the
existing --start-now / --start-on-login CLI flags that the Windows
branch already respects.
The fix mirrors the Windows path: read CLI flags first, prompt only
when flags are not provided AND stdin is a TTY, and fall back to True
defaults for non-TTY contexts. The argparse help strings are promoted
from SUPPRESS to visible so users can discover the flags.
Fixes#42065
Long-lived helpers spawned indirectly by tool calls (adb, platform
bridges) were left in the service cgroup after the gateway's main
process exited. When the kernel rejected the deferred cgroup-wide kill
with EINVAL, systemd blocked Restart=always for 6+ minutes, taking
down all platforms and cron windows (#37454).
Add a small ExecStopPost helper (gateway.cgroup_cleanup) that walks
cgroup.procs and sends per-PID SIGKILLs — a different kernel code path
than cgroup.kill, so it succeeds where the cgroup-wide write failed.
KillMode=mixed is preserved so the gateway still reaps its own
tool-call children before systemd intervenes (#8202).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Non-root users picking 'System service' in the setup wizard were handed a
'sudo hermes gateway install --system --run-as-user <you>' recipe that fails
on most distros: sudo's secure_path strips ~/.local/bin (pipx/uv installs),
so 'sudo hermes' is command-not-found. Worse, it funnels a non-root user
toward a system install they shouldn't be doing from a user session.
Now prompt_linux_gateway_install_scope() only offers system scope when
os.geteuid()==0. Non-root sessions get user-service or skip, with a tip to
re-run as root for a boot service. The non-root branch in
install_linux_gateway_from_setup becomes a defensive guard that refuses
without printing any self-elevation recipe. Gated the matching deferral hint
in setup.py behind root too.
* fix(windows): stop terminal-window popups from background spawns
Native-Windows desktop/gateway users saw cmd/conhost windows flash on
gateway restart, image paste, the dashboard Projects tree, voice notes,
and ~5 min after closing the app (detached cron). Two root causes:
- Console-subsystem exes (taskkill, schtasks, wmic, netstat, tasklist,
agent-browser, git, ffmpeg, powershell, git-bash) spawned via raw
subprocess allocate a fresh console when the launching process has
none (pythonw desktop backend / detached gateway) - even with output
captured.
- uv venv pythonw shims re-exec console python.exe, so Python children
get a console regardless of how they're launched.
Fixes:
- Single hidden-spawn primitive (_subprocess_compat.run/.popen) that ORs
CREATE_NO_WINDOW on Windows, no-op on POSIX. Route every Hermes-owned
console-exe spawn through it.
- FreeConsole() catch-all in hermes_bootstrap: any Python child that
exclusively owns an auto-allocated console detaches it at startup
(GetConsoleProcessList()==1 gate leaves shared interactive consoles
untouched).
- Replace PowerShell/wmic gateway PID scans with in-process psutil.
- Skip schtasks queries on non-interactive desktop restarts.
- Prefer native agent-browser .exe over .cmd shims.
- Guard test bans raw subprocess spawns of the Windows-only console
tools repo-wide so the popup class can't regress.
* fix(windows): scope FreeConsole to background entry points; fix merge fallout
Console detach review (per #53810 feedback): GetConsoleProcessList()==1 can't
tell a uv pythonw->python phantom console apart from a user opening the
interactive CLI/TUI in its own fresh console (double-click, shortcut, ConPTY) —
both report a single attached process with a tty. Running FreeConsole() in the
import-time bootstrap therefore risked detaching a legitimately-interactive
terminal.
- Extract FreeConsole into explicit hermes_bootstrap.detach_orphan_console();
remove it from apply_windows_utf8_bootstrap() (import side effect).
- Call it only from known background mains: gateway run, dashboard backend
(start_server, what the desktop spawns), cron standalone, tui_gateway entry,
slash worker. Interactive CLI/TUI never calls it.
- Behavior-contract tests: frees only when solo owner, leaves shared console,
no-op without console / on POSIX, and asserts it's not an import side effect.
Merge fallout from origin/main (#53791):
- local.py: 3-way merge left a dangling **_popen_kwargs (NameError crashing
every terminal init). _subprocess_compat.popen already hides the window, so
drop it.
- discord adapter: merge stacked an undefined windows_hide_flags() onto the
primitive call; drop the redundant arg.
- test_gateway: scan now goes psutil-first (zero spawn); rewrite the
case-variant test to drive that production path.
* test(claw): mock _subprocess_compat.run seam for Windows process scan
claw.py's Windows tasklist/powershell scan routes through the hidden-spawn
primitive; the tests still patched claw_mod.subprocess, so on win32 the mock
was never hit and real spawns returned nothing. Patch the actual seam.
* fix(windows): stop subprocess console-window popups + add CI guard
The single biggest source of Windows 'terminal popup' bug reports was bare
subprocess.run/Popen calls spawning a console window. The compat helpers
(windows_hide_flags / windows_detach_popen_kwargs) already existed but the
footgun checker had no rule to stop new bare calls from reintroducing the flash.
- scripts/check-windows-footguns.py: new AST-based rule flagging subprocess
calls that can create a new console — output-redirection-aware (capture/
redirect/check_output exempt) and POSIX-only-program-aware (launchctl/
systemctl/brew/etc. exempt). Comprehensive on real popups, no annotation
burden on calls that can't flash.
- Swept all genuine window-spawning sites through windows_hide_flags()/
windows_detach_popen_kwargs(); marked intentionally-visible launches
(editor/terminal/foreground re-exec) with '# windows-footgun: ok'.
- tests/scripts/test_windows_footgun_subprocess_rule.py: behavior-contract
tests + full-repo cleanliness invariant.
- CONTRIBUTING.md: documents the rule + the helper pattern.
* test: accept creationflags kwarg in psutil_android fake_subprocess_run
The Windows no-window sweep added creationflags=windows_hide_flags() to
install_psutil_android.py's subprocess.run call; the test's fake stub had a
fixed (cmd) signature and raised TypeError on the new kwarg.
kshitij
Rob Hilgefort
HexLab98
Kshitij Kothari
RelaxJonh
Brooklyn Nicholson
ethernet
Jeff Watts
teknium1
annguyenNous
webtecnica
Stoltemberg
web3blind
Cossackx
StellarisW
Burke Autrey
embwl0x
kshitijk4poor
Ben Barclay
Kevin Rajaram
Jack Earnest
Fabio Fernandes Valente
HiaHia
izumi0uu
liuhao1024
PRATHAMESH75