The reload retry loop treated `launchctl list <label>` exit 0 as success,
but exit 0 also covers a registered-but-not-running definition (macOS 26+
`state = not running`) — the same trap _probe_launchd_service_running
already guards against. Require a PID so success means launchd is
supervising a live process, in both the Python loop and the shell helper.
Verified against live launchd: a RunAtLoad=false job reports exit 0 with
no PID, which the old check accepted and the new one rejects.
Note this is NOT what distinguishes a draining instance — measured, the
label deregisters within ~1s of bootout while the old process drains on.
Waiting for the old PID to exit is what covers that.
Reload chose the in-process bootout/bootstrap path based on POSIX
ancestry, but bootout tears down the job's process coalition, and
coalition membership is inherited at spawn and survives reparenting.
A gateway-spawned process reparented to PID 1 is no longer an ancestor
yet still dies with the coalition, so the retry loop was killed
mid-bootstrap and nothing re-registered the label (KeepAlive can't
revive a job launchd no longer knows about).
- always prefer the detached transient-job helper; it's also correct
when genuinely outside the coalition, just asynchronous
- wait for the old gateway PID to exit before bootstrapping; bootout
only sends SIGTERM and every bootstrap during the drain fails EIO
- fall through to the in-process path when the helper can't spawn
instead of leaving the plist rewritten but never reloaded
Two live failures on the same guard (cron/lifecycle_guard.py), both of
which blocked legitimate diagnostics from inside the gateway:
1. Crash class: the referenced-script walk read compiled binaries as if
they were shell scripts. Reading/inspecting a referenced file is now
best-effort by construction: executable magic numbers (ELF, PE,
Mach-O fat/thin) short-circuit before any full read via a 4KB sniff,
NUL-bearing heads are skipped as non-scripts, and unreadable paths of
every kind (NUL bytes in the token, ENAMETOOLONG, missing files)
degrade to "nothing to scan" instead of raising. A second fail-safe
layer wraps the pure-string fallback so the boundary function stays
total even if the tokenizer itself fails.
2. False-positive class: the lifecycle regex matched its command shapes
inside DATA arguments — SQL string literals passed to sqlite3/psql
and grep/rg/journalctl patterns hunting for the lifecycle string in
logs. Added a fail-closed second-pass exemption: on a raw regex hit,
re-scan with data-sink executables' arguments masked; only a match
that survives (i.e. sits in command position) blocks. Masking is
skipped for pipes into shells/xargs, command/process substitution,
sqlite3 dot-commands and psql backslash escapes, so it can only ever
allow, never miss.
Behavioral tests: exact live false-positive shapes as negatives, the
smuggling shapes as positives, the kill-primitive positive catalog
unchanged, and an adversarial never-raises suite (NUL bytes, non-UTF-8,
/dev/*, directories, missing files, magic-prefix binaries).
3-reviewer simplify pass (reuse/quality/efficiency) findings:
- cron/scheduler.py _run_job_script: the ORIGINAL that
lifecycle_guard._resolve_script_path documents mirroring had the exact
same unguarded expanduser() — a NUL-bearing script value survives
creation (the guard treats it as nothing-to-scan) and crashed the
scheduler at fire time with ValueError instead of a clean job failure.
Same ingestion contract applied; regression test added.
- lifecycle_guard._resolve_script_path: get_hermes_home() -> Path.home()
raises RuntimeError when neither HERMES_HOME nor HOME resolves
(arbitrary-UID containers); the cron entry point called it bare.
Caught -> None; totality test added.
- terminal_tool: stale 'cat ...' docstring updated to the bounded
head -c form.
- lifecycle_guard: dead 'script_text and' condition dropped (guarded by
'if not script_text: continue' directly above).
Efficiency reviewer: no material findings (measured — encode/expand
costs negligible vs walk I/O, no timing regression vs base).
The guard feeds untrusted byte streams (tokenized binaries, remote cat
output) into OS-path and shell-text operations; every incident so far
(#76762, #77703, #77780, #78256, #77729) was hot-fixed with an except at
whichever frame crashed that week. tilllt's regression suite on #79454
showed 4 members of the class still open on merged main. Close the class
at three boundaries instead:
- _expand_candidate_path(): single ingestion chokepoint for path
candidates — reject NUL/empty tokens before any Path OS call and
tolerate ValueError/RuntimeError/OSError from expanduser (T1/T2, plus
the HOME-unset launchd crash). Both _resolve_terminal_script_path and
_resolve_script_path now go through it.
- _sanitize_remote_script_text(): apply the local-read contract (NUL =
binary = nothing to scan; >1MiB = fail closed) to whatever any
read_remote_script callback returns, at the recursion boundary — the
guard stops trusting its callbacks (T3/T4).
- contains_gateway_lifecycle_command_or_referenced_script() is now total
by construction: direct regex scans (pure string ops) run first; the
best-effort filesystem walk is wrapped so an unexpected failure logs a
warning and falls back to the direct-scan verdict instead of killing
every terminal command until gateway restart.
terminal_tool's remote fallback also bounds the read at the source
(head -c 1MiB+1 instead of cat), so a 166MB ELF never crosses the wire —
the superlinear-shlex 30-minute stall from #79838's field report drops
to a 0.02s fail-closed verdict.
Regression tests: tilllt's T1-T4 adopted verbatim, plus an adversarial
never-raises sweep (NUL paths, unset HOME, over-long paths) and a
walk-crash fallback test.
/heartbeat every <interval> <prompt> gives the current session one
recurring instruction. When the session is idle and the interval has
elapsed, the prompt is injected as a plain user turn — same
conversation, same context, prompt cache and role alternation
untouched.
- CLI: idle-poll watchdog thread (wake-word watchdog pattern) feeding
_pending_input; gateway: single gateway-wide async poller injecting
through the adapter FIFO. Busy sessions coalesce their tick to the
next idle poll.
- Missed ticks coalesce (anchor resets on fire) — a busy hour yields
ONE heartbeat turn, never a backlog. Real user messages always win.
- 60s interval floor; injected prompt carries a don't-invent-work
guard so idle heartbeats don't generate busywork.
- State persists in SessionDB.state_meta (heartbeat:<session_id>),
survives /resume, migrates across compression session rotations
alongside /goal state.
- Session-scoped and in-process by design — durable cross-process
schedules remain the cron subsystem's job (docs draw the boundary).
- Slack stays under the 50-slash cap via /hermes heartbeat; ghost-text
suggester now prefers the shortest prefix match so /he still
suggests /help.
Adapted from the session-heartbeat concept in Prime Intellect's
Prime-Agent (/heartbeat).
/goal gate add <command> attaches shell commands to the active goal.
Gates run at turn boundary BEFORE the LLM judge: a failing gate skips
the judge entirely and feeds its exit code + bounded output tail back
as the continuation prompt, so the agent iterates against concrete
evidence instead of a prose verdict.
- Unchanged-workspace skip: a gate that failed on an identical
workspace (git HEAD + status fingerprint) is not re-run — the
recorded failure replays and the attempt count advances.
- Bounded retries (default 3) + per-gate timeout (default 300s);
exhaustion auto-pauses the goal like the turn budget does.
- Gates persist in SessionDB.state_meta with the goal (survive
/resume and compression rotation); pre-gate goal rows load
unchanged.
- /goal gate [list|add|remove|clear] on CLI + gateway; 'gate' added
to the mid-run control-verb whitelist (gates only run at turn
boundary, so editing the list mid-run is safe).
Adapted from the quality-gate concept in Prime Intellect's Prime-Agent
(--autonomous-gate).
Residual #76762 class: _read_referenced_script caught OSError from
os.open but not ValueError, so a path token carrying an embedded NUL
(tokenized binary-adjacent command text) crashed the terminal tool's
lifecycle guard with 'ValueError: embedded null byte' instead of being
skipped as nothing-to-scan. Reproduced live against main. Same
treatment the resolve()-time site already has; two sabotage-verified
regressions added.
Both files were routed through the shared atomic writers earlier in this
branch. tempfile.mkstemp creates the temp file 0600 and the atomic swap
carries that mode onto the target, so the *create* paths silently tightened
two files that previously landed at the umask default:
- web_routers/profiles.py: the dashboard persona editor's first-ever Save has
no prior SOUL.md to copy permissions from, so the existing guard skipped the
chmod entirely -- contradicting the comment directly below it, which states
profile SOUL.md is created 0644 and is not run through _secure_file.
- profile_distribution.py: atomic_yaml_write only restores a mode it captured
from a file that already existed. _materialize() calls write_manifest() with
no manifest on disk whenever a distribution declares an explicit
distribution_owned allowlist that omits distribution.yaml, so the staged
copy is never placed in the profile.
Both are fixed with a local chmod at the two sites this branch regressed;
utils.py's public mode semantics are left alone. profiles.py now also
distinguishes "no file yet" (FileNotFoundError -> 0644) from "stat failed for
some other reason" (-> leave the mode alone rather than guess at it).
uninstall.py and xai_retirement.py have no create path and are unchanged: the
former captures prior_mode unconditionally after a successful read_text(), and
the latter runs require_readable_config_before_write() first.
The four new mode-preservation guards assert POSIX permission bits, which
Windows does not model (os.chmod only toggles the read-only flag there).
Guard them the way the suite already guards POSIX-specific semantics so the
tests stay meaningful on Linux/macOS without failing for Windows contributors.
The symlink cases stay unguarded, matching the existing symlink tests in
tests/hermes_cli/.
`utils.atomic_write_text`'s docstring states the invariant: it exists "so that
every destructive file rewrite in the codebase shares one implementation."
Four full-file rewrites of *existing user-authored files* still bypass it and
use a bare truncating `open(path, "w")` / `Path.write_text()`, which truncates
the target before the new content is produced. A crash, SIGINT, or ENOSPC
mid-write therefore leaves the file empty or half-written.
In all four cases the read half degrades silently to a default rather than
erroring, so the damage is invisible and the next write cements it:
* `xai_retirement.apply_migration()` rewrites the user's config.yaml. Merged
commit beaa1a08e added a readability guard here and noted the writer "lives
outside the atomic_yaml_write path, so the chokepoint didn't cover it"; this
closes the durability half it left open. `--no-backup` is a documented flag,
so on that path the truncated file is the only copy that exists, and the
loader returns early on `doc is None` — the next run reports nothing to
migrate rather than surfacing the damage.
* `uninstall.remove_path_from_shell_configs()` rewrites the user's shell rc
(~/.bashrc, ~/.zshrc, ...). Hermes does not own these files and this function
takes no backup; the enclosing `except Exception` downgrades a partial write
to a warning, so the next login just starts a bare shell.
* `web_routers.profiles.update_profile_soul()` replaces SOUL.md from the
dashboard editor. The paired GET reports an unreadable file as
`{"content": "", "exists": False}`, so an interrupted save presents as "your
persona was never set" and the editor's next Save persists the empty document.
* `profile_distribution.write_manifest()` rewrites distribution.yaml on every
install/update. `read_manifest` treats an unparseable manifest as "not a
distribution", silently dropping update tracking and env_requires.
The xAI migration keeps its ruamel round-trip dumper (comments, key order and
quoting must survive) and now serializes to a string before handing the bytes
to the shared writer. `write_manifest` moves to `atomic_yaml_write`, whose
SafeDumper output the manifest already round-trips through, retiring the local
`_dump_yaml` helper.
`atomic_write_text` recreates the target from a 0600 temp file, so each of its
call sites re-applies the file's previous permission bits: `_secure_file`
deliberately leaves config.yaml alone under managed (NixOS 0640) and container
installs, shell rc files are normally 0644, and profile SOUL.md is created 0644
and never secured. `atomic_yaml_write` already preserves mode and owner itself.
Routing through `atomic_replace` also keeps a symlinked config.yaml or ~/.zshrc
(dotfiles repo, managed deployment) pointing at the real file.
Tests: one regression test per site fails on clean main (the interrupted write
completes there and destroys the file) and passes here; the remaining cases are
behaviour guards covering symlink survival, permission preservation, comment
round-tripping, and the existing happy paths.
- fetch_models(): accept base_url kwarg (interface grew on main since May)
- runtime_provider: config-driven loopback base_url now reaches the local
no-auth placeholder before the usable-secret gate (added on main in the
interim, would otherwise AuthError on keyless local setups)
- test: fetch is now called with base_url by the generic live-fetch path
The dashboard now mints an action_id per backend update, hands it to the
spawned `hermes update` via HERMES_ACTION_ID, and reuses an in-flight
update action instead of spawning a duplicate. The updater prints a
bounded `=== hermes-update completed <id> ===` receipt on every success
path — normal, zip, dependency-repair, and the no-op "Already up to
date!" path that previously ended with no terminal marker at all
(#58764) — so the Desktop can prove completion across the dashboard
restart boundary instead of guessing from stale log text.
Co-authored-by: Vitor Cepeda Lopes <vitor@vitorcepedalopes.com>
Co-authored-by: doncazper <caztronics@yahoo.com>
On Linux a freshly-built desktop app had no presence in the application
launcher: no Hermes in the KDE/GNOME menu, no icon, nothing to pin. Users
had to hand-write ~/.local/share/applications/hermes.desktop and remember
to reindex the menu caches themselves.
`hermes desktop` now writes that entry itself (best-effort, idempotent,
never blocking a launch), and `hermes uninstall --gui` removes it again.
Both fields that matter are absolute:
- Exec — the launcher runs with a minimal environment and no shell PATH
customizations, so a bare `hermes desktop` silently fails for anyone
whose hermes lives in ~/.local/bin or a venv. We resolve the real binary
via relaunch.resolve_hermes_bin(), falling back to an absolute
interpreter + `-m hermes_cli.main`.
- Icon — an unqualified name only resolves against an indexed icon theme,
which we are not in. The spec allows an absolute path, so we point at
apps/desktop/assets/icon.png in the checkout. No copy is installed: Exec
already depends on that same tree, so a second copy would add bytes and
an uninstall step without surviving anything Exec wouldn't.
Menu-cache refresh is tool-gated — update-desktop-database, then
kbuildsycoca6 or kbuildsycoca5 — each only when the binary is actually on
PATH, because most desktops ship none of them and a missing one is not an
error. The entry is only rewritten when its contents change, so a launch
doesn't churn the caches every run.
Verified on NixOS: the generated entry passes desktop-file-validate, a
real kbuildsycoca6 on PATH is invoked with --noincremental, a real
update-desktop-database writes mimeinfo.cache, absent tools are skipped
cleanly, and removal leaves the checkout's icon untouched.
The gateway terminal guard crashed with 'ValueError: embedded null byte'
(command never ran, exit_code -1) when a command invoked an ELF binary by
full path. _read_referenced_script correctly rejects the binary locally
(NUL in first chunk), but the read_remote_script fallback
(_read_script_in_env) then re-read the SAME file's bytes without a NUL
guard, decoded them, and fed machine code back into the scanner, which
re-tokenized it into a bogus NUL-bearing path and crashed at os.open.
- _read_script_in_env: skip content containing a NUL byte on both the
local-read and remote-cat branches (mirrors _read_referenced_script:
a binary is nothing to scan), so binary never re-enters the guard.
- _read_referenced_script: tolerate ValueError from os.open on a
NUL-in-path, alongside the existing OSError guard, so the guard can
never crash the terminal tool regardless of input.
Extends the #76762 NUL-safety fix (local path only) to the gateway's
remote-read fallback path.
_skin_set has no try/except around yaml.safe_load, so invalid YAML raises
and aborts the command. The {} fallback comes only from safe_load()
returning None on a zero-length file — which is exactly the state a torn,
unsynced write leaves behind, so the data-loss chain is unchanged.
`write_profile_meta` and `hermes skin set` are both read-modify-write
helpers that rewrite a user-visible YAML file with a bare truncating
write, bypassing `utils.atomic_yaml_write` — the shared helper whose
docstring states that "every destructive file rewrite in the codebase
shares one implementation".
Both read halves swallow a parse error and fall back to `{}`, so a
truncated file is not transient corruption. The next call reads `{}` and
silently, permanently drops every field the caller did not explicitly
pass:
* `write_profile_meta` promises "unspecified fields preserve existing
values". After an interrupted write, a follow-up call that only sets
`description_auto` erases the profile's `description` — it vanishes
from `hermes profile list` and never comes back.
* `_skin_set` exists so that "changing one token never disturbs the rest
of the look". `path.write_text(...)` neither fsyncs nor swaps
atomically, so a crash or power loss can leave `<skin>.yaml`
zero-length; the next tweak then rewrites from empty and the whole
palette is gone. The gateway's skin watcher repaints live surfaces
from this file within ~1s, so a half-written file is observable.
Routing both through `atomic_yaml_write` gives temp file + fsync +
`atomic_replace`, which also preserves a symlinked target (GitHub
#16743) and restores owner/mode, and emits emoji descriptions as real
UTF-8 instead of `\UXXXXXXXX` escapes (GitHub #51356).
Supersedes #51808, which fixed the unicode-escaping symptom alone by
adding `allow_unicode=True` to the same `yaml.safe_dump` call.
A dispatched console handler that calls sys.exit("message") or
raise SystemExit("message") sets exc.code to a string. int(exc.code or 0)
then raises ValueError, which is not a ConsoleCommandError, so it escapes
execute()'s handler and crashes the local REPL on an ordinary user mistake
(e.g. removing a credential that does not exist). Treat a string exit code
as a status-1 failure carrying that message.
Timing out a bounded git probe must not leave helper descendants
(credential helpers, git-remote-https, hook children) running after the
probe fails open. bounded_git_probe now spawns the child in its own
process group on POSIX (process_group=0), and _kill_git_process_tree
signals the whole group with os.killpg — gated on the child actually
leading its own group (pgid == pid), so a shared-group spawn can never
take down unrelated processes. Windows keeps the existing taskkill /T /F
tree kill.
Proven live on main: a fake git that forks a 300s descendant left the
descendant running after the probe timeout; with the fix the descendant
dies with the launcher. Fast path and fail-open contract unchanged.
Port of openai/codex#36793 (Terminate timed-out Git process trees).
Selecting an NVIDIA NIM model whose id reached config without the nvidia/
prefix produced a bare "HTTP 404: 404 page not found" — retried three times,
never naming the model. It reads exactly like an outage or an auth failure,
which is where the Discord thread spent its time before the id was spotted.
normalize_model_for_provider() had no branch for nvidia, so a bare id passed
straight through to the API. Repair it from the provider's curated catalogue:
a bare name that matches exactly one entry modulo the prefix gets it back.
That's a lookup, not a guess — build.nvidia.com also fronts local NIM
containers and third-party models, and anything absent from the catalogue is
left alone. Because the repair runs on every runtime setup, an already-broken
config self-heals on the next turn and prints what it changed.
If a bare id still reaches the wire, the 404 now explains itself. The
classifier consults the same catalogue: a prefix-less id the provider only
serves as vendor/model is a deterministic failure, so it classifies as
model_not_found instead of burning three retries on a retryable "unknown",
and the error trace names the id to use.
Fixes#78796
hermes debug share runs on the backend. A desktop app connected to a
remote, docker, or SSH backend writes desktop.log on the client machine,
so the bundle can never contain it — and the report rendered that as a
bare "(file not found)", which reads as "the app logged nothing" and
sends triage after a client-side bug it cannot see.
Name the writer and the path to collect by hand. Backend-written logs
are unchanged, a present desktop.log is still captured, and an empty one
still reports "(file empty)" — the app ran and logged nothing is a
different fact from the file being on another host.
Teknium's review on #63457: existing tests pin the final boolean but not
that the slow PROVIDER_REGISTRY sweep is skipped. Add three tests that
booby-trap hermes_cli.auth.get_auth_status and verify
_has_any_provider_configured() short-circuits on:
- config.yaml model.provider
- config.yaml base_url/api_key (custom endpoint shape)
- auth.json active_provider (sweep-only call-pattern guard)
Mutation-checked: reverting the reorder makes all three fail.
test_custom_provider_context_avoids_false_shrink_warning used
qwen3.8-max-preview as a slug that deliberately falls through to the
generic 'qwen' 131K catalog match. The new qwen3.8-max
DEFAULT_CONTEXT_LENGTHS entry (1M) now substring-matches the preview
slug too, so the no-custom-providers branch stopped warning. Swap the
fixture to qwen3.9-max-preview, which still hits the generic fallback
— the test's intent (custom_providers threading) is unchanged.
Simplify-pass follow-up on the #70782 salvage: _bearer_auth_headers
hard-emits ${MCP_<NAME>_API_KEY} but install_entry only persists
auth.env-declared vars — a manifest naming its key differently (the
shipped n8n style) would install cleanly yet send a literal-placeholder
header at connect time (silent 401, the #37792 bug class). Enforce the
naming contract at parse time. Also pins the secret-stays-in-.env
property in the install test (raw config.yaml carries the template,
never the secret). Mutation-checked: validation disabled -> guard test
fails.
When an optional-mcps manifest declares transport.type=http with
auth.type=api_key, install_entry() prompts for the key and saves it to
.env, but _build_server_config() only handled the oauth case — the
api_key case produced a bare url entry with no headers, so every
request to the server was unauthenticated (-> 401).
Reuse _bearer_auth_headers(entry.name) from mcp_config.py so the
catalog path emits the same 'Authorization: Bearer ${MCP_..._API_KEY}'
template as the manual 'hermes mcp add --url' path.
Salvaged from #70782 (production hunk applied clean; tests re-anchored
onto current main). Credit: JonthanaHanh.
Three fixes for the Desktop/TUI cold-start stall where the event loop
is blocked for ~14s between HERMES_BACKEND_READY and the first
prompt (#60800):
1. copilot_auth: skip subprocess fallback when any
Copilot env var is explicitly set (even if invalid). The user
expressed token intent via env var; silently substituting a CLI
token is surprising and the subprocess adds up to 5s on Windows.
2. tui_gateway/ws: run resolve_skin() via asyncio.to_thread so config
loading + skin engine init do not block the WS read loop during
the cold-start RPC burst.
3. web_server: extend _warm_gateway_module to pre-import the heavy
module chains (auth, copilot_auth, runtime_provider, skin_engine,
inventory, model_switch) that the first WS connection + RPC burst
would otherwise import on the loop thread. These trigger .pyc
compilation and Defender scans on Windows (15-30s per the existing
comment) and were not covered by the original gateway-only warm.
Tests: 5 new tests in test_cold_start_gil_stall.py + 2 new tests in
test_copilot_auth.py. All 36 copilot_auth tests + 16 ws/web_server
tests pass.
rob-maron
Rob Hilgefort
Teknium
kshitij
briandevans
Justin Bennington
Jeffrey Quesnelle
Brooklyn Nicholson
ethernet
PRATHAMESH75
golldyck
Alex Fournier
HexLab98
Bryan Bednarski
kshitij
Hao Wang
JonthanaHanh
jinglun010
joncaldwell90
embwl0x