Four fix-forwards from the adversarial post-merge audit of the Aug 7
unreviewed merge batch:
- estop (#81148): is_engaged() now fails SAFE (engaged) on stat errors;
the gateway estop gate lets recognized slash commands and replies owned
by in-flight work (update prompts, clarify, slash-confirm, tool
approvals, running sessions) through instead of consuming them; new
gateway /pause [reason|off] command gives messaging-only operators an
in-band engage/resume path (busy_policy=dispatch so it works mid-run).
- cron monitor mode (#81138): execution-mode invariants (monitor x
no_agent, monitor_script x monitor_url, no_agent-requires-script) now
have ONE owner (_validate_job_mode_invariants) called from BOTH
create_job and update_job, so the create-time invariant can no longer
be silently violated through the update door.
- cron notepad (#81139): remove_job now clears the job's notepad rows
(clear_notepad was dead code -> orphaned KV state forever); clear is
best-effort and no-ops without creating notepad.db.
- delegation batch gate (#81141): template-marker regex narrowed to
multi-word placeholder shapes only (<feature name>, {file_path}) so
generics (Vec<T>), HTML tags, JSON snippets, glob braces and f-string
style no longer reject legitimate batches; duplicate-goal rejection
removed (best-of-N fan-outs are legitimate).
The salvaged bytes path (_extract_anydoc_bytes) bypassed the coverage
check added in #81680. Materialize transferred PDF bytes in a host temp
file for the pdftotext scan, and name the backend-visible path in the
recovery command rather than the temp file.
Video (plugins/video_gen/fal): Seedance 2.5, MiniMax H3, Seedance 2.0
Mini, FLUX 3, Grok Imagine 1.5, Gemini Omni Flash (i2v-only). New
family capability flags:
- duration_int: endpoints that take duration as a JSON integer
- resolution_aliases: maps 720p/1080p-style values onto non-standard
enums (H3's 768P/2K/4K)
- image_drop_keys: strips keys the family's i2v endpoint rejects
(aspect_ratio on Seedance 2.5 / H3 / Grok 1.5)
Image (tools/image_generation_tool): Seedream 5.0 Pro (+edit) and
Lite, Ideogram V4 instant + fast, Qwen Image 3 (+edit), MAI Image 2.5
Pro, Nano Banana 2 Lite (+edit), Recraft V4.1.
Every new endpoint live-tested against fal.run through the real
payload builders + submit path: 18/18 pass (t2v, i2v, t2i, and edit
probes). Note: several new endpoints return HTTP 409 from the Nous
Portal FAL proxy allowlist until it is updated portal-side; BYOK
FAL_KEY works today and the existing 4xx guidance message covers it.
Force-redact every terminal exception and traceback field before JSON serialization, including environment creation, background startup, exhausted foreground retries, and the outer catch-all. Preserve the current command-aware, opt-out-respecting redact_terminal_output(output, command) behavior for successful output.
anydoc converts the PDF text layer only and emits no image placeholders
or page markers, so a mostly-scanned PDF extracts 'successfully' into
section headers with empty bodies — silent data loss the model cannot
detect. Count per-page text via poppler pdftotext and prepend an
EXTRACTION COVERAGE WARNING naming the empty pages and the recovery
path (pdftoppm + vision_analyze, or the ocr-and-documents skill).
Found on a 311-page HOA resale package where 198 scanned pages
(CC&Rs, Bylaws, Articles, insurance certs) vanished without a trace.
Closes the last two emission gaps from #77484:
- tools/terminal_tool.py: both exception paths (generic except and
TERMINAL_DEGRADED_MODE=fail) returned raw str(e) + traceback.format_exc()
to the model — only the logger copy was redacted. Exception text can
embed the failing command line and any secrets inline in it; both fields
now pass through redact_sensitive_text.
- acp_adapter/entry.py: _setup_logging cleared root handlers and installed
a plain logging.Formatter, bypassing redaction entirely on ACP stderr.
Now uses RedactingFormatter like every other logging surface.
The other three gaps from #77484 (process(list), *_KEY regex variants,
control-char splits) were fixed in #80964/#80965.
image_generate and video_generate forwarded model-supplied local paths to
provider plugins, which read them off the HOST filesystem regardless of
terminal backend — inconsistent with the confinement boundary vision/video
analysis enforce (GHSA-gpxw-6wxv-w3qq), and broken for sandbox-only files.
New dispatch-layer chokepoint (_confine_source_images): under a non-local
backend, path-like image_url / reference_image_urls resolve through
tools.image_source (media-cache host reads, bounded in-sandbox exec-read,
lazy env bring-up, credential guard, 50MB cap) and reach every provider as
data: URLs — which all backends already accept. URLs/data: pass through;
local backend is a no-op. xai_video_edit/extend already require public
HTTPS URLs, so no change needed there.
Under the Docker terminal backend, vision_analyze's first exec-read
sometimes returned empty / non-zero against a freshly started container,
producing 'could not read <path> inside the sandbox' on a file the agent
could cat seconds later. Cold pipe setup on the first exec against a
new container, not a permissions or mount problem.
Retry once after a short delay (150 ms covers Docker exec warm-up
without making a real failure feel sluggish). When every attempt still
fails, fold the container's first stderr line into the raised error so
the user can tell 'no such file' from 'permission denied' instead of
staring at one opaque message.
Tests cover the retry-then-succeed path, the diagnostic-on-exhausted
path, and confirm the existing single-attempt raise is preserved.
Follow-up on the salvaged commit: replace the hand-rolled file_ops python3
exec-read with tools.image_source.resolve_image_source(permitted=('video',)),
so video_analyze gets the same pipeline as vision_analyze — media-cache host
reads, bounded head -c sandbox exec (no python3 dependency in the sandbox
image, no unbounded base64 stream), lazy env bring-up (#62825), the
credential-read guard, and the 50MB ingest cap.
The _bash_exec_payload delegation rejected short-option bundles with
letters outside bash's alphabet, so 'zsh -yc', 'dash -Vc' and 'ksh -Gc'
scripts stopped being scanned — a fail-open regression for shells the
guard's _SHELL_EXECUTABLES explicitly covers. Try the bash grammar
first (catches operand-hidden -c), then fall back to the permissive
positional scan; a block-guard fails closed.
bisect sat in _KNOWN_GIT_BUILTINS and was allowed in the running source
root, yet it repeatedly checks out commits — the exact module-version
skew this guard exists to prevent. Move it to _WORKTREE_MUTATIONS.
The guard's _shell_script_arg treated any leading option containing 'c'
as -c and looked no further, so 'bash -o pipefail -c "git checkout
main"' returned None and the script was never scanned (fail-open).
approval.py's _bash_exec_payload already parses bash's real option
grammar (-O/-o consume operands, short-option bundles, --init-file);
delegate to it instead of keeping a second, weaker parser.
`worktree` sat in _KNOWN_GIT_BUILTINS, so the guard returned safe for the
whole family. That allowed `git worktree remove [--force] <root>` and
`git worktree move <root> <dest>` against the very checkout this process
runs from, which the guard already treats as a source root when its .git
is a linked-worktree file.
Both name their target as an argument rather than acting on the cwd, so
they also slipped past the "is the cwd inside root" gate when run from
outside. Resolve the target against the command's cwd and block it when
it lands on the running root, from any directory. `worktree add`, list,
prune, lock, unlock, and operations on other worktrees stay allowed.
Wire the self-repo guard in next to the gateway lifecycle hard-block,
before the force check — force=True cannot make the command safe, only
delay the crash. Local backend only: sandboxed backends cannot reach the
host checkout. The block message explains the version-skew mechanism and
redirects to git worktree add / a temp clone, or running the command
outside hermes with a restart after.
When hermes runs from a source/editable install, a git checkout/reset/pull
in its own repo swaps code on disk under the live interpreter. Modules
imported before the switch stay old while later lazy imports load new code,
producing delayed signature TypeErrors and tracebacks that don't match the
source, typically losing the in-flight turn.
New tools/self_repo_guard.py detects working-tree/ref mutations (checkout,
switch, reset, rebase, merge, pull, restore, stash, clean, cherry-pick,
revert) whose target repo is the source root the process runs from, via
cwd, git -C, cd chains, and subshell segments. Read-only git, commits,
fetch, and git worktree add stay allowed; packaged installs (no .git) are
inert.
dispatch() called logger.exception with the exception interpolated into the
message. exc_info renders the same exception again in the traceback, so a
failing tool wrote its error body to the log twice. Every tool exception
passes through this one handler, so a large HTTP error body from any tool
landed here at full size.
Bound the message copy. The traceback still renders the exception once,
which is what an operator needs to place the failure.
Same double-write @arimu1 fixed in the vision, image, and TTS handlers in
#75938.
The debug line that fires when an error body is truncated interpolated the
whole untruncated body, so capping the model-facing copy still wrote the
original to the log. A large HTTP error body — a Cloudflare challenge page
or a proxy 502 — reached the log at full size on every failed call.
Log a bounded prefix instead. It stays longer than the model-facing cap so
an operator still has something to diagnose with, but it no longer grows
with the size of the response body.
Reported for the logging handlers in #75938 by @arimu1; the same pattern
was present here.
tool_error() caps its message at _MAX_TOOL_ERROR_CHARS (2048), logging
the full body at DEBUG before trimming the context-bound copy.
Handlers that serialize exceptions directly -- json.dumps({"error":
str(exc), ...}) -- bypass that helper, so _normalize_handler_result
also runs every string result through _bound_json_error_result: if it
parses as a JSON object with an oversized string error field, only
that field is trimmed and the payload re-serialized. Non-error
results, non-JSON strings, and multimodal envelopes pass untouched.
Open the selected microphone at its reported default rate and convert each capture block to the 16 kHz frame expected by wake-word engines. Add a regression covering a 48 kHz WASAPI device.
Co-authored-by: clyu168 <clyu168@126.com>
An auth failure on the very first connect returned out of the run loop
instead of parking. That ended the run task, and the task is the only
listener on _reconnect_event — so the server stayed dead for the life of
the process. `hermes mcp login`, a /mcp refresh, and the 300s self-probe
all had nothing left to wake, and the only cure was a full restart.
_classify_mcp_failure already calls 401/403 "permanent" and documents
that run() parks those immediately; the early return above it meant auth
was the one permanent failure that never got there. Park it with the
others and keep the tailored log line, now pointing at `hermes mcp
login <server>`.
The windows-compat change-detector checked for the literal string
'start_new_session=True', but the systemd scope isolation path
conditionally uses start_new_session=False (the scope creates its own
session/cgroup). Assert 'start_new_session=' instead — the value may
now be a variable.
When Hermes runs as a systemd gateway with MemoryHigh/MemoryMax limits,
local background terminal commands (terminal(background=true)) inherit the
gateway's cgroup. A memory-heavy executor (Codex, tests, Node) can push
the whole cgroup past MemoryMax and trigger systemd-oomd to kill the
ENTIRE gateway — taking down the messaging control plane and silently
losing the active turn.
Root cause: tools/process_registry.py::spawn_local() uses
start_new_session=True (creates a process session/group, NOT a resource
cgroup). The spawned process tree stays in the gateway's systemd cgroup.
Fix: when running under a service manager (detected via the existing
is_gateway_supervisor_process() helper), wrap the pipe-mode spawn command
in 'systemd-run --user --scope --unit=hermes-worker-<id>' so the worker
gets its own transient cgroup. An OOM in the worker then kills only the
worker, not the gateway.
The systemd-run availability is probed once (a no-op /bin/true in a
transient scope) and cached, because the binary can exist on PATH while
the user D-Bus session is unavailable (system services, containers). If
unavailable, fall back to the current start_new_session=True behavior
with a debug log.
Scope: this covers the common background pipe-mode path. PTY mode
(PtyProcess.spawn) is left as future work — it uses a different spawn
mechanism and is used for interactive CLI tools where cgroup isolation
has additional considerations.
vision_analyze reads container-only images by exec-reading them inside the
sandbox, but unlike terminal_tool it never triggered environment creation. Under
a non-local backend (ssh, docker, ...), a session whose first action was
vision_analyze on a remote path failed with 'no active sandbox session' until an
unrelated terminal command happened to establish the connection.
Add terminal_tool.ensure_task_env(task_id), a public lazy get-or-create that
reuses the terminal tool's own creation machinery, and call it from
image_source._resolve_container_fallback before the in-sandbox read. Extract the
ssh/container config-dict builders so both paths derive settings identically
(no duplication). Best-effort and fail-closed: a failed bring-up leaves the
existing 'no active sandbox' error intact, never a host read.
Fixes#62825
The modal/browserbase test file replaces tools.environments.base with a
SimpleNamespace stub; terminal_tool now imports EnvironmentConnectionError
from that module, so the stub must provide it too.
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.
Now:
- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
carrying a reason + retry_hint. Subclassing RuntimeError keeps every
existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
(missing exe, non-executable exe, daemon timeout, `docker version`
failure).
- terminal_tool catches EnvironmentConnectionError and returns a
structured tool result the model can act on:
{"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
The failed backend is evicted from the environment cache so a later
call retries from scratch — recovery is automatic once the backend is
reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
only infrastructure failures classify as degraded.
Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.
Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
Per-task `output_schema` (JSON Schema object) on task items plus the
top-level single-goal form — a one-time static addition to the tool
schema (never varies per call).
- Child side: the schema is appended to the child's context as an
explicit OUTPUT CONTRACT block before spawn.
- Completion side: the parent validates the child's final answer with
jsonschema; on failure it sends exactly ONE bounded retry turn
carrying the validation errors verbatim (no schema re-paste).
- Result entries gain schema_valid (+ schema_retries, and schema_errors
on final failure) ONLY when a schema was requested; schema-less calls
keep a byte-identical result shape.
- Malformed schemas are rejected loudly at dispatch (coerce_output_schema
meta-validates via jsonschema's validator_for/check_schema).
- New helpers in tools/delegation_output_schema.py: coerce, contract
block, fence/prose-tolerant extraction+validation, retry message.
Pattern from: github/copilot-cli ctx.agent(prompt,{schema}) — PATTERN
ONLY, zero code/prompt text copied (proprietary); proven consumer:
delegate-task-output-patterns skill.
Tests: tests/tools/test_delegate_output_schema.py (24 tests — valid
first try, invalid->retry->valid, invalid twice -> schema_valid false +
errors surfaced, retry-exception degrade, no-schema legacy shape pin,
dispatch rejection, contract plumbing). Delegation suite: 221/221 green.
Add an optional `region: [x1, y1, x2, y2]` parameter to vision_analyze
(pixel coordinates in the ORIGINAL image space). The crop is applied
with Pillow BEFORE the downscale/embed-cap pipeline, so the cropped
region gets the full resolution budget — a zoom for reading small text
or UI details after a full shot.
- New `_crop_image_region` helper: clamps out-of-bounds coordinates to
the image, rejects zero-area/inverted/malformed regions with an error
naming the actual image dimensions so the model can retry sensibly.
- Wired into both the native fast path (`_vision_analyze_native`) and
the legacy aux-LLM path (`vision_analyze_tool`).
- Schema gains one static optional param (byte-stable thereafter); the
description documents the intended flow: full shot first, then zoom.
- No region supplied = behavior unchanged (regression-guarded).
Tests: tests/tools/test_vision_region.py (11 tests — crop applied,
clamping, zero-area rejection with dims, malformed input, pre-downscale
full-budget zoom, schema shape, handler pass-through, no-region
unchanged). Widened one narrow fake_native stub in test_vision_tools.py
to be kwargs-tolerant.
Ported from: QwenLM/qwen-code zoom-image.ts (Apache-2.0)
write_file/patch targeting AGENTS.md, CLAUDE.md, SOUL.md, .cursorrules, or a
project-local .hermes config dir now ALWAYS prompt the human for approval —
even under --yolo/auto-approve — and fail closed when no human channel
exists. These files steer future agent behavior, so an injected write to
them is a prompt-injection persistence vector.
Design:
- New _check_protected_instruction_write() in tools/file_tools.py, a
sibling of _check_sensitive_path that returns approval-required rather
than a hard error. It realpaths before matching (symlink lesson from
#41351), matches basenames case-insensitively in ANY directory, rejects
'./x/../AGENTS.md' traversal via normpath, and gates files whose
immediate parent dir is `.hermes` (project-local config) while exempting
the authoritative ~/.hermes home (governed by its own guards).
- Approval is ONE-OPERATION only: no session/permanent persistence, no
yolo bypass — intentionally does not route through _run_approval_gate.
Gateway sessions get the button round-trip with allow_permanent and
allow_session both False; CLI uses the per-thread approval callback;
no channel at all = BLOCKED (fail closed).
- Multi-file V4A patches: ONE protected file gates the ENTIRE patch (a
single prompt lists all protected targets; deny applies nothing).
- Config: security.protected_instruction_files (default true) and
security.protected_instruction_extra_patterns (fnmatch on basename).
Config read failure keeps the gate ON.
Tests: 22 new cases in tests/tools/test_file_write_safety.py covering the
adversarial checklist — deny/approve/yolo-bypass attempt, symlink at a
protected target, case variants, relative traversal, arbitrary-directory
basenames, project-local .hermes, checkout-nested-under-~/.hermes
non-gating, patch replace + V4A multi-file atomicity, gateway round-trip,
fail-closed with no human, config off/extra patterns.
Ported from: RooCodeInc/Roo-Code RooProtectedController (Apache-2.0);
companion: #58631 (terminal vector), symlink lesson from #41351.
Adds a per-server `trust: full|untrusted` config key
(mcp_servers.<name>.trust). On an untrusted server, every write-capable
tool call — any tool whose discovery-time annotations do not carry
readOnlyHint=True — routes through the existing approval surface
(tools.approval.request_elicitation_consent, same lazy-import +
surface-routing pattern the MCP elicitation handler uses) before the RPC
fires. Denied/cancelled/errored approvals fail closed: the RPC never
runs, including the lazy first-use server spawn.
Design points:
- Classification happens at CALL TIME from metadata captured at
DISCOVERY (_record_tool_trust_metadata in _register_server_tools and
the lazy cache-registration path). No toolset/schema mutation, so the
toolset stays byte-stable and prompt caching is preserved.
- readOnlyHint is a server-supplied HINT: on an untrusted server a lying
server can at most skip approval for tools it claims read-only — it
can never widen access. Trust tiering itself is operator config.
- Missing/malformed annotations => write-capable (fail closed).
- Unrecognized trust values => untrusted (fail closed); missing key =>
full (backward compatible, documented in mcp-config-reference).
- The schema cache now persists readOnlyHint so lazy-registered servers
gate identically on next startup without spawning.
Tests: tests/tools/test_mcp_trust_gating.py (11 tests, TDD red->green):
approval invoked + accept proceeds, deny/cancel blocks RPC, readOnlyHint
=true skips gate, trusted/unconfigured servers skip gate, explicit
readOnlyHint=false gated, approval exception fails closed, trust
normalization, discovery-time capture (SDK objects and cached dicts).
Ported from: cloudflare-os classifyTool() (Apache-2.0), corroborated by
Claude Cowork (idea-level).
MCP servers generated from Rust/TypeScript union types commonly emit
closed value sets as const unions:
{"anyOf": [{"const": "red"}, {"const": "green"}, {"const": "blue"}]}
Strict tool-calling backends reject or mishandle these; the equivalent
property-level enum form is universally supported. Add
collapse_const_unions() to tools/schema_sanitizer.py and wire it into
the _normalize_mcp_input_schema discovery pipeline after the nullable
strip.
Rules:
- Collapse only when EVERY non-null branch is a pure const of the same
primitive type (bool never merges with integer).
- Mixed unions, non-uniform const types, and mismatched declared types
pass through untouched.
- A single {"type": "null"} branch is tolerated: consts -> enum,
null -> nullable: true hint (matches strip_nullable_unions, which
leaves null+multi-const unions alone by its one-non-null-branch rule).
- Outer title/description/default/examples carried onto the replacement.
- Deterministic, branch-order-preserving, non-mutating — applied at
discovery only, so schemas stay byte-stable per conversation.
Ported from: block/goose tool_schema_normalize.rs (Apache-2.0)
Two small config-gated features:
1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
true, config.yaml): a running card with broken claim bookkeeping
(claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
restore) is invisible to all existing recovery paths
(release_stale_claims requires claim_expires NOT NULL,
detect_crashed_workers requires host-local lock + pid,
detect_stale_running is config-disabled by default) and shows Running
forever. New reconcile_orphaned_running() pass in kanban_db.py runs
each dispatch_once tick: requeues orphans to ready with an explanatory
comment, closes any leaked run, emits a 'reconciled' event, and defers
when the recorded PID is still alive on this host (never requeue
beside a live worker). Surfaced via DispatchResult.reconciled_orphans.
2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
config.yaml): optional {name, value_from: static|profile, value}
mapping; the header is attached to that server's HTTP/SSE transport
requests. 'static' sends the config value; 'profile' resolves the
active Hermes profile name once at connect time (no per-call
mutation). Explicit per-server headers of the same name (any casing)
win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.
Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.
Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
Each serialized result entry now carries cost_usd (rounded to 6 dp)
and cost_status (the child's session_cost_status — 'estimated',
'reported', 'included', or 'unknown') alongside tokens/api_calls/
duration, so the parent model can see what each delegation cost.
The internal _child_cost_usd field is still stripped before
serialization and the parent session cost rollup is untouched.
Tool schema is unchanged (byte-stable).
Inspired by: Perplexity Agent API result shape (idea-level)
Reject malformed tasks=[...] batches before any child agent is spawned:
- exact-duplicate goals (case/whitespace-normalized), error names both
task indices
- placeholder goals: bare 'TODO', bare 'task N', unexpanded <...> or
{...} template markers, or goals shorter than 10 chars after strip
- 1-task batches, with an error pointing the model at the single
`goal` form instead
All checks are batch-only — the single-goal form is exempt by design
(short goals like goal="test" are valid there). Error strings are
actionable: each tells the model exactly how to fix the call.
Tool schema is unchanged (byte-stable); validation is runtime-only in
the existing batch-validation region.
Existing tests using terse batch goals ("A"/"B"/"C") updated to
realistic distinct goals per the new contract.
Inspired by: MoonshotAI/kimi-code agent-swarm.md validation rules (MIT)
Review pass on the idle-unload feature found two material concurrency
bugs; both fixed here with a regression guard:
1. Unload-vs-use null deref (HIGH): _transcribe_local re-read the
module global _local_model at the transcribe call site. An idle
unload firing between the model load and transcribe() evaluated
None.transcribe → AttributeError → user-visible 'Local
transcription failed'. The window was real: the idle timer was only
touched AFTER a successful transcription, so a voice note arriving
exactly as the timeout expired raced the watcher directly.
Fix: bind a strong local reference under the model lock and use it
for the whole transcription (the watcher can null the global at any
time; this in-flight call keeps its instance — the generator holds
self, so no use-after-free). Also touch the idle timer at the START
of transcription so a long in-flight transcribe can't be counted as
idle time. The CUDA-fallback retry path gets the same treatment
(locked global write, local ref use).
2. Watcher replacement race + response-path join (MEDIUM/HIGH): the
old design stopped and re-started the watcher after EVERY
transcription with an unlocked set/join(5)/clear/start sequence on
shared globals. Two concurrent voice messages could interleave to
leave TWO live watchers (one with a stale, shorter timeout — a
raised unload_after_idle_seconds could still unload on the old
value), and the join(timeout=5) sat on the user-visible response
path (a watcher blocked on _local_model_lock during a concurrent
multi-second model load stalls the reply up to 5s).
Fix: single long-lived watcher under a management lock — started
only when none is alive (per-transcription cost: one lock + one
is_alive check), re-reads the configured timeout from config every
cycle (config edits now apply within one 30s interval, without
waiting for the next voice message — previously undocumented), and
stands down without unloading when the timeout is set to 0
mid-idle.
Tests: 17 now — idempotent start (same thread, no churn), config
re-read + stand-down-when-disabled, and the race guard
(unload firing mid-transcription must not fail the in-flight call).
The race guard is mutation-verified: reverting the fix (re-reading
the global at the call site) makes it fail with the exact NoneType
error; the fixed code passes.
The local faster-whisper model singleton (_local_model) is loaded once
and never released — the 'base' model holds ~370 MB of RAM/VRAM for
the entire lifetime of the process, even when no voice messages arrive
for hours or days. On long-running gateway processes (especially with
local LLMs competing for the same GPU) this is wasteful.
Add a config-driven idle unload: after stt.local.unload_after_idle_seconds
(default 0 = never) of no transcription activity, a lightweight daemon
thread sets _local_model = None so the Python GC can reclaim the
ctranslate2 objects. The next voice message reloads the model
transparently (the existing lazy-load path handles it).
The watcher:
- Checks every 30s whether idle time exceeds the configured threshold
- Acquires _local_model_lock before unloading (prevents races with
concurrent transcriptions that are mid-load)
- Exits immediately if the model is already None (unloaded by another
path, e.g. the CUDA fallback eviction)
- Is restarted by each transcription with the current config value,
so changing stt.local.unload_after_idle_seconds in config.yaml takes
effect on the next voice message without a process restart
Default is 0 (never unload) — zero behavior change for existing users.
Recommended value for gateway processes: 300 (5 minutes).
15 tests: config resolution (garbage/negative/None fallbacks), unload
safety (already-None, lock acquisition), touch timestamp, watcher
lifecycle (unload after timeout, no unload within timeout, exits when
model already None, stopped on new start). Existing STT test suite
unchanged.
Three-reviewer pass (reuse / quality / efficiency) on the trim diff;
four findings folded:
1. Short-clip input gate (efficiency, HIGH): the trim previously paid
the full ffmpeg encode before the <10%-saving discard check — every
dense conversational voice note burned 3 subprocess spawns + a
complete re-encode on the synchronous response path for nothing.
New _CLOUD_TRIM_MIN_INPUT_SECONDS=12 gate: below it, savings can't
matter (a >=10% saving is ~1s of audio, and several providers bill
a per-request minimum anyway — Groq bills 10s minimum), so the
whole pipeline is skipped using the duration we already probed.
Typical 5-10s voice notes now pay 1 ffprobe (~50ms), not 3 spawns +
encode (~0.3-1s; multi-second on small-VPS gateway hosts).
2. Shared encode profile (reuse, HIGH): the trim's ffmpeg command
duplicated _transcode_audio_for_stt's encode byte-for-byte (same
16kHz/mono/AAC-32k/faststart args, same subprocess.run kwargs).
Extracted _STT_M4A_ENCODE_ARGS + _run_ffmpeg_stt_encode(ffmpeg,
in, out, audio_filter=None); both call sites now share one owner,
so codec/bitrate/timeout changes can't drift between the paths.
3. is_truthy_value for the enable flag (quality, MEDIUM): raw
bool(cfg.get(...)) treated a YAML string "false" as enabled — the
exact bug class utils.is_truthy_value (already imported, already
used by is_stt_enabled and the xai/elevenlabs flags) exists for.
4. All-silence guard scales with keep_ms (quality, LOW): the fixed
0.3s floor equals the default keep window, so an output consisting
solely of one kept pause could pass as "speech"; now
max(0.3, 2*keep_seconds).
Also: _probe_audio_duration docstring documents it as the canonical
sync seconds-probe (gateway/run.py and the Telegram adapter carry
local variants of the same ffprobe invocation).
Tests: 24 now — YAML-string-false disables; short clips skip the
encode entirely (encoder mock asserted not-called); E2E fixtures
moved past the input gate. E2E re-verified: 13.2s note -> 6.2s
(-53%), 8s clip skipped with 1 probe.
Local faster-whisper gets Silero VAD (bf8004e3a) so silence never
reaches the model. Cloud providers got no such protection: the raw
file uploads untouched, so every second of silence in a voice note is
paid for twice — upload time and per-audio-minute billing — and cloud
Whisper hallucinates junk tokens on silent stretches exactly like
local Whisper did before the VAD hardening. A 13s voice note with two
long pauses is billed as 13s of audio to transcribe ~6s of speech.
Close the gap client-side: before uploading to a built-in cloud
provider (groq/openai/mistral/xai/elevenlabs/deepinfra), collapse long
pauses with ffmpeg's silenceremove filter, keeping
stt.cloud_trim_keep_ms (default 300) of every pause so word boundaries
and natural pacing survive. Uses ffmpeg, already a dependency of this
exact path via _transcode_audio_for_stt — no new dependency.
The trim is strictly best-effort — ALL of these upload the original
untouched, transcription never fails because of the trim:
- stt.cloud_trim_silence: false
- ffmpeg/ffprobe missing, trim failure, or timeout
- trimmed result ~empty (mostly-silence clip: the provider, not a
client-side dB heuristic, decides whether it contains speech)
- trim saves <10% (re-encoding for nothing)
Command-type and plugin providers are deliberately NOT trimmed: they
may wrap local CLIs that want the original bytes or run their own VAD.
E2E (real ffmpeg + faster-whisper): 13.2s voice note with 7s pause ->
6.2s upload (-53%); transcript of trimmed audio matches the original
on both utterances. Dense-speech and all-silence WAVs correctly fall
back to the original. 22 unit+E2E tests; STT/voice suite failures
identical to upstream/main baseline (all pre-existing).
_download_image() and _download_video() both used client.get() +
response.content, buffering the entire media body into memory before
checking the size cap. A server that omits Content-Length could send
an arbitrarily large payload, causing OOM.
Extract _stream_download_to_file() shared helper: streams via
client.stream() + aiter_bytes(), writes chunks to a temp file, enforces
the running byte count against the cap after each chunk, and atomically
replaces onto the destination on success. Cleans up the temp file on
failure. Uses utils.atomic_replace() for cross-device/symlink safety.
Malformed Content-Length values are now caught and ignored instead of
crashing with ValueError; the streaming cap is the authoritative guard.
Approach adapted from PR #10440 by @WuKongAI-CMU (closed as stale —
14923 commits behind, reverted 32 commits of vision_tools.py evolution
including SSRF-safe client, retry classification, and lazy imports).
Closes#10440
_write_checkpoint persisted s.command verbatim to ~/.hermes/processes.json.
Recovery only uses command for display/logging (the process is already
running; adoption re-validates PID + start time, never re-runs the
command), so masking is lossless.
- is_tts_echo sliding window: return True immediately when ratio >= threshold
instead of scanning all remaining windows. Common echo case drops from
~4s to <1ms for long spoken text (found by /simplify-code efficiency review).
- Clear _voice_barge_phase in _voice_submit_barge_utterance finally block
alongside _voice_barge_capture, preventing stale phase from a previous
trip affecting a future call.
- Add _voice_last_tts_text and _voice_barge_phase to _make_voice_cli test
helper so it matches __init__ state.
Reviewer feedback on the fragment-echo fallback added in 24730b10c:
- A short playback-phase transcript (e.g. a genuine one-word "yes")
could trivially match a same-length window of a longer spoken reply
at ratio 1.0 and be wrongly dropped as a self-capture. The fallback
now requires the transcript to be at least
MIN_FRAGMENT_LENGTH_FOR_ECHO characters before it runs.
- The fallback split on whitespace, so it never engaged for
no-whitespace languages (both transcript and spoken text collapse to
a single "word"). Switched the sliding window to be character-based
instead of word-based, matching the function's tokenization-independent
contract.
Adds regression tests for both cases.
is_tts_echo() compared the captured transcript against the *entire*
spoken text with a whole-string similarity ratio, which only scores high
when the two strings are close in length. But a playback-phase barge
capture is cut immediately when the trigger fires and only spans the
pre-roll buffer plus time-to-silence, so a genuine self-capture is
typically a short fragment of a longer reply, not a near-verbatim repeat
of the whole thing -- for any response longer than a clause, the
length-diluted ratio fell below threshold and the echo sailed through
ungated.
When the whole-string check misses, also slide a window sized to the
transcript's word count across the spoken text and compare against each
window, so a short fragment echoed from within a much longer
multi-sentence reply is still caught. Add regression tests for a short
fragment matched at the start and in the middle of a longer reply.
The full-duplex barge-in listener added in 5081551f0 stays active during
TTS playback with no acoustic echo cancellation. On some speaker/mic
combinations, TTS bleed alone crosses the barge threshold, gets
transcribed, and is queued as the next user turn -- whose reply is then
spoken, captured, and queued again, producing an unbounded TTS -> STT ->
TTS feedback loop (#75780).
Add a fail-closed transcript-level guard: when a barge trip happens during
the playback phase, compare the captured transcript against the TTS text
Hermes just spoke (tools/voice_mode.is_tts_echo, a language-agnostic
character-level similarity ratio). A close match is dropped instead of
queued, and the mic is handed back to the normal continuous-listening
loop. Generation-phase trips (no TTS playing, so no bleed is possible)
are unaffected.
Fold the remaining simplify-code reuse finding: teams' _install_hint()
duplicated lazy_deps' spec-fetch + quote + join (feature_install_command
already builds pip commands from LAZY_DEPS). Add a venv_pip=True variant
to feature_install_command — sys.executable -m pip targeting, correct in
every install layout and immune to PEP 668 — and shrink the teams helper
to a one-line call. Also gives matrix and the other platforms a shared
derived hint to adopt later. New test mutation-checked (fails when
venv_pip returns the uv form).
Salvaged from PR #57342 by @liuhao1024 (with the injection-scan half
from PR #57360 by @ghedeselmabot): cronjob(action='run', prompt=...)
silently discarded the prompt argument — per-run context never
reached the spawned cron session.
The prompt is now threaded as extra_prompt through the whole chain
(cronjob run action → _try_dispatch_background_run/_execute_job_now →
_run_claimed_job → run_one_job → run_job → _build_job_prompt) and
appended to the stored prompt under a '## Run Context' header for
that single fire only — never persisted to the job definition. It
passes the same strict _scan_cron_prompt injection scan as stored
prompts before firing, and works identically on the background and
sync fallback paths.
Test fakes across tests/cron/ updated to accept the new kwargs
(sibling-test blast radius from the signature change).
Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
Salvaged from PR #53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.
Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, #60432) and mark_running_jobs_interrupted,
which previously could not see them.
The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.
Co-authored-by: izumi0uu <izumi0uu@gmail.com>
A manual cronjob run executed the job synchronously on the calling
agent's tool thread. A cron job is a full agent run that routinely
takes minutes to hours, so the parent turn sat inside ONE tool call
the whole time: uninterruptible (the interrupt flag is only checked
between loop iterations) and serial (a batch of manual runs executed
one by one). A Telegram session that kicked off dozens of new jobs
'right now' was wedged for hours ignoring every interrupt.
action='run' now rides the async-delegation rail delegate_task
background mode uses: the at-most-once claim is taken synchronously
(so paused/missing/already-firing jobs still report immediately),
the run executes on the shared daemon executor, the tool returns at
once with a delegation handle, and the job's outcome re-enters the
conversation as a type='async_delegation' completion event through
the existing completion-queue drains (CLI + gateway) — preserving
message-role alternation and the prompt cache.
Sync fallbacks preserved:
- no routable session (direct Python callers, hermes cron run)
- async delivery unsupported (hermes -z, cron child sessions,
Kanban workers, stateless HTTP)
- dispatch pool at capacity (claim already taken — runs inline
rather than stranding it)
The completion block reports ok/failure, delivery target, next
scheduled run, and an excerpt of the job's saved output.
The pane, in-app browser, and reaction tools were gated on HERMES_DESKTOP=1 —
an env var set only on backends Electron spawns itself (local and SSH). A
desktop client connected to a plain URL gateway or Hermes Cloud lost all six:
they were stripped from the schema before the model saw them, on the same
backend whose platform hint was telling it "you are chatting inside the Hermes
desktop app". open_preview, read_preview, read_terminal, close_terminal,
focus_pane, and react_to_message were all silently absent.
The client is not the host. Capability now resolves from the session's own
source, which session.create already carries:
- The six tools move into a `desktop_ui` toolset, off _HERMES_CORE_TOOLS so no
other platform pays their schema.
- _gui_surface_toolsets(platform) folds `desktop_ui` (and the existing
`project` tools) into the GUI gateway's resolution when the session's
platform is the desktop app — the same answer on every topology.
- check_fn drops the env probe. It kept the one thing that is genuinely a
per-process/user fact: react_to_message's display.message_reactions opt-in,
which the desktop mirrors onto whichever gateway it is connected to.
react_to_message was doubly broken: it read that toggle behind the env gate, so
even a local-backend user's Settings toggle could not reach a remote session.
The embedded terminal pane keeps working correctly the other way round: it runs
`hermes --tui` against a desktop-spawned backend, and a tui-sourced session
gets no GUI tools even though HERMES_DESKTOP=1 is set on that process.
The turn finalizer already hands back steer text that queued after the
final tool batch — result["pending_steer"], with the comment "hand it
back to the caller so it can be delivered as the next user turn instead
of being silently lost." Every interactive surface honors that contract
(cli.py, gateway/run.py, tui_gateway/server.py all requeue it). The
delegation layer doesn't: _run_single_child never reads it, so a steer
queued into a delegated child that finishes first vanishes with no trace
in the completion entry. There is also no sanctioned sender: the registry
has interrupt_subagent() but no redirection-side mirror, and session.steer
cannot reach children (lazy watch sessions have agent=None, so it 4010s).
Complete the contract for delegated children — both halves:
- steer_subagent(subagent_id, text): redirection-side mirror of
interrupt_subagent(). Resolves the live child in _active_subagents and
queues text via AIAgent.steer(). True means queued, not delivered.
- missed_steer retention: when the child's result carries pending_steer,
_run_single_child names it on the completion entry (missed_steer field
plus a summary note) so the parent can re-issue the guidance instead of
trusting it landed. This is what makes adding a sender safe: without it
the finish-before-drain race silently loses the text — the exact loss
the finalizer contract exists to prevent.
- subagent.steer gateway RPC beside subagent.interrupt so programmatic
hosts (dashboard, voice layers, ACP bridges) get an in-tree caller;
catalogued in programmatic-integration.md.
- docs: "Steering a Running Subagent" section in delegation.md covering
the queued-vs-delivered semantics.
Tests: registry-level steer coverage (delivery, unknown id, empty text,
dead record, raising agent), the finish-before-drain race retaining
missed_steer, and the RPC contract (4000/4002 validation, queued and
rejected envelopes).
The anydoc path from #79781 passed every covered file straight to
to_markdown with no pre-check. anydoc loads the whole document through
its Rust core and the read_file char budget only applies after
conversion, so one large PDF or deck could pin a tool turn and spike
RAM.
_extract_anydoc now rejects inputs over MAX_ANYDOC_BYTES (50 MB) with
ExtractionError before calling the converter, which routes them to the
existing read_file fallthrough instead of converting. No timeout is
added: the conversion is a synchronous Rust call that cannot be
cancelled from Python, so a thread-based deadline would bound the wait
but leave the RAM burn running in the background.
The first _anydoc() load cached None on any failure (network blip,
missing wheel, pip race), so one bad first try disabled document
extraction for the rest of the process. Long-lived gateway and desktop
workers never recovered.
Failed loads now cool down for ANYDOC_RETRY_SECONDS and retry instead
of sticking, and a lock serializes first use so parallel readers cannot
double-install or race a failure into the cache. Successful loads are
still cached for the process lifetime.
read_file's auto-extraction covered only the stdlib trio (.ipynb/.docx/
.xlsx). firecrawl-anydoc (MIT, Rust core, imports as `anydoc`) converts
Word, PowerPoint, Excel — including legacy .doc/.ppt/.xls — OpenDocument,
RTF, EPUB, and PDF to clean Markdown through one shared document model.
Wiring follows the footprint ladder: no new tool, no hard dependency.
- tools/read_extract.py gains an ANYDOC_EXTENSIONS set that is active
only when the converter imports; the stdlib extractors remain
authoritative for their three formats so behavior is identical with
or without the package.
- tools/lazy_deps.py adds tool.doc_extract (firecrawl-anydoc==0.1.6),
installed on first read of such a file with prompt=False so read_file
can never block. Lazy-only for now: the package's first release was
2026-08-04, inside uv's 14-day exclude-newer quarantine, so the
mirrored pyproject extra lands after it clears.
- Any anydoc ConvertError maps to ExtractionError, falling back to the
existing path/binary handling instead of erroring the tool.
Tests: real-binding suite skips cleanly when the wheel is absent
(verified: 15 passed/3 skipped without it, 18 passed with it), plus an
absent-dep contract class that pins the fallback regardless of local
install state.
* feat(agent): read_preview — the desktop-gated tool that reads the in-app browser
The agent could open the preview pane (open_preview) and read the embedded
terminal (read_terminal), but the browser it had just opened was a black box —
'what does this page say?' had no answer. read_preview mirrors read_terminal
end to end: HERMES_DESKTOP-gated via check_fn (zero schema footprint outside
the GUI), dispatched through the same agent callback pattern, windowed with
start/count so a long page pages instead of flooding context.
* feat(gateway): preview.read blocking bridge
Same lifecycle as terminal.read: the tool blocks on preview.read.request, the
renderer answers preview.read.respond (allow_expired — a slow page extraction
losing the 45s race must not surface a raw 4009), and a timeout emits
preview.read.expire so late answers resolve quietly.
* feat(desktop): the renderer serializes the active preview tab for the agent
preview-reader.ts is the preview analog of the terminal's buffer registry: the
URL pane registers a page reader (webview executeJavaScript → title + visible
innerText) keyed by tab id; readActivePreview resolves the ACTIVE tab, windows
the text (24k cap per read), and answers file/artifact tabs with identity plus
a note pointing at the tool that reads that content directly. The gateway
event handler answers preview.read.request beside terminal.read.request.
With capture:auto the desktop always preferred client streaming, so a local
desktop with a working backend mic silently switched from PortAudio to
getUserMedia default-device — dropping wake_word.input_device selection
(#74363). A ready backend input now wins under auto; client capture is the
fallback for a preferring surface on a mic-less backend, and capture:client
still forces streaming.
Also removes the dead auto branch (both arms returned local) and lets the
client-feed test skip cleanly when numpy is absent.
Remote headless backends have no PortAudio mic, so "hey hermes" fails even
when openWakeWord is installed. Let the desktop stream 16 kHz int16 PCM via
wake.feed while detection stays server-side.
- wake_word.capture: auto|local|client (+ GUI client_capture prefer)
- WakeWordDetector external_audio queue + feed_audio API
- wake.feed RPC; wake.start/status report capture + frame_length
- Desktop getUserMedia feeder; stop on wake.detected, re-arm after voice
- Docs + unit tests (26 pass in tests/tools/test_wake_word.py)
Top-level delegate_task runs in the background, and the 450s progress-stall
monitor only sees api_call_count / tool / last_activity_ts. Subagents use
non-streaming direct_api_call, which previously touched activity once and then
went silent — so a healthy local GGUF / long-prefill wait looked frozen and
was interrupted around ~450s as "Operation interrupted: waiting for model
response", even when child_timeout_seconds was raised. Refresh activity while
the inline request is open, and treat last_activity_ts advances as sync
heartbeat progress too.
- write_file: encode content once, share bytes between bytes_written and
the sha256 verification (drops a second full-content encode per write)
- patch_parser: replace the except-TypeError retry around
write_file(pre_content=...) with signature-based feature detection so a
TypeError raised inside a capable implementation propagates instead of
triggering a duplicate write; tests for both duck-typing contracts
- tests: real-ops V4A BOM round-trip + _file_has_bom disk-probe guard
(the teknium1-review regression previously only covered by a fake)
- comment: document dirs_created's long-standing "parent ensured" meaning
Bug 1 (UTF-8 BOM loss on V4A UPDATE):
_file_has_bom() trusted pre_content for BOM detection, but the most
common pre_content provider — read_file_raw() — deliberately strips
BOMs so the agent never sees U+FEFF glyphs. Passing BOM-stripped
content through pre_content caused a false-negative: the method
returned False and write_file() silently removed the marker on rewrite.
Fix: _file_has_bom() now always probes the first 3 bytes on disk
(head -c 3), ignoring pre_content for BOM purposes. pre_content is
still used by two other consumers — line-ending detection and lint/LSP
delta computation — neither of which is affected by BOM stripping.
Bug 2 (backward compatibility):
_apply_update() called write_file(path, content, pre_content=...) as a
keyword argument. Duck-typed file_ops implementations that only
implement the two-argument write_file(path, content) contract would
raise TypeError.
Fix: wrap the call in try/except TypeError, falling back to the
two-argument form when the keyword is not accepted.
Also declare tomli in pyproject.toml (pre-existing conditional import
for pre-3.11 Python, caught by the pre-commit dep scan after staging
file_operations.py).
Tests:
Add TestV4ABomRoundTrip with two cases:
- UPDATE on BOM-bearing file preserves the marker
- UPDATE on plain file does not inject a BOM
Addresses teknium1 review on PR #55661.
write_file currently spawns up to 6 subprocesses per call:
1. mkdir -p (separate call before atomic write)
2. cat (to read pre-content for lint/BOM/line-ending detection)
3. _atomic_write (mktemp + write + mv — the essential one)
4. wc -c (to measure bytes written)
5. _check_lint_delta (post-write lint — also essential)
6. LSP snapshot (also essential)
This PR removes three of them without changing any observable behavior:
1. Fold mkdir -p into _atomic_write shell script (−1 subprocess/write)
The atomic write script already runs a single shell; adding mkdir -p
to it costs zero extra processes.
2. Add optional pre_content parameter to write_file (−1 subprocess/patch)
patch_replace and V4A _apply_update already read the file for fuzzy
matching. Passing that content as pre_content skips the redundant cat
inside write_file. Fully backward-compatible: callers that don't pass
pre_content still read from disk as before.
3. Replace wc -c with len(content.encode('utf-8')) (−1 subprocess/write)
We already have the content in memory; encoding it to get the byte count
is equivalent to wc -c for UTF-8 text.
4. Remove redundant _check_lint loop in apply_v4a_operations (−N subprocesses/V4A)
write_file already runs _check_lint_delta internally. The old code ran a
bare _check_lint(f) loop over all modified files — a re-read + re-lint
without post_content context. Now lint results propagate from write_file
via a four-tuple return, zeroing out the extra subprocesses.
Net effect:
- write_file: 6 → 3 subprocesses per call (new files)
- patch_replace: 6 → 5 subprocesses per call (pre_content skips cat)
- V4A multi-file patches: saves 1 subprocess per modified file
- A typical 4-file V4A patch drops from ~28 to ~16 subprocess calls
build_local_transcribe_kwargs read stt.local.no_speech_prob_threshold /
stt.local.logprob_threshold only for Hermes' post-filter
(_is_hallucinated_segment). faster-whisper's model.transcribe() never
received them, so its internal defaults (no_speech_threshold=0.6,
log_prob_threshold=-1.0) always applied and silently dropped
low-confidence segments before they reached the post-filter — making
those config knobs dead for the first gate.
Non-English speech decodes at a lower avg_logprob, so the English-tuned
defaults discard whole utterances (empty transcript despite correct
capture and language detection). Map the same config values through to
model.transcribe() so both gates stay in sync and the knobs work.
Defaults are unchanged, so behavior is identical unless a user tunes them.
Fixes#74178
Five review findings folded:
- schema cache writes via utils.atomic_json_write (fsync; was bare
tmp+replace), file moved to cache/mcp_schema_cache.json with 0o600
(sibling precedent: registry discovery cache)
- phantom-tool reconciliation: after a lazy server's first-use connect,
cached tools the live server no longer offers are deregistered (were
permanent registry ghosts burning circuit-breaker strikes on every
'Unknown tool' round-trip); stale fingerprint logged
- cache-load path now runs _scan_mcp_description like the eager path
(cache file is user-writable JSON; defense-in-depth)
- write-through skips the disk rewrite when the entry is unchanged
(a flapping stdio server was rewriting byte-identical JSON per
revival)
- _lazy_server_fingerprints no longer write-only dead state (consumed
by the reconciliation logging)
444 mcp tests green (440 pre-fold + 4 new guards); phantom-dereg and
write-skip mutation-checked.
Wires the fingerprint-keyed schema cache (previous commit, @Vansh5632's
design from #56832) into the startup path, re-derived onto main's
current connect machinery:
- register_mcp_servers: servers with mcp_servers.<name>.lazy=true whose
config fingerprint matches a valid cache entry register tools from
cache WITHOUT spawning; miss/stale falls back to eager connect.
- First tool use routes through _ensure_lazy_server_connected, which
composes with the connect cooldown (#50394) and _server_connecting
dedup rather than duplicating the connect path.
- resource/prompt utility handlers (list_resources/get_prompt) also
connect-on-first-use — closes the gap flagged in the original
sweeper review.
- Write-through: a live connect refreshes the cache entry.
Config gate is per-server, default OFF, matching the
idle_timeout_seconds key pattern. 24 lazy/cache tests + 440 mcp-wide
green; mutation-checked (cache-read disabled -> registration test
fails; connect bypassed -> 3 first-use tests fail).
Salvage of #48637 (Fixes#48628). On a NixOS-style install the venv's
site-packages lives in the read-only store, so ensure()'s
uv -> pip -> ensurepip ladder spends ~15s bootstrapping ensurepip only
to fail against a target it can never write. Fail fast with an
actionable message pointing at the system package manager.
Retargeted onto current main (the PR's base predates the durable-target
subsystem by ~8.1K commits) with two corrections to the original:
- Gate on _lazy_install_target() is None. The container deployment sets
HERMES_MANAGED=true AND HERMES_LAZY_INSTALL_TARGET (a writable
volume); the original guard would have blocked installs that path
legitimately satisfies, breaking the NixOS-container mode.
- Reason string starts with 'unsupported ' because
refresh_active_features classifies FeatureUnavailable by that prefix;
the original wording made 'hermes update' report a hard failure
instead of a skip.
Placed after _unsupported_feature_reason so a platform-specific reason
(more actionable) wins, and so ensure() agrees with
refresh_active_features, which pre-checks that same function.
Extracted from #54314 (@flag0x369), re-derived onto current main: macOS
ships bash 3.2 as /bin/bash, which lacks $BASHPID entirely — the
variable expands to empty string, collapsing every concurrent writer's
'unique' temp path onto the same file (torn snapshot writes under
concurrency). mktemp allocates per-writer unique paths portably.
Live-verified: /bin/bash -c 'echo $BASHPID' prints empty on this box.
The workdir allowlist regex was ASCII-only, so perfectly normal
non-ASCII workdirs (Chinese Obsidian vault paths, accented dirnames)
were rejected with 'disallowed character'. Replace the regex with a
per-character check that accepts Unicode letters/digits (str.isalnum)
plus the same safe ASCII punctuation set, while still rejecting shell
metacharacters, control characters (newlines/tabs), and NUL.
Salvaged from PR #54314.
Co-authored-by: kshitij <82637225+kshitijk4poor@users.noreply.github.com>
Recalled session messages can carry raw ANSI escape sequences (e.g.
archived terminal output), which then re-enter the model's context.
Strip them in _shape_message before content is truncated/returned,
reusing tools.ansi_strip.strip_ansi.
Re-applied onto current main (the original hunk predates the
max_content_len truncation in _shape_message; stripping happens on the
raw content before truncation so escape bytes never count against the
budget). Extracted from #40276.
A raw newline in the _CMDPOS start-position class made ANY multi-line
quoted argument look like a command boundary, so hermes send message
bodies, multi-line git commit -m messages, and heredoc text that merely
mentioned dangerous command names tripped the unconditional hardline
blocklist and could not run at all.
Mask newlines inside single/double quotes (detection-only, mirroring the
quote tracking in _iter_shell_command_starts) before building detection
variants. Real threats keep blocking: unquoted newlines stay command
separators, command substitutions inside quotes still anchor, and
_mark_command_starts still re-inserts newlines at genuine quote-aware
command starts. Masking runs on the RAW command before normalization,
which strips escapes and would otherwise corrupt quote state.
Regression tests cover both directions: multi-line quoted data passes
(hermes send, git commit -m, heredocs); bare/chained/substituted
shutdown-class and rm-floor commands still block.
The top-level delegate_task description repeated content the model already
receives through parameter descriptions: the concurrency limit (tasks param),
the full nesting clause (role param), context-passing guidance (goal/context
params), and background semantics (background param). Every API call paid for
the duplication (~4,000 chars).
The description now carries only what exists nowhere else in the schema:
use/don't-use routing (execute_code, cronjob), the no-poll rule, the
non-durability warning, the self-report verification contract with concrete
verbs, the language-passing example, the leaf blocked-tool list, and model
inheritance. 3,963 -> 1,704 chars (~570 tokens saved per API call), and the
top-level text is now static (dynamic limits flow only through the two param
descriptions, which are already rebuilt per get_definitions() call).
A/B benchmark across 4 models (gpt-4o, gpt-4o-mini, claude-haiku-4.5,
llama-3.3-70b) showed the naive compaction in PR #72813 regressed weaker
models on exactly the passages it cut (side-effect verification 8/8->0/8 on
gpt-4o-mini; language passing 3/3->0/3 on haiku-4.5). This version keeps
those benchmark-sensitive hooks verbatim.
Tests pin the contracts at keyword level (not prose-literal) plus a size
ceiling, and verify dynamic limits still reach the model via the tasks/role
param descriptions.
Refs #72737, supersedes the delegate_task half of PR #72813.
The universal sync fallback in stream_tts_to_speaker ran strictly serially
per sentence — synthesize, play, and only then start synthesizing the next
sentence — so every sentence boundary added a full synthesis-time of dead
air. Chunked streamers (elevenlabs/openai/gemini/xai) already avoid this;
every other provider (edge, piper, plugin providers) paid it on each reply
in voice mode and the wake-word loop.
_SyncSentencePipeline overlaps the two: one single-threaded synthesis
worker (sentences stay FIFO; providers never see concurrent calls from
this loop — same effective concurrency as before) feeds one playback
worker through a small bounded queue, so sentence n+1 synthesizes while
sentence n plays. Lookahead is bounded (backpressure + at most a couple of
temp files), stop_event short-circuits both stages, synthesis failures are
isolated per sentence, temp files are always unlinked, and the finally
block flushes the pipeline BEFORE tts_done_event fires so continuous voice
mode never reopens the mic over its own voice. synthesize/play are
resolved late so existing monkeypatch-based tests work unchanged.
Measured with a real local model provider (OmniVoice plugin, Apple
Silicon), same 3-sentence reply, playback simulated at the produced clips'
true durations, best-of-2 interleaved runs under identical load:
serial pipelined
time to first word 10.8s 4.4s
mid-reply dead air 11.2s 1.8s (second gap: 0.03s)
full reply wall 33.2s 17.0s
Tests: 4 new (timestamp-proven overlap, order + per-sentence failure
isolation, stop skips queued playback, temp-file hygiene); the existing
sync-fallback and display-callback tests pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When an approval prompt expired without a response, every CLI-side path
collapsed the timeout into the same 'deny' choice as an explicit user
refusal, so the agent was told the user denied the action when the user
simply never answered. The gateway wait already distinguished the two
('timed out without user response... Silence is not consent.'); this
brings the CLI/TUI/ACP surfaces to parity.
- prompt_dangerous_approval(): input()-path expiry now returns a distinct
'timeout' choice (still fail-closed).
- cli.py _approval_callback + hermes_cli/callbacks.py approval_callback:
deadline expiry returns 'timeout' instead of 'deny'.
- check_all_command_guards / _run_approval_gate CLI tails: 'timeout' maps
to outcome='timeout' with a 'timed out without user response... Silence
is not consent.' BLOCKED message (matching the gateway wording);
explicit deny keeps outcome='denied' and gains user_consent=False for
shape parity.
- computer_use: 'timeout' verdict threads through the CLI adapter and
yields a 'prompt timed out — the user did not respond' error instead of
'denied by user'.
- ACP permissions bridge: FutureTimeout returns 'timeout' (other failures
still 'deny'); elicitation maps 'timeout' to 'cancel' like the gateway's
unresolved outcome; codex wire mapping documents deny/timeout→decline.
- write_approval already treats unknown choices as 'stage, not drop', so
a timeout now stages the memory write instead of silently refusing it.
Every timeout path remains fail-closed — the action never runs; only the
classification reported to the agent changes.
Every tool schema ships on every API call. The terminal schema was
5,641 chars (~1,410 tokens) and execute_code 2,842 (~710) — the two
largest core tools, padded with repeated war stories and triple-stated
rules. Schema token audit across 88 tools: ~33k tokens total.
This trims prose while preserving every hard rule (each still stated
exactly once):
- terminal description 2,324 -> 1,233 chars: tool-redirect lines
collapsed to one sentence; background/notify guidance deduplicated
(was stated in desc + 2 params); PTY/pager rules merged.
- background/notify_on_complete/watch_patterns params 692/508/1,114 ->
~330/250/490 chars: kept the mutual-exclusion contracts, the
rate-limit consequence, and the bounded-vs-long-lived distinction;
dropped narrative repetition.
- execute_code description tightened (helper docs inlined to one line
each; when-to-use kept).
Net: terminal schema 5,641 -> 3,386 chars, execute_code 2,842 -> 2,522
— ~700 tokens saved on EVERY request with the terminal+code toolsets.
One test updated (pinned a removed phrase; now pins the rule's new
phrasing).
Truncated terminal output was information LOSS: the middle was gone
and the only recovery was re-running the command (data: 1,394
truncation markers in a 250k-call window, with re-runs and grep
retries chained behind the big ones).
Truncation is now deferred retrieval (opencode/goose/qwen-code
pattern, codex's original_token_count idea):
- tools/environments/base.py: _BoundedOutputCollector gains an
optional spill tee — when foreground output overflows the capture
window, the FULL stream is teed to
~/.hermes/cache/terminal-output/out-*.log (lazy file creation with
backlog backfill, 5MB hard cap, 7-day opportunistic cleanup,
disk errors never break execution). All three _wait_for_process
returns attach {output_total_chars, full_output_path} via a shared
finalizer.
- tools/terminal_tool.py: redacts the spill with the same
redact_terminal_output pass as the visible output (no secret
persists unmasked), then surfaces output_total_chars,
full_output_path, and a truncation_note pointing at
search_files/read_file instead of a re-run.
Non-truncated results are byte-identical; internal unbounded
consumers (file-ops cat reads, RPC reads) are untouched (spill only
arms with bounded_capture=true).
'Found N matches for old_string' (190+ occurrences in a 250k-window)
previously reported only the count, forcing a re-read of the file to
find the occurrences before retrying. The error now appends up to 5
'L<line>: <snippet>' rows (80-char cap per snippet, overflow noted),
so the model can disambiguate in ONE follow-up — add neighboring
context or choose replace_all — without the intermediate read.
Applies to both patch modes (replace + V4A) since they share
fuzzy_find_and_replace.
#77128 fixed the orphaned zero-match probe but shipped no tests, so the
same class of break can recur: an early return anywhere in the rg branch
silently makes the whole steering tier unreachable.
Asserts hint wiring per search engine with the probe stubbed to a
sentinel (the probe itself needs rg, so a real-text parity assertion
fails on the grep leg for an unrelated reason), plus the negative case
and the rg newline-warning skip that the early return originally
existed to preserve.
Sabotage-verified against 794d6c434e: restoring the early return turns
4 red; a naive fix that also drops the rg newline guard turns 1 red;
attaching the hint when matches exist turns 2 red.
The top execute_code failure shapes in production (state.db mining)
are sandbox-contract confusions, not logic bugs: importing tools that
aren't in the sandbox from hermes_tools (23x in one window, incl.
importing the built-in helpers json_parse/shell_quote/retry), importing
third-party packages absent from the sandbox interpreter (matplotlib
6x), and indexing tool-result dicts as strings. The stderr traceback
alone sends models into re-diagnosis loops.
Failed scripts (exit != 0) now carry one actionable 'hint' field:
- unavailable hermes_tools import -> lists the tools that ARE
importable in this session + points to normal tool calls otherwise;
- built-in helper import -> 'no import needed, call it directly';
- ModuleNotFoundError -> 'sandbox has stdlib only; use terminal() with
the project venv for third-party packages';
- string-indexing errors -> 'tool functions return dicts, do not
json.loads them'.
Bounded 4KB stderr scan, first match wins, never raises; successful
scripts and unknown failures are untouched.
A regex \n (or a raw newline) in a search_files content pattern cannot
match in rg's default line-oriented mode. It previously either
hard-errored ('the literal "\n" is not allowed in a regex' — 17
occurrences in the production window) or, after the newline-warning
patch, returned 0 matches with an explanation — either way the model's
cross-line search intent required a manual workaround.
The rg engine now detects the pattern shape (_pattern_has_regex_newline,
already used by the warning path: odd-backslash \n escape or raw
newline; escaped \\n literals excluded) and enables -U/--multiline up
front, noting the mode switch in the result warning. Plain patterns are
untouched; the old line-oriented explanation is retained for the grep
fallback engine, which has no multiline mode.
skill_view re-sent full skill content on every call: ~286k tokens of
verbatim repeat views in a 400k-msg production window (one session
loaded the same skill 9 times), and a single repeat view of a large
skill costs ~25k tokens.
Mirrors read_file's proven unchanged-stub pattern: a per-task cache
keyed on (resolved name, file_path) with an mtime+size fingerprint of
the served file. On a repeat view of an UNCHANGED file, return a short
stub pointing at the earlier result. This does NOT violate the
skills-are-loaded-fully rule — the stub only ever replaces content
that is already fully present earlier in the same conversation, and:
- any on-disk change (patch, external edit) invalidates the entry;
- context compression clears the cache (wired next to
reset_file_dedup in conversation_compression.py) so post-compression
re-views return full content;
- setup-needed views are never deduped (readiness can change without
the file changing);
- no task_id -> no dedup; caches are task-isolated; 200-entry cap.
Live E2E: repeat view of hermes-agent-dev 99,739 chars -> 374-char
stub.
write_file confirmed only SIZE (wc -c) after writing — never content.
Models compensated by re-reading files immediately after writing them
(154 verify-reads in a 400k-msg production window), and a corrupted
write (truncated pipe, backend FS oddity) could silently pass.
The write path now compares the on-disk sha256 against the intended
content (one shell call). Three outcomes:
- match -> result carries verified: true; the schema tells the model
an explicit contract: do NOT re-read to check the write landed.
- mismatch -> hard error ('The write did not persist correctly'),
mirroring patch_replace's existing post-write verification.
- backend can't hash (no sha256sum) -> flag omitted, write unaffected.
Hashes the shim-adjusted content (after CRLF/BOM preservation) so
Windows-line-ending and BOM round-trips verify correctly; surrogatepass
encoding matches the rest of the codebase's hashing of model text.
When old_string survives all 9 fuzzy strategies without a match but
the closest candidate line matches after stripping whitespace, the
failure is whitespace-shaped (tabs vs spaces, indent depth). The
did-you-mean hint now appends a two-line diagnosis with leading
whitespace made visible:
Whitespace difference detected (→ = tab, · = space):
file has: →def start(self):
you sent: ····def start(self):
Use the exact whitespace shown in 'file has'.
Pattern ported from crush's diagnoseMismatch (agent-codebase survey) —
it converts the residual dead-end error into a one-turn fix. Only the
leading run is visualized (interior spacing stays readable); content-
shaped misses and raw-exact candidates are unchanged.
Follow-up to the recovery-recipe commit on this branch, per review:
instead of only TELLING the model to re-author the payload via
write_file (2 turns), materialize the blocked command to
~/.hermes/cache/blocked-scripts/blocked-*.sh and point the recovery
at it directly: 'saved to <path> - review it, then run
terminal(command="bash <path>")' (1 turn).
Safety posture is unchanged or better:
- Nothing is executed here; the file is only written.
- The bash <path> follow-up goes through the normal execution
pipeline, including the referenced-script content guard, which
inspects script files named in commands - the payload is MORE
visible to policy than it was inline.
- Genuine hardline blocks (destructive ops) never save anything
(test-asserted).
- Save failures fall back to the previous manual write_file recipe.
- 7-day opportunistic cleanup of saved payloads.
Two block classes from production mining (250k-call window) that
models answered with blind rephrase-retries:
1. Parser-limit / malformed-payload hardline blocks (198x): these fire
on oversized inline payloads (heredocs, giant one-liners), not on a
forbidden operation - but the message read like a permanent ban.
The block now appends: 'RECOVERY: ... write the script to a file
with write_file, then run bash /path/script.sh - do not retry
inline.' Genuine hardline blocks (destructive filesystem
operations) are unchanged.
2. Backgrounding-wrapper blocks (200x): the guidance now spells out
the exact corrected call shape - 're-send WITHOUT the wrapper as
terminal(command="<cmd>", background=true,
notify_on_complete=true)' - instead of describing the feature
abstractly.
process(action='wait') hitting its window returned status='timeout'
with a terse note — models read it as an error and re-issued identical
waits (process is the #1 exact-duplicate tool call in production: 511
dupes in a 400k-msg window; wait is 57% of all process actions).
The timeout result now carries:
- process_running: true — machine-readable 'this is a status, not a
failure'
- an explicit note: 'Wait window of Ns elapsed — the process is still
running. This is not an error. Uptime: Ms.' plus the right next step:
when notify_on_complete is set, 'you will be notified on exit — do
more work instead of waiting again'; otherwise a pointer to
notify_on_complete for next time.
- the clamp note (requested > max) now composes with the status note
instead of replacing it.
Exited/interrupted results are unchanged.
Third zero-match steering tier: when a content search finds nothing in
visible files, probe once with rg --hidden --no-ignore --count-matches.
If the pattern exists only in dotdirs or gitignored files, the result
says so and tells the model to search the hidden path explicitly.
Found live by the benchmark battery: a task with a match inside
.hidden/ returned a bare 0 and the model missed the file entirely in
2/3 baseline runs.
Two dead-turn classes from production mining (state.db, recent window):
1. 13.9% of 19.6k content searches return 0 matches with no steering.
Now a 0-match content search runs one cheap rg -i --count-matches
probe (plus an rg -F probe when the pattern has regex metachars) and
attaches what it found: '0 exact matches, but N case-insensitive
matches — casing may be wrong' / 'N literal matches — metacharacters
need escaping'. True zero-match results stay clean (no noise).
2. 122 'Path not found' failures came from models passing several paths
in ONE path string ('dir1 dir2 dir3', comma lists). Instead of
failing wholesale, split the string, search every path that exists,
merge results, and report skipped parts in a warning. Single-path
misses keep the existing Similar-paths hint; all-missing multi-path
strings still error.
Both probes are bounded (count-only rg, 30s timeout, max 2 invocations)
and wrapped so a probe failure can never break the search result.
Production mining (state.db, 400k-msg window): 60.2% of 104k terminal
calls carry a defensive 'cd X && ' prefix (~925k tokens of pure prefix)
and 2,462 failed calls led with cd — the model cannot see cwd state, so
it re-asserts it on every call and runs pwd/ls diagnostics after
directory changes.
The result dict now includes a 'cwd' field whenever the session cwd
after the command differs from the cwd it started in (cd, pushd,
chained cd). Stable-cwd commands are unchanged (no field, no noise).
Per-command workdir overrides stay transient by contract and never
echo. Schema note added so models learn to trust session cwd instead
of prefixing. Pattern borrowed from crush's <cwd> injection.
realpath comparison avoids false echoes through symlinks; the echo is
wrapped defensively so a backend without .cwd can never break the
result path.
The #1 patch failure class in production (state.db mining, 250k-window)
is a re-send of an edit that already landed: 'old_string and new_string
are identical' (299 occurrences) plus a share of hunk-not-found errors
where the new text is already in the file. These errored, sending
models into re-read/re-patch loops.
New tools/fuzzy_match.is_already_applied(content, old, new) — a
conservative check requiring (1) non-trivial new_string (>=8 chars),
(2) EXACT presence of new_string, (3) old_string gone (unless
identical). Wired into three sites:
- patch_replace (replace mode): returns success + no_change: true +
an explicit note instead of the identical-strings / no-match error.
- V4A validation phase: an already-applied hunk validates as a no-op
so multi-hunk patches no longer fail wholesale when one hunk landed
in a prior call.
- V4A apply phase: mirrors the same skip so the two phases agree.
Genuine no-matches (new text absent) and half-applied renames (old
text still present) keep their error behavior — covered by tests.
Production mining (state.db, 28.5k read_file calls in the recent
window) shows 74.3% of reads truncated — nearly all by the 500-line
default, not the char budget (22,443 line-limit vs 13 char-budget
truncations). That churn produced 12,229 redundant re-reads, and after
a truncated result the most common next move was fleeing to terminal
cat/sed (4,608 times) — the pagination contract was not trusted.
Median truncated file is 2,422 total lines, so a 2000-line default
makes 44% of today's truncated first-reads complete in one call while
the unchanged ~100K-char budget still caps worst-case result size
(same ceiling as before: 500 lines x 2000-char line cap = the same
100K). Schema max was already 2000.
Touchpoints: DEFAULT_READ_LIMIT + both read_file signatures
(file_operations.py), read_file_tool + schema text (file_tools.py),
execute_code sandbox stub docs (code_execution_tool.py), 3 tests
pinning the old default.
When a command exits non-zero, scan the first 4KB of output for
well-known failure shapes and attach one short, actionable recovery
hint to the tool result ('hint' field):
- gh 'Unknown JSON field' (9.2k occurrences in a 250k-call window)
- git merge conflicts (1.2k) — stop verbatim retries
- command not found (1.0k), incl. python->python3 and pip->pip3
- ModuleNotFoundError (739) — venv activation guidance
- 'already exists' (633), gh rate limits (133), permission denied
- exit-code-only tier: 124 timeout, 126 not-executable, 137 SIGKILL
Hints are suppressed when the existing exit_code_meaning tier already
explains the code (grep=1 etc). Pattern order = production frequency
from state.db mining; first match wins; pure function, no I/O.
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
Review follow-ups on the #25387 salvage:
1. CRITICAL: a cached miss survived out-of-band file creation (terminal
command, external process) for the full 60s TTL — breaking the common
agent pattern 'check for file -> create it -> read it' (live-repro'd).
Serve-side existence guard: one ~free stat before serving a cached
miss; if the path now exists the entry is evicted and the real read
runs. Also fixes the search-root variant (write under a cached-missing
directory). Both mutation-checked.
2. notify_other_tool_call now clears the task's not_found entries too
(belt: the dispatcher calls it for every non-read tool).
3. Tracking parity: the record sites no longer early-return. On upstream,
error results flow through consecutive-loop detection and dedup
bookkeeping; short-circuiting skipped that and broke
TestDedupInvalidationTaskResolution when preceded by
TestSilentFileMisplacementE2E (bisected: the early return at the
read record site was the trigger). Recording is now
side-effect-identical to upstream; serving from the cache remains the
optimization. Also reuse the already-computed _resolved instead of
resolving a second time.
When read_file or search hits a non-existent path, ShellFileOperations
spawns a subprocess to stat the path and another to walk the parent
directory for "did you mean..." suggestions. A typo'd path retried 13
times (observed in the wild) costs 26 subprocess invocations + 13 ls
walks for a result we already know.
Add a per-task negative-result cache keyed by (op, resolved_path) with
a 60s TTL and a hard cap of 500 entries. On hit, return the cached
error JSON immediately and skip the subprocess + suggestion walk.
The cache is namespaced by operation ("read" vs "search") because the
two callers return different error JSON shapes ("File not found:" vs
"Path not found:"). Eviction:
* TTL (60s) — short, so a path that appears later isn't masked.
* write_file / patch on the same path — _invalidate_dedup_for_path
now also drops the negative-cache entry so a freshly-written file
is read from disk on the next call instead of returning a stale
"not found" stub.
Tests in tests/tools/test_file_tools.py cover:
* read cache hit skips the subprocess on retry
* cache is per-task (no cross-task pollution)
* successful reads do not poison the cache
* search cache hit skips the subprocess on retry
* read and search caches are namespaced (different error shapes)
* write_file invalidates the read negative cache
* TTL expiry evicts stale entries
Salvaged from #56982 (@rayjun): the live piece of the PR. The
hermes_cli/config.py get_env_value scope-honoring change and its
test_env_load_cache.py tests were already merged via ed1170cd8b
(#76462) and are dropped here.
tools/xai_http.py::get_env_value wrapped the scope-aware
hermes_cli.config.get_env_value in except Exception + a raw os.environ
fallback — swallowing UnscopedSecretError and borrowing the process
env, so a multiplexed xAI credential read could silently pick up
another profile's XAI_API_KEY. Narrow the except to ImportError (the
only legitimate degraded case) so get_secret's verdict propagates: an
unscoped multiplexed read fails closed, and a scoped miss returns the
default instead of the foreign environ value.
Co-authored-by: rayjun <rayjun0412@gmail.com>
Whole-bug-class follow-up to the profiles/ gate: cron/, platforms/
pairing, and legacy pairing/ ran chown_hermes_tree unconditionally on
every boot with the identical warm-boot cost profile. Same
tree_has_non_hermes_owner gate; find evaluates the top directory first
and -quits on the first mismatch, so a mis-owned tree short-circuits in
O(1) while a clean tree pays one read-only walk instead of a full
chown -R inode rewrite.
The terminal-command guard path loaded config 2-3x per invocation via
load_config(), which pays a defensive deepcopy of the entire config on
every call (~356us of the ~376us warm-cache cost measured on a real
config.yaml). All six swapped call sites were audited read-only — every
caller takes scalar reads or iterates the returned structures; none
mutate (the save path at save_permanent_allowlist keeps load_config) —
so they now use load_config_readonly(), the API built for exactly this
(precedent: #74211, #74322; the one unsafe-site lesson from #56085's
salvage is covered by the mutation audit and a cache-integrity test).
Measured (real config.yaml, warm cache): load_config 376.0us ->
load_config_readonly 19.9us (18.9x); full guard pass
check_all_command_guards('ls -la','local') 930.7us -> 241.8us (3.85x).
Tests: new test_approval_config_readonly.py drives the real functions
against a temp HERMES_HOME — readonly call counts per function, a
no-deepcopy pin for the full guard pass, and cache-identity/integrity
checks. Existing test mocks retargeted from load_config to
load_config_readonly (same injection intent). Note: 6
test_approval_mode_parity failures are pre-existing ordering flakes —
identical with the change stashed on clean main.
kill_started_since duplicated kill_all's collect-under-lock/kill-outside-lock
loop line for line; it is now a thin delegate through new kill_all kwargs
(exclude_ids, source, consume_output). Public signatures unchanged — existing
callers and test monkeypatch seams keep working. kill_process's docstring now
names the deliberate consume_output=True exception for abandoned-turn reaping
so the deviation isn't 'fixed' later.
An agent turn can spawn a long-running background subprocess (e.g.
`next build`) and later be abandoned via inactivity timeout, /stop,
/new, or a client disconnect. Before this fix the gateway interrupted
the agent loop but never touched the subprocess: it kept running
inside the gateway's cgroup, unbounded, until memory pressure starved
the event loop and made every platform/cron look hung (#76115).
The process registry already knew how to kill a process tree — the
missing piece was per-turn ownership: nothing distinguished a process
that predates the turn (must survive), a process the turn started and
finished successfully (must survive), and a process an abandoned turn
left running (must be reaped).
- tools/process_registry.py: snapshot_running_ids() captures a turn's
starting baseline; kill_started_since() reaps only IDs created after
it, scoped to one task_id.
- gateway/turn_context.py: TurnContext carries process_task_id +
process_baseline so the timeout/interrupt paths can reach them.
- gateway/run.py: baseline is snapshotted right before the turn's
executor task starts; the inactivity-timeout path and the explicit
/stop|/new|disconnect interrupt path both reap via the same helper.
A daemon-thread watchdog backs up the asyncio-based timeout poll,
since a starved event loop is exactly the failure mode this bug
causes. The turn's own worker clears its ownership markers the
instant it finishes, closing a race where a /stop landing right
after normal completion could reap a background process the turn
deliberately left running.
Related but insufficient on their own: #37454 (cgroup ExecStopPost
reaper only fires on service restart) and #68915 (orphaned-pipe
grandchild detection, a registry bug not a turn-lifecycle gap).
Neither ties process cleanup to turn abandonment.
- logger.warning when the 6h ceiling stops the heartbeat (matches the
delegate_task stale-stop precedent) so the eventual watchdog reap is
explainable from logs instead of silent
- new mutation-checked test: past the ceiling the heartbeat stops while
the job still completes
- clearer assertion messages (surface res on failure)
- heartbeat loop continues past a raising activity callback instead of
silently stopping (matches delegate_task / touch_activity_if_due
swallow-and-continue semantics) — one transient error must not drop
watchdog protection for the rest of a long job
- hard 6h elapsed ceiling so a wedged job under HERMES_CRON_TIMEOUT=0
(unlimited child watchdog) cannot mask the gateway watchdog forever
- public get_activity_callback() accessor in tools/environments/base.py
instead of importing the private _get_activity_callback cross-module
- tests: deterministic heartbeat test (event-gated, no timing sleep),
no-callback test now asserts the thread is truly never created, new
exception-survival guard; dead started event removed
- fix comment: delegate_task heartbeat cadence is 30s, not 10s
Follow-up to the salvaged #76000 guard:
- refuse installing a skill INTO an existing skill directory (hybrid
skill-plus-category dirs whose later update/uninstall rmtree would
destroy the nested skill — sibling case of #75983)
- refuse a stray regular file at the install path with the caller's
ValueError contract instead of an uncaught NotADirectoryError
- regression tests: nested-only category (skills at depth >= 2),
category-inside-skill, file collision
Start from main's 13 tests (renamed test_openai_available_reflects_key
to test_openai_available_reflects_audio_key_resolution, added 4 new
tests for xai oauth, elevenlabs secret resolver, openai configured
key, stream cap). Append 12 new regression tests from PR #71084 for
the prefetch pipeline, PCM misalignment, and PortAudio resilience.
Patch platform.system in stream-path tests for main's macOS guard.
Three lower-severity core-tool robustness fixes from a targeted audit, each
reproduced live:
1. terminal_tool did not validate non-positive timeouts. 'timeout or default'
silently coerced 0 to the config default (0 can't mean 'no timeout'), and a
negative value is truthy so it flowed into 'deadline = now + timeout' and
fired an immediate '-Ns' timeout. Reject timeout <= 0 with a clear message.
2. fuzzy_find_and_replace accepted a whitespace-only old_string, which matches
trivially (blank line / run of spaces) and mass-replaces under replace_all
or raises an opaque ambiguity error. Reject it alongside the empty check.
3. The '/private/var/' sensitive-path prefix over-blocked ALL macOS temp-file
writes: , /tmp, and /var/folders realpath into /private/var/folders
on macOS (and paths are resolved through symlinks), and /private/var/tmp is
a normal temp dir. Narrowed to the genuinely-sensitive subtrees
(/private/var/db, /private/var/root); /etc and /private/etc stay blocked.
All verified with sabotage-checked regression tests. 85 terminal/fuzzy/file
tests pass; normal timeouts, legit replacements, and /var + /boot + /etc
blocking are unaffected.
Two V4A parse/validate bugs found in a core-tools audit, reproduced live:
1. CRLF patch body injected stray carriage returns. parse_v4a_patch split
on '\n' only, so a CRLF-encoded patch kept '\r' inside every HunkLine
and wrote mixed line endings into an LF file; the anchored Begin/End
markers could also fail to match because of the trailing '\r'. Strip a
trailing '\r' from each line at split time.
2. Move-then-Update of the same file was rejected. _validate_operations read
the UPDATE target from disk before the MOVE ran, so 'Move a->b' + 'Update
b' failed validation with 'b: file not found'. Added a small pending-move
overlay so UPDATE/DELETE/MOVE reads during validation see prior ops'
effects (moved-in destinations resolve, moved-away sources read as gone),
while a genuine 'destination already exists' conflict is still caught.
Both verified with sabotage-checked regression tests. 113 patch/fuzzy/file
tests pass.
Strategy 9 (context_aware, the last-resort fuzzy strategy used by
patch_replace, V4A UPDATE hunks, and skill_manage) had two serious flaws,
both reproduced live against current main:
1. CORRECTNESS: it accepted a block when >=50% of its lines were >=0.80
similar. A 2-line pattern with one real line and one garbage line matched,
silently deleting the non-matching line and persisting a wrong edit as
success. Now requires the first AND last lines to anchor-match and EVERY
non-blank pattern line to be >=0.80 similar — one garbage line disqualifies
the block.
2. PERFORMANCE: it scored every content window with per-line SequenceMatcher,
so every failed match paid O(file_lines x pattern_lines) — measured ~5.5s
for a single 40-line no-match on a 10k-line file, per hunk. The first/last
line anchor pre-filter skips non-candidate windows: same case now ~160ms
(34x faster).
Also gate replace_all: a similarity-based strategy (block_anchor,
context_aware) with multiple matches under replace_all would overwrite every
approximate block, not just exact ones. Now refused with a clear error
directing the caller to precise text.
All verified with sabotage-checked regression tests (fail against the old
50% logic). 158 file/patch/fuzzy tests pass; legit fuzzy edits (indent drift,
unique near-match) unaffected.
Two DATA-LOSS bugs in ShellFileOperations found in a core-tools audit,
each reproduced live against current main:
1. Non-UTF-8 file content silently corrupted on read->write. The terminal
env decodes stdout with errors='replace', so a latin-1/8859 file's bytes
arrive as U+FFFD before _is_likely_binary inspects them. U+FFFD is
'printable', so the >30%-non-printable check never flagged it, and the
agent would read the mojibake and write it back, permanently replacing the
original bytes. Fix: treat a sample containing U+FFFD as binary (read-only).
2. Writing through a symlink destroyed the link and orphaned the target. The
atomic temp-file + 'mv -f' swap replaced the symlink itself with a plain
file; the real target was never updated. Fix: resolve the link with
readlink -f/realpath first and recompute the temp dir from the resolved
target so the mv stays same-filesystem atomic. Broken links fall back to
the original path (no regression).
Both verified with sabotage-checked regression tests (fail without the fix).
Proper UTF-8 text (incl. non-ASCII) and plain-file writes are unaffected.
Two independent HIGH-severity correctness bugs found in a core-tools audit,
each reproduced live against current main:
1. Read-dedup was never evicted after a write on non-default tasks.
_invalidate_dedup_for_path looked up the read-tracker under the correct
task_id but resolved the path with _resolve_path(filepath) — which
DEFAULTS task_id='default'. The dedup cache is keyed by the task-resolved
absolute path, so for any task whose workspace cwd differs from the process
cwd (every -w worktree / Desktop / ACP session using relative paths) the
computed key never matched and the stale entry was never removed. A
read_file after a write_file/patch could then return the OLD content stub
when mtime coincided. Fix: pass task_id through.
2. A per-command workdir override permanently hijacked the session cwd.
The post-command dual-write unconditionally recorded env.cwd (stamped to
the transient workdir) into the durable session-cwd store, so every later
command that omitted workdir inherited the one-off directory — contradicting
the documented 'Working directory for this command' contract. Fix: skip the
session-cwd record when workdir was explicitly supplied.
Both verified with sabotage-checked regression tests (fail without the fix).
patch_tool resolved V4A header paths against the task workspace for
locking, staleness, and reporting, but handed the original (often
relative) patch text to file_ops.patch_v4a — which re-resolved headers
against the backend env's own cwd. When the two diverge (the git-worktree
cwd bug), a relative header landed in a different directory than
everything the tool locked and reported: a silent wrong-file write.
Rewrite Update/Add/Delete/Move File headers to the resolved absolute
paths before apply, only for host-filesystem backends (container/remote
namespaces keep their own paths). Header patterns mirror patch_parser
(no-space ***Update File: form) and cover Move File: src -> dst.
Salvage of #53176 by @necoweb3, reimplemented onto current main (the
original branch predates the sensitive-path/Move-header extraction and
per-path locking now in patch_tool).
Co-authored-by: necoweb3 <sswdarius@gmail.com>
The boundary scan in parse_v4a_patch used substring matching, so a
content line mentioning "*** End Patch" (docs about the patch format,
nested patch text) truncated the patch, and "*** Begin Patch" in
content reset the start boundary — silently dropping already-parsed
operations while reporting success. Match only whole-line markers at
column 0, preserving the no-space "***Begin Patch" tolerance.
Replace the bare /init ENTRYPOINT with entrypoint-dispatch.sh: exec
/init + main-wrapper when the image owns PID 1, fall back to a direct
stage2 bootstrap (with the s6 helper PATH restored) on wrapped runtimes
where s6-overlay-suexec would abort with 'can only run as pid 1'
(Fly Machines, docker run --init, podman/FreeBSD setups).
Cherry-picked from PR #43763 by @konsisumer, conflicts with current
main resolved (tests/test_dockerfile_tini_compat_shim.py was moved to
tests/docker/, container_boot argv tests were reshaped upstream).
Fixes#38349
Two amplifiers behind the 779K api.osv.dev DNS queries/16h report:
1. tools/osv_check.py: check_package_for_malware() hit OSV on EVERY
call. MCP reconnect ladders, stdio recycles, and parked-server
self-probes re-run the preflight for the same package on every spawn
attempt, so a flapping server became a sustained OSV query/DNS
stream. Verdicts (clean or blocked) are now cached for 1h
(OSV_CHECK_CACHE_TTL to tune); network failures stay uncached so
fail-open never masks a real advisory once connectivity returns.
2. hermes_cli/security_audit.py: cmd_security_audit() ran full
component discovery twice per audit (_count_components + run_audit).
Discovery now runs once via _discover_components() and run_audit()
accepts the pre-discovered list.
Both regression tests fail against the previous code (verified via
sabotage run).
michaelsam94
Teknium
fangliquanflq
Coy Geek
Ahmett101
dsad
kshitij
Erosika
Gille
Brooklyn Nicholson
Dominic Bejar
toprakeker
GodsBoy
PRATHAMESH75
kshitij
Soheil Fakour
chelsealong
liuhao1024
Fly
SmokeDev
Adolanium
Kevin Yin
Jeffrey Quesnelle
Andrew
xxxigm
Alex Fournier
阿泥豆
EndeavorYen
Carbon
Jakub Wolniewicz
f1aggo_macair
Xue-1997
Mahdi Hedhli
xrazai
Kent
Ray
LeonSGP43
spfcraze
joaomarcos
webtecnica
x7peeps
Christopher
tachyon-r
konsisumer