Guard both extra lookups with isinstance(dict) before merging, so a
truthy non-mapping `extra` value (e.g. `extra: "true"`) degrades to the
base Telegram hint instead of raising TypeError and aborting
system-prompt construction. Keep the narrowed except ImportError.
Add an integration test exercising the real config path (HERMES_HOME +
gateway.platforms.telegram.extra.rich_messages) and a regression test
for the malformed-extra fail-open path. The integration test fails on
main and passes with the fix.
Commit b45a217e0 gated the TELEGRAM_RICH_MESSAGES_HINT extension behind
a config read at the top-level ``platforms.telegram.extra.rich_messages``
key, but the Telegram adapter reads the same setting from the canonical
``gateway.platforms.telegram.extra.rich_messages`` path. When users set
the setting in the canonical location (the only one documented), the
lookup returned None and the extension never fired — the model degraded
pipe tables to bullet lists, task lists to plain dashes, and never
produced <details> blocks or block math.
Fix: merge both ``gateway.platforms.telegram.extra`` and the top-level
``platforms.telegram.extra`` with the same precedence the adapter uses
(top-level leaf wins), so config-wizard writes and dashboard-setup keys
are visible alongside the canonical gateway location. Narrow the
except-guard to ImportError so real config-stack failures surface.
The _standalone_send() function (used by the send_message tool for
proactive/scheduled sends) has the same bare `@<id> text` bug that
send() had before PR #44444. SimpleX's `@<x>` syntax resolves x as a
display name, not a contactId — the daemon silently drops messages
when it cannot find a contact named "6".
Use the structured `/_send @<id> json [...]` form, matching what
send_image, send_document, and the send() fix already use.
Fixes#46265
Adds regression test for the case where both disk and live start_time
are known integers but differ (e.g. stale value from a previous run).
The self-PID short-circuit must fire regardless — start_time only
guards PID reuse for *other* PIDs. Inspired by #81495's test case.
Open the selected microphone at its reported default rate and convert each capture block to the 16 kHz frame expected by wake-word engines. Add a regression covering a 48 kHz WASAPI device.
Co-authored-by: clyu168 <clyu168@126.com>
pause_job already sets enabled=false atomically with state/paused_at, but
get_due_jobs only checked enabled — so a contradictory record
(enabled=true + paused_at/state=paused) still fired. That was the 07-30
outage failure mode: list looked frozen, fleet kept merging.
- is_job_runnable / effective_job_state: pause markers gate fire; display
derives from the scheduler-honoured enabled flag so half-paused never
renders as [paused]
- get_due_jobs self-heals enabled=false + logs error on contradiction
- claim_job_for_fire uses is_job_runnable (paused_at counts too)
- list/format paths use effective_job_state
- behavioural tests: pause blocks due fire; half-pause self-disables
With sessions.max_*_messages: 0 the guards previously still ran an
unbounded COUNT (full lineage for resume) — the exact pathological work
disabling them is meant to avoid. Live callers use the raise side
effect only, so return 0 without touching the messages table.
OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
The mid-setup CLI resume path loads only the tip session's rows, so
gate it with a tip-only count instead of the full-lineage count (which
over-rejected heavily-compressed sessions). Transient guard failures
(locked DB, adaptor stores) now log and proceed instead of blocking
resume with a new error.
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
- Heartbeat tests: holder signals a threading.Event after acquiring
_SKILLS_PROFILE_LOCK; the scenario waits on it via run_in_executor
instead of sleeping 50ms and hoping.
- Fix the TestConfigMutationLock comment to describe the probabilistic
slow-save interleave the code actually implements.
- New regression test: PUT /api/dashboard/plugin-providers must hold
_CONFIG_MUTATION_LOCK — a concurrent locked writer survives.
Two follow-ups to the off-loop move, from external review (both verified,
the second larger than reported):
- Config read-modify-write handlers moved to worker threads could now
interleave — _CONFIG_LOCK covers each load/save individually, never the
span between them; the event loop used to serialize these accidentally.
New _CONFIG_MUTATION_LOCK (worker-threads only, so it can never block
the loop) held across the whole load→mutate→save span in all seven RMW
handlers. update_config_raw skipped: it's a full-document replace with
no server-side read, so a lock cannot close its client-side window.
- The review flagged two skills routes still taking _SKILLS_PROFILE_LOCK
on the event loop; a systematic audit of hermes_cli/web_routers/ found
24 on-loop routes (skills 5, mcp 9, tools 10, cron 1). All moved to the
same inner-_run + asyncio.to_thread pattern, mutating ones under the
mutation lock, uniform lock order (_SKILLS_PROFILE_LOCK →
_CONFIG_MUTATION_LOCK). Await-safe _config_profile_scope routes, plain
def routes, and already-threaded routes unchanged.
Regression tests: concurrent theme+font updates both survive (fails with
the lock nulled: "theme write lost to a concurrent font write"); event
loop stays responsive while the profile lock is held during GET
/api/skills. 214 tests passing across the touched suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diagnostics loop watchdog caught GET /api/config freezing the gateway
event loop for >1s, stack-sampled blocking on _SKILLS_PROFILE_LOCK inside
_profile_scope. Any async handler that entered _profile_scope (process-wide
threading lock) or called load_config()/save_config() on-loop could stall
every chat and WebSocket at once while a slow lock-holder ran.
Move 28 such handlers to the existing inner-_run + asyncio.to_thread
pattern (contextvar-safe: the whole scope enter/body/exit stays inside one
worker thread). Handlers using the await-safe _config_profile_scope, plain
def endpoints (FastAPI threadpool), and tui_gateway's contextvar-only
decorator are unaffected and unchanged.
Regression test holds _SKILLS_PROFILE_LOCK in a thread while calling
GET /api/config and asserts an event-loop heartbeat keeps ticking; it fails
against the pre-fix code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).
Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).
Fixes#10809
An auth failure on the very first connect returned out of the run loop
instead of parking. That ended the run task, and the task is the only
listener on _reconnect_event — so the server stayed dead for the life of
the process. `hermes mcp login`, a /mcp refresh, and the 300s self-probe
all had nothing left to wake, and the only cure was a full restart.
_classify_mcp_failure already calls 401/403 "permanent" and documents
that run() parks those immediately; the early return above it meant auth
was the one permanent failure that never got there. Park it with the
others and keep the tailored log line, now pointing at `hermes mcp
login <server>`.
The windows-compat change-detector checked for the literal string
'start_new_session=True', but the systemd scope isolation path
conditionally uses start_new_session=False (the scope creates its own
session/cgroup). Assert 'start_new_session=' instead — the value may
now be a variable.
When Hermes runs as a systemd gateway with MemoryHigh/MemoryMax limits,
local background terminal commands (terminal(background=true)) inherit the
gateway's cgroup. A memory-heavy executor (Codex, tests, Node) can push
the whole cgroup past MemoryMax and trigger systemd-oomd to kill the
ENTIRE gateway — taking down the messaging control plane and silently
losing the active turn.
Root cause: tools/process_registry.py::spawn_local() uses
start_new_session=True (creates a process session/group, NOT a resource
cgroup). The spawned process tree stays in the gateway's systemd cgroup.
Fix: when running under a service manager (detected via the existing
is_gateway_supervisor_process() helper), wrap the pipe-mode spawn command
in 'systemd-run --user --scope --unit=hermes-worker-<id>' so the worker
gets its own transient cgroup. An OOM in the worker then kills only the
worker, not the gateway.
The systemd-run availability is probed once (a no-op /bin/true in a
transient scope) and cached, because the binary can exist on PATH while
the user D-Bus session is unavailable (system services, containers). If
unavailable, fall back to the current start_new_session=True behavior
with a debug log.
Scope: this covers the common background pipe-mode path. PTY mode
(PtyProcess.spawn) is left as future work — it uses a different spawn
mechanism and is used for interactive CLI tools where cgroup isolation
has additional considerations.
- Fix _usage_audit_path() to use _get_hermes_home() instead of hardcoded
Path.home() / '.hermes' (profile-safe resolution, sweeper finding)
- Rewrite skip_background_review tests to exercise finalize_turn() directly
instead of duplicating the guard expression (sweeper finding)
- Fix response_silent audit field to use _is_cron_silence_response()
instead of the buggy SILENT_MARKER substring check it was meant to
replace (simplify-code review finding)
- Remove dead 'model' in locals() guard — model is always in scope
before the try block (simplify-code review finding)
- Extract _stub_agent_for_finalize() helper to eliminate ~40 lines of
copy-pasted agent stubbing in tests (simplify-code review finding)
- Clean up 'Phase 0.5' instrumentation comments
Phase 0.5 of the Hermes Agent token leak mitigation plan: append a single
JSONL line to ~/.hermes/cron/usage_audit.jsonl after every cron LLM
invocation, capturing prompt/completion/total tokens, model, duration_ms,
deliver target, and error (when raised). Read from agent.session_*_tokens
which run_conversation already returns in its result dict.
Without this, we have no measured baseline to attribute token deltas to
subsequent mitigation phases. The plan's hard gate: observability lands
before any mitigation phase.
Writer NEVER raises — wrapped in a single try/except that logs a warning
on any json.dumps / mkdir / open failure so an audit-log bug cannot
break a cron job. Failure-path audit guard via locals() check covers
exceptions that fire before the fire_id is assigned.
No new dependencies, no new env vars (the plan rejected one in v2).
Tests: 7 new unit tests in tests/cron/test_usage_audit_logger.py covering
the success path, missing token info, swallowed writer exception, parent
dir creation, multiple appends, and unicode preservation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 of the Hermes Agent token leak mitigation plan
(ralplan-hermes-token-leaks.md §3.9). Adds a boolean kwarg
`skip_background_review` (default False) to AIAgent.__init__ that
suppresses the end-of-turn _spawn_background_review fork.
Each background review fork instantiates a new AIAgent with its own
~15K input tokens + up to 8 LLM iterations, accumulating ~30K tokens
per event in the worst case. On cron sessions there is no
human-in-the-loop benefit from the review (no skill-creation pressure,
nobody curating MEMORY.md), so the cost is pure waste.
The end-of-turn guard now reads:
if (final_response and not interrupted
and not getattr(self, "skip_background_review", False)
and (_should_review_memory or _should_review_skills)):
skip_memory=True already disables the memory-review trigger; this
flag is the explicit single-switch off for both review paths.
Defaults to False, so behavior is unchanged for gateway/CLI callers
that omit the kwarg.
Tests: 5 new unit tests in tests/agent/test_skip_background_review.py
covering the default value, flag persistence, the gate short-circuit,
the gate fall-through, and a source-text assertion that the cron
scheduler sets the flag to True (separate commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_fallback_session_info` returned `_default_session_cwd()` — the directory the
gateway process happened to start in — so a session resumed without a built
agent told its client the wrong workspace, and the desktop Files pane painted
the wrong project even after the renderer rebound correctly.
Return the session's own cwd and always emit `branch` ("" outside a git repo)
so a client can clear a stale label instead of retaining it. This matches the
contract `_lazy_session_info` already follows a few hundred lines above.
Co-authored-by: ZHJay <ZHJay@users.noreply.github.com>
The main-checkout test compared the two probe roots with raw string
equality. When they differed only in separator spelling, the repo's own
checkout was misclassified as a linked worktree: it fell through to the
worktree branch and was labeled by directory basename. The sidebar then
showed one checkout twice — a dir-labeled lane plus the branch-labeled
`main` lane built from the same sessions.
Compare with `_path_key` so platform path identity decides, matching how
every other path comparison in this module is already keyed.
Tests cover the single-checkout case and the main + linked-worktree case;
both fail before this change (the lane comes back labeled `repo`, not
`main`).
Fleet audit showed these task skills are commonly needed across users;
shipping bundled per Teknium's direction. Docs and tests follow the
bundled paths.
Surfaced during the post-merge review pass on our own #81113 follow-up:
cached_fetch_api_models gained _cache_entry_valid (numeric-'at'
validation) but its sibling cached_provider_model_ids still did
float(entry.get('at', 0)), which raises ValueError/TypeError on a
hand-edited or corrupted provider_models_cache.json row and propagates
uncaught into the /model picker call sites. Same fix, same helper:
corrupt rows are now a cache miss (live fetch), never an exception.
Both wrappers now share the identical validity predicate, closing the
divergence the 'mirrors' docstring promised away.
Also two test nits from the same review: unused OrderedDict import
dropped and the drain-order assertion strengthened to pin LRU-first
FIFO order in tests/gateway/test_agent_cache_pressure.py.
Mutation-checked: restoring the raising float() form makes the new
corrupt-at tests fail.
Rescope: hermes verify fills only the runtime-smoke gap and plugs into
the pieces Hermes already has instead of standing beside them.
- agent/verification_evidence.py: record_verify_run() — explicit ledger
write for hermes verify results (shared _insert_evidence factored out
of record_terminal_result). Passing runs mark the workspace passed
like scripts/run_tests.sh; failures are recorded; --phase/--skip-start
runs are recorded as targeted scope.
- hermes_cli/verify_cmd.py: record results into the ledger on completion
(fail-silent, HERMES_SESSION_ID attribution); on the detect path merge
detect_project_facts verify commands the recipe missed into the
recipe's test list (never applied to a saved manifest).
- agent/verification_stop.py: recipe-aware nudge — when the workspace
has a runnable recipe (start command or .hermes/environment.json),
suggest hermes verify --json as the preferred full check; cheap,
try/except-guarded detection that can never break the nudge path.
- agent/verify/recipes.py: document layer ownership (coding_context =
cheap prompt facts; verify/recipes = deep runtime recipe).
- tests/verify/test_ledger_and_nudge_integration.py: 17 tests covering
ledger pass/fail recording, the closed edit->nudge->verify->satisfied
loop, recipe-aware nudge wording + fail-silence, and the facts merge.
vision_analyze reads container-only images by exec-reading them inside the
sandbox, but unlike terminal_tool it never triggered environment creation. Under
a non-local backend (ssh, docker, ...), a session whose first action was
vision_analyze on a remote path failed with 'no active sandbox session' until an
unrelated terminal command happened to establish the connection.
Add terminal_tool.ensure_task_env(task_id), a public lazy get-or-create that
reuses the terminal tool's own creation machinery, and call it from
image_source._resolve_container_fallback before the in-sandbox read. Extract the
ssh/container config-dict builders so both paths derive settings identically
(no duplication). Best-effort and fail-closed: a failed bring-up leaves the
existing 'no active sandbox' error intact, never a host read.
Fixes#62825
The modal/browserbase test file replaces tools.environments.base with a
SimpleNamespace stub; terminal_tool now imports EnvironmentConnectionError
from that module, so the stub must provide it too.
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.
Now:
- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
carrying a reason + retry_hint. Subclassing RuntimeError keeps every
existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
(missing exe, non-executable exe, daemon timeout, `docker version`
failure).
- terminal_tool catches EnvironmentConnectionError and returns a
structured tool result the model can act on:
{"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
The failed backend is evicted from the environment cache so a later
call retries from scratch — recovery is automatic once the backend is
reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
only infrastructure failures classify as degraded.
Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.
Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
Adds a bounded, read-only health probe per CONFIGURED tool backend, run
only when the user explicitly passes `--live` (real network calls):
- Firecrawl: credit-usage metadata GET (auth check, no scrape spend)
- FAL: models metadata GET (never a generation)
- Browser: headless launch + about:blank + close (full cleanup)
- MCP: initialize + tools/list per configured server (reuses the
`hermes mcp test` machinery in mcp_config._probe_single_server)
- TTS/STT: provider models/voices list GET (openai/groq/elevenlabs);
local providers (edge/piper/faster-whisper/...) skipped
Invariants:
- Opt-in only: zero probes without --live (default False)
- Bounded: sequential, per-probe timeout (doctor.live_probe_timeout,
default 10s, config.yaml knob)
- Never mutates state; unconfigured backends skip with a note
- Failure isolation: every probe wrapped in a catch-all; a probe crash
can never break the doctor run; failures append to the issues summary
New: hermes_cli/doctor_live.py, tests/hermes_cli/test_doctor_live.py
(23 tests, probes mocked at the HTTP/client seam).
Wired: --live flag in subcommands/doctor.py; run_doctor calls
maybe_run_live_checks after all static checks.
Coordination: PR #70124 (--probe-routes) probes LLM routes; this flag
probes TOOL backends — different surface, no code-region collision
(the run_doctor hook here sits at the end-of-run summary, not the
API Connectivity section #70124 extends).
Inspired by: paradigmxyz/centaur tool-health-smoke (MIT/Apache-2.0);
sibling: #70124 (LLM route probes — different surface)
Per-task `output_schema` (JSON Schema object) on task items plus the
top-level single-goal form — a one-time static addition to the tool
schema (never varies per call).
- Child side: the schema is appended to the child's context as an
explicit OUTPUT CONTRACT block before spawn.
- Completion side: the parent validates the child's final answer with
jsonschema; on failure it sends exactly ONE bounded retry turn
carrying the validation errors verbatim (no schema re-paste).
- Result entries gain schema_valid (+ schema_retries, and schema_errors
on final failure) ONLY when a schema was requested; schema-less calls
keep a byte-identical result shape.
- Malformed schemas are rejected loudly at dispatch (coerce_output_schema
meta-validates via jsonschema's validator_for/check_schema).
- New helpers in tools/delegation_output_schema.py: coerce, contract
block, fence/prose-tolerant extraction+validation, retry message.
Pattern from: github/copilot-cli ctx.agent(prompt,{schema}) — PATTERN
ONLY, zero code/prompt text copied (proprietary); proven consumer:
delegate-task-output-patterns skill.
Tests: tests/tools/test_delegate_output_schema.py (24 tests — valid
first try, invalid->retry->valid, invalid twice -> schema_valid false +
errors surfaced, retry-exception degrade, no-schema legacy shape pin,
dispatch rejection, contract plumbing). Delegation suite: 221/221 green.
Add an optional `region: [x1, y1, x2, y2]` parameter to vision_analyze
(pixel coordinates in the ORIGINAL image space). The crop is applied
with Pillow BEFORE the downscale/embed-cap pipeline, so the cropped
region gets the full resolution budget — a zoom for reading small text
or UI details after a full shot.
- New `_crop_image_region` helper: clamps out-of-bounds coordinates to
the image, rejects zero-area/inverted/malformed regions with an error
naming the actual image dimensions so the model can retry sensibly.
- Wired into both the native fast path (`_vision_analyze_native`) and
the legacy aux-LLM path (`vision_analyze_tool`).
- Schema gains one static optional param (byte-stable thereafter); the
description documents the intended flow: full shot first, then zoom.
- No region supplied = behavior unchanged (regression-guarded).
Tests: tests/tools/test_vision_region.py (11 tests — crop applied,
clamping, zero-area rejection with dims, malformed input, pre-downscale
full-budget zoom, schema shape, handler pass-through, no-region
unchanged). Widened one narrow fake_native stub in test_vision_tools.py
to be kwargs-tolerant.
Ported from: QwenLM/qwen-code zoom-image.ts (Apache-2.0)
write_file/patch targeting AGENTS.md, CLAUDE.md, SOUL.md, .cursorrules, or a
project-local .hermes config dir now ALWAYS prompt the human for approval —
even under --yolo/auto-approve — and fail closed when no human channel
exists. These files steer future agent behavior, so an injected write to
them is a prompt-injection persistence vector.
Design:
- New _check_protected_instruction_write() in tools/file_tools.py, a
sibling of _check_sensitive_path that returns approval-required rather
than a hard error. It realpaths before matching (symlink lesson from
#41351), matches basenames case-insensitively in ANY directory, rejects
'./x/../AGENTS.md' traversal via normpath, and gates files whose
immediate parent dir is `.hermes` (project-local config) while exempting
the authoritative ~/.hermes home (governed by its own guards).
- Approval is ONE-OPERATION only: no session/permanent persistence, no
yolo bypass — intentionally does not route through _run_approval_gate.
Gateway sessions get the button round-trip with allow_permanent and
allow_session both False; CLI uses the per-thread approval callback;
no channel at all = BLOCKED (fail closed).
- Multi-file V4A patches: ONE protected file gates the ENTIRE patch (a
single prompt lists all protected targets; deny applies nothing).
- Config: security.protected_instruction_files (default true) and
security.protected_instruction_extra_patterns (fnmatch on basename).
Config read failure keeps the gate ON.
Tests: 22 new cases in tests/tools/test_file_write_safety.py covering the
adversarial checklist — deny/approve/yolo-bypass attempt, symlink at a
protected target, case variants, relative traversal, arbitrary-directory
basenames, project-local .hermes, checkout-nested-under-~/.hermes
non-gating, patch replace + V4A multi-file atomicity, gateway round-trip,
fail-closed with no human, config off/extra patterns.
Ported from: RooCodeInc/Roo-Code RooProtectedController (Apache-2.0);
companion: #58631 (terminal vector), symlink lesson from #41351.
Adds a per-server `trust: full|untrusted` config key
(mcp_servers.<name>.trust). On an untrusted server, every write-capable
tool call — any tool whose discovery-time annotations do not carry
readOnlyHint=True — routes through the existing approval surface
(tools.approval.request_elicitation_consent, same lazy-import +
surface-routing pattern the MCP elicitation handler uses) before the RPC
fires. Denied/cancelled/errored approvals fail closed: the RPC never
runs, including the lazy first-use server spawn.
Design points:
- Classification happens at CALL TIME from metadata captured at
DISCOVERY (_record_tool_trust_metadata in _register_server_tools and
the lazy cache-registration path). No toolset/schema mutation, so the
toolset stays byte-stable and prompt caching is preserved.
- readOnlyHint is a server-supplied HINT: on an untrusted server a lying
server can at most skip approval for tools it claims read-only — it
can never widen access. Trust tiering itself is operator config.
- Missing/malformed annotations => write-capable (fail closed).
- Unrecognized trust values => untrusted (fail closed); missing key =>
full (backward compatible, documented in mcp-config-reference).
- The schema cache now persists readOnlyHint so lazy-registered servers
gate identically on next startup without spawning.
Tests: tests/tools/test_mcp_trust_gating.py (11 tests, TDD red->green):
approval invoked + accept proceeds, deny/cancel blocks RPC, readOnlyHint
=true skips gate, trusted/unconfigured servers skip gate, explicit
readOnlyHint=false gated, approval exception fails closed, trust
normalization, discovery-time capture (SDK objects and cached dicts).
Ported from: cloudflare-os classifyTool() (Apache-2.0), corroborated by
Claude Cowork (idea-level).
MCP servers generated from Rust/TypeScript union types commonly emit
closed value sets as const unions:
{"anyOf": [{"const": "red"}, {"const": "green"}, {"const": "blue"}]}
Strict tool-calling backends reject or mishandle these; the equivalent
property-level enum form is universally supported. Add
collapse_const_unions() to tools/schema_sanitizer.py and wire it into
the _normalize_mcp_input_schema discovery pipeline after the nullable
strip.
Rules:
- Collapse only when EVERY non-null branch is a pure const of the same
primitive type (bool never merges with integer).
- Mixed unions, non-uniform const types, and mismatched declared types
pass through untouched.
- A single {"type": "null"} branch is tolerated: consts -> enum,
null -> nullable: true hint (matches strip_nullable_unions, which
leaves null+multi-const unions alone by its one-non-null-branch rule).
- Outer title/description/default/examples carried onto the replacement.
- Deterministic, branch-order-preserving, non-mutating — applied at
discovery only, so schemas stay byte-stable per conversation.
Ported from: block/goose tool_schema_normalize.rs (Apache-2.0)
Two small config-gated features:
1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
true, config.yaml): a running card with broken claim bookkeeping
(claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
restore) is invisible to all existing recovery paths
(release_stale_claims requires claim_expires NOT NULL,
detect_crashed_workers requires host-local lock + pid,
detect_stale_running is config-disabled by default) and shows Running
forever. New reconcile_orphaned_running() pass in kanban_db.py runs
each dispatch_once tick: requeues orphans to ready with an explanatory
comment, closes any leaked run, emits a 'reconciled' event, and defers
when the recorded PID is still alive on this host (never requeue
beside a live worker). Surfaced via DispatchResult.reconciled_orphans.
2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
config.yaml): optional {name, value_from: static|profile, value}
mapping; the header is attached to that server's HTTP/SSE transport
requests. 'static' sends the config value; 'profile' resolves the
active Hermes profile name once at connect time (no per-call
mutation). Explicit per-server headers of the same name (any casing)
win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.
Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.
Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
Resumable ESTOP sentinel at $HERMES_HOME/ESTOP that halts NEW work only:
- agent/estop.py: sentinel engage/disengage/is_engaged (single stat, no
caching), optional reason + timestamp stored as JSON, paused_reply()
notice, check_paused() log-once-per-engagement helper. Corrupt/empty
sentinel still pauses (fail safe); a `touch ~/.hermes/ESTOP` works.
- cron/scheduler.py: tick() skips dispatch while engaged (logged once per
engagement, not per tick). Due jobs simply wait for the next tick after
resume — in-flight runs are never touched.
- gateway/kanban_watchers.py: dispatcher skips auto-decompose and worker
spawning while engaged; zombie reaping still runs and running workers
finish naturally.
- gateway/run.py: new gateway turns (post-auth, non-internal) get a brief
"Hermes is paused" reply instead of an agent run. Internal events
(in-flight background completions) bypass the gate.
- hermes_cli/subcommands/pause.py: `hermes pause [--reason]` and
`hermes resume`, wired into main() and _BUILTIN_SUBCOMMANDS.
- hermes_cli/status.py: `hermes status` shows a PAUSED banner (one stat).
- tests/test_estop.py: 20 tests — sentinel lifecycle, reason surfacing,
log-once, cron skip + resume, kanban gate, gateway paused reply +
internal bypass, CLI idempotence, builtin-set parity, status line.
Never kills in-flight work; resumable with no restart. Footprint ladder:
CLI command only, no new model tool, no new env vars.
Ported from: gastownhall/gastown estop.go (MIT); related prior art:
#26778 (/panic — kill/exit semantics, deliberately different: ours is a
resumable pause), #44617 (interrupt in-flight cron — out of scope here).
Each serialized result entry now carries cost_usd (rounded to 6 dp)
and cost_status (the child's session_cost_status — 'estimated',
'reported', 'included', or 'unknown') alongside tokens/api_calls/
duration, so the parent model can see what each delegation cost.
The internal _child_cost_usd field is still stripped before
serialization and the parent session cost rollup is untouched.
Tool schema is unchanged (byte-stable).
Inspired by: Perplexity Agent API result shape (idea-level)
Reject malformed tasks=[...] batches before any child agent is spawned:
- exact-duplicate goals (case/whitespace-normalized), error names both
task indices
- placeholder goals: bare 'TODO', bare 'task N', unexpanded <...> or
{...} template markers, or goals shorter than 10 chars after strip
- 1-task batches, with an error pointing the model at the single
`goal` form instead
All checks are batch-only — the single-goal form is exempt by design
(short goals like goal="test" are valid there). Error strings are
actionable: each tells the model exactly how to fix the call.
Tool schema is unchanged (byte-stable); validation is runtime-only in
the existing batch-validation region.
Existing tests using terse batch goals ("A"/"B"/"C") updated to
realistic distinct goals per the new contract.
Inspired by: MoonshotAI/kimi-code agent-swarm.md validation rules (MIT)
Validate a job's configuration BEFORE any agent machinery is constructed:
- missing provider API key (AuthError from a read-only
resolve_runtime_provider probe; skipped when a fallback_providers chain
is configured, since auth-fallback may rescue the run)
- attached skill not ready (skill_view readiness_status=setup_needed —
missing required env vars / commands / credential files)
- delivery platform unknown or unconnected (deliver=local/origin/all are
never checked; gateway-config load failures fail open)
On a failing check run_job returns a [blocked_config]-marked error without
constructing AIAgent/MCP/etc, so a misconfigured job never burns an LLM
call. run_one_job records last_status='blocked_config' and delivers the
alert exactly ONCE across ticks (persisted preflight_alerted bit — the
alert-once shape from the #73506 dead-pin auto-pause); the next healthy
run clears the marker so a future break re-alerts. Every preflight check
fails open: only an affirmative misconfiguration verdict blocks.
Config: cron.preflight (default true); `cron.preflight: false` restores
the old fail-during-run behavior. Documented in the cron user guide and
config defaults.
mark_job_run gains an optional status= override (unblocked call shape
unchanged) and drops preflight_alerted on any successful run.
Tests: tests/cron/test_preflight_config.py (blocked_config + no agent +
single alert across two ticks, healthy job unaffected, recovery clears
dedup, fallback-chain rescue, opt-out restores old behavior, skill
readiness miss, unknown delivery platform, deliver=local never loads
gateway config). Full tests/cron/ + cronjob tool suite green (525 tests).
Ported from: paperclipai/paperclip execution-semantics §5 (MIT);
in-repo precedent: #27948, #73506
- cron/notepad.py: SQLite-backed cron_notepad(job_id, key, value,
updated_at) store in its own profile-local db (cron/notepad.db),
following the executions.py connection/transaction pattern. APIs:
set_note/get_note/delete_note/list_notes/clear_notepad +
render_notepad_section. Documented size caps: 16KB per value,
128-char keys, 64KB per job total; oversized writes raise ValueError.
- cron/scheduler.py: inject non-empty notepads into the job prompt at
the context_from data-injection seam as a clearly-labeled
"Job notepad (persistent across runs)" section that also documents
the CLI write path. Empty notepad renders "" — byte-stable prompts
for jobs that never use the feature.
- hermes_cli/cron.py + hermes_cli/subcommands/cron.py:
`hermes cron notepad <job_id> [get|set|delete|list]` under the
existing cron subcommand tree (no new top-level command, no new
model tool — the agent writes via terminal + CLI).
- tests/cron/test_notepad.py: CRUD, durability, cap enforcement,
prompt injection, byte-stable empty case, read-failure resilience,
CLI handler + dispatch (TDD; watched fail first).
Inspired by: Amp (Sourcegraph) cron notepad (idea-level, proprietary —
zero code).
Add monitor-mode cron jobs: a cheap monitor source (monitor_script or
monitor_url) runs on every tick BEFORE any agent machinery is built.
Its output is hashed as exact bytes and compared to the hash stored
from the last agent-triggering tick:
- unchanged -> agent run suppressed entirely (no LLM, no delivery);
the tick is recorded as a silent no_change run visible in the
executions ledger doc
- changed -> a MONITOR CHANGE DETECTED block (capped unified diff of
previous vs current output + the new output) is injected into the
prompt via the existing extra_prompt seam, then a normal agent run
- first run -> always runs the agent with a baseline block
- source failure -> delivered as an ERROR alert, never treated as a
change; the stored hash is untouched so recovery to prior output
still suppresses
Implementation:
- cron/monitor.py (new): hash/diff/URL-fetch/state persistence.
monitor_script reuses _run_job_script (same ~/.hermes/scripts/
containment + interpreter rules); monitor_url is a bounded GET
(30s, 256KB, http/https only). Output is exact bytes by design —
scripts should emit stable output (documented).
- cron/jobs.py: additive job fields monitor_script / monitor_url /
monitor_state {last_output_hash, last_changed_at}. JSON job records
need no migration. create-time validation: sources are mutually
exclusive and incompatible with no_agent=True.
- cron/scheduler.py: one tight monitor gate in run_job between the
no_agent short-circuit and the LLM path (outside sibling-lane
regions). State persists in jobs.json + a per-job snapshot file, so
suppression survives scheduler restarts.
- tools/cronjob_tools.py: additive optional monitor_script/monitor_url
params on the cronjob tool (create + update, empty string clears),
path containment validated at the API boundary, surfaced in
_format_job.
- hermes_cli: --monitor-script/--monitor-url on `hermes cron create`
and `hermes cron edit`; `hermes cron list` shows the monitor source
and last-changed time.
Tests (tests/cron/test_monitor_kind.py, TDD): unchanged suppresses,
changed injects diff, first run always runs, hash persists across
module reload (restart), script failure is error-not-change with hash
untouched, create/update validation, tool wiring + path-escape reject.
Inspired by: ChatGPT Work monitor tasks (idea-level, docs-only);
enabler: #80774
Answers "what would the approval system do with this command?" without
executing it, prompting anyone, or persisting anything. Composes the
REAL runtime evaluators from tools/approval.py in the same order as
check_all_command_guards: container-skip gate, hardline blocklist,
sudo-stdin guard, user approvals.deny rules, yolo/mode-off bypass,
permanent command_allowlist, dangerous-pattern detection. Because the
same functions run — including _command_detection_variants's
normalization/de-obfuscation path — an obfuscated command gets exactly
the verdict its plain form would get at runtime, and the output shows
the normalized-variant trace the detectors actually evaluated.
- hermes_cli/approvals_test.py: evaluate_command() + text/JSON output.
Script-friendly exit codes: 0 allow, 1 usage, 2 ask-approval, 3 deny
(hardline / sudo-stdin / user deny rule).
- hermes_cli/subcommands/approvals.py: `test` subparser with --env-type
(default local), --json, and a REMAINDER command (dest command_words —
NOT "command", which main.py's startup path reads as the top-level
subcommand name).
- hermes_cli/approvals_suggest.py: dispatch `test` and mention it in the
bare-`hermes approvals` usage text.
- tests/hermes_cli/test_approvals_test.py: verdict matrix (benign /
hardline / dangerous / user-deny from config / container skip /
mode=off vs hardline), obfuscated==plain verdict parity with
normalized trace, spy proof that the real runtime detectors are the
ones invoked, read-only invariants (nothing executed; prompt and
persistence paths rigged to explode), JSON shape, dispatcher and
parser wiring.
Read-only by construction: only detection/matching functions are
called; the approval gate, prompts, gateway notify, and allowlist
writers are never reached.
Inspired by: Amp `permissions test` (idea-level, proprietary — zero code)
- Give cached_fetch_api_models the same stale-while-revalidate tier as
cached_provider_model_ids: TTL-expired entries within the 7d window are
served instantly while a background refresh rewrites the cache —
without this, every /model open an hour into the session re-blocked on
the live probe (#72762's stall class, deferred).
- Generalize _spawn_swr_refresh(cache_key, refresh_fn) so non-slug
custom:<base_url> keys reuse the same inflight-dedupe scaffolding;
slug behavior unchanged (default refresh_fn preserved).
- Convert the missed sibling site: acp_adapter/server.py
_named_custom_provider_catalogs() live-probed every custom_providers
row's /v1/models per ACP catalog build.
- Extract _cache_entry_valid() (the fp/models predicate existed 4x) and
validate 'at' is numeric so hand-edited/corrupt cache JSON degrades to
a live fetch instead of raising through the picker's blanket except.
- Flatten the dead api_mode conditional (fetch_api_models declares
api_mode=None; branch was behaviorally inert).
- Tests: 4 new guards (stale-serve, stale-window cutoff, generalized SWR
write-through, corrupt-at degradation) — stale-serve and corrupt-at
mutation-checked; 2 existing tests updated for the new behavior.
Custom OpenAI-compatible endpoints (named custom_providers rows, bare
provider: custom, and per-endpoint-map entries) called fetch_api_models()
directly at three call sites in model_switch.py, with no disk cache — unlike
first-class providers, which go through cached_provider_model_ids(). Every
plain /model open live-probed the active custom endpoint's /v1/models,
regardless of how recently it had already been probed.
Adds cached_fetch_api_models() in hermes_cli/models.py: a TTL disk-cache
wrapper keyed on custom:<base_url> (custom endpoints have no
PROVIDER_REGISTRY slug to key on) and fingerprinted on api_key/api_mode/
headers, with the same stale-beats-nothing fallback policy as
cached_provider_model_ids(). Routes all three probe call sites through it.
Since prewarm_picker_cache_async() already calls list_authenticated_providers()
with probe_custom_providers defaulting True, this also fixes the endpoint
being warmed on boot (populating the disk cache) instead of that work being
discarded on every open — any custom endpoint (an LLM gateway, a
self-hosted vLLM/SGLang server, etc.), not just one specific provider.
Fixes#72762. Salvaged from #72810 per review feedback: extracts just the
verified custom-endpoint cache fix with real cache-contract test coverage
(hit/stale/rotation/refresh/fallback), leaving the credential-pool and
Copilot-token-exchange costs described in the issue for separate follow-up.
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
The per-session agent cache is capped at 128 entries with a 1h idle TTL, and
neither bound knows how many bytes it holds. Each cached agent pins
_session_messages -- the full transcript including tool output, tens of MB on
a session with 100+ tool calls -- so a gateway serving many chats keeps every
warm transcript resident: agents that took a turn inside the TTL are never
idle-swept, and the idle sweep additionally defers finalizable sessions until
they expire. RSS climbs until the cgroup throttles and SIGTERM can no longer
flush inside systemd's stop timeout.
Add the missing bound. Each session-expiry watcher tick compares the process's
anonymous RSS against a budget and, when over, sheds LRU agents through the
same soft-eviction path the cap enforcer uses, then runs malloc_trim so the
freed arenas actually return to the OS. Evicted sessions rebuild their
transcript from the persisted session on the next turn.
Three classes of session are never shed: agents mid-turn, the most recently
used ones, and any session whose transcript has not finished reaching disk
(_last_flushed_db_idx vs len(_session_messages) -- the same divergence the FTS
write-corruption guard reacts to when it preserves live history).
memory_high_mb defaults to "auto", deriving the budget from the cgroup limit
the gateway runs under, so a MemoryHigh/MemoryMax on the unit is respected
without a second number to keep in sync. The two existing bounds become
configurable alongside it under agent.agent_cache.
protect_recent is clamped to half the cache: a couple of sessions can exhaust
the budget on their own, and a fixed MRU guard would then protect everything
and leave the gateway climbing with nothing it would shed.
Fixes#80764
When the Nous Portal returns paid_service_access.allowed=false with
reason=member_spend_cap_exceeded, Hermes was falling through to the
generic 'no active subscription or usable credits' message — even
though the user has ample purchased credits and the real blocker is
an org-level per-member spend cap.
This adds a dedicated branch that surfaces the actual cause: names the
spend cap, shows the cap/spend amounts, and tells the user to ask their
org admin to raise it. Also adds member_spend_cap_exceeded to the
billing error code set so the error classifier and auth error formatter
route it through the Nous entitlement message path.
Review finding: the close block's comment promises a raising
session_db.close() is swallowed with the flag already cleared (no re-close
on a second agent.close()), but nothing pinned it. One test with a raising
_RecordingDB proves both halves.
Follow-up to the review on the session.resume ownership fix. Closing the
pre-transfer early returns left two gaps, both real.
1. The transfer had no owner on the other side. Once ownership moved to the
agent, teardown ran AIAgent.close() (via _teardown_session on session.close
and the orphaned-session reaper), which called session_db.end_session() —
that finalizes the session ROW, not the connection. A successfully resumed
profile session kept its dedicated handle, its db/-wal/-shm fds and its
background token-writer thread for the life of the gateway.
AIAgent now carries an explicit _owns_session_db, defaulting False so the
SHARED launch handle — which outlives every agent and backs every other live
session — is still never closed there. Only the dedicated-open sites set it,
at the point ownership actually changes hands.
2. session.resume was not the only profile-scoped open with no close on its
failure paths. Covered here with the same flag, via a _transfer_db_to_agent
helper that refuses the transfer unless the agent really holds that handle:
- the deferred builder (_start_agent_build), including the session-reaped-
mid-build case, where the built agent is discarded and never torn down, so
transferring to it would leak exactly as before;
- session.branch's branch_db;
- the compute host's per-profile open;
- AIAgent's own lazy open in _get_session_db_for_recall, which no other
object ever references and so was unconditionally abandoned.
Where a handle has already reached a registered session, the drop is
unconditional and the transfer is best-effort on top: a refused transfer leaves
the old leak, which is survivable, whereas closing under a live session is the
permanent "Cannot operate on a closed database" break the original patch exists
to avoid.
Tests: tests/tui_gateway/test_session_db_ownership_teardown.py (new, 14).
11 of the 14 fail without this change; the 3 that pass are the "must NOT close"
guards, which hold in both directions by design.
Review-pass follow-ups (three parallel reviewers, findings verified):
- hermes_state_search.py list_recent_user_messages now drops legacy
standalone compaction handoffs in the decode loop (SQL can't see them:
durable role=user, no display_kind). Closes the /undo N pairing skew
where the in-memory count (new predicate) and the DB soft-delete pick
(old predicate) targeted different turns on legacy sessions. Fetches
with headroom so the requested limit is still honored. 3 new tests,
mutation-checked (no-op'ing the skip fails 2/3).
- _should_skip_model_call_for_reference_handoff: single drive-check scan
(was two — once inside the restore helper, once after); the restore
helper no longer re-scans and its return value now decides the verdict.
- _final_response_from_messages replaced by the _HANDOFF_SKIP_FINAL_RESPONSE
constant it always returned (parameter was unused).
- _handoff_carries_live_user_content delegates to the canonical
_strip_context_summary_handoff_message — also fixes the edge where a
merged-shaped row with an EMPTY preserved prior tail was wrongly
treated as carrying live content.
- Site-level guard test for rollback.restore with a legacy handoff row
(predicate-in-context, complements the unit tests).
Pin #80622 invariants: handoff alone must not drive a model call after
stop, pending real users are restored, and synthetic compaction rows are
never treated as user-originated turns. Also give micro-compaction
enough passes to pay back the longer SUMMARY_PREFIX marker overhead.
The rough preflight estimate intentionally overestimates, but not by a
fixed margin: CJK text is counted at ~1.7x its o200k cost and
Responses-mode reasoning replay blobs at several times their billed
cost. Heavy sessions show rough estimates 2-3x real usage and compact
at 35-55% of the real window, stalling turns for minutes and discarding
detail (churn), because the defer guard only tolerated 5% rough growth
and sessions that never compressed had no baseline at all.
Pair every request's rough estimate (note_request_rough_estimate,
recorded in the conversation loop right after the pressure estimate)
with the provider's real prompt_tokens in update_from_response(), then
defer preflight while projected real usage — last real + rough growth
since that reading — stays under the threshold. Rough growth is itself
an overestimate of real growth, so the projection is an upper bound and
deferring below the threshold is safe; the provider's context-overflow
handler remains the backstop.
The baseline no longer ratchets on defer: it is refreshed by the
response pairing, and advancing it without a matching real reading
would shrink apparent growth and defer on stale data.
Review pass on the idle-unload feature found two material concurrency
bugs; both fixed here with a regression guard:
1. Unload-vs-use null deref (HIGH): _transcribe_local re-read the
module global _local_model at the transcribe call site. An idle
unload firing between the model load and transcribe() evaluated
None.transcribe → AttributeError → user-visible 'Local
transcription failed'. The window was real: the idle timer was only
touched AFTER a successful transcription, so a voice note arriving
exactly as the timeout expired raced the watcher directly.
Fix: bind a strong local reference under the model lock and use it
for the whole transcription (the watcher can null the global at any
time; this in-flight call keeps its instance — the generator holds
self, so no use-after-free). Also touch the idle timer at the START
of transcription so a long in-flight transcribe can't be counted as
idle time. The CUDA-fallback retry path gets the same treatment
(locked global write, local ref use).
2. Watcher replacement race + response-path join (MEDIUM/HIGH): the
old design stopped and re-started the watcher after EVERY
transcription with an unlocked set/join(5)/clear/start sequence on
shared globals. Two concurrent voice messages could interleave to
leave TWO live watchers (one with a stale, shorter timeout — a
raised unload_after_idle_seconds could still unload on the old
value), and the join(timeout=5) sat on the user-visible response
path (a watcher blocked on _local_model_lock during a concurrent
multi-second model load stalls the reply up to 5s).
Fix: single long-lived watcher under a management lock — started
only when none is alive (per-transcription cost: one lock + one
is_alive check), re-reads the configured timeout from config every
cycle (config edits now apply within one 30s interval, without
waiting for the next voice message — previously undocumented), and
stands down without unloading when the timeout is set to 0
mid-idle.
Tests: 17 now — idempotent start (same thread, no churn), config
re-read + stand-down-when-disabled, and the race guard
(unload firing mid-transcription must not fail the in-flight call).
The race guard is mutation-verified: reverting the fix (re-reading
the global at the call site) makes it fail with the exact NoneType
error; the fixed code passes.
The local faster-whisper model singleton (_local_model) is loaded once
and never released — the 'base' model holds ~370 MB of RAM/VRAM for
the entire lifetime of the process, even when no voice messages arrive
for hours or days. On long-running gateway processes (especially with
local LLMs competing for the same GPU) this is wasteful.
Add a config-driven idle unload: after stt.local.unload_after_idle_seconds
(default 0 = never) of no transcription activity, a lightweight daemon
thread sets _local_model = None so the Python GC can reclaim the
ctranslate2 objects. The next voice message reloads the model
transparently (the existing lazy-load path handles it).
The watcher:
- Checks every 30s whether idle time exceeds the configured threshold
- Acquires _local_model_lock before unloading (prevents races with
concurrent transcriptions that are mid-load)
- Exits immediately if the model is already None (unloaded by another
path, e.g. the CUDA fallback eviction)
- Is restarted by each transcription with the current config value,
so changing stt.local.unload_after_idle_seconds in config.yaml takes
effect on the next voice message without a process restart
Default is 0 (never unload) — zero behavior change for existing users.
Recommended value for gateway processes: 300 (5 minutes).
15 tests: config resolution (garbage/negative/None fallbacks), unload
safety (already-None, lock acquisition), touch timestamp, watcher
lifecycle (unload after timeout, no unload within timeout, exits when
model already None, stopped on new start). Existing STT test suite
unchanged.
Three-reviewer pass (reuse / quality / efficiency) on the trim diff;
four findings folded:
1. Short-clip input gate (efficiency, HIGH): the trim previously paid
the full ffmpeg encode before the <10%-saving discard check — every
dense conversational voice note burned 3 subprocess spawns + a
complete re-encode on the synchronous response path for nothing.
New _CLOUD_TRIM_MIN_INPUT_SECONDS=12 gate: below it, savings can't
matter (a >=10% saving is ~1s of audio, and several providers bill
a per-request minimum anyway — Groq bills 10s minimum), so the
whole pipeline is skipped using the duration we already probed.
Typical 5-10s voice notes now pay 1 ffprobe (~50ms), not 3 spawns +
encode (~0.3-1s; multi-second on small-VPS gateway hosts).
2. Shared encode profile (reuse, HIGH): the trim's ffmpeg command
duplicated _transcode_audio_for_stt's encode byte-for-byte (same
16kHz/mono/AAC-32k/faststart args, same subprocess.run kwargs).
Extracted _STT_M4A_ENCODE_ARGS + _run_ffmpeg_stt_encode(ffmpeg,
in, out, audio_filter=None); both call sites now share one owner,
so codec/bitrate/timeout changes can't drift between the paths.
3. is_truthy_value for the enable flag (quality, MEDIUM): raw
bool(cfg.get(...)) treated a YAML string "false" as enabled — the
exact bug class utils.is_truthy_value (already imported, already
used by is_stt_enabled and the xai/elevenlabs flags) exists for.
4. All-silence guard scales with keep_ms (quality, LOW): the fixed
0.3s floor equals the default keep window, so an output consisting
solely of one kept pause could pass as "speech"; now
max(0.3, 2*keep_seconds).
Also: _probe_audio_duration docstring documents it as the canonical
sync seconds-probe (gateway/run.py and the Telegram adapter carry
local variants of the same ffprobe invocation).
Tests: 24 now — YAML-string-false disables; short clips skip the
encode entirely (encoder mock asserted not-called); E2E fixtures
moved past the input gate. E2E re-verified: 13.2s note -> 6.2s
(-53%), 8s clip skipped with 1 probe.
Local faster-whisper gets Silero VAD (bf8004e3a) so silence never
reaches the model. Cloud providers got no such protection: the raw
file uploads untouched, so every second of silence in a voice note is
paid for twice — upload time and per-audio-minute billing — and cloud
Whisper hallucinates junk tokens on silent stretches exactly like
local Whisper did before the VAD hardening. A 13s voice note with two
long pauses is billed as 13s of audio to transcribe ~6s of speech.
Close the gap client-side: before uploading to a built-in cloud
provider (groq/openai/mistral/xai/elevenlabs/deepinfra), collapse long
pauses with ffmpeg's silenceremove filter, keeping
stt.cloud_trim_keep_ms (default 300) of every pause so word boundaries
and natural pacing survive. Uses ffmpeg, already a dependency of this
exact path via _transcode_audio_for_stt — no new dependency.
The trim is strictly best-effort — ALL of these upload the original
untouched, transcription never fails because of the trim:
- stt.cloud_trim_silence: false
- ffmpeg/ffprobe missing, trim failure, or timeout
- trimmed result ~empty (mostly-silence clip: the provider, not a
client-side dB heuristic, decides whether it contains speech)
- trim saves <10% (re-encoding for nothing)
Command-type and plugin providers are deliberately NOT trimmed: they
may wrap local CLIs that want the original bytes or run their own VAD.
E2E (real ffmpeg + faster-whisper): 13.2s voice note with 7s pause ->
6.2s upload (-53%); transcript of trimmed audio matches the original
on both utterances. Dense-speech and all-silence WAVs correctly fall
back to the original. 22 unit+E2E tests; STT/voice suite failures
identical to upstream/main baseline (all pre-existing).
_download_image() and _download_video() both used client.get() +
response.content, buffering the entire media body into memory before
checking the size cap. A server that omits Content-Length could send
an arbitrarily large payload, causing OOM.
Extract _stream_download_to_file() shared helper: streams via
client.stream() + aiter_bytes(), writes chunks to a temp file, enforces
the running byte count against the cap after each chunk, and atomically
replaces onto the destination on success. Cleans up the temp file on
failure. Uses utils.atomic_replace() for cross-device/symlink safety.
Malformed Content-Length values are now caught and ignored instead of
crashing with ValueError; the streaming cap is the authoritative guard.
Approach adapted from PR #10440 by @WuKongAI-CMU (closed as stale —
14923 commits behind, reverted 32 commits of vision_tools.py evolution
including SSRF-safe client, retry classification, and lazy imports).
Closes#10440
_fts_teardown_trash_step deleted rows via 'WHERE key IN (SELECT key
LIMIT N)' — each chunk's subquery re-scanned from the start of the
table, so chunk k skipped past (k-1)xN already-deleted rows: O(n²)
total row visits. On a v22 shadow table with ~230K rows that is on the
order of 10^8 row visits, turning optimize-storage teardown into a
multi-hour grind on slow disks, with a write lock held per chunk.
Single-column INTEGER-PK trash tables now drain via a fts_teardown_<tbl>_progress
high-water marker mirroring fts_rebuild_step: each chunk claims rows
past the marker (SELECT ... WHERE key > ? ORDER BY key LIMIT N), deletes
the claimed range, and publishes the new marker in the same transaction.
Per-chunk work is bounded → O(n) total.
TEXT-PK tables (the FTS config shadow table, pk like 'version') and
compound-key tables fall back to the legacy chunked delete — those are
small by construction.
Fixes#79324
Follow-up to the #80376 salvage:
- TurnLeaseToken.degraded is dead code since acquire() started raising
TurnLeaseTimeoutError: the only constructor site passes the default and
repo-wide there are zero external readers. Remove the field, its ctor
param, the repr segment, and the always-False guards in rebind()/release();
the class docstring's 'retained for compatibility' claim described
consumers that do not exist.
- Revert pure black-rewrap churn in test_config_env_bridge_authority.py and
test_turn_lease.py (hunks on functions this change does not touch), keeping
the functional Windows-env/encoding additions.
- Turn the default-value test into an invariant: config default must equal
gateway.turn_lease.DEFAULT_LEASE_WAIT instead of pinning the 1800 literal.
- Rewrap the dangling 'Released' comment line in gateway/run.py.
Verified: 20/20 targeted gateway tests, ruff clean; mutation check — with
gateway/turn_lease.py reverted to pre-fix main the module fails, restored it
passes.
prompt.submit honored truncate_before_user_ordinal on every request. A
client that carried a leftover ordinal into an ordinary send therefore
issued something the gateway could not tell apart from a real rewind —
same method, same shape, an in-range target — and the cut was applied
with replace_messages(), which DELETEs the durable rows. One report lost
244 messages (296 -> 52) with no prompt and nothing to restore from.
The existing guard only covered ordinal 0, where the cut empties the
transcript; a mid-session ordinal sailed straight through. Only the
client knows whether a submit is a rewind, an edit, or a regenerate, so
require it to say so: an ordinal without confirm_truncate is refused on
4029 and neither memory nor the DB is touched. Desktop sends the flag
from the one place that builds these params, so every rewind path is
covered and a stale build fails closed with an actionable error instead
of quietly deleting a conversation.
Sibling of the chat_completions zero-byte-args fix (previous commits):
a clean SSE close after content_block_start(tool_use) but before any
input_json_delta / message_delta yields an SDK final-message snapshot
whose content is NON-empty (the tool_use block is present, input={})
and whose stop_reason is None. That shape sailed past both
empty-stream guards (they only fire on empty content) and executed the
tool with empty input — no retry, no error: the same silent-data-loss
class as #80498, one provider transport over.
A legitimate completion always carries a stop_reason, so a
tool_use-bearing message without one is a mid-tool-call stream drop.
Raise EmptyStreamError for it, riding the same bounded stream-retry
(HERMES_STREAM_RETRIES) the eventless-stream case already uses.
Gate checked on both return paths (raw SDK snapshot and
accumulator-modified message). Regression tests cover the dropped
shape (mutation-verified: disabling the gate fails exactly that test),
the legitimate tool_use completion, and the text-only no-stop_reason
shape (pre-existing behavior preserved).
Locks in two gaps left by 015a114a2 (#80623): a mixed response where one
tool call completes validly while a sibling has zero argument bytes still
gets discarded whole via the shared partial-stream-stub path, and the
zero-byte trigger now has an end-to-end test through run_conversation's
retry loop, not just at the chat_completion_helpers unit level.
When the stream closes right after a tool call's name arrives but
before any argument bytes are delivered, has_truncated_tool_args
was never set (the existing check required a non-empty, whitespace-
stripped arguments buffer). The call fell through to a normal "stop"
finish_reason, later coerced to "{}" at dispatch and executed
silently with no arguments and no retry.
Route this case through the same dropped-mid-tool-call stub/retry
path already used for partially-truncated JSON.
Post-merge review of aecb9ca89 found the join guard over-broad: skipping
the join whenever ANY fragment self-matches _PREFIX_RE reopened a leak
for non-newline splits — sk-<15 chars>ESC<25 chars> masked only the
self-matching head and left the 25-char tail in cleartext (fully masked
before the guard; main never masked this shape at all, so the merged
state was still >= main, but the salvage's own coverage regressed).
Skip the join only when the span crosses a line boundary (\n / \r) —
that is the shape where adjacent legitimate text gets swallowed
(ghp_<token>-then-'button [ref=e3]' annotation bug). ESC/zero-width
controls never legitimately separate a token from prose, so joining
there is safe and restores full-tail masking.
Both legs mutation-checked: reverting to the unconditional skip fails
the new tail-mask test; removing the guard fails the annotation test.
Follow-up to the salvaged registration contract:
- share one _raise_if_cron_registration_error() helper for the two
byte-identical dashboard 424 except-blocks (web_server + cron router,
via the existing late() seam)
- add endpoint-level 424 coverage for /api/cron/blueprints/instantiate
(previously only the sync worker was tested)
- give chat/CLI surfaces a human-facing user_message() (job name, no
exception class name) and add a recovery hint (pause/resume or update
re-registers via provider reconcile) to the model/REST message
- consolidate five inline provider test doubles into one ABC-subclassing
make_cron_provider conftest factory; the web_server test double now
subclasses CronScheduler so an ABC rename fails loudly
- narrow the wrapper facade to keyword-only (**kwargs) and route the
tool's partial-failure return through tool_error()
Follow-up hardening on the salvaged #80687 shrink-merge guard, folding in
the best part of the competing #80703 (credit: @JoaoMarcos44):
- Stat-stamp fast path: load_jobs() inside a _jobs_lock() section records
jobs.json's (mtime_ns, size, ino) BEFORE reading; the save-path merge
and the post-stage verify skip their full read+parse when the stamp
still matches. The healthy no-race save (every mark_job_run /
claim_dispatch / heartbeat / advance_next_runs tick persist) now costs
one stat() instead of up to two full JSON parses.
- Fail-safe stamp discipline: the stamp is captured pre-read (a sibling
racing the load leaves it older than disk -> mismatch -> merge runs),
includes st_ino (mkstemp+rename always allocates a new inode, so
coarse-mtime filesystems cannot false-match), resets on section
entry/exit, and is INVALIDATED - never refreshed - after any write in
the section (a refresh would let a nested create_job be clobbered by
an outer caller's stale payload; probe-verified both directions).
- _merge_unexpected_disk_jobs no longer mutates the caller's list in
place - it returns a new list when anything was recovered, and logs the
recovered ids.
- The tolerant read cascade (utf-8-sig + strict=False fallback) is
factored into one shared _parse_jobs_file used by both load_jobs and
_peek_jobs_unlocked, so future encoding/shape fixes land once. The
peek's repair-free re-entrancy contract is now documented - a repairing
read on the save path would recurse through _save_jobs_unlocked (the
exact defect the stamp-reconcile approach in #80703 had).
4 new regression tests (fast path, no-mutation, corrupt-file save,
nested-create-vs-stale-outer-save), each verified to fail against the
implementation it guards.
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
CI slice 1/12 caught a regression in _mask_control_split_tokens: a
COMPLETE prefix token at end-of-line followed by ordinary text (browser
accessibility annotations: 'ghp_<tok>\nbutton [ref=e3]: Copy') was
joined across the newline into one stripped-copy match, and the mask
swallowed the adjacent line ('button' disappeared).
Join only when no fragment inside the span matches _PREFIX_RE on its
own — a self-matching fragment is already handled by the ordinary
prefix pass, so joining can only cause damage. All smuggling shapes
(ESC/ZWSP/newline splits with under-length fragments) still mask;
regression test added and mutation-checked (fails without the guard).
_write_checkpoint persisted s.command verbatim to ~/.hermes/processes.json.
Recovery only uses command for display/logging (the process is already
running; adoption re-validates PID + start time, never re-runs the
command), so masking is lossless.
A masked secret's visible head/tail could carry control bytes (newline,
NUL, DEL, C1 0x80-0x9F, zero-width) into config/status/dump output.
Strip every control incl. \n/\t (display differs from redact_sensitive_text,
which preserves \n/\t as line structure) before slicing; all-control values
return the configured empty fallback.
Consolidates the previously-closed #58079 approach (strip controls before
masking) - supersedes it.
- Import file_safety._BLOCKED_PROJECT_ENV_BASENAMES instead of copying
it (comment-enforced parallel lists drift); lookup is now
case-insensitive to match file_safety's .lower() semantics (cat .ENV
on macOS/Windows case-insensitive filesystems reads the same secrets).
- Strip shell quotes plain split() leaves attached (cat ".env").
- Drop the dead _ENV_FILE_EXCLUDE_SUFFIXES logic (exact-basename
membership already excludes templates) and the stray blank-line noise.
- Document the defense-in-depth limits (sudo/full-path/substitution
readers) mirroring is_env_dump_command's precedent, and correct the
docstring overclaim about name-independence.
- Annotate command as str | None (tests pass None).
Terminal output from file-read commands (cat, head, tail, ...) uses
code_file=True, which skips the generic ENV-assignment redaction pass.
Reading a .env file through the terminal therefore leaked any key whose
value has no recognized vendor prefix (Mistral, Gemini AQ.*, tvly-dev-,
bu_, Spotify client secrets).
Detect file-read commands targeting .env-style basenames (mirroring
agent/file_safety's blocked list) and route them to code_file=False so
the existing ENV pass masks opaque values. Templates (.env.example,
.env.sample, ...) are excluded.
Salvaged from #61352 (145 commits of drift; conflict with the test-prune
wave resolved by NOT resurrecting pruned tests). Authored by @ShaoRou459.
Closes#61352
The api_messages build used a shallow msg.copy(), decoupling only
top-level fields. Every nested container (tool_calls entries and their
function dicts, multimodal content-part lists, reasoning_details) stayed
aliased to the persisted history, so ANY in-place transform on the send
copy silently rewrote the stored transcript.
Probed every send-path transform against that aliasing shape on main:
content strip loop safe (top-level reassign)
_canonicalize_api_tool_calls (repair) LEAKED <- #80616's fix
_sanitize_messages_surrogates LEAKED (multimodal parts,
tc ids/args, reasoning)
_sanitize_messages_non_ascii LEAKED (multimodal parts)
_sanitize_api_messages safe
_drop_thinking_only_and_merge_users safe
The retry loop already believed the copies were independent - it
sanitizes messages AND api_messages separately (~L3555) - so the
aliasing was accidental everywhere.
Fix at the chokepoint: _clone_message_for_send clones every container
(dict/list) recursively while sharing immutable leaves, so every
downstream in-place transform - current and future - is safe by
construction. Cost is container-count, not string-bytes: 100KB argument
strings and base64 payloads are shared (measured ~0.5ms vs ~0.1ms per
1500-message build; noise next to one json round-trip). Same clone
applied to the prefill-message insert (same class, same pipeline).
The class-wide invariant test runs the full send-path transform
pipeline over an adversarial fixture (malformed args, surrogates,
non-ASCII, multimodal parts, reasoning fields) and asserts the history
stays byte-identical; an AST contract pins the build-site wiring so the
shallow copy can't quietly return. Both mutation-verified: reverting
the clone to shallow fails 4 isolation tests, unwiring the build site
fails the AST contract.
0xGr1mm's branch fix (previous commit) remains as defense in depth at
the exact site the #80498 incident hit; his regression tests and the
class-wide invariant give layered coverage.
`_canonicalize_api_tool_calls` promises copy-on-write in its own docstring
— "the persisted history is untouched" — and the call site repeats it:
"Operates on api_messages (the API copy) so the original conversation
history in `messages` is untouched."
The canonicalize branch keeps that promise (`tc = {**tc, "function": {...}}`).
The repair branch does not:
except Exception:
tc["function"]["arguments"] = _repair_tool_call_arguments(...)
`api_messages` is built with `msg.copy()` — a SHALLOW per-message copy — so
every `tool_calls` entry is the same dict object the persisted history
holds. Assigning into `tc["function"]` therefore writes through to the
stored turn. The sibling loop two lines above only touches `am["content"]`,
one level deep, which is why the aliasing never showed up there.
On the unrepairable path `_repair_tool_call_arguments` returns "{}", so
that write replaces the model's real arguments with an empty object in the
transcript. A stream that dies mid `write_file` loses the file content it
had already streamed — the reported symptom in #80498, where a chapter
draft was silently reduced to `{}` and only a WARNING remained:
Unrepairable tool_call arguments for write_file — replaced with empty
object (was: {"content": "# 骨架-第25章\n> 承接...)
Mirror the canonicalize branch: build a new tool-call dict instead of
assigning into the shared one. The API copy still carries "{}" — the
repair's whole purpose is to never ship broken JSON — but the history keeps
what the model actually sent, so the transcript, session persistence and
any later retry still have it.
The in-place write was not an oversight in isolation: it predates the memo
refactor, which preserved it deliberately for byte-parity. The existing
`test_history_not_mutated` asserts exactly this invariant but restricts
itself to valid arguments, and its docstring records the gap — "(Malformed
args take the in-place repair path — pre-existing behavior)". That is why
a test file whose header already claims "the persisted history is never
mutated (copy-on-write preserved)" stayed green through the bug.
Four tests close it: history keeps the original bytes, the send copy is
still repaired, a broken call does not disturb its siblings, and repeated
sends stay lossless. On unpatched main three of them fail; the parity and
complexity tests are unaffected because the difference is only observable
when the history list is separate from the send copy — which is the shape
production uses.
Refs #80498
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The PR renamed _resolve_auto to _resolve_auto_route (3-tuple return).
This test patches the resolver to mock the auto-detection chain, but was
still patching the old _resolve_auto name. The patch never fired, so the
test fell through to the real auto-detection (which finds no providers
in CI's hermetic env). Update patch target to _resolve_auto_route with
the 3-tuple return value.
- state_db fixture: drop the HERMES_HOME setenv + sys.modules purge
(SessionDB takes an explicit path; tests/conftest.py already sandboxes
HERMES_HOME; the purge risks split-class identity for other modules
holding the old hermes_state reference) — matches the plain
tests/hermes_state/ sibling fixture pattern.
- probe guard asserts on the CALL ('has_archived_messages(') instead of a
local-variable name — a reintroduced probe under any rename now trips
it (mutation-checked: renamed-probe reintroduction fails the guard;
restored stack green).
- _archived_count uses the public get_messages(include_inactive=True)
instead of poking db._lock/_conn.
Follow-up hardening on top of the salvaged #80809 watchdog, porting the
locked state-machine design from #75301 (credit: @Zeraphim):
- All lifecycle transitions (stale/cancelled/done) now happen under
request_client_lock. A user or monitor interrupt marks the request
'cancelled' so a racing stale timer can no longer misclassify the kill
as provider staleness and feed a false +1 into the #58962 cross-turn
circuit breaker.
- 'done' is set under the lock on completion, so a late timer callback
that lost the race to a successful response is inert instead of
leaving a spurious streak=1 behind the reset.
- Registration race closed: if the budget expires while the client is
still being constructed, _make_client aborts the freshly-registered
client and fails the call with a retryable TimeoutError instead of
opening a brand-new socket after the only watchdog already fired.
- _resolve_direct_stale_timeout now fails closed: a raising resolver
propagates (same as the worker path) instead of being swallowed into
an infinite budget that would silently disarm the watchdog and
reinstate the very hang #80759 is about.
4 new regression tests, each verified to fail against the pre-fix
watchdog implementation.
Co-authored-by: Zeraphim <diamantejc87@gmail.com>
Cron turns and delegated children are routed onto direct_api_call, which
ran the request inline with no stale detector. The abort plumbing was
registered but nothing ever invoked it, so a provider that accepted the
request and then went silent — connection held open, zero bytes, no
error — hung the run until an external actor killed it, which also
orphaned the execution row. The httpx read timeout is not a usable bound
(1800s default, and this failure mode never trips it), and the job-level
inactivity monitor was observed not to fire.
Arm a watchdog timer on the same budget the interrupt worker's poll loop
uses, so these turns get exactly the patience every other non-streaming
request already gets. On expiry it only aborts the in-flight sockets
through the already-registered hook — it never issues a request, so the
inline / no-worker property that fixes the nested-pool deadlock is
preserved — bumps the cross-turn stale circuit breaker, and surfaces a
retryable TimeoutError so the outer loop reconnects on a fresh pool.
Fixes#80759
#80216 fixed /retry (and a follow-up fixed yuanbao recall) destroying
soft-archived active=0/compacted=1 in-place-compaction rows via the
destructive replace_messages default. Two sibling sites still carried the
same class:
- acp_adapter/session.py _persist (non-owned-agent branch): probed
has_archived_messages and FAILED OPEN into the destructive full replace
on any probe error; the probe can also race a concurrent
archive_and_compact. Now passes active_only=True unconditionally — on a
fresh create/fork every row is active=1 so behavior is identical, and
the probe (its only production caller) is deleted.
- tui_gateway/methods_prompt.py edit/regenerate truncation: bare
replace_messages() deleted the archived transcript of a compacted
session on every edit/regenerate. Now active_only=True.
hermes_state.has_archived_messages docstring updated (probe is now
test/diagnostic-only). Test stubs in test_tui_gateway_server.py accept the
new kwarg. New regression tests: real-SQLite archive-survival for both
write shapes, fresh-session equivalence (the claim the unconditional
switch rests on), and source-level guards pinning that neither site
re-grows the fail-open probe (both mutation-checked: revert either fix and
its guard fails).
- is_tts_echo sliding window: return True immediately when ratio >= threshold
instead of scanning all remaining windows. Common echo case drops from
~4s to <1ms for long spoken text (found by /simplify-code efficiency review).
- Clear _voice_barge_phase in _voice_submit_barge_utterance finally block
alongside _voice_barge_capture, preventing stale phase from a previous
trip affecting a future call.
- Add _voice_last_tts_text and _voice_barge_phase to _make_voice_cli test
helper so it matches __init__ state.
Reviewer feedback on the fragment-echo fallback added in 24730b10c:
- A short playback-phase transcript (e.g. a genuine one-word "yes")
could trivially match a same-length window of a longer spoken reply
at ratio 1.0 and be wrongly dropped as a self-capture. The fallback
now requires the transcript to be at least
MIN_FRAGMENT_LENGTH_FOR_ECHO characters before it runs.
- The fallback split on whitespace, so it never engaged for
no-whitespace languages (both transcript and spoken text collapse to
a single "word"). Switched the sliding window to be character-based
instead of word-based, matching the function's tokenization-independent
contract.
Adds regression tests for both cases.
is_tts_echo() compared the captured transcript against the *entire*
spoken text with a whole-string similarity ratio, which only scores high
when the two strings are close in length. But a playback-phase barge
capture is cut immediately when the trigger fires and only spans the
pre-roll buffer plus time-to-silence, so a genuine self-capture is
typically a short fragment of a longer reply, not a near-verbatim repeat
of the whole thing -- for any response longer than a clause, the
length-diluted ratio fell below threshold and the echo sailed through
ungated.
When the whole-string check misses, also slide a window sized to the
transcript's word count across the spoken text and compare against each
window, so a short fragment echoed from within a much longer
multi-sentence reply is still caught. Add regression tests for a short
fragment matched at the start and in the middle of a longer reply.
The full-duplex barge-in listener added in 5081551f0 stays active during
TTS playback with no acoustic echo cancellation. On some speaker/mic
combinations, TTS bleed alone crosses the barge threshold, gets
transcribed, and is queued as the next user turn -- whose reply is then
spoken, captured, and queued again, producing an unbounded TTS -> STT ->
TTS feedback loop (#75780).
Add a fail-closed transcript-level guard: when a barge trip happens during
the playback phase, compare the captured transcript against the TTS text
Hermes just spoke (tools/voice_mode.is_tts_echo, a language-agnostic
character-level similarity ratio). A close match is dropped instead of
queued, and the mic is handed back to the normal continuous-listening
loop. Generation-phase trips (no TTS playing, so no bleed is possible)
are unaffected.
_handle_media_message had the same gap as _handle_text_message: it didn't
set user_id/user_name or any reply_to_* fields on MessageEvent. A user
replying to a photo/video/audio in Matrix got null sender metadata and
null reply context — the exact bug PR #80293 fixes for text messages.
Mirrors the text handler's reply fallback parsing + sender propagation
into the media handler, and adds a test covering a media reply event.
Also fixes test env mutation: _make_adapter now accepts monkeypatch
and uses monkeypatch.setenv() instead of bare os.environ assignment,
preventing env var leakage across tests in the same xdist worker.
The Matrix adapter built MessageEvent from inbound room events but dropped
the sender's MXID and display name on the event itself -- only 'source'
carried them. Other adapters (signal/slack/telegram/discord/mattermost/irc)
have the same gap; this PR fixes matrix and adds the supporting
top-level MessageEvent fields so the rest can follow.
Downstream effects for matrix specifically:
- gateway prompt assembly can now read event.user_name (or source)
without having to dig into source per platform
- reply context (reply_to_text / reply_to_author_id /
reply_to_author_name) is parsed from the inline > <@user:server> ...
Matrix fallback format before stripping, instead of discarded
- the gateway's existing [Replying to: "..."] renderer can now show
who the user was replying to (was always anonymous for matrix)
MessageEvent gains two optional top-level fields (user_id, user_name,
both default None) so non-IM producers (cron/webhook/autonomous) remain
unaffected. Source still carries the same values for callers that
already read from there.
Tests cover:
- non-reply message carries sender user_id/user_name on MessageEvent
- different senders (alice, bob) both propagate
- reply message carries reply_to_message_id + reply_to_text +
reply_to_author_id + reply_to_author_name, parsed from the
> <@carol:example.org> original question\n\nactual reply shape
- non-reply message does NOT spuriously set reply_to_* fields
Sibling matrix tests (148 across test_matrix*.py) remain green.
Authored by WintleChoung <cwt@users.noreply.github.com>
Salvaged from PR #80293.
Widen #79293's trailing-in-flight guard from 'last message is assistant'
to 'last non-tool message is assistant': a multi-call batch snapshotted
between the executor's per-result appends looks like
[..., assistant(c1,c2,c3), tool(c1)] — c2/c3 are pending, not orphaned,
but the tail-only guard missed that shape and stripped them (same silent
result loss as the original bug, via concurrent /compress or the gateway
hygiene pass).
Preserving is safe on both shapes: the pre-API chokepoint
(sanitize_api_messages step 2) injects stub results for any call that
genuinely never gets an answer, while stripping a live call silently
loses its late result.
test_sanitizer_strips_orphaned_keeps_valid's mixed valid/orphan shape
moves mid-list — at the tail it is byte-identical to a live partial
batch and the sanitizer now correctly presumes in-flight there.
New regression test fails without the walk-back (c2/c3 stripped),
passes with it.
Tool_executor.py appends role=tool results AFTER running each call. When
context compression fires mid-chain, the trailing assistant(tool_calls)
message is a pending request whose result has not yet been appended.
_sanitize_tool_pairs previously stripped it as an 'orphan', so when the
executor later appended the real result, repair_message_sequence dropped
it as unmatched and the completed side effect (and final synthesis) was
lost. Preserve the trailing in-flight call verbatim; only genuinely
orphaned calls in the discarded region are stripped.
Adds regression tests: three unit tests for _sanitize_tool_pairs plus an
end-to-end test reproducing compression -> side-effect completion ->
result-returned flow. Confirmed failing on pre-fix code, passing with
the fix.
Review follow-up for the salvaged #79881/#63963 stack: the shutdown
interrupt fires exactly once, but work can materialize AFTER that one
shot on BOTH sibling paths:
- a /v1/runs task admitted before the drain populates
_active_run_agents only once _create_agent returns
(queued-before-agent window);
- a _running_agents entry claimed as _AGENT_PENDING_SENTINEL is
promoted to the real agent by track_agent() on its own schedule,
after the one-shot walk skipped the sentinel.
Either way the settle loop waited on work nothing signaled, and the
turn went straight to the post-interrupt tool-subprocess kill — the
exact amputation the fix exists to avoid, in a rarer window.
If any work is still live when the settle loop exits, re-invoke
_interrupt_running_agents (which already skips sentinels and folds in
the API-server helper) so late-materializing agents on either path get
the cooperative interrupt. Regression test drives the real stop() path
with an accelerated loop clock and asserts exactly two interrupt
signals.
The shutdown drain ACCOUNTS for API-server work but never INTERRUPTS it.
`_drain_active_agents()` folds `_active_api_run_count()` into both its wait
loop and its `timed_out` verdict, while `_interrupt_running_agents()` iterates
`self._running_agents` only -- a dict no API turn ever enters, because the
API server owns its own agent lifecycle. `gateway/run.py` states the gap
against itself: "API-server / desk sessions have the same structural gap
(#63529)."
The user-visible result is that every gateway restart with a live API or
desktop turn burns the full drain timeout and then runs
`_kill_tool_subprocesses("post-interrupt")`, which amputates the turn's tool
subprocesses with no cooperative interrupt and no resume marker.
There are seven API agent-entry points. Six funnel through `_run_agent()`
(both session-chat routes, and `/v1/chat/completions` + `/v1/responses` in
streaming and non-streaming form) and are counted by `_inflight_agent_runs`;
the seventh, `/v1/runs`, runs its own lifecycle and is counted through
`_active_run_tasks`. None of the six has a run_id, so the run_id-keyed
`_active_run_agents` cannot reach them, and only two pass `agent_ref` -- which
lands in a caller-local list, not a registry.
So register once at the single unconditional creation site inside
`_run_agent`, beside the existing `_publish_turn_process_ownership()` call,
and unregister in the same `finally` that already clears it. That one
symmetric pair covers all six callers. The registry is adapter-owned and
keyed by object identity, kept separate from `_active_run_agents` because
that dict is run_id-keyed and scoped to the public `/v1/runs` stop API.
`interrupt_active_runs()` then walks both registries, deduped by identity, so
the interrupt set matches the set the drain waits on. The settle window after
the interrupt now polls API work as well: the interrupt is cooperative, and
without this the window closes the instant `_running_agents` is empty -- which
it always is for API turns -- and the tool kill lands on a turn that was asked
to stop microseconds earlier.
/retry truncates the live transcript to before the last user message
and persists it via SessionStore.rewrite_transcript, which calls
replace_messages() with the default active_only=False. That DELETEs
every row for the session, including the soft-archived
active=0/compacted=1 rows that in-place compaction keeps on disk
(#38763), so any /retry after a compaction permanently wiped the
archived history. #57803 named this call site as a residual gap after
its global-default approach was rejected; the TUI sibling was fixed
in #80195.
The handler now probes has_archived_messages() (new SessionStore
wrapper, auto-exposed through AsyncSessionStore) and passes
active_only=True when archives exist, so only the live rows are
replaced. rewrite_transcript gains an active_only parameter that
defaults to False, keeping the destructive semantics yuanbao recall
redaction depends on. Also corrects the rewrite_transcript docstring,
which still listed /undo as a caller even though /undo soft-archives
via rewind_session.
The regression test drives _handle_retry_command against a real
SessionStore and SessionDB seeded with archived compaction rows and
asserts the archives survive.
Fold the remaining simplify-code reuse finding: teams' _install_hint()
duplicated lazy_deps' spec-fetch + quote + join (feature_install_command
already builds pip commands from LAZY_DEPS). Add a venv_pip=True variant
to feature_install_command — sys.executable -m pip targeting, correct in
every install layout and immune to PEP 668 — and shrink the teams helper
to a one-line call. Also gives matrix and the other platforms a shared
derived hint to adopt later. New test mutation-checked (fails when
venv_pip returns the uv form).
- matrix/dingtalk: extract deps-only installers (ensure_matrix_deps,
ensure_dingtalk_deps) and register THOSE as ensure_deps_fn — the prior
check_*_requirements combined credential env checks with the install,
so a platform configured via PlatformConfig.extra (which is_connected
accepts) would pass enablement, reach create_adapter(), and have the
'installer' veto on env-var grounds before installing anything —
re-creating the #79812 deadlock for extra-configured setups. The
combined deps+credentials functions remain for setup/status callers.
- matrix/feishu passive probes: use the existing lazy_deps.is_available()
instead of hand-rolling 'not feature_missing(...)' (reuse finding).
- teams: module docstring no longer recommends bare system pip (the
PEP 668 trap purged everywhere else); docs troubleshooting row updated
to match the new hint text.
- wecom_callback: drop dead 'global ET, DEFUSEDXML_AVAILABLE'
(ensure_and_bind mutates the module dict directly; nothing assigns).
- tests: parametrized wiring contract for all 8 lazy-installable
platforms — ensure_deps_fn present and distinct from check_fn
(behavior contract, not identity snapshot, so renames don't churn it).
- gateway/config.py: rewrite the stale enablement-pass header comment that
still described check_fn as 'the single source of truth for are-my-env-
vars-set' / 'lazy-installs it' — both false under the new contract.
- teams: check_requirements docstring wrongly claimed credential checks
(body checks only SDK/aiohttp presence); derive install_hint from the
canonical LAZY_DEPS pins + sys.executable instead of hardcoding
'~/.hermes/hermes-agent/venv/bin/pip' and version pins (wrong under
HERMES_HOME overrides / profile installs; pins go stale on CVE bumps);
connect() fatal-error hints now point at the venv pip instead of bare
system pip (the PEP 668 trap the docs warn about).
- teams docs: drop exact version pins from the two manual-install commands
(LAZY_DEPS is the source of truth; unpinned installs still work and the
text can't go stale).
- hermes_cli/status.py: per-entry exception guard around check_fn so one
raising probe can't abort the listing of all remaining plugin platforms
(aligns with the other three call sites).
- tests: rename test_register_check_fn_is_active_lazy_installer ->
test_register_splits_passive_probe_from_active_installer (name said the
opposite of what it verifies).
PlatformEntry.check_fn served three contradictory roles: adapter-creation
gate, config auto-enablement gate, and status display. Plugins had to pick
one function for all three:
- Active installer as check_fn (discord/slack/telegram/matrix/dingtalk/
feishu): every status display could pip-install SDKs as a side effect
(the desktop 94% boot-loop class).
- Passive probe as check_fn (teams, wecom_callback): create_adapter()
returned None before connect() could lazy-install, so the SDK never
installed (#79812 deadlock; wecom_callback's platform.wecom_callback
LAZY_DEPS entry was dead code).
The split makes both call sites correct by construction:
- check_fn is now contractually PASSIVE (probe only, never installs).
- New optional PlatformEntry.ensure_deps_fn is the ACTIVE installer;
create_adapter() runs it exactly when check_fn is False — the platform
is enabled+configured and the gateway is about to connect it.
- Config enablement keeps a configured platform whose deps are missing
but installable; the install itself is deferred to create_adapter().
- Status surfaces (_platform_status, hermes status) read only the
passive probe and can never trigger pip.
Migrated all lazy-installable platform plugins to the split; platforms
with no optional deps (irc/ntfy/buzz/simplex/line/a2a/...) are unchanged
— no ensure_deps_fn means a False check_fn stays a hard block.
wecom_callback gains a working installer for the first time.
Builds on @xxxigm's #79812 (both commits cherry-picked with authorship
preserved), reworking the check_fn swap into the two-field split so the
Teams fix doesn't reintroduce install-on-status.
Platform registry create_adapter() gates on check_fn before the adapter
exists, so wiring the passive probe permanently blocked connect() and
the existing check_teams_requirements() lazy-install never ran.
Adversarial review of the salvaged recovery found a reachable fail-open:
compression continuations inherit the rotated agent's model_config
verbatim (publish_compression_child callers pass
agent._session_init_model_config), so a delegate subagent's continuation
carries _delegate_from=<the delegate's own parent>. The marker-PRESENCE
filters in reopen_orphaned_compression_session and
find_live_compression_child misclassified such a REAL continuation as a
delegate child:
- reopen: parent 'orphaned' -> reopened while a live continuation exists
-> two live heads in one lineage (verified with a live repro)
- find_live: adoption misses the continuation (fail-closed, masked the
fork pre-PR; the PR made it active)
Fix: markers only disqualify a child when they point at the queried
parent (shared _NON_CONTINUATION_CHILD_FILTER_SQL fragment, also
resolving the duplicated-SQL drift risk flagged by the reuse reviewer).
Both directions regression-tested: reopen fails closed on an
inherited-marker continuation; find_live adopts it.
Also from review: reopen-failure log raised debug->warning (the failure
hard-fails the turn moments later), commit-semantics hardening comment
on the lease DELETE path, blank-line nit.
The three read-only projection walks (get_compression_tip,
list_sessions_rich chain, resume walk) share the marker-presence shape
but fail closed (skip a continuation -> resume shows the parent), and
the fixed adoption path self-heals that case at turn start; left as-is.
Salvaged from PR #57342 by @liuhao1024 (with the injection-scan half
from PR #57360 by @ghedeselmabot): cronjob(action='run', prompt=...)
silently discarded the prompt argument — per-run context never
reached the spawned cron session.
The prompt is now threaded as extra_prompt through the whole chain
(cronjob run action → _try_dispatch_background_run/_execute_job_now →
_run_claimed_job → run_one_job → run_job → _build_job_prompt) and
appended to the stored prompt under a '## Run Context' header for
that single fire only — never persisted to the job definition. It
passes the same strict _scan_cron_prompt injection scan as stored
prompts before firing, and works identically on the background and
sync fallback paths.
Test fakes across tests/cron/ updated to accept the new kwargs
(sibling-test blast radius from the signature change).
Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
Pins the scheduler-boundary contract: extra_prompt is appended under
'## Run Context', does not mutate job['prompt'], and the header is
absent when extra_prompt is omitted.
Addresses review feedback from harjothkhara on PR #57342.
Salvaged from PR #53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.
Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, #60432) and mark_running_jobs_interrupted,
which previously could not see them.
The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.
Co-authored-by: izumi0uu <izumi0uu@gmail.com>
A manual cronjob run executed the job synchronously on the calling
agent's tool thread. A cron job is a full agent run that routinely
takes minutes to hours, so the parent turn sat inside ONE tool call
the whole time: uninterruptible (the interrupt flag is only checked
between loop iterations) and serial (a batch of manual runs executed
one by one). A Telegram session that kicked off dozens of new jobs
'right now' was wedged for hours ignoring every interrupt.
action='run' now rides the async-delegation rail delegate_task
background mode uses: the at-most-once claim is taken synchronously
(so paused/missing/already-firing jobs still report immediately),
the run executes on the shared daemon executor, the tool returns at
once with a delegation handle, and the job's outcome re-enters the
conversation as a type='async_delegation' completion event through
the existing completion-queue drains (CLI + gateway) — preserving
message-role alternation and the prompt cache.
Sync fallbacks preserved:
- no routable session (direct Python callers, hermes cron run)
- async delivery unsupported (hermes -z, cron child sessions,
Kanban workers, stateless HTTP)
- dispatch pool at capacity (claim already taken — runs inline
rather than stranding it)
The completion block reports ok/failure, delivery target, next
scheduled run, and an excerpt of the job's saved output.
Inspired by virgiliojr94/book-to-skill (MIT): /learn now picks the skill
shape by the source. Workflows and small sources still get one tight
SKILL.md; books, paper stacks, specs, and large doc corpora get a
knowledge-base layout — a lean always-loaded SKILL.md index plus one
distilled file per chapter/topic under references/, loaded on demand via
skill_view so query cost stays proportional to the answer.
- agent/learn_prompt.py: new _KNOWLEDGE_SKILL_STANDARDS block (index +
per-chapter references/, structure-not-summary distillation, never
reproduce source passages, fold-in instead of duplicating) and a
_SOURCE_HYGIENE block pinning extracted source text as data and
dropping invisible/bidi Unicode (Trojan Source class). Clarified that
the ~200-line cap and hub-skill ban apply to SKILL.md itself, not a
knowledge skill's own references/ files.
- tests: contracts for the knowledge-base layout, the three embedded
standards blocks, and the source-hygiene coverage.
- docs: skills.md documents the knowledge-base shape.