The _standalone_send() function (used by the send_message tool for
proactive/scheduled sends) has the same bare `@<id> text` bug that
send() had before PR #44444. SimpleX's `@<x>` syntax resolves x as a
display name, not a contactId — the daemon silently drops messages
when it cannot find a contact named "6".
Use the structured `/_send @<id> json [...]` form, matching what
send_image, send_document, and the send() fix already use.
Fixes#46265
Adds regression test for the case where both disk and live start_time
are known integers but differ (e.g. stale value from a previous run).
The self-PID short-circuit must fire regardless — start_time only
guards PID reuse for *other* PIDs. Inspired by #81495's test case.
After Discord reconnect, on-disk start_time can be null while the live
record has a fingerprint. Requiring equality made the gateway treat its
own PID as a foreign token holder (#81468).
Open the selected microphone at its reported default rate and convert each capture block to the 16 kHz frame expected by wake-word engines. Add a regression covering a 48 kHz WASAPI device.
Co-authored-by: clyu168 <clyu168@126.com>
Two claim-failure diagnostic paths (cronjob_tools.py:629,921) still used
the old inline 'not enabled or state==paused' check. After get_job()
normalizes via effective_job_state, a half-paused record has
state='scheduled' and enabled=True, so the inline check returned False —
mislabeling the job as 'already being fired' instead of 'paused/disabled'.
Also hoists effective_job_state/is_job_runnable to the top-level import in
cronjob_tools.py (was function-local) and updates console_engine.py's
_format_job to use effective_job_state instead of the old inline
state-or-enabled derivation — a fourth display path the original PR missed.
Follow-up to PR #81287.
pause_job already sets enabled=false atomically with state/paused_at, but
get_due_jobs only checked enabled — so a contradictory record
(enabled=true + paused_at/state=paused) still fired. That was the 07-30
outage failure mode: list looked frozen, fleet kept merging.
- is_job_runnable / effective_job_state: pause markers gate fire; display
derives from the scheduler-honoured enabled flag so half-paused never
renders as [paused]
- get_due_jobs self-heals enabled=false + logs error on contradiction
- claim_job_for_fire uses is_job_runnable (paused_at counts too)
- list/format paths use effective_job_state
- behavioural tests: pause blocks due fire; half-pause self-disables
SessionResumeTooLargeError said 'across its lineage' even when the CLI
mid-setup path counted only the tip segment; the exception now takes a
scope phrase.
With sessions.max_*_messages: 0 the guards previously still ran an
unbounded COUNT (full lineage for resume) — the exact pathological work
disabling them is meant to avoid. Live callers use the raise side
effect only, so return 0 without touching the messages table.
The salvage's getLatestSessionMessages/getAllSessionMessages split left
two desktop test files mocking the old getSessionMessages name (vi.mock
partial-mock let the real function through, so calls hit the un-mocked
path). Also braces the new single-line if guards per the curly lint
rule.
OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
The mid-setup CLI resume path loads only the tip session's rows, so
gate it with a tip-only count instead of the full-lineage count (which
over-rejected heavily-compressed sessions). Transient guard failures
(locked DB, adaptor stores) now log and proceed instead of blocking
resume with a new error.
The remote interceptor rebuilt session/messages requests from pathname
only, silently dropping limit/offset/order. Against a paginating remote
backend, getAllSessionMessages would refetch the same default page until
the safe-load guard threw, breaking export/artifacts/branch for remote
sessions over one page.
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
Thread.start() failure is RuntimeError; catching BaseException here
swallowed KeyboardInterrupt/SystemExit without re-raise (unlike _worker,
which forwards them into the future).
- Heartbeat tests: holder signals a threading.Event after acquiring
_SKILLS_PROFILE_LOCK; the scenario waits on it via run_in_executor
instead of sleeping 50ms and hoping.
- Fix the TestConfigMutationLock comment to describe the probabilistic
slow-save interleave the code actually implements.
- New regression test: PUT /api/dashboard/plugin-providers must hold
_CONFIG_MUTATION_LOCK — a concurrent locked writer survives.
Two follow-ups to the off-loop move, from external review (both verified,
the second larger than reported):
- Config read-modify-write handlers moved to worker threads could now
interleave — _CONFIG_LOCK covers each load/save individually, never the
span between them; the event loop used to serialize these accidentally.
New _CONFIG_MUTATION_LOCK (worker-threads only, so it can never block
the loop) held across the whole load→mutate→save span in all seven RMW
handlers. update_config_raw skipped: it's a full-document replace with
no server-side read, so a lock cannot close its client-side window.
- The review flagged two skills routes still taking _SKILLS_PROFILE_LOCK
on the event loop; a systematic audit of hermes_cli/web_routers/ found
24 on-loop routes (skills 5, mcp 9, tools 10, cron 1). All moved to the
same inner-_run + asyncio.to_thread pattern, mutating ones under the
mutation lock, uniform lock order (_SKILLS_PROFILE_LOCK →
_CONFIG_MUTATION_LOCK). Await-safe _config_profile_scope routes, plain
def routes, and already-threaded routes unchanged.
Regression tests: concurrent theme+font updates both survive (fails with
the lock nulled: "theme write lost to a concurrent font write"); event
loop stays responsive while the profile lock is held during GET
/api/skills. 214 tests passing across the touched suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diagnostics loop watchdog caught GET /api/config freezing the gateway
event loop for >1s, stack-sampled blocking on _SKILLS_PROFILE_LOCK inside
_profile_scope. Any async handler that entered _profile_scope (process-wide
threading lock) or called load_config()/save_config() on-loop could stall
every chat and WebSocket at once while a slow lock-holder ran.
Move 28 such handlers to the existing inner-_run + asyncio.to_thread
pattern (contextvar-safe: the whole scope enter/body/exit stays inside one
worker thread). Handlers using the await-safe _config_profile_scope, plain
def endpoints (FastAPI threadpool), and tui_gateway's contextvar-only
decorator are unaffected and unchanged.
Regression test holds _SKILLS_PROFILE_LOCK in a thread while calling
GET /api/config and asserts an event-loop heartbeat keeps ticking; it fails
against the pre-fix code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).
Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).
Fixes#10809
An auth failure on the very first connect returned out of the run loop
instead of parking. That ended the run task, and the task is the only
listener on _reconnect_event — so the server stayed dead for the life of
the process. `hermes mcp login`, a /mcp refresh, and the 300s self-probe
all had nothing left to wake, and the only cure was a full restart.
_classify_mcp_failure already calls 401/403 "permanent" and documents
that run() parks those immediately; the early return above it meant auth
was the one permanent failure that never got there. Park it with the
others and keep the tailored log line, now pointing at `hermes mcp
login <server>`.
The windows-compat change-detector checked for the literal string
'start_new_session=True', but the systemd scope isolation path
conditionally uses start_new_session=False (the scope creates its own
session/cgroup). Assert 'start_new_session=' instead — the value may
now be a variable.
The entry_data/lock_held path (used by mark_turn_active/clear_turn_active)
was eagerly constructing a full O(n) snapshot of all routing entries under
the lock on every turn, even when the DB upsert would succeed. Defer the
fallback_data construction to the except branch where it's actually needed,
and respect lock_held to avoid re-acquiring a non-reentrant _lock.
When Hermes runs as a systemd gateway with MemoryHigh/MemoryMax limits,
local background terminal commands (terminal(background=true)) inherit the
gateway's cgroup. A memory-heavy executor (Codex, tests, Node) can push
the whole cgroup past MemoryMax and trigger systemd-oomd to kill the
ENTIRE gateway — taking down the messaging control plane and silently
losing the active turn.
Root cause: tools/process_registry.py::spawn_local() uses
start_new_session=True (creates a process session/group, NOT a resource
cgroup). The spawned process tree stays in the gateway's systemd cgroup.
Fix: when running under a service manager (detected via the existing
is_gateway_supervisor_process() helper), wrap the pipe-mode spawn command
in 'systemd-run --user --scope --unit=hermes-worker-<id>' so the worker
gets its own transient cgroup. An OOM in the worker then kills only the
worker, not the gateway.
The systemd-run availability is probed once (a no-op /bin/true in a
transient scope) and cached, because the binary can exist on PATH while
the user D-Bus session is unavailable (system services, containers). If
unavailable, fall back to the current start_new_session=True behavior
with a debug log.
Scope: this covers the common background pipe-mode path. PTY mode
(PtyProcess.spawn) is left as future work — it uses a different spawn
mechanism and is used for interactive CLI tools where cgroup isolation
has additional considerations.
- Fix _usage_audit_path() to use _get_hermes_home() instead of hardcoded
Path.home() / '.hermes' (profile-safe resolution, sweeper finding)
- Rewrite skip_background_review tests to exercise finalize_turn() directly
instead of duplicating the guard expression (sweeper finding)
- Fix response_silent audit field to use _is_cron_silence_response()
instead of the buggy SILENT_MARKER substring check it was meant to
replace (simplify-code review finding)
- Remove dead 'model' in locals() guard — model is always in scope
before the try block (simplify-code review finding)
- Extract _stub_agent_for_finalize() helper to eliminate ~40 lines of
copy-pasted agent stubbing in tests (simplify-code review finding)
- Clean up 'Phase 0.5' instrumentation comments
liuhao1024
kshitij
HexLab98
Gille
rjvandeve
hermes-seaeye[bot]
kinsolee
HenryG
Royalaid
gnanirahulnutakki
brooklyn!
rob-maron
Dominic Bejar
toprakeker