The per-session agent cache is capped at 128 entries with a 1h idle TTL, and
neither bound knows how many bytes it holds. Each cached agent pins
_session_messages -- the full transcript including tool output, tens of MB on
a session with 100+ tool calls -- so a gateway serving many chats keeps every
warm transcript resident: agents that took a turn inside the TTL are never
idle-swept, and the idle sweep additionally defers finalizable sessions until
they expire. RSS climbs until the cgroup throttles and SIGTERM can no longer
flush inside systemd's stop timeout.
Add the missing bound. Each session-expiry watcher tick compares the process's
anonymous RSS against a budget and, when over, sheds LRU agents through the
same soft-eviction path the cap enforcer uses, then runs malloc_trim so the
freed arenas actually return to the OS. Evicted sessions rebuild their
transcript from the persisted session on the next turn.
Three classes of session are never shed: agents mid-turn, the most recently
used ones, and any session whose transcript has not finished reaching disk
(_last_flushed_db_idx vs len(_session_messages) -- the same divergence the FTS
write-corruption guard reacts to when it preserves live history).
memory_high_mb defaults to "auto", deriving the budget from the cgroup limit
the gateway runs under, so a MemoryHigh/MemoryMax on the unit is respected
without a second number to keep in sync. The two existing bounds become
configurable alongside it under agent.agent_cache.
protect_recent is clamped to half the cache: a couple of sessions can exhaust
the budget on their own, and a fixed MRU guard would then protect everything
and leave the gateway climbing with nothing it would shed.
Fixes#80764
The docs promised 'frees ~370MB RAM' — measured behavior on macOS/CPU
is that ctranslate2's allocator keeps the freed pages (RSS doesn't
visibly shrink); the concrete win is VRAM release on CUDA hosts and
process-internal reuse on CPU. Say exactly that instead.
The local faster-whisper model singleton (_local_model) is loaded once
and never released — the 'base' model holds ~370 MB of RAM/VRAM for
the entire lifetime of the process, even when no voice messages arrive
for hours or days. On long-running gateway processes (especially with
local LLMs competing for the same GPU) this is wasteful.
Add a config-driven idle unload: after stt.local.unload_after_idle_seconds
(default 0 = never) of no transcription activity, a lightweight daemon
thread sets _local_model = None so the Python GC can reclaim the
ctranslate2 objects. The next voice message reloads the model
transparently (the existing lazy-load path handles it).
The watcher:
- Checks every 30s whether idle time exceeds the configured threshold
- Acquires _local_model_lock before unloading (prevents races with
concurrent transcriptions that are mid-load)
- Exits immediately if the model is already None (unloaded by another
path, e.g. the CUDA fallback eviction)
- Is restarted by each transcription with the current config value,
so changing stt.local.unload_after_idle_seconds in config.yaml takes
effect on the next voice message without a process restart
Default is 0 (never unload) — zero behavior change for existing users.
Recommended value for gateway processes: 300 (5 minutes).
15 tests: config resolution (garbage/negative/None fallbacks), unload
safety (already-None, lock acquisition), touch timestamp, watcher
lifecycle (unload after timeout, no unload within timeout, exits when
model already None, stopped on new start). Existing STT test suite
unchanged.
Local faster-whisper gets Silero VAD (bf8004e3a) so silence never
reaches the model. Cloud providers got no such protection: the raw
file uploads untouched, so every second of silence in a voice note is
paid for twice — upload time and per-audio-minute billing — and cloud
Whisper hallucinates junk tokens on silent stretches exactly like
local Whisper did before the VAD hardening. A 13s voice note with two
long pauses is billed as 13s of audio to transcribe ~6s of speech.
Close the gap client-side: before uploading to a built-in cloud
provider (groq/openai/mistral/xai/elevenlabs/deepinfra), collapse long
pauses with ffmpeg's silenceremove filter, keeping
stt.cloud_trim_keep_ms (default 300) of every pause so word boundaries
and natural pacing survive. Uses ffmpeg, already a dependency of this
exact path via _transcode_audio_for_stt — no new dependency.
The trim is strictly best-effort — ALL of these upload the original
untouched, transcription never fails because of the trim:
- stt.cloud_trim_silence: false
- ffmpeg/ffprobe missing, trim failure, or timeout
- trimmed result ~empty (mostly-silence clip: the provider, not a
client-side dB heuristic, decides whether it contains speech)
- trim saves <10% (re-encoding for nothing)
Command-type and plugin providers are deliberately NOT trimmed: they
may wrap local CLIs that want the original bytes or run their own VAD.
E2E (real ffmpeg + faster-whisper): 13.2s voice note with 7s pause ->
6.2s upload (-53%); transcript of trimmed audio matches the original
on both utterances. Dense-speech and all-silence WAVs correctly fall
back to the original. 22 unit+E2E tests; STT/voice suite failures
identical to upstream/main baseline (all pre-existing).
Remote headless backends have no PortAudio mic, so "hey hermes" fails even
when openWakeWord is installed. Let the desktop stream 16 kHz int16 PCM via
wake.feed while detection stays server-side.
- wake_word.capture: auto|local|client (+ GUI client_capture prefer)
- WakeWordDetector external_audio queue + feed_audio API
- wake.feed RPC; wake.start/status report capture + frame_length
- Desktop getUserMedia feeder; stop on wake.detected, re-arm after voice
- Docs + unit tests (26 pass in tests/tools/test_wake_word.py)
request_restart was calling stop() immediately, so the requesting turn stayed
in the drain wait set and got force-killed at restart_drain_timeout. Wait for
active work to reach zero first, then stop against an idle gateway.
- Config docs now describe session_stall_timeout precisely: a RECOVERY
notifier for an in-process AIAgent with an adapter-queued follow-up —
not a general gateway/session stall detector — with a per-AIAgent scan
cadence (not globally coordinated per durable session).
- import_sessions documents the deliberate export-includes /
import-resets asymmetry for the activity fields (no resurrected
'working' labels on machines where no agent runs), with a regression
pinning both halves.
- Strip trailing whitespace in contributors/emails/fangliquan@qq.com
(git diff --check housekeeping).
PR #76354 review, scope/contract items + housekeeping.
The post-begin_commit() waiter previously called unbounded future.result(),
so the advertised compression.context_total_ceiling_seconds was silently
unenforced for commit-phase hangs. The commit still must complete (abandoning
an in-flight SessionDB mutation would diverge live messages from durable
state), but the wait is now bounded in increments against the remaining
ceiling: on ceiling breach the overrun is logged (WARNING escalating to
ERROR), surfaced once through the user-visible warning channel via the new
on_commit_overrun callback (wired to _emit_warning in run_agent.py), and the
host keeps waiting in bounded slices until the commit finishes.
Documented guarantee (config comment + docs, en/zh): summary phase bounded
by the ceiling; commit phase logged + surfaced if it exceeds it — never
silently hung, never abandoned mid-commit.
Test updated to assert the surfacing fires (previously accepted a silent
over-ceiling wait); adds coverage that a raising overrun callback cannot
break the commit wait.
Once begin_commit() wins, SessionDB mutation cannot be fence-cancelled;
document that context_total_ceiling_seconds covers the summary phase only
and pin the hang-wait contract in tests.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
Add config-driven glibc malloc_trim for long-lived Hermes processes:
- hermes_cli/mem_trim.py: trim_memory() with configurable cooldown,
RSS snapshot telemetry, and forced-trim INFO logging
- gateway/run.py: periodic trim in gateway housekeeping loop
- tui_gateway/server.py: trim in idle reaper (~every 5 min)
- tui_gateway/slash_worker.py: trim on turn boundary
- run_agent.py: force trim on agent close
- hermes_cli/config.py: context.memory_trim config section
(enabled, cooldown_seconds, log_every_n, info_log_min_delta_mb)
CSA tier-4 reviewed (4 rounds, 0 HIGH/MEDIUM/CRITICAL remaining).
Supersedes PR #63708 + #64591 with enhanced telemetry and gateway/slash_worker coverage.
Adds a new config option terminal.font_family that lets users customize the
CSS font-family for the desktop app's embedded xterm.js terminal.
Previously the font was hardcoded in use-terminal-session.ts:
'JetBrains Mono', 'Cascadia Code', 'SF Mono', Menlo, Consolas, monospace
Now the value from config.yaml (terminal.font_family) is threaded through:
useHermesConfig → PersistentTerminal → TerminalTab → useTerminalSession
When font_family is empty or unset (default), the built-in fallback is used,
preserving backward compatibility. Users with Nerd Fonts installed (e.g.
CaskaydiaCoveNerdFont) can now set:
terminal:
font_family: 'CaskaydiaCoveNerdFont', 'JetBrains Mono', monospace
Closes: #terminal-font-config
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.
The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused #56105 to be reverted in
#56126.
Salvage of #33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.
Co-authored-by: BB-light <BB-light@users.noreply.github.com>
Non-interactive sessions (hermes chat -q, hermes -z) snapshot the tool
registry at AIAgent construction time. If background MCP discovery hasn't
finished, MCP tools are invisible for the entire session — and unlike
interactive mode, there is no between-turns late-binding refresh to recover.
Root cause: wait_for_mcp_discovery() only joins an already-created discovery
thread, so it no-ops if a direct/single-query path reaches agent construction
before MCP startup created that thread. Oneshot._run_agent() didn't call it
at all.
Fix:
- Add ensure_mcp_discovery_before_agent_build() helper to mcp_startup.py:
idempotently starts discovery if needed + bounded wait. Fail-open on errors.
- Add single_query parameter to _resolve_discovery_timeout/wait_for_mcp_discovery:
uses mcp_single_query_discovery_timeout (default 15s) instead of the
interactive mcp_discovery_timeout (1.5s) because one-shot sessions have no
second turn to recover.
- Wire into CLI _init_agent (single_query from _single_query_mode flag set
in cli.py's single-query path) and oneshot._run_agent (single_query=True).
- Interactive sessions unchanged: keep 1.5s bound (between-turns refresh covers).
Closes#38448, #51316, #37013, #68137
Composite salvage of #60017 (chrishart0), #51322 (Bartok9), #38620 (buptwz),
#43544 (halonke), #36882 (vanhoof).
Port from nanocoai/nanoclaw#2748: Docker's built-in 64 MB /dev/shm silently
breaks shared-memory-hungry workloads inside the sandbox — Chromium/Playwright
renderers crash tabs, and PyTorch DataLoader workers die with 'bus error' /
'insufficient shared memory'. tmpfs is lazily allocated, so the higher ceiling
costs nothing until actually used, and usage still counts against the
container's --memory cgroup limit.
- tools/environments/docker.py: --shm-size 1g in resource args (not
cgroup-gated; tmpfs mount option). Skipped when docker_extra_args already
sets --shm-size, or when configured empty/'0' (Docker default).
- terminal.docker_shm_size config key (DEFAULT_CONFIG + all three
config->TERMINAL_DOCKER_SHM_SIZE env bridges: CLI, gateway, config.py map)
- tests: default emit, custom value, opt-out, extra_args precedence,
helper edge cases (sabotage-verified: default/custom tests fail without
the emit)
The on/off switch was the only knob. A pass fired after every completed turn,
absorbed exactly one exchange, and there was no way to ask for less. Since a
pass is also what breaks the prompt-cache prefix, "how often does it run" and
"how often do I pay a cache break" are the same question, and it had no answer.
Add `compression.micro_compact_every_n_turns` (default 1, clamped to >= 1). At 1
the behaviour is what it was; at 5 you get a fifth of the breaks and a fifth of
the reclaim rate. The counter advances per invocation rather than per committed
pass, so a turn that finds nothing to absorb still moves the cadence along and
cannot wedge it, and a bogus 0 or negative degrades to "every turn" instead of
silently disabling compaction.
Also expose `micro_compact_defrag_threshold_tokens`, which has been a hardcoded
attribute on the compressor with no path from config since it was added.
This does not give micro-compaction the prune's reclaim-size gate -- a pass
still commits whatever the single absorbed exchange saved. It makes the break
frequency tunable, which reaches the same end by absorbing less rather than by
waiting for a bigger win. The docs now say that plainly, including that a
reclaim threshold is the obvious follow-up and does not exist yet.
Tests cover the skip-until-due window, the cursor and prefix staying untouched
on skipped turns, the clamp, and that the feature is off unless enabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review raised whether default-on can be reconciled with the prompt-cache
contract in AGENTS.md, which permits mutating past context only for context
compression and treats per-conversation caching as sacred. It cannot, and the
codebase already says so in its own words.
A micro-compaction pass rewrites already-sent history, so it invalidates the
cached prefix every turn rather than at an episodic boundary. That is the exact
cost the proactive prune gates against: `proactive_prune_min_reclaim_tokens`
exists, per its own config comment, to keep rewrites to "one big episodic break
instead of a tiny break every tool iteration." Micro-compaction has no
equivalent gate -- one exchange per turn means one break per turn, by design.
Default to off. An operator who wants the amortized stall can opt in with
`compression.micro_compact: true` and accept the tradeoff knowingly; nobody
inherits a per-turn cache break from installing an update.
Also register the key in config_defaults so it is discoverable and picked up by
the update path's new-options check -- it was previously read by agent_init but
declared nowhere -- and document the cache cost in docs/micro-compaction.md
instead of only the benefit. The measurements behind the feature (occupancy
plateau, zero batch compactions) never priced cache invalidation, and the doc
now says which numbers a reader would need to measure to justify enabling it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
database.wal_autocheckpoint / database.journal_size_limit are now real
schema keys (default None = SQLite defaults) so the dashboard config
schema doesn't produce a single-field 'database' category, and the two
pragmas apply_database_pragmas reads are discoverable/documented.
Use database.journal_mode as the sole non-secret operator setting, preserve the vulnerable-SQLite safety gate and existing WAL databases, validate explicit DELETE results, document the active config path, and cover real SQLite openers with behavioral tests.
Replaces the half-duplex per-playback barge monitors with ONE listener
that runs for the entire agent turn in continuous voice mode: armed at
utterance-submit, disarmed when the turn is fully done (response + TTS
finished). Fixes Teknium's live report that voice interruption never
works: (a) not while the LLM is generating, (b) not while TTS plays.
Root causes:
- HALF-DUPLEX GAP: the barge monitor only spawned when TTS playback
STARTED (cli.py streaming/whole-file paths, gateway _tts_stream_begin).
During LLM generation there was NO microphone listener at all.
- PLAYBACK DEAFNESS: the monitor calibrated its VAD noise floor WHILE the
speaker was blasting TTS (speaker bleed baked into the floor), then
multiplied it by 8x with a 1s strictly-consecutive block requirement —
normal speech could rarely reach the trigger, and the 2s grace
swallowed early interjections.
New model — tools/voice_mode.full_duplex_listen():
- Pre-playback calibration: quiet-room noise floor established at turn
start and HELD through playback (never recalibrated against bleed).
- Phase-aware trigger: generation = floor x voice.barge_in_threshold_multiplier
(new config, default 3.0, justified by synthetic-frame tests);
playback = additionally clamped to a 1500-RMS minimum so bleed alone
can't trip; 4000-RMS ceiling keeps speech always reachable.
- Windowed-majority detection (>=80% of a 300ms window) instead of the
strictly-consecutive counter that reset on intra-word energy dips.
- Grace on playback ONSET only (voice.barge_in_grace_seconds, default
down 2.0 -> 0.5) — suppresses the onset transient, not the mic.
- Debug diagnostics at every decision point (calibrated floor, per-window
RMS above 50% of trigger, trip/no-trip, grace suppressions) — always
logger.debug, mirrored to stderr under HERMES_VOICE_DEBUG=1.
Phase behavior (CLI cli.py + tui_gateway/server.py, same model):
- generation: speech interrupts the in-flight turn via the SAME seam the
typed/Ctrl+C interrupt uses (agent.interrupt()), cuts any pending TTS
pipeline so the stale reply never plays, and submits the captured
interjection (pre-roll capture, first syllable kept) as the next turn.
- playback: cuts TTS (streaming pipeline stop + fallback speak stop
events + file player) and submits the capture.
- stop phrase honored in BOTH phases: mid-generation 'stop' interrupts
the turn AND ends the voice chat (stop everything).
- one listener instance spans generation -> playback (no re-arm race);
double-arm refused (CLI _voice_fd_active / gateway _fd_listener_active).
Gateway specifics: _arm_full_duplex_listener() at _run_prompt_submit turn
start and inside _tts_stream_begin; _speak_text_with_barge registers its
(stop, done) pair in _fd_speak_pipelines so fallback speaks are cut and
tracked; _tts_stream_barge_in_monitor kept as a shim that arms the new
listener. Desktop renderer owns its own mic path (voice-barge-in.ts) and
is unaffected; if desktop backend-mic mode is used it inherits via the
gateway.
Tests: full_duplex_listen synthetic-RMS suite (speech-over-bleed trips,
bleed alone doesn't, quiet floor held through playback, grace window,
multiplier math 3x vs 8x, windowed-majority dips), CLI listener phase
tests (generation interrupt seam, playback cut, lifecycle spans phases,
double-arm, config forwarding, stop-phrase-mid-generation), gateway
generation-phase interrupt + stop-phrase tests. Generation-interrupt
test sabotage-verified.
HexLab98
kshitij
texasich
Teknium
Ryder Freeman
Andrew
Carbon
fangliquanflq
fangliquanflq
Hermes agent
webtecnica
baoyu0
BB-light
kshitij
Michael Jordan
Jasmine Naderi
Gille