With capture:auto the desktop always preferred client streaming, so a local
desktop with a working backend mic silently switched from PortAudio to
getUserMedia default-device — dropping wake_word.input_device selection
(#74363). A ready backend input now wins under auto; client capture is the
fallback for a preferring surface on a mic-less backend, and capture:client
still forces streaming.
Also removes the dead auto branch (both arms returned local) and lets the
client-feed test skip cleanly when numpy is absent.
Remote headless backends have no PortAudio mic, so "hey hermes" fails even
when openWakeWord is installed. Let the desktop stream 16 kHz int16 PCM via
wake.feed while detection stays server-side.
- wake_word.capture: auto|local|client (+ GUI client_capture prefer)
- WakeWordDetector external_audio queue + feed_audio API
- wake.feed RPC; wake.start/status report capture + frame_length
- Desktop getUserMedia feeder; stop on wake.detected, re-arm after voice
- Docs + unit tests (26 pass in tests/tools/test_wake_word.py)
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
Two follow-ups from the voice PR (#70509).
1. macOS ARM64 onnx migration. Existing users who pinned
openwakeword.inference_framework=onnx before the tflite fix landed kept a
wake word that arms but never fires (ONNX's embedding model is broken on
Apple Silicon, upstream #336). New resolve_inference_framework() honors an
explicit framework everywhere ONNX actually works, but coerces the one
provably-dead combination (explicit onnx + macOS ARM64) to tflite with a
one-time warning. No config mutation; empty still falls back to the platform
default. Both read sites (engine init + requirements check) route through
the shared resolver.
2. Voice turn-timeout leak. Each listen cycle reassigned turnTimeoutRef
without clearing the prior 60s timer, so a stale timer from an earlier
cycle could fire handleTurn() mid-way through a later listen — after enough
idle re-listens this wedged the loop into a non-re-arming state (the
'voice chat deactivates after ~a minute' report). Clear before re-arm.
Tests: 64 wake tests (added onnx-coercion / intel-kept / tflite-kept /
empty-default cases; updated the stale 'explicit onnx kept on ARM64' test
that encoded the old broken behavior), 39 desktop voice/wake vitest, tsc +
eslint clean.
'hey hor' triggered the wake word: the default sensitivity was 0.5, which
for openWakeWord IS the raw per-frame score threshold — openWakeWord's own
permissive baseline that near-misses clear. Raised the default to 0.6 so
phonetic near-misses fall short while real 'hey hermes' (typically 0.9+)
still fires easily.
Also fixed a real cross-engine inconsistency found while checking: the
sensitivity knob is documented 'higher = stricter' and behaves that way
for openWakeWord (threshold = sensitivity) and sherpa (0.05 + 0.4*s), but
Porcupine's own 'sensitivities' param runs the opposite way (higher = MORE
false alarms, per Picovoice). Turning sensitivity up made Porcupine looser
— backwards. Now inverted (1 - sensitivity) so 'higher = stricter' holds
for every engine.
- tools/wake_word.py: default 0.6; _sensitivity fallback uses _DEFAULTS;
Porcupine sensitivity inverted with rationale
- hermes_cli/config.py + docs: default + consistent-direction note
- tests: Porcupine inversion, default>=0.6 regression, fallback-to-default
openWakeWord scores one ~80ms frame at a time and the detector fired the
instant a SINGLE frame crossed threshold — so a stray phoneme in background
conversation could trigger the wake word unintentionally (reported in
testing). A real utterance of the phrase holds a high score across several
consecutive frames; an ambient blip spikes just one.
_OpenWakeWordEngine now requires N consecutive over-threshold frames
(wake_word.confirmation_frames, default 3) before firing. The streak resets
on any sub-threshold frame and on engine reset() (pause/resume), so a
pre-pause frame can't count toward a post-resume fire. confirmation_frames=1
restores the old single-frame behavior; clamped 1..10.
Only openWakeWord is affected — sherpa (streaming transducer) and porcupine
decode the whole phrase internally and already reject single-frame spikes.
- tools/wake_word.py: _confirmation_frames() accessor, streak logic in
process()/reset(), config default
- hermes_cli/config.py: wake_word.confirmation_frames documented default
- tests: 5 new (spike rejected, sustained fires once, =1 legacy behavior,
reset clears streak, config clamp) — 58 wake tests green
- docs: 'Reducing false triggers on ambient speech' section
openWakeWord's ONNX backend returns near-zero scores on Apple Silicon
(dscripka/openWakeWord#336), so "Hey Hermes" never crossed the 0.5
threshold: the listener armed, the microphone worked, and nothing fired.
Bisecting the pipeline puts the fault in exactly one stage — feeding the
same audio through both backends, the melspectrogram front-end is
bit-identical (maxdiff 0.00000) and the wake classifier agrees on
identical features, while the shared embedding model diverges by 45.44.
Cross-feeding confirms it: tflite features scored through the *onnx*
classifier give 0.9948 vs 0.000009 for onnx features. A telling
secondary symptom is that scores fall as input gets louder (0.5x ->
0.00031, 8x -> 0.000066), which is garbage inference rather than a weak
detection.
Selecting tflite in config alone does not fix it. openWakeWord hardcodes
`import tflite_runtime.interpreter` but declares tflite-runtime for
`platform_system == "Linux"` only; on macOS the equivalent wheel is
ai-edge-litert, so that import always fails and model.py silently
downgrades back to onnx. The result is a detector that reports itself
listening and can never fire.
- default the backend per platform (tflite on macOS ARM64, onnx
elsewhere) instead of hardcoding onnx, and pick the matching bundled
model artifact
- bridge tflite_runtime -> ai_edge_litert through sys.modules, in-process,
with no writes to site-packages
- refuse the silent onnx downgrade on macOS ARM64 and report the missing
runtime through check_wake_word_requirements() so the GUI surfaces an
actionable hint rather than arming a dead ear
- lazy-install ai-edge-litert via its own feature key, because lazy-dep
specs cannot carry PEP 508 markers (_spec_is_safe rejects ";")
An explicit `inference_framework` in config still wins, so anyone pinning
a backend keeps it.
Verified on macOS 26.5.2 / M-series: "hey hermes" scores 0.0005 on onnx
and 0.9423 on tflite from the same clip, with cross-phrase controls at
0.0003. Live over-the-air through the real microphone fires 4/4
utterances (peak 0.9532).
Two fixes from live testing (Teknium):
1. Desktop installs now ship the wake/voice stacks up front.
install.sh + install.ps1 desktop stages run 'uv pip install
-e .[wake,voice]' (best-effort, lazy-install remains the fallback)
before building the app, so the first ear-click arms instantly
instead of sitting through a multi-minute onnxruntime download.
CLI-only installs keep the lazy path — [all] curation unchanged.
2. The vanished ear: the STT/TTS gate made wake.status call
check_tts_requirements(), whose edge path runs _import_edge_tts →
lazy_deps.ensure — a synchronous PIP INSTALL inside a status poll.
On a venv without edge-tts that blew the desktop's 30s RPC timeout,
armWakeWord caught the error, the atom never learned enabled=true,
and the ear unmounted. _tts_ready is now a pure probe: deps missing
+ lazy installs allowed counts as ready (installs at first speak)
WITHOUT touching pip; check_tts_requirements only runs once deps
are present. Regression test asserts the probe never calls it while
deps are missing.
The wake loop is wake → record → STT → agent → TTS. Arming without
either end configured gives a mic that hears you and then does nothing
perceivable — a useless experience. check_wake_word_requirements now
probes both (same probes /voice uses: stt.enabled + provider != none;
check_tts_requirements) and refuses with a pointer to `hermes tools`
naming exactly which half is missing. The desktop ear hides (available:
false already hides the button), /wake on prints the hint on CLI/TUI.
wake.start also validates requirements BEFORE persisting
wake_word.enabled, so a refused gesture can't leave config claiming on
while nothing can ever arm.
Tests: per-half and both-missing hint assertions; existing requirements
tests pinned via _voice_loop_ready so they don't depend on the test
venv's installed voice stack. E2E: stt.enabled=false in a real config
-> unavailable with the speech-to-text hint.
Internal testing (macOS): clicking the ear froze the button for ~30s,
then it went blue but 'hey hermes' never fired even though STT worked.
Two distinct bugs:
1. Frozen-then-timeout button: first-use wake.start lazy-installs the
detection engine (onnxruntime is a large wheel), which blows the
desktop's default 30s WS request timeout — the RPC 'failed' client-
side while the backend kept installing and armed later on its own.
wake.start now gets a 180s budget and the pending state says
'arming — first use may take a minute while the engine installs'
instead of a silent disabled button.
2. Armed but deaf: macOS grants mic permission per PROCESS. The
renderer having mic access (STT working) does not grant the Python
backend anything — CoreAudio hands an unentitled process a 'working'
stream that delivers zeros forever, so the listener looks healthy
and can never hear the phrase. The detector now tracks frame peaks:
10s of consecutive near-zero frames sets audio_silent, logged with
the exact macOS Settings path, cleared automatically when audio
appears. Surfaced everywhere: wake.status (audio_silent + hint),
desktop ear tooltip (kept visible while listening), /wake status on
TUI (⚠ line) and classic CLI, plus a docs troubleshooting section.
Tests: detector silent-flag set/recover cycle (fake silent/loud
streams), desktop tooltip keeps the dead-mic hint while listening.
44 wake Python tests, 22 desktop + 14 TUI vitest green.
check_wake_word_requirements() gated 'available' on the audio probe,
but the probe imports sounddevice + numpy — two of the packages the
lazy installer would install. On a fresh machine deps_ok was False, so
audio_ok was always False and /wake on printed the manual pip hint and
bailed before the engine constructors' lazy_deps.ensure() could run.
Now the audio probe only runs once deps are installed; with deps
missing and lazy installs allowed (the default), /wake on proceeds and
ensure() installs the pinned engine deps in-process — no restart. The
manual pip hint remains for security.allow_lazy_installs=false, and a
mic hint still blocks when deps are present but no audio device works.
The CLI announces the one-time engine install so the pause is explained.
96-utterance TTS matrix (6 enrolled profile phrases x 4 voices/accents +
4 negative phrases x 4 voices) through the real engine: the old mapping
(default threshold 0.35) missed 3/24 positives; remapping so sensitivity
0.5 lands on sherpa's recommended 0.25 recovers 2 of 3 while keeping
0/16 false fires. Detection 23/24, routing accuracy 23/23.
One sherpa listener now enrolls every wake-enabled profile's phrase
(defaulting to "hey <profile name>") and reports WHICH phrase fired.
wake.detected gains a profile field; the desktop live-switches to the
matching profile (same path as the profile rail), opens a fresh session
there, and starts hands-free voice. The single-profile CLI/TUI print the
hermes -p switch command for foreign-profile phrases instead of
answering as the wrong profile. Opt out per listener with
wake_word.profile_routing: false.
New "sherpa" wake_word provider: the configured phrase is BPE-tokenized
at runtime against a small streaming zipformer KWS model (~13 MB English,
one-time download cached under HERMES_HOME), so ANY typed phrase works
with zero training — including per-profile phrases like "hey coder".
wake.sherpa lazy-dep group + [wake] extra grow sherpa-onnx/sentencepiece;
requirements probe routes per provider; sensitivity maps onto sherpa
keywords_threshold. E2E-verified on real audio (target phrase fires,
foreign phrase stays silent, reset drops buffered state).
From #53378: ships hey_hermes.onnx/.tflite (openWakeWord pipeline,
Apache-2.0) under tools/wakewords/, resolves the default (and hey_hermes
aliases) to the bundled file, ensures openWakeWord base feature models
are fetched for custom paths too, and updates config defaults + docs
from hey_jarvis to hey hermes.
Two bugs surfaced by the desktop wake conversation:
1. Runaway loop: wake -> voice -> resume -> wake fired again within
~200ms. openWakeWord keeps its rolling feature buffer across
pause/resume, so on resume it immediately re-scored the "hey jarvis"
captured before the pause and re-fired, reopening a session and
restarting voice in a tight cycle. Reset the engine buffer on every
detector (re)start so resume begins from clean audio.
2. Empty-transcript toast: a silent re-listen returns
success:false / "… STT returned empty transcript", which the desktop
transcribe endpoint turned into a 400 -> thrown error -> "Voice
transcription failed" notification on every silent gap. Treat an empty
transcript as no-speech: return {ok, transcript: ""} so the voice loop
quietly re-listens. Real failures still 4xx/5xx.
The detector logged listen/detect/close at debug, invisible at the
default level. Promote listen-start, phrase-detected, stream-closed, and
the wake.start outcome (disabled / unavailable / listening) to INFO, and
log wake.detected emission, so a non-triggering setup is diagnosable from
gateway/gui.log without flipping global log levels.
Makes the wake word a tri-surface feature with one configurable owner.
- wake_word.surface ("auto" | "cli" | "tui" | "gui") + shared
wake_surface_enabled() gate consulted by every surface, so exactly one
place owns the listener and the new session it opens.
- tui_gateway: wake.start/stop/pause/resume/status RPCs + a wake.detected
event, sharing one server-side detector for both TUI and desktop. The
detector yields the mic to voice.record (pause on capture start, resume
on terminal) and to the desktop's browser mic (wake.pause/resume).
- TUI (Ink): arm wake.start on gateway.ready; on wake.detected open a
fresh session and start voice capture.
- Desktop (Electron): arm wake.start on connect; on wake.detected open a
fresh session.
- CLI now gates on wake_surface_enabled("cli"); /wake status shows surface.
- Tests for the surface gate; docs cover the surface knob + cross-surface.
Adds an opt-in, on-device hotword listener for the CLI. With
wake_word.enabled (or /wake on), Hermes listens in the background for a
wake phrase; on detection it starts a fresh session, captures one
utterance through the existing voice pipeline, and answers — the
"Hey Siri" pattern.
- tools/wake_word.py: provider-pluggable detector (openWakeWord, free
local default; Porcupine, premium) over the shared 16 kHz sounddevice
capture path. Background daemon thread with pause/resume so it yields
the mic during a voice turn.
- CLI wiring: startup listener (off-thread), on-wake flow, an idle
watchdog that resumes the detector after each turn, cleanup hook, and
a /wake [on|off|status] command.
- config.yaml wake_word section; PORCUPINE_ACCESS_KEY as an optional
secret. Engines lazy-install via the [wake] extra.
- Hands a transcript to the input queue exactly like voice mode, so no
system-prompt/cache mutation. No new core model tool.
- Tests (mocked, no live audio/network) + feature docs.
Brooklyn Nicholson
Andrew
Gille
teknium1
Ben Barclay
Omid Saadat