The Discord semantic thread-rename lane resolved the target thread from
`_relay_auto_thread_info`, which read a single-slot-per-parent-chat cache
(`adapter._auto_thread_by_chat[chat_id]`, populated from connector
SendResult feedback). When two auto-threads spawned from the SAME parent
channel, the second send overwrote the first's slot and the title turn's
read raced the write — so only the FIRST thread in a channel ever got its
semantic rename. Staging repro 2026-08-02: message A's thread renamed to
"A Hundred Word Sword Story", sibling message B's thread stayed stuck at
the raw first-words name.
The connector now stamps `prospective_thread_id` on the inbound (the anchor
message id, which is the id of the thread it will auto-create) — shipped for
per-thread session keying. Reuse it here: it is deterministic and
per-message, so it names the EXACT thread even when several auto-threads
share one channel. `_relay_auto_thread_info` returns it directly (with an
empty initial-name marker) and never consults the collision-prone per-chat
cache; the connector's own created-name guard (`prefer_connector_created`)
still enforces no-clobber, so no initial name is needed gateway-side. The
send-result cache path stays as a fallback for older connectors that don't
stamp the field.
Tests: two new cases in test_relay_threads.py — prospective id wins over a
poisoned cache entry, and two sibling threads in one channel each rename to
their own thread id. Full gateway session + relay suites green (211 passed).
- Guard the thread-id-as-chat_id normalization to Discord only; Slack
and Telegram adapters use parent_channel as chat_id for thread messages,
so the unconditional version broke their handoff keys.
- Apply the same Discord-specific guard to _seed_cron_thread_session in
cron/scheduler.py (sibling site with the same bug, docstring said
'Mirrors _process_handoff').
- Replace the change-detector test with contract tests that verify the
actual invariant: Discord handoff key == organic thread key, Slack
handoff key still uses parent channel (non-regression).
A CLI→Discord handoff creates a dedicated thread and re-binds the CLI
session to it. It built the destination SessionSource with
chat_id = home.chat_id (the PARENT channel) while marking it
chat_type="thread" with thread_id set.
But platform adapters build organic in-thread messages with
chat_id = <thread id> (see the Discord adapter's on_message and
_build_thread_event paths). build_session_key therefore produced two
different keys for the same thread:
handoff: agent:main:<platform>🧵{parent}:{thread}
organic: agent:main:<platform>🧵{thread}:{thread}
So the next real user reply in the handoff thread resolved to a
DIFFERENT session_key and spawned a fresh session instead of continuing
the handed-off one — observed as a stray auto-titled session plus a
session_search fallback (the new session had no prior context).
Fix: for a thread destination, key on the thread's own id so the
synthetic handoff turn and later user replies share one session_key,
matching how adapters key organic in-thread messages.
Adds tests/gateway/test_handoff_thread_session_key.py, which asserts the
handoff key is byte-identical to the organic in-thread key (fails on the
old parent-channel keying, passes on the fix).
The _sync_codex_entry_from_auth_store source guard returned early for
source='manual:device_code', which is the recommended quarantine-safe
configuration (hermes auth add openai-codex produces SOURCE_MANUAL_DEVICE_CODE).
The PR's fix for refresh_token adoption was unreachable for these entries.
Widen the guard to accept both 'device_code' and 'manual:device_code'.
Follow-up to #70111. Issue reporter (imgyf) confirmed this caused a
12-of-16 fleet outage on Aug 1.
Co-authored-by: imgyf <imgyf@users.noreply.github.com>
Two defects in the openai-codex credential pool recovery path:
Defect 1 — adoption path silently no-ops when store_access is empty
_sync_codex_entry_from_auth_store() skipped adoption when the auth
store had no access_token (only last_refresh). When another process
rotated the token pair, the stale profile's entry kept the consumed
refresh_token and replayed it, getting refresh_token_reused and going
terminally DEAD.
Fix: also adopt when store_refresh differs from entry_refresh, even
when store_access is empty. Keep the entry's existing access_token
in that case (store_access or entry.access_token).
Defect 2 — false 'auth refreshed' success log
_try_refresh_codex_client_credentials() returned True whenever
resolve_codex_runtime_credentials() returned any non-empty credentials,
including the same stale token when the underlying refresh failed.
The conversation loop then logged 'auth refreshed after 401' right
before the retry failed with the identical token_expired.
Fix: compare the access token before/after the refresh. If unchanged,
return False so the 401-retry path logs the truth.
Fixes#70097
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
Completes the cache-parity bug class from #76938: the parent's request
body diverges from the fork's not only at the ephemeral system prompt
but also at prefill messages (inserted right after the system message
at API-call time) and, on OpenRouter, at upstream-provider selection
(prompt caches live per upstream; an unpinned fork can be routed to a
different upstream and miss a byte-identical prefix).
Also hardens the tests: pairwise asserts instead of re-implementing the
production prompt join, and routed-path omission guards for the whole
gated kwarg family.
_format_price_per_mtok collapsed any per-Mtok price below one cent to
"$0.00" (and readers treated near-zero as free). Nous Portal's DeepSeek
V4 Flash 0731 promo prices cache hits at $0.0018/Mtok, which rendered as
free in the /model picker and hermes model listings.
Prices under $0.01/Mtok now widen precision to the first significant
digit plus one, with trailing zeros trimmed: 0.0000000018/tok →
$0.0018. Standard prices keep the aligned two-decimal format; exact
zero still renders as "free".
Simplify-pass follow-up on the #68157 salvage (regression-neutral \u2014 the
old temp-file version was equally bare): capture ffmpeg's -loglevel
error output so a CalledProcessError carries the real message, and log
it at the voice-input catch site. Parity with transcription_tools'
ffmpeg call sites. Live-verified: forced ffmpeg failure produces the
captured message in the exception.
pcm_to_wav staged every captured utterance in a NamedTemporaryFile just to
hand ffmpeg an input path, then unlinked it. Feed the PCM to ffmpeg's stdin
instead: one fewer file created, written, read back and removed per voice
utterance, and the try/finally cleanup goes away with it.
The WAV output deliberately still goes to output_path rather than being
captured from stdout. ffmpeg cannot seek on a pipe, so a piped WAV is
written with placeholder 0xFFFFFFFF RIFF/data chunk sizes -- Python's wave
module then reports 2147483647 frames for a 1s clip, and strict readers
misjudge the length. Writing to the real path lets ffmpeg seek back and
patch the header.
Tests cover both halves: that the PCM goes over stdin with no temp file,
and (when ffmpeg is installed) that the resulting header reports the true
frame count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Pass extra_exclude={pid} to _reap_unsupervised_gateway_orphans so the
killed PID isn't double-killed during the sweep (#75936).
- Add extra_exclude param to _reap_unsupervised_gateway_orphans signature.
- Replace bare except:pass with logger.debug for diagnosability.
- Fix existing test (mock _reap_unsupervised_gateway_orphans so it
doesn't scan real processes and trigger conftest live-system guard).
- Add regression test asserting the killed PID is excluded from the sweep.
stop_profile_gateway() only killed the single PID recorded in the pid
file. On repeated restarts, the new process overwrites the pid file
before the old one exits, making older gateway instances invisible
to subsequent stops. Each restart stacked another orphan — observed
as N threads and N replies per Discord message.
After killing the recorded PID, also call _reap_unsupervised_gateway_orphans()
to sweep any remaining gateway processes for this profile. The reap function
already handles the no-systemd guard and SIGTERM+SIGKILL escalation.
Fixes#75936
Re-applied from #50508 onto current main: the executor moved from
tools/ to agent/ and the JSON-error site was refactored away into
_parse_tool_arguments, but these 8 f-string debug calls survived the
move verbatim. Lazy %s formatting skips string interpolation when
DEBUG is off — the Tool-result line interpolates the FULL tool output
(can be 100KB+) on every tool call otherwise.
_handle_cron_fire verified the NAS-minted fire JWT by calling the
fire-verifier inline on the event loop. That verifier resolves the NAS
signing key from a JWKS URL — a synchronous HTTP GET on a cache miss (a
cold PyJWKClient, or a rotated kid the cached client doesn't know) — so a
slow or rate-limited portal stalls the whole event loop and starves every
other adapter sharing it. #64641 already documented this exact symptom
(relay 504s on high-job-count instances) and cut the fetch frequency by
caching the client per URL, but the residual cache-miss fetch still ran
inline on the loop.
Dispatch the verifier the same way the platform HTTP event verifier was
hardened: await a coroutine verifier directly, run a sync one via
asyncio.to_thread so its blocking I/O stays off the loop, and fail closed
(reject with 401, never admit the fire) if the verifier raises — this is
the only inbound that can trigger remote job execution. The verifier's
JWK-client cache is already thread-safe (threading.Lock), so moving the
call to a worker thread is safe.
Adds regression tests: a sync verifier runs on a worker thread rather
than the loop thread, a crashing verifier yields 401 with no fire, and a
coroutine verifier is awaited.
/api/status is the desktop's boot liveness probe (polled ~1/s) but since
#60537 every call ran a full topology scan — per-profile yaml.safe_load
(pure-Python loader), psutil process probes, realpath walks — in the
default executor. On multi-profile installs concurrent polls pile up and
hold the GIL 14-16s, starving the event loop: the WS sidecar cannot
flush gateway.ready, the desktop times out into the next stall, and boot
escalates to the 'Hermes couldn't start' overlay (#60800).
Memoize the scan behind a 10s TTL with a collapse lock so concurrent
polls share one scan. Topology only changes on gateway start/stop, so a
<=10s stale badge is an acceptable trade for not starving the loop. The
cache also keys on the collector's identity: tests monkeypatch
_collect_profile_gateway_topology per case, and the identity check keeps
them hermetic (a swapped collector is a miss) without a reset hook.
py-spy captures during a failing boot land in _profile_platform_ports ->
yaml.safe_load on executor threads (7 profiles, Windows). After: one
cold-start scan, zero recurring stalls, desktop boots.
- Short-circuit the candidate waterfall on HTTP 401/403: an auth wall
proves the endpoint family exists, so probing the alternate URL just
doubles the wasted wait (the reported endpoint takes ~10s to return
401 without a key).
- Stream the probe so 4xx never downloads a slow error body; responses
are closed on every exit path.
- Regression tests: single-call assertion on 401/403 (fails on main),
negative-cache reuse, 404 waterfall preserved, no .json() on 4xx.
Fixes#69905
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simplify-pass advisory on the #25387 salvage: _read_tracker_lock is one
global lock guarding every task's read/search bookkeeping (15 sites). A
hung stat on a dead network mount inside it would stall all of them.
Check-then-recheck matches the dedup mtime pattern 30 lines below: read
the entry under the lock, stat outside, reacquire only to evict.
Review follow-ups on the #25387 salvage:
1. CRITICAL: a cached miss survived out-of-band file creation (terminal
command, external process) for the full 60s TTL — breaking the common
agent pattern 'check for file -> create it -> read it' (live-repro'd).
Serve-side existence guard: one ~free stat before serving a cached
miss; if the path now exists the entry is evicted and the real read
runs. Also fixes the search-root variant (write under a cached-missing
directory). Both mutation-checked.
2. notify_other_tool_call now clears the task's not_found entries too
(belt: the dispatcher calls it for every non-read tool).
3. Tracking parity: the record sites no longer early-return. On upstream,
error results flow through consecutive-loop detection and dedup
bookkeeping; short-circuiting skipped that and broke
TestDedupInvalidationTaskResolution when preceded by
TestSilentFileMisplacementE2E (bisected: the early return at the
read record site was the trigger). Recording is now
side-effect-identical to upstream; serving from the cache remains the
optimization. Also reuse the already-computed _resolved instead of
resolving a second time.
When read_file or search hits a non-existent path, ShellFileOperations
spawns a subprocess to stat the path and another to walk the parent
directory for "did you mean..." suggestions. A typo'd path retried 13
times (observed in the wild) costs 26 subprocess invocations + 13 ls
walks for a result we already know.
Add a per-task negative-result cache keyed by (op, resolved_path) with
a 60s TTL and a hard cap of 500 entries. On hit, return the cached
error JSON immediately and skip the subprocess + suggestion walk.
The cache is namespaced by operation ("read" vs "search") because the
two callers return different error JSON shapes ("File not found:" vs
"Path not found:"). Eviction:
* TTL (60s) — short, so a path that appears later isn't masked.
* write_file / patch on the same path — _invalidate_dedup_for_path
now also drops the negative-cache entry so a freshly-written file
is read from disk on the next call instead of returning a stale
"not found" stub.
Tests in tests/tools/test_file_tools.py cover:
* read cache hit skips the subprocess on retry
* cache is per-task (no cross-task pollution)
* successful reads do not poison the cache
* search cache hit skips the subprocess on retry
* read and search caches are namespaced (different error shapes)
* write_file invalidates the read negative cache
* TTL expiry evicts stale entries
Efficiency-pass follow-up on the #66355 salvage: force=True bypassed the
cooldown entirely, and AIAgent.close() fires a forced trim for EVERY
in-process child subagent close (delegate_tool child.close(), parent
close step 5). A delegate batch of N children closing back-to-back in
the gateway process stacked N+1 uncooled full gc.collect()+malloc_trim
passes (50-500ms each with a large live heap). Forced trims now honor a
5s floor — bursts coalesce, the parent's final close-trim still fires.
Guard test mutation-checked (floor zeroed -> test fails).
Simplify-pass follow-up on the #66355 salvage:
1. _config_settings runs on EVERY trim attempt (before the cooldown
check) and only reads — swap load_config for load_config_readonly.
Deep-copying the whole config per attempt generates exactly the
allocator garbage this module exists to release. Tests re-seamed.
2. Trim-failure logs demoted warning->debug at all 3 periodic sites
(gateway housekeeping, idle reaper, slash worker): sibling failure
branches in the same loops log at debug, and a persistent failure
(e.g. broken import after a partial update) would otherwise warn
every 60s forever.
3. The frame-inspection test now asserts the expected locals exist
before reading them — a rename in _run_prompt_submit fails the test
loudly instead of vacuously passing on None.
Add config-driven glibc malloc_trim for long-lived Hermes processes:
- hermes_cli/mem_trim.py: trim_memory() with configurable cooldown,
RSS snapshot telemetry, and forced-trim INFO logging
- gateway/run.py: periodic trim in gateway housekeeping loop
- tui_gateway/server.py: trim in idle reaper (~every 5 min)
- tui_gateway/slash_worker.py: trim on turn boundary
- run_agent.py: force trim on agent close
- hermes_cli/config.py: context.memory_trim config section
(enabled, cooldown_seconds, log_every_n, info_log_min_delta_mb)
CSA tier-4 reviewed (4 rounds, 0 HIGH/MEDIUM/CRITICAL remaining).
Supersedes PR #63708 + #64591 with enhanced telemetry and gateway/slash_worker coverage.
The detached meet_bot child inherits the process environment, not the parent's contextvar secret scope, so process_manager.start() now resolves HERMES_MEET_REALTIME_KEY/OPENAI_API_KEY through get_secret in the parent and passes it explicitly in the child env (spawn-wrap shape). Adds the consolidated per-family regression test file.
NOTION_API_KEY/LINEAR_API_KEY defaults, DEEPINFRA_API_KEY model gate, FAL_KEY via the file's own _scoped_credential helper, XAI/VERCEL/DAYTONA presence checks, GITHUB_TOKEN/GH_TOKEN + GitHub App creds in tirith/skills_hub, and the masked HERMES_API_KEY display in tui_gateway config.show all route through get_secret.
OPENAI_API_KEY, DEEPINFRA_API_KEY and KREA_API_KEY now resolve via get_secret; the OpenAI client is constructed with the scoped key explicitly instead of relying on the SDK's implicit environ read.
Route HINDSIGHT_API_KEY/HINDSIGHT_LLM_API_KEY, HONCHO_API_KEY, SUPERMEMORY_API_KEY, MEM0_API_KEY and RETAINDB_API_KEY reads through agent.secret_scope.get_secret so multiplexed turns resolve the active profile's key instead of the process environment. Also guard supermemory post_setup's os.environ write on not is_multiplex_active() — writing one profile's key into the process-global env pollutes sibling profiles; the single-profile convenience path is unchanged.
Review follow-up on the #62871 salvage (simplify pass, HIGH):
1. Ops unresolved at the wait deadline were RETAINED in the pending set.
A permanently failing status endpoint (auth error, endless 500s, or a
server that loses ops without 404) would grow the set forever and make
EVERY later prefetch burn the full 10s budget re-polling it — and
prefetch()'s bounded 3s join sits on the reply path, so that money-quote
'adds no response latency' claim breaks. Timed-out ops are now dropped
(identical degradation to prefetch_waits_for_retain=False: possibly
stale recall) with a WARNING so persistent server trouble is visible.
Guard test mutation-checked (fails with eviction disabled).
2. Status polls now spaced 0.5s (was 0.05s shared with the local drain
poll): a wedged op cost up to ~200 get_operation_status round trips
per prefetch; now ~20 max over the default 10s budget.
Address PR #62871 review: with the default retain_async=True, aretain_batch
returns when the write is accepted, not when it's durable/recall-visible, so
draining the local writer queue (task_done) is not a read-after-write signal.
The next-turn prefetch could still recall before the just-completed turn was
observable on the server.
- Track the async operation_id/operation_ids returned by aretain_batch
- _wait_for_retains_drained now applies two ordered, budget-bounded barriers:
(1) local writer queue drains, then (2) tracked server-side async ops report
completion via operations.get_operation_status (an explicit read-after-write
condition). NotFound (completed+evicted) counts as done; transient errors
keep waiting until the deadline
- Completed ops are removed from the pending set so later prefetches don't
re-poll them; the whole wait stays off the reply path
- Add TestPrefetchServerRetainVisibility: op-id tracking (single/multiple),
no-op tracking when retain_async=False, prefetch waiting for server
completion before recall, timeout fallback on a wedged op, and NotFound /
transient-error status handling
Async retain already keeps the memory WRITE off the reply path (writes drain
on the single writer thread while the user gets their response immediately).
This closes the remaining retain/prefetch race: the next turn's warm prefetch
runs on its own thread and could recall BEFORE the just-enqueued retain write
lands, silently dropping the latest turn from recall.
- The background prefetch now waits (bounded) for pending retains to drain
before recalling, so warmed context includes the just-completed turn.
- The wait runs only on the background prefetch thread, never the reply path,
so it adds zero latency to the user's response and loses no writes.
- Bounded by prefetch_retain_drain_timeout (default 10s) and polls
unfinished_tasks so a wedged write can't hang the prefetch.
- New config keys: prefetch_waits_for_retain (default true),
prefetch_retain_drain_timeout (default 10.0).
When hrr_dim=1 the prefixed float32 blob (4+4=8 bytes) collides in
size with a raw float64 blob (1×8=8 bytes), making the format
discriminator in bytes_to_phases ambiguous — a legacy blob starting
with HRR1 would be misread as a prefixed float32 vector.
- phases_to_bytes now accepts an optional dim and falls back to
writing raw float64 when the two blob sizes are equal.
- bytes_to_phases prefers the legacy float64 interpretation when
sizes collide and dim is provided, since phases_to_bytes never
writes a prefixed float32 blob at dim=1.
- Three regression tests cover dim=1 write, round-trip, and the
legacy-prefix collision case.
Addresses hermes-sweeper review on PR #30499.
Review follow-up on the #64169 salvage: _save_entry duplicated
_snapshot_routing_locked's counter-bump line verbatim. The stale-write
protection is a total order over ONE counter — extract
_next_routing_generation_locked() so the two allocation sites can't
drift apart silently.
The steady-state turn only bumps updated_at/last_prompt_tokens on one
routing entry, but persisted it through the full index rewrite twice
per turn (get_or_create_session's healthy-path bump + update_session):
every entry re-serialized, DELETE+INSERT of every gateway_routing row,
and a multi-MB sessions.json dump+fsync — ~50ms p50 at ~1,100 routing
keys in production, out of ~175ms total per-turn gateway persistence.
Metadata-only saves now UPSERT the single row via the existing
HermesDB.save_gateway_routing_entry (<1ms). Structural transitions
(create/recover/reset/switch/prune, compression-tip heals) keep the
full rewrite, which also refreshes the legacy sessions.json mirror.
Correctness: each fast save allocates a per-entry revision from the
routing generation counter under _lock, so fast and full snapshots are
totally ordered by number. Under _save_lock the UPSERT is skipped when
a newer full snapshot or a newer fast save of the same key has already
persisted, and a delayed full rewrite folds in fast records serialized
after its snapshot before writing — an older snapshot can never
overwrite a newer one, in either direction. update_session snapshots
peer fields under _lock so a concurrent reset cannot record a torn
peer row; no DB or a failed UPSERT falls back to the full rewrite so
DB-less installs keep sessions.json durable every turn.
Exercises representative migrated sites per cluster: scoped value wins,
scoped miss no-borrow under multiplex, and unscoped-multiplex behavior per
pattern (in-turn get_secret vs Slack-pattern startup fallback).
AZURE_CLIENT_SECRET and AZURE_FEDERATED_TOKEN_FILE presence checks in the
Entra diagnostics path now read through the profile secret scope (Slack
pattern) instead of raw os.environ.
- discord/telegram/slack/matrix standalone senders read bot tokens via
get_secret (in-turn: they run inside an installed scope — cron scheduler
and delegate spawns propagate it), never borrowing another profile's
env-bridged token under multiplex.
- telegram webhook-secret and matrix access-token/password startup reads
use the Slack pattern (#59739): get_secret, falling back to os.environ
only on UnscopedSecretError.
Ben Barclay
kshitij
Alejandro Moreno
kshitij
JonthanaHanh
Jason Chai
Quark Assistant
Teknium
Jesse Casco
wayne1992127
RelaxJonh
annguyenNous
AlexFucuson9
Frowtek
lost9999
AMATH
Josh Tsai
Kent
Ryder Freeman
And
JabberELF
Soju06