The session event stream (api_server.py:~2236) was the one genuinely
unicode-distinct SSE writer — json.dumps(payload, ensure_ascii=False) +
.encode('utf-8'). Every other writer uses plain json.dumps. Route it
through _sse_frame(..., ensure_ascii=False) so _sse_frame is now the single
source of truth for ALL SSE frame serialization in the module (chat-
completion, responses._write_event, /v1/runs, and the session stream).
Byte-identical for non-ASCII payloads: verified against the historical
inline encoder (raw bytes preserved). The ensure_ascii=False path is now
exercised by test_sse_frame_ensure_ascii_false_reproduces_session_event_stream.
_extend _sse_frame with an explicit ensure_ascii param (default True,
byte-identical to a bare json.dumps) and route the two sibling writers
through it: _write_sse_responses._write_event and the /v1/runs event
stream. This completes the dedup PR #65009 — previously only the five
_write_sse_chat_completion sites used the helper, leaving the other two
writers on inline json.dumps with no shared shape.
No behavior change: every writer's emitted bytes are unchanged (verified
byte-for-byte, including non-ASCII payloads where the default
ensure_ascii=True matches the original inline encoders). The ensure_ascii
option is exposed so a future writer can opt into raw non-ASCII bytes
without fractalizing the format again.
Adds tests/gateway/test_sse_frame.py asserting the byte-contract
invariant between _sse_frame and the historical inline encoders.
_write_sse_chat_completion had five near-identical
f"data: {json.dumps(...)}\n\n".encode() (and one event-tagged variant)
scattered across its role/content/finish/error chunk writes. Pure
extract-method, no behavior change: encoding is byte-identical for every
call site touched.
Left the pre-serialized-string writers elsewhere (_write_event's
json.dumps(..., ensure_ascii=False) path, the /v1/runs SSE writer) alone
— routing them through this helper's plain json.dumps(data) would
silently change their unicode-escaping behavior, which is out of scope
for a pure dedup.
_write_sse_chat_completion and _write_sse_responses bridged their
stream_delta_callback queue into the event loop via
`await loop.run_in_executor(None, lambda: stream_q.get(timeout=0.5))`
in a while-True poll — a thread-pool round trip on every 0.5s tick even
when idle, plus up to 500ms of tail latency between a delta landing in
the queue and it reaching the SSE response.
Add ThreadSafeAsyncQueue (asyncio.Queue + a put_threadsafe() that wraps
call_soon_threadsafe), used by both streaming producer closures
(_on_delta, tool start/complete callbacks — all invoked from the worker
thread running run_conversation via loop.run_in_executor). Consumers
now do a plain `await asyncio.wait_for(stream_q.get(), timeout=0.5)` —
woken immediately when a delta arrives, no executor hop, no poll
interval.
Updated tests/gateway/test_sse_agent_cancel.py's 7 call sites to
construct ThreadSafeAsyncQueue inside the running loop (required, since
it captures asyncio.get_running_loop() at construction) instead of a
bare queue.Queue() at test-method scope.
Sibling site to the disconnect() fix: go_dormant() still did an
unbounded await self._ws.close(), the exact same pattern bounded in
disconnect(). go_dormant runs on the scale-to-zero suspend path (Fly
autostop), which also has timeout constraints. Apply the same 1s
wait_for treatment using _TEARDOWN_AWAIT_TIMEOUT_S.
Found during review of PR #78027.
Offload manual /compress temporary-agent cleanup through the existing
bounded off-loop helper so a slow agent.close() cannot freeze the
gateway event loop, heartbeat, or platform polling.
Guarantee Relay transport teardown even when the runner cancels
adapter.disconnect() during go_idle: shielded finally, 2s drain-path
idle ACK budget under the 5s outer disconnect budget, and bounded
supervisor/reader/ws.close awaits.
Original commits:
- fix(gateway): offload manual /compress cleanup from the event loop
- fix(gateway): tear down Relay transport even if go_idle is cancelled
- fix(gateway): keep Relay disconnect budgets inside the runner window
By @Dannyzen (PR #78027), salvaged onto current main.
When a Discord channel message initiates a relay auto-thread, the thread does
not exist at ingest (source.thread_id is None) — the connector creates it on
its FIRST send and auto-threads any outbound carrying the reply anchor. The
final reply carries that anchor, so it lands in the thread. But the
tool-progress / status bubbles (the "Searching the web for..." updates and the
streaming preamble) were sent with _progress_metadata=None and
_progress_reply_to=None: _resolve_progress_thread_id returns None for Discord
(only slack/mattermost get a synthetic thread), so the progress send had no
anchor and the connector posted it FLAT in the parent channel. Result: the
search-status updates leaked outside the thread while the answer threaded
(staging repro 2026-08-02).
The connector now stamps prospective_thread_id on the inbound (the anchor
message id == the id of the thread it will create). Reuse it: when a
relay-delivered Discord channel-initiate carries prospective_thread_id and has
no real thread yet, carry the reply anchor (event_message_id) on both the
progress metadata (reply_to_message_id) and the progress reply_to, so the
connector routes the progress bubble into the SAME auto-thread as the final
reply. Applied to both the tool-progress path (_progress_metadata /
_progress_reply_to) and the status/interim callback path
(_status_thread_metadata). Events already arriving in a real thread, DMs, and
non-relay sources are untouched (guarded on delivered_via_upstream_relay +
prospective_thread_id + not thread_id).
Tests: two new cases in test_run_progress_topics.py — a relay Discord
channel-initiate asserts every progress send carries the anchor (reply_to +
metadata.reply_to_message_id + non_conversational), and an event already in a
real thread asserts the synthetic-anchor path does NOT engage. Full gateway
progress + relay + session suites green (228 passed).
Salvage of #26860 (hunk 2, ported \u2014 the PR's base predates the current
gateway layout by ~11.9K commits). Messaging platforms can set
gateway.platforms.<key>.skip_context_files: true to skip the
filesystem-heavy context-file discovery (SOUL.md, AGENTS.md,
.cursorrules walks) during AIAgent construction \u2014 10-100x slower
stat()/walk costs on Windows made this a real per-turn tax. Soul
identity is still loaded (single small file), so the persona survives.
The flag participates in _agent_config_signature so toggling it
rebuilds the cached agent instead of silently reusing a prompt built
under the other setting (prompt-cache correctness).
The PR's hunk 1 (mtime-caching the per-turn dotenv reload) was dropped:
df51ad797 mtime-cached load_config/read_raw_config and c2eda92fd
removed the per-turn deepcopies, capturing most of that win; the
function has since gained a multiplex early-return and managed-scope
overlay that the original whole-function skip would have bypassed.
Simplify-pass folds on the #23254 salvage:
- REUSE (HIGH): append_messages_batch now delegates row serialization to
the pre-existing _insert_message_rows helper (already shared by
replace_messages / archive_and_compact / portability import) instead
of adding a third serialization path (_prepare_message_row +
_MESSAGE_INSERT_SQL are gone). One row-writer for every multi-row
path; the row-ID return was consumed by no production caller, so the
batch returns the inserted count.
- QUALITY (HIGH): the compression-lock + compression-closed admission
guards are extracted into _check_transcript_write_guards, shared by
append_message and append_messages_batch (previously duplicated 23
lines that had already needed targeted fixes, #74478). The role-gated
reasoning filtering is no longer duplicated in run_agent.py — it
lives at its one site inside _insert_message_rows.
- EFFICIENCY (MEDIUM, measured): unbounded seed copies hold one BEGIN
IMMEDIATE for seconds (10k rows ~= 2.4s; FTS triggers dominate) and
monopolize the in-process write lock. append_messages_batch grows a
chunk_rows param; all seed/copy call sites use chunk_rows=500. Same
recovery semantics as the old per-row loops, bounded lock holds.
- REUSE (MEDIUM): the two remaining per-row branch-copy loops found by
the pass (gateway/slash_commands.py /branch, hermes_cli
cli_commands_mixin.py branch) are converted to chunked batches too
(AsyncSessionDB's generic to_thread forwarder covers the async site).
Turn-flush benchmark unchanged after the refactor: 2.43 -> 0.87 ms
median per 5-message flush (64% faster).
Follow-up on the salvaged pair: the original guard's `not msg_id` arm let an
id-less internal/synthetic event erase a tracking entry a concurrently-queued
id-bearing message's drain task still needs for recall matching (id-less
events never write entries in _dispatch_inbound_event, so they must never
pop). Tests cover: normal cleanup, id-less non-erasure, overwritten-entry
ownership handoff, TTL eviction + fresh-entry survival.
_build_msg_body_with_mentions() checks the TTL of each _member_cache
entry and returns an empty member list when the entry is stale, but
never removes the entry from the dict. Over time every group_code the
bot has ever queried accumulates a permanent entry, retaining the full
member list (potentially thousands of records per group) until
disconnect().
Fix: delete the stale entry at the point it is detected as expired.
The next call to get_group_member_list_raw() for the same group will
repopulate the cache with fresh data as before.
Symmetric with the existing TTL pattern in MessageDeduplicator, which
evicts on access.
_dispatch_inbound_event() writes session_key → msg_id/raw_text into
_processing_msg_ids and _processing_msg_texts so RecallGuardMiddleware
can find and interrupt the currently-processing message. These entries
were never removed after a message finished processing, causing both
dicts to grow unboundedly — one persistent entry per unique session key
for the lifetime of the bot.
Fix: clear both entries in the _process_message_background() finally
block, after super() returns. The guard compares the stored msg_id
against event.message_id before popping: a concurrent pending message
may have already overwritten the entry in _dispatch_inbound_event while
we were running, in which case the drain task owns it and we must not
clear it. When msg_id is absent (nothing was written at dispatch time)
the pop is a safe no-op.
Note: _msg_content_cache already bounds itself to 200 entries at the
same write site; _processing_msg_ids and _processing_msg_texts had no
such bound.
Two places were using SELECT COUNT(*) when they only needed a boolean:
- has_any_sessions() called session_count() > 1 (full table scan)
- delete_session() used SELECT COUNT(*) WHERE id=? (full matching scan)
Fix:
- Add session_count_ge(n) to SessionDB — short-circuits via
SELECT 1 FROM sessions LIMIT n, returns bool
- has_any_sessions() uses session_count_ge(2) instead of session_count() > 1
- delete_session() uses SELECT 1 ... LIMIT 1 with fetchone() is None
- Add tests for session_count_ge
Re-derivation of PR #65645 onto current main: _build_gateway_agent_history
already runs inside a turn whose config was loaded once into
ctx.user_config; re-reading config from disk via _load_gateway_config()
per turn is redundant. Reuse the loaded turn config.
The runtime footer (`/footer`) shows what model ran and how full the context
is, but not how long the turn took. On a messaging platform there is no
progress bar and no shell timer — a turn that took 4 seconds and one that took
four minutes produce visually identical replies. Users comparing models,
providers, or reasoning levels have no at-a-glance signal for the one
dimension they most often care about, and "was that slow or did I imagine it?"
is unanswerable after the fact.
Adds a `latency` field to the existing footer machinery, rendering the
wall-clock duration of the agent run: `<1s`, `22s`, `1m05s`.
`gateway/run.py` measures with `time.monotonic()` immediately around the
`self._run_agent(...)` await in `_handle_message_with_agent` — the same
function that already builds the footer, so the value is the user-perceived
turn duration (monotonic, so it is immune to wall-clock/NTP adjustment).
`latency` is deliberately NOT in `_DEFAULT_FIELDS`. It is opt-in via
`display.runtime_footer.fields`. Every existing footer — and every footer a
user has today without touching config — renders byte-identically.
This is enforced by tests, not just asserted:
- `test_latency_not_in_default_fields` pins the default tuple.
- `test_resolve_footer_config_default_fields_exclude_latency` pins what
config resolution produces for an untouched config.
- `test_default_footer_renders_byte_identically` pins five exact output
strings for default-config renders **while supplying `turn_seconds`** —
proving that even when the caller measures timing, a default-configured
footer does not show it.
- `test_default_build_footer_line_ignores_turn_seconds` asserts
`build_footer_line(...) == build_footer_line(..., turn_seconds=125.0)`
under default fields.
Adding `latency` to `_DEFAULT_FIELDS` fails 11 of these tests.
No new config surface (reuses `display.runtime_footer.fields`), no new env
vars, no new core tool, no new model-facing schema. One new module-private
helper (`_format_latency`), one new keyword argument threaded through the two
existing footer functions, and 3 lines in `gateway/run.py`.
`turn_seconds` defaults to `None` and the field is skipped when it is `None`
or negative, so any call site that does not measure timing keeps working
unchanged.
`tests/gateway/test_runtime_footer.py` (+185): `_format_latency` boundary
table (sub-second, rounding at 59.4/59.6, the `m{:02d}s` zero-pad, 60m), the
render/skip/opt-in matrix, field-order placement, `build_footer_line`
threading, and the byte-stability block above.
RED-proved by mutation — each of these breaks tests:
- `latency` added to `_DEFAULT_FIELDS` → 11 failures
- dropping the `turn_seconds is not None and >= 0` guard → 2 failures
- `{sec:02d}` → `{sec}` → 6 failures
- `build_footer_line` not threading `turn_seconds` → 1 failure
51 passed in `tests/gateway/test_runtime_footer.py`; 54 passed across the
footer blast radius. `ruff check` clean.
CI slices failed the offload tests with 0.5s witness timeouts: on a
loaded shared runner the event loop thread can take >0.5s to get
scheduled even when NOT blocked, making the probe report a false
positive. A genuinely blocked loop can never set the progress event at
any timeout (the witness coroutine can't run at all), so 5s only
absorbs scheduler flake without weakening the invariant. Mutation
re-verified: reverting the offload still fails all 4 tests.
The sweep-path test parametrizes over _runner/_adapter, which live on
TestGatewayRedeliverySweep; main later added
TestUnconnectedPlatformKeepsItsBudget at the cherry-pick anchor point and
the test landed in that class, where the helpers don't exist
(AttributeError x2). Placement-only move.
request_restart was calling stop() immediately, so the requesting turn stayed
in the drain wait set and got force-killed at restart_drain_timeout. Wait for
active work to reach zero first, then stop against an idle gateway.
- tests/agent/test_session_activity.py asserts against
ACTIVITY_DESCRIPTION_MAX instead of the literal 120.
- The session-stall WARNING log line names its config knob
(agent.session_stall_timeout) so operators can find the setting.
- hermes_state.py: collapse the triple blank line near line 191.
- hermes_cli/status.py no longer imports the private
hermes_cli.main._relative_time: the helper moved to a public home
(hermes_cli.timefmt.relative_time); main._relative_time stays as a
thin back-compat wrapper (sessions_cmd and external patchers keep
working).
The fence-cancel poll loops (sync host wait in conversation_compression,
async hygiene wait in gateway/run) spun at 1kHz while the worker held
the fence through its lock-setup window — which rides SessionDB write
patience and can last seconds. 25ms keeps sub-tick cancel latency
without the spin.
A wedged adapter transport (network hang, dead websocket) previously
blocked _check_session_stalls forever: sibling candidates in the same
pass were never evaluated and the watcher stopped ticking. Wrap the
send in asyncio.wait_for (15s); on timeout log a WARNING and do NOT
latch, so the next tick retries. Regression uses a never-resolving fake
adapter and proves the pass completes, a healthy sibling candidate is
still notified in the same pass, and the watcher ticks again
(sabotage-verified against the unbounded send).
The stall watchdog gathered pending/activity candidates and later sent
the recovery notification from that aging snapshot — an agent that made
progress (or drained its queue) between the scan and the send received a
false stall notice mid-recovery.
Re-read the adapter pending slot, the overflow queue, and the live
activity snapshot immediately before delivery; abort the send and re-arm
the latch (pop it) when the candidate is no longer stale, so a future
genuine episode still notifies.
Race regressions: progress between scan and send aborts delivery;
pending-drained between scan and send aborts delivery; a genuinely
still-stale candidate is still delivered exactly once.
PR #76354 review, 'watchdog can send /new using a stale snapshot' /
merge gate 8.
Both progress-aware waits (sync compress wrapper and gateway session
hygiene) slept a FULL idle interval and only then compared progress, so
progress early in an interval let silence approach 2x the configured
idle timeout before the waiter noticed. Compute each wait slice as
idle_timeout - elapsed_since_last_progress instead.
Regression: a worker that reports progress early and then goes silent is
timed out in ~1x the idle budget, not ~2x.
PR #76354 review, 'idle timeout can allow nearly twice that silence'.
A host timeout previously left the timed-out worker holding the durable
per-session compression lock AND refreshing its lease indefinitely, so a
truly hung summary blocked every later compression attempt; and a LATE
successful summary could clear the failure cooldown the host had just
recorded.
Transplant the lease-cancellation invariants from PR #71569
(@ciabata-git): the worker publishes an idempotent, holder-scoped release
hook on the fence once it owns the durable lock (begin_lock_setup /
register_cancelled_lock_release close the acquire→publish race), the
refresher start is serialized against the release path, and the host
invokes the hook on idle timeout, hygiene timeout, and every unwind
(revoke_commit_admission now also releases). ABA safety: the SessionDB
release is holder-qualified (DELETE ... WHERE holder = ?), so a stale
release can never free a replacement holder's lease.
State ordering: the compressor consults a fence-cancellation check BEFORE
clearing the failure cooldown, so a late worker cannot undo the host's
timeout cooldown; the check is installed only for the fenced call and
removed in a finally.
Regression implements the reviewer's exact 5-step scenario: summary
blocked indefinitely → host timeout → a NEW compressor acquires the
durable lock while the old summary is STILL blocked → old worker released
→ it cannot clear cooldown, release the new holder's lease, or publish
stale state.
PR #76354 review, blocking finding 4 / merge gates 4 + 5.
Co-authored-by: ciabata-git <ciabata-git@users.noreply.github.com>
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.
Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.
Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.
Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.
PR #76354 review, blocking finding 2 / merge gate 2.
begin_commit() retains the fence lock until finish_commit(), so a hung
SessionDB commit made try_cancel_before_commit() return None forever and
the host spun ahead of the overrun-warning loop — a genuinely hung commit
stayed unbounded AND silent. Add a lock-free phase marker (threading.Event
set inside begin_commit while the lock is held, readable without it) and
break the host spin on commit_in_flight so the bounded overrun loop — and
its WARNING + on_commit_overrun surfacing — is reachable WHILE the commit
is still blocked. Applies to both the sync compress wrapper and the
gateway session-hygiene wait.
Regression asserts the warning and callback fire while the event-gated
fake commit is still blocked; the test releases the worker only after
those assertions (addresses helix4u's released-before-asserting callout).
PR #76354 review, blocking finding 1 / merge gate 1.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
Builds on the adapter list_channels() hook (cherry-picked from #43545 by
@Guoen0):
- plugins/platforms/simplex: implement list_channels() — enumerates
contacts (/contacts) and groups (/groups) over the live daemon
WebSocket into the channel directory. Returns None when the WS is
down so the directory falls back to session discovery instead of
wiping known targets.
- hermes send --list: merge configured-but-undiscovered platforms into
the listing. Previously a platform configured only via env (e.g. a
fresh SimpleX setup used for outbound sends) was silently omitted,
leaving users guessing at platform names.
- format_directory_for_display(): accept an explicit platforms view and
render empty platforms with a targeting hint instead of hiding them.
- docs: simplex hermes-send section.
Reported by Fedpostoffice on Discord (simplex missing from
hermes send --list; guessed platform names simplex-chat/simplex-relay).
The inverse of the inbound webhook platform: hooks.outbound in
config.yaml lists HTTP targets + the plugin-hook events they subscribe
to (on_session_end, subagent_stop, post_tool_call, ...). Each firing
POSTs a JSON payload (same top-level shape as shell hooks' stdin wire)
signed GitHub-style with HMAC-SHA256 (X-Hermes-Signature-256).
Rides the existing hook bus — notify-only callbacks registered on the
plugin manager at the same CLI/gateway/main entry points as shell
hooks. Delivery is fire-and-forget via a bounded queue + single daemon
worker thread, so a dead endpoint can never stall a tool call. Bounded
retries (5xx/conn errors once; 4xx never). secret_env preferred over
inline secret. HERMES_SAFE_MODE skips registration. hermes hooks list
shows outbound targets with signed/UNSIGNED status.
Zero new model tools, zero new subsystems.
The Discord semantic thread-rename lane resolved the target thread from
`_relay_auto_thread_info`, which read a single-slot-per-parent-chat cache
(`adapter._auto_thread_by_chat[chat_id]`, populated from connector
SendResult feedback). When two auto-threads spawned from the SAME parent
channel, the second send overwrote the first's slot and the title turn's
read raced the write — so only the FIRST thread in a channel ever got its
semantic rename. Staging repro 2026-08-02: message A's thread renamed to
"A Hundred Word Sword Story", sibling message B's thread stayed stuck at
the raw first-words name.
The connector now stamps `prospective_thread_id` on the inbound (the anchor
message id, which is the id of the thread it will auto-create) — shipped for
per-thread session keying. Reuse it here: it is deterministic and
per-message, so it names the EXACT thread even when several auto-threads
share one channel. `_relay_auto_thread_info` returns it directly (with an
empty initial-name marker) and never consults the collision-prone per-chat
cache; the connector's own created-name guard (`prefer_connector_created`)
still enforces no-clobber, so no initial name is needed gateway-side. The
send-result cache path stays as a fallback for older connectors that don't
stamp the field.
Tests: two new cases in test_relay_threads.py — prospective id wins over a
poisoned cache entry, and two sibling threads in one channel each rename to
their own thread id. Full gateway session + relay suites green (211 passed).
- Guard the thread-id-as-chat_id normalization to Discord only; Slack
and Telegram adapters use parent_channel as chat_id for thread messages,
so the unconditional version broke their handoff keys.
- Apply the same Discord-specific guard to _seed_cron_thread_session in
cron/scheduler.py (sibling site with the same bug, docstring said
'Mirrors _process_handoff').
- Replace the change-detector test with contract tests that verify the
actual invariant: Discord handoff key == organic thread key, Slack
handoff key still uses parent channel (non-regression).
A CLI→Discord handoff creates a dedicated thread and re-binds the CLI
session to it. It built the destination SessionSource with
chat_id = home.chat_id (the PARENT channel) while marking it
chat_type="thread" with thread_id set.
But platform adapters build organic in-thread messages with
chat_id = <thread id> (see the Discord adapter's on_message and
_build_thread_event paths). build_session_key therefore produced two
different keys for the same thread:
handoff: agent:main:<platform>🧵{parent}:{thread}
organic: agent:main:<platform>🧵{thread}:{thread}
So the next real user reply in the handoff thread resolved to a
DIFFERENT session_key and spawned a fresh session instead of continuing
the handed-off one — observed as a stray auto-titled session plus a
session_search fallback (the new session had no prior context).
Fix: for a thread destination, key on the thread's own id so the
synthetic handoff turn and later user replies share one session_key,
matching how adapters key organic in-thread messages.
Adds tests/gateway/test_handoff_thread_session_key.py, which asserts the
handoff key is byte-identical to the organic in-thread key (fails on the
old parent-channel keying, passes on the fix).
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
_handle_cron_fire verified the NAS-minted fire JWT by calling the
fire-verifier inline on the event loop. That verifier resolves the NAS
signing key from a JWKS URL — a synchronous HTTP GET on a cache miss (a
cold PyJWKClient, or a rotated kid the cached client doesn't know) — so a
slow or rate-limited portal stalls the whole event loop and starves every
other adapter sharing it. #64641 already documented this exact symptom
(relay 504s on high-job-count instances) and cut the fetch frequency by
caching the client per URL, but the residual cache-miss fetch still ran
inline on the loop.
Dispatch the verifier the same way the platform HTTP event verifier was
hardened: await a coroutine verifier directly, run a sync one via
asyncio.to_thread so its blocking I/O stays off the loop, and fail closed
(reject with 401, never admit the fire) if the verifier raises — this is
the only inbound that can trigger remote job execution. The verifier's
JWK-client cache is already thread-safe (threading.Lock), so moving the
call to a worker thread is safe.
Adds regression tests: a sync verifier runs on a worker thread rather
than the loop thread, a crashing verifier yields 401 with no fire, and a
coroutine verifier is awaited.
Simplify-pass follow-up on the #66355 salvage:
1. _config_settings runs on EVERY trim attempt (before the cooldown
check) and only reads — swap load_config for load_config_readonly.
Deep-copying the whole config per attempt generates exactly the
allocator garbage this module exists to release. Tests re-seamed.
2. Trim-failure logs demoted warning->debug at all 3 periodic sites
(gateway housekeeping, idle reaper, slash worker): sibling failure
branches in the same loops log at debug, and a persistent failure
(e.g. broken import after a partial update) would otherwise warn
every 60s forever.
3. The frame-inspection test now asserts the expected locals exist
before reading them — a rename in _run_prompt_submit fails the test
loudly instead of vacuously passing on None.
Add config-driven glibc malloc_trim for long-lived Hermes processes:
- hermes_cli/mem_trim.py: trim_memory() with configurable cooldown,
RSS snapshot telemetry, and forced-trim INFO logging
- gateway/run.py: periodic trim in gateway housekeeping loop
- tui_gateway/server.py: trim in idle reaper (~every 5 min)
- tui_gateway/slash_worker.py: trim on turn boundary
- run_agent.py: force trim on agent close
- hermes_cli/config.py: context.memory_trim config section
(enabled, cooldown_seconds, log_every_n, info_log_min_delta_mb)
CSA tier-4 reviewed (4 rounds, 0 HIGH/MEDIUM/CRITICAL remaining).
Supersedes PR #63708 + #64591 with enhanced telemetry and gateway/slash_worker coverage.
Review follow-up on the #64169 salvage: _save_entry duplicated
_snapshot_routing_locked's counter-bump line verbatim. The stale-write
protection is a total order over ONE counter — extract
_next_routing_generation_locked() so the two allocation sites can't
drift apart silently.
The steady-state turn only bumps updated_at/last_prompt_tokens on one
routing entry, but persisted it through the full index rewrite twice
per turn (get_or_create_session's healthy-path bump + update_session):
every entry re-serialized, DELETE+INSERT of every gateway_routing row,
and a multi-MB sessions.json dump+fsync — ~50ms p50 at ~1,100 routing
keys in production, out of ~175ms total per-turn gateway persistence.
Metadata-only saves now UPSERT the single row via the existing
HermesDB.save_gateway_routing_entry (<1ms). Structural transitions
(create/recover/reset/switch/prune, compression-tip heals) keep the
full rewrite, which also refreshes the legacy sessions.json mirror.
Correctness: each fast save allocates a per-entry revision from the
routing generation counter under _lock, so fast and full snapshots are
totally ordered by number. Under _save_lock the UPSERT is skipped when
a newer full snapshot or a newer fast save of the same key has already
persisted, and a delayed full rewrite folds in fast records serialized
after its snapshot before writing — an older snapshot can never
overwrite a newer one, in either direction. update_session snapshots
peer fields under _lock so a concurrent reset cannot record a torn
peer row; no DB or a failed UPSERT falls back to the full rewrite so
DB-less installs keep sessions.json durable every turn.
- gateway/authz_mixin.py: group chat allowlists, {PLATFORM}_ALLOW_BOTS, and
pairing-mode allowlist presence checks now go through the file's own
_platform_gate_env helper (scoped-authoritative under multiplex).
- gateway/pairing.py: grant-mirror/revoke allowlist READS go through
get_secret (Slack pattern for unscoped admin/CLI callers); writes still
use save_env_value with a TODO for profile-aware writes.
Review follow-up: the adapter-level resolver alone left three paths
reading per-profile QQ_* values from raw os.getenv, so a secondary
multiplex profile's scoped opt-in or credentials were ignored (or the
primary's environ values leaked in):
- gateway/authz_mixin.py: route the per-platform allow-all flag and the
per-platform/group allowlist + allow-bots reads through the
scope-aware gateway.config._getenv. Deployment-global GATEWAY_* reads
intentionally stay on os.getenv. This makes the same fix effective
for every own-policy platform, not just QQ; unscoped behavior is
byte-identical to os.getenv.
- gateway/run.py (_own_policy_open_startup_violation): resolve the
per-platform dm/group policy and allow-all opt-in via _getenv; the
secondary-profile caller already runs inside _profile_runtime_scope.
- tools/send_message_tool.py (_send_qqbot): the QQ_APP_ID /
QQ_CLIENT_SECRET fallbacks now honor the active profile scope.
Tests: tests/gateway/test_qqbot_scope_paths.py covers all three paths
end-to-end (scope wins, no environ inheritance for non-opted profiles,
single-profile environ fallback unchanged); the STT suite now asserts
QQ_STT_BASE_URL and QQ_STT_MODEL scoping alongside the API key. All
five scoped-behavior tests fail on the previous commit and pass here.
zabih-sudo
kshitij
Dannyzen
Ben Barclay
314574126
kshitijk4poor
EloquentBrush
skywind
MaartenDMT
Kyzcreig
szzhoujiarui
ibaldr89
Coffee☕️
HexLab98
Teknium
fangliquanflq
Guoen
kshitij
Alejandro Moreno
AlexFucuson9
Frowtek
Ryder Freeman
Soju06
Da7-Tech