Follow-up to c0d974b19 (#79741). Three review findings against that commit,
none of which change the escalation behaviour it shipped.
1. The recovery decision lived inline in `_handle_message_with_agent`, a
~2000-line async method, so the only way to pin it was a test that read
`inspect.getsource(...)` and asserted on substrings. AGENTS.md bans reading
source in tests outright and names this exact situation: "if the logic lives
inline in a god-file (gateway/run.py) and extracting it feels disruptive:
that's the actual signal to do the extraction, not to regex around it."
Those tests were not merely stylistically wrong, they were actively harmful.
One asserted the substring `_new_tokens < _approx_tokens` was PRESENT -- so
it passed while the gate had the bug that substring represents, and had to be
edited when the gate was fixed. It failed on correct code and passed on
broken code, in one assertion.
Extracted `hygiene_compaction_recovered()` as a module-level pure predicate
and replaced the three source-reading tests with eight direct unit tests.
The extraction immediately earned itself: the new tests caught a `NameError`
(the predicate called `compression_made_progress` while the module bound it
under an alias) that a source-text assertion cannot see, because the symbol
is spelled correctly in the source and only fails at runtime.
2. The gate inferred "did the transcript actually get rewritten" from a numeric
side effect -- the degenerate "did not rotate or compact in place" path
(#21301) reuses the pre-compression counts -- when the booleans
`_hyg_rotated` / `_hyg_in_place` were already in scope and explicitly set
False on that path. The predicate now takes them directly, so a future edit
that re-estimates instead of reusing the old counts cannot silently defeat
the escalation.
3. `_record_hygiene_cooldown` passed no `error` to
`record_compression_failure_cooldown`, which writes `compression_failure_error`
unconditionally -- so a hygiene failure clobbered to NULL whatever reason the
in-conversation path had recorded, and readers then show the user "unknown
error" (agent/manual_compression_feedback.py, gateway/slash_commands.py). The
reason was already in hand at both call sites. Pre-existing from #74136 but
amplified by escalation: a blank reason on a 45-minute cooldown is far more
user-visible than on a 5-minute one.
Also: the ladder docstring described the compressor's absolute 60/300/900s
ladder while the constant is multipliers (1, 3, 9); the config docs still
described `hygiene_failure_cooldown_seconds` as a flat interval rather than the
first rung of a capped ladder; and `PersistentState.hygiene_failure_streak` now
documents that it is process-local by design -- keying on `session_key` is what
survives compaction rotation, which the persisted `compression_*_streak`
columns cannot express since they key on the rotating `session_id`. Making it
durable is a schema change, tracked on #79624 rather than smuggled in here.
Also replaces the file's hand-written `_Runner` stub with
`object.__new__(GatewayRunner)` (already the idiom elsewhere in the same file).
The stub reimplemented `_session_state` and `_peek_session_state`, so the tests
exercised copies that could drift from production; using the real class
immediately made one assertion stronger -- on a fresh runner `_sessions` does not
exist at all until something materialises it, so the reset provably did not even
create the map.
A review pass on this follow-up then caught that the CALL SITE was still
unbound: deleting the whole `if not _hyg_aborted: if
hygiene_compaction_recovered(...)` block left every ladder test green, because
the unit tests prove the predicate correct without proving it is wired in. The
merged commit had the same gap and its only cover was the banned source-reading
test. `test_session_hygiene_forces_in_place_compaction_with_bound_session_db`
now spies the reset on a genuine in-place compaction, so deleting the wiring
fails. Two earlier attempts at this test did NOT close the gap -- asserting on
streak VALUES passes either way, since the streak is 0 whether or not the gate
ran; only a positive spy assertion on a recovering run detects the deletion.
Same pass also corrected an overstatement: point (2) is hardening, not a live
bug. The degenerate path also sets `_new_count = _msg_count` and `_new_tokens =
_approx_tokens`, and `compression_made_progress(n, n, t, t)` is always False, so
the merged code already declined to reset there. A 200k-trial fuzz over the
reachable state space found zero behavioural disagreements between the merged
gate and this one. The guard's value is surviving a future edit that stops
reusing those counts.
Tests: 28 in tests/gateway/test_hygiene_failure_cooldown_ladder.py (8 new unit
tests for the predicate, 3 for reason forwarding, 3 source-reading tests
deleted). All 5 mutations caught -- including one that restores the hand-rolled
comparison and one that removes the rotated/in_place guard. Two mutations
initially SURVIVED and exposed vacuous tests of my own: the no-rewrite test used
counts the progress predicate already rejects, so it passed without binding the
guard at all; it now passes counts that read as progress on their own, proving
the guard is what rejects them. gateway hygiene + session-state + agent
compression-progress suites: 54 passed; ruff clean.
Refs #79624
/refine [focus] fires the existing background review fork
(AIAgent._spawn_background_review) immediately instead of waiting for
the automatic 10-turn memory / 10-iteration skill nudge counters.
Optional focus instructions are appended to the review prompt so the
fork prioritizes what the user asked for (e.g. '/refine save the
deploy workflow as a skill').
- New optional focus parameter threaded through
_spawn_background_review -> spawn_background_review_thread.
Automatic post-turn reviews pass None and their prompts are
byte-identical to before.
- CLI handler snapshots conversation_history; gateway handler pulls
the idle session's cached AIAgent from _agent_cache (rejected while
the agent is running).
- Review runs in a daemon thread against the snapshot — live
conversation, message alternation, and prompt cache untouched.
- Slack stays under the 50-slash cap via /hermes refine.
Adapted from the /refine concept in Prime Intellect's Prime-Agent
(Continual Harness); Hermes' equivalent durable state is the
memory + skill stores, so the review fork is the natural target.
/heartbeat every <interval> <prompt> gives the current session one
recurring instruction. When the session is idle and the interval has
elapsed, the prompt is injected as a plain user turn — same
conversation, same context, prompt cache and role alternation
untouched.
- CLI: idle-poll watchdog thread (wake-word watchdog pattern) feeding
_pending_input; gateway: single gateway-wide async poller injecting
through the adapter FIFO. Busy sessions coalesce their tick to the
next idle poll.
- Missed ticks coalesce (anchor resets on fire) — a busy hour yields
ONE heartbeat turn, never a backlog. Real user messages always win.
- 60s interval floor; injected prompt carries a don't-invent-work
guard so idle heartbeats don't generate busywork.
- State persists in SessionDB.state_meta (heartbeat:<session_id>),
survives /resume, migrates across compression session rotations
alongside /goal state.
- Session-scoped and in-process by design — durable cross-process
schedules remain the cron subsystem's job (docs draw the boundary).
- Slack stays under the 50-slash cap via /hermes heartbeat; ghost-text
suggester now prefers the shortest prefix match so /he still
suggests /help.
Adapted from the session-heartbeat concept in Prime Intellect's
Prime-Agent (/heartbeat).
/goal gate add <command> attaches shell commands to the active goal.
Gates run at turn boundary BEFORE the LLM judge: a failing gate skips
the judge entirely and feeds its exit code + bounded output tail back
as the continuation prompt, so the agent iterates against concrete
evidence instead of a prose verdict.
- Unchanged-workspace skip: a gate that failed on an identical
workspace (git HEAD + status fingerprint) is not re-run — the
recorded failure replays and the attempt count advances.
- Bounded retries (default 3) + per-gate timeout (default 300s);
exhaustion auto-pauses the goal like the turn budget does.
- Gates persist in SessionDB.state_meta with the goal (survive
/resume and compression rotation); pre-gate goal rows load
unchanged.
- /goal gate [list|add|remove|clear] on CLI + gateway; 'gate' added
to the mid-run control-verb whitelist (gates only run at turn
boundary, so editing the list mid-run is safe).
Adapted from the quality-gate concept in Prime Intellect's Prime-Agent
(--autonomous-gate).
A gateway session whose summary model keeps timing out no longer retries
compaction on the same fixed interval forever.
The in-agent compressor already escalates repeat summary timeouts
60 -> 300 -> 900s (ContextCompressor.record_timeout_failure), but that ladder
reads the in-memory _consecutive_timeout_failures counter and
bind_session_state() zeroes it (context_compressor.py:1645). Session hygiene
constructs a FRESH AIAgent for every run (gateway/run.py:16820) and re-binds
state each time, so from the gateway that streak is structurally always 0 --
only the flat hygiene_failure_cooldown_seconds (300s) could ever be recorded.
Issue #79624 reported exactly that steady state: an oversized session
(1053 messages, ~119.5k tokens) whose aux model always timed out, re-attempting
compaction every 300s across five days until the reporter deleted the session
by hand.
Track the streak on PersistentState instead, which outlives the per-run agent
and is not cleared by turn/boundary resets, so consecutive hygiene failures
climb 300 -> 900 -> 2700s and then saturate. Both failure sites (progress
timeout and aborted compression) feed it; a real compression resets it, so a
session that recovers starts from the first rung again. The ladder multiplies
the configured base, so operators who tuned
hygiene_failure_cooldown_seconds keep their first rung. Per-session, so one
wedged chat cannot penalize other conversations.
Deliberately NOT changed, since each is a maintainer policy call rather than a
defect (all three are written up on #79624):
- no durable failure-streak column, so escalation still resets on restart
- the gateway 30s / in-agent 120s / aux-client 300s-floor timeout mismatch
- no `hermes doctor` check or `hermes sessions list` marker for a session
stuck in a compression-failure cooldown
Note the reported exit(1) is NOT a crash: it is the deliberate
_signal_initiated_shutdown path (gateway/run.py:26746-26751, #5646) that lets
systemd Restart=on-failure revive the gateway after a bare SIGTERM, and it
fires on every `systemctl restart` independently of compaction. The compaction
log lines appear after the shutdown line because the gateway-owned executor is
torn down with shutdown(wait=False, cancel_futures=True) (run.py:21164), so an
in-flight turn keeps logging during teardown. Full analysis on the issue.
Post-review hardening (Phase 2c + /simplify-code found five real defects in the
first cut):
- the recovery gate hand-rolled `_new_tokens < _approx_tokens` when a canonical
predicate already existed: `compression_made_progress` (agent/turn_context.py,
#39548). They disagree on 3 of 5 cases -- the hand-rolled form misses a
row-count win when the summary keeps the token estimate flat, misses one
where the summary is slightly MORE verbose (so a genuinely recovered session
would keep escalating forever), and counts a sub-5% wobble as recovery. Now
reuses the shared predicate, promoted from `_compression_made_progress` to a
public name with the old private name kept as a back-compat alias so the
existing importer (tests/agent/test_protected_tail_pressure_61932.py) and any
patcher of that symbol keep working.
- the reset was gated on "not aborted", but the degenerate "did not rotate or
compact in place" branch (#21301) is NOT aborted and yields zero reduction,
so a session wedged there reset its streak every run and could never
escalate -- silently defeating the fix. Now gated on real progress.
- no absolute ceiling: base * 9 reaches 9h at an operator base of 3600s,
indistinguishable from "compaction switched off". Added
_HYGIENE_COOLDOWN_MAX_SECONDS = 3600, mirroring the in-file
_RECONNECT_BACKOFF_CAP precedent.
- the reset used the get-or-create accessor to write a 0 that was already 0,
materialising a _sessions entry (never evicted). Now peeks.
- the abort verdict was probed twice, leaving the reset/record mutual
exclusion implicit; a future await between the probes would have broken it
silently. Computed once into _hyg_aborted.
Tests: 19 new in tests/gateway/test_hygiene_failure_cooldown_ladder.py --
ladder escalation, saturation, the absolute cap, per-session isolation,
reset-on-recovery, custom/zero base, PersistentState scoping (a mutation moving
the field to TurnState fails), degraded runners, the progress gate, the exact
progress-predicate semantics the gate depends on, and end-to-end that the
escalated value is what reaches the state DB. All 12 mutations caught, including
ones that restore the flat cooldown (the original bug), ungate the reset, swap
the canonical predicate back for the hand-rolled comparison, remove the cap, and
share the streak globally; the harness hard-errors when a mutation cannot be
applied, since a silently no-op mutation check is worse than none -- an earlier
version of it WAS silently no-opping after a refactor. The gate's contract test
slices by AST node span rather than a fixed character count, which had already
truncated once as the block grew. gateway hygiene + session-state + the three
touched agent compression suites: 50 passed; ruff clean.
E2E with real imports demonstrates the premise rather than asserting it:
bind_session_state zeroes the in-agent counter, and the recorded deadlines go
300 -> 900 -> 2700 -> 2700 -> 2700s where they were previously a flat 300s.
Reported by @yucezerey (#79624), whose state.db column dump and
"deleting the session fixed it" datapoint made the real mechanism findable.
The salvaged fix changed only the gateway's verdict: a payload-less
multi-message split stopped inheriting legacy trust. But six code paths set
_turn_split_delivery, and only one of them was taught to record a payload, so
the remaining five swapped the swallow for the opposite defect.
Fix the producers instead of only distrusting them at the boundary:
- _send_or_edit failed-final-edit branch: record the visible payload on split
turns too. It deliberately skipped recording, which now reads as a mismatch
and re-sends an answer already on screen -- reintroducing the duplicate
#45517 fixed (#36965 / #25349).
- _send_fallback_final (x2) and _send_empty_fallback_final: route through
_record_turn_final_payload instead of assigning _delivered_final_text
directly. On a split turn their final_text is only the trailing chunk, so a
fully delivered heads+tail reply recorded a tail-only payload and was
re-sent in full.
- _try_fresh_final: refuse the fresh-final route once a head chunk is sealed.
It replaces every tracked preview with one message, which only holds the
whole answer on a single-message turn. After a split it deleted the sealed
heads while sending just the tail, so the complete reply was still lost --
on Telegram, the default finalize route and the shape #78541 reports.
- Set _turn_split_delivery at seal time rather than after the tail send, so
the tail's own finalize sees the split state. The sibling overflow path
already did this; the divergence is what let fresh-final delete the heads.
- run.py stale-finalize reconciliation: skip the in-place edit on a split
delivery. message_id is only the LAST chunk there, so editing it with the
complete response repeated every sealed head's text inside the tail
message. Fall through to the normal final send.
Also drop a dead `or "".join(chunks)` fallback (all growth funnels through
_append_accumulated, so the ledger is never empty at that call site, and joined
chunks carry injected fence markers that could never match final_response), and
document that _record_turn_final_payload intentionally ignores its argument on
split turns.
Tests: four end-to-end cases driving the real overflow-split loop instead of
hand-setting private flags -- complete split still suppresses (no duplicate),
split missing a tail does not suppress, fresh-final keeps sealed heads, and a
flood-controlled final edit after a split stays suppressed. Each was
mutation-checked: reverting any individual fix turns its test red.
The pre-existing gateway-boundary test asserted the recovery *route* (the
reconcile edit) rather than the guarantee. Relaxed to the real contract: either
_run_agent puts the complete text on the wire, or it declines to claim delivery
so the caller's normal final send does.
Co-authored-by: HexLab98 <liruixinch@outlook.com>
Record an unsplit stream ledger for multi-message deliveries and refuse
legacy trust when split delivery left no payload, so Telegram group
sessions no longer suppress a complete reply after an early/partial
finalize (#78541).
When a Discord channel message initiates a relay auto-thread, the thread does
not exist at ingest (source.thread_id is None) — the connector creates it on
its FIRST send and auto-threads any outbound carrying the reply anchor. The
final reply carries that anchor, so it lands in the thread. But the
tool-progress / status bubbles (the "Searching the web for..." updates and the
streaming preamble) were sent with _progress_metadata=None and
_progress_reply_to=None: _resolve_progress_thread_id returns None for Discord
(only slack/mattermost get a synthetic thread), so the progress send had no
anchor and the connector posted it FLAT in the parent channel. Result: the
search-status updates leaked outside the thread while the answer threaded
(staging repro 2026-08-02).
The connector now stamps prospective_thread_id on the inbound (the anchor
message id == the id of the thread it will create). Reuse it: when a
relay-delivered Discord channel-initiate carries prospective_thread_id and has
no real thread yet, carry the reply anchor (event_message_id) on both the
progress metadata (reply_to_message_id) and the progress reply_to, so the
connector routes the progress bubble into the SAME auto-thread as the final
reply. Applied to both the tool-progress path (_progress_metadata /
_progress_reply_to) and the status/interim callback path
(_status_thread_metadata). Events already arriving in a real thread, DMs, and
non-relay sources are untouched (guarded on delivered_via_upstream_relay +
prospective_thread_id + not thread_id).
Tests: two new cases in test_run_progress_topics.py — a relay Discord
channel-initiate asserts every progress send carries the anchor (reply_to +
metadata.reply_to_message_id + non_conversational), and an event already in a
real thread asserts the synthetic-anchor path does NOT engage. Full gateway
progress + relay + session suites green (228 passed).
Salvage of #26860 (hunk 2, ported \u2014 the PR's base predates the current
gateway layout by ~11.9K commits). Messaging platforms can set
gateway.platforms.<key>.skip_context_files: true to skip the
filesystem-heavy context-file discovery (SOUL.md, AGENTS.md,
.cursorrules walks) during AIAgent construction \u2014 10-100x slower
stat()/walk costs on Windows made this a real per-turn tax. Soul
identity is still loaded (single small file), so the persona survives.
The flag participates in _agent_config_signature so toggling it
rebuilds the cached agent instead of silently reusing a prompt built
under the other setting (prompt-cache correctness).
The PR's hunk 1 (mtime-caching the per-turn dotenv reload) was dropped:
df51ad797 mtime-cached load_config/read_raw_config and c2eda92fd
removed the per-turn deepcopies, capturing most of that win; the
function has since gained a multiplex early-return and managed-scope
overlay that the original whole-function skip would have bypassed.
Re-derivation of PR #65645 onto current main: _build_gateway_agent_history
already runs inside a turn whose config was loaded once into
ctx.user_config; re-reading config from disk via _load_gateway_config()
per turn is redundant. Reuse the loaded turn config.
The runtime footer (`/footer`) shows what model ran and how full the context
is, but not how long the turn took. On a messaging platform there is no
progress bar and no shell timer — a turn that took 4 seconds and one that took
four minutes produce visually identical replies. Users comparing models,
providers, or reasoning levels have no at-a-glance signal for the one
dimension they most often care about, and "was that slow or did I imagine it?"
is unanswerable after the fact.
Adds a `latency` field to the existing footer machinery, rendering the
wall-clock duration of the agent run: `<1s`, `22s`, `1m05s`.
`gateway/run.py` measures with `time.monotonic()` immediately around the
`self._run_agent(...)` await in `_handle_message_with_agent` — the same
function that already builds the footer, so the value is the user-perceived
turn duration (monotonic, so it is immune to wall-clock/NTP adjustment).
`latency` is deliberately NOT in `_DEFAULT_FIELDS`. It is opt-in via
`display.runtime_footer.fields`. Every existing footer — and every footer a
user has today without touching config — renders byte-identically.
This is enforced by tests, not just asserted:
- `test_latency_not_in_default_fields` pins the default tuple.
- `test_resolve_footer_config_default_fields_exclude_latency` pins what
config resolution produces for an untouched config.
- `test_default_footer_renders_byte_identically` pins five exact output
strings for default-config renders **while supplying `turn_seconds`** —
proving that even when the caller measures timing, a default-configured
footer does not show it.
- `test_default_build_footer_line_ignores_turn_seconds` asserts
`build_footer_line(...) == build_footer_line(..., turn_seconds=125.0)`
under default fields.
Adding `latency` to `_DEFAULT_FIELDS` fails 11 of these tests.
No new config surface (reuses `display.runtime_footer.fields`), no new env
vars, no new core tool, no new model-facing schema. One new module-private
helper (`_format_latency`), one new keyword argument threaded through the two
existing footer functions, and 3 lines in `gateway/run.py`.
`turn_seconds` defaults to `None` and the field is skipped when it is `None`
or negative, so any call site that does not measure timing keeps working
unchanged.
`tests/gateway/test_runtime_footer.py` (+185): `_format_latency` boundary
table (sub-second, rounding at 59.4/59.6, the `m{:02d}s` zero-pad, 60m), the
render/skip/opt-in matrix, field-order placement, `build_footer_line`
threading, and the byte-stability block above.
RED-proved by mutation — each of these breaks tests:
- `latency` added to `_DEFAULT_FIELDS` → 11 failures
- dropping the `turn_seconds is not None and >= 0` guard → 2 failures
- `{sec:02d}` → `{sec}` → 6 failures
- `build_footer_line` not threading `turn_seconds` → 1 failure
51 passed in `tests/gateway/test_runtime_footer.py`; 54 passed across the
footer blast radius. `ruff check` clean.
CI slices failed the offload tests with 0.5s witness timeouts: on a
loaded shared runner the event loop thread can take >0.5s to get
scheduled even when NOT blocked, making the probe report a false
positive. A genuinely blocked loop can never set the progress event at
any timeout (the witness coroutine can't run at all), so 5s only
absorbs scheduler flake without weakening the invariant. Mutation
re-verified: reverting the offload still fails all 4 tests.
The sweep-path test parametrizes over _runner/_adapter, which live on
TestGatewayRedeliverySweep; main later added
TestUnconnectedPlatformKeepsItsBudget at the cherry-pick anchor point and
the test landed in that class, where the helpers don't exist
(AttributeError x2). Placement-only move.
request_restart was calling stop() immediately, so the requesting turn stayed
in the drain wait set and got force-killed at restart_drain_timeout. Wait for
active work to reach zero first, then stop against an idle gateway.
- tests/agent/test_session_activity.py asserts against
ACTIVITY_DESCRIPTION_MAX instead of the literal 120.
- The session-stall WARNING log line names its config knob
(agent.session_stall_timeout) so operators can find the setting.
- hermes_state.py: collapse the triple blank line near line 191.
- hermes_cli/status.py no longer imports the private
hermes_cli.main._relative_time: the helper moved to a public home
(hermes_cli.timefmt.relative_time); main._relative_time stays as a
thin back-compat wrapper (sessions_cmd and external patchers keep
working).
The fence-cancel poll loops (sync host wait in conversation_compression,
async hygiene wait in gateway/run) spun at 1kHz while the worker held
the fence through its lock-setup window — which rides SessionDB write
patience and can last seconds. 25ms keeps sub-tick cancel latency
without the spin.
A wedged adapter transport (network hang, dead websocket) previously
blocked _check_session_stalls forever: sibling candidates in the same
pass were never evaluated and the watcher stopped ticking. Wrap the
send in asyncio.wait_for (15s); on timeout log a WARNING and do NOT
latch, so the next tick retries. Regression uses a never-resolving fake
adapter and proves the pass completes, a healthy sibling candidate is
still notified in the same pass, and the watcher ticks again
(sabotage-verified against the unbounded send).
The stall watchdog gathered pending/activity candidates and later sent
the recovery notification from that aging snapshot — an agent that made
progress (or drained its queue) between the scan and the send received a
false stall notice mid-recovery.
Re-read the adapter pending slot, the overflow queue, and the live
activity snapshot immediately before delivery; abort the send and re-arm
the latch (pop it) when the candidate is no longer stale, so a future
genuine episode still notifies.
Race regressions: progress between scan and send aborts delivery;
pending-drained between scan and send aborts delivery; a genuinely
still-stale candidate is still delivered exactly once.
PR #76354 review, 'watchdog can send /new using a stale snapshot' /
merge gate 8.
Both progress-aware waits (sync compress wrapper and gateway session
hygiene) slept a FULL idle interval and only then compared progress, so
progress early in an interval let silence approach 2x the configured
idle timeout before the waiter noticed. Compute each wait slice as
idle_timeout - elapsed_since_last_progress instead.
Regression: a worker that reports progress early and then goes silent is
timed out in ~1x the idle budget, not ~2x.
PR #76354 review, 'idle timeout can allow nearly twice that silence'.
A host timeout previously left the timed-out worker holding the durable
per-session compression lock AND refreshing its lease indefinitely, so a
truly hung summary blocked every later compression attempt; and a LATE
successful summary could clear the failure cooldown the host had just
recorded.
Transplant the lease-cancellation invariants from PR #71569
(@ciabata-git): the worker publishes an idempotent, holder-scoped release
hook on the fence once it owns the durable lock (begin_lock_setup /
register_cancelled_lock_release close the acquire→publish race), the
refresher start is serialized against the release path, and the host
invokes the hook on idle timeout, hygiene timeout, and every unwind
(revoke_commit_admission now also releases). ABA safety: the SessionDB
release is holder-qualified (DELETE ... WHERE holder = ?), so a stale
release can never free a replacement holder's lease.
State ordering: the compressor consults a fence-cancellation check BEFORE
clearing the failure cooldown, so a late worker cannot undo the host's
timeout cooldown; the check is installed only for the fenced call and
removed in a finally.
Regression implements the reviewer's exact 5-step scenario: summary
blocked indefinitely → host timeout → a NEW compressor acquires the
durable lock while the old summary is STILL blocked → old worker released
→ it cannot clear cooldown, release the new holder's lease, or publish
stale state.
PR #76354 review, blocking finding 4 / merge gates 4 + 5.
Co-authored-by: ciabata-git <ciabata-git@users.noreply.github.com>
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.
Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.
Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.
Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.
PR #76354 review, blocking finding 2 / merge gate 2.
begin_commit() retains the fence lock until finish_commit(), so a hung
SessionDB commit made try_cancel_before_commit() return None forever and
the host spun ahead of the overrun-warning loop — a genuinely hung commit
stayed unbounded AND silent. Add a lock-free phase marker (threading.Event
set inside begin_commit while the lock is held, readable without it) and
break the host spin on commit_in_flight so the bounded overrun loop — and
its WARNING + on_commit_overrun surfacing — is reachable WHILE the commit
is still blocked. Applies to both the sync compress wrapper and the
gateway session-hygiene wait.
Regression asserts the warning and callback fire while the event-gated
fake commit is still blocked; the test releases the worker only after
those assertions (addresses helix4u's released-before-asserting callout).
PR #76354 review, blocking finding 1 / merge gate 1.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
The inverse of the inbound webhook platform: hooks.outbound in
config.yaml lists HTTP targets + the plugin-hook events they subscribe
to (on_session_end, subagent_stop, post_tool_call, ...). Each firing
POSTs a JSON payload (same top-level shape as shell hooks' stdin wire)
signed GitHub-style with HMAC-SHA256 (X-Hermes-Signature-256).
Rides the existing hook bus — notify-only callbacks registered on the
plugin manager at the same CLI/gateway/main entry points as shell
hooks. Delivery is fire-and-forget via a bounded queue + single daemon
worker thread, so a dead endpoint can never stall a tool call. Bounded
retries (5xx/conn errors once; 4xx never). secret_env preferred over
inline secret. HERMES_SAFE_MODE skips registration. hermes hooks list
shows outbound targets with signed/UNSIGNED status.
Zero new model tools, zero new subsystems.
The Discord semantic thread-rename lane resolved the target thread from
`_relay_auto_thread_info`, which read a single-slot-per-parent-chat cache
(`adapter._auto_thread_by_chat[chat_id]`, populated from connector
SendResult feedback). When two auto-threads spawned from the SAME parent
channel, the second send overwrote the first's slot and the title turn's
read raced the write — so only the FIRST thread in a channel ever got its
semantic rename. Staging repro 2026-08-02: message A's thread renamed to
"A Hundred Word Sword Story", sibling message B's thread stayed stuck at
the raw first-words name.
The connector now stamps `prospective_thread_id` on the inbound (the anchor
message id, which is the id of the thread it will auto-create) — shipped for
per-thread session keying. Reuse it here: it is deterministic and
per-message, so it names the EXACT thread even when several auto-threads
share one channel. `_relay_auto_thread_info` returns it directly (with an
empty initial-name marker) and never consults the collision-prone per-chat
cache; the connector's own created-name guard (`prefer_connector_created`)
still enforces no-clobber, so no initial name is needed gateway-side. The
send-result cache path stays as a fallback for older connectors that don't
stamp the field.
Tests: two new cases in test_relay_threads.py — prospective id wins over a
poisoned cache entry, and two sibling threads in one channel each rename to
their own thread id. Full gateway session + relay suites green (211 passed).
- Guard the thread-id-as-chat_id normalization to Discord only; Slack
and Telegram adapters use parent_channel as chat_id for thread messages,
so the unconditional version broke their handoff keys.
- Apply the same Discord-specific guard to _seed_cron_thread_session in
cron/scheduler.py (sibling site with the same bug, docstring said
'Mirrors _process_handoff').
- Replace the change-detector test with contract tests that verify the
actual invariant: Discord handoff key == organic thread key, Slack
handoff key still uses parent channel (non-regression).
A CLI→Discord handoff creates a dedicated thread and re-binds the CLI
session to it. It built the destination SessionSource with
chat_id = home.chat_id (the PARENT channel) while marking it
chat_type="thread" with thread_id set.
But platform adapters build organic in-thread messages with
chat_id = <thread id> (see the Discord adapter's on_message and
_build_thread_event paths). build_session_key therefore produced two
different keys for the same thread:
handoff: agent:main:<platform>🧵{parent}:{thread}
organic: agent:main:<platform>🧵{thread}:{thread}
So the next real user reply in the handoff thread resolved to a
DIFFERENT session_key and spawned a fresh session instead of continuing
the handed-off one — observed as a stray auto-titled session plus a
session_search fallback (the new session had no prior context).
Fix: for a thread destination, key on the thread's own id so the
synthetic handoff turn and later user replies share one session_key,
matching how adapters key organic in-thread messages.
Adds tests/gateway/test_handoff_thread_session_key.py, which asserts the
handoff key is byte-identical to the organic in-thread key (fails on the
old parent-channel keying, passes on the fix).
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
Simplify-pass follow-up on the #66355 salvage:
1. _config_settings runs on EVERY trim attempt (before the cooldown
check) and only reads — swap load_config for load_config_readonly.
Deep-copying the whole config per attempt generates exactly the
allocator garbage this module exists to release. Tests re-seamed.
2. Trim-failure logs demoted warning->debug at all 3 periodic sites
(gateway housekeeping, idle reaper, slash worker): sibling failure
branches in the same loops log at debug, and a persistent failure
(e.g. broken import after a partial update) would otherwise warn
every 60s forever.
3. The frame-inspection test now asserts the expected locals exist
before reading them — a rename in _run_prompt_submit fails the test
loudly instead of vacuously passing on None.
Add config-driven glibc malloc_trim for long-lived Hermes processes:
- hermes_cli/mem_trim.py: trim_memory() with configurable cooldown,
RSS snapshot telemetry, and forced-trim INFO logging
- gateway/run.py: periodic trim in gateway housekeeping loop
- tui_gateway/server.py: trim in idle reaper (~every 5 min)
- tui_gateway/slash_worker.py: trim on turn boundary
- run_agent.py: force trim on agent close
- hermes_cli/config.py: context.memory_trim config section
(enabled, cooldown_seconds, log_every_n, info_log_min_delta_mb)
CSA tier-4 reviewed (4 rounds, 0 HIGH/MEDIUM/CRITICAL remaining).
Supersedes PR #63708 + #64591 with enhanced telemetry and gateway/slash_worker coverage.
Review follow-up: the adapter-level resolver alone left three paths
reading per-profile QQ_* values from raw os.getenv, so a secondary
multiplex profile's scoped opt-in or credentials were ignored (or the
primary's environ values leaked in):
- gateway/authz_mixin.py: route the per-platform allow-all flag and the
per-platform/group allowlist + allow-bots reads through the
scope-aware gateway.config._getenv. Deployment-global GATEWAY_* reads
intentionally stay on os.getenv. This makes the same fix effective
for every own-policy platform, not just QQ; unscoped behavior is
byte-identical to os.getenv.
- gateway/run.py (_own_policy_open_startup_violation): resolve the
per-platform dm/group policy and allow-all opt-in via _getenv; the
secondary-profile caller already runs inside _profile_runtime_scope.
- tools/send_message_tool.py (_send_qqbot): the QQ_APP_ID /
QQ_CLIENT_SECRET fallbacks now honor the active profile scope.
Tests: tests/gateway/test_qqbot_scope_paths.py covers all three paths
end-to-end (scope wins, no environ inheritance for non-opted profiles,
single-profile environ fallback unchanged); the STT suite now asserts
QQ_STT_BASE_URL and QQ_STT_MODEL scoping alongside the API key. All
five scoped-behavior tests fail on the previous commit and pass here.
_abandon_timed_out_gateway_turn landed on main (eb4772ec2) after this
PR's base and still used the soft interrupt(). Every other inactivity-
timeout surface in this change (cron, gateway executor poll, delegate
children) treats a timeout as an explicit stop that may cancel a
protected compression summary — widen the same fix to this sibling.
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.
Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.
Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
Two follow-ups from review of the salvaged fix:
- _reap_gateway_turn_processes now returns 0 for a blank task_id.
ProcessSession.task_id defaults to empty for sessionless callers, so a
blank turn id would match (and kill) every unrelated empty-task process
instead of the turn's own.
- The asyncio poll loop checks executor completion BEFORE the watchdog's
timeout flag. When both race in the same window, the completed run has
already persisted its real reply to session history; surfacing the
'agent inactive' diagnostic would contradict the stored transcript.
This matches _abandon_timed_out_gateway_turn's own worker-done-wins
tiebreak.
Addresses the hermes-sweeper review on #76188:
1. task_id is session-scoped (task_id == session_id), not turn-scoped,
and the reap runs on a detached thread. A replacement turn could
claim the same session and spawn a legitimate process before the
previous turn's reaper thread actually enumerates its targets,
killing that new process by mistake.
Fixed by gating the reap on the existing run_generation mechanism
(_is_session_run_current) instead of inventing a new ownership
token: the timeout path captures its own run_generation at turn
start, the interrupt path captures the generation immediately after
invalidating it. If a newer turn has since claimed the session, the
reap is skipped — that newer turn owns its own baseline, so nothing
is left permanently unreaped.
2. gateway/platforms/api_server.py's SSE handlers for chat-completions
and the /api/sessions responses endpoint run their own agent
lifecycle via _run_agent() and never passed through TurnRunner, so
client-disconnect abandonment there had no baseline and no reap —
contradicting the PR's stated disconnect coverage. Both disconnect
handlers now snapshot/reap through the same
tools.process_registry primitives, via a small
_reap_disconnected_agent_processes() helper shared by both call
sites.
An agent turn can spawn a long-running background subprocess (e.g.
`next build`) and later be abandoned via inactivity timeout, /stop,
/new, or a client disconnect. Before this fix the gateway interrupted
the agent loop but never touched the subprocess: it kept running
inside the gateway's cgroup, unbounded, until memory pressure starved
the event loop and made every platform/cron look hung (#76115).
The process registry already knew how to kill a process tree — the
missing piece was per-turn ownership: nothing distinguished a process
that predates the turn (must survive), a process the turn started and
finished successfully (must survive), and a process an abandoned turn
left running (must be reaped).
- tools/process_registry.py: snapshot_running_ids() captures a turn's
starting baseline; kill_started_since() reaps only IDs created after
it, scoped to one task_id.
- gateway/turn_context.py: TurnContext carries process_task_id +
process_baseline so the timeout/interrupt paths can reach them.
- gateway/run.py: baseline is snapshotted right before the turn's
executor task starts; the inactivity-timeout path and the explicit
/stop|/new|disconnect interrupt path both reap via the same helper.
A daemon-thread watchdog backs up the asyncio-based timeout poll,
since a starved event loop is exactly the failure mode this bug
causes. The turn's own worker clears its ownership markers the
instant it finishes, closing a race where a /stop landing right
after normal completion could reap a background process the turn
deliberately left running.
Related but insufficient on their own: #37454 (cgroup ExecStopPost
reaper only fires on service restart) and #68915 (orphaned-pipe
grandchild detection, a registry bug not a turn-lifecycle gap).
Neither ties process cleanup to turn abandonment.
Live staging (2026-08-01, on a fresh instance where title generation
finally succeeded): the rename lane fired end to end, but the connector
declined the op with "discord egress declined: target not routed to an
onboarded tenant". The trace logs added earlier pinpointed it:
discord auto-thread rename: thread=... lane=relay new_title='...'
relay thread_rename declined ...: target not routed to an onboarded tenant
discord auto-thread rename result: thread=... applied=False
Root cause: the connector's routedEgressGuard resolves the owning tenant
from the outbound metadata's scope_id (guild) or user_id (author). The
adapter builds those via _with_scope(chat_id), reading per-chat caches
keyed by the PARENT channel chat_id learned at inbound. The relay rename
lane called rename_thread WITHOUT parent_chat_id, so chat_id defaulted to
the THREAD id — a key the caches never held — and the op shipped with no
discriminator. resolveTenant returned undefined and egress was declined
before the op ever reached the (now-durable) no-clobber guard.
This was the true terminal blocker: every earlier fix (send-result
feedback, registration/poll ordering, connector-owned guard, durable
Redis store) was correct but sat DOWNSTREAM of this egress-routing
decline, so none of them could take effect.
Fix: the relay lane passes parent_chat_id=source.chat_id (the relay
source's chat_id IS the parent channel; the thread came from send-result
feedback). _with_scope then resolves scope_id/user_id from the
parent-channel caches and the connector routes the op to the tenant.
Scoped to the relay lane only (use_connector_guard); the native lane
renames via the direct Discord API and needs no discriminator.
Tests: adapter-level — a rename passing parent_chat_id carries the cached
scope_id, one keyed on the thread id alone does not (the regression
shape); lane-level — the late-feedback test now asserts parent_chat_id
flows through as the parent channel. Relay suite 150 passed; ruff +
footguns clean.
Connector-compatible with the deployed egress guard; no gateway-gateway
change needed.
A successful finalize edit can carry only the last streamed preview
snapshot: deltas generated between the last preview edit and stream
completion never reach any Bot API call, yet final_response_sent /
final_content_delivered were set from the call's success and suppressed
the gateway's normal final send — losing the tail permanently.
The stream consumer now records the exact cleaned payload of every
turn-final delivery (delivered_final_matches tri-state), and gateway/run.py
reconciles that record against the completed final_response before
trusting either suppression flag. On a demonstrable mismatch it edits the
streamed message up to the complete response, falling back to the normal
final send if the edit fails. Multi-message split deliveries and legacy
paths without a record keep the existing flag-trusting behavior, so
overflow splits and the failed-finalize handling (#51828/#33793) are
untouched.
Fixes#71643
Live staging (2026-08-01): relay semantic thread rename still declined
silently despite both #74482 and #75581 deployed — thread kept its
initial-words name, session title generated fine. Root cause is the
no-clobber guard string mismatch (see paired gateway-gateway PR): the
gateway can't reproduce the thread's initial name byte-for-byte, so the
connector's only_if_current_name check always failed.
- relay rename lane now passes prefer_connector_created=True instead of
the fragile initial-name string; the connector resolves the guard from
its own created-name memory. Native-marker lane keeps the legacy
only_if_current_name string (source carries the real initial name).
- rename_thread: prefer_connector_created param -> only_if_connector_created
on the wire, precedence over the legacy string.
- INFO logs at rename dispatch (thread/lane/new_title) and result
(applied=bool): the whole failure hunt needed telemetry the gateway
never emitted — this makes the outcome visible in fly logs.
Tests: connector-guard wire shape + precedence over legacy string; the
title-turn race test updated to assert the connector-owned guard. Relay
suite 149 passed; ruff + footguns clean.
The session hygiene compression path tracked its per-session failure
cooldown in an in-memory dict (_hygiene_compression_failure_cooldowns).
When the gateway restarted, the dict was gone, so the next message
re-triggered the same failing compression, wedging session storage.
The state DB already has a persistent column
(sessions.compression_failure_cooldown_until) and full read/write/clear
methods (record_/get_/clear_compression_failure_cooldown in hermes_state.py)
used by the in-conversation compression path (context_compressor.py) but
not by the session hygiene path in gateway/run.py.
Fix: replace the in-memory dict with calls to the persistent DB methods:
- Cooldown check: use get_compression_failure_cooldown instead of dict lookup
- Timeout failure: use record_compression_failure_cooldown instead of dict write
- Abort failure: use record_compression_failure_cooldown instead of dict write
After a restart, a session whose compression is in cooldown is now skipped
for the cooldown's remaining duration rather than re-attempted immediately.
Fixes#74136
Two call sites in _busy_steer_command were assigning directly to
adapter._pending_messages[quick_key], which overwrites the FIFO head
when a message is already enqueued. Changed both to self._enqueue_fifo()
which preserves the pending slot and appends to the overflow tail.
Regression test verifies both the pending-sentinel and no-steer() paths.
Answering "Always Approve" on the /clear, /new, /reset and /undo
confirmation calls save_config_value("approvals.destructive_slash_confirm",
False) and then appended "Future /clear, /new, /reset, and /undo will run
without confirmation" unconditionally.
save_config_value catches its own exceptions and reports the outcome in the
return value, so the caller's try/except could never observe a failed write,
and the return value was ignored. On any install whose config.yaml is not
writable the user was told the preference stuck when it had not, and the
prompt returned on the next restart with no explanation.
Check the return value. The approved action still runs either way, but when
the write did not land, say so and point at the config key instead of
promising an opt-out that was never written.
kshitij
Teknium
HexLab98
Ben Barclay
314574126
MaartenDMT
Kyzcreig
kshitijk4poor
ibaldr89
Coffee☕️
fangliquanflq
kshitij
Alejandro Moreno
Ryder Freeman
Da7-Tech
Shaun Prince
Hermes agent
joaomarcos
Bao
tachyon-r
Shakti Prasad Mohapatra
Baophan00
MaxFreedomPollard