hermes-agent/gateway
kshitij aaf9688519 refactor(gateway): extract the hygiene recovery gate and forward the failure reason
Follow-up to c0d974b19 (#79741). Three review findings against that commit,
none of which change the escalation behaviour it shipped.

1. The recovery decision lived inline in `_handle_message_with_agent`, a
   ~2000-line async method, so the only way to pin it was a test that read
   `inspect.getsource(...)` and asserted on substrings. AGENTS.md bans reading
   source in tests outright and names this exact situation: "if the logic lives
   inline in a god-file (gateway/run.py) and extracting it feels disruptive:
   that's the actual signal to do the extraction, not to regex around it."

   Those tests were not merely stylistically wrong, they were actively harmful.
   One asserted the substring `_new_tokens < _approx_tokens` was PRESENT -- so
   it passed while the gate had the bug that substring represents, and had to be
   edited when the gate was fixed. It failed on correct code and passed on
   broken code, in one assertion.

   Extracted `hygiene_compaction_recovered()` as a module-level pure predicate
   and replaced the three source-reading tests with eight direct unit tests.
   The extraction immediately earned itself: the new tests caught a `NameError`
   (the predicate called `compression_made_progress` while the module bound it
   under an alias) that a source-text assertion cannot see, because the symbol
   is spelled correctly in the source and only fails at runtime.

2. The gate inferred "did the transcript actually get rewritten" from a numeric
   side effect -- the degenerate "did not rotate or compact in place" path
   (#21301) reuses the pre-compression counts -- when the booleans
   `_hyg_rotated` / `_hyg_in_place` were already in scope and explicitly set
   False on that path. The predicate now takes them directly, so a future edit
   that re-estimates instead of reusing the old counts cannot silently defeat
   the escalation.

3. `_record_hygiene_cooldown` passed no `error` to
   `record_compression_failure_cooldown`, which writes `compression_failure_error`
   unconditionally -- so a hygiene failure clobbered to NULL whatever reason the
   in-conversation path had recorded, and readers then show the user "unknown
   error" (agent/manual_compression_feedback.py, gateway/slash_commands.py). The
   reason was already in hand at both call sites. Pre-existing from #74136 but
   amplified by escalation: a blank reason on a 45-minute cooldown is far more
   user-visible than on a 5-minute one.

Also: the ladder docstring described the compressor's absolute 60/300/900s
ladder while the constant is multipliers (1, 3, 9); the config docs still
described `hygiene_failure_cooldown_seconds` as a flat interval rather than the
first rung of a capped ladder; and `PersistentState.hygiene_failure_streak` now
documents that it is process-local by design -- keying on `session_key` is what
survives compaction rotation, which the persisted `compression_*_streak`
columns cannot express since they key on the rotating `session_id`. Making it
durable is a schema change, tracked on #79624 rather than smuggled in here.

Also replaces the file's hand-written `_Runner` stub with
`object.__new__(GatewayRunner)` (already the idiom elsewhere in the same file).
The stub reimplemented `_session_state` and `_peek_session_state`, so the tests
exercised copies that could drift from production; using the real class
immediately made one assertion stronger -- on a fresh runner `_sessions` does not
exist at all until something materialises it, so the reset provably did not even
create the map.

A review pass on this follow-up then caught that the CALL SITE was still
unbound: deleting the whole `if not _hyg_aborted: if
hygiene_compaction_recovered(...)` block left every ladder test green, because
the unit tests prove the predicate correct without proving it is wired in. The
merged commit had the same gap and its only cover was the banned source-reading
test. `test_session_hygiene_forces_in_place_compaction_with_bound_session_db`
now spies the reset on a genuine in-place compaction, so deleting the wiring
fails. Two earlier attempts at this test did NOT close the gap -- asserting on
streak VALUES passes either way, since the streak is 0 whether or not the gate
ran; only a positive spy assertion on a recovering run detects the deletion.

Same pass also corrected an overstatement: point (2) is hardening, not a live
bug. The degenerate path also sets `_new_count = _msg_count` and `_new_tokens =
_approx_tokens`, and `compression_made_progress(n, n, t, t)` is always False, so
the merged code already declined to reset there. A 200k-trial fuzz over the
reachable state space found zero behavioural disagreements between the merged
gate and this one. The guard's value is surviving a future edit that stops
reusing those counts.

Tests: 28 in tests/gateway/test_hygiene_failure_cooldown_ladder.py (8 new unit
tests for the predicate, 3 for reason forwarding, 3 source-reading tests
deleted). All 5 mutations caught -- including one that restores the hand-rolled
comparison and one that removes the rotated/in_place guard. Two mutations
initially SURVIVED and exposed vacuous tests of my own: the no-rewrite test used
counts the progress predicate already rejects, so it passed without binding the
guard at all; it now passes counts that read as progress on their own, proving
the guard is what rejects them. gateway hygiene + session-state + agent
compression-progress suites: 54 passed; ruff clean.

Refs #79624
2026-08-06 15:04:53 +05:30
..
assets chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
builtin_hooks
…
platforms refactor(gateway): route session event stream through _sse_frame (ensure_ascii=False) 2026-08-04 13:07:44 +05:30
relay fix(gateway): bound go_dormant ws.close with teardown timeout 2026-08-04 11:17:11 +05:30
__init__.py
…
authz_mixin.py fix(secrets): route authz gate and pairing allowlist reads through the profile secret scope 2026-08-02 10:02:33 -07:00
cgroup_cleanup.py fix: satisfy ruff encoding + windows-footgun lints for cgroup reaper 2026-06-28 02:05:50 -07:00
channel_directory.py feat(gateway): simplex channel enumeration + show configured platforms in hermes send --list 2026-08-02 15:08:45 -07:00
code_skew.py fix(gateway): refuse model switch on stale checkout to avoid env_float ImportError 2026-06-24 04:16:54 +05:30
config.py chore: remove unused imports and dead locals (ruff F401/F841 sweep) 2026-07-29 11:53:39 -07:00
cwd_placeholder.py fix(gateway): resolve terminal.cwd placeholders per backend and mount mode 2026-07-04 13:28:47 -07:00
dead_targets.py fix(gateway): add utf-8 encoding to dead target registry 2026-07-24 15:47:12 -07:00
delivery.py fix(gateway): pass encoding="utf-8" to read_text/write_text in update path (#37423) 2026-07-24 15:47:12 -07:00
delivery_ledger.py fix(gateway,tools,agent): close leaked SQLite connections in delivery, delegation, and verification ledgers 2026-07-24 15:55:08 -07:00
display_config.py fix(slack): quiet Slack display defaults — no heartbeat/busy-ack breadcrumbs in channels 2026-07-23 11:34:11 -07:00
drain_control.py feat(gateway): suppress home-channel shutdown broadcast on flagged drains (#54824) 2026-06-29 12:18:11 -07:00
hooks.py feat(hooks): expose thread_id and chat_type in agent:start/end context (#41672) 2026-06-07 19:16:36 -07:00
kanban_watchers.py fix: use lazy logging with %s formatting in logger calls 2026-08-02 23:17:40 +05:30
lifecycle_ledger.py fix(gateway): treat pid=None lifecycle sentinel as unknown ownership in mark_exited 2026-07-28 22:41:09 -07:00
memory_monitor.py
…
message_timestamps.py feat(gateway): inject stable human-readable message timestamps 2026-06-16 15:49:59 -07:00
mirror.py refactor: consolidate gateway session metadata into state.db (#58899) 2026-07-05 14:01:03 -07:00
pairing.py fix(secrets): route authz gate and pairing allowlist reads through the profile secret scope 2026-08-02 10:02:33 -07:00
platform_registry.py fix(teams): suppress SDK import-time dotenv instead of clearing environ 2026-08-01 15:34:19 -07:00
profile_routing.py fix(gateway): profile routing — conjunctive matching + universal gateway_runner 2026-07-15 09:50:05 -07:00
readiness.py fix(gateway): close readiness-probe SQLite connection deterministically 2026-07-24 15:55:08 -07:00
response_filters.py refactor(gateway): promote autonomous silence matcher to shared response_filters helper 2026-07-26 16:55:42 -07:00
restart.py fix(gateway): defer in-band restart until active turns finish (#77184) 2026-08-03 09:57:58 +05:30
restart_loop_guard.py fix(gateway,cron): guard cron model-tool path + add auto-resume loop breaker (#30719) 2026-07-01 02:48:36 -07:00
rich_sent_store.py style(profile): frame comments around what the code does 2026-06-30 15:30:06 -07:00
run.py refactor(gateway): extract the hygiene recovery gate and forward the failure reason 2026-08-06 15:04:53 +05:30
runtime_footer.py feat(gateway): add opt-in 'latency' runtime footer field 2026-08-03 17:16:57 +05:30
scale_to_zero.py feat(gateway): scale-to-zero idle detection + dormant-quiesce (Phase 0) 2026-06-24 18:47:18 -07:00
session.py perf: replace COUNT(*) with LIMIT-based existence checks 2026-08-03 17:31:29 +05:30
session_context.py fix(cron): scope cron approval context per session 2026-08-03 00:25:20 +05:30
session_stall.py feat(gateway): session activity watchdog, stall notify, compress timeout (#72424) 2026-08-02 16:16:36 -07:00
session_state.py refactor(gateway): extract the hygiene recovery gate and forward the failure reason 2026-08-06 15:04:53 +05:30
shutdown_flush.py fix: consolidate agent-history preservation into shutdown_flush module 2026-07-29 16:45:35 +05:30
shutdown_forensics.py fix: add explicit UTF-8 encoding to all subprocess text=True calls (#53428) 2026-07-24 11:45:57 -07:00
shutdown_watchdog.py fix(gateway): detect and report unclean shutdowns via lifecycle ledger (NS-608) 2026-07-28 22:41:09 -07:00
slash_access.py
…
slash_commands.py feat: /refine — run the memory/skill self-improvement review on demand 2026-08-05 22:40:51 -07:00
status.py fix(windows): native Windows correctness for CLI, gateway status, banner, and WSL browser paths 2026-07-29 23:16:18 -07:00
status_phrases.py chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
sticker_cache.py
…
stream_consumer.py fix(gateway): don't claim deleted head chunks as delivered in the empty-fallback recovery 2026-08-06 03:13:32 +05:30
stream_dispatch.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
stream_events.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
streaming_tts_consumer.py feat(gateway): streaming TTS adapter contract and consumer (#60671) 2026-07-28 22:31:40 -07:00
systemd_notify.py fix(gateway): recover Discord websocket and event-loop stalls 2026-07-18 20:01:55 +05:30
turn_context.py fix(gateway): reap only the background processes an abandoned turn created 2026-08-02 14:23:55 +05:30
turn_lease.py fix(gateway): per-session turn lease + conversation-scope funnel (#64934) (#67401) 2026-07-19 03:49:29 -07:00
wake.py fix(gateway): deliver kanban/delegate wake-ups to api_server sessions 2026-07-23 11:55:17 -07:00
whatsapp_identity.py fix(whatsapp): resolve LID aliases on modern platforms/ session layout 2026-06-28 02:05:26 -07:00