hermes-agent/gateway
joaomarcos bcdfdd51e5 fix(gateway): make the restart-loop breaker see slow crash cycles (#81642)
The auto-resume restart-loop breaker (#30719, defense-3) pruned its boot
log against an absolute `window_seconds` (default 60s). That prune is
period-sensitive: a crash cycle slower than the window drops its own
history on every boot, so the counter never leaves 1 and the breaker can
never trip, no matter how long the loop runs.

The cycle reported in #81642 is ~150s — a wedged event loop, the liveness
watchdog hard-exiting at ~90s, a supervisor respawn, and auto-resume
replaying the same session that wedges it again. Structurally invisible to
a 60s window: `gateway/restart_loop.json` kept a single timestamp across 15
kills in one morning. Because every cycle leaves a gateway that cannot
process SIGTERM, `hermes update` has no drainable gateway to stop, which is
the reported hang.

Chain boots on the inter-boot GAP instead of an absolute window: two boots
belong to the same loop when they are no more than `max_gap_seconds` apart
(default 300s, floored by `window_seconds` so widening the window never
makes the breaker less sensitive). The verdict becomes period-agnostic —
the original ~10s respawn loop still trips in 3 boots, and so does a 150s
one — while a boot after real quiet resets the chain, so occasional
operator restarts still never accumulate. The persisted chain is capped at
50 entries.

- gateway/restart_loop_guard.py: gap-chained pruning (`_chain_ending_at`),
  `DEFAULT_MAX_GAP_SECONDS`, `max_gap_seconds` kwarg on the three entry
  points, clock-step tolerance, bounded state file
- gateway/run.py: `_restart_loop_guard_config` reads and returns
  `max_gap_seconds`; the auto-resume call site passes it through
- hermes_cli/config_defaults.py: `gateway.restart_loop_guard.max_gap_seconds`

Tests: 7 new cases in TestRestartLoopGuard covering the slow cycle, chain
persistence, quiet-period reset, the #30719 fast loop, the config knob, the
window floor, and the disabled breaker. Verified RED before the fix (the
slow-cycle case asserted `[1300.0] == [1000.0, 1150.0, 1300.0]`, exactly
the single-timestamp state file from the report) and GREEN after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 21:40:23 +05:30
..
assets chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
builtin_hooks
…
platforms fix(api-server): resolve reasoning for the request's model, not model.default 2026-08-08 11:13:27 -07:00
relay fix(gateway): rename a Discord thread once, after the reply lands 2026-08-09 04:33:58 -05:00
__init__.py
…
agent_cache_pressure.py fix: follow-ups for salvaged PR #80795 2026-08-07 21:02:33 +05:30
authz_mixin.py fix(secrets): route authz gate and pairing allowlist reads through the profile secret scope 2026-08-02 10:02:33 -07:00
cgroup_cleanup.py fix: satisfy ruff encoding + windows-footgun lints for cgroup reaper 2026-06-28 02:05:50 -07:00
channel_directory.py perf(slack): parallelize conversations.info lookups with asyncio.gather 2026-08-07 18:40:33 +05:30
code_skew.py fix(gateway): refuse model switch on stale checkout to avoid env_float ImportError 2026-06-24 04:16:54 +05:30
config.py fix: address self-review findings on the check_fn/ensure_deps_fn split 2026-08-07 13:28:43 +05:30
cwd_placeholder.py fix(gateway): resolve terminal.cwd placeholders per backend and mount mode 2026-07-04 13:28:47 -07:00
dead_targets.py fix(gateway): add utf-8 encoding to dead target registry 2026-07-24 15:47:12 -07:00
delivery.py fix(gateway): pass encoding="utf-8" to read_text/write_text in update path (#37423) 2026-07-24 15:47:12 -07:00
delivery_ledger.py fix(gateway,tools,agent): close leaked SQLite connections in delivery, delegation, and verification ledgers 2026-07-24 15:55:08 -07:00
display_config.py fix(slack): quiet Slack display defaults — no heartbeat/busy-ack breadcrumbs in channels 2026-07-23 11:34:11 -07:00
drain_control.py feat(gateway): suppress home-channel shutdown broadcast on flagged drains (#54824) 2026-06-29 12:18:11 -07:00
hooks.py fix: handle replacement transforms after CLI streaming 2026-08-08 14:06:30 +05:30
kanban_watchers.py feat(kanban,mcp): orphaned-card reconciliation + per-server MCP identity header 2026-08-07 08:58:20 -07:00
lifecycle_ledger.py fix(gateway): treat pid=None lifecycle sentinel as unknown ownership in mark_exited 2026-07-28 22:41:09 -07:00
memory_monitor.py Port from cline/cline#10343: periodic gateway memory logging (#27102) 2026-05-16 12:55:23 -07:00
message_timestamps.py feat(gateway): inject stable human-readable message timestamps 2026-06-16 15:49:59 -07:00
mirror.py refactor: consolidate gateway session metadata into state.db (#58899) 2026-07-05 14:01:03 -07:00
pairing.py fix(secrets): route authz gate and pairing allowlist reads through the profile secret scope 2026-08-02 10:02:33 -07:00
platform_registry.py fix(gateway): split check_fn (passive probe) from ensure_deps_fn (active installer) 2026-08-07 13:28:43 +05:30
profile_routing.py fix(gateway): profile routing — conjunctive matching + universal gateway_runner 2026-07-15 09:50:05 -07:00
readiness.py fix(gateway): close readiness-probe SQLite connection deterministically 2026-07-24 15:55:08 -07:00
response_filters.py refactor(gateway): promote autonomous silence matcher to shared response_filters helper 2026-07-26 16:55:42 -07:00
restart.py fix(gateway): defer in-band restart until active turns finish (#77184) 2026-08-03 09:57:58 +05:30
restart_loop_guard.py fix(gateway): make the restart-loop breaker see slow crash cycles (#81642) 2026-08-09 21:40:23 +05:30
rich_sent_store.py style(profile): frame comments around what the code does 2026-06-30 15:30:06 -07:00
run.py fix(gateway): make the restart-loop breaker see slow crash cycles (#81642) 2026-08-09 21:40:23 +05:30
runtime_footer.py feat(gateway): add opt-in 'latency' runtime footer field 2026-08-03 17:16:57 +05:30
scale_to_zero.py feat(gateway): scale-to-zero idle detection + dormant-quiesce (Phase 0) 2026-06-24 18:47:18 -07:00
session.py fix: defer O(n) fallback_data construction to failure path in _save_entry 2026-08-08 01:12:16 +05:30
session_context.py fix(cron): scope cron approval context per session 2026-08-03 00:25:20 +05:30
session_stall.py feat(gateway): session activity watchdog, stall notify, compress timeout (#72424) 2026-08-02 16:16:36 -07:00
session_state.py refactor(gateway): extract the hygiene recovery gate and forward the failure reason 2026-08-06 15:04:53 +05:30
shutdown_flush.py fix: consolidate agent-history preservation into shutdown_flush module 2026-07-29 16:45:35 +05:30
shutdown_forensics.py fix: add explicit UTF-8 encoding to all subprocess text=True calls (#53428) 2026-07-24 11:45:57 -07:00
shutdown_watchdog.py fix(gateway): detect and report unclean shutdowns via lifecycle ledger (NS-608) 2026-07-28 22:41:09 -07:00
slash_access.py
…
slash_commands.py fix: finish the missing-encoding sweep — BOM-tolerant reads for user-edited stores 2026-08-08 12:32:23 -07:00
status.py fix(gateway): self-reacquire scoped lock by PID alone 2026-08-08 13:50:08 +05:30
status_phrases.py chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
sticker_cache.py fix: guard yaml.safe_load, flock unlock, TOCTOU races, and atomic writes 2026-05-19 00:12:41 -07:00
stream_consumer.py fix(gateway): cover all finalized stream sends 2026-08-08 16:59:42 -07:00
stream_dispatch.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
stream_events.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
streaming_tts_consumer.py feat(gateway): streaming TTS adapter contract and consumer (#60671) 2026-07-28 22:31:40 -07:00
systemd_notify.py fix(gateway): recover Discord websocket and event-loop stalls 2026-07-18 20:01:55 +05:30
turn_context.py fix(gateway): reap only the background processes an abandoned turn created 2026-08-02 14:23:55 +05:30
turn_lease.py refactor(gateway): drop dead degraded token field; de-churn salvage diff 2026-08-07 18:23:57 +05:30
wake.py fix(gateway): deliver kanban/delegate wake-ups to api_server sessions 2026-07-23 11:55:17 -07:00
whatsapp_identity.py fix(whatsapp): resolve LID aliases on modern platforms/ session layout 2026-06-28 02:05:26 -07:00