hermes-agent/agent
kshitij c0d974b19f fix(gateway): escalate the session-hygiene compaction cooldown on repeat failures
A gateway session whose summary model keeps timing out no longer retries
compaction on the same fixed interval forever.

The in-agent compressor already escalates repeat summary timeouts
60 -> 300 -> 900s (ContextCompressor.record_timeout_failure), but that ladder
reads the in-memory _consecutive_timeout_failures counter and
bind_session_state() zeroes it (context_compressor.py:1645). Session hygiene
constructs a FRESH AIAgent for every run (gateway/run.py:16820) and re-binds
state each time, so from the gateway that streak is structurally always 0 --
only the flat hygiene_failure_cooldown_seconds (300s) could ever be recorded.
Issue #79624 reported exactly that steady state: an oversized session
(1053 messages, ~119.5k tokens) whose aux model always timed out, re-attempting
compaction every 300s across five days until the reporter deleted the session
by hand.

Track the streak on PersistentState instead, which outlives the per-run agent
and is not cleared by turn/boundary resets, so consecutive hygiene failures
climb 300 -> 900 -> 2700s and then saturate. Both failure sites (progress
timeout and aborted compression) feed it; a real compression resets it, so a
session that recovers starts from the first rung again. The ladder multiplies
the configured base, so operators who tuned
hygiene_failure_cooldown_seconds keep their first rung. Per-session, so one
wedged chat cannot penalize other conversations.

Deliberately NOT changed, since each is a maintainer policy call rather than a
defect (all three are written up on #79624):
  - no durable failure-streak column, so escalation still resets on restart
  - the gateway 30s / in-agent 120s / aux-client 300s-floor timeout mismatch
  - no `hermes doctor` check or `hermes sessions list` marker for a session
    stuck in a compression-failure cooldown

Note the reported exit(1) is NOT a crash: it is the deliberate
_signal_initiated_shutdown path (gateway/run.py:26746-26751, #5646) that lets
systemd Restart=on-failure revive the gateway after a bare SIGTERM, and it
fires on every `systemctl restart` independently of compaction. The compaction
log lines appear after the shutdown line because the gateway-owned executor is
torn down with shutdown(wait=False, cancel_futures=True) (run.py:21164), so an
in-flight turn keeps logging during teardown. Full analysis on the issue.

Post-review hardening (Phase 2c + /simplify-code found five real defects in the
first cut):
  - the recovery gate hand-rolled `_new_tokens < _approx_tokens` when a canonical
    predicate already existed: `compression_made_progress` (agent/turn_context.py,
    #39548). They disagree on 3 of 5 cases -- the hand-rolled form misses a
    row-count win when the summary keeps the token estimate flat, misses one
    where the summary is slightly MORE verbose (so a genuinely recovered session
    would keep escalating forever), and counts a sub-5% wobble as recovery. Now
    reuses the shared predicate, promoted from `_compression_made_progress` to a
    public name with the old private name kept as a back-compat alias so the
    existing importer (tests/agent/test_protected_tail_pressure_61932.py) and any
    patcher of that symbol keep working.
  - the reset was gated on "not aborted", but the degenerate "did not rotate or
    compact in place" branch (#21301) is NOT aborted and yields zero reduction,
    so a session wedged there reset its streak every run and could never
    escalate -- silently defeating the fix. Now gated on real progress.
  - no absolute ceiling: base * 9 reaches 9h at an operator base of 3600s,
    indistinguishable from "compaction switched off". Added
    _HYGIENE_COOLDOWN_MAX_SECONDS = 3600, mirroring the in-file
    _RECONNECT_BACKOFF_CAP precedent.
  - the reset used the get-or-create accessor to write a 0 that was already 0,
    materialising a _sessions entry (never evicted). Now peeks.
  - the abort verdict was probed twice, leaving the reset/record mutual
    exclusion implicit; a future await between the probes would have broken it
    silently. Computed once into _hyg_aborted.

Tests: 19 new in tests/gateway/test_hygiene_failure_cooldown_ladder.py --
ladder escalation, saturation, the absolute cap, per-session isolation,
reset-on-recovery, custom/zero base, PersistentState scoping (a mutation moving
the field to TurnState fails), degraded runners, the progress gate, the exact
progress-predicate semantics the gate depends on, and end-to-end that the
escalated value is what reaches the state DB. All 12 mutations caught, including
ones that restore the flat cooldown (the original bug), ungate the reset, swap
the canonical predicate back for the hand-rolled comparison, remove the cap, and
share the streak globally; the harness hard-errors when a mutation cannot be
applied, since a silently no-op mutation check is worse than none -- an earlier
version of it WAS silently no-opping after a refactor. The gate's contract test
slices by AST node span rather than a fixed character count, which had already
truncated once as the block grew. gateway hygiene + session-state + the three
touched agent compression suites: 50 passed; ruff clean.

E2E with real imports demonstrates the premise rather than asserting it:
bind_session_state zeroes the in-agent counter, and the recorded deadlines go
300 -> 900 -> 2700 -> 2700 -> 2700s where they were previously a flat 300s.

Reported by @yucezerey (#79624), whose state.db column dump and
"deleting the session fixed it" datapoint made the real mechanism findable.
2026-08-06 05:00:48 +05:30
..
lsp fix(runtime): resolve Hermes-managed Node and uv before bare PATH 2026-08-01 21:17:51 -04:00
monitoring feat(monitoring): add hermes.gateway.background_delegations (unit/slot count) 2026-07-25 02:30:20 +00:00
pet fix: remove dead f-string prefixes via ruff F541 (216 sites) (#52336) 2026-07-05 13:42:46 -07:00
proxy_sources fix(windows): sweep remaining unguarded text-mode subprocess sites codebase-wide 2026-07-24 11:45:57 -07:00
secret_sources fix(gateway): hydrate cold profile secret sources 2026-08-02 00:11:50 -07:00
transports feat(providers): env-var metadata, config-driven local no-auth, reasoning-effort clamp for Actual 2026-08-05 14:08:32 -07:00
__init__.py
…
account_usage.py fix(auth): detect upstream Codex quota resets and lift stale pool cooldowns (#69494) 2026-07-22 10:58:22 -07:00
agent_init.py Hermes can read the in-app browser (#79482) 2026-08-05 16:35:00 +00:00
agent_runtime_helpers.py Hermes can read the in-app browser (#79482) 2026-08-05 16:35:00 +00:00
anthropic_adapter.py fix: prefer explicit anthropic api key 2026-08-04 00:12:45 +05:30
async_utils.py refactor(gateway): dedupe detached-task consumer + reconnect backoff policy 2026-07-18 20:01:55 +05:30
aux_accounting.py feat(analytics): record auxiliary model usage per task in session accounting (#65537) 2026-07-16 04:23:12 -07:00
auxiliary_client.py feat(providers): add Actual Computer inference provider 2026-08-05 14:08:32 -07:00
azure_identity_adapter.py fix(secrets): scope-aware Azure credential presence reads in identity diagnostics 2026-08-02 10:02:33 -07:00
backend_identity.py refactor(fallback): single owner for backend identity and failure-scoped skips 2026-07-26 23:41:54 -07:00
background_review.py fix(background-review): inherit prefill messages + OpenRouter provider pins on non-routed forks 2026-08-02 23:48:42 +05:30
battery.py feat(status-bar): add /battery toggle for a color-coded battery read-out 2026-07-21 13:54:11 -05:00
bedrock_adapter.py feat(bedrock): add Converse API prompt caching (cachePoint) 2026-07-23 11:45:07 -07:00
billing_links.py feat(billing): shared cross-surface out-of-credits signal 2026-07-22 18:08:59 -05:00
billing_usage.py chore: remove unused imports and dead locals (ruff F401/F841 sweep) 2026-07-29 11:53:39 -07:00
billing_view.py chore: remove unused imports and dead locals (ruff F401/F841 sweep) 2026-07-29 11:53:39 -07:00
bounded_response.py fix(agents): bound streaming error-response body reads 2026-07-05 14:00:20 -07:00
browser_provider.py fix(browser): replace expired cloud sessions 2026-08-02 11:18:41 +05:30
browser_registry.py
…
chat_completion_helpers.py feat(providers): env-var metadata, config-driven local no-auth, reasoning-effort clamp for Actual 2026-08-05 14:08:32 -07:00
codex_responses_adapter.py fix(codex): defang reserved Harmony tokens in requests 2026-07-31 22:53:20 -07:00
codex_runtime.py fix(codex): don't retry after a valid response.completed on drain error 2026-07-29 16:41:39 -03:00
coding_context.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00
context_breakdown.py feat(cli,gateway): unify /context into a visual context-usage breakdown 2026-07-26 18:06:21 -07:00
context_compressor.py fix(compression): durable-sync the prune runway on model switch + fast no-op for incapable stores 2026-08-06 02:22:08 +05:30
context_engine.py fix(context-engine): snapshot select_context read-only inputs; scope on_turn_complete coverage doc 2026-07-23 19:44:35 -07:00
context_references.py fix(context): keep @ref tokens inline instead of stripping them from the prompt 2026-07-27 22:55:18 -05:00
conversation_compression.py fix(compression): durable-sync the prune runway on model switch + fast no-op for incapable stores 2026-08-06 02:22:08 +05:30
conversation_loop.py fix(cache): make proactive pruning durable and cache-aware 2026-08-06 02:22:08 +05:30
copilot_acp_client.py fix(core,cli,gateway,plugins): add encoding='utf-8' to read_text() calls 2026-07-24 17:10:39 -07:00
credential_persistence.py feat(auth): make xAI Grok OAuth device-code-only, drop loopback login 2026-07-02 13:17:41 -07:00
credential_pool.py fix(credential-pool): bench a billing 403 fully, even as the sole key 2026-08-04 23:33:39 +05:30
credential_sources.py fix(windows): sweep remaining bare read_text/write_text sites + linter rule 2026-07-24 17:10:39 -07:00
credits_tracker.py fix(credits): reintroduce grant_spent, gated on an in-session crossing (#73634) 2026-07-28 21:36:23 +00:00
curator.py perf(curator): trim dead tool-schema from the LLM review fork 2026-08-03 18:21:17 +05:30
curator_backup.py fix(curator): restore the real skills tree when a rollback extract dies part-way 2026-07-31 22:34:57 -07:00
delegation_context.py fix(cron): don't let a cron job inherit a kanban worker's dispatcher identity 2026-08-06 03:16:26 +05:30
display.py refactor(discord): simplify tool preview links 2026-08-02 21:48:06 -07:00
error_classifier.py fix(models): a model id missing its vendor prefix says so instead of 404ing (#78856) 2026-08-04 19:35:57 +00:00
errors.py fix(moa): surface stale presets without retries 2026-07-17 13:49:12 -07:00
file_safety.py fix(secrets): harden encrypted Bitwarden cache 2026-07-22 04:40:07 -07:00
gemini_native_adapter.py fix(gemini): sweep hardcoded Gemini default models to gemini-3.6-flash (#32360) 2026-07-28 18:17:56 -07:00
gemini_schema.py fix(gemini): preserve typed enum constraints as strings 2026-07-26 20:59:16 -07:00
i18n.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00
image_gen_provider.py feat(image-gen): add image-to-image / editing to image_generate (#48705) 2026-06-18 22:13:07 -07:00
image_gen_registry.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00
image_routing.py feat(image_routing): accept vision alias for custom provider models 2026-07-23 08:32:09 -07:00
insights.py refactor(insights): drop consumer-less get_skill_breakdown alias (simplify-pass) 2026-08-03 18:45:43 +05:30
interrupt_compat.py fix(compression): let explicit interrupts cancel safely 2026-08-02 22:15:20 +05:30
iteration_budget.py feat: raise default tool-calling iteration limit from 90 to 500 2026-07-26 12:54:45 -07:00
jiter_preload.py
…
kanban_stop.py follow-up: integrate agent nudge + dispatcher retry docs and tests 2026-07-14 16:47:33 +05:30
learn_prompt.py fix(learn): honor requirements mixed with sources in /learn requests (#55956) 2026-06-30 16:56:01 -07:00
learning_graph.py fix(learning_graph): guard non-dict metadata so /journey can't crash 2026-07-01 16:25:48 -05:00
learning_graph_render.py chore: remove unused imports and dead locals (ruff F401/F841 sweep) 2026-07-29 11:53:39 -07:00
learning_mutations.py refactor(journey): route memory mutations through MemoryStore atomic I/O 2026-06-30 15:16:21 -05:00
lmstudio_reasoning.py fix(lmstudio): clamp max/ultra reasoning effort to LM Studio's ceiling 2026-07-16 07:57:51 -07:00
manual_compression_feedback.py fix(compress): classify unconfirmed lock-acquire failures and cover all manual-compress surfaces 2026-07-23 08:19:14 -07:00
markdown_tables.py
…
memory_manager.py fix(memory): honor disabled toolsets for provider tools 2026-07-24 13:00:53 +05:30
memory_provider.py fix(openviking): integrate reliability and configuration hardening 2026-08-03 20:35:47 +05:30
message_content.py fix(openviking): preserve structured sync attribution 2026-06-19 15:23:41 +08:00
message_sanitization.py refactor(agent): single-owner call_id + reasoning_content sanitization policies (wire-parity verified) 2026-07-29 12:29:39 -07:00
moa_loop.py perf(moa): cache resolved preset + per-slot runtime to cut cold-start latency (#66793) 2026-08-03 22:43:15 +05:30
moa_trace.py fix(moa): capture streamed aggregator output into full-turn traces (#56312) 2026-07-01 04:07:46 -07:00
model_metadata.py fix(model_metadata): read llama.cpp context from meta.n_ctx + accept sole model 2026-08-04 11:08:32 +05:30
models_dev.py Revert "remove Vercel AI Gateway and Vercel Sandbox (#33067)" 2026-07-29 19:48:37 -07:00
moonshot_schema.py fix(agent): inject empty required array on Moonshot object schemas 2026-07-20 11:05:52 -07:00
nous_rate_guard.py fix: decode config and state files as UTF-8 on non-UTF-8 locales 2026-07-24 17:10:39 -07:00
onboarding.py feat(surfaces): route busy-input corrections through active-turn redirect 2026-07-22 12:10:58 -05:00
oneshot.py feat(agent): one-shot LLM helper + llm.oneshot gateway RPC (#51261) 2026-06-23 08:01:50 +00:00
outbound_webhooks.py fix(hooks): flush outbound queue at interpreter exit 2026-08-02 15:01:11 -07:00
plugin_llm.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00
portal_tags.py feat(portal): ambient conversation context entangles aux/MoA/delegate calls 2026-07-16 01:13:43 -07:00
process_bootstrap.py fix(agent): apply pool-level keepalive to the process_bootstrap sibling builder 2026-07-05 03:14:55 -07:00
prompt_builder.py fix(agent): prevent historical steer replay 2026-08-05 13:00:47 +05:30
prompt_caching.py perf(prompt-caching): make PromptCachePlan.marker_count lazy 2026-08-01 15:39:58 +05:30
rate_limit_tracker.py
…
reactions.py feat(agent): core affection reaction detector + reaction_callback 2026-07-10 05:41:59 -05:00
reasoning_timeouts.py fix(compression): coherent context_length setter — no-op guard, re-floor on new window, un-strandable init log 2026-07-28 20:04:58 +05:30
redact.py perf(redact): eliminate exponential backtracking in config-key patterns 2026-08-01 15:58:25 +05:30
relay_llm.py Merge upstream main into model metrics 2026-08-02 20:10:47 -07:00
relay_runtime.py fix(relay): preserve skipped turn context 2026-08-03 13:43:17 -06:00
relay_tools.py Reapply "feat(observability): integrate NeMo Relay runtime and shared metrics" 2026-07-27 21:10:51 -07:00
replay_cleanup.py refactor: shared helpers for api_content sidecar pop/drop/extract 2026-07-19 08:25:35 +05:30
retry_utils.py refactor: single shared Retry-After parser 2026-07-29 10:13:50 -07:00
runtime_cwd.py fix(agent): scope install-tree guard to fallback-picked cwds, allow cli/tui in-tree dev 2026-07-16 04:32:23 -07:00
secret_scope.py fix(secrets): allowlist API_SERVER listener settings as global deployment env 2026-08-02 09:59:52 -07:00
session_activity.py refactor(agent): pin session activity heartbeat cadence + harden best-effort write 2026-08-02 16:16:36 -07:00
shell_hooks.py feat(hooks): outbound webhooks — push signed lifecycle events to external HTTP endpoints 2026-08-02 15:01:11 -07:00
skill_bundles.py feat(observability): add Relay skill metrics 2026-07-29 12:20:34 -07:00
skill_commands.py feat(observability): add Relay skill metrics 2026-07-29 12:20:34 -07:00
skill_preprocessing.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00
skill_utils.py fix(cron): don't let a cron job inherit a kanban worker's dispatcher identity 2026-08-06 03:16:26 +05:30
ssl_guard.py fix(ssl): detect and repair a missing certifi cacert.pem via existing venv-repair infra 2026-07-24 15:53:38 -07:00
ssl_verify.py fix(agent): honor custom CA certs on aux client + harden TLS resolution 2026-07-02 04:51:56 +05:30
stream_diag.py
…
stream_single_writer.py fix(streaming): make the single-writer fence best-effort so a missing guard can't crash a turn (#66448) 2026-07-17 18:31:09 +00:00
subagent_lifecycle.py fix(compression): let explicit interrupts cancel safely 2026-08-02 22:15:20 +05:30
subdirectory_hints.py fix(context): dedupe subdirectory hints by content digest and skip backup/vendor dirs 2026-08-03 20:43:34 +05:30
subscription_view.py feat(cli): plan catalog on Free + plan= deep link + top-up/auto-refill copy split (#68689) 2026-07-22 08:11:09 +05:30
system_prompt.py docs(system_prompt): fix stale reconstruct_static_prefix docstring example 2026-08-03 23:23:39 +05:30
think_scrubber.py fix(agent): re-arm think scrubber boundary after stream flush 2026-07-16 01:07:29 +05:30
thinking_timeout_guidance.py fix(agent): detect thinking-timeout for reasoning models and surface actionable guidance instead of misleading file-write advice 2026-06-25 19:00:48 -07:00
thread_scoped_output.py fix(bg-review): scope stdout/stderr silencing to the worker thread (#55966) 2026-06-30 17:28:33 -07:00
title_generator.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00
tool_dispatch_helpers.py feat(image): parallelize image_generate batches 2026-08-03 22:53:32 +05:30
tool_executor.py fix(agent): keep the start-order gate under the batch deadline and abort abandoned workers 2026-08-06 03:53:38 +05:30
tool_guardrails.py refactor(guardrails): make runaway-loop caps per-turn, not session-total 2026-07-26 21:27:45 -07:00
tool_result_classification.py fix(agent): preserve none vs unknown tool effects (#61783) 2026-07-11 05:41:58 -07:00
trace_upload.py fix(windows): sweep remaining unguarded text-mode subprocess sites codebase-wide 2026-07-24 11:45:57 -07:00
trajectory.py
…
transcription_provider.py
…
transcription_registry.py fix(deepinfra): harden multimodal provider routing 2026-07-14 02:59:39 +05:30
tts_provider.py
…
tts_registry.py feat(providers): Support DeepInfra as an LLM provider 2026-07-14 02:59:39 +05:30
turn_context.py fix(gateway): escalate the session-hygiene compaction cooldown on repeat failures 2026-08-06 05:00:48 +05:30
turn_finalizer.py fix(agent): invalidate flush-scan cursor at the defrag marker-pop sibling site 2026-07-31 23:18:12 -07:00
turn_retry_state.py fix(copilot): recover from stale/degraded token 400 AND expired IDE-token 401 2026-07-31 22:31:09 -07:00
turn_summary.py fix(cli): report unknown line deltas for content-free diffs instead of +0 -0 2026-07-26 17:47:51 -07:00
usage_pricing.py Revert "remove Vercel AI Gateway and Vercel Sandbox (#33067)" 2026-07-29 19:48:37 -07:00
verification_evidence.py fix(state): make journal mode canonical and behaviorally verified 2026-07-29 18:13:09 -07:00
verification_stop.py flux3 messaging system fixes 2026-07-30 15:20:09 -07:00
verify_hooks.py feat(agent): add pre_verify hook and verify-on-stop coding guidance 2026-06-30 00:59:29 -05:00
vertex_adapter.py security(vertex): route credential/project/region resolution through the profile secret scope 2026-07-02 06:07:56 +05:30
video_gen_provider.py feat(providers): Support DeepInfra as an LLM provider 2026-07-14 02:59:39 +05:30
video_gen_registry.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00
web_search_provider.py fix(web): widen config-aware env resolution to exa/parallel/tavily/brave-free providers 2026-07-06 02:42:24 -07:00
web_search_registry.py perf: use load_config_readonly() at read-only call sites in agent/ 2026-07-29 15:28:15 -07:00