hermes-agent/gateway
Teknium 5c29566e8d feat(terminal): graceful degradation for remote backend connection failures
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.

Now:

- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
  carrying a reason + retry_hint. Subclassing RuntimeError keeps every
  existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
  bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
  (missing exe, non-executable exe, daemon timeout, `docker version`
  failure).
- terminal_tool catches EnvironmentConnectionError and returns a
  structured tool result the model can act on:
    {"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
  The failed backend is evicted from the environment cache so a later
  call retries from scratch — recovery is automatic once the backend is
  reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
  config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
  sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
  TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
  historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
  only infrastructure failures classify as degraded.

Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.

Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
2026-08-07 09:07:55 -07:00
..
assets
…
builtin_hooks remove: BOOT.md built-in hook (#17093) 2026-04-28 09:50:27 -07:00
platforms fix(cron): surface initial scheduler registration failures 2026-08-07 17:45:06 +05:30
relay
…
__init__.py
…
agent_cache_pressure.py fix: follow-ups for salvaged PR #80795 2026-08-07 21:02:33 +05:30
authz_mixin.py
…
cgroup_cleanup.py
…
channel_directory.py perf(slack): parallelize conversations.info lookups with asyncio.gather 2026-08-07 18:40:33 +05:30
code_skew.py
…
config.py fix: address self-review findings on the check_fn/ensure_deps_fn split 2026-08-07 13:28:43 +05:30
cwd_placeholder.py
…
dead_targets.py
…
delivery.py
…
delivery_ledger.py
…
display_config.py
…
drain_control.py
…
hooks.py
…
kanban_watchers.py feat(kanban,mcp): orphaned-card reconciliation + per-server MCP identity header 2026-08-07 08:58:20 -07:00
lifecycle_ledger.py
…
memory_monitor.py
…
message_timestamps.py
…
mirror.py
…
pairing.py
…
platform_registry.py fix(gateway): split check_fn (passive probe) from ensure_deps_fn (active installer) 2026-08-07 13:28:43 +05:30
profile_routing.py
…
readiness.py
…
response_filters.py
…
restart.py
…
restart_loop_guard.py
…
rich_sent_store.py
…
run.py feat(terminal): graceful degradation for remote backend connection failures 2026-08-07 09:07:55 -07:00
runtime_footer.py
…
scale_to_zero.py
…
session.py fix(yuanbao): preserve archived history on recall redaction 2026-08-07 13:56:18 +05:30
session_context.py
…
session_stall.py
…
session_state.py refactor(gateway): extract the hygiene recovery gate and forward the failure reason 2026-08-06 15:04:53 +05:30
shutdown_flush.py
…
shutdown_forensics.py
…
shutdown_watchdog.py
…
slash_access.py
…
slash_commands.py fix(agent): finish the #80622 bug class — sibling predicates, refund ordering, prompt carve-out, honest skip response 2026-08-07 19:44:35 +05:30
status.py
…
status_phrases.py
…
sticker_cache.py
…
stream_consumer.py fix(gateway): don't claim deleted head chunks as delivered in the empty-fallback recovery 2026-08-06 03:13:32 +05:30
stream_dispatch.py
…
stream_events.py
…
streaming_tts_consumer.py
…
systemd_notify.py
…
turn_context.py
…
turn_lease.py refactor(gateway): drop dead degraded token field; de-churn salvage diff 2026-08-07 18:23:57 +05:30
wake.py
…
whatsapp_identity.py
…