Both session recovery paths (the startup stale-entry repoint and the
lazy in-message recovery) rebuilt the routing entry with updated_at=now
and never consulted _should_reset, so an opt-in idle/daily session_reset
policy was silently dead across any gateway restart: a recovered session
always looked freshly active, and since every subsequent message bumps
updated_at, a session recovered stale could then never age out at all.
Fix in three parts:
- _create_entry_from_recovered_row derives updated_at from the durable
last_activity_at the finder already returns on the row (no extra DB
round-trip; the original PR added SessionDB.get_last_activity for
this, unnecessary post-#82633), falling back to created_at. An
invalid or missing started_at now maps to epoch 0 instead of now — an
invalid durable timestamp must look old, never freshly active.
reset_had_activity is set from the row's durable activity/message
signals so the continuity hint stays accurate.
- _recover_session_from_db evaluates _should_reset on the rebuilt entry:
an overdue session is durably promoted to a reset boundary
(promote_to_session_reset, falling back to end_session) and the stale
mapping is dropped instead of repointed.
- _query_recoverable_session no longer reopens the row; the
get_or_create_session recovery phase evaluates _should_reset first and
either feeds the normal auto-reset create path (reset notice,
prev_session_id continuity, durable promotion) or reopens and
publishes the recovered entry exactly as before.
Behavior is unchanged under the default session_reset mode "none":
_should_reset returns None there, so recovery still resumes every
recoverable row — only users who opted into idle/daily resets see the
policy actually applied across restarts.
Cherry-picked from #78618 and adapted to the #82633 finder.
(cherry picked from commit 31c71f7629)
Complete the /branch routing-identity fix (salvaged from PR #62278 by
@jcjc81): in addition to user_id/session_key/chat_id/chat_type/thread_id,
forward origin_json and display_name at create_session() time, matching
the reset-path db_create_kwargs pattern (#82633) so the branch row is
born with full identity — no backfill gap for state.db consumers
(mcp_serve, mirror, channel directory) if a crash lands before
switch_session().
The obsolete compression-rotation half of #62278 was dropped: rotation
now goes exclusively through publish_compression_child, which already
copies all identity columns in-transaction.
The sweeper flagged two gaps in the routing-columns fix:
1. /branch create_session() omitted user_id and session_key — the
fallback lookup path (find_latest_gateway_session_for_peer) requires
user_id to match the complete peer tuple when session_key lookup fails,
and /resume IDOR guards reject sessions without matching user_id.
2. Compression-rotation create_session() omitted agent._user_id — same
problem: rotated child cannot satisfy persisted /resume ownership proof
before the later gateway backfill.
Forward user_id and session_key at CREATE time in both call sites so
the child row is immediately fully routable with zero backfill gap.
Extended tests: compression rotation asserts user_id is carried (and None
for CLI sessions). Branch routing asserts both user_id and session_key on
the child row before switch_session runs.
Same defect as the compression-rotation fix in the prior commit, found
during a full-audit of every create_session() call site per the repo's
'fix the whole bug class, sibling call paths included' contribution
guidance.
_handle_branch_command() (gateway/slash_commands.py) creates the branched
child session via create_session() without chat_id/chat_type/thread_id.
The routing columns are only backfilled later, when switch_session() runs
at the end of the function and calls _record_gateway_session_peer(). In
between, the function copies the parent's conversation history to the new
session_id one message at a time, with each append_message() call
independently try/excepted (best-effort) — a crash/kill anywhere in that
window leaves the branched session permanently unroutable, same failure
mode as the compression bug: NULL chat_id/thread_id can never be found by
find_latest_gateway_session_for_peer, AND unreachable via /resume's IDOR
guard (which requires the row's chat_id/thread_id to match the caller's).
Fix: forward source.chat_id/chat_type/thread_id at create_session() time,
mirroring the existing correct pattern already used by /title's
auto-create path a few hundred lines up in the same file (which has an
explicit IDOR-scoping comment justifying it).
Tests: tests/gateway/test_branch_routing_columns.py drives the real
_handle_branch_command against a real SessionStore + SessionDB (SQLite in
tmp_path, no DB/session-store mocks). Patches switch_session to simulate a
crash landing before it runs (the exact gap the routing columns need to
survive), then asserts the branched child's chat_id/chat_type/thread_id
are already correct in state.db at that point. RED verified against
unpatched code (assert None == '170829464'), GREEN after the fix.
Regression: 102/102 across the new test + pre-existing /branch, session
boundary, compression rotation, DM thread seeding, session API, and
resume-command suites. Broader tests/gateway/ -k "branch or session_api or
resume or topic_mode or session_boundary" sweep: 255/255 passed, 1
(unrelated) skip.
When the per-session pending transcript queue hits _MAX_PENDING_PER_SESSION
(200) while the session DB is broken, the gateway previously popped the
oldest message and discarded it permanently — silent user data loss during
live operation (#78182). The on-disk pending spool only ran at shutdown via
flush_pending_to_file.
Extend that existing spool machinery for runtime drops:
- gateway/shutdown_flush.py: add spool_dropped_transcript_message() and
drain_transcript_spool(), reusing _get_flush_dir/_write_payload (same
atomic-JSON pending_messages/ spool format). recover_pending_to_db()
now also replays transcript_cap_drop payloads left over across restarts.
- gateway/session.py: on cap eviction, spool the dropped message and log a
WARNING that includes the spool path; if spooling fails, degrade to the
previous drop-and-warn behavior. On the next fully successful transcript
flush for that session, drain and replay spooled messages in drop order;
replay failures keep the spool files for the next attempt.
- tests/gateway/test_pending_queue_spool.py: drop→spool→drain roundtrip,
per-session drain isolation, spool-failure degradation, replay-failure
retention, and spool primitive ordering/reason filtering.
No new config; extends existing flush_pending_to_file infrastructure per
AGENTS.md guidance.
Refs #82616, #78182
Root cause of #82616: gateway session identity (session_key/chat_id/
origin_json) was written best-effort in a separate UPDATE after row
creation, both reset-path DB writes swallowed failures silently
(logger.debug / bare print), transcript reads ignored the reroute map
that writes follow, and restart recovery ranked candidate rows by
started_at while hard-rejecting empty rows. A single failed write could
therefore strand the live conversation in an unroutable orphan row while
a days-old zombie kept the routing key — after any gateway restart the
chat silently resumed the zombie (user-visible context loss, 5 confirmed
incidents on one install since June).
Four class fixes:
1. Identity lands atomically in the session INSERT: origin_json and
display_name join _insert_session_row's column list + COALESCE
backfill; both gateway creation paths (get_or_create + reset) pass
full identity including parent_session_id lineage (fixes#12857).
2. record_gateway_session_peer self-heals: when the target row is
missing (failed/deferred create, crash window) it INSERTs the row
with full identity instead of silently no-opping — every per-turn
peer refresh is now a repair opportunity, and an identity-less lazy
writer (update_token_counts/record_auxiliary_usage) can never leave
a gateway session permanently unroutable.
3. load_transcript follows the write-side reroute chain and the durable
compression tip before querying, so reads can no longer return 0
rows for a session whose messages live under its compression child;
read exceptions are WARNING, distinguishable from an empty result.
4. find_latest_gateway_session_for_peer ranks by
COALESCE(last_activity_at, started_at) (message-bearing rows first)
and returns an empty-but-keyed row instead of None — a zombie
predecessor can no longer beat the live conversation, and recovery
never mints a fresh id when a keyed row exists.
Reset-path DB write failures now log at WARNING with the routing
consequence spelled out.
Tests: tests/gateway/test_session_continuity_82616.py (11 tests) —
sabotage-verified: 6/11 fail without the fixes. E2E incident replay
(real SessionDB, temp HERMES_HOME) confirms the production shape now
resolves to the live session.
Fixes#82616. Related: #12857, #78182 (read-path half), #79576.
Personality persistence used to be split per surface: the TUI/desktop wrote
the NAME to display.personality while the CLI/gateway wrote rendered TEXT
into agent.system_prompt (and their /personality none only blanked the
text, leaving the name behind). When #81946 made display.personality
authoritative everywhere, stale names written long ago resurrected
personalities users had turned off - kawaii defaulting on after updating.
- hermes_cli/personality.py: new single owner of personality state.
Built-in personality definitions, neutral-name normalization, rendering,
availability (built-ins overlaid by agent.personalities), overlay
resolution, and the ONLY sanctioned persistence path
(persist_personality -> display.personality; never agent.system_prompt).
- v34 config migration: one-time reset of display.personality to none
(announcing which personality was cleared and how to re-enable), plus a
scrub of agent.system_prompt when it verbatim-equals a known personality
render (machine-written by the old CLI/gateway). Hand-written manual
prompts are never touched.
- All surfaces rewired through the module: CLI /personality (incl. active
marker in the list), gateway /personality, TUI config.set + slash path
(which previously applied without persisting), TUI config.get (reports
the EFFECTIVE personality), completer, hermes config display, and the
tui_gateway health probe.
- cli.py/config duplicates removed: built-ins now defined once; the
desktop mirrors them from one lib module (src/lib/personalities.ts).
- Docs updated: selection lives in display.personality, built-ins always
available, one-time reset note.
asyncio.ensure_future(result) creates a task with only a weak ref in
the event loop's task table. After the carrier raises CancelledError,
the local 'task' variable goes out of scope and the loop can GC the
handler before it finishes — the exact 'handler killed mid-flight'
class we are fixing, just via GC instead of cancellation.
Add _detached_fatal_tasks set on BasePlatformAdapter (matching the
gateway-level pattern in _handle_adapter_fatal_error). Uses getattr
fallback for test stubs built via object.__new__().
When an adapter escalates a retryable fatal error from inside one of its
own tasks (e.g. Telegram's _polling_error_task after exhausting polling
network retries), the gateway's _handle_adapter_fatal_error tears the
adapter down via disconnect() — which cancels that very task. The
propagating CancelledError killed the handler between popping the
adapter from the adapter map and queueing the platform in
_failed_platforms, leaving a zombie gateway: process alive, zero
connected platforms, zero pending retries, until a manual restart.
Run the handler as a detached task under asyncio.shield so carrier
cancellation no longer aborts teardown/queueing mid-flight. The carrier
still observes CancelledError (teardown semantics unchanged); only the
handler is protected. A done-callback consumes the detached task's
exception to avoid 'Task exception was never retrieved' noise.
Fixes#81335
The auto-resume restart-loop breaker (#30719, defense-3) pruned its boot
log against an absolute `window_seconds` (default 60s). That prune is
period-sensitive: a crash cycle slower than the window drops its own
history on every boot, so the counter never leaves 1 and the breaker can
never trip, no matter how long the loop runs.
The cycle reported in #81642 is ~150s — a wedged event loop, the liveness
watchdog hard-exiting at ~90s, a supervisor respawn, and auto-resume
replaying the same session that wedges it again. Structurally invisible to
a 60s window: `gateway/restart_loop.json` kept a single timestamp across 15
kills in one morning. Because every cycle leaves a gateway that cannot
process SIGTERM, `hermes update` has no drainable gateway to stop, which is
the reported hang.
Chain boots on the inter-boot GAP instead of an absolute window: two boots
belong to the same loop when they are no more than `max_gap_seconds` apart
(default 300s, floored by `window_seconds` so widening the window never
makes the breaker less sensitive). The verdict becomes period-agnostic —
the original ~10s respawn loop still trips in 3 boots, and so does a 150s
one — while a boot after real quiet resets the chain, so occasional
operator restarts still never accumulate. The persisted chain is capped at
50 entries.
- gateway/restart_loop_guard.py: gap-chained pruning (`_chain_ending_at`),
`DEFAULT_MAX_GAP_SECONDS`, `max_gap_seconds` kwarg on the three entry
points, clock-step tolerance, bounded state file
- gateway/run.py: `_restart_loop_guard_config` reads and returns
`max_gap_seconds`; the auto-resume call site passes it through
- hermes_cli/config_defaults.py: `gateway.restart_loop_guard.max_gap_seconds`
Tests: 7 new cases in TestRestartLoopGuard covering the slow cycle, chain
persistence, quiet-period reset, the #30719 fast loop, the config knob, the
window floor, and the disabled breaker. Verified RED before the fix (the
slow-cycle case asserted `[1300.0] == [1000.0, 1150.0, 1300.0]`, exactly
the single-timestamp state file from the report) and GREEN after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Titling is two-stage — a slice of the user's own words lands inline, the
model's version replaces it a second later — and the platform rename lanes
fired on both. That is two rate-limited calls to reach one name, and
Discord allows two channel renames per ten minutes, so the throwaway could
be the one that survived. The callback now carries which stage it is, and
the lanes take the model's.
The relay lane also asked where the reply landed at title time, which is
before the model has answered: it polled the send-result cache for ten
seconds and read the timeout as "never auto-threaded", so any turn with
tool calls in it silently kept its raw thread name. Wait on the send
itself instead — the adapter already owns that cache, so it can say when a
reply arrives and, just as usefully, that one arrived carrying nothing.
Adds the failed-result guard the salvage review called for:
_deliver_queued_first_response now takes deliver_media and the queued
follow-up call site passes deliver_media=not _delivery_result.get('failed').
A failed turn still delivers its normalized failure text (pinned by
test_run_agent_sends_normalized_failure_before_queued_followup), but its
attachments are no longer uploaded as if the turn succeeded — mirroring
the completed-turn path's 'not agent_result.get(failed)' guard.
Regression test added.
Ensure queued follow-up resends keep MEDIA-backed attachments by replaying the
first response through the gateway's text-plus-media delivery flow instead of a
plain adapter text send.
Apply the same metadata invariant to sealed split chunks, keep expect_edits on live previews, and exercise the real Telegram adapter path with rich messages enabled but rich drafts disabled.\n\nCredits PR #78525 by @Slobaka for the reproduced rich_messages/rich_drafts combination.
A finalized native draft is the first persistent send and will not be edited again. Omitting expect_edits lets Telegram use sendRichMessage for the persistent final instead of degrading tables through MarkdownV2.\n\nAdapted from PR #46536.
Titling fired on the first response, so a session sat unnamed for the whole
opening turn - p50 151s, p90 1212s across real sessions, because a turn is
tool calls, not one round-trip. A turn that failed or was interrupted never
got a title at all. Four surfaces each carried their own copy of the call.
Move it into the shared turn prologue and split it in two: a deterministic
title derived from the user's opening message, written inline before the
model runs, then one small-model call that upgrades it. The response is
constrained to a JSON object so there is no preamble to strip, and control
wrappers are stripped rather than refused, so a slash command titles as
what the user asked for instead of the command itself.
Complements the cherry-picked contributor fixes and closes out the
remaining sites of the 'missing explicit encoding' bug class, which is
now permanently gated by ruff PLW1514 (enabled repo-wide in
pyproject.toml and enforced by the blocking `ruff check .` step in
.github/workflows/lint.yml):
- tools/memory_tool.py: read MEMORY.md/USER.md via utf-8-sig so a
Notepad BOM never glues U+FEFF onto the first entry (issue #10878,
PR #10888 by @easyvibecoding — strict-decode contract of
_read_raw_checked preserved rather than errors="replace", so
undecodable files still refuse read-modify-write instead of being
lossily rewritten). Regression tests included.
- tools/skills_tool.py: SKILL.md and skill file reads pinned to
utf-8-sig + errors="replace" — deterministic across platforms instead
of the locale fallback proposed in PR #51701 (superseded: falling back
to cp1252/GBK makes the same skill render differently per host); .env
reader aligned with the canonical utf-8-sig dialect in hermes_cli/config.py.
- agent/shell_hooks.py, hermes_cli/main.py, gateway/slash_commands.py:
explicit utf-8 on the remaining fdopen/open text-mode sites flagged by
the AlexFucuson9 sweep series (#56033#56940#65565#66782#66791).
Co-authored-by: easyvibecoding <easyvibecoding@users.noreply.github.com>
Co-authored-by: AlexFucuson9 <AlexFucuson9@users.noreply.github.com>
Co-authored-by: flyingdoubleg <wangzhe00zju@gmail.com>
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Own the surrogate-crash class at three chokepoints instead of leaf sites:
- finalize_turn scrubs final_response once where model text leaves the
conversation loop — covers oneshot stdout (#80366), NIM/any-provider
responses (#19819), and every delivery consumer of the turn result.
- _sanitize_gateway_final_response scrubs at the gateway chat-surface
boundary — Telegram utf16_len (#55309) and Signal formatting (#55143)
can no longer see a lone surrogate; raw-text surfaces keep passthrough.
- run_conversation walks the fully-built api_kwargs with
_sanitize_structure_surrogates so tool descriptions (session_search,
#50959) and every other request-body leaf are JSON-encodable before
any provider sees them.
Regression tests pin all three chokepoints plus helper semantics.
Cherry-picked alongside #79240 (TheophilusChinomona) and #80374
(rainbowgore) whose commits precede this one with authorship preserved.
Persist display.personality only; apply rendered text as an in-session
overlay across CLI, TUI config.set, and gateway /personality.
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: EMT5320 <1908937833@qq.com>
e81d18dfb collapsed six per-surface copies of reasoning resolution onto
resolve_reasoning_config() and, in its own words, "fixes the gateway
resolving reasoning against config model.default instead of the session's
effective model". It did not touch gateway/platforms/api_server.py, which
kept that defect.
_create_agent() called GatewayRunner._load_reasoning_config() with no
model on its first line — before the model precedence chain (browser lock
-> session /model -> session row -> route -> per-request -> defaults) has
run. Per-model agent.reasoning_overrides therefore keyed off model.default
on the one surface where every request names its own model: a request for
a model with an override silently got the global effort instead.
Resolve after the chain settles, so the override follows the model the
request actually runs. An explicit per-request reasoning parameter still
takes precedence over config.
The existing test stub for _load_reasoning_config took no arguments (it
mirrored the old call); it now matches the real signature, as the sibling
stub in the same file already did.
Follow-ups on the salvaged commit (#37207 by @charzhou):
- Persistent /root home mount translates too: an agent writing
/root/out.png produced a real host file under
<sandbox>/docker/default/home the gateway could not find.
- /root/.hermes cache mounts translate to the HOST cache (longest-prefix
beats the home mount), so MEDIA:<agent_visible_image> paths deliver.
- /root/.hermes/* OUTSIDE a cache mount never translates through the home
mount: those are the sandbox's credential copies (.env, auth.json) that
sit outside the host-side denylist prefixes — fail closed.
- Run the idempotent terminal-config->env bridge before mount parsing so
in-process gateways (Desktop backend, hermes serve) see the active
backend and docker_volumes (covers #42299's /output case there too).
Translate MEDIA paths under configured Docker volume mounts (and the
default persistent /workspace) to host paths before media delivery
validation, using longest container-prefix match so host:/workspace and
/output export mounts work.
Salvage of PR #17973 by @TKCen (Sebastian Hänisch), re-implemented on
current main to preserve speed/instructions/provider params,
prepare_spoken_text normalization, OPUS_VOICE_PLATFORMS, is_write_denied
path security, microsecond timestamps, and the streaming-TTS gate.
- Split long TTS text into provider-safe chunks instead of truncating
- Pack generated audio against platform upload limits (Discord 10MB,
Telegram 50MB, configurable via tts.delivery_profiles)
- Combine chunks with ffmpeg (OGG/Opus re-encoded, MP3 stream-copied)
- Multi-file delivery when combination fails or would exceed limits
- Remove hard [:4000] truncation from all callers (cli.py, voice.py,
gateway/run.py, gateway/platforms/base.py)
- Gemini TTS raises ValueError instead of silently truncating when
composed prompt exceeds the provider limit
Simplify-code fixes: removed dead all_touched_paths set, added
try/finally for scratch file cleanup on exception, clean error response
on chunk failure instead of leaking stale file_path.
The non-streaming /v1/responses path built function_call and
function_call_output output items with no status field (and no item id),
while the SSE streaming path correctly emits status in_progress ->
completed. Spec-strict OpenAI clients reading the non-streaming output
array could interpret the status-less function_call items as pending
calls the CLIENT must execute — but these tools were already executed
server-side by the Hermes agent and are replayed for structured tool UI
only. Reported by a community user whose GPT-5.6 client concluded 'a
server should not tell an OpenAI client to execute a tool the server
already executed itself'.
- _extract_output_items now stamps status: completed and spec-shaped
item ids (fc_/fco_) on replayed items, matching the streaming path
- test updated to pin status + id shape
- docs example updated + explicit note that output tool calls are
replayed, never pending
Four fix-forwards from the adversarial post-merge audit of the Aug 7
unreviewed merge batch:
- estop (#81148): is_engaged() now fails SAFE (engaged) on stat errors;
the gateway estop gate lets recognized slash commands and replies owned
by in-flight work (update prompts, clarify, slash-confirm, tool
approvals, running sessions) through instead of consuming them; new
gateway /pause [reason|off] command gives messaging-only operators an
in-band engage/resume path (busy_policy=dispatch so it works mid-run).
- cron monitor mode (#81138): execution-mode invariants (monitor x
no_agent, monitor_script x monitor_url, no_agent-requires-script) now
have ONE owner (_validate_job_mode_invariants) called from BOTH
create_job and update_job, so the create-time invariant can no longer
be silently violated through the update door.
- cron notepad (#81139): remove_job now clears the job's notepad rows
(clear_notepad was dead code -> orphaned KV state forever); clear is
best-effort and no-ops without creating notepad.db.
- delegation batch gate (#81141): template-marker regex narrowed to
multi-word placeholder shapes only (<feature name>, {file_path}) so
generics (Vec<T>), HTML tags, JSON snippets, glob braces and f-string
style no longer reject legitimate batches; duplicate-goal rejection
removed (best-of-N fan-outs are legitimate).
- Move classify_persistence_error into hermes_state beside is_disk_full_error
and delegate the disk bucket to it (fixes 'ENOSPC writing state.db' and
'not enough space' classifying as unknown). run_agent keeps a thin lazy
delegating wrapper so the documented import path and fast import survive.
- Classify CompressionSessionBusyError (and its RPC-wrapped message forms)
as 'locked': the motivating #81227 failure mode stringifies to 'is being
compressed by another writer', which the substring heuristic missed.
- Export PERSISTENCE_ERROR_CAUSES and iterate it in the cron explainer
suppression instead of a hardcoded tuple, so a future cause bucket cannot
silently desynchronize cron delivery.
- Hedge the gateway locked/unknown recovery wording ('should already be
saved' instead of 'was recorded') to match the explainer - the early
turn-start persist may also have failed.
- Drop STATE_DB_WAL_WARN_BYTES (speculative dead constant with no consumer;
the pre-existing 50 MB doctor WAL check covers the warning).
- Tests: compression-busy classification, is_disk_full_error delegation,
causes-tuple coverage; mutation-checked red-green.
Two defects in _normalize_empty_agent_response surfaced together during a
state.db lock-contention incident on an enterprise Slack deployment:
- the error lookup used dict.get's default, which an explicit
'error': None value bypasses, rendering 'The request failed: None' /
'unknown error';
- persistence failures fell through to the generic branch, whose 'use
/reset' advice is harmful for this failure mode (destroys conversation
context, fixes nothing).
Persistence-failed turns (failure_reason session_persistence_failed:*,
with a legacy fallback on the error text) now get a dedicated message:
storage was temporarily unavailable, the message was recorded, send it
again — with a disk-specific variant. No /reset suggestion. All other
branches unchanged.
agent_result is guaranteed non-None at this point — line 17926 calls
.get() on it unconditionally, and line 18099 in the same block does
the same without the guard. The if/else was dead defensive code
inconsistent with surrounding access patterns.
Gateway hook plugins have no way to know which LLM model or provider was used
for a turn. The agent result dict already contains model and provider from
finalize_turn(), but neither field was forwarded into the agent:end hook
context.
Add agent_result.get("model") and agent_result.get("provider") to the
agent:end emit payload so gateway hooks can read them via context.get().
After Discord reconnect, on-disk start_time can be null while the live
record has a fingerprint. Requiring equality made the gateway treat its
own PID as a foreign token holder (#81468).
Thread.start() failure is RuntimeError; catching BaseException here
swallowed KeyboardInterrupt/SystemExit without re-raise (unlike _worker,
which forwards them into the future).
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).
Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).
Fixes#10809
The entry_data/lock_held path (used by mark_turn_active/clear_turn_active)
was eagerly constructing a full O(n) snapshot of all routing entries under
the lock on every turn, even when the DB upsert would succeed. Defer the
fallback_data construction to the except branch where it's actually needed,
and respect lock_held to avoid re-acquiring a non-reentrant _lock.
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.
Now:
- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
carrying a reason + retry_hint. Subclassing RuntimeError keeps every
existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
(missing exe, non-executable exe, daemon timeout, `docker version`
failure).
- terminal_tool catches EnvironmentConnectionError and returns a
structured tool result the model can act on:
{"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
The failed backend is evicted from the environment cache so a later
call retries from scratch — recovery is automatic once the backend is
reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
only infrastructure failures classify as degraded.
Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.
Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
Two small config-gated features:
1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
true, config.yaml): a running card with broken claim bookkeeping
(claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
restore) is invisible to all existing recovery paths
(release_stale_claims requires claim_expires NOT NULL,
detect_crashed_workers requires host-local lock + pid,
detect_stale_running is config-disabled by default) and shows Running
forever. New reconcile_orphaned_running() pass in kanban_db.py runs
each dispatch_once tick: requeues orphans to ready with an explanatory
comment, closes any leaked run, emits a 'reconciled' event, and defers
when the recorded PID is still alive on this host (never requeue
beside a live worker). Surfaced via DispatchResult.reconciled_orphans.
2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
config.yaml): optional {name, value_from: static|profile, value}
mapping; the header is attached to that server's HTTP/SSE transport
requests. 'static' sends the config value; 'profile' resolves the
active Hermes profile name once at connect time (no per-call
mutation). Explicit per-server headers of the same name (any casing)
win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.
Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.
Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
Resumable ESTOP sentinel at $HERMES_HOME/ESTOP that halts NEW work only:
- agent/estop.py: sentinel engage/disengage/is_engaged (single stat, no
caching), optional reason + timestamp stored as JSON, paused_reply()
notice, check_paused() log-once-per-engagement helper. Corrupt/empty
sentinel still pauses (fail safe); a `touch ~/.hermes/ESTOP` works.
- cron/scheduler.py: tick() skips dispatch while engaged (logged once per
engagement, not per tick). Due jobs simply wait for the next tick after
resume — in-flight runs are never touched.
- gateway/kanban_watchers.py: dispatcher skips auto-decompose and worker
spawning while engaged; zombie reaping still runs and running workers
finish naturally.
- gateway/run.py: new gateway turns (post-auth, non-internal) get a brief
"Hermes is paused" reply instead of an agent run. Internal events
(in-flight background completions) bypass the gate.
- hermes_cli/subcommands/pause.py: `hermes pause [--reason]` and
`hermes resume`, wired into main() and _BUILTIN_SUBCOMMANDS.
- hermes_cli/status.py: `hermes status` shows a PAUSED banner (one stat).
- tests/test_estop.py: 20 tests — sentinel lifecycle, reason surfacing,
log-once, cron skip + resume, kanban gate, gateway paused reply +
internal bypass, CLI idempotence, builtin-set parity, status line.
Never kills in-flight work; resumable with no restart. Footprint ladder:
CLI command only, no new model tool, no new env vars.
Ported from: gastownhall/gastown estop.go (MIT); related prior art:
#26778 (/panic — kill/exit semantics, deliberately different: ours is a
resumable pause), #44617 (interrupt in-flight cron — out of scope here).
Hill Chitsanupong
TomAce7
Teknium
Jason
kshitij
gregorustar-maker
joaomarcos
Brooklyn Nicholson
StellarisW
Kong
liuhao1024
Hermes Agent
Drexuxux
CharZhou
Victor Kyriazakos
Ken Weiner
HexLab98
kinsolee
HenryG
gnanirahulnutakki
Gille
Dominic Bejar