The non-streaming /v1/responses path built function_call and
function_call_output output items with no status field (and no item id),
while the SSE streaming path correctly emits status in_progress ->
completed. Spec-strict OpenAI clients reading the non-streaming output
array could interpret the status-less function_call items as pending
calls the CLIENT must execute — but these tools were already executed
server-side by the Hermes agent and are replayed for structured tool UI
only. Reported by a community user whose GPT-5.6 client concluded 'a
server should not tell an OpenAI client to execute a tool the server
already executed itself'.
- _extract_output_items now stamps status: completed and spec-shaped
item ids (fc_/fco_) on replayed items, matching the streaming path
- test updated to pin status + id shape
- docs example updated + explicit note that output tool calls are
replayed, never pending
Four fix-forwards from the adversarial post-merge audit of the Aug 7
unreviewed merge batch:
- estop (#81148): is_engaged() now fails SAFE (engaged) on stat errors;
the gateway estop gate lets recognized slash commands and replies owned
by in-flight work (update prompts, clarify, slash-confirm, tool
approvals, running sessions) through instead of consuming them; new
gateway /pause [reason|off] command gives messaging-only operators an
in-band engage/resume path (busy_policy=dispatch so it works mid-run).
- cron monitor mode (#81138): execution-mode invariants (monitor x
no_agent, monitor_script x monitor_url, no_agent-requires-script) now
have ONE owner (_validate_job_mode_invariants) called from BOTH
create_job and update_job, so the create-time invariant can no longer
be silently violated through the update door.
- cron notepad (#81139): remove_job now clears the job's notepad rows
(clear_notepad was dead code -> orphaned KV state forever); clear is
best-effort and no-ops without creating notepad.db.
- delegation batch gate (#81141): template-marker regex narrowed to
multi-word placeholder shapes only (<feature name>, {file_path}) so
generics (Vec<T>), HTML tags, JSON snippets, glob braces and f-string
style no longer reject legitimate batches; duplicate-goal rejection
removed (best-of-N fan-outs are legitimate).
- Move classify_persistence_error into hermes_state beside is_disk_full_error
and delegate the disk bucket to it (fixes 'ENOSPC writing state.db' and
'not enough space' classifying as unknown). run_agent keeps a thin lazy
delegating wrapper so the documented import path and fast import survive.
- Classify CompressionSessionBusyError (and its RPC-wrapped message forms)
as 'locked': the motivating #81227 failure mode stringifies to 'is being
compressed by another writer', which the substring heuristic missed.
- Export PERSISTENCE_ERROR_CAUSES and iterate it in the cron explainer
suppression instead of a hardcoded tuple, so a future cause bucket cannot
silently desynchronize cron delivery.
- Hedge the gateway locked/unknown recovery wording ('should already be
saved' instead of 'was recorded') to match the explainer - the early
turn-start persist may also have failed.
- Drop STATE_DB_WAL_WARN_BYTES (speculative dead constant with no consumer;
the pre-existing 50 MB doctor WAL check covers the warning).
- Tests: compression-busy classification, is_disk_full_error delegation,
causes-tuple coverage; mutation-checked red-green.
Two defects in _normalize_empty_agent_response surfaced together during a
state.db lock-contention incident on an enterprise Slack deployment:
- the error lookup used dict.get's default, which an explicit
'error': None value bypasses, rendering 'The request failed: None' /
'unknown error';
- persistence failures fell through to the generic branch, whose 'use
/reset' advice is harmful for this failure mode (destroys conversation
context, fixes nothing).
Persistence-failed turns (failure_reason session_persistence_failed:*,
with a legacy fallback on the error text) now get a dedicated message:
storage was temporarily unavailable, the message was recorded, send it
again — with a disk-specific variant. No /reset suggestion. All other
branches unchanged.
agent_result is guaranteed non-None at this point — line 17926 calls
.get() on it unconditionally, and line 18099 in the same block does
the same without the guard. The if/else was dead defensive code
inconsistent with surrounding access patterns.
Gateway hook plugins have no way to know which LLM model or provider was used
for a turn. The agent result dict already contains model and provider from
finalize_turn(), but neither field was forwarded into the agent:end hook
context.
Add agent_result.get("model") and agent_result.get("provider") to the
agent:end emit payload so gateway hooks can read them via context.get().
After Discord reconnect, on-disk start_time can be null while the live
record has a fingerprint. Requiring equality made the gateway treat its
own PID as a foreign token holder (#81468).
Thread.start() failure is RuntimeError; catching BaseException here
swallowed KeyboardInterrupt/SystemExit without re-raise (unlike _worker,
which forwards them into the future).
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).
Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).
Fixes#10809
The entry_data/lock_held path (used by mark_turn_active/clear_turn_active)
was eagerly constructing a full O(n) snapshot of all routing entries under
the lock on every turn, even when the DB upsert would succeed. Defer the
fallback_data construction to the except branch where it's actually needed,
and respect lock_held to avoid re-acquiring a non-reentrant _lock.
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.
Now:
- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
carrying a reason + retry_hint. Subclassing RuntimeError keeps every
existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
(missing exe, non-executable exe, daemon timeout, `docker version`
failure).
- terminal_tool catches EnvironmentConnectionError and returns a
structured tool result the model can act on:
{"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
The failed backend is evicted from the environment cache so a later
call retries from scratch — recovery is automatic once the backend is
reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
only infrastructure failures classify as degraded.
Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.
Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
Two small config-gated features:
1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
true, config.yaml): a running card with broken claim bookkeeping
(claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
restore) is invisible to all existing recovery paths
(release_stale_claims requires claim_expires NOT NULL,
detect_crashed_workers requires host-local lock + pid,
detect_stale_running is config-disabled by default) and shows Running
forever. New reconcile_orphaned_running() pass in kanban_db.py runs
each dispatch_once tick: requeues orphans to ready with an explanatory
comment, closes any leaked run, emits a 'reconciled' event, and defers
when the recorded PID is still alive on this host (never requeue
beside a live worker). Surfaced via DispatchResult.reconciled_orphans.
2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
config.yaml): optional {name, value_from: static|profile, value}
mapping; the header is attached to that server's HTTP/SSE transport
requests. 'static' sends the config value; 'profile' resolves the
active Hermes profile name once at connect time (no per-call
mutation). Explicit per-server headers of the same name (any casing)
win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.
Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.
Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
Resumable ESTOP sentinel at $HERMES_HOME/ESTOP that halts NEW work only:
- agent/estop.py: sentinel engage/disengage/is_engaged (single stat, no
caching), optional reason + timestamp stored as JSON, paused_reply()
notice, check_paused() log-once-per-engagement helper. Corrupt/empty
sentinel still pauses (fail safe); a `touch ~/.hermes/ESTOP` works.
- cron/scheduler.py: tick() skips dispatch while engaged (logged once per
engagement, not per tick). Due jobs simply wait for the next tick after
resume — in-flight runs are never touched.
- gateway/kanban_watchers.py: dispatcher skips auto-decompose and worker
spawning while engaged; zombie reaping still runs and running workers
finish naturally.
- gateway/run.py: new gateway turns (post-auth, non-internal) get a brief
"Hermes is paused" reply instead of an agent run. Internal events
(in-flight background completions) bypass the gate.
- hermes_cli/subcommands/pause.py: `hermes pause [--reason]` and
`hermes resume`, wired into main() and _BUILTIN_SUBCOMMANDS.
- hermes_cli/status.py: `hermes status` shows a PAUSED banner (one stat).
- tests/test_estop.py: 20 tests — sentinel lifecycle, reason surfacing,
log-once, cron skip + resume, kanban gate, gateway paused reply +
internal bypass, CLI idempotence, builtin-set parity, status line.
Never kills in-flight work; resumable with no restart. Footprint ladder:
CLI command only, no new model tool, no new env vars.
Ported from: gastownhall/gastown estop.go (MIT); related prior art:
#26778 (/panic — kill/exit semantics, deliberately different: ours is a
resumable pause), #44617 (interrupt in-flight cron — out of scope here).
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
The per-session agent cache is capped at 128 entries with a 1h idle TTL, and
neither bound knows how many bytes it holds. Each cached agent pins
_session_messages -- the full transcript including tool output, tens of MB on
a session with 100+ tool calls -- so a gateway serving many chats keeps every
warm transcript resident: agents that took a turn inside the TTL are never
idle-swept, and the idle sweep additionally defers finalizable sessions until
they expire. RSS climbs until the cgroup throttles and SIGTERM can no longer
flush inside systemd's stop timeout.
Add the missing bound. Each session-expiry watcher tick compares the process's
anonymous RSS against a budget and, when over, sheds LRU agents through the
same soft-eviction path the cap enforcer uses, then runs malloc_trim so the
freed arenas actually return to the OS. Evicted sessions rebuild their
transcript from the persisted session on the next turn.
Three classes of session are never shed: agents mid-turn, the most recently
used ones, and any session whose transcript has not finished reaching disk
(_last_flushed_db_idx vs len(_session_messages) -- the same divergence the FTS
write-corruption guard reacts to when it preserves live history).
memory_high_mb defaults to "auto", deriving the budget from the cgroup limit
the gateway runs under, so a MemoryHigh/MemoryMax on the unit is respected
without a second number to keep in sync. The two existing bounds become
configurable alongside it under agent.agent_cache.
protect_recent is clamped to half the cache: a couple of sessions can exhaust
the budget on their own, and a fixed MRU guard would then protect everything
and leave the gateway climbing with nothing it would shed.
Fixes#80764
Follow-ups on top of the salvaged #80696 fix (review findings):
- Sibling sites: rollback.restore, gateway /retry, CLI /retry and /undo N,
and both CLI resume turn counters now use is_user_originated_turn so
legacy-persisted standalone handoffs (durable role=user, no display_kind)
can never be truncation targets or counted as user turns (#80622
suggested regression 4, dispatcher-wide).
- Site-1 guard: hoist the api_call_count decrement + iteration-budget
refund above the break so a skipped turn no longer leaks a budget unit
and finalize_turn logs the true call count (matches the ollama early-exit
and the site-2 sibling).
- Site-2 guard: run the handoff guard BEFORE reanchoring so a restored
user ask is what the anchor lands on, not a stale pre-restore index.
- SUMMARY_PREFIX: add the mid-tool-loop carve-out the code-side guard
already implements, so a literal-minded model doesn't halt an in-flight
exchange after in-place compaction.
- Skip path returns a short compaction status instead of replaying the
previous turn's answer (finalize_turn would append it as a fresh
assistant row — duplicate prose in transcript and delivery).
The dedup fix in #80679 reduced ~1.6k sequential API calls to ~30,
but those ~30 calls were still sequential. Extract the per-base-ID
resolution into an async helper and run all lookups concurrently
via asyncio.gather. Turns a 6-15s serial block into <1s.
Follow-up to #80700:
1. _await_disconnect_step was missing the try/except CancelledError around
asyncio.wait() that _await_adapter_cleanup_with_timeout already has.
When the outer fatal-handler timeout cancels disconnect() mid-step,
asyncio.wait does NOT cancel its inner task — the task was orphaned
with no observer. Add the same cancel+detach+re-raise pattern.
2. _queue_retryable_fatal_platform omitted credential_claim/listener_claim
keys that all 3 startup-path queue sites include. These are consumed by
the multiplex reservation logic to prevent secondary profiles from
taking the endpoint while a primary is queued. Pre-existing latent bug
— now fixed since the extraction makes it trivial.
After a network outage the Telegram fatal handler could hang inside
disconnect() and never populate _failed_platforms, so the reconnect
watcher had nothing to retry and the process stayed permanently deaf.
Queue retryable platforms before any disconnect await, bound the fatal
handler with an outer detach deadline, and release the Telegram token
lock / PTB close steps with detach-on-timeout so recovery cannot stall.
Follow-up to the #80376 salvage:
- TurnLeaseToken.degraded is dead code since acquire() started raising
TurnLeaseTimeoutError: the only constructor site passes the default and
repo-wide there are zero external readers. Remove the field, its ctor
param, the repr segment, and the always-False guards in rebind()/release();
the class docstring's 'retained for compatibility' claim described
consumers that do not exist.
- Revert pure black-rewrap churn in test_config_env_bridge_authority.py and
test_turn_lease.py (hunks on functions this change does not touch), keeping
the functional Windows-env/encoding additions.
- Turn the default-value test into an invariant: config default must equal
gateway.turn_lease.DEFAULT_LEASE_WAIT instead of pinning the 1800 literal.
- Rewrap the dangling 'Released' comment line in gateway/run.py.
Verified: 20/20 targeted gateway tests, ruff clean; mutation check — with
gateway/turn_lease.py reverted to pre-fix main the module fails, restored it
passes.
The Matrix adapter built MessageEvent from inbound room events but dropped
the sender's MXID and display name on the event itself -- only 'source'
carried them. Other adapters (signal/slack/telegram/discord/mattermost/irc)
have the same gap; this PR fixes matrix and adds the supporting
top-level MessageEvent fields so the rest can follow.
Downstream effects for matrix specifically:
- gateway prompt assembly can now read event.user_name (or source)
without having to dig into source per platform
- reply context (reply_to_text / reply_to_author_id /
reply_to_author_name) is parsed from the inline > <@user:server> ...
Matrix fallback format before stripping, instead of discarded
- the gateway's existing [Replying to: "..."] renderer can now show
who the user was replying to (was always anonymous for matrix)
MessageEvent gains two optional top-level fields (user_id, user_name,
both default None) so non-IM producers (cron/webhook/autonomous) remain
unaffected. Source still carries the same values for callers that
already read from there.
Tests cover:
- non-reply message carries sender user_id/user_name on MessageEvent
- different senders (alice, bob) both propagate
- reply message carries reply_to_message_id + reply_to_text +
reply_to_author_id + reply_to_author_name, parsed from the
> <@carol:example.org> original question\n\nactual reply shape
- non-reply message does NOT spuriously set reply_to_* fields
Sibling matrix tests (148 across test_matrix*.py) remain green.
Authored by WintleChoung <cwt@users.noreply.github.com>
Salvaged from PR #80293.
Review follow-up for the salvaged #79881/#63963 stack: the shutdown
interrupt fires exactly once, but work can materialize AFTER that one
shot on BOTH sibling paths:
- a /v1/runs task admitted before the drain populates
_active_run_agents only once _create_agent returns
(queued-before-agent window);
- a _running_agents entry claimed as _AGENT_PENDING_SENTINEL is
promoted to the real agent by track_agent() on its own schedule,
after the one-shot walk skipped the sentinel.
Either way the settle loop waited on work nothing signaled, and the
turn went straight to the post-interrupt tool-subprocess kill — the
exact amputation the fix exists to avoid, in a rarer window.
If any work is still live when the settle loop exits, re-invoke
_interrupt_running_agents (which already skips sentinels and folds in
the API-server helper) so late-materializing agents on either path get
the cooperative interrupt. Regression test drives the real stop() path
with an accelerated loop clock and asserts exactly two interrupt
signals.
The shutdown drain ACCOUNTS for API-server work but never INTERRUPTS it.
`_drain_active_agents()` folds `_active_api_run_count()` into both its wait
loop and its `timed_out` verdict, while `_interrupt_running_agents()` iterates
`self._running_agents` only -- a dict no API turn ever enters, because the
API server owns its own agent lifecycle. `gateway/run.py` states the gap
against itself: "API-server / desk sessions have the same structural gap
(#63529)."
The user-visible result is that every gateway restart with a live API or
desktop turn burns the full drain timeout and then runs
`_kill_tool_subprocesses("post-interrupt")`, which amputates the turn's tool
subprocesses with no cooperative interrupt and no resume marker.
There are seven API agent-entry points. Six funnel through `_run_agent()`
(both session-chat routes, and `/v1/chat/completions` + `/v1/responses` in
streaming and non-streaming form) and are counted by `_inflight_agent_runs`;
the seventh, `/v1/runs`, runs its own lifecycle and is counted through
`_active_run_tasks`. None of the six has a run_id, so the run_id-keyed
`_active_run_agents` cannot reach them, and only two pass `agent_ref` -- which
lands in a caller-local list, not a registry.
So register once at the single unconditional creation site inside
`_run_agent`, beside the existing `_publish_turn_process_ownership()` call,
and unregister in the same `finally` that already clears it. That one
symmetric pair covers all six callers. The registry is adapter-owned and
keyed by object identity, kept separate from `_active_run_agents` because
that dict is run_id-keyed and scoped to the public `/v1/runs` stop API.
`interrupt_active_runs()` then walks both registries, deduped by identity, so
the interrupt set matches the set the drain waits on. The settle window after
the interrupt now polls API work as well: the interrupt is cooperative, and
without this the window closes the instant `_running_agents` is empty -- which
it always is for API turns -- and the tool kill lands on a turn that was asked
to stop microseconds earlier.
Sibling-site fix for #80216: yuanbao recall redaction also calls
rewrite_transcript() and was subject to the same archived-history
data loss when active_only defaulted to False. Pass active_only=True
at both yuanbao call sites — load_transcript only returns active
rows, so the redacted content is in the active set and the archived
pre-compaction history should survive the rewrite.
Also drops the stale 'callers that mean to purge (e.g. yuanbao
recall redaction) keep the default' note from the rewrite_transcript
docstring — no caller intentionally purges archived rows.
/retry truncates the live transcript to before the last user message
and persists it via SessionStore.rewrite_transcript, which calls
replace_messages() with the default active_only=False. That DELETEs
every row for the session, including the soft-archived
active=0/compacted=1 rows that in-place compaction keeps on disk
(#38763), so any /retry after a compaction permanently wiped the
archived history. #57803 named this call site as a residual gap after
its global-default approach was rejected; the TUI sibling was fixed
in #80195.
The handler now probes has_archived_messages() (new SessionStore
wrapper, auto-exposed through AsyncSessionStore) and passes
active_only=True when archives exist, so only the live rows are
replaced. rewrite_transcript gains an active_only parameter that
defaults to False, keeping the destructive semantics yuanbao recall
redaction depends on. Also corrects the rewrite_transcript docstring,
which still listed /undo as a caller even though /undo soft-archives
via rewind_session.
The regression test drives _handle_retry_command against a real
SessionStore and SessionDB seeded with archived compaction rows and
asserts the archives survive.
- gateway/config.py: rewrite the stale enablement-pass header comment that
still described check_fn as 'the single source of truth for are-my-env-
vars-set' / 'lazy-installs it' — both false under the new contract.
- teams: check_requirements docstring wrongly claimed credential checks
(body checks only SDK/aiohttp presence); derive install_hint from the
canonical LAZY_DEPS pins + sys.executable instead of hardcoding
'~/.hermes/hermes-agent/venv/bin/pip' and version pins (wrong under
HERMES_HOME overrides / profile installs; pins go stale on CVE bumps);
connect() fatal-error hints now point at the venv pip instead of bare
system pip (the PEP 668 trap the docs warn about).
- teams docs: drop exact version pins from the two manual-install commands
(LAZY_DEPS is the source of truth; unpinned installs still work and the
text can't go stale).
- hermes_cli/status.py: per-entry exception guard around check_fn so one
raising probe can't abort the listing of all remaining plugin platforms
(aligns with the other three call sites).
- tests: rename test_register_check_fn_is_active_lazy_installer ->
test_register_splits_passive_probe_from_active_installer (name said the
opposite of what it verifies).
PlatformEntry.check_fn served three contradictory roles: adapter-creation
gate, config auto-enablement gate, and status display. Plugins had to pick
one function for all three:
- Active installer as check_fn (discord/slack/telegram/matrix/dingtalk/
feishu): every status display could pip-install SDKs as a side effect
(the desktop 94% boot-loop class).
- Passive probe as check_fn (teams, wecom_callback): create_adapter()
returned None before connect() could lazy-install, so the SDK never
installed (#79812 deadlock; wecom_callback's platform.wecom_callback
LAZY_DEPS entry was dead code).
The split makes both call sites correct by construction:
- check_fn is now contractually PASSIVE (probe only, never installs).
- New optional PlatformEntry.ensure_deps_fn is the ACTIVE installer;
create_adapter() runs it exactly when check_fn is False — the platform
is enabled+configured and the gateway is about to connect it.
- Config enablement keeps a configured platform whose deps are missing
but installable; the install itself is deferred to create_adapter().
- Status surfaces (_platform_status, hermes status) read only the
passive probe and can never trigger pip.
Migrated all lazy-installable platform plugins to the split; platforms
with no optional deps (irc/ntfy/buzz/simplex/line/a2a/...) are unchanged
— no ensure_deps_fn means a False check_fn stays a hard block.
wecom_callback gains a working installer for the first time.
Builds on @xxxigm's #79812 (both commits cherry-picked with authorship
preserved), reworking the check_fn swap into the two-field split so the
Teams fix doesn't reintroduce install-on-status.
Pinned was capped at half the viewport by its own nested scroller, so past
roughly a dozen pins the rest were reachable only by scrolling inside a
scroller — a pin you have to go hunting for isn't doing its job.
Drop the cap and let the section grow into the sidebar's existing scroll,
and stop virtualizing Pinned: virtualization needs a bounded viewport to
measure against, which is exactly what's being removed. No count badge, no
"show more" — pin as many as you want and they all render.
Also back-fill pins on the API-server list route, which was the one list
path still windowing purely on recency.
PATCH /api/sessions/{id} only accepted title and end_reason, so the
`pinned` flag the desktop sends was rejected as an unsupported field —
and the client swallows that error. Pins lived in one app's localStorage
and never reached state.db, which also meant the server-side auto-archive
sweep was free to hide the chats a pin exists to keep.
Accept pinned and archived as booleans, route them to the SessionDB
setters that already existed, and include both in the serialized session
so clients can reconcile against server truth.
Forensics on a live developer machine found pytest fixture rows inside the
REAL ~/.hermes/state.db — sessions with chat_id 'chat-1', '123', 'wx-chat',
and gateway_routing rows whose scope was literally under /tmp/pytest-of-*/.
A pytest-spawned process also opened the live DB and flipped its journal
mode (journal_mode=DELETE fallback on SQLite 3.50.4) under the WAL-mode
gateway writer, destroying committed transcripts ("Persisted transcript
lagged live cached history ... possible FTS write corruption", 15+
occurrences). The existing live-system guard covers kill primitives but not
the SessionDB/SessionStore write paths.
Root cause (leak vector): the session-level HERMES_HOME sandbox in
tests/conftest.py only created a tempdir when HERMES_HOME was UNSET. On a
machine where the shell (e.g. gateway-launched, or an exported
HERMES_HOME=~/.hermes) hands pytest the production home, the sandbox was
skipped entirely — every argless SessionDB()/SessionStore() and every
collection-time DEFAULT_DB_PATH froze onto the real state.db.
Fixes (fail the class, one owner):
* hermes_state._ensure_test_isolation(): single choke point wired into
SessionDB.__init__ (every construction, incl. read_only). Under pytest
(PYTEST_CURRENT_TEST / PYTEST_VERSION — inherited by subprocess
children), a db path resolving to <real-root>/state.db or
<real-root>/profiles/<name>/state.db raises RuntimeError('live-system
guard: ...') before any connection, mkdir, or journal-mode pragma.
* tests/conftest.py: session sandbox now also tempdir-redirects a pre-set
HERMES_HOME that points at the production root (the actual escape
vector); kanban deny-list capture updated to match. New autouse
_state_db_write_guard fixture honors the existing
@pytest.mark.live_system_guard_bypass marker as the escape hatch and
feeds custom (non-~/.hermes) production roots into the guard deny-list.
* gateway/session.py: SessionStore.__init__ no longer swallows the guard's
RuntimeError into the JSONL fallback — guard trips are loud.
* tests/hermes_state/test_live_db_isolation_guard.py: behavioral
regression tests — production paths (direct, profile, read-only,
unnormalized, default-resolution) raise; tmp HERMES_HOME works; bypass
marker works; SessionStore re-raises guard errors but still degrades on
ordinary failures; subprocess child without HERMES_HOME is refused while
a hermetic child succeeds.
No new HERMES_* env vars; no hardcoded ~/.hermes (platform root comes from
hermes_constants._get_platform_default_hermes_home()).
Follow-up to c0d974b19 (#79741). Three review findings against that commit,
none of which change the escalation behaviour it shipped.
1. The recovery decision lived inline in `_handle_message_with_agent`, a
~2000-line async method, so the only way to pin it was a test that read
`inspect.getsource(...)` and asserted on substrings. AGENTS.md bans reading
source in tests outright and names this exact situation: "if the logic lives
inline in a god-file (gateway/run.py) and extracting it feels disruptive:
that's the actual signal to do the extraction, not to regex around it."
Those tests were not merely stylistically wrong, they were actively harmful.
One asserted the substring `_new_tokens < _approx_tokens` was PRESENT -- so
it passed while the gate had the bug that substring represents, and had to be
edited when the gate was fixed. It failed on correct code and passed on
broken code, in one assertion.
Extracted `hygiene_compaction_recovered()` as a module-level pure predicate
and replaced the three source-reading tests with eight direct unit tests.
The extraction immediately earned itself: the new tests caught a `NameError`
(the predicate called `compression_made_progress` while the module bound it
under an alias) that a source-text assertion cannot see, because the symbol
is spelled correctly in the source and only fails at runtime.
2. The gate inferred "did the transcript actually get rewritten" from a numeric
side effect -- the degenerate "did not rotate or compact in place" path
(#21301) reuses the pre-compression counts -- when the booleans
`_hyg_rotated` / `_hyg_in_place` were already in scope and explicitly set
False on that path. The predicate now takes them directly, so a future edit
that re-estimates instead of reusing the old counts cannot silently defeat
the escalation.
3. `_record_hygiene_cooldown` passed no `error` to
`record_compression_failure_cooldown`, which writes `compression_failure_error`
unconditionally -- so a hygiene failure clobbered to NULL whatever reason the
in-conversation path had recorded, and readers then show the user "unknown
error" (agent/manual_compression_feedback.py, gateway/slash_commands.py). The
reason was already in hand at both call sites. Pre-existing from #74136 but
amplified by escalation: a blank reason on a 45-minute cooldown is far more
user-visible than on a 5-minute one.
Also: the ladder docstring described the compressor's absolute 60/300/900s
ladder while the constant is multipliers (1, 3, 9); the config docs still
described `hygiene_failure_cooldown_seconds` as a flat interval rather than the
first rung of a capped ladder; and `PersistentState.hygiene_failure_streak` now
documents that it is process-local by design -- keying on `session_key` is what
survives compaction rotation, which the persisted `compression_*_streak`
columns cannot express since they key on the rotating `session_id`. Making it
durable is a schema change, tracked on #79624 rather than smuggled in here.
Also replaces the file's hand-written `_Runner` stub with
`object.__new__(GatewayRunner)` (already the idiom elsewhere in the same file).
The stub reimplemented `_session_state` and `_peek_session_state`, so the tests
exercised copies that could drift from production; using the real class
immediately made one assertion stronger -- on a fresh runner `_sessions` does not
exist at all until something materialises it, so the reset provably did not even
create the map.
A review pass on this follow-up then caught that the CALL SITE was still
unbound: deleting the whole `if not _hyg_aborted: if
hygiene_compaction_recovered(...)` block left every ladder test green, because
the unit tests prove the predicate correct without proving it is wired in. The
merged commit had the same gap and its only cover was the banned source-reading
test. `test_session_hygiene_forces_in_place_compaction_with_bound_session_db`
now spies the reset on a genuine in-place compaction, so deleting the wiring
fails. Two earlier attempts at this test did NOT close the gap -- asserting on
streak VALUES passes either way, since the streak is 0 whether or not the gate
ran; only a positive spy assertion on a recovering run detects the deletion.
Same pass also corrected an overstatement: point (2) is hardening, not a live
bug. The degenerate path also sets `_new_count = _msg_count` and `_new_tokens =
_approx_tokens`, and `compression_made_progress(n, n, t, t)` is always False, so
the merged code already declined to reset there. A 200k-trial fuzz over the
reachable state space found zero behavioural disagreements between the merged
gate and this one. The guard's value is surviving a future edit that stops
reusing those counts.
Tests: 28 in tests/gateway/test_hygiene_failure_cooldown_ladder.py (8 new unit
tests for the predicate, 3 for reason forwarding, 3 source-reading tests
deleted). All 5 mutations caught -- including one that restores the hand-rolled
comparison and one that removes the rotated/in_place guard. Two mutations
initially SURVIVED and exposed vacuous tests of my own: the no-rewrite test used
counts the progress predicate already rejects, so it passed without binding the
guard at all; it now passes counts that read as progress on their own, proving
the guard is what rejects them. gateway hygiene + session-state + agent
compression-progress suites: 54 passed; ruff clean.
Refs #79624
/refine [focus] fires the existing background review fork
(AIAgent._spawn_background_review) immediately instead of waiting for
the automatic 10-turn memory / 10-iteration skill nudge counters.
Optional focus instructions are appended to the review prompt so the
fork prioritizes what the user asked for (e.g. '/refine save the
deploy workflow as a skill').
- New optional focus parameter threaded through
_spawn_background_review -> spawn_background_review_thread.
Automatic post-turn reviews pass None and their prompts are
byte-identical to before.
- CLI handler snapshots conversation_history; gateway handler pulls
the idle session's cached AIAgent from _agent_cache (rejected while
the agent is running).
- Review runs in a daemon thread against the snapshot — live
conversation, message alternation, and prompt cache untouched.
- Slack stays under the 50-slash cap via /hermes refine.
Adapted from the /refine concept in Prime Intellect's Prime-Agent
(Continual Harness); Hermes' equivalent durable state is the
memory + skill stores, so the review fork is the natural target.
Teknium
kshitij
Victor Kyriazakos
Ken Weiner
HexLab98
kinsolee
HenryG
gnanirahulnutakki
Gille
Dominic Bejar
kyssta-exe
poisdahl
texasich
WintleChoung
kshitij
briandevans
dsad
Adolanium
Brooklyn Nicholson