- Widen the scheduler-internal timeout classification to the sibling
TERMINAL_CWD lock-wait TimeoutError (#79768), which also matched the
generic 'timed out' branch and was delivered as a provider timeout.
- Reconcile the drift-guard alert with #72056's lifecycle-aware
remediation: finite one-shots are told to recreate the job, not to
update a consumed one.
- Scrub environment-specific references from comments/docstrings.
The generic failure summarizer caps unrecognized errors at 180 chars,
which cut the drift alert off mid-sentence before the pin command. The
drift branch now formats its own delivery from the guard's full message,
so the one alert the operator gets actually contains the fix.
A fleet-wide inference config change previously produced one 'Skipped to
prevent unintended spend' alert per unpinned job per tick — 40 jobs meant
40 alerts every tick until each was re-pinned (Coatue field report,
2026-08-11). The #44585 guard now reuses the #73506 alert-once shape the
preflight path already established: a persisted drift_alerted bit on the
job record, a [drift_skip:silent] marker on repeat ticks that suppresses
delivery, and the bit clears on the next successful run so a future drift
re-alerts. Only the drift branch consults the bit — every other failure
keeps alerting per tick.
The alert text also now says it is sent once, so operators know the job
stays skipped silently until pinned or restored.
A cron that dies on a provider timeout with no fallback chain configured
now tells the operator exactly how to fix it: `hermes fallback add` for a
personal chain, or the cron.model + cron.model_provider fleet defaults for
operator-managed fleets. The exhausted-chain branch stays terse — the chain
is intact there and no config command applies.
Field-reported: users hitting the empty-chain failure could not self-serve
from the alert text alone.
_summarize_cron_failure_for_delivery() unconditionally said 'Fallback
chain was exhausted or unavailable.' on every provider failure, even
when fallback_providers is empty (the default -- confirmed empty on
both the root and cto profile config.yaml). That phrasing implies a
fallback was attempted and failed, which sent the operator debugging
the wrong thing.
Add _fallback_chain_phrase(): reads the effective chain via
get_fallback_chain(load_config()) and returns 'No fallback chain
configured.' when it's empty, or the original wording when a chain
exists. Fails open to the original wording on any config read error.
The scheduler's own inactivity-watchdog mislabeling (idle-timeout
reported as provider timeout) was already fixed in a prior commit on
this branch; this closes the second half of t_29b8da55.
Data pull requested by the task (grep errors.log across profiles +
root for 'Provider has been unresponsive' + model=, 2026-07-21 to
2026-08-06): 9 stall events total, 5 on claude-sonnet-5, 4 on
claude-haiku-4-5, spread across 6 different cron jobs. No material
haiku-specific instability -- sonnet-5 stalls at least as often on the
cron path in this sample. Reporting per acceptance criteria; not
worth a routing change on this evidence.
Follow-ups to the salvaged #84009 commits:
- Add 'session_switch' to _RESET_END_REASONS: a reset continuation's
parent can be promoted to session_switch (resume the reset parent,
then switch away), which permanently hid pre-marker legacy children —
reopen-time stamping cannot rescue them because the parent is being
ended, not reopened. Probe-verified before/after.
- Share the legacy reset-child heuristic via _legacy_reset_child_sql()
so _RESET_CHILD_SQL and reopen_session()'s stamping UPDATE cannot
drift, and derive find_latest_gateway_session_for_peer's two recovery
fence literals from _RESET_END_REASONS_SQL (was a third hand-written
copy of the same set).
- Exclude reset children (marker + legacy shape) from the
resolve_resume_session_id forward walker: resuming a reset parent
could redirect into the post-reset conversation — the exact context
the user reset away. Regression tests cover both shapes plus the
walker's original compression-tip behavior; mutation-checked.
The previous gate compared session.user_peer_id against a fresh
_resolve_user_peer_id() call on the same manager. Both values come from
the same resolver with the same inputs, so a non-owner triggering a new
session in a shared channel passed the check and received the owner's
MEMORY.md/USER.md under their peer.
The owner is now a config fact: _declared_owner_peer_id() returns the
sanitized peerName, and migration runs only when the session's user peer
is that peer. Without a declared peerName, migration runs only when no
runtime gateway identity is present (the single-operator CLI path).
Aliases still work: a platform ID mapped onto peerName resolves to the
owner peer before the comparison.
Tests now derive each session's user peer from the real resolver instead
of hand-picking mismatched ids, so the non-owner test fails against the
old gate.
The owner gate from #82038 compared against config.peer_name directly,
which is None for most single-user setups — sanitizing None would raise
and the gate never accounted for pinned/runtime/aliased identities.
Resolve the owner the same way sessions do, and add the non-owner skip
regression test the original PR shipped without.
Co-authored-by: menhguin <menhguin@users.noreply.github.com>
sync_turn called manager._flush_session() directly, which flushes
synchronously every turn no matter what writeFrequency says — the
"async", "session", and every-N-turns modes were dead configuration
on the main turn path. Route through save(), the dispatcher that
actually implements those modes.
Same bug class reported in #19650 (starship-s) and #72708 (Diaspar4u);
this takes the minimal one-line routing fix without their broader
lifecycle refactors.
Co-authored-by: starship-s <45587122+starship-s@users.noreply.github.com>
Provider shutdown() only called manager.flush_all(), which drains the
queue but never joins the async-writer thread — manager.shutdown()
exists and nothing called it. The writer thread could still be blocked
in httpx I/O at interpreter exit (the #37632 crash class). Now
shutdown() calls manager.shutdown() (flush + join) when persistence is
enabled, and a new manager.stop_async_writer() (join only, no flush)
when saveMessages is false, so containment and clean teardown compose.
Salvages #67559 — original gated sync_turn/on_memory_write/on_session_end but missed shutdown(), whose flush_all() still persisted on exit. hermes-sweeper review (salvageability=high) flagged this as the one gap.
Guard sits after the worker-thread joins, not at the top: cleanup is independent of persistence, and a top-of-method return would leak _prefetch_thread/_sync_thread. Adds TestShutdown and clarifies the saveMessages=false README row.
Credit @Matroskin86 (original PR author).
The saveMessages knob has been parsed by HonchoClientConfig since its
introduction but was never consumed: sync_turn, on_memory_write and
on_session_end persisted to Honcho regardless. With saveMessages=false the
provider now never writes automatically (raw turns, memory-write conclusion
mirroring, session-end flush) while read/tools paths stay fully functional.
Guard uses getattr with a True default so legacy/injected configs keep the
old behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018giroL5zeMPnPxERxAxXHY
Two silent-auth-failure paths from #36098 (also #66125):
- the local-URL guard only escaped the 'local' placeholder when the
HOST BLOCK had apiKey. A top-level apiKey in honcho.json — explicit
user intent, and what 'hermes honcho setup' writes for single-host
configs — was dropped on the floor, so AUTH_USE_AUTH self-hosts
401'd on every request. Now any explicit key in honcho.json (host
block or top level) is honored; only env-sourced keys are still
treated as likely-cloud and skipped for local URLs.
- named-profile host blocks do not inherit the default host's apiKey
(credential isolation is by design), but the failure was silent:
the profile ran unauthenticated and every tool said 'no context'.
Affirm isolation and warn loudly at config-resolution time instead,
the outcome #66125 proposed if inheritance is rejected.
_all_profile_host_configs() built per-profile host keys inline as
f"{HOST}.{profile}" ("hermes.work") while profile_host_key() — used by
honcho status/enable/sync and the runtime memory plugin — produces the
underscore form ("hermes_work"). The lookup always missed, so
'hermes honcho peers' showed "(not set)" / leaked the raw malformed key
into the AI-peer column for every non-default profile. Profile names
needing sanitization (dots/spaces) were doubly broken.
Verified live: with hosts["hermes_work"] populated, cmd_peers showed
'work ... hermes.work' before the fix and 'work ... hermes' after.
Tests: host keys match the writer form, sanitized profile names resolve,
peers output shows populated identities with no key leak, and clean
fallback for profiles without a block.
Salvage of #2757 by @teyrebaz33 — rebased onto current Honcho plugin layout.
Stray control characters (e.g. terminal escapes pasted into HONCHO_BASE_URL
or config baseUrl) are dropped with a warning so SDK construction cannot
crash startup on Invalid non-printable ASCII character errors.
Adds the regression test #37671 shipped without (dot-form legacy host
block must keep its explicit apiKey on local base_urls instead of
silently degrading to the 'local' placeholder and 401ing every write),
its inverse (no host key -> placeholder), and an invariant test pinning
the full resolution order the three adopted fixes compose into:
host block > endpoint.baseUrl > flat root > HONCHO_BASE_URL > HONCHO_URL.
HonchoClientConfig.from_global_config() only consulted top-level
baseUrl / base_url / HONCHO_BASE_URL in ~/.honcho/config.json. The
Honcho SDK's native config format — and what Claude Desktop writes —
nests the URL at endpoint.baseUrl. Users with that config format had
their self-hosted Honcho container silently ignored: every honcho_*
call routed to https://api.honcho.dev with a workspace_id that does not
exist there, so tools returned empty data with no error anywhere.
Resolution order in from_global_config(), highest first:
1. endpoint.baseUrl (SDK-native, what Claude Desktop writes)
2. baseUrl / base_url (root-level, existing behavior)
3. HONCHO_BASE_URL (existing env var)
4. HONCHO_URL (the SDK's own env var, honcho/client.py:234)
HONCHO_URL is also read in from_env(). from_global_config() delegates to
from_env() whenever the config file is missing or unreadable, so an env
fallback wired into only one of the two would silently do nothing for
users with no config file.
A non-dict endpoint value falls through cleanly rather than raising.
Existing users are unaffected — the new sources are consulted only when
the existing ones resolve to None.
The INFO log for the base_url-unset case now says so explicitly instead
of printing only the host. The SDK resolves that case from its own
ENVIRONMENTS map (honcho/client.py:36-39), which for environment=
production means the public cloud; a self-hosted user whose config was
not picked up otherwise sees a healthy-looking startup line.
Closes#43800.
Drives the real resolution chain against real honcho.json files under
temp HERMES_HOMEs with the same ContextVar override the multiplexer and
dashboard use. Pins:
- #69123's minimal repro: two profile scopes get distinct clients with
their own workspaces and bearers
- the daemon-thread case: a bound config acquires its profile's client
from a thread that cannot see the ContextVar, and
spawn_context_thread carries the override where a plain Thread
(control test) does not
- credential identity: account swap on the same path/host creates a new
client and EVICTS the old slot; the OAuth fingerprint survives
access-token rotation but changes on re-auth; timeout changes rebuild
via the key
- provenance capture and its stability outside the profile scope
Two-profile repro shape from #69142 (NaMinhyeok); scenario set extends
the multiplex isolation tests from #81401 (angel12).
Co-authored-by: NaMinhyeok <NaMinhyeok@users.noreply.github.com>
Co-authored-by: angel12 <angel12@users.noreply.github.com>
Ports the test coverage from PR #29496 onto the salvaged implementation:
adapter-level serialization/workspace isolation/disconnect sealing, and
gateway-level ID-correlated concurrent duplicate tools plus the editable
text fallback when the native stream fails.
Slack's Agents & AI Apps feature ships a native streaming surface that
renders a live-typing message instead of the edit-based progressive
updates the adapter used until now.
The adapter now implements the existing draft-streaming interface:
- supports_draft_streaming() opts in whenever the app is connected and
native streaming hasn't been detected as unavailable.
- send_draft() starts a stream on the first frame (chat.startStream,
anchored to the resolved thread_ts, with recipient_team_id/user_id
for channel streams) and appends only the delta on subsequent frames
(chat.appendStream is append-only). The consumer's trailing cursor
glyph is stripped before delta computation.
- Unlike Telegram drafts (ephemeral, replaced by a real sendMessage),
a Slack stream IS the final message. send() therefore intercepts the
turn-final delivery for a chat with an active stream whose streamed
text is a prefix of the final content, and seals it via
chat.stopStream with the remaining delta instead of posting a
duplicate. Rich Block Kit (when enabled) is applied to the sealed
message via chat_update, mirroring the finalize path in edit_message.
- Feature-gate errors from chat.startStream (not_allowed,
missing_scope, unknown_method, ...) are cached on the adapter so
subsequent runs skip straight to edit-based streaming with a single
warning naming the fix (enable Agents & AI Apps for the app);
transient errors only disable drafts for the current run via the
consumer's existing send_draft failure handling.
- Segment breaks (new draft_id) and disconnect() seal any open stream
so chats are never left with a dangling live-typing indicator.
No consumer or config changes: streaming.transport auto/draft now
lights up native streaming on Slack through the same interface
Telegram drafts use, and the edit-based path remains the fallback.
Follow-ups on salvaged #81419:
- Honor the plugins.enabled allow-list / plugins.disabled deny-list (same
opt-in contract as the general PluginManager) — installed != loaded.
- Skip callables that require arguments: general plugins share the
hermes_agent.plugins group with register(ctx) targets; invoking them
zero-arg would TypeError-spam every startup.
- Fix test docstring (entry points are discovered FIRST, lowest precedence)
and docs mechanism wording; document the config gate.
- New tests: opt-in gate, deny-list, register(ctx) never invoked.
E2E-verified with a real pip-built package against a temp HERMES_HOME.
Model-provider discovery was filesystem-only (bundled dir, $HERMES_HOME,
legacy providers/*.py). The general PluginManager scans the
hermes_agent.plugins entry-point group but deliberately does NOT import
kind=model-provider manifests (providers/ owns their lifecycle), so a
pip-installed provider was recorded yet never called register_provider() —
it never appeared in the picker, contradicting the 'Distribute via pip' docs.
Add a _discover_entry_point_providers() step that scans the
hermes_agent.plugins group and imports each entry, supporting both a
module:func callable target and a bare self-registering module target.
- Runs BEFORE filesystem plugins (lowest precedence): last-writer-wins means
bundled/$HERMES_HOME profiles always override a pip provider of the same
name, so a third-party package cannot hijack a first-party provider id.
- Per-entry failures are isolated (logged + skipped), so one broken package
can't break discovery.
- Docs updated to describe the real mechanism; tests cover callable + module
targets, failure isolation, and first-party precedence.
Spill files (terminal overflow, hook context, web_extract full text,
subagent summaries) were written with plain open()/write_text into
predictable directories. A pre-planted symlink at any of those paths
redirected the write onto an arbitrary user-owned file, and raw
pre-redaction terminal/hook spills landed world-readable under the
default umask.
New tools/spill_safety.py helpers create files with
O_CREAT|O_EXCL|O_NOFOLLOW (a link-shaped path fails the write instead of
following it) and overwrite via lstat-checked unlink + exclusive
re-create, so even the redaction rewrite cannot be diverted. Private
tier (0o700 dir / 0o600 file) covers raw terminal and hook spills;
cache/web and cache/delegation keep umask perms because those dirs are
bind-mounted into remote backends that must read them.
Pattern borrowed from DeepSeek Harness dsh-spill-local (MIT):
private root + exclusive owner-only opens for spill artifacts.
save_context_length() and _invalidate_cached_context_length() did an
unguarded read-modify-write into $HERMES_HOME/context_length_cache.yaml.
The plain `open(path, "w")` truncates the file before the dump runs. If
the process is killed mid-dump, the file is left empty or partial. The
next _load_context_cache() swallows the YAML error and returns {} —
silently wiping every persisted context length. A concurrent process
reading between truncate and dump-complete also sees a torn file.
After the cache is lost, every model re-probes the network, and when a
probe fails it falls back to the generic 256K default — so a user on a
1M-window model ends up with a wrong, short context window.
Hermes routinely runs several processes against one shared $HERMES_HOME
(a cron agent plus an interactive session, multiple gateway sessions),
so this is hit in normal use.
Switch both writers to the existing utils.atomic_yaml_write helper
(temp file + fsync + os.replace, symlink- and mode-preserving). The real
file is only ever swapped from a fully written temp file, so an
interrupted write leaves the previous cache intact and readers never see
a partial file. Matches the atomic-write pattern already used for
auth.json, config.yaml, and other persisted state.
Makes the persistent model context-length cache write crash-safe. The
old non-atomic write could truncate or wipe the entire cache on an
interrupted or concurrent write, which then forces models onto the wrong
fallback context window. The fix routes both cache writers through the
repo's atomic temp-file + os.replace helper.
N/A
- [x] 🐛 Bug fix (non-breaking change that fixes an issue)
- [ ] ✨ New feature (non-breaking change that adds functionality)
- [ ] 🔒 Security fix
- [ ] 📝 Documentation update
- [ ] ✅ Tests (adding or improving test coverage)
- [ ] ♻️ Refactor (no behavior change)
- [ ] 🎯 New skill (bundled or hub)
- `agent/model_metadata.py`: `save_context_length()` and
`_invalidate_cached_context_length()` now write via
`utils.atomic_yaml_write` instead of a truncating `open(path, "w")`.
Added the `atomic_yaml_write` import.
- `tests/agent/test_model_metadata.py`: added
`test_write_failure_leaves_existing_cache_intact` — simulates a crash
during the atomic swap and asserts the existing cache survives
byte-for-byte with no stray temp file.
1. `pytest tests/agent/test_model_metadata.py -q` — 98 pass, including
the new crash-safety test.
2. The new test seeds a valid cache, forces the swap step to raise, and
confirms the file is not truncated and no `.cache_*.tmp` is left.
3. `ruff check agent/model_metadata.py` passes.
- [x] I've read the Contributing Guide
- [x] My commit messages follow Conventional Commits (`fix(scope):`, etc.)
- [x] I searched for existing PRs to make sure this isn't a duplicate
- [x] My PR contains **only** changes related to this fix
- [x] I've run the affected tests (`pytest tests/agent/test_model_metadata.py -q`) and they pass
- [x] I've added tests for my changes
- [x] I've tested on my platform: macOS 15 (Darwin 25.5)
- [x] I've updated relevant documentation (README, `docs/`, docstrings) — or N/A
- [x] I've updated `cli-config.yaml.example` if I added/changed config keys — or N/A
- [x] I've updated `CONTRIBUTING.md` or `AGENTS.md` if I changed architecture or workflows — or N/A
- [x] I've considered cross-platform impact (Windows, macOS) — the helper uses os.replace, which is atomic on both
- [x] I've updated tool descriptions/schemas if I changed tool behavior — or N/A
_resolve_thread_id() falls back to _last_inbound_thread[chat_id] when no
explicit thread is present. That fallback exists for interactive DMs, where
Google Chat spawns a fresh thread per top-level user message and the adapter
drops thread_id to keep the session key stable. It also fired for cron
deliveries, which carry job_id in their metadata but no thread: the output
landed as a reply inside the last inbound thread instead of starting a new
top-level message.
Bypass the _last_inbound_thread fallback when metadata has a job_id (i.e. the
message is an automated cron delivery), so cron output posts at top level
unless an explicit thread is requested.
_inherit_notify_subs (link_tasks / triage-decompose / create-parents path)
copied only platform/chat/thread/user/profile, dropping chat_type,
user_id_alt, delivery_mode, and delivery_metadata. A DM-originated child
completion then fell back to chat_type='group' and woke a fresh
group-scoped session instead of the originating DM; Telegram DM-topic subs
lost their persisted reply-fallback metadata (issue #73030).
Consolidates the duplicated inline inheritance block in create_task onto
the single-owner helper — one inheritance path, every column, ONE owner.
Sabotage-verified regression tests for both the link_tasks and
create-with-parents paths.
The NS-570 epoch stamp clears a drain marker that survives a machine
restart — but it assumes every drain-gated action ends in a restart. When
a maintenance action completes WITHOUT recreating the container and the
writer never cancels the drain, the orphaned marker still carries the
current epoch, so the 1s drain watcher honours it forever and the gateway
bounces every inbound message with the 'draining for a maintenance
action' text (observed in the field: a Hermes Cloud instance refused all
Telegram turns for ~3 days).
The marker already records requested_at; now the readers check it. A
marker older than DRAIN_REQUEST_MAX_AGE_SECONDS (1h) reads as stale in
drain_requested() and drain_notification_suppressed(), with a loud
warning log. Leniency mirrors the epoch check: a missing or unparseable
timestamp still reads as drain-active (fail-safe toward quiescing), and
a legitimately long drain keeps a sanctioned keep-alive — re-calling
write_drain_request() refreshes requested_at.
Fixes#85433
An empty/blank model id reaching get_model_context_length() can't be
meaningfully resolved — and it's worse than a miss: the endpoint
metadata fuzzy matcher ('model in key or key in model') is vacuously
true for "", so it matches an ARBITRARY catalog entry from the live
/v1/models response and returns whatever context length that entry
happens to have, persisting it under a junk '@<base_url>' cache key.
This started failing CI on main when the Nous portal catalog changed:
tests/run_agent/test_primary_runtime_restore.py constructs agents with
model='' against the live portal URL, the arbitrary match now lands on
a 32K entry, and init_agent raises the 64K-floor ValueError
(test_allowed_for_nous_anthropic_messages, red on every PR's slice).
Guard early: a blank model id falls back to DEFAULT_FALLBACK_CONTEXT
immediately, before any cache write or network probe.
Salvaged from #65515 by @whirmill (rebased onto current main; the
guard now sits after the malformed-base_url normalization added since,
and carries an explanatory comment for the fuzzy-match footgun).
Fixes the red slice on #85444, #85452 and every other open PR.
Co-authored-by: whirmill <5079591+whirmill@users.noreply.github.com>
Before delivery_mode existed the notifier woke unconditionally when the task
carried a session_id — pre-existing gateway subscriptions had de facto active
wake. The column's 'notify' default alone would silently disable that on
upgrade. Backfill gateway rows to notify+wake on first-add only (tui stays
notify); explicit user downgrades are never overwritten by re-migration.
Sabotage-verified regression tests included.
Salvage of #37865 by @verybigdog. Adds delivery_mode (notify / notify+wake / wake)
on kanban notify subscriptions, persists chat_type + user_id_alt so a woken turn
reconstructs the creator's real session key, inherits the return path to child
tasks, and keeps wake out of the model-exposed send_message schema.
Original commits were authored under a local placeholder identity
(hermes-agent@users.noreply.local); re-attributed to the contributor's
public email.
Third lane of the same contract (found in live staging validation):
/sethome run as a top-level relay-fronted Slack DM message captured the
adapter's session-keying thread stamp (the /sethome message's own id)
into the persisted HomeChannel.thread_id and its legacy env mirror.
Every bare-platform delivery (deliver="slack") then resolved home chat +
home thread and landed inside the ephemeral thread around the old
/sethome message. Extracted _home_thread_from_source with the same
synthetic-stamp recognition as cron origin capture; a /sethome run
inside a genuine thread keeps that thread as the home target. Users
repair an already-poisoned home target by rerunning /sethome.
Bug 1: relay-fronted Slack in thread-per-message mode stamps each top-level
message's own id as source.thread_id (session KEYING, native thread_ts
parity). Cron origin capture persisted that stamp as durable routing, so
every delivery landed inside the ephemeral thread spawned around the
creation message instead of the top-level conversation. Fix at the source:
_origin_from_env drops a Slack thread id equal to the creation message's
own id (genuine in-thread creations keep theirs). Fire-time repair for
already-persisted jobs: deliver=origin and the explicit-target Slack
re-attach treat an origin thread as stale when the origin chat is the
configured Slack home chat — top-level (or the home target's configured
thread) wins; non-home working threads are preserved.
Bug 2: _preflight_check_delivery and cron_delivery_targets validated
deliver prefixes against get_connected_platforms(), which only sees
natively configured platforms — a relay-only deployment ({relay}) rejected
'slack:CHAT' with 'no gateway credentials configured' although fire-time
routing (resolve_delivery_transport + RelayAdapter.fronts_platform)
delivers it. New gateway.relay.relay_fronted_platforms() (env-derived from
GATEWAY_RELAY_PLATFORMS — the same source that seeds the live adapter's
identity set, so validation and routing cannot disagree) is unioned into
the connected set when the relay is connected. Native topologies keep the
strict credential check unchanged.
1. Use open_credentialed_url() instead of bare urlopen() in
templates.py apply_template() and probe_existing_customization().
Both send Authorization: Bearer headers; bare urlopen forwards
credentials on cross-origin redirects. The codebase has
open_credentialed_url() in hermes_cli/urllib_security.py that
strips credentials on cross-origin redirects — used by 4 other
modules.
2. Guard unavailable_reason() with the dedup set check before
calling it. The gateway builds a fresh AIAgent per message, so
without this guard unavailable_reason() (which calls _load_config()
→ stat + file read + JSON parse, and _check_local_runtime() →
importlib probes) runs on every gateway turn for an unavailable
provider, even though the warning is deduped after the first.
3. Move INDICATOR_GLYPH from Hindsight's eye emoji to a generic
brain (🧠) in core (agent/memory_provider.py). Hindsight overrides
with its own _HINDSIGHT_GLYPH (👁️) in recall_status() and
_emit_saving_indicator(). Other memory providers no longer inherit
Hindsight's brand mark as the default glyph.
Bundles previously-separate Hindsight/memory PRs into a single review surface:
- opt-in synchronous recall (recall_sync) — recall the injected memory in-turn instead of next-turn prefetch (#5820)
- actionable error when local_embedded runtime is missing — tells the user which package to install (#7718)
- default retain_source to 'hermes' so every stored memory self-identifies its provenance
- offer a starter memory template during hermes memory setup, plus warn before overwriting an already-configured bank
- warn when a configured memory provider reports unavailable (#2765)
- deterministic 'recalled N memories' recall indicator — Hermes itself emits a status line when auto-recall injects memory
- 'saving to memory' retain indicator — emitted the moment a turn is dispatched to the writer
Authored by @benfrank241 (ben.bartholomew@vectorize.io).
Salvaged from PR #74379.
notify_session_compacted closed the old session scope immediately on a
legacy rotating compaction. A compaction can complete while a turn is
still live on the old session; closing then pops the session scope under
the live turn scope, violating the stack's LIFO order — the exact
invariant the rest of the segmentation feature protects.
Now: when the old session has an active turn, set close_pending instead;
that turn's end_turn consumes the flag after its own turn scope pops and
it unregisters from the active-turn table. Sabotage-verified: the new
test fails without the fix.
Continuous gateway sessions keep the Relay session scope open for days;
close-driven export means the session root span and out-of-turn marks
never export until /new or idle-end, and a crash loses the open segment
entirely.
Opt-in segmentation (both defaults OFF => scope lifecycle byte-identical
to today):
gateway.telemetry.session_segments.on_compaction: false
gateway.telemetry.session_segments.max_turns: 0
Rotation closes the current session scope and pushes the next segment
(same session_id attribute, plus hermes.session.segment=N and
segment_reason=compaction|max_turns) ONLY at a turn boundary in
begin_turn — never mid-turn (scope stack is LIFO). Compaction completion
just flags rotate_pending (observer semantics, nothing on the compaction
critical path); legacy rotating compaction closes the orphaned old
session scope so its segment exports. Both native calls ride the
existing bounded scope-op executor: a wedged rotation costs one segment
span, never the agent. Segment bookkeeping advances even on native
failure so a degraded rotation cannot retry every turn.
Follow-up fixes from /hermes-pr-review + /simplify-code on PR #83437:
1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True)
— the plugin's 11-pattern list was a strict subset of the 50+ patterns in
agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens,
HuggingFace tokens, DB connection strings, and Telegram bot tokens would
all leak through the plugin's list but are caught by the existing redactor.
Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py.
2. Remove dead 'not isinstance(client, object)' check in on_session_finalize —
always False for any Python value.
3. Fix MoAClient.last_reference_metrics() to call the public
self.chat.completions.last_reference_metrics() instead of reaching into
the private _last_reference_metrics attribute via getattr.
4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass
pre_coerced=input_messages to _messages_for_langfuse_input to avoid
double-coercion + double _capture_content serialization per API request.
5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py
for consistency with the other HERMES_LANGFUSE_* env vars.
6. Fix test_sanitized_mode_redacts_secrets test data — the old samples
('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too
short to match the regex thresholds and never actually tested redaction.
Updated to realistic-length secrets and changed assertions to check that
the output differs from input (redact_sensitive_text masks rather than
inserting the literal string 'REDACTED').
Salvaged from PR #83437 by @erosika, with adopted fixes from @bgodlin (#81054),
@aldoeliacim (#82332), @nftpoetrist (#42326), @rodboev (#39653), @FnExpress
(#64292, supersedes #32175 by @db-aeon), @Per0-1 (#61166), @NaMinhyeok (#64797),
and @liuhao1024 (#43130).
Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two
attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes
8 prior community PRs with interaction-fix follow-ups.
Model attribution: on_pre_llm_request and on_post_llm_call now prefer the
wire value (request body model, response model) over the agent attribute,
which goes stale after /model switch or provider fallback.
Cost total: both cost paths now send a summed total alongside the per-type
breakdown, since Langfuse does not derive calculatedTotalCost from
cost_details keys. Subscription-included routes send no cost keys at all.
New coverage: api_request_error closes failed generations with ERROR level;
on_session_finalize/on_session_end close dangling traces for tool-only and
interrupted turns; subagent_start/subagent_stop trace delegated children as
spans; MoA advisor fan-out emits one generation per advisor priced at the
advisor's own model.
Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default
sanitized). Sanitized mode redacts secret patterns before truncation.
Adopted lifecycle fixes: shutdown client at session finalize when
reason=shutdown (not on session rotation); atexit finalizer ends open root
spans for short-lived processes; root context manager exited to prevent
interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock;
reasoning_content surfaced in traces; system prompt included in generation
input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io.
Closes#29482, #43129, #72661.
Supersedes #81054, #82332, #42326, #39653, #64292, #32175, #61166, #64797, #43130.
Partially addresses #67544 (capture modes + secret redaction; user_id remains open).
The custom + explicit_base_url branch of resolve_provider_client()
unconditionally rewrote a trailing /anthropic to /v1 via
_to_openai_base_url(), even when api_mode was anthropic_messages. The
Anthropic wrapper then never saw the real /anthropic path, so auxiliary
tasks (title generation, compression, vision, web_extract,
session_search) hit .../v1/chat/completions on a Messages-only endpoint
and failed.
Guard the wrap base on api_mode: for anthropic_messages, pass the raw
/anthropic base to _wrap_if_needed (which builds the Anthropic wrapper),
while the plain OpenAI client keeps the /v1-rewritten base so the
OpenAI-wire fallback (used when the anthropic SDK is unavailable) never
lands on /anthropic/chat/completions.
Refs #16254
Review on #74658 flagged that the response-shape suite exercised identity,
ephemeral and upload paths but left two changed call sites untested:
- create_handoff_thread's seed-message ts (adapter.py:2262), which anchors
every subsequent handoff send onto the thread;
- the standalone media branch's chat_postMessage reads (adapter.py:8721 text
post, :8749 caption fallback), where an SDK-shaped reply used to drop the
ts and report a caption-only delivery as 'nothing deliverable'.
Both new cases run against the hand-rolled stand-in and the real
AsyncSlackResponse. Verified they fail against the pre-fix adapter.
Co-authored-by: Junie <junie@jetbrains.com>
Slack Web API calls return `SlackResponse`/`AsyncSlackResponse`, which are
mapping-like but not `dict` subclasses, so every `isinstance(resp, dict)`
gate took its "unexpected shape" branch at runtime: user and channel names
collapsed to raw IDs, every user resolved as a non-bot (defeating the
allow_bots loop guard), ephemeral replies were reported as failures, and
uploads/caption fallbacks lost their message_id.
Normalize responses through a single `_slack_response_payload()` helper
(dict passes through, SDK response yields `.data`, anything else yields
`{}` so callers keep their fallbacks) and use it at every call site.
Existing Slack tests injected plain dicts, which is why the defect was
invisible; the new tests run each behavioral case against a real
`AsyncSlackResponse` as well.
Replaces the per-model _model_name_suggests_grok_4_3/_grok_4_6/
_minimax_m3 stale-cache predicates with one generic
_stale_pre_catalog_cache_entry() guard driven by
_PRE_CATALOG_STALE_KEYS. A cached context length is dropped when the
model resolves (longest-key-first, same as step 8) to a listed catalog
key and the cached value is at or below what the old resolution path
could have produced (largest shorter matching catch-all, or the 256K
fallback).
Also covers qwen3.6-plus, grok-4-fast, and grok-4.20 (the models
PR #37684 requested guards for), absorbing that PR.
_model_name_suggests_minimax_m3 is kept for its two non-cache callers
(models.dev underreport guard, cache-control gating in
agent_runtime_helpers).
docs.x.ai (2026-08-12): grok-4.6 is the flagship, 500K context.
Live GET /v1/models lists grok-4.6 at context_length 500000
(no grok-4.6-latest alias).
#84661 landed the catalog. Main already lists native grok-4.6
on the xAI picker. This is only the leftover cache guard
(same pattern as grok-4.3): pre-catalog builds persisted the
grok-4 catch-all (256K).
'' is a substring of every catalog key, so _resolve_endpoint_context_length
with an empty model name "matched" whatever the endpoint listed first —
on the Nous portal that is currently a 32K embedding model, which poisoned
the resolved context length and made AIAgent init fail the 64K minimum.
This is what turned tests/run_agent/test_primary_runtime_restore.py::
TestTryRecoverPrimaryTransport::test_allowed_for_nous_anthropic_messages
red on every PR (CI slice 7/12) after the portal catalog reordered.
Single-model endpoints still resolve with an empty name (unambiguous);
non-empty names keep the substring fuzzy match.
voice.toggle is now pool-routed (returns None from dispatch), so the
audio playback guard test must call handle_request directly to get
the response dict.
wake.start calls check_wake_word_requirements() → _stt_ready() →
_get_provider() → _try_lazy_install_stt() → ensure("stt.faster_whisper")
(same synchronous subprocess install chain as voice.toggle), and
start_listening() → _build_engine() whose constructors call
lazy_deps.ensure("wake.openwakeword" / "wake.sherpa" / …).
wake.status calls check_wake_word_requirements() too and is polled
by the desktop on every gateway-ready. Same bug class as #21123 /
#50005 — sibling to the voice RPC fix in the prior commit.
Update existing wake.start test call sites from server.dispatch() to
_dispatch_sync() since dispatch() now returns None for pool-routed
methods. Extend the pool-routing regression test to cover wake RPCs.
voice.toggle (status) triggers check_voice_requirements() -> STT provider
auto-detect -> a synchronous faster-whisper lazy install (uv/pip subprocess
with a 300s timeout). Inline on the WS reader thread it stalls handle_ws
before it reads the next frame, so prompt.submit / session.list queued
behind a voice.toggle sit unread and the desktop 'send message' appears
dead for minutes while the install churns (reproduced: voice.toggle ->
session.list 40s+ timeout).
Route voice.toggle/voice.record/voice.tts to the RPC pool (same bug class
as #21123 / #50005) so a slow lazy install can't block message handling.
Adapt the voice handler tests to drive the handler inline via a small
_dispatch_sync helper (preserving transport-binding semantics) since
dispatch() now returns None for pool-routed methods, and add a regression
test asserting the voice RPCs stay pool-routed.
* feat: server-side ui_meta on profiles.list/configure
Roster UIs built on profiles.* have per-profile presentation state
(avatar, accent color, display title, pet) with nowhere server-side to
live — client plugin storage paints a different roster on every
machine. profiles.configure now accepts ui_meta (merged key-wise into
profile.yaml's ui_meta block via the existing atomic_yaml_write path,
null deletes a key, 64KB cap since it rides every roster paint) and
profiles.list returns the block per row. Consumers namespace under
their own key. No new files or config; profiles without the block are
unchanged.
* test: stop primary-runtime-restore tests probing live endpoints
_make_agent left the compressor's lazy context-length resolution
unmocked; for reachable base_urls (the nous portal test) the endpoint's
32K answer for the empty test model trips agent_init's 64K floor and
fails the suite on network behavior. Pin get_model_context_length in
the fixture.
Review caught a real gap: action='update' also accepts deliver, and the
tool description explicitly steers agents toward update-over-create — so
a cron-context agent updating a job to deliver='origin' would recreate
exactly the dangling literal-origin shape the create-path resolution
prevents (stored 'origin' on an origin-less job → fire-time home-channel
guessing or silent drop).
Wrap the update site in the same resolver. Semantics follow the create
precedent: in cron context, 'origin' means 'my run's target', resolved
concretely at mutation time; outside cron context updates are
byte-identical to before.
A job created from within a cron run must never store the literal
'origin' delivery target: the creating session is ephemeral, so by fire
time there is no origin to resolve and the scheduler falls back to
guessing a home channel. With agent scheduling enabled
(cron.allow_agent_scheduling), a scheduled agent creating follow-up jobs
would silently produce exactly that dangling shape.
Resolve at create time instead, in cron context only: 'origin' elements
(and an omitted deliver) are replaced with the creating run's concrete
target from the per-run HERMES_CRON_AUTO_DELIVER_* contextvars —
platform:chat_id[:thread_id], or 'local' when the creating run has no
concrete target. Explicit values ('local', 'all', platform:chat_id
targets) pass through verbatim, including inside comma lists. Chat and
CLI creates are byte-identical to before: the resolver is a no-op
outside cron-context sessions (HERMES_CRON_SESSION unset).
Cron-spawned agents have the cronjob toolset unconditionally denied, so
scheduled agents cannot create, tune, or remove jobs even when an
operator wants exactly that (reconciler-style jobs that manage a team's
cron table, follow-up one-shots scheduled from within scheduled work).
The denial is loop-prevention policy, not a security boundary: an agent
with the terminal toolset can already shell out to the CLI, so the
workaround exists but skips every limit and accounting layer.
Add cron.allow_agent_scheduling (config.yaml, default false — byte-exact
current behavior). When enabled, only 'cronjob' leaves the cron-context
denylist; 'messaging' and 'clarify' remain denied as interactivity
constraints, and the user-level agent.disabled_toolsets layering is
unchanged, so a user denylist entry still beats the gate. The cronjob
tool description now states the real policy and the quota bounds instead
of a blanket prohibition.
Per review: keep fall-through, claim, first-valid-wins, skipped-result
warning, malformed-result isolation, sanitization, and the end-to-end
synthetic-plugin test; drop the auxiliary variants. Compress the hooks.md
section to prose with a minimal return example.
Applies the batch-disposition SALVAGE conditions from #64231: the hook id
moves to the taxonomy transform-family name, and run-all-then-pick-first
dispatch now logs a runtime warning when a valid-but-losing classification
is skipped (the #64714 skipped-transform rule). Chaining semantics are
stated explicitly at the VALID_HOOKS entry, the dispatch helper docstring,
and the hooks.md catalog row and detail section.
Rebased onto current main, where the OpenRouter tool-use 404 is now
handled natively (the bundled demo's exact reason to exist), so the demo
plugin is removed per the standalone-repo policy and every test now uses
a synthetic unclaimed error (fake provider, neutral message, no status
code) that no present or future built-in rule can claim.
classify_api_error is now explicitly Python-plugin-only: VALID_HOOKS
doubles as the shell-hook allow-list, but the shell response parser has
no channel for the classification directive, so shell registrations are
refused at config parse with a warning instead of being silently
ignored (new SHELL_UNSUPPORTED_HOOKS set + regression test).
The hook is documented in the hooks reference as the third
behavior-changing hook, with the full kwargs contract, return shape,
and the Python-only note.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnMCvi2vXqfs996AjVeF2F
Adds a plugin seam at the top of agent/error_classifier.classify_api_error()
(step 0, before the built-in pipeline) so model-provider plugins can classify
their provider's error quirks without patching core:
- New "classify_api_error" entry in VALID_HOOKS. Callbacks receive the parsed
error context (provider, model, status_code, error_type, error_code,
error_message, error_body, error, approx_tokens, context_length,
num_messages), self-scope on `provider`, and return None to pass or a dict
{"reason": "<FailoverReason name>", ...optional recovery-hint overrides}.
- get_plugin_error_classification() helper mirrors
get_pre_tool_call_block_message(): first valid result wins, invalid dicts
and unknown reasons are skipped, callback exceptions are isolated — a
broken plugin can never break classification. Zero behavior change when no
plugin claims the error (all 179 existing classifier tests pass untouched).
- Bundled reference plugin `openrouter-tool-use-404` (opt-in, like all
bundled standalone plugins) re-implements PR #58451: OpenRouter's
"No endpoints found that support tool use" 404 carries no
_MODEL_NOT_FOUND_PATTERNS signal, so it classifies as unknown/retryable
and the retry loop burns 3-5 attempts on a deterministic rejection.
The plugin classifies it as model_not_found (retryable=False,
should_fallback=True) so the fast-fallback path fires immediately —
demonstrating a waiting core PR converted to a publishable plugin.
Motivation: ~10 open PRs are single-provider error-classification patches
(#58451, #58355, #58502, #58474, #58366, ...). This hook turns that whole
class of contribution into plugin territory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWMcB7RPSYUpsXDfBgwjzM
Per review: keep the load-bearing tests (fire+payload per hook, the
lock-probe contract test, misbehaving-subscriber isolation, no-subscriber
short-circuit, mutation-boundary coverage) and drop the auxiliary
variants; compress the hooks.md additions to a single catalog-row set
plus a compact bullet section.
Implements the remaining observers from RFC #58548 (@thebizfixer),
accepted as the design basis in the #64231 batch disposition:
- on_kanban_worker_spawned: fires in the dispatch loop after spawn_fn
returns and the worker PID is durably persisted (the RFC timing
contract), in both the ready and review lanes.
- on_kanban_worker_exited: tick-derived from detect_crashed_workers;
fires after every reclaim/accounting txn has committed, carrying
exit_kind / exit_code / outcome / retry_status.
- on_kanban_worker_stale_claim: fires when release_stale_claims
reclaims a TTL-expired claim; live-PID claim extensions and deferred
reclaims stay silent.
- on_kanban_task_updated: task-mutation boundary observer carrying
changed_fields (field names only); fired by assign_task,
set_model_override, and set_reasoning_effort, and by the dashboard
plugin API's direct-SQL priority/title/body editors (single and
bulk) through the new kanban_db.notify_task_updated seam.
- on_kanban_dispatch_tick: re-port of PR #56066 (@laboratoiresonore),
renamed per the taxonomy and fired strictly AFTER _dispatch_tick_lock
is released; the sweeper found the original fired inside the lock,
where a slow subscriber could extend the single-writer critical
section and stall a sibling dispatcher.
All five are observer-only (return values ignored), fire after the
relevant write txn commits, and short-circuit on has_hook() so nothing
is built when no consumer registers; every fire site is fully
best-effort so a broken plugin can never break dispatch or a task
mutation. No config surface added. Existing plugins and hook payloads
are untouched.
Mutation-boundary scope: every user-facing task-FIELD editor fires
(assignee, priority, title, body, model/provider override, reasoning
effort). Deliberately not wired: status transitions (they belong to
the lifecycle hook family), dispatcher bookkeeping columns
(worker_pid, workspace_path, claim columns — surfaced through the
worker hooks instead), link/comment/attachment tables (not task-row
writes), and the dispatcher's default-assignee auto-assign (already
surfaced via DispatchResult.auto_assigned_default in the tick
payload). notify_task_updated is the seam for wiring further paths.
Docs: new rows plus a detail section in the shipped plugin-hook catalog.
Tests: 30 new (9 worker lifecycle, 8 dispatch tick, 8 task updated,
5 dashboard mutation boundary), including a lock-probe contract test
that fails if the tick hook ever fires inside the dispatch lock.
Refs: RFC #58548, #64231 batch disposition, folds #56066.
delegate_task gains a control plane: action='list' / 'steer' / 'stop'
let the parent agent see, redirect, and early-stop its own running
subagents mid-flight — the model-facing counterpart of the TUI's
delegation.pause / subagent.interrupt / subagent.steer RPCs.
- action='list': live children of this conversation's spawn tree
(ids, goal, status, running_seconds, accepting_steer, live
transcript path). Ownership is enforced via a _delegate_parent_ref
weakref chain stamped at child build time, so a conversation can
only control its own descendants, never a sibling tree.
- action='steer': queues text into a running child via the existing
steer_subagent() registry path (delivered at the child's next tool
boundary; missed steers surface as missed_steer in the completion).
- action='stop': interrupt_subagent() — child stops at its next
iteration boundary, partial result still re-enters as a completion.
- Spawn dispatch response now includes subagent_ids + control hint.
- Control actions run synchronously (never backgrounded) and bypass
the spawn pause gate and depth limit; they also never consume the
per-turn subagent spawn cap, and remain usable once the cap is hit
(that is when stop matters most).
- Small-model robustness (found live with gpt-5.4-mini on Nous
Portal): tasks=[] alongside goal no longer trips the "Batch mode
requires at least 2 tasks" gate — treated as single-goal.
- CLI display: control calls render as "steer sa-…" / "list" instead
of an empty goal.
Live-tested E2E on Nous Portal (fable-5 + gpt-5.4-mini): full
spawn→list→steer→stop cycle, plus a steer-efficacy run where the
child acked the steer mid-essay and switched topics before finishing.
The session chat stream registered its wrapper task in _active_run_tasks,
but that turn is already counted by active_agent_work_count() via
_inflight_agent_runs (_run_agent) — the drain saw 2 for one turn
(test_session_chat_sse_turn_is_interrupted). Keep only the agent-ref
registration; run-scoped steer control doesn't need the task entry.
Adds POST /v1/runs/{run_id}/steer and bridges Browser-Extension/WebUI
session chat streams into the active run registry so live runs on those
surfaces are steerable too.
- steer accepted only while run status is exactly 'running'; stop/stopping/
terminal states return 409 run_not_accepting_steer even while cooperative
shutdown retains the agent reference
- session SSE disconnect/cancellation interrupts and drains the executor-
backed run instead of cancelling only the async wrapper; control refs stay
registered until the turn actually exits
- undelivered steer text (accepted after the final response) is preserved as
pending_steer on the terminal run.completed event/status so clients can
replay it as the next user turn instead of losing it
- docs for the endpoint, next-tool-boundary delivery, acceptance-vs-delivery
semantics
Salvaged from PR #54466 by @abundantbeing.
The cherry-picked guard sat inside the codex-items block, which (a) is
skipped entirely in codex_responses mode (conversation_loop passes
drop_codex_reasoning_items=False there) and (b) is unreachable for
carriers whose adapter-joined commentary populates msg['reasoning'] —
the string-reasoning branch returns True first. Hoist the checkpoint
check above every reasoning branch so no carrier shape can be dropped,
in any api_mode. Adds the two carrier-shape tests that pin exactly this
(both fail with the guard in its original position).
A type="compaction" item is the server-side stand-in for history that has
already been pruned, and it rides the same codex_reasoning_items sidecar as
per-turn reasoning. e00965a7e taught compaction pruning to filter that
sidecar instead of popping it so checkpoints survive on every retained
message.
The thinking-only sanitizer reaches the same sidecar from the other
direction and asks a coarser question: does any item have type ==
"reasoning"? A commentary turn carrying both a reasoning item and a
checkpoint answers yes, so the whole message is dropped from the wire copy
and the only copy of the checkpoint goes with it. The request then carries
neither the compacted history nor the checkpoint standing in for it.
Extract has_compaction_checkpoint() into agent/native_compaction.py — the
module that owns the concept, and where merge_interim_reasoning_items()
already spelled the same predicate inline — and consult it before the
thinking-only verdict. A reasoning-only carrier is still dropped.
Two reliability gaps from #82777:
1. Rejection matcher required only a field-name mention, so a transient
5xx/timeout whose body echoed the request (which contains
context_management) permanently downgraded native compaction for the
session. Now requires rejection language (unknown/unsupported/invalid/...)
alongside the field name, and when a parsed HTTP status is available,
400 specifically — non-400 statuses never match. Message-only transports
(no status attribute) keep working unchanged.
2. compression.codex_responses_native was coerced with bool(), so the
strings "false"/"off" enabled the feature. Now uses the shared
utils.is_truthy_value helper.
Conversation-loop call site passes api_error.status_code through.
Sabotage-verified: reverting the matcher to field-name-only fails the new
echo and non-400 tests.
Simplify-pass follow-ups on the salvage stack (all guard tests re-run,
mutation-checked):
1. conversation_loop.py: moved `_preflight_compression_blocked = False`
from 9 per-site copies into the restart_with_rebuilt_messages handler
(its single consumer). Besides removing the 9 duplicated blocks, this
fixes a 10th pre-existing retry-loop site (content-filter stall
failover, #32421) that set the flag and broke WITHOUT clearing the
preflight block — a content-filter failover previously restarted with
preflight compression still blocked against the fallback's smaller
window, the same #84733 bug class. The outer-loop empty-response site
keeps its own clear (it never passes through the handler). New AST
guard test_restart_handler_clears_preflight_block pins the hoisted
clear (mutation-checked).
2. agent_runtime_helpers.py: extracted _raw_cache_ttl_from_config() —
prompt_caching_disabled_from_config and configured_cache_ttl were
verbatim copies of the same config read. Added VALID_CACHE_TTLS.
3. prompt_caching.py: added is_qwen_model() next to
ALIBABA_FAMILY_PROVIDERS; effective_cache_ttl and
anthropic_prompt_cache_policy now share both the family set and the
qwen predicate — neither can desync.
4. Guard-test hardening: assert every _try_activate_fallback reference
is a direct `if agent._try_activate_fallback(...):` site, so a future
`activated = ...` form can't silently escape the restart-discipline
guard.
Follow-ups on the salvaged #84782 (webtecnica):
1. conversation_loop.py: the empty-response fallback site sits directly
in the OUTER iteration loop, not the retry loop. The salvaged commit's
`break` there exited the conversation loop and ended the turn without
ever calling the just-activated fallback (caught by CI:
test_empty_response_triggers_fallback_provider). Restored `continue`
(which already re-runs the pre-API preflight at the top of the next
outer iteration) while keeping the `_preflight_compression_blocked`
reset. The other 9 sites are inside the retry loop, where `break` to
the restart_with_rebuilt_messages handler is correct.
2. test_prompt_cache_ttl_propagation.py: made the AST guard loop-aware —
retry-loop sites must break, outer-loop sites must continue (the old
assertion pinned the bug in (1)). Mutation-checked both directions.
3. test_failover_identity.py: added `model` to the SimpleNamespace agent
fixture — _redecorate_prompt_cache_for_provider now reads agent.model
for the per-destination TTL clamp (2 CI failures).
4. prompt_caching.py / agent_runtime_helpers.py: single source of truth
for the alibaba-family provider set — ALIBABA_FAMILY_PROVIDERS lives
in prompt_caching and anthropic_prompt_cache_policy imports it, so the
cache-policy opt-in and the TTL clamp can never desync.
5. auxiliary_client.py: threaded the configured tier into
_replan_synchronous_cache_sections via new configured_cache_ttl()
(no live agent on that path) — the aux half of #84733's report also
stopped regressing 1h to 5m. Guarded by
TestAuxFallbackReplanThreadsTtl (mutation-checked).
6. Dropped the redundant `or "5m"` at the two threaded call sites —
effective_cache_ttl already resolves None to "5m", and the `or`
masked the cache-disabled (None) semantics.
Choosing Computer Use should be a config flip, not a hunt for
'hermes computer-use install'. Three provisioning rungs:
- install.sh / install.ps1 pre-install cua-driver (best-effort,
non-fatal, time-boxed at 660s above the upstream installer's 600s
lock window; --skip-computer-use / -SkipComputerUse to opt out;
Termux and unwritable-/Applications skipped cleanly)
- PUT /api/tools/toolsets/{name} (dashboard + desktop toggle) spawns
the background 'hermes tools post-setup cua_driver' action when the
toolset is enabled while the binary is missing — previously the
toggle 'saved' but the tool never appeared in the schema because
check_computer_use_requirements() couldn't find the binary
- hermes tools interactive flow already installed via
_toolset_needs_configuration_prompt/_POST_SETUP_INSTALLED (unchanged)
Docs: computer-use.md enabling section rewritten around the new flow;
installation.md documents --skip-computer-use.
_update_node_dependencies() installs the unified closure, but update then
calls _build_web_ui(), whose 'npm ci --workspace web' pass deleted
node_modules and re-reified only the web closure — pruning root
devDependencies and the ui-tui hoisted deps the previous step just
installed, while exiting 0. Since the manifests digest was already
recorded, later no-op updates skipped the repair.
Reported by @andrexibiza in the #44772 final review (P1). Reproduced
E2E: '--workspace web' alone removes typescript-eslint/@eslint/js from
root node_modules; the unified closure restores them.
Guards: ui-tui only named when its manifest exists (prebuilt checkouts),
web-own-lockfile (#42973) and Termux (#38772) paths unchanged.
check_subprocess_stdin.py already had a full-repo-scan pytest wrapper
(test_subprocess_stdin_guard.py), so a plain pytest run catches a
regression there without anyone remembering to run the script by
hand. check-windows-footguns.py had no equivalent (only a narrow
single-rule test existed), which is why the bare os.killpg/
signal.SIGKILL regression in the npx-agent-browser hardening commit
shipped past local testing and was only caught by CI running the
script directly. New test_windows_footguns_full_repo_scan.py mirrors
the stdin guard's exact pattern to close that asymmetry.
Also adds direct coverage for _kill_process_tree's getattr fallback
when os.killpg is missing, and asserts warm_agent_browser_npx_cache's
Popen call passes stdin=subprocess.DEVNULL as a literal argument.
ensure_browser() (install.sh) and Install-AgentBrowser (install.ps1)
are reached only via the explicit --ensure browser / -Ensure browser
on-demand mode, itself only triggered by an actual browser-tool call's
lazy-install fallback or `hermes acp --setup-browser`. agent-browser
already resolves via npx in that same fallback before ever reaching
these scripts, so eagerly npm-installing a second, separately
version-pinned copy here was redundant and an extra credential/
supply-chain surface for a path npx already covers. Chromium
acquisition for this on-demand path is now deferred entirely to
_maybe_autoinstall_chromium's existing lazy fallback. camofox's
install and system-browser detection/configuration are unaffected.
install.ps1 also drops the now-dead -SkipChromium switch, confirmed
unused at its one call site.
Full rewrite of test_browser_npx_warmup.py for the Popen-based
credential-scrubbing, PATH-propagation, and process-tree-kill rework:
argv shape, env scrubbing, PATH merge for managed-only npx, POSIX
process-group creation, Windows CREATE_NEW_PROCESS_GROUP, whole-tree
kill (not just the PID) on timeout with a bounded post-kill drain, and
_kill_process_tree's own POSIX/Windows/failure paths directly.
Also fixes test_windows_subprocess_no_window_flags.py's matching
regression test, which still mocked subprocess.run and a shutil.which
signature that didn't accept the path= kwarg _resolve_npx_bin's
extended-path rung now passes; its creationflags assertion becomes a
bitwise check since Windows now ORs CREATE_NEW_PROCESS_GROUP in
alongside the console-hiding flag.
Root package.json still owns devDependencies (the shared ESLint flat
config every workspace's eslint.config.mjs imports) even though
agent-browser and @streamdown/math were already removed from root
dependencies. The scoped `npm ci --workspace ui-tui --workspace web`
prunes them the same way it used to prune those; --include-workspace-root
protects them without reintroducing apps/desktop into the install.
- --ignore-scripts on every real npx agent-browser invocation.
AGENT_BROWSER_NPX_SPEC is a floating ^0.26.0 range, not an exact
pin, and none of these sites passed it (unlike install.sh/
install.ps1's own npm install of the same package). Verified against
the real CLI: `npx --ignore-scripts --prefer-offline -y
"agent-browser@^0.26.0" --version` resolves cleanly on npm
11.19.0/node 26.
- _resolve_npx_bin() now checks the Hermes-managed/extended search
before a bare ambient PATH lookup, validating each candidate with
node_tool_runnable before trusting it — a bare PATH-first lookup let
a broken system npx shadow a healthy managed one with no recovery.
- warm_agent_browser_npx_cache() now runs a credential-scrubbed,
PATH-propagated environment (matching every other agent-browser
subprocess spawn) instead of inheriting the full parent environment
including every provider/gateway credential Hermes holds, and kills
the whole process tree (not just the top-level npx PID) on timeout
via the new _kill_process_tree helper, since a surviving descendant
can otherwise hold a capture pipe open past the nominal deadline.
_browser_available()'s npx rung was missing the bare-npx-on-Termux
guard its sibling probes (dep_ensure, nous_subscription) already
apply, so it could report the browser probe available on Termux when
local mode would actually reject the bare npx fallback and fail on
first use.
Also adds argv-level coverage for the two real npx launch sites
(_run_browser_command, _run_chrome_fallback_command) and an
end-to-end test proving _find_agent_browser's lazy-install fallback
and ensure_dependency("browser")'s npx check terminate without
recursion.
The tools.browser_tool import-failure fallback in _has_agent_browser
dropped the Windows-installer managed-PATH probe and replaced a
PATHEXT-aware shutil.which lookup with a bare Path.exists() check,
reintroducing the .cmd-shim miss that probe was added to fix.
Both probes only checked PATH and node_modules, so they disagreed with
`hermes doctor` on npx-only installs (#43564): doctor --live reported
the browser probe unavailable, and ensure_dependency("browser") could
shell out to install.sh on installs doctor already reports healthy.
Git-clone installs resolving agent-browser via bare npx floated latest
with no integrity check, while install.sh/install.ps1 installs stayed
pinned to ^0.26.0. Pin the npx spec to match. Also extract the
"npx agent-browser" sentinel comparison (6 call sites across two
packages) into a named constant/predicate, fix a PATH-priority
inversion where a broken system npx could shadow a healthy
Hermes-managed one at the two real npx launch sites, and stop
`hermes doctor --fix` from counting a bonus npx cache warm as a fixed
issue on an otherwise-healthy run.
_run_npm_watching_for_engine_failure routes capture_output=False npm
invocations (the path _update_node_dependencies always uses) through
subprocess.Popen instead of subprocess.run. The
TestUpdateNodeDependencies mocks still patched subprocess.run, so they
fell through to the real, conftest-guarded Popen and tried to exec a
nonexistent /usr/bin/npm.
warm_agent_browser_npx_cache() spawns a resolved npx.cmd via
subprocess.run with a list arg and no shell=True, which Windows still
routes through cmd.exe. Without creationflags=windows_hide_flags(),
that can flash a console window during hermes update/doctor --fix,
same as the existing agent-browser subprocess spawn elsewhere in this
file already guards against.
Adds a regression test to the cross-cutting Windows no-window-flags
audit suite so a future refactor can't silently drop the flag again.
_find_agent_browser's extended-PATH branch now calls
shutil.which(name, path=extended_path), which broke two
post_setup_gating tests mocking shutil.which with name-only lambdas.
Update those mocks and two similarly-shaped chromium test mocks that
were latent landmines, and add coverage for cascade branches (local
node_modules/.bin, validate=False paths, and
_agent_browser_candidate_present) that had none.
The truthful per-provider readiness work (#67201) gates the desktop
Capabilities panel on _has_agent_browser, which only probes PATH and
node_modules/.bin. Now that agent-browser is no longer a root
package.json dependency (#43564), npx-only installs report needs_setup
in the panel while the browser tools themselves resolve fine at
runtime — and existing installs flip to needs_setup as soon as a
hermes update prunes node_modules.
Mirror the local-CLI tail of check_browser_requirements: resolve via
_find_agent_browser(validate=False), honor the Termux bare-npx
carve-out, and keep the old probe as the import-failure fallback.
Existing shutil.which test stubs gain the real signature so the
cascade's path= keyword calls don't break them.
`hermes update` was pruning root-level Node dependencies (agent-browser)
because npm ci always wipes and reifies node_modules according to its
active filter -- no root-first/workspace-first ordering or flag
combination (--workspaces=false, --include-workspace-root, etc.) can
reliably keep a root-only package.json dependency from being pruned by
a subsequent workspace-scoped npm ci. Confirmed empirically and via
npm/cli source (isArboristCmd hardcodes includeWorkspaceRoot=false for
ci/install), so no amount of install-order juggling fixes this for good.
Instead of chasing install order, remove the root-only dependencies
that made the npm step fragile in the first place:
- agent-browser is no longer a root package.json dependency. It
resolves lazily via `npx agent-browser` (tools/browser_tool.py
already had this as a fallback; it's now the primary path).
warm_agent_browser_npx_cache() is called fire-and-forget from both
`hermes update` and `hermes doctor --fix` to keep npx's cache warm,
preserving the "available before any session starts" property
agent-browser had as an eager dependency without re-entangling it
with the npm workspace graph.
- @streamdown/math moves to apps/desktop/package.json, where it's
actually imported (markdown-text.tsx, katex-memo.ts) -- it was
never used anywhere else and was subject to the same pruning risk.
- _update_node_dependencies() collapses to a single
`npm ci --workspace ui-tui --workspace web` call now that root has
no dependencies of its own to protect, and keeps its original spot
ahead of `_build_web_ui()` at both call sites in update_cmd.py --
with no root-only dependencies left to protect, there's no reason
for the Node refresh and the web build to run in any particular
order relative to each other.
- hermes_cli/tools_config.py's post-setup Chromium-install path and
hermes_cli/doctor.py's agent-browser check both now resolve through
the same PATH -> Homebrew/Hermes-managed-node -> npx cascade
(_find_agent_browser / _resolve_npx_bin) instead of hand-rolling
their own node_modules/.bin lookups, so they can't diverge from what
browser tools actually invoke at runtime.
- tests-js/package-json-lazy-deps.test.ts gets a lockfile-level check
mirroring the existing camofox one, so a future regression that
reintroduces agent-browser into package-lock.json fails this test
directly instead of relying on manual review to catch it.
Fixes#43564.
Simplify-code findings:
- _authenticated_json: replace manual header construction with
self._headers(include_tenant=False) — eliminates duplication with
_headers() and includes Content-Type consistently.
- _health_requires_credentials: replace getattr(exc, 'status_code')
with _status_code_from_error(exc) for consistency with the existing
error-classification utility. Drop the fragile string-matching
fallback — _parse_response always sets status_code on
_OpenVikingHTTPError, so 401/403 check is sufficient.
- Relax test header assertions to check presence/absence of specific
headers rather than exact dict equality, so they survive the
header-construction refactor.
Hosted OpenViking (Volcengine) rejects anonymous GET /health with
AuthenticationError, which made the provider look unhealthy and silently
disabled automatic memory mirroring. Keep the anonymous probe first for
identity safety, then retry once with the configured API key only when
the server demands credentials.
Fixes#78410
OpenCode identifies clients by request headers, the same way OpenRouter
does. Our opencode-zen and opencode-go profiles never set any, so every
request went out with the OpenAI SDK default "OpenAI/Python x.y.z"
User-Agent and OpenCode had no way to tell the traffic was Hermes Agent.
Two changes:
- Add HTTP-Referer, X-Title, and a HermesAgent User-Agent to both
OpenCode profiles through profile.default_headers, the same path
Fireworks uses. This covers chat_completions, codex_responses,
auxiliary clients, model switches, and the models catalog fetch.
- Merge the same headers in build_anthropic_client for opencode.ai
base URLs. The Anthropic Messages route (Claude on Zen, MiniMax and
Qwen on Go) builds its client there and never sees profile headers.
Verified against the live Go relay with a real key. Both wire formats
return HTTP 200 and the requests now carry X-Title "Hermes Agent",
HTTP-Referer, and User-Agent HermesAgent/0.20.0.
Live verification (gpt-5.6 @ api.openai.com) proved the Responses server
renders NOTHING placed before a replayed compaction checkpoint: a fact
stated in a pre-checkpoint input item is invisible to the model, while the
same item after the checkpoint recalls perfectly. Hermes was replaying the
full pre-checkpoint transcript anyway — dead upload weight, and worse, every
plaintext user ask from before the boundary silently vanished from the
model's view, surviving only inside the opaque server summary. That is the
goal-drift failure mode reported against native compaction sessions.
Codex CLI never hits this because it rebuilds history client-side after
compaction, retaining user messages verbatim under a token budget. This
change is the wire-level equivalent: when a replayed checkpoint is present,
_chat_messages_to_responses_input restructures the input as
[newest checkpoint run] + [retained pre-checkpoint user messages,
newest-first within a 64K-token budget] + [post-checkpoint tail]
Histories without a checkpoint are returned unchanged, so non-native
sessions see a byte-identical wire.
Webhook agent runs default to the constrained hermes-webhook toolset
(web/vision/clarify) because payloads can carry untrusted third-party
content. That default is right for public webhooks but wrong for trusted
local pushes (e.g. an OOM monitor daemon that needs the agent to run
ps/free/py-spy): the only workaround was widening platform_toolsets.webhook,
which elevates EVERY webhook route at once.
This adds a 'toolsets' key on individual webhook route configs (static
routes in config.yaml and dynamic subscriptions in
webhook_subscriptions.json) that replaces the platform-level resolution
for that route only:
- BasePlatformAdapter.toolsets_for_source(): per-source override hook,
default None (no behavior change for any other platform).
- WebhookAdapter.toolsets_for_source(): maps the session chat_id
(webhook:{route}:{delivery_id}) back to its route config and returns
the route's toolsets list.
- GatewayRunner._resolve_enabled_toolsets_for_source(): shared resolver
used by both agent-run call sites; validates the override through the
SAME _get_platform_tools path as platform config, so unknown names and
platform-restricted toolsets (e.g. discord_admin) are dropped rather
than trusted.
Deliberately NOT exposed via 'hermes webhook subscribe': granting elevated
tools is a manual config edit only, so an agent-created subscription
cannot self-grant terminal at runtime.
Cache-safe: the toolset list is resolved before agent construction and is
constant for a route, so the per-session agent signature and frozen system
prompt are unaffected mid-conversation.
Follow-up to the salvaged #80489 substring-fallback removal: the
structured 401/403 branch still exited with a bare return, leaving
_running True — dead listener, healthy-looking is_connected(), gateway
never told (the zombie half of the bug, OOF-156 class). It now sets a
non-retryable mattermost_auth_error with token guidance and notifies
the gateway fatal handler.
Also: pytest.importorskip for aiohttp in the verifier probe file
(module-level import crashed collection in envs without the optional
dep), and probe fixtures updated for the escalation attributes.
Independent-verifier boundary probes for commit fdd1a11ac5, covering
cases the implementer's regression tests did not exercise:
- WSServerHandshakeError(status=403) also stops the loop (only 401 tested)
- WSServerHandshakeError(status=500) does NOT stop the loop (structured
check must not over-match on type alone)
- transient error containing the word 'unauthorized' (not digit substring)
now retries correctly
- 5 consecutive transient errors all retry, not just the first
Verified these 2nd/4th tests fail against the pre-fix baseline commit
(01a1037d1e) and pass against the fix (fdd1a11ac5), confirming they
have real signal.
The WS reconnect loop had a fallback check that looked for "401", "403",
or "unauthorized" as substrings anywhere in an exception's string form.
A transient error whose message happens to contain those digits (a proxy
body, a stack trace, anything) got treated as a permanent auth failure
and stopped reconnection for good.
I removed the substring fallback and kept only the structured check:
aiohttp.WSServerHandshakeError with status in {401, 403}. That's the only
signal that reliably means the server rejected our credentials.
Added two regression tests: one proving a transient error containing
"401" in its text still retries, and one confirming the existing
_closing early-return path is untouched by the removal.
Teknium
S
Victor Kyriazakos
Alexey (CTO)
kshitij
embwl0x
Erosika
赵桂雄
eapwrk
Dillon Townsel
spfcraze
Bartok9
LeonSGP43
Rob Sherman
Simon van Laak
Arman
luoxiao6645
Beto de Paola
sasquatch9818
fangliquan
andyst-dev
whirmill
verybigdog
Ben
Nikita Barkov
Julientalbot
hustwkr
uperLu
webdevtodayjason
Jon Komet
Drexuxux
webtecnica
Zak B. Elep
Christopher
Slobaka
Adolanium
Stephen Chin