A pool with only one usable (non-DEAD) credential has nothing to rotate to.
On a transient throttle (429 rate-limit, 403 edge-throttle, 5xx) the offending
key was benched for a full hour (EXHAUSTED_TTL_429/DEFAULT), so single-key /
no-fallback setups got an hour of hard failures for a throttle that resets in
seconds. The pool already special-cases 401 to recover quickly for single-key
setups; extend that to transient throttles when the credential is the sole
non-DEAD entry. 402 (billing/quota) keeps the full bench — a quick retry can't
help. Provider-supplied reset_at still overrides.
Adds tests covering sole 429/403 recovery, 402 full-bench, and multi-key
(no early recovery).
hermes debug share runs on the backend. A desktop app connected to a
remote, docker, or SSH backend writes desktop.log on the client machine,
so the bundle can never contain it — and the report rendered that as a
bare "(file not found)", which reads as "the app logged nothing" and
sends triage after a client-side bug it cannot see.
Name the writer and the path to collect by hand. Backend-written logs
are unchanged, a present desktop.log is still captured, and an empty one
still reports "(file empty)" — the app ran and logged nothing is a
different fact from the file being on another host.
* fix(agent): adopt .env credential/base-url edits at the turn boundary
A Settings save (desktop PUT /api/env, hermes setup) updates .env and
the saving process's os.environ, but a live session worker keeps the
base_url/api_key captured at agent init until restart — an open chat
silently kept calling the old endpoint (e.g. a local-server key sent to
api.openai.com, failing with an opaque 401).
Add AIAgent._try_refresh_env_client_credentials(), called at the start
of each conversation turn: re-resolve the provider's env-sourced
credentials (load_env() is mtime-memoized, so an unchanged file costs
one stat()) and rebuild the client via the existing
_replace_primary_openai_client machinery when the user edited them.
The refresh reacts only to env edits — resolved values changed since
the last look — never to mere divergence from the agent's current
values: credential-pool rotation and failover legitimately move the
session off the env credential, and stomping those back would flap.
Config model.base_url / pool custom endpoints keep precedence: edits
are only adopted while the session still runs on the registry default
or the previously-seen env value.
Lift _get_env_prefer_dotenv out of _seed_from_env to module level
(get_env_prefer_dotenv) so both the pool seeder and the per-turn
refresh share the same .env-over-os.environ resolution, including the
op:// indirection handling.
Fixes#67821
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(agent): address sweeper review on env credential refresh
- Cover named custom providers (#67935): provider="custom" has no
PROVIDER_REGISTRY entry, so resolve the config block's key_env through
the same lookup the runtime resolver uses.
- Make the edit baseline transactional: a failed client rebuild rolls the
agent back and leaves _env_creds_seen un-advanced so the unchanged edit
is retried next turn.
- Recompute route-derived TLS material and default headers on a base-url
change, via a _reapply_route_client_config helper shared with
credential-pool rotation so the two paths cannot drift.
- Rebase onto main: get_env_prefer_dotenv keeps the scoped _get_secret
semantics from the profile-isolation fix (no raw os.environ reads).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: map jskang@lablup.com to rapsealk
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
* feat(desktop): expose native OS notifications to plugins via ctx.notifyNative
Desktop plugins can toast in-app (host.notify) but have no sanctioned way to
reach the OS notification pipeline the app's own approval/turn alerts use, so
a plugin surfacing a genuinely notable background event (e.g. a discovery
plugin finding a match) stays invisible once the user steps away from Hermes.
Add a curated per-plugin door instead of exporting the raw dispatcher:
- ctx.notifyNative({ title, body?, silent? }) on PluginContext — attributed
to the plugin id, routed through dispatchNativeNotification so every
existing gate applies (master + per-kind prefs, post-connect baseline,
away-from-app gating, throttle).
- New 'plugin' native-notification kind with its own Settings ▸ Notifications
toggle (default on), so users silence plugins without losing app alerts.
- New optional `tag` discriminator on the notify payload keys the renderer
throttle and main-process cross-window dedupe per plugin, so two plugins
can't collapse each other's session-less notifications.
Consumer: the Index Network desktop plugin wants background opportunity
alerts; anything in ~/.hermes/desktop-plugins gets the same door.
* feat(desktop): ctx.os — the curated OS door for plugins
Fold ctx.notifyNative into a ctx.os namespace so every way a plugin
reaches outside the app window lives behind one attributed door instead
of accreting one top-level ctx method per capability:
- ctx.os.notify — the native-notification door from the previous commit,
unchanged semantics (plugin kind pref, away-gating, per-plugin throttle).
- ctx.os.openExternal / ctx.os.revealPath / ctx.os.writeClipboard — the
existing window.hermesDesktop bridge capabilities, now sanctioned and
result-shaped: each resolves false (never throws) when the bridge or
member is missing, so a plugin branches on the result instead of
sniffing the preload surface or crashing on an older shell.
No new Electron surface: everything routes through bridge members the
app already ships; the notification path keeps every existing gate.
---------
Co-authored-by: seref <1573640+serefyarar@users.noreply.github.com>
* fix(credential-pool): clear exhaustion state on key rotation
When a user rotates an API key (e.g. via `hermes setup` after hitting a
rate limit), _upsert_entry updates the access_token on the existing pool
entry but preserves the stale last_status=exhausted from the old key.
On the next session the pool finds the entry, sees it exhausted, and
returns no usable credentials — even though the new key is valid.
Fix: when access_token changes on an existing entry, reset last_status,
last_error_code, last_error_reason, last_error_message, and
last_error_reset_at. The exhaustion state belongs to the old key, not
the new one.
* chore: add pasevin@gmail.com to AUTHOR_MAP
* fix: clear last_status_at on key rotation, remove unused pytest import
Address review feedback from teknium1 on PR #22622:
- Add last_status_at=None to the reset block (matches all other
token-sync reset paths in credential_pool.py)
- Assert last_status_at is None in the regression test
- Remove unused pytest import flagged by ruff + ty
Fold ctx.notifyNative into a ctx.os namespace so every way a plugin
reaches outside the app window lives behind one attributed door instead
of accreting one top-level ctx method per capability:
- ctx.os.notify — the native-notification door from the previous commit,
unchanged semantics (plugin kind pref, away-gating, per-plugin throttle).
- ctx.os.openExternal / ctx.os.revealPath / ctx.os.writeClipboard — the
existing window.hermesDesktop bridge capabilities, now sanctioned and
result-shaped: each resolves false (never throws) when the bridge or
member is missing, so a plugin branches on the result instead of
sniffing the preload surface or crashing on an older shell.
No new Electron surface: everything routes through bridge members the
app already ships; the notification path keeps every existing gate.
Show earlier spends the already-materialized DOM budget first and only asks the
session store for another page once that is exhausted, so the click stays cheap
and the store window stays as small as it can be.
Paging has no ceiling: each expand grows the window by one budget page until the
whole transcript is loaded. Branch persistence stays wired throughout —
setMessages is never dropped, so switchToBranch and applyBranchVisibility keep
working on a windowed session.
Co-authored-by: HexLab <8422520+HexLab98@users.noreply.github.com>
An oversized session rebuilt an unbounded runtime repository on every store
update and exhausted the renderer's V8 heap, crash-looping the window. The DOM
budget in thread/list.tsx bounds what PAINTS, but every message was still
normalized into the repository first, so a session only had to be heavy — not
visible — to kill the renderer.
selectTranscriptWindow keeps the tail that fits one render-weight page. Weight,
not message count: measured against a real 1,175-session store, a 400-message
cap disengages on 37 sessions that are heavy but short (one is 133 messages /
1.05MB) while firing on 92 long-but-light sessions that were never at risk.
The cut aligns off branch-group boundaries. useRuntimeMessageRepository records
a group's fork point the first time it sees the group, so a window starting
mid-group would re-parent the surviving branches to whatever happened to
precede them.
Co-authored-by: HexLab <8422520+HexLab98@users.noreply.github.com>
messageRenderWeight moves out of thread/list.tsx into lib/render-weight.ts.
The DOM page budget already spends render cost rather than message count —
the store window added next needs the same currency, and one weight function
keeps the two layers from drifting apart.
No behavior change.
Desktop plugins can toast in-app (host.notify) but have no sanctioned way to
reach the OS notification pipeline the app's own approval/turn alerts use, so
a plugin surfacing a genuinely notable background event (e.g. a discovery
plugin finding a match) stays invisible once the user steps away from Hermes.
Add a curated per-plugin door instead of exporting the raw dispatcher:
- ctx.notifyNative({ title, body?, silent? }) on PluginContext — attributed
to the plugin id, routed through dispatchNativeNotification so every
existing gate applies (master + per-kind prefs, post-connect baseline,
away-from-app gating, throttle).
- New 'plugin' native-notification kind with its own Settings ▸ Notifications
toggle (default on), so users silence plugins without losing app alerts.
- New optional `tag` discriminator on the notify payload keys the renderer
throttle and main-process cross-window dedupe per plugin, so two plugins
can't collapse each other's session-less notifications.
Consumer: the Index Network desktop plugin wants background opportunity
alerts; anything in ~/.hermes/desktop-plugins gets the same door.
Hitting ⌘1 (or cycling ⌃Tab onto the main tab) while Capabilities /
Messaging / Artifacts covered the workspace looked dead: the workspace
pane was already the zone's active tab behind the page, so fronting it
changed nothing on screen.
activateTreeTabSlot / cycleTreeTabInFocusedZone now return the activated
pane id, and the keybind handlers route back to the loaded session (or
the new-chat draft) when the landing pane is the workspace under a full
page — the same rule openSession already applies.
hermes model saves custom_providers models: {default: {context_length}} for
local Ollama. That dict shape was treated as an explicit catalog, so no-key
endpoints skipped live /v1/models probing and Desktop/Telegram only showed
the saved default — Refresh could not help. Keep list/string shapes as
allowlists; pin dict catalogs with discover_models: false.
Drop the manual web.search_backend / web.backend config-reading block
that duplicated _read_config_key in web_search_registry.py. The function
now delegates directly to get_active_search_provider() (which reads the
same config keys via the registry's canonical resolver) and falls back
to _get_search_backend() only when the registry has no providers loaded.
Also updates the TestXaiWebSearchBackendPreference tests to monkeypatch
the registry instead of load_config_readonly, and adds two new tests for
the legacy fallback path (no provider registered -> _get_search_backend).
Lock in backend preference, wire-name aliasing, and normalize mapping
so configured non-xai search providers stay on the Hermes client path.
Also init conflict-recovery generation on the telegram bare-adapter
helper so CI polling progress tests do not AttributeError.
When Grok runs on xAI Responses, only swap to native server-side
web_search when the active/configured backend is xai. For Firecrawl
and other Hermes providers, keep client dispatch under a renamed wire
tool so Grok cannot hijack web_search and ignore user config.
Sibling site missed by PR #54022 — /api/console WebSocket in
HermesConsoleModal.tsx has the same buildWsUrl → stale-token → 4401
close path as the PTY and events WebSockets. Without this guard,
opening the console after a dashboard restart shows 'Console closed
(4401). auth: token_mismatch' with no recovery.
react-router v7 exports MemoryRouter from 'react-router', not
'react-router-dom'. The test was written when the repo still imported
from 'react-router-dom' (4000+ commits ago).
Loopback dashboard tabs now share one one-shot stale-token recovery path across REST 401s, the PTY socket, the structured event socket, and the shared JSON-RPC gateway wrapper. The shared client exposes only an optional close-event interception hook; the dashboard remains responsible for deciding that loopback 4401 means reload.
Constraint: Current main delegates the web gateway to apps/shared JsonRpcGatewayClient, and #54022 review requires a shared-client-compatible close-code hook plus direct ChatSidebar event-socket coverage.
Rejected: Restore the dashboard's old direct WebSocket implementation | stale against the shared JSON-RPC client and would duplicate transport behavior.
Confidence: high
Scope-risk: moderate
Directive: Keep stale-token policy dashboard-specific; the shared JSON-RPC client should expose close events without learning dashboard auth semantics.
Tested: npm --workspace web test (21 files, 106 tests); focused stale-token tests (5 files, 14 tests); npm --workspace web run typecheck; npm --workspace @hermes/shared run lint; npm --workspace @hermes/shared run typecheck; focused web eslint; git diff --check.
Not-tested: Manual browser smoke test across a real dashboard restart.
#75017: Telegram polling conflict retry used drop_pending_updates=False,
starting a new getUpdates session that immediately got 409'd by the
previous still-expiring session — creating the very conflict it was
trying to recover from. Switch to drop_pending_updates=True so Telegram
terminates stale sessions. Also add a recovery-generation guard so the
first transient getUpdates success after a retry doesn't reset the
conflict counter back to 0 (defense-in-depth from PR #75096).
#75153: The WAL-reset warning always said 'hermes update can repair'
even for git/pip/system Python installs where it can't. Now uses
detect_install_method() + recommended_update_command_for_method() to
give a context-appropriate hint (hermes update for git, docker pull for
docker, nix message for nix, generic install hint as fallback).
requires-python is >=3.11 so tomllib is always in stdlib; the
tomli fallback branch in _lint_toml_inproc was unreachable. Removes
the dependency from pyproject.toml + uv.lock and deletes the dead
try/except ImportError fallback in the code.
- write_file: encode content once, share bytes between bytes_written and
the sha256 verification (drops a second full-content encode per write)
- patch_parser: replace the except-TypeError retry around
write_file(pre_content=...) with signature-based feature detection so a
TypeError raised inside a capable implementation propagates instead of
triggering a duplicate write; tests for both duck-typing contracts
- tests: real-ops V4A BOM round-trip + _file_has_bom disk-probe guard
(the teknium1-review regression previously only covered by a fake)
- comment: document dirs_created's long-standing "parent ensured" meaning
Content that flowed through a surrogateescape decode (backend output via
patch_replace) can carry lone surrogates; a strict encode raises
UnicodeEncodeError where the old wc -c path could not. Mirrors the
existing sha256 verification encode.
Bug 1 (UTF-8 BOM loss on V4A UPDATE):
_file_has_bom() trusted pre_content for BOM detection, but the most
common pre_content provider — read_file_raw() — deliberately strips
BOMs so the agent never sees U+FEFF glyphs. Passing BOM-stripped
content through pre_content caused a false-negative: the method
returned False and write_file() silently removed the marker on rewrite.
Fix: _file_has_bom() now always probes the first 3 bytes on disk
(head -c 3), ignoring pre_content for BOM purposes. pre_content is
still used by two other consumers — line-ending detection and lint/LSP
delta computation — neither of which is affected by BOM stripping.
Bug 2 (backward compatibility):
_apply_update() called write_file(path, content, pre_content=...) as a
keyword argument. Duck-typed file_ops implementations that only
implement the two-argument write_file(path, content) contract would
raise TypeError.
Fix: wrap the call in try/except TypeError, falling back to the
two-argument form when the keyword is not accepted.
Also declare tomli in pyproject.toml (pre-existing conditional import
for pre-3.11 Python, caught by the pre-commit dep scan after staging
file_operations.py).
Tests:
Add TestV4ABomRoundTrip with two cases:
- UPDATE on BOM-bearing file preserves the marker
- UPDATE on plain file does not inject a BOM
Addresses teknium1 review on PR #55661.
write_file currently spawns up to 6 subprocesses per call:
1. mkdir -p (separate call before atomic write)
2. cat (to read pre-content for lint/BOM/line-ending detection)
3. _atomic_write (mktemp + write + mv — the essential one)
4. wc -c (to measure bytes written)
5. _check_lint_delta (post-write lint — also essential)
6. LSP snapshot (also essential)
This PR removes three of them without changing any observable behavior:
1. Fold mkdir -p into _atomic_write shell script (−1 subprocess/write)
The atomic write script already runs a single shell; adding mkdir -p
to it costs zero extra processes.
2. Add optional pre_content parameter to write_file (−1 subprocess/patch)
patch_replace and V4A _apply_update already read the file for fuzzy
matching. Passing that content as pre_content skips the redundant cat
inside write_file. Fully backward-compatible: callers that don't pass
pre_content still read from disk as before.
3. Replace wc -c with len(content.encode('utf-8')) (−1 subprocess/write)
We already have the content in memory; encoding it to get the byte count
is equivalent to wc -c for UTF-8 text.
4. Remove redundant _check_lint loop in apply_v4a_operations (−N subprocesses/V4A)
write_file already runs _check_lint_delta internally. The old code ran a
bare _check_lint(f) loop over all modified files — a re-read + re-lint
without post_content context. Now lint results propagate from write_file
via a four-tuple return, zeroing out the extra subprocesses.
Net effect:
- write_file: 6 → 3 subprocesses per call (new files)
- patch_replace: 6 → 5 subprocesses per call (pre_content skips cat)
- V4A multi-file patches: saves 1 subprocess per modified file
- A typical 4-file V4A patch drops from ~28 to ~16 subprocess calls
#71775 moved deferred single-use-token refreshes outside the pool lock
(correct — they hold a cross-process flock plus network I/O). But
_refresh_entry_impl's three terminal-auth-failure quarantine paths do a
bare read-modify-write of self._entries. Those used to run with the
caller holding self._lock; on the deferred path they run unlocked, so a
concurrent mutation between the read and the write is silently lost.
Wrap all three in 'with self._lock' (an RLock, so locked callers
re-enter safely) and correct the _refresh_pending_entries docstring,
which claimed the mutations were already self-locking.
Post-merge gate-sweep finding on the #71775 salvage (#77714).
Sibling to the acquire_lease re-select fix.
select() re-selects once deferred single-use-token refreshes complete;
acquire_lease() performed the refresh but returned its pre-refresh
answer. Since _acquire_lease_under_lock returns early exactly when a
refresh is pending (if not available: return None, pending_refresh),
a pool whose entries all needed a refresh always returned None — the
caller failed an answerable request right after the refresh succeeded.
Retry once, only when the first pass was empty and a refresh ran.
Post-merge gate-sweep finding on the #71775 salvage (#77714).
CI caught a missed caller-shape update. Both PRODUCTION callers of
_write_sse_chat_completion / _write_sse_responses were converted to
ThreadSafeAsyncQueue, but two pre-existing tests in
tests/gateway/test_api_server.py construct the writer's queue
themselves and still passed a stdlib queue.Queue.
The consumer now does 'await asyncio.wait_for(stream_q.get(), ...)',
which on a queue.Queue blocks the thread forever:
test_stream_cancelled_persists_incomplete_snapshot hung until
pytest-timeout killed it (CI reported the whole file as 'no tests
ran (timeout before collection)'). The sibling disconnect test only
survived because it pre-fills before the first await.
tests/gateway/test_api_server.py: 99 passed (was 1 failed + a 60s
hang); with the SSE/api_server suites: 147 passed.
Gate finding (/simplify-code pass): both cross-thread tests passed
loop=loop explicitly, but no production caller does — all six
(_on_delta, _on_tool_*) rely on the queue resolving its own
_loop_ref in __init__. The kwarg made the tests vacuous: a broken
_loop_ref still passed them.
Dropping the kwarg exercises the real path. Verified by mutation:
with self._loop_ref = asyncio.new_event_loop() (wrong loop), both
tests now FAIL; they passed before this change.
CI caught a split defect in this salvage: the PR's long-reasoning tail
test was kept but its production hunk was dropped as 'cosmetics'. It
isn't — cleanThinkingText runs several full-string regex passes and
reasoning grows on every streamed token, so re-cleaning the whole
accumulated string per chunk is O(n) per token / O(n^2) per stream.
Only the tail is displayed (boundedLiveRenderText caps it downstream),
so bound the input to 1.5x LIVE_RENDER_MAX_CHARS first.
Restores the one text.ts hunk from cd99e65fc (author preserved); the
italic-thinking display change and profile script from that commit
remain out of scope.
The conflict-marker strip fused test_agent_task_raises with the body of
test_failed_result_dict — restore both as separate tests (content from
the PR head, verified verbatim).
Addresses teknium1 sweeper review (2026-07-30) requiring coverage of:
1. ThreadSafeAsyncQueue.put_threadsafe() off-loop boundary: a real
daemon thread pushes into the queue from outside the owning event loop
while the consumer awaits get(), mirroring the run_conversation
worker-thread producer path. Includes a 20-concurrent-thread
no-drop regression.
2. Long-reasoning bound stability for thinkingPreview: 100k-char input
plus empty/collapsed cases must not crash and must retain the visible
tail marker inside the bounded 24k clean window.
The session event stream (api_server.py:~2236) was the one genuinely
unicode-distinct SSE writer — json.dumps(payload, ensure_ascii=False) +
.encode('utf-8'). Every other writer uses plain json.dumps. Route it
through _sse_frame(..., ensure_ascii=False) so _sse_frame is now the single
source of truth for ALL SSE frame serialization in the module (chat-
completion, responses._write_event, /v1/runs, and the session stream).
Byte-identical for non-ASCII payloads: verified against the historical
inline encoder (raw bytes preserved). The ensure_ascii=False path is now
exercised by test_sse_frame_ensure_ascii_false_reproduces_session_event_stream.
_extend _sse_frame with an explicit ensure_ascii param (default True,
byte-identical to a bare json.dumps) and route the two sibling writers
through it: _write_sse_responses._write_event and the /v1/runs event
stream. This completes the dedup PR #65009 — previously only the five
_write_sse_chat_completion sites used the helper, leaving the other two
writers on inline json.dumps with no shared shape.
No behavior change: every writer's emitted bytes are unchanged (verified
byte-for-byte, including non-ASCII payloads where the default
ensure_ascii=True matches the original inline encoders). The ensure_ascii
option is exposed so a future writer can opt into raw non-ASCII bytes
without fractalizing the format again.
Adds tests/gateway/test_sse_frame.py asserting the byte-contract
invariant between _sse_frame and the historical inline encoders.
_write_sse_chat_completion had five near-identical
f"data: {json.dumps(...)}\n\n".encode() (and one event-tagged variant)
scattered across its role/content/finish/error chunk writes. Pure
extract-method, no behavior change: encoding is byte-identical for every
call site touched.
Left the pre-serialized-string writers elsewhere (_write_event's
json.dumps(..., ensure_ascii=False) path, the /v1/runs SSE writer) alone
— routing them through this helper's plain json.dumps(data) would
silently change their unicode-escaping behavior, which is out of scope
for a pure dedup.
_write_sse_chat_completion and _write_sse_responses bridged their
stream_delta_callback queue into the event loop via
`await loop.run_in_executor(None, lambda: stream_q.get(timeout=0.5))`
in a while-True poll — a thread-pool round trip on every 0.5s tick even
when idle, plus up to 500ms of tail latency between a delta landing in
the queue and it reaching the SSE response.
Add ThreadSafeAsyncQueue (asyncio.Queue + a put_threadsafe() that wraps
call_soon_threadsafe), used by both streaming producer closures
(_on_delta, tool start/complete callbacks — all invoked from the worker
thread running run_conversation via loop.run_in_executor). Consumers
now do a plain `await asyncio.wait_for(stream_q.get(), timeout=0.5)` —
woken immediately when a delta arrives, no executor hop, no poll
interval.
Updated tests/gateway/test_sse_agent_cancel.py's 7 call sites to
construct ThreadSafeAsyncQueue inside the running loop (required, since
it captures asyncio.get_running_loop() at construction) instead of a
bare queue.Queue() at test-method scope.
Regression coverage requested in review of #74357: mock
window.hermesDesktop.onWindowStateChanged (pattern from
persistent.test.tsx) and assert minimized/hidden clears the spinner
interval while restore resumes it; also cover document.visibilityState
hidden/visible via visibilitychange.
Advance the existing animated status glyph through its DOM text node instead of React state, and pause its timer for hidden panes or inactive windows. Cover frame advancement, zero update-phase commits, and timer suspension with behavior tests.
A.Alzaro
brooklyn!
Jeongseok Kang
Aleksandr Pasevin
seref
Alex Fournier
Jeffrey Quesnelle
HexLab98
Bryan Bednarski
kshitij
xxxigm
izumi0uu
kshitij
阿泥豆
zabih-sudo
hermes-seaeye[bot]
Mykola Bashmakov