Drop the manual web.search_backend / web.backend config-reading block
that duplicated _read_config_key in web_search_registry.py. The function
now delegates directly to get_active_search_provider() (which reads the
same config keys via the registry's canonical resolver) and falls back
to _get_search_backend() only when the registry has no providers loaded.
Also updates the TestXaiWebSearchBackendPreference tests to monkeypatch
the registry instead of load_config_readonly, and adds two new tests for
the legacy fallback path (no provider registered -> _get_search_backend).
When Grok runs on xAI Responses, only swap to native server-side
web_search when the active/configured backend is xai. For Firecrawl
and other Hermes providers, keep client dispatch under a renamed wire
tool so Grok cannot hijack web_search and ignore user config.
#71775 moved deferred single-use-token refreshes outside the pool lock
(correct — they hold a cross-process flock plus network I/O). But
_refresh_entry_impl's three terminal-auth-failure quarantine paths do a
bare read-modify-write of self._entries. Those used to run with the
caller holding self._lock; on the deferred path they run unlocked, so a
concurrent mutation between the read and the write is silently lost.
Wrap all three in 'with self._lock' (an RLock, so locked callers
re-enter safely) and correct the _refresh_pending_entries docstring,
which claimed the mutations were already self-locking.
Post-merge gate-sweep finding on the #71775 salvage (#77714).
Sibling to the acquire_lease re-select fix.
select() re-selects once deferred single-use-token refreshes complete;
acquire_lease() performed the refresh but returned its pre-refresh
answer. Since _acquire_lease_under_lock returns early exactly when a
refresh is pending (if not available: return None, pending_refresh),
a pool whose entries all needed a refresh always returned None — the
caller failed an answerable request right after the refresh succeeded.
Retry once, only when the first pass was empty and a refresh ran.
Post-merge gate-sweep finding on the #71775 salvage (#77714).
The output-cap error handler already computes request_input_estimate at
line 4722 via estimate_request_tokens_rough(api_messages, tools=...).
The new compression block ~50 lines below was calling the same function
with the same inputs again. Reuse the existing local.
The output-cap retry loop reduced max_tokens by 64 tokens per attempt but
never called _compress_context(), so the compressor never fired. Input
growth (~65 tokens/attempt) canceled the savings, leaving the session
stuck at 200,001 tokens — 1 over the 200,000 ceiling.
The fix adds compression to the output-cap retry path. The compressor
drops the middle window, freeing ~50% of tokens. If compression makes
>=5% savings, the session continues; otherwise vision payloads are
stripped or the session ends with compression_exhausted=True.
Also adds CHANGELOG.md entry and bug fix report.
The bracketed-marker regex was inlined in conversation_loop.py as
re.fullmatch(r"\[...", ...) while hermes_state.py defines the same
pattern as _STALE_TOOL_CALL_MARKER_RE. Both must agree on what counts
as a stale marker — a drift here means the runtime guard silently
disagrees with the load-on-read repair and CLI purge in hermes_state.
Consolidate onto a single compiled constant (_STALE_MARKER_RE) at
module level in conversation_loop.py, with a comment noting it must
mirror _STALE_TOOL_CALL_MARKER_RE in hermes_state.py. A direct import
from hermes_state was tried first but caused a regression: hermes_state
initializes DEFAULT_DB_PATH = get_hermes_home() / 'state.db' at module
import time, which breaks tests that monkeypatch get_hermes_home() to
return a str (test_slash_worker_accepts_profile_home).
Follow-up to PR #78175 (@JoaoMarcos44).
Local tool-call templates can emit a bare bracketed token (e.g. "[memory]")
as assistant content alongside a function call. The loop treated that
protocol scaffolding as visible content: it got cached as the post-tool
fallback, and when the next turn came back empty, the marker was replayed
as the final response and written into the persisted transcript. Later
context compaction preserved that history, letting the model repeat the
marker in subsequent turns.
Detect content that is only a bracketed marker (`[name]`) when the
response also carries tool_calls, and drop it before it can be cached
or persisted. Scoped narrowly: only fires alongside tool_calls, so a
genuine final response of "[memory]" without a tool call is unaffected.
Empty model.base_url plus a runtime custom-provider URL was treated as a
route mismatch, so gateway session-reset banners dropped model.context_length
and fell back to the Qwen family default (131K) while /status still showed
the configured 262K pin.
Qwen3.8 Max is live on both OpenRouter and the Nous portal
(qwen/qwen3.8-max, 1M context, 131K max output). Per the
newest-max-replaces-last-max convention, it takes qwen3.7-max's slot
in both curated lists.
- hermes_cli/models.py: OPENROUTER_MODELS + _PROVIDER_MODELS[nous]
swap qwen/qwen3.7-max -> qwen/qwen3.8-max
- agent/model_metadata.py: DEFAULT_CONTEXT_LENGTHS entry for
qwen3.8-max at 1,000,000 (verified against OpenRouter live
metadata and Nous /v1/models 2026-08-03)
- tests/test_empty_model_fallback.py: swap incidental catalog fixture
to the surviving slug
- website/static/api/model-catalog.json: regenerated
Pricing snapshot skipped: both routes bill via official_models_api
(live pricing), verified with resolve_billing_route. Reasoning
timeout floor already covered by the qwen3 prefix (180s).
Cherry-picked from PR #58560 by @itsflownium, adapted to current main
(_getenv instead of os.getenv). Moves ANTHROPIC_API_KEY check ahead of
Claude Code credential file and credential_pool auto-discovery so an
explicitly configured key is never shadowed by auto-discovered OAuth.
Fixes#58546
Simplify-pass finding: the safety note still cited 'skills edited' as a stable-tier input whose change mismatches the rebuilt prefix — after this PR a skill edit changes only the volatile tail (that's the point). Swap the example for genuinely stable-tier inputs.
The skills index is runtime-mutable: the agent adds and patches skills mid-session, so it is not byte-stable. Keeping it in the stable band breaks that band prefix-cache contract, because every skill edit changes the stable band and invalidates the entire cached prefix in front of it. Move it to the front of the volatile band so the stable scaffold (identity, tool guidance, model guidance) stays cacheable across skill edits.
The PR's concurrency wrapper splits call_llm into a semaphore-guarded
entry + _call_llm_impl; main added extra_headers to call_llm's
signature after the PR's base, so the split has to forward it too
(dropped silently otherwise — Azure Foundry and custom-endpoint
callers set it).
A successful compaction frees the largest allocation a long session ever
drops (the compressed-away message dicts), but Python's arena allocator
keeps those pages in the heap — RSS retains the pre-compaction
high-water mark until exit. #76905's trim_memory lifecycle covers the
gateway/TUI housekeeping loops but not the CLI compression path.
Call trim_memory(reason='post-compression') at the compression-success
point in ContextCompressor.compress(), following the house pattern
(lazy import in try, debug-level log on failure). The helper is
glibc-gated, config-gated and rate-limited, so it is a safe no-op on
other platforms and cannot fail compression.
Re-expresses the intent of #70782 (JonthanaHanh), which reached for a
bare gc.collect(); trim_memory is the house mechanism and already
wraps a collect.
Review follow-up on the #30223 salvage: the original changed the base
cooldown from 60s to 1800s, benching the primary for 30 minutes on the
FIRST 429 (30x regression in primary-restore latency) and breaking the
existing test_rate_limit_exhaustion_keeps_60s_cooldown contract.
Keep upstream's 60s base and escalate per consecutive rate-limit:
60s -> 2m -> 4m -> 8m -> ... capped at 4h. Counter still resets on
successful primary restore (cicae's mechanism, unchanged).
New tests: escalation doubling, 14400s cap, reset-on-restore.
Existing 60s contract test passes UNCHANGED. Mutation-checked:
escalation disabled -> 2 fail; reset disabled -> 1 fails.
Replace the fixed 60-second cooldown with exponential backoff:
30min → 1h → 2h → 4h cap.
The counter is reset by restore_primary_runtime on successful
primary-provider recovery, so the backoff is strictly for
consecutive failures within a single degradation window.
Closes#29702
Salvage of #71282 (Fixes#71281): a routable-but-dead endpoint (corp
LAN address while off-VPN) blackholes TCP SYNs, so every probe in the
model-metadata waterfall waits out its full connect timeout — 20+
seconds of stall per startup across detect_local_server_type,
fetch_endpoint_model_metadata, and the per-model probes.
A module-level blackhole cache keyed on host:port is populated when
any probe observes a ConnectTimeout (httpx or requests; read timeouts
deliberately excluded — an accepted connection is not a blackhole) and
consulted at the top of each guarded function. 30s TTL: long enough to
collapse one startup burst, short enough that VPN recovery is picked
up without a restart. Guard ordering: blackhole check -> disk L2 ->
HTTP waterfall, and a blackholed leg aborts the remaining legs instead
of letting each stall in turn.
Squash of the PR's two real commits (the branch's merge commits made
it un-rebase-merge-able; content verified identical via merge-tree).
CI slice 3/7 failures: run_conversation tests pass MagicMock base_urls
through the metadata probe path; re.sub raised TypeError where the old
code let non-strings flow through. Preserve that contract.
fetch_endpoint_model_metadata's generic (non-LM-Studio) /models fetch and
its llama.cpp /v1/props context-length follow-up built request URLs
straight from the unrewritten candidate, unlike every other local-probe
site. Both retained the multi-second dual-stack IPv6 connect penalty
that _localhost_to_ipv4() exists to skip (measured on macOS: localhost
32.9ms vs 127.0.0.1 0.1ms on a dead port; ~2s on Windows). normalized
stays the cache key so caching behavior is unchanged; only the outbound
request target is rewritten.
Re-derived from PR #61528 onto current main (original no longer applied
cleanly).
SubdirectoryHintTracker re-injected identical context files whenever the same
AGENTS.md was reachable through more than one path. Symlinked shared
workspaces, hardlinks, and timestamped backup copies all alias a single file,
so a normal session could ship the same 8KB of instructions two or three
times. Nothing deduped it and nothing excluded directories that only ever
hold copies.
Two changes:
* Track a sha256 of every injected hint body. Repeat content is skipped, and
the working directory's own context file is seeded at construction so the
copy prompt_builder already loaded at startup is never sent again.
* Skip directories that hold copies rather than authoritative context
(backups, node_modules, venv, site-packages, .git, .Trash, vendor, caches).
Screening is relative to working_dir, so a project that legitimately lives
under vendor/ keeps discovering its own subdirectory hints.
Measured on a real session that touched a symlinked shared workspace:
3 injections / ~24,000 chars before, 1 injection / 8,112 chars after.
14 new tests cover symlink aliasing, byte-identical copies, working-dir
seeding, distinct content still being injected, each excluded directory name,
excluded ancestors, and the working-dir-inside-excluded-name case.
Cross-PR interaction fix: #77714 (salvage of #71775) changed
_available_entries to return (available, pending_refresh) while #77631
(salvage of #67642) added next_available_at() which still truthiness-
tests the bare return. A non-empty tuple is always truthy — even
([], []) — so the reset-aware gate silently returned None ('no wait
info') for every exhausted pool, disabling the feature #77631 shipped.
Unpack the tuple and test the available list.
Also adapts the lock-probe test for the RLock introduced by #77714
(same-thread non-blocking acquire always succeeds on an RLock; probe
from a helper thread instead).
Review folds on the #71775 salvage (dossier findings 1+2):
- self._lock becomes an RLock and the mutation primitives
(_replace_entry, _persist) are now self-locking, so the deferred
single-use-token refresh path — which deliberately runs its
cross-process flock + OAuth network I/O OUTSIDE the pool lock —
still serializes its pool mutations against concurrent
select()/rotation. In-lock callers re-acquire reentrantly.
- Dropped _refresh_pending_entries' redundant second _replace_entry:
_refresh_entry already merges the refreshed entry internally.
Adds tests/agent/test_credential_pool_deferred_refresh.py pinning both
invariants: select() must NOT hold the lock during the refresh window
(the PR's whole point), and the post-refresh mutations MUST contend on
the lock (blocking-thread probe).
select() and acquire_lease() held self._lock during the entire
_available_entries() loop, which for openai-codex and xai-oauth providers
includes a cross-process file lock (_auth_store_lock) plus OAuth token
refresh HTTP POST. The lock timeout can exceed 20 seconds, blocking all
credential pool consumers across every gateway thread and subagent.
Collect single-use-token refresh entries under the lock, then execute the
refreshes outside it. On success the refreshed entry is merged back into
the pool and re-selected. Non-single-use providers (anthropic, nous)
continue refreshing inside the lock since their refresh is a simple HTTP
POST with no cross-process coordination.
Review fold on the #67642 salvage: next_available_at() called
_available_entries() — which prunes DEAD entries, syncs tokens, and
persists — and iterated self._entries with no lock, racing concurrent
select()/rotation exactly as has_available()'s comment warns. Wrap the
method body in self._lock and pin it with a non-blocking-acquire probe
test.
restore_primary_runtime retries the primary every turn once the 60s
transient cooldown clears. For subscription-window limits (Claude
Pro/Max 5h windows, Codex weekly caps) the reset is hours or days away,
so every retry is a guaranteed failure costing two provider switches
and two prompt-cache invalidations per turn.
Add CredentialPool.next_available_at() (earliest reset across exhausted
entries; None when available now or no reset info) and gate the restore
on it: skip while the primary's pool says nobody can serve, restore on
the first turn after the reset elapses. Fail-open: any gate error or
missing reset info falls through to the existing per-turn retry, so
recovery can never be later than today. Cross-provider fallbacks
consult the PRIMARY's pool (not the attached fallback pool), reusing
the loaded pool for the existing rebind to keep auth reads at one per
restore.
The 2-line alias had zero production consumers (web_server calls
get_usage_breakdown directly). Tests rewired onto the real API; the
contracts they pin are unchanged. Stale test docstring fixed.
The curator LLM review loop (_run_llm_review) built its AIAgent without
enabled_toolsets, so it advertised the full default catalog (~30 tools plus the
context_engine lcm_* family) on every call. The fork uses only four tools, fixed
by its own system prompt, with no dispatch path to the rest, so ~26 tool schemas
shipped on every request as dead weight: ~7K input tokens per call on a loop that
makes 50-100 calls per consolidation pass.
Restrict the fork to enabled_toolsets=["skills", "terminal"], the same tools the
prompt already names. Behavior-neutral: the prompt held the model to these tools
and nothing routed calls to the others. Mirrors the background_review fork
(background_review.py:788-794). Call-site only; AIAgent already forwards the kwarg.
Adds test_review_fork_restricts_toolsets_to_skills_and_terminal (captures the
constructor kwarg) and test_review_fork_toolset_surface_is_skills_plus_terminal
(pins the resolved surface).
Rebase fold on the salvaged gate:
- is_trivial_prompt/TRIVIAL_PROMPT_RE move to agent/memory_provider (the
ABC both the core gate and providers already import) — one source of
truth; honcho's _TRIVIAL_PROMPT_RE now aliases it, turn_context and the
queue_prefetch_all warm path (a sibling site main grew after the PR's
base) both use it
- tests: gate tests at the prefetch call site (mutation-checked), shared
classifier tests incl. prefix-collision guards (k8s/yolo/note/supper),
and honcho dialectic-machinery tests re-driven with a substantive prompt
("hello" became trivial by design — those tests exercise thread cadence,
not the classifier)
Salvage of PR #25350 (commits 88ffede2d + 2b848a0b2 + 3136dc63a, squashed
and ported): the run_agent.py prefetch site the PR gated has since moved
into agent/turn_context.py's build_turn_context(), so the trivial-query
gate lands there instead.
- Gate the per-turn memory_manager.prefetch_all() on a trivial-prompt
check so greetings/acknowledgements ('hi!', 'thanks', 'ok') no longer
block the turn on provider network round-trips or inject stale context.
- Extend honcho's _TRIVIAL_PROMPT_RE with greetings and a trailing
punctuation class so 'hey!' / 'hello.' classify as trivial.
- Add honcho classifier tests for greeting forms.
Simplify-pass fold: the four copy-pasted .replace blocks meant a\nfifth pinned statement could forget its strip line — a hard 'no such\nindex' crash on read-only DBs, the exact bug the fallback prevents.\nLoop over the attribute names instead.
The INDEXED BY pin is a hard dependency -- SQLite raises 'no such
index' when the named index is missing. That happens in production:
the web dashboard's usage analytics (_get_usage_analytics,
_get_models_analytics) open state.db read_only=True, which skips
_init_schema, so a DB last written by a pre-index version has no
idx_messages_assistant_calls_by_session and every insights call
crashes with OperationalError (reproduced E2E).
Probe sqlite_master once in __init__ and strip the pin from the four
prepared statements when absent -- identical rows, optimizer-chosen
plan, no crash. Replaces the change-detector test that froze the
crash as intended behavior with a fallback-equivalence test.
Review follow-up (#67341): on a freshly initialized state.db (before
ANALYZE has run) the source-filtered branches of _get_tool_usage /
_get_skill_usage did not select idx_messages_assistant_calls_by_session
— the optimizer drove from idx_sessions_source_id and probed each
session's messages via idx_messages_session_active, scanning non
tool-call rows. Pin the index with INDEXED BY on all four fixed-predicate
branches so the plan is deterministic for both the unfiltered and
source-filtered scopes without depending on statistics.
Safe because the index is declared in SCHEMA_SQL (created by every
read-write SessionDB._init_schema) and every InsightsEngine caller opens
a read-write SessionDB; read-only attachments (which skip schema init)
are never used for insights.
Extract the four queries into class constants and add tests: query-plan
coverage for both scopes without ANALYZE, row-level equivalence between
pinned and un-pinned forms, and an assertion that INDEXED BY fails loudly
if the index is absent.
Review follow-up on the #56798 salvage: the gate shipped fully dormant
(no provider profile sets supports_prompt_cache_key, no production
caller passes it, and no plain 'openai' profile exists to set it on) —
AGENTS.md rejects dead code wired in without E2E proof.
Activate the one endpoint where the field is first-class: exact-host
api.openai.com (OpenAI documents prompt_cache_key; GPT-5.6+ docs
recommend it for cache routing). Deliberately NOT substring matching —
Azure/OpenAI-compat endpoints may reject unknown fields and stay
opt-in via the flag. 4 new tests (imply + 3 spoof/proxy/Azure
negatives); mutation-checked (substring-weakened host check fails the
spoof tests).
Empty content retries previously fired back-to-back with no delay,
wasting up to 3 rapid API calls, and could not be cancelled mid-wait.
Apply the same jittered_backoff() already used for rate-limit and
API-error retries, sleeping in small increments so a user interrupt
aborts the wait instead of blocking until it elapses.
Fixes#35230
Review fold on the #76341 salvage: the substring test ('gh' in
source.lower()) classified GH_TOKEN and GITHUB_TOKEN as gh_cli, so a
user's env-var-specific suppression was silently bypassed (and
suppressing gh_cli silently dropped env tokens). Pre-existing bug on
main, but the PR's early gate makes the classification decide whether
the exchange runs at all. Match resolve_copilot_token's exact
'gh auth token' sentinel instead.
Adds 3 regression tests: env-var suppression gates the exchange,
gh_cli suppression doesn't swallow env tokens, all-sources suppression
skips the resolve subprocess entirely. Also corrects the ~13s comment
(actual worst case ~35s: 3x10s timeouts + 4.5s backoff).
The all-sources suppression gate now runs before resolve_copilot_token(),
which shells out to `gh auth token` (~30ms) on every pool load. A user
who suppressed every copilot source (hermes auth remove copilot gh_cli
suppresses gh_cli + all env variants) still paid the subprocess spawn on
every load — model picker open, /model, agent startup.
Enumerate the same source space credential_sources._remove_copilot_gh
suppresses and bail before any work when all are suppressed. Measured:
model.options payload build drops from ~0.46s to ~0.26s cold for an
all-suppressed user; resolve_copilot_token() is no longer called at all.
The copilot branch of _seed_from_singletons ran the suppression gate
_after get_copilot_api_token(), which retries the network exchange 3x
with backoff (~13s worst case). A source the user already suppressed
(hermes auth remove copilot gh_cli) still burned the full exchange dead
time on every pool load — model picker open, /model, agent startup —
only to have the entry discarded afterwards.
Move the _is_suppressed() gate ahead of the network call, matching the
early-gate pattern every other singleton branch uses. Suppressed copilot
sources now skip the exchange entirely. Measured: model.options payload
build drops from ~13s to ~0.2-0.4s for a user with copilot suppressed.
Add regression test test_load_pool_skips_exchange_for_suppressed_copilot
asserting the exchange is never invoked for a suppressed source.
OpenCode Zen's relay rejects the Anthropic-style content block format
that cache markers produce (content becomes a block array instead of a
plain string), causing HTTP 400 with "content must be string, not block
array" for DeepSeek models.
Reverts the DeepSeek addition from commit 6b6435a874 while preserving
the Qwen/Alibaba caching path which continues to work.
Fixes#77217
_convert_user_message hand-inlined the same blank-text-filter +
cache_control-relocation + placeholder-fallback logic that
_fix_blank_text_blocks_in_list (added in the cherry-picked commit)
implements as a reusable helper. Replace the inline copy with a call
to the helper, eliminating ~35 lines of duplication.
Follow-up fix on top of PR #77134 by @pooyan6.
Root cause: two independent bugs in convert_messages_to_anthropic()
(agent/anthropic_adapter.py), the final conversion step before every
Anthropic messages.create() call, both producing HTTP 400 "text content
blocks must contain non-whitespace text":
1. _ensure_leading_user_turn() synthesized a filler user turn with
content [{"type": "text", "text": " "}] (a single space) whenever the
built payload didn't start with role=user (e.g. after context
compaction leaves a leading assistant summary). The space is itself
whitespace-only, so the guard traded a "leading assistant turn" 400
for the "text content blocks" 400 it now hits. Fixed to reuse the
existing non-blank _EMPTY_TEXT_PLACEHOLDER ("(empty)").
2. _convert_user_message() filtered blank text blocks from list-type
user content with an all-or-nothing check:
all(blank for b in blocks if b.type == "text"). This is vacuously
true when a message has zero text-type blocks (silently destroying
valid non-text blocks like images/documents it never inspected), and
false as soon as any single text block is non-blank — which let a
*sibling* blank text block sit untouched next to valid content and
reach Anthropic as-is. Replaced with per-block filtering (mirroring
the assistant-side logic already in _convert_assistant_message),
preserving all non-blank/non-text blocks and relocating any
cache_control marker carried by a dropped block.
Also added _scrub_blank_text_blocks(), a final defense-in-depth pass run
as the last step of convert_messages_to_anthropic() (after every other
transform, including nested tool_result content lists) so a blank text
block from any current or future producer never reaches the wire. It
logs only structural metadata (message index, role, content location,
block index/type) — never message text, tool arguments, tokens, or
credentials.
An earlier local patch to sanitize_api_messages() (agent_runtime_
helpers.py) attempted to fix this by rewriting blank assistant content
before the OpenAI->Anthropic conversion step, but the real leaks were
introduced downstream of that sanitizer, inside the Anthropic-specific
converter itself — the patch never touched the actual defect and has
been fully reverted (agent_runtime_helpers.py is back to its committed
state; verified via `git diff` showing no changes).
Verified against a real Telegram message end-to-end: the gateway no
longer produces the "text content blocks must contain non-whitespace
text" error on a fresh conversation turn.
Testing:
- 9 new end-to-end regression tests in test_anthropic_adapter.py
(TestFinalPayloadHasNoBlankTextBlocks) covering content="",
content=" ", content=[{"type":"text","text":""}], mixed blank+valid
text, blank text next to a valid tool block, an assistant tool-call
message with blank content, the leading-synthesized-user-turn case,
and a blank text block nested inside a tool_result's own content list.
- Fixed one pre-existing test that had asserted the broken " " filler
behavior as correct.
- Full tests/agent/ + tests/run_agent/ suite (4671 tests) run against
both the patched tree and a stashed pre-fix baseline: identical 148
pre-existing failures in both runs (unrelated subsystems — codex
app-server integration, credential-pool interrupt handling, OpenAI
client lifecycle), zero failures unique to either side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Post-tool compression path passed context_compressor.last_prompt_tokens (0 in the no-usage
fallback) to _compress_context instead of the overhead-aware _real_tokens computed just above
— same tool-blind bug as the overflow handlers (upstream PR #77169 review, teknium1). Also adds
production-path regression tests asserting the 413, context-overflow (two wordings), and
Anthropic long-context recovery handlers pass estimate_request_tokens_rough(..., tools=...)
(sentinel-patched) to _compress_context.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root fix (Option A) for design-session "Context compression exhausted" crashes. The three
compression-retry handlers after an API overflow/413/long-context error (conversation_loop.py
~4229/4488/4747) passed the tool-BLIND messages-only estimate (approx_tokens) to _compress_context,
so hermes-lcm's forced-overflow recovery armed on the message count and missed overflows driven by
tool-schema/system overhead. Now they pass estimate_request_tokens_rough(api_messages, tools=...)
— the same overhead-aware estimator already used at :4580 — so recovery arms on the TRUE request
size; LCM's _overflow_recovery_assembly_cap self-subtracts the overhead so the full request fits.
Empirically validated on real failed session 6dddf1a67b76 (LCM engine, floor=24000/cap=248000):
observed 256,359 >= 248,000 -> arms; recovery 231,313->208,559 msg-tokens -> full request
233,605 < 272,000 FITS. Prior messages-only path did NOT arm (231K < 248K) and crashed.
Durable copy: ~/.hermes/local-patches/optionA-overflow-overhead-aware.patch (survives hermes update
reset). Upstream PR pending. classify_api_error call at :3667 intentionally unchanged (not recovery).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Simplify-pass follow-up on the #69653 salvage: the original code built
these patterns from a name loop; hand-expanding them into 10 literals
lost that single source. Tag-name tuples restore it (adding a 6th
reasoning tag is now a one-place change), and the gnarly named-function
pattern regained a pointer to its step-1c rationale. Byte-equivalence
of every rebuilt pattern verified programmatically (alternation-order
neutrality probed: the \b and > anchors make order irrelevant).
strip_think_blocks passed the same response-scrubbing strings through
re's pattern dispatcher on every response. Skills Guard repeated the
same work for 121 patterns against every scanned line.
Compile the existing expressions once and reuse Pattern.sub/search. Keep
each generic tool-call tag in its own paired expression so mismatched
openers retain their payload while existing stray-closer cleanup remains
unchanged.
Part of #33208
Salvaged from #32713 by @ErnestHysa.
Co-authored-by: ErnestHysa <takis312@hotmail.com>
A session's YOLO bypass lived only in the in-memory
tools.approval._session_yolo set (or the process-frozen --yolo env
var), so resuming a session in a fresh process silently reverted the
user's /yolo ON — dangerous commands started prompting again.
Persist a yolo_mode flag in the session row's model_config JSON and
restore it on every CLI resume path:
- SessionDB.set_session_yolo() merges the flag into model_config
(same lineage-preserving merge as update_session_runtime_lock);
SessionDB.session_yolo_enabled() reads it back, false on any parse
failure.
- /yolo toggle persists ON and OFF through the new helper; the
compression/branch session-id rotation carries the flag onto the
continuation row.
- --yolo launches record the flag at session creation (agent_init),
and a /yolo toggled before the lazily-created row exists is carried
into the creation-time model_config (_ensure_db_session).
- HermesCLI._restore_session_yolo() re-enables the bypass on startup
--resume/-c, the deferred init path, and mid-chat /resume, with a
visible '⚡ YOLO mode restored from session' notice. No-op under a
frozen process-wide --yolo and never enables on absent/garbage flags.
When an approval prompt expired without a response, every CLI-side path
collapsed the timeout into the same 'deny' choice as an explicit user
refusal, so the agent was told the user denied the action when the user
simply never answered. The gateway wait already distinguished the two
('timed out without user response... Silence is not consent.'); this
brings the CLI/TUI/ACP surfaces to parity.
- prompt_dangerous_approval(): input()-path expiry now returns a distinct
'timeout' choice (still fail-closed).
- cli.py _approval_callback + hermes_cli/callbacks.py approval_callback:
deadline expiry returns 'timeout' instead of 'deny'.
- check_all_command_guards / _run_approval_gate CLI tails: 'timeout' maps
to outcome='timeout' with a 'timed out without user response... Silence
is not consent.' BLOCKED message (matching the gateway wording);
explicit deny keeps outcome='denied' and gains user_consent=False for
shape parity.
- computer_use: 'timeout' verdict threads through the CLI adapter and
yields a 'prompt timed out — the user did not respond' error instead of
'denied by user'.
- ACP permissions bridge: FutureTimeout returns 'timeout' (other failures
still 'deny'); elicitation maps 'timeout' to 'cancel' like the gateway's
unresolved outcome; codex wire mapping documents deny/timeout→decline.
- write_approval already treats unknown choices as 'stage, not drop', so
a timeout now stages the memory write instead of silently refusing it.
Every timeout path remains fail-closed — the action never runs; only the
classification reported to the agent changes.
The fail-fast admission path (bounded compress pool, F6) only logged a
WARNING; in the compression-attempt telemetry stream a wedged pool
looked like compression simply stopped being attempted. Emit the
existing attempt telemetry with failure_class='pool_saturated'
(commit_status=aborted, split_status=aborted) on refusal, following
_emit_compression_attempt_telemetry's existing call shape. Regression
extends the F6 saturation test (sabotage-verified).
The fence-cancel poll loops (sync host wait in conversation_compression,
async hygiene wait in gateway/run) spun at 1kHz while the worker held
the fence through its lock-setup window — which rides SessionDB write
patience and can last seconds. 25ms keeps sub-tick cancel latency
without the spin.
The compression heartbeat's terminal 'context compression completed'
stamp force-persists against the PARENT session id (agent.session_id at
stamp time). After the out-of-place rotation the parent is archived but
kept advertising a fresh last_activity_at + terminal label forever.
Clear the parent row's activity labels best-effort after a committed
rotation (keeps last_activity_at so idle clocks stay continuous; the
child carries live labels). Regression asserts the archived parent's
labels are cleared while the child's lineage is intact
(sabotage-verified).
revoke_commit_admission() used to invoke the holder-qualified lease
release unconditionally — including while an admitted commit was still
mutating SessionDB — letting a second compressor acquire the durable
lock mid-commit and interleave with the first commit's writes.
The admission_revoked flag store stays lock-free, but the lease-release
decision now coordinates with the fence lock:
- revoke acquires the fence lock non-blocking; on success no commit can
be in flight (an admitted commit retains the lock until finish_commit)
and the release runs immediately, still under the lock so a racing
begin_commit cannot slip between the check and the release.
- on failure the release is deferred: finish_commit() re-checks
_admission_revoked and performs it AFTER the commit completes (prompt
even if the worker thread is later parked), and the begin_commit
refusal path does the same for a revoke that lost the race to a
transient lock-setup/cancel boundary. All paths are idempotent with
the worker's own outer cleanup (DB release is holder-qualified).
Invariant encoded + tested: no second compressor can acquire the durable
lock while an admitted commit is still mutating; after a post-revoke
commit finishes the lease is released promptly. Both regressions
(revoke-during-commit deferral, revoke-before-commit immediate release +
refused begin_commit) are sabotage-verified.
Rebase onto origin/main brought in 'let explicit interrupts cancel safely',
which predates this branch's pooled progress-timeout + F1-F6 fence rework.
Reconcile the two:
- begin_commit(cancel_event) re-checks the hard-cancel Event under the
fence lock again (lost in the mechanical rebase).
- compress_context: restore aux_interrupt_protection around the summary
call, the post-return frozen-cause AuxiliaryExplicitCancellation check,
and the full rollback/telemetry handler for explicit interrupts.
- run_agent._compress_context: recreate the per-attempt fence registration
(_active_compression_commit_fence) that hard_interrupt() uses to
serialize cancel admission, and thread that exact fence through both
the direct and pooled paths (run_compress_context_with_progress_timeout
now accepts an external fence).
- test: the pooled worker isolates the live transcript (F3), so the
hard-interrupt rollback regression mutates the engine's input snapshot
rather than reaching around it to the caller's list.
Both progress-aware waits (sync compress wrapper and gateway session
hygiene) slept a FULL idle interval and only then compared progress, so
progress early in an interval let silence approach 2x the configured
idle timeout before the waiter noticed. Compute each wait slice as
idle_timeout - elapsed_since_last_progress instead.
Regression: a worker that reports progress early and then goes silent is
timed out in ~1x the idle budget, not ~2x.
PR #76354 review, 'idle timeout can allow nearly twice that silence'.
The process-wide 4-worker pool retained the stdlib executor's unbounded
queue: four hung summaries wedged every slot, a fifth compression queued
silently, waited out its whole budget without starting, and remained
eligible to run later as an expensive stale job whose fence was already
cancelled (the first fence check used to sit AFTER the summary call).
- Bounded admission: submission fails fast (messages returned unchanged,
loud warning) when all pool slots are occupied; slots are freed by a
future done-callback. Recovery contract documented at the constant: new
work fails fast while wedged, wedged workers are fence-cancelled and
restore service when they return; a worker that never returns costs its
slot — bounded, observable degradation instead of unbounded queueing.
- Not-yet-started futures are cancel()ed on timeout.
- The cancelled fence is checked BEFORE any expensive summary work, both
in the pooled wrapper (stale queued job) and inside compress_context
(pre-summary gate), so a stale job never burns an LLM call or acquires
session state.
Saturation regression: 4 event-blocked summaries wedge the pool, a 5th
submission fails fast (asserted while the four are provably still
blocked), the refused job never runs after worker recovery, and a fresh
submission after recovery succeeds.
PR #76354 review, blocking finding 6 / merge gate 7.
A host timeout previously left the timed-out worker holding the durable
per-session compression lock AND refreshing its lease indefinitely, so a
truly hung summary blocked every later compression attempt; and a LATE
successful summary could clear the failure cooldown the host had just
recorded.
Transplant the lease-cancellation invariants from PR #71569
(@ciabata-git): the worker publishes an idempotent, holder-scoped release
hook on the fence once it owns the durable lock (begin_lock_setup /
register_cancelled_lock_release close the acquire→publish race), the
refresher start is serialized against the release path, and the host
invokes the hook on idle timeout, hygiene timeout, and every unwind
(revoke_commit_admission now also releases). ABA safety: the SessionDB
release is holder-qualified (DELETE ... WHERE holder = ?), so a stale
release can never free a replacement holder's lease.
State ordering: the compressor consults a fence-cancellation check BEFORE
clearing the failure cooldown, so a late worker cannot undo the host's
timeout cooldown; the check is installed only for the fenced call and
removed in a finally.
Regression implements the reviewer's exact 5-step scenario: summary
blocked indefinitely → host timeout → a NEW compressor acquires the
durable lock while the old summary is STILL blocked → old worker released
→ it cannot clear cooldown, release the new holder's lease, or publish
stale state.
PR #76354 review, blocking finding 4 / merge gates 4 + 5.
Co-authored-by: ciabata-git <ciabata-git@users.noreply.github.com>
The pooled worker closure captured the caller's live `messages` list and
compress_context explicitly supports plugin/legacy context engines that
mutate that list in place — so after a host timeout, a late engine could
rewrite the live conversation (roles, ordering, persisted content)
concurrently with the resumed turn.
The worker now deep-snapshots the transcript on the worker thread before
any engine code runs; the caller's list object is never handed to pooled
code. Results reach caller-visible state only through the returned value
of an ADMITTED commit (the host discards results on timeout/cancel), and
durable SessionDB mutation was already gated behind the commit fence.
No-op passes map the unchanged snapshot back to the caller's original
list so identity-based no-op detection and flush dedup keep working.
Document the thread-safety contract for context-engine and
memory-provider extension points (they now run on pooled threads) in the
module docstring and the context-engine plugin guide.
Regression: an in-place-mutating engine plus host timeout proves the
caller's live transcript is byte-identical WHILE the worker is still
blocked inside the engine (released only after the assertions).
PR #76354 review, blocking finding 3 / merge gate 3.
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.
Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.
Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.
Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.
PR #76354 review, blocking finding 2 / merge gate 2.
begin_commit() retains the fence lock until finish_commit(), so a hung
SessionDB commit made try_cancel_before_commit() return None forever and
the host spun ahead of the overrun-warning loop — a genuinely hung commit
stayed unbounded AND silent. Add a lock-free phase marker (threading.Event
set inside begin_commit while the lock is held, readable without it) and
break the host spin on commit_in_flight so the bounded overrun loop — and
its WARNING + on_commit_overrun surfacing — is reachable WHILE the commit
is still blocked. Applies to both the sync compress wrapper and the
gateway session-hygiene wait.
Regression asserts the warning and callback fire while the event-gated
fake commit is still blocked; the test releases the worker only after
those assertions (addresses helix4u's released-before-asserting callout).
PR #76354 review, blocking finding 1 / merge gate 1.
Heartbeat write discipline for the durable SessionDB activity projection:
- Pin the cadence in a named constant
(SESSION_ACTIVITY_HEARTBEAT_MIN_INTERVAL_SECONDS = 60s, contract >= 30s,
deliberately config-independent so no compression.*/agent.* setting can
turn the heartbeat into a high-frequency writer on the contended
SessionDB write path).
- The write already rides the standard _execute_write patience path via
SessionDB.touch_session_activity — verified, now documented in the
docstring.
- Best-effort hardening: a failed heartbeat write never raises into the
agent loop; the bare 'pass' becomes an explicit debug log with traceback.
- Tests: direct proof that a heartbeat DB failure doesn't propagate,
the cadence constant is pinned >= 30s, and the rate limiter keys off
the shared constant (boundary tested on both sides of the window).
The post-begin_commit() waiter previously called unbounded future.result(),
so the advertised compression.context_total_ceiling_seconds was silently
unenforced for commit-phase hangs. The commit still must complete (abandoning
an in-flight SessionDB mutation would diverge live messages from durable
state), but the wait is now bounded in increments against the remaining
ceiling: on ceiling breach the overrun is logged (WARNING escalating to
ERROR), surfaced once through the user-visible warning channel via the new
on_commit_overrun callback (wired to _emit_warning in run_agent.py), and the
host keeps waiting in bounded slices until the commit finishes.
Documented guarantee (config comment + docs, en/zh): summary phase bounded
by the ceiling; commit phase logged + surfaced if it exceeds it — never
silently hung, never abandoned mid-commit.
Test updated to assert the surfacing fires (previously accepted a silent
over-ceiling wait); adds coverage that a raising overrun callback cannot
break the commit wait.
Once begin_commit() wins, SessionDB mutation cannot be fence-cancelled;
document that context_total_ceiling_seconds covers the summary phase only
and pin the hang-wait contract in tests.
Host progress timeout leaves compress_context running on a daemon worker while
the live turn continues. Latch heartbeat silence on fence cancel or terminal
timeout/cooldown provenance so a later UNKNOWN stamp cannot re-arm
agent.compression and poison stall clocks.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
skill_view re-sent full skill content on every call: ~286k tokens of
verbatim repeat views in a 400k-msg production window (one session
loaded the same skill 9 times), and a single repeat view of a large
skill costs ~25k tokens.
Mirrors read_file's proven unchanged-stub pattern: a per-task cache
keyed on (resolved name, file_path) with an mtime+size fingerprint of
the served file. On a repeat view of an UNCHANGED file, return a short
stub pointing at the earlier result. This does NOT violate the
skills-are-loaded-fully rule — the stub only ever replaces content
that is already fully present earlier in the same conversation, and:
- any on-disk change (patch, external edit) invalidates the entry;
- context compression clears the cache (wired next to
reset_file_dedup in conversation_compression.py) so post-compression
re-views return full content;
- setup-needed views are never deduped (readiness can change without
the file changing);
- no task_id -> no dedup; caches are task-isolated; 200-entry cap.
Live E2E: repeat view of hermes-agent-dev 99,739 chars -> 374-char
stub.
The delivery worker is a daemon thread, so a short-lived process
(hermes chat -q, a cron session) could exit right after firing
on_session_end — silently dropping the headline event. Register a
bounded atexit flush (5s) when the worker starts: a dead endpoint can
delay exit slightly, never hang it.
Live-verified: hermes chat -q now delivers on_session_start,
post_tool_call, and on_session_end to a real receiver; regression test
runs a subprocess that exits without flushing (sabotage-verified).
- delivery_id is now generated once per firing and used for both the
X-Hermes-Delivery header and the signed body's delivery_id field —
previously they were two different uuid4s, breaking receiver-side
dedupe as documented.
- 3xx responses are no longer followed: urllib's default redirect
handler converts a redirected POST into a body-less GET, silently
dropping the signed payload. Redirects now log a misconfiguration
warning and count as delivery failure (no retry).
- Docs: receiver-side replay-protection guidance (dedupe on
delivery_id, timestamp freshness window) + redirect semantics.
- Tests: 5xx retry count, redirect-not-followed (sabotage-verified),
header/body delivery_id equality.
The inverse of the inbound webhook platform: hooks.outbound in
config.yaml lists HTTP targets + the plugin-hook events they subscribe
to (on_session_end, subagent_stop, post_tool_call, ...). Each firing
POSTs a JSON payload (same top-level shape as shell hooks' stdin wire)
signed GitHub-style with HMAC-SHA256 (X-Hermes-Signature-256).
Rides the existing hook bus — notify-only callbacks registered on the
plugin manager at the same CLI/gateway/main entry points as shell
hooks. Delivery is fire-and-forget via a bounded queue + single daemon
worker thread, so a dead endpoint can never stall a tool call. Bounded
retries (5xx/conn errors once; 4xx never). secret_env preferred over
inline secret. HERMES_SAFE_MODE skips registration. hermes hooks list
shows outbound targets with signed/UNSIGNED status.
Zero new model tools, zero new subsystems.
The _sync_codex_entry_from_auth_store source guard returned early for
source='manual:device_code', which is the recommended quarantine-safe
configuration (hermes auth add openai-codex produces SOURCE_MANUAL_DEVICE_CODE).
The PR's fix for refresh_token adoption was unreachable for these entries.
Widen the guard to accept both 'device_code' and 'manual:device_code'.
Follow-up to #70111. Issue reporter (imgyf) confirmed this caused a
12-of-16 fleet outage on Aug 1.
Co-authored-by: imgyf <imgyf@users.noreply.github.com>
Two defects in the openai-codex credential pool recovery path:
Defect 1 — adoption path silently no-ops when store_access is empty
_sync_codex_entry_from_auth_store() skipped adoption when the auth
store had no access_token (only last_refresh). When another process
rotated the token pair, the stale profile's entry kept the consumed
refresh_token and replayed it, getting refresh_token_reused and going
terminally DEAD.
Fix: also adopt when store_refresh differs from entry_refresh, even
when store_access is empty. Keep the entry's existing access_token
in that case (store_access or entry.access_token).
Defect 2 — false 'auth refreshed' success log
_try_refresh_codex_client_credentials() returned True whenever
resolve_codex_runtime_credentials() returned any non-empty credentials,
including the same stale token when the underlying refresh failed.
The conversation loop then logged 'auth refreshed after 401' right
before the retry failed with the identical token_expired.
Fix: compare the access token before/after the refresh. If unchanged,
return False so the 401-retry path logs the truth.
Fixes#70097
Completes the cache-parity bug class from #76938: the parent's request
body diverges from the fork's not only at the ephemeral system prompt
but also at prefill messages (inserted right after the system message
at API-call time) and, on OpenRouter, at upstream-provider selection
(prompt caches live per upstream; an unpinned fork can be routed to a
different upstream and miss a byte-identical prefix).
Also hardens the tests: pairwise asserts instead of re-implementing the
production prompt join, and routed-path omission guards for the whole
gated kwarg family.
Re-applied from #50508 onto current main: the executor moved from
tools/ to agent/ and the JSON-error site was refactored away into
_parse_tool_arguments, but these 8 f-string debug calls survived the
move verbatim. Lazy %s formatting skips string interpolation when
DEBUG is off — the Tool-result line interpolates the FULL tool output
(can be 100KB+) on every tool call otherwise.
- Short-circuit the candidate waterfall on HTTP 401/403: an auth wall
proves the endpoint family exists, so probing the alternate URL just
doubles the wasted wait (the reported endpoint takes ~10s to return
401 without a key).
- Stream the probe so 4xx never downloads a slow error body; responses
are closed on every exit path.
- Regression tests: single-call assertion on 401/403 (fails on main),
negative-cache reuse, 404 waterfall preserved, no .json() on 4xx.
Fixes#69905
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AZURE_CLIENT_SECRET and AZURE_FEDERATED_TOKEN_FILE presence checks in the
Entra diagnostics path now read through the profile secret scope (Slack
pattern) instead of raw os.environ.
Fixes#69379 (v2026.7.20 Docker multiplex regression: the scoped runner
reload dropped API_SERVER_* set via the container environment, silently
losing the api_server platform) by the canonical mechanism — corrects
the direction of #69524, which patched gateway/config._getenv to fall
through to os.environ on EVERY scoped miss, re-opening the
cross-profile borrow for all credentials.
API_SERVER_ENABLED / API_SERVER_HOST / API_SERVER_PORT /
API_SERVER_CORS_ORIGINS are deployment listener settings (Docker
compose environment: block, systemd Environment=), not profile
secrets: they join _GLOBAL_ENV_EXACT so get_secret reads them from
os.environ regardless of scope. API_SERVER_KEY is deliberately NOT
allowlisted — it IS a credential and stays profile-scoped, which keeps
tests/gateway/test_config.py's secondary-profile isolation semantics
intact (a secondary profile without the key still doesn't bind a
listener).
Ports #69524's regression test in the corrected form: container-env
API_SERVER_* stays visible during the scoped runner reload while the
key resolves through the profile scope; plus unit tests locking the
allowlist membership and the deliberate exclusion of API_SERVER_KEY.
Unquoted values truncate only at a # preceded by whitespace (so
KEY=foo#bar stays intact); quoted values scan escape-aware for the
matching close quote, keep through it, and drop a trailing '# ...'
remainder. Verified empirically against python-dotenv 1.2.2 on a
10-case corpus (full parity). Supersedes the approach in #57718, whose
scanner corrupted foo#bar-style values.
save_env_value writes values containing " or \ as escaped
double-quoted strings, and every other .env reader in the repo
(load_env/_parse_env_value, python-dotenv) reverses those escapes.
load_env_file — the parser behind build_profile_secret_scope, which
wraps every cron job and every multiplexed gateway turn — only stripped
the outer quotes, leaving the escapes literal. A credential containing
a double quote or backslash (JSON service-account blobs, generated
secrets) authenticates fine interactively but 401s under scoped
resolution, with no error pointing at the cause. Parse values with the
canonical _parse_env_value so all readers agree byte-for-byte.
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.
Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.
Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
Review follow-up on the #76098 salvage: the 4096-entry count bound alone
doesn't bound MEMORY — write_file/patch argument strings run 100KB+, so
a long-lived gateway process under sustained heavy write workloads could
pin ~800MB of evicted-session strings. A 32MB byte budget extends the
existing FIFO eviction; common-case args (0.5-2KB) never hit it. New
guard test mutation-checked (fails with the byte leg disabled).
The pre-send normalization pass re-canonicalized every historical tool
call's argument JSON on every API-call iteration — quadratic in session
tool-call count. Route it through a bounded value-keyed memo (the
_MSG_TOKENS_CACHE idiom from agent/model_metadata.py): per-iteration
cost is now proportional to new tool calls, not all of them.
Measured (simulated growing session, repo venv): session-total
canonicalization cost 1056 ms -> 58 ms at 500 tool calls x 2 KB args
(18.3x), 5744 ms -> 79 ms at 500 x 16 KB (72.5x). Byte-parity with the
pre-fix logic is asserted at every iteration (unicode, nested,
malformed, empty, non-string args), and a call-count test proves
json.loads invocations went from K(K+1)/2 to K per session.
- Merge _aux_free_only() + _aux_openrouter_model() into single
_aux_openrouter_settings() that reads config once via
load_config_readonly (avoids double deepcopy).
- Remove 15-line block comment and 5-line inline comment that
restated what the code already says.
- Trim module docstring from 10 lines to 3.
- Update test patches to target load_config_readonly.
Follow-ups from review of #76113:
- Extract cache_ttl_means_disabled() as the single disable-synonym
predicate; agent_init and prompt_caching_disabled_from_config both use
it so the two detection sites can no longer drift (drift would recreate
the #76085 bug class).
- Mirror _run_reference's not-None injection guard in
aggregate_moa_context (stamping None was a harmless no-op copy).
- Replace a vacuous trailing test assertion with the intended
input-non-mutation check; drop a stray blank line.
- Add a predicate-parity regression test (unknown TTL values keep
caching enabled, matching historical agent_init semantics).
Absorb the useful deltas from the parallel #76121 approach: a single
blank_cache_policy_stub factory so _cache_disabled cannot be left off
hand-rolled SimpleNamespaces, and pin the live agent disable onto MoA
advisor fan-out and one-shot aggregate_moa_context decoration so those
paths track conversation state rather than a fresh config re-read.
Keeps the earlier tri-state prepared-aggregator no-agent fix. Adds
factory and synthesis/advisor regressions.
Coordinates with #76121 / #76085.
Co-authored-by: JoaoMarcos44 <87440198+JoaoMarcos44@users.noreply.github.com>
Prepared-aggregator facades built via __new__ lack _agent. Accessing
self._agent raised inside the planner try and bool-coercion of a missing
snapshot forced False, suppressing config fallback for cache_ttl=off.
Pass a tri-state value and add a no-agent/config-off regression.
Blank SimpleNamespace stubs used by MoA decoration and
plan_cache_sections_for_destination never set _cache_disabled, so
anthropic_prompt_cache_policy re-injected cache_control markers after
operators turned caching off. Stamp the disable onto those stubs from
an explicit flag or the live config, and pass the agent flag from the
MoA aggregator path.
Fixes#76085
Hermes installs runtimes for itself — `uv` at `$HERMES_HOME/bin/uv`, Node
at `$HERMES_HOME/node` — and neither directory is on an arbitrary
process's PATH. Every `shutil.which("node"/"npm"/"npx"/"uv")` in Hermes's
own code therefore has two failure modes: the managed runtime is invisible,
so the caller reports "not installed" or degrades to a slower tier on a
machine that has exactly what it needed; and when a system copy also
exists, the one Hermes does not own wins.
Routed the Hermes-owned call sites through managed-aware resolvers:
- `agent/lsp/install.py`, `hermes_cli/dep_ensure.py`, `hermes_cli/main.py`
(`_make_tui_argv`), `hermes_cli/tools_config.py` (`_run_post_setup`) now
use `find_node_executable()`.
- `hermes_cli/tools_config.py::_pip_install` and `hermes_cli/setup.py`'s
vercel install use `ensure_uv()` (installing uv is in scope during setup,
and the Windows installer's `uv venv` does not seed pip, so the fallback
tier is "No module named pip"). `tools/lazy_deps.py` uses `resolve_uv()`
— a lookup, not a bootstrap, because it runs mid-turn for an optional
dependency and downloading a runtime as a side effect exceeds what the
caller asked for.
- `hermes_cli/gateway.py`: extracted `_append_node_dir_for_service()`,
shared by the systemd unit and launchd plist generators, which appends
the managed dirs before the PATH-resolved one. A service definition is
written once and survives reboots, so resolving a system Node that
happens to lead the installing shell's PATH bakes the wrong interpreter
in permanently. Managed dirs are profile-scoped, so each profile's unit
still names its own Node; the existing symlink-parent rule (don't
`.resolve()`) is preserved verbatim.
- `tools/environments/local.py`: the terminal tool's subshell PATH gains
the managed dirs, appended alongside the sane entries rather than
prepended — a tool the user deliberately put on their own PATH still
wins, and the managed one only fills a gap. This is also what makes the
bare `which("uv")` in `tools/env_probe.py` correct: that probe reports
the environment the *model* sees, and the model can only run what is on
that subshell's PATH.
`scripts/install.ps1`: the persisted User PATH update becomes
`Set-ManagedNodeFirstOnUserPath`, a move-to-front rather than an
add-if-missing. Installs made by an older install.ps1 already have the
managed dir in User PATH — at the tail, behind a system Node — and an
add-if-missing check sees it present and leaves that ordering in place
forever, so the users the bug hurt would never be repaired. Unrelated
entries keep their relative order (empty segments included; a trailing
`;` is legal and the installer's other PATH code preserves them),
duplicates collapse, and it writes only when the string actually changes.
Tests:
- `tests/test_managed_runtime_resolution.py` — AST guard that fails any
new bare `which()` for a managed runtime, with a short justified
allow-list and a companion test that fails when an allow-list entry goes
stale. Reading source is banned by AGENTS.md and this is the documented
exception: the property is "no call site anywhere spells it this way",
which no runtime seam can observe.
- `scripts/ci/test_install_ps1_path_migration.ps1` — behavioral, not a
source regex: it lifts the real `Set-ManagedNodeFirstOnUserPath` out of
install.ps1's AST and rewrites only the two registry calls into an
in-memory store, so the shipped split/dedupe/prepend/change-detection
logic executes for real. Not in the default lane (Linux runners have no
PowerShell host); runs under `pwsh`. 13/13 assertions pass.
The sequential executor's KeyboardInterrupt handlers emitted a cancelled
post-tool-call event for the current tool, called agent.interrupt(), then
re-raised — WITHOUT appending a tool result message for the interrupted call
or any remaining calls in the batch. The assistant tool-call turn was left
with no matching tool results, a message-role alternation violation that
malforms the next provider request (relying on downstream repair passes to
patch it, which don't run on every path).
The cooperative-interrupt block (_interrupt_requested) and the concurrent
executor already emit a result for every call_id; this brings the two hard-
interrupt handlers into line via a shared _append_cancelled_tool_results
helper that appends a cancelled result for the current + remaining calls
before re-raising.
Verified live before/after (0 tool results -> 3 for a 3-call batch
interrupted on the first tool) and with a sabotage-checked regression test.
52 interrupt/executor tests pass.
Salvaged from #51604 (@JoaoMarcos44, issue #51603): resolve_anthropic_token()
and run_oauth_setup_token() in agent/anthropic_adapter.py read
ANTHROPIC_TOKEN / CLAUDE_CODE_OAUTH_TOKEN / ANTHROPIC_API_KEY via bare
os.getenv(), bypassing agent.secret_scope — a cross-profile over-read in
multiplex mode. Every other provider routes through
runtime_provider._getenv -> get_secret; the adapter now does the same via
a local _getenv wrapper (identical to os.getenv when multiplexing is off,
scope-authoritative + fail-closed when on).
Dropped from the original PR: the cron scheduler hunks (superseded by
fdab380a1a which installs the per-job profile scope) and the unrelated
hermes_logging Windows hunk (scope creep).
Includes the PR's RED->GREEN scope-isolation test file (6 tests).
The parallel tool-batch planner treated search_files as unconditionally
parallel-safe (_PARALLEL_SAFE_TOOLS) with no path reservation, so a
batch of patch(path=X) + search_files(path=dir(X)) landed in one
concurrent segment and the search could observe pre-mutation file
content — a same-block write->read stale-read race.
Fix the class, not the site: path-scoped reservations now carry a
reader/writer role.
- search_files joins _PATH_SCOPED_TOOLS as a READER, reserving its
search root (default '.', matching the tool's default) instead of
bypassing path checks entirely.
- Overlap only conflicts when a WRITER is on either side: a write into
a searched/read subtree splits segments (ordered behind the write),
while reader<->reader overlap — previously split needlessly — now
stays parallel (concurrent reads commute).
- write_file/patch keep their existing writer barrier semantics.
Prior art surveyed for this design: Codex CLI's RwLock read/write
barrier (readers share, writers exclusive), Claude Code's
isConcurrencySafe partitioning, and gemini-cli's contiguous
parallelizable batching — all converge on reader-shared/writer-
exclusive with contiguous-order preservation, which this planner
already had for read_file/write_file/patch; this closes the
search_files gap and adds the missing reader/reader concession.
Verified by sabotage run (tests fail against the old planner) and an
E2E script exercising the real planner + real file I/O.
- Reuse telemetry['middle_window_tokens'] for the skip's middle estimate
(is-None fallback to a fresh estimate) so log and telemetry agree
- Declare prellm_skip_count in the base telemetry schema (fixed shape)
- Defer _derive_auto_focus_topic into the non-skip branch (user-turn scan
was wasted work on every skip)
- Document the skip in compress()'s Algorithm list and force: arg doc
- Drop dead call_llm patches from 7 tests (unreachable with
_generate_summary mocked)
When the middle section is < 10% of threshold tokens AND at least one prior
real-usage ineffectiveness strike has been recorded, skip the expensive LLM
summarization call and fall through to the deterministic message-dropping
path. Without this guard, a tool-heavy session where the protected tail
already holds most of the tokens can burn 500+ seconds on a summary call
that replaces a few lightweight messages with negligible token savings.
Key design decisions per GottZ review on PR #60451:
1. Separate _prellm_skip_count counter — never increments
_ineffective_compression_count (the strike counter that latches at >=2
to disable compression entirely). One real strike + one skip must NOT
permanently lock out compression until /new.
2. feasibility_skip sentinel flag — exempts skips from the abort branch
(abort_on_summary_failure / _last_summary_auth_failure /
_last_summary_network_failure). A stale failure flag from a prior
cycle must not turn a deliberate skip into a full abort.
3. reason=None for feasibility-skip fallbacks — a stale _last_summary_error
from an earlier real failure must not be embedded into the skip's
deterministic fallback marker.
4. info-level logging for feasibility-skip fallbacks (not warning) — this
is an intentional optimization, not a failure.
Skipped when force=True (manual /compress) so auth/error handling paths
are always exercised on explicit user request.
Adds 6 regression tests (TestPreLlmFeasibilityCheck) covering:
- Strike counter isolation
- Stale auth/network failure flag immunity
- force=True bypass
- No-skip when no prior strikes
- Counter reset on session reset
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: TRON <tron-agent@agentmail.to>
_CFG_DOTTED_RE's nested quantifier (?:[A-Za-z0-9_\-]+\.)+ backtracks
exponentially on long non-matching dotted runs (doubles every ~4
segments). Flatten it and use possessive quantifiers (py3.11+) in
_CFG_DOTTED_RE and _YAML_ASSIGN_RE wherever the successor is disjoint.
Zero behavior change: equivalence fuzz-verified over 120k structured
and random inputs comparing full sub() output including groups. Adds a
ReDoS regression test.
_call_fallback_candidate_sync replans messages/tools for each resolved
destination, but its async mirror still shipped the caller's decorated
sections verbatim — the primary destination's markers (including a
direct-native tool marker) leaked to fallback candidates with different
cache contracts, and the relay saw the display label instead of the
resolved provider/api_mode. Mirror the sync path: resolve the
destination, replan both sections, thread provider/api_mode through
_relay_async_completion, and replan again for the auth-refresh retry
client. Mutation-checked: the new parity test fails on the verbatim
pass-through shape.
Follow-up to #76032 (#20880).
The count walked every message part and tool schema on every request but
is consumed only by tests. Compute it on demand via a property instead.
Follow-up to #76032 (#20880).
Three copies of the same logic landed with #76032:
- MoA's _call_prepared_aggregator and auxiliary_client's
_replan_synchronous_cache_sections both implemented stub → policy →
strip → plan for a resolved destination. Extract
plan_cache_sections_for_destination() into agent_runtime_helpers (which
already owns the policy functions) and route both through it. Also
removes a redundant full-transcript deepcopy+strip per request (the
caller pre-stripped what build_prompt_cache_plan strips again).
- The fallback_chain[N] label regex + chain-entry lookup lived in
_fallback_entry_timeout AND _fallback_destination. Extract
_fallback_chain_entry() and reuse.
MoA's cache-plan failure log is promoted debug → warning: the call-block
site skips MoA, so this block is the aggregator's only decoration path —
a silent failure ships an undecorated request (the 0%-cache MoA bug class).
Behavior-preserving; 195 targeted tests green.
Follow-up to #76032 (#20880).
_apply_static_prefix_marker duplicated _apply_system_cache_markers' split
logic minus its empty-suffix guard: when the stored system prompt equals
the static prefix exactly, the tool-cache plan emitted a two-part split
with a trailing empty text block — HTTP 400 on native Anthropic. Fold the
tool-cache layout into the existing helper via mark_suffix /
fallback_to_whole flags; the empty-suffix case now marks the prompt as
one whole block. Behavior-parity verified against the merged planner for
every non-empty-suffix shape.
Follow-up to #76032 (#20880).
Review pass 2 (reuse reviewer HIGH): the step-3b probe-down fallback for
custom/local endpoints returns the same silent 256K default but only
logged at INFO - invisible by default, and it is the MORE common path
for small local models (the exact users the warning exists for).
Extract _warn_context_length_fallback() (deduped per model+base_url)
and call it from both fallback sites, per the fix-the-whole-bug-class
rule. Regression test drives the custom-endpoint path and fails without
the widening (mutation-checked).
Review follow-up:
- Warn once per (model, base_url) at the step-9 fallback via a module-level
dedup set (established _WARNED_* idiom). The fallback result is
deliberately never cached, so the un-deduped warning fired on every
resolution - e.g. once per gateway message via the session-hygiene path.
- Replace the three inline-mock pool-cleanup tests (which reproduced the
try/except block against a MagicMock and passed even with the production
code reverted) with a parametrized test that drives the real
BatchRunner.run() with a patched Pool; drop the CPython stdlib
signature change-detector test.
- Add a once-per-model warning regression test; clean up dead imports.
All tests verified to fail against pre-PR batch_runner.py/model_metadata.py
and pass with the fix (mutation check).
Salvage of #6629 by aaronlab (kshitijk4poor reworked against current main).
Three concerns from the original PR, reworked to address review feedback:
1. Context-length fallback diagnostic (agent/model_metadata.py):
get_model_context_length() silently returned 256K when all 9 detection
methods failed. Users with small-context models (8K, 32K) would get 256K
silently, causing hard-to-debug API context-length errors. Added a
warning log at the step 9 fallback with model name, base_url, and the
correct config override hint (model.context_length, not context_length).
The token-estimation ceiling-division fix from the original PR already
landed on main (5c2ecdec) with CJK handling — not duplicated here.
2. Fsync for batch trajectory writes (batch_runner.py):
Trajectory entries were written without flush/fsync, but the checkpoint
immediately marked them as completed. A crash between write and disk
sync would leave the checkpoint claiming completion with no trajectory
data on disk. Added flush() + os.fsync() before checkpoint update.
3. Pool cleanup on interruption (batch_runner.py):
Ctrl+C during pool.imap_unordered() relied on context manager cleanup
which can hang. Added explicit pool.terminate() + pool.join() for both
KeyboardInterrupt and Exception paths. The original PR used
pool.join(timeout=10) which is invalid — CPython's Pool.join() takes
no timeout parameter. Fixed to use pool.join() without arguments.
Tests:
- test_warning_emitted_on_fallback: verifies warning fires at step 9
- test_no_warning_when_cached: verifies no false warning when cache hits
- test_trajectory_entry_is_synced_to_disk: verifies os.fsync is called
- test_pool_terminate_called_on_exception: verifies cleanup on RuntimeError
- test_pool_terminate_called_on_keyboard_interrupt: verifies cleanup on Ctrl+C
- test_pool_join_called_without_timeout: verifies no timeout arg to join()
- test_real_pool_join_accepts_no_timeout: integration check on CPython API
Co-authored-by: Aaron Lab <aaronlab@users.noreply.github.com>
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.
The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused #56105 to be reverted in
#56126.
Salvage of #33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.
Co-authored-by: BB-light <BB-light@users.noreply.github.com>
The micro-compaction defrag pass (_defrag_rolling_summary) rewrites the
newest MICRO marker's content and pops _DB_PERSISTED_MARKER from the
LIVE dict in place — the same in-place pop class finalize_turn's fill
site was fixed for in #75170. Without invalidation the bounded
flush-scan cursor identity-skips the rewritten marker row and the
defragged rolling summary never reaches state.db (resume rehydrates a
stale summary).
The compressor holds no agent reference, so the pop site raises
_flush_scan_cursor_invalidated and the finalize_turn micro-compaction
block consumes it, setting agent._db_flush_scan_prefix = None.
The module-scope pop sites (context_compressor.py:175/224) operate on
fresh copies — identity-breaking by construction — and need no flag.
Follow-up to #75170 (fix-the-class sweep of _DB_PERSISTED_MARKER
in-place pops).
The bounded flush-scan in _flush_messages_to_session_db_unlocked skips
the identity-matched prefix of its previous snapshot, on the documented
assumption that no code path pops _DB_PERSISTED_MARKER from a live dict
in place. finalize_turn's pure-tool-call-tail fill is exactly that path:
it pops the marker so the filled content gets re-persisted — but the
cursor then skips the row anyway, so the delivered final response never
reaches state.db and /resume replays content="" (the #43849/#44100
class resurfacing via the perf cursor). Invalidate the cursor at the
pop site so the filled row is re-examined.
Companion to the preflight fix: _estimate_msg_budget_tokens charged
reasoning_details at chars/4 via _REPLAY_BUDGET_KEYS, so the signed/base64
envelope (measured 72% of the reasoning mass on Anthropic-wire sessions)
consumed the tail budget and _find_tail_cut_by_tokens summarized away real
transcript to make room for tokens that are never sent (69 messages on the
measured session; up to ~4.8x budget inflation on thinking-heavy histories).
Per the #51800 counter-argument, actual thinking TEXT stays visible to the
budget: _reasoning_details_text_chars counts thinking/text/summary fields
and skips signature/data/encrypted blobs, and the text is skipped entirely
when reasoning/reasoning_content already carries the identical prose (so it
is charged once, not twice). codex_reasoning_items remains fully charged —
Codex Responses genuinely replays it every request (#55572).
Sabotage-verified: restoring reasoning_details to _REPLAY_BUDGET_KEYS fails
the new envelope and double-charge tests.
The reasoning_details field (OpenRouter/Anthropic thinking blocks +
opaque cryptographic signature blobs) inflates the rough token estimate
by ~4x. Providers do not bill these envelope bytes as prompt tokens.
In a measured Kimi K3 session, reasoning_details held 2,124K chars
vs 281K chars of actual thinking text. The estimator reported ~533K
tokens when real prompt_tokens was ~140K — triggering compression at
~27% of the configured threshold.
Fix: skip reasoning_details in both _estimate_message_chars and
_estimate_message_tokens_without_images, alongside the existing
_anthropic_content_blocks exclusion.
Fixes#73298
Fix#75588
## Root cause
When a short conversation ends in a tool-call/result group and the
protected head alignment reaches the end of the message list,
_find_tail_cut_by_tokens() could return len(messages) + 1. This
happened because the final return used max(cut_idx, head_end + 1)
which could push past the array length when head_end >= len(messages).
The out-of-range value then propagated into _find_context_summaries()
which iterated range(start, end) and indexed messages[idx] without
clamping, raising IndexError and failing the active gateway turn.
## Fix
Two-layer defense:
1. Source fix: _find_tail_cut_by_tokens() now clamps its return to
min(n, ...) so it never exceeds len(messages).
2. Defensive clamp: _find_context_summaries() now bounds start/end
to [0, len(messages)] so even if a future caller passes bad values,
it cannot crash.
## Verification
- 7 new regression tests for the exact boundary conditions
- All 214 existing test_context_compressor.py tests pass
Review follow-up on #75102. The shadow substituted the sidecar whenever
the ``api_content`` key was merely PRESENT, but the wire only substitutes
a non-empty string sidecar on a user/assistant row (see
``turn_context.substitute_api_content``). For any other shape the sidecar
is popped and discarded while the clean ``content`` is sent -- so the
shadow dropped real content from the estimate and UNDERcounted, the
dangerous direction: compaction fires too late and the turn dies on a
hard context-length error instead of merely compressing early.
Gate the substitution on the same predicate, and cover the divergent
shapes (None, empty string, int, list, non-user/assistant role) with a
test that fails against the unconditional version.
Also rename the image test: it never carried a sidecar, so it was not
testing what its name claimed. It is a non-regression pin on the flat
per-image accounting that moved into ``_wire_message_shadow()``, and is
now named for that.
`api_content` is a SUBSTITUTE for `content`, not an addition to it.
`turn_context.substitute_api_content()` pops the sidecar and overwrites
`content` at every API-bound message-build site (the `api_messages` build
in `conversation_loop`, the max-iterations summary in
`chat_completion_helpers`, the chat-completions transport), so exactly one
of the two is ever sent to the provider.
The preflight estimator counted both, because both `_estimate_message_chars`
and `_estimate_message_tokens_without_images` walked every key of the
persisted dict with a single-entry denylist (`_anthropic_content_blocks`).
Any message whose sidecar differs from its clean stored content was counted
twice — exactly 2.00x on a 40KB sidecar.
The sidecar exists to keep the provider prompt-cache prefix byte-stable, so
it is written on precisely the long, cache-pinned messages where the
doubling hurts most. Because `estimate_messages_tokens_rough()` also feeds
the compaction threshold via `context_compressor` and `conversation_loop`,
the inflated estimate makes compression fire on phantom bytes.
Fix: substitute rather than sum, mirroring the wire. The two estimator
helpers had drifted into near-identical copies of the same shadow-building
loop, so this factors the shared logic into `_wire_message_shadow()` and
fixes the class once instead of patching one site and leaving the other.
Image accounting is unchanged: base64 payloads are still replaced with a
placeholder and charged at the flat `_count_image_tokens` rate, and the
`_multimodal` text_summary path is preserved.
Tests: three cases in `TestEstimateMessagesTokensRough` — sidecar equal to
content is counted once, a sidecar that DIFFERS is still counted (a lower
bound, so it fails if the field were dropped rather than substituted, which
would undercount the real request), and a sidecar cannot smuggle raw base64
past the flat image rate.
Verified on Linux (Python 3.11): 53 passed in
tests/agent/test_model_metadata.py, 57 passed with
tests/agent/test_context_breakdown.py, 656 passed / 3 skipped across the
compression/context/token/estimate/prune surface of tests/agent.
Mutation-tested: reverting the substitution fails the new equality test.
`scripts/check-windows-footguns.py` is not applicable — no file I/O,
process management, terminal handling, subprocesses, or signals.
agent_init.py's init-time fallback and chat_completion_helpers.py's
try_activate_fallback() still read key_env via raw os.getenv(), missing the
per-profile secret scope installed by the multiplexed gateway (same bug
fixed for fallback_config.py/auxiliary_client.py in this PR). Both now
delegate to hermes_cli.fallback_config.resolve_entry_api_key(), and the
Ollama Cloud OLLAMA_API_KEY read now goes through
agent.secret_scope.get_secret() too.
agent_init.py's fallback loop had no try/except around key resolution
(unlike the other three call sites), so a fail-closed UnscopedSecretError
under multiplexing would have crashed init instead of skipping to the next
fallback entry — added the same skip-and-continue handling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
resolve_entry_api_key() and the duplicated _fallback_entry_api_key()
read key_env via a raw os.getenv(), bypassing per-profile secret
scoping in the multiplexed gateway. Under multiplexing this can hand
a fallback request another profile's credential. Both now resolve
through agent.secret_scope.get_secret(), which reads the active
profile scope when multiplexing is on and falls back to os.environ
unchanged when it's off, so single-profile behavior is preserved.
Closes#74311
shutil.move() moves into an existing destination directory instead of
replacing it. When the snapshot extract failed after creating some of its
output, the recovery path moved each staged entry onto a path the extract
had already created, burying the user's own skill one level deeper
(skills/alpha/alpha/) and leaving the snapshot's partial content in its
place. rollback() then returned "snapshot extract failed (state restored)".
Clear the failed extract's output before moving the staged copies back:
entries the original tree never had are dropped, and each staged entry's
destination is removed before the move. When an entry still cannot be
restored, keep the staging dir and name the entries in the message rather
than reporting a restore that did not happen.
AIAgent.__init__ calls set_current_session_id(self.session_id), which
mutated both the task-local ContextVar and the process-global os.environ.
Because _build_child_agent wraps construction in delegated_child_context(),
the ContextVar write is harmless (task-local), but the os.environ write
clobbered the parent's HERMES_SESSION_ID for the rest of the process —
leaking the child id into parent tools and subprocesses spawned after
the child was built.
Root cause of HermesPRDelegationSessionContext: parent
20260729_212118_5d797e dispatched child 20260730_160515_736ea1; later
parent terminal inherited HERMES_SESSION_ID=the child.
Fix: set_current_session_id() skips the process-global os.environ write
when called from within a delegated_child_context(). The child's own
tools and subprocesses still resolve their id through the ContextVar
(task-local), while the parent's process-wide env keeps the parent's
session identity. Root agents (CLI, gateway, cron) retain both paths.
Adds 7 regression tests covering single child, concurrent children (8
parallel), parent-tool observation after construction, and root-agent
session rotation backward compatibility. All pass; ruff clean.
_sync_device_code_entry_to_auth_store used key-presence on the profile
store to decide whether to write-through rotated tokens to the global
root. _store_provider_state unconditionally creates that key, so every
refresh after the first self-disabled the write-through — root kept a
revoked refresh token and every other profile died with
refresh_token_reused / invalid_grant.
Fix: use _load_provider_state_with_source to learn where the grant was
resolved from. When the source is the global root, write back only to
root and skip _store_provider_state so the profile never accrues a
shadowing providers.<id> key that blocks future root fallback.
Add regression test verifying write-through fires on refresh 2+, not
just the first call.