Port from openclaw/openclaw#110518 / #110956: some models reuse one call id
for different tool calls in a single batch (native Kimi Responses replays,
Ollama-compatible endpoints, degraded models at long context). Hermes kept
both calls but the pre-API sanitizer then dropped the later call/result pair
per id (#58327), so the second call's output silently vanished from every
replayed payload — the model never saw it and confabulated.
_uniquify_tool_call_ids renames later collisions to a deterministic <id>_d<n>
suffix at ingestion, before validation/dispatch/history build, so both pairs
survive. Composite Responses ids collide on the call half and keep their
response-item half. Deterministic suffixes preserve prompt-cache prefix
stability (no random UUIDs).
test_interrupt_child_during_api_call pinned the OLD worker-thread contract:
a bare time.sleep(5) fake request that only 'interrupts fast' because the
worker thread gets abandoned. Delegated children now run the request INLINE
(#60203) and interrupt responsiveness comes from the cross-thread socket
abort (_abort_request_openai_client) — which a sleep can't feel. Wire the
fake request to an abort event that raises ConnectionError when the child's
abort hook fires, exactly as a real httpx recv unblocks on socket shutdown.
Sabotage-verified: disabling the abort hook fails the test at the full 5s;
with it the interrupt lands in ~10ms.
The whole-prompt scan for "Current working directory:" matched USER project
content, not just Hermes' own host-info line. The prompt embeds AGENTS.md /
CLAUDE.md / .cursorrules in the context tier, which sits AFTER the host-info
block, and line_value() takes the LAST match — so any project file containing
a line starting with that label won the scan.
The comparison then ran runtime state against project prose, a mismatch that
never clears: the stored prompt was rejected on EVERY turn, rebuilding the
system prompt each message and destroying the prefix cache for the whole
session. Strictly worse than the staleness the check exists to catch, and
invisible to CI because no test encoded the invariant.
"last match wins" was only ever safe because Model/Provider/Platform live in
the volatile tier at the very END of the prompt. The cwd line is the one field
read from the STABLE tier near the start, so it needs an anchored read:
locate Hermes' own "User home directory:" line and take the working-directory
line that follows it.
Verified against the real builder on a real AIAgent, not a stub:
- stable cwd, clean project -> prompt REUSED (0 rebuilds)
- stable cwd, AGENTS.md naming the
field -> prompt REUSED (was: rebuild/turn)
- drifted cwd -> rebuilt
- drifted cwd, AGENTS.md naming the
NEW cwd -> rebuilt (prose can't mask drift)
Also re-verified no spurious rebuild across all five cwd-resolution paths
(pinned contextvar, TERMINAL_CWD, launch dir, symlinked workspace,
trailing-slash cwd).
The contributor's fixtures omitted the host-info anchor, so they silently
stopped exercising the cwd path once the read was anchored; reshaped them to
match real prompt structure via a _host_block() helper.
The three refresh tests asserted the replaced shared client gets
close()d — the exact cross-thread close#70773 removes. They now pin
the new contract: close() is NOT called from the refresh path; the
old client is retired (sockets shutdown, FD release deferred to GC).
Widen the #70773 fix beyond the three in-request cleanup sites removed in
the cherry-picked commit: every remaining path that swaps out the shared
OpenAI client could still hard-close its pool from a thread that doesn't
own the in-flight sockets (credential rotation/refresh on the turn thread,
dead-connection cleanup, gateway cache eviction, transport recovery) —
the same FD-recycle corruption vector, just rarer.
Add AIAgent._retire_shared_openai_client(): shutdown(SHUT_RDWR) all pooled
sockets (FD-safe from any thread, unblocks in-flight readers) but never
call client.close() — FD release is deferred to GC, which cannot run until
every borrowing thread has unwound its SSL BIO. Refcounting is the
ownership handshake; with no borrowers the FDs are released immediately.
Wired into:
- _replace_primary_openai_client (rotation/refresh/dead-conn cleanup)
- try_recover_primary_transport (primary_recovery)
- release_clients (gateway cache_evict)
agent.close() keeps the hard close: full teardown is a real session
boundary where no request may be in flight.
Tests: new tests/run_agent/test_70773_shared_client_fd_corruption.py
covers the three watchdog/retry sites plus retire semantics; existing
close-assertions updated to pin retire-not-close.
Review follow-up for the dropped tool-call recovery (#69630): the
re-prompt pair was tagged _dropped_toolcall_nudge, but that marker was
not part of the ephemeral-scaffolding contract. _persist_session /
_flush_messages_to_session_db would therefore write the synthetic
'issue the actual tool call now' user message (and the narration-only
interim assistant turn) as real transcript rows — a resumed session
could replay the internal retry instruction as user-authored context
and prompt unsolicited tool use.
- Add _dropped_toolcall_nudge to _EPHEMERAL_SCAFFOLDING_FLAGS
(run_agent.py) so both SQLite and JSON persistence skip the pair.
- Add it to _SYNTHETIC_USER_FLAGS (conversation_compression.py) so the
compressor never treats the nudge as human intent.
- Flag the interim assistant half of the pair too, and include the
marker in the finalization scaffolding pop so a genuine turn end
strips the pair from the live transcript (mirrors the
_empty_recovery_synthetic pattern).
- Regression tests: flagged messages classify as ephemeral, the
returned transcript contains no scaffolding, and the turn tail stays
on the real assistant answer.
The initial fix guarded the no-tool-calls else branch, but that branch only
SETS final_response — the turn actually finalizes later, in a separate block
after final_msg is built. Runs that reached finalization via that path exited
without the guard ever running (observed live: a scheduled PR reviewer stalled
at tool_turns=1-2 with zero recovery nudges).
Move the recovery to the finalization chokepoint (right after final_msg is
built), so it catches every path that ends a turn. Single guard now:
- increments a consecutive-stall counter and re-prompts (bounded to 3),
- resets on any successful tool round, and
- resets on a genuine (non-mismatch) turn end,
so it guards each stall independently without capping the whole run and
without looping forever.
Verified live: the scheduled reviewer now recovers through the stalls and
submits real reviews — PR 57800 APPROVED, PR 54826 COMMENTED — one PR per run.
Some providers (observed: claude-opus-4.8 / claude-sonnet-4.5 on GitHub
Copilot, ~2026-07) return finish_reason="tool_calls" while the parsed
tool_calls array is empty — the model signalled it wanted to act but the
payload shipped no call. The conversation loop took the no-tool-calls
else branch, treated the turn's narration as the final answer, and exited
with the task unstarted.
On unattended multi-step jobs this is silent failure: a scheduled PR
reviewer, for example, would narrate "Let me verify the PR..." and stop
at tool_turns=0 every run, never submitting a review, while the job still
reported success.
Fix: in the no-tool-calls branch, detect the provider contract violation
(finish_reason == "tool_calls" with zero tool_calls) and re-prompt the
model to emit the call instead of exiting. Bounded to 3 consecutive
stalls; the budget resets after any successful tool round so it guards
each stall independently rather than capping the whole run. The narration
may live in content or only in the reasoning field (empty content) — the
guard keys on the finish_reason/tool_calls mismatch, so both are covered.
A genuine finish_reason="stop" text turn is unaffected.
Verified live: a scheduled reviewer that died at tool_turns=0 every run
now recovers through 20+ stalls per run and submits real reviews.
Tests: tests/run_agent/test_dropped_tool_call_recovery.py covers the
re-prompt, the empty-content case, the clean-stop control, and the
bounded-loop guard.
profile_name was only written on the agent's initial lazy create
(e8b7ce8c1); every parented child row — compression rotation, TUI
/branch, desktop branch first-persist — was created without it. A
non-default profile's lineage therefore turned NULL on its first
compression or branch and aggregated as "default" in unified session
lists, completing the cross-profile session-jump.
Fix the class at the DB layer: _insert_session_row's parent backfill now
COALESCEs profile_name from the parent alongside cwd/git_* (#64709
pattern), so any parented child inherits its lineage's owning profile.
Stamp it explicitly at the three create sites as well — compression
rotation (mirroring _ensure_db_session), TUI session.branch, and the
TUI first-prompt row persist — so rows are self-describing even when the
parent row predates the profile_name column.
Flips the default fan-out cadence from per_iteration (advisors re-run on
every tool iteration, multiplying advisor spend by tool-loop depth) to
user_turn (advisors run once on the first message of each user turn; the
acting aggregator works the rest of the tool loop with that turn's
advice). Until per-mode benchmarks justify a costlier default, MoA
defaults to the cheapest, lowest-impact cadence (#67199).
One default for everyone — no split legacy/new-preset semantics; presets
that want per-step advising set fanout: per_iteration explicitly. All
three modes (user_turn / per_iteration / every_n:N) remain selectable;
every_n:1 still collapses to per_iteration (semantic identity), while
unparseable values now fall to user_turn (the default).
Docs updated with a default-change note; the per-iteration rerun test
pins its mode explicitly.
Co-authored-by: skyer-flyyy <188930297+skyer-flyyy@users.noreply.github.com>
_render_tool_calls only handled dict-shaped entries; a SimpleNamespace-
shaped tool_call (SDK-style stream-stitched responses) rendered as
'[called tool: tool]', silently losing the function name and arguments
from the advisory view. Handle both shapes (including a namespace-shaped
nested function inside a dict entry).
One-hunk hardening salvaged from closed#59712.
Co-authored-by: SquabbyZ <601709253@qq.com>
Reference models may have a smaller context window than the aggregator
(e.g. kimi-k2.7-code @ 262K advising a glm-5.2 @ 1M conversation).
Without context-length protection, a reference whose window is exceeded
gets a hard HTTP 400 from the provider, which _run_reference's
try/except silently converts to a [failed: …] note — the MoA turn
silently degrades to fewer references (#60345).
Redesigned implementation of #60387:
- Estimate AFTER the advisory system prompt is prepended, so the
request that is actually sent is what gets budgeted.
- Reserve output headroom: the preset's reference_max_tokens when set,
else an 8192-token constant, plus a 10% estimator-error fraction.
- Trim on advisory-view boundaries (text-only user/assistant turns; no
tool-result frames to orphan), preserving the system prompt, the
user-first invariant after every pop (never assistant-first), and the
trailing synthetic user turn.
- Cache get_model_context_length per (provider, model) in a per-fan-out
dict shared across the worker threads, so a turn resolves each
window once instead of probing metadata sources
per-reference-per-iteration (failures are cached too).
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Follow-ups for salvaged #56344:
- A reference that completes between the interrupt check and the reap
keeps its REAL output and accounting (the provider call billed) instead
of being zeroed with a placeholder.
- A reference still in flight at interrupt time gets a placeholder in the
results, but its future now carries a done-callback that folds the
eventual real usage/cost into the facade's pending accounting
(late_accounting_sink -> _record_late_reference_accounting), so billed
spend is never silently dropped. Pending totals are folded (not
overwritten) and guarded by a lock since done-callbacks fire on
executor worker threads.
- Interrupted placeholder results are no longer written into the facade's
turn-scoped reference cache: a cache HIT never re-runs references, so
caching a partial snapshot would replay '[skipped: interrupted by
user]' notes for the rest of the turn. The cache is left empty and the
next create() re-runs the fan-out.
agent/tool_executor.py's concurrent tool batch checks agent._interrupt_requested
and aborts the wait early; agent/moa_loop.py's _run_references_parallel had
no equivalent, so a MoA-enabled turn blocked on ThreadPoolExecutor.result()
until every reference model finished or hit its own individual
auxiliary.moa_reference timeout -- there was no way for the user to abort a
live turn mid-fanout.
Thread an optional `agent` parameter through aggregate_moa_context ->
_run_references_parallel (used when MoA references run alongside the main
model) and MoAClient/MoAChatCompletions (used when the MoA preset itself is
the acting model), then poll concurrent.futures.wait() in
_REFERENCE_POLL_INTERVAL_S slices instead of blocking on future.result() per
reference, checking agent._interrupt_requested each cycle.
Deliberately scoped to interrupt/cancel only -- no new or changed timeout
value, so this doesn't overlap open PRs #53784/#53875 (which lower the
per-reference timeout default but don't add interrupt support). `agent` is
optional and defaults to None, so any caller that doesn't pass it keeps
today's uninterruptible blocking behavior unchanged.
Extends the all-references-failed short-circuit (#56975) to the
persistent `provider: moa` facade path: MoAChatCompletions.create()
previously attached 'use the reference responses below' guidance built
entirely from failure sentinels and called the aggregator with it. Now
an all-failed turn attaches either the sanitized unavailability notice
(loud policy) or nothing (silent policy), and the aggregator — which IS
the acting model — simply acts alone. Advisor accounting for the failed
fan-out is still recorded.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
When every MoA reference model returns a failure (HTTP error, timeout,
etc.) or is skipped by the recursion guard, the one-shot aggregator
synthesis call is now skipped entirely. Previously it would try to
synthesise a wall of failure sentinels, which could block for the full
provider timeout (observed ~6 min on SenseNova) before returning a
non-retryable error that left the session hanging.
The early return carries the sanitized unavailability notice (never raw
provider error text, per the failed-reference containment) so the main
agent loop can still act in single-model mode.
Salvaged from #56975, reworked atop the _is_failed_reference helpers.
Follow-ups for salvaged #53784:
- reference_timeout now defaults to None = no per-preset override, so the
reference fan-out inherits auxiliary.moa_reference.timeout (900s default)
via call_llm's own per-task timeout resolution. The PR's 30.0s default
would have cut off long-thinking advisors mid-response, and its 300s max
cap capped legitimate explicit values — both removed. Explicit per-preset
values are still honored as-is.
- _is_failed_reference also treats '[skipped: …]' recursion-guard notes as
internal sentinels, keeping them out of both aggregator prompts.
- Dashboard/desktop TS types updated to number | null; web_server validator
accepts null/empty as 'inherit'.
Adds moa.privacy_filter ('' | display | full, default off — issue #59959):
- display: redact user-visible surfaces only (reference blocks emitted to
the UI + saved MoA trace records, including per-advisor full input/output
and the aggregator-input copy); the aggregator sees raw advisor text so
synthesis quality is unaffected.
- full: additionally redact the advisor text injected into the aggregator
prompt, on both the persistent facade path and the one-shot /moa
synthesis path (the issue's literal ask). Legacy boolean true maps here.
Secret/credential shapes (API-key prefixes, JWTs, private keys, DB
connection strings) are delegated to the central redactor
(agent.redact.redact_sensitive_text, force=True + code_file=True); the MoA
filter adds only email and clearly delimited phone-number patterns. No
bare 10-digit matching: line numbers, timestamps, epoch values, git SHAs,
IPs, versions, and source-code assignments in code-review-shaped advisory
text pass through byte-identical. The reference cache always holds raw
text — redaction happens at each consuming surface, so a mid-session mode
change never leaks or double-redacts.
Reworked from PR #60463: replaced its hand-rolled pattern list (which
matched bare digit runs and re-implemented key shapes) with central-
redactor reuse + safe patterns, and split the single boolean into
display/full modes. Credited for the feature framing.
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Extends the fanout enum with 'every_n:<N>' (N >= 2): advisors run on the
first iteration of each user turn and every Nth tool iteration after it;
off-cadence iterations REUSE the cached guidance from the last on-cadence
run via the same cache mechanism the user_turn fanout uses, so the
aggregator still gets advice on every step. The cadence counter is scoped
per user turn (resets on a new user message) and only advances when the
advisory state actually changes, so streaming retries never consume a
cadence slot. Mapping form {mode: every_n, n: N} normalizes to the
canonical string. Unknown/degenerate values fall back to per_iteration.
Addresses issue #63393 (advisor fan-out multiplies turn latency/cost by
the tool-iteration count). Redesigned from PR #63448: the submitted shape
skipped references entirely on off-cadence iterations (aggregator ran
advice-less); this version keeps the last advice in play, credited for
the idea and cadence framing.
Config-gated, default-off (default fanout remains per_iteration).
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Adds test_moa_gemini_aggregator_sanitize_uses_real_model: drives a full MoA
tool-call turn (virtual-provider mode) with a Gemini aggregator and asserts
the strict-API sanitize pass is invoked with the resolved aggregator model
(gemini-3-pro-preview), never the virtual preset name once a slot is
resolved — the exact path that stripped extra_content/thought_signature and
made Gemini aggregators 400 (#65092).
Writing the test surfaced a gap in the salvaged #66212 fix: in virtual-
provider MoA mode (provider=moa, no moa_config threaded through
run_conversation) the conversation-loop branch never fired because it only
consulted moa_config. Extend it to fall back to the facade's
last_aggregator_slot — the same source the handle_max_iterations fix uses —
so both MoA entry modes resolve the real aggregator model.
Also adds the contributors/emails mapping for the #15676 credit base.
Follow-ups to the salvaged core of #60293:
- Gate the x-initiator header on _normalize_aux_provider() instead of a
literal 'copilot' string compare, so slot configs spelled github /
github-copilot / github-models / copilot-acp / mixed case all get the
user-turn attribution.
- Thread extra_headers through _retry_same_provider_sync/_async so the
credential-refresh and pool-rotation retry rebuilds don't silently drop
the header (the rebuilt kwargs previously started from scratch).
- Add a transport-boundary test asserting the header reaches the SDK
client's create() kwargs (no call_llm mocking), an alias-spelling
matrix test, and a retry-rebuild preservation test.
Follow-up to the salvaged core of #53802: a naive MoAClient(preset) rebuild
restores a working facade but silently drops the reference_callback relay
wired in agent_init, so moa.reference / moa.aggregating display events stop
reaching every frontend for the rest of the session.
Introduce agent.moa_loop.build_moa_facade(agent, preset) as the single
construction point for the MoA facade and use it at:
- initial client construction (agent_init.py)
- turn-start fallback restore (restore_primary_runtime)
- transient transport recovery (try_recover_primary_transport — previously
fell through to _create_openai_client with MoA's empty client_kwargs and
died with 'api_key client option must be set')
- mid-session model switches (switch_model)
The relay reads agent.tool_progress_callback at emit time, so callbacks
attached after construction are picked up automatically.
Adds test_moa_restored_facade_still_emits_reference_events covering event
delivery through a restored facade.
Follow-ups on top of the cherry-picked #62644 mechanism, porting it to
current main and closing the salvage-review requirements:
- proactive_prune_min_reclaim_tokens (default 4096): a prune only COMMITS
when it reclaims a meaningful token batch, measured on the pruned output.
A committed prune rewrites already-sent history and invalidates the
provider prompt-cache prefix; this hysteresis gate keeps those breaks
episodic/amortized (like a compression boundary) instead of firing every
tool iteration. 0 disables the gate. (Design point credited to the
#62389 review cycle's prune_minimum_tokens.)
- Standard no-op caller contract: every skip path returns the INPUT list
object; the loop commits only on 'result is not messages' + non-zero count.
- Loop call is getattr+callable guarded (plugin engines predating the hook,
SimpleNamespace test doubles) and exception-swallowed at debug level.
- Config parse follows the compression.max_attempts hardened semantics:
booleans rejected, fractional floats rejected, integral floats/numeric
strings accepted; negative trigger = disabled.
- cli-config.yaml.example documented (all three keys) and gateway
_CACHE_BUSTING_CONFIG_KEYS extended so hot-reload rebuilds the agent.
- Tests: min-reclaim gate both directions, input-object no-op contract,
no-orphan tool_call_id pairing in BOTH directions (#69830 pin rule),
default-off zero-behavior-change pin, config parse seam, and behavioral
loop-wiring tests (consulted/commit/no-op/absent-method/raising).
Interrupted turns can seed a speculative display token count before the provider receives the request. Restore that display-only seed when interruption wins the race, while preserving completed post-compaction state and treating a successful provider response independently of optional usage metadata.
Constraint: #54776 remains reproducible on current main, while review #4702305384 identifies anti-thrashing rollback as stale and usage receipt as an unreliable response-completion signal.
Rejected: Restore anti-thrashing counters from a preflight snapshot | current main derives their verdict from real provider usage after a completed compaction boundary.
Confidence: high
Scope-risk: narrow
Directive: Keep interrupted preflight rollback display-only, and never infer provider completion from the presence of usage metadata.
Tested: ./.venv/bin/python -m pytest -q tests/run_agent/test_413_compression.py (29 passed); turn-finalizer/conversation-loop tests (31 passed); context-compressor targeted tests (12 passed); infinite-compaction targeted tests (3 passed); ruff; git diff --check.
Not-tested: End-to-end interactive interrupt through CLI or gateway transport.
Gateway half of the #49874 salvage: pass compression_deferred through
both _run_agent_inner result dicts and guard the compression-exhausted
auto-reset block with it — a lock-contended defer keeps the session
intact (the concurrent compressor is actively shrinking it) instead of
wiping it via reset_session.
Regression tests:
- tests/run_agent/test_compression_lock_defer.py — provider-mock 413 and
400-overflow turns whose compression pass lost the lock end as
compression_deferred (failed=False, no compression_exhausted); flag
unset keeps the terminal exhaustion path byte-identical; type-pin
tests vs MagicMock agents and junk flag values; cap=1 e2e proving the
refunded pre-API defer leaves the budget for the provider-proven
413 retry.
- tests/agent/test_preflight_lock_defer.py — a lock-skipped preflight
pass stops the loop WITHOUT arming preflight_compression_blocked;
plain no-op still arms it; MagicMock junk does not defer.
- tests/gateway/test_compression_deferred_soft_result.py — AST pin that
the deferred branch guards the auto-reset chain and performs no
session mutation (mirrors test_35809_auto_reset_clean_context.py).
* fix(desktop): hide persisted agent-only history scaffolding
Filter verification-stop nudges and context-compaction handoffs at the
stored-history mapper boundary. Preserve a real reply when a compaction
handoff shares its stored message.
* test(desktop): build persisted E2E sessions through the real agent
Drive tui_gateway.entry over its stdio JSON-RPC transport against the mock
provider, wait for real completion events, and persist normal session history
through AIAgent and SessionDB. Migrate resume and hidden-history coverage,
including real compression and live verify-on-stop scaffolding, then remove
the unused direct SessionDB import scripts.
* fix(desktop): use the provisioned Python for real-session E2Es
Run the stdio gateway through uv's synced project environment outside the
Nix dev shell, while retaining the fully provisioned Nix Python when the
shell advertises HERMES_PYTHON_SRC_ROOT.
* fix(nix): expose the provisioned Python environment to uv
Mark the Nix-built Python environment active in the dev shell so the shared
E2E session builder can always run through `uv run --active --no-sync`.
* fix(timeline): persist typed display events
* fix(timeline): strip display-only fields from provider payloads, preserve through rewrites, fix /resume display history
Three review findings from PR #69771:
1. Provider payload leak: display_kind and display_metadata were forwarded
to the provider API as unknown message fields. Strict OpenAI-compatible
backends can reject the next request after a model switch or resumed
typed event. Strip both from the per-request api_msg copy in
conversation_loop alongside the existing api_content pop.
2. Rewrite/import data loss: _insert_message_rows preserved display_kind
but silently dropped display_metadata. After replace_messages,
archive_and_compact, or session import, async-delegation completion
events lost their task counts and fell back to generic display text.
Add display_metadata to the INSERT columns and bind tuple.
3. CLI /resume stale recap: startup --resume A set _resume_display_history
from A's lineage. A subsequent in-session /resume B loaded B only into
conversation_history via get_messages_as_conversation, leaving the stale
A display projection. _display_resumed_history preferentially read the
stale attribute, showing A's recap for B. Switch /resume to
get_resume_conversations and update _resume_display_history alongside
conversation_history.
Tests: 890 Python (5 files), 35 desktop TS — all green.
* feat(tui): render typed display events as ◈ markers in the Ink TUI
The TUI was not handling display_kind at all — model switch markers and
async delegation completions rendered as opaque user messages with the
full [System: ...] text, and hidden compaction handoffs were visible.
Wire display_kind through the full TUI chain:
- _history_to_messages (tui_gateway/server.py) forwards display_kind
and display_metadata to the gateway transcript payload.
- GatewayTranscriptMessage (gatewayTypes.ts) gains both fields.
- Msg.kind (types.ts) gains 'event' value.
- toTranscriptMessages (domain/messages.ts) maps:
- hidden → skip entirely
- model_switch → event "model changed"
- async_delegation_complete → event "N background agents finished"
(or "background agent work finished" without metadata)
- messageGroup (blockLayout.ts) routes event to its own group, with
SELF_SPACED + PAINTS_TRAILING_GAP so it owns its margins.
- messageLine.tsx renders event-kind as a dim ◈ marker with no gutter,
matching the CLI's ◈ event rendering.
- 4 new TUI tests for hidden/model_switch/async_delegation mapping.
TUI typecheck: clean. TUI lint: 0 errors (2 pre-existing warnings).
TUI tests: 9 passed (1 pre-existing failure on main, unrelated).
Follow-up to the #58738 salvage: the pre-exhausted check now enumerates
pool.entries() to find the failing key, so the MagicMock pool double must
expose entries as a callable, not a bare list.
Review follow-up: the auth path called pool.try_refresh_current() before
the hinted rotation, so a stale current() pointer could force-refresh a
different, healthy entry — consuming its single-use refresh token, or
(for non-OAuth entries, where a forced refresh marks the entry exhausted
outright) killing it entirely before api_key_hint was ever consulted.
Use try_refresh_matching(api_key_hint=...) to resolve and refresh the
entry that supplied the failing key under the pool lock, falling back to
the previous behavior when no key is known.
Adds a regression test with current() deliberately pointed at the
healthy entry: on the old code the healthy entry is exhausted by the
forced refresh and the pool ends up fully offline; with the fix the
failing entry is exhausted and recovery rotates to the healthy one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
recover_with_credential_pool identified "which credential failed" via
pool.current(), a shared mutable pointer that is advanced by every
select() (round-robin rotation, concurrent turns, and other processes
reloading the pool reset it to None). By the time recovery ran, it
routinely pointed at a different, healthy entry — mark_exhausted_and_rotate
then stamped the failing request's error message and reset time onto that
innocent entry. With round_robin and one hard-capped key this
deterministically exhausted the healthy key too and took the entire pool
offline ("no available entries") from a single rate-limited credential.
mark_exhausted_and_rotate already supports api_key_hint for exactly this
(the auxiliary-client path passes it); the main conversation-loop path
never did. Pass agent.api_key — kept in sync with the entry in use by
_swap_credential — as the hint on all four rotation call sites, and make
the "already exhausted → rotate immediately" pre-check look up the failing
entry by key with the same fallback to current().
Adds regression tests that fail on the old attribution logic: a fresh
pool (current() is None) failing on key B must mark entry B, never
entry A.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up for the salvaged #35191: the mid-turn pre-API pressure emit in
conversation_loop.py and the idle-resume emit in turn_context.py were not
routed through automatic_compaction_status_message, so an engine with
emit_automatic_compaction_status=False still leaked those lines. Both now
resolve through the hook (phases "pre_api" and "idle") while keeping the
#69550 template constants as the default wording. Suppression also skips the
#69546 structured 'compacted' terminal edge for compress-phase events that
opened no visible phase; failure warnings (_emit_warning) remain never
suppressible, pinned by test.
Follow-up for salvaged #58629: thread the previous flush baseline through
the idle-compaction caller in turn_context.py (the one caller the original
PR predates), and add regression coverage that every early-return path in
compress_context (breaker skip, no-progress, plus the completed-rotation
boundary) resets the per-attempt in-place outcome so a stale
_last_compaction_in_place from an earlier successful in-place compaction
can never baseline unflushed turns as persisted.
Context engines that override ``should_compress_preflight()`` (e.g. the
hermes-lcm plugin's incremental leaf-chunk compaction) never had their
hook fired by ``run_conversation`` because the preflight block exited
early once the hardcoded ``>= threshold_tokens`` check failed. As a
result, ``LCM_DEFERRED_MAINTENANCE_ENABLED=1`` and friends were inert
and accumulated raw_backlog debt indefinitely.
Add an ``elif`` branch that delegates to the engine's preflight hook
when the legacy threshold check does not fire. The default
``ContextEngine.should_compress_preflight()`` returns ``False`` so the
built-in ``ContextCompressor`` is unaffected; engines opting in get a
chance to ingest messages and request a single ``compress()`` pass for
deferred maintenance. Exceptions are swallowed at debug level so a
buggy engine cannot break an otherwise-healthy turn.
Closes#20316
Invalid-tool exhaustion and truncated-tool early returns skipped finalize_turn, leaving role=tool transcripts that become tool→user on the next turn for strict providers. Call close_interrupted_tool_sequence before persist on those paths (same as interrupt aborts).
Follow-up to the #43755 salvage:
- Update the strict _Pool doubles in tests/run_agent/test_run_agent.py to
accept api_key_hint and assert it carries the agent's failed key.
- Add a real-CredentialPool regression (no mocks) proving the hint routes
exhaustion to the entry whose key actually failed, not pool.current(),
plus the no-hint baseline (#43747 wrong-entry marking).
A follow-up sent while the model is still generating previously ended the
turn: Hermes kept only the visible partial text (reasoning was display-only),
cleared the loop, and replayed the message as a fresh next turn. If the
correction referred to something that only appeared in the thinking stream,
the model no longer had that context.
Add `AIAgent.redirect(text)`: a corrective interrupt distinct from a hard
stop. It cancels only the in-flight model request (not tool workers or child
agents), stashes the correction under a lock shared with `interrupt()` so a
concurrent `/stop` always wins, and lets the loop rebuild the same logical
iteration. `_apply_active_turn_redirect()` checkpoints the reasoning that was
actually shown to the user plus any visible partial text as an ordinary
assistant message, then appends the correction as a real user turn — never
replaying incomplete signed/encrypted provider reasoning, and keeping strict
role alternation and prompt-cache stability intact. During tool execution it
degrades to `steer()` so a running tool finishes at a safe boundary.
`_fire_reasoning_delta` now only records reasoning that a display callback
actually consumed, so `show_reasoning: false` never leaks hidden provider
thinking into the persisted transcript.
Follow-up to the salvaged contributor commit, closing the three gaps
flagged in the sweeper review:
1. Init ordering: assign compression.model_thresholds to a selected
plugin context engine BEFORE the initial update_model() call in
agent_init.py, so the initial model's override applies from init
(previously it only took effect after the first /model switch).
Base-class ContextEngine.update_model() now snapshots the
pre-override percent once so repeated switches fall back to the
engine's configured threshold, not a previous model's override.
2. DEFAULT_CONFIG: add compression.model_thresholds (empty map) to
hermes_cli/config.py — additive key, no _config_version bump.
3. Docs: document the key in
website/docs/developer-guide/context-compression-and-caching.md
(yaml example, parameter table, dedicated section) and update the
plugin-boundary note in context-engine-plugin.md to state the
explicit context-engine contract for model_thresholds.
Adds tests/run_agent/test_per_model_threshold_init_ordering.py:
plugin-engine AIAgent init regression (override applies at init,
empty map unchanged), DEFAULT_CONFIG key presence, floor interaction
on the model-switch path (override below the small-context floor is
raised to the floor; above the floor wins), and base-class config
snapshot across repeated switches. Also maps @bennybuoy in
contributors/emails/.
Addresses teknium1 review feedback on PR #60781:
1. Gateway cache invalidation: added ('compression', 'model_thresholds')
to _CACHE_BUSTING_CONFIG_KEYS so a live config edit to the map
invalidates the cached compressor (previously kept stale thresholds).
2. Integrated resolver with small-context floor: per-model overrides are
resolved FIRST, then the existing 75% floor for <512K models is applied
on top. The floor is no longer replaced — it stacks. An override below
75% on a small-context model still gets floored to 75% (raise-only);
an override above 75% wins.
3. Clean rebase on upstream main — no unrelated deletions or anti-thrashing
changes. Only the per-model threshold feature is added.
Changes:
- resolve_model_threshold() module-level helper (longest substring match)
- ContextCompressor.__init__ accepts model_thresholds dict
- _base_threshold_percent stores the per-model resolved value
- _config_threshold_percent stores the raw config value (fallback base)
- update_model() re-resolves on /model switch, falls back to config value
- ContextEngine base class update_model() applies overrides for plugin engines
- agent_init.py reads compression.model_thresholds from config, passes to ctor
- gateway/run.py cache busting key added
- cli-config.yaml.example documents the feature
- 17 tests covering resolve helper, compressor init (large/small context,
override above/below floor), update_model (re-resolve, fallback), base class
Co-authored-by: Copilot <copilot@github.com>
Review follow-up on #67431 (hermes-sweeper):
- The viability check compared the floored percentage against the raw
context window, but the built-in trigger recomputation also applies the
output-token reservation, the 64K floor, and the degenerate-window
guard (_compute_threshold_tokens). Mirror that math exactly, so e.g. a
200K window with max_tokens=120K recomputes to max(0.75*80K, 64K)=64K
and the suggestion is correctly KEPT for an 80K aux model instead of
being suppressed by the raw-window percentage.
- Gate the built-in policy behind isinstance(ContextCompressor): external
context engines own compaction policy (#44439), so plugin engines keep
the plain suggestion untouched.
- The non-viable explanation now names the recomputed trigger instead of
hardcoding the 75%/512K wording, so it stays accurate when the
reservation (not the percentage floor) is what makes the value
unreachable.
Tests: reservation-viability regression and plugin-engine passthrough,
per the review; the floored-branch assertion updated to the recomputed
number.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The auxiliary-compression feasibility warning computes its
compression.threshold suggestion as aux_context / main_context,
independently of ContextCompressor._effective_threshold_percent()'s
raise-only small-context floor. For main windows under 512K the floor
raises any configured value below 75% back up, so a suggestion like
'threshold: 0.40' is silently ignored and the same warning returns every
session.
Derive the suggestion's viability through the compressor's own floor
logic: offer the 'lower the threshold' option only when the floored value
still fits the auxiliary model's context; otherwise recommend only a
larger compression model and explain the floor, so the guidance is always
actionable.
Tests: the updated auto-correct test pins the floored branch (no
threshold suggestion, floor explained); two new tests pin the surviving
suggestion at/above the floor on a small window and below 75% on a
512K+ window where no floor applies. The updated test fails against the
previous code.
Fixes#67422
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Refresh the agent activity tracker while context compression is blocked in the auxiliary summarizer so gateway watchdogs do not report inactivity during long compactions.
Add regression coverage for successful heartbeats, exception cleanup, touch failures, and strict-signature compressor fallback.
(cherry picked from commit c09e58b7709cc60c5b454701f0ecf840e759222f)
Follow-up to the salvaged #64010 (Kenmege) and #63870 (dombejar) commits,
making one resolved compression.max_attempts cap govern ALL per-turn
compression attempt sites:
- conversation_loop: resolve max_compression_attempts ONCE at turn start
(it was previously re-resolved inside the API-call loop) and route the
pre-API pressure gate through it — that gate still hardcoded
'compression_attempts < 3' and logged 'attempt=%s/3'.
- conversation_loop: the salvaged post-tool compaction gate now uses the
resolved cap instead of a hardcoded 3.
- turn_context: the preflight compaction loop was 'for _pass in range(3)';
it now sizes itself from the same resolved cap.
- agent_init: harden the max_attempts parser — reject booleans (bool
subclasses int; 'true' would coerce to 1), reject fractional floats
instead of truncating them, keep accepting integral floats and numeric
strings; anything else falls back to 3 (floor 1, ceiling 10 unchanged).
- tests: replace #63870's inspect.getsource source-shape test with
behavioral loop tests (post-tool compaction fires <= cap times per turn,
shares its budget with the pre-API gate, resets between turns); add an
e2e test proving a 4th preflight pass runs at config cap=6 while the
unset default still stops at 3; extend the #64010 config tests with the
bool/float parser semantics.
Salvages #64010 by @Kenmege and #63870 by @dombejar.
* feat(nous): send top-level session_id for provider sticky routing
The Nous Portal profile only embedded the session id inside portal tags,
so Claude traffic through the portal had no sticky-routing key. Multi-turn
sessions could reroute between upstream endpoints (Anthropic/Vertex/
Bedrock), cold-writing a fresh prompt cache on every reroute since each
provider's cache is instance-local.
Mirror the OpenRouter profile: emit extra_body.session_id whenever the
agent has one, pinning every turn of a session to the same endpoint so
explicit cache_control breakpoints stay warm.
* test: expect top-level session_id in Nous max-iterations summary body
Sibling site of the profile change — the max-iterations summary path
builds its request through the same NousProfile.build_extra_body(), so
its exact-shape assertion now includes the sticky-routing session_id
when the agent has a session.
Salvage follow-up for PR #68899:
- Restore .rstrip('/') on base_url in _swap_credential (both anthropic
and OpenAI paths) to match every other assignment site. The route
identity comparison still uses normalize_route_base_url which handles
trailing slash correctly.
- Extract should_clear_context_pin() into hermes_cli/route_identity.py,
consolidating 7 copy-pasted call sites across cli.py, gateway/run.py,
gateway/slash_commands.py, and hermes_cli/model_switch.py into a
single fail-closed helper.
C1 (anthropic path TLS re-application): pre-existing gap — the Anthropic
adapter (build_anthropic_client) has no TLS customization support at
all, so this is out of scope for this salvage.
* fix(desktop): preserve interim assistant text wiped at message.complete
When the agent emits interim text (commentary alongside tool calls, or the
attempted final answer before a verify-on-stop nudge), all UI surfaces
streamed it live but then wiped it at message.complete — keeping only the
final response. The user saw text appear during inference, then disappear.
This is the complete fix across all three layers: agent core, gateway
transport, and all UI surfaces (desktop + Ink TUI).
The verify-on-stop and pre_verify paths flagged the assistant's attempted
final answer as _verification_stop_synthetic, suppressing it from both
state.db and the UI. The user only saw the terse post-verification reply.
Now the assistant response is real content: it's persisted to state.db and
emitted as an interim message via _emit_interim_assistant_message(force_display=True)
before the verification loop runs. Only the synthetic nudge messages keep
the synthetic flags. The turn finalizer drops nudges from live history and
compares content (not just role) to avoid duplicating a published candidate.
Message sequence repair collapses verification candidates in the
consecutive-assistant merge.
Wire agent.interim_assistant_callback both at construction (_agent_cbs())
and per-turn (defense-in-depth), emitting a new message.interim event with
{text, already_streamed}. Gated on display.interim_assistant_messages
(default true). Cleared in the finally block so a stale closure can't
fire on a later turn.
Add message.interim to the GatewayEventName union (apps/shared) and a
typed payload to the TUI's GatewayEvent discriminated union.
The TUI already had the segment-anchoring machinery (flushStreamingSegment +
finalTail) but had no handler for message.interim. Added recordInterimMessage
+ interimBoundaryIndex to seal segments mid-turn, and updated
recordMessageComplete to only dedupe segments after the interim boundary.
Replaced the fragile sealed-set approach with a proper interimBoundaryPending
state flag on ClientSessionState. finalizeInterimAssistantMessage finalizes
the streaming bubble in place (or creates a standalone one), rotates the
stream ID so next deltas create a new bubble, and sets the flag. When the
final text equals an already-sealed interim, they stay as distinct messages.
Extracted mergeFinalAssistantText() as a pure function in chat-messages.ts,
used by both completeAssistantMessage and finalizeInterimAssistantMessage.
Split the bidirectional dedup predicate: reasoning is a restatement only when
the final FULLY covers it. A short final ("Done.") no longer swallows a
longer reasoning block that merely starts with it.
Honor display.interim_assistant_messages (default true) across all layers:
the tui_gateway gates the callback, the desktop wires it to a nanostores
atom via use-hermes-config. Updated hermes_cli/config.py and
cli-config.yaml.example comments to document the Desktop behavior.
_split_segment_tokens now accepts posix=False and _find_ad_hoc_match tries
both posix modes so ad-hoc verification scripts with Windows backslash
paths are matched correctly. (response_previewed forwarding from #53553
is not included — our emit-interim + persist approach makes it unnecessary
since the attempted answer is now surfaced before the verification loop.)
- tsc: clean (desktop + TUI + shared)
- vitest desktop: 73/73 pass (7 interim-sealing + 5 mergeFinalAssistantText + 4 config atom)
- vitest TUI: 83/83 pass (4 new message.interim tests)
- python: 390 tests pass (340 tui_gateway + 33 verification/finalizer + 6 config gating + 3 evidence + 8 continuation budget)
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
* fix: prefix-match interim streamed content to avoid benign duplicate bubbles
_interim_content_was_streamed used exact equality (streamed == visible_content),
so a final response that was the streamed text plus a trailing delta — or a
partial stream before the verify nudge fired — failed the match and left
_response_was_previewed false. The turn then showed two bubbles (interim +
identical final) instead of settling the interim in place.
Relax to a prefix check (visible_content.startswith(streamed)) in both the
core match and the desktop's settle-in-place gate. The TUI already used
prefix matching via finalTail. The reverse direction (streamed longer than
final) is intentionally not matched — that could suppress a needed resend
in the gateway path where already_streamed=True calls on_segment_break().
* test(desktop): add partial-stream-then-nudge dedup edge case
Third edge case for the interim-sealing dedup: model streams part of its
answer via message.delta, verify nudge fires, interim seals the streamed
prefix, then the final response is the same text plus a trailing delta.
Asserts one bubble (not two) containing the full final text.
Acceptance protocol #2 — covers all three dedup edges:
1. interim == final (existing)
2. interim = strict prefix of final (existing)
3. partial-stream-then-nudge (this commit)
---------
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
When a streaming response ends cleanly (HTTP 200) with no finish_reason
after delivering text but no tool calls, the chunk collector silently
stamps finish_reason='stop' and the conversation loop presents truncated
text as a complete response.
Three stream-drop paths now exist after chunk collection:
1. Zero-chunk → EmptyStreamError, retried (existing)
2. Tool-call in progress, no finish_reason → partial-stream-stub (existing)
3. Text-only, no finish_reason → partial-stream-stub (NEW — this fix)
Path 3 routes through the same PARTIAL_STREAM_STUB_ID + FINISH_REASON_LENGTH
machinery as path 2. The conversation loop shows 'Stream interrupted —
requesting continuation' and injects a continue prompt, giving the model
a chance to resume where the stream dropped.
Observed with DeepSeek provider where CloudFront drops SSE streams
mid-response after delivering partial text.
The salvaged retention check compared the built-in memory snapshot
before vs after the disk reload. That holds for a long-lived CLI agent,
but on fresh-agent surfaces (gateway per-turn agents, TUI) the cached
prompt is restored from the session DB and can predate mid-session
memory writes that the fresh MemoryStore already absorbed at init: the
snapshot is then identical on both sides of the reload while the prompt
itself is stale, so compression would retain (and re-persist via
update_system_prompt) a prompt missing the new memory for the life of
the session.
Replace the equality check with a containment check
(_cached_prompt_reflects_builtin_memory): retain the cached prompt only
when the freshly-reloaded rendered blocks appear verbatim inside it,
and rebuild when a leftover block header remains for a target whose
entries have since been emptied or disabled. Block headers are shared
via MEMORY_BLOCK_HEADERS in tools/memory_tool.py so the check stays in
lockstep with MemoryStore._render_block.
Adds regression guards for the gateway stale-restore path and the
emptied-memory leftover-block path; verified with a real-MemoryStore
E2E matrix (9 scenarios) against a temp HERMES_HOME.
_is_real_user_message no longer accepts a user-role compaction summary as
the human anchor, so _compress_context restores the original user turn
after the summary; update the lifecycle-status test's expectation.
Direct-Anthropic requests used a single shared _anthropic_client, and the
stale/interrupt watchdog closed + rebuilt it from the poll (stranger) thread
at four sites (non-streaming stale/interrupt, streaming stale/interrupt).
Closing a client whose TLS socket a worker thread was still reading released
the FD from a stranger thread; the kernel recycled it under a live SSL BIO,
which then wrote a 24-byte TLS record into an unrelated SQLite header
(cron/executions.db), bricking every cron on the profile. Same shape as the
OpenAI-only #29507 fix, but the Anthropic path never got the owner-thread
contract.
Extend the #29507 ownership contract to Anthropic: build a per-request client
(_create_request_anthropic_client), register it with the request-client
holder tagged by kind, and route _close_request_client_once by kind — a
stranger thread only shuts the request client's sockets down
(_abort_request_anthropic_client), while the owning worker performs the SDK
close (_close_request_anthropic_client). The shared _anthropic_client is now
never closed from inside a request (streaming or non-streaming), including the
worker retry-cleanup sites, since each attempt builds a fresh request client.
The #28161 no-hang guarantee is preserved: the poll-thread socket abort
unblocks the worker immediately.
Salvages the approach from #51688 (@raymondyan-zhijie), reimplemented onto
current main (non-streaming dispatch was refactored into
_dispatch_nonstreaming_api_request; streaming grew _cancel_current_stream_attempt
and worker retry-cleanup sites). Tests updated to the request-local mechanism
(incl. replacing a banned source-reading test with a behavior test) plus new
regression coverage proving the watchdog aborts the request client and never
touches the shared client.
Co-authored-by: raymondyan-zhijie <32435458+raymondyan-zhijie@users.noreply.github.com>
Second call site (non-streaming / gateway path) now flattens list-type
content with flatten_message_text before the inline <think> regex and the
surrogate sanitizer, matching the interim-text fix from the prior commit.
Adds regression tests (tests/run_agent/test_66267_multimodal_interim.py)
covering:
- build_assistant_message with list content does not raise TypeError
- inline <think> inside list content is extracted + stripped correctly
- _interim_assistant_visible_text is safe for tool messages (list content)
- duplicate_previous_interim dedup guards against tool messages
Verified the tests fail without the fix (TypeError: expected string... got
'list') and pass with it.
_bedrock_reasoning_stale_floor only matched dashed floor-table keys (opus
'claude-opus-4-6'), so sonnet reasoning models keyed with a dotted version in the
shared table ('claude-sonnet-4.5'/'4.6') got no reasoning floor from the dashed
Bedrock inference-profile id — premature stale-abort if the base timeout is set
below 180s. Generate both version-separator forms (digit-dash-digit <-> digit-dot-
digit, via lookbehind/lookahead so only version numbers flip) and try all
candidates; matches the table however each model is keyed, no edit to the shared
table. Verified: opus->240 (unchanged), sonnet-4.5/4.6->180, haiku->None. New
TestBedrockReasoningStaleFloor (8 cases).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-applied against v0.18.2. Upstream now covers the OpenAI-path stall watchdog
(cross-turn give-up breaker #58962) and the tool-batch deadline, so those are
dropped. Two gaps remain:
- Bedrock streaming had NO liveness watchdog and was excluded from the #58962
breaker (it returns before _check_stale_giveup). Add an on_event hook to
stream_converse_with_callbacks (fires per yielded event = true wire-level
liveness), drive a stale timer from it, and wire Bedrock INTO the existing
breaker: entry _check_stale_giveup, _bump_stale_streak on stall, raise to end
the call (invalidate_runtime_client can't abort the in-flight botocore stream,
so the streak escalates across turns like the OpenAI path), and reset the
streak on success.
- local providers still got float('inf') stale timeout (watchdog disabled) — give
a finite ceiling (HERMES_LOCAL_STREAM_STALE_TIMEOUT, default 900s).
- tests: on_event per-event + swallow; Bedrock stall bumps streak + aborts;
pre-elevated streak aborts at entry; success resets streak.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
strip_think_blocks() ran re.sub() directly on content that could be a
list of blocks (Anthropic via OpenRouter returns assistant content as
[{type:text,...},{type:thinking,...}]). A list reaching re.sub raised
'TypeError: expected string or bytes-like object, got list', which the
outer conversation loop swallowed and retried forever — the observed
infinite 'preparing terminal...' loop that re-emitted the same
assistant text every iteration.
The live-turn path normalized list content to a string, but
_interim_assistant_visible_text reads a *stored* history message whose
content was persisted as a list and passes it straight into the shared
strip_think_blocks helper. Fix at the shared choke point: coerce
list/dict content to visible text (dropping reasoning blocks, which is
the function's job) before any regex runs, so every caller is safe.
Fold #62349's broader provider-boundary handling into the header fix: bound top-level and xAI override keys again at preflight after middleware, preserve unrelated headers, and cover boundaries and collisions.
Co-authored-by: Nick Taylor <nicktaylor@TheWorldofNick-Lappy.local>
Widen the #65991 single-writer fence to run_codex_stream: each codex
attempt claims the delta sink before consuming events, and the consume
loop's interrupt_check now also stops the instant a newer attempt
supersedes this one. Parity with the chat_completions / anthropic /
bedrock paths from the salvaged fix.
Two regression tests: superseded codex stream is fenced mid-stream;
sole-writer codex stream delivers unchanged.
Assert that a superseded stream (older writer token, other thread) is fenced
from the delta sink, the active writer is never fenced, a non-claiming thread
is never treated as a writer, and the real consume loop stops the instant it is
superseded — so two streams can never interleave into one turn (#65991).
Port from anomalyco/opencode#36130: the Responses spec carries streaming
error details at the top level of the error frame, but the official OpenAI
SDK and several OpenAI-compatible proxies wrap them in an HTTP-style nested
envelope ({"type": "error", "error": {code, message, param}}).
_raise_stream_error only read top-level fields, so nested-envelope frames
collapsed to the generic 'stream emitted error event' placeholder with
code=None — the error classifier never saw the provider's real failure
reason, misrouting rate-limit / context-overflow / entitlement errors into
the generic retry path.
Top-level fields keep precedence; the envelope is a fallback. Null-tolerant
for spec-compliant frames with explicit nulls.
Follow-ups on top of @xxxigm's salvaged bridge (#33294):
- Remove the now-dead narrow item/started-only mapper from #38835
(_codex_note_to_tool_progress) — the full bridge supersedes it and
keeps the same tool-name contract; its tests are repointed at the
bridge helpers.
- Preserve main's request_routing/approval-bypass wiring on the
CodexAppServerSession constructor (landed after the PR was filed).
- Gate agentMessage interim delivery on display.show_commentary so the
app-server runtime honors the same toggle as the codex_responses
commentary channel (tool progress is unaffected).
- Add json import (bridge helpers use json.dumps) and modernize the
wiring test's stub agent for main's usage-accounting attributes.
Commentary delivery is on by default; users who find the extra mid-turn
narration noisy can set display.show_commentary: false to restore the
previous behavior (commentary routed to the reasoning channel, visible
only with show_reasoning).
- hermes_cli/config.py: display.show_commentary default true
- agent/agent_init.py: wire config -> agent.show_commentary
- run_agent.py: gate structured commentary extraction on the flag
- agent/codex_runtime.py: gate live-stream commentary callback (falls
back to legacy reasoning-channel routing when off)
- docs + 2 tests (interim path off, live stream fallback)
Also adds AUTHOR_MAP entries for davidrobertson and 100yenadmin.
Keep each xAI OAuth auth-add login as an independent manual device-code pool entry and recognize xAI personal-team spending-limit 403 responses as billing exhaustion. Preserve the structured top-level error message so the failed credential is quarantined and the next healthy account is selected without attempting a pointless token refresh.
Route direct xAI HTTP consumers through the credential pool as well. Proactive and 401-reactive refreshes update the exact issuing manual entry, preserve validated xAI base URL overrides, and serialize single-use refresh-token rotation across concurrent pool instances.
Add current-main regression coverage for both the registered provider
profile and core GitHub Responses path while leaving live catalog loading
to the complementary catalog-resolution work in #51953.
_extract_parallel_scope_path used Path.cwd() (process cwd) instead of the
tool's actual execution cwd, and os.path.abspath() instead of os.path.realpath(),
so symlink aliases and relative/absolute path pairs that resolve to the same
physical file were treated as distinct targets and placed in the same parallel
segment. On case-insensitive platforms (Windows) os.path.normcase() was also
absent, allowing Foo.txt and foo.txt to race.
Changes:
- agent/tool_dispatch_helpers.py: introduce _canonical_path(raw_path,
execution_cwd) applying expanduser->abspath->realpath->normcase; thread
execution_cwd through _extract_parallel_scope_path and
_plan_tool_batch_segments
- agent/tool_executor.py: pass get_active_env(effective_task_id).cwd as
execution_cwd to _plan_tool_batch_segments; add pathlib.Path import
- run_agent.py: pass active env cwd to _plan_tool_batch_segments at the
second call site inside _execute_tool_calls
- tests/run_agent/test_tool_batch_segmentation.py: add 5 regression tests
covering relative/absolute same target, symlink alias, execution_cwd vs
process cwd, symlink parent + nonexistent write target, and Windows
case-insensitive alias (skipped on non-Windows)
Fixes a file-corruption / lost-update race introduced by the mixed
tool-batch segmentation feature (perf commit #64460).
* feat(analytics): record auxiliary model usage per task in session accounting
Auxiliary LLM calls (vision, compression, title_generation, web_extract,
session_search, ...) discarded their token usage, leaving dashboard
analytics blind to aux model spend (issue #23270).
- hermes_state.py: session_model_usage gains a task PK dimension
(''=main loop) via v22 table-rebuild migration (SQLite can't alter a
PK); record_auxiliary_usage() writes per-(model,provider,task) deltas
WITHOUT touching sessions counters (gateway overwrites those with
absolute main-loop totals — folding aux in would double-count or be
clobbered). Aux rows never inherit the session's main-loop route.
- agent/aux_accounting.py: ContextVar ambient accounting context
(mirrors the portal_tags conversation context); record_aux_usage()
normalizes usage via usage_pricing.normalize_usage, estimates cost,
and is strictly best-effort. moa_reference/moa_aggregator excluded —
conversation_loop already folds MoA usage+cost into the main delta.
- agent/auxiliary_client.py: _validate_llm_response is the recording
chokepoint — every successful non-streaming aux response passes
through it exactly once, sync and async, including fallback paths
(model read from the response itself stays accurate across
fallbacks).
- run_agent.py: run_conversation publishes/resets the accounting
context; agent/title_generator.py republishes on its bare thread.
- hermes_cli/web_server.py: /api/analytics/usage folds aux rows into
by_model (aux-only models finally appear) and adds a by_task
summary; /api/analytics/models surfaces aux rows on the Models page.
Design per review of PR #62850 by @eeksock (thread-local + separate
auxiliary_usage table): rebuilt on ContextVar (async-safe — thread-local
cross-attributes concurrent coroutines on one event loop) and the
existing session_model_usage table instead of a parallel accounting
path, extended beyond vision to every aux task, and wired the analytics
endpoints so the dashboard actually shows it. Credit to @eeksock for
the approach and @tboatman for the detailed root-cause analysis.
* test(moa): match _validate_llm_response mock to new accounting-hint signature
* test(aux): accept accounting-hint kwargs in remaining _validate_llm_response mocks
The end-to-end _build_api_kwargs parity test asserted the Nous Portal
tags exactly equal the base two-tag list. With the per-session
conversation tag, a real agent (which has a session_id) now emits a
third `conversation=<session_id>` tag. Assert against
nous_portal_tags(session_id=agent.session_id) so the check stays exact.
Two consecutive incomplete assistant interims with identical visible
content (content + reasoning) are collapsed even when opaque provider
state (encrypted reasoning item ids, message item phases) drifts per
continuation — previously that drift defeated dedup and caused message
storms (#52711). The latest opaque payload is written onto the existing
message in place, so continuation replay still uses fresh provider
state.
Salvage note: the original PR also made the 3-retry continuation cap
cumulative per turn (no reset on progress). That half is intentionally
dropped — a legitimate long turn alternating incomplete/progress would
hard-fail at 3 cumulative, changing semantics beyond the reported bug.
Map Codex Responses status=incomplete with incomplete_details.reason=content_filter to finish_reason=content_filter so the existing refusal/fallback path runs instead of burning incomplete continuation attempts.
Community reports of GPT 'infinitely thinking' are usually a slow or
overloaded provider plus silent retry machinery: the CLI/TUI/Desktop
spinner shows a generic 'cogitating...' verb for the whole wait and the
gateway heartbeat says only 'Working — N min'.
Add AIAgent._emit_wait_notice(): rewrites the live spinner/status line
(thinking_callback → CLI prompt_toolkit widget, thinking.delta → TUI +
Desktop) and updates the activity tracker (included in the gateway's
'⏳ Working — N min' heartbeat). Wired at the four wait points:
- non-streaming wait loop: after 30s with no response, the line becomes
'⏳ waiting on <model> — Ns with no response yet (provider may be slow
or overloaded; auto-reconnect at Ns)'
- streaming wait loop: same explanation after 30s with no chunks,
including the long-thinking case
- TTFB / stale-stream kills: '⚠ no response from provider in Ns —
reconnecting...'
- Codex continuation retries: '↻ model returned reasoning with no final
answer — asking it to continue (n/3)' instead of silence
Notices are fail-open (display errors never break the wait loop) and
gateway sessions without a display callback still get the improved
activity description.
grok-4.x on the xAI /v1/responses surface sometimes ends a turn with only
reasoning items — no message output item, no tool calls — and those
reasoning items carry no encrypted_content. Two compounding problems:
1. The model occasionally emits its final answer INSIDE the reasoning
channel, delimited by grok's internal "<response>" tag. The answer
exists but is classified reasoning-only → finish_reason=incomplete.
2. An interim assistant message holding only plain-text reasoning replays
as nothing in _chat_messages_to_responses_input, so every continuation
request is byte-identical to the one that just failed. The model
deterministically repeats the reasoning-only response until the retry
budget is exhausted and the turn dies with "Codex response remained
incomplete after 3 continuation attempts".
Fixes:
- _normalize_codex_response (xai_responses only): salvage the
<response>-delimited tail from the reasoning text and promote it to
assistant content; the untagged prefix stays as thinking text.
- Codex-incomplete continuation path: when the interim message has
nothing the input converter will replay (no content, no encrypted
reasoning items, no message items), append a user-role nudge so the
retry actually differs and explicitly asks for the final answer /
pending tool call. Mirrors the existing _get_continuation_prompt
pattern used for length truncation.
Observed live with grok-4.20 on xai-oauth (2026-07-13); sibling of the
grok-composer web_search incomplete-loop fix in transports/codex.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to the salvaged #64449: the status-trusting branch flipped
github_responses to 'stop' alongside unknown relays. Copilot fronts the
same OpenAI model family as codex_backend and shows the same
reasoning-only 'still thinking' degeneration, so it stays on the
continuation path. Only unrecognized (other:*) backends trust
response.status='completed' as terminal.
The chat_completions path raises EmptyStreamError when a stream yields
no chunks and no finish_reason (#64420). _call_anthropic() had no
equivalent, and the eventless failure surfaces differently by client:
- Real Anthropic SDK: no message_start means no final-message snapshot,
so get_final_message() raises a bare AssertionError — not in the
retry loop's transient set, so it burned no retries and surfaced raw.
- OpenAI-compat shims: may fabricate a contentless Message with no
stop_reason (or return None), flowing out as a 'successful' empty turn.
Track whether any stream event arrived and normalize both shapes to
EmptyStreamError, giving the anthropic path the same transient retry
budget in the shared _call() retry loop. A real completed response
always carries a stop_reason, so the guard cannot fire on legitimate
turns (including eventless mocks with stop_reason='end_turn').
Follow-up to #64420.
Follow-up to @michaelHMK's cherry-picked fix for #50892:
- agent/tool_dispatch_helpers.py: guard the segment planner's except-path
debug log against non-string arguments (the planner replaced the old
_should_parallelize_tool_batch body after #64460 and inherited the same
latent arguments[:200] slice).
- tests: the PR's executor-coercion hunks were dropped as redundant —
both executors now route through _parse_tool_arguments, which already
rejects null/non-object args with a structured error result instead of
coercing to {} (the 'we do not repair bad model outputs' contract).
Reworked the salvaged tests to pin the current behavior: None args are
rejected without dispatch, valid siblings still run, the planner treats
them as a barrier without raising, and the mainline run_conversation
path (which normalizes None to '{}' before dispatch) stays crash-free
under verbose logging.
A model response containing several parallel-safe reads plus one unsafe
tool used to lose ALL concurrency: _should_parallelize_tool_batch was
all-or-nothing, so a single barrier call (terminal, clarify, unknown
tool, malformed args) forced the entire batch onto the sequential path.
_plan_tool_batch_segments now splits the batch into ordered segments:
maximal contiguous runs of parallel-safe calls execute on the existing
concurrent path, barrier calls on the sequential path, strictly in the
model's emission order. Invariants preserved:
- one tool result per call, appended in emission order (segments are
contiguous, so no result reordering across a barrier)
- side-effect boundaries: no call starts before an earlier barrier ends
- overlapping file targets split into separate ordered parallel runs
- turn-end budget enforcement + /steer injection run exactly once per
batch (segment executors run with finalize=False; the segmented
dispatcher owns the whole-turn finalize)
- interrupt during segment k drains segments k+1..n with cancelled
results, keeping one result per tool_call_id
Homogeneous batches keep their original single-path dispatch (zero
behavior delta); _should_parallelize_tool_batch remains as a thin view
over the planner for existing callers and tests.
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.
Example:
agent:
reasoning_effort: "medium" # global default
reasoning_overrides:
"openrouter/anthropic/claude-opus-4.5": "xhigh"
"openai/gpt-5": "low"
"claude-sonnet-4.6": "high" # bare model name also works
The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.
Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)
Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
+ _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
+ Re-resolves reasoning_config for the fallback model (best-effort)
Closes#21256 (per-model reasoning_effort defaults).
Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
Follow-up to the cherry-picked empty-user-turn drop: the placeholder
introduced in 8582f35d9 fired for whitespace-only STRING turns too
(content=' ' flattens to non-stripping text but isn't in the
(None, '', []) exclusion set), fabricating an attachment note for a turn
that carried nothing. Gate the placeholder on isinstance(content, list)
so only genuinely structured (e.g. image-only) turns get it; empty and
whitespace-only string turns now fall through to the drop path.
Edge cases verified: trailing empty user turn still ends the view on the
synthetic advisory marker; an all-empty transcript degenerates to [].
MoA's _reference_messages() unconditionally appended every user-role
message to the advisory view sent to reference models, even when the
message content was an empty string or a non-string/multimodal payload
that the text-extraction step flattens to "".
Strict providers (Kimi/Moonshot, and others that enforce non-empty user
content) reject such a message with:
400 Invalid request: the message at position N with role 'user'
must not be empty
Lenient providers (DeepSeek) accept it, so an identical rendered view
passes on one reference and 400s on another within the same fan-out —
the user sees "kimi doesn't support MoA" when the real cause is an empty
user turn leaking into the advisory transcript.
Skip empty user turns, mirroring the existing behavior for empty
assistant turns (which are already dropped when they carry no parts).
The end-on-user invariant is preserved: the synthetic advisory-request
user turn is still appended when the view would otherwise end on an
assistant turn.
Adds a regression test asserting the advisory view contains no empty
user turn and still ends on a user turn.
Review follow-up to the reasoning_config cache-parity fix:
- Only inherit the parent's reasoning_config when the fork runs on the
parent's model (not routed). On the routed aux path
(auxiliary.background_review.{provider,model}) the cache is cold
regardless, so parity buys nothing, and the parent's effort vocabulary
can be invalid for the routed model/provider: OpenRouter
extra_body.reasoning.effort is forwarded unclamped
(chat_completions.py) and codex_responses only maps max/ultra for
gpt-5.6 — an exotic parent effort routed to a strict provider could
400 the review. Mirrors the existing 'not _routed' gate on
_cached_system_prompt / session_start three lines below.
- Add a routed-path regression test asserting reasoning_config is
omitted from the fork kwargs when _resolve_review_runtime returns
routed=True.
- Extract the four copy-pasted recorder stubs in
test_background_review_cache_parity.py into a single
_make_recorder_class() factory so a new fork attribute needs one stub
edit, not four.
PR #17276 painstakingly pinned `_cached_system_prompt`, `session_start`,
`session_id`, and the toolset config on the background-review fork so its
outbound request body would byte-match the parent's and hit Anthropic's
exact-prefix cache. The contributor measured a ~26% end-to-end cost
reduction on Sonnet 4.5.
That optimization is currently being silently undone by a missing
`reasoning_config` kwarg. The fork's `AIAgent(...)` call omits it, so the
fork's `reasoning_config` defaults to `None`. `anthropic_adapter.build_anthropic_kwargs`
(line ~2165) then short-circuits the `thinking` / `output_config` block,
and the fork's request body lands in a DIFFERENT Anthropic cache namespace
from the parent's.
Result on the wire: 0 `cache_read_input_tokens`, full `cache_creation_input_tokens`
of the entire parent prefix — every single background review.
7 days of midagent.db traffic from one host running stock Hermes against
Anthropic Sonnet:
```
Background-review FIRST calls (the moment a review fork is born):
count = 68
cache_write tokens = 7,004,297
cache_read tokens = 1,016,335
Cost on Sonnet ($3.75/M write vs $0.30/M read):
Spent on these writes: $26.27
Cost if they had hit parent cache instead: $2.10
WASTED: $24.16 / week / user
```
That is from one user. Multiply by Hermes's installed base for the full
impact.
Tested against api.anthropic.com directly (see refs/api-tests/ in the
attached investigation repo if needed):
| pair | cache_r | cache_w |
|---------------------------------------------|---------|---------|
| parent fresh | 0 | 24,047 |
| parent same again | 24,047 | 0 |
| fork: appends 2 new tail msgs, thinking ON | 24,047 | 22 |
| fork: appends 2 new tail msgs, thinking OFF | 0 | 24,047 |
Same fork-shape request, only difference is `thinking`. With the fix,
the fork hits the parent's full prefix and only writes the delta
(the `Review the conversation above…` prompt block, ~3-5K tokens).
One line in `agent/background_review.py`: pass
`reasoning_config=getattr(agent, "reasoning_config", None)` to the
`AIAgent(...)` constructor of the review fork. A short comment block
above it explains why so the next person who reads this code doesn't
re-introduce the regression.
`tests/run_agent/test_background_review_cache_parity.py` already covers
the system-prompt / session-id / toolset-config parity contracts that
PR #17276 introduced. I added:
* a `reasoning_config` attribute to `_make_agent_stub` so the stub has
a non-None parent value the test can verify is propagated.
* `test_review_fork_inherits_parent_reasoning_config()` — asserts the
fork's `AIAgent(...)` kwargs carry the parent's `reasoning_config`.
Pre-fix this test fails with `None vs expected {'enabled': True, 'effort': 'medium'}`;
post-fix all 4 tests in the file pass.
```
$ python -m pytest tests/run_agent/test_background_review_cache_parity.py -v
test_review_fork_inherits_parent_cached_system_prompt PASSED
test_review_fork_pins_session_start_and_session_id PASSED
test_review_fork_inherits_parent_toolset_config PASSED
test_review_fork_inherits_parent_reasoning_config PASSED ← new
```
Also runs against the broader background-review test suite:
`test_background_review.py` (4), `test_background_review_summary.py` (8),
`test_background_review_toolset_restriction.py` (3) — 19/19 pass.
`agent/curator.py:1691` has the same omission for the umbrella-curation
fork, but curator's prompt is "curate all skills" — it shares no prefix
with any user conversation, so cache-parity is a non-issue there. Worth
auditing if the curator ever takes a parent conversation as input, but
not part of this PR.
The `agent/auxiliary_client.py:1006` `reasoning_config=None` hardcode is
intentional (title/summary one-shots on short prompts — per-call cost
of namespace flip is negligible) and is also out of scope.
Provider auto-detection (URL-based inference for Anthropic, OpenAI Codex,
and xAI endpoints) runs before credential-pool validation in AIAgent init,
but #63048 placed the pool validation before auto-detection. When the agent
is constructed with provider=None and a recognized endpoint URL, the pool
is validated against an empty provider identity and discarded, even though
auto-detection correctly resolves the provider moments later.
Fix: move the credential-pool validation block to after the URL-based
auto-detection chain. The pool is stored on the agent before
auto-detection; validation now checks the resolved provider and only
nullifies agent._credential_pool when the pool's scoped provider genuinely
doesn't match.
Regression test covers all three auto-detection paths:
- Anthropic (api.anthropic.com)
- OpenAI Codex (chatgpt.com/backend-api/codex)
- xAI (api.x.ai)
Fixes#63425.
Salvage of #63888. The original fix clears stale _last_content_with_tools
on substantive tool-only turns but doesn't clear _mute_post_response, which
a prior housekeeping turn may have set. This suppresses tool progress
output via _vprint until the no-tool-call branch resets it at line ~4834
— after all tools have finished executing.
Fix: also reset _mute_post_response = False when clearing stale fallback.
Added test: verify pure housekeeping turns (content + only housekeeping
tools) still set the fallback correctly — the original use case the
fallback was designed for.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
A cached _last_content_with_tools response from a housekeeping-only turn
could survive a later substantive tool-only turn. When the model returned
an empty response, Hermes incorrectly finalized the older housekeeping
narration instead of invoking the post-tool empty-response nudge.
Production impact: scheduled cron jobs could return early without completing
their actual work (e.g., daily report job returning a housekeeping message
instead of producing the report artifact).
Root cause: The fallback state was only updated when a turn had both
content AND tool_calls. A turn with tool_calls but empty visible content
would skip state updates entirely, leaving stale fallback state intact.
Fix: Classify tools in every tool-call turn (regardless of visible content).
When any tool is substantive (non-housekeeping), clear the older fallback state
before processing later empty responses. This prevents two-turn-old housekeeping
narration from being treated as if it belonged to the immediately preceding
substantive tool turn.
Regression test added: tests/run_agent/test_conversation_fallback_state.py
Fixes#63860
Cache-decorated turns (apply_anthropic_cache_control converts string
content to [{type: text, ..., cache_control}] lists — applied BEFORE the
MoA facade since the #57675 cache-cold fix) and multimodal turns
(text + image_url parts) flattened to empty strings in
_reference_messages, which only read str content. On turn 1 of a
provider:moa session with a Claude aggregator the references received a
single EMPTY user message: Anthropic-side providers 400'd ('messages: at
least one message is required') while tolerant models answered 'no user
request is present' (live incident Jul 14 2026, preset 'closed').
Fixes, in totality:
- _reference_messages: extract visible text via
agent/message_content.flatten_message_text for user/assistant/tool
turns (skips image parts, so no base64 leaks into the advisory view);
decorated and undecorated transcripts now produce a byte-identical
advisory view (advisor cache prefix stays stable).
- image-only user turns get a placeholder instead of an empty message
(Anthropic rejects empty text blocks) or a silently dropped turn
(would break user/assistant alternation).
- degenerate-case fallback flattens structured content too.
- _attach_reference_guidance: a decorated/multimodal trailing user turn
now receives the guidance as a NEW text part appended AFTER the
cache_control-marked part (cached prefix byte-stable) instead of
falling through to a second consecutive user message (strict providers
reject user/user).
- conversation_loop MoA injection: multimodal user turns get the MoA
context appended as a trailing text part instead of being dropped;
user_prompt for the one-shot path flattens content lists instead of
str()-ing them (which leaked base64 payloads into the prompt).
Live-verified on the 'closed' preset (real OpenRouter wire, 2 user
turns, tool loop): all 4 reference calls carry the full document +
rendered tool state, end on user, zero tool-role/tool_calls; advisor
cache_write 7968 then cache_read 5909+; aggregator cache_read
14880-15237 on iterations 2+.
Co-authored-by: bo.fu <bo.fu@meituan.com>
Salvage of #63862. is_output_cap_error() returns False for vLLM/LM Studio
error messages that contain 'prompt contains ... input tokens' (treated as
input-overflow signal). But parse_available_output_tokens_from_error() CAN
extract a valid available_tokens from those same messages. The
compression-disabled guard only checked is_output_cap_error(), so vLLM/LM
Studio users with compression off still got a terminal failure instead of
the max-tokens retry.
Fix: also exempt when parse_available_output_tokens_from_error() returns a
value — that function determines whether the retry path can actually handle
the error, so it's the right predicate for the exemption.
Added test: verify vLLM-format error with compression_disabled=False still
triggers the max-tokens retry path.
Co-authored-by: dmabry <dmabry@users.noreply.github.com>
Bundle Fireworks AI as a first-class BYOK provider across the CLI, web/TUI,
and desktop onboarding.
- New model-provider plugin with attribution headers (HTTP-Referer / X-Title)
so Fireworks can attribute Hermes traffic; PAYG-safe default aux + fallback
models (accounts/fireworks/models/...), IDs tracking fw-ai/fireconnect.
- Registered in CANONICAL_PROVIDERS so it appears in the CLI/web/TUI pickers.
- Alias wiring (fireworks-ai, fw) into both CLI resolvers.
- First-class wiring: OPTIONAL_ENV_VARS, HERMES_OVERLAYS (FIREWORKS_BASE_URL
override), doctor env hints. Live catalog + model_metadata are auto-derived.
- doctor: treat Fireworks' native slash-form IDs (accounts/fireworks/...) as
valid, not aggregator vendor prefixes, so it no longer tells Fireworks users
to switch to openrouter or drop the prefix.
- picker: plugin providers with no static curated list now lead with their
profile fallback_models, so the default is an agentic chat model instead of
whatever the live catalog returns first (Fireworks listed an image model,
flux-*, ahead of its chat models).
- Desktop onboarding: Fireworks as a RECOMMENDED hero card with the official
Fireworks logomark and a brand-purple badge, routing to the BYOK key form;
i18n in en/ja/zh/zh-hant.
- Tests: profile contract, first-class wiring (both resolvers, overlay, config,
doctor incl. the slash-form regression, aux headers, credentials), discovery
spot-check, and a live smoke test driven through the Hermes runtime.
Fire Pass (fpk_) support is coming soon; the future wiring is kept as a
commented-out scaffold in the plugin.
force_close_tcp_sockets stayed shutdown-only after #29507 to avoid
cross-thread FD recycle. That left CLOSED sockets unreclaimed when
httpx.close() skipped already-shutdown sockets under long-lived
gateways (~1 CLOSED fd / 6 min via proxy).
Add release_fds= for the owning-thread dispose path only; abort still
defaults to shutdown-only.
Only restore held verification text when the loop genuinely ends through budget exhaustion. Preserve later interrupts and failures, keep generated-summary fragment explanations, and add regression coverage for both contracts.
Treat intent-ack continuation text as non-final so last-turn exhaustion requests a real summary instead of surfacing a premature promise. Keep iteration-limit fallback text free of the abnormal-fragment explainer.
Track held-back verification responses explicitly so budget exhaustion returns the composed report without a second model call. Keep unrelated error and recovery exit reasons intact, preserve Kanban timeout accounting, and cover the real run_conversation paths.
* test: deflake CI and dev-machine flaky tests in bulk
Fixes ten distinct flake sources found by mining recent CI failures and
running the full suite on a dev machine with real user state:
CI-observed races:
- tests/conftest.py live-system guard: allow signal 0 (pure liveness
probe) through _guarded_kill/_guarded_killpg. psutil.pid_exists()
probes a just-killed grandchild reparented to init; the subtree check
fails for it and the guard RuntimeError'd
test_entire_tree_is_sigkilled_not_just_parent intermittently on
unrelated PRs.
Hermeticity flakes (fail on dev machines with real state, pass on CI):
- agent/coding_context.py: _marker_root() now skips the shared temp
root (tempfile.gettempdir()) like it skips $HOME — a stray
/tmp/package.json flipped every tmp_path test into the coding
posture (9 failures in test_coding_context.py).
- test_agent_guardrails.py: pin MAX_CONCURRENT_CHILDREN=3 via autouse
monkeypatch instead of freezing the user's real config value at
import time (import-time vs call-time config mismatch).
- test_web_tools_config.py: TestCheckWebApiKey now neutralizes the
ddgs package probe and registry providers — the optional ddgs
package in a dev venv lit up the fallback backend.
- test_credential_pool.py: block claude_code/hermes-oauth credential
autodiscovery in the two pool-merge tests that assert exact id
lists (a real ~/.claude/.credentials.json seeded an extra entry).
- test_modal_sandbox_fixes.py: clear _permanent_approved /
_session_approved — the user's real command_allowlist silently
approved the guard-escalation commands under test.
- test_setup_irc.py: stub prompt_checklist to select only the IRC row;
the non-TTY cancel fallback re-ran the real configured platforms'
interactive setup_fn, which hit input() under captured stdin.
- test_doctor.py: TestGitHubTokenCheck now patches the module-level
HERMES_HOME constant (the file's established pattern) instead of
only setenv — doctor was running PRAGMA integrity_check against the
real multi-GB state.db and blowing the 300s per-file budget.
Latent atexit-duplication (same _enter_buffered_busy class as #34217):
- test_undo_command.py: drop importlib.reload(tui_gateway.server) in
fixture teardown; reload re-registers the module's atexit hooks.
- test_session_platform_resolution.py: drop per-test reload of
tui_gateway.server; every resolver reads env at call time.
* test: sentinel model value in ignore-user-config fallback assertion
With HERMES_IGNORE_USER_CONFIG=1, load_cli_config() falls back to the
repo-root cli-config.yaml (untracked, gitignored). On a dev machine that
file can legitimately set the same popular model the test hardcoded
(anthropic/claude-sonnet-4.6), flipping the != assertion locally while
CI (no cli-config.yaml) stayed green. Use an impossible sentinel model
name instead.
Three tests for the #61099 salvage: OpenRouter attribution headers
present after switching to openrouter.ai, Kimi User-Agent sentinel
present after switching to api.kimi.com, and stale headers cleared
when switching to a provider with no URL-specific headers.
2/3 fail on unpatched main (DID NOT ATTACH), confirming the bug.
switch_model() unconditionally set agent.provider but only set
agent.base_url when the resolved value was truthy. When a real
provider change resolved an empty base_url (e.g. minimax after
copilot), the agent ended up with provider="minimax" but
base_url still pointing at api.githubcopilot.com. That incoherent
pair then got snapshotted into agent._primary_runtime, so it kept
re-applying on every subsequent turn via restore_primary_runtime()
until the process restarted.
try_activate_fallback() and _swap_credential() were audited and
confirmed unaffected: both always derive base_url from an actually
constructed client, never from a possibly-empty resolver hint.
Fix: when base_url is empty AND the provider is genuinely changing,
raise ValueError instead of silently keeping the old provider's URL.
This routes through switch_model()'s existing snapshot/rollback
path, and callers (tui_gateway/server.py's _apply_model_switch)
already catch and surface a clean "switch failed, staying on X"
message. Re-selecting the SAME provider with an empty base_url
(credential-only refresh) still keeps the current URL, unchanged.
Fixes#47828
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)
Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by #57229 which
already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
behavior change reverting deliberate e8c3ac2f5; belongs in its own
PR with its own justification if pursued.
Conflict resolution: preserved main's newer timestamp-stripping
(#47868) inside the copy-on-write path.
Review finding: the substring check ('kimi' or 'moonshot' in model)
under-matches bare release slugs like k2-thinking that the repo's
canonical _model_name_is_kimi_family matcher (anthropic_adapter.py)
already covers. Reuse it instead of a second ad-hoc matcher; add a
regression test for the bare-slug case.
Adapted from PR #26014's test file to the canonical seam
(tests/run_agent/test_anthropic_prompt_cache_policy.py _make_agent
helper) instead of a new top-level file with sys.path manipulation.
Covers: kimi-k2.6 + moonshot-v1 on OpenRouter (envelope layout),
kimi via Nous Portal, and the non-OpenRouter negative case.
- Return the boundary snapshot from
_launch_session_boundary_memory_flush as a local value instead of
staging it on self._session_boundary_snapshot. The instance-attr
handoff could leak (no memory manager configured) or mis-fire a
stale snapshot on a later /new if an exception hit between staging
and consumption. A local variable eliminates the class; the helper
also returns None when no memory manager is configured so
new_session takes the inline-switch path.
- Drop the now-dead session_id kwarg from commit_memory_session:
after the redesign no production caller passes it (gateway, TUI,
compression all use the default), and speculative params are
rejected per AGENTS.md. The explicit-old-session need is served by
cli.py's direct engine call + commit_session_boundary_async.
- Drop the dead providers snapshot in commit_session_boundary_async
(only the emptiness check used it).
- Tests updated accordingly (dead-kwarg test removed, snapshot
assertion now covered by return-value contract).
Phase-2 gates: 2a tests/cli 1048 passed + 6 memory files 137 passed;
2b programmatic live smoke 0.38ms non-blocking caller, end→switch→sync
ordering verified; 2c structured 4-angle review — no Criticals, these
warnings fixed.
When an API error carries an httpx.Response whose body was consumed via
iter_bytes() during streaming error handling (e.g. GeminiAPIError from
agent/gemini_native_adapter.py), accessing .text raises
httpx.ResponseNotRead. The secondary exception replaced the real,
already-computed provider error (429 free-tier quota guidance) with the
generic 'Attempted to access streaming response content' message on
every turn.
Guard the .text access so it degrades to an empty snippet and falls
through to the str(error) fallback, which carries the full original
message. Mirrors the existing guards in
agent/error_classifier.py::_extract_error_body() and
agent/gemini_native_adapter.py::gemini_http_error().
Fixes#59769
Salvaged from PR #59868 (guard + regression test); the unrelated
desktop Ctrl-C fix bundled in that PR was intentionally dropped and is
triaged separately.
Sessions on sub-512K-context models were spending most of their wall-clock
re-summarizing: the 50% trigger left too little post-compaction headroom
(the incompressible floor — system prompt, tool schemas, protected tail,
rolling summary — ate most of the reclaimed space), so compaction re-fired
every 1-2 turns. Three compounding defects fixed:
- Threshold floor: models with context windows below 512K now trigger at
>=75% of the window (raise-only — a higher configured value or per-model
autoraise like Codex gpt-5.5's 85% always wins). Re-derived on
update_model() in both directions.
- No max_tokens on the summary call: the summary budget is prompt guidance
only ("Target ~N tokens"). The wire cap truncated summaries mid-section
on the Anthropic Messages / NVIDIA NIM paths (thinking models burn the
cap on reasoning first), yielding truncated or thinking-only summaries
and compaction loops. Summary token ceiling lowered 12K -> 10K to keep
the guidance within the intended 1K-10K envelope.
- Reasoning traces excluded end-to-end: inline <think>/<reasoning> blocks
are now stripped from assistant content before serialization to the
summarizer, and from the summarizer's own output before the summary is
stored (previously a thinking summarizer model's trace was persisted in
_previous_summary and re-fed into every iterative update, compounding
bloat). Native reasoning fields were already excluded.
Verified E2E with real imports against a temp HERMES_HOME: threshold table
across 64K-1M windows, override interactions (user 0.85 wins, spark 0.70
raised, gpt-5.5 0.85 kept), full compress() round-trip with a thinking
summarizer, and wire-kwargs capture proving no max_tokens is sent.
Review findings on the salvaged #60332 breaker, fixed as follow-ups:
- restore_primary_runtime() now resets the streak (third provider-swap
path; without it a recovered primary was short-circuited before a
single attempt and could never be re-proven healthy except via /model).
- interruptible_api_call (non-streaming) now carries the same breaker
(guard at entry, bump on stale_call_kill, reset on success). Quiet-mode
/ subagent / headless sessions — the profile most like #58962's
unattended 494-failure session — take this path and had the identical
infinite stale-retry class.
- Partial-stream stub return now resets the streak (chunks were received,
provider demonstrably responsive).
- Consolidated the triple-duplicated counter arithmetic into shared
helpers (_stale_streak/_bump_stale_streak/_reset_stale_streak/
_check_stale_giveup) with one canonical comment block; error message
now says 'consecutive stale attempts' (the counter counts kills, not
turns — a single turn can produce several).
4 new tests (restore resets / no-op restore keeps latch / non-streaming
short-circuit / non-streaming success reset).
Follow-up for the salvaged #60332 circuit breaker. The breaker latches:
once the streak trips, interruptible_streaming_api_call raises before any
stream is attempted, so the on-success reset can never run again. The
error text tells the user to switch models and retry — but neither
switch_model() nor try_activate_fallback() cleared the streak, so a
freshly selected healthy provider kept short-circuiting forever (only
/new recovered), and the automatic fallback chain was wedged the same way.
Reset the streak at both swap sites (after a successful rebuild only;
rollback/exhaustion paths keep the latch). 4 tests.
A session wedged against an unresponsive OpenAI-compatible provider can hit the stale-stream detector on every turn and loop forever, burning the full 180s x retries each turn with no response. Issue #58962 reports 494 consecutive failures over 3+ days on a single session.
The streaming retry path already caps retries WITHIN a turn (HERMES_STREAM_RETRIES, default 2) but has no cross-turn cap. Once a session's conversation state makes every turn stale, it retries indefinitely across turns and never notifies the user.
Add a per-session consecutive-stale-stream counter on the agent:
- incremented on every stale-stream kill in the outer poll loop;
- reset to 0 only when a stream actually completes;
- when it reaches HERMES_STREAM_STALE_GIVEUP (default 5), the next turn aborts immediately with a clear, actionable RuntimeError instead of spending 180s x retries again.
This is distinct from the existing stale-stream work (local-provider hard ceiling #44938, backoff/parse-error #60031): those bound a single hung stream, while this bounds repeated cross-turn staleness and surfaces a user-visible error.
Adds tests/run_agent/test_stream_stale_circuit_breaker.py covering the short-circuit, the success-reset, and the increment.
Drop the Responses-API native compaction path and its opt-in umbrella
flag from the salvaged feature. On the Codex OAuth chat route Hermes
owns the message list and the summary compressor works (and stays
provider-portable — encrypted compaction items would lock the session
history to chatgpt.com and break /model switches and provider
fallback). On the app-server runtime (codex CLI/agent) the codex agent
owns the real thread context, so thread/compact/start is the only
mechanism that can actually shrink it (#36801) — that path is now the
default behavior for codex_app_server sessions, controlled by
compression.codex_app_server_auto (native|hermes|off), no umbrella
flag.
Removed: responses.compact() call path, codex_compaction_items replay/
persistence plumbing, codex_native_compaction + codex_responses_threshold
config keys, desktop settings fields, and their tests. Kept: everything
app-server (compact_thread(), compaction notifications, bookkeeping,
docs, tests) plus cache-busting keys for the surviving knobs.
The provider-mismatch guard now checks pool_provider and
current_provider != pool_provider. MagicMock.provider returns
a truthy child mock by default, which would trigger the guard
and skip the pool recovery tests. Set pool.provider='' explicitly.
When context.engine selects a plugin engine (e.g. LCM), the host
compression threshold — including the Codex gpt-5.5 50% -> 85%
autoraise — only configures the built-in ContextCompressor and never
reaches the plugin. The autoraise notice still fired, telling the user
auto-compaction was raised when nothing actually changed, and the
startup context-limit line printed the host percent next to the
engine's own threshold_tokens, contradicting itself.
- Clear _compression_threshold_autoraised when a plugin engine is
selected, suppressing both the CLI startup notice and the gateway
turn-1 replay via _compression_warning.
- Print the active engine's own threshold_percent in the startup
context-limit line so percent and token count agree.
- Built-in behavior is preserved, including the fallback path where a
configured engine fails to load and the built-in compressor takes
over.
Fixes#44439
gpt-5.3-codex-spark has a native 128K context window but the default
50% compaction trigger fires at ~64K, wasting half the usable window
before the session has accumulated enough turns to summarize
meaningfully. This raises the trigger to 70% (~90K) on the Codex OAuth
route only, leaving ~38K headroom for the summary and continued
conversation before the 128K hard limit.
The override is not gated by allow_codex_gpt55_autoraise because 128K
is the model's native window (unlike gpt-5.5's artificial 272K Codex
cap). Non-Codex routes are unaffected.
Also adds a boundary regression test verifying the short-session
scenario from the issue always yields a non-empty compressible window
(no silent context wipe).
Teknium
Alex Fournier
MorAlekss
konsisumer
Anthony Ruiz
Jash Lee
Israel Lot
xxxigm
Brooklyn Nicholson
srojk34
liuhao1024
robbyczgw-cla
oppenheimor
dyreckt
kosta
Dineth Hettiarachchi
izumi0uu
ethernet
schattenan
Stephen Schoettler
Brett Bonner
Brandon Zarnitz
Fangliquan
Yingliang Zhang
Ben Kamholtz
Sora-bluesky
3ASiC
Jakub Wolniewicz
WeiYusc
Dom Bejar
kshitijk4poor
cucurigoo
Jan-Stefan Janetzky
ajzrva-sys
Enzo Adami
davidb73-hub
nanami7777777
Burke Autrey
HexLab98
Eva
David Robertson
cresslank
Bryan Nathan
Fli
sprmn24
Marcelo
Jai Suphavadeeprasit
Sk
Ignacio Pastor
webtecnica
Simplelife
Michael Huang
ScotterMonk
neo
Ziliang Peng
dmabry
Alex Jestin Taylor
giggling-ginger
Gille
joaomarcos
pedrommaiaa
Umi4Life
Kshitij Kapoor
Tosko4
gauravsaxena1997
doncazper
dsad
hmirin
isheng
AIalliAI
Tranquil-Flow