Commit Graph

271 Commits

Author SHA1 Message Date
brooklyn! 2c1d585002
Merge pull request #69655 from NousResearch/bb/out-of-credits-ux
feat(billing): consistent out-of-credits UX across CLI, TUI, and desktop
2026-07-22 21:03:38 -05:00
Teknium 9981242f88 fix(gateway): widen compression noise filter to all routine status lines
Post-sweep audit of every compression status emission (conversation_loop,
turn_context, conversation_compression): the buffered overflow/attempt-cap
retry chatter (🗜️ 'Context too large…', 'Compressed X → Y, retrying…',
'Context reduced to…'), the #69332-reworded auto-lower notice
("Auto-lowered this session's threshold…"), the aux-provider-unavailable
notice, and the concurrent-compression skip all leaked past
_TELEGRAM_NOISY_STATUS_RE on chat platforms. Add anchored alternatives for
each; the ', retrying'/'— compressing' anchors keep manual /compress
feedback ('Compressed: 30 → 12 messages') and failure/abort notices
visible per the deliberate carve-outs.

Also extract every routine compression status string into importable
template constants in agent/conversation_compression.py (single source of
truth shared by all emission sites), so tests can iterate the actual
emitted wording instead of hand-copied literals.
2026-07-22 17:14:36 -07:00
Brooklyn Nicholson 960d339f86 feat(billing): shared cross-surface out-of-credits signal
Detect a billing wall once (agent/error_classifier → FailoverReason.billing) and
map it to a recovery link + label in one place, then carry that structured
BillingBlock to every surface instead of re-parsing free-form error text per
surface.

- agent/billing_links.py: provider-agnostic slug/host → (label, billing URL)
  table (single source of truth), Nous-aware (is_nous routes to the in-app flow);
  unknown providers degrade to a readable label with no invented URL.
- conversation_loop: both billing exit paths return a billing_block through one
  helper; the guidance message carries the derived URL for every provider.
- gateway forwards billing_block on message.complete (it was dropped).
- @hermes/shared: BillingBlock type shared by desktop + TUI.
2026-07-22 18:08:59 -05:00
Brooklyn Nicholson cbf5b05c70 feat(agent): add active-turn redirect core primitive
A follow-up sent while the model is still generating previously ended the
turn: Hermes kept only the visible partial text (reasoning was display-only),
cleared the loop, and replayed the message as a fresh next turn. If the
correction referred to something that only appeared in the thinking stream,
the model no longer had that context.

Add `AIAgent.redirect(text)`: a corrective interrupt distinct from a hard
stop. It cancels only the in-flight model request (not tool workers or child
agents), stashes the correction under a lock shared with `interrupt()` so a
concurrent `/stop` always wins, and lets the loop rebuild the same logical
iteration. `_apply_active_turn_redirect()` checkpoints the reasoning that was
actually shown to the user plus any visible partial text as an ordinary
assistant message, then appends the correction as a real user turn — never
replaying incomplete signed/encrypted provider reasoning, and keeping strict
role alternation and prompt-cache stability intact. During tool execution it
degrades to `steer()` so a running tool finishes at a safe boundary.

`_fire_reasoning_delta` now only records reasoning that a display callback
actually consumed, so `show_reasoning: false` never leaks hidden provider
thinking into the persisted transcript.
2026-07-22 12:02:40 -05:00
Alex Fournier 761fce7f79 Merge origin/main into feat/hermes-relay-shared-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-22 08:55:25 -07:00
Jakub Wolniewicz 4c2e34f07d fix(moa): measure advisor guidance before compression 2026-07-22 06:57:54 -07:00
Teknium 1c2faedd88 fix(compression): unify the attempt cap across every compression site
Follow-up to the salvaged #64010 (Kenmege) and #63870 (dombejar) commits,
making one resolved compression.max_attempts cap govern ALL per-turn
compression attempt sites:

- conversation_loop: resolve max_compression_attempts ONCE at turn start
  (it was previously re-resolved inside the API-call loop) and route the
  pre-API pressure gate through it — that gate still hardcoded
  'compression_attempts < 3' and logged 'attempt=%s/3'.
- conversation_loop: the salvaged post-tool compaction gate now uses the
  resolved cap instead of a hardcoded 3.
- turn_context: the preflight compaction loop was 'for _pass in range(3)';
  it now sizes itself from the same resolved cap.
- agent_init: harden the max_attempts parser — reject booleans (bool
  subclasses int; 'true' would coerce to 1), reject fractional floats
  instead of truncating them, keep accepting integral floats and numeric
  strings; anything else falls back to 3 (floor 1, ceiling 10 unchanged).
- tests: replace #63870's inspect.getsource source-shape test with
  behavioral loop tests (post-tool compaction fires <= cap times per turn,
  shares its budget with the pre-API gate, resets between turns); add an
  e2e test proving a 4th preflight pass runs at config cap=6 while the
  unset default still stops at 3; extend the #64010 config tests with the
  bool/float parser semantics.

Salvages #64010 by @Kenmege and #63870 by @dombejar.
2026-07-22 06:56:42 -07:00
Dom Bejar fca883f06f fix(compaction): cap post-tool attempts per turn 2026-07-22 06:56:42 -07:00
Kennedy Umege 644b397fb2 fix(agent): make the compression retry cap config-driven (compression.max_attempts)
The conversation loop hardcodes max_compression_attempts = 3. Sessions
that legitimately need more rounds are stranded: on a restart history
reload, incompressible tool schemas can keep the per-request estimate
above the compressor threshold even though the message floor compresses
correctly, so three rounds cannot clear it and the turn dies with
"Context length exceeded: max compression attempts (3) reached" — the
same failure class as #62605, where the rough estimate similarly leaves
3 retries short.

Make the cap a config key, compression.max_attempts:

- default 3 = identical to today, so an unset key is behavior-neutral;
- parsed and validated in agent_init alongside the other compression.*
  keys (>= 1, hard-capped at 10, non-integer values fall back to 3),
  attached as agent.max_compression_attempts;
- the loop reads it via getattr(agent, "max_compression_attempts", 3),
  so objects without the attribute keep the prior behavior;
- documented in the DEFAULT_CONFIG compression block.

Tests pin the parse/validate/attach seam: default preserved, custom
value honored, floor and ceiling enforced, garbage tolerated, and the
loop-side getattr degradation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 06:56:42 -07:00
cucurigoo 377244f7c8 fix(compression): prevent stale-budget retry loops 2026-07-22 11:19:37 +05:30
Teknium 279be8211d Revert "fix(agent): circuit-break AttributeError from commit-splice and detect code skew"
This reverts commit 3a9b9d65d5.
2026-07-21 03:26:17 -07:00
x7peeps 3a9b9d65d5 fix(agent): circuit-break AttributeError from commit-splice and detect code skew
Fix #68178

The git-install auto-updater rewrites source while the desktop backend
is live. Because agent/conversation_loop.py is imported lazily on the
first API call, a process can end up running two different commits
spliced together — one commit's AIAgent against another commit's
conversation_loop. When the interface differs, every turn fails
permanently with an AttributeError, and the loop retries indefinitely,
burning provider API calls (576 failures, 149 wasted API calls observed).

Three-prong fix:

1. Circuit-break AttributeError on agent objects: the outer-loop error
   classifier now detects AttributeError targeting agent/run_agent
   modules and breaks immediately instead of continuing the retry loop.

2. Code skew detection for desktop/serve backend: run_agent.py now
   snapshots the checkout revision at import time and exposes a cheap
   per-iteration check that the conversation loop uses to refuse new
   work with a clear 'restart required' message before the lazy import
   can crash.

3. Informative error message: when code skew is detected, the user
   gets a clear explanation of the mismatch (boot revision vs current
   revision) and actionable guidance to restart the application.
2026-07-21 12:11:46 +05:30
Alex Fournier 67863777ca Merge remote-tracking branch 'origin/main' into tmp/hermes-relay-pr-merge-20260717
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-20 11:07:39 -07:00
Alex Fournier a15b98f414 fix(runtime): complete logical LLM calls after acceptance
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-20 11:01:14 -07:00
ethernet c363db81e0
fix(desktop, ink): don't wipe messages before final message (#65919)
* fix(desktop): preserve interim assistant text wiped at message.complete

When the agent emits interim text (commentary alongside tool calls, or the
attempted final answer before a verify-on-stop nudge), all UI surfaces
streamed it live but then wiped it at message.complete — keeping only the
final response. The user saw text appear during inference, then disappear.

This is the complete fix across all three layers: agent core, gateway
transport, and all UI surfaces (desktop + Ink TUI).

The verify-on-stop and pre_verify paths flagged the assistant's attempted
final answer as _verification_stop_synthetic, suppressing it from both
state.db and the UI. The user only saw the terse post-verification reply.

Now the assistant response is real content: it's persisted to state.db and
emitted as an interim message via _emit_interim_assistant_message(force_display=True)
before the verification loop runs. Only the synthetic nudge messages keep
the synthetic flags. The turn finalizer drops nudges from live history and
compares content (not just role) to avoid duplicating a published candidate.
Message sequence repair collapses verification candidates in the
consecutive-assistant merge.

Wire agent.interim_assistant_callback both at construction (_agent_cbs())
and per-turn (defense-in-depth), emitting a new message.interim event with
{text, already_streamed}. Gated on display.interim_assistant_messages
(default true). Cleared in the finally block so a stale closure can't
fire on a later turn.

Add message.interim to the GatewayEventName union (apps/shared) and a
typed payload to the TUI's GatewayEvent discriminated union.

The TUI already had the segment-anchoring machinery (flushStreamingSegment +
finalTail) but had no handler for message.interim. Added recordInterimMessage
+ interimBoundaryIndex to seal segments mid-turn, and updated
recordMessageComplete to only dedupe segments after the interim boundary.

Replaced the fragile sealed-set approach with a proper interimBoundaryPending
state flag on ClientSessionState. finalizeInterimAssistantMessage finalizes
the streaming bubble in place (or creates a standalone one), rotates the
stream ID so next deltas create a new bubble, and sets the flag. When the
final text equals an already-sealed interim, they stay as distinct messages.

Extracted mergeFinalAssistantText() as a pure function in chat-messages.ts,
used by both completeAssistantMessage and finalizeInterimAssistantMessage.
Split the bidirectional dedup predicate: reasoning is a restatement only when
the final FULLY covers it. A short final ("Done.") no longer swallows a
longer reasoning block that merely starts with it.

Honor display.interim_assistant_messages (default true) across all layers:
the tui_gateway gates the callback, the desktop wires it to a nanostores
atom via use-hermes-config. Updated hermes_cli/config.py and
cli-config.yaml.example comments to document the Desktop behavior.

_split_segment_tokens now accepts posix=False and _find_ad_hoc_match tries
both posix modes so ad-hoc verification scripts with Windows backslash
paths are matched correctly. (response_previewed forwarding from #53553
is not included — our emit-interim + persist approach makes it unnecessary
since the attempted answer is now surfaced before the verification loop.)

- tsc: clean (desktop + TUI + shared)
- vitest desktop: 73/73 pass (7 interim-sealing + 5 mergeFinalAssistantText + 4 config atom)
- vitest TUI: 83/83 pass (4 new message.interim tests)
- python: 390 tests pass (340 tui_gateway + 33 verification/finalizer + 6 config gating + 3 evidence + 8 continuation budget)

Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>

* fix: prefix-match interim streamed content to avoid benign duplicate bubbles

_interim_content_was_streamed used exact equality (streamed == visible_content),
so a final response that was the streamed text plus a trailing delta — or a
partial stream before the verify nudge fired — failed the match and left
_response_was_previewed false. The turn then showed two bubbles (interim +
identical final) instead of settling the interim in place.

Relax to a prefix check (visible_content.startswith(streamed)) in both the
core match and the desktop's settle-in-place gate. The TUI already used
prefix matching via finalTail. The reverse direction (streamed longer than
final) is intentionally not matched — that could suppress a needed resend
in the gateway path where already_streamed=True calls on_segment_break().

* test(desktop): add partial-stream-then-nudge dedup edge case

Third edge case for the interim-sealing dedup: model streams part of its
answer via message.delta, verify nudge fires, interim seals the streamed
prefix, then the final response is the same text plus a trailing delta.
Asserts one bubble (not two) containing the full final text.

Acceptance protocol #2 — covers all three dedup edges:
  1. interim == final (existing)
  2. interim = strict prefix of final (existing)
  3. partial-stream-then-nudge (this commit)

---------

Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
2026-07-20 11:42:29 -04:00
Brooklyn Nicholson b0f60622aa style(agent): tighten request-estimate comment 2026-07-19 19:54:31 -05:00
Brooklyn Nicholson c1c4e56e7e perf(agent): drop per-call base64 re-serialization from request-size estimate
Every API iteration computed `total_chars = sum(len(str(msg)) ...)`, which
str()-serializes the ENTIRE history — including base64 images and large tool
results — just to take its length, then called estimate_request_tokens_rough,
which walked the messages a SECOND time (it re-runs estimate_messages_tokens_rough
internally, already computed one line above).

Now derive both from one image-stripped message estimate:
  approx_tokens = estimate_messages_tokens_rough(api_messages)   # once
  request_pressure_tokens = approx_tokens + tools_tokens          # == old value
  total_chars = approx_tokens * 4                                 # log/metric only

request_pressure_tokens is byte-identical to the old
estimate_request_tokens_rough(api_messages, tools=agent.tools or None) (no
system_prompt arg → messages + tools). total_chars only feeds a verbose log and
the pre-api-request hook's request_char_count, so a rough proxy is fine and it no
longer balloons on image turns. On the TTFT critical path for every call.

tests/agent/test_model_metadata.py + test_compressor_image_tokens.py green.
2026-07-19 19:48:12 -05:00
Alex Fournier 2c8fd38f2e fix(observability): defer logical LLM completion until validation
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 09:00:27 -04:00
Alex Fournier 55b7903cf7 fix(observability): harden Relay terminal lifecycle metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:59:42 -04:00
Alex Fournier 4dedaa4237 refactor(runtime): consolidate Relay lifecycle ownership
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:59:06 -04:00
Alex Fournier 9bc521b138 refactor(runtime): manage Relay LLM tool and subagent execution
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:57:38 -04:00
Soju06 7b3dcee928 feat(cache): persist the exact bytes sent to the API in an api_content sidecar
The first LLM call of every gateway turn gets ~0% provider prompt-cache
hit rate (in-turn calls: 97-99%) because the bytes sent for a turn's
user message are not the bytes replayed next turn: memory-prefetch and
pre_llm_call context are injected into the API copy only, the #48677
persist override writes cleaned content to the DB row, and
get_messages_as_conversation sanitize/strips user and assistant content
on load. Any of these diverges the request prefix at that message and
re-prefills everything after it — measured 27.9s for the first call vs
2.4-5.8s cached at a median ~156k-token context.

Persist what you send: a nullable messages.api_content column stores the
exact content string sent to the API when it differs from the clean
stored content, and replay substitutes it verbatim (no sanitize, no
strip). The injection composition lives in one helper
(turn_context.compose_user_api_content); the turn prologue stamps its
output onto the live user message, the api_messages build sends the
stamped bytes, and every outgoing copy pops the field so it never
reaches a provider. The crash-resilience user-turn persist moves after
prefetch/pre_llm_call so the user row is written once with its final
sidecar; _ensure_db_session stays before preflight compression (session
rotation needs the parent row under PRAGMA foreign_keys=ON). The
current-turn index trackers are re-anchored after compaction rebuilds
the message list, in-place preflight compaction backfills the stamp onto
the already-inserted row, and gateway replay forwards the sidecar only
when the replay pipeline did not rewrite the content. Rewrite paths that
would leave stale bytes (historical image strip, merge-summary-into-
tail, consecutive-user repair merge, stale-confirmation redaction) drop
the sidecar; the chat-completions transport and the max-iterations
summary path strip it defensively. codex_app_server and MoA turns are
excluded from stamping because their wire bytes differ from the
composition. A missing or dropped sidecar degrades to today's behavior
(one cache-boundary miss), never to wrong content.
2026-07-19 08:25:35 +05:30
kshitijk4poor bd3d16a490 fix: salvage follow-up — remove redundant coercion, fix classifier, restore None guard
1. Remove PR's redundant strip_think_blocks coercion (Teknium's fix
   296494db0 already handles this at the same chokepoint with superior
   logic that drops thinking/reasoning blocks).

2. Restore 'if not content: return ' guard at top of strip_think_blocks
   that was lost during cherry-pick auto-merge. Without it, None content
   hits str(None) → 'None' string instead of returning empty.

3. Fix error classifier design flaw: remove 'conversation_loop' and
   'run_agent' from _local_processing_modules — these are the container
   modules for the try/except, so every exception passes through them,
   making _hit_local always True and misclassifying transient API/network
   errors as non-retryable local bugs.

4. Move module sets to module-level frozenset constants (_LOCAL_PROCESSING_MODULES,
   _API_CALL_MODULES) instead of rebuilding on every exception.

5. Replace traceback.extract_tb() with raw tb walk — avoids disk I/O for
   source lines that are never used.

6. Remove unused 'import traceback'.

7. Fix docstring corruption: 3 lines where think tags were replaced
   with Chinese characters during the PR's editing.
2026-07-18 19:39:07 +05:30
nanami7777777 aef0fe6f27 fix: handle multimodal content in interim assistant text and avoid retrying local processing errors (#66267) 2026-07-18 19:39:07 +05:30
Teknium 348e9912ff
fix(agent): execute valid tool calls in mixed batches with invalid names (#66317)
Degrading models (observed with gpt-5.6 past ~350K input) emit tool-call
batches like 6 valid named calls + 1 blank-name call. Previously the
whole turn was voided — every valid call got 'Skipped: another tool call
in this turn used an invalid name' — and three such batches tripped the
3-strike stop, killing sessions that were still making progress.

Now a mixed batch error-results ONLY the invalid call(s) (terse
anti-priming error for blank names per #47967, catalog dump for typos)
and dispatches the valid subset for execution. The assistant message
keeps every emitted call so provider-side tool_call/result pairing stays
intact. The 3-strike counter only advances when a turn contains NO valid
call, so a fully-degenerate model still stops while a mostly-coherent
one keeps working. Broken JSON args on a never-executing invalid call no
longer trigger the whole-turn JSON retry loop.

Field evidence: July 2026 debug bundle showed gpt-5.6-sol emitting
6-call batches with one blank-name rider at 559K/384K-token context in
two separate sessions; 13 valid tool calls were discarded before the
session stopped as partial.
2026-07-17 06:48:42 -07:00
Eva 7041c56cdf feat(agent): stream Codex commentary separately 2026-07-16 23:27:12 -07:00
Eva b008131b54 fix(agent): harden Codex commentary interim delivery 2026-07-16 23:27:12 -07:00
JiaDe-Wu 5e6a0d9eea fix(bedrock): streaming fallback to Converse API + image base64 decode + bearer token routing
Three fixes for the Bedrock Claude path:

1. Streaming fallback: When AnthropicBedrock SDK raises 'Unexpected event
   order' (SDK misparses Bedrock error events as message_start), auto-switch
   to native Converse API for the rest of the session instead of failing
   after 3 retries.

2. Image base64 decode (#33317): data URL payloads were passed as base64
   strings to source.bytes, but boto3 re-encodes at the wire layer. Now
   decoded to raw bytes before passing to Converse API.

3. Bearer token routing (#28156): Users with AWS_BEARER_TOKEN_BEDROCK are
   now routed through Converse API regardless of model, since the
   AnthropicBedrock SDK only supports SigV4 signing.

3 new tests. 121 bedrock_adapter tests passing.
2026-07-15 09:59:38 -07:00
liuhao1024 6a8e7069b4 fix(agent): dedup codex incomplete interims on visible content
Two consecutive incomplete assistant interims with identical visible
content (content + reasoning) are collapsed even when opaque provider
state (encrypted reasoning item ids, message item phases) drifts per
continuation — previously that drift defeated dedup and caused message
storms (#52711). The latest opaque payload is written onto the existing
message in place, so continuation replay still uses fresh provider
state.

Salvage note: the original PR also made the 3-retry continuation cap
cumulative per turn (no reset on progress). That half is intentionally
dropped — a legitimate long turn alternating incomplete/progress would
hard-fail at 3 cumulative, changing semantics beyond the reported bug.
2026-07-15 09:49:11 -07:00
Sk fe1ab949fd fix(agent): treat Codex incomplete content filter as refusal
Map Codex Responses status=incomplete with incomplete_details.reason=content_filter to finish_reason=content_filter so the existing refusal/fallback path runs instead of burning incomplete continuation attempts.
2026-07-15 09:47:37 -07:00
Teknium 569b912d7d
feat(agent): explain long provider waits on the live status line (#64775)
Community reports of GPT 'infinitely thinking' are usually a slow or
overloaded provider plus silent retry machinery: the CLI/TUI/Desktop
spinner shows a generic 'cogitating...' verb for the whole wait and the
gateway heartbeat says only 'Working — N min'.

Add AIAgent._emit_wait_notice(): rewrites the live spinner/status line
(thinking_callback → CLI prompt_toolkit widget, thinking.delta → TUI +
Desktop) and updates the activity tracker (included in the gateway's
'⏳ Working — N min' heartbeat). Wired at the four wait points:

- non-streaming wait loop: after 30s with no response, the line becomes
  '⏳ waiting on <model> — Ns with no response yet (provider may be slow
  or overloaded; auto-reconnect at Ns)'
- streaming wait loop: same explanation after 30s with no chunks,
  including the long-thinking case
- TTFB / stale-stream kills: '⚠ no response from provider in Ns —
  reconnecting...'
- Codex continuation retries: '↻ model returned reasoning with no final
  answer — asking it to continue (n/3)' instead of silence

Notices are fail-open (display errors never break the wait loop) and
gateway sessions without a display callback still get the improved
activity description.
2026-07-15 00:14:05 -07:00
Teknium 2fc0e3d1aa fix(codex): guard the continuation nudge against role-alternation violations
Follow-up to the salvaged #63690: when the interim assistant message is
too empty to append (no content and no reasoning of any kind), the last
message in history is still the prior user/tool turn — appending the
user-role nudge there would create a user→user or tool→user sequence
that strict providers reject. Only append the nudge when the last
message is an assistant turn. Also add IpastorSan to AUTHOR_MAP.
2026-07-15 00:13:24 -07:00
Ignacio Pastor 05d1ca549b fix(codex): rescue reasoning-only turns that die with 'remained incomplete after 3 continuation attempts'
grok-4.x on the xAI /v1/responses surface sometimes ends a turn with only
reasoning items — no message output item, no tool calls — and those
reasoning items carry no encrypted_content. Two compounding problems:

1. The model occasionally emits its final answer INSIDE the reasoning
   channel, delimited by grok's internal "<response>" tag. The answer
   exists but is classified reasoning-only → finish_reason=incomplete.

2. An interim assistant message holding only plain-text reasoning replays
   as nothing in _chat_messages_to_responses_input, so every continuation
   request is byte-identical to the one that just failed. The model
   deterministically repeats the reasoning-only response until the retry
   budget is exhausted and the turn dies with "Codex response remained
   incomplete after 3 continuation attempts".

Fixes:
- _normalize_codex_response (xai_responses only): salvage the
  <response>-delimited tail from the reasoning text and promote it to
  assistant content; the untagged prefix stays as thinking text.
- Codex-incomplete continuation path: when the interim message has
  nothing the input converter will replay (no content, no encrypted
  reasoning items, no message items), append a user-role nudge so the
  retry actually differs and explicitly asks for the final answer /
  pending tool call. Mirrors the existing _get_continuation_prompt
  pattern used for length truncation.

Observed live with grok-4.20 on xai-oauth (2026-07-13); sibling of the
grok-composer web_search incomplete-loop fix in transports/codex.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 00:13:24 -07:00
Michael Huang 94456a1288 Handle null tool call arguments 2026-07-14 21:46:41 -07:00
webtecnica 398cf40c08 fix(nous): restore inference-api.nousresearch.com base_url
The upstream migration to inference.nousresearch.com broke routing for
inference-api.nousresearch.com. Restore the correct base_url and drop
the stale alias match in _is_nous_inference_route().

Fixes #60715
2026-07-14 21:31:04 -07:00
kshitij 0684506072
Merge pull request #64004 from kshitijk4poor/salvage/63274-cli-close-persist
fix(cli): persist close transcript without history alias
2026-07-14 16:45:35 +05:30
mdc2122 03fbf6edbb fix(kanban): nudge workers that exit without complete/block
Add a bounded turn-end stop guard for kanban workers. When a worker
tries to exit with finish_reason=stop without having called
kanban_complete or kanban_block, inject up to two synthetic nudges
so the conversation loop continues instead of exiting cleanly (which
the dispatcher records as protocol_violation).

Mirrors the existing verify-on-stop pattern: same ephemeral scaffolding
flag (_kanban_stop_synthetic), same role-alternation contract, same
_pending_verification_response fallback for budget exhaustion.

Disabled by default (gated on HERMES_KANBAN_TASK env var set by the
dispatcher); kill switch via HERMES_KANBAN_STOP_NUDGE=0.

Salvaged from #62262 by @mdc2122. The original branch was 272 commits
behind main with ~538 files of stale-base reversions; this salvage
applies only the 4 substantive files (agent/kanban_stop.py,
conversation_loop.py insertion, run_agent.py _EPHEMERAL_SCAFFOLDING_FLAGS,
tests/agent/test_kanban_stop.py).
2026-07-14 16:34:00 +05:30
kshitijk4poor 3c2886f599 fix(conversation): clear _mute_post_response on substantive tool-only turn
Salvage of #63888. The original fix clears stale _last_content_with_tools
on substantive tool-only turns but doesn't clear _mute_post_response, which
a prior housekeeping turn may have set. This suppresses tool progress
output via _vprint until the no-tool-call branch resets it at line ~4834
— after all tools have finished executing.

Fix: also reset _mute_post_response = False when clearing stale fallback.

Added test: verify pure housekeeping turns (content + only housekeeping
tools) still set the fallback correctly — the original use case the
fallback was designed for.

Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
2026-07-14 16:33:14 +05:30
liuhao1024 8a7d32d4e4 fix(conversation): clear stale housekeeping fallback on substantive tool-only turns
A cached _last_content_with_tools response from a housekeeping-only turn
could survive a later substantive tool-only turn. When the model returned
an empty response, Hermes incorrectly finalized the older housekeeping
narration instead of invoking the post-tool empty-response nudge.

Production impact: scheduled cron jobs could return early without completing
their actual work (e.g., daily report job returning a housekeeping message
instead of producing the report artifact).

Root cause: The fallback state was only updated when a turn had both
content AND tool_calls. A turn with tool_calls but empty visible content
would skip state updates entirely, leaving stale fallback state intact.

Fix: Classify tools in every tool-call turn (regardless of visible content).
When any tool is substantive (non-housekeeping), clear the older fallback state
before processing later empty responses. This prevents two-turn-old housekeeping
narration from being treated as if it belonged to the immediately preceding
substantive tool turn.

Regression test added: tests/run_agent/test_conversation_fallback_state.py

Fixes #63860
2026-07-14 16:33:14 +05:30
Teknium 8582f35d96
fix(moa): flatten structured message content in the advisory view (#64319)
Cache-decorated turns (apply_anthropic_cache_control converts string
content to [{type: text, ..., cache_control}] lists — applied BEFORE the
MoA facade since the #57675 cache-cold fix) and multimodal turns
(text + image_url parts) flattened to empty strings in
_reference_messages, which only read str content. On turn 1 of a
provider:moa session with a Claude aggregator the references received a
single EMPTY user message: Anthropic-side providers 400'd ('messages: at
least one message is required') while tolerant models answered 'no user
request is present' (live incident Jul 14 2026, preset 'closed').

Fixes, in totality:
- _reference_messages: extract visible text via
  agent/message_content.flatten_message_text for user/assistant/tool
  turns (skips image parts, so no base64 leaks into the advisory view);
  decorated and undecorated transcripts now produce a byte-identical
  advisory view (advisor cache prefix stays stable).
- image-only user turns get a placeholder instead of an empty message
  (Anthropic rejects empty text blocks) or a silently dropped turn
  (would break user/assistant alternation).
- degenerate-case fallback flattens structured content too.
- _attach_reference_guidance: a decorated/multimodal trailing user turn
  now receives the guidance as a NEW text part appended AFTER the
  cache_control-marked part (cached prefix byte-stable) instead of
  falling through to a second consecutive user message (strict providers
  reject user/user).
- conversation_loop MoA injection: multimodal user turns get the MoA
  context appended as a trailing text part instead of being dropped;
  user_prompt for the one-shot path flattens content lists instead of
  str()-ing them (which leaked base64 payloads into the prompt).

Live-verified on the 'closed' preset (real OpenRouter wire, 2 user
turns, tool loop): all 4 reference calls carry the full document +
rendered tool state, end on user, zero tool-role/tool_calls; advisor
cache_write 7968 then cache_read 5909+; aggregator cache_read
14880-15237 on iterations 2+.

Co-authored-by: bo.fu <bo.fu@meituan.com>
2026-07-14 03:22:48 -07:00
kshitijk4poor 2ccfdb2db4 fix(agent): exempt parseable vLLM/LM Studio output-cap errors from compression-disabled guard
Salvage of #63862. is_output_cap_error() returns False for vLLM/LM Studio
error messages that contain 'prompt contains ... input tokens' (treated as
input-overflow signal). But parse_available_output_tokens_from_error() CAN
extract a valid available_tokens from those same messages. The
compression-disabled guard only checked is_output_cap_error(), so vLLM/LM
Studio users with compression off still got a terminal failure instead of
the max-tokens retry.

Fix: also exempt when parse_available_output_tokens_from_error() returns a
value — that function determines whether the retry path can actually handle
the error, so it's the right predicate for the exemption.

Added test: verify vLLM-format error with compression_disabled=False still
triggers the max-tokens retry path.

Co-authored-by: dmabry <dmabry@users.noreply.github.com>
2026-07-14 03:58:54 +05:30
dmabry 127f6e1514 fix: exempt output-cap errors from compression-disabled guard 2026-07-14 03:58:54 +05:30
dmabry 57f1814832 fix: use provider available_out + request estimate for output-cap retry cap
The branch computed safe_out from estimate_messages_tokens_rough(messages),
but the provider rejected the larger api_messages request (system prompt,
injected context, tool schemas). When API-only content is large, safe_out
could far exceed the provider's available_tokens.

Compute safe_out from estimate_request_tokens_rough(api_messages, tools=...)
and keep provider available_out as an upper bound. Do not alter context_length
or trigger compression for output-cap errors.

Add production-path run_conversation tests that assert the retry API call's
max_tokens, including a case where a large system prompt makes messages-only
estimation undercount the real request.

Fixes #55546
2026-07-14 03:58:54 +05:30
dmabry 62ea800586 fix: recalculate safe_out from current input on each output-cap retry (#55546)
The retry loop computed safe_out from the error's available_tokens,
which reflected the *previous* request. Between retries the agent
appends tool results and error text, so the real input token count
grows. Deriving safe_out from the stale budget meant every retry
still exceeded the context ceiling by 1+ tokens, burning through the
3-attempt limit.

Compute safe_out from estimate_messages_tokens_rough(messages) so
the cap tracks the growing input on each retry attempt.
2026-07-14 03:58:54 +05:30
kshitijk4poor 0b422559f3 fix(session): preserve clean multimodal persistence override 2026-07-14 03:32:45 +05:30
kshitijk4poor 83000c7295 fix(compaction): clear stale anti-thrash verdicts 2026-07-11 22:59:49 +05:30
kshitijk4poor bce17bf6a2 fix(codex): enforce Copilot replay policy at dispatch
Reapply the endpoint-aware preflight after request and execution
middleware so no override can reintroduce a connection-scoped ID.
2026-07-11 12:09:27 +05:30
kshitijk4poor 8acac440f4 fix(cron): keep inline dispatch behind the agent call seam 2026-07-11 11:07:34 +05:30
HexLab98 d00c15c0c5 fix(cron): run gateway cron LLM calls synchronously (#62151)
Cron jobs in the gateway process wedged before HTTP on later non-streaming
API calls because interruptible_api_call spawned a daemon worker inside
nested cron thread pools. Route cron platform turns through direct_api_call
on the conversation thread instead.
2026-07-11 11:07:34 +05:30
kshitijk4poor f46e7647eb fix(agent): clear stale intermediate acknowledgments
Treat intent-ack continuation text as non-final so last-turn exhaustion requests a real summary instead of surfacing a premature promise. Keep iteration-limit fallback text free of the abnormal-fragment explainer.
2026-07-10 11:53:58 +05:30
kshitijk4poor 1453431881 refactor(agent): scope pending fallback to verification
Name the continuation fallback for its actual verification-only provenance so unrelated continuation paths cannot accidentally inherit its cron-delivery semantics.
2026-07-10 11:53:58 +05:30
kshitijk4poor cd7c203ab9 fix(agent): preserve gated responses without masking failures
Track held-back verification responses explicitly so budget exhaustion returns the composed report without a second model call. Keep unrelated error and recovery exit reasons intact, preserve Kanban timeout accounting, and cover the real run_conversation paths.
2026-07-10 11:53:58 +05:30
HexLab98 3eb937a498 fix(cron): preserve composed reports when verify-on-stop exhausts budget
Clear stale final_response before verify-on-stop/pre_verify loop
continues, and normalize iteration-limit exit reasons in turn_finalizer
when a composed answer survives with unknown/budget_exhausted.

Fixes #61631
2026-07-10 11:53:58 +05:30
Umi4Life 3fe7f6d27a fix: preserve fallback switch notice on successful fallback
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 13:40:19 +05:30
kshitijk4poor 45f5a6e659 refactor(retry): single-source Z.AI overload short-attempts + drop change-detector assert
Follow-up on the salvage of #59523. Two low-risk cleanups surfaced by review:

- Extract _ZAI_CODING_OVERLOAD_SHORT_ATTEMPTS as a module constant so
  adaptive_rate_limit_backoff() and zai_coding_overload_retry_ceiling()
  share one source of truth. Previously both hardcoded short_attempts=3
  independently; tuning one without the other would silently desync the
  retry ceiling from the backoff schedule.
- Replace the tautological formula-mirroring assert in
  test_zai_overload_retry_ceiling_exceeds_short_attempts with a behavior
  invariant (ceiling leaves headroom for every long-backoff entry), per the
  repo's contracts-over-snapshots testing rule.
2026-07-07 11:57:01 +05:30
xxxigm 1c702aa73e fix(agent): run Z.AI overload adaptive backoff on the overloaded path
Z.AI Coding Plan GLM-5.2 reports server overload as HTTP 429 code 1305
("temporarily overloaded"). classify_api_error routes that to
FailoverReason.overloaded (so a valid credential pool isn't burned), but
the adaptive Z.AI backoff was gated on is_rate_limited — which excludes
overloaded — so it never ran (policy=default) and the request failed after
a few quick short retries.

Two compounding causes, both fixed here:

1. Detect the Z.AI overload 429 directly and let its adaptive backoff run
   on the overloaded path, not only the rate_limit path.
2. Raise the retry ceiling for this narrow case via
   zai_coding_overload_retry_ceiling(). The long-backoff tier
   (30/60/90/120s) starts after short_attempts (3) retries, but the default
   api_max_retries is also 3, so the loop always gave up before the long
   tier could run — leaving the whole long-backoff schedule as dead code.

Scope is limited to the existing narrow is_zai_coding_overload_error match,
so other providers' 429/503/529 handling is unchanged.
2026-07-07 11:44:58 +05:30
Teknium ba31699091
chore(providers): remove dead cloudcode-pa quota-fallback branches (#51489)
The google-antigravity and google-gemini-cli OAuth providers were removed
in #50492. They were the only producers of a cloudcode-pa:// base_url, so
the account-level-quota early-returns in _pool_may_recover_from_rate_limit
and _credential_pool_may_recover_rate_limit are now unreachable.

- Drop the dead cloudcode-pa:// checks and the now-unused provider/base_url
  params on _pool_may_recover_from_rate_limit (only caller updated).
- Prune the obsolete CloudCode-specific regression tests; keep the live
  single/multi-entry pool-rotation invariants (#11314).
2026-07-05 13:44:33 -07:00
Teknium 55e3ee1ab8
fix: remove dead f-string prefixes via ruff F541 (216 sites) (#52336)
ruff check --fix --select F541 . on current main. Pure prefix removals;
adjacent-string concatenations keep the f only on interpolating fragments.
No string content or live placeholder altered.
2026-07-05 13:42:46 -07:00
Teknium 4751af0a0b
feat(errors): fail fast on TLS certificate verification failures with fix hints (#57992)
Inspired by Claude Code v2.1.199 (July 2, 2026): SSL certificate errors
(TLS-inspecting proxies, missing CA bundles, expired certs) no longer
burn retries before showing actionable guidance — they fail immediately
with the fix hint.

- agent/error_classifier.py: new FailoverReason.ssl_cert_verification +
  _SSL_CERT_VERIFY_PATTERNS, checked BEFORE the transient-SSL patterns
  (cert-verify messages also contain '[SSL:' and previously retried
  forever as timeout). Non-retryable, no compression, no fallback churn.
- agent/conversation_loop.py: dedicated status line + per-cause fix
  hints (corporate proxy CA bundle, certifi refresh, self-signed local
  endpoints) on the non-retryable abort path.
- 7 new tests incl. regression guards (transient alerts still retry,
  large-session cert failure doesn't trigger compression).
2026-07-05 02:05:51 -07:00
Teknium 6f052b7ff1
fix(copilot): set x-initiator per turn so user prompts bill as premium requests (salvage #4097) (#58544)
* fix(cli): set correct x-initiator header per Copilot turn

copilot_default_headers() always hardcoded x-initiator: agent, but
GitHub Copilot billing requires "user" for user-initiated prompts and
"agent" for tool/follow-up calls. This caused premium requests to never
be consumed correctly, risking billing issues or account bans.

Adds is_agent_turn param to copilot_default_headers() and injects
extra_headers={"x-initiator": "user"} on the first API call of each
user turn when targeting Copilot URLs. The flag flips to False after
injection so subsequent calls (tool use, streaming fallback) default
back to "agent".

Fixes #3040

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore(release): add AUTHOR_MAP entry for @tjp2021 (PR #4097 salvage)

---------

Co-authored-by: Tim <tim@iteachyouai.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-05 00:44:15 -07:00
kshitijk4poor 90f84144ed refactor: gate pre-API compaction through the preflight guard chain
Self-review (hermes-pr-review Phase 2) flagged the mid-turn pre-API
compaction for reuse/duplication (W1/W2); fixing that surfaced a
regression against a deliberate existing feature, now also fixed.

The block now mirrors the turn-prologue preflight's guard chain exactly
(agent/turn_context.py) instead of a hand-rolled pressure limit:
  1. should_defer_preflight_to_real_usage(rough) — defer when the rough
     estimate is known-noisy vs a recent real provider prompt that fit
     under threshold (schema overhead / post-compaction over-count, #36718).
  2. get_active_compression_failure_cooldown() — skip during a same-session
     compression-failure cooldown.
  3. should_compress(rough) — reuses the canonical threshold_tokens (output
     room already reserved by _compute_threshold_tokens) plus its summary-LLM
     cooldown + anti-thrash guards (#11529).

Dropped the seven inline _reserve/_output_pressure locals (W2: they
re-derived _compute_threshold_tokens and omitted its 85% degenerate-window
fallback). compression_attempts stays as the hard per-turn backstop.

Without guard (1) the block fired a compaction the preflight deliberately
defers, breaking test_413_compression::test_preflight_defers_when_recent_
real_usage_fit (ValueError from the mocked _compress_context). Verified:
test_413_compression 26/26, codex 82/82, and anti-thrash engaged
(_ineffective_compression_count=2) still suppresses the block (0 calls).
2026-07-04 14:15:49 +05:30
kshitijk4poor 475dd97263 fix: re-baseline flush cursor after mid-turn pre-API compaction
The salvaged max-output/pressure fix set conversation_history=None after
the new pre-API compaction. That is only correct for legacy session-
rotation. Under the default in-place compaction (compression.in_place:
True), archive_and_compact inserts the compacted rows into the session DB
directly without stamping them with the intrinsic persisted-marker, so a
subsequent flush with conversation_history=None re-appends them — doubling
the active context and retriggering compression (the early-persist
duplicate-row trap).

Use conversation_history_after_compression(agent, messages), matching the
two existing compaction sites (post-response should_compress and the
turn-prologue preflight), which returns None for rotation and
list(messages) for in-place so the compacted dicts are skipped by identity.

Adds a regression test with a real SessionDB + real archive_and_compact
that asserts the compacted summary row is persisted exactly once (fails
with 2 copies on the None variant).
2026-07-04 13:55:26 +05:30
eliteworkstation94-ai 1f430e1aa2 fix: recover Codex max-output truncation 2026-07-04 13:55:26 +05:30
Teknium 372f8195c7
fix(moa): default temperatures to unset — provider default, like single-model agents (#57440)
A single-model Hermes agent never sends temperature; the provider default
applies. MoA hardcoded reference_temperature=0.6 / aggregator_temperature=0.4,
and the coercion float(preset.get(key, 0.6) or 0.6) made unset IMPOSSIBLE to
express: absent, null, empty, and even an explicit 0 all collapsed to the
baked-in default. Every MoA advisor and aggregator therefore ran at 0.6/0.4
while the same model running solo used the provider default — silently
skewing solo-vs-MoA comparisons and overriding provider-tuned defaults.

- moa_config normalization: temperatures coerce to None when absent/blank/
  invalid (new _coerce_float_or_none); explicit values incl. 0 honored.
- moa_loop: _preset_temperature() resolves preset values; None flows to
  call_llm, which already omits the parameter when None (same contract as
  max_tokens). Aggregator still inherits the acting agent's own configured
  temperature when the preset doesn't pin one.
- conversation_loop (context-mode MoA): same resolution, no more hardcoded
  0.6/0.4 at the call site.
- DEFAULT_CONFIG preset + web_server payload models + docs updated: unset
  is the default, pinning stays available.
2026-07-03 00:22:49 -07:00
Teknium 543d305bbb
feat(moa): add reference_max_tokens to cap advisor output and cut turn latency (#56756)
MoA per-turn latency is dominated by advisor GENERATION: turn wall time
correlates ~0.88 with output tokens and ~-0.03 with input tokens (measured over
52 turns). Each turn waits for the slowest advisor to finish writing, and
advisors were uncapped — writing multi-thousand-token essays the aggregator
only needs the gist of.

Add an opt-in per-preset reference_max_tokens knob (mirrors reference_temperature)
that caps ADVISOR output only; the acting aggregator is never capped. Default
None = uncapped, so existing presets are byte-for-byte unchanged (no regression).
Wired through both MoA execution paths (MoAChatCompletions.create and
aggregate_moa_context).

E2E: same task, closed preset uncapped vs reference_max_tokens=600 -> 59s to 33s
(~44% faster), final answer identical/correct.

- hermes_cli/moa_config.py: _coerce_int_or_none helper + reference_max_tokens
  in _normalize_preset/_default_preset/flattened view
- agent/moa_loop.py: read preset.reference_max_tokens, pass to reference fan-out
- agent/conversation_loop.py: pass reference_max_tokens on the per-turn path
- tests + docs
2026-07-02 00:16:35 -07:00
Teknium aa605b66c8
fix(moa): price aggregator turn at its real model so session cost isn't advisor-only (#56394)
On the MoA path agent.model/provider are the virtual preset name (e.g.
"closed") and "moa", which have no pricing entry. estimate_usage_cost()
returned None for the aggregator turn, so the `if amount_usd is not None`
guard skipped it and the session's estimated_cost_usd reflected only the
advisor fan-out — a ~50% undercount when the aggregator does the full acting
loop (verified: $0.91 advisor-only vs $1.96 true, aggregator = 54%).

MoAChatCompletions.create() now stashes the resolved aggregator slot as
last_aggregator_slot (exposed via MoAClient); conversation_loop reads it to
price the aggregator turn at its real model/provider. cost_source flips from
'none' to 'provider_models_api'.
2026-07-01 06:02:33 -07:00
Steve Lawton c73e74386b feat(vertex): add Google Vertex AI provider for Gemini (OAuth2)
Adds Vertex AI as a first-class provider for Gemini models via Vertex's
OpenAI-compatible endpoint. Vertex authenticates with short-lived OAuth2
access tokens (service-account JSON or ADC), not a static API key — the
missing piece behind the recurring requests (#13484, #12639, #56259).

- agent/vertex_adapter.py: OAuth2 token minting + refresh-on-expiry
  (5-min margin), ADC->service-account fallback, global vs regional
  endpoint URLs. Config precedence: env var > config.yaml > default.
- plugins/model-providers/vertex/: provider profile (auth_type=vertex),
  reuses Gemini's extra_body.google.thinking_config translation.
- runtime_provider: vertex short-circuit BEFORE the credential pool so a
  credentials-file path is never mistaken for a static API key; mints a
  fresh token + computes base_url per resolve.
- run_agent + conversation_loop: _try_refresh_vertex_client_credentials()
  re-mints the token and rebuilds the client on a mid-session 401, so a
  long-lived gateway agent survives token expiry (~1h).
- auxiliary_client: vertex auth_type branch for side-LLM tasks.
- config.yaml: vertex.project_id / vertex.region (non-secret, bridged to
  env); credential path stays in .env (VERTEX_CREDENTIALS_PATH).
- setup wizard + model picker: dedicated _model_flow_vertex; curated
  google/gemini-* model list; --provider choices.
- pricing/metadata: Vertex prices off the gemini docs snapshot; endpoint
  host auto-maps to the vertex provider (no probe spam).
- lazy_deps + pyproject [vertex] extra: google-auth, opt-in only.
- docs: guides/google-vertex.md + providers page; tests for adapter +
  runtime resolution.

Salvages and modernizes #8427 by @slawt onto current main: rewired from
the legacy PROVIDER_REGISTRY path to the provider-profile architecture,
moved non-secret config out of .env into config.yaml, and added the
per-turn 401 token-refresh the original lacked.
2026-07-01 05:25:33 -07:00
Dutch Dim 154c382d65 fix(gateway): recover from truncated responses 2026-07-01 17:08:50 +05:30
Teknium 5de65624d1
fix(moa): capture streamed aggregator output into full-turn traces (#56312)
MoA full-turn traces (moa.save_traces) recorded the aggregator's acting
output only on the non-streaming path, where it's captured inline at
call time. On the streaming path — which every hermes chat --query run
and every live gateway/CLI turn takes — the aggregator's raw token
stream is handed to the live consumer, so the trace left output=null and
only pointed at the session-db assistant row. An offline audit of a
benchmark run (HermesBench drives --query) then couldn't see what the
aggregator produced without hand-joining to state.db.

Capture the resolved streamed acting text at trace-flush time (the agent
already holds it in _current_streamed_assistant_text) and fold it into
the trace, so the record is self-contained in both modes. New
output_location value inline_from_stream marks a streamed turn whose text
was captured this way; a genuinely empty acting turn (pure tool call)
still points at the session db, matching state.db exactly.

Touches only the trace side-channel — no change to the acting path,
message history, role alternation, or prompt cache.

- agent/moa_loop.py: consume_and_save_trace(..., aggregator_output_fallback)
  on both the facade and the MoAClient wrapper; prefer inline capture,
  fall back to the resolved streamed text.
- agent/moa_trace.py: embed the fallback; add inline_from_stream location.
- agent/conversation_loop.py: pass _current_streamed_assistant_text at flush.
- tests: 5 cases across streaming / non-streaming / empty-fallback / no-double-write.
2026-07-01 04:07:46 -07:00
Tranquil-Flow 122e5bc037 fix(agent): retry 413 after stripping vision payloads (#47339)
When text compression can't reduce a 413 request further, evict base64
image parts from tool messages and retry once instead of dead-ending
with 'Payload too large and cannot compress further.'

A 413 is a request-body byte-size limit, not a token limit. browser_vision
screenshots (2-5MB base64 each) keep the HTTP body oversized even after
aggressive summarization. The strip pass passes remember_model=False so a
413 does not poison _no_list_tool_content_models — that set is for providers
that reject list-type tool content, a distinct failure mode.

Cherry-picked from #47397 by Tranquil-Flow; placed onto main's current
token-aware 413 recovery else branch.
2026-07-01 03:18:41 -07:00
Teknium 913e661a09
fix(cache): stop verification-loop synthetic nudges from persisting (#56194)
verify_on_stop / pre_verify append a synthetic assistant "done" plus a
synthetic user nudge to keep the agent going one more turn before it can
claim completion. Both were flagged (_verification_stop_synthetic on the
nudge only), but the flags were never registered in
_EPHEMERAL_SCAFFOLDING_FLAGS, so the central _is_ephemeral_scaffolding()
filter that guards both persistence sinks (SQLite flush + JSON snapshot)
let them through. The resumed transcript then inherited loop-only
scaffolding, invalidating the prompt-prefix cache on later turns.

- add _verification_stop_synthetic and _pre_verify_synthetic to
  _EPHEMERAL_SCAFFOLDING_FLAGS (the single chokepoint both sinks use)
- flag the blocked attempt assistant message too, not just the nudge, so
  the whole synthetic pair drops together and persistence does not keep a
  premature done with the nudge stripped (assistant to assistant adjacency)

The API-payload leak claimed in the report is already handled: the
chat_completions transport strips every underscore-prefixed message key
before the wire, so the marker never reaches strict providers.

Reported by patppham.
2026-07-01 02:26:06 -07:00
Omar Baradei 053424c486 fix(agent): preserve final_response on failure returns
AIAgent.run_conversation() promises a dict with final_response, but 16
terminal-failure branches returned dicts that either omitted the key or
set it to None. Callers that index result['final_response'] directly
(run_agent.py chat() + the __main__ printer) turn a real provider/context
failure into an opaque KeyError instead of surfacing the actionable error.

Every offending branch already carried usable 'error' text, so this
mirrors that text into final_response for all 16 sites (8 that omitted the
key, 8 that returned None). Adds an AST regression test that fails if any
run_conversation() dict return omits final_response or sets it to a literal
None, and tightens the invalid-response test to assert final_response == error.
2026-07-01 02:04:28 -07:00
teknium1 e00800fc89 feat(classifier): Anthropic-specific guidance for subscription exhaustion
When an Anthropic Claude Pro/Max OAuth subscription hits the "out of extra
usage" 400 (now classified as billing), surface actionable guidance pointing
at claude.ai/settings/usage and the cycle-reset option instead of the generic
"add credits with that provider" line — which does not apply to a
subscription. Folds in the UX from #40073 (@harsh-matchmyflight) without the
extra FailoverReason enum; the billing reclass already provides the recovery
behavior.
2026-07-01 01:36:34 -07:00
Teknium 2e8748ed22
feat(moa): opt-in full-turn trace persistence to JSONL (#56101)
Adds moa.save_traces (default off). When on, every MoA turn that runs the
reference fan-out appends one JSON line to
<hermes_home>/moa-traces/<session_id>.jsonl capturing the TRUE FULL turn:
each reference model's exact input messages (system advisory prompt + full
advisory view, not the truncated display preview) + full output + usage +
per-advisor cost, and the aggregator's exact input (including the injected
reference-context guidance block) + output. Lets MoA runs be audited and
improved offline — what every model saw, said, and cost.

- agent/moa_trace.py: config-gated JSONL writer, profile-aware path via
  get_hermes_home(), best-effort (never breaks a turn), moa.trace_dir override.
- agent/moa_loop.py: _RefAccounting now carries full input/output/model/
  provider/temperature; create() stashes the full turn on a cache MISS
  (once per turn, never on the cache-HIT repeat iterations); non-streaming
  aggregator output captured inline, streaming marked + pointed at the
  session assistant message. consume_and_save_trace(session_id) flushes it.
- agent/conversation_loop.py: flushes the trace with the live session_id
  right after MoA usage consumption. No-op for non-MoA clients.
- hermes_cli/config.py: moa.save_traces + moa.trace_dir defaults.

Traces are a side channel — NOT the messages table, never in replay, safe
to delete. Off by default; only overhead when off is one config read on a
MoA cache-MISS turn.

Tests: full-trace-when-enabled (per-ref input+output+cost, aggregator
input-with-guidance + output), nothing-when-disabled. Live E2E through
run_conversation confirmed the loop wiring writes the file.
2026-07-01 00:09:42 -07:00
Teknium 3bdb23de10
fix(moa): count reference (advisor) fan-out token usage + cost (#56087)
MoA ran the reference models before the aggregator but returned only the
aggregator's usage to the loop — _run_reference discarded each advisor
response's .usage entirely. Session accounting (state.db, /insights, cost)
therefore undercounted every MoA turn by the whole reference fan-out, which
is usually the bulk of the spend and scales with advisor count.

- _run_reference normalizes each advisor's usage with ITS OWN resolved
  provider/api_mode and prices it at ITS OWN model rate (correct cache-read/
  cache-write split), returning a _RefAccounting(usage, cost).
- create() sums advisor usage + cost once per turn (cache MISS only, so a
  repeat tool-iteration reusing cached advice does not double-charge) and
  exposes it via MoAClient.consume_reference_usage().
- conversation_loop folds advisor tokens into the reported/persisted token
  counts and adds advisor cost (priced per-advisor) on top of the
  aggregator cost, in both the in-memory session totals and the state.db
  per-call delta. Aggregator cost is still priced on aggregator-only usage
  so advisor tokens are never repriced at the aggregator rate.
- CanonicalUsage gains __add__ for per-bucket summing.

Tests: advisor usage/cost capture, per-turn sum + consume-clears +
cache-hit no-double-charge, CanonicalUsage.__add__.
2026-06-30 23:08:37 -07:00
Jeff Watts 4d2351a528 feat(moa): stream the aggregator response to the user
MoA sessions could not stream: the gateway streaming toggle was a no-op for
provider "moa", so users saw nothing until the entire response finished — minutes
of silence on long turns. The aggregator's reply was always fetched whole.

Root cause was twofold:
  1. conversation_loop hard-disabled streaming for provider in {"copilot-acp",
     "moa"} (MoA grouped with the ACP client, whose facade isn't a stream).
  2. MoAChatCompletions.create() fetched the aggregator response whole via
     call_llm(), which had no streaming mode.

For provider "moa", _create_request_openai_client() returns the MoAClient facade
itself, so the existing streaming consumer already calls
MoAChatCompletions.create(stream=True). We reuse that battle-tested consumer
(text-delta delivery, tool_call reassembly, stale-stream detection, non-streaming
fallback) instead of adding a parallel streaming path.

Changes:
  - call_llm() gains stream/stream_options. When streaming it returns the raw SDK
    stream iterator directly, bypassing _validate_llm_response and the
    temperature/max_tokens/payment fallback chain (which assume a complete
    response). The caller owns reassembly and fallback.
  - MoAChatCompletions.create() runs the references first (unchanged), then when
    stream=True returns the aggregator's raw stream, forwarding stream_options and
    the consumer's per-request read timeout. stream=False is byte-identical to
    before (no stream/stream_options/timeout forwarded).
  - conversation_loop streams MoA only when a display/TTS consumer is present;
    quiet/subagent/health-check paths keep the complete-response path.

Tests: tests/run_agent/test_moa_streaming.py — create() stream/non-stream
branches, stream_options + timeout forwarding, call_llm raw-stream return vs
validated non-stream. Existing MoA tests unchanged (20 passed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 12:07:01 -07:00
Zane Ding ac380050ea fix(credential-pool): distinguish OpenRouter upstream 429s from account 429s
OpenRouter returns 429 in two shapes: an account-level throttle on the
user's key, and an upstream-provider throttle (DeepSeek/Anthropic/etc.
rate-limiting OpenRouter's aggregate traffic). The classifier treated
both identically and rotated/exhausted OPENROUTER_API_KEY on every 429 —
burning the key for ~24min and silently disabling auxiliary features
(compression, summarization, vision) on an upstream throttle where the
key was healthy.

Add a FailoverReason.upstream_rate_limit classified from OpenRouter's
unambiguous wrapper message "Provider returned error" (the same signal
the metadata-raw parser already trusts). Recovery skips credential
rotation and defers to the fallback chain to switch models instead.

Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>
2026-06-30 03:57:14 -07:00
Teknium c7e0bdef9a
fix(agent): stop over-cap max_tokens 400s from death-looping into compression (#55570)
An over-cap model.max_tokens produces a provider 400 that mentions
max_tokens, which trips _CONTEXT_OVERFLOW_PATTERNS and is classified as
context_overflow. On providers whose wording isn't recognized by
parse_available_output_tokens_from_error() (e.g. DashScope/Qwen:
"Range of max_tokens should be [1, 65536]") the smart-retry is skipped
and the error falls into the compression fallback, which re-sends the
same oversized max_tokens, fails identically, and loops until
"cannot compress further" on a tiny conversation (#55546).

Root-cause fix for the whole class, not just DashScope:
- parse_available_output_tokens_from_error(): recognize the DashScope
  "Range of max_tokens should be [1, N]" form and return N (smart-retry
  then caps output and retries WITHOUT compressing).
- new is_output_cap_error(): broader yes/no gate for output-cap 400s.
  In the loop, when the error is output-cap-shaped but unparseable, fail
  fast with an actionable message (lower model.max_tokens) instead of
  routing into compression. Mirrors the existing GPT-5 max_tokens guard.

Real input overflows and GPT-5 unsupported-param 400s are unchanged.
2026-06-30 03:26:41 -07:00
Brooklyn Nicholson a10113658b feat(agent): add pre_verify hook and verify-on-stop coding guidance
Add a `pre_verify` user/plugin/shell hook fired once per turn when the agent
edited code and is about to finish, after the existing verify-on-stop guard. A
hook can keep the agent going one more turn (run a check, defer it, tidy the
diff) by returning {"action":"continue","message":...} (the Claude-Code Stop
shape {"decision":"block","reason":...} is accepted too). Hooks receive coding,
attempt, final_response, and sorted changed_paths so they can self-scope and
self-throttle; the path is bounded by agent.max_verify_nudges and preserves
message-role alternation.

Hermes still ships its default coding guidance (agent.verify_guidance, on by
default), but it now rides the evidence-based verify-on-stop missing-evidence
nudge instead of a separate default pre_verify continuation, so it costs no
extra model turn of its own. Guidance reuses the shared utils.is_truthy_value
parser rather than a local copy.
2026-06-30 00:59:29 -05:00
teknium1 578e3989d4 fix(agent): route content-filter stream stalls to fallback chain (#32421)
When a provider's output-layer safety filter (MiniMax "output new_sensitive
(1027)", Azure content_filter, etc.) kills a streaming response after deltas
were already sent, interruptible_streaming_api_call swallows the raw error
into a finish_reason=length partial-stream stub. The conversation loop then
burned 3 continuation retries against the SAME primary — re-hitting the
content-deterministic filter every time — and gave up with "Response remained
truncated after 3 continuation attempts", never consulting fallback_providers.

Builds on @595650661's classifier change (cherry-picked) so error_classifier
recognizes the filter; then:
- chat_completion_helpers: run the swallowed error through error_classifier at
  the stub-creation point and stamp _content_filter_terminated on the stub
  (single source of truth — no parallel pattern list).
- conversation_loop: read the tag and activate the fallback chain BEFORE
  burning any continuation retries; roll partial content back to the last
  clean turn and re-issue against the new provider (restart_with_rebuilt_messages).
  Plain network stalls are unaffected (only content_policy_blocked is tagged).

Credits #32479 (@sweetcornna) and #33845 (@Tranquil-Flow) which fixed the
same issue via the stub-tag and loop-escalation approaches respectively.

Live E2E confirmed: before, _try_activate_fallback called 0x; after, fallback
fires on the first stub and the fallback provider completes the turn.
2026-06-28 01:15:21 -07:00
sweetcornna 2701ea2f0c fix(agent): reopen fallback chain after primary recovery 2026-06-28 00:57:42 -07:00
infinitycrew39 e860a40e14 fix(agent,gateway): surface partial-stream recovery and bound detached restart
Salvage of NousResearch/hermes-agent#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
2026-06-27 22:03:14 -07:00
Teknium d43e0cf304
fix(agent): config-driven intent-ack continuation for all api_modes (#27881) (#53943)
* fix(agent): config-driven intent-ack continuation for all api_modes (#27881)

The agent could end a turn after only stating intent ('I will run a health
check...') without executing the announced tool call, forcing the user to
re-prompt. A continuation guard that catches this and nudges the model to
proceed already existed but was hard-gated to the codex_responses api_mode,
so Gemini/Claude/OpenRouter turns never benefited.

- New agent.intent_ack_continuation config (default 'auto' = codex-only,
  byte-stable for existing conversations). 'true'/model-list opts every
  api_mode in; 'false' disables. Mirrors agent.tool_use_enforcement's shape.
- looks_like_codex_intermediate_ack gains require_workspace (default True).
  The opted-in path drops the codebase/filesystem requirement so general
  autonomous workflows (server ops, deploys, API calls) are caught, not just
  coding tasks. Future-ack + action-verb + short-content + no-prior-tool
  guards still apply; the 2-nudge-per-turn cap is unchanged.
- Resolution centralized in intent_ack_continuation_mode (off/codex_only/all).

* docs(infographic): intent-ack continuation (#27881)
2026-06-27 20:46:00 -07:00
konsisumer 1ab35ba25d fix(anthropic): stop SDK auto-retry double-firing and raise Retry-After cap to 600s
The Anthropic SDK clients were built without max_retries, so the SDK
default (max_retries=2) retried 429/5xx with its own backoff that ignores
Retry-After — double-retrying inside hermes's outer loop and burning
request slots against a bucket that won't refill for minutes. Set
max_retries=0 on all Anthropic/AnthropicBedrock client constructions so
the outer conversation loop (which already honors Retry-After) owns retry.

Also raise the Retry-After cap in the conversation loop from 120s to 600s.
Anthropic Tier 1 input-token buckets reset in ~171s, so the 120s cap made
hermes retry before the reset window and re-trip the limit.

Refs #26293
2026-06-27 19:23:15 -07:00
LeonSGP43 32732a8f83 fix(agent): cap same-entry credential refreshes so fallback can activate (#26080)
A persistent upstream 401 on a single-entry OAuth pool (common for Claude
Max subscribers) made the credential-pool recovery spin forever:
try_refresh_current() re-mints a fresh token and reports success on every
401, so recover_with_credential_pool returned True and the retry loop
continue'd without ever incrementing retry_count or reaching the
auth-failover block. The configured fallback_model never activated and the
agent appeared to hang.

Cap consecutive successful same-entry refreshes (keyed by provider +
pool-entry id) at 2; once exceeded, treat the credential as unrecoverable
and return not-recovered so the loop falls through to
_try_activate_fallback. The 429/billing paths already rotate-or-fall-through
correctly (mark_exhausted_and_rotate returns None on a single entry), so
only the auth-refresh branch needed the cap.

Co-authored-by: Hermes Agent <hermes@nousresearch.com>
2026-06-27 19:20:07 -07:00
linyubin c946e6709f fix(agent): activate fallback on persistent transport failures (#22277)
Eager fallback previously fired only on rate_limit/billing. A stale-
detector-killed hung stream classifies as FailoverReason.timeout
(retryable=True) and the retry loop re-hit the same dead primary until
the budget exhausted -- 3 x ~180-300s stale kills compounding into a
15+ min silent hang while the configured fallback chain sat idle.

Extend the existing eager-fallback gate to also cover timeout and
overloaded, but only after one real retry (retry_count >= 2) so genuine
transient hiccups still recover on the primary. Reuses the same
pool-recovery guard and state-reset as the rate_limit branch -- no new
config flag, no change to the rate-limit intent.

Salvaged from PR #50228 by @linyubin. Closes #22277.

Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>
2026-06-27 19:12:21 -07:00
Teknium 1a570dae00
fix(image-routing): unblock message queue on OpenRouter 'no endpoints' image 404 (#53901)
The agent's image-rejection fallback strips images and retries text-only when
a provider rejects image content, which is what lets the gateway drain its
queued messages. The fallback only fires on a hardcoded phrase list, and the
OpenRouter wording — HTTP 404 'No endpoints found that support image input' —
was missing. For OpenRouter-routed non-vision models the fallback never fired,
the retry loop re-sent the same rejected request until exhaustion, and every
subsequent message (including plain text) stayed queued behind the stuck turn.

Add the phrase to _IMAGE_REJECTION_PHRASES (the 404 already passes the 4xx
gate). Add a positive test and a guard test so the sibling OpenRouter
'no endpoints ... data policy / guardrail' 404s do NOT get their images
stripped.

Fixes #21160. Reported by @liu14goal14-ux; PR #21198 by @ygd58.
2026-06-27 19:07:02 -07:00
Teknium 02b32e2d7c
fix(moa): call reference + aggregator models through their provider's real route (#53580)
MoA was calling reference and aggregator models through a bare
call_llm(provider=slot["provider"], model=slot["model"]) with a forced
temperature and a forced max_tokens (the preset's hardcoded 4096). That left
base_url/api_key/api_mode unresolved — so the auxiliary auto-detector guessed
the API surface instead of using the provider's real runtime, and the 4096 cap
truncated long aggregator syntheses.

A MoA slot is just a model selection and must be called the same way any model
is called elsewhere. Each slot is now resolved through resolve_runtime_provider
(the canonical provider→api_mode/base_url/api_key resolver the CLI, gateway, and
delegate_task all use) via a new _slot_runtime() helper, and the resolved
endpoint is passed into call_llm. So a reference/aggregator gets its provider's
actual API surface — MiniMax → anthropic_messages, GPT-5/o-series →
max_completion_tokens, custom endpoints → their base_url — identical to how that
model is handled as the acting model.

MoA also no longer imposes its own output cap: max_tokens defaults to None
(omitted → the model's real maximum) for references and is passed through from
the caller for the aggregator. The preset's hardcoded 4096 is gone. The
max_tokens preset config field is left in place (config/web/desktop unchanged);
it is simply no longer applied as a forced cap.

Tests: slots route through resolve_runtime_provider with resolved base_url/
api_key; resolution errors fall back to bare provider/model; neither call
carries an output cap even when the preset config still contains max_tokens.
2026-06-27 04:39:42 -07:00
diamondeyesfox 8df231c941 fix(agent): rebaseline in-place compression flushes 2026-06-27 03:04:26 -07:00
Teknium 217047de2d
fix(agent): silence verification-stop loop status line (#53223)
The verify-on-stop guard (#52296) printed '↻ Verification required before
finishing' to the terminal on every internal nudge turn, adding noise to
CLI/gateway sessions whenever code was edited without fresh passing checks.
Demote the user-facing status emit to a logger.debug breadcrumb — the loop
still nudges the model to verify before finishing, just silently.
2026-06-26 11:52:11 -07:00
Brooklyn Nicholson 8233598e64 fix(interrupt): keep partial streamed reply when stopped mid-response
Stopping a turn while the model is streaming (stop/esc to redirect) raised
InterruptedError, set final_response to the throwaway "waiting for model
response" sentinel, and persisted messages WITHOUT the assistant text that
was already streamed to the screen. The next turn then had no record of the
half-finished reply, so the model appeared to "forget" what it just said.

Recover the on-screen text from _current_streamed_assistant_text in the
InterruptedError branch and append it as the assistant turn (and surface it
as final_response). The metadata sentinel is kept only when nothing was
streamed yet, preserving the ACP/client suppression behavior.

Completes the partial-stream recovery from 397eae5d9 (which wired the same
_current_streamed_assistant_text salvage into the connection-failure twin
but missed the user-interrupt path). The lossy handler dates to c98ee9852.
2026-06-25 23:54:20 -05:00
DavidMetcalfe 865a09a610 fix(agent): detect thinking-timeout for reasoning models and surface actionable guidance instead of misleading file-write advice
Two-part fix:

Part 1 (classifier override at agent/error_classifier.py:720-738):
A transport disconnect on a reasoning model — even on a large session —
now routes to FailoverReason.timeout instead of context_overflow. Without
this, large-session reasoning-model disconnects route to the compression
branch and silently delete conversation history on a phantom
context-length error. The override is strictly targeted: non-reasoning
models (gpt-4o, claude-3-5-sonnet, llama-3.3-70b, etc.) still route to
context_overflow on large sessions — the existing intentional behavior
for chat models whose proxy doesn't idle-kill during prefill/generation.

Part 2 (new agent/thinking_timeout_guidance.py + integration at
agent/conversation_loop.py:3488-3567):
New is_thinking_timeout() and build_thinking_timeout_guidance() helpers.
When a known reasoning model (NVIDIA Nemotron 3 Ultra, OpenAI o1/o3,
Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning)
hits a transport-kill on a small session (classifier says timeout
directly) or after Part 1 routes correctly (large session), the user
now sees reasoning-specific guidance with three actionable workarounds
in priority order:

  1. Set providers.<provider>.models.<model>.stale_timeout_seconds: 900
     in ~/.hermes/config.yaml (Hermes's built-in floor is already 600s
     for known reasoning models; raise further if upstream is even
     tighter).
  2. Lower reasoning_budget or set reasoning_effort: medium on this
     model if the provider supports it.
  3. Use a smaller / faster reasoning model if the task doesn't
     require deep thinking.

The new guidance takes precedence via if/elif over the existing
_is_stream_drop block, so a reasoning-model user with a transport-kill
message sees actionable advice instead of the misleading "try
execute_code with Python's open() for large files" advice (which is
correct for the unrelated large-file-write stream-drop case but
actively wrong for the thinking-timeout case).

Verified:
- 478 tests passing across 9 directly-relevant files (49 new + 429
  existing, zero regressions).
- Ruff lint clean on all 4 modified/new files.
- Negative test: 6 parametrized regression guards confirm non-reasoning
  models still route to context_overflow on large sessions; 4
  parametrized gates confirm non-timeout classifier reasons never
  trigger the guidance; 5 parametrized cases confirm non-transport
  messages never trigger it.
- Regression guard: new guidance message does NOT contain
  "execute_code" or "open()" — the misleading advice is fully
  replaced, not appended alongside.
- Cross-vendor dual review via agy -p:
  - Gemini 3.5 Flash (Medium) — passed: true, zero blockers, one
    SHOULD-FIX (vprint block duplication — fixed by extracting
    detection into a helper module).
  - GPT-OSS 120B (Medium) — passed: true, zero blockers, two nits
    (test placement — adopted at tests/agent/test_thinking_timeout_guidance.py;
    primary-model capture — accepted as non-issue per Flash's nit).

Dependency note for maintainers:
This PR includes agent/reasoning_timeouts.py (the reasoning-model
allowlist module from PR #52238) because the Layer 1 override is
load-bearing on get_reasoning_stale_timeout_floor(). After PR #52238
lands on main, this PR's duplicate agent/reasoning_timeouts.py should
be rebased away. Either PR can land first; the other rebase is
mechanical.

Fixes #52271.
2026-06-25 19:00:48 -07:00
Teknium c6575df927
feat(moa): expose MoA presets as selectable virtual models (#46081)
* feat(moa): expose MoA presets as selectable virtual models

Reconstructed onto current main (PR #46081's base had diverged with no common
ancestor, marking the PR dirty so CI never dispatched). MoA is now a virtual
provider: each named preset is a selectable model under provider 'moa', and the
preset's aggregator is the acting model that answers and calls tools.

Reference models fan out in parallel via a bounded ThreadPoolExecutor (the same
batch pattern delegate_task uses) — all references dispatched at once, collected
when every one finishes, then handed to the aggregator. Output order is
preserved, failures and the MoA-recursion guard stay isolated per reference.

- Removed the old mixture_of_agents model tool and moa toolset.
- Added moa as a virtual provider in the provider/model inventory.
- /moa is shortcut behavior over model selection (default preset / named preset
  / one-shot prompt).
- Dashboard + Desktop manage named presets; presets appear in model pickers.
- Parallel reference fan-out in agent/moa_loop.py with regression test.

* fix(moa): thread moa_config through _run_agent to _run_agent_inner

The reconstructed gateway MoA wiring declared moa_config on _run_agent (the
profile-scoping wrapper) and used it inside _run_agent_inner, but the wrapper
never forwarded it — _run_agent_inner had no such parameter, so the runtime hit
NameError: name 'moa_config' is not defined on the compression-failure session
sync path. Add moa_config to _run_agent_inner's signature and forward it from
both wrapper call sites (multiplex and non-multiplex). Caught by
tests/gateway/test_compression_failure_session_sync.py on CI shard test(4).

* fix(moa): classify moa as a virtual provider in the catalog

The moa virtual provider has no PROVIDER_REGISTRY/ProviderProfile entry, so
provider_catalog() fell through to the default auth_type="api_key" with no
env vars — tripping two catalog invariants:
  - test_provider_catalog: api_key providers must expose a credential env var
  - test_provider_parity: every hermes-model provider must be desktop-configurable

moa already declares auth_type="virtual" in HERMES_OVERLAYS; consult that
overlay as an auth_type fallback so the catalog reports moa as virtual (no real
credential, no network endpoint). Exempt virtual providers from the desktop
parity union check the same way 'custom' is exempt — derived from the catalog,
not a hardcoded slug, so future virtual providers are covered too.
2026-06-25 13:52:06 -07:00
Brooklyn Nicholson 2d286a6d00 fix(agent): close tool-call sequence on all interrupt aborts, not just finalize_turn
#48879 closed the tool-call sequence on interrupt inside finalize_turn so a
/stop after a tool no longer persists a `tool` tail that the next user message
turns into a `tool -> user` role-alternation violation (which strict providers
like Gemini/Claude react to by hallucinating a continuation and ignoring prior
context — what users see as "lost context after stop").

But the retry-wait, error-handling, and post-error retry-wait interrupt aborts
in conversation_loop return early and never reach finalize_turn, so they still
persisted and returned a raw `tool` tail. Interrupting during provider
backoff/rate-limiting (common under heavy work) hit exactly this path.

Extract the close into a shared close_interrupted_tool_sequence helper and apply
it at every interrupt abort (finalize_turn + the three early returns) so the
whole bug class is fixed, not just the one site.
2026-06-25 12:24:34 -05:00
benbenwyb 6f2b2a1f34 fix: handle named custom providers and Z.AI overload retries 2026-06-25 00:17:17 -07:00
Brooklyn Nicholson 2f1a47b90e feat(agent): require verification before finishing edits
Make verification closure the default coding behavior after landed file edits while keeping bounded retries and config/env switches for users who need to disable it.
2026-06-24 23:02:48 -05:00
konsisumer 190b01c553 fix(agent): persist tool calls before turn-end flush
Co-authored-by: kshitijk4poor <82637225+kshitijk4poor@users.noreply.github.com>
2026-06-24 02:15:57 +05:30
David Gutowsky 87b60ae49a no-mistakes(review): guard token-delta status msg on actual compression in overflow handler 2026-06-22 15:23:24 +05:30
David Gutowsky 47b6b4cf85 fix #39550: detect token-only compression success
Compression can materially reduce request size (tool-result pruning,
in-place summarization) without reducing message count. The two
compression-success checks in conversation_loop.py (413 handler and
context-overflow handler) only compared len(messages) to detect
success, missing token-only compression.

Now re-estimates tokens after compress_context() returns and treats
any >=5% reduction as a successful compression pass. Error logs
also use the post-compression token count instead of the stale
pre-compression estimate.

Fixes: #39550
2026-06-22 15:23:24 +05:30
teknium1 f22dd8a75a fix(agent): fail over to fallback provider on persistent auth failure (401/403)
When the active provider returns a 401/403 that survives its per-provider
credential-refresh attempt (revoked OAuth, blocked/expired key, or an
account pinned to a dead/staging inference endpoint), the conversation
loop now escalates to the configured fallback chain instead of dead-ending.

Before: the generic failover dispatch fired only for {rate_limit, billing};
auth/auth_permanent fell through to 'switch providers manually' advice and
never called _try_activate_fallback(). A user whose primary credential was
broken kept thrashing on the same dead credential every turn — the main
agent appeared 'stuck in fallback mode' while never actually failing over.
This also affected auxiliary tasks (compression, vision, title-gen), since
auto-resolved aux follows the main provider.

After: a persistent auth failure with a configured fallback chain switches
to the next provider (mirroring the rate-limit/billing failover path),
guarded one-shot per attempt by TurnRetryState.auth_failover_attempted.
When no fallback is configured the behavior is unchanged — it falls through
to the existing terminal handling and provider-specific troubleshooting
guidance.

Tests: test_auth_provider_failover.py — 401/403 classify as auth, the
gating condition fires only with a chain present + guard unset, the guard
blocks repeats, and non-auth (500) errors do not trigger auth failover.
2026-06-20 11:38:01 -07:00
Sancho c884ff64ea fix(agent): keep system-prompt model identity in sync across provider failover
The session-stable system prompt embeds Model:/Provider: identity lines,
but mid-turn failover (try_activate_fallback) swaps the runtime without
touching them, so a fallback model misreports itself as the primary when
asked "what model are you?".

rewrite_prompt_model_identity() rewrites the last occurrence of each line
on _cached_system_prompt when a fallback activates (and back on restore,
byte-identical so the primary's prefix cache still hits). The rewrite is
never persisted to the session DB. _sync_failover_system_message() patches
the in-flight api_messages[0] at all 8 failover sites so the current turn
ships the corrected identity. Cache-safe: the fallback's prefix cache is
cold on a model switch anyway.

Co-authored-by: Hermes Agent <noreply@nousresearch.com>
2026-06-20 10:46:01 -07:00
kshitijk4poor d0622cafab refactor(agent): reuse hoisted summary in content-policy branch
The non-retryable abort path now computes _nonretryable_summary once and
reuses it at the emit sites and the returned error field. The
content-policy-blocked return branch still recomputed the identical
value into a separate _summary local, half-honoring the 'summarize once'
intent. _summarize_api_error is a pure staticmethod and api_error is
never reassigned in this block, so _summary was provably byte-identical
to _nonretryable_summary. Reuse the hoisted value and drop the redundant
call. Behavior-preserving.
2026-06-18 15:46:47 +05:30
xxxigm b892ee2bcf fix(agent): summarize non-retryable API errors so raw HTML never leaks
When a non-retryable client error aborts the turn (e.g. a Codex/Cloudflare
HTTP 403 "managed challenge" page), the conversation loop returned the
failure dict with `error: str(api_error)` — the entire ~60KB HTML page.
Downstream consumers deliver that field verbatim: a cron job dumped a
Cloudflare challenge page to Discord, where it was split into ~31 messages.

The sibling "max retries exhausted" path already collapses such bodies via
`_summarize_api_error` (which extracts the <title> / status from HTML error
pages). This makes the non-retryable path consistent: compute the summary
once and use it for both the status emit and the returned `error`.
2026-06-18 15:46:19 +05:30
Teknium 020e59d3cf
fix(agent): dampen empty-name phantom tool-call loop (#47967) (#48109)
Weak open models (mimo, nemotron-class) that see tool-call XML/JSON sitting in
file contents or tool output get primed and emit their own structured tool
calls mimicking the payload — usually with an empty/whitespace name. Those
calls can't be fuzzy-repaired toward a real tool, so the dispatch loop returns
an error and the model retries. Before this fix, every empty-name error dumped
the full tool catalog back to the model, which fed the priming loop more names
to mimic and inflated context 3-4x across the retry budget.

A blank/whitespace-only tool name now gets a terse anti-priming error that
tells the model in-context tool-call syntax is DATA, with no catalog dump. A
genuinely-wrong-but-nonempty name (a real typo) still gets the full catalog so
the model can self-correct.

Not a sandbox/auth boundary issue: Hermes never parses tool-call text from
content into executable calls (structured tool_calls only; the lone text->call
parser is the Copilot ACP transport and it also rejects empty names). The
reporter's own debug dump confirms the injection never executed.

Behavior-contract test added: empty-name -> terse error, no catalog; nonempty
unknown -> catalog preserved. Exercised end-to-end via run_conversation against
an in-process mock provider.
2026-06-17 17:32:14 -07:00
Wolfram Ravenwolf bd7fc8fdcd feat(gateway): inject stable human-readable message timestamps
Consolidates these related Amy fork patches:
- 429830f39 feat(gateway): inject message timestamps into user messages for LLM context
- 3c3d6fac0 fix: handle both ISO string and epoch float timestamps in history replay
- 2874f7725 feat: human-friendly timestamp format with weekday and timezone name
- 3735f4c8b fix: render gateway message timestamps once
2026-06-16 15:49:59 -07:00
Brooklyn Nicholson 7d938cc5c9 fix(desktop): keep live model switch metadata truthful
A live config.set model switch already moved the next API call to the new model,
but the conversation could still restore an old sessions.system_prompt snapshot
whose Model/Provider lines named the previous runtime. That made "what model are
you?" answer from stale metadata even while inference ran on the new model.

After a live switch we now refresh the stored system prompt and append a real
system-history pivot (not a fake user turn) so the transcript itself records the
new model/provider. Restore also rejects already-stale prompt snapshots when
their Model/Provider lines disagree with the runtime, so existing bad sessions
self-heal.
2026-06-16 09:50:17 -05:00
kshitijk4poor 10bd01972b refactor(agent): share the content_policy_blocked result builder + recovery hint
The HTTP-200 refusal handler (finish_reason=content_filter) and the
exception-path handler (a provider moderation error classified as
content_policy_blocked) independently built the same terminal turn result —
the same {final_response, messages, api_calls, completed:False, failed:True,
error:'content_policy_blocked: ...'} dict — and ended their user-facing
message with the same 'Try rephrasing... hermes fallback add' trailer, copied
verbatim. The two copies could drift.

Funnel both through a shared _content_policy_blocked_result() builder and a
shared _CONTENT_POLICY_RECOVERY_HINT constant. Also collapse the HTTP-200
path's two near-identical with/without-explanation templates into one (compute
the detail fragment once) and pass reason=FailoverReason.content_policy_blocked
.value to the error hook instead of a hand-written string literal, matching the
sibling hook call.

Behavior-preserving: the provider/refusal lead-in wording stays distinct (a
provider safety filter vs the model declining are genuinely different signals),
the with-text and exception messages are byte-identical to before, and the
no-explanation case only gains a paragraph break for consistency. Surfaced by
the simplify-code reuse/quality reviewers.

The efficiency reviewer's 'redundant normalize_response' flag was deliberately
NOT applied: that branch is cold (refusal-only) and pure-CPU, and reusing the
sibling-branch normalized locals would risk a NameError on the codex_responses
path (which sets finish_reason without normalizing) — re-normalizing is the
robust choice.
2026-06-14 12:19:19 +05:30
SHL0MS bb46bf8ce4 fix(agent): surface model refusals instead of retrying them as errors
A Claude refusal (HTTP 200, stop_reason="refusal", empty content) was
laundered into a generic retry loop and surfaced as a misleading
"rate limited / invalid response" or "no content after retries" error,
burning paid attempts reproducing a deterministic refusal.

This hit two distinct paths:

- Direct Anthropic (anthropic_messages): validate_response rejected the
  empty-content refusal *before* normalize_response mapped refusal ->
  content_filter, so it fell into the invalid-response retry loop.
- Nous Portal / OpenAI-compatible (chat_completions): the portal surfaces
  a Claude refusal via message.refusal with empty content, which sailed
  past validation and died in the empty-response retry loop.

Fix (one unified content_filter dispatch for all backends):
- AnthropicTransport.validate_response: accept empty content when
  stop_reason == "refusal" so it flows to normalize_response.
- ChatCompletionsTransport.normalize_response: promote message.refusal to
  content + a content_filter finish reason.
- conversation_loop: handle finish_reason == "content_filter" - fire the
  api_request_error hook (content_policy_blocked), try a configured
  fallback once, else return a clear terminal refusal message. Never retry
  a deterministic refusal.

Supersedes #43084, which fixed only the direct-Anthropic path and could
not reach the chat_completions/portal path.

Tests: transport-level (validate_response refusal, message.refusal
promotion) + end-to-end loop (refusal surfaced, exactly one API call).

(cherry picked from commit 01f546f92c)
2026-06-14 12:10:08 +05:30
brooklyn! 4b5ba112ad
fix: shrink images to reported provider dimension limit (#45979)
Parse provider-reported image pixel ceilings so many-image Anthropic requests can recover by shrinking Retina screenshots below the stricter limit instead of retrying the same rejected payload.
2026-06-14 01:07:43 -05:00
Teknium 069bfd6545 fix(agent): keep Codex reasoning replay on Codex path 2026-06-13 14:35:00 -07:00
Teknium 8905ee6b8a fix(agent): rewind flush cursor exactly when repair compacts before the cursor
Follow-up to the #44837 clamp: a min() clamp only fixes cursor overshoot
past the new end of the list. When repair_message_sequence drops/merges
messages at indexes below the cursor, the clamp leaves the cursor pointing
past unflushed rows and the turn-end flush silently skips them.

Extract repair_message_sequence_with_cursor(): snapshot the flushed prefix
by object identity before repair, then recompute the cursor as the count
of surviving flushed messages. Falls back to the clamp when no snapshot is
available. Keeps the safety guard in _flush_messages_to_session_db.

Adds targeted tests for overshoot, before-cursor compaction, no-repair,
bare-agent, and the flush guard.
2026-06-12 16:29:01 -07:00
kyssta-exe 5d0408d9fe fix(agent): clamp flush cursor after repair_message_sequence compaction (#44837) 2026-06-12 16:29:01 -07:00
Aðalsteinn Helgason 2714fc8396 fix(agent): re-enter retry loop on genuine Nous 429 so fallback guard runs
The genuine-rate-limit branch set retry_count = max_retries before
continue, intending the top-of-loop Nous guard to handle fallback or
bail cleanly. But the loop condition is retry_count < max_retries, so
the guard never ran: no fallback activation, no clean rate-limit
message — just the generic retry-exhaustion error.

Set retry_count = max(0, max_retries - 1) so the loop body runs exactly
once more and the guard sees the breaker state recorded moments earlier.

Extracted from the #44061 bugfix rollup by @AIalliAI.
2026-06-12 12:21:29 -07:00
0xyg3n 9f95f72b98 fix(agent): strip api_messages in thinking-signature recovery so the retry actually omits thinking blocks
The thinking-signature recovery in agent/conversation_loop.py popped
reasoning_details from messages, then continued to retry. That had two
defects.

First, the strip never reached the wire payload. api_messages is built
once at the start of the turn by shallow-copying every entry in messages
(line 919 area). Each api_messages entry has its own reference to the
same reasoning_details list. When build_api_kwargs runs on every retry
iteration of the inner while-loop, it consumes api_messages, not
messages. Popping reasoning_details from messages left api_messages
untouched, so the retry's request still carried the same thinking
blocks Anthropic had just rejected. The classifier latched
thinking_sig_retry_attempted = True after the first attempt, and the
loop terminated with max_retries_exhausted on the same 400.

Second, the pop mutated the canonical message list. messages is the
same list _persist_session writes to state.db and the session
transcript, so a single recovery permanently wiped every signed
thinking block from the stored conversation. Subsequent turns reloaded
the stripped state, hit the same 400 ('invalid signature' or 'cannot
be modified', see #24107), and the agent stopped responding entirely.
Cascading compaction-ended sessions then chained off the corrupted
parent and the affected chat could not produce a response on any
future turn.

Move the strip onto api_messages, which is the API-call-time list
rebuilt into kwargs on every retry. messages is no longer touched, so
disk I/O stays clean and the recovery actually reaches the wire.

Observed against the native Anthropic Messages API on claude-opus-4-7
and claude-opus-4-8 with the interleaved-thinking-2025-05-14 beta on
hermes-agent 0.12.0 and 0.14.0. PR #24107 narrows the trigger; this
change makes the recovery do what it always claimed to do, and
prevents the destructive aftermath.

Tests cover the api_messages strip in isolation: pop on a shallow copy
does not affect the source, the canonical messages list survives the
strip, idempotency on a duplicate firing path, and a no-op when no
reasoning_details exist on the messages.

Related: #24107, #26959, #17861.
2026-06-10 12:39:44 -07:00
teknium1 55b83c3d99 refactor(agent): extract run_conversation post-loop tail into finalize_turn (god-file Phase 1)
Lift the post-loop finalization tail out of run_conversation into
agent/turn_finalizer.py:finalize_turn. Behavior-neutral; run_conversation
4204 -> 3846 LOC, conversation_loop.py 4578 -> 4220.

The region (everything after the main tool-calling while loop): budget-exhaustion
summary, trajectory save, session persist, turn diagnostics, response transforms,
result-dict assembly, steer drain, and the memory/skill review trigger. Lifted
verbatim into a synchronous single-return free function; the 12 post-loop locals
it reads are passed as keyword args and the assembled result dict is returned to
run_conversation (which returns it to the caller). All agent.* side effects fire
exactly as before.

Imports: os + _summarize_user_message_for_log at module top; logger lazy from
agent.conversation_loop (preserves the gateway... err 'agent.conversation_loop'
logger name, no import cycle).

Validation: 1609/1609 tests/run_agent/ pass; live PTY agent turn PASS.
2026-06-08 09:42:23 -07:00
teknium1 524453dab5 refactor(agent): consolidate inner-retry-loop recovery flags into TurnRetryState (god-file Phase 1b)
run_conversation's inner retry loop tracked recovery state in ~15 scattered
bare booleans (per-provider OAuth refresh guards, format-recovery guards,
restart signals). They are now fields on a single TurnRetryState dataclass the
loop mutates in place (_retry.<flag>), giving the recovery bookkeeping a named,
testable home.

Loop-control vars (retry_count, max_retries, max_compression_attempts) stay as
plain locals — they're while-mechanics, not recovery bookkeeping.

Behavior-neutral: pure local→attribute rewrite of 42 references; kwarg NAMES
preserved (e.g. has_retried_429=_retry.has_retried_429). Live simple + tool
turns OK.

Validation: tests/run_agent/ 1615 passed / 0 failed under per-file process
isolation; new test_turn_retry_state.py pins the field contract.
2026-06-07 22:42:05 -07:00
teknium1 f5bd09af4b refactor(acp): share interrupt-sentinel prefix, simplify guard
Replace the ACP-local prefix/suffix matcher + helper with a single
startswith() check against INTERRUPT_WAITING_FOR_MODEL_PREFIX, now
defined once in conversation_loop.py where the sentinel is produced.
Keeps the source of truth in one place so the guard cannot drift if
the status string changes. Net -17 LOC in server.py.

Also add lsaether to release.py AUTHOR_MAP.
2026-06-07 22:20:43 -07:00
teknium1 54870847cb refactor(agent): extract run_conversation prologue into agent/turn_context.py
Phase 1 of the god-file decomposition plan. run_conversation's ~470-line
once-per-turn setup block (stdio guarding, retry-counter resets, user-message
sanitization, todo/nudge hydration, system-prompt restore-or-build,
crash-resilience persistence, preflight compression, the pre_llm_call hook, and
external-memory prefetch) is moved verbatim into build_turn_context(), which
returns a TurnContext dataclass the loop unpacks.

Behavior-neutral move-and-name refactor: the builder mutates `agent` exactly as
the inline code did; only the locals the loop reads back are returned.

- run_conversation: 4602 -> 4217 LOC (-385)
- agent/conversation_loop.py: 4965 -> ~4580 LOC
- new agent/turn_context.py: focused, dependency-injected, unit-tested in isolation

Tests: tests/run_agent/ 1570 passed / 0 failed under per-file process isolation.
Relocation follow-ups: 413_compression mocks now patch both module references;
nudge/on_turn_start source-inspection guards point at the extracted module.
2026-06-07 22:17:35 -07:00
bmoore210 330ca4585b fix: harden gateway startup and turn persistence
Persist the inbound user turn before provider/tool execution so a crash
before run_conversation() (e.g. provider/httpx client init failure) keeps
the inbound message in the transcript. Repair stale/missing SSL_CERT_FILE
state on gateway startup, and avoid duplicate gateway fallback writes.
2026-06-07 02:15:23 -07:00
Teknium 3c8f1dee8d fix(compression): don't overwrite the -1 post-compression sentinel in preflight seed (#36718)
compress_context() sets last_prompt_tokens=-1 right after compression to
mark "no real API usage yet". The preflight display-seed used
`_preflight_tokens > (last_prompt_tokens or 0)`, and `(-1 or 0)` is -1
(truthy), so any positive rough estimate clobbered the sentinel with a
schema-inflated count — re-triggering compression on the next turn.
Treat any negative value as "no real data yet" and skip the seed.

Salvaged from #40246 as the minimal root-cause fix. The original also
added an `_awaiting_suppression_count` bounded-window state machine to
should_compress() across 3 files; left out here to keep blast radius
small — the sentinel guard alone fixes the re-fire. The suppression
window can be added separately if the usage=None-stub edge case warrants it.

Co-authored-by: davidgut1982 <davidgut1982@users.noreply.github.com>
2026-06-07 01:56:51 -07:00
kshitij d4a7bfd3aa
Merge pull request #29724 from bbednarski9/bbednarski/nmf-41B-nemoflow-plugin
feat(middleware): add adaptive middleware to hermes-agent, consumed by NeMo-Relay
2026-06-06 10:46:41 -07:00
Siddharth Balyan fcb1944b4f
feat(credits): usage-aware credits — in-session notices, /usage view, dev readout (#40011)
* feat(tui): HERMES_DEV_CREDITS live-spend dev readout (L0 tracer for usage-aware credits)

L0 of the usage-aware-credits feature: a dev-only, env-gated tracer that
exercises the real header -> CreditsState -> TUI pipe end-to-end behind
HERMES_DEV_CREDITS, de-risking the L1/L5 build before the notice policy exists.

- agent/credits_tracker.py: CreditsState + parse_credits_headers (headers are
  strings -> paid_access via == "true", never bool(); retain-last-known; only
  subscription_micros may be negative; *_usd kept verbatim).
- run_agent.py: _capture_credits / get_credits_state / get_credits_spent_micros,
  session-start baseline latch, + dev-gated "credits" capture log.
- agent/chat_completion_helpers.py: capture on the streaming response.
- agent/agent_init.py: init _credits_state + _credits_session_start_micros.
- tui_gateway/server.py: _get_usage emits dev_credits_spent_micros only when flagged.
- ui-tui appChrome.tsx / types.ts: cents delta status segment + "(dev credits)" banner.

Off by default; silent for normal users. Validated live against staging
(capture log delta matches the TUI segment). Throwaway consumer (readout/log/
banner); credits_tracker + the capture plumbing are the real feature foundation.

* test(credits): lock parser under 9-state matrix + harden validation (L2)

Add tests/agent/test_credits_tracker.py with 92 tests covering the 9-state
matrix (healthy, sub_90pct, grant_exhausted, purchased_only, tool_pool_free,
depleted, debt, missing, no_org) plus validation edge cases: version strict==1
with warn-once latch for v>1, bool-string trap (paid_access/tool_pool_gated_off
== "true"/"false", never bool()), half-pair subscription limit treated as
both-absent while parse succeeds, USD regex ^-?\d+\.\d{2}$, non-int micros
→ None, negative non-subscription micros → None, as_of_ms junk → None, zero
limit ZeroDivision guard.

Harden agent/credits_tracker.py to match the spec:
- Add tool_pool_micros/tool_pool_gated_off/from_header fields to CreditsState
- Add depleted property (== not paid_access, never remaining==0)
- Change used_fraction guard to key off subscription_limit_micros (the actual
  denominator) not denominator_kind (metadata)
- Replace fail-soft _safe_int with a sentinel-returning variant; full validation
  now returns None on any malformed field rather than silently defaulting
- Add module-level warn-once latch for version > 1
- Add USD regex validation; add denominator_kind allow-list check
- Parse x-nous-tool-pool-* prefix headers (not x-nous-credits-tool-pool-*)

* feat(credits): notice spine — AgentNotice + notice_callback/notice_clear_callback + TUI binding (L1)

L1 of usage-aware credits: the driver-agnostic notice delivery spine that L4's
policy will fire through and L5's TUI render will consume.

- agent/credits_tracker.py: AgentNotice dataclass (text/level/kind/ttl_ms/key/id;
  kind defaults "sticky", kept TTL-expressive for a future config seam).
- run_agent.py: AIAgent gains notice_callback + notice_clear_callback slots and
  _emit_notice / _emit_notice_clear emitters (swallow all callback errors — a
  notice must never break the agent loop; no-op when unbound).
- agent/agent_init.py: thread both callbacks through init_agent.
- tui_gateway/server.py: bind both in _agent_cbs → notification.show / notification.clear
  WS events (snake_case payload, matching the existing gateway-event convention).
- ui-tui/src/gatewayTypes.ts: notification.show / notification.clear arms on GatewayEvent.
- tests/run_agent/test_notice_spine.py: 15 tests (emitter fire + fail-open + no-op,
  signature threading, TUI binding payload shape).

Messaging push is out of v1 (binds neither callback). CLI binding + the TUI render/
decode land with L4 (firing) and L5 (render) so turn-end flush is wired correctly.

* feat(credits): threshold reconciliation policy + tests (L4.1)

* feat(credits): wire threshold policy into capture + latch (L4.2)

After a fresh header parse, _capture_credits runs evaluate_credits_notices against
the agent's _credits_latch and emits the result — clears first, then shows (so a
recovered depletion clears before the "restored" success lands, and depleted wins
the latest-wins slot). Gated on a bound notice_callback: messaging (no callbacks)
still caches state for /usage but runs no policy. Parse stays fail-open (miss →
keep last-known); the eval/emit path warns on failure rather than swallowing, so a
depletion-notice bug can't vanish silently.

- run_agent.py: _capture_credits split into parse (swallow→miss) + policy (warn);
  latch lazy-guarded (object.__new__ safety).
- agent/agent_init.py: init agent._credits_latch = {"active": set(), "seen_below_90": False}.

* feat(tui): render credits notices in the status bar (L5, Strategy B)

The TUI now renders the notification.show / notification.clear gateway events the
agent emits — a level-colored notice overrides the status/verb slot when not busy.

- Notice state machine on turnController (pendingNotice + dedicated noticeTimer +
  show/clear/applyNotice/flushPendingNotice/clearNoticeState). createGatewayEventHandler
  decodes the events and delegates.
- Render priority busy > notice > status (appChrome StatusRule); notice text rendered
  verbatim (its glyph comes from the policy), shrinkable so it never clips model│ctx;
  dev-credits banner + Δ segment preserved. UiState.notice is snake_case (matches wire).
- Busy-wins: a notice arriving mid-turn is held and flushed at the THREE turn-end sites
  (recordMessageComplete / interruptTurn / recordError) — never idle(), which reset()
  also calls (would leak across sessions); reset() clears instead.
- Dedicated noticeTimer (never statusTimer); TTL starts on visibility with an id-guard;
  latest-wins cancels the prior timer; clear is key-matched (no-op on mismatch); a sticky
  survives a turn (flush no-ops with no pending); session reset clears (no cross-session leak).
- 20 tests (handler/turnController logic incl. R3-C2 timer isolation + render priority).

* feat(credits): cold-start seed for new Nous sessions (L3)

A genuinely-new Nous session has no inference header yet, so seed credits state from
the authoritative GET /api/oauth/account snapshot at session start (in the new-session
branch of _restore_or_build_system_prompt — inline, since the on_session_start plugin
hook gets no agent reference). The seed runs the shared notice policy, so a session that
opens already depleted warns IMMEDIATELY rather than only after the first turn.

- Maps the nested account fields (paid_service_access → paid_access; total_usable /
  subscription / purchased on paid_service_access_info; rollover on subscription), each
  None-guarded; float dollars → micros via round(d*1e6), *_usd left "" (render formats
  from micros — never synthesize a verbatim usd from a float).
- Magnitudes-only: no monthlyCredits on the endpoint → subscription_limit_* unset →
  used_fraction None → no warn90 from the seed (% only once a header lands, per D-E).
- Provider-guarded to Nous; fail-open (any error leaves _credits_state None, never
  blocks startup); paid_access unknown ⇒ True (never falsely depleted).
- run_agent.py: extracted the warm-path policy/emit block into a shared
  _emit_credits_notices() so capture and the seed fire notices identically.

* feat(credits): /usage Nous credits magnitudes view + recovery trigger (L6)

Add Nous credit dollar magnitudes to /usage (subscription / top-up / total
+ rollover + renewal + portal CTA), magnitudes-only per v1 (no % until the
account endpoint exposes a denominator). Reuses the existing account-usage
render machinery via a new pure build_nous_credits_snapshot() that maps a
NousPortalAccountInfo to an AccountUsageSnapshot; no nous branch is added to
fetch_account_usage (keeps the per-provider boundary intact).

CLI /usage also doubles as a depletion-recovery trigger: a force_fresh
account fetch, kept in a SEPARATE local so it never clobbers the
header-sourced agent._credits_state (which alone carries used_fraction). If
paid access recovered while credits.depleted is latched and a notice
consumer is bound, it reuses agent._emit_credits_notices() to clear it.
Gateway /usage displays magnitudes only — messaging binds no notice
consumer, so it performs no recovery emit.

Fail-open throughout: any portal hiccup leaves /usage unaffected.

* refactor(credits): dedupe HERMES_DEV_CREDITS flag parse via shared helpers

The dev-flag truthy check was inlined in three places. Replace with the shared
utils.is_truthy_value (run_agent.py, tui_gateway/server.py — also drops a
redundant inline `import os`) and a hoisted DEV_CREDITS_MODE export in
ui-tui/src/config/env.ts (consumed by appChrome, which also stops recomputing the
env check on every render). Behaviour-preserving; identical truthy set.

* fix(credits): cut dead /usage recovery trigger + bound portal fetches (L6 review)

Adversarial review found the /usage depletion-recovery trigger dead AND broken:
the CLI binds no notice_clear_callback, the TUI runs /usage in a separate
slash-worker subprocess (its own agent/latch), and the no-clobber rule made it
evaluate stale paid_access anyway. Recovery already happens on the next inference
(warm path), so the trigger was redundant — remove it and stop the depleted
notice over-promising.

- cli.py: remove the dead recovery block; bound the /usage portal fetch with a
  10s wall-clock timeout (ThreadPoolExecutor) like the per-provider fetch —
  urllib's per-socket timeout is not a wall-clock guarantee.
- agent/credits_tracker.py: reword the depleted CTA to "run /usage for balance"
  (no false recovery promise; /usage shows fresh magnitudes, sticky clears next turn).
- agent/conversation_loop.py: same wall-clock timeout on the cold-start seed fetch
  so a stalled portal can't hang session startup; tidy its time import.

* chore(credits): dev notice-state fixtures (HERMES_DEV_CREDITS_FIXTURE)

Throwaway dev scaffolding to exercise the notice pipeline without real spend or
Redis seeding. Set HERMES_DEV_CREDITS_FIXTURE to a state name (healthy / sub_90pct
/ grant_exhausted / depleted / clear) or a file path whose contents name a state
(re-read each turn → flip states live for recovery testing). _capture_credits
injects the chosen CreditsState instead of parsing real headers and runs the
shared notice policy. Deletable with the rest of the HERMES_DEV_CREDITS scaffolding.

* feat(credits): /usage monthly-grant % gauge

The portal /api/oauth/account subscription block now carries monthly_credits
(the per-period grant allowance, the % denominator). The consumer parsed
monthly_charge but dropped monthly_credits, so /usage stayed magnitudes-only.

Capture monthly_credits into NousPortalSubscriptionInfo + _subscription_from_payload.
build_nous_credits_snapshot emits a Subscription usage window (real % used, routed
through the existing render machinery) when monthly_credits is a finite positive
denominator and credits_remaining is finite and <= cap; otherwise it degrades to
magnitudes-only (older portals, rollover-over-cap, or non-finite payloads).

Guards (adversarial-review-driven): reject non-finite operands (json.loads parses
bare NaN/Infinity by default → would render $nan + a false 100% used), reject
bools, guard div-by-zero (cap>0), and suppress the gauge when remaining > cap
(rollover spanning the period makes the cap a nonsensical denominator → the
$X-of-$Y detail would read as a contradiction). Debt (remaining<0) clamps to 100%.

Money rule preserved: the ratio + magnitudes are computed from numeric float
account fields via display formatting, never by parsing a server *_usd string
(there are none on these dataclasses).

13 gauge tests added (tests/agent/test_nous_credits_gauge.py).

* fix(credits): show /usage Nous block whenever a Nous account is present

/usage runs in a slash-worker subprocess whose resolved inference provider is
often not "nous" even when the user has a Nous account, so gating the Nous
credits block on (provider == "nous") hid it entirely — the account data was
fully available but never rendered.

Gate instead on "a Nous account is logged in": a cheap local auth-state lookup
(get_provider_auth_state('nous') has an access_token) decides whether to attempt
the portal fetch, regardless of which provider inference runs on. In the gateway
the block is also lifted out of the 'if provider:' scope so a Nous-credentialled
user with another (or no) resident inference provider still sees their balance.
Fail-open and the per-fetch wall-clock timeout are preserved.

* fix(credits): show /usage Nous block when there's no live agent (TUI slash-worker)

In the TUI, /usage runs in a slash-worker subprocess that resumes the session
WITHOUT building an agent (self.agent is None), so _show_usage early-returned
"(._.) No active agent" before ever reaching the Nous credits block — which is
agent-independent (a portal fetch gated on Nous auth-state). Extract the block
into _print_nous_credits_block() and run it at the no-agent / no-calls
early-returns too (returns True if it printed, so the fallback message only
shows when there's genuinely nothing).

Verified live against staging: the block + monthly-grant gauge now render in the
slash-worker /usage path (previously hidden). The plain CLI REPL + messaging
paths are unchanged (they have a live agent).

* feat(credits): escalating 50/75/90 usage bands (single status line)

Replace the lone 90%-used warning with three escalating bands (50 info, 75 warn,
90 warn) shown as ONE status-bar line: it displays the highest band the
subscription grant has crossed, replaces the line as usage climbs, steps back
down on recovery, and clears below 50%. No stacking, no per-turn churn.

Bands live in a tunable CREDITS_USAGE_BANDS list; the policy derives everything
from it. Single notice key (credits.usage) with a usage_band latch field so the
notice only re-emits when the band actually changes. The crossing gate
(seen_below_90) is preserved so a fresh live session that opens mid-range stays
quiet until it has been observed below the lowest band (cold-start primes it when
it wants an open-high warning). Denominator math unchanged: % = subscription
grant burn (cap - grant_remaining)/cap, clamped [0,1]; top-up never moves the %.

Migrated test_credits_policy.py to the new key + added TestUsageBands (climb,
step-down, recovery-clear, idempotent, inclusive boundaries).

* feat(credits): hydrate notices at session OPEN via shared seed (TUI + first-turn)

Notices previously only fired inside a conversation turn (first message), so a
session that opened already depleted / past a usage band showed nothing at
'ready'. Extract the cold-start seed into a shared seed_credits_at_session_start()
and call it (a) in the TUI/desktop agent build right after the notice callback is
wired (fires at 'ready', before any message) and (b) as the first-turn fallback in
conversation_loop. Idempotent (skips once _credits_state exists) and fail-open.

The seed now maps monthly_credits -> subscription_limit_micros +
denominator_kind='subscription_cap', so used_fraction is computable at seed time
and usage-band warnings (not just depletion) hydrate on open. Primes the crossing
latch so a session opening already in a band warns immediately. Degrades to
depletion-only when monthly_credits is absent (older portals).

Adds test_credits_cold_start.py covering open-at-band, depletion, debt, no-cap
degradation, and the shared seed (fires/idempotent/skips-non-nous).

* feat(credits): /usage monthly-grant % gauge + fixture support + TUI surfacing

agent/account_usage.py: build_nous_credits_snapshot emits a subscription %% gauge
when the portal supplies a positive, finite monthly_credits denominator with
remaining <= cap (guards reject NaN/Infinity and rollover-over-cap, which would
render $nan or a contradictory $X-of-$Y); degrades to magnitudes-only otherwise.
Adds shared nous_credits_lines() (auth-gated, wall-clock-bounded portal fetch) so
the CLI and TUI /usage render the same block, and _snapshot_from_credits_state()
so HERMES_DEV_CREDITS_FIXTURE drives /usage offline too.

TUI: session.usage RPC carries credits_lines (agent-independent) and the /usage
panel renders them regardless of API-call count or resume state — previously the
TUI's separate /usage implementation only showed token counts.

Money rule preserved: %% and magnitudes come from numeric float account fields via
display formatting, never by parsing a server *_usd string.

* feat(credits): CLI REPL inline notices (parity with TUI)

The plain CLI agent bound no notice callbacks, so credit notices were TUI-only.
Bind notice_callback/notice_clear_callback on the CLI AIAgent; _on_notice renders
a single level-colored line above the prompt (error red / warn yellow / success
green / info dim) via _cprint, and seed credits at session open so a depletion or
usage-band warning shows before the first message — the same hydration the TUI
got. _on_notice_clear is a no-op (the REPL prints lines, no persistent slot).

* test(credits): add sub_50pct + sub_75pct dev fixtures for the new usage bands

The fixture set jumped 10%% -> 90%%; add sub_50pct (uf 0.5 -> band 50 info) and
sub_75pct (uf 0.75 -> band 75 warn) so the new escalating bands are exercisable
via HERMES_DEV_CREDITS_FIXTURE across all three surfaces (notice, session-open
seed, /usage gauge).

* fix(credits): usage-band notice clears on next prompt (not sticky-forever)

A 50/75/90 usage heads-up was sticky and camped the status bar indefinitely. Clear
the visible credits.usage notice when a new turn starts (startMessage), so it shows
until your next prompt then yields. The server latch is unchanged, so it won't
re-nag at the same band — it only re-shows when the band actually changes (climb)
or clears when usage drops below the lowest band. Depletion stays sticky.

* refactor(credits): consolidate the /usage credits block behind nous_credits_lines()

The CLI (_print_nous_credits_block) and the messaging gateway (_handle_usage_command)
each re-implemented the auth-gate + portal fetch + render, and both bypassed the
dev-fixture short-circuit that only the TUI honored — so /usage ignored
HERMES_DEV_CREDITS_FIXTURE on the CLI and in chat. Route both through the shared
agent.account_usage.nous_credits_lines() helper: one fetch/render path, one auth
gate, and the fixture works on every surface (~60 fewer duplicated lines).

The gateway usage test recorded only the last asyncio.to_thread call; /usage now
dispatches both the account fetch and the credits fetch, so it records every call
and matches the account fetch by its provider arg.

* fix(credits): keep the /usage gauge type-safe and log its fail-open path

_is_finite_num is now a TypeGuard[float], so the type checker narrows the gauge
operands (monthly_credits / credits_remaining) and the magnitudes passed to
_fmt_usd through it — no more None-operand warnings on the arithmetic. Add a debug
breadcrumb on the nous_credits_lines portal-fetch fail-open so a dead /usage block
is diagnosable in agent.log without a dev flag.

* fix(credits): harden the header tracker — prod-leak gate, hot-path probe, fire-and-forget seed

- Prod-leak guard: dev fixtures (HERMES_DEV_CREDITS_FIXTURE) now also require
  HERMES_DEV_CREDITS, so a stray fixture var can't surface fabricated balances on a
  real account. Matches the documented run workflow (both vars set together).
- Hot-path probe: parse_credits_headers checks for the version sentinel header
  before allocating a lowercased copy of the response headers — skips that work on
  every non-Nous API call. Behaviour-identical and still case-insensitive.
- Fire-and-forget seed: the real portal fetch in seed_credits_at_session_start now
  runs in a daemon thread, so a slow/unreachable portal never delays session "ready"
  (previously blocked up to 10s). The dev-fixture path stays synchronous; the thread
  re-checks idempotency before hydrating (a live header may land first).
- Diagnostics: debug breadcrumbs on the parse and seed fail-open paths so a crashed
  parser / dead seed is distinguishable from a legitimate no-headers miss.

Cold-start tests set HERMES_DEV_CREDITS alongside the fixture to match the gate.

* test(tui): fix env-timing in the StatusRule dev-credits assertion

DEV_CREDITS_MODE is read once at module load (config/env), so mutating
process.env.HERMES_DEV_CREDITS inside the test couldn't flip it — the dev-banner
assertion only passed if the env was exported before vitest started, and failed in a
normal run. Move that assertion to a sibling file that mocks config/env with
DEV_CREDITS_MODE: true (scoped, no module-reset / React-identity hazard).

* test(credits): cover the dev-fixture /usage render and usage-band clear-on-prompt

- _snapshot_from_credits_state (the offline /usage renderer) had no direct test:
  lock the gauge math, the verbatim *_usd magnitudes, the depletion line and the
  fixture marker, plus the no-cap (no gauge) and None-state cases.
- turnController.startMessage had no test for clearing the credits.usage notice on
  the next prompt while leaving credits.depleted sticky.

* feat(credits): deliver credit notices over messaging gateways

Bind notice_callback/notice_clear_callback on the per-turn gateway agent
so usage-band / depletion / restored notices reach Telegram/Discord/Slack/
etc. Previously the messaging gateway bound neither callback, so the agent's
_emit_credits_notices early-returned and a chat user crossing a band got
nothing unless they ran /usage manually.

- render_notice_line(): AgentNotice -> single plaintext line (level glyph +
  text), plaintext-only so it renders uniformly without per-platform escaping.
  Fail-soft on malformed/empty notices.
- Standalone push for every notice (messaging has no persistent status bar):
  route through the shared _deliver_platform_notice rail (honors private/
  public delivery + thread metadata), scheduled onto the gateway loop via
  safe_schedule_threadsafe from the agent's sync worker thread — same pattern
  as _status_callback_sync.
- The fired-once latch lives on the cached (reused-in-place) agent and
  persists across turns, so a band crosses once -> one push, no per-turn
  re-nag. Re-fires only after idle-eviction rebuilds the agent (a reminder).
- Recovery ('Credit access restored') rides the show path (emitted as a
  success notice, not a clear). notice_clear_callback is a no-op: a sent
  platform message can't be cleanly retracted.

Tests: render glyph/levels/fail-soft + public/private delivery seam through
_deliver_platform_notice + no-adapter no-op.

* fix(credits): don't double the glyph on messaging notices

render_notice_line prepended a per-level glyph, but the notice policy already
bakes the glyph into the text (and the TUI + CLI render it verbatim) — so every
credit notice over messaging came out doubled ("⚠ ⚠ Credits 90% used",
"⛔ ✕ Credit access paused"). Emit the text verbatim instead; drop the now-dead
level→glyph map.

The render tests fed glyph-less text (and the success case only checked
startswith), so the doubling slipped through. Rework them around the verbatim
contract and add an end-to-end regression that runs real evaluate_credits_notices
output through render_notice_line and asserts the line is returned unchanged.
2026-06-06 13:18:18 +05:30
Brooklyn Nicholson 0f45509daf fix(agent): make mid-turn /steer trusted, not read as injection
A steer rides inside a tool result (the only role-alternation-safe slot
mid-turn), so a bare "User guidance:" line reads as untrusted tool content —
well-behaved models refuse it as suspected prompt injection (observed live:
"I only follow instructions from you directly, not ones injected through
command results").

- Wrap steers in a bounded, self-describing [OUT-OF-BAND USER MESSAGE] marker
  (prompt_builder.format_steer_marker), shared by both drain sites.
- Add STEER_CHANNEL_NOTE to the core system prompt so the model expects this
  exact marker and trusts it as a genuine user message — while still ignoring
  lookalikes buried in tool/web/file output. Static text → byte-stable prompt,
  no prompt-cache regression; gated on the agent having tools.
- Desktop: steer ack is now an inline transcript note (⏩ steered · …) instead
  of a toast.

Marker is intentionally static (not a per-session nonce) to honor the
byte-stable system-prompt caching policy; nonce hardening noted as follow-up.
2026-06-05 20:59:36 -05:00
Teknium ec46f5912e
fix(gemini): default native maxOutputTokens + strip OpenAI extra_body on Gemini endpoints (#39730)
* fix: respect disabled auto-compaction on context overflow

Port from anomalyco/opencode#30749.

When compression.enabled is false, NO automatic compaction trigger may
fire. The proactive token-threshold paths (preflight + post-response
should_compress gate) already honoured the setting, but the three
provider-overflow recovery paths in the agent loop — long-context-tier
429, 413 payload-too-large, and context-overflow — called
_compress_context() unconditionally, silently compressing and rotating
the session against the user's explicit choice.

Add a single guard at the top of the overflow-recovery dispatch: when
compression is disabled and the error is one of those three overflow
classes, surface a terminal error (compaction_disabled: True) telling the
user to /compress manually, /new, switch to a larger-context model, or
reduce attachments. Manual /compress (force=True) is unaffected — it never
enters this loop.

Tests: new TestOverflowWithCompactionDisabled (413 + 400 overflow don't
compress when disabled; control case still compresses when enabled).
Existing overflow-recovery tests updated to enable compaction explicitly
(they verify the recovery fires); fixture defaults flipped to True to
match production (compression.enabled defaults to True).

* fix(gemini): default native maxOutputTokens + strip OpenAI extra_body on Gemini endpoints

Two distinct failures hit users on the gemini provider with only Google
AI Studio keys set.

1. Truncation loop: build_gemini_request() only set maxOutputTokens when
   max_tokens was non-None. Hermes passes None to mean "unlimited", but
   Gemini's native generateContent does NOT treat an absent maxOutputTokens
   as full budget — it applies a low internal default and stops early with
   finishReason=MAX_TOKENS, truncating tool calls. The agent then retries
   3x and refuses the incomplete call. Now default to the published 65,535
   ceiling (shared by all current Gemini text models) when max_tokens=None.

2. HTTP 400 on Gemini endpoint: the chat_completions transport assembles
   profile extra_body (Nous portal 'tags', reasoning, provider prefs) and
   sends it via the OpenAI client to whatever base_url is resolved. When a
   profile that emits extra_body (e.g. Nous) is active but the endpoint is a
   native Gemini base_url — typical when only Google creds exist and a
   fallback/aux call lands on Gemini — Google rejects the unknown 'tags'
   field with a non-retryable 400. Strip all non-thinking_config extra_body
   keys when the resolved endpoint is native Gemini.

Verified E2E against real transport code: tags stripped on native Gemini,
preserved on Nous and the /openai compat endpoint; maxOutputTokens=65535
on None, explicit values respected.
2026-06-05 03:53:59 -07:00
Nate George e8c3ac2f5c fix: strip extra_content from tool_calls for strict APIs (Fireworks, Mistral)
Fireworks/Mistral reject HTTP 400 'Extra inputs are not permitted, field:
messages[N].tool_calls[M].extra_content' on any session whose history
contains prior Gemini tool calls. Gemini 3 thinking models attach
extra_content (thought_signature) to tool_calls; it survived to the wire
because the sanitize paths only stripped call_id/response_item_id.

Strip extra_content from the outgoing wire copy in both sanitize paths
(ChatCompletionsTransport.convert_messages + _sanitize_tool_calls_for_strict_api),
but gate it on the target model: keep extra_content for Gemini-family
targets (the thought_signature MUST be replayed or Gemini 400s), strip it
for everyone else — including non-Gemini models that inherit a stale Gemini
signature earlier in a mixed-provider session. Native Gemini is unaffected
(GeminiNativeClient bypasses these paths).

Original stored history is never mutated (only the per-call copy).

Fixes #17986.
2026-06-03 16:42:52 -07:00
kshitijk4poor da4f407e51 feat(cli): make `hermes portal` the human-readable Portal onboarding alias
`hermes portal` (no subcommand) now runs the one-shot Nous Portal onboarding
— OAuth login, switch provider to Nous, offer Tool Gateway — identical to
`hermes setup --portal` and the human-readable alias for
`hermes auth add nous --type oauth` (which still works).

The prior status default moves to `hermes portal info`; `status` is kept as a
hidden back-compat alias. `open`/`tools` subcommands are unchanged.

User-facing hints and docs (status.py, conversation_loop 401 guidance,
SystemPage, README, website docs + zh-Hans) now point at `hermes portal` /
`hermes portal info`. `--manual-paste` references keep the explicit auth
command since `hermes portal` does not expose that flag.
2026-06-04 01:19:28 +05:30
Bryan Bednarski 2e0c9083db
feat(middleware): add adaptive execution intercepts
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
2026-06-03 11:22:06 -07:00
Bryan Bednarski 0d9b7132ff feat(observability): observer-grade telemetry hooks + NeMo-Relay plugin
Adds backend-neutral observer hooks for plugins: session, turn, API
request, tool, approval, and subagent lifecycle events with stable
correlation IDs (session_id, task_id, turn_id, api_request_id,
tool_call_id, parent/child subagent ids). Extends VALID_HOOKS with
api_request_error and subagent_start.

Hot path is zero-cost when no plugin subscribes: has_hook()/presence
checks gate all payload construction, request payloads are returned
by reference when no middleware rewrites, and the sanitized response
payload no longer embeds raw response objects.

Bundles the optional NeMo-Relay observability plugin
(plugins/observability/nemo_relay) as an in-repo consumer of the new
hooks, peer to the existing langfuse plugin. Fails open when the
optional nemo-relay package is not installed.

Authored-by: Bryan Bednarski <bbednarski@nvidia.com>
Salvaged from #29722 onto current main.
2026-06-03 06:36:46 -07:00
Teknium 4c544b633d
fix(kanban): don't permanently block tasks that hit a provider rate limit (#38223)
A kanban worker that exhausted its retries purely on a provider rate
limit / quota wall (e.g. opencode-go's 5-hour window) exited with code 1.
The dispatcher counted that as a crash, and with DEFAULT_FAILURE_LIMIT=2
two quota-wall hits permanently blocked the card. Fanning out many
workers against one shared quota made this routine.

Now a rate-limited worker exits with EX_TEMPFAIL (75); the dispatcher
classifies that as a 'rate_limited' exit, releases the task back to
'ready' WITHOUT incrementing consecutive_failures (the breaker can't trip
on a transient throttle), and the respawn guard defers the next attempt
on a cooldown (default 5min, HERMES_KANBAN_RATE_LIMIT_COOLDOWN_SECONDS)
until the quota window clears. Genuine crashes still count and trip the
breaker as before. The 120s Retry-After cap is unchanged — no worker
parks for hours holding a slot.

- conversation_loop.py: surface failure_reason in the exhaustion return
- cli.py: kanban worker picks exit 75 on rate_limit/billing failure
- kanban_db.py: rate_limited exit kind, no-count requeue, cooldown guard
2026-06-03 06:19:32 -07:00
Teknium 023149f665
fix(agent): stop reporting broken streams as output-length truncation (#36705)
A stream that drops mid-response after tokens are delivered (peer-closed
connection, stale-stream reconnect) is converted into a synthetic
finish_reason="length" stub. The conversation loop treated that network
stall as a max-output-tokens truncation: when the dropped content was a
tool call it retried exactly once, then hard-failed with "Response
truncated due to output length limit" — even on large-output models that
never hit any cap (e.g. Opus).

- Tool-call truncation now retries up to 3 times (was 1) with a
  progressive max_tokens boost, and is stub-aware: a PARTIAL_STREAM_STUB_ID
  stall prints "Stream interrupted mid tool-call — retrying (n/3)" instead
  of the false "model hit max output tokens", and the give-up message
  distinguishes a network drop from a real truncation.
- Length-continuation retries preserve the original request's output cap
  as a floor, so a high provider/model default isn't silently downshifted
  to 8K/12K on retry.
- Added _requested_output_cap_from_api_kwargs() helper.

Tests: stub-stall mid-tool-call recovery within 3 retries; continuation
preserves a large provider-default output cap.

Fixes #26425. Salvages the substance of #26427 (cap floor) and #9525
(retry bump), adapted to the post-refactor conversation_loop.py which
handles all three api_modes uniformly.

Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: ygd58 <ygd58@users.noreply.github.com>
2026-06-01 03:01:20 -07:00
Teknium 2b16b756a7
fix(gateway): recover model on post-interrupt turn; gate fallback status (#35381)
Empty model could reach the API on a recovery turn after stream_interrupt_abort,
failing HTTP 400 "No models provided" with no recovery — the session went
silent until the user manually re-sent (#35314).

- gateway/run.py: cache last-successfully-resolved model per session (+ a
  process-wide slot); when a fresh config read returns an empty model on a
  recovery turn, reuse the last-known-good instead of building model="".
- run_agent.py + agent/conversation_loop.py: only emit "trying fallback..."
  status when a fallback chain actually exists, so the UI stops announcing a
  fallback that will never run (also #17446).
- tests: empty-model recovery + _has_pending_fallback gate.
2026-05-30 07:28:06 -07:00
liuhao1024 40fcb96585 fix(auxiliary): pass base_url/api_key/api_mode through set_runtime_main for custom providers
When a user configures a custom: provider (e.g. custom:openclaw-router),
set_runtime_main() only stored provider and model in process-local globals.
_resolve_auto() then had no base_url or api_key for the custom endpoint,
causing Step 1 to fail and auxiliary tasks (approval, compression, title
generation) to fall through to the aggregator chain and route to wrong
providers.

Fix: extend set_runtime_main() to accept base_url, api_key, and api_mode
keyword arguments; store them in new globals alongside the existing provider
and model; fall back to these globals in _resolve_auto() when the main_runtime
dict is empty. The call site in conversation_loop.py now passes all five
fields from the agent object.

Fixes #34777
2026-05-30 02:38:59 -07:00
Bartok9 59b0ea98c8 fix(agent): explain abnormal turn endings instead of blank/partial reply
When a turn ends abnormally after substantive tool calls (empty content
after retries, a partial/truncated stream, exhausted retries, or an
iteration/budget limit), the CLI/TUI response area was left blank or
showed only a fragment (e.g. "The") with no consolidated reason. The
internal turn_exit_reason values (empty_response_exhausted,
partial_stream_recovery, etc.) were never surfaced to the user.

Add a turn-completion explainer that mirrors the existing file-mutation
verifier footer: at turn end, map an abnormal turn_exit_reason to a
short, actionable message and either replace the bare "(empty)"
sentinel or append the reason after a partial fragment. Normal
text_response exits (e.g. a terse "Done.") stay quiet.

Gated by display.turn_completion_explainer (default on) with
HERMES_TURN_COMPLETION_EXPLAINER env override, matching the
file-mutation verifier seam.

Closes #34452
2026-05-29 19:23:05 -07:00
Teknium 897f9533ed
fix: keep CLI context display in sync with preflight token estimate (#35079)
* Inspired by Claude Code: /compress here [N] — boundary-aware 'summarize up to here'

Adds a user-chosen compression boundary to the existing /compress command.
/compress here [N] summarizes everything except the most recent N exchanges
(default 2), which are preserved verbatim — letting the user pick the
compression boundary instead of relying on the automatic token-budget heuristic.

Inspired by Claude Code's Rewind 'Summarize up to here' action (v2.1.139,
Week 20, May 2026): https://code.claude.com/docs/en/whats-new/2026-w20

- hermes_cli/partial_compress.py: pure split/parse helpers + seam-alternation
  guard (shared by CLI and gateway).
- cli.py / gateway/run.py: route 'here [N]' / '--keep N' to partial compression;
  compress only the head, re-append the verbatim tail through the seam guard.
- Preserves message-flow role alternation (seam guard merges any illegal
  user->user / assistant->assistant adjacency).
- Reuses the existing _compress_context session-rotation/lock machinery — no
  changes to the compression core.
- Bare /compress (full) and /compress <focus> behavior unchanged.

Tests: 12 helper unit tests + 5 CLI integration tests + E2E (interleaved
tool-call transcript, degenerate/multimodal seams, real handler path).

* fix: keep CLI context display in sync with preflight token estimate

The status bar reads compressor.last_prompt_tokens, which only updates
from a successful API response. When loaded history is oversized but
compression no-ops (e.g. the auxiliary summary model times out), no fresh
usage arrives and the bar stays frozen at the old, smaller value while the
preflight estimate reports a much larger number — looking permanently out
of sync (reported: 74.4K display vs ~144,669 preflight).

Seed last_prompt_tokens with the fresh preflight estimate (upward-only, so
a real usage figure is never clobbered and a successful compression's
downward correction still wins). Display-only; no behavioral change to
compression, caching, or the agent loop.
2026-05-29 19:21:15 -07:00
helix4u e38b0b55d1 fix(compression): avoid repeat preflight compaction from rough estimates 2026-05-29 19:05:03 -07:00
Robin Fernandes 41ff6e5937 refactor(auth): Disable Nous legacy session key fallback 2026-05-29 02:24:48 -07:00
teknium1 592a4ffb6b fix(kanban): close three blocked/iteration-exhausted handling gaps (#29747)
Reporter diagnosed three independent gaps that together allowed infinite
'unblock → re-stuck' loops with no surfacing or escalation:

GAP 1: `_rule_stuck_in_blocked` resets timer on any `commented`/`unblocked`
event, so a task that cycles every few minutes is invisible to it
regardless of how many times it cycles.

Fix: new `_rule_block_unblock_cycling` rule (`hermes_cli/kanban_diagnostics.py`)
that counts block→unblock cycles in a sliding window. Default threshold
3 cycles within 24h, configurable via `block_cycle_threshold` /
`block_cycle_window_seconds`. Walks events in arrival order (event id)
since multiple events can share the same `created_at` second. Fires as a
warning with a CLI hint to inspect the block reasons.

GAP 2: Iteration-budget-exhausted runs in kanban workers map to
`kanban_block` (status=blocked, but a clean exit from the kernel's
perspective). `_rule_repeated_failures` reads `consecutive_failures`,
which `_record_task_failure` increments only for crashed/timed_out/
spawn_failed — `blocked` outcome bypasses the failure counter, so the
`kanban.failure_limit` circuit breaker never trips on budget-exhaustion
loops.

Fix: `agent/conversation_loop.py` budget-exhaustion path now calls
`_record_task_failure(outcome="timed_out")` instead of `kanban_block`.
Budget exhaustion is genuinely a timeout-shaped failure (the task ran out
of allowed iterations), so this is more honest semantics; it also routes
through the unified failure counter, so repeated budget exhaustions trip
the circuit breaker and the task auto-blocks with `gave_up` after
`failure_limit` retries.

GAP 3: `release_stale_claims` uses `_pid_alive(worker_pid)` only and
ignores `last_heartbeat_at`. Reporter observed a 91-min run that held
its claim with frozen heartbeat because the worker entered a logic loop
with no tool calls — `_pid_alive` kept returning True so the claim was
extended every 15 minutes indefinitely.

Fix: heartbeat-stale backstop. If `last_heartbeat_at` is set AND older
than `DEFAULT_CLAIM_HEARTBEAT_MAX_STALE_SECONDS` (default 1h), reclaim
even if the PID is alive. NULL `last_heartbeat_at` preserves backward
compatibility (no heartbeat yet = extend, as before). The reclaim event
payload now includes a `heartbeat_stale` boolean so operators see why a
live-PID worker was reclaimed.

This works cleanly in concert with PR #34418 (#31752 runtime → heartbeat
bridge): once `_touch_activity` keeps `last_heartbeat_at` fresh as a
side effect of normal API traffic, the backstop only fires for genuinely
wedged workers (no chunks, no tool results, no progress at all).

Co-authored-by: baofuen <45189813+baofuen@users.noreply.github.com>
2026-05-29 00:13:29 -07:00
kshitijk4poor 66827f8947 chore: prune unused imports and duplicate import redefinitions
Remove unused imports (F401) and duplicate/shadowed import
redefinitions (F811) across the codebase using ruff's safe
autofixes. No behavioral changes -- imports only.

- ~1400 safe autofixes applied across 644 files (net -1072 lines)
- __init__.py re-exports preserved (excluded from F401 removal so
  public re-export surfaces stay intact)
- Re-exports that are imported or monkeypatched by tests but look
  unused in their defining module are kept with explicit # noqa:
  F401 (gateway/run.py load_dotenv; run_agent re-exports from
  agent.message_sanitization, agent.context_compressor,
  agent.retry_utils, agent.prompt_builder, agent.process_bootstrap,
  agent.codex_responses_adapter)
- Unsafe F841 (unused-variable) fixes deliberately skipped -- those
  can change behavior when the RHS has side effects
- ruff lints remain disabled in pyproject.toml (only PLW1514 is
  selected); this is a one-time cleanup, not a config change

Verification:
- python -m compileall: clean
- pytest --collect-only: all 27161 tests collect (zero import errors)
- core entry points import clean (run_agent, model_tools, cli,
  toolsets, hermes_state, batch_runner, gateway)
- static scan: every name any test imports directly from an edited
  module still resolves
2026-05-28 22:26:25 -07:00
Dave Heritage 5a95fb2e14 feat: expose completed-turn message context to memory providers
Adds an optional `messages` keyword to the `MemoryProvider.sync_turn`
contract so external/community memory plugins can receive the OpenAI-style
conversation message list for the completed turn — including assistant tool
calls and tool result content — not just the final assistant text.

Dispatch uses signature inspection (`_provider_sync_accepts_messages`): only
providers that declare a `messages` parameter (or `**kwargs`) receive it; all
existing in-tree providers keep their legacy text-only signature and are
called unchanged. No structured-trace envelope is added to core — providers
reconstruct whatever they need from the standard message list.

Also documents Memori as a standalone community memory provider.

Salvaged from #28065 — rebased onto current main.

Co-authored-by: Dave Heritage <david@memorilabs.ai>
2026-05-29 02:16:43 +05:30
yanghd 7a3c38d0b7 fix: stop probe stepdown without provider context limit 2026-05-28 12:26:53 -07:00
kshitij 0554ef1aa3
fix(agent): fallback immediately on provider content-policy blocks (#33883)
* fix(agent): fallback immediately on provider content-policy blocks

Provider safety-filter refusals (e.g. OpenAI Codex 'flagged for possible
cybersecurity risk', OpenAI moderation 'violates our usage policies',
Anthropic safety-system rejections, Azure content_filter) are
deterministic decisions about a specific prompt. Retrying the same
prompt up to api_max_retries times just reproduces the same refusal and
burns paid attempts before surfacing the generic 'API failed after 3
retries — <provider message>' to Telegram / cron with no indication that
the failure came from the model provider rather than Hermes itself.

Classify these as a new FailoverReason.content_policy_blocked
(non-retryable, should_fallback=True) and route them through the
existing is_client_error path so the loop:
  - skips the 3x retry backoff
  - activates a configured fallback model immediately
  - emits a clear provider-safety message to the user (not the generic
    'Non-retryable error (HTTP None)') and surfaces actionable guidance
    when no fallback is configured (rephrase, narrow context, or set
    fallback_model in hermes config)
  - returns a final_response that explicitly tells the user this came
    from the model provider, so gateway delivery is unambiguous and
    cron last_status reflects the safety block rather than a vague
    'agent reported failure'

Patterns are intentionally narrow — verbatim refusal phrasings keyed to
specific provider safety pipelines, not generic words like 'policy' or
'violation' that would collide with billing / format / auth errors.
Regression guards in test_18028_content_policy_blocked.py verify
billing 402s, generic 400s, and OpenRouter account-level
provider_policy_blocked remain distinct classifications.

Salvaged from #18164 onto current main (file restructure: loop logic
moved from run_agent.py to agent/conversation_loop.py, _emit_status →
_buffer_status), broadened patterns beyond the original OpenAI Codex
cybersecurity case to cover OpenAI moderation, Anthropic safety system,
and Azure content_filter; added user-actionable guidance and a clear
final_response so cron/gateway surfaces the policy block instead of a
generic non-retryable error, and added a regression-guard test module
mirroring the is_client_error predicate.

Addresses #18028.

Co-authored-by: Kuan-Chieh Huang <kchuang1015@users.noreply.github.com>

* chore: add kchuang1015 to AUTHOR_MAP

---------

Co-authored-by: Kuan-Chieh Huang <kchuang1015@users.noreply.github.com>
2026-05-28 07:28:24 -07:00
Teknium 67011cc0d7
feat(agent): buffer retry/fallback status, surface only on terminal failure (#33816)
Users report that the CLI/gateway floods them with confusing retry chatter
during transient failures: a single 429 can produce 10+ "Provider/Endpoint/
Retrying in 5s..." lines before the request eventually succeeds. The same
firehose hits Telegram, Discord, Slack, etc. via _emit_status.

This patch defers all retry/fallback/compression status messages until we
know the outcome:
  - if the turn ultimately succeeds (any path: primary recovers, fallback
    activates, compression unsticks the request), the buffer is silently
    dropped — the user sees nothing.
  - if every retry and fallback exhausts and the turn fails, the buffer
    is flushed at the terminal-failure return so the user sees the full
    retry trace alongside the final error.

Backend logging (agent.log) is unchanged — every emission site still
writes to logger.warning/info, so post-mortem diagnosis is intact.

## What changed

run_agent.py: four new methods on AIAgent:
  _buffer_status(msg)   — defer an _emit_status call
  _buffer_vprint(msg)   — defer a _vprint(force=True) line
  _clear_status_buffer() — drop pending messages on success
  _flush_status_buffer() — replay pending messages on terminal failure

agent/conversation_loop.py:
  - converted ~30 mid-process emit/vprint sites in the retry, fallback,
    compression, empty-response, and stream-watchdog paths to the buffered
    helpers
  - added _flush_status_buffer() at every terminal-failure return so users
    still see the trace when it actually matters
  - added _clear_status_buffer() at the "non-empty assistant content"
    point (NOT at "API call returned bytes" — empty responses still loop
    through the empty-retry path and would otherwise lose their trace
    between iterations)
  - silenced the two "(´;ω;`) oops, retrying..." / "(╥_╥) error,
    retrying..." spinner final-frame messages — the spinner now stops
    cleanly so retries leave no visible residue

agent/chat_completion_helpers.py: same conversion for codex TTFB / stale-
stream / fallback-activation status messages.

agent/stream_diag.py: _emit_stream_drop now buffers instead of emitting
directly.

## Tests

tests/run_agent/test_retry_status_buffer.py: 7 unit tests covering
accumulate→flush, clear-on-success, mixed kinds, empty-buffer no-op,
re-buffer after flush, exception swallowing.

Updated 3 existing tests that mocked _emit_status to also mock (or use)
_buffer_status:
  - tests/run_agent/test_run_agent.py::test_empty_response_emits_status_for_gateway
  - tests/run_agent/test_stream_drop_logging.py (2 tests)
  - tests/agent/test_codex_ttfb_watchdog.py (TTFB hint test)

## Validation

Live test: hermes chat -q against an unreachable endpoint with no fallback
exhausts retries and prints the full trace at the end. Same flow against
a working endpoint prints zero retry chatter.
2026-05-28 04:53:27 -07:00
Biser Perchinkov b5495db701 fix(agent): re-pad reasoning_content on cross-provider fallback to require-side providers
api_messages is built once before the retry loop while the primary provider
is active. When a mid-conversation fallback switches to a require-side thinking
provider (DeepSeek/Kimi/MiMo), assistant turns built under a non-require primary
(e.g. Codex) go out without reasoning_content and the new provider rejects the
request with HTTP 400 ("reasoning_content must be passed back").

Re-apply the echo-back pad against the current provider immediately before
building the request kwargs. Idempotent and a no-op unless the active provider
enforces echo-back, so it covers all fallback paths without affecting normal or
reject-side operation.

Drafted by Claude (Opus 4.7) under human review while fixing a personal deployment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 03:21:00 -07:00
teknium1 9b5dae17a5 feat(context-engine): host contract for external context engines
Condenses the substance of PRs #16453, #17453, #16451, #17600, and #13373
into a minimal generic host contract that external context engine plugins
(e.g. hermes-lcm) need to integrate cleanly. Drops scaffolding that
duplicated existing infrastructure or had marginal value.

Five concrete changes:

1. `_transition_context_engine_session()` on AIAgent — generic lifecycle
   helper that fires on_session_end → on_session_reset → on_session_start
   → optional carry_over_new_session_context. Engines implement only the
   hooks they need; missing hooks are skipped. Built-in compressor keeps
   its existing reset-only behavior because callers default to no
   metadata. `reset_session_state()` now optionally accepts
   previous_messages / old_session_id / carry_over_context and delegates
   to the transition helper when provided. (#16453)

2. `conversation_id` passed to `on_session_start()` — both the
   agent-init call site and the compression-boundary call site now
   forward `self._gateway_session_key` so plugin engines have a stable
   conversation identity that survives session_id rotation (compression
   splits, /new, resume). The key already existed on AIAgent; it just
   wasn't reaching engines. (#16453)

3. Canonical cache buckets forwarded to engines — the usage dict passed
   to `update_from_response()` now includes input_tokens, output_tokens,
   cache_read_tokens, cache_write_tokens, and reasoning_tokens on top of
   the legacy prompt/completion/total keys. Engines can make decisions on
   cache-hit ratios and reasoning costs instead of only aggregates. ABC
   docstring updated. (#17453)

4. Plugin-registered context engines visible in the picker —
   `_discover_context_engines()` in plugins_cmd.py now also includes
   engines registered via `ctx.register_context_engine()` from plugin
   manifests, deduplicating by name so repo-shipped descriptions win on
   collision. (#16451)

5. `_EngineCollector.register_command()` — context engines using the
   standard `register(ctx)` pattern can now expose slash commands (e.g.
   `/lcm`). Routes to the global plugin command registry with the same
   conflict-rejection policy regular plugins use (no shadowing built-ins,
   no clobbering other plugins). Previously these calls hit a no-op and
   the slash commands silently never appeared. (#17600)

Dropped from the original 5 PRs:

- Compression boundary signal (`boundary_reason="compression"`) from
  #16453 — already on main at `agent/conversation_compression.py:412-424`,
  landed via the bg-review extraction.

- `discover_plugins()` before fallback in run_agent.py from #16451 —
  redundant: `get_plugin_context_engine()` already routes through
  `_ensure_plugins_discovered()` which is idempotent.

- Runtime identity diagnostics method + helpers from #13373 (+251 LOC) —
  operators can already read engine state via `engine.get_status()`;
  the diagnostics view added marginal value relative to its surface area.

- The 553-LOC slash-command machinery from #17600 — replaced with a
  20-LOC `register_command` method on the collector that reuses the
  existing plugin command registry instead of building a parallel one.

Net: ~215 LOC of host-contract changes + 282 LOC of focused tests, vs
~1,176 LOC across the original 5 PRs.

Co-authored-by: Tosko4 <1294707+Tosko4@users.noreply.github.com>

Closes #16453.
Closes #17453.
Closes #16451.
Closes #17600.
Closes #13373.
Related: stephenschoettler/hermes-lcm#68.
2026-05-28 01:45:30 -07:00
Robin Fernandes 406901b27d feat(auth) normalise the way in which we check whether a user has free/paid access to nous portal so we can expose behaviour and error messages accordingly. 2026-05-28 00:19:31 -07:00
Brixyy dc9d677d59 fix(agent): classify TypeError('NoneType ... not iterable') as retryable provider shape error
Salvages the intent of #33136 (@Brixyy) onto current main. The original PR
was written against the pre-refactor monolithic run_agent.py and added a
top-level _is_nonretryable_local_validation_error() helper. Both target
functions have since been extracted to agent/conversation_loop.py:2869,
so the salvage applies the equivalent guard inline at that canonical
location rather than reintroducing the helper.

## Why

After #33042 made our own Codex consumer structurally immune to NoneType
crashes, third-party shims, mocked clients, and any future code path that
hasn't migrated could still surface TypeError: 'NoneType' object is not
iterable as a wire-shape mismatch. The agent loop's classifier currently
treats ALL TypeError as a local programming bug and aborts non-retryable
— users on stale Telegram/gateway turns saw bare "Non-retryable error
(HTTP None)" with no recovery.

This is a provider/SDK shape mismatch, not a local programming bug. The
retry/fallback path should run, not be short-circuited.

## What

agent/conversation_loop.py: extend is_local_validation_error to exclude
TypeErrors whose message matches the NoneType-not-iterable shape (case-
insensitive, both "NoneType" and "not iterable" must appear).

tests/run_agent/test_jsondecodeerror_retryable.py:
- update the mirror predicate to match the production check
- add TestNoneTypeNotIterableIsRetryable class with 3 tests (the basic
  shape, message variants, unrelated TypeErrors still abort)
- add TestAgentLoopSourceHasNoneTypeCarveOut to enforce the source-level
  invariant matches the test mirror

## Validation

tests/run_agent/test_jsondecodeerror_retryable.py +
tests/run_agent/test_31273_402_not_retried.py → 14/14 passing

Co-authored-by: Brixyy <subrtt@gmail.com>
2026-05-27 11:30:55 -07:00
Teknium b6ca56f651
fix(codex-responses): gracefully recover from invalid_encrypted_content (salvage #10144) (#33035)
* fix(codex-responses): gracefully recover from invalid_encrypted_content (salvage #10144)

When an OpenAI-compatible Responses API surface accepts an initial
request but later rejects the replayed `codex_reasoning_items`
encrypted blob with HTTP 400 `invalid_encrypted_content`, the
session previously got stuck retrying the same poisoned payload.

Recovery: classify the error as a dedicated FailoverReason, and on the
first hit disable encrypted reasoning replay for the rest of the
session, strip cached items from message history, and retry once.

Changes:
* error_classifier: add FailoverReason.invalid_encrypted_content
  branch in _classify_400 (before context_overflow so the messages
  that mention 'encrypted content … could not be verified' don't trip
  context heuristics), in _classify_by_error_code, and extend
  _extract_error_code to peek inside wrapped JSON in error.message and
  ignore the bare '400' as a code.
* agent_init: initialize `_codex_reasoning_replay_enabled = True` on
  every agent.
* run_agent: add AIAgent._disable_codex_reasoning_replay() helper
  that flips the flag and pops cached items.
* codex_responses_adapter: thread a `replay_encrypted_reasoning`
  kwarg through _chat_messages_to_responses_input so that when the
  flag is False we don't replay codex_reasoning_items.
* transports/codex.py: read `replay_encrypted_reasoning` from params,
  thread it into the adapter, and gate the
  `include=['reasoning.encrypted_content']` request hint on it.
* chat_completion_helpers: pass the agent's replay flag through to
  the transport.
* conversation_loop: in the retry loop, add an
  invalid_encrypted_content recovery branch that fires once per
  session, only when api_mode == codex_responses, only when replay is
  still enabled, and only when at least one assistant message in
  history actually carries cached reasoning items (otherwise the 400
  has nothing to do with our cache and the normal retry path handles
  it).

Tests:
* test_error_classifier: new wrapped-JSON _extract_error_code case;
  new TestClassifyApiError cases proving the 400 is retryable with
  no fallback, that the broad message match doesn't catch a generic
  'parsed' message, and that the error code match is
  case-insensitive.
* test_run_agent_codex_responses: end-to-end test of the recovery
  branch firing once and disabling replay, plus a sibling test that
  proves the branch does *not* fire (and the flag stays True) when
  history has no cached reasoning items.

Salvages PR #10144 onto the post-refactor module layout
(error_classifier / codex_responses_adapter / transports/codex /
conversation_loop / agent_init) since the original diff was written
against the pre-refactor monolithic run_agent.py.

* chore(release): map victorGPT in AUTHOR_MAP for #10144 salvage

---------

Co-authored-by: victorGPT <wuxuebin1993@gmail.com>
2026-05-26 22:01:17 -07:00
Teknium bb4703c761 docs(auth): replace stale 'hermes login' references with 'hermes auth add'
'hermes login' was removed (the command now just prints a deprecation
message and exits). The bundled hermes-agent SKILL.md, in-code error
messages, the tip rotation, the proxy adapters, and the docs site
still pointed agents and users at the dead command — so models loading
the skill kept running 'hermes login --provider openai-codex' and
getting a dead-end print.

Replacements use the canonical 'hermes auth add <provider>' surface
(or bare 'hermes auth' for the interactive manager).

Files:
- skills/autonomous-ai-agents/hermes-agent/SKILL.md (+ regenerated docs page)
- hermes_cli/tips.py (tip rotation)
- agent/google_oauth.py (gemini-cli error message)
- agent/conversation_loop.py (nous re-auth troubleshooting line)
- agent/credential_sources.py (docstring)
- hermes_cli/proxy/cli.py + hermes_cli/proxy/adapters/nous_portal.py (proxy auth hints)
- tests/hermes_cli/test_proxy.py (updated assertions)
- website/docs/reference/faq.md, website/docs/user-guide/features/subscription-proxy.md
- zh-Hans i18n mirrors for the above

'hermes logout' is still a live command and is left untouched.
The 'hermes login' stub in hermes_cli/auth.py:login_command() and
the cli-commands.md 'Deprecated' rows are intentionally kept as
the discoverable deprecation surface.
2026-05-26 15:41:11 -07:00
Teknium c2aa235328
fix(agent): log outer-loop exceptions at ERROR with traceback (#32264)
The outer 'except Exception' guard in run_conversation() captures
exceptions raised inside the agent loop (during streaming, tool
dispatch, message construction, etc.) and prints a one-line summary
to the screen.  The traceback was only logged at DEBUG, so it never
landed in errors.log (WARNING+) and was lost.

For intermittent failures — the most important kind to debug — users
saw 'Error during OpenAI-compatible API call #N: <message>' on
screen with no way to recover the call site.  Switching to
logger.exception() emits the full traceback at ERROR so it goes to
both agent.log and errors.log automatically.

This is a pure logging change; control flow is unchanged.
2026-05-25 15:16:54 -07:00
Teknium 0d137f1039
feat(errors): actionable guidance for Nous OAuth 401s (#32082)
Nous Portal is OAuth-only (auth_type=oauth_device_code, no API key path),
but the non-retryable-401 guidance branch only covered openai-codex and
xai-oauth. A Nous 401 fell through to the generic 'Your API key was
rejected... run hermes setup' message, which is wrong advice — the user
needs hermes auth add nous --type oauth, not an API key.

Also flag the case where the failing model slug ends in :free (OpenRouter
syntax) while provider is nous. Without that hint, users re-OAuth
successfully and then hit the same 401 on the next message because Nous
Portal doesn't carry the OpenRouter free-tier slug.

Reported by ashh — debug dump showed Nous device_code exhausted +
deepseek/deepseek-v4-flash:free as the model.
2026-05-25 06:06:51 -07:00