Commit Graph

2766 Commits

Author SHA1 Message Date
Teknium 3fb066c372 fix(agent): dotenv-compatible inline-comment stripping in load_env_file
Unquoted values truncate only at a # preceded by whitespace (so
KEY=foo#bar stays intact); quoted values scan escape-aware for the
matching close quote, keep through it, and drop a trailing '# ...'
remainder. Verified empirically against python-dotenv 1.2.2 on a
10-case corpus (full parity). Supersedes the approach in #57718, whose
scanner corrupted foo#bar-style values.
2026-08-02 09:58:59 -07:00
spfcraze 55abf206bf fix(agent): unescape quoted .env values in secret_scope.load_env_file
save_env_value writes values containing " or \ as escaped
double-quoted strings, and every other .env reader in the repo
(load_env/_parse_env_value, python-dotenv) reverses those escapes.
load_env_file — the parser behind build_profile_secret_scope, which
wraps every cron job and every multiplexed gateway turn — only stripped
the outer quotes, leaving the escapes literal. A credential containing
a double quote or backslash (JSON service-account blobs, generated
secrets) authenticates fine interactively but 401s under scoped
resolution, with no error pointing at the cause. Parse values with the
canonical _parse_env_value so all readers agree byte-for-byte.
2026-08-02 09:58:59 -07:00
Fangliquan 696fae8e78 fix(agent): strip UTF-8 BOM when loading profile .env into secret scope
Windows editors often save .env with a leading BOM; plain utf-8 left U+FEFF
on the first key so multiplex get_secret missed that credential.
2026-08-02 09:58:59 -07:00
Shaun Prince d15b638a88 fix(compression): let explicit interrupts cancel safely
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.

Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.

Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
2026-08-02 22:15:20 +05:30
kshitij e8656bedfe fix(agent): byte budget for the canon-args memo
Review follow-up on the #76098 salvage: the 4096-entry count bound alone
doesn't bound MEMORY — write_file/patch argument strings run 100KB+, so
a long-lived gateway process under sustained heavy write workloads could
pin ~800MB of evicted-session strings. A 32MB byte budget extends the
existing FIFO eviction; common-case args (0.5-2KB) never hit it. New
guard test mutation-checked (fails with the byte leg disabled).
2026-08-02 21:17:12 +05:30
spfcraze cf803603dc perf(agent): memoize send-path tool-call argument canonicalization
The pre-send normalization pass re-canonicalized every historical tool
call's argument JSON on every API-call iteration — quadratic in session
tool-call count. Route it through a bounded value-keyed memo (the
_MSG_TOKENS_CACHE idiom from agent/model_metadata.py): per-iteration
cost is now proportional to new tool calls, not all of them.

Measured (simulated growing session, repo venv): session-total
canonicalization cost 1056 ms -> 58 ms at 500 tool calls x 2 KB args
(18.3x), 5744 ms -> 79 ms at 500 x 16 KB (72.5x). Byte-parity with the
pre-fix logic is asserted at every iteration (unicode, nested,
malformed, empty, non-string args), and a call-count test proves
json.loads invocations went from K(K+1)/2 to K per session.
2026-08-02 21:17:12 +05:30
Bao 6ab390a476 fix(gateway): hydrate cold profile secret sources 2026-08-02 00:11:50 -07:00
kshitij 077317a0f6 simplify: collapse aux config reads, trim comments, use load_config_readonly
- Merge _aux_free_only() + _aux_openrouter_model() into single
  _aux_openrouter_settings() that reads config once via
  load_config_readonly (avoids double deepcopy).
- Remove 15-line block comment and 5-line inline comment that
  restated what the code already says.
- Trim module docstring from 10 lines to 3.
- Update test patches to target load_config_readonly.
2026-08-02 12:19:04 +05:30
webtecnica c19c63d9e6 fix(agent): make auxiliary auto-chain fallback configurable and free-only (#75803) 2026-08-02 12:19:04 +05:30
kshitijk4poor 88a629b9dc refactor(agent): share the cache_ttl disable predicate across init and stub paths
Follow-ups from review of #76113:
- Extract cache_ttl_means_disabled() as the single disable-synonym
  predicate; agent_init and prompt_caching_disabled_from_config both use
  it so the two detection sites can no longer drift (drift would recreate
  the #76085 bug class).
- Mirror _run_reference's not-None injection guard in
  aggregate_moa_context (stamping None was a harmless no-op copy).
- Replace a vacuous trailing test assertion with the intended
  input-non-mutation check; drop a stray blank line.
- Add a predicate-parity regression test (unknown TTL values keep
  caching enabled, matching historical agent_init semantics).
2026-08-02 11:52:59 +05:30
686f6c61 8ee51747fe fix(agent): consolidate cache-disable stubs with blank_cache_policy_stub
Absorb the useful deltas from the parallel #76121 approach: a single
blank_cache_policy_stub factory so _cache_disabled cannot be left off
hand-rolled SimpleNamespaces, and pin the live agent disable onto MoA
advisor fan-out and one-shot aggregate_moa_context decoration so those
paths track conversation state rather than a fresh config re-read.

Keeps the earlier tri-state prepared-aggregator no-agent fix. Adds
factory and synthesis/advisor regressions.

Coordinates with #76121 / #76085.

Co-authored-by: JoaoMarcos44 <87440198+JoaoMarcos44@users.noreply.github.com>
2026-08-02 11:52:59 +05:30
686f6c61 8e1a351e82 fix(agent): preserve None cache_disabled when MoA has no agent
Prepared-aggregator facades built via __new__ lack _agent. Accessing
self._agent raised inside the planner try and bool-coercion of a missing
snapshot forced False, suppressing config fallback for cache_ttl=off.
Pass a tri-state value and add a no-agent/config-off regression.
2026-08-02 11:52:59 +05:30
686f6c61 8a29703a1b fix(agent): honor prompt_caching.cache_ttl=off on stub policy paths
Blank SimpleNamespace stubs used by MoA decoration and
plan_cache_sections_for_destination never set _cache_disabled, so
anthropic_prompt_cache_policy re-injected cache_control markers after
operators turned caching off. Stamp the disable onto those stubs from
an explicit flag or the live config, and pass the agent flag from the
MoA aggregator path.

Fixes #76085
2026-08-02 11:52:59 +05:30
Gille 58e85f4314 fix(browser): replace expired cloud sessions 2026-08-02 11:18:41 +05:30
ethernet 25d0bcd424 fix(runtime): resolve Hermes-managed Node and uv before bare PATH
Hermes installs runtimes for itself — `uv` at `$HERMES_HOME/bin/uv`, Node
at `$HERMES_HOME/node` — and neither directory is on an arbitrary
process's PATH. Every `shutil.which("node"/"npm"/"npx"/"uv")` in Hermes's
own code therefore has two failure modes: the managed runtime is invisible,
so the caller reports "not installed" or degrades to a slower tier on a
machine that has exactly what it needed; and when a system copy also
exists, the one Hermes does not own wins.

Routed the Hermes-owned call sites through managed-aware resolvers:

- `agent/lsp/install.py`, `hermes_cli/dep_ensure.py`, `hermes_cli/main.py`
  (`_make_tui_argv`), `hermes_cli/tools_config.py` (`_run_post_setup`) now
  use `find_node_executable()`.
- `hermes_cli/tools_config.py::_pip_install` and `hermes_cli/setup.py`'s
  vercel install use `ensure_uv()` (installing uv is in scope during setup,
  and the Windows installer's `uv venv` does not seed pip, so the fallback
  tier is "No module named pip"). `tools/lazy_deps.py` uses `resolve_uv()`
  — a lookup, not a bootstrap, because it runs mid-turn for an optional
  dependency and downloading a runtime as a side effect exceeds what the
  caller asked for.
- `hermes_cli/gateway.py`: extracted `_append_node_dir_for_service()`,
  shared by the systemd unit and launchd plist generators, which appends
  the managed dirs before the PATH-resolved one. A service definition is
  written once and survives reboots, so resolving a system Node that
  happens to lead the installing shell's PATH bakes the wrong interpreter
  in permanently. Managed dirs are profile-scoped, so each profile's unit
  still names its own Node; the existing symlink-parent rule (don't
  `.resolve()`) is preserved verbatim.
- `tools/environments/local.py`: the terminal tool's subshell PATH gains
  the managed dirs, appended alongside the sane entries rather than
  prepended — a tool the user deliberately put on their own PATH still
  wins, and the managed one only fills a gap. This is also what makes the
  bare `which("uv")` in `tools/env_probe.py` correct: that probe reports
  the environment the *model* sees, and the model can only run what is on
  that subshell's PATH.

`scripts/install.ps1`: the persisted User PATH update becomes
`Set-ManagedNodeFirstOnUserPath`, a move-to-front rather than an
add-if-missing. Installs made by an older install.ps1 already have the
managed dir in User PATH — at the tail, behind a system Node — and an
add-if-missing check sees it present and leaves that ordering in place
forever, so the users the bug hurt would never be repaired. Unrelated
entries keep their relative order (empty segments included; a trailing
`;` is legal and the installer's other PATH code preserves them),
duplicates collapse, and it writes only when the string actually changes.

Tests:

- `tests/test_managed_runtime_resolution.py` — AST guard that fails any
  new bare `which()` for a managed runtime, with a short justified
  allow-list and a companion test that fails when an allow-list entry goes
  stale. Reading source is banned by AGENTS.md and this is the documented
  exception: the property is "no call site anywhere spells it this way",
  which no runtime seam can observe.
- `scripts/ci/test_install_ps1_path_migration.ps1` — behavioral, not a
  source regex: it lifts the real `Set-ManagedNodeFirstOnUserPath` out of
  install.ps1's AST and rewrites only the two registry calls into an
  in-memory store, so the shipped split/dedupe/prepend/change-detection
  logic executes for real. Not in the default lane (Linux runners have no
  PowerShell host); runs under `pwsh`. 13/13 assertions pass.
2026-08-01 21:17:51 -04:00
Teknium 38c09e5d73 fix(tool-executor): emit tool results on hard interrupt to keep alternation
The sequential executor's KeyboardInterrupt handlers emitted a cancelled
post-tool-call event for the current tool, called agent.interrupt(), then
re-raised — WITHOUT appending a tool result message for the interrupted call
or any remaining calls in the batch. The assistant tool-call turn was left
with no matching tool results, a message-role alternation violation that
malforms the next provider request (relying on downstream repair passes to
patch it, which don't run on every path).

The cooperative-interrupt block (_interrupt_requested) and the concurrent
executor already emit a result for every call_id; this brings the two hard-
interrupt handlers into line via a shared _append_cancelled_tool_results
helper that appends a cancelled result for the current + remaining calls
before re-raising.

Verified live before/after (0 tool results -> 3 for a 3-call batch
interrupted on the first tool) and with a sabotage-checked regression test.
52 interrupt/executor tests pass.
2026-08-01 16:42:57 -07:00
joaomarcos 18e0683bfc fix(auth): route anthropic adapter credential reads through the profile secret scope
Salvaged from #51604 (@JoaoMarcos44, issue #51603): resolve_anthropic_token()
and run_oauth_setup_token() in agent/anthropic_adapter.py read
ANTHROPIC_TOKEN / CLAUDE_CODE_OAUTH_TOKEN / ANTHROPIC_API_KEY via bare
os.getenv(), bypassing agent.secret_scope — a cross-profile over-read in
multiplex mode. Every other provider routes through
runtime_provider._getenv -> get_secret; the adapter now does the same via
a local _getenv wrapper (identical to os.getenv when multiplexing is off,
scope-authoritative + fail-closed when on).

Dropped from the original PR: the cron scheduler hunks (superseded by
fdab380a1a which installs the per-job profile scope) and the unrelated
hermes_logging Windows hunk (scope creep).

Includes the PR's RED->GREEN scope-isolation test file (6 tests).
2026-08-01 16:42:51 -07:00
Fangliquan 87bc710609 fix(agent): scope parallel batches from V4A patch headers 2026-08-01 11:40:59 -07:00
Teknium cb1e059a98 fix(agent): reader/writer path roles in parallel batch planner — search_files no longer races batched writes
The parallel tool-batch planner treated search_files as unconditionally
parallel-safe (_PARALLEL_SAFE_TOOLS) with no path reservation, so a
batch of patch(path=X) + search_files(path=dir(X)) landed in one
concurrent segment and the search could observe pre-mutation file
content — a same-block write->read stale-read race.

Fix the class, not the site: path-scoped reservations now carry a
reader/writer role.

- search_files joins _PATH_SCOPED_TOOLS as a READER, reserving its
  search root (default '.', matching the tool's default) instead of
  bypassing path checks entirely.
- Overlap only conflicts when a WRITER is on either side: a write into
  a searched/read subtree splits segments (ordered behind the write),
  while reader<->reader overlap — previously split needlessly — now
  stays parallel (concurrent reads commute).
- write_file/patch keep their existing writer barrier semantics.

Prior art surveyed for this design: Codex CLI's RwLock read/write
barrier (readers share, writers exclusive), Claude Code's
isConcurrencySafe partitioning, and gemini-cli's contiguous
parallelizable batching — all converge on reader-shared/writer-
exclusive with contiguous-order preservation, which this planner
already had for read_file/write_file/patch; this closes the
search_files gap and adds the missing reader/reader concession.

Verified by sabotage run (tests fail against the old planner) and an
E2E script exercising the real planner + real file I/O.
2026-08-01 10:46:25 -07:00
kshitijk4poor 85e0073902 refactor(compression): fold simplify-pass findings into feasibility skip
- Reuse telemetry['middle_window_tokens'] for the skip's middle estimate
  (is-None fallback to a fresh estimate) so log and telemetry agree
- Declare prellm_skip_count in the base telemetry schema (fixed shape)
- Defer _derive_auto_focus_topic into the non-skip branch (user-turn scan
  was wasted work on every skip)
- Document the skip in compress()'s Algorithm list and force: arg doc
- Drop dead call_llm patches from 7 tests (unreachable with
  _generate_summary mocked)
2026-08-01 16:00:52 +05:30
TRON 8daf03063d fix(compression): add pre-LLM feasibility check to skip costly no-op summaries
When the middle section is < 10% of threshold tokens AND at least one prior
real-usage ineffectiveness strike has been recorded, skip the expensive LLM
summarization call and fall through to the deterministic message-dropping
path.  Without this guard, a tool-heavy session where the protected tail
already holds most of the tokens can burn 500+ seconds on a summary call
that replaces a few lightweight messages with negligible token savings.

Key design decisions per GottZ review on PR #60451:

1. Separate _prellm_skip_count counter — never increments
   _ineffective_compression_count (the strike counter that latches at >=2
   to disable compression entirely).  One real strike + one skip must NOT
   permanently lock out compression until /new.

2. feasibility_skip sentinel flag — exempts skips from the abort branch
   (abort_on_summary_failure / _last_summary_auth_failure /
   _last_summary_network_failure).  A stale failure flag from a prior
   cycle must not turn a deliberate skip into a full abort.

3. reason=None for feasibility-skip fallbacks — a stale _last_summary_error
   from an earlier real failure must not be embedded into the skip's
   deterministic fallback marker.

4. info-level logging for feasibility-skip fallbacks (not warning) — this
   is an intentional optimization, not a failure.

Skipped when force=True (manual /compress) so auth/error handling paths
are always exercised on explicit user request.

Adds 6 regression tests (TestPreLlmFeasibilityCheck) covering:
- Strike counter isolation
- Stale auth/network failure flag immunity
- force=True bypass
- No-skip when no prior strikes
- Counter reset on session reset

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: TRON <tron-agent@agentmail.to>
2026-08-01 16:00:52 +05:30
carlotestor 13ad903a3c perf(redact): eliminate exponential backtracking in config-key patterns
_CFG_DOTTED_RE's nested quantifier (?:[A-Za-z0-9_\-]+\.)+ backtracks
exponentially on long non-matching dotted runs (doubles every ~4
segments). Flatten it and use possessive quantifiers (py3.11+) in
_CFG_DOTTED_RE and _YAML_ASSIGN_RE wherever the successor is disjoint.

Zero behavior change: equivalence fuzz-verified over 120k structured
and random inputs comparing full sub() output including groups. Adds a
ReDoS regression test.
2026-08-01 15:58:25 +05:30
kshitij 536754919d fix(auxiliary): replan cache sections on the async fallback path too
_call_fallback_candidate_sync replans messages/tools for each resolved
destination, but its async mirror still shipped the caller's decorated
sections verbatim — the primary destination's markers (including a
direct-native tool marker) leaked to fallback candidates with different
cache contracts, and the relay saw the display label instead of the
resolved provider/api_mode. Mirror the sync path: resolve the
destination, replan both sections, thread provider/api_mode through
_relay_async_completion, and replan again for the auth-refresh retry
client. Mutation-checked: the new parity test fails on the verbatim
pass-through shape.

Follow-up to #76032 (#20880).
2026-08-01 15:39:58 +05:30
kshitij e7340ea281 perf(prompt-caching): make PromptCachePlan.marker_count lazy
The count walked every message part and tool schema on every request but
is consumed only by tests. Compute it on demand via a property instead.

Follow-up to #76032 (#20880).
2026-08-01 15:39:58 +05:30
kshitij af06308425 refactor(prompt-caching): collapse triplicated destination-plan and label parsing
Three copies of the same logic landed with #76032:
- MoA's _call_prepared_aggregator and auxiliary_client's
  _replan_synchronous_cache_sections both implemented stub → policy →
  strip → plan for a resolved destination. Extract
  plan_cache_sections_for_destination() into agent_runtime_helpers (which
  already owns the policy functions) and route both through it. Also
  removes a redundant full-transcript deepcopy+strip per request (the
  caller pre-stripped what build_prompt_cache_plan strips again).
- The fallback_chain[N] label regex + chain-entry lookup lived in
  _fallback_entry_timeout AND _fallback_destination. Extract
  _fallback_chain_entry() and reuse.

MoA's cache-plan failure log is promoted debug → warning: the call-block
site skips MoA, so this block is the aggregator's only decoration path —
a silent failure ships an undecorated request (the 0%-cache MoA bug class).

Behavior-preserving; 195 targeted tests green.

Follow-up to #76032 (#20880).
2026-08-01 15:39:58 +05:30
kshitij 7ae4a5efba fix(prompt-caching): consolidate static-prefix split, guard empty volatile suffix
_apply_static_prefix_marker duplicated _apply_system_cache_markers' split
logic minus its empty-suffix guard: when the stored system prompt equals
the static prefix exactly, the tool-cache plan emitted a two-part split
with a trailing empty text block — HTTP 400 on native Anthropic. Fold the
tool-cache layout into the existing helper via mark_suffix /
fallback_to_whole flags; the empty-suffix case now marks the prompt as
one whole block. Behavior-parity verified against the merged planner for
every non-empty-suffix shape.

Follow-up to #76032 (#20880).
2026-08-01 15:39:58 +05:30
kshitijk4poor e078c8c6ef fix: widen fallback warning to the sibling custom-endpoint 256K path
Review pass 2 (reuse reviewer HIGH): the step-3b probe-down fallback for
custom/local endpoints returns the same silent 256K default but only
logged at INFO - invisible by default, and it is the MORE common path
for small local models (the exact users the warning exists for).

Extract _warn_context_length_fallback() (deduped per model+base_url)
and call it from both fallback sites, per the fix-the-whole-bug-class
rule. Regression test drives the custom-endpoint path and fails without
the widening (mutation-checked).
2026-08-01 15:05:05 +05:30
kshitijk4poor 4c2d0c7fd8 refactor: dedupe fallback warning per model, drive pool-cleanup tests through real run()
Review follow-up:
- Warn once per (model, base_url) at the step-9 fallback via a module-level
  dedup set (established _WARNED_* idiom). The fallback result is
  deliberately never cached, so the un-deduped warning fired on every
  resolution - e.g. once per gateway message via the session-hygiene path.
- Replace the three inline-mock pool-cleanup tests (which reproduced the
  try/except block against a MagicMock and passed even with the production
  code reverted) with a parametrized test that drives the real
  BatchRunner.run() with a patched Pool; drop the CPython stdlib
  signature change-detector test.
- Add a once-per-model warning regression test; clean up dead imports.

All tests verified to fail against pre-PR batch_runner.py/model_metadata.py
and pass with the fix (mutation check).
2026-08-01 15:05:05 +05:30
kshitijk4poor a1ff62a139 fix: context-length fallback logging, batch trajectory durability, pool cleanup
Salvage of #6629 by aaronlab (kshitijk4poor reworked against current main).

Three concerns from the original PR, reworked to address review feedback:

1. Context-length fallback diagnostic (agent/model_metadata.py):
   get_model_context_length() silently returned 256K when all 9 detection
   methods failed. Users with small-context models (8K, 32K) would get 256K
   silently, causing hard-to-debug API context-length errors. Added a
   warning log at the step 9 fallback with model name, base_url, and the
   correct config override hint (model.context_length, not context_length).
   The token-estimation ceiling-division fix from the original PR already
   landed on main (5c2ecdec) with CJK handling — not duplicated here.

2. Fsync for batch trajectory writes (batch_runner.py):
   Trajectory entries were written without flush/fsync, but the checkpoint
   immediately marked them as completed. A crash between write and disk
   sync would leave the checkpoint claiming completion with no trajectory
   data on disk. Added flush() + os.fsync() before checkpoint update.

3. Pool cleanup on interruption (batch_runner.py):
   Ctrl+C during pool.imap_unordered() relied on context manager cleanup
   which can hang. Added explicit pool.terminate() + pool.join() for both
   KeyboardInterrupt and Exception paths. The original PR used
   pool.join(timeout=10) which is invalid — CPython's Pool.join() takes
   no timeout parameter. Fixed to use pool.join() without arguments.

Tests:
  - test_warning_emitted_on_fallback: verifies warning fires at step 9
  - test_no_warning_when_cached: verifies no false warning when cache hits
  - test_trajectory_entry_is_synced_to_disk: verifies os.fsync is called
  - test_pool_terminate_called_on_exception: verifies cleanup on RuntimeError
  - test_pool_terminate_called_on_keyboard_interrupt: verifies cleanup on Ctrl+C
  - test_pool_join_called_without_timeout: verifies no timeout arg to join()
  - test_real_pool_join_accepts_no_timeout: integration check on CPython API

Co-authored-by: Aaron Lab <aaronlab@users.noreply.github.com>
2026-08-01 15:05:05 +05:30
Rod Boev 9fc12bf7a4 perf(prompt-caching): preserve tool-loop cache boundaries (#20880) 2026-08-01 14:27:12 +05:30
BB-light e9d52d2bda fix(caching): honor prompt_caching.cache_ttl disable in config
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.

The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused #56105 to be reverted in
#56126.

Salvage of #33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.

Co-authored-by: BB-light <BB-light@users.noreply.github.com>
2026-08-01 14:14:15 +05:30
Teknium d5463e5f6d fix(agent): invalidate flush-scan cursor at the defrag marker-pop sibling site
The micro-compaction defrag pass (_defrag_rolling_summary) rewrites the
newest MICRO marker's content and pops _DB_PERSISTED_MARKER from the
LIVE dict in place — the same in-place pop class finalize_turn's fill
site was fixed for in #75170. Without invalidation the bounded
flush-scan cursor identity-skips the rewritten marker row and the
defragged rolling summary never reaches state.db (resume rehydrates a
stale summary).

The compressor holds no agent reference, so the pop site raises
_flush_scan_cursor_invalidated and the finalize_turn micro-compaction
block consumes it, setting agent._db_flush_scan_prefix = None.

The module-scope pop sites (context_compressor.py:175/224) operate on
fresh copies — identity-breaking by construction — and need no flag.

Follow-up to #75170 (fix-the-class sweep of _DB_PERSISTED_MARKER
in-place pops).
2026-07-31 23:18:12 -07:00
spfcraze 2aaeee2ee5 fix(agent): invalidate flush-scan cursor when finalizer pops db marker
The bounded flush-scan in _flush_messages_to_session_db_unlocked skips
the identity-matched prefix of its previous snapshot, on the documented
assumption that no code path pops _DB_PERSISTED_MARKER from a live dict
in place. finalize_turn's pure-tool-call-tail fill is exactly that path:
it pops the marker so the filled content gets re-persisted — but the
cursor then skips the row anyway, so the delivered final response never
reaches state.db and /resume replays content="" (the #43849/#44100
class resurfacing via the perf cursor). Invalidate the cursor at the
pop site so the filled row is re-examined.
2026-07-31 23:18:12 -07:00
Teknium 3b9cf56aff fix(agent): exclude reasoning_details envelope from tail-budget walk (#73298)
Companion to the preflight fix: _estimate_msg_budget_tokens charged
reasoning_details at chars/4 via _REPLAY_BUDGET_KEYS, so the signed/base64
envelope (measured 72% of the reasoning mass on Anthropic-wire sessions)
consumed the tail budget and _find_tail_cut_by_tokens summarized away real
transcript to make room for tokens that are never sent (69 messages on the
measured session; up to ~4.8x budget inflation on thinking-heavy histories).

Per the #51800 counter-argument, actual thinking TEXT stays visible to the
budget: _reasoning_details_text_chars counts thinking/text/summary fields
and skips signature/data/encrypted blobs, and the text is skipped entirely
when reasoning/reasoning_content already carries the identical prose (so it
is charged once, not twice). codex_reasoning_items remains fully charged —
Codex Responses genuinely replays it every request (#55572).

Sabotage-verified: restoring reasoning_details to _REPLAY_BUDGET_KEYS fails
the new envelope and double-charge tests.
2026-07-31 23:16:58 -07:00
JonthanaHanh 530503a6a5 fix: exclude reasoning_details from preflight token estimate
The reasoning_details field (OpenRouter/Anthropic thinking blocks +
opaque cryptographic signature blobs) inflates the rough token estimate
by ~4x. Providers do not bill these envelope bytes as prompt tokens.

In a measured Kimi K3 session, reasoning_details held 2,124K chars
vs 281K chars of actual thinking text. The estimator reported ~533K
tokens when real prompt_tokens was ~140K — triggering compression at
~27% of the configured threshold.

Fix: skip reasoning_details in both _estimate_message_chars and
_estimate_message_tokens_without_images, alongside the existing
_anthropic_content_blocks exclusion.

Fixes #73298
2026-07-31 23:16:58 -07:00
Doud-FR 3127ddcb64 fix(agent): preserve a non-empty user query after compression 2026-07-31 23:16:58 -07:00
x7peeps a1f70343fd fix(agent): clamp tail-cut boundary and summary-scan indices to prevent IndexError
Fix #75588

## Root cause

When a short conversation ends in a tool-call/result group and the
protected head alignment reaches the end of the message list,
_find_tail_cut_by_tokens() could return len(messages) + 1. This
happened because the final return used max(cut_idx, head_end + 1)
which could push past the array length when head_end >= len(messages).

The out-of-range value then propagated into _find_context_summaries()
which iterated range(start, end) and indexed messages[idx] without
clamping, raising IndexError and failing the active gateway turn.

## Fix

Two-layer defense:
1. Source fix: _find_tail_cut_by_tokens() now clamps its return to
   min(n, ...) so it never exceeds len(messages).
2. Defensive clamp: _find_context_summaries() now bounds start/end
   to [0, len(messages)] so even if a future caller passes bad values,
   it cannot crash.

## Verification
- 7 new regression tests for the exact boundary conditions
- All 214 existing test_context_compressor.py tests pass
2026-07-31 23:16:58 -07:00
Xipong 9ceb0858ab fix(codex): defang reserved Harmony tokens in requests 2026-07-31 22:53:20 -07:00
wz-heng 6b6435a874 feat(cache): enable DeepSeek caching on OpenCode 2026-07-31 22:45:07 -07:00
Israel Lot 5c45d9c208 fix(agent): mirror substitute_api_content's guard in the estimator shadow
Review follow-up on #75102. The shadow substituted the sidecar whenever
the ``api_content`` key was merely PRESENT, but the wire only substitutes
a non-empty string sidecar on a user/assistant row (see
``turn_context.substitute_api_content``). For any other shape the sidecar
is popped and discarded while the clean ``content`` is sent -- so the
shadow dropped real content from the estimate and UNDERcounted, the
dangerous direction: compaction fires too late and the turn dies on a
hard context-length error instead of merely compressing early.

Gate the substitution on the same predicate, and cover the divergent
shapes (None, empty string, int, list, non-user/assistant role) with a
test that fails against the unconditional version.

Also rename the image test: it never carried a sidecar, so it was not
testing what its name claimed. It is a non-regression pin on the flat
per-image accounting that moved into ``_wire_message_shadow()``, and is
now named for that.
2026-08-01 11:10:15 +05:30
Israel Lot e3bc517034 fix(agent): stop double-counting api_content in the token estimator
`api_content` is a SUBSTITUTE for `content`, not an addition to it.
`turn_context.substitute_api_content()` pops the sidecar and overwrites
`content` at every API-bound message-build site (the `api_messages` build
in `conversation_loop`, the max-iterations summary in
`chat_completion_helpers`, the chat-completions transport), so exactly one
of the two is ever sent to the provider.

The preflight estimator counted both, because both `_estimate_message_chars`
and `_estimate_message_tokens_without_images` walked every key of the
persisted dict with a single-entry denylist (`_anthropic_content_blocks`).
Any message whose sidecar differs from its clean stored content was counted
twice — exactly 2.00x on a 40KB sidecar.

The sidecar exists to keep the provider prompt-cache prefix byte-stable, so
it is written on precisely the long, cache-pinned messages where the
doubling hurts most. Because `estimate_messages_tokens_rough()` also feeds
the compaction threshold via `context_compressor` and `conversation_loop`,
the inflated estimate makes compression fire on phantom bytes.

Fix: substitute rather than sum, mirroring the wire. The two estimator
helpers had drifted into near-identical copies of the same shadow-building
loop, so this factors the shared logic into `_wire_message_shadow()` and
fixes the class once instead of patching one site and leaving the other.

Image accounting is unchanged: base64 payloads are still replaced with a
placeholder and charged at the flat `_count_image_tokens` rate, and the
`_multimodal` text_summary path is preserved.

Tests: three cases in `TestEstimateMessagesTokensRough` — sidecar equal to
content is counted once, a sidecar that DIFFERS is still counted (a lower
bound, so it fails if the field were dropped rather than substituted, which
would undercount the real request), and a sidecar cannot smuggle raw base64
past the flat image rate.

Verified on Linux (Python 3.11): 53 passed in
tests/agent/test_model_metadata.py, 57 passed with
tests/agent/test_context_breakdown.py, 656 passed / 3 skipped across the
compression/context/token/estimate/prune surface of tests/agent.
Mutation-tested: reverting the substitution fails the new equality test.
`scripts/check-windows-footguns.py` is not applicable — no file I/O,
process management, terminal handling, subprocesses, or signals.
2026-08-01 11:10:15 +05:30
brooklyn! c74f4c5335
Merge pull request #75890 from NousResearch/bb/disk-full-toast
Toast when a send fails because the disk is full
2026-08-01 00:36:21 -05:00
Teknium d1cdfcd38a
Merge pull request #74348 from JoaoMarcos44/fix/ia-03-codex-post-terminal-retry
fix(codex): stop duplicating billed inferences on post-terminal drain errors
2026-07-31 22:35:44 -07:00
joaomarcos 854007d1c3 fix(auth): route remaining main-agent fallback key reads through secret_scope
agent_init.py's init-time fallback and chat_completion_helpers.py's
try_activate_fallback() still read key_env via raw os.getenv(), missing the
per-profile secret scope installed by the multiplexed gateway (same bug
fixed for fallback_config.py/auxiliary_client.py in this PR). Both now
delegate to hermes_cli.fallback_config.resolve_entry_api_key(), and the
Ollama Cloud OLLAMA_API_KEY read now goes through
agent.secret_scope.get_secret() too.

agent_init.py's fallback loop had no try/except around key resolution
(unlike the other three call sites), so a fail-closed UnscopedSecretError
under multiplexing would have crashed init instead of skipping to the next
fallback entry — added the same skip-and-continue handling.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-31 22:35:26 -07:00
joaomarcos d52a1c25e0 fix(auth): resolve fallback api keys through secret_scope, not raw env
resolve_entry_api_key() and the duplicated _fallback_entry_api_key()
read key_env via a raw os.getenv(), bypassing per-profile secret
scoping in the multiplexed gateway. Under multiplexing this can hand
a fallback request another profile's credential. Both now resolve
through agent.secret_scope.get_secret(), which reads the active
profile scope when multiplexing is on and falls back to os.environ
unchanged when it's off, so single-profile behavior is preserved.

Closes #74311
2026-07-31 22:35:26 -07:00
MaxFreedomPollard 65e9ece964 fix(curator): restore the real skills tree when a rollback extract dies part-way
shutil.move() moves into an existing destination directory instead of
replacing it. When the snapshot extract failed after creating some of its
output, the recovery path moved each staged entry onto a path the extract
had already created, burying the user's own skill one level deeper
(skills/alpha/alpha/) and leaving the snapshot's partial content in its
place. rollback() then returned "snapshot extract failed (state restored)".

Clear the failed extract's output before moving the staged copies back:
entries the original tree never had are dropped, and each staged entry's
destination is removed before the move. When an entry still cannot be
restored, keep the staging dir and name the entries in the message rather
than reporting a restore that did not happen.
2026-07-31 22:34:57 -07:00
Zeraphim 2ace68ad37 fix(background-review): reject unresolved failures as skills 2026-07-31 22:33:03 -07:00
Xipong 0af8fb05bf fix(delegation): prevent child HERMES_SESSION_ID leak into parent process env
AIAgent.__init__ calls set_current_session_id(self.session_id), which
mutated both the task-local ContextVar and the process-global os.environ.
Because _build_child_agent wraps construction in delegated_child_context(),
the ContextVar write is harmless (task-local), but the os.environ write
clobbered the parent's HERMES_SESSION_ID for the rest of the process —
leaking the child id into parent tools and subprocesses spawned after
the child was built.

Root cause of HermesPRDelegationSessionContext: parent
20260729_212118_5d797e dispatched child 20260730_160515_736ea1; later
parent terminal inherited HERMES_SESSION_ID=the child.

Fix: set_current_session_id() skips the process-global os.environ write
when called from within a delegated_child_context(). The child's own
tools and subprocesses still resolve their id through the ContextVar
(task-local), while the parent's process-wide env keeps the parent's
session identity. Root agents (CLI, gateway, cron) retain both paths.

Adds 7 regression tests covering single child, concurrent children (8
parallel), parent-tool observation after construction, and root-agent
session rotation backward compatibility. All pass; ruff clean.
2026-07-31 22:32:55 -07:00
Xipong 0fd0db1a8c fix(agent): preserve /steer through turn-budget enforcement 2026-07-31 22:32:52 -07:00
praneshnikhar 4e6299af48 fix(credential_pool): use source-path-based write-through to root (#74339)
_sync_device_code_entry_to_auth_store used key-presence on the profile
store to decide whether to write-through rotated tokens to the global
root.  _store_provider_state unconditionally creates that key, so every
refresh after the first self-disabled the write-through — root kept a
revoked refresh token and every other profile died with
refresh_token_reused / invalid_grant.

Fix: use _load_provider_state_with_source to learn where the grant was
resolved from.  When the source is the global root, write back only to
root and skip _store_provider_state so the profile never accrues a
shadowing providers.<id> key that blocks future root fallback.

Add regression test verifying write-through fires on refresh 2+, not
just the first call.
2026-07-31 22:32:48 -07:00
Gille 29eac371d1 fix(context): persist NVIDIA DeepSeek endpoint limit 2026-07-31 22:31:22 -07:00
Teknium 1737741730 fix(copilot): follow-ups for salvaged PR #58743
- Bound ALL reads of the on-disk JWT store through one _read_jwt_store()
  helper (load, eviction, save-merge) — the 1 MiB cap previously only
  covered the load path; eviction and save could still parse an
  oversized/corrupt store and rewrite it back out (sweeper finding).
- Fix the class, not the site: the recovery gates checked the literal
  provider == "copilot" while /model and profile configs can leave the
  alias spelling in place (the reporter's own log shows provider=copilot
  AND provider=github-copilot in one session — the aliased turns would
  have silently skipped recovery). Single owner:
  AIAgent._is_copilot_provider() (slug aliases + Copilot base-URL
  fallback), used by both run_agent recovery methods and both
  conversation_loop gates.
- Update the salvaged 401 test to current main's client-retirement
  contract (release deferred to GC — no synchronous .close()).
- Add copilot_stale_cred_retry_attempted to the TurnRetryState field
  contract test; add bounded-store and alias-gate regression tests.
2026-07-31 22:31:09 -07:00
dstkwll 7779409a76 fix(copilot): recover from stale/degraded token 400 AND expired IDE-token 401
Copilot degrades in two related ways that both abort a turn as non-retryable
and only clear on a gateway restart (a cold process re-runs the token exchange):

1. HTTP 400 model_not_available_for_integrator / model_not_supported — a
   raw/degraded token routes to the restricted copilot-language-server
   integrator whose allowlist omits enterprise-only models (e.g.
   claude-opus-4.8). Because it is a 400 (not 401), the existing 401 refresh
   path never fired. Prevented (retry-with-backoff exchange + on-disk JWT
   persistence + header guard at the client chokepoint) and self-healed at
   runtime (single-shot forced re-exchange + client rebuild + retry before
   fallback).

2. HTTP 401 'IDE token expired: unauthorized: token expired' — the short-TTL
   *exchanged* IDE token expires mid-turn. The clean-401 path DID fire and call
   _try_refresh_copilot_client_credentials(), but that method only re-resolved
   the stable raw ghu_ token and rebuilt the client — it never evicted the
   cached exchanged JWT or forced a fresh exchange, so the retry put the SAME
   expired token back on the wire, 401'd again, and the single-shot guard
   aborted the turn. Fix: force a fresh IDE-token exchange (evict cached JWT via
   evict_cached_exchanged_token + re-mint via get_copilot_api_token) before the
   client rebuild, mirroring the merged auxiliary-path recovery (#59837) and the
   400 recovery in this same PR. Graceful fallback to the resolved token if the
   exchange endpoint is unreachable; picks up the enterprise base_url on
   re-exchange.

Brings main-loop clean-401 recovery to parity with the merged auxiliary path
(#59837), using the newer on-disk-aware evict helper. Companion context: #58743
(this PR, expanded), #51313, #63204 (which assumed the 401 path already
recovered — it reached the method but the method was too weak).

Tests: exchange retry/persist round-trip, restart-blip disk reuse, stale-cred
400 classifier, 400 recovery, and 3 new 401 cases (fresh exchanged token on the
wire; network-blip fallback to resolved token). 58 copilot tests green on
current main.
2026-07-31 22:31:09 -07:00
Brooklyn Nicholson 24f346ee77 fix(gateway): fail prompt.submit when session storage hits a full disk
Disk-full / ENOSPC / SQLITE_FULL on first-message session persist used to be
swallowed as a debug log while prompt.submit still returned streaming, so the
send vanished with no error. Re-raise those failures, return a real RPC error,
and stamp session_persistence_failed turns with error so clients get a terminal
error frame.
2026-08-01 00:27:17 -05:00
Kyle 48ded07155
fix(codex): handle completed auxiliary responses 2026-07-31 21:52:36 -07:00
knocksen 743c4906d9
fix(moa): restore a valid preset after fallback model drift
build_moa_facade() reused agent.model as the preset name; a session
that had drifted to a fallback model crashed on restore with
MoAPresetNotFoundError. Validate the resolved preset against the
configured presets and fall back to the default preset.

Salvaged from PR #74903 by @liusencomic-cyber.
2026-07-31 21:52:36 -07:00
knocksen 716d274f58
test(moa): cover direct provider streaming for Codex-shim aggregators
Narrow the moa_aggregator Relay bypass to CodexAuxiliaryClient and add
coverage that call_llm(stream=True) returns the provider's direct
create() result for Responses-shim clients.

Salvaged from PR #74903 by @liusencomic-cyber.
2026-07-31 21:52:35 -07:00
knocksen 8e191af0fb
fix(moa): stream completed aggregator responses safely
Convert a completed MoA aggregator response into one valid Chat
Completions delta chunk at the MoA facade boundary, normalize completed
message.tool_calls into indexed stream deltas, and classify these local
MoA adapter-shape errors as non-fallback format errors so a local
compatibility bug cannot silently drift the user's MoA route to a
single model (#55933 follow-up).

Salvaged from PR #74903 by @liusencomic-cyber.
2026-07-31 21:52:35 -07:00
AKAZIK-py c23ff21d7c
fix(moa): carry completed responses through the managed Relay path
Under managed Relay execution the provider factory runs lazily inside
provider_stream() on the Relay session's event loop. The MoA facade's
auxiliary call_llm(stream=True) is invoked from that callback, so the
eager final_response check added for the non-managed path never fires:
the inner ManagedLlmStream is returned to the outer stream, which then
synchronously iterates it on the same loop thread and dies with
RuntimeError: Cannot run the event loop while another loop is running
(the completed response effectively trapped one level deeper).

stream_current() now detects a running event loop and returns the raw
factory result instead of nesting a ManagedLlmStream: the outer managed
stream already provides Relay tracking for the enclosing attempt, and
its own completed_response_predicate traps the completed response as
final_response — the same contract the main streaming worker consumes
(chat_completion_helpers reads stream.final_response after the chunk
loop). Nested managed streams remain supported for genuinely streaming
providers via the outer stream's own iteration.

Adds managed-execution regressions using the retained relay_turn
fixture: direct completed-response trapping, and the nested
facade-shaped stream_current call.
2026-07-31 21:52:35 -07:00
AKAZIK-py 654915f187
fix(moa): unwrap completed responses in the auxiliary streaming path 2026-07-31 21:52:35 -07:00
teknium1 dc87d15586 feat(terminal): raise Docker sandbox /dev/shm to 1g by default (configurable)
Port from nanocoai/nanoclaw#2748: Docker's built-in 64 MB /dev/shm silently
breaks shared-memory-hungry workloads inside the sandbox — Chromium/Playwright
renderers crash tabs, and PyTorch DataLoader workers die with 'bus error' /
'insufficient shared memory'. tmpfs is lazily allocated, so the higher ceiling
costs nothing until actually used, and usage still counts against the
container's --memory cgroup limit.

- tools/environments/docker.py: --shm-size 1g in resource args (not
  cgroup-gated; tmpfs mount option). Skipped when docker_extra_args already
  sets --shm-size, or when configured empty/'0' (Docker default).
- terminal.docker_shm_size config key (DEFAULT_CONFIG + all three
  config->TERMINAL_DOCKER_SHM_SIZE env bridges: CLI, gateway, config.py map)
- tests: default emit, custom value, opt-out, extra_args precedence,
  helper edge cases (sabotage-verified: default/custom tests fail without
  the emit)
2026-07-31 21:31:51 -07:00
teknium1 950fe236d0 fix(security): extend secret redaction to GitLab token families
Port from openclaw/openclaw#112954. The redactor knew GitHub, Slack,
Google, Stripe, AWS access-key-ID and ~25 other vendor prefixes but had
zero GitLab coverage — glpat-/gloas-/gldt-/glrt-/glrtr-/glcbt-/glptt-/
glft-/glimt-/glagent-/glsoat-/glffct-/glwt- tokens and legacy GR1348941
runner registration tokens passed through display and log surfaces
verbatim. Follow-up explicitly invited when #4541 was closed.

Each pattern keeps a full literal prefix so the _PREFIX_SUBSTRINGS
pre-screen (derived at module load) stays false-negative-free; routable
runner tokens allow dotted segments. Sibling site: skills_guard's
credential-exposure scan gains a gitlab_token_leaked pattern.
2026-07-31 21:31:10 -07:00
Alex Fournier e3d10f0a8f Merge current tool metrics into skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:54:38 -07:00
Alex Fournier 1318c213b0 Merge current model route metrics into tool metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:54:13 -07:00
Alex Fournier 0de9b65c23 Merge upstream main into model route metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:53:12 -07:00
Alex Fournier 3d5fcab70f Merge updated tool metrics into skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	hermes_cli/observability/schemas/hermes.shared_metrics.v1.schema.json
#	scripts/smoke_nemo_relay_shared_metrics.py
#	tests/agent/test_skill_commands.py
#	tests/hermes_cli/test_relay_shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics_runtime.py
#	tests/tools/test_skill_manager_tool.py
#	tests/tools/test_skill_usage.py
#	tests/tools/test_skills_tool.py
2026-07-31 07:30:30 -07:00
Alex Fournier aef76ba398 Merge updated model metrics into tool metrics
# Conflicts:
#	hermes_cli/observability/shared_metrics_contract.py
#	hermes_cli/observability/shared_metrics_subscriber.py
#	scripts/smoke_nemo_relay_shared_metrics.py
#	tests/hermes_cli/test_plugins.py
#	tests/hermes_cli/test_relay_shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics_runtime.py
#	tests/run_agent/test_run_agent.py
#	tests/test_model_tools.py
#	tests/tools/test_approval.py
2026-07-31 07:18:02 -07:00
rob-maron 126ff7071b
Portal free user vision fix + flux3 polling improvements (#75448)
* flux3 polling improvments

* poll gap to 4s

* back to 5s

* vision model fix

* minor fix
2026-07-31 10:17:55 -04:00
Alex Fournier 0558ea0c48 Merge upstream main into feat/hermes-relay-model-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:04:37 -07:00
Alex Fournier 43d29a37c8 fix(observability): include auxiliary model routes
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 06:58:41 -07:00
kshitijk4poor 53559aaf86 fix(agent): protect batch-compaction markers from micro supersede/defrag
Phase 2 review findings on the salvage branch:

C1 (critical): batch and micro summary markers share
COMPRESSED_SUMMARY_METADATA_KEY, and compress() never reset micro state.
After micro absorbed exchanges 1..k, a batch compaction summarizing
1..m (m>k) could fire; the next micro pass's supersede then dropped the
batch marker (whose content the stale rolling summary does NOT contain)
and archive_and_compact immediately made the loss durable. Defrag had
the same hazard: it rewrote "the newest marker" even if that was a
batch marker. Empirically confirmed with a probe (batch marker content
destroyed in one pass).

Fix, three parts:
- Micro-created markers now carry MICRO_COMPACT_MARKER_KEY; supersede
  and defrag only ever touch micro-tagged markers. Rehydration in
  _resolve_compact_cursor tags the marker it absorbs (containment
  proof), which safely covers adopting a batch marker as the new
  rolling base after a reset.
- compress() success path resets micro rolling summary/cursor state so
  a stale summary can never claim cumulativeness over a batch marker.
- Regression tests for both directions plus the reset.

W4: _splice_micro_compact_result no longer strips _db_persisted stamps
from surviving messages. Micro archives in place under the SAME session
id (unlike batch's child-session rotation, #57491), so surviving stamps
are accurate; stripping them meant an archive_and_compact failure left
every previously-persisted message unstamped and the next append-only
flush re-inserted them all as duplicate active rows.

W5: finalize_turn micro gate now checks agent._persist_disabled —
persistence-isolated fork agents (background review) must not burn an
aux call per review turn, and must never archive_and_compact the
canonical session rows if their compressor ever gains a DB binding.

W1: _serialize_one_exchange now delegates to _serialize_for_summary
(was a ~70-line near-verbatim copy; one serializer, one place to fix).

S4: _find_one_exchange boundary guard rejects only assistant/tool
boundaries (the actual alternation hazard) instead of requiring user —
a stray mid-list system/injected message can no longer wedge the
cursor forever.

5 new regression tests; 38 micro/prune tests, 400 compression-suite
tests, 61 finalize/persist tests pass; ruff clean.
2026-07-31 17:44:19 +05:30
kshitijk4poor c696a5fd9c fix(agent): harden the finalize-turn micro-compaction gate against duck-typed compressors
tests/run_agent/test_proactive_prune_loop_wiring.py builds agents with a
MagicMock compressor; getattr(mock, '_micro_compact_enabled', False)
returns a truthy auto-attribute, so the hook called _micro_compact on the
mock and spliced its (empty-iterating) return over the transcript —
wiping all messages before persist (CI slice 7/8 failure).

Gate now requires _micro_compact_enabled is True, a callable
_micro_compact, and a non-empty list result before touching messages.
Same hardening protects production plugin context engines that don't
subclass ContextCompressor.
2026-07-31 17:44:19 +05:30
kshitijk4poor b8bfd68af1 fix(agent): make micro-compaction alternation-safe and defrag user-preserving
Two integration bugs found during review of #74522, both confirmed with
empirical probes against the production message-repair path:

1. Alternation: the summary marker was role="user" and an exchange was a
   single assistant+tools group, so splicing between two user turns produced
   user -> marker(user) -> user. The pre-request repair_message_sequence pass
   (conversation_loop.py, runs before EVERY API call) then merged the marker
   into the neighbouring real user message: metadata gone, cursor
   unrecoverable on resume, and the summary text duplicated into the
   transcript on every later pass (the transcript GREW every turn).
   Fix: an exchange is now a full agent turn (assistant + tools + follow-up
   assistant iterations, bounded by user messages), the marker is
   assistant-role, and superseding an old marker deliberately merges the two
   adjacent real user turns (plain-text \n\n-join, identical to repair
   pass 2) so the returned transcript is alternation-valid by construction.
   Probe result: repairs 0 (was 2), marker survives, no summary leakage.

2. Defrag destroyed user messages: _defrag_rolling_summary serialized the
   whole remaining middle (user turns included) and spliced it away —
   8 of 10 user prompts destroyed in one pass, contradicting the feature's
   "your messages are never compacted" invariant. Fix: defrag now
   re-summarizes only the rolling summary TEXT and rewrites the marker
   content in place; transcript shape, cursor, and user turns untouched.
   Probe result: 10 of 10 user prompts survive.

Also: marker provenance is now COMPRESSED_SUMMARY_HAS_USER_TURN_KEY=False —
micro markers absorb only assistant/tool content (#64650 invariant), and
real user turns remain in the transcript for provenance detection.

Adds 5 regression tests (repair-pass integration, alternation on
multi-iteration tool turns, defrag user survival, defrag input scope,
marker provenance); updates the two existing tests and the design doc to
the corrected semantics. 28 tests pass.
2026-07-31 17:44:19 +05:30
Michael Jordan 9ca4ee72ca feat(agent): make the micro-compaction cadence configurable
The on/off switch was the only knob. A pass fired after every completed turn,
absorbed exactly one exchange, and there was no way to ask for less. Since a
pass is also what breaks the prompt-cache prefix, "how often does it run" and
"how often do I pay a cache break" are the same question, and it had no answer.

Add `compression.micro_compact_every_n_turns` (default 1, clamped to >= 1). At 1
the behaviour is what it was; at 5 you get a fifth of the breaks and a fifth of
the reclaim rate. The counter advances per invocation rather than per committed
pass, so a turn that finds nothing to absorb still moves the cadence along and
cannot wedge it, and a bogus 0 or negative degrades to "every turn" instead of
silently disabling compaction.

Also expose `micro_compact_defrag_threshold_tokens`, which has been a hardcoded
attribute on the compressor with no path from config since it was added.

This does not give micro-compaction the prune's reclaim-size gate -- a pass
still commits whatever the single absorbed exchange saved. It makes the break
frequency tunable, which reaches the same end by absorbing less rather than by
waiting for a bigger win. The docs now say that plainly, including that a
reclaim threshold is the obvious follow-up and does not exist yet.

Tests cover the skip-until-due window, the cursor and prefix staying untouched
on skipped turns, the clamp, and that the feature is off unless enabled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan d19c182879 fix(agent): ship micro-compaction opt-in, not default-on
Review raised whether default-on can be reconciled with the prompt-cache
contract in AGENTS.md, which permits mutating past context only for context
compression and treats per-conversation caching as sacred. It cannot, and the
codebase already says so in its own words.

A micro-compaction pass rewrites already-sent history, so it invalidates the
cached prefix every turn rather than at an episodic boundary. That is the exact
cost the proactive prune gates against: `proactive_prune_min_reclaim_tokens`
exists, per its own config comment, to keep rewrites to "one big episodic break
instead of a tiny break every tool iteration." Micro-compaction has no
equivalent gate -- one exchange per turn means one break per turn, by design.

Default to off. An operator who wants the amortized stall can opt in with
`compression.micro_compact: true` and accept the tradeoff knowingly; nobody
inherits a per-turn cache break from installing an update.

Also register the key in config_defaults so it is discoverable and picked up by
the update path's new-options check -- it was previously read by agent_init but
declared nowhere -- and document the cache cost in docs/micro-compaction.md
instead of only the benefit. The measurements behind the feature (occupancy
plateau, zero batch compactions) never priced cache invalidation, and the doc
now says which numbers a reader would need to measure to justify enabling it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan 60781a0cc8 fix(agent): do not destroy compacted history when a session resumes
The rolling summary lives only in memory. A resumed session starts with an
empty one while the marker carrying every previously absorbed exchange is
still in the transcript. The first pass after a resume therefore built a
marker from a single exchange and superseded the marker holding the entire
history -- silently discarding everything micro-compaction had accumulated.

This was introduced by the supersede fix. Before it, markers piled up
wastefully, but nothing was ever lost.

Two changes, so a single failure cannot lose data:

Rehydrate. When the cursor is recovered by scanning the transcript -- the
resume path -- also recover the rolling summary from that marker, so the
next pass merges into the existing history instead of replacing it.
Extraction uses rfind for the heading because SUMMARY_PREFIX references the
heading text itself, so the first occurrence is inside the preamble.

Gate superseding. Earlier markers are dropped only when this pass's summary
is demonstrably cumulative, i.e. the rolling summary was non-empty going in.
If rehydration ever fails, the pass keeps both markers: wasteful, but the
history survives.

Tests cover the resume path, the failed-rehydration fallback, and the
round trip of a summary through a marker.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan d626528560 fix(agent): derive the micro-compaction cursor from the spliced list
The cursor was set to the pre-splice `exchange_end`. A splice collapses the
absorbed span -- an assistant plus its tool results, often four or more
messages -- into a single marker, and may also drop a superseded marker
further back, so every index after it shifts.

The stale cursor therefore overshot, landing inside a *later* exchange's
tool group. The next pass's `_find_one_exchange` walked forward from there
to the following assistant, so the exchange it had landed inside was never
absorbed at all. On tool-bearing conversations micro-compaction was
silently doing roughly half the work it should.

Traced on a 3-tool-per-exchange transcript: the cursor sat at index 6 when
the marker was at 2, and the message count stalled at 32 instead of
continuing to 28.

Derive the cursor from the marker's actual position in the spliced result
instead, which is self-correcting regardless of how much the splice moved.
Apply it on the defrag path too, which had the same staleness.

Found by a randomized long-horizon harness (480 conversation shapes x 25
passes, varying tool-group sizes and summarizer failure modes) asserting
structural and progress invariants after every pass. Existing tests missed
it because their fixtures have no tool results, so the absorbed span is one
message and nothing shifts. The regression test uses tool groups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan ac48add3a7 feat(agent): report context occupancy, not just tokens saved
Tokens saved is the wrong headline for this feature. Micro-compaction is
not an efficiency optimisation — the same summarization work happens either
way. What it buys is (a) that work amortized across turns instead of one
stall, and (b) a window kept low enough that a session runs much further
before needing a hard compaction at all.

Neither shows up in "net tokens saved". A session can save nothing on paper
and still be a clear win on both counts.

So the telemetry now carries occupancy: tokens_after as a share of the
compaction threshold, plus the threshold and resolved window it was
computed from. That is the number that says whether a session has headroom
left. The report leads with it, and cross-references the batch
`compression_attempt` lines already in the log so it can show how often the
long pause actually fired — ideally never.

Occupancy is read from the cached threshold only. The public
`threshold_tokens` property resolves lazily and can issue a synchronous
/models probe (#32221); telemetry must never be the thing that blocks a
turn, so an unresolved window reports null. In practice a pass has already
resolved it via the tail calculation, so the field is populated. A test
pins the no-forcing behaviour directly against the emitter.

The report is pure ASCII: `scripts/check_subprocess_stdin.py` currently
dies on a cp1252 console before printing its results, and a diagnostic tool
that crashes on the platform it is diagnosing is worse than no tool.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan cac9526d2a feat(agent): token telemetry for micro-compaction
The existing log line reports message counts, which is the least
informative number available here: absorbing one tool-heavy exchange can
drop hundreds of tokens while moving the count by one. There was no way to
answer "is this actually helping?" from a real session.

Emit one content-free JSON line per pass, in the same shape as the batch
compaction telemetry: before/after tokens, the delta, the size of the
absorbed exchange, the rolling summary size, duration, and running
per-session totals so a whole run can be read off the last line. No
transcript content rides along.

Add scripts/micro_compaction_report.py to aggregate those lines into
passes, outcome mix, net tokens saved, mean exchange size and durations,
with an optional per-session breakdown.

Measuring it immediately surfaced something worth documenting: the first
pass in a session normally *costs* tokens. The summary marker carries a
fixed ~400 tokens of scaffolding, paid on pass one against a single
absorbed exchange. From pass two the marker is replaced rather than added,
so the overhead is already paid and each exchange is close to pure saving.
Break-even is typically the second or third pass. Tests cover the
telemetry contract, the cumulative totals, and that first-pass/later-pass
shape so nobody reads a single turn and concludes it made things worse.

The estimator costs ~5 ms at 600 messages and ~20 ms at 1200, taken twice
per pass, post-turn — and only once an exchange is actually in hand, so
turns that no-op early pay nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan a72f898d0f docs(agent): drop a fabricated issue reference
The `_micro_compact` docstring cited "#82483" for the resume double-load
problem. No such issue exists — the repository's highest number is 74323,
so the reference was invented rather than looked up.

The reasoning it was attached to is correct and stays: the session flush is
append-only, so an in-memory splice alone leaves the original rows active
and a resume loads both the summary and the messages it replaced. Only the
citation was wrong.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan 214b5d8612 docs(agent): state that user turns are never micro-compacted
`_find_one_exchange`'s docstring described an exchange as "(optional) user
message + assistant message + its tool results", but the walk skips past
user messages and starts at the assistant, so user turns are never absorbed
into the rolling summary.

The code is right and the docstring was wrong. Assistant output is largely
an account of what was done and survives summarising with little loss. The
user's messages are the intent everything else is derived from and cannot be
reconstructed from the work that followed — paraphrasing "use the existing
helper, don't add a new one" into a summary is how an agent ends up doing
the opposite six turns later. They are also cheap: a prompt is normally a
tiny fraction of what one tool result costs.

Correct the docstring, document the property (and its cost — a floor on how
small the middle can get, since user turns accumulate), and add a test so it
stays deliberate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan 186cad02f9 feat(agent): per-turn micro-compaction to amortize context compression
Batch compaction pauses a session for one large summarization once the
window fills. Micro-compaction spreads that cost out: after each completed
turn, `finalize_turn` folds the single oldest un-absorbed exchange
(assistant message plus its tool results) into a rolling summary, so the
work happens in small increments during post-turn idle time instead of one
long stall.

Mechanics:
  - a cursor tracks the first message not yet absorbed, recovered from the
    transcript's last summary marker when in-memory state is unavailable;
  - protected head and tail windows are never touched, so the system prompt
    and recent turns stay verbatim;
  - the absorbed span is replaced by a marker carrying the usual
    `_compressed_summary` metadata, so resume, handoff and `/compress`
    treat it exactly like a batch summary;
  - `archive_and_compact` keeps the session DB in step, otherwise the
    append-only flush would leave the original rows active and a resume
    would double-load both summary and originals;
  - when the rolling summary itself passes a token threshold it is
    defragged: re-summarized in one shot and the cursor jumps to the tail;
  - an exchange the summarizer can't handle is retried a bounded number of
    times, then skipped, so one poison exchange can't stall every turn.

Keep only the newest summary marker. The rolling summary is cumulative, so
each marker already contains everything the previous ones held; leaving them
stacked near-duplicate copies of the same text, each with its own heading and
end-marker scaffolding, and the transcript grew on every turn instead of
shrinking. Measured over six turns on a 12-exchange conversation with tool
output: 4104 -> 4797 tokens before, 4104 -> 2572 after.

Off switch: `compression.micro_compact: false` (default on).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
joaomarcos b0785fc1a1 Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts:
#	tests/run_agent/test_run_agent_codex_responses.py
2026-07-31 02:43:12 -03:00
rob-maron 4c7cc62f9f flux3 messaging system fixes 2026-07-30 15:20:09 -07:00
Matt Ezell 7965462d6c fix(compression): choose summary role by template-visible alternation
The compaction summary's role was selected against the LITERAL
neighbouring messages (compressed[-1] / tail_messages[0]). Mistral-family
chat templates (Devstral, Mistral Small 3.x, Magistral) enforce
user/assistant alternation but exempt the tool flow (tool results and
assistant messages carrying tool_calls) from the check, so a protected
head ending [user, assistant(tool_calls), tool] pinned the summary to
role="user" while the last role the template counts is "user": the
backend rejects the whole request with a Jinja alternation error
(HTTP 500). The summary persists in the stored conversation, every
retry replays the identical poisoned history, and the session is
permanently unrecoverable. Fires on EVERY compaction against a
Mistral-strict backend, captured byte-exact via a tee-proxy in front of
a llama.cpp/llama-swap Devstral deployment.

Fix: compute both neighbour roles through _template_visible_role(),
which skips template-exempt messages. The #52160 (Anthropic user-first)
and #58753 (zero-user-turn) forced-user guards are preserved; their
forced shapes (summary-user followed only by exempt messages) are
alternation-safe. When the visible head ends "assistant" and the
visible tail opens "user", no standalone role can alternate and the
existing merge-into-tail fallback now correctly fires (the literal
logic emitted a standalone user summary there: a second poisoning
shape).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGN45sMMbwM8cW9T9ga4ou
2026-07-30 21:07:36 +05:30
Brooklyn Nicholson 7d92056c49 feat(gateway): iMessage-style message reactions — storage, RPC, agent tool, model context
Reactions live in the existing messages.display_metadata JSON column (no new
table), with iOS Tapback semantics enforced DB-side: one reaction per author
per message, re-tap retracts, different emoji replaces. The desktop catches up
to the reaction contract five platform adapters already ship.

- SessionDB: set/get_message_reaction, latest_message_row_id (role + offset +
  require_text so invisible tool-call-only rows are never targeted),
  take_unseen_reactions (announce-exactly-once), get_message_role
- message.react RPC: accepts row_id or newest_role for live messages that
  haven't learned their durable id yet
- react_to_message tool: desktop-gated (check_fn), defaults to the user's
  latest visible message, messages_back for retroactive reactions
- Model context rides run_message only (beside the speech-interrupted note):
  the persisted prompt stays clean, so no [The user reacted …] scaffolding in
  transcripts, and no cached prefix ever changes
- Resume projection forwards row_id + reactions; _row_id is stripped from
  outgoing API copies next to display_metadata
2026-07-30 00:08:28 -05:00
Teknium c770515e2b modernize re-added Vercel integrations: SDK 0.7.2, telemetry off, sibling-site wiring
- Bump vercel SDK pin 0.5.7 -> 0.7.2 (pyproject, lazy_deps) and regenerate uv.lock
- Disable the SDK's new default-on telemetry (VERCEL_TELEMETRY_DISABLED=1
  set before import, user-overridable) per the no-opt-out-telemetry policy
- Move _model_flow_ai_gateway into hermes_cli/model_setup_flows.py (god-file
  decomposition landed after the removal)
- Widen post-removal backend sets that vercel_sandbox missed: terminal_tool
  container_backend + _CONTAINER_BACKENDS, file_tools fallback set,
  env_probe._REMOTE_BACKENDS, approval._should_skip_container_guards,
  prompt_builder probe container_config
- Add terminal.vercel_runtime to config_defaults + TERMINAL_CONFIG_ENV_MAP
- Re-add vercel dependency group to nix #full variant (reverts #33773 workaround)
- Update restored tests to current contracts: upload-only credential sync-back
  (bcfc7458fa), registry-derived provider env list, parametrized backend fixture,
  drop tests superseded on main (slack wizard move #41112, nous status format)
2026-07-29 19:48:37 -07:00
Teknium ad12df6ba4 Revert "remove Vercel AI Gateway and Vercel Sandbox (#33067)"
This reverts commit febc4cfec0.
2026-07-29 19:48:37 -07:00
Jasmine Naderi 91351b7b77 fix(state): make journal mode canonical and behaviorally verified
Use database.journal_mode as the sole non-secret operator setting, preserve the vulnerable-SQLite safety gate and existing WAL databases, validate explicit DELETE results, document the active config path, and cover real SQLite openers with behavioral tests.
2026-07-29 18:13:09 -07:00
Jasmine Naderi 04ec841462 fix(state): configurable journal_mode + centralize all DB openers
Add HERMES_JOURNAL_MODE env / database.journal_mode config for
virtiofs/NFS/SMB where WAL is not crash-safe. Route 5 bypass openers
through apply_wal_with_fallback so a single setting covers every .db
(#68545).
2026-07-29 18:13:09 -07:00
Gabriel Stoltemberg 59ee85ed50 perf: use load_config_readonly() at read-only call sites in agent/
Salvaged from #56085 (@Stoltemberg), rebased onto current main: sites
main had already converted (credential_pool, auxiliary_client MoA
paths, model_metadata, moa_loop, agent_runtime_helpers) resolve to
main's versions; the remaining ~29 read-only sites across 16 agent/
files swap to the no-deepcopy readonly loader (~135us saved per call).

Full per-site mutation audit performed (every enclosing function read,
escapes traced): 23 SAFE, 5 ESCAPES with read-only consumers, 1 UNSAFE
path (init_agent -> get_compatible_custom_providers -> normalizer
in-place alias writes) fixed by the preceding no-mutate commits, which
make the normalizer copy-safe for ALL callers.
2026-07-29 15:28:15 -07:00
Soju06 ce9f6712ff refactor(agent): remove the inter-tool delay
The 1.0s sleep between sequential tool calls has been present since the
initial commit with no documented rationale. It sleeps between local
tool executions — the next LLM request goes out only after the whole
batch — so it rate-limits nothing, and the parallel read-only path
already runs with no delay. Every multi-tool turn pays (N-1) seconds
of dead time. Remove the sleep, the internal tool_delay plumbing, and
dead test assignments. AIAgent.__init__ keeps tool_delay as a
deprecated no-op keyword for one release so existing programmatic
callers construct cleanly; passing it emits a DeprecationWarning.
2026-07-29 15:21:26 -07:00
Teknium 1a088989bc
Merge pull request #66730 from NousResearch/feat/hsp-sync-client
feat(sync): HSP/1 personal skill sync client (M1 client)
2026-07-29 15:20:35 -07:00
Teknium 28524adb0e fix(tests): eliminate flaky/broken tests — shadow sys.path inserts, unmocked network in compressor tests, stale-SDK feishu pin guard, quadratic redact regexes
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
  it prepended the tests/ dir itself to sys.path, so 'import agent' /
  'import hermes_cli' resolved to the test packages and collection died
  with ModuleNotFoundError depending on import order (2 files failed in
  every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
  unmocked: each burned ~50s attempting live LLM traffic through the
  relay before falling back (572s file — the slowest in the suite, and
  flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
  the compressor's redaction pass on large payloads —
  _STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
  prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
  output-equivalence fuzz-verified on 20k random strings), and the
  _CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
  pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
  signature check; the repo pins lark-oapi==1.6.8 but stale local
  installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
  agent.redact + agent.credential_persistence in the fake agent package
  (empty __path__ blocks all real agent.* imports added since the fake
  was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
  2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
  baseline run (passes instantly when the box is quiet).
2026-07-29 15:12:28 -07:00
Ben Barclay f5b68ad58b Merge origin/main into feat/hsp-sync-client
Resolves the PR's conflict with main (2252 commits). Two conflicts, both
"each side added an independent block in the same place" — kept both:

- gateway/run.py — the housekeeping loop. This branch adds the Skill Sync
  pulls inside the CURATOR_EVERY branch (12-space indent); main adds a
  stale-session auto-archive as a sibling `if` at loop level (8-space).
  Different scopes, so the naive union would have mis-nested the archive
  block into the curator branch; kept each at its own indent level.
- tools/skill_manager_tool.py — the _edit_skill result dict. This branch
  appends the org auto-propose note; main appends
  _add_description_prompt_preview(). Independent, order-insensitive.

No behaviour dropped from either side.

Verified: 3552 passed / 0 failed across 63 suites (scope regenerated to
include main's new maybe_auto_archive / _add_description_prompt_preview
consumers) via scripts/run_tests.sh. `hermes sync` and `hermes sync status`
still work against a live token, resolving the production plane default.

The Pyright Optional-parameter warnings in skill_manager_tool.py are
pre-existing on main (`content: str = None` etc.), not introduced here.
2026-07-29 13:01:36 -07:00
joaomarcos 7b5997f111 fix(codex): don't retry after a valid response.completed on drain error
A transport error during the post-terminal SSE drain (used only to let the
Relay transport finalize the attempt) was sharing exception handling with
the pre-terminal assembly path, so it discarded an already-completed,
already-billed response and opened a brand-new physical request. Give the
drain step its own non-fatal error handling: log a finalization warning
and still return the terminal response.

Closes #74310
2026-07-29 16:41:39 -03:00
teknium1 f8758dcaf8 refactor(agent): single-owner call_id + reasoning_content sanitization policies (wire-parity verified) 2026-07-29 12:29:39 -07:00
Alex Fournier f1fd678e44 feat(observability): add Relay skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 12:20:34 -07:00
Francesco Bonacci c268397752 feat(computer_use): align cua-driver 0.10 permission modes 2026-07-29 12:19:37 -07:00
Francesco Bonacci 847e401b74 feat(computer_use): align cua-driver 0.9 contracts
Salvaged from PR #67807 by @f-trycua onto current main.

- Foreground gate: discover delivery_mode support from the live tools/list
  inputSchema.properties (fail closed), not the never-shipped
  input.delivery_mode capability token
- bring_to_front: standalone strict-schema MCP tool (inject_session=False),
  separate approval scope, requires foreground
- Verdict precedence: confirmed > unverifiable (verify before retry) >
  suspected_noop/refusal (escalate); surfaced as explicit verdict field
- Typed cua_browser_* route inside computer_use (browser_route.py) with
  exact-binding, adapter-injected session, snapshot-scoped refs
- Per-Hermes-session backend isolation + release_computer_use_session seam
  wired into AIAgent.close()
- Recorded 0.9 tools/list fixture replaces fabricated capability tokens
2026-07-29 12:19:37 -07:00
teknium1 30c783589c perf(agent): cursor/memo optimizations for per-iteration full-history walks (byte-parity proven)
Three provably-safe optimizations for O(n)-per-iteration history walks:

1. sanitize_tool_call_arguments: optional identity-keyed cursor (strong
   refs to the exact validated message objects) skips re-json.loads-ing
   already-validated history each loop iteration. Any list rewrite
   (compression, repair, undo, steer) breaks the identity prefix match
   and forces re-scan from the divergence point. Wired via a per-agent
   cursor dict in conversation_loop.

2. estimate_messages_tokens_rough: per-message memo keyed on a deep
   identity fingerprint (strings pinned by strong reference so id()
   aliasing is impossible; scalars by value; dicts/lists structurally
   with key order). Equal fingerprints imply identical str(shadow)
   bytes, hence identical estimates. Unfingerprintable shapes fall
   through to direct compute. Bounded FIFO cache (4096 entries).

3. _flush_messages_to_session_db_unlocked: bounded scan that skips the
   identity-matched prefix of the previous successful flush's snapshot.
   Snapshot only taken on full success; cleared on exception. Compression
   rewrites use fresh copies, breaking identity and forcing full re-scan.

Parity proven in tests/agent/test_cursor_optimizations_parity.py:
500-message synthetic histories with tool calls, malformed args, unicode,
element-wise old==new across 3 iterations incl. simulated compression.

Measured (median of 5): sanitize 0.097ms->0.011ms, tokens 1.145ms->0.853ms,
persist-scan 179.5us->10.0us at 500 messages.
2026-07-29 11:54:18 -07:00
teknium1 5b751dc0ad chore: remove unused imports and dead locals (ruff F401/F841 sweep)
Cleans F401 unused imports and F841 dead local assignments across
root *.py, agent/, hermes_cli/, tools/, gateway/, cron/, tui_gateway/
(tests/, plugins/, skills/ excluded).

Intentionally KEPT (false positives / test-patch surfaces):
- agent/transports/__init__.py package re-exports
- cli.py browser_connect re-exports (DEFAULT_BROWSER_CDP_URL area,
  used by tests/cli/test_cli_browser_connect.py)
- hermes_cli/main.py _prompt_auth_credentials_choice /
  _model_flow_bedrock_api_key (accessed via main_mod attr in tests)
- gateway/run.py aliased replay_cleanup + whatsapp_identity re-exports
  and _PORT_BINDING_PLATFORM_VALUES (test-referenced)
- hermes_cli/web_server.py get_running_pid (tests monkeypatch it) and
  _OAUTH_TOKEN_URL availability probe
- hermes_cli/config.py get_process_hermes_home re-export (noqa'd F811
  chain) and yaml availability-probe import
- hermes_cli/nous_subscription.py managed_nous_tools_enabled
  (tests patch hermes_cli.nous_subscription.managed_nous_tools_enabled)
- try/except ImportError availability probes (env_loader, tts_tool,
  mcp_tool, web_server anthropic OAuth block)
- tools/web_tools.py noqa F401 re-exports
- hermes_cli/setup_whatsapp_cloud.py:263 'proceed' skipped: possible
  missing-guard bug, flagged for separate review
- unused function parameters (signature changes out of scope)

Side-effect RHS calls preserved where only the binding was dead
(e.g. web_server proc = _spawn_hermes_action -> bare call).
2026-07-29 11:53:39 -07:00
teknium1 c2eda92fd0 perf(config): stop deepcopying config on per-turn read-only paths
Four hot-path consumers paid a full config deepcopy per read:

- telemetry gate relay_shared_metrics.enabled() — runs 2-3x per agent
  turn (2x per API call from lifecycle hooks + 1x per tool call) and
  called read_raw_config(), which deepcopies the whole raw config every
  call. New read_raw_config_readonly() serves the cached dict directly:
  248 us -> 4.6 us per call (54x) on Teknium's real 77-key config.
- interruptible_streaming_api_call local-endpoint stale-timeout branch
  called load_config() once per API call for every local-model user.
- gateway get_inbound_media_max_bytes() + _get_ephemeral_system_ttl_default()
  called load_config() on per-message paths. All three switched to
  load_config_readonly() (345 us -> 12 us; PR #28866 lineage).

Together these account for ~90% of the ~1,900 deepcopy primitives per
turn measured in the 26-call stubbed-LLM profile.

read_raw_config_readonly() keeps the (mtime_ns, size) freshness key so
config edits are picked up next call, and preserves the identity
invariant (cache-miss returns the same object later hits serve) —
regression-tested with 'is', per the PR #28866 identity-bug lesson.
The mutable read_raw_config() is unchanged for save-path callers.

581 targeted tests green (config, relay metrics x2, ephemeral reply,
platform base, new readonly suite).
2026-07-29 11:33:41 -07:00
Alex Fournier 8b0c3da8c0 feat(observability): aggregate bounded tool metrics 2026-07-29 11:25:55 -07:00
teknium1 eff3b11eb2 refactor: complete approval mode/timeout resolution migration to tools/approval.py core (TUI + codex surfaces)
TUI (tui_gateway/server.py _load_approval_mode): now delegates to
tools.approval._get_approval_mode instead of re-reading config raw via
_load_cfg + _deep_merge(DEFAULT_CONFIG, ...) and normalizing locally.
Behavior fix, not pure refactor: the canonical load_config path applies
managed-scope config overlays and ${VAR} env expansion, plus a legacy
max_turns lift, which the TUI's raw YAML read bypassed — under a managed
config that sets approvals.mode, the TUI previously reported/toggled a
different mode than the approval gate actually enforced. Both surfaces
now agree by construction. Name/signature and the mode-vocabulary clamp
are preserved.

Codex (agent/transports/codex_app_server_session.py): read confirmed the
_decide_exec_approval/_decide_apply_patch_approval paths carry NO
Hermes-side mode/timeout reads — the Hermes resolution already flows in
from agent/codex_runtime.py via tools.approval.is_approval_bypass_active()
(auto_approve_* routing) and via the shared approval-gate callback. So no
code extraction was needed; added docstrings pinning that invariant and a
cross-reference on the protocol-semantic choice mapping
(_approval_choice_to_codex_decision), which intentionally stays local.

Adds tests/tools/test_approval_mode_parity.py: cross-surface invariant
test asserting the core resolver, the TUI path, and the codex bypass
derivation agree for synthetic configs (unset defaults, global mode set,
YAML-bool off, malformed values, whitespace/case), plus a delegation-seam
test proving the TUI has no independent config read left.

Note: gateway/run.py has sibling raw reads but is intentionally untouched
(multiple in-flight PRs); flagged as follow-up.
2026-07-29 11:19:24 -07:00
teknium1 bc747001ee perf(imports): lazy-load heavy SDKs off the cold-start waterfall
Four deferrals following the established truthy-skip / PEP 562
lazy-load patterns (PRs #22681/#22859 lineage). Rebased over #74194,
which independently landed the browser_tool half of this work — that
file is dropped here; the remaining four modules are untouched by it:

- tools/vision_tools.py: defer agent.auxiliary_client
  (credential_pool -> hermes_cli.auth -> httpx -> rich, ~50 ms) to
  first vision handler call. async_call_llm /
  extract_content_or_reasoning stay patchable module attributes;
  injected test mocks win over the loader.
- agent/model_metadata.py: defer 'requests' (+urllib3, ~27 ms of the
  'import cli' waterfall) to the fetch functions. PEP 562 __getattr__
  keeps patch('agent.model_metadata.requests.get') working.
- tools/browser_supervisor.py: websockets (~22 ms) imports on first
  CDP connect; ClientConnection type under TYPE_CHECKING.
- cron/jobs.py: croniter (~15 ms) resolves on first cron-expression
  use; HAS_CRONITER stays monkeypatchable (None = unprobed sentinel).

A/B vs current main incl. #74194 (median of 7, cold subprocess):
  import cli          147 -> 132 ms  (-10%)
  import model_tools  244 -> 224 ms  (-8%)
  import run_agent    264 -> 244 ms  (-8%)

Lazy-verify: importing the four modules no longer pulls requests /
croniter / websockets into sys.modules. 369 targeted tests green
post-rebase.
2026-07-29 10:54:04 -07:00
teknium1 d7a4065568 perf(local-endpoints): disk L2 for server-type + ollama ctx probes, faster timeouts
Local-model users paid a fresh probe waterfall on EVERY CLI cold start
inside AIAgent.__init__: detect_local_server_type (up to 4 HTTP GETs,
2s timeout each on a hung server) + /api/show (3s timeout). The
existing caches were in-process only, so back-to-back invocations
(chat -q, cron ticks, subagents) re-paid the network every time.

- New 300s-TTL disk L2 at HERMES_HOME/cache/local_endpoint_probes.json
  for detect_local_server_type verdicts and query_ollama_num_ctx
  results. Only SUCCESSFUL probes persist (a down server never pins a
  negative verdict); stale entries pruned on write; corrupted cache
  degrades to a miss; atomic writes. 300s is strictly fresher than the
  1h in-process TTL that already accepts server-swap staleness.
- models.dev fetch timeout 15 -> (5, 10) connect/read tuple: a
  blackholed connect stalled the first-turn critical path 15s; now
  fails in 5s (matches the OpenRouter fetch convention, #46620).
- _auto_detect_local_model timeout 5 -> (2, 3): runs inside
  _get_model_config() at startup against a LOCAL endpoint; a hung local
  server cost 5s before the banner.

E2E (real HTTP server, two fresh subprocesses, isolated HERMES_HOME):
proc1 = 2 HTTP hits, proc2 = 0 HTTP hits, identical results
(ollama/131072), probe wall 74.5 -> 35.5 ms. 222 targeted tests green
incl. 9 new disk-L2 contract tests.
2026-07-29 10:52:13 -07:00
teknium1 0034da2975 perf(stream): replace per-chunk repr() with delta-length byte estimate
The streaming hot loop computed len(repr(chunk)) on EVERY chunk to feed
the retry-diagnostic byte counter — a full recursive pydantic repr at
5.5-8.8 us per chunk (measured), ~20-30 ms of pure CPU per 3,000-chunk
response, paid on every streaming response on every platform.

New _estimate_chunk_bytes() sizes the chunk from its delta payload
strings (content / reasoning / tool-call arguments) plus a 40-byte
framing floor: 2.1-2.4 us per chunk (~3x cheaper), independent of
pydantic field count, never raises on unknown shapes (Anthropic events,
stub providers fall back to the floor). Both call sites switched
(chat-completions loop + anthropic event loop).

The counter feeds only the stream-retry diagnostic log line
(agent/stream_diag.py) — an estimate proportional to traffic preserves
its purpose (distinguishing 'died at 0 bytes' from 'died mid-stream').

6 new contract tests; 64 targeted stream tests green.
2026-07-29 10:21:59 -07:00
teknium1 3d48f893da refactor: single build_subprocess_env() factory for all child-process spawns (profile + secret-scrub single owner) 2026-07-29 10:14:11 -07:00
teknium1 6f7f7cd064 refactor: use shared strip_ansi for inline ANSI regexes 2026-07-29 10:13:50 -07:00
teknium1 7c198c5e44 refactor: single shared Retry-After parser 2026-07-29 10:13:50 -07:00
teknium1 19055492aa fix: route stray HERMES_HOME hardcodes through get_hermes_home() (profile + native-Windows safety) 2026-07-29 09:33:48 -07:00
victor-kyriazakos 1773752c8c Merge remote-tracking branch 'origin/main' into feat/gateway-health-diagnostics-monitoring
# Conflicts:
#	uv.lock
2026-07-29 15:37:14 +00:00
kshitijk4poor 222ea2b6c9 refactor: fold simplify-code review findings
- extract _commit_registry/_note_refresh_failure shared by the background
  worker and foreground stage-4 (identical 4-step success + failure paths
  were duplicated); worker now commits under _models_dev_fetch_lock so a
  failing background refresh can never re-arm the backoff immediately
  after a successful force_refresh committed (unsynchronized-write race)
- add should_clear_context_pin_async to hermes_cli/route_identity.py
  (matching the get_model_context_length_async precedent) and use it at
  the 4 async gateway sites instead of inline asyncio.to_thread wraps;
  the sync _format_session_info site keeps the sync call (already
  off-loop via its callers' to_thread)
- test the background-refresh success path (the PR's primary new
  behavior): disk saved, mem cache swapped, backoff cleared, in_flight
  reset — mutation-checked
- replace the race-prone spin-wait on _models_dev_refresh_in_flight with
  a named-thread join in the backoff test
2026-07-29 17:13:48 +05:30
kshitijk4poor ccf7129ed0 fix: restore zero-arg fetch_models_dev call on default paths
The branched call shape in get_provider_info/get_provider is deliberate:
~69 test sites across tests/hermes_cli and tests/gateway monkeypatch
fetch_models_dev (and get_provider_info) with zero/single-arg lambdas.
Passing allow_network= unconditionally broke 5 tests in CI slices 2/3/7.
Documented the constraint inline.
2026-07-29 17:13:48 +05:30
kshitijk4poor 11ca7eedf0 fix: follow-up hardening for salvaged #73621 + #35853
- _mark_stale_cache_grace only moves cache_time forward so a completed
  background refresh is never rewound to a 5-minute grace window
- clear _models_dev_refresh_in_flight if Thread.start() raises so a
  one-off thread-exhaustion failure doesn't disable refresh forever
- move empty-registry validation into _fetch_models_dev_from_network
  (was duplicated in the background worker and the foreground fetch)
- pass allow_network through as a plain kwarg in get_provider_info and
  hermes_cli.providers.get_provider instead of the branched call shape
- refresh the stale module docstring (no bundled snapshot exists; the
  resolution order now describes stale-serve + background refresh)
2026-07-29 17:13:48 +05:30
zapabob a479a1599f fix(models): use stale cache before models.dev refresh 2026-07-29 17:13:48 +05:30
StellarisW 8c50aaceb6 fix(gateway): keep models.dev refreshes off event loop 2026-07-29 17:13:48 +05:30
kshitijk4poor 24a56f027c fix(lsp): expose lsp.idle_timeout in config, harden reaper, log reaps
Follow-ups on top of @DonutsDelivery's salvaged reaper commit (#64141):

- create_from_config() now parses lsp.idle_timeout (invalid values fall
  back to DEFAULT_IDLE_TIMEOUT) — previously the constructor knob was
  unreachable from config.yaml (config exposure adapted from #36892 by
  @0xbWy and #68091 by @9miya20)
- canonical default declared in hermes_cli DEFAULT_CONFIG so config
  discovery surfaces the knob (per sweeper review note on #64980)
- reaper loop survives transient sweep errors instead of dying and
  silently re-opening the leak (gap flagged in #68091 review)
- eventlog.log_reaped(): one INFO line per sweep + clears the
  log_active announce cache so respawns re-announce at INFO
- docs: replace the stale 'no idle-timeout reaper' paragraph with the
  new lifecycle description + config reference
- tests: reuse-refresh protection (the regression teknium's sweeper
  requested on #64141), reaper-survives-error, config propagation,
  invalid-value fallback, DEFAULT_CONFIG/manager-constant sync
2026-07-29 17:11:30 +05:30
DonutsDelivery d7578018c5 fix(lsp): reap idle language servers 2026-07-29 17:11:30 +05:30
Teknium 63fc810b95 fix(gemini): sweep hardcoded Gemini default models to gemini-3.6-flash (#32360)
gemini-2.5-flash shuts down Oct 16 2026 (Google deprecation schedule)
and gemini-3-flash-preview is superseded. Update every hardcoded
default to the current GA flash model:

- gemini_native_adapter: probe_gemini_tier + _create_chat_completion
  default params; free-tier guidance de-pinned from a specific model's
  RPD number so it doesn't stale again
- auxiliary_client: gemini/kilocode fallback aux models,
  _OPENROUTER_MODEL, _NOUS_MODEL -> google/gemini-3.6-flash
  (verified live on both OpenRouter and Nous portal /models)
- provider plugins: kilocode + vertex default_aux_model
- hindsight memory plugin: gemini provider default
- setup wizard gemini list: 3-flash-preview -> 3.6-flash (matches the
  curated picker catalog)
- tests: aux-client assertions that pinned the old default literal now
  reference the _NOUS_MODEL constant, so the next default bump can't
  break them (change-detector cleanup)

Fixes #32360.
2026-07-28 18:17:56 -07:00
Atakan 01cb38e8ec fix(gemini): preserve bridged tool response name 2026-07-28 14:37:56 -07:00
Siddharth Balyan 3cedac00b7
fix(credits): reintroduce grant_spent, gated on an in-session crossing (#73634)
* Revert "fix(credits): remove the 'Grant spent · $X top-up left' notice"

This reverts commit 5dc6a14c14.

* fix(credits): reintroduce grant_spent behind an in-session crossing gate

The removed notice nagged because its condition is a steady state for
accounts living on top-up, the latch is per-session, and the cold-start seed
runs the policy at every session open — every session re-announced
'Grant spent · $X top-up left'. Reintroduce it gated so only a session that
WATCHES the grant run out announces:

- seen_grant_unspent crossing gate, mirroring seen_below_90: opens when the
  session observes the grant meaningfully unspent (>= GRANT_UNSPENT_MIN_MICROS,
  1 cent — portal-seeded states derive micros from float dollars and can carry
  sub-cent residue where headers report exactly spent). Seeds never prime it.
- The gate guards only the show branch and is consumed by the announcement —
  one announcement per crossing. Header flicker (used_fraction None and back)
  clears the sticky line but cannot re-announce; a renewal re-opens the gate.
- new_credits_latch() centralizes the latch shape (agent build, lazy re-init,
  and the policy test helper all build through it).
- Tests: steady-state-open stays silent (seed + policy seams), live crossing
  announces once, flicker/renewal/residual cases locked; the rendering test
  primes the gate and asserts its leg count so a gate regression cannot
  silently shrink coverage.
2026-07-28 21:36:23 +00:00
atakan g 0c2d9aee0b fix(model-metadata): prefer local Ollama num_ctx 2026-07-28 14:18:18 -07:00
kshitijk4poor f1eef85877 fix: replace RuntimeError with graceful degradation for unverified LM Studio loads
Salvage of PR #52188. The original PR raised RuntimeError when LM Studio
load was rejected or unverifiable, which would abort agent startup on
transient network failures. Replace with logger.warning + fallback to
configured context length, preserving the old graceful-degradation behavior.
2026-07-29 02:19:13 +05:30
LunarNexus 8c12fa7cf0 fix(lmstudio): respect applied runtime context 2026-07-29 02:19:13 +05:30
PRATHAMESH75 e45f2b39e2 fix(codex): clamp oversized Responses call_id so MCP tools don't brick sessions (#73492)
The codex app-server namespaces MCP tool call ids as
codex_mcp__<server>__<tool>_<codex_call_id>. With an exec-<uuid> component the
built-in hermes-tools server alone overflows the Responses API's 64-char
call_id limit, so the request 400s with a non-retryable "string too long".
The offending item sits near the head of the transcript and replays every
turn, permanently bricking the session — the only recovery is /reset.

Sibling defect to #10788, which clamped input[*].id via
_MAX_RESPONSES_ITEM_ID_LENGTH. Apply the same treatment to call_id at both
Responses emit sites in _chat_messages_to_responses_input: a deterministic
surrogate (call_ + sha256[:32]) for ids over the limit, short ids unchanged.
Because the surrogate is a pure function of the original id, a function_call
and its matching function_call_output — which carry the same original id — map
to the same surrogate and stay paired without correlating the two items.
2026-07-29 00:47:54 +05:30
Teknium d259b24ede feat(pricing): add gemini-3.6-flash and gemini-3.5-flash-lite official snapshot entries
Follow-up to the #60063 salvage: the curated gemini list now carries
gemini-3.6-flash (aux default, #70416) and the vertex list carries
gemini-3.5-flash-lite (#68767) — both need snapshot pricing so direct
Gemini/Vertex sessions don't report cost=unknown.

Rates verified against https://ai.google.dev/gemini-api/docs/pricing
(2026-07-28): 3.6-flash $1.50/$7.50, cache read $0.15;
3.5-flash-lite $0.30/$2.50, cache read $0.03.
2026-07-28 11:54:46 -07:00
Ivan Miao 26dd976fbe fix: align Gemini billing pricing with provider catalog 2026-07-28 11:54:46 -07:00
Ivan Miao ecebff82d2 fix: support Gemini billing route mapping and pricing update 2026-07-28 11:54:46 -07:00
Teknium 5dc6a14c14 fix(credits): remove the 'Grant spent · $X top-up left' notice
The grant_spent notice fired for every subscription user with top-up
funds the moment their cap was reached and camped in the CLI/TUI status
bar and desktop toasts with no action to take — the account keeps
working off top-up. Remove it everywhere:

- agent/credits_tracker.py: drop grant_cond + the emit/clear block;
  the dev fixture state now (correctly) produces no notice
- TUI: keep the turn-start clear of credits.grant_spent as back-compat
  for older backends that still emit the key
- Desktop: drop the demo step and stale comment references
- Docs/config comments: remove grant-spent from credits_notices text
- Tests updated: policy/cold-start now assert the key never fires

Usage bands, depleted, and restored notices are unchanged; /usage still
reports the full balance breakdown.
2026-07-28 11:21:44 -07:00
Alex Fournier 3b55419cb1 fix(relay): close logical calls in stack order
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 10:19:53 -07:00
Alex Fournier 5e34fa2d5c fix(relay): preserve provider stream errors
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 09:58:43 -07:00
Alex Fournier b1a5d67e71 Merge upstream main into fix/hermes-relay-anthropic-context
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 08:10:58 -07:00
kshitijk4poor 49f50c68a0 fix(compression): coherent context_length setter — no-op guard, re-floor on new window, un-strandable init log
Review-pass findings on the lazy-init deferral:

- No-op guard: the codex app-server usage callback assigns
  compressor.context_length on EVERY response (same window each time).
  The setter unconditionally invalidated the derived budgets, wiping
  runtime corrections applied directly to threshold_tokens /
  tail_token_budget (aux-context threshold sync) — those persisted on
  main's eager init. Same-value assignment is now a no-op.

- Re-floor on genuinely new window: the setter invalidates budgets but
  previously kept the stale threshold_percent, so a codex window switch
  recomputed threshold_tokens from the new window with the old model's
  floored percent. Re-apply the raise-only small-context floor from
  _base_threshold_percent so percent and tokens derive from the same
  window (guarded with getattr for object.__new__ test instances).

- Init log extracted to _emit_init_summary_once() and also fired from
  the setter path, so a consumer assigning context_length before any
  read no longer strands the startup line forever.

- threshold_tokens getter resolves the window into a local before
  reading threshold_percent — correctness no longer depends on
  left-to-right argument evaluation order.

- reasoning_timeouts: fix inaccurate 'tuples are immutable' comment
  (the container is a list; safety comes from build-once-at-import),
  document why the slug stays in the tuple.

Adds TestContextLengthSetterCoherence (3 tests): same-value assignment
preserves overrides; new-window assignment re-floors both directions.
2026-07-28 20:04:58 +05:30
kshitijk4poor e762a5a473 fix(compression): copy-on-write in image-shrink recovery so degraded images never reach stored history
With the selective prompt-cache copy (#57046), un-marked messages on the
decorated api_messages list share their nested content parts with the
persistent conversation history — the per-message copy in
conversation_loop is shallow and decoration now deep-copies only the
marked messages. try_shrink_image_parts_in_messages previously wrote the
re-encoded image INTO the aliased part/source dicts, so an
image-too-large retry on an Anthropic route would silently replace the
original image bytes in agent.messages (and persist the degraded copy).

Replace the in-place writes with copy-on-write: rebuild the content list
with fresh part/source/image_url dicts and reassign msg['content'] — a
top-level write on the per-call copy that never reaches history.

Adds two regression tests simulating the aliasing; both fail against the
old in-place implementation (mutation-verified).
2026-07-28 20:04:58 +05:30
kshitijk4poor 44bd0521a4 fix(compression): keep ContextCompressor init non-blocking when quiet_mode=False
The lazy-init change in #32221 deferred get_model_context_length() out of
ContextCompressor.__init__, but the "Context compressor initialized" log
(emitted only when quiet_mode=False) reads self.context_length,
self.threshold_tokens and self.tail_token_budget. Those property reads
resolve the deferred value, so the synchronous model-metadata probe still
ran inside __init__ on the interactive-CLI path — the exact blocking the
PR removed, just narrowed to the non-quiet path.

Emit the informative line once, on first context-length resolution, so
construction stays non-blocking on every path. Add a regression test
asserting no probe fires in __init__ with quiet_mode=False and that the
init log is emitted exactly once on first access.
2026-07-28 20:04:58 +05:30
Rod Boev 958a81a1e4 perf(agent): defer synchronous httpx.post out of AIAgent.__init__ (#32221)
(cherry picked from commit 1fe63238b012e4326c975b2b1a303f6f27aa5102)
2026-07-28 20:04:58 +05:30
Gabriel Stoltemberg abc2069f49 perf: pre-compute sorted reasoning timeout floors at module level
_match_any() was re-sorting _REASONING_STALE_TIMEOUT_FLOORS (21 elements)
on every call. This function runs per API turn via error_classifier,
chat_completion_helpers, and thinking_timeout_guidance.

Also fixes thread-safety: the old _PATTERN_CACHE was a mutable dict
accessed from multiple threads without locking. Pre-compiling all
patterns at module load time eliminates both the per-call sort and
the TOCTOU race condition. The resulting list is effectively
immutable after import, safe for free-threaded Python 3.13+.

(cherry picked from commit e8b006b853)
2026-07-28 20:04:58 +05:30
Koho Zheng 43a9bd9e0b perf(caching): selective copy instead of deepcopy entire history
Replaces full deepcopy with selective shallow copy in apply_anthropic_cache_control.
Only the 4 messages that receive cache_control markers are deep-copied; the rest
stay as references.

Measured on a 100-message conversation (typical long session):
- Before: ~15ms per call
- After: ~2ms per call
- 7.5x faster, scales with conversation length

Memory impact is also significant — no need to duplicate dozens of unchanged
messages on every turn.

The contract is unchanged: callers still get an independent message list
they can mutate. Only messages we modify (by injecting cache markers) are
copied. The rest are shared references to immutable history entries, which
is safe since the agent never mutates past turns.

(cherry picked from commit 17892df6cc)
2026-07-28 20:04:58 +05:30
Soju06 82e2c9ce40 perf(agent): reuse the per-request OpenAI wire client across sequential LLM calls
Every LLM call built a fresh openai.OpenAI wire client (new httpx pool,
TCP+TLS handshake, measured 19.2ms p50 / 35.5ms p95 per call at ~5 calls
per tool-loop turn) and closed it when the request finished. Cache ONE
reusable wire client on the agent, keyed by the effective client kwargs:

- _create_request_openai_client hands back the cached client when the
  effective kwargs are identical; any change (credential rotation,
  provider failover, vision default_headers) evicts and rebuilds.
- Only a request that produced a response reports a reuse close reason
  (request_complete / stream_request_complete); error unwinds report
  *_error_cleanup and really close, so a retry after a request error
  always builds a fresh pool.
- Cross-thread aborts poison the slot: a pool whose sockets were
  shutdown(SHUT_RDWR) from a stranger thread is never reused (#29507) —
  the owner-thread close discards it and the next create rebuilds. The
  holder read and the abort are atomic (under the holder lock) at all
  three abort sites, so a late abort can never poison the NEXT request's
  checked-out client.
- Worker-side interrupt breaks close the half-read SSE stream on the
  owning thread before building the partial response; a failed close
  poisons the slot (otherwise each interrupt leaked one checked-out
  connection until the pool hit PoolTimeout). run_codex_stream gets the
  same poison-on-close-failure handling.
- Single checked-out slot (in_use): a concurrent call gets an untracked
  client with the old per-request lifecycle.
- release_clients() / close() really close the cached client when idle;
  if a worker has it checked out they abort the sockets and detach the
  slot, deferring the FD release to the worker's own close.
- MoA facade and Mock passthroughs never enter the cache; max_retries=0
  is preserved on all request clients.
2026-07-28 19:06:59 +05:30
Soju06 174ad45939 perf(state): apply per-call token accounting on a background single-writer queue
Every API call in the tool loop persisted its token/cost delta by
calling SessionDB.update_token_counts() synchronously on the turn
thread — a BEGIN IMMEDIATE sessions UPDATE plus a session_model_usage
upsert, measured in production at p50 3.3ms / p95 70.4ms per call and
up to 299ms against a cold multi-GB state.db. The tool loop stalls for
that long between calls, N times per multi-tool turn.

SessionDB gains queue_token_counts(): same signature and semantics as
update_token_counts(), but the critical path is a deque append plus a
condvar notify. A lazily started daemon thread applies deltas in
enqueue order through the existing update_token_counts ->
_execute_write path, so the established self._lock / BEGIN IMMEDIATE /
jitter-retry discipline is unchanged. When a backlog forms, adjacent
same-route incremental deltas coalesce into one UPDATE: token and
api-call fields sum, cost fields sum None-preservingly (an all-None
run stays None so COALESCE keeps the stored value), and absolute=True
deltas never merge and act as ordering barriers. Route equality is
required for a merge because those fields feed COALESCE backfill, the
last-non-None-wins status fields, and the per-model usage attribution
key — a merged apply is row-equivalent to sequential applies.

Correctness and durability:

- flush_token_counts() gives read-your-writes to token/cost readers
  (get_session, list_sessions_rich, _get_session_rich_row,
  list_gateway_sessions, InsightsEngine.generate) — a plain attribute
  check when nothing is queued. The writer sets its busy flag before
  popping the queue so the lock-free fast path can never miss an
  in-flight batch.
- update_session_model / update_session_billing_route /
  update_session_meta write the sessions row synchronously, bypassing
  the queue, so they flush it first: a still-queued first-of-session
  delta carries the pre-switch route, and applying it after the switch
  UPDATE would trip the first_accounted_route branch (api_call_count
  == 0 plus a route mismatch) and resurrect the old model/provider.
- AIAgent._persist_session flushes at turn finalize and every
  error-exit persist point; close() stops and drains the writer before
  the WAL checkpoint; an atexit hook (registered on first enqueue,
  unregistered on close so closed instances are not pinned until
  interpreter exit) drains at shutdown. Worst-case crash loss is the
  in-flight call's delta — the same window as the old inline write.
- A flush trusts a live stop-flagged writer (its loop drains before
  exiting) and only drains on the caller's thread when the writer is
  dead or never started, claiming the same busy flag so concurrent
  flushes wait instead of racing an in-flight batch.
- After close() has stopped the writer, queue_token_counts applies the
  delta inline instead of parking it on a queue nothing will drain; a
  closed-connection failure then raises at the call site, which
  already guards for it, exactly like the old synchronous path.
- Writer apply failures are logged and never raise into a turn; the
  writer thread survives and keeps applying.

Call sites switched to the queue: the per-call site in
agent/conversation_loop.py and both codex app-server sites in
agent/codex_runtime.py. In-memory per-turn counters
(agent.session_estimated_cost_usd etc.) stay synchronous, so live turn
displays never see the queue.

Tests: tests/agent/test_async_token_accounting.py (19 tests: enqueue
ordering, absolute-as-barrier, backlog coalescing with exact sums,
coalesced-vs-sequential row equivalence, merge unit rules, None-cost
preservation, read-your-writes, flush vs stop-flagged/concurrent
drains, inline apply after writer stop, close/atexit durability,
_persist_session drain, writer failure isolation);
tests/run_agent/test_token_persistence_non_cli.py updated to the
queue_token_counts contract.
2026-07-28 18:47:54 +05:30
Victor Kyriazakos 6a174e9967 Merge origin/main into feat/gateway-health-diagnostics
# Conflicts:
#	cron/executions.py
#	cron/jobs.py
2026-07-28 13:09:17 +00:00
HexLab98 173fec8c03 fix(agent): shut down sockets on httpx mount pools during interrupt abort
With HTTP(S)_PROXY (and similar mounted transports), live connections sit
on client._mounts rather than the default _transport. force_close_tcp_sockets
only walked the default pool, so stranger-thread interrupt abort logged
tcp_force_closed=0 and left the request alive for minutes (#72975). Also
walk nested proxy _connection wrappers and WARNING when an abort finds no
sockets.
2026-07-28 18:07:44 +05:30
Brooklyn Nicholson b3ba5570c9 Merge origin/main into bb/skill-chip-only
Keep skill-invocation projection helpers alongside the auto-continue
legacy display typing from #73250.
2026-07-28 04:06:32 -05:00
Brooklyn Nicholson bf0871fbf8 fix(agent): type a synthesized user turn when its row is written
The auto-continue recovery note was typed only after run_conversation
returned, so its row sat untyped for the whole turn — and permanently
when the continuation was itself killed, which is the case it exists
for. persist_user_display_kind stamps the type on the live message
before the crash persist writes it, in the same insert as the content.
The flush also carries display_metadata through, which it was dropping.
2026-07-28 03:54:16 -05:00
Brooklyn Nicholson dd39ec2694 fix(gateway): project a /skill turn onto its invocation for every client
A slash-skill invocation is persisted expanded — activation note plus the
entire skill body. _history_to_messages is the single display projection
every surface reads, so that payload rendered as a chat bubble anywhere a
session was resumed.

Project it here onto the invocation the user typed, and tag skill/bundle
dispatches with the same string so the live send matches. Rewind and
regenerate replay from what the transcript shows, so re-expand the
invocation server-side before running the turn: the replayed prompt is
identical to the original and no client ever holds the body.
2026-07-28 03:44:17 -05:00
Brooklyn Nicholson c883367bd2 fix(agent): keep interrupt-checkpoint scaffolding out of the steered transcript
A mid-stream steer persists an interrupted-turn checkpoint so the model knows
its reply was cut off. That scaffolding — "[This response was interrupted by a
user correction.]" and the "Visible response before the interruption:" header —
was written straight into message content, so every reload painted the raw
machinery as an assistant bubble (and merged it into the preceding tool-call
bubble). Steered transcripts became unreadable.

Reuse the existing display/replay split instead of inventing new surface:
- Carry the scaffolded form in the server-only api_content sidecar (the exact
  bytes replayed to the provider), keep content the user's/agent's real words.
- When nothing reached the screen there is no clean form, so mark the row
  display_kind=hidden — replayed to the model, dropped by every transcript
  surface, exactly like compaction-reference rows.
- Honor display_kind=hidden in the gateway's _history_to_messages projection
  (it only sniffed the [System: convention), so the checkpoint can't leak
  through the live/resume path to the TUI/CLI either.

The model still receives the full interrupted context on the wire; the
transcript shows the partial reply and the user's correction.
2026-07-28 01:05:02 -05:00
brooklyn! 3c388db06b
Merge pull request #73128 from NousResearch/bb/steer-mid-stream
fix(agent): mid-stream steering survives retry/backoff; interrupt sentinel stays out of the transcript
2026-07-27 23:59:52 -05:00
Brooklyn Nicholson 12cd4ab423 fix(agent): steering corrections survive retry/backoff, and the interrupt sentinel stays out of the transcript
A mid-stream steer/redirect cancels only the live model request and queues
the correction for a rebuild. But the retry-wait, error-handling, and
backoff-wait paths all treated the cancellation bit as a hard stop:
clear_interrupt() destroyed the pending correction and the turn died with
"Operation interrupted…" — the user's message silently lost. All three
sites now preserve the redirect and rebuild the iteration from it, exactly
like the InterruptedError handler.

tui_gateway also gets the two suppressions its sibling surfaces already
had: the "Operation interrupted: waiting for model response (…)" sentinel
is cancellation metadata and no longer ships as assistant prose in
message.complete (gateway/run.py and ACP already suppress it), and a
leftover pending_steer returned by the turn is requeued as the next prompt
instead of dropped (cli.py and gateway/run.py already do this).

session.steer now records the correction on the inflight turn like
session.redirect does, so a resume/reconnect mid-turn rebuilds the steered
user bubble instead of losing it.
2026-07-27 23:47:57 -05:00
Alex Fournier eda54775e2 fix(relay): isolate streaming callback contexts
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 21:11:04 -07:00
Alex Fournier 14bed44c8c Reapply "feat(observability): integrate NeMo Relay runtime and shared metrics"
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 21:10:51 -07:00
Brooklyn Nicholson 045811f5be fix(context): keep @ref tokens inline instead of stripping them from the prompt
Expanding an @file:/@folder: reference deleted the token from the message
it was typed in, leaving a hole in the sentence and no anchor for clients
to chip. The attached context block still names each ref, so nothing is
lost by leaving the token where the user put it.
2026-07-27 22:55:18 -05:00
Hermes Agent 725c7ba534 refactor: single owner for empty-content wire repair (class fix)
The concept 'never send a turn that strict wire validation rejects as
empty' was forked across four sites, each with its own predicate and its
own blind spots:

1. build_assistant_message write-time ' ' pad — broke codex commentary
   turns (content:'' is a designed state), and a DB-side pad can't
   survive _rows_to_conversation's whitespace strip anyway. REMOVED.
2. conversation_loop send-time ' ' pad — main-loop only (summary path
   uncovered), ordering-fragile (had to run after whitespace
   normalization), assistant-only. REMOVED.
3. stream-stub '[response interrupted]' substitution — defeated the
   loop's empty-stub guard (the stub no longer looked empty, entered
   history, and the placeholder leaked into the stitched final
   response via truncated_response_parts). REMOVED.
4. repair_empty_non_final_messages in sanitize_api_messages — the
   unconditional pre-send chokepoint shared by the main loop AND the
   summary path, covers user and assistant turns, non-final only,
   copy-on-write. This is now the SINGLE OWNER.

The owner's payload predicate (_msg_has_payload) is extended to treat
codex_message_items / codex_reasoning_items as payload, so
designed-empty codex commentary turns are never rewritten on any
api_mode — the failure shape that broke site 1 in CI is encoded in the
owner, not special-cased at a call site.

Tests updated to pin the new contracts: builder stores textless turns
as-is; the empty stream stub stays recognizably empty for the loop
guard; poisoned resumed histories are repaired to the placeholder at
the send boundary; codex item carriers are never rewritten.
Sabotage-verified: unwiring the owner fails 3 regression tests.
2026-07-27 20:27:56 -07:00
Aaron Weiker df45811198 fix(errors): self-heal empty-content non-final messages before send
Third layer of the empty-stub fix: full self-recovery. A poisoned transcript
(empty assistant stub or empty user turn already persisted before the write-
time guard, or fed in from a host history) previously 400'd every subsequent
request until it scrolled out — needing a manual DB edit + gateway restart.

sanitize_api_messages() (the unconditional pre-send chokepoint) now repairs
empty non-final messages on the per-call copy by substituting a minimal
'[response interrupted]' placeholder, so the session recovers itself IN MEMORY
on the very next send. The final message is left untouched (empty final
assistant is legal); stored history is never mutated; reasoning-only and
tool_call turns are preserved (negative controls).

Tests: production-shape repro (tool -> empty assistant -> user), empty-user
case, non-destructive guarantee, and negative controls. RED verified by
disabling the wire-in.
2026-07-27 20:27:56 -07:00
Aaron Weiker 4587d77e0e fix(errors): log when the empty-stub guard and malformed-body classification fire
Add distinct, greppable log lines at both fix sites so the condition is
observable in the field instead of only inferable from the absence of the old
'Cannot compress further' spiral:

- chat_completion_helpers: warn when an empty partial-stream stub is replaced
  with the placeholder (0 chars recovered, no tool call).
- error_classifier: warn when a malformed-body 400 is classified as
  format_error rather than context overflow, with num_messages/approx_tokens.

Extend the litellm-shape classifier test to assert the warning is emitted.
2026-07-27 20:27:56 -07:00
Aaron Weiker 207a6c969d fix(errors): stop empty-content stream stubs from poisoning the transcript
A stream that dies after delivering deltas but with 0 recovered chars (and
no captured tool call) produced a content-less assistant stub. That empty
non-final message is invalid for the Anthropic schema, so every later request
400s with 'all messages must have non-empty content' (INVALID_REQUEST_BODY).
On a large session the 400 was misclassified as context overflow, dropping the
loop into a compression spiral that ends in 'Cannot compress further' — a
misleading context-size error on a session nowhere near its limit.

Root cause: substitute a minimal '[response interrupted]' placeholder so the
stub is never empty.

Misclassification: add _INVALID_MESSAGE_BODY_PATTERNS (checked before the
context-overflow heuristic) to classify these as non-retryable format_error,
and teach the body extractors the litellm/Bedrock errorMessage/errorCode/
errorArgs shape so descriptive proxy errors are not mistaken for generic ones.

Adds RED-verified regression tests for the stub path and three classifier tests
(including a guard that real overflows still compress).
2026-07-27 20:27:56 -07:00
Jeffrey Quesnelle 841a5a744a
Revert "feat(observability): integrate NeMo Relay runtime and shared metrics" 2026-07-27 22:28:08 -04:00
Hermes Agent 5e88745f12 fix: exempt codex_responses from the empty-assistant-content pads
Commentary-phase Codex turns persist with content:'' by design (their
text is delivered via the interim assistant callback), and the Responses
wire has no 'assistant must not be empty' validation — padding them
broke test_run_conversation_codex_continues_after_commentary_phase_message
in CI. Both the builder pad and the send-time pad now skip
api_mode=codex_responses. Keying on the ACTIVE api_mode preserves the
repair for codex-written sessions replayed through a strict
chat-completions provider.
2026-07-27 19:23:34 -07:00
0xprincess 309f06b044 fix: prevent session poisoning from empty partial-stream-stub assistant turns
A mid-tool-call stream drop with no delivered text produces a
partial-stream stub carrying content:'' and tool_calls=None.  The
conversation loop's truncation path appended it to history as
{"role":"assistant","content":""} before the continuation nudge, and
strict providers (Moonshot/Kimi via OpenRouter) reject empty assistant
content with HTTP 400 ("the message at position N with role 'assistant'
must not be empty") on the next replay.  Because the message is
persisted, every subsequent turn re-failed — the session was
unrecoverable.

Three layers, smallest blast radius first:

1. conversation_loop (length path): an EMPTY partial-stream stub is no
   longer appended as an interim assistant message; only the
   continuation user-message is.  Stubs that delivered partial text are
   still persisted so continuation stitching is unchanged.

2. chat_completion_helpers.build_assistant_message: never serialize a
   textless assistant turn with content:'' — pad to a single space, the
   same trick as the reasoning_content pad (#15250, #17400).  Tool-call
   turns are exempt (content:'' alongside tool_calls is accepted
   everywhere).

3. conversation_loop send boundary: pad a textless assistant turn's
   empty content to a single space AFTER all content-mutating passes
   (surrogate sanitize, whitespace normalization, thinking-only drops),
   before token estimation.  This is the durable repair for sessions
   ALREADY poisoned by older builds: the persisted stub rows are rebuilt
   to '' on every reload (_rows_to_conversation strips whitespace, so a
   DB-side pad can't survive) and only a send-time pad repairs them.

Verified: 485 tests pass across the four affected files; live replay of
a real poisoned session's resumed history against Moonshot via
OpenRouter returns HTTP 200 (was HTTP 400).
2026-07-27 19:23:34 -07:00
Alex Fournier 147e451cc8 Merge upstream main into feat/hermes-relay-shared-metrics 2026-07-27 17:54:42 -07:00
Jaaneek 5cffc53194 fix(xai): send Hermes-Agent User-Agent on chat/completions
Direct tool HTTP calls already identified as Hermes-Agent, but the main
OpenAI-SDK chat path still sent OpenAI/Python. Set Hermes-Agent/<ver> for
api.x.ai clients (xai + xai-oauth) so normal text traffic is attributed correctly.
2026-07-27 17:47:28 -07:00
Alex Fournier e23ef9f312 Merge upstream main into feat/hermes-relay-shared-metrics 2026-07-27 17:09:18 -07:00
teknium1 cf0c42fa0b fix(agent): never replay chain-of-thought in the active-turn redirect checkpoint
A /steer redirect during a thinking phase serialized the streamed
reasoning into the persisted assistant checkpoint ('Reasoning shown
before the interruption: ...'). An assistant turn exposing its own
chain-of-thought reads to Anthropic's output classifier as
reasoning-injection/prefill jailbreak, so every subsequent call on the
session deterministically returned 'Provider returned an empty
response' — and because the checkpoint is persisted and replayed, no
retry, nudge, or empty-recovery branch could ever escape it. Four
sessions were permanently bricked this way in the week of Jul 21-27
(42+ blocked calls; every reasoning-free checkpoint that week was
untouched — same mechanism as the prefill.json incident).

Class fix: streamed reasoning is now display-only state. The
_current_streamed_reasoning_text accumulator is removed entirely
(producer in _fire_reasoning_delta, resets, and init), so no future
path can serialize CoT into replayable content. The checkpoint keeps
only the visible response text; the model regenerates its reasoning on
the retried turn. Invariant documented in _apply_active_turn_redirect.

Regression tests: CoT never appears in either checkpoint shape,
reasoning-only interrupts produce a bare checkpoint, reasoning deltas
stay display-only.
2026-07-27 16:58:55 -07:00
Ben Barclay 981feb6730 feat(skills): org skills are editable in place; local edits survive org updates
The read-only org mirror broke the learning loop precisely where it matters
most. The system prompt tells every agent to patch a skill the moment it
finds a gap, and shared skills are the ones the most people use — but every
write to _org/ was refused, and the curator was excluded from them outright.
So org skills froze while personal skills kept improving, and the offered
alternative ("fork it into a personal skill, then propose the fork") is not
something an agent does mid-task. The refusal WAS the feature; improvements
were simply lost, and manual forks would have fragmented the shared set.

Edit in place:
- skill_manage patch/edit/write_file now work on org skills. Only delete is
  still refused (the mirror is a view of org HEAD — a local delete returns on
  the next pull; removing a shared skill is an admin action).
- Org skills are curation-eligible again, so the curator can improve the
  highest-leverage skills in the system instead of skipping them.
- The load-time provenance header now says edits are allowed and kept,
  instead of instructing the agent not to edit.

Local edits are never overwritten:
- pull_org_skills previously rmtree'd each skill dir and re-materialized it,
  silently destroying local work on the next session start. It now records a
  content fingerprint per skill (.org-baseline.json) when it writes one, and
  SKIPS any skill whose local content diverges from that baseline.
- When upstream ALSO changed such a skill, it is reported in the pull
  result's "conflicted" list and left untouched for the user to resolve
  deliberately (propose the local version, or delete it and re-pull to take
  theirs). A missing baseline is treated as unmodified so pre-existing
  mirrors do not raise phantom conflicts.
- Fingerprints are content-based (path + bytes, sorted), so a touch/mtime
  change is not mistaken for an edit.

Sharing back:
- Default: the edit stays local and the tool result tells the user to run
  "hermes skills propose <skill>".
- Opt-in sync.org_auto_propose / HERMES_SYNC_ORG_AUTO_PROPOSE submits each
  edit immediately. Defaults OFF — pushing every agent edit to a whole
  organisation is not a safe default. A failed submission never fails the
  edit; the change is saved and can be proposed later.
- "hermes sync status" lists org skills with unshared local edits;
  "hermes sync pull" reports conflicts it declined to overwrite.

Tests: 25 in the namespace suite (was 15). The two that asserted the old
read-only behaviour now assert the opposite. New coverage for edit-applied,
share-back guidance, delete-still-refused, curation-allowed, edit detection,
missing-baseline tolerance, mtime-insensitivity, and the auto-propose
default. 428 passed / 0 failed via scripts/run_tests.sh.

Verified through the REAL pull path against a mock plane: pull v1 -> edit in
place -> upstream ships v2 -> pull leaves the local edit intact, reports the
conflict, and surfaces it in status.
2026-07-28 09:08:04 +10:00
Alex Fournier 70d8db7409 Merge upstream main into feat/hermes-relay-shared-metrics 2026-07-27 15:08:25 -07:00
kshitijk4poor 551e1c6d64 refactor(agent): direct flag access in redecoration, matching call-block site
The call-block decoration reads agent._use_prompt_caching / _cache_ttl /
_use_native_cache_layout directly; the redecoration helper wrapped each in
getattr with divergent defaults (e.g. or-'5m' vs verbatim _cache_ttl).
The flags are unconditionally initialized on AIAgent, so the defaults
served only test fixtures and would mask a real init bug as silent
cache-off. Align with the house style.
2026-07-28 01:10:05 +05:30
kshitijk4poor 708390f47d refactor(moa): co-locate guidance peel with attach, add round-trip contract
_peel_moa_guidance hand-implemented the inverse of moa_loop's
_attach_reference_guidance from a different module — a drifting separator
or shape would make the peel silently no-op and put the last cache
breakpoint on the turn-varying guidance block (the #72626 bug class).
Move the inverse into moa_loop.peel_reference_guidance directly adjacent
to the attach, keep a thin wrapper in conversation_loop, and pin the
contract with a round-trip test over all three attach shapes.

Also fix the empty-list residue: peeling a guidance-only content part now
drops the whole message (mirroring the appended-user-message shape)
instead of leaving an empty-content user turn behind.
2026-07-28 01:10:05 +05:30
kshitijk4poor f9be15d0f9 fix(agent): rebase MoA prepared request even when guidance is empty
guidance=None is a real prepared shape (all references failed / silent
degraded policy builds prepared_request without attaching guidance), and
the MoA facade sends prepared['messages'] — not api_kwargs['messages'].
Gating the rebase on 'and guidance' left the stale decoration in the
prepared object for the no-guidance MoA sub-path, so #72626 persisted
there. rebase_prepared_request already handles falsy guidance (copies
messages, skips the attach).
2026-07-28 01:10:05 +05:30
kshitijk4poor bfd82660b5 refactor(agent): share static-prefix reconstruction, memoize failed rebuilds
The static-prefix reconstruction pattern (build_system_prompt_parts ->
['stable'] -> startswith gate -> fail-open) existed in three copies:
session restore (conversation_loop), compression keep-prompt path
(conversation_compression), and the new failover redecoration helper.
Hoist it into agent/system_prompt.reconstruct_static_prefix and call it
from all three sites.

Also memoize failed rebuilds per stored prompt (_static_rebuild_failed_for):
the redecoration chokepoint runs at the top of every retry attempt, and a
persistent stable-tier mismatch (restored session whose SOUL.md/skills
changed since save) would otherwise re-run the full prompt build — SOUL.md,
context files, memory I/O — on every attempt of every API call for the
life of the session. A legitimately changed stored prompt retries once.
2026-07-28 01:10:05 +05:30
kshitijk4poor 2322f0dcca fix(agent): restrict strip flatten to decoration-produced shapes
strip_anthropic_cache_control flattened ANY pure-text multi-part content
list with a separator-less join. Decoration only ever produces a single
text part or the 2-part [static, volatile] system split; organic
multi-part text (merged user turns, imported transcripts) got word-jammed
and parts carrying extra keys (citations) were silently dropped — on the
common no-failover path, since redecoration runs on every attempt.

Restrict the flatten to the exact decoration-produced shapes and make
marker removal copy-on-write on part dicts (the per-call message copy is
shallow, so parts alias the persistent history).
2026-07-28 01:10:05 +05:30
HexLab98 3e86df2753 fix(agent): redecorate prompt-cache breakpoints after provider failover
try_activate_fallback refreshes the cache policy flags for the new
provider, but the retry loop reused the primary's decorated api_messages.
Cache-off→cache-on shipped zero breakpoints; cache-on→cache-off left
stale markers. Strip and re-render at each retry attempt (same chokepoint
as reasoning-echo reapply), peel/rebase MoA guidance so the last marker
stays off the turn-varying block, and rebuild the static system prefix
when caching becomes active mid-turn (#72626).
2026-07-28 01:10:05 +05:30
kshitijk4poor 2c1809e6ca revert: PR #72817 — session activity watchdog, stall notify, compress timeout
Reverting #72817 (salvage of #72424) pending further review.
All 4 commits reverted: feat, refactor, chore (contributor map), CI fix.
2026-07-28 00:15:00 +05:00
kshitijk4poor c135b8543b refactor: reuse existing utilities in salvaged PR #72424
Three code-reuse fixes applied during salvage:

1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
   the relative-time formatting logic in hermes_cli/status.py.

2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
   to deduplicate the two nearly-identical try/except blocks that stamp
   compression timeout/abort provenance in the hygiene path.

3. Add ContextCompressor.record_timeout_failure() method and use it from
   the in-agent compress_context timeout callback instead of re-implementing
   the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
   exception handler already has this ladder — now both paths share one
   method.
2026-07-28 00:44:02 +05:30
fangliquanflq cfb206fe2e feat(gateway): session activity watchdog, stall notify, compress timeout (#72424)
Three mechanisms to detect and notify when gateway sessions stall silently:

1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
   and hermes status show progress during long turns without new message rows.

2. Stall watchdog: when a busy session has pending inbound and the shared
   activity clock is idle past agent.session_stall_timeout (default 300),
   log a WARNING and notify the user once to try /new. Notify-only; does
   not kill the turn.

3. Compaction timeout: fenceless compress_context callers get a progress-aware
   host budget (compression.context_timeout_seconds default 120 idle,
   compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
   cancel via commit fence, skip compaction without dropping messages, and
   continue the turn.

Closes #72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).

Cherry-picked from PR #72424 by @fangliquanflq.
2026-07-28 00:44:02 +05:30
Alex Fournier 8314854d52 Merge origin/main into feat/hermes-relay-shared-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 11:50:52 -07:00
kshitijk4poor 8e934e84ac fix: follow-ups for salvaged PR #72425
- codex app-server sibling path: surface a WARNING (was silent debug) when
  the projected-message flush fails — same bug class as the main fix, but
  codex output has already streamed so fail-closed and agent_persisted=False
  are both wrong here (#860/#42039 duplicate-write hazard); loud durability
  gap logging instead.
- map session_persistence_failed in _format_turn_completion_explanation so
  the user sees an actionable reason instead of 'The request failed:
  unknown error' + explainer test.
- contributors/emails mapping for elco@thedaoist.gg (attribution CI).
2026-07-28 00:14:19 +05:30
elcocoel 858bedea02 fix(session): persist tool activity before projection 2026-07-28 00:14:19 +05:30
kshitijk4poor 29abaeb98f fix(timeouts): add claude-fable to reasoning stale-timeout floor table
claude-fable-5 is a Mythos-class reasoning model (1M context, 128K output,
adaptive thinking per anthropic_adapter.py) but was missing from the
_REASONING_STALE_TIMEOUT_FLOORS table. Without a floor entry it got the
default 180s stale timeout (300s with context scaling), which is too short
for fable-5's thinking phase on large contexts.

Each stale kill bumped the cross-turn circuit breaker streak; after 5
consecutive kills _check_stale_giveup() fired immediately (elapsed: 0.00s),
aborting all calls with "Provider has been unresponsive for 5 consecutive
stale attempts." Users with 191K-token contexts hit this reliably.

Add ("claude-fable", 600) — deep-reasoning tier alongside o1/deepseek-r1/
nemotron-3-ultra. The claude-fable slug matches claude-fable-5 and future
variants via the existing right-anchor regex.
2026-07-27 23:09:47 +05:00
Alex Fournier 8ac686fa27 Merge remote-tracking branch 'origin/main' into fix/hermes-relay-review-round3
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	agent/chat_completion_helpers.py
2026-07-27 10:11:53 -07:00
rob-maron 02d5e23085 nous portal anthropic wire 2026-07-27 11:53:48 -04:00
Alex Fournier 8699ff58ed fix(relay): close child sessions inside turn ownership
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:20:05 -07:00
Alex Fournier 3be8a2f5f9 refactor(relay): serialize provider objects by capability
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:18:41 -07:00
Alex Fournier bfae9c2572 fix(relay): ignore rewrites after codec baseline failure
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:17:21 -07:00
Alex Fournier ec6c881453 fix(relay): preserve buffered provider stream chunks
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:16:16 -07:00
Alex Fournier bc597571fe perf(relay): bypass inactive execution adapters
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:14:50 -07:00
Alex Fournier 4fe4b0dca7 Merge origin/main into feat/hermes-relay-shared-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:12:37 -07:00
kshitij 8eaaa5021c fix(compression): update _pre_msg_count after durable adoption
Update _pre_msg_count after adopting the durable transcript so the
post-compression log reflects the correct pre-adoption message count.
Also use the existing _live_child_id() helper in the updated test
instead of hand-rolled child-id extraction.

Follow-up to #72631.
2026-07-27 19:34:54 +05:30
xxxigm 74ae2d3bf2 fix(compression): default in_place to True to match DEFAULT_CONFIG
is_truthy_value(..., default=False) and getattr(..., False) disagreed with
compression.in_place: true from #38763, so partial/failed config loads fell
back into rotation mode and re-armed the pre-lease drift path. Also report
compression.in_place in hermes dump overrides so stale false values are visible.
2026-07-27 19:34:54 +05:30
xxxigm e0a9a11466 fix(compression): adopt durable history when the session grows pre-lease
Busy sessions (memory review / shared session writers) kept outrunning the
in-memory snapshot, so rotation-mode compress aborted every attempt with
"changed before lease acquisition" and surfaced as a fake "No changes from
compression". Adopt the durable transcript and continue compressing instead
of returning the stale snapshot unchanged.
2026-07-27 19:34:54 +05:30
teknium1 39b5965569 refactor(fallback): single owner for backend identity and failure-scoped skips
Every fallback/dedup/skip decision asks one question — 'is this candidate
the same backend as the one that failed, along the axis that failure
invalidated?' — but it was re-implemented inline at six sites across four
subsystems, each comparing whatever string was locally convenient. Each
incident fixed one site while the others kept the bug: #22548, #70893,
#59561, #72468, #62984/#54250/#57584.

agent/backend_identity.py now owns the concept: BackendIdentity (provider /
model / base_url axes), FailureScope (MODEL / CREDENTIAL / ENDPOINT — each
failure class invalidates a different axis), and should_skip_candidate().
Unknown axes never manufacture a skip (over-skipping strands failover; a
wrong try costs one RTT).

Migrated sites:
- chat_completion_helpers.try_activate_fallback: replaces the provider+model
  early-exit (the #62984 bug: ignored base_url, stranding multi-endpoint
  pools) AND _fallback_entry_is_same_backend_by_base_url (deleted)
- auxiliary_client._try_configured_fallback_chain +
  _try_main_agent_model_fallback: replace label/model comparisons; auth and
  payment map to CREDENTIAL scope, keeping the #59561 carve-out
- hermes_cli/fallback_cmd add: primary-match + duplicate checks now identity-
  aware (#54250/#57584): same provider+model on a different explicit
  base_url is a pool entry, not a duplicate

_mark_provider_unhealthy stays label-keyed deliberately: its only triggers
are confirmed 402s, which ARE credential-scoped.

Owner-level tests pin each incident's semantics by number; sabotage-verified
(removing the base_url axis fails the #62984 test).
2026-07-26 23:41:54 -07:00
Ben Barclay 797c52b571 fix(sync): wire org skill pull into the runtime; scrub internal jargon
Two defects found by manual testing on the branch.

1. ORG SYNC NEVER RAN. The org pull/mirror/gating machinery was fully
   implemented and unit-tested but had ZERO runtime callers —
   maybe_pull_org_skills() was referenced only inside a comment, and
   `hermes sync` had no org path at all. Every code path fell through to
   personal sync (refs/user/<sub>/), so org skills never loaded and the
   feature looked like 'everything syncs to my personal org' even with a
   valid org token. The unit tests could not catch this: they invoked the
   functions directly, which is exactly the gap they left open.

   - cli.py session startup now calls maybe_pull_org_skills() alongside the
     personal maybe_pull_skills(), fail-quiet.
   - Auto-pull is gated on real org membership: resolve_org_identity()
     requires an org role on the token, only issued for multi-member orgs,
     so a solo account never reaches the network.
   - `hermes sync pull` refreshes the org mirror too (one pull, both
     surfaces) and reports what it refreshed.
   - `hermes sync status` exposes org_available/org_id/org_role/org_skills
     plus a plain-language summary, so a user can tell whether the org
     workflow applies instead of it being invisible.

2. INTERNAL JARGON LEAKED TO USERS. Help text and errors exposed internal
   milestone/spec coordinates: 'Propose a skill ... (M2)', 'Personal skill
   sync (HSP/1)', 'DEV-PHASE gate closed: your token lacks
   tool_gateway_admin', 'contract §4.3', and an inert message describing our
   internal personal-vs-multi-member design split. All rewritten in user
   language. Feature-local comments/docstrings lost their internal
   coordinates (§N, M1/M2, design.md, PR numbers) while keeping the
   explanatory prose. Pre-existing issue references elsewhere in the tree
   were deliberately left untouched.

Tests: 4 new guards, including two that assert the CALL SITES exist so the
org pull cannot silently become dead code again (verified failing when the
wiring is removed) and one that fails if user-facing help leaks jargon.
344 passed across the sync/skills/prompt suites.

Verified against live staging with a real org token: sync status reports
org_available=true, org_role=OWNER; sync pull performs the org refresh; the
.active_org marker is written with the org id from the token.
2026-07-27 16:40:16 +10:00
teknium1 98fe6d0a8e fix(delegation): integrate lifecycle refactor with tool-history + daemon pool
Follow-ups on the salvaged #63359:
- _finalize_child_results carries tool_call_history on subagent_stop
  (the #62011/#72403 field landed after the PR branched; the shared
  pipeline must emit it for both delegate_task and plugin-launched
  children). Lifecycle test updated for the new payload field.
- The lifecycle executor uses DaemonThreadPoolExecutor — a wedged or
  abandoned child must never block interpreter exit at atexit-join time
  (same rationale as _run_single_child's timeout executor and the
  async-delegation pool).
- delegate_task's batch path keeps live-transcript wiring while routing
  child construction through the shared
  _build_child_preserving_parent_tools helper.
2026-07-26 23:27:30 -07:00
Tony Simons f60abd6e37 fix subagent lifecycle ownership invariants 2026-07-26 23:27:30 -07:00
Tony Simons 1865fb5fcd feat(plugins): add public subagent lifecycle API 2026-07-26 23:27:30 -07:00
teknium1 0a2c245cd6 fix(auxiliary): reach the main agent model when a sibling aux model fails on the same provider
Widen okalentiev's failed_model narrowing (#59561) to
_try_main_agent_model_fallback. The safety-net layer still skipped on a
provider-label match alone, so single-provider users whose aux compression
model and main model share one custom endpoint had ZERO fallbacks: the aux
model timing out exhausted the chain in one hop and compression aborted.

Real incident (0.19.0 debug dump): aux zai-org/glm-5.2 hung 324s and timed
out while main mindai/macaron-v1-venti on the SAME endpoint was serving
448K-token turns — the label-only skip discarded the one viable summarizer,
the session wedged over threshold, and the anti-thrash breaker tripped.

Same convention as the chain fix: model-specific failures (timeout,
connection, rate limit) pass failed_model so only the exact failed model is
skipped; provider-wide failures (auth 401 / payment 402) pass None and keep
the whole-provider skip. Both sync and async call_llm sites pass it.

Sabotage-verified: the new regression test fails on the provider-only skip.
2026-07-26 22:33:06 -07:00
Oleksii Kalentiev 83f20e07e0 fix(auxiliary): keep provider-wide skip for auth/payment failures in fallback chain
Follow-up to the same-provider fallback fix: narrowing the configured-chain
skip to the exact failed model is only correct for model-specific failures.
Auth (401) and payment (402) errors are provider-wide — every model on the
provider shares the same broken credentials/account — so trying a sibling
model can't recover and merely burns another doomed request before the
aux task fails. Worse, returning that sibling client bypasses the
main-agent-model safety net that a provider-wide skip would have reached.

Only forward failed_model to _try_configured_fallback_chain for
model-specific failures (timeout, connection, rate limit, model-incompatible,
invalid response). Auth/payment keep failed_model=None (whole-provider skip),
preserving the pre-existing safety-net behaviour for credential/billing
failures.

Adds an integration test that a timeout forwards the failed model, and
updates the payment-error test to assert failed_model=None.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 22:33:06 -07:00
Oleksii Kalentiev e8f8b34b0c fix(auxiliary): don't skip sibling models when a configured fallback_chain reuses the same provider
_try_configured_fallback_chain skipped every fallback_chain entry whose
provider matched the one that just failed. A chain that intentionally lists
several models under the same provider (e.g. two more NVIDIA NIM models
after the primary NIM model times out) was therefore skipped wholesale,
falling straight through to the main-agent-model safety net instead of
trying the other configured models on that provider.

Add failed_model so the skip narrows to the exact (provider, model) pair
that failed. Callers that only know the provider (client-build failures,
where the whole provider is unreachable regardless of model) keep the old
provider-wide skip; the two runtime request-error call sites (call_llm,
async_call_llm) now pass the model that just failed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-07-26 22:33:06 -07:00
teknium1 cb06017b1d refactor(guardrails): make runaway-loop caps per-turn, not session-total
Per Teknium: the caps should bound a single agent loop, not accumulate
over the whole session. Rename SessionCapConfig -> LoopCapConfig and the
config section session_caps -> loop_caps; move the counters into
reset_for_turn (invoked per turn via turn_context) so each turn starts
with a fresh budget; retune defaults 200 -> 50 (a single turn issuing 50
web searches / spawning 50 subagents is already pathological). Block
codes session_*_cap -> loop_*_cap and messages updated to drop the
/new-resets-the-budget guidance (irrelevant now that it resets per turn).
Tests flipped: the old persists-across-turn-resets assertion becomes
resets-each-turn.
2026-07-26 21:27:45 -07:00
teknium1 b68787ad25 Inspired by Claude Code: session-wide runaway-loop caps for web_search and delegate_task
Add per-session lifetime caps on web_search calls and subagent spawns
(defaults 200/200, matching Claude Code v2.1.212). Unlike the existing
per-turn tool-loop guardrails, these count over the whole session and
reset only when a fresh agent is built (/new, /clear). Hitting a cap
blocks the offending call and halts the turn cleanly.

- agent/tool_guardrails.py: SessionCapConfig + session counters on the
  controller (in __init__, not reset_for_turn, so they persist across
  turns). before_call() enforces caps first, independent of
  hard_stop_enabled. delegate_task batches count each task.
- hermes_cli/config.py: tool_loop_guardrails.session_caps defaults.
- docs + tests (unit + E2E validated against a real AIAgent).
2026-07-26 21:27:45 -07:00
teknium1 8fbe2e388f feat(tool_search): probe-validate blind tool_call args against the deferred schema
Port from nearai/ironclaw#5149 (the describe-first live-hardening fix in
their progressive tool disclosure work): when a model invokes a deferred
tool through the tool_call bridge without the schema-required arguments,
return the tool's parameter schema instead of dispatching blind.

Pre-fix, a blind call produced an opaque downstream failure
("[TOOL_ERROR] Tool execution failed: KeyError: 'document_id'") that
teaches the model nothing about what the tool expects — IronClaw observed
cheap models looping ~30 identical invalid calls until the iteration
budget died. Post-fix, the model repairs the call in one round-trip.

- tools/tool_search.py: new validate_deferred_call_args() — key-absence
  check of schema 'required' fields only; no type checking (coerce_tool_args
  already repairs types downstream); fails open on any validator error so
  it can never block a legitimate dispatch.
- model_tools.py: probe after the scope gate in the bridge dispatch.
- agent/tool_executor.py: probe in both unwrap sites (concurrent +
  sequential) before the underlying tool replaces the bridge; sequential
  path flattens the payload to match its {"error": str} wrapping.
- tests: TestDeferredCallSchemaProbe — blind call returns schema (not
  KeyError), valid/optional calls dispatch, unvalidatable tools fail open,
  out-of-scope rejection unchanged.
2026-07-26 20:59:36 -07:00
Teknium 9b97dea1e6 fix(skills): parse stored GitHub credentials without scanner false positives
Co-authored-by: Syed Annas <28944679+AnnasMazhar@users.noreply.github.com>
Co-authored-by: Bryan Neva <13835061+bryanneva@users.noreply.github.com>
2026-07-26 20:59:26 -07:00
Teknium 214ae7b77c feat(agent): surface a labeled reasoning excerpt at the empty-response terminal
When the empty-response ladder is fully exhausted (thinking-prefill
continuation, empty-content retries, provider fallback) and the model
produced structured reasoning but never any visible text, deliver a
clearly labeled excerpt of that reasoning instead of a bare '(empty)' —
the reasoning frequently contains the actual answer.

Delivery-only by design: raw chain-of-thought is never promoted to a
normal answer earlier in the ladder (prefill continuation still gets
first crack, retries and fallback still run), transcript persistence
semantics are untouched (the '(empty)' sentinel scaffolding keeps its
replay-safety behavior), and a truly empty exhaustion still returns the
existing terminal.

Idea credit: PR #48795 (@ligl0325) proposed falling back to
reasoning_content on empty content; this lands the safe kernel of that
idea at the one point in the ladder where it is strictly an improvement.
2026-07-26 20:59:21 -07:00
Teknium a751924c04 fix(gemini): preserve typed enum constraints as strings
Port from openclaw/openclaw#104567: Gemini requires enum metadata to be strings even when the declared tool parameter type is numeric or boolean.
2026-07-26 20:59:16 -07:00
Teknium b41eee450b fix(redact): stop masking prose words that embed a secret keyword (Secretary, tokenizer, author=)
Port from nearai/ironclaw#6129: their sensitive-marker scrubber matched
markers as bare substrings, so tool results containing 'Secretary of the
Treasury' were scrubbed as 'secret' on replay, evicting legitimate content
and forcing the model into a re-fetch loop. Hermes' lowercase/dotted/YAML
config-key redaction patterns (_CFG_DOTTED_RE, _CFG_ANCHORED_RE,
_YAML_ASSIGN_RE) had the same false-positive class: their key classes allow
arbitrary alphanumeric affixes around the keyword, so ordinary document
text like 'Secretary: J.Smith', 'tokenizer: cl100k_base' (HF model cards),
and BibTeX 'author=Smith' got value-masked on the surfaces that run these
passes (browser snapshots, log lines, kanban summaries, CLI-echoed output).

Fix: post-match word-boundary validation of the keyword occurrence inside
the matched key. Boundaries: key edges, non-letters (_ - . digits),
camelCase transitions (clientSecret, secretKey, APIToken), plural 's'
(secrets:, tokens:). Concatenated real-world compounds keep matching via
explicit alternatives (authtoken, authkey, secretkey, accesstoken). ALL-CAPS
keys keep legacy embedded matching (MYTOKEN=...) — all-caps is almost never
prose, same rationale as _ENV_ASSIGN_RE. Same discipline the file already
applies to exact-match body/query keys (ported from ironclaw#2529) and the
deliberate 'auth' exclusion that keeps 'author:' from matching.
2026-07-26 20:59:11 -07:00
Teknium 474c84ed8d fix(agent): uniquify duplicate tool-call ids to keep call/result pairing lossless
Port from openclaw/openclaw#110518 / #110956: some models reuse one call id
for different tool calls in a single batch (native Kimi Responses replays,
Ollama-compatible endpoints, degraded models at long context). Hermes kept
both calls but the pre-API sanitizer then dropped the later call/result pair
per id (#58327), so the second call's output silently vanished from every
replayed payload — the model never saw it and confabulated.

_uniquify_tool_call_ids renames later collisions to a deterministic <id>_d<n>
suffix at ingestion, before validation/dispatch/history build, so both pairs
survive. Composite Responses ids collide on the call half and keep their
response-item half. Deterministic suffixes preserve prompt-cache prefix
stability (no random UUIDs).
2026-07-26 20:59:06 -07:00
Teknium d4381f0e39 fix(gemini): explain legacy Standard-key 401 rejections with migration guidance
Port from Kilo-Org/kilocode#12162.

Google began rejecting unrestricted legacy 'Standard' Google Cloud API keys
on the Gemini API on June 19, 2026 (all Standard keys stop working in
September 2026). The rejection is a 401 whose message misleadingly tells the
user to supply an OAuth 2 access token. gemini_http_error() now appends
actionable guidance (check key type in AI Studio, mint a new Gemini API key,
temporary restriction bridge) on that narrow shape — matched via
google.rpc.ErrorInfo reason ACCESS_TOKEN_TYPE_UNSUPPORTED or the
'Expected OAuth 2 access token' signature. Plain invalid keys
(API_KEY_INVALID) keep their existing message.

Also fixes a latent sibling gap: _summarize_api_error() preferred re-extracting
the raw response body for errors carrying .response, which stripped adapter-
composed guidance (this one AND the existing free-tier 429 guidance) from the
user-facing summary. GeminiAPIError now surfaces its composed message.
2026-07-26 20:58:57 -07:00
Teknium 53bfe40a35 fix(errors): classify throttle messages before token-overflow patterns; add new overflow shapes
Port from anomalyco/opencode#37848 (+ dev-branch twin #37840): expand
context-overflow patterns and guard against rate-limit messages that
mention tokens.

- 'Throttling error: Too many tokens, please wait before trying again.'
  (AWS Bedrock / proxy shape) classified as context_overflow and routed a
  healthy session into compression on every throttle. Added 'throttling'
  to _RATE_LIMIT_PATTERNS, which the message-only path checks BEFORE the
  overflow list.
- 'Input length N exceeds the maximum allowed input length of M tokens.'
  (Together/Fireworks shape) fell through to unknown — no compression
  recovery. Added 'maximum allowed input length' to overflow patterns.
- 'request_too_large' / 'Request exceeds the maximum size' (Anthropic 413
  type re-wrapped without a status code by aggregators/proxies) fell
  through to unknown. Added to _PAYLOAD_TOO_LARGE_PATTERNS.

All three shapes proven live on main before the fix; 265 classifier +
bedrock tests and 238 sibling rate-guard/compression tests pass.
2026-07-26 20:58:48 -07:00
teknium1 ece050ac30 fix(delegation): route delegated-child API calls inline to avoid nested-pool wedge (#60203)
Root cause: delegate_task children run through three nested daemon-thread
layers (async-delegation executor -> per-child timeout executor -> the
interrupt worker interruptible_api_call spawns). After multi-day gateway
uptime the deepest layer wedges BEFORE the socket opens — the same
fingerprint as the gateway-cron hang (#62151): zero stale-detector output
(the worker never reaches dispatch), all providers, foreground/restart
works. The cron fix (should_use_direct_api_call) explicitly excluded
delegation 'for lack of evidence' — #60203 is that evidence.

- should_use_direct_api_call: extend the inline gate to delegated
  children, detected via the delegation ContextVar set by
  _run_single_child (platform='subagent' stamp as fallback). Scope
  unchanged otherwise: chat_completions wire only; Codex/Anthropic/
  Bedrock/MoA keep their established workers. Interrupts still work —
  the inline path registers _active_request_abort, which interrupt()
  invokes cross-thread (same mechanism the #72227 stall monitor uses).
- _dump_subagent_timeout_diagnostic: dump ALL thread stacks (bounded,
  40), not just the conversation worker — a pre-HTTP wedge is
  indistinguishable from a slow provider without seeing where the
  nested helper threads sit.
2026-07-26 20:37:51 -07:00
embwl0x e369d6ea3f feat(delegate): expose redacted child tool history 2026-07-26 20:36:47 -07:00
Jeffrey Quesnelle 9216198601
Merge branch 'main' into feat/hermes-relay-shared-metrics 2026-07-26 23:14:26 -04:00
dsad 3c4220cd9c fix(agent): keep system cache breakpoints across provider failover
`apply_anthropic_cache_control` runs once per call block, before the retry
loop, and splits the system prompt into `[static prefix, volatile tail]` text
blocks carrying the cache_control breakpoints.

A failover fires *inside* that retry loop, and `_sync_failover_system_message`
assigns a bare string over `api_messages[0]["content"]` to refresh the
`Model:`/`Provider:` identity lines. That drops the block list and both
breakpoints. `convert_messages_to_anthropic` only emits system cache blocks for
list content (`isinstance(content, list)`), so the retried request ships
`system` as one plain string: zero breakpoints, nothing written to cache, and
the whole system prompt re-billed at full write price. The next call misses
too, so a failover costs two full-price prompts instead of one write + one
read.

Measured with the real functions, same history, native Anthropic layout:

  normal turn       system=BLOCK LIST(2)  system breakpoints=2
  after failover    system=plain string   system breakpoints=0

`_sync_failover_system_message`'s own docstring notes this fires on "every
gateway turn, since fallback re-activates per message while the primary is
down" -- so a gateway running on a degraded primary pays it on every message.

Fix: rewrite the decorated blocks in place instead of flattening them.
`rewrite_prompt_model_identity` only touches the LAST `Model:`/`Provider:`
lines, and those live in the volatile tail, so the static prefix stays
byte-identical and its cache entry keeps matching -- the failover retry keeps
both breakpoints AND the warm prefix. Shapes we cannot safely patch fall back
to the existing plain-string assignment.

This is the same class the retry loop already guards against eight lines below
the gap, for reasoning fields:

    # api_messages is built once, before this retry loop, while the primary
    # provider is active. [...] so the fallback request isn't sent with stale,
    # primary-shaped reasoning fields.
    agent._reapply_reasoning_echo_for_provider(api_messages)

There was no equivalent for cache decoration.
2026-07-26 19:42:43 -07:00
dsad c3e99fce49 fix(anthropic): keep the assistant cache breakpoint on the ordered-replay path
`apply_anthropic_cache_control` marks an assistant turn with non-empty text by
writing `cache_control` INTO `content` -- `_apply_cache_marker`'s list branch
puts it on the last content block, not at the top level.

`_convert_assistant_message`'s ordered-replay branch rebuilds the message from
`anthropic_content_blocks` and returns early. Its only cache sources are
`_relocated_replay_cache_control` (markers rescued from blocks the replay
sanitizer dropped) and the top-level `m["cache_control"]`; it never reads
`m["content"]`. So for an assistant turn that interleaves signed thinking with
a tool_use AND has preamble text, the breakpoint is dropped.

It is burned, not relocated: `_can_carry_marker` returns True for this message
(non-empty content), so the breakpoint budget already counted it. Hermes'
accounting believes the marker landed.

Measured with the real functions, native layout, same history, only
`anthropic_content_blocks` differing:

    normal path   4 breakpoints   assistant text block carries cache_control
    replay path   3 breakpoints   assistant carries NONE

Under the static-prefix layout only two conversation-tier breakpoints exist, so
this halves them. Nothing is logged and the request succeeds with identical
model output, which is why it survives: it recurs on every request for the life
of any Claude thinking+tools session, since the flag rides the message in
`agent.messages`.

#56195 fixed the complementary shape -- blank assistant content, where the
marker lands top-level -- and its regression test pins `"content": ""`. The
non-empty case is the one a Claude 4.5/4.6 tool turn normally produces (a
sentence of preamble before the call) and was never covered.

Harvest an in-`content` marker next to the existing top-level lookup and hand
it to the same `_apply_assistant_cache_control_to_last_cacheable_block` helper.
2026-07-26 19:29:51 -07:00
teknium1 07370a9dba feat(cli,gateway): unify /context into a visual context-usage breakdown
Extends the cherry-picked /context command (PR #52184) and prompt-size
attribution helpers (PR #66656) into one visual context view across
surfaces, and absorbs the per-component budget-visibility goal of the
/tokens proposal (PR #48470):

- agent/context_breakdown.py: pure renderers over the existing payload —
  a 5x20 glyph block grid (1 cell ~= 1% of the model window), an
  'Estimated usage by category' table with free space, and expanded
  per-skill / per-toolset listings via compute_context_details(), which
  reuses the prompt-size attribution mechanism (skills index-line bytes +
  registry tool->toolset map) converted to the same chars/4 heuristic.
- cli.py: /context [all] renders grid + category table (+ expanded
  listings) from the live agent and in-memory conversation history.
- gateway/slash_commands.py: /context appends the plain-text category
  table (no grid — monospace not guaranteed on messaging platforms);
  /context all adds the expanded listings. Fail-open: breakdown errors
  never break the gauge.
- hermes_cli/commands.py: /context gains the 'all' subcommand; /version
  demoted to /hermes version on Slack to keep the 50-slash cap.
- tests: renderer unit tests against synthetic payloads, registry test,
  gateway /context + /context all + failure-degradation handler tests.
- docs: slash-commands reference + CLI guide entries.

Read-only and locally computed: no provider calls, no prompt-cache impact.

Co-authored-by: RemyFevry <29257684+RemyFevry@users.noreply.github.com>
Co-authored-by: joelbrilliant <joelbrilliant1@gmail.com>
Co-authored-by: CharlesMcquade <6466275+CharlesMcquade@users.noreply.github.com>
2026-07-26 18:06:21 -07:00
teknium1 9ca33680ea fix(cli): report unknown line deltas for content-free diffs instead of +0 -0
Found by E2E-rendering the collector against realistic tool payloads.
2026-07-26 17:47:51 -07:00
teknium1 ce997f9e62 feat(cli): per-turn summary line and live token flow in the spinner
Ports Claude Code's post-turn accounting (Edited N files +X -Y · Worked for Ns). Display-only, quiet-mode aware, config-gated.
2026-07-26 17:47:51 -07:00
teknium1 0b670f0539 fix(clarify): route multi_select through current dispatch paths and harden callback detection
Follow-ups to the salvaged #23768 commit, which targeted a pre-79559214
codebase:

- agent/tool_executor.py + agent/agent_runtime_helpers.py: pass
  multi_select at both current clarify dispatch points (the PR's
  run_agent.py edits landed on dead code paths).
- tools/clarify_tool.py: replace the broad TypeError-retry in
  _invoke_callback with inspect.signature detection, so a compatible
  callback that raises TypeError internally is not invoked twice
  (addresses hermes-sweeper review feedback on #23768).
- tests: cover single-invocation on internal TypeError, legacy 2-arg
  callbacks, **kwargs callbacks, and registry handler multi_select
  pass-through (schema arg → handler → callback).
2026-07-26 17:46:55 -07:00
Siddharth Balyan 59529afee0
feat(billing): carry the payment-method union through to clients (#71542)
* feat(billing): add payment_method union to the billing-state wire type

* feat(billing): carry the payment-method union through the gateway

NAS now sends a typed `paymentMethod` union on /api/billing/state alongside
the legacy `card` field. The gateway parses payloads field-by-field, so the
new field was dropped on the floor before reaching TUI/Desktop.

Parse it into PaymentMethodInfo and re-emit it as snake_case `payment_method`,
matching the translation the rest of this payload already does. The payment
method id is deliberately not carried through — clients have no use for it.

No client rendering changes: `card` stays populated for cards, so every
existing consumer behaves exactly as before and the new field is inert until
a surface opts into reading it.

* fix(billing): send only the fields each payment-method kind declares

The serializer emitted every key for every kind, so a Link method went out
carrying brand, last4 and wallet set to null. That contradicts the shared
type, where each kind declares its own fields: a client testing `'brand' in
pm` would read every Link method as a card, and one trusting the declared
non-null `brand` could crash on it.

Send each kind's own fields, and forward an unrecognized kind by name alone
so a client that predates it can still say something honest. The shared type
gains the matching fallback arm its own comment already promised.

Tests now follow a payload from the server response through to the client
wire for each kind, rather than checking parsing and serializing separately —
which is why the old expectation locked in the wrong shape without noticing.

* fix(billing): keep the payment-method kind narrowable

Typing the fallback arm's kind as `string & {}` borrowed a trick that only
works on unions of plain strings. On a union of objects it makes the
discriminant non-literal, so TypeScript stops narrowing on every arm — even
`if (pm.kind === 'card')` no longer gives you `brand`. The first client to
use this would have hit a compile error and reached for a cast.

An unrecognised kind now arrives as `unknown`, carrying the real name
alongside it, so every arm has a literal discriminant. A type-level test
pins this: it fails to compile if the discriminant stops narrowing.

The parser settles which kind it is, the way the card parser already does,
so the record cannot hold fields that do not belong to its kind and the
serializer no longer re-checks. The type comment also stops claiming `card`
is a safe signal — it is null for Link, so `!card` does not mean "nothing on
file".
2026-07-27 01:49:43 +05:30
teknium1 1b081e4891 feat: raise default tool-calling iteration limit from 90 to 500
The default max_iterations/agent.max_turns budget was set when long
agentic runs were rare; complex tasks now routinely exceed 90 tool
calls. Raise the default to 500 across every surface that hardcodes
the fallback: AIAgent constructor, DEFAULT_CONFIG, CLI resolution
chain, gateway env bridge, cron scheduler, and TUI gateway. Explicit
user config values are unaffected (deep-merge preserves them; no
_config_version bump needed).

Docs (en + zh-Hans), CLI help text, tips, and pinned tests updated
to match.
2026-07-26 12:54:45 -07:00
Alex Fournier 45580cc93a Merge origin/main into feat/hermes-relay-shared-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-26 09:21:19 -07:00
Brooklyn Nicholson dcf5fc0eea fix(sessions): stop titling a session after the skill it invoked
generate_title() sent the first 500 characters of the user turn to the
auxiliary model. On a /skill invocation those characters are the skill's own
opening prose, so the session got named after the skill instead of the request
— /work sessions came back as "Isolated Git Worktree Setup".

Route the turn through describe_skill_invocation() first, so the titler sees
what the user typed. Also keep only the first line of the response: a model
that ignores "return ONLY the title" and answers the prompt would otherwise
have a shell transcript stored as the title, truncated mid-command.
2026-07-26 02:58:16 -05:00
Brooklyn Nicholson e4724ea455 feat(skills): describe a /skill turn the way the user typed it
A /skill invocation expands into a message that embeds the whole skill body.
Anything that summarizes a user turn from its raw content reads the skill's
prose as if the user had written it.

describe_skill_invocation() sits next to the existing extractor and reuses its
markers, returning `/work — fix the title leak` for an invocation with an
instruction and `/work` for a bare one. It also exports the SQL LIKE pattern
and excerpt-joint sentinel that listing queries need to recognize scaffolding
before a row reaches Python.
2026-07-26 02:58:12 -05:00
teknium1 72c013b662 docs(compression): correct the stale in_place default in comments
2107b86024 flipped compression.in_place to True but left both explanatory
comments reading "Default False during rollout". The contradiction is
load-bearing: it is why two recent PRs (#71747, #48951) were built on the
premise that agent.session_id still rotates at every compaction.

Comment-only; no behavior change.
2026-07-25 22:47:07 -07:00
joaomarcos 11c487e409 fix(compression-lock): reclaim crashed holders instead of stranding the lease
The compression lock was gated so that any holder on a Windows host was
assumed alive until the full TTL expired. A crashed holder therefore
stranded every other agent on the session for the whole lease window.

Probe liveness with psutil.pid_exists, which is safe on nt. Keep the
os.kill(pid, 0) path strictly for POSIX: on Windows signal 0 maps to
CTRL_C_EVENT (bpo-14484) and can kill the target's console group, so with
psutil absent the only safe answer stays 'assume alive' and let the TTL
run out.

Also carry the commit fence through cancelled compressions so an aborted
run leaves the lock reacquirable rather than half-held.
2026-07-25 22:47:07 -07:00
Teknium 243a01d5d7 fix(curator): make the autonomous write policy consistent (#67140)
The background write guard decided ownership from `isinstance(usage_rec, dict)`,
so a local skill with NO usage record passed. That successful write called
bump_patch(), which created a `created_by: null` record — and the identical
write was refused from then on. "Allowed exactly once, then never" is a race
with our own bookkeeping, not a policy. Reproduced on main: patch #1 succeeds,
patch #2 with the same arguments is refused.

Option B from the issue. Option A (split `session_review` from
`scheduled_curator` and let the session fork patch user-owned skills it
consulted) would widen autonomous write permission onto skills the user owns
with no user present to consent — wrong direction for a no-user-present actor.

- skill_manager_tool: missing and explicit-null records now resolve
  IDENTICALLY, both fail closed. The refusal names the reason and points at
  `hermes curator adopt <name>`.
- background_review: both review prompts told the reviewer to patch any skill
  consulted in the session and claimed pinned skills could be improved, while
  enforcement refused both. Prompts now list pinned, external, and user-owned
  skills as protected, and tell the reviewer to RECOMMEND adoption instead of
  attempting a write that will be refused.
- skill_usage: document that `created_by` is a curator-management policy flag,
  not a provenance claim, and add `is_curator_managed()` so call sites read as
  the question they ask. Field name retained — it is on disk in every
  `.usage.json` and renaming would strand those records.
- curator CLI: `hermes curator list-unmanaged` itemizes unmanaged skills with
  the reason each is unmanaged (completes the #67139 spec).

Foreground writes are untouched: a user-directed edit to a user-owned skill
still works, including on pinned skills.

Sibling tests: 9 failures in test_skill_manager_tool.py were fixtures that
created record-less skills to exercise OTHER guards (consolidation-delete,
read-before-write) and relied on ownership falling through. Fixed at the
fixture, since the real curator only ever operates on managed sediment. One
test asserted the old "manually authored" wording; rewritten to assert the
behavior contract instead of the string.

Validation: 274 targeted tests + all 7 background-review files (60 tests) pass.
E2E on a temp HERMES_HOME (30 checks) covers the flip, foreground writes,
adoption unblocking, pin semantics, prompt/enforcement parity, and the new verb.
Each new test sabotage-verified: revert the fix, confirm it goes red.

Fixes #67140
2026-07-25 19:27:17 -07:00
teknium1 cfd2a38223 fix(conversation): anchor the cwd staleness read to the host-info block
The whole-prompt scan for "Current working directory:" matched USER project
content, not just Hermes' own host-info line. The prompt embeds AGENTS.md /
CLAUDE.md / .cursorrules in the context tier, which sits AFTER the host-info
block, and line_value() takes the LAST match — so any project file containing
a line starting with that label won the scan.

The comparison then ran runtime state against project prose, a mismatch that
never clears: the stored prompt was rejected on EVERY turn, rebuilding the
system prompt each message and destroying the prefix cache for the whole
session. Strictly worse than the staleness the check exists to catch, and
invisible to CI because no test encoded the invariant.

"last match wins" was only ever safe because Model/Provider/Platform live in
the volatile tier at the very END of the prompt. The cwd line is the one field
read from the STABLE tier near the start, so it needs an anchored read:
locate Hermes' own "User home directory:" line and take the working-directory
line that follows it.

Verified against the real builder on a real AIAgent, not a stub:
  - stable cwd, clean project           -> prompt REUSED (0 rebuilds)
  - stable cwd, AGENTS.md naming the
    field                               -> prompt REUSED (was: rebuild/turn)
  - drifted cwd                         -> rebuilt
  - drifted cwd, AGENTS.md naming the
    NEW cwd                             -> rebuilt (prose can't mask drift)

Also re-verified no spurious rebuild across all five cwd-resolution paths
(pinned contextvar, TERMINAL_CWD, launch dir, symlinked workspace,
trailing-slash cwd).

The contributor's fixtures omitted the host-info anchor, so they silently
stopped exercising the cwd path once the read was anchored; reshaped them to
match real prompt structure via a _host_block() helper.
2026-07-25 19:25:34 -07:00
MorAlekss 7a61b66dac fix(conversation): use resolve_agent_cwd() and Platform line for stored-prompt staleness check 2026-07-25 19:25:34 -07:00
MorAlekss a2b0d6d8e4 fix(conversation): prevent stale prefix cache on cwd or runtime surface change 2026-07-25 19:25:34 -07:00
webtecnica b9fedab47a fix: curator labels bundled skills as agent-created (#64393) 2026-07-25 18:10:24 -07:00
Teknium 9de7dfe1cc feat(compression): stream the summary call on every compression path
The progress-hook streaming from #71508 only activated when a
CompressionCommitFence was present (gateway session hygiene). CLI
/compress and in-loop auto-compression still used the plain
non-streaming summary call, where the SDK timeout is inactivity-based —
a byte-trickling provider that keeps the connection alive could outlive
auxiliary.compression.timeout indefinitely (the gap #69192/#41397 were
built to close).

Fenceless compression callers now install a no-op progress hook, which
routes their summary call onto the same streamed path: the configured
timeout acts on inactivity (slow models finish instead of being cut
off mid-generation), and a degenerate trickle stream is bounded by the
streamed total ceiling (max(600s, 4× the task timeout)) instead of
running forever. No config knob needed — the ceiling machinery ships
with the streaming layer and applies uniformly.

Supersedes the opt-in wall-clock deadline approaches in PR #69192
(@JabberELF) and PR #41397: same guarantee (bounded total compression
wall time even while bytes move) without a daemonized watchdog thread
or a new config surface, and without punishing slow-but-healthy models.
2026-07-25 14:58:04 -07:00
kudi88 5121a2a20e feat(aux): force streaming for providers that reject non-stream requests
Some OpenAI-compatible endpoints — notably Tencent Copilot
(copilot.tencent.com) — only accept streaming chat requests; any
non-streaming call returns HTTP 400 (code 11101, 'Non-stream chat
request is currently not supported'). The main conversation loop already
streams, so interactive chat works, but every auxiliary task (title
generation, compression, web extraction) used the non-streaming path and
failed on each call.

_provider_requires_stream() detects stream-only endpoints
(copilot.tencent.com built in, plus user-configurable
auxiliary.stream_only_base_urls substring markers in config.yaml).
Matching sync auxiliary calls route through _create_with_progress
(force_stream=True) and async calls through the new
_acreate_with_stream, aggregating the chunk stream — including tool-call
deltas and reasoning deltas — into a complete response via the shared
_ChatStreamAccumulator.

Salvaged from PR #60686 by @kudi88 onto the progress-aware streaming
machinery from #71508, addressing both sweeper-review gaps: the async
path now consumes the stream with 'async for' (awaiting create() and
iterating synchronously raised on AsyncOpenAI streams), and tool-call
deltas are reassembled instead of dropped (MCP passes tools= through
call_llm). Under force_stream there is no silent non-streaming retry —
a stream-only provider rejects those by definition, so the original
error surfaces to the normal recovery chains.
2026-07-25 14:58:04 -07:00
Teknium 32fd9d65cf feat(compression): progress-aware timeouts — stop punishing slow summary models
The gateway's pre-agent session-hygiene compression killed the summary
call at a fixed 30s wall-clock deadline (compression.hygiene_timeout_seconds),
regardless of whether the summary model was hung or merely slow. A reasoning
model happily streaming a large summary was cut off mid-generation, the user
got '⚠️ Context compression timed out after 30.0s', and a 300s failure
cooldown left the session oversized — a doom loop for slow-but-healthy
auxiliary models.

Timeouts are now liveness-based instead of wall-clock-based:

- agent/auxiliary_client.py: new thread-local aux_progress_hook. When
  installed (only by context compression today), the primary call_llm
  attempt streams (stream=True) and aggregates chunks back into a complete
  response, ticking the hook per chunk. The configured timeout then acts
  per stream read (idle) instead of as a total budget. Providers that
  reject streaming fall back to the plain non-streaming call; auth/payment/
  rate-limit/transport errors propagate unchanged into the existing
  recovery chains. Codex Responses (per SSE event) and Anthropic Messages
  (per stream event, via the new create_anthropic_message on_stream_event
  callback) tick the same hook from inside their wire adapters.

- agent/conversation_compression.py: CompressionCommitFence gains
  touch_progress()/seconds_since_progress(); compress_context() installs
  fence.touch_progress as the progress hook around the compress call.

- gateway/run.py: the hygiene wait loop treats hygiene_timeout_seconds as
  an INACTIVITY budget — while the fence reports fresh progress the wait
  extends, bounded by the new compression.hygiene_total_ceiling_seconds
  (default 600s, clamped >= the idle budget) so a degenerate trickle
  stream still dies. The timeout warning now says the summary model
  produced no output, which is the only case that still triggers it.

- config/docs: hygiene_total_ceiling_seconds added to DEFAULT_CONFIG and
  configuration.md; hygiene_timeout_seconds documented as inactivity-based.

Tests: tests/agent/test_aux_progress_streaming.py (hook plumbing, stream
aggregation incl. tool-call deltas and reasoning deltas, rejection
fallback, ceiling kill, fence progress surface); two new gateway tests
prove a slow-but-streaming worker survives past the fixed timeout
(sabotage-verified: fails with the old fixed deadline) and a
forever-trickling worker is still cut off at the ceiling.
2026-07-25 12:26:28 -07:00
Victor Kyriazakos cc80dba96f feat(monitoring): add hermes.gateway.background_delegations (unit/slot count)
Complements the task-granular background_work with the async-delegation
UNIT count (each dispatch/batch = 1), recovering the pool-slot semantics
active_count() gives. Together: background_work = real concurrent subagent
load (batch expanded), background_delegations = slot pressure to alert
against delegation.max_concurrent_children. Registered in metric_names,
documented, and covered by a task-vs-unit contract test.
2026-07-25 02:30:20 +00:00
Brooklyn Nicholson 6811a79b62 fix(desktop): quote persisted @image: paths so spaced paths render
The unquoted alternative in the directive pattern is `\S+`, so a ref built
by string interpolation truncates at the first space and strands the tail
as loose text next to a broken thumbnail. Composer images live in the app's
userData dir, which on macOS is `~/Library/Application Support/<App>/` — so
every pasted or dropped image hit this.

Adds format_reference_value next to REFERENCE_PATTERN, mirroring
formatRefValue in the desktop's directive-text.tsx, and covers the
round-trip through the parser.
2026-07-24 21:20:52 -05:00
Victor Kyriazakos 0ef0816284 feat(monitoring): count background_work task-granular (expand delegate batches)
A delegate_task fan-out batch occupies ONE async-pool slot by design, so
active_count() (unit/slot count) reports a 3-task batch as 1 — which
undercounts real concurrent subagent load on the background_work metric.
Add active_task_count() that expands a running batch to its child count
(N-task batch -> N, single -> 1) and switch the background_work reader to
it. active_count() is unchanged (capacity semantics preserved). Adds a
contract test for the unit-vs-task distinction and documents both counts.
2026-07-25 02:12:12 +00:00
Victor Kyriazakos 9edcb7b091 feat(monitoring): emit hermes.gateway.background_work (subagent/bg jobs)
active_agents counts foreground turns + cron + API runs but never the
backgrounded delegate_task subagents / terminal(background) processes /
kanban workers tracked only for scale-to-zero. Emit them as a distinct
content-free gauge so the fleet dashboard can show subagent/background
load per peer. Best-effort, sums async_delegation.active_count +
process_registry.count_running; 0 if a source is unavailable.

(cherry picked from commit 1b7ec684e253651a5aa5760b3bd527a3073ce207)
2026-07-25 01:54:31 +00:00
Victor Kyriazakos 6de6e36dbe fix(monitoring): surface cron snapshot failures at WARNING (content-free)
The cron health snapshot failure path logged only at DEBUG, so a cron
telemetry regression would silently drop all hermes.cron.* metrics while
gateway health metrics kept flowing. Promote to WARNING with the
exception *type* name only (no message text, preserving the no-raw-error
contract); keep the traceback on DEBUG.

(cherry picked from commit 1c9a3e737b7c57fa9886bc927f1623753a9a811b)
2026-07-25 01:54:31 +00:00
teknium1 75e0d52034 fix(windows): sweep remaining bare read_text/write_text sites + linter rule
AST-driven pass over every Path.read_text()/write_text() without an
explicit encoding= across non-test code: 71 sites in 34 files
(skills_hub, hermes_cli/main+profiles+service_manager+container_boot,
mem0/hindsight/honcho plugins, achievements dashboard, release/CI
scripts, productivity+comfyui skill helpers, agent/*). Verified zero
positional-encoding collisions before insertion; per-file compile()
check after.

Adds a check-windows-footguns rule flagging bare single-line
read_text/write_text (multi-line forms stay covered by the AST guard
test from #38985). Together with the salvaged contributor commits this
retires the ~169-site bare file-I/O class (#37423's long tail).
2026-07-24 17:10:39 -07:00
AlexFucuson9 cf35fd6de5 fix(core,cli,gateway,plugins): add encoding='utf-8' to read_text() calls
Path.read_text() without an explicit encoding uses the platform's
default encoding. On Windows this is typically cp1252 or mbcs, which
causes UnicodeDecodeError or silent data corruption when reading
UTF-8 content (JSON files, user text, config with non-ASCII chars).

This is the read-side companion to the write_text() encoding fix.
Fixed the most critical locations that read JSON data, user content,
and config files across 14 files with 31 call sites.

Pattern: .read_text() → .read_text(encoding='utf-8')
         json.loads(path.read_text()) → json.loads(path.read_text(encoding='utf-8'))
2026-07-24 17:10:39 -07:00
CoreyNoDream 9f6b2a64e0 fix: decode config and state files as UTF-8 on non-UTF-8 locales
Several file-I/O call sites still use open() / Path.read_text() /
Path.write_text() without an explicit encoding, so they fall back to
the platform default. On Windows CN/JP/KR locales (GBK/CP932/CP949)
any non-ASCII byte in a config/state/user-content file raises
UnicodeDecodeError or UnicodeEncodeError and crashes the caller.

to the remaining hot paths:

- agent/copilot_acp_client.py:  fs/read_text_file and fs/write_text_file
                                (Copilot's read_file / write_file tools,
                                 directly reported in #18637 bug 2)
- agent/model_metadata.py:      context-length YAML cache load + two
                                save sites (context probing is on the
                                call path of every model invocation)
- agent/nous_rate_guard.py:     cross-session rate-limit JSON state
                                (read + atomic write via os.fdopen)
- cron/scheduler.py:            user config.yaml read in run_job
- gateway/delivery.py:          cron output writes for AI-generated
                                content, very likely non-ASCII

yaml.dump call sites also gain allow_unicode=True so the emitted
YAML preserves non-ASCII chars as-is instead of emitting \u escape
sequences.

Adds regression tests that monkeypatch builtins.open / Path.read_text
/ Path.write_text to simulate a GBK locale: each test raises
UnicodeDecodeError / UnicodeEncodeError unless the caller explicitly
passes encoding='utf-8'. Verified that the tests fail on main and
pass with this change, on Linux as well as on Windows.

Refs #18637
2026-07-24 17:10:39 -07:00
teknium1 1608a46884 fix(agent): retire replaced shared OpenAI clients instead of cross-thread pool close
Widen the #70773 fix beyond the three in-request cleanup sites removed in
the cherry-picked commit: every remaining path that swaps out the shared
OpenAI client could still hard-close its pool from a thread that doesn't
own the in-flight sockets (credential rotation/refresh on the turn thread,
dead-connection cleanup, gateway cache eviction, transport recovery) —
the same FD-recycle corruption vector, just rarer.

Add AIAgent._retire_shared_openai_client(): shutdown(SHUT_RDWR) all pooled
sockets (FD-safe from any thread, unblocks in-flight readers) but never
call client.close() — FD release is deferred to GC, which cannot run until
every borrowing thread has unwound its SSL BIO. Refcounting is the
ownership handshake; with no borrowers the FDs are released immediately.

Wired into:
- _replace_primary_openai_client (rotation/refresh/dead-conn cleanup)
- try_recover_primary_transport (primary_recovery)
- release_clients (gateway cache_evict)

agent.close() keeps the hard close: full teardown is a real session
boundary where no request may be in flight.

Tests: new tests/run_agent/test_70773_shared_client_fd_corruption.py
covers the three watchdog/retry sites plus retire semantics; existing
close-assertions updated to pin retire-not-close.
2026-07-24 16:04:48 -07:00
JonthanaHanh e51abb266d fix(agent): prevent shared OpenAI client FD-recycle corruption from stale stream watchdog
The streaming stale watchdog was calling
_replace_primary_openai_client() from its polling thread, which closes
the shared client's connection pool. Worker threads from previous
stale-killed attempts may still be unwinding their SSL BIOs, causing
TLS application-data to overwrite SQLite file headers via FD reuse.

This is the same corruption vector documented in #67142 for Anthropic,
where the fix was to never close the shared client from a non-owner
thread. Apply the same pattern to the OpenAI-wire path:

- Stale stream watchdog: skip shared client replacement
- Mid-tool-retry cleanup: skip shared client replacement
- Stream retry cleanup: skip shared client replacement

The request-local client is already closed via _close_request_client_once.
The shared client is replaced lazily by _ensure_primary_openai_client
on the next request, which runs on the owning thread.

Closes #70773.
2026-07-24 16:04:48 -07:00
teknium1 18af81bb5b fix(caching): reconstruct static system prefix on session restore and post-compression reuse
Follow-up to the cherry-picked #68258 base: the cross-session-stable
prefix (_cached_system_prompt_static) was only recorded on fresh
builds, so two paths silently degraded to the legacy single-breakpoint
layout (flagged in review of #68258/#69341/#69704):

- Session restore: gateway surfaces build a fresh AIAgent per turn and
  restore the persisted prompt verbatim from the session DB; the static
  prefix stayed None from turn 2 onward, flip-flopping the wire layout.
- Post-compression cached-prompt reuse: _invalidate_system_prompt()
  clears the static prefix, and the keep-cached-prompt branch never
  restored it.

Both sites now reconstruct the stable tier and adopt it ONLY when the
authoritative prompt string literally startswith() it — stable-tier
drift (skills edited, identity changed) falls back to the legacy layout
with the stored bytes untouched. Fail-open on any builder error. The
restore-path rebuild is gated on _use_prompt_caching so non-Anthropic
routes skip it entirely.

Refs #68191

Co-authored-by: JonthanaHanh <92574114+JonthanaHanh@users.noreply.github.com>
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
2026-07-24 16:01:38 -07:00
Drexuxux fb1b89b09e fix(prompt-caching): inject cache breakpoints after message normalization
The conversation loop normalizes message text right before the API call so
the request prefix is byte-identical across turns -- the stated reason is
KV cache reuse on local inference servers and better cache hit rates on
cloud providers. Cache breakpoints were injected *before* that pass, which
defeats it.

`_apply_cache_marker` rewrites a plain-string `content` into a
`[{"type": "text", ...}]` block. The normalization pass is guarded on
`isinstance(content, str)`, so every message that just got marked is
silently skipped by it and keeps its raw leading/trailing whitespace. A
message is only marked while it sits in the last-3 window, so:

    turn N      in the window  -> marked, content "file1\nfile2\n"
    turn N+1    rolled out     -> plain,  content "file1\nfile2"

The same logical message is sent with different bytes on consecutive
turns. The prefix stops matching at that position -- which is inside the
span the breakpoints were placed to protect -- so the reusable prefix
collapses back toward the system breakpoint on every turn. Tool results
carry a trailing newline almost by default (any shell command output), so
this is the common case, not an edge case.

Move the injection below every message mutation. Besides fixing the
whitespace divergence this stops breakpoints from being spent on messages
that the orphan sweep or the thinking-only drop is about to remove or
merge away -- a marker on a dropped message is a wasted breakpoint out of
the four available.

Nothing between the old and new call sites reads `cache_control`, and the
mutators now see the plain-string shapes they were written against.
2026-07-24 16:01:38 -07:00
konsisumer 9fdadf0cd7 fix(agent): cache static system prompt prefixes 2026-07-24 16:01:38 -07:00
teknium1 c54fe5b33f fix: pre-lease drift guard must not fire on in-place compaction or mutated snapshots
The salvaged drift check compared durable rows to the in-memory snapshot
by content and ran in both modes. Two problems:
1. In-place compaction (the default) archives non-destructively — drift
   cannot lose data there, and the strict-prefix content comparison
   failed against seeded histories, aborting every in-place compaction
   (5 test failures in test_in_place_compaction.py).
2. Content equality wedges on sessions with legal in-memory mutation of
   past turns (multimodal compression, retry replacement) — the same
   permanent-abort shape as #14694.

Now rotation-only and length-based: abort only when the durable parent
has MORE rows than the snapshot (a writer committed in the lease window).
Dead helper _durable_history_matches_snapshot removed.
2026-07-24 16:00:34 -07:00
Anthony Ruiz 0ee8d41878 fix(compression): recover rotated session lineage 2026-07-24 16:00:34 -07:00
teknium1 62bec4b3f8 fix(compression): add recovery path to anti-thrash auto-compaction block
When two consecutive compactions each failed to clear the threshold, the
anti-thrashing breaker blocked automatic compaction PERMANENTLY for the
life of the session: nothing decremented _ineffective_compression_count
(or _fallback_compression_streak) while blocked, so a session whose
middle region was briefly too small to compact never auto-compacted
again — it grew unbounded until the provider's hard context limit, and
only /new or /reset recovered it.

Recovery is a probation probe, not amnesty: after
_ANTI_THRASH_RECOVERY_SECONDS (300s) of continuous block the gate grants
exactly ONE attempt by dropping tripped counters to 1 strike (persisted,
so sibling agents on the same session row — gateway hygiene — unblock
too). An ineffective probe re-trips the guard on the next real-usage
verdict and the next recovery waits a full fresh window, so the worst
case in a truly incompressible session is one compaction attempt per
window — bounded, not thrash.

The recovery clock is armed lazily on the first BLOCKED evaluation and
is deliberately not durable: a restart that loads a durable tripped
counter (#69872) starts a full fresh window blocked, preserving the
restart-must-never-disarm contract (#54923).

Fixes #14694
2026-07-24 15:57:09 -07:00
teknium1 5a0f51325c fix(agent): make dropped tool-call nudge pair ephemeral scaffolding
Review follow-up for the dropped tool-call recovery (#69630): the
re-prompt pair was tagged _dropped_toolcall_nudge, but that marker was
not part of the ephemeral-scaffolding contract. _persist_session /
_flush_messages_to_session_db would therefore write the synthetic
'issue the actual tool call now' user message (and the narration-only
interim assistant turn) as real transcript rows — a resumed session
could replay the internal retry instruction as user-authored context
and prompt unsolicited tool use.

- Add _dropped_toolcall_nudge to _EPHEMERAL_SCAFFOLDING_FLAGS
  (run_agent.py) so both SQLite and JSON persistence skip the pair.
- Add it to _SYNTHETIC_USER_FLAGS (conversation_compression.py) so the
  compressor never treats the nudge as human intent.
- Flag the interim assistant half of the pair too, and include the
  marker in the finalization scaffolding pop so a genuine turn end
  strips the pair from the live transcript (mirrors the
  _empty_recovery_synthetic pattern).
- Regression tests: flagged messages classify as ephemeral, the
  returned transcript contains no scaffolding, and the turn tail stays
  on the real assistant answer.
2026-07-24 15:56:39 -07:00
Jash Lee 923704c7c2 fix(agent): move dropped tool-call recovery to the finalization chokepoint
The initial fix guarded the no-tool-calls else branch, but that branch only
SETS final_response — the turn actually finalizes later, in a separate block
after final_msg is built. Runs that reached finalization via that path exited
without the guard ever running (observed live: a scheduled PR reviewer stalled
at tool_turns=1-2 with zero recovery nudges).

Move the recovery to the finalization chokepoint (right after final_msg is
built), so it catches every path that ends a turn. Single guard now:
- increments a consecutive-stall counter and re-prompts (bounded to 3),
- resets on any successful tool round, and
- resets on a genuine (non-mismatch) turn end,
so it guards each stall independently without capping the whole run and
without looping forever.

Verified live: the scheduled reviewer now recovers through the stalls and
submits real reviews — PR 57800 APPROVED, PR 54826 COMMENTED — one PR per run.
2026-07-24 15:56:39 -07:00