Commit Graph

1678 Commits

Author SHA1 Message Date
Teknium 0628e33347 fix(agent): clear archived parent's activity labels after rotation (re-review #4)
The compression heartbeat's terminal 'context compression completed'
stamp force-persists against the PARENT session id (agent.session_id at
stamp time). After the out-of-place rotation the parent is archived but
kept advertising a fresh last_activity_at + terminal label forever.
Clear the parent row's activity labels best-effort after a committed
rotation (keeps last_activity_at so idle clocks stay continuous; the
child carries live labels). Regression asserts the archived parent's
labels are cleared while the child's lineage is intact
(sabotage-verified).
2026-08-02 16:16:36 -07:00
Teknium 0277cc48bd fix(agent): never release the durable compression lease mid-commit (re-review #1)
revoke_commit_admission() used to invoke the holder-qualified lease
release unconditionally — including while an admitted commit was still
mutating SessionDB — letting a second compressor acquire the durable
lock mid-commit and interleave with the first commit's writes.

The admission_revoked flag store stays lock-free, but the lease-release
decision now coordinates with the fence lock:
- revoke acquires the fence lock non-blocking; on success no commit can
  be in flight (an admitted commit retains the lock until finish_commit)
  and the release runs immediately, still under the lock so a racing
  begin_commit cannot slip between the check and the release.
- on failure the release is deferred: finish_commit() re-checks
  _admission_revoked and performs it AFTER the commit completes (prompt
  even if the worker thread is later parked), and the begin_commit
  refusal path does the same for a revoke that lost the race to a
  transient lock-setup/cancel boundary. All paths are idempotent with
  the worker's own outer cleanup (DB release is holder-qualified).

Invariant encoded + tested: no second compressor can acquire the durable
lock while an admitted commit is still mutating; after a post-revoke
commit finishes the lease is released promptly. Both regressions
(revoke-during-commit deferral, revoke-before-commit immediate release +
refused begin_commit) are sabotage-verified.
2026-08-02 16:16:36 -07:00
Teknium 15267a1d2d fix(agent): reconcile explicit hard-cancel (main d15b638a88) with pooled fence rework
Rebase onto origin/main brought in 'let explicit interrupts cancel safely',
which predates this branch's pooled progress-timeout + F1-F6 fence rework.
Reconcile the two:

- begin_commit(cancel_event) re-checks the hard-cancel Event under the
  fence lock again (lost in the mechanical rebase).
- compress_context: restore aux_interrupt_protection around the summary
  call, the post-return frozen-cause AuxiliaryExplicitCancellation check,
  and the full rollback/telemetry handler for explicit interrupts.
- run_agent._compress_context: recreate the per-attempt fence registration
  (_active_compression_commit_fence) that hard_interrupt() uses to
  serialize cancel admission, and thread that exact fence through both
  the direct and pooled paths (run_compress_context_with_progress_timeout
  now accepts an external fence).
- test: the pooled worker isolates the live transcript (F3), so the
  hard-interrupt rollback regression mutates the engine's input snapshot
  rather than reaching around it to the caller's list.
2026-08-02 16:16:36 -07:00
Teknium fe2fc57240 test: skip live aux feasibility probe in worker-isolation regressions
Same hermetic-CI trap the concurrent-fork suite already guards against:
without credentials the one-time feasibility probe aborts compression
before the stubbed engine starts, so the F3/F4 blocked-state assertions
went vacuous-false in CI while passing on credentialed dev machines.
2026-08-02 16:16:36 -07:00
Teknium 99100843c6 fix(agent,gateway): charge the idle wait from the last progress event (review S3)
Both progress-aware waits (sync compress wrapper and gateway session
hygiene) slept a FULL idle interval and only then compared progress, so
progress early in an interval let silence approach 2x the configured
idle timeout before the waiter noticed. Compute each wait slice as
idle_timeout - elapsed_since_last_progress instead.

Regression: a worker that reports progress early and then goes silent is
timed out in ~1x the idle budget, not ~2x.

PR #76354 review, 'idle timeout can allow nearly twice that silence'.
2026-08-02 16:16:36 -07:00
Teknium abc0db8cde fix(agent): bounded admission + stale-job cancellation for the compress pool (review F6)
The process-wide 4-worker pool retained the stdlib executor's unbounded
queue: four hung summaries wedged every slot, a fifth compression queued
silently, waited out its whole budget without starting, and remained
eligible to run later as an expensive stale job whose fence was already
cancelled (the first fence check used to sit AFTER the summary call).

- Bounded admission: submission fails fast (messages returned unchanged,
  loud warning) when all pool slots are occupied; slots are freed by a
  future done-callback. Recovery contract documented at the constant: new
  work fails fast while wedged, wedged workers are fence-cancelled and
  restore service when they return; a worker that never returns costs its
  slot — bounded, observable degradation instead of unbounded queueing.
- Not-yet-started futures are cancel()ed on timeout.
- The cancelled fence is checked BEFORE any expensive summary work, both
  in the pooled wrapper (stale queued job) and inside compress_context
  (pre-summary gate), so a stale job never burns an LLM call or acquires
  session state.

Saturation regression: 4 event-blocked summaries wedge the pool, a 5th
submission fails fast (asserted while the four are provably still
blocked), the refused job never runs after worker recovery, and a fresh
submission after recovery succeeds.

PR #76354 review, blocking finding 6 / merge gate 7.
2026-08-02 16:16:36 -07:00
Teknium 36a60c5edc fix(agent): rebind the caller's session ContextVar after out-of-place rotation (review F5)
Session rotation runs on the pooled worker thread, whose copied context
gets the child id — the CALLER's ContextVar still holds the parent, and
get_session_env() prefers a bound ContextVar over os.environ. Tools and
subprocesses invoked on the caller thread after a compression.in_place=false
rotation therefore saw the STALE parent HERMES_SESSION_ID.

After the pooled wrapper returns, rebind the session id in the caller's
own context (set_current_session_id) alongside the existing logging
repair; idempotent when no rotation happened.

Behavioral regression: with the gateway-style bound session context, a
post-compression get_session_env("HERMES_SESSION_ID") read on the caller
thread now returns the child id.

PR #76354 review, blocking finding 5 / merge gate 6.
2026-08-02 16:16:36 -07:00
Teknium fdeb09a596 fix(agent): holder-qualified durable lease cancellation, cooldown ordering (review F4)
A host timeout previously left the timed-out worker holding the durable
per-session compression lock AND refreshing its lease indefinitely, so a
truly hung summary blocked every later compression attempt; and a LATE
successful summary could clear the failure cooldown the host had just
recorded.

Transplant the lease-cancellation invariants from PR #71569
(@ciabata-git): the worker publishes an idempotent, holder-scoped release
hook on the fence once it owns the durable lock (begin_lock_setup /
register_cancelled_lock_release close the acquire→publish race), the
refresher start is serialized against the release path, and the host
invokes the hook on idle timeout, hygiene timeout, and every unwind
(revoke_commit_admission now also releases). ABA safety: the SessionDB
release is holder-qualified (DELETE ... WHERE holder = ?), so a stale
release can never free a replacement holder's lease.

State ordering: the compressor consults a fence-cancellation check BEFORE
clearing the failure cooldown, so a late worker cannot undo the host's
timeout cooldown; the check is installed only for the fenced call and
removed in a finally.

Regression implements the reviewer's exact 5-step scenario: summary
blocked indefinitely → host timeout → a NEW compressor acquires the
durable lock while the old summary is STILL blocked → old worker released
→ it cannot clear cooldown, release the new holder's lease, or publish
stale state.

PR #76354 review, blocking finding 4 / merge gates 4 + 5.

Co-authored-by: ciabata-git <ciabata-git@users.noreply.github.com>
2026-08-02 16:16:36 -07:00
Teknium 971d81f892 fix(agent): isolate the pooled compression worker from the live transcript (review F3)
The pooled worker closure captured the caller's live `messages` list and
compress_context explicitly supports plugin/legacy context engines that
mutate that list in place — so after a host timeout, a late engine could
rewrite the live conversation (roles, ordering, persisted content)
concurrently with the resumed turn.

The worker now deep-snapshots the transcript on the worker thread before
any engine code runs; the caller's list object is never handed to pooled
code. Results reach caller-visible state only through the returned value
of an ADMITTED commit (the host discards results on timeout/cancel), and
durable SessionDB mutation was already gated behind the commit fence.
No-op passes map the unchanged snapshot back to the caller's original
list so identity-based no-op detection and flush dedup keep working.

Document the thread-safety contract for context-engine and
memory-provider extension points (they now run on pooled threads) in the
module docstring and the context-engine plugin guide.

Regression: an in-place-mutating engine plus host timeout proves the
caller's live transcript is byte-identical WHILE the worker is still
blocked inside the engine (released only after the assertions).

PR #76354 review, blocking finding 3 / merge gate 3.
2026-08-02 16:16:36 -07:00
Teknium efdd229884 fix(agent): revoke commit admission on every host unwind (review F2)
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.

Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.

Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.

Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.

PR #76354 review, blocking finding 2 / merge gate 2.
2026-08-02 16:16:36 -07:00
Teknium 980aea225e fix(agent): observe commit phase without the fence lock (review F1)
begin_commit() retains the fence lock until finish_commit(), so a hung
SessionDB commit made try_cancel_before_commit() return None forever and
the host spun ahead of the overrun-warning loop — a genuinely hung commit
stayed unbounded AND silent. Add a lock-free phase marker (threading.Event
set inside begin_commit while the lock is held, readable without it) and
break the host spin on commit_in_flight so the bounded overrun loop — and
its WARNING + on_commit_overrun surfacing — is reachable WHILE the commit
is still blocked. Applies to both the sync compress wrapper and the
gateway session-hygiene wait.

Regression asserts the warning and callback fire while the event-gated
fake commit is still blocked; the test releases the worker only after
those assertions (addresses helix4u's released-before-asserting callout).

PR #76354 review, blocking finding 1 / merge gate 1.
2026-08-02 16:16:36 -07:00
Teknium a05b102d0d test(agent): deflake concurrent-fork suite by skipping live aux feasibility probe
The stubbed-compressor fixture still let the one-time compression-model
feasibility probe run inside the first _compress_context call. On machines
with real credentials configured, that probe resolves a live auxiliary
provider (credential pool seeding, Copilot token exchange over HTTPS),
which nondeterministically exceeds the 2s event-timing budget in
test_fence_cancelled_compression_leaves_lock_reacquirable (reproduced on
PR #73031's own head). Mark the probe done in the shared fixture: these
tests exercise locking/fencing/rotation, never aux feasibility. Suite
runtime drops from ~90s to ~4s.
2026-08-02 16:16:36 -07:00
Teknium 2fb0aa1c0e fix(agent): enforce bounded, surfaced commit-phase waits past context_total_ceiling_seconds
The post-begin_commit() waiter previously called unbounded future.result(),
so the advertised compression.context_total_ceiling_seconds was silently
unenforced for commit-phase hangs. The commit still must complete (abandoning
an in-flight SessionDB mutation would diverge live messages from durable
state), but the wait is now bounded in increments against the remaining
ceiling: on ceiling breach the overrun is logged (WARNING escalating to
ERROR), surfaced once through the user-visible warning channel via the new
on_commit_overrun callback (wired to _emit_warning in run_agent.py), and the
host keeps waiting in bounded slices until the commit finishes.

Documented guarantee (config comment + docs, en/zh): summary phase bounded
by the ceiling; commit phase logged + surfaced if it exceeds it — never
silently hung, never abandoned mid-commit.

Test updated to assert the surfacing fires (previously accepted a silent
over-ceiling wait); adds coverage that a raising overrun callback cannot
break the commit wait.
2026-08-02 16:16:36 -07:00
fangliquanflq 06c7f9b26f fix(agent): clarify compress_context ceiling is pre-commit only
Once begin_commit() wins, SessionDB mutation cannot be fence-cancelled;
document that context_total_ceiling_seconds covers the summary phase only
and pin the hang-wait contract in tests.
2026-08-02 16:16:36 -07:00
fangliquanflq 240148b440 fix(agent): force-persist compression completed past SessionDB rate limit
{id: #72016}
2026-08-02 16:16:36 -07:00
fangliquanflq 61e722261c fix(agent): silence detached compression heartbeat after host timeout
Host progress timeout leaves compress_context running on a daemon worker while
the live turn continues. Latch heartbeat silence on fence cancel or terminal
timeout/cooldown provenance so a later UNKNOWN stamp cannot re-arm
agent.compression and poison stall clocks.
2026-08-02 16:16:36 -07:00
kshitijk4poor 962e4538da refactor: reuse existing utilities in salvaged PR #72424
Three code-reuse fixes applied during salvage:

1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
   the relative-time formatting logic in hermes_cli/status.py.

2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
   to deduplicate the two nearly-identical try/except blocks that stamp
   compression timeout/abort provenance in the hygiene path.

3. Add ContextCompressor.record_timeout_failure() method and use it from
   the in-agent compress_context timeout callback instead of re-implementing
   the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
   exception handler already has this ladder — now both paths share one
   method.
2026-08-02 16:16:36 -07:00
fangliquanflq c2088efe9e feat(gateway): session activity watchdog, stall notify, compress timeout (#72424)
Three mechanisms to detect and notify when gateway sessions stall silently:

1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
   and hermes status show progress during long turns without new message rows.

2. Stall watchdog: when a busy session has pending inbound and the shared
   activity clock is idle past agent.session_stall_timeout (default 300),
   log a WARNING and notify the user once to try /new. Notify-only; does
   not kill the turn.

3. Compaction timeout: fenceless compress_context callers get a progress-aware
   host budget (compression.context_timeout_seconds default 120 idle,
   compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
   cancel via commit fence, skip compaction without dropping messages, and
   continue the turn.

Closes #72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).

Cherry-picked from PR #72424 by @fangliquanflq.
2026-08-02 16:16:36 -07:00
Teknium 5b4d20b524 fix(hooks): flush outbound queue at interpreter exit
The delivery worker is a daemon thread, so a short-lived process
(hermes chat -q, a cron session) could exit right after firing
on_session_end — silently dropping the headline event. Register a
bounded atexit flush (5s) when the worker starts: a dead endpoint can
delay exit slightly, never hang it.

Live-verified: hermes chat -q now delivers on_session_start,
post_tool_call, and on_session_end to a real receiver; regression test
runs a subprocess that exits without flushing (sabotage-verified).
2026-08-02 15:01:11 -07:00
Teknium 86fd6da1dc fix(hooks): single delivery_id across header+body, never follow redirects
- delivery_id is now generated once per firing and used for both the
  X-Hermes-Delivery header and the signed body's delivery_id field —
  previously they were two different uuid4s, breaking receiver-side
  dedupe as documented.
- 3xx responses are no longer followed: urllib's default redirect
  handler converts a redirected POST into a body-less GET, silently
  dropping the signed payload. Redirects now log a misconfiguration
  warning and count as delivery failure (no retry).
- Docs: receiver-side replay-protection guidance (dedupe on
  delivery_id, timestamp freshness window) + redirect semantics.
- Tests: 5xx retry count, redirect-not-followed (sabotage-verified),
  header/body delivery_id equality.
2026-08-02 15:01:11 -07:00
Teknium 3829e34e23 feat(hooks): outbound webhooks — push signed lifecycle events to external HTTP endpoints
The inverse of the inbound webhook platform: hooks.outbound in
config.yaml lists HTTP targets + the plugin-hook events they subscribe
to (on_session_end, subagent_stop, post_tool_call, ...). Each firing
POSTs a JSON payload (same top-level shape as shell hooks' stdin wire)
signed GitHub-style with HMAC-SHA256 (X-Hermes-Signature-256).

Rides the existing hook bus — notify-only callbacks registered on the
plugin manager at the same CLI/gateway/main entry points as shell
hooks. Delivery is fire-and-forget via a bounded queue + single daemon
worker thread, so a dead endpoint can never stall a tool call. Bounded
retries (5xx/conn errors once; 4xx never). secret_env preferred over
inline secret. HERMES_SAFE_MODE skips registration. hermes hooks list
shows outbound targets with signed/UNSIGNED status.

Zero new model tools, zero new subsystems.
2026-08-02 15:01:11 -07:00
Josh Tsai 013779924f fix(agent): fail fast on custom-provider /models auth errors
- Short-circuit the candidate waterfall on HTTP 401/403: an auth wall
  proves the endpoint family exists, so probing the alternate URL just
  doubles the wasted wait (the reported endpoint takes ~10s to return
  401 without a key).
- Stream the probe so 4xx never downloads a slow error body; responses
  are closed on every exit path.
- Regression tests: single-call assertion on 401/403 (fails on main),
  negative-cache reuse, 404 waterfall preserved, no .json() on 4xx.

Fixes #69905

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-02 22:47:48 +05:30
Teknium 2c9bbda072 test(secrets): consolidated tier-1 migration regression suite
Exercises representative migrated sites per cluster: scoped value wins,
scoped miss no-borrow under multiplex, and unscoped-multiplex behavior per
pattern (in-turn get_secret vs Slack-pattern startup fallback).
2026-08-02 10:02:33 -07:00
Teknium 44640149f7 fix(secrets): allowlist API_SERVER listener settings as global deployment env
Fixes #69379 (v2026.7.20 Docker multiplex regression: the scoped runner
reload dropped API_SERVER_* set via the container environment, silently
losing the api_server platform) by the canonical mechanism — corrects
the direction of #69524, which patched gateway/config._getenv to fall
through to os.environ on EVERY scoped miss, re-opening the
cross-profile borrow for all credentials.

API_SERVER_ENABLED / API_SERVER_HOST / API_SERVER_PORT /
API_SERVER_CORS_ORIGINS are deployment listener settings (Docker
compose environment: block, systemd Environment=), not profile
secrets: they join _GLOBAL_ENV_EXACT so get_secret reads them from
os.environ regardless of scope. API_SERVER_KEY is deliberately NOT
allowlisted — it IS a credential and stays profile-scoped, which keeps
tests/gateway/test_config.py's secondary-profile isolation semantics
intact (a secondary profile without the key still doesn't bind a
listener).

Ports #69524's regression test in the corrected form: container-env
API_SERVER_* stays visible during the scoped runner reload while the
key resolves through the profile scope; plus unit tests locking the
allowlist membership and the deliberate exclusion of API_SERVER_KEY.
2026-08-02 09:59:52 -07:00
Teknium 3fb066c372 fix(agent): dotenv-compatible inline-comment stripping in load_env_file
Unquoted values truncate only at a # preceded by whitespace (so
KEY=foo#bar stays intact); quoted values scan escape-aware for the
matching close quote, keep through it, and drop a trailing '# ...'
remainder. Verified empirically against python-dotenv 1.2.2 on a
10-case corpus (full parity). Supersedes the approach in #57718, whose
scanner corrupted foo#bar-style values.
2026-08-02 09:58:59 -07:00
spfcraze 55abf206bf fix(agent): unescape quoted .env values in secret_scope.load_env_file
save_env_value writes values containing " or \ as escaped
double-quoted strings, and every other .env reader in the repo
(load_env/_parse_env_value, python-dotenv) reverses those escapes.
load_env_file — the parser behind build_profile_secret_scope, which
wraps every cron job and every multiplexed gateway turn — only stripped
the outer quotes, leaving the escapes literal. A credential containing
a double quote or backslash (JSON service-account blobs, generated
secrets) authenticates fine interactively but 401s under scoped
resolution, with no error pointing at the cause. Parse values with the
canonical _parse_env_value so all readers agree byte-for-byte.
2026-08-02 09:58:59 -07:00
Fangliquan 696fae8e78 fix(agent): strip UTF-8 BOM when loading profile .env into secret scope
Windows editors often save .env with a leading BOM; plain utf-8 left U+FEFF
on the first key so multiplex get_secret missed that credential.
2026-08-02 09:58:59 -07:00
Shaun Prince d15b638a88 fix(compression): let explicit interrupts cancel safely
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.

Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.

Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
2026-08-02 22:15:20 +05:30
kshitij e8656bedfe fix(agent): byte budget for the canon-args memo
Review follow-up on the #76098 salvage: the 4096-entry count bound alone
doesn't bound MEMORY — write_file/patch argument strings run 100KB+, so
a long-lived gateway process under sustained heavy write workloads could
pin ~800MB of evicted-session strings. A 32MB byte budget extends the
existing FIFO eviction; common-case args (0.5-2KB) never hit it. New
guard test mutation-checked (fails with the byte leg disabled).
2026-08-02 21:17:12 +05:30
spfcraze cf803603dc perf(agent): memoize send-path tool-call argument canonicalization
The pre-send normalization pass re-canonicalized every historical tool
call's argument JSON on every API-call iteration — quadratic in session
tool-call count. Route it through a bounded value-keyed memo (the
_MSG_TOKENS_CACHE idiom from agent/model_metadata.py): per-iteration
cost is now proportional to new tool calls, not all of them.

Measured (simulated growing session, repo venv): session-total
canonicalization cost 1056 ms -> 58 ms at 500 tool calls x 2 KB args
(18.3x), 5744 ms -> 79 ms at 500 x 16 KB (72.5x). Byte-parity with the
pre-fix logic is asserted at every iteration (unicode, nested,
malformed, empty, non-string args), and a call-count test proves
json.loads invocations went from K(K+1)/2 to K per session.
2026-08-02 21:17:12 +05:30
kshitij 077317a0f6 simplify: collapse aux config reads, trim comments, use load_config_readonly
- Merge _aux_free_only() + _aux_openrouter_model() into single
  _aux_openrouter_settings() that reads config once via
  load_config_readonly (avoids double deepcopy).
- Remove 15-line block comment and 5-line inline comment that
  restated what the code already says.
- Trim module docstring from 10 lines to 3.
- Update test patches to target load_config_readonly.
2026-08-02 12:19:04 +05:30
webtecnica c19c63d9e6 fix(agent): make auxiliary auto-chain fallback configurable and free-only (#75803) 2026-08-02 12:19:04 +05:30
kshitijk4poor 88a629b9dc refactor(agent): share the cache_ttl disable predicate across init and stub paths
Follow-ups from review of #76113:
- Extract cache_ttl_means_disabled() as the single disable-synonym
  predicate; agent_init and prompt_caching_disabled_from_config both use
  it so the two detection sites can no longer drift (drift would recreate
  the #76085 bug class).
- Mirror _run_reference's not-None injection guard in
  aggregate_moa_context (stamping None was a harmless no-op copy).
- Replace a vacuous trailing test assertion with the intended
  input-non-mutation check; drop a stray blank line.
- Add a predicate-parity regression test (unknown TTL values keep
  caching enabled, matching historical agent_init semantics).
2026-08-02 11:52:59 +05:30
686f6c61 8ee51747fe fix(agent): consolidate cache-disable stubs with blank_cache_policy_stub
Absorb the useful deltas from the parallel #76121 approach: a single
blank_cache_policy_stub factory so _cache_disabled cannot be left off
hand-rolled SimpleNamespaces, and pin the live agent disable onto MoA
advisor fan-out and one-shot aggregate_moa_context decoration so those
paths track conversation state rather than a fresh config re-read.

Keeps the earlier tri-state prepared-aggregator no-agent fix. Adds
factory and synthesis/advisor regressions.

Coordinates with #76121 / #76085.

Co-authored-by: JoaoMarcos44 <87440198+JoaoMarcos44@users.noreply.github.com>
2026-08-02 11:52:59 +05:30
686f6c61 8e1a351e82 fix(agent): preserve None cache_disabled when MoA has no agent
Prepared-aggregator facades built via __new__ lack _agent. Accessing
self._agent raised inside the planner try and bool-coercion of a missing
snapshot forced False, suppressing config fallback for cache_ttl=off.
Pass a tri-state value and add a no-agent/config-off regression.
2026-08-02 11:52:59 +05:30
686f6c61 67db87009e test(agent): drop unused pytest import from cache-disable tests
Avoid F401 from ruff/pyflakes on the #76085 regression file.
2026-08-02 11:52:59 +05:30
686f6c61 8a29703a1b fix(agent): honor prompt_caching.cache_ttl=off on stub policy paths
Blank SimpleNamespace stubs used by MoA decoration and
plan_cache_sections_for_destination never set _cache_disabled, so
anthropic_prompt_cache_policy re-injected cache_control markers after
operators turned caching off. Stamp the disable onto those stubs from
an explicit flag or the live config, and pass the agent flag from the
MoA aggregator path.

Fixes #76085
2026-08-02 11:52:59 +05:30
joaomarcos 18e0683bfc fix(auth): route anthropic adapter credential reads through the profile secret scope
Salvaged from #51604 (@JoaoMarcos44, issue #51603): resolve_anthropic_token()
and run_oauth_setup_token() in agent/anthropic_adapter.py read
ANTHROPIC_TOKEN / CLAUDE_CODE_OAUTH_TOKEN / ANTHROPIC_API_KEY via bare
os.getenv(), bypassing agent.secret_scope — a cross-profile over-read in
multiplex mode. Every other provider routes through
runtime_provider._getenv -> get_secret; the adapter now does the same via
a local _getenv wrapper (identical to os.getenv when multiplexing is off,
scope-authoritative + fail-closed when on).

Dropped from the original PR: the cron scheduler hunks (superseded by
fdab380a1a which installs the per-job profile scope) and the unrelated
hermes_logging Windows hunk (scope creep).

Includes the PR's RED->GREEN scope-isolation test file (6 tests).
2026-08-01 16:42:51 -07:00
kshitijk4poor 85e0073902 refactor(compression): fold simplify-pass findings into feasibility skip
- Reuse telemetry['middle_window_tokens'] for the skip's middle estimate
  (is-None fallback to a fresh estimate) so log and telemetry agree
- Declare prellm_skip_count in the base telemetry schema (fixed shape)
- Defer _derive_auto_focus_topic into the non-skip branch (user-turn scan
  was wasted work on every skip)
- Document the skip in compress()'s Algorithm list and force: arg doc
- Drop dead call_llm patches from 7 tests (unreachable with
  _generate_summary mocked)
2026-08-01 16:00:52 +05:30
TRON 8daf03063d fix(compression): add pre-LLM feasibility check to skip costly no-op summaries
When the middle section is < 10% of threshold tokens AND at least one prior
real-usage ineffectiveness strike has been recorded, skip the expensive LLM
summarization call and fall through to the deterministic message-dropping
path.  Without this guard, a tool-heavy session where the protected tail
already holds most of the tokens can burn 500+ seconds on a summary call
that replaces a few lightweight messages with negligible token savings.

Key design decisions per GottZ review on PR #60451:

1. Separate _prellm_skip_count counter — never increments
   _ineffective_compression_count (the strike counter that latches at >=2
   to disable compression entirely).  One real strike + one skip must NOT
   permanently lock out compression until /new.

2. feasibility_skip sentinel flag — exempts skips from the abort branch
   (abort_on_summary_failure / _last_summary_auth_failure /
   _last_summary_network_failure).  A stale failure flag from a prior
   cycle must not turn a deliberate skip into a full abort.

3. reason=None for feasibility-skip fallbacks — a stale _last_summary_error
   from an earlier real failure must not be embedded into the skip's
   deterministic fallback marker.

4. info-level logging for feasibility-skip fallbacks (not warning) — this
   is an intentional optimization, not a failure.

Skipped when force=True (manual /compress) so auth/error handling paths
are always exercised on explicit user request.

Adds 6 regression tests (TestPreLlmFeasibilityCheck) covering:
- Strike counter isolation
- Stale auth/network failure flag immunity
- force=True bypass
- No-skip when no prior strikes
- Counter reset on session reset

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: TRON <tron-agent@agentmail.to>
2026-08-01 16:00:52 +05:30
kshitij 321cbcc2e2 test(redact): add YAML ReDoS test, strengthen existing test with keyword
- Relax timing bound from 1.0s to 2.0s (CI machine robustness).
- Add test_long_dotted_run_with_keyword_completes_fast: includes a secret
  keyword so the pre-gate does NOT skip _CFG_DOTTED_RE — directly exercises
  the possessive-quantifier regex, not just the pre-gate.
- Add test_yaml_assign_redos_resistance: _YAML_ASSIGN_RE was modified but
  had no ReDoS test — add 100-line stress input.
- Add test_yaml_assign_secret_still_redacted: verify YAML matching behavior
  preserved with possessive quantifiers.
2026-08-01 15:58:25 +05:30
carlotestor 13ad903a3c perf(redact): eliminate exponential backtracking in config-key patterns
_CFG_DOTTED_RE's nested quantifier (?:[A-Za-z0-9_\-]+\.)+ backtracks
exponentially on long non-matching dotted runs (doubles every ~4
segments). Flatten it and use possessive quantifiers (py3.11+) in
_CFG_DOTTED_RE and _YAML_ASSIGN_RE wherever the successor is disjoint.

Zero behavior change: equivalence fuzz-verified over 120k structured
and random inputs comparing full sub() output including groups. Adds a
ReDoS regression test.
2026-08-01 15:58:25 +05:30
kshitij 536754919d fix(auxiliary): replan cache sections on the async fallback path too
_call_fallback_candidate_sync replans messages/tools for each resolved
destination, but its async mirror still shipped the caller's decorated
sections verbatim — the primary destination's markers (including a
direct-native tool marker) leaked to fallback candidates with different
cache contracts, and the relay saw the display label instead of the
resolved provider/api_mode. Mirror the sync path: resolve the
destination, replan both sections, thread provider/api_mode through
_relay_async_completion, and replan again for the auth-refresh retry
client. Mutation-checked: the new parity test fails on the verbatim
pass-through shape.

Follow-up to #76032 (#20880).
2026-08-01 15:39:58 +05:30
kshitij 7ae4a5efba fix(prompt-caching): consolidate static-prefix split, guard empty volatile suffix
_apply_static_prefix_marker duplicated _apply_system_cache_markers' split
logic minus its empty-suffix guard: when the stored system prompt equals
the static prefix exactly, the tool-cache plan emitted a two-part split
with a trailing empty text block — HTTP 400 on native Anthropic. Fold the
tool-cache layout into the existing helper via mark_suffix /
fallback_to_whole flags; the empty-suffix case now marks the prompt as
one whole block. Behavior-parity verified against the merged planner for
every non-empty-suffix shape.

Follow-up to #76032 (#20880).
2026-08-01 15:39:58 +05:30
kshitijk4poor e078c8c6ef fix: widen fallback warning to the sibling custom-endpoint 256K path
Review pass 2 (reuse reviewer HIGH): the step-3b probe-down fallback for
custom/local endpoints returns the same silent 256K default but only
logged at INFO - invisible by default, and it is the MORE common path
for small local models (the exact users the warning exists for).

Extract _warn_context_length_fallback() (deduped per model+base_url)
and call it from both fallback sites, per the fix-the-whole-bug-class
rule. Regression test drives the custom-endpoint path and fails without
the widening (mutation-checked).
2026-08-01 15:05:05 +05:30
kshitijk4poor 4c2d0c7fd8 refactor: dedupe fallback warning per model, drive pool-cleanup tests through real run()
Review follow-up:
- Warn once per (model, base_url) at the step-9 fallback via a module-level
  dedup set (established _WARNED_* idiom). The fallback result is
  deliberately never cached, so the un-deduped warning fired on every
  resolution - e.g. once per gateway message via the session-hygiene path.
- Replace the three inline-mock pool-cleanup tests (which reproduced the
  try/except block against a MagicMock and passed even with the production
  code reverted) with a parametrized test that drives the real
  BatchRunner.run() with a patched Pool; drop the CPython stdlib
  signature change-detector test.
- Add a once-per-model warning regression test; clean up dead imports.

All tests verified to fail against pre-PR batch_runner.py/model_metadata.py
and pass with the fix (mutation check).
2026-08-01 15:05:05 +05:30
kshitijk4poor a1ff62a139 fix: context-length fallback logging, batch trajectory durability, pool cleanup
Salvage of #6629 by aaronlab (kshitijk4poor reworked against current main).

Three concerns from the original PR, reworked to address review feedback:

1. Context-length fallback diagnostic (agent/model_metadata.py):
   get_model_context_length() silently returned 256K when all 9 detection
   methods failed. Users with small-context models (8K, 32K) would get 256K
   silently, causing hard-to-debug API context-length errors. Added a
   warning log at the step 9 fallback with model name, base_url, and the
   correct config override hint (model.context_length, not context_length).
   The token-estimation ceiling-division fix from the original PR already
   landed on main (5c2ecdec) with CJK handling — not duplicated here.

2. Fsync for batch trajectory writes (batch_runner.py):
   Trajectory entries were written without flush/fsync, but the checkpoint
   immediately marked them as completed. A crash between write and disk
   sync would leave the checkpoint claiming completion with no trajectory
   data on disk. Added flush() + os.fsync() before checkpoint update.

3. Pool cleanup on interruption (batch_runner.py):
   Ctrl+C during pool.imap_unordered() relied on context manager cleanup
   which can hang. Added explicit pool.terminate() + pool.join() for both
   KeyboardInterrupt and Exception paths. The original PR used
   pool.join(timeout=10) which is invalid — CPython's Pool.join() takes
   no timeout parameter. Fixed to use pool.join() without arguments.

Tests:
  - test_warning_emitted_on_fallback: verifies warning fires at step 9
  - test_no_warning_when_cached: verifies no false warning when cache hits
  - test_trajectory_entry_is_synced_to_disk: verifies os.fsync is called
  - test_pool_terminate_called_on_exception: verifies cleanup on RuntimeError
  - test_pool_terminate_called_on_keyboard_interrupt: verifies cleanup on Ctrl+C
  - test_pool_join_called_without_timeout: verifies no timeout arg to join()
  - test_real_pool_join_accepts_no_timeout: integration check on CPython API

Co-authored-by: Aaron Lab <aaronlab@users.noreply.github.com>
2026-08-01 15:05:05 +05:30
Rod Boev 9fc12bf7a4 perf(prompt-caching): preserve tool-loop cache boundaries (#20880) 2026-08-01 14:27:12 +05:30
Teknium d5463e5f6d fix(agent): invalidate flush-scan cursor at the defrag marker-pop sibling site
The micro-compaction defrag pass (_defrag_rolling_summary) rewrites the
newest MICRO marker's content and pops _DB_PERSISTED_MARKER from the
LIVE dict in place — the same in-place pop class finalize_turn's fill
site was fixed for in #75170. Without invalidation the bounded
flush-scan cursor identity-skips the rewritten marker row and the
defragged rolling summary never reaches state.db (resume rehydrates a
stale summary).

The compressor holds no agent reference, so the pop site raises
_flush_scan_cursor_invalidated and the finalize_turn micro-compaction
block consumes it, setting agent._db_flush_scan_prefix = None.

The module-scope pop sites (context_compressor.py:175/224) operate on
fresh copies — identity-breaking by construction — and need no flag.

Follow-up to #75170 (fix-the-class sweep of _DB_PERSISTED_MARKER
in-place pops).
2026-07-31 23:18:12 -07:00
spfcraze 2aaeee2ee5 fix(agent): invalidate flush-scan cursor when finalizer pops db marker
The bounded flush-scan in _flush_messages_to_session_db_unlocked skips
the identity-matched prefix of its previous snapshot, on the documented
assumption that no code path pops _DB_PERSISTED_MARKER from a live dict
in place. finalize_turn's pure-tool-call-tail fill is exactly that path:
it pops the marker so the filled content gets re-persisted — but the
cursor then skips the row anyway, so the delivered final response never
reaches state.db and /resume replays content="" (the #43849/#44100
class resurfacing via the perf cursor). Invalidate the cursor at the
pop site so the filled row is re-examined.
2026-07-31 23:18:12 -07:00
Teknium 3b9cf56aff fix(agent): exclude reasoning_details envelope from tail-budget walk (#73298)
Companion to the preflight fix: _estimate_msg_budget_tokens charged
reasoning_details at chars/4 via _REPLAY_BUDGET_KEYS, so the signed/base64
envelope (measured 72% of the reasoning mass on Anthropic-wire sessions)
consumed the tail budget and _find_tail_cut_by_tokens summarized away real
transcript to make room for tokens that are never sent (69 messages on the
measured session; up to ~4.8x budget inflation on thinking-heavy histories).

Per the #51800 counter-argument, actual thinking TEXT stays visible to the
budget: _reasoning_details_text_chars counts thinking/text/summary fields
and skips signature/data/encrypted blobs, and the text is skipped entirely
when reasoning/reasoning_content already carries the identical prose (so it
is charged once, not twice). codex_reasoning_items remains fully charged —
Codex Responses genuinely replays it every request (#55572).

Sabotage-verified: restoring reasoning_details to _REPLAY_BUDGET_KEYS fails
the new envelope and double-charge tests.
2026-07-31 23:16:58 -07:00
Doud-FR 3127ddcb64 fix(agent): preserve a non-empty user query after compression 2026-07-31 23:16:58 -07:00
Teknium d2098331e7 test(agent): add compress()-level E2E regression for short tool-suffix OOB (#75588)
Follow-up to the boundary clamps: drive the REAL compress() pipeline over
the live 8-message transcript shape from #75588 (system + tool-only
suffix, aligned head == len(messages)). Asserts no exception, the summary
LLM is never invoked when the window is out of range, and the transcript
is returned unchanged. Sanity-checks the clamped tail-cut boundary so a
future regression of the n+1 floor is caught even where downstream
callsite clamps mask the IndexError.
2026-07-31 23:16:58 -07:00
x7peeps a1f70343fd fix(agent): clamp tail-cut boundary and summary-scan indices to prevent IndexError
Fix #75588

## Root cause

When a short conversation ends in a tool-call/result group and the
protected head alignment reaches the end of the message list,
_find_tail_cut_by_tokens() could return len(messages) + 1. This
happened because the final return used max(cut_idx, head_end + 1)
which could push past the array length when head_end >= len(messages).

The out-of-range value then propagated into _find_context_summaries()
which iterated range(start, end) and indexed messages[idx] without
clamping, raising IndexError and failing the active gateway turn.

## Fix

Two-layer defense:
1. Source fix: _find_tail_cut_by_tokens() now clamps its return to
   min(n, ...) so it never exceeds len(messages).
2. Defensive clamp: _find_context_summaries() now bounds start/end
   to [0, len(messages)] so even if a future caller passes bad values,
   it cannot crash.

## Verification
- 7 new regression tests for the exact boundary conditions
- All 214 existing test_context_compressor.py tests pass
2026-07-31 23:16:58 -07:00
Xipong 9ceb0858ab fix(codex): defang reserved Harmony tokens in requests 2026-07-31 22:53:20 -07:00
Israel Lot 5c45d9c208 fix(agent): mirror substitute_api_content's guard in the estimator shadow
Review follow-up on #75102. The shadow substituted the sidecar whenever
the ``api_content`` key was merely PRESENT, but the wire only substitutes
a non-empty string sidecar on a user/assistant row (see
``turn_context.substitute_api_content``). For any other shape the sidecar
is popped and discarded while the clean ``content`` is sent -- so the
shadow dropped real content from the estimate and UNDERcounted, the
dangerous direction: compaction fires too late and the turn dies on a
hard context-length error instead of merely compressing early.

Gate the substitution on the same predicate, and cover the divergent
shapes (None, empty string, int, list, non-user/assistant role) with a
test that fails against the unconditional version.

Also rename the image test: it never carried a sidecar, so it was not
testing what its name claimed. It is a non-regression pin on the flat
per-image accounting that moved into ``_wire_message_shadow()``, and is
now named for that.
2026-08-01 11:10:15 +05:30
Israel Lot e3bc517034 fix(agent): stop double-counting api_content in the token estimator
`api_content` is a SUBSTITUTE for `content`, not an addition to it.
`turn_context.substitute_api_content()` pops the sidecar and overwrites
`content` at every API-bound message-build site (the `api_messages` build
in `conversation_loop`, the max-iterations summary in
`chat_completion_helpers`, the chat-completions transport), so exactly one
of the two is ever sent to the provider.

The preflight estimator counted both, because both `_estimate_message_chars`
and `_estimate_message_tokens_without_images` walked every key of the
persisted dict with a single-entry denylist (`_anthropic_content_blocks`).
Any message whose sidecar differs from its clean stored content was counted
twice — exactly 2.00x on a 40KB sidecar.

The sidecar exists to keep the provider prompt-cache prefix byte-stable, so
it is written on precisely the long, cache-pinned messages where the
doubling hurts most. Because `estimate_messages_tokens_rough()` also feeds
the compaction threshold via `context_compressor` and `conversation_loop`,
the inflated estimate makes compression fire on phantom bytes.

Fix: substitute rather than sum, mirroring the wire. The two estimator
helpers had drifted into near-identical copies of the same shadow-building
loop, so this factors the shared logic into `_wire_message_shadow()` and
fixes the class once instead of patching one site and leaving the other.

Image accounting is unchanged: base64 payloads are still replaced with a
placeholder and charged at the flat `_count_image_tokens` rate, and the
`_multimodal` text_summary path is preserved.

Tests: three cases in `TestEstimateMessagesTokensRough` — sidecar equal to
content is counted once, a sidecar that DIFFERS is still counted (a lower
bound, so it fails if the field were dropped rather than substituted, which
would undercount the real request), and a sidecar cannot smuggle raw base64
past the flat image rate.

Verified on Linux (Python 3.11): 53 passed in
tests/agent/test_model_metadata.py, 57 passed with
tests/agent/test_context_breakdown.py, 656 passed / 3 skipped across the
compression/context/token/estimate/prune surface of tests/agent.
Mutation-tested: reverting the substitution fails the new equality test.
`scripts/check-windows-footguns.py` is not applicable — no file I/O,
process management, terminal handling, subprocesses, or signals.
2026-08-01 11:10:15 +05:30
MaxFreedomPollard 7729c183b4 test: restore four silently shadowed definitions and guard against more
Python keeps only the last definition of a name in a scope, so a duplicate
silently deletes the first. Four had accumulated, and two cost real coverage.

tests/agent/test_auxiliary_client.py grew a second _clean_env autouse fixture
alongside an NVIDIA feature. Being module-level with the same name, it
replaced the original for all 158 tests in the file: the ANTHROPIC_API_KEY,
ANTHROPIC_TOKEN, CLAUDE_CODE_OAUTH_TOKEN, OPENAI_MODEL, LLM_MODEL and
NOUS_INFERENCE_BASE_URL stripping went away, and so did the _aux_unhealthy_*
cache reset between tests. Two tests had already started clearing that cache
by hand to work around the leak, one of them with a comment describing the
pollution. The NVIDIA keys are folded into the original fixture and the
duplicate removed.

tests/gateway/test_mattermost.py had two copies of
test_progress_send_with_invalid_thread_root_never_falls_back_flat. The
surviving copy omitted the recorded 400 "invalid root_id" state, so the case
the name describes never ran. Both now run under distinct names.

The other two were harmless but hid the pattern: an exact duplicate in
test_tts_media_routing.py, and a dead _codex_auth_store in
test_credential_pool.py that nothing calls.

tests/test_no_shadowed_test_definitions.py walks every test module and fails
on any repeated definition in one scope, exempting the property/setter/
register family and throwaway _ callbacks. It fails on main listing exactly
these four.
2026-07-31 22:35:16 -07:00
MaxFreedomPollard 65e9ece964 fix(curator): restore the real skills tree when a rollback extract dies part-way
shutil.move() moves into an existing destination directory instead of
replacing it. When the snapshot extract failed after creating some of its
output, the recovery path moved each staged entry onto a path the extract
had already created, burying the user's own skill one level deeper
(skills/alpha/alpha/) and leaving the snapshot's partial content in its
place. rollback() then returned "snapshot extract failed (state restored)".

Clear the failed extract's output before moving the staged copies back:
entries the original tree never had are dropped, and each staged entry's
destination is removed before the move. When an entry still cannot be
restored, keep the staging dir and name the entries in the message rather
than reporting a restore that did not happen.
2026-07-31 22:34:57 -07:00
praneshnikhar 35b44e0d5f fix(auth): apply same source-path write-through fix to non-pool xAI path
_save_xai_oauth_tokens had the identical self-sealing bug as
_sync_device_code_entry_to_auth_store: key-presence check before
_store_provider_state, which unconditionally creates the key.
Use _load_provider_state_with_source to decide write-through from
the actual grant source, not key presence.

Also update regression test per review: use the real
_write_through_provider_state_to_global_root helper and assert
rotated token pair values in the root store after each refresh
instead of just counting mock calls.
2026-07-31 22:32:48 -07:00
praneshnikhar 4e6299af48 fix(credential_pool): use source-path-based write-through to root (#74339)
_sync_device_code_entry_to_auth_store used key-presence on the profile
store to decide whether to write-through rotated tokens to the global
root.  _store_provider_state unconditionally creates that key, so every
refresh after the first self-disabled the write-through — root kept a
revoked refresh token and every other profile died with
refresh_token_reused / invalid_grant.

Fix: use _load_provider_state_with_source to learn where the grant was
resolved from.  When the source is the global root, write back only to
root and skip _store_provider_state so the profile never accrues a
shadowing providers.<id> key that blocks future root fallback.

Add regression test verifying write-through fires on refresh 2+, not
just the first call.
2026-07-31 22:32:48 -07:00
Gille 29eac371d1 fix(context): persist NVIDIA DeepSeek endpoint limit 2026-07-31 22:31:22 -07:00
Teknium 1737741730 fix(copilot): follow-ups for salvaged PR #58743
- Bound ALL reads of the on-disk JWT store through one _read_jwt_store()
  helper (load, eviction, save-merge) — the 1 MiB cap previously only
  covered the load path; eviction and save could still parse an
  oversized/corrupt store and rewrite it back out (sweeper finding).
- Fix the class, not the site: the recovery gates checked the literal
  provider == "copilot" while /model and profile configs can leave the
  alias spelling in place (the reporter's own log shows provider=copilot
  AND provider=github-copilot in one session — the aliased turns would
  have silently skipped recovery). Single owner:
  AIAgent._is_copilot_provider() (slug aliases + Copilot base-URL
  fallback), used by both run_agent recovery methods and both
  conversation_loop gates.
- Update the salvaged 401 test to current main's client-retirement
  contract (release deferred to GC — no synchronous .close()).
- Add copilot_stale_cred_retry_attempted to the TurnRetryState field
  contract test; add bounded-store and alias-gate regression tests.
2026-07-31 22:31:09 -07:00
Kyle 48ded07155
fix(codex): handle completed auxiliary responses 2026-07-31 21:52:36 -07:00
knocksen 716d274f58
test(moa): cover direct provider streaming for Codex-shim aggregators
Narrow the moa_aggregator Relay bypass to CodexAuxiliaryClient and add
coverage that call_llm(stream=True) returns the provider's direct
create() result for Responses-shim clients.

Salvaged from PR #74903 by @liusencomic-cyber.
2026-07-31 21:52:35 -07:00
AKAZIK-py c23ff21d7c
fix(moa): carry completed responses through the managed Relay path
Under managed Relay execution the provider factory runs lazily inside
provider_stream() on the Relay session's event loop. The MoA facade's
auxiliary call_llm(stream=True) is invoked from that callback, so the
eager final_response check added for the non-managed path never fires:
the inner ManagedLlmStream is returned to the outer stream, which then
synchronously iterates it on the same loop thread and dies with
RuntimeError: Cannot run the event loop while another loop is running
(the completed response effectively trapped one level deeper).

stream_current() now detects a running event loop and returns the raw
factory result instead of nesting a ManagedLlmStream: the outer managed
stream already provides Relay tracking for the enclosing attempt, and
its own completed_response_predicate traps the completed response as
final_response — the same contract the main streaming worker consumes
(chat_completion_helpers reads stream.final_response after the chunk
loop). Nested managed streams remain supported for genuinely streaming
providers via the outer stream's own iteration.

Adds managed-execution regressions using the retained relay_turn
fixture: direct completed-response trapping, and the nested
facade-shaped stream_current call.
2026-07-31 21:52:35 -07:00
AKAZIK-py 52705e496f
test(moa): cover call_llm(stream=True) completed-response unwrap at the outermost seam 2026-07-31 21:52:35 -07:00
AKAZIK-py 654915f187
fix(moa): unwrap completed responses in the auxiliary streaming path 2026-07-31 21:52:35 -07:00
teknium1 950fe236d0 fix(security): extend secret redaction to GitLab token families
Port from openclaw/openclaw#112954. The redactor knew GitHub, Slack,
Google, Stripe, AWS access-key-ID and ~25 other vendor prefixes but had
zero GitLab coverage — glpat-/gloas-/gldt-/glrt-/glrtr-/glcbt-/glptt-/
glft-/glimt-/glagent-/glsoat-/glffct-/glwt- tokens and legacy GR1348941
runner registration tokens passed through display and log surfaces
verbatim. Follow-up explicitly invited when #4541 was closed.

Each pattern keeps a full literal prefix so the _PREFIX_SUBSTRINGS
pre-screen (derived at module load) stays false-negative-free; routable
runner tokens allow dotted segments. Sibling site: skills_guard's
credential-exposure scan gains a gitlab_token_leaked pattern.
2026-07-31 21:31:10 -07:00
Alex Fournier ac1045e571 Merge current skill metrics into client resource metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:55:02 -07:00
Alex Fournier e3d10f0a8f Merge current tool metrics into skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:54:38 -07:00
Alex Fournier 0de9b65c23 Merge upstream main into model route metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:53:12 -07:00
Alex Fournier 9b8d4133db Merge updated skill metrics into client resource metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	docs/observability/relay-shared-metrics.md
#	hermes_cli/observability/shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics.py
2026-07-31 07:36:57 -07:00
Alex Fournier 3d5fcab70f Merge updated tool metrics into skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	hermes_cli/observability/schemas/hermes.shared_metrics.v1.schema.json
#	scripts/smoke_nemo_relay_shared_metrics.py
#	tests/agent/test_skill_commands.py
#	tests/hermes_cli/test_relay_shared_metrics.py
#	tests/hermes_cli/test_relay_shared_metrics_runtime.py
#	tests/tools/test_skill_manager_tool.py
#	tests/tools/test_skill_usage.py
#	tests/tools/test_skills_tool.py
2026-07-31 07:30:30 -07:00
rob-maron 126ff7071b
Portal free user vision fix + flux3 polling improvements (#75448)
* flux3 polling improvments

* poll gap to 4s

* back to 5s

* vision model fix

* minor fix
2026-07-31 10:17:55 -04:00
Alex Fournier 0558ea0c48 Merge upstream main into feat/hermes-relay-model-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 07:04:37 -07:00
Alex Fournier 43d29a37c8 fix(observability): include auxiliary model routes
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-31 06:58:41 -07:00
kshitijk4poor 53559aaf86 fix(agent): protect batch-compaction markers from micro supersede/defrag
Phase 2 review findings on the salvage branch:

C1 (critical): batch and micro summary markers share
COMPRESSED_SUMMARY_METADATA_KEY, and compress() never reset micro state.
After micro absorbed exchanges 1..k, a batch compaction summarizing
1..m (m>k) could fire; the next micro pass's supersede then dropped the
batch marker (whose content the stale rolling summary does NOT contain)
and archive_and_compact immediately made the loss durable. Defrag had
the same hazard: it rewrote "the newest marker" even if that was a
batch marker. Empirically confirmed with a probe (batch marker content
destroyed in one pass).

Fix, three parts:
- Micro-created markers now carry MICRO_COMPACT_MARKER_KEY; supersede
  and defrag only ever touch micro-tagged markers. Rehydration in
  _resolve_compact_cursor tags the marker it absorbs (containment
  proof), which safely covers adopting a batch marker as the new
  rolling base after a reset.
- compress() success path resets micro rolling summary/cursor state so
  a stale summary can never claim cumulativeness over a batch marker.
- Regression tests for both directions plus the reset.

W4: _splice_micro_compact_result no longer strips _db_persisted stamps
from surviving messages. Micro archives in place under the SAME session
id (unlike batch's child-session rotation, #57491), so surviving stamps
are accurate; stripping them meant an archive_and_compact failure left
every previously-persisted message unstamped and the next append-only
flush re-inserted them all as duplicate active rows.

W5: finalize_turn micro gate now checks agent._persist_disabled —
persistence-isolated fork agents (background review) must not burn an
aux call per review turn, and must never archive_and_compact the
canonical session rows if their compressor ever gains a DB binding.

W1: _serialize_one_exchange now delegates to _serialize_for_summary
(was a ~70-line near-verbatim copy; one serializer, one place to fix).

S4: _find_one_exchange boundary guard rejects only assistant/tool
boundaries (the actual alternation hazard) instead of requiring user —
a stray mid-list system/injected message can no longer wedge the
cursor forever.

5 new regression tests; 38 micro/prune tests, 400 compression-suite
tests, 61 finalize/persist tests pass; ruff clean.
2026-07-31 17:44:19 +05:30
kshitijk4poor b8bfd68af1 fix(agent): make micro-compaction alternation-safe and defrag user-preserving
Two integration bugs found during review of #74522, both confirmed with
empirical probes against the production message-repair path:

1. Alternation: the summary marker was role="user" and an exchange was a
   single assistant+tools group, so splicing between two user turns produced
   user -> marker(user) -> user. The pre-request repair_message_sequence pass
   (conversation_loop.py, runs before EVERY API call) then merged the marker
   into the neighbouring real user message: metadata gone, cursor
   unrecoverable on resume, and the summary text duplicated into the
   transcript on every later pass (the transcript GREW every turn).
   Fix: an exchange is now a full agent turn (assistant + tools + follow-up
   assistant iterations, bounded by user messages), the marker is
   assistant-role, and superseding an old marker deliberately merges the two
   adjacent real user turns (plain-text \n\n-join, identical to repair
   pass 2) so the returned transcript is alternation-valid by construction.
   Probe result: repairs 0 (was 2), marker survives, no summary leakage.

2. Defrag destroyed user messages: _defrag_rolling_summary serialized the
   whole remaining middle (user turns included) and spliced it away —
   8 of 10 user prompts destroyed in one pass, contradicting the feature's
   "your messages are never compacted" invariant. Fix: defrag now
   re-summarizes only the rolling summary TEXT and rewrites the marker
   content in place; transcript shape, cursor, and user turns untouched.
   Probe result: 10 of 10 user prompts survive.

Also: marker provenance is now COMPRESSED_SUMMARY_HAS_USER_TURN_KEY=False —
micro markers absorb only assistant/tool content (#64650 invariant), and
real user turns remain in the transcript for provenance detection.

Adds 5 regression tests (repair-pass integration, alternation on
multi-iteration tool turns, defrag user survival, defrag input scope,
marker provenance); updates the two existing tests and the design doc to
the corrected semantics. 28 tests pass.
2026-07-31 17:44:19 +05:30
Michael Jordan 9ca4ee72ca feat(agent): make the micro-compaction cadence configurable
The on/off switch was the only knob. A pass fired after every completed turn,
absorbed exactly one exchange, and there was no way to ask for less. Since a
pass is also what breaks the prompt-cache prefix, "how often does it run" and
"how often do I pay a cache break" are the same question, and it had no answer.

Add `compression.micro_compact_every_n_turns` (default 1, clamped to >= 1). At 1
the behaviour is what it was; at 5 you get a fifth of the breaks and a fifth of
the reclaim rate. The counter advances per invocation rather than per committed
pass, so a turn that finds nothing to absorb still moves the cadence along and
cannot wedge it, and a bogus 0 or negative degrades to "every turn" instead of
silently disabling compaction.

Also expose `micro_compact_defrag_threshold_tokens`, which has been a hardcoded
attribute on the compressor with no path from config since it was added.

This does not give micro-compaction the prune's reclaim-size gate -- a pass
still commits whatever the single absorbed exchange saved. It makes the break
frequency tunable, which reaches the same end by absorbing less rather than by
waiting for a bigger win. The docs now say that plainly, including that a
reclaim threshold is the obvious follow-up and does not exist yet.

Tests cover the skip-until-due window, the cursor and prefix staying untouched
on skipped turns, the clamp, and that the feature is off unless enabled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan 60781a0cc8 fix(agent): do not destroy compacted history when a session resumes
The rolling summary lives only in memory. A resumed session starts with an
empty one while the marker carrying every previously absorbed exchange is
still in the transcript. The first pass after a resume therefore built a
marker from a single exchange and superseded the marker holding the entire
history -- silently discarding everything micro-compaction had accumulated.

This was introduced by the supersede fix. Before it, markers piled up
wastefully, but nothing was ever lost.

Two changes, so a single failure cannot lose data:

Rehydrate. When the cursor is recovered by scanning the transcript -- the
resume path -- also recover the rolling summary from that marker, so the
next pass merges into the existing history instead of replacing it.
Extraction uses rfind for the heading because SUMMARY_PREFIX references the
heading text itself, so the first occurrence is inside the preamble.

Gate superseding. Earlier markers are dropped only when this pass's summary
is demonstrably cumulative, i.e. the rolling summary was non-empty going in.
If rehydration ever fails, the pass keeps both markers: wasteful, but the
history survives.

Tests cover the resume path, the failed-rehydration fallback, and the
round trip of a summary through a marker.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan d626528560 fix(agent): derive the micro-compaction cursor from the spliced list
The cursor was set to the pre-splice `exchange_end`. A splice collapses the
absorbed span -- an assistant plus its tool results, often four or more
messages -- into a single marker, and may also drop a superseded marker
further back, so every index after it shifts.

The stale cursor therefore overshot, landing inside a *later* exchange's
tool group. The next pass's `_find_one_exchange` walked forward from there
to the following assistant, so the exchange it had landed inside was never
absorbed at all. On tool-bearing conversations micro-compaction was
silently doing roughly half the work it should.

Traced on a 3-tool-per-exchange transcript: the cursor sat at index 6 when
the marker was at 2, and the message count stalled at 32 instead of
continuing to 28.

Derive the cursor from the marker's actual position in the spliced result
instead, which is self-correcting regardless of how much the splice moved.
Apply it on the defrag path too, which had the same staleness.

Found by a randomized long-horizon harness (480 conversation shapes x 25
passes, varying tool-group sizes and summarizer failure modes) asserting
structural and progress invariants after every pass. Existing tests missed
it because their fixtures have no tool results, so the absorbed span is one
message and nothing shifts. The regression test uses tool groups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan ac48add3a7 feat(agent): report context occupancy, not just tokens saved
Tokens saved is the wrong headline for this feature. Micro-compaction is
not an efficiency optimisation — the same summarization work happens either
way. What it buys is (a) that work amortized across turns instead of one
stall, and (b) a window kept low enough that a session runs much further
before needing a hard compaction at all.

Neither shows up in "net tokens saved". A session can save nothing on paper
and still be a clear win on both counts.

So the telemetry now carries occupancy: tokens_after as a share of the
compaction threshold, plus the threshold and resolved window it was
computed from. That is the number that says whether a session has headroom
left. The report leads with it, and cross-references the batch
`compression_attempt` lines already in the log so it can show how often the
long pause actually fired — ideally never.

Occupancy is read from the cached threshold only. The public
`threshold_tokens` property resolves lazily and can issue a synchronous
/models probe (#32221); telemetry must never be the thing that blocks a
turn, so an unresolved window reports null. In practice a pass has already
resolved it via the tail calculation, so the field is populated. A test
pins the no-forcing behaviour directly against the emitter.

The report is pure ASCII: `scripts/check_subprocess_stdin.py` currently
dies on a cp1252 console before printing its results, and a diagnostic tool
that crashes on the platform it is diagnosing is worse than no tool.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan cac9526d2a feat(agent): token telemetry for micro-compaction
The existing log line reports message counts, which is the least
informative number available here: absorbing one tool-heavy exchange can
drop hundreds of tokens while moving the count by one. There was no way to
answer "is this actually helping?" from a real session.

Emit one content-free JSON line per pass, in the same shape as the batch
compaction telemetry: before/after tokens, the delta, the size of the
absorbed exchange, the rolling summary size, duration, and running
per-session totals so a whole run can be read off the last line. No
transcript content rides along.

Add scripts/micro_compaction_report.py to aggregate those lines into
passes, outcome mix, net tokens saved, mean exchange size and durations,
with an optional per-session breakdown.

Measuring it immediately surfaced something worth documenting: the first
pass in a session normally *costs* tokens. The summary marker carries a
fixed ~400 tokens of scaffolding, paid on pass one against a single
absorbed exchange. From pass two the marker is replaced rather than added,
so the overhead is already paid and each exchange is close to pure saving.
Break-even is typically the second or third pass. Tests cover the
telemetry contract, the cumulative totals, and that first-pass/later-pass
shape so nobody reads a single turn and concludes it made things worse.

The estimator costs ~5 ms at 600 messages and ~20 ms at 1200, taken twice
per pass, post-turn — and only once an exchange is actually in hand, so
turns that no-op early pay nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan 214b5d8612 docs(agent): state that user turns are never micro-compacted
`_find_one_exchange`'s docstring described an exchange as "(optional) user
message + assistant message + its tool results", but the walk skips past
user messages and starts at the assistant, so user turns are never absorbed
into the rolling summary.

The code is right and the docstring was wrong. Assistant output is largely
an account of what was done and survives summarising with little loss. The
user's messages are the intent everything else is derived from and cannot be
reconstructed from the work that followed — paraphrasing "use the existing
helper, don't add a new one" into a summary is how an agent ends up doing
the opposite six turns later. They are also cheap: a prompt is normally a
tiny fraction of what one tool result costs.

Correct the docstring, document the property (and its cost — a floor on how
small the middle can get, since user turns accumulate), and add a test so it
stays deliberate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Michael Jordan 186cad02f9 feat(agent): per-turn micro-compaction to amortize context compression
Batch compaction pauses a session for one large summarization once the
window fills. Micro-compaction spreads that cost out: after each completed
turn, `finalize_turn` folds the single oldest un-absorbed exchange
(assistant message plus its tool results) into a rolling summary, so the
work happens in small increments during post-turn idle time instead of one
long stall.

Mechanics:
  - a cursor tracks the first message not yet absorbed, recovered from the
    transcript's last summary marker when in-memory state is unavailable;
  - protected head and tail windows are never touched, so the system prompt
    and recent turns stay verbatim;
  - the absorbed span is replaced by a marker carrying the usual
    `_compressed_summary` metadata, so resume, handoff and `/compress`
    treat it exactly like a batch summary;
  - `archive_and_compact` keeps the session DB in step, otherwise the
    append-only flush would leave the original rows active and a resume
    would double-load both summary and originals;
  - when the rolling summary itself passes a token threshold it is
    defragged: re-summarized in one shot and the cursor jumps to the tail;
  - an exchange the summarizer can't handle is retried a bounded number of
    times, then skipped, so one poison exchange can't stall every turn.

Keep only the newest summary marker. The rolling summary is cumulative, so
each marker already contains everything the previous ones held; leaving them
stacked near-duplicate copies of the same text, each with its own heading and
end-marker scaffolding, and the transcript grew on every turn instead of
shrinking. Measured over six turns on a 12-exchange conversation with tool
output: 4104 -> 4797 tokens before, 4104 -> 2572 after.

Off switch: `compression.micro_compact: false` (default on).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 17:44:19 +05:30
Alex Fournier c0369f0891 Merge remote-tracking branch 'origin/main' into feat/hermes-relay-model-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-30 11:12:51 -07:00
Matt Ezell 7965462d6c fix(compression): choose summary role by template-visible alternation
The compaction summary's role was selected against the LITERAL
neighbouring messages (compressed[-1] / tail_messages[0]). Mistral-family
chat templates (Devstral, Mistral Small 3.x, Magistral) enforce
user/assistant alternation but exempt the tool flow (tool results and
assistant messages carrying tool_calls) from the check, so a protected
head ending [user, assistant(tool_calls), tool] pinned the summary to
role="user" while the last role the template counts is "user": the
backend rejects the whole request with a Jinja alternation error
(HTTP 500). The summary persists in the stored conversation, every
retry replays the identical poisoned history, and the session is
permanently unrecoverable. Fires on EVERY compaction against a
Mistral-strict backend, captured byte-exact via a tee-proxy in front of
a llama.cpp/llama-swap Devstral deployment.

Fix: compute both neighbour roles through _template_visible_role(),
which skips template-exempt messages. The #52160 (Anthropic user-first)
and #58753 (zero-user-turn) forced-user guards are preserved; their
forced shapes (summary-user followed only by exempt messages) are
alternation-safe. When the visible head ends "assistant" and the
visible tail opens "user", no standalone role can alternate and the
existing merge-into-tail fallback now correctly fires (the literal
logic emitted a standalone user summary there: a second poisoning
shape).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGN45sMMbwM8cW9T9ga4ou
2026-07-30 21:07:36 +05:30
Kyzcreig 9c28600e77 test: prove concurrency with barriers/witnesses instead of wall-clock bounds
Replace OS-scheduler-dependent elapsed-time ceilings with deterministic
concurrency proofs in three tests:

- test_context_refs_concurrent: asyncio.Barrier rendezvous — all 3 URL
  fetches must be in flight simultaneously before any returns.
- test_memory_boundary_commit: positive non-blocking witness — the
  provider call list must still be empty when the async commit returns.
- test_mem0_v3 slow-prefetch: threading.Event park/release — prefetch
  must return while the backend search is still parked.

The tests/tools/test_mcp_tool.py hunk from the original PR is dropped:
main no longer carries the 'elapsed < 2.5' assertion it targeted
(superseded by a delay-relative bound).

Salvaged from #71913.
2026-07-29 21:30:53 -07:00
Teknium ad12df6ba4 Revert "remove Vercel AI Gateway and Vercel Sandbox (#33067)"
This reverts commit febc4cfec0.
2026-07-29 19:48:37 -07:00
y0shualee 8d009e4f3e test: guard browser and Keychain side effects suite-wide (#35404)
Re-port of PR #35464 onto the rewritten hermetic conftest:
- autouse _neutralize_webbrowser fixture records open/open_new/open_new_tab
  and webbrowser.get() instead of launching a real browser
- autouse _neutralize_macos_keychain_creds defaults the Anthropic Keychain
  reader to None, with an opt-in allow_macos_keychain marker
- regression tests in tests/test_hermetic_side_effect_guards.py
- tests/agent/test_anthropic_keychain.py opts in via pytestmark

Salvaged-from: #35464
Co-authored-by: y0shualee <yuxiangl490@gmail.com>
2026-07-29 18:55:10 -07:00
Kyzcreig bcec6c8d39 fix(test): hermetic env-detection tests — pin container/supervisor/HOME probes (#422)
The restart-routing, systemd-support, and subprocess-HOME tests asserted
branch behavior but left part of the real probe surface unmocked, so they
fail when the suite itself runs inside a container (self-hosted CI) or a
launchd-descended shell:

- /restart routing tests: the handler also consults the real /.dockerenv —
  extract the inline probe to gateway.restart.is_container_restart_context()
  (patchable seam, no behavior change) and pin it False; scrub ALL four
  supervisor env markers (ambient XPC_SERVICE_NAME on macOS flipped one).
- supports_systemd_services tests: pin shutil.which('systemctl') and
  is_container() so the test asserts the branch, not the host.
- copilot ACP real-HOME test: pin is_container() (auto mode prefers profile
  home in containers) and scrub ambient HERMES_REAL_HOME/TERMINAL_HOME_MODE.

97 tests green on macOS dev box AND inside a docker CI runner container.

Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
2026-07-29 18:55:10 -07:00
Teknium 67435c9a02 fix(tests): restore original module identities after vision-routing reloads (#61597)
_fresh_modules() in test_vision_routing_31179.py deleted agent.image_routing
(+siblings) from sys.modules and never restored the ORIGINAL modules — the
reloaded copies leaked. Any later same-process test holding function refs to
the original module (test_image_routing.py) then patched the reloaded copy
via mock.patch string targets, making patches invisible: order-dependent
failures (repro: 31179 file first → 2 failures; reversed → green).

Autouse fixture now snapshots the affected sys.modules entries and restores
them on teardown. Both orders + solo runs verified green.

Masked by the per-file-isolated canonical runner; bites anyone running
pytest tests/agent/ directly.
2026-07-29 18:55:10 -07:00
Teknium 3913ac2ba0
test: pair load_config_readonly stubs at pruned-file sites (post-merge with main's readonly sweep)
Main's 243c9182b1/a16fd675df/7142dc4580 added load_config_readonly
sibling stubs across 38 files; our pruned versions of 11 of those files
kept only the load_config stubs. Re-applied the pairing at every
surviving site (26 patch()/setattr sites) — same return_value/
side_effect as the adjacent load_config stub. 494 tests green across
the 11 files.
2026-07-29 15:37:24 -07:00
teknium1 a16fd675df test: pair attribute-form load_config stubs with readonly siblings
Second stub shape the first sweep missed: monkeypatch.setattr(config_mod,
"load_config", ...) — attribute-form instead of string-form. Four files
(compression_max_attempts, preflight_compression_cap_e2e,
codex_gpt55_autoraise_notice, proactive_prune_config) stub agent_init
config reads that now go through load_config_readonly(). Swept the whole
tree for the attribute form; remaining hits stub modules that still use
the mutable loader. 1,210 tests green across the 19-file re-sweep.
2026-07-29 15:28:15 -07:00
teknium1 243c9182b1 test: patch load_config_readonly alongside load_config across affected suites
The readonly swaps mean agent/ modules now read config through
load_config_readonly(); tests that stubbed only load_config stopped
intercepting those reads. Added sibling readonly stubs (same
return_value/side_effect/lambda) at every affected site across 11 test
files, found via a tree-wide sweep of load_config stubs cross-referenced
against the swapped modules. 3,915 tests green across the 38-file
sibling sweep.
2026-07-29 15:28:15 -07:00
Teknium 7c5a98d888
Merge remote-tracking branch 'origin/main' into tests/prune-low-value
# Conflicts:
#	tests/run_agent/test_conversation_fallback_state.py
2026-07-29 15:24:14 -07:00
Teknium 1a088989bc
Merge pull request #66730 from NousResearch/feat/hsp-sync-client
feat(sync): HSP/1 personal skill sync client (M1 client)
2026-07-29 15:20:35 -07:00
Teknium 28524adb0e fix(tests): eliminate flaky/broken tests — shadow sys.path inserts, unmocked network in compressor tests, stale-SDK feishu pin guard, quadratic redact regexes
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
  it prepended the tests/ dir itself to sys.path, so 'import agent' /
  'import hermes_cli' resolved to the test packages and collection died
  with ModuleNotFoundError depending on import order (2 files failed in
  every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
  unmocked: each burned ~50s attempting live LLM traffic through the
  relay before falling back (572s file — the slowest in the suite, and
  flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
  the compressor's redaction pass on large payloads —
  _STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
  prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
  output-equivalence fuzz-verified on 20k random strings), and the
  _CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
  pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
  signature check; the repo pins lark-oapi==1.6.8 but stale local
  installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
  agent.redact + agent.credential_persistence in the fake agent package
  (empty __path__ blocks all real agent.* imports added since the fake
  was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
  2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
  baseline run (passes instantly when the box is quiet).
2026-07-29 15:12:28 -07:00
Teknium 40ec417881
Merge remote-tracking branch 'origin/main' into tests/prune-low-value
# Conflicts:
#	tests/hermes_cli/test_install_cua_driver.py
#	tests/run_agent/test_codex_app_server_integration.py
#	tests/test_tui_gateway_server.py
#	tests/tools/test_computer_use_delivery_ladder.py
#	tests/tools/test_zombie_process_cleanup.py
2026-07-29 14:12:25 -07:00
Teknium 39975613b1
test: prune wave 2 + speed fixes — 28,106 → 19,757 test functions, suite wall 315s → 294s
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.

Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
  (DEFAULT_CONFIG smart-approval leaked in) — pinned approval
  mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
  silently probed live provider catalogs (~2s/test) — stubbed
  cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
  shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
  MagicMock agents — interrupt.side_effect now clears _running_agents
  (22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
  (24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
  patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
  5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
  with rationale comment — the watchdog-dump assertion needs the loop
  blocked well past deadline+grace under parallel load (flaked once in
  the 40-worker verification run at a 0.2s margin).

Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
2026-07-29 13:39:40 -07:00
Teknium 6b81590c55
test: prune low-value tests suite-wide (wave 1) — 46,820 → 28,106 test functions
Systematic prune per AGENTS.md test policy, one pass over every major
test tree (gateway, hermes_cli, tools, agent, run_agent, plugins, cli,
cron, tui_gateway, honcho/openviking, root-level):

- DELETE: source-reading tests (read_text/getsource on prod files),
  change-detector tests (exact catalog counts, model-name snapshots,
  config version literals), mock-echo tests (assert a mock returns what
  it was told), assertion-free/trivial tests, near-duplicate
  parametrizations (boundaries + one representative kept), async/sync
  twin duplicates, cosmetic within-file variations.
- KEEP (mandatory): security/redaction/approval guards, message-role
  alternation invariants, prompt-caching/deterministic-call-id
  invariants, issue-number regression tests (deduped), E2E tests.
- 6 test files deleted outright (script-style/no-assert or fully
  redundant); conftest.py, fakes/, fixtures/ untouched.
- tests/acp/conftest.py added: autouse fixture stubs the live
  models.dev/GitHub/Copilot/Anthropic inventory fetches that ACP server
  tests performed on every session create — test_server.py 147s → 3.4s,
  and the tests are now genuinely hermetic.
- Sleep-based slowness shrunk where safe (codex_ttfb_watchdog,
  compression_concurrent_fork, etc.); no wall-clock assertion tightened.

Verification: full hermetic suite via scripts/run_tests.sh —
2439 files, 31,130 tests passed, 0 failed, 0 flaky retries, 315s wall
(baseline: 583s wall, 13,564s subprocess CPU).
2026-07-29 13:10:23 -07:00
Ben Barclay f5b68ad58b Merge origin/main into feat/hsp-sync-client
Resolves the PR's conflict with main (2252 commits). Two conflicts, both
"each side added an independent block in the same place" — kept both:

- gateway/run.py — the housekeeping loop. This branch adds the Skill Sync
  pulls inside the CURATOR_EVERY branch (12-space indent); main adds a
  stale-session auto-archive as a sibling `if` at loop level (8-space).
  Different scopes, so the naive union would have mis-nested the archive
  block into the curator branch; kept each at its own indent level.
- tools/skill_manager_tool.py — the _edit_skill result dict. This branch
  appends the org auto-propose note; main appends
  _add_description_prompt_preview(). Independent, order-insensitive.

No behaviour dropped from either side.

Verified: 3552 passed / 0 failed across 63 suites (scope regenerated to
include main's new maybe_auto_archive / _add_description_prompt_preview
consumers) via scripts/run_tests.sh. `hermes sync` and `hermes sync status`
still work against a live token, resolving the production plane default.

The Pyright Optional-parameter warnings in skill_manager_tool.py are
pre-existing on main (`content: str = None` etc.), not introduced here.
2026-07-29 13:01:36 -07:00
teknium1 f8758dcaf8 refactor(agent): single-owner call_id + reasoning_content sanitization policies (wire-parity verified) 2026-07-29 12:29:39 -07:00
Alex Fournier f1fd678e44 feat(observability): add Relay skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 12:20:34 -07:00
Teknium 3dd8059a05
fix(tests): eliminate flaky/broken tests — shadow sys.path inserts, unmocked network in compressor tests, stale-SDK feishu pin guard, quadratic redact regexes
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
  it prepended the tests/ dir itself to sys.path, so 'import agent' /
  'import hermes_cli' resolved to the test packages and collection died
  with ModuleNotFoundError depending on import order (2 files failed in
  every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
  unmocked: each burned ~50s attempting live LLM traffic through the
  relay before falling back (572s file — the slowest in the suite, and
  flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
  the compressor's redaction pass on large payloads —
  _STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
  prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
  output-equivalence fuzz-verified on 20k random strings), and the
  _CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
  pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
  signature check; the repo pins lark-oapi==1.6.8 but stale local
  installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
  agent.redact + agent.credential_persistence in the fake agent package
  (empty __path__ blocks all real agent.* imports added since the fake
  was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
  2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
  baseline run (passes instantly when the box is quiet).
2026-07-29 12:18:07 -07:00
teknium1 30c783589c perf(agent): cursor/memo optimizations for per-iteration full-history walks (byte-parity proven)
Three provably-safe optimizations for O(n)-per-iteration history walks:

1. sanitize_tool_call_arguments: optional identity-keyed cursor (strong
   refs to the exact validated message objects) skips re-json.loads-ing
   already-validated history each loop iteration. Any list rewrite
   (compression, repair, undo, steer) breaks the identity prefix match
   and forces re-scan from the divergence point. Wired via a per-agent
   cursor dict in conversation_loop.

2. estimate_messages_tokens_rough: per-message memo keyed on a deep
   identity fingerprint (strings pinned by strong reference so id()
   aliasing is impossible; scalars by value; dicts/lists structurally
   with key order). Equal fingerprints imply identical str(shadow)
   bytes, hence identical estimates. Unfingerprintable shapes fall
   through to direct compute. Bounded FIFO cache (4096 entries).

3. _flush_messages_to_session_db_unlocked: bounded scan that skips the
   identity-matched prefix of the previous successful flush's snapshot.
   Snapshot only taken on full success; cleared on exception. Compression
   rewrites use fresh copies, breaking identity and forcing full re-scan.

Parity proven in tests/agent/test_cursor_optimizations_parity.py:
500-message synthetic histories with tool calls, malformed args, unicode,
element-wise old==new across 3 iterations incl. simulated compression.

Measured (median of 5): sanitize 0.097ms->0.011ms, tokens 1.145ms->0.853ms,
persist-scan 179.5us->10.0us at 500 messages.
2026-07-29 11:54:18 -07:00
Alex Fournier dc4714b1e0 feat(observability): report model and provider usage 2026-07-29 11:24:13 -07:00
teknium1 c92e2c0fbf test: age the disk L2 entry too in the server-swap redetection test
The TTL-expiry test aged only the in-process probe cache; with the new
disk L2 the fresh disk verdict (correctly) served 'ollama' and the swap
to lm-studio wasn't re-detected. In real time-flow the 300s disk TTL
always lapses before the 1h in-proc TTL — the test now compresses both
expiries, matching the scenario it describes. 25/25 pass.
2026-07-29 10:52:13 -07:00
teknium1 d7a4065568 perf(local-endpoints): disk L2 for server-type + ollama ctx probes, faster timeouts
Local-model users paid a fresh probe waterfall on EVERY CLI cold start
inside AIAgent.__init__: detect_local_server_type (up to 4 HTTP GETs,
2s timeout each on a hung server) + /api/show (3s timeout). The
existing caches were in-process only, so back-to-back invocations
(chat -q, cron ticks, subagents) re-paid the network every time.

- New 300s-TTL disk L2 at HERMES_HOME/cache/local_endpoint_probes.json
  for detect_local_server_type verdicts and query_ollama_num_ctx
  results. Only SUCCESSFUL probes persist (a down server never pins a
  negative verdict); stale entries pruned on write; corrupted cache
  degrades to a miss; atomic writes. 300s is strictly fresher than the
  1h in-process TTL that already accepts server-swap staleness.
- models.dev fetch timeout 15 -> (5, 10) connect/read tuple: a
  blackholed connect stalled the first-turn critical path 15s; now
  fails in 5s (matches the OpenRouter fetch convention, #46620).
- _auto_detect_local_model timeout 5 -> (2, 3): runs inside
  _get_model_config() at startup against a LOCAL endpoint; a hung local
  server cost 5s before the banner.

E2E (real HTTP server, two fresh subprocesses, isolated HERMES_HOME):
proc1 = 2 HTTP hits, proc2 = 0 HTTP hits, identical results
(ollama/131072), probe wall 74.5 -> 35.5 ms. 222 targeted tests green
incl. 9 new disk-L2 contract tests.
2026-07-29 10:52:13 -07:00
teknium1 0034da2975 perf(stream): replace per-chunk repr() with delta-length byte estimate
The streaming hot loop computed len(repr(chunk)) on EVERY chunk to feed
the retry-diagnostic byte counter — a full recursive pydantic repr at
5.5-8.8 us per chunk (measured), ~20-30 ms of pure CPU per 3,000-chunk
response, paid on every streaming response on every platform.

New _estimate_chunk_bytes() sizes the chunk from its delta payload
strings (content / reasoning / tool-call arguments) plus a 40-byte
framing floor: 2.1-2.4 us per chunk (~3x cheaper), independent of
pydantic field count, never raises on unknown shapes (Anthropic events,
stub providers fall back to the floor). Both call sites switched
(chat-completions loop + anthropic event loop).

The counter feeds only the stream-retry diagnostic log line
(agent/stream_diag.py) — an estimate proportional to traffic preserves
its purpose (distinguishing 'died at 0 bytes' from 'died mid-stream').

6 new contract tests; 64 targeted stream tests green.
2026-07-29 10:21:59 -07:00
teknium1 3d48f893da refactor: single build_subprocess_env() factory for all child-process spawns (profile + secret-scrub single owner) 2026-07-29 10:14:11 -07:00
kshitijk4poor 222ea2b6c9 refactor: fold simplify-code review findings
- extract _commit_registry/_note_refresh_failure shared by the background
  worker and foreground stage-4 (identical 4-step success + failure paths
  were duplicated); worker now commits under _models_dev_fetch_lock so a
  failing background refresh can never re-arm the backoff immediately
  after a successful force_refresh committed (unsynchronized-write race)
- add should_clear_context_pin_async to hermes_cli/route_identity.py
  (matching the get_model_context_length_async precedent) and use it at
  the 4 async gateway sites instead of inline asyncio.to_thread wraps;
  the sync _format_session_info site keeps the sync call (already
  off-loop via its callers' to_thread)
- test the background-refresh success path (the PR's primary new
  behavior): disk saved, mem cache swapped, backoff cleared, in_flight
  reset — mutation-checked
- replace the race-prone spin-wait on _models_dev_refresh_in_flight with
  a named-thread join in the backoff test
2026-07-29 17:13:48 +05:30
kshitijk4poor ccf7129ed0 fix: restore zero-arg fetch_models_dev call on default paths
The branched call shape in get_provider_info/get_provider is deliberate:
~69 test sites across tests/hermes_cli and tests/gateway monkeypatch
fetch_models_dev (and get_provider_info) with zero/single-arg lambdas.
Passing allow_network= unconditionally broke 5 tests in CI slices 2/3/7.
Documented the constraint inline.
2026-07-29 17:13:48 +05:30
kshitijk4poor 11ca7eedf0 fix: follow-up hardening for salvaged #73621 + #35853
- _mark_stale_cache_grace only moves cache_time forward so a completed
  background refresh is never rewound to a 5-minute grace window
- clear _models_dev_refresh_in_flight if Thread.start() raises so a
  one-off thread-exhaustion failure doesn't disable refresh forever
- move empty-registry validation into _fetch_models_dev_from_network
  (was duplicated in the background worker and the foreground fetch)
- pass allow_network through as a plain kwarg in get_provider_info and
  hermes_cli.providers.get_provider instead of the branched call shape
- refresh the stale module docstring (no bundled snapshot exists; the
  resolution order now describes stale-serve + background refresh)
2026-07-29 17:13:48 +05:30
zapabob a479a1599f fix(models): use stale cache before models.dev refresh 2026-07-29 17:13:48 +05:30
StellarisW 8c50aaceb6 fix(gateway): keep models.dev refreshes off event loop 2026-07-29 17:13:48 +05:30
kshitijk4poor 24a56f027c fix(lsp): expose lsp.idle_timeout in config, harden reaper, log reaps
Follow-ups on top of @DonutsDelivery's salvaged reaper commit (#64141):

- create_from_config() now parses lsp.idle_timeout (invalid values fall
  back to DEFAULT_IDLE_TIMEOUT) — previously the constructor knob was
  unreachable from config.yaml (config exposure adapted from #36892 by
  @0xbWy and #68091 by @9miya20)
- canonical default declared in hermes_cli DEFAULT_CONFIG so config
  discovery surfaces the knob (per sweeper review note on #64980)
- reaper loop survives transient sweep errors instead of dying and
  silently re-opening the leak (gap flagged in #68091 review)
- eventlog.log_reaped(): one INFO line per sweep + clears the
  log_active announce cache so respawns re-announce at INFO
- docs: replace the stale 'no idle-timeout reaper' paragraph with the
  new lifecycle description + config reference
- tests: reuse-refresh protection (the regression teknium's sweeper
  requested on #64141), reaper-survives-error, config propagation,
  invalid-value fallback, DEFAULT_CONFIG/manager-constant sync
2026-07-29 17:11:30 +05:30
DonutsDelivery d7578018c5 fix(lsp): reap idle language servers 2026-07-29 17:11:30 +05:30
Teknium 63fc810b95 fix(gemini): sweep hardcoded Gemini default models to gemini-3.6-flash (#32360)
gemini-2.5-flash shuts down Oct 16 2026 (Google deprecation schedule)
and gemini-3-flash-preview is superseded. Update every hardcoded
default to the current GA flash model:

- gemini_native_adapter: probe_gemini_tier + _create_chat_completion
  default params; free-tier guidance de-pinned from a specific model's
  RPD number so it doesn't stale again
- auxiliary_client: gemini/kilocode fallback aux models,
  _OPENROUTER_MODEL, _NOUS_MODEL -> google/gemini-3.6-flash
  (verified live on both OpenRouter and Nous portal /models)
- provider plugins: kilocode + vertex default_aux_model
- hindsight memory plugin: gemini provider default
- setup wizard gemini list: 3-flash-preview -> 3.6-flash (matches the
  curated picker catalog)
- tests: aux-client assertions that pinned the old default literal now
  reference the _NOUS_MODEL constant, so the next default bump can't
  break them (change-detector cleanup)

Fixes #32360.
2026-07-28 18:17:56 -07:00
Ben Barclay 4f990ec09e refactor(sync): put every Skill Sync verb under `hermes sync`; drop HSP naming
Encapsulates the feature behind one command for launch, and adopts the
official product name.

One command:
- `propose` moves from `hermes skills propose` to `hermes sync propose`, so
  the whole feature is one command to learn and one to document. Its handler
  moves from cmd_skills to cmd_sync accordingly.
- The `hermes sync` parser now documents both halves plainly: personal sync
  across your devices, and sharing with your organisation. Added an examples
  epilog; rewrote the verb help in user language ("Include a skill in your
  sync" rather than "Opt a skill into sync").
- Every user-facing string that pointed at `hermes skills propose` now points
  at `hermes sync propose` (8 sites, including the agent-visible guidance
  returned by skill_manage and the org provenance header).

This also clears the way for #39343, which adds its own top-level `sync` for
git-repo profile backup — that feature nests under `skills`, this one owns
`sync`.

Naming:
- HSP / "Hermes Sync Protocol" is gone from prose, docstrings, and comments.
  The feature is "Skill Sync".
- Public identifiers renamed: HSPClient -> SyncClient, HSPError -> SyncError,
  HSPConflict -> SyncConflict, hsp_address -> wire_address, HSP_VERSION ->
  WIRE_VERSION.
- The WIRE names are deliberately NOT renamed: the `hsp_version` capability
  field and the `x-hsp-object-type` response header are set by the deployed
  gateway-gateway sync plane (verified in src/sync/syncRouter.ts), so
  renaming them client-side would break sync against a live server. A comment
  at the version constant records why they differ from the product name.
- The version-mismatch error is now actionable ("this server speaks sync
  version X, but this Hermes speaks Y — update Hermes to sync with it")
  instead of leaking the protocol acronym.

Also fixes a wiring gap found on the way: the gateway housekeeping tick
pulled personal skills but never org skills — the same defect already fixed
for the CLI. Org pull now runs there too, gated on real org membership.

Tests: the jargon guard now also fails on a bare "HSP". The two tests that
asserted the old cross-command structure are replaced by three asserting the
new one (propose IS under sync, propose is NOT under skills, sync usage
lists it). 2294 passed / 0 failed across all 51 suites that import the
changed modules, via scripts/run_tests.sh.

Verified by running the real CLI: `hermes sync --help` lists all eight verbs,
`hermes skills --help` no longer mentions propose, `hermes sync propose
--help` parses, and `hermes sync status` still reports live org state.
2026-07-29 07:47:06 +10:00
Atakan 01cb38e8ec fix(gemini): preserve bridged tool response name 2026-07-28 14:37:56 -07:00
Siddharth Balyan 3cedac00b7
fix(credits): reintroduce grant_spent, gated on an in-session crossing (#73634)
* Revert "fix(credits): remove the 'Grant spent · $X top-up left' notice"

This reverts commit 5dc6a14c14.

* fix(credits): reintroduce grant_spent behind an in-session crossing gate

The removed notice nagged because its condition is a steady state for
accounts living on top-up, the latch is per-session, and the cold-start seed
runs the policy at every session open — every session re-announced
'Grant spent · $X top-up left'. Reintroduce it gated so only a session that
WATCHES the grant run out announces:

- seen_grant_unspent crossing gate, mirroring seen_below_90: opens when the
  session observes the grant meaningfully unspent (>= GRANT_UNSPENT_MIN_MICROS,
  1 cent — portal-seeded states derive micros from float dollars and can carry
  sub-cent residue where headers report exactly spent). Seeds never prime it.
- The gate guards only the show branch and is consumed by the announcement —
  one announcement per crossing. Header flicker (used_fraction None and back)
  clears the sticky line but cannot re-announce; a renewal re-opens the gate.
- new_credits_latch() centralizes the latch shape (agent build, lazy re-init,
  and the policy test helper all build through it).
- Tests: steady-state-open stays silent (seed + policy seams), live crossing
  announces once, flicker/renewal/residual cases locked; the rendering test
  primes the gate and asserts its leg count so a gate regression cannot
  silently shrink coverage.
2026-07-28 21:36:23 +00:00
atakan g 713982a8f8 test(model-metadata): cover local Ollama fallback 2026-07-28 14:18:18 -07:00
atakan g 0c2d9aee0b fix(model-metadata): prefer local Ollama num_ctx 2026-07-28 14:18:18 -07:00
PRATHAMESH75 e45f2b39e2 fix(codex): clamp oversized Responses call_id so MCP tools don't brick sessions (#73492)
The codex app-server namespaces MCP tool call ids as
codex_mcp__<server>__<tool>_<codex_call_id>. With an exec-<uuid> component the
built-in hermes-tools server alone overflows the Responses API's 64-char
call_id limit, so the request 400s with a non-retryable "string too long".
The offending item sits near the head of the transcript and replays every
turn, permanently bricking the session — the only recovery is /reset.

Sibling defect to #10788, which clamped input[*].id via
_MAX_RESPONSES_ITEM_ID_LENGTH. Apply the same treatment to call_id at both
Responses emit sites in _chat_messages_to_responses_input: a deterministic
surrogate (call_ + sha256[:32]) for ids over the limit, short ids unchanged.
Because the surrogate is a pure function of the original id, a function_call
and its matching function_call_output — which carry the same original id — map
to the same surrogate and stay paired without correlating the two items.
2026-07-29 00:47:54 +05:30
Ivan Miao 26dd976fbe fix: align Gemini billing pricing with provider catalog 2026-07-28 11:54:46 -07:00
Ivan Miao ecebff82d2 fix: support Gemini billing route mapping and pricing update 2026-07-28 11:54:46 -07:00
Teknium 5dc6a14c14 fix(credits): remove the 'Grant spent · $X top-up left' notice
The grant_spent notice fired for every subscription user with top-up
funds the moment their cap was reached and camped in the CLI/TUI status
bar and desktop toasts with no action to take — the account keeps
working off top-up. Remove it everywhere:

- agent/credits_tracker.py: drop grant_cond + the emit/clear block;
  the dev fixture state now (correctly) produces no notice
- TUI: keep the turn-start clear of credits.grant_spent as back-compat
  for older backends that still emit the key
- Desktop: drop the demo step and stale comment references
- Docs/config comments: remove grant-spent from credits_notices text
- Tests updated: policy/cold-start now assert the key never fires

Usage bands, depleted, and restored notices are unchanged; /usage still
reports the full balance breakdown.
2026-07-28 11:21:44 -07:00
Alex Fournier 5e34fa2d5c fix(relay): preserve provider stream errors
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 09:58:43 -07:00
Alex Fournier b1a5d67e71 Merge upstream main into fix/hermes-relay-anthropic-context
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 08:10:58 -07:00
kshitijk4poor 2e9559adf0 test(compression): resolve context_length inside mock in overflow-warning fixture
CI shard 4 caught 4 failures the local baseline diff missed (local env
pollution made them look pre-existing): _make_compressor() constructs
under a get_model_context_length mock but the lazy deferral (#32221)
pushed resolution past the with block, so the 96K window / 75% floor the
tests rely on never materialized. Resolve inside the mock.
2026-07-28 20:04:58 +05:30
kshitijk4poor 49f50c68a0 fix(compression): coherent context_length setter — no-op guard, re-floor on new window, un-strandable init log
Review-pass findings on the lazy-init deferral:

- No-op guard: the codex app-server usage callback assigns
  compressor.context_length on EVERY response (same window each time).
  The setter unconditionally invalidated the derived budgets, wiping
  runtime corrections applied directly to threshold_tokens /
  tail_token_budget (aux-context threshold sync) — those persisted on
  main's eager init. Same-value assignment is now a no-op.

- Re-floor on genuinely new window: the setter invalidates budgets but
  previously kept the stale threshold_percent, so a codex window switch
  recomputed threshold_tokens from the new window with the old model's
  floored percent. Re-apply the raise-only small-context floor from
  _base_threshold_percent so percent and tokens derive from the same
  window (guarded with getattr for object.__new__ test instances).

- Init log extracted to _emit_init_summary_once() and also fired from
  the setter path, so a consumer assigning context_length before any
  read no longer strands the startup line forever.

- threshold_tokens getter resolves the window into a local before
  reading threshold_percent — correctness no longer depends on
  left-to-right argument evaluation order.

- reasoning_timeouts: fix inaccurate 'tuples are immutable' comment
  (the container is a list; safety comes from build-once-at-import),
  document why the slug stays in the tuple.

Adds TestContextLengthSetterCoherence (3 tests): same-value assignment
preserves overrides; new-window assignment re-floors both directions.
2026-07-28 20:04:58 +05:30
kshitijk4poor 5ce8e34767 test(compression): resolve context_length inside mocks across suites hit by lazy-init deferral
Baseline diff vs clean upstream/main (182 pre-existing environmental
failures on both sides) showed exactly 6 PR-caused failures, all the same
mechanism: tests construct ContextCompressor under a
get_model_context_length mock and read threshold_percent/threshold_tokens
after the with block, or build via object.__new__ and assign
context_length AFTER the derived budgets (the setter now resets the
lazily-cached budgets).

- test_compression_small_ctx_threshold_floor: resolve inside _make()
- test_cjk_token_estimation: resolve inside mock
- test_per_model_compression_threshold: resolve inside mock (2 tests)
- test_pre_compress_memory_context: assign context_length before
  threshold/tail/summary budgets; add summary_target_ratio
2026-07-28 20:04:58 +05:30
kshitijk4poor fd53fa3eac test(compression): resolve context_length inside mocks for tests added on main since PR base
TestThresholdTokensCap and TestLazyContextResolution landed on main after
the #38991 lazy-init base; they construct ContextCompressor under a
get_model_context_length patch and read threshold_tokens after the with
block. With deferred resolution the probe now fires lazily, so resolve
inside the mock (same pattern as the rest of the suite) and give the
lazy-resolution mock a real return_value.
2026-07-28 20:04:58 +05:30
kshitijk4poor 4da1abf789 test(caching): guard prompt-cache shallow-copy mutation-safety + byte-equivalence
Self-review found the deepcopy->shallow-copy change in
apply_anthropic_cache_control (#57046) had no test pinning the
"prompt caching is sacred / never mutate the caller's list" invariant.
The old test_returns_deep_copy only exercised the single marked message,
never an un-marked shared reference, so a regression that deep-copied too
little would pass the whole suite.

Add two tests: (1) caller list + every element left byte-identical after
the call, un-marked middle messages returned as shared references, marked
messages fresh copies, and mutating a returned marked message does not leak
upstream; (2) structural byte-equivalence vs a reference full-deepcopy
implementation across both native_anthropic modes and two TTLs.

Mutation-verified: neutering the per-message deepcopy makes test (1) fail.
2026-07-28 20:04:58 +05:30
kshitijk4poor 44bd0521a4 fix(compression): keep ContextCompressor init non-blocking when quiet_mode=False
The lazy-init change in #32221 deferred get_model_context_length() out of
ContextCompressor.__init__, but the "Context compressor initialized" log
(emitted only when quiet_mode=False) reads self.context_length,
self.threshold_tokens and self.tail_token_budget. Those property reads
resolve the deferred value, so the synchronous model-metadata probe still
ran inside __init__ on the interactive-CLI path — the exact blocking the
PR removed, just narrowed to the non-quiet path.

Emit the informative line once, on first context-length resolution, so
construction stays non-blocking on every path. Add a regression test
asserting no probe fires in __init__ with quiet_mode=False and that the
init log is emitted exactly once on first access.
2026-07-28 20:04:58 +05:30
Rod Boev f8d6f79c1a test(agent): fix lazy-init rebase test fallout (#32221)
(cherry picked from commit 920013f8d9300f66276808bc945789011cdb396f)
2026-07-28 20:04:58 +05:30
Rod Boev 958a81a1e4 perf(agent): defer synchronous httpx.post out of AIAgent.__init__ (#32221)
(cherry picked from commit 1fe63238b012e4326c975b2b1a303f6f27aa5102)
2026-07-28 20:04:58 +05:30
Soju06 82e2c9ce40 perf(agent): reuse the per-request OpenAI wire client across sequential LLM calls
Every LLM call built a fresh openai.OpenAI wire client (new httpx pool,
TCP+TLS handshake, measured 19.2ms p50 / 35.5ms p95 per call at ~5 calls
per tool-loop turn) and closed it when the request finished. Cache ONE
reusable wire client on the agent, keyed by the effective client kwargs:

- _create_request_openai_client hands back the cached client when the
  effective kwargs are identical; any change (credential rotation,
  provider failover, vision default_headers) evicts and rebuilds.
- Only a request that produced a response reports a reuse close reason
  (request_complete / stream_request_complete); error unwinds report
  *_error_cleanup and really close, so a retry after a request error
  always builds a fresh pool.
- Cross-thread aborts poison the slot: a pool whose sockets were
  shutdown(SHUT_RDWR) from a stranger thread is never reused (#29507) —
  the owner-thread close discards it and the next create rebuilds. The
  holder read and the abort are atomic (under the holder lock) at all
  three abort sites, so a late abort can never poison the NEXT request's
  checked-out client.
- Worker-side interrupt breaks close the half-read SSE stream on the
  owning thread before building the partial response; a failed close
  poisons the slot (otherwise each interrupt leaked one checked-out
  connection until the pool hit PoolTimeout). run_codex_stream gets the
  same poison-on-close-failure handling.
- Single checked-out slot (in_use): a concurrent call gets an untracked
  client with the old per-request lifecycle.
- release_clients() / close() really close the cached client when idle;
  if a worker has it checked out they abort the sockets and detach the
  slot, deferring the FD release to the worker's own close.
- MoA facade and Mock passthroughs never enter the cache; max_retries=0
  is preserved on all request clients.
2026-07-28 19:06:59 +05:30
kshitijk4poor a41dc65ba7 test: guard coalescing field lists against update_token_counts drift
Follow-up to PR #64171. The _TOKEN_DELTA_* classification must exactly
cover update_token_counts' keyword surface: an unclassified kwarg is
silently kept only from the first delta of a merged run. Introspect the
live signature and fail with a pointed message when a future kwarg is
added without classification (or a classified field is removed).
2026-07-28 18:47:54 +05:30
kshitijk4poor 927633272f fix: claim busy before clearing queue in _stop_token_writer drain
Follow-up to PR #64171. The writer loop and flush_token_counts'
caller-drain both set _token_writer_busy BEFORE popping the queue —
that ordering is what makes flush's lock-free fast path (reads queue
then busy, no cond held) sound. _stop_token_writer's leftover drain did
it backwards (clear queue, then set busy), leaving a few-bytecode window
where a concurrent flush could observe 'empty and idle' and return True
with the popped batch still unapplied. Shutdown-only staleness, no data
loss — but the protocol now matches at all three drain sites.

Test: concurrent flush during a stop-drain mid-apply must time out
(False), never report drained.
2026-07-28 18:47:54 +05:30
kshitijk4poor e49705d6af fix: respawn dead token writer and never let coalescing kill it
Follow-up to PR #64171. Two writer-death hardening gaps:

- queue_token_counts only spawned the writer when the thread object was
  None, so a writer that died from an unexpected exception could never be
  replaced: deltas piled up on the uncapped deque until a reader's flush
  drained them synchronously. Respawn on 'not thread.is_alive()' instead.
  (atexit re-registration on respawn is safe: unregister removes all equal
  bound-method registrations and the drain hook is idempotent.)

- _coalesce_token_deltas ran outside the per-delta try/except in
  _apply_token_batch, so a merge bug (e.g. an unclassified future kwarg
  summing None + 0) would escape and kill the writer thread. Wrap it and
  fall back to applying the raw batch — coalescing is an optimization,
  never load-bearing.

Tests: coalesce-failure fallback + dead-writer respawn.
2026-07-28 18:47:54 +05:30
Soju06 174ad45939 perf(state): apply per-call token accounting on a background single-writer queue
Every API call in the tool loop persisted its token/cost delta by
calling SessionDB.update_token_counts() synchronously on the turn
thread — a BEGIN IMMEDIATE sessions UPDATE plus a session_model_usage
upsert, measured in production at p50 3.3ms / p95 70.4ms per call and
up to 299ms against a cold multi-GB state.db. The tool loop stalls for
that long between calls, N times per multi-tool turn.

SessionDB gains queue_token_counts(): same signature and semantics as
update_token_counts(), but the critical path is a deque append plus a
condvar notify. A lazily started daemon thread applies deltas in
enqueue order through the existing update_token_counts ->
_execute_write path, so the established self._lock / BEGIN IMMEDIATE /
jitter-retry discipline is unchanged. When a backlog forms, adjacent
same-route incremental deltas coalesce into one UPDATE: token and
api-call fields sum, cost fields sum None-preservingly (an all-None
run stays None so COALESCE keeps the stored value), and absolute=True
deltas never merge and act as ordering barriers. Route equality is
required for a merge because those fields feed COALESCE backfill, the
last-non-None-wins status fields, and the per-model usage attribution
key — a merged apply is row-equivalent to sequential applies.

Correctness and durability:

- flush_token_counts() gives read-your-writes to token/cost readers
  (get_session, list_sessions_rich, _get_session_rich_row,
  list_gateway_sessions, InsightsEngine.generate) — a plain attribute
  check when nothing is queued. The writer sets its busy flag before
  popping the queue so the lock-free fast path can never miss an
  in-flight batch.
- update_session_model / update_session_billing_route /
  update_session_meta write the sessions row synchronously, bypassing
  the queue, so they flush it first: a still-queued first-of-session
  delta carries the pre-switch route, and applying it after the switch
  UPDATE would trip the first_accounted_route branch (api_call_count
  == 0 plus a route mismatch) and resurrect the old model/provider.
- AIAgent._persist_session flushes at turn finalize and every
  error-exit persist point; close() stops and drains the writer before
  the WAL checkpoint; an atexit hook (registered on first enqueue,
  unregistered on close so closed instances are not pinned until
  interpreter exit) drains at shutdown. Worst-case crash loss is the
  in-flight call's delta — the same window as the old inline write.
- A flush trusts a live stop-flagged writer (its loop drains before
  exiting) and only drains on the caller's thread when the writer is
  dead or never started, claiming the same busy flag so concurrent
  flushes wait instead of racing an in-flight batch.
- After close() has stopped the writer, queue_token_counts applies the
  delta inline instead of parking it on a queue nothing will drain; a
  closed-connection failure then raises at the call site, which
  already guards for it, exactly like the old synchronous path.
- Writer apply failures are logged and never raise into a turn; the
  writer thread survives and keeps applying.

Call sites switched to the queue: the per-call site in
agent/conversation_loop.py and both codex app-server sites in
agent/codex_runtime.py. In-memory per-turn counters
(agent.session_estimated_cost_usd etc.) stay synchronous, so live turn
displays never see the queue.

Tests: tests/agent/test_async_token_accounting.py (19 tests: enqueue
ordering, absolute-as-barrier, backlog coalescing with exact sums,
coalesced-vs-sequential row equivalence, merge unit rules, None-cost
preservation, read-your-writes, flush vs stop-flagged/concurrent
drains, inline apply after writer stop, close/atexit durability,
_persist_session drain, writer failure isolation);
tests/run_agent/test_token_persistence_non_cli.py updated to the
queue_token_counts contract.
2026-07-28 18:47:54 +05:30
Brooklyn Nicholson bf0871fbf8 fix(agent): type a synthesized user turn when its row is written
The auto-continue recovery note was typed only after run_conversation
returned, so its row sat untyped for the whole turn — and permanently
when the continuation was itself killed, which is the case it exists
for. persist_user_display_kind stamps the type on the live message
before the crash persist writes it, in the same insert as the content.
The flush also carries display_metadata through, which it was dropping.
2026-07-28 03:54:16 -05:00
Alex Fournier eda54775e2 fix(relay): isolate streaming callback contexts
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 21:11:04 -07:00
Alex Fournier 14bed44c8c Reapply "feat(observability): integrate NeMo Relay runtime and shared metrics"
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 21:10:51 -07:00
Brooklyn Nicholson 045811f5be fix(context): keep @ref tokens inline instead of stripping them from the prompt
Expanding an @file:/@folder: reference deleted the token from the message
it was typed in, leaving a hole in the sentence and no anchor for clients
to chip. The attached context block still names each ref, so nothing is
lost by leaving the token where the user put it.
2026-07-27 22:55:18 -05:00
Aaron Weiker 4587d77e0e fix(errors): log when the empty-stub guard and malformed-body classification fire
Add distinct, greppable log lines at both fix sites so the condition is
observable in the field instead of only inferable from the absence of the old
'Cannot compress further' spiral:

- chat_completion_helpers: warn when an empty partial-stream stub is replaced
  with the placeholder (0 chars recovered, no tool call).
- error_classifier: warn when a malformed-body 400 is classified as
  format_error rather than context overflow, with num_messages/approx_tokens.

Extend the litellm-shape classifier test to assert the warning is emitted.
2026-07-27 20:27:56 -07:00
Aaron Weiker 207a6c969d fix(errors): stop empty-content stream stubs from poisoning the transcript
A stream that dies after delivering deltas but with 0 recovered chars (and
no captured tool call) produced a content-less assistant stub. That empty
non-final message is invalid for the Anthropic schema, so every later request
400s with 'all messages must have non-empty content' (INVALID_REQUEST_BODY).
On a large session the 400 was misclassified as context overflow, dropping the
loop into a compression spiral that ends in 'Cannot compress further' — a
misleading context-size error on a session nowhere near its limit.

Root cause: substitute a minimal '[response interrupted]' placeholder so the
stub is never empty.

Misclassification: add _INVALID_MESSAGE_BODY_PATTERNS (checked before the
context-overflow heuristic) to classify these as non-retryable format_error,
and teach the body extractors the litellm/Bedrock errorMessage/errorCode/
errorArgs shape so descriptive proxy errors are not mistaken for generic ones.

Adds RED-verified regression tests for the stub path and three classifier tests
(including a guard that real overflows still compress).
2026-07-27 20:27:56 -07:00