- Bound ALL reads of the on-disk JWT store through one _read_jwt_store()
helper (load, eviction, save-merge) — the 1 MiB cap previously only
covered the load path; eviction and save could still parse an
oversized/corrupt store and rewrite it back out (sweeper finding).
- Fix the class, not the site: the recovery gates checked the literal
provider == "copilot" while /model and profile configs can leave the
alias spelling in place (the reporter's own log shows provider=copilot
AND provider=github-copilot in one session — the aliased turns would
have silently skipped recovery). Single owner:
AIAgent._is_copilot_provider() (slug aliases + Copilot base-URL
fallback), used by both run_agent recovery methods and both
conversation_loop gates.
- Update the salvaged 401 test to current main's client-retirement
contract (release deferred to GC — no synchronous .close()).
- Add copilot_stale_cred_retry_attempted to the TurnRetryState field
contract test; add bounded-store and alias-gate regression tests.
Narrow the moa_aggregator Relay bypass to CodexAuxiliaryClient and add
coverage that call_llm(stream=True) returns the provider's direct
create() result for Responses-shim clients.
Salvaged from PR #74903 by @liusencomic-cyber.
Under managed Relay execution the provider factory runs lazily inside
provider_stream() on the Relay session's event loop. The MoA facade's
auxiliary call_llm(stream=True) is invoked from that callback, so the
eager final_response check added for the non-managed path never fires:
the inner ManagedLlmStream is returned to the outer stream, which then
synchronously iterates it on the same loop thread and dies with
RuntimeError: Cannot run the event loop while another loop is running
(the completed response effectively trapped one level deeper).
stream_current() now detects a running event loop and returns the raw
factory result instead of nesting a ManagedLlmStream: the outer managed
stream already provides Relay tracking for the enclosing attempt, and
its own completed_response_predicate traps the completed response as
final_response — the same contract the main streaming worker consumes
(chat_completion_helpers reads stream.final_response after the chunk
loop). Nested managed streams remain supported for genuinely streaming
providers via the outer stream's own iteration.
Adds managed-execution regressions using the retained relay_turn
fixture: direct completed-response trapping, and the nested
facade-shaped stream_current call.
Port from openclaw/openclaw#112954. The redactor knew GitHub, Slack,
Google, Stripe, AWS access-key-ID and ~25 other vendor prefixes but had
zero GitLab coverage — glpat-/gloas-/gldt-/glrt-/glrtr-/glcbt-/glptt-/
glft-/glimt-/glagent-/glsoat-/glffct-/glwt- tokens and legacy GR1348941
runner registration tokens passed through display and log surfaces
verbatim. Follow-up explicitly invited when #4541 was closed.
Each pattern keeps a full literal prefix so the _PREFIX_SUBSTRINGS
pre-screen (derived at module load) stays false-negative-free; routable
runner tokens allow dotted segments. Sibling site: skills_guard's
credential-exposure scan gains a gitlab_token_leaked pattern.
Phase 2 review findings on the salvage branch:
C1 (critical): batch and micro summary markers share
COMPRESSED_SUMMARY_METADATA_KEY, and compress() never reset micro state.
After micro absorbed exchanges 1..k, a batch compaction summarizing
1..m (m>k) could fire; the next micro pass's supersede then dropped the
batch marker (whose content the stale rolling summary does NOT contain)
and archive_and_compact immediately made the loss durable. Defrag had
the same hazard: it rewrote "the newest marker" even if that was a
batch marker. Empirically confirmed with a probe (batch marker content
destroyed in one pass).
Fix, three parts:
- Micro-created markers now carry MICRO_COMPACT_MARKER_KEY; supersede
and defrag only ever touch micro-tagged markers. Rehydration in
_resolve_compact_cursor tags the marker it absorbs (containment
proof), which safely covers adopting a batch marker as the new
rolling base after a reset.
- compress() success path resets micro rolling summary/cursor state so
a stale summary can never claim cumulativeness over a batch marker.
- Regression tests for both directions plus the reset.
W4: _splice_micro_compact_result no longer strips _db_persisted stamps
from surviving messages. Micro archives in place under the SAME session
id (unlike batch's child-session rotation, #57491), so surviving stamps
are accurate; stripping them meant an archive_and_compact failure left
every previously-persisted message unstamped and the next append-only
flush re-inserted them all as duplicate active rows.
W5: finalize_turn micro gate now checks agent._persist_disabled —
persistence-isolated fork agents (background review) must not burn an
aux call per review turn, and must never archive_and_compact the
canonical session rows if their compressor ever gains a DB binding.
W1: _serialize_one_exchange now delegates to _serialize_for_summary
(was a ~70-line near-verbatim copy; one serializer, one place to fix).
S4: _find_one_exchange boundary guard rejects only assistant/tool
boundaries (the actual alternation hazard) instead of requiring user —
a stray mid-list system/injected message can no longer wedge the
cursor forever.
5 new regression tests; 38 micro/prune tests, 400 compression-suite
tests, 61 finalize/persist tests pass; ruff clean.
Two integration bugs found during review of #74522, both confirmed with
empirical probes against the production message-repair path:
1. Alternation: the summary marker was role="user" and an exchange was a
single assistant+tools group, so splicing between two user turns produced
user -> marker(user) -> user. The pre-request repair_message_sequence pass
(conversation_loop.py, runs before EVERY API call) then merged the marker
into the neighbouring real user message: metadata gone, cursor
unrecoverable on resume, and the summary text duplicated into the
transcript on every later pass (the transcript GREW every turn).
Fix: an exchange is now a full agent turn (assistant + tools + follow-up
assistant iterations, bounded by user messages), the marker is
assistant-role, and superseding an old marker deliberately merges the two
adjacent real user turns (plain-text \n\n-join, identical to repair
pass 2) so the returned transcript is alternation-valid by construction.
Probe result: repairs 0 (was 2), marker survives, no summary leakage.
2. Defrag destroyed user messages: _defrag_rolling_summary serialized the
whole remaining middle (user turns included) and spliced it away —
8 of 10 user prompts destroyed in one pass, contradicting the feature's
"your messages are never compacted" invariant. Fix: defrag now
re-summarizes only the rolling summary TEXT and rewrites the marker
content in place; transcript shape, cursor, and user turns untouched.
Probe result: 10 of 10 user prompts survive.
Also: marker provenance is now COMPRESSED_SUMMARY_HAS_USER_TURN_KEY=False —
micro markers absorb only assistant/tool content (#64650 invariant), and
real user turns remain in the transcript for provenance detection.
Adds 5 regression tests (repair-pass integration, alternation on
multi-iteration tool turns, defrag user survival, defrag input scope,
marker provenance); updates the two existing tests and the design doc to
the corrected semantics. 28 tests pass.
The on/off switch was the only knob. A pass fired after every completed turn,
absorbed exactly one exchange, and there was no way to ask for less. Since a
pass is also what breaks the prompt-cache prefix, "how often does it run" and
"how often do I pay a cache break" are the same question, and it had no answer.
Add `compression.micro_compact_every_n_turns` (default 1, clamped to >= 1). At 1
the behaviour is what it was; at 5 you get a fifth of the breaks and a fifth of
the reclaim rate. The counter advances per invocation rather than per committed
pass, so a turn that finds nothing to absorb still moves the cadence along and
cannot wedge it, and a bogus 0 or negative degrades to "every turn" instead of
silently disabling compaction.
Also expose `micro_compact_defrag_threshold_tokens`, which has been a hardcoded
attribute on the compressor with no path from config since it was added.
This does not give micro-compaction the prune's reclaim-size gate -- a pass
still commits whatever the single absorbed exchange saved. It makes the break
frequency tunable, which reaches the same end by absorbing less rather than by
waiting for a bigger win. The docs now say that plainly, including that a
reclaim threshold is the obvious follow-up and does not exist yet.
Tests cover the skip-until-due window, the cursor and prefix staying untouched
on skipped turns, the clamp, and that the feature is off unless enabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rolling summary lives only in memory. A resumed session starts with an
empty one while the marker carrying every previously absorbed exchange is
still in the transcript. The first pass after a resume therefore built a
marker from a single exchange and superseded the marker holding the entire
history -- silently discarding everything micro-compaction had accumulated.
This was introduced by the supersede fix. Before it, markers piled up
wastefully, but nothing was ever lost.
Two changes, so a single failure cannot lose data:
Rehydrate. When the cursor is recovered by scanning the transcript -- the
resume path -- also recover the rolling summary from that marker, so the
next pass merges into the existing history instead of replacing it.
Extraction uses rfind for the heading because SUMMARY_PREFIX references the
heading text itself, so the first occurrence is inside the preamble.
Gate superseding. Earlier markers are dropped only when this pass's summary
is demonstrably cumulative, i.e. the rolling summary was non-empty going in.
If rehydration ever fails, the pass keeps both markers: wasteful, but the
history survives.
Tests cover the resume path, the failed-rehydration fallback, and the
round trip of a summary through a marker.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cursor was set to the pre-splice `exchange_end`. A splice collapses the
absorbed span -- an assistant plus its tool results, often four or more
messages -- into a single marker, and may also drop a superseded marker
further back, so every index after it shifts.
The stale cursor therefore overshot, landing inside a *later* exchange's
tool group. The next pass's `_find_one_exchange` walked forward from there
to the following assistant, so the exchange it had landed inside was never
absorbed at all. On tool-bearing conversations micro-compaction was
silently doing roughly half the work it should.
Traced on a 3-tool-per-exchange transcript: the cursor sat at index 6 when
the marker was at 2, and the message count stalled at 32 instead of
continuing to 28.
Derive the cursor from the marker's actual position in the spliced result
instead, which is self-correcting regardless of how much the splice moved.
Apply it on the defrag path too, which had the same staleness.
Found by a randomized long-horizon harness (480 conversation shapes x 25
passes, varying tool-group sizes and summarizer failure modes) asserting
structural and progress invariants after every pass. Existing tests missed
it because their fixtures have no tool results, so the absorbed span is one
message and nothing shifts. The regression test uses tool groups.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tokens saved is the wrong headline for this feature. Micro-compaction is
not an efficiency optimisation — the same summarization work happens either
way. What it buys is (a) that work amortized across turns instead of one
stall, and (b) a window kept low enough that a session runs much further
before needing a hard compaction at all.
Neither shows up in "net tokens saved". A session can save nothing on paper
and still be a clear win on both counts.
So the telemetry now carries occupancy: tokens_after as a share of the
compaction threshold, plus the threshold and resolved window it was
computed from. That is the number that says whether a session has headroom
left. The report leads with it, and cross-references the batch
`compression_attempt` lines already in the log so it can show how often the
long pause actually fired — ideally never.
Occupancy is read from the cached threshold only. The public
`threshold_tokens` property resolves lazily and can issue a synchronous
/models probe (#32221); telemetry must never be the thing that blocks a
turn, so an unresolved window reports null. In practice a pass has already
resolved it via the tail calculation, so the field is populated. A test
pins the no-forcing behaviour directly against the emitter.
The report is pure ASCII: `scripts/check_subprocess_stdin.py` currently
dies on a cp1252 console before printing its results, and a diagnostic tool
that crashes on the platform it is diagnosing is worse than no tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The existing log line reports message counts, which is the least
informative number available here: absorbing one tool-heavy exchange can
drop hundreds of tokens while moving the count by one. There was no way to
answer "is this actually helping?" from a real session.
Emit one content-free JSON line per pass, in the same shape as the batch
compaction telemetry: before/after tokens, the delta, the size of the
absorbed exchange, the rolling summary size, duration, and running
per-session totals so a whole run can be read off the last line. No
transcript content rides along.
Add scripts/micro_compaction_report.py to aggregate those lines into
passes, outcome mix, net tokens saved, mean exchange size and durations,
with an optional per-session breakdown.
Measuring it immediately surfaced something worth documenting: the first
pass in a session normally *costs* tokens. The summary marker carries a
fixed ~400 tokens of scaffolding, paid on pass one against a single
absorbed exchange. From pass two the marker is replaced rather than added,
so the overhead is already paid and each exchange is close to pure saving.
Break-even is typically the second or third pass. Tests cover the
telemetry contract, the cumulative totals, and that first-pass/later-pass
shape so nobody reads a single turn and concludes it made things worse.
The estimator costs ~5 ms at 600 messages and ~20 ms at 1200, taken twice
per pass, post-turn — and only once an exchange is actually in hand, so
turns that no-op early pay nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`_find_one_exchange`'s docstring described an exchange as "(optional) user
message + assistant message + its tool results", but the walk skips past
user messages and starts at the assistant, so user turns are never absorbed
into the rolling summary.
The code is right and the docstring was wrong. Assistant output is largely
an account of what was done and survives summarising with little loss. The
user's messages are the intent everything else is derived from and cannot be
reconstructed from the work that followed — paraphrasing "use the existing
helper, don't add a new one" into a summary is how an agent ends up doing
the opposite six turns later. They are also cheap: a prompt is normally a
tiny fraction of what one tool result costs.
Correct the docstring, document the property (and its cost — a floor on how
small the middle can get, since user turns accumulate), and add a test so it
stays deliberate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Batch compaction pauses a session for one large summarization once the
window fills. Micro-compaction spreads that cost out: after each completed
turn, `finalize_turn` folds the single oldest un-absorbed exchange
(assistant message plus its tool results) into a rolling summary, so the
work happens in small increments during post-turn idle time instead of one
long stall.
Mechanics:
- a cursor tracks the first message not yet absorbed, recovered from the
transcript's last summary marker when in-memory state is unavailable;
- protected head and tail windows are never touched, so the system prompt
and recent turns stay verbatim;
- the absorbed span is replaced by a marker carrying the usual
`_compressed_summary` metadata, so resume, handoff and `/compress`
treat it exactly like a batch summary;
- `archive_and_compact` keeps the session DB in step, otherwise the
append-only flush would leave the original rows active and a resume
would double-load both summary and originals;
- when the rolling summary itself passes a token threshold it is
defragged: re-summarized in one shot and the cursor jumps to the tail;
- an exchange the summarizer can't handle is retried a bounded number of
times, then skipped, so one poison exchange can't stall every turn.
Keep only the newest summary marker. The rolling summary is cumulative, so
each marker already contains everything the previous ones held; leaving them
stacked near-duplicate copies of the same text, each with its own heading and
end-marker scaffolding, and the transcript grew on every turn instead of
shrinking. Measured over six turns on a 12-exchange conversation with tool
output: 4104 -> 4797 tokens before, 4104 -> 2572 after.
Off switch: `compression.micro_compact: false` (default on).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The compaction summary's role was selected against the LITERAL
neighbouring messages (compressed[-1] / tail_messages[0]). Mistral-family
chat templates (Devstral, Mistral Small 3.x, Magistral) enforce
user/assistant alternation but exempt the tool flow (tool results and
assistant messages carrying tool_calls) from the check, so a protected
head ending [user, assistant(tool_calls), tool] pinned the summary to
role="user" while the last role the template counts is "user": the
backend rejects the whole request with a Jinja alternation error
(HTTP 500). The summary persists in the stored conversation, every
retry replays the identical poisoned history, and the session is
permanently unrecoverable. Fires on EVERY compaction against a
Mistral-strict backend, captured byte-exact via a tee-proxy in front of
a llama.cpp/llama-swap Devstral deployment.
Fix: compute both neighbour roles through _template_visible_role(),
which skips template-exempt messages. The #52160 (Anthropic user-first)
and #58753 (zero-user-turn) forced-user guards are preserved; their
forced shapes (summary-user followed only by exempt messages) are
alternation-safe. When the visible head ends "assistant" and the
visible tail opens "user", no standalone role can alternate and the
existing merge-into-tail fallback now correctly fires (the literal
logic emitted a standalone user summary there: a second poisoning
shape).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGN45sMMbwM8cW9T9ga4ou
Replace OS-scheduler-dependent elapsed-time ceilings with deterministic
concurrency proofs in three tests:
- test_context_refs_concurrent: asyncio.Barrier rendezvous — all 3 URL
fetches must be in flight simultaneously before any returns.
- test_memory_boundary_commit: positive non-blocking witness — the
provider call list must still be empty when the async commit returns.
- test_mem0_v3 slow-prefetch: threading.Event park/release — prefetch
must return while the backend search is still parked.
The tests/tools/test_mcp_tool.py hunk from the original PR is dropped:
main no longer carries the 'elapsed < 2.5' assertion it targeted
(superseded by a delay-relative bound).
Salvaged from #71913.
Re-port of PR #35464 onto the rewritten hermetic conftest:
- autouse _neutralize_webbrowser fixture records open/open_new/open_new_tab
and webbrowser.get() instead of launching a real browser
- autouse _neutralize_macos_keychain_creds defaults the Anthropic Keychain
reader to None, with an opt-in allow_macos_keychain marker
- regression tests in tests/test_hermetic_side_effect_guards.py
- tests/agent/test_anthropic_keychain.py opts in via pytestmark
Salvaged-from: #35464
Co-authored-by: y0shualee <yuxiangl490@gmail.com>
The restart-routing, systemd-support, and subprocess-HOME tests asserted
branch behavior but left part of the real probe surface unmocked, so they
fail when the suite itself runs inside a container (self-hosted CI) or a
launchd-descended shell:
- /restart routing tests: the handler also consults the real /.dockerenv —
extract the inline probe to gateway.restart.is_container_restart_context()
(patchable seam, no behavior change) and pin it False; scrub ALL four
supervisor env markers (ambient XPC_SERVICE_NAME on macOS flipped one).
- supports_systemd_services tests: pin shutil.which('systemctl') and
is_container() so the test asserts the branch, not the host.
- copilot ACP real-HOME test: pin is_container() (auto mode prefers profile
home in containers) and scrub ambient HERMES_REAL_HOME/TERMINAL_HOME_MODE.
97 tests green on macOS dev box AND inside a docker CI runner container.
Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
_fresh_modules() in test_vision_routing_31179.py deleted agent.image_routing
(+siblings) from sys.modules and never restored the ORIGINAL modules — the
reloaded copies leaked. Any later same-process test holding function refs to
the original module (test_image_routing.py) then patched the reloaded copy
via mock.patch string targets, making patches invisible: order-dependent
failures (repro: 31179 file first → 2 failures; reversed → green).
Autouse fixture now snapshots the affected sys.modules entries and restores
them on teardown. Both orders + solo runs verified green.
Masked by the per-file-isolated canonical runner; bites anyone running
pytest tests/agent/ directly.
Main's 243c9182b1/a16fd675df/7142dc4580 added load_config_readonly
sibling stubs across 38 files; our pruned versions of 11 of those files
kept only the load_config stubs. Re-applied the pairing at every
surviving site (26 patch()/setattr sites) — same return_value/
side_effect as the adjacent load_config stub. 494 tests green across
the 11 files.
Second stub shape the first sweep missed: monkeypatch.setattr(config_mod,
"load_config", ...) — attribute-form instead of string-form. Four files
(compression_max_attempts, preflight_compression_cap_e2e,
codex_gpt55_autoraise_notice, proactive_prune_config) stub agent_init
config reads that now go through load_config_readonly(). Swept the whole
tree for the attribute form; remaining hits stub modules that still use
the mutable loader. 1,210 tests green across the 19-file re-sweep.
The readonly swaps mean agent/ modules now read config through
load_config_readonly(); tests that stubbed only load_config stopped
intercepting those reads. Added sibling readonly stubs (same
return_value/side_effect/lambda) at every affected site across 11 test
files, found via a tree-wide sweep of load_config stubs cross-referenced
against the swapped modules. 3,915 tests green across the 38-file
sibling sweep.
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
Resolves the PR's conflict with main (2252 commits). Two conflicts, both
"each side added an independent block in the same place" — kept both:
- gateway/run.py — the housekeeping loop. This branch adds the Skill Sync
pulls inside the CURATOR_EVERY branch (12-space indent); main adds a
stale-session auto-archive as a sibling `if` at loop level (8-space).
Different scopes, so the naive union would have mis-nested the archive
block into the curator branch; kept each at its own indent level.
- tools/skill_manager_tool.py — the _edit_skill result dict. This branch
appends the org auto-propose note; main appends
_add_description_prompt_preview(). Independent, order-insensitive.
No behaviour dropped from either side.
Verified: 3552 passed / 0 failed across 63 suites (scope regenerated to
include main's new maybe_auto_archive / _add_description_prompt_preview
consumers) via scripts/run_tests.sh. `hermes sync` and `hermes sync status`
still work against a live token, resolving the production plane default.
The Pyright Optional-parameter warnings in skill_manager_tool.py are
pre-existing on main (`content: str = None` etc.), not introduced here.
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
Three provably-safe optimizations for O(n)-per-iteration history walks:
1. sanitize_tool_call_arguments: optional identity-keyed cursor (strong
refs to the exact validated message objects) skips re-json.loads-ing
already-validated history each loop iteration. Any list rewrite
(compression, repair, undo, steer) breaks the identity prefix match
and forces re-scan from the divergence point. Wired via a per-agent
cursor dict in conversation_loop.
2. estimate_messages_tokens_rough: per-message memo keyed on a deep
identity fingerprint (strings pinned by strong reference so id()
aliasing is impossible; scalars by value; dicts/lists structurally
with key order). Equal fingerprints imply identical str(shadow)
bytes, hence identical estimates. Unfingerprintable shapes fall
through to direct compute. Bounded FIFO cache (4096 entries).
3. _flush_messages_to_session_db_unlocked: bounded scan that skips the
identity-matched prefix of the previous successful flush's snapshot.
Snapshot only taken on full success; cleared on exception. Compression
rewrites use fresh copies, breaking identity and forcing full re-scan.
Parity proven in tests/agent/test_cursor_optimizations_parity.py:
500-message synthetic histories with tool calls, malformed args, unicode,
element-wise old==new across 3 iterations incl. simulated compression.
Measured (median of 5): sanitize 0.097ms->0.011ms, tokens 1.145ms->0.853ms,
persist-scan 179.5us->10.0us at 500 messages.
The TTL-expiry test aged only the in-process probe cache; with the new
disk L2 the fresh disk verdict (correctly) served 'ollama' and the swap
to lm-studio wasn't re-detected. In real time-flow the 300s disk TTL
always lapses before the 1h in-proc TTL — the test now compresses both
expiries, matching the scenario it describes. 25/25 pass.
Local-model users paid a fresh probe waterfall on EVERY CLI cold start
inside AIAgent.__init__: detect_local_server_type (up to 4 HTTP GETs,
2s timeout each on a hung server) + /api/show (3s timeout). The
existing caches were in-process only, so back-to-back invocations
(chat -q, cron ticks, subagents) re-paid the network every time.
- New 300s-TTL disk L2 at HERMES_HOME/cache/local_endpoint_probes.json
for detect_local_server_type verdicts and query_ollama_num_ctx
results. Only SUCCESSFUL probes persist (a down server never pins a
negative verdict); stale entries pruned on write; corrupted cache
degrades to a miss; atomic writes. 300s is strictly fresher than the
1h in-process TTL that already accepts server-swap staleness.
- models.dev fetch timeout 15 -> (5, 10) connect/read tuple: a
blackholed connect stalled the first-turn critical path 15s; now
fails in 5s (matches the OpenRouter fetch convention, #46620).
- _auto_detect_local_model timeout 5 -> (2, 3): runs inside
_get_model_config() at startup against a LOCAL endpoint; a hung local
server cost 5s before the banner.
E2E (real HTTP server, two fresh subprocesses, isolated HERMES_HOME):
proc1 = 2 HTTP hits, proc2 = 0 HTTP hits, identical results
(ollama/131072), probe wall 74.5 -> 35.5 ms. 222 targeted tests green
incl. 9 new disk-L2 contract tests.
The streaming hot loop computed len(repr(chunk)) on EVERY chunk to feed
the retry-diagnostic byte counter — a full recursive pydantic repr at
5.5-8.8 us per chunk (measured), ~20-30 ms of pure CPU per 3,000-chunk
response, paid on every streaming response on every platform.
New _estimate_chunk_bytes() sizes the chunk from its delta payload
strings (content / reasoning / tool-call arguments) plus a 40-byte
framing floor: 2.1-2.4 us per chunk (~3x cheaper), independent of
pydantic field count, never raises on unknown shapes (Anthropic events,
stub providers fall back to the floor). Both call sites switched
(chat-completions loop + anthropic event loop).
The counter feeds only the stream-retry diagnostic log line
(agent/stream_diag.py) — an estimate proportional to traffic preserves
its purpose (distinguishing 'died at 0 bytes' from 'died mid-stream').
6 new contract tests; 64 targeted stream tests green.
- extract _commit_registry/_note_refresh_failure shared by the background
worker and foreground stage-4 (identical 4-step success + failure paths
were duplicated); worker now commits under _models_dev_fetch_lock so a
failing background refresh can never re-arm the backoff immediately
after a successful force_refresh committed (unsynchronized-write race)
- add should_clear_context_pin_async to hermes_cli/route_identity.py
(matching the get_model_context_length_async precedent) and use it at
the 4 async gateway sites instead of inline asyncio.to_thread wraps;
the sync _format_session_info site keeps the sync call (already
off-loop via its callers' to_thread)
- test the background-refresh success path (the PR's primary new
behavior): disk saved, mem cache swapped, backoff cleared, in_flight
reset — mutation-checked
- replace the race-prone spin-wait on _models_dev_refresh_in_flight with
a named-thread join in the backoff test
The branched call shape in get_provider_info/get_provider is deliberate:
~69 test sites across tests/hermes_cli and tests/gateway monkeypatch
fetch_models_dev (and get_provider_info) with zero/single-arg lambdas.
Passing allow_network= unconditionally broke 5 tests in CI slices 2/3/7.
Documented the constraint inline.
- _mark_stale_cache_grace only moves cache_time forward so a completed
background refresh is never rewound to a 5-minute grace window
- clear _models_dev_refresh_in_flight if Thread.start() raises so a
one-off thread-exhaustion failure doesn't disable refresh forever
- move empty-registry validation into _fetch_models_dev_from_network
(was duplicated in the background worker and the foreground fetch)
- pass allow_network through as a plain kwarg in get_provider_info and
hermes_cli.providers.get_provider instead of the branched call shape
- refresh the stale module docstring (no bundled snapshot exists; the
resolution order now describes stale-serve + background refresh)
Follow-ups on top of @DonutsDelivery's salvaged reaper commit (#64141):
- create_from_config() now parses lsp.idle_timeout (invalid values fall
back to DEFAULT_IDLE_TIMEOUT) — previously the constructor knob was
unreachable from config.yaml (config exposure adapted from #36892 by
@0xbWy and #68091 by @9miya20)
- canonical default declared in hermes_cli DEFAULT_CONFIG so config
discovery surfaces the knob (per sweeper review note on #64980)
- reaper loop survives transient sweep errors instead of dying and
silently re-opening the leak (gap flagged in #68091 review)
- eventlog.log_reaped(): one INFO line per sweep + clears the
log_active announce cache so respawns re-announce at INFO
- docs: replace the stale 'no idle-timeout reaper' paragraph with the
new lifecycle description + config reference
- tests: reuse-refresh protection (the regression teknium's sweeper
requested on #64141), reaper-survives-error, config propagation,
invalid-value fallback, DEFAULT_CONFIG/manager-constant sync
gemini-2.5-flash shuts down Oct 16 2026 (Google deprecation schedule)
and gemini-3-flash-preview is superseded. Update every hardcoded
default to the current GA flash model:
- gemini_native_adapter: probe_gemini_tier + _create_chat_completion
default params; free-tier guidance de-pinned from a specific model's
RPD number so it doesn't stale again
- auxiliary_client: gemini/kilocode fallback aux models,
_OPENROUTER_MODEL, _NOUS_MODEL -> google/gemini-3.6-flash
(verified live on both OpenRouter and Nous portal /models)
- provider plugins: kilocode + vertex default_aux_model
- hindsight memory plugin: gemini provider default
- setup wizard gemini list: 3-flash-preview -> 3.6-flash (matches the
curated picker catalog)
- tests: aux-client assertions that pinned the old default literal now
reference the _NOUS_MODEL constant, so the next default bump can't
break them (change-detector cleanup)
Fixes#32360.
Encapsulates the feature behind one command for launch, and adopts the
official product name.
One command:
- `propose` moves from `hermes skills propose` to `hermes sync propose`, so
the whole feature is one command to learn and one to document. Its handler
moves from cmd_skills to cmd_sync accordingly.
- The `hermes sync` parser now documents both halves plainly: personal sync
across your devices, and sharing with your organisation. Added an examples
epilog; rewrote the verb help in user language ("Include a skill in your
sync" rather than "Opt a skill into sync").
- Every user-facing string that pointed at `hermes skills propose` now points
at `hermes sync propose` (8 sites, including the agent-visible guidance
returned by skill_manage and the org provenance header).
This also clears the way for #39343, which adds its own top-level `sync` for
git-repo profile backup — that feature nests under `skills`, this one owns
`sync`.
Naming:
- HSP / "Hermes Sync Protocol" is gone from prose, docstrings, and comments.
The feature is "Skill Sync".
- Public identifiers renamed: HSPClient -> SyncClient, HSPError -> SyncError,
HSPConflict -> SyncConflict, hsp_address -> wire_address, HSP_VERSION ->
WIRE_VERSION.
- The WIRE names are deliberately NOT renamed: the `hsp_version` capability
field and the `x-hsp-object-type` response header are set by the deployed
gateway-gateway sync plane (verified in src/sync/syncRouter.ts), so
renaming them client-side would break sync against a live server. A comment
at the version constant records why they differ from the product name.
- The version-mismatch error is now actionable ("this server speaks sync
version X, but this Hermes speaks Y — update Hermes to sync with it")
instead of leaking the protocol acronym.
Also fixes a wiring gap found on the way: the gateway housekeeping tick
pulled personal skills but never org skills — the same defect already fixed
for the CLI. Org pull now runs there too, gated on real org membership.
Tests: the jargon guard now also fails on a bare "HSP". The two tests that
asserted the old cross-command structure are replaced by three asserting the
new one (propose IS under sync, propose is NOT under skills, sync usage
lists it). 2294 passed / 0 failed across all 51 suites that import the
changed modules, via scripts/run_tests.sh.
Verified by running the real CLI: `hermes sync --help` lists all eight verbs,
`hermes skills --help` no longer mentions propose, `hermes sync propose
--help` parses, and `hermes sync status` still reports live org state.
* Revert "fix(credits): remove the 'Grant spent · $X top-up left' notice"
This reverts commit 5dc6a14c14.
* fix(credits): reintroduce grant_spent behind an in-session crossing gate
The removed notice nagged because its condition is a steady state for
accounts living on top-up, the latch is per-session, and the cold-start seed
runs the policy at every session open — every session re-announced
'Grant spent · $X top-up left'. Reintroduce it gated so only a session that
WATCHES the grant run out announces:
- seen_grant_unspent crossing gate, mirroring seen_below_90: opens when the
session observes the grant meaningfully unspent (>= GRANT_UNSPENT_MIN_MICROS,
1 cent — portal-seeded states derive micros from float dollars and can carry
sub-cent residue where headers report exactly spent). Seeds never prime it.
- The gate guards only the show branch and is consumed by the announcement —
one announcement per crossing. Header flicker (used_fraction None and back)
clears the sticky line but cannot re-announce; a renewal re-opens the gate.
- new_credits_latch() centralizes the latch shape (agent build, lazy re-init,
and the policy test helper all build through it).
- Tests: steady-state-open stays silent (seed + policy seams), live crossing
announces once, flicker/renewal/residual cases locked; the rendering test
primes the gate and asserts its leg count so a gate regression cannot
silently shrink coverage.
The codex app-server namespaces MCP tool call ids as
codex_mcp__<server>__<tool>_<codex_call_id>. With an exec-<uuid> component the
built-in hermes-tools server alone overflows the Responses API's 64-char
call_id limit, so the request 400s with a non-retryable "string too long".
The offending item sits near the head of the transcript and replays every
turn, permanently bricking the session — the only recovery is /reset.
Sibling defect to #10788, which clamped input[*].id via
_MAX_RESPONSES_ITEM_ID_LENGTH. Apply the same treatment to call_id at both
Responses emit sites in _chat_messages_to_responses_input: a deterministic
surrogate (call_ + sha256[:32]) for ids over the limit, short ids unchanged.
Because the surrogate is a pure function of the original id, a function_call
and its matching function_call_output — which carry the same original id — map
to the same surrogate and stay paired without correlating the two items.
The grant_spent notice fired for every subscription user with top-up
funds the moment their cap was reached and camped in the CLI/TUI status
bar and desktop toasts with no action to take — the account keeps
working off top-up. Remove it everywhere:
- agent/credits_tracker.py: drop grant_cond + the emit/clear block;
the dev fixture state now (correctly) produces no notice
- TUI: keep the turn-start clear of credits.grant_spent as back-compat
for older backends that still emit the key
- Desktop: drop the demo step and stale comment references
- Docs/config comments: remove grant-spent from credits_notices text
- Tests updated: policy/cold-start now assert the key never fires
Usage bands, depleted, and restored notices are unchanged; /usage still
reports the full balance breakdown.
CI shard 4 caught 4 failures the local baseline diff missed (local env
pollution made them look pre-existing): _make_compressor() constructs
under a get_model_context_length mock but the lazy deferral (#32221)
pushed resolution past the with block, so the 96K window / 75% floor the
tests rely on never materialized. Resolve inside the mock.
Review-pass findings on the lazy-init deferral:
- No-op guard: the codex app-server usage callback assigns
compressor.context_length on EVERY response (same window each time).
The setter unconditionally invalidated the derived budgets, wiping
runtime corrections applied directly to threshold_tokens /
tail_token_budget (aux-context threshold sync) — those persisted on
main's eager init. Same-value assignment is now a no-op.
- Re-floor on genuinely new window: the setter invalidates budgets but
previously kept the stale threshold_percent, so a codex window switch
recomputed threshold_tokens from the new window with the old model's
floored percent. Re-apply the raise-only small-context floor from
_base_threshold_percent so percent and tokens derive from the same
window (guarded with getattr for object.__new__ test instances).
- Init log extracted to _emit_init_summary_once() and also fired from
the setter path, so a consumer assigning context_length before any
read no longer strands the startup line forever.
- threshold_tokens getter resolves the window into a local before
reading threshold_percent — correctness no longer depends on
left-to-right argument evaluation order.
- reasoning_timeouts: fix inaccurate 'tuples are immutable' comment
(the container is a list; safety comes from build-once-at-import),
document why the slug stays in the tuple.
Adds TestContextLengthSetterCoherence (3 tests): same-value assignment
preserves overrides; new-window assignment re-floors both directions.
Baseline diff vs clean upstream/main (182 pre-existing environmental
failures on both sides) showed exactly 6 PR-caused failures, all the same
mechanism: tests construct ContextCompressor under a
get_model_context_length mock and read threshold_percent/threshold_tokens
after the with block, or build via object.__new__ and assign
context_length AFTER the derived budgets (the setter now resets the
lazily-cached budgets).
- test_compression_small_ctx_threshold_floor: resolve inside _make()
- test_cjk_token_estimation: resolve inside mock
- test_per_model_compression_threshold: resolve inside mock (2 tests)
- test_pre_compress_memory_context: assign context_length before
threshold/tail/summary budgets; add summary_target_ratio
TestThresholdTokensCap and TestLazyContextResolution landed on main after
the #38991 lazy-init base; they construct ContextCompressor under a
get_model_context_length patch and read threshold_tokens after the with
block. With deferred resolution the probe now fires lazily, so resolve
inside the mock (same pattern as the rest of the suite) and give the
lazy-resolution mock a real return_value.
Self-review found the deepcopy->shallow-copy change in
apply_anthropic_cache_control (#57046) had no test pinning the
"prompt caching is sacred / never mutate the caller's list" invariant.
The old test_returns_deep_copy only exercised the single marked message,
never an un-marked shared reference, so a regression that deep-copied too
little would pass the whole suite.
Add two tests: (1) caller list + every element left byte-identical after
the call, un-marked middle messages returned as shared references, marked
messages fresh copies, and mutating a returned marked message does not leak
upstream; (2) structural byte-equivalence vs a reference full-deepcopy
implementation across both native_anthropic modes and two TTLs.
Mutation-verified: neutering the per-message deepcopy makes test (1) fail.
The lazy-init change in #32221 deferred get_model_context_length() out of
ContextCompressor.__init__, but the "Context compressor initialized" log
(emitted only when quiet_mode=False) reads self.context_length,
self.threshold_tokens and self.tail_token_budget. Those property reads
resolve the deferred value, so the synchronous model-metadata probe still
ran inside __init__ on the interactive-CLI path — the exact blocking the
PR removed, just narrowed to the non-quiet path.
Emit the informative line once, on first context-length resolution, so
construction stays non-blocking on every path. Add a regression test
asserting no probe fires in __init__ with quiet_mode=False and that the
init log is emitted exactly once on first access.
Every LLM call built a fresh openai.OpenAI wire client (new httpx pool,
TCP+TLS handshake, measured 19.2ms p50 / 35.5ms p95 per call at ~5 calls
per tool-loop turn) and closed it when the request finished. Cache ONE
reusable wire client on the agent, keyed by the effective client kwargs:
- _create_request_openai_client hands back the cached client when the
effective kwargs are identical; any change (credential rotation,
provider failover, vision default_headers) evicts and rebuilds.
- Only a request that produced a response reports a reuse close reason
(request_complete / stream_request_complete); error unwinds report
*_error_cleanup and really close, so a retry after a request error
always builds a fresh pool.
- Cross-thread aborts poison the slot: a pool whose sockets were
shutdown(SHUT_RDWR) from a stranger thread is never reused (#29507) —
the owner-thread close discards it and the next create rebuilds. The
holder read and the abort are atomic (under the holder lock) at all
three abort sites, so a late abort can never poison the NEXT request's
checked-out client.
- Worker-side interrupt breaks close the half-read SSE stream on the
owning thread before building the partial response; a failed close
poisons the slot (otherwise each interrupt leaked one checked-out
connection until the pool hit PoolTimeout). run_codex_stream gets the
same poison-on-close-failure handling.
- Single checked-out slot (in_use): a concurrent call gets an untracked
client with the old per-request lifecycle.
- release_clients() / close() really close the cached client when idle;
if a worker has it checked out they abort the sockets and detach the
slot, deferring the FD release to the worker's own close.
- MoA facade and Mock passthroughs never enter the cache; max_retries=0
is preserved on all request clients.
Follow-up to PR #64171. The _TOKEN_DELTA_* classification must exactly
cover update_token_counts' keyword surface: an unclassified kwarg is
silently kept only from the first delta of a merged run. Introspect the
live signature and fail with a pointed message when a future kwarg is
added without classification (or a classified field is removed).
Follow-up to PR #64171. The writer loop and flush_token_counts'
caller-drain both set _token_writer_busy BEFORE popping the queue —
that ordering is what makes flush's lock-free fast path (reads queue
then busy, no cond held) sound. _stop_token_writer's leftover drain did
it backwards (clear queue, then set busy), leaving a few-bytecode window
where a concurrent flush could observe 'empty and idle' and return True
with the popped batch still unapplied. Shutdown-only staleness, no data
loss — but the protocol now matches at all three drain sites.
Test: concurrent flush during a stop-drain mid-apply must time out
(False), never report drained.
Follow-up to PR #64171. Two writer-death hardening gaps:
- queue_token_counts only spawned the writer when the thread object was
None, so a writer that died from an unexpected exception could never be
replaced: deltas piled up on the uncapped deque until a reader's flush
drained them synchronously. Respawn on 'not thread.is_alive()' instead.
(atexit re-registration on respawn is safe: unregister removes all equal
bound-method registrations and the drain hook is idempotent.)
- _coalesce_token_deltas ran outside the per-delta try/except in
_apply_token_batch, so a merge bug (e.g. an unclassified future kwarg
summing None + 0) would escape and kill the writer thread. Wrap it and
fall back to applying the raw batch — coalescing is an optimization,
never load-bearing.
Tests: coalesce-failure fallback + dead-writer respawn.
Every API call in the tool loop persisted its token/cost delta by
calling SessionDB.update_token_counts() synchronously on the turn
thread — a BEGIN IMMEDIATE sessions UPDATE plus a session_model_usage
upsert, measured in production at p50 3.3ms / p95 70.4ms per call and
up to 299ms against a cold multi-GB state.db. The tool loop stalls for
that long between calls, N times per multi-tool turn.
SessionDB gains queue_token_counts(): same signature and semantics as
update_token_counts(), but the critical path is a deque append plus a
condvar notify. A lazily started daemon thread applies deltas in
enqueue order through the existing update_token_counts ->
_execute_write path, so the established self._lock / BEGIN IMMEDIATE /
jitter-retry discipline is unchanged. When a backlog forms, adjacent
same-route incremental deltas coalesce into one UPDATE: token and
api-call fields sum, cost fields sum None-preservingly (an all-None
run stays None so COALESCE keeps the stored value), and absolute=True
deltas never merge and act as ordering barriers. Route equality is
required for a merge because those fields feed COALESCE backfill, the
last-non-None-wins status fields, and the per-model usage attribution
key — a merged apply is row-equivalent to sequential applies.
Correctness and durability:
- flush_token_counts() gives read-your-writes to token/cost readers
(get_session, list_sessions_rich, _get_session_rich_row,
list_gateway_sessions, InsightsEngine.generate) — a plain attribute
check when nothing is queued. The writer sets its busy flag before
popping the queue so the lock-free fast path can never miss an
in-flight batch.
- update_session_model / update_session_billing_route /
update_session_meta write the sessions row synchronously, bypassing
the queue, so they flush it first: a still-queued first-of-session
delta carries the pre-switch route, and applying it after the switch
UPDATE would trip the first_accounted_route branch (api_call_count
== 0 plus a route mismatch) and resurrect the old model/provider.
- AIAgent._persist_session flushes at turn finalize and every
error-exit persist point; close() stops and drains the writer before
the WAL checkpoint; an atexit hook (registered on first enqueue,
unregistered on close so closed instances are not pinned until
interpreter exit) drains at shutdown. Worst-case crash loss is the
in-flight call's delta — the same window as the old inline write.
- A flush trusts a live stop-flagged writer (its loop drains before
exiting) and only drains on the caller's thread when the writer is
dead or never started, claiming the same busy flag so concurrent
flushes wait instead of racing an in-flight batch.
- After close() has stopped the writer, queue_token_counts applies the
delta inline instead of parking it on a queue nothing will drain; a
closed-connection failure then raises at the call site, which
already guards for it, exactly like the old synchronous path.
- Writer apply failures are logged and never raise into a turn; the
writer thread survives and keeps applying.
Call sites switched to the queue: the per-call site in
agent/conversation_loop.py and both codex app-server sites in
agent/codex_runtime.py. In-memory per-turn counters
(agent.session_estimated_cost_usd etc.) stay synchronous, so live turn
displays never see the queue.
Tests: tests/agent/test_async_token_accounting.py (19 tests: enqueue
ordering, absolute-as-barrier, backlog coalescing with exact sums,
coalesced-vs-sequential row equivalence, merge unit rules, None-cost
preservation, read-your-writes, flush vs stop-flagged/concurrent
drains, inline apply after writer stop, close/atexit durability,
_persist_session drain, writer failure isolation);
tests/run_agent/test_token_persistence_non_cli.py updated to the
queue_token_counts contract.
The auto-continue recovery note was typed only after run_conversation
returned, so its row sat untyped for the whole turn — and permanently
when the continuation was itself killed, which is the case it exists
for. persist_user_display_kind stamps the type on the live message
before the crash persist writes it, in the same insert as the content.
The flush also carries display_metadata through, which it was dropping.
Expanding an @file:/@folder: reference deleted the token from the message
it was typed in, leaving a hole in the sentence and no anchor for clients
to chip. The attached context block still names each ref, so nothing is
lost by leaving the token where the user put it.
Add distinct, greppable log lines at both fix sites so the condition is
observable in the field instead of only inferable from the absence of the old
'Cannot compress further' spiral:
- chat_completion_helpers: warn when an empty partial-stream stub is replaced
with the placeholder (0 chars recovered, no tool call).
- error_classifier: warn when a malformed-body 400 is classified as
format_error rather than context overflow, with num_messages/approx_tokens.
Extend the litellm-shape classifier test to assert the warning is emitted.
A stream that dies after delivering deltas but with 0 recovered chars (and
no captured tool call) produced a content-less assistant stub. That empty
non-final message is invalid for the Anthropic schema, so every later request
400s with 'all messages must have non-empty content' (INVALID_REQUEST_BODY).
On a large session the 400 was misclassified as context overflow, dropping the
loop into a compression spiral that ends in 'Cannot compress further' — a
misleading context-size error on a session nowhere near its limit.
Root cause: substitute a minimal '[response interrupted]' placeholder so the
stub is never empty.
Misclassification: add _INVALID_MESSAGE_BODY_PATTERNS (checked before the
context-overflow heuristic) to classify these as non-retryable format_error,
and teach the body extractors the litellm/Bedrock errorMessage/errorCode/
errorArgs shape so descriptive proxy errors are not mistaken for generic ones.
Adds RED-verified regression tests for the stub path and three classifier tests
(including a guard that real overflows still compress).
The read-only org mirror broke the learning loop precisely where it matters
most. The system prompt tells every agent to patch a skill the moment it
finds a gap, and shared skills are the ones the most people use — but every
write to _org/ was refused, and the curator was excluded from them outright.
So org skills froze while personal skills kept improving, and the offered
alternative ("fork it into a personal skill, then propose the fork") is not
something an agent does mid-task. The refusal WAS the feature; improvements
were simply lost, and manual forks would have fragmented the shared set.
Edit in place:
- skill_manage patch/edit/write_file now work on org skills. Only delete is
still refused (the mirror is a view of org HEAD — a local delete returns on
the next pull; removing a shared skill is an admin action).
- Org skills are curation-eligible again, so the curator can improve the
highest-leverage skills in the system instead of skipping them.
- The load-time provenance header now says edits are allowed and kept,
instead of instructing the agent not to edit.
Local edits are never overwritten:
- pull_org_skills previously rmtree'd each skill dir and re-materialized it,
silently destroying local work on the next session start. It now records a
content fingerprint per skill (.org-baseline.json) when it writes one, and
SKIPS any skill whose local content diverges from that baseline.
- When upstream ALSO changed such a skill, it is reported in the pull
result's "conflicted" list and left untouched for the user to resolve
deliberately (propose the local version, or delete it and re-pull to take
theirs). A missing baseline is treated as unmodified so pre-existing
mirrors do not raise phantom conflicts.
- Fingerprints are content-based (path + bytes, sorted), so a touch/mtime
change is not mistaken for an edit.
Sharing back:
- Default: the edit stays local and the tool result tells the user to run
"hermes skills propose <skill>".
- Opt-in sync.org_auto_propose / HERMES_SYNC_ORG_AUTO_PROPOSE submits each
edit immediately. Defaults OFF — pushing every agent edit to a whole
organisation is not a safe default. A failed submission never fails the
edit; the change is saved and can be proposed later.
- "hermes sync status" lists org skills with unshared local edits;
"hermes sync pull" reports conflicts it declined to overwrite.
Tests: 25 in the namespace suite (was 15). The two that asserted the old
read-only behaviour now assert the opposite. New coverage for edit-applied,
share-back guidance, delete-still-refused, curation-allowed, edit detection,
missing-baseline tolerance, mtime-insensitivity, and the auto-propose
default. 428 passed / 0 failed via scripts/run_tests.sh.
Verified through the REAL pull path against a mock plane: pull v1 -> edit in
place -> upstream ships v2 -> pull leaves the local edit intact, reports the
conflict, and surfaces it in status.
`hermes sync` gave no hint that org sharing exists, so there was no path
from 'I want to share this with my team' to `hermes skills propose` — sync
looked like the only sharing surface while being personal-only (it always
CAS-es refs/user/<owner>/HEAD).
- Bare `hermes sync` usage and the `--help` epilog now state that these
commands are personal-only and name `hermes skills propose <skill>` as
the org path, noting the approval step and that org skills arrive
automatically and are read-only locally.
- Scrubbed the remaining internal jargon from the sync module docstring
(M1-D, DEV-PHASE, HSP/1) missed by the earlier pass, which only covered
help= strings.
Tests: 2 new guards asserting both surfaces reference the org command.
373 passed / 0 failed via scripts/run_tests.sh. Both outputs verified by
running the actual commands.
_peel_moa_guidance hand-implemented the inverse of moa_loop's
_attach_reference_guidance from a different module — a drifting separator
or shape would make the peel silently no-op and put the last cache
breakpoint on the turn-varying guidance block (the #72626 bug class).
Move the inverse into moa_loop.peel_reference_guidance directly adjacent
to the attach, keep a thin wrapper in conversation_loop, and pin the
contract with a round-trip test over all three attach shapes.
Also fix the empty-list residue: peeling a guidance-only content part now
drops the whole message (mirroring the appended-user-message shape)
instead of leaving an empty-content user turn behind.
guidance=None is a real prepared shape (all references failed / silent
degraded policy builds prepared_request without attaching guidance), and
the MoA facade sends prepared['messages'] — not api_kwargs['messages'].
Gating the rebase on 'and guidance' left the stale decoration in the
prepared object for the no-guidance MoA sub-path, so #72626 persisted
there. rebase_prepared_request already handles falsy guidance (copies
messages, skips the attach).
The static-prefix reconstruction pattern (build_system_prompt_parts ->
['stable'] -> startswith gate -> fail-open) existed in three copies:
session restore (conversation_loop), compression keep-prompt path
(conversation_compression), and the new failover redecoration helper.
Hoist it into agent/system_prompt.reconstruct_static_prefix and call it
from all three sites.
Also memoize failed rebuilds per stored prompt (_static_rebuild_failed_for):
the redecoration chokepoint runs at the top of every retry attempt, and a
persistent stable-tier mismatch (restored session whose SOUL.md/skills
changed since save) would otherwise re-run the full prompt build — SOUL.md,
context files, memory I/O — on every attempt of every API call for the
life of the session. A legitimately changed stored prompt retries once.
strip_anthropic_cache_control flattened ANY pure-text multi-part content
list with a separator-less join. Decoration only ever produces a single
text part or the 2-part [static, volatile] system split; organic
multi-part text (merged user turns, imported transcripts) got word-jammed
and parts carrying extra keys (citations) were silently dropped — on the
common no-failover path, since redecoration runs on every attempt.
Restrict the flatten to the exact decoration-produced shapes and make
marker removal copy-on-write on part dicts (the per-call message copy is
shallow, so parts alias the persistent history).
Add strip_anthropic_cache_control coverage and policy-change cases
(cache-off→on, on→off, native→envelope, MoA guidance outside marker)
that TestSyncFailoverPreservesCacheDecoration did not exercise.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
claude-fable-5 is a Mythos-class reasoning model (1M context, 128K output,
adaptive thinking per anthropic_adapter.py) but was missing from the
_REASONING_STALE_TIMEOUT_FLOORS table. Without a floor entry it got the
default 180s stale timeout (300s with context scaling), which is too short
for fable-5's thinking phase on large contexts.
Each stale kill bumped the cross-turn circuit breaker streak; after 5
consecutive kills _check_stale_giveup() fired immediately (elapsed: 0.00s),
aborting all calls with "Provider has been unresponsive for 5 consecutive
stale attempts." Users with 191K-token contexts hit this reliably.
Add ("claude-fable", 600) — deep-reasoning tier alongside o1/deepseek-r1/
nemotron-3-ultra. The claude-fable slug matches claude-fable-5 and future
variants via the existing right-anchor regex.
Update _pre_msg_count after adopting the durable transcript so the
post-compression log reflects the correct pre-adoption message count.
Also use the existing _live_child_id() helper in the updated test
instead of hand-rolled child-id extraction.
Follow-up to #72631.
Busy sessions (memory review / shared session writers) kept outrunning the
in-memory snapshot, so rotation-mode compress aborted every attempt with
"changed before lease acquisition" and surfaced as a fake "No changes from
compression". Adopt the durable transcript and continue compressing instead
of returning the stale snapshot unchanged.
Every fallback/dedup/skip decision asks one question — 'is this candidate
the same backend as the one that failed, along the axis that failure
invalidated?' — but it was re-implemented inline at six sites across four
subsystems, each comparing whatever string was locally convenient. Each
incident fixed one site while the others kept the bug: #22548, #70893,
#59561, #72468, #62984/#54250/#57584.
agent/backend_identity.py now owns the concept: BackendIdentity (provider /
model / base_url axes), FailureScope (MODEL / CREDENTIAL / ENDPOINT — each
failure class invalidates a different axis), and should_skip_candidate().
Unknown axes never manufacture a skip (over-skipping strands failover; a
wrong try costs one RTT).
Migrated sites:
- chat_completion_helpers.try_activate_fallback: replaces the provider+model
early-exit (the #62984 bug: ignored base_url, stranding multi-endpoint
pools) AND _fallback_entry_is_same_backend_by_base_url (deleted)
- auxiliary_client._try_configured_fallback_chain +
_try_main_agent_model_fallback: replace label/model comparisons; auth and
payment map to CREDENTIAL scope, keeping the #59561 carve-out
- hermes_cli/fallback_cmd add: primary-match + duplicate checks now identity-
aware (#54250/#57584): same provider+model on a different explicit
base_url is a pool entry, not a duplicate
_mark_provider_unhealthy stays label-keyed deliberately: its only triggers
are confirmed 402s, which ARE credential-scoped.
Owner-level tests pin each incident's semantics by number; sabotage-verified
(removing the base_url axis fails the #62984 test).
Two defects found by manual testing on the branch.
1. ORG SYNC NEVER RAN. The org pull/mirror/gating machinery was fully
implemented and unit-tested but had ZERO runtime callers —
maybe_pull_org_skills() was referenced only inside a comment, and
`hermes sync` had no org path at all. Every code path fell through to
personal sync (refs/user/<sub>/), so org skills never loaded and the
feature looked like 'everything syncs to my personal org' even with a
valid org token. The unit tests could not catch this: they invoked the
functions directly, which is exactly the gap they left open.
- cli.py session startup now calls maybe_pull_org_skills() alongside the
personal maybe_pull_skills(), fail-quiet.
- Auto-pull is gated on real org membership: resolve_org_identity()
requires an org role on the token, only issued for multi-member orgs,
so a solo account never reaches the network.
- `hermes sync pull` refreshes the org mirror too (one pull, both
surfaces) and reports what it refreshed.
- `hermes sync status` exposes org_available/org_id/org_role/org_skills
plus a plain-language summary, so a user can tell whether the org
workflow applies instead of it being invisible.
2. INTERNAL JARGON LEAKED TO USERS. Help text and errors exposed internal
milestone/spec coordinates: 'Propose a skill ... (M2)', 'Personal skill
sync (HSP/1)', 'DEV-PHASE gate closed: your token lacks
tool_gateway_admin', 'contract §4.3', and an inert message describing our
internal personal-vs-multi-member design split. All rewritten in user
language. Feature-local comments/docstrings lost their internal
coordinates (§N, M1/M2, design.md, PR numbers) while keeping the
explanatory prose. Pre-existing issue references elsewhere in the tree
were deliberately left untouched.
Tests: 4 new guards, including two that assert the CALL SITES exist so the
org pull cannot silently become dead code again (verified failing when the
wiring is removed) and one that fails if user-facing help leaks jargon.
344 passed across the sync/skills/prompt suites.
Verified against live staging with a real org token: sync status reports
org_available=true, org_role=OWNER; sync pull performs the org refresh; the
.active_org marker is written with the org id from the token.
Follow-ups on the salvaged #63359:
- _finalize_child_results carries tool_call_history on subagent_stop
(the #62011/#72403 field landed after the PR branched; the shared
pipeline must emit it for both delegate_task and plugin-launched
children). Lifecycle test updated for the new payload field.
- The lifecycle executor uses DaemonThreadPoolExecutor — a wedged or
abandoned child must never block interpreter exit at atexit-join time
(same rationale as _run_single_child's timeout executor and the
async-delegation pool).
- delegate_task's batch path keeps live-transcript wiring while routing
child construction through the shared
_build_child_preserving_parent_tools helper.
Widen okalentiev's failed_model narrowing (#59561) to
_try_main_agent_model_fallback. The safety-net layer still skipped on a
provider-label match alone, so single-provider users whose aux compression
model and main model share one custom endpoint had ZERO fallbacks: the aux
model timing out exhausted the chain in one hop and compression aborted.
Real incident (0.19.0 debug dump): aux zai-org/glm-5.2 hung 324s and timed
out while main mindai/macaron-v1-venti on the SAME endpoint was serving
448K-token turns — the label-only skip discarded the one viable summarizer,
the session wedged over threshold, and the anti-thrash breaker tripped.
Same convention as the chain fix: model-specific failures (timeout,
connection, rate limit) pass failed_model so only the exact failed model is
skipped; provider-wide failures (auth 401 / payment 402) pass None and keep
the whole-provider skip. Both sync and async call_llm sites pass it.
Sabotage-verified: the new regression test fails on the provider-only skip.
Follow-up to the same-provider fallback fix: narrowing the configured-chain
skip to the exact failed model is only correct for model-specific failures.
Auth (401) and payment (402) errors are provider-wide — every model on the
provider shares the same broken credentials/account — so trying a sibling
model can't recover and merely burns another doomed request before the
aux task fails. Worse, returning that sibling client bypasses the
main-agent-model safety net that a provider-wide skip would have reached.
Only forward failed_model to _try_configured_fallback_chain for
model-specific failures (timeout, connection, rate limit, model-incompatible,
invalid response). Auth/payment keep failed_model=None (whole-provider skip),
preserving the pre-existing safety-net behaviour for credential/billing
failures.
Adds an integration test that a timeout forwards the failed model, and
updates the payment-error test to assert failed_model=None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_try_configured_fallback_chain skipped every fallback_chain entry whose
provider matched the one that just failed. A chain that intentionally lists
several models under the same provider (e.g. two more NVIDIA NIM models
after the primary NIM model times out) was therefore skipped wholesale,
falling straight through to the main-agent-model safety net instead of
trying the other configured models on that provider.
Add failed_model so the skip narrows to the exact (provider, model) pair
that failed. Callers that only know the provider (client-build failures,
where the whole provider is unreachable regardless of model) keep the old
provider-wide skip; the two runtime request-error call sites (call_llm,
async_call_llm) now pass the model that just failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Teknium: the caps should bound a single agent loop, not accumulate
over the whole session. Rename SessionCapConfig -> LoopCapConfig and the
config section session_caps -> loop_caps; move the counters into
reset_for_turn (invoked per turn via turn_context) so each turn starts
with a fresh budget; retune defaults 200 -> 50 (a single turn issuing 50
web searches / spawning 50 subagents is already pathological). Block
codes session_*_cap -> loop_*_cap and messages updated to drop the
/new-resets-the-budget guidance (irrelevant now that it resets per turn).
Tests flipped: the old persists-across-turn-resets assertion becomes
resets-each-turn.
Add per-session lifetime caps on web_search calls and subagent spawns
(defaults 200/200, matching Claude Code v2.1.212). Unlike the existing
per-turn tool-loop guardrails, these count over the whole session and
reset only when a fresh agent is built (/new, /clear). Hitting a cap
blocks the offending call and halts the turn cleanly.
- agent/tool_guardrails.py: SessionCapConfig + session counters on the
controller (in __init__, not reset_for_turn, so they persist across
turns). before_call() enforces caps first, independent of
hard_stop_enabled. delegate_task batches count each task.
- hermes_cli/config.py: tool_loop_guardrails.session_caps defaults.
- docs + tests (unit + E2E validated against a real AIAgent).
Port from nearai/ironclaw#6129: their sensitive-marker scrubber matched
markers as bare substrings, so tool results containing 'Secretary of the
Treasury' were scrubbed as 'secret' on replay, evicting legitimate content
and forcing the model into a re-fetch loop. Hermes' lowercase/dotted/YAML
config-key redaction patterns (_CFG_DOTTED_RE, _CFG_ANCHORED_RE,
_YAML_ASSIGN_RE) had the same false-positive class: their key classes allow
arbitrary alphanumeric affixes around the keyword, so ordinary document
text like 'Secretary: J.Smith', 'tokenizer: cl100k_base' (HF model cards),
and BibTeX 'author=Smith' got value-masked on the surfaces that run these
passes (browser snapshots, log lines, kanban summaries, CLI-echoed output).
Fix: post-match word-boundary validation of the keyword occurrence inside
the matched key. Boundaries: key edges, non-letters (_ - . digits),
camelCase transitions (clientSecret, secretKey, APIToken), plural 's'
(secrets:, tokens:). Concatenated real-world compounds keep matching via
explicit alternatives (authtoken, authkey, secretkey, accesstoken). ALL-CAPS
keys keep legacy embedded matching (MYTOKEN=...) — all-caps is almost never
prose, same rationale as _ENV_ASSIGN_RE. Same discipline the file already
applies to exact-match body/query keys (ported from ironclaw#2529) and the
deliberate 'auth' exclusion that keeps 'author:' from matching.
Port from Kilo-Org/kilocode#12162.
Google began rejecting unrestricted legacy 'Standard' Google Cloud API keys
on the Gemini API on June 19, 2026 (all Standard keys stop working in
September 2026). The rejection is a 401 whose message misleadingly tells the
user to supply an OAuth 2 access token. gemini_http_error() now appends
actionable guidance (check key type in AI Studio, mint a new Gemini API key,
temporary restriction bridge) on that narrow shape — matched via
google.rpc.ErrorInfo reason ACCESS_TOKEN_TYPE_UNSUPPORTED or the
'Expected OAuth 2 access token' signature. Plain invalid keys
(API_KEY_INVALID) keep their existing message.
Also fixes a latent sibling gap: _summarize_api_error() preferred re-extracting
the raw response body for errors carrying .response, which stripped adapter-
composed guidance (this one AND the existing free-tier 429 guidance) from the
user-facing summary. GeminiAPIError now surfaces its composed message.
Port from anomalyco/opencode#37848 (+ dev-branch twin #37840): expand
context-overflow patterns and guard against rate-limit messages that
mention tokens.
- 'Throttling error: Too many tokens, please wait before trying again.'
(AWS Bedrock / proxy shape) classified as context_overflow and routed a
healthy session into compression on every throttle. Added 'throttling'
to _RATE_LIMIT_PATTERNS, which the message-only path checks BEFORE the
overflow list.
- 'Input length N exceeds the maximum allowed input length of M tokens.'
(Together/Fireworks shape) fell through to unknown — no compression
recovery. Added 'maximum allowed input length' to overflow patterns.
- 'request_too_large' / 'Request exceeds the maximum size' (Anthropic 413
type re-wrapped without a status code by aggregators/proxies) fell
through to unknown. Added to _PAYLOAD_TOO_LARGE_PATTERNS.
All three shapes proven live on main before the fix; 265 classifier +
bedrock tests and 238 sibling rate-guard/compression tests pass.
`apply_anthropic_cache_control` runs once per call block, before the retry
loop, and splits the system prompt into `[static prefix, volatile tail]` text
blocks carrying the cache_control breakpoints.
A failover fires *inside* that retry loop, and `_sync_failover_system_message`
assigns a bare string over `api_messages[0]["content"]` to refresh the
`Model:`/`Provider:` identity lines. That drops the block list and both
breakpoints. `convert_messages_to_anthropic` only emits system cache blocks for
list content (`isinstance(content, list)`), so the retried request ships
`system` as one plain string: zero breakpoints, nothing written to cache, and
the whole system prompt re-billed at full write price. The next call misses
too, so a failover costs two full-price prompts instead of one write + one
read.
Measured with the real functions, same history, native Anthropic layout:
normal turn system=BLOCK LIST(2) system breakpoints=2
after failover system=plain string system breakpoints=0
`_sync_failover_system_message`'s own docstring notes this fires on "every
gateway turn, since fallback re-activates per message while the primary is
down" -- so a gateway running on a degraded primary pays it on every message.
Fix: rewrite the decorated blocks in place instead of flattening them.
`rewrite_prompt_model_identity` only touches the LAST `Model:`/`Provider:`
lines, and those live in the volatile tail, so the static prefix stays
byte-identical and its cache entry keeps matching -- the failover retry keeps
both breakpoints AND the warm prefix. Shapes we cannot safely patch fall back
to the existing plain-string assignment.
This is the same class the retry loop already guards against eight lines below
the gap, for reasoning fields:
# api_messages is built once, before this retry loop, while the primary
# provider is active. [...] so the fallback request isn't sent with stale,
# primary-shaped reasoning fields.
agent._reapply_reasoning_echo_for_provider(api_messages)
There was no equivalent for cache decoration.
`_apply_persist_user_message_override` replaces the anchored user message's
content with the clean transcript text. Its DB-write twin refuses to do that
for a compaction-merged row, and says why:
# Preflight compaction can re-anchor the override index at a message
# whose content was MERGED with the compaction summary
# (merge-summary-into-tail). Overwriting that with the clean gateway text
# would silently drop the summary from the durable transcript.
... and not msg.get(COMPRESSED_SUMMARY_METADATA_KEY)
The live mutation has no such guard, and `finalize_turn` runs it FIRST -- it
calls `_apply_persist_user_message_override(messages)` and only then
`_persist_session(...)`. So the row is already clobbered by the time the
DB-side guard looks at it, and the clobbered list is also what is returned as
the continuation history the next turn is built from.
The anchor lands on the merged row by design, not by accident:
`reanchor_current_turn_user_idx` prefers the last user row whose content
matches this turn's text and "fall[s] back to the last user message when no
exact match survives (merge-summary-into-tail rewrites the content ...)" --
after a merge the exact match cannot survive, so the fallback selects exactly
the merged row.
Effect: on the turn a compaction fires, the `[MERGED PRIOR CONTEXT]` summary --
the entire pre-compaction conversation -- is deleted from the history the
session continues from. It is silent, there is no error, and the archived
pre-compaction turns are already rotated away. The durable child row still has
the summary, so the live session and a later `/resume` now replay different
bytes for the same row.
Add the same guard to the live path. The paired timestamp override is
unrelated and still applies.
Both halves of this invariant are now covered:
`TestFlushCompressedSummaryOverrideGuard` already asserted it for the DB write;
the two new tests assert it for the in-memory path, plus a negative control
that an ordinary turn is still cleaned in place.
`apply_anthropic_cache_control` marks an assistant turn with non-empty text by
writing `cache_control` INTO `content` -- `_apply_cache_marker`'s list branch
puts it on the last content block, not at the top level.
`_convert_assistant_message`'s ordered-replay branch rebuilds the message from
`anthropic_content_blocks` and returns early. Its only cache sources are
`_relocated_replay_cache_control` (markers rescued from blocks the replay
sanitizer dropped) and the top-level `m["cache_control"]`; it never reads
`m["content"]`. So for an assistant turn that interleaves signed thinking with
a tool_use AND has preamble text, the breakpoint is dropped.
It is burned, not relocated: `_can_carry_marker` returns True for this message
(non-empty content), so the breakpoint budget already counted it. Hermes'
accounting believes the marker landed.
Measured with the real functions, native layout, same history, only
`anthropic_content_blocks` differing:
normal path 4 breakpoints assistant text block carries cache_control
replay path 3 breakpoints assistant carries NONE
Under the static-prefix layout only two conversation-tier breakpoints exist, so
this halves them. Nothing is logged and the request succeeds with identical
model output, which is why it survives: it recurs on every request for the life
of any Claude thinking+tools session, since the flag rides the message in
`agent.messages`.
#56195 fixed the complementary shape -- blank assistant content, where the
marker lands top-level -- and its regression test pins `"content": ""`. The
non-empty case is the one a Claude 4.5/4.6 tool turn normally produces (a
sentence of preamble before the call) and was never covered.
Harvest an in-`content` marker next to the existing top-level lookup and hand
it to the same `_apply_assistant_cache_control_to_last_cacheable_block` helper.
Extends the cherry-picked /context command (PR #52184) and prompt-size
attribution helpers (PR #66656) into one visual context view across
surfaces, and absorbs the per-component budget-visibility goal of the
/tokens proposal (PR #48470):
- agent/context_breakdown.py: pure renderers over the existing payload —
a 5x20 glyph block grid (1 cell ~= 1% of the model window), an
'Estimated usage by category' table with free space, and expanded
per-skill / per-toolset listings via compute_context_details(), which
reuses the prompt-size attribution mechanism (skills index-line bytes +
registry tool->toolset map) converted to the same chars/4 heuristic.
- cli.py: /context [all] renders grid + category table (+ expanded
listings) from the live agent and in-memory conversation history.
- gateway/slash_commands.py: /context appends the plain-text category
table (no grid — monospace not guaranteed on messaging platforms);
/context all adds the expanded listings. Fail-open: breakdown errors
never break the gauge.
- hermes_cli/commands.py: /context gains the 'all' subcommand; /version
demoted to /hermes version on Slack to keep the 50-slash cap.
- tests: renderer unit tests against synthetic payloads, registry test,
gateway /context + /context all + failure-degradation handler tests.
- docs: slash-commands reference + CLI guide entries.
Read-only and locally computed: no provider calls, no prompt-cache impact.
Co-authored-by: RemyFevry <29257684+RemyFevry@users.noreply.github.com>
Co-authored-by: joelbrilliant <joelbrilliant1@gmail.com>
Co-authored-by: CharlesMcquade <6466275+CharlesMcquade@users.noreply.github.com>
* feat(billing): add payment_method union to the billing-state wire type
* feat(billing): carry the payment-method union through the gateway
NAS now sends a typed `paymentMethod` union on /api/billing/state alongside
the legacy `card` field. The gateway parses payloads field-by-field, so the
new field was dropped on the floor before reaching TUI/Desktop.
Parse it into PaymentMethodInfo and re-emit it as snake_case `payment_method`,
matching the translation the rest of this payload already does. The payment
method id is deliberately not carried through — clients have no use for it.
No client rendering changes: `card` stays populated for cards, so every
existing consumer behaves exactly as before and the new field is inert until
a surface opts into reading it.
* fix(billing): send only the fields each payment-method kind declares
The serializer emitted every key for every kind, so a Link method went out
carrying brand, last4 and wallet set to null. That contradicts the shared
type, where each kind declares its own fields: a client testing `'brand' in
pm` would read every Link method as a card, and one trusting the declared
non-null `brand` could crash on it.
Send each kind's own fields, and forward an unrecognized kind by name alone
so a client that predates it can still say something honest. The shared type
gains the matching fallback arm its own comment already promised.
Tests now follow a payload from the server response through to the client
wire for each kind, rather than checking parsing and serializing separately —
which is why the old expectation locked in the wrong shape without noticing.
* fix(billing): keep the payment-method kind narrowable
Typing the fallback arm's kind as `string & {}` borrowed a trick that only
works on unions of plain strings. On a union of objects it makes the
discriminant non-literal, so TypeScript stops narrowing on every arm — even
`if (pm.kind === 'card')` no longer gives you `brand`. The first client to
use this would have hit a compile error and reached for a cast.
An unrecognised kind now arrives as `unknown`, carrying the real name
alongside it, so every arm has a literal discriminant. A type-level test
pins this: it fails to compile if the discriminant stops narrowing.
The parser settles which kind it is, the way the card parser already does,
so the record cannot hold fields that do not belong to its kind and the
serializer no longer re-checks. The type comment also stops claiming `card`
is a safe signal — it is null for Link, so `!card` does not mean "nothing on
file".
generate_title() sent the first 500 characters of the user turn to the
auxiliary model. On a /skill invocation those characters are the skill's own
opening prose, so the session got named after the skill instead of the request
— /work sessions came back as "Isolated Git Worktree Setup".
Route the turn through describe_skill_invocation() first, so the titler sees
what the user typed. Also keep only the first line of the response: a model
that ignores "return ONLY the title" and answers the prompt would otherwise
have a shell transcript stored as the title, truncated mid-command.
A /skill invocation expands into a message that embeds the whole skill body.
Anything that summarizes a user turn from its raw content reads the skill's
prose as if the user had written it.
describe_skill_invocation() sits next to the existing extractor and reuses its
markers, returning `/work — fix the title leak` for an invocation with an
instruction and `/work` for a bare one. It also exports the SQL LIKE pattern
and excerpt-joint sentinel that listing queries need to recognize scaffolding
before a row reaches Python.
The compression lock was gated so that any holder on a Windows host was
assumed alive until the full TTL expired. A crashed holder therefore
stranded every other agent on the session for the whole lease window.
Probe liveness with psutil.pid_exists, which is safe on nt. Keep the
os.kill(pid, 0) path strictly for POSIX: on Windows signal 0 maps to
CTRL_C_EVENT (bpo-14484) and can kill the target's console group, so with
psutil absent the only safe answer stays 'assume alive' and let the TTL
run out.
Also carry the commit fence through cancelled compressions so an aborted
run leaves the lock reacquirable rather than half-held.
The Nous Portal profile publishes a top-level `session_id` that the Portal
uses as its sticky routing key, pinning a conversation to one upstream
endpoint so explicit Anthropic `cache_control` breakpoints stay warm
(those caches are instance-local, so a reroute cold-writes instead of reads).
`NousProfile.build_extra_body` sourced that key only from the explicit
`session_id` argument. Auxiliary call sites — compression, title generation,
vision, web_extract, session_search, MoA slots — funnel through
`agent.auxiliary_client`, which has no session handle and never passes one.
They carried the `conversation=` tag but NO sticky key at all, so each one
routed independently of the conversation it belonged to.
Resolve the key the way `nous_portal_tags` already resolves that tag:
ambient lineage-ROOT contextvar first, explicit argument as fallback. Fixes
every aux call site at once with no per-call-site plumbing.
Also publish the root in the `_compress_context` forwarder when nothing is
ambient. Out-of-turn entry points (`/compact`, the gateway `/compress`
command and its hygiene sweep, partial head compression) call it outside
`run_conversation`'s scope, so the summarizer's aux call ran untagged and
unkeyed; the caller's value is restored in `finally` so a compaction never
leaks its tag.
Scope note: this does NOT keep the compaction turn's own prompt cache warm.
Compaction replaces the history with a summary and rebuilds the system
prompt, so that request is a cold write on any endpoint — what a stable key
buys is the turns AFTER compaction reading the cache it wrote. Under the
default `compression.in_place: true` (#38763) the session id does not rotate
at compaction, so for the main loop the ambient root and the explicit
argument already agree; the ambient resolution additionally holds the key
stable for installs that opt back into rotating compaction and across
delegate-subagent trees.
Salvaged from #71747. Subsumes the aux-call half of #70883 (@webtecnica),
which threaded `session_id` through `call_llm` to close the same gap.
Tests: 4 cases in tests/agent/test_portal_tags.py; both fix halves verified
to fail against the pre-fix behavior via sabotage runs.
Some OpenAI-compatible endpoints — notably Tencent Copilot
(copilot.tencent.com) — only accept streaming chat requests; any
non-streaming call returns HTTP 400 (code 11101, 'Non-stream chat
request is currently not supported'). The main conversation loop already
streams, so interactive chat works, but every auxiliary task (title
generation, compression, web extraction) used the non-streaming path and
failed on each call.
_provider_requires_stream() detects stream-only endpoints
(copilot.tencent.com built in, plus user-configurable
auxiliary.stream_only_base_urls substring markers in config.yaml).
Matching sync auxiliary calls route through _create_with_progress
(force_stream=True) and async calls through the new
_acreate_with_stream, aggregating the chunk stream — including tool-call
deltas and reasoning deltas — into a complete response via the shared
_ChatStreamAccumulator.
Salvaged from PR #60686 by @kudi88 onto the progress-aware streaming
machinery from #71508, addressing both sweeper-review gaps: the async
path now consumes the stream with 'async for' (awaiting create() and
iterating synchronously raised on AsyncOpenAI streams), and tool-call
deltas are reassembled instead of dropped (MCP passes tools= through
call_llm). Under force_stream there is no silent non-streaming retry —
a stream-only provider rejects those by definition, so the original
error surfaces to the normal recovery chains.
The gateway's pre-agent session-hygiene compression killed the summary
call at a fixed 30s wall-clock deadline (compression.hygiene_timeout_seconds),
regardless of whether the summary model was hung or merely slow. A reasoning
model happily streaming a large summary was cut off mid-generation, the user
got '⚠️ Context compression timed out after 30.0s', and a 300s failure
cooldown left the session oversized — a doom loop for slow-but-healthy
auxiliary models.
Timeouts are now liveness-based instead of wall-clock-based:
- agent/auxiliary_client.py: new thread-local aux_progress_hook. When
installed (only by context compression today), the primary call_llm
attempt streams (stream=True) and aggregates chunks back into a complete
response, ticking the hook per chunk. The configured timeout then acts
per stream read (idle) instead of as a total budget. Providers that
reject streaming fall back to the plain non-streaming call; auth/payment/
rate-limit/transport errors propagate unchanged into the existing
recovery chains. Codex Responses (per SSE event) and Anthropic Messages
(per stream event, via the new create_anthropic_message on_stream_event
callback) tick the same hook from inside their wire adapters.
- agent/conversation_compression.py: CompressionCommitFence gains
touch_progress()/seconds_since_progress(); compress_context() installs
fence.touch_progress as the progress hook around the compress call.
- gateway/run.py: the hygiene wait loop treats hygiene_timeout_seconds as
an INACTIVITY budget — while the fence reports fresh progress the wait
extends, bounded by the new compression.hygiene_total_ceiling_seconds
(default 600s, clamped >= the idle budget) so a degenerate trickle
stream still dies. The timeout warning now says the summary model
produced no output, which is the only case that still triggers it.
- config/docs: hygiene_total_ceiling_seconds added to DEFAULT_CONFIG and
configuration.md; hygiene_timeout_seconds documented as inactivity-based.
Tests: tests/agent/test_aux_progress_streaming.py (hook plumbing, stream
aggregation incl. tool-call deltas and reasoning deltas, rejection
fallback, ceiling kill, fence progress surface); two new gateway tests
prove a slow-but-streaming worker survives past the fixed timeout
(sabotage-verified: fails with the old fixed deadline) and a
forever-trickling worker is still cut off at the ceiling.
The unquoted alternative in the directive pattern is `\S+`, so a ref built
by string interpolation truncates at the first space and strands the tail
as loose text next to a broken thumbnail. Composer images live in the app's
userData dir, which on macOS is `~/Library/Application Support/<App>/` — so
every pasted or dropped image hit this.
Adds format_reference_value next to REFERENCE_PATTERN, mirroring
formatRefValue in the desktop's directive-text.tsx, and covers the
round-trip through the parser.
Several file-I/O call sites still use open() / Path.read_text() /
Path.write_text() without an explicit encoding, so they fall back to
the platform default. On Windows CN/JP/KR locales (GBK/CP932/CP949)
any non-ASCII byte in a config/state/user-content file raises
UnicodeDecodeError or UnicodeEncodeError and crashes the caller.
to the remaining hot paths:
- agent/copilot_acp_client.py: fs/read_text_file and fs/write_text_file
(Copilot's read_file / write_file tools,
directly reported in #18637 bug 2)
- agent/model_metadata.py: context-length YAML cache load + two
save sites (context probing is on the
call path of every model invocation)
- agent/nous_rate_guard.py: cross-session rate-limit JSON state
(read + atomic write via os.fdopen)
- cron/scheduler.py: user config.yaml read in run_job
- gateway/delivery.py: cron output writes for AI-generated
content, very likely non-ASCII
yaml.dump call sites also gain allow_unicode=True so the emitted
YAML preserves non-ASCII chars as-is instead of emitting \u escape
sequences.
Adds regression tests that monkeypatch builtins.open / Path.read_text
/ Path.write_text to simulate a GBK locale: each test raises
UnicodeDecodeError / UnicodeEncodeError unless the caller explicitly
passes encoding='utf-8'. Verified that the tests fail on main and
pass with this change, on Linux as well as on Windows.
Refs #18637
Follow-up to the cherry-picked #68258 base: the cross-session-stable
prefix (_cached_system_prompt_static) was only recorded on fresh
builds, so two paths silently degraded to the legacy single-breakpoint
layout (flagged in review of #68258/#69341/#69704):
- Session restore: gateway surfaces build a fresh AIAgent per turn and
restore the persisted prompt verbatim from the session DB; the static
prefix stayed None from turn 2 onward, flip-flopping the wire layout.
- Post-compression cached-prompt reuse: _invalidate_system_prompt()
clears the static prefix, and the keep-cached-prompt branch never
restored it.
Both sites now reconstruct the stable tier and adopt it ONLY when the
authoritative prompt string literally startswith() it — stable-tier
drift (skills edited, identity changed) falls back to the legacy layout
with the stored bytes untouched. Fail-open on any builder error. The
restore-path rebuild is gated on _use_prompt_caching so non-Anthropic
routes skip it entirely.
Refs #68191
Co-authored-by: JonthanaHanh <92574114+JonthanaHanh@users.noreply.github.com>
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
The conversation loop normalizes message text right before the API call so
the request prefix is byte-identical across turns -- the stated reason is
KV cache reuse on local inference servers and better cache hit rates on
cloud providers. Cache breakpoints were injected *before* that pass, which
defeats it.
`_apply_cache_marker` rewrites a plain-string `content` into a
`[{"type": "text", ...}]` block. The normalization pass is guarded on
`isinstance(content, str)`, so every message that just got marked is
silently skipped by it and keeps its raw leading/trailing whitespace. A
message is only marked while it sits in the last-3 window, so:
turn N in the window -> marked, content "file1\nfile2\n"
turn N+1 rolled out -> plain, content "file1\nfile2"
The same logical message is sent with different bytes on consecutive
turns. The prefix stops matching at that position -- which is inside the
span the breakpoints were placed to protect -- so the reusable prefix
collapses back toward the system breakpoint on every turn. Tool results
carry a trailing newline almost by default (any shell command output), so
this is the common case, not an edge case.
Move the injection below every message mutation. Besides fixing the
whitespace divergence this stops breakpoints from being spent on messages
that the orphan sweep or the thinking-only drop is about to remove or
merge away -- a marker on a dropped message is a wasted breakpoint out of
the four available.
Nothing between the old and new call sites reads `cache_control`, and the
mutators now see the plain-string shapes they were written against.
When two consecutive compactions each failed to clear the threshold, the
anti-thrashing breaker blocked automatic compaction PERMANENTLY for the
life of the session: nothing decremented _ineffective_compression_count
(or _fallback_compression_streak) while blocked, so a session whose
middle region was briefly too small to compact never auto-compacted
again — it grew unbounded until the provider's hard context limit, and
only /new or /reset recovered it.
Recovery is a probation probe, not amnesty: after
_ANTI_THRASH_RECOVERY_SECONDS (300s) of continuous block the gate grants
exactly ONE attempt by dropping tripped counters to 1 strike (persisted,
so sibling agents on the same session row — gateway hygiene — unblock
too). An ineffective probe re-trips the guard on the next real-usage
verdict and the next recovery waits a full fresh window, so the worst
case in a truly incompressible session is one compaction attempt per
window — bounded, not thrash.
The recovery clock is armed lazily on the first BLOCKED evaluation and
is deliberately not durable: a restart that loads a durable tripped
counter (#69872) starts a full fresh window blocked, preserving the
restart-must-never-disarm contract (#54923).
Fixes#14694
Three durable ledgers used `with _connect() as conn:` where the sqlite3
connection context manager commits/rolls back but never closes, leaking the
db/-wal/-shm file descriptors on every call. On a long-running gateway this
exhausts RLIMIT_NOFILE and fails unrelated components with
`[Errno 24] Too many open files`. Same bug class as the cron execution ledger
(#69567 / PR #69594), which the connection helpers here are modeled on.
Fix: route every ledger operation through a `_transaction()` context manager
that guarantees `conn.close()` on exit. `_connect()` keeps its
schema-on-connect contract (several tests call it directly) and now self-closes
if schema init fails.
Adds per-module regression tests asserting every opened connection is closed,
including the no-op-update and exception-mid-transaction paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the cherry-picked #68633 commits, closing the final open
review point (egilewski): _relocated_replay_cache_control was applied
only inside `if replayed:`. When anthropic_content_blocks contained
only a blank cache-marked text block, `replayed` came out empty, the
function fell through to the main path's placeholder, and the cache
marker was lost; signed thinking + a blank marked text block likewise
returned with no cacheable carrier for the relocated marker.
The replay branch now appends the non-whitespace "(empty)" placeholder
when no cacheable (text/tool_use) block survives the blank filter and a
blank text block was dropped (or a marker needs a carrier) — so replay
stays schema-valid on Bedrock/strict endpoints and the breakpoint
survives on the placeholder.
Also reconciles the block-level tests from #69517 with the new
drop-then-fallback contract (blank blocks are dropped at the block
level; the message-level result is still always non-blank).
Refs #69512
Co-authored-by: ygd58 <buraysandro9@gmail.com>
Follow-up per independent review of #68633 (GPT-5.6-sol-xhigh in Codex,
reviewer egilewski) on this PR.
Two real bugs in the blank-text-block filtering added by that fix:
1. `effective = blocks or content` fell back to the RAW, unfiltered
`content` variable whenever every block was filtered out as blank --
which happens precisely when the entire message content WAS the
blank/whitespace payload the filter exists to remove (a sole blank
text block, a sole cache-marked blank block, or standalone
whitespace scalar content with no tool_calls). The fallback silently
restored the exact invalid content the filtering just stripped,
leaving the message provider-invalid.
Fixed: `effective = blocks if blocks else [{"type": "text", "text":
"(empty)"}]` -- never falls back to raw `content`. Also moved the
cache_control application (both the relocated-from-a-dropped-block
marker and the message-level marker) to run against `effective`
instead of the pre-fallback `blocks`, so a cache marker on a block
that was the ONLY content still lands on the (empty) placeholder
rather than being silently lost when `blocks` was empty at the
point it would otherwise have been applied.
2. The normal-path blank-text check used `(blk.get("text") or "").strip()`,
which is not type-safe for a truthy NON-string, non-None text value
(e.g. an int or dict from an invalid upstream payload) -- `or`
doesn't substitute for a truthy value, so `(7 or "").strip()` still
raises AttributeError. Now checks `isinstance(text, str)` first,
matching the replay path's `_sanitize_replay_block()`, which the
reviewer confirmed was already correctly type-safe.
Added regression tests for: sole blank list block, sole whitespace
scalar content, sole cache-marked blank block (marker relocation to
the placeholder), a truthy non-string (int) text value both mixed with
a surviving tool_use and as the sole content, and a dict-valued text
field. 7/7 new tests pass; 193/193 in the full
tests/agent/test_anthropic_adapter.py file; 23/23 in
tests/agent/test_prompt_caching.py (unaffected, confirmed).
Ports #63228 forward onto current main per teknium1's review.
Bedrock and strict Anthropic-compatible endpoints reject text blocks
where text is empty or whitespace-only with HTTP 400. The normal
list-content path extended blocks without filtering, and the
ordered-replay fast path (_sanitize_replay_block) returned blank text
blocks unfiltered.
Per review, fixes three gaps in the original port:
1. Type safety: the normal-path filter used blk.get('text', '').strip(),
which crashes with AttributeError when text is explicitly None (not
absent) -- .get()'s default only applies when the key is missing.
_convert_content_part_to_anthropic() can preserve None from an
invalid upstream input text block. Now uses
(blk.get('text') or '').strip() on both paths.
2. Cache marker loss: prompt_caching.py's _apply_cache_marker() sets
cache_control directly on content[-1] for list content. If that last
part happens to be blank text, dropping it without relocating
cache_control silently loses the breakpoint. Both the normal and
replay paths now capture a dropped block's cache_control and reapply
it to the new last surviving cacheable block via the existing
_apply_assistant_cache_control_to_last_cacheable_block() helper
(setdefault semantics, so it never clobbers a legitimately-placed
marker).
3. Scalar whitespace: the non-list content branch
(blocks.append({'type': 'text', 'text': str(content)})) accepted a
truthy whitespace-only string unfiltered. Now filtered the same way
as list-content blocks.
8/8 new tests pass in TestBlankTextBlockFiltering (including None-safety,
scalar-whitespace, and cache_control-relocation regressions on both
paths); 186/186 in the full tests/agent/test_anthropic_adapter.py file.
Teknium
Kyle
knocksen
AKAZIK-py
Alex Fournier
rob-maron
kshitijk4poor
Michael Jordan
Matt Ezell
Kyzcreig
y0shualee
Ben Barclay
zapabob
StellarisW
DonutsDelivery
Atakan
Siddharth Balyan
PRATHAMESH75
Ivan Miao
kshitijk4poor
Rod Boev
Soju06
Brooklyn Nicholson
Aaron Weiker
Jeffrey Quesnelle
kshitijk4poor
HexLab98
fangliquanflq
xxxigm
Tony Simons
Oleksii Kalentiev
embwl0x
Jeffrey Quesnelle
dsad
joaomarcos
kudi88
CoreyNoDream
Drexuxux
konsisumer
Anthony Ruiz
Dhruv Raajeev
ygd58
Israel Lot