Follow-ups on the salvaged bounded-read fix:
- refresh flow: the non-200 branch reads a STREAMED body, which fails
(ReadError/StreamClosed) once the httpx.Client context has exited —
moved inside the context. Repro + regression test use a real socket
server (MockTransport buffers in memory and cannot catch this).
- truncation guard: >limit bodies end with ...[truncated] (mutation-checked
against the is_stream_consumed fallback).
- test mocks now model the streamed-read surface (is_stream_consumed,
iter_bytes, client.send) so non-200 paths exercise the real bounded read.
The as_completed drain + `with` join made the parallel version WORSE than
sequential main in the common case (first endpoint succeeds fast, others
slow/unreachable): main returned at first success, the parallel version
waited for every straggler. Now: after each completion, walk endpoints in
priority order and return as soon as a success is unbeatable (all
higher-priority probes already finished); pool uses shutdown(wait=False) so
losers drain in the background. Mutation-checked: removing the early exit
makes the new timing test fail (8.2s vs <1.5s).
Rebase fold: the original PR predates ZAI_ENDPOINTS growing per-endpoint
probe_models lists; the parallel worker now preserves that candidate-model
fallback loop (was: scalar model). Tests (both mutation-checked):
- candidate-model fallback within one endpoint worker
- ZAI_ENDPOINTS priority order wins over completion order
- all-fail returns None
Follow-up on the salvaged pair: the original guard's `not msg_id` arm let an
id-less internal/synthetic event erase a tracking entry a concurrently-queued
id-bearing message's drain task still needs for recall matching (id-less
events never write entries in _dispatch_inbound_event, so they must never
pop). Tests cover: normal cleanup, id-less non-erasure, overwritten-entry
ownership handoff, TTL eviction + fresh-entry survival.
Adds two regression tests for the #59077 batch: (1) _get_session_rich_rows_batch(compact_rows=True) uses the schema-derived compact projection (no system_prompt, git_branch/git_repo_root kept); (2) list_sessions_rich(compact_rows=True) threads compact_rows through the compression-tip projection call site. Mutation-checked: hardcoding compact_rows=False at the call site fails test 2.
list_sessions_rich()'s compression-root projection called
_get_session_rich_row() once per root — a separate single-row query per
compression root on every session-list render. Resolve every tip id
first, then fetch all tip rows in one WHERE id IN (...) query via the
new _get_session_rich_rows_batch().
_get_session_rich_row() is now a thin wrapper over the batch method, so
the enriched SELECT (preview + last_active) lives in exactly one place —
future column changes (e.g. #42196's include_system_prompt) only touch
one query.
get_compression_tip()'s chain walk is untouched; it's a genuine
per-session graph walk with branch/delegate-exclusion and race handling,
and batching it safely is out of scope here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The INDEXED BY pin is a hard dependency -- SQLite raises 'no such
index' when the named index is missing. That happens in production:
the web dashboard's usage analytics (_get_usage_analytics,
_get_models_analytics) open state.db read_only=True, which skips
_init_schema, so a DB last written by a pre-index version has no
idx_messages_assistant_calls_by_session and every insights call
crashes with OperationalError (reproduced E2E).
Probe sqlite_master once in __init__ and strip the pin from the four
prepared statements when absent -- identical rows, optimizer-chosen
plan, no crash. Replaces the change-detector test that froze the
crash as intended behavior with a fallback-equivalence test.
Review follow-up (#67341): on a freshly initialized state.db (before
ANALYZE has run) the source-filtered branches of _get_tool_usage /
_get_skill_usage did not select idx_messages_assistant_calls_by_session
— the optimizer drove from idx_sessions_source_id and probed each
session's messages via idx_messages_session_active, scanning non
tool-call rows. Pin the index with INDEXED BY on all four fixed-predicate
branches so the plan is deterministic for both the unfiltered and
source-filtered scopes without depending on statistics.
Safe because the index is declared in SCHEMA_SQL (created by every
read-write SessionDB._init_schema) and every InsightsEngine caller opens
a read-write SessionDB; read-only attachments (which skip schema init)
are never used for insights.
Extract the four queries into class constants and add tests: query-plan
coverage for both scopes without ANALYZE, row-level equivalence between
pinned and un-pinned forms, and an assertion that INDEXED BY fails loudly
if the index is absent.
InsightsEngine._get_tool_usage and _get_skill_usage scan messages for
role='assistant' AND tool_calls IS NOT NULL, but no index aligns with
that predicate, so SQLite scans the full messages table on a large
state.db. Add a partial index over exactly those rows.
role and tool_calls are base columns in the messages table, so the index
lives in SCHEMA_SQL (created on both fresh and existing databases via the
executescript on every open) rather than DEFERRED_INDEX_SQL.
Adds schema regression coverage (fresh + reopened DB, plan uses the index)
and an Insights regression test proving tool/skill output is identical with
and without the index present.
Fixes#67341
Two places were using SELECT COUNT(*) when they only needed a boolean:
- has_any_sessions() called session_count() > 1 (full table scan)
- delete_session() used SELECT COUNT(*) WHERE id=? (full matching scan)
Fix:
- Add session_count_ge(n) to SessionDB — short-circuits via
SELECT 1 FROM sessions LIMIT n, returns bool
- has_any_sessions() uses session_count_ge(2) instead of session_count() > 1
- delete_session() uses SELECT 1 ... LIMIT 1 with fetchone() is None
- Add tests for session_count_ge
CI exposed the whole class: feishu tests across MANY files (thread
routing, text batching, sdk executor, ...) inject a mock _client and
skip connect(), so the deferred import leaves the request-builder
globals None. Replace the single-file setUpModule with a session-scoped
autouse conftest fixture that binds the globals once when lark_oapi is
installed; when it isn't, the affected tests already skip via their own
skipUnless guards. Full tests/gateway run: zero failures beyond main's
pre-existing baseline (sorted failure-diff).
Salvage of #57657, ported onto the plugin layout (the adapter moved
from gateway/platforms/feishu.py to plugins/platforms/feishu/adapter.py
since the PR's base). lark_oapi takes seconds to import and holds the
GIL doing it; the module-level import made every gateway boot pay that
cost even with Feishu unconfigured.
- _load_lark_oapi() with double-checked locking binds the SDK globals
on first use; connect() and _standalone_send() call it via
asyncio.to_thread so the loop never blocks on the import.
- probe_bot() also calls _load_lark_oapi() (sync context) so the SDK
probe path is preserved rather than silently degrading to the HTTP
fallback before a first connect.
- check_feishu_requirements() is install-only and no longer rebinds
globals; test_feishu.py gets a setUpModule that binds them eagerly
for tests that inject fake clients.
Includes the dedicated lazy-import test file (check-does-not-import,
connect-loads-on-worker-thread).
Re-derivation of #33612 by @LeonSGP43 onto the rewritten scheduler (the
original is 10,692 commits behind; its tick() no longer exists in that
shape, so this is a fresh minimal fix crediting the PR's idea).
The gateway's built-in ticker calls tick(verbose=False) every 60s. The
idle early-return was gated on 'verbose and not due_jobs', so idle
GATEWAY ticks fell through to load_config() + worker-pool resolution
every minute. Return early on ANY idle tick; keep the post-tick MCP
orphan sweep (main intentionally reaps orphaned stdio children on idle
ticks).
3 new tests; mutation-checked (restoring the verbose-gated guard fails
the config-skip test). 66 scheduler tests green.
Re-derivation of PR #39399 onto current main: pass --prefer-offline to
the web-UI workspace install (both silent and verbose arms of
_install_web_deps) and to the update-time Node dependency refresh in
_update_node_dependencies, so npm reuses its local cache instead of
re-fetching metadata. Test expectations updated to match, mirroring the
PR's own test-update commit.
Review follow-up on the #56798 salvage: the gate shipped fully dormant
(no provider profile sets supports_prompt_cache_key, no production
caller passes it, and no plain 'openai' profile exists to set it on) —
AGENTS.md rejects dead code wired in without E2E proof.
Activate the one endpoint where the field is first-class: exact-host
api.openai.com (OpenAI documents prompt_cache_key; GPT-5.6+ docs
recommend it for cache routing). Deliberately NOT substring matching —
Azure/OpenAI-compat endpoints may reject unknown fields and stay
opt-in via the flag. 4 new tests (imply + 3 spoof/proxy/Azure
negatives); mutation-checked (substring-weakened host check fails the
spoof tests).
The runtime footer (`/footer`) shows what model ran and how full the context
is, but not how long the turn took. On a messaging platform there is no
progress bar and no shell timer — a turn that took 4 seconds and one that took
four minutes produce visually identical replies. Users comparing models,
providers, or reasoning levels have no at-a-glance signal for the one
dimension they most often care about, and "was that slow or did I imagine it?"
is unanswerable after the fact.
Adds a `latency` field to the existing footer machinery, rendering the
wall-clock duration of the agent run: `<1s`, `22s`, `1m05s`.
`gateway/run.py` measures with `time.monotonic()` immediately around the
`self._run_agent(...)` await in `_handle_message_with_agent` — the same
function that already builds the footer, so the value is the user-perceived
turn duration (monotonic, so it is immune to wall-clock/NTP adjustment).
`latency` is deliberately NOT in `_DEFAULT_FIELDS`. It is opt-in via
`display.runtime_footer.fields`. Every existing footer — and every footer a
user has today without touching config — renders byte-identically.
This is enforced by tests, not just asserted:
- `test_latency_not_in_default_fields` pins the default tuple.
- `test_resolve_footer_config_default_fields_exclude_latency` pins what
config resolution produces for an untouched config.
- `test_default_footer_renders_byte_identically` pins five exact output
strings for default-config renders **while supplying `turn_seconds`** —
proving that even when the caller measures timing, a default-configured
footer does not show it.
- `test_default_build_footer_line_ignores_turn_seconds` asserts
`build_footer_line(...) == build_footer_line(..., turn_seconds=125.0)`
under default fields.
Adding `latency` to `_DEFAULT_FIELDS` fails 11 of these tests.
No new config surface (reuses `display.runtime_footer.fields`), no new env
vars, no new core tool, no new model-facing schema. One new module-private
helper (`_format_latency`), one new keyword argument threaded through the two
existing footer functions, and 3 lines in `gateway/run.py`.
`turn_seconds` defaults to `None` and the field is skipped when it is `None`
or negative, so any call site that does not measure timing keeps working
unchanged.
`tests/gateway/test_runtime_footer.py` (+185): `_format_latency` boundary
table (sub-second, rounding at 59.4/59.6, the `m{:02d}s` zero-pad, 60m), the
render/skip/opt-in matrix, field-order placement, `build_footer_line`
threading, and the byte-stability block above.
RED-proved by mutation — each of these breaks tests:
- `latency` added to `_DEFAULT_FIELDS` → 11 failures
- dropping the `turn_seconds is not None and >= 0` guard → 2 failures
- `{sec:02d}` → `{sec}` → 6 failures
- `build_footer_line` not threading `turn_seconds` → 1 failure
51 passed in `tests/gateway/test_runtime_footer.py`; 54 passed across the
footer blast radius. `ruff check` clean.
The existing test only mutated the miss-path return; a mutation to
'return _PROVIDER_LIST_CACHE' (aliasing the global cache) survived the
suite. One line pins the cached-return copy. Mutation-checked.
The retry loop gates on real time.time() < sleep_end; with sleep mocked
to a no-op the test hot-spun 7.5 wall-clock seconds. Advance a fake
clock by each sleep amount instead (pattern precedent:
test_session_activity_persist.py).
Empty content retries previously fired back-to-back with no delay,
wasting up to 3 rapid API calls, and could not be cancelled mid-wait.
Apply the same jittered_backoff() already used for rate-limit and
API-error retries, sleeping in small increments so a user interrupt
aborts the wait instead of blocking until it elapses.
Fixes#35230
Review fold on the #76341 salvage: the substring test ('gh' in
source.lower()) classified GH_TOKEN and GITHUB_TOKEN as gh_cli, so a
user's env-var-specific suppression was silently bypassed (and
suppressing gh_cli silently dropped env tokens). Pre-existing bug on
main, but the PR's early gate makes the classification decide whether
the exchange runs at all. Match resolve_copilot_token's exact
'gh auth token' sentinel instead.
Adds 3 regression tests: env-var suppression gates the exchange,
gh_cli suppression doesn't swallow env tokens, all-sources suppression
skips the resolve subprocess entirely. Also corrects the ~13s comment
(actual worst case ~35s: 3x10s timeouts + 4.5s backoff).
The copilot branch of _seed_from_singletons ran the suppression gate
_after get_copilot_api_token(), which retries the network exchange 3x
with backoff (~13s worst case). A source the user already suppressed
(hermes auth remove copilot gh_cli) still burned the full exchange dead
time on every pool load — model picker open, /model, agent startup —
only to have the entry discarded afterwards.
Move the _is_suppressed() gate ahead of the network call, matching the
early-gate pattern every other singleton branch uses. Suppressed copilot
sources now skip the exchange entirely. Measured: model.options payload
build drops from ~13s to ~0.2-0.4s for a user with copilot suppressed.
Add regression test test_load_pool_skips_exchange_for_suppressed_copilot
asserting the exchange is never invoked for a suppressed source.
build_local_transcribe_kwargs read stt.local.no_speech_prob_threshold /
stt.local.logprob_threshold only for Hermes' post-filter
(_is_hallucinated_segment). faster-whisper's model.transcribe() never
received them, so its internal defaults (no_speech_threshold=0.6,
log_prob_threshold=-1.0) always applied and silently dropped
low-confidence segments before they reached the post-filter — making
those config knobs dead for the first gate.
Non-English speech decodes at a lower avg_logprob, so the English-tuned
defaults discard whole utterances (empty transcript despite correct
capture and language detection). Map the same config values through to
model.transcribe() so both gates stay in sync and the knobs work.
Defaults are unchanged, so behavior is identical unless a user tunes them.
Fixes#74178
Five review findings folded:
- schema cache writes via utils.atomic_json_write (fsync; was bare
tmp+replace), file moved to cache/mcp_schema_cache.json with 0o600
(sibling precedent: registry discovery cache)
- phantom-tool reconciliation: after a lazy server's first-use connect,
cached tools the live server no longer offers are deregistered (were
permanent registry ghosts burning circuit-breaker strikes on every
'Unknown tool' round-trip); stale fingerprint logged
- cache-load path now runs _scan_mcp_description like the eager path
(cache file is user-writable JSON; defense-in-depth)
- write-through skips the disk rewrite when the entry is unchanged
(a flapping stdio server was rewriting byte-identical JSON per
revival)
- _lazy_server_fingerprints no longer write-only dead state (consumed
by the reconciliation logging)
444 mcp tests green (440 pre-fold + 4 new guards); phantom-dereg and
write-skip mutation-checked.
Wires the fingerprint-keyed schema cache (previous commit, @Vansh5632's
design from #56832) into the startup path, re-derived onto main's
current connect machinery:
- register_mcp_servers: servers with mcp_servers.<name>.lazy=true whose
config fingerprint matches a valid cache entry register tools from
cache WITHOUT spawning; miss/stale falls back to eager connect.
- First tool use routes through _ensure_lazy_server_connected, which
composes with the connect cooldown (#50394) and _server_connecting
dedup rather than duplicating the connect path.
- resource/prompt utility handlers (list_resources/get_prompt) also
connect-on-first-use — closes the gap flagged in the original
sweeper review.
- Write-through: a live connect refreshes the cache entry.
Config gate is per-server, default OFF, matching the
idle_timeout_seconds key pattern. 24 lazy/cache tests + 440 mcp-wide
green; mutation-checked (cache-read disabled -> registration test
fails; connect bypassed -> 3 first-use tests fail).
OpenCode Zen's relay rejects the Anthropic-style content block format
that cache markers produce (content becomes a block array instead of a
plain string), causing HTTP 400 with "content must be string, not block
array" for DeepSeek models.
Reverts the DeepSeek addition from commit 6b6435a874 while preserving
the Qwen/Alibaba caching path which continues to work.
Fixes#77217
Root cause: two independent bugs in convert_messages_to_anthropic()
(agent/anthropic_adapter.py), the final conversion step before every
Anthropic messages.create() call, both producing HTTP 400 "text content
blocks must contain non-whitespace text":
1. _ensure_leading_user_turn() synthesized a filler user turn with
content [{"type": "text", "text": " "}] (a single space) whenever the
built payload didn't start with role=user (e.g. after context
compaction leaves a leading assistant summary). The space is itself
whitespace-only, so the guard traded a "leading assistant turn" 400
for the "text content blocks" 400 it now hits. Fixed to reuse the
existing non-blank _EMPTY_TEXT_PLACEHOLDER ("(empty)").
2. _convert_user_message() filtered blank text blocks from list-type
user content with an all-or-nothing check:
all(blank for b in blocks if b.type == "text"). This is vacuously
true when a message has zero text-type blocks (silently destroying
valid non-text blocks like images/documents it never inspected), and
false as soon as any single text block is non-blank — which let a
*sibling* blank text block sit untouched next to valid content and
reach Anthropic as-is. Replaced with per-block filtering (mirroring
the assistant-side logic already in _convert_assistant_message),
preserving all non-blank/non-text blocks and relocating any
cache_control marker carried by a dropped block.
Also added _scrub_blank_text_blocks(), a final defense-in-depth pass run
as the last step of convert_messages_to_anthropic() (after every other
transform, including nested tool_result content lists) so a blank text
block from any current or future producer never reaches the wire. It
logs only structural metadata (message index, role, content location,
block index/type) — never message text, tool arguments, tokens, or
credentials.
An earlier local patch to sanitize_api_messages() (agent_runtime_
helpers.py) attempted to fix this by rewriting blank assistant content
before the OpenAI->Anthropic conversion step, but the real leaks were
introduced downstream of that sanitizer, inside the Anthropic-specific
converter itself — the patch never touched the actual defect and has
been fully reverted (agent_runtime_helpers.py is back to its committed
state; verified via `git diff` showing no changes).
Verified against a real Telegram message end-to-end: the gateway no
longer produces the "text content blocks must contain non-whitespace
text" error on a fresh conversation turn.
Testing:
- 9 new end-to-end regression tests in test_anthropic_adapter.py
(TestFinalPayloadHasNoBlankTextBlocks) covering content="",
content=" ", content=[{"type":"text","text":""}], mixed blank+valid
text, blank text next to a valid tool block, an assistant tool-call
message with blank content, the leading-synthesized-user-turn case,
and a blank text block nested inside a tool_result's own content list.
- Fixed one pre-existing test that had asserted the broken " " filler
behavior as correct.
- Full tests/agent/ + tests/run_agent/ suite (4671 tests) run against
both the patched tree and a stashed pre-fix baseline: identical 148
pre-existing failures in both runs (unrelated subsystems — codex
app-server integration, credential-pool interrupt handling, OpenAI
client lifecycle), zero failures unique to either side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Salvage of #48637 (Fixes#48628). On a NixOS-style install the venv's
site-packages lives in the read-only store, so ensure()'s
uv -> pip -> ensurepip ladder spends ~15s bootstrapping ensurepip only
to fail against a target it can never write. Fail fast with an
actionable message pointing at the system package manager.
Retargeted onto current main (the PR's base predates the durable-target
subsystem by ~8.1K commits) with two corrections to the original:
- Gate on _lazy_install_target() is None. The container deployment sets
HERMES_MANAGED=true AND HERMES_LAZY_INSTALL_TARGET (a writable
volume); the original guard would have blocked installs that path
legitimately satisfies, breaking the NixOS-container mode.
- Reason string starts with 'unsupported ' because
refresh_active_features classifies FeatureUnavailable by that prefix;
the original wording made 'hermes update' report a hard failure
instead of a skip.
Placed after _unsupported_feature_reason so a platform-specific reason
(more actionable) wins, and so ensure() agrees with
refresh_active_features, which pre-checks that same function.
Extracted from #54314 (@flag0x369), re-derived onto current main: macOS
ships bash 3.2 as /bin/bash, which lacks $BASHPID entirely — the
variable expands to empty string, collapsing every concurrent writer's
'unique' temp path onto the same file (torn snapshot writes under
concurrency). mktemp allocates per-writer unique paths portably.
Live-verified: /bin/bash -c 'echo $BASHPID' prints empty on this box.
The workdir allowlist regex was ASCII-only, so perfectly normal
non-ASCII workdirs (Chinese Obsidian vault paths, accented dirnames)
were rejected with 'disallowed character'. Replace the regex with a
per-character check that accepts Unicode letters/digits (str.isalnum)
plus the same safe ASCII punctuation set, while still rejecting shell
metacharacters, control characters (newlines/tabs), and NUL.
Salvaged from PR #54314.
Co-authored-by: kshitij <82637225+kshitijk4poor@users.noreply.github.com>
Recalled session messages can carry raw ANSI escape sequences (e.g.
archived terminal output), which then re-enter the model's context.
Strip them in _shape_message before content is truncated/returned,
reusing tools.ansi_strip.strip_ansi.
Re-applied onto current main (the original hunk predates the
max_content_len truncation in _shape_message; stripping happens on the
raw content before truncation so escape bytes never count against the
budget). Extracted from #40276.
The picker path fetches the Copilot /models catalog multiple times per
process (list_authenticated_providers -> provider_model_ids ->
_fetch_github_models, plus get_copilot_model_context / normalize
helpers). Cache the filtered catalog at module level with a short TTL
so repeated picker opens do not pay a TLS handshake each time.
Fold-fixes on top of the original patch:
- key the cache by api_key so a mid-process credential swap never
serves the previous account's catalog
- use time.monotonic() so wall-clock adjustments cannot extend the TTL
- deep-copy on store/serve so callers cannot mutate cached entries
- tests updated to patch _urlopen_model_catalog_request (main routes
catalog fetches through open_credentialed_url now), plus TTL-expiry
and credential-change coverage
Extracted from #40276.
Salvage of #69926: omit_messages support ported from the PR's
tui_gateway/server.py base onto the post-split methods_session.py
layout. When a Desktop client passes omit_messages=true on
session.resume / session.activate, the RPC returns messages: [] with
messages_omitted: true and an accurate message_count, skipping the
potentially multi-megabyte compression-lineage serialization over the
WebSocket; Desktop hydrates the transcript via the authenticated REST
route in parallel.
The PR's bundled cron-outputs endpoint and codex quiet-timeout bump
were dropped from this salvage as unrelated (invited back separately).
mark_job_run popped a finite one-shot from jobs.json the moment its
repeat limit was reached and returned early — discarding the
last_status / last_error / last_delivery_error it had just written.
Every finished one-shot vanished from `cronjob action=list` with no
inspectable record, and a delivery failure (agent succeeded, platform
send failed) was silently thrown away with it.
Changes:
- mark_job_run now retires a limit-reached one-shot as a terminal
record (state="completed", enabled=False, next_run_at=None) —
mirroring the existing next_run_at-is-None terminal branch — so the
final status and any delivery error persist and surface in the
cronjob tool's list output (which already emits last_delivery_error
and defaults to include_disabled=True).
- claim_dispatch's stale-job cleanup marks already-ran jobs completed
instead of popping them; genuinely wedged claims (last_run_at never
written) are still removed with the operator-visible diagnostic.
- Retention sweep in the due scan prunes completed one-shot records
older than cron.completed_retention_days (default 7; non-positive
disables) so jobs.json cannot grow unboundedly. Recurring jobs and
non-terminal one-shots are never candidates.
Tests: completion retains record + delivery error, list surfaces it,
completed jobs never re-dispatch, sweep prunes old / keeps recent /
ignores recurring / honors the disable knob; recurring lifecycle
unchanged.
A raw newline in the _CMDPOS start-position class made ANY multi-line
quoted argument look like a command boundary, so hermes send message
bodies, multi-line git commit -m messages, and heredoc text that merely
mentioned dangerous command names tripped the unconditional hardline
blocklist and could not run at all.
Mask newlines inside single/double quotes (detection-only, mirroring the
quote tracking in _iter_shell_command_starts) before building detection
variants. Real threats keep blocking: unquoted newlines stay command
separators, command substitutions inside quotes still anchor, and
_mark_command_starts still re-inserts newlines at genuine quote-aware
command starts. Masking runs on the RAW command before normalization,
which strips escapes and would otherwise corrupt quote state.
Regression tests cover both directions: multi-line quoted data passes
(hermes send, git commit -m, heredocs); bare/chained/substituted
shutdown-class and rm-floor commands still block.
The top-level delegate_task description repeated content the model already
receives through parameter descriptions: the concurrency limit (tasks param),
the full nesting clause (role param), context-passing guidance (goal/context
params), and background semantics (background param). Every API call paid for
the duplication (~4,000 chars).
The description now carries only what exists nowhere else in the schema:
use/don't-use routing (execute_code, cronjob), the no-poll rule, the
non-durability warning, the self-report verification contract with concrete
verbs, the language-passing example, the leaf blocked-tool list, and model
inheritance. 3,963 -> 1,704 chars (~570 tokens saved per API call), and the
top-level text is now static (dynamic limits flow only through the two param
descriptions, which are already rebuilt per get_definitions() call).
A/B benchmark across 4 models (gpt-4o, gpt-4o-mini, claude-haiku-4.5,
llama-3.3-70b) showed the naive compaction in PR #72813 regressed weaker
models on exactly the passages it cut (side-effect verification 8/8->0/8 on
gpt-4o-mini; language passing 3/3->0/3 on haiku-4.5). This version keeps
those benchmark-sensitive hooks verbatim.
Tests pin the contracts at keyword level (not prose-literal) plus a size
ceiling, and verify dynamic limits still reach the model via the tasks/role
param descriptions.
Refs #72737, supersedes the delegate_task half of PR #72813.
Two consumer tests assumed a send completes synchronously within the
handler turn: the continuation-drain test polled on handler-call count
then immediately asserted on adapter.sent, and the split-brain heal
test drained with bare zero-delay yields. With ledger calls hopping to
worker threads around each send, the reply can land microseconds after
those checks. Poll for the actual sends with a bounded 2s window —
same invariants, scheduling-robust.
CI slices failed the offload tests with 0.5s witness timeouts: on a
loaded shared runner the event loop thread can take >0.5s to get
scheduled even when NOT blocked, making the probe report a false
positive. A genuinely blocked loop can never set the progress event at
any timeout (the witness coroutine can't run at all), so 5s only
absorbs scheduler flake without weakening the invariant. Mutation
re-verified: reverting the offload still fails all 4 tests.
The sweep-path test parametrizes over _runner/_adapter, which live on
TestGatewayRedeliverySweep; main later added
TestUnconnectedPlatformKeepsItsBudget at the cherry-pick anchor point and
the test landed in that class, where the helpers don't exist
(AttributeError x2). Placement-only move.
The original PR #77274 used positional slicing (current_history[len(history):])
to detect the model-switch-only mutation. But _append_model_switch_marker
strips prior markers in-place before appending the new one, so when a prior
marker existed (every switch after the first in a session), the net length
delta is zero and the slice produces an empty list — the merge path is dead
code for the common case.
Replace with a content-based diff: strip markers from both the turn-start
snapshot and the current history, then check that the non-marker content is
identical. This correctly handles the strip-and-replace behavior.
Also guard against auto-compression making result["messages"] shorter than
the turn-start history — use the full result as the base when that happens.
Added test covering both no-prior-marker and prior-marker cases.
The universal sync fallback in stream_tts_to_speaker ran strictly serially
per sentence — synthesize, play, and only then start synthesizing the next
sentence — so every sentence boundary added a full synthesis-time of dead
air. Chunked streamers (elevenlabs/openai/gemini/xai) already avoid this;
every other provider (edge, piper, plugin providers) paid it on each reply
in voice mode and the wake-word loop.
_SyncSentencePipeline overlaps the two: one single-threaded synthesis
worker (sentences stay FIFO; providers never see concurrent calls from
this loop — same effective concurrency as before) feeds one playback
worker through a small bounded queue, so sentence n+1 synthesizes while
sentence n plays. Lookahead is bounded (backpressure + at most a couple of
temp files), stop_event short-circuits both stages, synthesis failures are
isolated per sentence, temp files are always unlinked, and the finally
block flushes the pipeline BEFORE tts_done_event fires so continuous voice
mode never reopens the mic over its own voice. synthesize/play are
resolved late so existing monkeypatch-based tests work unchanged.
Measured with a real local model provider (OmniVoice plugin, Apple
Silicon), same 3-sentence reply, playback simulated at the produced clips'
true durations, best-of-2 interleaved runs under identical load:
serial pipelined
time to first word 10.8s 4.4s
mid-reply dead air 11.2s 1.8s (second gap: 0.03s)
full reply wall 33.2s 17.0s
Tests: 4 new (timestamp-proven overlap, order + per-sentence failure
isolation, stop skips queued playback, temp-file hygiene); the existing
sync-fallback and display-callback tests pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wrap the masked-link destination in angle brackets so Discord does not
unfurl an OG-preview embed under every tool progress bubble. quote()
percent-encodes any <> inside the URL itself, so the wrapper cannot be
broken out of.
The gateway lifecycle guard (cron/lifecycle_guard.py) applied shell-style
tokenization and script-reference resolution to non-shell content, with two
regressions:
#77131 - every .py cron script using pathlib division was hard-blocked:
Path.home() / ".hermes" / ".env" tokenizes the bare "/" operator as an
executable path, which resolves to the filesystem root; the regular-file
check then fails closed as unsafe. Since Python runs under the
interpreter, never through a POSIX shell, the shell-script reference walk
is a false-positive generator on Python sources. check_gateway_lifecycle
now skips the walk for *.py scripts (the direct command regex still scans
the full text), and _iter_referenced_shell_scripts skips pure-separator
tokens.
#76762 - terminal commands invoking a binary by absolute path (e.g.
/usr/bin/python3) crashed the guard with ValueError: embedded null byte:
the walk read the binary's bytes, decoded them as text, and re-tokenized
machine code; the recursion then hit Path.resolve() on a NUL-bearing
path while only OSError was caught. _read_referenced_script now skips
NUL-containing files (binaries are not referenced shell scripts) and
resolve() tolerates ValueError.
Shell scripts (.sh/.bash/.zsh) keep the full deep scan; literal lifecycle
commands in .py scripts are still blocked by the direct regex. New tests
cover all four behaviors.
The account was written under the new pickle key before sessions were
re-pickled. The account is effectively the migration's commit marker —
once it reads under the current key, the fast path short-circuits every
later startup — so a sweep that errored or was interrupted left the
remaining legacy-key sessions stranded permanently with no retry.
Sweep first, commit the account last, and return False on sweep failure
so the migration is retried on the next start.
Also corrects the unreadable-row log: it claimed rows were being dropped
while no DELETE was ever issued. Such rows are left in place (already
unusable; deleting crypto material on a guess is not worth it) and the
message now says so.
Adds session-sweep coverage, which was previously absent: rows rewritten
under the current key, rows already current left alone, unreadable rows
left in place, and a failed sweep that leaves the account uncommitted.
The existing migration test now fakes the olm C-extension so the suite
no longer requires libolm.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Olm account pickle key is derived from the account ID plus the
configured device ID (acct:device_id). If the crypto store's account was
created before MATRIX_DEVICE_ID was set — e.g. the very first password
login, where the device ID is only known after connecting — it gets
pickled under "<acct>:default". Setting MATRIX_DEVICE_ID afterwards (a
reasonable thing to do once you know the device ID you want to pin)
changes the derived pickle key, and every subsequent unpickle attempt
fails with BAD_ACCOUNT_KEY. In optional-E2EE mode that failure is
swallowed and encryption silently stays disabled instead of surfacing an
actionable error.
_migrate_legacy_crypto_pickle() detects the BAD_ACCOUNT_KEY failure,
tries the known legacy pickle keys, and re-pickles the account (plus
every stored olm/megolm session — sessions share the same pickle key, so
migrating only the account would leave them unreadable on the next
decrypt and silently break key sharing with peers) under the current
key. It only reports failure when no known key can unpickle the
account, with a log message pointing at what changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
connect() resolved client.device_id as `self._device_id or resolved_device_id`,
so a configured MATRIX_DEVICE_ID masked the live whoami() device. With
persisted A, configured A and a rotated token reporting B, the reset
compared A to A and never fired — exactly the token-rotation case this PR
claims to handle.
An access token is bound to one device and the homeserver only accepts key
uploads for that device, so a configured value naming a different one
cannot work. The live whoami() device now wins on conflict and logs an
error naming both. The configured value is still preferred when whoami()
reports no device.
Adds a connect()-level regression for persisted A + configured A +
whoami B, and corrects test_connect_uses_configured_device_id_over_whoami,
whose stated premise this inverts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The crypto store is keyed by Matrix user ID, not device ID, so swapping in
a new access token (which mints a new device_id) silently inherits the
previous device's Olm account. That account's identity keys can never be
published under the new device ID, and the pickle key embeds the old
device ID anyway — the result is stale-key mismatches and cross-signing
signatures the homeserver refuses to replace, degrading E2EE in ways that
are hard to diagnose (peers silently withhold room keys).
_reset_crypto_store_if_device_changed() compares the store's persisted
device ID against the live one at connect time and wipes the store on
mismatch, so a fresh Olm account is generated for the new device instead
of reusing stale key material.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Post-tool compression path passed context_compressor.last_prompt_tokens (0 in the no-usage
fallback) to _compress_context instead of the overhead-aware _real_tokens computed just above
— same tool-blind bug as the overflow handlers (upstream PR #77169 review, teknium1). Also adds
production-path regression tests asserting the 413, context-overflow (two wordings), and
Anthropic long-context recovery handlers pass estimate_request_tokens_rough(..., tools=...)
(sentinel-patched) to _compress_context.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drain reserves a slice of the shutdown budget so flush_all_sessions
still runs when in-flight turns outlast the window. But that flush was
unconditional: a session whose turn was still running got its one-shot
_finalize_session spent mid-turn, and the executor.shutdown(wait=False,
cancel_futures=True) immediately after does not join the turn. The
session was then permanently un-finalizable and its active-session lease
had been released out from under live work — the same persistence and
lifecycle race the drain exists to close, just relocated past the
deadline instead of removed.
Give _turn_futures a session association (Future -> sid, the same key
space as server._sessions) at both submit sites, and on deadline expiry
exclude the sids whose futures are still running from the flush. Those
sessions are retained unfinalized and therefore recoverable; sessions
with no live turn finalize exactly as before. The done-callback now pops
under the lock, since a bare dict.pop is not the drop-in set.discard was.
wait semantics, the reserve math and the bounded per-tick sleep are
unchanged, so this adds no shutdown latency. All three shutdown callers
(orphan, sigterm, and the tight stdin_closed wait=2.0 path) funnel
through this one function and are covered.
The drain loop slept a flat 0.05s per tick, so it could overshoot its deadline
by up to one tick and spend part of the reserve withheld for
flush_all_sessions(). For a small `wait` the reserve is itself half the budget,
so a single overshoot can consume all of it: at wait=0.34 the drain budget is
0.17s but the loop requested 4 x 0.05 = 0.20s of sleep.
Clamp each tick to the remaining time. The new test asserts on the summed
*requested* sleep rather than wall-clock, which is deterministic: every sleep is
bounded by the strictly-decreasing remainder, so the total can never exceed the
drain budget regardless of how the scheduler interleaves.
ComputeHost.shutdown() called flush_all_sessions() before its own in-flight
turn drain loop. server._finalize_session latches on session["_finalized"]
and every later call returns immediately, so that one flush was spent while
turns were still producing output: the unflushed tail was never persisted,
commit_memory_session wrote long-term memory from a truncated transcript, the
session's DB row was marked ended while it was live, on_session_end fired with
completed=False/interrupted=True against a running session, and the
active-session lease was released out from under a turn. The drain loop exists
precisely so that mid-turn work survives a teardown; finalizing first defeated
it. Reachable from all three teardown paths: the parent/orphan guard (which
os._exit(0)s immediately after), the SIGTERM/SIGINT handler, and stdin close.
Drain first, then flush. A slice of the caller's budget (_FLUSH_RESERVE_SECS,
never more than half of it so a short explicit wait still gets a real drain) is
withheld from the drain so the flush still runs when turns outlast the window:
HostSupervisor SIGKILLs the host _SHUTDOWN_TIMEOUT_SECS after SIGTERM — 10.0s,
the same value as shutdown()'s default wait — so a drain allowed to consume the
whole budget would leave the durability write racing that kill. `wait` itself is
unchanged, so total shutdown latency and the SIGTERM->SIGKILL margin are
unchanged.
strip_think_blocks passed the same response-scrubbing strings through
re's pattern dispatcher on every response. Skills Guard repeated the
same work for 121 patterns against every scanned line.
Compile the existing expressions once and reuse Pattern.sub/search. Keep
each generic tool-call tag in its own paired expression so mismatched
openers retain their payload while existing stray-closer cleanup remains
unchanged.
Part of #33208
Salvaged from #32713 by @ErnestHysa.
Co-authored-by: ErnestHysa <takis312@hotmail.com>
Deferred model switches append a marker and bump history_version at
turn start; the dispatcher was snapshotting history before that
mutation, so the version-mismatch guard rejected the turn's own
result as a stale/concurrent write. Move the snapshot to after
_apply_pending_model_switch/_sync_agent_model_with_config, under
history_lock, so the turn's own preparatory mutation is included in
its baseline while the anti-stale guard still catches real external
writes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A session's YOLO bypass lived only in the in-memory
tools.approval._session_yolo set (or the process-frozen --yolo env
var), so resuming a session in a fresh process silently reverted the
user's /yolo ON — dangerous commands started prompting again.
Persist a yolo_mode flag in the session row's model_config JSON and
restore it on every CLI resume path:
- SessionDB.set_session_yolo() merges the flag into model_config
(same lineage-preserving merge as update_session_runtime_lock);
SessionDB.session_yolo_enabled() reads it back, false on any parse
failure.
- /yolo toggle persists ON and OFF through the new helper; the
compression/branch session-id rotation carries the flag onto the
continuation row.
- --yolo launches record the flag at session creation (agent_init),
and a /yolo toggled before the lazily-created row exists is carried
into the creation-time model_config (_ensure_db_session).
- HermesCLI._restore_session_yolo() re-enables the bypass on startup
--resume/-c, the deferred init path, and mid-chat /resume, with a
visible '⚡ YOLO mode restored from session' notice. No-op under a
frozen process-wide --yolo and never enables on absent/garbage flags.
When an approval prompt expired without a response, every CLI-side path
collapsed the timeout into the same 'deny' choice as an explicit user
refusal, so the agent was told the user denied the action when the user
simply never answered. The gateway wait already distinguished the two
('timed out without user response... Silence is not consent.'); this
brings the CLI/TUI/ACP surfaces to parity.
- prompt_dangerous_approval(): input()-path expiry now returns a distinct
'timeout' choice (still fail-closed).
- cli.py _approval_callback + hermes_cli/callbacks.py approval_callback:
deadline expiry returns 'timeout' instead of 'deny'.
- check_all_command_guards / _run_approval_gate CLI tails: 'timeout' maps
to outcome='timeout' with a 'timed out without user response... Silence
is not consent.' BLOCKED message (matching the gateway wording);
explicit deny keeps outcome='denied' and gains user_consent=False for
shape parity.
- computer_use: 'timeout' verdict threads through the CLI adapter and
yields a 'prompt timed out — the user did not respond' error instead of
'denied by user'.
- ACP permissions bridge: FutureTimeout returns 'timeout' (other failures
still 'deny'); elicitation maps 'timeout' to 'cancel' like the gateway's
unresolved outcome; codex wire mapping documents deny/timeout→decline.
- write_approval already treats unknown choices as 'stage, not drop', so
a timeout now stages the memory write instead of silently refusing it.
Every timeout path remains fail-closed — the action never runs; only the
classification reported to the agent changes.
Follow-up for the salvaged #29239: regression test drives the real
gateway _reload_runtime_env_preserving_config_authority() path with a
stale .env TERMINAL_ENV=docker vs config.yaml terminal.backend=local,
and the hermes debug dump test that pinned the old stale-env-wins
symptom now pins the fixed contract (config wins, override line kept
as defense-in-depth for post-load env mutation).
A leftover TERMINAL_ENV in ~/.hermes/.env (written by `hermes setup` or
shell exports) was silently overriding terminal.backend in config.yaml,
so users switching from docker to local saw `hermes config show` agree
with their change while the gateway / cron / batch_runner still ran
against the old backend.
load_hermes_dotenv now re-applies config.yaml's terminal.* values on top
of whatever the .env files set, so the documented source of truth wins
for every entrypoint that goes through the loader.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
Live-run findings from a real fact-checking task (ankylosing spondylitis
genetics, 7 authoritative sources) against the new mode:
- Verbatim check rejected a legitimate quote because web_extract returns
markdown: the MedlinePlus sentence is "including _[ERAP1](https://...)_,
_[IL1A](...)_" on the wire but plain prose to a reader. The agent was
forced onto a weaker evidence fragment — the opposite of the point.
Matching now canonicalizes inline links to their label and drops
emphasis/code markers and backslash escapes on both sides, so quoting
the sentence a reader sees works. Paraphrases are still rejected.
- Escaped asterisks (HLA-B\*27) no longer have to be reproduced in the
quote, so extractor artifacts stop leaking into rendered evidence.
- New `render --replace-in <draft>`: rewrites a draft's Sources block in
place, idempotently. Previously the only path was hand-slicing the
file, which also tripped over the emitted heading being `## Sources`
while the prose said "Sources:".
- verify stats: report the provenance total that the percentage is
actually computed from (cited + [unverified], counted once), and print
the line as `info:` instead of `warn:` when nothing is wrong. The old
line printed 17 cited / 2 unverified next to 72%, which does not
reconcile — a sentence can be both.
- SKILL.md documents the emitted heading, --replace-in, and exactly what
counts as a prose sentence for --min-coverage.
7 new tests (47 total) using the real MedlinePlus/Frontiers markup;
6 sabotage runs, all red.
Extends the citation ledger with evidence-backed fact-checking:
- New `quote` subcommand attaches verbatim supporting quotes to a
source; the quote is rejected unless it appears verbatim
(whitespace/case-insensitive) in the fetched page text, so a
paraphrase or misremembered figure cannot masquerade as evidence.
- `verify --evidence` fails a draft whose cited sources carry no
attached quote.
- `render --style evidence` prints each source's quotes beneath its
URL, showing the claim -> source -> exact-text chain.
- `[unverified]` marker declares model-knowledge claims; counts toward
--min-coverage so provenance is declared for every sentence without
forcing fake citations.
- SKILL.md: new Fact-Checking Mode section + pitfalls; version 1.1.0.
- 10 new tests (40 total), all proven live by sabotage runs.
Covers the fact-checking/evidence-transparency half of #28289.
- tests/agent/test_session_activity.py asserts against
ACTIVITY_DESCRIPTION_MAX instead of the literal 120.
- The session-stall WARNING log line names its config knob
(agent.session_stall_timeout) so operators can find the setting.
- hermes_state.py: collapse the triple blank line near line 191.
- hermes_cli/status.py no longer imports the private
hermes_cli.main._relative_time: the helper moved to a public home
(hermes_cli.timefmt.relative_time); main._relative_time stays as a
thin back-compat wrapper (sessions_cmd and external patchers keep
working).
test_progress_extends_idle_budget_until_success raced wall-clock: the
0.1s-idle/0.04s-tick shape left ~60ms of slack per tick, so one slow
scheduler pass on a loaded CI box lapsed the idle budget mid-loop.
Widened to 0.5s idle / 0.1s ticks (5x per-tick margin, total runtime
still <1s) per the FLAKY policy's minimum-margin guidance.
The fail-fast admission path (bounded compress pool, F6) only logged a
WARNING; in the compression-attempt telemetry stream a wedged pool
looked like compression simply stopped being attempted. Emit the
existing attempt telemetry with failure_class='pool_saturated'
(commit_status=aborted, split_status=aborted) on refusal, following
_emit_compression_attempt_telemetry's existing call shape. Regression
extends the F6 saturation test (sabotage-verified).
The compression heartbeat's terminal 'context compression completed'
stamp force-persists against the PARENT session id (agent.session_id at
stamp time). After the out-of-place rotation the parent is archived but
kept advertising a fresh last_activity_at + terminal label forever.
Clear the parent row's activity labels best-effort after a committed
rotation (keeps last_activity_at so idle clocks stay continuous; the
child carries live labels). Regression asserts the archived parent's
labels are cleared while the child's lineage is intact
(sabotage-verified).
A wedged adapter transport (network hang, dead websocket) previously
blocked _check_session_stalls forever: sibling candidates in the same
pass were never evaluated and the watcher stopped ticking. Wrap the
send in asyncio.wait_for (15s); on timeout log a WARNING and do NOT
latch, so the next tick retries. Regression uses a never-resolving fake
adapter and proves the pass completes, a healthy sibling candidate is
still notified in the same pass, and the watcher ticks again
(sabotage-verified against the unbounded send).
revoke_commit_admission() used to invoke the holder-qualified lease
release unconditionally — including while an admitted commit was still
mutating SessionDB — letting a second compressor acquire the durable
lock mid-commit and interleave with the first commit's writes.
The admission_revoked flag store stays lock-free, but the lease-release
decision now coordinates with the fence lock:
- revoke acquires the fence lock non-blocking; on success no commit can
be in flight (an admitted commit retains the lock until finish_commit)
and the release runs immediately, still under the lock so a racing
begin_commit cannot slip between the check and the release.
- on failure the release is deferred: finish_commit() re-checks
_admission_revoked and performs it AFTER the commit completes (prompt
even if the worker thread is later parked), and the begin_commit
refusal path does the same for a revoke that lost the race to a
transient lock-setup/cancel boundary. All paths are idempotent with
the worker's own outer cleanup (DB release is holder-qualified).
Invariant encoded + tested: no second compressor can acquire the durable
lock while an admitted commit is still mutating; after a post-revoke
commit finishes the lease is released promptly. Both regressions
(revoke-during-commit deferral, revoke-before-commit immediate release +
refused begin_commit) are sabotage-verified.
Rebase onto origin/main brought in 'let explicit interrupts cancel safely',
which predates this branch's pooled progress-timeout + F1-F6 fence rework.
Reconcile the two:
- begin_commit(cancel_event) re-checks the hard-cancel Event under the
fence lock again (lost in the mechanical rebase).
- compress_context: restore aux_interrupt_protection around the summary
call, the post-return frozen-cause AuxiliaryExplicitCancellation check,
and the full rollback/telemetry handler for explicit interrupts.
- run_agent._compress_context: recreate the per-attempt fence registration
(_active_compression_commit_fence) that hard_interrupt() uses to
serialize cancel admission, and thread that exact fence through both
the direct and pooled paths (run_compress_context_with_progress_timeout
now accepts an external fence).
- test: the pooled worker isolates the live transcript (F3), so the
hard-interrupt rollback regression mutates the engine's input snapshot
rather than reaching around it to the caller's list.
Same hermetic-CI trap the concurrent-fork suite already guards against:
without credentials the one-time feasibility probe aborts compression
before the stubbed engine starts, so the F3/F4 blocked-state assertions
went vacuous-false in CI while passing on credentialed dev machines.
- Config docs now describe session_stall_timeout precisely: a RECOVERY
notifier for an in-process AIAgent with an adapter-queued follow-up —
not a general gateway/session stall detector — with a per-AIAgent scan
cadence (not globally coordinated per durable session).
- import_sessions documents the deliberate export-includes /
import-resets asymmetry for the activity fields (no resurrected
'working' labels on machines where no agent runs), with a regression
pinning both halves.
- Strip trailing whitespace in contributors/emails/fangliquan@qq.com
(git diff --check housekeeping).
PR #76354 review, scope/contract items + housekeeping.
The stall watchdog gathered pending/activity candidates and later sent
the recovery notification from that aging snapshot — an agent that made
progress (or drained its queue) between the scan and the send received a
false stall notice mid-recovery.
Re-read the adapter pending slot, the overflow queue, and the live
activity snapshot immediately before delivery; abort the send and re-arm
the latch (pop it) when the candidate is no longer stale, so a future
genuine episode still notifies.
Race regressions: progress between scan and send aborts delivery;
pending-drained between scan and send aborts delivery; a genuinely
still-stale candidate is still delivered exactly once.
PR #76354 review, 'watchdog can send /new using a stale snapshot' /
merge gate 8.
Activity heartbeat writes and turn-end label clears ran synchronously on
the response-critical path with the full ~20s routine write-patience
budget — under contention an otherwise-finished reply could stall for
seconds just to update observation labels, mimicking the very stall the
watchdog detects.
touch_session_activity and clear_session_activity_labels now use a
dedicated 0.5s patience budget (they are observation-only; the next
heartbeat window retries naturally), and a no-op label clear (labels
already empty) skips the write transaction entirely.
Regressions: with another connection holding BEGIN IMMEDIATE, both writes
give up well under the routine budget; the no-op clear performs zero
write transactions.
PR #76354 review, 'activity writes are synchronous on critical paths' /
merge gate 9.
Both progress-aware waits (sync compress wrapper and gateway session
hygiene) slept a FULL idle interval and only then compared progress, so
progress early in an interval let silence approach 2x the configured
idle timeout before the waiter noticed. Compute each wait slice as
idle_timeout - elapsed_since_last_progress instead.
Regression: a worker that reports progress early and then goes silent is
timed out in ~1x the idle budget, not ~2x.
PR #76354 review, 'idle timeout can allow nearly twice that silence'.
The process-wide 4-worker pool retained the stdlib executor's unbounded
queue: four hung summaries wedged every slot, a fifth compression queued
silently, waited out its whole budget without starting, and remained
eligible to run later as an expensive stale job whose fence was already
cancelled (the first fence check used to sit AFTER the summary call).
- Bounded admission: submission fails fast (messages returned unchanged,
loud warning) when all pool slots are occupied; slots are freed by a
future done-callback. Recovery contract documented at the constant: new
work fails fast while wedged, wedged workers are fence-cancelled and
restore service when they return; a worker that never returns costs its
slot — bounded, observable degradation instead of unbounded queueing.
- Not-yet-started futures are cancel()ed on timeout.
- The cancelled fence is checked BEFORE any expensive summary work, both
in the pooled wrapper (stale queued job) and inside compress_context
(pre-summary gate), so a stale job never burns an LLM call or acquires
session state.
Saturation regression: 4 event-blocked summaries wedge the pool, a 5th
submission fails fast (asserted while the four are provably still
blocked), the refused job never runs after worker recovery, and a fresh
submission after recovery succeeds.
PR #76354 review, blocking finding 6 / merge gate 7.
Session rotation runs on the pooled worker thread, whose copied context
gets the child id — the CALLER's ContextVar still holds the parent, and
get_session_env() prefers a bound ContextVar over os.environ. Tools and
subprocesses invoked on the caller thread after a compression.in_place=false
rotation therefore saw the STALE parent HERMES_SESSION_ID.
After the pooled wrapper returns, rebind the session id in the caller's
own context (set_current_session_id) alongside the existing logging
repair; idempotent when no rotation happened.
Behavioral regression: with the gateway-style bound session context, a
post-compression get_session_env("HERMES_SESSION_ID") read on the caller
thread now returns the child id.
PR #76354 review, blocking finding 5 / merge gate 6.
A host timeout previously left the timed-out worker holding the durable
per-session compression lock AND refreshing its lease indefinitely, so a
truly hung summary blocked every later compression attempt; and a LATE
successful summary could clear the failure cooldown the host had just
recorded.
Transplant the lease-cancellation invariants from PR #71569
(@ciabata-git): the worker publishes an idempotent, holder-scoped release
hook on the fence once it owns the durable lock (begin_lock_setup /
register_cancelled_lock_release close the acquire→publish race), the
refresher start is serialized against the release path, and the host
invokes the hook on idle timeout, hygiene timeout, and every unwind
(revoke_commit_admission now also releases). ABA safety: the SessionDB
release is holder-qualified (DELETE ... WHERE holder = ?), so a stale
release can never free a replacement holder's lease.
State ordering: the compressor consults a fence-cancellation check BEFORE
clearing the failure cooldown, so a late worker cannot undo the host's
timeout cooldown; the check is installed only for the fenced call and
removed in a finally.
Regression implements the reviewer's exact 5-step scenario: summary
blocked indefinitely → host timeout → a NEW compressor acquires the
durable lock while the old summary is STILL blocked → old worker released
→ it cannot clear cooldown, release the new holder's lease, or publish
stale state.
PR #76354 review, blocking finding 4 / merge gates 4 + 5.
Co-authored-by: ciabata-git <ciabata-git@users.noreply.github.com>
The pooled worker closure captured the caller's live `messages` list and
compress_context explicitly supports plugin/legacy context engines that
mutate that list in place — so after a host timeout, a late engine could
rewrite the live conversation (roles, ordering, persisted content)
concurrently with the resumed turn.
The worker now deep-snapshots the transcript on the worker thread before
any engine code runs; the caller's list object is never handed to pooled
code. Results reach caller-visible state only through the returned value
of an ADMITTED commit (the host discards results on timeout/cancel), and
durable SessionDB mutation was already gated behind the commit fence.
No-op passes map the unchanged snapshot back to the caller's original
list so identity-based no-op detection and flush dedup keep working.
Document the thread-safety contract for context-engine and
memory-provider extension points (they now run on pooled threads) in the
module docstring and the context-engine plugin guide.
Regression: an in-place-mutating engine plus host timeout proves the
caller's live transcript is byte-identical WHILE the worker is still
blocked inside the engine (released only after the assertions).
PR #76354 review, blocking finding 3 / merge gate 3.
The sync compress wrapper only handled concurrent.futures.TimeoutError;
KeyboardInterrupt, task cancellation, or any other exception while
waiting let the host unwind while the detached worker kept full commit
authority — it could later enter the commit fence and mutate durable
state (in-place archival, session rotation) behind the caller's back.
Wrap the whole host wait in try/finally: any exit that did not settle the
worker (returned result or won the fence race) revokes future commit
admission via a new lock-free CompressionCommitFence.revoke_commit_admission()
(begin_commit re-checks the flag under the fence lock, so no admitted
commit is ever abandoned mid-mutation). The gateway hygiene wait gets the
same guarantee via a BaseException handler that revokes admission and
defers helper cleanup until the worker actually returns.
Reconciliation with PR #74449 (suparious): that PR routes EXPLICIT host
interrupts into auxiliary-call cancellation; this change is the
complementary host-side guarantee that no unwind — explicit or not —
leaves an unfenced worker. The two compose (fence revocation here is the
outer safety net; #74449's aux cancellation remains the fast path) rather
than duplicating one another.
Regressions: KeyboardInterrupt and generic-exception unwinds assert the
fence is revoked WHILE the worker is still blocked pre-commit, then
release the worker and prove begin_commit() is refused.
PR #76354 review, blocking finding 2 / merge gate 2.
begin_commit() retains the fence lock until finish_commit(), so a hung
SessionDB commit made try_cancel_before_commit() return None forever and
the host spun ahead of the overrun-warning loop — a genuinely hung commit
stayed unbounded AND silent. Add a lock-free phase marker (threading.Event
set inside begin_commit while the lock is held, readable without it) and
break the host spin on commit_in_flight so the bounded overrun loop — and
its WARNING + on_commit_overrun surfacing — is reachable WHILE the commit
is still blocked. Applies to both the sync compress wrapper and the
gateway session-hygiene wait.
Regression asserts the warning and callback fire while the event-gated
fake commit is still blocked; the test releases the worker only after
those assertions (addresses helix4u's released-before-asserting callout).
PR #76354 review, blocking finding 1 / merge gate 1.
The stubbed-compressor fixture still let the one-time compression-model
feasibility probe run inside the first _compress_context call. On machines
with real credentials configured, that probe resolves a live auxiliary
provider (credential pool seeding, Copilot token exchange over HTTPS),
which nondeterministically exceeds the 2s event-timing budget in
test_fence_cancelled_compression_leaves_lock_reacquirable (reproduced on
PR #73031's own head). Mark the probe done in the shared fixture: these
tests exercise locking/fencing/rotation, never aux feasibility. Suite
runtime drops from ~90s to ~4s.
Heartbeat write discipline for the durable SessionDB activity projection:
- Pin the cadence in a named constant
(SESSION_ACTIVITY_HEARTBEAT_MIN_INTERVAL_SECONDS = 60s, contract >= 30s,
deliberately config-independent so no compression.*/agent.* setting can
turn the heartbeat into a high-frequency writer on the contended
SessionDB write path).
- The write already rides the standard _execute_write patience path via
SessionDB.touch_session_activity — verified, now documented in the
docstring.
- Best-effort hardening: a failed heartbeat write never raises into the
agent loop; the bare 'pass' becomes an explicit debug log with traceback.
- Tests: direct proof that a heartbeat DB failure doesn't propagate,
the cadence constant is pinned >= 30s, and the rate limiter keys off
the shared constant (boundary tested on both sides of the window).
The post-begin_commit() waiter previously called unbounded future.result(),
so the advertised compression.context_total_ceiling_seconds was silently
unenforced for commit-phase hangs. The commit still must complete (abandoning
an in-flight SessionDB mutation would diverge live messages from durable
state), but the wait is now bounded in increments against the remaining
ceiling: on ceiling breach the overrun is logged (WARNING escalating to
ERROR), surfaced once through the user-visible warning channel via the new
on_commit_overrun callback (wired to _emit_warning in run_agent.py), and the
host keeps waiting in bounded slices until the commit finishes.
Documented guarantee (config comment + docs, en/zh): summary phase bounded
by the ceiling; commit phase logged + surfaced if it exceeds it — never
silently hung, never abandoned mid-commit.
Test updated to assert the surfacing fires (previously accepted a silent
over-ceiling wait); adds coverage that a raising overrun callback cannot
break the commit wait.
Once begin_commit() wins, SessionDB mutation cannot be fence-cancelled;
document that context_total_ceiling_seconds covers the summary phase only
and pin the hang-wait contract in tests.
Host progress timeout leaves compress_context running on a daemon worker while
the live turn continues. Latch heartbeat silence on fence cancel or terminal
timeout/cooldown provenance so a later UNKNOWN stamp cannot re-arm
agent.compression and poison stall clocks.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
Every tool schema ships on every API call. The terminal schema was
5,641 chars (~1,410 tokens) and execute_code 2,842 (~710) — the two
largest core tools, padded with repeated war stories and triple-stated
rules. Schema token audit across 88 tools: ~33k tokens total.
This trims prose while preserving every hard rule (each still stated
exactly once):
- terminal description 2,324 -> 1,233 chars: tool-redirect lines
collapsed to one sentence; background/notify guidance deduplicated
(was stated in desc + 2 params); PTY/pager rules merged.
- background/notify_on_complete/watch_patterns params 692/508/1,114 ->
~330/250/490 chars: kept the mutual-exclusion contracts, the
rate-limit consequence, and the bounded-vs-long-lived distinction;
dropped narrative repetition.
- execute_code description tightened (helper docs inlined to one line
each; when-to-use kept).
Net: terminal schema 5,641 -> 3,386 chars, execute_code 2,842 -> 2,522
— ~700 tokens saved on EVERY request with the terminal+code toolsets.
One test updated (pinned a removed phrase; now pins the rule's new
phrasing).
Truncated terminal output was information LOSS: the middle was gone
and the only recovery was re-running the command (data: 1,394
truncation markers in a 250k-call window, with re-runs and grep
retries chained behind the big ones).
Truncation is now deferred retrieval (opencode/goose/qwen-code
pattern, codex's original_token_count idea):
- tools/environments/base.py: _BoundedOutputCollector gains an
optional spill tee — when foreground output overflows the capture
window, the FULL stream is teed to
~/.hermes/cache/terminal-output/out-*.log (lazy file creation with
backlog backfill, 5MB hard cap, 7-day opportunistic cleanup,
disk errors never break execution). All three _wait_for_process
returns attach {output_total_chars, full_output_path} via a shared
finalizer.
- tools/terminal_tool.py: redacts the spill with the same
redact_terminal_output pass as the visible output (no secret
persists unmasked), then surfaces output_total_chars,
full_output_path, and a truncation_note pointing at
search_files/read_file instead of a re-run.
Non-truncated results are byte-identical; internal unbounded
consumers (file-ops cat reads, RPC reads) are untouched (spill only
arms with bounded_capture=true).
'Found N matches for old_string' (190+ occurrences in a 250k-window)
previously reported only the count, forcing a re-read of the file to
find the occurrences before retrying. The error now appends up to 5
'L<line>: <snippet>' rows (80-char cap per snippet, overflow noted),
so the model can disambiguate in ONE follow-up — add neighboring
context or choose replace_all — without the intermediate read.
Applies to both patch modes (replace + V4A) since they share
fuzzy_find_and_replace.
#77128 fixed the orphaned zero-match probe but shipped no tests, so the
same class of break can recur: an early return anywhere in the rg branch
silently makes the whole steering tier unreachable.
Asserts hint wiring per search engine with the probe stubbed to a
sentinel (the probe itself needs rg, so a real-text parity assertion
fails on the grep leg for an unrelated reason), plus the negative case
and the rg newline-warning skip that the early return originally
existed to preserve.
Sabotage-verified against 794d6c434e: restoring the early return turns
4 red; a naive fix that also drops the rg newline guard turns 1 red;
attaching the hint when matches exist turns 2 red.
The top execute_code failure shapes in production (state.db mining)
are sandbox-contract confusions, not logic bugs: importing tools that
aren't in the sandbox from hermes_tools (23x in one window, incl.
importing the built-in helpers json_parse/shell_quote/retry), importing
third-party packages absent from the sandbox interpreter (matplotlib
6x), and indexing tool-result dicts as strings. The stderr traceback
alone sends models into re-diagnosis loops.
Failed scripts (exit != 0) now carry one actionable 'hint' field:
- unavailable hermes_tools import -> lists the tools that ARE
importable in this session + points to normal tool calls otherwise;
- built-in helper import -> 'no import needed, call it directly';
- ModuleNotFoundError -> 'sandbox has stdlib only; use terminal() with
the project venv for third-party packages';
- string-indexing errors -> 'tool functions return dicts, do not
json.loads them'.
Bounded 4KB stderr scan, first match wins, never raises; successful
scripts and unknown failures are untouched.
A regex \n (or a raw newline) in a search_files content pattern cannot
match in rg's default line-oriented mode. It previously either
hard-errored ('the literal "\n" is not allowed in a regex' — 17
occurrences in the production window) or, after the newline-warning
patch, returned 0 matches with an explanation — either way the model's
cross-line search intent required a manual workaround.
The rg engine now detects the pattern shape (_pattern_has_regex_newline,
already used by the warning path: odd-backslash \n escape or raw
newline; escaped \\n literals excluded) and enables -U/--multiline up
front, noting the mode switch in the result warning. Plain patterns are
untouched; the old line-oriented explanation is retained for the grep
fallback engine, which has no multiline mode.
skill_view re-sent full skill content on every call: ~286k tokens of
verbatim repeat views in a 400k-msg production window (one session
loaded the same skill 9 times), and a single repeat view of a large
skill costs ~25k tokens.
Mirrors read_file's proven unchanged-stub pattern: a per-task cache
keyed on (resolved name, file_path) with an mtime+size fingerprint of
the served file. On a repeat view of an UNCHANGED file, return a short
stub pointing at the earlier result. This does NOT violate the
skills-are-loaded-fully rule — the stub only ever replaces content
that is already fully present earlier in the same conversation, and:
- any on-disk change (patch, external edit) invalidates the entry;
- context compression clears the cache (wired next to
reset_file_dedup in conversation_compression.py) so post-compression
re-views return full content;
- setup-needed views are never deduped (readiness can change without
the file changing);
- no task_id -> no dedup; caches are task-isolated; 200-entry cap.
Live E2E: repeat view of hermes-agent-dev 99,739 chars -> 374-char
stub.
write_file confirmed only SIZE (wc -c) after writing — never content.
Models compensated by re-reading files immediately after writing them
(154 verify-reads in a 400k-msg production window), and a corrupted
write (truncated pipe, backend FS oddity) could silently pass.
The write path now compares the on-disk sha256 against the intended
content (one shell call). Three outcomes:
- match -> result carries verified: true; the schema tells the model
an explicit contract: do NOT re-read to check the write landed.
- mismatch -> hard error ('The write did not persist correctly'),
mirroring patch_replace's existing post-write verification.
- backend can't hash (no sha256sum) -> flag omitted, write unaffected.
Hashes the shim-adjusted content (after CRLF/BOM preservation) so
Windows-line-ending and BOM round-trips verify correctly; surrogatepass
encoding matches the rest of the codebase's hashing of model text.
When old_string survives all 9 fuzzy strategies without a match but
the closest candidate line matches after stripping whitespace, the
failure is whitespace-shaped (tabs vs spaces, indent depth). The
did-you-mean hint now appends a two-line diagnosis with leading
whitespace made visible:
Whitespace difference detected (→ = tab, · = space):
file has: →def start(self):
you sent: ····def start(self):
Use the exact whitespace shown in 'file has'.
Pattern ported from crush's diagnoseMismatch (agent-codebase survey) —
it converts the residual dead-end error into a one-turn fix. Only the
leading run is visualized (interior spacing stays readable); content-
shaped misses and raw-exact candidates are unchanged.
Follow-up to the recovery-recipe commit on this branch, per review:
instead of only TELLING the model to re-author the payload via
write_file (2 turns), materialize the blocked command to
~/.hermes/cache/blocked-scripts/blocked-*.sh and point the recovery
at it directly: 'saved to <path> - review it, then run
terminal(command="bash <path>")' (1 turn).
Safety posture is unchanged or better:
- Nothing is executed here; the file is only written.
- The bash <path> follow-up goes through the normal execution
pipeline, including the referenced-script content guard, which
inspects script files named in commands - the payload is MORE
visible to policy than it was inline.
- Genuine hardline blocks (destructive ops) never save anything
(test-asserted).
- Save failures fall back to the previous manual write_file recipe.
- 7-day opportunistic cleanup of saved payloads.
Two block classes from production mining (250k-call window) that
models answered with blind rephrase-retries:
1. Parser-limit / malformed-payload hardline blocks (198x): these fire
on oversized inline payloads (heredocs, giant one-liners), not on a
forbidden operation - but the message read like a permanent ban.
The block now appends: 'RECOVERY: ... write the script to a file
with write_file, then run bash /path/script.sh - do not retry
inline.' Genuine hardline blocks (destructive filesystem
operations) are unchanged.
2. Backgrounding-wrapper blocks (200x): the guidance now spells out
the exact corrected call shape - 're-send WITHOUT the wrapper as
terminal(command="<cmd>", background=true,
notify_on_complete=true)' - instead of describing the feature
abstractly.
process(action='wait') hitting its window returned status='timeout'
with a terse note — models read it as an error and re-issued identical
waits (process is the #1 exact-duplicate tool call in production: 511
dupes in a 400k-msg window; wait is 57% of all process actions).
The timeout result now carries:
- process_running: true — machine-readable 'this is a status, not a
failure'
- an explicit note: 'Wait window of Ns elapsed — the process is still
running. This is not an error. Uptime: Ms.' plus the right next step:
when notify_on_complete is set, 'you will be notified on exit — do
more work instead of waiting again'; otherwise a pointer to
notify_on_complete for next time.
- the clamp note (requested > max) now composes with the status note
instead of replacing it.
Exited/interrupted results are unchanged.
Third zero-match steering tier: when a content search finds nothing in
visible files, probe once with rg --hidden --no-ignore --count-matches.
If the pattern exists only in dotdirs or gitignored files, the result
says so and tells the model to search the hidden path explicitly.
Found live by the benchmark battery: a task with a match inside
.hidden/ returned a bare 0 and the model missed the file entirely in
2/3 baseline runs.
Two dead-turn classes from production mining (state.db, recent window):
1. 13.9% of 19.6k content searches return 0 matches with no steering.
Now a 0-match content search runs one cheap rg -i --count-matches
probe (plus an rg -F probe when the pattern has regex metachars) and
attaches what it found: '0 exact matches, but N case-insensitive
matches — casing may be wrong' / 'N literal matches — metacharacters
need escaping'. True zero-match results stay clean (no noise).
2. 122 'Path not found' failures came from models passing several paths
in ONE path string ('dir1 dir2 dir3', comma lists). Instead of
failing wholesale, split the string, search every path that exists,
merge results, and report skipped parts in a warning. Single-path
misses keep the existing Similar-paths hint; all-missing multi-path
strings still error.
Both probes are bounded (count-only rg, 30s timeout, max 2 invocations)
and wrapped so a probe failure can never break the search result.
Two bugs reported by gfdsa (PR #41711 comment, Jul 12) that break the
official a2a-sdk 1.1.0 Python client:
1. Task objects serialized non-spec createdAt/lastModified fields.
The A2A v1.0 Task proto (lf.a2a.v1.Task) only has id, contextId,
status, artifacts, history, metadata. Strict ProtoJSON parsers
reject unknown fields with ParseError. Removed both fields from
build_task(); created_at param kept for call-site compatibility.
2. SSE streaming frames were not JSON-RPC wrapped. A2A v1.0 §9.4
requires data: {"jsonrpc":"2.0","id":...,"result":{StreamResponse}}.
sse_data() now accepts req_id and wraps in JSON-RPC envelope.
sse_done() changed from 'data: {}' to SSE comment ': done' so
SDK doesn't try to parse an empty JSON-RPC response.
All call sites in adapter.py (_emit_terminal, _rpc_message_stream,
_rpc_tasks_subscribe) updated to thread req_id through.
Tests updated: 153 pass (151 unit + 17 integration, including 2 new
tests for JSON-RPC envelope wrapping and fallback behavior).
Refs: gfdsa/a2a-hermes reproduction repo
## File/data Parts (v1.0 unified Part)
- file_part(url=, raw=, filename=, media_type=) builds v1.0 file Parts
- data_part(data, media_type=) builds v1.0 data Parts
- message_with_parts(role, parts, context_id=) builds Messages with mixed Part types
- extract_text now renders file/data Parts into the text stream:
- File with URL: '[file: name] https://url (mediaType)'
- File with raw: '[file: name] N bytes base64-encoded (mediaType)'
- Data: '[data (mediaType)]\n{json}'
- v0.3 file (file.fileWithUri) and data (kind=data) still accepted
- Outbound replies stay text-only (agent produces text)
## Push notification config full CRUD
- get_push_config(task_id, config_id) — retrieve by task, optionally by configId
- list_push_configs(task_id) — list all configs for a task (max 1 per task)
- delete_push_config(task_id, config_id) — remove a config
- New JSON-RPC methods: tasks/pushNotificationConfig/get, /list, /delete
- New adapter handlers: _rpc_push_config_get, _list, _delete
- All return spec-shaped PushNotificationConfig with configId + createdAt
## Tests
- 6 new unit tests for Part builders + extract_text with file/data
- 13 new unit tests for push config get/list/delete (happy + error paths)
- 2 new integration tests over real HTTP:
- test_mixed_parts_delivered_to_agent: file URL + data JSON reach agent
- test_push_config_crud_over_http: full create→get→list→delete cycle
- Old test_extract_text_skips_non_text_parts replaced (now renders, not skips)
Total: 151 tests (134 unit + 17 integration), 0 failed.
DESIGN.md updated: file/data Parts and push config CRUD removed from
out-of-scope list.
Consolidates 5 follow-up PRs onto the a2a-work branch:
1. Reply-capture fix (#56437): adapter.send() now only resolves the
blocked RPC Future when metadata['notify'] is True (the gateway's
final-reply marker). Interim sends no longer short-circuit the
response. Also accepts **kwargs in connect() for reconnect compat.
2. Slash command passthrough (#53743): wrap_inbound() passes /-prefixed
text through unwrapped so the gateway command processor sees it.
Fixes /sethome deadlock during A2A onboarding. Documented security
trade-off (bearer auth at network layer compensates).
3. Routable URL in Agent Card (#53736): _build_card() now derives URL
from A2A_PUBLIC_URL env > X-Forwarded-Host/Host header > bind host.
Fixes k8s bug where Agent Card advertised 0.0.0.0.
4. contextId multi-turn memory (#53756): _handle_inbound_task() now
checks top-level params.contextId first (A2A spec), falls back to
params.message.contextId (legacy). Outbound a2a_call also sends
contextId at both top-level and inside message.
5. Type checker fixes (#53759): TypedDict for _SCHEMAS, _FunctionSchema,
_ToolSchema. Removes str() band-aid casts.
All 45 tests pass including new tests for each fix.
Zero core files modified — only plugins/platforms/a2a/ and tests/.
Credits: @davidrobertson (#56437), @knoal (#53736, #53743, #53756,
#53759), @kuangmi-bit (slash command bug report), @gfdsa (k8s URL bug
report), @shivasymbl (#45996 userContext OBO).
Live Tier-3 testing (CLI agent -> a2a tools -> live peer gateway -> model)
surfaced two bugs the kwarg-style unit tests masked:
1. registry.dispatch calls handlers as handler(args, **kwargs) — args is the
whole dict positional. The handlers used keyword params (url=, agent=), so
the dict bound to the first param and .strip() raised
'dict object has no attribute strip'. Rewrote all three handlers to take
args: dict (matching the spotify/google_meet convention). Added a
registry-dispatch regression test that exercises the real call path the
direct-kwarg tests never hit.
2. The model repeatedly reached for agent_name= instead of agent= (6 retries
before success). Accept agent_name/name and message/text/task aliases so a
reasonable guess succeeds first try.
Verified live: client agent discovers the peer's Agent Card, calls it, and
gets the reply back (PONG round-trip confirmed on both client audit log and
peer conversation log). 39 plugin tests pass.
Single platform-adapter plugin under plugins/platforms/a2a/ — zero core
edits — that supersedes the entire A2A PR/issue cluster. Built on the
ctx.register_platform + ctx.register_tool surface the codebase now exposes.
Outbound (a2a toolset): a2a_discover / a2a_call / a2a_list let the agent
call any A2A-compliant peer over JSON-RPC message/send. Inbound (platform
adapter): a stdlib http.server serves an Agent Card at
/.well-known/agent.json and routes incoming tasks into the agent's LIVE
gateway session (the #11025 insight) — same agent, full memory — returning
the reply over A2A.
Security on by default: no bearer token => 127.0.0.1-only bind; constant-
time bearer auth; inbound prompt-injection filtering + untrusted-peer
framing; outbound credential redaction; append-only audit log; per-context
conversation persistence outside the compaction pipeline.
Stdlib only (no a2a-sdk). 37 tests incl. a live HTTP round-trip
(card + message/send + reply) and a bearer-auth 401 path.
Production mining (state.db, 400k-msg window): 60.2% of 104k terminal
calls carry a defensive 'cd X && ' prefix (~925k tokens of pure prefix)
and 2,462 failed calls led with cd — the model cannot see cwd state, so
it re-asserts it on every call and runs pwd/ls diagnostics after
directory changes.
The result dict now includes a 'cwd' field whenever the session cwd
after the command differs from the cwd it started in (cd, pushd,
chained cd). Stable-cwd commands are unchanged (no field, no noise).
Per-command workdir overrides stay transient by contract and never
echo. Schema note added so models learn to trust session cwd instead
of prefixing. Pattern borrowed from crush's <cwd> injection.
realpath comparison avoids false echoes through symlinks; the echo is
wrapped defensively so a backend without .cwd can never break the
result path.
The #1 patch failure class in production (state.db mining, 250k-window)
is a re-send of an edit that already landed: 'old_string and new_string
are identical' (299 occurrences) plus a share of hunk-not-found errors
where the new text is already in the file. These errored, sending
models into re-read/re-patch loops.
New tools/fuzzy_match.is_already_applied(content, old, new) — a
conservative check requiring (1) non-trivial new_string (>=8 chars),
(2) EXACT presence of new_string, (3) old_string gone (unless
identical). Wired into three sites:
- patch_replace (replace mode): returns success + no_change: true +
an explicit note instead of the identical-strings / no-match error.
- V4A validation phase: an already-applied hunk validates as a no-op
so multi-hunk patches no longer fail wholesale when one hunk landed
in a prior call.
- V4A apply phase: mirrors the same skip so the two phases agree.
Genuine no-matches (new text absent) and half-applied renames (old
text still present) keep their error behavior — covered by tests.
Production mining (state.db, 28.5k read_file calls in the recent
window) shows 74.3% of reads truncated — nearly all by the 500-line
default, not the char budget (22,443 line-limit vs 13 char-budget
truncations). That churn produced 12,229 redundant re-reads, and after
a truncated result the most common next move was fleeing to terminal
cat/sed (4,608 times) — the pagination contract was not trusted.
Median truncated file is 2,422 total lines, so a 2000-line default
makes 44% of today's truncated first-reads complete in one call while
the unchanged ~100K-char budget still caps worst-case result size
(same ceiling as before: 500 lines x 2000-char line cap = the same
100K). Schema max was already 2000.
Touchpoints: DEFAULT_READ_LIMIT + both read_file signatures
(file_operations.py), read_file_tool + schema text (file_tools.py),
execute_code sandbox stub docs (code_execution_tool.py), 3 tests
pinning the old default.
Builds on the adapter list_channels() hook (cherry-picked from #43545 by
@Guoen0):
- plugins/platforms/simplex: implement list_channels() — enumerates
contacts (/contacts) and groups (/groups) over the live daemon
WebSocket into the channel directory. Returns None when the WS is
down so the directory falls back to session discovery instead of
wiping known targets.
- hermes send --list: merge configured-but-undiscovered platforms into
the listing. Previously a platform configured only via env (e.g. a
fresh SimpleX setup used for outbound sends) was silently omitted,
leaving users guessing at platform names.
- format_directory_for_display(): accept an explicit platforms view and
render empty platforms with a targeting hint instead of hiding them.
- docs: simplex hermes-send section.
Reported by Fedpostoffice on Discord (simplex missing from
hermes send --list; guessed platform names simplex-chat/simplex-relay).
When a command exits non-zero, scan the first 4KB of output for
well-known failure shapes and attach one short, actionable recovery
hint to the tool result ('hint' field):
- gh 'Unknown JSON field' (9.2k occurrences in a 250k-call window)
- git merge conflicts (1.2k) — stop verbatim retries
- command not found (1.0k), incl. python->python3 and pip->pip3
- ModuleNotFoundError (739) — venv activation guidance
- 'already exists' (633), gh rate limits (133), permission denied
- exit-code-only tier: 124 timeout, 126 not-executable, 137 SIGKILL
Hints are suppressed when the existing exit_code_meaning tier already
explains the code (grep=1 etc). Pattern order = production frequency
from state.db mining; first match wins; pure function, no I/O.
The delivery worker is a daemon thread, so a short-lived process
(hermes chat -q, a cron session) could exit right after firing
on_session_end — silently dropping the headline event. Register a
bounded atexit flush (5s) when the worker starts: a dead endpoint can
delay exit slightly, never hang it.
Live-verified: hermes chat -q now delivers on_session_start,
post_tool_call, and on_session_end to a real receiver; regression test
runs a subprocess that exits without flushing (sabotage-verified).
- delivery_id is now generated once per firing and used for both the
X-Hermes-Delivery header and the signed body's delivery_id field —
previously they were two different uuid4s, breaking receiver-side
dedupe as documented.
- 3xx responses are no longer followed: urllib's default redirect
handler converts a redirected POST into a body-less GET, silently
dropping the signed payload. Redirects now log a misconfiguration
warning and count as delivery failure (no retry).
- Docs: receiver-side replay-protection guidance (dedupe on
delivery_id, timestamp freshness window) + redirect semantics.
- Tests: 5xx retry count, redirect-not-followed (sabotage-verified),
header/body delivery_id equality.
The inverse of the inbound webhook platform: hooks.outbound in
config.yaml lists HTTP targets + the plugin-hook events they subscribe
to (on_session_end, subagent_stop, post_tool_call, ...). Each firing
POSTs a JSON payload (same top-level shape as shell hooks' stdin wire)
signed GitHub-style with HMAC-SHA256 (X-Hermes-Signature-256).
Rides the existing hook bus — notify-only callbacks registered on the
plugin manager at the same CLI/gateway/main entry points as shell
hooks. Delivery is fire-and-forget via a bounded queue + single daemon
worker thread, so a dead endpoint can never stall a tool call. Bounded
retries (5xx/conn errors once; 4xx never). secret_env preferred over
inline secret. HERMES_SAFE_MODE skips registration. hermes hooks list
shows outbound targets with signed/UNSIGNED status.
Zero new model tools, zero new subsystems.
Answers and written deliverables that rest on retrieved information now get
inline numbered citations plus a mechanically-rendered Sources list, with a
persistent ledger that makes a hallucinated citation detectable.
- skills/research/grounded-citations/scripts/sources.py: stdlib citation
ledger (add/ingest/list/render/verify) at
$HERMES_HOME/cache/citations/ledger.json, profile-aware, O_EXCL-locked so
parallel subagents sharing a ledger can't collide on ids
- SKILL.md: cite-while-drafting procedure, register-at-retrieval rule,
pitfalls, verification gate
- references/citation-formats.md: per-target placement (markdown, LaTeX/PDF,
docx, pptx, xlsx, wiki, BibTeX handoff to research-paper-writing)
- references/grounding-rationale.md: why numbered ids (ALCE 2305.14627,
WebGPT 2112.09332, Perplexity marker conventions), and how this relates to
the in-process registry in PR #44833
- tests/skills/test_grounded_citations_skill.py: 30 tests
The LRU-cap test in tests/tui_gateway/test_protocol.py stubs
_close_session_by_id with a two-arg lambda; the reaper now passes a
revalidation predicate. Widen the stub signature.
The Discord semantic thread-rename lane resolved the target thread from
`_relay_auto_thread_info`, which read a single-slot-per-parent-chat cache
(`adapter._auto_thread_by_chat[chat_id]`, populated from connector
SendResult feedback). When two auto-threads spawned from the SAME parent
channel, the second send overwrote the first's slot and the title turn's
read raced the write — so only the FIRST thread in a channel ever got its
semantic rename. Staging repro 2026-08-02: message A's thread renamed to
"A Hundred Word Sword Story", sibling message B's thread stayed stuck at
the raw first-words name.
The connector now stamps `prospective_thread_id` on the inbound (the anchor
message id, which is the id of the thread it will auto-create) — shipped for
per-thread session keying. Reuse it here: it is deterministic and
per-message, so it names the EXACT thread even when several auto-threads
share one channel. `_relay_auto_thread_info` returns it directly (with an
empty initial-name marker) and never consults the collision-prone per-chat
cache; the connector's own created-name guard (`prefer_connector_created`)
still enforces no-clobber, so no initial name is needed gateway-side. The
send-result cache path stays as a fallback for older connectors that don't
stamp the field.
Tests: two new cases in test_relay_threads.py — prospective id wins over a
poisoned cache entry, and two sibling threads in one channel each rename to
their own thread id. Full gateway session + relay suites green (211 passed).
- Guard the thread-id-as-chat_id normalization to Discord only; Slack
and Telegram adapters use parent_channel as chat_id for thread messages,
so the unconditional version broke their handoff keys.
- Apply the same Discord-specific guard to _seed_cron_thread_session in
cron/scheduler.py (sibling site with the same bug, docstring said
'Mirrors _process_handoff').
- Replace the change-detector test with contract tests that verify the
actual invariant: Discord handoff key == organic thread key, Slack
handoff key still uses parent channel (non-regression).
A CLI→Discord handoff creates a dedicated thread and re-binds the CLI
session to it. It built the destination SessionSource with
chat_id = home.chat_id (the PARENT channel) while marking it
chat_type="thread" with thread_id set.
But platform adapters build organic in-thread messages with
chat_id = <thread id> (see the Discord adapter's on_message and
_build_thread_event paths). build_session_key therefore produced two
different keys for the same thread:
handoff: agent:main:<platform>🧵{parent}:{thread}
organic: agent:main:<platform>🧵{thread}:{thread}
So the next real user reply in the handoff thread resolved to a
DIFFERENT session_key and spawned a fresh session instead of continuing
the handed-off one — observed as a stray auto-titled session plus a
session_search fallback (the new session had no prior context).
Fix: for a thread destination, key on the thread's own id so the
synthetic handoff turn and later user replies share one session_key,
matching how adapters key organic in-thread messages.
Adds tests/gateway/test_handoff_thread_session_key.py, which asserts the
handoff key is byte-identical to the organic in-thread key (fails on the
old parent-channel keying, passes on the fix).
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
Completes the cache-parity bug class from #76938: the parent's request
body diverges from the fork's not only at the ephemeral system prompt
but also at prefill messages (inserted right after the system message
at API-call time) and, on OpenRouter, at upstream-provider selection
(prompt caches live per upstream; an unpinned fork can be routed to a
different upstream and miss a byte-identical prefix).
Also hardens the tests: pairwise asserts instead of re-implementing the
production prompt join, and routed-path omission guards for the whole
gated kwarg family.
_format_price_per_mtok collapsed any per-Mtok price below one cent to
"$0.00" (and readers treated near-zero as free). Nous Portal's DeepSeek
V4 Flash 0731 promo prices cache hits at $0.0018/Mtok, which rendered as
free in the /model picker and hermes model listings.
Prices under $0.01/Mtok now widen precision to the first significant
digit plus one, with trailing zeros trimmed: 0.0000000018/tok →
$0.0018. Standard prices keep the aligned two-decimal format; exact
zero still renders as "free".
pcm_to_wav staged every captured utterance in a NamedTemporaryFile just to
hand ffmpeg an input path, then unlinked it. Feed the PCM to ffmpeg's stdin
instead: one fewer file created, written, read back and removed per voice
utterance, and the try/finally cleanup goes away with it.
The WAV output deliberately still goes to output_path rather than being
captured from stdout. ffmpeg cannot seek on a pipe, so a piped WAV is
written with placeholder 0xFFFFFFFF RIFF/data chunk sizes -- Python's wave
module then reports 2147483647 frames for a 1s clip, and strict readers
misjudge the length. Writing to the real path lets ffmpeg seek back and
patch the header.
Tests cover both halves: that the PCM goes over stdin with no temp file,
and (when ffmpeg is installed) that the resulting header reports the true
frame count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Pass extra_exclude={pid} to _reap_unsupervised_gateway_orphans so the
killed PID isn't double-killed during the sweep (#75936).
- Add extra_exclude param to _reap_unsupervised_gateway_orphans signature.
- Replace bare except:pass with logger.debug for diagnosability.
- Fix existing test (mock _reap_unsupervised_gateway_orphans so it
doesn't scan real processes and trigger conftest live-system guard).
- Add regression test asserting the killed PID is excluded from the sweep.
_handle_cron_fire verified the NAS-minted fire JWT by calling the
fire-verifier inline on the event loop. That verifier resolves the NAS
signing key from a JWKS URL — a synchronous HTTP GET on a cache miss (a
cold PyJWKClient, or a rotated kid the cached client doesn't know) — so a
slow or rate-limited portal stalls the whole event loop and starves every
other adapter sharing it. #64641 already documented this exact symptom
(relay 504s on high-job-count instances) and cut the fetch frequency by
caching the client per URL, but the residual cache-miss fetch still ran
inline on the loop.
Dispatch the verifier the same way the platform HTTP event verifier was
hardened: await a coroutine verifier directly, run a sync one via
asyncio.to_thread so its blocking I/O stays off the loop, and fail closed
(reject with 401, never admit the fire) if the verifier raises — this is
the only inbound that can trigger remote job execution. The verifier's
JWK-client cache is already thread-safe (threading.Lock), so moving the
call to a worker thread is safe.
Adds regression tests: a sync verifier runs on a worker thread rather
than the loop thread, a crashing verifier yields 401 with no fire, and a
coroutine verifier is awaited.
/api/status is the desktop's boot liveness probe (polled ~1/s) but since
#60537 every call ran a full topology scan — per-profile yaml.safe_load
(pure-Python loader), psutil process probes, realpath walks — in the
default executor. On multi-profile installs concurrent polls pile up and
hold the GIL 14-16s, starving the event loop: the WS sidecar cannot
flush gateway.ready, the desktop times out into the next stall, and boot
escalates to the 'Hermes couldn't start' overlay (#60800).
Memoize the scan behind a 10s TTL with a collapse lock so concurrent
polls share one scan. Topology only changes on gateway start/stop, so a
<=10s stale badge is an acceptable trade for not starving the loop. The
cache also keys on the collector's identity: tests monkeypatch
_collect_profile_gateway_topology per case, and the identity check keeps
them hermetic (a swapped collector is a miss) without a reset hook.
py-spy captures during a failing boot land in _profile_platform_ports ->
yaml.safe_load on executor threads (7 profiles, Windows). After: one
cold-start scan, zero recurring stalls, desktop boots.
- Short-circuit the candidate waterfall on HTTP 401/403: an auth wall
proves the endpoint family exists, so probing the alternate URL just
doubles the wasted wait (the reported endpoint takes ~10s to return
401 without a key).
- Stream the probe so 4xx never downloads a slow error body; responses
are closed on every exit path.
- Regression tests: single-call assertion on 401/403 (fails on main),
negative-cache reuse, 404 waterfall preserved, no .json() on 4xx.
Fixes#69905
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-ups on the #25387 salvage:
1. CRITICAL: a cached miss survived out-of-band file creation (terminal
command, external process) for the full 60s TTL — breaking the common
agent pattern 'check for file -> create it -> read it' (live-repro'd).
Serve-side existence guard: one ~free stat before serving a cached
miss; if the path now exists the entry is evicted and the real read
runs. Also fixes the search-root variant (write under a cached-missing
directory). Both mutation-checked.
2. notify_other_tool_call now clears the task's not_found entries too
(belt: the dispatcher calls it for every non-read tool).
3. Tracking parity: the record sites no longer early-return. On upstream,
error results flow through consecutive-loop detection and dedup
bookkeeping; short-circuiting skipped that and broke
TestDedupInvalidationTaskResolution when preceded by
TestSilentFileMisplacementE2E (bisected: the early return at the
read record site was the trigger). Recording is now
side-effect-identical to upstream; serving from the cache remains the
optimization. Also reuse the already-computed _resolved instead of
resolving a second time.
kshitijk4poor
kshitij
jasoisjaso
PRATHAMESH75
skywind
baau
Rod Boev
Georgio Constantinou
Kyzcreig
Zeheng Huang
arimu1
szzhoujiarui
wangyunyou
JonthanaHanh
Pooyan6
liuhao1024
f1aggo_macair
Xue-1997
Daniel Blank
Teknium
ibaldr89
kshitij
Mahdi Hedhli
Coffee☕️
CriptoGus
ckaznocha
LFDM Core
briandevans
HexLab98
Eugeniusz Gilewski
joaomarcos
Alex Fournier
Jiahui-Gu
fangliquanflq
fangliquanflq
null-runner
Ben Kamholtz
Kevin (OpenClaw Bot)
David Robertson
Guoen
xrazai
Ben Barclay
kshitij
Alejandro Moreno
Quark Assistant
Jesse Casco
wayne1992127
Frowtek
lost9999
Josh Tsai