Both sidecars answered the same question in their own way. Which
directory does this Node child run from, when some installs put the
source tree somewhere nothing can write?
gateway/sidecar_runtime.py answers it once. Four rungs:
1. An operator override.
2. A writable source.
3. A read-only source whose baked deps match the lockfile.
4. A read-only source that must move to $HERMES_HOME/sidecars/<name>
before npm can run.
Node sets the shape of that last rung. Its ESM resolver reads
node_modules only from the directories above the importing file, and
NODE_PATH applies to CommonJS alone. Measured on Node 26: an ESM import
with NODE_PATH pointing at the packages fails, and the same import from
a directory beside them works. Both sidecars are "type": "module", so
the entry file and the packages must share a tree. A copy is the only
arrangement Node accepts.
_MIRROR_FILES is gone. It named the files to copy, so it had to name
every module the entry file imports. It was wrong twice. It listed the
deleted spectrum patch, and it omitted send-format.mjs and
stream-staleness.mjs. Both faults appear only on a read-only install.
The resolver copies the tree instead, without node_modules, and the test
compares the mirror against the source tree rather than against a second
list. A mutation that returns to a fixed list fails it.
The copy uses shutil.copy, not copy2. copy2 gives the mirror file the
mtime of the source, and a Nix store source has mtime = epoch. A
refreshed lockfile then always predates npm's install marker, and
deps_are_current() keeps stale node_modules through every upgrade,
which is the fault this resolver exists to fix. A plain copy stamps
the copy time, so a content change always postdates the previous
install. npm's hidden node_modules/.package-lock.json cannot replace
the content comparison: it is a different document, and npm matches it
semantically, not byte for byte.
WhatsApp gains what it never had: a staleness check and a refresh. Its
resolver returned any existing mirror without comparing it against the
lockfile, so an upgrade kept the old node_modules. This is a behaviour
change.
The mirrors move to $HERMES_HOME/sidecars/. The Baileys credentials live
in $HERMES_HOME/whatsapp/session, so a paired account is not affected.
`hermes doctor` reports the mirrors the old resolvers left at
$HERMES_HOME/photon/sidecar and $HERMES_HOME/scripts/whatsapp-bridge.
Nothing reads them now, and each one can hold a node_modules of some
hundred MB. --fix removes them.
_sidecar_deps_stale and deps_are_current read the same two files with
opposite missing-file answers, on purpose. Each one points at the other
and says why.
The container bakes both sidecars now. It baked Photon and left WhatsApp
to install at run time.
Four majors, 8.0.0 to 12.7.0.
The mixed text and attachment patch is gone, because upstream does the
work now. Hermes carried patch-spectrum-mixed-attachments.mjs to rewrite
the compiled iMessage mappers. A bubble with text and an attachment
returned only the attachment. The typed text never reached the agent.
spectrum-ts 12 builds the parts with toOrderedParts(text, attachments),
and it reads better than the patch did. The patch always put the text
first. Upstream splits on the object replacement character that Apple
writes at each attachment position, so the parts keep the order the
sender typed. Ran the real mapper against four shapes: text with one
attachment, text between two attachments, an attachment alone, and text
alone. The text survives in each.
The patch anchors do not match 12.7.0 in any case. The first one fails
with "expected exactly one rebuild text capture match, found 0".
Removed with it:
- The postinstall hook.
- The call in index.mjs that ran the patch on each start, and refused
to start when it threw.
- The spawn in adapter.py. It ran node and waited up to 10s on every
_start_sidecar, which includes every reconnect.
- The copy in the Dockerfile, and test_spectrum_patch.py.
Confirmed the sidecar reaches the Photon API on 12.7.0. With test
credentials it stops at the same SpectrumCloudError 422 as 8.0.0, from
the same call, so only the credentials are wrong. Each symbol index.mjs
imports still resolves.
tools/lazy_deps.py held a table of about 40 features, each with its own
literal pip specs. pyproject.toml declares the same packages as extras,
so every pin existed twice and the two copies drifted.
Each feature now names an extra, and the specs come from pyproject at
run time. The table is 218 lines shorter. A test asserts that each
feature names an extra that exists and resolves to at least one spec, so
a typo cannot ship.
A wheel install, such as Nix, has no pyproject.toml beside the code.
There the same table comes from the dist metadata: each spec of an
extra is one Requires-Dist line, and its marker names the extra.
Without this fallback, each entry point raised on a Nix install, and
ensure() raised even for a feature whose packages the build baked in
through extraDependencyGroups. That call must be a no-op.
is_available() and feature_install_command() catch the failure as well
now. Their callers sit in status paths with no try/except, and their
contracts are bool and Optional[str].
The security overrides already come from pyproject (the previous
commit). This commit moves the reader onto the shared _pyproject()
cache and the shared temp-file writer.
The tier-0 installer, `uv sync --extra <name>`, names the project with
--project. uv reads the project from its working directory, and the
agent runs from the user's working directory, not from the install
tree. Without the flag the sync failed outside a checkout, and the pip
ladder always ran instead.
install_specs gets the same managed-install guard as ensure(). A Nix
venv is in the read-only store, so the pip ladder could only fail with
EROFS after a 15s ensurepip attempt. It reports the Nix remedy instead.
A durable install target overrides the guard, as it does in ensure(),
because the NixOS container module sets HERMES_MANAGED=true with a
writable target.
Spec parsing goes to packaging.requirements.Requirement, which is
already a core dependency. The hand-written version kept the
environment marker attached to the version. SpecifierSet raised on it,
so _is_satisfied answered True for every installed version of a marked
package. Such a package can never upgrade.
Reading the specs from an extra exposed a second fault, in the record of
which features are active. active_features read specs[0] as the anchor
package, and extra composition put sounddevice there for [voice] and for
each wake extra. One local STT install then marked every audio feature
active, and `hermes update` installed the wake engines that the user
never enabled.
ensure() records each feature it serves in
$HERMES_HOME/lazy-features.json, and active_features reads that record.
A recorded feature still needs its anchor package installed, so an
uninstalled backend does not come back. The anchor is the first pin
written directly in the extra, not the first spec after expansion. A
test asserts that no two extras share an anchor.
There is no seeding for an install that predates the record. Its first
`hermes update` refreshes nothing. ensure() then repairs a stale pin at
each backend's start and records the feature, and the next update covers
it.
[stt-whisper] splits out of [voice]. faster-whisper transcribes audio
files and needs no microphone and no PortAudio, so the Docker image can
bake it. [voice] composes [stt-whisper] and [audio-io] and stays the
microphone stack. stt.faster_whisper maps to the new extra.
Removed with the table:
- The literal pin list in plugins/platforms/google_chat/oauth.py. Its
pip path targeted /nix/store on a Nix install, which is read-only.
- The bare honcho-ai fallback in the honcho setup. An unpinned install
accepts whatever PyPI serves, which is the hole this branch closes.
Both call sites report the remedy for the deployment instead, through
the now-public managed_install_reason.
- install_deps() in the google-workspace skill. The SDKs ship in the
[google] extra, so a stripped environment is a broken install. The
repair is `hermes update`. A pip run from the script writes to
whichever interpreter it runs under, which is not always the one
Hermes uses.
- tests/test_runtime_pins_are_locked.py, which scanned first-party
source for pin literals. There are none left to find.
- The spec shape check in install_specs. The same plugin.yaml hands
external_dependencies[].install to bash with shell=True, and the
plugin's __init__.py is imported. Anyone who can write that file
already runs code as the user.
`uv pip install` and `pip install` do not read [tool.uv]
override-dependencies from pyproject.toml. A backend whose transitive
deps cap a security-pinned package below its patched floor therefore
downgrades the core venv the first time that backend is enabled.
The measured case: the core venv ships cryptography 50.0.0. The first
DingTalk install pulls alibabacloud-tea-openapi 0.4.5, which caps
cryptography<49, and the resolver moves cryptography back to 48.0.1 —
with its three advisories. Pinning the floor next to the specs is not
a fix: the resolver satisfies it by walking tea-openapi back to
0.3.16, a two-year-old sdist build, and pinning both is unsatisfiable.
tools/lazy_deps.py now reads override-dependencies from pyproject.toml
and hands the list to both installer tiers: uv gets it as --overrides,
pip gets it as --constraint. pyproject.toml is the one source of
truth, so there is no second list to keep in sync. Lazy installs only
run from a source checkout — the one wheel-shaped install, Nix, seals
its venv and cannot lazy-install — so the file is always on disk.
This also covers the pynacl override: a lazy discord.py install caps
pynacl below the patched 1.6 floor, and would move the core venv back
to 1.5.0.
New tests hold the contract: the reader returns the pyproject list
verbatim, and both installer tiers receive it.
cryptography 48.0.1 carries three advisories (GHSA-m2h6-j472-rp4c,
GHSA-jwv3-5hgf-82ww, CVE-2026-69247). msal and alibabacloud-tea-openapi
cap cryptography below 49, so the bump needs an override-dependencies
entry in [tool.uv] to take effect.
The cap is conservative, not a real limit: we installed tea-openapi
against cryptography 50 and its client ran with no errors.
This override only governs `uv lock` / `uv sync`. The lazy-install
path does not read [tool.uv] and can still downgrade the pin; the next
commit closes that path.
aiohttp moves to 3.14.3 in the same pass, for GHSA-9548-qrrj-x5pj.
When the per-session pending transcript queue hits _MAX_PENDING_PER_SESSION
(200) while the session DB is broken, the gateway previously popped the
oldest message and discarded it permanently — silent user data loss during
live operation (#78182). The on-disk pending spool only ran at shutdown via
flush_pending_to_file.
Extend that existing spool machinery for runtime drops:
- gateway/shutdown_flush.py: add spool_dropped_transcript_message() and
drain_transcript_spool(), reusing _get_flush_dir/_write_payload (same
atomic-JSON pending_messages/ spool format). recover_pending_to_db()
now also replays transcript_cap_drop payloads left over across restarts.
- gateway/session.py: on cap eviction, spool the dropped message and log a
WARNING that includes the spool path; if spooling fails, degrade to the
previous drop-and-warn behavior. On the next fully successful transcript
flush for that session, drain and replay spooled messages in drop order;
replay failures keep the spool files for the next attempt.
- tests/gateway/test_pending_queue_spool.py: drop→spool→drain roundtrip,
per-session drain isolation, spool-failure degradation, replay-failure
retention, and spool primitive ordering/reason filtering.
No new config; extends existing flush_pending_to_file infrastructure per
AGENTS.md guidance.
Refs #82616, #78182
When state.db's write path fails (corrupt FTS, or a crash landing between
routing publication and row creation), the live gateway conversation can end
up in a session row that never received its identity columns: session_key,
chat_id, chat_type and origin_json are all NULL. In-memory routing hides the
damage for as long as the gateway stays up. After a restart the chat is
resolved from the DB, and find_latest_gateway_session_for_peer cannot see
that row — both of its queries match on the very columns it lacks — so the
chat resumes the last keyed sibling instead, days older. The messages were
never lost, only unreachable.
Hardening the write side cannot reach a row that is already damaged, so add
the offline repair path the tracking issue asks for:
- SessionDB.find_orphaned_gateway_sessions() reports message-bearing rows
with no session_key, and names the predecessor each one continues only
when the evidence is unambiguous — a recorded parent_session_id
("lineage"), or exactly one keyed row of the same source and compatible
user_id that fell quiet within 15 minutes of the orphan's start
("contiguity"). Contested pairs are reported with a reason and left alone:
a wrong adoption would splice one person's conversation into another
person's chat. Branch, delegate and tool rows are excluded — they are
unkeyed by design, not by damage.
- SessionDB.adopt_orphaned_gateway_session() stamps the orphan from the
predecessor (never overwriting a column that already has a value), records
the lineage, and retires the predecessor under end_reason
'superseded_by_repair' — a reason recovery does not treat as resumable, so
the repaired row wins the chat from then on. The pair is re-verified inside
the write transaction, making a concurrent heal a no-op rather than a
conflicting write.
- `hermes sessions repair-routing` drives both. It reports without touching
the database; --apply confirms first and warns that a running gateway
still holds the old mapping in memory.
Refs #82616.
tui_gateway/server.py:_save_cfg called yaml.safe_dump on a deep-loaded
config dict, which reordered top-level keys alphabetically, stripped
every user-edited comment, and re-escaped non-ASCII (kaomoji/Chinese)
personality prompts to \uXXXX. Every TUI setting change - /personality,
/reasoning, /details_mode, /skin, /prompt - rewrote the file top to
bottom.
Changes:
* Add atomic_roundtrip_yaml_save(path, new_state) in utils.py - a
comment-, ordering-, and unicode-preserving full-state replacement
for yaml.safe_dump(cfg, f). Uses ruamel round-trip mode like the
existing atomic_roundtrip_yaml_update, but accepts the whole cfg
dict so callers that mutate multiple keys before saving (the
_save_cfg pattern) don't have to be rewritten. Recurses into nested
dicts, deletes keys missing from new_state (preserves the
cfg.pop()-then-save semantic), and overwrites lists/scalars
wholesale.
* Fail closed on an unreadable existing config.yaml the same way
hermes_cli.config.atomic_config_write does, via a lazy import of
require_readable_config_before_write (avoids a module-level circular
import, since hermes_cli.config itself imports from utils). Also
preserves both file mode and owner across the write, matching the
existing atomic_roundtrip_yaml_update contract.
* Force-quote any new string value that YAML 1.1 would misparse as a
bool/null (yes/no/on/off/true/false/null/~). ruamel's round-trip
dumper resolves against the YAML 1.2 core schema and emits these
unquoted, but PyYAML-based readers elsewhere in the codebase parse
under YAML 1.1 rules - so an unquoted `approvals.mode: off` would
silently round-trip back as the boolean False.
* tui_gateway/server.py:_save_cfg now delegates to
atomic_roundtrip_yaml_save. Drop-in - all call sites (/personality,
/reasoning, /details_mode, /prompt, etc.) inherit comment
preservation and the fail-closed contract.
Tests:
* tests/test_utils_atomic_roundtrip_yaml_save.py - unit tests covering
create-from-empty, top-level key-order preservation, comment
preservation, readable Unicode, append-new-keys, delete-missing-keys,
scalar/list overwrite, nested-dict recursion, refusal on an
unreadable existing config, and owner preservation.
* tests/test_atomic_replace_symlinks.py - owner-preservation regression
test mirroring the existing atomic_roundtrip_yaml_update coverage.
* tests/test_tui_gateway_server.py - 4 new tests pinning _save_cfg
comment preservation, top-level key-order preservation, and
unicode-readability under unrelated writes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The CI run stayed in progress until its last job ended. Two advisory jobs
set that time: the review-comment poller (40 minutes) and the Docker image
build (45 minutes). Neither job was required to merge.
GitHub refuses `gh run rerun` on a run that is in progress. Thus a reviewer
who added the `ci-reviewed` label had to wait for the two slow jobs, and
label-rerun.yml carried a 2100-second wait loop for this reason. The fast
required jobs were ready long before.
Each slow job now runs in its own workflow:
- docker.yml owns its `pull_request` trigger and does its own change
detection. The new `detect` job runs the same composite action with the
same condition that ci.yml applied, so a tests-only PR still skips the
build. The `workflow_call` trigger is gone.
- ci-review-comment.yml starts on `workflow_run` when CI starts. It reads
the workflow and the scripts from the default branch, which is the trust
boundary that the old job got from its `ref: default_branch` checkout.
The poller reads job results through the API, so it can report on a run
that it does not belong to. `WATCH_WORKFLOWS` names sibling workflows for
the same commit, and `select_watched_runs` keeps the newest run for each
name. Thus the comment still shows the Docker results. The list is
newline-separated, because a workflow name can contain a comma.
The poller always exits 0 now. It reports on the CI run from a different
run, so a failed CI job is not a failure of the poller. The CI run has its
own gate for that.
Also correct a parse error in label-rerun.yml. STATUS came from the already
truncated RUN_ID, so its value was the run id and never "completed". Thus
the wait branch always ran.
ci.yml no longer needs `packages: write`, because the image build has left.
Root cause of #82616: gateway session identity (session_key/chat_id/
origin_json) was written best-effort in a separate UPDATE after row
creation, both reset-path DB writes swallowed failures silently
(logger.debug / bare print), transcript reads ignored the reroute map
that writes follow, and restart recovery ranked candidate rows by
started_at while hard-rejecting empty rows. A single failed write could
therefore strand the live conversation in an unroutable orphan row while
a days-old zombie kept the routing key — after any gateway restart the
chat silently resumed the zombie (user-visible context loss, 5 confirmed
incidents on one install since June).
Four class fixes:
1. Identity lands atomically in the session INSERT: origin_json and
display_name join _insert_session_row's column list + COALESCE
backfill; both gateway creation paths (get_or_create + reset) pass
full identity including parent_session_id lineage (fixes#12857).
2. record_gateway_session_peer self-heals: when the target row is
missing (failed/deferred create, crash window) it INSERTs the row
with full identity instead of silently no-opping — every per-turn
peer refresh is now a repair opportunity, and an identity-less lazy
writer (update_token_counts/record_auxiliary_usage) can never leave
a gateway session permanently unroutable.
3. load_transcript follows the write-side reroute chain and the durable
compression tip before querying, so reads can no longer return 0
rows for a session whose messages live under its compression child;
read exceptions are WARNING, distinguishable from an empty result.
4. find_latest_gateway_session_for_peer ranks by
COALESCE(last_activity_at, started_at) (message-bearing rows first)
and returns an empty-but-keyed row instead of None — a zombie
predecessor can no longer beat the live conversation, and recovery
never mints a fresh id when a keyed row exists.
Reset-path DB write failures now log at WARNING with the routing
consequence spelled out.
Tests: tests/gateway/test_session_continuity_82616.py (11 tests) —
sabotage-verified: 6/11 fail without the fixes. E2E incident replay
(real SessionDB, temp HERMES_HOME) confirms the production shape now
resolves to the live session.
Fixes#82616. Related: #12857, #78182 (read-path half), #79576.
Personality persistence used to be split per surface: the TUI/desktop wrote
the NAME to display.personality while the CLI/gateway wrote rendered TEXT
into agent.system_prompt (and their /personality none only blanked the
text, leaving the name behind). When #81946 made display.personality
authoritative everywhere, stale names written long ago resurrected
personalities users had turned off - kawaii defaulting on after updating.
- hermes_cli/personality.py: new single owner of personality state.
Built-in personality definitions, neutral-name normalization, rendering,
availability (built-ins overlaid by agent.personalities), overlay
resolution, and the ONLY sanctioned persistence path
(persist_personality -> display.personality; never agent.system_prompt).
- v34 config migration: one-time reset of display.personality to none
(announcing which personality was cleared and how to re-enable), plus a
scrub of agent.system_prompt when it verbatim-equals a known personality
render (machine-written by the old CLI/gateway). Hand-written manual
prompts are never touched.
- All surfaces rewired through the module: CLI /personality (incl. active
marker in the list), gateway /personality, TUI config.set + slash path
(which previously applied without persisting), TUI config.get (reports
the EFFECTIVE personality), completer, hermes config display, and the
tui_gateway health probe.
- cli.py/config duplicates removed: built-ins now defined once; the
desktop mirrors them from one lib module (src/lib/personalities.ts).
- Docs updated: selection lives in display.personality, built-ins always
available, one-time reset note.
asyncio.ensure_future(result) creates a task with only a weak ref in
the event loop's task table. After the carrier raises CancelledError,
the local 'task' variable goes out of scope and the loop can GC the
handler before it finishes — the exact 'handler killed mid-flight'
class we are fixing, just via GC instead of cancellation.
Add _detached_fatal_tasks set on BasePlatformAdapter (matching the
gateway-level pattern in _handle_adapter_fatal_error). Uses getattr
fallback for test stubs built via object.__new__().
When an adapter escalates a retryable fatal error from inside one of its
own tasks (e.g. Telegram's _polling_error_task after exhausting polling
network retries), the gateway's _handle_adapter_fatal_error tears the
adapter down via disconnect() — which cancels that very task. The
propagating CancelledError killed the handler between popping the
adapter from the adapter map and queueing the platform in
_failed_platforms, leaving a zombie gateway: process alive, zero
connected platforms, zero pending retries, until a manual restart.
Run the handler as a detached task under asyncio.shield so carrier
cancellation no longer aborts teardown/queueing mid-flight. The carrier
still observes CancelledError (teardown semantics unchanged); only the
handler is protected. A done-callback consumes the detached task's
exception to avoid 'Task exception was never retrieved' noise.
Fixes#81335
- Remove dead use_systemd_scope = False assignment (leftover from
the old try/except pattern, immediately overwritten).
- Update stale log label supervisor= -> in_supervised_gateway=
to match the renamed variable.
- Convert autouse _mark_gateway_process fixture to opt-in
_gateway_identity so negative tests start from a clean slate
instead of undoing the fixture's env/PID mocks.
- Parametrize 4 near-duplicate negative tests (2 scenarios x
pipe/PTY) into 2 parametrized tests, reducing ~130 lines to ~80.
76 tests pass, ruff clean, net -32 LOC.
The auto-resume restart-loop breaker (#30719, defense-3) pruned its boot
log against an absolute `window_seconds` (default 60s). That prune is
period-sensitive: a crash cycle slower than the window drops its own
history on every boot, so the counter never leaves 1 and the breaker can
never trip, no matter how long the loop runs.
The cycle reported in #81642 is ~150s — a wedged event loop, the liveness
watchdog hard-exiting at ~90s, a supervisor respawn, and auto-resume
replaying the same session that wedges it again. Structurally invisible to
a 60s window: `gateway/restart_loop.json` kept a single timestamp across 15
kills in one morning. Because every cycle leaves a gateway that cannot
process SIGTERM, `hermes update` has no drainable gateway to stop, which is
the reported hang.
Chain boots on the inter-boot GAP instead of an absolute window: two boots
belong to the same loop when they are no more than `max_gap_seconds` apart
(default 300s, floored by `window_seconds` so widening the window never
makes the breaker less sensitive). The verdict becomes period-agnostic —
the original ~10s respawn loop still trips in 3 boots, and so does a 150s
one — while a boot after real quiet resets the chain, so occasional
operator restarts still never accumulate. The persisted chain is capped at
50 entries.
- gateway/restart_loop_guard.py: gap-chained pruning (`_chain_ending_at`),
`DEFAULT_MAX_GAP_SECONDS`, `max_gap_seconds` kwarg on the three entry
points, clock-step tolerance, bounded state file
- gateway/run.py: `_restart_loop_guard_config` reads and returns
`max_gap_seconds`; the auto-resume call site passes it through
- hermes_cli/config_defaults.py: `gateway.restart_loop_guard.max_gap_seconds`
Tests: 7 new cases in TestRestartLoopGuard covering the slow cycle, chain
persistence, quiet-period reset, the #30719 fast loop, the config knob, the
window floor, and the disabled breaker. Verified RED before the fix (the
slow-cycle case asserted `[1300.0] == [1000.0, 1150.0, 1300.0]`, exactly
the single-timestamp state file from the report) and GREEN after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the legacy ghost-row filter from inside the api_messages loop to
BEFORE repair_message_sequence_with_cursor. Dropping a ghost assistant
row between two user messages creates user→user which the repair can
now fix (previously the repair ran first and missed it).
Promote '[This response was interrupted by a user correction.]' to
module-level _INTERRUPT_SCAFFOLD_MARKER constant — used in both
_apply_active_turn_redirect (checkpoint_parts) and the ghost filter,
so they can never drift.
Update ghost-row test: the two consecutive user messages are now
merged by repair, so check for content as substring.
Sessions already poisoned by the incomplete #73146 else branch still replay
hidden assistant rows whose content is the raw interrupt scaffold. Skip those
rows when building provider messages so old state.db history cannot keep
seeding the echo loop.
The incomplete #73146 else branch still wrote the interrupt checkpoint into
the placeholder assistant row. Mid-tool steers then replayed that scaffold as
the model's own prior reply, which it echoed into a self-replicating ghost
loop. Carry the scaffold only on the user correction's api_content, matching
the assistant-tail branch.
The warm covered session cwds, but build_tree also resolves each declared
project folder and each discovered repo root. Those were the last probes
running one directory at a time while the sidebar showed a skeleton.
The project tree probes every distinct session cwd, and on a long-lived
history most of those directories are deleted worktrees — `git -C` there
can only fail, at the price of a fork each. Stat first.
The second elision is `common_repo_root`: only repos have a common dir,
and the parallel warm never covers that probe because `resolve()` reaches
it only for cwds that already resolved. Every non-repo cwd was therefore
paying a serial `git` spawn on the discovery pass.
Fix: _LENGTH_CONTINUATION_DROPPED_TOOLS_PREFIX ended with '(' but
_get_continuation_prompt still had f'({tool_list})', producing
'((write_file)' instead of '(write_file)'. Removed the '(' from
the prefix constant — the parenthesis belongs in the interpolation.
Widened: promoted the empty-response nudge (line 6993,
'You just executed tool calls but returned an empty response...')
to _EMPTY_TOOL_RESPONSE_NUDGE constant and added it to the
classifier's recognition set. Same bug class — its
_empty_recovery_synthetic metadata flag doesn't survive SessionDB
projection either.
Test: added parametrize case for the empty-response nudge (7→8 cases).
E2E: verified byte-for-byte string equivalence for all nudge constants.
aed114a69 taught _is_synthetic_compression_user_turn to recognize the
max-iteration nudge as ephemeral runtime scaffolding rather than a human
turn, since its role="user" metadata flag doesn't survive SessionDB
projection and a crash/interrupt mid-turn can persist it durably — becoming
the compaction anchor / auto-focus topic in place of the real task.
conversation_loop.py's retry loop appends several more role="user" rows
with the exact same "ephemeral, metadata-tag-only" shape, none of them
recognized by the classifier:
- The three _get_continuation_prompt variants (length-continuation nudge,
tagged _length_continuation_nudge) — two fixed strings plus a third that
interpolates the dropped-tool-call list.
- _CODEX_INCOMPLETE_NUDGE (codex/responses reasoning-only retry).
- The codex ack-continuation nudge (acknowledgment-only reply re-prompt).
- The dropped-tool-call nudge (tagged _dropped_toolcall_nudge) — persisted
across up to 3 consecutive retries before the finalization pop-loop
strips it; an interrupt/crash before that pop can persist it same as the
max-iteration case.
Promote the previously-inline nudge strings to named module-level constants
in conversation_loop.py (single source of truth for both construction and
recognition), then extend the classifier to recognize all of them — exact
match for the five fixed-content nudges, a stable-prefix check for the
dropped-tool-call continuation variant (its tool list is interpolated so it
can't be exact-matched, same treatment TODO_INJECTION_HEADER already gets).
Imported lazily inside the classifier to avoid a module-load-order cycle —
conversation_loop.py already imports FROM context_compressor.py at call
time for the same reason.
Simplify registration/unregistration to match delegate_tool.py's
hasattr+getattr pattern instead of over-defensive try/except Exception
blocks. Delete inspect.getsource() change-detector test (breaks on
rename, proves nothing the behavioral test doesn't cover).
Net: -73 lines, +35 lines = -38 lines.
A background memory/skill review (agent/background_review.py) forks a
second, complete AIAgent in a daemon thread that deliberately shares the
live agent's own session_id for prompt-cache warmth. Nothing previously
stopped a user's next live turn from starting while that fork was still
mid-conversation, letting both stream against the same session_id and
credentials concurrently. That produced two observable failures:
- Doubled prompt-token accounting on the live turn's own calls (the two
concurrent request/response streams under one session_id confuse the
token-usage bookkeeping), triggering premature context compression.
- A lockup that a normal interrupt could not clear: the review fork is a
fully independent AIAgent with its own _interrupt_requested flag, and
was never added to the parent's _active_children list -- the only list
AIAgent.interrupt() actually walks for cross-agent cancellation -- so a
live-turn Ctrl+C had no propagation path to it at all.
Fix, three files:
1. agent/agent_init.py -- add _background_review_agent /
_background_review_lock tracking state to every AIAgent, mirroring the
existing _active_children pattern.
2. agent/background_review.py -- the review fork now registers itself on
the parent's _active_children right after construction (reusing the
same list/lock interrupt() already fans out to for real subagent
delegation), and unregisters on every exit path (success, the
tool-whitelist finally, and the outer exception safety-net). All
registration is defensive (getattr/try-except) so an AIAgent built
without going through agent_init.py's setup degrades to "no
cross-turn cancellation" instead of aborting the whole review.
3. agent/conversation_loop.py -- at the very start of every
run_conversation() turn, if a prior background review is still
in-flight, it is now proactively cancelled via interrupt() before the
live turn proceeds -- fire-and-forget, non-blocking, adds no latency.
Adds 3 regression tests to tests/run_agent/test_background_review.py,
confirmed to fail against the pre-fix code via a scripted revert.
Verified: ruff clean on all touched files; 66/66 background-review and
interrupt-propagation tests pass; 256/256 across turn_finalizer +
run_agent regression suites; no fork-only symbols in the diff.
Follow-ups from review of #82049:
- extract append_user_instruction() into agent/skill_commands so the
stable-prefix construction cannot drift between the skill and cron
builders (the registered prefix must stay a byte-prefix of the built
message); cron no longer imports the private _SINGLE_SKILL_INSTRUCTION
- add the startswith guard to the skill builder registration site,
matching the stronger cron guard
- rename _MAX_BYTES to _MAX_CHARS (sum(map(len, ...)) counts characters,
not bytes) and correct the comment
- collapse find_stable_prefix's two-lock dance into a single critical
section (scan is <=32 short-circuiting startswith calls, measured
2-4us; drops the snapshot copy and the TOCTOU re-check)
- document the split-shape lifetime (marked-endpoint window) in the
module docstring
- add a contract test for the helper's byte-prefix invariant
(mutation-checked)
Follow-up review of the builder-declared cache boundary (#81867) found three
ways the split could silently stop paying off, or keep paying more than it
should, on a long-lived gateway process.
Flattening no longer consults the registry. `strip_anthropic_cache_control`
matched the decorated split by looking the first block up in the prefix
registry, so a mid-turn failover that re-decorates a request built many
messages earlier (#72626) would fail to flatten once _MAX_ENTRIES newer
scaffolds had been registered in between, and would hand the next provider
the two-part shape instead of the canonical string. The split is now matched
by its shape: a marker on the *first* part of a user message is something no
other decoration produces (list content otherwise gets its marker on the last
part, and the two-part [static, volatile] split is role-gated to system), so
the ""-join stays provably byte-exact without any process state. This drops
`is_registered_stable_prefix` and one lock acquisition per stripped message.
Lookups now refresh LRU position. A scaffold fired every minute by cron could
be evicted by a burst of one-off skill invocations while still being the
hottest prefix in the process, silently reverting it to whole-message caching.
Registration now also evicts by total retained bytes (4 MiB). Entries hold
whole expanded skill bodies, so a 32-entry cap alone does not bound memory.
The newest entry is always kept, so a single oversized scaffold still gets a
boundary instead of disabling the split.
Tests: eviction-then-failover round-trip, LRU refresh on hit, byte-cap
eviction, and oversized-single-entry survival.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Webhook/cron skill invocations concatenate a large static scaffold
(activation note + expanded skill body) with a small volatile tail
(ticket payload, timestamps) into one user string, and the Anthropic
cache planner marked that whole string as a single atomic block — so a
few changed tail bytes forced a full cache rewrite on every invocation.
Instead of re-parsing scaffold marker strings out of the message at
request time (fragile when a payload or skill body quotes the marker),
the builders now register the exact stable-prefix bytes in a small
process-local LRU registry at construction time. The cache planner
splits a registered user string into [marked stable prefix, unmarked
volatile tail] request-locally; canonical session history stays a plain
string, and the failover stripper flattens the split back byte-exactly
via an O(1) registry lookup. Unregistered messages keep the existing
whole-message policy.
Covers the single-skill builder (webhook + slash command + TUI) and the
cron job prompt assembler (multi-skill, bundles, skipped-skill notice),
with registration guarded against injection-scanner sanitization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Folds the model-switch fix in with the untitled retry. They answer
different halves and each is wrong alone: counting alone left a session
that merely opened with machinery nameless forever, because nothing
reconsidered it, and the stored title alone would never title at all on a
store too old to report one. Skip only when both agree — past the opening
turn, and already named.
Counting a turn now judges a multimodal one on its text, so "here's a
screenshot, fix the login" counts as the question it is rather than
reading as machinery and undercounting the conversation.
Co-authored-by: yy28 <yy28@vip.sina.com>
Switching models before sending the first real message titled the session
"[System: The active model for this chat has…" instead of the user's actual
question.
`_append_model_switch_marker` persists its notice with `role="user"` because
strict OpenAI-compatible providers reject a system message that is not first
(#48338). Titling had no way to tell that apart from a genuine opening turn,
which caused two distinct failures:
1. `_MACHINE_PREFIXES` did not cover the marker. Its `[System: ` prefix
matches none of `[CONTEXT COMPACTION`, `[Runtime note:`, or `[SYSTEM]`
(different case, no closing bracket), so `is_titleable_user_message()`
returned True and the marker was formatted into the title.
2. `maybe_auto_title()` counted the marker as a user message. With the marker
present, the first real question arrived at `user_msg_count == 2` and the
`> 1` guard returned early, so the session was never titled at all and its
`title` column stayed NULL. Fixing only (1) would therefore have traded a
wrong title for a permanently missing one.
Add the marker prefix to `_MACHINE_PREFIXES` (kept in sync with
`tui_gateway.server._MODEL_SWITCH_MARKER_PREFIX`) and count only titleable
user messages when detecting the opening turn.
The guard stays narrow: ordinary user text that happens to start with
"[System:" still titles normally.
Adds 6 regression tests, verified to fail without the fix.
Two lookalike gaps found auditing the titler.
_MACHINE_PREFIXES missed the compressor's legacy summary opener and the
"[System note:" injections, so a compacted or resumed session could be
named after the note that carried it. Take the summary prefix from the
compressor that emits it rather than keeping a fourth local copy.
The fast-model exclude list covered embedders but not the other non-chat
siblings a provider names after its chat model — "gpt-4o-mini-tts"
satisfies the "-mini" rung and cannot answer a prompt.
An opener is not always titleable — an image with no caption, a compaction
handoff, a bare slash command — and those sessions stayed unnamed for
life, because the guard that stops re-titling a named session also stopped
the nameless one from ever asking again. Let a later turn name a session
that still has no title.
The derived title also ran the collision dedupe inline on the turn.
It is a slice of the user's own words, so it collides constantly — people
open sessions with "hi" — and resolving "hi #47" is a widening scan on the
critical path for a name the model replaces a second later. Decline it
there and let the background stage, which can afford the scan, pick it up.
The fast-model picker reads /v1/models to find the small model a provider
currently serves, and it asked anonymously. Most of those endpoints need a
key, so the fetch 401'd and the empty result read as "this provider has no
small model" — the picker fell back to its curated list and never noticed.
Worse, a failed fetch cached its empty result forever, so one bad moment
during startup disabled live model discovery for the life of the process,
and the processes that read this run for weeks. Give the failure an expiry
and pass the provider's credentials.
The bare family rungs (-mini, -flash, haiku) also picked whichever id
sorted first, which is the oldest generation a provider still serves:
gpt-3.5-mini over gpt-5.4-mini, claude-3-haiku over claude-haiku-4.5.
Compare the digit runs as numbers so the rung meant to keep us current
does.
Titling is two-stage — a slice of the user's own words lands inline, the
model's version replaces it a second later — and the platform rename lanes
fired on both. That is two rate-limited calls to reach one name, and
Discord allows two channel renames per ten minutes, so the throwaway could
be the one that survived. The callback now carries which stage it is, and
the lanes take the model's.
The relay lane also asked where the reply landed at title time, which is
before the model has answered: it polled the send-result cache for ten
seconds and read the timeout as "never auto-threaded", so any turn with
tool calls in it silently kept its raw thread name. Wait on the send
itself instead — the adapter already owns that cache, so it can say when a
reply arrives and, just as usefully, that one arrived carrying nothing.
The turn prologue titles every session, and it is shared by every agent —
including the ones no person is reading. A cron job already names its own
session after the job in its finally block, so the titler spent a side-LLM
call per fire to write the delivery scaffolding over it for the length of
the run. A delegated child's session is hidden from every picker, so a
batch at max_concurrent_children paid N title calls for N names nobody
opens.
Both are the same class of run that already sets skip_memory to stay off
the auxiliary path, so keep the titler off it too.
_PROVIDER_PREFIXES was a hand-maintained frozenset, so providers that ship
as plugins (bundled like fireworks, or user plugins under
$HERMES_HOME/plugins/model-providers/) were never recognised as
provider: prefixes in model strings, and metadata/context-window lookups
received the unstripped string. Mirror the _URL_TO_PROVIDER auto-extend
that already sits below it: add each registered profile's name and
aliases after discovery. The _OLLAMA_TAG_PATTERN guard keeps model:tag
strings intact.
Fixes#66106
The #70716 regression fix changes popen_start_new_session from False to
True in the systemd-scope branch. Update the assertion in
test_wraps_in_systemd_scope_when_supervisor_and_available and the
docstring in test_systemd_post_spawn_failure_never_kills_gateway_process_group.
/home and /Users hold home directories; neither is a workspace. A session
whose cwd was one of them got promoted to its own auto project, so the
sidebar listed a lowercase "home" row beside the synthetic Home bucket.
Both POSIX spellings are excluded on every host — macOS ships an empty
/home autofs stub, and container/remote shells hand back Linux paths — as
are the filesystem root and the parent of $HOME.
The existing test asserted supports_draft_streaming returns True with
rich_messages=True and rich_drafts=False, but the PR's gate now makes
it return False. The test already force-sets _use_draft_streaming=True,
so the assertion was redundant — updated to reflect the new behavior.
Also removed redundant manual attribute overrides in test 3 where
_make_adapter(extra={'rich_messages': False}) already sets the flag.
When rich_messages is on and rich_drafts is off, transport=auto used
sendMessageDraft (MarkdownV2 tables→bullets) then finalized via
sendRichMessage. Users saw a crooked first bubble and a second wiki-style
final. Decline drafts in that config so auto uses edit-in-place + rich
finalize on one message.
Fixes#78524
Agent Plugins v1 packages with 'streamable-http' mcp.json entries now load
through Hermes' existing URL-based MCP client instead of being reported and
skipped. The stdio-only limitation was the agreed follow-up slice from
PR #81196.
Boundary rules from the v1 spec (§7.2.1) are enforced:
- URL must be absolute http(s), no user information, no fragment; plain
HTTP only for localhost/loopback hosts.
- Configured package headers are never forwarded across a cross-origin
redirect: translation marks entries strict_redirect_headers, and the
redirect hook in the native runtime strips those headers (plus
Authorization) whenever a redirect leaves the original origin. On mcp <
1.24.0, where the client cannot hook redirects, such servers fail closed
with an actionable upgrade message.
- Legacy 'sse' entries remain reported and skipped.
The redirect hook is extracted into a testable module-level factory
(_make_redirect_header_stripper); default behavior for native config
servers is unchanged (Authorization-only stripping).
The native Relay pipeline binds its Futures to the event loop that
entered run_in_session_async. While a managed tool callback executes,
that loop is blocked until the callback returns — so a nested managed
relay call made from inside the callback (vision_analyze's auxiliary
LLM call on a worker-thread loop) awaits a Future that can never
complete: 'RuntimeError: Future attached to a different loop', or a
deadlock, plus 'Event loop is closed' at shutdown when the orphaned
future completes late. (#77244)
Fix: managed_callback_guard, a ContextVar depth marker set around every
Hermes callback the relay adapters hand to the native pipeline
(relay_tools.execute invoke, relay_llm execute/execute_async invoke,
ManagedLlmStream run_callback). resolve_execution_context returns the
no-relay triple while the marker is set, so nested calls run unmanaged.
The marker propagates through contextvars.copy_context() into the
worker threads tools use for their internal async work.
Top-level turn LLM calls and tool wraps stay fully managed — verified
live: vision_analyze works under active shared metrics while the main
turn still records managed llm.execute events.
Alternative fixes considered and rejected: removing
retain_managed_execution (kills the shared-metrics managed pipeline)
and gating on main-thread identity (managed tool wraps legitimately
run on the run_agent thread, so that gate disables relay everywhere).
The Anthropic SDK's streaming accumulator builds ParsedMessage snapshots
whose ParsedTextBlock content doesn't match the generic union pydantic
expects, so model_dump() on stream events (message_stop) emits
PydanticSerializationUnexpectedValue UserWarnings straight into the
user's CLI output mid-response.
Pass warnings=False at every helper that dumps arbitrary SDK models
(relay_llm/_jsonable, relay_tools/_jsonable, anthropic_adapter
_to_plain_data, run_agent _hook_jsonable, chat_completion_helpers
extra_content/reasoning_details sites, chat_completions transport),
with a TypeError fallback for duck-typed model_dump implementations.
Adds regression tests including a precondition test that proves the
fixture still trips the warning without suppression.
ethernet
izumi0uu
Teknium
joaomarcos
Phil Thomas
kshitij
gregorustar-maker
miura
bgrablin
HexLab98
Brooklyn Nicholson
pierrenode
Adam Durham
yy28
Oliver Mee
Slobaka