Both sidecars answered the same question in their own way. Which
directory does this Node child run from, when some installs put the
source tree somewhere nothing can write?
gateway/sidecar_runtime.py answers it once. Four rungs:
1. An operator override.
2. A writable source.
3. A read-only source whose baked deps match the lockfile.
4. A read-only source that must move to $HERMES_HOME/sidecars/<name>
before npm can run.
Node sets the shape of that last rung. Its ESM resolver reads
node_modules only from the directories above the importing file, and
NODE_PATH applies to CommonJS alone. Measured on Node 26: an ESM import
with NODE_PATH pointing at the packages fails, and the same import from
a directory beside them works. Both sidecars are "type": "module", so
the entry file and the packages must share a tree. A copy is the only
arrangement Node accepts.
_MIRROR_FILES is gone. It named the files to copy, so it had to name
every module the entry file imports. It was wrong twice. It listed the
deleted spectrum patch, and it omitted send-format.mjs and
stream-staleness.mjs. Both faults appear only on a read-only install.
The resolver copies the tree instead, without node_modules, and the test
compares the mirror against the source tree rather than against a second
list. A mutation that returns to a fixed list fails it.
The copy uses shutil.copy, not copy2. copy2 gives the mirror file the
mtime of the source, and a Nix store source has mtime = epoch. A
refreshed lockfile then always predates npm's install marker, and
deps_are_current() keeps stale node_modules through every upgrade,
which is the fault this resolver exists to fix. A plain copy stamps
the copy time, so a content change always postdates the previous
install. npm's hidden node_modules/.package-lock.json cannot replace
the content comparison: it is a different document, and npm matches it
semantically, not byte for byte.
WhatsApp gains what it never had: a staleness check and a refresh. Its
resolver returned any existing mirror without comparing it against the
lockfile, so an upgrade kept the old node_modules. This is a behaviour
change.
The mirrors move to $HERMES_HOME/sidecars/. The Baileys credentials live
in $HERMES_HOME/whatsapp/session, so a paired account is not affected.
`hermes doctor` reports the mirrors the old resolvers left at
$HERMES_HOME/photon/sidecar and $HERMES_HOME/scripts/whatsapp-bridge.
Nothing reads them now, and each one can hold a node_modules of some
hundred MB. --fix removes them.
_sidecar_deps_stale and deps_are_current read the same two files with
opposite missing-file answers, on purpose. Each one points at the other
and says why.
The container bakes both sidecars now. It baked Photon and left WhatsApp
to install at run time.
tools/lazy_deps.py held a table of about 40 features, each with its own
literal pip specs. pyproject.toml declares the same packages as extras,
so every pin existed twice and the two copies drifted.
Each feature now names an extra, and the specs come from pyproject at
run time. The table is 218 lines shorter. A test asserts that each
feature names an extra that exists and resolves to at least one spec, so
a typo cannot ship.
A wheel install, such as Nix, has no pyproject.toml beside the code.
There the same table comes from the dist metadata: each spec of an
extra is one Requires-Dist line, and its marker names the extra.
Without this fallback, each entry point raised on a Nix install, and
ensure() raised even for a feature whose packages the build baked in
through extraDependencyGroups. That call must be a no-op.
is_available() and feature_install_command() catch the failure as well
now. Their callers sit in status paths with no try/except, and their
contracts are bool and Optional[str].
The security overrides already come from pyproject (the previous
commit). This commit moves the reader onto the shared _pyproject()
cache and the shared temp-file writer.
The tier-0 installer, `uv sync --extra <name>`, names the project with
--project. uv reads the project from its working directory, and the
agent runs from the user's working directory, not from the install
tree. Without the flag the sync failed outside a checkout, and the pip
ladder always ran instead.
install_specs gets the same managed-install guard as ensure(). A Nix
venv is in the read-only store, so the pip ladder could only fail with
EROFS after a 15s ensurepip attempt. It reports the Nix remedy instead.
A durable install target overrides the guard, as it does in ensure(),
because the NixOS container module sets HERMES_MANAGED=true with a
writable target.
Spec parsing goes to packaging.requirements.Requirement, which is
already a core dependency. The hand-written version kept the
environment marker attached to the version. SpecifierSet raised on it,
so _is_satisfied answered True for every installed version of a marked
package. Such a package can never upgrade.
Reading the specs from an extra exposed a second fault, in the record of
which features are active. active_features read specs[0] as the anchor
package, and extra composition put sounddevice there for [voice] and for
each wake extra. One local STT install then marked every audio feature
active, and `hermes update` installed the wake engines that the user
never enabled.
ensure() records each feature it serves in
$HERMES_HOME/lazy-features.json, and active_features reads that record.
A recorded feature still needs its anchor package installed, so an
uninstalled backend does not come back. The anchor is the first pin
written directly in the extra, not the first spec after expansion. A
test asserts that no two extras share an anchor.
There is no seeding for an install that predates the record. Its first
`hermes update` refreshes nothing. ensure() then repairs a stale pin at
each backend's start and records the feature, and the next update covers
it.
[stt-whisper] splits out of [voice]. faster-whisper transcribes audio
files and needs no microphone and no PortAudio, so the Docker image can
bake it. [voice] composes [stt-whisper] and [audio-io] and stays the
microphone stack. stt.faster_whisper maps to the new extra.
Removed with the table:
- The literal pin list in plugins/platforms/google_chat/oauth.py. Its
pip path targeted /nix/store on a Nix install, which is read-only.
- The bare honcho-ai fallback in the honcho setup. An unpinned install
accepts whatever PyPI serves, which is the hole this branch closes.
Both call sites report the remedy for the deployment instead, through
the now-public managed_install_reason.
- install_deps() in the google-workspace skill. The SDKs ship in the
[google] extra, so a stripped environment is a broken install. The
repair is `hermes update`. A pip run from the script writes to
whichever interpreter it runs under, which is not always the one
Hermes uses.
- tests/test_runtime_pins_are_locked.py, which scanned first-party
source for pin literals. There are none left to find.
- The spec shape check in install_specs. The same plugin.yaml hands
external_dependencies[].install to bash with shell=True, and the
plugin's __init__.py is imported. Anyone who can write that file
already runs code as the user.
When the per-session pending transcript queue hits _MAX_PENDING_PER_SESSION
(200) while the session DB is broken, the gateway previously popped the
oldest message and discarded it permanently — silent user data loss during
live operation (#78182). The on-disk pending spool only ran at shutdown via
flush_pending_to_file.
Extend that existing spool machinery for runtime drops:
- gateway/shutdown_flush.py: add spool_dropped_transcript_message() and
drain_transcript_spool(), reusing _get_flush_dir/_write_payload (same
atomic-JSON pending_messages/ spool format). recover_pending_to_db()
now also replays transcript_cap_drop payloads left over across restarts.
- gateway/session.py: on cap eviction, spool the dropped message and log a
WARNING that includes the spool path; if spooling fails, degrade to the
previous drop-and-warn behavior. On the next fully successful transcript
flush for that session, drain and replay spooled messages in drop order;
replay failures keep the spool files for the next attempt.
- tests/gateway/test_pending_queue_spool.py: drop→spool→drain roundtrip,
per-session drain isolation, spool-failure degradation, replay-failure
retention, and spool primitive ordering/reason filtering.
No new config; extends existing flush_pending_to_file infrastructure per
AGENTS.md guidance.
Refs #82616, #78182
Root cause of #82616: gateway session identity (session_key/chat_id/
origin_json) was written best-effort in a separate UPDATE after row
creation, both reset-path DB writes swallowed failures silently
(logger.debug / bare print), transcript reads ignored the reroute map
that writes follow, and restart recovery ranked candidate rows by
started_at while hard-rejecting empty rows. A single failed write could
therefore strand the live conversation in an unroutable orphan row while
a days-old zombie kept the routing key — after any gateway restart the
chat silently resumed the zombie (user-visible context loss, 5 confirmed
incidents on one install since June).
Four class fixes:
1. Identity lands atomically in the session INSERT: origin_json and
display_name join _insert_session_row's column list + COALESCE
backfill; both gateway creation paths (get_or_create + reset) pass
full identity including parent_session_id lineage (fixes#12857).
2. record_gateway_session_peer self-heals: when the target row is
missing (failed/deferred create, crash window) it INSERTs the row
with full identity instead of silently no-opping — every per-turn
peer refresh is now a repair opportunity, and an identity-less lazy
writer (update_token_counts/record_auxiliary_usage) can never leave
a gateway session permanently unroutable.
3. load_transcript follows the write-side reroute chain and the durable
compression tip before querying, so reads can no longer return 0
rows for a session whose messages live under its compression child;
read exceptions are WARNING, distinguishable from an empty result.
4. find_latest_gateway_session_for_peer ranks by
COALESCE(last_activity_at, started_at) (message-bearing rows first)
and returns an empty-but-keyed row instead of None — a zombie
predecessor can no longer beat the live conversation, and recovery
never mints a fresh id when a keyed row exists.
Reset-path DB write failures now log at WARNING with the routing
consequence spelled out.
Tests: tests/gateway/test_session_continuity_82616.py (11 tests) —
sabotage-verified: 6/11 fail without the fixes. E2E incident replay
(real SessionDB, temp HERMES_HOME) confirms the production shape now
resolves to the live session.
Fixes#82616. Related: #12857, #78182 (read-path half), #79576.
asyncio.ensure_future(result) creates a task with only a weak ref in
the event loop's task table. After the carrier raises CancelledError,
the local 'task' variable goes out of scope and the loop can GC the
handler before it finishes — the exact 'handler killed mid-flight'
class we are fixing, just via GC instead of cancellation.
Add _detached_fatal_tasks set on BasePlatformAdapter (matching the
gateway-level pattern in _handle_adapter_fatal_error). Uses getattr
fallback for test stubs built via object.__new__().
When an adapter escalates a retryable fatal error from inside one of its
own tasks (e.g. Telegram's _polling_error_task after exhausting polling
network retries), the gateway's _handle_adapter_fatal_error tears the
adapter down via disconnect() — which cancels that very task. The
propagating CancelledError killed the handler between popping the
adapter from the adapter map and queueing the platform in
_failed_platforms, leaving a zombie gateway: process alive, zero
connected platforms, zero pending retries, until a manual restart.
Run the handler as a detached task under asyncio.shield so carrier
cancellation no longer aborts teardown/queueing mid-flight. The carrier
still observes CancelledError (teardown semantics unchanged); only the
handler is protected. A done-callback consumes the detached task's
exception to avoid 'Task exception was never retrieved' noise.
Fixes#81335
Titling is two-stage — a slice of the user's own words lands inline, the
model's version replaces it a second later — and the platform rename lanes
fired on both. That is two rate-limited calls to reach one name, and
Discord allows two channel renames per ten minutes, so the throwaway could
be the one that survived. The callback now carries which stage it is, and
the lanes take the model's.
The relay lane also asked where the reply landed at title time, which is
before the model has answered: it polled the send-result cache for ten
seconds and read the timeout as "never auto-threaded", so any turn with
tool calls in it silently kept its raw thread name. Wait on the send
itself instead — the adapter already owns that cache, so it can say when a
reply arrives and, just as usefully, that one arrived carrying nothing.
The existing test asserted supports_draft_streaming returns True with
rich_messages=True and rich_drafts=False, but the PR's gate now makes
it return False. The test already force-sets _use_draft_streaming=True,
so the assertion was redundant — updated to reflect the new behavior.
Also removed redundant manual attribute overrides in test 3 where
_make_adapter(extra={'rich_messages': False}) already sets the flag.
When rich_messages is on and rich_drafts is off, transport=auto used
sendMessageDraft (MarkdownV2 tables→bullets) then finalized via
sendRichMessage. Users saw a crooked first bubble and a second wiki-style
final. Decline drafts in that config so auto uses edit-in-place + rich
finalize on one message.
Fixes#78524
Adds the failed-result guard the salvage review called for:
_deliver_queued_first_response now takes deliver_media and the queued
follow-up call site passes deliver_media=not _delivery_result.get('failed').
A failed turn still delivers its normalized failure text (pinned by
test_run_agent_sends_normalized_failure_before_queued_followup), but its
attachments are no longer uploaded as if the turn succeeded — mirroring
the completed-turn path's 'not agent_result.get(failed)' guard.
Regression test added.
Ensure queued follow-up resends keep MEDIA-backed attachments by replaying the
first response through the gateway's text-plus-media delivery flow instead of a
plain adapter text send.
Apply the same metadata invariant to sealed split chunks, keep expect_edits on live previews, and exercise the real Telegram adapter path with rich messages enabled but rich drafts disabled.\n\nCredits PR #78525 by @Slobaka for the reproduced rich_messages/rich_drafts combination.
A finalized native draft is the first persistent send and will not be edited again. Omitting expect_edits lets Telegram use sendRichMessage for the persistent final instead of degrading tables through MarkdownV2.\n\nAdapted from PR #46536.
Unknown charset labels (QQ Mail's RFC 1428 'unknown-8bit' placeholder,
misspelled names, garbage encoded-word charsets) raised LookupError from
bytes.decode — errors='replace' only guards decode errors, not a missing
codec — aborting the whole fetch batch. UIDs are marked seen before the
fetch, so the crash permanently dropped every message in the batch.
- _safe_decode(): alias table (unknown-8bit→utf-8, gb2312/gbk→gb18030,
ks_c_5601-1987→cp949, ...) then utf-8, then latin-1 last resort.
- _decode_header_value(): wraps decode_header() so a malformed RFC 2047
header degrades to the raw string instead of crashing.
- _extract_text_body(): all three decode sites now use _safe_decode.
Fixes#35901, fixes#55381, fixes#55383.
Cron and agent output that contains emoji, CJK, or accented text is
silently lost on Windows. When a job's output exceeds the platform limit
(MAX_PLATFORM_OUTPUT = 4000), DeliveryRouter._deliver_to_platform saves
the full text to disk and sends a truncated preview with a "full output
saved to ..." pointer. That save used Path.write_text(content) with no
encoding, so on Windows it encodes through the platform code page
(cp1252) and raises UnicodeEncodeError on any non-ASCII character. The
exception propagates out of _deliver_to_platform and deliver() records
the target as failed, so the whole truncate-and-send path aborts: the
user receives nothing — even though an ASCII payload of the same size
would deliver fine — and the promised backup file is never written. The
sibling local-file path (_deliver_local) had the identical defect. The
Windows-footgun CI gate misses this because it only inspects open() /
Path.open(), not Path.write_text().
Both writes now pass encoding="utf-8" explicitly so output is persisted
consistently across platforms.
Fixes silent loss of non-ASCII cron/agent output on Windows. The two
on-disk writes in the delivery router (`_deliver_to_platform`'s full
output save and `_deliver_local`'s file save) now write UTF-8 instead of
the platform-default code page, so emoji/CJK/accented output is saved
and delivered the same on Windows as on macOS/Linux.
N/A
- [x] 🐛 Bug fix (non-breaking change that fixes an issue)
- `gateway/delivery.py`: pass `encoding="utf-8"` to the `write_text`
call in `_save_full_output` (oversized-output backup) and the one in
`_deliver_local` (local file delivery).
- `tests/gateway/test_delivery.py`: add two regression tests that
simulate a non-UTF-8 Windows code page and assert oversized non-ASCII
output is still delivered and the backup/local files round-trip as
UTF-8.
1. `scripts/run_tests.sh tests/gateway/test_delivery.py` — 25 passing.
2. Revert either `encoding="utf-8"` argument and re-run: the two new
tests fail with `UnicodeEncodeError` from the cp1252 codec, proving
they catch the regression.
3. `python scripts/check-windows-footguns.py gateway/delivery.py` and
`ruff check gateway/delivery.py tests/gateway/test_delivery.py` both
pass.
- [x] I've read the Contributing Guide
- [x] My commit messages follow Conventional Commits
- [x] I searched for existing PRs to make sure this isn't a duplicate
- [x] My PR contains only changes related to this fix
- [x] I've run the gateway delivery tests and all tests pass
- [x] I've added tests for my changes
- [x] I've tested on my platform: macOS 15 (Darwin 25.5)
- [x] I've updated relevant documentation (README, docs/, docstrings) — or N/A
- [x] I've updated cli-config.yaml.example if I added/changed config keys — or N/A
- [x] I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
- [x] I've considered cross-platform impact (Windows, macOS) — this fix is specifically about Windows code-page encoding
- [x] I've updated tool descriptions/schemas if I changed tool behavior — or N/A
e81d18dfb collapsed six per-surface copies of reasoning resolution onto
resolve_reasoning_config() and, in its own words, "fixes the gateway
resolving reasoning against config model.default instead of the session's
effective model". It did not touch gateway/platforms/api_server.py, which
kept that defect.
_create_agent() called GatewayRunner._load_reasoning_config() with no
model on its first line — before the model precedence chain (browser lock
-> session /model -> session row -> route -> per-request -> defaults) has
run. Per-model agent.reasoning_overrides therefore keyed off model.default
on the one surface where every request names its own model: a request for
a model with an override silently got the global effort instead.
Resolve after the chain settles, so the override follows the model the
request actually runs. An explicit per-request reasoning parameter still
takes precedence over config.
The existing test stub for _load_reasoning_config took no arguments (it
mirrored the old call); it now matches the real signature, as the sibling
stub in the same file already did.
Follow-ups on the salvaged commit (#37207 by @charzhou):
- Persistent /root home mount translates too: an agent writing
/root/out.png produced a real host file under
<sandbox>/docker/default/home the gateway could not find.
- /root/.hermes cache mounts translate to the HOST cache (longest-prefix
beats the home mount), so MEDIA:<agent_visible_image> paths deliver.
- /root/.hermes/* OUTSIDE a cache mount never translates through the home
mount: those are the sandbox's credential copies (.env, auth.json) that
sit outside the host-side denylist prefixes — fail closed.
- Run the idempotent terminal-config->env bridge before mount parsing so
in-process gateways (Desktop backend, hermes serve) see the active
backend and docker_volumes (covers #42299's /output case there too).
Translate MEDIA paths under configured Docker volume mounts (and the
default persistent /workspace) to host paths before media delivery
validation, using longest container-prefix match so host:/workspace and
/output export mounts work.
_humanize_user_mentions rewrites <@UID> to @DisplayName by passing the
resolved name as re.sub's replacement, where re parses it as a template.
A display name is arbitrary user-set text, so the escapes in it are the
user's characters, not regex syntax:
dev\ops -> re.error: bad escape \o
a\1b -> re.error: invalid group reference 1
\g<0> -> expands to the whole match, silently putting the opaque
<@UID> back — the token this method exists to remove
The trigger-text call site sits in _handle_slack_message outside any try,
and both Bolt event handlers await it bare, so the raise takes the whole
inbound message down: every message mentioning that person is dropped.
Pass the replacement as a function instead — re does no template parsing
on the return value, so the name lands verbatim. Same shape the Matrix
adapter already uses for its outbound mention rewrite.
The non-streaming /v1/responses path built function_call and
function_call_output output items with no status field (and no item id),
while the SSE streaming path correctly emits status in_progress ->
completed. Spec-strict OpenAI clients reading the non-streaming output
array could interpret the status-less function_call items as pending
calls the CLIENT must execute — but these tools were already executed
server-side by the Hermes agent and are replayed for structured tool UI
only. Reported by a community user whose GPT-5.6 client concluded 'a
server should not tell an OpenAI client to execute a tool the server
already executed itself'.
- _extract_output_items now stamps status: completed and spec-shaped
item ids (fc_/fco_) on replayed items, matching the streaming path
- test updated to pin status + id shape
- docs example updated + explicit note that output tool calls are
replayed, never pending
Two defects in _normalize_empty_agent_response surfaced together during a
state.db lock-contention incident on an enterprise Slack deployment:
- the error lookup used dict.get's default, which an explicit
'error': None value bypasses, rendering 'The request failed: None' /
'unknown error';
- persistence failures fell through to the generic branch, whose 'use
/reset' advice is harmful for this failure mode (destroys conversation
context, fixes nothing).
Persistence-failed turns (failure_reason session_persistence_failed:*,
with a legacy fallback on the error text) now get a dedicated message:
storage was temporarily unavailable, the message was recorded, send it
again — with a disk-specific variant. No /reset suggestion. All other
branches unchanged.
The _standalone_send() function (used by the send_message tool for
proactive/scheduled sends) has the same bare `@<id> text` bug that
send() had before PR #44444. SimpleX's `@<x>` syntax resolves x as a
display name, not a contactId — the daemon silently drops messages
when it cannot find a contact named "6".
Use the structured `/_send @<id> json [...]` form, matching what
send_image, send_document, and the send() fix already use.
Fixes#46265
Adds regression test for the case where both disk and live start_time
are known integers but differ (e.g. stale value from a previous run).
The self-PID short-circuit must fire regardless — start_time only
guards PID reuse for *other* PIDs. Inspired by #81495's test case.
Replace the 'describe everything in thorough detail' auto image-preprocess
prompt with a concise 2-4 sentence summary prompt so image-bearing gateway
messages stop generating ~2000-char descriptions (35s+ on local models).
Prompt-only variant of #10852: the max_tokens=500 cap and the
preserve_max_tokens aux-client plumbing from the original PR are
intentionally dropped to stay compatible with the max-tokens-knob policy
direction (#75253 removes hardcoded vision caps).
Fixes#10809
Surfaced during the post-merge review pass on our own #81113 follow-up:
cached_fetch_api_models gained _cache_entry_valid (numeric-'at'
validation) but its sibling cached_provider_model_ids still did
float(entry.get('at', 0)), which raises ValueError/TypeError on a
hand-edited or corrupted provider_models_cache.json row and propagates
uncaught into the /model picker call sites. Same fix, same helper:
corrupt rows are now a cache miss (live fetch), never an exception.
Both wrappers now share the identical validity predicate, closing the
divergence the 'mirrors' docstring promised away.
Also two test nits from the same review: unused OrderedDict import
dropped and the drain-order assertion strengthened to pin LRU-first
FIFO order in tests/gateway/test_agent_cache_pressure.py.
Mutation-checked: restoring the raising float() form makes the new
corrupt-at tests fail.
Two small config-gated features:
1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
true, config.yaml): a running card with broken claim bookkeeping
(claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
restore) is invisible to all existing recovery paths
(release_stale_claims requires claim_expires NOT NULL,
detect_crashed_workers requires host-local lock + pid,
detect_stale_running is config-disabled by default) and shows Running
forever. New reconcile_orphaned_running() pass in kanban_db.py runs
each dispatch_once tick: requeues orphans to ready with an explanatory
comment, closes any leaked run, emits a 'reconciled' event, and defers
when the recorded PID is still alive on this host (never requeue
beside a live worker). Surfaced via DispatchResult.reconciled_orphans.
2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
config.yaml): optional {name, value_from: static|profile, value}
mapping; the header is attached to that server's HTTP/SSE transport
requests. 'static' sends the config value; 'profile' resolves the
active Hermes profile name once at connect time (no per-call
mutation). Explicit per-server headers of the same name (any casing)
win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.
Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.
Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
- Drain the eviction plan (pop + del) before trim_memory: the batch
thread previously held every evicted agent in its local list while
gc.collect + malloc_trim ran, so the in-pass trim freed almost
nothing, the next tick re-read a still-high RSS, and the valve
over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
shallow copy of the flushed transcript (stamped on every successful
flush) sharing every message dict — and pressure-evictable agents
have flushed by definition, so it pinned the multi-MB content strings
on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
instead of bare AgentCacheBounds(): the dataclass default disables
the pressure pass, but an absent config section means 'auto' — a
transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
un-flushed persistence (e.g. session DB never initialized — NFS
HERMES_HOME) from mid-turn agents, so operators can diagnose why the
valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
systemd unit's MemoryHigh=/MemoryMax= is detected — the root
memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
mutation-checked (revert each fix -> its guard fails).
The per-session agent cache is capped at 128 entries with a 1h idle TTL, and
neither bound knows how many bytes it holds. Each cached agent pins
_session_messages -- the full transcript including tool output, tens of MB on
a session with 100+ tool calls -- so a gateway serving many chats keeps every
warm transcript resident: agents that took a turn inside the TTL are never
idle-swept, and the idle sweep additionally defers finalizable sessions until
they expire. RSS climbs until the cgroup throttles and SIGTERM can no longer
flush inside systemd's stop timeout.
Add the missing bound. Each session-expiry watcher tick compares the process's
anonymous RSS against a budget and, when over, sheds LRU agents through the
same soft-eviction path the cap enforcer uses, then runs malloc_trim so the
freed arenas actually return to the OS. Evicted sessions rebuild their
transcript from the persisted session on the next turn.
Three classes of session are never shed: agents mid-turn, the most recently
used ones, and any session whose transcript has not finished reaching disk
(_last_flushed_db_idx vs len(_session_messages) -- the same divergence the FTS
write-corruption guard reacts to when it preserves live history).
memory_high_mb defaults to "auto", deriving the budget from the cgroup limit
the gateway runs under, so a MemoryHigh/MemoryMax on the unit is respected
without a second number to keep in sync. The two existing bounds become
configurable alongside it under agent.agent_cache.
protect_recent is clamped to half the cache: a couple of sessions can exhaust
the budget on their own, and a fixed MRU guard would then protect everything
and leave the gateway climbing with nothing it would shed.
Fixes#80764
ethernet
Teknium
kshitij
gregorustar-maker
Brooklyn Nicholson
Slobaka
StellarisW
Kong
liuhao1024
Hermes Agent
峯岸 亮
kernel-t1
Drexuxux
CharZhou
Victor Kyriazakos
Ken Weiner
HexLab98
kinsolee
HenryG
gnanirahulnutakki
Gille
Dominic Bejar
kyssta-exe