Three portability bugs the Windows development machine hid:
- The version check's subprocess.run used text=True without an explicit
encoding, which decodes with the locale codepage and crashes on
non-UTF-8 bytes (the repo's Windows-footgun lint). Pass
encoding='utf-8', errors='replace'.
- test_sha256_mismatch_rejects hardcoded the Windows asset name
(bin-win-*.zip); Linux runners resolve bin-ubuntu-*.tar.gz, so the
poisoned download was never the file under verification and the
expected rejection never fired. Resolve the asset name the way the
installer does.
- test_download_job_lifecycle_with_sha_failure let variant selection
price against the host machine; a GPU-less CI runner honestly refuses
every build (409) before the download path under test is reached. Pin
a generous budget — the test is about hash failure, not selection.
User documentation for the managed runtime: the install -> download ->
use flow, how hardware-aware model selection works, the memory
guarantees (fit pills, context growth, the 64K floor), the system
resources statusbar item, the local_runtime config reference, and using
an existing llama-server instead of the managed one. Cross-linked from
configuring-models, the desktop guide, and both manual local-LLM guides
(Ollama, Mac), which stay authoritative for manual setups.
The desktop surface for the managed local runtime:
- Local Models pane (Settings -> Providers): install the runtime, browse
the catalog with per-machine fit pills (green fits-your-GPU / amber
uses-system-RAM / red too-big, plus context start/max and vision),
download with live byte progress, Use to make a model the default,
eject and delete. Rows show residency live while the pane is visible —
a stale 'Not in memory' next to a full GPU reads as a broken feature.
- Downloads and activations run through an app-level job store, so
closing the pane (or reloading the app) never orphans a 20 GB
download; completion and failure surface as toasts wherever the user
is.
- Onboarding and the providers Accounts page offer 'Run models locally —
no account needed' alongside cloud providers.
- System resources statusbar item (hidden by default): GPU utilization,
GPU memory, and RAM, polled only while visible.
- i18n for en/zh/zh-hant/ja; Badge gains a success variant so fit state
reads as a real traffic light.
A curated model catalog priced for the machine it's viewed on, and the
/api/local-models/* routes the desktop consumes.
- catalog: each model ships a quant ladder (Q8 down to Q4, best first)
with exact sizes and sha256s pinned from Hugging Face LFS metadata.
Selection picks the highest-quality build whose weights + 64K-floor KV
fit GPU memory entirely; machines that can't get Q4 spilled to system
RAM; refusal only when even Q4 exceeds GPU + RAM. Nothing below Q4
ships — the quality loss is too severe for a first local-AI
experience. Split-GGUF variants, vision projectors, and spec-decode
draft models download as a unit, each file verified.
- routes: status (cheap, poll-safe), hardware, catalog (each row carries
plain-language fit facts the UI shows verbatim), download jobs with
aggregate byte progress that survive the pane unmounting, activate
(start server, set as main model — the click is the opt-in), eject,
delete (removes every staged file), and server on/off. Downloads run
8 parallel ranged connections and verify sha256 before use; a running
router only scans models at spawn, so staging changes bounce it.
- inventory: staged local models appear as a provider row in the model
picker payload every surface consumes; no credential — a local server
is authenticated by reachability.
Catalog reachability (repos, filenames, live-sha drift) is covered by an
opt-in network test gated on HERMES_TEST_NETWORK=1.
Local models get one context contract: any model runs at any window up to
its native max; hardware and session depth only change speed. No knobs.
- gguf + estimator: a stdlib GGUF reader feeds a per-layer context-memory
estimator that prices dense, sliding-window, and recurrent/hybrid
layers separately. Per-architecture cost spreads ~40x (dense 144 KiB
per token vs hybrid ~3 KiB), so per-layer pricing is what makes
1M-token windows a launch decision instead of a guess. Estimator
accuracy vs real models: worst case ~8%.
- context_policy: models launch at the largest window that fits GPU
memory entirely, floored at 64K. On Windows/WDDM, over-allocating VRAM
silently slows decode ~9x, so every window grant re-fits against live
memory. When weights exceed VRAM, spill placement pins expert/FFN
weights to host RAM so attention and KV stay resident (~1.75x over
naive spill); speculative decoding turns on only for spilled configs.
- growth: when a session reaches its window's edge, Hermes grows the
window toward native max instead of compressing — both compression
gates try growth first, and compression becomes the move of last
resort at native max, at the ~6 tok/s speed floor, or when physics
says stop. Growth re-prefills server-side; the conversation history
never mutates, so prompt caching is unaffected. Grown windows persist
per model and re-fit honestly on every boot.
- presets: launch decisions travel to the router as a generated
--models-preset INI; catalog sampling defaults merge under policy keys
(policy wins), vision projectors and spec-decode drafts attach when
present.
- model_metadata: the context meter reads the granted window from the
running server, per router child, so the UI shows the window the model
actually has.
Hermes can now bring its own inference engine. New hermes_cli/local_runtime
package:
- binaries: resolve and download official llama.cpp release builds for the
host platform (CUDA/Metal/Vulkan/HIP/CPU), sha256-verified, with N-1 tag
retention for rollback and honest errors for platform gaps.
- supervisor: spawn one llama-server in router mode with a generated API
key; crash-restart with backoff (router only — child failures surface,
never auto-retry); readiness proven by a real generation rather than a
health probe; idle models unload after 15 minutes and reload on demand.
- detect: fingerprint an already-running llama-server via /props so an
external server is used instead of starting a second one. Servers that
merely speak /v1 (Ollama, LM Studio) don't false-positive.
- endpoint resolution: a llamacpp-flavored provider with no explicit
base_url resolves managed-first, detected-external second; an explicit
base_url always wins. No new provider surface — the existing custom
provider aliases carry it.
- lifecycle: the backend boots the server when the user has opted in and
shuts it down with the app so no orphan pins GPU memory. Config lives
in the local_runtime section; deliberately no context or VRAM knobs.
The server binds 127.0.0.1 (never localhost — the name resolution adds
~2s per request on Windows) and models load on first inference rather
than at boot.
Typing '@t' in the composer only ever completed paths/directives —
agent profiles were invisible even though multi-agent UIs (Bot Mode)
route @<profile> as a handoff. complete.path now surfaces matching
profiles for bare-word @queries, ranked above file hits (there are at
most a handful), plus the full list on a bare '@'. The primary profile
is also offered as '@hermes' when no real profile claims that name.
@kind: directive queries are untouched.
Verified live: '@t' -> @turqoise first, '@her' -> @hermes,
'@file:tur' unaffected.
The command wrapper prints the cwd marker after the command returns. A
killed or timed-out command emits no marker, so ``env.cwd`` still holds the
directory of the last command to FINISH. One local environment serves every
session, because ``_resolve_container_task_id`` collapses cwd-only overrides
to ``"default"``. That leftover directory is therefore routinely another
session's.
The post-command dual-write copied ``env.cwd`` into the interrupted session's
durable record. Every later command in that session then ran in the foreign
directory, and the cwd echo told the model it had moved there. A desktop chat
silently re-homed into a worktree that another chat had opened.
Report the observation instead of inferring it. The marker parse now sets
``result["cwd_observed"]``, and both the record write and the echo read that
flag. The local override clears the flag when it rolls back a path that does
not exist, because the restored value is also unobserved. When a command
reports no cwd, the session keeps the directory it already had.
This needs no second session to be wrong: a lone session that interrupts a
command re-adopts a stale value too. A second session only makes the wrong
directory belong to somebody else.
The same class of write exists in the file-tools rescue for a reaped
environment (#26211). That rescue copied the cached snapshot of the shared
``env.cwd`` into the session record. The rescue is now fill-only: it writes
the snapshot when the session has no record, and it never overwrites a
record that the session wrote for itself.
The tests drive ``terminal_tool`` itself through an interrupt, not a copy of
its gate. Review found that a revert of either call site passed the first
version of the tests. Each gate now has a test that fails when the gate is
removed (verified by mutation).
Two exact-dict assertions in the Vercel sandbox tests now assert the two
fields they care about, so a new result key does not fail them.
Extends the NS-656 memory-pressure surface to cover disk exhaustion
(OOF-2 / OOF-107 lineage: agents fill their data volume — SQLite writes
fail, sessions stop persisting — while every dashboard looks healthy).
- gateway/disk_status.py (new): collect_disk_status() samples
shutil.disk_usage(HERMES_HOME) and classifies pressure
(critical: <256 MB free or >=95% used; elevated: <512 MB free).
Never raises — degrades to pressure="unknown" with null telemetry,
same contract as collect_memory_status().
- /api/status: sibling `disk` block next to `memory`, advisory only —
not folded into component/overall health.
- web: DiskPressureStatus type; MemoryPressureBanner generalized to a
resource banner with worst-first triggers (disk critical > memory
critical > OOM restart > disk elevated > memory elevated) and
cascading dismissals — hiding the top trigger surfaces the next one
instead of silencing everything. All dismissals stay boot_id-scoped.
- i18n: diskCriticalBanner / diskElevatedBanner (en, optional fields
with English fallback per existing pattern).
Tests: gateway/test_disk_status.py (14), web_server disk-block
presence/degradation, banner disk trigger/priority/dismissal-cascade
suite (21 total).
Review edge case: OOM dismissal was boot-keyed but live critical/elevated
dismissals were keyed only by severity. Dismiss critical, gateway reboots,
next poll still critical with no observed "ok" in between — the new
incident stayed hidden, and masked the OOM notice too, since critical
takes precedence.
Every dismissal key now embeds boot_id, so a gateway restart invalidates
prior dismissals of any kind. Within a boot, semantics are unchanged:
severity-scoped masking, escalation re-opens, confirmed "ok" recovery
clears live dismissals, "unknown" clears nothing. Missing boot_id
(pre-NS-656 image) degrades to a shared per-severity bucket as before.
Addresses the human review findings on the memory-pressure feature:
* [P2] Dismissal hid later incidents of the same kind. The gateway now
publishes `boot_id` (the lifecycle sentinel's started_at — changes on
every gateway life) in the /api/status memory block, and the dashboard
keys OOM-restart dismissal on it: acknowledging one restart no longer
mutes the NEXT one (the OOM-loop case this banner exists for). Live
pressure dismissals now also reset once pressure is demonstrably back
to "ok" — "unknown" (stale heartbeat) is absence of evidence and
clears nothing. Dismissal storage moved to a JSON list; old bare-string
entries fail JSON.parse and degrade to a clean reset.
* [P2] suspected_oom is a heuristic (unclean exit + low-memory final
heartbeat), not proof the OOM killer acted — banner copy now says
"restarted unexpectedly, most likely because it ran out of memory"
instead of stating OOM as fact.
* [P3] Mobile header clearance was applied per-banner (mt-14 on both
MemoryPressureBanner and ProfileScopeBanner) AND on the content
(pt-14), double/triple-stacking 56px gaps when banners were visible.
Replaced with a single h-14 spacer above the banner stack.
api.ts already declares MemoryStatus for the /api/memory providers
endpoint; the NS-656 pressure block reused the name, and TypeScript
declaration merging fused the two shapes — 'tsc -b' in the Docker image
build failed on every test fixture. Renamed to MemoryPressureStatus.
Hosted agents can be OOM-killed hourly while the dashboard and the NAS
agent card both look perfectly healthy — every memory signal the gateway
already produces (heartbeat mem samples, lifecycle-ledger unclean-exit
verdicts, cache-pressure evictions) dies in server-side log files. The
BlueAtlas incident (NS-608) ran for three days like this.
This is the read-side fix:
* New gateway/memory_status.py distills the existing 30s loop heartbeat
(gateway RSS + system MemAvailable/MemTotal + swap) and the lifecycle
sentinel into a compact `memory` block: pressure ok/elevated/critical/
unknown, coarse MB numbers, and last-boot unclean/suspected-OOM flags.
Pure file reads, no new sampling, no gateway IPC. Stale (>150s) or
future-dated heartbeats degrade pressure to "unknown" so a dead
gateway's final gasp can't render a live "critical" banner forever.
Critical thresholds mirror the ledger's OOM-suspicion heuristics: if a
level would make a later unclean death "suspected OOM", warn at that
level while the process is still alive.
* lifecycle_ledger.record_startup now carries prior_unclean_exit /
prior_suspected_oom onto the reclaimed sentinel — previously the
verdict survived only in append-only diag prose. Flags age out on the
next sentinel rewrite (scoped to the life after the crash).
* /api/status serves the block (profile-aware, executor-offloaded,
fail-safe to pressure=unknown). Deliberately NOT folded into
components/overall: memory pressure is advisory, and flipping overall
to "degraded" on it would page NAS's availability sweep for a
condition the eviction valve is already handling. Public-safety:
coarse numbers/enums/booleans only — same disclosure class as the
existing nous_session_valid field, added for the same NAS-sweep
audience.
* Dashboard: new MemoryPressureBanner (app-shell, next to
ProfileScopeBanner) with worst-first trigger precedence
(critical > suspected-OOM restart > elevated), per-trigger
session-scoped dismissal, and escalation re-opening past a dismissal.
i18n keys optional with English fallbacks, matching the
managingProfileBanner convention.
Tests: gateway/test_memory_status.py (classification bands, staleness,
clock skew, corrupt files, bool-is-not-int), lifecycle sentinel
carry-forward, /api/status contract (block always present, collector
crash degrades instead of 500), and 7 banner component tests.
NAS-side ingestion (agent-card notice + memory-tier upsell) ships
separately.
Refs NS-656; context: NS-608, NS-657, OOF-77.
Azure Foundry's OpenAI-compatible Responses surface rejects the post-tool
follow-up payload with HTTP 400 `invalid_payload` when a replayed encrypted
`reasoning` item is sent alongside `function_call` / `function_call_output`.
The initial function-call request and ordinary multi-turn continuity are both
accepted, so the failure only appears after the first tool executes.
Detect the Foundry endpoint in `ResponsesApiTransport.build_kwargs` and drop
only the encrypted reasoning replay on that follow-up turn, leaving
function_call / function_call_output continuity intact.
Salvage of #59981, rebuilt on current main. Same root cause and fix direction
as the original, which was correct; this version resolves three defects:
- No `chat_completion_helpers.py` change. main already forwards `provider`
and `base_url` to the Responses transport, so the original's re-added
arguments produced `SyntaxError: keyword argument repeated: provider` on
merge. Dropping the hunk removed the syntax error and the conflict.
- Host matching uses `utils.base_url_host_matches`, not a substring test.
`".services.ai.azure.com" in base_url` also matches URLs carrying the
domain in a path or query segment, which would silently disable reasoning
replay on an unrelated provider.
- The post-tool predicate tests the trailing messages, not the whole history.
Scanning for any tool call plus any tool result made it sticky: one tool
call early in a conversation suppressed reasoning on every later turn.
- Tool calls pair on `call_id` as well as `id`. Responses histories carry the
function call id in `call_id` while `id` holds the response item id
(`fc_...`). Identity is resolved via the converter's own
`_split_responses_tool_id`, covering composite `"call_x|fc_y"` ids and bare
`fc_` ids on both sides of the pairing.
Tests: 27 cases across the transport and the live `build_api_kwargs` bridge,
including six parametrized tool-call id shapes, non-Foundry host lookalikes,
the sticky-history guard, parallel tool results, and an unpaired tool result.
Each guard was confirmed to catch its defect by reverting the fix.
Verified with `scripts/run_tests.sh tests/agent/ tests/run_agent/`:
532 files, 5602 tests passed, 0 failed.
Not verified against a live Azure Foundry endpoint — no credentials. The
original HTTP 400 reproduction and post-fix Foundry Project / Azure Container
Apps harness runs are @AshuJoshi's, from #59981. This change is verified at
the payload-construction layer only.
Closes#59981.
Co-authored-by: Ashu Joshi <AshuJoshi@users.noreply.github.com>
* fix: mirror voice config (stt/tts/voice) into profiles created via profiles.create
Desktop dictation is profile-scoped: /api/audio/transcribe resolves the
stt section inside the TARGET profile's home. Profiles created through
profiles.create got only a model section, so dictation and TTS silently
fell back to defaults (local whisper, often not installed) — 'voice
dictation doesn't work in bot mode but is fine in regular mode'.
Mirror the launch profile's stt/tts/voice sections (key-wise, never
overwriting sections the clone already has) under the same
mirror_credentials flag that gates .env/auth mirroring, and report it
as mirrored.voice in the receipt.
* guard: route voice-config mirror through canonical loaders
read_user_config_raw (write-back round-trip; load_config would merge
DEFAULT_CONFIG and no-op the mirror) + save_config under the target
profile's HERMES_HOME override — same mechanism as _write_profile_model.
Satisfies test_config_read_guard.
npx hyperframes preview starts a long-lived next-server that keeps
chrome-headless-shell render workers resident. On GPU-less hosts (WSL,
containers, CI) each idle worker falls back to software WebGL (swiftshader)
and busy-spins a CPU core; a preview left open stacks these up until the
host is wedged. The skill never said preview was long-lived, so leaking
was the default outcome.
Add a Cleanup section + pitfall to SKILL.md and a Runaway CPU
troubleshooting entry with diagnose/fix/avoid steps.
Both connect providers captured `sessionId` when the suggestion was built
and ignored the one the pill hands `invoke`. An offer that outlived a
session switch therefore aimed its `reload.mcp` at the session the draft was
sampled in, so the chat the user actually clicked from resumed without the
tools the pill just said were ready.
Prefer the invoking pill's session; the captured one stays as the fallback.
Phase lived in a `Record<key, phase>` that only ever grew, keyed by
`provider:id` — keys that repeat constantly, since a provider withdraws and
re-offers the same suggestion whenever the draft loses and regains its
trigger. A leftover `done` then painted a genuine new offer as "Added
GitHub" and swallowed clicks, because only `idle` invokes.
The same map outlived a session switch. One composer stays mounted across
it, so connecting GitHub in one chat left the next chat's real offer inert.
Withdrawal also stranded in-flight work. The pill is the only cancel
affordance — clicking a working pill sets the flag the provider polls — so
once it left the strip an OAuth flow could poll forever, hold the server's
in-progress slot against a retry, and resolve into a config write with no UI
left to narrate or roll it back.
Phase now lives and dies with the pill: withdrawn keys drop their phase and
flip their cancel flag, unmount cancels everything in flight, and the strip
remounts per session.
The bus's change gate compared offers by `provider:id` alone. Providers
rebuild their suggestion objects on every draft sample, so that key is equal
constantly and the write bailed out — pinning the FIRST object for the life
of the offer.
Two consequences, both user-visible. The pill keeps painting a stale reason
("you mentioned linear" after the user pasted a linear.app link), and it
keeps calling a stale `invoke` closure — work built for a draft that no
longer exists.
Compare the fields the pill actually renders instead. The reference-identity
bail-out survives for the common case (same draft, same match, no re-render),
which is what the gate was there for.
Vercel, Supabase, Netlify, Hugging Face, Asana, Intercom, Airtable,
Webflow, PayPal, and Square join the directory. Every entry is a
vendor-operated remote with its docs page linked, same URL-only rule
as the founding eight. Trigger notes where words are ambiguous:
'square' the English word never fires (squareup only), and
vercel.app/netlify.app deploy-preview hosts are deliberately absent
(a pasted preview link is about the site, not the platform). Brand
glyphs wired for all newcomers.
Two precision guards on the draft-keyword providers, both aimed at the
same annoyance: a pill firing while the trigger is still under the caret.
- Completed-word guard (mcp + skill): a whole-word keyword hit only
counts once at least one character follows it, so the debounce
elapsing mid-thought no longer pops a pill for the word being typed.
Pasted-URL host hits are exempt: pasting is deliberate and the URL
routinely ends the draft.
- Workspace homonym guard (skill): a skill named like the session's
working directory is the project's name, not a request. Working in
~/www/hermes-agent no longer floats 'Use skill: hermes-agent' on
every mention of the repo.
Under a global SSH/remote gateway, resolveProfileBackendRoute routes every
profile to the shared primary backend (case 3) and getConnection() returns
the primary descriptor tagged with the profile. ensureGatewayForProfile
still dialed a per-profile secondary socket at that descriptor; over SSH
the duplicate dial fails (per-backend tunnel/ticket) and the closed socket
became the ACTIVE gateway — every profile except the primary showed
'Hermes gateway is not connected' even though the primary socket was open.
Detect the shared-primary route and activate the primary socket instead;
$activeGatewayProfile still tracks the selected profile so per-request
?profile= scoping is unchanged. Hover pre-warm no-ops on this route.
Local pooled profiles and per-profile remote overrides are untouched
(pinned by test).
- web_server CONFIG_SCHEMA: fold the one-field models_dev category
(models_dev.url) into the agent tab via _CATEGORY_MERGE, matching the
established pattern for single-field categories (slice 7,
test_no_single_field_categories).
- image_routing._lookup_supports_vision: pass allow_network=True to
get_model_capabilities. The vision-capability lookup runs when an
image actually needs routing (not per conversation turn), and the
#31179 text-only-main guard depends on catalog data — with the new
allow_network=False default a cold cache returned 'unknown', which
falls back to attempting the call and reintroduced the #31179
failure shape (slice 8, test_text_only_main_skipped_when_no_
aggregator). This preserves that path's historical
network-on-cold-cache behavior; the fetch stays 4h-TTL cached and
backoff-limited.
- Cold force_refresh (fresh CLI process, e.g. hermes config refresh)
now hydrates the memory cache from disk before fetching, so the
conditional GET actually fires on the flow the feature was built for
instead of silently re-downloading the full ~2 MB registry
(empirically probed: If-None-Match sent, 304 serves disk data).
- Conditional-GET decision is passed in explicitly
(_fetch_models_dev_from_network(conditional=...)) by callers holding
the fetch lock, removing the hidden read of module globals inside
the fetch; the background worker now fetches INSIDE the lock,
symmetric with foreground (true singleflight — no concurrent
double-download, no fetching against mid-commit etag state).
- Corrupt disk cache is QUARANTINED (renamed to .json.corrupt) rather
than left in place: rejection becomes a one-time event instead of a
re-read + re-parse + warning + unlink on every hot-path call while
offline (probed: 1 warning across 5 calls, was 5).
- Dropped the dead _DEFAULT_MODELS_DEV_URL constant; module and
function docstrings updated to match the servable-cache conditional
semantics.
- Conditional GET now requires a servable in-memory registry: an
If-None-Match sent while holding no cache invited a 304 against
nothing, permanently serving {} with a blocking foreground fetch on
every call (the exact #35838 class this PR fixes). Empirically
repro'd and verified fixed (corrupt cache + stale sidecar: was 3
calls -> {} forever; now 1 unconditional fetch -> real data).
- ETag persists atomically WITH the cache body via
_commit_registry -> _save_disk_cache(data, etag), wiring up the
previously-dead etag param; the sidecar can no longer get ahead of
the registry it vouches for. _save_etag now uses
utils.atomic_write_text (unique tempnames + fsync) instead of a
hand-rolled fixed-name .tmp replace.
- Corrupt/unreadable disk cache clears the ETag sidecar so the
refetch is unconditional; _confirm_cache_not_modified keeps a
defense-in-depth guard (clear sidecar + arm backoff) should a 304
ever land on an empty registry.
- allow_network=True paths use the zero-arg fetch_models_dev() call
shape at all sites (was 1 of 5) — ~46 test sites monkeypatch it
with zero-arg lambdas; the unconditional kwarg broke
test_xiaomi_provider (verified fail->pass).
- _get_models_dev_url falls back to the MODELS_DEV_URL module global
(not the constant) so existing patch sites keep working.
- Tests: replaced two mock-riddled corrupt-cache tests with real
tmp_path file tests; added regression tests for the 304/empty-cache
loop, sidecar clearing, and conditional-GET gating.
Harden the models.dev catalog refresh path (#35838) with three missing
pieces:
1. ETag conditional GET — every network request sends If-None-Match
with the last-known ETag (persisted alongside the cache file). A 304
Not Modified re-confirms the existing cache without re-downloading
the full ~2 MB registry. This makes the 4-hour TTL effectively free
to maintain.
2. No-network-on-hot-paths invariant — allow_network=False is now the
default for every query function called on the conversation hot path:
get_model_capabilities, get_model_info, lookup_models_dev_context,
_get_provider_models. These are called during vision routing, image
routing, cost-guard checks, and context-length resolution on every
turn — they must never block on the network. Interactive flows
(model picker, model switch) explicitly pass allow_network=True.
3. Mirror URL override — models_dev.url in config.yaml lets deployments
point at a self-hosted mirror without code changes. Follows the same
pattern as model_catalog.url.
Additional hardening:
- Cache TTL bumped from 1h to 4h (ETag makes refresh cheap)
- Corrupt/empty disk cache is rejected with a warning instead of being
served as {} and silently breaking provider/model resolution
- _validate_registry() guards against non-dict and empty-dict payloads
Fixes#35838
When a prompt_toolkit run_in_terminal cooked->raw restore is lost (cancelled
coroutine, racing chained cross-thread windows from background-review
summaries / process-notification prints), the tty stays in cooked mode while
the Application still expects raw. The kernel line-buffers keystrokes and the
CLI appears to stop taking input even though the event loop is healthy.
Observed live 2026-08-13: interactive session left in 'icanon echo' after a
background skill-review fork + notify_on_complete turn; only an external
stty rescue restored input.
Fix: _heal_cooked_mode_drift() re-applies prompt_toolkit's own raw-mode flag
surgery when stdin's lflag has drifted cooked, and process_loop's idle branch
runs a rate-limited _check_termios_drift() watchdog that skips legitimate
cooked windows (app._running_in_terminal), agent-running phases, non-tty
stdin, and Windows.
patch_session_model_config merges key-level and only deletes on explicit
None. Dropping falsy values from the top-level patch let a previous
switch's api_mode/base_url survive the next switch — TUI/desktop resume
then restored e.g. openrouter with anthropic_messages wire mode, and a
failed bare-custom heal produced a stale-provider/new-endpoint route.
Write absent top-level values as explicit None so each switch fully
replaces the persisted route. Regression test against a real SessionDB;
mutation-checked. Also correct the heal comment (CLI is deliberately
stricter than the TUI recovery, which keeps bare custom with a base_url).
- Bare 'custom' from ModelSwitchResult.target_provider is the resolved
billing class, not a routable identity; persisting it verbatim made a
later --resume hard-fail once the config default moved off the custom
endpoint. Heal to custom:<name> via canonical_custom_identity at
persist time, and again on restore for rows written by older builds
(mirrors tui_gateway's _stored_session_runtime_overrides recovery).
- --global switches now also update the session row: the row records
what THIS session runs, otherwise resume restored the stale
creation-time model over the user's new global choice.
- Only adopt resolved credential_pool alongside its api_key (don't null
the ambient pool when resolution returns no credentials).
- 3 new tests; healing path mutation-checked.
- Extract the two duplicated /model session-persist blocks into
_persist_model_switch_to_session; persist the route BOTH nested
(gateway_runtime, CLI reader) and top-level (TUI gateway's
_stored_session_runtime_overrides reader) so a CLI switch also
survives a desktop/TUI session.resume.
- Add SessionDB.session_gateway_runtime as the canonical tolerant
row-level route reader (session_yolo_enabled precedent); use it in
_restore_session_model instead of hand-rolled JSON parsing.
- Clear stale launch-time _explicit_api_key/_explicit_base_url when
resume restores a different provider (same leak guard
_apply_model_switch_result already has).
- 12 new tests incl. a real-SessionDB round trip; mutation-checked.
Two bugs caused resumed sessions to use the config default model instead
of the model the session was actually using:
1. CLI /model switch didn't persist the new model to the session DB row.
The gateway calls update_session_model() after a /model switch, but
the CLI path only updated in-memory state and the agent's runtime —
it never wrote the new model to the sessions.model column. So the DB
row always kept the original model from session creation.
2. Resume didn't restore model/provider from the session DB row.
_preload_resumed_session and _init_agent restored CWD and YOLO from
session_meta, but never read session_meta['model'] back into
self.model/self.provider. So even if the DB had the right model,
resume would use whatever was in config.yaml.
Fix:
- _handle_model_switch / _apply_model_switch_result: call
update_session_model() after a session-scoped /model switch (skipped
for --once and --global), mirroring the gateway's behavior.
- New _restore_session_model() method: restores model/provider from
session_meta on resume, with provider/base_url/api_mode from
model_config.gateway_runtime. Also swaps the running agent in-place
for mid-chat /resume.
- Call _restore_session_model() from all three resume paths:
_preload_resumed_session, _init_agent, and _handle_resume_command.
- Track _explicit_model_override flag so -m/--model on the CLI overrides
resume (user intent wins). Cleared on /new.
Follow-ups on salvaged #85287:
- discover_entrypoint_manifests() now carries BOTH the import-free kind
classification (from #85527) and capability declarations — the two
contracts compose in one function instead of the capability rewrite
dropping classification.
- Per-entry exception isolation: one malformed distribution no longer
blanks every other plugin's manifest (same contract as
providers/__init__.py entry-point scan).
- Documented the hermes_agent.plugin_capabilities group in the plugin
developer guide (pyproject example).
- Single-source the included note as _INCLUDED_NOTE and attach it at
BOTH status='included' sites (the zero-amount pricing-entry branch
previously returned the same status with no note).
- Docstring/comment precision on format_cost_label: the fallback
triggers on 4dp ROUNDING to 0.0000 (banker's rounding includes the
exact $0.00005 boundary), not truncation; note why the rendered-label
guard beats a naive Decimal threshold.
- Tests: replaced a dead assertion with the exact-boundary case
($0.00005), fixed an overclaiming comment, aligned the terminal
cost column.
- Insights formatters now route aggregate estimated cost through the
shared format_cost_label() instead of hardcoded 2dp — a sub-cent
aggregate (one cheap DeepSeek session, ~$0.0046) no longer renders
'Estimated: ~$0.00', the exact bug class this PR fixes (#79220).
- format_cost_label: positive amounts below $0.00005 render '~$<0.0001'
instead of the zero-looking '~$0.0000' 4dp truncation artifact.
- Renamed _format_cost_label -> format_cost_label (now a cross-module
shared helper).
- Tests: renamed test_gateway_format_hides_cost ->
test_gateway_format_hides_cache_details and
test_no_cost_section_when_all_zero ->
test_unknown_bucket_shown_for_costless_session (names contradicted
behavior); restored a real assertion in the custom-models test that
had been weakened to a comment; added sub-cent-aggregate and 4dp-floor
contract tests (mutation-checked).
Three cost-display honesty fixes:
1. Sub-cent cost label rendering (#79220) — _format_cost_label() scales
precision to magnitude: zero renders as '$0.00', sub-cent (< $0.01)
renders at 4 decimal places (e.g. '~$0.0046'), normal costs keep 2dp.
This fixes the bug where DeepSeek per-turn costs of $0.004640 rendered
as '~$0.00' despite amount_usd carrying full Decimal precision.
2. Cost bucket surfacing (#77223) — insights format_terminal and
format_gateway now display three cost buckets: estimated (with dollar
figure), included (session count, labeled 'subscription — no provider
invoice'), and unknown (session count, labeled 'no pricing data').
Previously, included and unknown sessions silently collapsed to $0 in
the aggregate view, hiding 315 of 473 sessions in the reporter's DB.
3. Subscription-included cost notes — estimate_usage_cost now attaches a
'subscription-included; no provider invoice for usage' note to
CostResult for subscription-included routes (openai-codex), so
consumers can distinguish 'free because subscription' from 'free
because $0 pricing'.
Fixes#79220Fixes#77223
- Deleted the id(cfg)-keyed _OVERRIDE_CACHE layer: id() is unique only
among live objects, so a config reload could serve stale overrides
forever when CPython reuses the freed dict's address. The upstream
load_config_readonly is already (mtime,size)-cached (~1 stat/hit), so
the local layer was redundant state with a correctness risk.
- _override_to_catalog_shape returns (patch, vision) instead of
smuggling an in-band _vision_override sentinel key through the merged
dict; removed the two dead call-site pops.
- _find_model_entry gains the :cloud/-cloud suffix fallback that
lookup_models_dev_context already had, so 'catalog hit' means the
same thing to every consumer — a suffix-keyed model (kimi-k2.6:cloud)
now counts as KNOWN and keeps its catalog capabilities instead of
being displaced by a fill-gap _default (mutation-checked contract
test added).
- get_model_info's unknown-model override path seeds the same safe
defaults as get_model_capabilities (200K ctx, tools on, 8192 out),
so a partial override no longer yields ctx=0/tools-off on that path
(contract test added); the DEFAULT_CONFIG defaults claim is now true
for both paths.
- Activated the previously-dead _MODELS_DEV_TO_PROVIDER reverse map
(lazily built, many-to-one aware) and used it in
_provider_override_section instead of a per-call linear scan.
Review follow-ups on the model_overrides feature:
- ONE canonical override schema everywhere. get_model_info previously
merged the override dict raw into the models.dev catalog shape
({**raw, **override}), so the documented context_window/supports_*
keys silently did nothing on that path (cost guard, inventory) while
working in capabilities/context paths — same config key, two
incompatible schemas. Overrides are now translated into the catalog
shape at the get_model_info boundary (_override_to_catalog_shape),
and sub-dicts (limit, modalities) are MERGED, not clobbered — an
override setting only context_window no longer wipes the catalog's
limit.output.
- _default is now a FILL-GAP default, not an override: it applies only
to models the catalog does not know (the #8731/#84482 self-unblock
path) and never displaces catalog data. A
_default: {context_window: 128000} can no longer clamp every model
of a provider. Explicit per-provider+model entries keep their
win-over-catalog semantics.
- Early-chain _override_context_window (model_metadata step 0b) is
explicit-only, so a _default can never preempt custom_providers
per-model settings or live probes; fill-gap defaults apply at the
lookup_models_dev_context catalog-miss boundary (step 5f) instead.
This fixes the precedence inversion where a provider/global _default
silently overrode an explicit per-endpoint per-model context_length.
- Provider keys accept BOTH id spaces (Hermes id and models.dev id:
copilot/github-copilot both work) and model ids match
case-insensitively, mirroring catalog lookup.
- Malformed override values (context_window: '512k') log a one-shot
warning instead of being silently swallowed.
- DEFAULT_CONFIG comment: removed the false family/dated-snapshot
inheritance claim, documented the recognized field list, fill-gap
semantics, and the id-space rule.
- Tests: rewritten for the new contracts (fill-gap invariants,
dual-id-space keys, sub-dict merge preservation, one-shot warning);
added a real-config-yaml e2e plumbing test (mutation-checked: fails
when the config key wiring is broken).
Add a unified model_overrides config section that lets users manually
declare context_window, max_output_tokens, capabilities, cost, and
family for any provider+model — winning over models.dev, OpenRouter, and
hardcoded defaults.
Resolution order (first hit wins):
1. model_overrides.<provider>.<model_id> (per-provider+model)
2. model_overrides.<provider>._default (per-provider default)
3. model_overrides._default (global default)
4. Normal catalog resolution
Key subtlety: an unknown model id (not in the
catalog) derives base metadata from sensible defaults before patching,
so overriding a model the catalog doesn't know yet is the supported
self-unblock path. This is exactly the #84482 scenario (Upstage
solar-pro4/syn-pro wrong context) and the #8731 scenario (custom/local
models with manual capability declaration).
Wired into:
- get_model_capabilities() — patches capability fields; unknown models
get safe defaults (tools on, vision/reasoning off) before patching
- lookup_models_dev_context() — context_window override, checked before
catalog lookup so it works even for providers not in PROVIDER_TO_MODELS_DEV
- get_model_info() — merges override dict onto catalog entry (shallow
merge); for unknown models, the override is the sole source of metadata
- get_model_context_length() — step 0b in the resolution pipeline,
before custom_providers (0c) and before any network probe
Config example:
model_overrides:
upstage:
solar-pro4:
context_window: 524288
syn-pro:
context_window: 65536
custom:my-local-vllm:
my-llava-model:
context_window: 8192
supports_vision: true
supports_reasoning: false
supports_tools: true
_default:
context_window: 128000
Fixes#8731Fixes#84482
Refs #47247
The set is no longer in parity with agent_init's fail-fast gate (which
still skips openrouter for a different reason: default route, not
unroutable). Say so instead of claiming parity.
BARE_BILLING_PROVIDERS incorrectly included "openrouter" alongside
"auto" and "custom". OpenRouter is a fully routable provider with
its own API key and base_url — sessions that used OpenRouter store
billing_provider="openrouter", and dropping it forces resume to the
current global model (e.g. a custom endpoint), which is the wrong
provider for the stored model.
Remove "openrouter" from the bare-bucket set so OpenRouter sessions
correctly restore their provider identity on resume.
Fixes#57588
cronjob(action=create) has supported a repeat cap since the tool
existed, but the ws handler dropped the param — UIs building on
cron.manage could not create run-N-times jobs. Forward it (digit
strings accepted, None keeps schedule-kind defaults). One-shot
relative schedules (bare 30m/2h) already flow through the schedule
string untouched.
Maintainer fixup on the #79787 salvage:
- An explicit fb.api_mode of "chat_completions" was silently overridden
by the codex_responses / bedrock re-detection pass (which only skipped
re-detection when the pre-computed mode was non-default). Track
explicitness in fb_api_mode_explicit and gate the whole re-detection
block on it.
- Replace the locals().get('fb_api_mode') dead-code hack with clean code
(fb_api_mode is always bound at that point).
- Restore the post-resolve /anthropic + api.anthropic.com host check for
named custom providers whose base_url comes from config rather than
the fallback entry (#32243, #49247), which the PR's restructure dropped.
- Add regression tests: explicit api_mode honored (incl. explicit
chat_completions not overridden), /anthropic-hint fallback detected
pre-rewrite, api_mode forwarded to resolve_provider_client, plain
fallback unchanged.
Fallback activation determined api_mode from the POST-rewrite client
base_url, losing the Anthropic wire signal for /anthropic endpoints
routed through provider 'custom', and never honored an explicit
fb.api_mode config field. Pre-compute fb_api_mode from the ORIGINAL
fallback base_url hint (before _to_openai_base_url rewriting), honor
the explicit api_mode config field, check provider name before the
base_url gate, and pass api_mode into resolve_provider_client at the
fallback call site.
Salvaged from PR #79787 (chat_completion_helpers.py hunks; the
auxiliary_client.py hunk is redundant with #85466's wrap_base fix).
emozilla
Teknium
ethernet
Shannon Sands
Ben Barclay
Gille
Brooklyn Nicholson
kshitij
Jeeves Assistant
liuhao1024
sjungwon03
Xie