The subprocess import test was creating .tmp-hermes-exec-ask-import/
in the repo root without cleanup. Switch to pytest's tmp_path fixture
so the temp directory is auto-cleaned and never appears as untracked.
Regression tests for silent pending_approval when ask-mode leaks into
interactive CLI, plus a Path-typed hermes_constants mock so the
slash-worker profile_home test survives per-file isolation.
test_slash_worker_accepts_profile_home mocks hermes_constants with
get_hermes_home=MagicMock(return_value="/tmp/hermes_test"), a str. In
production get_hermes_home() returns a Path, and hermes_state.py's
module-level DEFAULT_DB_PATH = get_hermes_home() / "state.db" does path
division. Under the str mock that becomes str / str, so importing
tui_gateway.server inside the patch raises TypeError and the test fails on
every main run (slice 4). Wrap the mock return in Path(...) so it matches the
real return type. Test-only; no production code change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Salvage of #79604 (webtecnica) + #85721 (pierrenode), combined and
rebased onto current main with simplify-code findings folded in.
#79604: update_session_model() wrote the model name to sessions.model
but never persisted the provider into model_config. On resume, the
runtime recombined the persisted model with the config.yaml primary
provider (which may not serve that model), producing auth errors.
Fix: add optional provider parameter to update_session_model, merged
into model_config via the shared _merge_model_config_json helper (not
hand-rolled SQL). Wire both gateway /model call sites to pass
result.target_provider.
#85721: session_gateway_runtime() had no billing_provider fallback.
A CLI session that never ran /model has no gateway_runtime or
top-level provider in model_config — billing_provider (written on
every session's first accounted API call) is the only durable record.
Fix: add billing_provider as the last-resort fallback in
session_gateway_runtime(), filtering bare billing buckets (auto/custom)
that are not routable identities.
Simplify-code findings addressed:
- Use _merge_model_config_json instead of 40 lines of branched SQL
- Share _BARE_BILLING_PROVIDERS from hermes_state.py (was duplicated
as a set in tui_gateway/server.py)
- Merge None-filtering from #85920 with the billing_provider fallback
into one coherent return path
Co-authored-by: pierrenode <298902573+pierrenode@users.noreply.github.com>
54cc39aa15 (distrust foreign pricing for custom providers) tested with
openai/gpt-5.5-pro fixtures; 83d373aae6 (salvaged #70324) made that exact
id warn unconditionally as a known-confusion model. Each was green alone;
together the distrust tests fail on every main run (slice 6).
Use a neutral fixture id for the distrust tests and add a regression test
pinning the composed behavior: the id-keyed nudge survives custom-provider
pricing distrust.
Run the expensive-model warning for explicit startup `-m` / `--provider`
overrides before the chat loop starts, and fail closed for non-interactive
invocations that select an expensive or known-confusing model.
Also classify Nous paid-model 404s that say credits are required as billing
exhaustion so they fail fast with billing guidance.
Tested:
- scripts/run_tests.sh tests/hermes_cli/test_cli_startup_model_cost_guard.py tests/hermes_cli/test_model_cost_guard.py tests/agent/test_error_classifier.py -- --tb=short -q
Custom providers (custom:xxx) serve their own pricing; models.dev stores
OpenRouter prices for the same model ids. The cost guard fired on that
foreign pricing and blocked composer/CLI model switches on custom
providers with a wildly wrong warning (#54348).
expensive_model_warning now only trusts model_info/models.dev pricing
when the provider maps to a models.dev provider and the info's
provider_id matches, and only consults the pricing-entry lookup when
the billing route is known. Salvaged from #54422; the PR's desktop-hook
half predates the use-model-controls rewrite and is superseded by the
hook's existing rollback handling.
The command path (/model <name> --provider <p>) called
_confirm_expensive_model_switch() inline. That modal blocks its calling
thread on a response queue (see _prompt_text_input_modal); on the
prompt_toolkit main thread the TUI event loop freezes, the modal never
renders, and the switch silently cancels after the 120s timeout — the
user sees a frozen terminal and 'Model switch cancelled.' without ever
seeing the warning. The picker path already dispatched confirm+apply on
a worker thread; the command path now mirrors that contract.
Extract the inline confirm+apply block into
_confirm_and_apply_cli_model_switch() (preserving --once restore and
persist semantics) and dispatch it on a daemon thread when a TUI app is
present, keeping the synchronous path for non-interactive/test use.
Tests: new test_model_switch_confirm_thread.py pins (a) confirm runs off
the main thread when _app is present, and (b) the no-app path stays
synchronous. Existing _StubCLI helpers forward to the extracted method.
Plain type=completion events built in _run_process_watcher carried only
session_key (chat/thread routing) with no spawning-session stamp, so after
/new (or a session switch) a completion notification from the OLD session
was injected into the chat's NEW session. Main already solved this exact
class for async delegations via the _classify_completion_target pre-flight
(_USER_BOUNDARY_END_REASONS drop on user-closed sessions, deliver on
idle-ends, follow the compression-tip chain), but the gate only ran for
type=async_delegation events.
Kernel salvage of #16455:
- Stamp the spawning conversation's session-db id (HERMES_SESSION_ID via
session-scoped env) on the ProcessSession and the pending_watchers entry
at spawn time in tools/terminal_tool.py; persist it through the process
registry checkpoint/restore so recovered watchers keep the stamp.
- Thread the stamp into the completion_evt built by _run_process_watcher
(watcher entry first, ProcessSession fallback for recovered watchers).
- In _deliver_completion_notification, run the SAME pre-flight classifier
for stamped type=completion events: terminal -> drop with a log (output
stays available via process(action='log')), retry -> False so the
watcher re-polls, deliver -> proceed. The policy has exactly one owner
(_classify_completion_target); nothing is forked. Unstamped legacy
events keep today's deliver-always behavior, and the async-delegation
path is untouched.
Based on the session-boundary approach from #16455 by @Tosko4 (original PR
was over-scoped across adapters/slash-commands/cron; this lands the kernel
only).
Tests: completion from a /new-closed session is dropped; completion after
an idle-end still delivers; unstamped legacy event delivers; retry verdict
returns retryable False without adapter injection; async_delegation gate
unchanged; stamp survives checkpoint recovery.
The async-delegation watcher drained the completion queue as a batch but
then delivered each event as its own synthetic turn, flooding the session
when a fan-out of background subagents finished together. Builds on the
per-process completion batching salvaged from PR #71898 (thanks
@yuzilongleif-collab) which coalesces concurrent _run_process_watcher
completions behind a short per-route fan-in window.
This commit adds the async-delegation half: group the drained batch by
full routing key (session_key + parent_session_id + platform/chat/thread/
user) and inject ONE consolidated turn per group. Durable-ack handling
stays honest: sibling rows are claimed up front via claim_event_delivery;
rows another consumer owns are excluded from the consolidated text (no
double-delivery); sibling claims are acknowledged only after adapter
acceptance and released (still pending) on failure. Events for different
sessions never coalesce, and a single-event group rides the existing
per-event path unchanged (latency and text identical).
Tests: 3 same-tick events -> exactly one adapter.handle_message carrying
all 3 results with all 3 durable rows delivered; 2 sessions -> 2 turns;
single-event path unchanged; failed batch releases claims and retries;
foreign-claimed sibling excluded and left pending.
Async-delegation batch completions and background watch notifications
re-enter the gateway as synthetic MessageEvent(internal=True) turns via
_inject_watch_notification, but were persisted as bare role='user' rows —
indistinguishable from real user input in transcripts and the desktop UI.
Thread the event's internal flag through to persistence: when
event.internal is set, the turn's persisted user row is stamped
display_kind='internal_notification' (the existing DB-only presentation
sidecar used by auto_continue / model_switch rows). Wired through
_run_agent → _run_agent_inner → TurnContext → run_conversation's
persist_user_display_kind, and onto the three gateway-side fallback user
rows (transient failure, no-new-messages, pre-run crash), whose
append_to_transcript writer now forwards display_kind/display_metadata
to SessionDB.append_message.
Invariants preserved: role stays 'user' (alternation untouched), no new
injections, no past-context mutation, and display_kind is already popped
from every provider-bound copy in conversation_loop, so replayed sessions
never leak the marker to the API.
Regression tests: internal turn marked, real user turn unmarked, fallback
rows marked/unmarked per event, and a DB round-trip proving replay keeps
role/content intact while the provider copy drops the marker.
Add TestNotificationRedaction class with two tests:
1. test_completion_notification_redacts_secret — verifies _move_to_finished
redacts API keys in completion notifications before enqueueing
2. test_watch_match_notification_redacts_secret — verifies _check_watch_patterns
redacts secrets in watch_match notifications before enqueueing
These tests cover the gap identified in #43025 where the explicit process
tool path (poll/log/wait) was redacted but the automatic notification
delivery path was not.
format_process_notification had no case for watch_overflow_tripped /
watch_overflow_released, so a watch-pattern notification flood surfaced
as '[IMPORTANT: Background process exited (exit code ?)]' — a phantom
exit notification for a process that never existed — while the actual
'watch flood, N notifications suppressed' summary in the event's
message field was silently dropped. The gateway delivery path was
worse: _drain_gateway_watch_events retained only watch_match and
watch_disabled, discarding overflow events entirely before formatting.
Route both event types through the message field in the shared
formatter and the gateway formatter, and retain them in the gateway
drain.
Adds hermes_cli/model_selection_guards.py: a single evaluation point that
runs every selection guard (cost + the new data-policy guard) and returns
the warnings that fired. All seven model-selection surfaces (CLI picker,
cli.py TUI modal, gateway typed /model, dashboard web_server, TUI gateway,
Telegram and Discord pickers) now call the registry instead of importing
model_cost_guard directly — so the data-training-tier warning from
PR #81416 fires everywhere at once, and future guards need zero surface
wiring.
Guard modules keep their public APIs; existing mock patch points
(hermes_cli.model_cost_guard.expensive_model_warning) remain valid.
muse-spark-1.2-contributor is heavily discounted BECAUSE Meta uses your
prompts and completions to train future models. Selecting it for the price
without realising the data trade-off is a footgun.
Add hermes_cli/model_data_policy_guard.py (mirrors model_cost_guard):
data_training_warning(model_id, provider, base_url) -> DataTrainingWarning|None,
driven by a vendor-agnostic rule table. The status is not machine-readable on
/v1/models or models.dev, so the v1 rule keys on the documented '-contributor'
model id (fires regardless of provider, so it also covers custom/gateway
routes). Message mirrors Meta's pricing-doc language and figures
(https://dev.meta.ai/docs/pricing-rate-limits/).
Wire it into the CLI model picker's confirm flow (auth.py) as a [y/N]
disclosure, chained after the expensive-model cost guard. Fires only on the
contributor tier; silent on muse-spark-1.1/1.2 and all other models.
The route dict in _persist_model_switch_to_session used filtering
(omits falsy values) while the top-level keys used (writes
explicit None to trigger deletion in _merge_model_config_json). This
asymmetry meant stale keys from a previous /model switch survived in
the nested gateway_runtime dict even after the fix in #85261 that
properly deleted them from the top-level keys.
Fix: build the route dict with and derive the top-level keys
from **route so both shapes always use identical deletion semantics.
Also filter None values in session_gateway_runtime's reader since
gateway_runtime is replaced as a whole dict (not deep-merged), so None
values written by the persist path survive in the nested dict.
Found by /simplify-code 3-reviewer review on #85261 (all 3 reviewers
converged on the route dict asymmetry as the verdict-relevant finding).
Two tests in TestReapUnsupervisedGatewayOrphansMacOS:
- test_macos_excludes_launchd_pid_from_kill: verifies a launchd-managed
PID is not SIGTERM'd while a real orphan is
- test_macos_no_orphans_when_only_launchd_gateway_running: verifies the
reaper returns False when the only gateway PID is launchd-managed
Both tests patch is_macos() to True and supports_systemd_services() to
False to simulate the macOS code path where the short-circuit does not
fire.
Background process completions on messaging platforms now default to a
one-line status message (✅/❌ + command + duration; failures append a
short output tail) instead of dumping the raw output buffer into the
chat. New display.background_process_notifications mode 'concise' is
the default; 'all' keeps the old raw-dump behavior for anyone who wants
it. Config migration v35 moves users still on the old implicit default
'all' to 'concise' on their next update; explicit result/error/off
choices are preserved.
A provider-confirmed rearm (#85846) resets the shared attempt budget, but
an earlier insufficient-progress verdict left _preflight_compression_blocked
armed, keeping the pre-API gate dark for the rest of the turn — a later
pressure spike could still grow unchecked until the provider overflow
handler fired. Clear the blocker and the stale pressure reading inside the
provider-confirmed rearm branch: the prompt is proven back below the
threshold, so the old verdict describes a request shape that no longer
exists.
Builds on @h-mascot's #84995, whose commit is preserved on this branch;
his rearm condition was superseded by #85846's latch-verified variant, but
the blocker-clear half was correct and is kept.
* fix(cli): --in accepts Git Bash / MSYS paths on Windows
Under Git Bash, 'hermes chat --in ~' reaches the CLI as /c/Users/<user>
(the shell expands ~ to an MSYS POSIX path; MSYS2 argument conversion
is disabled for native executables), and the isdir check failed with
'--in directory not found: /c/Users/...'. Route the value through the
existing _msys_to_windows_path translator (MSYS + Cygwin + WSL drive
spellings; no-op elsewhere) before expanduser/abspath.
Hit live: Bot Mode's agent-messaging protocol delivers with --in ~, so
every bot-to-bot send from a Git Bash-driven agent failed on Windows.
Tests pin both the translation cases and (source-level) the call site
actually using it.
* test: assert the MSYS translation, not platform abspath
The prior assertion ran os.path.abspath on the translated Windows path,
which on the Linux CI runner (posixpath) treats 'C:\Users\alice' as
relative and prepends the runner cwd. Pin the translation output and
ntpath absoluteness instead — same contract, platform-independent.
Windows contributors' tools default to CRLF; without repo-level
normalization an edit becomes a whole-file phantom diff (583-line
'change' observed today from one 2-line edit), string-match patch
tooling breaks on invisible \r, and review is polluted. Extend the
existing LF rules (shell/Docker) to every source/text extension:
normalize at check-in AND check out as LF so working trees match the
index on all platforms. *.ps1 stays CRLF (PowerShell 5.1 tooling).
git add --renormalize: exactly one tracked file had mixed endings
(tests/tools/test_windows_agent_loop_papercuts.py) — normalized here,
so no phantom diffs land on anyone's next commit.
A marathon tool turn burned all compression_attempts on *successful*
pre-API compactions; the gate then went permanently dark and the context
grew unchecked until the provider rejected the request terminally
("Context length exceeded: max compression attempts (3) reached", session
f087963205f9, 2026-08-01). The budget now refunds at loop top when the
assembled request is back under threshold * 0.8 AND the compressor's own
should_compress() agrees the pressure is gone.
Anti-thrash intent of the cap is preserved (#11529): no-progress passes
never reach the refund margin, divergent-signal cases (should_compress
still True) keep the budget burnt, and the insufficient-progress blocker
is untouched.
Validation: new behavioral suite (7) + all 130 compression/context tests
via scripts/run_tests_hermetic.py.
Rückbau: Commit revertieren; kein Zustand, keine Migration.
(cherry picked from commit 041b489d56)
Stop freezing the xAI/xAI-OAuth catalog at import so /model and setup
pick up new Grok IDs after the models.dev cache refreshes. Put xai and
xai-oauth on the shared picker-time models.dev merge path and pin
grok-4.6 as the default headline model.
Box cloud content management via the official @box/cli through the
terminal tool: files, folders, sharing, search, metadata, Box AI,
Hubs, bulk operations, webhooks, and a REST fallback via box request.
OAuth-only auth; SKILL.md routes to ten scoped reference files.
Salvaged from PR #52107 by @iskysun96.
Regression tests salvaged from PR #70522 by @JoaoMarcos44. The Qwen flat
cached_tokens behavior is provided by the shared top-level fallback from
PR #66105 (@mehmetkr-31); the codex cache_write_tokens read landed in the
previous commit.
Salvaged from PR #85702 by @JoaoMarcos44, composed onto the mapping-safe
_usage_get reads (PR #74591 by @RelaxJonh) and the flat cached_tokens /
Anthropic-name fallbacks (PRs #66105, #52571):
- cache-write precedence in the chat_completions branch:
details.cache_write_tokens > details.cache_creation_input_tokens >
usage.cache_creation_input_tokens > usage.cache_write_tokens
- codex_responses branch reads details.cache_write_tokens (GPT-5.6+
documented name) with cache_creation_tokens fallback (from PR #70522)
- _usage_count(): clamp malformed negative counters to 0
- all reads in every branch are mapping-safe via _usage_get
When the Responses API returns usage as a plain dict (e.g. from a
middleware or proxy that deserialises JSON to dict instead of a typed
SDK object), normalize_usage() used getattr() exclusively, which
silently returned 0 for every field on a dict.
Add _usage_get() helper that reads via .get() for dicts and getattr()
for attribute-style objects. All accessor sites in normalize_usage()
now use this helper, so token counts and cost are correct regardless
of the usage object's type.
Regression tests: two new tests feed the same payload as both a dict
and a SimpleNamespace through the codex_responses and
chat_completions branches, asserting identical output and non-zero
values.
Kimi/Moonshot's native API (api.moonshot.cn / .ai) reports context-cache hits
as a top-level ``usage.cached_tokens``. The chat-completions branch of
normalize_usage() walks a fallback chain of
prompt_tokens_details.cached_tokens -> cache_read_input_tokens ->
prompt_cache_hit_tokens; none of those names match, so direct Kimi sessions
normalized to cache_read_tokens=0. The hits were invisible in accounting and
the cached prefix was billed at the full input rate.
Appended as the last link in that chain, so it only fills a genuine zero and
cannot override a provider that reports the nested OpenAI shape or DeepSeek's
prompt_cache_hit_tokens.
Rebuilt on current main rather than rebased — the branch was ~3400 commits
behind. The DeepSeek half of the original branch is dropped: 03c0b00f4
(#65678) landed prompt_cache_hit_tokens on main, so this is Kimi-only as the
review asked. The scripts/release.py addition to the frozen LEGACY_AUTHOR_MAP
is dropped too; contributors/emails/mehmet.kar@std.yildiz.edu.tr already
exists on main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The command wrapper prints the cwd marker after the command returns. A
killed or timed-out command emits no marker, so ``env.cwd`` still holds the
directory of the last command to FINISH. One local environment serves every
session, because ``_resolve_container_task_id`` collapses cwd-only overrides
to ``"default"``. That leftover directory is therefore routinely another
session's.
The post-command dual-write copied ``env.cwd`` into the interrupted session's
durable record. Every later command in that session then ran in the foreign
directory, and the cwd echo told the model it had moved there. A desktop chat
silently re-homed into a worktree that another chat had opened.
Report the observation instead of inferring it. The marker parse now sets
``result["cwd_observed"]``, and both the record write and the echo read that
flag. The local override clears the flag when it rolls back a path that does
not exist, because the restored value is also unobserved. When a command
reports no cwd, the session keeps the directory it already had.
This needs no second session to be wrong: a lone session that interrupts a
command re-adopts a stale value too. A second session only makes the wrong
directory belong to somebody else.
The same class of write exists in the file-tools rescue for a reaped
environment (#26211). That rescue copied the cached snapshot of the shared
``env.cwd`` into the session record. The rescue is now fill-only: it writes
the snapshot when the session has no record, and it never overwrites a
record that the session wrote for itself.
The tests drive ``terminal_tool`` itself through an interrupt, not a copy of
its gate. Review found that a revert of either call site passed the first
version of the tests. Each gate now has a test that fails when the gate is
removed (verified by mutation).
Two exact-dict assertions in the Vercel sandbox tests now assert the two
fields they care about, so a new result key does not fail them.
Extends the NS-656 memory-pressure surface to cover disk exhaustion
(OOF-2 / OOF-107 lineage: agents fill their data volume — SQLite writes
fail, sessions stop persisting — while every dashboard looks healthy).
- gateway/disk_status.py (new): collect_disk_status() samples
shutil.disk_usage(HERMES_HOME) and classifies pressure
(critical: <256 MB free or >=95% used; elevated: <512 MB free).
Never raises — degrades to pressure="unknown" with null telemetry,
same contract as collect_memory_status().
- /api/status: sibling `disk` block next to `memory`, advisory only —
not folded into component/overall health.
- web: DiskPressureStatus type; MemoryPressureBanner generalized to a
resource banner with worst-first triggers (disk critical > memory
critical > OOM restart > disk elevated > memory elevated) and
cascading dismissals — hiding the top trigger surfaces the next one
instead of silencing everything. All dismissals stay boot_id-scoped.
- i18n: diskCriticalBanner / diskElevatedBanner (en, optional fields
with English fallback per existing pattern).
Tests: gateway/test_disk_status.py (14), web_server disk-block
presence/degradation, banner disk trigger/priority/dismissal-cascade
suite (21 total).
Addresses the human review findings on the memory-pressure feature:
* [P2] Dismissal hid later incidents of the same kind. The gateway now
publishes `boot_id` (the lifecycle sentinel's started_at — changes on
every gateway life) in the /api/status memory block, and the dashboard
keys OOM-restart dismissal on it: acknowledging one restart no longer
mutes the NEXT one (the OOM-loop case this banner exists for). Live
pressure dismissals now also reset once pressure is demonstrably back
to "ok" — "unknown" (stale heartbeat) is absence of evidence and
clears nothing. Dismissal storage moved to a JSON list; old bare-string
entries fail JSON.parse and degrade to a clean reset.
* [P2] suspected_oom is a heuristic (unclean exit + low-memory final
heartbeat), not proof the OOM killer acted — banner copy now says
"restarted unexpectedly, most likely because it ran out of memory"
instead of stating OOM as fact.
* [P3] Mobile header clearance was applied per-banner (mt-14 on both
MemoryPressureBanner and ProfileScopeBanner) AND on the content
(pt-14), double/triple-stacking 56px gaps when banners were visible.
Replaced with a single h-14 spacer above the banner stack.
Hosted agents can be OOM-killed hourly while the dashboard and the NAS
agent card both look perfectly healthy — every memory signal the gateway
already produces (heartbeat mem samples, lifecycle-ledger unclean-exit
verdicts, cache-pressure evictions) dies in server-side log files. The
BlueAtlas incident (NS-608) ran for three days like this.
This is the read-side fix:
* New gateway/memory_status.py distills the existing 30s loop heartbeat
(gateway RSS + system MemAvailable/MemTotal + swap) and the lifecycle
sentinel into a compact `memory` block: pressure ok/elevated/critical/
unknown, coarse MB numbers, and last-boot unclean/suspected-OOM flags.
Pure file reads, no new sampling, no gateway IPC. Stale (>150s) or
future-dated heartbeats degrade pressure to "unknown" so a dead
gateway's final gasp can't render a live "critical" banner forever.
Critical thresholds mirror the ledger's OOM-suspicion heuristics: if a
level would make a later unclean death "suspected OOM", warn at that
level while the process is still alive.
* lifecycle_ledger.record_startup now carries prior_unclean_exit /
prior_suspected_oom onto the reclaimed sentinel — previously the
verdict survived only in append-only diag prose. Flags age out on the
next sentinel rewrite (scoped to the life after the crash).
* /api/status serves the block (profile-aware, executor-offloaded,
fail-safe to pressure=unknown). Deliberately NOT folded into
components/overall: memory pressure is advisory, and flipping overall
to "degraded" on it would page NAS's availability sweep for a
condition the eviction valve is already handling. Public-safety:
coarse numbers/enums/booleans only — same disclosure class as the
existing nous_session_valid field, added for the same NAS-sweep
audience.
* Dashboard: new MemoryPressureBanner (app-shell, next to
ProfileScopeBanner) with worst-first trigger precedence
(critical > suspected-OOM restart > elevated), per-trigger
session-scoped dismissal, and escalation re-opening past a dismissal.
i18n keys optional with English fallbacks, matching the
managingProfileBanner convention.
Tests: gateway/test_memory_status.py (classification bands, staleness,
clock skew, corrupt files, bool-is-not-int), lifecycle sentinel
carry-forward, /api/status contract (block always present, collector
crash degrades instead of 500), and 7 banner component tests.
NAS-side ingestion (agent-card notice + memory-tier upsell) ships
separately.
Refs NS-656; context: NS-608, NS-657, OOF-77.
Azure Foundry's OpenAI-compatible Responses surface rejects the post-tool
follow-up payload with HTTP 400 `invalid_payload` when a replayed encrypted
`reasoning` item is sent alongside `function_call` / `function_call_output`.
The initial function-call request and ordinary multi-turn continuity are both
accepted, so the failure only appears after the first tool executes.
Detect the Foundry endpoint in `ResponsesApiTransport.build_kwargs` and drop
only the encrypted reasoning replay on that follow-up turn, leaving
function_call / function_call_output continuity intact.
Salvage of #59981, rebuilt on current main. Same root cause and fix direction
as the original, which was correct; this version resolves three defects:
- No `chat_completion_helpers.py` change. main already forwards `provider`
and `base_url` to the Responses transport, so the original's re-added
arguments produced `SyntaxError: keyword argument repeated: provider` on
merge. Dropping the hunk removed the syntax error and the conflict.
- Host matching uses `utils.base_url_host_matches`, not a substring test.
`".services.ai.azure.com" in base_url` also matches URLs carrying the
domain in a path or query segment, which would silently disable reasoning
replay on an unrelated provider.
- The post-tool predicate tests the trailing messages, not the whole history.
Scanning for any tool call plus any tool result made it sticky: one tool
call early in a conversation suppressed reasoning on every later turn.
- Tool calls pair on `call_id` as well as `id`. Responses histories carry the
function call id in `call_id` while `id` holds the response item id
(`fc_...`). Identity is resolved via the converter's own
`_split_responses_tool_id`, covering composite `"call_x|fc_y"` ids and bare
`fc_` ids on both sides of the pairing.
Tests: 27 cases across the transport and the live `build_api_kwargs` bridge,
including six parametrized tool-call id shapes, non-Foundry host lookalikes,
the sticky-history guard, parallel tool results, and an unpaired tool result.
Each guard was confirmed to catch its defect by reverting the fix.
Verified with `scripts/run_tests.sh tests/agent/ tests/run_agent/`:
532 files, 5602 tests passed, 0 failed.
Not verified against a live Azure Foundry endpoint — no credentials. The
original HTTP 400 reproduction and post-fix Foundry Project / Azure Container
Apps harness runs are @AshuJoshi's, from #59981. This change is verified at
the payload-construction layer only.
Closes#59981.
Co-authored-by: Ashu Joshi <AshuJoshi@users.noreply.github.com>
- Cold force_refresh (fresh CLI process, e.g. hermes config refresh)
now hydrates the memory cache from disk before fetching, so the
conditional GET actually fires on the flow the feature was built for
instead of silently re-downloading the full ~2 MB registry
(empirically probed: If-None-Match sent, 304 serves disk data).
- Conditional-GET decision is passed in explicitly
(_fetch_models_dev_from_network(conditional=...)) by callers holding
the fetch lock, removing the hidden read of module globals inside
the fetch; the background worker now fetches INSIDE the lock,
symmetric with foreground (true singleflight — no concurrent
double-download, no fetching against mid-commit etag state).
- Corrupt disk cache is QUARANTINED (renamed to .json.corrupt) rather
than left in place: rejection becomes a one-time event instead of a
re-read + re-parse + warning + unlink on every hot-path call while
offline (probed: 1 warning across 5 calls, was 5).
- Dropped the dead _DEFAULT_MODELS_DEV_URL constant; module and
function docstrings updated to match the servable-cache conditional
semantics.
- Conditional GET now requires a servable in-memory registry: an
If-None-Match sent while holding no cache invited a 304 against
nothing, permanently serving {} with a blocking foreground fetch on
every call (the exact #35838 class this PR fixes). Empirically
repro'd and verified fixed (corrupt cache + stale sidecar: was 3
calls -> {} forever; now 1 unconditional fetch -> real data).
- ETag persists atomically WITH the cache body via
_commit_registry -> _save_disk_cache(data, etag), wiring up the
previously-dead etag param; the sidecar can no longer get ahead of
the registry it vouches for. _save_etag now uses
utils.atomic_write_text (unique tempnames + fsync) instead of a
hand-rolled fixed-name .tmp replace.
- Corrupt/unreadable disk cache clears the ETag sidecar so the
refetch is unconditional; _confirm_cache_not_modified keeps a
defense-in-depth guard (clear sidecar + arm backoff) should a 304
ever land on an empty registry.
- allow_network=True paths use the zero-arg fetch_models_dev() call
shape at all sites (was 1 of 5) — ~46 test sites monkeypatch it
with zero-arg lambdas; the unconditional kwarg broke
test_xiaomi_provider (verified fail->pass).
- _get_models_dev_url falls back to the MODELS_DEV_URL module global
(not the constant) so existing patch sites keep working.
- Tests: replaced two mock-riddled corrupt-cache tests with real
tmp_path file tests; added regression tests for the 304/empty-cache
loop, sidecar clearing, and conditional-GET gating.
Harden the models.dev catalog refresh path (#35838) with three missing
pieces:
1. ETag conditional GET — every network request sends If-None-Match
with the last-known ETag (persisted alongside the cache file). A 304
Not Modified re-confirms the existing cache without re-downloading
the full ~2 MB registry. This makes the 4-hour TTL effectively free
to maintain.
2. No-network-on-hot-paths invariant — allow_network=False is now the
default for every query function called on the conversation hot path:
get_model_capabilities, get_model_info, lookup_models_dev_context,
_get_provider_models. These are called during vision routing, image
routing, cost-guard checks, and context-length resolution on every
turn — they must never block on the network. Interactive flows
(model picker, model switch) explicitly pass allow_network=True.
3. Mirror URL override — models_dev.url in config.yaml lets deployments
point at a self-hosted mirror without code changes. Follows the same
pattern as model_catalog.url.
Additional hardening:
- Cache TTL bumped from 1h to 4h (ETag makes refresh cheap)
- Corrupt/empty disk cache is rejected with a warning instead of being
served as {} and silently breaking provider/model resolution
- _validate_registry() guards against non-dict and empty-dict payloads
Fixes#35838
When a prompt_toolkit run_in_terminal cooked->raw restore is lost (cancelled
coroutine, racing chained cross-thread windows from background-review
summaries / process-notification prints), the tty stays in cooked mode while
the Application still expects raw. The kernel line-buffers keystrokes and the
CLI appears to stop taking input even though the event loop is healthy.
Observed live 2026-08-13: interactive session left in 'icanon echo' after a
background skill-review fork + notify_on_complete turn; only an external
stty rescue restored input.
Fix: _heal_cooked_mode_drift() re-applies prompt_toolkit's own raw-mode flag
surgery when stdin's lflag has drifted cooked, and process_loop's idle branch
runs a rate-limited _check_termios_drift() watchdog that skips legitimate
cooked windows (app._running_in_terminal), agent-running phases, non-tty
stdin, and Windows.
patch_session_model_config merges key-level and only deletes on explicit
None. Dropping falsy values from the top-level patch let a previous
switch's api_mode/base_url survive the next switch — TUI/desktop resume
then restored e.g. openrouter with anthropic_messages wire mode, and a
failed bare-custom heal produced a stale-provider/new-endpoint route.
Write absent top-level values as explicit None so each switch fully
replaces the persisted route. Regression test against a real SessionDB;
mutation-checked. Also correct the heal comment (CLI is deliberately
stricter than the TUI recovery, which keeps bare custom with a base_url).
- Bare 'custom' from ModelSwitchResult.target_provider is the resolved
billing class, not a routable identity; persisting it verbatim made a
later --resume hard-fail once the config default moved off the custom
endpoint. Heal to custom:<name> via canonical_custom_identity at
persist time, and again on restore for rows written by older builds
(mirrors tui_gateway's _stored_session_runtime_overrides recovery).
- --global switches now also update the session row: the row records
what THIS session runs, otherwise resume restored the stale
creation-time model over the user's new global choice.
- Only adopt resolved credential_pool alongside its api_key (don't null
the ambient pool when resolution returns no credentials).
- 3 new tests; healing path mutation-checked.
- Extract the two duplicated /model session-persist blocks into
_persist_model_switch_to_session; persist the route BOTH nested
(gateway_runtime, CLI reader) and top-level (TUI gateway's
_stored_session_runtime_overrides reader) so a CLI switch also
survives a desktop/TUI session.resume.
- Add SessionDB.session_gateway_runtime as the canonical tolerant
row-level route reader (session_yolo_enabled precedent); use it in
_restore_session_model instead of hand-rolled JSON parsing.
- Clear stale launch-time _explicit_api_key/_explicit_base_url when
resume restores a different provider (same leak guard
_apply_model_switch_result already has).
- 12 new tests incl. a real-SessionDB round trip; mutation-checked.
kshitij
xxxigm
Laura
webtecnica
Teknium
lkz-de
Dustin Persek
Turgut Kural
dsad
yuzilongleif-collab
yuzilongleif-collab
spfcraze
Lidang-Jiang
fangliquanflq
Beto de Paola
Shedrack Eze
Henry Mascot
Gille
ai-ag2026
Jaaneek
chriskim
joaomarcos
RelaxJonh
mehmetkr-31
ethernet
Shannon Sands
Ben Barclay