The sole-credential cooldown sized the bench from the raw HTTP status, but
403 is overloaded: error_classifier maps OpenRouter's "key limit exceeded"
and xAI's spending-limit block to FailoverReason.billing, while an edge
throttle with the same status is transient. Only 402 was excluded from the
short cooldown, so a spent account on a single key retried every 60 seconds
and re-failed forever.
Thread the classified reason from recover_with_credential_pool through
mark_exhausted_and_rotate to _exhausted_ttl. Billing keeps the full bench
regardless of status; everything else transient still recovers in 60s. The
verdict is stored on the entry (_EXTRA_KEYS, so it persists to auth.json) —
without that a restart would re-read a bare 403 and downgrade the bench.
Tests: sole billing-403 stays benched, survives reload, unclassified 403
still recovers; call-site coverage that the reason actually reaches the pool.
Three existing kwargs assertions updated for the new argument.
Replace the fixed 60-second cooldown with exponential backoff:
30min → 1h → 2h → 4h cap.
The counter is reset by restore_primary_runtime on successful
primary-provider recovery, so the backoff is strictly for
consecutive failures within a single degradation window.
Closes#29702
restore_primary_runtime retries the primary every turn once the 60s
transient cooldown clears. For subscription-window limits (Claude
Pro/Max 5h windows, Codex weekly caps) the reset is hours or days away,
so every retry is a guaranteed failure costing two provider switches
and two prompt-cache invalidations per turn.
Add CredentialPool.next_available_at() (earliest reset across exhausted
entries; None when available now or no reset info) and gate the restore
on it: skip while the primary's pool says nobody can serve, restore on
the first turn after the reset elapses. Fail-open: any gate error or
missing reset info falls through to the existing per-turn retry, so
recovery can never be later than today. Cross-provider fallbacks
consult the PRIMARY's pool (not the attached fallback pool), reusing
the loaded pool for the existing rebind to keep auth reads at one per
restore.
OpenCode Zen's relay rejects the Anthropic-style content block format
that cache markers produce (content becomes a block array instead of a
plain string), causing HTTP 400 with "content must be string, not block
array" for DeepSeek models.
Reverts the DeepSeek addition from commit 6b6435a874 while preserving
the Qwen/Alibaba caching path which continues to work.
Fixes#77217
Simplify-pass follow-up on the #69653 salvage: the original code built
these patterns from a name loop; hand-expanding them into 10 literals
lost that single source. Tag-name tuples restore it (adding a 6th
reasoning tag is now a one-place change), and the gnarly named-function
pattern regained a pointer to its step-1c rationale. Byte-equivalence
of every rebuilt pattern verified programmatically (alternation-order
neutrality probed: the \b and > anchors make order irrelevant).
strip_think_blocks passed the same response-scrubbing strings through
re's pattern dispatcher on every response. Skills Guard repeated the
same work for 121 patterns against every scanned line.
Compile the existing expressions once and reuse Pattern.sub/search. Keep
each generic tool-call tag in its own paired expression so mismatched
openers retain their payload while existing stray-closer cleanup remains
unchanged.
Part of #33208
Salvaged from #32713 by @ErnestHysa.
Co-authored-by: ErnestHysa <takis312@hotmail.com>
Follow-ups from review of #76113:
- Extract cache_ttl_means_disabled() as the single disable-synonym
predicate; agent_init and prompt_caching_disabled_from_config both use
it so the two detection sites can no longer drift (drift would recreate
the #76085 bug class).
- Mirror _run_reference's not-None injection guard in
aggregate_moa_context (stamping None was a harmless no-op copy).
- Replace a vacuous trailing test assertion with the intended
input-non-mutation check; drop a stray blank line.
- Add a predicate-parity regression test (unknown TTL values keep
caching enabled, matching historical agent_init semantics).
Absorb the useful deltas from the parallel #76121 approach: a single
blank_cache_policy_stub factory so _cache_disabled cannot be left off
hand-rolled SimpleNamespaces, and pin the live agent disable onto MoA
advisor fan-out and one-shot aggregate_moa_context decoration so those
paths track conversation state rather than a fresh config re-read.
Keeps the earlier tri-state prepared-aggregator no-agent fix. Adds
factory and synthesis/advisor regressions.
Coordinates with #76121 / #76085.
Co-authored-by: JoaoMarcos44 <87440198+JoaoMarcos44@users.noreply.github.com>
Blank SimpleNamespace stubs used by MoA decoration and
plan_cache_sections_for_destination never set _cache_disabled, so
anthropic_prompt_cache_policy re-injected cache_control markers after
operators turned caching off. Stamp the disable onto those stubs from
an explicit flag or the live config, and pass the agent flag from the
MoA aggregator path.
Fixes#76085
Three copies of the same logic landed with #76032:
- MoA's _call_prepared_aggregator and auxiliary_client's
_replan_synchronous_cache_sections both implemented stub → policy →
strip → plan for a resolved destination. Extract
plan_cache_sections_for_destination() into agent_runtime_helpers (which
already owns the policy functions) and route both through it. Also
removes a redundant full-transcript deepcopy+strip per request (the
caller pre-stripped what build_prompt_cache_plan strips again).
- The fallback_chain[N] label regex + chain-entry lookup lived in
_fallback_entry_timeout AND _fallback_destination. Extract
_fallback_chain_entry() and reuse.
MoA's cache-plan failure log is promoted debug → warning: the call-block
site skips MoA, so this block is the aggregator's only decoration path —
a silent failure ships an undecorated request (the 0%-cache MoA bug class).
Behavior-preserving; 195 targeted tests green.
Follow-up to #76032 (#20880).
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.
The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused #56105 to be reverted in
#56126.
Salvage of #33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.
Co-authored-by: BB-light <BB-light@users.noreply.github.com>
Copilot degrades in two related ways that both abort a turn as non-retryable
and only clear on a gateway restart (a cold process re-runs the token exchange):
1. HTTP 400 model_not_available_for_integrator / model_not_supported — a
raw/degraded token routes to the restricted copilot-language-server
integrator whose allowlist omits enterprise-only models (e.g.
claude-opus-4.8). Because it is a 400 (not 401), the existing 401 refresh
path never fired. Prevented (retry-with-backoff exchange + on-disk JWT
persistence + header guard at the client chokepoint) and self-healed at
runtime (single-shot forced re-exchange + client rebuild + retry before
fallback).
2. HTTP 401 'IDE token expired: unauthorized: token expired' — the short-TTL
*exchanged* IDE token expires mid-turn. The clean-401 path DID fire and call
_try_refresh_copilot_client_credentials(), but that method only re-resolved
the stable raw ghu_ token and rebuilt the client — it never evicted the
cached exchanged JWT or forced a fresh exchange, so the retry put the SAME
expired token back on the wire, 401'd again, and the single-shot guard
aborted the turn. Fix: force a fresh IDE-token exchange (evict cached JWT via
evict_cached_exchanged_token + re-mint via get_copilot_api_token) before the
client rebuild, mirroring the merged auxiliary-path recovery (#59837) and the
400 recovery in this same PR. Graceful fallback to the resolved token if the
exchange endpoint is unreachable; picks up the enterprise base_url on
re-exchange.
Brings main-loop clean-401 recovery to parity with the merged auxiliary path
(#59837), using the newer on-disk-aware evict helper. Companion context: #58743
(this PR, expanded), #51313, #63204 (which assumed the 401 path already
recovered — it reached the method but the method was too weak).
Tests: exchange retry/persist round-trip, restart-blip disk reuse, stale-cred
400 classifier, 400 recovery, and 3 new 401 cases (fresh exchanged token on the
wire; network-blip fallback to resolved token). 58 copilot tests green on
current main.
Three provably-safe optimizations for O(n)-per-iteration history walks:
1. sanitize_tool_call_arguments: optional identity-keyed cursor (strong
refs to the exact validated message objects) skips re-json.loads-ing
already-validated history each loop iteration. Any list rewrite
(compression, repair, undo, steer) breaks the identity prefix match
and forces re-scan from the divergence point. Wired via a per-agent
cursor dict in conversation_loop.
2. estimate_messages_tokens_rough: per-message memo keyed on a deep
identity fingerprint (strings pinned by strong reference so id()
aliasing is impossible; scalars by value; dicts/lists structurally
with key order). Equal fingerprints imply identical str(shadow)
bytes, hence identical estimates. Unfingerprintable shapes fall
through to direct compute. Bounded FIFO cache (4096 entries).
3. _flush_messages_to_session_db_unlocked: bounded scan that skips the
identity-matched prefix of the previous successful flush's snapshot.
Snapshot only taken on full success; cleared on exception. Compression
rewrites use fresh copies, breaking identity and forcing full re-scan.
Parity proven in tests/agent/test_cursor_optimizations_parity.py:
500-message synthetic histories with tool calls, malformed args, unicode,
element-wise old==new across 3 iterations incl. simulated compression.
Measured (median of 5): sanitize 0.097ms->0.011ms, tokens 1.145ms->0.853ms,
persist-scan 179.5us->10.0us at 500 messages.
Salvage of PR #52188. The original PR raised RuntimeError when LM Studio
load was rejected or unverifiable, which would abort agent startup on
transient network failures. Replace with logger.warning + fallback to
configured context length, preserving the old graceful-degradation behavior.
With HTTP(S)_PROXY (and similar mounted transports), live connections sit
on client._mounts rather than the default _transport. force_close_tcp_sockets
only walked the default pool, so stranger-thread interrupt abort logged
tcp_force_closed=0 and left the request alive for minutes (#72975). Also
walk nested proxy _connection wrappers and WARNING when an abort finds no
sockets.
The concept 'never send a turn that strict wire validation rejects as
empty' was forked across four sites, each with its own predicate and its
own blind spots:
1. build_assistant_message write-time ' ' pad — broke codex commentary
turns (content:'' is a designed state), and a DB-side pad can't
survive _rows_to_conversation's whitespace strip anyway. REMOVED.
2. conversation_loop send-time ' ' pad — main-loop only (summary path
uncovered), ordering-fragile (had to run after whitespace
normalization), assistant-only. REMOVED.
3. stream-stub '[response interrupted]' substitution — defeated the
loop's empty-stub guard (the stub no longer looked empty, entered
history, and the placeholder leaked into the stitched final
response via truncated_response_parts). REMOVED.
4. repair_empty_non_final_messages in sanitize_api_messages — the
unconditional pre-send chokepoint shared by the main loop AND the
summary path, covers user and assistant turns, non-final only,
copy-on-write. This is now the SINGLE OWNER.
The owner's payload predicate (_msg_has_payload) is extended to treat
codex_message_items / codex_reasoning_items as payload, so
designed-empty codex commentary turns are never rewritten on any
api_mode — the failure shape that broke site 1 in CI is encoded in the
owner, not special-cased at a call site.
Tests updated to pin the new contracts: builder stores textless turns
as-is; the empty stream stub stays recognizably empty for the loop
guard; poisoned resumed histories are repaired to the placeholder at
the send boundary; codex item carriers are never rewritten.
Sabotage-verified: unwiring the owner fails 3 regression tests.
Third layer of the empty-stub fix: full self-recovery. A poisoned transcript
(empty assistant stub or empty user turn already persisted before the write-
time guard, or fed in from a host history) previously 400'd every subsequent
request until it scrolled out — needing a manual DB edit + gateway restart.
sanitize_api_messages() (the unconditional pre-send chokepoint) now repairs
empty non-final messages on the per-call copy by substituting a minimal
'[response interrupted]' placeholder, so the session recovers itself IN MEMORY
on the very next send. The final message is left untouched (empty final
assistant is legal); stored history is never mutated; reasoning-only and
tool_call turns are preserved (negative controls).
Tests: production-shape repro (tool -> empty assistant -> user), empty-user
case, non-destructive guarantee, and negative controls. RED verified by
disabling the wire-in.
Follow-ups to the salvaged #23768 commit, which targeted a pre-79559214
codebase:
- agent/tool_executor.py + agent/agent_runtime_helpers.py: pass
multi_select at both current clarify dispatch points (the PR's
run_agent.py edits landed on dead code paths).
- tools/clarify_tool.py: replace the broad TypeError-retry in
_invoke_callback with inspect.signature detection, so a compatible
callback that raises TypeError internally is not invoked twice
(addresses hermes-sweeper review feedback on #23768).
- tests: cover single-invocation on internal TypeError, legacy 2-arg
callbacks, **kwargs callbacks, and registry handler multi_select
pass-through (schema arg → handler → callback).
Widen the #70773 fix beyond the three in-request cleanup sites removed in
the cherry-picked commit: every remaining path that swaps out the shared
OpenAI client could still hard-close its pool from a thread that doesn't
own the in-flight sockets (credential rotation/refresh on the turn thread,
dead-connection cleanup, gateway cache eviction, transport recovery) —
the same FD-recycle corruption vector, just rarer.
Add AIAgent._retire_shared_openai_client(): shutdown(SHUT_RDWR) all pooled
sockets (FD-safe from any thread, unblocks in-flight readers) but never
call client.close() — FD release is deferred to GC, which cannot run until
every borrowing thread has unwound its SSL BIO. Refcounting is the
ownership handshake; with no borrowers the FDs are released immediately.
Wired into:
- _replace_primary_openai_client (rotation/refresh/dead-conn cleanup)
- try_recover_primary_transport (primary_recovery)
- release_clients (gateway cache_evict)
agent.close() keeps the hard close: full teardown is a real session
boundary where no request may be in flight.
Tests: new tests/run_agent/test_70773_shared_client_fd_corruption.py
covers the three watchdog/retry sites plus retire semantics; existing
close-assertions updated to pin retire-not-close.
Replace 3 duplicated entry_id resolution blocks (try/except +
entry_id_for_api_key + fallback to None) in agent_init.py,
chat_completion_helpers.py, and switch_model with a single
sync_credential_pool_entry_id(agent) function in agent_runtime_helpers.
Follow-up to #70323.
Track the selected credential by stable pool entry ID so token refreshes and shared cursor movement cannot detach failures from the entry that issued them. Stop unmatched single-entry pools from reporting a no-op rotation as successful recovery.
Co-authored-by: Maxim Esipov <maksesipov@gmail.com>
Follow-up to the salvaged core of #53802: a naive MoAClient(preset) rebuild
restores a working facade but silently drops the reference_callback relay
wired in agent_init, so moa.reference / moa.aggregating display events stop
reaching every frontend for the rest of the session.
Introduce agent.moa_loop.build_moa_facade(agent, preset) as the single
construction point for the MoA facade and use it at:
- initial client construction (agent_init.py)
- turn-start fallback restore (restore_primary_runtime)
- transient transport recovery (try_recover_primary_transport — previously
fell through to _create_openai_client with MoA's empty client_kwargs and
died with 'api_key client option must be set')
- mid-session model switches (switch_model)
The relay reads agent.tool_progress_callback at emit time, so callbacks
attached after construction are picked up automatically.
Adds test_moa_restored_facade_still_emits_reference_events covering event
delivery through a restored facade.
Review follow-up: the auth path called pool.try_refresh_current() before
the hinted rotation, so a stale current() pointer could force-refresh a
different, healthy entry — consuming its single-use refresh token, or
(for non-OAuth entries, where a forced refresh marks the entry exhausted
outright) killing it entirely before api_key_hint was ever consulted.
Use try_refresh_matching(api_key_hint=...) to resolve and refresh the
entry that supplied the failing key under the pool lock, falling back to
the previous behavior when no key is known.
Adds a regression test with current() deliberately pointed at the
healthy entry: on the old code the healthy entry is exhausted by the
forced refresh and the pool ends up fully offline; with the fix the
failing entry is exhausted and recovery rotates to the healthy one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
recover_with_credential_pool identified "which credential failed" via
pool.current(), a shared mutable pointer that is advanced by every
select() (round-robin rotation, concurrent turns, and other processes
reloading the pool reset it to None). By the time recovery ran, it
routinely pointed at a different, healthy entry — mark_exhausted_and_rotate
then stamped the failing request's error message and reset time onto that
innocent entry. With round_robin and one hard-capped key this
deterministically exhausted the healthy key too and took the entire pool
offline ("no available entries") from a single rate-limited credential.
mark_exhausted_and_rotate already supports api_key_hint for exactly this
(the auxiliary-client path passes it); the main conversation-loop path
never did. Pass agent.api_key — kept in sync with the entry in use by
_swap_credential — as the hint on all four rotation call sites, and make
the "already exhausted → rotate immediately" pre-check look up the failing
entry by key with the same fallback to current().
Adds regression tests that fail on the old attribution logic: a fresh
pool (current() is None) failing on key B must mark entry B, never
entry A.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
recover_with_credential_pool() called mark_exhausted_and_rotate() without
api_key_hint, causing it to fall back to current() or _select_unlocked().
When a prior rotation left current() as None, _select_unlocked() returned
the NEXT (healthy) entry instead of the one that actually failed — marking
the wrong credential as exhausted (#43747).
Extract the current API key from agent.api_key (or pool.current().runtime_api_key
as fallback) and pass it as api_key_hint to all 4 call sites.
* fix(desktop): preserve interim assistant text wiped at message.complete
When the agent emits interim text (commentary alongside tool calls, or the
attempted final answer before a verify-on-stop nudge), all UI surfaces
streamed it live but then wiped it at message.complete — keeping only the
final response. The user saw text appear during inference, then disappear.
This is the complete fix across all three layers: agent core, gateway
transport, and all UI surfaces (desktop + Ink TUI).
The verify-on-stop and pre_verify paths flagged the assistant's attempted
final answer as _verification_stop_synthetic, suppressing it from both
state.db and the UI. The user only saw the terse post-verification reply.
Now the assistant response is real content: it's persisted to state.db and
emitted as an interim message via _emit_interim_assistant_message(force_display=True)
before the verification loop runs. Only the synthetic nudge messages keep
the synthetic flags. The turn finalizer drops nudges from live history and
compares content (not just role) to avoid duplicating a published candidate.
Message sequence repair collapses verification candidates in the
consecutive-assistant merge.
Wire agent.interim_assistant_callback both at construction (_agent_cbs())
and per-turn (defense-in-depth), emitting a new message.interim event with
{text, already_streamed}. Gated on display.interim_assistant_messages
(default true). Cleared in the finally block so a stale closure can't
fire on a later turn.
Add message.interim to the GatewayEventName union (apps/shared) and a
typed payload to the TUI's GatewayEvent discriminated union.
The TUI already had the segment-anchoring machinery (flushStreamingSegment +
finalTail) but had no handler for message.interim. Added recordInterimMessage
+ interimBoundaryIndex to seal segments mid-turn, and updated
recordMessageComplete to only dedupe segments after the interim boundary.
Replaced the fragile sealed-set approach with a proper interimBoundaryPending
state flag on ClientSessionState. finalizeInterimAssistantMessage finalizes
the streaming bubble in place (or creates a standalone one), rotates the
stream ID so next deltas create a new bubble, and sets the flag. When the
final text equals an already-sealed interim, they stay as distinct messages.
Extracted mergeFinalAssistantText() as a pure function in chat-messages.ts,
used by both completeAssistantMessage and finalizeInterimAssistantMessage.
Split the bidirectional dedup predicate: reasoning is a restatement only when
the final FULLY covers it. A short final ("Done.") no longer swallows a
longer reasoning block that merely starts with it.
Honor display.interim_assistant_messages (default true) across all layers:
the tui_gateway gates the callback, the desktop wires it to a nanostores
atom via use-hermes-config. Updated hermes_cli/config.py and
cli-config.yaml.example comments to document the Desktop behavior.
_split_segment_tokens now accepts posix=False and _find_ad_hoc_match tries
both posix modes so ad-hoc verification scripts with Windows backslash
paths are matched correctly. (response_previewed forwarding from #53553
is not included — our emit-interim + persist approach makes it unnecessary
since the attempted answer is now surfaced before the verification loop.)
- tsc: clean (desktop + TUI + shared)
- vitest desktop: 73/73 pass (7 interim-sealing + 5 mergeFinalAssistantText + 4 config atom)
- vitest TUI: 83/83 pass (4 new message.interim tests)
- python: 390 tests pass (340 tui_gateway + 33 verification/finalizer + 6 config gating + 3 evidence + 8 continuation budget)
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
* fix: prefix-match interim streamed content to avoid benign duplicate bubbles
_interim_content_was_streamed used exact equality (streamed == visible_content),
so a final response that was the streamed text plus a trailing delta — or a
partial stream before the verify nudge fired — failed the match and left
_response_was_previewed false. The turn then showed two bubbles (interim +
identical final) instead of settling the interim in place.
Relax to a prefix check (visible_content.startswith(streamed)) in both the
core match and the desktop's settle-in-place gate. The TUI already used
prefix matching via finalTail. The reverse direction (streamed longer than
final) is intentionally not matched — that could suppress a needed resend
in the gateway path where already_streamed=True calls on_segment_break().
* test(desktop): add partial-stream-then-nudge dedup edge case
Third edge case for the interim-sealing dedup: model streams part of its
answer via message.delta, verify nudge fires, interim seals the streamed
prefix, then the final response is the same text plus a trailing delta.
Asserts one bubble (not two) containing the full final text.
Acceptance protocol #2 — covers all three dedup edges:
1. interim == final (existing)
2. interim = strict prefix of final (existing)
3. partial-stream-then-nudge (this commit)
---------
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
Background-review forks share the live parent's session_id for prompt-cache
warmth but are _persist_disabled — they can never write to the transcript.
Without this, every review fork on an active session would (a) trip a false
cross-agent overlap warning against the parent's real turn, sending the
#64934 route investigation the wrong way, and (b) pop the parent's in-flight
slot at its own persist, making a real overlap right after a review go
unreported. Both legs now skip persist-disabled agents symmetrically.
+2 tests.
note_turn_start kept its in-flight marker on the agent object, but the
gateway caches agents per routing key (_agent_cache) while transcripts
are owned by session_id — and switch_session (/resume from a second
surface, CLI-continuity rebinding, async-delegation pinning,
topic-binding tip-walks) maps multiple routing keys onto one session_id
without any cross-key check. Two keys mapped to one session run
concurrent turns on two different agent objects, so the per-agent
tripwire could never fire for exactly the dispatch route #64934 is
waiting to identify.
Add a module-level session_id-keyed in-flight registry alongside the
per-agent marker. Same philosophy as the original tripwire: log-only,
takes ownership on overlap, under-reports rather than double-reports
(a same-agent overlap warns once, not twice). The persist-time clear
pops the session id stamped at turn start, so a mid-turn compression
rotation of agent.session_id cannot strand the slot.
Deduplicates the sidecar handling across 9 sites:
- substitute_api_content(): 2 API-bound pop+substitute sites
(chat_completion_helpers, transport — conversation_loop keeps its
inline pop because the current-turn compose fallback needs the value)
- drop_stale_api_content(): 4 rewrite-drop sites
(context_compressor x2, replay_cleanup, agent_runtime_helpers)
- extract_api_content_sidecar(): 3 gateway forwarding sites
(gateway/session, gateway/slash_commands, cli_commands_mixin)
Also: restore the eager _pending_cli_user_message clear on the early
row-creation try (was lost when the crash-persist moved after prefetch —
a crash in compression between the two tries would leave a stale staged
input), and fix a comment indentation nit in run_agent.py.
Co-authored-by: Soju06 <qlskssk@gmail.com>
The first LLM call of every gateway turn gets ~0% provider prompt-cache
hit rate (in-turn calls: 97-99%) because the bytes sent for a turn's
user message are not the bytes replayed next turn: memory-prefetch and
pre_llm_call context are injected into the API copy only, the #48677
persist override writes cleaned content to the DB row, and
get_messages_as_conversation sanitize/strips user and assistant content
on load. Any of these diverges the request prefix at that message and
re-prefills everything after it — measured 27.9s for the first call vs
2.4-5.8s cached at a median ~156k-token context.
Persist what you send: a nullable messages.api_content column stores the
exact content string sent to the API when it differs from the clean
stored content, and replay substitutes it verbatim (no sanitize, no
strip). The injection composition lives in one helper
(turn_context.compose_user_api_content); the turn prologue stamps its
output onto the live user message, the api_messages build sends the
stamped bytes, and every outgoing copy pops the field so it never
reaches a provider. The crash-resilience user-turn persist moves after
prefetch/pre_llm_call so the user row is written once with its final
sidecar; _ensure_db_session stays before preflight compression (session
rotation needs the parent row under PRAGMA foreign_keys=ON). The
current-turn index trackers are re-anchored after compaction rebuilds
the message list, in-place preflight compaction backfills the stamp onto
the already-inserted row, and gateway replay forwards the sidecar only
when the replay pipeline did not rewrite the content. Rewrite paths that
would leave stale bytes (historical image strip, merge-summary-into-
tail, consecutive-user repair merge, stale-confirmation redaction) drop
the sidecar; the chat-completions transport and the max-iterations
summary path strip it defensively. codex_app_server and MoA turns are
excluded from stamping because their wire bytes differ from the
composition. A missing or dropped sidecar degrades to today's behavior
(one cache-boundary miss), never to wrong content.
1. Remove PR's redundant strip_think_blocks coercion (Teknium's fix
296494db0 already handles this at the same chokepoint with superior
logic that drops thinking/reasoning blocks).
2. Restore 'if not content: return ' guard at top of strip_think_blocks
that was lost during cherry-pick auto-merge. Without it, None content
hits str(None) → 'None' string instead of returning empty.
3. Fix error classifier design flaw: remove 'conversation_loop' and
'run_agent' from _local_processing_modules — these are the container
modules for the try/except, so every exception passes through them,
making _hit_local always True and misclassifying transient API/network
errors as non-retryable local bugs.
4. Move module sets to module-level frozenset constants (_LOCAL_PROCESSING_MODULES,
_API_CALL_MODULES) instead of rebuilding on every exception.
5. Replace traceback.extract_tb() with raw tb walk — avoids disk I/O for
source lines that are never used.
6. Remove unused 'import traceback'.
7. Fix docstring corruption: 3 lines where think tags were replaced
with Chinese characters during the PR's editing.
Brooklyn Nicholson
cicav
WojtekMR3
JonthanaHanh
kshitij
Eugeniusz Gilewski
AlexFucuson9
686f6c61
Rod Boev
BB-light
wz-heng
dstkwll
teknium1
LunarNexus
Alex Fournier
HexLab98
Aaron Weiker
Jeffrey Quesnelle
rob-maron
Jeffrey Quesnelle
kshitij
Gille
kosta
schattenan
liuhao1024
ethernet
hotragn
Soju06
nanami7777777