Commit Graph

1684 Commits

Author SHA1 Message Date
686f6c61 bf7c716648 fix(agent): rebind pool entry id after env credential refresh
Per-turn .env adoption could rewrite agent.api_key while leaving
_credential_pool_entry_id on a previously rotated fallback. The next 429
then marked the healthy fallback exhausted via credential_id precedence
(#79156).

- Sync pool entry id after a successful env credential refresh
- First look does not stomp a pool-rotated key with the env primary
- mark_exhausted_and_rotate prefers api_key_hint when it disagrees with
  credential_id

Fixes #79156
2026-08-08 19:17:02 -07:00
Teknium 212e84176d fix(compression): charge stale thinking to the tail budget only on the newest assistant turn (#73624)
Generic thinking fields (reasoning / reasoning_content + the
reasoning_details text charge) are replayed for at most the NEWEST
assistant turn on every transport: Anthropic strips all-but-newest at
convert time, Bedrock Converse never replays thinking, and strict
chat-completions providers reject or one-space-pad the field. The tail
budget walks charged them on every message anyway, spending 19-24% of
the budget (per the issue's 1,025-message measurement) on bytes that
provably never reach the wire — so the tail cut landed early and each
compaction discarded more real transcript than configured.

_estimate_msg_budget_tokens now partitions the replay keys:

* _ALWAYS_REPLAYED_BUDGET_KEYS (codex_reasoning_items,
  codex_message_items) — charged unconditionally. These ride the wire
  on every retained turn (#55572), and codex_reasoning_items now also
  carries native server-side compaction checkpoints (#81747).
* _NEWEST_TURN_ONLY_BUDGET_KEYS (reasoning, reasoning_content) + the
  reasoning_details text charge — charged only for the newest assistant
  turn via charge_stale_thinking, resolved by the three budget walks
  (tail cut, raw-budget re-walk, proactive-prune boundary).

Default stays the conservative full charge for callers without
turn-position context. A partition invariant test pins that any future
_REPLAY_BUDGET_KEYS entry must be classified into exactly one class.

Direction credit: #73669 (@x7peeps) and #73730 (@webtecnica) both
attacked this; the keep_open reviews asked for provider/API-mode-aware
accounting that keeps Codex carriers charged — this implements that
shape.
2026-08-08 17:42:34 -07:00
Brooklyn Nicholson f726090d48 feat(sessions): name a session the moment it starts
Titling fired on the first response, so a session sat unnamed for the whole
opening turn - p50 151s, p90 1212s across real sessions, because a turn is
tool calls, not one round-trip. A turn that failed or was interrupted never
got a title at all. Four surfaces each carried their own copy of the call.

Move it into the shared turn prologue and split it in two: a deterministic
title derived from the user's opening message, written inline before the
model runs, then one small-model call that upgrades it. The response is
constrained to a JSON object so there is no preamble to strip, and control
wrappers are stripped rather than refused, so a slash command titles as
what the user asked for instead of the command itself.
2026-08-08 17:07:21 -05:00
Brooklyn Nicholson e358eaf44a perf(sessions): resolve the titling model from the provider's live catalog
Titling ran on the user's main chat model, so a five-word title was billed
to a frontier reasoning model and inherited its latency. Pinning a cheap
model id instead just moves the problem: the hardcoded default was already
dead upstream and every call paid a 404 before the retry net caught it.

Match model FAMILIES against the provider's live /v1/models catalog,
preferring rolling '-latest' aliases where a provider publishes them, and
order the families by measured latency. Nothing to bump when a provider
ships a new mini/flash/haiku. Opt-in per task, so compression, vision, and
search keep 'auto means my chat model'.
2026-08-08 17:07:21 -05:00
Teknium e00965a7e8 fix(compression): correct prune boundary + exempt native compaction checkpoints
Two corrections on top of the #71077 base (the whole bug class):

1. Turn boundary = last USER message, not last assistant message. A Codex
   turn spans several assistant messages (assistant+tool_calls -> tool ->
   ... -> final assistant) whose reasoning items must replay together; the
   last-assistant boundary would strip reasoning mid-chain from the active
   turn (the gap flagged in PR #71077 review).

2. type="compaction" checkpoints (native server-side compaction, PR #81747)
   are exempt: they carry already-pruned history, not per-turn reasoning.
   Pruning filters items instead of popping the sidecar key.

Sibling site fixed in the same class: the Codex incomplete-continuation
dedup path blind-overwrote codex_reasoning_items on visually-duplicate
interim messages, which would drop the only copy of a checkpoint captured
on the earlier response. Extracted merge_interim_reasoning_items() into
agent/native_compaction.py; newer reasoning wins, prior checkpoints are
preserved unless the newer payload carries its own.
2026-08-08 14:09:41 -07:00
Teknium 566b5b16a9 fix(agent,gateway): class-level lone-surrogate chokepoints (#80366 #55143 #55309 #50959 #19819)
Own the surrogate-crash class at three chokepoints instead of leaf sites:

- finalize_turn scrubs final_response once where model text leaves the
  conversation loop — covers oneshot stdout (#80366), NIM/any-provider
  responses (#19819), and every delivery consumer of the turn result.
- _sanitize_gateway_final_response scrubs at the gateway chat-surface
  boundary — Telegram utf16_len (#55309) and Signal formatting (#55143)
  can no longer see a lone surrogate; raw-text surfaces keep passthrough.
- run_conversation walks the fully-built api_kwargs with
  _sanitize_structure_surrogates so tool descriptions (session_search,
  #50959) and every other request-body leaf are JSON-encodable before
  any provider sees them.

Regression tests pin all three chokepoints plus helper semantics.
Cherry-picked alongside #79240 (TheophilusChinomona) and #80374
(rainbowgore) whose commits precede this one with authorship preserved.
2026-08-08 12:31:19 -07:00
Teknium 5e1b50115f feat(compression): native OpenAI Responses server-side compaction for gpt-5.6
Opt-in via compression.codex_responses_native (default: false). When enabled,
gpt-5.6-family models on the direct OpenAI API (api.openai.com) or a ChatGPT
Codex subscription send context_management=[{type: compaction,
compact_threshold: N}] on Responses requests. OpenAI compacts server-side and
returns an encrypted compaction output item; Hermes captures it into the
existing codex_reasoning_items sidecar and replays it on later turns in place
of the pruned history — inheriting persistence, session replay, the
cross-issuer guard, and the encrypted-replay kill switch with zero new state.

Scope is deliberately hard-gated (agent/native_compaction.py, re-checked per
request): gpt-5.6 family only — gpt-5.1/5.2 fail server-side on the field
(HTTP 500 / stream stall, no structured rejection; live-verified) — and
direct OpenAI/Codex routes only; xAI, GitHub/Copilot, OpenRouter, relays,
and local servers never see the field.

Hermes' local compression stays armed as the fallback owner: the native
threshold is clamped ~8K tokens below the local trigger so the server
compacts first, and a structured provider rejection of context_management
disables native compaction for the session and retries without it
(one-shot guard in TurnRetryState).

Live-verified E2E on api.openai.com/gpt-5.6: server compaction fired at a
4K threshold, checkpoints captured and replayed, recall preserved across
3 turns; gpt-5.1 with the flag enabled stays clean (field never sent).

Direction credit: PR #76950 by @laryhorb explored native Responses
compaction; this is a minimal reimplementation on current main.
2026-08-08 11:24:45 -07:00
kshitij 206f74baac Revert "feat: add DCP context engine"
This reverts commit d7072ab914.
2026-08-08 23:09:43 +05:30
kshitij 9841a6c651 fix: rewire DCP context engine to current main architecture
Fixes 13 issues found in PR #20774 review:

1. Wiring: engine selection moved from run_agent.py to agent/agent_init.py
   (where init_agent lives on current main). Transform hook moved from
   run_agent.py to agent/conversation_loop.py (where run_conversation lives).

2. Prompt caching: replace copy.deepcopy with copy-on-write (shallow list
   copy + clone only messages that are mutated). Use last_prompt_tokens
   from update_from_response instead of re-estimating tokens every call.
   System extension injection is idempotent (one-time cache break).

3. Signature mismatch: _message_signature renamed to _content_signature
   and now excludes tool_calls/tool_call_id from the hash. This prevents
   mismatches when _canonicalize_api_tool_calls re-serializes argument
   JSON with sort_keys=True on the API copy.

4. update_model: accepts api_mode parameter (required by agent_init.py).

5. Reconciled with select_context: transform_api_messages is a separate
   hook that runs AFTER select_context and sanitization, before
   prompt-cache marker placement. Both hooks coexist with clear ordering.

6. Dedup/purge: kept as DCP-specific strategies (different semantics from
   ContextCompressor._prune_old_tool_results — DCP deduplicates by
   tool+args signature, not by content hash).

7. Removed copy.deepcopy: replaced with shallow list copy + copy-on-write
   via _clone_if_needed. Only messages that are actually mutated get
   cloned.

8. Removed redundant _ensure_refs call: _match_api_messages_to_refs no
   longer calls _ensure_refs (the caller already called it).

9. _message_key still uses index (needed for positional ref assignment),
   but _content_signature is cached per id(msg) to avoid re-hashing.

10. _inject_nudge: only injects into user messages, never falls back to
    non-user messages (prevents role semantics violations).

11. Memory: _evict_inactive_blocks bounds blocks_by_id to
    _MAX_INACTIVE_BLOCKS (50) deactivated blocks.

12. Merged _range_tool_schema and _message_tool_schema into a single
    _compress_tool_schema. Merged _handle_range_compress and
    _handle_message_compress into _handle_compress.

13. Dropped DCP_CONTEXT_ENGINE_PR_SPEC.md (temporary file, not for tree).

Config defaults kept minimal in hermes_cli/config_defaults.py (only
the keys the engine actually reads, not the full DCP-compatible surface).

Closes #20717
2026-08-08 23:08:49 +05:30
Jack Maloney d7072ab914 feat: add DCP context engine
Cherry-picked from PR #20774 by @jmmaloney4 (jmmaloney4@gmail.com).
Original commits: 4420e0b0, 44247545, bac2955d.

DCP-style model-guided context engine behind context.engine: dcp.
Adds compress tool, outbound API-call transforms, automatic dedup/purge,
and DCP-compatible config surface.

Closes #20717
Co-Authored-By: Jack Maloney <jmmaloney4@gmail.com>
2026-08-08 23:08:49 +05:30
Drexuxux d135f64b51 fix(curator): protect cron skills referenced by absolute path
4c2961c51 added referenced_skill_names() so the curator never archives a
skill a cron job depends on — paused jobs and infrequent schedules would
otherwise age their skills out and the next run fails to load them.

62972060c then taught the scheduler that jobs may store ABSOLUTE skill
paths, normalizing them through normalize_skill_lookup_name before
skill_view. The protection set kept returning the raw string, so it now
holds a full path while the curator matches it against bare skill names.
Those jobs silently lost their protection: the skill is archived, and the
next fire logs a warning and runs the job without its instructions.

Canonicalize each reference the same way the scheduler resolves it, with
a deferred import and a verbatim fallback so a resolver failure can never
drop a name (referenced_skill_names has exactly one caller, the curator's
protection lookup, so nothing else sees the change).
2026-08-08 06:04:14 -07:00
webtecnica 464e7e4e5f fix(docker): read attached binary files in backend (#76577) 2026-08-08 05:44:18 -07:00
fangliquanflq fb4664f79d fix(learn): process large sources incrementally 2026-08-08 05:09:43 -07:00
fangliquanflq 57ca5995c6 fix(learn): extend existing skills during relearning 2026-08-08 05:09:43 -07:00
Teknium 1dee73400e Inspired by Cursor: fail-closed hook semantics + exit-code-2 blocking 2026-08-08 05:02:04 -07:00
Teknium 2e2fcc09ff Port from superagent-ai/grok-cli: directory-chain AGENTS.md loading 2026-08-08 04:31:45 -07:00
WolftacDigital 4af8fb2148 fix(ssl_guard): tolerate truststore SSLContext.get_ca_certs() NotImplementedError on Windows
On Windows, truststore.inject_into_ssl() replaces ssl.SSLContext with an
OS-trust-store-backed context whose get_ca_certs() raises NotImplementedError
(empty message). The ssl_guard's _validate_bundle_path() called get_ca_certs()
unguarded, crashing every fresh agent init with an opaque
'Failed to initialize OpenAI client:' error.

create_default_context(cafile=...) already validates that the bundle is
parseable, so we skip the post-load introspection rather than treat the
NotImplementedError as a failure.

Cherry-picked from PR #49945 with comment trimmed.

Co-authored-by: WolftacDigital <jonathan@wolftacdigital.com>
2026-08-08 15:10:24 +05:30
kshitij d81f2f49ea refactor(compression): fold simplify findings — dedup floor constant, drift-guard test
- Wire the Pass-1 dedup floor (len < 200) to the shared _PRUNE_MIN_CHARS
  constant it was already documented as matching, and use the constant in
  the remaining test literal.
- Restructure the clarify 'resolved' computation (is_answer_shaped +
  sentinel check) instead of compute-then-flip.
- Add a live producer->recognizer drift guard: the REAL oneshot no-user
  callback's output must be recognized as a sentinel, so producer wording
  drift fails a test instead of silently reintroducing false attribution.
- Document the any()-poisoning semantic for multi-select sentinel lists.
2026-08-08 14:27:31 +05:30
kshitij 39056e8de4 fix(compression): filter clarify non-response sentinels; share prune floor constant
Follow-up to the salvaged #81244 commits:

- Timeout/no-user clarify callbacks (CLI timeout, gateway timeout and
  delivery failure, oneshot no-user) embed sentinel prose as
  user_response; quoting those as '[clarify] user responded: ...' would
  be false attribution. Route them to the generic summary path.
- Extract the shared _PRUNE_MIN_CHARS = 200 floor (prune default +
  proactive clamp) and cap the clarify summary at _PRUNE_MIN_CHARS - 1,
  removing the knife-edge equality the summary's survival depended on
  and keeping it out of the >=200-char dedup pass.
- Tests: 4 sentinel shapes + multi-select sentinel; mutation-checked.
2026-08-08 14:27:31 +05:30
Crypto Intern 3090e9e871 fix: reject forged clarify summaries 2026-08-08 14:27:31 +05:30
Crypto Intern cf1863c878 test(compression): cover clarify persistence path 2026-08-08 14:27:31 +05:30
Crypto Intern 6433d5723f fix(compression): make clarify summaries UTF-8 safe 2026-08-08 14:27:31 +05:30
Crypto Intern d6511aecb6 fix(compression): preserve clarify responses 2026-08-08 14:27:31 +05:30
PRATHAMESH75 aed114a69b fix(agent): treat max-iteration nudge as synthetic during compaction
handle_max_iterations() appends its runtime summary request as a plain
role="user" row, which SessionDB persists verbatim. On later compaction the
synthetic-turn filters only recognized compaction summaries, continuation
rows, and todo snapshots, so the nudge could be selected as the latest
actionable user turn — becoming the task snapshot / auto-focus input and
getting summarized as "User asked: ...", demoting the real human task.

Metadata flags do not survive SessionDB projection (the reason the existing
markers are content-based), so recognition must key off stable content.
Extract the nudge into a shared MAX_ITERATIONS_SUMMARY_REQUEST constant and
teach _is_synthetic_compression_user_turn() to recognize it, mirroring the
continuation/todo markers. Every _is_actionable_user_turn call site already
pairs the synthetic guard, so the single recognizer change covers anchor
selection, auto-focus, and real-user-turn detection.

Fixes #78580
2026-08-08 14:24:39 +05:30
bex 95520b812f fix(agent): fail open on malformed telegram extra config
Guard both extra lookups with isinstance(dict) before merging, so a
truthy non-mapping `extra` value (e.g. `extra: "true"`) degrades to the
base Telegram hint instead of raising TypeError and aborting
system-prompt construction. Keep the narrowed except ImportError.

Add an integration test exercising the real config path (HERMES_HOME +
gateway.platforms.telegram.extra.rich_messages) and a regression test
for the malformed-extra fail-open path. The integration test fails on
main and passes with the fix.
2026-08-08 13:57:44 +05:30
bex 9f582aca1d fix(agent): read Telegram rich_messages config from correct path
Commit b45a217e0 gated the TELEGRAM_RICH_MESSAGES_HINT extension behind
a config read at the top-level ``platforms.telegram.extra.rich_messages``
key, but the Telegram adapter reads the same setting from the canonical
``gateway.platforms.telegram.extra.rich_messages`` path.  When users set
the setting in the canonical location (the only one documented), the
lookup returned None and the extension never fired — the model degraded
pipe tables to bullet lists, task lists to plain dashes, and never
produced <details> blocks or block math.

Fix: merge both ``gateway.platforms.telegram.extra`` and the top-level
``platforms.telegram.extra`` with the same precedence the adapter uses
(top-level leaf wins), so config-wizard writes and dashboard-setup keys
are visible alongside the canonical gateway location.  Narrow the
except-guard to ImportError so real config-stack failures surface.
2026-08-08 13:57:44 +05:30
kshitij 7307f88993 fix: follow-up for salvaged PR #18255
- Fix _usage_audit_path() to use _get_hermes_home() instead of hardcoded
  Path.home() / '.hermes' (profile-safe resolution, sweeper finding)
- Rewrite skip_background_review tests to exercise finalize_turn() directly
  instead of duplicating the guard expression (sweeper finding)
- Fix response_silent audit field to use _is_cron_silence_response()
  instead of the buggy SILENT_MARKER substring check it was meant to
  replace (simplify-code review finding)
- Remove dead 'model' in locals() guard — model is always in scope
  before the try block (simplify-code review finding)
- Extract _stub_agent_for_finalize() helper to eliminate ~40 lines of
  copy-pasted agent stubbing in tests (simplify-code review finding)
- Clean up 'Phase 0.5' instrumentation comments
2026-08-08 00:07:14 +05:30
0xarkstar eaeba6474f feat(agent): add skip_background_review flag to AIAgent constructor
Phase 8 of the Hermes Agent token leak mitigation plan
(ralplan-hermes-token-leaks.md §3.9). Adds a boolean kwarg
`skip_background_review` (default False) to AIAgent.__init__ that
suppresses the end-of-turn _spawn_background_review fork.

Each background review fork instantiates a new AIAgent with its own
~15K input tokens + up to 8 LLM iterations, accumulating ~30K tokens
per event in the worst case. On cron sessions there is no
human-in-the-loop benefit from the review (no skill-creation pressure,
nobody curating MEMORY.md), so the cost is pure waste.

The end-of-turn guard now reads:

    if (final_response and not interrupted
            and not getattr(self, "skip_background_review", False)
            and (_should_review_memory or _should_review_skills)):

skip_memory=True already disables the memory-review trigger; this
flag is the explicit single-switch off for both review paths.

Defaults to False, so behavior is unchanged for gateway/CLI callers
that omit the kwarg.

Tests: 5 new unit tests in tests/agent/test_skip_background_review.py
covering the default value, flag persistence, the gate short-circuit,
the gate fall-through, and a source-text assertion that the cron
scheduler sets the flag to True (separate commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-08-08 00:07:14 +05:30
Teknium 94bc3194b3 feat(delegation): validate batch task quality before spawning children
Reject malformed tasks=[...] batches before any child agent is spawned:

- exact-duplicate goals (case/whitespace-normalized), error names both
  task indices
- placeholder goals: bare 'TODO', bare 'task N', unexpanded <...> or
  {...} template markers, or goals shorter than 10 chars after strip
- 1-task batches, with an error pointing the model at the single
  `goal` form instead

All checks are batch-only — the single-goal form is exempt by design
(short goals like goal="test" are valid there). Error strings are
actionable: each tells the model exactly how to fix the call.

Tool schema is unchanged (byte-stable); validation is runtime-only in
the existing batch-validation region.

Existing tests using terse batch goals ("A"/"B"/"C") updated to
realistic distinct goals per the new contract.

Inspired by: MoonshotAI/kimi-code agent-swarm.md validation rules (MIT)
2026-08-07 08:57:57 -07:00
HexLab98 b9636b1047 test(agent): cover reference-only handoff sole-active-turn regression
Pin #80622 invariants: handoff alone must not drive a model call after
stop, pending real users are restored, and synthetic compaction rows are
never treated as user-originated turns. Also give micro-compaction
enough passes to pay back the longer SUMMARY_PREFIX marker overhead.
2026-08-07 19:44:35 +05:30
Alan Hsu 6d89b10653 fix(agent): project real usage in preflight defer instead of fixed growth tolerance
The rough preflight estimate intentionally overestimates, but not by a
fixed margin: CJK text is counted at ~1.7x its o200k cost and
Responses-mode reasoning replay blobs at several times their billed
cost. Heavy sessions show rough estimates 2-3x real usage and compact
at 35-55% of the real window, stalling turns for minutes and discarding
detail (churn), because the defer guard only tolerated 5% rough growth
and sessions that never compressed had no baseline at all.

Pair every request's rough estimate (note_request_rough_estimate,
recorded in the conversation loop right after the pressure estimate)
with the provider's real prompt_tokens in update_from_response(), then
defer preflight while projected real usage — last real + rough growth
since that reading — stays under the threshold. Rough growth is itself
an overestimate of real growth, so the projection is an upper bound and
deferring below the threshold is safe; the provider's context-overflow
handler remains the backstop.

The baseline no longer ratchets on defer: it is refreshed by the
response pairing, and advancing it without a matching real reading
would shrink apparent growth and defer on stale data.
2026-08-07 19:44:29 +05:30
kshitij 9377c5a539 fix(redact): narrow control-split join guard to line-crossing spans
Post-merge review of aecb9ca89 found the join guard over-broad: skipping
the join whenever ANY fragment self-matches _PREFIX_RE reopened a leak
for non-newline splits — sk-<15 chars>ESC<25 chars> masked only the
self-matching head and left the 25-char tail in cleartext (fully masked
before the guard; main never masked this shape at all, so the merged
state was still >= main, but the salvage's own coverage regressed).

Skip the join only when the span crosses a line boundary (\n / \r) —
that is the shape where adjacent legitimate text gets swallowed
(ghp_<token>-then-'button [ref=e3]' annotation bug). ESC/zero-width
controls never legitimately separate a token from prose, so joining
there is safe and restores full-tail masking.

Both legs mutation-checked: reverting to the unconditional skip fails
the new tail-mask test; removing the guard fails the annotation test.
2026-08-07 17:50:03 +05:30
kshitij aecb9ca894 fix(redact): don't join across controls when a fragment already matches
CI slice 1/12 caught a regression in _mask_control_split_tokens: a
COMPLETE prefix token at end-of-line followed by ordinary text (browser
accessibility annotations: 'ghp_<tok>\nbutton [ref=e3]: Copy') was
joined across the newline into one stripped-copy match, and the mask
swallowed the adjacent line ('button' disappeared).

Join only when no fragment inside the span matches _PREFIX_RE on its
own — a self-matching fragment is already handled by the ordinary
prefix pass, so joining can only cause damage. All smuggling shapes
(ESC/ZWSP/newline splits with under-length fragments) still mask;
regression test added and mutation-checked (fails without the guard).
2026-08-07 17:01:23 +05:30
Soheil Fakour e9d1551e65 fix(redact): strip control chars from mask_secret display (#55319, #55321)
A masked secret's visible head/tail could carry control bytes (newline,
NUL, DEL, C1 0x80-0x9F, zero-width) into config/status/dump output.
Strip every control incl. \n/\t (display differs from redact_sensitive_text,
which preserves \n/\t as line structure) before slicing; all-control values
return the configured empty fallback.

Consolidates the previously-closed #58079 approach (strip controls before
masking) - supersedes it.
2026-08-07 17:01:23 +05:30
Soheil Fakour 5444f6853b test(redact): harden new #77484 tests - assert fragments, opaque values (review) 2026-08-07 17:01:23 +05:30
Soheil Fakour 8563fe3435 fix(redact): close emission gaps - env suffix keys, control-char splits, process(list) (#77484) 2026-08-07 17:01:23 +05:30
kshitij 15d7103aa7 fix: harden .env-read detection — review follow-ups for #61352
- Import file_safety._BLOCKED_PROJECT_ENV_BASENAMES instead of copying
  it (comment-enforced parallel lists drift); lookup is now
  case-insensitive to match file_safety's .lower() semantics (cat .ENV
  on macOS/Windows case-insensitive filesystems reads the same secrets).
- Strip shell quotes plain split() leaves attached (cat ".env").
- Drop the dead _ENV_FILE_EXCLUDE_SUFFIXES logic (exact-basename
  membership already excludes templates) and the stray blank-line noise.
- Document the defense-in-depth limits (sudo/full-path/substitution
  readers) mirroring is_env_dump_command's precedent, and correct the
  docstring overclaim about name-independence.
- Annotate command as str | None (tests pass None).
2026-08-07 16:58:25 +05:30
Peter cf755f5c42 fix: redact .env terminal output via detection instead of known-env-var list
Terminal output from file-read commands (cat, head, tail, ...) uses
code_file=True, which skips the generic ENV-assignment redaction pass.
Reading a .env file through the terminal therefore leaked any key whose
value has no recognized vendor prefix (Mistral, Gemini AQ.*, tvly-dev-,
bu_, Spotify client secrets).

Detect file-read commands targeting .env-style basenames (mirroring
agent/file_safety's blocked list) and route them to code_file=False so
the existing ENV pass masks opaque values. Templates (.env.example,
.env.sample, ...) are excluded.

Salvaged from #61352 (145 commits of drift; conflict with the test-prune
wave resolved by NOT resurrecting pruned tests). Authored by @ShaoRou459.

Closes #61352
2026-08-07 16:58:25 +05:30
kshitij c18e19c3c7 fix(agent): make the send-path copy structural — close the write-through class
The api_messages build used a shallow msg.copy(), decoupling only
top-level fields. Every nested container (tool_calls entries and their
function dicts, multimodal content-part lists, reasoning_details) stayed
aliased to the persisted history, so ANY in-place transform on the send
copy silently rewrote the stored transcript.

Probed every send-path transform against that aliasing shape on main:

  content strip loop                       safe (top-level reassign)
  _canonicalize_api_tool_calls (repair)    LEAKED  <- #80616's fix
  _sanitize_messages_surrogates            LEAKED  (multimodal parts,
                                                    tc ids/args, reasoning)
  _sanitize_messages_non_ascii             LEAKED  (multimodal parts)
  _sanitize_api_messages                   safe
  _drop_thinking_only_and_merge_users      safe

The retry loop already believed the copies were independent - it
sanitizes messages AND api_messages separately (~L3555) - so the
aliasing was accidental everywhere.

Fix at the chokepoint: _clone_message_for_send clones every container
(dict/list) recursively while sharing immutable leaves, so every
downstream in-place transform - current and future - is safe by
construction. Cost is container-count, not string-bytes: 100KB argument
strings and base64 payloads are shared (measured ~0.5ms vs ~0.1ms per
1500-message build; noise next to one json round-trip). Same clone
applied to the prefill-message insert (same class, same pipeline).

The class-wide invariant test runs the full send-path transform
pipeline over an adversarial fixture (malformed args, surrogates,
non-ASCII, multimodal parts, reasoning fields) and asserts the history
stays byte-identical; an AST contract pins the build-site wiring so the
shallow copy can't quietly return. Both mutation-verified: reverting
the clone to shallow fails 4 isolation tests, unwiring the build site
fails the AST contract.

0xGr1mm's branch fix (previous commit) remains as defense in depth at
the exact site the #80498 incident hit; his regression tests and the
class-wide invariant give layered coverage.
2026-08-07 16:57:11 +05:30
Gr1mmJ4w e60ca1c6ca fix(agent): stop the send-path repair from rewriting persisted history
`_canonicalize_api_tool_calls` promises copy-on-write in its own docstring
— "the persisted history is untouched" — and the call site repeats it:
"Operates on api_messages (the API copy) so the original conversation
history in `messages` is untouched."

The canonicalize branch keeps that promise (`tc = {**tc, "function": {...}}`).
The repair branch does not:

    except Exception:
        tc["function"]["arguments"] = _repair_tool_call_arguments(...)

`api_messages` is built with `msg.copy()` — a SHALLOW per-message copy — so
every `tool_calls` entry is the same dict object the persisted history
holds. Assigning into `tc["function"]` therefore writes through to the
stored turn. The sibling loop two lines above only touches `am["content"]`,
one level deep, which is why the aliasing never showed up there.

On the unrepairable path `_repair_tool_call_arguments` returns "{}", so
that write replaces the model's real arguments with an empty object in the
transcript. A stream that dies mid `write_file` loses the file content it
had already streamed — the reported symptom in #80498, where a chapter
draft was silently reduced to `{}` and only a WARNING remained:

    Unrepairable tool_call arguments for write_file — replaced with empty
    object (was: {"content": "# 骨架-第25章\n> 承接...)

Mirror the canonicalize branch: build a new tool-call dict instead of
assigning into the shared one. The API copy still carries "{}" — the
repair's whole purpose is to never ship broken JSON — but the history keeps
what the model actually sent, so the transcript, session persistence and
any later retry still have it.

The in-place write was not an oversight in isolation: it predates the memo
refactor, which preserved it deliberately for byte-parity. The existing
`test_history_not_mutated` asserts exactly this invariant but restricts
itself to valid arguments, and its docstring records the gap — "(Malformed
args take the in-place repair path — pre-existing behavior)". That is why
a test file whose header already claims "the persisted history is never
mutated (copy-on-write preserved)" stayed green through the bug.

Four tests close it: history keeps the original bytes, the send copy is
still repaired, a broken call does not disturb its siblings, and repeated
sends stay lossless. On unpatched main three of them fail; the parity and
complexity tests are unaffected because the difference is only observable
when the history list is separate from the send copy — which is the shape
production uses.

Refs #80498

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 16:57:11 +05:30
kshitij 8b6dd27cdb test(auxiliary): update _resolve_auto patch to _resolve_auto_route
The PR renamed _resolve_auto to _resolve_auto_route (3-tuple return).
This test patches the resolver to mock the auto-detection chain, but was
still patching the old _resolve_auto name. The patch never fired, so the
test fell through to the real auto-detection (which finds no providers
in CI's hermetic env). Update patch target to _resolve_auto_route with
the 3-tuple return value.
2026-08-07 16:56:37 +05:30
Gille 293e67328c fix(agent): preserve auto-routed provider identity 2026-08-07 16:56:37 +05:30
kshitij 03beb662e8 fix: cover the partial multi-call batch in the in-flight exemption
Widen #79293's trailing-in-flight guard from 'last message is assistant'
to 'last non-tool message is assistant': a multi-call batch snapshotted
between the executor's per-result appends looks like
[..., assistant(c1,c2,c3), tool(c1)] — c2/c3 are pending, not orphaned,
but the tail-only guard missed that shape and stripped them (same silent
result loss as the original bug, via concurrent /compress or the gateway
hygiene pass).

Preserving is safe on both shapes: the pre-API chokepoint
(sanitize_api_messages step 2) injects stub results for any call that
genuinely never gets an answer, while stripping a live call silently
loses its late result.

test_sanitizer_strips_orphaned_keeps_valid's mixed valid/orphan shape
moves mid-list — at the tail it is byte-identical to a live partial
batch and the sanitizer now correctly presumes in-flight there.

New regression test fails without the walk-back (c2/c3 stripped),
passes with it.
2026-08-07 14:13:32 +05:30
Shotflame 788b8ab497 fix(compress): preserve in-flight tool chain across context compression (#79278)
Tool_executor.py appends role=tool results AFTER running each call. When
context compression fires mid-chain, the trailing assistant(tool_calls)
message is a pending request whose result has not yet been appended.
_sanitize_tool_pairs previously stripped it as an 'orphan', so when the
executor later appended the real result, repair_message_sequence dropped
it as unmatched and the completed side effect (and final synthesis) was
lost. Preserve the trailing in-flight call verbatim; only genuinely
orphaned calls in the discarded region are stripped.

Adds regression tests: three unit tests for _sanitize_tool_pairs plus an
end-to-end test reproducing compression -> side-effect completion ->
result-returned flow. Confirmed failing on pre-fix code, passing with
the fix.
2026-08-07 14:13:32 +05:30
izumi0uu 988f2baaf8 fix(sessions): recover compression parents without continuations 2026-08-07 13:24:56 +05:30
Teknium 32e7fb07a0 feat(/learn): expansive knowledge-base skills for books and large corpora
Inspired by virgiliojr94/book-to-skill (MIT): /learn now picks the skill
shape by the source. Workflows and small sources still get one tight
SKILL.md; books, paper stacks, specs, and large doc corpora get a
knowledge-base layout — a lean always-loaded SKILL.md index plus one
distilled file per chapter/topic under references/, loaded on demand via
skill_view so query cost stays proportional to the answer.

- agent/learn_prompt.py: new _KNOWLEDGE_SKILL_STANDARDS block (index +
  per-chapter references/, structure-not-summary distillation, never
  reproduce source passages, fold-in instead of duplicating) and a
  _SOURCE_HYGIENE block pinning extracted source text as data and
  dropping invisible/bidi Unicode (Trojan Source class). Clarified that
  the ~200-line cap and hub-skill ban apply to SKILL.md itself, not a
  knowledge skill's own references/ files.
- tests: contracts for the knowledge-base layout, the three embedded
  standards blocks, and the source-hygiene coverage.
- docs: skills.md documents the knowledge-base shape.
2026-08-06 22:14:52 -07:00
brooklyn! 55505be152
Merge pull request #80770 from NousResearch/bb/desktop-session-integrity
fix: preserve session history when a turn crashes
2026-08-06 22:12:22 -06:00
Brooklyn Nicholson fc05247be8 fix: preserve session history when a turn crashes 2026-08-06 23:08:23 -05:00
Brooklyn Nicholson 0f83661808 fix(reasoning): keep gpt-5.x summary parts as separate blocks on the chat wire
Reasoning-summary models emit one reasoning_content delta per completed
summary part, each a self-contained bold heading. The Responses API delimits
those parts with summary_index; the OpenAI chat wire carries no such field —
verified live against Nous Portal, whose reasoning chunks contain nothing but
delta.reasoning_content — so concatenating them glued every part into one
unspaced, half-bold paragraph.

Re-derive the boundary from the signal the wire does carry: a delta opening a
closed bold heading against a mid-line tail. This matches Hermes own Responses
adapter, which already joins its summary parts with a blank line.
2026-08-06 22:02:37 -05:00
Teknium 8f2712725a feat: /refine — run the memory/skill self-improvement review on demand
/refine [focus] fires the existing background review fork
(AIAgent._spawn_background_review) immediately instead of waiting for
the automatic 10-turn memory / 10-iteration skill nudge counters.
Optional focus instructions are appended to the review prompt so the
fork prioritizes what the user asked for (e.g. '/refine save the
deploy workflow as a skill').

- New optional focus parameter threaded through
  _spawn_background_review -> spawn_background_review_thread.
  Automatic post-turn reviews pass None and their prompts are
  byte-identical to before.
- CLI handler snapshots conversation_history; gateway handler pulls
  the idle session's cached AIAgent from _agent_cache (rejected while
  the agent is running).
- Review runs in a daemon thread against the snapshot — live
  conversation, message alternation, and prompt cache untouched.
- Slack stays under the 50-slash cap via /hermes refine.

Adapted from the /refine concept in Prime Intellect's Prime-Agent
(Continual Harness); Hermes' equivalent durable state is the
memory + skill stores, so the review fork is the natural target.
2026-08-05 22:40:51 -07:00