Commit Graph

10895 Commits

Author SHA1 Message Date
GodsBoy e288d93fc1 fix(review): harden portable plugin boundaries 2026-08-07 09:44:21 -07:00
GodsBoy ca78c6d7a6 feat(plugins): load portable agent components 2026-08-07 09:44:21 -07:00
GodsBoy c5117655b6 feat(plugins): validate portable agent packages 2026-08-07 09:44:21 -07:00
PRATHAMESH75 6e87d43a57 fix(tools): lazily bring up sandbox for vision_analyze reads
vision_analyze reads container-only images by exec-reading them inside the
sandbox, but unlike terminal_tool it never triggered environment creation. Under
a non-local backend (ssh, docker, ...), a session whose first action was
vision_analyze on a remote path failed with 'no active sandbox session' until an
unrelated terminal command happened to establish the connection.

Add terminal_tool.ensure_task_env(task_id), a public lazy get-or-create that
reuses the terminal tool's own creation machinery, and call it from
image_source._resolve_container_fallback before the in-sandbox read. Extract the
ssh/container config-dict builders so both paths derive settings identically
(no duplication). Best-effort and fail-closed: a failed bring-up leaves the
existing 'no active sandbox' error intact, never a host read.

Fixes #62825
2026-08-07 09:11:48 -07:00
Teknium bc80a0be5c test: stub EnvironmentConnectionError in environments.base module stub
The modal/browserbase test file replaces tools.environments.base with a
SimpleNamespace stub; terminal_tool now imports EnvironmentConnectionError
from that module, so the stub must provide it too.
2026-08-07 09:07:55 -07:00
Teknium 5c29566e8d feat(terminal): graceful degradation for remote backend connection failures
Connection-class infrastructure failures on remote terminal backends (SSH
host unreachable/timed out, Docker daemon down or missing, remote file
sync failing on a dead link) previously surfaced to the model as raised
RuntimeError tracebacks. The model got a stack blob with no guidance and
the failure was indistinguishable from a tool bug.

Now:

- New EnvironmentConnectionError(RuntimeError) in tools/environments/base.py
  carrying a reason + retry_hint. Subclassing RuntimeError keeps every
  existing catcher working.
- ssh.py classifies connect-refused, connect-timeout, scp, remote mkdir,
  bulk upload/download, and remote rm failures as connection errors.
- docker.py classifies all four _ensure_docker_available() failure paths
  (missing exe, non-executable exe, daemon timeout, `docker version`
  failure).
- terminal_tool catches EnvironmentConnectionError and returns a
  structured tool result the model can act on:
    {"status": "degraded", "reason": ..., "retry_hint": ..., "exit_code": -1}
  The failed backend is evicted from the environment cache so a later
  call retries from scratch — recovery is automatic once the backend is
  reachable again.
- Config gate terminal.degraded_mode: warn|fail (default warn) in
  config.yaml, bridged as TERMINAL_DEGRADED_MODE across all four bridge
  sites (cli.py env_mappings, gateway/run.py _terminal_env_map,
  TERMINAL_CONFIG_ENV_MAP, DEFAULT_CONFIG). "fail" preserves the
  historical error+traceback tool result.
- Command failures (nonzero exit, command-not-found) are NOT touched —
  only infrastructure failures classify as degraded.

Tests: tests/tools/test_terminal_degraded_mode.py (15 tests) covering
exception classification for ssh+docker, structured degraded results,
no-caching of degraded envs, recovery after the backend returns,
nonzero-exit results unaffected, fail-mode preservation, invalid-mode
fallback to warn, and the four-site config bridge invariant.

Inspired by: Claude Cowork degraded-backend behavior (idea-level,
docs-only evidence).
2026-08-07 09:07:55 -07:00
Teknium 1006faa6f8 feat(doctor): add opt-in `hermes doctor --live` real-call backend probes
Adds a bounded, read-only health probe per CONFIGURED tool backend, run
only when the user explicitly passes `--live` (real network calls):

- Firecrawl: credit-usage metadata GET (auth check, no scrape spend)
- FAL: models metadata GET (never a generation)
- Browser: headless launch + about:blank + close (full cleanup)
- MCP: initialize + tools/list per configured server (reuses the
  `hermes mcp test` machinery in mcp_config._probe_single_server)
- TTS/STT: provider models/voices list GET (openai/groq/elevenlabs);
  local providers (edge/piper/faster-whisper/...) skipped

Invariants:
- Opt-in only: zero probes without --live (default False)
- Bounded: sequential, per-probe timeout (doctor.live_probe_timeout,
  default 10s, config.yaml knob)
- Never mutates state; unconfigured backends skip with a note
- Failure isolation: every probe wrapped in a catch-all; a probe crash
  can never break the doctor run; failures append to the issues summary

New: hermes_cli/doctor_live.py, tests/hermes_cli/test_doctor_live.py
(23 tests, probes mocked at the HTTP/client seam).
Wired: --live flag in subcommands/doctor.py; run_doctor calls
maybe_run_live_checks after all static checks.

Coordination: PR #70124 (--probe-routes) probes LLM routes; this flag
probes TOOL backends — different surface, no code-region collision
(the run_doctor hook here sits at the end-of-run summary, not the
API Connectivity section #70124 extends).

Inspired by: paradigmxyz/centaur tool-health-smoke (MIT/Apache-2.0);
sibling: #70124 (LLM route probes — different surface)
2026-08-07 09:07:48 -07:00
Teknium 0ebaa490b5 test: use valid 2-task batches in schema-rejection tests
The batch quality gate (#81141) now rejects 1-task batches before
schema coercion runs; exercise schema rejection with a valid batch.
2026-08-07 09:07:42 -07:00
Teknium d6ee58b583 feat(delegation): optional structured-output schema on delegate_task
Per-task `output_schema` (JSON Schema object) on task items plus the
top-level single-goal form — a one-time static addition to the tool
schema (never varies per call).

- Child side: the schema is appended to the child's context as an
  explicit OUTPUT CONTRACT block before spawn.
- Completion side: the parent validates the child's final answer with
  jsonschema; on failure it sends exactly ONE bounded retry turn
  carrying the validation errors verbatim (no schema re-paste).
- Result entries gain schema_valid (+ schema_retries, and schema_errors
  on final failure) ONLY when a schema was requested; schema-less calls
  keep a byte-identical result shape.
- Malformed schemas are rejected loudly at dispatch (coerce_output_schema
  meta-validates via jsonschema's validator_for/check_schema).
- New helpers in tools/delegation_output_schema.py: coerce, contract
  block, fence/prose-tolerant extraction+validation, retry message.

Pattern from: github/copilot-cli ctx.agent(prompt,{schema}) — PATTERN
ONLY, zero code/prompt text copied (proprietary); proven consumer:
delegate-task-output-patterns skill.

Tests: tests/tools/test_delegate_output_schema.py (24 tests — valid
first try, invalid->retry->valid, invalid twice -> schema_valid false +
errors surfaced, retry-exception degrade, no-schema legacy shape pin,
dispatch rejection, contract plumbing). Delegation suite: 221/221 green.
2026-08-07 09:07:42 -07:00
Teknium e166159f26 feat(vision): optional region zoom crop on vision_analyze
Add an optional `region: [x1, y1, x2, y2]` parameter to vision_analyze
(pixel coordinates in the ORIGINAL image space). The crop is applied
with Pillow BEFORE the downscale/embed-cap pipeline, so the cropped
region gets the full resolution budget — a zoom for reading small text
or UI details after a full shot.

- New `_crop_image_region` helper: clamps out-of-bounds coordinates to
  the image, rejects zero-area/inverted/malformed regions with an error
  naming the actual image dimensions so the model can retry sensibly.
- Wired into both the native fast path (`_vision_analyze_native`) and
  the legacy aux-LLM path (`vision_analyze_tool`).
- Schema gains one static optional param (byte-stable thereafter); the
  description documents the intended flow: full shot first, then zoom.
- No region supplied = behavior unchanged (regression-guarded).

Tests: tests/tools/test_vision_region.py (11 tests — crop applied,
clamping, zero-area rejection with dims, malformed input, pre-downscale
full-budget zoom, schema shape, handler pass-through, no-region
unchanged). Widened one narrow fake_native stub in test_vision_tools.py
to be kwargs-tolerant.

Ported from: QwenLM/qwen-code zoom-image.ts (Apache-2.0)
2026-08-07 08:58:49 -07:00
Teknium fe66596df3 feat(security): protected agent-instruction files always require write approval
write_file/patch targeting AGENTS.md, CLAUDE.md, SOUL.md, .cursorrules, or a
project-local .hermes config dir now ALWAYS prompt the human for approval —
even under --yolo/auto-approve — and fail closed when no human channel
exists. These files steer future agent behavior, so an injected write to
them is a prompt-injection persistence vector.

Design:
- New _check_protected_instruction_write() in tools/file_tools.py, a
  sibling of _check_sensitive_path that returns approval-required rather
  than a hard error. It realpaths before matching (symlink lesson from
  #41351), matches basenames case-insensitively in ANY directory, rejects
  './x/../AGENTS.md' traversal via normpath, and gates files whose
  immediate parent dir is `.hermes` (project-local config) while exempting
  the authoritative ~/.hermes home (governed by its own guards).
- Approval is ONE-OPERATION only: no session/permanent persistence, no
  yolo bypass — intentionally does not route through _run_approval_gate.
  Gateway sessions get the button round-trip with allow_permanent and
  allow_session both False; CLI uses the per-thread approval callback;
  no channel at all = BLOCKED (fail closed).
- Multi-file V4A patches: ONE protected file gates the ENTIRE patch (a
  single prompt lists all protected targets; deny applies nothing).
- Config: security.protected_instruction_files (default true) and
  security.protected_instruction_extra_patterns (fnmatch on basename).
  Config read failure keeps the gate ON.

Tests: 22 new cases in tests/tools/test_file_write_safety.py covering the
adversarial checklist — deny/approve/yolo-bypass attempt, symlink at a
protected target, case variants, relative traversal, arbitrary-directory
basenames, project-local .hermes, checkout-nested-under-~/.hermes
non-gating, patch replace + V4A multi-file atomicity, gateway round-trip,
fail-closed with no human, config off/extra patterns.

Ported from: RooCodeInc/Roo-Code RooProtectedController (Apache-2.0);
companion: #58631 (terminal vector), symlink lesson from #41351.
2026-08-07 08:58:38 -07:00
Teknium c8369e37f4 feat(mcp): trust-tier gating for write-capable MCP tools via readOnlyHint
Adds a per-server `trust: full|untrusted` config key
(mcp_servers.<name>.trust). On an untrusted server, every write-capable
tool call — any tool whose discovery-time annotations do not carry
readOnlyHint=True — routes through the existing approval surface
(tools.approval.request_elicitation_consent, same lazy-import +
surface-routing pattern the MCP elicitation handler uses) before the RPC
fires. Denied/cancelled/errored approvals fail closed: the RPC never
runs, including the lazy first-use server spawn.

Design points:
- Classification happens at CALL TIME from metadata captured at
  DISCOVERY (_record_tool_trust_metadata in _register_server_tools and
  the lazy cache-registration path). No toolset/schema mutation, so the
  toolset stays byte-stable and prompt caching is preserved.
- readOnlyHint is a server-supplied HINT: on an untrusted server a lying
  server can at most skip approval for tools it claims read-only — it
  can never widen access. Trust tiering itself is operator config.
- Missing/malformed annotations => write-capable (fail closed).
- Unrecognized trust values => untrusted (fail closed); missing key =>
  full (backward compatible, documented in mcp-config-reference).
- The schema cache now persists readOnlyHint so lazy-registered servers
  gate identically on next startup without spawning.

Tests: tests/tools/test_mcp_trust_gating.py (11 tests, TDD red->green):
approval invoked + accept proceeds, deny/cancel blocks RPC, readOnlyHint
=true skips gate, trusted/unconfigured servers skip gate, explicit
readOnlyHint=false gated, approval exception fails closed, trust
normalization, discovery-time capture (SDK objects and cached dicts).

Ported from: cloudflare-os classifyTool() (Apache-2.0), corroborated by
Claude Cowork (idea-level).
2026-08-07 08:58:32 -07:00
Teknium 37cc999926 feat(mcp): collapse const-only anyOf/oneOf unions to property enums
MCP servers generated from Rust/TypeScript union types commonly emit
closed value sets as const unions:

    {"anyOf": [{"const": "red"}, {"const": "green"}, {"const": "blue"}]}

Strict tool-calling backends reject or mishandle these; the equivalent
property-level enum form is universally supported. Add
collapse_const_unions() to tools/schema_sanitizer.py and wire it into
the _normalize_mcp_input_schema discovery pipeline after the nullable
strip.

Rules:
- Collapse only when EVERY non-null branch is a pure const of the same
  primitive type (bool never merges with integer).
- Mixed unions, non-uniform const types, and mismatched declared types
  pass through untouched.
- A single {"type": "null"} branch is tolerated: consts -> enum,
  null -> nullable: true hint (matches strip_nullable_unions, which
  leaves null+multi-const unions alone by its one-non-null-branch rule).
- Outer title/description/default/examples carried onto the replacement.
- Deterministic, branch-order-preserving, non-mutating — applied at
  discovery only, so schemas stay byte-stable per conversation.

Ported from: block/goose tool_schema_normalize.rs (Apache-2.0)
2026-08-07 08:58:25 -07:00
Teknium 9fad45fcda feat(kanban,mcp): orphaned-card reconciliation + per-server MCP identity header
Two small config-gated features:

1. Kanban orphaned-card reconciliation (kanban.reconcile_orphans, default
   true, config.yaml): a running card with broken claim bookkeeping
   (claim_lock or claim_expires NULL — crash mid-claim, manual SQL, DB
   restore) is invisible to all existing recovery paths
   (release_stale_claims requires claim_expires NOT NULL,
   detect_crashed_workers requires host-local lock + pid,
   detect_stale_running is config-disabled by default) and shows Running
   forever. New reconcile_orphaned_running() pass in kanban_db.py runs
   each dispatch_once tick: requeues orphans to ready with an explanatory
   comment, closes any leaked run, emits a 'reconciled' event, and defers
   when the recorded PID is still alive on this host (never requeue
   beside a live worker). Surfaced via DispatchResult.reconciled_orphans.

2. Per-server MCP identity header (mcp_servers.<name>.identity_header,
   config.yaml): optional {name, value_from: static|profile, value}
   mapping; the header is attached to that server's HTTP/SSE transport
   requests. 'static' sends the config value; 'profile' resolves the
   active Hermes profile name once at connect time (no per-call
   mutation). Explicit per-server headers of the same name (any casing)
   win. Invalid blocks warn-and-ignore; stdio servers warn-and-ignore.

Tests: tests/gateway/test_kanban_reconcile_orphans.py (9),
tests/tools/test_mcp_identity_header.py (13), all written first (RED)
then implemented (GREEN). No new HERMES_* env vars.

Inspired by: openai/symphony tracker reconciliation (Apache-2.0) +
Poke per-user MCP identity (idea-level).
2026-08-07 08:58:20 -07:00
Teknium 5db1b72b1f feat(cli): global emergency stop — `hermes pause` / `hermes resume`
Resumable ESTOP sentinel at $HERMES_HOME/ESTOP that halts NEW work only:

- agent/estop.py: sentinel engage/disengage/is_engaged (single stat, no
  caching), optional reason + timestamp stored as JSON, paused_reply()
  notice, check_paused() log-once-per-engagement helper. Corrupt/empty
  sentinel still pauses (fail safe); a `touch ~/.hermes/ESTOP` works.
- cron/scheduler.py: tick() skips dispatch while engaged (logged once per
  engagement, not per tick). Due jobs simply wait for the next tick after
  resume — in-flight runs are never touched.
- gateway/kanban_watchers.py: dispatcher skips auto-decompose and worker
  spawning while engaged; zombie reaping still runs and running workers
  finish naturally.
- gateway/run.py: new gateway turns (post-auth, non-internal) get a brief
  "Hermes is paused" reply instead of an agent run. Internal events
  (in-flight background completions) bypass the gate.
- hermes_cli/subcommands/pause.py: `hermes pause [--reason]` and
  `hermes resume`, wired into main() and _BUILTIN_SUBCOMMANDS.
- hermes_cli/status.py: `hermes status` shows a PAUSED banner (one stat).
- tests/test_estop.py: 20 tests — sentinel lifecycle, reason surfacing,
  log-once, cron skip + resume, kanban gate, gateway paused reply +
  internal bypass, CLI idempotence, builtin-set parity, status line.

Never kills in-flight work; resumable with no restart. Footprint ladder:
CLI command only, no new model tool, no new env vars.

Ported from: gastownhall/gastown estop.go (MIT); related prior art:
#26778 (/panic — kill/exit semantics, deliberately different: ours is a
resumable pause), #44617 (interrupt in-flight cron — out of scope here).
2026-08-07 08:58:14 -07:00
Teknium d7635e43bb feat(delegation): surface per-delegation cost in the result entry
Each serialized result entry now carries cost_usd (rounded to 6 dp)
and cost_status (the child's session_cost_status — 'estimated',
'reported', 'included', or 'unknown') alongside tokens/api_calls/
duration, so the parent model can see what each delegation cost.

The internal _child_cost_usd field is still stripped before
serialization and the parent session cost rollup is untouched.
Tool schema is unchanged (byte-stable).

Inspired by: Perplexity Agent API result shape (idea-level)
2026-08-07 08:58:02 -07:00
Teknium 94bc3194b3 feat(delegation): validate batch task quality before spawning children
Reject malformed tasks=[...] batches before any child agent is spawned:

- exact-duplicate goals (case/whitespace-normalized), error names both
  task indices
- placeholder goals: bare 'TODO', bare 'task N', unexpanded <...> or
  {...} template markers, or goals shorter than 10 chars after strip
- 1-task batches, with an error pointing the model at the single
  `goal` form instead

All checks are batch-only — the single-goal form is exempt by design
(short goals like goal="test" are valid there). Error strings are
actionable: each tells the model exactly how to fix the call.

Tool schema is unchanged (byte-stable); validation is runtime-only in
the existing batch-validation region.

Existing tests using terse batch goals ("A"/"B"/"C") updated to
realistic distinct goals per the new contract.

Inspired by: MoonshotAI/kimi-code agent-swarm.md validation rules (MIT)
2026-08-07 08:57:57 -07:00
Teknium ed903f953e feat(cron): pre-dispatch configuration validation (blocked_config + alert-once)
Validate a job's configuration BEFORE any agent machinery is constructed:

- missing provider API key (AuthError from a read-only
  resolve_runtime_provider probe; skipped when a fallback_providers chain
  is configured, since auth-fallback may rescue the run)
- attached skill not ready (skill_view readiness_status=setup_needed —
  missing required env vars / commands / credential files)
- delivery platform unknown or unconnected (deliver=local/origin/all are
  never checked; gateway-config load failures fail open)

On a failing check run_job returns a [blocked_config]-marked error without
constructing AIAgent/MCP/etc, so a misconfigured job never burns an LLM
call. run_one_job records last_status='blocked_config' and delivers the
alert exactly ONCE across ticks (persisted preflight_alerted bit — the
alert-once shape from the #73506 dead-pin auto-pause); the next healthy
run clears the marker so a future break re-alerts. Every preflight check
fails open: only an affirmative misconfiguration verdict blocks.

Config: cron.preflight (default true); `cron.preflight: false` restores
the old fail-during-run behavior. Documented in the cron user guide and
config defaults.

mark_job_run gains an optional status= override (unblocked call shape
unchanged) and drops preflight_alerted on any successful run.

Tests: tests/cron/test_preflight_config.py (blocked_config + no agent +
single alert across two ticks, healthy job unaffected, recovery clears
dedup, fallback-chain rescue, opt-out restores old behavior, skill
readiness miss, unknown delivery platform, deliver=local never loads
gateway config). Full tests/cron/ + cronjob tool suite green (525 tests).

Ported from: paperclipai/paperclip execution-semantics §5 (MIT);
in-repo precedent: #27948, #73506
2026-08-07 08:57:53 -07:00
Teknium 04e8a661f2 feat(cron): per-job durable notepad — KV scratchpad surviving scheduled runs
- cron/notepad.py: SQLite-backed cron_notepad(job_id, key, value,
  updated_at) store in its own profile-local db (cron/notepad.db),
  following the executions.py connection/transaction pattern. APIs:
  set_note/get_note/delete_note/list_notes/clear_notepad +
  render_notepad_section. Documented size caps: 16KB per value,
  128-char keys, 64KB per job total; oversized writes raise ValueError.
- cron/scheduler.py: inject non-empty notepads into the job prompt at
  the context_from data-injection seam as a clearly-labeled
  "Job notepad (persistent across runs)" section that also documents
  the CLI write path. Empty notepad renders "" — byte-stable prompts
  for jobs that never use the feature.
- hermes_cli/cron.py + hermes_cli/subcommands/cron.py:
  `hermes cron notepad <job_id> [get|set|delete|list]` under the
  existing cron subcommand tree (no new top-level command, no new
  model tool — the agent writes via terminal + CLI).
- tests/cron/test_notepad.py: CRUD, durability, cap enforcement,
  prompt injection, byte-stable empty case, read-failure resilience,
  CLI handler + dispatch (TDD; watched fail first).

Inspired by: Amp (Sourcegraph) cron notepad (idea-level, proprietary —
zero code).
2026-08-07 08:57:48 -07:00
Teknium 6dff2109aa feat(cron): monitor-mode jobs — hash-suppressed change detection
Add monitor-mode cron jobs: a cheap monitor source (monitor_script or
monitor_url) runs on every tick BEFORE any agent machinery is built.
Its output is hashed as exact bytes and compared to the hash stored
from the last agent-triggering tick:

- unchanged  -> agent run suppressed entirely (no LLM, no delivery);
  the tick is recorded as a silent no_change run visible in the
  executions ledger doc
- changed    -> a MONITOR CHANGE DETECTED block (capped unified diff of
  previous vs current output + the new output) is injected into the
  prompt via the existing extra_prompt seam, then a normal agent run
- first run  -> always runs the agent with a baseline block
- source failure -> delivered as an ERROR alert, never treated as a
  change; the stored hash is untouched so recovery to prior output
  still suppresses

Implementation:
- cron/monitor.py (new): hash/diff/URL-fetch/state persistence.
  monitor_script reuses _run_job_script (same ~/.hermes/scripts/
  containment + interpreter rules); monitor_url is a bounded GET
  (30s, 256KB, http/https only). Output is exact bytes by design —
  scripts should emit stable output (documented).
- cron/jobs.py: additive job fields monitor_script / monitor_url /
  monitor_state {last_output_hash, last_changed_at}. JSON job records
  need no migration. create-time validation: sources are mutually
  exclusive and incompatible with no_agent=True.
- cron/scheduler.py: one tight monitor gate in run_job between the
  no_agent short-circuit and the LLM path (outside sibling-lane
  regions). State persists in jobs.json + a per-job snapshot file, so
  suppression survives scheduler restarts.
- tools/cronjob_tools.py: additive optional monitor_script/monitor_url
  params on the cronjob tool (create + update, empty string clears),
  path containment validated at the API boundary, surfaced in
  _format_job.
- hermes_cli: --monitor-script/--monitor-url on `hermes cron create`
  and `hermes cron edit`; `hermes cron list` shows the monitor source
  and last-changed time.

Tests (tests/cron/test_monitor_kind.py, TDD): unchanged suppresses,
changed injects diff, first run always runs, hash persists across
module reload (restart), script failure is error-not-change with hash
untouched, create/update validation, tool wiring + path-escape reject.

Inspired by: ChatGPT Work monitor tasks (idea-level, docs-only);
enabler: #80774
2026-08-07 08:57:44 -07:00
Teknium 563f0a6fde feat(cli): add `hermes approvals test` — dry-run approval verdict CLI
Answers "what would the approval system do with this command?" without
executing it, prompting anyone, or persisting anything. Composes the
REAL runtime evaluators from tools/approval.py in the same order as
check_all_command_guards: container-skip gate, hardline blocklist,
sudo-stdin guard, user approvals.deny rules, yolo/mode-off bypass,
permanent command_allowlist, dangerous-pattern detection. Because the
same functions run — including _command_detection_variants's
normalization/de-obfuscation path — an obfuscated command gets exactly
the verdict its plain form would get at runtime, and the output shows
the normalized-variant trace the detectors actually evaluated.

- hermes_cli/approvals_test.py: evaluate_command() + text/JSON output.
  Script-friendly exit codes: 0 allow, 1 usage, 2 ask-approval, 3 deny
  (hardline / sudo-stdin / user deny rule).
- hermes_cli/subcommands/approvals.py: `test` subparser with --env-type
  (default local), --json, and a REMAINDER command (dest command_words —
  NOT "command", which main.py's startup path reads as the top-level
  subcommand name).
- hermes_cli/approvals_suggest.py: dispatch `test` and mention it in the
  bare-`hermes approvals` usage text.
- tests/hermes_cli/test_approvals_test.py: verdict matrix (benign /
  hardline / dangerous / user-deny from config / container skip /
  mode=off vs hardline), obfuscated==plain verdict parity with
  normalized trace, spy proof that the real runtime detectors are the
  ones invoked, read-only invariants (nothing executed; prompt and
  persistence paths rigged to explode), JSON shape, dispatcher and
  parser wiring.

Read-only by construction: only detection/matching functions are
called; the approval gate, prompts, gateway notify, and allowlist
writers are never reached.

Inspired by: Amp `permissions test` (idea-level, proprietary — zero code)
2026-08-07 08:57:39 -07:00
kshitij 7cf71c32bb fix: follow-ups for salvaged PR #80740
- Give cached_fetch_api_models the same stale-while-revalidate tier as
  cached_provider_model_ids: TTL-expired entries within the 7d window are
  served instantly while a background refresh rewrites the cache —
  without this, every /model open an hour into the session re-blocked on
  the live probe (#72762's stall class, deferred).
- Generalize _spawn_swr_refresh(cache_key, refresh_fn) so non-slug
  custom:<base_url> keys reuse the same inflight-dedupe scaffolding;
  slug behavior unchanged (default refresh_fn preserved).
- Convert the missed sibling site: acp_adapter/server.py
  _named_custom_provider_catalogs() live-probed every custom_providers
  row's /v1/models per ACP catalog build.
- Extract _cache_entry_valid() (the fp/models predicate existed 4x) and
  validate 'at' is numeric so hand-edited/corrupt cache JSON degrades to
  a live fetch instead of raising through the picker's blanket except.
- Flatten the dead api_mode conditional (fetch_api_models declares
  api_mode=None; branch was behaviorally inert).
- Tests: 4 new guards (stale-serve, stale-window cutoff, generalized SWR
  write-through, corrupt-at degradation) — stale-serve and corrupt-at
  mutation-checked; 2 existing tests updated for the new behavior.
2026-08-07 21:02:40 +05:30
Prashant Jain fb435aae97 perf(model): disk-cache custom-provider /v1/models probes
Custom OpenAI-compatible endpoints (named custom_providers rows, bare
provider: custom, and per-endpoint-map entries) called fetch_api_models()
directly at three call sites in model_switch.py, with no disk cache — unlike
first-class providers, which go through cached_provider_model_ids(). Every
plain /model open live-probed the active custom endpoint's /v1/models,
regardless of how recently it had already been probed.

Adds cached_fetch_api_models() in hermes_cli/models.py: a TTL disk-cache
wrapper keyed on custom:<base_url> (custom endpoints have no
PROVIDER_REGISTRY slug to key on) and fingerprinted on api_key/api_mode/
headers, with the same stale-beats-nothing fallback policy as
cached_provider_model_ids(). Routes all three probe call sites through it.

Since prewarm_picker_cache_async() already calls list_authenticated_providers()
with probe_custom_providers defaulting True, this also fixes the endpoint
being warmed on boot (populating the disk cache) instead of that work being
discarded on every open — any custom endpoint (an LLM gateway, a
self-hosted vLLM/SGLang server, etc.), not just one specific provider.

Fixes #72762. Salvaged from #72810 per review feedback: extracts just the
verified custom-endpoint cache fix with real cache-contract test coverage
(hit/stale/rotation/refresh/fallback), leaving the credential-pool and
Copilot-token-exchange costs described in the issue for separate follow-up.
2026-08-07 21:02:40 +05:30
kshitij 83bad5cdda fix: follow-ups for salvaged PR #80795
- Drain the eviction plan (pop + del) before trim_memory: the batch
  thread previously held every evicted agent in its local list while
  gc.collect + malloc_trim ran, so the in-pass trim freed almost
  nothing, the next tick re-read a still-high RSS, and the valve
  over-evicted an extra batch of warm prompt caches per cycle.
- Clear _db_flush_scan_prefix in _release_evicted_agent_soft: it is a
  shallow copy of the flushed transcript (stamped on every successful
  flush) sharing every message dict — and pressure-evictable agents
  have flushed by definition, so it pinned the multi-MB content strings
  on exactly the agents the valve targets.
- Config-read failure now falls back to resolve_agent_cache_bounds({})
  instead of bare AgentCacheBounds(): the dataclass default disables
  the pressure pass, but an absent config section means 'auto' — a
  transient read failure must not permanently switch off the OOM valve.
- protect_recent: false (YAML bool; False == 0) keeps the default MRU
  protection instead of silently disabling it.
- 'No evictable session' warning now distinguishes sessions blocked on
  un-flushed persistence (e.g. session DB never initialized — NFS
  HERMES_HOME) from mid-turn agents, so operators can diagnose why the
  valve isn't shedding instead of being pointed at running turns.
- _cgroup_limit_bytes checks the process's own cgroup (via the existing
  gateway.cgroup_cleanup._own_cgroup_path) before the root files, so a
  systemd unit's MemoryHigh=/MemoryMax= is detected — the root
  memory.high/max read 'max' on those deployments.
- Tests: 5 new guards; drain-before-trim and scan-prefix-clear
  mutation-checked (revert each fix -> its guard fails).
2026-08-07 21:02:33 +05:30
HexLab98 6bbe55dd09 fix(gateway): bound the agent cache by memory, not just count and age
The per-session agent cache is capped at 128 entries with a 1h idle TTL, and
neither bound knows how many bytes it holds. Each cached agent pins
_session_messages -- the full transcript including tool output, tens of MB on
a session with 100+ tool calls -- so a gateway serving many chats keeps every
warm transcript resident: agents that took a turn inside the TTL are never
idle-swept, and the idle sweep additionally defers finalizable sessions until
they expire. RSS climbs until the cgroup throttles and SIGTERM can no longer
flush inside systemd's stop timeout.

Add the missing bound. Each session-expiry watcher tick compares the process's
anonymous RSS against a budget and, when over, sheds LRU agents through the
same soft-eviction path the cap enforcer uses, then runs malloc_trim so the
freed arenas actually return to the OS. Evicted sessions rebuild their
transcript from the persisted session on the next turn.

Three classes of session are never shed: agents mid-turn, the most recently
used ones, and any session whose transcript has not finished reaching disk
(_last_flushed_db_idx vs len(_session_messages) -- the same divergence the FTS
write-corruption guard reacts to when it preserves live history).

memory_high_mb defaults to "auto", deriving the budget from the cgroup limit
the gateway runs under, so a MemoryHigh/MemoryMax on the unit is respected
without a second number to keep in sync. The two existing bounds become
configurable alongside it under agent.agent_cache.

protect_recent is clamped to half the cache: a couple of sessions can exhaust
the budget on their own, and a fixed MRU guard would then protect everything
and leave the gateway climbing with nothing it would shed.

Fixes #80764
2026-08-07 21:02:33 +05:30
kshitij 4a3942d948 fix: show explicit member spend cap message instead of 'no credits'
When the Nous Portal returns paid_service_access.allowed=false with
reason=member_spend_cap_exceeded, Hermes was falling through to the
generic 'no active subscription or usable credits' message — even
though the user has ample purchased credits and the real blocker is
an org-level per-member spend cap.

This adds a dedicated branch that surfaces the actual cause: names the
spend cap, shows the cap/spend amounts, and tells the user to ask their
org admin to raise it. Also adds member_spend_cap_exceeded to the
billing error code set so the error classifier and auth error formatter
route it through the Nous entitlement message path.
2026-08-07 19:47:40 +05:30
kshitij 813793db23 test(tui_gateway): pin the raising-close swallow + no-retry contract
Review finding: the close block's comment promises a raising
session_db.close() is swallowed with the flag already cleared (no re-close
on a second agent.close()), but nothing pinned it. One test with a raising
_RecordingDB proves both halves.
2026-08-07 19:44:41 +05:30
Yishova be14a4bee3 tui_gateway: close dedicated profile SessionDB handles at teardown too
Follow-up to the review on the session.resume ownership fix. Closing the
pre-transfer early returns left two gaps, both real.

1. The transfer had no owner on the other side. Once ownership moved to the
   agent, teardown ran AIAgent.close() (via _teardown_session on session.close
   and the orphaned-session reaper), which called session_db.end_session() —
   that finalizes the session ROW, not the connection. A successfully resumed
   profile session kept its dedicated handle, its db/-wal/-shm fds and its
   background token-writer thread for the life of the gateway.

   AIAgent now carries an explicit _owns_session_db, defaulting False so the
   SHARED launch handle — which outlives every agent and backs every other live
   session — is still never closed there. Only the dedicated-open sites set it,
   at the point ownership actually changes hands.

2. session.resume was not the only profile-scoped open with no close on its
   failure paths. Covered here with the same flag, via a _transfer_db_to_agent
   helper that refuses the transfer unless the agent really holds that handle:

   - the deferred builder (_start_agent_build), including the session-reaped-
     mid-build case, where the built agent is discarded and never torn down, so
     transferring to it would leak exactly as before;
   - session.branch's branch_db;
   - the compute host's per-profile open;
   - AIAgent's own lazy open in _get_session_db_for_recall, which no other
     object ever references and so was unconditionally abandoned.

Where a handle has already reached a registered session, the drop is
unconditional and the transfer is best-effort on top: a refused transfer leaves
the old leak, which is survivable, whereas closing under a live session is the
permanent "Cannot operate on a closed database" break the original patch exists
to avoid.

Tests: tests/tui_gateway/test_session_db_ownership_teardown.py (new, 14).
11 of the 14 fail without this change; the 3 that pass are the "must NOT close"
guards, which hold in both directions by design.
2026-08-07 19:44:41 +05:30
Yishova 79625e3c0b tui_gateway: session.resume abandons the profile SessionDB it opens 2026-08-07 19:44:41 +05:30
kshitij fecba5afcc refactor(agent): fold simplify findings — DB picker parity, single scan, canonical strip delegation
Review-pass follow-ups (three parallel reviewers, findings verified):

- hermes_state_search.py list_recent_user_messages now drops legacy
  standalone compaction handoffs in the decode loop (SQL can't see them:
  durable role=user, no display_kind). Closes the /undo N pairing skew
  where the in-memory count (new predicate) and the DB soft-delete pick
  (old predicate) targeted different turns on legacy sessions. Fetches
  with headroom so the requested limit is still honored. 3 new tests,
  mutation-checked (no-op'ing the skip fails 2/3).
- _should_skip_model_call_for_reference_handoff: single drive-check scan
  (was two — once inside the restore helper, once after); the restore
  helper no longer re-scans and its return value now decides the verdict.
- _final_response_from_messages replaced by the _HANDOFF_SKIP_FINAL_RESPONSE
  constant it always returned (parameter was unused).
- _handoff_carries_live_user_content delegates to the canonical
  _strip_context_summary_handoff_message — also fixes the edge where a
  merged-shaped row with an EMPTY preserved prior tail was wrongly
  treated as carrying live content.
- Site-level guard test for rollback.restore with a legacy handoff row
  (predicate-in-context, complements the unit tests).
2026-08-07 19:44:35 +05:30
HexLab98 b9636b1047 test(agent): cover reference-only handoff sole-active-turn regression
Pin #80622 invariants: handoff alone must not drive a model call after
stop, pending real users are restored, and synthetic compaction rows are
never treated as user-originated turns. Also give micro-compaction
enough passes to pay back the longer SUMMARY_PREFIX marker overhead.
2026-08-07 19:44:35 +05:30
Alan Hsu 6d89b10653 fix(agent): project real usage in preflight defer instead of fixed growth tolerance
The rough preflight estimate intentionally overestimates, but not by a
fixed margin: CJK text is counted at ~1.7x its o200k cost and
Responses-mode reasoning replay blobs at several times their billed
cost. Heavy sessions show rough estimates 2-3x real usage and compact
at 35-55% of the real window, stalling turns for minutes and discarding
detail (churn), because the defer guard only tolerated 5% rough growth
and sessions that never compressed had no baseline at all.

Pair every request's rough estimate (note_request_rough_estimate,
recorded in the conversation loop right after the pressure estimate)
with the provider's real prompt_tokens in update_from_response(), then
defer preflight while projected real usage — last real + rough growth
since that reading — stays under the threshold. Rough growth is itself
an overestimate of real growth, so the projection is an upper bound and
deferring below the threshold is safe; the provider's context-overflow
handler remains the backstop.

The baseline no longer ratchets on defer: it is refreshed by the
response pairing, and advancing it without a matching real reading
would shrink apparent growth and defer on stale data.
2026-08-07 19:44:29 +05:30
kshitij 72c63aa586 fix(stt): close idle-unload races — strong model ref, single long-lived watcher
Review pass on the idle-unload feature found two material concurrency
bugs; both fixed here with a regression guard:

1. Unload-vs-use null deref (HIGH): _transcribe_local re-read the
   module global _local_model at the transcribe call site. An idle
   unload firing between the model load and transcribe() evaluated
   None.transcribe → AttributeError → user-visible 'Local
   transcription failed'. The window was real: the idle timer was only
   touched AFTER a successful transcription, so a voice note arriving
   exactly as the timeout expired raced the watcher directly.
   Fix: bind a strong local reference under the model lock and use it
   for the whole transcription (the watcher can null the global at any
   time; this in-flight call keeps its instance — the generator holds
   self, so no use-after-free). Also touch the idle timer at the START
   of transcription so a long in-flight transcribe can't be counted as
   idle time. The CUDA-fallback retry path gets the same treatment
   (locked global write, local ref use).

2. Watcher replacement race + response-path join (MEDIUM/HIGH): the
   old design stopped and re-started the watcher after EVERY
   transcription with an unlocked set/join(5)/clear/start sequence on
   shared globals. Two concurrent voice messages could interleave to
   leave TWO live watchers (one with a stale, shorter timeout — a
   raised unload_after_idle_seconds could still unload on the old
   value), and the join(timeout=5) sat on the user-visible response
   path (a watcher blocked on _local_model_lock during a concurrent
   multi-second model load stalls the reply up to 5s).
   Fix: single long-lived watcher under a management lock — started
   only when none is alive (per-transcription cost: one lock + one
   is_alive check), re-reads the configured timeout from config every
   cycle (config edits now apply within one 30s interval, without
   waiting for the next voice message — previously undocumented), and
   stands down without unloading when the timeout is set to 0
   mid-idle.

Tests: 17 now — idempotent start (same thread, no churn), config
re-read + stand-down-when-disabled, and the race guard
(unload firing mid-transcription must not fail the in-flight call).
The race guard is mutation-verified: reverting the fix (re-reading
the global at the call site) makes it fail with the exact NoneType
error; the fixed code passes.
2026-08-07 19:26:12 +05:30
kshitij 7b006ea6e8 feat(stt): idle unload for local whisper model
The local faster-whisper model singleton (_local_model) is loaded once
and never released — the 'base' model holds ~370 MB of RAM/VRAM for
the entire lifetime of the process, even when no voice messages arrive
for hours or days. On long-running gateway processes (especially with
local LLMs competing for the same GPU) this is wasteful.

Add a config-driven idle unload: after stt.local.unload_after_idle_seconds
(default 0 = never) of no transcription activity, a lightweight daemon
thread sets _local_model = None so the Python GC can reclaim the
ctranslate2 objects. The next voice message reloads the model
transparently (the existing lazy-load path handles it).

The watcher:
  - Checks every 30s whether idle time exceeds the configured threshold
  - Acquires _local_model_lock before unloading (prevents races with
    concurrent transcriptions that are mid-load)
  - Exits immediately if the model is already None (unloaded by another
    path, e.g. the CUDA fallback eviction)
  - Is restarted by each transcription with the current config value,
    so changing stt.local.unload_after_idle_seconds in config.yaml takes
    effect on the next voice message without a process restart

Default is 0 (never unload) — zero behavior change for existing users.
Recommended value for gateway processes: 300 (5 minutes).

15 tests: config resolution (garbage/negative/None fallbacks), unload
safety (already-None, lock acquisition), touch timestamp, watcher
lifecycle (unload after timeout, no unload within timeout, exits when
model already None, stopped on new start). Existing STT test suite
unchanged.
2026-08-07 19:26:12 +05:30
kshitij 3277eb8872 refactor(stt): fold review findings into the cloud silence trim
Three-reviewer pass (reuse / quality / efficiency) on the trim diff;
four findings folded:

1. Short-clip input gate (efficiency, HIGH): the trim previously paid
   the full ffmpeg encode before the <10%-saving discard check — every
   dense conversational voice note burned 3 subprocess spawns + a
   complete re-encode on the synchronous response path for nothing.
   New _CLOUD_TRIM_MIN_INPUT_SECONDS=12 gate: below it, savings can't
   matter (a >=10% saving is ~1s of audio, and several providers bill
   a per-request minimum anyway — Groq bills 10s minimum), so the
   whole pipeline is skipped using the duration we already probed.
   Typical 5-10s voice notes now pay 1 ffprobe (~50ms), not 3 spawns +
   encode (~0.3-1s; multi-second on small-VPS gateway hosts).

2. Shared encode profile (reuse, HIGH): the trim's ffmpeg command
   duplicated _transcode_audio_for_stt's encode byte-for-byte (same
   16kHz/mono/AAC-32k/faststart args, same subprocess.run kwargs).
   Extracted _STT_M4A_ENCODE_ARGS + _run_ffmpeg_stt_encode(ffmpeg,
   in, out, audio_filter=None); both call sites now share one owner,
   so codec/bitrate/timeout changes can't drift between the paths.

3. is_truthy_value for the enable flag (quality, MEDIUM): raw
   bool(cfg.get(...)) treated a YAML string "false" as enabled — the
   exact bug class utils.is_truthy_value (already imported, already
   used by is_stt_enabled and the xai/elevenlabs flags) exists for.

4. All-silence guard scales with keep_ms (quality, LOW): the fixed
   0.3s floor equals the default keep window, so an output consisting
   solely of one kept pause could pass as "speech"; now
   max(0.3, 2*keep_seconds).

Also: _probe_audio_duration docstring documents it as the canonical
sync seconds-probe (gateway/run.py and the Telegram adapter carry
local variants of the same ffprobe invocation).

Tests: 24 now — YAML-string-false disables; short clips skip the
encode entirely (encoder mock asserted not-called); E2E fixtures
moved past the input gate. E2E re-verified: 13.2s note -> 6.2s
(-53%), 8s clip skipped with 1 probe.
2026-08-07 19:26:04 +05:30
kshitij a683ef95d2 feat(stt): pre-upload silence trim for cloud providers
Local faster-whisper gets Silero VAD (bf8004e3a) so silence never
reaches the model. Cloud providers got no such protection: the raw
file uploads untouched, so every second of silence in a voice note is
paid for twice — upload time and per-audio-minute billing — and cloud
Whisper hallucinates junk tokens on silent stretches exactly like
local Whisper did before the VAD hardening. A 13s voice note with two
long pauses is billed as 13s of audio to transcribe ~6s of speech.

Close the gap client-side: before uploading to a built-in cloud
provider (groq/openai/mistral/xai/elevenlabs/deepinfra), collapse long
pauses with ffmpeg's silenceremove filter, keeping
stt.cloud_trim_keep_ms (default 300) of every pause so word boundaries
and natural pacing survive. Uses ffmpeg, already a dependency of this
exact path via _transcode_audio_for_stt — no new dependency.

The trim is strictly best-effort — ALL of these upload the original
untouched, transcription never fails because of the trim:
  - stt.cloud_trim_silence: false
  - ffmpeg/ffprobe missing, trim failure, or timeout
  - trimmed result ~empty (mostly-silence clip: the provider, not a
    client-side dB heuristic, decides whether it contains speech)
  - trim saves <10% (re-encoding for nothing)

Command-type and plugin providers are deliberately NOT trimmed: they
may wrap local CLIs that want the original bytes or run their own VAD.

E2E (real ffmpeg + faster-whisper): 13.2s voice note with 7s pause ->
6.2s upload (-53%); transcript of trimmed audio matches the original
on both utterances. Dense-speech and all-silence WAVs correctly fall
back to the original. 22 unit+E2E tests; STT/voice suite failures
identical to upstream/main baseline (all pre-existing).
2026-08-07 19:26:04 +05:30
kshitij b7eb97a835 fix(vision): stream image and video downloads with chunk-by-chunk size cap
_download_image() and _download_video() both used client.get() +
response.content, buffering the entire media body into memory before
checking the size cap. A server that omits Content-Length could send
an arbitrarily large payload, causing OOM.

Extract _stream_download_to_file() shared helper: streams via
client.stream() + aiter_bytes(), writes chunks to a temp file, enforces
the running byte count against the cap after each chunk, and atomically
replaces onto the destination on success. Cleans up the temp file on
failure. Uses utils.atomic_replace() for cross-device/symlink safety.

Malformed Content-Length values are now caught and ignored instead of
crashing with ValueError; the streaming cap is the authoritative guard.

Approach adapted from PR #10440 by @WuKongAI-CMU (closed as stale —
14923 commits behind, reverted 32 commits of vision_tools.py evolution
including SSRF-safe client, retry classification, and lazy imports).

Closes #10440
2026-08-07 18:50:39 +05:30
Chen Jin 23dce021a5 perf(fts): drain trash tables with a high-water marker instead of re-scanning
_fts_teardown_trash_step deleted rows via 'WHERE key IN (SELECT key
LIMIT N)' — each chunk's subquery re-scanned from the start of the
table, so chunk k skipped past (k-1)xN already-deleted rows: O(n²)
total row visits. On a v22 shadow table with ~230K rows that is on the
order of 10^8 row visits, turning optimize-storage teardown into a
multi-hour grind on slow disks, with a write lock held per chunk.

Single-column INTEGER-PK trash tables now drain via a fts_teardown_<tbl>_progress
high-water marker mirroring fts_rebuild_step: each chunk claims rows
past the marker (SELECT ... WHERE key > ? ORDER BY key LIMIT N), deletes
the claimed range, and publishes the new marker in the same transaction.
Per-chunk work is bounded → O(n) total.

TEXT-PK tables (the FTS config shadow table, pk like 'version') and
compound-key tables fall back to the legacy chunked delete — those are
small by construction.

Fixes #79324
2026-08-07 18:40:47 +05:30
kyssta-exe 025fc7e74b fix(slack): dedupe thread-qualified channel lookups (#80668) 2026-08-07 18:40:33 +05:30
HexLab98 95e78556f4 test(gateway): cover fatal-handler queue-before-disconnect (#80598) 2026-08-07 18:32:45 +05:30
kshitij 2a0d0bc698 refactor(gateway): drop dead degraded token field; de-churn salvage diff
Follow-up to the #80376 salvage:

- TurnLeaseToken.degraded is dead code since acquire() started raising
  TurnLeaseTimeoutError: the only constructor site passes the default and
  repo-wide there are zero external readers. Remove the field, its ctor
  param, the repr segment, and the always-False guards in rebind()/release();
  the class docstring's 'retained for compatibility' claim described
  consumers that do not exist.
- Revert pure black-rewrap churn in test_config_env_bridge_authority.py and
  test_turn_lease.py (hunks on functions this change does not touch), keeping
  the functional Windows-env/encoding additions.
- Turn the default-value test into an invariant: config default must equal
  gateway.turn_lease.DEFAULT_LEASE_WAIT instead of pinning the 1800 literal.
- Rewrap the dangling 'Released' comment line in gateway/run.py.

Verified: 20/20 targeted gateway tests, ruff clean; mutation check — with
gateway/turn_lease.py reverted to pre-fix main the module fails, restored it
passes.
2026-08-07 18:23:57 +05:30
poisdahl 3a3aed3c1f fix(gateway): keep pending turn lease acquires registered 2026-08-07 18:23:57 +05:30
texasich b3e9e91709 fix(gateway): configure turn lease timeout via yaml 2026-08-07 18:23:57 +05:30
texasich b2b681fefd fix(gateway): harden turn-lease timeout rejection 2026-08-07 18:23:57 +05:30
texasich 29af112cd4 fix(gateway): fail closed when session turn lease times out 2026-08-07 18:23:57 +05:30
HexLab98 c24ff38c51 fix(gateway): make a history-dropping submit prove it meant to
prompt.submit honored truncate_before_user_ordinal on every request. A
client that carried a leftover ordinal into an ordinary send therefore
issued something the gateway could not tell apart from a real rewind —
same method, same shape, an in-range target — and the cut was applied
with replace_messages(), which DELETEs the durable rows. One report lost
244 messages (296 -> 52) with no prompt and nothing to restore from.

The existing guard only covered ordinal 0, where the cut empties the
transcript; a mid-session ordinal sailed straight through. Only the
client knows whether a submit is a rewind, an edit, or a regenerate, so
require it to say so: an ordinal without confirm_truncate is refused on
4029 and neither memory nor the DB is touched. Desktop sends the flag
from the one place that builds these params, so every rewind path is
covered and a stale build fails closed with an actionable error instead
of quietly deleting a conversation.
2026-08-07 18:20:19 +05:30
kshitij 30679b876c test(cron): pin fail-closed TERMINAL_CWD lock timeout behavior
- reader/writer run_job timeout paths fail loudly (writer additionally
  proven to never mutate the active holder's TERMINAL_CWD override -
  the fail-open design clobbered it)
- waiter whose holder finishes inside the bound still proceeds
- bound derivation from HERMES_CRON_TIMEOUT (floor, margin, 0/garbage)

The run_job fail-fast test shape follows @necoweb3's #63959.

Co-authored-by: dsad <sswdarius@gmail.com>
2026-08-07 18:11:41 +05:30
dsad 5fcca432f5 test(cron): cover bounded TERMINAL_CWD lock acquisition
Lock-primitive timeout tests from #63959, applied onto the timeout API
that landed via #80912.
2026-08-07 18:11:41 +05:30
kshitij 458ce7b2b4 fix(streaming): close the same mid-tool-call drop gap on the Anthropic path
Sibling of the chat_completions zero-byte-args fix (previous commits):
a clean SSE close after content_block_start(tool_use) but before any
input_json_delta / message_delta yields an SDK final-message snapshot
whose content is NON-empty (the tool_use block is present, input={})
and whose stop_reason is None. That shape sailed past both
empty-stream guards (they only fire on empty content) and executed the
tool with empty input — no retry, no error: the same silent-data-loss
class as #80498, one provider transport over.

A legitimate completion always carries a stop_reason, so a
tool_use-bearing message without one is a mid-tool-call stream drop.
Raise EmptyStreamError for it, riding the same bounded stream-retry
(HERMES_STREAM_RETRIES) the eventless-stream case already uses.

Gate checked on both return paths (raw SDK snapshot and
accumulator-modified message). Regression tests cover the dropped
shape (mutation-verified: disabling the gate fails exactly that test),
the legitimate tool_use completion, and the text-only no-stop_reason
shape (pre-existing behavior preserved).
2026-08-07 18:07:19 +05:30
joaomarcos e6f31b07cb test(streaming): cover mixed tool-call and retry-exhaustion paths for #80498
Locks in two gaps left by 015a114a2 (#80623): a mixed response where one
tool call completes validly while a sibling has zero argument bytes still
gets discarded whole via the shared partial-stream-stub path, and the
zero-byte trigger now has an end-to-end test through run_conversation's
retry loop, not just at the chat_completion_helpers unit level.
2026-08-07 18:07:19 +05:30
joaomarcos f734578033 fix(streaming): flag empty tool-call args on clean stream end (#80498)
When the stream closes right after a tool call's name arrives but
before any argument bytes are delivered, has_truncated_tool_args
was never set (the existing check required a non-empty, whitespace-
stripped arguments buffer). The call fell through to a normal "stop"
finish_reason, later coerced to "{}" at dispatch and executed
silently with no arguments and no retry.

Route this case through the same dropped-mid-tool-call stub/retry
path already used for partially-truncated JSON.
2026-08-07 18:07:19 +05:30
kshitij 9377c5a539 fix(redact): narrow control-split join guard to line-crossing spans
Post-merge review of aecb9ca89 found the join guard over-broad: skipping
the join whenever ANY fragment self-matches _PREFIX_RE reopened a leak
for non-newline splits — sk-<15 chars>ESC<25 chars> masked only the
self-matching head and left the 25-char tail in cleartext (fully masked
before the guard; main never masked this shape at all, so the merged
state was still >= main, but the salvage's own coverage regressed).

Skip the join only when the span crosses a line boundary (\n / \r) —
that is the shape where adjacent legitimate text gets swallowed
(ghp_<token>-then-'button [ref=e3]' annotation bug). ESC/zero-width
controls never legitimately separate a token from prose, so joining
there is safe and restores full-tail masking.

Both legs mutation-checked: reverting to the unconditional skip fails
the new tail-mask test; removing the guard fails the annotation test.
2026-08-07 17:50:03 +05:30
kshitij afb46fdab4 refactor(cron): polish registration partial-failure surfaces
Follow-up to the salvaged registration contract:
- share one _raise_if_cron_registration_error() helper for the two
  byte-identical dashboard 424 except-blocks (web_server + cron router,
  via the existing late() seam)
- add endpoint-level 424 coverage for /api/cron/blueprints/instantiate
  (previously only the sync worker was tested)
- give chat/CLI surfaces a human-facing user_message() (job name, no
  exception class name) and add a recovery hint (pause/resume or update
  re-registers via provider reconcile) to the model/REST message
- consolidate five inline provider test doubles into one ABC-subclassing
  make_cron_provider conftest factory; the web_server test double now
  subclasses CronScheduler so an ABC rename fails loudly
- narrow the wrapper facade to keyword-only (**kwargs) and route the
  tool's partial-failure return through tool_error()
2026-08-07 17:45:06 +05:30
Gille f346458f29 fix(cron): surface initial scheduler registration failures 2026-08-07 17:45:06 +05:30
kshitij 261aef5268 perf(cron): stat-stamp fast path for the shrink-merge; no caller-list mutation
Follow-up hardening on the salvaged #80687 shrink-merge guard, folding in
the best part of the competing #80703 (credit: @JoaoMarcos44):

- Stat-stamp fast path: load_jobs() inside a _jobs_lock() section records
  jobs.json's (mtime_ns, size, ino) BEFORE reading; the save-path merge
  and the post-stage verify skip their full read+parse when the stamp
  still matches. The healthy no-race save (every mark_job_run /
  claim_dispatch / heartbeat / advance_next_runs tick persist) now costs
  one stat() instead of up to two full JSON parses.
- Fail-safe stamp discipline: the stamp is captured pre-read (a sibling
  racing the load leaves it older than disk -> mismatch -> merge runs),
  includes st_ino (mkstemp+rename always allocates a new inode, so
  coarse-mtime filesystems cannot false-match), resets on section
  entry/exit, and is INVALIDATED - never refreshed - after any write in
  the section (a refresh would let a nested create_job be clobbered by
  an outer caller's stale payload; probe-verified both directions).
- _merge_unexpected_disk_jobs no longer mutates the caller's list in
  place - it returns a new list when anything was recovered, and logs the
  recovered ids.
- The tolerant read cascade (utf-8-sig + strict=False fallback) is
  factored into one shared _parse_jobs_file used by both load_jobs and
  _peek_jobs_unlocked, so future encoding/shape fixes land once. The
  peek's repair-free re-entrancy contract is now documented - a repairing
  read on the save path would recurse through _save_jobs_unlocked (the
  exact defect the stamp-reconcile approach in #80703 had).

4 new regression tests (fast path, no-mutation, corrupt-file save,
nested-create-vs-stale-outer-save), each verified to fail against the
implementation it guards.

Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
2026-08-07 17:29:49 +05:30
HexLab98 5511ec623b test(cron): cover jobs.json shrink-merge against concurrent creates
Regress the #80624 no_agent watchdog clobber: a stale empty save must
not wipe a concurrent create, while intentional remove/replace still work.
2026-08-07 17:29:49 +05:30
kshitij aecb9ca894 fix(redact): don't join across controls when a fragment already matches
CI slice 1/12 caught a regression in _mask_control_split_tokens: a
COMPLETE prefix token at end-of-line followed by ordinary text (browser
accessibility annotations: 'ghp_<tok>\nbutton [ref=e3]: Copy') was
joined across the newline into one stripped-copy match, and the mask
swallowed the adjacent line ('button' disappeared).

Join only when no fragment inside the span matches _PREFIX_RE on its
own — a self-matching fragment is already handled by the ordinary
prefix pass, so joining can only cause damage. All smuggling shapes
(ESC/ZWSP/newline splits with under-length fragments) still mask;
regression test added and mutation-checked (fails without the guard).
2026-08-07 17:01:23 +05:30
Soheil Fakour 8969ebac1c fix(secrets): redact command in process checkpoint file (#77484)
_write_checkpoint persisted s.command verbatim to ~/.hermes/processes.json.
Recovery only uses command for display/logging (the process is already
running; adoption re-validates PID + start time, never re-runs the
command), so masking is lossless.
2026-08-07 17:01:23 +05:30
Soheil Fakour e9d1551e65 fix(redact): strip control chars from mask_secret display (#55319, #55321)
A masked secret's visible head/tail could carry control bytes (newline,
NUL, DEL, C1 0x80-0x9F, zero-width) into config/status/dump output.
Strip every control incl. \n/\t (display differs from redact_sensitive_text,
which preserves \n/\t as line structure) before slicing; all-control values
return the configured empty fallback.

Consolidates the previously-closed #58079 approach (strip controls before
masking) - supersedes it.
2026-08-07 17:01:23 +05:30
Soheil Fakour 5444f6853b test(redact): harden new #77484 tests - assert fragments, opaque values (review) 2026-08-07 17:01:23 +05:30
Soheil Fakour 8563fe3435 fix(redact): close emission gaps - env suffix keys, control-char splits, process(list) (#77484) 2026-08-07 17:01:23 +05:30
kshitij 15d7103aa7 fix: harden .env-read detection — review follow-ups for #61352
- Import file_safety._BLOCKED_PROJECT_ENV_BASENAMES instead of copying
  it (comment-enforced parallel lists drift); lookup is now
  case-insensitive to match file_safety's .lower() semantics (cat .ENV
  on macOS/Windows case-insensitive filesystems reads the same secrets).
- Strip shell quotes plain split() leaves attached (cat ".env").
- Drop the dead _ENV_FILE_EXCLUDE_SUFFIXES logic (exact-basename
  membership already excludes templates) and the stray blank-line noise.
- Document the defense-in-depth limits (sudo/full-path/substitution
  readers) mirroring is_env_dump_command's precedent, and correct the
  docstring overclaim about name-independence.
- Annotate command as str | None (tests pass None).
2026-08-07 16:58:25 +05:30
Peter cf755f5c42 fix: redact .env terminal output via detection instead of known-env-var list
Terminal output from file-read commands (cat, head, tail, ...) uses
code_file=True, which skips the generic ENV-assignment redaction pass.
Reading a .env file through the terminal therefore leaked any key whose
value has no recognized vendor prefix (Mistral, Gemini AQ.*, tvly-dev-,
bu_, Spotify client secrets).

Detect file-read commands targeting .env-style basenames (mirroring
agent/file_safety's blocked list) and route them to code_file=False so
the existing ENV pass masks opaque values. Templates (.env.example,
.env.sample, ...) are excluded.

Salvaged from #61352 (145 commits of drift; conflict with the test-prune
wave resolved by NOT resurrecting pruned tests). Authored by @ShaoRou459.

Closes #61352
2026-08-07 16:58:25 +05:30
kshitij c18e19c3c7 fix(agent): make the send-path copy structural — close the write-through class
The api_messages build used a shallow msg.copy(), decoupling only
top-level fields. Every nested container (tool_calls entries and their
function dicts, multimodal content-part lists, reasoning_details) stayed
aliased to the persisted history, so ANY in-place transform on the send
copy silently rewrote the stored transcript.

Probed every send-path transform against that aliasing shape on main:

  content strip loop                       safe (top-level reassign)
  _canonicalize_api_tool_calls (repair)    LEAKED  <- #80616's fix
  _sanitize_messages_surrogates            LEAKED  (multimodal parts,
                                                    tc ids/args, reasoning)
  _sanitize_messages_non_ascii             LEAKED  (multimodal parts)
  _sanitize_api_messages                   safe
  _drop_thinking_only_and_merge_users      safe

The retry loop already believed the copies were independent - it
sanitizes messages AND api_messages separately (~L3555) - so the
aliasing was accidental everywhere.

Fix at the chokepoint: _clone_message_for_send clones every container
(dict/list) recursively while sharing immutable leaves, so every
downstream in-place transform - current and future - is safe by
construction. Cost is container-count, not string-bytes: 100KB argument
strings and base64 payloads are shared (measured ~0.5ms vs ~0.1ms per
1500-message build; noise next to one json round-trip). Same clone
applied to the prefill-message insert (same class, same pipeline).

The class-wide invariant test runs the full send-path transform
pipeline over an adversarial fixture (malformed args, surrogates,
non-ASCII, multimodal parts, reasoning fields) and asserts the history
stays byte-identical; an AST contract pins the build-site wiring so the
shallow copy can't quietly return. Both mutation-verified: reverting
the clone to shallow fails 4 isolation tests, unwiring the build site
fails the AST contract.

0xGr1mm's branch fix (previous commit) remains as defense in depth at
the exact site the #80498 incident hit; his regression tests and the
class-wide invariant give layered coverage.
2026-08-07 16:57:11 +05:30
Gr1mmJ4w e60ca1c6ca fix(agent): stop the send-path repair from rewriting persisted history
`_canonicalize_api_tool_calls` promises copy-on-write in its own docstring
— "the persisted history is untouched" — and the call site repeats it:
"Operates on api_messages (the API copy) so the original conversation
history in `messages` is untouched."

The canonicalize branch keeps that promise (`tc = {**tc, "function": {...}}`).
The repair branch does not:

    except Exception:
        tc["function"]["arguments"] = _repair_tool_call_arguments(...)

`api_messages` is built with `msg.copy()` — a SHALLOW per-message copy — so
every `tool_calls` entry is the same dict object the persisted history
holds. Assigning into `tc["function"]` therefore writes through to the
stored turn. The sibling loop two lines above only touches `am["content"]`,
one level deep, which is why the aliasing never showed up there.

On the unrepairable path `_repair_tool_call_arguments` returns "{}", so
that write replaces the model's real arguments with an empty object in the
transcript. A stream that dies mid `write_file` loses the file content it
had already streamed — the reported symptom in #80498, where a chapter
draft was silently reduced to `{}` and only a WARNING remained:

    Unrepairable tool_call arguments for write_file — replaced with empty
    object (was: {"content": "# 骨架-第25章\n> 承接...)

Mirror the canonicalize branch: build a new tool-call dict instead of
assigning into the shared one. The API copy still carries "{}" — the
repair's whole purpose is to never ship broken JSON — but the history keeps
what the model actually sent, so the transcript, session persistence and
any later retry still have it.

The in-place write was not an oversight in isolation: it predates the memo
refactor, which preserved it deliberately for byte-parity. The existing
`test_history_not_mutated` asserts exactly this invariant but restricts
itself to valid arguments, and its docstring records the gap — "(Malformed
args take the in-place repair path — pre-existing behavior)". That is why
a test file whose header already claims "the persisted history is never
mutated (copy-on-write preserved)" stayed green through the bug.

Four tests close it: history keeps the original bytes, the send copy is
still repaired, a broken call does not disturb its siblings, and repeated
sends stay lossless. On unpatched main three of them fail; the parity and
complexity tests are unaffected because the difference is only observable
when the history list is separate from the send copy — which is the shape
production uses.

Refs #80498

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 16:57:11 +05:30
kshitij 8b6dd27cdb test(auxiliary): update _resolve_auto patch to _resolve_auto_route
The PR renamed _resolve_auto to _resolve_auto_route (3-tuple return).
This test patches the resolver to mock the auto-detection chain, but was
still patching the old _resolve_auto name. The patch never fired, so the
test fell through to the real auto-detection (which finds no providers
in CI's hermetic env). Update patch target to _resolve_auto_route with
the 3-tuple return value.
2026-08-07 16:56:37 +05:30
Gille 293e67328c fix(agent): preserve auto-routed provider identity 2026-08-07 16:56:37 +05:30
kshitij 427584b768
Merge pull request #80933 from kshitijk4poor/fix/replace-messages-archive-siblings
fix(state): finish the #80216 bug class — archive-preserving rewrites at the ACP and TUI sibling sites
2026-08-07 15:45:38 +05:30
kshitij a82910c37b test: fold review findings — plain fixture, call-shaped probe guard, public-API row counting
- state_db fixture: drop the HERMES_HOME setenv + sys.modules purge
  (SessionDB takes an explicit path; tests/conftest.py already sandboxes
  HERMES_HOME; the purge risks split-class identity for other modules
  holding the old hermes_state reference) — matches the plain
  tests/hermes_state/ sibling fixture pattern.
- probe guard asserts on the CALL ('has_archived_messages(') instead of a
  local-variable name — a reintroduced probe under any rename now trips
  it (mutation-checked: renamed-probe reintroduction fails the guard;
  restored stack green).
- _archived_count uses the public get_messages(include_inactive=True)
  instead of poking db._lock/_conn.
2026-08-07 15:42:49 +05:30
kshitij 1e5b507440 fix(cron): move watchdog state under the request lock; fail closed on resolver errors
Follow-up hardening on top of the salvaged #80809 watchdog, porting the
locked state-machine design from #75301 (credit: @Zeraphim):

- All lifecycle transitions (stale/cancelled/done) now happen under
  request_client_lock. A user or monitor interrupt marks the request
  'cancelled' so a racing stale timer can no longer misclassify the kill
  as provider staleness and feed a false +1 into the #58962 cross-turn
  circuit breaker.
- 'done' is set under the lock on completion, so a late timer callback
  that lost the race to a successful response is inert instead of
  leaving a spurious streak=1 behind the reset.
- Registration race closed: if the budget expires while the client is
  still being constructed, _make_client aborts the freshly-registered
  client and fails the call with a retryable TimeoutError instead of
  opening a brand-new socket after the only watchdog already fired.
- _resolve_direct_stale_timeout now fails closed: a raising resolver
  propagates (same as the worker path) instead of being swallowed into
  an infinite budget that would silently disarm the watchdog and
  reinstate the very hang #80759 is about.

4 new regression tests, each verified to fail against the pre-fix
watchdog implementation.

Co-authored-by: Zeraphim <diamantejc87@gmail.com>
2026-08-07 15:32:37 +05:30
HexLab98 cb066a971b fix(cron): bound the inline non-streaming call with a stale watchdog
Cron turns and delegated children are routed onto direct_api_call, which
ran the request inline with no stale detector. The abort plumbing was
registered but nothing ever invoked it, so a provider that accepted the
request and then went silent — connection held open, zero bytes, no
error — hung the run until an external actor killed it, which also
orphaned the execution row. The httpx read timeout is not a usable bound
(1800s default, and this failure mode never trips it), and the job-level
inactivity monitor was observed not to fire.

Arm a watchdog timer on the same budget the interrupt worker's poll loop
uses, so these turns get exactly the patience every other non-streaming
request already gets. On expiry it only aborts the in-flight sockets
through the already-registered hook — it never issues a request, so the
inline / no-worker property that fixes the nested-pool deadlock is
preserved — bumps the cross-turn stale circuit breaker, and surfaces a
retryable TimeoutError so the outer loop reconnects on a fresh pool.

Fixes #80759
2026-08-07 15:32:37 +05:30
kshitij ee6d79648a fix(state): finish the #80216 bug class — archive-preserving rewrites at the two remaining sibling sites
#80216 fixed /retry (and a follow-up fixed yuanbao recall) destroying
soft-archived active=0/compacted=1 in-place-compaction rows via the
destructive replace_messages default. Two sibling sites still carried the
same class:

- acp_adapter/session.py _persist (non-owned-agent branch): probed
  has_archived_messages and FAILED OPEN into the destructive full replace
  on any probe error; the probe can also race a concurrent
  archive_and_compact. Now passes active_only=True unconditionally — on a
  fresh create/fork every row is active=1 so behavior is identical, and
  the probe (its only production caller) is deleted.
- tui_gateway/methods_prompt.py edit/regenerate truncation: bare
  replace_messages() deleted the archived transcript of a compacted
  session on every edit/regenerate. Now active_only=True.

hermes_state.has_archived_messages docstring updated (probe is now
test/diagnostic-only). Test stubs in test_tui_gateway_server.py accept the
new kwarg. New regression tests: real-SQLite archive-survival for both
write shapes, fresh-session equivalence (the claim the unconditional
switch rests on), and source-level guards pinning that neither site
re-grows the fail-open probe (both mutation-checked: revert either fix and
its guard fails).
2026-08-07 14:42:32 +05:30
kshitij 20e01f935b fix(voice): early-exit sliding window on match, clear barge phase in finally, fix test helper
- is_tts_echo sliding window: return True immediately when ratio >= threshold
  instead of scanning all remaining windows. Common echo case drops from
  ~4s to <1ms for long spoken text (found by /simplify-code efficiency review).
- Clear _voice_barge_phase in _voice_submit_barge_utterance finally block
  alongside _voice_barge_capture, preventing stale phase from a previous
  trip affecting a future call.
- Add _voice_last_tts_text and _voice_barge_phase to _make_voice_cli test
  helper so it matches __init__ state.
2026-08-07 14:38:02 +05:30
chelsealong 979bf0cc4a fix(voice): require minimum evidence for fragment echo matching, use char windows
Reviewer feedback on the fragment-echo fallback added in 24730b10c:
- A short playback-phase transcript (e.g. a genuine one-word "yes")
  could trivially match a same-length window of a longer spoken reply
  at ratio 1.0 and be wrongly dropped as a self-capture. The fallback
  now requires the transcript to be at least
  MIN_FRAGMENT_LENGTH_FOR_ECHO characters before it runs.
- The fallback split on whitespace, so it never engaged for
  no-whitespace languages (both transcript and spoken text collapse to
  a single "word"). Switched the sliding window to be character-based
  instead of word-based, matching the function's tokenization-independent
  contract.

Adds regression tests for both cases.
2026-08-07 14:38:02 +05:30
chelsealong b7bff6f2d7 fix(voice): catch short echoed fragments of longer multi-sentence TTS replies
is_tts_echo() compared the captured transcript against the *entire*
spoken text with a whole-string similarity ratio, which only scores high
when the two strings are close in length. But a playback-phase barge
capture is cut immediately when the trigger fires and only spans the
pre-roll buffer plus time-to-silence, so a genuine self-capture is
typically a short fragment of a longer reply, not a near-verbatim repeat
of the whole thing -- for any response longer than a clause, the
length-diluted ratio fell below threshold and the echo sailed through
ungated.

When the whole-string check misses, also slide a window sized to the
transcript's word count across the spoken text and compare against each
window, so a short fragment echoed from within a much longer
multi-sentence reply is still caught. Add regression tests for a short
fragment matched at the start and in the middle of a longer reply.
2026-08-07 14:38:02 +05:30
chelsealong d4a753ea42 fix(voice): drop playback-phase barge transcripts that echo Hermes' own TTS
The full-duplex barge-in listener added in 5081551f0 stays active during
TTS playback with no acoustic echo cancellation. On some speaker/mic
combinations, TTS bleed alone crosses the barge threshold, gets
transcribed, and is queued as the next user turn -- whose reply is then
spoken, captured, and queued again, producing an unbounded TTS -> STT ->
TTS feedback loop (#75780).

Add a fail-closed transcript-level guard: when a barge trip happens during
the playback phase, compare the captured transcript against the TTS text
Hermes just spoke (tools/voice_mode.is_tts_echo, a language-agnostic
character-level similarity ratio). A close match is dropped instead of
queued, and the mic is handed back to the normal continuous-listening
loop. Generation-phase trips (no TTS playing, so no bleed is possible)
are unaffected.
2026-08-07 14:38:02 +05:30
kshitij e8511efe75 fix(matrix): propagate sender + reply context to media MessageEvents too
_handle_media_message had the same gap as _handle_text_message: it didn't
set user_id/user_name or any reply_to_* fields on MessageEvent. A user
replying to a photo/video/audio in Matrix got null sender metadata and
null reply context — the exact bug PR #80293 fixes for text messages.

Mirrors the text handler's reply fallback parsing + sender propagation
into the media handler, and adds a test covering a media reply event.

Also fixes test env mutation: _make_adapter now accepts monkeypatch
and uses monkeypatch.setenv() instead of bare os.environ assignment,
preventing env var leakage across tests in the same xdist worker.
2026-08-07 14:34:55 +05:30
WintleChoung e245a98781 fix(matrix): propagate sender MXID + reply context to MessageEvent
The Matrix adapter built MessageEvent from inbound room events but dropped
the sender's MXID and display name on the event itself -- only 'source'
carried them. Other adapters (signal/slack/telegram/discord/mattermost/irc)
have the same gap; this PR fixes matrix and adds the supporting
top-level MessageEvent fields so the rest can follow.

Downstream effects for matrix specifically:
  - gateway prompt assembly can now read event.user_name (or source)
    without having to dig into source per platform
  - reply context (reply_to_text / reply_to_author_id /
    reply_to_author_name) is parsed from the inline > <@user:server> ...
    Matrix fallback format before stripping, instead of discarded
  - the gateway's existing [Replying to: "..."] renderer can now show
    who the user was replying to (was always anonymous for matrix)

MessageEvent gains two optional top-level fields (user_id, user_name,
both default None) so non-IM producers (cron/webhook/autonomous) remain
unaffected. Source still carries the same values for callers that
already read from there.

Tests cover:
  - non-reply message carries sender user_id/user_name on MessageEvent
  - different senders (alice, bob) both propagate
  - reply message carries reply_to_message_id + reply_to_text +
    reply_to_author_id + reply_to_author_name, parsed from the
    > <@carol:example.org> original question\n\nactual reply shape
  - non-reply message does NOT spuriously set reply_to_* fields

Sibling matrix tests (148 across test_matrix*.py) remain green.

Authored by WintleChoung <cwt@users.noreply.github.com>
Salvaged from PR #80293.
2026-08-07 14:34:55 +05:30
kshitij 03beb662e8 fix: cover the partial multi-call batch in the in-flight exemption
Widen #79293's trailing-in-flight guard from 'last message is assistant'
to 'last non-tool message is assistant': a multi-call batch snapshotted
between the executor's per-result appends looks like
[..., assistant(c1,c2,c3), tool(c1)] — c2/c3 are pending, not orphaned,
but the tail-only guard missed that shape and stripped them (same silent
result loss as the original bug, via concurrent /compress or the gateway
hygiene pass).

Preserving is safe on both shapes: the pre-API chokepoint
(sanitize_api_messages step 2) injects stub results for any call that
genuinely never gets an answer, while stripping a live call silently
loses its late result.

test_sanitizer_strips_orphaned_keeps_valid's mixed valid/orphan shape
moves mid-list — at the tail it is byte-identical to a live partial
batch and the sanitizer now correctly presumes in-flight there.

New regression test fails without the walk-back (c2/c3 stripped),
passes with it.
2026-08-07 14:13:32 +05:30
Shotflame 788b8ab497 fix(compress): preserve in-flight tool chain across context compression (#79278)
Tool_executor.py appends role=tool results AFTER running each call. When
context compression fires mid-chain, the trailing assistant(tool_calls)
message is a pending request whose result has not yet been appended.
_sanitize_tool_pairs previously stripped it as an 'orphan', so when the
executor later appended the real result, repair_message_sequence dropped
it as unmatched and the completed side effect (and final synthesis) was
lost. Preserve the trailing in-flight call verbatim; only genuinely
orphaned calls in the discarded region are stripped.

Adds regression tests: three unit tests for _sanitize_tool_pairs plus an
end-to-end test reproducing compression -> side-effect completion ->
result-returned flow. Confirmed failing on pre-fix code, passing with
the fix.
2026-08-07 14:13:32 +05:30
kshitij 416d2a0157 fix(gateway): re-signal interrupts when work is still live at settle-window exit
Review follow-up for the salvaged #79881/#63963 stack: the shutdown
interrupt fires exactly once, but work can materialize AFTER that one
shot on BOTH sibling paths:

- a /v1/runs task admitted before the drain populates
  _active_run_agents only once _create_agent returns
  (queued-before-agent window);
- a _running_agents entry claimed as _AGENT_PENDING_SENTINEL is
  promoted to the real agent by track_agent() on its own schedule,
  after the one-shot walk skipped the sentinel.

Either way the settle loop waited on work nothing signaled, and the
turn went straight to the post-interrupt tool-subprocess kill — the
exact amputation the fix exists to avoid, in a rarer window.

If any work is still live when the settle loop exits, re-invoke
_interrupt_running_agents (which already skips sentinels and folds in
the API-server helper) so late-materializing agents on either path get
the cooperative interrupt. Regression test drives the real stop() path
with an accelerated loop clock and asserts exactly two interrupt
signals.
2026-08-07 14:11:18 +05:30
briandevans d9ddfb23d5 fix(gateway): interrupt every in-flight API turn on shutdown, not just /v1/runs
The shutdown drain ACCOUNTS for API-server work but never INTERRUPTS it.
`_drain_active_agents()` folds `_active_api_run_count()` into both its wait
loop and its `timed_out` verdict, while `_interrupt_running_agents()` iterates
`self._running_agents` only -- a dict no API turn ever enters, because the
API server owns its own agent lifecycle. `gateway/run.py` states the gap
against itself: "API-server / desk sessions have the same structural gap
(#63529)."

The user-visible result is that every gateway restart with a live API or
desktop turn burns the full drain timeout and then runs
`_kill_tool_subprocesses("post-interrupt")`, which amputates the turn's tool
subprocesses with no cooperative interrupt and no resume marker.

There are seven API agent-entry points. Six funnel through `_run_agent()`
(both session-chat routes, and `/v1/chat/completions` + `/v1/responses` in
streaming and non-streaming form) and are counted by `_inflight_agent_runs`;
the seventh, `/v1/runs`, runs its own lifecycle and is counted through
`_active_run_tasks`. None of the six has a run_id, so the run_id-keyed
`_active_run_agents` cannot reach them, and only two pass `agent_ref` -- which
lands in a caller-local list, not a registry.

So register once at the single unconditional creation site inside
`_run_agent`, beside the existing `_publish_turn_process_ownership()` call,
and unregister in the same `finally` that already clears it. That one
symmetric pair covers all six callers. The registry is adapter-owned and
keyed by object identity, kept separate from `_active_run_agents` because
that dict is run_id-keyed and scoped to the public `/v1/runs` stop API.

`interrupt_active_runs()` then walks both registries, deduped by identity, so
the interrupt set matches the set the drain waits on. The settle window after
the interrupt now polls API work as well: the interrupt is cooperative, and
without this the window closes the instant `_running_agents` is empty -- which
it always is for API turns -- and the tool kill lands on a turn that was asked
to stop microseconds earlier.
2026-08-07 14:11:18 +05:30
dsad 51fa7db469 fix(gateway): interrupt api server runs on shutdown timeout 2026-08-07 14:11:18 +05:30
poisdahl 30c1421acf fix(gateway): make retry archive preservation fail-safe 2026-08-07 13:56:18 +05:30
Adolanium 56fbac6b38 fix(gateway): preserve archived compaction history on /retry
/retry truncates the live transcript to before the last user message
and persists it via SessionStore.rewrite_transcript, which calls
replace_messages() with the default active_only=False. That DELETEs
every row for the session, including the soft-archived
active=0/compacted=1 rows that in-place compaction keeps on disk
(#38763), so any /retry after a compaction permanently wiped the
archived history. #57803 named this call site as a residual gap after
its global-default approach was rejected; the TUI sibling was fixed
in #80195.

The handler now probes has_archived_messages() (new SessionStore
wrapper, auto-exposed through AsyncSessionStore) and passes
active_only=True when archives exist, so only the live rows are
replaced. rewrite_transcript gains an active_only parameter that
defaults to False, keeping the destructive semantics yuanbao recall
redaction depends on. Also corrects the rewrite_transcript docstring,
which still listed /undo as a caller even though /undo soft-archives
via rewind_session.

The regression test drives _handle_retry_command against a real
SessionStore and SessionDB seeded with archived compaction rows and
asserts the archives survive.
2026-08-07 13:56:18 +05:30
kshitij 99237a4444 refactor: derive teams install hint via feature_install_command(venv_pip=True)
Fold the remaining simplify-code reuse finding: teams' _install_hint()
duplicated lazy_deps' spec-fetch + quote + join (feature_install_command
already builds pip commands from LAZY_DEPS). Add a venv_pip=True variant
to feature_install_command — sys.executable -m pip targeting, correct in
every install layout and immune to PEP 668 — and shrink the teams helper
to a one-line call.  Also gives matrix and the other platforms a shared
derived hint to adopt later.  New test mutation-checked (fails when
venv_pip returns the uv form).
2026-08-07 13:28:43 +05:30
kshitij f5784617e8 refactor: fold simplify-code review findings
- matrix/dingtalk: extract deps-only installers (ensure_matrix_deps,
  ensure_dingtalk_deps) and register THOSE as ensure_deps_fn — the prior
  check_*_requirements combined credential env checks with the install,
  so a platform configured via PlatformConfig.extra (which is_connected
  accepts) would pass enablement, reach create_adapter(), and have the
  'installer' veto on env-var grounds before installing anything —
  re-creating the #79812 deadlock for extra-configured setups.  The
  combined deps+credentials functions remain for setup/status callers.
- matrix/feishu passive probes: use the existing lazy_deps.is_available()
  instead of hand-rolling 'not feature_missing(...)' (reuse finding).
- teams: module docstring no longer recommends bare system pip (the
  PEP 668 trap purged everywhere else); docs troubleshooting row updated
  to match the new hint text.
- wecom_callback: drop dead 'global ET, DEFUSEDXML_AVAILABLE'
  (ensure_and_bind mutates the module dict directly; nothing assigns).
- tests: parametrized wiring contract for all 8 lazy-installable
  platforms — ensure_deps_fn present and distinct from check_fn
  (behavior contract, not identity snapshot, so renames don't churn it).
2026-08-07 13:28:43 +05:30
kshitij a658dfe509 fix: address self-review findings on the check_fn/ensure_deps_fn split
- gateway/config.py: rewrite the stale enablement-pass header comment that
  still described check_fn as 'the single source of truth for are-my-env-
  vars-set' / 'lazy-installs it' — both false under the new contract.
- teams: check_requirements docstring wrongly claimed credential checks
  (body checks only SDK/aiohttp presence); derive install_hint from the
  canonical LAZY_DEPS pins + sys.executable instead of hardcoding
  '~/.hermes/hermes-agent/venv/bin/pip' and version pins (wrong under
  HERMES_HOME overrides / profile installs; pins go stale on CVE bumps);
  connect() fatal-error hints now point at the venv pip instead of bare
  system pip (the PEP 668 trap the docs warn about).
- teams docs: drop exact version pins from the two manual-install commands
  (LAZY_DEPS is the source of truth; unpinned installs still work and the
  text can't go stale).
- hermes_cli/status.py: per-entry exception guard around check_fn so one
  raising probe can't abort the listing of all remaining plugin platforms
  (aligns with the other three call sites).
- tests: rename test_register_check_fn_is_active_lazy_installer ->
  test_register_splits_passive_probe_from_active_installer (name said the
  opposite of what it verifies).
2026-08-07 13:28:43 +05:30
kshitij 0d32607c62 fix(gateway): split check_fn (passive probe) from ensure_deps_fn (active installer)
PlatformEntry.check_fn served three contradictory roles: adapter-creation
gate, config auto-enablement gate, and status display. Plugins had to pick
one function for all three:

- Active installer as check_fn (discord/slack/telegram/matrix/dingtalk/
  feishu): every status display could pip-install SDKs as a side effect
  (the desktop 94% boot-loop class).
- Passive probe as check_fn (teams, wecom_callback): create_adapter()
  returned None before connect() could lazy-install, so the SDK never
  installed (#79812 deadlock; wecom_callback's platform.wecom_callback
  LAZY_DEPS entry was dead code).

The split makes both call sites correct by construction:

- check_fn is now contractually PASSIVE (probe only, never installs).
- New optional PlatformEntry.ensure_deps_fn is the ACTIVE installer;
  create_adapter() runs it exactly when check_fn is False — the platform
  is enabled+configured and the gateway is about to connect it.
- Config enablement keeps a configured platform whose deps are missing
  but installable; the install itself is deferred to create_adapter().
- Status surfaces (_platform_status, hermes status) read only the
  passive probe and can never trigger pip.

Migrated all lazy-installable platform plugins to the split; platforms
with no optional deps (irc/ntfy/buzz/simplex/line/a2a/...) are unchanged
— no ensure_deps_fn means a False check_fn stays a hard block.
wecom_callback gains a working installer for the first time.

Builds on @xxxigm's #79812 (both commits cherry-picked with authorship
preserved), reworking the check_fn swap into the two-field split so the
Teams fix doesn't reintroduce install-on-status.
2026-08-07 13:28:43 +05:30
xxxigm 98408f713b fix(teams): lazy-install SDK via registry check_fn
Platform registry create_adapter() gates on check_fn before the adapter
exists, so wiring the passive probe permanently blocked connect() and
the existing check_teams_requirements() lazy-install never ran.
2026-08-07 13:28:43 +05:30
kshitij a0801b878a fix: bind continuation-marker exclusions to the queried parent (fail-open fix)
Adversarial review of the salvaged recovery found a reachable fail-open:
compression continuations inherit the rotated agent's model_config
verbatim (publish_compression_child callers pass
agent._session_init_model_config), so a delegate subagent's continuation
carries _delegate_from=<the delegate's own parent>. The marker-PRESENCE
filters in reopen_orphaned_compression_session and
find_live_compression_child misclassified such a REAL continuation as a
delegate child:

- reopen: parent 'orphaned' -> reopened while a live continuation exists
  -> two live heads in one lineage (verified with a live repro)
- find_live: adoption misses the continuation (fail-closed, masked the
  fork pre-PR; the PR made it active)

Fix: markers only disqualify a child when they point at the queried
parent (shared _NON_CONTINUATION_CHILD_FILTER_SQL fragment, also
resolving the duplicated-SQL drift risk flagged by the reuse reviewer).
Both directions regression-tested: reopen fails closed on an
inherited-marker continuation; find_live adopts it.

Also from review: reopen-failure log raised debug->warning (the failure
hard-fails the turn moments later), commit-semantics hardening comment
on the lease DELETE path, blank-line nit.

The three read-only projection walks (get_compression_tip,
list_sessions_rich chain, resume walk) share the marker-presence shape
but fail closed (skip a continuation -> resume shows the parent), and
the fixed adoption path self-heals that case at turn start; left as-is.
2026-08-07 13:24:56 +05:30
izumi0uu 95a7058e4b fix(sessions): fence expired orphan recovery leases 2026-08-07 13:24:56 +05:30
izumi0uu 988f2baaf8 fix(sessions): recover compression parents without continuations 2026-08-07 13:24:56 +05:30
liuhao1024 66c60f81b6 fix(cron): thread per-run prompt through cronjob(action='run') (#57331)
Salvaged from PR #57342 by @liuhao1024 (with the injection-scan half
from PR #57360 by @ghedeselmabot): cronjob(action='run', prompt=...)
silently discarded the prompt argument — per-run context never
reached the spawned cron session.

The prompt is now threaded as extra_prompt through the whole chain
(cronjob run action → _try_dispatch_background_run/_execute_job_now →
_run_claimed_job → run_one_job → run_job → _build_job_prompt) and
appended to the stored prompt under a '## Run Context' header for
that single fire only — never persisted to the job definition. It
passes the same strict _scan_cron_prompt injection scan as stored
prompts before firing, and works identically on the background and
sync fallback paths.

Test fakes across tests/cron/ updated to accept the new kwargs
(sibling-test blast radius from the signature change).

Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
2026-08-06 23:14:55 -07:00
liuhao1024 fa96419993 test(cron): add _build_job_prompt extra_prompt regression tests
Pins the scheduler-boundary contract: extra_prompt is appended under
'## Run Context', does not mutate job['prompt'], and the header is
absent when extra_prompt is omitted.

Addresses review feedback from harjothkhara on PR #57342.
2026-08-06 23:14:55 -07:00
Fly 7a5fe00244 fix(cron): deliver manual runs on gateway loop 2026-08-06 23:14:55 -07:00
Teknium 3671c9f188 fix: share in-flight cron dedupe between ticker and manual runs
Salvaged from PR #53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.

Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, #60432) and mark_running_jobs_interrupted,
which previously could not see them.

The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.

Co-authored-by: izumi0uu <izumi0uu@gmail.com>
2026-08-06 22:19:36 -07:00
Teknium 7ab42dda60 fix: dispatch cronjob(action='run') to the background like delegate_task
A manual cronjob run executed the job synchronously on the calling
agent's tool thread. A cron job is a full agent run that routinely
takes minutes to hours, so the parent turn sat inside ONE tool call
the whole time: uninterruptible (the interrupt flag is only checked
between loop iterations) and serial (a batch of manual runs executed
one by one). A Telegram session that kicked off dozens of new jobs
'right now' was wedged for hours ignoring every interrupt.

action='run' now rides the async-delegation rail delegate_task
background mode uses: the at-most-once claim is taken synchronously
(so paused/missing/already-firing jobs still report immediately),
the run executes on the shared daemon executor, the tool returns at
once with a delegation handle, and the job's outcome re-enters the
conversation as a type='async_delegation' completion event through
the existing completion-queue drains (CLI + gateway) — preserving
message-role alternation and the prompt cache.

Sync fallbacks preserved:
- no routable session (direct Python callers, hermes cron run)
- async delivery unsupported (hermes -z, cron child sessions,
  Kanban workers, stateless HTTP)
- dispatch pool at capacity (claim already taken — runs inline
  rather than stranding it)

The completion block reports ok/failure, delivery target, next
scheduled run, and an excerpt of the job's saved output.
2026-08-06 22:19:36 -07:00
Teknium 32e7fb07a0 feat(/learn): expansive knowledge-base skills for books and large corpora
Inspired by virgiliojr94/book-to-skill (MIT): /learn now picks the skill
shape by the source. Workflows and small sources still get one tight
SKILL.md; books, paper stacks, specs, and large doc corpora get a
knowledge-base layout — a lean always-loaded SKILL.md index plus one
distilled file per chapter/topic under references/, loaded on demand via
skill_view so query cost stays proportional to the answer.

- agent/learn_prompt.py: new _KNOWLEDGE_SKILL_STANDARDS block (index +
  per-chapter references/, structure-not-summary distillation, never
  reproduce source passages, fold-in instead of duplicating) and a
  _SOURCE_HYGIENE block pinning extracted source text as data and
  dropping invisible/bidi Unicode (Trojan Source class). Clarified that
  the ~200-line cap and hub-skill ban apply to SKILL.md itself, not a
  knowledge skill's own references/ files.
- tests: contracts for the knowledge-base layout, the three embedded
  standards blocks, and the source-hygiene coverage.
- docs: skills.md documents the knowledge-base shape.
2026-08-06 22:14:52 -07:00
Jeffrey Quesnelle db407c8c93
Merge pull request #80797 from NousResearch/fix/sidebar-stale-schema-probe
fix(desktop): derive the stale-schema read probe from SCHEMA_SQL
2026-08-07 00:56:48 -04:00
emozilla bdee48928f fix(dashboard): derive the stale-schema read probe from SCHEMA_SQL
After `hermes update`, the desktop sidebar showed "No sessions yet" until
the user's first message. #72424 added sessions.last_activity_at, which
list_sessions_rich now selects — but column adds only land through
_reconcile_columns() in the writable _init_schema, and read-only opens
skip that by design. Every sidebar read path opens state.db read-only, so
each poll raised "no such column: s.last_activity_at" until the first
prompt's lazy session-row persist forced a writable open and reconciled.

A heal for exactly this class already existed (_open_session_db_for_profile
probes the read-only handle and does a one-time writable reopen on
staleness), but its probe was a hand-written four-column list that never
learned last_activity_at — it went stale three days after shipping. And the
batched sidebar route (/api/profiles/sessions/sidebar) bypassed the helper
entirely, swallowing per-profile failures into an errors array the desktop
never surfaces, so the incident produced an empty sidebar with clean logs.

The fix removes the maintenance burden instead of paying it once more:

- hermes_state_schema.schema_read_probe_statements() derives one
  `SELECT <every declared column> FROM <table> LIMIT 0` per table from
  SCHEMA_SQL via the existing _parse_schema_columns() — the same source of
  truth the writable reconciler diffs against, so any future ADD COLUMN is
  probed with no list to update. Column references are table-qualified:
  an unqualified double-quoted identifier that fails to resolve silently
  degrades to a string literal (SQLite's double-quoted-string misfeature)
  and would make the probe pass on exactly the store it exists to catch.

- web_server splits the heal into a path-level _open_session_db_at_path
  (semantics unchanged) so the cross-profile session routes can share it;
  both profiles.py loops and _count_status_active_sessions (the remaining
  raw read-only sibling) now open through it. The heal stays a helper
  rather than a SessionDB classmethod on purpose: escalation-to-writable
  must remain an explicit caller decision — update_cmd.py opens read-only
  mid-update and must never write.

- Exhaustion guard: if the writable heal SUCCEEDS and the re-probe still
  fails (a schema problem ADD COLUMN cannot express), the store is marked
  exhausted — warn once, skip the probe, serve reads probe-less — instead
  of re-running the full writable init on every poll against a possibly
  live DB. A FAILED writable open (transient lock) is deliberately not
  recorded, so the next poll retries the heal.

- The per-profile swallow sites in profiles.py now also log a deduplicated
  warning, so a persistent read failure is loud in errors.log even though
  the response errors array stays invisible to the sidebar.

Tests: probe/SCHEMA_SQL coverage invariants (tests/test_schema_read_probe.py),
last_activity_at added to the /api/sessions heal parametrize, a sidebar-route
heal test reproducing the shipped symptom (errors == [] and the session
returned against a store missing the column), and an exhaustion test pinning
exactly one writable open. The sidebar and last_activity_at tests fail on
main.
2026-08-07 00:41:58 -04:00
brooklyn! 55505be152
Merge pull request #80770 from NousResearch/bb/desktop-session-integrity
fix: preserve session history when a turn crashes
2026-08-06 22:12:22 -06:00
Brooklyn Nicholson fc05247be8 fix: preserve session history when a turn crashes 2026-08-06 23:08:23 -05:00
brooklyn! 623d5c93e0
Merge pull request #80736 from NousResearch/bb/reasoning-summary-blocks
Reasoning steps read as separate blocks again instead of one glued paragraph
2026-08-06 22:02:13 -06:00
Brooklyn Nicholson 6bb630ef78 fix(codex): split reasoning summary parts on summary_index
The native Responses stream does carry summary_index, so the part boundary is
structured data here rather than something to infer. Break on a change of
index, and leave streams that send no index (plain reasoning_text) untouched.
2026-08-06 22:02:46 -05:00
Brooklyn Nicholson 0f83661808 fix(reasoning): keep gpt-5.x summary parts as separate blocks on the chat wire
Reasoning-summary models emit one reasoning_content delta per completed
summary part, each a self-contained bold heading. The Responses API delimits
those parts with summary_index; the OpenAI chat wire carries no such field —
verified live against Nous Portal, whose reasoning chunks contain nothing but
delta.reasoning_content — so concatenating them glued every part into one
unspaced, half-bold paragraph.

Re-derive the boundary from the signal the wire does carry: a delta opening a
closed bold heading against a mid-line tail. This matches Hermes own Responses
adapter, which already joins its summary parts with a blank line.
2026-08-06 22:02:37 -05:00
brooklyn! 4b0238578e
Merge pull request #80711 from NousResearch/bb/sidebar-pin-sort
Pin as many sessions as you want, and they stay where you put them
2026-08-06 20:33:14 -06:00
Brooklyn Nicholson cef7d1a1e1 fix(api): persist session pins instead of 400ing them
PATCH /api/sessions/{id} only accepted title and end_reason, so the
`pinned` flag the desktop sends was rejected as an unsupported field —
and the client swallows that error. Pins lived in one app's localStorage
and never reached state.db, which also meant the server-side auto-archive
sweep was free to hide the chats a pin exists to keep.

Accept pinned and archived as booleans, route them to the SessionDB
setters that already existed, and include both in the serialized session
so clients can reconcile against server truth.
2026-08-06 21:10:54 -05:00
rob-maron 226b095a59
Fireworks user agent (#80422) 2026-08-07 01:49:57 +00:00
Brooklyn Nicholson 7ad9ace2cc fix(agent): the desktop's tools reach it on remote and cloud backends too
The pane, in-app browser, and reaction tools were gated on HERMES_DESKTOP=1 —
an env var set only on backends Electron spawns itself (local and SSH). A
desktop client connected to a plain URL gateway or Hermes Cloud lost all six:
they were stripped from the schema before the model saw them, on the same
backend whose platform hint was telling it "you are chatting inside the Hermes
desktop app". open_preview, read_preview, read_terminal, close_terminal,
focus_pane, and react_to_message were all silently absent.

The client is not the host. Capability now resolves from the session's own
source, which session.create already carries:

- The six tools move into a `desktop_ui` toolset, off _HERMES_CORE_TOOLS so no
  other platform pays their schema.
- _gui_surface_toolsets(platform) folds `desktop_ui` (and the existing
  `project` tools) into the GUI gateway's resolution when the session's
  platform is the desktop app — the same answer on every topology.
- check_fn drops the env probe. It kept the one thing that is genuinely a
  per-process/user fact: react_to_message's display.message_reactions opt-in,
  which the desktop mirrors onto whichever gateway it is connected to.

react_to_message was doubly broken: it read that toggle behind the env gate, so
even a local-backend user's Settings toggle could not reach a remote session.

The embedded terminal pane keeps working correctly the other way round: it runs
`hermes --tui` against a desktop-spawned backend, and a tui-sourced session
gets no GUI tools even though HERMES_DESKTOP=1 is set on that process.
2026-08-06 19:35:47 -06:00
Rob Hilgefort 65b7151dbd fix(launchd): require a supervised PID to call a reload successful
The reload retry loop treated `launchctl list <label>` exit 0 as success,
but exit 0 also covers a registered-but-not-running definition (macOS 26+
`state = not running`) — the same trap _probe_launchd_service_running
already guards against. Require a PID so success means launchd is
supervising a live process, in both the Python loop and the shell helper.

Verified against live launchd: a RunAtLoad=false job reports exit 0 with
no PID, which the old check accepted and the new one rejects.

Note this is NOT what distinguishes a draining instance — measured, the
label deregisters within ~1s of bootout while the old process drains on.
Waiting for the old PID to exit is what covers that.
2026-08-07 06:55:27 +05:30
Rob Hilgefort a1e4c905f5 fix(launchd): stop stranding gateway label on plist reload
Reload chose the in-process bootout/bootstrap path based on POSIX
ancestry, but bootout tears down the job's process coalition, and
coalition membership is inherited at spawn and survives reparenting.
A gateway-spawned process reparented to PID 1 is no longer an ancestor
yet still dies with the coalition, so the retry loop was killed
mid-bootstrap and nothing re-registered the label (KeepAlive can't
revive a job launchd no longer knows about).

- always prefer the detached transient-job helper; it's also correct
  when genuinely outside the coalition, just asynchronous
- wait for the old gateway PID to exit before bootstrapping; bootout
  only sends SIGTERM and every bootstrap during the drain fails EIO
- fall through to the in-process path when the helper can't spawn
  instead of leaving the plist rewritten but never reloaded
2026-08-07 06:55:27 +05:30
SmokeDev 9d4ef04ed0 fix(delegation): bind steering to session generation 2026-08-06 09:40:27 -07:00
SmokeDev a94ebf5f5e fix(delegation): harden steer lifecycle ownership 2026-08-06 09:40:27 -07:00
SmokeDev 60e1f7517c fix(delegation): surface a child's undelivered steer instead of dropping it
The turn finalizer already hands back steer text that queued after the
final tool batch — result["pending_steer"], with the comment "hand it
back to the caller so it can be delivered as the next user turn instead
of being silently lost." Every interactive surface honors that contract
(cli.py, gateway/run.py, tui_gateway/server.py all requeue it). The
delegation layer doesn't: _run_single_child never reads it, so a steer
queued into a delegated child that finishes first vanishes with no trace
in the completion entry. There is also no sanctioned sender: the registry
has interrupt_subagent() but no redirection-side mirror, and session.steer
cannot reach children (lazy watch sessions have agent=None, so it 4010s).

Complete the contract for delegated children — both halves:

- steer_subagent(subagent_id, text): redirection-side mirror of
  interrupt_subagent(). Resolves the live child in _active_subagents and
  queues text via AIAgent.steer(). True means queued, not delivered.
- missed_steer retention: when the child's result carries pending_steer,
  _run_single_child names it on the completion entry (missed_steer field
  plus a summary note) so the parent can re-issue the guidance instead of
  trusting it landed. This is what makes adding a sender safe: without it
  the finish-before-drain race silently loses the text — the exact loss
  the finalizer contract exists to prevent.
- subagent.steer gateway RPC beside subagent.interrupt so programmatic
  hosts (dashboard, voice layers, ACP bridges) get an in-tree caller;
  catalogued in programmatic-integration.md.
- docs: "Steering a Running Subagent" section in delegation.md covering
  the queued-vs-delivered semantics.

Tests: registry-level steer coverage (delivery, unknown id, empty text,
dead record, raising agent), the finish-before-drain race retaining
missed_steer, and the RPC contract (4000/4002 validation, queued and
rejected envelopes).
2026-08-06 09:40:27 -07:00
Teknium 19fc9c103e fix(tests): fail hard when pytest resolves the production state.db (live-DB isolation guard)
Forensics on a live developer machine found pytest fixture rows inside the
REAL ~/.hermes/state.db — sessions with chat_id 'chat-1', '123', 'wx-chat',
and gateway_routing rows whose scope was literally under /tmp/pytest-of-*/.
A pytest-spawned process also opened the live DB and flipped its journal
mode (journal_mode=DELETE fallback on SQLite 3.50.4) under the WAL-mode
gateway writer, destroying committed transcripts ("Persisted transcript
lagged live cached history ... possible FTS write corruption", 15+
occurrences). The existing live-system guard covers kill primitives but not
the SessionDB/SessionStore write paths.

Root cause (leak vector): the session-level HERMES_HOME sandbox in
tests/conftest.py only created a tempdir when HERMES_HOME was UNSET. On a
machine where the shell (e.g. gateway-launched, or an exported
HERMES_HOME=~/.hermes) hands pytest the production home, the sandbox was
skipped entirely — every argless SessionDB()/SessionStore() and every
collection-time DEFAULT_DB_PATH froze onto the real state.db.

Fixes (fail the class, one owner):

* hermes_state._ensure_test_isolation(): single choke point wired into
  SessionDB.__init__ (every construction, incl. read_only). Under pytest
  (PYTEST_CURRENT_TEST / PYTEST_VERSION — inherited by subprocess
  children), a db path resolving to <real-root>/state.db or
  <real-root>/profiles/<name>/state.db raises RuntimeError('live-system
  guard: ...') before any connection, mkdir, or journal-mode pragma.
* tests/conftest.py: session sandbox now also tempdir-redirects a pre-set
  HERMES_HOME that points at the production root (the actual escape
  vector); kanban deny-list capture updated to match. New autouse
  _state_db_write_guard fixture honors the existing
  @pytest.mark.live_system_guard_bypass marker as the escape hatch and
  feeds custom (non-~/.hermes) production roots into the guard deny-list.
* gateway/session.py: SessionStore.__init__ no longer swallows the guard's
  RuntimeError into the JSONL fallback — guard trips are loud.
* tests/hermes_state/test_live_db_isolation_guard.py: behavioral
  regression tests — production paths (direct, profile, read-only,
  unnormalized, default-resolution) raise; tmp HERMES_HOME works; bypass
  marker works; SessionStore re-raises guard errors but still degrades on
  ordinary failures; subprocess child without HERMES_HOME is refused while
  a hermetic child succeeds.

No new HERMES_* env vars; no hardcoded ~/.hermes (platform root comes from
hermes_constants._get_platform_default_hermes_home()).
2026-08-06 07:49:50 -07:00
Teknium c4aea32317 fix(db): never downgrade journal mode on a database with concurrent openers
The WAL-reset-vulnerability gate (#70055 lineage) could flip a LIVE WAL
database to journal_mode=DELETE while another process was writing to it.
Observed on state.db (Aug 5): a pytest process on the repo .venv (SQLite
3.50.4, vulnerable) opened the live ~/.hermes/state.db while the gateway
(SQLite 3.53.1, WAL) held it, downgraded the journal mode, and destroyed
the gateway's committed-but-uncheckpointed WAL transactions (disk rows
went 10 -> 0 while memory held 185). cron/executions.db already had the
"leave WAL in place, no live downgrade under concurrent openers" rule via
the on-disk WAL probe; state.db and every other store shared the hole
whenever the mode PROBE itself was blocked by a concurrent opener's locks
("could not read the mode" was treated as "not WAL" -> flip anyway).

Generalized in the single journal-mode owner (apply_wal_with_fallback),
covering ALL call sites (state.db, kanban.db, projects.db,
cron/executions.db, delivery_ledger, async_delegation,
verification_evidence, discord recovery, response_store.db,
memory_store.db):

- _set_journal_mode_no_wait(): the only journal-mode switch primitive for
  non-WAL targets. Forces busy_timeout=0 around the pragma so SQLite's own
  exclusivity requirement for leaving WAL becomes the concurrent-opener
  detector — any other opener (this process or another) makes the flip
  fail immediately instead of waiting out a busy timeout and sneaking the
  flip in under a live writer.
- Vulnerable-SQLite gate: an unreadable journal mode (probe blocked) now
  means "ownership not provably exclusive" — leave the mode untouched and
  warn, never flip. A lock conflict on the flip itself likewise leaves the
  mode alone.
- Configured journal_mode=delete: refuses (raises) rather than downgrading
  blind when the mode cannot be verified under a concurrent opener.
- Filesystem-incompat fallback: re-raises instead of downgrading when the
  on-disk mode cannot be verified.
- New/exclusively-owned DBs on vulnerable builds behave exactly as before
  (DELETE gate retained per #70055).

Behavioral tests use a REAL second process (and a real second connection
holding an exclusive lock) with the blocked-state assertions running WHILE
the holder owns the DB, plus exclusive-ownership downgrade-still-happens
coverage.
2026-08-06 07:49:42 -07:00
Teknium 70de958921 fix(cron): lifecycle guard — never crash on binary referenced paths, stop matching lifecycle words inside SQL/text
Two live failures on the same guard (cron/lifecycle_guard.py), both of
which blocked legitimate diagnostics from inside the gateway:

1. Crash class: the referenced-script walk read compiled binaries as if
   they were shell scripts. Reading/inspecting a referenced file is now
   best-effort by construction: executable magic numbers (ELF, PE,
   Mach-O fat/thin) short-circuit before any full read via a 4KB sniff,
   NUL-bearing heads are skipped as non-scripts, and unreadable paths of
   every kind (NUL bytes in the token, ENAMETOOLONG, missing files)
   degrade to "nothing to scan" instead of raising. A second fail-safe
   layer wraps the pure-string fallback so the boundary function stays
   total even if the tokenizer itself fails.

2. False-positive class: the lifecycle regex matched its command shapes
   inside DATA arguments — SQL string literals passed to sqlite3/psql
   and grep/rg/journalctl patterns hunting for the lifecycle string in
   logs. Added a fail-closed second-pass exemption: on a raw regex hit,
   re-scan with data-sink executables' arguments masked; only a match
   that survives (i.e. sits in command position) blocks. Masking is
   skipped for pipes into shells/xargs, command/process substitution,
   sqlite3 dot-commands and psql backslash escapes, so it can only ever
   allow, never miss.

Behavioral tests: exact live false-positive shapes as negatives, the
smuggling shapes as positives, the kill-primitive positive catalog
unchanged, and an adversarial never-raises suite (NUL bytes, non-UTF-8,
/dev/*, directories, missing files, magic-prefix binaries).
2026-08-06 07:49:35 -07:00
kshitij 863e313185 fix: close simplify-pass findings — scheduler sibling site + home-unresolvable totality
3-reviewer simplify pass (reuse/quality/efficiency) findings:

- cron/scheduler.py _run_job_script: the ORIGINAL that
  lifecycle_guard._resolve_script_path documents mirroring had the exact
  same unguarded expanduser() — a NUL-bearing script value survives
  creation (the guard treats it as nothing-to-scan) and crashed the
  scheduler at fire time with ValueError instead of a clean job failure.
  Same ingestion contract applied; regression test added.
- lifecycle_guard._resolve_script_path: get_hermes_home() -> Path.home()
  raises RuntimeError when neither HERMES_HOME nor HOME resolves
  (arbitrary-UID containers); the cron entry point called it bare.
  Caught -> None; totality test added.
- terminal_tool: stale 'cat ...' docstring updated to the bounded
  head -c form.
- lifecycle_guard: dead 'script_text and' condition dropped (guarded by
  'if not script_text: continue' directly above).

Efficiency reviewer: no material findings (measured — encode/expand
costs negligible vs walk I/O, no timing regression vs base).
2026-08-06 17:36:40 +05:30
kshitij c8d48b8b13 fix(cron): make the lifecycle guard total — sanitize at ingestion, not per-syscall
The guard feeds untrusted byte streams (tokenized binaries, remote cat
output) into OS-path and shell-text operations; every incident so far
(#76762, #77703, #77780, #78256, #77729) was hot-fixed with an except at
whichever frame crashed that week. tilllt's regression suite on #79454
showed 4 members of the class still open on merged main. Close the class
at three boundaries instead:

- _expand_candidate_path(): single ingestion chokepoint for path
  candidates — reject NUL/empty tokens before any Path OS call and
  tolerate ValueError/RuntimeError/OSError from expanduser (T1/T2, plus
  the HOME-unset launchd crash). Both _resolve_terminal_script_path and
  _resolve_script_path now go through it.
- _sanitize_remote_script_text(): apply the local-read contract (NUL =
  binary = nothing to scan; >1MiB = fail closed) to whatever any
  read_remote_script callback returns, at the recursion boundary — the
  guard stops trusting its callbacks (T3/T4).
- contains_gateway_lifecycle_command_or_referenced_script() is now total
  by construction: direct regex scans (pure string ops) run first; the
  best-effort filesystem walk is wrapped so an unexpected failure logs a
  warning and falls back to the direct-scan verdict instead of killing
  every terminal command until gateway restart.

terminal_tool's remote fallback also bounds the read at the source
(head -c 1MiB+1 instead of cat), so a 166MB ELF never crosses the wire —
the superlinear-shlex 30-minute stall from #79838's field report drops
to a 0.02s fail-closed verdict.

Regression tests: tilllt's T1-T4 adopted verbatim, plus an adversarial
never-raises sweep (NUL paths, unset HOME, over-long paths) and a
walk-crash fallback test.
2026-08-06 17:36:40 +05:30
kshitij ea0d54db1d refactor: fold /simplify-code findings
- Single source for the approval-derived bound: public human_wait_ceiling()
  in tools/approval.py; the gate's lock-timeout helper delegates to it
  instead of re-deriving timeout + margin (was duplicated in two modules
  and reached for a private _get_approval_timeout).
- Shared _clamped_window_seconds() for the close-time accrual and the
  open-window read, so the two clamps are identical by construction.
- Gate __init__ grows session_key kwarg; tests construct via the real
  constructor instead of mutating privates post-hoc.
- Gateway test resolves its pending approval via resolve_gateway_approval()
  (the production /deny path) instead of hand-rolling queue-entry internals.
- Docstring accuracy: human_wait_seconds monotonicity caveat under cap
  eviction; s/pre_tool_block/pre_tool_call/ hook name.
2026-08-06 17:03:10 +05:30
kshitij 10fb01e725 fix: harden human-wait tracker from review findings
Review-driven follow-up to the #79719 fix:

- Clamp the CLOSE-side accrual too: a wedged window that eventually closed
  used to inject its full unclamped overstay into completed_seconds,
  retroactively extending a running batch's deadline by hours. Both clamps
  now share one ceiling helper (_human_wait_ceiling = approvals.timeout +
  HUMAN_WAIT_MARGIN_S), and the gate's lock-timeout uses the same margin
  constant so the bounds cannot drift apart.

- Evict idle sessions until the table is under the cap (was: at most one
  per insert, so churn could outgrow _HUMAN_WAIT_MAX_SESSIONS). Entries
  with an open window are still never evicted.

- Log (debug) instead of silently swallowing a failed session-key snapshot
  in the gate constructor.

Tests: close-side clamp regression + table-cap assertion added; suite at
17 passed.
2026-08-06 17:03:10 +05:30
kshitij 3305cfd2bb fix(agent): measure batch-deadline exclusion at the human wait, not authorization-gate residency
A tool wedged inside _ConcurrentToolAuthorizationGate hung the whole turn
forever (#79719): excluded_seconds() measured residency in gate.run() —
arbitrary code — so an open window grew 1:1 with wall clock and the batch
deadline's remaining was constant (remaining = deadline - window_started;
now cancels out). A hanging pre_tool_call plugin or an approval round-trip
to a dead client defeated the deadline entirely. The serialization lock was
also an unbounded acquire, so every other worker needing authorization
parked behind the wedged holder forever.

Fix, in two halves:

- tools/approval.py grows per-session human-wait accounting
  (human_wait_window / human_wait_seconds). The two places that are
  verifiably blocked on a HUMAN — the CLI approval prompt and the gateway
  approval poll loop — mark their own windows. Both are intrinsically
  bounded by approvals.timeout; the open-window read is additionally
  clamped to that timeout plus a margin as belt-and-braces.

- _ConcurrentToolAuthorizationGate keeps only serialization, with a bounded
  acquire (approvals.timeout + 60s; on expiry the prompt runs unserialized —
  the same degradation the start-order gate accepted in #79705).
  excluded_seconds() becomes a baseline-delta read of the session's
  human-wait total.

A wedged plugin now contributes nothing to the exclusion, so the batch
times out at the normal deadline with correctly labeled results, while a
genuine approval wait — which can legitimately exceed any fixed bound —
still extends the deadline in full. E2E (real AIAgent, worktree imports):
wedged-plugin batch on main never ends (>30s observed, 3s deadline); with
the fix it ends at 3.0s. A 4s simulated approval over a 2s deadline
completes without a timeout label.

Closes #79719
2026-08-06 17:03:10 +05:30
kshitij aaf9688519 refactor(gateway): extract the hygiene recovery gate and forward the failure reason
Follow-up to c0d974b19 (#79741). Three review findings against that commit,
none of which change the escalation behaviour it shipped.

1. The recovery decision lived inline in `_handle_message_with_agent`, a
   ~2000-line async method, so the only way to pin it was a test that read
   `inspect.getsource(...)` and asserted on substrings. AGENTS.md bans reading
   source in tests outright and names this exact situation: "if the logic lives
   inline in a god-file (gateway/run.py) and extracting it feels disruptive:
   that's the actual signal to do the extraction, not to regex around it."

   Those tests were not merely stylistically wrong, they were actively harmful.
   One asserted the substring `_new_tokens < _approx_tokens` was PRESENT -- so
   it passed while the gate had the bug that substring represents, and had to be
   edited when the gate was fixed. It failed on correct code and passed on
   broken code, in one assertion.

   Extracted `hygiene_compaction_recovered()` as a module-level pure predicate
   and replaced the three source-reading tests with eight direct unit tests.
   The extraction immediately earned itself: the new tests caught a `NameError`
   (the predicate called `compression_made_progress` while the module bound it
   under an alias) that a source-text assertion cannot see, because the symbol
   is spelled correctly in the source and only fails at runtime.

2. The gate inferred "did the transcript actually get rewritten" from a numeric
   side effect -- the degenerate "did not rotate or compact in place" path
   (#21301) reuses the pre-compression counts -- when the booleans
   `_hyg_rotated` / `_hyg_in_place` were already in scope and explicitly set
   False on that path. The predicate now takes them directly, so a future edit
   that re-estimates instead of reusing the old counts cannot silently defeat
   the escalation.

3. `_record_hygiene_cooldown` passed no `error` to
   `record_compression_failure_cooldown`, which writes `compression_failure_error`
   unconditionally -- so a hygiene failure clobbered to NULL whatever reason the
   in-conversation path had recorded, and readers then show the user "unknown
   error" (agent/manual_compression_feedback.py, gateway/slash_commands.py). The
   reason was already in hand at both call sites. Pre-existing from #74136 but
   amplified by escalation: a blank reason on a 45-minute cooldown is far more
   user-visible than on a 5-minute one.

Also: the ladder docstring described the compressor's absolute 60/300/900s
ladder while the constant is multipliers (1, 3, 9); the config docs still
described `hygiene_failure_cooldown_seconds` as a flat interval rather than the
first rung of a capped ladder; and `PersistentState.hygiene_failure_streak` now
documents that it is process-local by design -- keying on `session_key` is what
survives compaction rotation, which the persisted `compression_*_streak`
columns cannot express since they key on the rotating `session_id`. Making it
durable is a schema change, tracked on #79624 rather than smuggled in here.

Also replaces the file's hand-written `_Runner` stub with
`object.__new__(GatewayRunner)` (already the idiom elsewhere in the same file).
The stub reimplemented `_session_state` and `_peek_session_state`, so the tests
exercised copies that could drift from production; using the real class
immediately made one assertion stronger -- on a fresh runner `_sessions` does not
exist at all until something materialises it, so the reset provably did not even
create the map.

A review pass on this follow-up then caught that the CALL SITE was still
unbound: deleting the whole `if not _hyg_aborted: if
hygiene_compaction_recovered(...)` block left every ladder test green, because
the unit tests prove the predicate correct without proving it is wired in. The
merged commit had the same gap and its only cover was the banned source-reading
test. `test_session_hygiene_forces_in_place_compaction_with_bound_session_db`
now spies the reset on a genuine in-place compaction, so deleting the wiring
fails. Two earlier attempts at this test did NOT close the gap -- asserting on
streak VALUES passes either way, since the streak is 0 whether or not the gate
ran; only a positive spy assertion on a recovering run detects the deletion.

Same pass also corrected an overstatement: point (2) is hardening, not a live
bug. The degenerate path also sets `_new_count = _msg_count` and `_new_tokens =
_approx_tokens`, and `compression_made_progress(n, n, t, t)` is always False, so
the merged code already declined to reset there. A 200k-trial fuzz over the
reachable state space found zero behavioural disagreements between the merged
gate and this one. The guard's value is surviving a future edit that stops
reusing those counts.

Tests: 28 in tests/gateway/test_hygiene_failure_cooldown_ladder.py (8 new unit
tests for the predicate, 3 for reason forwarding, 3 source-reading tests
deleted). All 5 mutations caught -- including one that restores the hand-rolled
comparison and one that removes the rotated/in_place guard. Two mutations
initially SURVIVED and exposed vacuous tests of my own: the no-rewrite test used
counts the progress predicate already rejects, so it passed without binding the
guard at all; it now passes counts that read as progress on their own, proving
the guard is what rejects them. gateway hygiene + session-state + agent
compression-progress suites: 54 passed; ruff clean.

Refs #79624
2026-08-06 15:04:53 +05:30
Teknium 8f2712725a feat: /refine — run the memory/skill self-improvement review on demand
/refine [focus] fires the existing background review fork
(AIAgent._spawn_background_review) immediately instead of waiting for
the automatic 10-turn memory / 10-iteration skill nudge counters.
Optional focus instructions are appended to the review prompt so the
fork prioritizes what the user asked for (e.g. '/refine save the
deploy workflow as a skill').

- New optional focus parameter threaded through
  _spawn_background_review -> spawn_background_review_thread.
  Automatic post-turn reviews pass None and their prompts are
  byte-identical to before.
- CLI handler snapshots conversation_history; gateway handler pulls
  the idle session's cached AIAgent from _agent_cache (rejected while
  the agent is running).
- Review runs in a daemon thread against the snapshot — live
  conversation, message alternation, and prompt cache untouched.
- Slack stays under the 50-slash cap via /hermes refine.

Adapted from the /refine concept in Prime Intellect's Prime-Agent
(Continual Harness); Hermes' equivalent durable state is the
memory + skill stores, so the review fork is the natural target.
2026-08-05 22:40:51 -07:00
Teknium 6518aa184e feat: /heartbeat — recurring session re-entry prompt fired when idle
/heartbeat every <interval> <prompt> gives the current session one
recurring instruction. When the session is idle and the interval has
elapsed, the prompt is injected as a plain user turn — same
conversation, same context, prompt cache and role alternation
untouched.

- CLI: idle-poll watchdog thread (wake-word watchdog pattern) feeding
  _pending_input; gateway: single gateway-wide async poller injecting
  through the adapter FIFO. Busy sessions coalesce their tick to the
  next idle poll.
- Missed ticks coalesce (anchor resets on fire) — a busy hour yields
  ONE heartbeat turn, never a backlog. Real user messages always win.
- 60s interval floor; injected prompt carries a don't-invent-work
  guard so idle heartbeats don't generate busywork.
- State persists in SessionDB.state_meta (heartbeat:<session_id>),
  survives /resume, migrates across compression session rotations
  alongside /goal state.
- Session-scoped and in-process by design — durable cross-process
  schedules remain the cron subsystem's job (docs draw the boundary).
- Slack stays under the 50-slash cap via /hermes heartbeat; ghost-text
  suggester now prefers the shortest prefix match so /he still
  suggests /help.

Adapted from the session-heartbeat concept in Prime Intellect's
Prime-Agent (/heartbeat).
2026-08-05 22:32:55 -07:00
Teknium 6e041d5244 feat(goals): quality gates — deterministic commands that must pass before /goal completes
/goal gate add <command> attaches shell commands to the active goal.
Gates run at turn boundary BEFORE the LLM judge: a failing gate skips
the judge entirely and feeds its exit code + bounded output tail back
as the continuation prompt, so the agent iterates against concrete
evidence instead of a prose verdict.

- Unchanged-workspace skip: a gate that failed on an identical
  workspace (git HEAD + status fingerprint) is not re-run — the
  recorded failure replays and the attempt count advances.
- Bounded retries (default 3) + per-gate timeout (default 300s);
  exhaustion auto-pauses the goal like the turn budget does.
- Gates persist in SessionDB.state_meta with the goal (survive
  /resume and compression rotation); pre-gate goal rows load
  unchanged.
- /goal gate [list|add|remove|clear] on CLI + gateway; 'gate' added
  to the mid-run control-verb whitelist (gates only run at turn
  boundary, so editing the list mid-run is safe).

Adapted from the quality-gate concept in Prime Intellect's Prime-Agent
(--autonomous-gate).
2026-08-05 22:32:39 -07:00
Adolanium ffdbc883ee fix(read_extract): cap anydoc input size before conversion
The anydoc path from #79781 passed every covered file straight to
to_markdown with no pre-check. anydoc loads the whole document through
its Rust core and the read_file char budget only applies after
conversion, so one large PDF or deck could pin a tool turn and spike
RAM.

_extract_anydoc now rejects inputs over MAX_ANYDOC_BYTES (50 MB) with
ExtractionError before calling the converter, which routes them to the
existing read_file fallthrough instead of converting. No timeout is
added: the conversion is a synchronous Rust call that cannot be
cancelled from Python, so a thread-based deadline would bound the wait
but leave the RAM burn running in the background.
2026-08-05 22:02:42 -07:00
Adolanium 997a913a58 fix(read_extract): retry anydoc init after failure instead of sticky disable
The first _anydoc() load cached None on any failure (network blip,
missing wheel, pip race), so one bad first try disabled document
extraction for the rest of the process. Long-lived gateway and desktop
workers never recovered.

Failed loads now cool down for ANYDOC_RETRY_SECONDS and retry instead
of sticking, and a lock serializes first use so parallel readers cannot
double-install or race a failure into the cache. Successful loads are
still cached for the process lifetime.
2026-08-05 21:59:34 -07:00
Teknium 4e7e103ba6 fix(gemini): interpose placeholder model turn between tool result and user text
Port from google-gemini/gemini-cli#28700: when an interrupted/failed turn
leaves history ending on an unanswered tool result and the user sends a new
message, fusing the two into one Gemini user content makes the model read the
trailing text as a continuation of the tool result — it 'finishes your
sentence' instead of answering.

Builds on #68863 (@rille111), which split the mixed functionResponse/text
merge but emitted two consecutive user contents — a shape Gemini's
alternation contract rejects with HTTP 400 on other request paths (#55125).
This follow-up interposes gemini-cli's INTERRUPTED_RESPONSE_PLACEHOLDER model
turn between the split contents so the request stays alternation-valid while
the user's message remains a turn of its own.
2026-08-05 17:21:01 -07:00
Rickard Robin 0afeaaa0a1 fix(gemini): prevent user message merge into adjacent function response
Do not fold a human user text turn into a preceding functionResponse
user content. Gemini 3 accepts that fold with HTTP 200 but then returns
an empty model response.

Contract:
- ordinary same-role merges remain (parallel tool results, back-to-back
  plain user texts) for Gemini alternation
- only mixed functionResponse/text user turns are split
2026-08-05 17:21:01 -07:00
kshitij 169758d42f perf(tests): cut test_hermes_state.py 52s -> 10s — kill sleep throttle + per-row seeding
test_hermes_state.py was the slowest file in the suite (46.7s in CI's
durations cache) and therefore the LPT floor: no test slice can finish
faster than its slowest file, which caps how far slicing the test matrix
wider can cut the merge-gate critical path.

Profiling (cProfile on the slowest tests) found the time was dead, not
work:

1. time.sleep in optimize_fts_storage's inter-chunk throttle — 4.1s of
   a 4.6s migration test. The throttle exists so a LIVE gateway/CLI
   sharing the DB isn't starved of the write lock; tests run against a
   private tmp-path DB with no concurrent process, so the sleep protects
   nobody. New autouse fixture zeroes _FTS_REBUILD_MIN_PAUSE /
   _FTS_REBUILD_DUTY_FACTOR for this file (~20s saved). No test asserts
   on wall-clock pacing, so nothing weakens.

2. TestGetMessagesPagination._seed appending 3000 messages one
   append_message (= one commit, and off WAL one fsync) at a time —
   ~10s of seeding before the query under test even ran. Switched to
   append_messages_batch (one write transaction), the API the docstring
   of which exists for exactly this shape. The perf contract the seed
   feeds still discriminates: measured 11 progress-handler steps on the
   indexed path vs 855 on the forced scan path, against the unchanged
   300 threshold.

Measured (local, 3 runs + canonical runner):
  before: 187 passed in 51.8s
  after:  187 passed in 9.1-16.1s (canonical scripts/run_tests.sh: 14.9s)

Zero production code touched; 187 tests before and after.
2026-08-06 05:38:04 +05:30
Teknium b2598b41e1 feat(read_file): widen document extraction to PDF/legacy Office/ODF/RTF/EPUB via optional anydoc
read_file's auto-extraction covered only the stdlib trio (.ipynb/.docx/
.xlsx). firecrawl-anydoc (MIT, Rust core, imports as `anydoc`) converts
Word, PowerPoint, Excel — including legacy .doc/.ppt/.xls — OpenDocument,
RTF, EPUB, and PDF to clean Markdown through one shared document model.

Wiring follows the footprint ladder: no new tool, no hard dependency.
- tools/read_extract.py gains an ANYDOC_EXTENSIONS set that is active
  only when the converter imports; the stdlib extractors remain
  authoritative for their three formats so behavior is identical with
  or without the package.
- tools/lazy_deps.py adds tool.doc_extract (firecrawl-anydoc==0.1.6),
  installed on first read of such a file with prompt=False so read_file
  can never block. Lazy-only for now: the package's first release was
  2026-08-04, inside uv's 14-day exclude-newer quarantine, so the
  mirrored pyproject extra lands after it clears.
- Any anydoc ConvertError maps to ExtractionError, falling back to the
  existing path/binary handling instead of erroring the tool.

Tests: real-binding suite skips cleanly when the wheel is absent
(verified: 15 passed/3 skipped without it, 18 passed with it), plus an
absent-dep contract class that pins the fallback regardless of local
install state.
2026-08-05 17:07:47 -07:00
kshitij 9baf92b7f3 test(search): update grep command mirrors to -rnHE for fidelity with production 2026-08-06 05:31:17 +05:30
Kevin Yin 7c6f9affd7 fix(file): align grep fallback regex behavior 2026-08-06 05:31:17 +05:30
Teknium 9a9cf6ae83 fix(cron): tolerate NUL bytes in referenced-script paths at os.open
Residual #76762 class: _read_referenced_script caught OSError from
os.open but not ValueError, so a path token carrying an embedded NUL
(tokenized binary-adjacent command text) crashed the terminal tool's
lifecycle guard with 'ValueError: embedded null byte' instead of being
skipped as nothing-to-scan. Reproduced live against main. Same
treatment the resolve()-time site already has; two sabotage-verified
regressions added.
2026-08-05 16:53:50 -07:00
Paolo Shamoon 03dc4aad52 fix: hide memory tool from cron agents
Cron agents are constructed with skip_memory=True, so the memory
backend is not initialised — exposing the memory tool only gives the
model an unbacked tool that fails at runtime with 'Memory is not
available.'  Add 'memory' to _resolve_cron_disabled_toolsets() so the
tool is stripped from the schema before the model can call it.

Fixes #38129.

Co-authored-by: Paolo Shamoon <Paolo@Dylans-Mac-Studio.local>
2026-08-06 05:20:56 +05:30
kshitij c0d974b19f fix(gateway): escalate the session-hygiene compaction cooldown on repeat failures
A gateway session whose summary model keeps timing out no longer retries
compaction on the same fixed interval forever.

The in-agent compressor already escalates repeat summary timeouts
60 -> 300 -> 900s (ContextCompressor.record_timeout_failure), but that ladder
reads the in-memory _consecutive_timeout_failures counter and
bind_session_state() zeroes it (context_compressor.py:1645). Session hygiene
constructs a FRESH AIAgent for every run (gateway/run.py:16820) and re-binds
state each time, so from the gateway that streak is structurally always 0 --
only the flat hygiene_failure_cooldown_seconds (300s) could ever be recorded.
Issue #79624 reported exactly that steady state: an oversized session
(1053 messages, ~119.5k tokens) whose aux model always timed out, re-attempting
compaction every 300s across five days until the reporter deleted the session
by hand.

Track the streak on PersistentState instead, which outlives the per-run agent
and is not cleared by turn/boundary resets, so consecutive hygiene failures
climb 300 -> 900 -> 2700s and then saturate. Both failure sites (progress
timeout and aborted compression) feed it; a real compression resets it, so a
session that recovers starts from the first rung again. The ladder multiplies
the configured base, so operators who tuned
hygiene_failure_cooldown_seconds keep their first rung. Per-session, so one
wedged chat cannot penalize other conversations.

Deliberately NOT changed, since each is a maintainer policy call rather than a
defect (all three are written up on #79624):
  - no durable failure-streak column, so escalation still resets on restart
  - the gateway 30s / in-agent 120s / aux-client 300s-floor timeout mismatch
  - no `hermes doctor` check or `hermes sessions list` marker for a session
    stuck in a compression-failure cooldown

Note the reported exit(1) is NOT a crash: it is the deliberate
_signal_initiated_shutdown path (gateway/run.py:26746-26751, #5646) that lets
systemd Restart=on-failure revive the gateway after a bare SIGTERM, and it
fires on every `systemctl restart` independently of compaction. The compaction
log lines appear after the shutdown line because the gateway-owned executor is
torn down with shutdown(wait=False, cancel_futures=True) (run.py:21164), so an
in-flight turn keeps logging during teardown. Full analysis on the issue.

Post-review hardening (Phase 2c + /simplify-code found five real defects in the
first cut):
  - the recovery gate hand-rolled `_new_tokens < _approx_tokens` when a canonical
    predicate already existed: `compression_made_progress` (agent/turn_context.py,
    #39548). They disagree on 3 of 5 cases -- the hand-rolled form misses a
    row-count win when the summary keeps the token estimate flat, misses one
    where the summary is slightly MORE verbose (so a genuinely recovered session
    would keep escalating forever), and counts a sub-5% wobble as recovery. Now
    reuses the shared predicate, promoted from `_compression_made_progress` to a
    public name with the old private name kept as a back-compat alias so the
    existing importer (tests/agent/test_protected_tail_pressure_61932.py) and any
    patcher of that symbol keep working.
  - the reset was gated on "not aborted", but the degenerate "did not rotate or
    compact in place" branch (#21301) is NOT aborted and yields zero reduction,
    so a session wedged there reset its streak every run and could never
    escalate -- silently defeating the fix. Now gated on real progress.
  - no absolute ceiling: base * 9 reaches 9h at an operator base of 3600s,
    indistinguishable from "compaction switched off". Added
    _HYGIENE_COOLDOWN_MAX_SECONDS = 3600, mirroring the in-file
    _RECONNECT_BACKOFF_CAP precedent.
  - the reset used the get-or-create accessor to write a 0 that was already 0,
    materialising a _sessions entry (never evicted). Now peeks.
  - the abort verdict was probed twice, leaving the reset/record mutual
    exclusion implicit; a future await between the probes would have broken it
    silently. Computed once into _hyg_aborted.

Tests: 19 new in tests/gateway/test_hygiene_failure_cooldown_ladder.py --
ladder escalation, saturation, the absolute cap, per-session isolation,
reset-on-recovery, custom/zero base, PersistentState scoping (a mutation moving
the field to TurnState fails), degraded runners, the progress gate, the exact
progress-predicate semantics the gate depends on, and end-to-end that the
escalated value is what reaches the state DB. All 12 mutations caught, including
ones that restore the flat cooldown (the original bug), ungate the reset, swap
the canonical predicate back for the hand-rolled comparison, remove the cap, and
share the streak globally; the harness hard-errors when a mutation cannot be
applied, since a silently no-op mutation check is worse than none -- an earlier
version of it WAS silently no-opping after a refactor. The gate's contract test
slices by AST node span rather than a fixed character count, which had already
truncated once as the block grew. gateway hygiene + session-state + the three
touched agent compression suites: 50 passed; ruff clean.

E2E with real imports demonstrates the premise rather than asserting it:
bind_session_state zeroes the in-agent counter, and the recorded deadlines go
300 -> 900 -> 2700 -> 2700 -> 2700s where they were previously a flat 300s.

Reported by @yucezerey (#79624), whose state.db column dump and
"deleting the session fixed it" datapoint made the real mechanism findable.
2026-08-06 05:00:48 +05:30
kshitij 43fc86562c fix(utils): tighten create_mode semantics and close the yaml 0600 transit window
Post-review fixes on the preserve_mode/create_mode follow-up:

- create_mode is now applied ONLY when the target does not exist, on
  both atomic_write_text and atomic_yaml_write. Previously
  atomic_write_text(path, s, create_mode=X) without preserve_mode would
  silently chmod an EXISTING file to X (docstring/code mismatch, latent
  trap -- no caller relied on it), and a stat failure on an existing
  file could fall through to create_mode instead of leaving the mode
  alone.

- atomic_yaml_write now fchmods the temp fd BEFORE the replace when a
  mode is known, matching atomic_write_text: a freshly created
  distribution.yaml no longer transits through mkstemp's 0600 (a crash
  between replace and chmod could previously leave it 0600 forever).
  The post-replace _restore_file_mode stays as the Windows path.

- fchmod moved inside the fdopen context in atomic_write_text, so a
  raising fchmod can no longer leak the fd.

Tests: create_mode-never-rewrites-existing guard (mutation-checked) and
a monkeypatch.delattr(os, 'fchmod') test covering the Windows
post-replace branch that the win32 module skip left uncovered.
2026-08-06 05:00:17 +05:30
kshitij 3556728a54 refactor(utils): move mode+owner preservation into atomic_write_text
Follow-up to the salvaged #79323 commits. The three hand-rolled
stat -> atomic_write_text -> chmod blocks (xai migration, uninstaller
shell-rc rewrite, dashboard SOUL.md editor) collapse into an opt-in
preserve_mode=True kwarg on utils.atomic_write_text, plus create_mode=
on both atomic_write_text and atomic_yaml_write for first-create paths
(SOUL.md first save, write_manifest's allowlist create path).

Beyond deduplication this closes two gaps the hand-rolled copies had:

- Owner preservation: the old in-place writes kept the inode, so file
  ownership survived root-run rewrites for free. atomic_write_text
  swaps in a new inode owned by the writing user, and the hand-rolled
  blocks restored only the mode -- a root-run 'hermes migrate xai' or
  sudo uninstall on a user-owned Docker/NAS volume would flip
  config.yaml / ~/.zshrc ownership to root. preserve_mode now routes
  through the same _preserve_file_owner/_restore_file_owner helpers
  atomic_yaml_write and atomic_json_write already use.

- chmod-after-replace window: the mode is applied to the temp fd via
  fchmod BEFORE the replace (mirroring atomic_json_write's mode= param),
  so the target never transits through mkstemp's 0600.

Also removes write_manifest's caller-side existed/chmod block (and its
small TOCTOU) in favor of atomic_yaml_write(create_mode=0o644), and
corrects the SOUL.md mode comment (the default profile's runtime seeder
does run it through _secure_file; named profiles do not).

preserve_mode defaults to False so the existing callers (memory store,
skill manager, cron, agent importer) keep their current semantics.

New tests in tests/test_atomic_write_text_metadata.py cover mode
preservation, owner restore through symlinks, fchmod-before-replace,
create_mode on both writers, and no-behavior-change without opt-in;
all mutation-checked.
2026-08-06 05:00:17 +05:30
briandevans 4541d30181 fix(cli): keep newly created SOUL.md and distribution.yaml at 0644
Both files were routed through the shared atomic writers earlier in this
branch. tempfile.mkstemp creates the temp file 0600 and the atomic swap
carries that mode onto the target, so the *create* paths silently tightened
two files that previously landed at the umask default:

- web_routers/profiles.py: the dashboard persona editor's first-ever Save has
  no prior SOUL.md to copy permissions from, so the existing guard skipped the
  chmod entirely -- contradicting the comment directly below it, which states
  profile SOUL.md is created 0644 and is not run through _secure_file.
- profile_distribution.py: atomic_yaml_write only restores a mode it captured
  from a file that already existed. _materialize() calls write_manifest() with
  no manifest on disk whenever a distribution declares an explicit
  distribution_owned allowlist that omits distribution.yaml, so the staged
  copy is never placed in the profile.

Both are fixed with a local chmod at the two sites this branch regressed;
utils.py's public mode semantics are left alone. profiles.py now also
distinguishes "no file yet" (FileNotFoundError -> 0644) from "stat failed for
some other reason" (-> leave the mode alone rather than guess at it).

uninstall.py and xai_retirement.py have no create path and are unchanged: the
former captures prior_mode unconditionally after a successful read_text(), and
the latter runs require_readable_config_before_write() first.
2026-08-06 05:00:17 +05:30
briandevans c005546cbc test(cli): skip the permission-preservation cases on Windows
The four new mode-preservation guards assert POSIX permission bits, which
Windows does not model (os.chmod only toggles the read-only flag there).
Guard them the way the suite already guards POSIX-specific semantics so the
tests stay meaningful on Linux/macOS without failing for Windows contributors.
The symlink cases stay unguarded, matching the existing symlink tests in
tests/hermes_cli/.
2026-08-06 05:00:17 +05:30
briandevans 67827dd99e fix(cli): route the remaining destructive user-file rewrites through atomic writes
`utils.atomic_write_text`'s docstring states the invariant: it exists "so that
every destructive file rewrite in the codebase shares one implementation."
Four full-file rewrites of *existing user-authored files* still bypass it and
use a bare truncating `open(path, "w")` / `Path.write_text()`, which truncates
the target before the new content is produced. A crash, SIGINT, or ENOSPC
mid-write therefore leaves the file empty or half-written.

In all four cases the read half degrades silently to a default rather than
erroring, so the damage is invisible and the next write cements it:

* `xai_retirement.apply_migration()` rewrites the user's config.yaml. Merged
  commit beaa1a08e added a readability guard here and noted the writer "lives
  outside the atomic_yaml_write path, so the chokepoint didn't cover it"; this
  closes the durability half it left open. `--no-backup` is a documented flag,
  so on that path the truncated file is the only copy that exists, and the
  loader returns early on `doc is None` — the next run reports nothing to
  migrate rather than surfacing the damage.
* `uninstall.remove_path_from_shell_configs()` rewrites the user's shell rc
  (~/.bashrc, ~/.zshrc, ...). Hermes does not own these files and this function
  takes no backup; the enclosing `except Exception` downgrades a partial write
  to a warning, so the next login just starts a bare shell.
* `web_routers.profiles.update_profile_soul()` replaces SOUL.md from the
  dashboard editor. The paired GET reports an unreadable file as
  `{"content": "", "exists": False}`, so an interrupted save presents as "your
  persona was never set" and the editor's next Save persists the empty document.
* `profile_distribution.write_manifest()` rewrites distribution.yaml on every
  install/update. `read_manifest` treats an unparseable manifest as "not a
  distribution", silently dropping update tracking and env_requires.

The xAI migration keeps its ruamel round-trip dumper (comments, key order and
quoting must survive) and now serializes to a string before handing the bytes
to the shared writer. `write_manifest` moves to `atomic_yaml_write`, whose
SafeDumper output the manifest already round-trips through, retiring the local
`_dump_yaml` helper.

`atomic_write_text` recreates the target from a 0600 temp file, so each of its
call sites re-applies the file's previous permission bits: `_secure_file`
deliberately leaves config.yaml alone under managed (NixOS 0640) and container
installs, shell rc files are normally 0644, and profile SOUL.md is created 0644
and never secured. `atomic_yaml_write` already preserves mode and owner itself.
Routing through `atomic_replace` also keeps a symlinked config.yaml or ~/.zshrc
(dotfiles repo, managed deployment) pointing at the real file.

Tests: one regression test per site fails on clean main (the interrupted write
completes there and destroys the file) and passes here; the remaining cases are
behaviour guards covering symlink survival, permission preservation, comment
round-tripping, and the existing happy paths.
2026-08-06 05:00:17 +05:30
kshitij 55e70f570e test(sessions): guard the Windows backslash child arm of _cwd_prefix_clause
The quadruple-backslash pattern arm is the trickiest byte sequence in the
fix and had no direct coverage — 'simplifying' it to a double backslash
would break Windows child matching with every test still green. Mutation
checked: weakening the arm fails this test.
2026-08-06 04:16:12 +05:30
Drexuxux b37de01926 fix(sessions): escape LIKE wildcards in the cwd-prefix clause
_cwd_prefix_clause builds "cwd is this directory or under it" for session
listing, workspace resume and prune/archive. The two LIKE arms bound the
raw prefix, so `_` and `%` acted as wildcards on a value that is a path:

  cwd_prefix="/home/me/my_project"
    main -> ['sibling', 'target']    # /home/me/myXproject/src matched too
    fix  -> ['target']

`_` matches any single character, so a same-length sibling directory with
children falls inside the pattern. prune_sessions() deletes the rows it
matches (and their on-disk transcripts), so an unrelated project's history
goes with it.

Escape the needle and pair both arms with ESCAPE, the convention the rest
of this file already uses; the literal separator backslash in the Windows
pattern is escaped for the same reason. The `=` arm is an exact compare and
keeps the raw prefix, so directory-and-children matching is unchanged.

Follow-up to the *_like filters in #78681, kept separate because this helper
is shared by four call sites beyond prune.
2026-08-06 04:16:12 +05:30
Drexuxux 1d2dabce56 fix(sessions): escape LIKE wildcards in prune/archive substring filters
_prune_filter_where documents title_like / model_like / branch_like as
"case-insensitive substring matches", and the CLI confirmation renders them
as "title contains 'X'". They were bound straight into a bare LIKE, so `_`
matched any single character and `%` any run.

The builder backs prune_sessions(), which deletes session rows and their
on-disk transcripts, so the over-match is unrecoverable: pruning
title_like="user_auth" also destroys "user-auth", "userXauth" and
"user auth". `_` is not exotic here -- git branch names and session titles
carry it routinely.

Escape the operator's needle and add ESCAPE '\' to the three clauses, the
same convention the rest of this file already uses for LIKE queries. Match
direction is unchanged for needles without wildcards.

Left alone: _cwd_prefix_clause has the same unescaped shape but is shared
by four call sites beyond prune, so it is a separate change.
2026-08-06 04:16:12 +05:30
kshitij 042a2cf3d7 fix(agent): keep the start-order gate under the batch deadline and abort abandoned workers
Follow-up to the salvaged start-order gate bound. Two gaps remained, both
reachable through the same knob.

1. The gate bound ignored the batch deadline it sits under. With
   HERMES_CONCURRENT_TOOL_TIMEOUT_S below 120s the deadline fired first, so
   the parked tools were still reported as "timed out" without ever running --
   the exact bug the bound exists to fix. The gate now clamps to
   min(120s, batch_timeout / 2), matching the sibling constant's documented
   habit of relating the two timeouts.

2. A gate-parked worker released purely by its own timeout could wake up after
   the batch was abandoned and dispatch its tool anyway: wasted work whose
   result nobody reads, a duplicate post_tool_call for a tool_call_id the turn
   already closed as timeout, and agent._current_tool left pointing at a dead
   tool for the rest of the session (the main thread's reset already ran).
   Abandonment is now a first-class wakeup: both abandon sites set an event and
   notify the condition, and a released worker raises _BatchAbandoned instead
   of dispatching. Parked threads are reclaimed in milliseconds rather than one
   full gate timeout plus a tool runtime.

Also names the tool in the gate-timeout warning. The closure's function_name
binds the last-parsed tool, so logging it directly would have printed the wrong
name; it is threaded through _begin_in_order instead.

Measured, 3-tool batch with the first tool wedged during dispatch:

                          main    PR as-is   with this commit
  dispatched in batch       0        0          tool_b, tool_c
  dispatched after return   0        2 (ghost)  0
  _current_tool leaked      no       "tool_b"   no

Adds tests/run_agent/test_start_order_gate.py (3 tests). Mutation-checked
against the parent commit: the starvation guard passes there (it binds the
salvaged fix), while the deadline-clamp and abandonment guards both fail,
reproducing the ghost dispatch as
"tool(s) dispatched after the batch was abandoned: [tool_a, tool_b]".
2026-08-06 03:53:38 +05:30
kshitij a88b76bf7d
Merge pull request #79693 from kshitijk4poor/fix/79669-empty-fallback-deleted-heads
fix(gateway): don't claim deleted head chunks as delivered in the empty-fallback recovery
2026-08-06 03:24:26 +05:30
kshitij 145e777314 test(cron): close a blind spot in the kanban env drift guard
The AST invariant only matched `env["HERMES_KANBAN_X"] = ...` subscript
assignments, so a future dispatcher var added via `env.update({...})`,
`env.setdefault(...)`, or an annotated subscript would have slipped past
the guard and leaked into cron sessions unnoticed.

None of those shapes exist in _default_spawn today; this is about the
guard staying trustworthy as that function evolves.

Verified by injecting an unregistered var into _default_spawn one shape at
a time and requiring the guard to fail: subscript assign, annotated
assign, update(dict literal), setdefault(literal), and update(kwarg) are
all detected. Source restored byte-identical after probing.
tests/cron/ 410 passed; ruff clean.
2026-08-06 03:16:26 +05:30
kshitij 80f37e36ed fix(cron): don't let a cron job inherit a kanban worker's dispatcher identity
A kanban worker that fires a cron job in-process no longer leaks its task
identity into the cron agent.

The worker is a normal `hermes chat -q` CLI agent whose default toolset
includes `cronjob`, running with HERMES_KANBAN_TASK legitimately set in its
own environment. `cronjob(action="run")` calls run_one_job() -> run_job()
in that same process, so the cron AIAgent was misidentified as that worker:
kanban toolset force-added, kanban-worker protocol injected into its system
prompt, and kanban_complete defaulting task_id to $HERMES_KANBAN_TASK --
letting an unrelated cron job close the worker's task and overwrite real
results.

Fixed with a ContextVar (`non_dispatcher_owned_context`), not by clearing
os.environ. The env is process-global and shared with three concurrent
readers that all need the real values:

  * the worker's own claim heartbeat -- run_agent._touch_activity ->
    heartbeat_current_worker_from_env reads TASK/CLAIM_LOCK/RUN_ID, and the
    cron-run heartbeat thread drives it every 10s. Clearing them silently
    no-ops the heartbeat, so after DEFAULT_CLAIM_TTL_SECONDS (15 min) the
    dispatcher reclaims a task whose worker is still alive and re-dispatches
    it -- the same duplicate-work failure from the other direction.
  * the gateway's kanban watchers, which do their own HERMES_KANBAN_BOARD
    save/restore around a slow decompose_task() LLM call.
  * concurrent cron jobs, which take a *shared* read lock
    (_terminal_cwd_lock.acquire_read) and so interleave: job A clears, job B
    snapshots empty, A restores, B clears and its restore no-ops -- the
    worker's identity is destroyed permanently.

`is_dispatcher_owned_worker_context()` is now the single predicate every
HERMES_KANBAN_* identity gate consults before trusting those vars. It also
closes a pre-existing gap in agent/skill_utils.py, which read the vars
without consulting the delegate_task ContextVar at all; the `kanban` verdict
additionally bypasses _ENV_DETECT_CACHE, since a context-dependent answer
must not be memoized process-wide.

HERMES_KANBAN_BOARD/DB/WORKSPACES_ROOT are left untouched, so the #20074
board pin and the dispatcher's path overrides keep working.

Tests: 18 new, including thread-isolation, concurrent-cron-jobs, and an AST
invariant over _default_spawn that fails if the dispatcher gains a var that
is neither identity-gated nor explicitly classified behaviour-only. All six
mutations are caught, including one that reintroduces the os.environ clear.
tests/cron/ + kanban suites 440 passed; model_tools/skill_utils/boards 63
passed; ruff clean.

Reported and diagnosed by Geoff Friesen (#78961), who identified the symptom
and the exact gating mechanism.

Co-authored-by: Geoff Friesen <gfriesen1@users.noreply.github.com>
2026-08-06 03:16:26 +05:30
kshitij 68ebb198c7 fix(gateway): don't claim deleted head chunks as delivered in the empty-fallback recovery
Follow-up to #79669. That PR routed the three fallback recorder sites through
_record_turn_final_payload so a split turn would record the unsplit ledger
instead of a tail-only payload. For two of them that is right. For
_send_empty_fallback_final it is wrong, and it reintroduces the #78541 swallow
at the one site that was supposed to be fixed.

_send_empty_fallback_final is a *replacement* recovery: it sends the completed
text as a fresh message and deletes every tracked segment preview -- which on an
overflow split includes the sealed head chunks. After it runs, the only thing
on screen is the message it just sent. Recording the ledger there claims
delivery for text the same function just removed, so delivered_final_matches()
returns True, the gateway suppresses its own send, and the user is left with a
fraction of the answer.

Observed with a probe driving the real run() loop (543-char reply, 475-char head
sealed then deleted, 67-char tail committed):

  before this fix        recorded=543  matches=True   -> suppressed, 67/543 on screen
  after  this fix        recorded=67   matches=False  -> gateway sends the full answer

Record final_text verbatim here instead. The sibling site in
_send_fallback_final keeps the recorder: its delete is gated on
`continuation == final_text` and targets only the single active partial, never
the sealed heads, so the ledger correctly describes what survives.

The distinction is whether a recovery ADDS to what is on screen or REPLACES it.
Additive paths may record the ledger; replacing paths must record only what they
leave behind. _try_fresh_final is the same shape and #79669 handled it by
refusing the route on split turns.

Test drives the real seal-then-delete sequence and asserts the mismatch, so the
gateway is required to re-send. Mutation-checked: restoring the recorder call
turns it red.
2026-08-06 03:13:32 +05:30
kshitij 392e3a8c53 fix(gateway): finish the split-delivery bug class so the fix cannot duplicate or still swallow
The salvaged fix changed only the gateway's verdict: a payload-less
multi-message split stopped inheriting legacy trust. But six code paths set
_turn_split_delivery, and only one of them was taught to record a payload, so
the remaining five swapped the swallow for the opposite defect.

Fix the producers instead of only distrusting them at the boundary:

- _send_or_edit failed-final-edit branch: record the visible payload on split
  turns too. It deliberately skipped recording, which now reads as a mismatch
  and re-sends an answer already on screen -- reintroducing the duplicate
  #45517 fixed (#36965 / #25349).
- _send_fallback_final (x2) and _send_empty_fallback_final: route through
  _record_turn_final_payload instead of assigning _delivered_final_text
  directly. On a split turn their final_text is only the trailing chunk, so a
  fully delivered heads+tail reply recorded a tail-only payload and was
  re-sent in full.
- _try_fresh_final: refuse the fresh-final route once a head chunk is sealed.
  It replaces every tracked preview with one message, which only holds the
  whole answer on a single-message turn. After a split it deleted the sealed
  heads while sending just the tail, so the complete reply was still lost --
  on Telegram, the default finalize route and the shape #78541 reports.
- Set _turn_split_delivery at seal time rather than after the tail send, so
  the tail's own finalize sees the split state. The sibling overflow path
  already did this; the divergence is what let fresh-final delete the heads.
- run.py stale-finalize reconciliation: skip the in-place edit on a split
  delivery. message_id is only the LAST chunk there, so editing it with the
  complete response repeated every sealed head's text inside the tail
  message. Fall through to the normal final send.

Also drop a dead `or "".join(chunks)` fallback (all growth funnels through
_append_accumulated, so the ledger is never empty at that call site, and joined
chunks carry injected fence markers that could never match final_response), and
document that _record_turn_final_payload intentionally ignores its argument on
split turns.

Tests: four end-to-end cases driving the real overflow-split loop instead of
hand-setting private flags -- complete split still suppresses (no duplicate),
split missing a tail does not suppress, fresh-final keeps sealed heads, and a
flood-controlled final edit after a split stays suppressed. Each was
mutation-checked: reverting any individual fix turns its test red.

The pre-existing gateway-boundary test asserted the recovery *route* (the
reconcile edit) rather than the guarantee. Relaxed to the real contract: either
_run_agent puts the complete text on the wire, or it declines to claim delivery
so the caller's normal final send does.

Co-authored-by: HexLab98 <liruixinch@outlook.com>
2026-08-06 02:54:32 +05:30
HexLab98 a2ca5c2a78 test(gateway): cover payload-less split-delivery final-send swallow
Add unit and GatewayRunner boundary coverage for the #78541 shape where
final_content_delivered is set via split delivery with no recorded
payload.
2026-08-06 02:54:32 +05:30
Teknium 5aa798fecc feat(skills): actual-setup optional skill + provider docs
- optional-skills/devops/actual-setup: field-tested setup skill contributed
  by shl0ms, updated for the first-class 'actual' provider (the original
  targeted a custom-provider config that now collides with the built-in name)
- docs: providers.md section + tables, environment-variables.md, quickstart.md
- tests/skills: frontmatter + first-class-provider conformance checks
2026-08-05 14:08:32 -07:00
Teknium e79f16cab6 feat(providers): env-var metadata, config-driven local no-auth, reasoning-effort clamp for Actual
- config_defaults: ACTUAL_API_KEY / ACTUAL_BASE_URL entries (setup wizard + hermes tools)
- codex transport: clamp xhigh->high, ultra->max for provider=actual (SGLang/vLLM
  backends reject the wider values with a wrapped HTTP 400)
- chat_completion_helpers: thread provider into Responses build_kwargs
- tests: transport clamp + config-driven local no-auth regression
2026-08-05 14:08:32 -07:00
Teknium b6d55a790e fix: adapt Actual provider salvage to current main
- fetch_models(): accept base_url kwarg (interface grew on main since May)
- runtime_provider: config-driven loopback base_url now reaches the local
  no-auth placeholder before the usable-secret gate (added on main in the
  interim, would otherwise AuthError on keyless local setups)
- test: fetch is now called with base_url by the generic live-fetch path
2026-08-05 14:08:32 -07:00
Justin Bennington a9acb400ba feat(providers): add Actual Computer inference provider 2026-08-05 14:08:32 -07:00
kshitij 241605d1ea fix(compression): durable-sync the prune runway on model switch + fast no-op for incapable stores
Three review follow-ups on the salvaged #79286 commit:

- update_model() zeroed the in-memory prune runway but left the durable
  model_config copy stale, breaking the method's own durable-sync
  discipline (the strike reset three lines above keeps its durable copy
  in sync). A restart after a model switch resurrected a runway
  computed under the old model's trigger sizes. New
  _clear_durable_proactive_prune_rearm() removes the persisted key via
  patch_session_model_config() without touching the transcript.

- The archive_and_compact capability check ran AFTER the expensive
  3-pass prune scan, so a duck-typed session store lacking the method
  paid the full scan on every eligible iteration forever with pruning
  permanently no-opping. Hoist it above the scan (all in-tree stores
  pass a real SessionDB; this only affects third-party stores).

- _load_proactive_prune_rearm_tokens now uses the shared
  get_session_model_config_value() accessor instead of inlining a 5th
  copy of the model_config JSON parse, matching its sibling loaders'
  typed-accessor pattern.

Also documents why the rotation-publish-failure branch restores only
the runway field rather than the full attempt snapshot.

Tests: model-switch durable clear, patch_session_model_config
merge/delete/no-op, and a guard proving incapable stores skip the scan.
2026-08-06 02:22:08 +05:30
Ryder Freeman bf6a210ab9 fix(cache): make proactive pruning durable and cache-aware 2026-08-06 02:22:08 +05:30
Jeffrey Quesnelle 6564f319a6
Merge pull request #69416 from afourniernv/feat/hermes-relay-install-activation-metrics
feat(observability): add Relay active install metrics
2026-08-05 14:09:25 -04:00
Jeffrey Quesnelle edf0a7e14b
Merge pull request #68978 from afourniernv/feat/hermes-relay-client-dimensions
feat(observability): add Relay client resource metrics
2026-08-05 14:02:28 -04:00
Jeffrey Quesnelle 0531aad55d
Merge pull request #68883 from afourniernv/feat/hermes-relay-skill-metrics
feat(observability): aggregate bounded skill metrics
2026-08-05 13:20:57 -04:00
brooklyn! 69bedb7bed
fix(desktop): stop reporting false failure on successful backend updates (#79513)
* feat(update): emit an action-scoped terminal receipt from hermes update

The dashboard now mints an action_id per backend update, hands it to the
spawned `hermes update` via HERMES_ACTION_ID, and reuses an in-flight
update action instead of spawning a duplicate. The updater prints a
bounded `=== hermes-update completed <id> ===` receipt on every success
path — normal, zip, dependency-repair, and the no-op "Already up to
date!" path that previously ended with no terminal marker at all
(#58764) — so the Desktop can prove completion across the dashboard
restart boundary instead of guessing from stale log text.

Co-authored-by: Vitor Cepeda Lopes <vitor@vitorcepedalopes.com>
Co-authored-by: doncazper <caztronics@yahoo.com>

* fix(desktop): make remote backend updates terminal-state driven

Remote backend updates failed with "Backend update failed." on nearly
every run: applyBackendUpdate() polled for only 30×1.5s ≈ 45s, then
read exit_code null off the still-running action and called it a
failure. Real updates (backup + uv sync + npm install + vite build)
routinely run longer, and the no-op "Already up to date" path never
restarted the gateway so the old return-check timed out too.

A still-running, reachable action is now never converted into failure
by an elapsed budget — only a nonzero exit is. The apply loop keeps one
in-flight promise, tolerates reconnects during the dashboard restart
without extending the fixed six-minute deadline forever, and confirms
success by the action-specific receipt that survives the restart,
falling back to proving the requested commit / up-to-date check for
older backends without action_id support. Inconclusive completion
fails closed.

Fixes #47359
Fixes #58764

Co-authored-by: Vitor Cepeda Lopes <vitor@vitorcepedalopes.com>
Co-authored-by: Mark Vlcek <markvlcek@gmail.com>
Co-authored-by: doncazper <caztronics@yahoo.com>

---------

Co-authored-by: Vitor Cepeda Lopes <vitor@vitorcepedalopes.com>
Co-authored-by: doncazper <caztronics@yahoo.com>
Co-authored-by: Mark Vlcek <markvlcek@gmail.com>
2026-08-05 10:42:02 -06:00
brooklyn! 64646dda56
Hermes can read the in-app browser (#79482)
* feat(agent): read_preview — the desktop-gated tool that reads the in-app browser

The agent could open the preview pane (open_preview) and read the embedded
terminal (read_terminal), but the browser it had just opened was a black box —
'what does this page say?' had no answer. read_preview mirrors read_terminal
end to end: HERMES_DESKTOP-gated via check_fn (zero schema footprint outside
the GUI), dispatched through the same agent callback pattern, windowed with
start/count so a long page pages instead of flooding context.

* feat(gateway): preview.read blocking bridge

Same lifecycle as terminal.read: the tool blocks on preview.read.request, the
renderer answers preview.read.respond (allow_expired — a slow page extraction
losing the 45s race must not surface a raw 4009), and a timeout emits
preview.read.expire so late answers resolve quietly.

* feat(desktop): the renderer serializes the active preview tab for the agent

preview-reader.ts is the preview analog of the terminal's buffer registry: the
URL pane registers a page reader (webview executeJavaScript → title + visible
innerText) keyed by tab id; readActivePreview resolves the ACTIVE tab, windows
the text (24k cap per read), and answers file/artifact tabs with identity plus
a note pointing at the tool that reads that content directly. The gateway
event handler answers preview.read.request beside terminal.read.request.
2026-08-05 16:35:00 +00:00
Brooklyn Nicholson 950b55d4d7 feat(update): emit an action-scoped terminal receipt from hermes update
The dashboard now mints an action_id per backend update, hands it to the
spawned `hermes update` via HERMES_ACTION_ID, and reuses an in-flight
update action instead of spawning a duplicate. The updater prints a
bounded `=== hermes-update completed <id> ===` receipt on every success
path — normal, zip, dependency-repair, and the no-op "Already up to
date!" path that previously ended with no terminal marker at all
(#58764) — so the Desktop can prove completion across the dashboard
restart boundary instead of guessing from stale log text.

Co-authored-by: Vitor Cepeda Lopes <vitor@vitorcepedalopes.com>
Co-authored-by: doncazper <caztronics@yahoo.com>
2026-08-05 10:34:18 -06:00
brooklyn! 7e16241825
feat(wake): hands-free wake word for remote desktop via client mic streaming (#79491)
* feat(wake): client-capture wake word for remote desktop

Remote headless backends have no PortAudio mic, so "hey hermes" fails even
when openWakeWord is installed. Let the desktop stream 16 kHz int16 PCM via
wake.feed while detection stays server-side.

- wake_word.capture: auto|local|client (+ GUI client_capture prefer)
- WakeWordDetector external_audio queue + feed_audio API
- wake.feed RPC; wake.start/status report capture + frame_length
- Desktop getUserMedia feeder; stop on wake.detected, re-arm after voice
- Docs + unit tests (26 pass in tests/tools/test_wake_word.py)

* fix(wake): address review on client-capture re-arm and feed queue

- wake.status reports effective capture from the armed detector (client vs
  local), plus frame_length/sample_rate; GUI status probes prefer client
- Gateway test doubles accept external_audio on start_listening
- Desktop PCM feeder uses a bounded ordered queue instead of dropping frames
  while a wake.feed RPC is in flight
- /wake on and status/re-arm paths pass client_capture so remote reattach works

* fix(wake): auto capture keeps the backend mic when one exists

With capture:auto the desktop always preferred client streaming, so a local
desktop with a working backend mic silently switched from PortAudio to
getUserMedia default-device — dropping wake_word.input_device selection
(#74363). A ready backend input now wins under auto; client capture is the
fallback for a preferring surface on a mic-less backend, and capture:client
still forces streaming.

Also removes the dead auto branch (both arms returned local) and lets the
client-feed test skip cleanly when numpy is absent.

* perf(desktop): coalesce wake.feed frames

Sending one 80 ms frame per RPC is ~12.5 gateway calls/s for as long as the
ear is armed. Drain up to 4 queued frames into a single wake.feed payload
(backend feed() already splits long buffers into engine frames) — ~3 RPCs/s
steady-state. Fix the wake.feed size-cap comment (64000 bytes = 2 s, not
0.5 s).

* docs(config): document wake_word.capture in cli-config.yaml.example

---------

Co-authored-by: Andrew <drew@kainotomic.com>
2026-08-05 10:32:38 -06:00
Brooklyn Nicholson 60808dcf72 fix(wake): auto capture keeps the backend mic when one exists
With capture:auto the desktop always preferred client streaming, so a local
desktop with a working backend mic silently switched from PortAudio to
getUserMedia default-device — dropping wake_word.input_device selection
(#74363). A ready backend input now wins under auto; client capture is the
fallback for a preferring surface on a mic-less backend, and capture:client
still forces streaming.

Also removes the dead auto branch (both arms returned local) and lets the
client-feed test skip cleanly when numpy is absent.
2026-08-05 10:06:49 -06:00
Andrew d401c27edf fix(wake): address review on client-capture re-arm and feed queue
- wake.status reports effective capture from the armed detector (client vs
  local), plus frame_length/sample_rate; GUI status probes prefer client
- Gateway test doubles accept external_audio on start_listening
- Desktop PCM feeder uses a bounded ordered queue instead of dropping frames
  while a wake.feed RPC is in flight
- /wake on and status/re-arm paths pass client_capture so remote reattach works
2026-08-05 10:03:55 -06:00
Andrew 105fbf6b7d feat(wake): client-capture wake word for remote desktop
Remote headless backends have no PortAudio mic, so "hey hermes" fails even
when openWakeWord is installed. Let the desktop stream 16 kHz int16 PCM via
wake.feed while detection stays server-side.

- wake_word.capture: auto|local|client (+ GUI client_capture prefer)
- WakeWordDetector external_audio queue + feed_audio API
- wake.feed RPC; wake.start/status report capture + frame_length
- Desktop getUserMedia feeder; stop on wake.detected, re-arm after voice
- Docs + unit tests (26 pass in tests/tools/test_wake_word.py)
2026-08-05 10:03:48 -06:00
ethernet eea6044098 feat(desktop): register a Linux launcher entry for `hermes desktop`
On Linux a freshly-built desktop app had no presence in the application
launcher: no Hermes in the KDE/GNOME menu, no icon, nothing to pin. Users
had to hand-write ~/.local/share/applications/hermes.desktop and remember
to reindex the menu caches themselves.

`hermes desktop` now writes that entry itself (best-effort, idempotent,
never blocking a launch), and `hermes uninstall --gui` removes it again.

Both fields that matter are absolute:

- Exec — the launcher runs with a minimal environment and no shell PATH
  customizations, so a bare `hermes desktop` silently fails for anyone
  whose hermes lives in ~/.local/bin or a venv. We resolve the real binary
  via relaunch.resolve_hermes_bin(), falling back to an absolute
  interpreter + `-m hermes_cli.main`.
- Icon — an unqualified name only resolves against an indexed icon theme,
  which we are not in. The spec allows an absolute path, so we point at
  apps/desktop/assets/icon.png in the checkout. No copy is installed: Exec
  already depends on that same tree, so a second copy would add bytes and
  an uninstall step without surviving anything Exec wouldn't.

Menu-cache refresh is tool-gated — update-desktop-database, then
kbuildsycoca6 or kbuildsycoca5 — each only when the binary is actually on
PATH, because most desktops ship none of them and a missing one is not an
error. The entry is only rewritten when its contents change, so a launch
doesn't churn the caches every run.

Verified on NixOS: the generated entry passes desktop-file-validate, a
real kbuildsycoca6 on PATH is invoked with --noincremental, a real
update-desktop-database writes mimeinfo.cache, absent tools are skipped
cleanly, and removal leaves the checkout's icon untouched.
2026-08-05 11:47:05 -04:00
ethernet ee7c614eef fix(ci): follow artifact download redirect without auth
The artifact download URL returns a 302 redirect to a signed blob URL.
urllib sent the Authorization header to the blob, and the blob rejected it
with a 401 error. The download now has two hops. The first hop authenticates
to the API. The second hop follows the redirect without the auth header.

The query runs?event=workflow_call returns nothing for this repository.
GitHub flattens reusable-workflow jobs and their artifacts into the caller
run. The fetch now lists the artifacts on the orchestrator run only. The
dead sub-run enumeration is gone. Two API calls per cycle are gone with it.

The 'artifact statuses updated' reason never appeared. The code updated the
count before the comparison. Now the code compares first and updates after.

The code rejects zip members that contain '..' or start with '/'.

tests/ci/test_live_comment.py is deleted. This repository does not keep
tests for CI infrastructure.
2026-08-05 11:16:18 -04:00
ethernet 1d7d0e41af ci: poll review statuses from artifacts every cycle
The live comment poller got its review statuses from two sources. The first
was the REVIEW_STATUSES environment variable, fixed at the start of the
comment-live job. The second was one ci-timings artifact, downloaded at the
end of the run. Status details (error messages, action_required items)
appeared only after all jobs finished. The job pass/fail results were visible
as each job completed.

Now every status-producing workflow_call uploads a small review-status
artifact when it completes. The poller lists all review-status-* artifacts
from the orchestrator run and its workflow_call runs every cycle. It
downloads each artifact and merges the statuses into the comment. A status
appears as soon as its job finishes.

Changes:
- live_comment.py: _fetch_artifact_statuses became fetch_all_review_statuses.
  The new function lists the artifacts via the API, downloads each one, and
  parses it. Removed the review_statuses_json parameter, the
  --review-statuses-file argument, and the subprocess import.
- ci.yml: removed the REVIEW_STATUSES environment variable, the inline Python
  merger, and the --review-statuses-file argument. Renamed the
  ci-timings-review-status artifact to review-status-ci-timings.
- Eight workflow_call files: added a step that writes review-status.json and
  uploads it as an artifact after each review_status output.
- test_live_comment.py: added tests for _parse_status_file and
  _merge_statuses.
2026-08-05 11:16:18 -04:00
PRATHAMESH75 49d8a155c4 fix(terminal): skip binary content on the referenced-script remote-read fallback (#77703)
The gateway terminal guard crashed with 'ValueError: embedded null byte'
(command never ran, exit_code -1) when a command invoked an ELF binary by
full path. _read_referenced_script correctly rejects the binary locally
(NUL in first chunk), but the read_remote_script fallback
(_read_script_in_env) then re-read the SAME file's bytes without a NUL
guard, decoded them, and fed machine code back into the scanner, which
re-tokenized it into a bogus NUL-bearing path and crashed at os.open.

- _read_script_in_env: skip content containing a NUL byte on both the
  local-read and remote-cat branches (mirrors _read_referenced_script:
  a binary is nothing to scan), so binary never re-enters the guard.
- _read_referenced_script: tolerate ValueError from os.open on a
  NUL-in-path, alongside the existing OSError guard, so the guard can
  never crash the terminal tool regardless of input.

Extends the #76762 NUL-safety fix (local path only) to the gateway's
remote-read fallback path.
2026-08-05 20:34:59 +05:30
brooklyn! ae6c2e57e1
Merge pull request #78682 from NousResearch/bb/win-8dot3-profile-paths
fix(install): Windows install completes on profiles with a space, dot, or accent in the username
2026-08-05 08:21:23 -06:00
kshitij 1be70d6354 fix: join heartbeat thread in finally + add error-path test
Add activity_hb.join(timeout=2.0) after activity_hb_stop.set() in
direct_api_call's finally block so the heartbeat thread is deterministically
stopped before client teardown. Add test verifying no stray _touch_activity
fires after direct_api_call raises an exception.

Follow-up to PR #78548 by @xxxigm.
2026-08-05 14:00:25 +05:30
xxxigm d55bc063f1 fix(delegation): keep subagents alive during slow model waits
Top-level delegate_task runs in the background, and the 450s progress-stall
monitor only sees api_call_count / tool / last_activity_ts. Subagents use
non-streaming direct_api_call, which previously touched activity once and then
went silent — so a healthy local GGUF / long-prefill wait looked frozen and
was interrupted around ~450s as "Operation interrupted: waiting for model
response", even when child_timeout_seconds was raised. Refresh activity while
the inline request is open, and treat last_activity_ts advances as sync
heartbeat progress too.
2026-08-05 14:00:25 +05:30
Ryder Freeman 84e93ffefb fix(state): stop delegate/tool children corrupting compression lineage
get_compression_lineage's forward walk accepted any non-branch child as
the compression continuation. Delegate subagent rows (_delegate_from)
and tool-tagged rows (source=tool) created before the real continuation
were picked as the lineage successor, so the lineage — and session .md
export built on it — followed a subagent's transcript instead of the
actual conversation continuation.

Rename _is_branch_child_row to _is_explicit_fork_child_row, treat
_delegate_from and source=tool rows as explicit forks alongside
_branched_from, and require _is_compression_child_row in the forward
walk instead of merely excluding branches.

Sliced from PR #79024 by @RyderFreeman4Logos (the cache-scope portion
of that PR is tracked separately in #79017).
2026-08-05 13:50:26 +05:30
burak33bb c4f3d5a313 fix(agent): prevent historical steer replay 2026-08-05 13:00:47 +05:30
joaomarcos 34c3f06f91 fix(cache): scope prompt_cache_key by session to stop cross-session bucket sharing
Cherry-picked from PR #78959 by @JoaoMarcos44 with authorship preserved.
Follow-up: hoist _cache_scope_from_session_id(session_id) to a local in
build_kwargs so it's computed once instead of 4 times per call.

Closes #78941. Closes #79012. Closes #79013. Closes #79014. Closes #79015.

Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
2026-08-05 12:42:46 +05:30
briandevans 63c0bb694e fix(cli): correct the skin_cmd fallback comment to match the actual read path
_skin_set has no try/except around yaml.safe_load, so invalid YAML raises
and aborts the command. The {} fallback comes only from safe_load()
returning None on a zero-length file — which is exactly the state a torn,
unsynced write leaves behind, so the data-loss chain is unchanged.
2026-08-05 11:59:33 +05:30
briandevans 649ce1f811 fix(cli): make profile.yaml and skin writes atomic to stop silent field loss
`write_profile_meta` and `hermes skin set` are both read-modify-write
helpers that rewrite a user-visible YAML file with a bare truncating
write, bypassing `utils.atomic_yaml_write` — the shared helper whose
docstring states that "every destructive file rewrite in the codebase
shares one implementation".

Both read halves swallow a parse error and fall back to `{}`, so a
truncated file is not transient corruption. The next call reads `{}` and
silently, permanently drops every field the caller did not explicitly
pass:

* `write_profile_meta` promises "unspecified fields preserve existing
  values". After an interrupted write, a follow-up call that only sets
  `description_auto` erases the profile's `description` — it vanishes
  from `hermes profile list` and never comes back.
* `_skin_set` exists so that "changing one token never disturbs the rest
  of the look". `path.write_text(...)` neither fsyncs nor swaps
  atomically, so a crash or power loss can leave `<skin>.yaml`
  zero-length; the next tweak then rewrites from empty and the whole
  palette is gone. The gateway's skin watcher repaints live surfaces
  from this file within ~1s, so a half-written file is observable.

Routing both through `atomic_yaml_write` gives temp file + fsync +
`atomic_replace`, which also preserves a symlinked target (GitHub
#16743) and restores owner/mode, and emits emoji descriptions as real
UTF-8 instead of `\UXXXXXXXX` escapes (GitHub #51356).

Supersedes #51808, which fixed the unicode-escaping symptom alone by
adding `allow_unicode=True` to the same `yaml.safe_dump` call.
2026-08-05 11:59:33 +05:30
golldyck 652ebc5899 fix(console): handle string SystemExit code in _capture_output
A dispatched console handler that calls sys.exit("message") or
raise SystemExit("message") sets exc.code to a string. int(exc.code or 0)
then raises ValueError, which is not a ConsoleCommandError, so it escapes
execute()'s handler and crashes the local REPL on an ordinary user mistake
(e.g. removing a credential that does not exist). Treat a string exit code
as a status-1 failure carrying that message.
2026-08-05 11:48:03 +05:30
Teknium 42e92c9c09 fix(git): kill the whole probe process tree on timeout (port of openai/codex#36793)
Timing out a bounded git probe must not leave helper descendants
(credential helpers, git-remote-https, hook children) running after the
probe fails open. bounded_git_probe now spawns the child in its own
process group on POSIX (process_group=0), and _kill_git_process_tree
signals the whole group with os.killpg — gated on the child actually
leading its own group (pgid == pid), so a shared-group spawn can never
take down unrelated processes. Windows keeps the existing taskkill /T /F
tree kill.

Proven live on main: a fake git that forks a 300s descendant left the
descendant running after the probe timeout; with the fix the descendant
dies with the launcher. Fast path and fail-open contract unchanged.

Port of openai/codex#36793 (Terminate timed-out Git process trees).
2026-08-04 17:33:36 -07:00
Alex Fournier 806c2b1fdc Merge updated client resource metrics into active-install metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-04 15:10:41 -07:00
Alex Fournier e7eaae2bd3 Merge latest skill metrics into client resource metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-04 15:07:34 -07:00
Alex Fournier 451a078a50 Merge latest origin/main into skill metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-04 15:06:36 -07:00
ethernet 3d9ec4d62e test(install): prove updating from a release reaches this commit
Nothing covered the update path, which is the worst thing to break: a broken
updater strands users on the version that cannot fix itself. `hermes update`
alone is ~2000 lines (hermes_cli/update_cmd.py) and had no end-to-end test.

tests/install/install-update-e2e.sh installs a genuine earlier Hermes through
the real one-liner (curl -fsSL https://…/install.sh | bash, served by
dev-sandbox's MITM proxy at the canonical URL, cloning "github.com" through the
upload-pack shim), which really installs uv, a managed Python, Node and the
venv. It then applies ONE update route and requires the checkout to land on this
commit with `hermes --version` still working -- so a pass means the venv and
entry point survived, not merely that git moved.

One route per run, each on a sandbox built from scratch. Sharing one install
across routes -- or rewinding with `git reset --hard` between them -- leaves the
second route running against a tree the first already updated (same venv, same
console script, same __pycache__), which is not the state any real user is in: a
route could pass only because its predecessor did the work, and a failure in the
first left the second exercising something undefined.

--install-ref chooses what to install first, so this covers "update from an
older release", not just from the tip. Installer flags are probed against the
target rather than assumed, because releases from months back predate flags
current Hermes takes for granted: --skip-browser is read out of that ref's own
install.sh, and `--yes` is asked of the installed `hermes update --help` (the
update subcommand has lived in main.py, subcommands/update.py and update_cmd.py
across the tags we sample, so a static parse rots silently -- and did). Without
those probes, old releases die on "Unknown option: --skip-browser" and
"unrecognized arguments: --yes" before doing any work.

Installer output is streamed through tee rather than captured: a real install of
uv, Python, Node and the venv IS the substance of this test, so it belongs in
the job log, not only in an artifact. pipefail keeps the installer's exit status
rather than tee's, so a failed install cannot look like a pass. The sandbox's
own proxy log is printed in full on failure, since a rejected TLS handshake
explains a failure that otherwise reads as a bare `curl: (35)`.

Deliberately reuses dev-sandbox rather than adding a second harness. An earlier
draft rewrote install.sh's hardcoded URLs with insteadOf and ran it against the
host; that tested the installer LESS faithfully (bash install.sh instead of the
real one-liner, host libs instead of a clean machine, ssh disabled to keep a
failed rewrite from reaching real GitHub) while duplicating a fake Internet we
already have.

Shell, not pytest, so scripts/run_tests.sh and run_tests_parallel.py stay
untouched: a pytest version needed an entry in the former's `env -i` credential
allowlist and a _SKIP_PARTS exclusion in the latter, and every meaningful line
was a command run inside the sandbox anyway.

Two guards, both earned during bring-up. It prefers the `sandbox` wrapper and
falls back to the raw script only when bwrap is on PATH (under Nix the wrapper
supplies the PATH and DEV_SANDBOX_* vars, so the bare script exits 127). And it
refuses to run on a dirty worktree: every dev-sandbox invocation re-derives fake
main from the working copy, so uncommitted changes move the update target
between the call that installs and the call that verifies -- a failure that
looks like a broken updater but is a moving reference.
2026-08-04 17:36:26 -04:00
Brooklyn Nicholson 34833303f5 ci(install): actually run the PowerShell installer tests
scripts/tests/ has held three PowerShell suites that no workflow ever
invoked -- there is no Windows runner in CI, so they have been inert since
they landed. A regression test nothing executes is worse than none: it
reads as coverage.

Adds a windows-latest job, gated on a new `installer` lane so it only fires
for PRs touching install.ps1 or its tests. The 8.3 suite runs under both
pwsh 7 and Windows PowerShell 5.1, since install.ps1 arrives via `irm | iex`
into whichever shell the user already has and 5.1 is what ships with Windows.

Only the 8.3 suite is wired up. The other two fail on main today for
unrelated reasons; they can join once they are fixed.
2026-08-04 15:34:32 -06:00
brooklyn! 43717123ca
fix(models): a model id missing its vendor prefix says so instead of 404ing (#78856)
Selecting an NVIDIA NIM model whose id reached config without the nvidia/
prefix produced a bare "HTTP 404: 404 page not found" — retried three times,
never naming the model. It reads exactly like an outage or an auth failure,
which is where the Discord thread spent its time before the id was spotted.

normalize_model_for_provider() had no branch for nvidia, so a bare id passed
straight through to the API. Repair it from the provider's curated catalogue:
a bare name that matches exactly one entry modulo the prefix gets it back.
That's a lookup, not a guess — build.nvidia.com also fronts local NIM
containers and third-party models, and anything absent from the catalogue is
left alone. Because the repair runs on every runtime setup, an already-broken
config self-heals on the next turn and prints what it changed.

If a bare id still reaches the wire, the 404 now explains itself. The
classifier consults the same catalogue: a prefix-less id the provider only
serves as vendor/model is a deterministic failure, so it classifies as
model_not_found instead of burning three retries on a retryable "unknown",
and the error trace names the id to use.

Fixes #78796
2026-08-04 19:35:57 +00:00
brooklyn! d98287fe3c
Merge pull request #78831 from NousResearch/bb/session-read-state
Sessions track read/unread
2026-08-04 13:20:26 -06:00
Jeffrey Quesnelle f40fbcf409
Merge pull request #68882 from afourniernv/feat/hermes-relay-tool-metrics
feat(observability): aggregate bounded tool metrics
2026-08-04 15:04:29 -04:00
Brooklyn Nicholson ec0c8d9c20 feat(state): sessions carry read/unread state
Adds a last_read_at watermark to the sessions table so surfaces (CLI,
TUI, desktop) can badge unread conversations. Read state derives from
the watermark vs latest activity, so new messages flip a conversation
back to unread with zero writes on the message path. NULL means never
tracked, so shipping the column doesn't badge pre-existing history.

set_session_read() stamps the whole compression lineage, matching the
archive/pin semantics; list_sessions_rich() rows carry a derived
`unread` key. DB layer only — no surface exposes it yet.
2026-08-04 12:32:27 -06:00
brooklyn! b3e45a3d46
Discord drops an empty outbound message instead of sending it (#78815)
* fix(discord): reject empty outbound messages

* test(discord): cover empty final reply backfill state

Missed-message backfill decides what to replay from discord_messages, so
a dropped final reply must be recorded as failed by the new guard the
same way the exception path records one — otherwise the reply is both
never sent and never retried.

Co-authored-by: Jony <619963502@qq.com>

* chore: map 619963502@qq.com to zyz619963502zyz for PR #73449 salvage

---------

Co-authored-by: Jony <619963502@qq.com>
2026-08-04 18:23:04 +00:00
Alex Fournier 44897dd6f0 Merge origin/main into feat/hermes-relay-client-dimensions
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-04 11:19:43 -07:00
Brooklyn Nicholson 9712b8f0cc test: teach the hand-rolled fake pools the failure_reason kwarg
Three fakes pin mark_exhausted_and_rotate's signature explicitly and broke on
the new argument. They now assert it rather than just tolerate it — the xAI
spending-limit case is exactly the billing-403 this fixes, so it should be
pinning `failure_reason == "billing"`.
2026-08-04 23:33:39 +05:30
Brooklyn Nicholson 9cd0338688 fix(credential-pool): bench a billing 403 fully, even as the sole key
The sole-credential cooldown sized the bench from the raw HTTP status, but
403 is overloaded: error_classifier maps OpenRouter's "key limit exceeded"
and xAI's spending-limit block to FailoverReason.billing, while an edge
throttle with the same status is transient. Only 402 was excluded from the
short cooldown, so a spent account on a single key retried every 60 seconds
and re-failed forever.

Thread the classified reason from recover_with_credential_pool through
mark_exhausted_and_rotate to _exhausted_ttl. Billing keeps the full bench
regardless of status; everything else transient still recovers in 60s. The
verdict is stored on the entry (_EXTRA_KEYS, so it persists to auth.json) —
without that a restart would re-read a bare 403 and downgrade the bench.

Tests: sole billing-403 stays benched, survives reload, unclassified 403
still recovers; call-site coverage that the reason actually reaches the pool.
Three existing kwargs assertions updated for the new argument.
2026-08-04 23:33:39 +05:30
kshitij d1eb08fcf3 fix: thread sole_credential into next_available_at sibling site
next_available_at() was computing the full 1-hour TTL for a sole
credential on a 429, contradicting the 60s cooldown in _available_entries.
The fallback restore gate (agent_runtime_helpers) uses next_available_at
to decide when to switch back from fallback to primary — so the agent
stayed on fallback for an hour instead of ~60s.

Add sole_credential computation in next_available_at mirroring
_available_entries, and a test verifying the short cooldown propagates.
2026-08-04 23:33:39 +05:30
A.Alzaro dcd7504349 fix(credential-pool): short cooldown for sole credential on transient throttle
A pool with only one usable (non-DEAD) credential has nothing to rotate to.
On a transient throttle (429 rate-limit, 403 edge-throttle, 5xx) the offending
key was benched for a full hour (EXHAUSTED_TTL_429/DEFAULT), so single-key /
no-fallback setups got an hour of hard failures for a throttle that resets in
seconds. The pool already special-cases 401 to recover quickly for single-key
setups; extend that to transient throttles when the credential is the sole
non-DEAD entry. 402 (billing/quota) keeps the full bench — a quick retry can't
help. Provider-supplied reset_at still overrides.

Adds tests covering sole 429/403 recovery, 402 full-bench, and multi-key
(no early recovery).
2026-08-04 23:33:39 +05:30
brooklyn! 97641a820d
fix(debug): say where a client-side log lives instead of "(file not found)" (#78687)
hermes debug share runs on the backend. A desktop app connected to a
remote, docker, or SSH backend writes desktop.log on the client machine,
so the bundle can never contain it — and the report rendered that as a
bare "(file not found)", which reads as "the app logged nothing" and
sends triage after a client-side bug it cannot see.

Name the writer and the path to collect by hand. Backend-written logs
are unchanged, a present desktop.log is still captured, and an empty one
still reports "(file empty)" — the app ran and logged nothing is a
different fact from the file being on another host.
2026-08-04 11:53:32 -06:00
Jeongseok Kang 2d70f56327
fix(agent): adopt .env credential/base-url edits at the turn boundary (#67843)
* fix(agent): adopt .env credential/base-url edits at the turn boundary

A Settings save (desktop PUT /api/env, hermes setup) updates .env and
the saving process's os.environ, but a live session worker keeps the
base_url/api_key captured at agent init until restart — an open chat
silently kept calling the old endpoint (e.g. a local-server key sent to
api.openai.com, failing with an opaque 401).

Add AIAgent._try_refresh_env_client_credentials(), called at the start
of each conversation turn: re-resolve the provider's env-sourced
credentials (load_env() is mtime-memoized, so an unchanged file costs
one stat()) and rebuild the client via the existing
_replace_primary_openai_client machinery when the user edited them.

The refresh reacts only to env edits — resolved values changed since
the last look — never to mere divergence from the agent's current
values: credential-pool rotation and failover legitimately move the
session off the env credential, and stomping those back would flap.
Config model.base_url / pool custom endpoints keep precedence: edits
are only adopted while the session still runs on the registry default
or the previously-seen env value.

Lift _get_env_prefer_dotenv out of _seed_from_env to module level
(get_env_prefer_dotenv) so both the pool seeder and the per-turn
refresh share the same .env-over-os.environ resolution, including the
op:// indirection handling.

Fixes #67821

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(agent): address sweeper review on env credential refresh

- Cover named custom providers (#67935): provider="custom" has no
  PROVIDER_REGISTRY entry, so resolve the config block's key_env through
  the same lookup the runtime resolver uses.
- Make the edit baseline transactional: a failed client rebuild rolls the
  agent back and leaves _env_creds_seen un-advanced so the unchanged edit
  is retried next turn.
- Recompute route-derived TLS material and default headers on a base-url
  change, via a _reapply_route_client_config helper shared with
  credential-pool rotation so the two paths cannot drift.
- Rebase onto main: get_env_prefer_dotenv keeps the scoped _get_secret
  semantics from the profile-isolation fix (no raw os.environ reads).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: map jskang@lablup.com to rapsealk

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
2026-08-04 17:53:17 +00:00