Bundled backends/platforms/providers load without a plugins.enabled
entry ('must just work'), but plugins.manage reported them 'not
enabled' — clients rendered running plugins with an OFF switch.
Surface the truthful default; explicit disable still wins.
plugins.manage list rows now carry the canonical registry key and a
portable flag (Agent Plugins v1 plugin.json packages), and toggles
address the key — bare names collide across category dirs
(image_gen/fal vs video_gen/fal), so name-addressed toggles flipped
both. Portable packages' in-memory MCP servers also fold into
enabled_mcp_server_names(); without that their tools registered with
the MCP runtime but never reached the model's schema.
Titling fired on the first response, so a session sat unnamed for the whole
opening turn - p50 151s, p90 1212s across real sessions, because a turn is
tool calls, not one round-trip. A turn that failed or was interrupted never
got a title at all. Four surfaces each carried their own copy of the call.
Move it into the shared turn prologue and split it in two: a deterministic
title derived from the user's opening message, written inline before the
model runs, then one small-model call that upgrades it. The response is
constrained to a JSON object so there is no preamble to strip, and control
wrappers are stripped rather than refused, so a slash command titles as
what the user asked for instead of the command itself.
* fix(model-picker): serve cached custom-provider catalog on no-probe opens
#58183 stopped GUI picker opens from live-probing saved custom
OpenAI-compatible endpoints so a stopped local server could not stall the
picker. It gated the whole discovery block, not just the network call, so
`cached_fetch_api_models()` was skipped too — and with it the catalog an
earlier probe had already written to `provider_models_cache.json`.
A custom endpoint that is not the current provider therefore renders only
the models named in its config entry. A local server with 8 models loaded
shows the 1 model that was saved when the provider was first added, on
every picker open, while an explicit Refresh shows all 8.
Add `cache_only` to `cached_fetch_api_models()`: answer from disk within
the existing stale-serve window, never fetch, never revalidate off-thread,
return None on a miss. Split the three call sites in
`list_authenticated_providers()` into what the user's config permits
(`discover_models`, an explicit `models:` allowlist) and how we may obtain
it, so suppressing the probe now downgrades to a cached read instead of
skipping discovery outright. `discover_models: false` still pins, and a
cache hit no longer writes back to config since the probe that populated
it already did.
The latency win stands: a cold cache is a miss, so picker opens against
offline endpoints still make zero network calls.
* test(model-picker): pin the cached-catalog contract for no-probe opens
Cover both halves of the invariant, since fixing either one alone
reintroduces a bug the other guards against.
`cache_only` on `cached_fetch_api_models()`: a fresh entry and an entry
past its TTL but inside the stale-serve window both serve; an entry beyond
that window, an empty cache, rotated credentials, `force_refresh`, and a
missing base_url are all misses — and none of them fetch or spawn a
background revalidation.
`list_authenticated_providers()` on the GUI path: a non-current endpoint
with a warm cache reports its full catalog across all three provider
shapes (`custom_providers`, `providers:`, bare `provider: custom`) with no
live fetch attempted. A cold cache keeps the configured list and still
makes no network call, which is the #58183 guarantee. `discover_models:
false` keeps pinning, and a cache hit does not write back to config.
* fix: persist discovered custom-provider models in the hermes model flow
The `hermes model` named-custom-provider flow (_model_flow_named_custom)
probes the endpoint and shows the full catalog, but never persists it to the
entry's `models:` list. No-probe surfaces (dashboard, desktop, ACP) call
build_models_payload(..., probe_custom_providers=False) and only render the
configured `models:` list, so a provider added via `hermes model` collapses
to the single `model:` default everywhere except the CLI. OpenAI-compatible
providers added via a probing picker already benefit from
_save_discovered_models_to_config; the CLI flow did not.
Persist the live catalog after a successful probe, mirroring the picker path
in model_switch.py. A failed save is non-fatal.
* fix(model-picker): stop an auto-saved catalog pinning a keyless endpoint
The cached-catalog read added for no-probe picker opens still sat behind
the no-key discovery gate, so it never reached the shape that motivated
it: a keyless local model server.
`bool(api_key) or not has_explicit_models` is a network-cost gate. It
exists so Hermes does not probe an endpoint it cannot authenticate to
when that endpoint already declares its catalog (5f00f36ba, 1039e90b5).
Reading a catalog an earlier probe already paid for costs nothing, so
the gate belongs on the probe, not on discovery as a whole.
Left on the discovery side it re-pins the endpoint it was meant to
spare. A successful probe calls `_save_discovered_models_to_config()`,
which writes a plain list into `models:` — exactly the shape
`_models_config_is_allowlist()` reads back as an explicit user
allowlist. A keyless server therefore froze on the catalog of its first
probe and could never widen again, which is the "lineup changes after
config was written" case. f66319097 already carved the dict shape out of
this trap for the same reason; the list shape is the other door into it.
Move the clause to `_probe_live` at both custom-endpoint sites. Probe
suppression is unchanged — verified byte-identical to main across the
keyed/keyless x declared/undeclared matrix — and `discover_models: false`
remains the documented way to pin a catalog.
* test(model-picker): cover the keyless auto-save pinning trap
Three tests around the gate move, each failing on the code before it:
- a keyless endpoint carrying an auto-saved `models:` list still reads
its full cached catalog
- the same row, cold cache and probing enabled, still makes zero live
fetches — the network-cost gate the clause exists for
- an end-to-end round trip: persist a probe result via
`_save_discovered_models_to_config()`, reload it, and assert the shape
we wrote does not read back as a user pin
The round-trip test guards the whole chain rather than one branch, so a
future change that makes the saved shape look like an intentional
allowlist fails here even if the gate logic is refactored.
* fix(model-picker): key the custom-endpoint model cache by api_mode
`cached_fetch_api_models()` fingerprints entries with `api_mode`, but no
call site in `list_authenticated_providers()` passed it, so every custom
row resolved to the `api_mode=None` fingerprint. Two rows sharing a
base_url and credential but differing by `api_mode` are deliberately
distinct picker rows — it is part of `group_key` at both sites — yet they
collapsed onto one cache entry.
That was latent while probing was the only way to fill a row: a mismatched
entry was overwritten by the row's own live fetch. Serving that entry
without a probe makes it visible, so an `anthropic_messages` row could
render the catalog an OpenAI-mode row cached against the same URL. The
wire protocols differ (`x-api-key` + `anthropic-version` vs
`Authorization: Bearer`), so those catalogs are not interchangeable.
Persist `api_mode` on the group at both grouping sites — it is already
part of `group_key`, so it is constant across the group — and pass it
into the cache read. Section 3b (bare `provider: custom`) has no
`api_mode` in scope and already reads with the empty-credential
fingerprint, so it is unchanged.
Reported by Copilot review on #81973.
---------
Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com>
Co-authored-by: Navlem <114683850+Navlem@users.noreply.github.com>
Widen of the PowerShell codepage cluster: the autostart registration
subprocess in tools_config.py was the last text=True capture in these
modules still decoding with the locale code page. Standardize on
encoding='utf-8', errors='replace' like the rest of the file (#53428).
A gate runs whatever command the operator configured, so its output is
arbitrary bytes. run_gate captured it with text=True and no encoding, which
decodes with locale.getpreferredencoding() under errors="strict".
One byte the decoder rejects — a test runner's checkmarks or CJK on a
non-UTF-8 Windows console, a stray binary byte anywhere in the stream — kills
subprocess's reader thread. proc.stdout comes back None, the `or ""` fallback
turns that into an empty tail, and an unhandled traceback is dumped to stderr.
The gate's pass/fail verdict still lands on the exit code, but the output tail
is exactly what the retry prompt feeds back so the agent can fix the failure.
With it empty the agent is told a gate failed and given nothing to act on, so
it burns every retry and the goal auto-pauses.
workspace_fingerprint has the same two calls; there a non-ASCII path in
`git status --porcelain` empties the fingerprint, silently disabling the
unchanged-gate skip that exists to stop a stalled agent re-running the same
red suite.
Decode as UTF-8 with errors="replace" — what git and modern toolchains emit,
and what 262 of the repo's 299 text-mode subprocess calls already do.
Complements the cherry-picked contributor fixes and closes out the
remaining sites of the 'missing explicit encoding' bug class, which is
now permanently gated by ruff PLW1514 (enabled repo-wide in
pyproject.toml and enforced by the blocking `ruff check .` step in
.github/workflows/lint.yml):
- tools/memory_tool.py: read MEMORY.md/USER.md via utf-8-sig so a
Notepad BOM never glues U+FEFF onto the first entry (issue #10878,
PR #10888 by @easyvibecoding — strict-decode contract of
_read_raw_checked preserved rather than errors="replace", so
undecodable files still refuse read-modify-write instead of being
lossily rewritten). Regression tests included.
- tools/skills_tool.py: SKILL.md and skill file reads pinned to
utf-8-sig + errors="replace" — deterministic across platforms instead
of the locale fallback proposed in PR #51701 (superseded: falling back
to cp1252/GBK makes the same skill render differently per host); .env
reader aligned with the canonical utf-8-sig dialect in hermes_cli/config.py.
- agent/shell_hooks.py, hermes_cli/main.py, gateway/slash_commands.py:
explicit utf-8 on the remaining fdopen/open text-mode sites flagged by
the AlexFucuson9 sweep series (#56033#56940#65565#66782#66791).
Co-authored-by: easyvibecoding <easyvibecoding@users.noreply.github.com>
Co-authored-by: AlexFucuson9 <AlexFucuson9@users.noreply.github.com>
Co-authored-by: flyingdoubleg <wangzhe00zju@gmail.com>
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Follow-up to the auth.json UTF-8 read fix in this PR. A repo-wide scan for
the same bug class found three more callers that read ~/.hermes/auth.json
via Path.read_text() with no encoding — same Windows cp1252 hazard:
- agent/auxiliary_client.py _read_nous_auth: a non-ASCII byte raised
UnicodeDecodeError, the broad except swallowed it, and Nous silently
stopped being available as the auxiliary (vision/summarization) provider.
- tools/xai_http.py has_xai_credentials: same failure mode — xAI OAuth
silently looked absent on Windows.
- hermes_cli/main.py is_setup_complete: same; has a config.yaml fallback so
the impact is milder, but the read is still wrong.
All three now use read_text(encoding="utf-8-sig"), matching _save_auth_store's
write encoding. A repo-wide grep confirms there are no remaining
json.loads(...read_text()) reads of auth.json without an explicit encoding.
Tests: rewrote the Windows-encoding regression tests to actually exercise the
bug on POSIX too — a new windows_default_encoding fixture forces a no-encoding
read_text() to decode as cp1252 (the Windows default), and _write_utf8 now
emits real non-ASCII UTF-8 bytes (ensure_ascii=False) so the bytes actually
trip cp1252. Verified each test fails when its fix is reverted (including
the two new sibling-reader tests).
The auth store readers (_load_auth_store, _import_codex_cli_tokens, and the
shared Nous store reader) called Path.read_text() with no encoding, so bytes
were decoded with locale.getpreferredencoding() — cp1252 on Windows. The
stores are *written* as UTF-8 (os.fdopen(..., encoding="utf-8")), so any
non-ASCII byte (a CJK or emoji credential label, an accented display name in
OAuth state) raised UnicodeDecodeError on read.
Worst case: _load_auth_store's broad except then copied the file to .corrupt
and returned an empty store, silently wiping every provider credential on the
next launch. The sibling reader at line 2161 already used
read_text(encoding="utf-8"), confirming the omission was unintentional.
Use utf-8-sig (matching the .env handling in config.py) so a BOM from a
Notepad-edited file is tolerated too.
Adds regression tests covering the UTF-8 round-trip with a non-ASCII label,
BOM tolerance, no-corrupt-on-valid-load, and that the readers pass an explicit
encoding (guard against future regressions). Verified the tests fail when the
fix is reverted.
Closes no issue — found via cross-platform code audit (the bug is not in the
issue tracker).
send_cmd._load_hermes_env intentionally reimplements a minimal dotenv
load (no secret-source pulls, no sanitize rewrite, get_hermes_home path
resolution incl. Windows/profile override), so the shared-loader BOM fix
is mirrored in place: utf-8-sig primary read, BOM strip before the
latin-1 stream fallback.
Claude-Session: https://claude.ai/code/session_01JPmJz5u1Bvtw4cCRvRWnYr
utf-8-sig only covers the primary decode. BOM + invalid UTF-8 (e.g.
PowerShell BOM + cp1252 body) forced latin-1, which kept EF BB BF as
part of the first key name and dropped the canonical name. Strip the
BOM before latin-1 decode and load via stream so override= is preserved.
PowerShell 5.1 Set-Content -Encoding UTF8 and Windows Notepad write a
UTF-8 BOM. load_dotenv(encoding="utf-8") kept U+FEFF on the first key
name, so the canonical name was absent from os.environ and Hermes looked
unconfigured with no error. utf-8-sig strips the BOM and is a no-op for
BOM-less UTF-8; latin-1 fallback unchanged.
Prevent UnicodeEncodeError when model text contains U+D800-range
surrogates by sanitizing to U+FFFD before writing to UTF-8 stdout.
Co-authored-by: Cursor <cursoragent@cursor.com>
Ports #68497 forward onto current main per teknium1's review.
input() can raise UnicodeDecodeError when the terminal encoding
cannot decode the byte sequence (e.g. a non-UTF-8 locale, or an
embedded terminal). The prior port targeted hermes_cli/main.py, the
pre-refactor location -- the update pipeline moved to
hermes_cli/update_cmd.py in 927463efcc.
Per review, fixed all three interactive update prompts that call
input() directly, not just the one this originally targeted:
1. Config-migration prompt (update_cmd.py:~3989): extends the existing
except EOFError to also catch UnicodeDecodeError, prints an
actionable 'hermes config migrate' hint, and falls through to the
skip branch (response=n).
2. Stash-restore prompt (_restore_stashed_changes, ~line 971): the raw
input() call here had NO exception guard at all -- not even for
EOFError. Added a try/except covering both EOFError and
UnicodeDecodeError, falling back to the existing skip-restore path
(changes remain safely in git stash, restorable manually).
3. Upstream-remote prompt (_sync_with_upstream_if_needed, ~line 1274):
already caught (EOFError, KeyboardInterrupt) but not
UnicodeDecodeError -- added it to the existing tuple.
Also dropped the incorrect #12884 reference (a TUI sticky-scroll
report, unrelated to this update-encoding issue, per the review).
4 new tests pass covering all three call sites (config-migration prompt
via cmd_update end to end, stash-restore and upstream-remote prompts
via direct unit tests against their own functions), plus an EOFError
sanity test confirming the stash-restore fix doesn't regress that case
either (it had no guard before). 6/6 in the full
tests/hermes_cli/test_update_yes_flag.py file (no regression).
Persist display.personality only; apply rendered text as an in-session
overlay across CLI, TUI config.set, and gateway /personality.
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: EMT5320 <1908937833@qq.com>
Keep agent.system_prompt user-owned; named personalities resolve as an
ephemeral overlay via display.personality.
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: EMT5320 <1908937833@qq.com>
Opt-in via compression.codex_responses_native (default: false). When enabled,
gpt-5.6-family models on the direct OpenAI API (api.openai.com) or a ChatGPT
Codex subscription send context_management=[{type: compaction,
compact_threshold: N}] on Responses requests. OpenAI compacts server-side and
returns an encrypted compaction output item; Hermes captures it into the
existing codex_reasoning_items sidecar and replays it on later turns in place
of the pruned history — inheriting persistence, session replay, the
cross-issuer guard, and the encrypted-replay kill switch with zero new state.
Scope is deliberately hard-gated (agent/native_compaction.py, re-checked per
request): gpt-5.6 family only — gpt-5.1/5.2 fail server-side on the field
(HTTP 500 / stream stall, no structured rejection; live-verified) — and
direct OpenAI/Codex routes only; xAI, GitHub/Copilot, OpenRouter, relays,
and local servers never see the field.
Hermes' local compression stays armed as the fallback owner: the native
threshold is clamped ~8K tokens below the local trigger so the server
compacts first, and a structured provider rejection of context_management
disables native compaction for the session and retries without it
(one-shot guard in TurnRetryState).
Live-verified E2E on api.openai.com/gpt-5.6: server compaction fired at a
4K threshold, checkpoints captured and replayed, recall preserved across
3 turns; gpt-5.1 with the flag enabled stays clean (field never sent).
Direction credit: PR #76950 by @laryhorb explored native Responses
compaction; this is a minimal reimplementation on current main.
Fixes 13 issues found in PR #20774 review:
1. Wiring: engine selection moved from run_agent.py to agent/agent_init.py
(where init_agent lives on current main). Transform hook moved from
run_agent.py to agent/conversation_loop.py (where run_conversation lives).
2. Prompt caching: replace copy.deepcopy with copy-on-write (shallow list
copy + clone only messages that are mutated). Use last_prompt_tokens
from update_from_response instead of re-estimating tokens every call.
System extension injection is idempotent (one-time cache break).
3. Signature mismatch: _message_signature renamed to _content_signature
and now excludes tool_calls/tool_call_id from the hash. This prevents
mismatches when _canonicalize_api_tool_calls re-serializes argument
JSON with sort_keys=True on the API copy.
4. update_model: accepts api_mode parameter (required by agent_init.py).
5. Reconciled with select_context: transform_api_messages is a separate
hook that runs AFTER select_context and sanitization, before
prompt-cache marker placement. Both hooks coexist with clear ordering.
6. Dedup/purge: kept as DCP-specific strategies (different semantics from
ContextCompressor._prune_old_tool_results — DCP deduplicates by
tool+args signature, not by content hash).
7. Removed copy.deepcopy: replaced with shallow list copy + copy-on-write
via _clone_if_needed. Only messages that are actually mutated get
cloned.
8. Removed redundant _ensure_refs call: _match_api_messages_to_refs no
longer calls _ensure_refs (the caller already called it).
9. _message_key still uses index (needed for positional ref assignment),
but _content_signature is cached per id(msg) to avoid re-hashing.
10. _inject_nudge: only injects into user messages, never falls back to
non-user messages (prevents role semantics violations).
11. Memory: _evict_inactive_blocks bounds blocks_by_id to
_MAX_INACTIVE_BLOCKS (50) deactivated blocks.
12. Merged _range_tool_schema and _message_tool_schema into a single
_compress_tool_schema. Merged _handle_range_compress and
_handle_message_compress into _handle_compress.
13. Dropped DCP_CONTEXT_ENGINE_PR_SPEC.md (temporary file, not for tree).
Config defaults kept minimal in hermes_cli/config_defaults.py (only
the keys the engine actually reads, not the full DCP-compatible surface).
Closes#20717
Salvage of PR #17973 by @TKCen (Sebastian Hänisch), re-implemented on
current main to preserve speed/instructions/provider params,
prepare_spoken_text normalization, OPUS_VOICE_PLATFORMS, is_write_denied
path security, microsecond timestamps, and the streaming-TTS gate.
- Split long TTS text into provider-safe chunks instead of truncating
- Pack generated audio against platform upload limits (Discord 10MB,
Telegram 50MB, configurable via tts.delivery_profiles)
- Combine chunks with ffmpeg (OGG/Opus re-encoded, MP3 stream-copied)
- Multi-file delivery when combination fails or would exceed limits
- Remove hard [:4000] truncation from all callers (cli.py, voice.py,
gateway/run.py, gateway/platforms/base.py)
- Gemini TTS raises ValueError instead of silently truncating when
composed prompt exceeds the provider limit
Simplify-code fixes: removed dead all_touched_paths set, added
try/finally for scratch file cleanup on exception, clean error response
on chunk failure instead of leaking stale file_path.
Four fix-forwards from the adversarial post-merge audit of the Aug 7
unreviewed merge batch:
- estop (#81148): is_engaged() now fails SAFE (engaged) on stat errors;
the gateway estop gate lets recognized slash commands and replies owned
by in-flight work (update prompts, clarify, slash-confirm, tool
approvals, running sessions) through instead of consuming them; new
gateway /pause [reason|off] command gives messaging-only operators an
in-band engage/resume path (busy_policy=dispatch so it works mid-run).
- cron monitor mode (#81138): execution-mode invariants (monitor x
no_agent, monitor_script x monitor_url, no_agent-requires-script) now
have ONE owner (_validate_job_mode_invariants) called from BOTH
create_job and update_job, so the create-time invariant can no longer
be silently violated through the update door.
- cron notepad (#81139): remove_job now clears the job's notepad rows
(clear_notepad was dead code -> orphaned KV state forever); clear is
best-effort and no-ops without creating notepad.db.
- delegation batch gate (#81141): template-marker regex narrowed to
multi-word placeholder shapes only (<feature name>, {file_path}) so
generics (Vec<T>), HTML tags, JSON snippets, glob braces and f-string
style no longer reject legitimate batches; duplicate-goal rejection
removed (best-of-N fan-outs are legitimate).
--resume latest resolves the most recent session through the same
workspace-scoped MRU lookup as -c (TUI source first under --tui, with
classic-CLI fallback). --in DIR chdirs before session resolution so the
lookup keys off DIR's workspace, and pins the session there by skipping
the recorded-cwd restore.
Requested by @Jeff9James: hermes --tui --resume latest --in ./dir
- Migrate the missed 6th inline formatter (update_cmd.py backup-size
display) to the shared helper.
- checkpoints._fmt_bytes: plain alias instead of a None-guard wrapper —
every caller feeds ints from checkpoint_manager (all size fields
initialize to 0), so the None path was dead defensive code.
- Drop the fallback= kwarg (zero production callers; '?' default is the
real inherited contract and stays).
- curator_backup + context_references: call format_bytes directly (single
internal call site each, zero external importers — alias was churn
avoidance with nothing to avoid). backup/_format_size and
doctor/_human_bytes keep their aliases (claw.py + tests pin the former;
three call sites use the latter).
- Reshape the loop so the trailing TB return is reachable (no dead line).
- Tests: replace alias-identity assertions (ossified the delegation
mechanism) with behavior-contract equality over a value sweep;
mutation-checked red-green.
- update_cmd parity: byte-identical B-GB vs the old inline loop; gains
the TB tier.
Five modules each carried a private near-identical human-readable byte
formatter (backup._format_size, checkpoints._fmt_bytes,
doctor._human_bytes, context_references._human_bytes,
curator_backup.format_size). Three of them silently topped out at GB and
rendered a 1 TiB value as '1024.0 GB'. All five now alias one shared
format_bytes in hermes_cli/sizefmt.py (sibling of timefmt.py, same
zero-dependency rationale), keeping each module's established local name
so no caller churns.
Deliberately NOT migrated (behavior differs on purpose):
- session_recovery._format_bytes: binary suffixes (KiB/MiB/GiB)
- qqbot chunked_upload.format_size: '100.0 B' one-decimal style, pinned
by its protocol tests
Net -33 production LOC before the new module; parity verified over a
16-value corpus against all five verbatim originals (only divergence:
the TB tier fix). Contract tests mutation-checked red-green.
_format_db_size reimplemented human-readable size formatting two
imports away from backup._format_size, which doctor already leans on
for _QUICK_STATE_FILES. Delegate and keep only the stat-failure wrap.
Sizes now scale units (KB/GB) instead of pinning everything to MB.
hermes doctor already warns when the linked SQLite carries the WAL-reset
bug, but it never said which databases are actually exposed. A database
already in WAL mode on a vulnerable runtime can still corrupt; one on a
rollback journal cannot. Doctor now lists each Hermes-managed database
with its journal mode next to the SQLite version line and marks the WAL
ones as exposed when the runtime is vulnerable.
The probe reads the 20-byte file header and checks byte 18 (2 = WAL,
1 = rollback journal). It deliberately avoids the SQLite engine: even a
read-only open creates -wal/-shm sidecars next to a WAL database, needs
directory write access, and can wait on locks. The header read does none
of that. It cannot tell delete from truncate/persist, so doctor reports
'rollback journal mode' rather than an exact mode name. Missing files
are skipped; empty, unreadable, or corrupt files are reported as
unreadable without failing doctor.
The database list reuses backup.py's _QUICK_STATE_FILES plus per-board
kanban databases. Exposure uses hermes_state.is_sqlite_wal_reset_vulnerable,
so the 3.50.7 and 3.44.6 backports count as fixed.
OpenCode Go serves GPT 5.6 Luna only via the Responses API per its
published endpoint table (https://opencode.ai/docs/go/#endpoints), but
opencode_model_api_mode() had no gpt- case in the Go branch, sending
Luna to /v1/chat/completions. The relay's shim streams full text but
never emits a finish_reason chunk, so every complete answer is
classified as a mid-stream drop and each turn fails with 'Response
remained truncated after 4 continuation attempts'.
Mirror the Zen branch: gpt- on Go -> codex_responses. Base URL needs
no change (normalize_opencode_base_url already keeps /v1 for
codex_responses). Extend test_opencode_go_api_modes_match_docs with
the Luna assertions.
- Move classify_persistence_error into hermes_state beside is_disk_full_error
and delegate the disk bucket to it (fixes 'ENOSPC writing state.db' and
'not enough space' classifying as unknown). run_agent keeps a thin lazy
delegating wrapper so the documented import path and fast import survive.
- Classify CompressionSessionBusyError (and its RPC-wrapped message forms)
as 'locked': the motivating #81227 failure mode stringifies to 'is being
compressed by another writer', which the substring heuristic missed.
- Export PERSISTENCE_ERROR_CAUSES and iterate it in the cron explainer
suppression instead of a hardcoded tuple, so a future cause bucket cannot
silently desynchronize cron delivery.
- Hedge the gateway locked/unknown recovery wording ('should already be
saved' instead of 'was recorded') to match the explainer - the early
turn-start persist may also have failed.
- Drop STATE_DB_WAL_WARN_BYTES (speculative dead constant with no consumer;
the pre-existing 50 MB doctor WAL check covers the warning).
- Tests: compression-busy classification, is_disk_full_error delegation,
causes-tuple coverage; mutation-checked red-green.
Review follow-ups from the pre-push falsification pass:
- collect_state_db_stats now routes through _connect_tracked_db so the
module's byte-probe guard sees the read-only connection (consistency
with the module's own ro-connection precedent; prevents a raw header
probe from cancelling this reader's locks in multi-threaded callers).
- Drop the new >256 MiB WAL warning from the stats renderer: doctor's
pre-existing 50 MB WAL check (with --fix checkpoint) already covers
WAL runaway, and two warnings for one condition is noise. The test now
locks in the dedup decision.
- Locked-cause explainer says the message 'should already be saved'
rather than overclaiming when the early turn-start persist also failed.
Operators had no Hermes surface showing state.db size, WAL health, index
family shape, or how many processes hold the database — all of which were
needed to diagnose a lock-contention incident on a 4.5 GB multi-writer
install.
Adds collect_state_db_stats() (strictly read-only URI connection, no
SessionDB instantiation, per-field best-effort) and a /proc-based
count_db_holders() to hermes_state, and wires a stats block into hermes
doctor's state.db section: logical size, pages/freelist, WAL size,
message/session counts, journal mode, holder count, FTS table presence
and deferred-rebuild status. Advisory warnings at >1 GiB (suggest
sessions.auto_prune and, when the v23 rebuild is pending or the legacy
trigram shape is detected, an offline 'hermes sessions optimize-storage')
and >256 MiB WAL (checkpoint health). Any stats failure degrades to a
single info line.
Two claim-failure diagnostic paths (cronjob_tools.py:629,921) still used
the old inline 'not enabled or state==paused' check. After get_job()
normalizes via effective_job_state, a half-paused record has
state='scheduled' and enabled=True, so the inline check returned False —
mislabeling the job as 'already being fired' instead of 'paused/disabled'.
Also hoists effective_job_state/is_job_runnable to the top-level import in
cronjob_tools.py (was function-local) and updates console_engine.py's
_format_job to use effective_job_state instead of the old inline
state-or-enabled derivation — a fourth display path the original PR missed.
Follow-up to PR #81287.
pause_job already sets enabled=false atomically with state/paused_at, but
get_due_jobs only checked enabled — so a contradictory record
(enabled=true + paused_at/state=paused) still fired. That was the 07-30
outage failure mode: list looked frozen, fleet kept merging.
- is_job_runnable / effective_job_state: pause markers gate fire; display
derives from the scheduler-honoured enabled flag so half-paused never
renders as [paused]
- get_due_jobs self-heals enabled=false + logs error on contradiction
- claim_job_for_fire uses is_job_runnable (paused_at counts too)
- list/format paths use effective_job_state
- behavioural tests: pause blocks due fire; half-pause self-disables
SessionResumeTooLargeError said 'across its lineage' even when the CLI
mid-setup path counted only the tip segment; the exception now takes a
scope phrase.
OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
The mid-setup CLI resume path loads only the tip session's rows, so
gate it with a tip-only count instead of the full-lineage count (which
over-rejected heavily-compressed sessions). Transient guard failures
(locked DB, adaptor stores) now log and proceed instead of blocking
resume with a new error.
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
Two follow-ups to the off-loop move, from external review (both verified,
the second larger than reported):
- Config read-modify-write handlers moved to worker threads could now
interleave — _CONFIG_LOCK covers each load/save individually, never the
span between them; the event loop used to serialize these accidentally.
New _CONFIG_MUTATION_LOCK (worker-threads only, so it can never block
the loop) held across the whole load→mutate→save span in all seven RMW
handlers. update_config_raw skipped: it's a full-document replace with
no server-side read, so a lock cannot close its client-side window.
- The review flagged two skills routes still taking _SKILLS_PROFILE_LOCK
on the event loop; a systematic audit of hermes_cli/web_routers/ found
24 on-loop routes (skills 5, mcp 9, tools 10, cron 1). All moved to the
same inner-_run + asyncio.to_thread pattern, mutating ones under the
mutation lock, uniform lock order (_SKILLS_PROFILE_LOCK →
_CONFIG_MUTATION_LOCK). Await-safe _config_profile_scope routes, plain
def routes, and already-threaded routes unchanged.
Regression tests: concurrent theme+font updates both survive (fails with
the lock nulled: "theme write lost to a concurrent font write"); event
loop stays responsive while the profile lock is held during GET
/api/skills. 214 tests passing across the touched suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diagnostics loop watchdog caught GET /api/config freezing the gateway
event loop for >1s, stack-sampled blocking on _SKILLS_PROFILE_LOCK inside
_profile_scope. Any async handler that entered _profile_scope (process-wide
threading lock) or called load_config()/save_config() on-loop could stall
every chat and WebSocket at once while a slow lock-holder ran.
Move 28 such handlers to the existing inner-_run + asyncio.to_thread
pattern (contextvar-safe: the whole scope enter/body/exit stays inside one
worker thread). Handlers using the await-safe _config_profile_scope, plain
def endpoints (FastAPI threadpool), and tui_gateway's contextvar-only
decorator are unaffected and unchanged.
Regression test holds _SKILLS_PROFILE_LOCK in a thread while calling
GET /api/config and asserts an event-loop heartbeat keeps ticking; it fails
against the pre-fix code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Surfaced during the post-merge review pass on our own #81113 follow-up:
cached_fetch_api_models gained _cache_entry_valid (numeric-'at'
validation) but its sibling cached_provider_model_ids still did
float(entry.get('at', 0)), which raises ValueError/TypeError on a
hand-edited or corrupted provider_models_cache.json row and propagates
uncaught into the /model picker call sites. Same fix, same helper:
corrupt rows are now a cache miss (live fetch), never an exception.
Both wrappers now share the identical validity predicate, closing the
divergence the 'mirrors' docstring promised away.
Also two test nits from the same review: unused OrderedDict import
dropped and the drain-order assertion strengthened to pin LRU-first
FIFO order in tests/gateway/test_agent_cache_pressure.py.
Mutation-checked: restoring the raising float() form makes the new
corrupt-at tests fail.
Rescope: hermes verify fills only the runtime-smoke gap and plugs into
the pieces Hermes already has instead of standing beside them.
- agent/verification_evidence.py: record_verify_run() — explicit ledger
write for hermes verify results (shared _insert_evidence factored out
of record_terminal_result). Passing runs mark the workspace passed
like scripts/run_tests.sh; failures are recorded; --phase/--skip-start
runs are recorded as targeted scope.
- hermes_cli/verify_cmd.py: record results into the ledger on completion
(fail-silent, HERMES_SESSION_ID attribution); on the detect path merge
detect_project_facts verify commands the recipe missed into the
recipe's test list (never applied to a saved manifest).
- agent/verification_stop.py: recipe-aware nudge — when the workspace
has a runnable recipe (start command or .hermes/environment.json),
suggest hermes verify --json as the preferred full check; cheap,
try/except-guarded detection that can never break the nudge path.
- agent/verify/recipes.py: document layer ownership (coding_context =
cheap prompt facts; verify/recipes = deep runtime recipe).
- tests/verify/test_ledger_and_nudge_integration.py: 17 tests covering
ledger pass/fail recording, the closed edit->nudge->verify->satisfied
loop, recipe-aware nudge wording + fail-silence, and the facts merge.
Teknium
cresslank
brooklyn!
Austin Pickett
Drexuxux
Nolan
Paulo Nascimento
rainbowgits
ygd58
kshitij
Erosika
Axmr1
Victor Kyriazakos
rjvandeve
kinsolee
Royalaid
Gille
GodsBoy