Adversarial review of the salvaged recovery found a reachable fail-open:
compression continuations inherit the rotated agent's model_config
verbatim (publish_compression_child callers pass
agent._session_init_model_config), so a delegate subagent's continuation
carries _delegate_from=<the delegate's own parent>. The marker-PRESENCE
filters in reopen_orphaned_compression_session and
find_live_compression_child misclassified such a REAL continuation as a
delegate child:
- reopen: parent 'orphaned' -> reopened while a live continuation exists
-> two live heads in one lineage (verified with a live repro)
- find_live: adoption misses the continuation (fail-closed, masked the
fork pre-PR; the PR made it active)
Fix: markers only disqualify a child when they point at the queried
parent (shared _NON_CONTINUATION_CHILD_FILTER_SQL fragment, also
resolving the duplicated-SQL drift risk flagged by the reuse reviewer).
Both directions regression-tested: reopen fails closed on an
inherited-marker continuation; find_live adopts it.
Also from review: reopen-failure log raised debug->warning (the failure
hard-fails the turn moments later), commit-semantics hardening comment
on the lease DELETE path, blank-line nit.
The three read-only projection walks (get_compression_tip,
list_sessions_rich chain, resume walk) share the marker-presence shape
but fail closed (skip a continuation -> resume shows the parent), and
the fixed adoption path self-heals that case at turn start; left as-is.
Installs whose state.db reached schema_version >= 22 before the task
dimension was added carry a 5-column PRIMARY KEY on
session_model_usage. The column reconciler ADDs task as a bare
nullable, but SQLite cannot ALTER a primary key, and the version-gated
v22 rebuild is unreachable (current_version < 22 already false), so
the composite 6-column key never lands. Every upsert in
_record_model_usage then fails with 'ON CONFLICT clause does not match
any PRIMARY KEY or UNIQUE constraint', aborting the enclosing write
transaction — token/cost accounting permanently dead (#73823).
Add an idempotent _heal_session_model_usage_pk() modeled on
_heal_gateway_routing_pk(), run unconditionally from _init_schema on
every open. Salvaged from #73838 with fix-ups:
- ported to SessionSchemaMixin in hermes_state_schema.py (the schema
code moved out of hermes_state.py in 21c7ae8563; the PR targeted the
old location)
- rebuild wrapped in a PRAGMA foreign_keys=OFF/ON window: the
connection enables FKs before _init_schema and OR IGNORE does NOT
suppress FK violations, so a single orphaned usage row (session
pruned while accounting was broken) would have aborted the heal
- COALESCE('') on the nullable reconciler-added task column (and the
billing columns) during the copy
- stale-v22+ regression tests: rebuilt PK + restored upsert, orphan
rows survive the FK window, healthy-DB no-op, no legacy leftover
Fixes#73823
Under dual gateway/agent WAL contention (FTS5 trigram sync holding the
write lock on large appends) the SQLite engine can raise a transient
'no more rows available' error. The exception CLASS varies with the
build — some surface it as InterfaceError, a SIBLING of DatabaseError —
so it escaped both existing retry branches in _execute_write on attempt
0 and killed the turn as session_persistence_failed even though the
identical write succeeds standalone.
Port of #74934 onto the deadline-patience rewrite (8da8a7887d): the
PR's attempt-counted constants (60 retries / 300ms jitter / 2.0s engine
timeout) predate that rewrite and are superseded by the patience
budget, so they are intentionally NOT carried over. Instead the check
is message-scoped and rides the existing deadline/patience loop:
- extract the jittered-sleep-within-deadline logic into a shared
_sleep_before_retry helper (behavior-preserving for locked/busy)
- retry 'no more rows available' from OperationalError, DatabaseError
(checked BEFORE the FTS-corruption rebuild path so it is not
misrouted), and a message-scoped sqlite3.Error catch-all
- any other error in any class propagates untouched on attempt 0
Tests: transient InterfaceError retried to success; unrelated
InterfaceError propagates immediately; DatabaseError variant retried;
exhausted patience surfaces the original error.
Disk-full / ENOSPC / SQLITE_FULL on first-message session persist used to be
swallowed as a debug log while prompt.submit still returned streaming, so the
send vanished with no error. Re-raise those failures, return a real RPC error,
and stamp session_persistence_failed turns with error so clients get a terminal
error frame.
A shared state.db is legitimately held for multi-second stretches by
sibling Hermes processes: VACUUM after auto-prune, the TRUNCATE WAL
checkpoint at close on a large WAL, offline recovery, or an older
still-running process whose FTS maintenance predates the bounded-merge
protocol (every `hermes update` leaves mixed-version processes sharing
the DB until the old ones exit).
The old retry budget was attempt-counted: 15 attempts x 20-150ms jitter
gives up after ~1.3s of waiting. Any hold longer than that surfaced as:
- append_message failing -> the conversation loop aborts the turn as
session_persistence_failed ('No reply: the turn was stopped because
session storage could not be written') on a perfectly healthy store;
- SessionDB() open failing -> the CLI disables persistence for the
entire run ('Failed to initialize SessionDB ... database is locked').
Both observed in production logs on 2026-07-29 (10.8 GB state.db, 9
concurrent hermes processes, three of them pre-dating the bounded-merge
fix pull).
Changes:
- _execute_write patience is now TIME-based with two budgets: routine
writes wait up to 20s; transcript-critical writes (append_message,
session-row creation — the ones whose failure aborts a user turn)
wait up to 60s. Jitter stays 20-150ms for the first 2s, then backs
off to 250ms-1s so a long hold isn't hammered with BEGIN IMMEDIATE.
- Exhausted patience raises an error that names the actual cause
(another process held the write lock; the database is healthy)
instead of a bare 'database is locked' that reads like disk damage —
and the turn-abort explainer inherits that clarity.
- SessionDB open now applies the same jittered patience to the
locked/busy class around connect+schema-init, instead of failing the
whole open (and disabling persistence for the run) on the first 1s
timeout. Non-lock errors, including the malformed-schema repair
class, propagate immediately as before.
Fixes#74478
The trigram MATCH branch in search_messages() had the same
OperationalError-only catch that #66420 fixed on the main FTS5 branch: a
corrupt messages_fts_trigram shadow table raises the malformed /
'fts5: corrupt structure record' class (sqlite3.DatabaseError, parent of
OperationalError), which propagated straight out of search_messages and
crashed CJK session/history search for read-only sessions.
Route that class through the shared one-shot _try_runtime_fts_rebuild()
and retry the trigram query (catch moved outside self._lock so
rebuild_fts() can re-acquire it, mirroring the main branch). If the
rebuild is refused (guard consumed / FTS disabled / different error) or
the retry fails, fall through to the existing LIKE substring fallback —
which reads only the canonical messages table — instead of raising, so
CJK search degrades gracefully rather than crashing.
Adds two regression tests: trigram search self-heals in place after
shadow-table corruption (answers from the rebuilt trigram index, not the
LIKE fallback), and degrades to LIKE without raising when the one-shot
rebuild was already consumed.
Follow-up to #66420; refs #66296#66724
Complements #66296 (self-heal on the write path): search_messages()'s main
FTS5 MATCH query caught only sqlite3.OperationalError (a query-syntax error →
return empty). A corrupt FTS index raises the malformed / "fts5: corrupt
structure record" class, which is a sqlite3.DatabaseError — the parent of
OperationalError, so it was NOT caught and propagated straight out of
search_messages, crashing session/history search.
The write path now rebuilds and retries on that class, but a read-only
session (cron/CLI history search, or a search issued before any write) never
triggers a write, so its search stayed broken until the next process restart
ran the offline repair.
Catch the DatabaseError corruption class on the search MATCH read too and
route it through the existing one-shot _try_runtime_fts_rebuild(), then retry
the query. The catch is moved outside `with self._lock` so rebuild_fts() can
re-acquire the lock (mirrors _execute_write). The one-shot guard is shared
with the write path, so a single instance never loops on a genuinely
unrecoverable index. OperationalError syntax handling is unchanged (caught
first).
Adds a regression test: with a corrupted messages_fts and no post-corruption
write, search_messages() rebuilds in place and returns the match; without the
fix it raises DatabaseError.
Complements the #65637 salvage (53d358838 + a9cc17fd8): the gateway
session store now retries transcript appends through its own queue, but
cron and CLI writers call SessionDB directly — a corrupt FTS index still
hard-failed their appends until the next process restart triggered the
offline repair.
_execute_write now detects the FTS-corruption error class (both the
generic 'database disk image is malformed' and newer SQLite's
'fts5: corrupt structure record' variant), performs a one-shot in-place
rebuild by delegating to the existing rebuild_fts(), and retries the
failed write. One-shot per instance so an unrecoverable database cannot
loop; lock/busy jitter-retry path untouched.
E2E-verified: corrupted messages_fts_data rejects appends; with this fix
the same append self-heals, persists, and FTS search works again.
izumi0uu
kshitij
RelaxJonh
Dannoob
Brooklyn Nicholson
Teknium
Anthony Ruiz
Frowtek