When state.db's write path fails (corrupt FTS, or a crash landing between
routing publication and row creation), the live gateway conversation can end
up in a session row that never received its identity columns: session_key,
chat_id, chat_type and origin_json are all NULL. In-memory routing hides the
damage for as long as the gateway stays up. After a restart the chat is
resolved from the DB, and find_latest_gateway_session_for_peer cannot see
that row — both of its queries match on the very columns it lacks — so the
chat resumes the last keyed sibling instead, days older. The messages were
never lost, only unreachable.
Hardening the write side cannot reach a row that is already damaged, so add
the offline repair path the tracking issue asks for:
- SessionDB.find_orphaned_gateway_sessions() reports message-bearing rows
with no session_key, and names the predecessor each one continues only
when the evidence is unambiguous — a recorded parent_session_id
("lineage"), or exactly one keyed row of the same source and compatible
user_id that fell quiet within 15 minutes of the orphan's start
("contiguity"). Contested pairs are reported with a reason and left alone:
a wrong adoption would splice one person's conversation into another
person's chat. Branch, delegate and tool rows are excluded — they are
unkeyed by design, not by damage.
- SessionDB.adopt_orphaned_gateway_session() stamps the orphan from the
predecessor (never overwriting a column that already has a value), records
the lineage, and retires the predecessor under end_reason
'superseded_by_repair' — a reason recovery does not treat as resumable, so
the repaired row wins the chat from then on. The pair is re-verified inside
the write transaction, making a concurrent heal a no-op rather than a
conflicting write.
- `hermes sessions repair-routing` drives both. It reports without touching
the database; --apply confirms first and warns that a running gateway
still holds the old mapping in memory.
Refs #82616.
Titling fired on the first response, so a session sat unnamed for the whole
opening turn - p50 151s, p90 1212s across real sessions, because a turn is
tool calls, not one round-trip. A turn that failed or was interrupted never
got a title at all. Four surfaces each carried their own copy of the call.
Move it into the shared turn prologue and split it in two: a deterministic
title derived from the user's opening message, written inline before the
model runs, then one small-model call that upgrades it. The response is
constrained to a JSON object so there is no preamble to strip, and control
wrappers are stripped rather than refused, so a slash command titles as
what the user asked for instead of the command itself.
purge_stale_tool_call_markers ran a permanent, irreversible UPDATE with
no backup — inconsistent with repair_state_db_schema's backup-by-default
convention for destructive state.db operations elsewhere in this file.
Take a full snapshot via VACUUM INTO (safe against a live connection,
unlike the raw-copy _backup_db_file used for malformed-schema repair)
before the write, timestamped beside state.db. Skipped when dry_run or
when there's nothing to change. Add --no-backup to `hermes sessions
clean-markers`, mirroring `sessions repair`.
Verified end-to-end: the CLI run against a real temp state.db produces
the backup file before printing the cleared-row count.
The load-on-read repair (_strip_stale_tool_call_markers) fixes affected
sessions in memory on every resume, but never touches the DB — long-lived
sessions re-scan and re-repair the same rows on every load, and the
contaminated bytes stay in state.db (and any backup/cache snapshot of it)
indefinitely.
Add SessionDB.purge_stale_tool_call_markers(dry_run=False): a one-time,
idempotent UPDATE that permanently blanks the content column on affected
rows. Only content is touched — tool_calls and every other column are
left untouched, so provider tool_call/tool_result pairing survives.
dry_run reads through the no-lock read path and never writes.
Wire it up as `hermes sessions clean-markers [--dry-run]`, mirroring the
existing optimize/repair subcommands. Verified end-to-end against a real
temp state.db: dry-run reports the row without writing, the real run
clears it and preserves tool_calls, and a second run is a no-op.
joaomarcos
Brooklyn Nicholson
teknium1