Review of this PR was right that maxsize=8 bounds the wrong thing. The
LifoQueue caps how many connections are RETURNED; _checkout_read_conn opened
unconditionally on a miss, so N readers arriving on a cold pool all missed, all
opened, and peaked at N. The surplus was closed on release, so nothing
accumulated forever -- but EMFILE is a peak-instant condition and the burst
that empties the pool is exactly the burst that exhausts the fd table, so the
original wedge was still reachable. Measured on the previous commit: 64
concurrent readers held 64 live connections at once.
A connection now holds a permit for its whole lifetime -- acquired in
_get_read_conn() before the open, released in _close_read_conn() after the
close -- so open+checked-out is bounded together. A pool hit costs no permit
because the connection it hands back already holds one, which leaves
_get_read_conn() as the only place that can open. The acquire is non-blocking:
past the ceiling readers fall back to the locked writer connection rather than
queueing, since blocking would convert descriptor exhaustion into a stall,
which is the same outage with a different stack trace. Same burst now peaks at
8. BoundedSemaphore rather than Semaphore so an unpaired release raises instead
of silently widening the ceiling.
Two latent leaks in the same function, found while doing this:
- a CJK extension load that failed after a successful open returned None
without closing the connection, leaking a descriptor the tracking registry
still counted -- the same leak shape one level down;
- any non-sqlite3.Error between open and return stranded a permit
permanently, which would ratchet the ceiling down to zero and silently
demote every later read to the writer lock.
On the test: the existing one joins every worker before counting, so it
measures the pool at rest and structurally cannot observe peak -- which is why
this got through. The new one uses a barrier so all 64 workers hold their
connections until every worker has checked out, making the count taken at that
moment the actual simultaneous peak. Verified it fails against the previous
commit (64 checked out, 65 live) and passes at 8/9. Also covers the
writer-connection fallback, permit recovery after a failed open, and that
close() releases exactly the permits it drained.
`projects.tree` answers for the backend's own profile, so the grouped
sidebar had nothing to draw once the user asked to see every profile.
Run the same authoritative builder once per profile against that
profile's state.db and merge the results by folder, so one checkout is
one group no matter how many profiles work in it, and the owning profile
rides on each session row where the badge and filter can read it.
Group totals are summed in SQL rather than over the loaded page — a
number that shrank as you scrolled would be worse than no number.
Scope the batched sidebar slices while we're here: cron and messaging
came back cross-profile unconditionally, which is why a concrete profile
showed another profile's Telegram threads and cronjobs.
Closes#65710Closes#42651Closes#70629
Review follow-ups on the composite salvage (whole-bug-class sweep):
- session.branch and _persist_branch_seed copied parent history without
display_kind/display_metadata, so a tagged timeline marker (personality
pivot, model switch, auto-continue) re-entered the branched session as a
bare role=user row after a restart — re-planting the phantom-ordinal
class this PR fixes. Both projection dicts now carry the tags; regression
asserts added to both branch tests (mutation-checked: fail without the
fix).
- ui-tui renderer learns display_kind=personality_switch (was falling
through to an opaque user bubble; desktop got the case in commit 1).
- programmatic-integration docs: document the two new 4004 refusals
(boolean ordinal, bare confirm_truncate).
- hermes_state comment: archived rows are searchable only with
include_inactive=True, not by default search — align comment with the
actual FTS filter.
- strip stray trailing blank line in test_tui_gateway_server.py
Guarding the *aim* of a rewind still leaves every other way of aiming it
wrong terminal. All three reported incidents (#70516, #80763, #82756) ended
at the same write — `replace_messages()` in the `prompt.submit` truncation
path — and all three were unrecoverable for the same reason: the rows are
DELETEd, which also evicts them from the FTS index, so there is no `active=0`
archive and nothing to restore from.
The codebase already draws this distinction and already has the safe half of
it. `archive_and_compact` is documented as "the durability-preserving
alternative to replace_messages"; `rewind_to_message` — the `/undo` path —
soft-deletes to `active=0, compacted=0` and keeps the rows "on disk for audit
/ forensic inspection". The desktop rewind is the same user-facing operation
as `/undo` and was the one taking the destructive branch.
`replace_messages(..., archive_dropped=True)` flips the DELETE to a
content-preserving `UPDATE messages SET active = 0`, reusing the existing
transaction and the existing `active=0, compacted=0` marking so the dropped
turns stay readable via `get_messages(..., include_inactive=True)` and stay
out of session search (`compacted=0` = "the user took it back", vs
compaction's `compacted=1` = "summarized away, still discoverable").
The live transcript is byte-identical either way — only the durability of the
dropped turns changes. The parameter defaults to False, so the fork handler,
the ACP adapter and `gateway/session.py` keep their current semantics
untouched; a test pins that.
`active_only=True` stays on the call: #80216 still applies, and archiving must
not disturb rows an earlier compaction deliberately archived.
Test doubles for `replace_messages` in the gateway suite are widened to the
real signature — they are stand-ins for SessionDB, and a double that does not
accept what production passes silently converts this write into a 5008.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find_latest_gateway_session_for_peer filtered non-recoverable rows out of
candidacy BEFORE ordering, so recovery could search behind a /new reset
boundary and resurrect an older still-open row for the same peer —
silently restoring the exact context the user reset.
Rebuilt against the #82633 finder (has-messages ranking +
COALESCE(last_activity_at, started_at) recency): the fence is expressed
as a NOT EXISTS guard inside both the exact-key and peer-fallback
queries — a candidate is rejected when an intentional boundary row
(session_reset / session_switch / idle / daily / suspended /
resume_pending_expired) for the same peer ended after the candidate's
last activity. If the conversation's most recent event is an intentional
reset, recovery returns nothing rather than reaching behind it.
Cherry-picked from #68617 and adapted to the rewritten finder.
(cherry picked from commit bb2c562a16)
When state.db's write path fails (corrupt FTS, or a crash landing between
routing publication and row creation), the live gateway conversation can end
up in a session row that never received its identity columns: session_key,
chat_id, chat_type and origin_json are all NULL. In-memory routing hides the
damage for as long as the gateway stays up. After a restart the chat is
resolved from the DB, and find_latest_gateway_session_for_peer cannot see
that row — both of its queries match on the very columns it lacks — so the
chat resumes the last keyed sibling instead, days older. The messages were
never lost, only unreachable.
Hardening the write side cannot reach a row that is already damaged, so add
the offline repair path the tracking issue asks for:
- SessionDB.find_orphaned_gateway_sessions() reports message-bearing rows
with no session_key, and names the predecessor each one continues only
when the evidence is unambiguous — a recorded parent_session_id
("lineage"), or exactly one keyed row of the same source and compatible
user_id that fell quiet within 15 minutes of the orphan's start
("contiguity"). Contested pairs are reported with a reason and left alone:
a wrong adoption would splice one person's conversation into another
person's chat. Branch, delegate and tool rows are excluded — they are
unkeyed by design, not by damage.
- SessionDB.adopt_orphaned_gateway_session() stamps the orphan from the
predecessor (never overwriting a column that already has a value), records
the lineage, and retires the predecessor under end_reason
'superseded_by_repair' — a reason recovery does not treat as resumable, so
the repaired row wins the chat from then on. The pair is re-verified inside
the write transaction, making a concurrent heal a no-op rather than a
conflicting write.
- `hermes sessions repair-routing` drives both. It reports without touching
the database; --apply confirms first and warns that a running gateway
still holds the old mapping in memory.
Refs #82616.
Root cause of #82616: gateway session identity (session_key/chat_id/
origin_json) was written best-effort in a separate UPDATE after row
creation, both reset-path DB writes swallowed failures silently
(logger.debug / bare print), transcript reads ignored the reroute map
that writes follow, and restart recovery ranked candidate rows by
started_at while hard-rejecting empty rows. A single failed write could
therefore strand the live conversation in an unroutable orphan row while
a days-old zombie kept the routing key — after any gateway restart the
chat silently resumed the zombie (user-visible context loss, 5 confirmed
incidents on one install since June).
Four class fixes:
1. Identity lands atomically in the session INSERT: origin_json and
display_name join _insert_session_row's column list + COALESCE
backfill; both gateway creation paths (get_or_create + reset) pass
full identity including parent_session_id lineage (fixes#12857).
2. record_gateway_session_peer self-heals: when the target row is
missing (failed/deferred create, crash window) it INSERTs the row
with full identity instead of silently no-opping — every per-turn
peer refresh is now a repair opportunity, and an identity-less lazy
writer (update_token_counts/record_auxiliary_usage) can never leave
a gateway session permanently unroutable.
3. load_transcript follows the write-side reroute chain and the durable
compression tip before querying, so reads can no longer return 0
rows for a session whose messages live under its compression child;
read exceptions are WARNING, distinguishable from an empty result.
4. find_latest_gateway_session_for_peer ranks by
COALESCE(last_activity_at, started_at) (message-bearing rows first)
and returns an empty-but-keyed row instead of None — a zombie
predecessor can no longer beat the live conversation, and recovery
never mints a fresh id when a keyed row exists.
Reset-path DB write failures now log at WARNING with the routing
consequence spelled out.
Tests: tests/gateway/test_session_continuity_82616.py (11 tests) —
sabotage-verified: 6/11 fail without the fixes. E2E incident replay
(real SessionDB, temp HERMES_HOME) confirms the production shape now
resolves to the live session.
Fixes#82616. Related: #12857, #78182 (read-path half), #79576.
A session row can say whether its work is open, merged or closed, and link
to it. The join is the session's own repo + branch, asked of GitHub in one
batched GraphQL request per repo (branch aliases, not a `gh pr list` page
that a busy repo crowds ours out of), through the remote-aware git facade so
a desktop on a remote gateway asks the backend's `gh`.
Two ways a session's branch can't answer, both covered:
- It ran on trunk. Fork PRs share our branch namespace, so asking about
`main` badges a stranger's PR onto it — trunk is never asked about, and
cross-repository PRs are dropped server-side either way.
- It worked in a worktree, so the branch it recorded at start isn't where
the PR came from. Creating a PR from the review pane binds the session to
the branch it actually used, and for sessions that predate that, the PR is
recovered from the transcript: `gh pr create` prints a bare PR url and
nothing else, so a tool result whose whole output is one is a claim rather
than a mention. Scanned read-only across profiles, once per session ever.
get_messages() only deserializes content and tool_calls; the structured
reasoning columns (reasoning_details, codex_reasoning_items,
codex_message_items) come back as the raw TEXT they were stored as.
Feeding those rows straight back into a write, which is exactly what
the POST /api/sessions/{id}/fork handler does by piping get_messages()
into replace_messages(), hit an unguarded json.dumps() and stored the
already-serialized string encoded a second time. On replay of the fork,
json.loads() then yields the inner string instead of a list, and every
consumer's isinstance(..., list) gate silently drops it: preserved
Anthropic thinking blocks, Codex encrypted-reasoning/message-item
replay, and OpenRouter multi-turn reasoning context are all lost after
a fork, with one more encoding layer added per fork.
The /branch copy loop had the same defect from the other side: it
forwarded reasoning but none of the structured columns, and both TUI
branch writers persisted role/content alone, dropping reasoning and
reasoning_content along with them.
Route the six dumps sites in append_message and _insert_message_rows
through a shared guard that keeps already-serialized strings as-is;
structured values from the live runtime are dumped exactly as before.
Forward the reasoning fields in all three branch writers, matching the
set gateway/slash_commands.py already forwards on its own /branch path.
A session title had no notion of who set it, so two bugs followed. An
auto-generated title could clobber a name the user typed, and every
compression rotation renumbered the conversation it forked - one piece of
work reaching 'Smallville Map Architecture Plan #10' in the sidebar.
Titles now carry a source (derived < llm < user) enforced by one
compare-and-swap, so an automatic write can only ever replace a title of
strictly lower authority. Compression carries the name across unchanged.
Legacy NULL rows rank as user, so auto-titling only fills genuinely
empty titles on existing data.
- Move classify_persistence_error into hermes_state beside is_disk_full_error
and delegate the disk bucket to it (fixes 'ENOSPC writing state.db' and
'not enough space' classifying as unknown). run_agent keeps a thin lazy
delegating wrapper so the documented import path and fast import survive.
- Classify CompressionSessionBusyError (and its RPC-wrapped message forms)
as 'locked': the motivating #81227 failure mode stringifies to 'is being
compressed by another writer', which the substring heuristic missed.
- Export PERSISTENCE_ERROR_CAUSES and iterate it in the cron explainer
suppression instead of a hardcoded tuple, so a future cause bucket cannot
silently desynchronize cron delivery.
- Hedge the gateway locked/unknown recovery wording ('should already be
saved' instead of 'was recorded') to match the explainer - the early
turn-start persist may also have failed.
- Drop STATE_DB_WAL_WARN_BYTES (speculative dead constant with no consumer;
the pre-existing 50 MB doctor WAL check covers the warning).
- Tests: compression-busy classification, is_disk_full_error delegation,
causes-tuple coverage; mutation-checked red-green.
Review follow-ups from the pre-push falsification pass:
- collect_state_db_stats now routes through _connect_tracked_db so the
module's byte-probe guard sees the read-only connection (consistency
with the module's own ro-connection precedent; prevents a raw header
probe from cancelling this reader's locks in multi-threaded callers).
- Drop the new >256 MiB WAL warning from the stats renderer: doctor's
pre-existing 50 MB WAL check (with --fix checkpoint) already covers
WAL runaway, and two warnings for one condition is noise. The test now
locks in the dedup decision.
- Locked-cause explainer says the message 'should already be saved'
rather than overclaiming when the early turn-start persist also failed.
Operators had no Hermes surface showing state.db size, WAL health, index
family shape, or how many processes hold the database — all of which were
needed to diagnose a lock-contention incident on a 4.5 GB multi-writer
install.
Adds collect_state_db_stats() (strictly read-only URI connection, no
SessionDB instantiation, per-field best-effort) and a /proc-based
count_db_holders() to hermes_state, and wires a stats block into hermes
doctor's state.db section: logical size, pages/freelist, WAL size,
message/session counts, journal mode, holder count, FTS table presence
and deferred-rebuild status. Advisory warnings at >1 GiB (suggest
sessions.auto_prune and, when the v23 rebuild is pending or the legacy
trigram shape is detected, an offline 'hermes sessions optimize-storage')
and >256 MiB WAL (checkpoint health). Any stats failure degrades to a
single info line.
SessionResumeTooLargeError said 'across its lineage' even when the CLI
mid-setup path counted only the tip segment; the exception now takes a
scope phrase.
With sessions.max_*_messages: 0 the guards previously still ran an
unbounded COUNT (full lineage for resume) — the exact pathological work
disabling them is meant to avoid. Live callers use the raise side
effect only, so return 0 without touching the messages table.
OFFSET paging made the streaming export O(n^2) on huge transcripts;
after_id keyset paging keeps each page seek O(1). Adds after_id to
SessionDB.get_messages (ascending-only, guarded against latest/offset
combos).
sessions.max_resume_messages / sessions.max_export_messages (default
20000, 0 disables) replace the hardcoded hard-rejects, and the CLI
'sessions export' guard becomes per-session instead of cumulative so
full-DB backups of many small sessions keep working. Error guidance now
points at the config override instead of the (corruption-only) repair
command.
#80216 fixed /retry (and a follow-up fixed yuanbao recall) destroying
soft-archived active=0/compacted=1 in-place-compaction rows via the
destructive replace_messages default. Two sibling sites still carried the
same class:
- acp_adapter/session.py _persist (non-owned-agent branch): probed
has_archived_messages and FAILED OPEN into the destructive full replace
on any probe error; the probe can also race a concurrent
archive_and_compact. Now passes active_only=True unconditionally — on a
fresh create/fork every row is active=1 so behavior is identical, and
the probe (its only production caller) is deleted.
- tui_gateway/methods_prompt.py edit/regenerate truncation: bare
replace_messages() deleted the archived transcript of a compacted
session on every edit/regenerate. Now active_only=True.
hermes_state.has_archived_messages docstring updated (probe is now
test/diagnostic-only). Test stubs in test_tui_gateway_server.py accept the
new kwarg. New regression tests: real-SQLite archive-survival for both
write shapes, fresh-session equivalence (the claim the unconditional
switch rests on), and source-level guards pinning that neither site
re-grows the fail-open probe (both mutation-checked: revert either fix and
its guard fails).
Adversarial review of the salvaged recovery found a reachable fail-open:
compression continuations inherit the rotated agent's model_config
verbatim (publish_compression_child callers pass
agent._session_init_model_config), so a delegate subagent's continuation
carries _delegate_from=<the delegate's own parent>. The marker-PRESENCE
filters in reopen_orphaned_compression_session and
find_live_compression_child misclassified such a REAL continuation as a
delegate child:
- reopen: parent 'orphaned' -> reopened while a live continuation exists
-> two live heads in one lineage (verified with a live repro)
- find_live: adoption misses the continuation (fail-closed, masked the
fork pre-PR; the PR made it active)
Fix: markers only disqualify a child when they point at the queried
parent (shared _NON_CONTINUATION_CHILD_FILTER_SQL fragment, also
resolving the duplicated-SQL drift risk flagged by the reuse reviewer).
Both directions regression-tested: reopen fails closed on an
inherited-marker continuation; find_live adopts it.
Also from review: reopen-failure log raised debug->warning (the failure
hard-fails the turn moments later), commit-semantics hardening comment
on the lease DELETE path, blank-line nit.
The three read-only projection walks (get_compression_tip,
list_sessions_rich chain, resume walk) share the marker-presence shape
but fail closed (skip a continuation -> resume shows the parent), and
the fixed adoption path self-heals that case at turn start; left as-is.
Tests like tests/gateway/test_goal_verdict_send.py monkeypatch Path.home()
to a tmpdir; resolving the guard's 'real root' through Path.home() made the
test's own hermetic home look like production (false positive). Resolve via
os.path.expanduser/LOCALAPPDATA instead — the hermetic conftest never
rewrites HOME, so this always names the actual production root.
Forensics on a live developer machine found pytest fixture rows inside the
REAL ~/.hermes/state.db — sessions with chat_id 'chat-1', '123', 'wx-chat',
and gateway_routing rows whose scope was literally under /tmp/pytest-of-*/.
A pytest-spawned process also opened the live DB and flipped its journal
mode (journal_mode=DELETE fallback on SQLite 3.50.4) under the WAL-mode
gateway writer, destroying committed transcripts ("Persisted transcript
lagged live cached history ... possible FTS write corruption", 15+
occurrences). The existing live-system guard covers kill primitives but not
the SessionDB/SessionStore write paths.
Root cause (leak vector): the session-level HERMES_HOME sandbox in
tests/conftest.py only created a tempdir when HERMES_HOME was UNSET. On a
machine where the shell (e.g. gateway-launched, or an exported
HERMES_HOME=~/.hermes) hands pytest the production home, the sandbox was
skipped entirely — every argless SessionDB()/SessionStore() and every
collection-time DEFAULT_DB_PATH froze onto the real state.db.
Fixes (fail the class, one owner):
* hermes_state._ensure_test_isolation(): single choke point wired into
SessionDB.__init__ (every construction, incl. read_only). Under pytest
(PYTEST_CURRENT_TEST / PYTEST_VERSION — inherited by subprocess
children), a db path resolving to <real-root>/state.db or
<real-root>/profiles/<name>/state.db raises RuntimeError('live-system
guard: ...') before any connection, mkdir, or journal-mode pragma.
* tests/conftest.py: session sandbox now also tempdir-redirects a pre-set
HERMES_HOME that points at the production root (the actual escape
vector); kanban deny-list capture updated to match. New autouse
_state_db_write_guard fixture honors the existing
@pytest.mark.live_system_guard_bypass marker as the escape hatch and
feeds custom (non-~/.hermes) production roots into the guard deny-list.
* gateway/session.py: SessionStore.__init__ no longer swallows the guard's
RuntimeError into the JSONL fallback — guard trips are loud.
* tests/hermes_state/test_live_db_isolation_guard.py: behavioral
regression tests — production paths (direct, profile, read-only,
unnormalized, default-resolution) raise; tmp HERMES_HOME works; bypass
marker works; SessionStore re-raises guard errors but still degrades on
ordinary failures; subprocess child without HERMES_HOME is refused while
a hermetic child succeeds.
No new HERMES_* env vars; no hardcoded ~/.hermes (platform root comes from
hermes_constants._get_platform_default_hermes_home()).
The WAL-reset-vulnerability gate (#70055 lineage) could flip a LIVE WAL
database to journal_mode=DELETE while another process was writing to it.
Observed on state.db (Aug 5): a pytest process on the repo .venv (SQLite
3.50.4, vulnerable) opened the live ~/.hermes/state.db while the gateway
(SQLite 3.53.1, WAL) held it, downgraded the journal mode, and destroyed
the gateway's committed-but-uncheckpointed WAL transactions (disk rows
went 10 -> 0 while memory held 185). cron/executions.db already had the
"leave WAL in place, no live downgrade under concurrent openers" rule via
the on-disk WAL probe; state.db and every other store shared the hole
whenever the mode PROBE itself was blocked by a concurrent opener's locks
("could not read the mode" was treated as "not WAL" -> flip anyway).
Generalized in the single journal-mode owner (apply_wal_with_fallback),
covering ALL call sites (state.db, kanban.db, projects.db,
cron/executions.db, delivery_ledger, async_delegation,
verification_evidence, discord recovery, response_store.db,
memory_store.db):
- _set_journal_mode_no_wait(): the only journal-mode switch primitive for
non-WAL targets. Forces busy_timeout=0 around the pragma so SQLite's own
exclusivity requirement for leaving WAL becomes the concurrent-opener
detector — any other opener (this process or another) makes the flip
fail immediately instead of waiting out a busy timeout and sneaking the
flip in under a live writer.
- Vulnerable-SQLite gate: an unreadable journal mode (probe blocked) now
means "ownership not provably exclusive" — leave the mode untouched and
warn, never flip. A lock conflict on the flip itself likewise leaves the
mode alone.
- Configured journal_mode=delete: refuses (raises) rather than downgrading
blind when the mode cannot be verified under a concurrent opener.
- Filesystem-incompat fallback: re-raises instead of downgrading when the
on-disk mode cannot be verified.
- New/exclusively-owned DBs on vulnerable builds behave exactly as before
(DELETE gate retained per #70055).
Behavioral tests use a REAL second process (and a real second connection
holding an exclusive lock) with the blocked-state assertions running WHILE
the holder owns the DB, plus exclusive-ownership downgrade-still-happens
coverage.
Follow-up to #79722, which introduced _escape_like in hermes_state.py for
the prune/archive filter fix. The same three-replace escape chain existed
as five more inline copies in hermes_state.py and two in
hermes_state_search.py (which must not import hermes_state — cycle).
Move the helper to hermes_state_common.escape_like (the module that exists
for exactly this) and route every copy through it:
- hermes_state.py: session-ID prefix resolution, find_session_by_title,
get_next_title_in_lineage, the _like_pattern closure in list projection,
and the kanban cwd retag
- hermes_state_search.py: the two LIKE-fallback token escapes
hermes_state re-imports it as _escape_like for back-compat. No behavior
change: every site produces byte-identical SQL patterns.
Follow-up on the salvage of #78681 + #78927: the second fix inlined the
exact body of the _escape_like helper the first fix introduced ten lines
above. Call the helper instead so there is one copy of the escaping rule.
_cwd_prefix_clause builds "cwd is this directory or under it" for session
listing, workspace resume and prune/archive. The two LIKE arms bound the
raw prefix, so `_` and `%` acted as wildcards on a value that is a path:
cwd_prefix="/home/me/my_project"
main -> ['sibling', 'target'] # /home/me/myXproject/src matched too
fix -> ['target']
`_` matches any single character, so a same-length sibling directory with
children falls inside the pattern. prune_sessions() deletes the rows it
matches (and their on-disk transcripts), so an unrelated project's history
goes with it.
Escape the needle and pair both arms with ESCAPE, the convention the rest
of this file already uses; the literal separator backslash in the Windows
pattern is escaped for the same reason. The `=` arm is an exact compare and
keeps the raw prefix, so directory-and-children matching is unchanged.
Follow-up to the *_like filters in #78681, kept separate because this helper
is shared by four call sites beyond prune.
_prune_filter_where documents title_like / model_like / branch_like as
"case-insensitive substring matches", and the CLI confirmation renders them
as "title contains 'X'". They were bound straight into a bare LIKE, so `_`
matched any single character and `%` any run.
The builder backs prune_sessions(), which deletes session rows and their
on-disk transcripts, so the over-match is unrecoverable: pruning
title_like="user_auth" also destroys "user-auth", "userXauth" and
"user auth". `_` is not exotic here -- git branch names and session titles
carry it routinely.
Escape the operator's needle and add ESCAPE '\' to the three clauses, the
same convention the rest of this file already uses for LIKE queries. Match
direction is unchanged for needles without wildcards.
Left alone: _cwd_prefix_clause has the same unescaped shape but is shared
by four call sites beyond prune, so it is a separate change.
archive_and_compact's new model_config_patch block was the third
near-identical SELECT -> tolerant-parse -> merge -> UPDATE copy in
hermes_state.py (update_session_runtime_lock and set_session_yolo carry
the other two). Extract _merge_model_config_json(conn, sid, patch,
on_missing=...) and route all three through it, preserving each
caller's missing-row policy (flag setters skip, archive raises).
Also adds the two small accessors the compressor needs:
- patch_session_model_config(): standalone atomic merge for callers
that must update model_config without rewriting the transcript
- get_session_model_config_value(): tolerant single-key read
Follow-up to the salvaged #79286 commit, per the repo's
extend-don't-duplicate rule.
get_compression_lineage's forward walk accepted any non-branch child as
the compression continuation. Delegate subagent rows (_delegate_from)
and tool-tagged rows (source=tool) created before the real continuation
were picked as the lineage successor, so the lineage — and session .md
export built on it — followed a subagent's transcript instead of the
actual conversation continuation.
Rename _is_branch_child_row to _is_explicit_fork_child_row, treat
_delegate_from and source=tool rows as explicit forks alongside
_branched_from, and require _is_compression_child_row in the forward
walk instead of merely excluding branches.
Sliced from PR #79024 by @RyderFreeman4Logos (the cache-scope portion
of that PR is tracked separately in #79017).
A session created in the wrong directory needs its cwd corrected after the
fact. session.cwd.set only reaches live runtime sessions, so cold rows were
stuck. The new RPC targets the persisted row by session_key, validates the
folder, and REPLACES the git branch/root identity (update_session_cwd grows a
replace_git_meta flag) so the project tree's grouping follows the move instead
of pinning the session under the project it left via a stale git_repo_root.
A live idle agent bound to the row is re-anchored through the runtime path;
a mid-turn session refuses with 'session busy'. Runs on the RPC pool — the
git probes are subprocesses.
Adds a last_read_at watermark to the sessions table so surfaces (CLI,
TUI, desktop) can badge unread conversations. Read state derives from
the watermark vs latest activity, so new messages flip a conversation
back to unread with zero writes on the message path. NULL means never
tracked, so shipping the column doesn't badge pre-existing history.
set_session_read() stamps the whole compression lineage, matching the
archive/pin semantics; list_sessions_rich() rows carry a derived
`unread` key. DB layer only — no surface exposes it yet.
#75017: Telegram polling conflict retry used drop_pending_updates=False,
starting a new getUpdates session that immediately got 409'd by the
previous still-expiring session — creating the very conflict it was
trying to recover from. Switch to drop_pending_updates=True so Telegram
terminates stale sessions. Also add a recovery-generation guard so the
first transient getUpdates success after a retry doesn't reset the
conflict counter back to 0 (defense-in-depth from PR #75096).
#75153: The WAL-reset warning always said 'hermes update can repair'
even for git/pip/system Python installs where it can't. Now uses
detect_install_method() + recommended_update_command_for_method() to
give a context-appropriate hint (hermes update for git, docker pull for
docker, nix message for nix, generic install hint as fallback).
purge_stale_tool_call_markers ran a permanent, irreversible UPDATE with
no backup — inconsistent with repair_state_db_schema's backup-by-default
convention for destructive state.db operations elsewhere in this file.
Take a full snapshot via VACUUM INTO (safe against a live connection,
unlike the raw-copy _backup_db_file used for malformed-schema repair)
before the write, timestamped beside state.db. Skipped when dry_run or
when there's nothing to change. Add --no-backup to `hermes sessions
clean-markers`, mirroring `sessions repair`.
Verified end-to-end: the CLI run against a real temp state.db produces
the backup file before printing the cleared-row count.
The load-on-read repair (_strip_stale_tool_call_markers) fixes affected
sessions in memory on every resume, but never touches the DB — long-lived
sessions re-scan and re-repair the same rows on every load, and the
contaminated bytes stay in state.db (and any backup/cache snapshot of it)
indefinitely.
Add SessionDB.purge_stale_tool_call_markers(dry_run=False): a one-time,
idempotent UPDATE that permanently blanks the content column on affected
rows. Only content is touched — tool_calls and every other column are
left untouched, so provider tool_call/tool_result pairing survives.
dry_run reads through the no-lock read path and never writes.
Wire it up as `hermes sessions clean-markers [--dry-run]`, mirroring the
existing optimize/repair subcommands. Verified end-to-end against a real
temp state.db: dry-run reports the row without writing, the real run
clears it and preserves tool_calls, and a second run is a no-op.
The conversation_loop fix (previous commit) stops new "[memory]"-style
bare tool-call markers from being cached/persisted, but sessions written
before that fix can still carry rows where a bare marker was saved as
the assistant's "final response".
Add a load-on-read repair pass in hermes_state.py, mirroring the existing
_strip_background_review_harness defense-in-depth: on session restore,
any assistant row whose content is only a bracketed marker (e.g.
"[memory]", "[skill_manage]") AND that carries tool_calls has its content
blanked before the history re-enters the model's context. The tool call
and its result are left untouched so provider tool_call/tool_result
pairing stays intact. Sessions with no affected rows pass through the
normal path unchanged.
get_messages_as_conversation, get_resume_conversations, and
get_ancestor_display_prefix still took self._lock — the same global
choke point the read-path split (WAL per-thread read-only connections)
was meant to remove from every recall/browse read. These three are the
hottest reads in the file: every session resume across the gateway,
CLI, and ACP adapter goes through one of them, so a resume racing a
burst of concurrent-session writer flushes still convoys behind them
exactly like the fixed paths used to.
_session_lineage_root_to_tip (the lineage walk shared by all three,
plus get_conversation_root) had its own independent self._lock use and
needed the same conversion — without it the outer functions still
blocked on the very first line.
Verified empirically: a reader thread calling all three functions
while another thread holds self._lock blocked for the writer's full
hold duration before the fix, and returned immediately after (SQLite
3.50.4 in this dev venv falls back to journal_mode=DELETE per the
WAL-reset-bug guard, so the requires_wal-marked regression test is
exercised via a local WAL-forced script instead; it still runs and
passes on any runtime where WAL is actually active).
Simplify-pass folds on the #23254 salvage:
- REUSE (HIGH): append_messages_batch now delegates row serialization to
the pre-existing _insert_message_rows helper (already shared by
replace_messages / archive_and_compact / portability import) instead
of adding a third serialization path (_prepare_message_row +
_MESSAGE_INSERT_SQL are gone). One row-writer for every multi-row
path; the row-ID return was consumed by no production caller, so the
batch returns the inserted count.
- QUALITY (HIGH): the compression-lock + compression-closed admission
guards are extracted into _check_transcript_write_guards, shared by
append_message and append_messages_batch (previously duplicated 23
lines that had already needed targeted fixes, #74478). The role-gated
reasoning filtering is no longer duplicated in run_agent.py — it
lives at its one site inside _insert_message_rows.
- EFFICIENCY (MEDIUM, measured): unbounded seed copies hold one BEGIN
IMMEDIATE for seconds (10k rows ~= 2.4s; FTS triggers dominate) and
monopolize the in-process write lock. append_messages_batch grows a
chunk_rows param; all seed/copy call sites use chunk_rows=500. Same
recovery semantics as the old per-row loops, bounded lock holds.
- REUSE (MEDIUM): the two remaining per-row branch-copy loops found by
the pass (gateway/slash_commands.py /branch, hermes_cli
cli_commands_mixin.py branch) are converted to chunked batches too
(AsyncSessionDB's generic to_thread forwarder covers the async site).
Turn-flush benchmark unchanged after the refactor: 2.43 -> 0.87 ms
median per 5-message flush (64% faster).
Re-derivation of #23254 (@devsart95) on today's flush loop. The turn
flush in _flush_messages_to_session_db wrote one BEGIN IMMEDIATE
transaction per message row; a typical agent turn (user + assistant +
tool results) paid 3-8 transactions -- and, off WAL (the default on
macOS while the WAL-reset guard is active), 3-8 fsyncs -- per turn.
Adds SessionDB.append_messages_batch: same row shape as append_message
(shared _prepare_message_row serializer + _MESSAGE_INSERT_SQL column
list, so the two writers cannot drift), same compression-lock and
compression-closed guards, one aggregated session-counter UPDATE, one
transaction for the whole batch. Row serialization stays outside the
write lock.
The flush loop now collects the turn's new rows and writes them in one
call. All-or-nothing pairs exactly with the persisted-marker stamping:
on failure no rows landed and no markers were stamped, so the next
flush re-writes the whole tail (same recovery contract as before,
minus the partial-prefix case that could double-count).
Measured (same harness, 5-message turn, journal_mode=DELETE,
synchronous=FULL): 2.32ms -> 0.83ms median per turn flush (64% faster,
5 fsyncs -> 1). On WAL the win is smaller but the atomicity fix holds.
Retarget #73639 onto the SessionDB mixin split (hermes_state_common /
hermes_state_schema). Fresh installs create UPDATE OF content/tool_*
triggers; existing broad AFTER UPDATE triggers are inspected and
replaced under schema init without an FTS rebuild (WHEN clauses already
guarded content correctness; OF skips non-content status writes that
saturated disk I/O on large state.db).
Tests: tests/test_fts_update_of_narrowing.py (4)
Addresses review from @teknium1 on PR #71755:
- Extended apply_database_pragmas() to handle cache_size, mmap_size,
and temp_store from config.yaml (alongside existing wal_autocheckpoint
and journal_size_limit). No hardcoded defaults — all values are
opt-in via config.yaml, avoiding policy conflicts with other PRs.
- Applied to ALL connection types: writer (_connect_and_init),
read_only cross-profile attach, and WAL per-thread readers
(_get_read_conn). Previously PRAGMAs only ran on the writer path.
- Removed inline PRAGMAs from _connect_and_init — single source of
truth in apply_database_pragmas().
- Documented config keys with examples in function docstring.
list_sessions_rich()'s compression-root projection called
_get_session_rich_row() once per root — a separate single-row query per
compression root on every session-list render. Resolve every tip id
first, then fetch all tip rows in one WHERE id IN (...) query via the
new _get_session_rich_rows_batch().
_get_session_rich_row() is now a thin wrapper over the batch method, so
the enriched SELECT (preview + last_active) lives in exactly one place —
future column changes (e.g. #42196's include_system_prompt) only touch
one query.
get_compression_tip()'s chain walk is untouched; it's a genuine
per-session graph walk with branch/delegate-exclusion and race handling,
and batching it safely is out of scope here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review fold-ins on top of #56768 (@Skywind5487):
- session_count_ge ran its query without self._lock, unlike every
sibling counter on SessionDB (session_count, session_count_by_source).
- Document the deliberate semantics change: session_count() defaults to
archived = 0, which is both the expensive part (full index scan,
measured 543us vs 4us on 20k sessions) and wrong for the only caller
(has_any_sessions asks 'has this install ever had sessions' -- an
archived session is still a created one).
Yishova
Brooklyn Nicholson
kshitij
joaomarcos
Tranquil-Flow
izumi0uu
Teknium
cryptoyasenka
Victor Kyriazakos
kinsolee
kshitij
Drexuxux
Ryder Freeman
pierrenode
devsart95
embwl0x
Jasmine Naderi
crayfish-ai
jasoisjaso