Cross-PR interaction fix: #77714 (salvage of #71775) changed
_available_entries to return (available, pending_refresh) while #77631
(salvage of #67642) added next_available_at() which still truthiness-
tests the bare return. A non-empty tuple is always truthy — even
([], []) — so the reset-aware gate silently returned None ('no wait
info') for every exhausted pool, disabling the feature #77631 shipped.
Unpack the tuple and test the available list.
Also adapts the lock-probe test for the RLock introduced by #77714
(same-thread non-blocking acquire always succeeds on an RLock; probe
from a helper thread instead).
Simplify-pass finding: _listing_group_label already falls back to 'other' for empty source names, and _classify_source guarantees source_name=='' only when source=='other' — both legs were dead by construction. Aligns the summary path's grouping with the listing path.
```
Error: optimization failed: no such table: messages_fts_trigram
No data was lost. Re-run to resume.
```
on any install where the trigram FTS index is legitimately absent. The failure is
deterministic — re-running can never make progress, because the crash happens at the same
point every time — so the database is permanently stuck on the legacy high-footprint FTS
layout with no supported way forward.
Observed on a 5.4 GB production `state.db`. After the fix the same database optimized
successfully and shrank to 3.3 GB.
The trigram index is absent whenever the runtime cannot maintain it. On a SQLite build
without the `trigram` tokenizer, `_ensure_fts_schema()` returns `False`, so `__init__`
leaves `self._trigram_available = False` and no `messages_fts_trigram` table on disk. This
is a **supported degraded runtime**, not damage — CJK/substring search falls back to
`LIKE` and everything else works normally. `_is_fts5_unavailable_error()` and
`_warn_trigram_unavailable()` exist specifically to make this path graceful.
Two code paths write the boundary sweep for the deferred FTS rebuild, and only one of them
respects that flag:
| Function | Trigram `INSERT` guarded? |
|---|---|
| `fts_rebuild_step()` | ✅ `if include_trigram:` where `include_trigram = self._trigram_available` |
| `_fts_rebuild_finish()` | ❌ unconditional |
`_fts_rebuild_finish()` runs the boundary sweep at the *end* of the backfill. Its
unguarded `INSERT INTO messages_fts_trigram …` raises `OperationalError`, which propagates
out of `optimize_fts_storage()` and aborts the entire optimization — *after* the backfill
has already completed. Hence the characteristic output showing 100% progress immediately
before the error:
```
Rebuilding index: 100% (909,671/909,671)
Error: optimization failed: no such table: messages_fts_trigram
```
There is a second, quieter consequence. The teardown phase that reclaims the demoted
`fts_v22_trash_*` shadow tables runs *after* the backfill phase in
`optimize_fts_storage()`. Because the crash happens before teardown is ever reached, those
tables are never emptied or dropped — so the space the migration was supposed to reclaim
stays allocated indefinitely, and the leftover trash tables look (misleadingly) like
evidence of a half-finished migration.
Build a populated v23 database, set the deferred-rebuild markers, then reopen it on a
runtime where `_ensure_fts_schema('messages_fts_trigram', …)` returns `False` (exactly
what a SQLite build without the trigram tokenizer produces) and call
`optimize_fts_storage()`:
```
[precondition] trigram absent, _trigram_available=False, rebuild pending ✓
RED ✗ optimize_fts_storage raised OperationalError: no such table: messages_fts_trigram
```
With this patch applied, unchanged harness:
```
optimize_fts_storage returned {'ok': True, 'vacuumed': None}
GREEN ✓ optimize ok; markers cleared; base FTS 'zebra' -> 200 hits
```
Full harness and transcripts in `TEST-EVIDENCE.md`.
Gate the sweep on `self._trigram_available`, exactly as `fts_rebuild_step()` already does:
```python
include_trigram = self._trigram_available
def _do(conn):
...
if include_trigram:
conn.execute("INSERT INTO messages_fts_trigram(...) ...")
```
The base `messages_fts` sweep and the marker cleanup are untouched, so the rebuild still
finalizes correctly and the index remains complete for every row it is responsible for.
The fix does not disable or weaken search to dodge the error — the regression tests assert
that base FTS still returns results afterwards.
`TestFtsRebuildFinishWithoutTrigram` in `tests/test_hermes_state.py`:
- `test_rebuild_finish_skips_trigram_when_unavailable` — drives `_fts_rebuild_finish()`
directly on a trigram-less runtime; asserts it completes, clears both rebuild markers,
and leaves base FTS searchable.
- `test_optimize_fts_storage_succeeds_without_trigram` — end-to-end through the public
`optimize_fts_storage()` entry point; asserts `ok=True`, markers cleared, search intact.
Both use the existing `_NoTrigramConnection` helper already in the file. Both fail on
`main` with `no such table: messages_fts_trigram` and pass with this patch.
`tests/test_hermes_state.py` passes in full (463 tests → 465 with these two). `ruff` clean.
This PR is the crash only.
A companion PR narrows `_db_opens_cleanly()` so that
`hermes sessions repair --check-only` stops reporting a write-broken FTS schema as
healthy — the gap that makes this class of problem hard to diagnose in the first place.
The two are independent and can land in either order.
Review folds on the #71775 salvage (dossier findings 1+2):
- self._lock becomes an RLock and the mutation primitives
(_replace_entry, _persist) are now self-locking, so the deferred
single-use-token refresh path — which deliberately runs its
cross-process flock + OAuth network I/O OUTSIDE the pool lock —
still serializes its pool mutations against concurrent
select()/rotation. In-lock callers re-acquire reentrantly.
- Dropped _refresh_pending_entries' redundant second _replace_entry:
_refresh_entry already merges the refreshed entry internally.
Adds tests/agent/test_credential_pool_deferred_refresh.py pinning both
invariants: select() must NOT hold the lock during the refresh window
(the PR's whole point), and the post-refresh mutations MUST contend on
the lock (blocking-thread probe).
select() and acquire_lease() held self._lock during the entire
_available_entries() loop, which for openai-codex and xai-oauth providers
includes a cross-process file lock (_auth_store_lock) plus OAuth token
refresh HTTP POST. The lock timeout can exceed 20 seconds, blocking all
credential pool consumers across every gateway thread and subagent.
Collect single-use-token refresh entries under the lock, then execute the
refreshes outside it. On success the refreshed entry is merged back into
the pool and re-selected. Non-single-use providers (anthropic, nous)
continue refreshing inside the lock since their refresh is a simple HTTP
POST with no cross-process coordination.
Review fold on the #67642 salvage: next_available_at() called
_available_entries() — which prunes DEAD entries, syncs tokens, and
persists — and iterated self._entries with no lock, racing concurrent
select()/rotation exactly as has_available()'s comment warns. Wrap the
method body in self._lock and pin it with a non-blocking-acquire probe
test.
restore_primary_runtime retries the primary every turn once the 60s
transient cooldown clears. For subscription-window limits (Claude
Pro/Max 5h windows, Codex weekly caps) the reset is hours or days away,
so every retry is a guaranteed failure costing two provider switches
and two prompt-cache invalidations per turn.
Add CredentialPool.next_available_at() (earliest reset across exhausted
entries; None when available now or no reset info) and gate the restore
on it: skip while the primary's pool says nobody can serve, restore on
the first turn after the reset elapses. Fail-open: any gate error or
missing reset info falls through to the existing per-turn retry, so
recovery can never be later than today. Cross-provider fallbacks
consult the PRIMARY's pool (not the attached fallback pool), reusing
the loaded pool for the existing rebind to keep auth reads at one per
restore.
Every hashed bundle chunk under /assets/ was served with no caching
directives, so each dashboard load re-fetched (or at best revalidated)
every JS/CSS chunk. Those filenames carry a Vite content hash — the
bytes behind a given URL can never change; a rebuild mints new
filenames referenced by a freshly served index.html.
Mark them Cache-Control: public, max-age=31536000, immutable:
- the /assets StaticFiles mount, via a subclass that stamps the header
on 200s only (404s stay uncached — a rebuild can create the file),
- serve_css, preserving its X-Forwarded-Prefix url() rewrites for
/fonts/, /fonts-terminal/, /ds-assets/, /assets/.
index.html keeps no-store, no-cache, must-revalidate — it is the
mutable entry point that binds users to the current hashes.
The original PR also added hand-rolled per-request gzip compression of
asset responses; that part is deliberately dropped. This server is a
localhost-default dashboard backend: compressing every response on the
CPU to save loopback bandwidth is a pessimization, and callers that
front it with a real proxy already get compression there.
Salvaged from PR #28543 (idea by @sea-monsters; gzip groups dropped as
described above).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sessionStorage seed set loading=false whenever any cache existed, which
defeats App.tsx's load-bearing pluginsLoading gate: with a cached manifest
that declares tab.override === "/chat", the persistent ChatPage host must
NOT mount before plugins resolve, or it spawns a PTY and gets yanked when
the override plugin takes over the route.
Seed loading=false from the cache only when no cached manifest overrides
/chat (canSeedLoadedFromCache); manifests are still seeded either way so
plugin routes register synchronously on refresh. Adds focused tests for
the gate, including the /chat-override case.
- Plugin manifests are now cached in sessionStorage on fetch.
- On refresh, plugin routes are registered synchronously from cache, preventing unwanted redirects to /sessions.
- Removes the !pluginsLoading guard from the catch-all route in App.tsx, as plugin routes are now always available on first render.
- Background fetch always updates the cache and routes, so new/removed plugins are reflected after reload.
- Resolves the race condition where plugin pages would redirect to /sessions on hard refresh.
le=100 would 422 real desktop callers: sessions-settings fetches
archived at limit=200, the command palette lists at 200, and the
electron remote-merge over-fetches limit+offset (exceeds 100 at
offset>=81, and its .catch(()=>null) silently drops remote sessions).
Clamp must sit above real client maxima. New test pins limit=200 w/
offset.
Re-derivation of aydnOktay's twin clamp PRs onto current main (the
session-list endpoints moved into web_routers/; the analytics endpoints
gained asyncio.to_thread wrappers since the originals):
- limit le=100 on /api/sessions, /api/sessions/search and the
/api/profiles/sessions fan-out (one unbounded request could drag every
session row + correlated-subquery preview work out of SQLite, times
every profile's state.db on the fan-out).
- days ge=1 le=365 on /api/analytics/usage + /api/analytics/models
(huge or non-positive values force full-history InsightsEngine work or
inverted windows; the UI only offers 7/30/90 presets).
FastAPI Query bounds reject at the validation layer (422). 8 new tests;
both clamp classes mutation-checked (clamp removed -> its tests fail).
The 2-line alias had zero production consumers (web_server calls
get_usage_breakdown directly). Tests rewired onto the real API; the
contracts they pin are unchanged. Stale test docstring fixed.
Follow-up to #77652: each runFlush registered a fresh requestAnimationFrame and never cancelled it. Chromium parks rAF callbacks for hidden renderers, so a long hidden stream at the 33ms floor accumulates thousands of parked closures that all fire in the first frame on refocus (all but one no-oping through the stale-frame guard). Track the pending handle, cancel it before requesting a new one (only the newest flush's measurement matters), and cancel on unmount.
On dashboard-only sessions nothing else executes check_fn warmers (they
live only in the tool-schema build), so the hub's read-only cache lookup
would report auth_required=False forever. On a cache miss, schedule a
deduplicated daemon-thread probe off the request path; the short hub TTL
surfaces the verdict on the next fetch.
Two fixes landed overlapping helpers on the same statement: the mid-turn
reply guard grew `isLiveProjectionRow` / `hasStreamedContent`, while the
inflight-dump guard grew `isLiveTailRow` / `hasStructuralParts`. Two
definitions of "is this row live" and "does it carry content" in one
function is how the next change silently reshapes one of them.
Collapse to a single module-level pair. `isLiveTailRow` now covers pending,
stream ids, inflight projections and sealed interim rows, so the reply guard
also stops treating an interim row as committed history; `hasStreamedContent`
is defined in terms of `hasStructuralParts`. Both text-extension checks route
through `isStrictAnswerTextExtension` rather than a bare `startsWith`.
Also hoists the live-tail lookup out of an inline IIFE and fixes the lint
warnings it carried.
Co-authored-by: 686f6c61 <github@00b.tech>
Structure-only same-turn carry used (live(previous) || live(message)), so a
new live text-only assistant at a compression-rewritten ordinal could inherit
reasoning/tool parts from an unrelated historical structured row.
Require the structure-bearing cached row itself to be live-tail (pending /
assistant-stream-* / interim). Add regressions for non-extending live dump
carry and the compression graft rejection.
Addresses salvage path on #76744 / #76444.
Only skip/graft structure for the current live assistant (stream id,
pending, or after the latest user), not completed historical tool rows.
Require live-tail identity for same-turn structure carry. Align journal
overlay with strict answer-text extension.
Addresses review + CI on #76744.
Skip pure-text inflight.assistant projections when the transcript already
has reasoning/tool-call structure, and only overlay journal answer text
on strict extension.
Fixes#76444
Follow-up to #77653: textWithoutReferenceLines built a fresh /g RegExp per call and hand-managed lastIndex — but it runs on both sides of every message comparison in the reconcile loops. An anchored non-global regex has no shared-lastIndex hazard and can be hoisted to module scope. Also removes textWithoutImageRefs, whose last production consumer #77653 replaced (kept IMAGE_REF_LINE_RE for extractImageRefs), and retargets its now-stale comment.
When a turn's reply commits under its own id, the settled local
`assistant-stream-*` row shifts one assistant ordinal earlier, so ordinal
pairing finds nothing at its slot and re-appends it — the same answer twice.
Drop a settled stream row only when the authoritative transcript already
carries that exact text. Keying `isPendingAssistant` on the explicit pending
flag alone would also have fixed this, but it discards the sibling case in
the same report: a reply that finished locally before the gateway committed
it, where the local row is the only copy that exists.
Co-authored-by: Dolverin <59100064+Dolverin@users.noreply.github.com>
Switching sessions while a turn streams (or right as it completes) could
leave the assistant reply missing until restart. Resume merges stored
history with the gateway's `inflight` projection, whose assistant row is
text-only and often an empty `assistant-stream-${sessionId}` shell; both
reconcile paths then dropped the local pending row that held the only copy
of the streamed text, reasoning and tool calls.
A shared pair of guards replaces the ad-hoc comparisons at all three sites.
`localPendingSupersedes` accepts the cached row only when it is the same
reply further along — an empty shell it has content for, or text it strictly
extends — so a longer unrelated row can no longer hijack an ordinal or reuse
a stream id, and a retained `inflight.error` snapshot is never mistaken for
an empty shell. `withAuthoritativeTurnState` then takes content from the
renderer while liveness, row id and reactions stay the backend's call, so a
settled shell cannot leave a finished reply spinning.
Co-authored-by: arimu1 <19286898+arimu1@users.noreply.github.com>
The curator LLM review loop (_run_llm_review) built its AIAgent without
enabled_toolsets, so it advertised the full default catalog (~30 tools plus the
context_engine lcm_* family) on every call. The fork uses only four tools, fixed
by its own system prompt, with no dispatch path to the rest, so ~26 tool schemas
shipped on every request as dead weight: ~7K input tokens per call on a loop that
makes 50-100 calls per consolidation pass.
Restrict the fork to enabled_toolsets=["skills", "terminal"], the same tools the
prompt already names. Behavior-neutral: the prompt held the model to these tools
and nothing routed calls to the others. Mirrors the background_review fork
(background_review.py:788-794). Call-site only; AIAgent already forwards the kwarg.
Adds test_review_fork_restricts_toolsets_to_skills_and_terminal (captures the
constructor kwarg) and test_review_fork_toolset_surface_is_skills_plus_terminal
(pins the resolved surface).
Re-derive of PR #38470 on today's main (its target file desktop-controller.tsx no longer exists after the contrib/ refactor; the three surviving ideas are applied at their new homes):
- incremental-external-store-runtime: the dep-less setAdapter effect ran every render; gate on [runtime, store] — behavior-preserving because __internal_setAdapter early-exits on identical store.
- ChatView is now memo()d, and session-tile hoists its inline arrow props to useCallbacks/module constants so the memo actually holds.
- Render-count regression test (mocked Thread) proves an unrelated parent re-render no longer re-renders the chat shell.
Credit: idea and original implementation by @hdd69 in #38470.
The comment above .btn-arc contained 'bg-*/', whose */ terminated the comment mid-sentence, leaving 'text-* variant utilities. */ .btn-arc {' as an invalid prelude — CSS error recovery can drop the whole .btn-arc rule. Reword so no */ appears inside the comment.
Extracted from #59352 by @rerdi92 (the rest of that PR — a month-stale icons.ts rewrite and a chunk-size warning-ceiling bump — is superseded/masking and was not salvaged).
With the full-jitter backoff (300ms base) six attempts can elapse in ~9s,
so the old RECONNECT_ESCALATE_AFTER=6 attempt threshold raised the
recoverable boot error during a brief post-boot blip — breaking the
'a remote that drops post-boot keeps looping with NO boot.error' contract.
Escalate after RECONNECT_ESCALATE_AFTER_MS (45s, matching the old
deterministic 1->15s ladder's calibration) elapsed since the first failed
reconnect of the episode. Reset on clean open, manual/wake reconnect, and
soft switch, preserving the reset-on-success path.
All three desktop reconnect loops (primary gateway boot, secondary
multi-profile gateway pool, plugin event socket) used bare exponential
backoff with no jitter. After a gateway restart every disconnected
client redials on the exact same schedule, so the reconnect attempts
land in lockstep instead of spreading out -- a burst that can starve
the gateway's file descriptors while it's still coming back up.
Add reconnect-backoff.ts implementing AWS-style full-jitter backoff
(random delay in [0, min(cap, base * 2^attempt))) and wire it into all
three call sites in place of their local Math.min/2**attempt math.
Manual reconnect paths already reset the attempt counter and bypass
the timer entirely -- unchanged.
scheduleDeltaFlush's adaptive floor is driven by lastFlushCostRef, but
runFlush only timed flushQueuedDeltas(), the synchronous store write.
While a session streams, syncSessionStateToView defers the $messages
publish (React commit + Streamdown re-parse) to its own rAF, so the
measured cost stayed near zero and the floor collapsed to the fixed
33ms path no matter how expensive the real commit was.
runFlush now records the write cost as a fallback, then extends the
measurement through a rAF registered after the view-sync one: it runs
in the same frame right after the deferred commit, and the rAF
timestamp marks frame start so only in-frame work is counted, not the
vsync wait. A stale callback from before a newer flush is ignored, and
a hidden renderer that never fires rAF keeps the write-cost fallback.
Simplify-pass finding: sharing only the REGEX left the wrapper logic
(empty/strip/slash checks) duplicated, half-defeating the no-drift goal.
The classmethod now calls agent/memory_provider.is_trivial_prompt directly;
_TRIVIAL_PROMPT_RE stays as a class attr for backward compatibility with
any external referents.
Rebase fold on the salvaged gate:
- is_trivial_prompt/TRIVIAL_PROMPT_RE move to agent/memory_provider (the
ABC both the core gate and providers already import) — one source of
truth; honcho's _TRIVIAL_PROMPT_RE now aliases it, turn_context and the
queue_prefetch_all warm path (a sibling site main grew after the PR's
base) both use it
- tests: gate tests at the prefetch call site (mutation-checked), shared
classifier tests incl. prefix-collision guards (k8s/yolo/note/supper),
and honcho dialectic-machinery tests re-driven with a substantive prompt
("hello" became trivial by design — those tests exercise thread cadence,
not the classifier)
Salvage of PR #25350 (commits 88ffede2d + 2b848a0b2 + 3136dc63a, squashed
and ported): the run_agent.py prefetch site the PR gated has since moved
into agent/turn_context.py's build_turn_context(), so the trivial-query
gate lands there instead.
- Gate the per-turn memory_manager.prefetch_all() on a trivial-prompt
check so greetings/acknowledgements ('hi!', 'thanks', 'ok') no longer
block the turn on provider network round-trips or inject stale context.
- Extend honcho's _TRIVIAL_PROMPT_RE with greetings and a trailing
punctuation class so 'hey!' / 'hello.' classify as trivial.
- Add honcho classifier tests for greeting forms.
kshitijk4poor
kshitij
Carbon
brooklyn!
Kyzcreig
dsad
WojtekMR3
Allard
youzi
aydnOktay
Rod Boev
Hafiz Ahmad Ashfaq
686f6c61
VittoriaLanzo
Lex
rerdi92
Hermes Coder
Paulo Nascimento
Adolanium
ayushere
Ayush Sahay Chaudhary
Jakub Wolniewicz