httpx timeout exceptions (ReadTimeout/ConnectTimeout/PoolTimeout/WriteTimeout)
stringify to an EMPTY string, so when one survived the retry loop the user saw
'API call failed after 6 retries: ' with nothing after the colon — a TLS
abort, a reset connection, and a deterministic read-timeout fire were all
indistinguishable (and invisible).
Ported Buzz's pure timeout_message classifier (crates/buzz-agent/src/llm.rs,
block/buzz#4959) to hermes-agent:
- agent/timeout_error_summary.py: pure classifier over the exception type —
connect-phase timeouts ('no connection established, check base_url') vs
read-phase timeouts ('no response received within <N>s — consider raising
providers.<provider>.request_timeout_seconds in ~/.hermes/config.yaml'),
embedding the configured timeout value (per-model/per-provider config
first, HERMES_API_TIMEOUT fallback) and the exact config knob.
- run_agent.py: _summarize_api_error() checks the timeout classifier first
(all later branches produce blank output for message-less exceptions);
gains optional provider/model kwargs, backward compatible.
- agent/conversation_loop.py: the three retry-loop summary sites pass
provider/model context.
Tests: 12 new (pure classifier + AIAgent integration + empty-str
precondition pin). Sabotage-verified: disabling the classifier fails the
regression tests with the historical blank summary (assert '').
Covers the behavior shipped in #80807 (background dispatch for
cronjob action='run') and #80838 (per-run '## Run Context' prompt,
gateway-loop delivery): immediate return with handle, completion
re-entering the conversation, in-flight dedupe, transient context
injection with prompt scanning, and the sync fallbacks.
Fixes#61495
When manually triggering cron jobs from a live Matrix session, delivery
would fail with "Timeout context manager should be used inside a task"
because the aiohttp.ClientTimeout context manager requires a proper asyncio
task context.
Use asyncio.wait_for() instead of aiohttp.ClientTimeout to avoid this error,
following the same pattern as the Weixin platform (gateway/platforms/weixin.py).
Changes:
- Remove aiohttp.ClientTimeout(total=30) from ClientSession constructor
- Wrap the send operation in a nested async function (_do_send)
- Use asyncio.wait_for(_do_send(), timeout=30) for timeout handling
- Catch asyncio.TimeoutError explicitly and return clear error message
Salvaged from PR #57342 by @liuhao1024 (with the injection-scan half
from PR #57360 by @ghedeselmabot): cronjob(action='run', prompt=...)
silently discarded the prompt argument — per-run context never
reached the spawned cron session.
The prompt is now threaded as extra_prompt through the whole chain
(cronjob run action → _try_dispatch_background_run/_execute_job_now →
_run_claimed_job → run_one_job → run_job → _build_job_prompt) and
appended to the stored prompt under a '## Run Context' header for
that single fire only — never persisted to the job definition. It
passes the same strict _scan_cron_prompt injection scan as stored
prompts before firing, and works identically on the background and
sync fallback paths.
Test fakes across tests/cron/ updated to accept the new kwargs
(sibling-test blast radius from the signature change).
Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
Pins the scheduler-boundary contract: extra_prompt is appended under
'## Run Context', does not mutate job['prompt'], and the header is
absent when extra_prompt is omitted.
Addresses review feedback from harjothkhara on PR #57342.
Salvaged from PR #53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.
Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, #60432) and mark_running_jobs_interrupted,
which previously could not see them.
The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.
Co-authored-by: izumi0uu <izumi0uu@gmail.com>
A manual cronjob run executed the job synchronously on the calling
agent's tool thread. A cron job is a full agent run that routinely
takes minutes to hours, so the parent turn sat inside ONE tool call
the whole time: uninterruptible (the interrupt flag is only checked
between loop iterations) and serial (a batch of manual runs executed
one by one). A Telegram session that kicked off dozens of new jobs
'right now' was wedged for hours ignoring every interrupt.
action='run' now rides the async-delegation rail delegate_task
background mode uses: the at-most-once claim is taken synchronously
(so paused/missing/already-firing jobs still report immediately),
the run executes on the shared daemon executor, the tool returns at
once with a delegation handle, and the job's outcome re-enters the
conversation as a type='async_delegation' completion event through
the existing completion-queue drains (CLI + gateway) — preserving
message-role alternation and the prompt cache.
Sync fallbacks preserved:
- no routable session (direct Python callers, hermes cron run)
- async delivery unsupported (hermes -z, cron child sessions,
Kanban workers, stateless HTTP)
- dispatch pool at capacity (claim already taken — runs inline
rather than stranding it)
The completion block reports ok/failure, delivery target, next
scheduled run, and an excerpt of the job's saved output.
Inspired by virgiliojr94/book-to-skill (MIT): /learn now picks the skill
shape by the source. Workflows and small sources still get one tight
SKILL.md; books, paper stacks, specs, and large doc corpora get a
knowledge-base layout — a lean always-loaded SKILL.md index plus one
distilled file per chapter/topic under references/, loaded on demand via
skill_view so query cost stays proportional to the answer.
- agent/learn_prompt.py: new _KNOWLEDGE_SKILL_STANDARDS block (index +
per-chapter references/, structure-not-summary distillation, never
reproduce source passages, fold-in instead of duplicating) and a
_SOURCE_HYGIENE block pinning extracted source text as data and
dropping invisible/bidi Unicode (Trojan Source class). Clarified that
the ~200-line cap and hub-skill ban apply to SKILL.md itself, not a
knowledge skill's own references/ files.
- tests: contracts for the knowledge-base layout, the three embedded
standards blocks, and the source-hygiene coverage.
- docs: skills.md documents the knowledge-base shape.
The in-repo skill-authoring skill taught the validator's ceilings (1024-char
descriptions, 'Use when ...' phrasing) instead of the repo's review standards,
so agents following it produced skills that fail review: 240+ char
descriptions, author 'Hermes Agent' with no human credit, no bundled-vs-
optional decision, dangling related_skills, no platforms audit, no tests, no
docs regen, and machine-local /home/bb/... paths baked into prose.
Rewritten to teach the hardline standards from AGENTS.md:
- description <= 60 chars, one sentence, ends with period
- author credits the human contributor first
- bundled vs optional tier decision (5+ sessions/month bar; default optional)
- no router/index/hub skills
- platforms: audited against actual scripts, POSIX-signal table
- related_skills must resolve in-repo
- Hermes-tool framing instead of raw shell prose
- tests at tests/skills/ + docs regen with scope discipline
- removed machine-local paths; validator limits marked as NOT the standard
After `hermes update`, the desktop sidebar showed "No sessions yet" until
the user's first message. #72424 added sessions.last_activity_at, which
list_sessions_rich now selects — but column adds only land through
_reconcile_columns() in the writable _init_schema, and read-only opens
skip that by design. Every sidebar read path opens state.db read-only, so
each poll raised "no such column: s.last_activity_at" until the first
prompt's lazy session-row persist forced a writable open and reconciled.
A heal for exactly this class already existed (_open_session_db_for_profile
probes the read-only handle and does a one-time writable reopen on
staleness), but its probe was a hand-written four-column list that never
learned last_activity_at — it went stale three days after shipping. And the
batched sidebar route (/api/profiles/sessions/sidebar) bypassed the helper
entirely, swallowing per-profile failures into an errors array the desktop
never surfaces, so the incident produced an empty sidebar with clean logs.
The fix removes the maintenance burden instead of paying it once more:
- hermes_state_schema.schema_read_probe_statements() derives one
`SELECT <every declared column> FROM <table> LIMIT 0` per table from
SCHEMA_SQL via the existing _parse_schema_columns() — the same source of
truth the writable reconciler diffs against, so any future ADD COLUMN is
probed with no list to update. Column references are table-qualified:
an unqualified double-quoted identifier that fails to resolve silently
degrades to a string literal (SQLite's double-quoted-string misfeature)
and would make the probe pass on exactly the store it exists to catch.
- web_server splits the heal into a path-level _open_session_db_at_path
(semantics unchanged) so the cross-profile session routes can share it;
both profiles.py loops and _count_status_active_sessions (the remaining
raw read-only sibling) now open through it. The heal stays a helper
rather than a SessionDB classmethod on purpose: escalation-to-writable
must remain an explicit caller decision — update_cmd.py opens read-only
mid-update and must never write.
- Exhaustion guard: if the writable heal SUCCEEDS and the re-probe still
fails (a schema problem ADD COLUMN cannot express), the store is marked
exhausted — warn once, skip the probe, serve reads probe-less — instead
of re-running the full writable init on every poll against a possibly
live DB. A FAILED writable open (transient lock) is deliberately not
recorded, so the next poll retries the heal.
- The per-profile swallow sites in profiles.py now also log a deduplicated
warning, so a persistent read failure is loud in errors.log even though
the response errors array stays invisible to the sidebar.
Tests: probe/SCHEMA_SQL coverage invariants (tests/test_schema_read_probe.py),
last_activity_at added to the /api/sessions heal parametrize, a sidebar-route
heal test reproducing the shipped symptom (errors == [] and the session
returned against a store missing the column), and an exhaustion test pinning
exactly one writable open. The sidebar and last_activity_at tests fail on
main.
It was a 1:1 rebuild of the core statusbar gateway item and shipped
enabled by default, so the pill showed up twice. Core chrome stays in
shell; demos that clone it belong in hermes-example-plugins.
Repairs what is already in the transcript: reasoning persisted before the
backend fix, and any provider still gluing its parts. Handles both shapes —
heading-onto-heading (the **** run) and prose-onto-heading (vercel/ai#6742).
Verified against 46 real glued messages from a gpt-5.6-sol session; all repair
cleanly and idempotently.
The native Responses stream does carry summary_index, so the part boundary is
structured data here rather than something to infer. Break on a change of
index, and leave streams that send no index (plain reasoning_text) untouched.
Reasoning-summary models emit one reasoning_content delta per completed
summary part, each a self-contained bold heading. The Responses API delimits
those parts with summary_index; the OpenAI chat wire carries no such field —
verified live against Nous Portal, whose reasoning chunks contain nothing but
delta.reasoning_content — so concatenating them glued every part into one
unspaced, half-bold paragraph.
Re-derive the boundary from the signal the wire does carry: a delta opening a
closed bold heading against a mid-line tail. This matches Hermes own Responses
adapter, which already joins its summary parts with a blank line.
Pinned was capped at half the viewport by its own nested scroller, so past
roughly a dozen pins the rest were reachable only by scrolling inside a
scroller — a pin you have to go hunting for isn't doing its job.
Drop the cap and let the section grow into the sidebar's existing scroll,
and stop virtualizing Pinned: virtualization needs a bounded viewport to
measure against, which is exactly what's being removed. No count badge, no
"show more" — pin as many as you want and they all render.
Also back-fill pins on the API-server list route, which was the one list
path still windowing purely on recency.
On real sessions the button showed up two or three turns from the bottom, over
a screen and a half of transcript that had barely painted anything.
The budget now spends paint weight, which is what the DOM actually mounts, and
600 units of it — 10-20 agentic turns measured, where a tool-heavy turn prices
at 30-90 and a plain exchange at 5-10. A floor of 8 turns covers the session of
enormous turns that a weight-only cut still truncates hard; it applies to a
real page only, so the small first-paint commit stays small and the backfill a
frame later fills the rest.
Measured on four stored sessions at the same budget: one went from 3 turns
visible to 12, another from 3 to 4, two unchanged. The store window still caps
what the DOM can reach at all.
One weight function served two budgets that protect different things. The
store window protects the heap: every message it admits is normalized into the
runtime repository whether or not the transcript collapses it, so it has to
price the payload it holds. The DOM budget protects the paint, and what a turn
mounts is decided by the grouping, not by the bytes behind it.
Charging the DOM budget for payload made it count work that never happens. A
settled run of twelve reads is one grey summary line, a thought is one
collapsed disclosure, a todo is hoisted out of the transcript, and an image is
one img however long its data URL — all of it priced as if fully expanded.
messageStoreWeight keeps the payload price for the window. messagePaintWeight
prices what mounts: collapsed rows flat, silent rows free, cards fixed, and
markdown and diffs by size, since those really do build DOM. Both share one
character ceiling per message rather than one per part.
The transcript decides what a tool call draws and the render budget has to
price it. Both sides need the same answer, so the classification moves out of
the tool renderer into its own module rather than the budget importing the
formatting and i18n weight of fallback-model to ask one question.
Adds isSilentTool for the rows that render nothing at all: todo is hoisted to
its own panel, and a reaction's UI is the emoji on the bubble.
Dragging one row switched the entire sidebar into a frozen manual mode
with no date dividers at all — permanently, for every session, because
the manual order replaced the recency sort outright instead of layering
on it. Chronology and ranking are separate concerns: keep the calendar
buckets where recency put them and apply the hand-picked order only
within a bucket, so a drag ranks a chat among its own day's chats and the
dividers survive.
Rows move as clusters, so a reorder can't strand a branch child from its
parent, and a session the saved order doesn't name keeps the slot recency
gave it. Two supporting fixes fall out: dnd-kit now receives the ids it
actually renders (it was handed the unrendered session order, so a drop
computed its target against a list the user wasn't looking at), and an
older page that loads no longer jumps above the hand-picked rows — new
ids fold in by position rather than all hoisting to the top.
Two ways a pin got misfiled. The duplicate: a pin is stored on the
durable lineage root, but recents, the messaging slice and the backend
project tree are three independent fetches and each can surface the same
conversation under either its live tip or its root — so the filter
compared one identity against the other, missed, and the session rendered
in both Pinned and its project group. Match on every id the pin is
reachable under.
The lost reorder: a drag only reports the pins whose row is loaded, and
setPinnedSessionOrder required that list to match the stored one in
length, so a single unresolved pin discarded the whole reorder. Treat it
as a permutation of a subset — re-slot the named ids, leave the rest.
The guard that stops a stale list page from reverting a fresh pin was
released on the PATCH's own ack. A list request issued just before the
write is slower than the write, so it lands after the ack still carrying
the old value, with no guard left to fence it: the pin flips back and the
next reconcile pushes that wrong value to the server, making it durable.
Keep the guard until a page actually confirms the value written, with a
cooldown so a row that never returns can't fence itself forever, and drop
it outright when the write fails — the server never changed, so it stays
authoritative.
Also reset the mirror bookkeeping on a gateway switch. mirrored/pending
are per-backend facts; carrying them across a re-home told us the pins
were already pushed to a backend that has never seen them.
The list endpoints deliberately back-fill pinned conversations past their
LIMIT, then the client sliced the response back down to that same limit
and threw them away — so only pins that happened to land inside the most
recent page ever rendered, which reads as a cap on how many sessions you
can pin.
Keep the back-filled rows when trimming, and discount them from the
"window came back full" test that drives Load more. Counting a back-fill
as a loaded row invented a page that could never be fetched, leaving a
Load more button that refetched the same rows forever.
PATCH /api/sessions/{id} only accepted title and end_reason, so the
`pinned` flag the desktop sends was rejected as an unsupported field —
and the client swallows that error. Pins lived in one app's localStorage
and never reached state.db, which also meant the server-side auto-archive
sweep was free to hide the chats a pin exists to keep.
Accept pinned and archived as booleans, route them to the SessionDB
setters that already existed, and include both in the serialized session
so clients can reconcile against server truth.
The six GUI tools moved out of `terminal` into their own toolset; the tables
still described them as check_fn-gated members of it, and as available to every
hermes-* platform bundle.
The rule the preview-tool bug broke, written down so the next GUI-adjacent tool
does not rediscover it: the client and the backend are separate machines, so
"was this process spawned by Electron?" cannot answer "is a GUI watching?".
Names the working pattern (toolset gates the surface, check_fn answers only
reachability or user opt-in), the process-wide check_fn TTL cache that makes it
the wrong home for a per-session answer, and the test that would have caught
it — assert the GUI session gets the tool with the env var absent.
The pane, in-app browser, and reaction tools were gated on HERMES_DESKTOP=1 —
an env var set only on backends Electron spawns itself (local and SSH). A
desktop client connected to a plain URL gateway or Hermes Cloud lost all six:
they were stripped from the schema before the model saw them, on the same
backend whose platform hint was telling it "you are chatting inside the Hermes
desktop app". open_preview, read_preview, read_terminal, close_terminal,
focus_pane, and react_to_message were all silently absent.
The client is not the host. Capability now resolves from the session's own
source, which session.create already carries:
- The six tools move into a `desktop_ui` toolset, off _HERMES_CORE_TOOLS so no
other platform pays their schema.
- _gui_surface_toolsets(platform) folds `desktop_ui` (and the existing
`project` tools) into the GUI gateway's resolution when the session's
platform is the desktop app — the same answer on every topology.
- check_fn drops the env probe. It kept the one thing that is genuinely a
per-process/user fact: react_to_message's display.message_reactions opt-in,
which the desktop mirrors onto whichever gateway it is connected to.
react_to_message was doubly broken: it read that toggle behind the env gate, so
even a local-backend user's Settings toggle could not reach a remote session.
The embedded terminal pane keeps working correctly the other way round: it runs
`hermes --tui` against a desktop-spawned backend, and a tui-sourced session
gets no GUI tools even though HERMES_DESKTOP=1 is set on that process.
Review follow-ups on the salvage:
- The helper script's success check accepted any "PID" line, including
the "PID" = -1 a recently-crashed job reports — while the in-process
path's _parse_launchd_pid_from_list_output rejects non-positive PIDs.
Both bash sites now require a positive PID (grep -qE '"PID" = [0-9]+;')
so the two paths enforce the same supervised-PID standard.
- _graceful_restart_via_sigusr1's drain-wait tail was a duplicate of the
new _wait_for_pid_exit — now delegates to it.
- Stale comments: the ancestry-detection framing at the top of the reload
block, and the exhaustion log's '(refresh ran outside gateway process
tree)' which is false on the new helper-spawn-failure fallback path
(now '(in-process fallback path)').
The reload retry loop treated `launchctl list <label>` exit 0 as success,
but exit 0 also covers a registered-but-not-running definition (macOS 26+
`state = not running`) — the same trap _probe_launchd_service_running
already guards against. Require a PID so success means launchd is
supervising a live process, in both the Python loop and the shell helper.
Verified against live launchd: a RunAtLoad=false job reports exit 0 with
no PID, which the old check accepted and the new one rejects.
Note this is NOT what distinguishes a draining instance — measured, the
label deregisters within ~1s of bootout while the old process drains on.
Waiting for the old PID to exit is what covers that.
Reload chose the in-process bootout/bootstrap path based on POSIX
ancestry, but bootout tears down the job's process coalition, and
coalition membership is inherited at spawn and survives reparenting.
A gateway-spawned process reparented to PID 1 is no longer an ancestor
yet still dies with the coalition, so the retry loop was killed
mid-bootstrap and nothing re-registered the label (KeepAlive can't
revive a job launchd no longer knows about).
- always prefer the detached transient-job helper; it's also correct
when genuinely outside the coalition, just asynchronous
- wait for the old gateway PID to exit before bootstrapping; bootout
only sends SIGTERM and every bootstrap during the drain fails EIO
- fall through to the in-process path when the helper can't spawn
instead of leaving the plist rewritten but never reloaded
Per the 'when in doubt, optional' rule — niche prediction-market data
skill that sees no regular use; belongs alongside stocks in the finance
optional category rather than the default bundle.
Install via: hermes skills install official/finance/polymarket
The turn finalizer already hands back steer text that queued after the
final tool batch — result["pending_steer"], with the comment "hand it
back to the caller so it can be delivered as the next user turn instead
of being silently lost." Every interactive surface honors that contract
(cli.py, gateway/run.py, tui_gateway/server.py all requeue it). The
delegation layer doesn't: _run_single_child never reads it, so a steer
queued into a delegated child that finishes first vanishes with no trace
in the completion entry. There is also no sanctioned sender: the registry
has interrupt_subagent() but no redirection-side mirror, and session.steer
cannot reach children (lazy watch sessions have agent=None, so it 4010s).
Complete the contract for delegated children — both halves:
- steer_subagent(subagent_id, text): redirection-side mirror of
interrupt_subagent(). Resolves the live child in _active_subagents and
queues text via AIAgent.steer(). True means queued, not delivered.
- missed_steer retention: when the child's result carries pending_steer,
_run_single_child names it on the completion entry (missed_steer field
plus a summary note) so the parent can re-issue the guidance instead of
trusting it landed. This is what makes adding a sender safe: without it
the finish-before-drain race silently loses the text — the exact loss
the finalizer contract exists to prevent.
- subagent.steer gateway RPC beside subagent.interrupt so programmatic
hosts (dashboard, voice layers, ACP bridges) get an in-tree caller;
catalogued in programmatic-integration.md.
- docs: "Steering a Running Subagent" section in delegation.md covering
the queued-vs-delivered semantics.
Tests: registry-level steer coverage (delivery, unknown id, empty text,
dead record, raising agent), the finish-before-drain race retaining
missed_steer, and the RPC contract (4000/4002 validation, queued and
rejected envelopes).
Tests like tests/gateway/test_goal_verdict_send.py monkeypatch Path.home()
to a tmpdir; resolving the guard's 'real root' through Path.home() made the
test's own hermetic home look like production (false positive). Resolve via
os.path.expanduser/LOCALAPPDATA instead — the hermetic conftest never
rewrites HOME, so this always names the actual production root.
Forensics on a live developer machine found pytest fixture rows inside the
REAL ~/.hermes/state.db — sessions with chat_id 'chat-1', '123', 'wx-chat',
and gateway_routing rows whose scope was literally under /tmp/pytest-of-*/.
A pytest-spawned process also opened the live DB and flipped its journal
mode (journal_mode=DELETE fallback on SQLite 3.50.4) under the WAL-mode
gateway writer, destroying committed transcripts ("Persisted transcript
lagged live cached history ... possible FTS write corruption", 15+
occurrences). The existing live-system guard covers kill primitives but not
the SessionDB/SessionStore write paths.
Root cause (leak vector): the session-level HERMES_HOME sandbox in
tests/conftest.py only created a tempdir when HERMES_HOME was UNSET. On a
machine where the shell (e.g. gateway-launched, or an exported
HERMES_HOME=~/.hermes) hands pytest the production home, the sandbox was
skipped entirely — every argless SessionDB()/SessionStore() and every
collection-time DEFAULT_DB_PATH froze onto the real state.db.
Fixes (fail the class, one owner):
* hermes_state._ensure_test_isolation(): single choke point wired into
SessionDB.__init__ (every construction, incl. read_only). Under pytest
(PYTEST_CURRENT_TEST / PYTEST_VERSION — inherited by subprocess
children), a db path resolving to <real-root>/state.db or
<real-root>/profiles/<name>/state.db raises RuntimeError('live-system
guard: ...') before any connection, mkdir, or journal-mode pragma.
* tests/conftest.py: session sandbox now also tempdir-redirects a pre-set
HERMES_HOME that points at the production root (the actual escape
vector); kanban deny-list capture updated to match. New autouse
_state_db_write_guard fixture honors the existing
@pytest.mark.live_system_guard_bypass marker as the escape hatch and
feeds custom (non-~/.hermes) production roots into the guard deny-list.
* gateway/session.py: SessionStore.__init__ no longer swallows the guard's
RuntimeError into the JSONL fallback — guard trips are loud.
* tests/hermes_state/test_live_db_isolation_guard.py: behavioral
regression tests — production paths (direct, profile, read-only,
unnormalized, default-resolution) raise; tmp HERMES_HOME works; bypass
marker works; SessionStore re-raises guard errors but still degrades on
ordinary failures; subprocess child without HERMES_HOME is refused while
a hermetic child succeeds.
No new HERMES_* env vars; no hardcoded ~/.hermes (platform root comes from
hermes_constants._get_platform_default_hermes_home()).
The WAL-reset-vulnerability gate (#70055 lineage) could flip a LIVE WAL
database to journal_mode=DELETE while another process was writing to it.
Observed on state.db (Aug 5): a pytest process on the repo .venv (SQLite
3.50.4, vulnerable) opened the live ~/.hermes/state.db while the gateway
(SQLite 3.53.1, WAL) held it, downgraded the journal mode, and destroyed
the gateway's committed-but-uncheckpointed WAL transactions (disk rows
went 10 -> 0 while memory held 185). cron/executions.db already had the
"leave WAL in place, no live downgrade under concurrent openers" rule via
the on-disk WAL probe; state.db and every other store shared the hole
whenever the mode PROBE itself was blocked by a concurrent opener's locks
("could not read the mode" was treated as "not WAL" -> flip anyway).
Generalized in the single journal-mode owner (apply_wal_with_fallback),
covering ALL call sites (state.db, kanban.db, projects.db,
cron/executions.db, delivery_ledger, async_delegation,
verification_evidence, discord recovery, response_store.db,
memory_store.db):
- _set_journal_mode_no_wait(): the only journal-mode switch primitive for
non-WAL targets. Forces busy_timeout=0 around the pragma so SQLite's own
exclusivity requirement for leaving WAL becomes the concurrent-opener
detector — any other opener (this process or another) makes the flip
fail immediately instead of waiting out a busy timeout and sneaking the
flip in under a live writer.
- Vulnerable-SQLite gate: an unreadable journal mode (probe blocked) now
means "ownership not provably exclusive" — leave the mode untouched and
warn, never flip. A lock conflict on the flip itself likewise leaves the
mode alone.
- Configured journal_mode=delete: refuses (raises) rather than downgrading
blind when the mode cannot be verified under a concurrent opener.
- Filesystem-incompat fallback: re-raises instead of downgrading when the
on-disk mode cannot be verified.
- New/exclusively-owned DBs on vulnerable builds behave exactly as before
(DELETE gate retained per #70055).
Behavioral tests use a REAL second process (and a real second connection
holding an exclusive lock) with the blocked-state assertions running WHILE
the holder owns the DB, plus exclusive-ownership downgrade-still-happens
coverage.
Two live failures on the same guard (cron/lifecycle_guard.py), both of
which blocked legitimate diagnostics from inside the gateway:
1. Crash class: the referenced-script walk read compiled binaries as if
they were shell scripts. Reading/inspecting a referenced file is now
best-effort by construction: executable magic numbers (ELF, PE,
Mach-O fat/thin) short-circuit before any full read via a 4KB sniff,
NUL-bearing heads are skipped as non-scripts, and unreadable paths of
every kind (NUL bytes in the token, ENAMETOOLONG, missing files)
degrade to "nothing to scan" instead of raising. A second fail-safe
layer wraps the pure-string fallback so the boundary function stays
total even if the tokenizer itself fails.
2. False-positive class: the lifecycle regex matched its command shapes
inside DATA arguments — SQL string literals passed to sqlite3/psql
and grep/rg/journalctl patterns hunting for the lifecycle string in
logs. Added a fail-closed second-pass exemption: on a raw regex hit,
re-scan with data-sink executables' arguments masked; only a match
that survives (i.e. sits in command position) blocks. Masking is
skipped for pipes into shells/xargs, command/process substitution,
sqlite3 dot-commands and psql backslash escapes, so it can only ever
allow, never miss.
Behavioral tests: exact live false-positive shapes as negatives, the
smuggling shapes as positives, the kill-primitive positive catalog
unchanged, and an adversarial never-raises suite (NUL bytes, non-UTF-8,
/dev/*, directories, missing files, magic-prefix binaries).
Teknium
liuhao1024
Fly
Jeffrey Quesnelle
brooklyn!
hermes-seaeye[bot]
rob-maron
kshitij
Rob Hilgefort
SmokeDev