The result-error path in _deliver_result is not inside an except block,
so sys.exc_info() always returns (None, None, None) — the condition was
always False. Simplify to a plain logger.error call with accurate comment.
hermes-cron-tick.service starts without TELEGRAM_HOME_CHANNEL/DISCORD_HOME_CHANNEL
in the unit env; the per-run load_hermes_dotenv reload lived only on the agent
path (after the no_agent short-circuit returns), so every deliver=telegram/all
script job failed with 'no delivery target resolved'. Load the dotenv at the top
of the no_agent branch; override=False keeps the gateway's in-process tick
behavior unchanged.
Follow-ups from review of #82049:
- extract append_user_instruction() into agent/skill_commands so the
stable-prefix construction cannot drift between the skill and cron
builders (the registered prefix must stay a byte-prefix of the built
message); cron no longer imports the private _SINGLE_SKILL_INSTRUCTION
- add the startswith guard to the skill builder registration site,
matching the stronger cron guard
- rename _MAX_BYTES to _MAX_CHARS (sum(map(len, ...)) counts characters,
not bytes) and correct the comment
- collapse find_stable_prefix's two-lock dance into a single critical
section (scan is <=32 short-circuiting startswith calls, measured
2-4us; drops the snapshot copy and the TOCTOU re-check)
- document the split-shape lifetime (marked-endpoint window) in the
module docstring
- add a contract test for the helper's byte-prefix invariant
(mutation-checked)
Webhook/cron skill invocations concatenate a large static scaffold
(activation note + expanded skill body) with a small volatile tail
(ticket payload, timestamps) into one user string, and the Anthropic
cache planner marked that whole string as a single atomic block — so a
few changed tail bytes forced a full cache rewrite on every invocation.
Instead of re-parsing scaffold marker strings out of the message at
request time (fragile when a payload or skill body quotes the marker),
the builders now register the exact stable-prefix bytes in a small
process-local LRU registry at construction time. The cache planner
splits a registered user string into [marked stable prefix, unmarked
volatile tail] request-locally; canonical session history stays a plain
string, and the failover stripper flattens the split back byte-exactly
via an O(1) registry lookup. Unregistered messages keep the existing
whole-message policy.
Covers the single-skill builder (webhook + slash command + TUI) and the
cron job prompt assembler (multi-skill, bundles, skipped-skill notice),
with registration guarded against injection-scanner sanitization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Move classify_persistence_error into hermes_state beside is_disk_full_error
and delegate the disk bucket to it (fixes 'ENOSPC writing state.db' and
'not enough space' classifying as unknown). run_agent keeps a thin lazy
delegating wrapper so the documented import path and fast import survive.
- Classify CompressionSessionBusyError (and its RPC-wrapped message forms)
as 'locked': the motivating #81227 failure mode stringifies to 'is being
compressed by another writer', which the substring heuristic missed.
- Export PERSISTENCE_ERROR_CAUSES and iterate it in the cron explainer
suppression instead of a hardcoded tuple, so a future cause bucket cannot
silently desynchronize cron delivery.
- Hedge the gateway locked/unknown recovery wording ('should already be
saved' instead of 'was recorded') to match the explainer - the early
turn-start persist may also have failed.
- Drop STATE_DB_WAL_WARN_BYTES (speculative dead constant with no consumer;
the pre-existing 50 MB doctor WAL check covers the warning).
- Tests: compression-busy classification, is_disk_full_error delegation,
causes-tuple coverage; mutation-checked red-green.
An enterprise deployment hit sustained SQLite write-lock contention on a
shared multi-gigabyte state.db (gateway + CLI processes writing
concurrently). Turns correctly failed closed with
session_persistence_failed, but the only user-facing wording claimed the
disk was full and the gateway rendered a generic failure.
The fast-fail semantics are deliberate and unchanged. This adds a pure
classifier (locked / disk / unknown) applied where the SQLite error is
still visible, threads the cause through the turn-completion explainer,
and stamps a machine-readable failure_reason
(session_persistence_failed:<cause>) plus a guaranteed non-empty error on
the result for downstream surfaces. The cron scheduler's explainer-text
suppression now matches every cause variant so refined wording cannot
leak into scheduled-job deliveries.
- Fix _usage_audit_path() to use _get_hermes_home() instead of hardcoded
Path.home() / '.hermes' (profile-safe resolution, sweeper finding)
- Rewrite skip_background_review tests to exercise finalize_turn() directly
instead of duplicating the guard expression (sweeper finding)
- Fix response_silent audit field to use _is_cron_silence_response()
instead of the buggy SILENT_MARKER substring check it was meant to
replace (simplify-code review finding)
- Remove dead 'model' in locals() guard — model is always in scope
before the try block (simplify-code review finding)
- Extract _stub_agent_for_finalize() helper to eliminate ~40 lines of
copy-pasted agent stubbing in tests (simplify-code review finding)
- Clean up 'Phase 0.5' instrumentation comments
Phase 0.5 of the Hermes Agent token leak mitigation plan: append a single
JSONL line to ~/.hermes/cron/usage_audit.jsonl after every cron LLM
invocation, capturing prompt/completion/total tokens, model, duration_ms,
deliver target, and error (when raised). Read from agent.session_*_tokens
which run_conversation already returns in its result dict.
Without this, we have no measured baseline to attribute token deltas to
subsequent mitigation phases. The plan's hard gate: observability lands
before any mitigation phase.
Writer NEVER raises — wrapped in a single try/except that logs a warning
on any json.dumps / mkdir / open failure so an audit-log bug cannot
break a cron job. Failure-path audit guard via locals() check covers
exceptions that fire before the fire_id is assigned.
No new dependencies, no new env vars (the plan rejected one in v2).
Tests: 7 new unit tests in tests/cron/test_usage_audit_logger.py covering
the success path, missing token info, swallowed writer exception, parent
dir creation, multiple appends, and unicode preservation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 wire-in + Vector 8 doc comment from
ralplan-hermes-token-leaks.md.
(1) Phase 8 wire-in: cron AIAgent construction now passes
skip_background_review=True. This suppresses the end-of-turn
skill/memory review fork (~30K tokens/event, ≤30K typical and
≤150K worst-case daily on bluenode) which has no human-in-the-loop
value for cron sessions.
(2) Vector 8 doc comment: a one-line comment immediately above the
AIAgent(...) construction documenting the verified-negative
finding that title generation does not run on the cron path
(maybe_auto_title is gateway/CLI-side only). Future contributors
won't accidentally introduce title-gen here without realizing it
would add ~600-1000 tokens/fire on a path that explicitly opts out
of memory/review/title overhead.
No new tests required for the doc comment (no behavior change). The
skip_background_review wiring is covered by the existing source-text
assertion in tests/agent/test_skip_background_review.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resumable ESTOP sentinel at $HERMES_HOME/ESTOP that halts NEW work only:
- agent/estop.py: sentinel engage/disengage/is_engaged (single stat, no
caching), optional reason + timestamp stored as JSON, paused_reply()
notice, check_paused() log-once-per-engagement helper. Corrupt/empty
sentinel still pauses (fail safe); a `touch ~/.hermes/ESTOP` works.
- cron/scheduler.py: tick() skips dispatch while engaged (logged once per
engagement, not per tick). Due jobs simply wait for the next tick after
resume — in-flight runs are never touched.
- gateway/kanban_watchers.py: dispatcher skips auto-decompose and worker
spawning while engaged; zombie reaping still runs and running workers
finish naturally.
- gateway/run.py: new gateway turns (post-auth, non-internal) get a brief
"Hermes is paused" reply instead of an agent run. Internal events
(in-flight background completions) bypass the gate.
- hermes_cli/subcommands/pause.py: `hermes pause [--reason]` and
`hermes resume`, wired into main() and _BUILTIN_SUBCOMMANDS.
- hermes_cli/status.py: `hermes status` shows a PAUSED banner (one stat).
- tests/test_estop.py: 20 tests — sentinel lifecycle, reason surfacing,
log-once, cron skip + resume, kanban gate, gateway paused reply +
internal bypass, CLI idempotence, builtin-set parity, status line.
Never kills in-flight work; resumable with no restart. Footprint ladder:
CLI command only, no new model tool, no new env vars.
Ported from: gastownhall/gastown estop.go (MIT); related prior art:
#26778 (/panic — kill/exit semantics, deliberately different: ours is a
resumable pause), #44617 (interrupt in-flight cron — out of scope here).
Validate a job's configuration BEFORE any agent machinery is constructed:
- missing provider API key (AuthError from a read-only
resolve_runtime_provider probe; skipped when a fallback_providers chain
is configured, since auth-fallback may rescue the run)
- attached skill not ready (skill_view readiness_status=setup_needed —
missing required env vars / commands / credential files)
- delivery platform unknown or unconnected (deliver=local/origin/all are
never checked; gateway-config load failures fail open)
On a failing check run_job returns a [blocked_config]-marked error without
constructing AIAgent/MCP/etc, so a misconfigured job never burns an LLM
call. run_one_job records last_status='blocked_config' and delivers the
alert exactly ONCE across ticks (persisted preflight_alerted bit — the
alert-once shape from the #73506 dead-pin auto-pause); the next healthy
run clears the marker so a future break re-alerts. Every preflight check
fails open: only an affirmative misconfiguration verdict blocks.
Config: cron.preflight (default true); `cron.preflight: false` restores
the old fail-during-run behavior. Documented in the cron user guide and
config defaults.
mark_job_run gains an optional status= override (unblocked call shape
unchanged) and drops preflight_alerted on any successful run.
Tests: tests/cron/test_preflight_config.py (blocked_config + no agent +
single alert across two ticks, healthy job unaffected, recovery clears
dedup, fallback-chain rescue, opt-out restores old behavior, skill
readiness miss, unknown delivery platform, deliver=local never loads
gateway config). Full tests/cron/ + cronjob tool suite green (525 tests).
Ported from: paperclipai/paperclip execution-semantics §5 (MIT);
in-repo precedent: #27948, #73506
- cron/notepad.py: SQLite-backed cron_notepad(job_id, key, value,
updated_at) store in its own profile-local db (cron/notepad.db),
following the executions.py connection/transaction pattern. APIs:
set_note/get_note/delete_note/list_notes/clear_notepad +
render_notepad_section. Documented size caps: 16KB per value,
128-char keys, 64KB per job total; oversized writes raise ValueError.
- cron/scheduler.py: inject non-empty notepads into the job prompt at
the context_from data-injection seam as a clearly-labeled
"Job notepad (persistent across runs)" section that also documents
the CLI write path. Empty notepad renders "" — byte-stable prompts
for jobs that never use the feature.
- hermes_cli/cron.py + hermes_cli/subcommands/cron.py:
`hermes cron notepad <job_id> [get|set|delete|list]` under the
existing cron subcommand tree (no new top-level command, no new
model tool — the agent writes via terminal + CLI).
- tests/cron/test_notepad.py: CRUD, durability, cap enforcement,
prompt injection, byte-stable empty case, read-failure resilience,
CLI handler + dispatch (TDD; watched fail first).
Inspired by: Amp (Sourcegraph) cron notepad (idea-level, proprietary —
zero code).
Add monitor-mode cron jobs: a cheap monitor source (monitor_script or
monitor_url) runs on every tick BEFORE any agent machinery is built.
Its output is hashed as exact bytes and compared to the hash stored
from the last agent-triggering tick:
- unchanged -> agent run suppressed entirely (no LLM, no delivery);
the tick is recorded as a silent no_change run visible in the
executions ledger doc
- changed -> a MONITOR CHANGE DETECTED block (capped unified diff of
previous vs current output + the new output) is injected into the
prompt via the existing extra_prompt seam, then a normal agent run
- first run -> always runs the agent with a baseline block
- source failure -> delivered as an ERROR alert, never treated as a
change; the stored hash is untouched so recovery to prior output
still suppresses
Implementation:
- cron/monitor.py (new): hash/diff/URL-fetch/state persistence.
monitor_script reuses _run_job_script (same ~/.hermes/scripts/
containment + interpreter rules); monitor_url is a bounded GET
(30s, 256KB, http/https only). Output is exact bytes by design —
scripts should emit stable output (documented).
- cron/jobs.py: additive job fields monitor_script / monitor_url /
monitor_state {last_output_hash, last_changed_at}. JSON job records
need no migration. create-time validation: sources are mutually
exclusive and incompatible with no_agent=True.
- cron/scheduler.py: one tight monitor gate in run_job between the
no_agent short-circuit and the LLM path (outside sibling-lane
regions). State persists in jobs.json + a per-job snapshot file, so
suppression survives scheduler restarts.
- tools/cronjob_tools.py: additive optional monitor_script/monitor_url
params on the cronjob tool (create + update, empty string clears),
path containment validated at the API boundary, surfaced in
_format_job.
- hermes_cli: --monitor-script/--monitor-url on `hermes cron create`
and `hermes cron edit`; `hermes cron list` shows the monitor source
and last-changed time.
Tests (tests/cron/test_monitor_kind.py, TDD): unchanged suppresses,
changed injects diff, first run always runs, hash persists across
module reload (restart), script failure is error-not-change with hash
untouched, create/update validation, tool wiring + path-escape reject.
Inspired by: ChatGPT Work monitor tasks (idea-level, docs-only);
enabler: #80774
- Share one HERMES_CRON_TIMEOUT parser (_cron_inactivity_seconds) between
run_job's inactivity monitor and the cwd-lock bound so the two sites
cannot drift - the bound must stay >= the inactivity limit or waiters
would fail while a healthy holder runs.
- Bind the timeout once per run_job; the raise reports the value that was
actually used for the wait.
- Timeout message covers the writer-blocked-by-readers path instead of
always blaming a workdir job.
- The finally's TERMINAL_CWD restore is now gated on _cwd_lock_acquired:
a fail-closed timeout raised before the env-set, so restoring there
replayed a pre-wait snapshot over the ACTIVE holder's live override
(pre-existing microsecond race from the #80912 snapshot placement).
- Comment accuracy: the bound is measured from the waiter's arrival; a
late-wedging or pre-agent-hung holder can outlive it.
The 120s bound added in #80912 proceeded WITHOUT the lock on timeout
(fail-open). That degraded mode fires on every overlap with a HEALTHY
long-running workdir job - the write lock is legitimately held for the
holder's entire agent run - and a workdir-less job that proceeds unlocked
executes its shell/file/code commands with the holder's process-global
TERMINAL_CWD override visible: silent wrong-directory execution, the
exact corruption _ReadWriteLock exists to prevent (see
test_reader_never_observes_writer_override). A degraded WRITER was worse:
it clobbered the active holder's override mid-run and later restored a
pre-wait snapshot over the holder's value.
Fail closed instead: on timeout the job errors loudly with an actionable
message (stagger the holder's schedule / drop its workdir) and is retried
on its next tick. A failed job is visible and recoverable; a job that ran
in the wrong directory is neither.
The bound is now derived from the cron inactivity limit
(HERMES_CRON_TIMEOUT, default 600s) + 60s margin instead of a flat 120s:
a wedged holder stops touching its activity clock, so the inactivity
monitor reaps it and releases the lock within that limit - waiters only
fail when even the monitor could not clear the holder. Healthy workdir
jobs shorter than the inactivity limit can no longer fail their waiters.
Design follows @necoweb3's #63959 (fail-closed semantic); its 30s flat
bound would have failed every waiter overlapping a healthy >30s workdir
run, which is why the bound is derived instead.
Follow-up to the salvaged registration contract:
- share one _raise_if_cron_registration_error() helper for the two
byte-identical dashboard 424 except-blocks (web_server + cron router,
via the existing late() seam)
- add endpoint-level 424 coverage for /api/cron/blueprints/instantiate
(previously only the sync worker was tested)
- give chat/CLI surfaces a human-facing user_message() (job name, no
exception class name) and add a recovery hint (pause/resume or update
re-registers via provider reconcile) to the model/REST message
- consolidate five inline provider test doubles into one ABC-subclassing
make_cron_provider conftest factory; the web_server test double now
subclasses CronScheduler so an ABC rename fails loudly
- narrow the wrapper facade to keyword-only (**kwargs) and route the
tool's partial-failure return through tool_error()
When acquire_write or acquire_read times out, call notify_all() before
returning False so waiters blocked by the timed-out thread's presence
(e.g. readers blocked by writer-preference _writers_waiting > 0) are
woken immediately instead of sleeping until the next external notify.
The _ReadWriteLock used for per-job TERMINAL_CWD serialization had
unbounded acquire_read() and acquire_write() — no timeout, no logging.
A wedged or extremely long-running workdir job silently parked every
concurrently-firing job behind the lock, leaving them stuck in
'running' with zero log output until gateway restart.
Changes:
- Add optional `timeout` parameter to _ReadWriteLock.acquire_read()
and acquire_write(), returning False on timeout
- Add _CWD_LOCK_TIMEOUT_SECONDS (120s) constant
- Use bounded acquire at the run_job() call sites with WARNING logging
on timeout, proceeding in degraded mode (same trade-off as #60703
for the cross-process flock)
- Guard release_write/release_read to only fire when the lock was
actually acquired
Degraded mode risks a leaked TERMINAL_CWD override into concurrent
jobs, which is strictly better than a permanently wedged scheduler.
Salvaged from PR #57342 by @liuhao1024 (with the injection-scan half
from PR #57360 by @ghedeselmabot): cronjob(action='run', prompt=...)
silently discarded the prompt argument — per-run context never
reached the spawned cron session.
The prompt is now threaded as extra_prompt through the whole chain
(cronjob run action → _try_dispatch_background_run/_execute_job_now →
_run_claimed_job → run_one_job → run_job → _build_job_prompt) and
appended to the stored prompt under a '## Run Context' header for
that single fire only — never persisted to the job definition. It
passes the same strict _scan_cron_prompt injection scan as stored
prompts before firing, and works identically on the background and
sync fallback paths.
Test fakes across tests/cron/ updated to accept the new kwargs
(sibling-test blast radius from the signature change).
Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
Salvaged from PR #53395 by @izumi0uu: the fire claim's 300s TTL is
routinely outlived by real cron jobs, so claim_job_for_fire alone
cannot stop a manual cronjob(action='run') from double-firing a job
the ticker (or another manual run) is still executing.
Extract the ticker's _submit_with_guard running-set check into shared
module-level helpers (try_register_running_job / release_running_job)
and register manual runs through the same set — one dedupe owner, no
drift. Manual runs also become visible to get_running_job_ids (the
gateway shutdown drain, #60432) and mark_running_jobs_interrupted,
which previously could not see them.
The background dispatch path pre-checks the running set so a mid-run
job reports 'already running' in the tool response immediately
instead of as a delayed error completion event; the authoritative
atomic check remains in _run_claimed_job on the worker.
Co-authored-by: izumi0uu <izumi0uu@gmail.com>
3-reviewer simplify pass (reuse/quality/efficiency) findings:
- cron/scheduler.py _run_job_script: the ORIGINAL that
lifecycle_guard._resolve_script_path documents mirroring had the exact
same unguarded expanduser() — a NUL-bearing script value survives
creation (the guard treats it as nothing-to-scan) and crashed the
scheduler at fire time with ValueError instead of a clean job failure.
Same ingestion contract applied; regression test added.
- lifecycle_guard._resolve_script_path: get_hermes_home() -> Path.home()
raises RuntimeError when neither HERMES_HOME nor HOME resolves
(arbitrary-UID containers); the cron entry point called it bare.
Caught -> None; totality test added.
- terminal_tool: stale 'cat ...' docstring updated to the bounded
head -c form.
- lifecycle_guard: dead 'script_text and' condition dropped (guarded by
'if not script_text: continue' directly above).
Efficiency reviewer: no material findings (measured — encode/expand
costs negligible vs walk I/O, no timing regression vs base).
Cron agents are constructed with skip_memory=True, so the memory
backend is not initialised — exposing the memory tool only gives the
model an unbacked tool that fails at runtime with 'Memory is not
available.' Add 'memory' to _resolve_cron_disabled_toolsets() so the
tool is stripped from the schema before the model can call it.
Fixes#38129.
Co-authored-by: Paolo Shamoon <Paolo@Dylans-Mac-Studio.local>
A kanban worker that fires a cron job in-process no longer leaks its task
identity into the cron agent.
The worker is a normal `hermes chat -q` CLI agent whose default toolset
includes `cronjob`, running with HERMES_KANBAN_TASK legitimately set in its
own environment. `cronjob(action="run")` calls run_one_job() -> run_job()
in that same process, so the cron AIAgent was misidentified as that worker:
kanban toolset force-added, kanban-worker protocol injected into its system
prompt, and kanban_complete defaulting task_id to $HERMES_KANBAN_TASK --
letting an unrelated cron job close the worker's task and overwrite real
results.
Fixed with a ContextVar (`non_dispatcher_owned_context`), not by clearing
os.environ. The env is process-global and shared with three concurrent
readers that all need the real values:
* the worker's own claim heartbeat -- run_agent._touch_activity ->
heartbeat_current_worker_from_env reads TASK/CLAIM_LOCK/RUN_ID, and the
cron-run heartbeat thread drives it every 10s. Clearing them silently
no-ops the heartbeat, so after DEFAULT_CLAIM_TTL_SECONDS (15 min) the
dispatcher reclaims a task whose worker is still alive and re-dispatches
it -- the same duplicate-work failure from the other direction.
* the gateway's kanban watchers, which do their own HERMES_KANBAN_BOARD
save/restore around a slow decompose_task() LLM call.
* concurrent cron jobs, which take a *shared* read lock
(_terminal_cwd_lock.acquire_read) and so interleave: job A clears, job B
snapshots empty, A restores, B clears and its restore no-ops -- the
worker's identity is destroyed permanently.
`is_dispatcher_owned_worker_context()` is now the single predicate every
HERMES_KANBAN_* identity gate consults before trusting those vars. It also
closes a pre-existing gap in agent/skill_utils.py, which read the vars
without consulting the delegate_task ContextVar at all; the `kanban` verdict
additionally bypasses _ENV_DETECT_CACHE, since a context-dependent answer
must not be memoized process-wide.
HERMES_KANBAN_BOARD/DB/WORKSPACES_ROOT are left untouched, so the #20074
board pin and the dispatcher's path overrides keep working.
Tests: 18 new, including thread-isolation, concurrent-cron-jobs, and an AST
invariant over _default_spawn that fails if the dispatcher gains a var that
is neither identity-gated nor explicitly classified behaviour-only. All six
mutations are caught, including one that reintroduces the os.environ clear.
tests/cron/ + kanban suites 440 passed; model_tools/skill_utils/boards 63
passed; ruff clean.
Reported and diagnosed by Geoff Friesen (#78961), who identified the symptom
and the exact gating mechanism.
Co-authored-by: Geoff Friesen <gfriesen1@users.noreply.github.com>
Re-derivation of #33612 by @LeonSGP43 onto the rewritten scheduler (the
original is 10,692 commits behind; its tick() no longer exists in that
shape, so this is a fresh minimal fix crediting the PR's idea).
The gateway's built-in ticker calls tick(verbose=False) every 60s. The
idle early-return was gated on 'verbose and not due_jobs', so idle
GATEWAY ticks fell through to load_config() + worker-pool resolution
every minute. Return early on ANY idle tick; keep the post-tick MCP
orphan sweep (main intentionally reaps orphaned stdio children on idle
ticks).
3 new tests; mutation-checked (restoring the verbose-gated guard fails
the config-skip test). 66 scheduler tests green.
- Guard the thread-id-as-chat_id normalization to Discord only; Slack
and Telegram adapters use parent_channel as chat_id for thread messages,
so the unconditional version broke their handoff keys.
- Apply the same Discord-specific guard to _seed_cron_thread_session in
cron/scheduler.py (sibling site with the same bug, docstring said
'Mirrors _process_handoff').
- Replace the change-detector test with contract tests that verify the
actual invariant: Discord handoff key == organic thread key, Slack
handoff key still uses parent channel (non-regression).
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.
Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.
Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
Review follow-ups on the #76287 salvage:
- scheduler.py no longer calls advance_next_run after the batch switch;
keeping the import invites a future test to patch the wrong seam.
- advance_next_run returns >= 1 instead of == 1 so a corrupted jobs file
with duplicate ids (both records advanced by the batch) still reports
True after advancing and saving.
The scheduler's pre-dispatch loop called advance_next_run per due job —
one full load_jobs() + one full save_jobs() of the jobs file each — so
N due jobs cost N reads + N writes of the whole file (gateway-restart
catch-up or co-scheduled bursts). advance_next_runs() does one load +
at most one save for the whole due set with identical per-job semantics;
advance_next_run() is now a thin wrapper over it.
Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms
(45x; 50 loads + 50 saves -> 1 + 1).
Tests: batch advances recurring and skips one-shots, single load + save
I/O pin (fails pre-fix — no such function), no save when nothing
advances, and per-job wrapper semantics unchanged. Related: #60946 and
#75833 both restructure this loop's call site for correctness — neither
addresses the I/O cost, and this batch primitive composes with either
dispatch design; happy to rebase onto whichever lands first.
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
Fixes#73973.
A finite one-shot whose dispatch was claimed (claim_dispatch increments
repeat.completed BEFORE execution) but whose run died before mark_job_run
was left permanently wedged: completed==times, last_run_at null, state
'scheduled'. The run-claim TTL blocked re-dispatch, and once it expired
the dispatch-limit guard silently removed the job with no output and no
error.
Two complementary fixes:
- cron/scheduler.py run_one_job: the outer handler now catches
BaseException, not just Exception. The inner run_job handler re-raises
CancelledError/KeyboardInterrupt/SystemExit after agent teardown, and
none of those are Exception subclasses, so the outer 'except Exception'
missed them and mark_job_run(False) was never called. Failures are now
recorded first (mark_job_run + finish_execution, each independently
guarded), then non-Exception BaseExceptions are re-raised to preserve
teardown semantics. Plain Exceptions keep the existing behavior
(recorded, return False, no re-raise). Empty str(e) (bare
CancelledError) falls back to the exception class name.
- cron/jobs.py: when either removal site (claim_dispatch or the
get_due_jobs dispatch-limit guard) drops a one-shot whose claimed run
never completed (last_run_at null), _write_wedged_oneshot_diagnostic
now writes an operator-visible .md into cron/output/<job_id>/ instead
of vanishing silently. Best-effort: diagnostics can never break the
removal. No diagnostic when last_run_at is set (normal completion
race).
Per-job cron inference pins are now user-owned: the agent-facing cronjob
tool schema no longer exposes model/provider/base_url, and the registered
handler ignores them even if a model hallucinates the old parameters.
Users set pins via the dashboard, hermes cron create/edit --model/--provider,
or jobs.json directly — and once set, a pin sticks until the user changes it.
Existing agent-era pins are grandfathered untouched.
New cron.model / cron.model_provider config keys give the cron fleet its
own default model, independent of the chat model. Fire-time resolution:
per-job pin > cron.model > HERMES_MODEL > model.default. An axis covered
by the explicit cron-fleet default is deliberate routing, not drift, so
the #44585 fail-closed guard skips it — switching your chat model with
/model or hermes model no longer breaks unpinned cron fleets.
- tools/cronjob_tools.py: drop model param from agent schema + handler;
remove now-dead _resolve_model_override
- cron/scheduler.py: cron.model/model_provider resolution + per-axis
drift-guard skip
- cron/jobs.py: snapshot resolution mirrors the new precedence
- hermes_cli/subcommands/cron.py + hermes_cli/cron.py: --model/--provider
on hermes cron create/edit
- hermes_cli/config.py: cron.model / cron.model_provider defaults
- docs: cron.md model-resolution tip rewritten
When an operator changes the global model/provider config, warn that
unpinned cron jobs with stored snapshots will fail-closed on their next
run. Adds a cron.model_drift_guard config opt-out (default true) for
fleets that should deliberately track changing global defaults.
Addresses #59031. Original PR #59177 by @doncazper.
Follow-up to the salvaged #71756: instead of webhook importing cron's
private _is_cron_silence_response, the loose autonomous-lane matcher now
lives in gateway/response_filters.py as is_autonomous_silence_response,
sharing LIVE_GATEWAY_SILENT_MARKERS with the interactive exact-marker
rule so the marker sets can never drift. Cron and webhook both delegate
to it. Interactive gateway behavior unchanged.
The default max_iterations/agent.max_turns budget was set when long
agentic runs were rare; complex tasks now routinely exceed 90 tool
calls. Raise the default to 500 across every surface that hardcodes
the fallback: AIAgent constructor, DEFAULT_CONFIG, CLI resolution
chain, gateway env bridge, cron scheduler, and TUI gateway. Explicit
user config values are unaffected (deep-merge preserves them; no
_config_version bump needed).
Docs (en + zh-Hans), CLI help text, tips, and pinned tests updated
to match.
Eliminates the separate import/set/clear dance for _SESSION_CWD by
passing cwd= directly to set_session_vars(), which already handles
the ContextVar set internally and clears it via clear_session_vars().
Also includes the exception message in the no_agent error path.
Follow-up to salvaged PR #70548 (#69396).
The cron scheduler was mutating process-global state in two places:
1. no_agent path called os.chdir() which changed the global process cwd,
leaking into concurrent gateway sessions.
2. The agent path set os.environ['TERMINAL_CWD'] which any gateway
session could read during context-file discovery via
resolve_context_cwd/build_context_files_prompt.
Fix:
- no_agent path: pass workdir as subprocess cwd parameter to
_run_job_script() instead of os.chdir(). The Python process cwd
is never mutated.
- Agent path: in addition to the lock-serialized TERMINAL_CWD,
also set the per-context _SESSION_CWD ContextVar from
agent.runtime_cwd. This ContextVar is scoped to the current
thread/context and NEVER leaks into other sessions.
resolve_context_cwd() checks _SESSION_CWD first, so the cron's
own context file discovery uses the correct workdir, while
gateway sessions (which have no override) fall through to their
own TERMINAL_CWD.
kshitij
aameobius
joaomarcos
Victor Kyriazakos
0xarkstar
Teknium
kshitij
Gille
JonthanaHanh
liuhao1024
Paolo Shamoon
Alex Fournier
kshitij
Shaun Prince
spfcraze
doncazper
Hermes Agent
Victor Kyriazakos