Two live failures on the same guard (cron/lifecycle_guard.py), both of
which blocked legitimate diagnostics from inside the gateway:
1. Crash class: the referenced-script walk read compiled binaries as if
they were shell scripts. Reading/inspecting a referenced file is now
best-effort by construction: executable magic numbers (ELF, PE,
Mach-O fat/thin) short-circuit before any full read via a 4KB sniff,
NUL-bearing heads are skipped as non-scripts, and unreadable paths of
every kind (NUL bytes in the token, ENAMETOOLONG, missing files)
degrade to "nothing to scan" instead of raising. A second fail-safe
layer wraps the pure-string fallback so the boundary function stays
total even if the tokenizer itself fails.
2. False-positive class: the lifecycle regex matched its command shapes
inside DATA arguments — SQL string literals passed to sqlite3/psql
and grep/rg/journalctl patterns hunting for the lifecycle string in
logs. Added a fail-closed second-pass exemption: on a raw regex hit,
re-scan with data-sink executables' arguments masked; only a match
that survives (i.e. sits in command position) blocks. Masking is
skipped for pipes into shells/xargs, command/process substitution,
sqlite3 dot-commands and psql backslash escapes, so it can only ever
allow, never miss.
Behavioral tests: exact live false-positive shapes as negatives, the
smuggling shapes as positives, the kill-primitive positive catalog
unchanged, and an adversarial never-raises suite (NUL bytes, non-UTF-8,
/dev/*, directories, missing files, magic-prefix binaries).
3-reviewer simplify pass (reuse/quality/efficiency) findings:
- cron/scheduler.py _run_job_script: the ORIGINAL that
lifecycle_guard._resolve_script_path documents mirroring had the exact
same unguarded expanduser() — a NUL-bearing script value survives
creation (the guard treats it as nothing-to-scan) and crashed the
scheduler at fire time with ValueError instead of a clean job failure.
Same ingestion contract applied; regression test added.
- lifecycle_guard._resolve_script_path: get_hermes_home() -> Path.home()
raises RuntimeError when neither HERMES_HOME nor HOME resolves
(arbitrary-UID containers); the cron entry point called it bare.
Caught -> None; totality test added.
- terminal_tool: stale 'cat ...' docstring updated to the bounded
head -c form.
- lifecycle_guard: dead 'script_text and' condition dropped (guarded by
'if not script_text: continue' directly above).
Efficiency reviewer: no material findings (measured — encode/expand
costs negligible vs walk I/O, no timing regression vs base).
- _sanitize_remote_script_text: compare re-encoded BYTES against the cap,
not characters — a >1MiB multibyte file truncated at the head -c byte
bound decodes to fewer chars than bytes and would have scanned the
truncated text instead of failing closed (the exact local/remote
divergence this PR closes).
- terminal_tool: replace the three hardcoded 1MiB literals with
lifecycle_guard._MAX_REFERENCED_SCRIPT_BYTES so the budget cannot
drift; use the redirect-safe 'head -c N < path' form from
tools/image_source.py so leading-dash paths stay out of argv.
- Public guard wrapper: drop the duplicate depth-0 direct scan — the
walk already runs it; the except-path now falls back to the pure
string scans, preserving the direct verdict when the walk crashes.
The guard feeds untrusted byte streams (tokenized binaries, remote cat
output) into OS-path and shell-text operations; every incident so far
(#76762, #77703, #77780, #78256, #77729) was hot-fixed with an except at
whichever frame crashed that week. tilllt's regression suite on #79454
showed 4 members of the class still open on merged main. Close the class
at three boundaries instead:
- _expand_candidate_path(): single ingestion chokepoint for path
candidates — reject NUL/empty tokens before any Path OS call and
tolerate ValueError/RuntimeError/OSError from expanduser (T1/T2, plus
the HOME-unset launchd crash). Both _resolve_terminal_script_path and
_resolve_script_path now go through it.
- _sanitize_remote_script_text(): apply the local-read contract (NUL =
binary = nothing to scan; >1MiB = fail closed) to whatever any
read_remote_script callback returns, at the recursion boundary — the
guard stops trusting its callbacks (T3/T4).
- contains_gateway_lifecycle_command_or_referenced_script() is now total
by construction: direct regex scans (pure string ops) run first; the
best-effort filesystem walk is wrapped so an unexpected failure logs a
warning and falls back to the direct-scan verdict instead of killing
every terminal command until gateway restart.
terminal_tool's remote fallback also bounds the read at the source
(head -c 1MiB+1 instead of cat), so a 166MB ELF never crosses the wire —
the superlinear-shlex 30-minute stall from #79838's field report drops
to a 0.02s fail-closed verdict.
Regression tests: tilllt's T1-T4 adopted verbatim, plus an adversarial
never-raises sweep (NUL paths, unset HOME, over-long paths) and a
walk-crash fallback test.
Cron agents are constructed with skip_memory=True, so the memory
backend is not initialised — exposing the memory tool only gives the
model an unbacked tool that fails at runtime with 'Memory is not
available.' Add 'memory' to _resolve_cron_disabled_toolsets() so the
tool is stripped from the schema before the model can call it.
Fixes#38129.
Co-authored-by: Paolo Shamoon <Paolo@Dylans-Mac-Studio.local>
A kanban worker that fires a cron job in-process no longer leaks its task
identity into the cron agent.
The worker is a normal `hermes chat -q` CLI agent whose default toolset
includes `cronjob`, running with HERMES_KANBAN_TASK legitimately set in its
own environment. `cronjob(action="run")` calls run_one_job() -> run_job()
in that same process, so the cron AIAgent was misidentified as that worker:
kanban toolset force-added, kanban-worker protocol injected into its system
prompt, and kanban_complete defaulting task_id to $HERMES_KANBAN_TASK --
letting an unrelated cron job close the worker's task and overwrite real
results.
Fixed with a ContextVar (`non_dispatcher_owned_context`), not by clearing
os.environ. The env is process-global and shared with three concurrent
readers that all need the real values:
* the worker's own claim heartbeat -- run_agent._touch_activity ->
heartbeat_current_worker_from_env reads TASK/CLAIM_LOCK/RUN_ID, and the
cron-run heartbeat thread drives it every 10s. Clearing them silently
no-ops the heartbeat, so after DEFAULT_CLAIM_TTL_SECONDS (15 min) the
dispatcher reclaims a task whose worker is still alive and re-dispatches
it -- the same duplicate-work failure from the other direction.
* the gateway's kanban watchers, which do their own HERMES_KANBAN_BOARD
save/restore around a slow decompose_task() LLM call.
* concurrent cron jobs, which take a *shared* read lock
(_terminal_cwd_lock.acquire_read) and so interleave: job A clears, job B
snapshots empty, A restores, B clears and its restore no-ops -- the
worker's identity is destroyed permanently.
`is_dispatcher_owned_worker_context()` is now the single predicate every
HERMES_KANBAN_* identity gate consults before trusting those vars. It also
closes a pre-existing gap in agent/skill_utils.py, which read the vars
without consulting the delegate_task ContextVar at all; the `kanban` verdict
additionally bypasses _ENV_DETECT_CACHE, since a context-dependent answer
must not be memoized process-wide.
HERMES_KANBAN_BOARD/DB/WORKSPACES_ROOT are left untouched, so the #20074
board pin and the dispatcher's path overrides keep working.
Tests: 18 new, including thread-isolation, concurrent-cron-jobs, and an AST
invariant over _default_spawn that fails if the dispatcher gains a var that
is neither identity-gated nor explicitly classified behaviour-only. All six
mutations are caught, including one that reintroduces the os.environ clear.
tests/cron/ + kanban suites 440 passed; model_tools/skill_utils/boards 63
passed; ruff clean.
Reported and diagnosed by Geoff Friesen (#78961), who identified the symptom
and the exact gating mechanism.
Co-authored-by: Geoff Friesen <gfriesen1@users.noreply.github.com>
The gateway terminal guard crashed with 'ValueError: embedded null byte'
(command never ran, exit_code -1) when a command invoked an ELF binary by
full path. _read_referenced_script correctly rejects the binary locally
(NUL in first chunk), but the read_remote_script fallback
(_read_script_in_env) then re-read the SAME file's bytes without a NUL
guard, decoded them, and fed machine code back into the scanner, which
re-tokenized it into a bogus NUL-bearing path and crashed at os.open.
- _read_script_in_env: skip content containing a NUL byte on both the
local-read and remote-cat branches (mirrors _read_referenced_script:
a binary is nothing to scan), so binary never re-enters the guard.
- _read_referenced_script: tolerate ValueError from os.open on a
NUL-in-path, alongside the existing OSError guard, so the guard can
never crash the terminal tool regardless of input.
Extends the #76762 NUL-safety fix (local path only) to the gateway's
remote-read fallback path.
Re-derivation of #33612 by @LeonSGP43 onto the rewritten scheduler (the
original is 10,692 commits behind; its tick() no longer exists in that
shape, so this is a fresh minimal fix crediting the PR's idea).
The gateway's built-in ticker calls tick(verbose=False) every 60s. The
idle early-return was gated on 'verbose and not due_jobs', so idle
GATEWAY ticks fell through to load_config() + worker-pool resolution
every minute. Return early on ANY idle tick; keep the post-tick MCP
orphan sweep (main intentionally reaps orphaned stdio children on idle
ticks).
3 new tests; mutation-checked (restoring the verbose-gated guard fails
the config-skip test). 66 scheduler tests green.
mark_job_run popped a finite one-shot from jobs.json the moment its
repeat limit was reached and returned early — discarding the
last_status / last_error / last_delivery_error it had just written.
Every finished one-shot vanished from `cronjob action=list` with no
inspectable record, and a delivery failure (agent succeeded, platform
send failed) was silently thrown away with it.
Changes:
- mark_job_run now retires a limit-reached one-shot as a terminal
record (state="completed", enabled=False, next_run_at=None) —
mirroring the existing next_run_at-is-None terminal branch — so the
final status and any delivery error persist and surface in the
cronjob tool's list output (which already emits last_delivery_error
and defaults to include_disabled=True).
- claim_dispatch's stale-job cleanup marks already-ran jobs completed
instead of popping them; genuinely wedged claims (last_run_at never
written) are still removed with the operator-visible diagnostic.
- Retention sweep in the due scan prunes completed one-shot records
older than cron.completed_retention_days (default 7; non-positive
disables) so jobs.json cannot grow unboundedly. Recurring jobs and
non-terminal one-shots are never candidates.
Tests: completion retains record + delivery error, list surfaces it,
completed jobs never re-dispatch, sweep prunes old / keeps recent /
ignores recurring / honors the disable knob; recurring lifecycle
unchanged.
On Linux, /usr/bin/python3 is >1MB, so the size check fired before
the NUL check could run — the binary was returned as unsafe=True
(blocked) instead of (None, False) (skip). Reorder: read the bounded
chunk first, check for NUL bytes (binary → skip), then check size
(oversized text → fail closed).
The gateway lifecycle guard (cron/lifecycle_guard.py) applied shell-style
tokenization and script-reference resolution to non-shell content, with two
regressions:
#77131 - every .py cron script using pathlib division was hard-blocked:
Path.home() / ".hermes" / ".env" tokenizes the bare "/" operator as an
executable path, which resolves to the filesystem root; the regular-file
check then fails closed as unsafe. Since Python runs under the
interpreter, never through a POSIX shell, the shell-script reference walk
is a false-positive generator on Python sources. check_gateway_lifecycle
now skips the walk for *.py scripts (the direct command regex still scans
the full text), and _iter_referenced_shell_scripts skips pure-separator
tokens.
#76762 - terminal commands invoking a binary by absolute path (e.g.
/usr/bin/python3) crashed the guard with ValueError: embedded null byte:
the walk read the binary's bytes, decoded them as text, and re-tokenized
machine code; the recursion then hit Path.resolve() on a NUL-bearing
path while only OSError was caught. _read_referenced_script now skips
NUL-containing files (binaries are not referenced shell scripts) and
resolve() tolerates ValueError.
Shell scripts (.sh/.bash/.zsh) keep the full deep scan; literal lifecycle
commands in .py scripts are still blocked by the direct regex. New tests
cover all four behaviors.
- Guard the thread-id-as-chat_id normalization to Discord only; Slack
and Telegram adapters use parent_channel as chat_id for thread messages,
so the unconditional version broke their handoff keys.
- Apply the same Discord-specific guard to _seed_cron_thread_session in
cron/scheduler.py (sibling site with the same bug, docstring said
'Mirrors _process_handoff').
- Replace the change-detector test with contract tests that verify the
actual invariant: Discord handoff key == organic thread key, Slack
handoff key still uses parent channel (non-regression).
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.
Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.
Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
Review follow-ups on the #76287 salvage:
- scheduler.py no longer calls advance_next_run after the batch switch;
keeping the import invites a future test to patch the wrong seam.
- advance_next_run returns >= 1 instead of == 1 so a corrupted jobs file
with duplicate ids (both records advanced by the batch) still reports
True after advancing and saving.
The scheduler's pre-dispatch loop called advance_next_run per due job —
one full load_jobs() + one full save_jobs() of the jobs file each — so
N due jobs cost N reads + N writes of the whole file (gateway-restart
catch-up or co-scheduled bursts). advance_next_runs() does one load +
at most one save for the whole due set with identical per-job semantics;
advance_next_run() is now a thin wrapper over it.
Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms
(45x; 50 loads + 50 saves -> 1 + 1).
Tests: batch advances recurring and skips one-shots, single load + save
I/O pin (fails pre-fix — no such function), no save when nothing
advances, and per-job wrapper semantics unchanged. Related: #60946 and
#75833 both restructure this loop's call site for correctness — neither
addresses the I/O cost, and this batch primitive composes with either
dispatch design; happy to rebase onto whichever lands first.
Extends the shared _GATEWAY_LIFECYCLE_PATTERN (used by BOTH the cron
creation-time guard in cron/lifecycle_guard.py and the terminal
execution-time hard-block in tools/terminal_tool.py) so Branch B covers
launchctl submit and bootstrap alongside kickstart/unload/load/stop/
restart, and normalizes POSIX shell line continuations before matching
so the exact multi-line reported shape in #62891 cannot slip past.
Also extends the execution-aware, label-independent detector
(contains_launchctl_submit_command, cherry-picked from #63272) to cover
launchctl bootstrap, since a neutral label like ai.hermes.svc-reload-tmp
defeats any label-anchored regex — the second production reproduction.
Regression tests cover both sites, including
`launchctl submit -l com.foo -- /path/gateway` and the bootstrap
variant, plus outside-gateway pass-through.
Branch B regex extension and continuation normalization drawn from
PR #62896; bootstrap coverage and test shapes drawn from PR #51003.
Co-authored-by: JackJin <1037461232@qq.com>
Co-authored-by: joelbrilliant <joelbrilliant1@gmail.com>
- Resolve guard cwd against get_session_cwd(session_key); fall back to env.cwd
when no session record exists yet, matching current main's per-session cwd
architecture.
- Make referenced-script reads backend-aware: local read first; if missing,
fall back to env.execute('cat ...') for SSH/Modal/Daytona backends.
- Reuse the recursive scanner in check_gateway_lifecycle so nested cron
wrapper scripts are caught, and resolve relative refs inside a script
against that script's directory.
- Add regression tests for remote-backend reads, two-session cwd, and nested
cron wrappers.
Verification: 80 passed tests/hermes_cli/test_gateway_restart_loop.py;
694 passed tests/cron; ruff + git diff --check clean.
Use database.journal_mode as the sole non-secret operator setting, preserve the vulnerable-SQLite safety gate and existing WAL databases, validate explicit DELETE results, document the active config path, and cover real SQLite openers with behavioral tests.
Add HERMES_JOURNAL_MODE env / database.journal_mode config for
virtiofs/NFS/SMB where WAL is not crash-safe. Route 5 bypass openers
through apply_wal_with_fallback so a single setting covers every .db
(#68545).
Four deferrals following the established truthy-skip / PEP 562
lazy-load patterns (PRs #22681/#22859 lineage). Rebased over #74194,
which independently landed the browser_tool half of this work — that
file is dropped here; the remaining four modules are untouched by it:
- tools/vision_tools.py: defer agent.auxiliary_client
(credential_pool -> hermes_cli.auth -> httpx -> rich, ~50 ms) to
first vision handler call. async_call_llm /
extract_content_or_reasoning stay patchable module attributes;
injected test mocks win over the loader.
- agent/model_metadata.py: defer 'requests' (+urllib3, ~27 ms of the
'import cli' waterfall) to the fetch functions. PEP 562 __getattr__
keeps patch('agent.model_metadata.requests.get') working.
- tools/browser_supervisor.py: websockets (~22 ms) imports on first
CDP connect; ClientConnection type under TYPE_CHECKING.
- cron/jobs.py: croniter (~15 ms) resolves on first cron-expression
use; HAS_CRONITER stays monkeypatchable (None = unprobed sentinel).
A/B vs current main incl. #74194 (median of 7, cold subprocess):
import cli 147 -> 132 ms (-10%)
import model_tools 244 -> 224 ms (-8%)
import run_agent 264 -> 244 ms (-8%)
Lazy-verify: importing the four modules no longer pulls requests /
croniter / websockets into sys.modules. 369 targeted tests green
post-rebase.
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
Fixes#73973.
A finite one-shot whose dispatch was claimed (claim_dispatch increments
repeat.completed BEFORE execution) but whose run died before mark_job_run
was left permanently wedged: completed==times, last_run_at null, state
'scheduled'. The run-claim TTL blocked re-dispatch, and once it expired
the dispatch-limit guard silently removed the job with no output and no
error.
Two complementary fixes:
- cron/scheduler.py run_one_job: the outer handler now catches
BaseException, not just Exception. The inner run_job handler re-raises
CancelledError/KeyboardInterrupt/SystemExit after agent teardown, and
none of those are Exception subclasses, so the outer 'except Exception'
missed them and mark_job_run(False) was never called. Failures are now
recorded first (mark_job_run + finish_execution, each independently
guarded), then non-Exception BaseExceptions are re-raised to preserve
teardown semantics. Plain Exceptions keep the existing behavior
(recorded, return False, no re-raise). Empty str(e) (bare
CancelledError) falls back to the exception class name.
- cron/jobs.py: when either removal site (claim_dispatch or the
get_due_jobs dispatch-limit guard) drops a one-shot whose claimed run
never completed (last_run_at null), _write_wedged_oneshot_diagnostic
now writes an operator-visible .md into cron/output/<job_id>/ instead
of vanishing silently. Best-effort: diagnostics can never break the
removal. No diagnostic when last_run_at is set (normal completion
race).
Per-job cron inference pins are now user-owned: the agent-facing cronjob
tool schema no longer exposes model/provider/base_url, and the registered
handler ignores them even if a model hallucinates the old parameters.
Users set pins via the dashboard, hermes cron create/edit --model/--provider,
or jobs.json directly — and once set, a pin sticks until the user changes it.
Existing agent-era pins are grandfathered untouched.
New cron.model / cron.model_provider config keys give the cron fleet its
own default model, independent of the chat model. Fire-time resolution:
per-job pin > cron.model > HERMES_MODEL > model.default. An axis covered
by the explicit cron-fleet default is deliberate routing, not drift, so
the #44585 fail-closed guard skips it — switching your chat model with
/model or hermes model no longer breaks unpinned cron fleets.
- tools/cronjob_tools.py: drop model param from agent schema + handler;
remove now-dead _resolve_model_override
- cron/scheduler.py: cron.model/model_provider resolution + per-axis
drift-guard skip
- cron/jobs.py: snapshot resolution mirrors the new precedence
- hermes_cli/subcommands/cron.py + hermes_cli/cron.py: --model/--provider
on hermes cron create/edit
- hermes_cli/config.py: cron.model / cron.model_provider defaults
- docs: cron.md model-resolution tip rewritten
When an operator changes the global model/provider config, warn that
unpinned cron jobs with stored snapshots will fail-closed on their next
run. Adds a cron.model_drift_guard config opt-out (default true) for
fleets that should deliberately track changing global defaults.
Addresses #59031. Original PR #59177 by @doncazper.
Follow-up to the salvaged #71756: instead of webhook importing cron's
private _is_cron_silence_response, the loose autonomous-lane matcher now
lives in gateway/response_filters.py as is_autonomous_silence_response,
sharing LIVE_GATEWAY_SILENT_MARKERS with the interactive exact-marker
rule so the marker sets can never drift. Cron and webhook both delegate
to it. Interactive gateway behavior unchanged.
The default max_iterations/agent.max_turns budget was set when long
agentic runs were rare; complex tasks now routinely exceed 90 tool
calls. Raise the default to 500 across every surface that hardcodes
the fallback: AIAgent constructor, DEFAULT_CONFIG, CLI resolution
chain, gateway env bridge, cron scheduler, and TUI gateway. Explicit
user config values are unaffected (deep-merge preserves them; no
_config_version bump needed).
Docs (en + zh-Hans), CLI help text, tips, and pinned tests updated
to match.
Eliminates the separate import/set/clear dance for _SESSION_CWD by
passing cwd= directly to set_session_vars(), which already handles
the ContextVar set internally and clears it via clear_session_vars().
Also includes the exception message in the no_agent error path.
Follow-up to salvaged PR #70548 (#69396).
The cron scheduler was mutating process-global state in two places:
1. no_agent path called os.chdir() which changed the global process cwd,
leaking into concurrent gateway sessions.
2. The agent path set os.environ['TERMINAL_CWD'] which any gateway
session could read during context-file discovery via
resolve_context_cwd/build_context_files_prompt.
Fix:
- no_agent path: pass workdir as subprocess cwd parameter to
_run_job_script() instead of os.chdir(). The Python process cwd
is never mutated.
- Agent path: in addition to the lock-serialized TERMINAL_CWD,
also set the per-context _SESSION_CWD ContextVar from
agent.runtime_cwd. This ContextVar is scoped to the current
thread/context and NEVER leaks into other sessions.
resolve_context_cwd() checks _SESSION_CWD first, so the cron's
own context file discovery uses the correct workdir, while
gateway sessions (which have no override) fall through to their
own TERMINAL_CWD.
Running any state-writing `hermes cron` CLI command as root (the
default for `docker exec`) rewrote jobs.json via mkstemp +
atomic_replace, leaving it root:root mode 600. The gateway's ticker
(uid 1000 via PUID/PGID) was then locked out of every tick with
PermissionError — silently: the liveness heartbeat stayed fresh,
`hermes cron status` opened with 'Gateway is running — cron jobs will
fire automatically', and in the field ~14h of scheduled jobs were
skipped before a human noticed the absence of messages.
Fixes, per the issue's suggested items 1 and 3:
1. Ownership preservation on save (cron/jobs.py): snapshot the owner
before the atomic replace; when the writer is privileged (euid 0)
and the previous owner differs, chown the rewritten file back.
First-time creation inherits the cron dir's owner. Unprivileged
writers never call chown. POSIX-only (guarded via os.name/getattr),
best-effort — a chown failure logs a warning but never breaks the
save. 0600 hardening is unchanged.
2. Zombie-ticker surfacing: the ticker loop (both single-profile and
multiplex paths) now persists the failure reason to a
ticker_last_error marker next to the heartbeat files on every
failed tick, and clears it on the next clean tick. `hermes cron
status` shows the recorded reason in its 'ticks may be failing'
branch, plus an actionable ownership hint when the error is a
PermissionError (recommend `docker exec -u <uid>:<gid>`).
Fixes#68483
cron/scheduler.py deliberately applies utf-8/replace only on Windows via
popen_kwargs (non-Windows keeps locale default per its test contract) —
drop the sweep's unconditional inline kwargs there. Update the gateway
force-kill kwarg snapshot for the new guard.
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.
Fixes#53428 (master tracker for Windows GBK locale crash).
Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
The multiplex cron path only used use_cron_store() to scope storage paths
(jobs.json, heartbeat files), but _get_lock_paths() and the agent execution
path in cron/scheduler.py resolve via _get_hermes_home() → get_hermes_home()
which checks _HERMES_HOME_OVERRIDE, a separate ContextVar. Without
set_hermes_home_override(), the .tick.lock, config.yaml, .env, and secrets
all resolved to the default profile instead of the per-profile home.
This matches the web_server.py pattern (line 11994) which sets both
set_hermes_home_override(home) AND use_cron_store(home), and the
_profile_runtime_scope pattern used for the multiplexed inbound path.
Found via 3-agent parallel review of salvaged PR #69529.
Teknium
kshitij
Paolo Shamoon
Jeffrey Quesnelle
PRATHAMESH75
Alex Fournier
kshitij
kshitij
CriptoGus
Shaun Prince
spfcraze
John Lussier
Jasmine Naderi
victor-kyriazakos
doncazper
Hermes Agent
joaomarcos
Victor Kyriazakos
Stoltemberg