Re-derivation of #33612 by @LeonSGP43 onto the rewritten scheduler (the
original is 10,692 commits behind; its tick() no longer exists in that
shape, so this is a fresh minimal fix crediting the PR's idea).
The gateway's built-in ticker calls tick(verbose=False) every 60s. The
idle early-return was gated on 'verbose and not due_jobs', so idle
GATEWAY ticks fell through to load_config() + worker-pool resolution
every minute. Return early on ANY idle tick; keep the post-tick MCP
orphan sweep (main intentionally reaps orphaned stdio children on idle
ticks).
3 new tests; mutation-checked (restoring the verbose-gated guard fails
the config-skip test). 66 scheduler tests green.
- Guard the thread-id-as-chat_id normalization to Discord only; Slack
and Telegram adapters use parent_channel as chat_id for thread messages,
so the unconditional version broke their handoff keys.
- Apply the same Discord-specific guard to _seed_cron_thread_session in
cron/scheduler.py (sibling site with the same bug, docstring said
'Mirrors _process_handoff').
- Replace the change-detector test with contract tests that verify the
actual invariant: Discord handoff key == organic thread key, Slack
handoff key still uses parent channel (non-regression).
Replace the process-global HERMES_CRON_SESSION env var with a per-session
ContextVar so a cron tick in the gateway process cannot leak into unrelated
live gateway/API/TUI turns. The cron scheduler now sets the ContextVar
inside the job's try/finally scope and resets it on cleanup. Gateway, API
server, ACP adapter, and TUI gateway all pass cron_session='' to explicitly
mark their sessions as non-cron, masking any stale process env.
Co-authored-by: hinablue <hinablue@gmail.com>
Closes#37968
Makes interrupt-protected context compression cancellable by an explicit
user or lifecycle stop, without weakening protection against ordinary
incoming messages, voice interjections, or active-turn redirects.
Separates explicit hard cancellation from ordinary interrupt/redirect
state with a dedicated threading.Event; introduces
AuxiliaryExplicitCancellation as an attempt-local frozen-cause signal;
isolates the synchronous provider callback in a bounded daemon worker
during protected compression; atomically linearizes Codex timeout
cleanup against explicit cancellation; propagates hard cancellation
through child agents and explicit stop surfaces; serializes hard-cancel
admission against compression commit admission with
CompressionCommitFence; aborts before session rotation or late DB commit,
restores in-place transcript mutations and compressor state, and releases
the heartbeat and compression lease.
Based on #74449 by @suparious. Resolved merge conflicts in
agent/context_compressor.py (feasibility check + try/except) and
tui_gateway/methods_session.py.
Review follow-ups on the #76287 salvage:
- scheduler.py no longer calls advance_next_run after the batch switch;
keeping the import invites a future test to patch the wrong seam.
- advance_next_run returns >= 1 instead of == 1 so a corrupted jobs file
with duplicate ids (both records advanced by the batch) still reports
True after advancing and saving.
The scheduler's pre-dispatch loop called advance_next_run per due job —
one full load_jobs() + one full save_jobs() of the jobs file each — so
N due jobs cost N reads + N writes of the whole file (gateway-restart
catch-up or co-scheduled bursts). advance_next_runs() does one load +
at most one save for the whole due set with identical per-job semantics;
advance_next_run() is now a thin wrapper over it.
Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms
(45x; 50 loads + 50 saves -> 1 + 1).
Tests: batch advances recurring and skips one-shots, single load + save
I/O pin (fails pre-fix — no such function), no save when nothing
advances, and per-job wrapper semantics unchanged. Related: #60946 and
#75833 both restructure this loop's call site for correctness — neither
addresses the I/O cost, and this batch primitive composes with either
dispatch design; happy to rebase onto whichever lands first.
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
Fixes#73973.
A finite one-shot whose dispatch was claimed (claim_dispatch increments
repeat.completed BEFORE execution) but whose run died before mark_job_run
was left permanently wedged: completed==times, last_run_at null, state
'scheduled'. The run-claim TTL blocked re-dispatch, and once it expired
the dispatch-limit guard silently removed the job with no output and no
error.
Two complementary fixes:
- cron/scheduler.py run_one_job: the outer handler now catches
BaseException, not just Exception. The inner run_job handler re-raises
CancelledError/KeyboardInterrupt/SystemExit after agent teardown, and
none of those are Exception subclasses, so the outer 'except Exception'
missed them and mark_job_run(False) was never called. Failures are now
recorded first (mark_job_run + finish_execution, each independently
guarded), then non-Exception BaseExceptions are re-raised to preserve
teardown semantics. Plain Exceptions keep the existing behavior
(recorded, return False, no re-raise). Empty str(e) (bare
CancelledError) falls back to the exception class name.
- cron/jobs.py: when either removal site (claim_dispatch or the
get_due_jobs dispatch-limit guard) drops a one-shot whose claimed run
never completed (last_run_at null), _write_wedged_oneshot_diagnostic
now writes an operator-visible .md into cron/output/<job_id>/ instead
of vanishing silently. Best-effort: diagnostics can never break the
removal. No diagnostic when last_run_at is set (normal completion
race).
Per-job cron inference pins are now user-owned: the agent-facing cronjob
tool schema no longer exposes model/provider/base_url, and the registered
handler ignores them even if a model hallucinates the old parameters.
Users set pins via the dashboard, hermes cron create/edit --model/--provider,
or jobs.json directly — and once set, a pin sticks until the user changes it.
Existing agent-era pins are grandfathered untouched.
New cron.model / cron.model_provider config keys give the cron fleet its
own default model, independent of the chat model. Fire-time resolution:
per-job pin > cron.model > HERMES_MODEL > model.default. An axis covered
by the explicit cron-fleet default is deliberate routing, not drift, so
the #44585 fail-closed guard skips it — switching your chat model with
/model or hermes model no longer breaks unpinned cron fleets.
- tools/cronjob_tools.py: drop model param from agent schema + handler;
remove now-dead _resolve_model_override
- cron/scheduler.py: cron.model/model_provider resolution + per-axis
drift-guard skip
- cron/jobs.py: snapshot resolution mirrors the new precedence
- hermes_cli/subcommands/cron.py + hermes_cli/cron.py: --model/--provider
on hermes cron create/edit
- hermes_cli/config.py: cron.model / cron.model_provider defaults
- docs: cron.md model-resolution tip rewritten
When an operator changes the global model/provider config, warn that
unpinned cron jobs with stored snapshots will fail-closed on their next
run. Adds a cron.model_drift_guard config opt-out (default true) for
fleets that should deliberately track changing global defaults.
Addresses #59031. Original PR #59177 by @doncazper.
Follow-up to the salvaged #71756: instead of webhook importing cron's
private _is_cron_silence_response, the loose autonomous-lane matcher now
lives in gateway/response_filters.py as is_autonomous_silence_response,
sharing LIVE_GATEWAY_SILENT_MARKERS with the interactive exact-marker
rule so the marker sets can never drift. Cron and webhook both delegate
to it. Interactive gateway behavior unchanged.
The default max_iterations/agent.max_turns budget was set when long
agentic runs were rare; complex tasks now routinely exceed 90 tool
calls. Raise the default to 500 across every surface that hardcodes
the fallback: AIAgent constructor, DEFAULT_CONFIG, CLI resolution
chain, gateway env bridge, cron scheduler, and TUI gateway. Explicit
user config values are unaffected (deep-merge preserves them; no
_config_version bump needed).
Docs (en + zh-Hans), CLI help text, tips, and pinned tests updated
to match.
Eliminates the separate import/set/clear dance for _SESSION_CWD by
passing cwd= directly to set_session_vars(), which already handles
the ContextVar set internally and clears it via clear_session_vars().
Also includes the exception message in the no_agent error path.
Follow-up to salvaged PR #70548 (#69396).
The cron scheduler was mutating process-global state in two places:
1. no_agent path called os.chdir() which changed the global process cwd,
leaking into concurrent gateway sessions.
2. The agent path set os.environ['TERMINAL_CWD'] which any gateway
session could read during context-file discovery via
resolve_context_cwd/build_context_files_prompt.
Fix:
- no_agent path: pass workdir as subprocess cwd parameter to
_run_job_script() instead of os.chdir(). The Python process cwd
is never mutated.
- Agent path: in addition to the lock-serialized TERMINAL_CWD,
also set the per-context _SESSION_CWD ContextVar from
agent.runtime_cwd. This ContextVar is scoped to the current
thread/context and NEVER leaks into other sessions.
resolve_context_cwd() checks _SESSION_CWD first, so the cron's
own context file discovery uses the correct workdir, while
gateway sessions (which have no override) fall through to their
own TERMINAL_CWD.
cron/scheduler.py deliberately applies utf-8/replace only on Windows via
popen_kwargs (non-Windows keeps locale default per its test contract) —
drop the sweep's unconditional inline kwargs there. Update the gateway
force-kill kwarg snapshot for the new guard.
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.
Fixes#53428 (master tracker for Windows GBK locale crash).
Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
The in_channel flat-delivery clear was gated on `runtime_adapter is not
None`, but the flat continuation session is seeded ONLY on the live-send
path, which also requires a running event loop. When an adapter is
present but the loop is absent/not-running, the live-send/seed block is
skipped and delivery falls through to the standalone path — clearing
thread_id there flattened an unseeded brief with no continuable session
behind it (and bypassed the D6 capability check).
Factor the live-send condition into `live_adapter_ready` and gate both
the thread_id clear and the live-send block on it, so the clear can no
longer drift from the seed. Add an adapter-present/no-running-loop
regression test (verified it fails against the un-scoped clear).
Addresses review r3609147550.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cron_continuable_surface=in_channel is meant to deliver a cron brief FLAT
into a channel (thread_id=None) so a plain channel reply continues the job
via the shared-channel session. The scheduler already seeds that flat
session (_seed_cron_channel_session) but never cleared the ORIGINAL
target/origin thread_id before routing the actual delivery, so a job
scheduled from inside a Slack thread still delivered into that origin
thread — the seeded continuable session then never matched where the brief
actually landed.
Clear thread_id once in_channel_surface is finalized, before it is re-read
for route_thread_id / the standalone fallback. Scope the clear to EXACTLY
the target that gets flat-seeded — the origin-continuable target
(mirror_this_target) on a live, in_channel-capable adapter
(runtime_adapter is not None):
- fan-out / broadcast / explicit-thread targets keep their thread_id — they
are not continuable and are never seeded, so flattening them would
reroute an explicit thread and could collapse two targets into duplicate
flat sends;
- the standalone no-live-adapter path keeps the origin thread — it can
never seed the flat session, and without an adapter the D6 capability
fail-safe can't run, so flattening there would both drop the brief out of
any continuable lane and bypass D6.
This keeps the clear in lockstep with the seed condition
(in_channel_surface and mirror_this_target) and the D6 fail-safe.
mirror_this_target / origin_user_id are computed earlier using the original
thread_id to match the origin conversation.
Regression tests in TestCronContinuableSurfaceInChannel:
- a job whose origin carries a thread_id must not forward that thread_id to
the live adapter (DeliveryRouter folds target.thread_id into send
metadata), and the seeded continuable session stays flat (thread_id=None);
- with no live adapter, the standalone send falls back to the origin thread
rather than flattening.
Two follow-ups to the per-job model pin surface (#67472 / #49948 review):
- cron/scheduler.py: pass target_model=<effective job model> to
resolve_runtime_provider() on the primary path, so providers with
model-specific api_mode routing derive the mode from the model the job
actually runs (per-job pin > env > config default) instead of the stale
persisted default. The auth-fallback path already did this for its
fb_model.
- hermes_cli/web_server.py: POST /api/cron/jobs (and its sync worker) no
longer hardcodes profile="default" when the request carries no profile
param. A pool backend scoped to a named profile now resolves its own
profile via get_active_profile_name(), so pre-profileScoped desktop
clients can't write a named profile's job into ~/.hermes. Unscoped /
custom HERMES_HOME keeps the legacy default fallback.
Tests: target_model capture test on run_job; two profile-default tests on
the create endpoint.
run_agent._dispatch_delegate_task forces background=True for every top-level
delegation, and async_delivery_supported() returns True for any session that
never binds the capability. On runners that cannot receive a completion after
their turn ends, that combination silently discards every subagent result: the
model gets a dispatch handle, ends its turn, and reports 'waiting for results'.
Two such runners never bind the capability:
* hermes -z (one-shot) prints one final response and exits. It bypasses cli.py,
so nothing drains process_registry.completion_queue (only the interactive
process_loop and the gateway watchers do).
* cron run_job clears the HERMES_SESSION_* routing keys, so a completion event
carries session_key="" — _enrich_async_delegation_routing cannot resolve it
and _inject_watch_notification drops it ("no routing metadata"). By then
run_job has already shipped the job's final response via _deliver_result;
there is no turn left to re-enter. Worse, get_current_session_key() can fall
back to the ambient os.environ HERMES_SESSION_KEY, so a cron subagent's output
can be routed into an unrelated user chat rather than merely dropped.
Add declare_stateless_channel() and bind it in both runners, routing
delegate_task to its existing inline/synchronous path — the same fallback the
stateless HTTP adapter already relies on, and the fix suggested in #63142. The
helper binds only the capability: set_session_vars() would also latch
_session_context_engaged, which a pure single-process one-shot must not trigger.
Also correct two agent-facing strings that hardcoded 'stateless HTTP API' as the
only channel without async delivery (delegate_tool, terminal_tool); they now name
the actual condition.
Repro (before): hermes -z 'Use delegate_task to spawn a subagent that replies
BANANA. Report its reply.' -> "Waiting for the subagent's response...", exit 0,
no BANANA. After: BANANA is returned in-turn.
Fixes#53027Fixes#63142
A fallback chain entry can name its API key via key_env (or the
api_key_env alias) per the fallback-providers docs, but only the gateway
path resolved it — TUI/desktop, cron, and CLI setup fallbacks ignored it,
so a fallback provider whose key lives in a non-standard env var never
resolved on those surfaces.
Centralize the inline-api_key-then-key_env lookup in
hermes_cli/fallback_config.resolve_entry_api_key() and use it at all four
fallback resolution sites (tui_gateway, cron scheduler, gateway runner,
CLI setup mixin); the CLI mixin also gains the base_url passthrough the
other surfaces already had.
Salvaged from PR #43861 (surgical reapply — the original branch predates
the #65264 fallback restructuring).
Only fallback resolution needs an explicit target model. Keep the primary resolver call compatible with existing callers and test doubles while retaining atomic provider/model fallback selection.
Switch provider and model together after setup-time auth failure. Serialize global auth-store merges under target-specific locks and preserve auth-to-shared lock ordering for profile OAuth refreshes.
Collapse the six per-surface copies of override-then-global resolution
(CLI startup, gateway, TUI, cron, /model switch, fallback activation)
onto one shared resolve_reasoning_config() in hermes_constants.
Also fixes the gateway resolving reasoning against config model.default
instead of the session's effective model: after a session-only /model
switch, the switched model's override now applies (gateway message paths
pass the resolved session model through _resolve_session_reasoning_config;
/reasoning status reads the session model override).
Cleanup: drop docs/PER_MODEL_REASONING.md (duplicates the website docs
page), drop the change-detector _config_version test (no bump needed —
deep-merge handles new keys), remove a stale plan-reference comment.
Adds chokepoint contract tests (13) and gateway session-effective-model
regression tests (2).
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.
Example:
agent:
reasoning_effort: "medium" # global default
reasoning_overrides:
"openrouter/anthropic/claude-opus-4.5": "xhigh"
"openai/gpt-5": "low"
"claude-sonnet-4.6": "high" # bare model name also works
The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.
Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)
Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
+ _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
+ Re-resolves reasoning_config for the fallback model (best-effort)
Closes#21256 (per-model reasoning_effort defaults).
Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
Salvage of #63935. The original fix read HERMES_CRON_SESSION_DB_TIMEOUT
from a bare env var, but AGENTS.md requires non-secret behavioral
settings to live in config.yaml with an env var bridge only for
backward compatibility.
Changes:
- Add cron.session_db_timeout_seconds to DEFAULT_CONFIG (default 10s)
- Resolution order: HERMES_CRON_SESSION_DB_TIMEOUT env override →
cron.session_db_timeout_seconds in config.yaml → 10s default
(mirrors the existing script_timeout_seconds pattern)
- 0 = unlimited (opt-in for debugging, skips the bound)
- Strengthen test: assert the warning is logged on invalid env value
(caplog was taken but never asserted)
- Add test: verify config.yaml resolution path works end-to-end
Co-authored-by: LoicHmh <26006141+LoicHmh@users.noreply.github.com>
run_job() constructs SessionDB() synchronously with no timeout of its
own, unlike the agent's run_conversation call further down, which is
already bounded by HERMES_CRON_TIMEOUT. A wedged sqlite3.connect (e.g.
a stale flock from a crashed sibling process) hangs this call
indefinitely.
That hang is invisible to every existing cron safeguard because it
happens before _submit_with_guard's future exists: the finally block
that discards the job ID from _running_job_ids never runs. The job
stays wedged "running" — every later tick logs "already running —
skipping" — until the whole gateway process is restarted.
Observed in production: a cron job's worker thread was confirmed via
a live py-spy thread dump to be parked inside SessionDB.__init__'s
sqlite3.connect for 3+ days, silently skipping every scheduled fire
in between across a gateway process that otherwise stayed healthy.
Bound the SessionDB() construction with its own timeout
(HERMES_CRON_SESSION_DB_TIMEOUT, default 10s), following the same
bounded-thread-pool pattern already used elsewhere in this file (the
delivery retry path, and the agent inactivity watchdog just below).
On timeout, log at ERROR and proceed with session_db=None instead of
degrading silently to debug level, since an actual hang here is a new
condition worth surfacing.
Adds tests/cron/test_sessiondb_init_hang.py, including an end-to-end
regression proving the dispatch guard is released and a subsequent
tick can fire the same job again after a simulated hang.
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
get_due_jobs()'s one-shot stale-entry recovery (#38758) treated an
expired run_claim (#59229) as proof the claiming tick died, but a run
stalled on network I/O — or a laptop asleep mid-run — legitimately
outlives the TTL while very much alive. The recovery then deleted the
job record mid-flight: list showed the job gone, and when the run
finished mark_job_run() found nothing to update, so last_run_at /
last_status / last_delivery_error were never recorded.
Two guards, per the liveness signals available:
- Same process (the common single-gateway case): before removing a
dispatch-limit-reached one-shot, consult the scheduler's running set
via a lazy import; if the job is still running here it is slow, not
stale — keep the entry.
- Cross process: run_job's monitor loop now refreshes run_claim.at
every 60s while the run is alive (including under
HERMES_CRON_TIMEOUT=0, which previously blocked without polling), so
an expired claim really does mean the owner died and the TTL stays a
dead-owner detector.
Fixes#62002
kshitij
kshitij
Shaun Prince
spfcraze
teknium1
victor-kyriazakos
doncazper
Hermes Agent
Victor Kyriazakos
Stoltemberg
wz-heng
Jake Long Vu
Wesley Simplicio
Vinay Bikkina
Colin Greig
Clifford Garwood
Gille
Trevor Gordon
Rage Lopez
Dan Schnurbusch
ScotterMonk
morluto
Minhao HU
PRATHAMESH75
embwl0x