the sparklines were stretched across the whole gantt bar, but the profiler
wraps a SINGLE step (.github/actions/profile runs it between start/stop), so
on a job dominated by checkout + uv sync + post-job cleanup the samples only
describe a slice in the middle. reproduced: a 30s profile inside a 100s job
whose profiled step ran t=60..90 drew at left 0% width 100% instead of
left 60% width 30% — putting a cpu spike visually under a step that never ran.
the profile json had no wall-clock anchor to place it with, only duration_s,
so emit started_at/completed_at as iso-8601 utc in the same shape as github's
job timestamps. monotonic() still drives the sampling loop (immune to clock
steps); the timestamps are purely for placement.
_profile_window_pct() converts that window into bar-relative percentages and
both overlay states now use it — the expanded holder directly, the collapsed
strip via a .res-clip wrapper so its 100%-width is relative to the window
rather than the bar. the two states are asserted to agree on the x-axis.
falls back to the full bar, i.e. exactly today's behaviour, when the profile
predates these fields, when the timestamps don't parse, or when the window
doesn't overlap the job at all (clock skew between the runner writing the
profile and github's timestamps). a profiler that outran the job's
completed_at is clamped to the bar, and a sub-percent window keeps a 0.5%
hairline so it can't collapse to invisible.
7 new tests. verified they discriminate: forcing the old always-stretch
behaviour fails 4 of them, while the 3 fallback tests keep passing since
full-bar is what they want. tests/ci 145/145.
ops/s alone can't tell you whether a device is saturated — it reads low on a
few large IOs that pin the disk at 100% busy, and high on many small cached
ones. sample io_ticks (diskstats field 13) instead: its delta over the
interval is device busy time, i.e. iostat's %util. take the busiest single
device rather than the sum, since summing across devices exceeds 100% on a
multi-disk node and means nothing as a saturation percentage.
divide by the real elapsed gap rather than the nominal interval — a loaded or
throttled runner drifts well past 1.0s and would otherwise report >100%.
emit a downsampled series (cpu/mem/disk, 0-100 ints) alongside the existing
summary, mean-bucketed to 180 points so a 40min job costs the same few KB as
a 40s one. the report inlines every profile into one self-contained html
file, so an unbounded 1Hz series would dominate its size. mean, not every Nth
sample: a spike that survives decimation by luck is misleading.
the report renders the series as svg sparklines over each gantt bar, in two
states off the same markup (3px strip when collapsed, full height when
expanded) via preserveAspectRatio=none. profiles predating the series field
degrade to table-only, no overlay.
tests/ci 62/62.
TestPrefetchServerRetainVisibility polled the mocked op-status endpoint at
the production _RETAIN_OP_POLL_INTERVAL_S of 0.5s, so each test burned ~1s
of real sleep plus a cross-thread hop to the shared hindsight event loop per
poll — all inside a 5.0s join on the background prefetch thread. On a busy
runner that thread is exactly what gets starved, so it outlived the join and
the assertion saw an empty order list: assert [] == ['recall'].
collapse the interval to 0.01s in these tests (against an AsyncMock there is
no server to be polite to) and route the joins through a helper that asserts
the thread actually finished, so a genuine wedge reports itself instead of
surfacing as a confusing empty-list mismatch.
also drop the two wall-clock assertions that measured runner load rather than
the contract under test: the drain-budget bound goes from 3.0s to budget+10s,
and the eviction test now asserts the status endpoint saw no further calls
(the actual "dropped ops aren't re-polled" contract) instead of timing the
second prefetch at <0.25s.
reproduced by pinning the tests to one core against N spinners on that same
core: at 16x oversubscription pre-fix failed 3/3 and post-fix passed 3/3.
class wall time drops 1.45s -> 0.44s for the recall test.
the idle deadline starts at popen, so the first window has to cover the
helper script's interpreter startup — wall clock the test doesn't control.
39975613b shrank the stt windows to 0.1s, which is under a cold `python -u`
on a loaded runner, so the child got killed before printing anything and
the assertion saw 'Terminated\n' instead of its progress line.
hoist the timings to named constants (idle 2.0s, 12 ticks x 0.25s) with a
setup assertion that total runtime still exceeds the idle window, so a
future speed pass can't silently tune the test into not exercising the
deadline reset. fix the same latent race in the sibling tts test it was
ported from (0.2s window vs a real spawn) before it fires too.
swept tests/ for the pattern; the only other hit (win_pty_bridge) is a
per-read poll inside a 5s outer loop, not a total budget.
Run a duplicate of CI on the new ARC (Actions Runner Controller) runners in
GKE, beside the existing CI. The duplicate does not change production CI.
Every workflow in .github/workflows/ that does not start with newci- is
byte-identical to main. Watch the shadow runs for a few days, then migrate.
The shadow set is 16 files: newci-ci.yml plus the 15 reusable workflows that
ci.yml calls on a pull request. Only pull-request workflows are copied.
js-autofix, deploy-site, and skills-index run on push or on a schedule. A
copy of those would push branches and deploy the site a second time.
Safety properties of the shadow:
- Concurrency groups are newci-prefixed. This is the important one. The
production groups use cancel-in-progress, so a shared group would let a
shadow run cancel the production run.
- Cache keys are newci-prefixed. The shadow cannot poison or evict a
production cache entry.
- Reusable-workflow calls point only at other newci-* files. No shadow job
calls a production workflow.
- The PR review comment runs with --dry-run. It prints the comment body to
the job log. Two pollers cannot fight over the hermes-ci-review-bot
comment.
- The gate job is renamed to "[newci] All checks pass (informational)". The
production check "All required checks pass" stays the only merge gate.
- The shadow runs on pull_request only. The push trigger is removed.
- docker publish and merge jobs are unreachable. Their conditions require a
push to main or a release.
Runner infrastructure, in the shadow copies only:
- Jobs go to three scale sets: arc-runner-small for short gate jobs,
arc-runner-set for general work, arc-runner-docker and arc-runner-arm64
for image builds. dind is only on the docker sets, so the other jobs stop
paying for a privileged sidecar.
- The runner image supplies node 26, npm 12, uv, Python, and ripgrep. The
setup-node, setup-uv, and per-job install steps are gone.
- Checkout uses a node-local git mirror, seeded from the runner pod env.
- buildx layer cache moved to Artifact Registry in us-central1, the same
region as the runners. Reads are keyless through GKE Workload Identity.
Writes use GitHub OIDC and happen only on main pushes and releases, so
pull-request code cannot write a layer that the publish job reads.
Merge-base work, in the shadow copies only:
- A new composite action, .github/actions/merge-base, deepens a shallow
clone until the two histories connect. fetch-depth: 0 fetches all ~1400
refs and measured 76-81s, against 3-6s for a shallow checkout.
- The action fails by default when no merge base exists. A three-dot diff
over a missing merge base scans nothing and reports clean, so the
supply-chain audit must stop. history-check sets fail-on-missing to false,
because absence is the result it measures.
- lint diffs against the base commit directly. The job checks out the PR
merge ref, so base.sha is already the correct comparison point.
- contributor-check uses origin/main..HEAD. The result equals the merge-base
form, and the extra git call also expanded a SHA without quotes.
Other changes:
- .github/actionlint.yaml declares the four ARC labels. actionlint knows
only GitHub-hosted labels, so every runs-on in the repo was reported as an
unknown label: 40 warnings that hid real findings.
- scripts/ci/resource_profile.py records CPU and memory for a job step. The
timing report shows the data per step.
- run_tests_parallel.py can list test files from the git index. The slice
generator then needs no blobs.
- Docker test files are split so boot-heavy tests run in parallel.
- Container-environment parity fixes in doctor, gateway, and skill_utils,
with tests.
To retire the shadow: delete .github/workflows/newci-*.yml.
Timing out a bounded git probe must not leave helper descendants
(credential helpers, git-remote-https, hook children) running after the
probe fails open. bounded_git_probe now spawns the child in its own
process group on POSIX (process_group=0), and _kill_git_process_tree
signals the whole group with os.killpg — gated on the child actually
leading its own group (pgid == pid), so a shared-group spawn can never
take down unrelated processes. Windows keeps the existing taskkill /T /F
tree kill.
Proven live on main: a fake git that forks a 300s descendant left the
descendant running after the probe timeout; with the fix the descendant
dies with the launcher. Fast path and fail-open contract unchanged.
Port of openai/codex#36793 (Terminate timed-out Git process trees).
Nothing covered the update path, which is the worst thing to break: a broken
updater strands users on the version that cannot fix itself. `hermes update`
alone is ~2000 lines (hermes_cli/update_cmd.py) and had no end-to-end test.
tests/install/install-update-e2e.sh installs a genuine earlier Hermes through
the real one-liner (curl -fsSL https://…/install.sh | bash, served by
dev-sandbox's MITM proxy at the canonical URL, cloning "github.com" through the
upload-pack shim), which really installs uv, a managed Python, Node and the
venv. It then applies ONE update route and requires the checkout to land on this
commit with `hermes --version` still working -- so a pass means the venv and
entry point survived, not merely that git moved.
One route per run, each on a sandbox built from scratch. Sharing one install
across routes -- or rewinding with `git reset --hard` between them -- leaves the
second route running against a tree the first already updated (same venv, same
console script, same __pycache__), which is not the state any real user is in: a
route could pass only because its predecessor did the work, and a failure in the
first left the second exercising something undefined.
--install-ref chooses what to install first, so this covers "update from an
older release", not just from the tip. Installer flags are probed against the
target rather than assumed, because releases from months back predate flags
current Hermes takes for granted: --skip-browser is read out of that ref's own
install.sh, and `--yes` is asked of the installed `hermes update --help` (the
update subcommand has lived in main.py, subcommands/update.py and update_cmd.py
across the tags we sample, so a static parse rots silently -- and did). Without
those probes, old releases die on "Unknown option: --skip-browser" and
"unrecognized arguments: --yes" before doing any work.
Installer output is streamed through tee rather than captured: a real install of
uv, Python, Node and the venv IS the substance of this test, so it belongs in
the job log, not only in an artifact. pipefail keeps the installer's exit status
rather than tee's, so a failed install cannot look like a pass. The sandbox's
own proxy log is printed in full on failure, since a rejected TLS handshake
explains a failure that otherwise reads as a bare `curl: (35)`.
Deliberately reuses dev-sandbox rather than adding a second harness. An earlier
draft rewrote install.sh's hardcoded URLs with insteadOf and ran it against the
host; that tested the installer LESS faithfully (bash install.sh instead of the
real one-liner, host libs instead of a clean machine, ssh disabled to keep a
failed rewrite from reaching real GitHub) while duplicating a fake Internet we
already have.
Shell, not pytest, so scripts/run_tests.sh and run_tests_parallel.py stay
untouched: a pytest version needed an entry in the former's `env -i` credential
allowlist and a _SKIP_PARTS exclusion in the latter, and every meaningful line
was a command run inside the sandbox anyway.
Two guards, both earned during bring-up. It prefers the `sandbox` wrapper and
falls back to the raw script only when bwrap is on PATH (under Nix the wrapper
supplies the PATH and DEV_SANDBOX_* vars, so the bare script exits 127). And it
refuses to run on a dirty worktree: every dev-sandbox invocation re-derives fake
main from the working copy, so uncommitted changes move the update target
between the call that installs and the call that verifies -- a failure that
looks like a broken updater but is a moving reference.
Selecting an NVIDIA NIM model whose id reached config without the nvidia/
prefix produced a bare "HTTP 404: 404 page not found" — retried three times,
never naming the model. It reads exactly like an outage or an auth failure,
which is where the Discord thread spent its time before the id was spotted.
normalize_model_for_provider() had no branch for nvidia, so a bare id passed
straight through to the API. Repair it from the provider's curated catalogue:
a bare name that matches exactly one entry modulo the prefix gets it back.
That's a lookup, not a guess — build.nvidia.com also fronts local NIM
containers and third-party models, and anything absent from the catalogue is
left alone. Because the repair runs on every runtime setup, an already-broken
config self-heals on the next turn and prints what it changed.
If a bare id still reaches the wire, the 404 now explains itself. The
classifier consults the same catalogue: a prefix-less id the provider only
serves as vendor/model is a deterministic failure, so it classifies as
model_not_found instead of burning three retries on a retryable "unknown",
and the error trace names the id to use.
Fixes#78796
Adds a last_read_at watermark to the sessions table so surfaces (CLI,
TUI, desktop) can badge unread conversations. Read state derives from
the watermark vs latest activity, so new messages flip a conversation
back to unread with zero writes on the message path. NULL means never
tracked, so shipping the column doesn't badge pre-existing history.
set_session_read() stamps the whole compression lineage, matching the
archive/pin semantics; list_sessions_rich() rows carry a derived
`unread` key. DB layer only — no surface exposes it yet.
* fix(discord): reject empty outbound messages
* test(discord): cover empty final reply backfill state
Missed-message backfill decides what to replay from discord_messages, so
a dropped final reply must be recorded as failed by the new guard the
same way the exception path records one — otherwise the reply is both
never sent and never retried.
Co-authored-by: Jony <619963502@qq.com>
* chore: map 619963502@qq.com to zyz619963502zyz for PR #73449 salvage
---------
Co-authored-by: Jony <619963502@qq.com>
Three fakes pin mark_exhausted_and_rotate's signature explicitly and broke on
the new argument. They now assert it rather than just tolerate it — the xAI
spending-limit case is exactly the billing-403 this fixes, so it should be
pinning `failure_reason == "billing"`.
The sole-credential cooldown sized the bench from the raw HTTP status, but
403 is overloaded: error_classifier maps OpenRouter's "key limit exceeded"
and xAI's spending-limit block to FailoverReason.billing, while an edge
throttle with the same status is transient. Only 402 was excluded from the
short cooldown, so a spent account on a single key retried every 60 seconds
and re-failed forever.
Thread the classified reason from recover_with_credential_pool through
mark_exhausted_and_rotate to _exhausted_ttl. Billing keeps the full bench
regardless of status; everything else transient still recovers in 60s. The
verdict is stored on the entry (_EXTRA_KEYS, so it persists to auth.json) —
without that a restart would re-read a bare 403 and downgrade the bench.
Tests: sole billing-403 stays benched, survives reload, unclassified 403
still recovers; call-site coverage that the reason actually reaches the pool.
Three existing kwargs assertions updated for the new argument.
next_available_at() was computing the full 1-hour TTL for a sole
credential on a 429, contradicting the 60s cooldown in _available_entries.
The fallback restore gate (agent_runtime_helpers) uses next_available_at
to decide when to switch back from fallback to primary — so the agent
stayed on fallback for an hour instead of ~60s.
Add sole_credential computation in next_available_at mirroring
_available_entries, and a test verifying the short cooldown propagates.
A pool with only one usable (non-DEAD) credential has nothing to rotate to.
On a transient throttle (429 rate-limit, 403 edge-throttle, 5xx) the offending
key was benched for a full hour (EXHAUSTED_TTL_429/DEFAULT), so single-key /
no-fallback setups got an hour of hard failures for a throttle that resets in
seconds. The pool already special-cases 401 to recover quickly for single-key
setups; extend that to transient throttles when the credential is the sole
non-DEAD entry. 402 (billing/quota) keeps the full bench — a quick retry can't
help. Provider-supplied reset_at still overrides.
Adds tests covering sole 429/403 recovery, 402 full-bench, and multi-key
(no early recovery).
hermes debug share runs on the backend. A desktop app connected to a
remote, docker, or SSH backend writes desktop.log on the client machine,
so the bundle can never contain it — and the report rendered that as a
bare "(file not found)", which reads as "the app logged nothing" and
sends triage after a client-side bug it cannot see.
Name the writer and the path to collect by hand. Backend-written logs
are unchanged, a present desktop.log is still captured, and an empty one
still reports "(file empty)" — the app ran and logged nothing is a
different fact from the file being on another host.
* fix(agent): adopt .env credential/base-url edits at the turn boundary
A Settings save (desktop PUT /api/env, hermes setup) updates .env and
the saving process's os.environ, but a live session worker keeps the
base_url/api_key captured at agent init until restart — an open chat
silently kept calling the old endpoint (e.g. a local-server key sent to
api.openai.com, failing with an opaque 401).
Add AIAgent._try_refresh_env_client_credentials(), called at the start
of each conversation turn: re-resolve the provider's env-sourced
credentials (load_env() is mtime-memoized, so an unchanged file costs
one stat()) and rebuild the client via the existing
_replace_primary_openai_client machinery when the user edited them.
The refresh reacts only to env edits — resolved values changed since
the last look — never to mere divergence from the agent's current
values: credential-pool rotation and failover legitimately move the
session off the env credential, and stomping those back would flap.
Config model.base_url / pool custom endpoints keep precedence: edits
are only adopted while the session still runs on the registry default
or the previously-seen env value.
Lift _get_env_prefer_dotenv out of _seed_from_env to module level
(get_env_prefer_dotenv) so both the pool seeder and the per-turn
refresh share the same .env-over-os.environ resolution, including the
op:// indirection handling.
Fixes#67821
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(agent): address sweeper review on env credential refresh
- Cover named custom providers (#67935): provider="custom" has no
PROVIDER_REGISTRY entry, so resolve the config block's key_env through
the same lookup the runtime resolver uses.
- Make the edit baseline transactional: a failed client rebuild rolls the
agent back and leaves _env_creds_seen un-advanced so the unchanged edit
is retried next turn.
- Recompute route-derived TLS material and default headers on a base-url
change, via a _reapply_route_client_config helper shared with
credential-pool rotation so the two paths cannot drift.
- Rebase onto main: get_env_prefer_dotenv keeps the scoped _get_secret
semantics from the profile-isolation fix (no raw os.environ reads).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: map jskang@lablup.com to rapsealk
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
* fix(credential-pool): clear exhaustion state on key rotation
When a user rotates an API key (e.g. via `hermes setup` after hitting a
rate limit), _upsert_entry updates the access_token on the existing pool
entry but preserves the stale last_status=exhausted from the old key.
On the next session the pool finds the entry, sees it exhausted, and
returns no usable credentials — even though the new key is valid.
Fix: when access_token changes on an existing entry, reset last_status,
last_error_code, last_error_reason, last_error_message, and
last_error_reset_at. The exhaustion state belongs to the old key, not
the new one.
* chore: add pasevin@gmail.com to AUTHOR_MAP
* fix: clear last_status_at on key rotation, remove unused pytest import
Address review feedback from teknium1 on PR #22622:
- Add last_status_at=None to the reset block (matches all other
token-sync reset paths in credential_pool.py)
- Assert last_status_at is None in the regression test
- Remove unused pytest import flagged by ruff + ty
Drop the manual web.search_backend / web.backend config-reading block
that duplicated _read_config_key in web_search_registry.py. The function
now delegates directly to get_active_search_provider() (which reads the
same config keys via the registry's canonical resolver) and falls back
to _get_search_backend() only when the registry has no providers loaded.
Also updates the TestXaiWebSearchBackendPreference tests to monkeypatch
the registry instead of load_config_readonly, and adds two new tests for
the legacy fallback path (no provider registered -> _get_search_backend).
Lock in backend preference, wire-name aliasing, and normalize mapping
so configured non-xai search providers stay on the Hermes client path.
Also init conflict-recovery generation on the telegram bare-adapter
helper so CI polling progress tests do not AttributeError.
#75017: Telegram polling conflict retry used drop_pending_updates=False,
starting a new getUpdates session that immediately got 409'd by the
previous still-expiring session — creating the very conflict it was
trying to recover from. Switch to drop_pending_updates=True so Telegram
terminates stale sessions. Also add a recovery-generation guard so the
first transient getUpdates success after a retry doesn't reset the
conflict counter back to 0 (defense-in-depth from PR #75096).
#75153: The WAL-reset warning always said 'hermes update can repair'
even for git/pip/system Python installs where it can't. Now uses
detect_install_method() + recommended_update_command_for_method() to
give a context-appropriate hint (hermes update for git, docker pull for
docker, nix message for nix, generic install hint as fallback).
- write_file: encode content once, share bytes between bytes_written and
the sha256 verification (drops a second full-content encode per write)
- patch_parser: replace the except-TypeError retry around
write_file(pre_content=...) with signature-based feature detection so a
TypeError raised inside a capable implementation propagates instead of
triggering a duplicate write; tests for both duck-typing contracts
- tests: real-ops V4A BOM round-trip + _file_has_bom disk-probe guard
(the teknium1-review regression previously only covered by a fake)
- comment: document dirs_created's long-standing "parent ensured" meaning
Bug 1 (UTF-8 BOM loss on V4A UPDATE):
_file_has_bom() trusted pre_content for BOM detection, but the most
common pre_content provider — read_file_raw() — deliberately strips
BOMs so the agent never sees U+FEFF glyphs. Passing BOM-stripped
content through pre_content caused a false-negative: the method
returned False and write_file() silently removed the marker on rewrite.
Fix: _file_has_bom() now always probes the first 3 bytes on disk
(head -c 3), ignoring pre_content for BOM purposes. pre_content is
still used by two other consumers — line-ending detection and lint/LSP
delta computation — neither of which is affected by BOM stripping.
Bug 2 (backward compatibility):
_apply_update() called write_file(path, content, pre_content=...) as a
keyword argument. Duck-typed file_ops implementations that only
implement the two-argument write_file(path, content) contract would
raise TypeError.
Fix: wrap the call in try/except TypeError, falling back to the
two-argument form when the keyword is not accepted.
Also declare tomli in pyproject.toml (pre-existing conditional import
for pre-3.11 Python, caught by the pre-commit dep scan after staging
file_operations.py).
Tests:
Add TestV4ABomRoundTrip with two cases:
- UPDATE on BOM-bearing file preserves the marker
- UPDATE on plain file does not inject a BOM
Addresses teknium1 review on PR #55661.
write_file currently spawns up to 6 subprocesses per call:
1. mkdir -p (separate call before atomic write)
2. cat (to read pre-content for lint/BOM/line-ending detection)
3. _atomic_write (mktemp + write + mv — the essential one)
4. wc -c (to measure bytes written)
5. _check_lint_delta (post-write lint — also essential)
6. LSP snapshot (also essential)
This PR removes three of them without changing any observable behavior:
1. Fold mkdir -p into _atomic_write shell script (−1 subprocess/write)
The atomic write script already runs a single shell; adding mkdir -p
to it costs zero extra processes.
2. Add optional pre_content parameter to write_file (−1 subprocess/patch)
patch_replace and V4A _apply_update already read the file for fuzzy
matching. Passing that content as pre_content skips the redundant cat
inside write_file. Fully backward-compatible: callers that don't pass
pre_content still read from disk as before.
3. Replace wc -c with len(content.encode('utf-8')) (−1 subprocess/write)
We already have the content in memory; encoding it to get the byte count
is equivalent to wc -c for UTF-8 text.
4. Remove redundant _check_lint loop in apply_v4a_operations (−N subprocesses/V4A)
write_file already runs _check_lint_delta internally. The old code ran a
bare _check_lint(f) loop over all modified files — a re-read + re-lint
without post_content context. Now lint results propagate from write_file
via a four-tuple return, zeroing out the extra subprocesses.
Net effect:
- write_file: 6 → 3 subprocesses per call (new files)
- patch_replace: 6 → 5 subprocesses per call (pre_content skips cat)
- V4A multi-file patches: saves 1 subprocess per modified file
- A typical 4-file V4A patch drops from ~28 to ~16 subprocess calls
#71775 moved deferred single-use-token refreshes outside the pool lock
(correct — they hold a cross-process flock plus network I/O). But
_refresh_entry_impl's three terminal-auth-failure quarantine paths do a
bare read-modify-write of self._entries. Those used to run with the
caller holding self._lock; on the deferred path they run unlocked, so a
concurrent mutation between the read and the write is silently lost.
Wrap all three in 'with self._lock' (an RLock, so locked callers
re-enter safely) and correct the _refresh_pending_entries docstring,
which claimed the mutations were already self-locking.
Post-merge gate-sweep finding on the #71775 salvage (#77714).
Sibling to the acquire_lease re-select fix.
select() re-selects once deferred single-use-token refreshes complete;
acquire_lease() performed the refresh but returned its pre-refresh
answer. Since _acquire_lease_under_lock returns early exactly when a
refresh is pending (if not available: return None, pending_refresh),
a pool whose entries all needed a refresh always returned None — the
caller failed an answerable request right after the refresh succeeded.
Retry once, only when the first pass was empty and a refresh ran.
Post-merge gate-sweep finding on the #71775 salvage (#77714).
CI caught a missed caller-shape update. Both PRODUCTION callers of
_write_sse_chat_completion / _write_sse_responses were converted to
ThreadSafeAsyncQueue, but two pre-existing tests in
tests/gateway/test_api_server.py construct the writer's queue
themselves and still passed a stdlib queue.Queue.
The consumer now does 'await asyncio.wait_for(stream_q.get(), ...)',
which on a queue.Queue blocks the thread forever:
test_stream_cancelled_persists_incomplete_snapshot hung until
pytest-timeout killed it (CI reported the whole file as 'no tests
ran (timeout before collection)'). The sibling disconnect test only
survived because it pre-fills before the first await.
tests/gateway/test_api_server.py: 99 passed (was 1 failed + a 60s
hang); with the SSE/api_server suites: 147 passed.
Gate finding (/simplify-code pass): both cross-thread tests passed
loop=loop explicitly, but no production caller does — all six
(_on_delta, _on_tool_*) rely on the queue resolving its own
_loop_ref in __init__. The kwarg made the tests vacuous: a broken
_loop_ref still passed them.
Dropping the kwarg exercises the real path. Verified by mutation:
with self._loop_ref = asyncio.new_event_loop() (wrong loop), both
tests now FAIL; they passed before this change.
The conflict-marker strip fused test_agent_task_raises with the body of
test_failed_result_dict — restore both as separate tests (content from
the PR head, verified verbatim).
Addresses teknium1 sweeper review (2026-07-30) requiring coverage of:
1. ThreadSafeAsyncQueue.put_threadsafe() off-loop boundary: a real
daemon thread pushes into the queue from outside the owning event loop
while the consumer awaits get(), mirroring the run_conversation
worker-thread producer path. Includes a 20-concurrent-thread
no-drop regression.
2. Long-reasoning bound stability for thinkingPreview: 100k-char input
plus empty/collapsed cases must not crash and must retain the visible
tail marker inside the bounded 24k clean window.
The session event stream (api_server.py:~2236) was the one genuinely
unicode-distinct SSE writer — json.dumps(payload, ensure_ascii=False) +
.encode('utf-8'). Every other writer uses plain json.dumps. Route it
through _sse_frame(..., ensure_ascii=False) so _sse_frame is now the single
source of truth for ALL SSE frame serialization in the module (chat-
completion, responses._write_event, /v1/runs, and the session stream).
Byte-identical for non-ASCII payloads: verified against the historical
inline encoder (raw bytes preserved). The ensure_ascii=False path is now
exercised by test_sse_frame_ensure_ascii_false_reproduces_session_event_stream.
_extend _sse_frame with an explicit ensure_ascii param (default True,
byte-identical to a bare json.dumps) and route the two sibling writers
through it: _write_sse_responses._write_event and the /v1/runs event
stream. This completes the dedup PR #65009 — previously only the five
_write_sse_chat_completion sites used the helper, leaving the other two
writers on inline json.dumps with no shared shape.
No behavior change: every writer's emitted bytes are unchanged (verified
byte-for-byte, including non-ASCII payloads where the default
ensure_ascii=True matches the original inline encoders). The ensure_ascii
option is exposed so a future writer can opt into raw non-ASCII bytes
without fractalizing the format again.
Adds tests/gateway/test_sse_frame.py asserting the byte-contract
invariant between _sse_frame and the historical inline encoders.
_write_sse_chat_completion and _write_sse_responses bridged their
stream_delta_callback queue into the event loop via
`await loop.run_in_executor(None, lambda: stream_q.get(timeout=0.5))`
in a while-True poll — a thread-pool round trip on every 0.5s tick even
when idle, plus up to 500ms of tail latency between a delta landing in
the queue and it reaching the SSE response.
Add ThreadSafeAsyncQueue (asyncio.Queue + a put_threadsafe() that wraps
call_soon_threadsafe), used by both streaming producer closures
(_on_delta, tool start/complete callbacks — all invoked from the worker
thread running run_conversation via loop.run_in_executor). Consumers
now do a plain `await asyncio.wait_for(stream_q.get(), timeout=0.5)` —
woken immediately when a delta arrives, no executor hop, no poll
interval.
Updated tests/gateway/test_sse_agent_cancel.py's 7 call sites to
construct ThreadSafeAsyncQueue inside the running loop (required, since
it captures asyncio.get_running_loop() at construction) instead of a
bare queue.Queue() at test-method scope.
Teknium's review on #63457: existing tests pin the final boolean but not
that the slow PROVIDER_REGISTRY sweep is skipped. Add three tests that
booby-trap hermes_cli.auth.get_auth_status and verify
_has_any_provider_configured() short-circuits on:
- config.yaml model.provider
- config.yaml base_url/api_key (custom endpoint shape)
- auth.json active_provider (sweep-only call-pattern guard)
Mutation-checked: reverting the reorder makes all three fail.
teknium's review gap on #72021: the helper's worker/once-guard was
covered, but nothing asserted the stdio TUI entry point actually
invokes prewarm_picker_cache_async() — or that it does so in the right
place. Add a focused entrypoint test that runs the real entry.main()
with stubbed collaborators (same monkeypatch-module-attrs harness as
test_tui_entry_mcp_owner.py), spies on the helper in
hermes_cli.model_switch (the lazy-import source), and asserts:
- prewarm fires exactly once, strictly AFTER the gateway.ready write
- startup stays non-blocking: main() reaches the stdin loop and
returns on EOF
- a prewarm failure is swallowed (fire-and-forget) without breaking
startup
Mutation-checked: deleting the prewarm hunk from entry.py fails both
tests.
The output-cap retry loop reduced max_tokens by 64 tokens per attempt but
never called _compress_context(), so the compressor never fired. Input
growth (~65 tokens/attempt) canceled the savings, leaving the session
stuck at 200,001 tokens — 1 over the 200,000 ceiling.
The fix adds compression to the output-cap retry path. The compressor
drops the middle window, freeing ~50% of tokens. If compression makes
>=5% savings, the session continues; otherwise vision payloads are
stripped or the session ends with compression_exhausted=True.
Also adds CHANGELOG.md entry and bug fix report.
purge_stale_tool_call_markers ran a permanent, irreversible UPDATE with
no backup — inconsistent with repair_state_db_schema's backup-by-default
convention for destructive state.db operations elsewhere in this file.
Take a full snapshot via VACUUM INTO (safe against a live connection,
unlike the raw-copy _backup_db_file used for malformed-schema repair)
before the write, timestamped beside state.db. Skipped when dry_run or
when there's nothing to change. Add --no-backup to `hermes sessions
clean-markers`, mirroring `sessions repair`.
Verified end-to-end: the CLI run against a real temp state.db produces
the backup file before printing the cleared-row count.
The load-on-read repair (_strip_stale_tool_call_markers) fixes affected
sessions in memory on every resume, but never touches the DB — long-lived
sessions re-scan and re-repair the same rows on every load, and the
contaminated bytes stay in state.db (and any backup/cache snapshot of it)
indefinitely.
Add SessionDB.purge_stale_tool_call_markers(dry_run=False): a one-time,
idempotent UPDATE that permanently blanks the content column on affected
rows. Only content is touched — tool_calls and every other column are
left untouched, so provider tool_call/tool_result pairing survives.
dry_run reads through the no-lock read path and never writes.
Wire it up as `hermes sessions clean-markers [--dry-run]`, mirroring the
existing optimize/repair subcommands. Verified end-to-end against a real
temp state.db: dry-run reports the row without writing, the real run
clears it and preserves tool_calls, and a second run is a no-op.
The conversation_loop fix (previous commit) stops new "[memory]"-style
bare tool-call markers from being cached/persisted, but sessions written
before that fix can still carry rows where a bare marker was saved as
the assistant's "final response".
Add a load-on-read repair pass in hermes_state.py, mirroring the existing
_strip_background_review_harness defense-in-depth: on session restore,
any assistant row whose content is only a bracketed marker (e.g.
"[memory]", "[skill_manage]") AND that carries tool_calls has its content
blanked before the history re-enters the model's context. The tool call
and its result are left untouched so provider tool_call/tool_result
pairing stays intact. Sessions with no affected rows pass through the
normal path unchanged.
Local tool-call templates can emit a bare bracketed token (e.g. "[memory]")
as assistant content alongside a function call. The loop treated that
protocol scaffolding as visible content: it got cached as the post-tool
fallback, and when the next turn came back empty, the marker was replayed
as the final response and written into the persisted transcript. Later
context compaction preserved that history, letting the model repeat the
marker in subsequent turns.
Detect content that is only a bracketed marker (`[name]`) when the
response also carries tool_calls, and drop it before it can be cached
or persisted. Scoped narrowly: only fires alongside tool_calls, so a
genuine final response of "[memory]" without a tool call is unaffected.
Salvage of #65260's b7d7cfd0e (ported — the PR's close() predates ~4K
commits of teardown-step churn, so the hunk is re-anchored after step
6b rather than cherry-picked).
agent/codex_runtime.py already drops _codex_session on turn crash and
on retirement, but AIAgent.close() — the hard teardown for /new,
/reset, and session expiry — had no owner for it, so the app-server
child process survived until interpreter exit. Long-lived gateways
accumulate one leaked subprocess per ended Codex session.
The attribute is cleared BEFORE close() so a concurrent reader can't
observe a half-closed session and a raising close() can't strand a
stale reference (tested).
Tests extend the author's original lifecycle test with the
raising-close and no-codex-session cases.
Offload manual /compress temporary-agent cleanup through the existing
bounded off-loop helper so a slow agent.close() cannot freeze the
gateway event loop, heartbeat, or platform polling.
Guarantee Relay transport teardown even when the runner cancels
adapter.disconnect() during go_idle: shielded finally, 2s drain-path
idle ACK budget under the 5s outer disconnect budget, and bounded
supervisor/reader/ws.close awaits.
Original commits:
- fix(gateway): offload manual /compress cleanup from the event loop
- fix(gateway): tear down Relay transport even if go_idle is cancelled
- fix(gateway): keep Relay disconnect budgets inside the runner window
By @Dannyzen (PR #78027), salvaged onto current main.
ethernet
Teknium
brooklyn!
Jeffrey Quesnelle
kshitij
A.Alzaro
Jeongseok Kang
Aleksandr Pasevin
HexLab98
Bryan Bednarski
xxxigm
kshitijk4poor
阿泥豆
zabih-sudo
BobClawblaw
Hermes Agent
joaomarcos
JR Razmus
Dannyzen
Jordan