Run a duplicate of CI on the new ARC (Actions Runner Controller) runners in
GKE, beside the existing CI. The duplicate does not change production CI.
Every workflow in .github/workflows/ that does not start with newci- is
byte-identical to main. Watch the shadow runs for a few days, then migrate.
The shadow set is 16 files: newci-ci.yml plus the 15 reusable workflows that
ci.yml calls on a pull request. Only pull-request workflows are copied.
js-autofix, deploy-site, and skills-index run on push or on a schedule. A
copy of those would push branches and deploy the site a second time.
Safety properties of the shadow:
- Concurrency groups are newci-prefixed. This is the important one. The
production groups use cancel-in-progress, so a shared group would let a
shadow run cancel the production run.
- Cache keys are newci-prefixed. The shadow cannot poison or evict a
production cache entry.
- Reusable-workflow calls point only at other newci-* files. No shadow job
calls a production workflow.
- The PR review comment runs with --dry-run. It prints the comment body to
the job log. Two pollers cannot fight over the hermes-ci-review-bot
comment.
- The gate job is renamed to "[newci] All checks pass (informational)". The
production check "All required checks pass" stays the only merge gate.
- The shadow runs on pull_request only. The push trigger is removed.
- docker publish and merge jobs are unreachable. Their conditions require a
push to main or a release.
Runner infrastructure, in the shadow copies only:
- Jobs go to three scale sets: arc-runner-small for short gate jobs,
arc-runner-set for general work, arc-runner-docker and arc-runner-arm64
for image builds. dind is only on the docker sets, so the other jobs stop
paying for a privileged sidecar.
- The runner image supplies node 26, npm 12, uv, Python, and ripgrep. The
setup-node, setup-uv, and per-job install steps are gone.
- Checkout uses a node-local git mirror, seeded from the runner pod env.
- buildx layer cache moved to Artifact Registry in us-central1, the same
region as the runners. Reads are keyless through GKE Workload Identity.
Writes use GitHub OIDC and happen only on main pushes and releases, so
pull-request code cannot write a layer that the publish job reads.
Merge-base work, in the shadow copies only:
- A new composite action, .github/actions/merge-base, deepens a shallow
clone until the two histories connect. fetch-depth: 0 fetches all ~1400
refs and measured 76-81s, against 3-6s for a shallow checkout.
- The action fails by default when no merge base exists. A three-dot diff
over a missing merge base scans nothing and reports clean, so the
supply-chain audit must stop. history-check sets fail-on-missing to false,
because absence is the result it measures.
- lint diffs against the base commit directly. The job checks out the PR
merge ref, so base.sha is already the correct comparison point.
- contributor-check uses origin/main..HEAD. The result equals the merge-base
form, and the extra git call also expanded a SHA without quotes.
Other changes:
- .github/actionlint.yaml declares the four ARC labels. actionlint knows
only GitHub-hosted labels, so every runs-on in the repo was reported as an
unknown label: 40 warnings that hid real findings.
- scripts/ci/resource_profile.py records CPU and memory for a job step. The
timing report shows the data per step.
- run_tests_parallel.py can list test files from the git index. The slice
generator then needs no blobs.
- Docker test files are split so boot-heavy tests run in parallel.
- Container-environment parity fixes in doctor, gateway, and skill_utils,
with tests.
To retire the shadow: delete .github/workflows/newci-*.yml.
Timing out a bounded git probe must not leave helper descendants
(credential helpers, git-remote-https, hook children) running after the
probe fails open. bounded_git_probe now spawns the child in its own
process group on POSIX (process_group=0), and _kill_git_process_tree
signals the whole group with os.killpg — gated on the child actually
leading its own group (pgid == pid), so a shared-group spawn can never
take down unrelated processes. Windows keeps the existing taskkill /T /F
tree kill.
Proven live on main: a fake git that forks a 300s descendant left the
descendant running after the probe timeout; with the fix the descendant
dies with the launcher. Fast path and fail-open contract unchanged.
Port of openai/codex#36793 (Terminate timed-out Git process trees).
allow you to simulate the whole official curl | bash installer,
and subsequent hermes updates.
Run development commands in a bubblewrap filesystem and network sandbox
with a local HTTPS MITM fixture server and a fake github
git-upload-pack transport.
Package the sandbox command and expose it from the nix devShell.
Stage the local installer at its canonical fake HTTPS URL and add a
persistent installation/update test path. Route root installs through
sandbox-owned filesystem locations and snapshot dirty source worktrees
into temporary fake commits so update tests can fast-forward without
changing the real checkout.
Includes a --install-ref sandbox installer mode that fetches any commit
(--from-main is a nice shorthand for local development) outside the
sealed sandbox, installs from that snapshot, and then promotes the fake
remote to the current worktree so update flows can be exercised with FF.
Notes on non-root sandboxes:
Giving a non-root sandbox a network is tricky.
slirp4netns joins the target userns and setuids to root before configuring the
netns, so the userns must map a uid 0; bwrap's --unshare-user maps exactly ONE
uid, so --uid 1000 leaves no root to become and slirp diedswith
`setns(CLONE_NEWNET): Operation not permitted`. Stage 1 builds the user+net
namespaces with `unshare` and two one-id ranges:
inner 0 -> a subuid, unused by the payload, present only so slirp can
become root
inner 1000 -> our real host uid
Mapping the payload to the *host* uid (not a second subuid) keeps everything the
sandbox writes owned by us, so `rm -rf` on a persistent sandbox still needs no
privileges. Stage 2 execs bwrap WITHOUT --unshare-user -- it only adds mount/pid
-- sidestepping bwrap's refusal to accept --uid outside a userns it created.
Costs a /etc/subuid range for the invoking user (we error with the exact line to
add) and util-linux `unshare`; `--root` needs neither.
Selecting an NVIDIA NIM model whose id reached config without the nvidia/
prefix produced a bare "HTTP 404: 404 page not found" — retried three times,
never naming the model. It reads exactly like an outage or an auth failure,
which is where the Discord thread spent its time before the id was spotted.
normalize_model_for_provider() had no branch for nvidia, so a bare id passed
straight through to the API. Repair it from the provider's curated catalogue:
a bare name that matches exactly one entry modulo the prefix gets it back.
That's a lookup, not a guess — build.nvidia.com also fronts local NIM
containers and third-party models, and anything absent from the catalogue is
left alone. Because the repair runs on every runtime setup, an already-broken
config self-heals on the next turn and prints what it changed.
If a bare id still reaches the wire, the 404 now explains itself. The
classifier consults the same catalogue: a prefix-less id the provider only
serves as vendor/model is a deterministic failure, so it classifies as
model_not_found instead of burning three retries on a retryable "unknown",
and the error trace names the id to use.
Fixes#78796
shutil.make_archive writes PAX with fractional-mtime records, which macOS
Archive Utility rejects ("Error 94 - Bad message") on double-click. Write
the profile archive with tarfile in GNU format instead: integer mtimes,
longlink for deep paths, extracts under Finder, bsdtar, and gnutar alike.
Verified against /usr/bin/tar (bsdtar) with >100-char member paths.
/export [profile] [-o output.tar.gz] bundles a profile into the shareable
archive; /import <archive> [--name <name>] adopts one as a new profile
(wrapper alias created when safe). Registry-driven, cli_only, so the CLI
and TUI both pick them up in autocomplete and help.
export_profile() accepts extra_files (root-relative filename -> text) so a
caller can stage companion files into the archive; the desktop uses it for
desktop.json, its appearance/interface overlay, now part of the default
profile's export allow-list.
New routes wrapping the existing hermes profile export/import machinery:
- POST /api/profiles/{name}/export (extra_files + optional output path)
- POST /api/profiles/import (returns the bundled desktop overlay)
- GET /api/profiles/{name}/desktop-overlay
Paths cross the API, not bytes - the desktop's native dialogs and its
local/pooled backends share a filesystem.
hermes debug share runs on the backend. A desktop app connected to a
remote, docker, or SSH backend writes desktop.log on the client machine,
so the bundle can never contain it — and the report rendered that as a
bare "(file not found)", which reads as "the app logged nothing" and
sends triage after a client-side bug it cannot see.
Name the writer and the path to collect by hand. Backend-written logs
are unchanged, a present desktop.log is still captured, and an empty one
still reports "(file empty)" — the app ran and logged nothing is a
different fact from the file being on another host.
hermes model saves custom_providers models: {default: {context_length}} for
local Ollama. That dict shape was treated as an explicit catalog, so no-key
endpoints skipped live /v1/models probing and Desktop/Telegram only showed
the saved default — Refresh could not help. Keep list/string shapes as
allowlists; pin dict catalogs with discover_models: false.
_has_any_provider_configured() probed every api_key provider (gh subprocess
for copilot alone takes 5s; full sweep ~18s) before consulting auth.json and
config.yaml, which are instant local reads. Desktop setup.status calls
blocked past the UI's timeout, causing the connect/disconnect boot loop.
Reorder so cheap local checks run first. Same semantics, ~35x faster here.
purge_stale_tool_call_markers ran a permanent, irreversible UPDATE with
no backup — inconsistent with repair_state_db_schema's backup-by-default
convention for destructive state.db operations elsewhere in this file.
Take a full snapshot via VACUUM INTO (safe against a live connection,
unlike the raw-copy _backup_db_file used for malformed-schema repair)
before the write, timestamped beside state.db. Skipped when dry_run or
when there's nothing to change. Add --no-backup to `hermes sessions
clean-markers`, mirroring `sessions repair`.
Verified end-to-end: the CLI run against a real temp state.db produces
the backup file before printing the cleared-row count.
The load-on-read repair (_strip_stale_tool_call_markers) fixes affected
sessions in memory on every resume, but never touches the DB — long-lived
sessions re-scan and re-repair the same rows on every load, and the
contaminated bytes stay in state.db (and any backup/cache snapshot of it)
indefinitely.
Add SessionDB.purge_stale_tool_call_markers(dry_run=False): a one-time,
idempotent UPDATE that permanently blanks the content column on affected
rows. Only content is touched — tool_calls and every other column are
left untouched, so provider tool_call/tool_result pairing survives.
dry_run reads through the no-lock read path and never writes.
Wire it up as `hermes sessions clean-markers [--dry-run]`, mirroring the
existing optimize/repair subcommands. Verified end-to-end against a real
temp state.db: dry-run reports the row without writing, the real run
clears it and preserves tool_calls, and a second run is a no-op.
Qwen3.8 Max is live on both OpenRouter and the Nous portal
(qwen/qwen3.8-max, 1M context, 131K max output). Per the
newest-max-replaces-last-max convention, it takes qwen3.7-max's slot
in both curated lists.
- hermes_cli/models.py: OPENROUTER_MODELS + _PROVIDER_MODELS[nous]
swap qwen/qwen3.7-max -> qwen/qwen3.8-max
- agent/model_metadata.py: DEFAULT_CONTEXT_LENGTHS entry for
qwen3.8-max at 1,000,000 (verified against OpenRouter live
metadata and Nous /v1/models 2026-08-03)
- tests/test_empty_model_fallback.py: swap incidental catalog fixture
to the surviving slug
- website/static/api/model-catalog.json: regenerated
Pricing snapshot skipped: both routes bill via official_models_api
(live pricing), verified with resolve_billing_route. Reasoning
timeout floor already covered by the qwen3 prefix (180s).
CI-caught: compute_prompt_breakdown still looked for <available_skills> in the stable tier, but #37117 moved it to volatile. Search volatile first, fall back to stable for older sessions.
Simplify-pass follow-up on the #70782 salvage: _bearer_auth_headers
hard-emits ${MCP_<NAME>_API_KEY} but install_entry only persists
auth.env-declared vars — a manifest naming its key differently (the
shipped n8n style) would install cleanly yet send a literal-placeholder
header at connect time (silent 401, the #37792 bug class). Enforce the
naming contract at parse time. Also pins the secret-stays-in-.env
property in the install test (raw config.yaml carries the template,
never the secret). Mutation-checked: validation disabled -> guard test
fails.
When an optional-mcps manifest declares transport.type=http with
auth.type=api_key, install_entry() prompts for the key and saves it to
.env, but _build_server_config() only handled the oauth case — the
api_key case produced a bare url entry with no headers, so every
request to the server was unauthenticated (-> 401).
Reuse _bearer_auth_headers(entry.name) from mcp_config.py so the
catalog path emits the same 'Authorization: Bearer ${MCP_..._API_KEY}'
template as the manual 'hermes mcp add --url' path.
Salvaged from #70782 (production hunk applied clean; tests re-anchored
onto current main). Credit: JonthanaHanh.
Review folds on the #60807 salvage:
- resolve_skin tests are behavioral (thread-ident probe + ready-frame
wiring check) instead of pure source inspection, per the #72720
pattern; a source assertion remains as belt-and-braces.
- The warm-list test does REAL imports and checks sys.modules —
_warm_gateway_module swallows ImportError by design, so the PR's
tracking-stub test would pass even with a typo'd module name.
- resolve_copilot_token logs a debug line when the env-var
short-circuit skips the gh-CLI fallback (behavioral change made
observable).
Three fixes for the Desktop/TUI cold-start stall where the event loop
is blocked for ~14s between HERMES_BACKEND_READY and the first
prompt (#60800):
1. copilot_auth: skip subprocess fallback when any
Copilot env var is explicitly set (even if invalid). The user
expressed token intent via env var; silently substituting a CLI
token is surprising and the subprocess adds up to 5s on Windows.
2. tui_gateway/ws: run resolve_skin() via asyncio.to_thread so config
loading + skin engine init do not block the WS read loop during
the cold-start RPC burst.
3. web_server: extend _warm_gateway_module to pre-import the heavy
module chains (auth, copilot_auth, runtime_provider, skin_engine,
inventory, model_switch) that the first WS connection + RPC burst
would otherwise import on the loop thread. These trigger .pyc
compilation and Defender scans on Windows (15-30s per the existing
comment) and were not covered by the original gateway-only warm.
Tests: 5 new tests in test_cold_start_gil_stall.py + 2 new tests in
test_copilot_auth.py. All 36 copilot_auth tests + 16 ws/web_server
tests pass.
Simplify-pass folds on the #23254 salvage:
- REUSE (HIGH): append_messages_batch now delegates row serialization to
the pre-existing _insert_message_rows helper (already shared by
replace_messages / archive_and_compact / portability import) instead
of adding a third serialization path (_prepare_message_row +
_MESSAGE_INSERT_SQL are gone). One row-writer for every multi-row
path; the row-ID return was consumed by no production caller, so the
batch returns the inserted count.
- QUALITY (HIGH): the compression-lock + compression-closed admission
guards are extracted into _check_transcript_write_guards, shared by
append_message and append_messages_batch (previously duplicated 23
lines that had already needed targeted fixes, #74478). The role-gated
reasoning filtering is no longer duplicated in run_agent.py — it
lives at its one site inside _insert_message_rows.
- EFFICIENCY (MEDIUM, measured): unbounded seed copies hold one BEGIN
IMMEDIATE for seconds (10k rows ~= 2.4s; FTS triggers dominate) and
monopolize the in-process write lock. append_messages_batch grows a
chunk_rows param; all seed/copy call sites use chunk_rows=500. Same
recovery semantics as the old per-row loops, bounded lock holds.
- REUSE (MEDIUM): the two remaining per-row branch-copy loops found by
the pass (gateway/slash_commands.py /branch, hermes_cli
cli_commands_mixin.py branch) are converted to chunked batches too
(AsyncSessionDB's generic to_thread forwarder covers the async site).
Turn-flush benchmark unchanged after the refactor: 2.43 -> 0.87 ms
median per 5-message flush (64% faster).
Every hashed bundle chunk under /assets/ was served with no caching
directives, so each dashboard load re-fetched (or at best revalidated)
every JS/CSS chunk. Those filenames carry a Vite content hash — the
bytes behind a given URL can never change; a rebuild mints new
filenames referenced by a freshly served index.html.
Mark them Cache-Control: public, max-age=31536000, immutable:
- the /assets StaticFiles mount, via a subclass that stamps the header
on 200s only (404s stay uncached — a rebuild can create the file),
- serve_css, preserving its X-Forwarded-Prefix url() rewrites for
/fonts/, /fonts-terminal/, /ds-assets/, /assets/.
index.html keeps no-store, no-cache, must-revalidate — it is the
mutable entry point that binds users to the current hashes.
The original PR also added hand-rolled per-request gzip compression of
asset responses; that part is deliberately dropped. This server is a
localhost-default dashboard backend: compressing every response on the
CPU to save loopback bandwidth is a pessimization, and callers that
front it with a real proxy already get compression there.
Salvaged from PR #28543 (idea by @sea-monsters; gzip groups dropped as
described above).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
le=100 would 422 real desktop callers: sessions-settings fetches
archived at limit=200, the command palette lists at 200, and the
electron remote-merge over-fetches limit+offset (exceeds 100 at
offset>=81, and its .catch(()=>null) silently drops remote sessions).
Clamp must sit above real client maxima. New test pins limit=200 w/
offset.
Re-derivation of aydnOktay's twin clamp PRs onto current main (the
session-list endpoints moved into web_routers/; the analytics endpoints
gained asyncio.to_thread wrappers since the originals):
- limit le=100 on /api/sessions, /api/sessions/search and the
/api/profiles/sessions fan-out (one unbounded request could drag every
session row + correlated-subquery preview work out of SQLite, times
every profile's state.db on the fan-out).
- days ge=1 le=365 on /api/analytics/usage + /api/analytics/models
(huge or non-positive values force full-history InsightsEngine work or
inverted windows; the UI only offers 7/30/90 presets).
FastAPI Query bounds reject at the validation layer (422). 8 new tests;
both clamp classes mutation-checked (clamp removed -> its tests fail).
On dashboard-only sessions nothing else executes check_fn warmers (they
live only in the tool-schema build), so the hub's read-only cache lookup
would report auth_required=False forever. On a cache miss, schedule a
deduplicated daemon-thread probe off the request path; the short hub TTL
surfaces the verdict on the next fetch.
Follow-ups on the salvaged bounded-read fix:
- refresh flow: the non-200 branch reads a STREAMED body, which fails
(ReadError/StreamClosed) once the httpx.Client context has exited —
moved inside the context. Repro + regression test use a real socket
server (MockTransport buffers in memory and cannot catch this).
- truncation guard: >limit bodies end with ...[truncated] (mutation-checked
against the is_stream_consumed fallback).
- test mocks now model the streamed-read surface (is_stream_consumed,
iter_bytes, client.send) so non-200 paths exercise the real bounded read.
The as_completed drain + `with` join made the parallel version WORSE than
sequential main in the common case (first endpoint succeeds fast, others
slow/unreachable): main returned at first success, the parallel version
waited for every straggler. Now: after each completion, walk endpoints in
priority order and return as soon as a success is unbeatable (all
higher-priority probes already finished); pool uses shutdown(wait=False) so
losers drain in the background. Mutation-checked: removing the early exit
makes the new timing test fail (8.2s vs <1.5s).
Z.AI has separate billing for general vs coding plans and global vs
China endpoints. On startup, detect_zai_endpoint() probes up to 4
endpoints sequentially with 8s timeout each, taking 8-9 seconds when
the first endpoints return non-200 (rate limited) before a working one
is found.
Replace the sequential loop with concurrent.futures.ThreadPoolExecutor
to probe all 4 endpoints in parallel. Results are returned in
ZAI_ENDPOINTS priority order so the preference chain is preserved.
Benchmark on macOS M4 Max, Python 3.11, Hermes v0.8.0:
Before: 8.8s (sequential: global=0.9s/429, cn=1.6s/429,
coding-global=4.3s/200, coding-cn=2.0s/200)
After: ~4.5s (single round-trip, bounded by slowest endpoint)
Signed-off-by: Merlin <merlin@merlin.me>
Re-derivation of PR #39399 onto current main: pass --prefer-offline to
the web-UI workspace install (both silent and verbose arms of
_install_web_deps) and to the update-time Node dependency refresh in
_update_node_dependencies, so npm reuses its local cache instead of
re-fetching metadata. Test expectations updated to match, mirroring the
PR's own test-update commit.
The picker path fetches the Copilot /models catalog multiple times per
process (list_authenticated_providers -> provider_model_ids ->
_fetch_github_models, plus get_copilot_model_context / normalize
helpers). Cache the filtered catalog at module level with a short TTL
so repeated picker opens do not pay a TLS handshake each time.
Fold-fixes on top of the original patch:
- key the cache by api_key so a mid-process credential swap never
serves the previous account's catalog
- use time.monotonic() so wall-clock adjustments cannot extend the TTL
- deep-copy on store/serve so callers cannot mutate cached entries
- tests updated to patch _urlopen_model_catalog_request (main routes
catalog fetches through open_credentialed_url now), plus TTL-expiry
and credential-change coverage
Extracted from #40276.
request_restart was calling stop() immediately, so the requesting turn stayed
in the drain wait set and got force-killed at restart_drain_timeout. Wait for
active work to reach zero first, then stop against an idle gateway.
A session's YOLO bypass lived only in the in-memory
tools.approval._session_yolo set (or the process-frozen --yolo env
var), so resuming a session in a fresh process silently reverted the
user's /yolo ON — dangerous commands started prompting again.
Persist a yolo_mode flag in the session row's model_config JSON and
restore it on every CLI resume path:
- SessionDB.set_session_yolo() merges the flag into model_config
(same lineage-preserving merge as update_session_runtime_lock);
SessionDB.session_yolo_enabled() reads it back, false on any parse
failure.
- /yolo toggle persists ON and OFF through the new helper; the
compression/branch session-id rotation carries the flag onto the
continuation row.
- --yolo launches record the flag at session creation (agent_init),
and a /yolo toggled before the lazily-created row exists is carried
into the creation-time model_config (_ensure_db_session).
- HermesCLI._restore_session_yolo() re-enables the bypass on startup
--resume/-c, the deferred init path, and mid-chat /resume, with a
visible '⚡ YOLO mode restored from session' notice. No-op under a
frozen process-wide --yolo and never enables on absent/garbage flags.
ethernet
Teknium
brooklyn!
Jeffrey Quesnelle
HexLab98
frohsinnllc
joaomarcos
Bryan Bednarski
Hao Wang
kshitij
kshitij
JonthanaHanh
jinglun010
joncaldwell90
embwl0x
ehz0ah
Carbon
youzi
aydnOktay
Rod Boev
Hafiz Ahmad Ashfaq
Jakub Wolniewicz
luyifan
light-merlin-dark
szzhoujiarui
Xue-1997