the sparklines were stretched across the whole gantt bar, but the profiler
wraps a SINGLE step (.github/actions/profile runs it between start/stop), so
on a job dominated by checkout + uv sync + post-job cleanup the samples only
describe a slice in the middle. reproduced: a 30s profile inside a 100s job
whose profiled step ran t=60..90 drew at left 0% width 100% instead of
left 60% width 30% — putting a cpu spike visually under a step that never ran.
the profile json had no wall-clock anchor to place it with, only duration_s,
so emit started_at/completed_at as iso-8601 utc in the same shape as github's
job timestamps. monotonic() still drives the sampling loop (immune to clock
steps); the timestamps are purely for placement.
_profile_window_pct() converts that window into bar-relative percentages and
both overlay states now use it — the expanded holder directly, the collapsed
strip via a .res-clip wrapper so its 100%-width is relative to the window
rather than the bar. the two states are asserted to agree on the x-axis.
falls back to the full bar, i.e. exactly today's behaviour, when the profile
predates these fields, when the timestamps don't parse, or when the window
doesn't overlap the job at all (clock skew between the runner writing the
profile and github's timestamps). a profiler that outran the job's
completed_at is clamped to the bar, and a sub-percent window keeps a 0.5%
hairline so it can't collapse to invisible.
7 new tests. verified they discriminate: forcing the old always-stretch
behaviour fails 4 of them, while the 3 fallback tests keep passing since
full-bar is what they want. tests/ci 145/145.
ops/s alone can't tell you whether a device is saturated — it reads low on a
few large IOs that pin the disk at 100% busy, and high on many small cached
ones. sample io_ticks (diskstats field 13) instead: its delta over the
interval is device busy time, i.e. iostat's %util. take the busiest single
device rather than the sum, since summing across devices exceeds 100% on a
multi-disk node and means nothing as a saturation percentage.
divide by the real elapsed gap rather than the nominal interval — a loaded or
throttled runner drifts well past 1.0s and would otherwise report >100%.
emit a downsampled series (cpu/mem/disk, 0-100 ints) alongside the existing
summary, mean-bucketed to 180 points so a 40min job costs the same few KB as
a 40s one. the report inlines every profile into one self-contained html
file, so an unbounded 1Hz series would dominate its size. mean, not every Nth
sample: a spike that survives decimation by luck is misleading.
the report renders the series as svg sparklines over each gantt bar, in two
states off the same markup (3px strip when collapsed, full height when
expanded) via preserveAspectRatio=none. profiles predating the series field
degrade to table-only, no overlay.
tests/ci 62/62.
main added install-e2e.yml + install-e2e-run.yml; mirror them as newci-*
so the migration gets the same signal it has for the other lanes. triggers
match production exactly (same tag filters, same 12h cadence, cron offset
:20 -> :25 so the two runs don't contend for the pool).
it cannot fail the production job. it's a separate workflow that production
never calls or reads, and belt-and-braces the e2e step is continue-on-error
so a failing leg is reported in the summary instead of reddening a check.
the tolerance lives on the STEP, not the job: job-level continue-on-error is
not a legal keyword on a job that calls a reusable workflow (only name/uses/
with/secrets/strategy/needs/if/concurrency/permissions are) — my first draft
had it on the matrix jobs and would not have parsed. noted in the header so
nobody "fixes" it back.
the legs are expected to SKIP for now. the e2e runs inside dev-sandbox.sh
(bubblewrap), which needs to remount / as slave and mount a fresh /proc, and
a stock ARC pod denies both. probed in-cluster on nous-gke-runner:
default pod Failed to make / slave: EPERM
capabilities.add: [SYS_ADMIN] Can't mount proc: EPERM
SYS_ADMIN + apparmor/seccomp Unconfined Can't mount proc: EPERM
privileged: true works
arc-runner-docker does NOT qualify — only its dind sidecar is privileged,
the runner container isn't. so a ~30s preflight job probes bwrap and skips
the legs with the capability matrix in the step summary, rather than burning
~11min per leg to fail at the same mount. when infra adds a privileged set,
pass its label as `runner` and the legs start running with no other change.
actionlint clean; summary script exercised for both skipped-legs and
failing-leg shapes (exits 0 in both).
TestPrefetchServerRetainVisibility polled the mocked op-status endpoint at
the production _RETAIN_OP_POLL_INTERVAL_S of 0.5s, so each test burned ~1s
of real sleep plus a cross-thread hop to the shared hindsight event loop per
poll — all inside a 5.0s join on the background prefetch thread. On a busy
runner that thread is exactly what gets starved, so it outlived the join and
the assertion saw an empty order list: assert [] == ['recall'].
collapse the interval to 0.01s in these tests (against an AsyncMock there is
no server to be polite to) and route the joins through a helper that asserts
the thread actually finished, so a genuine wedge reports itself instead of
surfacing as a confusing empty-list mismatch.
also drop the two wall-clock assertions that measured runner load rather than
the contract under test: the drain-budget bound goes from 3.0s to budget+10s,
and the eviction test now asserts the status endpoint saw no further calls
(the actual "dropped ops aren't re-polled" contract) instead of timing the
second prefetch at <0.25s.
reproduced by pinning the tests to one core against N spinners on that same
core: at 16x oversubscription pre-fix failed 3/3 and post-fix passed 3/3.
class wall time drops 1.45s -> 0.44s for the recall test.
the idle deadline starts at popen, so the first window has to cover the
helper script's interpreter startup — wall clock the test doesn't control.
39975613b shrank the stt windows to 0.1s, which is under a cold `python -u`
on a loaded runner, so the child got killed before printing anything and
the assertion saw 'Terminated\n' instead of its progress line.
hoist the timings to named constants (idle 2.0s, 12 ticks x 0.25s) with a
setup assertion that total runtime still exceeds the idle window, so a
future speed pass can't silently tune the test into not exercising the
deadline reset. fix the same latent race in the sibling tts test it was
ported from (0.2s window vs a real spawn) before it fires too.
swept tests/ for the pattern; the only other hit (win_pty_bridge) is a
per-read poll inside a 5s outer loop, not a total budget.
Run a duplicate of CI on the new ARC (Actions Runner Controller) runners in
GKE, beside the existing CI. The duplicate does not change production CI.
Every workflow in .github/workflows/ that does not start with newci- is
byte-identical to main. Watch the shadow runs for a few days, then migrate.
The shadow set is 16 files: newci-ci.yml plus the 15 reusable workflows that
ci.yml calls on a pull request. Only pull-request workflows are copied.
js-autofix, deploy-site, and skills-index run on push or on a schedule. A
copy of those would push branches and deploy the site a second time.
Safety properties of the shadow:
- Concurrency groups are newci-prefixed. This is the important one. The
production groups use cancel-in-progress, so a shared group would let a
shadow run cancel the production run.
- Cache keys are newci-prefixed. The shadow cannot poison or evict a
production cache entry.
- Reusable-workflow calls point only at other newci-* files. No shadow job
calls a production workflow.
- The PR review comment runs with --dry-run. It prints the comment body to
the job log. Two pollers cannot fight over the hermes-ci-review-bot
comment.
- The gate job is renamed to "[newci] All checks pass (informational)". The
production check "All required checks pass" stays the only merge gate.
- The shadow runs on pull_request only. The push trigger is removed.
- docker publish and merge jobs are unreachable. Their conditions require a
push to main or a release.
Runner infrastructure, in the shadow copies only:
- Jobs go to three scale sets: arc-runner-small for short gate jobs,
arc-runner-set for general work, arc-runner-docker and arc-runner-arm64
for image builds. dind is only on the docker sets, so the other jobs stop
paying for a privileged sidecar.
- The runner image supplies node 26, npm 12, uv, Python, and ripgrep. The
setup-node, setup-uv, and per-job install steps are gone.
- Checkout uses a node-local git mirror, seeded from the runner pod env.
- buildx layer cache moved to Artifact Registry in us-central1, the same
region as the runners. Reads are keyless through GKE Workload Identity.
Writes use GitHub OIDC and happen only on main pushes and releases, so
pull-request code cannot write a layer that the publish job reads.
Merge-base work, in the shadow copies only:
- A new composite action, .github/actions/merge-base, deepens a shallow
clone until the two histories connect. fetch-depth: 0 fetches all ~1400
refs and measured 76-81s, against 3-6s for a shallow checkout.
- The action fails by default when no merge base exists. A three-dot diff
over a missing merge base scans nothing and reports clean, so the
supply-chain audit must stop. history-check sets fail-on-missing to false,
because absence is the result it measures.
- lint diffs against the base commit directly. The job checks out the PR
merge ref, so base.sha is already the correct comparison point.
- contributor-check uses origin/main..HEAD. The result equals the merge-base
form, and the extra git call also expanded a SHA without quotes.
Other changes:
- .github/actionlint.yaml declares the four ARC labels. actionlint knows
only GitHub-hosted labels, so every runs-on in the repo was reported as an
unknown label: 40 warnings that hid real findings.
- scripts/ci/resource_profile.py records CPU and memory for a job step. The
timing report shows the data per step.
- run_tests_parallel.py can list test files from the git index. The slice
generator then needs no blobs.
- Docker test files are split so boot-heavy tests run in parallel.
- Container-environment parity fixes in doctor, gateway, and skill_utils,
with tests.
To retire the shadow: delete .github/workflows/newci-*.yml.
Timing out a bounded git probe must not leave helper descendants
(credential helpers, git-remote-https, hook children) running after the
probe fails open. bounded_git_probe now spawns the child in its own
process group on POSIX (process_group=0), and _kill_git_process_tree
signals the whole group with os.killpg — gated on the child actually
leading its own group (pgid == pid), so a shared-group spawn can never
take down unrelated processes. Windows keeps the existing taskkill /T /F
tree kill.
Proven live on main: a fake git that forks a 300s descendant left the
descendant running after the probe timeout; with the fix the descendant
dies with the launcher. Fast path and fail-open contract unchanged.
Port of openai/codex#36793 (Terminate timed-out Git process trees).
Wires tests/install/install-update-e2e.sh into CI as a reusable workflow plus a
caller that fans out over real releases, because that is the question users care
about: can someone on a version they actually installed get to this commit?
install-e2e-run.yml takes `route` and `install-ref`, so the combinations that
matter are expressible without duplicating runner setup. Each leg is independent
-- its own runner, its own sandbox, its own install, nothing shared or rewound.
The starting versions are chosen at runtime by scripts/sandbox/pick-release-tags.sh:
newest, oldest, and an evenly spaced spread between (5 by default). Choosing at
runtime rather than hardcoding keeps the matrix honest -- a pinned list stops
covering the newest release the day after it ships, and pins an "oldest" long
after anyone still runs it. Newest catches "did the last release break
updating?", oldest is the longest upgrade jump still possible, and the spread
samples the migrations in between (config-schema bumps, venv layout changes,
dependency floors). Tags are read from the checkout with `git tag --list`, not
`git ls-remote`: the job has the repository already, so this needs no network,
works offline and on a fork, and takes 8ms. The repo is derived from the
script's own resolved path rather than $PWD, so a copy cannot silently report a
different checkout's tags. The pick-releases job takes the checkout that suits
it -- blob:none filter, sparse-checkout of just that script, and fetch-tags,
since tags are the entire input and the default shallow checkout has none.
Triggers match the shape of the work:
* every 12 hours, so upstream drift (a new uv, a Node bump, a PyPI change)
surfaces on a schedule instead of in someone's review cycle;
* on release tags, the moment the set of versions users can update FROM
changes and the moment a broken updater would strand them;
* manually, with the route and the sample size as inputs.
Not on pull_request: a leg is ~9 minutes of real toolchain installation and the
matrix multiplies it. fail-fast is off so one broken release does not mask the
others, and max-parallel caps the fan-out so a run does not hammer the runners
or PyPI. The tag list is resolved once and shared by both route matrices, so the
two routes cover the same versions.
Artifact names include the sanitized install-ref, since a matrix runs the
reusable workflow several times per route and same-named artifacts collide; that
name is built in a step because Actions expressions have no string-replace
function. The name step runs with `if: always()`, since a failing leg is exactly
when its logs are wanted.
.gitignore covers .hermes-sandbox-e2e*/ rather than the bare directory: the
per-route sandbox trees (-update, -installer) fell outside it, so the sandbox
made the worktree dirty and dev-sandbox reacted by snapshotting the working copy
into a fresh fake-main commit on every invocation.
Nothing covered the update path, which is the worst thing to break: a broken
updater strands users on the version that cannot fix itself. `hermes update`
alone is ~2000 lines (hermes_cli/update_cmd.py) and had no end-to-end test.
tests/install/install-update-e2e.sh installs a genuine earlier Hermes through
the real one-liner (curl -fsSL https://…/install.sh | bash, served by
dev-sandbox's MITM proxy at the canonical URL, cloning "github.com" through the
upload-pack shim), which really installs uv, a managed Python, Node and the
venv. It then applies ONE update route and requires the checkout to land on this
commit with `hermes --version` still working -- so a pass means the venv and
entry point survived, not merely that git moved.
One route per run, each on a sandbox built from scratch. Sharing one install
across routes -- or rewinding with `git reset --hard` between them -- leaves the
second route running against a tree the first already updated (same venv, same
console script, same __pycache__), which is not the state any real user is in: a
route could pass only because its predecessor did the work, and a failure in the
first left the second exercising something undefined.
--install-ref chooses what to install first, so this covers "update from an
older release", not just from the tip. Installer flags are probed against the
target rather than assumed, because releases from months back predate flags
current Hermes takes for granted: --skip-browser is read out of that ref's own
install.sh, and `--yes` is asked of the installed `hermes update --help` (the
update subcommand has lived in main.py, subcommands/update.py and update_cmd.py
across the tags we sample, so a static parse rots silently -- and did). Without
those probes, old releases die on "Unknown option: --skip-browser" and
"unrecognized arguments: --yes" before doing any work.
Installer output is streamed through tee rather than captured: a real install of
uv, Python, Node and the venv IS the substance of this test, so it belongs in
the job log, not only in an artifact. pipefail keeps the installer's exit status
rather than tee's, so a failed install cannot look like a pass. The sandbox's
own proxy log is printed in full on failure, since a rejected TLS handshake
explains a failure that otherwise reads as a bare `curl: (35)`.
Deliberately reuses dev-sandbox rather than adding a second harness. An earlier
draft rewrote install.sh's hardcoded URLs with insteadOf and ran it against the
host; that tested the installer LESS faithfully (bash install.sh instead of the
real one-liner, host libs instead of a clean machine, ssh disabled to keep a
failed rewrite from reaching real GitHub) while duplicating a fake Internet we
already have.
Shell, not pytest, so scripts/run_tests.sh and run_tests_parallel.py stay
untouched: a pytest version needed an entry in the former's `env -i` credential
allowlist and a _SKIP_PARTS exclusion in the latter, and every meaningful line
was a command run inside the sandbox anyway.
Two guards, both earned during bring-up. It prefers the `sandbox` wrapper and
falls back to the raw script only when bwrap is on PATH (under Nix the wrapper
supplies the PATH and DEV_SANDBOX_* vars, so the bare script exits 127). And it
refuses to run on a dirty worktree: every dev-sandbox invocation re-derives fake
main from the working copy, so uncommitted changes move the update target
between the call that installs and the call that verifies -- a failure that
looks like a broken updater but is a moving reference.
allow you to simulate the whole official curl | bash installer,
and subsequent hermes updates.
Run development commands in a bubblewrap filesystem and network sandbox
with a local HTTPS MITM fixture server and a fake github
git-upload-pack transport.
Package the sandbox command and expose it from the nix devShell.
Stage the local installer at its canonical fake HTTPS URL and add a
persistent installation/update test path. Route root installs through
sandbox-owned filesystem locations and snapshot dirty source worktrees
into temporary fake commits so update tests can fast-forward without
changing the real checkout.
Includes a --install-ref sandbox installer mode that fetches any commit
(--from-main is a nice shorthand for local development) outside the
sealed sandbox, installs from that snapshot, and then promotes the fake
remote to the current worktree so update flows can be exercised with FF.
Notes on non-root sandboxes:
Giving a non-root sandbox a network is tricky.
slirp4netns joins the target userns and setuids to root before configuring the
netns, so the userns must map a uid 0; bwrap's --unshare-user maps exactly ONE
uid, so --uid 1000 leaves no root to become and slirp diedswith
`setns(CLONE_NEWNET): Operation not permitted`. Stage 1 builds the user+net
namespaces with `unshare` and two one-id ranges:
inner 0 -> a subuid, unused by the payload, present only so slirp can
become root
inner 1000 -> our real host uid
Mapping the payload to the *host* uid (not a second subuid) keeps everything the
sandbox writes owned by us, so `rm -rf` on a persistent sandbox still needs no
privileges. Stage 2 execs bwrap WITHOUT --unshare-user -- it only adds mount/pid
-- sidestepping bwrap's refusal to accept --uid outside a userns it created.
Costs a /etc/subuid range for the invoking user (we error with the exact line to
add) and util-linux `unshare`; `--root` needs neither.
Selecting an NVIDIA NIM model whose id reached config without the nvidia/
prefix produced a bare "HTTP 404: 404 page not found" — retried three times,
never naming the model. It reads exactly like an outage or an auth failure,
which is where the Discord thread spent its time before the id was spotted.
normalize_model_for_provider() had no branch for nvidia, so a bare id passed
straight through to the API. Repair it from the provider's curated catalogue:
a bare name that matches exactly one entry modulo the prefix gets it back.
That's a lookup, not a guess — build.nvidia.com also fronts local NIM
containers and third-party models, and anything absent from the catalogue is
left alone. Because the repair runs on every runtime setup, an already-broken
config self-heals on the next turn and prints what it changed.
If a bare id still reaches the wire, the 404 now explains itself. The
classifier consults the same catalogue: a prefix-less id the provider only
serves as vendor/model is a deterministic failure, so it classifies as
model_not_found instead of burning three retries on a retryable "unknown",
and the error trace names the id to use.
Fixes#78796
Selecting an NVIDIA NIM model whose id reached config without the nvidia/
prefix produced a bare "HTTP 404: 404 page not found" — retried three times,
never naming the model. It reads exactly like an outage or an auth failure,
which is where the Discord thread spent its time before the id was spotted.
normalize_model_for_provider() had no branch for nvidia, so a bare id passed
straight through to the API. Repair it from the provider's curated catalogue:
a bare name that matches exactly one entry modulo the prefix gets it back.
That's a lookup, not a guess — build.nvidia.com also fronts local NIM
containers and third-party models, and anything absent from the catalogue is
left alone. Because the repair runs on every runtime setup, an already-broken
config self-heals on the next turn and prints what it changed.
If a bare id still reaches the wire, the 404 now explains itself. The
classifier consults the same catalogue: a prefix-less id the provider only
serves as vendor/model is a deterministic failure, so it classifies as
model_not_found instead of burning three retries on a retryable "unknown",
and the error trace names the id to use.
Fixes#78796
shutil.make_archive writes PAX with fractional-mtime records, which macOS
Archive Utility rejects ("Error 94 - Bad message") on double-click. Write
the profile archive with tarfile in GNU format instead: integer mtimes,
longlink for deep paths, extracts under Finder, bsdtar, and gnutar alike.
Verified against /usr/bin/tar (bsdtar) with >100-char member paths.
'Move to project' submenu in the session actions menu (kebab and right-click,
via the shared MenuKit) listing every project with a folder except the current
owner. Picking one calls session.workspace.move at the project root, mirrors
the new cwd/branch/root into the $sessions cache, and refreshes the tree so
the row hops immediately.
A session created in the wrong directory needs its cwd corrected after the
fact. session.cwd.set only reaches live runtime sessions, so cold rows were
stuck. The new RPC targets the persisted row by session_key, validates the
folder, and REPLACES the git branch/root identity (update_session_cwd grows a
replace_git_meta flag) so the project tree's grouping follows the move instead
of pinning the session under the project it left via a stale git_repo_root.
A live idle agent bound to the row is re-anchored through the runtime path;
a mid-turn session refuses with 'session busy'. Runs on the RPC pool — the
git probes are subprocesses.
Chrome's grammar on every zone tab strip: Shift-click ranges from the
anchor, ⌥-click (Ctrl-click off-Mac — ⌘ stays close, ⌃ stays the macOS
context menu) toggles, plain click collapses back to one tab. Selected
tabs wear an accent wash; dragging any of them carries the block — the
ghost chip counts it — into a strip slot, a zone edge, or a Shift-span,
so three tabs land in a new zone as one gesture.
movePanes/reorderPanesInGroup/mergeZonesWithPane now take a block of pane
ids in strip order: the lead pane decides the drop geometry (slot, split,
span-merge) and the rest stack in behind it, with the pressed tab fronting
at the destination. The tab-selection store holds the block (Chrome
grammar: toggle, anchor range, collapse on plain click), drag-session
carries it — every dragged tab dims, the insertion slot skips the whole
block, and a landed drop spends the selection while a deny-area release
keeps it for a retry.
Adds a last_read_at watermark to the sessions table so surfaces (CLI,
TUI, desktop) can badge unread conversations. Read state derives from
the watermark vs latest activity, so new messages flip a conversation
back to unread with zero writes on the message path. NULL means never
tracked, so shipping the column doesn't badge pre-existing history.
set_session_read() stamps the whole compression lineage, matching the
archive/pin semantics; list_sessions_rich() rows carry a derived
`unread` key. DB layer only — no surface exposes it yet.
* fix(discord): reject empty outbound messages
* test(discord): cover empty final reply backfill state
Missed-message backfill decides what to replay from discord_messages, so
a dropped final reply must be recorded as failed by the new guard the
same way the exception path records one — otherwise the reply is both
never sent and never retried.
Co-authored-by: Jony <619963502@qq.com>
* chore: map 619963502@qq.com to zyz619963502zyz for PR #73449 salvage
---------
Co-authored-by: Jony <619963502@qq.com>
Export stages desktop.json (skin + mode, bundled user-theme definitions,
rail color, layout tree) into the CLI's own profile archive; import applies
it, so the receiver gets the whole look as a ready-to-use profile. Doors:
Export/Import profile... in Cmd-K, an import button beside the rail's +,
and Export in each profile square's context menu. New selectSavePath IPC
(native save dialog); credentials never leave the machine (CLI filter).
/export [profile] [-o output.tar.gz] bundles a profile into the shareable
archive; /import <archive> [--name <name>] adopts one as a new profile
(wrapper alias created when safe). Registry-driven, cli_only, so the CLI
and TUI both pick them up in autocomplete and help.
export_profile() accepts extra_files (root-relative filename -> text) so a
caller can stage companion files into the archive; the desktop uses it for
desktop.json, its appearance/interface overlay, now part of the default
profile's export allow-list.
New routes wrapping the existing hermes profile export/import machinery:
- POST /api/profiles/{name}/export (extra_files + optional output path)
- POST /api/profiles/import (returns the bundled desktop overlay)
- GET /api/profiles/{name}/desktop-overlay
Paths cross the API, not bytes - the desktop's native dialogs and its
local/pooled backends share a filesystem.
Three fakes pin mark_exhausted_and_rotate's signature explicitly and broke on
the new argument. They now assert it rather than just tolerate it — the xAI
spending-limit case is exactly the billing-403 this fixes, so it should be
pinning `failure_reason == "billing"`.
The sole-credential cooldown sized the bench from the raw HTTP status, but
403 is overloaded: error_classifier maps OpenRouter's "key limit exceeded"
and xAI's spending-limit block to FailoverReason.billing, while an edge
throttle with the same status is transient. Only 402 was excluded from the
short cooldown, so a spent account on a single key retried every 60 seconds
and re-failed forever.
Thread the classified reason from recover_with_credential_pool through
mark_exhausted_and_rotate to _exhausted_ttl. Billing keeps the full bench
regardless of status; everything else transient still recovers in 60s. The
verdict is stored on the entry (_EXTRA_KEYS, so it persists to auth.json) —
without that a restart would re-read a bare 403 and downgrade the bench.
Tests: sole billing-403 stays benched, survives reload, unclassified 403
still recovers; call-site coverage that the reason actually reaches the pool.
Three existing kwargs assertions updated for the new argument.
next_available_at() was computing the full 1-hour TTL for a sole
credential on a 429, contradicting the 60s cooldown in _available_entries.
The fallback restore gate (agent_runtime_helpers) uses next_available_at
to decide when to switch back from fallback to primary — so the agent
stayed on fallback for an hour instead of ~60s.
Add sole_credential computation in next_available_at mirroring
_available_entries, and a test verifying the short cooldown propagates.
A pool with only one usable (non-DEAD) credential has nothing to rotate to.
On a transient throttle (429 rate-limit, 403 edge-throttle, 5xx) the offending
key was benched for a full hour (EXHAUSTED_TTL_429/DEFAULT), so single-key /
no-fallback setups got an hour of hard failures for a throttle that resets in
seconds. The pool already special-cases 401 to recover quickly for single-key
setups; extend that to transient throttles when the credential is the sole
non-DEAD entry. 402 (billing/quota) keeps the full bench — a quick retry can't
help. Provider-supplied reset_at still overrides.
Adds tests covering sole 429/403 recovery, 402 full-bench, and multi-key
(no early recovery).
hermes debug share runs on the backend. A desktop app connected to a
remote, docker, or SSH backend writes desktop.log on the client machine,
so the bundle can never contain it — and the report rendered that as a
bare "(file not found)", which reads as "the app logged nothing" and
sends triage after a client-side bug it cannot see.
Name the writer and the path to collect by hand. Backend-written logs
are unchanged, a present desktop.log is still captured, and an empty one
still reports "(file empty)" — the app ran and logged nothing is a
different fact from the file being on another host.
* fix(agent): adopt .env credential/base-url edits at the turn boundary
A Settings save (desktop PUT /api/env, hermes setup) updates .env and
the saving process's os.environ, but a live session worker keeps the
base_url/api_key captured at agent init until restart — an open chat
silently kept calling the old endpoint (e.g. a local-server key sent to
api.openai.com, failing with an opaque 401).
Add AIAgent._try_refresh_env_client_credentials(), called at the start
of each conversation turn: re-resolve the provider's env-sourced
credentials (load_env() is mtime-memoized, so an unchanged file costs
one stat()) and rebuild the client via the existing
_replace_primary_openai_client machinery when the user edited them.
The refresh reacts only to env edits — resolved values changed since
the last look — never to mere divergence from the agent's current
values: credential-pool rotation and failover legitimately move the
session off the env credential, and stomping those back would flap.
Config model.base_url / pool custom endpoints keep precedence: edits
are only adopted while the session still runs on the registry default
or the previously-seen env value.
Lift _get_env_prefer_dotenv out of _seed_from_env to module level
(get_env_prefer_dotenv) so both the pool seeder and the per-turn
refresh share the same .env-over-os.environ resolution, including the
op:// indirection handling.
Fixes#67821
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(agent): address sweeper review on env credential refresh
- Cover named custom providers (#67935): provider="custom" has no
PROVIDER_REGISTRY entry, so resolve the config block's key_env through
the same lookup the runtime resolver uses.
- Make the edit baseline transactional: a failed client rebuild rolls the
agent back and leaves _env_creds_seen un-advanced so the unchanged edit
is retried next turn.
- Recompute route-derived TLS material and default headers on a base-url
change, via a _reapply_route_client_config helper shared with
credential-pool rotation so the two paths cannot drift.
- Rebase onto main: get_env_prefer_dotenv keeps the scoped _get_secret
semantics from the profile-isolation fix (no raw os.environ reads).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: map jskang@lablup.com to rapsealk
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
* feat(desktop): expose native OS notifications to plugins via ctx.notifyNative
Desktop plugins can toast in-app (host.notify) but have no sanctioned way to
reach the OS notification pipeline the app's own approval/turn alerts use, so
a plugin surfacing a genuinely notable background event (e.g. a discovery
plugin finding a match) stays invisible once the user steps away from Hermes.
Add a curated per-plugin door instead of exporting the raw dispatcher:
- ctx.notifyNative({ title, body?, silent? }) on PluginContext — attributed
to the plugin id, routed through dispatchNativeNotification so every
existing gate applies (master + per-kind prefs, post-connect baseline,
away-from-app gating, throttle).
- New 'plugin' native-notification kind with its own Settings ▸ Notifications
toggle (default on), so users silence plugins without losing app alerts.
- New optional `tag` discriminator on the notify payload keys the renderer
throttle and main-process cross-window dedupe per plugin, so two plugins
can't collapse each other's session-less notifications.
Consumer: the Index Network desktop plugin wants background opportunity
alerts; anything in ~/.hermes/desktop-plugins gets the same door.
* feat(desktop): ctx.os — the curated OS door for plugins
Fold ctx.notifyNative into a ctx.os namespace so every way a plugin
reaches outside the app window lives behind one attributed door instead
of accreting one top-level ctx method per capability:
- ctx.os.notify — the native-notification door from the previous commit,
unchanged semantics (plugin kind pref, away-gating, per-plugin throttle).
- ctx.os.openExternal / ctx.os.revealPath / ctx.os.writeClipboard — the
existing window.hermesDesktop bridge capabilities, now sanctioned and
result-shaped: each resolves false (never throws) when the bridge or
member is missing, so a plugin branches on the result instead of
sniffing the preload surface or crashing on an older shell.
No new Electron surface: everything routes through bridge members the
app already ships; the notification path keeps every existing gate.
---------
Co-authored-by: seref <1573640+serefyarar@users.noreply.github.com>
* fix(credential-pool): clear exhaustion state on key rotation
When a user rotates an API key (e.g. via `hermes setup` after hitting a
rate limit), _upsert_entry updates the access_token on the existing pool
entry but preserves the stale last_status=exhausted from the old key.
On the next session the pool finds the entry, sees it exhausted, and
returns no usable credentials — even though the new key is valid.
Fix: when access_token changes on an existing entry, reset last_status,
last_error_code, last_error_reason, last_error_message, and
last_error_reset_at. The exhaustion state belongs to the old key, not
the new one.
* chore: add pasevin@gmail.com to AUTHOR_MAP
* fix: clear last_status_at on key rotation, remove unused pytest import
Address review feedback from teknium1 on PR #22622:
- Add last_status_at=None to the reset block (matches all other
token-sync reset paths in credential_pool.py)
- Assert last_status_at is None in the regression test
- Remove unused pytest import flagged by ruff + ty
Fold ctx.notifyNative into a ctx.os namespace so every way a plugin
reaches outside the app window lives behind one attributed door instead
of accreting one top-level ctx method per capability:
- ctx.os.notify — the native-notification door from the previous commit,
unchanged semantics (plugin kind pref, away-gating, per-plugin throttle).
- ctx.os.openExternal / ctx.os.revealPath / ctx.os.writeClipboard — the
existing window.hermesDesktop bridge capabilities, now sanctioned and
result-shaped: each resolves false (never throws) when the bridge or
member is missing, so a plugin branches on the result instead of
sniffing the preload surface or crashing on an older shell.
No new Electron surface: everything routes through bridge members the
app already ships; the notification path keeps every existing gate.
Show earlier spends the already-materialized DOM budget first and only asks the
session store for another page once that is exhausted, so the click stays cheap
and the store window stays as small as it can be.
Paging has no ceiling: each expand grows the window by one budget page until the
whole transcript is loaded. Branch persistence stays wired throughout —
setMessages is never dropped, so switchToBranch and applyBranchVisibility keep
working on a windowed session.
Co-authored-by: HexLab <8422520+HexLab98@users.noreply.github.com>
An oversized session rebuilt an unbounded runtime repository on every store
update and exhausted the renderer's V8 heap, crash-looping the window. The DOM
budget in thread/list.tsx bounds what PAINTS, but every message was still
normalized into the repository first, so a session only had to be heavy — not
visible — to kill the renderer.
selectTranscriptWindow keeps the tail that fits one render-weight page. Weight,
not message count: measured against a real 1,175-session store, a 400-message
cap disengages on 37 sessions that are heavy but short (one is 133 messages /
1.05MB) while firing on 92 long-but-light sessions that were never at risk.
The cut aligns off branch-group boundaries. useRuntimeMessageRepository records
a group's fork point the first time it sees the group, so a window starting
mid-group would re-parent the surviving branches to whatever happened to
precede them.
Co-authored-by: HexLab <8422520+HexLab98@users.noreply.github.com>
messageRenderWeight moves out of thread/list.tsx into lib/render-weight.ts.
The DOM page budget already spends render cost rather than message count —
the store window added next needs the same currency, and one weight function
keeps the two layers from drifting apart.
No behavior change.
Desktop plugins can toast in-app (host.notify) but have no sanctioned way to
reach the OS notification pipeline the app's own approval/turn alerts use, so
a plugin surfacing a genuinely notable background event (e.g. a discovery
plugin finding a match) stays invisible once the user steps away from Hermes.
Add a curated per-plugin door instead of exporting the raw dispatcher:
- ctx.notifyNative({ title, body?, silent? }) on PluginContext — attributed
to the plugin id, routed through dispatchNativeNotification so every
existing gate applies (master + per-kind prefs, post-connect baseline,
away-from-app gating, throttle).
- New 'plugin' native-notification kind with its own Settings ▸ Notifications
toggle (default on), so users silence plugins without losing app alerts.
- New optional `tag` discriminator on the notify payload keys the renderer
throttle and main-process cross-window dedupe per plugin, so two plugins
can't collapse each other's session-less notifications.
Consumer: the Index Network desktop plugin wants background opportunity
alerts; anything in ~/.hermes/desktop-plugins gets the same door.
Hitting ⌘1 (or cycling ⌃Tab onto the main tab) while Capabilities /
Messaging / Artifacts covered the workspace looked dead: the workspace
pane was already the zone's active tab behind the page, so fronting it
changed nothing on screen.
activateTreeTabSlot / cycleTreeTabInFocusedZone now return the activated
pane id, and the keybind handlers route back to the loaded session (or
the new-chat draft) when the landing pane is the workspace under a full
page — the same rule openSession already applies.
hermes model saves custom_providers models: {default: {context_length}} for
local Ollama. That dict shape was treated as an explicit catalog, so no-key
endpoints skipped live /v1/models probing and Desktop/Telegram only showed
the saved default — Refresh could not help. Keep list/string shapes as
allowlists; pin dict catalogs with discover_models: false.
ethernet
Teknium
brooklyn!
Jeffrey Quesnelle
kshitij
A.Alzaro
Jeongseok Kang
Aleksandr Pasevin
seref
HexLab98