Complements the cherry-picked contributor fixes and closes out the
remaining sites of the 'missing explicit encoding' bug class, which is
now permanently gated by ruff PLW1514 (enabled repo-wide in
pyproject.toml and enforced by the blocking `ruff check .` step in
.github/workflows/lint.yml):
- tools/memory_tool.py: read MEMORY.md/USER.md via utf-8-sig so a
Notepad BOM never glues U+FEFF onto the first entry (issue #10878,
PR #10888 by @easyvibecoding — strict-decode contract of
_read_raw_checked preserved rather than errors="replace", so
undecodable files still refuse read-modify-write instead of being
lossily rewritten). Regression tests included.
- tools/skills_tool.py: SKILL.md and skill file reads pinned to
utf-8-sig + errors="replace" — deterministic across platforms instead
of the locale fallback proposed in PR #51701 (superseded: falling back
to cp1252/GBK makes the same skill render differently per host); .env
reader aligned with the canonical utf-8-sig dialect in hermes_cli/config.py.
- agent/shell_hooks.py, hermes_cli/main.py, gateway/slash_commands.py:
explicit utf-8 on the remaining fdopen/open text-mode sites flagged by
the AlexFucuson9 sweep series (#56033#56940#65565#66782#66791).
Co-authored-by: easyvibecoding <easyvibecoding@users.noreply.github.com>
Co-authored-by: AlexFucuson9 <AlexFucuson9@users.noreply.github.com>
Co-authored-by: flyingdoubleg <wangzhe00zju@gmail.com>
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
_remove_env_source() decides whether a credential var lives in ~/.hermes/.env
or the shell by scanning the .env with env_path.read_text(errors="replace") —
no encoding. read_text() with no encoding falls back to the system locale
(cp1252/GBK on Windows) and never strips a BOM.
The canonical .env readers in hermes_cli/config.py all use
encoding="utf-8-sig" precisely because 'users may edit .env in Notepad which
adds one' (a BOM), and doctor.py documents that .env is written as UTF-8
everywhere. This sibling reader diverged: on a Notepad-edited .env the BOM
prefixes the first line, so line.strip().startswith(f"{env_var}=") is False
for the first variable — the detector reports a .env-backed key as a phantom
shell export and prints a misleading 'still set in your shell environment'
hint on .
Match the canonical reader (utf-8-sig + errors=replace). Adds a regression
test with a BOM'd .env.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review asked for a BOM-prefixed JSON case alongside the existing
control-character coverage. The sandbox script now also feeds
json_parse a \ufeff-prefixed document and asserts the parsed value
round-trips (fails against the pre-fix helper, passes with the
BOM strip).
json_parse used json.loads(strict=False), which relaxes control
characters but rejects a leading UTF-8 BOM (U+FEFF). Windows CLI
tools and some files prepend a BOM, causing JSONDecodeError on
otherwise valid JSON output.
Strip a leading BOM before calling json.loads when the input is
a string with a U+FEFF prefix.
Original PR by @woxinwuhen713-bit (#57870).
Co-Authored-By: Claude <noreply@anthropic.com>
tools/managed_tool_gateway._read_nous_provider_state read auth.json with a
bare read_text(), which on Windows decodes as cp1252 and raises on any
non-ASCII byte (e.g. an accented Nous provider label). The broad except
swallowed it and returned None, so the gateway treated Nous as
unconfigured — the same Windows UTF-8 hazard the other auth.json readers
in this PR already fix.
Add encoding="utf-8-sig" (consistent with the sibling readers) plus a
non-ASCII regression test reusing the windows_default_encoding fixture.
This covers the one auth.json reader in #66782 not already handled here
(tools/managed_tool_gateway.py:40); the other two readers #66782 touches
(agent/auxiliary_client.py, tools/xai_http.py) are already fixed in this
PR. RED-verified.
Address review feedback on #58158: the regression suite covered four of
the changed readers but not _read_shared_nous_state (auth.py) or
_has_any_provider_configured (main.py), which also read UTF-8 stores the
Windows cp1252 default can corrupt.
Add a non-ASCII UTF-8 regression case for each, reusing the existing
windows_default_encoding fixture and _write_utf8 helper:
- _read_shared_nous_state: a nous_auth.json with an accented display_name
and valid tokens must round-trip intact (not return None). Pins
HERMES_SHARED_AUTH_DIR to tmp to satisfy the shared-store seat belt.
- _has_any_provider_configured: an auth.json whose active provider carries
a CJK label must still report a configured provider (the read must not
raise into the swallowing except). get_auth_status is faked so the
result is driven by the read, and provider env vars are cleared to reach
the auth.json branch.
Both tests are RED-verified — they fail when the respective
read_text(encoding=...) is reverted.
Follow-up to the auth.json UTF-8 read fix in this PR. A repo-wide scan for
the same bug class found three more callers that read ~/.hermes/auth.json
via Path.read_text() with no encoding — same Windows cp1252 hazard:
- agent/auxiliary_client.py _read_nous_auth: a non-ASCII byte raised
UnicodeDecodeError, the broad except swallowed it, and Nous silently
stopped being available as the auxiliary (vision/summarization) provider.
- tools/xai_http.py has_xai_credentials: same failure mode — xAI OAuth
silently looked absent on Windows.
- hermes_cli/main.py is_setup_complete: same; has a config.yaml fallback so
the impact is milder, but the read is still wrong.
All three now use read_text(encoding="utf-8-sig"), matching _save_auth_store's
write encoding. A repo-wide grep confirms there are no remaining
json.loads(...read_text()) reads of auth.json without an explicit encoding.
Tests: rewrote the Windows-encoding regression tests to actually exercise the
bug on POSIX too — a new windows_default_encoding fixture forces a no-encoding
read_text() to decode as cp1252 (the Windows default), and _write_utf8 now
emits real non-ASCII UTF-8 bytes (ensure_ascii=False) so the bytes actually
trip cp1252. Verified each test fails when its fix is reverted (including
the two new sibling-reader tests).
The auth store readers (_load_auth_store, _import_codex_cli_tokens, and the
shared Nous store reader) called Path.read_text() with no encoding, so bytes
were decoded with locale.getpreferredencoding() — cp1252 on Windows. The
stores are *written* as UTF-8 (os.fdopen(..., encoding="utf-8")), so any
non-ASCII byte (a CJK or emoji credential label, an accented display name in
OAuth state) raised UnicodeDecodeError on read.
Worst case: _load_auth_store's broad except then copied the file to .corrupt
and returned an empty store, silently wiping every provider credential on the
next launch. The sibling reader at line 2161 already used
read_text(encoding="utf-8"), confirming the omission was unintentional.
Use utf-8-sig (matching the .env handling in config.py) so a BOM from a
Notepad-edited file is tolerated too.
Adds regression tests covering the UTF-8 round-trip with a non-ASCII label,
BOM tolerance, no-corrupt-on-valid-load, and that the readers pass an explicit
encoding (guard against future regressions). Verified the tests fail when the
fix is reverted.
Closes no issue — found via cross-platform code audit (the bug is not in the
issue tracker).
send_cmd._load_hermes_env intentionally reimplements a minimal dotenv
load (no secret-source pulls, no sanitize rewrite, get_hermes_home path
resolution incl. Windows/profile override), so the shared-loader BOM fix
is mirrored in place: utf-8-sig primary read, BOM strip before the
latin-1 stream fallback.
Claude-Session: https://claude.ai/code/session_01JPmJz5u1Bvtw4cCRvRWnYr
utf-8-sig only covers the primary decode. BOM + invalid UTF-8 (e.g.
PowerShell BOM + cp1252 body) forced latin-1, which kept EF BB BF as
part of the first key name and dropped the canonical name. Strip the
BOM before latin-1 decode and load via stream so override= is preserved.
PowerShell 5.1 Set-Content -Encoding UTF8 and Windows Notepad write a
UTF-8 BOM. load_dotenv(encoding="utf-8") kept U+FEFF on the first key
name, so the canonical name was absent from os.environ and Hermes looked
unconfigured with no error. utf-8-sig strips the BOM and is a no-op for
BOM-less UTF-8; latin-1 fallback unchanged.
Own the surrogate-crash class at three chokepoints instead of leaf sites:
- finalize_turn scrubs final_response once where model text leaves the
conversation loop — covers oneshot stdout (#80366), NIM/any-provider
responses (#19819), and every delivery consumer of the turn result.
- _sanitize_gateway_final_response scrubs at the gateway chat-surface
boundary — Telegram utf16_len (#55309) and Signal formatting (#55143)
can no longer see a lone surrogate; raw-text surfaces keep passthrough.
- run_conversation walks the fully-built api_kwargs with
_sanitize_structure_surrogates so tool descriptions (session_search,
#50959) and every other request-body leaf are JSON-encodable before
any provider sees them.
Regression tests pin all three chokepoints plus helper semantics.
Cherry-picked alongside #79240 (TheophilusChinomona) and #80374
(rainbowgore) whose commits precede this one with authorship preserved.
Prevent UnicodeEncodeError when model text contains U+D800-range
surrogates by sanitizing to U+FFFD before writing to UTF-8 stdout.
Co-authored-by: Cursor <cursoragent@cursor.com>
Unknown charset labels (QQ Mail's RFC 1428 'unknown-8bit' placeholder,
misspelled names, garbage encoded-word charsets) raised LookupError from
bytes.decode — errors='replace' only guards decode errors, not a missing
codec — aborting the whole fetch batch. UIDs are marked seen before the
fetch, so the crash permanently dropped every message in the batch.
- _safe_decode(): alias table (unknown-8bit→utf-8, gb2312/gbk→gb18030,
ks_c_5601-1987→cp949, ...) then utf-8, then latin-1 last resort.
- _decode_header_value(): wraps decode_header() so a malformed RFC 2047
header degrades to the raw string instead of crashing.
- _extract_text_body(): all three decode sites now use _safe_decode.
Fixes#35901, fixes#55381, fixes#55383.
Follow-up per review of #74631.
The prior assertion (call_count == 0 OR interactive != True) also
passed if an unintended non-interactive migration occurred, which the
safe fallback (response='n') is supposed to prevent entirely. Replaced
with mock_migrate.assert_not_called().
6/6 pass in the full tests/hermes_cli/test_update_yes_flag.py file.
Ports #68497 forward onto current main per teknium1's review.
input() can raise UnicodeDecodeError when the terminal encoding
cannot decode the byte sequence (e.g. a non-UTF-8 locale, or an
embedded terminal). The prior port targeted hermes_cli/main.py, the
pre-refactor location -- the update pipeline moved to
hermes_cli/update_cmd.py in 927463efcc.
Per review, fixed all three interactive update prompts that call
input() directly, not just the one this originally targeted:
1. Config-migration prompt (update_cmd.py:~3989): extends the existing
except EOFError to also catch UnicodeDecodeError, prints an
actionable 'hermes config migrate' hint, and falls through to the
skip branch (response=n).
2. Stash-restore prompt (_restore_stashed_changes, ~line 971): the raw
input() call here had NO exception guard at all -- not even for
EOFError. Added a try/except covering both EOFError and
UnicodeDecodeError, falling back to the existing skip-restore path
(changes remain safely in git stash, restorable manually).
3. Upstream-remote prompt (_sync_with_upstream_if_needed, ~line 1274):
already caught (EOFError, KeyboardInterrupt) but not
UnicodeDecodeError -- added it to the existing tuple.
Also dropped the incorrect #12884 reference (a TUI sticky-scroll
report, unrelated to this update-encoding issue, per the review).
4 new tests pass covering all three call sites (config-migration prompt
via cmd_update end to end, stash-restore and upstream-remote prompts
via direct unit tests against their own functions), plus an EOFError
sanity test confirming the stash-restore fix doesn't regress that case
either (it had no guard before). 6/6 in the full
tests/hermes_cli/test_update_yes_flag.py file (no regression).
Cron and agent output that contains emoji, CJK, or accented text is
silently lost on Windows. When a job's output exceeds the platform limit
(MAX_PLATFORM_OUTPUT = 4000), DeliveryRouter._deliver_to_platform saves
the full text to disk and sends a truncated preview with a "full output
saved to ..." pointer. That save used Path.write_text(content) with no
encoding, so on Windows it encodes through the platform code page
(cp1252) and raises UnicodeEncodeError on any non-ASCII character. The
exception propagates out of _deliver_to_platform and deliver() records
the target as failed, so the whole truncate-and-send path aborts: the
user receives nothing — even though an ASCII payload of the same size
would deliver fine — and the promised backup file is never written. The
sibling local-file path (_deliver_local) had the identical defect. The
Windows-footgun CI gate misses this because it only inspects open() /
Path.open(), not Path.write_text().
Both writes now pass encoding="utf-8" explicitly so output is persisted
consistently across platforms.
Fixes silent loss of non-ASCII cron/agent output on Windows. The two
on-disk writes in the delivery router (`_deliver_to_platform`'s full
output save and `_deliver_local`'s file save) now write UTF-8 instead of
the platform-default code page, so emoji/CJK/accented output is saved
and delivered the same on Windows as on macOS/Linux.
N/A
- [x] 🐛 Bug fix (non-breaking change that fixes an issue)
- `gateway/delivery.py`: pass `encoding="utf-8"` to the `write_text`
call in `_save_full_output` (oversized-output backup) and the one in
`_deliver_local` (local file delivery).
- `tests/gateway/test_delivery.py`: add two regression tests that
simulate a non-UTF-8 Windows code page and assert oversized non-ASCII
output is still delivered and the backup/local files round-trip as
UTF-8.
1. `scripts/run_tests.sh tests/gateway/test_delivery.py` — 25 passing.
2. Revert either `encoding="utf-8"` argument and re-run: the two new
tests fail with `UnicodeEncodeError` from the cp1252 codec, proving
they catch the regression.
3. `python scripts/check-windows-footguns.py gateway/delivery.py` and
`ruff check gateway/delivery.py tests/gateway/test_delivery.py` both
pass.
- [x] I've read the Contributing Guide
- [x] My commit messages follow Conventional Commits
- [x] I searched for existing PRs to make sure this isn't a duplicate
- [x] My PR contains only changes related to this fix
- [x] I've run the gateway delivery tests and all tests pass
- [x] I've added tests for my changes
- [x] I've tested on my platform: macOS 15 (Darwin 25.5)
- [x] I've updated relevant documentation (README, docs/, docstrings) — or N/A
- [x] I've updated cli-config.yaml.example if I added/changed config keys — or N/A
- [x] I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
- [x] I've considered cross-platform impact (Windows, macOS) — this fix is specifically about Windows code-page encoding
- [x] I've updated tool descriptions/schemas if I changed tool behavior — or N/A
- description 210 -> 57 chars; author credits Ben Barclay (benbarclay) first
- optional-skills/creative/ (marketing vertical, narrowest audience of
the batch)
- dropped phantom 'image-generation-workflow' ref; visuals via the
image_generate tool
- honest handoff language: platforms without connectors end at approved
drafts marked handed-off, never claimed as published
- tests (10) incl. phantom-ref and honest-handoff guards
- docs regen scoped: per-skill page + one catalog row + one sidebar line
Persist display.personality only; apply rendered text as an in-session
overlay across CLI, TUI config.set, and gateway /personality.
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: EMT5320 <1908937833@qq.com>
Keep agent.system_prompt user-owned; named personalities resolve as an
ephemeral overlay via display.personality.
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: EMT5320 <1908937833@qq.com>
Skill polish:
- description 208 -> 57 chars; author credits Ben Barclay (benbarclay) first
- connector framing (google-workspace, obsidian, notion, email-inbox-triage)
- modern section order; boilerplate folded into step-local rules
Blueprint wiring (completes the batch's recipe integration):
- weekly-review blueprint loads weekly-review-planning; prompt follows the
skill's seven-section shape, drafts-only
- morning-brief blueprint loads google-workspace; prompt points at
references/daily-brief.md when connected
- important-mail blueprint loads email-inbox-triage
- blueprints index regenerated
Tests: 13 skill tests + two catalog invariants (every blueprint skills=
entry resolves to a real bundled skill; the four task blueprints are wired
to their procedure skills). 32 green across both files.
Opt-in via compression.codex_responses_native (default: false). When enabled,
gpt-5.6-family models on the direct OpenAI API (api.openai.com) or a ChatGPT
Codex subscription send context_management=[{type: compaction,
compact_threshold: N}] on Responses requests. OpenAI compacts server-side and
returns an encrypted compaction output item; Hermes captures it into the
existing codex_reasoning_items sidecar and replays it on later turns in place
of the pruned history — inheriting persistence, session replay, the
cross-issuer guard, and the encrypted-replay kill switch with zero new state.
Scope is deliberately hard-gated (agent/native_compaction.py, re-checked per
request): gpt-5.6 family only — gpt-5.1/5.2 fail server-side on the field
(HTTP 500 / stream stall, no structured rejection; live-verified) — and
direct OpenAI/Codex routes only; xAI, GitHub/Copilot, OpenRouter, relays,
and local servers never see the field.
Hermes' local compression stays armed as the fallback owner: the native
threshold is clamped ~8K tokens below the local trigger so the server
compacts first, and a structured provider rejection of context_management
disables native compaction for the session and retries without it
(one-shot guard in TurnRetryState).
Live-verified E2E on api.openai.com/gpt-5.6: server compaction fired at a
4K threshold, checkpoints captured and replayed, recall preserved across
3 turns; gpt-5.1 with the flag enabled stays clean (field never sent).
Direction credit: PR #76950 by @laryhorb explored native Responses
compaction; this is a minimal reimplementation on current main.
The salvaged fix covered run_workflow.py and hardware_check.py. The same
locale-default read of user-authored workflow JSON exists in five sibling
scripts (auto_fix_deps, check_deps, extract_schema, health_check,
run_batch) — same bug class, same utf-8-sig fix. Invariant test extended
to pin all nine read sites.
The pdf half of the original PR is superseded: those scripts were
replaced wholesale by the clean-room rewrite (#81890), which ships
UTF-8-explicit I/O enforced by its own invariant test.
The bundled comfyui and pdf skills read and write text files with the
locale-default codec. Both declare platforms: [linux, macos, windows], so
these paths run on hosts where that codec is not UTF-8 (cp1252 on US
Windows, cp936 on Chinese Windows, ASCII under LC_ALL=C).
Readers (the live bugs):
- run_workflow.py load_schema() and the main() workflow read parse
user-authored JSON. A non-ASCII label crashes json.load with
UnicodeDecodeError under a non-UTF-8 locale, and a file saved from a
Windows GUI editor carries a UTF-8 BOM that json.load rejects with
JSONDecodeError. Both are read as utf-8-sig, which is BOM-tolerant and
identical to utf-8 on BOM-less input. This differs from adecb0d1a,
which used plain utf-8 for the pdf form JSON; those payloads are
agent-authored and BOM-free by construction, these are not.
- hardware_check.py reads /proc/version and /proc/meminfo. Both are
Linux-gated so Windows never reaches them, but the C locale defaults to
ASCII, so they pin plain utf-8. No BOM is possible on /proc.
Writers (not currently broken):
- extract_form_structure.py and extract_form_field_info.py write their
JSON with json.dump, whose default ensure_ascii=True keeps the bytes
pure ASCII. Pinned anyway because the codec is the writer's contract,
not a property of what the caller happens to dump.
wf_path.open() is a Path.open() site that check-windows-footguns.py
deliberately does not flag (per the rule comment: "Path.open() is ALSO
affected ... and can be audited separately"). It is fixed here because it
is the same bug 156 lines from a site the checker does flag, and line 623
of the same file already uses read_text(encoding="utf-8").
Adds tests/skills/test_comfyui_skill.py with contract assertions plus two
live regressions that run load_schema in a child interpreter under
LC_ALL=C with PYTHONUTF8=0, and extends the office skill tests with writer
contract assertions. All 8 new tests fail without this change.
Note that pyproject.toml exempts skills/** from ruff PLW1514
(unspecified-encoding) because skill scripts are partly user-authored.
This change does not touch that exemption; the sites are fixed by hand,
the same way adecb0d1a did.
The local-search plugin shipped a ~16 MB client-side lunr index that
every visitor downloaded and hydrated before their first result — slow
on any connection, painful on poor ones, and another lazy-loaded chunk
that died during deploy skew windows. DocSearch answers from Algolia's
servers: no client index, instant results at any docs size.
- themeConfig.algolia with public search-only credentials (admin key is
not in the repo); contextualSearch keeps en/zh-Hans results separated
via the crawler's docusaurus_tag facets
- drop @easyops-cn/docusaurus-search-local from package.json + lockfile
- index live and verified: 9,404 records, query 'telegram' returns 374
hits with correct URLs
e81d18dfb collapsed six per-surface copies of reasoning resolution onto
resolve_reasoning_config() and, in its own words, "fixes the gateway
resolving reasoning against config model.default instead of the session's
effective model". It did not touch gateway/platforms/api_server.py, which
kept that defect.
_create_agent() called GatewayRunner._load_reasoning_config() with no
model on its first line — before the model precedence chain (browser lock
-> session /model -> session row -> route -> per-request -> defaults) has
run. Per-model agent.reasoning_overrides therefore keyed off model.default
on the one surface where every request names its own model: a request for
a model with an override silently got the global effort instead.
Resolve after the chain settles, so the override follows the model the
request actually runs. An explicit per-request reasoning parameter still
takes precedence over config.
The existing test stub for _load_reasoning_config took no arguments (it
mirrored the old call); it now matches the real signature, as the sibling
stub in the same file already did.
The _POSIX_PRIMITIVES tuple holds search-pattern STRINGS the linter
greps for in skill scripts — 'os.setsid' / 'signal.SIGKILL' are data,
not calls. Add inline # windows-footgun: ok suppressions.
Adds tools/skill_linter.py — a soft companion to the hard frontmatter
validator. It encodes the CONTRIBUTING 'Skill authoring standards
(HARDLINE)' conventions that today only a human reviewer catches:
- shell-utility references in prose (`grep`/`sed`/`cat`...) that should
name the native tool (search_files/patch/read_file)
- missing version/author/license/metadata.hermes block
- name != directory, invalid name format
- description over the 60-char prompt budget, marketing words
- dangling references/ links, forbidden scaffolding files
- POSIX-only script primitives without a platforms: gate
Findings are ADVISORY. skill_manage(create) attaches them as
lint_warnings + lint_hint in the success result; nothing is blocked
(the hard rejects already run in _validate_frontmatter). A CLI
(python -m tools.skill_linter <dir>) exits 1 only on ERROR-severity
findings so CI can gate on structural breakage without failing on nits.
Calibrated against the bundled skills/ tree: 76 advisory findings, exit 0,
no false positives after excluding repo-root scripts/ refs.
Inspired by MiniMax Code's skill-creator lint step; adapted to our
existing validator + skill_utils rather than a parallel system.
Teknium
solyanviktor-star
nankingjing
nankingjing
Nolan
Paulo Nascimento
rainbowgits
Theophilus Chinomona
ygd58
Hermes Agent
峯岸 亮
Kailigithub
kernel-t1
zcj1122
brooklyn!
Ben Barclay
William Chastain
Drexuxux
hermes-seaeye[bot]