tools/managed_tool_gateway._read_nous_provider_state read auth.json with a
bare read_text(), which on Windows decodes as cp1252 and raises on any
non-ASCII byte (e.g. an accented Nous provider label). The broad except
swallowed it and returned None, so the gateway treated Nous as
unconfigured — the same Windows UTF-8 hazard the other auth.json readers
in this PR already fix.
Add encoding="utf-8-sig" (consistent with the sibling readers) plus a
non-ASCII regression test reusing the windows_default_encoding fixture.
This covers the one auth.json reader in #66782 not already handled here
(tools/managed_tool_gateway.py:40); the other two readers #66782 touches
(agent/auxiliary_client.py, tools/xai_http.py) are already fixed in this
PR. RED-verified.
Address review feedback on #58158: the regression suite covered four of
the changed readers but not _read_shared_nous_state (auth.py) or
_has_any_provider_configured (main.py), which also read UTF-8 stores the
Windows cp1252 default can corrupt.
Add a non-ASCII UTF-8 regression case for each, reusing the existing
windows_default_encoding fixture and _write_utf8 helper:
- _read_shared_nous_state: a nous_auth.json with an accented display_name
and valid tokens must round-trip intact (not return None). Pins
HERMES_SHARED_AUTH_DIR to tmp to satisfy the shared-store seat belt.
- _has_any_provider_configured: an auth.json whose active provider carries
a CJK label must still report a configured provider (the read must not
raise into the swallowing except). get_auth_status is faked so the
result is driven by the read, and provider env vars are cleared to reach
the auth.json branch.
Both tests are RED-verified — they fail when the respective
read_text(encoding=...) is reverted.
Follow-up to the auth.json UTF-8 read fix in this PR. A repo-wide scan for
the same bug class found three more callers that read ~/.hermes/auth.json
via Path.read_text() with no encoding — same Windows cp1252 hazard:
- agent/auxiliary_client.py _read_nous_auth: a non-ASCII byte raised
UnicodeDecodeError, the broad except swallowed it, and Nous silently
stopped being available as the auxiliary (vision/summarization) provider.
- tools/xai_http.py has_xai_credentials: same failure mode — xAI OAuth
silently looked absent on Windows.
- hermes_cli/main.py is_setup_complete: same; has a config.yaml fallback so
the impact is milder, but the read is still wrong.
All three now use read_text(encoding="utf-8-sig"), matching _save_auth_store's
write encoding. A repo-wide grep confirms there are no remaining
json.loads(...read_text()) reads of auth.json without an explicit encoding.
Tests: rewrote the Windows-encoding regression tests to actually exercise the
bug on POSIX too — a new windows_default_encoding fixture forces a no-encoding
read_text() to decode as cp1252 (the Windows default), and _write_utf8 now
emits real non-ASCII UTF-8 bytes (ensure_ascii=False) so the bytes actually
trip cp1252. Verified each test fails when its fix is reverted (including
the two new sibling-reader tests).
The auth store readers (_load_auth_store, _import_codex_cli_tokens, and the
shared Nous store reader) called Path.read_text() with no encoding, so bytes
were decoded with locale.getpreferredencoding() — cp1252 on Windows. The
stores are *written* as UTF-8 (os.fdopen(..., encoding="utf-8")), so any
non-ASCII byte (a CJK or emoji credential label, an accented display name in
OAuth state) raised UnicodeDecodeError on read.
Worst case: _load_auth_store's broad except then copied the file to .corrupt
and returned an empty store, silently wiping every provider credential on the
next launch. The sibling reader at line 2161 already used
read_text(encoding="utf-8"), confirming the omission was unintentional.
Use utf-8-sig (matching the .env handling in config.py) so a BOM from a
Notepad-edited file is tolerated too.
Adds regression tests covering the UTF-8 round-trip with a non-ASCII label,
BOM tolerance, no-corrupt-on-valid-load, and that the readers pass an explicit
encoding (guard against future regressions). Verified the tests fail when the
fix is reverted.
Closes no issue — found via cross-platform code audit (the bug is not in the
issue tracker).
Teknium
Nolan