fix: resolve tiktoken encoding without constructing the embedding client (#955)

EmbeddingClient.encoding forced full client construction, which raises
'OpenAI API key is required' even though tiktoken needs no credentials.
The document dedup tie-break (src/crud/document.py) only needs .encoding
for token counting, so any test hitting that path fails in environments
without embedding keys — notably CI for pull requests from forks, where
repo secrets are unavailable (e.g. #908's test-python job failing on
tests/crud/test_document.py::test_duplicate_rejection_reinforces_existing).

Resolve the encoding from the configured model directly, falling back to
cl100k_base, and only reuse the underlying client's encoding when it has
already been constructed.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
adavyas 2026-08-05 12:28:39 -07:00 committed by GitHub
parent e6d4d78ba1
commit ca32f50797
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 13 additions and 2 deletions

View File

@ -677,8 +677,19 @@ class EmbeddingClient:
@property
def encoding(self) -> tiktoken.Encoding:
"""Get the tiktoken encoding."""
return self._get_client().encoding
"""Get the tiktoken encoding.
Resolved without constructing the underlying client: tiktoken needs no
API key, and token-counting callers (e.g. the document dedup tie-break)
must work in environments with no embedding credentials, such as CI for
pull requests from forks.
"""
if self._instance is not None:
return self._instance.encoding
try:
return tiktoken.encoding_for_model(self._resolve_runtime_config().model)
except KeyError:
return tiktoken.get_encoding("cl100k_base")
# Shared singleton embedding client instance