The embedding client resolved every model's tokenizer through tiktoken,
silently falling back to cl100k_base for models tiktoken doesn't know
(e.g. baai/bge-m3). cl100k_base undercounts vs the model's real
tokenizer on technical/mixed text (runtime-measured +44% for bge-m3),
so prepare_chunks emits "within-limit" chunks the provider then rejects
with HTTP 400. The reconciler retries the unchanged payload 20 times
over ~3h, marks MessageEmbedding.sync_state='failed', and the message
is permanently excluded from vector search (search.py filters
embedding IS NOT NULL).
Add EMBEDDING_MODEL_CONFIG__TOKENIZER: unset keeps tiktoken
auto-detection (backwards compatible); tiktoken:<encoding>, hf:<repo>,
or file:<path> select an explicit tokenizer. HF/file tokenizers use the
optional honcho[tokenizers] extra. The HuggingFace adapter encodes
without special tokens and reserves the special-token overhead
([CLS]/[SEP]) from the chunk budget so provider-side counts stay
exactly within limit. Unknown models now log a warning pointing at the
new setting. Invalid specs raise ValidationException (repo-standard).
The singleton rebuild signature includes tokenizer so runtime config
changes take effect.
Runtime-verified end-to-end without a live provider: 24,360 chars of
technical text with bge-m3 went from 1 chunk (8,355 real tokens > 8,192
-> provider 400 -> failed) to 2 chunks (8,192 / 1,803, both within
limit).
Out of scope (noted for follow-up): recovery/reindex of existing failed
rows, scripts/generate_message_embeddings.py chunk-identity bug,
ConclusionCreate o200k_base validator alignment, typed
dimension-vs-token-limit exceptions, live-embedding CI matrix.
Fixes#827
e83ab842c1