e83ab842c1
The embedding client resolved every model's tokenizer through tiktoken, silently falling back to cl100k_base for models tiktoken doesn't know (e.g. baai/bge-m3). cl100k_base undercounts vs the model's real tokenizer on technical/mixed text (runtime-measured +44% for bge-m3), so prepare_chunks emits "within-limit" chunks the provider then rejects with HTTP 400. The reconciler retries the unchanged payload 20 times over ~3h, marks MessageEmbedding.sync_state='failed', and the message is permanently excluded from vector search (search.py filters embedding IS NOT NULL). Add EMBEDDING_MODEL_CONFIG__TOKENIZER: unset keeps tiktoken auto-detection (backwards compatible); tiktoken:<encoding>, hf:<repo>, or file:<path> select an explicit tokenizer. HF/file tokenizers use the optional honcho[tokenizers] extra. The HuggingFace adapter encodes without special tokens and reserves the special-token overhead ([CLS]/[SEP]) from the chunk budget so provider-side counts stay exactly within limit. Unknown models now log a warning pointing at the new setting. Invalid specs raise ValidationException (repo-standard). The singleton rebuild signature includes tokenizer so runtime config changes take effect. Runtime-verified end-to-end without a live provider: 24,360 chars of technical text with bge-m3 went from 1 chunk (8,355 real tokens > 8,192 -> provider 400 -> failed) to 2 chunks (8,192 / 1,803, both within limit). Out of scope (noted for follow-up): recovery/reindex of existing failed rows, scripts/generate_message_embeddings.py chunk-identity bug, ConclusionCreate o200k_base validator alignment, typed dimension-vs-token-limit exceptions, live-embedding CI matrix. Fixes #827 |
||
|---|---|---|
| .. | ||
| api-reference | ||
| contributing | ||
| documentation | ||
| guides | ||
| migrations | ||
| README.md | ||
| openapi.json | ||
README.md
This subdirectory contains the peer-paradigm documentation for Honcho (Honcho v2.0.0 onwards).