e83ab842c1
The embedding client resolved every model's tokenizer through tiktoken, silently falling back to cl100k_base for models tiktoken doesn't know (e.g. baai/bge-m3). cl100k_base undercounts vs the model's real tokenizer on technical/mixed text (runtime-measured +44% for bge-m3), so prepare_chunks emits "within-limit" chunks the provider then rejects with HTTP 400. The reconciler retries the unchanged payload 20 times over ~3h, marks MessageEmbedding.sync_state='failed', and the message is permanently excluded from vector search (search.py filters embedding IS NOT NULL). Add EMBEDDING_MODEL_CONFIG__TOKENIZER: unset keeps tiktoken auto-detection (backwards compatible); tiktoken:<encoding>, hf:<repo>, or file:<path> select an explicit tokenizer. HF/file tokenizers use the optional honcho[tokenizers] extra. The HuggingFace adapter encodes without special tokens and reserves the special-token overhead ([CLS]/[SEP]) from the chunk budget so provider-side counts stay exactly within limit. Unknown models now log a warning pointing at the new setting. Invalid specs raise ValidationException (repo-standard). The singleton rebuild signature includes tokenizer so runtime config changes take effect. Runtime-verified end-to-end without a live provider: 24,360 chars of technical text with bge-m3 went from 1 chunk (8,355 real tokens > 8,192 -> provider 400 -> failed) to 2 chunks (8,192 / 1,803, both within limit). Out of scope (noted for follow-up): recovery/reindex of existing failed rows, scripts/generate_message_embeddings.py chunk-identity bug, ConclusionCreate o200k_base validator alignment, typed dimension-vs-token-limit exceptions, live-embedding CI matrix. Fixes #827 |
||
|---|---|---|
| .. | ||
| changelog | ||
| images | ||
| logo | ||
| snippets | ||
| v1 | ||
| v2 | ||
| v3 | ||
| README.md | ||
| bun.lock | ||
| docs.json | ||
| favicon.svg | ||
| package.json | ||
README.md
Honcho Docs
These docs are built using Next.js via mintlify.
Setting Up Honcho's Docs Locally
- Clone the repository:
git clone git@github.com:plastic-labs/honcho.git
- Navigate into the
docsfolder:
cd honcho/docs/
The docs folder contains the markdown files that make up the documentation. The majority of the files are in the pages directory. Some notable files in this folder include:
- Verify that you have Node.js and npm installed in your system. You can check by running:
node --version
npm --version
-
If not installed, download Node.js and npm from the respective official websites.
-
Once you have Node.js and npm running, proceed to install
pnpm- another package manager that helps to manage project dependencies:
npm install -g pnpm
- Install the project dependencies using pnpm:
pnpm i
- After the successful installation of the project dependencies, start the local server:
pnpm dev
Now, you should be able to view the docs on your local environment by visiting http://localhost:3000. You can explore the different markdown files and make changes as you see fit.