fix(embedding): run warmup in a worker thread to avoid blocking the event loop

warmup() can download an hf: tokenizer, which is blocking network I/O. Both call sites live in async functions (the API lifespan and run_deriver), so a slow download would stall the event loop during startup, delaying telemetry tasks and signal handling. Wrap both calls in asyncio.to_thread; warmup itself synchronizes via the singleton lock, so running it in a worker thread is safe.
This commit is contained in:
Vansh-Sharma27 2026-07-26 19:27:54 +05:30
parent 8c0709eec6
commit 07cc9e1886
No known key found for this signature in database
2 changed files with 8 additions and 3 deletions

View File

@ -73,9 +73,11 @@ async def run_deriver():
await validate_embedding_schema(engine)
# Eagerly build the embedding client so an hf: tokenizer download
# happens here, not on the first reconciler call under the lock.
# Run it in a worker thread so the blocking download doesn't stall
# the event loop.
from src.embedding_client import embedding_client
embedding_client.warmup()
await asyncio.to_thread(embedding_client.warmup)
await main()
finally:
# Shutdown telemetry (flush CloudEvents buffer)

View File

@ -1,3 +1,4 @@
import asyncio
import logging
import re
import time
@ -118,8 +119,10 @@ async def lifespan(_: FastAPI):
await validate_embedding_schema(engine)
# Eagerly build the embedding client so an hf: tokenizer download happens
# here, not on the first request under the singleton lock.
embedding_client.warmup()
# here, not on the first request under the singleton lock. Run it in a
# worker thread: the download is blocking network I/O and would otherwise
# stall the event loop (signal handling, telemetry) during startup.
await asyncio.to_thread(embedding_client.warmup)
try:
await init_cache()