fix(embedding): run warmup in a worker thread to avoid blocking the event loop
warmup() can download an hf: tokenizer, which is blocking network I/O. Both call sites live in async functions (the API lifespan and run_deriver), so a slow download would stall the event loop during startup, delaying telemetry tasks and signal handling. Wrap both calls in asyncio.to_thread; warmup itself synchronizes via the singleton lock, so running it in a worker thread is safe.
This commit is contained in:
parent
8c0709eec6
commit
07cc9e1886
|
|
@ -73,9 +73,11 @@ async def run_deriver():
|
|||
await validate_embedding_schema(engine)
|
||||
# Eagerly build the embedding client so an hf: tokenizer download
|
||||
# happens here, not on the first reconciler call under the lock.
|
||||
# Run it in a worker thread so the blocking download doesn't stall
|
||||
# the event loop.
|
||||
from src.embedding_client import embedding_client
|
||||
|
||||
embedding_client.warmup()
|
||||
await asyncio.to_thread(embedding_client.warmup)
|
||||
await main()
|
||||
finally:
|
||||
# Shutdown telemetry (flush CloudEvents buffer)
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
|
|
@ -118,8 +119,10 @@ async def lifespan(_: FastAPI):
|
|||
await validate_embedding_schema(engine)
|
||||
|
||||
# Eagerly build the embedding client so an hf: tokenizer download happens
|
||||
# here, not on the first request under the singleton lock.
|
||||
embedding_client.warmup()
|
||||
# here, not on the first request under the singleton lock. Run it in a
|
||||
# worker thread: the download is blocking network I/O and would otherwise
|
||||
# stall the event loop (signal handling, telemetry) during startup.
|
||||
await asyncio.to_thread(embedding_client.warmup)
|
||||
|
||||
try:
|
||||
await init_cache()
|
||||
|
|
|
|||
Loading…
Reference in New Issue