RomanNum3ral
/
cai
mirror of https://github.com/aliasrobotics/cai.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

fix: remove duplicate litellm stream call in fetch_response_litellm_openai 

The streaming path called litellm.acompletion twice with the same kwargs.
The first result was assigned to a variable that is never read, and only
the second call's stream was returned. Every streamed request therefore
consumed two provider requests (e.g. two NVIDIA NIM requests) with the
same API key, halving effective per-key rate limits and leaking the
first stream. Keep a single acompletion call for the streamed path; same
fix in the tool_call_id truncation retry branch.
This commit is contained in:
UnaiAlias 2026-07-31 18:24:17 +00:00
parent 62871b6f5a
commit ffc404f10f
1 changed files with 0 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/cai/sdk/agents/models/chatcompletions/litellm_adapter.py
View File

@ -68,7 +68,6 @@ async def fetch_response_litellm_openai(
"""
try:
if stream:
ret = await litellm.acompletion(**kwargs)
stream_obj = await litellm.acompletion(**kwargs)
return _build_response_obj(model_name, model_settings, tool_choice, parallel_tool_calls), stream_obj
else:
@ -102,7 +101,6 @@ async def fetch_response_litellm_openai(
kwargs["messages"] = messages
if stream:
ret = await litellm.acompletion(**kwargs)
stream_obj = await litellm.acompletion(**kwargs)
return _build_response_obj(model_name, model_settings, tool_choice, parallel_tool_calls), stream_obj
else: