mirror of https://github.com/aliasrobotics/cai.git
4546033b8e
Adds cai.util.nim_rotation which cycles NVIDIA_NIM_API_KEY_1..N per request, keeping under NIM per-key rate limits (~40 req/min). Rotation applies in the OpenAI-compatible LLM path and the direct httpx completion path when the API base is a NIM endpoint (api.nvidia.com). Also defaults cache_write_tokens to 0 in the streamed usage object: litellm CustomResponseUsage now requires the field while NIM usage payloads omit it, which previously aborted streaming at end of response. |
||
|---|---|---|
| .. | ||
| test_cyber_prompt_layering.py | ||
| test_gateway_rate_limiter.py | ||
| test_nim_rotation.py | ||
| test_session_compact.py | ||
| test_wait_hints_compact.py | ||
| test_worker_silence.py | ||