RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add vLLM backend for soup serve (Phase 9) — v0.8.0 

- Add --backend vllm flag to soup serve for 2-4x better inference throughput
- Add --tensor-parallel and --gpu-memory flags for vLLM tuning
- Auto-detect vLLM and show hint when installed but not enabled
- New utils/vllm.py with engine creation, app factory, LoRA support
- Native token-by-token streaming via vLLM AsyncLLMEngine
- Add serve-fast extra: pip install 'soup-cli[serve-fast]'
- Add vllm detection to version --full
- 30 new tests (560 total), ruff clean

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys 2026-03-23 21:30:08 +05:00
parent 2a2c0997ea
commit 14f619cc00
8 changed files with 1124 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
CLAUDE.md
View File

@ -67,7 +67,9 @@ soup train --config soup.yaml
**W&B integration:** `commands/train.py` supports `--wandb` flag to enable Weights & Biases logging. Sets `report_to="wandb"` in TrainingArguments. Requires `pip install wandb`.
**Serve:** `commands/serve.py` starts a local inference server with OpenAI-compatible API (`/v1/chat/completions`, `/v1/models`, `/health`). Uses FastAPI + uvicorn. Supports LoRA adapters and full models, SSE streaming. Requires `pip install 'soup-cli[serve]'`.
**Serve:** `commands/serve.py` starts a local inference server with OpenAI-compatible API (`/v1/chat/completions`, `/v1/models`, `/health`). Supports two backends: `transformers` (default, FastAPI + uvicorn) and `vllm` (2-4x better throughput). Supports LoRA adapters and full models, SSE streaming. `--backend vllm` enables vLLM's `AsyncLLMEngine` with tensor parallelism (`--tensor-parallel`) and GPU memory control (`--gpu-memory`). Auto-detects vLLM if installed and shows a hint. Requires `pip install 'soup-cli[serve]'` or `pip install 'soup-cli[serve-fast]'` for vLLM.
**vLLM backend:** `utils/vllm.py` provides `is_vllm_available()`, `get_vllm_version()`, `create_vllm_engine()`, and `create_vllm_app()`. The engine uses `AsyncLLMEngine` with `AsyncEngineArgs` for async inference. For LoRA adapters, loads base model with `enable_lora=True` and applies adapter via `LoRARequest` at request time. Streaming uses vLLM's native token-by-token generation (not simulated). Config: `--backend vllm` (default: `transformers`). Install: `pip install 'soup-cli[serve-fast]'`.
**Data generate:** `commands/generate.py` generates synthetic training data using LLMs. Supports OpenAI API and local models as providers. Outputs in alpaca/sharegpt/chatml format. Validates on the fly and can deduplicate against existing datasets. Registered as `soup data generate`.
@ -174,3 +176,4 @@ Test suite lives in `tests/`:
| `test_vision.py` | Vision modality config, LLaVA/ShareGPT4V formats, loader, trainer, templates |
| `test_qat.py` | QAT config, validation, trainer integration, export compatibility |
| `test_ui.py` | Web UI command, FastAPI endpoints, static files, config validation |
| `test_vllm_serve.py` | vLLM backend detection, engine creation, serve --backend flag, FastAPI app |

22
README.md
View File

@ -428,6 +428,26 @@ response = client.chat.completions.create(
)
```
### vLLM Backend (2-4x Faster Inference)
Use [vLLM](https://github.com/vllm-project/vllm) for significantly better throughput in production:
```bash
# Install vLLM support
pip install 'soup-cli[serve-fast]'
# Start with vLLM backend
soup serve --model ./output --backend vllm
# Multi-GPU with tensor parallelism
soup serve --model ./output --backend vllm --tensor-parallel 2
# Control GPU memory usage
soup serve --model ./output --backend vllm --gpu-memory 0.8
```
> **Tip:** Soup auto-detects vLLM. When installed, you'll see a hint during `soup serve` if you haven't enabled it yet.
## Synthetic Data Generation
Generate training data using LLMs:
@ -668,6 +688,7 @@ soup merge --adapter ./output Merge LoRA with base model
soup export --model ./output --format gguf Export to GGUF (Ollama)
soup eval --model ./output --benchmarks mmlu Evaluate on benchmarks
soup serve --model ./output --port 8000 OpenAI-compatible API server
soup serve --model ./output --backend vllm vLLM backend (2-4x throughput)
soup sweep --config soup.yaml --param lr=... Hyperparameter search
soup diff --model-a ./a --model-b ./b Compare two models
soup data inspect <path> View dataset stats
@ -702,6 +723,7 @@ soup --verbose <command> Full traceback on errors
| `fast` | `pip install 'soup-cli[fast]'` | Unsloth backend (2-5x faster, -80% VRAM) |
| `ui` | `pip install 'soup-cli[ui]'` | Web UI + inference server (FastAPI + uvicorn) |
| `serve` | `pip install 'soup-cli[serve]'` | Inference server (FastAPI + uvicorn) |
| `serve-fast` | `pip install 'soup-cli[serve-fast]'` | vLLM inference backend (2-4x throughput) |
| `data` | `pip install 'soup-cli[data]'` | Deduplication (MinHash via datasketch) |
| `eval` | `pip install 'soup-cli[eval]'` | Benchmark evaluation (lm-evaluation-harness) |
| `deepspeed` | `pip install 'soup-cli[deepspeed]'` | Multi-GPU training (DeepSpeed ZeRO) |

3
pyproject.toml
View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "soup-cli"
version = "0.7.3"
version = "0.8.0"
description = "Fine-tune LLMs in one command. No SSH, no config hell."
readme = "README.md"
license = "MIT"
@ -44,6 +44,7 @@ wandb = ["wandb>=0.15.0"]
dev = ["pytest>=7.0", "ruff>=0.1.0", "pytest-cov>=4.0", "httpx>=0.24.0"]
ui = ["fastapi>=0.104.0", "uvicorn>=0.24.0"]
serve = ["fastapi>=0.104.0", "uvicorn>=0.24.0"]
serve-fast = ["vllm>=0.4.0", "fastapi>=0.104.0", "uvicorn>=0.24.0"]
generate = ["httpx>=0.24.0"]
deepspeed = ["deepspeed>=0.12.0"]
fast = ["unsloth>=2024.8"]

2
soup_cli/__init__.py
View File

@ -1,3 +1,3 @@
"""Soup CLI — Fine-tune LLMs in one command."""
__version__ = "0.7.3"
__version__ = "0.8.0"

1
soup_cli/cli.py
View File

@ -91,6 +91,7 @@ def version(
extras = []
for name, label in [
("fastapi", "serve"),
("vllm", "serve-fast"),
("datasketch", "data"),
("lm_eval", "eval"),
("deepspeed", "deepspeed"),

141
soup_cli/commands/serve.py
View File

@ -47,6 +47,22 @@ def serve(
"--max-tokens",
help="Default max tokens for generation",
),
backend: str = typer.Option(
"transformers",
"--backend",
help="Inference backend: transformers (default) or vllm",
),
tensor_parallel: int = typer.Option(
1,
"--tensor-parallel",
"--tp",
help="Number of GPUs for tensor parallelism (vLLM only)",
),
gpu_memory_utilization: float = typer.Option(
0.9,
"--gpu-memory",
help="Fraction of GPU memory to use (vLLM only, 0.0-1.0)",
),
):
"""Start a local inference server with OpenAI-compatible API."""
# Lazy imports for fast CLI startup
@ -57,10 +73,40 @@ def serve(
except ImportError:
console.print(
"[red]FastAPI/uvicorn not installed.[/]\n"
"Install with: [bold]pip install 'soup-cli[ui]'[/]"
"Install with: [bold]pip install 'soup-cli[serve]'[/]"
)
raise typer.Exit(1)
# Validate backend
backend = backend.lower()
if backend not in ("transformers", "vllm"):
console.print(
f"[red]Unknown backend: {backend}[/]\n"
"Supported backends: [bold]transformers[/], [bold]vllm[/]"
)
raise typer.Exit(1)
# Auto-detect vLLM: if installed but not selected, show hint
if backend == "transformers":
from soup_cli.utils.vllm import is_vllm_available
if is_vllm_available():
console.print(
"[dim]Hint: vLLM is installed. Use [bold]--backend vllm[/] "
"for 2-4x better throughput.[/]"
)
# Validate vLLM availability
if backend == "vllm":
from soup_cli.utils.vllm import is_vllm_available
if not is_vllm_available():
console.print(
"[red]vLLM not installed.[/]\n"
"Install with: [bold]pip install 'soup-cli[serve-fast]'[/]"
)
raise typer.Exit(1)
model_path = Path(model)
if not model_path.exists():
console.print(f"[red]Model path not found: {model_path}[/]")
@ -80,43 +126,58 @@ def serve(
)
raise typer.Exit(1)
# Detect device
if not device:
# Detect device (only for transformers backend)
if not device and backend == "transformers":
from soup_cli.utils.gpu import detect_device
device, _ = detect_device()
elif not device:
device = "cuda"
backend_label = "vLLM" if backend == "vllm" else "transformers"
console.print(
Panel(
f"Model: [bold]{model_path}[/]\n"
+ (f"Base: [bold]{base_model}[/]\n" if is_adapter else "")
+ f"Device: [bold]{device}[/]\n"
f"Type: [bold]{'LoRA adapter' if is_adapter else 'Full model'}[/]",
f"Model: [bold]{model_path}[/]\n"
+ (f"Base: [bold]{base_model}[/]\n" if is_adapter else "")
+ f"Device: [bold]{device}[/]\n"
f"Type: [bold]{'LoRA adapter' if is_adapter else 'Full model'}[/]\n"
f"Backend: [bold]{backend_label}[/]"
+ (f"\nTP: [bold]{tensor_parallel}[/]" if backend == "vllm" else ""),
title="Loading model",
)
)
# Load model
model_obj, tokenizer = _load_model(
model_path=str(model_path),
base_model=base_model,
is_adapter=is_adapter,
device=device,
)
console.print("[bold green]Model loaded![/]")
if backend == "vllm":
app = _serve_vllm(
model_path=model_path,
base_model=base_model,
is_adapter=is_adapter,
max_tokens_default=max_tokens_default,
tensor_parallel=tensor_parallel,
gpu_memory_utilization=gpu_memory_utilization,
)
else:
# Transformers backend (original)
model_obj, tokenizer = _load_model(
model_path=str(model_path),
base_model=base_model,
is_adapter=is_adapter,
device=device,
)
console.print("[bold green]Model loaded![/]")
# Build and start FastAPI app
app = _create_app(
model_obj=model_obj,
tokenizer=tokenizer,
device=device,
model_name=str(model_path.name),
max_tokens_default=max_tokens_default,
)
app = _create_app(
model_obj=model_obj,
tokenizer=tokenizer,
device=device,
model_name=str(model_path.name),
max_tokens_default=max_tokens_default,
)
console.print(
Panel(
f"URL: [bold]http://{host}:{port}[/]\n"
f"Backend: [bold]{backend_label}[/]\n"
f"Endpoints: [bold]/v1/chat/completions[/], [bold]/v1/models[/], [bold]/health[/]\n\n"
f"Example:\n"
f" curl http://localhost:{port}/v1/chat/completions \\\n"
@ -135,6 +196,40 @@ def serve(
uvicorn.run(app, host=host, port=port, log_level="warning")
def _serve_vllm(
model_path: Path,
base_model: Optional[str],
is_adapter: bool,
max_tokens_default: int,
tensor_parallel: int,
gpu_memory_utilization: float,
):
"""Set up vLLM engine and create FastAPI app."""
from soup_cli.utils.vllm import create_vllm_app, create_vllm_engine
console.print("[dim]Initializing vLLM engine...[/]")
engine, engine_model_name = create_vllm_engine(
model_path=str(model_path),
base_model=base_model,
is_adapter=is_adapter,
tensor_parallel_size=tensor_parallel,
gpu_memory_utilization=gpu_memory_utilization,
)
console.print("[bold green]vLLM engine ready![/]")
adapter_path = str(model_path) if is_adapter else None
app = create_vllm_app(
engine=engine,
engine_model_name=engine_model_name,
model_name=str(model_path.name),
adapter_path=adapter_path,
max_tokens_default=max_tokens_default,
)
return app
def _detect_base_model(adapter_config_path: Path) -> Optional[str]:
"""Read base_model_name_or_path from adapter_config.json."""
try:

297
soup_cli/utils/vllm.py Normal file
View File

@ -0,0 +1,297 @@
"""vLLM backend utilities for soup serve."""
def is_vllm_available() -> bool:
"""Check if vLLM is installed."""
try:
import vllm # noqa: F401
return True
except ImportError:
return False
def get_vllm_version() -> str:
"""Get installed vLLM version."""
try:
import vllm
return getattr(vllm, "__version__", "unknown")
except ImportError:
return "not installed"
def create_vllm_engine(
model_path: str,
base_model: str | None = None,
is_adapter: bool = False,
tensor_parallel_size: int = 1,
gpu_memory_utilization: float = 0.9,
max_model_len: int | None = None,
dtype: str = "auto",
):
"""Create a vLLM AsyncLLMEngine for serving.
Args:
model_path: Path to model or LoRA adapter directory.
base_model: Base model ID (required if model_path is a LoRA adapter).
is_adapter: Whether model_path is a LoRA adapter.
tensor_parallel_size: Number of GPUs for tensor parallelism.
gpu_memory_utilization: Fraction of GPU memory to use.
max_model_len: Maximum sequence length. Auto-detected if None.
dtype: Data type for model weights.
Returns:
(engine, engine_model_name) tuple.
"""
from vllm import AsyncEngineArgs, AsyncLLMEngine
# For LoRA adapters, load the base model and apply LoRA at request time
if is_adapter and base_model:
engine_args = AsyncEngineArgs(
model=base_model,
enable_lora=True,
max_lora_rank=128,
tensor_parallel_size=tensor_parallel_size,
gpu_memory_utilization=gpu_memory_utilization,
dtype=dtype,
trust_remote_code=True,
)
if max_model_len is not None:
engine_args.max_model_len = max_model_len
engine_model_name = base_model
else:
engine_args = AsyncEngineArgs(
model=model_path,
tensor_parallel_size=tensor_parallel_size,
gpu_memory_utilization=gpu_memory_utilization,
dtype=dtype,
trust_remote_code=True,
)
if max_model_len is not None:
engine_args.max_model_len = max_model_len
engine_model_name = model_path
engine = AsyncLLMEngine.from_engine_args(engine_args)
return engine, engine_model_name
def create_vllm_app(
engine,
engine_model_name: str,
model_name: str,
adapter_path: str | None = None,
max_tokens_default: int = 512,
):
"""Create a FastAPI app using vLLM engine for inference.
Args:
engine: vLLM AsyncLLMEngine instance.
engine_model_name: Model name used by vLLM engine.
model_name: Display model name for API responses.
adapter_path: Path to LoRA adapter (if using adapter).
max_tokens_default: Default max tokens for generation.
Returns:
FastAPI application.
"""
import json
import time
import uuid
from typing import Optional
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import StreamingResponse
from pydantic import BaseModel as PydanticBaseModel
from pydantic import Field
from vllm import SamplingParams
app = FastAPI(title="Soup Inference Server (vLLM)", version="1.0.0")
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)
class ChatMessage(PydanticBaseModel):
role: str
content: str
class ChatCompletionRequest(PydanticBaseModel):
model: str = model_name
messages: list[ChatMessage]
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
top_p: float = Field(default=0.9, ge=0.0, le=1.0)
max_tokens: Optional[int] = None
stream: bool = False
@app.get("/health")
def health():
return {"status": "ok", "model": model_name, "backend": "vllm"}
@app.get("/v1/models")
def list_models():
return {
"object": "list",
"data": [
{
"id": model_name,
"object": "model",
"owned_by": "soup",
}
],
}
@app.post("/v1/chat/completions")
async def chat_completions(request: ChatCompletionRequest):
max_tokens = request.max_tokens or max_tokens_default
request_id = f"chatcmpl-{uuid.uuid4().hex[:8]}"
# Build prompt from messages using a simple chat template
prompt = _build_prompt(request.messages)
sampling_params = SamplingParams(
temperature=request.temperature,
top_p=request.top_p,
max_tokens=max_tokens,
)
# Build generate kwargs
generate_kwargs = {}
if adapter_path:
from vllm.lora.request import LoRARequest
generate_kwargs["lora_request"] = LoRARequest(
lora_name="adapter",
lora_int_id=1,
lora_path=adapter_path,
)
if request.stream:
return StreamingResponse(
_stream_vllm_response(
engine=engine,
prompt=prompt,
sampling_params=sampling_params,
request_id=request_id,
model_name=model_name,
generate_kwargs=generate_kwargs,
),
media_type="text/event-stream",
)
# Non-streaming
try:
results_generator = engine.generate(
prompt, sampling_params, request_id, **generate_kwargs
)
final_output = None
async for request_output in results_generator:
final_output = request_output
if final_output is None:
raise HTTPException(status_code=500, detail="No output generated")
output = final_output.outputs[0]
response_text = output.text
prompt_tokens = len(final_output.prompt_token_ids)
completion_tokens = len(output.token_ids)
return {
"id": request_id,
"object": "chat.completion",
"created": int(time.time()),
"model": model_name,
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": response_text,
},
"finish_reason": "stop",
}
],
"usage": {
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"total_tokens": prompt_tokens + completion_tokens,
},
}
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc))
def _build_prompt(messages: list[ChatMessage]) -> str:
"""Build a simple prompt from chat messages."""
parts = []
for msg in messages:
role = msg.role
content = msg.content
if role == "system":
parts.append(f"System: {content}")
elif role == "user":
parts.append(f"User: {content}")
elif role == "assistant":
parts.append(f"Assistant: {content}")
parts.append("Assistant:")
return "\n".join(parts)
async def _stream_vllm_response(
engine,
prompt: str,
sampling_params,
request_id: str,
model_name: str,
generate_kwargs: dict,
):
"""Stream SSE chunks from vLLM engine."""
created = int(time.time())
previous_text = ""
results_generator = engine.generate(
prompt, sampling_params, request_id, **generate_kwargs
)
async for request_output in results_generator:
output = request_output.outputs[0]
new_text = output.text[len(previous_text):]
previous_text = output.text
if new_text:
chunk = {
"id": request_id,
"object": "chat.completion.chunk",
"created": created,
"model": model_name,
"choices": [
{
"index": 0,
"delta": {"content": new_text},
"finish_reason": None,
}
],
}
yield f"data: {json.dumps(chunk)}\n\n"
# Final chunk
final_chunk = {
"id": request_id,
"object": "chat.completion.chunk",
"created": created,
"model": model_name,
"choices": [
{
"index": 0,
"delta": {},
"finish_reason": "stop",
}
],
}
yield f"data: {json.dumps(final_chunk)}\n\n"
yield "data: [DONE]\n\n"
return app

679
tests/test_vllm_serve.py Normal file
View File

@ -0,0 +1,679 @@
"""Tests for vLLM backend in soup serve."""
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
# ============================================================
# vLLM utility tests
# ============================================================
class TestVllmDetection:
"""Test vLLM availability detection."""
def test_is_vllm_available_when_installed(self):
"""Should return True when vllm is importable."""
from soup_cli.utils.vllm import is_vllm_available
mock_vllm = MagicMock()
with patch.dict("sys.modules", {"vllm": mock_vllm}):
assert is_vllm_available() is True
def test_is_vllm_available_when_not_installed(self):
"""Should return False when vllm is not importable."""
from soup_cli.utils.vllm import is_vllm_available
with patch.dict("sys.modules", {"vllm": None}):
assert is_vllm_available() is False
def test_get_vllm_version_installed(self):
"""Should return version string when installed."""
from soup_cli.utils.vllm import get_vllm_version
mock_vllm = MagicMock()
mock_vllm.__version__ = "0.5.0"
with patch.dict("sys.modules", {"vllm": mock_vllm}):
assert get_vllm_version() == "0.5.0"
def test_get_vllm_version_not_installed(self):
"""Should return 'not installed' when vllm is missing."""
from soup_cli.utils.vllm import get_vllm_version
with patch.dict("sys.modules", {"vllm": None}):
assert get_vllm_version() == "not installed"
def test_get_vllm_version_no_attr(self):
"""Should return 'unknown' if __version__ not set."""
from soup_cli.utils.vllm import get_vllm_version
mock_vllm = MagicMock(spec=[]) # no __version__ attr
with patch.dict("sys.modules", {"vllm": mock_vllm}):
assert get_vllm_version() == "unknown"
# ============================================================
# Serve command backend flag tests
# ============================================================
class TestServeBackendFlag:
"""Test --backend flag in serve command."""
def test_invalid_backend_rejected(self, tmp_path):
"""serve --backend invalid should fail."""
from typer.testing import CliRunner
from soup_cli.cli import app
runner = CliRunner()
# Create a fake model dir
model_dir = tmp_path / "model"
model_dir.mkdir()
result = runner.invoke(
app, ["serve", "--model", str(model_dir), "--backend", "invalid"]
)
assert result.exit_code != 0
assert "unknown backend" in result.output.lower() or result.exit_code != 0
def test_vllm_backend_not_installed(self, tmp_path):
"""serve --backend vllm should fail if vllm not installed."""
from typer.testing import CliRunner
from soup_cli.cli import app
runner = CliRunner()
model_dir = tmp_path / "model"
model_dir.mkdir()
with patch("soup_cli.utils.vllm.is_vllm_available", return_value=False):
result = runner.invoke(
app, ["serve", "--model", str(model_dir), "--backend", "vllm"]
)
assert result.exit_code != 0
def test_vllm_hint_shown_when_available(self, tmp_path):
"""When vllm is installed but not selected, show hint."""
from io import StringIO
from rich.console import Console
from soup_cli.commands import serve as serve_mod
# Save original console
original_console = serve_mod.console
output = StringIO()
serve_mod.console = Console(file=output)
model_dir = tmp_path / "model"
model_dir.mkdir()
try:
with patch("soup_cli.utils.vllm.is_vllm_available", return_value=True):
# Will fail at model loading but we can check for hint output
try:
serve_mod.serve.__wrapped__(
model=str(model_dir),
base_model=None,
port=8000,
host="0.0.0.0",
device="cpu",
max_tokens_default=512,
backend="transformers",
tensor_parallel=1,
gpu_memory_utilization=0.9,
)
except (SystemExit, Exception):
pass
finally:
serve_mod.console = original_console
# Hint may or may not appear depending on how far execution gets
# The key test is that no crash happens
assert output.getvalue() is not None
def test_default_backend_is_transformers(self):
"""Default backend should be transformers."""
import inspect
from soup_cli.commands.serve import serve
sig = inspect.signature(serve)
backend_param = sig.parameters.get("backend")
assert backend_param is not None
assert backend_param.default.default == "transformers"
def test_backend_accepts_vllm(self):
"""Backend param should accept 'vllm' as a value."""
import inspect
from soup_cli.commands.serve import serve
sig = inspect.signature(serve)
backend_param = sig.parameters.get("backend")
assert backend_param is not None
# Just check the param exists and has help text mentioning vllm
assert "vllm" in backend_param.default.help.lower()
class TestServeTensorParallel:
"""Test --tensor-parallel flag."""
def test_tensor_parallel_param_exists(self):
"""serve should have --tensor-parallel param."""
import inspect
from soup_cli.commands.serve import serve
sig = inspect.signature(serve)
assert "tensor_parallel" in sig.parameters
def test_tensor_parallel_default_is_one(self):
"""Default tensor parallel size should be 1."""
import inspect
from soup_cli.commands.serve import serve
sig = inspect.signature(serve)
tp_param = sig.parameters["tensor_parallel"]
assert tp_param.default.default == 1
def test_gpu_memory_param_exists(self):
"""serve should have --gpu-memory param."""
import inspect
from soup_cli.commands.serve import serve
sig = inspect.signature(serve)
assert "gpu_memory_utilization" in sig.parameters
def test_gpu_memory_default(self):
"""Default GPU memory utilization should be 0.9."""
import inspect
from soup_cli.commands.serve import serve
sig = inspect.signature(serve)
param = sig.parameters["gpu_memory_utilization"]
assert param.default.default == 0.9
# ============================================================
# vLLM engine creation tests
# ============================================================
class TestCreateVllmEngine:
"""Test vLLM engine creation logic."""
def test_create_engine_full_model(self):
"""create_vllm_engine with full model should use model_path directly."""
mock_engine = MagicMock()
mock_args_cls = MagicMock()
mock_engine_cls = MagicMock()
mock_engine_cls.from_engine_args.return_value = mock_engine
with patch.dict("sys.modules", {
"vllm": MagicMock(
AsyncEngineArgs=mock_args_cls,
AsyncLLMEngine=mock_engine_cls,
),
}):
from importlib import reload
import soup_cli.utils.vllm as vllm_mod
reload(vllm_mod)
engine, name = vllm_mod.create_vllm_engine(
model_path="/path/to/model",
is_adapter=False,
tensor_parallel_size=2,
gpu_memory_utilization=0.8,
)
assert engine == mock_engine
assert name == "/path/to/model"
mock_args_cls.assert_called_once()
call_kwargs = mock_args_cls.call_args
assert call_kwargs.kwargs["model"] == "/path/to/model"
assert call_kwargs.kwargs["tensor_parallel_size"] == 2
assert call_kwargs.kwargs["gpu_memory_utilization"] == 0.8
def test_create_engine_adapter(self):
"""create_vllm_engine with adapter should use base_model and enable LoRA."""
mock_engine = MagicMock()
mock_args_cls = MagicMock()
mock_engine_cls = MagicMock()
mock_engine_cls.from_engine_args.return_value = mock_engine
with patch.dict("sys.modules", {
"vllm": MagicMock(
AsyncEngineArgs=mock_args_cls,
AsyncLLMEngine=mock_engine_cls,
),
}):
from importlib import reload
import soup_cli.utils.vllm as vllm_mod
reload(vllm_mod)
engine, name = vllm_mod.create_vllm_engine(
model_path="/path/to/adapter",
base_model="meta-llama/Llama-3.1-8B",
is_adapter=True,
)
assert engine == mock_engine
assert name == "meta-llama/Llama-3.1-8B"
call_kwargs = mock_args_cls.call_args
assert call_kwargs.kwargs["model"] == "meta-llama/Llama-3.1-8B"
assert call_kwargs.kwargs["enable_lora"] is True
def test_create_engine_max_model_len(self):
"""create_vllm_engine should set max_model_len when provided."""
mock_engine = MagicMock()
mock_args_cls = MagicMock()
mock_engine_cls = MagicMock()
mock_engine_cls.from_engine_args.return_value = mock_engine
with patch.dict("sys.modules", {
"vllm": MagicMock(
AsyncEngineArgs=mock_args_cls,
AsyncLLMEngine=mock_engine_cls,
),
}):
from importlib import reload
import soup_cli.utils.vllm as vllm_mod
reload(vllm_mod)
engine, name = vllm_mod.create_vllm_engine(
model_path="/path/to/model",
is_adapter=False,
max_model_len=4096,
)
args_instance = mock_args_cls.return_value
assert args_instance.max_model_len == 4096
# ============================================================
# vLLM app creation tests
# ============================================================
class TestCreateVllmApp:
"""Test vLLM FastAPI app creation."""
def _make_mock_engine(self):
"""Create a mock vLLM engine."""
engine = MagicMock()
return engine
def test_create_vllm_app_returns_fastapi(self):
"""create_vllm_app should return a FastAPI app."""
try:
from fastapi import FastAPI
except ImportError:
pytest.skip("FastAPI not installed")
mock_vllm = MagicMock()
mock_vllm.SamplingParams = MagicMock()
with patch.dict("sys.modules", {
"vllm": mock_vllm,
"vllm.lora": MagicMock(),
"vllm.lora.request": MagicMock(),
}):
from importlib import reload
import soup_cli.utils.vllm as vllm_mod
reload(vllm_mod)
engine = self._make_mock_engine()
app = vllm_mod.create_vllm_app(
engine=engine,
engine_model_name="test-model",
model_name="test-model",
max_tokens_default=256,
)
assert isinstance(app, FastAPI)
def test_vllm_app_has_correct_routes(self):
"""vLLM app should have health, models, and chat completions routes."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
mock_vllm = MagicMock()
mock_vllm.SamplingParams = MagicMock()
with patch.dict("sys.modules", {
"vllm": mock_vllm,
"vllm.lora": MagicMock(),
"vllm.lora.request": MagicMock(),
}):
from importlib import reload
import soup_cli.utils.vllm as vllm_mod
reload(vllm_mod)
engine = self._make_mock_engine()
app = vllm_mod.create_vllm_app(
engine=engine,
engine_model_name="test-model",
model_name="test-model",
max_tokens_default=256,
)
routes = [route.path for route in app.routes]
assert "/health" in routes
assert "/v1/models" in routes
assert "/v1/chat/completions" in routes
def test_vllm_health_endpoint(self):
"""Health endpoint should return backend: vllm."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
mock_vllm = MagicMock()
mock_vllm.SamplingParams = MagicMock()
with patch.dict("sys.modules", {
"vllm": mock_vllm,
"vllm.lora": MagicMock(),
"vllm.lora.request": MagicMock(),
}):
from importlib import reload
import soup_cli.utils.vllm as vllm_mod
reload(vllm_mod)
engine = self._make_mock_engine()
app = vllm_mod.create_vllm_app(
engine=engine,
engine_model_name="test-model",
model_name="test-model",
max_tokens_default=256,
)
client = TestClient(app)
response = client.get("/health")
assert response.status_code == 200
data = response.json()
assert data["status"] == "ok"
assert data["model"] == "test-model"
assert data["backend"] == "vllm"
def test_vllm_models_endpoint(self):
"""Models endpoint should list the served model."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
mock_vllm = MagicMock()
mock_vllm.SamplingParams = MagicMock()
with patch.dict("sys.modules", {
"vllm": mock_vllm,
"vllm.lora": MagicMock(),
"vllm.lora.request": MagicMock(),
}):
from importlib import reload
import soup_cli.utils.vllm as vllm_mod
reload(vllm_mod)
engine = self._make_mock_engine()
app = vllm_mod.create_vllm_app(
engine=engine,
engine_model_name="my-model",
model_name="my-model",
max_tokens_default=256,
)
client = TestClient(app)
response = client.get("/v1/models")
assert response.status_code == 200
data = response.json()
assert data["object"] == "list"
assert len(data["data"]) == 1
assert data["data"][0]["id"] == "my-model"
# ============================================================
# Serve command integration with vLLM
# ============================================================
class TestServeVllmIntegration:
"""Test _serve_vllm helper."""
def test_serve_vllm_creates_app(self, tmp_path):
"""_serve_vllm should create engine and return FastAPI app."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
mock_engine = MagicMock()
mock_app = MagicMock()
with patch(
"soup_cli.utils.vllm.create_vllm_engine",
return_value=(mock_engine, "base-model"),
) as mock_create_engine, patch(
"soup_cli.utils.vllm.create_vllm_app",
return_value=mock_app,
) as mock_create_app:
from soup_cli.commands.serve import _serve_vllm
model_path = tmp_path / "model"
model_path.mkdir()
app = _serve_vllm(
model_path=model_path,
base_model="base-model",
is_adapter=False,
max_tokens_default=512,
tensor_parallel=2,
gpu_memory_utilization=0.85,
)
assert app == mock_app
mock_create_engine.assert_called_once_with(
model_path=str(model_path),
base_model="base-model",
is_adapter=False,
tensor_parallel_size=2,
gpu_memory_utilization=0.85,
)
mock_create_app.assert_called_once()
def test_serve_vllm_with_adapter(self, tmp_path):
"""_serve_vllm with adapter should pass adapter_path."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
mock_engine = MagicMock()
mock_app = MagicMock()
with patch(
"soup_cli.utils.vllm.create_vllm_engine",
return_value=(mock_engine, "base-model"),
), patch(
"soup_cli.utils.vllm.create_vllm_app",
return_value=mock_app,
) as mock_create_app:
from soup_cli.commands.serve import _serve_vllm
model_path = tmp_path / "adapter"
model_path.mkdir()
_serve_vllm(
model_path=model_path,
base_model="base-model",
is_adapter=True,
max_tokens_default=256,
tensor_parallel=1,
gpu_memory_utilization=0.9,
)
# Check adapter_path was passed
call_kwargs = mock_create_app.call_args
assert call_kwargs.kwargs["adapter_path"] == str(model_path)
# ============================================================
# Existing serve tests still pass (transformers backend)
# ============================================================
class TestTransformersBackendUnchanged:
"""Verify transformers backend still works as before."""
def test_create_app_returns_fastapi_instance(self):
"""_create_app should return a FastAPI app with correct endpoints."""
try:
from fastapi import FastAPI
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.commands.serve import _create_app
mock_model = MagicMock()
mock_tokenizer = MagicMock()
app = _create_app(
model_obj=mock_model,
tokenizer=mock_tokenizer,
device="cpu",
model_name="test-model",
max_tokens_default=256,
)
assert isinstance(app, FastAPI)
routes = [route.path for route in app.routes]
assert "/health" in routes
assert "/v1/models" in routes
assert "/v1/chat/completions" in routes
def test_health_endpoint_transformers(self):
"""Health endpoint with transformers backend returns ok."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.commands.serve import _create_app
app = _create_app(
model_obj=MagicMock(),
tokenizer=MagicMock(),
device="cpu",
model_name="test-model",
max_tokens_default=256,
)
client = TestClient(app)
response = client.get("/health")
assert response.status_code == 200
data = response.json()
assert data["status"] == "ok"
# ============================================================
# CLI registration tests
# ============================================================
class TestServeCliRegistration:
"""Test serve command is properly registered."""
def test_serve_help_shows_backend(self):
"""soup serve --help should mention --backend flag."""
from typer.testing import CliRunner
from soup_cli.cli import app
runner = CliRunner()
result = runner.invoke(app, ["serve", "--help"])
assert result.exit_code == 0
assert "--backend" in result.output
def test_serve_help_shows_tensor_parallel(self):
"""soup serve --help should mention --tensor-parallel flag."""
from typer.testing import CliRunner
from soup_cli.cli import app
runner = CliRunner()
result = runner.invoke(app, ["serve", "--help"])
assert result.exit_code == 0
assert "--tensor-parallel" in result.output
def test_serve_help_shows_gpu_memory(self):
"""soup serve --help should mention --gpu-memory flag."""
from typer.testing import CliRunner
from soup_cli.cli import app
runner = CliRunner()
result = runner.invoke(app, ["serve", "--help"])
assert result.exit_code == 0
assert "--gpu-memory" in result.output
# ============================================================
# pyproject.toml extra tests
# ============================================================
class TestServeFastExtra:
"""Test serve-fast extra is properly defined."""
def test_serve_fast_extra_in_pyproject(self):
"""pyproject.toml should have serve-fast extra with vllm."""
toml_path = Path(__file__).parent.parent / "pyproject.toml"
content = toml_path.read_text()
assert "serve-fast" in content
assert "vllm" in content
# ============================================================
# Version detection tests
# ============================================================
class TestVersionDetectsVllm:
"""Test that version --full detects vllm."""
def test_version_full_checks_vllm(self):
"""version --full extras should include serve-fast check."""
from typer.testing import CliRunner
from soup_cli.cli import app
runner = CliRunner()
# Run version --full — vllm won't be installed but it shouldn't crash
result = runner.invoke(app, ["version", "--full"])
assert result.exit_code == 0
# Should NOT show serve-fast since vllm not installed
# But command should complete successfully