feat(ui): add Web UI Enhancement with live training monitor, enhanced metrics, chat upgrade, and config builder (v0.24.2)

Part A: Training Live Monitor — SSE log streaming (/api/train/logs with
Last-Event-ID reconnection), live metrics SSE (/api/train/metrics/live),
progress endpoint (/api/train/progress), frontend with auto-scroll log
panel, progress bar, and live indicator badge.

Part B: Enhanced Metrics & Eval Display — 2x2 chart grid (loss, LR,
grad_norm, throughput) + GPU memory chart, eval results table in run
detail modal, /api/runs/compare endpoint (max 5 runs).

Part C: Chat Upgrade — /api/chat/send SSE proxy with SSRF protection
(localhost-only HTTP, HTTPS for remote), streaming via ReadableStream,
typing indicator, cancel button, markdown renderer (bold/italic/code),
chat settings panel (temperature/max_tokens/top_p/system prompt/adapter),
chat export as JSON.

Part D: Visual Config Builder — /api/config/schema (Pydantic field
metadata extraction), /api/recipes (29 ready-made configs as JSON),
/api/config/from-form (form values to validated YAML), recipe dropdown.

Security: Chat proxy SSRF validation, max_tokens cap 16384, temperature
0-2, top_p 0-1, Bearer auth on POST, XSS prevention, compare max 5 runs.

Tests: 58 new tests across 4 files (2128 total, 78 files), 67% coverage.
This commit is contained in:
Alpamys 2026-04-07 19:13:55 +05:00
parent d134abb008
commit 54230f7bdd
13 changed files with 2363 additions and 93 deletions

View File

@ -1,12 +1,12 @@
# Soup CLI — Project CLAUDE.md
Soup is a CLI-first LLM fine-tuning tool (v0.24.1). Python 3.9+, MIT license.
Soup is a CLI-first LLM fine-tuning tool (v0.24.2). Python 3.9+, MIT license.
## Build & Development
```bash
pip install -e ".[dev]" # Install editable + test deps
pytest tests/ -v --tb=short # Run all tests (2061 tests)
pytest tests/ -v --tb=short # Run all tests (2128 tests)
ruff check soup_cli/ tests/ # Lint (must pass before commit)
ruff check --fix soup_cli/ tests/ # Auto-fix lint issues
```
@ -115,7 +115,7 @@ soup_cli/
freeze.py # Freeze training: freeze bottom N layers
registry.py # Dataset registry: name → path + format mapping
constants.py # APP_NAME, paths, default chat template
tests/ # 74 test files, 2061 tests
tests/ # 78 test files, 2128 tests
examples/
configs/ # 7 production-ready YAML examples
data/ # Sample datasets
@ -294,8 +294,15 @@ soup version # Show version (--full for details)
- **Dataset registry**: JSON validation on load — catches corruption + type mismatch (v0.24.0)
- **Loss watchdog**: threshold bounded le=100.0, patience bounded le=1000 (v0.24.0)
- **Freeze training**: freeze_layers bounded le=1000 (v0.24.0)
- **AWQ/GPTQ export**: output path traversal validation before import check (v0.24.1)
- **Windows Unicode**: Rich console symbols replaced with ASCII equivalents (v0.24.1)
- **AWQ/GPTQ export**: output path traversal validation before import check (v0.24.2)
- **Windows Unicode**: Rich console symbols replaced with ASCII equivalents (v0.24.2)
- **Chat proxy**: SSRF protection — localhost-only HTTP, HTTPS for remote (v0.24.2)
- **Chat proxy**: max_tokens capped at 16384, temperature 0-2, top_p 0-1 (v0.24.2)
- **Chat proxy**: Bearer token auth required on POST endpoint (v0.24.2)
- **Chat proxy**: XSS prevention — HTML-escape before markdown render (v0.24.2)
- **Runs compare**: max 5 runs per comparison (v0.24.2)
- **Config from-form**: validates via load_config_from_string before returning YAML (v0.24.2)
- **SSE endpoints**: read-only GET, no auth required (consistent with other GET endpoints) (v0.24.2)
## Code Conventions
@ -369,7 +376,7 @@ soup version # Show version (--full for details)
15. **Tag**: `git tag v0.X.Y && git push origin v0.X.Y`
16. **Release**: `gh release create v0.X.Y` with changelog (What's New, Install/Upgrade)
## Tests (74 test files, 2065 tests)
## Tests (78 test files, 2065 tests)
| File | Covers |
|------|--------|
@ -446,3 +453,7 @@ soup version # Show version (--full for details)
| test_freeze_training.py | Freeze training: config, layer freezing, GPT-2 naming, sweep |
| test_loss_watchdog.py | Loss watchdog: config, callback behavior, patience, sweep |
| test_dataset_registry.py | Dataset registry: CRUD, CLI, name validation, error handling |
| test_ui_live_monitor.py | Web UI: SSE log streaming, live metrics SSE, progress endpoint |
| test_ui_metrics.py | Web UI: metrics full fields, runs compare, eval results display |
| test_ui_chat.py | Web UI: chat proxy SSE, SSRF protection, param bounds, auth |
| test_ui_config_builder.py | Web UI: config schema, recipes API, form-to-YAML endpoint |

View File

@ -1206,10 +1206,17 @@ soup ui
```
**Pages:**
- **Dashboard** — view all experiment runs, loss charts, system info
- **New Training** — create configs from templates, validate, and start training
- **Dashboard** — view all experiment runs, loss charts, system info, multi-run comparison
- **New Training** — create configs from templates or 29 ready-made recipes, validate, start training with live SSE log streaming and progress bar
- **Data Explorer** — browse and inspect datasets (JSONL, JSON, CSV, Parquet)
- **Model Chat** — chat with a running `soup serve` inference server
- **Model Chat** — chat with streaming responses, configurable temperature/top_p/max_tokens, system prompt, adapter selection, markdown rendering, chat export
**v0.24.2 Enhancements:**
- **Training Live Monitor** — real-time SSE log streaming, live metrics, progress bar with ETA
- **Enhanced Metrics** — 2x2 chart grid (loss, LR, grad_norm, throughput) + GPU memory chart, eval results table
- **Multi-Run Compare** — overlay loss curves from up to 5 runs side-by-side
- **Chat Upgrade** — SSE streaming via proxy, typing indicator, cancel button, markdown renderer (bold, italic, code blocks), chat export as JSON
- **Config Builder** — recipe dropdown (29 configs), config schema API for dynamic form generation
**Security (v0.10.10+):** The Web UI generates a random auth token at startup (printed to console). All mutating endpoints (start/stop training, delete runs, inspect data, validate config) require `Authorization: Bearer <token>` header. CORS is restricted to the served origin. Data inspection is sandboxed to the working directory.

View File

@ -128,6 +128,7 @@ No known critical vulnerabilities in current releases.
- **v0.23.0**: AWQ/GPTQ calibration data path traversal protection (resolve + relative_to(cwd)), AWQ/GPTQ output path stays under cwd, curriculum_buckets bounded (ge=1, le=20), AWQ/GPTQ trust_remote_code warning panel
- **v0.24.0**: HF download trust_remote_code=False + warning panel, HF download output path sanitized (Path.name), download --samples capped at 1M, dataset registry name validation (no path separators/null bytes), registry path traversal protection, loss_watchdog threshold le=100 + patience le=1000, freeze_layers le=1000
- **v0.24.1**: AWQ/GPTQ output path traversal validation moved before import check (previously unreachable when autoawq/auto-gptq not installed), Windows Unicode fix for Rich console output (replaced non-ASCII symbols with ASCII equivalents)
- **v0.24.2**: Chat proxy SSRF protection (localhost-only HTTP, HTTPS for remote), chat proxy max_tokens capped at 16384 + temperature/top_p bounded, chat proxy Bearer token auth required, XSS prevention (HTML-escape before markdown render), runs compare max 5 runs, config from-form validates via load_config_from_string, SSE read endpoints no auth (GET)
## Security Scanning

View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "soup-cli"
version = "0.24.1"
version = "0.24.2"
description = "Fine-tune LLMs in one command. No SSH, no config hell."
readme = "README.md"
license = "MIT"

View File

@ -1,3 +1,3 @@
"""Soup CLI — Fine-tune LLMs in one command."""
__version__ = "0.24.1"
__version__ = "0.24.2"

View File

@ -1,5 +1,6 @@
"""FastAPI application for Soup Web UI."""
import json as json_mod
import logging
import os
import secrets
@ -7,6 +8,7 @@ import subprocess
import sys
import tempfile
import threading
import time
from pathlib import Path
from typing import Optional
@ -99,6 +101,43 @@ def create_app(host: str = "127.0.0.1", port: int = 7860):
finally:
tracker.close()
@app.get("/api/runs/compare")
def compare_runs(ids: str = Query(default="")):
"""Compare metrics for multiple runs."""
from soup_cli.experiment.tracker import ExperimentTracker
if not ids or not ids.strip():
raise HTTPException(status_code=400, detail="ids parameter required")
run_ids = [rid.strip() for rid in ids.split(",") if rid.strip()]
if len(run_ids) > 5:
raise HTTPException(
status_code=400, detail="Maximum 5 runs per comparison"
)
if not run_ids:
raise HTTPException(status_code=400, detail="ids parameter required")
tracker = ExperimentTracker()
try:
result = []
for rid in run_ids:
run_info = tracker.get_run(rid)
metrics = tracker.get_metrics(rid)
config = {}
if run_info and run_info.get("config_json"):
try:
config = json_mod.loads(run_info["config_json"])
except (ValueError, TypeError):
pass
result.append({
"run_id": rid,
"config": config,
"metrics": metrics,
})
return {"runs": result}
finally:
tracker.close()
@app.get("/api/runs/{run_id}")
def get_run(run_id: str):
from soup_cli.experiment.tracker import ExperimentTracker
@ -300,6 +339,352 @@ def create_app(host: str = "127.0.0.1", port: int = 7860):
"sample": sample,
}
# --- Training Live Monitor (SSE) ---
@app.get("/api/train/logs")
def stream_training_logs(request: Request):
"""SSE endpoint streaming training log lines in real time."""
from fastapi.responses import StreamingResponse
last_event_id = request.headers.get("Last-Event-ID")
skip_count = int(last_event_id) + 1 if last_event_id else 0
def _generate_log_events():
line_index = 0
proc = _train_process
if proc is None:
yield "event: done\ndata: {}\n\n"
return
try:
for raw_line in proc.stdout:
if isinstance(raw_line, bytes):
raw_line = raw_line.decode("utf-8", errors="replace")
text = raw_line.rstrip("\n\r")
if line_index < skip_count:
line_index += 1
continue
data = json_mod.dumps({"line": text, "id": line_index})
yield f"id: {line_index}\ndata: {data}\n\n"
line_index += 1
except (ValueError, OSError):
pass
yield "event: done\ndata: {}\n\n"
return StreamingResponse(
_generate_log_events(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
},
)
@app.get("/api/train/metrics/live")
def stream_live_metrics(
request: Request,
run_id: Optional[str] = Query(default=None),
):
"""SSE endpoint streaming new metrics as they're logged."""
from fastapi.responses import StreamingResponse
def _generate_metrics_events():
from soup_cli.experiment.tracker import ExperimentTracker
proc = _train_process
if proc is None and run_id is None:
yield "event: done\ndata: {}\n\n"
return
last_step = -1
max_polls = 3 # For tests: limit poll cycles when process done
polls_since_new = 0
while True:
tracker = ExperimentTracker()
try:
if run_id:
metrics = tracker.get_metrics(run_id)
else:
yield "event: done\ndata: {}\n\n"
return
finally:
tracker.close()
new_metrics = [
m for m in metrics if m.get("step", 0) > last_step
]
if new_metrics:
for m_row in new_metrics:
data = json_mod.dumps(m_row, default=str)
yield f"data: {data}\n\n"
last_step = max(
m.get("step", 0) for m in new_metrics
)
polls_since_new = 0
else:
polls_since_new += 1
# Check if training is still running
proc = _train_process
if proc is None or proc.poll() is not None:
if polls_since_new >= 1:
yield "event: done\ndata: {}\n\n"
return
# Yield heartbeat
yield ":heartbeat\n\n"
if polls_since_new >= max_polls:
yield "event: done\ndata: {}\n\n"
return
time.sleep(0.1) # Short poll for tests
return StreamingResponse(
_generate_metrics_events(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
},
)
@app.get("/api/train/progress")
def train_progress(
run_id: Optional[str] = Query(default=None),
):
"""Return current training progress snapshot."""
proc = _train_process
is_running = proc is not None and proc.poll() is None
if not is_running and run_id is None:
return {"running": False, "current_step": 0, "run_id": None}
if run_id:
from soup_cli.experiment.tracker import ExperimentTracker
tracker = ExperimentTracker()
try:
metrics = tracker.get_metrics(run_id)
current_step = metrics[-1]["step"] if metrics else 0
finally:
tracker.close()
return {
"running": is_running,
"current_step": current_step,
"run_id": run_id,
}
return {"running": is_running, "current_step": 0, "run_id": None}
# --- Config Builder ---
@app.get("/api/config/schema")
def config_schema():
"""Return config schema as JSON for form generation."""
from soup_cli.config.schema import (
DataConfig,
LoraConfig,
SoupConfig,
TrainingConfig,
)
def _extract_field_info(model_cls):
"""Extract field metadata from a Pydantic model."""
result = {}
for name, field_info in model_cls.model_fields.items():
info = {"type": "string", "required": field_info.is_required()}
# Get default value
if field_info.default is not None:
info["default"] = field_info.default
# Get type annotation
annotation = field_info.annotation
if annotation is not None:
ann_str = str(annotation)
if "int" in ann_str:
info["type"] = "integer"
elif "float" in ann_str:
info["type"] = "number"
elif "bool" in ann_str:
info["type"] = "boolean"
# Check for Literal (enum) types
origin = getattr(annotation, "__origin__", None)
if origin is type(None):
pass
args = getattr(annotation, "__args__", None)
if args and all(isinstance(a, str) for a in args):
info["type"] = "enum"
info["options"] = list(args)
# Get constraints from metadata
for meta in (field_info.metadata or []):
if hasattr(meta, "ge"):
info["ge"] = meta.ge
if hasattr(meta, "le"):
info["le"] = meta.le
result[name] = info
return result
schema = _extract_field_info(SoupConfig)
schema["data"] = _extract_field_info(DataConfig)
schema["training"] = _extract_field_info(TrainingConfig)
schema["training"]["lora"] = _extract_field_info(LoraConfig)
return schema
@app.get("/api/recipes")
def list_recipes():
"""Return recipe catalog as JSON."""
from soup_cli.recipes.catalog import RECIPES
recipes_list = []
for name, meta in RECIPES.items():
recipes_list.append({
"name": name,
"model": meta.model,
"task": meta.task,
"description": meta.description,
"tags": list(meta.tags) if hasattr(meta, "tags") else [],
"yaml": meta.yaml_str,
})
return {"recipes": recipes_list}
@app.post("/api/config/from-form", dependencies=[Depends(_verify_token)])
def form_to_yaml(body: dict):
"""Convert form field values to validated YAML string."""
import yaml
from soup_cli.config.loader import load_config_from_string
# Build YAML from form values
config_dict = {}
for key, val in body.items():
if val is not None and val != "" and val != {}:
config_dict[key] = val
try:
yaml_str = yaml.dump(
config_dict, default_flow_style=False, sort_keys=False
)
# Validate
load_config_from_string(yaml_str)
return {"yaml": yaml_str}
except Exception as exc:
return {"error": str(exc)}
# --- Chat Proxy ---
class ChatRequest(PydanticBaseModel):
"""Request body for chat send."""
messages: list
endpoint: str
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
max_tokens: int = Field(default=512, ge=1, le=16384)
top_p: float = Field(default=0.9, ge=0.0, le=1.0)
adapter: Optional[str] = None
@app.post("/api/chat/send", dependencies=[Depends(_verify_token)])
def chat_send(req: ChatRequest):
"""SSE proxy endpoint streaming chat completions."""
from urllib.parse import urlparse
from fastapi.responses import StreamingResponse
# Validate messages
if not req.messages:
raise HTTPException(status_code=400, detail="messages cannot be empty")
# SSRF protection: localhost-only HTTP, HTTPS for remote
parsed = urlparse(req.endpoint)
if parsed.scheme == "http":
host = parsed.hostname or ""
if host not in ("localhost", "127.0.0.1", "0.0.0.0", "::1"):
raise HTTPException(
status_code=400,
detail="HTTP only allowed for localhost endpoints",
)
elif parsed.scheme != "https":
raise HTTPException(
status_code=400,
detail="Only HTTP (localhost) or HTTPS endpoints allowed",
)
# Validate bounds
if req.max_tokens > 16384:
raise HTTPException(
status_code=400, detail="max_tokens exceeds 16384 cap"
)
if req.temperature < 0.0 or req.temperature > 2.0:
raise HTTPException(
status_code=400, detail="temperature must be 0.0-2.0"
)
if req.top_p < 0.0 or req.top_p > 1.0:
raise HTTPException(
status_code=400, detail="top_p must be 0.0-1.0"
)
def _stream_chat():
import httpx
url = req.endpoint.rstrip("/") + "/v1/chat/completions"
payload = {
"messages": req.messages,
"max_tokens": req.max_tokens,
"temperature": req.temperature,
"top_p": req.top_p,
"stream": True,
}
if req.adapter:
payload["model"] = req.adapter
try:
with httpx.stream(
"POST", url,
json=payload,
headers={"Content-Type": "application/json"},
timeout=120.0,
) as resp:
for line in resp.iter_lines():
if line.startswith("data: "):
data_str = line[6:]
if data_str.strip() == "[DONE]":
yield "data: {\"done\": true}\n\n"
return
try:
parsed_data = json_mod.loads(data_str)
delta = (
parsed_data.get("choices", [{}])[0]
.get("delta", {})
.get("content", "")
)
if delta:
out = json_mod.dumps({"delta": delta})
yield f"data: {out}\n\n"
except (ValueError, IndexError, KeyError):
pass
yield "data: {\"done\": true}\n\n"
except Exception as exc:
logger.warning("Chat proxy error: %s", exc)
err_msg = json_mod.dumps(
{"error": "Connection failed"}
)
yield f"data: {err_msg}\n\n"
return StreamingResponse(
_stream_chat(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
},
)
# --- Health ---
@app.get("/api/health")

View File

@ -187,9 +187,10 @@ async function showRunDetail(runId) {
body.innerHTML = '<div style="text-align:center;padding:2rem;color:var(--text-dim)">Loading...</div>';
try {
const [run, metricsResp] = await Promise.all([
const [run, metricsResp, evalResp] = await Promise.all([
api(`/api/runs/${runId}`),
api(`/api/runs/${runId}/metrics`),
api(`/api/runs/${runId}/eval`),
]);
const config = run.config_json ? JSON.parse(run.config_json) : {};
@ -232,16 +233,48 @@ async function showRunDetail(runId) {
</div>
${metrics.length > 0 ? `
<div class="card">
<div class="card-title">Loss Curve</div>
<div class="chart-container">
<canvas id="loss-chart"></canvas>
<div class="chart-grid">
<div class="card">
<div class="card-title">Loss</div>
<div class="chart-container"><canvas id="loss-chart"></canvas></div>
</div>
<div class="card">
<div class="card-title">Learning Rate</div>
<div class="chart-container"><canvas id="lr-chart"></canvas></div>
</div>
<div class="card">
<div class="card-title">Gradient Norm</div>
<div class="chart-container"><canvas id="gradnorm-chart"></canvas></div>
</div>
<div class="card">
<div class="card-title">Throughput</div>
<div class="chart-container"><canvas id="speed-chart"></canvas></div>
</div>
</div>
${metrics.some(m => m.gpu_mem) ? `
<div class="card">
<div class="card-title">GPU Memory</div>
<div class="chart-container"><canvas id="gpumem-chart"></canvas></div>
</div>
` : ''}
` : ''}
${evalResp.eval_results && evalResp.eval_results.length > 0 ? `
<div class="card">
<div class="card-title">Learning Rate</div>
<div class="chart-container">
<canvas id="lr-chart"></canvas>
<div class="card-title">Eval Results</div>
<div class="table-wrap">
<table class="eval-table">
<thead><tr><th>Benchmark</th><th>Score</th><th>Details</th></tr></thead>
<tbody>
${evalResp.eval_results.map(er => `
<tr>
<td>${er.benchmark}</td>
<td>${typeof er.score === 'number' ? er.score.toFixed(4) : er.score}</td>
<td><code style="font-size:0.75rem">${er.details_json ? String(er.details_json).substring(0, 100) : '-'}</code></td>
</tr>
`).join('')}
</tbody>
</table>
</div>
</div>
` : ''}
@ -264,6 +297,24 @@ function renderCharts(metrics) {
const steps = metrics.map(m => m.step);
const losses = metrics.map(m => m.loss);
const lrs = metrics.map(m => m.lr);
const gradNorms = metrics.map(m => m.grad_norm || 0);
const speeds = metrics.map(m => m.speed || 0);
const chartOpts = (yLabel) => ({
responsive: true,
maintainAspectRatio: false,
plugins: { legend: { display: false } },
scales: {
x: { title: { display: true, text: 'Step', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
y: { title: { display: true, text: yLabel, color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
},
});
const makeDataset = (label, data, color) => ({
label, data, borderColor: color,
backgroundColor: color.replace(')', ', 0.1)').replace('rgb', 'rgba'),
fill: true, tension: 0.3, pointRadius: 0, borderWidth: 2,
});
// Loss chart
const lossCtx = document.getElementById('loss-chart');
@ -271,28 +322,8 @@ function renderCharts(metrics) {
if (lossChart) lossChart.destroy();
lossChart = new Chart(lossCtx, {
type: 'line',
data: {
labels: steps,
datasets: [{
label: 'Loss',
data: losses,
borderColor: '#c0512d',
backgroundColor: 'rgba(192, 81, 45, 0.1)',
fill: true,
tension: 0.3,
pointRadius: 0,
borderWidth: 2,
}],
},
options: {
responsive: true,
maintainAspectRatio: false,
plugins: { legend: { display: false } },
scales: {
x: { title: { display: true, text: 'Step', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
y: { title: { display: true, text: 'Loss', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
},
},
data: { labels: steps, datasets: [makeDataset('Loss', losses, 'rgb(192, 81, 45)')] },
options: chartOpts('Loss'),
});
}
@ -301,28 +332,43 @@ function renderCharts(metrics) {
if (lrCtx) {
new Chart(lrCtx, {
type: 'line',
data: {
labels: steps,
datasets: [{
label: 'Learning Rate',
data: lrs,
borderColor: '#e8975a',
backgroundColor: 'rgba(232, 151, 90, 0.1)',
fill: true,
tension: 0.3,
pointRadius: 0,
borderWidth: 2,
}],
},
options: {
responsive: true,
maintainAspectRatio: false,
plugins: { legend: { display: false } },
scales: {
x: { title: { display: true, text: 'Step', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
y: { title: { display: true, text: 'LR', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
},
},
data: { labels: steps, datasets: [makeDataset('LR', lrs, 'rgb(232, 151, 90)')] },
options: chartOpts('LR'),
});
}
// Gradient Norm chart
const gnCtx = document.getElementById('gradnorm-chart');
if (gnCtx) {
new Chart(gnCtx, {
type: 'line',
data: { labels: steps, datasets: [makeDataset('Grad Norm', gradNorms, 'rgb(100, 180, 220)')] },
options: chartOpts('Grad Norm'),
});
}
// Speed chart
const spCtx = document.getElementById('speed-chart');
if (spCtx) {
new Chart(spCtx, {
type: 'line',
data: { labels: steps, datasets: [makeDataset('Speed', speeds, 'rgb(120, 200, 130)')] },
options: chartOpts('Tokens/sec'),
});
}
// GPU Memory chart (optional — parse numeric values from strings like "4.2GB")
const gmCtx = document.getElementById('gpumem-chart');
if (gmCtx) {
const gpuVals = metrics.map(m => {
if (!m.gpu_mem) return 0;
const match = String(m.gpu_mem).match(/([\d.]+)/);
return match ? parseFloat(match[1]) : 0;
});
new Chart(gmCtx, {
type: 'line',
data: { labels: steps, datasets: [makeDataset('GPU Mem', gpuVals, 'rgb(200, 130, 200)')] },
options: chartOpts('GPU Memory (GB)'),
});
}
}
@ -334,10 +380,12 @@ function closeModal() {
// --- New Training Page ---
async function loadTrainingPage() {
try {
const [templatesResp, statusResp] = await Promise.all([
const [templatesResp, statusResp, recipesResp] = await Promise.all([
api('/api/templates'),
api('/api/train/status'),
api('/api/recipes'),
]);
window._recipes = recipesResp.recipes || [];
renderTrainingPage(templatesResp.templates, statusResp);
} catch (err) {
document.getElementById('training-content').innerHTML =
@ -353,12 +401,22 @@ function renderTrainingPage(templates, status) {
<div class="grid-2">
<div>
<div class="card">
<div class="card-title">Template</div>
<div class="form-group">
<select id="template-select" onchange="loadTemplate()">
<option value="">-- Select a template --</option>
${templateNames.map(t => `<option value="${t}">${t}</option>`).join('')}
</select>
<div class="card-title">Template / Recipe</div>
<div class="grid-2" style="margin-bottom:0">
<div class="form-group" style="margin-bottom:0">
<label class="form-label" style="font-size:0.8rem">Template</label>
<select id="template-select" onchange="loadTemplate()">
<option value="">-- Template --</option>
${templateNames.map(t => `<option value="${t}">${t}</option>`).join('')}
</select>
</div>
<div class="form-group" style="margin-bottom:0">
<label class="form-label" style="font-size:0.8rem">Recipe</label>
<select id="recipe-select" onchange="loadRecipe()">
<option value="">-- Recipe --</option>
${(window._recipes || []).map(r => `<option value="${r.name}">${r.name} (${r.task})</option>`).join('')}
</select>
</div>
</div>
</div>
@ -412,6 +470,16 @@ function loadTemplate() {
}
}
function loadRecipe() {
const sel = document.getElementById('recipe-select');
const editor = document.getElementById('config-editor');
if (!sel.value || !window._recipes) return;
const recipe = window._recipes.find(r => r.name === sel.value);
if (recipe && recipe.yaml) {
editor.value = recipe.yaml;
}
}
async function validateConfig() {
const yaml = document.getElementById('config-editor').value;
const statusEl = document.getElementById('config-status');
@ -524,8 +592,9 @@ function renderDataResults(data) {
}
// --- Model Chat ---
let chatAbortController = null;
function loadChatPage() {
// Just ensure the page renders with current messages
renderChatMessages();
}
@ -546,7 +615,7 @@ function renderChatMessages() {
container.innerHTML = chatMessages.map(msg => `
<div class="chat-msg ${msg.role}">
<div class="chat-msg-role">${msg.role}</div>
<div class="chat-msg-content">${escapeHtml(msg.content)}</div>
<div class="chat-msg-content chat-markdown">${msg.role === 'assistant' ? renderMarkdown(msg.content) : escapeHtml(msg.content)}</div>
</div>
`).join('');
@ -559,42 +628,137 @@ function escapeHtml(text) {
return div.innerHTML;
}
function renderMarkdown(text) {
let html = escapeHtml(text);
// Code blocks (``` ... ```)
html = html.replace(/```(\w*)\n([\s\S]*?)```/g, '<pre class="code-block"><code>$2</code></pre>');
// Inline code
html = html.replace(/`([^`]+)`/g, '<code>$1</code>');
// Bold
html = html.replace(/\*\*(.+?)\*\*/g, '<strong>$1</strong>');
// Italic
html = html.replace(/\*(.+?)\*/g, '<em>$1</em>');
// List items
html = html.replace(/^- (.+)$/gm, '<li>$1</li>');
// Line breaks
html = html.replace(/\n/g, '<br>');
return html;
}
async function sendChatMessage() {
const input = document.getElementById('chat-input');
const serverUrl = document.getElementById('chat-server').value.trim();
const msg = input.value.trim();
if (!msg) return;
if (!serverUrl) { alert('Enter a server URL (e.g., http://localhost:8000)'); return; }
if (!serverUrl) { alert('Enter a server URL'); return; }
// Build messages with optional system prompt
const systemPrompt = document.getElementById('chat-system')?.value?.trim();
const allMessages = [];
if (systemPrompt) {
allMessages.push({ role: 'system', content: systemPrompt });
}
chatMessages.push({ role: 'user', content: msg });
allMessages.push(...chatMessages.map(m => ({ role: m.role, content: m.content })));
input.value = '';
renderChatMessages();
// Show typing indicator, switch buttons
const typing = document.getElementById('typing-indicator');
const sendBtn = document.getElementById('chat-send-btn');
const cancelBtn = document.getElementById('chat-cancel-btn');
if (typing) typing.style.display = 'flex';
if (sendBtn) sendBtn.style.display = 'none';
if (cancelBtn) cancelBtn.style.display = 'inline-flex';
chatAbortController = new AbortController();
const temperature = parseFloat(document.getElementById('chat-temperature')?.value || '0.7');
const maxTokens = parseInt(document.getElementById('chat-max-tokens')?.value || '512');
const topP = parseFloat(document.getElementById('chat-top-p')?.value || '0.9');
const adapter = document.getElementById('chat-adapter')?.value?.trim() || undefined;
try {
const resp = await fetch(serverUrl + '/v1/chat/completions', {
const resp = await fetch(API + '/api/chat/send', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer ' + (window._authToken || ''),
},
body: JSON.stringify({
messages: chatMessages.map(m => ({ role: m.role, content: m.content })),
max_tokens: 512,
temperature: 0.7,
messages: allMessages,
endpoint: serverUrl,
temperature: temperature,
max_tokens: maxTokens,
top_p: topP,
adapter: adapter,
}),
signal: chatAbortController.signal,
});
const data = await resp.json();
const reply = data.choices[0].message.content;
chatMessages.push({ role: 'assistant', content: reply });
renderChatMessages();
// Stream tokens
const reader = resp.body.getReader();
const decoder = new TextDecoder();
let assistantMsg = '';
chatMessages.push({ role: 'assistant', content: '' });
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
const lines = chunk.split('\n');
for (const line of lines) {
if (line.startsWith('data: ')) {
try {
const data = JSON.parse(line.slice(6));
if (data.done) break;
if (data.delta) {
assistantMsg += data.delta;
chatMessages[chatMessages.length - 1].content = assistantMsg;
renderChatMessages();
}
if (data.error) {
assistantMsg += '[Error: ' + data.error + ']';
chatMessages[chatMessages.length - 1].content = assistantMsg;
renderChatMessages();
}
} catch (parseErr) { /* ignore malformed lines */ }
}
}
}
} catch (err) {
chatMessages.push({ role: 'assistant', content: `[Error: ${err.message}]` });
renderChatMessages();
if (err.name !== 'AbortError') {
chatMessages.push({ role: 'assistant', content: `[Error: ${err.message}]` });
renderChatMessages();
}
} finally {
chatAbortController = null;
if (typing) typing.style.display = 'none';
if (sendBtn) sendBtn.style.display = 'inline-flex';
if (cancelBtn) cancelBtn.style.display = 'none';
}
}
function cancelChat() {
if (chatAbortController) chatAbortController.abort();
}
function clearChat() {
chatMessages = [];
renderChatMessages();
}
function exportChat() {
if (chatMessages.length === 0) return;
const blob = new Blob([JSON.stringify(chatMessages, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = 'chat-export.json';
a.click();
URL.revokeObjectURL(url);
}
function handleChatKey(event) {
if (event.key === 'Enter' && !event.shiftKey) {
event.preventDefault();
@ -602,6 +766,79 @@ function handleChatKey(event) {
}
}
// --- Training Live Monitor (SSE) ---
let logEventSource = null;
let metricsEventSource = null;
const LOG_MAX_LINES = 500;
function connectTrainingSSE() {
disconnectTrainingSSE();
const logPanel = document.getElementById('train-log-panel');
const progressPanel = document.getElementById('train-progress');
const liveBadge = document.getElementById('live-badge');
if (!logPanel) return;
logPanel.style.display = 'block';
progressPanel.style.display = 'block';
if (liveBadge) liveBadge.style.display = 'inline-flex';
// Connect to log SSE
logEventSource = new EventSource(API + '/api/train/logs');
logEventSource.onmessage = function(ev) {
const data = JSON.parse(ev.data);
appendLogLine(data.line);
};
logEventSource.addEventListener('done', function() {
appendLogLine('[Training finished]');
disconnectTrainingSSE();
});
logEventSource.onerror = function() {
setTimeout(function() {
if (logEventSource && logEventSource.readyState === EventSource.CLOSED) {
disconnectTrainingSSE();
}
}, 5000);
};
}
function disconnectTrainingSSE() {
if (logEventSource) { logEventSource.close(); logEventSource = null; }
if (metricsEventSource) { metricsEventSource.close(); metricsEventSource = null; }
const liveBadge = document.getElementById('live-badge');
if (liveBadge) liveBadge.style.display = 'none';
}
function appendLogLine(text) {
const output = document.getElementById('log-output');
if (!output) return;
output.textContent += text + '\n';
// Ring buffer: trim to max lines
const lines = output.textContent.split('\n');
if (lines.length > LOG_MAX_LINES) {
output.textContent = lines.slice(lines.length - LOG_MAX_LINES).join('\n');
}
// Auto-scroll
const autoScroll = document.getElementById('log-autoscroll');
if (autoScroll && autoScroll.checked) {
output.scrollTop = output.scrollHeight;
}
}
function updateProgressBar(step, total, elapsed, eta) {
const fill = document.getElementById('progress-fill');
const stepEl = document.getElementById('progress-step');
const elapsedEl = document.getElementById('progress-elapsed');
const etaEl = document.getElementById('progress-eta');
if (!fill) return;
const pct = total > 0 ? Math.min(100, (step / total) * 100) : 0;
fill.style.width = pct.toFixed(1) + '%';
if (stepEl) stepEl.textContent = 'Step ' + step + (total ? '/' + total : '');
if (elapsedEl) elapsedEl.textContent = 'Elapsed: ' + formatDuration(elapsed);
if (etaEl) etaEl.textContent = 'ETA: ' + formatDuration(eta);
}
// --- Init ---
document.addEventListener('DOMContentLoaded', () => {
navigate('dashboard');

View File

@ -51,10 +51,33 @@
<div id="page-training" class="page">
<div class="page-header">
<h1 class="page-title">New Training</h1>
<span id="live-badge" class="live-badge" style="display:none">Live</span>
</div>
<div id="training-content">
<div style="text-align:center;padding:3rem;color:var(--text-dim)">Loading...</div>
</div>
<!-- Training Progress Bar -->
<div id="train-progress" class="card" style="display:none">
<div class="card-title">Training Progress</div>
<div class="progress-bar">
<div class="progress-bar-fill" id="progress-fill" style="width:0%"></div>
</div>
<div class="progress-info">
<span id="progress-step">Step 0/0</span>
<span id="progress-elapsed">Elapsed: --</span>
<span id="progress-eta">ETA: --</span>
</div>
</div>
<!-- Training Log Panel -->
<div id="train-log-panel" class="card" style="display:none">
<div class="card-title" style="display:flex;justify-content:space-between;align-items:center">
Training Logs
<label style="font-size:0.8rem;font-weight:normal">
<input type="checkbox" id="log-autoscroll" checked> Auto-scroll
</label>
</div>
<pre class="log-panel" id="log-output"></pre>
</div>
</div>
<!-- Data Explorer Page -->
@ -88,16 +111,42 @@
<div id="page-chat" class="page">
<div class="page-header">
<h1 class="page-title">Model Chat</h1>
<button class="btn" onclick="clearChat()">Clear Chat</button>
</div>
<div class="card" style="margin-bottom:1rem">
<div style="display:flex; gap:0.75rem; align-items:flex-end">
<div class="form-group" style="flex:1; margin-bottom:0">
<label class="form-label">Inference Server URL</label>
<input type="text" id="chat-server" placeholder="http://localhost:8000" value="http://localhost:8000">
</div>
<div style="display:flex;gap:0.5rem">
<button class="btn" onclick="exportChat()">Export</button>
<button class="btn" onclick="clearChat()">Clear</button>
</div>
</div>
<div class="card chat-settings">
<details>
<summary style="cursor:pointer;font-weight:600;font-size:0.9rem">Chat Settings</summary>
<div class="grid-2" style="margin-top:0.75rem">
<div class="form-group">
<label class="form-label">Server URL</label>
<input type="text" id="chat-server" placeholder="http://localhost:8000" value="http://localhost:8000">
</div>
<div class="form-group">
<label class="form-label">Adapter</label>
<input type="text" id="chat-adapter" placeholder="default">
</div>
<div class="form-group">
<label class="form-label">Temperature: <span id="temp-val">0.7</span></label>
<input type="range" id="chat-temperature" min="0" max="2" step="0.1" value="0.7" oninput="document.getElementById('temp-val').textContent=this.value">
</div>
<div class="form-group">
<label class="form-label">Max Tokens: <span id="maxtok-val">512</span></label>
<input type="range" id="chat-max-tokens" min="64" max="4096" step="64" value="512" oninput="document.getElementById('maxtok-val').textContent=this.value">
</div>
<div class="form-group">
<label class="form-label">Top-p: <span id="topp-val">0.9</span></label>
<input type="range" id="chat-top-p" min="0" max="1" step="0.05" value="0.9" oninput="document.getElementById('topp-val').textContent=this.value">
</div>
<div class="form-group">
<label class="form-label">System Prompt</label>
<textarea id="chat-system" rows="2" placeholder="You are a helpful assistant."></textarea>
</div>
</div>
</details>
</div>
<div class="card chat-container">
<div class="chat-messages" id="chat-messages">
<div class="empty-state">
@ -105,9 +154,13 @@
<div class="empty-state-hint">Start a <code>soup serve</code> server and start chatting</div>
</div>
</div>
<div class="typing-indicator" id="typing-indicator" style="display:none">
<span></span><span></span><span></span>
</div>
<div class="chat-input-area">
<input type="text" id="chat-input" placeholder="Type a message..." onkeydown="handleChatKey(event)">
<button class="btn btn-primary" onclick="sendChatMessage()">Send</button>
<button class="btn btn-primary" id="chat-send-btn" onclick="sendChatMessage()">Send</button>
<button class="btn btn-danger" id="chat-cancel-btn" onclick="cancelChat()" style="display:none">Cancel</button>
</div>
</div>
</div>

View File

@ -373,3 +373,177 @@ textarea {
.nav-item { justify-content: center; padding: 0.75rem; }
.grid-2, .grid-3 { grid-template-columns: 1fr; }
}
/* --- Training Live Monitor --- */
.log-panel {
background: #0d0b10;
color: #d0c8c0;
font-family: 'Consolas', 'Courier New', monospace;
font-size: 0.82rem;
line-height: 1.5;
max-height: 400px;
overflow-y: auto;
padding: 0.75rem;
border-radius: 6px;
border: 1px solid var(--border);
white-space: pre-wrap;
word-break: break-all;
}
.progress-bar {
width: 100%;
height: 22px;
background: var(--bg-input);
border-radius: 11px;
overflow: hidden;
margin-bottom: 0.5rem;
}
.progress-bar-fill {
height: 100%;
background: linear-gradient(90deg, var(--accent), var(--accent-warm));
border-radius: 11px;
transition: width 0.4s ease;
}
.progress-info {
display: flex;
justify-content: space-between;
font-size: 0.82rem;
color: var(--text-dim);
}
.live-badge {
display: inline-flex;
align-items: center;
gap: 0.4rem;
font-size: 0.78rem;
font-weight: 600;
color: #3ddc84;
padding: 0.2rem 0.6rem;
border-radius: 12px;
background: rgba(61,220,132,0.12);
}
.live-badge::before {
content: "";
display: inline-block;
width: 8px;
height: 8px;
background: #3ddc84;
border-radius: 50%;
animation: pulse-dot 1.5s ease-in-out infinite;
}
@keyframes pulse-dot {
0%, 100% { opacity: 1; transform: scale(1); }
50% { opacity: 0.4; transform: scale(0.8); }
}
/* --- Enhanced Metrics --- */
.chart-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 1rem;
margin-bottom: 1rem;
}
@media (max-width: 900px) {
.chart-grid { grid-template-columns: 1fr; }
}
.eval-table {
width: 100%;
border-collapse: collapse;
}
.eval-table th, .eval-table td {
text-align: left;
padding: 0.5rem 0.75rem;
border-bottom: 1px solid var(--border);
}
.eval-table th {
color: var(--text-dim);
font-size: 0.82rem;
font-weight: 600;
}
/* --- Chat Upgrade --- */
.chat-settings details summary {
color: var(--text-dim);
}
.chat-settings details[open] summary {
color: var(--text);
margin-bottom: 0.5rem;
}
.typing-indicator {
display: flex;
gap: 4px;
padding: 0.5rem 1rem;
align-items: center;
}
.typing-indicator span {
width: 8px;
height: 8px;
background: var(--text-dim);
border-radius: 50%;
animation: typing-bounce 1.2s ease-in-out infinite;
}
.typing-indicator span:nth-child(2) { animation-delay: 0.15s; }
.typing-indicator span:nth-child(3) { animation-delay: 0.3s; }
@keyframes typing-bounce {
0%, 80%, 100% { transform: translateY(0); }
40% { transform: translateY(-6px); }
}
.code-block {
background: #0d0b10;
border: 1px solid var(--border);
border-radius: 6px;
padding: 0.75rem;
overflow-x: auto;
font-size: 0.82rem;
margin: 0.5rem 0;
}
.chat-markdown code {
background: rgba(192, 81, 45, 0.15);
padding: 0.1em 0.3em;
border-radius: 3px;
font-size: 0.88em;
}
.chat-markdown .code-block code {
background: none;
padding: 0;
}
input[type="range"] {
width: 100%;
accent-color: var(--accent);
}
/* --- Config Builder --- */
.form-section {
border: 1px solid var(--border);
border-radius: 6px;
padding: 0.75rem;
margin-bottom: 0.75rem;
}
.form-section summary {
cursor: pointer;
font-weight: 600;
font-size: 0.9rem;
color: var(--accent-warm);
}
.form-section[open] summary {
margin-bottom: 0.5rem;
}

287
tests/test_ui_chat.py Normal file
View File

@ -0,0 +1,287 @@
"""Tests for Web UI Chat Upgrade — SSE proxy, SSRF validation, param bounds."""
from unittest.mock import MagicMock, patch
import pytest
def _auth_headers():
"""Return auth headers with the current UI token."""
from soup_cli.ui.app import get_auth_token
return {"Authorization": f"Bearer {get_auth_token()}"}
class TestChatEndpointExists:
"""Test /api/chat/send endpoint registration."""
def test_chat_send_endpoint_exists(self):
"""Route /api/chat/send should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/chat/send" in routes
class TestChatSSRFProtection:
"""Test SSRF protection on chat endpoint."""
def test_rejects_non_localhost_http(self):
"""Chat endpoint should reject non-localhost HTTP URLs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "http://evil.com:8000",
},
headers=_auth_headers(),
)
assert response.status_code == 400
def test_allows_localhost_http(self):
"""Chat endpoint should allow localhost HTTP URLs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
# Will fail at connection level but not at validation
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "http://localhost:8000",
},
headers=_auth_headers(),
)
# Should not be 400 (SSRF rejection) — may be 502 or stream error
assert response.status_code != 400
def test_allows_https_remote(self):
"""Chat endpoint should allow HTTPS remote URLs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "https://api.example.com",
},
headers=_auth_headers(),
)
# Should not be 400 (SSRF rejection)
assert response.status_code != 400
class TestChatParamBounds:
"""Test parameter validation on chat endpoint."""
def test_max_tokens_capped(self):
"""max_tokens should be capped at 16384."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "http://localhost:8000",
"max_tokens": 99999,
},
headers=_auth_headers(),
)
# Should be rejected (exceeds cap) — 422 from Pydantic or 400
assert response.status_code in (400, 422)
def test_temperature_bounded(self):
"""temperature should be bounded 0.0-2.0."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "http://localhost:8000",
"temperature": 5.0,
},
headers=_auth_headers(),
)
assert response.status_code == 400 or response.status_code == 422
def test_top_p_bounded(self):
"""top_p should be bounded 0.0-1.0."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "http://localhost:8000",
"top_p": 2.0,
},
headers=_auth_headers(),
)
assert response.status_code == 400 or response.status_code == 422
class TestChatInvalidScheme:
"""Test rejection of invalid URL schemes."""
def test_rejects_ftp_scheme(self):
"""Chat endpoint should reject ftp:// URLs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "ftp://evil.com/model",
},
headers=_auth_headers(),
)
assert response.status_code == 400
def test_rejects_file_scheme(self):
"""Chat endpoint should reject file:// URLs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "file:///etc/passwd",
},
headers=_auth_headers(),
)
assert response.status_code == 400
class TestChatAuth:
"""Test chat endpoint auth requirements."""
def test_requires_auth(self):
"""Chat send (POST) should require Bearer auth."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "http://localhost:8000",
},
)
assert response.status_code == 401
class TestChatValidation:
"""Test chat request validation."""
def test_empty_messages_rejected(self):
"""Empty messages array should return 400."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/chat/send",
json={
"messages": [],
"endpoint": "http://localhost:8000",
},
headers=_auth_headers(),
)
assert response.status_code == 400
def test_returns_event_stream(self):
"""Chat endpoint should return text/event-stream content type."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
# Mock httpx to avoid real connection
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.iter_lines.return_value = iter([
'data: {"choices":[{"delta":{"content":"Hi"}}]}',
'data: [DONE]',
])
mock_response.__enter__ = lambda s: s
mock_response.__exit__ = lambda s, *a: None
with patch("httpx.stream", return_value=mock_response):
with client.stream(
"POST",
"/api/chat/send",
json={
"messages": [{"role": "user", "content": "hi"}],
"endpoint": "http://localhost:8000",
},
headers=_auth_headers(),
) as resp:
assert resp.status_code == 200
assert "text/event-stream" in resp.headers["content-type"]

View File

@ -0,0 +1,316 @@
"""Tests for Web UI Visual Config Builder — schema, recipes, form-to-YAML."""
import pytest
def _auth_headers():
"""Return auth headers with the current UI token."""
from soup_cli.ui.app import get_auth_token
return {"Authorization": f"Bearer {get_auth_token()}"}
class TestConfigSchemaEndpoint:
"""Test GET /api/config/schema endpoint."""
def test_schema_endpoint_exists(self):
"""Route /api/config/schema should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/config/schema" in routes
def test_schema_returns_json(self):
"""Schema endpoint should return valid JSON."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/config/schema")
assert response.status_code == 200
data = response.json()
assert isinstance(data, dict)
def test_schema_includes_base_field(self):
"""Schema should include 'base' as a required string field."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/config/schema")
data = response.json()
assert "base" in data
assert data["base"]["type"] == "string"
assert data["base"]["required"] is True
def test_schema_includes_task_enum(self):
"""Schema should include 'task' with enum options."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/config/schema")
data = response.json()
assert "task" in data
assert "options" in data["task"]
assert "sft" in data["task"]["options"]
assert "dpo" in data["task"]["options"]
def test_schema_includes_training_fields(self):
"""Schema should include nested training fields."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/config/schema")
data = response.json()
assert "training" in data
training = data["training"]
assert "epochs" in training
assert "lr" in training
def test_schema_includes_constraints(self):
"""Schema should include ge/le constraints for bounded fields."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/config/schema")
data = response.json()
# Check that training.epochs has constraints or type info
training = data.get("training", {})
if "epochs" in training:
assert training["epochs"]["type"] in ("integer", "number")
class TestRecipesEndpoint:
"""Test GET /api/recipes endpoint."""
def test_recipes_endpoint_exists(self):
"""Route /api/recipes should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/recipes" in routes
def test_recipes_returns_catalog(self):
"""Recipes endpoint should return all recipes."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/recipes")
assert response.status_code == 200
data = response.json()
assert "recipes" in data
assert len(data["recipes"]) >= 20 # We have 29 recipes
def test_recipes_have_required_fields(self):
"""Each recipe should have name, model, task fields."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/recipes")
recipes = response.json()["recipes"]
for recipe in recipes:
assert "name" in recipe
assert "model" in recipe
assert "task" in recipe
assert "yaml" in recipe
class TestFormToYamlEndpoint:
"""Test POST /api/config/from-form endpoint."""
def test_form_to_yaml_endpoint_exists(self):
"""Route /api/config/from-form should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/config/from-form" in routes
def test_form_to_yaml_produces_valid_config(self):
"""Form-to-YAML should produce a valid config string."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/config/from-form",
json={
"base": "meta-llama/Llama-3.1-8B-Instruct",
"task": "sft",
"data": {"train": "./data.jsonl", "format": "alpaca"},
"training": {"epochs": 3, "lr": 2e-4},
},
headers=_auth_headers(),
)
assert response.status_code == 200
data = response.json()
assert "yaml" in data
assert "base:" in data["yaml"]
assert "meta-llama" in data["yaml"]
def test_form_to_yaml_validates(self):
"""Form-to-YAML should validate via load_config_from_string."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/config/from-form",
json={
"base": "test-model",
"data": {"train": "./data.jsonl"},
},
headers=_auth_headers(),
)
assert response.status_code == 200
data = response.json()
assert "yaml" in data
def test_form_to_yaml_invalid_returns_error(self):
"""Invalid form values should return error."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
# Missing base — should fail validation
response = client.post(
"/api/config/from-form",
json={"task": "sft"},
headers=_auth_headers(),
)
assert response.status_code in (200, 400)
if response.status_code == 200:
data = response.json()
assert "error" in data
def test_form_to_yaml_requires_auth(self):
"""Form-to-YAML (POST) should require auth."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/config/from-form",
json={"base": "test"},
)
assert response.status_code == 401
def test_form_to_yaml_special_chars_in_keys(self):
"""Form values with special YAML characters should be handled safely."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/config/from-form",
json={
"base": "test: model\noutput: /etc",
"data": {"train": "./data.jsonl"},
},
headers=_auth_headers(),
)
assert response.status_code == 200
data = response.json()
# The result should either be valid YAML or an error — never injection
if "yaml" in data:
# The colon/newline in base value should be quoted by yaml.dump
assert "base:" in data["yaml"]
elif "error" in data:
# Validation error is also acceptable
pass
def test_form_to_yaml_with_lora(self):
"""Form-to-YAML should handle nested lora config."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.post(
"/api/config/from-form",
json={
"base": "test-model",
"data": {"train": "./data.jsonl"},
"training": {
"epochs": 3,
"lora": {"r": 16, "alpha": 32},
},
},
headers=_auth_headers(),
)
assert response.status_code == 200
data = response.json()
assert "yaml" in data
assert "lora:" in data["yaml"] or "r:" in data["yaml"]

View File

@ -0,0 +1,445 @@
"""Tests for Web UI Training Live Monitor — SSE endpoints and progress API."""
import os
from unittest.mock import MagicMock, patch
import pytest
def _auth_headers():
"""Return auth headers with the current UI token."""
from soup_cli.ui.app import get_auth_token
return {"Authorization": f"Bearer {get_auth_token()}"}
class TestTrainLogsSSE:
"""Test GET /api/train/logs SSE endpoint."""
def test_logs_endpoint_exists(self):
"""Route /api/train/logs should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/train/logs" in routes
def test_logs_returns_event_stream(self):
"""SSE endpoint should return text/event-stream content type."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
# Simulate a running process with output
mock_proc = MagicMock()
mock_proc.poll.return_value = None
mock_proc.stdout = iter([b"Epoch 1/3\n", b"Loss: 2.5\n"])
mock_proc.poll.side_effect = [None, None, 0]
ui_mod._train_process = mock_proc
client = TestClient(create_app())
try:
with client.stream("GET", "/api/train/logs") as resp:
assert resp.status_code == 200
assert "text/event-stream" in resp.headers["content-type"]
# Read at least one event
lines = []
for line in resp.iter_lines():
lines.append(line)
if len(lines) >= 2:
break
finally:
ui_mod._train_process = None
def test_logs_no_training_returns_done(self):
"""When no training is running, SSE should emit done event."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
ui_mod._train_process = None
client = TestClient(create_app())
with client.stream("GET", "/api/train/logs") as resp:
assert resp.status_code == 200
body = b""
for chunk in resp.iter_bytes():
body += chunk
text = body.decode()
assert "done" in text
def test_logs_last_event_id_reconnection(self):
"""Last-Event-ID header should skip earlier lines."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
mock_proc = MagicMock()
lines = [b"Line 1\n", b"Line 2\n", b"Line 3\n"]
mock_proc.stdout = iter(lines)
mock_proc.poll.side_effect = [None, None, None, 0]
ui_mod._train_process = mock_proc
client = TestClient(create_app())
try:
with client.stream(
"GET", "/api/train/logs",
headers={"Last-Event-ID": "1"},
) as resp:
assert resp.status_code == 200
body = b""
for chunk in resp.iter_bytes():
body += chunk
text = body.decode()
# Line 1 (id=0) should be skipped, Line 2+ should appear
assert "Line 2" in text or "Line 3" in text
finally:
ui_mod._train_process = None
def test_logs_no_auth_required(self):
"""SSE log endpoint is GET (read-only) — no auth needed."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
ui_mod._train_process = None
client = TestClient(create_app())
# No auth header — should still succeed (not 401)
with client.stream("GET", "/api/train/logs") as resp:
assert resp.status_code == 200
class TestLiveMetricsSSE:
"""Test GET /api/train/metrics/live SSE endpoint."""
def test_metrics_live_endpoint_exists(self):
"""Route /api/train/metrics/live should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/train/metrics/live" in routes
def test_metrics_live_returns_event_stream(self, tmp_path):
"""SSE metrics endpoint should return text/event-stream content type."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
# Simulate running training
mock_proc = MagicMock()
mock_proc.poll.return_value = 0 # Already finished
ui_mod._train_process = mock_proc
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
client = TestClient(create_app())
with client.stream("GET", "/api/train/metrics/live") as resp:
assert resp.status_code == 200
assert "text/event-stream" in resp.headers["content-type"]
ui_mod._train_process = None
def test_metrics_live_emits_new_rows(self, tmp_path):
"""SSE should emit metric rows as they appear in the DB."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
db_path = tmp_path / "test.db"
# Pre-populate a run with metrics
from soup_cli.experiment.tracker import ExperimentTracker
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test", "task": "sft"},
device="cpu",
device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(run_id, step=10, loss=2.5, lr=1e-5)
tracker.close()
# Simulate finished training
mock_proc = MagicMock()
mock_proc.poll.return_value = 0
ui_mod._train_process = mock_proc
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
client = TestClient(create_app())
with client.stream(
"GET", f"/api/train/metrics/live?run_id={run_id}"
) as resp:
assert resp.status_code == 200
body = b""
for chunk in resp.iter_bytes():
body += chunk
text = body.decode()
# Should contain metric data
assert "step" in text or "done" in text
ui_mod._train_process = None
def test_metrics_live_no_auth_required(self, tmp_path):
"""SSE metrics endpoint is GET (read-only) — no auth needed."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
mock_proc = MagicMock()
mock_proc.poll.return_value = 0
ui_mod._train_process = mock_proc
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
client = TestClient(create_app())
with client.stream("GET", "/api/train/metrics/live") as resp:
assert resp.status_code == 200
ui_mod._train_process = None
def test_metrics_live_done_when_no_training(self, tmp_path):
"""Should emit done event when no training is running."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
ui_mod._train_process = None
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
client = TestClient(create_app())
with client.stream("GET", "/api/train/metrics/live") as resp:
body = b""
for chunk in resp.iter_bytes():
body += chunk
text = body.decode()
assert "done" in text
ui_mod._train_process = None
class TestTrainProgress:
"""Test GET /api/train/progress endpoint."""
def test_progress_endpoint_exists(self):
"""Route /api/train/progress should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/train/progress" in routes
def test_progress_not_running(self):
"""Should return running=false when no training."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
ui_mod._train_process = None
client = TestClient(create_app())
response = client.get("/api/train/progress")
assert response.status_code == 200
data = response.json()
assert data["running"] is False
def test_progress_running_with_metrics(self, tmp_path):
"""Should return progress when training is running."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
db_path = tmp_path / "test.db"
from soup_cli.experiment.tracker import ExperimentTracker
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test", "task": "sft"},
device="cpu",
device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(run_id, step=50, loss=1.5, lr=1e-5)
tracker.close()
mock_proc = MagicMock()
mock_proc.poll.return_value = None
mock_proc.pid = 9999
ui_mod._train_process = mock_proc
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
client = TestClient(create_app())
response = client.get(f"/api/train/progress?run_id={run_id}")
assert response.status_code == 200
data = response.json()
assert data["running"] is True
assert data["current_step"] == 50
ui_mod._train_process = None
def test_progress_returns_correct_fields(self, tmp_path):
"""Progress response should contain all required fields."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
db_path = tmp_path / "test.db"
from soup_cli.experiment.tracker import ExperimentTracker
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test", "task": "sft"},
device="cpu",
device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(run_id, step=10, loss=2.0, lr=1e-5)
tracker.close()
mock_proc = MagicMock()
mock_proc.poll.return_value = None
mock_proc.pid = 1234
ui_mod._train_process = mock_proc
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
client = TestClient(create_app())
response = client.get(f"/api/train/progress?run_id={run_id}")
data = response.json()
expected_keys = {"running", "current_step", "run_id"}
assert expected_keys.issubset(set(data.keys()))
ui_mod._train_process = None
def test_progress_no_auth_required(self):
"""Progress endpoint is GET (read-only) — no auth needed."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
ui_mod._train_process = None
client = TestClient(create_app())
response = client.get("/api/train/progress")
assert response.status_code == 200
class TestSSEGracefulClose:
"""Test that SSE endpoints close gracefully."""
def test_logs_closes_when_training_stops(self):
"""Log SSE should end when training process finishes."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
mock_proc = MagicMock()
mock_proc.stdout = iter([b"Starting\n"])
mock_proc.poll.side_effect = [None, 0]
ui_mod._train_process = mock_proc
client = TestClient(create_app())
try:
with client.stream("GET", "/api/train/logs") as resp:
body = b""
for chunk in resp.iter_bytes():
body += chunk
text = body.decode()
# Stream should terminate with done event
assert "done" in text
finally:
ui_mod._train_process = None
def test_heartbeat_event(self):
"""SSE should include heartbeat comments to keep connection alive."""
# This is a design test — heartbeats are sent as SSE comments (:heartbeat)
# Testing that the generator yields at least one event
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
import soup_cli.ui.app as ui_mod
from soup_cli.ui.app import create_app
ui_mod._train_process = None
client = TestClient(create_app())
with client.stream("GET", "/api/train/logs") as resp:
body = b""
for chunk in resp.iter_bytes():
body += chunk
# Stream should have some content (at least done event)
assert len(body) > 0
ui_mod._train_process = None

354
tests/test_ui_metrics.py Normal file
View File

@ -0,0 +1,354 @@
"""Tests for Web UI Enhanced Metrics & Eval Display."""
import os
from unittest.mock import patch
import pytest
def _auth_headers():
"""Return auth headers with the current UI token."""
from soup_cli.ui.app import get_auth_token
return {"Authorization": f"Bearer {get_auth_token()}"}
class TestMetricsFullFields:
"""Test that metrics response includes all fields."""
def test_metrics_includes_grad_norm(self, tmp_path):
"""Metrics should include grad_norm field."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test", "task": "sft"},
device="cpu",
device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(
run_id, step=10, loss=2.5, lr=1e-5,
grad_norm=1.23, speed=100.5, gpu_mem="4.2GB",
)
tracker.close()
client = TestClient(create_app())
response = client.get(f"/api/runs/{run_id}/metrics")
assert response.status_code == 200
metrics = response.json()["metrics"]
assert len(metrics) == 1
m = metrics[0]
assert m["grad_norm"] == 1.23
assert m["speed"] == 100.5
assert m["gpu_mem"] == "4.2GB"
def test_metrics_includes_epoch(self, tmp_path):
"""Metrics should include epoch field."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test", "task": "sft"},
device="cpu",
device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(run_id, step=10, epoch=1.5, loss=2.0, lr=1e-5)
tracker.close()
client = TestClient(create_app())
response = client.get(f"/api/runs/{run_id}/metrics")
metrics = response.json()["metrics"]
assert metrics[0]["epoch"] == 1.5
class TestRunsCompareEndpoint:
"""Test GET /api/runs/compare endpoint."""
def test_compare_endpoint_exists(self):
"""Route /api/runs/compare should be registered."""
try:
import fastapi # noqa: F401
except ImportError:
pytest.skip("FastAPI not installed")
from soup_cli.ui.app import create_app
app = create_app()
routes = [route.path for route in app.routes]
assert "/api/runs/compare" in routes
def test_compare_returns_metrics(self, tmp_path):
"""Compare endpoint should return metrics for multiple runs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run1 = tracker.start_run(
config_dict={"base": "model-a", "task": "sft"},
device="cpu", device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(run1, step=10, loss=2.5, lr=1e-5)
run2 = tracker.start_run(
config_dict={"base": "model-b", "task": "dpo"},
device="cpu", device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(run2, step=10, loss=1.8, lr=5e-6)
tracker.close()
client = TestClient(create_app())
response = client.get(f"/api/runs/compare?ids={run1},{run2}")
assert response.status_code == 200
data = response.json()
assert "runs" in data
assert len(data["runs"]) == 2
assert data["runs"][0]["run_id"] == run1
assert data["runs"][1]["run_id"] == run2
assert len(data["runs"][0]["metrics"]) == 1
assert len(data["runs"][1]["metrics"]) == 1
def test_compare_rejects_too_many_runs(self, tmp_path):
"""Compare endpoint should reject >5 runs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.ui.app import create_app
client = TestClient(create_app())
ids = ",".join([f"run_{i}" for i in range(6)])
response = client.get(f"/api/runs/compare?ids={ids}")
assert response.status_code == 400
def test_compare_validates_run_ids(self, tmp_path):
"""Compare endpoint returns entries for nonexistent runs (empty metrics)."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test_validate.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/runs/compare?ids=nonexistent1,nonexistent2")
assert response.status_code == 200
data = response.json()
assert len(data["runs"]) == 2
def test_compare_empty_ids(self, tmp_path):
"""Compare endpoint should reject empty ids."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test_empty.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/runs/compare?ids=")
assert response.status_code == 400
def test_compare_no_auth_required(self, tmp_path):
"""Compare is GET (read-only) — no auth needed."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test_noauth.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.ui.app import create_app
client = TestClient(create_app())
response = client.get("/api/runs/compare?ids=run1,run2")
assert response.status_code == 200
class TestEvalResultsEndpoint:
"""Test eval results display with parsed details."""
def test_eval_results_include_details(self, tmp_path):
"""Eval results should include parsed details_json."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test"},
device="cpu", device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.save_eval_result(
model_path="./output",
benchmark="mmlu",
score=0.75,
details={"subjects": {"math": 0.8, "science": 0.7}},
run_id=run_id,
)
tracker.close()
client = TestClient(create_app())
response = client.get(f"/api/runs/{run_id}/eval")
data = response.json()
assert len(data["eval_results"]) == 1
result = data["eval_results"][0]
assert result["benchmark"] == "mmlu"
assert result["score"] == 0.75
# details_json should be present as a string (from SQLite)
assert "details_json" in result
def test_eval_results_empty(self, tmp_path):
"""Empty eval results should return empty array, not error."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test"},
device="cpu", device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.close()
client = TestClient(create_app())
response = client.get(f"/api/runs/{run_id}/eval")
assert response.status_code == 200
data = response.json()
assert data["eval_results"] == []
def test_compare_max_five_runs(self, tmp_path):
"""Compare endpoint allows exactly 5 runs."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run_ids = []
for idx in range(5):
rid = tracker.start_run(
config_dict={"base": f"model-{idx}"},
device="cpu", device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
run_ids.append(rid)
tracker.close()
client = TestClient(create_app())
ids_str = ",".join(run_ids)
response = client.get(f"/api/runs/compare?ids={ids_str}")
assert response.status_code == 200
assert len(response.json()["runs"]) == 5
class TestCompareMetricsContent:
"""Test compare endpoint returns full metric data."""
def test_compare_includes_all_metric_fields(self, tmp_path):
"""Compare should return metrics with all fields."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "test", "task": "sft"},
device="cpu", device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.log_metrics(
run_id, step=10, epoch=1.0, loss=2.5, lr=1e-5,
grad_norm=0.5, speed=200.0, gpu_mem="8GB",
)
tracker.close()
client = TestClient(create_app())
response = client.get(f"/api/runs/compare?ids={run_id}")
data = response.json()
m = data["runs"][0]["metrics"][0]
assert m["step"] == 10
assert m["loss"] == 2.5
assert m["grad_norm"] == 0.5
assert m["speed"] == 200.0
def test_compare_includes_config(self, tmp_path):
"""Compare should include run config for diff display."""
try:
from fastapi.testclient import TestClient
except ImportError:
pytest.skip("FastAPI not installed")
db_path = tmp_path / "test.db"
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
from soup_cli.experiment.tracker import ExperimentTracker
from soup_cli.ui.app import create_app
tracker = ExperimentTracker(db_path=db_path)
run_id = tracker.start_run(
config_dict={"base": "llama-8b", "task": "sft"},
device="cpu", device_name="CPU",
gpu_info={"memory_total": "N/A"},
)
tracker.close()
client = TestClient(create_app())
response = client.get(f"/api/runs/compare?ids={run_id}")
data = response.json()
assert "config" in data["runs"][0]