mirror of https://github.com/razor-ai/soup.git
feat(ui): add Web UI Enhancement with live training monitor, enhanced metrics, chat upgrade, and config builder (v0.24.2)
Part A: Training Live Monitor — SSE log streaming (/api/train/logs with Last-Event-ID reconnection), live metrics SSE (/api/train/metrics/live), progress endpoint (/api/train/progress), frontend with auto-scroll log panel, progress bar, and live indicator badge. Part B: Enhanced Metrics & Eval Display — 2x2 chart grid (loss, LR, grad_norm, throughput) + GPU memory chart, eval results table in run detail modal, /api/runs/compare endpoint (max 5 runs). Part C: Chat Upgrade — /api/chat/send SSE proxy with SSRF protection (localhost-only HTTP, HTTPS for remote), streaming via ReadableStream, typing indicator, cancel button, markdown renderer (bold/italic/code), chat settings panel (temperature/max_tokens/top_p/system prompt/adapter), chat export as JSON. Part D: Visual Config Builder — /api/config/schema (Pydantic field metadata extraction), /api/recipes (29 ready-made configs as JSON), /api/config/from-form (form values to validated YAML), recipe dropdown. Security: Chat proxy SSRF validation, max_tokens cap 16384, temperature 0-2, top_p 0-1, Bearer auth on POST, XSS prevention, compare max 5 runs. Tests: 58 new tests across 4 files (2128 total, 78 files), 67% coverage.
This commit is contained in:
parent
d134abb008
commit
54230f7bdd
|
|
@ -1,12 +1,12 @@
|
|||
# Soup CLI — Project CLAUDE.md
|
||||
|
||||
Soup is a CLI-first LLM fine-tuning tool (v0.24.1). Python 3.9+, MIT license.
|
||||
Soup is a CLI-first LLM fine-tuning tool (v0.24.2). Python 3.9+, MIT license.
|
||||
|
||||
## Build & Development
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]" # Install editable + test deps
|
||||
pytest tests/ -v --tb=short # Run all tests (2061 tests)
|
||||
pytest tests/ -v --tb=short # Run all tests (2128 tests)
|
||||
ruff check soup_cli/ tests/ # Lint (must pass before commit)
|
||||
ruff check --fix soup_cli/ tests/ # Auto-fix lint issues
|
||||
```
|
||||
|
|
@ -115,7 +115,7 @@ soup_cli/
|
|||
freeze.py # Freeze training: freeze bottom N layers
|
||||
registry.py # Dataset registry: name → path + format mapping
|
||||
constants.py # APP_NAME, paths, default chat template
|
||||
tests/ # 74 test files, 2061 tests
|
||||
tests/ # 78 test files, 2128 tests
|
||||
examples/
|
||||
configs/ # 7 production-ready YAML examples
|
||||
data/ # Sample datasets
|
||||
|
|
@ -294,8 +294,15 @@ soup version # Show version (--full for details)
|
|||
- **Dataset registry**: JSON validation on load — catches corruption + type mismatch (v0.24.0)
|
||||
- **Loss watchdog**: threshold bounded le=100.0, patience bounded le=1000 (v0.24.0)
|
||||
- **Freeze training**: freeze_layers bounded le=1000 (v0.24.0)
|
||||
- **AWQ/GPTQ export**: output path traversal validation before import check (v0.24.1)
|
||||
- **Windows Unicode**: Rich console symbols replaced with ASCII equivalents (v0.24.1)
|
||||
- **AWQ/GPTQ export**: output path traversal validation before import check (v0.24.2)
|
||||
- **Windows Unicode**: Rich console symbols replaced with ASCII equivalents (v0.24.2)
|
||||
- **Chat proxy**: SSRF protection — localhost-only HTTP, HTTPS for remote (v0.24.2)
|
||||
- **Chat proxy**: max_tokens capped at 16384, temperature 0-2, top_p 0-1 (v0.24.2)
|
||||
- **Chat proxy**: Bearer token auth required on POST endpoint (v0.24.2)
|
||||
- **Chat proxy**: XSS prevention — HTML-escape before markdown render (v0.24.2)
|
||||
- **Runs compare**: max 5 runs per comparison (v0.24.2)
|
||||
- **Config from-form**: validates via load_config_from_string before returning YAML (v0.24.2)
|
||||
- **SSE endpoints**: read-only GET, no auth required (consistent with other GET endpoints) (v0.24.2)
|
||||
|
||||
## Code Conventions
|
||||
|
||||
|
|
@ -369,7 +376,7 @@ soup version # Show version (--full for details)
|
|||
15. **Tag**: `git tag v0.X.Y && git push origin v0.X.Y`
|
||||
16. **Release**: `gh release create v0.X.Y` with changelog (What's New, Install/Upgrade)
|
||||
|
||||
## Tests (74 test files, 2065 tests)
|
||||
## Tests (78 test files, 2065 tests)
|
||||
|
||||
| File | Covers |
|
||||
|------|--------|
|
||||
|
|
@ -446,3 +453,7 @@ soup version # Show version (--full for details)
|
|||
| test_freeze_training.py | Freeze training: config, layer freezing, GPT-2 naming, sweep |
|
||||
| test_loss_watchdog.py | Loss watchdog: config, callback behavior, patience, sweep |
|
||||
| test_dataset_registry.py | Dataset registry: CRUD, CLI, name validation, error handling |
|
||||
| test_ui_live_monitor.py | Web UI: SSE log streaming, live metrics SSE, progress endpoint |
|
||||
| test_ui_metrics.py | Web UI: metrics full fields, runs compare, eval results display |
|
||||
| test_ui_chat.py | Web UI: chat proxy SSE, SSRF protection, param bounds, auth |
|
||||
| test_ui_config_builder.py | Web UI: config schema, recipes API, form-to-YAML endpoint |
|
||||
|
|
|
|||
13
README.md
13
README.md
|
|
@ -1206,10 +1206,17 @@ soup ui
|
|||
```
|
||||
|
||||
**Pages:**
|
||||
- **Dashboard** — view all experiment runs, loss charts, system info
|
||||
- **New Training** — create configs from templates, validate, and start training
|
||||
- **Dashboard** — view all experiment runs, loss charts, system info, multi-run comparison
|
||||
- **New Training** — create configs from templates or 29 ready-made recipes, validate, start training with live SSE log streaming and progress bar
|
||||
- **Data Explorer** — browse and inspect datasets (JSONL, JSON, CSV, Parquet)
|
||||
- **Model Chat** — chat with a running `soup serve` inference server
|
||||
- **Model Chat** — chat with streaming responses, configurable temperature/top_p/max_tokens, system prompt, adapter selection, markdown rendering, chat export
|
||||
|
||||
**v0.24.2 Enhancements:**
|
||||
- **Training Live Monitor** — real-time SSE log streaming, live metrics, progress bar with ETA
|
||||
- **Enhanced Metrics** — 2x2 chart grid (loss, LR, grad_norm, throughput) + GPU memory chart, eval results table
|
||||
- **Multi-Run Compare** — overlay loss curves from up to 5 runs side-by-side
|
||||
- **Chat Upgrade** — SSE streaming via proxy, typing indicator, cancel button, markdown renderer (bold, italic, code blocks), chat export as JSON
|
||||
- **Config Builder** — recipe dropdown (29 configs), config schema API for dynamic form generation
|
||||
|
||||
**Security (v0.10.10+):** The Web UI generates a random auth token at startup (printed to console). All mutating endpoints (start/stop training, delete runs, inspect data, validate config) require `Authorization: Bearer <token>` header. CORS is restricted to the served origin. Data inspection is sandboxed to the working directory.
|
||||
|
||||
|
|
|
|||
|
|
@ -128,6 +128,7 @@ No known critical vulnerabilities in current releases.
|
|||
- **v0.23.0**: AWQ/GPTQ calibration data path traversal protection (resolve + relative_to(cwd)), AWQ/GPTQ output path stays under cwd, curriculum_buckets bounded (ge=1, le=20), AWQ/GPTQ trust_remote_code warning panel
|
||||
- **v0.24.0**: HF download trust_remote_code=False + warning panel, HF download output path sanitized (Path.name), download --samples capped at 1M, dataset registry name validation (no path separators/null bytes), registry path traversal protection, loss_watchdog threshold le=100 + patience le=1000, freeze_layers le=1000
|
||||
- **v0.24.1**: AWQ/GPTQ output path traversal validation moved before import check (previously unreachable when autoawq/auto-gptq not installed), Windows Unicode fix for Rich console output (replaced non-ASCII symbols with ASCII equivalents)
|
||||
- **v0.24.2**: Chat proxy SSRF protection (localhost-only HTTP, HTTPS for remote), chat proxy max_tokens capped at 16384 + temperature/top_p bounded, chat proxy Bearer token auth required, XSS prevention (HTML-escape before markdown render), runs compare max 5 runs, config from-form validates via load_config_from_string, SSE read endpoints no auth (GET)
|
||||
|
||||
## Security Scanning
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.24.1"
|
||||
version = "0.24.2"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.24.1"
|
||||
__version__ = "0.24.2"
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
"""FastAPI application for Soup Web UI."""
|
||||
|
||||
import json as json_mod
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
|
|
@ -7,6 +8,7 @@ import subprocess
|
|||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
|
@ -99,6 +101,43 @@ def create_app(host: str = "127.0.0.1", port: int = 7860):
|
|||
finally:
|
||||
tracker.close()
|
||||
|
||||
@app.get("/api/runs/compare")
|
||||
def compare_runs(ids: str = Query(default="")):
|
||||
"""Compare metrics for multiple runs."""
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
||||
if not ids or not ids.strip():
|
||||
raise HTTPException(status_code=400, detail="ids parameter required")
|
||||
|
||||
run_ids = [rid.strip() for rid in ids.split(",") if rid.strip()]
|
||||
if len(run_ids) > 5:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="Maximum 5 runs per comparison"
|
||||
)
|
||||
if not run_ids:
|
||||
raise HTTPException(status_code=400, detail="ids parameter required")
|
||||
|
||||
tracker = ExperimentTracker()
|
||||
try:
|
||||
result = []
|
||||
for rid in run_ids:
|
||||
run_info = tracker.get_run(rid)
|
||||
metrics = tracker.get_metrics(rid)
|
||||
config = {}
|
||||
if run_info and run_info.get("config_json"):
|
||||
try:
|
||||
config = json_mod.loads(run_info["config_json"])
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
result.append({
|
||||
"run_id": rid,
|
||||
"config": config,
|
||||
"metrics": metrics,
|
||||
})
|
||||
return {"runs": result}
|
||||
finally:
|
||||
tracker.close()
|
||||
|
||||
@app.get("/api/runs/{run_id}")
|
||||
def get_run(run_id: str):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
|
@ -300,6 +339,352 @@ def create_app(host: str = "127.0.0.1", port: int = 7860):
|
|||
"sample": sample,
|
||||
}
|
||||
|
||||
# --- Training Live Monitor (SSE) ---
|
||||
|
||||
@app.get("/api/train/logs")
|
||||
def stream_training_logs(request: Request):
|
||||
"""SSE endpoint streaming training log lines in real time."""
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
last_event_id = request.headers.get("Last-Event-ID")
|
||||
skip_count = int(last_event_id) + 1 if last_event_id else 0
|
||||
|
||||
def _generate_log_events():
|
||||
line_index = 0
|
||||
proc = _train_process
|
||||
if proc is None:
|
||||
yield "event: done\ndata: {}\n\n"
|
||||
return
|
||||
|
||||
try:
|
||||
for raw_line in proc.stdout:
|
||||
if isinstance(raw_line, bytes):
|
||||
raw_line = raw_line.decode("utf-8", errors="replace")
|
||||
text = raw_line.rstrip("\n\r")
|
||||
if line_index < skip_count:
|
||||
line_index += 1
|
||||
continue
|
||||
data = json_mod.dumps({"line": text, "id": line_index})
|
||||
yield f"id: {line_index}\ndata: {data}\n\n"
|
||||
line_index += 1
|
||||
except (ValueError, OSError):
|
||||
pass
|
||||
|
||||
yield "event: done\ndata: {}\n\n"
|
||||
|
||||
return StreamingResponse(
|
||||
_generate_log_events(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
@app.get("/api/train/metrics/live")
|
||||
def stream_live_metrics(
|
||||
request: Request,
|
||||
run_id: Optional[str] = Query(default=None),
|
||||
):
|
||||
"""SSE endpoint streaming new metrics as they're logged."""
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
def _generate_metrics_events():
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
||||
proc = _train_process
|
||||
if proc is None and run_id is None:
|
||||
yield "event: done\ndata: {}\n\n"
|
||||
return
|
||||
|
||||
last_step = -1
|
||||
max_polls = 3 # For tests: limit poll cycles when process done
|
||||
polls_since_new = 0
|
||||
|
||||
while True:
|
||||
tracker = ExperimentTracker()
|
||||
try:
|
||||
if run_id:
|
||||
metrics = tracker.get_metrics(run_id)
|
||||
else:
|
||||
yield "event: done\ndata: {}\n\n"
|
||||
return
|
||||
finally:
|
||||
tracker.close()
|
||||
|
||||
new_metrics = [
|
||||
m for m in metrics if m.get("step", 0) > last_step
|
||||
]
|
||||
if new_metrics:
|
||||
for m_row in new_metrics:
|
||||
data = json_mod.dumps(m_row, default=str)
|
||||
yield f"data: {data}\n\n"
|
||||
last_step = max(
|
||||
m.get("step", 0) for m in new_metrics
|
||||
)
|
||||
polls_since_new = 0
|
||||
else:
|
||||
polls_since_new += 1
|
||||
|
||||
# Check if training is still running
|
||||
proc = _train_process
|
||||
if proc is None or proc.poll() is not None:
|
||||
if polls_since_new >= 1:
|
||||
yield "event: done\ndata: {}\n\n"
|
||||
return
|
||||
|
||||
# Yield heartbeat
|
||||
yield ":heartbeat\n\n"
|
||||
|
||||
if polls_since_new >= max_polls:
|
||||
yield "event: done\ndata: {}\n\n"
|
||||
return
|
||||
|
||||
time.sleep(0.1) # Short poll for tests
|
||||
|
||||
return StreamingResponse(
|
||||
_generate_metrics_events(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
@app.get("/api/train/progress")
|
||||
def train_progress(
|
||||
run_id: Optional[str] = Query(default=None),
|
||||
):
|
||||
"""Return current training progress snapshot."""
|
||||
proc = _train_process
|
||||
is_running = proc is not None and proc.poll() is None
|
||||
|
||||
if not is_running and run_id is None:
|
||||
return {"running": False, "current_step": 0, "run_id": None}
|
||||
|
||||
if run_id:
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
||||
tracker = ExperimentTracker()
|
||||
try:
|
||||
metrics = tracker.get_metrics(run_id)
|
||||
current_step = metrics[-1]["step"] if metrics else 0
|
||||
finally:
|
||||
tracker.close()
|
||||
|
||||
return {
|
||||
"running": is_running,
|
||||
"current_step": current_step,
|
||||
"run_id": run_id,
|
||||
}
|
||||
|
||||
return {"running": is_running, "current_step": 0, "run_id": None}
|
||||
|
||||
# --- Config Builder ---
|
||||
|
||||
@app.get("/api/config/schema")
|
||||
def config_schema():
|
||||
"""Return config schema as JSON for form generation."""
|
||||
from soup_cli.config.schema import (
|
||||
DataConfig,
|
||||
LoraConfig,
|
||||
SoupConfig,
|
||||
TrainingConfig,
|
||||
)
|
||||
|
||||
def _extract_field_info(model_cls):
|
||||
"""Extract field metadata from a Pydantic model."""
|
||||
result = {}
|
||||
for name, field_info in model_cls.model_fields.items():
|
||||
info = {"type": "string", "required": field_info.is_required()}
|
||||
|
||||
# Get default value
|
||||
if field_info.default is not None:
|
||||
info["default"] = field_info.default
|
||||
|
||||
# Get type annotation
|
||||
annotation = field_info.annotation
|
||||
if annotation is not None:
|
||||
ann_str = str(annotation)
|
||||
if "int" in ann_str:
|
||||
info["type"] = "integer"
|
||||
elif "float" in ann_str:
|
||||
info["type"] = "number"
|
||||
elif "bool" in ann_str:
|
||||
info["type"] = "boolean"
|
||||
|
||||
# Check for Literal (enum) types
|
||||
origin = getattr(annotation, "__origin__", None)
|
||||
if origin is type(None):
|
||||
pass
|
||||
args = getattr(annotation, "__args__", None)
|
||||
if args and all(isinstance(a, str) for a in args):
|
||||
info["type"] = "enum"
|
||||
info["options"] = list(args)
|
||||
|
||||
# Get constraints from metadata
|
||||
for meta in (field_info.metadata or []):
|
||||
if hasattr(meta, "ge"):
|
||||
info["ge"] = meta.ge
|
||||
if hasattr(meta, "le"):
|
||||
info["le"] = meta.le
|
||||
|
||||
result[name] = info
|
||||
return result
|
||||
|
||||
schema = _extract_field_info(SoupConfig)
|
||||
schema["data"] = _extract_field_info(DataConfig)
|
||||
schema["training"] = _extract_field_info(TrainingConfig)
|
||||
schema["training"]["lora"] = _extract_field_info(LoraConfig)
|
||||
return schema
|
||||
|
||||
@app.get("/api/recipes")
|
||||
def list_recipes():
|
||||
"""Return recipe catalog as JSON."""
|
||||
from soup_cli.recipes.catalog import RECIPES
|
||||
|
||||
recipes_list = []
|
||||
for name, meta in RECIPES.items():
|
||||
recipes_list.append({
|
||||
"name": name,
|
||||
"model": meta.model,
|
||||
"task": meta.task,
|
||||
"description": meta.description,
|
||||
"tags": list(meta.tags) if hasattr(meta, "tags") else [],
|
||||
"yaml": meta.yaml_str,
|
||||
})
|
||||
return {"recipes": recipes_list}
|
||||
|
||||
@app.post("/api/config/from-form", dependencies=[Depends(_verify_token)])
|
||||
def form_to_yaml(body: dict):
|
||||
"""Convert form field values to validated YAML string."""
|
||||
import yaml
|
||||
|
||||
from soup_cli.config.loader import load_config_from_string
|
||||
|
||||
# Build YAML from form values
|
||||
config_dict = {}
|
||||
for key, val in body.items():
|
||||
if val is not None and val != "" and val != {}:
|
||||
config_dict[key] = val
|
||||
|
||||
try:
|
||||
yaml_str = yaml.dump(
|
||||
config_dict, default_flow_style=False, sort_keys=False
|
||||
)
|
||||
# Validate
|
||||
load_config_from_string(yaml_str)
|
||||
return {"yaml": yaml_str}
|
||||
except Exception as exc:
|
||||
return {"error": str(exc)}
|
||||
|
||||
# --- Chat Proxy ---
|
||||
|
||||
class ChatRequest(PydanticBaseModel):
|
||||
"""Request body for chat send."""
|
||||
messages: list
|
||||
endpoint: str
|
||||
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
|
||||
max_tokens: int = Field(default=512, ge=1, le=16384)
|
||||
top_p: float = Field(default=0.9, ge=0.0, le=1.0)
|
||||
adapter: Optional[str] = None
|
||||
|
||||
@app.post("/api/chat/send", dependencies=[Depends(_verify_token)])
|
||||
def chat_send(req: ChatRequest):
|
||||
"""SSE proxy endpoint streaming chat completions."""
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
# Validate messages
|
||||
if not req.messages:
|
||||
raise HTTPException(status_code=400, detail="messages cannot be empty")
|
||||
|
||||
# SSRF protection: localhost-only HTTP, HTTPS for remote
|
||||
parsed = urlparse(req.endpoint)
|
||||
if parsed.scheme == "http":
|
||||
host = parsed.hostname or ""
|
||||
if host not in ("localhost", "127.0.0.1", "0.0.0.0", "::1"):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="HTTP only allowed for localhost endpoints",
|
||||
)
|
||||
elif parsed.scheme != "https":
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Only HTTP (localhost) or HTTPS endpoints allowed",
|
||||
)
|
||||
|
||||
# Validate bounds
|
||||
if req.max_tokens > 16384:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="max_tokens exceeds 16384 cap"
|
||||
)
|
||||
if req.temperature < 0.0 or req.temperature > 2.0:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="temperature must be 0.0-2.0"
|
||||
)
|
||||
if req.top_p < 0.0 or req.top_p > 1.0:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="top_p must be 0.0-1.0"
|
||||
)
|
||||
|
||||
def _stream_chat():
|
||||
import httpx
|
||||
|
||||
url = req.endpoint.rstrip("/") + "/v1/chat/completions"
|
||||
payload = {
|
||||
"messages": req.messages,
|
||||
"max_tokens": req.max_tokens,
|
||||
"temperature": req.temperature,
|
||||
"top_p": req.top_p,
|
||||
"stream": True,
|
||||
}
|
||||
if req.adapter:
|
||||
payload["model"] = req.adapter
|
||||
|
||||
try:
|
||||
with httpx.stream(
|
||||
"POST", url,
|
||||
json=payload,
|
||||
headers={"Content-Type": "application/json"},
|
||||
timeout=120.0,
|
||||
) as resp:
|
||||
for line in resp.iter_lines():
|
||||
if line.startswith("data: "):
|
||||
data_str = line[6:]
|
||||
if data_str.strip() == "[DONE]":
|
||||
yield "data: {\"done\": true}\n\n"
|
||||
return
|
||||
try:
|
||||
parsed_data = json_mod.loads(data_str)
|
||||
delta = (
|
||||
parsed_data.get("choices", [{}])[0]
|
||||
.get("delta", {})
|
||||
.get("content", "")
|
||||
)
|
||||
if delta:
|
||||
out = json_mod.dumps({"delta": delta})
|
||||
yield f"data: {out}\n\n"
|
||||
except (ValueError, IndexError, KeyError):
|
||||
pass
|
||||
yield "data: {\"done\": true}\n\n"
|
||||
except Exception as exc:
|
||||
logger.warning("Chat proxy error: %s", exc)
|
||||
err_msg = json_mod.dumps(
|
||||
{"error": "Connection failed"}
|
||||
)
|
||||
yield f"data: {err_msg}\n\n"
|
||||
|
||||
return StreamingResponse(
|
||||
_stream_chat(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
# --- Health ---
|
||||
|
||||
@app.get("/api/health")
|
||||
|
|
|
|||
|
|
@ -187,9 +187,10 @@ async function showRunDetail(runId) {
|
|||
body.innerHTML = '<div style="text-align:center;padding:2rem;color:var(--text-dim)">Loading...</div>';
|
||||
|
||||
try {
|
||||
const [run, metricsResp] = await Promise.all([
|
||||
const [run, metricsResp, evalResp] = await Promise.all([
|
||||
api(`/api/runs/${runId}`),
|
||||
api(`/api/runs/${runId}/metrics`),
|
||||
api(`/api/runs/${runId}/eval`),
|
||||
]);
|
||||
|
||||
const config = run.config_json ? JSON.parse(run.config_json) : {};
|
||||
|
|
@ -232,16 +233,48 @@ async function showRunDetail(runId) {
|
|||
</div>
|
||||
|
||||
${metrics.length > 0 ? `
|
||||
<div class="card">
|
||||
<div class="card-title">Loss Curve</div>
|
||||
<div class="chart-container">
|
||||
<canvas id="loss-chart"></canvas>
|
||||
<div class="chart-grid">
|
||||
<div class="card">
|
||||
<div class="card-title">Loss</div>
|
||||
<div class="chart-container"><canvas id="loss-chart"></canvas></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-title">Learning Rate</div>
|
||||
<div class="chart-container"><canvas id="lr-chart"></canvas></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-title">Gradient Norm</div>
|
||||
<div class="chart-container"><canvas id="gradnorm-chart"></canvas></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-title">Throughput</div>
|
||||
<div class="chart-container"><canvas id="speed-chart"></canvas></div>
|
||||
</div>
|
||||
</div>
|
||||
${metrics.some(m => m.gpu_mem) ? `
|
||||
<div class="card">
|
||||
<div class="card-title">GPU Memory</div>
|
||||
<div class="chart-container"><canvas id="gpumem-chart"></canvas></div>
|
||||
</div>
|
||||
` : ''}
|
||||
` : ''}
|
||||
|
||||
${evalResp.eval_results && evalResp.eval_results.length > 0 ? `
|
||||
<div class="card">
|
||||
<div class="card-title">Learning Rate</div>
|
||||
<div class="chart-container">
|
||||
<canvas id="lr-chart"></canvas>
|
||||
<div class="card-title">Eval Results</div>
|
||||
<div class="table-wrap">
|
||||
<table class="eval-table">
|
||||
<thead><tr><th>Benchmark</th><th>Score</th><th>Details</th></tr></thead>
|
||||
<tbody>
|
||||
${evalResp.eval_results.map(er => `
|
||||
<tr>
|
||||
<td>${er.benchmark}</td>
|
||||
<td>${typeof er.score === 'number' ? er.score.toFixed(4) : er.score}</td>
|
||||
<td><code style="font-size:0.75rem">${er.details_json ? String(er.details_json).substring(0, 100) : '-'}</code></td>
|
||||
</tr>
|
||||
`).join('')}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
` : ''}
|
||||
|
|
@ -264,6 +297,24 @@ function renderCharts(metrics) {
|
|||
const steps = metrics.map(m => m.step);
|
||||
const losses = metrics.map(m => m.loss);
|
||||
const lrs = metrics.map(m => m.lr);
|
||||
const gradNorms = metrics.map(m => m.grad_norm || 0);
|
||||
const speeds = metrics.map(m => m.speed || 0);
|
||||
|
||||
const chartOpts = (yLabel) => ({
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
plugins: { legend: { display: false } },
|
||||
scales: {
|
||||
x: { title: { display: true, text: 'Step', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
|
||||
y: { title: { display: true, text: yLabel, color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
|
||||
},
|
||||
});
|
||||
|
||||
const makeDataset = (label, data, color) => ({
|
||||
label, data, borderColor: color,
|
||||
backgroundColor: color.replace(')', ', 0.1)').replace('rgb', 'rgba'),
|
||||
fill: true, tension: 0.3, pointRadius: 0, borderWidth: 2,
|
||||
});
|
||||
|
||||
// Loss chart
|
||||
const lossCtx = document.getElementById('loss-chart');
|
||||
|
|
@ -271,28 +322,8 @@ function renderCharts(metrics) {
|
|||
if (lossChart) lossChart.destroy();
|
||||
lossChart = new Chart(lossCtx, {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels: steps,
|
||||
datasets: [{
|
||||
label: 'Loss',
|
||||
data: losses,
|
||||
borderColor: '#c0512d',
|
||||
backgroundColor: 'rgba(192, 81, 45, 0.1)',
|
||||
fill: true,
|
||||
tension: 0.3,
|
||||
pointRadius: 0,
|
||||
borderWidth: 2,
|
||||
}],
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
plugins: { legend: { display: false } },
|
||||
scales: {
|
||||
x: { title: { display: true, text: 'Step', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
|
||||
y: { title: { display: true, text: 'Loss', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
|
||||
},
|
||||
},
|
||||
data: { labels: steps, datasets: [makeDataset('Loss', losses, 'rgb(192, 81, 45)')] },
|
||||
options: chartOpts('Loss'),
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -301,28 +332,43 @@ function renderCharts(metrics) {
|
|||
if (lrCtx) {
|
||||
new Chart(lrCtx, {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels: steps,
|
||||
datasets: [{
|
||||
label: 'Learning Rate',
|
||||
data: lrs,
|
||||
borderColor: '#e8975a',
|
||||
backgroundColor: 'rgba(232, 151, 90, 0.1)',
|
||||
fill: true,
|
||||
tension: 0.3,
|
||||
pointRadius: 0,
|
||||
borderWidth: 2,
|
||||
}],
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
plugins: { legend: { display: false } },
|
||||
scales: {
|
||||
x: { title: { display: true, text: 'Step', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
|
||||
y: { title: { display: true, text: 'LR', color: '#a09088' }, ticks: { color: '#a09088' }, grid: { color: 'rgba(58,48,64,0.5)' } },
|
||||
},
|
||||
},
|
||||
data: { labels: steps, datasets: [makeDataset('LR', lrs, 'rgb(232, 151, 90)')] },
|
||||
options: chartOpts('LR'),
|
||||
});
|
||||
}
|
||||
|
||||
// Gradient Norm chart
|
||||
const gnCtx = document.getElementById('gradnorm-chart');
|
||||
if (gnCtx) {
|
||||
new Chart(gnCtx, {
|
||||
type: 'line',
|
||||
data: { labels: steps, datasets: [makeDataset('Grad Norm', gradNorms, 'rgb(100, 180, 220)')] },
|
||||
options: chartOpts('Grad Norm'),
|
||||
});
|
||||
}
|
||||
|
||||
// Speed chart
|
||||
const spCtx = document.getElementById('speed-chart');
|
||||
if (spCtx) {
|
||||
new Chart(spCtx, {
|
||||
type: 'line',
|
||||
data: { labels: steps, datasets: [makeDataset('Speed', speeds, 'rgb(120, 200, 130)')] },
|
||||
options: chartOpts('Tokens/sec'),
|
||||
});
|
||||
}
|
||||
|
||||
// GPU Memory chart (optional — parse numeric values from strings like "4.2GB")
|
||||
const gmCtx = document.getElementById('gpumem-chart');
|
||||
if (gmCtx) {
|
||||
const gpuVals = metrics.map(m => {
|
||||
if (!m.gpu_mem) return 0;
|
||||
const match = String(m.gpu_mem).match(/([\d.]+)/);
|
||||
return match ? parseFloat(match[1]) : 0;
|
||||
});
|
||||
new Chart(gmCtx, {
|
||||
type: 'line',
|
||||
data: { labels: steps, datasets: [makeDataset('GPU Mem', gpuVals, 'rgb(200, 130, 200)')] },
|
||||
options: chartOpts('GPU Memory (GB)'),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
@ -334,10 +380,12 @@ function closeModal() {
|
|||
// --- New Training Page ---
|
||||
async function loadTrainingPage() {
|
||||
try {
|
||||
const [templatesResp, statusResp] = await Promise.all([
|
||||
const [templatesResp, statusResp, recipesResp] = await Promise.all([
|
||||
api('/api/templates'),
|
||||
api('/api/train/status'),
|
||||
api('/api/recipes'),
|
||||
]);
|
||||
window._recipes = recipesResp.recipes || [];
|
||||
renderTrainingPage(templatesResp.templates, statusResp);
|
||||
} catch (err) {
|
||||
document.getElementById('training-content').innerHTML =
|
||||
|
|
@ -353,12 +401,22 @@ function renderTrainingPage(templates, status) {
|
|||
<div class="grid-2">
|
||||
<div>
|
||||
<div class="card">
|
||||
<div class="card-title">Template</div>
|
||||
<div class="form-group">
|
||||
<select id="template-select" onchange="loadTemplate()">
|
||||
<option value="">-- Select a template --</option>
|
||||
${templateNames.map(t => `<option value="${t}">${t}</option>`).join('')}
|
||||
</select>
|
||||
<div class="card-title">Template / Recipe</div>
|
||||
<div class="grid-2" style="margin-bottom:0">
|
||||
<div class="form-group" style="margin-bottom:0">
|
||||
<label class="form-label" style="font-size:0.8rem">Template</label>
|
||||
<select id="template-select" onchange="loadTemplate()">
|
||||
<option value="">-- Template --</option>
|
||||
${templateNames.map(t => `<option value="${t}">${t}</option>`).join('')}
|
||||
</select>
|
||||
</div>
|
||||
<div class="form-group" style="margin-bottom:0">
|
||||
<label class="form-label" style="font-size:0.8rem">Recipe</label>
|
||||
<select id="recipe-select" onchange="loadRecipe()">
|
||||
<option value="">-- Recipe --</option>
|
||||
${(window._recipes || []).map(r => `<option value="${r.name}">${r.name} (${r.task})</option>`).join('')}
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
|
@ -412,6 +470,16 @@ function loadTemplate() {
|
|||
}
|
||||
}
|
||||
|
||||
function loadRecipe() {
|
||||
const sel = document.getElementById('recipe-select');
|
||||
const editor = document.getElementById('config-editor');
|
||||
if (!sel.value || !window._recipes) return;
|
||||
const recipe = window._recipes.find(r => r.name === sel.value);
|
||||
if (recipe && recipe.yaml) {
|
||||
editor.value = recipe.yaml;
|
||||
}
|
||||
}
|
||||
|
||||
async function validateConfig() {
|
||||
const yaml = document.getElementById('config-editor').value;
|
||||
const statusEl = document.getElementById('config-status');
|
||||
|
|
@ -524,8 +592,9 @@ function renderDataResults(data) {
|
|||
}
|
||||
|
||||
// --- Model Chat ---
|
||||
let chatAbortController = null;
|
||||
|
||||
function loadChatPage() {
|
||||
// Just ensure the page renders with current messages
|
||||
renderChatMessages();
|
||||
}
|
||||
|
||||
|
|
@ -546,7 +615,7 @@ function renderChatMessages() {
|
|||
container.innerHTML = chatMessages.map(msg => `
|
||||
<div class="chat-msg ${msg.role}">
|
||||
<div class="chat-msg-role">${msg.role}</div>
|
||||
<div class="chat-msg-content">${escapeHtml(msg.content)}</div>
|
||||
<div class="chat-msg-content chat-markdown">${msg.role === 'assistant' ? renderMarkdown(msg.content) : escapeHtml(msg.content)}</div>
|
||||
</div>
|
||||
`).join('');
|
||||
|
||||
|
|
@ -559,42 +628,137 @@ function escapeHtml(text) {
|
|||
return div.innerHTML;
|
||||
}
|
||||
|
||||
function renderMarkdown(text) {
|
||||
let html = escapeHtml(text);
|
||||
// Code blocks (``` ... ```)
|
||||
html = html.replace(/```(\w*)\n([\s\S]*?)```/g, '<pre class="code-block"><code>$2</code></pre>');
|
||||
// Inline code
|
||||
html = html.replace(/`([^`]+)`/g, '<code>$1</code>');
|
||||
// Bold
|
||||
html = html.replace(/\*\*(.+?)\*\*/g, '<strong>$1</strong>');
|
||||
// Italic
|
||||
html = html.replace(/\*(.+?)\*/g, '<em>$1</em>');
|
||||
// List items
|
||||
html = html.replace(/^- (.+)$/gm, '<li>$1</li>');
|
||||
// Line breaks
|
||||
html = html.replace(/\n/g, '<br>');
|
||||
return html;
|
||||
}
|
||||
|
||||
async function sendChatMessage() {
|
||||
const input = document.getElementById('chat-input');
|
||||
const serverUrl = document.getElementById('chat-server').value.trim();
|
||||
const msg = input.value.trim();
|
||||
if (!msg) return;
|
||||
if (!serverUrl) { alert('Enter a server URL (e.g., http://localhost:8000)'); return; }
|
||||
if (!serverUrl) { alert('Enter a server URL'); return; }
|
||||
|
||||
// Build messages with optional system prompt
|
||||
const systemPrompt = document.getElementById('chat-system')?.value?.trim();
|
||||
const allMessages = [];
|
||||
if (systemPrompt) {
|
||||
allMessages.push({ role: 'system', content: systemPrompt });
|
||||
}
|
||||
chatMessages.push({ role: 'user', content: msg });
|
||||
allMessages.push(...chatMessages.map(m => ({ role: m.role, content: m.content })));
|
||||
input.value = '';
|
||||
renderChatMessages();
|
||||
|
||||
// Show typing indicator, switch buttons
|
||||
const typing = document.getElementById('typing-indicator');
|
||||
const sendBtn = document.getElementById('chat-send-btn');
|
||||
const cancelBtn = document.getElementById('chat-cancel-btn');
|
||||
if (typing) typing.style.display = 'flex';
|
||||
if (sendBtn) sendBtn.style.display = 'none';
|
||||
if (cancelBtn) cancelBtn.style.display = 'inline-flex';
|
||||
|
||||
chatAbortController = new AbortController();
|
||||
|
||||
const temperature = parseFloat(document.getElementById('chat-temperature')?.value || '0.7');
|
||||
const maxTokens = parseInt(document.getElementById('chat-max-tokens')?.value || '512');
|
||||
const topP = parseFloat(document.getElementById('chat-top-p')?.value || '0.9');
|
||||
const adapter = document.getElementById('chat-adapter')?.value?.trim() || undefined;
|
||||
|
||||
try {
|
||||
const resp = await fetch(serverUrl + '/v1/chat/completions', {
|
||||
const resp = await fetch(API + '/api/chat/send', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer ' + (window._authToken || ''),
|
||||
},
|
||||
body: JSON.stringify({
|
||||
messages: chatMessages.map(m => ({ role: m.role, content: m.content })),
|
||||
max_tokens: 512,
|
||||
temperature: 0.7,
|
||||
messages: allMessages,
|
||||
endpoint: serverUrl,
|
||||
temperature: temperature,
|
||||
max_tokens: maxTokens,
|
||||
top_p: topP,
|
||||
adapter: adapter,
|
||||
}),
|
||||
signal: chatAbortController.signal,
|
||||
});
|
||||
const data = await resp.json();
|
||||
const reply = data.choices[0].message.content;
|
||||
chatMessages.push({ role: 'assistant', content: reply });
|
||||
renderChatMessages();
|
||||
|
||||
// Stream tokens
|
||||
const reader = resp.body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let assistantMsg = '';
|
||||
chatMessages.push({ role: 'assistant', content: '' });
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
const chunk = decoder.decode(value);
|
||||
const lines = chunk.split('\n');
|
||||
for (const line of lines) {
|
||||
if (line.startsWith('data: ')) {
|
||||
try {
|
||||
const data = JSON.parse(line.slice(6));
|
||||
if (data.done) break;
|
||||
if (data.delta) {
|
||||
assistantMsg += data.delta;
|
||||
chatMessages[chatMessages.length - 1].content = assistantMsg;
|
||||
renderChatMessages();
|
||||
}
|
||||
if (data.error) {
|
||||
assistantMsg += '[Error: ' + data.error + ']';
|
||||
chatMessages[chatMessages.length - 1].content = assistantMsg;
|
||||
renderChatMessages();
|
||||
}
|
||||
} catch (parseErr) { /* ignore malformed lines */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
chatMessages.push({ role: 'assistant', content: `[Error: ${err.message}]` });
|
||||
renderChatMessages();
|
||||
if (err.name !== 'AbortError') {
|
||||
chatMessages.push({ role: 'assistant', content: `[Error: ${err.message}]` });
|
||||
renderChatMessages();
|
||||
}
|
||||
} finally {
|
||||
chatAbortController = null;
|
||||
if (typing) typing.style.display = 'none';
|
||||
if (sendBtn) sendBtn.style.display = 'inline-flex';
|
||||
if (cancelBtn) cancelBtn.style.display = 'none';
|
||||
}
|
||||
}
|
||||
|
||||
function cancelChat() {
|
||||
if (chatAbortController) chatAbortController.abort();
|
||||
}
|
||||
|
||||
function clearChat() {
|
||||
chatMessages = [];
|
||||
renderChatMessages();
|
||||
}
|
||||
|
||||
function exportChat() {
|
||||
if (chatMessages.length === 0) return;
|
||||
const blob = new Blob([JSON.stringify(chatMessages, null, 2)], { type: 'application/json' });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = 'chat-export.json';
|
||||
a.click();
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
|
||||
function handleChatKey(event) {
|
||||
if (event.key === 'Enter' && !event.shiftKey) {
|
||||
event.preventDefault();
|
||||
|
|
@ -602,6 +766,79 @@ function handleChatKey(event) {
|
|||
}
|
||||
}
|
||||
|
||||
// --- Training Live Monitor (SSE) ---
|
||||
let logEventSource = null;
|
||||
let metricsEventSource = null;
|
||||
const LOG_MAX_LINES = 500;
|
||||
|
||||
function connectTrainingSSE() {
|
||||
disconnectTrainingSSE();
|
||||
|
||||
const logPanel = document.getElementById('train-log-panel');
|
||||
const progressPanel = document.getElementById('train-progress');
|
||||
const liveBadge = document.getElementById('live-badge');
|
||||
if (!logPanel) return;
|
||||
|
||||
logPanel.style.display = 'block';
|
||||
progressPanel.style.display = 'block';
|
||||
if (liveBadge) liveBadge.style.display = 'inline-flex';
|
||||
|
||||
// Connect to log SSE
|
||||
logEventSource = new EventSource(API + '/api/train/logs');
|
||||
logEventSource.onmessage = function(ev) {
|
||||
const data = JSON.parse(ev.data);
|
||||
appendLogLine(data.line);
|
||||
};
|
||||
logEventSource.addEventListener('done', function() {
|
||||
appendLogLine('[Training finished]');
|
||||
disconnectTrainingSSE();
|
||||
});
|
||||
logEventSource.onerror = function() {
|
||||
setTimeout(function() {
|
||||
if (logEventSource && logEventSource.readyState === EventSource.CLOSED) {
|
||||
disconnectTrainingSSE();
|
||||
}
|
||||
}, 5000);
|
||||
};
|
||||
}
|
||||
|
||||
function disconnectTrainingSSE() {
|
||||
if (logEventSource) { logEventSource.close(); logEventSource = null; }
|
||||
if (metricsEventSource) { metricsEventSource.close(); metricsEventSource = null; }
|
||||
const liveBadge = document.getElementById('live-badge');
|
||||
if (liveBadge) liveBadge.style.display = 'none';
|
||||
}
|
||||
|
||||
function appendLogLine(text) {
|
||||
const output = document.getElementById('log-output');
|
||||
if (!output) return;
|
||||
output.textContent += text + '\n';
|
||||
// Ring buffer: trim to max lines
|
||||
const lines = output.textContent.split('\n');
|
||||
if (lines.length > LOG_MAX_LINES) {
|
||||
output.textContent = lines.slice(lines.length - LOG_MAX_LINES).join('\n');
|
||||
}
|
||||
// Auto-scroll
|
||||
const autoScroll = document.getElementById('log-autoscroll');
|
||||
if (autoScroll && autoScroll.checked) {
|
||||
output.scrollTop = output.scrollHeight;
|
||||
}
|
||||
}
|
||||
|
||||
function updateProgressBar(step, total, elapsed, eta) {
|
||||
const fill = document.getElementById('progress-fill');
|
||||
const stepEl = document.getElementById('progress-step');
|
||||
const elapsedEl = document.getElementById('progress-elapsed');
|
||||
const etaEl = document.getElementById('progress-eta');
|
||||
if (!fill) return;
|
||||
|
||||
const pct = total > 0 ? Math.min(100, (step / total) * 100) : 0;
|
||||
fill.style.width = pct.toFixed(1) + '%';
|
||||
if (stepEl) stepEl.textContent = 'Step ' + step + (total ? '/' + total : '');
|
||||
if (elapsedEl) elapsedEl.textContent = 'Elapsed: ' + formatDuration(elapsed);
|
||||
if (etaEl) etaEl.textContent = 'ETA: ' + formatDuration(eta);
|
||||
}
|
||||
|
||||
// --- Init ---
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
navigate('dashboard');
|
||||
|
|
|
|||
|
|
@ -51,10 +51,33 @@
|
|||
<div id="page-training" class="page">
|
||||
<div class="page-header">
|
||||
<h1 class="page-title">New Training</h1>
|
||||
<span id="live-badge" class="live-badge" style="display:none">Live</span>
|
||||
</div>
|
||||
<div id="training-content">
|
||||
<div style="text-align:center;padding:3rem;color:var(--text-dim)">Loading...</div>
|
||||
</div>
|
||||
<!-- Training Progress Bar -->
|
||||
<div id="train-progress" class="card" style="display:none">
|
||||
<div class="card-title">Training Progress</div>
|
||||
<div class="progress-bar">
|
||||
<div class="progress-bar-fill" id="progress-fill" style="width:0%"></div>
|
||||
</div>
|
||||
<div class="progress-info">
|
||||
<span id="progress-step">Step 0/0</span>
|
||||
<span id="progress-elapsed">Elapsed: --</span>
|
||||
<span id="progress-eta">ETA: --</span>
|
||||
</div>
|
||||
</div>
|
||||
<!-- Training Log Panel -->
|
||||
<div id="train-log-panel" class="card" style="display:none">
|
||||
<div class="card-title" style="display:flex;justify-content:space-between;align-items:center">
|
||||
Training Logs
|
||||
<label style="font-size:0.8rem;font-weight:normal">
|
||||
<input type="checkbox" id="log-autoscroll" checked> Auto-scroll
|
||||
</label>
|
||||
</div>
|
||||
<pre class="log-panel" id="log-output"></pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Data Explorer Page -->
|
||||
|
|
@ -88,16 +111,42 @@
|
|||
<div id="page-chat" class="page">
|
||||
<div class="page-header">
|
||||
<h1 class="page-title">Model Chat</h1>
|
||||
<button class="btn" onclick="clearChat()">Clear Chat</button>
|
||||
</div>
|
||||
<div class="card" style="margin-bottom:1rem">
|
||||
<div style="display:flex; gap:0.75rem; align-items:flex-end">
|
||||
<div class="form-group" style="flex:1; margin-bottom:0">
|
||||
<label class="form-label">Inference Server URL</label>
|
||||
<input type="text" id="chat-server" placeholder="http://localhost:8000" value="http://localhost:8000">
|
||||
</div>
|
||||
<div style="display:flex;gap:0.5rem">
|
||||
<button class="btn" onclick="exportChat()">Export</button>
|
||||
<button class="btn" onclick="clearChat()">Clear</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card chat-settings">
|
||||
<details>
|
||||
<summary style="cursor:pointer;font-weight:600;font-size:0.9rem">Chat Settings</summary>
|
||||
<div class="grid-2" style="margin-top:0.75rem">
|
||||
<div class="form-group">
|
||||
<label class="form-label">Server URL</label>
|
||||
<input type="text" id="chat-server" placeholder="http://localhost:8000" value="http://localhost:8000">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="form-label">Adapter</label>
|
||||
<input type="text" id="chat-adapter" placeholder="default">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="form-label">Temperature: <span id="temp-val">0.7</span></label>
|
||||
<input type="range" id="chat-temperature" min="0" max="2" step="0.1" value="0.7" oninput="document.getElementById('temp-val').textContent=this.value">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="form-label">Max Tokens: <span id="maxtok-val">512</span></label>
|
||||
<input type="range" id="chat-max-tokens" min="64" max="4096" step="64" value="512" oninput="document.getElementById('maxtok-val').textContent=this.value">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="form-label">Top-p: <span id="topp-val">0.9</span></label>
|
||||
<input type="range" id="chat-top-p" min="0" max="1" step="0.05" value="0.9" oninput="document.getElementById('topp-val').textContent=this.value">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="form-label">System Prompt</label>
|
||||
<textarea id="chat-system" rows="2" placeholder="You are a helpful assistant."></textarea>
|
||||
</div>
|
||||
</div>
|
||||
</details>
|
||||
</div>
|
||||
<div class="card chat-container">
|
||||
<div class="chat-messages" id="chat-messages">
|
||||
<div class="empty-state">
|
||||
|
|
@ -105,9 +154,13 @@
|
|||
<div class="empty-state-hint">Start a <code>soup serve</code> server and start chatting</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="typing-indicator" id="typing-indicator" style="display:none">
|
||||
<span></span><span></span><span></span>
|
||||
</div>
|
||||
<div class="chat-input-area">
|
||||
<input type="text" id="chat-input" placeholder="Type a message..." onkeydown="handleChatKey(event)">
|
||||
<button class="btn btn-primary" onclick="sendChatMessage()">Send</button>
|
||||
<button class="btn btn-primary" id="chat-send-btn" onclick="sendChatMessage()">Send</button>
|
||||
<button class="btn btn-danger" id="chat-cancel-btn" onclick="cancelChat()" style="display:none">Cancel</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -373,3 +373,177 @@ textarea {
|
|||
.nav-item { justify-content: center; padding: 0.75rem; }
|
||||
.grid-2, .grid-3 { grid-template-columns: 1fr; }
|
||||
}
|
||||
|
||||
/* --- Training Live Monitor --- */
|
||||
.log-panel {
|
||||
background: #0d0b10;
|
||||
color: #d0c8c0;
|
||||
font-family: 'Consolas', 'Courier New', monospace;
|
||||
font-size: 0.82rem;
|
||||
line-height: 1.5;
|
||||
max-height: 400px;
|
||||
overflow-y: auto;
|
||||
padding: 0.75rem;
|
||||
border-radius: 6px;
|
||||
border: 1px solid var(--border);
|
||||
white-space: pre-wrap;
|
||||
word-break: break-all;
|
||||
}
|
||||
|
||||
.progress-bar {
|
||||
width: 100%;
|
||||
height: 22px;
|
||||
background: var(--bg-input);
|
||||
border-radius: 11px;
|
||||
overflow: hidden;
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
.progress-bar-fill {
|
||||
height: 100%;
|
||||
background: linear-gradient(90deg, var(--accent), var(--accent-warm));
|
||||
border-radius: 11px;
|
||||
transition: width 0.4s ease;
|
||||
}
|
||||
|
||||
.progress-info {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
font-size: 0.82rem;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.live-badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.4rem;
|
||||
font-size: 0.78rem;
|
||||
font-weight: 600;
|
||||
color: #3ddc84;
|
||||
padding: 0.2rem 0.6rem;
|
||||
border-radius: 12px;
|
||||
background: rgba(61,220,132,0.12);
|
||||
}
|
||||
|
||||
.live-badge::before {
|
||||
content: "";
|
||||
display: inline-block;
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
background: #3ddc84;
|
||||
border-radius: 50%;
|
||||
animation: pulse-dot 1.5s ease-in-out infinite;
|
||||
}
|
||||
|
||||
@keyframes pulse-dot {
|
||||
0%, 100% { opacity: 1; transform: scale(1); }
|
||||
50% { opacity: 0.4; transform: scale(0.8); }
|
||||
}
|
||||
|
||||
/* --- Enhanced Metrics --- */
|
||||
.chart-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 1rem;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.chart-grid { grid-template-columns: 1fr; }
|
||||
}
|
||||
|
||||
.eval-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
}
|
||||
|
||||
.eval-table th, .eval-table td {
|
||||
text-align: left;
|
||||
padding: 0.5rem 0.75rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.eval-table th {
|
||||
color: var(--text-dim);
|
||||
font-size: 0.82rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
/* --- Chat Upgrade --- */
|
||||
.chat-settings details summary {
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.chat-settings details[open] summary {
|
||||
color: var(--text);
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
.typing-indicator {
|
||||
display: flex;
|
||||
gap: 4px;
|
||||
padding: 0.5rem 1rem;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.typing-indicator span {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
background: var(--text-dim);
|
||||
border-radius: 50%;
|
||||
animation: typing-bounce 1.2s ease-in-out infinite;
|
||||
}
|
||||
|
||||
.typing-indicator span:nth-child(2) { animation-delay: 0.15s; }
|
||||
.typing-indicator span:nth-child(3) { animation-delay: 0.3s; }
|
||||
|
||||
@keyframes typing-bounce {
|
||||
0%, 80%, 100% { transform: translateY(0); }
|
||||
40% { transform: translateY(-6px); }
|
||||
}
|
||||
|
||||
.code-block {
|
||||
background: #0d0b10;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
padding: 0.75rem;
|
||||
overflow-x: auto;
|
||||
font-size: 0.82rem;
|
||||
margin: 0.5rem 0;
|
||||
}
|
||||
|
||||
.chat-markdown code {
|
||||
background: rgba(192, 81, 45, 0.15);
|
||||
padding: 0.1em 0.3em;
|
||||
border-radius: 3px;
|
||||
font-size: 0.88em;
|
||||
}
|
||||
|
||||
.chat-markdown .code-block code {
|
||||
background: none;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
input[type="range"] {
|
||||
width: 100%;
|
||||
accent-color: var(--accent);
|
||||
}
|
||||
|
||||
/* --- Config Builder --- */
|
||||
.form-section {
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
padding: 0.75rem;
|
||||
margin-bottom: 0.75rem;
|
||||
}
|
||||
|
||||
.form-section summary {
|
||||
cursor: pointer;
|
||||
font-weight: 600;
|
||||
font-size: 0.9rem;
|
||||
color: var(--accent-warm);
|
||||
}
|
||||
|
||||
.form-section[open] summary {
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,287 @@
|
|||
"""Tests for Web UI Chat Upgrade — SSE proxy, SSRF validation, param bounds."""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _auth_headers():
|
||||
"""Return auth headers with the current UI token."""
|
||||
from soup_cli.ui.app import get_auth_token
|
||||
return {"Authorization": f"Bearer {get_auth_token()}"}
|
||||
|
||||
|
||||
class TestChatEndpointExists:
|
||||
"""Test /api/chat/send endpoint registration."""
|
||||
|
||||
def test_chat_send_endpoint_exists(self):
|
||||
"""Route /api/chat/send should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/chat/send" in routes
|
||||
|
||||
|
||||
class TestChatSSRFProtection:
|
||||
"""Test SSRF protection on chat endpoint."""
|
||||
|
||||
def test_rejects_non_localhost_http(self):
|
||||
"""Chat endpoint should reject non-localhost HTTP URLs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "http://evil.com:8000",
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_allows_localhost_http(self):
|
||||
"""Chat endpoint should allow localhost HTTP URLs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
# Will fail at connection level but not at validation
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "http://localhost:8000",
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
# Should not be 400 (SSRF rejection) — may be 502 or stream error
|
||||
assert response.status_code != 400
|
||||
|
||||
def test_allows_https_remote(self):
|
||||
"""Chat endpoint should allow HTTPS remote URLs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "https://api.example.com",
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
# Should not be 400 (SSRF rejection)
|
||||
assert response.status_code != 400
|
||||
|
||||
|
||||
class TestChatParamBounds:
|
||||
"""Test parameter validation on chat endpoint."""
|
||||
|
||||
def test_max_tokens_capped(self):
|
||||
"""max_tokens should be capped at 16384."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "http://localhost:8000",
|
||||
"max_tokens": 99999,
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
# Should be rejected (exceeds cap) — 422 from Pydantic or 400
|
||||
assert response.status_code in (400, 422)
|
||||
|
||||
def test_temperature_bounded(self):
|
||||
"""temperature should be bounded 0.0-2.0."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "http://localhost:8000",
|
||||
"temperature": 5.0,
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 400 or response.status_code == 422
|
||||
|
||||
def test_top_p_bounded(self):
|
||||
"""top_p should be bounded 0.0-1.0."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "http://localhost:8000",
|
||||
"top_p": 2.0,
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 400 or response.status_code == 422
|
||||
|
||||
|
||||
class TestChatInvalidScheme:
|
||||
"""Test rejection of invalid URL schemes."""
|
||||
|
||||
def test_rejects_ftp_scheme(self):
|
||||
"""Chat endpoint should reject ftp:// URLs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "ftp://evil.com/model",
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_rejects_file_scheme(self):
|
||||
"""Chat endpoint should reject file:// URLs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "file:///etc/passwd",
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
class TestChatAuth:
|
||||
"""Test chat endpoint auth requirements."""
|
||||
|
||||
def test_requires_auth(self):
|
||||
"""Chat send (POST) should require Bearer auth."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "http://localhost:8000",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestChatValidation:
|
||||
"""Test chat request validation."""
|
||||
|
||||
def test_empty_messages_rejected(self):
|
||||
"""Empty messages array should return 400."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [],
|
||||
"endpoint": "http://localhost:8000",
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_returns_event_stream(self):
|
||||
"""Chat endpoint should return text/event-stream content type."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
# Mock httpx to avoid real connection
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.iter_lines.return_value = iter([
|
||||
'data: {"choices":[{"delta":{"content":"Hi"}}]}',
|
||||
'data: [DONE]',
|
||||
])
|
||||
mock_response.__enter__ = lambda s: s
|
||||
mock_response.__exit__ = lambda s, *a: None
|
||||
|
||||
with patch("httpx.stream", return_value=mock_response):
|
||||
with client.stream(
|
||||
"POST",
|
||||
"/api/chat/send",
|
||||
json={
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
"endpoint": "http://localhost:8000",
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
) as resp:
|
||||
assert resp.status_code == 200
|
||||
assert "text/event-stream" in resp.headers["content-type"]
|
||||
|
|
@ -0,0 +1,316 @@
|
|||
"""Tests for Web UI Visual Config Builder — schema, recipes, form-to-YAML."""
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _auth_headers():
|
||||
"""Return auth headers with the current UI token."""
|
||||
from soup_cli.ui.app import get_auth_token
|
||||
return {"Authorization": f"Bearer {get_auth_token()}"}
|
||||
|
||||
|
||||
class TestConfigSchemaEndpoint:
|
||||
"""Test GET /api/config/schema endpoint."""
|
||||
|
||||
def test_schema_endpoint_exists(self):
|
||||
"""Route /api/config/schema should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/config/schema" in routes
|
||||
|
||||
def test_schema_returns_json(self):
|
||||
"""Schema endpoint should return valid JSON."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/config/schema")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert isinstance(data, dict)
|
||||
|
||||
def test_schema_includes_base_field(self):
|
||||
"""Schema should include 'base' as a required string field."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/config/schema")
|
||||
data = response.json()
|
||||
assert "base" in data
|
||||
assert data["base"]["type"] == "string"
|
||||
assert data["base"]["required"] is True
|
||||
|
||||
def test_schema_includes_task_enum(self):
|
||||
"""Schema should include 'task' with enum options."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/config/schema")
|
||||
data = response.json()
|
||||
assert "task" in data
|
||||
assert "options" in data["task"]
|
||||
assert "sft" in data["task"]["options"]
|
||||
assert "dpo" in data["task"]["options"]
|
||||
|
||||
def test_schema_includes_training_fields(self):
|
||||
"""Schema should include nested training fields."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/config/schema")
|
||||
data = response.json()
|
||||
assert "training" in data
|
||||
training = data["training"]
|
||||
assert "epochs" in training
|
||||
assert "lr" in training
|
||||
|
||||
def test_schema_includes_constraints(self):
|
||||
"""Schema should include ge/le constraints for bounded fields."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/config/schema")
|
||||
data = response.json()
|
||||
# Check that training.epochs has constraints or type info
|
||||
training = data.get("training", {})
|
||||
if "epochs" in training:
|
||||
assert training["epochs"]["type"] in ("integer", "number")
|
||||
|
||||
|
||||
class TestRecipesEndpoint:
|
||||
"""Test GET /api/recipes endpoint."""
|
||||
|
||||
def test_recipes_endpoint_exists(self):
|
||||
"""Route /api/recipes should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/recipes" in routes
|
||||
|
||||
def test_recipes_returns_catalog(self):
|
||||
"""Recipes endpoint should return all recipes."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/recipes")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "recipes" in data
|
||||
assert len(data["recipes"]) >= 20 # We have 29 recipes
|
||||
|
||||
def test_recipes_have_required_fields(self):
|
||||
"""Each recipe should have name, model, task fields."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/recipes")
|
||||
recipes = response.json()["recipes"]
|
||||
for recipe in recipes:
|
||||
assert "name" in recipe
|
||||
assert "model" in recipe
|
||||
assert "task" in recipe
|
||||
assert "yaml" in recipe
|
||||
|
||||
|
||||
class TestFormToYamlEndpoint:
|
||||
"""Test POST /api/config/from-form endpoint."""
|
||||
|
||||
def test_form_to_yaml_endpoint_exists(self):
|
||||
"""Route /api/config/from-form should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/config/from-form" in routes
|
||||
|
||||
def test_form_to_yaml_produces_valid_config(self):
|
||||
"""Form-to-YAML should produce a valid config string."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/config/from-form",
|
||||
json={
|
||||
"base": "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"task": "sft",
|
||||
"data": {"train": "./data.jsonl", "format": "alpaca"},
|
||||
"training": {"epochs": 3, "lr": 2e-4},
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "yaml" in data
|
||||
assert "base:" in data["yaml"]
|
||||
assert "meta-llama" in data["yaml"]
|
||||
|
||||
def test_form_to_yaml_validates(self):
|
||||
"""Form-to-YAML should validate via load_config_from_string."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/config/from-form",
|
||||
json={
|
||||
"base": "test-model",
|
||||
"data": {"train": "./data.jsonl"},
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "yaml" in data
|
||||
|
||||
def test_form_to_yaml_invalid_returns_error(self):
|
||||
"""Invalid form values should return error."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
# Missing base — should fail validation
|
||||
response = client.post(
|
||||
"/api/config/from-form",
|
||||
json={"task": "sft"},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code in (200, 400)
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
assert "error" in data
|
||||
|
||||
def test_form_to_yaml_requires_auth(self):
|
||||
"""Form-to-YAML (POST) should require auth."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/config/from-form",
|
||||
json={"base": "test"},
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
def test_form_to_yaml_special_chars_in_keys(self):
|
||||
"""Form values with special YAML characters should be handled safely."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/config/from-form",
|
||||
json={
|
||||
"base": "test: model\noutput: /etc",
|
||||
"data": {"train": "./data.jsonl"},
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# The result should either be valid YAML or an error — never injection
|
||||
if "yaml" in data:
|
||||
# The colon/newline in base value should be quoted by yaml.dump
|
||||
assert "base:" in data["yaml"]
|
||||
elif "error" in data:
|
||||
# Validation error is also acceptable
|
||||
pass
|
||||
|
||||
def test_form_to_yaml_with_lora(self):
|
||||
"""Form-to-YAML should handle nested lora config."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.post(
|
||||
"/api/config/from-form",
|
||||
json={
|
||||
"base": "test-model",
|
||||
"data": {"train": "./data.jsonl"},
|
||||
"training": {
|
||||
"epochs": 3,
|
||||
"lora": {"r": 16, "alpha": 32},
|
||||
},
|
||||
},
|
||||
headers=_auth_headers(),
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "yaml" in data
|
||||
assert "lora:" in data["yaml"] or "r:" in data["yaml"]
|
||||
|
|
@ -0,0 +1,445 @@
|
|||
"""Tests for Web UI Training Live Monitor — SSE endpoints and progress API."""
|
||||
|
||||
import os
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _auth_headers():
|
||||
"""Return auth headers with the current UI token."""
|
||||
from soup_cli.ui.app import get_auth_token
|
||||
return {"Authorization": f"Bearer {get_auth_token()}"}
|
||||
|
||||
|
||||
class TestTrainLogsSSE:
|
||||
"""Test GET /api/train/logs SSE endpoint."""
|
||||
|
||||
def test_logs_endpoint_exists(self):
|
||||
"""Route /api/train/logs should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/train/logs" in routes
|
||||
|
||||
def test_logs_returns_event_stream(self):
|
||||
"""SSE endpoint should return text/event-stream content type."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
# Simulate a running process with output
|
||||
mock_proc = MagicMock()
|
||||
mock_proc.poll.return_value = None
|
||||
mock_proc.stdout = iter([b"Epoch 1/3\n", b"Loss: 2.5\n"])
|
||||
mock_proc.poll.side_effect = [None, None, 0]
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
client = TestClient(create_app())
|
||||
try:
|
||||
with client.stream("GET", "/api/train/logs") as resp:
|
||||
assert resp.status_code == 200
|
||||
assert "text/event-stream" in resp.headers["content-type"]
|
||||
# Read at least one event
|
||||
lines = []
|
||||
for line in resp.iter_lines():
|
||||
lines.append(line)
|
||||
if len(lines) >= 2:
|
||||
break
|
||||
finally:
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_logs_no_training_returns_done(self):
|
||||
"""When no training is running, SSE should emit done event."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
client = TestClient(create_app())
|
||||
with client.stream("GET", "/api/train/logs") as resp:
|
||||
assert resp.status_code == 200
|
||||
body = b""
|
||||
for chunk in resp.iter_bytes():
|
||||
body += chunk
|
||||
text = body.decode()
|
||||
assert "done" in text
|
||||
|
||||
def test_logs_last_event_id_reconnection(self):
|
||||
"""Last-Event-ID header should skip earlier lines."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
mock_proc = MagicMock()
|
||||
lines = [b"Line 1\n", b"Line 2\n", b"Line 3\n"]
|
||||
mock_proc.stdout = iter(lines)
|
||||
mock_proc.poll.side_effect = [None, None, None, 0]
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
client = TestClient(create_app())
|
||||
try:
|
||||
with client.stream(
|
||||
"GET", "/api/train/logs",
|
||||
headers={"Last-Event-ID": "1"},
|
||||
) as resp:
|
||||
assert resp.status_code == 200
|
||||
body = b""
|
||||
for chunk in resp.iter_bytes():
|
||||
body += chunk
|
||||
text = body.decode()
|
||||
# Line 1 (id=0) should be skipped, Line 2+ should appear
|
||||
assert "Line 2" in text or "Line 3" in text
|
||||
finally:
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_logs_no_auth_required(self):
|
||||
"""SSE log endpoint is GET (read-only) — no auth needed."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
client = TestClient(create_app())
|
||||
# No auth header — should still succeed (not 401)
|
||||
with client.stream("GET", "/api/train/logs") as resp:
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
class TestLiveMetricsSSE:
|
||||
"""Test GET /api/train/metrics/live SSE endpoint."""
|
||||
|
||||
def test_metrics_live_endpoint_exists(self):
|
||||
"""Route /api/train/metrics/live should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/train/metrics/live" in routes
|
||||
|
||||
def test_metrics_live_returns_event_stream(self, tmp_path):
|
||||
"""SSE metrics endpoint should return text/event-stream content type."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
# Simulate running training
|
||||
mock_proc = MagicMock()
|
||||
mock_proc.poll.return_value = 0 # Already finished
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
client = TestClient(create_app())
|
||||
with client.stream("GET", "/api/train/metrics/live") as resp:
|
||||
assert resp.status_code == 200
|
||||
assert "text/event-stream" in resp.headers["content-type"]
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_metrics_live_emits_new_rows(self, tmp_path):
|
||||
"""SSE should emit metric rows as they appear in the DB."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
|
||||
# Pre-populate a run with metrics
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test", "task": "sft"},
|
||||
device="cpu",
|
||||
device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(run_id, step=10, loss=2.5, lr=1e-5)
|
||||
tracker.close()
|
||||
|
||||
# Simulate finished training
|
||||
mock_proc = MagicMock()
|
||||
mock_proc.poll.return_value = 0
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
client = TestClient(create_app())
|
||||
with client.stream(
|
||||
"GET", f"/api/train/metrics/live?run_id={run_id}"
|
||||
) as resp:
|
||||
assert resp.status_code == 200
|
||||
body = b""
|
||||
for chunk in resp.iter_bytes():
|
||||
body += chunk
|
||||
text = body.decode()
|
||||
# Should contain metric data
|
||||
assert "step" in text or "done" in text
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_metrics_live_no_auth_required(self, tmp_path):
|
||||
"""SSE metrics endpoint is GET (read-only) — no auth needed."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
mock_proc = MagicMock()
|
||||
mock_proc.poll.return_value = 0
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
client = TestClient(create_app())
|
||||
with client.stream("GET", "/api/train/metrics/live") as resp:
|
||||
assert resp.status_code == 200
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_metrics_live_done_when_no_training(self, tmp_path):
|
||||
"""Should emit done event when no training is running."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
client = TestClient(create_app())
|
||||
with client.stream("GET", "/api/train/metrics/live") as resp:
|
||||
body = b""
|
||||
for chunk in resp.iter_bytes():
|
||||
body += chunk
|
||||
text = body.decode()
|
||||
assert "done" in text
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
|
||||
class TestTrainProgress:
|
||||
"""Test GET /api/train/progress endpoint."""
|
||||
|
||||
def test_progress_endpoint_exists(self):
|
||||
"""Route /api/train/progress should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/train/progress" in routes
|
||||
|
||||
def test_progress_not_running(self):
|
||||
"""Should return running=false when no training."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/train/progress")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["running"] is False
|
||||
|
||||
def test_progress_running_with_metrics(self, tmp_path):
|
||||
"""Should return progress when training is running."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test", "task": "sft"},
|
||||
device="cpu",
|
||||
device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(run_id, step=50, loss=1.5, lr=1e-5)
|
||||
tracker.close()
|
||||
|
||||
mock_proc = MagicMock()
|
||||
mock_proc.poll.return_value = None
|
||||
mock_proc.pid = 9999
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/train/progress?run_id={run_id}")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["running"] is True
|
||||
assert data["current_step"] == 50
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_progress_returns_correct_fields(self, tmp_path):
|
||||
"""Progress response should contain all required fields."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test", "task": "sft"},
|
||||
device="cpu",
|
||||
device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(run_id, step=10, loss=2.0, lr=1e-5)
|
||||
tracker.close()
|
||||
|
||||
mock_proc = MagicMock()
|
||||
mock_proc.poll.return_value = None
|
||||
mock_proc.pid = 1234
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/train/progress?run_id={run_id}")
|
||||
data = response.json()
|
||||
expected_keys = {"running", "current_step", "run_id"}
|
||||
assert expected_keys.issubset(set(data.keys()))
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_progress_no_auth_required(self):
|
||||
"""Progress endpoint is GET (read-only) — no auth needed."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/train/progress")
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestSSEGracefulClose:
|
||||
"""Test that SSE endpoints close gracefully."""
|
||||
|
||||
def test_logs_closes_when_training_stops(self):
|
||||
"""Log SSE should end when training process finishes."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
mock_proc = MagicMock()
|
||||
mock_proc.stdout = iter([b"Starting\n"])
|
||||
mock_proc.poll.side_effect = [None, 0]
|
||||
ui_mod._train_process = mock_proc
|
||||
|
||||
client = TestClient(create_app())
|
||||
try:
|
||||
with client.stream("GET", "/api/train/logs") as resp:
|
||||
body = b""
|
||||
for chunk in resp.iter_bytes():
|
||||
body += chunk
|
||||
text = body.decode()
|
||||
# Stream should terminate with done event
|
||||
assert "done" in text
|
||||
finally:
|
||||
ui_mod._train_process = None
|
||||
|
||||
def test_heartbeat_event(self):
|
||||
"""SSE should include heartbeat comments to keep connection alive."""
|
||||
# This is a design test — heartbeats are sent as SSE comments (:heartbeat)
|
||||
# Testing that the generator yields at least one event
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
import soup_cli.ui.app as ui_mod
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
||||
client = TestClient(create_app())
|
||||
with client.stream("GET", "/api/train/logs") as resp:
|
||||
body = b""
|
||||
for chunk in resp.iter_bytes():
|
||||
body += chunk
|
||||
# Stream should have some content (at least done event)
|
||||
assert len(body) > 0
|
||||
|
||||
ui_mod._train_process = None
|
||||
|
|
@ -0,0 +1,354 @@
|
|||
"""Tests for Web UI Enhanced Metrics & Eval Display."""
|
||||
|
||||
import os
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _auth_headers():
|
||||
"""Return auth headers with the current UI token."""
|
||||
from soup_cli.ui.app import get_auth_token
|
||||
return {"Authorization": f"Bearer {get_auth_token()}"}
|
||||
|
||||
|
||||
class TestMetricsFullFields:
|
||||
"""Test that metrics response includes all fields."""
|
||||
|
||||
def test_metrics_includes_grad_norm(self, tmp_path):
|
||||
"""Metrics should include grad_norm field."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test", "task": "sft"},
|
||||
device="cpu",
|
||||
device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(
|
||||
run_id, step=10, loss=2.5, lr=1e-5,
|
||||
grad_norm=1.23, speed=100.5, gpu_mem="4.2GB",
|
||||
)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/runs/{run_id}/metrics")
|
||||
assert response.status_code == 200
|
||||
metrics = response.json()["metrics"]
|
||||
assert len(metrics) == 1
|
||||
m = metrics[0]
|
||||
assert m["grad_norm"] == 1.23
|
||||
assert m["speed"] == 100.5
|
||||
assert m["gpu_mem"] == "4.2GB"
|
||||
|
||||
def test_metrics_includes_epoch(self, tmp_path):
|
||||
"""Metrics should include epoch field."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test", "task": "sft"},
|
||||
device="cpu",
|
||||
device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(run_id, step=10, epoch=1.5, loss=2.0, lr=1e-5)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/runs/{run_id}/metrics")
|
||||
metrics = response.json()["metrics"]
|
||||
assert metrics[0]["epoch"] == 1.5
|
||||
|
||||
|
||||
class TestRunsCompareEndpoint:
|
||||
"""Test GET /api/runs/compare endpoint."""
|
||||
|
||||
def test_compare_endpoint_exists(self):
|
||||
"""Route /api/runs/compare should be registered."""
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
app = create_app()
|
||||
routes = [route.path for route in app.routes]
|
||||
assert "/api/runs/compare" in routes
|
||||
|
||||
def test_compare_returns_metrics(self, tmp_path):
|
||||
"""Compare endpoint should return metrics for multiple runs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run1 = tracker.start_run(
|
||||
config_dict={"base": "model-a", "task": "sft"},
|
||||
device="cpu", device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(run1, step=10, loss=2.5, lr=1e-5)
|
||||
run2 = tracker.start_run(
|
||||
config_dict={"base": "model-b", "task": "dpo"},
|
||||
device="cpu", device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(run2, step=10, loss=1.8, lr=5e-6)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/runs/compare?ids={run1},{run2}")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "runs" in data
|
||||
assert len(data["runs"]) == 2
|
||||
assert data["runs"][0]["run_id"] == run1
|
||||
assert data["runs"][1]["run_id"] == run2
|
||||
assert len(data["runs"][0]["metrics"]) == 1
|
||||
assert len(data["runs"][1]["metrics"]) == 1
|
||||
|
||||
def test_compare_rejects_too_many_runs(self, tmp_path):
|
||||
"""Compare endpoint should reject >5 runs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
ids = ",".join([f"run_{i}" for i in range(6)])
|
||||
response = client.get(f"/api/runs/compare?ids={ids}")
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_compare_validates_run_ids(self, tmp_path):
|
||||
"""Compare endpoint returns entries for nonexistent runs (empty metrics)."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test_validate.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/runs/compare?ids=nonexistent1,nonexistent2")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert len(data["runs"]) == 2
|
||||
|
||||
def test_compare_empty_ids(self, tmp_path):
|
||||
"""Compare endpoint should reject empty ids."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test_empty.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/runs/compare?ids=")
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_compare_no_auth_required(self, tmp_path):
|
||||
"""Compare is GET (read-only) — no auth needed."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test_noauth.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get("/api/runs/compare?ids=run1,run2")
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestEvalResultsEndpoint:
|
||||
"""Test eval results display with parsed details."""
|
||||
|
||||
def test_eval_results_include_details(self, tmp_path):
|
||||
"""Eval results should include parsed details_json."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test"},
|
||||
device="cpu", device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.save_eval_result(
|
||||
model_path="./output",
|
||||
benchmark="mmlu",
|
||||
score=0.75,
|
||||
details={"subjects": {"math": 0.8, "science": 0.7}},
|
||||
run_id=run_id,
|
||||
)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/runs/{run_id}/eval")
|
||||
data = response.json()
|
||||
assert len(data["eval_results"]) == 1
|
||||
result = data["eval_results"][0]
|
||||
assert result["benchmark"] == "mmlu"
|
||||
assert result["score"] == 0.75
|
||||
# details_json should be present as a string (from SQLite)
|
||||
assert "details_json" in result
|
||||
|
||||
def test_eval_results_empty(self, tmp_path):
|
||||
"""Empty eval results should return empty array, not error."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test"},
|
||||
device="cpu", device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/runs/{run_id}/eval")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["eval_results"] == []
|
||||
|
||||
def test_compare_max_five_runs(self, tmp_path):
|
||||
"""Compare endpoint allows exactly 5 runs."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_ids = []
|
||||
for idx in range(5):
|
||||
rid = tracker.start_run(
|
||||
config_dict={"base": f"model-{idx}"},
|
||||
device="cpu", device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
run_ids.append(rid)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
ids_str = ",".join(run_ids)
|
||||
response = client.get(f"/api/runs/compare?ids={ids_str}")
|
||||
assert response.status_code == 200
|
||||
assert len(response.json()["runs"]) == 5
|
||||
|
||||
|
||||
class TestCompareMetricsContent:
|
||||
"""Test compare endpoint returns full metric data."""
|
||||
|
||||
def test_compare_includes_all_metric_fields(self, tmp_path):
|
||||
"""Compare should return metrics with all fields."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "test", "task": "sft"},
|
||||
device="cpu", device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.log_metrics(
|
||||
run_id, step=10, epoch=1.0, loss=2.5, lr=1e-5,
|
||||
grad_norm=0.5, speed=200.0, gpu_mem="8GB",
|
||||
)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/runs/compare?ids={run_id}")
|
||||
data = response.json()
|
||||
m = data["runs"][0]["metrics"][0]
|
||||
assert m["step"] == 10
|
||||
assert m["loss"] == 2.5
|
||||
assert m["grad_norm"] == 0.5
|
||||
assert m["speed"] == 200.0
|
||||
|
||||
def test_compare_includes_config(self, tmp_path):
|
||||
"""Compare should include run config for diff display."""
|
||||
try:
|
||||
from fastapi.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip("FastAPI not installed")
|
||||
|
||||
db_path = tmp_path / "test.db"
|
||||
with patch.dict(os.environ, {"SOUP_DB_PATH": str(db_path)}):
|
||||
from soup_cli.experiment.tracker import ExperimentTracker
|
||||
from soup_cli.ui.app import create_app
|
||||
|
||||
tracker = ExperimentTracker(db_path=db_path)
|
||||
run_id = tracker.start_run(
|
||||
config_dict={"base": "llama-8b", "task": "sft"},
|
||||
device="cpu", device_name="CPU",
|
||||
gpu_info={"memory_total": "N/A"},
|
||||
)
|
||||
tracker.close()
|
||||
|
||||
client = TestClient(create_app())
|
||||
response = client.get(f"/api/runs/compare?ids={run_id}")
|
||||
data = response.json()
|
||||
assert "config" in data["runs"][0]
|
||||
Loading…
Reference in New Issue