RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
442 Commits 1 Branch 0 Tags 10 MiB
Commit Graph

5 Commits

Author SHA1 Message Date
Alpamys 892fd33f9e feat(trainers): v0.35.0 — Trainer Coverage (closes #60, #61, #45) 
Wires v0.28.0 speed/memory features into every transformer-backend
trainer (grpo / kto / orpo / simpo / ipo / ppo / reward_model /
embedding) plus closes the v0.33.0 #43 oversight where dpo / pretrain
accepted activation_offloading without installing offload hooks.

Auto-quant --auto-quant now forwards the picked candidate's
quantization to vLLM via an explicit named parameter (kwarg-splat
hazard removed). Kernel auto-compose runs a forward-only benchmark
loop on the trainer's actual model under torch.no_grad() so live
training gradients aren't polluted (this was a critical-class bug
caught by code-review pre-tag and fixed before merge).

Schema gate lifted with distinct MLX-backend vs unknown-task error
messages so users get the right fix. fp8 / int8 QAT guard fixed in
6 trainers (the legacy unguarded `if tcfg.quantization_aware:` would
have crashed the int8 path with the string "fp8").

Net +187 tests (3928 -> 4115). New file
tests/test_trainer_coverage_v035.py provides a parametrised matrix
proof that every trainer x every feature is exercised on every CI
matrix job. All four review-agent waves (python / code / security /
tdd) clean with every CRITICAL -> LOW finding fixed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 15:01:42 +05:00
Alpamys e509614a9f feat(inference): v0.30.0 — Inference Excellence 
- Prefix caching (--prefix-cache) wired to vLLM enable_prefix_caching.
- Speculative decoding auto-pairing (--auto-spec) with curated target→draft
  map for Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek
  V3/R1, Gemma 2/3. Targets without a known draft (≤8B) fall through.
- Dynamic LoRA hot-swap: POST /v1/adapters/activate/{name} + /deactivate.
  Name regex enforced by FastAPI Path(pattern=…); active state protected
  by threading.Lock; CORS restricted to loopback origins (hot-swap mutates
  state without auth).
- Structured output: --structured-output json|regex with --json-schema
  (cwd-confined via is_under_cwd; 64KB cap; top-level type required) and
  --regex-pattern (2048-char cap, null-byte reject, must compile).
  Constrained token sampling deferred to v0.30.1; constraint descriptor
  exposed on app.state.
- Continuous-batching dashboard: --dashboard + /metrics endpoint. Thread
  safe ServerMetrics with bounded deque for latencies; record_latency
  runs in finally so failure paths are not a blind spot.
- OpenTelemetry tracing: --trace --trace-endpoint. OTLP endpoint SSRF
  hardened matching v0.29.0 HF_ENDPOINT (scheme allowlist, 0.0.0.0
  rejected, RFC1918/link-local/cloud-metadata via ipaddress.ip_address,
  plain HTTP loopback-only). build_tracer idempotent — only installs
  provider when current is ProxyTracerProvider/NoOpTracerProvider. Span
  context via contextlib.ExitStack so __exit__ receives real exc info.
- Auto-quant picker API: --auto-quant flag + Candidate dataclass +
  pick_best() shipped (generator-safe, first-wins tie-break, matches
  v0.28.0 kernel_picker precedent). Live eval loop deferred to v0.30.1;
  flag prints a yellow deferral warning so it is never a silent no-op.

Tests: 2801 → 2886 (+85). New tests/test_inference_advanced.py covers
all 7 parts plus review-driven negatives: Llama 3.3/4 pairing, Mistral
Large, DeepSeek V3/R1, Gemma 3 targets; /v1/adapters/deactivate;
activate-when-no-adapters → 404; metrics concurrent track_request;
OTLP private-IP / 0.0.0.0 / missing-host; pick_best empty-list and
all-failed; NaN score/latency rejection; --json-schema outside cwd;
--structured-output json without schema; --auto-quant deferral warning.

Review agents run (python-review, security-review, code-review, tdd-guide
+ manual verification-loop) and every finding addressed:
- HIGH: pick_best iterator exhaustion (materialise to list).
- HIGH: --auto-quant silent no-op (yellow warning).
- HIGH: --structured-output json silent no-op (fail-fast requires schema).
- HIGH: --json-schema path traversal (is_under_cwd containment).
- HIGH: OTel span swallowed HTTPException (ExitStack migration).
- HIGH: wildcard CORS on unauthed POST (loopback-only regex).
- MEDIUM: OTLP RFC1918/link-local rejection.
- MEDIUM: tracer provider idempotency.
- MEDIUM: record_latency in finally (no failure-path blind spot).
- LOW: active_state threading.Lock; deque replaces O(n) pop(0).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-24 23:39:04 +05:00
Alpamys cbc0a0e558 v0.16.0: embedding models, ONNX/TensorRT export, speculative decoding 
New features:
- task: embedding — fine-tune sentence embedding models (BGE, E5, GTE)
  with contrastive, triplet, or cosine loss and configurable pooling
- soup export --format onnx — ONNX export via optimum
- soup export --format tensorrt — TensorRT-LLM export for GPU inference
- soup serve --speculative-decoding — draft model for 2-3x faster generation
  (transformers assisted generation + vLLM native speculative decoding)
- soup init --template embedding — new template for embedding fine-tuning

Security:
- ONNX export: removed unconditional trust_remote_code, added warning
- Speculative decoding: SSRF protection (URL blocked), warning panel
- vLLM speculative: URL validation rejects http:// schemes
- TensorRT export: separated try/except per subprocess call
- Embedding config: Literal constraints, margin gt=0 validation

1270 tests, 52 test files, 58% coverage
 2026-03-26 12:41:39 +05:00
Alpamys 930489a82e Fix CI: strip ANSI codes in serve help assertions for Rich output 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-23 21:39:04 +05:00
Alpamys 14f619cc00 Add vLLM backend for soup serve (Phase 9) — v0.8.0 
- Add --backend vllm flag to soup serve for 2-4x better inference throughput
- Add --tensor-parallel and --gpu-memory flags for vLLM tuning
- Auto-detect vLLM and show hint when installed but not enabled
- New utils/vllm.py with engine creation, app factory, LoRA support
- Native token-by-token streaming via vLLM AsyncLLMEngine
- Add serve-fast extra: pip install 'soup-cli[serve-fast]'
- Add vllm detection to version --full
- 30 new tests (560 total), ruff clean

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-23 21:30:08 +05:00