Alpamys
|
e509614a9f
|
feat(inference): v0.30.0 — Inference Excellence
- Prefix caching (--prefix-cache) wired to vLLM enable_prefix_caching.
- Speculative decoding auto-pairing (--auto-spec) with curated target→draft
map for Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek
V3/R1, Gemma 2/3. Targets without a known draft (≤8B) fall through.
- Dynamic LoRA hot-swap: POST /v1/adapters/activate/{name} + /deactivate.
Name regex enforced by FastAPI Path(pattern=…); active state protected
by threading.Lock; CORS restricted to loopback origins (hot-swap mutates
state without auth).
- Structured output: --structured-output json|regex with --json-schema
(cwd-confined via is_under_cwd; 64KB cap; top-level type required) and
--regex-pattern (2048-char cap, null-byte reject, must compile).
Constrained token sampling deferred to v0.30.1; constraint descriptor
exposed on app.state.
- Continuous-batching dashboard: --dashboard + /metrics endpoint. Thread
safe ServerMetrics with bounded deque for latencies; record_latency
runs in finally so failure paths are not a blind spot.
- OpenTelemetry tracing: --trace --trace-endpoint. OTLP endpoint SSRF
hardened matching v0.29.0 HF_ENDPOINT (scheme allowlist, 0.0.0.0
rejected, RFC1918/link-local/cloud-metadata via ipaddress.ip_address,
plain HTTP loopback-only). build_tracer idempotent — only installs
provider when current is ProxyTracerProvider/NoOpTracerProvider. Span
context via contextlib.ExitStack so __exit__ receives real exc info.
- Auto-quant picker API: --auto-quant flag + Candidate dataclass +
pick_best() shipped (generator-safe, first-wins tie-break, matches
v0.28.0 kernel_picker precedent). Live eval loop deferred to v0.30.1;
flag prints a yellow deferral warning so it is never a silent no-op.
Tests: 2801 → 2886 (+85). New tests/test_inference_advanced.py covers
all 7 parts plus review-driven negatives: Llama 3.3/4 pairing, Mistral
Large, DeepSeek V3/R1, Gemma 3 targets; /v1/adapters/deactivate;
activate-when-no-adapters → 404; metrics concurrent track_request;
OTLP private-IP / 0.0.0.0 / missing-host; pick_best empty-list and
all-failed; NaN score/latency rejection; --json-schema outside cwd;
--structured-output json without schema; --auto-quant deferral warning.
Review agents run (python-review, security-review, code-review, tdd-guide
+ manual verification-loop) and every finding addressed:
- HIGH: pick_best iterator exhaustion (materialise to list).
- HIGH: --auto-quant silent no-op (yellow warning).
- HIGH: --structured-output json silent no-op (fail-fast requires schema).
- HIGH: --json-schema path traversal (is_under_cwd containment).
- HIGH: OTel span swallowed HTTPException (ExitStack migration).
- HIGH: wildcard CORS on unauthed POST (loopback-only regex).
- MEDIUM: OTLP RFC1918/link-local rejection.
- MEDIUM: tracer provider idempotency.
- MEDIUM: record_latency in finally (no failure-path blind spot).
- LOW: active_state threading.Lock; deque replaces O(n) pop(0).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-04-24 23:39:04 +05:00 |
|
Alpamys
|
dee9317dde
|
feat: v0.22.0 — Training Profiler, Multi-Adapter Serving, Data Sampling, Adapter Management
New commands:
- `soup profile` — estimate memory, speed, GPU requirements before training
(--config, --gpu, --json flags)
- `soup adapters list/info/compare` — LoRA adapter management
- `soup data sample` — intelligent dataset sampling (random/diverse/hard strategies)
- `soup serve --adapters` — multi-adapter serving with adapter selection
New files:
- soup_cli/utils/profiler.py — memory/speed estimation engine
- soup_cli/commands/profile.py — profile CLI command
- soup_cli/commands/adapters.py — adapter management CLI
Security:
- Multi-adapter: adapter path traversal protection (resolve + relative_to)
- Multi-adapter: adapter name validation (alphanumeric + hyphens only)
- Multi-adapter: unknown adapter → 404, no adapter name leakage in errors
- Multi-adapter: /v1/adapters returns names only (no filesystem paths)
- Multi-adapter: --adapters rejected for non-transformers backends
- Data sample: output path confinement (resolve + relative_to(cwd))
101 new tests (1890 total), 66 test files, 65.5% coverage, ruff clean.
|
2026-04-03 12:54:24 +05:00 |