mirror of https://github.com/razor-ai/soup.git
9 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
ff55e751ab |
fix(v0.33.0): review-wave findings (CRITICAL + HIGH + MEDIUM + LOW)
Addresses findings from 5-agent review wave (python-reviewer, code-reviewer, security-reviewer, tdd-guide, smoke-verification). CRITICAL: - cans/run.py _deploy_target ollama path: rglob *.gguf result is now realpath+commonpath checked against extract_dir before forwarding to `soup deploy ollama --gguf`. Prevents a crafted symlink in the can from making rglob point at an arbitrary on-disk path. HIGH: - cans/publish.py: removed dead update_repo_settings + bare-except tag block (was a no-op network round-trip). Tag attachment via README front-matter is documented as a v0.33.x docs follow-up. - registry/attach.py lookup_entry_by_output_dir: emits ResourceWarning when the 1000-row scan limit is hit (was a silent miss). - data/collators.py CrossDocCollator: stops mutating input dicts via pop() — uses get + dict comprehension. HF Dataset rows are cached and reused; mutation broke subsequent batches silently. Bare-except now logs at DEBUG level so production degradation is inspectable. - monitoring/callback.py _write_spike_recovery_hint: added is_under_cwd guard. args.output_dir came from raw HF TrainingArguments without separate path-containment check. - trainer/rewards.py MACOS_SANDBOX_PROFILE: narrowed (allow mach-lookup) to a 3-name allowlist (SecurityServer, notification_center, opendirectoryd.libinfo). Broad mach-lookup permitted DNS / NSURLSession via launchd, defeating (deny network*). - cans/run.py: PermissionError → ValueError so a caller wrapping in `except OSError` cannot silently swallow the consent gate. PermissionError is an OSError subclass. - commands/can.py run_cmd: assigns result=None up front + explicit None guard so a future _fail bypass cannot trigger NameError on result. - utils/v028_features.py: added type annotations on apply_v028_speed_memory (model: Any, tcfg: TrainingConfig via TYPE_CHECKING, console: Console) and warn_unsupported_features. - cans/run.py: confirm_callback now annotated Callable[[Manifest], bool] for IDE introspection. - tests/test_part_b.py reexec test: drops env-var contamination (RANK/WORLD_SIZE/LOCAL_RANK/ACCELERATE_*) before run, patches imported names on train module, and forces assertion that os.execvp was called — no more silent skip-on-bypass. - tests/test_part_d.py: added TestGenerateResponseSignature source-level guard that catches the lenient logits_processor mock silently passing. MEDIUM: - cans/run.py _run_subprocess: catches subprocess.TimeoutExpired and returns rc=124 (coreutils convention) so callers see a clean CanRunResult instead of an unhandled traceback after the 24h cap. - cans/run.py: temp dir created via mkdtemp is now cleaned up on extract_can failure (try/except + cleanup_extract_dir). - cans/run.py cleanup_extract_dir: switched startswith path check to os.path.commonpath (project-standard idiom; Windows-safe). - cans/schema.py DeployTarget._safe_relpath: normalises mixed separators before splitting on '/' so foo/..\bar can no longer bypass the .. check. - utils/lr_finder.py run_lr_sweep: removed redundant local `import math as _math` (math already at module level). LOW: - eval/gate.py _parse_judge_url: removed bare http:// catchall after scheme allowlist. Defence-in-depth for callers that bypass the Pydantic GateTask validator. - utils/auto_quant.py evaluate_candidate: latency mean now divides by *completed* prompts (excludes crashed). Crashed candidate no longer appears artificially fast. - utils/auto_quant.py Candidate.__post_init__: explicitly rejects bool in score / latency_ms (bool is a subclass of int, was sneaking past). - utils/mii.py: removed `noqa: F401` on Optional import (now actually used in type annotation since we restored it). Tests added (+7, total 3811→3818): - test_part_a_wave1: attach_artifact outside-cwd rejection. - test_part_a_wave2: PermissionError→ValueError migration in 2 tests. - test_part_c: CrossDocCollator mismatched doc_lengths fallback, does-not-mutate-input-dict regression guard. - test_part_d: source-level _generate_response signature guard. - test_part_e: should_recover at max_attempts, outside-cwd skip. Lint: clean. Full suite: 3818 passed in 156s. Findings deliberately not actioned (with rationale): - code-review M1 (mii Pydantic at import-time): forward-ref resolution requires module-level definitions for FastAPI; documented in mii.py. - code-review M4 (supports_v028_features vs validator divergence): the v0.33.0 schema validator was renamed to _validate_v028_speed_memory_supported_tasks and now imports supports_v028_features — they cannot drift. - python-review LOW (_deploy_target vllm silent no-op): documented in the docstring as advisory; logging requires a console arg the helper does not currently take. - security-review LOW 8/9 (TOCTOU window, CLONE_NEWPID): theoretical; documented in CLAUDE.md security section in the next commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
e406688f2d |
feat(training): stability auto-tuning live (v0.33.0 Part E)
Closes #56, #57, #58, #59. #56 Live --find-lr in-process LR-sweep: - New utils/lr_finder.run_lr_sweep(model, dataloader, schedule, optimizer_factory, device): per-step LR mutation + forward + backward, records loss until exhaustion or NaN/Inf divergence. - commands/train.py wires it via _live_lr_sweep_from_config (loads model + tokenizer + first N rows of cfg.data.train), with synthetic-curve fallback so users without GPU/torch still get a parseable report. #57 Loss-spike recovery hint: - SoupTrainerCallback gains spike_recovery / spike_recovery_max_attempts / spike_recovery_lr_decay; on watchdog fire writes output_dir/spike_recovery.json with previous_lr, recommended_lr (per SpikeRecoveryStrategy.compute_new_lr), should_recover, attempts. A wrapper / re-launch can resume with the decayed LR. Live optimizer rewind is intentionally NOT done — HF Trainer has no safe public API for mid-loop optimizer-state mutation; the JSON hint is the contract. #58 auto_mixed_precision push to TrainingArguments: - New SFTTrainerWrapper._resolve_mixed_precision: when tcfg.auto_mixed_precision is True, queries torch.cuda compute capability and calls pick_mixed_precision(base, cc) to set bf16=/fp16= flags. CPU short-circuits to (False, False). When the flag is False, legacy default preserved (bf16=cuda). #59 Grad-accum advisory (Phase 1): - SoupTrainerCallback gains grad_accum_auto_tune / grad_accum_pressure_threshold / grad_accum_total_vram_gb / grad_accum_current_steps / grad_accum_current_batch. - on_log samples torch.cuda.max_memory_allocated each step; if GradAccumMonitor.should_adjust crosses the threshold once, prints (batch, accum) -> (new_batch, new_accum) advisory and short-circuits (one-shot). Phase 2 (live DataLoader rebuild) needs a small TRL upstream PR — tracked as a known limitation. Wiring: - soup_cli/trainer/sft.py: _resolve_mixed_precision helper, batch_size preserved on self, SoupTrainerCallback constructor passes through new spike + grad-accum knobs. - soup_cli/monitoring/callback.py: rich Console import added (was previously module-relative); spike + grad-accum state fields and one-shot helpers. Tests: +15 in tests/test_part_e.py covering the LR-sweep loop with mocked model + optimizer (records, divergence break), mixed-precision resolver across cpu/cuda + auto-flag combinations + qwen2 fp16 quirk on Ampere, spike recovery hint write + attempts increment + disabled no-op, grad-accum advisory one-shot semantics + threshold + cuda-absent + disabled. Known limitations (release notes): - #57 spike recovery is a JSON hint, not in-process optimizer rewind - #59 Phase 2 (live DataLoader rebuild on advisory) deferred Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
ddab34115c |
feat(v0.26.0): Parts B-E — Eval Gate, Trace-to-Pref, Quant-Check, Soup Cans
Closes the v0.26.0 "Red and Blue Ocean" flywheel after Part A (Registry): Train (eval-gated) -> Registry -> Deploy (quant-check) -> Trace-to-Pref -> Train. Part B — Eval-Gated Training: - soup_cli/config/schema.py: EvalGateConfig (enabled/suite/every_n_epochs/ regression_threshold/baseline/on_regression) + TrainingConfig.eval_gate field - soup_cli/eval/gate.py: EvalSuite, GateTask, run_gate, resolve_baseline, load_suite; baselines from registry:// or file - soup_cli/monitoring/callback.py: on_epoch_end + _run_eval_gate with fail-safe error handling (structured errors treated as regressions under on_regression=stop) - soup_cli/commands/train.py: --gate <suite.yaml> shortcut flag - soup_cli/commands/eval.py: gate subcommand (stub generator; live scoring v0.26.1) Part C — Trace-to-Preference: - soup_cli/data/traces/: parse_langchain, parse_openai, parse_soup_serve; build_pairs from thumbs_up / regenerations / user_edit - soup_cli/commands/data.py: from-traces + review subcommands - PII warning panel, 100,000-line cap, path containment, Literal validation Part D — Quant-Lobotomy Checker: - soup_cli/eval/quant_check.py: classify_delta (OK/MINOR/MAJOR), run_quant_check, resolve_model_ref with artifact kinds filter, table/json/markdown renderers - soup_cli/commands/eval.py: quant-check subcommand Part E — Soup Cans: - soup_cli/cans/: Manifest + DataRef (Pydantic v2); pack_entry + fork_can (100MB cap, dunder-key guard); safe tar extraction (filter='data' on py3.12+, narrow fallback, manual symlink rejection + commonpath check) - soup_cli/commands/can.py: pack/inspect/verify/fork subcommands Shared utility: - soup_cli/utils/paths.py: single is_under_cwd helper replacing 5 duplicates (os.path.realpath + commonpath — Windows 8.3 short-name safe) Tests: 103 new (29 eval_gate + 24 trace_to_pref + 23 quant_check + 27 cans) Full suite: 2511 passed on Windows Python 3.10. Security hardening (review-driven, all severities fixed): - EvalGateConfig bounds; GateTask null-byte + judge URL scheme allowlist - Narrow except in _safe_extract so TarError from filter='data' is not swallowed - resolve_model_ref artifact kinds filter (avoid wrong artifact) - Manifest.author cap + null/newline rejection; created_at ISO-8601 validation - fork_can dunder-key + null-byte rejection (prototype pollution prevention) - fork_can size cap (100MB matches pack_entry) - inspect_can/read_config refuse paths outside cwd Docs: - README.md: v0.26.0 "New in" block (flywheel); 43 recipes; all new commands in All Commands list; version examples bumped to 0.26.0; Windows-safe arrows - CLAUDE.md: architecture + test table + schema + CLI + security section extended with B/C/D/E; phase vs Part terminology clarified; release checklist step 18 adds Known Limitations section; step 20 adds comment template; step 21 adds completeness check via gh issue list --milestone - SECURITY.md: per-Part security notes (B/C/D/E) under v0.26.0 - CONTRIBUTING.md: test count + directory tree updates Local smoke: version, eval gate, eval quant-check (table + json), data from-traces, data review, can pack/inspect/verify/fork — all happy-path end-to-end. Fixed Unicode arrows (U+2192) in can.py + gate.py that crashed on Windows CP1252 consoles. Deferred to v0.26.1 (known limitations, filed as issues post-release): - eval gate/quant-check live model scoring (stub generator currently) - data from-traces quality.py judge validation; serve --trace-log collector - can run + can publish + orchestrator - eval --attach-to-registry flag; export auto-artifact registration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1b6b428aaa |
feat: v0.24.0 — Dataset Hub, Freeze Training, Loss Watchdog, Dataset Registry
Part A: HuggingFace Dataset browser - soup data search: search HF Hub for datasets (sort by downloads/likes) - soup data preview: preview remote dataset metadata, splits, features - soup data download: stream HF dataset to local JSONL (with format conversion) - Security: trust_remote_code=False, path traversal protection, samples cap at 1M Part B: Freeze training (like LLaMA-Factory finetuning_type: freeze) - freeze_layers / freeze_ratio config fields - soup_cli/utils/freeze.py: detect layers, freeze bottom N - Wired into SFT trainer before LoRA application - Supports LLaMA (layers.N) and GPT-2 (h.N) naming Part C: Loss watchdog (like Axolotl loss_watchdog_threshold) - loss_watchdog, loss_watchdog_threshold, loss_watchdog_patience config - Implemented in SoupTrainerCallback with patience counter - Rich warning panel (stops Live display first), fires only once - Wired into all 11 trainers via callback kwargs Part D: Dataset info registry - soup data register/unregister/registry commands - ~/.soup/datasets.json local name→path+format mapping - Name validation, path traversal protection, Rich markup escaping 82 new tests (2061 total), 74 test files. |
|
|
Alpamys
|
c46265fd18 |
feat: add eval platform with custom evals, LLM judge, human eval, leaderboard (v0.19.0)
Full-featured evaluation system with 7 subcommands: - soup eval benchmark: standard benchmarks via lm-evaluation-harness - soup eval custom: custom JSONL eval tasks with 4 scoring modes - soup eval judge: LLM-as-a-judge (OpenAI/Ollama/server backends) - soup eval auto: automatic post-training evaluation from config - soup eval compare: side-by-side eval comparison with regression detection - soup eval leaderboard: local model leaderboard with JSON/CSV export - soup eval human: terminal A/B comparison with Elo ratings New modules: soup_cli/eval/ (custom.py, judge.py, human.py, leaderboard.py) Config: EvalConfig added to schema.py (auto_eval, benchmarks, custom_tasks, judge) Callback: SoupTrainerCallback.on_train_end triggers auto-eval when configured Security: SSRF protection on judge API, ReDoS guard on regex scoring, API key isolation per provider, 10k task/prompt caps, read-only SQL queries 1585 tests, 58 test files, ruff clean |
|
|
Alpamys
|
4010798e2b |
Fix PyTorch 2.7 compatibility: total_mem → total_memory (v0.2.1)
PyTorch 2.7+ renamed `get_device_properties().total_mem` to `total_memory`. Fixed in gpu.py and callback.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
|
|
Alpamys
|
2aaa87fb4e |
Phase 2: experiment tracking, data tools, model evaluation
- Add SQLite experiment tracker (~/.soup/experiments.db) with auto-logging of config, per-step metrics, hardware info, and eval results - Add soup runs commands: list, show (with plotext loss curves), compare, delete - Integrate tracker into soup train (auto start_run/finish_run/fail_run) - Add soup data convert (alpaca/sharegpt/chatml bidirectional conversion) - Add soup data merge (concatenate datasets with optional shuffle) - Add soup data dedup (MinHash near-duplicate removal via datasketch) - Add soup data stats (length percentiles, token counts, language detection) - Add soup eval (lm-evaluation-harness wrapper with tracker integration) - Add reverse format conversion: messages_to_format() in data/formats.py - Add extended_stats() to data/validator.py - Update monitoring callback to log metrics to tracker - Add plotext to deps, datasketch as optional [data] dep - Update README and CLAUDE.md with Phase 2 docs - 70 tests passing, ruff clean Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
|
|
Alpamys
|
7433029d19 |
Fix all ruff lint errors and failing test
- Fix 23 ruff errors: line too long, unused imports, ambiguous vars - Fix validator: empty string is valid data, only count None as empty - Remove unused imports in display.py and validator.py - Rename ambiguous `l` vars to `part`, `entry`, `length` - Break long lines in callback.py, display.py, sft.py, constants.py All 20 tests passing, ruff clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
|
|
Alpamys
|
d6e932a1d3 |
Initial project setup: CLI skeleton + config + trainer + data pipeline
- Typer CLI: soup init, soup train, soup data inspect/validate - Pydantic config schema with YAML loader and validation - Data pipeline: JSONL/JSON/CSV/Parquet + HuggingFace datasets - Format detection: Alpaca, ShareGPT, ChatML (auto-detect) - SFT trainer wrapper over transformers + peft + trl - QLoRA/LoRA support with auto batch size estimation - GPU detection (CUDA/MPS/CPU) and memory calculation - Rich live terminal dashboard for training monitoring - Config templates: chat, code, medical - Tests (pytest) + GitHub Actions CI - MIT license Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |