Alpamys
|
fcf4b33394
|
feat(train): native Spectrum targeted training — soup spectrum scan + training.unfrozen_parameters (v0.71.23)
Closes #266. `soup spectrum scan` streams safetensors per-tensor (no model
load, CPU-friendly) and computes a singular-value SNR per weight matrix
(Marchenko-Pastur, arXiv:2406.06623), emitting a ready-to-paste
training.unfrozen_parameters patch. The SFT trainer freezes all params then
unfreezes the matched set (full FT, LoRA off).
- utils/spectrum_scan.py: pure-numpy transpose-invariant SNR kernel +
per-tensor safetensors streaming (2^31 SVD cap, symlink skip) + cache
(~/.soup/spectrum, SOUP_SPECTRUM_CACHE_DIR containment) + hardened
hubs.snapshot_download.
- commands/spectrum.py: soup spectrum scan (SNR table + YAML patch).
- schema: training.unfrozen_parameters (caps/NUL/invalid-regex/ReDoS reject)
+ gates (sft/transformers/text/quantization=none; mutually exclusive with
LoRA features / freeze_layers / freeze_ratio / train_router_only /
expand_layers).
- trainer/sft.py: full-FT branch via apply_unfrozen_parameters +
enable_input_require_grads (fixes grad-checkpointing through frozen
embeddings).
Existing spectrum trainer-plugin untouched (back-compat); LISA -> #267.
Live-validated on Windows + RTX 3050: CPU scan of SmolLM2-135M + top-25%
unfrozen full-FT train (loss 3.455 -> 0.719).
Note: after the version bump the editable install metadata was stale
(0.71.17); pip install -e . --force-reinstall --no-deps re-synced it so
test_cli_subprocess::test_version passes.
+94 tests in tests/test_v07123.py (14184 -> 14278).
|
2026-06-12 17:40:46 +05:00 |