Alpamys
|
e6a9c087c3
|
feat(lora): v0.39.0 — LoRA Quality (PiSSA + ReLoRA + per-pattern rank + surgical patches + templates registry)
Five PEFT-surface improvements that LlamaFactory and Axolotl maintain:
- LoraConfig.init_strategy Literal["random","pissa","olora"]; PiSSA SVD init
via PEFT init_lora_weights="pissa". Back-compat: use_olora=True aligns to
init_strategy="olora" via dict-copy model_validator(mode="before"); explicit
conflict (use_olora=True + init_strategy="pissa"/"random") rejected.
Mutual-exclusion vs DoRA / VeRA.
- ReLoRA callback (utils/relora.py): frozen ReLoRAPolicy with bounds-checked
steps [1, 1e7] / warmup_ratio [0,1] / prune_ratio (0,1) (strict — prevents
zero-everything footgun); magnitude_prune_tensor (in-place torch.kthvalue,
rejects non-Tensor / single-element short-circuit); duck-typed
ReLoRACallback (no transformers import at module load). TrainingConfig
fields relora_steps / relora_warmup_ratio / relora_reset_optimizer /
relora_prune_ratio. SoupConfig _validate_relora_supported_tasks gates to
task=sft + transformers backend with distinct MLX-backend error message;
multi-trainer expansion deferred to v0.39.1 (mirrors v0.27.0 MII /
v0.37.0 multipack / v0.38.0 quant menu stub-then-live pattern).
- LoraConfig.rank_pattern / alpha_pattern Optional[Dict[str,int]]; field
validator caps at 256 keys × value (0, 1024], rejects bool / null-byte /
empty key. Cross-validator rejects with use_vera=True (VeRA shares one
rank). peft_builder propagates into LoraConfig init_kwargs.
- utils/peft_patches.py: is_gemma4_model uses regex word boundary
(?:^|[^a-z0-9])gemma-?4(?:[^a-z0-9]|$) so "ungemma4ed" no longer matches.
apply_gemma4_clippable_patch swaps ClippableLinear → nn.Linear by class
name (weight-copy fallback logs at DEBUG). strip_lora_dropout_for_3d_experts
zeroes lora_dropout.p on 3-D weights (handles ModuleDict variant for
PEFT >=0.10). apply_surgical_patches orchestrator validates model_name.
Wired into sft.py _setup_transformers with is_gemma4_model gate before
the pre-LoRA swap; post-LoRA 3-D dropout strip runs unconditionally
(architecture-detected internally).
- 16 inline templates migrated to soup_cli/templates/*.yaml + manifest.json
+ load_template loader (path-traversal-rejecting name validator;
os.path.realpath + commonpath containment so a tampered manifest cannot
read files outside the package directory; 256 KB file-size cap with
inline fallback). Inline TEMPLATES kept with deprecation comment
(planned removal v0.41.0+); test_templates_yaml asserts byte-equality
of all 16 inline ↔ YAML pairs to prevent silent drift.
Net +164 tests (4374 → 4538). All 5 review-agent waves clean before tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-01 16:18:07 +05:00 |