Alpamys
|
42b56f1570
|
fix: rename APIs to match test plan, fix RoPE factor detection
- Rename is_liger_available -> check_liger_available
- Rename detect_flash_attention -> check_flash_attn_available
- Rename is_ring_attention_available -> check_ring_attention_available
- Rename is_sglang_available -> check_sglang_available
- Rename compute_coherence_scores -> compute_coherence_score
- Rename FSDP keys: fsdp_full_shard -> full_shard, etc.
- Fix get_rope_scaling_config to accept factor-style args (e.g., 4.0)
- Update all callers, tests, and README
- 1371 tests pass, ruff clean, 58.81% coverage
|
2026-03-26 15:14:24 +05:00 |
|
Alpamys
|
fd2513aa8b
|
v0.15.0: performance + long-context fine-tuning
Features:
- Liger Kernel integration (fused RMSNorm, SwiGLU, CrossEntropy, RoPE)
- FlashAttention v2/v3 auto-detection via attn_implementation
- FSDP2 support alongside DeepSpeed (--fsdp flag: full_shard, shard_grad, full_offload)
- Ring FlashAttention for sequence parallelism across GPUs
- 128k+ context fine-tuning with RoPE scaling (linear, dynamic, yarn, longrope)
- Gradient checkpointing for memory-efficient long-sequence training
- New 'longcontext' template: soup init --template longcontext
Security:
- rope_scaling_type validated via Literal constraint (no free-form strings)
- max_length bounded ge=64, le=1048576 (prevents OOM/corruption)
- FSDP config key allowlist prevents injection of unexpected TrainingArguments
- Liger exception handling narrowed (no silent CUDA error swallowing)
New files: utils/liger.py, utils/flash_attn.py, utils/fsdp.py,
utils/ring_attention.py, utils/long_context.py, tests/test_performance.py
Modified: all 10 trainers (fsdp_config support), schema.py, train.py, pyproject.toml
91 new tests (1182 total), 58.5% coverage, ruff clean
|
2026-03-26 11:26:48 +05:00 |