RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
442 Commits 1 Branch 0 Tags 10 MiB
Commit Graph

442 Commits

Author SHA1 Message Date
Alpamys 114225ef59 test: add TDD review gap tests — malformed responses, URL hardcoding, shared utils 
Address TDD review findings: test Anthropic hardcoded URL, malformed
response handling for all 3 providers, shared parse_json_array utility.
13 new tests, 1682 total.
 2026-04-01 18:11:37 +05:00
Alpamys 6157bbb4b5 fix: implement rate limiting, add trust_remote_code warning, fix dedup logging 
- Implement sleep-based rate limiting using --requests-per-minute flag
  (was declared but never enforced).
- Add Rich warning panel before loading local models with
  trust_remote_code=True, matching SGLang/audio precedent.
- Replace silent ValueError pass in dedup pipeline with debug logging.
 2026-04-01 18:07:47 +05:00
Alpamys 68d958d14c fix: address python review — extract parse_json_array, narrow exceptions 
- Extract _parse_json_array into soup_cli/data/providers/_utils.py to
  avoid circular imports between generate.py and provider modules.
- Narrow bare except Exception in detect_ollama to httpx.HTTPError/OSError
  with debug logging instead of silent swallow.
 2026-04-01 18:04:24 +05:00
Alpamys 011ebb6478 docs: update test counts and security notes after path confinement fix 2026-04-01 17:57:29 +05:00
Alpamys 5ecfb0b29c fix: strengthen path confinement in generate command (security review) 
Replace simple '..' check with resolve() + relative_to(cwd) for output
path. Add same confinement guard to --seed, --dedup-with, and --context
file paths. Add _path_within_cwd helper. 4 new security tests.
 2026-04-01 17:54:02 +05:00
Alpamys ea8f785b50 feat: add synth data gen pro with multi-provider, templates, quality pipeline (v0.20.0) 
New providers: Ollama (localhost-only), Anthropic Claude (env-only API key),
vLLM (SSRF-protected). Domain templates: code, conversation, qa, preference,
reasoning. Quality pipeline: --validate, --filter, --dedup, --quality-pipeline.
84 new tests, 1669 total. Security: SSRF protection on all providers, output
path traversal prevention, rate limiting.
 2026-04-01 17:44:23 +05:00
Alpamys 45522ef4e7 fix: use ANSI-safe assertions in eval human help test (macOS CI fix) 
Rich markup wraps --model-a with ANSI codes on macOS, breaking the
substring check. Strip ANSI codes before asserting, matching the
existing pattern in test_speculative_decoding.py and test_deploy_ollama.py.
 2026-04-01 14:51:38 +05:00
Alpamys c46265fd18 feat: add eval platform with custom evals, LLM judge, human eval, leaderboard (v0.19.0) 
Full-featured evaluation system with 7 subcommands:
- soup eval benchmark: standard benchmarks via lm-evaluation-harness
- soup eval custom: custom JSONL eval tasks with 4 scoring modes
- soup eval judge: LLM-as-a-judge (OpenAI/Ollama/server backends)
- soup eval auto: automatic post-training evaluation from config
- soup eval compare: side-by-side eval comparison with regression detection
- soup eval leaderboard: local model leaderboard with JSON/CSV export
- soup eval human: terminal A/B comparison with Elo ratings

New modules: soup_cli/eval/ (custom.py, judge.py, human.py, leaderboard.py)
Config: EvalConfig added to schema.py (auto_eval, benchmarks, custom_tasks, judge)
Callback: SoupTrainerCallback.on_train_end triggers auto-eval when configured

Security: SSRF protection on judge API, ReDoS guard on regex scoring,
API key isolation per provider, 10k task/prompt caps, read-only SQL queries

1585 tests, 58 test files, ruff clean
 2026-04-01 14:47:08 +05:00
Alpamys eb6d1e8671 docs: update version refs to v0.18.2 in CLAUDE.md 2026-04-01 14:03:57 +05:00
Alpamys e072c9d844 chore: bump version to v0.18.2 2026-04-01 14:02:33 +05:00
Alpamys a55f6745e9 fix: use ANSI-safe assertions in deploy help tests (macOS CI fix) 
Rich markup in Typer help output inserts ANSI escape codes around
--flag names on macOS, breaking exact string matches. Check for
lowercase words instead of --prefixed flags.
 2026-04-01 14:01:39 +05:00
Alpamys ad819e32af chore: bump version to v0.18.1 
Post-review fixes since v0.18.0 tag:
- Validate model name before ollama rm
- Fix reversed mock decorator argument order in tests
- Auto-detect chat template in export --deploy path
- Add 7 edge-case tests (OSError, timeout, boundary)
 2026-04-01 13:58:46 +05:00
Alpamys 7416ccdf74 test: add edge-case tests for Ollama deploy (TDD review findings) 
Add 7 tests for previously uncovered branches:
- deploy_to_ollama OSError path
- remove_model timeout and OSError paths
- list_soup_models timeout and nonzero returncode
- validate_model_name 128-char boundary
- detect_ollama version-in-stderr fallback
 2026-04-01 13:55:12 +05:00
Alpamys 4df66414ce fix: auto-detect chat template in export --deploy ollama path 
Use _auto_detect_template() from soup.yaml instead of hardcoding chatml
in the --deploy ollama flow. Falls back to chatml if detection fails.
 2026-04-01 13:53:48 +05:00
Alpamys 1d84595938 fix: correct mock parameter names and add assertion in deploy tests 
Fix reversed @patch decorator argument binding in 4 tests and add
mock_deploy_fn.assert_called_once() in test_export_deploy_ollama_success.
 2026-04-01 13:53:03 +05:00
Alpamys ba7a6b1ee1 fix: validate model name before ollama rm (security review finding) 
Add validate_model_name() check in deploy --remove path to prevent
passing unsanitized names to ollama rm subprocess. Adds test coverage.
 2026-04-01 13:51:15 +05:00
Alpamys f98519ef87 feat: add Ollama integration — deploy GGUF models in one command (v0.18.0) 
New commands:
- `soup deploy ollama` — deploy GGUF to local Ollama with auto-template detection
- `soup deploy ollama --list` / `--remove` — manage Soup-deployed models
- `soup export --deploy ollama` — export + auto-deploy in one step

New files:
- soup_cli/utils/ollama.py — detect, deploy, list, remove, Modelfile generation
- soup_cli/commands/deploy.py — Typer command group with Rich panels
- tests/test_deploy_ollama.py — 78 tests covering all paths

Security hardening:
- GGUF path traversal protection + .gguf extension validation
- Model name validation (no path separators, null bytes)
- Modelfile parameter key allowlist prevents directive injection
- Parameter value newline/null sanitization
- Subprocess calls use list args (no shell injection)
- Warning panel before overwriting existing Ollama models

1449 tests, 57 test files, all passing.
 2026-04-01 13:47:40 +05:00
Alpamys 986f8cb26c feat: add GitHub repo link to CLI output, bump version to v0.17.3 
Show GitHub URL in `soup version`, `soup version --full`, `soup doctor`,
and `soup --help` so users can find and star the repo.
Extract URL to GITHUB_URL constant in utils/constants.py.
 2026-03-26 15:48:04 +05:00
Alpamys 6687f92682 docs: add Troubleshooting section for torch DLL and dual Python issues 2026-03-26 15:20:36 +05:00
Alpamys ab611ce9a5 chore: bump version to v0.17.2 2026-03-26 15:16:04 +05:00
Alpamys 42b56f1570 fix: rename APIs to match test plan, fix RoPE factor detection 
- Rename is_liger_available -> check_liger_available
- Rename detect_flash_attention -> check_flash_attn_available
- Rename is_ring_attention_available -> check_ring_attention_available
- Rename is_sglang_available -> check_sglang_available
- Rename compute_coherence_scores -> compute_coherence_score
- Rename FSDP keys: fsdp_full_shard -> full_shard, etc.
- Fix get_rope_scaling_config to accept factor-style args (e.g., 4.0)
- Update all callers, tests, and README
- 1371 tests pass, ruff clean, 58.81% coverage
 2026-03-26 15:14:24 +05:00
Alpamys e30a637f48 chore: bump version to v0.17.1 
- Fix outdated __version__ reference in CLAUDE.md (was showing 0.14.2)
- Version bump in pyproject.toml, __init__.py, CLAUDE.md, plan.md
- All 1369 tests pass, ruff clean, 58.80% coverage
 2026-03-26 14:53:55 +05:00
Alpamys 705e6b260a fix: skip SGLang FastAPI tests when fastapi not installed (CI fix) 
SGLang app tests require fastapi which is an optional dependency.
Skip these tests gracefully on CI environments without fastapi installed.
 2026-03-26 14:19:43 +05:00
Alpamys 20c2f4e515 fix: use AutoModel for audio, is_relative_to path check, early librosa import 
- Use AutoModel instead of AutoModelForCausalLM for audio-language models
  (Qwen2-Audio, Whisper don't work with causal LM auto class)
- Use Path.is_relative_to() for path traversal check (symlink-safe, Python 3.9+)
- Fail fast with helpful error if librosa not installed before dataset processing
 2026-03-26 13:59:46 +05:00
Alpamys fb1f49afc8 test: add missing coverage for _parse_json_array, _validate_example, SSRF guards 
Addresses TDD review findings:
- Tests for _parse_json_array (markdown fences, NDJSON fallback, empty, invalid)
- Tests for _validate_example (alpaca, sharegpt, chatml, unknown format)
- Tests for _generate_server SSRF validation (scheme whitelist, remote HTTP block)
- Tests for SGLang SSRF validation (URL model path rejection)
- Fix vacuous test_server_provider_accepted assertion

1369 tests, 58.84% coverage.
 2026-03-26 13:57:03 +05:00
Alpamys bc743e2814 fix: pad token perf in perplexity scoring, SGLang SSRF validation 
- Use -100 ignore_index for pad tokens in perplexity computation (avoids
  redundant softmax on padding positions)
- Block URL-based model paths in SGLang create_sglang_runtime (SSRF protection)
 2026-03-26 13:51:57 +05:00
Alpamys 0b7759898c fix: address review findings — immutable rows, response guard, GPU cleanup 
- Stop mutating dataset rows in-place in _validate_audio_files (use shallow copy)
- Guard _generate_server response parsing against unexpected JSON shape
- Add empty dataset guard in _prepare_audio_dataset
- Free GPU memory after perplexity scoring in compute_perplexity_scores
 2026-03-26 13:49:40 +05:00
Alpamys 3d66b41d00 v0.17.0: data quality filters, audio modality, SGLang backend, server provider 
New features:
- soup data filter: quality filters with perplexity and coherence scoring
- modality: audio — Qwen2-Audio, Whisper fine-tuning with audio data format
- --backend sglang for soup serve (SGLang high-throughput inference)
- --provider server for soup data generate (local OpenAI-compatible servers)
- Audio template: soup init --template audio

Security hardening:
- Server provider SSRF validation (scheme whitelist, localhost-only HTTP)
- Audio file path traversal protection (resolved paths confined to audio_dir)
- trust_remote_code warning panels for audio models and SGLang runtime

1348 tests, 56 test files, 58.8% coverage, ruff clean.
 2026-03-26 13:46:17 +05:00
Alpamys 83213e86c4 fix: strip ANSI escape codes in CLI help flag tests 
Rich inserts color codes between flag name parts (e.g. --speculative
becomes \x1b[1;36m-\x1b[0m\x1b[1;36m-speculative\x1b[0m), so plain
substring match fails in CI. Strip ANSI before asserting.
 2026-03-26 12:54:38 +05:00
Alpamys ff88a2c525 fix: resolve CI failures on macOS — speculative decoding flag collision + missing fastapi 
- Rename --spec-tokens to --num-speculative-tokens to avoid prefix
  collision with --speculative-decoding in Typer help rendering
- Add pytest.skip for _create_app tests when fastapi is not installed
 2026-03-26 12:46:29 +05:00
Alpamys cbc0a0e558 v0.16.0: embedding models, ONNX/TensorRT export, speculative decoding 
New features:
- task: embedding — fine-tune sentence embedding models (BGE, E5, GTE)
  with contrastive, triplet, or cosine loss and configurable pooling
- soup export --format onnx — ONNX export via optimum
- soup export --format tensorrt — TensorRT-LLM export for GPU inference
- soup serve --speculative-decoding — draft model for 2-3x faster generation
  (transformers assisted generation + vLLM native speculative decoding)
- soup init --template embedding — new template for embedding fine-tuning

Security:
- ONNX export: removed unconditional trust_remote_code, added warning
- Speculative decoding: SSRF protection (URL blocked), warning panel
- vLLM speculative: URL validation rejects http:// schemes
- TensorRT export: separated try/except per subprocess call
- Embedding config: Literal constraints, margin gt=0 validation

1270 tests, 52 test files, 58% coverage
 2026-03-26 12:41:39 +05:00
Alpamys bee13c22f0 docs: update SECURITY, CONTRIBUTING, examples README to v0.15.0 
- SECURITY.md: supported versions updated to v0.15.x, added v0.15.0 hardening history
- CONTRIBUTING.md: utils list updated with new modules, test count 1182, templates 13
- examples/README.md: added long-context fine-tuning section (#8)
 2026-03-26 11:33:11 +05:00
Alpamys fd2513aa8b v0.15.0: performance + long-context fine-tuning 
Features:
- Liger Kernel integration (fused RMSNorm, SwiGLU, CrossEntropy, RoPE)
- FlashAttention v2/v3 auto-detection via attn_implementation
- FSDP2 support alongside DeepSpeed (--fsdp flag: full_shard, shard_grad, full_offload)
- Ring FlashAttention for sequence parallelism across GPUs
- 128k+ context fine-tuning with RoPE scaling (linear, dynamic, yarn, longrope)
- Gradient checkpointing for memory-efficient long-sequence training
- New 'longcontext' template: soup init --template longcontext

Security:
- rope_scaling_type validated via Literal constraint (no free-form strings)
- max_length bounded ge=64, le=1048576 (prevents OOM/corruption)
- FSDP config key allowlist prevents injection of unexpected TrainingArguments
- Liger exception handling narrowed (no silent CUDA error swallowing)

New files: utils/liger.py, utils/flash_attn.py, utils/fsdp.py,
  utils/ring_attention.py, utils/long_context.py, tests/test_performance.py
Modified: all 10 trainers (fsdp_config support), schema.py, train.py, pyproject.toml

91 new tests (1182 total), 58.5% coverage, ruff clean
 2026-03-26 11:26:48 +05:00
Alpamys e82685c555 u 2026-03-26 10:57:10 +05:00
Alpamys 67adbff558 fix: add future annotations for Python 3.9 compatibility 
`dict | None` syntax requires Python 3.10+. Adding
`from __future__ import annotations` fixes collection on 3.9.
 2026-03-26 10:50:46 +05:00
Alpamys 0ebe59ae00 fix: subprocess tests accept Typer no_args_is_help exit code 2 
Typer returns exit code 2 (not 0) when no_args_is_help=True and no
arguments are provided. Fix test_no_args_shows_help and
test_data_no_args_shows_help to accept both 0 and 2.
 2026-03-26 10:45:03 +05:00
Alpamys 1718578a1a test: add subprocess CLI tests + cross-platform CI matrix 
- Add test_cli_subprocess.py (69 tests): real subprocess execution
  testing entry points, encoding, paths, Unicode, platform regressions
- CI matrix: ubuntu/windows/macos × Python 3.9/3.11/3.12 (9 jobs)
- CI: add coverage reporting with Codecov upload
- Update CLAUDE.md and CONTRIBUTING.md test counts (1022 → 1091)
 2026-03-26 10:24:58 +05:00
Alpamys f5ad0f5a45 docs: update SECURITY, CONTRIBUTING, examples README to v0.14.3 
- SECURITY.md: update supported versions to v0.14.x, add security hardening history
- CONTRIBUTING.md: update test counts (47 files, 1022 tests), add all trainers, fix project structure
- examples/README.md: add KTO/ORPO/SimPO/IPO, pre-training, MoE, batch inference sections
- CLAUDE.md: add SECURITY/CONTRIBUTING/examples to release checklist
 2026-03-25 23:33:46 +05:00
Alpamys dc2ad877db v0.14.3: fix plotext histogram on Windows (cp1251/cp1252 encoding) 
- Check stdout encoding instead of type to detect non-UTF-8 consoles
- Redirect stdout to UTF-8 TextIOWrapper before plotext renders
- Add unit test that simulates cp1251 stdout with plotext
- 1022 tests total
 2026-03-25 23:23:38 +05:00
Alpamys 2ed542318c v0.14.2: fix validate auto-detect, Windows histogram, UI auth docs 
- soup data validate: default --format changed from 'alpaca' to 'auto',
  uses detect_format() to auto-detect dataset format
- soup data stats: force UTF-8 stdout on Windows for plotext histograms
- soup ui: add --show-token flag, document auth token in --help
- 7 new tests (BUG-013/014/015), 1021 tests total
 2026-03-25 23:10:07 +05:00
Alpamys 61a8a1fb56 docs: add Pre-training and MoE sections to README 
Add dedicated sections for continued pre-training (task: pretrain,
plaintext format) and MoE model support (ScatterMoE LoRA, auto-detection).
Update task lists to include Pretrain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 22:41:45 +05:00
Alpamys 232a3b143c docs: update version reference to 0.14.1 in README 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 22:40:12 +05:00
Alpamys 11e075dd94 v0.14.1: review fixes + expanded test coverage 
Post-release fixes from code review agents:
- Guard _output_dir in PretrainTrainerWrapper.train()
- 12 new tests: MoE integration, DeepSeek naming, model types, Unicode
- 1014 total tests, 58.31% coverage

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 22:38:41 +05:00
Alpamys 8dc2e1ebc7 test: add MoE integration, DeepSeek naming, and model type coverage 
Address TDD reviewer findings:
- MoE integration tests for _setup_transformers (moe_lora, aux_loss, non-MoE fallback)
- DeepSeek w1/w2/w3 expert naming discovery tests
- Parametrized tests for jetmoe, arctic, grok, qwen2_moe, deepseek_v2 types
- Plaintext line-level chunking and Unicode loading tests

1014 total tests, 58.31% coverage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 22:34:55 +05:00
Alpamys 4d34924d8f fix: guard _output_dir in PretrainTrainerWrapper.train() 
Prevent silent failure if setup() partially completes by checking
both self.trainer and self._output_dir before proceeding.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 22:29:29 +05:00
Alpamys 15a6daf342 feat: v0.14.0 — pre-training + MoE support 
Add continued pre-training task and Mixture of Experts model support:

- `task: pretrain` for continued pre-training on raw text data
- `plaintext` data format ({"text": "..."} JSONL or .txt files)
- MoE model detection (Mixtral, Qwen3 MoE, DeepSeek V3, DBRX, OLMoE)
- ScatterMoE LoRA (`moe_lora: true`) targets expert FFN + attention layers
- `moe_aux_loss_coeff` for router load-balancing loss
- Templates: `soup init --template pretrain` and `--template moe`
- 85 new tests across test_pretrain.py and test_moe.py (1002 total)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 22:26:01 +05:00
Alpamys b0f17d83af chore: add .claude/settings.json to gitignore 2026-03-25 20:19:44 +05:00
Alpamys c7d4b10712 docs: update version reference to 0.13.2 and correct test file count to 45 2026-03-25 19:37:34 +05:00
Alpamys d0be93af68 docs: expand Supported Models to 340k+ HF models with VRAM guide 
- Lead with 340,000+ text-generation models count from HuggingFace Hub
- Expand recommended models table: 15 families with sizes and use cases
- Add Vision Models table (LLaMA-Vision, Qwen2-VL, Pixtral)
- Add Quick Size Guide: VRAM -> max model size with examples
- Fix version references to v0.13.2 across CLAUDE.md and README
 2026-03-25 19:05:25 +05:00
Alpamys 9451b980d4 fix: rename shadowed builtins and add type annotations in infer.py 
- Rename `input` param to `input_file` (avoid shadowing builtin)
- Rename `output` param to `output_file` for consistency
- Rename loop variable `line = line.strip()` to `raw_line`/`line`
- Add return type annotation to `_load_model() -> tuple`
 2026-03-25 18:59:33 +05:00