mirror of https://github.com/razor-ai/soup.git
158 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
ed5fc3a8b3 |
feat(precision,rollout): live fp8/nvfp4 + vLLM sleep + openenv rollout + apple-adapter + delinearize-llama4 (v0.71.21)
Closes #141, #124, #125, #228, #97. - #141: apply_fp8_attention (torchao float8 on attention projections, Hopper gate) + apply_nvfp4 (NVFP4Config, Blackwell gate); partial-conversion honesty; wired into the v0.28 speed/memory pipeline with yellow-advisory degrade. - #124: vllm_sleep_mode live - create_vllm_engine(sleep_mode=True) + vllm_sleep_cycle ctx (wake in finally) + TRL GRPOConfig hook probe. - #125: openenv rollout fully live via training.rollout_func module:fn resolver; rows replace the prompt dataset; art/ruler/nemo_gym honest dep gates + _EXTERNAL_ROLLOUT_RUNNERS seam. Real GRPO train on SmolLM2-135M. - #228: convert_apple_adapter live - PEFT LoRA <-> mlx-lm (both matrices transpose, bf16 upcast, adapters.safetensors + num_layers, npz legacy read, np.ascontiguousarray fix for safetensors non-contiguous mangling); *-to-apple upstream-gated exit 3. - #97: delinearize-llama4 live - [E*din,dout] -> [E,din,dout] per shard, config.json expert-count probe + --num-experts, sidecar copy, atomic writes. Review waves: 3 HIGH + ~8 MEDIUM + ~12 LOW fixed. Tests: 13874 -> 14084 (+210 in tests/test_v07121.py). Full suite: 13967 passed, 117 skipped. ruff clean. |
|
|
Alpamys
|
a4dfbb308c |
feat(trainer): live TTS / BitNet / MoE-expert-quant trainers (v0.71.20)
Lift three v0.52.0 schema-only NotImplementedError stubs to real code. - #131 TTS: TTSTrainerWrapper(SFTTrainerWrapper) — TTS fine-tune = next-token CE over [text][audio-codec-token] chat; per-family emotion templating (Orpheus/Oute) + codec special-token registration. Pre-encoded chat path live-validated on SmolLM2-135M-Instruct; live-codec (data.format=audio) hardware-gated per family. - #134 BitNet: BitNetTrainerWrapper gated on onebitllms; export --format bitnet|tq1_0 runs real llama.cpp TQ1_0 ternary GGUF export. - #136 MoE: apply_moe_expert_quant swaps fused-MoE experts to bnb Linear4bit/ Linear8bitLt (pre-LoRA); train_router_only freezes experts (post-LoRA). Live-validated on RTX 3050 (dequant err 0.0155). Review fixes: H1 explicit Params4bit/Int8Params weight-carry; H2 quant pre-LoRA / freeze post-LoRA + skip PEFT-wrapped modules; M4 device-aware placement. Tests 13807 -> 13874 (+69 in test_v07120.py, -2 lifted stubs in test_v0520.py). |
|
|
Alpamys
|
f51331d637 |
feat(quant,multipack): vision/audio Quant Menu + multipack FSDP sharding (v0.71.19)
#81 Quant Menu for vision/audio modality - config/schema: drop the `modality != "text"` rejection in _validate_quant_menu_supported_tasks (mlx-backend gate retained) so the full Quant Menu (gptq/awq/hqq:Nbit/aqlm/eetq/mxfp4/fp8) applies to vision+audio. - trainer/sft: _setup_vision_transformers + _setup_audio_transformers call build_quantization_config_for_loader (strict superset of the inline 4bit/8bit BNB blocks they replaced); drop BitsAndBytesConfig import; retain the prepare_model_for_kbit_training gate on (4bit,8bit,mxfp4). #80 multipack DataLoader sharding under FSDP/DeepSpeed/DDP - utils/multipack_trainer: get_train_dataloader routes the multipack DataLoader through accelerator.prepare when num_processes > 1 so accelerate's BatchSamplerShard shards whole FFD bins across ranks (even_batches=True avoids the epoch-boundary collective hang; seed identical across ranks, no `+ rank`). Single-process path unchanged. Defence-in-depth guard against an unconfigured MagicMock num_processes. Tests: +37 in tests/test_v07119.py (13770 -> 13807). ruff clean. Supersedes the v0.40.5 vision/audio Quant-Menu and v0.40.4 multipack-FSDP known-limitations. Full multi-GPU validation remains an INFRA-BLOCKED QA item. |
|
|
Alpamys
|
70fd5ee9f3 |
feat(distill,agent,cloud): on-policy MiniLLM + aligned ULD + agent sandbox eval + Modal cloud (v0.71.18)
Closes #257, #258, #110, #16. - #257 MiniLLM true on-policy rollout: minillm_on_policy_rollout (Gu et al. §3.1 autoregressive teacher-mixed rollout, reverse-KL on full distributions, grad-to-student-only) + on_policy_term + training.minillm_on_policy / minillm_rollout_length, wired into DistillTrainer.compute_loss. - #258 cross-tokenizer ULD wasserstein_aligned: align_token_sequences (difflib char-span) + aggregate_aligned_logits + uld_aligned_loss for fully-disjoint tokenizers, wired into DistillTrainer. - #110 soup agent eval --sandbox: build_eval_stub (base64-embed-as-data) + run_eval_in_sandbox (v0.25 RLVR isolation + SANDBOX_NETWORK_GUARD) + classify_sandbox_outcome (ok/tool_error/timeout/arg_error). - #16 soup train --cloud modal: render a Modal app from soup.yaml (config base64-embedded), plan-only default, --cloud-submit token-gated; [modal] extra. +114 tests (tests/test_v07118.py); 13656 -> 13770. Step-6 smoke on real input (Windows + RTX 3050): Modal stub render, real subprocess sandbox scorecard, on-policy distill (tiny-gpt2), cross-tokenizer aligned ULD (GPT-2 + Llama). |
|
|
Alpamys
|
0d55ba9e46 |
feat(serve): serve-time MoLE + per-request vector banks + epoch RAFT shuffle (v0.71.17)
Closes #259 (soup serve --mole: load base + N frozen task LoRAs + mole_gate.pt, blend per-token at decode; train writes mole_manifest.json). Closes #260 (soup serve --bank active user per-request via contextvars.ContextVar, no cross-request leak; streaming path re-selects in-context). Closes #253 (data.raft_epoch_shuffle: re-permute golden/distractor docs each epoch; epoch=0 == legacy order). Closes #254 (soup diagnose --citation-style / --shuffle-seed into the live probe). Fix: MoLE train() returns initial_loss/final_loss/total_steps/duration_secs so task=moe_lora_routing completes cleanly (surfaced by the #259 smoke). Validated live on SmolLM2-135M (RTX 3050). 13595 -> 13656 tests. |
|
|
Alpamys
|
28a2d8cf82 |
feat(edit): GPT-2 Conv1D edits, covariance ROME, atomic governor, Mixtral LongLoRA (v0.71.16)
Knowledge edit depth — 4-issue patch (validated on real gpt2 + SmolLM2-135M, RTX 3050). - #251 GPT-2 (transformer.h / mlp.c_proj Conv1D) support in ROME/MEMIT/AlphaEdit: transpose-aware _rank1_update + _alphaedit_project + MEMIT dim-check + PEFT unwrap. - #250 covariance-preconditioned ROME via --cov-corpus (estimate_key_covariance + C^-1 k* solve; fail-loud reject for non-rome; cwd-contained O_NOFOLLOW loader). - #252 atomic EditGovernor increment: save_state merges this run's delta under the cross-process lock (baseline-delta, mirrors namespace_pin). - #147 Mixtral in the LongLoRA allowlist (is_mixtral_model + MixtralAttention forward override + _SEPARATE_QKV_FAMILIES). Tests: 13511 -> 13595 (+81 in tests/test_v07116.py). Full suite green; ruff clean. |
|
|
Alpamys
|
d5f98c67d5 |
fix(loop): config-render, cmaes base-reuse, cost gate, energy hand-off, rank guard (v0.71.15)
#261 iterative_dpo._default_train_fn rendered output as a {dir: ...} mapping that SoupConfig rejected; render it flat so the spawned soup train succeeds. #246 CMA-ES merge now loads the base model once and reuses it across the candidate population (_CachedBaseScorer) instead of reloading per candidate. #245 soup loop estimate_cost wires run_cost.estimate_run_cost_usd off the last completed run instead of a 0.0 placeholder; never crashes the daemon. #244 soup train --track-energy --energy-out persists the measurement JSON so soup bom emit --energy can attach it to an ML-BOM. #170 --diagnose-gate is RANK-aware: gate once per cluster (RANK==0), not per node. Tests: 13476 -> 13511 (+35 in tests/test_v07115.py). Validated end-to-end on SmolLM2-135M / RTX 3050. |
|
|
Alpamys
|
0b621076b9 |
docs: reposition tagline to "fine-tune and post-train"
Soup spans the full post-training stack — SFT + preference/RL (DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO) + distillation + unlearning + knowledge-edit + steering + RAFT/RA-DIT — not just fine-tuning. "Fine-tune" alone undersells the RL/alignment surface and the post-training category. Update the canonical tagline to "Fine-tune and post-train LLMs in one command. No SSH, no config hell." across the five places it appears: README hero, PyPI description (pyproject), `soup --help` epilog + callback docstring (cli.py), package docstring (__init__), and the auto-generated HF model card (push.py). Searchable "fine-tune" keyword kept as the verb; "post-train" added as the category claim. Copy-only; no version bump (description ships with next release). |
|
|
Alpamys
|
6ec36b5ff6 |
feat: live FSDP-shard consolidation + serve KV-cache type + ONNX QA (v0.71.14)
Close the doable tail of the export-QA + deferred-stub family. - #96 consolidate_shards: lazy-torch safetensors merge, TOCTOU-hardened (enforce_under_cwd_and_no_symlink, weights_only=True, atomic_write_bytes), 16 GiB/shard cap, dup-key shape-conflict reject. --plan-only flag. - #140 apply_kv_cache_type -> KvCacheRuntime; soup serve --kv-cache-type (bf16/f16 dtype, q8_0 quantized cache + hqq probe, fp8 Hopper-gated). - #71 ONNX export QA: tiny-GPT2 PASS, TinyLlama host-RAM-bound (qa doc). - #70/#72/#144/#74/#79 deferred to INFRA-BLOCKED tail (kept open). Tests 13430 -> 13476 (+46). ruff clean. Suite green, 78.52% cov. |
|
|
Alpamys
|
f528da5328 |
feat(prompt-compile): live soup compile / distill-prompt / compile-tools / local-rl train (v0.71.13)
Lift the v0.68.0 deferred-stub family to live (closes #225, #226, #227, #229): - #229 local-rl train --once: harvest thumbs -> DPO/KTO/ORPO train via a soup train subprocess (argv list, no shell); state table tracks last_train_at (skip-on-no-new-thumbs + skip-on-insufficient-pairs); no --once renders a systemd/launchd nightly scheduler scaffold. New local_rl_scheduler.py. - #226 distill-prompt: call the teacher once per trace (Ollama/Anthropic/vLLM) and write a real dataset (sft/kl -> messages; preference -> chosen/rejected). - #225 compile / #227 compile-tools: live DSPy/GEPA/TextGrad dispatch behind the new [compile] extra with a friendly ImportError when absent; injectable seams. Security: reject \n/\r in the model id + shell-quote ExecStart args (systemd injection defence). Fix: render train output as a plain string (schema-valid), with a regression test against SoupConfig. Tests 13329 -> 13424. Smoked end-to-end: real DPO train on SmolLM2-135M (RTX 3050) + real Ollama teacher distillation. |
|
|
Alpamys
|
a66e4b9ebe |
feat: architecture + distill + adapter-train live wiring (v0.71.12)
Lift seven schema-only stubs to live, validated on tiny models: - #145 distill_mode token|sequence — sequence-level teacher-continuation KD - #146 classifier LoRA — frozen encoder + adapter (classifier/reranker/cross_encoder) - #148 LLaMA Pro block expansion — per-arch (Llama/Qwen/Mistral) zero-init blocks - #158 LongLoRA S2 — shifted-sparse attention on Q/K projections (Llama/Mistral/Qwen/Phi) - #84 Mixture-of-Depths — per-layer top-k token router (use_mod; Llama/Qwen/Mistral) - #221 VeRA/VB-LoRA serving — soup serve --bank, per-user delta via X-User-Id header - #222 MoLE — task=moe_lora_routing, per-token gate over N frozen task LoRAs (gate-only) Tests 13203 -> 13329 (+126; tests/test_v07112.py). ruff clean, coverage 78.49%. |
|
|
Alpamys
|
f316d334bc |
feat(rl): live GRPO/RL callbacks — reward-hack, echo-trap, RL ckpt, ULD, MiniLLM, iterative-DPO (v0.71.11)
Lifts the v0.70.0 schema-only build_*_callback / build_uld_projection / run_iterative_dpo stubs. Validated end-to-end on SmolLM2-135M. Closes #235, #236, #237, #238, #239, #240, #159, #160 - #235 RewardHackCallback: info_rm cluster-sep / rm_ensemble divergence, OK/WARN/HACK, halt on HACK. Shared thread-safe RLSignalBuffer captures per-step rewards by wrapping the reward fns (no TRL monkeypatching). - #236 ULD: Wasserstein-1 / top-k aligned distill loss in DistillTrainer. - #237 MiniLLM: teacher-mixed length-normalised reverse-KL + pretrain anchor. - #238 RLCheckpointCallback: adapter + optimizer.pt + manifest + keep_last prune. - #239 run_iterative_dpo: sample -> RM-score -> build-pairs -> DPO-train per round. - #240 EchoTrapCallback: n-gram repetition OK/WARN/TRAP, halt on TRAP. - #159 one-shot WARNING when a GRPO variant compute_loss falls back to super(). - #160 in-place ref-model EMA (no state_dict round-trip) + 0-overlap warning. Tests 13142 -> 13203 (+62 in tests/test_v07111.py). |
|
|
Alpamys
|
a2287dd6a5 |
feat(rag): RAFT span-mask trainer + RA-DIT auto-link + live steering + eval citation (v0.71.10)
Lifts the v0.62.0 RAG-family schema-only stubs to live, validated on SmolLM2-135M: - #199 RAFT: data.format=raft trains answer-only (prompt span masked to -100, [doc-N] citation ids, deterministic doc shuffle by raft_shuffle_seed); rows whose prompt fills max_length are dropped with a warning. New utils/raft.py + trainer/raft.py (RaftDataCollator + weighted-CE _RaftTrainer). - #200 soup ra-dit: one-shot two-stage orchestrator (train retriever -> record it as the generator's paired retriever -> train generator); a generator-stage `soup train` with no retriever set auto-links the latest RA-DIT retriever from the Registry. New utils/ra_dit_run.py + commands/ra_dit.py. - #201 soup steer train/apply + soup serve --steer: live CAA/ITI/RepE fit from {positive, negative} pairs + decode-time forward hook (transformers backend). Lifts the steering.py apply_steering/build_steering_vector stubs. - #202 soup eval citation + citation-span per-token loss boost + 7th `citation` failure mode in soup diagnose. New commands/_eval_v07110.py + diagnose/citation.py. Review fixes (3 agents, all CRITICAL->LOW): markup-escaped autolink advisory; shared enforce_under_cwd_and_no_symlink + O_NOFOLLOW on every new file read; steering-artifact containment; honest RA-DIT docs (records pairing, no weight fusion); public validate_ra_dit_config_path + render_raft_prompt; repe/iti require >=2 pairs; eval citation --shuffle-seed. Full suite: 13034 passed, 106 skipped (13142 collected). ruff clean. |
|
|
Alpamys
|
96c339f184 |
feat(edit): live ROME/MEMIT/AlphaEdit + GRACE + NPO/SimNPO/RMU unlearn (v0.71.9)
Closes #193, #194, #196, #197, #203. - #194 utils/edit_kernels.py: covariance-free rank-1 ROME/MEMIT/AlphaEdit; apply_edit live (load -> optimise residual -> rank-1 update -> save); edit diff live before/after generation. - #196 EditGovernorStore SQLite persistence + cross-process lock. - #197 apply_edit consults the governor (check_can_edit before, record after). - #203 GraceCodebook + apply_grace_edit + install_grace_hook + Registry kinds. - #193 utils/unlearn_kernels.py (NPO/SimNPO/RMU) + live UnlearnTrainerWrapper + soup train --task unlearn. Validated on SmolLM2-135M: ROME 0.0016->0.96, NPO/SimNPO forget loss down. +81 tests (tests/test_v0719.py). 2 review waves, all findings fixed. |
|
|
Alpamys
|
823456c1a5 |
feat(probe): real probe weights, SAE auto-download, truth/harm, interference --measure, capture-activations (v0.71.8)
Closes #216, #217, #218, #219. Partial #215 (calibrated vectors upstream-gated). - #215 probe_kernel.py: compute_contrast_probe + load_probe_weights (.npz/.npy/.safetensors, O_NOFOLLOW, allow_pickle=False, cwd-contained); soup probe sleeper --weights. Synthetic seed fallback retained. - #216 hubs.snapshot_download (SSRF-hardened, home/cwd/tmp cache, TOFU gate) + sae_diff.download_sae (allowlist-before-network + symlink-escape guard); soup probe sae-diff --auto-download. - #217 truth_probe.py + harm_probe.py over probe_kernel; soup probe truth/harm; probe pack ships truth+harm per base. - #218 interference_live.measure_interference_losses (live PEFT multi-adapter, add_weighted_adapter cat off-diagonal); soup probe interference --measure. - #219 live_eval.extract_layer_activations + resolve_layer_module PEFT-fallback; soup train --capture-activations writes <output>/activations/activations.json. Test count 12771 -> 12917 (+146 in tests/test_v0718.py). Step-6 smoke on SmolLM2-135M (RTX 3050) green; caught + fixed a PEFT-wrapper layer-resolution bug. |
|
|
Alpamys
|
f097528ac0 |
feat(eval): live eval runners — advise/tunability/capability/behavior/diagnose (v0.71.7)
Closes #161, #162, #208, #211, #212, #165. New utils/live_eval.py shared model-loading layer (lazy torch/transformers/peft): load_model_and_tokenizer, make_generator/make_multi_generator, compute_eval_loss, lora_probe, measure_logit_agreement, token_f1. - #161 soup advise --probe-model: live zero/few-shot token-F1 + LoRA probe - #162 base_model_proximity via held-out logit agreement - #208 soup tunability --live: per-candidate LoRA probe - #211 soup eval capability --live --model: lm-eval-harness per task (per-task isolation) - #212 soup eval behavior --base-model: live pre/post battery diff - #165 soup diagnose --base-model: utils/diagnose/live.py runs all 6 probes live Heuristic/neutral paths preserved when no model is supplied. Both new JSONL readers open with O_NOFOLLOW after cwd-containment (TOCTOU close). +68 tests (12703 -> 12771). Smoked end-to-end on SmolLM2-135M (RTX 3050). |
|
|
Alpamys
|
a1463bf716 |
feat(v0.71.6): live build runner + Magpie generator + 2PL/3PL IRT + augment fix
Lift the v0.69.0 deferred stubs to live + extend IRT + fix a real bug: - #231 soup build materialises (5 built-in transforms, table/view/incremental with SQLite-tracked config-fingerprint cache key, atomic JSONL, --output-dir) - #232 soup data gen-magpie live (ollama/vllm raw-completion harvest; anthropic rejected; optional --quality-filter; dedup-before-response) - #167 tokenizer-aware memorization probe (sub-word/BPE overlap, library-only) - #213 soup eval irt-subset --model 2pl|3pl (joint coordinate-ascent MLE) - #75 fix soup data augment --provider ollama|vllm ImportError + QA log Security: validate_ollama_url/validate_vllm_url reject 0.0.0.0; augment output containment+symlink reject; magpie response-body cap. Tests 12581 -> 12703 (+122 in tests/test_v0716.py). Full suite green, ruff clean. |
|
|
Alpamys
|
1f63393421 |
feat(v0.71.5): ingest/data/prompt/drift polish
Closes #157, #205, #207, #149, #164, #163. Defers #204 (live SaaS pull — paid accounts, infra-blocked, kept open). - #164: get_metric_series falls back to eval_results when metrics is empty - #163: build_verdict confidence biased by advise_history (same project+choice, >=3 precedents); decision never changes - #207: shared utils/webhooks.py (SSRF-hardened) + --slack-url/--discord-url on ingest/prune-prompt/ab/active-sample; ab fires only on a decision - #205: soup prune-prompt --tokenizer (token-prefix detect + decode remainder, boundary-safe) - #149: DynamicCurriculumCallback buckets by loss/perplexity percentile; length keeps round-robin - #157: soup data push/forge --hub modelscope|modelers (data score N/A) 107 new tests in tests/test_v0715.py (12474 -> 12581). ruff clean. |
|
|
Alpamys
|
5652215d4a |
feat(adapters,loop): v0.71.4 — live canary verdict + cmaes merge + PR push + pre-wired loop + can lineage + branch↔registry
Closes #172, #173, #176, #177, #220, #223. - #172 soup adapters merge --canary/--strict-verdict: live OK/MINOR/MAJOR verdict (was UNKNOWN stub) - #220 soup adapters merge --strategy cmaes: live merge→score→write-best loop (was plan-only) - #223 soup adapters pr --push owner/repo#N: post PR comment via gh api - #176 soup loop --pre-wired: real traces→DPO→eval-gate→canary stages - #177 soup loop --pack-cans / replay --extract: iterations as Soup Cans + Registry lineage DAG - #173 soup adapters branch --from-registry / --attach-to-registry Security: backdoor-scan + license gates now run for ALL merge strategies (incl cmaes); loop canary deploy restricted to loopback/RFC1918; gh child env from allowlist; canary read uses O_NOFOLLOW+fstat (TOCTOU); pack-entry failure rolls back registry entry. Tests: 12342 → 12474 (+130 in tests/test_v0714.py). |
|
|
Alpamys
|
21a2bf8e8c |
feat(governance,energy): v0.71.3 — annex PDF, audit auto-log, energy hook, can v3, airgap receipt
Closes #180 #181 #182 #183 #184 #188. - #180 EnergyTracker (codecarbon offline) + `soup train --track-energy`; [carbon] extra - #181 PDF Annex XI/XII (reportlab) + paths.atomic_write_bytes; [pdf] extra - #182 Soup Can manifest v3 + attestations field + `can pack --attest` - #183 per-command audit-log auto-instrumentation (--no-audit-log / SOUP_NO_AUDIT_LOG) - #184 auto-populate Annex top_domains from the training JSONL - #188 embed repro-receipt into `soup airgap-bundle` New [pdf]+[carbon] extras (reportlab also in [dev]). +83 tests (12259 -> 12342), 79.10% coverage. Reviewed (security/code/python/tdd): security M1 raw-size gate on --attest before parse, python H1/H2 type hints, +22 negative tests. |
|
|
Alpamys
|
9300ee3412 |
feat(governance,sign): v0.71.2 — ed25519 signing + supply-chain gates
ed25519 signing (#179/#185 ed25519 half): new utils/signing.py + [sign] extra. `soup adapters sign/verify --backend ed25519` (--key/--generate-key/ --public-key) and `soup attest emit --sign ed25519` + new `soup attest verify`. Sigstore keyless stays infra-blocked (OIDC/Fulcio/Rekor network). #186 namespace-pin gate wired into download_repo (anti-AI-Jacking; fail-open) #187 adapters merge auto-detects each adapter's license #190 license-override reason recorded to the audit log #191 NamespacePinStore WAL + busy_timeout + cross-process lock #192 merge refuses scan-FAIL inputs unless --allow-unscanned +106 tests (12153 -> 12259); full suite 79.07% cov. cryptography added to [dev] for CI. 5 review waves, all findings fixed. |
|
|
Alpamys
|
514761c89a |
feat(env,lock,serve,eval): v0.71.1 — quick wins + wiring (7 closures)
Closes #195 #210 #214 #224 #230 #233 #209. - soup env fix: print-only install-plan renderer from soup-env.lock (uv-pip / requirements; non-pip entries surfaced as comments). (#209) - soup lock write --env-lock: auto-derive --env-hash from soup-env.lock via new compute_env_hash (excludes created_at). (#224) - soup serve --record-thumbs <db>: capture thumbs-up/down into the local-RL SQLite + POST /v1/thumbs (transformers backend). (#230) - Judge-calibration persistence: JudgeCalibrationReport.to_dict + write/load_judge_calibration + judge_calibration registry kind; load re-validates the frozen dataclass with cwd/symlink containment. (#214) - soup completions: introspect a base model's real LoRA target modules (config-only AutoConfig, local_files_only, never networks/raises). (#210) - Bundled MUSE + WMDP unlearning eval fixtures; WMDP forget rows ship REDACTED (Soup never bundles verbatim hazardous content). (#195) - build_dag.validate_build_source: cwd-containment + symlink rejection. (#233) Review-fix hardening (consolidated python+code+security+tdd, 0 CRIT/0 HIGH): load_judge_calibration containment + friendly missing-field ValueError; serve thumbs success-print escape; env_fix --output Optional[str]; empty --env-hash auto-derives; render_install_plan PEP 440 docstring note. Tests: 12071 -> 12134 (12044 passed, 90 skipped, 2 deselected). |
|
|
Alpamys
|
894cb632dd |
chore: release v0.71.0 — split heavy deps into [train] extra
Heavy training stack (torch, transformers, peft, trl, datasets, bitsandbytes, accelerate) moves out of the core install into a new [train] optional-dependency extra. `pip install soup-cli` is now a light CLI + data-tools install with no PyTorch; `pip install 'soup-cli[train]'` adds the training stack. - pyproject: new [train] + [all] extras; [dev] self-references [train] so CI (`pip install -e ".[dev]"`) still gets torch. Pins unchanged. - errors.py: missing torch/transformers/peft/trl/datasets/bitsandbytes/ accelerate now surface a single 'install soup-cli[train]' fix. - Dockerfile: install soup-cli[train,serve,data,eval] so the GPU image can still fine-tune. - README + docs/models.md: split install into light core vs [train]. - CHANGELOG: cut [0.71.0]; bump version 0.70.0 -> 0.71.0. |
|
|
Alpamys
|
30cfe5b5be |
chore: project hygiene — py.typed, pre-commit, mypy CI, CHANGELOG, slim SECURITY.md
- add src/soup_cli/py.typed (PEP 561); verified it ships in the built wheel - add .pre-commit-config.yaml (ruff lint+format + standard file-hygiene hooks) - add mypy>=1.8.0 + pre-commit to the [dev] extra; lenient [tool.mypy] config - add a non-blocking type-check CI job (mypy, continue-on-error: true) - add CHANGELOG.md (Keep a Changelog; [Unreleased] + link to GitHub Releases) - replace the ~221KB per-version security log in SECURITY.md with a concise policy - raise the coverage gate 50% -> 77% (measured 79% on the suite; real-2 margin) No version bump: hygiene/docs only — rides into the 0.71.0 deps-split release. |
|
|
Alpamys
|
06d8ea7cc5 |
chore: migrate to src-layout
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.
- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
CODEOWNERS patterns, and the PR-template checklist all repointed to
src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.
Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
|
|
|
Alpamys
|
aeaa24eae2 |
chore: drop Python 3.9, add AGENTS.md, register unit/integration markers
- pyproject: requires-python >=3.10, ruff target py310, register unit/integration pytest markers alongside smoke - CI: matrix 3.10/3.11/3.12 (drops 3.9 — EOL October 2025, ~33% CI time/cost savings) - README, CONTRIBUTING: bump Python references 3.9 -> 3.10 - AGENTS.md: new root entry-point for tool-agnostic AI coding agents (Codex, Cursor, Aider) — points to .claude/CLAUDE.md for full guide Acts on external feedback re: project conventions for OSS contributors. |
|
|
Alpamys
|
74edac95d1 |
feat(v0.70.0): Loop Hardening — reward-hacking + ULD + MiniLLM + RL ckpt + iterative DPO + echo-trap
Six-part schema-only release shipping the axis-3 + axis-13 training-loop hardening. Every live trainer-callback / math kernel is deferred to v0.70.1 per the project's established stub-then-live cadence (matches v0.50.0 / v0.62.0 / v0.69.0). Part A — Reward-hacking detector (soup_cli/utils/reward_hacking.py): InfoRM Cluster-Separation Index (Wang et al. 2024 arXiv:2402.09345) + RM-ensemble pairwise variance + OK/WARN/HACK taxonomy at 0.10/0.30 thresholds. TrainingConfig.reward_hack_detector / reward_hack_halt; SoupConfig task-gate (grpo/ppo only, mlx rejected, halt requires detector). Part B — Cross-tokenizer ULD (soup_cli/utils/uld.py): Universal Logit Distillation (Boizard et al. 2024 arXiv:2402.12030). wasserstein + topk_align allowlist; ULDConfig frozen with topk/strategy cross-validators; vocab-size cap 262144. Schema-gated to task='distill'. Part C — MiniLLM reverse-KL on-policy distillation (soup_cli/utils/minillm.py): Gu et al. 2024 (arXiv:2306.08543) — bundles teacher-mixed sampling + length-norm + pretrain-loss anchor stability tricks. Anchor weight↔path mutual-requirement cross-validators reject silent no-op combos. Part D — Mid-epoch RL checkpoint (soup_cli/utils/rl_checkpoint.py): Optimizer-state serialization TorchTune explicitly punts. RLCheckpointConfig + RLCheckpointState frozen + JSON-serialisable manifest. RL-task gate. Part E — Iterative DPO loop driver (soup_cli/utils/iterative_dpo.py + commands/iterative_dpo.py): sample → RM-score → re-pair → retrain over N rounds. IterativeDPOPlan with consecutive-round_index invariant. New `soup iterative-dpo` CLI; --plan-only live, runner deferred. Part F — RAGEN echo-trap detector (soup_cli/utils/echo_trap.py): Zhu et al. 2025 (arXiv:2504.14437) — n-gram trajectory-repetition kernels with DoS caps (max 32 ngram_n, 1M tokens, 100k trajectories). OK/WARN/TRAP at 0.30/0.60. Composes with v0.53.11 #127 GRPOStabilityCallback. Cross-cutting hardening: - 6 new util modules + 1 new top-level CLI + 11 new TrainingConfig fields + 6 new SoupConfig cross-validators + 3 new field validators - Closed allowlists (frozenset) + MappingProxyType registries everywhere - Frozen dataclasses with post-init validation on every public record - Bool-as-int rejection on every numeric (matches v0.30.0 / v0.41.0 policy) - math.isfinite NaN/Inf rejection on every float - Null-byte rejection + per-field length caps on every string - No top-level torch imports (4 source-grep regression tests) - Deferred-live stubs validate inputs FIRST then raise NotImplementedError with explicit v0.70.1 marker - CLI exit codes split: 2 = validation rejection, 3 = deferred-live Test count: 11487 → 11824 (+337 net). 12-invariant self-review against the full project checklist (closed allowlists, frozen dataclasses, MappingProxyType, bool-as-int rejection, finite check, null-byte, length caps, no top-level torch, TypeError/ValueError split, deferred-live, tuples-not-lists, CLI exit codes) all green across all 6 Parts. Manual CPU smokes (Step 6): every CLI happy + failure path exercised — `soup iterative-dpo --plan-only` 3-round plan rendered end-to-end with per-round artifacts; 5 happy-path YAML loads + 5 failure-mode rejections across reward_hack / uld / minillm / rl_checkpoint / echo_trap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
49943a5af6 |
feat(v0.69.0): Data Engineering Pro — soup build + expect + gen-magpie + persona-mix + brain-rot
5 parts shipping axis-2 (dbt-for-SFT) + axis-13 (data ops): - soup build — dbt-for-SFT DAG with refs / incremental materialization / content-hash row-diff kernel (run_build live runner deferred → v0.69.1) - soup expect <data> <suite> — LIVE expectations suite: PII / token-length / refusal / chosen-vs-rejected judge; exit 3 on suite failure - soup data gen-magpie — Magpie synthetic generator plan (live → v0.69.1) - soup data persona-mix — Persona-Hub × style sampler with bundled 12×5 set, atomic JSONL write (LIVE) - soup data brain-rot — arXiv 2510.13928 detector with --strict CI gate, worst-signal composite (LIVE) Centralised TOCTOU defence behind utils/paths.enforce_under_cwd_and_no_symlink in build_dag / expectations / expect.py (code-review CRIT — replaces 3 duplicate os.lstat + S_ISLNK + realpath + is_under_cwd blocks). DoS caps on every new JSONL loader (brain-rot 1 GiB + 1M rows; persona-mix 100 MiB + 100k entries). persona-mix --output TOCTOU symlink rejection. magpie quality_filter validator + expectations._dispatch_expectation raw-args pass-through (no int/float coercion bypass). BuildModel seed/derived cross-validator rejects ambiguous shapes at schema load. Review-fix coverage across 4 waves (security + code + python + TDD): 1 CRITICAL + 4 HIGH + 5 MEDIUM + 4 LOW. Test count: 11225 → 11487 (+262 net across 5 new files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
aa71658f50 |
feat(v0.68.0): Anti-trend Insurance — compile (DSPy/GEPA) + distill-prompt + compile-tools + apple-adapter + local-rl
5 commands that hedge Soup against paradigm shifts. If 1M-context kills FT, `soup compile` (DSPy + GEPA + TextGrad prompt-program compilation) takes its place. If teams hit prompt-cost walls, `soup distill-prompt` bridges to small FT. If only Apple Foundation Models win on-device, `soup apple-adapter` ships the converter+signing surface. If personal-LLM flywheels become the shape, `soup local-rl` captures thumbs into SQLite and emits DPO pairs. - Part A: `soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|...]` - Part B: `soup distill-prompt --traces <jsonl> --teacher --student --strategy` - Part C: `soup compile-tools <spec.json|yaml> --eval <jsonl>` - Part D: `soup apple-adapter <source-dir> --direction hf-to-mlx|... --output` - Part E: `soup local-rl init/status/record/harvest/train` (LIVE except train) Schema + path containment + symlink rejection + atomic-write surface ship now; live runners for Parts A/B/C/D + Part E nightly scheduler deferred to v0.68.1 (stub-then-live, mirrors v0.50.0 / v0.61.0 / v0.62.0 / v0.67.0 cadence). Test count: 11021 -> 11225 (+204). Review-fix: 0 CRIT + 4 HIGH + 10 MED + 4 LOW. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
32145097ec |
feat(v0.67.0): Adapter Lifecycle Finish — CMA-ES merge + VeRA bank + MoLE + PRs + soup.lock + bisect
Six surfaces close v0.57: - Part A: pure-Python rank-mu CMA-ES evolutionary merge (cmaes_merge.py) + soup adapters merge --strategy cmaes --eval <s> --budget 1h - Part B: VeRA / VB-LoRA vector-bank schema + atomic JSON I/O (vector_bank.py) - Part C: MoLE per-token routing schema + new task='moe_lora_routing' (mole_routing.py) - Part D: GitHub-shaped adapter PR renderer (adapter_pr.py) + soup adapters pr <title> --base-sha --adapter --eval --samples - Part E: soup.lock shared run lockfile (soup_lock.py + commands/lock.py) + soup lock write/show/check (exit 3 on drift) - Part F: training-history binary search (adapter_bisect.py) + soup adapters bisect <ckpts> --eval-command "..." Live wiring deferred to v0.67.1: CMA-ES eval-suite auto-bind, VeRA serving, MoLE gating kernel. +185 tests (10836 -> 11021) across 7 new test files. Review-fix coverage from 2 sequential waves (security + tdd-guide). All step-6 smokes green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
a015ccc812 |
feat(v0.66.0): Post-train X-rays — SAE diff + live blame + sleeper probe + interference matrix + probe pack
Extends `soup diagnose` from 6 failure modes to 10. Closes v0.57 #171 — live blame runner replaces the NotImplementedError stub. - `soup probe sae-diff`: SAE feature attribution (pure-numpy; HF_HUB_ALLOWLIST) - `soup adapters blame --top-k 50`: live DataInf influence runner (closes #171; replaces v0.57 stub with cos(grad_row, grad_probe) × |grad_row|) - `soup probe sleeper`: calibrated defection probe (6 bundled bases; OK/MINOR/MAJOR at 1%/5%; exit 2 on MAJOR) - `soup probe interference`: pairwise N×N matrix (OK/MINOR/MAJOR at 5%/20%; exit 2 on MAJOR worst-pair) - `soup probe pack`: per-base probe manifest assembler Review-fix coverage across 3 sequential waves: 0 CRITICAL + 9 HIGH + 14 MEDIUM + 5 LOW. Notable hardening: - TOCTOU O_NOFOLLOW probe-open in load_sae_weights + _count_dataset_rows - hashlib.sha256 replaces process-salted hash() for CI reproducibility - Rich-markup escape on adapter / verdict / description / layer - TypeError on bool/non-str verdict before membership check - Non-numeric loss rejection in `probe interference` CLI - 10M-row hard reject (no silent truncate); 100k synthetic-probe cap - _LOWER_INDEX MappingProxyType for O(1) case-insensitive lookup - Mapping from collections.abc (PEP 585); frozenset[str] type params - Frozen dataclasses + FrozenInstanceError regression tests Note: Windows cp1251 print on stdout-capturing Python wrappers can crash on Rich's '→' arrow output; the soup CLI itself uses force_utf8_stdio. Test count: 10577 → 10836 (+259 net across test_v0660_part_{a-e}.py, test_v0660_cli.py, test_v0660_followups.py). Full suite green (10836 passed, 81 skipped); ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
1e822af461 |
feat(v0.65.0): Eval Depth — judge calibration + behaviour battery + capability suite + CheckList DSL + IRT subset
5 LIVE parts closing axis 4 of the roadmap. Evals as first-class surface, not afterthought:
- Judge calibration: SCOPE/CJE-style bidirectional pairwise judging in eval/calibrate.py
with PairwiseJudgement / fit_position_bias / conformal_threshold +
ensure_judge_calibrated production gate that refuses to score with an uncalibrated
judge (RuntimeError on None / calibrated=False / low agreement / extreme bias).
- Behaviour battery (soup eval behavior): closed allowlist of XSTest / HarmBench /
JailbreakBench / ELEPHANT / SycEval with 5 tiny bundled redacted probe sets under
soup_cli/data/_fixtures/behavior/. Word-boundary regex agreement rejects
"safe" in "unsafe" false positives. Pre/post diff with OK/MINOR/MAJOR verdict
(matches v0.26 / v0.56 taxonomy).
- Capability auto-suite (soup eval capability): MMLU-Pro / GPQA / BBEH / AIME /
MATH-500 / HumanEval+ / SWE-bench-Verified with full / fast / math / code profile
selector. Emits (benchmark, lm-eval task) manifest for downstream
soup eval benchmark chaining.
- CheckList DSL (soup eval checklist): Ribeiro et al. 2020 MFT / INV / DIR test kinds
rendered from YAML. Word-boundary matching prevents "and" matching "sand".
Per-test pass/fail + OK/MINOR/MAJOR overall verdict.
- IRT eval-cost optimizer (soup eval irt-subset): 1PL Rasch closed-form fit on
per-item correctness signals + high-info subset selector (full / small / tiny
profiles). 5-10x cut in eval bills without losing ranking power.
Cross-cutting hardening (review-fix coverage across 2 review waves):
- TOCTOU defence: every new read path uses O_NOFOLLOW + os.fstat on SAME fd
(load_checklist_spec, load_response_rows, _read_evidence_json). Earlier
double-lstat-on-path was a race the attacker could win by swapping the file
between calls.
- Namespace-package safety: load_battery_probes uses importlib.resources.files
Traversable / op + as_file (was Path(os.path.join(str(pkg_root), ...)) which
silently fails is_file() on MultiplexedPath installs).
- Word-boundary regex agreement in behavior_battery + checklist_dsl.
- CLI _validate_run_id gate; 16 MiB --evidence cap with O_NOFOLLOW;
_MAX_ROWS=1_000_000 cap counts skipped lines toward total in load_response_rows.
- INV empty-string normalisation no longer spuriously passes.
- _write_json_output / _read_evidence_json / _validate_run_id dedup helpers in
commands/_eval_v0650.py.
Review fixes: 0 CRITICAL + 6 HIGH + 9 MEDIUM + 7 LOW resolved across 2 waves.
Test count: 10306 -> 10577 (+271 net across test_v0650_part_{a,b,c,d,e}.py +
test_v0650_followups.py). Full suite 10577/10577 passing. 0 regressions.
Step 6 smoke: every new CLI command + 3 failure modes exercised end-to-end
(behavior with --evidence happy + MAJOR exit 2; capability fast with output;
checklist with real YAML; irt-subset on 600-row synthetic data; unknown
battery / size / kind all exit 2; outside-cwd evidence rejected).
Known limitations:
1. Live lm-eval-harness invocation deferred — soup eval capability emits the
manifest for downstream soup eval benchmark chaining (Typer commands aren't
safe to re-enter; matches v0.46.0 / v0.44.0 design).
2. Live model-driven soup eval behavior deferred — without --evidence, emits a
neutral OK report (v0.65.1).
3. Behaviour battery probe sets ship as tiny redacted placeholders — operators
pull real harmful prompts from upstream papers.
4. IRT model is 1PL Rasch only (2PL / 3PL deferred to v0.65.x).
Step 6 quirk worth noting: --evidence containment rejects /tmp/ on Windows
WSL bash; operators must run from cwd or pass cwd-contained paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
8b5991674b |
feat(v0.64.0): Pre-flight & Tooling — tunability, plan/apply, env, hardware-fit, completions, license-advisor
Six new top-level commands close axis 1 + 11 of the roadmap: pick the right base, lock the env, refuse OOMs before launch, and clear license-clean deploys. - soup tunability: probe-train 8 candidate bases (Qwen3-0.6/1.7B, Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) -> Pareto frontier over (delta x cost x license). Live LoRA probe -> v0.64.1. - soup plan / soup apply: Terraform-shape lock-and-execute. `apply` refuses on drift between soup.yaml and soup.tfstate (exit 3). - soup env lock / status / check: hermetic env lockfile via importlib.metadata across 15 ABI-sensitive packages + Python + CUDA. `env check` exits 3 on drift. - Hardware-fit calculator: static analytical 5-bucket VRAM predictor with 10% safety margin + actionable hint on OOM. - soup completions bash|zsh|fish: sourceable shell completion scripts; recipe names auto-complete from the 115-recipe catalogue. - soup license-advisor: per-deploy-target license matrix (b2c/defense/embedded) + Llama community + 700M MAU gate (exit 3). Composes with v0.60 license-conflict matrix. Tests: 10035 -> 10306 (+271 net in 7 new files). Review-fix coverage: 0 CRITICAL + 6 HIGH + 8 MEDIUM + 4 LOW across consolidated code+security+TDD review wave. Every HIGH lands a regression test in tests/test_v0640_followups.py (POSIX-skipped symlink rejection, containment-before-existence ordering, drift-refusal exit-3 end-to-end). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
40bd6251a2 |
feat(v0.63.0): Production Trace Ecosystem — soup ingest + prune-prompt + active-sample + ab + drift-alarm
5 new top-level commands close axis 7 of the roadmap. Every Part LIVE on
day one (no deferred stubs):
- soup ingest: universal trace importer (Langfuse / LangSmith / Helicone /
OpenPipe / OTel / OpenAI Stored Completions). 6 adapters
+ frozen TraceRecord with MappingProxyType-wrapped metadata.
Zero credential-handling threat surface — Soup parses the
JSONL export, never makes the SaaS network call.
- soup prune-prompt: detect + strip a shared system-prompt prefix so the
FT model internalises it (OpenPipe's signature trick,
OSS). Binary-search over up to 32 templates finds the
longest threshold-meeting prefix.
- soup data active-sample: surface top-uncertainty prod traces for human
review. Max-entropy on single rm_score or
pairwise disagreement on dual rm_scores.
- soup ab: Wald sequential SPRT for the point alternative. LLR is a
martingale under H0 so Type-I error is controlled at every
stopping time per the optional stopping theorem.
- soup drift-alarm: rolling KL on whitespace-tokenised output distribution
+ SSRF-hardened Slack/Discord webhook (full parity with
v0.51.0 validate_hub_endpoint). Exit 3 on drift for
cron-friendly automation.
Test count: 9816 -> 10035 (+219 net across 6 new test files).
Review-fix coverage (code-reviewer + tdd-guide returned actionable;
python-reviewer + security-reviewer agents context-thrashed on the large
CLAUDE.md release-notes history — matches the v0.58.0 / v0.59.0 / v0.60.0
/ v0.61.0 / v0.62.0 idiom; verified manually):
- 1 CRITICAL: mSPRT log-likelihood-ratio sign error drove Type-I error
to 1.0 as n grew. Replaced with Wald's classic point-
alternative SPRT (martingale under H0).
- 2 HIGH: detect_common_prefix early-exit on 100% match returned the
shortest qualifying prefix instead of the longest;
_MAX_SCAN_ROWS DoS cap used 'pass' instead of 'break'.
- 3 MEDIUM: TraceRecord.metadata now MappingProxyType-wrapped post-init
(frozen-dataclass mutation hazard); _AUTH_ENV table
deduplicated; drift_alarm precedence parens on SSRF gate.
- 2 LOW: pooled_se dead-branch refactor; mean_uncertainty NaN guard.
- 8 follow-up tests: msprt zero-variance, partial-majority binary-search
activation, score_uncertainty exact boundaries, rolling_kl identical
+ disjoint, validate_budget + validate_threshold exact endpoints,
_signal_from_thumbs boundaries, no-heavy-top-level-imports source-grep
guard across all 5 new util modules.
Step 6 smoke verified for all 5 commands + 6 failure-mode rejection
paths.
CRLF gotcha note for future maintainers: PowerShell wrote the smoke
fixtures with a UTF-8 BOM on Windows during Step 6 — switched to
inline Python for the fixture write. Production CLI input handling is
already BOM-tolerant (utf-8-sig in JSONL loaders via v0.40.1 Part E).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
0d6f95181a |
feat(v0.62.0): RAG & Activation Steering — RAFT + RA-DIT + soup steer + citation-faithful + GRACE codebook
5 Parts shipping wedge 14 of the roadmap (RAG-aware fine-tuning +
activation steering + lifelong edit codebook). Schema-only release;
live training loops + decode-hook intervention + codebook lookup all
land in v0.62.1 (mirrors v0.50.0 / v0.52.0 / v0.61.0 stub-then-live).
Part A — RAFT data format: new data.format='raft' schema +
_convert_raft validator (64 KiB per-field cap, 64-distractor cap,
null-byte rejection on every field) + raft-llama3-8b recipe.
Part B — RA-DIT two-stage: TrainingConfig.ra_dit_stage Literal
{retriever, generator} + ra_dit_retriever_model field + closed
allowlist + cross-validator enforcing stage to base-task pairing
(retriever to embedding, generator to sft) + 2 recipes.
Part C — soup steer (CAA / ITI / RepE): closed-allowlist control-vector
methods + validate_steering_method/name/strength + Typer subcommands
train/apply/list + soup serve --steer/--steer-strength flags +
steering_vector Registry artifact kind. apply_steering +
build_steering_vector deferred-live stubs raise NotImplementedError
with v0.62.1 marker after validating inputs.
Part D — Citation-faithful FT: score_citations precision/recall/F1
kernel + extract_citation_ids public API + citation_faithful /
citation_style / citation_recall_threshold schema. Cross-validator:
citation_faithful=true requires data.format='raft' AND task in
{sft, pretrain} (silent-no-op footgun rejection mirroring v0.52.0
distill / classifier task-gate policy).
Part E — GRACE codebook: GraceCodebookConfig + bounded size [1, 100k]
+ bounded dim [1, 16384]. Extends v0.61.0 SUPPORTED_EDIT_METHODS
allowlist with 'grace'; apply_edit routes grace plans to v0.62.1
marker while legacy rome/memit/alphaedit retain v0.61.1 marker
(regression-guarded via TestEditMarkerRegressionGuard).
Test count: 9571 -> 9786 (+215 net). 4 review-agent waves resolved
0 CRITICAL + 0 HIGH + 4 MEDIUM + 11 LOW (broken list_steers registry
context-manager + dict-key access; missing version bump;
citation_faithful task-gate; shared TOCTOU helper delegation; Rich
markup escape on --steer exception messages; --base length cap +
null-byte rejection; typing.Iterable -> collections.abc.Iterable
migration; except Exception -> except ImportError narrowing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
740832e1b4 |
feat(unlearn/edit): v0.61.0 — Unlearning & Knowledge Edit (NPO/SimNPO/RMU + ROME/MEMIT/AlphaEdit)
5 Parts shipping schema + CLI surface for two of the most under-served axes in fine-tuning: GDPR right-to-be-forgotten unlearning (the legal-liability axis upstream TRL avoids) and surgical knowledge editing (research-coded everywhere, productized nowhere). Schema-only release; live trainer + kernel wiring deferred to v0.61.1 (matches established v0.50.0 / v0.52.0 / v0.53.0 stub-then-live cadence). Part A — task='unlearn' + NPO/SimNPO/RMU allowlist + UnlearnTrainerWrapper + data.forget_set / data.retain_set + training.unlearn_method/_alpha Part B — soup eval unlearning (TOFU/MUSE/WMDP) with Forget Quality + Model Utility + PrivLeak kernels + OK/MINOR/MAJOR taxonomy; bundled TOFU mini-fixture under soup_cli/data/_fixtures/unlearning/ Part C — soup edit set (ROME/MEMIT/AlphaEdit) + EditPlan + per-method default layer; --plan-only ships live, apply_edit kernel deferred Part D — Sequential edit governor: norm-blowup detection (OK/WARN/BLOWUP), auto-switch ROME→AlphaEdit at edit#10 or BLOWUP, refuses past cap Part E — soup edit diff: cwd-contained probe loader, atomic JSONL out, shape + table renderer (live before/after generation v0.61.1) Net: +125 tests (9446 → 9571), +5 utility modules + 1 trainer wrapper + 2 commands. Review-fix coverage: 0 CRITICAL + 5 HIGH + 11 MEDIUM + 11 LOW. All ruff + pytest green; Step 6 smokes (CLI plumbing + happy paths + 5 schema rejection paths) confirmed end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
f3c40e7753 |
feat(security): v0.60.0 — Supply Chain Security wedge (adapter scan/sign/verify, strict-safetensors, namespace-pin, license-matrix, airgap-bundle)
Six controls that hosted vendors structurally can't provide: - soup adapters scan: spectral backdoor scanner (rank-1 dominance + energy concentration + NaN/Inf + Frobenius outlier via robust median+MAD); pure numpy, reuses v0.57.0 adapter_diff loader - soup adapters sign / verify: Merkle-root manifest in .soup-signature.json; recursive file enumeration (catches nested tokenizer/ tamper); sigstore + ed25519 backends stub-then-live (v0.60.1) - soup adapters check-safetensors: closed 8-entry unsafe-extension allowlist; strict exit 3 for CI gating - NamespacePinStore: TOFU SQLite anti-AI-Jacking; author + created_at fingerprint compared via datetime.fromisoformat for offset-aware order; bool opt-in rejected so --allow-namespace-shift cannot be a free-for-all - License-conflict matrix: 33 SPDX-ish ids in MappingProxyType compat table; soup adapters merge --license <id> --license-override <reason> gate - soup airgap-bundle: signed tarball with deterministic dataset labeling (sorted basename, NOT argv order); TOCTOU lstat+S_ISLNK on parent + output; atomic os.replace; tarfile.data_filter for future extractall callers Test count: 9294 -> 9446 (+152 net across 6 new test files). Review-fix coverage across 5 waves (python / security / code / tdd / smoke): 0 CRITICAL + 12 HIGH + 11 MEDIUM + 6 LOW fixed before commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
6d44f0f931 |
feat(governance): v0.59.0 — CycloneDX/SPDX BOM + in-toto/SLSA-3 attest + Annex XI/XII + audit-log + repro-receipt + energy schema
Six Parts ship the procurement-floor moat — every Soup run can now emit
the formats regulated orgs demand, with no SaaS structurally able to
follow. Pure orchestration on top of v0.26 Registry + v0.34 cost
tracker + v0.56 diagnose — schema + atomic-write surface only; live
Sigstore signing, CodeCarbon hook, PDF rendering deferred to v0.59.1.
Part A — `soup bom emit` CycloneDX 1.6 ML-BOM + SPDX 2.3 AI-profile
dual emitter from any RegistryEntry. SHA-256 validation on every sha
field, license-id chain, base-model component with hash, per-artifact
file components, energy properties under metadata.properties.
Part B — `soup attest emit` in-toto v1 Statement wrapping SLSA-3
provenance v1 predicate. Stage allowlist (extract/train/eval/export/
publish), subject SHA locked to 64-hex, builder_id capped, SignatureBackend
enum with UNSIGNED live + SIGSTORE/ED25519 stubs raising NotImplementedError
with explicit v0.59.1 marker.
Part C — `soup train --annex-xi` EU AI Act Annex XI Sections 1+2 +
Annex XII Article 53(1)(d) markdown auto-doc. Top-10 domain cap,
modality breakdown, FLOPs/kWh/CO2. `_md_escape` neutralises |[](){}!<>
plus newline/CR/tab in every operator-controlled field — defends
against forged-heading + Markdown-link injection in downstream PDF/HTML
renderers (mirrors v0.29.0 model-card v2 policy).
Part D — `soup audit-log tail/rotate` HIPAA/SOC2-shaped JSONL with
PII redaction across every string field via v0.40.3 _SECRET_RE policy.
POSIX O_NOFOLLOW on append + 0o600 perms + lstat-based symlink rejection
at rotation backup path (no lexists race). SOUP_AUDIT_LOG_PATH env
override containment-checked to $HOME / $CWD / $TMPDIR.
Part E — `soup train --repro-receipt` SR 11-7-style receipt: seeds
(torch/numpy/python), kernel versions (CUDA/cuDNN/NCCL via best-effort
torch probes), GPU model + driver, OS + arch, Python version. Atomic
write, cwd-contained.
Part F — CodeCarbon hook schema + electricityMap SSRF validator with
full parity to v0.51.0 hubs.validate_hub_endpoint (scheme allowlist,
loopback-only HTTP, RFC1918 / link-local / reserved / multicast IP
rejection via ipaddress.ip_address, control-char + null-byte
rejection). PUE math + attach_energy populating BomEntry.
Cross-cutting: new paths.atomic_write_text shared TOCTOU-safe helper
centralises the v0.33.0 #22 / v0.43.0 / v0.55.0 / v0.56.0 / v0.57.0
/ v0.58.0 atomic-write pattern from four separate copies into one
single-source-of-truth (mirrors v0.40.6 / v0.53.5 peft_wiring policy).
Four review waves (python-reviewer + general-purpose security/code/tdd):
0 CRITICAL + 8 HIGH + 12 MEDIUM + 4 LOW resolved before commit.
HIGH fixes: audit-log lstat-before-write TOCTOU, O_NOFOLLOW on
append, redaction extended to host_id/operator_id/command, audit-log
env override containment, bom artifact size_bytes validation,
BomEntry attach_energy type-hint fix, default_log_path public symbol,
duplicated seeds validation removed.
Test count 9193 → 9294 (+99 net in tests/test_v0590.py; 93 pass +
6 POSIX-skipped on Windows for symlink rejection branches). v0.58.0
floor-check assertions widened from exact-match in test_v0580.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
b344aa881a |
feat(loop): soup loop CLI-first data flywheel capstone (v0.58.0)
Connects 8 existing uniques into one workflow: production traces -> preference pairs -> Eval-Gated DPO -> canary deploy -> rollback, all from a single CLI with budget guardrails and per-iteration replay. Modules (live): - utils/loop_state.py: LoopState frozen + atomic .soup/loop.yaml I/O - utils/canary_router.py: deterministic SHA-256 routing + BucketStats - utils/loop_budget.py: parse_budget_string + check_budget + UTC rollover - utils/loop_iteration.py: IterationRecord + write/read/list manifests - utils/loop_daemon.py: WatchConfig + run_once + watch daemon - commands/loop.py: init / status / pause / resume / watch / canary / replay Three review waves fixed 1 CRITICAL + 7 HIGH + 9 MEDIUM + 2 LOW total: python-review wave 1 (BucketStats lock scope + TOCTOU lstat-before-write on _check_path + init_state + NUL-byte on _bucket_for_key); code-review wave 2 (watch preserves paused / budget-skip writes no manifest / canary autoroll persisted to LoopState / route() math.ceil for sub-bucket predictability / parse_budget_string usd-only friendly error / list_iterations swallows OSError / module-top replace import); security + tdd wave 3 (_check_dir TOCTOU mirrors _check_path pattern, exact-boundary tests at _MAX_STR_FIELD=512 and _MAX_FILE_BYTES=1 MiB, bool-rejection on 4 counters, empty-string rejection on 3 optional-str). verification-loop: manual CPU smoke covering init / status / pause / resume / watch --max-iterations / canary / replay end-to-end. Notes: - ASCII arrows (->) in user-facing help text (CI test_help_output_is_ascii_safe). - Source-grep tests use Path(__file__).resolve().parent.parent for cwd- independence (defends against monkeypatch.chdir side-effects from earlier tests in the suite). - Stage callbacks ship as no-op stubs; v0.26 trace-to-pref / eval-gate / v0.30 multi-adapter deploy wiring is operator-driven via WatchConfig fields. Pre-wired versions tracked for v0.58.1. Test count: 8998 -> 9193 (+195 net in tests/test_v0580.py). Lint clean. Full repo pytest green (9105 pass + 53 skipped pre-fixes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
7da82d355a |
feat(adapters): v0.57.0 — git for LoRA (diff, merge, blame, branch)
Ships "soup adapters {diff,merge,blame,branch,checkout,branches}" — git-shaped
UX for LoRA adapter management. Pure-numpy math (no torch); TOCTOU defence on
every path; atomic writes throughout.
Part A — adapters diff (utils/adapter_diff.py, ~270 LOC):
Per-layer ΔW Frobenius norm + relative drift; effective-rank delta via
SVD entropy; top-K changed projections; JSON/Markdown/table output.
Part B — adapters merge (utils/adapter_merge.py, ~280 LOC):
Four strategies — linear (weighted avg), ties (Yadav et al. — trim/
elect-sign/disjoint), dare (Yu et al. — drop+rescale, deterministic),
svd (low-rank reconstruction). MergeReport.verdict='UNKNOWN' is a stub;
live canary verdict via v0.55 eval gate ships in v0.57.1.
Part C — adapters blame (utils/blame.py, ~190 LOC):
Leave-one-out plan emitter + budget tracker (parse_budget mirrors v0.48.0
data_mix idiom). Per-shard work table + feasibility check. Live ablation
runner raises NotImplementedError v0.57.1 (mirrors v0.27.0 / v0.50.0 /
v0.56.0 stub-then-live pattern).
Part D — adapters branch / checkout / branches (utils/adapter_branch.py,
~230 LOC):
SHA-256 snapshot pointers under ~/.soup/branches/ (SOUP_BRANCHES_DIR
override, $HOME/$CWD/$TMPDIR-bounded). Drift detection on checkout —
refuses restore when source SHA != snapshot SHA. CRLF/null-byte
rejection on env override (mirrors v0.51.0 hub-endpoint policy).
5-agent review-fix wave (1 CRITICAL + 9 HIGH + 11 MEDIUM + 4 LOW):
- TIES tied-sign defaults to +1 (np.sign(0)==0 would silently zero all
tied parameters)
- load_branch / delete_branch reject symlinks via os.lstat + S_ISLNK
before read/unlink
- merge output safetensors + adapter_config.json atomic writes with
symlink target rejection at output path; source config size-capped
at 256 KB
- compute_adapter_diff weights-file path symlink-rejected via lstat
BEFORE is_file() (defends against .safetensors -> /etc/passwd escape)
- _count_dataset_rows opens via realpath captured at containment check
(closes TOCTOU window)
- diff --output write is atomic (tempfile + os.replace)
- SOUP_BRANCHES_DIR rejects every C0 control char, not just null
- SUPPORTED_STRATEGIES is now frozenset (matches v0.41.0+ allowlist
policy); STRATEGY_ORDER tuple preserved for canonical iteration
- 5× pytest.raises(Exception) tightened to FrozenInstanceError
- 2 zero-assertion Part D tests converted to real assertions
- Added: bool base_model rejection, top_k boundary 1/201, density=1.0
inclusive bound, inf weight rejection, tied-sign positive default,
bool False for num_shards/budget_seconds, POSIX symlink rejections
for diff weights / merge output / load_branch / delete_branch,
no-top-level-torch source-grep guards, traversal delete_branch.
Plus v0.56.0 follow-up: test_v0560.py version-floor tests widened from
exact-match to floor-check (matches v0.51.0 / v0.54.0 idiom — every
subsequent release would otherwise edit this one line).
Test count: 8849 → 8998 (+149 net in 4 new files; 4 POSIX-only symlink
tests skipped on Windows). Full suite green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
72189baba7 |
feat(diagnose): soup diagnose — post-training model report card (v0.56.0)
Six failure-mode probes (forgetting / refusal / format / mode_collapse / memorization / contamination) + FailureReport frozen dataclass + SVG badge + soup train --diagnose-gate. Same OK/MINOR/MAJOR taxonomy as v0.26.0 Quant-Lobotomy. - soup diagnose <run-id> [--evidence|--output|--badge|--attach-to-registry] - soup train --diagnose-gate <evidence.json> refuses MAJOR runs - diagnose_report added to registry._VALID_KINDS Review wave (4 agents): 4 HIGH + 8 MEDIUM + 2 LOW addressed — atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), realpath containment, evidence size cap, contamination combined-complexity cap, ReDoS probe, extras null-byte sanitisation, extract_row_text centralisation, tokenize delegates to _eval_text. Test count: 8676 -> 8849 (+123 in test_v0560.py + 50 net adjustments). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
58d7d510bf |
feat(eval): soup eval design — derive evals from data (v0.55.0)
Trainer libraries help you RUN evals — none help you DEFINE them.
v0.55.0 closes that gap with 5 new subcommands:
- soup eval design <data> --goal "..." → goal-conditioned EvalDesign
(TF-IDF salience + scorer dispatch)
- soup eval discover <data> → held-out canaries + memorization probes
(farthest-first Jaccard clustering)
- soup eval lock + soup eval coverage → SHA-256-checksummed artifact +
gap analysis vs v0.54.0 task taxonomy
- soup eval gate-install --baseline R → pre-push regression gate
(paired-bootstrap CI, shlex.quote)
- soup eval against B --candidate C → run-vs-run paired-bootstrap CI
Heuristic / CPU-only — no GPU required. Lazy imports across all 6 new
modules so `soup --help` startup remains < 200 ms.
New registry artifact kinds: eval_suite, canaries.
New tracker accessor: ExperimentTracker.get_metric_series(run_id, metric).
Security policy (all atomic-write + read surfaces):
- cwd containment via os.path.realpath + commonpath
- unconditional os.lstat + stat.S_ISLNK rejection (TOCTOU defence)
- atomic write via tempfile.mkstemp + os.replace
- shlex.quote for shell-script generation (NO hand-rolled escape)
- MappingProxyType on every registry / metric / scorer map
- frozen dataclass on every public return type
- bool-as-int rejection on every numeric input
- DoS caps: 10k subsample for TF-IDF + clustering hot paths
Review-fix coverage across 4 agents (python / security / code / tdd):
0 CRITICAL + 7 HIGH + 11 MEDIUM + 6 LOW resolved before commit.
Tests: 8571 → 8676 (+105 net).
Lint: ruff clean.
Smoke: every CLI command + every failure mode exercised in /tmp/soup_smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
600686cd70 |
feat(advise): soup advise — pre-flight decision (v0.54.0)
`soup advise <data.jsonl> --goal "..."` returns one of PROMPT_ENG / RAG / SFT / DPO / GRPO with a confidence, reason, and reverse-when criterion BEFORE the user spends 8 hours on a GPU. Layer above autopilot — autopilot picks hyperparams AFTER the training decision; advise picks the training decision itself. Three Parts: - Part A: Verdict engine — TASK_CATEGORIES + CHOICES allowlists, frozen Verdict / DatasetProfile / ROIEstimate dataclasses, pure- Python classify_task + compute_dataset_profile + build_verdict rubric (DPO / GRPO floor 500 / PROMPT_ENG floor 50 / RAG / SFT). - Part B: Probe runner — synth_probe_baselines + synth_probe_lora_delta heuristic stubs with forward-compat model/device/lr/timeout_seconds kwargs (v0.54.1 lifts to live model loading per stub-then-live cadence used by v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus). - Part C: Cross-project learning — ~/.soup/advise_history.jsonl with cross-process file locking (fcntl on POSIX, sidecar <path>.lock + msvcrt on Windows). `soup advise compare` reads history; env override SOUP_ADVISE_HISTORY_PATH containment-checked to $HOME / $CWD / tempdir (mirrors v0.36.0 SOUP_BATCH_CACHE_PATH policy). CLI: Typer subcommand group `run` / `explain` / `compare` plus argv preprocessor in cli.py that maps `soup advise data.jsonl` → `soup advise run data.jsonl`. Scoped to argv[1] == "advise" only (code-review HIGH fix — defends against rewrites when an unrelated arg contains the literal string "advise"). Schema: AdviseConfig (goal / probe / record) field on SoupConfig honors the plan's cross-cutting bullet. Security: cwd-containment + os.lstat + S_ISLNK symlink reject on every path input; atomic writes via tempfile.mkstemp + os.replace on scratch + history; per-line 64 KB cap + 16 MiB file cap on history reads; bool / finite / NUL / oversize guards on every public input; Rich markup escape on user-controlled output. Reviewed by python / code / security / tdd / architect agents — every finding fixed before commit (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW). Test count: 8400 → 8571 (+136 in tests/test_v0540.py, +35 net adjustments to v0.53.x version-pin assertions to forward-compat >=). Note: Windows CRLF / LF warnings during stage are .gitattributes- governed and benign. CI runs on ubuntu-latest / windows-latest / macos-latest × Python 3.9 / 3.11 / 3.12. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
cfdabf2b3b |
feat(v0.53.11): GRPO Plus finish + preference live
Closes v0.50.1 (#123, #126, #127), v0.49.1 (#119), v0.40.1 (#68). #123 — live math kernels for 6 GRPO variants (gspo/dapo/dr_grpo/bnpo/ two_sided/rft) + `_GRPOTrainerVariant` HF Trainer subclass via `make_grpo_trainer_variant` factory. Variant compute_loss reads kernel inputs FIRST (no double-forward); falls back to super() only on missing attrs. Case-insensitive variant normalisation before lru_cache. #126 — PRMTrainerWrapper + `_PRMTrainer` HF Trainer subclass with real compute_loss (gather hidden states at step_positions -> reward_head -> MSE via compute_prm_loss). Dataset wrapped in datasets.Dataset.from_list for HF Trainer compatibility. Bool-before-isinstance guard on batch_size. #127 — GRPOStabilityCallback inherits transformers.TrainerCallback (lazy), live EMA ref-model update in on_step_end with strict=True + fallback-to-strict=False-with-WARNING on key mismatch (silent corruption defence). math.isfinite guard on alpha. #119 — LongLoRA forward override via LongLoRAForwardOverride context manager with idempotent install (_soup_longlora_patched marker prevents re-entry double-wrap), 256-char class name cap on regex match, restore on __exit__ AND on exception. #68 — true per-batch weighted-sum preference combine reading policy/ref logps from TRL inputs + each compute_*_term kernel + combine_losses. Explicit None checks on trainer attrs (no `or` on possibly-tensor), DEBUG log on per-term skip. Review fixes from 4 agents (python/code/security/tdd): 10 HIGH + 8 MEDIUM + 7 LOW — see CLAUDE.md v0.53.11 entry for the full list. Test count: 8330 -> 8400 (+75 in test_v05311.py: 54 initial + 21 review-fix coverage gaps). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
76f033a6ff |
feat(v0.53.10): Quick wins + packaging + UX wiring
7 issues closed: - #150 [mix] pyproject extra bundles scikit-optimize so `soup data mix --optimize` runs the Bayesian loop instead of the v0.48.0 Dirichlet fallback; new describe_default_optimizer() helper labels the active backend without paying skopt's import cost. - #113 [data-pro] extras (langdetect + presidio-analyzer) with lazy fall-through helpers in utils/data_score (broader language coverage + Presidio entity recognition on top of the v0.47.0 regex baseline). Llama-Guard-3-1B documented as a manual recipe (license + size). - #154 SOUP_POSTHOG_KEY / SOUP_POSTHOG_ENDPOINT env override via sentinel-based explicit-vs-env precedence; HTTPS-only + RFC1918/link-local rejection on the endpoint; null-byte / control-char / >256-char rejection on the key. - #152 --hub flag plumbed on chat / serve / infer / merge / export / push via shared utils/hubs.apply_hub_to_cli_model + prefetch_model_from_hub helpers; push uses upload_repo (skips HF-specific Collections + model-card auto-render on non-HF hubs). - #153 `soup data download --hub modelscope|modelers` live SDK (lifts the v0.53.8 advisory-only path); friendly ImportError advisory when the SDK is missing. - #155 Web UI Tool Outputs panel — `loadToolOutputs` polls /api/tool-outputs every 3s; XSS-safe DOM-built table (textContent per cell, no innerHTML for user-controlled fields); Bearer token threaded via the v0.53.9 window._authToken bootstrap. - #156 SoupTrainerCallback.on_step_end records tool_calls counts from kwargs['inputs'] into the global tool buffer. Best-effort (# noqa: BLE001 per project policy — training must never crash). 13 review-fixes applied (4 HIGH / 5 MEDIUM / 4 LOW): - HIGH PostHog explicit-endpoint precedence sentinel - HIGH absolute path leak in local_path advisory reduced to relpath - HIGH Rich markup escape on base / local_path / cache_dir - HIGH callback # noqa: BLE001 per project policy - MED `import time` moved out of try block - MED oversize key + explicit-empty key rejection tests - MED source-grep regression guards (advisory-removal, helper imports across 5 non-push commands) - MED `prefetch_model_from_hub` outside-cwd cache_root rejection - LOW empty-list + bool-True tool_calls no-op tests - LOW push.py uses upload_repo + validate_hub_name regression guard Test count: 8285 -> 8330 (+45 in tests/test_v05310.py). Full suite green; ruff clean; on Win+Py3.10. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
bde9d149a0 |
feat(v0.53.9): Live Dashboard + UX + Bench + Standalone CLIs
Eight features that close out the v0.44.x live-monitoring deferrals plus a long tail of standalone CLI wins: - #94 /api/train/stream async SSE with per-subscriber cursor + JS EventSource consumer; SoupTrainerCallback pushes TrainEvent on each on_log. - #95 soup ui --public derives LAN IP via SOCK_DGRAM connect-trick, prints scannable QR; --auth-token override; SPA bootstrap hydrates window._authToken from ?token= + sessionStorage and cleans the URL via history.replaceState; CORS regex auto-widens to loopback + RFC1918 in public mode; set_auth_token rotation race fixed via threading.Lock. - #98 soup serve --reasoning-parser strips <think>...</think> (and OpenThinker tags); pre-compiled regex with marker-token fast-path + 1 MiB cap + leading-newline-only strip. - #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON endpoint; best-effort observation hook in callback.on_log. - #15 soup tokenizer train: BPE training CLI with raw-path lstat symlink rejection, 50 MiB total / 8 KiB per-line caps, post-mkdir output-dir re-check, --special-token NUL/oversize dedup, vocab bounds [256, 200000]. - #26 soup bench --p50 --p95 renders extra per-prompt tail-latency Rich table; --prompts-file gains symlink rejection. - #28 soup bench --backend auto: MLX weights.npz probe (per-entry lstat) -> config.json model_type keyword -> transformers fallback; SOUP_BENCH_BACKEND env hint. - #12 examples/synthetic_workflow.{md,yaml} end-to-end walkthrough. Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the python / code / security / tdd review agents. Notable HIGH: - QR token now consumed by SPA (was unreachable previously). - set_auth_token rotation lock-protected, 8-thread stress tested. - Tokenizer input + output symlink TOCTOU defence on raw path. - SSE generator switched to async (asyncio.sleep) for non-blocking multi-subscriber operation. - CORS regex for --public LAN mode (the old fixed allowlist of http://0.0.0.0:port never matched a real Origin header). - _has_mlx_weights per-entry lstat so a symlinked weights.npz can't trigger MLX dispatch. Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the relaxed v0.53.8 version-pin asserts in tests/test_v0538.py). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
53bb82afeb |
fix(v0.53.8.1): hatch artifacts directive — fix PyPI duplicate-filename 400
v0.53.8 PyPI publish failed with: 400 Invalid distribution file. ZIP archive not accepted: Duplicate filename in local headers Root cause: `[tool.hatch.build.targets.wheel.force-include]` shipped `soup_cli/data/_fixtures/` AND `packages = ["soup_cli"]` recursed into the same path, so both the wheel and sdist contained each JSONL twice. Fix: switch from force-include to `artifacts = [...]` which adds non-Python files to the existing package tree exactly once. Standard hatchling pattern for shipping data files inside an already-packaged directory. Version bumped to v0.53.8.1 (patch) — same code surface, just a build config fix. v0.53.8 GitHub release remains as the feature changelog; PyPI ships under v0.53.8.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
6d2170c4f3 |
feat(v0.53.8): Remote data + Hubs + Trackers (wave 2) — 6 features
- #85 fsspec live loaders — data/loader.py routes the v0.42.0 fsspec scheme allowlist (s3:// / gs:// / gcs:// / az:// / abfs:// / abfss:// / oci://) through fsspec.open with validate_remote_uri containment BEFORE connection. Friendly Rich panel names the pip install advisory when the backend SDK is missing. Threads data.streaming + data.buffer_size. Row count capped at 1M. - #130 Hub dispatcher live — utils/hubs.download_repo() and upload_repo() lazy-import per backend (huggingface_hub / modelscope / openmind_hub). Shared _validate_repo_id_shape (bool / null-byte / leading-slash / .. / control-char / oversize) + cwd containment on local_dir / folder_path. commands/train.py pre-fetches non-HF base into .soup_hub_cache/ (sanitised slug, idempotent on resume, cfg.base updated via model_copy). soup data download --hub flag plumbed. Multi-command rollout for chat / serve / infer / merge / export / push tracked for v0.53.9. - #89 [trackers] pyproject extra bundles mlflow / swanlab / trackio; tracker_missing_dep_message surfaces a friendly pip install advisory via importlib.util.find_spec (non-executing probe). - #90 utils/trackers.send_telemetry_payload — opt-IN via SOUP_TELEMETRY=1; lazy httpx; 1s hard timeout; HTTPS-only with SSRF re-validation (mirrors v0.51.0 hub endpoint policy); silent-fail on every exception. - #93 Fixtures migrated to soup_cli/data/_fixtures/ — zipapp / namespace-package safe via [tool.hatch.build.targets.wheel.force-include]; _bundle_source_path falls back to examples/data/ for editable installs. - #69 utils/hf_space.detect_space_sdk(requirements_text) — picks "streamlit" / "gradio" from the rendered requirements.txt; closes the v0.40.2 known limitation that custom Spaces always defaulted to gradio. Wired into commands/deploy.py. Review pass: python-review + code-review + security-review ran in parallel; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). Highlights: cwd-containment on local_dir/folder_path, Windows ..\ traversal defence on .soup_hub_cache slug, Pydantic model_copy(update=...) instead of attribute mutation, idempotent pre-fetch via cache probe, 1M-row cap on remote materialisation, SSRF re-validation on telemetry endpoint override, 256 KB cap on detect_space_sdk input, modelscope.push_model commit_message kwarg removed (would TypeError at runtime), find_spec instead of __import__ to avoid swanlab side-effects. Test count: 8162 -> 8257 (+66 in tests/test_v0538.py + 29 net adjustments). Lint clean. CPU smoke: version, --help, load_config_from_string with hub: modelscope passes; mlx + non-HF rejected; data download --hub modelscope advisory rendered; detect_space_sdk live on real requirements.txt bodies; package-data fixtures resolve from soup_cli/data/_fixtures/. v0.53.7 known limitation #1 (bash 501 marker) bumped to v0.53.9. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
18d8b36114 |
feat(v0.53.7): Data Forge + Pipeline live (wave 1) — 11 items
Closes v0.47.0 deferrals (#111, #112), community QA (#75), v0.42.1 wave 1 (#87, #86, #88), and 5 v0.53.6 stub-to-live carry-overs (#102 vLLM parity + SSE streaming, #103 tool HTTP endpoints, #105 instantiate_trainer_plugins, #106 run_recipe DAG runner). - #88 markdown ingest heading split (split_markdown_by_headings) - #112 soup data decontaminate --benchmark-file (cwd-contained operator corpus + symlink rejection) - #87 prompt_strategy live resolver (resolve_prompt_strategy + lru_cache, importlib-based; per-row hook in sft_format.py) - #86 soup data preprocess AOT tokenize (atomic Arrow shard write + cache metadata sidecar; SFT + Pretrain wrappers short-circuit on format='pre_tokenized' + tokenized_path with cache-hash gate) - #111 forge --judge-provider {ollama,anthropic,vllm} live (lazy v0.20.0 providers; SSRF parity) - #75 QA log entry for synth-data provider manual smoke - #106 run_recipe LIVE for 6 NODE_KINDS (seed / llm_text / code / judge / validator / sampler); atomic checkpoint via tempfile.mkstemp; resume rehydrates predecessor outputs from per-node sidecar JSONL; lstat-on- raw-path symlink rejection (v0.33.0 #22 TOCTOU parity); failed_reason path-redacted - #105 instantiate_trainer_plugins LIVE for cce_plugin / grokfast / spectrum / llmcompressor / sonicmoe / math_verify (lazy imports, friendly pip-install advisory on missing dep) - #103 POST /v1/tools/python + /v1/tools/web_search LIVE (Bearer auth gate, deny-by-default domain allowlist, 5s timeout, 5-result cap). bash reverted to HTTP 501 — security review caught /bin/sh -c child escapes RLVR sandbox's OS-level isolation; deferred to v0.53.8. - #102 vLLM /v1/messages parity LIVE on both backends; CORS loopback-only - #102 Anthropic-shape SSE streaming on /v1/messages LIVE; Cache-Control: no-store; CRLF/NUL/oversize strip on model+msg_id (header injection) Review fixes (1 CRITICAL + 11 HIGH + 17 MEDIUM + 10 LOW from python-reviewer + code-reviewer + security-reviewer + tdd-guide) all addressed in this commit. Test count: 8051 → 8162 (+111 in tests/test_v0537.py). Test files: 189 → 190. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|
|
Alpamys
|
70326f1aa7 |
feat(v0.53.6): plugin callback + Anthropic /v1/messages + n-gram spec + 3 stubs
Ships v0.53.6 "Plugin + Agent + Anthropic API" — 6 features, 3 live and 3 stub-then-live with v0.53.7 markers. LIVE - #101 SoupPluginCallback fans HF Trainer events to enabled Soup plugin hooks (pre_train/post_train/pre_step/post_step). Hook exceptions swallowed at WARNING. Hook snapshot collected once and passed to ctor (race-free per code-review fix). Wired into all 13 transformer-backend trainers via utils/peft_wiring.attach_plugin_callback. - #102 POST /v1/messages on transformers backend reuses the v0.45.0 anthropic_messages converter + existing chat handler. Streaming -> 501. Validation errors -> generic 'Invalid request' body, details at DEBUG (security-review redaction fix). - #104 n-gram speculative decoding: NgramSpecConfig.num_draft_tokens threaded through model.generate(prompt_lookup_num_tokens=N). Mutually exclusive with assistant_model. STUB-THEN-LIVE (v0.53.7) - #103 /v1/tools/{python,bash,web_search} return HTTP 501. - #105 instantiate_trainer_plugins validates then NotImplementedError. - #106 run_recipe + 'soup data recipe --execute --output <dir>' with CLI-side is_under_cwd containment before the live runner. Tests: 7998 -> 8051 (+53 in tests/test_v0536.py). Lint clean. Three review agents (python / code / security); every HIGH/MEDIUM fixed. TDD coverage gaps closed (ngram-None regression, max_tokens cap, run_recipe boundaries, console-print failure swallow). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |