`soup advise <data.jsonl> --goal "..."` returns one of PROMPT_ENG /
RAG / SFT / DPO / GRPO with a confidence, reason, and reverse-when
criterion BEFORE the user spends 8 hours on a GPU. Layer above
autopilot — autopilot picks hyperparams AFTER the training decision;
advise picks the training decision itself.
Three Parts:
- Part A: Verdict engine — TASK_CATEGORIES + CHOICES allowlists,
frozen Verdict / DatasetProfile / ROIEstimate dataclasses, pure-
Python classify_task + compute_dataset_profile + build_verdict
rubric (DPO / GRPO floor 500 / PROMPT_ENG floor 50 / RAG / SFT).
- Part B: Probe runner — synth_probe_baselines + synth_probe_lora_delta
heuristic stubs with forward-compat model/device/lr/timeout_seconds
kwargs (v0.54.1 lifts to live model loading per stub-then-live
cadence used by v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus).
- Part C: Cross-project learning — ~/.soup/advise_history.jsonl with
cross-process file locking (fcntl on POSIX, sidecar <path>.lock +
msvcrt on Windows). `soup advise compare` reads history; env
override SOUP_ADVISE_HISTORY_PATH containment-checked to $HOME /
$CWD / tempdir (mirrors v0.36.0 SOUP_BATCH_CACHE_PATH policy).
CLI: Typer subcommand group `run` / `explain` / `compare` plus argv
preprocessor in cli.py that maps `soup advise data.jsonl` →
`soup advise run data.jsonl`. Scoped to argv[1] == "advise" only
(code-review HIGH fix — defends against rewrites when an unrelated
arg contains the literal string "advise").
Schema: AdviseConfig (goal / probe / record) field on SoupConfig
honors the plan's cross-cutting bullet.
Security: cwd-containment + os.lstat + S_ISLNK symlink reject on
every path input; atomic writes via tempfile.mkstemp + os.replace
on scratch + history; per-line 64 KB cap + 16 MiB file cap on
history reads; bool / finite / NUL / oversize guards on every public
input; Rich markup escape on user-controlled output.
Reviewed by python / code / security / tdd / architect agents — every
finding fixed before commit (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW).
Test count: 8400 → 8571 (+136 in tests/test_v0540.py, +35 net
adjustments to v0.53.x version-pin assertions to forward-compat >=).
Note: Windows CRLF / LF warnings during stage are .gitattributes-
governed and benign. CI runs on ubuntu-latest / windows-latest /
macos-latest × Python 3.9 / 3.11 / 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes v0.50.1 (#123, #126, #127), v0.49.1 (#119), v0.40.1 (#68).
#123 — live math kernels for 6 GRPO variants (gspo/dapo/dr_grpo/bnpo/
two_sided/rft) + `_GRPOTrainerVariant` HF Trainer subclass via
`make_grpo_trainer_variant` factory. Variant compute_loss reads kernel
inputs FIRST (no double-forward); falls back to super() only on missing
attrs. Case-insensitive variant normalisation before lru_cache.
#126 — PRMTrainerWrapper + `_PRMTrainer` HF Trainer subclass with real
compute_loss (gather hidden states at step_positions -> reward_head ->
MSE via compute_prm_loss). Dataset wrapped in datasets.Dataset.from_list
for HF Trainer compatibility. Bool-before-isinstance guard on batch_size.
#127 — GRPOStabilityCallback inherits transformers.TrainerCallback
(lazy), live EMA ref-model update in on_step_end with strict=True +
fallback-to-strict=False-with-WARNING on key mismatch (silent corruption
defence). math.isfinite guard on alpha.
#119 — LongLoRA forward override via LongLoRAForwardOverride context
manager with idempotent install (_soup_longlora_patched marker prevents
re-entry double-wrap), 256-char class name cap on regex match, restore
on __exit__ AND on exception.
#68 — true per-batch weighted-sum preference combine reading policy/ref
logps from TRL inputs + each compute_*_term kernel + combine_losses.
Explicit None checks on trainer attrs (no `or` on possibly-tensor),
DEBUG log on per-term skip.
Review fixes from 4 agents (python/code/security/tdd): 10 HIGH + 8
MEDIUM + 7 LOW — see CLAUDE.md v0.53.11 entry for the full list.
Test count: 8330 -> 8400 (+75 in test_v05311.py: 54 initial + 21
review-fix coverage gaps).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpamys