The list/info portions of 4f54179 follow the structure of @dreamer0129's
PR #175 draft. Adding Co-Authored-By trailer so GitHub contributors graph
reflects the original draft author.
Co-Authored-By: dreamer0129 <185807357+dreamer0129@users.noreply.github.com>
Closes#174. Picked up after PR #175 (dreamer0129) went quiet — the
list/info path was clean, but compare() escaped only inside the
highlight branch, leaving a shared crafted base_model = "[link=evil]
click[/]" un-escaped on equal-value rows.
Fix follows the "escape always at the value layer, decoration wraps
after" pattern mirroring v0.57.0 `adapters diff` / `info`:
- list_adapters: wrap base / lora_r / peft_type / rel_path with
rich.markup.escape() before table.add_row(); also escape
adapter_path in the JSONDecodeError fallback.
- info: wrap base_model / peft_type / task_type / lora_r / lora_alpha
/ lora_dropout / modules_str inside the Rich Panel f-string; also
escape adapter_path.name in the Panel title.
- compare: escape val1_str / val2_str unconditionally; [yellow]
highlight wraps already-escaped values when they differ. Equal-value
rows now also escape (was the v0.57.0 known-limitation gap).
+4 regression tests in tests/test_adapters.py::TestAdaptersMarkupEscape:
- test_list_escapes_crafted_base_model — asserts no ANSI hyperlink
sequence (\x1b]8;) leaks from a crafted [link=http://evil/...] payload.
- test_info_escapes_crafted_base_model — same assertion for Panel.
- test_compare_escapes_equal_crafted_values — the specific regression
for the PR #175 review gap (identical crafted values on both sides
must NOT smuggle live markup through the equal-branch).
- test_compare_escapes_differing_crafted_values — highlight branch
also escapes.
Closes v0.57.0 Known Limitation (9).
Verified locally:
- ruff check soup_cli/commands/adapters.py tests/test_adapters.py -> clean
- pytest tests/test_adapters.py --no-cov -> 20 passed
CI on ubuntu / macOS exposed a missed TOCTOU detail: `os.lstat(realpath(path))`
silently resolves symlinks before the lstat, so `S_ISLNK` never trips and a
symlinked input path passes the rejection. Windows CI was skipped (POSIX-only
symlink test), so the bug didn't surface in local smoke.
Three fixes, matching the v0.53.7 #106 project policy of "lstat the RAW path
before realpath":
- `edit_diff.load_probes` — lstat path first, then realpath after rejection.
- `unlearning_eval.load_evidence_file` — same.
- `unlearning_eval.get_fixture_path` — lstat raw candidate before realpath.
No new tests — the existing `test_symlink_rejected` covers it; it now passes
on POSIX where it previously failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three failures on CI run 26084542388 — all are version-pin / Rich-wrap
artefacts, not real regressions in v0.60.0 functionality:
- test_v0560 test_pyproject_version: regex-based >=0.56 floor check
(was substring `version = "0.5`)
- test_v0590 test_version_is_0_59 -> test_version_is_at_least_0_59:
>=0.59 floor (matches v0.51/v0.54 floor-check idiom)
- test_v0600_part_e merge_help_lists_license_flags: strip ANSI codes
before substring check (Rich splits `--license` across `\x1b[1;36m`
escapes in the wrapped Typer table)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v0.59.0 What's New block claimed +93 new tests, but the actual count
is +99 new tests in tests/test_v0590.py (9193 -> 9294). Numbers in
CLAUDE.md / SECURITY.md / CONTRIBUTING.md / the release notes already
have the correct count.
Docs-only hotfix; does not require a version bump (per release checklist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous test_default_log_path_rejects_null_byte_env used monkeypatch.setenv
to inject a null byte into SOUP_AUDIT_LOG_PATH and expected default_log_path
to fall back gracefully. But the OS layer rejects null bytes in env vars on
every platform we ship on:
- POSIX (Linux/macOS): `ValueError: embedded null byte`
- Windows: `ValueError: embedded null character`
The setenv call itself raises, never reaching default_log_path. Split into two
tests that hit the actual validation surfaces:
1. test_default_log_path_rejects_null_byte_override — calls the private
_validate_log_path_override helper directly with a null-byte string and
asserts it returns None (so the caller falls back to the safe default).
2. test_default_log_path_handles_env_read_value_error — monkeypatches
os.environ.get to raise ValueError, exercising the defence-in-depth
try/except around the env read in default_log_path().
Both tests pass on Linux + macOS + Windows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three macOS-3.11 CI failures in test_v0590.py post-merge:
1+2. test_train_annex_xi_flag_present_in_help / test_train_repro_receipt_flag_present_in_help
— Typer's Rich-renderer wraps long lines and inserts ANSI colour codes
BETWEEN the two dashes of `--annex-xi` / `--repro-receipt`, so the
literal substring match fails. Strip ANSI escape codes via regex before
asserting; also accept the bare option name as a defence-in-depth
fallback against future Rich line-wrap quirks.
3. test_default_log_path_rejects_null_byte_env — POSIX `os.environ.get` raises
`ValueError("embedded null byte")` when the env value contains a NUL
character, while Windows allows the read. Wrap the env read in
`try/except ValueError` so the function falls back to the safe default
(~/.soup/audit.jsonl) on either platform.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six Parts ship the procurement-floor moat — every Soup run can now emit
the formats regulated orgs demand, with no SaaS structurally able to
follow. Pure orchestration on top of v0.26 Registry + v0.34 cost
tracker + v0.56 diagnose — schema + atomic-write surface only; live
Sigstore signing, CodeCarbon hook, PDF rendering deferred to v0.59.1.
Part A — `soup bom emit` CycloneDX 1.6 ML-BOM + SPDX 2.3 AI-profile
dual emitter from any RegistryEntry. SHA-256 validation on every sha
field, license-id chain, base-model component with hash, per-artifact
file components, energy properties under metadata.properties.
Part B — `soup attest emit` in-toto v1 Statement wrapping SLSA-3
provenance v1 predicate. Stage allowlist (extract/train/eval/export/
publish), subject SHA locked to 64-hex, builder_id capped, SignatureBackend
enum with UNSIGNED live + SIGSTORE/ED25519 stubs raising NotImplementedError
with explicit v0.59.1 marker.
Part C — `soup train --annex-xi` EU AI Act Annex XI Sections 1+2 +
Annex XII Article 53(1)(d) markdown auto-doc. Top-10 domain cap,
modality breakdown, FLOPs/kWh/CO2. `_md_escape` neutralises |[](){}!<>
plus newline/CR/tab in every operator-controlled field — defends
against forged-heading + Markdown-link injection in downstream PDF/HTML
renderers (mirrors v0.29.0 model-card v2 policy).
Part D — `soup audit-log tail/rotate` HIPAA/SOC2-shaped JSONL with
PII redaction across every string field via v0.40.3 _SECRET_RE policy.
POSIX O_NOFOLLOW on append + 0o600 perms + lstat-based symlink rejection
at rotation backup path (no lexists race). SOUP_AUDIT_LOG_PATH env
override containment-checked to $HOME / $CWD / $TMPDIR.
Part E — `soup train --repro-receipt` SR 11-7-style receipt: seeds
(torch/numpy/python), kernel versions (CUDA/cuDNN/NCCL via best-effort
torch probes), GPU model + driver, OS + arch, Python version. Atomic
write, cwd-contained.
Part F — CodeCarbon hook schema + electricityMap SSRF validator with
full parity to v0.51.0 hubs.validate_hub_endpoint (scheme allowlist,
loopback-only HTTP, RFC1918 / link-local / reserved / multicast IP
rejection via ipaddress.ip_address, control-char + null-byte
rejection). PUE math + attach_energy populating BomEntry.
Cross-cutting: new paths.atomic_write_text shared TOCTOU-safe helper
centralises the v0.33.0 #22 / v0.43.0 / v0.55.0 / v0.56.0 / v0.57.0
/ v0.58.0 atomic-write pattern from four separate copies into one
single-source-of-truth (mirrors v0.40.6 / v0.53.5 peft_wiring policy).
Four review waves (python-reviewer + general-purpose security/code/tdd):
0 CRITICAL + 8 HIGH + 12 MEDIUM + 4 LOW resolved before commit.
HIGH fixes: audit-log lstat-before-write TOCTOU, O_NOFOLLOW on
append, redaction extended to host_id/operator_id/command, audit-log
env override containment, bom artifact size_bytes validation,
BomEntry attach_energy type-hint fix, default_log_path public symbol,
duplicated seeds validation removed.
Test count 9193 → 9294 (+99 net in tests/test_v0590.py; 93 pass +
6 POSIX-skipped on Windows for symlink rejection branches). v0.58.0
floor-check assertions widened from exact-match in test_v0580.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #178 wired the live NCCL bandwidth measurement on top of the
v0.43.0 nccl_bandwidth_check reference table. Four polish items
on top of the merged version:
- Bump timing samples 5 -> 10 and report MEDIAN rather than mean.
Median is robust to one-off jitter (kernel preemption, page
faults, OS scheduler hiccups) that can scew a 5-sample mean by
10-30%. Median of 10 gives a much more honest GB/s number.
- Bump warmup 1 -> 3 iters. First all_reduce in a fresh process
pays for CUDA kernel JIT plus the initial NCCL collective
handshake; one warmup isn't always enough to amortise that out.
3 warmups is the standard NVIDIA recommendation for collective
benchmarks.
- Time each iteration separately (collect a per-iter list, then
median) instead of averaging total elapsed / iters. Same headline
cost, but exposes per-call variance for future percentile output.
- Snapshot + restore MASTER_ADDR / MASTER_PORT around the spawn.
The worker sets them unconditionally, so prior to this patch a
second doctor invocation in the same process (test harness,
notebook, future TUI auto-refresh) would inherit stale env. Now
we restore exactly the prior state (including absent vars).
Plus three module-level constants (_NCCL_BENCHMARK_TENSOR_BYTES
/ _WARMUP_ITERS / _TIMED_ITERS) with comments explaining each
number so future tuning is informed, not magic. Status line now
prints the actual params so users see what we measured.
Existing tests (mock mp.spawn at the outer boundary) keep passing
unchanged; no new tests needed for this polish layer.
* feat(doctor): add --nccl flag to measure and validate multi-GPU bandwidth
* test(doctor): add mocked CUDA tests to verify --nccl skip and success behaviors
* docs(readme): document the new --nccl bandwidth check flag for the doctor command
Step 9 follow-up: add dedicated `## Adapter Management (git for LoRA)`
section to the README body so the surface is self-contained when the
single-slot `## What's New` block is overwritten in v0.58.0.
Step 10 follow-up: add v0.57.0 to the SECURITY.md supported-versions
list + a detailed entry in the per-version fix notes covering all
9 HIGH fixes (TIES sign-tie default, 4× symlink TOCTOU rejections,
atomic writes, env CRLF rejection, allowlist policy migration, etc.)
and the 7 known limitations.
Tracked follow-ups filed as GitHub issues #171–#174:
#171 — live blame ablation runner
#172 — merge canary verdict via v0.55 eval gate
#173 — branch pointers → v0.26 Registry lineage
#174 — Rich-markup backfill for legacy adapters list/info/compare
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POSIX setenv (and Windows equivalent) reject null bytes + control chars
at the syscall boundary, so monkeypatch.setenv("SOUP_BRANCHES_DIR",
"/some\x00path") raises ValueError on every CI runner before our code
ever sees the env var.
Stub os.environ.get directly so the helper's rejection branch is
exercised exactly as it would be if the env var arrived through some
other channel (subprocess env inheritance, in-process programmatic
mutation, etc).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rich-wrap-CI workaround — same fix pattern as v0.55.0 / v0.56.0:
CliRunner output contains ANSI color escapes that break literal
'--top-k' in output substring matches because Rich renders option
names as -\x1b[0m\x1b[1;36m-top-k.
Adds _ANSI_RE + _strip_ansi() helper to each of the 4 test files
(test_v0570_part_{a,b,c,d}.py) and routes every help-output
substring assertion through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #169 wired LOCAL_RANK==0 guard on _run_diagnose_gate so distributed
launches only run the gate on one worker per machine. Two minor polish
items on top of the merged version:
- Wrap the int() parse in try/except ValueError. A malformed LOCAL_RANK
(garbage value from a misconfigured launcher) would previously crash
the post-training gate. Falling back to True is safer than silently
skipping the gate -- over-running is recoverable, under-running hides
failures.
- Expand the docstring to explain why we use LOCAL_RANK (per-machine)
rather than RANK (global): the gate reads the local output_dir, so
one gate per machine is the right granularity for typical single-
machine multi-GPU runs. Documents the choice for future readers.
- Add a focused test (test_diagnose_gate_handles_malformed_local_rank)
asserting the safe fallback path.
Rich's CliRunner output on CI carries ANSI escape codes that split long
option names like `--badge` and `--diagnose-gate` across colour-reset
boundaries (`-\x1b[0m\x1b[1;36m-badge`), breaking naive `"--badge" in
result.output` substring checks. Same fix pattern as v0.55.0 CI hotfix.
Failures: tests/test_v0560.py::TestCli::test_diagnose_help and
TestTrainDiagnoseGate::test_help_lists_flag on all 9 CI matrix cells.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 macOS CI failures from the v0.55.0 push — Rich wraps option names
with ANSI escapes when the terminal is narrow (macOS CI runners
default to a smaller width than Linux/Windows), so substring searches
like `"--goal" in result.output` fail because the actual output
contains `\x1b[1;36m-\x1b[0m\x1b[1;36m-goal\x1b[0m`.
Project precedent: v0.53.5 / v0.53.6 / v0.53.8 / v0.53.9 all hit the
same pattern; tests/test_auto_tuning.py and tests/test_eval_platform.py
already ship `_ANSI_RE` + `_strip_ansi` helpers.
Failures fixed:
tests/test_v0550.py::TestCLIPlumbing::test_eval_design_help
tests/test_v0550.py::TestEvalAgainst::test_against_help
tests/test_v0550_followups.py::TestEvalAgainst::test_against_cli_help_lists_flag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two failures on ubuntu/macos/windows × py3.9/3.11/3.12 after v0.54.0
push:
1. test_env_null_byte_falls_back: monkeypatch.setenv can't set raw
NUL into the OS env layer (POSIX execve + Win32 SetEnv both
refuse). Switched to a temporary `advise_history.os.environ` swap
so the helper's defence-in-depth NUL guard is still exercised
without going through the C env layer.
2. test_default_missing_data: Click 8.0–8.1 returns rc=0 on
`no_args_is_help=True` invocations; Click 8.2+ returns rc=2 (the
"missing command" convention). CI runners had the newer Click;
dev box had the older. Accept both renderings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`soup advise <data.jsonl> --goal "..."` returns one of PROMPT_ENG /
RAG / SFT / DPO / GRPO with a confidence, reason, and reverse-when
criterion BEFORE the user spends 8 hours on a GPU. Layer above
autopilot — autopilot picks hyperparams AFTER the training decision;
advise picks the training decision itself.
Three Parts:
- Part A: Verdict engine — TASK_CATEGORIES + CHOICES allowlists,
frozen Verdict / DatasetProfile / ROIEstimate dataclasses, pure-
Python classify_task + compute_dataset_profile + build_verdict
rubric (DPO / GRPO floor 500 / PROMPT_ENG floor 50 / RAG / SFT).
- Part B: Probe runner — synth_probe_baselines + synth_probe_lora_delta
heuristic stubs with forward-compat model/device/lr/timeout_seconds
kwargs (v0.54.1 lifts to live model loading per stub-then-live
cadence used by v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus).
- Part C: Cross-project learning — ~/.soup/advise_history.jsonl with
cross-process file locking (fcntl on POSIX, sidecar <path>.lock +
msvcrt on Windows). `soup advise compare` reads history; env
override SOUP_ADVISE_HISTORY_PATH containment-checked to $HOME /
$CWD / tempdir (mirrors v0.36.0 SOUP_BATCH_CACHE_PATH policy).
CLI: Typer subcommand group `run` / `explain` / `compare` plus argv
preprocessor in cli.py that maps `soup advise data.jsonl` →
`soup advise run data.jsonl`. Scoped to argv[1] == "advise" only
(code-review HIGH fix — defends against rewrites when an unrelated
arg contains the literal string "advise").
Schema: AdviseConfig (goal / probe / record) field on SoupConfig
honors the plan's cross-cutting bullet.
Security: cwd-containment + os.lstat + S_ISLNK symlink reject on
every path input; atomic writes via tempfile.mkstemp + os.replace
on scratch + history; per-line 64 KB cap + 16 MiB file cap on
history reads; bool / finite / NUL / oversize guards on every public
input; Rich markup escape on user-controlled output.
Reviewed by python / code / security / tdd / architect agents — every
finding fixed before commit (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW).
Test count: 8400 → 8571 (+136 in tests/test_v0540.py, +35 net
adjustments to v0.53.x version-pin assertions to forward-compat >=).
Note: Windows CRLF / LF warnings during stage are .gitattributes-
governed and benign. CI runs on ubuntu-latest / windows-latest /
macos-latest × Python 3.9 / 3.11 / 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes v0.50.1 (#123, #126, #127), v0.49.1 (#119), v0.40.1 (#68).
#123 — live math kernels for 6 GRPO variants (gspo/dapo/dr_grpo/bnpo/
two_sided/rft) + `_GRPOTrainerVariant` HF Trainer subclass via
`make_grpo_trainer_variant` factory. Variant compute_loss reads kernel
inputs FIRST (no double-forward); falls back to super() only on missing
attrs. Case-insensitive variant normalisation before lru_cache.
#126 — PRMTrainerWrapper + `_PRMTrainer` HF Trainer subclass with real
compute_loss (gather hidden states at step_positions -> reward_head ->
MSE via compute_prm_loss). Dataset wrapped in datasets.Dataset.from_list
for HF Trainer compatibility. Bool-before-isinstance guard on batch_size.
#127 — GRPOStabilityCallback inherits transformers.TrainerCallback
(lazy), live EMA ref-model update in on_step_end with strict=True +
fallback-to-strict=False-with-WARNING on key mismatch (silent corruption
defence). math.isfinite guard on alpha.
#119 — LongLoRA forward override via LongLoRAForwardOverride context
manager with idempotent install (_soup_longlora_patched marker prevents
re-entry double-wrap), 256-char class name cap on regex match, restore
on __exit__ AND on exception.
#68 — true per-batch weighted-sum preference combine reading policy/ref
logps from TRL inputs + each compute_*_term kernel + combine_losses.
Explicit None checks on trainer attrs (no `or` on possibly-tensor),
DEBUG log on per-term skip.
Review fixes from 4 agents (python/code/security/tdd): 10 HIGH + 8
MEDIUM + 7 LOW — see CLAUDE.md v0.53.11 entry for the full list.
Test count: 8330 -> 8400 (+75 in test_v05311.py: 54 initial + 21
review-fix coverage gaps).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 issues closed:
- #150 [mix] pyproject extra bundles scikit-optimize so `soup data mix
--optimize` runs the Bayesian loop instead of the v0.48.0 Dirichlet
fallback; new describe_default_optimizer() helper labels the active
backend without paying skopt's import cost.
- #113 [data-pro] extras (langdetect + presidio-analyzer) with lazy
fall-through helpers in utils/data_score (broader language coverage +
Presidio entity recognition on top of the v0.47.0 regex baseline).
Llama-Guard-3-1B documented as a manual recipe (license + size).
- #154 SOUP_POSTHOG_KEY / SOUP_POSTHOG_ENDPOINT env override via
sentinel-based explicit-vs-env precedence; HTTPS-only +
RFC1918/link-local rejection on the endpoint; null-byte / control-char
/ >256-char rejection on the key.
- #152 --hub flag plumbed on chat / serve / infer / merge / export /
push via shared utils/hubs.apply_hub_to_cli_model +
prefetch_model_from_hub helpers; push uses upload_repo (skips
HF-specific Collections + model-card auto-render on non-HF hubs).
- #153 `soup data download --hub modelscope|modelers` live SDK
(lifts the v0.53.8 advisory-only path); friendly ImportError
advisory when the SDK is missing.
- #155 Web UI Tool Outputs panel — `loadToolOutputs` polls
/api/tool-outputs every 3s; XSS-safe DOM-built table (textContent
per cell, no innerHTML for user-controlled fields); Bearer token
threaded via the v0.53.9 window._authToken bootstrap.
- #156 SoupTrainerCallback.on_step_end records tool_calls counts
from kwargs['inputs'] into the global tool buffer. Best-effort
(# noqa: BLE001 per project policy — training must never crash).
13 review-fixes applied (4 HIGH / 5 MEDIUM / 4 LOW):
- HIGH PostHog explicit-endpoint precedence sentinel
- HIGH absolute path leak in local_path advisory reduced to relpath
- HIGH Rich markup escape on base / local_path / cache_dir
- HIGH callback # noqa: BLE001 per project policy
- MED `import time` moved out of try block
- MED oversize key + explicit-empty key rejection tests
- MED source-grep regression guards (advisory-removal, helper imports
across 5 non-push commands)
- MED `prefetch_model_from_hub` outside-cwd cache_root rejection
- LOW empty-list + bool-True tool_calls no-op tests
- LOW push.py uses upload_repo + validate_hub_name regression guard
Test count: 8285 -> 8330 (+45 in tests/test_v05310.py).
Full suite green; ruff clean; on Win+Py3.10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's Rich pipeline emits styled output that splits `--vocab-size` into
multiple ANSI-bracketed spans (e.g. `\x1b[36m-\x1b[0m\x1b[36m-vocab\x1b[0m\x1b[36m-size\x1b[0m`),
breaking naive `"--vocab-size" in result.output` checks. Locally Rich
auto-detects non-TTY and skips the codes, so the regression only shows
on CI (ubuntu/macos/windows × 3.9/3.11/3.12).
Fix: small `_plain()` helper using `re.sub(r"\x1b\[[0-9;]*m", "", ...)`
applied to the 7 failing assertions. Same approach already used in
several other v0.5x test modules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eight features that close out the v0.44.x live-monitoring deferrals
plus a long tail of standalone CLI wins:
- #94 /api/train/stream async SSE with per-subscriber cursor + JS
EventSource consumer; SoupTrainerCallback pushes TrainEvent on
each on_log.
- #95 soup ui --public derives LAN IP via SOCK_DGRAM connect-trick,
prints scannable QR; --auth-token override; SPA bootstrap
hydrates window._authToken from ?token= + sessionStorage and
cleans the URL via history.replaceState; CORS regex auto-widens
to loopback + RFC1918 in public mode; set_auth_token rotation
race fixed via threading.Lock.
- #98 soup serve --reasoning-parser strips <think>...</think> (and
OpenThinker tags); pre-compiled regex with marker-token
fast-path + 1 MiB cap + leading-newline-only strip.
- #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON
endpoint; best-effort observation hook in callback.on_log.
- #15 soup tokenizer train: BPE training CLI with raw-path lstat
symlink rejection, 50 MiB total / 8 KiB per-line caps,
post-mkdir output-dir re-check, --special-token NUL/oversize
dedup, vocab bounds [256, 200000].
- #26 soup bench --p50 --p95 renders extra per-prompt tail-latency
Rich table; --prompts-file gains symlink rejection.
- #28 soup bench --backend auto: MLX weights.npz probe (per-entry
lstat) -> config.json model_type keyword -> transformers
fallback; SOUP_BENCH_BACKEND env hint.
- #12 examples/synthetic_workflow.{md,yaml} end-to-end walkthrough.
Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the
python / code / security / tdd review agents. Notable HIGH:
- QR token now consumed by SPA (was unreachable previously).
- set_auth_token rotation lock-protected, 8-thread stress tested.
- Tokenizer input + output symlink TOCTOU defence on raw path.
- SSE generator switched to async (asyncio.sleep) for non-blocking
multi-subscriber operation.
- CORS regex for --public LAN mode (the old fixed allowlist of
http://0.0.0.0:port never matched a real Origin header).
- _has_mlx_weights per-entry lstat so a symlinked weights.npz can't
trigger MLX dispatch.
Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the
relaxed v0.53.8 version-pin asserts in tests/test_v0538.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local training runs write to ./out/ by default (e.g. .checkpoint.json
from soup train). Add it to the existing training-outputs gitignore
block alongside output/ and checkpoints/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.53.8 PyPI publish failed with:
400 Invalid distribution file. ZIP archive not accepted:
Duplicate filename in local headers
Root cause: `[tool.hatch.build.targets.wheel.force-include]` shipped
`soup_cli/data/_fixtures/` AND `packages = ["soup_cli"]` recursed into
the same path, so both the wheel and sdist contained each JSONL twice.
Fix: switch from force-include to `artifacts = [...]` which adds
non-Python files to the existing package tree exactly once. Standard
hatchling pattern for shipping data files inside an already-packaged
directory.
Version bumped to v0.53.8.1 (patch) — same code surface, just a build
config fix. v0.53.8 GitHub release remains as the feature changelog;
PyPI ships under v0.53.8.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five v0.53.8 CI failures (ubuntu/macos × py3.9/3.11/3.12):
1. test_help_lists_hub_flag — Typer's Rich-rendered help wraps long
option help across ANSI box-drawing lines; "--hub" appears as
"│ --\nhub" in the CI terminal renderer. Strip ANSI + collapse
whitespace before asserting.
2-5. test_pyproject_version / test_*_extra_present / test_force_include
— used `Path("pyproject.toml")` (relative to cwd). CI invokes pytest
from a different cwd than the repo root on at least one matrix
entry. Switched to a `_repo_root()` helper that derives from
`__file__` (matches v0.43.0 Part D demo_bundles approach).
Local re-run: 66/66 v0.53.8 tests pass after the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- #85 fsspec live loaders — data/loader.py routes the v0.42.0 fsspec
scheme allowlist (s3:// / gs:// / gcs:// / az:// / abfs:// / abfss:// /
oci://) through fsspec.open with validate_remote_uri containment
BEFORE connection. Friendly Rich panel names the pip install
advisory when the backend SDK is missing. Threads data.streaming
+ data.buffer_size. Row count capped at 1M.
- #130 Hub dispatcher live — utils/hubs.download_repo() and
upload_repo() lazy-import per backend (huggingface_hub /
modelscope / openmind_hub). Shared _validate_repo_id_shape (bool /
null-byte / leading-slash / .. / control-char / oversize) + cwd
containment on local_dir / folder_path. commands/train.py pre-fetches
non-HF base into .soup_hub_cache/ (sanitised slug, idempotent on
resume, cfg.base updated via model_copy). soup data download --hub
flag plumbed. Multi-command rollout for chat / serve / infer / merge
/ export / push tracked for v0.53.9.
- #89 [trackers] pyproject extra bundles mlflow / swanlab / trackio;
tracker_missing_dep_message surfaces a friendly pip install advisory
via importlib.util.find_spec (non-executing probe).
- #90 utils/trackers.send_telemetry_payload — opt-IN via SOUP_TELEMETRY=1;
lazy httpx; 1s hard timeout; HTTPS-only with SSRF re-validation
(mirrors v0.51.0 hub endpoint policy); silent-fail on every exception.
- #93 Fixtures migrated to soup_cli/data/_fixtures/ — zipapp /
namespace-package safe via [tool.hatch.build.targets.wheel.force-include];
_bundle_source_path falls back to examples/data/ for editable installs.
- #69 utils/hf_space.detect_space_sdk(requirements_text) — picks
"streamlit" / "gradio" from the rendered requirements.txt; closes
the v0.40.2 known limitation that custom Spaces always defaulted to
gradio. Wired into commands/deploy.py.
Review pass: python-review + code-review + security-review ran in
parallel; 16 findings fixed (3 HIGH + 8 MEDIUM + 5 LOW). Highlights:
cwd-containment on local_dir/folder_path, Windows ..\ traversal
defence on .soup_hub_cache slug, Pydantic model_copy(update=...)
instead of attribute mutation, idempotent pre-fetch via cache probe,
1M-row cap on remote materialisation, SSRF re-validation on
telemetry endpoint override, 256 KB cap on detect_space_sdk input,
modelscope.push_model commit_message kwarg removed (would TypeError
at runtime), find_spec instead of __import__ to avoid swanlab
side-effects.
Test count: 8162 -> 8257 (+66 in tests/test_v0538.py + 29 net adjustments).
Lint clean. CPU smoke: version, --help, load_config_from_string with
hub: modelscope passes; mlx + non-HF rejected; data download --hub
modelscope advisory rendered; detect_space_sdk live on real
requirements.txt bodies; package-data fixtures resolve from
soup_cli/data/_fixtures/.
v0.53.7 known limitation #1 (bash 501 marker) bumped to v0.53.9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
test_load_jsonl_rows_rejects_symlink: when the symlink target is outside
cwd, is_under_cwd's realpath resolution catches it before the lstat check
fires. Both rejections are valid security guards; broaden the regex to
match either error message ("symlink" or "under cwd").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI run 25805598433 failed across all 9 OS×Python cells:
- TestToolEndpointsLive / TestAnthropicMessagesStreaming /
TestReviewFixesVllmAnthropicLive ModuleNotFoundError: fastapi (CI does
not install [serve] extra) → autouse fixture pytest.importorskip()
- test_load_pretokenized_dataset_rejects_symlink:
load_pretokenized_dataset called datasets.load_from_disk on the symlink
target before the lstat check ran → moved the lstat + S_ISLNK check
to the entry of the helper so symlinks reject before any load attempt
- test_redact_exc_message_handles_windows_paths: hardened _redact_exc_message
to strip both POSIX absolute paths and Windows-style paths regardless
of host platform
Local pytest tests/test_v0537.py: 112 passed, 7 skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI failures on `tests/test_v0536.py`:
1. `ModuleNotFoundError: No module named 'fastapi'` on Ubuntu/macOS runners
— fastapi is in the [serve] extra, not [dev]. Added
`pytest.importorskip("fastapi")` to all 7 tests that use TestClient
(matches the pattern in `_build_app`).
2. `--execute` / `--output` / `v0.53.7` substring checks failed because
CI terminals render Typer/Rich help text with style spans (`-` and
`-execute` end up in separate `\x1b[...]m` runs). Added `_strip_ansi`
helper + wrapped 5 substring checks. Same fix pattern as the v0.53.5
`--live` CI fix on test_v0535.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rich splits the `--live` token across ANSI colour codes on narrow CI
terminals (`-\x1b[0m\x1b[1;36m-live`), so the raw-output substring
assertion failed on macOS/Windows runners but passed locally on a wide
terminal. Strip ANSI escapes before the check — matches how earlier test
files (e.g. test_v0402_part_b) handle Rich-coloured `--help` output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six closes lifting the v0.49.0 LongLoRA hardening + v0.41.0 LLaMA Pro
deferred stubs, plus a UX upgrade to the CUDA-OOM friendly message:
- #11 utils/errors.py: OOM hint now names --batch-size / --grad-accum
- #122 flash_attn.is_flash_attn_v3_available() + LongLoRA+FA3 schema reject
- #120 LongLoRA arch allowlist expansion (Mistral / Qwen / Phi); Mixtral
intentionally excluded (regex matches the bare 'mistral' token only)
- #121 apply_long_context_config auto-detects 'llama3' when caller passes
rope_scaling_type=None and the model config carries a Llama 3.1
rope_scaling block
- #83 block_expansion.expand_model_blocks LIVE (deepcopy last-N blocks,
zero-init residual projections, append, bump num_hidden_layers) +
apply_llama_pro_freeze + shared apply_block_expansion_if_configured
helper wired into SFT + Pretrain (mirrors v0.40.6 peft_wiring
centralisation policy so SFT and Pretrain stay in lock-step)
- #74 HF push surface QA — test plan recorded in tests/qa/v053_qa.md;
live execution against a private HF repo deferred to a credentialed
contributor
Review pipeline (python / code / security / tdd agents) ran; every
CRITICAL -> LOW finding addressed:
- bool-first guards in _check_model_name (defends against int subclass)
- is_supported_longlora_arch defensive non-string surface (returns False,
never raises) matching v0.53.3 is_known_vlm_base policy
- _truncate_for_message(value, limit=64) bounds the base echo in
LongLoRA error messages (security MEDIUM, mirrors v0.34.0 crash.py)
- null-byte + non-string TypeError guards on validate_longlora_compat
task / backend params (matches v0.50.0 validate_long_context_grpo_compat)
- _get_layers_module uses explicit `is None` not falsy shortcut (defends
against nn.Module.__bool__ overrides on subclasses)
- _zero_init_block_residual returns bool + warnings.warn when neither
standard projection matches the cloned block (non-Llama-shaped arches
still train but lose the LLaMA Pro identity-init guarantee)
Test count: 7879 -> 7935 (+56 net; +49 in new tests/test_v0534.py).
Lint clean. CPU smoke verified on a real transformers.LlamaForCausalLM:
4 -> 6 layers, down_proj + o_proj actually zeroed on PyTorch tensors,
old blocks frozen + new blocks trainable, forward pass finite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two surgical fixes from the v0.50.0 GRPO Plus deferred-stub family land:
- #128 grpo_fp16 routing: GRPOTrainerWrapper._build_precision_kwargs
returns {fp16, bf16} per (device, grpo_fp16) matrix (CPU/MPS/XPU →
both False, CUDA + grpo_fp16=True → fp16/!bf16, default CUDA →
legacy bf16). SoupConfig._validate_grpo_fp16_amp_exclusive rejects
the silent-mutex combo with auto_mixed_precision=True; short-circuits
when task != 'grpo' so the v0.50.0 task-gate diagnosis fires first.
- #129 vision-GRPO base probe: KNOWN_VLM_REGEX covers 10 VLM families
(Qwen2-VL/Qwen2.5-VL/QVQ/Pixtral/InternVL/Llama-3.2-Vision/LLaVA/
MiniCPM-V/Idefics/ShareGPT4V/Fuyu) with word-boundary anchors;
is_known_vlm_base returns False (never raises) on bad input;
validate_vision_grpo_compat now accepts optional base kwarg with
64-char error-message truncation. YAML pairing vision_grpo: true
with a non-VLM base is rejected at schema load with a friendly
families listing instead of a cryptic runtime AttributeError.
Scope: 4 larger v0.53.3 items (#127 stability callback, #123 GRPO
variant losses, #126 PRMTrainerWrapper, #68 multi-objective preference
live combine) are scope-deferred to v0.53.4 — each warrants its own
focused release per the v0.40.x stub-then-live cadence.
Tests: 7842 -> 7879 (+37 in tests/test_v0533.py). Four review agents
(python/code/security/tdd) ran; every HIGH/MEDIUM/LOW finding fixed
(task-gate priority short-circuit, MPS branch documented, 64-char
error truncation, QVQ regex coverage, 512-byte boundary test).
Two pre-existing v0.50.0 Part E tests migrated `base: test-llama` ->
`base: Qwen/Qwen2-VL-7B-Instruct` to clear the new probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three CLI tests added in the previous commit (test_help_lists_measure_flag,
test_save_format_help_lists_flag, test_torchao_help_lists_quant_config)
assumed the literal option name (e.g. `--measure`, `--save-format`,
`--quant-config`) would appear contiguously in CliRunner-captured
output. On CI runners the terminal defaults to 80-col and Rich wraps
long option names across lines, splitting the literal string.
Fix: walk `typer.main.get_command(app).params` and collect every `opt`
+ `secondary_opt` into a set, then assert membership. The test now
verifies what we actually care about (the option is registered) without
depending on Rich's wrapping behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lift six v0.53.0 deferred stubs from NotImplementedError to live wiring:
- #82 autopilot pre-quantized base detection
utils name regex over gptq/awq/aqlm/eetq/fp8/mxfp4 with word-boundary
anchoring + HQQ Nbit extraction + config.json quantization_config probe
(cwd-contained + symlink-rejected). decide_quantization() short-circuits
the VRAM heuristic when prequantized is set. autopilot pipeline auto-
applies so TheBloke/Llama-2-7B-Chat-GPTQ is recommended gptq instead of
4bit-on-top-of-quantized.
- #142 merge_4bit + export_torchao live writers
soup merge --save-format {fp16|4bit|4bit_forced}: single BNB-4bit
merged checkpoint without the dequant->merge->requant cycle (fixes
wrong-name llm_int8_skip_modules to bnb_4bit_skip_modules per code-
review). soup export --format torchao --quant-config <yaml>: torchao
.quantize_ + save_pretrained with per-scheme closed kwarg allowlist
(Int4WeightOnly accepts {group_size, inner_k_tiles}, NVFP4 accepts
nothing extra; dunder + unknown keys rejected per security-review H1).
load_quant_config enforces yaml.safe_load + 256 KB cap + extension
allowlist + cwd containment + S_ISLNK rejection.
- #139 export_advanced_gguf via llama.cpp imatrix
3-stage pipeline: convert_hf_to_gguf.py -> optional imatrix ->
quantize. argv-list subprocess (no shell), 30-min timeout, realpath-
verified convert script stays inside llama_cpp_dir (security-review
M5). _prepare_calibration_text accepts JSONL with text/prompt/content
field aliases + raw text fallback; strips null bytes, collapses
newlines, 8 KB per-line + 50 MB total cap (security-review M1); POSIX
O_NOFOLLOW closes the TOCTOU window between dispatch-time check and
open() (security-review M3). UD- prefix stripped before passing to
llama-quantize. _safe_stderr Rich-escapes subprocess stderr before
embedding in RuntimeError (security-review L4).
- #109 soup deploy autopilot --measure
Live Quant-Lobotomy scorecard: classifies each candidate quant OK /
MINOR / MAJOR (thresholds 2% / 5% mirror v0.26.0 Part D). Results
cached at ~/.soup/deploy_autopilot_cache.json (atomic write, 0o600
perms on POSIX, S_ISLNK rejection on BOTH load and save). pick_best
soft-fallback now picks max-by-delta (was max-by-after) matching the
v0.33.0 #54 design intent. _DEPLOY_MEASURE_BEFORE_GEN / _AFTER_FACTORY
module-level hooks act as the stop-gap escape hatch until v0.46.1
ships first-party transformers / vLLM generator factories.
- #70/#72 manual QA log scripted at tests/qa/v053_qa.md with exact
reproduction recipes + acceptance criteria for the CUDA + llama.cpp
smokes that can't run on the CI runners.
Shared cleanup:
- soup_cli/utils/paths.enforce_under_cwd_and_no_symlink consolidates the
v0.33.0 #22 TOCTOU pattern previously copy-pasted in save_formats.py
and gguf_quant.py (code-review HIGH fix).
Reviews ran: python / code / security / tdd. Every CRITICAL / HIGH /
MEDIUM / LOW finding fixed or documented.
Test count: 7610 -> 7722 (+112 across 4 new files).
Known limitations: live GPU + bitsandbytes / torchao smokes for the
new merge / export paths remain pending (recipes in QA log); injected-
generator escape hatch is non-public until v0.46.1; cache key truncates
base_sha to 16 hex (1-in-2^32 collision floor).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpamys
Vivaan Dhawan
Salil M
Yixuan Xu