Closes#241.
Adds opt-in tokenizer-aware n-gram path for the v0.70.0 Part F echo-trap detector. The existing whitespace `score_echo_signal` is unchanged; callers opt in via the new `score_trajectory_repetition_tokenized` / `score_echo_signal_tokenized` helpers or the `--echo-trap-tokenizer-aware` train flag.
Acceptance criterion from #241 verified: synthetic case where decoded strings differ by punctuation but token-id sequence repeats — whitespace path returns OK, tokenizer-aware path returns TRAP.
Post-release audit caught two pre-existing doc drifts that v0.70.0 should
have closed in the initial commit:
1. README.md `## All Commands` table was missing the 6 new v0.70.0 surfaces
(`soup iterative-dpo`, `soup train --reward-hack-detector`,
`--uld-strategy`, `--minillm-enabled`, `--rl-checkpoint-save-every-steps`,
`--echo-trap-enabled`). Added 6 lines mirroring the v0.69.0 entry format.
2. CONTRIBUTING.md `utils/` directory tree had stale drift from v0.66.0 →
v0.70.0 (24 modules missing across 5 releases). Brought in sync:
v0.66 (sae_diff, sleeper_probe, interference, probe_pack), v0.67
(cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock,
adapter_bisect), v0.68 (prompt_compile, prompt_distill, compile_tools,
apple_adapter, local_rl), v0.69 (build_dag, expectations, magpie,
persona_hub, brain_rot), v0.70 (reward_hacking, uld, minillm,
rl_checkpoint, iterative_dpo, echo_trap).
No code changes — docs-only hotfix per the checklist
"CI-only / docs-only hotfixes" rule. No version bump, no tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v0.67.0 and v0.68.0 releases shipped 14 new commands but neither
release updated the README `## All Commands` reference table per Release
Checklist Step 9. Adds the missing rows:
- v0.67.0 (5): adapters merge --strategy cmaes / adapters pr / adapters
bisect / lock write / lock show + check.
- v0.68.0 (9): compile / distill-prompt / compile-tools / apple-adapter +
local-rl init / status / record / harvest / train.
Docs-only — no source code change, no test count delta, no version bump
per the checklist's CI-only / docs-only hotfix policy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 commands that hedge Soup against paradigm shifts. If 1M-context kills FT,
`soup compile` (DSPy + GEPA + TextGrad prompt-program compilation) takes its
place. If teams hit prompt-cost walls, `soup distill-prompt` bridges to small
FT. If only Apple Foundation Models win on-device, `soup apple-adapter` ships
the converter+signing surface. If personal-LLM flywheels become the shape,
`soup local-rl` captures thumbs into SQLite and emits DPO pairs.
- Part A: `soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|...]`
- Part B: `soup distill-prompt --traces <jsonl> --teacher --student --strategy`
- Part C: `soup compile-tools <spec.json|yaml> --eval <jsonl>`
- Part D: `soup apple-adapter <source-dir> --direction hf-to-mlx|... --output`
- Part E: `soup local-rl init/status/record/harvest/train` (LIVE except train)
Schema + path containment + symlink rejection + atomic-write surface ship now;
live runners for Parts A/B/C/D + Part E nightly scheduler deferred to v0.68.1
(stub-then-live, mirrors v0.50.0 / v0.61.0 / v0.62.0 / v0.67.0 cadence).
Test count: 11021 -> 11225 (+204). Review-fix: 0 CRIT + 4 HIGH + 10 MED + 4 LOW.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- SECURITY.md: add v0.67.0 to supported-versions list
- CONTRIBUTING.md: bump test count 244 files / 10836 tests -> 251 / 11021
Docs-only follow-up to v0.67.0 release; no code change, no version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Was `assert soup_cli.__version__ == "0.64.0"` — broke on v0.65.0 bump on CI
across all 9 platform×Python matrix jobs. The test intent is "v0.64.0 has
shipped at least once"; switch to a tuple floor check matching the v0.51 /
v0.54 / v0.57 / v0.60 idiom.
Caught by CI red on v0.65.0 push to main; local pytest passed because we
ran the v0.65 test files in isolation per the Release Checklist Step 4
``pytest --no-cov`` invocation. Lesson: include the full suite in step 4
or grep for ``soup_cli.__version__ ==`` exact-equality assertions in any
future version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six new top-level commands close axis 1 + 11 of the roadmap: pick the
right base, lock the env, refuse OOMs before launch, and clear
license-clean deploys.
- soup tunability: probe-train 8 candidate bases (Qwen3-0.6/1.7B,
Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) ->
Pareto frontier over (delta x cost x license). Live LoRA probe -> v0.64.1.
- soup plan / soup apply: Terraform-shape lock-and-execute. `apply`
refuses on drift between soup.yaml and soup.tfstate (exit 3).
- soup env lock / status / check: hermetic env lockfile via
importlib.metadata across 15 ABI-sensitive packages + Python + CUDA.
`env check` exits 3 on drift.
- Hardware-fit calculator: static analytical 5-bucket VRAM predictor
with 10% safety margin + actionable hint on OOM.
- soup completions bash|zsh|fish: sourceable shell completion scripts;
recipe names auto-complete from the 115-recipe catalogue.
- soup license-advisor: per-deploy-target license matrix
(b2c/defense/embedded) + Llama community + 700M MAU gate (exit 3).
Composes with v0.60 license-conflict matrix.
Tests: 10035 -> 10306 (+271 net in 7 new files).
Review-fix coverage: 0 CRITICAL + 6 HIGH + 8 MEDIUM + 4 LOW across
consolidated code+security+TDD review wave. Every HIGH lands a regression
test in tests/test_v0640_followups.py (POSIX-skipped symlink rejection,
containment-before-existence ordering, drift-refusal exit-3 end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docs-only hotfix — adds the v0.63.0 commands (soup ingest /
prune-prompt / data active-sample / ab / drift-alarm) to the
"## All Commands" code-block reference card. The release v0.63.0 had
dedicated ## feature sections + the ## What's New block updated, but
the All Commands quick-reference was missed during the doc pass.
No source code changed — no version bump, no tag, no release.
Per CLAUDE.md hotfix policy: docs-only commits don't ship to PyPI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 new top-level commands close axis 7 of the roadmap. Every Part LIVE on
day one (no deferred stubs):
- soup ingest: universal trace importer (Langfuse / LangSmith / Helicone /
OpenPipe / OTel / OpenAI Stored Completions). 6 adapters
+ frozen TraceRecord with MappingProxyType-wrapped metadata.
Zero credential-handling threat surface — Soup parses the
JSONL export, never makes the SaaS network call.
- soup prune-prompt: detect + strip a shared system-prompt prefix so the
FT model internalises it (OpenPipe's signature trick,
OSS). Binary-search over up to 32 templates finds the
longest threshold-meeting prefix.
- soup data active-sample: surface top-uncertainty prod traces for human
review. Max-entropy on single rm_score or
pairwise disagreement on dual rm_scores.
- soup ab: Wald sequential SPRT for the point alternative. LLR is a
martingale under H0 so Type-I error is controlled at every
stopping time per the optional stopping theorem.
- soup drift-alarm: rolling KL on whitespace-tokenised output distribution
+ SSRF-hardened Slack/Discord webhook (full parity with
v0.51.0 validate_hub_endpoint). Exit 3 on drift for
cron-friendly automation.
Test count: 9816 -> 10035 (+219 net across 6 new test files).
Review-fix coverage (code-reviewer + tdd-guide returned actionable;
python-reviewer + security-reviewer agents context-thrashed on the large
CLAUDE.md release-notes history — matches the v0.58.0 / v0.59.0 / v0.60.0
/ v0.61.0 / v0.62.0 idiom; verified manually):
- 1 CRITICAL: mSPRT log-likelihood-ratio sign error drove Type-I error
to 1.0 as n grew. Replaced with Wald's classic point-
alternative SPRT (martingale under H0).
- 2 HIGH: detect_common_prefix early-exit on 100% match returned the
shortest qualifying prefix instead of the longest;
_MAX_SCAN_ROWS DoS cap used 'pass' instead of 'break'.
- 3 MEDIUM: TraceRecord.metadata now MappingProxyType-wrapped post-init
(frozen-dataclass mutation hazard); _AUTH_ENV table
deduplicated; drift_alarm precedence parens on SSRF gate.
- 2 LOW: pooled_se dead-branch refactor; mean_uncertainty NaN guard.
- 8 follow-up tests: msprt zero-variance, partial-majority binary-search
activation, score_uncertainty exact boundaries, rolling_kl identical
+ disjoint, validate_budget + validate_threshold exact endpoints,
_signal_from_thumbs boundaries, no-heavy-top-level-imports source-grep
guard across all 5 new util modules.
Step 6 smoke verified for all 5 commands + 6 failure-mode rejection
paths.
CRLF gotcha note for future maintainers: PowerShell wrote the smoke
fixtures with a UTF-8 BOM on Windows during Step 6 — switched to
inline Python for the fixture write. Production CLI input handling is
already BOM-tolerant (utf-8-sig in JSONL loaders via v0.40.1 Part E).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The list/info portions of 4f54179 follow the structure of @dreamer0129's
PR #175 draft. Adding Co-Authored-By trailer so GitHub contributors graph
reflects the original draft author.
Co-Authored-By: dreamer0129 <185807357+dreamer0129@users.noreply.github.com>
Closes#174. Picked up after PR #175 (dreamer0129) went quiet — the
list/info path was clean, but compare() escaped only inside the
highlight branch, leaving a shared crafted base_model = "[link=evil]
click[/]" un-escaped on equal-value rows.
Fix follows the "escape always at the value layer, decoration wraps
after" pattern mirroring v0.57.0 `adapters diff` / `info`:
- list_adapters: wrap base / lora_r / peft_type / rel_path with
rich.markup.escape() before table.add_row(); also escape
adapter_path in the JSONDecodeError fallback.
- info: wrap base_model / peft_type / task_type / lora_r / lora_alpha
/ lora_dropout / modules_str inside the Rich Panel f-string; also
escape adapter_path.name in the Panel title.
- compare: escape val1_str / val2_str unconditionally; [yellow]
highlight wraps already-escaped values when they differ. Equal-value
rows now also escape (was the v0.57.0 known-limitation gap).
+4 regression tests in tests/test_adapters.py::TestAdaptersMarkupEscape:
- test_list_escapes_crafted_base_model — asserts no ANSI hyperlink
sequence (\x1b]8;) leaks from a crafted [link=http://evil/...] payload.
- test_info_escapes_crafted_base_model — same assertion for Panel.
- test_compare_escapes_equal_crafted_values — the specific regression
for the PR #175 review gap (identical crafted values on both sides
must NOT smuggle live markup through the equal-branch).
- test_compare_escapes_differing_crafted_values — highlight branch
also escapes.
Closes v0.57.0 Known Limitation (9).
Verified locally:
- ruff check soup_cli/commands/adapters.py tests/test_adapters.py -> clean
- pytest tests/test_adapters.py --no-cov -> 20 passed
CI on ubuntu / macOS exposed a missed TOCTOU detail: `os.lstat(realpath(path))`
silently resolves symlinks before the lstat, so `S_ISLNK` never trips and a
symlinked input path passes the rejection. Windows CI was skipped (POSIX-only
symlink test), so the bug didn't surface in local smoke.
Three fixes, matching the v0.53.7 #106 project policy of "lstat the RAW path
before realpath":
- `edit_diff.load_probes` — lstat path first, then realpath after rejection.
- `unlearning_eval.load_evidence_file` — same.
- `unlearning_eval.get_fixture_path` — lstat raw candidate before realpath.
No new tests — the existing `test_symlink_rejected` covers it; it now passes
on POSIX where it previously failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three failures on CI run 26084542388 — all are version-pin / Rich-wrap
artefacts, not real regressions in v0.60.0 functionality:
- test_v0560 test_pyproject_version: regex-based >=0.56 floor check
(was substring `version = "0.5`)
- test_v0590 test_version_is_0_59 -> test_version_is_at_least_0_59:
>=0.59 floor (matches v0.51/v0.54 floor-check idiom)
- test_v0600_part_e merge_help_lists_license_flags: strip ANSI codes
before substring check (Rich splits `--license` across `\x1b[1;36m`
escapes in the wrapped Typer table)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v0.59.0 What's New block claimed +93 new tests, but the actual count
is +99 new tests in tests/test_v0590.py (9193 -> 9294). Numbers in
CLAUDE.md / SECURITY.md / CONTRIBUTING.md / the release notes already
have the correct count.
Docs-only hotfix; does not require a version bump (per release checklist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous test_default_log_path_rejects_null_byte_env used monkeypatch.setenv
to inject a null byte into SOUP_AUDIT_LOG_PATH and expected default_log_path
to fall back gracefully. But the OS layer rejects null bytes in env vars on
every platform we ship on:
- POSIX (Linux/macOS): `ValueError: embedded null byte`
- Windows: `ValueError: embedded null character`
The setenv call itself raises, never reaching default_log_path. Split into two
tests that hit the actual validation surfaces:
1. test_default_log_path_rejects_null_byte_override — calls the private
_validate_log_path_override helper directly with a null-byte string and
asserts it returns None (so the caller falls back to the safe default).
2. test_default_log_path_handles_env_read_value_error — monkeypatches
os.environ.get to raise ValueError, exercising the defence-in-depth
try/except around the env read in default_log_path().
Both tests pass on Linux + macOS + Windows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three macOS-3.11 CI failures in test_v0590.py post-merge:
1+2. test_train_annex_xi_flag_present_in_help / test_train_repro_receipt_flag_present_in_help
— Typer's Rich-renderer wraps long lines and inserts ANSI colour codes
BETWEEN the two dashes of `--annex-xi` / `--repro-receipt`, so the
literal substring match fails. Strip ANSI escape codes via regex before
asserting; also accept the bare option name as a defence-in-depth
fallback against future Rich line-wrap quirks.
3. test_default_log_path_rejects_null_byte_env — POSIX `os.environ.get` raises
`ValueError("embedded null byte")` when the env value contains a NUL
character, while Windows allows the read. Wrap the env read in
`try/except ValueError` so the function falls back to the safe default
(~/.soup/audit.jsonl) on either platform.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six Parts ship the procurement-floor moat — every Soup run can now emit
the formats regulated orgs demand, with no SaaS structurally able to
follow. Pure orchestration on top of v0.26 Registry + v0.34 cost
tracker + v0.56 diagnose — schema + atomic-write surface only; live
Sigstore signing, CodeCarbon hook, PDF rendering deferred to v0.59.1.
Part A — `soup bom emit` CycloneDX 1.6 ML-BOM + SPDX 2.3 AI-profile
dual emitter from any RegistryEntry. SHA-256 validation on every sha
field, license-id chain, base-model component with hash, per-artifact
file components, energy properties under metadata.properties.
Part B — `soup attest emit` in-toto v1 Statement wrapping SLSA-3
provenance v1 predicate. Stage allowlist (extract/train/eval/export/
publish), subject SHA locked to 64-hex, builder_id capped, SignatureBackend
enum with UNSIGNED live + SIGSTORE/ED25519 stubs raising NotImplementedError
with explicit v0.59.1 marker.
Part C — `soup train --annex-xi` EU AI Act Annex XI Sections 1+2 +
Annex XII Article 53(1)(d) markdown auto-doc. Top-10 domain cap,
modality breakdown, FLOPs/kWh/CO2. `_md_escape` neutralises |[](){}!<>
plus newline/CR/tab in every operator-controlled field — defends
against forged-heading + Markdown-link injection in downstream PDF/HTML
renderers (mirrors v0.29.0 model-card v2 policy).
Part D — `soup audit-log tail/rotate` HIPAA/SOC2-shaped JSONL with
PII redaction across every string field via v0.40.3 _SECRET_RE policy.
POSIX O_NOFOLLOW on append + 0o600 perms + lstat-based symlink rejection
at rotation backup path (no lexists race). SOUP_AUDIT_LOG_PATH env
override containment-checked to $HOME / $CWD / $TMPDIR.
Part E — `soup train --repro-receipt` SR 11-7-style receipt: seeds
(torch/numpy/python), kernel versions (CUDA/cuDNN/NCCL via best-effort
torch probes), GPU model + driver, OS + arch, Python version. Atomic
write, cwd-contained.
Part F — CodeCarbon hook schema + electricityMap SSRF validator with
full parity to v0.51.0 hubs.validate_hub_endpoint (scheme allowlist,
loopback-only HTTP, RFC1918 / link-local / reserved / multicast IP
rejection via ipaddress.ip_address, control-char + null-byte
rejection). PUE math + attach_energy populating BomEntry.
Cross-cutting: new paths.atomic_write_text shared TOCTOU-safe helper
centralises the v0.33.0 #22 / v0.43.0 / v0.55.0 / v0.56.0 / v0.57.0
/ v0.58.0 atomic-write pattern from four separate copies into one
single-source-of-truth (mirrors v0.40.6 / v0.53.5 peft_wiring policy).
Four review waves (python-reviewer + general-purpose security/code/tdd):
0 CRITICAL + 8 HIGH + 12 MEDIUM + 4 LOW resolved before commit.
HIGH fixes: audit-log lstat-before-write TOCTOU, O_NOFOLLOW on
append, redaction extended to host_id/operator_id/command, audit-log
env override containment, bom artifact size_bytes validation,
BomEntry attach_energy type-hint fix, default_log_path public symbol,
duplicated seeds validation removed.
Test count 9193 → 9294 (+99 net in tests/test_v0590.py; 93 pass +
6 POSIX-skipped on Windows for symlink rejection branches). v0.58.0
floor-check assertions widened from exact-match in test_v0580.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #178 wired the live NCCL bandwidth measurement on top of the
v0.43.0 nccl_bandwidth_check reference table. Four polish items
on top of the merged version:
- Bump timing samples 5 -> 10 and report MEDIAN rather than mean.
Median is robust to one-off jitter (kernel preemption, page
faults, OS scheduler hiccups) that can scew a 5-sample mean by
10-30%. Median of 10 gives a much more honest GB/s number.
- Bump warmup 1 -> 3 iters. First all_reduce in a fresh process
pays for CUDA kernel JIT plus the initial NCCL collective
handshake; one warmup isn't always enough to amortise that out.
3 warmups is the standard NVIDIA recommendation for collective
benchmarks.
- Time each iteration separately (collect a per-iter list, then
median) instead of averaging total elapsed / iters. Same headline
cost, but exposes per-call variance for future percentile output.
- Snapshot + restore MASTER_ADDR / MASTER_PORT around the spawn.
The worker sets them unconditionally, so prior to this patch a
second doctor invocation in the same process (test harness,
notebook, future TUI auto-refresh) would inherit stale env. Now
we restore exactly the prior state (including absent vars).
Plus three module-level constants (_NCCL_BENCHMARK_TENSOR_BYTES
/ _WARMUP_ITERS / _TIMED_ITERS) with comments explaining each
number so future tuning is informed, not magic. Status line now
prints the actual params so users see what we measured.
Existing tests (mock mp.spawn at the outer boundary) keep passing
unchanged; no new tests needed for this polish layer.
* feat(doctor): add --nccl flag to measure and validate multi-GPU bandwidth
* test(doctor): add mocked CUDA tests to verify --nccl skip and success behaviors
* docs(readme): document the new --nccl bandwidth check flag for the doctor command
Step 9 follow-up: add dedicated `## Adapter Management (git for LoRA)`
section to the README body so the surface is self-contained when the
single-slot `## What's New` block is overwritten in v0.58.0.
Step 10 follow-up: add v0.57.0 to the SECURITY.md supported-versions
list + a detailed entry in the per-version fix notes covering all
9 HIGH fixes (TIES sign-tie default, 4× symlink TOCTOU rejections,
atomic writes, env CRLF rejection, allowlist policy migration, etc.)
and the 7 known limitations.
Tracked follow-ups filed as GitHub issues #171–#174:
#171 — live blame ablation runner
#172 — merge canary verdict via v0.55 eval gate
#173 — branch pointers → v0.26 Registry lineage
#174 — Rich-markup backfill for legacy adapters list/info/compare
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POSIX setenv (and Windows equivalent) reject null bytes + control chars
at the syscall boundary, so monkeypatch.setenv("SOUP_BRANCHES_DIR",
"/some\x00path") raises ValueError on every CI runner before our code
ever sees the env var.
Stub os.environ.get directly so the helper's rejection branch is
exercised exactly as it would be if the env var arrived through some
other channel (subprocess env inheritance, in-process programmatic
mutation, etc).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rich-wrap-CI workaround — same fix pattern as v0.55.0 / v0.56.0:
CliRunner output contains ANSI color escapes that break literal
'--top-k' in output substring matches because Rich renders option
names as -\x1b[0m\x1b[1;36m-top-k.
Adds _ANSI_RE + _strip_ansi() helper to each of the 4 test files
(test_v0570_part_{a,b,c,d}.py) and routes every help-output
substring assertion through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #169 wired LOCAL_RANK==0 guard on _run_diagnose_gate so distributed
launches only run the gate on one worker per machine. Two minor polish
items on top of the merged version:
- Wrap the int() parse in try/except ValueError. A malformed LOCAL_RANK
(garbage value from a misconfigured launcher) would previously crash
the post-training gate. Falling back to True is safer than silently
skipping the gate -- over-running is recoverable, under-running hides
failures.
- Expand the docstring to explain why we use LOCAL_RANK (per-machine)
rather than RANK (global): the gate reads the local output_dir, so
one gate per machine is the right granularity for typical single-
machine multi-GPU runs. Documents the choice for future readers.
- Add a focused test (test_diagnose_gate_handles_malformed_local_rank)
asserting the safe fallback path.
Rich's CliRunner output on CI carries ANSI escape codes that split long
option names like `--badge` and `--diagnose-gate` across colour-reset
boundaries (`-\x1b[0m\x1b[1;36m-badge`), breaking naive `"--badge" in
result.output` substring checks. Same fix pattern as v0.55.0 CI hotfix.
Failures: tests/test_v0560.py::TestCli::test_diagnose_help and
TestTrainDiagnoseGate::test_help_lists_flag on all 9 CI matrix cells.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 macOS CI failures from the v0.55.0 push — Rich wraps option names
with ANSI escapes when the terminal is narrow (macOS CI runners
default to a smaller width than Linux/Windows), so substring searches
like `"--goal" in result.output` fail because the actual output
contains `\x1b[1;36m-\x1b[0m\x1b[1;36m-goal\x1b[0m`.
Project precedent: v0.53.5 / v0.53.6 / v0.53.8 / v0.53.9 all hit the
same pattern; tests/test_auto_tuning.py and tests/test_eval_platform.py
already ship `_ANSI_RE` + `_strip_ansi` helpers.
Failures fixed:
tests/test_v0550.py::TestCLIPlumbing::test_eval_design_help
tests/test_v0550.py::TestEvalAgainst::test_against_help
tests/test_v0550_followups.py::TestEvalAgainst::test_against_cli_help_lists_flag
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two failures on ubuntu/macos/windows × py3.9/3.11/3.12 after v0.54.0
push:
1. test_env_null_byte_falls_back: monkeypatch.setenv can't set raw
NUL into the OS env layer (POSIX execve + Win32 SetEnv both
refuse). Switched to a temporary `advise_history.os.environ` swap
so the helper's defence-in-depth NUL guard is still exercised
without going through the C env layer.
2. test_default_missing_data: Click 8.0–8.1 returns rc=0 on
`no_args_is_help=True` invocations; Click 8.2+ returns rc=2 (the
"missing command" convention). CI runners had the newer Click;
dev box had the older. Accept both renderings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`soup advise <data.jsonl> --goal "..."` returns one of PROMPT_ENG /
RAG / SFT / DPO / GRPO with a confidence, reason, and reverse-when
criterion BEFORE the user spends 8 hours on a GPU. Layer above
autopilot — autopilot picks hyperparams AFTER the training decision;
advise picks the training decision itself.
Three Parts:
- Part A: Verdict engine — TASK_CATEGORIES + CHOICES allowlists,
frozen Verdict / DatasetProfile / ROIEstimate dataclasses, pure-
Python classify_task + compute_dataset_profile + build_verdict
rubric (DPO / GRPO floor 500 / PROMPT_ENG floor 50 / RAG / SFT).
- Part B: Probe runner — synth_probe_baselines + synth_probe_lora_delta
heuristic stubs with forward-compat model/device/lr/timeout_seconds
kwargs (v0.54.1 lifts to live model loading per stub-then-live
cadence used by v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus).
- Part C: Cross-project learning — ~/.soup/advise_history.jsonl with
cross-process file locking (fcntl on POSIX, sidecar <path>.lock +
msvcrt on Windows). `soup advise compare` reads history; env
override SOUP_ADVISE_HISTORY_PATH containment-checked to $HOME /
$CWD / tempdir (mirrors v0.36.0 SOUP_BATCH_CACHE_PATH policy).
CLI: Typer subcommand group `run` / `explain` / `compare` plus argv
preprocessor in cli.py that maps `soup advise data.jsonl` →
`soup advise run data.jsonl`. Scoped to argv[1] == "advise" only
(code-review HIGH fix — defends against rewrites when an unrelated
arg contains the literal string "advise").
Schema: AdviseConfig (goal / probe / record) field on SoupConfig
honors the plan's cross-cutting bullet.
Security: cwd-containment + os.lstat + S_ISLNK symlink reject on
every path input; atomic writes via tempfile.mkstemp + os.replace
on scratch + history; per-line 64 KB cap + 16 MiB file cap on
history reads; bool / finite / NUL / oversize guards on every public
input; Rich markup escape on user-controlled output.
Reviewed by python / code / security / tdd / architect agents — every
finding fixed before commit (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW).
Test count: 8400 → 8571 (+136 in tests/test_v0540.py, +35 net
adjustments to v0.53.x version-pin assertions to forward-compat >=).
Note: Windows CRLF / LF warnings during stage are .gitattributes-
governed and benign. CI runs on ubuntu-latest / windows-latest /
macos-latest × Python 3.9 / 3.11 / 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes v0.50.1 (#123, #126, #127), v0.49.1 (#119), v0.40.1 (#68).
#123 — live math kernels for 6 GRPO variants (gspo/dapo/dr_grpo/bnpo/
two_sided/rft) + `_GRPOTrainerVariant` HF Trainer subclass via
`make_grpo_trainer_variant` factory. Variant compute_loss reads kernel
inputs FIRST (no double-forward); falls back to super() only on missing
attrs. Case-insensitive variant normalisation before lru_cache.
#126 — PRMTrainerWrapper + `_PRMTrainer` HF Trainer subclass with real
compute_loss (gather hidden states at step_positions -> reward_head ->
MSE via compute_prm_loss). Dataset wrapped in datasets.Dataset.from_list
for HF Trainer compatibility. Bool-before-isinstance guard on batch_size.
#127 — GRPOStabilityCallback inherits transformers.TrainerCallback
(lazy), live EMA ref-model update in on_step_end with strict=True +
fallback-to-strict=False-with-WARNING on key mismatch (silent corruption
defence). math.isfinite guard on alpha.
#119 — LongLoRA forward override via LongLoRAForwardOverride context
manager with idempotent install (_soup_longlora_patched marker prevents
re-entry double-wrap), 256-char class name cap on regex match, restore
on __exit__ AND on exception.
#68 — true per-batch weighted-sum preference combine reading policy/ref
logps from TRL inputs + each compute_*_term kernel + combine_losses.
Explicit None checks on trainer attrs (no `or` on possibly-tensor),
DEBUG log on per-term skip.
Review fixes from 4 agents (python/code/security/tdd): 10 HIGH + 8
MEDIUM + 7 LOW — see CLAUDE.md v0.53.11 entry for the full list.
Test count: 8330 -> 8400 (+75 in test_v05311.py: 54 initial + 21
review-fix coverage gaps).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 issues closed:
- #150 [mix] pyproject extra bundles scikit-optimize so `soup data mix
--optimize` runs the Bayesian loop instead of the v0.48.0 Dirichlet
fallback; new describe_default_optimizer() helper labels the active
backend without paying skopt's import cost.
- #113 [data-pro] extras (langdetect + presidio-analyzer) with lazy
fall-through helpers in utils/data_score (broader language coverage +
Presidio entity recognition on top of the v0.47.0 regex baseline).
Llama-Guard-3-1B documented as a manual recipe (license + size).
- #154 SOUP_POSTHOG_KEY / SOUP_POSTHOG_ENDPOINT env override via
sentinel-based explicit-vs-env precedence; HTTPS-only +
RFC1918/link-local rejection on the endpoint; null-byte / control-char
/ >256-char rejection on the key.
- #152 --hub flag plumbed on chat / serve / infer / merge / export /
push via shared utils/hubs.apply_hub_to_cli_model +
prefetch_model_from_hub helpers; push uses upload_repo (skips
HF-specific Collections + model-card auto-render on non-HF hubs).
- #153 `soup data download --hub modelscope|modelers` live SDK
(lifts the v0.53.8 advisory-only path); friendly ImportError
advisory when the SDK is missing.
- #155 Web UI Tool Outputs panel — `loadToolOutputs` polls
/api/tool-outputs every 3s; XSS-safe DOM-built table (textContent
per cell, no innerHTML for user-controlled fields); Bearer token
threaded via the v0.53.9 window._authToken bootstrap.
- #156 SoupTrainerCallback.on_step_end records tool_calls counts
from kwargs['inputs'] into the global tool buffer. Best-effort
(# noqa: BLE001 per project policy — training must never crash).
13 review-fixes applied (4 HIGH / 5 MEDIUM / 4 LOW):
- HIGH PostHog explicit-endpoint precedence sentinel
- HIGH absolute path leak in local_path advisory reduced to relpath
- HIGH Rich markup escape on base / local_path / cache_dir
- HIGH callback # noqa: BLE001 per project policy
- MED `import time` moved out of try block
- MED oversize key + explicit-empty key rejection tests
- MED source-grep regression guards (advisory-removal, helper imports
across 5 non-push commands)
- MED `prefetch_model_from_hub` outside-cwd cache_root rejection
- LOW empty-list + bool-True tool_calls no-op tests
- LOW push.py uses upload_repo + validate_hub_name regression guard
Test count: 8285 -> 8330 (+45 in tests/test_v05310.py).
Full suite green; ruff clean; on Win+Py3.10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's Rich pipeline emits styled output that splits `--vocab-size` into
multiple ANSI-bracketed spans (e.g. `\x1b[36m-\x1b[0m\x1b[36m-vocab\x1b[0m\x1b[36m-size\x1b[0m`),
breaking naive `"--vocab-size" in result.output` checks. Locally Rich
auto-detects non-TTY and skips the codes, so the regression only shows
on CI (ubuntu/macos/windows × 3.9/3.11/3.12).
Fix: small `_plain()` helper using `re.sub(r"\x1b\[[0-9;]*m", "", ...)`
applied to the 7 failing assertions. Same approach already used in
several other v0.5x test modules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eight features that close out the v0.44.x live-monitoring deferrals
plus a long tail of standalone CLI wins:
- #94 /api/train/stream async SSE with per-subscriber cursor + JS
EventSource consumer; SoupTrainerCallback pushes TrainEvent on
each on_log.
- #95 soup ui --public derives LAN IP via SOCK_DGRAM connect-trick,
prints scannable QR; --auth-token override; SPA bootstrap
hydrates window._authToken from ?token= + sessionStorage and
cleans the URL via history.replaceState; CORS regex auto-widens
to loopback + RFC1918 in public mode; set_auth_token rotation
race fixed via threading.Lock.
- #98 soup serve --reasoning-parser strips <think>...</think> (and
OpenThinker tags); pre-compiled regex with marker-token
fast-path + 1 MiB cap + leading-newline-only strip.
- #100 ToolOutputsBuffer global singleton + /api/tool-outputs JSON
endpoint; best-effort observation hook in callback.on_log.
- #15 soup tokenizer train: BPE training CLI with raw-path lstat
symlink rejection, 50 MiB total / 8 KiB per-line caps,
post-mkdir output-dir re-check, --special-token NUL/oversize
dedup, vocab bounds [256, 200000].
- #26 soup bench --p50 --p95 renders extra per-prompt tail-latency
Rich table; --prompts-file gains symlink rejection.
- #28 soup bench --backend auto: MLX weights.npz probe (per-entry
lstat) -> config.json model_type keyword -> transformers
fallback; SOUP_BENCH_BACKEND env hint.
- #12 examples/synthetic_workflow.{md,yaml} end-to-end walkthrough.
Review fixes: 0 CRITICAL + 11 HIGH + 14 MEDIUM + 9 LOW across the
python / code / security / tdd review agents. Notable HIGH:
- QR token now consumed by SPA (was unreachable previously).
- set_auth_token rotation lock-protected, 8-thread stress tested.
- Tokenizer input + output symlink TOCTOU defence on raw path.
- SSE generator switched to async (asyncio.sleep) for non-blocking
multi-subscriber operation.
- CORS regex for --public LAN mode (the old fixed allowlist of
http://0.0.0.0:port never matched a real Origin header).
- _has_mlx_weights per-entry lstat so a symlinked weights.npz can't
trigger MLX dispatch.
Test count: 8257 -> 8285 (+57 in tests/test_v0539.py, minus the
relaxed v0.53.8 version-pin asserts in tests/test_v0538.py).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local training runs write to ./out/ by default (e.g. .checkpoint.json
from soup train). Add it to the existing training-outputs gitignore
block alongside output/ and checkpoints/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.53.8 PyPI publish failed with:
400 Invalid distribution file. ZIP archive not accepted:
Duplicate filename in local headers
Root cause: `[tool.hatch.build.targets.wheel.force-include]` shipped
`soup_cli/data/_fixtures/` AND `packages = ["soup_cli"]` recursed into
the same path, so both the wheel and sdist contained each JSONL twice.
Fix: switch from force-include to `artifacts = [...]` which adds
non-Python files to the existing package tree exactly once. Standard
hatchling pattern for shipping data files inside an already-packaged
directory.
Version bumped to v0.53.8.1 (patch) — same code surface, just a build
config fix. v0.53.8 GitHub release remains as the feature changelog;
PyPI ships under v0.53.8.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five v0.53.8 CI failures (ubuntu/macos × py3.9/3.11/3.12):
1. test_help_lists_hub_flag — Typer's Rich-rendered help wraps long
option help across ANSI box-drawing lines; "--hub" appears as
"│ --\nhub" in the CI terminal renderer. Strip ANSI + collapse
whitespace before asserting.
2-5. test_pyproject_version / test_*_extra_present / test_force_include
— used `Path("pyproject.toml")` (relative to cwd). CI invokes pytest
from a different cwd than the repo root on at least one matrix
entry. Switched to a `_repo_root()` helper that derives from
`__file__` (matches v0.43.0 Part D demo_bundles approach).
Local re-run: 66/66 v0.53.8 tests pass after the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Shivam
Alpamys
Vivaan Dhawan
Salil M
Yixuan Xu