Lift the v0.68.0 deferred-stub family to live (closes#225, #226, #227, #229):
- #229 local-rl train --once: harvest thumbs -> DPO/KTO/ORPO train via a
soup train subprocess (argv list, no shell); state table tracks last_train_at
(skip-on-no-new-thumbs + skip-on-insufficient-pairs); no --once renders a
systemd/launchd nightly scheduler scaffold. New local_rl_scheduler.py.
- #226 distill-prompt: call the teacher once per trace (Ollama/Anthropic/vLLM)
and write a real dataset (sft/kl -> messages; preference -> chosen/rejected).
- #225 compile / #227 compile-tools: live DSPy/GEPA/TextGrad dispatch behind the
new [compile] extra with a friendly ImportError when absent; injectable seams.
Security: reject \n/\r in the model id + shell-quote ExecStart args (systemd
injection defence). Fix: render train output as a plain string (schema-valid),
with a regression test against SoupConfig.
Tests 13329 -> 13424. Smoked end-to-end: real DPO train on SmolLM2-135M (RTX 3050)
+ real Ollama teacher distillation.
CI renders Typer help with ANSI colour codes under FORCE_COLOR that split the
leading `--` from the flag name, so raw-substring assertions on `--steer`
(serve) and `--output`/`--top-k` (steer train) passed locally but failed in CI.
Strip ANSI before the membership check (same fix as the v0.71.1 --record-thumbs
help assert). Local + FORCE_COLOR=1: 142/142 pass. tests-only, no version bump.
POSIX os.putenv forbids null bytes in env values, so
monkeypatch.setenv(SOUP_EDIT_GOVERNOR_DB, 'x\x00.db') raised
ValueError at setenv time on ubuntu/macos before the code under
test ran (windows tolerated it). Assert _validate_governor_db_override
rejects the null byte directly; the validated-None fallback branch is
already covered cross-platform by test_env_override_out_of_bounds_falls_back.
CI (FORCE_COLOR) makes Rich/Typer split flag tokens at colorized hyphens
(--auto-download -> -auto -download) and auto-highlight `=` in error text
(name=path), so contiguous-substring asserts fail. Add the _clean_help helper
(strip ANSI + all whitespace, matching the v0.71.1 / test_v0717 pattern) and
apply it to the sae-diff / train / sleeper / interference --help asserts plus
the bad-adapter-spec name=path error assert. Reproduced + verified with
FORCE_COLOR=1 locally. No source change; test count unchanged.
Rich splits `--pre-wired` / `--pack-cans` / `--push` with ANSI escapes under
CI FORCE_COLOR; _clean_help() strips them before the substring check (same
fix family as v0.71.1/v0.71.3). No src change.
CI (FORCE_COLOR) renders --track-energy / --no-audit-log as split ANSI colour
segments, and monkeypatch.setenv with a null byte raises at setup on POSIX
(Windows tolerated both). Strip ANSI via a shared `_plain()` helper for every
--help substring assert, and rewrite the never-raises audit test to monkeypatch
append_audit_event to throw instead of injecting a null-byte env path.
CI installs [dev] with FORCE_COLOR, so Rich colorizes Typer --help and
splits an option name like --key into ANSI-wrapped segments
(\x1b[1;36m-\x1b[0m\x1b[1;36m-key\x1b[0m). The 4 raw-substring help
asserts passed locally (no color) but failed on all 9 CI test jobs.
Add a module-level _strip_ansi() helper and route the sign/verify/merge/
attest-emit --help substring checks through it (mirrors the v0.71.1
test_serve --record-thumbs fix). Confirmed locally under FORCE_COLOR=1:
all 4 pass; ANSI-strip alone is sufficient (no flag line-wraps).
Test-only change on the unreleased v0.71.2 — no version bump.
The v0.71.1 release commit (514761c) went red on CI for two reasons:
- test_flag_in_help asserted a raw "--record-thumbs" substring, but Rich
splits an option name's dashes with ANSI codes under CI's FORCE_COLOR
(it passes locally without color). Strip ANSI before the substring check.
- Coverage fell to 76.96% (< 77% gate): CI installs [dev], which has no
FastAPI, so the new /v1/thumbs endpoint + record-thumbs startup block in
serve.py are uncovered there. Restore the gate honestly (no lowering, no
pragma) by adding 19 genuine no-FastAPI tests for previously-uncovered
pure-CLI paths: lock show / lock check (no-drift / drift exit 3 / missing),
env check (no-drift / missing / drift exit 3), env fix error branches,
env lock null-byte output, and load_evidence_file (the
`eval unlearning --evidence` loader).
CI-equivalent (no-fastapi) coverage: 76.96% -> 77.24%. Tests: 12134 -> 12153.
Heavy training stack (torch, transformers, peft, trl, datasets,
bitsandbytes, accelerate) moves out of the core install into a new
[train] optional-dependency extra. `pip install soup-cli` is now a
light CLI + data-tools install with no PyTorch; `pip install
'soup-cli[train]'` adds the training stack.
- pyproject: new [train] + [all] extras; [dev] self-references [train]
so CI (`pip install -e ".[dev]"`) still gets torch. Pins unchanged.
- errors.py: missing torch/transformers/peft/trl/datasets/bitsandbytes/
accelerate now surface a single 'install soup-cli[train]' fix.
- Dockerfile: install soup-cli[train,serve,data,eval] so the GPU image
can still fine-tune.
- README + docs/models.md: split install into light core vs [train].
- CHANGELOG: cut [0.71.0]; bump version 0.70.0 -> 0.71.0.
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.
- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
CODEOWNERS patterns, and the PR-template checklist all repointed to
src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.
Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
Rich on narrow Windows columns splits `--lang` across colour-cycle ANSI
escapes (`\x1b[..m-\x1b[..m-lang`), so the literal substring check fails
on CI even though the rendered help renders correctly for humans. CI was
red on every commit landing after #234 hit a runner with that exact
column width + Python 3.9 + Rich combination.
Mirrors the `_ANSI_RE` strip pattern in tests/test_auto_tuning.py — flat
regex over the captured output before the `in` check.
v0.63.0 `score_uncertainty` raised on K>2 and `_row_uncertainty` fell back
to a monotone-broken `max(scores) - min(scores)`. Now generalises to K<=32
via population variance scaled by 4 — adding a fresh RM score equal to the
running mean strictly decreases uncertainty (the new contribution to the
sum-of-squares is zero while the denominator grows), so consensus on
redundant evidence can never spike the score.
K=1 max-entropy and K=2 disagreement formulas preserved verbatim (existing
operator dashboards depend on the |s1 - s2| value). Cap stays at K=32 for
DoS defence. _row_uncertainty K>2 path now routes through score_uncertainty
inside an isolated try/except — bad rows return 0.0 instead of crashing the
batch.
PEP 585 modernisation: collections.abc imports + list[...] annotations
(safe because `from __future__ import annotations` is in scope).
math.fsum used for the variance accumulation to keep rounding error
sub-ULP at K=32.
Tests: +32 net (25 in new tests/test_v0631_206.py + 7 TDD review-fix
followups). Full suite 11941 -> 11973 pass.
Closes#206.
Extends v0.69.0 Part E score_triviality + score_popularity_signal to
non-English corpora. New utils/brain_rot_lang.py ships a MappingProxyType
registry of frozen BrainRotLangBundle for en/es/fr/de/ru. Every public
scorer accepts an optional lang kwarg (default None preserves v0.69.0
English behaviour). The "auto" sentinel routes through the v0.53.10
[data-pro] langdetect helper with silent fallback to English on
missing-package / detector-exception / unsupported-code.
soup data brain-rot gains --lang en|es|fr|de|ru|auto, strictly validated
at the CLI boundary (exit 2 on typos). Per-row resolution backed by
eager _validate_lang_arg on dataset scorers so empty rows cannot
bypass shape checks.
Closes#234.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#241.
Adds opt-in tokenizer-aware n-gram path for the v0.70.0 Part F echo-trap detector. The existing whitespace `score_echo_signal` is unchanged; callers opt in via the new `score_trajectory_repetition_tokenized` / `score_echo_signal_tokenized` helpers or the `--echo-trap-tokenizer-aware` train flag.
Acceptance criterion from #241 verified: synthetic case where decoded strings differ by punctuation but token-id sequence repeats — whitespace path returns OK, tokenizer-aware path returns TRAP.
5 commands that hedge Soup against paradigm shifts. If 1M-context kills FT,
`soup compile` (DSPy + GEPA + TextGrad prompt-program compilation) takes its
place. If teams hit prompt-cost walls, `soup distill-prompt` bridges to small
FT. If only Apple Foundation Models win on-device, `soup apple-adapter` ships
the converter+signing surface. If personal-LLM flywheels become the shape,
`soup local-rl` captures thumbs into SQLite and emits DPO pairs.
- Part A: `soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|...]`
- Part B: `soup distill-prompt --traces <jsonl> --teacher --student --strategy`
- Part C: `soup compile-tools <spec.json|yaml> --eval <jsonl>`
- Part D: `soup apple-adapter <source-dir> --direction hf-to-mlx|... --output`
- Part E: `soup local-rl init/status/record/harvest/train` (LIVE except train)
Schema + path containment + symlink rejection + atomic-write surface ship now;
live runners for Parts A/B/C/D + Part E nightly scheduler deferred to v0.68.1
(stub-then-live, mirrors v0.50.0 / v0.61.0 / v0.62.0 / v0.67.0 cadence).
Test count: 11021 -> 11225 (+204). Review-fix: 0 CRIT + 4 HIGH + 10 MED + 4 LOW.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Was `assert soup_cli.__version__ == "0.64.0"` — broke on v0.65.0 bump on CI
across all 9 platform×Python matrix jobs. The test intent is "v0.64.0 has
shipped at least once"; switch to a tuple floor check matching the v0.51 /
v0.54 / v0.57 / v0.60 idiom.
Caught by CI red on v0.65.0 push to main; local pytest passed because we
ran the v0.65 test files in isolation per the Release Checklist Step 4
``pytest --no-cov`` invocation. Lesson: include the full suite in step 4
or grep for ``soup_cli.__version__ ==`` exact-equality assertions in any
future version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six new top-level commands close axis 1 + 11 of the roadmap: pick the
right base, lock the env, refuse OOMs before launch, and clear
license-clean deploys.
- soup tunability: probe-train 8 candidate bases (Qwen3-0.6/1.7B,
Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) ->
Pareto frontier over (delta x cost x license). Live LoRA probe -> v0.64.1.
- soup plan / soup apply: Terraform-shape lock-and-execute. `apply`
refuses on drift between soup.yaml and soup.tfstate (exit 3).
- soup env lock / status / check: hermetic env lockfile via
importlib.metadata across 15 ABI-sensitive packages + Python + CUDA.
`env check` exits 3 on drift.
- Hardware-fit calculator: static analytical 5-bucket VRAM predictor
with 10% safety margin + actionable hint on OOM.
- soup completions bash|zsh|fish: sourceable shell completion scripts;
recipe names auto-complete from the 115-recipe catalogue.
- soup license-advisor: per-deploy-target license matrix
(b2c/defense/embedded) + Llama community + 700M MAU gate (exit 3).
Composes with v0.60 license-conflict matrix.
Tests: 10035 -> 10306 (+271 net in 7 new files).
Review-fix coverage: 0 CRITICAL + 6 HIGH + 8 MEDIUM + 4 LOW across
consolidated code+security+TDD review wave. Every HIGH lands a regression
test in tests/test_v0640_followups.py (POSIX-skipped symlink rejection,
containment-before-existence ordering, drift-refusal exit-3 end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 new top-level commands close axis 7 of the roadmap. Every Part LIVE on
day one (no deferred stubs):
- soup ingest: universal trace importer (Langfuse / LangSmith / Helicone /
OpenPipe / OTel / OpenAI Stored Completions). 6 adapters
+ frozen TraceRecord with MappingProxyType-wrapped metadata.
Zero credential-handling threat surface — Soup parses the
JSONL export, never makes the SaaS network call.
- soup prune-prompt: detect + strip a shared system-prompt prefix so the
FT model internalises it (OpenPipe's signature trick,
OSS). Binary-search over up to 32 templates finds the
longest threshold-meeting prefix.
- soup data active-sample: surface top-uncertainty prod traces for human
review. Max-entropy on single rm_score or
pairwise disagreement on dual rm_scores.
- soup ab: Wald sequential SPRT for the point alternative. LLR is a
martingale under H0 so Type-I error is controlled at every
stopping time per the optional stopping theorem.
- soup drift-alarm: rolling KL on whitespace-tokenised output distribution
+ SSRF-hardened Slack/Discord webhook (full parity with
v0.51.0 validate_hub_endpoint). Exit 3 on drift for
cron-friendly automation.
Test count: 9816 -> 10035 (+219 net across 6 new test files).
Review-fix coverage (code-reviewer + tdd-guide returned actionable;
python-reviewer + security-reviewer agents context-thrashed on the large
CLAUDE.md release-notes history — matches the v0.58.0 / v0.59.0 / v0.60.0
/ v0.61.0 / v0.62.0 idiom; verified manually):
- 1 CRITICAL: mSPRT log-likelihood-ratio sign error drove Type-I error
to 1.0 as n grew. Replaced with Wald's classic point-
alternative SPRT (martingale under H0).
- 2 HIGH: detect_common_prefix early-exit on 100% match returned the
shortest qualifying prefix instead of the longest;
_MAX_SCAN_ROWS DoS cap used 'pass' instead of 'break'.
- 3 MEDIUM: TraceRecord.metadata now MappingProxyType-wrapped post-init
(frozen-dataclass mutation hazard); _AUTH_ENV table
deduplicated; drift_alarm precedence parens on SSRF gate.
- 2 LOW: pooled_se dead-branch refactor; mean_uncertainty NaN guard.
- 8 follow-up tests: msprt zero-variance, partial-majority binary-search
activation, score_uncertainty exact boundaries, rolling_kl identical
+ disjoint, validate_budget + validate_threshold exact endpoints,
_signal_from_thumbs boundaries, no-heavy-top-level-imports source-grep
guard across all 5 new util modules.
Step 6 smoke verified for all 5 commands + 6 failure-mode rejection
paths.
CRLF gotcha note for future maintainers: PowerShell wrote the smoke
fixtures with a UTF-8 BOM on Windows during Step 6 — switched to
inline Python for the fixture write. Production CLI input handling is
already BOM-tolerant (utf-8-sig in JSONL loaders via v0.40.1 Part E).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#174. Picked up after PR #175 (dreamer0129) went quiet — the
list/info path was clean, but compare() escaped only inside the
highlight branch, leaving a shared crafted base_model = "[link=evil]
click[/]" un-escaped on equal-value rows.
Fix follows the "escape always at the value layer, decoration wraps
after" pattern mirroring v0.57.0 `adapters diff` / `info`:
- list_adapters: wrap base / lora_r / peft_type / rel_path with
rich.markup.escape() before table.add_row(); also escape
adapter_path in the JSONDecodeError fallback.
- info: wrap base_model / peft_type / task_type / lora_r / lora_alpha
/ lora_dropout / modules_str inside the Rich Panel f-string; also
escape adapter_path.name in the Panel title.
- compare: escape val1_str / val2_str unconditionally; [yellow]
highlight wraps already-escaped values when they differ. Equal-value
rows now also escape (was the v0.57.0 known-limitation gap).
+4 regression tests in tests/test_adapters.py::TestAdaptersMarkupEscape:
- test_list_escapes_crafted_base_model — asserts no ANSI hyperlink
sequence (\x1b]8;) leaks from a crafted [link=http://evil/...] payload.
- test_info_escapes_crafted_base_model — same assertion for Panel.
- test_compare_escapes_equal_crafted_values — the specific regression
for the PR #175 review gap (identical crafted values on both sides
must NOT smuggle live markup through the equal-branch).
- test_compare_escapes_differing_crafted_values — highlight branch
also escapes.
Closes v0.57.0 Known Limitation (9).
Verified locally:
- ruff check soup_cli/commands/adapters.py tests/test_adapters.py -> clean
- pytest tests/test_adapters.py --no-cov -> 20 passed
Three failures on CI run 26084542388 — all are version-pin / Rich-wrap
artefacts, not real regressions in v0.60.0 functionality:
- test_v0560 test_pyproject_version: regex-based >=0.56 floor check
(was substring `version = "0.5`)
- test_v0590 test_version_is_0_59 -> test_version_is_at_least_0_59:
>=0.59 floor (matches v0.51/v0.54 floor-check idiom)
- test_v0600_part_e merge_help_lists_license_flags: strip ANSI codes
before substring check (Rich splits `--license` across `\x1b[1;36m`
escapes in the wrapped Typer table)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous test_default_log_path_rejects_null_byte_env used monkeypatch.setenv
to inject a null byte into SOUP_AUDIT_LOG_PATH and expected default_log_path
to fall back gracefully. But the OS layer rejects null bytes in env vars on
every platform we ship on:
- POSIX (Linux/macOS): `ValueError: embedded null byte`
- Windows: `ValueError: embedded null character`
The setenv call itself raises, never reaching default_log_path. Split into two
tests that hit the actual validation surfaces:
1. test_default_log_path_rejects_null_byte_override — calls the private
_validate_log_path_override helper directly with a null-byte string and
asserts it returns None (so the caller falls back to the safe default).
2. test_default_log_path_handles_env_read_value_error — monkeypatches
os.environ.get to raise ValueError, exercising the defence-in-depth
try/except around the env read in default_log_path().
Both tests pass on Linux + macOS + Windows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three macOS-3.11 CI failures in test_v0590.py post-merge:
1+2. test_train_annex_xi_flag_present_in_help / test_train_repro_receipt_flag_present_in_help
— Typer's Rich-renderer wraps long lines and inserts ANSI colour codes
BETWEEN the two dashes of `--annex-xi` / `--repro-receipt`, so the
literal substring match fails. Strip ANSI escape codes via regex before
asserting; also accept the bare option name as a defence-in-depth
fallback against future Rich line-wrap quirks.
3. test_default_log_path_rejects_null_byte_env — POSIX `os.environ.get` raises
`ValueError("embedded null byte")` when the env value contains a NUL
character, while Windows allows the read. Wrap the env read in
`try/except ValueError` so the function falls back to the safe default
(~/.soup/audit.jsonl) on either platform.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six Parts ship the procurement-floor moat — every Soup run can now emit
the formats regulated orgs demand, with no SaaS structurally able to
follow. Pure orchestration on top of v0.26 Registry + v0.34 cost
tracker + v0.56 diagnose — schema + atomic-write surface only; live
Sigstore signing, CodeCarbon hook, PDF rendering deferred to v0.59.1.
Part A — `soup bom emit` CycloneDX 1.6 ML-BOM + SPDX 2.3 AI-profile
dual emitter from any RegistryEntry. SHA-256 validation on every sha
field, license-id chain, base-model component with hash, per-artifact
file components, energy properties under metadata.properties.
Part B — `soup attest emit` in-toto v1 Statement wrapping SLSA-3
provenance v1 predicate. Stage allowlist (extract/train/eval/export/
publish), subject SHA locked to 64-hex, builder_id capped, SignatureBackend
enum with UNSIGNED live + SIGSTORE/ED25519 stubs raising NotImplementedError
with explicit v0.59.1 marker.
Part C — `soup train --annex-xi` EU AI Act Annex XI Sections 1+2 +
Annex XII Article 53(1)(d) markdown auto-doc. Top-10 domain cap,
modality breakdown, FLOPs/kWh/CO2. `_md_escape` neutralises |[](){}!<>
plus newline/CR/tab in every operator-controlled field — defends
against forged-heading + Markdown-link injection in downstream PDF/HTML
renderers (mirrors v0.29.0 model-card v2 policy).
Part D — `soup audit-log tail/rotate` HIPAA/SOC2-shaped JSONL with
PII redaction across every string field via v0.40.3 _SECRET_RE policy.
POSIX O_NOFOLLOW on append + 0o600 perms + lstat-based symlink rejection
at rotation backup path (no lexists race). SOUP_AUDIT_LOG_PATH env
override containment-checked to $HOME / $CWD / $TMPDIR.
Part E — `soup train --repro-receipt` SR 11-7-style receipt: seeds
(torch/numpy/python), kernel versions (CUDA/cuDNN/NCCL via best-effort
torch probes), GPU model + driver, OS + arch, Python version. Atomic
write, cwd-contained.
Part F — CodeCarbon hook schema + electricityMap SSRF validator with
full parity to v0.51.0 hubs.validate_hub_endpoint (scheme allowlist,
loopback-only HTTP, RFC1918 / link-local / reserved / multicast IP
rejection via ipaddress.ip_address, control-char + null-byte
rejection). PUE math + attach_energy populating BomEntry.
Cross-cutting: new paths.atomic_write_text shared TOCTOU-safe helper
centralises the v0.33.0 #22 / v0.43.0 / v0.55.0 / v0.56.0 / v0.57.0
/ v0.58.0 atomic-write pattern from four separate copies into one
single-source-of-truth (mirrors v0.40.6 / v0.53.5 peft_wiring policy).
Four review waves (python-reviewer + general-purpose security/code/tdd):
0 CRITICAL + 8 HIGH + 12 MEDIUM + 4 LOW resolved before commit.
HIGH fixes: audit-log lstat-before-write TOCTOU, O_NOFOLLOW on
append, redaction extended to host_id/operator_id/command, audit-log
env override containment, bom artifact size_bytes validation,
BomEntry attach_energy type-hint fix, default_log_path public symbol,
duplicated seeds validation removed.
Test count 9193 → 9294 (+99 net in tests/test_v0590.py; 93 pass +
6 POSIX-skipped on Windows for symlink rejection branches). v0.58.0
floor-check assertions widened from exact-match in test_v0580.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): add --nccl flag to measure and validate multi-GPU bandwidth
* test(doctor): add mocked CUDA tests to verify --nccl skip and success behaviors
* docs(readme): document the new --nccl bandwidth check flag for the doctor command
Alpamys
gittihub-jpg
Shivam
Vivaan Dhawan
Salil M