CI renders Typer help with ANSI colour codes under FORCE_COLOR that split the
leading `--` from the flag name, so raw-substring assertions on `--steer`
(serve) and `--output`/`--top-k` (steer train) passed locally but failed in CI.
Strip ANSI before the membership check (same fix as the v0.71.1 --record-thumbs
help assert). Local + FORCE_COLOR=1: 142/142 pass. tests-only, no version bump.
POSIX os.putenv forbids null bytes in env values, so
monkeypatch.setenv(SOUP_EDIT_GOVERNOR_DB, 'x\x00.db') raised
ValueError at setenv time on ubuntu/macos before the code under
test ran (windows tolerated it). Assert _validate_governor_db_override
rejects the null byte directly; the validated-None fallback branch is
already covered cross-platform by test_env_override_out_of_bounds_falls_back.
CI (FORCE_COLOR) makes Rich/Typer split flag tokens at colorized hyphens
(--auto-download -> -auto -download) and auto-highlight `=` in error text
(name=path), so contiguous-substring asserts fail. Add the _clean_help helper
(strip ANSI + all whitespace, matching the v0.71.1 / test_v0717 pattern) and
apply it to the sae-diff / train / sleeper / interference --help asserts plus
the bad-adapter-spec name=path error assert. Reproduced + verified with
FORCE_COLOR=1 locally. No source change; test count unchanged.
Rich splits `--pre-wired` / `--pack-cans` / `--push` with ANSI escapes under
CI FORCE_COLOR; _clean_help() strips them before the substring check (same
fix family as v0.71.1/v0.71.3). No src change.
CI (FORCE_COLOR) renders --track-energy / --no-audit-log as split ANSI colour
segments, and monkeypatch.setenv with a null byte raises at setup on POSIX
(Windows tolerated both). Strip ANSI via a shared `_plain()` helper for every
--help substring assert, and rewrite the never-raises audit test to monkeypatch
append_audit_event to throw instead of injecting a null-byte env path.
CI installs [dev] with FORCE_COLOR, so Rich colorizes Typer --help and
splits an option name like --key into ANSI-wrapped segments
(\x1b[1;36m-\x1b[0m\x1b[1;36m-key\x1b[0m). The 4 raw-substring help
asserts passed locally (no color) but failed on all 9 CI test jobs.
Add a module-level _strip_ansi() helper and route the sign/verify/merge/
attest-emit --help substring checks through it (mirrors the v0.71.1
test_serve --record-thumbs fix). Confirmed locally under FORCE_COLOR=1:
all 4 pass; ANSI-strip alone is sufficient (no flag line-wraps).
Test-only change on the unreleased v0.71.2 — no version bump.
The v0.71.1 release commit (514761c) went red on CI for two reasons:
- test_flag_in_help asserted a raw "--record-thumbs" substring, but Rich
splits an option name's dashes with ANSI codes under CI's FORCE_COLOR
(it passes locally without color). Strip ANSI before the substring check.
- Coverage fell to 76.96% (< 77% gate): CI installs [dev], which has no
FastAPI, so the new /v1/thumbs endpoint + record-thumbs startup block in
serve.py are uncovered there. Restore the gate honestly (no lowering, no
pragma) by adding 19 genuine no-FastAPI tests for previously-uncovered
pure-CLI paths: lock show / lock check (no-drift / drift exit 3 / missing),
env check (no-drift / missing / drift exit 3), env fix error branches,
env lock null-byte output, and load_evidence_file (the
`eval unlearning --evidence` loader).
CI-equivalent (no-fastapi) coverage: 76.96% -> 77.24%. Tests: 12134 -> 12153.
- SECURITY.md: supported window 0.70.x -> 0.71.x
- CONTRIBUTING.md: dev-deps list now lists mypy + pre-commit and notes the
v0.71.0 deps-split ([dev] self-references [train], so torch & co are pulled in)
Heavy training stack (torch, transformers, peft, trl, datasets,
bitsandbytes, accelerate) moves out of the core install into a new
[train] optional-dependency extra. `pip install soup-cli` is now a
light CLI + data-tools install with no PyTorch; `pip install
'soup-cli[train]'` adds the training stack.
- pyproject: new [train] + [all] extras; [dev] self-references [train]
so CI (`pip install -e ".[dev]"`) still gets torch. Pins unchanged.
- errors.py: missing torch/transformers/peft/trl/datasets/bitsandbytes/
accelerate now surface a single 'install soup-cli[train]' fix.
- Dockerfile: install soup-cli[train,serve,data,eval] so the GPU image
can still fine-tune.
- README + docs/models.md: split install into light core vs [train].
- CHANGELOG: cut [0.71.0]; bump version 0.70.0 -> 0.71.0.
CLAUDE.md and .claude/plan.md are maintainer-local (gitignored), so any
committed file that links to or tells contributors to read/update them is
broken for anyone cloning the public repo. Repoint everything at the public
docs/ split (and the schema / CHANGELOG where relevant).
- CONTRIBUTING.md: "add a trainer/format" + "Version Bump Process" steps now
say README.md + the relevant docs/ page (was CLAUDE.md); removed the
"See CLAUDE.md for the complete release checklist" pointer (replaced with a
one-line public note: bump -> tag v* -> CI -> PyPI OIDC publish); reworded
the test-table note off CLAUDE.md
- AGENTS.md: removed the maintainer-local CLAUDE.md parenthetical
- advise.py / runs.py / quant_menu.py: reword docstring/comments that cited
.claude/plan.md or CLAUDE.md (comment-only; no behaviour change)
Acceptance: grep "CLAUDE.md" across *.md (excl .claude/) is now empty; no
committed file references .claude/CLAUDE.md or .claude/plan.md.
The repo moved to src-layout and trimmed README into a 238-line front door
with the feature reference under docs/, but several committed files still
referenced bare soup_cli/ paths or linked the gitignored .claude/CLAUDE.md
(which 404s for anyone cloning the public repo).
- docs/: `soup_cli/{plugins,templates,ui/plugins}/...` path refs -> `src/soup_cli/...`
(import statements `from soup_cli...` left unchanged — package name is still soup_cli)
- AGENTS.md: point external agents at public docs/, CONTRIBUTING.md, and the
config schema; note CLAUDE.md is a maintainer-local (gitignored) file
- CONTRIBUTING.md + .github/pull_request_template.md: PR checklist now says
"README.md and the matching page under docs/" (kept in sync); Questions
section links docs/ instead of the gitignored CLAUDE.md
- examples/README.md: fix two broken ../CLAUDE.md links -> config schema source
+ docs/ feature reference
- .gitignore: add root-anchored /_*.py temp-script guard + trailing newline
mypy findings are surfaced as a warning annotation + step log instead of
failing the job. The workflow was already green (job-level continue-on-error),
but the type-check job itself rendered as a red X in the commit checks. Move
the tolerance to the step and emit a :⚠️: so the check stays green while
type annotations are adopted incrementally.
The README had grown to 5046 lines (195 sections) — roughly one deep-dive per
feature accreted over 70 releases. Split it into a concise front door plus a
public docs/ tree:
- README (5046 -> 238 lines): hero, why, quickstart, config, a Documentation
map, data formats, common commands, models, Docker, requirements, dev.
- docs/*.md: all 185 feature sections preserved verbatim, grouped into 10 themed
guides + an index. Every original line is accounted for (content-conservation
checked); all 235 internal links + anchors verified to resolve.
- un-gitignore docs/ (it was empty); fix a pre-existing dangling
docs/QUANTIZATION.md link; correct the stale `ruff check soup_cli/` ->
`src/soup_cli/` reference in the Development section.
No version bump: docs-only — rides into the 0.71.0 deps-split release.
- add src/soup_cli/py.typed (PEP 561); verified it ships in the built wheel
- add .pre-commit-config.yaml (ruff lint+format + standard file-hygiene hooks)
- add mypy>=1.8.0 + pre-commit to the [dev] extra; lenient [tool.mypy] config
- add a non-blocking type-check CI job (mypy, continue-on-error: true)
- add CHANGELOG.md (Keep a Changelog; [Unreleased] + link to GitHub Releases)
- replace the ~221KB per-version security log in SECURITY.md with a concise policy
- raise the coverage gate 50% -> 77% (measured 79% on the suite; real-2 margin)
No version bump: hygiene/docs only — rides into the 0.71.0 deps-split release.
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.
- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
CODEOWNERS patterns, and the PR-template checklist all repointed to
src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.
Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
Rich on narrow Windows columns splits `--lang` across colour-cycle ANSI
escapes (`\x1b[..m-\x1b[..m-lang`), so the literal substring check fails
on CI even though the rendered help renders correctly for humans. CI was
red on every commit landing after #234 hit a runner with that exact
column width + Python 3.9 + Rich combination.
Mirrors the `_ANSI_RE` strip pattern in tests/test_auto_tuning.py — flat
regex over the captured output before the `in` check.
v0.63.0 `score_uncertainty` raised on K>2 and `_row_uncertainty` fell back
to a monotone-broken `max(scores) - min(scores)`. Now generalises to K<=32
via population variance scaled by 4 — adding a fresh RM score equal to the
running mean strictly decreases uncertainty (the new contribution to the
sum-of-squares is zero while the denominator grows), so consensus on
redundant evidence can never spike the score.
K=1 max-entropy and K=2 disagreement formulas preserved verbatim (existing
operator dashboards depend on the |s1 - s2| value). Cap stays at K=32 for
DoS defence. _row_uncertainty K>2 path now routes through score_uncertainty
inside an isolated try/except — bad rows return 0.0 instead of crashing the
batch.
PEP 585 modernisation: collections.abc imports + list[...] annotations
(safe because `from __future__ import annotations` is in scope).
math.fsum used for the variance accumulation to keep rounding error
sub-ULP at K=32.
Tests: +32 net (25 in new tests/test_v0631_206.py + 7 TDD review-fix
followups). Full suite 11941 -> 11973 pass.
Closes#206.
Extends v0.69.0 Part E score_triviality + score_popularity_signal to
non-English corpora. New utils/brain_rot_lang.py ships a MappingProxyType
registry of frozen BrainRotLangBundle for en/es/fr/de/ru. Every public
scorer accepts an optional lang kwarg (default None preserves v0.69.0
English behaviour). The "auto" sentinel routes through the v0.53.10
[data-pro] langdetect helper with silent fallback to English on
missing-package / detector-exception / unsupported-code.
soup data brain-rot gains --lang en|es|fr|de|ru|auto, strictly validated
at the CLI boundary (exit 2 on typos). Per-row resolution backed by
eager _validate_lang_arg on dataset scorers so empty rows cannot
bypass shape checks.
Closes#234.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#241.
Adds opt-in tokenizer-aware n-gram path for the v0.70.0 Part F echo-trap detector. The existing whitespace `score_echo_signal` is unchanged; callers opt in via the new `score_trajectory_repetition_tokenized` / `score_echo_signal_tokenized` helpers or the `--echo-trap-tokenizer-aware` train flag.
Acceptance criterion from #241 verified: synthetic case where decoded strings differ by punctuation but token-id sequence repeats — whitespace path returns OK, tokenizer-aware path returns TRAP.
Post-release audit caught two pre-existing doc drifts that v0.70.0 should
have closed in the initial commit:
1. README.md `## All Commands` table was missing the 6 new v0.70.0 surfaces
(`soup iterative-dpo`, `soup train --reward-hack-detector`,
`--uld-strategy`, `--minillm-enabled`, `--rl-checkpoint-save-every-steps`,
`--echo-trap-enabled`). Added 6 lines mirroring the v0.69.0 entry format.
2. CONTRIBUTING.md `utils/` directory tree had stale drift from v0.66.0 →
v0.70.0 (24 modules missing across 5 releases). Brought in sync:
v0.66 (sae_diff, sleeper_probe, interference, probe_pack), v0.67
(cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock,
adapter_bisect), v0.68 (prompt_compile, prompt_distill, compile_tools,
apple_adapter, local_rl), v0.69 (build_dag, expectations, magpie,
persona_hub, brain_rot), v0.70 (reward_hacking, uld, minillm,
rl_checkpoint, iterative_dpo, echo_trap).
No code changes — docs-only hotfix per the checklist
"CI-only / docs-only hotfixes" rule. No version bump, no tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v0.67.0 and v0.68.0 releases shipped 14 new commands but neither
release updated the README `## All Commands` reference table per Release
Checklist Step 9. Adds the missing rows:
- v0.67.0 (5): adapters merge --strategy cmaes / adapters pr / adapters
bisect / lock write / lock show + check.
- v0.68.0 (9): compile / distill-prompt / compile-tools / apple-adapter +
local-rl init / status / record / harvest / train.
Docs-only — no source code change, no test count delta, no version bump
per the checklist's CI-only / docs-only hotfix policy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 commands that hedge Soup against paradigm shifts. If 1M-context kills FT,
`soup compile` (DSPy + GEPA + TextGrad prompt-program compilation) takes its
place. If teams hit prompt-cost walls, `soup distill-prompt` bridges to small
FT. If only Apple Foundation Models win on-device, `soup apple-adapter` ships
the converter+signing surface. If personal-LLM flywheels become the shape,
`soup local-rl` captures thumbs into SQLite and emits DPO pairs.
- Part A: `soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|...]`
- Part B: `soup distill-prompt --traces <jsonl> --teacher --student --strategy`
- Part C: `soup compile-tools <spec.json|yaml> --eval <jsonl>`
- Part D: `soup apple-adapter <source-dir> --direction hf-to-mlx|... --output`
- Part E: `soup local-rl init/status/record/harvest/train` (LIVE except train)
Schema + path containment + symlink rejection + atomic-write surface ship now;
live runners for Parts A/B/C/D + Part E nightly scheduler deferred to v0.68.1
(stub-then-live, mirrors v0.50.0 / v0.61.0 / v0.62.0 / v0.67.0 cadence).
Test count: 11021 -> 11225 (+204). Review-fix: 0 CRIT + 4 HIGH + 10 MED + 4 LOW.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- SECURITY.md: add v0.67.0 to supported-versions list
- CONTRIBUTING.md: bump test count 244 files / 10836 tests -> 251 / 11021
Docs-only follow-up to v0.67.0 release; no code change, no version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Was `assert soup_cli.__version__ == "0.64.0"` — broke on v0.65.0 bump on CI
across all 9 platform×Python matrix jobs. The test intent is "v0.64.0 has
shipped at least once"; switch to a tuple floor check matching the v0.51 /
v0.54 / v0.57 / v0.60 idiom.
Caught by CI red on v0.65.0 push to main; local pytest passed because we
ran the v0.65 test files in isolation per the Release Checklist Step 4
``pytest --no-cov`` invocation. Lesson: include the full suite in step 4
or grep for ``soup_cli.__version__ ==`` exact-equality assertions in any
future version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six new top-level commands close axis 1 + 11 of the roadmap: pick the
right base, lock the env, refuse OOMs before launch, and clear
license-clean deploys.
- soup tunability: probe-train 8 candidate bases (Qwen3-0.6/1.7B,
Llama-3.2-1/3B, Gemma-3-E2B, Phi-4-mini, SmolLM3, Qwen2.5-1.5B) ->
Pareto frontier over (delta x cost x license). Live LoRA probe -> v0.64.1.
- soup plan / soup apply: Terraform-shape lock-and-execute. `apply`
refuses on drift between soup.yaml and soup.tfstate (exit 3).
- soup env lock / status / check: hermetic env lockfile via
importlib.metadata across 15 ABI-sensitive packages + Python + CUDA.
`env check` exits 3 on drift.
- Hardware-fit calculator: static analytical 5-bucket VRAM predictor
with 10% safety margin + actionable hint on OOM.
- soup completions bash|zsh|fish: sourceable shell completion scripts;
recipe names auto-complete from the 115-recipe catalogue.
- soup license-advisor: per-deploy-target license matrix
(b2c/defense/embedded) + Llama community + 700M MAU gate (exit 3).
Composes with v0.60 license-conflict matrix.
Tests: 10035 -> 10306 (+271 net in 7 new files).
Review-fix coverage: 0 CRITICAL + 6 HIGH + 8 MEDIUM + 4 LOW across
consolidated code+security+TDD review wave. Every HIGH lands a regression
test in tests/test_v0640_followups.py (POSIX-skipped symlink rejection,
containment-before-existence ordering, drift-refusal exit-3 end-to-end).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docs-only hotfix — adds the v0.63.0 commands (soup ingest /
prune-prompt / data active-sample / ab / drift-alarm) to the
"## All Commands" code-block reference card. The release v0.63.0 had
dedicated ## feature sections + the ## What's New block updated, but
the All Commands quick-reference was missed during the doc pass.
No source code changed — no version bump, no tag, no release.
Per CLAUDE.md hotfix policy: docs-only commits don't ship to PyPI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 new top-level commands close axis 7 of the roadmap. Every Part LIVE on
day one (no deferred stubs):
- soup ingest: universal trace importer (Langfuse / LangSmith / Helicone /
OpenPipe / OTel / OpenAI Stored Completions). 6 adapters
+ frozen TraceRecord with MappingProxyType-wrapped metadata.
Zero credential-handling threat surface — Soup parses the
JSONL export, never makes the SaaS network call.
- soup prune-prompt: detect + strip a shared system-prompt prefix so the
FT model internalises it (OpenPipe's signature trick,
OSS). Binary-search over up to 32 templates finds the
longest threshold-meeting prefix.
- soup data active-sample: surface top-uncertainty prod traces for human
review. Max-entropy on single rm_score or
pairwise disagreement on dual rm_scores.
- soup ab: Wald sequential SPRT for the point alternative. LLR is a
martingale under H0 so Type-I error is controlled at every
stopping time per the optional stopping theorem.
- soup drift-alarm: rolling KL on whitespace-tokenised output distribution
+ SSRF-hardened Slack/Discord webhook (full parity with
v0.51.0 validate_hub_endpoint). Exit 3 on drift for
cron-friendly automation.
Test count: 9816 -> 10035 (+219 net across 6 new test files).
Review-fix coverage (code-reviewer + tdd-guide returned actionable;
python-reviewer + security-reviewer agents context-thrashed on the large
CLAUDE.md release-notes history — matches the v0.58.0 / v0.59.0 / v0.60.0
/ v0.61.0 / v0.62.0 idiom; verified manually):
- 1 CRITICAL: mSPRT log-likelihood-ratio sign error drove Type-I error
to 1.0 as n grew. Replaced with Wald's classic point-
alternative SPRT (martingale under H0).
- 2 HIGH: detect_common_prefix early-exit on 100% match returned the
shortest qualifying prefix instead of the longest;
_MAX_SCAN_ROWS DoS cap used 'pass' instead of 'break'.
- 3 MEDIUM: TraceRecord.metadata now MappingProxyType-wrapped post-init
(frozen-dataclass mutation hazard); _AUTH_ENV table
deduplicated; drift_alarm precedence parens on SSRF gate.
- 2 LOW: pooled_se dead-branch refactor; mean_uncertainty NaN guard.
- 8 follow-up tests: msprt zero-variance, partial-majority binary-search
activation, score_uncertainty exact boundaries, rolling_kl identical
+ disjoint, validate_budget + validate_threshold exact endpoints,
_signal_from_thumbs boundaries, no-heavy-top-level-imports source-grep
guard across all 5 new util modules.
Step 6 smoke verified for all 5 commands + 6 failure-mode rejection
paths.
CRLF gotcha note for future maintainers: PowerShell wrote the smoke
fixtures with a UTF-8 BOM on Windows during Step 6 — switched to
inline Python for the fixture write. Production CLI input handling is
already BOM-tolerant (utf-8-sig in JSONL loaders via v0.40.1 Part E).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpamys
gittihub-jpg
Shivam