Commit Graph

4 Commits

Author SHA1 Message Date
Alpamys 06d8ea7cc5 chore: migrate to src-layout
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.

- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
  The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
  and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
  CODEOWNERS patterns, and the PR-template checklist all repointed to
  src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
  path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
  over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
  references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
  the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.

Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
2026-05-31 12:40:06 +05:00
Alpamys 72189baba7 feat(diagnose): soup diagnose — post-training model report card (v0.56.0)
Six failure-mode probes (forgetting / refusal / format / mode_collapse /
memorization / contamination) + FailureReport frozen dataclass + SVG
badge + soup train --diagnose-gate. Same OK/MINOR/MAJOR taxonomy as
v0.26.0 Quant-Lobotomy.

- soup diagnose <run-id> [--evidence|--output|--badge|--attach-to-registry]
- soup train --diagnose-gate <evidence.json> refuses MAJOR runs
- diagnose_report added to registry._VALID_KINDS

Review wave (4 agents): 4 HIGH + 8 MEDIUM + 2 LOW addressed —
atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), realpath
containment, evidence size cap, contamination combined-complexity cap,
ReDoS probe, extras null-byte sanitisation, extract_row_text
centralisation, tokenize delegates to _eval_text.

Test count: 8676 -> 8849 (+123 in test_v0560.py + 50 net adjustments).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 15:03:00 +05:00
Alpamys 04c504e761 test(eval): strip ANSI before help-output substring asserts (v0.55.0 CI fix)
3 macOS CI failures from the v0.55.0 push — Rich wraps option names
with ANSI escapes when the terminal is narrow (macOS CI runners
default to a smaller width than Linux/Windows), so substring searches
like `"--goal" in result.output` fail because the actual output
contains `\x1b[1;36m-\x1b[0m\x1b[1;36m-goal\x1b[0m`.

Project precedent: v0.53.5 / v0.53.6 / v0.53.8 / v0.53.9 all hit the
same pattern; tests/test_auto_tuning.py and tests/test_eval_platform.py
already ship `_ANSI_RE` + `_strip_ansi` helpers.

Failures fixed:
  tests/test_v0550.py::TestCLIPlumbing::test_eval_design_help
  tests/test_v0550.py::TestEvalAgainst::test_against_help
  tests/test_v0550_followups.py::TestEvalAgainst::test_against_cli_help_lists_flag

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 13:10:53 +05:00
Alpamys 58d7d510bf feat(eval): soup eval design — derive evals from data (v0.55.0)
Trainer libraries help you RUN evals — none help you DEFINE them.
v0.55.0 closes that gap with 5 new subcommands:

- soup eval design <data> --goal "..."  → goal-conditioned EvalDesign
                                          (TF-IDF salience + scorer dispatch)
- soup eval discover <data>             → held-out canaries + memorization probes
                                          (farthest-first Jaccard clustering)
- soup eval lock + soup eval coverage   → SHA-256-checksummed artifact +
                                          gap analysis vs v0.54.0 task taxonomy
- soup eval gate-install --baseline R   → pre-push regression gate
                                          (paired-bootstrap CI, shlex.quote)
- soup eval against B --candidate C     → run-vs-run paired-bootstrap CI

Heuristic / CPU-only — no GPU required. Lazy imports across all 6 new
modules so `soup --help` startup remains < 200 ms.

New registry artifact kinds: eval_suite, canaries.
New tracker accessor: ExperimentTracker.get_metric_series(run_id, metric).

Security policy (all atomic-write + read surfaces):
  - cwd containment via os.path.realpath + commonpath
  - unconditional os.lstat + stat.S_ISLNK rejection (TOCTOU defence)
  - atomic write via tempfile.mkstemp + os.replace
  - shlex.quote for shell-script generation (NO hand-rolled escape)
  - MappingProxyType on every registry / metric / scorer map
  - frozen dataclass on every public return type
  - bool-as-int rejection on every numeric input
  - DoS caps: 10k subsample for TF-IDF + clustering hot paths

Review-fix coverage across 4 agents (python / security / code / tdd):
0 CRITICAL + 7 HIGH + 11 MEDIUM + 6 LOW resolved before commit.

Tests: 8571 → 8676 (+105 net).
Lint: ruff clean.
Smoke: every CLI command + every failure mode exercised in /tmp/soup_smoke.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 12:44:14 +05:00