Commit Graph

4 Commits

Author SHA1 Message Date
Alpamys 06d8ea7cc5 chore: migrate to src-layout
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.

- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
  The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
  and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
  CODEOWNERS patterns, and the PR-template checklist all repointed to
  src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
  path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
  over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
  references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
  the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.

Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
2026-05-31 12:40:06 +05:00
Alpamys 58d7d510bf feat(eval): soup eval design — derive evals from data (v0.55.0)
Trainer libraries help you RUN evals — none help you DEFINE them.
v0.55.0 closes that gap with 5 new subcommands:

- soup eval design <data> --goal "..."  → goal-conditioned EvalDesign
                                          (TF-IDF salience + scorer dispatch)
- soup eval discover <data>             → held-out canaries + memorization probes
                                          (farthest-first Jaccard clustering)
- soup eval lock + soup eval coverage   → SHA-256-checksummed artifact +
                                          gap analysis vs v0.54.0 task taxonomy
- soup eval gate-install --baseline R   → pre-push regression gate
                                          (paired-bootstrap CI, shlex.quote)
- soup eval against B --candidate C     → run-vs-run paired-bootstrap CI

Heuristic / CPU-only — no GPU required. Lazy imports across all 6 new
modules so `soup --help` startup remains < 200 ms.

New registry artifact kinds: eval_suite, canaries.
New tracker accessor: ExperimentTracker.get_metric_series(run_id, metric).

Security policy (all atomic-write + read surfaces):
  - cwd containment via os.path.realpath + commonpath
  - unconditional os.lstat + stat.S_ISLNK rejection (TOCTOU defence)
  - atomic write via tempfile.mkstemp + os.replace
  - shlex.quote for shell-script generation (NO hand-rolled escape)
  - MappingProxyType on every registry / metric / scorer map
  - frozen dataclass on every public return type
  - bool-as-int rejection on every numeric input
  - DoS caps: 10k subsample for TF-IDF + clustering hot paths

Review-fix coverage across 4 agents (python / security / code / tdd):
0 CRITICAL + 7 HIGH + 11 MEDIUM + 6 LOW resolved before commit.

Tests: 8571 → 8676 (+105 net).
Lint: ruff clean.
Smoke: every CLI command + every failure mode exercised in /tmp/soup_smoke.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 12:44:14 +05:00
Alpamys 9e18643ea0 test(advise): fix cross-platform CI failures in test_v0540
Two failures on ubuntu/macos/windows × py3.9/3.11/3.12 after v0.54.0
push:

1. test_env_null_byte_falls_back: monkeypatch.setenv can't set raw
   NUL into the OS env layer (POSIX execve + Win32 SetEnv both
   refuse). Switched to a temporary `advise_history.os.environ` swap
   so the helper's defence-in-depth NUL guard is still exercised
   without going through the C env layer.

2. test_default_missing_data: Click 8.0–8.1 returns rc=0 on
   `no_args_is_help=True` invocations; Click 8.2+ returns rc=2 (the
   "missing command" convention). CI runners had the newer Click;
   dev box had the older. Accept both renderings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 21:58:58 +05:00
Alpamys 600686cd70 feat(advise): soup advise — pre-flight decision (v0.54.0)
`soup advise <data.jsonl> --goal "..."` returns one of PROMPT_ENG /
RAG / SFT / DPO / GRPO with a confidence, reason, and reverse-when
criterion BEFORE the user spends 8 hours on a GPU. Layer above
autopilot — autopilot picks hyperparams AFTER the training decision;
advise picks the training decision itself.

Three Parts:
- Part A: Verdict engine — TASK_CATEGORIES + CHOICES allowlists,
  frozen Verdict / DatasetProfile / ROIEstimate dataclasses, pure-
  Python classify_task + compute_dataset_profile + build_verdict
  rubric (DPO / GRPO floor 500 / PROMPT_ENG floor 50 / RAG / SFT).
- Part B: Probe runner — synth_probe_baselines + synth_probe_lora_delta
  heuristic stubs with forward-compat model/device/lr/timeout_seconds
  kwargs (v0.54.1 lifts to live model loading per stub-then-live
  cadence used by v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus).
- Part C: Cross-project learning — ~/.soup/advise_history.jsonl with
  cross-process file locking (fcntl on POSIX, sidecar <path>.lock +
  msvcrt on Windows). `soup advise compare` reads history; env
  override SOUP_ADVISE_HISTORY_PATH containment-checked to $HOME /
  $CWD / tempdir (mirrors v0.36.0 SOUP_BATCH_CACHE_PATH policy).

CLI: Typer subcommand group `run` / `explain` / `compare` plus argv
preprocessor in cli.py that maps `soup advise data.jsonl` →
`soup advise run data.jsonl`. Scoped to argv[1] == "advise" only
(code-review HIGH fix — defends against rewrites when an unrelated
arg contains the literal string "advise").

Schema: AdviseConfig (goal / probe / record) field on SoupConfig
honors the plan's cross-cutting bullet.

Security: cwd-containment + os.lstat + S_ISLNK symlink reject on
every path input; atomic writes via tempfile.mkstemp + os.replace
on scratch + history; per-line 64 KB cap + 16 MiB file cap on
history reads; bool / finite / NUL / oversize guards on every public
input; Rich markup escape on user-controlled output.

Reviewed by python / code / security / tdd / architect agents — every
finding fixed before commit (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW).

Test count: 8400 → 8571 (+136 in tests/test_v0540.py, +35 net
adjustments to v0.53.x version-pin assertions to forward-compat >=).

Note: Windows CRLF / LF warnings during stage are .gitattributes-
governed and benign. CI runs on ubuntu-latest / windows-latest /
macos-latest × Python 3.9 / 3.11 / 3.12.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 21:48:19 +05:00