diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bfa264c..c61040e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ soup_cli/ templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (257 files, 11225 tests) +tests/ - Test suite (262 files, 11487 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index cabaa23..3115702 100644 --- a/README.md +++ b/README.md @@ -42,44 +42,51 @@ soup train Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases). -**v0.68.0 — Anti-trend Insurance: `soup compile` (DSPy/GEPA) + `soup distill-prompt` + `soup compile-tools` + `soup apple-adapter` + `soup local-rl` daemon.** Five bets that hedge Soup against paradigm shifts. If 1M-token contexts kill fine-tuning, `soup compile` (DSPy + GEPA prompt-program compilation) takes its place. If teams hit prompt-cost walls, `soup distill-prompt` bridges to small FT. If only Apple Foundation Models win on-device, `soup apple-adapter` ships the converter+signing surface. If personal-LLM flywheels become the shape, `soup local-rl` daemon captures thumbs into SQLite and emits DPO pairs nightly. None of these exist as CLI workflows in the OSS or hosted ecosystem — DSPy/GEPA are libraries, prompt distillation is unowned, Apple FoundationModels adapter format is undocumented territory, personal-LLM flywheels are research-stage. Schema-only release for Parts A/B/C/D — live wiring deferred to v0.68.1. Part E ships LIVE except the nightly train scheduler. +**v0.69.0 — Data Engineering Pro: `soup build` (dbt-for-SFT) + `soup expect` + `soup data gen-magpie` + `soup data persona-mix` + `soup data brain-rot`.** Five surfaces that turn dataset preparation into a first-class engineering workflow. `soup build` is a dbt-style DAG of dataset transforms with `ref()`-connected models, `incremental` materialization, and content-hash-based row-diff so re-runs re-tokenize only changed rows. `soup expect` ships Great-Expectations for chat data — `expect_no_pii` / `expect_token_length_between` / `expect_no_refusal_pattern` / `expect_chosen_preferred_over_rejected_by_judge` — with exit code 3 on suite failure so CI pipelines can gate on dataset quality regressions. `soup data gen-magpie` plans the Magpie synthetic generator (chat-template-prefix harvest, reuses v0.20 providers). `soup data persona-mix` samples a prompt × persona × style matrix with a bundled 12-persona / 5-style diversity set + topic-entropy metric. `soup data brain-rot` implements the arXiv 2510.13928 detector with OK/MINOR/MAJOR verdicts that compose with v0.47 educational scorer; `--strict` mode exits 3 when too many rows score MAJOR. -- **`soup compile --eval [--optimizer mipro|gepa|textgrad|copro|bootstrap_fewshot] [--plan-only]`** — DSPy / GEPA prompt-program compiler. Closed allowlist over the canonical 5 optimisers. `CompilePlan` is frozen and re-validated on construction — direct-construction bugs that bypass `build_compile_plan` still produce loud failures. `CompileResult` rejects NaN / ±Inf scores via `math.isfinite`, bool-as-int, and negative iterations. Live orchestrator (DSPy / GEPA / TextGrad) deferred to v0.68.1; the CLI renders the validated plan today and exits 3 on the deferred path (mirrors v0.50.0 / v0.61.0 stub-then-live cadence). -- **`soup distill-prompt --traces --teacher --student --strategy sft|preference|kl`** — Take prompt-heavy traces from GPT-5 / Claude calls and prepare a distillation dataset targeting a small student model. Bridge between prompt-engineering and FT worlds. Composes with v0.70 cross-tokenizer KD when that ships. Schema + path containment + symlink rejection ship now; the teacher/student tokeniser-bridge runner lands in v0.68.1. -- **`soup compile-tools --eval [--optimizer textgrad|gepa]`** — TextGrad / GEPA tool-schema optimiser. Reads OpenAPI / MCP / GraphQL specs and prepares to optimise tool descriptions via textual gradients. Composes with v0.46 Agent Forge — Agent Forge parses the spec, `soup compile-tools` optimises the descriptions. `validate_spec_path` enforces JSON / YAML extension allowlist + cwd containment + symlink rejection. -- **`soup apple-adapter --direction hf-to-mlx|mlx-to-hf|hf-to-apple|mlx-to-apple --output [--sign]`** — Adapter format conversion + optional v0.60 Merkle-root signing. Extends v0.25 MLX backend. `validate_source_adapter` enforces cwd containment + `stat.S_ISDIR` (must be a directory) + symlink rejection. The `--sign` flag flows the converted adapter through v0.60 signing for distribution / sigstore. -- **`soup local-rl init / status / record / harvest / train`** — Personal-LLM flywheel daemon. Smaller-scope cousin of v0.58 `soup loop` — runs locally on a single workstation, trains your personal model from your own feedback. **LIVE today**: `init` creates the SQLite schema (`interactions` + `thumbs` with CHECK constraints + `0o600` perms on POSIX); `record` appends a thumbs-up/down row via parameterised query; `status` renders a Rich table of counters; `harvest` walks the thumbs table and emits `{prompt, chosen, rejected}` JSONL pairs (last-writes-win dedup so duplicate thumbs collapse to one pair per prompt). The `train` subcommand (Ollama / MLX × DPO/KTO/ORPO) is deferred to v0.68.1 — operators today can harvest into `dpo_pairs.jsonl` and feed it to existing `soup train --task dpo`. -- **+204 new tests** (11021 → 11225) across 6 new test files (5 part files + 1 followups covering every TDD-review finding). Review-fix coverage: 0 CRITICAL + 4 HIGH (validate_student_id rejection-matrix parity / Part C CLI unknown-optimizer exit 2 / harvest_dpo_pairs edge cases / POSIX symlink skip predicate uses `sys.platform`) + 10 MEDIUM (CompileResult ±Inf rejection + bool-score / validate_eval_suite_path symlink test / Part C empty + oversize on tool-optimizer / Part C null-byte + symlink on spec_path / TestBuildToolCompilePlan factory / Part D non-string + oversize on direction / TestBuildAppleAdapterPlan factory / Part E backend + train_method full rejection matrix / record_thumb response null-byte + oversize + empty / SUPPORTED_LOCAL_RL_TRAIN_METHODS immutability) + 4 LOW (TestInitDb column-level schema via PRAGMA / Part D output_dir validation / validate_db_path public + in __all__ / DpoPair frozen). Manual CPU smokes (Step 6): every command's `--help` + happy-path `--plan-only` + deferred-live exit 3 + validation rejection exit 2; full `soup local-rl init → record up → record down → status → harvest` end-to-end producing valid DPO JSONL. +- **`soup build [--dry-run]`** — dbt-for-SFT DAG parser + topological sort + plan rendering. Each model declares `kind: incremental|table|view`, a `transform`, and either a `source:` (seed) or `refs: [...]` (derived). Cross-validators reject ambiguous shapes (no-refs + no-source = degenerate; refs + source = mutually exclusive). `compute_row_hash` + `incremental_diff(prev, new) -> IncrementalDiffReport(added, changed, removed, unchanged)` are the kernel that lets the live runner (v0.69.1) re-tokenize only changed rows. `BuildModel` and `BuildPlan` are frozen dataclasses; `load_build_yaml` delegates to `paths.enforce_under_cwd_and_no_symlink` (TOCTOU defence). Live runner deferred to v0.69.1; today `--dry-run` renders the plan + exits 0, no flag exits 3 with the deferred-live marker. +- **`soup expect `** — Great Expectations for chat data. Four built-in expectation functions, each a pure function returning a frozen `ExpectationResult`. Composes with v0.47.0 `data_score.detect_pii` (Presidio backend when `[data-pro]` installed), v0.56.0 `diagnose.refusal.looks_like_refusal`, and v0.19.0 judge backends (operator-injected callable for the chosen-vs-rejected judge). `_dispatch_expectation` passes args through raw so per-expectation validators (bool-rejection, NaN-rejection, range checks) fire authentically — no silent int/float coercion bypasses the validator. Exit 0 = pass, 2 = validation rejection, 3 = suite failure. +- **`soup data gen-magpie --base --provider ollama|anthropic|vllm --target N [--plan-only]`** — Magpie technique (Xu et al. 2024) schema + plan. Feeds the chat-template prefix only to an aligned base model and harvests user-side turns via v0.20 providers. `MagpieConfig` frozen, `validate_magpie_provider` closed allowlist, `validate_target_rows` ∈ [1, 1_000_000] bool-rejected. Live generation loop (provider calls + v0.47 quality filter) deferred to v0.69.1. +- **`soup data persona-mix --prompts --n N --output `** — Persona-Hub-style diversity sampler. Reads `--prompts` JSONL, multiplies through bundled 12-persona × 5-style matrix (or operator-supplied `--personas` / `--styles` JSONL), writes `{prompt, persona, style}` per row via atomic `tempfile.mkstemp + os.replace`. Deterministic by `--seed`. New `compute_topic_diversity` Shannon-entropy kernel for downstream gating. Centralised TOCTOU policy via `enforce_under_cwd_and_no_symlink` on every read AND write path. +- **`soup data brain-rot [--strict] [--max-major-fraction 0.25]`** — arXiv 2510.13928 brain-rot detector. Two orthogonal slop scorers: `score_triviality` (low diversity / excessive `!!`/`??` punctuation / `lol/omg/lmao` density / length penalty) and `score_popularity_signal` (clickbait phrase substrings + emoji density). Per-row composite = `1.0 - max(triviality, popularity)`. Same OK/MINOR/MAJOR taxonomy as v0.26 / v0.56 / v0.65 (≥0.85 OK, ≥0.60 MINOR, else MAJOR). `--strict` exits 3 when MAJOR-row fraction exceeds the threshold so training pipelines can refuse to materialise slop datasets. `--max-major-fraction` validated at the CLI boundary (NaN / Inf / out-of-range / bool rejected) before any scoring. +- **+264 new tests** (11225 → 11487) across 5 part files. Review-fix coverage: 1 CRITICAL (centralised duplicate cwd+symlink blocks behind `paths.enforce_under_cwd_and_no_symlink` in build_dag / expectations / expect.py — code-review HIGH) + 4 HIGH (brain-rot loader DoS caps; persona-mix `--output` symlink TOCTOU rejection; persona-mix `_load_jsonl_field` DoS caps; magpie `quality_filter` validator + `_dispatch_expectation` int/float-coercion bypass) + 5 MEDIUM (BuildModel seed/derived cross-validator + docs; `--max-major-fraction` CLI-boundary validation; `expect` skipped-line WARNING) + 4 LOW (boundary tests at exact threshold ± ε on classify_brain_rot; BuildPlan FrozenInstanceError; version_bumped checks in B/C/D/E; lazy-yaml import source-grep). Manual CPU smokes (Step 6): every CLI happy + failure path exercised, including 3-stage build DAG dry-run, PII suite exit 3, magpie plan-only, persona-mix atomic write to JSONL, brain-rot MAJOR-on-slop exit 3. -## Anti-trend Insurance +## Data Engineering Pro -The v0.68.0 release ships 5 commands that hedge Soup against paradigm shifts. Each is one of the 5 "if X kills FT, Soup still survives" bets. +The v0.69.0 release ships 5 surfaces that turn dataset prep from "throw a JSONL at the trainer" into a first-class engineering workflow. ```bash -# DSPy / GEPA prompt-program compilation — replaces FT if 1M-context wins -soup compile my_program.py --eval evals.json --optimizer mipro --max-iters 10 +# dbt-for-SFT — DAG of dataset transforms with incremental materialization +cat > build.yaml << 'EOF' +models: + - {name: raw, kind: incremental, source: data/raw.jsonl, transform: identity} + - {name: filtered, kind: incremental, refs: [raw], transform: filter_low_quality} + - {name: tokenized, kind: incremental, refs: [filtered], transform: tokenize} +EOF +soup build build.yaml --dry-run # validate topology + plan +# soup build build.yaml # live materialise (v0.69.1) -# Distill prompt-heavy traces into small FT — bridges prompt-eng to FT worlds -soup distill-prompt --traces traces.jsonl --teacher anthropic/claude-3-5-sonnet \ - --student meta-llama/Llama-3.2-1B --strategy sft +# Expectations suite — Great Expectations for chat data +cat > suite.yaml << 'EOF' +expectations: + - {name: expect_no_pii} + - {name: expect_token_length_between, args: {min_tokens: 16, max_tokens: 4096}} + - {name: expect_no_refusal_pattern} +EOF +soup expect data.jsonl suite.yaml # exit 3 on suite failure -# Tool-schema textual-gradient optimization — composes with `soup agent` (v0.46) -soup compile-tools api_spec.yaml --eval tool_eval.jsonl --optimizer textgrad +# Magpie synthetic data — chat-template-prefix harvest (plan, runner v0.69.1) +soup data gen-magpie --base meta-llama/Llama-3.1-8B-Instruct \ + --provider ollama --target 1000 --plan-only -# HF / MLX / Apple FoundationModels adapter conversion + optional signing -soup apple-adapter ./my_adapter --direction hf-to-mlx --output ./mlx_adapter -soup apple-adapter ./my_adapter --direction hf-to-apple --output ./apple_adapter --sign +# Persona-Hub diversity — prompt × persona × style matrix sampling +soup data persona-mix --prompts prompts.jsonl --n 500 --output mixed.jsonl -# Personal-LLM flywheel — capture thumbs into SQLite, harvest DPO pairs nightly -soup local-rl init --db local_rl.db -soup local-rl record --db local_rl.db --prompt "What is X?" --response "..." --thumb up -soup local-rl record --db local_rl.db --prompt "What is X?" --response "wrong" --thumb down -soup local-rl status --db local_rl.db -soup local-rl harvest --db local_rl.db -o dpo_pairs.jsonl -# soup local-rl train (nightly DPO/KTO/ORPO) — v0.68.1 +# Brain-rot detector (arXiv 2510.13928) — refuses to train on excessive slop +soup data brain-rot data.jsonl --strict --max-major-fraction 0.10 ``` -Live runners for Parts A/B/C/D + the local-rl nightly scheduler land in v0.68.1. All schema + validation + path containment + symlink-rejection + atomic-write surfaces ship today, so misconfigured commands fail loudly at config load. +Every command applies the project-wide TOCTOU policy (`os.lstat + S_ISLNK` symlink rejection before any open) and cwd containment via the shared `paths.enforce_under_cwd_and_no_symlink` helper. Live runners for `soup build` and `soup data gen-magpie` land in v0.69.1; the other three are LIVE today. ## Why Soup? @@ -3768,6 +3775,11 @@ soup local-rl status --db Print interactions / thumbs-up / t soup local-rl record --db --prompt --response --thumb up|down Append thumbs record soup local-rl harvest --db -o Harvest DPO pairs from thumbs into JSONL soup local-rl train --db --backend ollama|mlx --model Nightly DPO/KTO/ORPO train (v0.68.1) +soup build [--dry-run] dbt-for-SFT DAG: validate + plan dataset transforms (v0.69.0) +soup expect Expectations suite: PII / token-length / refusal / judge (v0.69.0) +soup data gen-magpie --base --provider ollama|anthropic|vllm --target N [--plan-only] Magpie synthetic generator (v0.69.0) +soup data persona-mix --prompts --n N --output Persona-Hub diversity sampler (v0.69.0) +soup data brain-rot [--strict] Brain-rot detector — arXiv 2510.13928 (v0.69.0) soup version [--full] [--json] Show version (--full: system info, --json: JSON output) soup --verbose Full traceback on errors ``` diff --git a/SECURITY.md b/SECURITY.md index bb0feb1..bfb8175 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,7 +9,8 @@ We provide security updates for the following versions: - **Versions older than 3 minor versions:** No support Example: -- v0.68.0 -- Full support (latest) +- v0.69.0 -- Full support (latest) +- v0.68.0 -- Full support - v0.67.0 -- Full support - v0.66.0 -- Full support - v0.65.0 -- Full support @@ -142,6 +143,8 @@ No known critical vulnerabilities in current releases. ### Security Hardening History +- **v0.69.0 — Data Engineering Pro**: 5 parts ship `soup build` (dbt-for-SFT DAG) + `soup expect` (expectations suite) + `soup data gen-magpie` + `soup data persona-mix` + `soup data brain-rot`. **CRITICAL — centralised TOCTOU helper**: replaces 3 duplicate `os.lstat + S_ISLNK + realpath + is_under_cwd` blocks in `build_dag.load_build_yaml` / `expectations.load_suite_yaml` / `expect._load_jsonl_rows` with the shared `paths.enforce_under_cwd_and_no_symlink` helper (v0.59.0 single-source-of-truth policy — mirrors v0.40.6 / v0.53.5 centralisation enforcement). **HIGH — DoS caps on every new JSONL loader**: brain-rot data path capped at 1 GiB + 1M rows, persona-mix `--prompts` / `--personas` / `--styles` capped at 100 MiB + 100k entries, expect data path capped at 1 GiB + 1M rows (mirrors v0.55 / v0.65 _MAX_ROWS policy). **HIGH — TOCTOU symlink rejection on every write target**: `persona-mix --output` and `soup expect` data input now both pass through `enforce_under_cwd_and_no_symlink` so a pre-placed `out.jsonl -> /etc/cron.d/x` cannot redirect atomic writes. **HIGH — validator bypass fixed**: `magpie.build_magpie_config` now explicit `isinstance(quality_filter, bool)` (was passing non-bool through to the dataclass); `expectations._dispatch_expectation` passes raw args to per-expectation validators (was `int()`/`float()`-coercing inputs which silently bypassed `_check_token_bound` bool-rejection). **MEDIUM — `BuildModel` cross-validator** rejects ambiguous shapes at schema load: `refs=() + source=None` (degenerate, no input) and `refs + source set` (ambiguous) both raise `ValueError` so misconfigured manifests fail loudly. **MEDIUM — `--max-major-fraction` CLI-boundary validation**: bool / NaN / Inf / out-of-range rejected before any scoring pass (security-review M2). **Closed-allowlist validators**: `SUPPORTED_MODEL_KINDS = {incremental, table, view}` (build_dag), `SUPPORTED_EXPECTATIONS = {expect_no_pii, expect_token_length_between, expect_no_refusal_pattern, expect_chosen_preferred_over_rejected_by_judge}`, `SUPPORTED_MAGPIE_PROVIDERS = {ollama, anthropic, vllm}`, `BRAIN_ROT_VERDICTS = (OK, MINOR, MAJOR)` — all bool-first / null-byte / oversize (≤32 / ≤128-char) / case-insensitive normalisation. **Brain-rot scoring is ReDoS-safe**: `_PUNCT_PATTERN` uses bounded `{2,}` quantifiers; `_CLICKBAIT_PHRASES` is substring scan (not regex); `_TEXT_FIELDS` capped at `_MAX_TEXT_LEN=65_536`. **No top-level heavy imports** — torch / transformers / peft / sentence_transformers / yaml never imported at module scope in any v0.69.0 utility module (source-grep regression tests). **Atomic writes** via `tempfile.mkstemp + os.replace` + parent-realpath in `persona-mix`. **Worst-signal composite** for brain-rot row score: `1.0 - max(triviality, popularity)` so a single strong slop signal drives the verdict down hard (mirrors v0.56.0 `overall_verdict` worst-case policy). **Exit-code policy**: `soup build` and `soup data gen-magpie` exit 3 on deferred-live (distinct from validation rejection exit 2); `soup expect` and `soup data brain-rot --strict` exit 3 on gate failure (matches v0.55 / v0.56 / v0.64 / v0.65 gate convention). **Test count**: 11225 → 11487 (+262 net in `tests/test_v0690_part_{a-e}.py`; 264 pass + 3 POSIX-skipped symlink tests on Windows). **Review-fix coverage across 4 review waves** (security + code + python + TDD; direct reviewer agents context-thrashed on 800+ KB CLAUDE.md — `general-purpose` agent with explicit "do not crawl, read only these 9 files" prompts produced equivalent findings consolidated): 1 CRITICAL (TOCTOU helper centralisation) + 4 HIGH (DoS caps, output symlink, persona loader caps, validator bypass) + 5 MEDIUM (BuildModel cross-validator, --max-major-fraction validation, expect-loader WARN-on-skip) + 4 LOW (boundary tests at OK/MINOR/MAJOR ± ε, BuildPlan FrozenInstanceError, version_bumped checks, lazy-yaml import grep). **Known limitations**: (1) Live `soup build` runner deferred to v0.69.1 — `run_build` raises NotImplementedError; `--dry-run` produces fully-validated topological plan today. (2) Live Magpie generator deferred to v0.69.1 — `run_magpie` raises NotImplementedError; `--plan-only` renders the resolved config today. (3) `BuildModel.source` schema-time containment NOT enforced (only shape) — live v0.69.1 runner must apply `enforce_under_cwd_and_no_symlink` before reading source paths; schema permits relative paths so a build can be planned offline before the data lands. (4) `expect_chosen_preferred_over_rejected_by_judge` advisory-pass when `judge_fn=None` — every row scores 1.0; production callers MUST supply a real judge. (5) Brain-rot detector is English-keyword-only — `_LOW_EFFORT_TOKENS` + `_CLICKBAIT_PHRASES` are English; entropy + punct-run signals remain language-agnostic. (6) Persona-Hub bundled 12-persona × 5-style set by default; full Tencent corpus (200k personas) NOT bundled — operators pass `--personas ` to swap in the full set. (v0.69.0) + - **v0.66.0 — Post-train X-rays**: 5 LIVE parts extend `soup diagnose` from 6 failure modes to 10 — SAE feature diff (Part A), live influence-function blame closing v0.57 #171 (Part B), sleeper-agent defection probe (Part C), pairwise adapter interference matrix (Part D), and probe pack manifest (Part E). **TOCTOU `O_NOFOLLOW` probe-open** — `load_sae_weights` (Part A) and `_count_dataset_rows` (Part B) both replace the v0.57 `os.path.realpath + open()` pattern (which left a race window between containment check and read) with `os.open(path, os.O_RDONLY | os.O_NOFOLLOW)`. A symlink swap between containment check and read now raises immediately rather than silently following the link. **Process-independent deterministic seeds** (HIGH H3) — `_probe_weights` (sleeper) and `_default_synthetic_probe` (blame) replace Python's process-salted `hash()` with `int(hashlib.sha256(key).hexdigest()[:16], 16)` (64-bit seed). CI reproducibility now works regardless of `PYTHONHASHSEED`. **Rich-markup escape on operator-controlled strings** (HIGH H2) — `render_matrix_markdown` (interference), `render_blame_markdown` (blame), and `render_pack_markdown` (probe_pack) all route adapter names / verdict / description / layer through a local `_md_escape` (`[` → `\\[`, `]` → `\\]`) before embedding into Rich markdown. A crafted adapter name like `"[link=evil]click[/]"` cannot inject markup or open file: URIs in the user's terminal. **TypeError on bool/non-str verdict** (HIGH H1) — `SleeperProbeResult.__post_init__` and `InterferenceCell.__post_init__` type-check `verdict` BEFORE the membership check, so `verdict=True` raises TypeError (was: misleading ValueError about membership). **Non-numeric loss rejection** (HIGH H3 CLI) — `soup probe interference` CLI rejects string / bool / list values in the `losses` payload before they reach the math kernel; JSON booleans (`true`/`false`) deserialise to Python `bool` which is a subclass of `int` so the explicit `isinstance(value, bool)` guard fires first. **10M-row hard rejection** (MEDIUM M4) — `_count_dataset_rows` now raises `ValueError("dataset has >10M rows; subsample before blame")` instead of silently truncating to 10M (which would have corrupted the plan). **Synthetic probe row cap** (MEDIUM M6) — `_DEFAULT_SYNTH_PROBE_CAP=100_000`: without an operator-supplied `probe_fn`, `_default_synthetic_probe` caps at 100k rows so a 10M-row dataset doesn't allocate 1.28 GB of float32 (10M × 32 × 4 B). **Description length cap** (MEDIUM M5) — `ProbeEntry._MAX_DESCRIPTION_LEN=4096`; operator-controlled description field cannot blow up Rich render or downstream JSON sinks. **Safetensors key-count pre-check** (MEDIUM) — `load_sae_weights` validates `len(f.keys()) > 64` BEFORE materializing any tensor; the DoS cap now defends against pathological files (was: cap fired AFTER N tensors were already loaded). **Frozen dataclasses everywhere** — `SaeFeatureChange`, `SaeFeatureDiffReport`, `RowInfluence`, `BlameResult`, `SleeperProbeSpec`, `SleeperProbeResult`, `InterferenceCell`, `InterferenceMatrix`, `ProbeEntry`, `ProbePack` all use `@dataclass(frozen=True)` + `__post_init__` validation; all report tests assert `pytest.raises(FrozenInstanceError)` (HIGH H1 wave 3 — replaces the previous `pytest.raises((AttributeError, Exception))` which would have silently passed if the dataclass dropped its frozen flag). **`MappingProxyType` for read-only registries** — `BUNDLED_PROBES`, `BUNDLED_PACKS`, `_LOWER_INDEX` (sleeper case-insensitive lookup, O(1) instead of O(N) loop). **No top-level heavy imports** — `torch` / `transformers` / `peft` / `safetensors` never imported at module scope in any v0.66 module (source-grep regression tests). **Test count**: 10577 → 10836 (+259 net across `tests/test_v0660_part_{a-e}.py` + `tests/test_v0660_cli.py` + `tests/test_v0660_followups.py`). **Review-fix coverage** across 3 sequential waves (python-review + code-review + security-review + tdd-guide): 0 CRITICAL + 9 HIGH + 14 MEDIUM + 5 LOW. (v0.66.0) - **v0.65.0 — Eval Depth**: 5 LIVE parts ship `soup eval behavior / capability / checklist / irt-subset` + judge calibration with conformal abstention. **TOCTOU `O_NOFOLLOW` + `os.fstat` on SAME fd** — `load_checklist_spec` + `load_response_rows` + `_read_evidence_json` all use `os.open(path, O_RDONLY | O_NOFOLLOW)` + `os.fstat(fd)` + `os.fdopen(fd, ..., closefd=True)` so the shared `enforce_under_cwd_and_no_symlink` helper's lstat-on-raw + the downstream open both target the SAME file descriptor (HIGH H-NEW-1 / H-NEW-2 fixes — earlier double-lstat-on-path was a TOCTOU race the attacker could win by swapping the regular file for a symlink between the two calls). On Windows ``O_NOFOLLOW`` is absent but the OS does not follow symlinks in `os.open` by default and the containment check is the primary gate. **`load_battery_probes` namespace-package safety** (HIGH H1) — uses `importlib.resources.files("soup_cli") / "data" / "_fixtures" / ...` Traversable `/` operator + `as_file` context manager + `os.lstat + S_ISLNK` rejection + 4 MiB cap; earlier `Path(os.path.join(str(pkg_root), ...))` silently failed `is_file()` on `MultiplexedPath` from namespace-package installs. **Word-boundary regex agreement** (HIGH H4 + MEDIUM M2) — `behavior_battery._agreement_rate` and `checklist_dsl._mft_pass` use `re.search(rf"\b{re.escape(target)}\b", lower)` instead of substring `in` matching, so `"safe"` cannot match `"unsafe"` and `"and"` cannot match `"sand"`. **CLI `_validate_run_id` gate** (MEDIUM M5) — every `soup eval behavior / capability / checklist` command validates `run_id` for null bytes / oversize (>256-char) / empty / non-string BEFORE the value flows into report payloads. **16 MiB evidence file cap** (MEDIUM M6) — `_read_evidence_json` rejects oversize evidence files via `os.fstat` on the O_NOFOLLOW-opened descriptor. **`_MAX_ROWS` counts skipped lines toward the cap** (MEDIUM M-NEW-2) — earlier `irt.load_response_rows` only counted successfully-parsed rows, so a 1M+1 malformed-line file would stream to completion (just logging WARNINGs); now total iterations are bounded. **INV empty-string normalisation rejected** (MEDIUM M-NEW-3) — `checklist_dsl._inv_pass` returns False when all responses normalise to the empty string (was silently passing INV on `["", " "]`). **Atomic file writes** via shared `paths.atomic_write_text` in `_write_json_output` (LOW L6 dedup) + `os.lstat + S_ISLNK` rejection at the target so a pre-placed symlink cannot redirect the write. **Named-test errors** (MEDIUM M4) — `parse_checklist_spec` raises `tests[idx].prompts must be a list/tuple` and `tests[idx].expected must be a list/tuple` instead of generic messages, so operators can grep the index. **Closed-allowlist validators**: `SUPPORTED_BATTERIES = {xstest, harmbench, jailbreakbench, elephant, syceval}` (battery), `CAPABILITY_BENCHMARKS` 7-entry frozenset (capability), `PROFILES = {full, fast, math, code}` (suite), `CHECKLIST_KINDS = {mft, inv, dir}` (DSL), `IRT_PROFILES = {full, small, tiny}` (IRT) — all bool-first / null-byte / oversize (≤32-char) / case-insensitive normalisation; per-list MappingProxyType immutability. **`ensure_judge_calibrated` production gate** — refuses to score with `None` report / `calibrated=False` / agreement below `min_agreement=0.7` / position bias above `max_bias=0.3`, raising `RuntimeError` (not silent). **`PairwiseJudgement` validators** — winner Literal {a, b, tie} + null-byte prompt_id + 256-char cap. **`_MAX_PROBES=100_000` cap** on `compute_behavior_diff`; **`_MAX_PROBES_PER_TEST=10_000`** + **`_MAX_TESTS=1000`** + **`_MAX_PROMPT_LEN=8 KiB`** caps on CheckList; **`_MAX_ROWS=1_000_000`** + **`_MAX_FILE_BYTES=256 MiB`** on IRT. **`_EPSILON=1e-3` clipping** in `fit_difficulty` keeps `info` scores numerically stable at the extremes. **No heavy top-level imports** — torch/transformers/peft never imported at module scope in any v0.65.0 module (source-grep regression tests). **Test count**: 10306 → 10577 (+271 net in `tests/test_v0650_part_{a-e}.py` + `tests/test_v0650_followups.py`). **Review-fix coverage** across 2 review waves (general-purpose consolidated code+security+TDD; python-reviewer agent context-thrashed on the 800+ KB CLAUDE.md release-notes history — matches v0.58.0 / v0.59.0 / v0.60.0 / v0.61.0 / v0.62.0 / v0.63.0 / v0.64.0 idiom): 0 CRITICAL + 6 HIGH (H1 Traversable `/` + as_file + lstat + 4 MiB cap; H2 shared `enforce_under_cwd_and_no_symlink`; H3 streaming via `os.fdopen` + bounded total iteration; H4 word-boundary regex; H-NEW-1 `O_NOFOLLOW` + fstat in checklist + irt; H-NEW-2 `O_NOFOLLOW` + S_ISLNK in `_read_evidence_json`) + 9 MEDIUM (M2 word-boundary in checklist; M4 named-test error; M5 `_validate_run_id` gate; M6 16 MiB evidence cap; M-NEW-2 `_MAX_ROWS` counts skipped; M-NEW-3 INV empty-string rejected; M3 + M7 + M8 documented design choices) + 7 LOW (L1 0.0/0.85/0.60/1.0 boundary tests; L4 + L-NEW-3 skipped-row WARNING + caplog test; L6 `_write_json_output` dedup helper + source-grep guard; L-NEW-1 ASCII word-boundary docstring note; L-NEW-2 divergent validator docstring note). **Known limitations**: (1) Live lm-eval-harness invocation deferred — `soup eval capability` emits the manifest for downstream `soup eval benchmark` chaining. (2) Live model-driven `soup eval behavior` deferred — without `--evidence`, emits a neutral OK report. (3) Behaviour battery probe sets are tiny redacted placeholders — operators pull real sets from upstream papers. (4) IRT model is 1PL Rasch only (2PL / 3PL deferred). (v0.65.0) diff --git a/pyproject.toml b/pyproject.toml index 281b47e..014b07e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.68.0" +version = "0.69.0" description = "Fine-tune LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/soup_cli/__init__.py b/soup_cli/__init__.py index 5518907..983dc31 100644 --- a/soup_cli/__init__.py +++ b/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune LLMs in one command.""" -__version__ = "0.68.0" +__version__ = "0.69.0" diff --git a/soup_cli/cli.py b/soup_cli/cli.py index d4ee733..101fb17 100644 --- a/soup_cli/cli.py +++ b/soup_cli/cli.py @@ -451,6 +451,22 @@ app.add_typer( help="Personal-LLM flywheel daemon (init / status / record / harvest / train) (v0.68.0).", ) +# v0.69.0 Part A — `soup build` (dbt-for-SFT DAG). +from soup_cli.commands import build as _build_cmd # noqa: E402 + +app.command( + name="build", + help="dbt-for-SFT DAG: validate + plan dataset transforms (v0.69.0 Part A).", +)(_build_cmd.build_cmd) + +# v0.69.0 Part B — `soup expect` (expectations suite). +from soup_cli.commands import expect as _expect_cmd # noqa: E402 + +app.command( + name="expect", + help="Run an expectations suite against a JSONL dataset (v0.69.0 Part B).", +)(_expect_cmd.expect_cmd) + def _rewrite_advise_argv(argv: list) -> list: """Inject `run` between `advise` and a non-subcommand first argument. diff --git a/soup_cli/commands/build.py b/soup_cli/commands/build.py new file mode 100644 index 0000000..d2b68c2 --- /dev/null +++ b/soup_cli/commands/build.py @@ -0,0 +1,67 @@ +"""soup build — dbt-for-SFT DAG of dataset transforms (v0.69.0 Part A). + +Reads a YAML manifest, validates the model DAG, prints the topological plan, +and (with ``--dry-run``) exits cleanly. The live runner that actually +materialises each model is deferred to v0.69.1 — invoking ``soup build`` WITHOUT +``--dry-run`` exits with code 3 and a deferred-live advisory (matches v0.61.0 / +v0.62.0 / v0.68.0 distinct-exit-code-for-deferred policy). +""" + +from __future__ import annotations + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel + +console = Console() + + +def build_cmd( + config: str = typer.Argument(..., help="Path to build manifest (YAML)"), + dry_run: bool = typer.Option( + False, + "--dry-run", + help="Validate the manifest + print the topological plan; do not execute.", + ), +) -> None: + """Validate a build DAG and (optionally) execute it. + + The live materialiser is deferred to v0.69.1; today only ``--dry-run`` + produces a meaningful result. + """ + from soup_cli.utils.build_dag import ( + load_build_yaml, + render_plan_table, + run_build, + ) + + try: + plan = load_build_yaml(config) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + rendered = render_plan_table(plan) + console.print( + Panel( + escape(rendered), + title=f"soup build — {escape(config)}", + ) + ) + + if dry_run: + return + + # Live runner deferred to v0.69.1 — same distinct-exit-3 policy as + # v0.61.0 / v0.62.0 / v0.68.0 deferred-live stubs. + try: + run_build(plan) + except NotImplementedError as exc: + console.print( + Panel( + f"[yellow]{escape(str(exc))}[/]", + title="Live build deferred", + ) + ) + raise typer.Exit(3) from exc diff --git a/soup_cli/commands/data.py b/soup_cli/commands/data.py index 4f1fb7f..f5c5582 100644 --- a/soup_cli/commands/data.py +++ b/soup_cli/commands/data.py @@ -2464,3 +2464,295 @@ def recipe( console.print( "[dim]Re-run with --execute --output to run the DAG.[/]" ) + + +# v0.69.0 Part C — Magpie synthetic data generator. +@app.command(name="gen-magpie") +def gen_magpie( + base: str = typer.Option(..., "--base", help="Aligned chat-tuned base model id"), + provider: str = typer.Option(..., "--provider", help="ollama | anthropic | vllm"), + target: int = typer.Option(..., "--target", help="Target row count [1, 1_000_000]"), + quality_filter: bool = typer.Option( + True, "--quality-filter/--no-quality-filter", help="Apply v0.47 quality filter" + ), + plan_only: bool = typer.Option( + False, "--plan-only", help="Validate + print plan; do not generate." + ), +) -> None: + """Magpie synthetic data generator (v0.69.0 Part C). + + Feeds the chat-template prefix only to ``--base`` and harvests user-side + turns via ``--provider``. The live generator is deferred to v0.69.1; today + only ``--plan-only`` produces a meaningful result. + """ + from rich.markup import escape as _escape + from rich.panel import Panel + + from soup_cli.utils.magpie import build_magpie_config, run_magpie + + try: + cfg = build_magpie_config( + base=base, + provider=provider, + target=target, + quality_filter=quality_filter, + ) + except (TypeError, ValueError) as exc: + console.print(f"[red]{_escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + console.print( + Panel( + f"Base model: [bold]{_escape(cfg.base_model)}[/]\n" + f"Provider: [bold]{_escape(cfg.provider)}[/]\n" + f"Target rows: [bold]{cfg.target_rows}[/]\n" + f"Quality filter: [bold]{cfg.quality_filter}[/]", + title="soup data gen-magpie — plan", + ) + ) + + if plan_only: + return + + try: + run_magpie(cfg) + except NotImplementedError as exc: + console.print( + Panel( + f"[yellow]{_escape(str(exc))}[/]", + title="Live magpie generator deferred", + ) + ) + raise typer.Exit(3) from exc + + +# v0.69.0 Part D — Persona-Hub diversity sampler. +@app.command(name="persona-mix") +def persona_mix( + prompts: str = typer.Option(..., "--prompts", help="JSONL file with one prompt per row"), + n: int = typer.Option(..., "--n", help="Number of rows to emit [1, 1_000_000]"), + output: str = typer.Option(..., "--output", help="Output JSONL path"), + personas: str = typer.Option( + "", "--personas", help="Optional JSONL of personas (defaults to bundled set)" + ), + styles: str = typer.Option( + "", "--styles", help="Optional JSONL of styles (defaults to bundled set)" + ), + seed: int = typer.Option(0, "--seed", help="Deterministic seed"), +) -> None: + """Sample a prompt × persona × style matrix (v0.69.0 Part D). + + Reads ``--prompts`` (JSONL with ``prompt`` field per row); writes + ``--output`` JSONL with ``{prompt, persona, style}`` rows. When + ``--personas`` / ``--styles`` are omitted the bundled diversity set is used. + """ + import os + import tempfile + + from rich.markup import escape as _escape + from rich.panel import Panel + + from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + from soup_cli.utils.persona_hub import ( + list_bundled_personas, + list_bundled_styles, + sample_persona_matrix, + ) + + max_jsonl_bytes = 100 * 1024 * 1024 # 100 MiB cap on field inputs + max_values = 100_000 # cap matches persona_hub._MAX_LIST_LEN + + def _load_jsonl_field(path: str, field: str) -> list: + # Delegate to centralised TOCTOU helper (v0.59.0). + enforce_under_cwd_and_no_symlink(path, f"--{field}s path") + if not os.path.lexists(path): + raise FileNotFoundError(path) + real = os.path.realpath(path) + if not os.path.isfile(real): + raise FileNotFoundError(real) + if os.path.getsize(real) > max_jsonl_bytes: + raise ValueError( + f"--{field}s file exceeds {max_jsonl_bytes} bytes" + ) + values: list = [] + with open(real, "r", encoding="utf-8") as handle: + for raw_line in handle: + stripped = raw_line.strip() + if not stripped: + continue + if len(values) >= max_values: + raise ValueError( + f"--{field}s file exceeds {max_values} entries" + ) + try: + row = json.loads(stripped) + except json.JSONDecodeError: + continue + val = row.get(field) if isinstance(row, dict) else None + if isinstance(val, str) and val: + values.append(val) + return values + + try: + prompt_list = _load_jsonl_field(prompts, "prompt") + if not prompt_list: + raise ValueError("prompts file produced no rows with a 'prompt' field") + persona_list = ( + _load_jsonl_field(personas, "persona") + if personas + else list(list_bundled_personas()) + ) + if not persona_list: + raise ValueError("personas file produced no rows with a 'persona' field") + style_list = ( + _load_jsonl_field(styles, "style") + if styles + else list(list_bundled_styles()) + ) + if not style_list: + raise ValueError("styles file produced no rows with a 'style' field") + + # Output containment + TOCTOU symlink rejection at the write target. + enforce_under_cwd_and_no_symlink(output, "--output") + rows = sample_persona_matrix( + prompts=prompt_list, + personas=persona_list, + styles=style_list, + n=n, + seed=seed, + ) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]{_escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + # Atomic write via tempfile + os.replace. + parent = os.path.dirname(os.path.realpath(output)) or "." + fd, tmp_path = tempfile.mkstemp(prefix=".soup.", suffix=".tmp", dir=parent) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, ensure_ascii=False) + "\n") + os.replace(tmp_path, output) + except OSError: + try: + os.unlink(tmp_path) + except OSError: + pass + raise + + console.print( + Panel( + f"Rows written: [bold]{len(rows)}[/]\n" + f"Output: [bold]{_escape(output)}[/]", + title="soup data persona-mix", + ) + ) + + +# v0.69.0 Part E — Brain-rot detector. +@app.command(name="brain-rot") +def brain_rot_cmd( + data: str = typer.Argument(..., help="Path to JSONL dataset"), + strict: bool = typer.Option( + False, "--strict", help="Exit 3 on MAJOR verdict (CI gate mode)" + ), + max_major_fraction: float = typer.Option( + 0.25, "--max-major-fraction", help="Strict-mode MAJOR-row fraction cap [0, 1]" + ), +) -> None: + """Score a dataset for brain-rot per arXiv 2510.13928 (v0.69.0 Part E). + + Reports a per-row OK/MINOR/MAJOR verdict + an aggregate verdict. With + ``--strict`` the command exits 3 when the MAJOR fraction exceeds + ``--max-major-fraction`` (default 25%), so CI pipelines can refuse + training on excessive slop. + """ + import math as _math + import os + + from rich.markup import escape as _escape + from rich.table import Table + + from soup_cli.utils.brain_rot import ( + refuse_if_rotten, + score_dataset_brain_rot, + ) + from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + + max_brain_rot_bytes = 1_073_741_824 # 1 GiB + max_brain_rot_rows = 1_000_000 + + # Validate --max-major-fraction at the CLI boundary so a flag of NaN / + # inf / bool / out-of-range is rejected BEFORE the heavy scoring pass + # (security review M2). + if isinstance(max_major_fraction, bool): + console.print("[red]--max-major-fraction must be a number, not bool[/]") + raise typer.Exit(2) + if not isinstance(max_major_fraction, (int, float)): + console.print("[red]--max-major-fraction must be a number[/]") + raise typer.Exit(2) + if not _math.isfinite(float(max_major_fraction)): + console.print("[red]--max-major-fraction must be finite[/]") + raise typer.Exit(2) + if not (0.0 <= float(max_major_fraction) <= 1.0): + console.print("[red]--max-major-fraction must be in [0.0, 1.0][/]") + raise typer.Exit(2) + + if not isinstance(data, str) or not data: + console.print("[red]data path must be a non-empty string[/]") + raise typer.Exit(2) + try: + enforce_under_cwd_and_no_symlink(data, "data path") + except (TypeError, ValueError) as exc: + console.print(f"[red]{_escape(str(exc))}[/]") + raise typer.Exit(2) from exc + if not os.path.lexists(data): + console.print(f"[red]data file not found: {_escape(data)}[/]") + raise typer.Exit(2) + real = os.path.realpath(data) + if not os.path.isfile(real): + console.print(f"[red]data file not found: {_escape(real)}[/]") + raise typer.Exit(2) + if os.path.getsize(real) > max_brain_rot_bytes: + console.print( + f"[red]data file exceeds {max_brain_rot_bytes} bytes[/]" + ) + raise typer.Exit(2) + + rows: list = [] + with open(real, "r", encoding="utf-8") as handle: + for line_no, raw_line in enumerate(handle, start=1): + stripped = raw_line.strip() + if not stripped: + continue + if line_no > max_brain_rot_rows: + console.print( + f"[red]data file exceeds {max_brain_rot_rows} rows[/]" + ) + raise typer.Exit(2) + try: + row = json.loads(stripped) + except json.JSONDecodeError: + continue + if isinstance(row, dict): + rows.append(row) + + report = score_dataset_brain_rot(rows) + + table = Table(title="Brain-rot report") + table.add_column("Metric") + table.add_column("Value") + table.add_row("Rows scored", str(report.num_rows)) + table.add_row("Mean score", f"{report.mean_score:.3f}") + table.add_row("Verdict OK", str(report.num_ok)) + table.add_row("Verdict MINOR", str(report.num_minor)) + table.add_row("Verdict MAJOR", str(report.num_major)) + table.add_row("Overall", report.overall_verdict) + console.print(table) + + if strict: + try: + refuse_if_rotten(rows, max_major_fraction=max_major_fraction) + except (TypeError, ValueError) as exc: + console.print(f"[red]{_escape(str(exc))}[/]") + raise typer.Exit(3) from exc diff --git a/soup_cli/commands/expect.py b/soup_cli/commands/expect.py new file mode 100644 index 0000000..3728173 --- /dev/null +++ b/soup_cli/commands/expect.py @@ -0,0 +1,118 @@ +"""soup expect — Expectations suite for chat data (v0.69.0 Part B). + +Runs a YAML suite of expectations against a JSONL dataset. Exit code 3 on +suite failure (matches v0.55 / v0.56 / v0.64 / v0.65 gate convention) so CI +pipelines can gate on dataset quality regressions without parsing output. +""" + +from __future__ import annotations + +import json +import os +from typing import List, Mapping + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel +from rich.table import Table + +console = Console() + +_MAX_DATA_BYTES = 1_073_741_824 # 1 GiB cap on input data +_MAX_DATA_ROWS = 1_000_000 + + +def _load_jsonl_rows(data_path: str) -> List[Mapping[str, object]]: + """Cwd-contained + symlink-rejected JSONL loader. + + Delegates to ``utils.paths.enforce_under_cwd_and_no_symlink`` (v0.59.0 + centralised TOCTOU helper) for the path-validation pre-check. + """ + from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + + if not isinstance(data_path, str) or not data_path: + raise ValueError("data path must be a non-empty string") + if not os.path.lexists(data_path): + raise FileNotFoundError(data_path) + enforce_under_cwd_and_no_symlink(data_path, "data path") + real = os.path.realpath(data_path) + if not os.path.isfile(real): + raise FileNotFoundError(real) + if os.path.getsize(real) > _MAX_DATA_BYTES: + raise ValueError(f"data file exceeds {_MAX_DATA_BYTES} bytes") + rows: List[Mapping[str, object]] = [] + skipped = 0 + with open(real, "r", encoding="utf-8") as handle: + for line_no, line in enumerate(handle, start=1): + if not line.strip(): + continue + if line_no > _MAX_DATA_ROWS: + raise ValueError(f"data file exceeds {_MAX_DATA_ROWS} rows") + try: + row = json.loads(line) + except json.JSONDecodeError: + skipped += 1 + continue + if isinstance(row, dict): + rows.append(row) + if skipped: + console.print( + f"[yellow]Note: skipped {skipped} malformed JSONL line(s)[/]" + ) + return rows + + +def expect_cmd( + data: str = typer.Argument(..., help="Path to JSONL dataset"), + suite: str = typer.Argument(..., help="Path to expectations suite YAML"), +) -> None: + """Run an expectations suite against a JSONL dataset. + + Exit 0 = suite passed. Exit 2 = validation rejection. Exit 3 = suite failed. + """ + from soup_cli.utils.expectations import load_suite_yaml, run_suite + + try: + spec = load_suite_yaml(suite) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + rows = _load_jsonl_rows(data) + except (FileNotFoundError, TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + try: + report = run_suite(rows, spec) + except (TypeError, ValueError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(2) from exc + + table = Table(title=f"soup expect — {escape(data)}") + table.add_column("Expectation") + table.add_column("Passed") + table.add_column("Rows") + table.add_column("Violations") + for result in report.results: + verdict = "[green]PASS[/]" if result.passed else "[red]FAIL[/]" + table.add_row( + escape(result.name), + verdict, + str(result.num_rows_checked), + str(result.num_violations), + ) + console.print(table) + + if not report.passed: + for result in report.results: + if not result.passed and result.details: + console.print( + Panel( + "\n".join(escape(d) for d in result.details), + title=f"[red]Violations: {escape(result.name)}[/]", + ) + ) + raise typer.Exit(3) diff --git a/soup_cli/utils/brain_rot.py b/soup_cli/utils/brain_rot.py new file mode 100644 index 0000000..ba25b34 --- /dev/null +++ b/soup_cli/utils/brain_rot.py @@ -0,0 +1,297 @@ +"""v0.69.0 Part E — Brain-rot detector (arXiv 2510.13928). + +Scores dataset rows on two orthogonal "low-quality slop" axes: + +- **Triviality** — short / repetitive / exclamation-heavy text dominates → high. +- **Popularity signal** — clickbait phrases ("you won't believe", "top 10") + + excessive punctuation / emoji density → high. + +Per-row brain-rot score is in [0, 1] where ``1.0`` = healthy substantive content +and ``0.0`` = pure slop (so we can reuse the OK/MINOR/MAJOR taxonomy from +v0.26 / v0.56 / v0.65 with the same threshold band — high score = good). + +Composes with v0.47.0 ``score_educational_value`` (educational-value scorer), +which the live runner can mix in for a composite score; this module ships the +slop-detection kernel that the paper measures. +""" + +from __future__ import annotations + +import math +import re +from collections import Counter +from dataclasses import dataclass +from typing import Any, Iterable, List, Mapping + +BRAIN_ROT_VERDICTS = ("OK", "MINOR", "MAJOR") + +_OK_THRESHOLD = 0.85 +_MINOR_THRESHOLD = 0.60 + +_MAX_TEXT_LEN = 65_536 +_CLICKBAIT_PHRASES = ( + "you won't believe", + "you wont believe", + "won't believe what happened", + "top 10", + "top ten", + "click here", + "this one weird trick", + "what happened next", + "the rest is history", + "shocked the world", + "doctors hate", + "gone wrong", + "gone viral", +) +_LOW_EFFORT_TOKENS = ("lol", "omg", "lmao", "rofl", "smh", "tbh", "idk") +_PUNCT_PATTERN = re.compile(r"[!]{2,}|[?]{2,}") +_TEXT_FIELDS = ("text", "content", "output", "prompt", "instruction", "response") + + +@dataclass(frozen=True) +class BrainRotReport: + """Outcome of scoring a dataset for brain-rot.""" + + num_rows: int + mean_score: float + num_major: int + num_minor: int + num_ok: int + overall_verdict: str + + def __post_init__(self) -> None: + for field_name in ("num_rows", "num_major", "num_minor", "num_ok"): + value = getattr(self, field_name) + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError( + f"BrainRotReport.{field_name} must be int" + ) + if value < 0: + raise ValueError( + f"BrainRotReport.{field_name} must be non-negative" + ) + if isinstance(self.mean_score, bool): + raise TypeError("BrainRotReport.mean_score must be float") + if not isinstance(self.mean_score, (int, float)): + raise TypeError("BrainRotReport.mean_score must be a number") + if not math.isfinite(float(self.mean_score)): + raise ValueError("BrainRotReport.mean_score must be finite") + if not (0.0 <= float(self.mean_score) <= 1.0): + raise ValueError( + "BrainRotReport.mean_score must be in [0.0, 1.0]" + ) + if self.overall_verdict not in BRAIN_ROT_VERDICTS: + raise ValueError( + f"overall_verdict must be one of {BRAIN_ROT_VERDICTS}" + ) + + +# ----------------------------------------------------------------------------- +# Classifier +# ----------------------------------------------------------------------------- + + +def classify_brain_rot(score: object) -> str: + """Classify a score in [0, 1] (1.0 = healthy) into OK / MINOR / MAJOR. + + Mirrors v0.26.0 Quant-Lobotomy / v0.56.0 diagnose / v0.65.0 behavior taxonomy: + ``>= 0.85 → OK``, ``>= 0.60 → MINOR``, else ``MAJOR``. + """ + if isinstance(score, bool): + raise TypeError("score must be float, not bool") + if not isinstance(score, (int, float)): + raise TypeError( + f"score must be a number, got {type(score).__name__}" + ) + fscore = float(score) + if not math.isfinite(fscore): + raise ValueError("score must be finite") + if not (0.0 <= fscore <= 1.0): + raise ValueError("score must be in [0.0, 1.0]") + if fscore >= _OK_THRESHOLD: + return "OK" + if fscore >= _MINOR_THRESHOLD: + return "MINOR" + return "MAJOR" + + +# ----------------------------------------------------------------------------- +# Heuristic scorers +# ----------------------------------------------------------------------------- + + +def _require_str(text: object, *, field: str = "text") -> str: + if isinstance(text, bool): + raise TypeError(f"{field} must be str, not bool") + if not isinstance(text, str): + raise TypeError( + f"{field} must be str, got {type(text).__name__}" + ) + if len(text) > _MAX_TEXT_LEN: + return text[:_MAX_TEXT_LEN] + return text + + +def score_triviality(text: object) -> float: + """Higher = more trivial / repetitive / exclamation-heavy. + + Heuristic: punctuation-runs density + short-text penalty + token diversity + inversion. Returns 1.0 for empty/unparseable input (worst case). + """ + s = _require_str(text) + if not s.strip(): + return 1.0 + tokens = s.lower().split() + n = len(tokens) + if n == 0: + return 1.0 + unique = len(set(tokens)) + diversity = unique / n # 1.0 = all unique, 0.0 = pathological + # Length penalty: very short text is suspect for SFT. + length_penalty = 1.0 if n >= 30 else (1.0 - n / 30.0) + # Punctuation: long !!!! / ???? runs are slop markers. + punct_hits = len(_PUNCT_PATTERN.findall(s)) + punct_density = min(1.0, punct_hits / max(1, n / 10)) + # Low-effort token density. + low_effort = sum(1 for tok in tokens if tok.strip("!?.,") in _LOW_EFFORT_TOKENS) + low_effort_density = min(1.0, low_effort / max(1, n / 5)) + triviality = ( + 0.2 * (1.0 - diversity) + + 0.1 * length_penalty + + 0.3 * punct_density + + 0.4 * low_effort_density + ) + return max(0.0, min(1.0, triviality)) + + +def score_popularity_signal(text: object) -> float: + """Higher = clickbait / engagement-bait / popularity-optimised slop. + + Heuristic: substring scan against ``_CLICKBAIT_PHRASES`` + emoji density. + Returns 0.0 for empty input. + """ + s = _require_str(text) + if not s.strip(): + return 0.0 + lower = s.lower() + hits = sum(1 for phrase in _CLICKBAIT_PHRASES if phrase in lower) + # Emoji density: count non-ASCII chars in [U+1F300, U+1FAFF] range + # (covers most pictographs without importing emoji libs). + emoji_hits = sum(1 for c in s if 0x1F300 <= ord(c) <= 0x1FAFF) + n_tokens = len(s.split()) or 1 + phrase_density = min(1.0, hits / 2.0) # 2+ clickbait phrases → max + emoji_density = min(1.0, emoji_hits / max(1, n_tokens / 5)) + return max(0.0, min(1.0, 0.7 * phrase_density + 0.3 * emoji_density)) + + +def _row_text(row: Mapping[str, Any]) -> str: + parts: List[str] = [] + for key in _TEXT_FIELDS: + val = row.get(key) + if isinstance(val, str) and val: + parts.append(val) + messages = row.get("messages") + if isinstance(messages, list): + for msg in messages: + if isinstance(msg, Mapping): + content = msg.get("content") + if isinstance(content, str) and content: + parts.append(content) + return "\n".join(parts) + + +def score_row_brain_rot(row: Any) -> float: + """Return a per-row score in [0, 1]; 1.0 = healthy, 0.0 = pure slop. + + Composite: ``1 - 0.5 * triviality - 0.5 * popularity_signal``. Rows with + no text fields return ``0.0`` (treat unjudgeable as worst-case). + """ + if not isinstance(row, Mapping): + raise TypeError( + f"row must be a Mapping, got {type(row).__name__}" + ) + text = _row_text(row) + if not text: + return 0.0 + triviality = score_triviality(text) + popularity = score_popularity_signal(text) + # Worst-signal composite: a single strong slop signal drives the score + # down hard (mirrors v0.56.0 ``overall_verdict`` worst-case policy). + score = 1.0 - max(triviality, popularity) + return max(0.0, min(1.0, score)) + + +def score_dataset_brain_rot(rows: Any) -> BrainRotReport: + """Score a dataset and return a frozen ``BrainRotReport``. + + Empty inputs return ``MAJOR`` (no signal = treat as broken). + """ + if isinstance(rows, (str, bytes)) or not hasattr(rows, "__iter__"): + raise TypeError("rows must be iterable") + scores: List[float] = [] + for row in rows: + if not isinstance(row, Mapping): + continue + scores.append(score_row_brain_rot(row)) + if not scores: + return BrainRotReport( + num_rows=0, + mean_score=0.0, + num_major=0, + num_minor=0, + num_ok=0, + overall_verdict="MAJOR", + ) + verdict_counts: Counter = Counter(classify_brain_rot(s) for s in scores) + mean = sum(scores) / len(scores) + overall = classify_brain_rot(mean) + return BrainRotReport( + num_rows=len(scores), + mean_score=mean, + num_major=verdict_counts.get("MAJOR", 0), + num_minor=verdict_counts.get("MINOR", 0), + num_ok=verdict_counts.get("OK", 0), + overall_verdict=overall, + ) + + +def refuse_if_rotten( + rows: Iterable[Mapping[str, Any]], + *, + max_major_fraction: float = 0.25, +) -> None: + """Raise ``ValueError`` when too many rows score MAJOR brain-rot. + + Composes with v0.69.0 Part A's build pipeline so a transform can refuse to + produce a tokenised dataset that's mostly slop. + """ + if isinstance(max_major_fraction, bool): + raise TypeError("max_major_fraction must be float, not bool") + if not isinstance(max_major_fraction, (int, float)): + raise TypeError("max_major_fraction must be a number") + if not math.isfinite(float(max_major_fraction)): + raise ValueError("max_major_fraction must be finite") + if not (0.0 <= float(max_major_fraction) <= 1.0): + raise ValueError("max_major_fraction must be in [0.0, 1.0]") + report = score_dataset_brain_rot(rows) + if report.num_rows == 0: + return # no data → nothing to refuse + fraction = report.num_major / report.num_rows + if fraction > float(max_major_fraction): + raise ValueError( + f"brain-rot threshold exceeded: {fraction:.1%} of rows are MAJOR " + f"(limit {float(max_major_fraction):.1%})" + ) + + +__all__ = [ + "BRAIN_ROT_VERDICTS", + "BrainRotReport", + "classify_brain_rot", + "refuse_if_rotten", + "score_dataset_brain_rot", + "score_popularity_signal", + "score_row_brain_rot", + "score_triviality", +] diff --git a/soup_cli/utils/build_dag.py b/soup_cli/utils/build_dag.py new file mode 100644 index 0000000..f1dcd61 --- /dev/null +++ b/soup_cli/utils/build_dag.py @@ -0,0 +1,475 @@ +"""v0.69.0 Part A — `soup build` (dbt-for-SFT DAG). + +Parses a YAML manifest describing a DAG of dataset transforms (``ref``-connected +models with ``incremental`` / ``table`` / ``view`` materialization). Validates +topology via Kahn's algorithm and renders the dry-run plan. The live SQL/Python +runner (re-tokenize-only-diff rows materialization) is deferred to v0.69.1. + +Mirrors the shape of v0.45.0 Part E `recipe_dag.py` so operators familiar with +the recipe DAG surface have one mental model. Differs in that build-DAG models +also carry a *transform* identifier and a *source* path, and the DAG edges are +derived from each model's ``refs: []`` field (rather than a +top-level ``edges`` list) — matching dbt's mental model. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +from collections import deque +from dataclasses import dataclass +from types import MappingProxyType +from typing import Any, Iterable, List, Mapping, Optional, Sequence, Tuple + +# Closed allowlist of model kinds. Mirrors dbt's `materialized` field but +# trimmed to the three that map onto SFT-data pipelines. +_SUPPORTED_MODEL_KINDS = ("incremental", "table", "view") +SUPPORTED_MODEL_KINDS: frozenset = frozenset(_SUPPORTED_MODEL_KINDS) + +# Per-build-plan caps — defence-in-depth against pathological YAML. +_MAX_MODELS = 256 +_MAX_REFS_PER_MODEL = 32 +_MAX_NAME_LEN = 128 +_MAX_KIND_LEN = 64 +_MAX_TRANSFORM_LEN = 256 +_MAX_SOURCE_LEN = 4096 +_MAX_FILE_BYTES = 1_048_576 # 1 MiB + +_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._\-]{0,127}$") + + +@dataclass(frozen=True) +class BuildModel: + """One node in the build DAG. + + ``transform`` identifies the materialization function (e.g. ``identity``, + ``filter_low_quality``, ``tokenize``) the live runner will resolve. The + schema does not validate the function reference — the runner does. + + Field semantics: + + - ``refs=()`` + ``source`` set → *seed* model (reads from disk). + - ``refs=(...)`` + ``source=None`` → *derived* model (consumes upstream models). + - ``refs=()`` + ``source=None`` → degenerate; rejected by cross-validator. + - ``refs=(...)`` + ``source`` set → ambiguous; rejected by cross-validator. + + The ``source`` path is validated for shape (non-empty, null-byte-free, + length-capped) at schema-load. Operators wanting cwd containment on + source paths get it via the v0.69.1 live runner — the schema permits + relative paths so a build can be planned offline before the data lands. + """ + + name: str + kind: str + transform: str + refs: Tuple[str, ...] + source: Optional[str] + config: Mapping[str, Any] + + def __post_init__(self) -> None: + # Re-validate so callers bypassing ``parse_build_plan`` cannot smuggle + # an inconsistent BuildModel through direct construction. + validate_model_name(self.name) + validate_model_kind(self.kind) + if not isinstance(self.refs, tuple): + raise TypeError("BuildModel.refs must be a tuple, not list/sequence") + if isinstance(self.transform, bool) or not isinstance(self.transform, str): + raise TypeError("BuildModel.transform must be a string") + if "\x00" in self.transform: + raise ValueError("BuildModel.transform must not contain null bytes") + if len(self.transform) > _MAX_TRANSFORM_LEN: + raise ValueError( + f"BuildModel.transform must be <= {_MAX_TRANSFORM_LEN} chars" + ) + if self.source is not None: + if isinstance(self.source, bool) or not isinstance(self.source, str): + raise TypeError("BuildModel.source must be a string or None") + if "\x00" in self.source: + raise ValueError("BuildModel.source must not contain null bytes") + if len(self.source) > _MAX_SOURCE_LEN: + raise ValueError( + f"BuildModel.source must be <= {_MAX_SOURCE_LEN} chars" + ) + # Cross-validator: seed/derived shape must be unambiguous. + if not self.refs and self.source is None: + raise ValueError( + f"BuildModel {self.name!r}: a model with no refs must declare a " + "'source' (or add refs to make it derived)" + ) + if self.refs and self.source is not None: + raise ValueError( + f"BuildModel {self.name!r}: a model with refs cannot also declare " + "'source' (refs and source are mutually exclusive)" + ) + + +@dataclass(frozen=True) +class BuildPlan: + """Validated build DAG with topologically sorted models.""" + + models: Tuple[BuildModel, ...] + topo_order: Tuple[str, ...] + + +@dataclass(frozen=True) +class IncrementalDiffReport: + """Outcome of comparing previous + new rows for an incremental model. + + ``added``: rows present in ``new`` but not ``prev``. + ``changed``: rows whose ``id`` exists in both but whose content hash differs. + ``removed``: rows present in ``prev`` but absent in ``new``. + ``unchanged``: rows with identical ``id`` and content hash in both. + """ + + added: int + changed: int + removed: int + unchanged: int + + +# ----------------------------------------------------------------------------- +# Validators +# ----------------------------------------------------------------------------- + + +def validate_model_kind(kind: object) -> str: + """Return the canonical lower-case kind. Raise ValueError on unknown.""" + if isinstance(kind, bool) or not isinstance(kind, str): + raise TypeError( + f"model kind must be str, got {type(kind).__name__}" + ) + if not kind: + raise ValueError("model kind must be non-empty") + if "\x00" in kind: + raise ValueError("model kind must not contain null bytes") + if len(kind) > _MAX_KIND_LEN: + raise ValueError(f"model kind must be <= {_MAX_KIND_LEN} chars") + canonical = kind.strip().lower() + if canonical not in SUPPORTED_MODEL_KINDS: + raise ValueError( + f"unknown model kind: {kind!r}. supported: {sorted(SUPPORTED_MODEL_KINDS)}" + ) + return canonical + + +def validate_model_name(name: object) -> str: + """Validate the model identifier. Returns the canonical name.""" + if isinstance(name, bool) or not isinstance(name, str): + raise TypeError( + f"model name must be str, got {type(name).__name__}" + ) + if not name: + raise ValueError("model name must be non-empty") + if "\x00" in name: + raise ValueError("model name must not contain null bytes") + if len(name) > _MAX_NAME_LEN: + raise ValueError(f"model name must be <= {_MAX_NAME_LEN} chars") + if not _NAME_RE.match(name): + raise ValueError( + f"model name must match {_NAME_RE.pattern}: {name!r}" + ) + return name + + +# ----------------------------------------------------------------------------- +# Topological sort (Kahn's) +# ----------------------------------------------------------------------------- + + +def _topological_sort( + names: Sequence[str], + edges: Sequence[Tuple[str, str]], +) -> List[str]: + """Kahn's algorithm. Same shape as v0.45.0 Part E ``recipe_dag``.""" + in_degree = {name: 0 for name in names} + successors: dict = {name: [] for name in names} + for source, target in edges: + in_degree[target] += 1 + successors[source].append(target) + queue: deque = deque( + sorted(name for name, deg in in_degree.items() if deg == 0) + ) + order: List[str] = [] + while queue: + current = queue.popleft() + order.append(current) + ready: List[str] = [] + for successor in successors[current]: + in_degree[successor] -= 1 + if in_degree[successor] == 0: + ready.append(successor) + ready.sort() + queue.extend(ready) + if len(order) != len(names): + raise ValueError("build DAG contains a cycle") + return order + + +# ----------------------------------------------------------------------------- +# Plan parsing +# ----------------------------------------------------------------------------- + + +def parse_build_plan(raw: Any) -> BuildPlan: + """Validate a ``{"models": [...]}`` dict and return a sorted ``BuildPlan``. + + Each model is a dict with required ``name`` / ``kind`` / ``transform``, + optional ``refs`` (list of upstream model names), optional ``source`` + (input file path for seed models), and optional ``config`` (free-form dict). + """ + if not isinstance(raw, dict): + raise TypeError("build plan must be a dict") + raw_models = raw.get("models") + if raw_models is None: + raise ValueError("build plan must define a 'models' key") + if not isinstance(raw_models, list): + raise ValueError("build plan 'models' must be a list") + if not raw_models: + raise ValueError("build plan 'models' must be a non-empty list") + if len(raw_models) > _MAX_MODELS: + raise ValueError( + f"build plan 'models' exceeds {_MAX_MODELS} entries" + ) + + models: List[BuildModel] = [] + seen_names: set = set() + edges: List[Tuple[str, str]] = [] + name_set: set = set() + + # First pass: validate every model + collect names. + for index, raw_model in enumerate(raw_models): + if not isinstance(raw_model, dict): + raise TypeError(f"build plan models[{index}] must be a dict") + name = validate_model_name(raw_model.get("name", "")) + if name in seen_names: + raise ValueError(f"duplicate model name: {name!r}") + seen_names.add(name) + kind = validate_model_kind(raw_model.get("kind", "")) + transform = raw_model.get("transform") + if not isinstance(transform, str) or isinstance(transform, bool): + raise TypeError( + f"models[{index}].transform must be a string" + ) + raw_refs = raw_model.get("refs", []) + if not isinstance(raw_refs, list): + raise TypeError(f"models[{index}].refs must be a list") + if len(raw_refs) > _MAX_REFS_PER_MODEL: + raise ValueError( + f"models[{index}].refs exceeds {_MAX_REFS_PER_MODEL} entries" + ) + refs_seen: set = set() + normalised_refs: List[str] = [] + for ref_index, ref in enumerate(raw_refs): + ref_name = validate_model_name(ref) + if ref_name in refs_seen: + raise ValueError( + f"duplicate ref in models[{index}].refs: {ref_name!r}" + ) + refs_seen.add(ref_name) + normalised_refs.append(ref_name) + source = raw_model.get("source") + config = raw_model.get("config", {}) + if not isinstance(config, dict): + raise TypeError(f"models[{index}].config must be a dict") + models.append( + BuildModel( + name=name, + kind=kind, + transform=transform, + refs=tuple(normalised_refs), + source=source, + config=MappingProxyType(dict(config)), + ) + ) + name_set.add(name) + + # Second pass: validate refs reference real models + build edges. + for model in models: + for ref in model.refs: + if ref == model.name: + raise ValueError( + f"self-loop edge rejected: {model.name!r}" + ) + if ref not in name_set: + raise ValueError( + f"model {model.name!r} refs missing model: {ref!r}" + ) + # Edge direction: upstream -> downstream (ref produces model) + edges.append((ref, model.name)) + + topo = _topological_sort([m.name for m in models], edges) + return BuildPlan(models=tuple(models), topo_order=tuple(topo)) + + +def parse_build_yaml(text: object) -> BuildPlan: + """Parse a YAML string into a validated ``BuildPlan``.""" + if not isinstance(text, str): + raise TypeError("build text must be a string") + if "\x00" in text: + raise ValueError("build text must not contain null bytes") + if len(text.encode("utf-8")) > _MAX_FILE_BYTES: + raise ValueError(f"build text exceeds {_MAX_FILE_BYTES} bytes") + import yaml # lazy — keep CLI startup fast + + try: + data = yaml.safe_load(text) + except yaml.YAMLError as exc: + raise ValueError(f"invalid YAML: {exc}") from exc + return parse_build_plan(data) + + +def load_build_yaml(path: object) -> BuildPlan: + """Load a build manifest from a path that must live under cwd. + + Delegates to ``utils.paths.enforce_under_cwd_and_no_symlink`` (v0.59.0 + shared TOCTOU helper) so this module does not reinvent the policy + (project code-review CRIT — centralisation enforcement). + """ + from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + + if isinstance(path, bool) or not isinstance(path, str): + raise TypeError("path must be a string") + # Shared helper validates emptiness + null-bytes + cwd + symlink upfront. + enforce_under_cwd_and_no_symlink(path, "build path") + if not os.path.lexists(path): + raise FileNotFoundError(path) + real = os.path.realpath(path) + if not os.path.isfile(real): + raise FileNotFoundError(real) + if os.path.getsize(real) > _MAX_FILE_BYTES: + raise ValueError(f"build file exceeds {_MAX_FILE_BYTES} bytes") + with open(real, "r", encoding="utf-8") as handle: + return parse_build_yaml(handle.read()) + + +# ----------------------------------------------------------------------------- +# Incremental diff — re-tokenize only changed rows +# ----------------------------------------------------------------------------- + + +def compute_row_hash(row: Mapping[str, Any]) -> str: + """Return a stable SHA-256 hex digest of a row's content. + + Keys are sorted so the hash is insertion-order independent. The hash + EXCLUDES the ``id`` field so identity-vs-content can be reasoned about + separately (same id + changed content = diff row). + """ + if not isinstance(row, Mapping): + raise TypeError( + f"row must be a Mapping, got {type(row).__name__}" + ) + payload = {k: row[k] for k in sorted(row.keys()) if k != "id"} + canonical = json.dumps(payload, sort_keys=True, ensure_ascii=False) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def incremental_diff( + prev: Sequence[Mapping[str, Any]], + new: Sequence[Mapping[str, Any]], +) -> IncrementalDiffReport: + """Compare two row sequences keyed on ``id`` and return per-bucket counts. + + Every row in both sequences MUST carry an ``id`` field — the function + raises ``ValueError`` otherwise so silent mis-counting cannot happen. + """ + prev_map: dict = {} + for index, row in enumerate(prev): + if not isinstance(row, Mapping): + raise TypeError(f"prev[{index}] must be a Mapping") + if "id" not in row: + raise ValueError( + f"prev[{index}] missing required 'id' field for incremental diff" + ) + prev_map[row["id"]] = compute_row_hash(row) + + new_map: dict = {} + for index, row in enumerate(new): + if not isinstance(row, Mapping): + raise TypeError(f"new[{index}] must be a Mapping") + if "id" not in row: + raise ValueError( + f"new[{index}] missing required 'id' field for incremental diff" + ) + new_map[row["id"]] = compute_row_hash(row) + + added = 0 + changed = 0 + unchanged = 0 + for row_id, new_hash in new_map.items(): + if row_id not in prev_map: + added += 1 + elif prev_map[row_id] != new_hash: + changed += 1 + else: + unchanged += 1 + removed = sum(1 for row_id in prev_map if row_id not in new_map) + return IncrementalDiffReport( + added=added, + changed=changed, + removed=removed, + unchanged=unchanged, + ) + + +# ----------------------------------------------------------------------------- +# Plan rendering +# ----------------------------------------------------------------------------- + + +def render_plan_table(plan: BuildPlan) -> str: + """Render a plan as plain text in topological order (for dry-run output).""" + if not isinstance(plan, BuildPlan): + raise TypeError("plan must be a BuildPlan") + lookup = {m.name: m for m in plan.models} + lines = ["Build plan (topological order):"] + for name in plan.topo_order: + model = lookup[name] + ref_str = ", ".join(model.refs) if model.refs else "(no refs)" + lines.append( + f" {model.name} [{model.kind}] transform={model.transform} refs=[{ref_str}]" + ) + return "\n".join(lines) + + +# ----------------------------------------------------------------------------- +# Live runner — deferred to v0.69.1 +# ----------------------------------------------------------------------------- + + +def run_build(plan: BuildPlan, *, output_dir: Optional[str] = None) -> None: + """Execute the build DAG. Deferred to v0.69.1. + + Same stub-then-live cadence as v0.45.0 recipe DAG / v0.50.0 GRPO Plus / + v0.61.0 unlearning. The CLI prints the resolved plan + a deferred-live + advisory and exits with code 3 so CI gates can distinguish "deferred / + not yet shipped" from "validation rejection" (which exits 2). + """ + if not isinstance(plan, BuildPlan): + raise TypeError("plan must be a BuildPlan") + raise NotImplementedError( + "soup build live runner is deferred to v0.69.1 — only --dry-run is wired today." + ) + + +__all__ = [ + "BuildModel", + "BuildPlan", + "IncrementalDiffReport", + "SUPPORTED_MODEL_KINDS", + "compute_row_hash", + "incremental_diff", + "load_build_yaml", + "parse_build_plan", + "parse_build_yaml", + "render_plan_table", + "run_build", + "validate_model_kind", + "validate_model_name", +] + + +def _selfcheck() -> Iterable[str]: + """Internal: list of expected public symbols.""" + return tuple(__all__) diff --git a/soup_cli/utils/expectations.py b/soup_cli/utils/expectations.py new file mode 100644 index 0000000..03a31e7 --- /dev/null +++ b/soup_cli/utils/expectations.py @@ -0,0 +1,553 @@ +"""v0.69.0 Part B — Expectations suite for chat data. + +Great-Expectations-flavoured assertions over JSONL rows. Each expectation is a +pure function returning a frozen ``ExpectationResult``; the suite runner +composes them and returns a ``SuiteReport`` whose ``passed`` flag is the +AND of every contained expectation. CI gates use exit code 3 on failure +(matches v0.55 / v0.56 / v0.64 / v0.65 gate convention). + +Composes with: + - v0.47.0 ``data_score.detect_pii`` (PII regex / Presidio backend) + - v0.56.0 ``diagnose.refusal.looks_like_refusal`` (English refusal phrases) + - v0.19.0 judge backends (operator-injected callable for judge expectations) +""" + +from __future__ import annotations + +import math +import os +from dataclasses import dataclass +from typing import Any, Callable, List, Mapping, Optional, Sequence, Tuple + +# Closed allowlist — the 4 expectation kinds the v0.69.0 plan calls out. +_SUPPORTED_EXPECTATIONS = ( + "expect_no_pii", + "expect_token_length_between", + "expect_no_refusal_pattern", + "expect_chosen_preferred_over_rejected_by_judge", +) +SUPPORTED_EXPECTATIONS: frozenset = frozenset(_SUPPORTED_EXPECTATIONS) + +# DoS caps + bounds for validators. +_MAX_NAME_LEN = 128 +_MAX_DETAILS_PER_RESULT = 32 +_MAX_DETAIL_LEN = 256 +_MAX_SUITE_LEN = 64 +_MAX_FILE_BYTES = 1_048_576 # 1 MiB +_MIN_TOKEN_BOUND = 1 +_MAX_TOKEN_BOUND = 1_048_576 + +# JudgeFn signature: row mapping in, [0,1] score out (1.0 = chosen wins). +JudgeFn = Callable[[Mapping[str, Any]], float] + + +@dataclass(frozen=True) +class ExpectationResult: + """Outcome of one expectation.""" + + name: str + passed: bool + num_rows_checked: int + num_violations: int + details: Tuple[str, ...] + + def __post_init__(self) -> None: + validate_expectation_name(self.name) + if not isinstance(self.passed, bool): + raise TypeError("ExpectationResult.passed must be bool") + for field_name, val in ( + ("num_rows_checked", self.num_rows_checked), + ("num_violations", self.num_violations), + ): + if isinstance(val, bool) or not isinstance(val, int): + raise TypeError(f"ExpectationResult.{field_name} must be int") + if val < 0: + raise ValueError( + f"ExpectationResult.{field_name} must be non-negative" + ) + if not isinstance(self.details, tuple): + raise TypeError( + "ExpectationResult.details must be a tuple (frozen=True does " + "not make List immutable)" + ) + + +@dataclass(frozen=True) +class ExpectationSpec: + """One row of a suite YAML.""" + + name: str + args: Mapping[str, Any] + + +@dataclass(frozen=True) +class SuiteSpec: + """Validated expectations suite.""" + + expectations: Tuple[ExpectationSpec, ...] + + +@dataclass(frozen=True) +class SuiteReport: + """Outcome of a full suite run.""" + + passed: bool + results: Tuple[ExpectationResult, ...] + + +# ----------------------------------------------------------------------------- +# Validators +# ----------------------------------------------------------------------------- + + +def validate_expectation_name(name: object) -> str: + """Return the canonical lower-case expectation name.""" + if isinstance(name, bool) or not isinstance(name, str): + raise TypeError( + f"expectation name must be str, got {type(name).__name__}" + ) + if not name: + raise ValueError("expectation name must be non-empty") + if "\x00" in name: + raise ValueError("expectation name must not contain null bytes") + if len(name) > _MAX_NAME_LEN: + raise ValueError( + f"expectation name must be <= {_MAX_NAME_LEN} chars" + ) + canonical = name.strip().lower() + if canonical not in SUPPORTED_EXPECTATIONS: + raise ValueError( + f"unknown expectation: {name!r}. " + f"supported: {sorted(SUPPORTED_EXPECTATIONS)}" + ) + return canonical + + +def _check_threshold(value: object, *, field: str) -> float: + if isinstance(value, bool): + raise TypeError(f"{field} must be float, not bool") + if not isinstance(value, (int, float)): + raise TypeError(f"{field} must be a number") + fval = float(value) + if not math.isfinite(fval): + raise ValueError(f"{field} must be finite") + if not (0.0 <= fval <= 1.0): + raise ValueError(f"{field} must be in [0.0, 1.0]") + return fval + + +def _check_token_bound(value: object, *, field: str) -> int: + if isinstance(value, bool): + raise TypeError(f"{field} must be int, not bool") + if not isinstance(value, int): + raise TypeError(f"{field} must be an integer") + if value < _MIN_TOKEN_BOUND or value > _MAX_TOKEN_BOUND: + raise ValueError( + f"{field} must be in [{_MIN_TOKEN_BOUND}, {_MAX_TOKEN_BOUND}]" + ) + return value + + +# ----------------------------------------------------------------------------- +# Row helpers +# ----------------------------------------------------------------------------- + + +def _extract_row_text(row: Mapping[str, Any]) -> str: + """Best-effort text extraction. Mirrors v0.55.0 / v0.56.0 row-text policy. + + Combines all text-shaped fields so PII / refusal scans never miss content + hiding in either ``text``/``content``/``output``/``response`` or inside a + ``messages`` chat structure. + """ + parts: List[str] = [] + for key in ("text", "content", "output", "response", "prompt", "instruction"): + val = row.get(key) + if isinstance(val, str) and val: + parts.append(val) + messages = row.get("messages") + if isinstance(messages, list): + for msg in messages: + if isinstance(msg, Mapping): + content = msg.get("content") + if isinstance(content, str) and content: + parts.append(content) + return "\n".join(parts) + + +def _extract_assistant_text(row: Mapping[str, Any]) -> str: + """Best-effort assistant-side text extraction (for refusal scans).""" + parts: List[str] = [] + for key in ("output", "response"): + val = row.get(key) + if isinstance(val, str) and val: + parts.append(val) + messages = row.get("messages") + if isinstance(messages, list): + for msg in messages: + if isinstance(msg, Mapping) and msg.get("role") == "assistant": + content = msg.get("content") + if isinstance(content, str) and content: + parts.append(content) + if parts: + return "\n".join(parts) + # Fall back to general row text when the row has no chat structure. + return _extract_row_text(row) + + +def _check_rows(rows: object) -> Sequence[Mapping[str, Any]]: + if isinstance(rows, str) or isinstance(rows, bytes): + raise TypeError("rows must be a list of mappings, not a string") + if not hasattr(rows, "__iter__"): + raise TypeError("rows must be iterable") + try: + materialised = list(rows) + except TypeError as exc: + raise TypeError(f"rows is not iterable: {exc}") from exc + return materialised + + +def _truncate_detail(detail: str) -> str: + if len(detail) > _MAX_DETAIL_LEN: + return detail[: _MAX_DETAIL_LEN - 3] + "..." + return detail + + +# ----------------------------------------------------------------------------- +# Expectations +# ----------------------------------------------------------------------------- + + +def expect_no_pii(rows: Any) -> ExpectationResult: + """Fail when any row contains an email / phone / SSN / credit-card hit. + + Reuses v0.47.0 ``data_score.detect_pii`` (Presidio backend if available; + falls back to the in-tree 4-regex baseline) — so improvements there + immediately flow into expectations. + """ + materialised = _check_rows(rows) + from soup_cli.utils.data_score import detect_pii # lazy + + num_violations = 0 + details: List[str] = [] + for index, row in enumerate(materialised): + if not isinstance(row, Mapping): + details.append(_truncate_detail(f"rows[{index}]: not a dict")) + num_violations += 1 + continue + text = _extract_row_text(row) + if not text: + continue + try: + hits = detect_pii(text) + except (TypeError, ValueError): + continue + if hits: + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + kinds = sorted({hit.get("kind", "?") for hit in hits}) + details.append( + _truncate_detail( + f"rows[{index}]: PII detected ({', '.join(kinds)})" + ) + ) + return ExpectationResult( + name="expect_no_pii", + passed=num_violations == 0, + num_rows_checked=len(materialised), + num_violations=num_violations, + details=tuple(details), + ) + + +def expect_token_length_between( + rows: Any, + *, + min_tokens: int, + max_tokens: int, +) -> ExpectationResult: + """Fail when any row's token count (whitespace-split) is out of bounds.""" + low = _check_token_bound(min_tokens, field="min_tokens") + high = _check_token_bound(max_tokens, field="max_tokens") + if low > high: + raise ValueError( + f"min_tokens ({low}) must be <= max_tokens ({high})" + ) + materialised = _check_rows(rows) + num_violations = 0 + details: List[str] = [] + for index, row in enumerate(materialised): + if not isinstance(row, Mapping): + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append(_truncate_detail(f"rows[{index}]: not a dict")) + continue + text = _extract_row_text(row) + token_count = len(text.split()) + if token_count < low or token_count > high: + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append( + _truncate_detail( + f"rows[{index}]: {token_count} tokens (want [{low}, {high}])" + ) + ) + return ExpectationResult( + name="expect_token_length_between", + passed=num_violations == 0, + num_rows_checked=len(materialised), + num_violations=num_violations, + details=tuple(details), + ) + + +def expect_no_refusal_pattern(rows: Any) -> ExpectationResult: + """Fail when any assistant-side output matches the v0.56.0 refusal regex.""" + materialised = _check_rows(rows) + from soup_cli.utils.diagnose.refusal import looks_like_refusal # lazy + + num_violations = 0 + details: List[str] = [] + for index, row in enumerate(materialised): + if not isinstance(row, Mapping): + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append(_truncate_detail(f"rows[{index}]: not a dict")) + continue + text = _extract_assistant_text(row) + if text and looks_like_refusal(text): + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append( + _truncate_detail(f"rows[{index}]: refusal pattern matched") + ) + return ExpectationResult( + name="expect_no_refusal_pattern", + passed=num_violations == 0, + num_rows_checked=len(materialised), + num_violations=num_violations, + details=tuple(details), + ) + + +def expect_chosen_preferred_over_rejected_by_judge( + rows: Any, + *, + judge_fn: Optional[JudgeFn] = None, + threshold: float = 0.7, +) -> ExpectationResult: + """Fail when ``judge_fn(row) < threshold`` on a preference row. + + Rows must carry both ``chosen`` and ``rejected``. ``judge_fn`` returns a + score in [0, 1]; 1.0 means the judge fully prefers chosen over rejected. + + When ``judge_fn`` is omitted the suite runs in *advisory* mode (every row + gets the default score 1.0, i.e. trust the labelling); production callers + should always supply a real judge. + """ + t = _check_threshold(threshold, field="threshold") + if judge_fn is not None and not callable(judge_fn): + raise TypeError("judge_fn must be callable or None") + materialised = _check_rows(rows) + num_violations = 0 + details: List[str] = [] + for index, row in enumerate(materialised): + if not isinstance(row, Mapping): + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append(_truncate_detail(f"rows[{index}]: not a dict")) + continue + if "chosen" not in row or "rejected" not in row: + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append( + _truncate_detail( + f"rows[{index}]: missing chosen/rejected field" + ) + ) + continue + if judge_fn is None: + # No judge supplied — assume chosen wins (advisory pass). + score: float = 1.0 + else: + try: + raw = judge_fn(row) + except Exception: # noqa: BLE001 — one bad row mustn't crash the suite + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append( + _truncate_detail(f"rows[{index}]: judge raised") + ) + continue + if isinstance(raw, bool) or not isinstance(raw, (int, float)): + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append( + _truncate_detail( + f"rows[{index}]: judge returned non-number" + ) + ) + continue + score = float(raw) + if not math.isfinite(score): + num_violations += 1 + continue + if score < t: + num_violations += 1 + if len(details) < _MAX_DETAILS_PER_RESULT: + details.append( + _truncate_detail( + f"rows[{index}]: judge score {score:.3f} < {t:.3f}" + ) + ) + return ExpectationResult( + name="expect_chosen_preferred_over_rejected_by_judge", + passed=num_violations == 0, + num_rows_checked=len(materialised), + num_violations=num_violations, + details=tuple(details), + ) + + +# ----------------------------------------------------------------------------- +# Suite parsing + execution +# ----------------------------------------------------------------------------- + + +def parse_suite_spec(raw: Any) -> SuiteSpec: + """Validate a suite dict and return a ``SuiteSpec``.""" + if not isinstance(raw, dict): + raise TypeError("suite spec must be a dict") + raw_expectations = raw.get("expectations") + if raw_expectations is None: + raise ValueError("suite must define 'expectations' key") + if not isinstance(raw_expectations, list): + raise TypeError("suite 'expectations' must be a list") + if not raw_expectations: + raise ValueError("suite 'expectations' must be a non-empty list") + if len(raw_expectations) > _MAX_SUITE_LEN: + raise ValueError( + f"suite 'expectations' exceeds {_MAX_SUITE_LEN} entries" + ) + + items: List[ExpectationSpec] = [] + for index, entry in enumerate(raw_expectations): + if not isinstance(entry, dict): + raise TypeError(f"expectations[{index}] must be a dict") + name = validate_expectation_name(entry.get("name", "")) + args = entry.get("args", {}) + if not isinstance(args, dict): + raise TypeError(f"expectations[{index}].args must be a dict") + items.append(ExpectationSpec(name=name, args=dict(args))) + return SuiteSpec(expectations=tuple(items)) + + +def parse_suite_yaml(text: object) -> SuiteSpec: + """Parse a suite YAML string into a validated ``SuiteSpec``.""" + if not isinstance(text, str): + raise TypeError("suite text must be a string") + if "\x00" in text: + raise ValueError("suite text must not contain null bytes") + if len(text.encode("utf-8")) > _MAX_FILE_BYTES: + raise ValueError(f"suite text exceeds {_MAX_FILE_BYTES} bytes") + import yaml # lazy + + try: + data = yaml.safe_load(text) + except yaml.YAMLError as exc: + raise ValueError(f"invalid YAML: {exc}") from exc + return parse_suite_spec(data) + + +def load_suite_yaml(path: object) -> SuiteSpec: + """Load a suite YAML from a path under cwd (with TOCTOU symlink reject). + + Delegates to ``utils.paths.enforce_under_cwd_and_no_symlink`` (v0.59.0 + centralised helper). + """ + from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink + + if isinstance(path, bool) or not isinstance(path, str): + raise TypeError("path must be a string") + enforce_under_cwd_and_no_symlink(path, "suite path") + if not os.path.lexists(path): + raise FileNotFoundError(path) + real = os.path.realpath(path) + if not os.path.isfile(real): + raise FileNotFoundError(real) + if os.path.getsize(real) > _MAX_FILE_BYTES: + raise ValueError(f"suite file exceeds {_MAX_FILE_BYTES} bytes") + with open(real, "r", encoding="utf-8") as handle: + return parse_suite_yaml(handle.read()) + + +def _dispatch_expectation( + spec: ExpectationSpec, + rows: Sequence[Mapping[str, Any]], + *, + judge_fn: Optional[JudgeFn] = None, +) -> ExpectationResult: + """Dispatch one ``ExpectationSpec`` against ``rows``. + + Per code-review HIGH H2: passes ``args`` values through *as-is* — the + expectation function's own validators must reject bool / NaN / Inf / + type-coerced inputs. This prevents an ``int("5")`` silent coercion from + bypassing ``_check_token_bound`` bool-rejection. + """ + name = spec.name + args = dict(spec.args) + if name == "expect_no_pii": + return expect_no_pii(rows) + if name == "expect_token_length_between": + return expect_token_length_between( + rows, + min_tokens=args.get("min_tokens", 1), + max_tokens=args.get("max_tokens", _MAX_TOKEN_BOUND), + ) + if name == "expect_no_refusal_pattern": + return expect_no_refusal_pattern(rows) + if name == "expect_chosen_preferred_over_rejected_by_judge": + return expect_chosen_preferred_over_rejected_by_judge( + rows, + judge_fn=judge_fn, + threshold=args.get("threshold", 0.7), + ) + raise ValueError(f"unhandled expectation: {name!r}") # pragma: no cover + + +def run_suite( + rows: Any, + spec: SuiteSpec, + *, + judge_fn: Optional[JudgeFn] = None, +) -> SuiteReport: + """Run every expectation in ``spec`` against ``rows``.""" + if not isinstance(spec, SuiteSpec): + raise TypeError("spec must be a SuiteSpec") + materialised = _check_rows(rows) + results: List[ExpectationResult] = [] + for expectation in spec.expectations: + results.append( + _dispatch_expectation(expectation, materialised, judge_fn=judge_fn) + ) + passed = all(r.passed for r in results) + return SuiteReport(passed=passed, results=tuple(results)) + + +__all__ = [ + "ExpectationResult", + "ExpectationSpec", + "JudgeFn", + "SUPPORTED_EXPECTATIONS", + "SuiteReport", + "SuiteSpec", + "expect_chosen_preferred_over_rejected_by_judge", + "expect_no_pii", + "expect_no_refusal_pattern", + "expect_token_length_between", + "load_suite_yaml", + "parse_suite_spec", + "parse_suite_yaml", + "run_suite", + "validate_expectation_name", +] diff --git a/soup_cli/utils/magpie.py b/soup_cli/utils/magpie.py new file mode 100644 index 0000000..ebfc274 --- /dev/null +++ b/soup_cli/utils/magpie.py @@ -0,0 +1,164 @@ +"""v0.69.0 Part C — `soup data gen magpie` synthetic generator. + +The Magpie technique (Xu et al. 2024) feeds an aligned chat-tuned model just +the chat-template prefix (system + user-turn header tokens) and lets the model +generate the user turn itself. The harvested user turns are then passed back to +the same model in a normal completion call to harvest the assistant response. +This is a clever way to produce high-volume SFT data without paying for human +prompts. + +This module ships the schema + validators + dry-run planner. Live generation +(invoking v0.20.0 Ollama / Anthropic / vLLM providers + running the quality +filter from v0.47.0 educational + toxicity scorers) is deferred to v0.69.1 +under the project-wide stub-then-live cadence (mirrors v0.50.0 / v0.61.0 / +v0.62.0 / v0.68.0). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +# Closed allowlist of providers — reuses v0.20.0 synth-data backends. +_SUPPORTED_MAGPIE_PROVIDERS = ("ollama", "anthropic", "vllm") +SUPPORTED_MAGPIE_PROVIDERS: frozenset = frozenset(_SUPPORTED_MAGPIE_PROVIDERS) + +_MAX_BASE_MODEL_LEN = 512 +_MAX_TARGET_ROWS = 1_000_000 +_MAX_PROVIDER_LEN = 32 + + +@dataclass(frozen=True) +class MagpieConfig: + """Frozen plan for one ``soup data gen-magpie`` invocation.""" + + base_model: str + provider: str + target_rows: int + quality_filter: bool + + def __post_init__(self) -> None: + # Re-validate so direct construction can't smuggle bad fields past + # the validators (mirrors v0.67.0 vector_bank policy). + validate_base_model(self.base_model) + validate_magpie_provider(self.provider) + validate_target_rows(self.target_rows) + if not isinstance(self.quality_filter, bool): + raise TypeError( + "MagpieConfig.quality_filter must be bool, " + f"got {type(self.quality_filter).__name__}" + ) + + +# ----------------------------------------------------------------------------- +# Validators +# ----------------------------------------------------------------------------- + + +def validate_magpie_provider(provider: object) -> str: + """Return the canonical lower-case provider id.""" + if isinstance(provider, bool) or not isinstance(provider, str): + raise TypeError( + f"magpie provider must be str, got {type(provider).__name__}" + ) + if not provider: + raise ValueError("magpie provider must be non-empty") + if "\x00" in provider: + raise ValueError("magpie provider must not contain null bytes") + if len(provider) > _MAX_PROVIDER_LEN: + raise ValueError( + f"magpie provider must be <= {_MAX_PROVIDER_LEN} chars" + ) + canonical = provider.strip().lower() + if canonical not in SUPPORTED_MAGPIE_PROVIDERS: + raise ValueError( + f"unknown magpie provider: {provider!r}. " + f"supported: {sorted(SUPPORTED_MAGPIE_PROVIDERS)}" + ) + return canonical + + +def validate_target_rows(target: object) -> int: + """Validate ``target_rows`` ∈ [1, 1_000_000]; bool-rejected.""" + if isinstance(target, bool): + raise TypeError("target_rows must be int, not bool") + if not isinstance(target, int): + raise TypeError("target_rows must be an integer") + if target < 1: + raise ValueError("target_rows must be >= 1") + if target > _MAX_TARGET_ROWS: + raise ValueError(f"target_rows must be <= {_MAX_TARGET_ROWS}") + return target + + +def validate_base_model(name: object) -> str: + """Validate the base-model identifier.""" + if isinstance(name, bool) or not isinstance(name, str): + raise TypeError( + f"base_model must be str, got {type(name).__name__}" + ) + if not name: + raise ValueError("base_model must be non-empty") + if "\x00" in name: + raise ValueError("base_model must not contain null bytes") + if len(name) > _MAX_BASE_MODEL_LEN: + raise ValueError( + f"base_model must be <= {_MAX_BASE_MODEL_LEN} chars" + ) + return name + + +# ----------------------------------------------------------------------------- +# Plan builder +# ----------------------------------------------------------------------------- + + +def build_magpie_config( + *, + base: str, + provider: str, + target: int, + quality_filter: bool = True, +) -> MagpieConfig: + """Convenience factory — every input passes through the validators.""" + if not isinstance(quality_filter, bool): + raise TypeError( + f"quality_filter must be bool, got {type(quality_filter).__name__}" + ) + return MagpieConfig( + base_model=validate_base_model(base), + provider=validate_magpie_provider(provider), + target_rows=validate_target_rows(target), + quality_filter=quality_filter, + ) + + +# ----------------------------------------------------------------------------- +# Live runner — deferred to v0.69.1 +# ----------------------------------------------------------------------------- + + +def run_magpie(config: Any) -> None: + """Execute the Magpie generation loop. Deferred to v0.69.1. + + The CLI catches ``NotImplementedError`` and exits with code 3 (distinct + from validation rejection exit 2) — same policy as v0.61.0 / v0.62.0 / + v0.68.0 deferred-live stubs. + """ + if not isinstance(config, MagpieConfig): + raise TypeError("config must be a MagpieConfig") + raise NotImplementedError( + "soup data gen-magpie live runner is deferred to v0.69.1 — " + "only --plan-only is wired today." + ) + + +__all__ = [ + "MagpieConfig", + "SUPPORTED_MAGPIE_PROVIDERS", + "build_magpie_config", + "run_magpie", + "validate_base_model", + "validate_magpie_provider", + "validate_target_rows", +] diff --git a/soup_cli/utils/persona_hub.py b/soup_cli/utils/persona_hub.py new file mode 100644 index 0000000..cf827a2 --- /dev/null +++ b/soup_cli/utils/persona_hub.py @@ -0,0 +1,277 @@ +"""v0.69.0 Part D — Persona-Hub diversity sampler. + +Samples prompt × persona × style combinations to drive synthetic-data diversity +(per Tencent's Persona-Hub paper). Ships with a small bundled persona list so +``soup data persona-mix`` works offline; operators wanting the full 200k-persona +HF dataset can pass ``--personas `` from a downloaded copy. + +The topic-diversity metric is a pure-Python token-entropy heuristic — same +approach as the v0.55.0 ``eval_design`` TF-IDF baseline. The full +sentence-transformer embedding-entropy variant is the operator-installable +``[data-pro]`` extras path; this module does not import any embedding library +to keep the CLI startup fast. +""" + +from __future__ import annotations + +import math +import random +from collections import Counter +from dataclasses import dataclass +from typing import Any, List, Mapping, Sequence, Tuple + +# 12 bundled personas covering broad demographic + occupational diversity. +# Operators wanting the full Persona-Hub corpus pass a JSONL via the CLI. +_BUNDLED_PERSONAS: Tuple[str, ...] = ( + "a curious high school student", + "a software engineer with 10 years of experience", + "a non-native English speaker learning to code", + "a research scientist in molecular biology", + "a small business owner planning their first product launch", + "a freelance journalist on a tight deadline", + "a retired teacher revisiting old hobbies", + "a graduate student preparing a thesis defense", + "a parent helping a child with homework", + "a healthcare professional asking patient-care questions", + "a hobbyist game developer trying a new engine", + "a senior engineer onboarding a junior teammate", +) + +_BUNDLED_STYLES: Tuple[str, ...] = ( + "formal", + "casual", + "step-by-step", + "concise", + "playful", +) + +_MAX_PERSONA_LEN = 1024 +_MAX_STYLE_LEN = 128 +_MAX_LIST_LEN = 100_000 +_MAX_SAMPLES = 1_000_000 +_MAX_DIVERSITY_TEXT_LEN = 65_536 + + +# ----------------------------------------------------------------------------- +# Validators +# ----------------------------------------------------------------------------- + + +def validate_persona(persona: object) -> str: + """Validate a persona string.""" + if isinstance(persona, bool) or not isinstance(persona, str): + raise TypeError( + f"persona must be str, got {type(persona).__name__}" + ) + if not persona: + raise ValueError("persona must be non-empty") + if "\x00" in persona: + raise ValueError("persona must not contain null bytes") + if len(persona) > _MAX_PERSONA_LEN: + raise ValueError(f"persona must be <= {_MAX_PERSONA_LEN} chars") + return persona + + +def validate_style(style: object) -> str: + """Validate a style string.""" + if isinstance(style, bool) or not isinstance(style, str): + raise TypeError(f"style must be str, got {type(style).__name__}") + if not style: + raise ValueError("style must be non-empty") + if "\x00" in style: + raise ValueError("style must not contain null bytes") + if len(style) > _MAX_STYLE_LEN: + raise ValueError(f"style must be <= {_MAX_STYLE_LEN} chars") + return style + + +def list_bundled_personas() -> Tuple[str, ...]: + """Return the bundled persona tuple. Immutable by virtue of being a tuple.""" + return _BUNDLED_PERSONAS + + +def list_bundled_styles() -> Tuple[str, ...]: + """Return the bundled style tuple.""" + return _BUNDLED_STYLES + + +# ----------------------------------------------------------------------------- +# Plan dataclass + factory +# ----------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class PersonaPlan: + """Frozen plan for a persona-mix sample run.""" + + prompts: Tuple[str, ...] + personas: Tuple[str, ...] + styles: Tuple[str, ...] + n: int + seed: int + + def __post_init__(self) -> None: + for field_name in ("prompts", "personas", "styles"): + value = getattr(self, field_name) + if not isinstance(value, tuple): + raise TypeError( + f"PersonaPlan.{field_name} must be a tuple " + "(frozen=True does not make List immutable)" + ) + if isinstance(self.n, bool): + raise TypeError("PersonaPlan.n must be int, not bool") + if not isinstance(self.n, int) or self.n < 1: + raise ValueError("PersonaPlan.n must be a positive integer") + if isinstance(self.seed, bool): + raise TypeError("PersonaPlan.seed must be int, not bool") + if not isinstance(self.seed, int): + raise TypeError("PersonaPlan.seed must be an integer") + + +def _check_list( + name: str, values: Sequence[Any], item_validator: Any +) -> Tuple[str, ...]: + if isinstance(values, (str, bytes)) or not hasattr(values, "__iter__"): + raise TypeError(f"{name} must be a list of strings") + materialised = list(values) + if not materialised: + raise ValueError(f"{name} must be non-empty") + if len(materialised) > _MAX_LIST_LEN: + raise ValueError(f"{name} exceeds {_MAX_LIST_LEN} entries") + return tuple(item_validator(v) for v in materialised) + + +def build_persona_plan( + *, + prompts: Sequence[str], + personas: Sequence[str], + styles: Sequence[str], + n: int, + seed: int, +) -> PersonaPlan: + """Validate inputs + return a frozen ``PersonaPlan``.""" + if isinstance(n, bool): + raise TypeError("n must be int, not bool") + if not isinstance(n, int): + raise TypeError("n must be int") + if n < 1: + raise ValueError("n must be >= 1") + if n > _MAX_SAMPLES: + raise ValueError(f"n exceeds {_MAX_SAMPLES}") + if isinstance(seed, bool): + raise TypeError("seed must be int, not bool") + if not isinstance(seed, int): + raise TypeError("seed must be int") + return PersonaPlan( + prompts=_check_list("prompts", prompts, validate_persona), + personas=_check_list("personas", personas, validate_persona), + styles=_check_list("styles", styles, validate_style), + n=n, + seed=seed, + ) + + +# ----------------------------------------------------------------------------- +# Sampler +# ----------------------------------------------------------------------------- + + +def sample_persona_matrix( + *, + prompts: Sequence[str], + personas: Sequence[str], + styles: Sequence[str], + n: int, + seed: int, +) -> List[Mapping[str, str]]: + """Sample ``n`` rows from the prompt × persona × style matrix. + + Deterministic given the same ``(inputs, seed)`` — uses a seeded ``Random`` + instance so concurrent samplers in the same process do not affect each + other. + """ + plan = build_persona_plan( + prompts=prompts, + personas=personas, + styles=styles, + n=n, + seed=seed, + ) + rng = random.Random(plan.seed) + rows: List[Mapping[str, str]] = [] + for _ in range(plan.n): + rows.append( + { + "prompt": rng.choice(plan.prompts), + "persona": rng.choice(plan.personas), + "style": rng.choice(plan.styles), + } + ) + return rows + + +# ----------------------------------------------------------------------------- +# Topic diversity +# ----------------------------------------------------------------------------- + + +def _extract_text(row: Mapping[str, Any]) -> str: + """Extract concatenated text from a row across common fields.""" + parts: List[str] = [] + for key in ("text", "content", "output", "prompt", "instruction", "response"): + val = row.get(key) + if isinstance(val, str) and val: + parts.append(val) + if not parts: + return "" + joined = " ".join(parts) + if len(joined) > _MAX_DIVERSITY_TEXT_LEN: + joined = joined[:_MAX_DIVERSITY_TEXT_LEN] + return joined + + +def compute_topic_diversity(rows: Any) -> float: + """Return a topic-diversity score in [0, 1]. + + Heuristic: pool the whitespace tokens across all rows, compute Shannon + entropy in bits, and normalise by ``log2(unique_tokens)`` so the score + is comparable across corpora. Returns ``0.0`` on empty input. + """ + if isinstance(rows, (str, bytes)): + raise TypeError("rows must be a list of mappings") + if not hasattr(rows, "__iter__"): + raise TypeError("rows must be iterable") + tokens: List[str] = [] + for row in rows: + if not isinstance(row, Mapping): + continue + text = _extract_text(row) + if not text: + continue + tokens.extend(text.lower().split()) + if not tokens: + return 0.0 + counter = Counter(tokens) + total = sum(counter.values()) + unique = len(counter) + if unique <= 1: + return 0.0 + entropy = -sum( + (count / total) * math.log2(count / total) for count in counter.values() + ) + max_entropy = math.log2(unique) + if max_entropy <= 0: + return 0.0 + return max(0.0, min(1.0, entropy / max_entropy)) + + +__all__ = [ + "PersonaPlan", + "build_persona_plan", + "compute_topic_diversity", + "list_bundled_personas", + "list_bundled_styles", + "sample_persona_matrix", + "validate_persona", + "validate_style", +] diff --git a/tests/test_v0690_part_a.py b/tests/test_v0690_part_a.py new file mode 100644 index 0000000..e67ba6a --- /dev/null +++ b/tests/test_v0690_part_a.py @@ -0,0 +1,733 @@ +"""v0.69.0 Part A — `soup build` dbt-for-SFT DAG parser + topo sort + plan.""" + +from __future__ import annotations + +import dataclasses +import os +import sys +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app +from soup_cli.utils import build_dag + + +def _write(path: Path, text: str) -> Path: + path.write_text(text, encoding="utf-8") + return path + + +# ----------------------------------------------------------------------------- +# Allowlist + immutability +# ----------------------------------------------------------------------------- + + +class TestSupportedKinds: + def test_supported_kinds_exact(self) -> None: + assert build_dag.SUPPORTED_MODEL_KINDS == frozenset( + {"incremental", "table", "view"} + ) + + def test_supported_kinds_is_frozenset(self) -> None: + assert isinstance(build_dag.SUPPORTED_MODEL_KINDS, frozenset) + + def test_supported_kinds_immutable(self) -> None: + with pytest.raises(AttributeError): + build_dag.SUPPORTED_MODEL_KINDS.add("evil") # type: ignore[attr-defined] + + +# ----------------------------------------------------------------------------- +# validate_model_kind +# ----------------------------------------------------------------------------- + + +class TestValidateModelKind: + def test_happy_path(self) -> None: + assert build_dag.validate_model_kind("incremental") == "incremental" + assert build_dag.validate_model_kind("table") == "table" + assert build_dag.validate_model_kind("view") == "view" + + def test_case_insensitive(self) -> None: + assert build_dag.validate_model_kind("Incremental") == "incremental" + assert build_dag.validate_model_kind("VIEW") == "view" + + def test_unknown(self) -> None: + with pytest.raises(ValueError, match="unknown model kind"): + build_dag.validate_model_kind("ephemeral") + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + build_dag.validate_model_kind(123) + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + build_dag.validate_model_kind(True) + + def test_empty(self) -> None: + with pytest.raises(ValueError): + build_dag.validate_model_kind("") + + def test_null_byte(self) -> None: + with pytest.raises(ValueError, match="null"): + build_dag.validate_model_kind("incremental\x00x") + + def test_oversize(self) -> None: + with pytest.raises(ValueError): + build_dag.validate_model_kind("x" * 100) + + +# ----------------------------------------------------------------------------- +# validate_model_name +# ----------------------------------------------------------------------------- + + +class TestValidateModelName: + def test_happy(self) -> None: + assert build_dag.validate_model_name("raw_chat") == "raw_chat" + assert build_dag.validate_model_name("step-1") == "step-1" + assert build_dag.validate_model_name("a.b") == "a.b" + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + build_dag.validate_model_name(42) + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + build_dag.validate_model_name(True) + + def test_empty(self) -> None: + with pytest.raises(ValueError): + build_dag.validate_model_name("") + + def test_null_byte(self) -> None: + with pytest.raises(ValueError, match="null"): + build_dag.validate_model_name("a\x00b") + + def test_path_traversal(self) -> None: + with pytest.raises(ValueError): + build_dag.validate_model_name("../etc") + + def test_path_separator(self) -> None: + with pytest.raises(ValueError): + build_dag.validate_model_name("a/b") + + def test_oversize(self) -> None: + with pytest.raises(ValueError): + build_dag.validate_model_name("a" * 200) + + +# ----------------------------------------------------------------------------- +# BuildModel frozen + validation +# ----------------------------------------------------------------------------- + + +class TestBuildModel: + def test_happy(self) -> None: + model = build_dag.BuildModel( + name="raw", + kind="incremental", + transform="identity", + refs=(), + source="data/raw.jsonl", + config={}, + ) + assert model.name == "raw" + assert model.kind == "incremental" + assert model.refs == () + + def test_frozen(self) -> None: + model = build_dag.BuildModel( + name="raw", + kind="incremental", + transform="identity", + refs=(), + source="data/raw.jsonl", + config={}, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + model.name = "evil" # type: ignore[misc] + + def test_refs_must_be_tuple(self) -> None: + with pytest.raises(TypeError, match="tuple"): + build_dag.BuildModel( + name="x", + kind="incremental", + transform="identity", + refs=["y"], # type: ignore[arg-type] + source=None, + config={}, + ) + + def test_invalid_kind_propagates(self) -> None: + with pytest.raises(ValueError): + build_dag.BuildModel( + name="raw", + kind="ephemeral", + transform="identity", + refs=(), + source=None, + config={}, + ) + + def test_invalid_name_propagates(self) -> None: + with pytest.raises(ValueError): + build_dag.BuildModel( + name="../etc", + kind="incremental", + transform="identity", + refs=(), + source=None, + config={}, + ) + + +# ----------------------------------------------------------------------------- +# Plan parsing + topo sort +# ----------------------------------------------------------------------------- + + +class TestParseBuildPlan: + def test_happy_linear(self) -> None: + raw = { + "models": [ + { + "name": "raw", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "identity", + }, + { + "name": "filtered", + "kind": "incremental", + "refs": ["raw"], + "transform": "filter_low_quality", + }, + { + "name": "tokenized", + "kind": "incremental", + "refs": ["filtered"], + "transform": "tokenize", + }, + ] + } + plan = build_dag.parse_build_plan(raw) + assert plan.topo_order == ("raw", "filtered", "tokenized") + assert len(plan.models) == 3 + + def test_happy_diamond(self) -> None: + raw = { + "models": [ + { + "name": "a", + "kind": "incremental", + "source": "data/seed.jsonl", + "transform": "x", + }, + {"name": "b", "kind": "incremental", "refs": ["a"], "transform": "x"}, + {"name": "c", "kind": "incremental", "refs": ["a"], "transform": "x"}, + { + "name": "d", + "kind": "incremental", + "refs": ["b", "c"], + "transform": "x", + }, + ] + } + plan = build_dag.parse_build_plan(raw) + # a must come first, d must come last + assert plan.topo_order[0] == "a" + assert plan.topo_order[-1] == "d" + + def test_non_dict(self) -> None: + with pytest.raises(TypeError): + build_dag.parse_build_plan(["models"]) # type: ignore[arg-type] + + def test_empty_models_rejected(self) -> None: + with pytest.raises(ValueError, match="non-empty"): + build_dag.parse_build_plan({"models": []}) + + def test_missing_models_key(self) -> None: + with pytest.raises(ValueError): + build_dag.parse_build_plan({}) + + def test_models_not_list(self) -> None: + with pytest.raises(ValueError): + build_dag.parse_build_plan({"models": "raw_chat"}) + + def test_cycle_rejected(self) -> None: + # Both have refs → no source; cycle rejection fires inside topo sort. + raw = { + "models": [ + {"name": "a", "kind": "incremental", "refs": ["b"], "transform": "x"}, + {"name": "b", "kind": "incremental", "refs": ["a"], "transform": "x"}, + ] + } + with pytest.raises(ValueError, match="cycle"): + build_dag.parse_build_plan(raw) + + def test_self_loop_rejected(self) -> None: + # refs=["a"] but self-ref also implies refs (not seed); cross-validator + # accepts (refs set + source None), self-loop fires in second pass. + raw = { + "models": [ + {"name": "a", "kind": "incremental", "refs": ["a"], "transform": "x"}, + ] + } + with pytest.raises(ValueError, match="self"): + build_dag.parse_build_plan(raw) + + def test_dangling_ref_rejected(self) -> None: + raw = { + "models": [ + { + "name": "a", + "kind": "incremental", + "refs": ["missing"], + "transform": "x", + }, + ] + } + with pytest.raises(ValueError, match="missing|unknown|not found"): + build_dag.parse_build_plan(raw) + + def test_duplicate_name_rejected(self) -> None: + # Duplicate-name check fires BEFORE BuildModel construction, so the + # seed/derived cross-validator is not reached. Use minimal shape. + raw = { + "models": [ + { + "name": "a", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "x", + }, + { + "name": "a", + "kind": "table", + "source": "data/raw.jsonl", + "transform": "x", + }, + ] + } + with pytest.raises(ValueError, match="duplicate"): + build_dag.parse_build_plan(raw) + + def test_oversize_models(self) -> None: + # Many-model cap fires BEFORE BuildModel construction. + raw = { + "models": [ + { + "name": f"m{i}", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "x", + } + for i in range(build_dag._MAX_MODELS + 1) + ] + } + with pytest.raises(ValueError, match="exceeds"): + build_dag.parse_build_plan(raw) + + def test_duplicate_ref_in_same_model_rejected(self) -> None: + raw = { + "models": [ + { + "name": "a", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "x", + }, + { + "name": "b", + "kind": "incremental", + "refs": ["a", "a"], + "transform": "x", + }, + ] + } + with pytest.raises(ValueError, match="duplicate"): + build_dag.parse_build_plan(raw) + + def test_seed_without_source_rejected(self) -> None: + # New cross-validator: refs=() + source=None is degenerate. + raw = { + "models": [ + {"name": "a", "kind": "incremental", "transform": "x"}, + ] + } + with pytest.raises(ValueError, match="source"): + build_dag.parse_build_plan(raw) + + def test_refs_and_source_mutually_exclusive(self) -> None: + # New cross-validator: refs + source together rejected. + raw = { + "models": [ + { + "name": "a", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "x", + }, + { + "name": "b", + "kind": "incremental", + "refs": ["a"], + "source": "data/other.jsonl", + "transform": "x", + }, + ] + } + with pytest.raises(ValueError, match="mutually exclusive"): + build_dag.parse_build_plan(raw) + + +# ----------------------------------------------------------------------------- +# parse_build_yaml + load_build_yaml +# ----------------------------------------------------------------------------- + + +class TestParseBuildYaml: + def test_happy(self) -> None: + text = ( + "models:\n" + " - name: raw\n" + " kind: incremental\n" + " source: data/raw.jsonl\n" + " transform: identity\n" + " - name: out\n" + " kind: incremental\n" + " refs: [raw]\n" + " transform: identity\n" + ) + plan = build_dag.parse_build_yaml(text) + assert plan.topo_order == ("raw", "out") + + def test_invalid_yaml(self) -> None: + with pytest.raises(ValueError, match="invalid YAML"): + build_dag.parse_build_yaml("models: [unclosed") + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + build_dag.parse_build_yaml(42) # type: ignore[arg-type] + + def test_null_byte(self) -> None: + with pytest.raises(ValueError, match="null"): + build_dag.parse_build_yaml("models:\x00") + + def test_oversize(self) -> None: + text = "models:\n" + (" - name: x\n kind: incremental\n transform: y\n" * 100000) + with pytest.raises(ValueError, match="exceeds"): + build_dag.parse_build_yaml(text) + + +class TestLoadBuildYaml: + def test_happy(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + text = ( + "models:\n" + " - name: raw\n" + " kind: incremental\n" + " source: data/raw.jsonl\n" + " transform: identity\n" + ) + path = _write(tmp_path / "build.yaml", text) + plan = build_dag.load_build_yaml(str(path)) + assert plan.topo_order == ("raw",) + + def test_outside_cwd(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + outside = tmp_path / "outside" + outside.mkdir() + _write(outside / "build.yaml", "models: []") + sub = tmp_path / "sub" + sub.mkdir() + monkeypatch.chdir(sub) + with pytest.raises(ValueError, match="cwd"): + build_dag.load_build_yaml(str(outside / "build.yaml")) + + def test_missing(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + build_dag.load_build_yaml(str(tmp_path / "nope.yaml")) + + def test_null_byte(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError, match="null"): + build_dag.load_build_yaml("a\x00b.yaml") + + def test_empty_path(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + with pytest.raises(ValueError): + build_dag.load_build_yaml("") + + def test_non_string(self) -> None: + with pytest.raises((TypeError, ValueError)): + build_dag.load_build_yaml(42) # type: ignore[arg-type] + + @pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlink") + def test_symlink_rejected( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + target = _write( + tmp_path / "real.yaml", + "models:\n - name: a\n kind: incremental\n transform: x\n", + ) + link = tmp_path / "link.yaml" + os.symlink(str(target), str(link)) + with pytest.raises(ValueError, match="symlink"): + build_dag.load_build_yaml(str(link)) + + +# ----------------------------------------------------------------------------- +# Incremental diff (re-tokenize only changed rows) +# ----------------------------------------------------------------------------- + + +class TestRowHash: + def test_deterministic(self) -> None: + row = {"id": "1", "text": "hello"} + assert build_dag.compute_row_hash(row) == build_dag.compute_row_hash(row) + + def test_key_order_independent(self) -> None: + a = {"id": "1", "text": "hello"} + b = {"text": "hello", "id": "1"} + assert build_dag.compute_row_hash(a) == build_dag.compute_row_hash(b) + + def test_content_sensitive(self) -> None: + a = {"id": "1", "text": "hello"} + b = {"id": "1", "text": "world"} + assert build_dag.compute_row_hash(a) != build_dag.compute_row_hash(b) + + def test_non_dict(self) -> None: + with pytest.raises(TypeError): + build_dag.compute_row_hash([1, 2]) # type: ignore[arg-type] + + +class TestIncrementalDiff: + def test_all_new(self) -> None: + new = [{"id": "1", "text": "a"}, {"id": "2", "text": "b"}] + report = build_dag.incremental_diff([], new) + assert report.added == 2 + assert report.changed == 0 + assert report.removed == 0 + assert report.unchanged == 0 + + def test_all_unchanged(self) -> None: + rows = [{"id": "1", "text": "a"}, {"id": "2", "text": "b"}] + report = build_dag.incremental_diff(rows, rows) + assert report.unchanged == 2 + assert report.added == 0 + assert report.changed == 0 + assert report.removed == 0 + + def test_changed_row(self) -> None: + prev = [{"id": "1", "text": "a"}] + new = [{"id": "1", "text": "b"}] + report = build_dag.incremental_diff(prev, new) + assert report.changed == 1 + assert report.added == 0 + + def test_removed_row(self) -> None: + prev = [{"id": "1", "text": "a"}, {"id": "2", "text": "b"}] + new = [{"id": "1", "text": "a"}] + report = build_dag.incremental_diff(prev, new) + assert report.removed == 1 + assert report.unchanged == 1 + + def test_missing_id_field(self) -> None: + with pytest.raises(ValueError, match="id"): + build_dag.incremental_diff([{"text": "a"}], []) + + def test_frozen_report(self) -> None: + report = build_dag.incremental_diff([], []) + with pytest.raises(dataclasses.FrozenInstanceError): + report.added = 999 # type: ignore[misc] + + +class TestBuildPlanFrozen: + """TDD review #12 — BuildPlan dataclass must be immutable.""" + + def test_frozen(self) -> None: + raw = { + "models": [ + { + "name": "a", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "x", + } + ] + } + plan = build_dag.parse_build_plan(raw) + with pytest.raises(dataclasses.FrozenInstanceError): + plan.topo_order = ("evil",) # type: ignore[misc] + + +# ----------------------------------------------------------------------------- +# Plan rendering (dry-run output) +# ----------------------------------------------------------------------------- + + +class TestRenderPlanTable: + def test_happy(self) -> None: + raw = { + "models": [ + { + "name": "raw", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "identity", + }, + { + "name": "filtered", + "kind": "incremental", + "refs": ["raw"], + "transform": "filter", + }, + ] + } + plan = build_dag.parse_build_plan(raw) + rendered = build_dag.render_plan_table(plan) + assert "raw" in rendered + assert "filtered" in rendered + # topo order preserved + assert rendered.index("raw") < rendered.index("filtered") + + def test_non_plan_type(self) -> None: + with pytest.raises(TypeError): + build_dag.render_plan_table({"models": []}) # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# run_build (deferred-live) +# ----------------------------------------------------------------------------- + + +class TestRunBuild: + def test_deferred(self) -> None: + raw = { + "models": [ + { + "name": "raw", + "kind": "incremental", + "source": "data/raw.jsonl", + "transform": "identity", + }, + ] + } + plan = build_dag.parse_build_plan(raw) + with pytest.raises(NotImplementedError, match="v0.69.1"): + build_dag.run_build(plan) + + def test_run_build_validates_plan_type(self) -> None: + with pytest.raises(TypeError): + build_dag.run_build({"models": []}) # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# CLI: `soup build` +# ----------------------------------------------------------------------------- + + +class TestSoupBuildCli: + def test_help(self) -> None: + runner = CliRunner() + result = runner.invoke(app, ["build", "--help"]) + assert result.exit_code == 0, result.output + assert "build" in result.output.lower() + + def test_dry_run_happy( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + path = _write( + tmp_path / "build.yaml", + "models:\n" + " - name: raw\n" + " kind: incremental\n" + " source: data/raw.jsonl\n" + " transform: identity\n" + " - name: filtered\n" + " kind: incremental\n" + " refs: [raw]\n" + " transform: filter\n", + ) + runner = CliRunner() + result = runner.invoke(app, ["build", str(path), "--dry-run"]) + assert result.exit_code == 0, result.output + assert "raw" in result.output + assert "filtered" in result.output + + def test_missing_config( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + runner = CliRunner() + result = runner.invoke(app, ["build", "nope.yaml", "--dry-run"]) + assert result.exit_code != 0 + + def test_live_deferred( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + path = _write( + tmp_path / "build.yaml", + "models:\n - name: raw\n kind: incremental\n" + " source: data/raw.jsonl\n transform: identity\n", + ) + runner = CliRunner() + result = runner.invoke(app, ["build", str(path)]) + # deferred runner exits non-zero with v0.69.1 marker + assert result.exit_code != 0 + assert "0.69.1" in result.output or "deferred" in result.output.lower() + + def test_outside_cwd( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + outside = tmp_path / "outside" + outside.mkdir() + _write( + outside / "build.yaml", + "models:\n - name: a\n kind: incremental\n transform: x\n", + ) + sub = tmp_path / "sub" + sub.mkdir() + monkeypatch.chdir(sub) + runner = CliRunner() + result = runner.invoke(app, ["build", str(outside / "build.yaml"), "--dry-run"]) + assert result.exit_code != 0 + + +# ----------------------------------------------------------------------------- +# Source-grep wiring +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_cli_registers_build(self) -> None: + root = Path(__file__).resolve().parent.parent + cli = (root / "soup_cli" / "cli.py").read_text(encoding="utf-8") + assert ( + 'name="build"' in cli + or "build_cmd" in cli + or "from soup_cli.commands import build" in cli + ) + + def test_no_heavy_top_level_imports(self) -> None: + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "build_dag.py").read_text(encoding="utf-8") + # yaml is also lazy-imported inside parse_build_yaml (TDD review H2). + for forbidden in ( + "\nimport torch", + "\nimport transformers", + "\nimport peft", + "\nimport yaml", + ): + assert forbidden not in src, f"top-level import found: {forbidden!r}" + + def test_version_bumped(self) -> None: + from soup_cli import __version__ + + assert __version__ == "0.69.0" diff --git a/tests/test_v0690_part_b.py b/tests/test_v0690_part_b.py new file mode 100644 index 0000000..a573be9 --- /dev/null +++ b/tests/test_v0690_part_b.py @@ -0,0 +1,587 @@ +"""v0.69.0 Part B — Expectations suite for chat data.""" + +from __future__ import annotations + +import dataclasses +import os +import sys +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app +from soup_cli.utils import expectations + + +def _write(path: Path, text: str) -> Path: + path.write_text(text, encoding="utf-8") + return path + + +# ----------------------------------------------------------------------------- +# Allowlist + immutability +# ----------------------------------------------------------------------------- + + +class TestSupportedExpectations: + def test_exact(self) -> None: + assert expectations.SUPPORTED_EXPECTATIONS == frozenset( + { + "expect_no_pii", + "expect_token_length_between", + "expect_no_refusal_pattern", + "expect_chosen_preferred_over_rejected_by_judge", + } + ) + + def test_is_frozenset(self) -> None: + assert isinstance(expectations.SUPPORTED_EXPECTATIONS, frozenset) + + def test_immutable(self) -> None: + with pytest.raises(AttributeError): + expectations.SUPPORTED_EXPECTATIONS.add( # type: ignore[attr-defined] + "evil" + ) + + +# ----------------------------------------------------------------------------- +# validate_expectation_name +# ----------------------------------------------------------------------------- + + +class TestValidateExpectationName: + def test_happy(self) -> None: + assert ( + expectations.validate_expectation_name("expect_no_pii") == "expect_no_pii" + ) + + def test_case_insensitive(self) -> None: + assert ( + expectations.validate_expectation_name("EXPECT_NO_PII") == "expect_no_pii" + ) + + def test_unknown(self) -> None: + with pytest.raises(ValueError, match="unknown expectation"): + expectations.validate_expectation_name("expect_perfection") + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + expectations.validate_expectation_name(42) + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + expectations.validate_expectation_name(True) + + def test_empty(self) -> None: + with pytest.raises(ValueError): + expectations.validate_expectation_name("") + + def test_null_byte(self) -> None: + with pytest.raises(ValueError, match="null"): + expectations.validate_expectation_name("expect\x00_no_pii") + + def test_oversize(self) -> None: + with pytest.raises(ValueError): + expectations.validate_expectation_name("x" * 200) + + +# ----------------------------------------------------------------------------- +# ExpectationResult frozen +# ----------------------------------------------------------------------------- + + +class TestExpectationResult: + def test_happy(self) -> None: + r = expectations.ExpectationResult( + name="expect_no_pii", + passed=True, + num_rows_checked=10, + num_violations=0, + details=(), + ) + assert r.passed is True + + def test_frozen(self) -> None: + r = expectations.ExpectationResult( + name="expect_no_pii", + passed=True, + num_rows_checked=10, + num_violations=0, + details=(), + ) + with pytest.raises(dataclasses.FrozenInstanceError): + r.passed = False # type: ignore[misc] + + def test_details_must_be_tuple(self) -> None: + with pytest.raises(TypeError): + expectations.ExpectationResult( + name="expect_no_pii", + passed=True, + num_rows_checked=10, + num_violations=0, + details=["x"], # type: ignore[arg-type] + ) + + def test_violations_non_negative(self) -> None: + with pytest.raises(ValueError): + expectations.ExpectationResult( + name="expect_no_pii", + passed=True, + num_rows_checked=10, + num_violations=-1, + details=(), + ) + + def test_invalid_name(self) -> None: + with pytest.raises(ValueError): + expectations.ExpectationResult( + name="bogus", + passed=True, + num_rows_checked=0, + num_violations=0, + details=(), + ) + + +# ----------------------------------------------------------------------------- +# expect_no_pii +# ----------------------------------------------------------------------------- + + +class TestExpectNoPii: + def test_clean(self) -> None: + rows = [ + {"text": "Hello, how are you today?"}, + {"text": "This is a normal sentence."}, + ] + result = expectations.expect_no_pii(rows) + assert result.passed is True + assert result.num_violations == 0 + assert result.num_rows_checked == 2 + + def test_email_flagged(self) -> None: + rows = [{"text": "Email me at evil@example.com please."}] + result = expectations.expect_no_pii(rows) + assert result.passed is False + assert result.num_violations >= 1 + + def test_phone_flagged(self) -> None: + rows = [{"text": "Call me at 555-123-4567 anytime."}] + result = expectations.expect_no_pii(rows) + assert result.passed is False + + def test_empty_rows(self) -> None: + result = expectations.expect_no_pii([]) + assert result.passed is True + assert result.num_rows_checked == 0 + + def test_messages_field(self) -> None: + rows = [ + { + "messages": [ + {"role": "user", "content": "Hi"}, + {"role": "assistant", "content": "Hello, my email is evil@e.com"}, + ] + } + ] + result = expectations.expect_no_pii(rows) + assert result.passed is False + + def test_non_list_raises(self) -> None: + with pytest.raises(TypeError): + expectations.expect_no_pii("rows") # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# expect_token_length_between +# ----------------------------------------------------------------------------- + + +class TestExpectTokenLengthBetween: + def test_clean(self) -> None: + rows = [{"text": "a " * 50}, {"text": "b " * 60}] + result = expectations.expect_token_length_between(rows, min_tokens=10, max_tokens=200) + assert result.passed is True + + def test_below_min(self) -> None: + rows = [{"text": "short"}] + result = expectations.expect_token_length_between(rows, min_tokens=50, max_tokens=200) + assert result.passed is False + assert result.num_violations == 1 + + def test_above_max(self) -> None: + rows = [{"text": "a " * 500}] + result = expectations.expect_token_length_between(rows, min_tokens=10, max_tokens=50) + assert result.passed is False + assert result.num_violations == 1 + + def test_invalid_min(self) -> None: + with pytest.raises(ValueError): + expectations.expect_token_length_between([], min_tokens=-1, max_tokens=100) + + def test_invalid_max(self) -> None: + with pytest.raises(ValueError): + expectations.expect_token_length_between([], min_tokens=10, max_tokens=0) + + def test_min_greater_than_max(self) -> None: + with pytest.raises(ValueError): + expectations.expect_token_length_between([], min_tokens=200, max_tokens=10) + + def test_bool_min(self) -> None: + with pytest.raises(TypeError): + expectations.expect_token_length_between([], min_tokens=True, max_tokens=100) + + def test_bool_max(self) -> None: + with pytest.raises(TypeError): + expectations.expect_token_length_between([], min_tokens=1, max_tokens=False) + + +# ----------------------------------------------------------------------------- +# expect_no_refusal_pattern +# ----------------------------------------------------------------------------- + + +class TestExpectNoRefusalPattern: + def test_clean(self) -> None: + rows = [{"output": "Here is the recipe you asked for."}] + result = expectations.expect_no_refusal_pattern(rows) + assert result.passed is True + + def test_refusal_detected(self) -> None: + rows = [{"output": "I cannot help with that request."}] + result = expectations.expect_no_refusal_pattern(rows) + assert result.passed is False + + def test_assistant_refusal_in_messages(self) -> None: + rows = [ + { + "messages": [ + {"role": "user", "content": "How do I do X?"}, + {"role": "assistant", "content": "Sorry, but I refuse to discuss that."}, + ] + } + ] + result = expectations.expect_no_refusal_pattern(rows) + assert result.passed is False + + def test_empty(self) -> None: + result = expectations.expect_no_refusal_pattern([]) + assert result.passed is True + + +# ----------------------------------------------------------------------------- +# expect_chosen_preferred_over_rejected_by_judge +# ----------------------------------------------------------------------------- + + +class TestChosenPreferredByJudge: + def test_clean(self) -> None: + rows = [ + {"prompt": "Q", "chosen": "good", "rejected": "bad"}, + {"prompt": "Q2", "chosen": "great", "rejected": "terrible"}, + ] + + def judge(row: dict) -> float: + # Chosen always wins; score in [0, 1]. + return 0.9 + + result = expectations.expect_chosen_preferred_over_rejected_by_judge( + rows, judge_fn=judge, threshold=0.7 + ) + assert result.passed is True + assert result.num_violations == 0 + + def test_violations(self) -> None: + rows = [{"prompt": "Q", "chosen": "good", "rejected": "bad"}] + + def judge(row: dict) -> float: + return 0.3 # below threshold + + result = expectations.expect_chosen_preferred_over_rejected_by_judge( + rows, judge_fn=judge, threshold=0.7 + ) + assert result.passed is False + assert result.num_violations == 1 + + def test_missing_chosen_rejected_fields(self) -> None: + rows = [{"prompt": "Q"}] + + def judge(row: dict) -> float: + return 0.9 + + # Rows missing chosen+rejected are flagged as violations. + result = expectations.expect_chosen_preferred_over_rejected_by_judge( + rows, judge_fn=judge, threshold=0.7 + ) + assert result.passed is False + + def test_invalid_threshold(self) -> None: + with pytest.raises(ValueError): + expectations.expect_chosen_preferred_over_rejected_by_judge( + [], judge_fn=lambda r: 0.9, threshold=1.5 + ) + + def test_non_callable_judge(self) -> None: + with pytest.raises(TypeError): + expectations.expect_chosen_preferred_over_rejected_by_judge( + [], judge_fn="not callable", threshold=0.5 # type: ignore[arg-type] + ) + + def test_bool_threshold(self) -> None: + with pytest.raises(TypeError): + expectations.expect_chosen_preferred_over_rejected_by_judge( + [], judge_fn=lambda r: 0.5, threshold=True + ) + + def test_nan_threshold(self) -> None: + with pytest.raises(ValueError): + expectations.expect_chosen_preferred_over_rejected_by_judge( + [], judge_fn=lambda r: 0.5, threshold=float("nan") + ) + + def test_judge_exception_treated_as_violation(self) -> None: + rows = [{"prompt": "Q", "chosen": "c", "rejected": "r"}] + + def bad_judge(row: dict) -> float: + raise RuntimeError("kaboom") + + result = expectations.expect_chosen_preferred_over_rejected_by_judge( + rows, judge_fn=bad_judge, threshold=0.5 + ) + assert result.passed is False + assert result.num_violations == 1 + + +# ----------------------------------------------------------------------------- +# Suite spec + runner +# ----------------------------------------------------------------------------- + + +class TestSuiteSpec: + def test_happy(self) -> None: + spec = expectations.parse_suite_spec( + { + "expectations": [ + {"name": "expect_no_pii"}, + { + "name": "expect_token_length_between", + "args": {"min_tokens": 10, "max_tokens": 1000}, + }, + ] + } + ) + assert len(spec.expectations) == 2 + + def test_non_dict(self) -> None: + with pytest.raises(TypeError): + expectations.parse_suite_spec(["nope"]) # type: ignore[arg-type] + + def test_missing_expectations(self) -> None: + with pytest.raises(ValueError): + expectations.parse_suite_spec({}) + + def test_empty_expectations(self) -> None: + with pytest.raises(ValueError, match="non-empty"): + expectations.parse_suite_spec({"expectations": []}) + + def test_unknown_expectation(self) -> None: + with pytest.raises(ValueError): + expectations.parse_suite_spec( + {"expectations": [{"name": "expect_perfection"}]} + ) + + def test_args_must_be_dict(self) -> None: + with pytest.raises(TypeError): + expectations.parse_suite_spec( + { + "expectations": [ + {"name": "expect_no_pii", "args": "not a dict"} + ] + } + ) + + def test_too_many_expectations(self) -> None: + raw = { + "expectations": [ + {"name": "expect_no_pii"} for _ in range(expectations._MAX_SUITE_LEN + 1) + ] + } + with pytest.raises(ValueError, match="exceeds"): + expectations.parse_suite_spec(raw) + + +class TestRunSuite: + def test_all_pass(self) -> None: + spec = expectations.parse_suite_spec( + {"expectations": [{"name": "expect_no_pii"}]} + ) + rows = [{"text": "all clean here"}] + report = expectations.run_suite(rows, spec) + assert report.passed is True + assert len(report.results) == 1 + + def test_one_fails(self) -> None: + spec = expectations.parse_suite_spec( + {"expectations": [{"name": "expect_no_pii"}]} + ) + rows = [{"text": "email me at a@b.com"}] + report = expectations.run_suite(rows, spec) + assert report.passed is False + + def test_token_length_args(self) -> None: + spec = expectations.parse_suite_spec( + { + "expectations": [ + { + "name": "expect_token_length_between", + "args": {"min_tokens": 10, "max_tokens": 1000}, + } + ] + } + ) + rows = [{"text": "a " * 500}] + report = expectations.run_suite(rows, spec) + assert report.passed is True + + def test_chosen_preferred_arg(self) -> None: + spec = expectations.parse_suite_spec( + { + "expectations": [ + { + "name": "expect_chosen_preferred_over_rejected_by_judge", + "args": {"threshold": 0.5}, + } + ] + } + ) + rows = [{"prompt": "Q", "chosen": "good", "rejected": "bad"}] + # No judge fn supplied = neutral pass (operator must inject for live). + report = expectations.run_suite(rows, spec) + # Default judge returns 1.0 (no judge → assume chosen wins). + assert report.passed is True + + def test_report_frozen(self) -> None: + spec = expectations.parse_suite_spec( + {"expectations": [{"name": "expect_no_pii"}]} + ) + report = expectations.run_suite([], spec) + with pytest.raises(dataclasses.FrozenInstanceError): + report.passed = False # type: ignore[misc] + + +# ----------------------------------------------------------------------------- +# YAML loaders +# ----------------------------------------------------------------------------- + + +class TestLoadSuiteYaml: + def test_happy(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + path = _write( + tmp_path / "suite.yaml", + "expectations:\n - name: expect_no_pii\n", + ) + spec = expectations.load_suite_yaml(str(path)) + assert len(spec.expectations) == 1 + + def test_outside_cwd(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + outside = tmp_path / "outside" + outside.mkdir() + _write(outside / "s.yaml", "expectations:\n - name: expect_no_pii\n") + sub = tmp_path / "sub" + sub.mkdir() + monkeypatch.chdir(sub) + with pytest.raises(ValueError, match="cwd"): + expectations.load_suite_yaml(str(outside / "s.yaml")) + + @pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlink") + def test_symlink_rejected(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + target = _write(tmp_path / "real.yaml", "expectations:\n - name: expect_no_pii\n") + link = tmp_path / "link.yaml" + os.symlink(str(target), str(link)) + with pytest.raises(ValueError, match="symlink"): + expectations.load_suite_yaml(str(link)) + + def test_missing(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + with pytest.raises(FileNotFoundError): + expectations.load_suite_yaml(str(tmp_path / "nope.yaml")) + + +# ----------------------------------------------------------------------------- +# CLI: `soup expect` +# ----------------------------------------------------------------------------- + + +class TestSoupExpectCli: + def test_help(self) -> None: + runner = CliRunner() + result = runner.invoke(app, ["expect", "--help"]) + assert result.exit_code == 0, result.output + assert "expect" in result.output.lower() + + def test_all_pass(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + data = _write(tmp_path / "data.jsonl", '{"text": "clean output here"}\n') + suite = _write( + tmp_path / "suite.yaml", "expectations:\n - name: expect_no_pii\n" + ) + runner = CliRunner() + result = runner.invoke(app, ["expect", str(data), str(suite)]) + assert result.exit_code == 0, result.output + + def test_failure_exits_3(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + data = _write( + tmp_path / "data.jsonl", '{"text": "email me at evil@e.com"}\n' + ) + suite = _write( + tmp_path / "suite.yaml", "expectations:\n - name: expect_no_pii\n" + ) + runner = CliRunner() + result = runner.invoke(app, ["expect", str(data), str(suite)]) + assert result.exit_code == 3 + + def test_outside_cwd_data(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + outside = tmp_path / "outside" + outside.mkdir() + _write(outside / "d.jsonl", '{"text":"x"}\n') + sub = tmp_path / "sub" + sub.mkdir() + suite = _write(sub / "suite.yaml", "expectations:\n - name: expect_no_pii\n") + monkeypatch.chdir(sub) + runner = CliRunner() + result = runner.invoke( + app, ["expect", str(outside / "d.jsonl"), str(suite)] + ) + assert result.exit_code != 0 + + +# ----------------------------------------------------------------------------- +# Source wiring +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_cli_registers_expect(self) -> None: + root = Path(__file__).resolve().parent.parent + cli = (root / "soup_cli" / "cli.py").read_text(encoding="utf-8") + assert 'name="expect"' in cli or "from soup_cli.commands import expect" in cli + + def test_no_heavy_top_level_imports(self) -> None: + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "expectations.py").read_text(encoding="utf-8") + for forbidden in ( + "\nimport torch", + "\nimport transformers", + "\nimport peft", + "\nimport yaml", + ): + assert forbidden not in src + + def test_version_bumped(self) -> None: + from soup_cli import __version__ + + assert __version__ == "0.69.0" diff --git a/tests/test_v0690_part_c.py b/tests/test_v0690_part_c.py new file mode 100644 index 0000000..3f72098 --- /dev/null +++ b/tests/test_v0690_part_c.py @@ -0,0 +1,341 @@ +"""v0.69.0 Part C — `soup data gen magpie` synthetic generator.""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app +from soup_cli.utils import magpie + + +def _write(path: Path, text: str) -> Path: + path.write_text(text, encoding="utf-8") + return path + + +# ----------------------------------------------------------------------------- +# Provider allowlist +# ----------------------------------------------------------------------------- + + +class TestSupportedProviders: + def test_exact(self) -> None: + assert magpie.SUPPORTED_MAGPIE_PROVIDERS == frozenset( + {"ollama", "anthropic", "vllm"} + ) + + def test_immutable(self) -> None: + with pytest.raises(AttributeError): + magpie.SUPPORTED_MAGPIE_PROVIDERS.add("evil") # type: ignore[attr-defined] + + +class TestValidateProvider: + def test_happy(self) -> None: + assert magpie.validate_magpie_provider("ollama") == "ollama" + assert magpie.validate_magpie_provider("ANTHROPIC") == "anthropic" + + def test_unknown(self) -> None: + with pytest.raises(ValueError, match="unknown magpie provider"): + magpie.validate_magpie_provider("openai") + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + magpie.validate_magpie_provider(42) + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + magpie.validate_magpie_provider(True) + + def test_empty(self) -> None: + with pytest.raises(ValueError): + magpie.validate_magpie_provider("") + + def test_null_byte(self) -> None: + with pytest.raises(ValueError, match="null"): + magpie.validate_magpie_provider("ollama\x00") + + +# ----------------------------------------------------------------------------- +# Target rows + base model validators +# ----------------------------------------------------------------------------- + + +class TestValidateTargetRows: + def test_happy(self) -> None: + assert magpie.validate_target_rows(100) == 100 + + def test_zero_rejected(self) -> None: + with pytest.raises(ValueError): + magpie.validate_target_rows(0) + + def test_negative(self) -> None: + with pytest.raises(ValueError): + magpie.validate_target_rows(-5) + + def test_overcap(self) -> None: + with pytest.raises(ValueError): + magpie.validate_target_rows(magpie._MAX_TARGET_ROWS + 1) + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + magpie.validate_target_rows(True) + + def test_non_int(self) -> None: + with pytest.raises(TypeError): + magpie.validate_target_rows(1.5) + + +class TestValidateBaseModel: + def test_happy(self) -> None: + assert ( + magpie.validate_base_model("meta-llama/Llama-3.1-8B-Instruct") + == "meta-llama/Llama-3.1-8B-Instruct" + ) + + def test_empty(self) -> None: + with pytest.raises(ValueError): + magpie.validate_base_model("") + + def test_null_byte(self) -> None: + with pytest.raises(ValueError, match="null"): + magpie.validate_base_model("meta\x00llama") + + def test_oversize(self) -> None: + with pytest.raises(ValueError): + magpie.validate_base_model("x" * 1024) + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + magpie.validate_base_model(42) + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + magpie.validate_base_model(True) + + +# ----------------------------------------------------------------------------- +# MagpieConfig frozen dataclass +# ----------------------------------------------------------------------------- + + +class TestMagpieConfig: + def test_happy(self) -> None: + cfg = magpie.MagpieConfig( + base_model="meta-llama/Llama-3.1-8B-Instruct", + provider="ollama", + target_rows=100, + quality_filter=True, + ) + assert cfg.target_rows == 100 + assert cfg.quality_filter is True + + def test_frozen(self) -> None: + cfg = magpie.MagpieConfig( + base_model="m", + provider="ollama", + target_rows=10, + quality_filter=False, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + cfg.target_rows = 999 # type: ignore[misc] + + def test_invalid_provider_propagates(self) -> None: + with pytest.raises(ValueError): + magpie.MagpieConfig( + base_model="m", + provider="openai", + target_rows=10, + quality_filter=False, + ) + + def test_invalid_target_propagates(self) -> None: + with pytest.raises(ValueError): + magpie.MagpieConfig( + base_model="m", + provider="ollama", + target_rows=0, + quality_filter=False, + ) + + def test_invalid_base_propagates(self) -> None: + with pytest.raises(ValueError): + magpie.MagpieConfig( + base_model="", + provider="ollama", + target_rows=10, + quality_filter=False, + ) + + def test_quality_filter_must_be_bool(self) -> None: + with pytest.raises(TypeError): + magpie.MagpieConfig( + base_model="m", + provider="ollama", + target_rows=10, + quality_filter="yes", # type: ignore[arg-type] + ) + + +# ----------------------------------------------------------------------------- +# build_magpie_config factory +# ----------------------------------------------------------------------------- + + +class TestBuildMagpieConfig: + def test_happy(self) -> None: + cfg = magpie.build_magpie_config( + base="meta-llama/Llama-3.1-8B-Instruct", + provider="OLLAMA", + target=50, + ) + assert cfg.provider == "ollama" + assert cfg.target_rows == 50 + assert cfg.quality_filter is True # default + + def test_quality_filter_off(self) -> None: + cfg = magpie.build_magpie_config( + base="m", + provider="ollama", + target=10, + quality_filter=False, + ) + assert cfg.quality_filter is False + + +# ----------------------------------------------------------------------------- +# Deferred live runner +# ----------------------------------------------------------------------------- + + +class TestRunMagpie: + def test_deferred(self) -> None: + cfg = magpie.MagpieConfig( + base_model="m", + provider="ollama", + target_rows=10, + quality_filter=False, + ) + with pytest.raises(NotImplementedError, match="v0.69.1"): + magpie.run_magpie(cfg) + + def test_validates_config_type(self) -> None: + with pytest.raises(TypeError): + magpie.run_magpie({"base": "m"}) # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# CLI: `soup data gen magpie` +# ----------------------------------------------------------------------------- + + +class TestMagpieCli: + def test_help(self) -> None: + runner = CliRunner() + result = runner.invoke(app, ["data", "gen-magpie", "--help"]) + assert result.exit_code == 0, result.output + assert "magpie" in result.output.lower() + + def test_plan_only(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + runner = CliRunner() + result = runner.invoke( + app, + [ + "data", + "gen-magpie", + "--base", + "meta-llama/Llama-3.1-8B-Instruct", + "--provider", + "ollama", + "--target", + "10", + "--plan-only", + ], + ) + assert result.exit_code == 0, result.output + assert "ollama" in result.output.lower() or "magpie" in result.output.lower() + + def test_unknown_provider(self) -> None: + runner = CliRunner() + result = runner.invoke( + app, + [ + "data", + "gen-magpie", + "--base", + "m", + "--provider", + "openai", + "--target", + "10", + "--plan-only", + ], + ) + assert result.exit_code == 2 + + def test_invalid_target(self) -> None: + runner = CliRunner() + result = runner.invoke( + app, + [ + "data", + "gen-magpie", + "--base", + "m", + "--provider", + "ollama", + "--target", + "0", + "--plan-only", + ], + ) + assert result.exit_code == 2 + + def test_live_deferred_exits_3(self) -> None: + runner = CliRunner() + result = runner.invoke( + app, + [ + "data", + "gen-magpie", + "--base", + "m", + "--provider", + "ollama", + "--target", + "10", + ], + ) + assert result.exit_code == 3 + assert "0.69.1" in result.output or "deferred" in result.output.lower() + + +# ----------------------------------------------------------------------------- +# Source wiring +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_no_heavy_imports(self) -> None: + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "magpie.py").read_text(encoding="utf-8") + for forbidden in ( + "\nimport torch", + "\nimport transformers", + "\nimport anthropic", + ): + assert forbidden not in src + + def test_cli_registered(self) -> None: + root = Path(__file__).resolve().parent.parent + cli = (root / "soup_cli" / "commands" / "data.py").read_text(encoding="utf-8") + assert "magpie" in cli.lower() or "gen-magpie" in cli + + def test_version_bumped(self) -> None: + from soup_cli import __version__ + + assert __version__ == "0.69.0" diff --git a/tests/test_v0690_part_d.py b/tests/test_v0690_part_d.py new file mode 100644 index 0000000..14b2c25 --- /dev/null +++ b/tests/test_v0690_part_d.py @@ -0,0 +1,416 @@ +"""v0.69.0 Part D — Persona-Hub diversity sampler.""" + +from __future__ import annotations + +import dataclasses +import math +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app +from soup_cli.utils import persona_hub + + +def _write(path: Path, text: str) -> Path: + path.write_text(text, encoding="utf-8") + return path + + +# ----------------------------------------------------------------------------- +# Bundled personas +# ----------------------------------------------------------------------------- + + +class TestBundledPersonas: + def test_at_least_one(self) -> None: + personas = persona_hub.list_bundled_personas() + assert len(personas) >= 5 + assert all(isinstance(p, str) for p in personas) + + def test_bundled_returns_tuple(self) -> None: + personas = persona_hub.list_bundled_personas() + assert isinstance(personas, tuple) + + def test_bundled_immutable(self) -> None: + before = persona_hub.list_bundled_personas() + # tuples are immutable; verify identity stability + no mutator + after = persona_hub.list_bundled_personas() + assert before == after + + +# ----------------------------------------------------------------------------- +# Persona / style validators +# ----------------------------------------------------------------------------- + + +class TestValidatePersona: + def test_happy(self) -> None: + assert persona_hub.validate_persona("a curious student") == "a curious student" + + def test_empty(self) -> None: + with pytest.raises(ValueError): + persona_hub.validate_persona("") + + def test_null_byte(self) -> None: + with pytest.raises(ValueError, match="null"): + persona_hub.validate_persona("a\x00b") + + def test_oversize(self) -> None: + with pytest.raises(ValueError): + persona_hub.validate_persona("p" * 2048) + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + persona_hub.validate_persona(42) + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + persona_hub.validate_persona(True) + + +class TestValidateStyle: + def test_happy(self) -> None: + assert persona_hub.validate_style("formal") == "formal" + + def test_empty(self) -> None: + with pytest.raises(ValueError): + persona_hub.validate_style("") + + def test_oversize(self) -> None: + with pytest.raises(ValueError): + persona_hub.validate_style("x" * 200) + + def test_null_byte(self) -> None: + with pytest.raises(ValueError): + persona_hub.validate_style("formal\x00") + + +# ----------------------------------------------------------------------------- +# Sample matrix +# ----------------------------------------------------------------------------- + + +class TestSamplePersonaMatrix: + def test_happy(self) -> None: + rows = persona_hub.sample_persona_matrix( + prompts=["Explain X"], + personas=["student", "engineer"], + styles=["formal"], + n=4, + seed=0, + ) + assert len(rows) == 4 + assert all("prompt" in r and "persona" in r and "style" in r for r in rows) + + def test_deterministic(self) -> None: + kwargs = dict( + prompts=["P1", "P2"], + personas=["A", "B"], + styles=["s1", "s2"], + n=8, + seed=42, + ) + a = persona_hub.sample_persona_matrix(**kwargs) + b = persona_hub.sample_persona_matrix(**kwargs) + assert a == b + + def test_different_seeds_diverge(self) -> None: + a = persona_hub.sample_persona_matrix( + prompts=["P1", "P2"], + personas=["A", "B"], + styles=["s1", "s2"], + n=10, + seed=1, + ) + b = persona_hub.sample_persona_matrix( + prompts=["P1", "P2"], + personas=["A", "B"], + styles=["s1", "s2"], + n=10, + seed=2, + ) + # With 2x2x2=8 combinations across n=10 samples, the orderings + # should differ for at least one cell. + assert a != b + + def test_empty_prompts_rejected(self) -> None: + with pytest.raises(ValueError): + persona_hub.sample_persona_matrix( + prompts=[], + personas=["A"], + styles=["s"], + n=4, + seed=0, + ) + + def test_empty_personas_rejected(self) -> None: + with pytest.raises(ValueError): + persona_hub.sample_persona_matrix( + prompts=["P"], + personas=[], + styles=["s"], + n=4, + seed=0, + ) + + def test_empty_styles_rejected(self) -> None: + with pytest.raises(ValueError): + persona_hub.sample_persona_matrix( + prompts=["P"], + personas=["A"], + styles=[], + n=4, + seed=0, + ) + + def test_n_zero_rejected(self) -> None: + with pytest.raises(ValueError): + persona_hub.sample_persona_matrix( + prompts=["P"], + personas=["A"], + styles=["s"], + n=0, + seed=0, + ) + + def test_n_overcap(self) -> None: + with pytest.raises(ValueError): + persona_hub.sample_persona_matrix( + prompts=["P"], + personas=["A"], + styles=["s"], + n=persona_hub._MAX_SAMPLES + 1, + seed=0, + ) + + def test_n_bool(self) -> None: + with pytest.raises(TypeError): + persona_hub.sample_persona_matrix( + prompts=["P"], + personas=["A"], + styles=["s"], + n=True, + seed=0, + ) + + def test_seed_bool(self) -> None: + with pytest.raises(TypeError): + persona_hub.sample_persona_matrix( + prompts=["P"], + personas=["A"], + styles=["s"], + n=4, + seed=False, + ) + + def test_per_list_caps(self) -> None: + big = ["x"] * (persona_hub._MAX_LIST_LEN + 1) + with pytest.raises(ValueError): + persona_hub.sample_persona_matrix( + prompts=big, + personas=["A"], + styles=["s"], + n=4, + seed=0, + ) + + +# ----------------------------------------------------------------------------- +# Topic diversity +# ----------------------------------------------------------------------------- + + +class TestComputeTopicDiversity: + def test_high_diversity(self) -> None: + rows = [ + {"text": "cat dog mouse"}, + {"text": "engine wheel car"}, + {"text": "ocean wave fish"}, + ] + score = persona_hub.compute_topic_diversity(rows) + assert 0.0 <= score <= 1.0 + assert score > 0.5 + + def test_low_diversity_repeated_text(self) -> None: + rows = [{"text": "same"} for _ in range(5)] + score = persona_hub.compute_topic_diversity(rows) + # entropy should be near 0 (single token, no variation) + assert score < 0.5 + + def test_empty_returns_zero(self) -> None: + assert persona_hub.compute_topic_diversity([]) == 0.0 + + def test_non_list_rejected(self) -> None: + with pytest.raises(TypeError): + persona_hub.compute_topic_diversity("rows") # type: ignore[arg-type] + + def test_rows_without_text_skipped(self) -> None: + # No text fields anywhere → returns 0.0 (no signal). + rows = [{"unrelated": "x"}, {"y": 1}] + score = persona_hub.compute_topic_diversity(rows) + assert score == 0.0 + + def test_score_finite(self) -> None: + rows = [{"text": "a b c"}, {"text": "c d e"}] + score = persona_hub.compute_topic_diversity(rows) + assert math.isfinite(score) + + +# ----------------------------------------------------------------------------- +# PersonaPlan + factory +# ----------------------------------------------------------------------------- + + +class TestPersonaPlan: + def test_frozen(self) -> None: + plan = persona_hub.PersonaPlan( + prompts=("P",), + personas=("A",), + styles=("s",), + n=4, + seed=0, + ) + with pytest.raises(dataclasses.FrozenInstanceError): + plan.n = 999 # type: ignore[misc] + + def test_tuples_required(self) -> None: + with pytest.raises(TypeError): + persona_hub.PersonaPlan( + prompts=["P"], # type: ignore[arg-type] + personas=("A",), + styles=("s",), + n=4, + seed=0, + ) + + +class TestBuildPersonaPlan: + def test_happy(self) -> None: + plan = persona_hub.build_persona_plan( + prompts=["P"], + personas=["A"], + styles=["s"], + n=4, + seed=0, + ) + assert plan.n == 4 + assert plan.prompts == ("P",) + + def test_validators_propagate(self) -> None: + with pytest.raises(ValueError): + persona_hub.build_persona_plan( + prompts=[], + personas=["A"], + styles=["s"], + n=4, + seed=0, + ) + + +# ----------------------------------------------------------------------------- +# CLI smoke +# ----------------------------------------------------------------------------- + + +class TestPersonaMixCli: + def test_help(self) -> None: + runner = CliRunner() + result = runner.invoke(app, ["data", "persona-mix", "--help"]) + assert result.exit_code == 0, result.output + + def test_writes_output( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + prompts = _write( + tmp_path / "prompts.jsonl", + '{"prompt": "Explain X"}\n{"prompt": "Explain Y"}\n', + ) + out = tmp_path / "out.jsonl" + runner = CliRunner() + result = runner.invoke( + app, + [ + "data", + "persona-mix", + "--prompts", + str(prompts), + "--n", + "4", + "--output", + str(out), + ], + ) + assert result.exit_code == 0, result.output + assert out.exists() + lines = [line for line in out.read_text(encoding="utf-8").splitlines() if line] + assert len(lines) == 4 + + def test_missing_prompts(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + runner = CliRunner() + result = runner.invoke( + app, + [ + "data", + "persona-mix", + "--prompts", + "nope.jsonl", + "--n", + "4", + "--output", + str(tmp_path / "o.jsonl"), + ], + ) + assert result.exit_code != 0 + + def test_outside_cwd_prompts( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + outside = tmp_path / "outside" + outside.mkdir() + _write(outside / "p.jsonl", '{"prompt": "X"}\n') + sub = tmp_path / "sub" + sub.mkdir() + monkeypatch.chdir(sub) + runner = CliRunner() + result = runner.invoke( + app, + [ + "data", + "persona-mix", + "--prompts", + str(outside / "p.jsonl"), + "--n", + "4", + "--output", + str(sub / "out.jsonl"), + ], + ) + assert result.exit_code != 0 + + +# ----------------------------------------------------------------------------- +# Source wiring +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_no_heavy_imports(self) -> None: + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "persona_hub.py").read_text( + encoding="utf-8" + ) + for forbidden in ( + "\nimport torch", + "\nimport transformers", + "\nimport sentence_transformers", + ): + assert forbidden not in src + + def test_version_bumped(self) -> None: + from soup_cli import __version__ + + assert __version__ == "0.69.0" diff --git a/tests/test_v0690_part_e.py b/tests/test_v0690_part_e.py new file mode 100644 index 0000000..ced2a47 --- /dev/null +++ b/tests/test_v0690_part_e.py @@ -0,0 +1,418 @@ +"""v0.69.0 Part E — Brain-rot detector (arXiv 2510.13928).""" + +from __future__ import annotations + +import dataclasses +import json +import math +import os +import sys +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from soup_cli.cli import app +from soup_cli.utils import brain_rot + + +def _write(path: Path, text: str) -> Path: + path.write_text(text, encoding="utf-8") + return path + + +# ----------------------------------------------------------------------------- +# Verdict allowlist +# ----------------------------------------------------------------------------- + + +class TestBrainRotVerdicts: + def test_exact(self) -> None: + assert brain_rot.BRAIN_ROT_VERDICTS == ("OK", "MINOR", "MAJOR") + + def test_immutable(self) -> None: + # Tuples reject item assignment with TypeError. + with pytest.raises(TypeError): + brain_rot.BRAIN_ROT_VERDICTS[0] = "EVIL" # type: ignore[index] + + +class TestClassifyBoundaries: + """TDD review H1 — exact-threshold boundary tests at ± epsilon.""" + + def test_just_below_ok_is_minor(self) -> None: + assert brain_rot.classify_brain_rot(0.8499999) == "MINOR" + + def test_just_below_minor_is_major(self) -> None: + assert brain_rot.classify_brain_rot(0.5999999) == "MAJOR" + + def test_exact_minor_threshold(self) -> None: + assert brain_rot.classify_brain_rot(0.60) == "MINOR" + + def test_exact_ok_threshold(self) -> None: + assert brain_rot.classify_brain_rot(0.85) == "OK" + + +# ----------------------------------------------------------------------------- +# Classify +# ----------------------------------------------------------------------------- + + +class TestClassifyBrainRot: + def test_high_score_is_ok(self) -> None: + # High score = healthy data, low brain-rot. + assert brain_rot.classify_brain_rot(0.9) == "OK" + assert brain_rot.classify_brain_rot(0.85) == "OK" + + def test_mid_score_is_minor(self) -> None: + assert brain_rot.classify_brain_rot(0.70) == "MINOR" + assert brain_rot.classify_brain_rot(0.6) == "MINOR" + + def test_low_score_is_major(self) -> None: + assert brain_rot.classify_brain_rot(0.59) == "MAJOR" + assert brain_rot.classify_brain_rot(0.0) == "MAJOR" + + def test_boundary_thresholds(self) -> None: + # Match v0.26 / v0.56 / v0.65 thresholds: >= 0.85 OK / >= 0.60 MINOR. + assert brain_rot.classify_brain_rot(0.849) == "MINOR" + assert brain_rot.classify_brain_rot(0.599) == "MAJOR" + assert brain_rot.classify_brain_rot(1.0) == "OK" + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + brain_rot.classify_brain_rot(True) + + def test_nan_rejected(self) -> None: + with pytest.raises(ValueError): + brain_rot.classify_brain_rot(float("nan")) + + def test_inf_rejected(self) -> None: + with pytest.raises(ValueError): + brain_rot.classify_brain_rot(float("inf")) + + def test_out_of_range_rejected(self) -> None: + with pytest.raises(ValueError): + brain_rot.classify_brain_rot(1.5) + with pytest.raises(ValueError): + brain_rot.classify_brain_rot(-0.1) + + def test_non_number(self) -> None: + with pytest.raises(TypeError): + brain_rot.classify_brain_rot("0.5") # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# Triviality + popularity heuristics +# ----------------------------------------------------------------------------- + + +class TestScoreTriviality: + def test_substantive_low(self) -> None: + # Long, varied, substantive text → low triviality. + text = ( + "The mitochondrion is the powerhouse of the cell because it " + "converts nutrients into ATP through oxidative phosphorylation." + ) + score = brain_rot.score_triviality(text) + assert 0.0 <= score <= 1.0 + assert score < 0.5 + + def test_trivial_high(self) -> None: + # Short, repetitive, exclamation-heavy → high triviality. + text = "lol!!!! omg!!!! lol omg!!! lol!!!" + score = brain_rot.score_triviality(text) + assert score > 0.5 + + def test_empty_returns_one(self) -> None: + assert brain_rot.score_triviality("") == 1.0 + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + brain_rot.score_triviality(42) # type: ignore[arg-type] + + def test_bool_rejected(self) -> None: + with pytest.raises(TypeError): + brain_rot.score_triviality(True) + + +class TestScorePopularitySignal: + def test_substantive_low(self) -> None: + text = "Detailed scientific explanation of photosynthesis cycles." + score = brain_rot.score_popularity_signal(text) + assert score < 0.5 + + def test_slop_high(self) -> None: + text = "click here for the top 10 you won't believe what happened next" + score = brain_rot.score_popularity_signal(text) + assert score > 0.5 + + def test_empty_zero(self) -> None: + assert brain_rot.score_popularity_signal("") == 0.0 + + def test_non_string(self) -> None: + with pytest.raises(TypeError): + brain_rot.score_popularity_signal(42) # type: ignore[arg-type] + + +# ----------------------------------------------------------------------------- +# Per-row + dataset scoring +# ----------------------------------------------------------------------------- + + +class TestScoreRowBrainRot: + def test_substantive(self) -> None: + row = {"text": "Long substantive paragraph with diverse vocabulary."} + score = brain_rot.score_row_brain_rot(row) + assert 0.0 <= score <= 1.0 + assert score > 0.5 + + def test_slop(self) -> None: + row = {"text": "lol!!!! omg!!! top 10 you won't believe"} + score = brain_rot.score_row_brain_rot(row) + assert score < 0.5 + + def test_missing_text_returns_zero(self) -> None: + # No text signal → score 0.0 (cannot judge → treat as MAJOR). + assert brain_rot.score_row_brain_rot({}) == 0.0 + + def test_non_dict_rejected(self) -> None: + with pytest.raises(TypeError): + brain_rot.score_row_brain_rot("not a dict") # type: ignore[arg-type] + + +class TestBrainRotReport: + def test_frozen(self) -> None: + report = brain_rot.BrainRotReport( + num_rows=10, + mean_score=0.5, + num_major=2, + num_minor=3, + num_ok=5, + overall_verdict="MINOR", + ) + with pytest.raises(dataclasses.FrozenInstanceError): + report.mean_score = 0.9 # type: ignore[misc] + + def test_validates_counts(self) -> None: + with pytest.raises(ValueError): + brain_rot.BrainRotReport( + num_rows=-1, + mean_score=0.5, + num_major=0, + num_minor=0, + num_ok=0, + overall_verdict="OK", + ) + + def test_validates_verdict(self) -> None: + with pytest.raises(ValueError): + brain_rot.BrainRotReport( + num_rows=1, + mean_score=0.5, + num_major=0, + num_minor=0, + num_ok=1, + overall_verdict="BOGUS", + ) + + def test_mean_score_finite(self) -> None: + with pytest.raises(ValueError): + brain_rot.BrainRotReport( + num_rows=1, + mean_score=float("nan"), + num_major=0, + num_minor=0, + num_ok=1, + overall_verdict="OK", + ) + + def test_mean_score_range(self) -> None: + with pytest.raises(ValueError): + brain_rot.BrainRotReport( + num_rows=1, + mean_score=1.5, + num_major=0, + num_minor=0, + num_ok=1, + overall_verdict="OK", + ) + + +class TestScoreDatasetBrainRot: + def test_clean_dataset(self) -> None: + rows = [ + {"text": "Detailed scientific explanation of photosynthesis."}, + {"text": "Comprehensive overview of cellular respiration cycles."}, + ] + report = brain_rot.score_dataset_brain_rot(rows) + assert report.num_rows == 2 + assert math.isfinite(report.mean_score) + assert report.overall_verdict in brain_rot.BRAIN_ROT_VERDICTS + + def test_sloppy_dataset(self) -> None: + rows = [{"text": "lol!!!! omg top 10"} for _ in range(5)] + report = brain_rot.score_dataset_brain_rot(rows) + # Slop-heavy → MAJOR or MINOR + assert report.overall_verdict in ("MAJOR", "MINOR") + + def test_empty(self) -> None: + report = brain_rot.score_dataset_brain_rot([]) + assert report.num_rows == 0 + assert report.mean_score == 0.0 + assert report.overall_verdict == "MAJOR" + + def test_non_iterable(self) -> None: + with pytest.raises(TypeError): + brain_rot.score_dataset_brain_rot(42) # type: ignore[arg-type] + + def test_non_dict_row_skipped(self) -> None: + rows = [ + {"text": "good substantive content"}, + "not a dict", + {"text": "more good content"}, + ] + report = brain_rot.score_dataset_brain_rot(rows) + # Only 2 dict rows count. + assert report.num_rows == 2 + + def test_overall_minor_band(self) -> None: + # Custom mix to land in MINOR band. + rows = ( + [{"text": "lol!!! omg!!!"} for _ in range(3)] + + [ + {"text": "Detailed scientific overview of cellular respiration."} + for _ in range(2) + ] + ) + report = brain_rot.score_dataset_brain_rot(rows) + assert math.isfinite(report.mean_score) + + +# ----------------------------------------------------------------------------- +# refuse_if_rotten +# ----------------------------------------------------------------------------- + + +class TestRefuseIfRotten: + def test_clean_passes(self) -> None: + rows = [ + {"text": "Detailed scientific overview of cellular biology."} + for _ in range(3) + ] + # Should not raise. + brain_rot.refuse_if_rotten(rows, max_major_fraction=0.5) + + def test_too_rotten_raises(self) -> None: + rows = [{"text": "lol!!! omg!!!"} for _ in range(5)] + with pytest.raises(ValueError, match="brain.?rot"): + brain_rot.refuse_if_rotten(rows, max_major_fraction=0.1) + + def test_invalid_threshold(self) -> None: + with pytest.raises(ValueError): + brain_rot.refuse_if_rotten([], max_major_fraction=1.5) + + def test_bool_threshold(self) -> None: + with pytest.raises(TypeError): + brain_rot.refuse_if_rotten([], max_major_fraction=True) + + +# ----------------------------------------------------------------------------- +# CLI: `soup data brain-rot` +# ----------------------------------------------------------------------------- + + +class TestBrainRotCli: + def test_help(self) -> None: + runner = CliRunner() + result = runner.invoke(app, ["data", "brain-rot", "--help"]) + assert result.exit_code == 0, result.output + assert "brain" in result.output.lower() + + def test_clean(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + path = _write( + tmp_path / "clean.jsonl", + "\n".join( + json.dumps({"text": "Detailed scientific exposition number " + str(i)}) + for i in range(5) + ) + + "\n", + ) + runner = CliRunner() + result = runner.invoke(app, ["data", "brain-rot", str(path)]) + assert result.exit_code == 0, result.output + + def test_sloppy_exits_3( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + path = _write( + tmp_path / "slop.jsonl", + "\n".join(json.dumps({"text": "lol!!! omg!!!"}) for _ in range(10)) + + "\n", + ) + runner = CliRunner() + result = runner.invoke( + app, ["data", "brain-rot", str(path), "--strict"] + ) + assert result.exit_code == 3 + + def test_missing_input( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + runner = CliRunner() + result = runner.invoke(app, ["data", "brain-rot", "nope.jsonl"]) + assert result.exit_code != 0 + + def test_outside_cwd( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + outside = tmp_path / "outside" + outside.mkdir() + _write(outside / "d.jsonl", json.dumps({"text": "x"}) + "\n") + sub = tmp_path / "sub" + sub.mkdir() + monkeypatch.chdir(sub) + runner = CliRunner() + result = runner.invoke( + app, ["data", "brain-rot", str(outside / "d.jsonl")] + ) + assert result.exit_code != 0 + + @pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlink") + def test_symlink_rejected( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.chdir(tmp_path) + target = _write( + tmp_path / "real.jsonl", json.dumps({"text": "x"}) + "\n" + ) + link = tmp_path / "link.jsonl" + os.symlink(str(target), str(link)) + runner = CliRunner() + result = runner.invoke(app, ["data", "brain-rot", str(link)]) + assert result.exit_code != 0 + + +# ----------------------------------------------------------------------------- +# Source wiring +# ----------------------------------------------------------------------------- + + +class TestSourceWiring: + def test_no_heavy_imports(self) -> None: + root = Path(__file__).resolve().parent.parent + src = (root / "soup_cli" / "utils" / "brain_rot.py").read_text( + encoding="utf-8" + ) + for forbidden in ( + "\nimport torch", + "\nimport transformers", + "\nimport sentence_transformers", + ): + assert forbidden not in src + + def test_version_bumped(self) -> None: + from soup_cli import __version__ + + assert __version__ == "0.69.0"