mirror of https://github.com/razor-ai/soup.git
feat(v0.69.0): Data Engineering Pro — soup build + expect + gen-magpie + persona-mix + brain-rot
5 parts shipping axis-2 (dbt-for-SFT) + axis-13 (data ops): - soup build — dbt-for-SFT DAG with refs / incremental materialization / content-hash row-diff kernel (run_build live runner deferred → v0.69.1) - soup expect <data> <suite> — LIVE expectations suite: PII / token-length / refusal / chosen-vs-rejected judge; exit 3 on suite failure - soup data gen-magpie — Magpie synthetic generator plan (live → v0.69.1) - soup data persona-mix — Persona-Hub × style sampler with bundled 12×5 set, atomic JSONL write (LIVE) - soup data brain-rot — arXiv 2510.13928 detector with --strict CI gate, worst-signal composite (LIVE) Centralised TOCTOU defence behind utils/paths.enforce_under_cwd_and_no_symlink in build_dag / expectations / expect.py (code-review CRIT — replaces 3 duplicate os.lstat + S_ISLNK + realpath + is_under_cwd blocks). DoS caps on every new JSONL loader (brain-rot 1 GiB + 1M rows; persona-mix 100 MiB + 100k entries). persona-mix --output TOCTOU symlink rejection. magpie quality_filter validator + expectations._dispatch_expectation raw-args pass-through (no int/float coercion bypass). BuildModel seed/derived cross-validator rejects ambiguous shapes at schema load. Review-fix coverage across 4 waves (security + code + python + TDD): 1 CRITICAL + 4 HIGH + 5 MEDIUM + 4 LOW. Test count: 11225 → 11487 (+262 net across 5 new files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alpamys
parent
1d4892ea80
commit
49943a5af6
|
|
@ -111,7 +111,7 @@ soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (257 files, 11225 tests)
|
||||
tests/ - Test suite (262 files, 11487 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
66
README.md
66
README.md
|
|
@ -42,44 +42,51 @@ soup train
|
|||
|
||||
Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
**v0.68.0 — Anti-trend Insurance: `soup compile` (DSPy/GEPA) + `soup distill-prompt` + `soup compile-tools` + `soup apple-adapter` + `soup local-rl` daemon.** Five bets that hedge Soup against paradigm shifts. If 1M-token contexts kill fine-tuning, `soup compile` (DSPy + GEPA prompt-program compilation) takes its place. If teams hit prompt-cost walls, `soup distill-prompt` bridges to small FT. If only Apple Foundation Models win on-device, `soup apple-adapter` ships the converter+signing surface. If personal-LLM flywheels become the shape, `soup local-rl` daemon captures thumbs into SQLite and emits DPO pairs nightly. None of these exist as CLI workflows in the OSS or hosted ecosystem — DSPy/GEPA are libraries, prompt distillation is unowned, Apple FoundationModels adapter format is undocumented territory, personal-LLM flywheels are research-stage. Schema-only release for Parts A/B/C/D — live wiring deferred to v0.68.1. Part E ships LIVE except the nightly train scheduler.
|
||||
**v0.69.0 — Data Engineering Pro: `soup build` (dbt-for-SFT) + `soup expect` + `soup data gen-magpie` + `soup data persona-mix` + `soup data brain-rot`.** Five surfaces that turn dataset preparation into a first-class engineering workflow. `soup build` is a dbt-style DAG of dataset transforms with `ref()`-connected models, `incremental` materialization, and content-hash-based row-diff so re-runs re-tokenize only changed rows. `soup expect` ships Great-Expectations for chat data — `expect_no_pii` / `expect_token_length_between` / `expect_no_refusal_pattern` / `expect_chosen_preferred_over_rejected_by_judge` — with exit code 3 on suite failure so CI pipelines can gate on dataset quality regressions. `soup data gen-magpie` plans the Magpie synthetic generator (chat-template-prefix harvest, reuses v0.20 providers). `soup data persona-mix` samples a prompt × persona × style matrix with a bundled 12-persona / 5-style diversity set + topic-entropy metric. `soup data brain-rot` implements the arXiv 2510.13928 detector with OK/MINOR/MAJOR verdicts that compose with v0.47 educational scorer; `--strict` mode exits 3 when too many rows score MAJOR.
|
||||
|
||||
- **`soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|textgrad|copro|bootstrap_fewshot] [--plan-only]`** — DSPy / GEPA prompt-program compiler. Closed allowlist over the canonical 5 optimisers. `CompilePlan` is frozen and re-validated on construction — direct-construction bugs that bypass `build_compile_plan` still produce loud failures. `CompileResult` rejects NaN / ±Inf scores via `math.isfinite`, bool-as-int, and negative iterations. Live orchestrator (DSPy / GEPA / TextGrad) deferred to v0.68.1; the CLI renders the validated plan today and exits 3 on the deferred path (mirrors v0.50.0 / v0.61.0 stub-then-live cadence).
|
||||
- **`soup distill-prompt --traces <jsonl> --teacher <m> --student <m> --strategy sft|preference|kl`** — Take prompt-heavy traces from GPT-5 / Claude calls and prepare a distillation dataset targeting a small student model. Bridge between prompt-engineering and FT worlds. Composes with v0.70 cross-tokenizer KD when that ships. Schema + path containment + symlink rejection ship now; the teacher/student tokeniser-bridge runner lands in v0.68.1.
|
||||
- **`soup compile-tools <spec.json|yaml> --eval <jsonl> [--optimizer textgrad|gepa]`** — TextGrad / GEPA tool-schema optimiser. Reads OpenAPI / MCP / GraphQL specs and prepares to optimise tool descriptions via textual gradients. Composes with v0.46 Agent Forge — Agent Forge parses the spec, `soup compile-tools` optimises the descriptions. `validate_spec_path` enforces JSON / YAML extension allowlist + cwd containment + symlink rejection.
|
||||
- **`soup apple-adapter <source-dir> --direction hf-to-mlx|mlx-to-hf|hf-to-apple|mlx-to-apple --output <dir> [--sign]`** — Adapter format conversion + optional v0.60 Merkle-root signing. Extends v0.25 MLX backend. `validate_source_adapter` enforces cwd containment + `stat.S_ISDIR` (must be a directory) + symlink rejection. The `--sign` flag flows the converted adapter through v0.60 signing for distribution / sigstore.
|
||||
- **`soup local-rl init / status / record / harvest / train`** — Personal-LLM flywheel daemon. Smaller-scope cousin of v0.58 `soup loop` — runs locally on a single workstation, trains your personal model from your own feedback. **LIVE today**: `init` creates the SQLite schema (`interactions` + `thumbs` with CHECK constraints + `0o600` perms on POSIX); `record` appends a thumbs-up/down row via parameterised query; `status` renders a Rich table of counters; `harvest` walks the thumbs table and emits `{prompt, chosen, rejected}` JSONL pairs (last-writes-win dedup so duplicate thumbs collapse to one pair per prompt). The `train` subcommand (Ollama / MLX × DPO/KTO/ORPO) is deferred to v0.68.1 — operators today can harvest into `dpo_pairs.jsonl` and feed it to existing `soup train --task dpo`.
|
||||
- **+204 new tests** (11021 → 11225) across 6 new test files (5 part files + 1 followups covering every TDD-review finding). Review-fix coverage: 0 CRITICAL + 4 HIGH (validate_student_id rejection-matrix parity / Part C CLI unknown-optimizer exit 2 / harvest_dpo_pairs edge cases / POSIX symlink skip predicate uses `sys.platform`) + 10 MEDIUM (CompileResult ±Inf rejection + bool-score / validate_eval_suite_path symlink test / Part C empty + oversize on tool-optimizer / Part C null-byte + symlink on spec_path / TestBuildToolCompilePlan factory / Part D non-string + oversize on direction / TestBuildAppleAdapterPlan factory / Part E backend + train_method full rejection matrix / record_thumb response null-byte + oversize + empty / SUPPORTED_LOCAL_RL_TRAIN_METHODS immutability) + 4 LOW (TestInitDb column-level schema via PRAGMA / Part D output_dir validation / validate_db_path public + in __all__ / DpoPair frozen). Manual CPU smokes (Step 6): every command's `--help` + happy-path `--plan-only` + deferred-live exit 3 + validation rejection exit 2; full `soup local-rl init → record up → record down → status → harvest` end-to-end producing valid DPO JSONL.
|
||||
- **`soup build <manifest.yaml> [--dry-run]`** — dbt-for-SFT DAG parser + topological sort + plan rendering. Each model declares `kind: incremental|table|view`, a `transform`, and either a `source:` (seed) or `refs: [...]` (derived). Cross-validators reject ambiguous shapes (no-refs + no-source = degenerate; refs + source = mutually exclusive). `compute_row_hash` + `incremental_diff(prev, new) -> IncrementalDiffReport(added, changed, removed, unchanged)` are the kernel that lets the live runner (v0.69.1) re-tokenize only changed rows. `BuildModel` and `BuildPlan` are frozen dataclasses; `load_build_yaml` delegates to `paths.enforce_under_cwd_and_no_symlink` (TOCTOU defence). Live runner deferred to v0.69.1; today `--dry-run` renders the plan + exits 0, no flag exits 3 with the deferred-live marker.
|
||||
- **`soup expect <data.jsonl> <suite.yaml>`** — Great Expectations for chat data. Four built-in expectation functions, each a pure function returning a frozen `ExpectationResult`. Composes with v0.47.0 `data_score.detect_pii` (Presidio backend when `[data-pro]` installed), v0.56.0 `diagnose.refusal.looks_like_refusal`, and v0.19.0 judge backends (operator-injected callable for the chosen-vs-rejected judge). `_dispatch_expectation` passes args through raw so per-expectation validators (bool-rejection, NaN-rejection, range checks) fire authentically — no silent int/float coercion bypasses the validator. Exit 0 = pass, 2 = validation rejection, 3 = suite failure.
|
||||
- **`soup data gen-magpie --base <m> --provider ollama|anthropic|vllm --target N [--plan-only]`** — Magpie technique (Xu et al. 2024) schema + plan. Feeds the chat-template prefix only to an aligned base model and harvests user-side turns via v0.20 providers. `MagpieConfig` frozen, `validate_magpie_provider` closed allowlist, `validate_target_rows` ∈ [1, 1_000_000] bool-rejected. Live generation loop (provider calls + v0.47 quality filter) deferred to v0.69.1.
|
||||
- **`soup data persona-mix --prompts <jsonl> --n N --output <jsonl>`** — Persona-Hub-style diversity sampler. Reads `--prompts` JSONL, multiplies through bundled 12-persona × 5-style matrix (or operator-supplied `--personas` / `--styles` JSONL), writes `{prompt, persona, style}` per row via atomic `tempfile.mkstemp + os.replace`. Deterministic by `--seed`. New `compute_topic_diversity` Shannon-entropy kernel for downstream gating. Centralised TOCTOU policy via `enforce_under_cwd_and_no_symlink` on every read AND write path.
|
||||
- **`soup data brain-rot <data.jsonl> [--strict] [--max-major-fraction 0.25]`** — arXiv 2510.13928 brain-rot detector. Two orthogonal slop scorers: `score_triviality` (low diversity / excessive `!!`/`??` punctuation / `lol/omg/lmao` density / length penalty) and `score_popularity_signal` (clickbait phrase substrings + emoji density). Per-row composite = `1.0 - max(triviality, popularity)`. Same OK/MINOR/MAJOR taxonomy as v0.26 / v0.56 / v0.65 (≥0.85 OK, ≥0.60 MINOR, else MAJOR). `--strict` exits 3 when MAJOR-row fraction exceeds the threshold so training pipelines can refuse to materialise slop datasets. `--max-major-fraction` validated at the CLI boundary (NaN / Inf / out-of-range / bool rejected) before any scoring.
|
||||
- **+264 new tests** (11225 → 11487) across 5 part files. Review-fix coverage: 1 CRITICAL (centralised duplicate cwd+symlink blocks behind `paths.enforce_under_cwd_and_no_symlink` in build_dag / expectations / expect.py — code-review HIGH) + 4 HIGH (brain-rot loader DoS caps; persona-mix `--output` symlink TOCTOU rejection; persona-mix `_load_jsonl_field` DoS caps; magpie `quality_filter` validator + `_dispatch_expectation` int/float-coercion bypass) + 5 MEDIUM (BuildModel seed/derived cross-validator + docs; `--max-major-fraction` CLI-boundary validation; `expect` skipped-line WARNING) + 4 LOW (boundary tests at exact threshold ± ε on classify_brain_rot; BuildPlan FrozenInstanceError; version_bumped checks in B/C/D/E; lazy-yaml import source-grep). Manual CPU smokes (Step 6): every CLI happy + failure path exercised, including 3-stage build DAG dry-run, PII suite exit 3, magpie plan-only, persona-mix atomic write to JSONL, brain-rot MAJOR-on-slop exit 3.
|
||||
|
||||
## Anti-trend Insurance
|
||||
## Data Engineering Pro
|
||||
|
||||
The v0.68.0 release ships 5 commands that hedge Soup against paradigm shifts. Each is one of the 5 "if X kills FT, Soup still survives" bets.
|
||||
The v0.69.0 release ships 5 surfaces that turn dataset prep from "throw a JSONL at the trainer" into a first-class engineering workflow.
|
||||
|
||||
```bash
|
||||
# DSPy / GEPA prompt-program compilation — replaces FT if 1M-context wins
|
||||
soup compile my_program.py --eval evals.json --optimizer mipro --max-iters 10
|
||||
# dbt-for-SFT — DAG of dataset transforms with incremental materialization
|
||||
cat > build.yaml << 'EOF'
|
||||
models:
|
||||
- {name: raw, kind: incremental, source: data/raw.jsonl, transform: identity}
|
||||
- {name: filtered, kind: incremental, refs: [raw], transform: filter_low_quality}
|
||||
- {name: tokenized, kind: incremental, refs: [filtered], transform: tokenize}
|
||||
EOF
|
||||
soup build build.yaml --dry-run # validate topology + plan
|
||||
# soup build build.yaml # live materialise (v0.69.1)
|
||||
|
||||
# Distill prompt-heavy traces into small FT — bridges prompt-eng to FT worlds
|
||||
soup distill-prompt --traces traces.jsonl --teacher anthropic/claude-3-5-sonnet \
|
||||
--student meta-llama/Llama-3.2-1B --strategy sft
|
||||
# Expectations suite — Great Expectations for chat data
|
||||
cat > suite.yaml << 'EOF'
|
||||
expectations:
|
||||
- {name: expect_no_pii}
|
||||
- {name: expect_token_length_between, args: {min_tokens: 16, max_tokens: 4096}}
|
||||
- {name: expect_no_refusal_pattern}
|
||||
EOF
|
||||
soup expect data.jsonl suite.yaml # exit 3 on suite failure
|
||||
|
||||
# Tool-schema textual-gradient optimization — composes with `soup agent` (v0.46)
|
||||
soup compile-tools api_spec.yaml --eval tool_eval.jsonl --optimizer textgrad
|
||||
# Magpie synthetic data — chat-template-prefix harvest (plan, runner v0.69.1)
|
||||
soup data gen-magpie --base meta-llama/Llama-3.1-8B-Instruct \
|
||||
--provider ollama --target 1000 --plan-only
|
||||
|
||||
# HF / MLX / Apple FoundationModels adapter conversion + optional signing
|
||||
soup apple-adapter ./my_adapter --direction hf-to-mlx --output ./mlx_adapter
|
||||
soup apple-adapter ./my_adapter --direction hf-to-apple --output ./apple_adapter --sign
|
||||
# Persona-Hub diversity — prompt × persona × style matrix sampling
|
||||
soup data persona-mix --prompts prompts.jsonl --n 500 --output mixed.jsonl
|
||||
|
||||
# Personal-LLM flywheel — capture thumbs into SQLite, harvest DPO pairs nightly
|
||||
soup local-rl init --db local_rl.db
|
||||
soup local-rl record --db local_rl.db --prompt "What is X?" --response "..." --thumb up
|
||||
soup local-rl record --db local_rl.db --prompt "What is X?" --response "wrong" --thumb down
|
||||
soup local-rl status --db local_rl.db
|
||||
soup local-rl harvest --db local_rl.db -o dpo_pairs.jsonl
|
||||
# soup local-rl train (nightly DPO/KTO/ORPO) — v0.68.1
|
||||
# Brain-rot detector (arXiv 2510.13928) — refuses to train on excessive slop
|
||||
soup data brain-rot data.jsonl --strict --max-major-fraction 0.10
|
||||
```
|
||||
|
||||
Live runners for Parts A/B/C/D + the local-rl nightly scheduler land in v0.68.1. All schema + validation + path containment + symlink-rejection + atomic-write surfaces ship today, so misconfigured commands fail loudly at config load.
|
||||
Every command applies the project-wide TOCTOU policy (`os.lstat + S_ISLNK` symlink rejection before any open) and cwd containment via the shared `paths.enforce_under_cwd_and_no_symlink` helper. Live runners for `soup build` and `soup data gen-magpie` land in v0.69.1; the other three are LIVE today.
|
||||
|
||||
## Why Soup?
|
||||
|
||||
|
|
@ -3768,6 +3775,11 @@ soup local-rl status --db <path> Print interactions / thumbs-up / t
|
|||
soup local-rl record --db <path> --prompt <q> --response <r> --thumb up|down Append thumbs record
|
||||
soup local-rl harvest --db <path> -o <pairs.jsonl> Harvest DPO pairs from thumbs into JSONL
|
||||
soup local-rl train --db <path> --backend ollama|mlx --model <id> Nightly DPO/KTO/ORPO train (v0.68.1)
|
||||
soup build <manifest.yaml> [--dry-run] dbt-for-SFT DAG: validate + plan dataset transforms (v0.69.0)
|
||||
soup expect <data.jsonl> <suite.yaml> Expectations suite: PII / token-length / refusal / judge (v0.69.0)
|
||||
soup data gen-magpie --base <m> --provider ollama|anthropic|vllm --target N [--plan-only] Magpie synthetic generator (v0.69.0)
|
||||
soup data persona-mix --prompts <jsonl> --n N --output <jsonl> Persona-Hub diversity sampler (v0.69.0)
|
||||
soup data brain-rot <data.jsonl> [--strict] Brain-rot detector — arXiv 2510.13928 (v0.69.0)
|
||||
soup version [--full] [--json] Show version (--full: system info, --json: JSON output)
|
||||
soup --verbose <command> Full traceback on errors
|
||||
```
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.68.0"
|
||||
version = "0.69.0"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.68.0"
|
||||
__version__ = "0.69.0"
|
||||
|
|
|
|||
|
|
@ -451,6 +451,22 @@ app.add_typer(
|
|||
help="Personal-LLM flywheel daemon (init / status / record / harvest / train) (v0.68.0).",
|
||||
)
|
||||
|
||||
# v0.69.0 Part A — `soup build` (dbt-for-SFT DAG).
|
||||
from soup_cli.commands import build as _build_cmd # noqa: E402
|
||||
|
||||
app.command(
|
||||
name="build",
|
||||
help="dbt-for-SFT DAG: validate + plan dataset transforms (v0.69.0 Part A).",
|
||||
)(_build_cmd.build_cmd)
|
||||
|
||||
# v0.69.0 Part B — `soup expect` (expectations suite).
|
||||
from soup_cli.commands import expect as _expect_cmd # noqa: E402
|
||||
|
||||
app.command(
|
||||
name="expect",
|
||||
help="Run an expectations suite against a JSONL dataset (v0.69.0 Part B).",
|
||||
)(_expect_cmd.expect_cmd)
|
||||
|
||||
|
||||
def _rewrite_advise_argv(argv: list) -> list:
|
||||
"""Inject `run` between `advise` and a non-subcommand first argument.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,67 @@
|
|||
"""soup build — dbt-for-SFT DAG of dataset transforms (v0.69.0 Part A).
|
||||
|
||||
Reads a YAML manifest, validates the model DAG, prints the topological plan,
|
||||
and (with ``--dry-run``) exits cleanly. The live runner that actually
|
||||
materialises each model is deferred to v0.69.1 — invoking ``soup build`` WITHOUT
|
||||
``--dry-run`` exits with code 3 and a deferred-live advisory (matches v0.61.0 /
|
||||
v0.62.0 / v0.68.0 distinct-exit-code-for-deferred policy).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
def build_cmd(
|
||||
config: str = typer.Argument(..., help="Path to build manifest (YAML)"),
|
||||
dry_run: bool = typer.Option(
|
||||
False,
|
||||
"--dry-run",
|
||||
help="Validate the manifest + print the topological plan; do not execute.",
|
||||
),
|
||||
) -> None:
|
||||
"""Validate a build DAG and (optionally) execute it.
|
||||
|
||||
The live materialiser is deferred to v0.69.1; today only ``--dry-run``
|
||||
produces a meaningful result.
|
||||
"""
|
||||
from soup_cli.utils.build_dag import (
|
||||
load_build_yaml,
|
||||
render_plan_table,
|
||||
run_build,
|
||||
)
|
||||
|
||||
try:
|
||||
plan = load_build_yaml(config)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
rendered = render_plan_table(plan)
|
||||
console.print(
|
||||
Panel(
|
||||
escape(rendered),
|
||||
title=f"soup build — {escape(config)}",
|
||||
)
|
||||
)
|
||||
|
||||
if dry_run:
|
||||
return
|
||||
|
||||
# Live runner deferred to v0.69.1 — same distinct-exit-3 policy as
|
||||
# v0.61.0 / v0.62.0 / v0.68.0 deferred-live stubs.
|
||||
try:
|
||||
run_build(plan)
|
||||
except NotImplementedError as exc:
|
||||
console.print(
|
||||
Panel(
|
||||
f"[yellow]{escape(str(exc))}[/]",
|
||||
title="Live build deferred",
|
||||
)
|
||||
)
|
||||
raise typer.Exit(3) from exc
|
||||
|
|
@ -2464,3 +2464,295 @@ def recipe(
|
|||
console.print(
|
||||
"[dim]Re-run with --execute --output <dir> to run the DAG.[/]"
|
||||
)
|
||||
|
||||
|
||||
# v0.69.0 Part C — Magpie synthetic data generator.
|
||||
@app.command(name="gen-magpie")
|
||||
def gen_magpie(
|
||||
base: str = typer.Option(..., "--base", help="Aligned chat-tuned base model id"),
|
||||
provider: str = typer.Option(..., "--provider", help="ollama | anthropic | vllm"),
|
||||
target: int = typer.Option(..., "--target", help="Target row count [1, 1_000_000]"),
|
||||
quality_filter: bool = typer.Option(
|
||||
True, "--quality-filter/--no-quality-filter", help="Apply v0.47 quality filter"
|
||||
),
|
||||
plan_only: bool = typer.Option(
|
||||
False, "--plan-only", help="Validate + print plan; do not generate."
|
||||
),
|
||||
) -> None:
|
||||
"""Magpie synthetic data generator (v0.69.0 Part C).
|
||||
|
||||
Feeds the chat-template prefix only to ``--base`` and harvests user-side
|
||||
turns via ``--provider``. The live generator is deferred to v0.69.1; today
|
||||
only ``--plan-only`` produces a meaningful result.
|
||||
"""
|
||||
from rich.markup import escape as _escape
|
||||
from rich.panel import Panel
|
||||
|
||||
from soup_cli.utils.magpie import build_magpie_config, run_magpie
|
||||
|
||||
try:
|
||||
cfg = build_magpie_config(
|
||||
base=base,
|
||||
provider=provider,
|
||||
target=target,
|
||||
quality_filter=quality_filter,
|
||||
)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{_escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Base model: [bold]{_escape(cfg.base_model)}[/]\n"
|
||||
f"Provider: [bold]{_escape(cfg.provider)}[/]\n"
|
||||
f"Target rows: [bold]{cfg.target_rows}[/]\n"
|
||||
f"Quality filter: [bold]{cfg.quality_filter}[/]",
|
||||
title="soup data gen-magpie — plan",
|
||||
)
|
||||
)
|
||||
|
||||
if plan_only:
|
||||
return
|
||||
|
||||
try:
|
||||
run_magpie(cfg)
|
||||
except NotImplementedError as exc:
|
||||
console.print(
|
||||
Panel(
|
||||
f"[yellow]{_escape(str(exc))}[/]",
|
||||
title="Live magpie generator deferred",
|
||||
)
|
||||
)
|
||||
raise typer.Exit(3) from exc
|
||||
|
||||
|
||||
# v0.69.0 Part D — Persona-Hub diversity sampler.
|
||||
@app.command(name="persona-mix")
|
||||
def persona_mix(
|
||||
prompts: str = typer.Option(..., "--prompts", help="JSONL file with one prompt per row"),
|
||||
n: int = typer.Option(..., "--n", help="Number of rows to emit [1, 1_000_000]"),
|
||||
output: str = typer.Option(..., "--output", help="Output JSONL path"),
|
||||
personas: str = typer.Option(
|
||||
"", "--personas", help="Optional JSONL of personas (defaults to bundled set)"
|
||||
),
|
||||
styles: str = typer.Option(
|
||||
"", "--styles", help="Optional JSONL of styles (defaults to bundled set)"
|
||||
),
|
||||
seed: int = typer.Option(0, "--seed", help="Deterministic seed"),
|
||||
) -> None:
|
||||
"""Sample a prompt × persona × style matrix (v0.69.0 Part D).
|
||||
|
||||
Reads ``--prompts`` (JSONL with ``prompt`` field per row); writes
|
||||
``--output`` JSONL with ``{prompt, persona, style}`` rows. When
|
||||
``--personas`` / ``--styles`` are omitted the bundled diversity set is used.
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
from rich.markup import escape as _escape
|
||||
from rich.panel import Panel
|
||||
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
from soup_cli.utils.persona_hub import (
|
||||
list_bundled_personas,
|
||||
list_bundled_styles,
|
||||
sample_persona_matrix,
|
||||
)
|
||||
|
||||
max_jsonl_bytes = 100 * 1024 * 1024 # 100 MiB cap on field inputs
|
||||
max_values = 100_000 # cap matches persona_hub._MAX_LIST_LEN
|
||||
|
||||
def _load_jsonl_field(path: str, field: str) -> list:
|
||||
# Delegate to centralised TOCTOU helper (v0.59.0).
|
||||
enforce_under_cwd_and_no_symlink(path, f"--{field}s path")
|
||||
if not os.path.lexists(path):
|
||||
raise FileNotFoundError(path)
|
||||
real = os.path.realpath(path)
|
||||
if not os.path.isfile(real):
|
||||
raise FileNotFoundError(real)
|
||||
if os.path.getsize(real) > max_jsonl_bytes:
|
||||
raise ValueError(
|
||||
f"--{field}s file exceeds {max_jsonl_bytes} bytes"
|
||||
)
|
||||
values: list = []
|
||||
with open(real, "r", encoding="utf-8") as handle:
|
||||
for raw_line in handle:
|
||||
stripped = raw_line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
if len(values) >= max_values:
|
||||
raise ValueError(
|
||||
f"--{field}s file exceeds {max_values} entries"
|
||||
)
|
||||
try:
|
||||
row = json.loads(stripped)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
val = row.get(field) if isinstance(row, dict) else None
|
||||
if isinstance(val, str) and val:
|
||||
values.append(val)
|
||||
return values
|
||||
|
||||
try:
|
||||
prompt_list = _load_jsonl_field(prompts, "prompt")
|
||||
if not prompt_list:
|
||||
raise ValueError("prompts file produced no rows with a 'prompt' field")
|
||||
persona_list = (
|
||||
_load_jsonl_field(personas, "persona")
|
||||
if personas
|
||||
else list(list_bundled_personas())
|
||||
)
|
||||
if not persona_list:
|
||||
raise ValueError("personas file produced no rows with a 'persona' field")
|
||||
style_list = (
|
||||
_load_jsonl_field(styles, "style")
|
||||
if styles
|
||||
else list(list_bundled_styles())
|
||||
)
|
||||
if not style_list:
|
||||
raise ValueError("styles file produced no rows with a 'style' field")
|
||||
|
||||
# Output containment + TOCTOU symlink rejection at the write target.
|
||||
enforce_under_cwd_and_no_symlink(output, "--output")
|
||||
rows = sample_persona_matrix(
|
||||
prompts=prompt_list,
|
||||
personas=persona_list,
|
||||
styles=style_list,
|
||||
n=n,
|
||||
seed=seed,
|
||||
)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{_escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
# Atomic write via tempfile + os.replace.
|
||||
parent = os.path.dirname(os.path.realpath(output)) or "."
|
||||
fd, tmp_path = tempfile.mkstemp(prefix=".soup.", suffix=".tmp", dir=parent)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
for row in rows:
|
||||
handle.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
os.replace(tmp_path, output)
|
||||
except OSError:
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
console.print(
|
||||
Panel(
|
||||
f"Rows written: [bold]{len(rows)}[/]\n"
|
||||
f"Output: [bold]{_escape(output)}[/]",
|
||||
title="soup data persona-mix",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# v0.69.0 Part E — Brain-rot detector.
|
||||
@app.command(name="brain-rot")
|
||||
def brain_rot_cmd(
|
||||
data: str = typer.Argument(..., help="Path to JSONL dataset"),
|
||||
strict: bool = typer.Option(
|
||||
False, "--strict", help="Exit 3 on MAJOR verdict (CI gate mode)"
|
||||
),
|
||||
max_major_fraction: float = typer.Option(
|
||||
0.25, "--max-major-fraction", help="Strict-mode MAJOR-row fraction cap [0, 1]"
|
||||
),
|
||||
) -> None:
|
||||
"""Score a dataset for brain-rot per arXiv 2510.13928 (v0.69.0 Part E).
|
||||
|
||||
Reports a per-row OK/MINOR/MAJOR verdict + an aggregate verdict. With
|
||||
``--strict`` the command exits 3 when the MAJOR fraction exceeds
|
||||
``--max-major-fraction`` (default 25%), so CI pipelines can refuse
|
||||
training on excessive slop.
|
||||
"""
|
||||
import math as _math
|
||||
import os
|
||||
|
||||
from rich.markup import escape as _escape
|
||||
from rich.table import Table
|
||||
|
||||
from soup_cli.utils.brain_rot import (
|
||||
refuse_if_rotten,
|
||||
score_dataset_brain_rot,
|
||||
)
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
max_brain_rot_bytes = 1_073_741_824 # 1 GiB
|
||||
max_brain_rot_rows = 1_000_000
|
||||
|
||||
# Validate --max-major-fraction at the CLI boundary so a flag of NaN /
|
||||
# inf / bool / out-of-range is rejected BEFORE the heavy scoring pass
|
||||
# (security review M2).
|
||||
if isinstance(max_major_fraction, bool):
|
||||
console.print("[red]--max-major-fraction must be a number, not bool[/]")
|
||||
raise typer.Exit(2)
|
||||
if not isinstance(max_major_fraction, (int, float)):
|
||||
console.print("[red]--max-major-fraction must be a number[/]")
|
||||
raise typer.Exit(2)
|
||||
if not _math.isfinite(float(max_major_fraction)):
|
||||
console.print("[red]--max-major-fraction must be finite[/]")
|
||||
raise typer.Exit(2)
|
||||
if not (0.0 <= float(max_major_fraction) <= 1.0):
|
||||
console.print("[red]--max-major-fraction must be in [0.0, 1.0][/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
if not isinstance(data, str) or not data:
|
||||
console.print("[red]data path must be a non-empty string[/]")
|
||||
raise typer.Exit(2)
|
||||
try:
|
||||
enforce_under_cwd_and_no_symlink(data, "data path")
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{_escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
if not os.path.lexists(data):
|
||||
console.print(f"[red]data file not found: {_escape(data)}[/]")
|
||||
raise typer.Exit(2)
|
||||
real = os.path.realpath(data)
|
||||
if not os.path.isfile(real):
|
||||
console.print(f"[red]data file not found: {_escape(real)}[/]")
|
||||
raise typer.Exit(2)
|
||||
if os.path.getsize(real) > max_brain_rot_bytes:
|
||||
console.print(
|
||||
f"[red]data file exceeds {max_brain_rot_bytes} bytes[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
|
||||
rows: list = []
|
||||
with open(real, "r", encoding="utf-8") as handle:
|
||||
for line_no, raw_line in enumerate(handle, start=1):
|
||||
stripped = raw_line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
if line_no > max_brain_rot_rows:
|
||||
console.print(
|
||||
f"[red]data file exceeds {max_brain_rot_rows} rows[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
try:
|
||||
row = json.loads(stripped)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(row, dict):
|
||||
rows.append(row)
|
||||
|
||||
report = score_dataset_brain_rot(rows)
|
||||
|
||||
table = Table(title="Brain-rot report")
|
||||
table.add_column("Metric")
|
||||
table.add_column("Value")
|
||||
table.add_row("Rows scored", str(report.num_rows))
|
||||
table.add_row("Mean score", f"{report.mean_score:.3f}")
|
||||
table.add_row("Verdict OK", str(report.num_ok))
|
||||
table.add_row("Verdict MINOR", str(report.num_minor))
|
||||
table.add_row("Verdict MAJOR", str(report.num_major))
|
||||
table.add_row("Overall", report.overall_verdict)
|
||||
console.print(table)
|
||||
|
||||
if strict:
|
||||
try:
|
||||
refuse_if_rotten(rows, max_major_fraction=max_major_fraction)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{_escape(str(exc))}[/]")
|
||||
raise typer.Exit(3) from exc
|
||||
|
|
|
|||
|
|
@ -0,0 +1,118 @@
|
|||
"""soup expect — Expectations suite for chat data (v0.69.0 Part B).
|
||||
|
||||
Runs a YAML suite of expectations against a JSONL dataset. Exit code 3 on
|
||||
suite failure (matches v0.55 / v0.56 / v0.64 / v0.65 gate convention) so CI
|
||||
pipelines can gate on dataset quality regressions without parsing output.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import List, Mapping
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
|
||||
console = Console()
|
||||
|
||||
_MAX_DATA_BYTES = 1_073_741_824 # 1 GiB cap on input data
|
||||
_MAX_DATA_ROWS = 1_000_000
|
||||
|
||||
|
||||
def _load_jsonl_rows(data_path: str) -> List[Mapping[str, object]]:
|
||||
"""Cwd-contained + symlink-rejected JSONL loader.
|
||||
|
||||
Delegates to ``utils.paths.enforce_under_cwd_and_no_symlink`` (v0.59.0
|
||||
centralised TOCTOU helper) for the path-validation pre-check.
|
||||
"""
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
if not isinstance(data_path, str) or not data_path:
|
||||
raise ValueError("data path must be a non-empty string")
|
||||
if not os.path.lexists(data_path):
|
||||
raise FileNotFoundError(data_path)
|
||||
enforce_under_cwd_and_no_symlink(data_path, "data path")
|
||||
real = os.path.realpath(data_path)
|
||||
if not os.path.isfile(real):
|
||||
raise FileNotFoundError(real)
|
||||
if os.path.getsize(real) > _MAX_DATA_BYTES:
|
||||
raise ValueError(f"data file exceeds {_MAX_DATA_BYTES} bytes")
|
||||
rows: List[Mapping[str, object]] = []
|
||||
skipped = 0
|
||||
with open(real, "r", encoding="utf-8") as handle:
|
||||
for line_no, line in enumerate(handle, start=1):
|
||||
if not line.strip():
|
||||
continue
|
||||
if line_no > _MAX_DATA_ROWS:
|
||||
raise ValueError(f"data file exceeds {_MAX_DATA_ROWS} rows")
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
skipped += 1
|
||||
continue
|
||||
if isinstance(row, dict):
|
||||
rows.append(row)
|
||||
if skipped:
|
||||
console.print(
|
||||
f"[yellow]Note: skipped {skipped} malformed JSONL line(s)[/]"
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def expect_cmd(
|
||||
data: str = typer.Argument(..., help="Path to JSONL dataset"),
|
||||
suite: str = typer.Argument(..., help="Path to expectations suite YAML"),
|
||||
) -> None:
|
||||
"""Run an expectations suite against a JSONL dataset.
|
||||
|
||||
Exit 0 = suite passed. Exit 2 = validation rejection. Exit 3 = suite failed.
|
||||
"""
|
||||
from soup_cli.utils.expectations import load_suite_yaml, run_suite
|
||||
|
||||
try:
|
||||
spec = load_suite_yaml(suite)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
try:
|
||||
rows = _load_jsonl_rows(data)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
try:
|
||||
report = run_suite(rows, spec)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
table = Table(title=f"soup expect — {escape(data)}")
|
||||
table.add_column("Expectation")
|
||||
table.add_column("Passed")
|
||||
table.add_column("Rows")
|
||||
table.add_column("Violations")
|
||||
for result in report.results:
|
||||
verdict = "[green]PASS[/]" if result.passed else "[red]FAIL[/]"
|
||||
table.add_row(
|
||||
escape(result.name),
|
||||
verdict,
|
||||
str(result.num_rows_checked),
|
||||
str(result.num_violations),
|
||||
)
|
||||
console.print(table)
|
||||
|
||||
if not report.passed:
|
||||
for result in report.results:
|
||||
if not result.passed and result.details:
|
||||
console.print(
|
||||
Panel(
|
||||
"\n".join(escape(d) for d in result.details),
|
||||
title=f"[red]Violations: {escape(result.name)}[/]",
|
||||
)
|
||||
)
|
||||
raise typer.Exit(3)
|
||||
|
|
@ -0,0 +1,297 @@
|
|||
"""v0.69.0 Part E — Brain-rot detector (arXiv 2510.13928).
|
||||
|
||||
Scores dataset rows on two orthogonal "low-quality slop" axes:
|
||||
|
||||
- **Triviality** — short / repetitive / exclamation-heavy text dominates → high.
|
||||
- **Popularity signal** — clickbait phrases ("you won't believe", "top 10") +
|
||||
excessive punctuation / emoji density → high.
|
||||
|
||||
Per-row brain-rot score is in [0, 1] where ``1.0`` = healthy substantive content
|
||||
and ``0.0`` = pure slop (so we can reuse the OK/MINOR/MAJOR taxonomy from
|
||||
v0.26 / v0.56 / v0.65 with the same threshold band — high score = good).
|
||||
|
||||
Composes with v0.47.0 ``score_educational_value`` (educational-value scorer),
|
||||
which the live runner can mix in for a composite score; this module ships the
|
||||
slop-detection kernel that the paper measures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Iterable, List, Mapping
|
||||
|
||||
BRAIN_ROT_VERDICTS = ("OK", "MINOR", "MAJOR")
|
||||
|
||||
_OK_THRESHOLD = 0.85
|
||||
_MINOR_THRESHOLD = 0.60
|
||||
|
||||
_MAX_TEXT_LEN = 65_536
|
||||
_CLICKBAIT_PHRASES = (
|
||||
"you won't believe",
|
||||
"you wont believe",
|
||||
"won't believe what happened",
|
||||
"top 10",
|
||||
"top ten",
|
||||
"click here",
|
||||
"this one weird trick",
|
||||
"what happened next",
|
||||
"the rest is history",
|
||||
"shocked the world",
|
||||
"doctors hate",
|
||||
"gone wrong",
|
||||
"gone viral",
|
||||
)
|
||||
_LOW_EFFORT_TOKENS = ("lol", "omg", "lmao", "rofl", "smh", "tbh", "idk")
|
||||
_PUNCT_PATTERN = re.compile(r"[!]{2,}|[?]{2,}")
|
||||
_TEXT_FIELDS = ("text", "content", "output", "prompt", "instruction", "response")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BrainRotReport:
|
||||
"""Outcome of scoring a dataset for brain-rot."""
|
||||
|
||||
num_rows: int
|
||||
mean_score: float
|
||||
num_major: int
|
||||
num_minor: int
|
||||
num_ok: int
|
||||
overall_verdict: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for field_name in ("num_rows", "num_major", "num_minor", "num_ok"):
|
||||
value = getattr(self, field_name)
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise TypeError(
|
||||
f"BrainRotReport.{field_name} must be int"
|
||||
)
|
||||
if value < 0:
|
||||
raise ValueError(
|
||||
f"BrainRotReport.{field_name} must be non-negative"
|
||||
)
|
||||
if isinstance(self.mean_score, bool):
|
||||
raise TypeError("BrainRotReport.mean_score must be float")
|
||||
if not isinstance(self.mean_score, (int, float)):
|
||||
raise TypeError("BrainRotReport.mean_score must be a number")
|
||||
if not math.isfinite(float(self.mean_score)):
|
||||
raise ValueError("BrainRotReport.mean_score must be finite")
|
||||
if not (0.0 <= float(self.mean_score) <= 1.0):
|
||||
raise ValueError(
|
||||
"BrainRotReport.mean_score must be in [0.0, 1.0]"
|
||||
)
|
||||
if self.overall_verdict not in BRAIN_ROT_VERDICTS:
|
||||
raise ValueError(
|
||||
f"overall_verdict must be one of {BRAIN_ROT_VERDICTS}"
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Classifier
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def classify_brain_rot(score: object) -> str:
|
||||
"""Classify a score in [0, 1] (1.0 = healthy) into OK / MINOR / MAJOR.
|
||||
|
||||
Mirrors v0.26.0 Quant-Lobotomy / v0.56.0 diagnose / v0.65.0 behavior taxonomy:
|
||||
``>= 0.85 → OK``, ``>= 0.60 → MINOR``, else ``MAJOR``.
|
||||
"""
|
||||
if isinstance(score, bool):
|
||||
raise TypeError("score must be float, not bool")
|
||||
if not isinstance(score, (int, float)):
|
||||
raise TypeError(
|
||||
f"score must be a number, got {type(score).__name__}"
|
||||
)
|
||||
fscore = float(score)
|
||||
if not math.isfinite(fscore):
|
||||
raise ValueError("score must be finite")
|
||||
if not (0.0 <= fscore <= 1.0):
|
||||
raise ValueError("score must be in [0.0, 1.0]")
|
||||
if fscore >= _OK_THRESHOLD:
|
||||
return "OK"
|
||||
if fscore >= _MINOR_THRESHOLD:
|
||||
return "MINOR"
|
||||
return "MAJOR"
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Heuristic scorers
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _require_str(text: object, *, field: str = "text") -> str:
|
||||
if isinstance(text, bool):
|
||||
raise TypeError(f"{field} must be str, not bool")
|
||||
if not isinstance(text, str):
|
||||
raise TypeError(
|
||||
f"{field} must be str, got {type(text).__name__}"
|
||||
)
|
||||
if len(text) > _MAX_TEXT_LEN:
|
||||
return text[:_MAX_TEXT_LEN]
|
||||
return text
|
||||
|
||||
|
||||
def score_triviality(text: object) -> float:
|
||||
"""Higher = more trivial / repetitive / exclamation-heavy.
|
||||
|
||||
Heuristic: punctuation-runs density + short-text penalty + token diversity
|
||||
inversion. Returns 1.0 for empty/unparseable input (worst case).
|
||||
"""
|
||||
s = _require_str(text)
|
||||
if not s.strip():
|
||||
return 1.0
|
||||
tokens = s.lower().split()
|
||||
n = len(tokens)
|
||||
if n == 0:
|
||||
return 1.0
|
||||
unique = len(set(tokens))
|
||||
diversity = unique / n # 1.0 = all unique, 0.0 = pathological
|
||||
# Length penalty: very short text is suspect for SFT.
|
||||
length_penalty = 1.0 if n >= 30 else (1.0 - n / 30.0)
|
||||
# Punctuation: long !!!! / ???? runs are slop markers.
|
||||
punct_hits = len(_PUNCT_PATTERN.findall(s))
|
||||
punct_density = min(1.0, punct_hits / max(1, n / 10))
|
||||
# Low-effort token density.
|
||||
low_effort = sum(1 for tok in tokens if tok.strip("!?.,") in _LOW_EFFORT_TOKENS)
|
||||
low_effort_density = min(1.0, low_effort / max(1, n / 5))
|
||||
triviality = (
|
||||
0.2 * (1.0 - diversity)
|
||||
+ 0.1 * length_penalty
|
||||
+ 0.3 * punct_density
|
||||
+ 0.4 * low_effort_density
|
||||
)
|
||||
return max(0.0, min(1.0, triviality))
|
||||
|
||||
|
||||
def score_popularity_signal(text: object) -> float:
|
||||
"""Higher = clickbait / engagement-bait / popularity-optimised slop.
|
||||
|
||||
Heuristic: substring scan against ``_CLICKBAIT_PHRASES`` + emoji density.
|
||||
Returns 0.0 for empty input.
|
||||
"""
|
||||
s = _require_str(text)
|
||||
if not s.strip():
|
||||
return 0.0
|
||||
lower = s.lower()
|
||||
hits = sum(1 for phrase in _CLICKBAIT_PHRASES if phrase in lower)
|
||||
# Emoji density: count non-ASCII chars in [U+1F300, U+1FAFF] range
|
||||
# (covers most pictographs without importing emoji libs).
|
||||
emoji_hits = sum(1 for c in s if 0x1F300 <= ord(c) <= 0x1FAFF)
|
||||
n_tokens = len(s.split()) or 1
|
||||
phrase_density = min(1.0, hits / 2.0) # 2+ clickbait phrases → max
|
||||
emoji_density = min(1.0, emoji_hits / max(1, n_tokens / 5))
|
||||
return max(0.0, min(1.0, 0.7 * phrase_density + 0.3 * emoji_density))
|
||||
|
||||
|
||||
def _row_text(row: Mapping[str, Any]) -> str:
|
||||
parts: List[str] = []
|
||||
for key in _TEXT_FIELDS:
|
||||
val = row.get(key)
|
||||
if isinstance(val, str) and val:
|
||||
parts.append(val)
|
||||
messages = row.get("messages")
|
||||
if isinstance(messages, list):
|
||||
for msg in messages:
|
||||
if isinstance(msg, Mapping):
|
||||
content = msg.get("content")
|
||||
if isinstance(content, str) and content:
|
||||
parts.append(content)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def score_row_brain_rot(row: Any) -> float:
|
||||
"""Return a per-row score in [0, 1]; 1.0 = healthy, 0.0 = pure slop.
|
||||
|
||||
Composite: ``1 - 0.5 * triviality - 0.5 * popularity_signal``. Rows with
|
||||
no text fields return ``0.0`` (treat unjudgeable as worst-case).
|
||||
"""
|
||||
if not isinstance(row, Mapping):
|
||||
raise TypeError(
|
||||
f"row must be a Mapping, got {type(row).__name__}"
|
||||
)
|
||||
text = _row_text(row)
|
||||
if not text:
|
||||
return 0.0
|
||||
triviality = score_triviality(text)
|
||||
popularity = score_popularity_signal(text)
|
||||
# Worst-signal composite: a single strong slop signal drives the score
|
||||
# down hard (mirrors v0.56.0 ``overall_verdict`` worst-case policy).
|
||||
score = 1.0 - max(triviality, popularity)
|
||||
return max(0.0, min(1.0, score))
|
||||
|
||||
|
||||
def score_dataset_brain_rot(rows: Any) -> BrainRotReport:
|
||||
"""Score a dataset and return a frozen ``BrainRotReport``.
|
||||
|
||||
Empty inputs return ``MAJOR`` (no signal = treat as broken).
|
||||
"""
|
||||
if isinstance(rows, (str, bytes)) or not hasattr(rows, "__iter__"):
|
||||
raise TypeError("rows must be iterable")
|
||||
scores: List[float] = []
|
||||
for row in rows:
|
||||
if not isinstance(row, Mapping):
|
||||
continue
|
||||
scores.append(score_row_brain_rot(row))
|
||||
if not scores:
|
||||
return BrainRotReport(
|
||||
num_rows=0,
|
||||
mean_score=0.0,
|
||||
num_major=0,
|
||||
num_minor=0,
|
||||
num_ok=0,
|
||||
overall_verdict="MAJOR",
|
||||
)
|
||||
verdict_counts: Counter = Counter(classify_brain_rot(s) for s in scores)
|
||||
mean = sum(scores) / len(scores)
|
||||
overall = classify_brain_rot(mean)
|
||||
return BrainRotReport(
|
||||
num_rows=len(scores),
|
||||
mean_score=mean,
|
||||
num_major=verdict_counts.get("MAJOR", 0),
|
||||
num_minor=verdict_counts.get("MINOR", 0),
|
||||
num_ok=verdict_counts.get("OK", 0),
|
||||
overall_verdict=overall,
|
||||
)
|
||||
|
||||
|
||||
def refuse_if_rotten(
|
||||
rows: Iterable[Mapping[str, Any]],
|
||||
*,
|
||||
max_major_fraction: float = 0.25,
|
||||
) -> None:
|
||||
"""Raise ``ValueError`` when too many rows score MAJOR brain-rot.
|
||||
|
||||
Composes with v0.69.0 Part A's build pipeline so a transform can refuse to
|
||||
produce a tokenised dataset that's mostly slop.
|
||||
"""
|
||||
if isinstance(max_major_fraction, bool):
|
||||
raise TypeError("max_major_fraction must be float, not bool")
|
||||
if not isinstance(max_major_fraction, (int, float)):
|
||||
raise TypeError("max_major_fraction must be a number")
|
||||
if not math.isfinite(float(max_major_fraction)):
|
||||
raise ValueError("max_major_fraction must be finite")
|
||||
if not (0.0 <= float(max_major_fraction) <= 1.0):
|
||||
raise ValueError("max_major_fraction must be in [0.0, 1.0]")
|
||||
report = score_dataset_brain_rot(rows)
|
||||
if report.num_rows == 0:
|
||||
return # no data → nothing to refuse
|
||||
fraction = report.num_major / report.num_rows
|
||||
if fraction > float(max_major_fraction):
|
||||
raise ValueError(
|
||||
f"brain-rot threshold exceeded: {fraction:.1%} of rows are MAJOR "
|
||||
f"(limit {float(max_major_fraction):.1%})"
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BRAIN_ROT_VERDICTS",
|
||||
"BrainRotReport",
|
||||
"classify_brain_rot",
|
||||
"refuse_if_rotten",
|
||||
"score_dataset_brain_rot",
|
||||
"score_popularity_signal",
|
||||
"score_row_brain_rot",
|
||||
"score_triviality",
|
||||
]
|
||||
|
|
@ -0,0 +1,475 @@
|
|||
"""v0.69.0 Part A — `soup build` (dbt-for-SFT DAG).
|
||||
|
||||
Parses a YAML manifest describing a DAG of dataset transforms (``ref``-connected
|
||||
models with ``incremental`` / ``table`` / ``view`` materialization). Validates
|
||||
topology via Kahn's algorithm and renders the dry-run plan. The live SQL/Python
|
||||
runner (re-tokenize-only-diff rows materialization) is deferred to v0.69.1.
|
||||
|
||||
Mirrors the shape of v0.45.0 Part E `recipe_dag.py` so operators familiar with
|
||||
the recipe DAG surface have one mental model. Differs in that build-DAG models
|
||||
also carry a *transform* identifier and a *source* path, and the DAG edges are
|
||||
derived from each model's ``refs: [<other-model>]`` field (rather than a
|
||||
top-level ``edges`` list) — matching dbt's mental model.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Any, Iterable, List, Mapping, Optional, Sequence, Tuple
|
||||
|
||||
# Closed allowlist of model kinds. Mirrors dbt's `materialized` field but
|
||||
# trimmed to the three that map onto SFT-data pipelines.
|
||||
_SUPPORTED_MODEL_KINDS = ("incremental", "table", "view")
|
||||
SUPPORTED_MODEL_KINDS: frozenset = frozenset(_SUPPORTED_MODEL_KINDS)
|
||||
|
||||
# Per-build-plan caps — defence-in-depth against pathological YAML.
|
||||
_MAX_MODELS = 256
|
||||
_MAX_REFS_PER_MODEL = 32
|
||||
_MAX_NAME_LEN = 128
|
||||
_MAX_KIND_LEN = 64
|
||||
_MAX_TRANSFORM_LEN = 256
|
||||
_MAX_SOURCE_LEN = 4096
|
||||
_MAX_FILE_BYTES = 1_048_576 # 1 MiB
|
||||
|
||||
_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._\-]{0,127}$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BuildModel:
|
||||
"""One node in the build DAG.
|
||||
|
||||
``transform`` identifies the materialization function (e.g. ``identity``,
|
||||
``filter_low_quality``, ``tokenize``) the live runner will resolve. The
|
||||
schema does not validate the function reference — the runner does.
|
||||
|
||||
Field semantics:
|
||||
|
||||
- ``refs=()`` + ``source`` set → *seed* model (reads from disk).
|
||||
- ``refs=(...)`` + ``source=None`` → *derived* model (consumes upstream models).
|
||||
- ``refs=()`` + ``source=None`` → degenerate; rejected by cross-validator.
|
||||
- ``refs=(...)`` + ``source`` set → ambiguous; rejected by cross-validator.
|
||||
|
||||
The ``source`` path is validated for shape (non-empty, null-byte-free,
|
||||
length-capped) at schema-load. Operators wanting cwd containment on
|
||||
source paths get it via the v0.69.1 live runner — the schema permits
|
||||
relative paths so a build can be planned offline before the data lands.
|
||||
"""
|
||||
|
||||
name: str
|
||||
kind: str
|
||||
transform: str
|
||||
refs: Tuple[str, ...]
|
||||
source: Optional[str]
|
||||
config: Mapping[str, Any]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Re-validate so callers bypassing ``parse_build_plan`` cannot smuggle
|
||||
# an inconsistent BuildModel through direct construction.
|
||||
validate_model_name(self.name)
|
||||
validate_model_kind(self.kind)
|
||||
if not isinstance(self.refs, tuple):
|
||||
raise TypeError("BuildModel.refs must be a tuple, not list/sequence")
|
||||
if isinstance(self.transform, bool) or not isinstance(self.transform, str):
|
||||
raise TypeError("BuildModel.transform must be a string")
|
||||
if "\x00" in self.transform:
|
||||
raise ValueError("BuildModel.transform must not contain null bytes")
|
||||
if len(self.transform) > _MAX_TRANSFORM_LEN:
|
||||
raise ValueError(
|
||||
f"BuildModel.transform must be <= {_MAX_TRANSFORM_LEN} chars"
|
||||
)
|
||||
if self.source is not None:
|
||||
if isinstance(self.source, bool) or not isinstance(self.source, str):
|
||||
raise TypeError("BuildModel.source must be a string or None")
|
||||
if "\x00" in self.source:
|
||||
raise ValueError("BuildModel.source must not contain null bytes")
|
||||
if len(self.source) > _MAX_SOURCE_LEN:
|
||||
raise ValueError(
|
||||
f"BuildModel.source must be <= {_MAX_SOURCE_LEN} chars"
|
||||
)
|
||||
# Cross-validator: seed/derived shape must be unambiguous.
|
||||
if not self.refs and self.source is None:
|
||||
raise ValueError(
|
||||
f"BuildModel {self.name!r}: a model with no refs must declare a "
|
||||
"'source' (or add refs to make it derived)"
|
||||
)
|
||||
if self.refs and self.source is not None:
|
||||
raise ValueError(
|
||||
f"BuildModel {self.name!r}: a model with refs cannot also declare "
|
||||
"'source' (refs and source are mutually exclusive)"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BuildPlan:
|
||||
"""Validated build DAG with topologically sorted models."""
|
||||
|
||||
models: Tuple[BuildModel, ...]
|
||||
topo_order: Tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IncrementalDiffReport:
|
||||
"""Outcome of comparing previous + new rows for an incremental model.
|
||||
|
||||
``added``: rows present in ``new`` but not ``prev``.
|
||||
``changed``: rows whose ``id`` exists in both but whose content hash differs.
|
||||
``removed``: rows present in ``prev`` but absent in ``new``.
|
||||
``unchanged``: rows with identical ``id`` and content hash in both.
|
||||
"""
|
||||
|
||||
added: int
|
||||
changed: int
|
||||
removed: int
|
||||
unchanged: int
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Validators
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_model_kind(kind: object) -> str:
|
||||
"""Return the canonical lower-case kind. Raise ValueError on unknown."""
|
||||
if isinstance(kind, bool) or not isinstance(kind, str):
|
||||
raise TypeError(
|
||||
f"model kind must be str, got {type(kind).__name__}"
|
||||
)
|
||||
if not kind:
|
||||
raise ValueError("model kind must be non-empty")
|
||||
if "\x00" in kind:
|
||||
raise ValueError("model kind must not contain null bytes")
|
||||
if len(kind) > _MAX_KIND_LEN:
|
||||
raise ValueError(f"model kind must be <= {_MAX_KIND_LEN} chars")
|
||||
canonical = kind.strip().lower()
|
||||
if canonical not in SUPPORTED_MODEL_KINDS:
|
||||
raise ValueError(
|
||||
f"unknown model kind: {kind!r}. supported: {sorted(SUPPORTED_MODEL_KINDS)}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def validate_model_name(name: object) -> str:
|
||||
"""Validate the model identifier. Returns the canonical name."""
|
||||
if isinstance(name, bool) or not isinstance(name, str):
|
||||
raise TypeError(
|
||||
f"model name must be str, got {type(name).__name__}"
|
||||
)
|
||||
if not name:
|
||||
raise ValueError("model name must be non-empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("model name must not contain null bytes")
|
||||
if len(name) > _MAX_NAME_LEN:
|
||||
raise ValueError(f"model name must be <= {_MAX_NAME_LEN} chars")
|
||||
if not _NAME_RE.match(name):
|
||||
raise ValueError(
|
||||
f"model name must match {_NAME_RE.pattern}: {name!r}"
|
||||
)
|
||||
return name
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Topological sort (Kahn's)
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _topological_sort(
|
||||
names: Sequence[str],
|
||||
edges: Sequence[Tuple[str, str]],
|
||||
) -> List[str]:
|
||||
"""Kahn's algorithm. Same shape as v0.45.0 Part E ``recipe_dag``."""
|
||||
in_degree = {name: 0 for name in names}
|
||||
successors: dict = {name: [] for name in names}
|
||||
for source, target in edges:
|
||||
in_degree[target] += 1
|
||||
successors[source].append(target)
|
||||
queue: deque = deque(
|
||||
sorted(name for name, deg in in_degree.items() if deg == 0)
|
||||
)
|
||||
order: List[str] = []
|
||||
while queue:
|
||||
current = queue.popleft()
|
||||
order.append(current)
|
||||
ready: List[str] = []
|
||||
for successor in successors[current]:
|
||||
in_degree[successor] -= 1
|
||||
if in_degree[successor] == 0:
|
||||
ready.append(successor)
|
||||
ready.sort()
|
||||
queue.extend(ready)
|
||||
if len(order) != len(names):
|
||||
raise ValueError("build DAG contains a cycle")
|
||||
return order
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Plan parsing
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_build_plan(raw: Any) -> BuildPlan:
|
||||
"""Validate a ``{"models": [...]}`` dict and return a sorted ``BuildPlan``.
|
||||
|
||||
Each model is a dict with required ``name`` / ``kind`` / ``transform``,
|
||||
optional ``refs`` (list of upstream model names), optional ``source``
|
||||
(input file path for seed models), and optional ``config`` (free-form dict).
|
||||
"""
|
||||
if not isinstance(raw, dict):
|
||||
raise TypeError("build plan must be a dict")
|
||||
raw_models = raw.get("models")
|
||||
if raw_models is None:
|
||||
raise ValueError("build plan must define a 'models' key")
|
||||
if not isinstance(raw_models, list):
|
||||
raise ValueError("build plan 'models' must be a list")
|
||||
if not raw_models:
|
||||
raise ValueError("build plan 'models' must be a non-empty list")
|
||||
if len(raw_models) > _MAX_MODELS:
|
||||
raise ValueError(
|
||||
f"build plan 'models' exceeds {_MAX_MODELS} entries"
|
||||
)
|
||||
|
||||
models: List[BuildModel] = []
|
||||
seen_names: set = set()
|
||||
edges: List[Tuple[str, str]] = []
|
||||
name_set: set = set()
|
||||
|
||||
# First pass: validate every model + collect names.
|
||||
for index, raw_model in enumerate(raw_models):
|
||||
if not isinstance(raw_model, dict):
|
||||
raise TypeError(f"build plan models[{index}] must be a dict")
|
||||
name = validate_model_name(raw_model.get("name", ""))
|
||||
if name in seen_names:
|
||||
raise ValueError(f"duplicate model name: {name!r}")
|
||||
seen_names.add(name)
|
||||
kind = validate_model_kind(raw_model.get("kind", ""))
|
||||
transform = raw_model.get("transform")
|
||||
if not isinstance(transform, str) or isinstance(transform, bool):
|
||||
raise TypeError(
|
||||
f"models[{index}].transform must be a string"
|
||||
)
|
||||
raw_refs = raw_model.get("refs", [])
|
||||
if not isinstance(raw_refs, list):
|
||||
raise TypeError(f"models[{index}].refs must be a list")
|
||||
if len(raw_refs) > _MAX_REFS_PER_MODEL:
|
||||
raise ValueError(
|
||||
f"models[{index}].refs exceeds {_MAX_REFS_PER_MODEL} entries"
|
||||
)
|
||||
refs_seen: set = set()
|
||||
normalised_refs: List[str] = []
|
||||
for ref_index, ref in enumerate(raw_refs):
|
||||
ref_name = validate_model_name(ref)
|
||||
if ref_name in refs_seen:
|
||||
raise ValueError(
|
||||
f"duplicate ref in models[{index}].refs: {ref_name!r}"
|
||||
)
|
||||
refs_seen.add(ref_name)
|
||||
normalised_refs.append(ref_name)
|
||||
source = raw_model.get("source")
|
||||
config = raw_model.get("config", {})
|
||||
if not isinstance(config, dict):
|
||||
raise TypeError(f"models[{index}].config must be a dict")
|
||||
models.append(
|
||||
BuildModel(
|
||||
name=name,
|
||||
kind=kind,
|
||||
transform=transform,
|
||||
refs=tuple(normalised_refs),
|
||||
source=source,
|
||||
config=MappingProxyType(dict(config)),
|
||||
)
|
||||
)
|
||||
name_set.add(name)
|
||||
|
||||
# Second pass: validate refs reference real models + build edges.
|
||||
for model in models:
|
||||
for ref in model.refs:
|
||||
if ref == model.name:
|
||||
raise ValueError(
|
||||
f"self-loop edge rejected: {model.name!r}"
|
||||
)
|
||||
if ref not in name_set:
|
||||
raise ValueError(
|
||||
f"model {model.name!r} refs missing model: {ref!r}"
|
||||
)
|
||||
# Edge direction: upstream -> downstream (ref produces model)
|
||||
edges.append((ref, model.name))
|
||||
|
||||
topo = _topological_sort([m.name for m in models], edges)
|
||||
return BuildPlan(models=tuple(models), topo_order=tuple(topo))
|
||||
|
||||
|
||||
def parse_build_yaml(text: object) -> BuildPlan:
|
||||
"""Parse a YAML string into a validated ``BuildPlan``."""
|
||||
if not isinstance(text, str):
|
||||
raise TypeError("build text must be a string")
|
||||
if "\x00" in text:
|
||||
raise ValueError("build text must not contain null bytes")
|
||||
if len(text.encode("utf-8")) > _MAX_FILE_BYTES:
|
||||
raise ValueError(f"build text exceeds {_MAX_FILE_BYTES} bytes")
|
||||
import yaml # lazy — keep CLI startup fast
|
||||
|
||||
try:
|
||||
data = yaml.safe_load(text)
|
||||
except yaml.YAMLError as exc:
|
||||
raise ValueError(f"invalid YAML: {exc}") from exc
|
||||
return parse_build_plan(data)
|
||||
|
||||
|
||||
def load_build_yaml(path: object) -> BuildPlan:
|
||||
"""Load a build manifest from a path that must live under cwd.
|
||||
|
||||
Delegates to ``utils.paths.enforce_under_cwd_and_no_symlink`` (v0.59.0
|
||||
shared TOCTOU helper) so this module does not reinvent the policy
|
||||
(project code-review CRIT — centralisation enforcement).
|
||||
"""
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
if isinstance(path, bool) or not isinstance(path, str):
|
||||
raise TypeError("path must be a string")
|
||||
# Shared helper validates emptiness + null-bytes + cwd + symlink upfront.
|
||||
enforce_under_cwd_and_no_symlink(path, "build path")
|
||||
if not os.path.lexists(path):
|
||||
raise FileNotFoundError(path)
|
||||
real = os.path.realpath(path)
|
||||
if not os.path.isfile(real):
|
||||
raise FileNotFoundError(real)
|
||||
if os.path.getsize(real) > _MAX_FILE_BYTES:
|
||||
raise ValueError(f"build file exceeds {_MAX_FILE_BYTES} bytes")
|
||||
with open(real, "r", encoding="utf-8") as handle:
|
||||
return parse_build_yaml(handle.read())
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Incremental diff — re-tokenize only changed rows
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def compute_row_hash(row: Mapping[str, Any]) -> str:
|
||||
"""Return a stable SHA-256 hex digest of a row's content.
|
||||
|
||||
Keys are sorted so the hash is insertion-order independent. The hash
|
||||
EXCLUDES the ``id`` field so identity-vs-content can be reasoned about
|
||||
separately (same id + changed content = diff row).
|
||||
"""
|
||||
if not isinstance(row, Mapping):
|
||||
raise TypeError(
|
||||
f"row must be a Mapping, got {type(row).__name__}"
|
||||
)
|
||||
payload = {k: row[k] for k in sorted(row.keys()) if k != "id"}
|
||||
canonical = json.dumps(payload, sort_keys=True, ensure_ascii=False)
|
||||
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def incremental_diff(
|
||||
prev: Sequence[Mapping[str, Any]],
|
||||
new: Sequence[Mapping[str, Any]],
|
||||
) -> IncrementalDiffReport:
|
||||
"""Compare two row sequences keyed on ``id`` and return per-bucket counts.
|
||||
|
||||
Every row in both sequences MUST carry an ``id`` field — the function
|
||||
raises ``ValueError`` otherwise so silent mis-counting cannot happen.
|
||||
"""
|
||||
prev_map: dict = {}
|
||||
for index, row in enumerate(prev):
|
||||
if not isinstance(row, Mapping):
|
||||
raise TypeError(f"prev[{index}] must be a Mapping")
|
||||
if "id" not in row:
|
||||
raise ValueError(
|
||||
f"prev[{index}] missing required 'id' field for incremental diff"
|
||||
)
|
||||
prev_map[row["id"]] = compute_row_hash(row)
|
||||
|
||||
new_map: dict = {}
|
||||
for index, row in enumerate(new):
|
||||
if not isinstance(row, Mapping):
|
||||
raise TypeError(f"new[{index}] must be a Mapping")
|
||||
if "id" not in row:
|
||||
raise ValueError(
|
||||
f"new[{index}] missing required 'id' field for incremental diff"
|
||||
)
|
||||
new_map[row["id"]] = compute_row_hash(row)
|
||||
|
||||
added = 0
|
||||
changed = 0
|
||||
unchanged = 0
|
||||
for row_id, new_hash in new_map.items():
|
||||
if row_id not in prev_map:
|
||||
added += 1
|
||||
elif prev_map[row_id] != new_hash:
|
||||
changed += 1
|
||||
else:
|
||||
unchanged += 1
|
||||
removed = sum(1 for row_id in prev_map if row_id not in new_map)
|
||||
return IncrementalDiffReport(
|
||||
added=added,
|
||||
changed=changed,
|
||||
removed=removed,
|
||||
unchanged=unchanged,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Plan rendering
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def render_plan_table(plan: BuildPlan) -> str:
|
||||
"""Render a plan as plain text in topological order (for dry-run output)."""
|
||||
if not isinstance(plan, BuildPlan):
|
||||
raise TypeError("plan must be a BuildPlan")
|
||||
lookup = {m.name: m for m in plan.models}
|
||||
lines = ["Build plan (topological order):"]
|
||||
for name in plan.topo_order:
|
||||
model = lookup[name]
|
||||
ref_str = ", ".join(model.refs) if model.refs else "(no refs)"
|
||||
lines.append(
|
||||
f" {model.name} [{model.kind}] transform={model.transform} refs=[{ref_str}]"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Live runner — deferred to v0.69.1
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_build(plan: BuildPlan, *, output_dir: Optional[str] = None) -> None:
|
||||
"""Execute the build DAG. Deferred to v0.69.1.
|
||||
|
||||
Same stub-then-live cadence as v0.45.0 recipe DAG / v0.50.0 GRPO Plus /
|
||||
v0.61.0 unlearning. The CLI prints the resolved plan + a deferred-live
|
||||
advisory and exits with code 3 so CI gates can distinguish "deferred /
|
||||
not yet shipped" from "validation rejection" (which exits 2).
|
||||
"""
|
||||
if not isinstance(plan, BuildPlan):
|
||||
raise TypeError("plan must be a BuildPlan")
|
||||
raise NotImplementedError(
|
||||
"soup build live runner is deferred to v0.69.1 — only --dry-run is wired today."
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BuildModel",
|
||||
"BuildPlan",
|
||||
"IncrementalDiffReport",
|
||||
"SUPPORTED_MODEL_KINDS",
|
||||
"compute_row_hash",
|
||||
"incremental_diff",
|
||||
"load_build_yaml",
|
||||
"parse_build_plan",
|
||||
"parse_build_yaml",
|
||||
"render_plan_table",
|
||||
"run_build",
|
||||
"validate_model_kind",
|
||||
"validate_model_name",
|
||||
]
|
||||
|
||||
|
||||
def _selfcheck() -> Iterable[str]:
|
||||
"""Internal: list of expected public symbols."""
|
||||
return tuple(__all__)
|
||||
|
|
@ -0,0 +1,553 @@
|
|||
"""v0.69.0 Part B — Expectations suite for chat data.
|
||||
|
||||
Great-Expectations-flavoured assertions over JSONL rows. Each expectation is a
|
||||
pure function returning a frozen ``ExpectationResult``; the suite runner
|
||||
composes them and returns a ``SuiteReport`` whose ``passed`` flag is the
|
||||
AND of every contained expectation. CI gates use exit code 3 on failure
|
||||
(matches v0.55 / v0.56 / v0.64 / v0.65 gate convention).
|
||||
|
||||
Composes with:
|
||||
- v0.47.0 ``data_score.detect_pii`` (PII regex / Presidio backend)
|
||||
- v0.56.0 ``diagnose.refusal.looks_like_refusal`` (English refusal phrases)
|
||||
- v0.19.0 judge backends (operator-injected callable for judge expectations)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable, List, Mapping, Optional, Sequence, Tuple
|
||||
|
||||
# Closed allowlist — the 4 expectation kinds the v0.69.0 plan calls out.
|
||||
_SUPPORTED_EXPECTATIONS = (
|
||||
"expect_no_pii",
|
||||
"expect_token_length_between",
|
||||
"expect_no_refusal_pattern",
|
||||
"expect_chosen_preferred_over_rejected_by_judge",
|
||||
)
|
||||
SUPPORTED_EXPECTATIONS: frozenset = frozenset(_SUPPORTED_EXPECTATIONS)
|
||||
|
||||
# DoS caps + bounds for validators.
|
||||
_MAX_NAME_LEN = 128
|
||||
_MAX_DETAILS_PER_RESULT = 32
|
||||
_MAX_DETAIL_LEN = 256
|
||||
_MAX_SUITE_LEN = 64
|
||||
_MAX_FILE_BYTES = 1_048_576 # 1 MiB
|
||||
_MIN_TOKEN_BOUND = 1
|
||||
_MAX_TOKEN_BOUND = 1_048_576
|
||||
|
||||
# JudgeFn signature: row mapping in, [0,1] score out (1.0 = chosen wins).
|
||||
JudgeFn = Callable[[Mapping[str, Any]], float]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExpectationResult:
|
||||
"""Outcome of one expectation."""
|
||||
|
||||
name: str
|
||||
passed: bool
|
||||
num_rows_checked: int
|
||||
num_violations: int
|
||||
details: Tuple[str, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
validate_expectation_name(self.name)
|
||||
if not isinstance(self.passed, bool):
|
||||
raise TypeError("ExpectationResult.passed must be bool")
|
||||
for field_name, val in (
|
||||
("num_rows_checked", self.num_rows_checked),
|
||||
("num_violations", self.num_violations),
|
||||
):
|
||||
if isinstance(val, bool) or not isinstance(val, int):
|
||||
raise TypeError(f"ExpectationResult.{field_name} must be int")
|
||||
if val < 0:
|
||||
raise ValueError(
|
||||
f"ExpectationResult.{field_name} must be non-negative"
|
||||
)
|
||||
if not isinstance(self.details, tuple):
|
||||
raise TypeError(
|
||||
"ExpectationResult.details must be a tuple (frozen=True does "
|
||||
"not make List immutable)"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExpectationSpec:
|
||||
"""One row of a suite YAML."""
|
||||
|
||||
name: str
|
||||
args: Mapping[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SuiteSpec:
|
||||
"""Validated expectations suite."""
|
||||
|
||||
expectations: Tuple[ExpectationSpec, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SuiteReport:
|
||||
"""Outcome of a full suite run."""
|
||||
|
||||
passed: bool
|
||||
results: Tuple[ExpectationResult, ...]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Validators
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_expectation_name(name: object) -> str:
|
||||
"""Return the canonical lower-case expectation name."""
|
||||
if isinstance(name, bool) or not isinstance(name, str):
|
||||
raise TypeError(
|
||||
f"expectation name must be str, got {type(name).__name__}"
|
||||
)
|
||||
if not name:
|
||||
raise ValueError("expectation name must be non-empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("expectation name must not contain null bytes")
|
||||
if len(name) > _MAX_NAME_LEN:
|
||||
raise ValueError(
|
||||
f"expectation name must be <= {_MAX_NAME_LEN} chars"
|
||||
)
|
||||
canonical = name.strip().lower()
|
||||
if canonical not in SUPPORTED_EXPECTATIONS:
|
||||
raise ValueError(
|
||||
f"unknown expectation: {name!r}. "
|
||||
f"supported: {sorted(SUPPORTED_EXPECTATIONS)}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def _check_threshold(value: object, *, field: str) -> float:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must be float, not bool")
|
||||
if not isinstance(value, (int, float)):
|
||||
raise TypeError(f"{field} must be a number")
|
||||
fval = float(value)
|
||||
if not math.isfinite(fval):
|
||||
raise ValueError(f"{field} must be finite")
|
||||
if not (0.0 <= fval <= 1.0):
|
||||
raise ValueError(f"{field} must be in [0.0, 1.0]")
|
||||
return fval
|
||||
|
||||
|
||||
def _check_token_bound(value: object, *, field: str) -> int:
|
||||
if isinstance(value, bool):
|
||||
raise TypeError(f"{field} must be int, not bool")
|
||||
if not isinstance(value, int):
|
||||
raise TypeError(f"{field} must be an integer")
|
||||
if value < _MIN_TOKEN_BOUND or value > _MAX_TOKEN_BOUND:
|
||||
raise ValueError(
|
||||
f"{field} must be in [{_MIN_TOKEN_BOUND}, {_MAX_TOKEN_BOUND}]"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Row helpers
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _extract_row_text(row: Mapping[str, Any]) -> str:
|
||||
"""Best-effort text extraction. Mirrors v0.55.0 / v0.56.0 row-text policy.
|
||||
|
||||
Combines all text-shaped fields so PII / refusal scans never miss content
|
||||
hiding in either ``text``/``content``/``output``/``response`` or inside a
|
||||
``messages`` chat structure.
|
||||
"""
|
||||
parts: List[str] = []
|
||||
for key in ("text", "content", "output", "response", "prompt", "instruction"):
|
||||
val = row.get(key)
|
||||
if isinstance(val, str) and val:
|
||||
parts.append(val)
|
||||
messages = row.get("messages")
|
||||
if isinstance(messages, list):
|
||||
for msg in messages:
|
||||
if isinstance(msg, Mapping):
|
||||
content = msg.get("content")
|
||||
if isinstance(content, str) and content:
|
||||
parts.append(content)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _extract_assistant_text(row: Mapping[str, Any]) -> str:
|
||||
"""Best-effort assistant-side text extraction (for refusal scans)."""
|
||||
parts: List[str] = []
|
||||
for key in ("output", "response"):
|
||||
val = row.get(key)
|
||||
if isinstance(val, str) and val:
|
||||
parts.append(val)
|
||||
messages = row.get("messages")
|
||||
if isinstance(messages, list):
|
||||
for msg in messages:
|
||||
if isinstance(msg, Mapping) and msg.get("role") == "assistant":
|
||||
content = msg.get("content")
|
||||
if isinstance(content, str) and content:
|
||||
parts.append(content)
|
||||
if parts:
|
||||
return "\n".join(parts)
|
||||
# Fall back to general row text when the row has no chat structure.
|
||||
return _extract_row_text(row)
|
||||
|
||||
|
||||
def _check_rows(rows: object) -> Sequence[Mapping[str, Any]]:
|
||||
if isinstance(rows, str) or isinstance(rows, bytes):
|
||||
raise TypeError("rows must be a list of mappings, not a string")
|
||||
if not hasattr(rows, "__iter__"):
|
||||
raise TypeError("rows must be iterable")
|
||||
try:
|
||||
materialised = list(rows)
|
||||
except TypeError as exc:
|
||||
raise TypeError(f"rows is not iterable: {exc}") from exc
|
||||
return materialised
|
||||
|
||||
|
||||
def _truncate_detail(detail: str) -> str:
|
||||
if len(detail) > _MAX_DETAIL_LEN:
|
||||
return detail[: _MAX_DETAIL_LEN - 3] + "..."
|
||||
return detail
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Expectations
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def expect_no_pii(rows: Any) -> ExpectationResult:
|
||||
"""Fail when any row contains an email / phone / SSN / credit-card hit.
|
||||
|
||||
Reuses v0.47.0 ``data_score.detect_pii`` (Presidio backend if available;
|
||||
falls back to the in-tree 4-regex baseline) — so improvements there
|
||||
immediately flow into expectations.
|
||||
"""
|
||||
materialised = _check_rows(rows)
|
||||
from soup_cli.utils.data_score import detect_pii # lazy
|
||||
|
||||
num_violations = 0
|
||||
details: List[str] = []
|
||||
for index, row in enumerate(materialised):
|
||||
if not isinstance(row, Mapping):
|
||||
details.append(_truncate_detail(f"rows[{index}]: not a dict"))
|
||||
num_violations += 1
|
||||
continue
|
||||
text = _extract_row_text(row)
|
||||
if not text:
|
||||
continue
|
||||
try:
|
||||
hits = detect_pii(text)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if hits:
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
kinds = sorted({hit.get("kind", "?") for hit in hits})
|
||||
details.append(
|
||||
_truncate_detail(
|
||||
f"rows[{index}]: PII detected ({', '.join(kinds)})"
|
||||
)
|
||||
)
|
||||
return ExpectationResult(
|
||||
name="expect_no_pii",
|
||||
passed=num_violations == 0,
|
||||
num_rows_checked=len(materialised),
|
||||
num_violations=num_violations,
|
||||
details=tuple(details),
|
||||
)
|
||||
|
||||
|
||||
def expect_token_length_between(
|
||||
rows: Any,
|
||||
*,
|
||||
min_tokens: int,
|
||||
max_tokens: int,
|
||||
) -> ExpectationResult:
|
||||
"""Fail when any row's token count (whitespace-split) is out of bounds."""
|
||||
low = _check_token_bound(min_tokens, field="min_tokens")
|
||||
high = _check_token_bound(max_tokens, field="max_tokens")
|
||||
if low > high:
|
||||
raise ValueError(
|
||||
f"min_tokens ({low}) must be <= max_tokens ({high})"
|
||||
)
|
||||
materialised = _check_rows(rows)
|
||||
num_violations = 0
|
||||
details: List[str] = []
|
||||
for index, row in enumerate(materialised):
|
||||
if not isinstance(row, Mapping):
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(_truncate_detail(f"rows[{index}]: not a dict"))
|
||||
continue
|
||||
text = _extract_row_text(row)
|
||||
token_count = len(text.split())
|
||||
if token_count < low or token_count > high:
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(
|
||||
_truncate_detail(
|
||||
f"rows[{index}]: {token_count} tokens (want [{low}, {high}])"
|
||||
)
|
||||
)
|
||||
return ExpectationResult(
|
||||
name="expect_token_length_between",
|
||||
passed=num_violations == 0,
|
||||
num_rows_checked=len(materialised),
|
||||
num_violations=num_violations,
|
||||
details=tuple(details),
|
||||
)
|
||||
|
||||
|
||||
def expect_no_refusal_pattern(rows: Any) -> ExpectationResult:
|
||||
"""Fail when any assistant-side output matches the v0.56.0 refusal regex."""
|
||||
materialised = _check_rows(rows)
|
||||
from soup_cli.utils.diagnose.refusal import looks_like_refusal # lazy
|
||||
|
||||
num_violations = 0
|
||||
details: List[str] = []
|
||||
for index, row in enumerate(materialised):
|
||||
if not isinstance(row, Mapping):
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(_truncate_detail(f"rows[{index}]: not a dict"))
|
||||
continue
|
||||
text = _extract_assistant_text(row)
|
||||
if text and looks_like_refusal(text):
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(
|
||||
_truncate_detail(f"rows[{index}]: refusal pattern matched")
|
||||
)
|
||||
return ExpectationResult(
|
||||
name="expect_no_refusal_pattern",
|
||||
passed=num_violations == 0,
|
||||
num_rows_checked=len(materialised),
|
||||
num_violations=num_violations,
|
||||
details=tuple(details),
|
||||
)
|
||||
|
||||
|
||||
def expect_chosen_preferred_over_rejected_by_judge(
|
||||
rows: Any,
|
||||
*,
|
||||
judge_fn: Optional[JudgeFn] = None,
|
||||
threshold: float = 0.7,
|
||||
) -> ExpectationResult:
|
||||
"""Fail when ``judge_fn(row) < threshold`` on a preference row.
|
||||
|
||||
Rows must carry both ``chosen`` and ``rejected``. ``judge_fn`` returns a
|
||||
score in [0, 1]; 1.0 means the judge fully prefers chosen over rejected.
|
||||
|
||||
When ``judge_fn`` is omitted the suite runs in *advisory* mode (every row
|
||||
gets the default score 1.0, i.e. trust the labelling); production callers
|
||||
should always supply a real judge.
|
||||
"""
|
||||
t = _check_threshold(threshold, field="threshold")
|
||||
if judge_fn is not None and not callable(judge_fn):
|
||||
raise TypeError("judge_fn must be callable or None")
|
||||
materialised = _check_rows(rows)
|
||||
num_violations = 0
|
||||
details: List[str] = []
|
||||
for index, row in enumerate(materialised):
|
||||
if not isinstance(row, Mapping):
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(_truncate_detail(f"rows[{index}]: not a dict"))
|
||||
continue
|
||||
if "chosen" not in row or "rejected" not in row:
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(
|
||||
_truncate_detail(
|
||||
f"rows[{index}]: missing chosen/rejected field"
|
||||
)
|
||||
)
|
||||
continue
|
||||
if judge_fn is None:
|
||||
# No judge supplied — assume chosen wins (advisory pass).
|
||||
score: float = 1.0
|
||||
else:
|
||||
try:
|
||||
raw = judge_fn(row)
|
||||
except Exception: # noqa: BLE001 — one bad row mustn't crash the suite
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(
|
||||
_truncate_detail(f"rows[{index}]: judge raised")
|
||||
)
|
||||
continue
|
||||
if isinstance(raw, bool) or not isinstance(raw, (int, float)):
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(
|
||||
_truncate_detail(
|
||||
f"rows[{index}]: judge returned non-number"
|
||||
)
|
||||
)
|
||||
continue
|
||||
score = float(raw)
|
||||
if not math.isfinite(score):
|
||||
num_violations += 1
|
||||
continue
|
||||
if score < t:
|
||||
num_violations += 1
|
||||
if len(details) < _MAX_DETAILS_PER_RESULT:
|
||||
details.append(
|
||||
_truncate_detail(
|
||||
f"rows[{index}]: judge score {score:.3f} < {t:.3f}"
|
||||
)
|
||||
)
|
||||
return ExpectationResult(
|
||||
name="expect_chosen_preferred_over_rejected_by_judge",
|
||||
passed=num_violations == 0,
|
||||
num_rows_checked=len(materialised),
|
||||
num_violations=num_violations,
|
||||
details=tuple(details),
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Suite parsing + execution
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_suite_spec(raw: Any) -> SuiteSpec:
|
||||
"""Validate a suite dict and return a ``SuiteSpec``."""
|
||||
if not isinstance(raw, dict):
|
||||
raise TypeError("suite spec must be a dict")
|
||||
raw_expectations = raw.get("expectations")
|
||||
if raw_expectations is None:
|
||||
raise ValueError("suite must define 'expectations' key")
|
||||
if not isinstance(raw_expectations, list):
|
||||
raise TypeError("suite 'expectations' must be a list")
|
||||
if not raw_expectations:
|
||||
raise ValueError("suite 'expectations' must be a non-empty list")
|
||||
if len(raw_expectations) > _MAX_SUITE_LEN:
|
||||
raise ValueError(
|
||||
f"suite 'expectations' exceeds {_MAX_SUITE_LEN} entries"
|
||||
)
|
||||
|
||||
items: List[ExpectationSpec] = []
|
||||
for index, entry in enumerate(raw_expectations):
|
||||
if not isinstance(entry, dict):
|
||||
raise TypeError(f"expectations[{index}] must be a dict")
|
||||
name = validate_expectation_name(entry.get("name", ""))
|
||||
args = entry.get("args", {})
|
||||
if not isinstance(args, dict):
|
||||
raise TypeError(f"expectations[{index}].args must be a dict")
|
||||
items.append(ExpectationSpec(name=name, args=dict(args)))
|
||||
return SuiteSpec(expectations=tuple(items))
|
||||
|
||||
|
||||
def parse_suite_yaml(text: object) -> SuiteSpec:
|
||||
"""Parse a suite YAML string into a validated ``SuiteSpec``."""
|
||||
if not isinstance(text, str):
|
||||
raise TypeError("suite text must be a string")
|
||||
if "\x00" in text:
|
||||
raise ValueError("suite text must not contain null bytes")
|
||||
if len(text.encode("utf-8")) > _MAX_FILE_BYTES:
|
||||
raise ValueError(f"suite text exceeds {_MAX_FILE_BYTES} bytes")
|
||||
import yaml # lazy
|
||||
|
||||
try:
|
||||
data = yaml.safe_load(text)
|
||||
except yaml.YAMLError as exc:
|
||||
raise ValueError(f"invalid YAML: {exc}") from exc
|
||||
return parse_suite_spec(data)
|
||||
|
||||
|
||||
def load_suite_yaml(path: object) -> SuiteSpec:
|
||||
"""Load a suite YAML from a path under cwd (with TOCTOU symlink reject).
|
||||
|
||||
Delegates to ``utils.paths.enforce_under_cwd_and_no_symlink`` (v0.59.0
|
||||
centralised helper).
|
||||
"""
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
if isinstance(path, bool) or not isinstance(path, str):
|
||||
raise TypeError("path must be a string")
|
||||
enforce_under_cwd_and_no_symlink(path, "suite path")
|
||||
if not os.path.lexists(path):
|
||||
raise FileNotFoundError(path)
|
||||
real = os.path.realpath(path)
|
||||
if not os.path.isfile(real):
|
||||
raise FileNotFoundError(real)
|
||||
if os.path.getsize(real) > _MAX_FILE_BYTES:
|
||||
raise ValueError(f"suite file exceeds {_MAX_FILE_BYTES} bytes")
|
||||
with open(real, "r", encoding="utf-8") as handle:
|
||||
return parse_suite_yaml(handle.read())
|
||||
|
||||
|
||||
def _dispatch_expectation(
|
||||
spec: ExpectationSpec,
|
||||
rows: Sequence[Mapping[str, Any]],
|
||||
*,
|
||||
judge_fn: Optional[JudgeFn] = None,
|
||||
) -> ExpectationResult:
|
||||
"""Dispatch one ``ExpectationSpec`` against ``rows``.
|
||||
|
||||
Per code-review HIGH H2: passes ``args`` values through *as-is* — the
|
||||
expectation function's own validators must reject bool / NaN / Inf /
|
||||
type-coerced inputs. This prevents an ``int("5")`` silent coercion from
|
||||
bypassing ``_check_token_bound`` bool-rejection.
|
||||
"""
|
||||
name = spec.name
|
||||
args = dict(spec.args)
|
||||
if name == "expect_no_pii":
|
||||
return expect_no_pii(rows)
|
||||
if name == "expect_token_length_between":
|
||||
return expect_token_length_between(
|
||||
rows,
|
||||
min_tokens=args.get("min_tokens", 1),
|
||||
max_tokens=args.get("max_tokens", _MAX_TOKEN_BOUND),
|
||||
)
|
||||
if name == "expect_no_refusal_pattern":
|
||||
return expect_no_refusal_pattern(rows)
|
||||
if name == "expect_chosen_preferred_over_rejected_by_judge":
|
||||
return expect_chosen_preferred_over_rejected_by_judge(
|
||||
rows,
|
||||
judge_fn=judge_fn,
|
||||
threshold=args.get("threshold", 0.7),
|
||||
)
|
||||
raise ValueError(f"unhandled expectation: {name!r}") # pragma: no cover
|
||||
|
||||
|
||||
def run_suite(
|
||||
rows: Any,
|
||||
spec: SuiteSpec,
|
||||
*,
|
||||
judge_fn: Optional[JudgeFn] = None,
|
||||
) -> SuiteReport:
|
||||
"""Run every expectation in ``spec`` against ``rows``."""
|
||||
if not isinstance(spec, SuiteSpec):
|
||||
raise TypeError("spec must be a SuiteSpec")
|
||||
materialised = _check_rows(rows)
|
||||
results: List[ExpectationResult] = []
|
||||
for expectation in spec.expectations:
|
||||
results.append(
|
||||
_dispatch_expectation(expectation, materialised, judge_fn=judge_fn)
|
||||
)
|
||||
passed = all(r.passed for r in results)
|
||||
return SuiteReport(passed=passed, results=tuple(results))
|
||||
|
||||
|
||||
__all__ = [
|
||||
"ExpectationResult",
|
||||
"ExpectationSpec",
|
||||
"JudgeFn",
|
||||
"SUPPORTED_EXPECTATIONS",
|
||||
"SuiteReport",
|
||||
"SuiteSpec",
|
||||
"expect_chosen_preferred_over_rejected_by_judge",
|
||||
"expect_no_pii",
|
||||
"expect_no_refusal_pattern",
|
||||
"expect_token_length_between",
|
||||
"load_suite_yaml",
|
||||
"parse_suite_spec",
|
||||
"parse_suite_yaml",
|
||||
"run_suite",
|
||||
"validate_expectation_name",
|
||||
]
|
||||
|
|
@ -0,0 +1,164 @@
|
|||
"""v0.69.0 Part C — `soup data gen magpie` synthetic generator.
|
||||
|
||||
The Magpie technique (Xu et al. 2024) feeds an aligned chat-tuned model just
|
||||
the chat-template prefix (system + user-turn header tokens) and lets the model
|
||||
generate the user turn itself. The harvested user turns are then passed back to
|
||||
the same model in a normal completion call to harvest the assistant response.
|
||||
This is a clever way to produce high-volume SFT data without paying for human
|
||||
prompts.
|
||||
|
||||
This module ships the schema + validators + dry-run planner. Live generation
|
||||
(invoking v0.20.0 Ollama / Anthropic / vLLM providers + running the quality
|
||||
filter from v0.47.0 educational + toxicity scorers) is deferred to v0.69.1
|
||||
under the project-wide stub-then-live cadence (mirrors v0.50.0 / v0.61.0 /
|
||||
v0.62.0 / v0.68.0).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
# Closed allowlist of providers — reuses v0.20.0 synth-data backends.
|
||||
_SUPPORTED_MAGPIE_PROVIDERS = ("ollama", "anthropic", "vllm")
|
||||
SUPPORTED_MAGPIE_PROVIDERS: frozenset = frozenset(_SUPPORTED_MAGPIE_PROVIDERS)
|
||||
|
||||
_MAX_BASE_MODEL_LEN = 512
|
||||
_MAX_TARGET_ROWS = 1_000_000
|
||||
_MAX_PROVIDER_LEN = 32
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MagpieConfig:
|
||||
"""Frozen plan for one ``soup data gen-magpie`` invocation."""
|
||||
|
||||
base_model: str
|
||||
provider: str
|
||||
target_rows: int
|
||||
quality_filter: bool
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Re-validate so direct construction can't smuggle bad fields past
|
||||
# the validators (mirrors v0.67.0 vector_bank policy).
|
||||
validate_base_model(self.base_model)
|
||||
validate_magpie_provider(self.provider)
|
||||
validate_target_rows(self.target_rows)
|
||||
if not isinstance(self.quality_filter, bool):
|
||||
raise TypeError(
|
||||
"MagpieConfig.quality_filter must be bool, "
|
||||
f"got {type(self.quality_filter).__name__}"
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Validators
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_magpie_provider(provider: object) -> str:
|
||||
"""Return the canonical lower-case provider id."""
|
||||
if isinstance(provider, bool) or not isinstance(provider, str):
|
||||
raise TypeError(
|
||||
f"magpie provider must be str, got {type(provider).__name__}"
|
||||
)
|
||||
if not provider:
|
||||
raise ValueError("magpie provider must be non-empty")
|
||||
if "\x00" in provider:
|
||||
raise ValueError("magpie provider must not contain null bytes")
|
||||
if len(provider) > _MAX_PROVIDER_LEN:
|
||||
raise ValueError(
|
||||
f"magpie provider must be <= {_MAX_PROVIDER_LEN} chars"
|
||||
)
|
||||
canonical = provider.strip().lower()
|
||||
if canonical not in SUPPORTED_MAGPIE_PROVIDERS:
|
||||
raise ValueError(
|
||||
f"unknown magpie provider: {provider!r}. "
|
||||
f"supported: {sorted(SUPPORTED_MAGPIE_PROVIDERS)}"
|
||||
)
|
||||
return canonical
|
||||
|
||||
|
||||
def validate_target_rows(target: object) -> int:
|
||||
"""Validate ``target_rows`` ∈ [1, 1_000_000]; bool-rejected."""
|
||||
if isinstance(target, bool):
|
||||
raise TypeError("target_rows must be int, not bool")
|
||||
if not isinstance(target, int):
|
||||
raise TypeError("target_rows must be an integer")
|
||||
if target < 1:
|
||||
raise ValueError("target_rows must be >= 1")
|
||||
if target > _MAX_TARGET_ROWS:
|
||||
raise ValueError(f"target_rows must be <= {_MAX_TARGET_ROWS}")
|
||||
return target
|
||||
|
||||
|
||||
def validate_base_model(name: object) -> str:
|
||||
"""Validate the base-model identifier."""
|
||||
if isinstance(name, bool) or not isinstance(name, str):
|
||||
raise TypeError(
|
||||
f"base_model must be str, got {type(name).__name__}"
|
||||
)
|
||||
if not name:
|
||||
raise ValueError("base_model must be non-empty")
|
||||
if "\x00" in name:
|
||||
raise ValueError("base_model must not contain null bytes")
|
||||
if len(name) > _MAX_BASE_MODEL_LEN:
|
||||
raise ValueError(
|
||||
f"base_model must be <= {_MAX_BASE_MODEL_LEN} chars"
|
||||
)
|
||||
return name
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Plan builder
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_magpie_config(
|
||||
*,
|
||||
base: str,
|
||||
provider: str,
|
||||
target: int,
|
||||
quality_filter: bool = True,
|
||||
) -> MagpieConfig:
|
||||
"""Convenience factory — every input passes through the validators."""
|
||||
if not isinstance(quality_filter, bool):
|
||||
raise TypeError(
|
||||
f"quality_filter must be bool, got {type(quality_filter).__name__}"
|
||||
)
|
||||
return MagpieConfig(
|
||||
base_model=validate_base_model(base),
|
||||
provider=validate_magpie_provider(provider),
|
||||
target_rows=validate_target_rows(target),
|
||||
quality_filter=quality_filter,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Live runner — deferred to v0.69.1
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_magpie(config: Any) -> None:
|
||||
"""Execute the Magpie generation loop. Deferred to v0.69.1.
|
||||
|
||||
The CLI catches ``NotImplementedError`` and exits with code 3 (distinct
|
||||
from validation rejection exit 2) — same policy as v0.61.0 / v0.62.0 /
|
||||
v0.68.0 deferred-live stubs.
|
||||
"""
|
||||
if not isinstance(config, MagpieConfig):
|
||||
raise TypeError("config must be a MagpieConfig")
|
||||
raise NotImplementedError(
|
||||
"soup data gen-magpie live runner is deferred to v0.69.1 — "
|
||||
"only --plan-only is wired today."
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"MagpieConfig",
|
||||
"SUPPORTED_MAGPIE_PROVIDERS",
|
||||
"build_magpie_config",
|
||||
"run_magpie",
|
||||
"validate_base_model",
|
||||
"validate_magpie_provider",
|
||||
"validate_target_rows",
|
||||
]
|
||||
|
|
@ -0,0 +1,277 @@
|
|||
"""v0.69.0 Part D — Persona-Hub diversity sampler.
|
||||
|
||||
Samples prompt × persona × style combinations to drive synthetic-data diversity
|
||||
(per Tencent's Persona-Hub paper). Ships with a small bundled persona list so
|
||||
``soup data persona-mix`` works offline; operators wanting the full 200k-persona
|
||||
HF dataset can pass ``--personas <jsonl>`` from a downloaded copy.
|
||||
|
||||
The topic-diversity metric is a pure-Python token-entropy heuristic — same
|
||||
approach as the v0.55.0 ``eval_design`` TF-IDF baseline. The full
|
||||
sentence-transformer embedding-entropy variant is the operator-installable
|
||||
``[data-pro]`` extras path; this module does not import any embedding library
|
||||
to keep the CLI startup fast.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import random
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, List, Mapping, Sequence, Tuple
|
||||
|
||||
# 12 bundled personas covering broad demographic + occupational diversity.
|
||||
# Operators wanting the full Persona-Hub corpus pass a JSONL via the CLI.
|
||||
_BUNDLED_PERSONAS: Tuple[str, ...] = (
|
||||
"a curious high school student",
|
||||
"a software engineer with 10 years of experience",
|
||||
"a non-native English speaker learning to code",
|
||||
"a research scientist in molecular biology",
|
||||
"a small business owner planning their first product launch",
|
||||
"a freelance journalist on a tight deadline",
|
||||
"a retired teacher revisiting old hobbies",
|
||||
"a graduate student preparing a thesis defense",
|
||||
"a parent helping a child with homework",
|
||||
"a healthcare professional asking patient-care questions",
|
||||
"a hobbyist game developer trying a new engine",
|
||||
"a senior engineer onboarding a junior teammate",
|
||||
)
|
||||
|
||||
_BUNDLED_STYLES: Tuple[str, ...] = (
|
||||
"formal",
|
||||
"casual",
|
||||
"step-by-step",
|
||||
"concise",
|
||||
"playful",
|
||||
)
|
||||
|
||||
_MAX_PERSONA_LEN = 1024
|
||||
_MAX_STYLE_LEN = 128
|
||||
_MAX_LIST_LEN = 100_000
|
||||
_MAX_SAMPLES = 1_000_000
|
||||
_MAX_DIVERSITY_TEXT_LEN = 65_536
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Validators
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_persona(persona: object) -> str:
|
||||
"""Validate a persona string."""
|
||||
if isinstance(persona, bool) or not isinstance(persona, str):
|
||||
raise TypeError(
|
||||
f"persona must be str, got {type(persona).__name__}"
|
||||
)
|
||||
if not persona:
|
||||
raise ValueError("persona must be non-empty")
|
||||
if "\x00" in persona:
|
||||
raise ValueError("persona must not contain null bytes")
|
||||
if len(persona) > _MAX_PERSONA_LEN:
|
||||
raise ValueError(f"persona must be <= {_MAX_PERSONA_LEN} chars")
|
||||
return persona
|
||||
|
||||
|
||||
def validate_style(style: object) -> str:
|
||||
"""Validate a style string."""
|
||||
if isinstance(style, bool) or not isinstance(style, str):
|
||||
raise TypeError(f"style must be str, got {type(style).__name__}")
|
||||
if not style:
|
||||
raise ValueError("style must be non-empty")
|
||||
if "\x00" in style:
|
||||
raise ValueError("style must not contain null bytes")
|
||||
if len(style) > _MAX_STYLE_LEN:
|
||||
raise ValueError(f"style must be <= {_MAX_STYLE_LEN} chars")
|
||||
return style
|
||||
|
||||
|
||||
def list_bundled_personas() -> Tuple[str, ...]:
|
||||
"""Return the bundled persona tuple. Immutable by virtue of being a tuple."""
|
||||
return _BUNDLED_PERSONAS
|
||||
|
||||
|
||||
def list_bundled_styles() -> Tuple[str, ...]:
|
||||
"""Return the bundled style tuple."""
|
||||
return _BUNDLED_STYLES
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Plan dataclass + factory
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PersonaPlan:
|
||||
"""Frozen plan for a persona-mix sample run."""
|
||||
|
||||
prompts: Tuple[str, ...]
|
||||
personas: Tuple[str, ...]
|
||||
styles: Tuple[str, ...]
|
||||
n: int
|
||||
seed: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for field_name in ("prompts", "personas", "styles"):
|
||||
value = getattr(self, field_name)
|
||||
if not isinstance(value, tuple):
|
||||
raise TypeError(
|
||||
f"PersonaPlan.{field_name} must be a tuple "
|
||||
"(frozen=True does not make List immutable)"
|
||||
)
|
||||
if isinstance(self.n, bool):
|
||||
raise TypeError("PersonaPlan.n must be int, not bool")
|
||||
if not isinstance(self.n, int) or self.n < 1:
|
||||
raise ValueError("PersonaPlan.n must be a positive integer")
|
||||
if isinstance(self.seed, bool):
|
||||
raise TypeError("PersonaPlan.seed must be int, not bool")
|
||||
if not isinstance(self.seed, int):
|
||||
raise TypeError("PersonaPlan.seed must be an integer")
|
||||
|
||||
|
||||
def _check_list(
|
||||
name: str, values: Sequence[Any], item_validator: Any
|
||||
) -> Tuple[str, ...]:
|
||||
if isinstance(values, (str, bytes)) or not hasattr(values, "__iter__"):
|
||||
raise TypeError(f"{name} must be a list of strings")
|
||||
materialised = list(values)
|
||||
if not materialised:
|
||||
raise ValueError(f"{name} must be non-empty")
|
||||
if len(materialised) > _MAX_LIST_LEN:
|
||||
raise ValueError(f"{name} exceeds {_MAX_LIST_LEN} entries")
|
||||
return tuple(item_validator(v) for v in materialised)
|
||||
|
||||
|
||||
def build_persona_plan(
|
||||
*,
|
||||
prompts: Sequence[str],
|
||||
personas: Sequence[str],
|
||||
styles: Sequence[str],
|
||||
n: int,
|
||||
seed: int,
|
||||
) -> PersonaPlan:
|
||||
"""Validate inputs + return a frozen ``PersonaPlan``."""
|
||||
if isinstance(n, bool):
|
||||
raise TypeError("n must be int, not bool")
|
||||
if not isinstance(n, int):
|
||||
raise TypeError("n must be int")
|
||||
if n < 1:
|
||||
raise ValueError("n must be >= 1")
|
||||
if n > _MAX_SAMPLES:
|
||||
raise ValueError(f"n exceeds {_MAX_SAMPLES}")
|
||||
if isinstance(seed, bool):
|
||||
raise TypeError("seed must be int, not bool")
|
||||
if not isinstance(seed, int):
|
||||
raise TypeError("seed must be int")
|
||||
return PersonaPlan(
|
||||
prompts=_check_list("prompts", prompts, validate_persona),
|
||||
personas=_check_list("personas", personas, validate_persona),
|
||||
styles=_check_list("styles", styles, validate_style),
|
||||
n=n,
|
||||
seed=seed,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Sampler
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def sample_persona_matrix(
|
||||
*,
|
||||
prompts: Sequence[str],
|
||||
personas: Sequence[str],
|
||||
styles: Sequence[str],
|
||||
n: int,
|
||||
seed: int,
|
||||
) -> List[Mapping[str, str]]:
|
||||
"""Sample ``n`` rows from the prompt × persona × style matrix.
|
||||
|
||||
Deterministic given the same ``(inputs, seed)`` — uses a seeded ``Random``
|
||||
instance so concurrent samplers in the same process do not affect each
|
||||
other.
|
||||
"""
|
||||
plan = build_persona_plan(
|
||||
prompts=prompts,
|
||||
personas=personas,
|
||||
styles=styles,
|
||||
n=n,
|
||||
seed=seed,
|
||||
)
|
||||
rng = random.Random(plan.seed)
|
||||
rows: List[Mapping[str, str]] = []
|
||||
for _ in range(plan.n):
|
||||
rows.append(
|
||||
{
|
||||
"prompt": rng.choice(plan.prompts),
|
||||
"persona": rng.choice(plan.personas),
|
||||
"style": rng.choice(plan.styles),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Topic diversity
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _extract_text(row: Mapping[str, Any]) -> str:
|
||||
"""Extract concatenated text from a row across common fields."""
|
||||
parts: List[str] = []
|
||||
for key in ("text", "content", "output", "prompt", "instruction", "response"):
|
||||
val = row.get(key)
|
||||
if isinstance(val, str) and val:
|
||||
parts.append(val)
|
||||
if not parts:
|
||||
return ""
|
||||
joined = " ".join(parts)
|
||||
if len(joined) > _MAX_DIVERSITY_TEXT_LEN:
|
||||
joined = joined[:_MAX_DIVERSITY_TEXT_LEN]
|
||||
return joined
|
||||
|
||||
|
||||
def compute_topic_diversity(rows: Any) -> float:
|
||||
"""Return a topic-diversity score in [0, 1].
|
||||
|
||||
Heuristic: pool the whitespace tokens across all rows, compute Shannon
|
||||
entropy in bits, and normalise by ``log2(unique_tokens)`` so the score
|
||||
is comparable across corpora. Returns ``0.0`` on empty input.
|
||||
"""
|
||||
if isinstance(rows, (str, bytes)):
|
||||
raise TypeError("rows must be a list of mappings")
|
||||
if not hasattr(rows, "__iter__"):
|
||||
raise TypeError("rows must be iterable")
|
||||
tokens: List[str] = []
|
||||
for row in rows:
|
||||
if not isinstance(row, Mapping):
|
||||
continue
|
||||
text = _extract_text(row)
|
||||
if not text:
|
||||
continue
|
||||
tokens.extend(text.lower().split())
|
||||
if not tokens:
|
||||
return 0.0
|
||||
counter = Counter(tokens)
|
||||
total = sum(counter.values())
|
||||
unique = len(counter)
|
||||
if unique <= 1:
|
||||
return 0.0
|
||||
entropy = -sum(
|
||||
(count / total) * math.log2(count / total) for count in counter.values()
|
||||
)
|
||||
max_entropy = math.log2(unique)
|
||||
if max_entropy <= 0:
|
||||
return 0.0
|
||||
return max(0.0, min(1.0, entropy / max_entropy))
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PersonaPlan",
|
||||
"build_persona_plan",
|
||||
"compute_topic_diversity",
|
||||
"list_bundled_personas",
|
||||
"list_bundled_styles",
|
||||
"sample_persona_matrix",
|
||||
"validate_persona",
|
||||
"validate_style",
|
||||
]
|
||||
|
|
@ -0,0 +1,733 @@
|
|||
"""v0.69.0 Part A — `soup build` dbt-for-SFT DAG parser + topo sort + plan."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
from soup_cli.utils import build_dag
|
||||
|
||||
|
||||
def _write(path: Path, text: str) -> Path:
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Allowlist + immutability
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSupportedKinds:
|
||||
def test_supported_kinds_exact(self) -> None:
|
||||
assert build_dag.SUPPORTED_MODEL_KINDS == frozenset(
|
||||
{"incremental", "table", "view"}
|
||||
)
|
||||
|
||||
def test_supported_kinds_is_frozenset(self) -> None:
|
||||
assert isinstance(build_dag.SUPPORTED_MODEL_KINDS, frozenset)
|
||||
|
||||
def test_supported_kinds_immutable(self) -> None:
|
||||
with pytest.raises(AttributeError):
|
||||
build_dag.SUPPORTED_MODEL_KINDS.add("evil") # type: ignore[attr-defined]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# validate_model_kind
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateModelKind:
|
||||
def test_happy_path(self) -> None:
|
||||
assert build_dag.validate_model_kind("incremental") == "incremental"
|
||||
assert build_dag.validate_model_kind("table") == "table"
|
||||
assert build_dag.validate_model_kind("view") == "view"
|
||||
|
||||
def test_case_insensitive(self) -> None:
|
||||
assert build_dag.validate_model_kind("Incremental") == "incremental"
|
||||
assert build_dag.validate_model_kind("VIEW") == "view"
|
||||
|
||||
def test_unknown(self) -> None:
|
||||
with pytest.raises(ValueError, match="unknown model kind"):
|
||||
build_dag.validate_model_kind("ephemeral")
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.validate_model_kind(123)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.validate_model_kind(True)
|
||||
|
||||
def test_empty(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.validate_model_kind("")
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
build_dag.validate_model_kind("incremental\x00x")
|
||||
|
||||
def test_oversize(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.validate_model_kind("x" * 100)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# validate_model_name
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateModelName:
|
||||
def test_happy(self) -> None:
|
||||
assert build_dag.validate_model_name("raw_chat") == "raw_chat"
|
||||
assert build_dag.validate_model_name("step-1") == "step-1"
|
||||
assert build_dag.validate_model_name("a.b") == "a.b"
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.validate_model_name(42)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.validate_model_name(True)
|
||||
|
||||
def test_empty(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.validate_model_name("")
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
build_dag.validate_model_name("a\x00b")
|
||||
|
||||
def test_path_traversal(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.validate_model_name("../etc")
|
||||
|
||||
def test_path_separator(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.validate_model_name("a/b")
|
||||
|
||||
def test_oversize(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.validate_model_name("a" * 200)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# BuildModel frozen + validation
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBuildModel:
|
||||
def test_happy(self) -> None:
|
||||
model = build_dag.BuildModel(
|
||||
name="raw",
|
||||
kind="incremental",
|
||||
transform="identity",
|
||||
refs=(),
|
||||
source="data/raw.jsonl",
|
||||
config={},
|
||||
)
|
||||
assert model.name == "raw"
|
||||
assert model.kind == "incremental"
|
||||
assert model.refs == ()
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
model = build_dag.BuildModel(
|
||||
name="raw",
|
||||
kind="incremental",
|
||||
transform="identity",
|
||||
refs=(),
|
||||
source="data/raw.jsonl",
|
||||
config={},
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
model.name = "evil" # type: ignore[misc]
|
||||
|
||||
def test_refs_must_be_tuple(self) -> None:
|
||||
with pytest.raises(TypeError, match="tuple"):
|
||||
build_dag.BuildModel(
|
||||
name="x",
|
||||
kind="incremental",
|
||||
transform="identity",
|
||||
refs=["y"], # type: ignore[arg-type]
|
||||
source=None,
|
||||
config={},
|
||||
)
|
||||
|
||||
def test_invalid_kind_propagates(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.BuildModel(
|
||||
name="raw",
|
||||
kind="ephemeral",
|
||||
transform="identity",
|
||||
refs=(),
|
||||
source=None,
|
||||
config={},
|
||||
)
|
||||
|
||||
def test_invalid_name_propagates(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.BuildModel(
|
||||
name="../etc",
|
||||
kind="incremental",
|
||||
transform="identity",
|
||||
refs=(),
|
||||
source=None,
|
||||
config={},
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Plan parsing + topo sort
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestParseBuildPlan:
|
||||
def test_happy_linear(self) -> None:
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "raw",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "identity",
|
||||
},
|
||||
{
|
||||
"name": "filtered",
|
||||
"kind": "incremental",
|
||||
"refs": ["raw"],
|
||||
"transform": "filter_low_quality",
|
||||
},
|
||||
{
|
||||
"name": "tokenized",
|
||||
"kind": "incremental",
|
||||
"refs": ["filtered"],
|
||||
"transform": "tokenize",
|
||||
},
|
||||
]
|
||||
}
|
||||
plan = build_dag.parse_build_plan(raw)
|
||||
assert plan.topo_order == ("raw", "filtered", "tokenized")
|
||||
assert len(plan.models) == 3
|
||||
|
||||
def test_happy_diamond(self) -> None:
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "a",
|
||||
"kind": "incremental",
|
||||
"source": "data/seed.jsonl",
|
||||
"transform": "x",
|
||||
},
|
||||
{"name": "b", "kind": "incremental", "refs": ["a"], "transform": "x"},
|
||||
{"name": "c", "kind": "incremental", "refs": ["a"], "transform": "x"},
|
||||
{
|
||||
"name": "d",
|
||||
"kind": "incremental",
|
||||
"refs": ["b", "c"],
|
||||
"transform": "x",
|
||||
},
|
||||
]
|
||||
}
|
||||
plan = build_dag.parse_build_plan(raw)
|
||||
# a must come first, d must come last
|
||||
assert plan.topo_order[0] == "a"
|
||||
assert plan.topo_order[-1] == "d"
|
||||
|
||||
def test_non_dict(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.parse_build_plan(["models"]) # type: ignore[arg-type]
|
||||
|
||||
def test_empty_models_rejected(self) -> None:
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
build_dag.parse_build_plan({"models": []})
|
||||
|
||||
def test_missing_models_key(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.parse_build_plan({})
|
||||
|
||||
def test_models_not_list(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.parse_build_plan({"models": "raw_chat"})
|
||||
|
||||
def test_cycle_rejected(self) -> None:
|
||||
# Both have refs → no source; cycle rejection fires inside topo sort.
|
||||
raw = {
|
||||
"models": [
|
||||
{"name": "a", "kind": "incremental", "refs": ["b"], "transform": "x"},
|
||||
{"name": "b", "kind": "incremental", "refs": ["a"], "transform": "x"},
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="cycle"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
def test_self_loop_rejected(self) -> None:
|
||||
# refs=["a"] but self-ref also implies refs (not seed); cross-validator
|
||||
# accepts (refs set + source None), self-loop fires in second pass.
|
||||
raw = {
|
||||
"models": [
|
||||
{"name": "a", "kind": "incremental", "refs": ["a"], "transform": "x"},
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="self"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
def test_dangling_ref_rejected(self) -> None:
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "a",
|
||||
"kind": "incremental",
|
||||
"refs": ["missing"],
|
||||
"transform": "x",
|
||||
},
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="missing|unknown|not found"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
def test_duplicate_name_rejected(self) -> None:
|
||||
# Duplicate-name check fires BEFORE BuildModel construction, so the
|
||||
# seed/derived cross-validator is not reached. Use minimal shape.
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "a",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "x",
|
||||
},
|
||||
{
|
||||
"name": "a",
|
||||
"kind": "table",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "x",
|
||||
},
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="duplicate"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
def test_oversize_models(self) -> None:
|
||||
# Many-model cap fires BEFORE BuildModel construction.
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": f"m{i}",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "x",
|
||||
}
|
||||
for i in range(build_dag._MAX_MODELS + 1)
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="exceeds"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
def test_duplicate_ref_in_same_model_rejected(self) -> None:
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "a",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "x",
|
||||
},
|
||||
{
|
||||
"name": "b",
|
||||
"kind": "incremental",
|
||||
"refs": ["a", "a"],
|
||||
"transform": "x",
|
||||
},
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="duplicate"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
def test_seed_without_source_rejected(self) -> None:
|
||||
# New cross-validator: refs=() + source=None is degenerate.
|
||||
raw = {
|
||||
"models": [
|
||||
{"name": "a", "kind": "incremental", "transform": "x"},
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="source"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
def test_refs_and_source_mutually_exclusive(self) -> None:
|
||||
# New cross-validator: refs + source together rejected.
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "a",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "x",
|
||||
},
|
||||
{
|
||||
"name": "b",
|
||||
"kind": "incremental",
|
||||
"refs": ["a"],
|
||||
"source": "data/other.jsonl",
|
||||
"transform": "x",
|
||||
},
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="mutually exclusive"):
|
||||
build_dag.parse_build_plan(raw)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# parse_build_yaml + load_build_yaml
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestParseBuildYaml:
|
||||
def test_happy(self) -> None:
|
||||
text = (
|
||||
"models:\n"
|
||||
" - name: raw\n"
|
||||
" kind: incremental\n"
|
||||
" source: data/raw.jsonl\n"
|
||||
" transform: identity\n"
|
||||
" - name: out\n"
|
||||
" kind: incremental\n"
|
||||
" refs: [raw]\n"
|
||||
" transform: identity\n"
|
||||
)
|
||||
plan = build_dag.parse_build_yaml(text)
|
||||
assert plan.topo_order == ("raw", "out")
|
||||
|
||||
def test_invalid_yaml(self) -> None:
|
||||
with pytest.raises(ValueError, match="invalid YAML"):
|
||||
build_dag.parse_build_yaml("models: [unclosed")
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.parse_build_yaml(42) # type: ignore[arg-type]
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
build_dag.parse_build_yaml("models:\x00")
|
||||
|
||||
def test_oversize(self) -> None:
|
||||
text = "models:\n" + (" - name: x\n kind: incremental\n transform: y\n" * 100000)
|
||||
with pytest.raises(ValueError, match="exceeds"):
|
||||
build_dag.parse_build_yaml(text)
|
||||
|
||||
|
||||
class TestLoadBuildYaml:
|
||||
def test_happy(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
text = (
|
||||
"models:\n"
|
||||
" - name: raw\n"
|
||||
" kind: incremental\n"
|
||||
" source: data/raw.jsonl\n"
|
||||
" transform: identity\n"
|
||||
)
|
||||
path = _write(tmp_path / "build.yaml", text)
|
||||
plan = build_dag.load_build_yaml(str(path))
|
||||
assert plan.topo_order == ("raw",)
|
||||
|
||||
def test_outside_cwd(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
_write(outside / "build.yaml", "models: []")
|
||||
sub = tmp_path / "sub"
|
||||
sub.mkdir()
|
||||
monkeypatch.chdir(sub)
|
||||
with pytest.raises(ValueError, match="cwd"):
|
||||
build_dag.load_build_yaml(str(outside / "build.yaml"))
|
||||
|
||||
def test_missing(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
build_dag.load_build_yaml(str(tmp_path / "nope.yaml"))
|
||||
|
||||
def test_null_byte(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
build_dag.load_build_yaml("a\x00b.yaml")
|
||||
|
||||
def test_empty_path(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(ValueError):
|
||||
build_dag.load_build_yaml("")
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises((TypeError, ValueError)):
|
||||
build_dag.load_build_yaml(42) # type: ignore[arg-type]
|
||||
|
||||
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlink")
|
||||
def test_symlink_rejected(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = _write(
|
||||
tmp_path / "real.yaml",
|
||||
"models:\n - name: a\n kind: incremental\n transform: x\n",
|
||||
)
|
||||
link = tmp_path / "link.yaml"
|
||||
os.symlink(str(target), str(link))
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
build_dag.load_build_yaml(str(link))
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Incremental diff (re-tokenize only changed rows)
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRowHash:
|
||||
def test_deterministic(self) -> None:
|
||||
row = {"id": "1", "text": "hello"}
|
||||
assert build_dag.compute_row_hash(row) == build_dag.compute_row_hash(row)
|
||||
|
||||
def test_key_order_independent(self) -> None:
|
||||
a = {"id": "1", "text": "hello"}
|
||||
b = {"text": "hello", "id": "1"}
|
||||
assert build_dag.compute_row_hash(a) == build_dag.compute_row_hash(b)
|
||||
|
||||
def test_content_sensitive(self) -> None:
|
||||
a = {"id": "1", "text": "hello"}
|
||||
b = {"id": "1", "text": "world"}
|
||||
assert build_dag.compute_row_hash(a) != build_dag.compute_row_hash(b)
|
||||
|
||||
def test_non_dict(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.compute_row_hash([1, 2]) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestIncrementalDiff:
|
||||
def test_all_new(self) -> None:
|
||||
new = [{"id": "1", "text": "a"}, {"id": "2", "text": "b"}]
|
||||
report = build_dag.incremental_diff([], new)
|
||||
assert report.added == 2
|
||||
assert report.changed == 0
|
||||
assert report.removed == 0
|
||||
assert report.unchanged == 0
|
||||
|
||||
def test_all_unchanged(self) -> None:
|
||||
rows = [{"id": "1", "text": "a"}, {"id": "2", "text": "b"}]
|
||||
report = build_dag.incremental_diff(rows, rows)
|
||||
assert report.unchanged == 2
|
||||
assert report.added == 0
|
||||
assert report.changed == 0
|
||||
assert report.removed == 0
|
||||
|
||||
def test_changed_row(self) -> None:
|
||||
prev = [{"id": "1", "text": "a"}]
|
||||
new = [{"id": "1", "text": "b"}]
|
||||
report = build_dag.incremental_diff(prev, new)
|
||||
assert report.changed == 1
|
||||
assert report.added == 0
|
||||
|
||||
def test_removed_row(self) -> None:
|
||||
prev = [{"id": "1", "text": "a"}, {"id": "2", "text": "b"}]
|
||||
new = [{"id": "1", "text": "a"}]
|
||||
report = build_dag.incremental_diff(prev, new)
|
||||
assert report.removed == 1
|
||||
assert report.unchanged == 1
|
||||
|
||||
def test_missing_id_field(self) -> None:
|
||||
with pytest.raises(ValueError, match="id"):
|
||||
build_dag.incremental_diff([{"text": "a"}], [])
|
||||
|
||||
def test_frozen_report(self) -> None:
|
||||
report = build_dag.incremental_diff([], [])
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
report.added = 999 # type: ignore[misc]
|
||||
|
||||
|
||||
class TestBuildPlanFrozen:
|
||||
"""TDD review #12 — BuildPlan dataclass must be immutable."""
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "a",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "x",
|
||||
}
|
||||
]
|
||||
}
|
||||
plan = build_dag.parse_build_plan(raw)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
plan.topo_order = ("evil",) # type: ignore[misc]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Plan rendering (dry-run output)
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRenderPlanTable:
|
||||
def test_happy(self) -> None:
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "raw",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "identity",
|
||||
},
|
||||
{
|
||||
"name": "filtered",
|
||||
"kind": "incremental",
|
||||
"refs": ["raw"],
|
||||
"transform": "filter",
|
||||
},
|
||||
]
|
||||
}
|
||||
plan = build_dag.parse_build_plan(raw)
|
||||
rendered = build_dag.render_plan_table(plan)
|
||||
assert "raw" in rendered
|
||||
assert "filtered" in rendered
|
||||
# topo order preserved
|
||||
assert rendered.index("raw") < rendered.index("filtered")
|
||||
|
||||
def test_non_plan_type(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.render_plan_table({"models": []}) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# run_build (deferred-live)
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRunBuild:
|
||||
def test_deferred(self) -> None:
|
||||
raw = {
|
||||
"models": [
|
||||
{
|
||||
"name": "raw",
|
||||
"kind": "incremental",
|
||||
"source": "data/raw.jsonl",
|
||||
"transform": "identity",
|
||||
},
|
||||
]
|
||||
}
|
||||
plan = build_dag.parse_build_plan(raw)
|
||||
with pytest.raises(NotImplementedError, match="v0.69.1"):
|
||||
build_dag.run_build(plan)
|
||||
|
||||
def test_run_build_validates_plan_type(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
build_dag.run_build({"models": []}) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CLI: `soup build`
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSoupBuildCli:
|
||||
def test_help(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["build", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "build" in result.output.lower()
|
||||
|
||||
def test_dry_run_happy(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = _write(
|
||||
tmp_path / "build.yaml",
|
||||
"models:\n"
|
||||
" - name: raw\n"
|
||||
" kind: incremental\n"
|
||||
" source: data/raw.jsonl\n"
|
||||
" transform: identity\n"
|
||||
" - name: filtered\n"
|
||||
" kind: incremental\n"
|
||||
" refs: [raw]\n"
|
||||
" transform: filter\n",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["build", str(path), "--dry-run"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "raw" in result.output
|
||||
assert "filtered" in result.output
|
||||
|
||||
def test_missing_config(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["build", "nope.yaml", "--dry-run"])
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_live_deferred(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = _write(
|
||||
tmp_path / "build.yaml",
|
||||
"models:\n - name: raw\n kind: incremental\n"
|
||||
" source: data/raw.jsonl\n transform: identity\n",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["build", str(path)])
|
||||
# deferred runner exits non-zero with v0.69.1 marker
|
||||
assert result.exit_code != 0
|
||||
assert "0.69.1" in result.output or "deferred" in result.output.lower()
|
||||
|
||||
def test_outside_cwd(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
_write(
|
||||
outside / "build.yaml",
|
||||
"models:\n - name: a\n kind: incremental\n transform: x\n",
|
||||
)
|
||||
sub = tmp_path / "sub"
|
||||
sub.mkdir()
|
||||
monkeypatch.chdir(sub)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["build", str(outside / "build.yaml"), "--dry-run"])
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source-grep wiring
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_cli_registers_build(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
cli = (root / "soup_cli" / "cli.py").read_text(encoding="utf-8")
|
||||
assert (
|
||||
'name="build"' in cli
|
||||
or "build_cmd" in cli
|
||||
or "from soup_cli.commands import build" in cli
|
||||
)
|
||||
|
||||
def test_no_heavy_top_level_imports(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "build_dag.py").read_text(encoding="utf-8")
|
||||
# yaml is also lazy-imported inside parse_build_yaml (TDD review H2).
|
||||
for forbidden in (
|
||||
"\nimport torch",
|
||||
"\nimport transformers",
|
||||
"\nimport peft",
|
||||
"\nimport yaml",
|
||||
):
|
||||
assert forbidden not in src, f"top-level import found: {forbidden!r}"
|
||||
|
||||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
|
|
@ -0,0 +1,587 @@
|
|||
"""v0.69.0 Part B — Expectations suite for chat data."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
from soup_cli.utils import expectations
|
||||
|
||||
|
||||
def _write(path: Path, text: str) -> Path:
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Allowlist + immutability
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSupportedExpectations:
|
||||
def test_exact(self) -> None:
|
||||
assert expectations.SUPPORTED_EXPECTATIONS == frozenset(
|
||||
{
|
||||
"expect_no_pii",
|
||||
"expect_token_length_between",
|
||||
"expect_no_refusal_pattern",
|
||||
"expect_chosen_preferred_over_rejected_by_judge",
|
||||
}
|
||||
)
|
||||
|
||||
def test_is_frozenset(self) -> None:
|
||||
assert isinstance(expectations.SUPPORTED_EXPECTATIONS, frozenset)
|
||||
|
||||
def test_immutable(self) -> None:
|
||||
with pytest.raises(AttributeError):
|
||||
expectations.SUPPORTED_EXPECTATIONS.add( # type: ignore[attr-defined]
|
||||
"evil"
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# validate_expectation_name
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateExpectationName:
|
||||
def test_happy(self) -> None:
|
||||
assert (
|
||||
expectations.validate_expectation_name("expect_no_pii") == "expect_no_pii"
|
||||
)
|
||||
|
||||
def test_case_insensitive(self) -> None:
|
||||
assert (
|
||||
expectations.validate_expectation_name("EXPECT_NO_PII") == "expect_no_pii"
|
||||
)
|
||||
|
||||
def test_unknown(self) -> None:
|
||||
with pytest.raises(ValueError, match="unknown expectation"):
|
||||
expectations.validate_expectation_name("expect_perfection")
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.validate_expectation_name(42)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.validate_expectation_name(True)
|
||||
|
||||
def test_empty(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.validate_expectation_name("")
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
expectations.validate_expectation_name("expect\x00_no_pii")
|
||||
|
||||
def test_oversize(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.validate_expectation_name("x" * 200)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# ExpectationResult frozen
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExpectationResult:
|
||||
def test_happy(self) -> None:
|
||||
r = expectations.ExpectationResult(
|
||||
name="expect_no_pii",
|
||||
passed=True,
|
||||
num_rows_checked=10,
|
||||
num_violations=0,
|
||||
details=(),
|
||||
)
|
||||
assert r.passed is True
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
r = expectations.ExpectationResult(
|
||||
name="expect_no_pii",
|
||||
passed=True,
|
||||
num_rows_checked=10,
|
||||
num_violations=0,
|
||||
details=(),
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
r.passed = False # type: ignore[misc]
|
||||
|
||||
def test_details_must_be_tuple(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.ExpectationResult(
|
||||
name="expect_no_pii",
|
||||
passed=True,
|
||||
num_rows_checked=10,
|
||||
num_violations=0,
|
||||
details=["x"], # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def test_violations_non_negative(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.ExpectationResult(
|
||||
name="expect_no_pii",
|
||||
passed=True,
|
||||
num_rows_checked=10,
|
||||
num_violations=-1,
|
||||
details=(),
|
||||
)
|
||||
|
||||
def test_invalid_name(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.ExpectationResult(
|
||||
name="bogus",
|
||||
passed=True,
|
||||
num_rows_checked=0,
|
||||
num_violations=0,
|
||||
details=(),
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# expect_no_pii
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExpectNoPii:
|
||||
def test_clean(self) -> None:
|
||||
rows = [
|
||||
{"text": "Hello, how are you today?"},
|
||||
{"text": "This is a normal sentence."},
|
||||
]
|
||||
result = expectations.expect_no_pii(rows)
|
||||
assert result.passed is True
|
||||
assert result.num_violations == 0
|
||||
assert result.num_rows_checked == 2
|
||||
|
||||
def test_email_flagged(self) -> None:
|
||||
rows = [{"text": "Email me at evil@example.com please."}]
|
||||
result = expectations.expect_no_pii(rows)
|
||||
assert result.passed is False
|
||||
assert result.num_violations >= 1
|
||||
|
||||
def test_phone_flagged(self) -> None:
|
||||
rows = [{"text": "Call me at 555-123-4567 anytime."}]
|
||||
result = expectations.expect_no_pii(rows)
|
||||
assert result.passed is False
|
||||
|
||||
def test_empty_rows(self) -> None:
|
||||
result = expectations.expect_no_pii([])
|
||||
assert result.passed is True
|
||||
assert result.num_rows_checked == 0
|
||||
|
||||
def test_messages_field(self) -> None:
|
||||
rows = [
|
||||
{
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hi"},
|
||||
{"role": "assistant", "content": "Hello, my email is evil@e.com"},
|
||||
]
|
||||
}
|
||||
]
|
||||
result = expectations.expect_no_pii(rows)
|
||||
assert result.passed is False
|
||||
|
||||
def test_non_list_raises(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.expect_no_pii("rows") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# expect_token_length_between
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExpectTokenLengthBetween:
|
||||
def test_clean(self) -> None:
|
||||
rows = [{"text": "a " * 50}, {"text": "b " * 60}]
|
||||
result = expectations.expect_token_length_between(rows, min_tokens=10, max_tokens=200)
|
||||
assert result.passed is True
|
||||
|
||||
def test_below_min(self) -> None:
|
||||
rows = [{"text": "short"}]
|
||||
result = expectations.expect_token_length_between(rows, min_tokens=50, max_tokens=200)
|
||||
assert result.passed is False
|
||||
assert result.num_violations == 1
|
||||
|
||||
def test_above_max(self) -> None:
|
||||
rows = [{"text": "a " * 500}]
|
||||
result = expectations.expect_token_length_between(rows, min_tokens=10, max_tokens=50)
|
||||
assert result.passed is False
|
||||
assert result.num_violations == 1
|
||||
|
||||
def test_invalid_min(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.expect_token_length_between([], min_tokens=-1, max_tokens=100)
|
||||
|
||||
def test_invalid_max(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.expect_token_length_between([], min_tokens=10, max_tokens=0)
|
||||
|
||||
def test_min_greater_than_max(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.expect_token_length_between([], min_tokens=200, max_tokens=10)
|
||||
|
||||
def test_bool_min(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.expect_token_length_between([], min_tokens=True, max_tokens=100)
|
||||
|
||||
def test_bool_max(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.expect_token_length_between([], min_tokens=1, max_tokens=False)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# expect_no_refusal_pattern
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExpectNoRefusalPattern:
|
||||
def test_clean(self) -> None:
|
||||
rows = [{"output": "Here is the recipe you asked for."}]
|
||||
result = expectations.expect_no_refusal_pattern(rows)
|
||||
assert result.passed is True
|
||||
|
||||
def test_refusal_detected(self) -> None:
|
||||
rows = [{"output": "I cannot help with that request."}]
|
||||
result = expectations.expect_no_refusal_pattern(rows)
|
||||
assert result.passed is False
|
||||
|
||||
def test_assistant_refusal_in_messages(self) -> None:
|
||||
rows = [
|
||||
{
|
||||
"messages": [
|
||||
{"role": "user", "content": "How do I do X?"},
|
||||
{"role": "assistant", "content": "Sorry, but I refuse to discuss that."},
|
||||
]
|
||||
}
|
||||
]
|
||||
result = expectations.expect_no_refusal_pattern(rows)
|
||||
assert result.passed is False
|
||||
|
||||
def test_empty(self) -> None:
|
||||
result = expectations.expect_no_refusal_pattern([])
|
||||
assert result.passed is True
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# expect_chosen_preferred_over_rejected_by_judge
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestChosenPreferredByJudge:
|
||||
def test_clean(self) -> None:
|
||||
rows = [
|
||||
{"prompt": "Q", "chosen": "good", "rejected": "bad"},
|
||||
{"prompt": "Q2", "chosen": "great", "rejected": "terrible"},
|
||||
]
|
||||
|
||||
def judge(row: dict) -> float:
|
||||
# Chosen always wins; score in [0, 1].
|
||||
return 0.9
|
||||
|
||||
result = expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
rows, judge_fn=judge, threshold=0.7
|
||||
)
|
||||
assert result.passed is True
|
||||
assert result.num_violations == 0
|
||||
|
||||
def test_violations(self) -> None:
|
||||
rows = [{"prompt": "Q", "chosen": "good", "rejected": "bad"}]
|
||||
|
||||
def judge(row: dict) -> float:
|
||||
return 0.3 # below threshold
|
||||
|
||||
result = expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
rows, judge_fn=judge, threshold=0.7
|
||||
)
|
||||
assert result.passed is False
|
||||
assert result.num_violations == 1
|
||||
|
||||
def test_missing_chosen_rejected_fields(self) -> None:
|
||||
rows = [{"prompt": "Q"}]
|
||||
|
||||
def judge(row: dict) -> float:
|
||||
return 0.9
|
||||
|
||||
# Rows missing chosen+rejected are flagged as violations.
|
||||
result = expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
rows, judge_fn=judge, threshold=0.7
|
||||
)
|
||||
assert result.passed is False
|
||||
|
||||
def test_invalid_threshold(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
[], judge_fn=lambda r: 0.9, threshold=1.5
|
||||
)
|
||||
|
||||
def test_non_callable_judge(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
[], judge_fn="not callable", threshold=0.5 # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def test_bool_threshold(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
[], judge_fn=lambda r: 0.5, threshold=True
|
||||
)
|
||||
|
||||
def test_nan_threshold(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
[], judge_fn=lambda r: 0.5, threshold=float("nan")
|
||||
)
|
||||
|
||||
def test_judge_exception_treated_as_violation(self) -> None:
|
||||
rows = [{"prompt": "Q", "chosen": "c", "rejected": "r"}]
|
||||
|
||||
def bad_judge(row: dict) -> float:
|
||||
raise RuntimeError("kaboom")
|
||||
|
||||
result = expectations.expect_chosen_preferred_over_rejected_by_judge(
|
||||
rows, judge_fn=bad_judge, threshold=0.5
|
||||
)
|
||||
assert result.passed is False
|
||||
assert result.num_violations == 1
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Suite spec + runner
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSuiteSpec:
|
||||
def test_happy(self) -> None:
|
||||
spec = expectations.parse_suite_spec(
|
||||
{
|
||||
"expectations": [
|
||||
{"name": "expect_no_pii"},
|
||||
{
|
||||
"name": "expect_token_length_between",
|
||||
"args": {"min_tokens": 10, "max_tokens": 1000},
|
||||
},
|
||||
]
|
||||
}
|
||||
)
|
||||
assert len(spec.expectations) == 2
|
||||
|
||||
def test_non_dict(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.parse_suite_spec(["nope"]) # type: ignore[arg-type]
|
||||
|
||||
def test_missing_expectations(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.parse_suite_spec({})
|
||||
|
||||
def test_empty_expectations(self) -> None:
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
expectations.parse_suite_spec({"expectations": []})
|
||||
|
||||
def test_unknown_expectation(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
expectations.parse_suite_spec(
|
||||
{"expectations": [{"name": "expect_perfection"}]}
|
||||
)
|
||||
|
||||
def test_args_must_be_dict(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
expectations.parse_suite_spec(
|
||||
{
|
||||
"expectations": [
|
||||
{"name": "expect_no_pii", "args": "not a dict"}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
def test_too_many_expectations(self) -> None:
|
||||
raw = {
|
||||
"expectations": [
|
||||
{"name": "expect_no_pii"} for _ in range(expectations._MAX_SUITE_LEN + 1)
|
||||
]
|
||||
}
|
||||
with pytest.raises(ValueError, match="exceeds"):
|
||||
expectations.parse_suite_spec(raw)
|
||||
|
||||
|
||||
class TestRunSuite:
|
||||
def test_all_pass(self) -> None:
|
||||
spec = expectations.parse_suite_spec(
|
||||
{"expectations": [{"name": "expect_no_pii"}]}
|
||||
)
|
||||
rows = [{"text": "all clean here"}]
|
||||
report = expectations.run_suite(rows, spec)
|
||||
assert report.passed is True
|
||||
assert len(report.results) == 1
|
||||
|
||||
def test_one_fails(self) -> None:
|
||||
spec = expectations.parse_suite_spec(
|
||||
{"expectations": [{"name": "expect_no_pii"}]}
|
||||
)
|
||||
rows = [{"text": "email me at a@b.com"}]
|
||||
report = expectations.run_suite(rows, spec)
|
||||
assert report.passed is False
|
||||
|
||||
def test_token_length_args(self) -> None:
|
||||
spec = expectations.parse_suite_spec(
|
||||
{
|
||||
"expectations": [
|
||||
{
|
||||
"name": "expect_token_length_between",
|
||||
"args": {"min_tokens": 10, "max_tokens": 1000},
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
rows = [{"text": "a " * 500}]
|
||||
report = expectations.run_suite(rows, spec)
|
||||
assert report.passed is True
|
||||
|
||||
def test_chosen_preferred_arg(self) -> None:
|
||||
spec = expectations.parse_suite_spec(
|
||||
{
|
||||
"expectations": [
|
||||
{
|
||||
"name": "expect_chosen_preferred_over_rejected_by_judge",
|
||||
"args": {"threshold": 0.5},
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
rows = [{"prompt": "Q", "chosen": "good", "rejected": "bad"}]
|
||||
# No judge fn supplied = neutral pass (operator must inject for live).
|
||||
report = expectations.run_suite(rows, spec)
|
||||
# Default judge returns 1.0 (no judge → assume chosen wins).
|
||||
assert report.passed is True
|
||||
|
||||
def test_report_frozen(self) -> None:
|
||||
spec = expectations.parse_suite_spec(
|
||||
{"expectations": [{"name": "expect_no_pii"}]}
|
||||
)
|
||||
report = expectations.run_suite([], spec)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
report.passed = False # type: ignore[misc]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# YAML loaders
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestLoadSuiteYaml:
|
||||
def test_happy(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = _write(
|
||||
tmp_path / "suite.yaml",
|
||||
"expectations:\n - name: expect_no_pii\n",
|
||||
)
|
||||
spec = expectations.load_suite_yaml(str(path))
|
||||
assert len(spec.expectations) == 1
|
||||
|
||||
def test_outside_cwd(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
_write(outside / "s.yaml", "expectations:\n - name: expect_no_pii\n")
|
||||
sub = tmp_path / "sub"
|
||||
sub.mkdir()
|
||||
monkeypatch.chdir(sub)
|
||||
with pytest.raises(ValueError, match="cwd"):
|
||||
expectations.load_suite_yaml(str(outside / "s.yaml"))
|
||||
|
||||
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlink")
|
||||
def test_symlink_rejected(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = _write(tmp_path / "real.yaml", "expectations:\n - name: expect_no_pii\n")
|
||||
link = tmp_path / "link.yaml"
|
||||
os.symlink(str(target), str(link))
|
||||
with pytest.raises(ValueError, match="symlink"):
|
||||
expectations.load_suite_yaml(str(link))
|
||||
|
||||
def test_missing(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
expectations.load_suite_yaml(str(tmp_path / "nope.yaml"))
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CLI: `soup expect`
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSoupExpectCli:
|
||||
def test_help(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["expect", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "expect" in result.output.lower()
|
||||
|
||||
def test_all_pass(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
data = _write(tmp_path / "data.jsonl", '{"text": "clean output here"}\n')
|
||||
suite = _write(
|
||||
tmp_path / "suite.yaml", "expectations:\n - name: expect_no_pii\n"
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["expect", str(data), str(suite)])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_failure_exits_3(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
data = _write(
|
||||
tmp_path / "data.jsonl", '{"text": "email me at evil@e.com"}\n'
|
||||
)
|
||||
suite = _write(
|
||||
tmp_path / "suite.yaml", "expectations:\n - name: expect_no_pii\n"
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["expect", str(data), str(suite)])
|
||||
assert result.exit_code == 3
|
||||
|
||||
def test_outside_cwd_data(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
_write(outside / "d.jsonl", '{"text":"x"}\n')
|
||||
sub = tmp_path / "sub"
|
||||
sub.mkdir()
|
||||
suite = _write(sub / "suite.yaml", "expectations:\n - name: expect_no_pii\n")
|
||||
monkeypatch.chdir(sub)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app, ["expect", str(outside / "d.jsonl"), str(suite)]
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source wiring
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_cli_registers_expect(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
cli = (root / "soup_cli" / "cli.py").read_text(encoding="utf-8")
|
||||
assert 'name="expect"' in cli or "from soup_cli.commands import expect" in cli
|
||||
|
||||
def test_no_heavy_top_level_imports(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "expectations.py").read_text(encoding="utf-8")
|
||||
for forbidden in (
|
||||
"\nimport torch",
|
||||
"\nimport transformers",
|
||||
"\nimport peft",
|
||||
"\nimport yaml",
|
||||
):
|
||||
assert forbidden not in src
|
||||
|
||||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
|
|
@ -0,0 +1,341 @@
|
|||
"""v0.69.0 Part C — `soup data gen magpie` synthetic generator."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
from soup_cli.utils import magpie
|
||||
|
||||
|
||||
def _write(path: Path, text: str) -> Path:
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Provider allowlist
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSupportedProviders:
|
||||
def test_exact(self) -> None:
|
||||
assert magpie.SUPPORTED_MAGPIE_PROVIDERS == frozenset(
|
||||
{"ollama", "anthropic", "vllm"}
|
||||
)
|
||||
|
||||
def test_immutable(self) -> None:
|
||||
with pytest.raises(AttributeError):
|
||||
magpie.SUPPORTED_MAGPIE_PROVIDERS.add("evil") # type: ignore[attr-defined]
|
||||
|
||||
|
||||
class TestValidateProvider:
|
||||
def test_happy(self) -> None:
|
||||
assert magpie.validate_magpie_provider("ollama") == "ollama"
|
||||
assert magpie.validate_magpie_provider("ANTHROPIC") == "anthropic"
|
||||
|
||||
def test_unknown(self) -> None:
|
||||
with pytest.raises(ValueError, match="unknown magpie provider"):
|
||||
magpie.validate_magpie_provider("openai")
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.validate_magpie_provider(42)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.validate_magpie_provider(True)
|
||||
|
||||
def test_empty(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.validate_magpie_provider("")
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
magpie.validate_magpie_provider("ollama\x00")
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Target rows + base model validators
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateTargetRows:
|
||||
def test_happy(self) -> None:
|
||||
assert magpie.validate_target_rows(100) == 100
|
||||
|
||||
def test_zero_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.validate_target_rows(0)
|
||||
|
||||
def test_negative(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.validate_target_rows(-5)
|
||||
|
||||
def test_overcap(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.validate_target_rows(magpie._MAX_TARGET_ROWS + 1)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.validate_target_rows(True)
|
||||
|
||||
def test_non_int(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.validate_target_rows(1.5)
|
||||
|
||||
|
||||
class TestValidateBaseModel:
|
||||
def test_happy(self) -> None:
|
||||
assert (
|
||||
magpie.validate_base_model("meta-llama/Llama-3.1-8B-Instruct")
|
||||
== "meta-llama/Llama-3.1-8B-Instruct"
|
||||
)
|
||||
|
||||
def test_empty(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.validate_base_model("")
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
magpie.validate_base_model("meta\x00llama")
|
||||
|
||||
def test_oversize(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.validate_base_model("x" * 1024)
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.validate_base_model(42)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.validate_base_model(True)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# MagpieConfig frozen dataclass
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestMagpieConfig:
|
||||
def test_happy(self) -> None:
|
||||
cfg = magpie.MagpieConfig(
|
||||
base_model="meta-llama/Llama-3.1-8B-Instruct",
|
||||
provider="ollama",
|
||||
target_rows=100,
|
||||
quality_filter=True,
|
||||
)
|
||||
assert cfg.target_rows == 100
|
||||
assert cfg.quality_filter is True
|
||||
|
||||
def test_frozen(self) -> None:
|
||||
cfg = magpie.MagpieConfig(
|
||||
base_model="m",
|
||||
provider="ollama",
|
||||
target_rows=10,
|
||||
quality_filter=False,
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
cfg.target_rows = 999 # type: ignore[misc]
|
||||
|
||||
def test_invalid_provider_propagates(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.MagpieConfig(
|
||||
base_model="m",
|
||||
provider="openai",
|
||||
target_rows=10,
|
||||
quality_filter=False,
|
||||
)
|
||||
|
||||
def test_invalid_target_propagates(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.MagpieConfig(
|
||||
base_model="m",
|
||||
provider="ollama",
|
||||
target_rows=0,
|
||||
quality_filter=False,
|
||||
)
|
||||
|
||||
def test_invalid_base_propagates(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
magpie.MagpieConfig(
|
||||
base_model="",
|
||||
provider="ollama",
|
||||
target_rows=10,
|
||||
quality_filter=False,
|
||||
)
|
||||
|
||||
def test_quality_filter_must_be_bool(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.MagpieConfig(
|
||||
base_model="m",
|
||||
provider="ollama",
|
||||
target_rows=10,
|
||||
quality_filter="yes", # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# build_magpie_config factory
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBuildMagpieConfig:
|
||||
def test_happy(self) -> None:
|
||||
cfg = magpie.build_magpie_config(
|
||||
base="meta-llama/Llama-3.1-8B-Instruct",
|
||||
provider="OLLAMA",
|
||||
target=50,
|
||||
)
|
||||
assert cfg.provider == "ollama"
|
||||
assert cfg.target_rows == 50
|
||||
assert cfg.quality_filter is True # default
|
||||
|
||||
def test_quality_filter_off(self) -> None:
|
||||
cfg = magpie.build_magpie_config(
|
||||
base="m",
|
||||
provider="ollama",
|
||||
target=10,
|
||||
quality_filter=False,
|
||||
)
|
||||
assert cfg.quality_filter is False
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Deferred live runner
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRunMagpie:
|
||||
def test_deferred(self) -> None:
|
||||
cfg = magpie.MagpieConfig(
|
||||
base_model="m",
|
||||
provider="ollama",
|
||||
target_rows=10,
|
||||
quality_filter=False,
|
||||
)
|
||||
with pytest.raises(NotImplementedError, match="v0.69.1"):
|
||||
magpie.run_magpie(cfg)
|
||||
|
||||
def test_validates_config_type(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
magpie.run_magpie({"base": "m"}) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CLI: `soup data gen magpie`
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestMagpieCli:
|
||||
def test_help(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["data", "gen-magpie", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "magpie" in result.output.lower()
|
||||
|
||||
def test_plan_only(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"data",
|
||||
"gen-magpie",
|
||||
"--base",
|
||||
"meta-llama/Llama-3.1-8B-Instruct",
|
||||
"--provider",
|
||||
"ollama",
|
||||
"--target",
|
||||
"10",
|
||||
"--plan-only",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "ollama" in result.output.lower() or "magpie" in result.output.lower()
|
||||
|
||||
def test_unknown_provider(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"data",
|
||||
"gen-magpie",
|
||||
"--base",
|
||||
"m",
|
||||
"--provider",
|
||||
"openai",
|
||||
"--target",
|
||||
"10",
|
||||
"--plan-only",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 2
|
||||
|
||||
def test_invalid_target(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"data",
|
||||
"gen-magpie",
|
||||
"--base",
|
||||
"m",
|
||||
"--provider",
|
||||
"ollama",
|
||||
"--target",
|
||||
"0",
|
||||
"--plan-only",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 2
|
||||
|
||||
def test_live_deferred_exits_3(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"data",
|
||||
"gen-magpie",
|
||||
"--base",
|
||||
"m",
|
||||
"--provider",
|
||||
"ollama",
|
||||
"--target",
|
||||
"10",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 3
|
||||
assert "0.69.1" in result.output or "deferred" in result.output.lower()
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source wiring
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_heavy_imports(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "magpie.py").read_text(encoding="utf-8")
|
||||
for forbidden in (
|
||||
"\nimport torch",
|
||||
"\nimport transformers",
|
||||
"\nimport anthropic",
|
||||
):
|
||||
assert forbidden not in src
|
||||
|
||||
def test_cli_registered(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
cli = (root / "soup_cli" / "commands" / "data.py").read_text(encoding="utf-8")
|
||||
assert "magpie" in cli.lower() or "gen-magpie" in cli
|
||||
|
||||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
|
|
@ -0,0 +1,416 @@
|
|||
"""v0.69.0 Part D — Persona-Hub diversity sampler."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
from soup_cli.utils import persona_hub
|
||||
|
||||
|
||||
def _write(path: Path, text: str) -> Path:
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Bundled personas
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBundledPersonas:
|
||||
def test_at_least_one(self) -> None:
|
||||
personas = persona_hub.list_bundled_personas()
|
||||
assert len(personas) >= 5
|
||||
assert all(isinstance(p, str) for p in personas)
|
||||
|
||||
def test_bundled_returns_tuple(self) -> None:
|
||||
personas = persona_hub.list_bundled_personas()
|
||||
assert isinstance(personas, tuple)
|
||||
|
||||
def test_bundled_immutable(self) -> None:
|
||||
before = persona_hub.list_bundled_personas()
|
||||
# tuples are immutable; verify identity stability + no mutator
|
||||
after = persona_hub.list_bundled_personas()
|
||||
assert before == after
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Persona / style validators
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidatePersona:
|
||||
def test_happy(self) -> None:
|
||||
assert persona_hub.validate_persona("a curious student") == "a curious student"
|
||||
|
||||
def test_empty(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.validate_persona("")
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError, match="null"):
|
||||
persona_hub.validate_persona("a\x00b")
|
||||
|
||||
def test_oversize(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.validate_persona("p" * 2048)
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
persona_hub.validate_persona(42)
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
persona_hub.validate_persona(True)
|
||||
|
||||
|
||||
class TestValidateStyle:
|
||||
def test_happy(self) -> None:
|
||||
assert persona_hub.validate_style("formal") == "formal"
|
||||
|
||||
def test_empty(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.validate_style("")
|
||||
|
||||
def test_oversize(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.validate_style("x" * 200)
|
||||
|
||||
def test_null_byte(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.validate_style("formal\x00")
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Sample matrix
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSamplePersonaMatrix:
|
||||
def test_happy(self) -> None:
|
||||
rows = persona_hub.sample_persona_matrix(
|
||||
prompts=["Explain X"],
|
||||
personas=["student", "engineer"],
|
||||
styles=["formal"],
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
assert len(rows) == 4
|
||||
assert all("prompt" in r and "persona" in r and "style" in r for r in rows)
|
||||
|
||||
def test_deterministic(self) -> None:
|
||||
kwargs = dict(
|
||||
prompts=["P1", "P2"],
|
||||
personas=["A", "B"],
|
||||
styles=["s1", "s2"],
|
||||
n=8,
|
||||
seed=42,
|
||||
)
|
||||
a = persona_hub.sample_persona_matrix(**kwargs)
|
||||
b = persona_hub.sample_persona_matrix(**kwargs)
|
||||
assert a == b
|
||||
|
||||
def test_different_seeds_diverge(self) -> None:
|
||||
a = persona_hub.sample_persona_matrix(
|
||||
prompts=["P1", "P2"],
|
||||
personas=["A", "B"],
|
||||
styles=["s1", "s2"],
|
||||
n=10,
|
||||
seed=1,
|
||||
)
|
||||
b = persona_hub.sample_persona_matrix(
|
||||
prompts=["P1", "P2"],
|
||||
personas=["A", "B"],
|
||||
styles=["s1", "s2"],
|
||||
n=10,
|
||||
seed=2,
|
||||
)
|
||||
# With 2x2x2=8 combinations across n=10 samples, the orderings
|
||||
# should differ for at least one cell.
|
||||
assert a != b
|
||||
|
||||
def test_empty_prompts_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=[],
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_empty_personas_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=["P"],
|
||||
personas=[],
|
||||
styles=["s"],
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_empty_styles_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=["P"],
|
||||
personas=["A"],
|
||||
styles=[],
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_n_zero_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=["P"],
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=0,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_n_overcap(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=["P"],
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=persona_hub._MAX_SAMPLES + 1,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_n_bool(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=["P"],
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=True,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
def test_seed_bool(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=["P"],
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=4,
|
||||
seed=False,
|
||||
)
|
||||
|
||||
def test_per_list_caps(self) -> None:
|
||||
big = ["x"] * (persona_hub._MAX_LIST_LEN + 1)
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.sample_persona_matrix(
|
||||
prompts=big,
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Topic diversity
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestComputeTopicDiversity:
|
||||
def test_high_diversity(self) -> None:
|
||||
rows = [
|
||||
{"text": "cat dog mouse"},
|
||||
{"text": "engine wheel car"},
|
||||
{"text": "ocean wave fish"},
|
||||
]
|
||||
score = persona_hub.compute_topic_diversity(rows)
|
||||
assert 0.0 <= score <= 1.0
|
||||
assert score > 0.5
|
||||
|
||||
def test_low_diversity_repeated_text(self) -> None:
|
||||
rows = [{"text": "same"} for _ in range(5)]
|
||||
score = persona_hub.compute_topic_diversity(rows)
|
||||
# entropy should be near 0 (single token, no variation)
|
||||
assert score < 0.5
|
||||
|
||||
def test_empty_returns_zero(self) -> None:
|
||||
assert persona_hub.compute_topic_diversity([]) == 0.0
|
||||
|
||||
def test_non_list_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
persona_hub.compute_topic_diversity("rows") # type: ignore[arg-type]
|
||||
|
||||
def test_rows_without_text_skipped(self) -> None:
|
||||
# No text fields anywhere → returns 0.0 (no signal).
|
||||
rows = [{"unrelated": "x"}, {"y": 1}]
|
||||
score = persona_hub.compute_topic_diversity(rows)
|
||||
assert score == 0.0
|
||||
|
||||
def test_score_finite(self) -> None:
|
||||
rows = [{"text": "a b c"}, {"text": "c d e"}]
|
||||
score = persona_hub.compute_topic_diversity(rows)
|
||||
assert math.isfinite(score)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# PersonaPlan + factory
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPersonaPlan:
|
||||
def test_frozen(self) -> None:
|
||||
plan = persona_hub.PersonaPlan(
|
||||
prompts=("P",),
|
||||
personas=("A",),
|
||||
styles=("s",),
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
plan.n = 999 # type: ignore[misc]
|
||||
|
||||
def test_tuples_required(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
persona_hub.PersonaPlan(
|
||||
prompts=["P"], # type: ignore[arg-type]
|
||||
personas=("A",),
|
||||
styles=("s",),
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
|
||||
class TestBuildPersonaPlan:
|
||||
def test_happy(self) -> None:
|
||||
plan = persona_hub.build_persona_plan(
|
||||
prompts=["P"],
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
assert plan.n == 4
|
||||
assert plan.prompts == ("P",)
|
||||
|
||||
def test_validators_propagate(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
persona_hub.build_persona_plan(
|
||||
prompts=[],
|
||||
personas=["A"],
|
||||
styles=["s"],
|
||||
n=4,
|
||||
seed=0,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CLI smoke
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPersonaMixCli:
|
||||
def test_help(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["data", "persona-mix", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_writes_output(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
prompts = _write(
|
||||
tmp_path / "prompts.jsonl",
|
||||
'{"prompt": "Explain X"}\n{"prompt": "Explain Y"}\n',
|
||||
)
|
||||
out = tmp_path / "out.jsonl"
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"data",
|
||||
"persona-mix",
|
||||
"--prompts",
|
||||
str(prompts),
|
||||
"--n",
|
||||
"4",
|
||||
"--output",
|
||||
str(out),
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert out.exists()
|
||||
lines = [line for line in out.read_text(encoding="utf-8").splitlines() if line]
|
||||
assert len(lines) == 4
|
||||
|
||||
def test_missing_prompts(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"data",
|
||||
"persona-mix",
|
||||
"--prompts",
|
||||
"nope.jsonl",
|
||||
"--n",
|
||||
"4",
|
||||
"--output",
|
||||
str(tmp_path / "o.jsonl"),
|
||||
],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_outside_cwd_prompts(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
_write(outside / "p.jsonl", '{"prompt": "X"}\n')
|
||||
sub = tmp_path / "sub"
|
||||
sub.mkdir()
|
||||
monkeypatch.chdir(sub)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"data",
|
||||
"persona-mix",
|
||||
"--prompts",
|
||||
str(outside / "p.jsonl"),
|
||||
"--n",
|
||||
"4",
|
||||
"--output",
|
||||
str(sub / "out.jsonl"),
|
||||
],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source wiring
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_heavy_imports(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "persona_hub.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
for forbidden in (
|
||||
"\nimport torch",
|
||||
"\nimport transformers",
|
||||
"\nimport sentence_transformers",
|
||||
):
|
||||
assert forbidden not in src
|
||||
|
||||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
|
|
@ -0,0 +1,418 @@
|
|||
"""v0.69.0 Part E — Brain-rot detector (arXiv 2510.13928)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from soup_cli.cli import app
|
||||
from soup_cli.utils import brain_rot
|
||||
|
||||
|
||||
def _write(path: Path, text: str) -> Path:
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Verdict allowlist
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBrainRotVerdicts:
|
||||
def test_exact(self) -> None:
|
||||
assert brain_rot.BRAIN_ROT_VERDICTS == ("OK", "MINOR", "MAJOR")
|
||||
|
||||
def test_immutable(self) -> None:
|
||||
# Tuples reject item assignment with TypeError.
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.BRAIN_ROT_VERDICTS[0] = "EVIL" # type: ignore[index]
|
||||
|
||||
|
||||
class TestClassifyBoundaries:
|
||||
"""TDD review H1 — exact-threshold boundary tests at ± epsilon."""
|
||||
|
||||
def test_just_below_ok_is_minor(self) -> None:
|
||||
assert brain_rot.classify_brain_rot(0.8499999) == "MINOR"
|
||||
|
||||
def test_just_below_minor_is_major(self) -> None:
|
||||
assert brain_rot.classify_brain_rot(0.5999999) == "MAJOR"
|
||||
|
||||
def test_exact_minor_threshold(self) -> None:
|
||||
assert brain_rot.classify_brain_rot(0.60) == "MINOR"
|
||||
|
||||
def test_exact_ok_threshold(self) -> None:
|
||||
assert brain_rot.classify_brain_rot(0.85) == "OK"
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Classify
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestClassifyBrainRot:
|
||||
def test_high_score_is_ok(self) -> None:
|
||||
# High score = healthy data, low brain-rot.
|
||||
assert brain_rot.classify_brain_rot(0.9) == "OK"
|
||||
assert brain_rot.classify_brain_rot(0.85) == "OK"
|
||||
|
||||
def test_mid_score_is_minor(self) -> None:
|
||||
assert brain_rot.classify_brain_rot(0.70) == "MINOR"
|
||||
assert brain_rot.classify_brain_rot(0.6) == "MINOR"
|
||||
|
||||
def test_low_score_is_major(self) -> None:
|
||||
assert brain_rot.classify_brain_rot(0.59) == "MAJOR"
|
||||
assert brain_rot.classify_brain_rot(0.0) == "MAJOR"
|
||||
|
||||
def test_boundary_thresholds(self) -> None:
|
||||
# Match v0.26 / v0.56 / v0.65 thresholds: >= 0.85 OK / >= 0.60 MINOR.
|
||||
assert brain_rot.classify_brain_rot(0.849) == "MINOR"
|
||||
assert brain_rot.classify_brain_rot(0.599) == "MAJOR"
|
||||
assert brain_rot.classify_brain_rot(1.0) == "OK"
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.classify_brain_rot(True)
|
||||
|
||||
def test_nan_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.classify_brain_rot(float("nan"))
|
||||
|
||||
def test_inf_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.classify_brain_rot(float("inf"))
|
||||
|
||||
def test_out_of_range_rejected(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.classify_brain_rot(1.5)
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.classify_brain_rot(-0.1)
|
||||
|
||||
def test_non_number(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.classify_brain_rot("0.5") # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Triviality + popularity heuristics
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestScoreTriviality:
|
||||
def test_substantive_low(self) -> None:
|
||||
# Long, varied, substantive text → low triviality.
|
||||
text = (
|
||||
"The mitochondrion is the powerhouse of the cell because it "
|
||||
"converts nutrients into ATP through oxidative phosphorylation."
|
||||
)
|
||||
score = brain_rot.score_triviality(text)
|
||||
assert 0.0 <= score <= 1.0
|
||||
assert score < 0.5
|
||||
|
||||
def test_trivial_high(self) -> None:
|
||||
# Short, repetitive, exclamation-heavy → high triviality.
|
||||
text = "lol!!!! omg!!!! lol omg!!! lol!!!"
|
||||
score = brain_rot.score_triviality(text)
|
||||
assert score > 0.5
|
||||
|
||||
def test_empty_returns_one(self) -> None:
|
||||
assert brain_rot.score_triviality("") == 1.0
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.score_triviality(42) # type: ignore[arg-type]
|
||||
|
||||
def test_bool_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.score_triviality(True)
|
||||
|
||||
|
||||
class TestScorePopularitySignal:
|
||||
def test_substantive_low(self) -> None:
|
||||
text = "Detailed scientific explanation of photosynthesis cycles."
|
||||
score = brain_rot.score_popularity_signal(text)
|
||||
assert score < 0.5
|
||||
|
||||
def test_slop_high(self) -> None:
|
||||
text = "click here for the top 10 you won't believe what happened next"
|
||||
score = brain_rot.score_popularity_signal(text)
|
||||
assert score > 0.5
|
||||
|
||||
def test_empty_zero(self) -> None:
|
||||
assert brain_rot.score_popularity_signal("") == 0.0
|
||||
|
||||
def test_non_string(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.score_popularity_signal(42) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Per-row + dataset scoring
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestScoreRowBrainRot:
|
||||
def test_substantive(self) -> None:
|
||||
row = {"text": "Long substantive paragraph with diverse vocabulary."}
|
||||
score = brain_rot.score_row_brain_rot(row)
|
||||
assert 0.0 <= score <= 1.0
|
||||
assert score > 0.5
|
||||
|
||||
def test_slop(self) -> None:
|
||||
row = {"text": "lol!!!! omg!!! top 10 you won't believe"}
|
||||
score = brain_rot.score_row_brain_rot(row)
|
||||
assert score < 0.5
|
||||
|
||||
def test_missing_text_returns_zero(self) -> None:
|
||||
# No text signal → score 0.0 (cannot judge → treat as MAJOR).
|
||||
assert brain_rot.score_row_brain_rot({}) == 0.0
|
||||
|
||||
def test_non_dict_rejected(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.score_row_brain_rot("not a dict") # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestBrainRotReport:
|
||||
def test_frozen(self) -> None:
|
||||
report = brain_rot.BrainRotReport(
|
||||
num_rows=10,
|
||||
mean_score=0.5,
|
||||
num_major=2,
|
||||
num_minor=3,
|
||||
num_ok=5,
|
||||
overall_verdict="MINOR",
|
||||
)
|
||||
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||
report.mean_score = 0.9 # type: ignore[misc]
|
||||
|
||||
def test_validates_counts(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.BrainRotReport(
|
||||
num_rows=-1,
|
||||
mean_score=0.5,
|
||||
num_major=0,
|
||||
num_minor=0,
|
||||
num_ok=0,
|
||||
overall_verdict="OK",
|
||||
)
|
||||
|
||||
def test_validates_verdict(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.BrainRotReport(
|
||||
num_rows=1,
|
||||
mean_score=0.5,
|
||||
num_major=0,
|
||||
num_minor=0,
|
||||
num_ok=1,
|
||||
overall_verdict="BOGUS",
|
||||
)
|
||||
|
||||
def test_mean_score_finite(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.BrainRotReport(
|
||||
num_rows=1,
|
||||
mean_score=float("nan"),
|
||||
num_major=0,
|
||||
num_minor=0,
|
||||
num_ok=1,
|
||||
overall_verdict="OK",
|
||||
)
|
||||
|
||||
def test_mean_score_range(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.BrainRotReport(
|
||||
num_rows=1,
|
||||
mean_score=1.5,
|
||||
num_major=0,
|
||||
num_minor=0,
|
||||
num_ok=1,
|
||||
overall_verdict="OK",
|
||||
)
|
||||
|
||||
|
||||
class TestScoreDatasetBrainRot:
|
||||
def test_clean_dataset(self) -> None:
|
||||
rows = [
|
||||
{"text": "Detailed scientific explanation of photosynthesis."},
|
||||
{"text": "Comprehensive overview of cellular respiration cycles."},
|
||||
]
|
||||
report = brain_rot.score_dataset_brain_rot(rows)
|
||||
assert report.num_rows == 2
|
||||
assert math.isfinite(report.mean_score)
|
||||
assert report.overall_verdict in brain_rot.BRAIN_ROT_VERDICTS
|
||||
|
||||
def test_sloppy_dataset(self) -> None:
|
||||
rows = [{"text": "lol!!!! omg top 10"} for _ in range(5)]
|
||||
report = brain_rot.score_dataset_brain_rot(rows)
|
||||
# Slop-heavy → MAJOR or MINOR
|
||||
assert report.overall_verdict in ("MAJOR", "MINOR")
|
||||
|
||||
def test_empty(self) -> None:
|
||||
report = brain_rot.score_dataset_brain_rot([])
|
||||
assert report.num_rows == 0
|
||||
assert report.mean_score == 0.0
|
||||
assert report.overall_verdict == "MAJOR"
|
||||
|
||||
def test_non_iterable(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.score_dataset_brain_rot(42) # type: ignore[arg-type]
|
||||
|
||||
def test_non_dict_row_skipped(self) -> None:
|
||||
rows = [
|
||||
{"text": "good substantive content"},
|
||||
"not a dict",
|
||||
{"text": "more good content"},
|
||||
]
|
||||
report = brain_rot.score_dataset_brain_rot(rows)
|
||||
# Only 2 dict rows count.
|
||||
assert report.num_rows == 2
|
||||
|
||||
def test_overall_minor_band(self) -> None:
|
||||
# Custom mix to land in MINOR band.
|
||||
rows = (
|
||||
[{"text": "lol!!! omg!!!"} for _ in range(3)]
|
||||
+ [
|
||||
{"text": "Detailed scientific overview of cellular respiration."}
|
||||
for _ in range(2)
|
||||
]
|
||||
)
|
||||
report = brain_rot.score_dataset_brain_rot(rows)
|
||||
assert math.isfinite(report.mean_score)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# refuse_if_rotten
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRefuseIfRotten:
|
||||
def test_clean_passes(self) -> None:
|
||||
rows = [
|
||||
{"text": "Detailed scientific overview of cellular biology."}
|
||||
for _ in range(3)
|
||||
]
|
||||
# Should not raise.
|
||||
brain_rot.refuse_if_rotten(rows, max_major_fraction=0.5)
|
||||
|
||||
def test_too_rotten_raises(self) -> None:
|
||||
rows = [{"text": "lol!!! omg!!!"} for _ in range(5)]
|
||||
with pytest.raises(ValueError, match="brain.?rot"):
|
||||
brain_rot.refuse_if_rotten(rows, max_major_fraction=0.1)
|
||||
|
||||
def test_invalid_threshold(self) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
brain_rot.refuse_if_rotten([], max_major_fraction=1.5)
|
||||
|
||||
def test_bool_threshold(self) -> None:
|
||||
with pytest.raises(TypeError):
|
||||
brain_rot.refuse_if_rotten([], max_major_fraction=True)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CLI: `soup data brain-rot`
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBrainRotCli:
|
||||
def test_help(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["data", "brain-rot", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "brain" in result.output.lower()
|
||||
|
||||
def test_clean(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = _write(
|
||||
tmp_path / "clean.jsonl",
|
||||
"\n".join(
|
||||
json.dumps({"text": "Detailed scientific exposition number " + str(i)})
|
||||
for i in range(5)
|
||||
)
|
||||
+ "\n",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["data", "brain-rot", str(path)])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
def test_sloppy_exits_3(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = _write(
|
||||
tmp_path / "slop.jsonl",
|
||||
"\n".join(json.dumps({"text": "lol!!! omg!!!"}) for _ in range(10))
|
||||
+ "\n",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app, ["data", "brain-rot", str(path), "--strict"]
|
||||
)
|
||||
assert result.exit_code == 3
|
||||
|
||||
def test_missing_input(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["data", "brain-rot", "nope.jsonl"])
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_outside_cwd(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
_write(outside / "d.jsonl", json.dumps({"text": "x"}) + "\n")
|
||||
sub = tmp_path / "sub"
|
||||
sub.mkdir()
|
||||
monkeypatch.chdir(sub)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
app, ["data", "brain-rot", str(outside / "d.jsonl")]
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
|
||||
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX symlink")
|
||||
def test_symlink_rejected(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
target = _write(
|
||||
tmp_path / "real.jsonl", json.dumps({"text": "x"}) + "\n"
|
||||
)
|
||||
link = tmp_path / "link.jsonl"
|
||||
os.symlink(str(target), str(link))
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(app, ["data", "brain-rot", str(link)])
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Source wiring
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSourceWiring:
|
||||
def test_no_heavy_imports(self) -> None:
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
src = (root / "soup_cli" / "utils" / "brain_rot.py").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
for forbidden in (
|
||||
"\nimport torch",
|
||||
"\nimport transformers",
|
||||
"\nimport sentence_transformers",
|
||||
):
|
||||
assert forbidden not in src
|
||||
|
||||
def test_version_bumped(self) -> None:
|
||||
from soup_cli import __version__
|
||||
|
||||
assert __version__ == "0.69.0"
|
||||
Loading…
Reference in New Issue