diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b43234..eea6bce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,65 @@ reproducing 70+ versions of notes. ## [Unreleased] +## [0.71.35] - 2026-07-15 + +### Added +- **Compliance templates — `soup init --template hipaa|soc2|eu-ai-act|sr-11-7`.** + Four regulation-shaped starting configs. Soup's compliance controls are CLI + flags/commands rather than config keys, so each template is a valid training + config plus header comments naming the exact commands for that regime + (PHI scrubbing + air-gap for HIPAA, BOM/attest/sign for SOC 2, Annex XI + + energy tracking for the EU AI Act, repro-receipt + diagnose/ship for SR 11-7). + Templates default to a license-clean Apache-2.0 base. +- **`soup card -o MODELCARD.md` — model-card autogen.** Turns a + Local Model Registry entry into a publishable, provenance-carrying HF model + card: base model, training config, eval scorecard, config/data hashes, + lineage (ancestors) and a table of every registered artifact. Adapter vs + full-model is inferred from registered artifacts, falling back to the training + config (LoRA rank, with Spectrum/LISA full-FT correctly treated as dense), so + the card sets the right `library_name` and never misreports the model type. +- **`soup push --card `** — render that registry-driven card and + upload it as `README.md`, overriding the auto-generated one. A bad ref fails + fast before any network call; HF hub only. +- **`soup ci init` — fine-tuning CI.** Writes `.github/workflows/soup-gate.yml`, + a PR gate chaining `soup data validate` → `soup expect` → + `soup ship --evidence` (exit 2 blocks the merge). Every interpolated path is + validated to stay under the repo root and shell-quoted; the branch and Python + version are regex-gated; the write is atomic, symlink-rejecting, and refuses + to clobber an existing workflow without `--force`. +- **Compliance quickstart** — a new [docs/compliance.md](docs/compliance.md) + walkthrough: template → PII scrub → train with receipt/Annex XI/energy → + registry → BOM + attestation → scan/sign/verify → air-gap → model card → CI gate. + +### Fixed +- **GGUF export now actually works on Windows** (validated end-to-end against a + locally-built llama.cpp: SmolLM2-135M → q4_0 / q4_k_m / q8_0 / f16 → `soup deploy + ollama` → live inference). Four real bugs, each of which independently broke the + path: + - **`soup export --format gguf` cloned llama.cpp into your current directory.** + `SOUP_DIR` is the bare name `.soup`, but the lookup used it relatively rather + than anchoring to `~` like the rest of the codebase — so the canonical + `~/.soup/llama.cpp` was never found and a fresh ~200 MB checkout was dropped + into whatever directory you ran from. + - **The first GGUF export downgraded your PyTorch and broke CUDA.** The auto-clone + ran `pip install -r /requirements.txt` into your interpreter, and + llama.cpp pins `torch~=2.2.1` against the CPU wheel index (observed: + torch 2.5.1+cu → 2.2.2+cpu, transformers 4.57 → 4.46). Soup now installs only + the convert script's extra dependencies, unpinned, and never touches torch. + - **A correctly-built llama.cpp was not found on Windows.** MSVC (like Xcode) is a + multi-config generator and emits `build/bin/Release/llama-quantize.exe`; only the + flat single-config layout was searched. + - **`soup deploy ollama` failed on a relative GGUF path** with "pull model manifest: + file does not exist" — Ollama resolves `FROM` against the Modelfile's directory, + and Soup writes the Modelfile to a temp dir. The Modelfile now emits an absolute path. +- **Model-card injection hardening (affects the pre-existing `soup push` card too).** + The `## Training` section interpolated `base` / `task` / `scheduler` / `recipe` + unescaped. Since `SoupConfig.base` and `scheduler` have no charset validator, a + crafted-but-valid config could smuggle raw HTML — or a backtick breaking out of + the surrounding code span — into a card published to the Hub. All values now go + through the markdown escaper, which additionally neutralises backticks and + strips C0/ESC control bytes. + ## [0.71.34] - 2026-07-15 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 24d0093..63220e7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -117,10 +117,10 @@ src/soup_cli/ data/traces/ - Trace-to-Preference harvester (v0.26.0) data/collators.py - CrossDocCollator for sample packing (v0.33.0) utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve - templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0) + templates/ - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35) ui/ - Web UI (FastAPI + HTML/JS SPA) -tests/ - Test suite (312 files, 15906 tests) +tests/ - Test suite (313 files, 16001 tests) examples/ - Real-world config examples and datasets ``` diff --git a/README.md b/README.md index 2bcabef..0a76564 100644 --- a/README.md +++ b/README.md @@ -49,18 +49,28 @@ infrastructure instead of improving models. Soup fixes that. ## What's New -**v0.71.34 — adapter algebra + LISA.** Compose fine-tunes like vectors, and train big-model quality on a small-model memory budget. +**v0.71.35 — the compliance pack.** Ship a regulated fine-tune with the paperwork it needs: start from a regulation-shaped config, publish a provenance-carrying model card, and gate every PR in CI. -- **`soup adapters arithmetic "coder + 0.5*math - toxic"`.** Task-vector algebra over LoRA - adapters (arXiv:2212.04089) — **add, scale, and NEGATE** trained behaviours into one merged - adapter. The math is done right: a LoRA's effective delta `ΔW = B·A` scales *linearly* with - each coefficient (subtracting an adapter actually removes its behaviour), not quadratically. - Same-base + backdoor-scan gated; mixed ranks refused with a clear message. -- **LISA — layerwise importance sampling (`training.lisa_enabled`).** Full-fine-tuning quality - at LoRA-like memory (arXiv:2403.17919): every N steps LISA re-activates a small random set of - decoder layers (embeddings + head always on) and freezes the rest. Live on a 4 GB GPU. -- **Verified on real models (RTX 3050).** LISA trains SmolLM2-135M end-to-end; `2·a` produces an - adapter whose ΔW is *exactly* 2× the original's — the merged adapter loads and serves. +- **`soup init --template hipaa|soc2|eu-ai-act|sr-11-7`.** Four regulation-shaped starting + configs, each documenting the exact commands for that regime — PHI scrubbing + air-gap, + BOM/attest/sign, EU Annex XI + energy tracking, or repro-receipt + diagnose/ship. +- **`soup card ` → `MODELCARD.md`.** Turn a registry entry into a publishable HF + model card: training config, eval scorecard, config/data hashes, lineage, and every + registered artifact. `soup push --card` uploads it as the README, so every public push + carries its provenance. +- **`soup ci init` — fine-tuning CI.** Writes a GitHub Actions gate that runs + `data validate` → `expect` → `ship --evidence` on each PR; a regression exits 2 and + blocks the merge. +- **Compliance quickstart** ([docs/compliance.md](docs/compliance.md)) — the whole path from + template to air-gapped, signed, attested, carded model. +- **GGUF export fixed on Windows.** Validated end-to-end for the first time (SmolLM2 → + q4_0/q4_k_m/q8_0/f16 → Ollama → live inference), which surfaced four real bugs: the export + cloned llama.cpp into your *current directory*, its first run **downgraded your PyTorch to + CPU-only** (breaking CUDA), a correctly-built llama.cpp wasn't found under MSVC, and + `deploy ollama` rejected relative GGUF paths. All fixed. +- **Hardening:** the model-card generator (including the existing `soup push` card) no longer + interpolates unescaped config values — a crafted `base`/`scheduler` can't smuggle HTML into + a card published on the Hub. ```bash soup adapters arithmetic "coder - toxic" \ diff --git a/docs/README.md b/docs/README.md index 867a34a..702b047 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,6 +14,7 @@ feature reference — every `soup` capability, grouped by area. | [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` | | [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge | | [Adapters, registry & governance](adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls | +| [Compliance & governance quickstart](compliance.md) | HIPAA/SOC2/EU-AI-Act/SR-11-7 `init` templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (`soup card`), CI gate (`soup ci init`) | | [Backends, platform & ops](backends-and-ops.md) | MLX/Unsloth backends, Modal cloud GPU training, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands | | [Command reference](commands.md) | The full `soup` command list | | [Supported models & extras](models.md) | Recommended model families, the VRAM size guide, the pip extras matrix | diff --git a/docs/commands.md b/docs/commands.md index c42cc34..1fd7500 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -8,6 +8,7 @@ ``` soup init [--template chat|code|...|audio] Create config +soup init --template hipaa|soc2|eu-ai-act|sr-11-7 Compliance-shaped starting config + the commands for that regime (v0.71.35) soup autopilot --model --data d.jsonl --goal Zero-configsoup train --config soup.yaml Start training soup train --config soup.yaml --tensorboard Train with TensorBoard logging soup train --config soup.yaml --fsdp full_shard Train with FSDP2 @@ -159,6 +160,9 @@ soup ship --evidence ev.json [--output v.json] Decide offline from pre-computed soup ship ... --task-mode judge_score --judge-model ollama://llama3.1 Leg-1 via LLM-as-a-judge soup ship ... --task-mode pairwise --judge-model ollama://llama3.1 Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31) soup ship ... --general-suite mmlu,gsm8k --baseline base.json lm-eval leg-2 + recorded base scores +soup card -o MODELCARD.md HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35) +soup push --model ./out --repo you/m --card Upload that registry-driven card as the README (HF only) (v0.71.35) +soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--branch main --python 3.11] [--force] Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35) soup mcp serve MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28) soup mcp serve --allow-mutating Also expose plan-only train_start / export tools (never execute) (v0.71.28) soup shrink --model --drop-ratio 0.25 --calib c.jsonl -o shrunk Depth-prune least-important layer block + SHIP/DON'T-SHIP ppl verdict (exit 0/2/1) (v0.71.29) diff --git a/docs/compliance.md b/docs/compliance.md new file mode 100644 index 0000000..22acafe --- /dev/null +++ b/docs/compliance.md @@ -0,0 +1,136 @@ +# Compliance & governance quickstart + +[← Back to the docs index](README.md) + +Soup ships an end-to-end compliance workflow: start from a regulation-shaped +config, train with provenance capture, sign + attest the artifact, keep an audit +trail, and move it across an air gap — then publish a documented model card and +gate future changes in CI. + +- **Pre-wired configs:** `soup init --template hipaa|soc2|eu-ai-act|sr-11-7` +- **Provenance:** `soup train --repro-receipt` · `soup bom emit` · `soup attest emit` +- **Integrity:** `soup adapters sign` / `verify` · `soup adapters scan` +- **Audit trail:** the HIPAA/SOC2 audit log is on by default (`soup audit-log tail`) +- **Air gap:** `soup airgap-bundle` +- **Publish:** `soup card ` → `MODELCARD.md` (or `soup push --card`) +- **CI gate:** `soup ci init` → a PR workflow that runs validate → expect → ship + +--- + +## 1. Start from a compliance template + +Each template is a normal training config plus header comments listing the exact +compliance commands to run around it. Pick the regime you operate under: + +```bash +soup init --template hipaa # Protected Health Information +soup init --template soc2 # SOC 2 Trust Services Criteria +soup init --template eu-ai-act # EU AI Act Annex XI/XII +soup init --template sr-11-7 # SR 11-7 Model Risk Management +``` + +The compliance controls are Soup **CLI flags/commands, not config keys** — the +template header documents which ones apply. The steps below are the common path. + +## 2. Clean the data before training + +```bash +soup data pii ./data/train.jsonl # flag emails / phones / SSNs / MRNs +soup data decontaminate ./data/train.jsonl # drop public-benchmark overlap +``` + +## 3. Train with a reproducibility receipt (+ Annex XI / energy for the EU) + +```bash +# SR 11-7 / SOC 2 / HIPAA: capture seeds, kernels, GPU, OS +soup train --config soup.yaml --repro-receipt receipt.json + +# EU AI Act: auto-generate the Annex XI/XII documentation + measure energy +soup train --config soup.yaml \ + --annex-xi annex_xi.md \ + --track-energy --energy-country DEU --energy-out energy.json +``` + +The audit log records every command automatically: + +```bash +soup audit-log tail # review the trail +soup audit-log rotate # force a rotation pass +``` + +## 4. Register the run, then emit BOM + attestation + +```bash +soup registry push --run-id --name my-model --tag v1 + +soup bom emit --name my-model --base-sha --config-sha \ + --energy energy.json --format both # CycloneDX + SPDX +soup attest emit --stage train --subject my-model --sha \ + --sign ed25519 --key key.pem # in-toto + SLSA-3 +``` + +## 5. Sign, scan, and verify the artifact + +```bash +soup adapters scan ./output # weight-space backdoor scan +soup adapters sign ./output --backend ed25519 --generate-key key.pem +soup adapters verify ./output --strict --public-key key.pub.pem +``` + +## 6. Air-gap transfer (optional) + +```bash +soup airgap-bundle --model ./output --output my-model.tar --repro-receipt receipt.json +``` + +## 7. Generate a documented model card + +Turn the registry entry into a provenance-rich `MODELCARD.md` — base model, +training config, eval scorecard, config/data hashes, lineage, and every +registered artifact: + +```bash +soup card my-model:v1 -o MODELCARD.md +# or, when uploading to the Hub, override the auto-generated card: +soup push --model ./output --repo you/my-model --card my-model:v1 +``` + +## 8. Gate future changes in CI + +Write a GitHub Actions workflow that blocks a PR unless the data validates, the +expectations suite passes, and the SHIP verdict is green: + +```bash +soup ci init --data data/train.jsonl --suite expectations.yaml --evidence ship_evidence.json +# writes .github/workflows/soup-gate.yml +``` + +The generated job runs, in order: + +``` +soup data validate # dataset format compliance +soup expect # PII / token-length / refusal / judge expectations +soup ship --evidence # SHIP / DON'T-SHIP (exit 2 blocks the merge) +``` + +A minimal `expectations.yaml` for the second step: + +```yaml +expectations: + - name: expect_no_pii + - name: expect_token_length_between + min_tokens: 1 + max_tokens: 512 +``` + +Supported names: `expect_no_pii`, `expect_token_length_between`, +`expect_no_refusal_pattern`, `expect_chosen_preferred_over_rejected_by_judge`. + +Every path is shell-quoted and validated to stay under the repo root, so the +rendered workflow is injection-safe. Edit the paths to match your repo. + +--- + +See also: [Adapters, registry & governance](adapters-and-governance.md) for the +full supply-chain command set, and [Evaluation & probes](evaluation.md) for the +`soup ship` verdict engine. diff --git a/pyproject.toml b/pyproject.toml index 57493de..f572ab2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "soup-cli" -version = "0.71.34" +version = "0.71.35" description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell." readme = "README.md" license = "Apache-2.0" diff --git a/src/soup_cli/__init__.py b/src/soup_cli/__init__.py index f2e9d14..92a9a9b 100644 --- a/src/soup_cli/__init__.py +++ b/src/soup_cli/__init__.py @@ -1,3 +1,3 @@ """Soup CLI — Fine-tune and post-train LLMs in one command.""" -__version__ = "0.71.34" +__version__ = "0.71.35" diff --git a/src/soup_cli/cli.py b/src/soup_cli/cli.py index 64a2138..eaefdca 100644 --- a/src/soup_cli/cli.py +++ b/src/soup_cli/cli.py @@ -21,7 +21,9 @@ from soup_cli.commands import ( # noqa: E402 autopilot, bench, can, + card, chat, + ci, cost, data, deploy, @@ -98,6 +100,8 @@ app.command()(cost.cost) app.command()(push.push) app.command(name="export")(export.export) app.command()(merge.merge) +app.command(name="card")(card.card) +app.add_typer(ci.app, name="ci", help="Fine-tuning CI: init a PR gate workflow.") app.add_typer( data.app, name="data", help="Dataset tools: inspect, convert, merge, dedup, validate, stats.", diff --git a/src/soup_cli/commands/card.py b/src/soup_cli/commands/card.py new file mode 100644 index 0000000..ae759f0 --- /dev/null +++ b/src/soup_cli/commands/card.py @@ -0,0 +1,316 @@ +"""soup card — generate a HuggingFace model card from a registry entry (v0.71.35). + +Turns a Local Model Registry entry into a ready-to-publish ``MODELCARD.md``: +base model, training config, eval scorecard, provenance (config/data hashes), +lineage (ancestors), and a table of every registered artifact (adapter / merged +model / GGUF / diagnose report / eval results). Every public push becomes a +documented, provenance-carrying release. + +Design notes: + * ``build_model_card`` is a pure function over plain dicts (no store / no + torch) so it is trivially unit-testable and reusable by ``soup push --card``. + * User-controlled text is escaped for its output target: registry ``notes`` is + free-form -> ``html.escape`` (blocks script/``javascript:`` injection on the + HF README viewer); markdown table cells -> ``_safe_md_cell`` (reused from + ``push.py``); YAML frontmatter scalars -> ``_yaml_dq`` (control-char strip + + double-quote escaping so a hostile ``base_model`` cannot inject sibling keys). + * Output is written through ``atomic_write_text`` (cwd-contained, + symlink-rejected), consistent with ``write_bom`` / ``write_attestation``. +""" + +from __future__ import annotations + +import html +import json +from typing import Any, Mapping, Sequence + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel + +from soup_cli.commands.push import ( + _render_eval_scorecard, + _render_training_section, + _safe_md_cell, +) +from soup_cli.registry.store import AmbiguousRefError, RegistryStore +from soup_cli.utils.paths import atomic_write_text + +console = Console() + +# Bounds on registry-sourced content (v0.71.35 security review). The registry +# places no length cap on `notes` / `base_model` and no row cap on artifacts, so +# a pathological entry could otherwise render an unbounded card. +_MAX_NOTES_CHARS = 8_000 +_MAX_ROWS = 200 + + +class CardError(ValueError): + """Raised when a registry entry cannot be resolved for a card.""" + + +def _truncate(text: str, limit: int) -> str: + return text if len(text) <= limit else text[:limit] + "\n\n[truncated]" + + +# Export kinds that prove the artifact is a standalone (non-adapter) model. +_DENSE_KINDS = frozenset({"merged", "gguf", "onnx", "awq", "gptq", "tensorrt", "edited_model"}) + + +def _is_adapter(artifacts: Sequence[Mapping[str, Any]], config: Mapping[str, Any]) -> bool: + """Decide whether the entry describes a LoRA adapter or a standalone model. + + A registered artifact is definitive; otherwise fall back to the training + config. This matters because the card is a provenance document — claiming + "Full model" for a LoRA (and emitting ``library_name: transformers`` instead + of ``peft``) is a false statement + a broken HF Hub card. Found by the + v0.71.35 Step-6 smoke: a real LoRA run with no artifacts attached rendered + as "Full model". + """ + kinds = {a.get("kind") for a in artifacts or []} + if "adapter" in kinds: + return True + if kinds & _DENSE_KINDS: + return False + training = config.get("training") or {} + if not isinstance(training, dict): + return False + # Spectrum / LISA are full-FT: LoRA is off even though the dumped config + # still carries a default `lora` block. + if training.get("unfrozen_parameters") or training.get("lisa_enabled"): + return False + lora = training.get("lora") or {} + if not isinstance(lora, dict): + return False + try: + return int(lora.get("r") or 0) > 0 + except (TypeError, ValueError): + return False + + +def _yaml_dq(value: object) -> str: + """Serialise ``value`` as a YAML double-quoted scalar, stripping control + chars and escaping ``\\`` / ``"`` so it cannot break out of the frontmatter. + """ + text = "".join(ch for ch in str(value) if ord(ch) >= 0x20) + text = text.replace("\\", "\\\\").replace('"', '\\"') + return f'"{text}"' + + +def build_model_card( + entry: Mapping[str, Any], + artifacts: Sequence[Mapping[str, Any]], + eval_results: Sequence[Mapping[str, Any]], + ancestors: Sequence[Mapping[str, Any]], +) -> str: + """Render a full HF model card (markdown) from a registry entry + joins. + + ``entry`` is a hydrated registry row; ``artifacts`` / ``eval_results`` / + ``ancestors`` are the corresponding ``RegistryStore`` joins (may be empty). + """ + name = str(entry.get("name") or "model") + base = str(entry.get("base_model") or "") + task = str(entry.get("task") or "sft") + notes = entry.get("notes") or "" + tags = entry.get("tags") or [] + created = str(entry.get("created_at") or "") + config_hash = str(entry.get("config_hash") or "") + data_hash = str(entry.get("data_hash") or "") + run_id = str(entry.get("run_id") or "") + + try: + config = json.loads(entry.get("config_json") or "{}") + if not isinstance(config, dict): + config = {} + except (TypeError, ValueError): + config = {} + training_cfg = dict(config) + training_cfg.setdefault("base", base) + training_cfg.setdefault("task", task) + + # Row caps: bound the card regardless of how many rows the registry holds. + artifacts = list(artifacts or [])[:_MAX_ROWS] + ancestors = list(ancestors or [])[:_MAX_ROWS] + eval_results = list(eval_results or [])[:_MAX_ROWS] + + scorecard: dict[str, Any] = {} + for row in eval_results or []: + bench = row.get("benchmark") + score = row.get("score") + if bench is not None and score is not None: + scorecard[str(bench)] = score + + is_adapter = _is_adapter(artifacts, config) + + # --- YAML frontmatter --------------------------------------------------- # + fm: list[str] = ["---", f"library_name: {'peft' if is_adapter else 'transformers'}"] + if base: + fm.append(f"base_model: {_yaml_dq(base)}") + fm.append("tags:") + for tag in ("soup-cli", "fine-tuned", "compliance"): + fm.append(f" - {tag}") + fm.append("---") + + lines: list[str] = list(fm) + lines += ["", f"# {_safe_md_cell(name)}", ""] + lines.append("Fine-tuned with [Soup CLI](https://github.com/MakazhanAlpamys/Soup).") + lines.append("") + + # --- Model details ------------------------------------------------------ # + lines += ["## Model Details", "", "| Field | Value |", "| --- | --- |"] + detail_rows = [ + ("Name", name), + ("Base model", base), + ("Task", task), + ("Type", "LoRA adapter" if is_adapter else "Full model"), + ] + if run_id: + detail_rows.append(("Run id", run_id)) + if created: + detail_rows.append(("Created", created)) + if tags: + detail_rows.append(("Tags", ", ".join(str(t) for t in tags))) + for key, val in detail_rows: + lines.append(f"| {key} | {_safe_md_cell(val)} |") + lines.append("") + + # --- Training ----------------------------------------------------------- # + training_section = _render_training_section(training_cfg) + if training_section: + lines.append(training_section) + + # --- Evaluation --------------------------------------------------------- # + eval_section = _render_eval_scorecard(scorecard) + if eval_section: + lines.append(eval_section) + + # --- Provenance --------------------------------------------------------- # + if config_hash or data_hash: + lines += ["## Provenance", "", "| Hash | Value |", "| --- | --- |"] + if config_hash: + lines.append(f"| config sha256 | `{_safe_md_cell(config_hash)}` |") + if data_hash: + lines.append(f"| data sha256 | `{_safe_md_cell(data_hash)}` |") + lines.append("") + + # --- Lineage ------------------------------------------------------------ # + if ancestors: + lines += ["## Lineage", "", "| Ancestor | Relation | Entry id |", "| --- | --- | --- |"] + for anc in ancestors: + lines.append( + f"| {_safe_md_cell(anc.get('name', ''))} " + f"| {_safe_md_cell(anc.get('relation', ''))} " + f"| `{_safe_md_cell(str(anc.get('id', ''))[:24])}` |" + ) + lines.append("") + + # --- Artifacts (provenance / compliance) -------------------------------- # + if artifacts: + lines += [ + "## Artifacts", + "", + "| Kind | File | SHA256 (prefix) |", + "| --- | --- | --- |", + ] + for art in artifacts: + path = str(art.get("path", "")) + filename = path.replace("\\", "/").rsplit("/", 1)[-1] + sha = str(art.get("sha256", ""))[:16] + lines.append( + f"| {_safe_md_cell(art.get('kind', ''))} " + f"| {_safe_md_cell(filename)} " + f"| `{_safe_md_cell(sha)}` |" + ) + lines.append("") + + # --- Notes -------------------------------------------------------------- # + if notes: + lines += ["## Notes", "", _truncate(html.escape(str(notes)), _MAX_NOTES_CHARS), ""] + + # --- Usage -------------------------------------------------------------- # + lines += ["## Usage", ""] + if is_adapter: + lines += [ + "```python", + "from peft import PeftModel", + "from transformers import AutoModelForCausalLM, AutoTokenizer", + "", + # json.dumps so a base_model containing a quote cannot produce a + # syntactically broken snippet (the value is inert docs, never run). + f"base = {json.dumps(_safe_md_cell(base) or 'BASE_MODEL')}", + "model = AutoModelForCausalLM.from_pretrained(base)", + 'model = PeftModel.from_pretrained(model, "YOUR_REPO")', + "tokenizer = AutoTokenizer.from_pretrained(base)", + "```", + "", + ] + else: + lines += [ + "```python", + "from transformers import AutoModelForCausalLM, AutoTokenizer", + "", + 'model = AutoModelForCausalLM.from_pretrained("YOUR_REPO")', + 'tokenizer = AutoTokenizer.from_pretrained("YOUR_REPO")', + "```", + "", + ] + + return "\n".join(lines) + "\n" + + +def build_card_for_ref(ref: str) -> str: + """Resolve ``ref`` in the registry and render its model card. + + Raises :class:`CardError` on ambiguous / missing refs (so callers such as + ``soup push --card`` can map it to an exit code). + """ + with RegistryStore() as store: + try: + eid = store.resolve(ref) + except AmbiguousRefError as exc: + raise CardError(str(exc)) from exc + if eid is None: + raise CardError(f"Registry entry not found: {ref}") + entry = store.get(eid) + if entry is None: + raise CardError(f"Registry entry not found: {ref}") + artifacts = store.get_artifacts(eid) + evals = store.get_eval_results(eid) + ancestors = store.get_ancestors(eid) + return build_model_card(entry, artifacts, evals, ancestors) + + +def card( + ref: str = typer.Argument(..., help="Registry entry id / prefix / name:tag"), + output: str = typer.Option( + "MODELCARD.md", + "-o", + "--output", + help="Output markdown path (must stay under the current directory)", + ), +) -> None: + """Generate a HuggingFace model card from a registry entry.""" + try: + markdown = build_card_for_ref(ref) + except CardError as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + + try: + written = atomic_write_text(markdown, output, field="card output") + except (ValueError, OSError) as exc: + console.print(f"[red]Cannot write card: {escape(str(exc))}[/]") + raise typer.Exit(1) from exc + + console.print( + Panel( + f"Model card written to [bold]{escape(written)}[/]", + title="soup card", + border_style="green", + ) + ) + + +# Optional re-exports for callers that want the raw pieces. +__all__ = ["build_model_card", "build_card_for_ref", "card", "CardError"] diff --git a/src/soup_cli/commands/ci.py b/src/soup_cli/commands/ci.py new file mode 100644 index 0000000..ba4ad16 --- /dev/null +++ b/src/soup_cli/commands/ci.py @@ -0,0 +1,65 @@ +"""soup ci — fine-tuning CI helpers (v0.71.35). + +``soup ci init`` writes a GitHub Actions workflow that gates every PR on +``soup data validate`` -> ``soup expect`` -> ``soup ship --evidence``. +""" + +from __future__ import annotations + +import typer +from rich.console import Console +from rich.markup import escape +from rich.panel import Panel + +from soup_cli.utils.ci_workflow import write_soup_gate_workflow + +console = Console() + +app = typer.Typer(no_args_is_help=True, help="Fine-tuning CI helpers.") + + +@app.command("init") +def init( + data: str = typer.Option( + "./data/train.jsonl", "--data", help="Training data path (repo-relative)" + ), + suite: str = typer.Option( + "expectations.yaml", "--suite", help="Expectations suite YAML (repo-relative)" + ), + evidence: str = typer.Option( + "ship_evidence.json", "--evidence", help="Ship evidence JSON (repo-relative)" + ), + python_version: str = typer.Option( + "3.11", "--python", help="Python version for the CI runner (e.g. 3.11)" + ), + branch: str = typer.Option("main", "--branch", help="Branch the workflow triggers on"), + output: str = typer.Option( + ".github/workflows/soup-gate.yml", "-o", "--output", help="Workflow output path" + ), + force: bool = typer.Option(False, "--force", "-f", help="Overwrite an existing workflow"), +) -> None: + """Write a GitHub Actions fine-tuning gate workflow.""" + try: + written = write_soup_gate_workflow( + data_path=data, + suite_path=suite, + evidence_path=evidence, + python_version=python_version, + branch=branch, + output_path=output, + overwrite=force, + ) + except (ValueError, TypeError, OSError) as exc: + console.print(f"[red]{escape(str(exc))}[/]") + raise typer.Exit(1) from exc + + console.print( + Panel( + f"Wrote [bold]{escape(written)}[/]\n\n" + "The gate runs: [cyan]soup data validate[/] -> [cyan]soup expect[/] -> " + "[cyan]soup ship --evidence[/].\n" + "Edit the paths in the workflow to match your repo.", + title="soup ci init", + border_style="green", + ) + ) diff --git a/src/soup_cli/commands/export.py b/src/soup_cli/commands/export.py index 094168d..96f61be 100644 --- a/src/soup_cli/commands/export.py +++ b/src/soup_cli/commands/export.py @@ -9,6 +9,7 @@ from typing import Optional import typer from rich.console import Console +from rich.markup import escape from rich.panel import Panel console = Console() @@ -404,6 +405,36 @@ def _merge_adapter( console.print("[green]Adapter merged successfully.[/]") +# llama.cpp's own requirements.txt pins `torch~=2.2.1` against the CPU wheel +# index plus an old `transformers`. Installing it into the user's interpreter +# silently DOWNGRADES a CUDA torch to CPU-only and breaks their training setup +# (observed live on Windows during the v0.71.35 GGUF validation: torch +# 2.5.1+cu -> 2.2.2+cpu, transformers 4.57 -> 4.46). Soup's `[train]` extra +# already provides torch / transformers / numpy, so install ONLY the extra +# packages the convert script needs, unpinned, and never touch the rest. +_CONVERT_EXTRA_DEPS = ("gguf", "sentencepiece", "protobuf") + + +def _install_convert_deps() -> None: + """Install the convert script's extra deps without disturbing torch.""" + try: + subprocess.run( + [sys.executable, "-m", "pip", "install", "-q", *_CONVERT_EXTRA_DEPS], + check=True, + capture_output=True, + text=True, + ) + except (subprocess.CalledProcessError, FileNotFoundError) as exc: + # Non-fatal: the user may already have them, or manage deps themselves. + detail = getattr(exc, "stderr", "") or type(exc).__name__ + console.print( + "[yellow]Could not auto-install the GGUF convert dependencies " + f"({', '.join(_CONVERT_EXTRA_DEPS)}).[/]\n" + f"[dim]{escape(str(detail)[:200])}[/]\n" + "Install them manually if the conversion fails." + ) + + def _find_llama_cpp(user_path: Optional[str] = None) -> Path: """Find or clone llama.cpp directory.""" from soup_cli.utils.constants import SOUP_DIR @@ -426,7 +457,12 @@ def _find_llama_cpp(user_path: Optional[str] = None) -> Path: return path # 3. Check ~/.soup/llama.cpp - soup_llama = Path(SOUP_DIR) / LLAMA_CPP_DIR_NAME + # SOUP_DIR is a bare name (".soup"), so it MUST be anchored to the home + # directory the way tracker.py / registry/store.py do. Using it relatively + # made the lookup cwd-dependent: llama.cpp was never found in the canonical + # ~/.soup, and the auto-clone dropped a fresh ~200 MB checkout into whatever + # directory the user happened to run from (v0.71.35 GGUF validation). + soup_llama = Path.home() / SOUP_DIR / LLAMA_CPP_DIR_NAME if soup_llama.exists() and (soup_llama / "convert_hf_to_gguf.py").exists(): return soup_llama @@ -442,15 +478,7 @@ def _find_llama_cpp(user_path: Optional[str] = None) -> Path: capture_output=True, text=True, ) - # Install Python requirements for the convert script - requirements = soup_llama / "requirements.txt" - if requirements.exists(): - subprocess.run( - [sys.executable, "-m", "pip", "install", "-r", str(requirements), "-q"], - check=True, - capture_output=True, - text=True, - ) + _install_convert_deps() console.print("[green]llama.cpp cloned successfully.[/]") return soup_llama except subprocess.CalledProcessError as exc: @@ -505,6 +533,14 @@ def _run_quantize(llama_dir: Path, input_path: Path, output_path: Path, quant_ty raise typer.Exit(1) +# MSVC / Xcode are multi-config generators: they nest binaries under a +# per-configuration subdirectory (build/bin/Release/llama-quantize.exe) rather +# than the flat build/bin/ that single-config generators (Make/Ninja) produce. +# Without these, `soup export --format gguf` cannot find a correctly-built +# llama.cpp on Windows (v0.71.35 GGUF-on-Windows validation, #70/#144). +_CMAKE_CONFIG_DIRS = ("Release", "RelWithDebInfo", "MinSizeRel", "Debug") + + def _find_quantize_binary(llama_dir: Path) -> Optional[Path]: """Find the llama-quantize binary.""" # Check common locations @@ -515,6 +551,11 @@ def _find_quantize_binary(llama_dir: Path) -> Optional[Path]: llama_dir / "llama-quantize.exe", llama_dir / "build" / "llama-quantize", ] + # Multi-config generator layouts (MSVC on Windows, Xcode on macOS). + for config in _CMAKE_CONFIG_DIRS: + candidates.append(llama_dir / "build" / "bin" / config / "llama-quantize") + candidates.append(llama_dir / "build" / "bin" / config / "llama-quantize.exe") + candidates.append(llama_dir / "build" / config / "llama-quantize.exe") for candidate in candidates: if candidate.exists(): return candidate diff --git a/src/soup_cli/commands/push.py b/src/soup_cli/commands/push.py index f82b746..eae18ba 100644 --- a/src/soup_cli/commands/push.py +++ b/src/soup_cli/commands/push.py @@ -66,6 +66,15 @@ def push( "Collections / model-card auto-render path (v0.53.10 #152)." ), ), + card: Optional[str] = typer.Option( + None, + "--card", + help=( + "Registry entry id / prefix / name:tag — render its provenance-rich " + "model card (via `soup card`) and upload it as README.md, overriding " + "the auto-generated card (v0.71.35). HF hub only." + ), + ), ): """Push a trained model to HuggingFace Hub (or alternate hub).""" # v0.53.10 #152 — validate hub at the CLI boundary; only HF is the @@ -163,6 +172,26 @@ def push( ) # --- Upload --- + # v0.71.35 — --card renders a provenance-rich card from a registry entry + # and uploads it as README.md (HF only). Resolve it up front so a bad ref + # fails fast, before any network upload. + card_override: Optional[str] = None + if card: + if hub_canonical != "hf": + console.print("[yellow]--card is HF-only; ignoring for non-HF hub.[/]") + else: + from rich.markup import escape + + from soup_cli.commands.card import CardError, build_card_for_ref + + try: + card_override = build_card_for_ref(card) + except CardError as exc: + # escape: an AmbiguousRefError message embeds registry-derived + # entry names (v0.71.35 security review). + console.print(f"[red]--card: {escape(str(exc))}[/]") + raise typer.Exit(1) from exc + # v0.53.10 #152 — non-HF hubs route through utils.hubs.upload_repo # before we reach the HF-specific Collections / model-card auto-render # path. Each backend lazy-imports its own SDK; missing-dep surfaces @@ -216,10 +245,18 @@ def push( commit_message=commit_message, ) - # Generate and upload model card if not present (v2 — includes - # training config and optional eval scorecard) + # Model card: --card overrides with a registry-driven card; + # otherwise fall back to the path-based v2 card when README is absent + # (v2 — includes training config and optional eval scorecard). readme_path = model_path / "README.md" - if not readme_path.exists(): + if card_override is not None: + api.upload_file( + path_or_fileobj=card_override.encode("utf-8"), + path_in_repo="README.md", + repo_id=repo, + commit_message="Add model card (soup card)", + ) + elif not readme_path.exists(): model_card = generate_model_card_v2( model_path, repo_id=repo, is_adapter=is_adapter, ) @@ -327,13 +364,19 @@ def _load_training_config(model_path: Path) -> dict: return {} -_UNSAFE_MD_CHARS = re.compile(r"[\|\[\]\(\)!\n\r\t<>]") +_UNSAFE_MD_CHARS = re.compile(r"[\|\[\]\(\)!\n\r\t<>`]") def _safe_md_cell(value: str) -> str: """Neutralise Markdown-active chars so ``value`` cannot inject table rows, - links, images, or raw HTML when rendered on HF Hub.""" - return _UNSAFE_MD_CHARS.sub(" ", str(value)).strip() + links, images, raw HTML, or break out of a code span when rendered on HF Hub. + + Also strips C0/ESC control bytes (v0.71.35 security review): the rendered + card is a file an operator may later ``cat``/``less``, where an embedded + ANSI/OSC sequence could manipulate the terminal. + """ + text = "".join(ch for ch in str(value) if ord(ch) >= 0x20 or ch in "\n\r\t") + return _UNSAFE_MD_CHARS.sub(" ", text).strip() def _render_eval_scorecard(eval_scorecard: Optional[dict]) -> str: @@ -353,20 +396,28 @@ def _render_eval_scorecard(eval_scorecard: Optional[dict]) -> str: def _render_training_section(training_cfg: dict) -> str: + """Render the ``## Training`` section. + + Every interpolated value is passed through :func:`_safe_md_cell` + (v0.71.35 security review): ``base`` / ``scheduler`` have no charset + validator in ``SoupConfig``, so a crafted-but-schema-valid config could + otherwise smuggle raw markdown/HTML (or a backtick that breaks out of the + surrounding code span) into a card published to the HF Hub. + """ if not training_cfg: return "" - task = training_cfg.get("task") or "sft" + task = _safe_md_cell(training_cfg.get("task") or "sft") training = training_cfg.get("training", {}) or {} - base = training_cfg.get("base") or "" + base = _safe_md_cell(training_cfg.get("base") or "") lines = ["## Training", "", f"- **Task:** {task}"] if base: lines.append(f"- **Base model:** `{base}`") for key in ("epochs", "lr", "batch_size", "optimizer", "scheduler"): if key in training: - lines.append(f"- **{key}:** {training[key]}") + lines.append(f"- **{key}:** {_safe_md_cell(training[key])}") recipe = training_cfg.get("recipe") if recipe: - lines.append(f"- **Recipe:** `{recipe}`") + lines.append(f"- **Recipe:** `{_safe_md_cell(recipe)}`") lines.append("") return "\n".join(lines) diff --git a/src/soup_cli/templates/eu-ai-act.yaml b/src/soup_cli/templates/eu-ai-act.yaml new file mode 100644 index 0000000..8423973 --- /dev/null +++ b/src/soup_cli/templates/eu-ai-act.yaml @@ -0,0 +1,45 @@ +# Soup compliance template: EU AI Act (Annex XI/XII technical documentation) +# Starting point for fine-tuning a model destined for the EU market. +# +# This file is a normal training config — the compliance controls are Soup +# CLI flags/commands, not config keys. Run them around your training: +# +# 1. Clean the training data (Annex XI data-governance obligations): +# soup data decontaminate ./data/train.jsonl # drop public-benchmark overlap +# soup data pii ./data/train.jsonl # flag personal data +# +# 2. Auto-generate the Annex XI/XII technical documentation + measure energy: +# soup train --config soup.yaml \ +# --annex-xi annex_xi.md \ # or annex_xi.pdf (pip install 'soup-cli[pdf]') +# --track-energy --energy-country DEU \ # codecarbon (pip install 'soup-cli[carbon]') +# --energy-out energy.json +# +# 3. Fold the measured energy into the ML-BOM (transparency): +# soup bom emit --name model --base-sha --config-sha \ +# --energy energy.json --format both +# +# 4. The audit log is ON by default (record-keeping obligations): +# soup audit-log tail +# +# License note: use a license-clean base for EU deployments (Apache-2.0 below). + +base: Qwen/Qwen2.5-7B-Instruct +task: sft + +data: + train: ./data/train.jsonl + format: alpaca + val_split: 0.1 + max_length: 2048 + +training: + epochs: 3 + lr: 2e-5 + batch_size: auto + lora: + r: 64 + alpha: 16 + target_modules: auto + quantization: 4bit + +output: ./output diff --git a/src/soup_cli/templates/hipaa.yaml b/src/soup_cli/templates/hipaa.yaml new file mode 100644 index 0000000..a12f748 --- /dev/null +++ b/src/soup_cli/templates/hipaa.yaml @@ -0,0 +1,46 @@ +# Soup compliance template: HIPAA (Protected Health Information) +# Starting point for fine-tuning on health data under HIPAA safeguards. +# +# This file is a normal training config — the compliance controls are Soup +# CLI flags/commands, not config keys. Run them around your training: +# +# 1. Scrub PHI BEFORE training (keep raw PHI off the training box): +# soup data pii ./data/train.jsonl # flag emails / phones / SSNs / MRNs +# soup data decontaminate ./data/train.jsonl # drop public-benchmark overlap +# +# 2. The HIPAA/SOC2 audit log is ON by default (do NOT pass --no-audit-log): +# soup audit-log tail # review the per-command trail +# +# 3. Train with a reproducibility receipt (who/what/when, SR 11-7 style): +# soup train --config soup.yaml --repro-receipt receipt.json +# +# 4. Prove provenance + integrity of the artifact you ship: +# soup bom emit --name phi-model --base-sha --config-sha --format both +# soup adapters sign ./output --backend ed25519 --generate-key key.pem +# soup attest emit --stage train --subject phi-model --sha --sign ed25519 --key key.pem +# +# 5. Move the model across a data diode without exposing PHI: +# soup airgap-bundle --model ./output --output phi-model.tar --repro-receipt receipt.json +# +# License note: use a license-clean base for regulated deployments (Apache-2.0 below). + +base: Qwen/Qwen2.5-7B-Instruct +task: sft + +data: + train: ./data/train.jsonl + format: alpaca + val_split: 0.1 + max_length: 2048 + +training: + epochs: 3 + lr: 2e-5 + batch_size: auto + lora: + r: 64 + alpha: 16 + target_modules: auto + quantization: 4bit + +output: ./output diff --git a/src/soup_cli/templates/manifest.json b/src/soup_cli/templates/manifest.json index 288904e..e71d1d2 100644 --- a/src/soup_cli/templates/manifest.json +++ b/src/soup_cli/templates/manifest.json @@ -5,6 +5,8 @@ "chat": "chat.yaml", "code": "code.yaml", "embedding": "embedding.yaml", + "eu-ai-act": "eu-ai-act.yaml", + "hipaa": "hipaa.yaml", "ipo": "ipo.yaml", "kto": "kto.yaml", "longcontext": "longcontext.yaml", @@ -15,6 +17,8 @@ "reasoning": "reasoning.yaml", "rlhf": "rlhf.yaml", "simpo": "simpo.yaml", + "soc2": "soc2.yaml", + "sr-11-7": "sr-11-7.yaml", "tool-calling": "tool-calling.yaml", "vision": "vision.yaml" }, diff --git a/src/soup_cli/templates/soc2.yaml b/src/soup_cli/templates/soc2.yaml new file mode 100644 index 0000000..273c25a --- /dev/null +++ b/src/soup_cli/templates/soc2.yaml @@ -0,0 +1,45 @@ +# Soup compliance template: SOC 2 (Trust Services Criteria) +# Starting point for fine-tuning under a SOC 2 change-management + audit regime. +# +# This file is a normal training config — the compliance controls are Soup +# CLI flags/commands, not config keys. Run them around your training: +# +# 1. The audit log is ON by default (evidence for the Security criterion): +# soup audit-log tail # tail the per-command trail +# soup audit-log rotate # force a rotation pass +# +# 2. Pin the exact inputs so a run is reproducible (Change Management): +# soup lock write --base-sha --dataset-sha --env-lock soup-env.lock +# soup train --config soup.yaml --repro-receipt receipt.json +# +# 3. Emit an ML-BOM + signed attestation for every shipped artifact: +# soup bom emit --name model --base-sha --config-sha --format both +# soup adapters sign ./output --backend ed25519 --generate-key key.pem +# soup adapters verify ./output --strict --public-key key.pub.pem +# soup attest emit --stage train --subject model --sha --sign ed25519 --key key.pem +# +# 4. Gate merges/deploys on a SHIP verdict (Availability / Processing Integrity): +# soup ship --evidence ev.json # exit 0 = SHIP, 2 = DON'T SHIP +# +# License note: use a license-clean base for regulated deployments (Apache-2.0 below). + +base: Qwen/Qwen2.5-7B-Instruct +task: sft + +data: + train: ./data/train.jsonl + format: alpaca + val_split: 0.1 + max_length: 2048 + +training: + epochs: 3 + lr: 2e-5 + batch_size: auto + lora: + r: 64 + alpha: 16 + target_modules: auto + quantization: 4bit + +output: ./output diff --git a/src/soup_cli/templates/sr-11-7.yaml b/src/soup_cli/templates/sr-11-7.yaml new file mode 100644 index 0000000..2e10535 --- /dev/null +++ b/src/soup_cli/templates/sr-11-7.yaml @@ -0,0 +1,43 @@ +# Soup compliance template: SR 11-7 (Model Risk Management, banking) +# Starting point for fine-tuning under a model-risk-management framework. +# +# This file is a normal training config — the compliance controls are Soup +# CLI flags/commands, not config keys. Run them around your training: +# +# 1. Capture a reproducibility receipt (seeds / kernels / GPU / OS): +# soup train --config soup.yaml --repro-receipt receipt.json +# +# 2. Independent effective-challenge: diagnose failure modes + SHIP gate: +# soup diagnose --output diag.json --badge diag.svg +# soup ship --base --adapter ./output --task-eval task.jsonl +# # exit 0 = SHIP, 2 = DON'T SHIP (leg 1 task-win AND leg 2 no regression) +# +# 3. Emit an ML-BOM + attestation for the model inventory / provenance: +# soup bom emit --name model --base-sha --config-sha --format both +# soup attest emit --stage train --subject model --sha --sign ed25519 --key key.pem +# +# 4. The audit log is ON by default (ongoing monitoring / record-keeping): +# soup audit-log tail +# +# License note: use a license-clean base for regulated deployments (Apache-2.0 below). + +base: Qwen/Qwen2.5-7B-Instruct +task: sft + +data: + train: ./data/train.jsonl + format: alpaca + val_split: 0.1 + max_length: 2048 + +training: + epochs: 3 + lr: 2e-5 + batch_size: auto + lora: + r: 64 + alpha: 16 + target_modules: auto + quantization: 4bit + +output: ./output diff --git a/src/soup_cli/utils/ci_workflow.py b/src/soup_cli/utils/ci_workflow.py new file mode 100644 index 0000000..8b0b18f --- /dev/null +++ b/src/soup_cli/utils/ci_workflow.py @@ -0,0 +1,179 @@ +"""Render a GitHub Actions "fine-tuning gate" workflow (v0.71.35). + +``soup ci init`` writes ``.github/workflows/soup-gate.yml`` — a CI job that +gates every PR on three Soup checks: + + soup data validate # dataset format compliance + soup expect # expectations suite (PII / length / refusal) + soup ship --evidence # SHIP / DON'T-SHIP verdict (exit 2 = block) + +The workflow body is a fixed YAML skeleton; only a handful of tokens are +interpolated. Following ``eval_gate_hook.py``: any value placed into a ``run:`` +shell step is passed through ``shlex.quote`` (+ control-char rejection) so a +crafted path cannot chain commands, and every path is validated to stay under +the repo root. There is NO top-level torch import. +""" + +from __future__ import annotations + +import os +import re +import shlex +import tempfile + +from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink, is_under_cwd + +# GitHub allows quite permissive ref names; we constrain to a safe subset that +# also cannot break the YAML scalar or a shell step. +_BRANCH_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._/-]{0,254}$") +_PY_RE = re.compile(r"^\d+\.\d+$") +_MAX_PATH_LEN = 4096 +_MAX_FILE_BYTES = 64 * 1024 + +_WORKFLOW_TEMPLATE = """\ +# Soup fine-tuning gate — generated by `soup ci init` (v0.71.35). +# Gates every PR on: data validate -> expectations -> SHIP verdict. +# Edit the paths below to match your repo. The `soup ship` step needs a +# committed evidence JSON (see `soup ship --evidence`); remove it if unused. +name: Soup Fine-tuning Gate + +on: + push: + branches: [{branch}] + pull_request: + branches: [{branch}] + +jobs: + soup-gate: + runs-on: ubuntu-latest + env: + PYTHONUTF8: "1" + PYTHONIOENCODING: "utf-8" + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "{python_version}" + - name: Install Soup CLI + # Core install is enough: `data validate`, `expect` and + # `ship --evidence` are all no-torch paths. + run: pip install soup-cli + - name: Validate training data + run: soup data validate {data} + - name: Run expectations suite + run: soup expect {data} {suite} + - name: SHIP / DON'T-SHIP gate + run: soup ship --evidence {evidence} +""" + + +def _validate_path(value: str, field: str) -> str: + if isinstance(value, bool) or not isinstance(value, str): + raise TypeError(f"{field} must be str") + if not value: + raise ValueError(f"{field} must be non-empty") + if "\x00" in value: + raise ValueError(f"{field} must not contain NUL") + if "\n" in value or "\r" in value: + raise ValueError(f"{field} must be a single line") + if len(value) > _MAX_PATH_LEN: + raise ValueError(f"{field} exceeds {_MAX_PATH_LEN} characters") + if not is_under_cwd(value): + raise ValueError(f"{field} must stay under the repository root") + return value + + +def _safe_shell_quote(value: str) -> str: + """``shlex.quote`` with a control-char rejection prelude (defence in depth).""" + if any(ord(ch) < 0x20 for ch in value): + raise ValueError("value contains control characters") + return shlex.quote(value) + + +def render_soup_gate_workflow( + *, + data_path: str, + suite_path: str, + evidence_path: str, + python_version: str = "3.11", + branch: str = "main", +) -> str: + """Render the workflow YAML body. Deterministic, no I/O. + + Every path is validated to stay under the repo root and shell-quoted before + it reaches a ``run:`` step; ``python_version`` / ``branch`` are regex-gated. + """ + data = _validate_path(data_path, "data_path") + suite = _validate_path(suite_path, "suite_path") + evidence = _validate_path(evidence_path, "evidence_path") + + if not isinstance(python_version, str) or not _PY_RE.match(python_version): + raise ValueError("python_version must look like '3.11'") + if not isinstance(branch, str) or not _BRANCH_RE.match(branch): + raise ValueError("branch has an unsupported name") + + return _WORKFLOW_TEMPLATE.format( + branch=branch, + python_version=python_version, + data=_safe_shell_quote(data), + suite=_safe_shell_quote(suite), + evidence=_safe_shell_quote(evidence), + ) + + +def write_soup_gate_workflow( + *, + data_path: str, + suite_path: str, + evidence_path: str, + python_version: str = "3.11", + branch: str = "main", + output_path: str = ".github/workflows/soup-gate.yml", + overwrite: bool = False, +) -> str: + """Render + atomically write the workflow. Returns the path written. + + Refuses to clobber an existing file unless ``overwrite=True``; rejects a + symlink destination (TOCTOU) and any path outside the repo root. + """ + body = render_soup_gate_workflow( + data_path=data_path, + suite_path=suite_path, + evidence_path=evidence_path, + python_version=python_version, + branch=branch, + ) + if isinstance(output_path, bool) or not isinstance(output_path, str): + raise TypeError("output_path must be str") + if not output_path: + raise ValueError("output_path must be non-empty") + if "\x00" in output_path: + raise ValueError("output_path must not contain NUL") + if not isinstance(overwrite, bool): + raise TypeError("overwrite must be bool") + # Single-source-of-truth containment + symlink/reparse-point rejection. + # A hand-rolled S_ISLNK check misses Windows junctions, so always delegate + # to the shared helper (v0.71.35 security review). + enforce_under_cwd_and_no_symlink(output_path, "output_path") + if os.path.lexists(output_path) and not overwrite: + raise ValueError("workflow already exists; pass --force to overwrite") + if len(body.encode("utf-8")) > _MAX_FILE_BYTES: + raise ValueError("rendered workflow exceeds 64 KiB cap") + + parent = os.path.dirname(os.path.abspath(output_path)) or "." + os.makedirs(parent, exist_ok=True) + fd, tmp = tempfile.mkstemp(prefix=".soup-gate.", dir=parent) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(body) + os.replace(tmp, output_path) + except Exception: + try: + os.unlink(tmp) + except OSError: + pass + raise + return output_path + + +__all__ = ["render_soup_gate_workflow", "write_soup_gate_workflow"] diff --git a/src/soup_cli/utils/ollama.py b/src/soup_cli/utils/ollama.py index 8c50e1c..c6d8898 100644 --- a/src/soup_cli/utils/ollama.py +++ b/src/soup_cli/utils/ollama.py @@ -1,5 +1,6 @@ """Ollama integration utilities — detect, deploy, list, remove models.""" +import os import re import subprocess from pathlib import Path @@ -167,7 +168,12 @@ def create_modelfile( Returns: Modelfile content as string. """ - lines = [f"FROM {gguf_path}"] + # `ollama create` resolves a relative FROM against the Modelfile's own + # directory, and we write the Modelfile to a temp dir — so a relative GGUF + # path made Ollama treat it as a remote model name and fail with + # "pull model manifest: file does not exist". Always emit an absolute path + # (v0.71.35 GGUF-on-Windows validation). + lines = [f"FROM {os.path.abspath(str(gguf_path))}"] # Template if template: diff --git a/tests/test_v07135.py b/tests/test_v07135.py new file mode 100644 index 0000000..492c764 --- /dev/null +++ b/tests/test_v07135.py @@ -0,0 +1,993 @@ +"""v0.71.35 — Compliance pack. + +Covers: + * Compliance init templates (hipaa / soc2 / eu-ai-act / sr-11-7) + * ``soup card`` — HF model-card autogen from a registry entry + * ``soup ci init`` — render .github/workflows/soup-gate.yml +""" + +from __future__ import annotations + +import json +import os + +import pytest +import yaml +from typer.testing import CliRunner + +from soup_cli.config.loader import load_config_from_string +from soup_cli.templates import list_templates, load_template + +runner = CliRunner() + +COMPLIANCE_TEMPLATES = ["hipaa", "soc2", "eu-ai-act", "sr-11-7"] + + +# --------------------------------------------------------------------------- # +# Compliance init templates +# --------------------------------------------------------------------------- # +class TestComplianceTemplates: + @pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES) + def test_template_is_listed(self, name): + assert name in list_templates() + + @pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES) + def test_template_loads(self, name): + body = load_template(name) + assert body is not None and body.strip() + + @pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES) + def test_template_parses_as_config(self, name): + """The YAML body must be a valid SoupConfig (train-time keys only).""" + body = load_template(name) + cfg = load_config_from_string(body) + assert cfg.base + assert cfg.task + + @pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES) + def test_template_has_compliance_guidance(self, name): + """Compliance behaviours are CLI flags/commands, not schema keys, so the + template must carry header-comment guidance pointing users at them.""" + body = load_template(name) + lowered = body.lower() + # every compliance template should mention the audit log + at least one + # provenance command + assert "audit" in lowered + assert any(tok in lowered for tok in ("bom", "attest", "repro-receipt", "sign")) + + def test_manifest_lists_compliance_templates(self): + import soup_cli.templates as tpl + + manifest = tpl._load_manifest() + for name in COMPLIANCE_TEMPLATES: + assert name in manifest["templates"] + + def test_init_writes_compliance_template(self, tmp_path, monkeypatch): + from soup_cli.cli import app + + monkeypatch.chdir(tmp_path) + out = tmp_path / "soup.yaml" + result = runner.invoke(app, ["init", "--template", "hipaa", "-o", str(out), "--force"]) + assert result.exit_code == 0, (result.output, repr(result.exception)) + assert out.exists() + cfg = load_config_from_string(out.read_text(encoding="utf-8")) + assert cfg.base + + +# --------------------------------------------------------------------------- # +# soup card — model-card autogen from a registry entry +# --------------------------------------------------------------------------- # +def _make_entry(db_path, *, notes=None, with_eval=False, with_artifact=None, parent=False): + """Seed a registry entry (and optional artifact / lineage) in a temp DB.""" + from soup_cli.registry.store import RegistryStore + + with RegistryStore(db_path=db_path) as store: + parent_id = None + if parent: + parent_id = store.push( + name="base-run", tag="v1", base_model="Qwen/Qwen2.5-7B-Instruct", + task="sft", run_id=None, config={"base": "Qwen/Qwen2.5-7B-Instruct"}, + notes=None, + ) + eid = store.push( + name="phi-model", tag="v1", base_model="Qwen/Qwen2.5-7B-Instruct", + task="sft", run_id=None, + config={"base": "Qwen/Qwen2.5-7B-Instruct", "task": "sft", + "training": {"epochs": 3, "lr": 2e-5}}, + notes=notes, + ) + if parent_id: + store.add_lineage(child_id=eid, parent_id=parent_id, relation="forked_from") + if with_artifact is not None: + store.add_artifact(entry_id=eid, kind=with_artifact[0], + path=with_artifact[1], enforce_cwd=True) + return eid + + +class TestBuildModelCard: + def test_pure_card_has_core_sections(self): + from soup_cli.commands.card import build_model_card + + entry = { + "id": "reg_x", "name": "my-model", "base_model": "Qwen/Qwen2.5-7B-Instruct", + "task": "sft", "created_at": "2026-07-15T00:00:00", "notes": "clean run", + "config_hash": "a" * 64, "data_hash": "b" * 64, "run_id": None, + "tags": ["prod"], + "config_json": json.dumps({"base": "Qwen/Qwen2.5-7B-Instruct", "task": "sft", + "training": {"epochs": 3, "lr": 2e-5}}), + } + md = build_model_card(entry, [], [], []) + assert md.startswith("---") # YAML frontmatter + assert "# my-model" in md + assert "Qwen/Qwen2.5-7B-Instruct" in md + assert "## Training" in md + + def test_notes_html_escaped(self): + from soup_cli.commands.card import build_model_card + + entry = { + "id": "reg_x", "name": "m", "base_model": "b", "task": "sft", + "created_at": "t", "notes": "", + "config_hash": "", "data_hash": "", "run_id": None, "tags": [], + "config_json": "{}", + } + md = build_model_card(entry, [], [], []) + assert " `' + entry = { + "id": "r", "name": "m", "base_model": "safe/base", "task": "sft", + "created_at": "t", "notes": None, "config_hash": "", "data_hash": "", + "run_id": None, "tags": [], + "config_json": json.dumps({ + "base": hostile, "task": "sft", + "training": {"epochs": 1, "scheduler": ""}, + }), + } + md = build_model_card(entry, [], [], []) + # The security property is that no HTML TAG survives — angle brackets and + # backticks are neutralised, so the payload is inert prose. (The bare + # word "onerror" surviving as plain text is harmless.) + assert "