RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

feat(compliance): init templates + soup card + soup ci init + GGUF-on-Windows (v0.71.35) 

Ship a regulated fine-tune with the paperwork it needs, plus GGUF export that
actually works on Windows.

Added:
* `soup init --template hipaa|soc2|eu-ai-act|sr-11-7` (templates 17 -> 21).
  Design constraint verified in-repo, not assumed: Soup's compliance controls
  are CLI flags/commands, NOT schema keys (audit_log/bom/attest/repro_receipt/
  annex_xi/track_energy/pii/decontaminate have zero matches in config/schema.py),
  so a template cannot "pre-wire audit-log on" as YAML. Each is a valid
  SoupConfig + header comments naming that regime's exact commands.
* `soup card <registry-id> -o MODELCARD.md` — registry entry -> publishable HF
  card (training config, eval scorecard, config/data hashes, lineage, artifacts).
  Pure build_model_card() over dicts, reused by `soup push --card`.
* `soup ci init` — writes .github/workflows/soup-gate.yml chaining
  data validate -> expect -> ship --evidence (exit 2 blocks the merge).
* docs/compliance.md quickstart.

Fixed (GGUF-on-Windows, validated end-to-end for the first time; closes the
CPU-validatable half of #70/#144). Four independently-fatal bugs:
* export cloned llama.cpp into the CURRENT directory: SOUP_DIR is the bare name
  ".soup" but was used relatively instead of anchored to home like tracker.py /
  registry/store.py, so ~/.soup/llama.cpp was never found.
* the first GGUF export DOWNGRADED the user's torch and broke CUDA: the
  auto-clone pip-installed llama.cpp's requirements.txt (pins torch~=2.2.1 from
  the CPU index) into the user's interpreter. Observed live: torch 2.5.1+cu ->
  2.2.2+cpu, transformers 4.57 -> 4.46. Now installs only gguf/sentencepiece/
  protobuf, unpinned, non-fatally.
* a correctly-built llama.cpp was not found on Windows: MSVC (like Xcode) is a
  multi-config generator emitting build/bin/Release/llama-quantize.exe.
* `soup deploy ollama` failed on a relative GGUF path ("pull model manifest:
  file does not exist") — ollama resolves FROM against the Modelfile's dir and
  Soup writes it to a temp dir. Modelfile now emits an absolute path.

Also fixed a pre-existing model-card injection hole (affects `soup push`'s own
auto-card): _render_training_section interpolated base/task/scheduler/recipe
unescaped, and SoupConfig.base/scheduler have no charset validator, so a
crafted-but-valid config could smuggle raw HTML or a code-span-breaking backtick
into a card published to the Hub.

Step-6 live smoke (real train -> registry push -> card) caught a bug 90 green
tests missed: is_adapter came only from registry artifacts, so a real LoRA run
with no artifacts rendered "Full model" + library_name: transformers — a false
claim in a provenance document.

5 sequential reviews, every finding fixed (code HIGH: the generated workflow ran
`pip install -e ".[dev]"`, which only works in the Soup source tree, breaking the
gate's first step for every downstream user; security HIGH/MEDIUM/3 LOW; tdd HIGH:
push --card was only --help-tested). All 5 new fixes mutation-verified as real
pins. Tests 15906 -> 16001 (+95); full suite 15872 passed / 126 skipped.
This commit is contained in:
Alpamys 2026-07-15 20:09:10 +05:00
parent ff4c9963cb
commit dc73e947e9
21 changed files with 2084 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

59
CHANGELOG.md
View File

@ -12,6 +12,65 @@ reproducing 70+ versions of notes.
## [Unreleased]
## [0.71.35] - 2026-07-15
### Added
- **Compliance templates — `soup init --template hipaa|soc2|eu-ai-act|sr-11-7`.**
Four regulation-shaped starting configs. Soup's compliance controls are CLI
flags/commands rather than config keys, so each template is a valid training
config plus header comments naming the exact commands for that regime
(PHI scrubbing + air-gap for HIPAA, BOM/attest/sign for SOC 2, Annex XI +
energy tracking for the EU AI Act, repro-receipt + diagnose/ship for SR 11-7).
Templates default to a license-clean Apache-2.0 base.
- **`soup card <registry-id> -o MODELCARD.md` — model-card autogen.** Turns a
Local Model Registry entry into a publishable, provenance-carrying HF model
card: base model, training config, eval scorecard, config/data hashes,
lineage (ancestors) and a table of every registered artifact. Adapter vs
full-model is inferred from registered artifacts, falling back to the training
config (LoRA rank, with Spectrum/LISA full-FT correctly treated as dense), so
the card sets the right `library_name` and never misreports the model type.
- **`soup push --card <registry-id>`** — render that registry-driven card and
upload it as `README.md`, overriding the auto-generated one. A bad ref fails
fast before any network call; HF hub only.
- **`soup ci init` — fine-tuning CI.** Writes `.github/workflows/soup-gate.yml`,
a PR gate chaining `soup data validate` → `soup expect` →
`soup ship --evidence` (exit 2 blocks the merge). Every interpolated path is
validated to stay under the repo root and shell-quoted; the branch and Python
version are regex-gated; the write is atomic, symlink-rejecting, and refuses
to clobber an existing workflow without `--force`.
- **Compliance quickstart** — a new [docs/compliance.md](docs/compliance.md)
walkthrough: template → PII scrub → train with receipt/Annex XI/energy →
registry → BOM + attestation → scan/sign/verify → air-gap → model card → CI gate.
### Fixed
- **GGUF export now actually works on Windows** (validated end-to-end against a
locally-built llama.cpp: SmolLM2-135M → q4_0 / q4_k_m / q8_0 / f16 → `soup deploy
ollama` → live inference). Four real bugs, each of which independently broke the
path:
- **`soup export --format gguf` cloned llama.cpp into your current directory.**
`SOUP_DIR` is the bare name `.soup`, but the lookup used it relatively rather
than anchoring to `~` like the rest of the codebase — so the canonical
`~/.soup/llama.cpp` was never found and a fresh ~200 MB checkout was dropped
into whatever directory you ran from.
- **The first GGUF export downgraded your PyTorch and broke CUDA.** The auto-clone
ran `pip install -r <llama.cpp>/requirements.txt` into your interpreter, and
llama.cpp pins `torch~=2.2.1` against the CPU wheel index (observed:
torch 2.5.1+cu → 2.2.2+cpu, transformers 4.57 → 4.46). Soup now installs only
the convert script's extra dependencies, unpinned, and never touches torch.
- **A correctly-built llama.cpp was not found on Windows.** MSVC (like Xcode) is a
multi-config generator and emits `build/bin/Release/llama-quantize.exe`; only the
flat single-config layout was searched.
- **`soup deploy ollama` failed on a relative GGUF path** with "pull model manifest:
file does not exist" — Ollama resolves `FROM` against the Modelfile's directory,
and Soup writes the Modelfile to a temp dir. The Modelfile now emits an absolute path.
- **Model-card injection hardening (affects the pre-existing `soup push` card too).**
The `## Training` section interpolated `base` / `task` / `scheduler` / `recipe`
unescaped. Since `SoupConfig.base` and `scheduler` have no charset validator, a
crafted-but-valid config could smuggle raw HTML — or a backtick breaking out of
the surrounding code span — into a card published to the Hub. All values now go
through the markdown escaper, which additionally neutralises backticks and
strips C0/ESC control bytes.
## [0.71.34] - 2026-07-15
### Added

4
CONTRIBUTING.md
View File

@ -117,10 +117,10 @@ src/soup_cli/
data/traces/ - Trace-to-Preference harvester (v0.26.0)
data/collators.py - CrossDocCollator for sample packing (v0.33.0)
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
templates/ - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
ui/ - Web UI (FastAPI + HTML/JS SPA)
tests/ - Test suite (312 files, 15906 tests)
tests/ - Test suite (313 files, 16001 tests)
examples/ - Real-world config examples and datasets
```

32
README.md
View File

@ -49,18 +49,28 @@ infrastructure instead of improving models. Soup fixes that.
## What's New
**v0.71.34 — adapter algebra + LISA.** Compose fine-tunes like vectors, and train big-model quality on a small-model memory budget.
**v0.71.35 — the compliance pack.** Ship a regulated fine-tune with the paperwork it needs: start from a regulation-shaped config, publish a provenance-carrying model card, and gate every PR in CI.
- **`soup adapters arithmetic "coder + 0.5*math - toxic"`.** Task-vector algebra over LoRA
adapters (arXiv:2212.04089) — **add, scale, and NEGATE** trained behaviours into one merged
adapter. The math is done right: a LoRA's effective delta `ΔW = B·A` scales *linearly* with
each coefficient (subtracting an adapter actually removes its behaviour), not quadratically.
Same-base + backdoor-scan gated; mixed ranks refused with a clear message.
- **LISA — layerwise importance sampling (`training.lisa_enabled`).** Full-fine-tuning quality
at LoRA-like memory (arXiv:2403.17919): every N steps LISA re-activates a small random set of
decoder layers (embeddings + head always on) and freezes the rest. Live on a 4 GB GPU.
- **Verified on real models (RTX 3050).** LISA trains SmolLM2-135M end-to-end; `2·a` produces an
adapter whose ΔW is *exactly* 2× the original's — the merged adapter loads and serves.
- **`soup init --template hipaa|soc2|eu-ai-act|sr-11-7`.** Four regulation-shaped starting
configs, each documenting the exact commands for that regime — PHI scrubbing + air-gap,
BOM/attest/sign, EU Annex XI + energy tracking, or repro-receipt + diagnose/ship.
- **`soup card <registry-id>` → `MODELCARD.md`.** Turn a registry entry into a publishable HF
model card: training config, eval scorecard, config/data hashes, lineage, and every
registered artifact. `soup push --card` uploads it as the README, so every public push
carries its provenance.
- **`soup ci init` — fine-tuning CI.** Writes a GitHub Actions gate that runs
`data validate` → `expect` → `ship --evidence` on each PR; a regression exits 2 and
blocks the merge.
- **Compliance quickstart** ([docs/compliance.md](docs/compliance.md)) — the whole path from
template to air-gapped, signed, attested, carded model.
- **GGUF export fixed on Windows.** Validated end-to-end for the first time (SmolLM2 →
q4_0/q4_k_m/q8_0/f16 → Ollama → live inference), which surfaced four real bugs: the export
cloned llama.cpp into your *current directory*, its first run **downgraded your PyTorch to
CPU-only** (breaking CUDA), a correctly-built llama.cpp wasn't found under MSVC, and
`deploy ollama` rejected relative GGUF paths. All fixed.
- **Hardening:** the model-card generator (including the existing `soup push` card) no longer
interpolates unescaped config values — a crafted `base`/`scheduler` can't smuggle HTML into
a card published on the Hub.
```bash
soup adapters arithmetic "coder - toxic" \

1
docs/README.md
View File

@ -14,6 +14,7 @@ feature reference — every `soup` capability, grouped by area.
| [Evaluation & probes](evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, `soup ship` verdict, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
| [Serving & export](serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
| [Adapters, registry & governance](adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls |
| [Compliance & governance quickstart](compliance.md) | HIPAA/SOC2/EU-AI-Act/SR-11-7 `init` templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (`soup card`), CI gate (`soup ci init`) |
| [Backends, platform & ops](backends-and-ops.md) | MLX/Unsloth backends, Modal cloud GPU training, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
| [Command reference](commands.md) | The full `soup` command list |
| [Supported models & extras](models.md) | Recommended model families, the VRAM size guide, the pip extras matrix |

4
docs/commands.md
View File

@ -8,6 +8,7 @@
```
soup init [--template chat|code|...|audio] Create config
soup init --template hipaa|soc2|eu-ai-act|sr-11-7 Compliance-shaped starting config + the commands for that regime (v0.71.35)
soup autopilot --model <id> --data d.jsonl --goal <g> Zero-configsoup train --config soup.yaml Start training
soup train --config soup.yaml --tensorboard Train with TensorBoard logging
soup train --config soup.yaml --fsdp full_shard Train with FSDP2
@ -159,6 +160,9 @@ soup ship --evidence ev.json [--output v.json] Decide offline from pre-computed
soup ship ... --task-mode judge_score --judge-model ollama://llama3.1 Leg-1 via LLM-as-a-judge
soup ship ... --task-mode pairwise --judge-model ollama://llama3.1 Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31)
soup ship ... --general-suite mmlu,gsm8k --baseline base.json lm-eval leg-2 + recorded base scores
soup card <registry-id> -o MODELCARD.md HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35)
soup push --model ./out --repo you/m --card <registry-id> Upload that registry-driven card as the README (HF only) (v0.71.35)
soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--branch main --python 3.11] [--force] Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35)
soup mcp serve MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28)
soup mcp serve --allow-mutating Also expose plan-only train_start / export tools (never execute) (v0.71.28)
soup shrink --model <id|path> --drop-ratio 0.25 --calib c.jsonl -o shrunk Depth-prune least-important layer block + SHIP/DON'T-SHIP ppl verdict (exit 0/2/1) (v0.71.29)

136
docs/compliance.md Normal file
View File

@ -0,0 +1,136 @@
# Compliance & governance quickstart
[← Back to the docs index](README.md)
Soup ships an end-to-end compliance workflow: start from a regulation-shaped
config, train with provenance capture, sign + attest the artifact, keep an audit
trail, and move it across an air gap — then publish a documented model card and
gate future changes in CI.
- **Pre-wired configs:** `soup init --template hipaa|soc2|eu-ai-act|sr-11-7`
- **Provenance:** `soup train --repro-receipt` · `soup bom emit` · `soup attest emit`
- **Integrity:** `soup adapters sign` / `verify` · `soup adapters scan`
- **Audit trail:** the HIPAA/SOC2 audit log is on by default (`soup audit-log tail`)
- **Air gap:** `soup airgap-bundle`
- **Publish:** `soup card <registry-id>` → `MODELCARD.md` (or `soup push --card`)
- **CI gate:** `soup ci init` → a PR workflow that runs validate → expect → ship
---
## 1. Start from a compliance template
Each template is a normal training config plus header comments listing the exact
compliance commands to run around it. Pick the regime you operate under:
```bash
soup init --template hipaa # Protected Health Information
soup init --template soc2 # SOC 2 Trust Services Criteria
soup init --template eu-ai-act # EU AI Act Annex XI/XII
soup init --template sr-11-7 # SR 11-7 Model Risk Management
```
The compliance controls are Soup **CLI flags/commands, not config keys** — the
template header documents which ones apply. The steps below are the common path.
## 2. Clean the data before training
```bash
soup data pii ./data/train.jsonl # flag emails / phones / SSNs / MRNs
soup data decontaminate ./data/train.jsonl # drop public-benchmark overlap
```
## 3. Train with a reproducibility receipt (+ Annex XI / energy for the EU)
```bash
# SR 11-7 / SOC 2 / HIPAA: capture seeds, kernels, GPU, OS
soup train --config soup.yaml --repro-receipt receipt.json
# EU AI Act: auto-generate the Annex XI/XII documentation + measure energy
soup train --config soup.yaml \
--annex-xi annex_xi.md \
--track-energy --energy-country DEU --energy-out energy.json
```
The audit log records every command automatically:
```bash
soup audit-log tail # review the trail
soup audit-log rotate # force a rotation pass
```
## 4. Register the run, then emit BOM + attestation
```bash
soup registry push --run-id <run-id> --name my-model --tag v1
soup bom emit --name my-model --base-sha <hex> --config-sha <hex> \
--energy energy.json --format both # CycloneDX + SPDX
soup attest emit --stage train --subject my-model --sha <hex> \
--sign ed25519 --key key.pem # in-toto + SLSA-3
```
## 5. Sign, scan, and verify the artifact
```bash
soup adapters scan ./output # weight-space backdoor scan
soup adapters sign ./output --backend ed25519 --generate-key key.pem
soup adapters verify ./output --strict --public-key key.pub.pem
```
## 6. Air-gap transfer (optional)
```bash
soup airgap-bundle --model ./output --output my-model.tar --repro-receipt receipt.json
```
## 7. Generate a documented model card
Turn the registry entry into a provenance-rich `MODELCARD.md` — base model,
training config, eval scorecard, config/data hashes, lineage, and every
registered artifact:
```bash
soup card my-model:v1 -o MODELCARD.md
# or, when uploading to the Hub, override the auto-generated card:
soup push --model ./output --repo you/my-model --card my-model:v1
```
## 8. Gate future changes in CI
Write a GitHub Actions workflow that blocks a PR unless the data validates, the
expectations suite passes, and the SHIP verdict is green:
```bash
soup ci init --data data/train.jsonl --suite expectations.yaml --evidence ship_evidence.json
# writes .github/workflows/soup-gate.yml
```
The generated job runs, in order:
```
soup data validate <data> # dataset format compliance
soup expect <data> <suite> # PII / token-length / refusal / judge expectations
soup ship --evidence <ev.json> # SHIP / DON'T-SHIP (exit 2 blocks the merge)
```
A minimal `expectations.yaml` for the second step:
```yaml
expectations:
- name: expect_no_pii
- name: expect_token_length_between
min_tokens: 1
max_tokens: 512
```
Supported names: `expect_no_pii`, `expect_token_length_between`,
`expect_no_refusal_pattern`, `expect_chosen_preferred_over_rejected_by_judge`.
Every path is shell-quoted and validated to stay under the repo root, so the
rendered workflow is injection-safe. Edit the paths to match your repo.
---
See also: [Adapters, registry & governance](adapters-and-governance.md) for the
full supply-chain command set, and [Evaluation & probes](evaluation.md) for the
`soup ship` verdict engine.

2
pyproject.toml
View File

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "soup-cli"
version = "0.71.34"
version = "0.71.35"
description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
readme = "README.md"
license = "Apache-2.0"

2
src/soup_cli/__init__.py
View File

@ -1,3 +1,3 @@
"""Soup CLI — Fine-tune and post-train LLMs in one command."""
__version__ = "0.71.34"
__version__ = "0.71.35"

4
src/soup_cli/cli.py
View File

@ -21,7 +21,9 @@ from soup_cli.commands import ( # noqa: E402
autopilot,
bench,
can,
card,
chat,
ci,
cost,
data,
deploy,
@ -98,6 +100,8 @@ app.command()(cost.cost)
app.command()(push.push)
app.command(name="export")(export.export)
app.command()(merge.merge)
app.command(name="card")(card.card)
app.add_typer(ci.app, name="ci", help="Fine-tuning CI: init a PR gate workflow.")
app.add_typer(
data.app, name="data",
help="Dataset tools: inspect, convert, merge, dedup, validate, stats.",

316
src/soup_cli/commands/card.py Normal file
View File

@ -0,0 +1,316 @@
"""soup card — generate a HuggingFace model card from a registry entry (v0.71.35).
Turns a Local Model Registry entry into a ready-to-publish ``MODELCARD.md``:
base model, training config, eval scorecard, provenance (config/data hashes),
lineage (ancestors), and a table of every registered artifact (adapter / merged
model / GGUF / diagnose report / eval results). Every public push becomes a
documented, provenance-carrying release.
Design notes:
* ``build_model_card`` is a pure function over plain dicts (no store / no
torch) so it is trivially unit-testable and reusable by ``soup push --card``.
* User-controlled text is escaped for its output target: registry ``notes`` is
free-form -> ``html.escape`` (blocks script/``javascript:`` injection on the
HF README viewer); markdown table cells -> ``_safe_md_cell`` (reused from
``push.py``); YAML frontmatter scalars -> ``_yaml_dq`` (control-char strip +
double-quote escaping so a hostile ``base_model`` cannot inject sibling keys).
* Output is written through ``atomic_write_text`` (cwd-contained,
symlink-rejected), consistent with ``write_bom`` / ``write_attestation``.
"""
from __future__ import annotations
import html
import json
from typing import Any, Mapping, Sequence
import typer
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
from soup_cli.commands.push import (
_render_eval_scorecard,
_render_training_section,
_safe_md_cell,
)
from soup_cli.registry.store import AmbiguousRefError, RegistryStore
from soup_cli.utils.paths import atomic_write_text
console = Console()
# Bounds on registry-sourced content (v0.71.35 security review). The registry
# places no length cap on `notes` / `base_model` and no row cap on artifacts, so
# a pathological entry could otherwise render an unbounded card.
_MAX_NOTES_CHARS = 8_000
_MAX_ROWS = 200
class CardError(ValueError):
"""Raised when a registry entry cannot be resolved for a card."""
def _truncate(text: str, limit: int) -> str:
return text if len(text) <= limit else text[:limit] + "\n\n[truncated]"
# Export kinds that prove the artifact is a standalone (non-adapter) model.
_DENSE_KINDS = frozenset({"merged", "gguf", "onnx", "awq", "gptq", "tensorrt", "edited_model"})
def _is_adapter(artifacts: Sequence[Mapping[str, Any]], config: Mapping[str, Any]) -> bool:
"""Decide whether the entry describes a LoRA adapter or a standalone model.
A registered artifact is definitive; otherwise fall back to the training
config. This matters because the card is a provenance document — claiming
"Full model" for a LoRA (and emitting ``library_name: transformers`` instead
of ``peft``) is a false statement + a broken HF Hub card. Found by the
v0.71.35 Step-6 smoke: a real LoRA run with no artifacts attached rendered
as "Full model".
"""
kinds = {a.get("kind") for a in artifacts or []}
if "adapter" in kinds:
return True
if kinds & _DENSE_KINDS:
return False
training = config.get("training") or {}
if not isinstance(training, dict):
return False
# Spectrum / LISA are full-FT: LoRA is off even though the dumped config
# still carries a default `lora` block.
if training.get("unfrozen_parameters") or training.get("lisa_enabled"):
return False
lora = training.get("lora") or {}
if not isinstance(lora, dict):
return False
try:
return int(lora.get("r") or 0) > 0
except (TypeError, ValueError):
return False
def _yaml_dq(value: object) -> str:
"""Serialise ``value`` as a YAML double-quoted scalar, stripping control
chars and escaping ``\\`` / ``"`` so it cannot break out of the frontmatter.
"""
text = "".join(ch for ch in str(value) if ord(ch) >= 0x20)
text = text.replace("\\", "\\\\").replace('"', '\\"')
return f'"{text}"'
def build_model_card(
entry: Mapping[str, Any],
artifacts: Sequence[Mapping[str, Any]],
eval_results: Sequence[Mapping[str, Any]],
ancestors: Sequence[Mapping[str, Any]],
) -> str:
"""Render a full HF model card (markdown) from a registry entry + joins.
``entry`` is a hydrated registry row; ``artifacts`` / ``eval_results`` /
``ancestors`` are the corresponding ``RegistryStore`` joins (may be empty).
"""
name = str(entry.get("name") or "model")
base = str(entry.get("base_model") or "")
task = str(entry.get("task") or "sft")
notes = entry.get("notes") or ""
tags = entry.get("tags") or []
created = str(entry.get("created_at") or "")
config_hash = str(entry.get("config_hash") or "")
data_hash = str(entry.get("data_hash") or "")
run_id = str(entry.get("run_id") or "")
try:
config = json.loads(entry.get("config_json") or "{}")
if not isinstance(config, dict):
config = {}
except (TypeError, ValueError):
config = {}
training_cfg = dict(config)
training_cfg.setdefault("base", base)
training_cfg.setdefault("task", task)
# Row caps: bound the card regardless of how many rows the registry holds.
artifacts = list(artifacts or [])[:_MAX_ROWS]
ancestors = list(ancestors or [])[:_MAX_ROWS]
eval_results = list(eval_results or [])[:_MAX_ROWS]
scorecard: dict[str, Any] = {}
for row in eval_results or []:
bench = row.get("benchmark")
score = row.get("score")
if bench is not None and score is not None:
scorecard[str(bench)] = score
is_adapter = _is_adapter(artifacts, config)
# --- YAML frontmatter --------------------------------------------------- #
fm: list[str] = ["---", f"library_name: {'peft' if is_adapter else 'transformers'}"]
if base:
fm.append(f"base_model: {_yaml_dq(base)}")
fm.append("tags:")
for tag in ("soup-cli", "fine-tuned", "compliance"):
fm.append(f" - {tag}")
fm.append("---")
lines: list[str] = list(fm)
lines += ["", f"# {_safe_md_cell(name)}", ""]
lines.append("Fine-tuned with [Soup CLI](https://github.com/MakazhanAlpamys/Soup).")
lines.append("")
# --- Model details ------------------------------------------------------ #
lines += ["## Model Details", "", "| Field | Value |", "| --- | --- |"]
detail_rows = [
("Name", name),
("Base model", base),
("Task", task),
("Type", "LoRA adapter" if is_adapter else "Full model"),
]
if run_id:
detail_rows.append(("Run id", run_id))
if created:
detail_rows.append(("Created", created))
if tags:
detail_rows.append(("Tags", ", ".join(str(t) for t in tags)))
for key, val in detail_rows:
lines.append(f"| {key} | {_safe_md_cell(val)} |")
lines.append("")
# --- Training ----------------------------------------------------------- #
training_section = _render_training_section(training_cfg)
if training_section:
lines.append(training_section)
# --- Evaluation --------------------------------------------------------- #
eval_section = _render_eval_scorecard(scorecard)
if eval_section:
lines.append(eval_section)
# --- Provenance --------------------------------------------------------- #
if config_hash or data_hash:
lines += ["## Provenance", "", "| Hash | Value |", "| --- | --- |"]
if config_hash:
lines.append(f"| config sha256 | `{_safe_md_cell(config_hash)}` |")
if data_hash:
lines.append(f"| data sha256 | `{_safe_md_cell(data_hash)}` |")
lines.append("")
# --- Lineage ------------------------------------------------------------ #
if ancestors:
lines += ["## Lineage", "", "| Ancestor | Relation | Entry id |", "| --- | --- | --- |"]
for anc in ancestors:
lines.append(
f"| {_safe_md_cell(anc.get('name', ''))} "
f"| {_safe_md_cell(anc.get('relation', ''))} "
f"| `{_safe_md_cell(str(anc.get('id', ''))[:24])}` |"
)
lines.append("")
# --- Artifacts (provenance / compliance) -------------------------------- #
if artifacts:
lines += [
"## Artifacts",
"",
"| Kind | File | SHA256 (prefix) |",
"| --- | --- | --- |",
]
for art in artifacts:
path = str(art.get("path", ""))
filename = path.replace("\\", "/").rsplit("/", 1)[-1]
sha = str(art.get("sha256", ""))[:16]
lines.append(
f"| {_safe_md_cell(art.get('kind', ''))} "
f"| {_safe_md_cell(filename)} "
f"| `{_safe_md_cell(sha)}` |"
)
lines.append("")
# --- Notes -------------------------------------------------------------- #
if notes:
lines += ["## Notes", "", _truncate(html.escape(str(notes)), _MAX_NOTES_CHARS), ""]
# --- Usage -------------------------------------------------------------- #
lines += ["## Usage", ""]
if is_adapter:
lines += [
"```python",
"from peft import PeftModel",
"from transformers import AutoModelForCausalLM, AutoTokenizer",
"",
# json.dumps so a base_model containing a quote cannot produce a
# syntactically broken snippet (the value is inert docs, never run).
f"base = {json.dumps(_safe_md_cell(base) or 'BASE_MODEL')}",
"model = AutoModelForCausalLM.from_pretrained(base)",
'model = PeftModel.from_pretrained(model, "YOUR_REPO")',
"tokenizer = AutoTokenizer.from_pretrained(base)",
"```",
"",
]
else:
lines += [
"```python",
"from transformers import AutoModelForCausalLM, AutoTokenizer",
"",
'model = AutoModelForCausalLM.from_pretrained("YOUR_REPO")',
'tokenizer = AutoTokenizer.from_pretrained("YOUR_REPO")',
"```",
"",
]
return "\n".join(lines) + "\n"
def build_card_for_ref(ref: str) -> str:
"""Resolve ``ref`` in the registry and render its model card.
Raises :class:`CardError` on ambiguous / missing refs (so callers such as
``soup push --card`` can map it to an exit code).
"""
with RegistryStore() as store:
try:
eid = store.resolve(ref)
except AmbiguousRefError as exc:
raise CardError(str(exc)) from exc
if eid is None:
raise CardError(f"Registry entry not found: {ref}")
entry = store.get(eid)
if entry is None:
raise CardError(f"Registry entry not found: {ref}")
artifacts = store.get_artifacts(eid)
evals = store.get_eval_results(eid)
ancestors = store.get_ancestors(eid)
return build_model_card(entry, artifacts, evals, ancestors)
def card(
ref: str = typer.Argument(..., help="Registry entry id / prefix / name:tag"),
output: str = typer.Option(
"MODELCARD.md",
"-o",
"--output",
help="Output markdown path (must stay under the current directory)",
),
) -> None:
"""Generate a HuggingFace model card from a registry entry."""
try:
markdown = build_card_for_ref(ref)
except CardError as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(1) from exc
try:
written = atomic_write_text(markdown, output, field="card output")
except (ValueError, OSError) as exc:
console.print(f"[red]Cannot write card: {escape(str(exc))}[/]")
raise typer.Exit(1) from exc
console.print(
Panel(
f"Model card written to [bold]{escape(written)}[/]",
title="soup card",
border_style="green",
)
)
# Optional re-exports for callers that want the raw pieces.
__all__ = ["build_model_card", "build_card_for_ref", "card", "CardError"]

65
src/soup_cli/commands/ci.py Normal file
View File

@ -0,0 +1,65 @@
"""soup ci — fine-tuning CI helpers (v0.71.35).
``soup ci init`` writes a GitHub Actions workflow that gates every PR on
``soup data validate`` -> ``soup expect`` -> ``soup ship --evidence``.
"""
from __future__ import annotations
import typer
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
from soup_cli.utils.ci_workflow import write_soup_gate_workflow
console = Console()
app = typer.Typer(no_args_is_help=True, help="Fine-tuning CI helpers.")
@app.command("init")
def init(
data: str = typer.Option(
"./data/train.jsonl", "--data", help="Training data path (repo-relative)"
),
suite: str = typer.Option(
"expectations.yaml", "--suite", help="Expectations suite YAML (repo-relative)"
),
evidence: str = typer.Option(
"ship_evidence.json", "--evidence", help="Ship evidence JSON (repo-relative)"
),
python_version: str = typer.Option(
"3.11", "--python", help="Python version for the CI runner (e.g. 3.11)"
),
branch: str = typer.Option("main", "--branch", help="Branch the workflow triggers on"),
output: str = typer.Option(
".github/workflows/soup-gate.yml", "-o", "--output", help="Workflow output path"
),
force: bool = typer.Option(False, "--force", "-f", help="Overwrite an existing workflow"),
) -> None:
"""Write a GitHub Actions fine-tuning gate workflow."""
try:
written = write_soup_gate_workflow(
data_path=data,
suite_path=suite,
evidence_path=evidence,
python_version=python_version,
branch=branch,
output_path=output,
overwrite=force,
)
except (ValueError, TypeError, OSError) as exc:
console.print(f"[red]{escape(str(exc))}[/]")
raise typer.Exit(1) from exc
console.print(
Panel(
f"Wrote [bold]{escape(written)}[/]\n\n"
"The gate runs: [cyan]soup data validate[/] -> [cyan]soup expect[/] -> "
"[cyan]soup ship --evidence[/].\n"
"Edit the paths in the workflow to match your repo.",
title="soup ci init",
border_style="green",
)
)

61
src/soup_cli/commands/export.py
View File

@ -9,6 +9,7 @@ from typing import Optional
import typer
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
console = Console()
@ -404,6 +405,36 @@ def _merge_adapter(
console.print("[green]Adapter merged successfully.[/]")
# llama.cpp's own requirements.txt pins `torch~=2.2.1` against the CPU wheel
# index plus an old `transformers`. Installing it into the user's interpreter
# silently DOWNGRADES a CUDA torch to CPU-only and breaks their training setup
# (observed live on Windows during the v0.71.35 GGUF validation: torch
# 2.5.1+cu -> 2.2.2+cpu, transformers 4.57 -> 4.46). Soup's `[train]` extra
# already provides torch / transformers / numpy, so install ONLY the extra
# packages the convert script needs, unpinned, and never touch the rest.
_CONVERT_EXTRA_DEPS = ("gguf", "sentencepiece", "protobuf")
def _install_convert_deps() -> None:
"""Install the convert script's extra deps without disturbing torch."""
try:
subprocess.run(
[sys.executable, "-m", "pip", "install", "-q", *_CONVERT_EXTRA_DEPS],
check=True,
capture_output=True,
text=True,
)
except (subprocess.CalledProcessError, FileNotFoundError) as exc:
# Non-fatal: the user may already have them, or manage deps themselves.
detail = getattr(exc, "stderr", "") or type(exc).__name__
console.print(
"[yellow]Could not auto-install the GGUF convert dependencies "
f"({', '.join(_CONVERT_EXTRA_DEPS)}).[/]\n"
f"[dim]{escape(str(detail)[:200])}[/]\n"
"Install them manually if the conversion fails."
)
def _find_llama_cpp(user_path: Optional[str] = None) -> Path:
"""Find or clone llama.cpp directory."""
from soup_cli.utils.constants import SOUP_DIR
@ -426,7 +457,12 @@ def _find_llama_cpp(user_path: Optional[str] = None) -> Path:
return path
# 3. Check ~/.soup/llama.cpp
soup_llama = Path(SOUP_DIR) / LLAMA_CPP_DIR_NAME
# SOUP_DIR is a bare name (".soup"), so it MUST be anchored to the home
# directory the way tracker.py / registry/store.py do. Using it relatively
# made the lookup cwd-dependent: llama.cpp was never found in the canonical
# ~/.soup, and the auto-clone dropped a fresh ~200 MB checkout into whatever
# directory the user happened to run from (v0.71.35 GGUF validation).
soup_llama = Path.home() / SOUP_DIR / LLAMA_CPP_DIR_NAME
if soup_llama.exists() and (soup_llama / "convert_hf_to_gguf.py").exists():
return soup_llama
@ -442,15 +478,7 @@ def _find_llama_cpp(user_path: Optional[str] = None) -> Path:
capture_output=True,
text=True,
)
# Install Python requirements for the convert script
requirements = soup_llama / "requirements.txt"
if requirements.exists():
subprocess.run(
[sys.executable, "-m", "pip", "install", "-r", str(requirements), "-q"],
check=True,
capture_output=True,
text=True,
)
_install_convert_deps()
console.print("[green]llama.cpp cloned successfully.[/]")
return soup_llama
except subprocess.CalledProcessError as exc:
@ -505,6 +533,14 @@ def _run_quantize(llama_dir: Path, input_path: Path, output_path: Path, quant_ty
raise typer.Exit(1)
# MSVC / Xcode are multi-config generators: they nest binaries under a
# per-configuration subdirectory (build/bin/Release/llama-quantize.exe) rather
# than the flat build/bin/ that single-config generators (Make/Ninja) produce.
# Without these, `soup export --format gguf` cannot find a correctly-built
# llama.cpp on Windows (v0.71.35 GGUF-on-Windows validation, #70/#144).
_CMAKE_CONFIG_DIRS = ("Release", "RelWithDebInfo", "MinSizeRel", "Debug")
def _find_quantize_binary(llama_dir: Path) -> Optional[Path]:
"""Find the llama-quantize binary."""
# Check common locations
@ -515,6 +551,11 @@ def _find_quantize_binary(llama_dir: Path) -> Optional[Path]:
llama_dir / "llama-quantize.exe",
llama_dir / "build" / "llama-quantize",
]
# Multi-config generator layouts (MSVC on Windows, Xcode on macOS).
for config in _CMAKE_CONFIG_DIRS:
candidates.append(llama_dir / "build" / "bin" / config / "llama-quantize")
candidates.append(llama_dir / "build" / "bin" / config / "llama-quantize.exe")
candidates.append(llama_dir / "build" / config / "llama-quantize.exe")
for candidate in candidates:
if candidate.exists():
return candidate

71
src/soup_cli/commands/push.py
View File

@ -66,6 +66,15 @@ def push(
"Collections / model-card auto-render path (v0.53.10 #152)."
),
),
card: Optional[str] = typer.Option(
None,
"--card",
help=(
"Registry entry id / prefix / name:tag — render its provenance-rich "
"model card (via `soup card`) and upload it as README.md, overriding "
"the auto-generated card (v0.71.35). HF hub only."
),
),
):
"""Push a trained model to HuggingFace Hub (or alternate hub)."""
# v0.53.10 #152 — validate hub at the CLI boundary; only HF is the
@ -163,6 +172,26 @@ def push(
)
# --- Upload ---
# v0.71.35 — --card renders a provenance-rich card from a registry entry
# and uploads it as README.md (HF only). Resolve it up front so a bad ref
# fails fast, before any network upload.
card_override: Optional[str] = None
if card:
if hub_canonical != "hf":
console.print("[yellow]--card is HF-only; ignoring for non-HF hub.[/]")
else:
from rich.markup import escape
from soup_cli.commands.card import CardError, build_card_for_ref
try:
card_override = build_card_for_ref(card)
except CardError as exc:
# escape: an AmbiguousRefError message embeds registry-derived
# entry names (v0.71.35 security review).
console.print(f"[red]--card: {escape(str(exc))}[/]")
raise typer.Exit(1) from exc
# v0.53.10 #152 — non-HF hubs route through utils.hubs.upload_repo
# before we reach the HF-specific Collections / model-card auto-render
# path. Each backend lazy-imports its own SDK; missing-dep surfaces
@ -216,10 +245,18 @@ def push(
commit_message=commit_message,
)
# Generate and upload model card if not present (v2 — includes
# training config and optional eval scorecard)
# Model card: --card <ref> overrides with a registry-driven card;
# otherwise fall back to the path-based v2 card when README is absent
# (v2 — includes training config and optional eval scorecard).
readme_path = model_path / "README.md"
if not readme_path.exists():
if card_override is not None:
api.upload_file(
path_or_fileobj=card_override.encode("utf-8"),
path_in_repo="README.md",
repo_id=repo,
commit_message="Add model card (soup card)",
)
elif not readme_path.exists():
model_card = generate_model_card_v2(
model_path, repo_id=repo, is_adapter=is_adapter,
)
@ -327,13 +364,19 @@ def _load_training_config(model_path: Path) -> dict:
return {}
_UNSAFE_MD_CHARS = re.compile(r"[\|\[\]\(\)!\n\r\t<>]")
_UNSAFE_MD_CHARS = re.compile(r"[\|\[\]\(\)!\n\r\t<>`]")
def _safe_md_cell(value: str) -> str:
"""Neutralise Markdown-active chars so ``value`` cannot inject table rows,
links, images, or raw HTML when rendered on HF Hub."""
return _UNSAFE_MD_CHARS.sub(" ", str(value)).strip()
links, images, raw HTML, or break out of a code span when rendered on HF Hub.
Also strips C0/ESC control bytes (v0.71.35 security review): the rendered
card is a file an operator may later ``cat``/``less``, where an embedded
ANSI/OSC sequence could manipulate the terminal.
"""
text = "".join(ch for ch in str(value) if ord(ch) >= 0x20 or ch in "\n\r\t")
return _UNSAFE_MD_CHARS.sub(" ", text).strip()
def _render_eval_scorecard(eval_scorecard: Optional[dict]) -> str:
@ -353,20 +396,28 @@ def _render_eval_scorecard(eval_scorecard: Optional[dict]) -> str:
def _render_training_section(training_cfg: dict) -> str:
"""Render the ``## Training`` section.
Every interpolated value is passed through :func:`_safe_md_cell`
(v0.71.35 security review): ``base`` / ``scheduler`` have no charset
validator in ``SoupConfig``, so a crafted-but-schema-valid config could
otherwise smuggle raw markdown/HTML (or a backtick that breaks out of the
surrounding code span) into a card published to the HF Hub.
"""
if not training_cfg:
return ""
task = training_cfg.get("task") or "sft"
task = _safe_md_cell(training_cfg.get("task") or "sft")
training = training_cfg.get("training", {}) or {}
base = training_cfg.get("base") or ""
base = _safe_md_cell(training_cfg.get("base") or "")
lines = ["## Training", "", f"- **Task:** {task}"]
if base:
lines.append(f"- **Base model:** `{base}`")
for key in ("epochs", "lr", "batch_size", "optimizer", "scheduler"):
if key in training:
lines.append(f"- **{key}:** {training[key]}")
lines.append(f"- **{key}:** {_safe_md_cell(training[key])}")
recipe = training_cfg.get("recipe")
if recipe:
lines.append(f"- **Recipe:** `{recipe}`")
lines.append(f"- **Recipe:** `{_safe_md_cell(recipe)}`")
lines.append("")
return "\n".join(lines)

45
src/soup_cli/templates/eu-ai-act.yaml Normal file
View File

@ -0,0 +1,45 @@
# Soup compliance template: EU AI Act (Annex XI/XII technical documentation)
# Starting point for fine-tuning a model destined for the EU market.
#
# This file is a normal training config — the compliance controls are Soup
# CLI flags/commands, not config keys. Run them around your training:
#
# 1. Clean the training data (Annex XI data-governance obligations):
# soup data decontaminate ./data/train.jsonl # drop public-benchmark overlap
# soup data pii ./data/train.jsonl # flag personal data
#
# 2. Auto-generate the Annex XI/XII technical documentation + measure energy:
# soup train --config soup.yaml \
# --annex-xi annex_xi.md \ # or annex_xi.pdf (pip install 'soup-cli[pdf]')
# --track-energy --energy-country DEU \ # codecarbon (pip install 'soup-cli[carbon]')
# --energy-out energy.json
#
# 3. Fold the measured energy into the ML-BOM (transparency):
# soup bom emit --name model --base-sha <hex> --config-sha <hex> \
# --energy energy.json --format both
#
# 4. The audit log is ON by default (record-keeping obligations):
# soup audit-log tail
#
# License note: use a license-clean base for EU deployments (Apache-2.0 below).
base: Qwen/Qwen2.5-7B-Instruct
task: sft
data:
train: ./data/train.jsonl
format: alpaca
val_split: 0.1
max_length: 2048
training:
epochs: 3
lr: 2e-5
batch_size: auto
lora:
r: 64
alpha: 16
target_modules: auto
quantization: 4bit
output: ./output

46
src/soup_cli/templates/hipaa.yaml Normal file
View File

@ -0,0 +1,46 @@
# Soup compliance template: HIPAA (Protected Health Information)
# Starting point for fine-tuning on health data under HIPAA safeguards.
#
# This file is a normal training config — the compliance controls are Soup
# CLI flags/commands, not config keys. Run them around your training:
#
# 1. Scrub PHI BEFORE training (keep raw PHI off the training box):
# soup data pii ./data/train.jsonl # flag emails / phones / SSNs / MRNs
# soup data decontaminate ./data/train.jsonl # drop public-benchmark overlap
#
# 2. The HIPAA/SOC2 audit log is ON by default (do NOT pass --no-audit-log):
# soup audit-log tail # review the per-command trail
#
# 3. Train with a reproducibility receipt (who/what/when, SR 11-7 style):
# soup train --config soup.yaml --repro-receipt receipt.json
#
# 4. Prove provenance + integrity of the artifact you ship:
# soup bom emit --name phi-model --base-sha <hex> --config-sha <hex> --format both
# soup adapters sign ./output --backend ed25519 --generate-key key.pem
# soup attest emit --stage train --subject phi-model --sha <hex> --sign ed25519 --key key.pem
#
# 5. Move the model across a data diode without exposing PHI:
# soup airgap-bundle --model ./output --output phi-model.tar --repro-receipt receipt.json
#
# License note: use a license-clean base for regulated deployments (Apache-2.0 below).
base: Qwen/Qwen2.5-7B-Instruct
task: sft
data:
train: ./data/train.jsonl
format: alpaca
val_split: 0.1
max_length: 2048
training:
epochs: 3
lr: 2e-5
batch_size: auto
lora:
r: 64
alpha: 16
target_modules: auto
quantization: 4bit
output: ./output

4
src/soup_cli/templates/manifest.json
View File

@ -5,6 +5,8 @@
"chat": "chat.yaml",
"code": "code.yaml",
"embedding": "embedding.yaml",
"eu-ai-act": "eu-ai-act.yaml",
"hipaa": "hipaa.yaml",
"ipo": "ipo.yaml",
"kto": "kto.yaml",
"longcontext": "longcontext.yaml",
@ -15,6 +17,8 @@
"reasoning": "reasoning.yaml",
"rlhf": "rlhf.yaml",
"simpo": "simpo.yaml",
"soc2": "soc2.yaml",
"sr-11-7": "sr-11-7.yaml",
"tool-calling": "tool-calling.yaml",
"vision": "vision.yaml"
},

45
src/soup_cli/templates/soc2.yaml Normal file
View File

@ -0,0 +1,45 @@
# Soup compliance template: SOC 2 (Trust Services Criteria)
# Starting point for fine-tuning under a SOC 2 change-management + audit regime.
#
# This file is a normal training config — the compliance controls are Soup
# CLI flags/commands, not config keys. Run them around your training:
#
# 1. The audit log is ON by default (evidence for the Security criterion):
# soup audit-log tail # tail the per-command trail
# soup audit-log rotate # force a rotation pass
#
# 2. Pin the exact inputs so a run is reproducible (Change Management):
# soup lock write --base-sha <hex> --dataset-sha <hex> --env-lock soup-env.lock
# soup train --config soup.yaml --repro-receipt receipt.json
#
# 3. Emit an ML-BOM + signed attestation for every shipped artifact:
# soup bom emit --name model --base-sha <hex> --config-sha <hex> --format both
# soup adapters sign ./output --backend ed25519 --generate-key key.pem
# soup adapters verify ./output --strict --public-key key.pub.pem
# soup attest emit --stage train --subject model --sha <hex> --sign ed25519 --key key.pem
#
# 4. Gate merges/deploys on a SHIP verdict (Availability / Processing Integrity):
# soup ship --evidence ev.json # exit 0 = SHIP, 2 = DON'T SHIP
#
# License note: use a license-clean base for regulated deployments (Apache-2.0 below).
base: Qwen/Qwen2.5-7B-Instruct
task: sft
data:
train: ./data/train.jsonl
format: alpaca
val_split: 0.1
max_length: 2048
training:
epochs: 3
lr: 2e-5
batch_size: auto
lora:
r: 64
alpha: 16
target_modules: auto
quantization: 4bit
output: ./output

43
src/soup_cli/templates/sr-11-7.yaml Normal file
View File

@ -0,0 +1,43 @@
# Soup compliance template: SR 11-7 (Model Risk Management, banking)
# Starting point for fine-tuning under a model-risk-management framework.
#
# This file is a normal training config — the compliance controls are Soup
# CLI flags/commands, not config keys. Run them around your training:
#
# 1. Capture a reproducibility receipt (seeds / kernels / GPU / OS):
# soup train --config soup.yaml --repro-receipt receipt.json
#
# 2. Independent effective-challenge: diagnose failure modes + SHIP gate:
# soup diagnose <run-id> --output diag.json --badge diag.svg
# soup ship --base <id> --adapter ./output --task-eval task.jsonl
# # exit 0 = SHIP, 2 = DON'T SHIP (leg 1 task-win AND leg 2 no regression)
#
# 3. Emit an ML-BOM + attestation for the model inventory / provenance:
# soup bom emit --name model --base-sha <hex> --config-sha <hex> --format both
# soup attest emit --stage train --subject model --sha <hex> --sign ed25519 --key key.pem
#
# 4. The audit log is ON by default (ongoing monitoring / record-keeping):
# soup audit-log tail
#
# License note: use a license-clean base for regulated deployments (Apache-2.0 below).
base: Qwen/Qwen2.5-7B-Instruct
task: sft
data:
train: ./data/train.jsonl
format: alpaca
val_split: 0.1
max_length: 2048
training:
epochs: 3
lr: 2e-5
batch_size: auto
lora:
r: 64
alpha: 16
target_modules: auto
quantization: 4bit
output: ./output

179
src/soup_cli/utils/ci_workflow.py Normal file
View File

@ -0,0 +1,179 @@
"""Render a GitHub Actions "fine-tuning gate" workflow (v0.71.35).
``soup ci init`` writes ``.github/workflows/soup-gate.yml`` — a CI job that
gates every PR on three Soup checks:
soup data validate <data> # dataset format compliance
soup expect <data> <suite> # expectations suite (PII / length / refusal)
soup ship --evidence <ev.json> # SHIP / DON'T-SHIP verdict (exit 2 = block)
The workflow body is a fixed YAML skeleton; only a handful of tokens are
interpolated. Following ``eval_gate_hook.py``: any value placed into a ``run:``
shell step is passed through ``shlex.quote`` (+ control-char rejection) so a
crafted path cannot chain commands, and every path is validated to stay under
the repo root. There is NO top-level torch import.
"""
from __future__ import annotations
import os
import re
import shlex
import tempfile
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink, is_under_cwd
# GitHub allows quite permissive ref names; we constrain to a safe subset that
# also cannot break the YAML scalar or a shell step.
_BRANCH_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._/-]{0,254}$")
_PY_RE = re.compile(r"^\d+\.\d+$")
_MAX_PATH_LEN = 4096
_MAX_FILE_BYTES = 64 * 1024
_WORKFLOW_TEMPLATE = """\
# Soup fine-tuning gate — generated by `soup ci init` (v0.71.35).
# Gates every PR on: data validate -> expectations -> SHIP verdict.
# Edit the paths below to match your repo. The `soup ship` step needs a
# committed evidence JSON (see `soup ship --evidence`); remove it if unused.
name: Soup Fine-tuning Gate
on:
push:
branches: [{branch}]
pull_request:
branches: [{branch}]
jobs:
soup-gate:
runs-on: ubuntu-latest
env:
PYTHONUTF8: "1"
PYTHONIOENCODING: "utf-8"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "{python_version}"
- name: Install Soup CLI
# Core install is enough: `data validate`, `expect` and
# `ship --evidence` are all no-torch paths.
run: pip install soup-cli
- name: Validate training data
run: soup data validate {data}
- name: Run expectations suite
run: soup expect {data} {suite}
- name: SHIP / DON'T-SHIP gate
run: soup ship --evidence {evidence}
"""
def _validate_path(value: str, field: str) -> str:
if isinstance(value, bool) or not isinstance(value, str):
raise TypeError(f"{field} must be str")
if not value:
raise ValueError(f"{field} must be non-empty")
if "\x00" in value:
raise ValueError(f"{field} must not contain NUL")
if "\n" in value or "\r" in value:
raise ValueError(f"{field} must be a single line")
if len(value) > _MAX_PATH_LEN:
raise ValueError(f"{field} exceeds {_MAX_PATH_LEN} characters")
if not is_under_cwd(value):
raise ValueError(f"{field} must stay under the repository root")
return value
def _safe_shell_quote(value: str) -> str:
"""``shlex.quote`` with a control-char rejection prelude (defence in depth)."""
if any(ord(ch) < 0x20 for ch in value):
raise ValueError("value contains control characters")
return shlex.quote(value)
def render_soup_gate_workflow(
*,
data_path: str,
suite_path: str,
evidence_path: str,
python_version: str = "3.11",
branch: str = "main",
) -> str:
"""Render the workflow YAML body. Deterministic, no I/O.
Every path is validated to stay under the repo root and shell-quoted before
it reaches a ``run:`` step; ``python_version`` / ``branch`` are regex-gated.
"""
data = _validate_path(data_path, "data_path")
suite = _validate_path(suite_path, "suite_path")
evidence = _validate_path(evidence_path, "evidence_path")
if not isinstance(python_version, str) or not _PY_RE.match(python_version):
raise ValueError("python_version must look like '3.11'")
if not isinstance(branch, str) or not _BRANCH_RE.match(branch):
raise ValueError("branch has an unsupported name")
return _WORKFLOW_TEMPLATE.format(
branch=branch,
python_version=python_version,
data=_safe_shell_quote(data),
suite=_safe_shell_quote(suite),
evidence=_safe_shell_quote(evidence),
)
def write_soup_gate_workflow(
*,
data_path: str,
suite_path: str,
evidence_path: str,
python_version: str = "3.11",
branch: str = "main",
output_path: str = ".github/workflows/soup-gate.yml",
overwrite: bool = False,
) -> str:
"""Render + atomically write the workflow. Returns the path written.
Refuses to clobber an existing file unless ``overwrite=True``; rejects a
symlink destination (TOCTOU) and any path outside the repo root.
"""
body = render_soup_gate_workflow(
data_path=data_path,
suite_path=suite_path,
evidence_path=evidence_path,
python_version=python_version,
branch=branch,
)
if isinstance(output_path, bool) or not isinstance(output_path, str):
raise TypeError("output_path must be str")
if not output_path:
raise ValueError("output_path must be non-empty")
if "\x00" in output_path:
raise ValueError("output_path must not contain NUL")
if not isinstance(overwrite, bool):
raise TypeError("overwrite must be bool")
# Single-source-of-truth containment + symlink/reparse-point rejection.
# A hand-rolled S_ISLNK check misses Windows junctions, so always delegate
# to the shared helper (v0.71.35 security review).
enforce_under_cwd_and_no_symlink(output_path, "output_path")
if os.path.lexists(output_path) and not overwrite:
raise ValueError("workflow already exists; pass --force to overwrite")
if len(body.encode("utf-8")) > _MAX_FILE_BYTES:
raise ValueError("rendered workflow exceeds 64 KiB cap")
parent = os.path.dirname(os.path.abspath(output_path)) or "."
os.makedirs(parent, exist_ok=True)
fd, tmp = tempfile.mkstemp(prefix=".soup-gate.", dir=parent)
try:
with os.fdopen(fd, "w", encoding="utf-8") as handle:
handle.write(body)
os.replace(tmp, output_path)
except Exception:
try:
os.unlink(tmp)
except OSError:
pass
raise
return output_path
__all__ = ["render_soup_gate_workflow", "write_soup_gate_workflow"]

8
src/soup_cli/utils/ollama.py
View File

@ -1,5 +1,6 @@
"""Ollama integration utilities — detect, deploy, list, remove models."""
import os
import re
import subprocess
from pathlib import Path
@ -167,7 +168,12 @@ def create_modelfile(
Returns:
Modelfile content as string.
"""
lines = [f"FROM {gguf_path}"]
# `ollama create` resolves a relative FROM against the Modelfile's own
# directory, and we write the Modelfile to a temp dir — so a relative GGUF
# path made Ollama treat it as a remote model name and fail with
# "pull model manifest: file does not exist". Always emit an absolute path
# (v0.71.35 GGUF-on-Windows validation).
lines = [f"FROM {os.path.abspath(str(gguf_path))}"]
# Template
if template:

993
tests/test_v07135.py Normal file
View File

@ -0,0 +1,993 @@
"""v0.71.35 — Compliance pack.
Covers:
* Compliance init templates (hipaa / soc2 / eu-ai-act / sr-11-7)
* ``soup card`` — HF model-card autogen from a registry entry
* ``soup ci init`` — render .github/workflows/soup-gate.yml
"""
from __future__ import annotations
import json
import os
import pytest
import yaml
from typer.testing import CliRunner
from soup_cli.config.loader import load_config_from_string
from soup_cli.templates import list_templates, load_template
runner = CliRunner()
COMPLIANCE_TEMPLATES = ["hipaa", "soc2", "eu-ai-act", "sr-11-7"]
# --------------------------------------------------------------------------- #
# Compliance init templates
# --------------------------------------------------------------------------- #
class TestComplianceTemplates:
@pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES)
def test_template_is_listed(self, name):
assert name in list_templates()
@pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES)
def test_template_loads(self, name):
body = load_template(name)
assert body is not None and body.strip()
@pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES)
def test_template_parses_as_config(self, name):
"""The YAML body must be a valid SoupConfig (train-time keys only)."""
body = load_template(name)
cfg = load_config_from_string(body)
assert cfg.base
assert cfg.task
@pytest.mark.parametrize("name", COMPLIANCE_TEMPLATES)
def test_template_has_compliance_guidance(self, name):
"""Compliance behaviours are CLI flags/commands, not schema keys, so the
template must carry header-comment guidance pointing users at them."""
body = load_template(name)
lowered = body.lower()
# every compliance template should mention the audit log + at least one
# provenance command
assert "audit" in lowered
assert any(tok in lowered for tok in ("bom", "attest", "repro-receipt", "sign"))
def test_manifest_lists_compliance_templates(self):
import soup_cli.templates as tpl
manifest = tpl._load_manifest()
for name in COMPLIANCE_TEMPLATES:
assert name in manifest["templates"]
def test_init_writes_compliance_template(self, tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
out = tmp_path / "soup.yaml"
result = runner.invoke(app, ["init", "--template", "hipaa", "-o", str(out), "--force"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert out.exists()
cfg = load_config_from_string(out.read_text(encoding="utf-8"))
assert cfg.base
# --------------------------------------------------------------------------- #
# soup card — model-card autogen from a registry entry
# --------------------------------------------------------------------------- #
def _make_entry(db_path, *, notes=None, with_eval=False, with_artifact=None, parent=False):
"""Seed a registry entry (and optional artifact / lineage) in a temp DB."""
from soup_cli.registry.store import RegistryStore
with RegistryStore(db_path=db_path) as store:
parent_id = None
if parent:
parent_id = store.push(
name="base-run", tag="v1", base_model="Qwen/Qwen2.5-7B-Instruct",
task="sft", run_id=None, config={"base": "Qwen/Qwen2.5-7B-Instruct"},
notes=None,
)
eid = store.push(
name="phi-model", tag="v1", base_model="Qwen/Qwen2.5-7B-Instruct",
task="sft", run_id=None,
config={"base": "Qwen/Qwen2.5-7B-Instruct", "task": "sft",
"training": {"epochs": 3, "lr": 2e-5}},
notes=notes,
)
if parent_id:
store.add_lineage(child_id=eid, parent_id=parent_id, relation="forked_from")
if with_artifact is not None:
store.add_artifact(entry_id=eid, kind=with_artifact[0],
path=with_artifact[1], enforce_cwd=True)
return eid
class TestBuildModelCard:
def test_pure_card_has_core_sections(self):
from soup_cli.commands.card import build_model_card
entry = {
"id": "reg_x", "name": "my-model", "base_model": "Qwen/Qwen2.5-7B-Instruct",
"task": "sft", "created_at": "2026-07-15T00:00:00", "notes": "clean run",
"config_hash": "a" * 64, "data_hash": "b" * 64, "run_id": None,
"tags": ["prod"],
"config_json": json.dumps({"base": "Qwen/Qwen2.5-7B-Instruct", "task": "sft",
"training": {"epochs": 3, "lr": 2e-5}}),
}
md = build_model_card(entry, [], [], [])
assert md.startswith("---") # YAML frontmatter
assert "# my-model" in md
assert "Qwen/Qwen2.5-7B-Instruct" in md
assert "## Training" in md
def test_notes_html_escaped(self):
from soup_cli.commands.card import build_model_card
entry = {
"id": "reg_x", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": "<script>alert(1)</script>",
"config_hash": "", "data_hash": "", "run_id": None, "tags": [],
"config_json": "{}",
}
md = build_model_card(entry, [], [], [])
assert "<script>" not in md
assert "&lt;script&gt;" in md
def test_eval_scorecard_rendered(self):
from soup_cli.commands.card import build_model_card
entry = {
"id": "r", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": None, "config_hash": "", "data_hash": "",
"run_id": "run1", "tags": [], "config_json": "{}",
}
evals = [{"benchmark": "mmlu", "score": 0.61}]
md = build_model_card(entry, [], evals, [])
assert "## Evaluation" in md
assert "mmlu" in md
assert "0.610" in md
def test_frontmatter_yaml_safe_base(self):
"""A hostile base_model must not break the YAML frontmatter."""
from soup_cli.commands.card import build_model_card
entry = {
"id": "r", "name": "m", "base_model": 'evil"\ninjected: true',
"task": "sft", "created_at": "t", "notes": None,
"config_hash": "", "data_hash": "", "run_id": None, "tags": [],
"config_json": "{}",
}
md = build_model_card(entry, [], [], [])
front = md.split("---", 2)[1]
parsed = yaml.safe_load(front)
assert "injected" not in parsed # injection neutralised
class TestCardCli:
def test_help(self):
from soup_cli.cli import app
result = runner.invoke(app, ["card", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_card_happy(self, tmp_path, monkeypatch):
from soup_cli.cli import app
db = tmp_path / "reg.db"
eid = _make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["card", eid, "-o", "CARD.md"])
assert result.exit_code == 0, (result.output, repr(result.exception))
card = (tmp_path / "CARD.md").read_text(encoding="utf-8")
assert "phi-model" in card
assert "Qwen/Qwen2.5-7B-Instruct" in card
def test_card_with_lineage_and_artifact(self, tmp_path, monkeypatch):
from soup_cli.cli import app
db = tmp_path / "reg.db"
monkeypatch.chdir(tmp_path)
gguf = tmp_path / "model.q4_k_m.gguf"
gguf.write_text("stub", encoding="utf-8")
eid = _make_entry(db, parent=True, with_artifact=("gguf", str(gguf)))
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
result = runner.invoke(app, ["card", eid, "-o", "CARD.md"])
assert result.exit_code == 0, (result.output, repr(result.exception))
card = (tmp_path / "CARD.md").read_text(encoding="utf-8")
assert "base-run" in card # ancestor
assert "model.q4_k_m.gguf" in card # artifact link
def test_card_not_found(self, tmp_path, monkeypatch):
from soup_cli.cli import app
db = tmp_path / "reg.db"
_make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["card", "reg_nonexistent", "-o", "CARD.md"])
assert result.exit_code == 1
assert "not found" in result.output.lower()
def test_card_output_outside_cwd_rejected(self, tmp_path, monkeypatch):
from soup_cli.cli import app
db = tmp_path / "reg.db"
eid = _make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
work = tmp_path / "work"
work.mkdir()
monkeypatch.chdir(work)
result = runner.invoke(app, ["card", eid, "-o", "../escape.md"])
assert result.exit_code == 1
assert not (tmp_path / "escape.md").exists()
class TestPushCardRider:
def test_push_has_card_option(self):
from soup_cli.cli import app
result = runner.invoke(app, ["push", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "--card" in result.output
def test_build_card_for_ref_unknown_raises(self, tmp_path, monkeypatch):
from soup_cli.commands.card import CardError, build_card_for_ref
db = tmp_path / "reg.db"
_make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
monkeypatch.chdir(tmp_path)
with pytest.raises(CardError):
build_card_for_ref("reg_does_not_exist")
# --------------------------------------------------------------------------- #
# soup ci init — render .github/workflows/soup-gate.yml
# --------------------------------------------------------------------------- #
class TestRenderWorkflow:
def test_render_is_valid_yaml_with_gate_steps(self):
from soup_cli.utils.ci_workflow import render_soup_gate_workflow
body = render_soup_gate_workflow(
data_path="./data/train.jsonl",
suite_path="expectations.yaml",
evidence_path="ship_evidence.json",
)
doc = yaml.safe_load(body)
assert "jobs" in doc
steps = doc["jobs"]["soup-gate"]["steps"]
runs = " ".join(s.get("run", "") for s in steps)
assert "soup data validate" in runs
assert "soup expect" in runs
assert "soup ship --evidence" in runs
def test_render_defaults_python_and_branch(self):
from soup_cli.utils.ci_workflow import render_soup_gate_workflow
body = render_soup_gate_workflow(
data_path="data/train.jsonl",
suite_path="suite.yaml",
evidence_path="ev.json",
)
yaml.safe_load(body) # must parse
assert '3.11' in body
# `on:` parses to True in YAML 1.1 (the "Norway problem"); assert via body
assert "pull_request" in body
def test_render_shell_quotes_injection(self):
from soup_cli.utils.ci_workflow import render_soup_gate_workflow
body = render_soup_gate_workflow(
data_path="data/train.jsonl; rm -rf /",
suite_path="suite.yaml",
evidence_path="ev.json",
)
# the malicious path must be shell-quoted as a single token, not left
# as a chainable command
assert "'data/train.jsonl; rm -rf /'" in body
@pytest.mark.parametrize("bad", ["", "a\nb", "a\x00b", "../escape.jsonl"])
def test_render_rejects_bad_path(self, bad):
from soup_cli.utils.ci_workflow import render_soup_gate_workflow
with pytest.raises((ValueError, TypeError)):
render_soup_gate_workflow(
data_path=bad, suite_path="s.yaml", evidence_path="e.json",
)
@pytest.mark.parametrize("bad_py", ["3", "3.x", "3.11; rm", "abc"])
def test_render_rejects_bad_python(self, bad_py):
from soup_cli.utils.ci_workflow import render_soup_gate_workflow
with pytest.raises(ValueError):
render_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
python_version=bad_py,
)
@pytest.mark.parametrize("bad_branch", ["a b", "a$b", "a;b", ""])
def test_render_rejects_bad_branch(self, bad_branch):
from soup_cli.utils.ci_workflow import render_soup_gate_workflow
with pytest.raises(ValueError):
render_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
branch=bad_branch,
)
class TestCiInitCli:
def test_help(self):
from soup_cli.cli import app
result = runner.invoke(app, ["ci", "init", "--help"])
assert result.exit_code == 0, (result.output, repr(result.exception))
def test_ci_init_writes_workflow(self, tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["ci", "init", "--data", "data/train.jsonl"])
assert result.exit_code == 0, (result.output, repr(result.exception))
wf = tmp_path / ".github" / "workflows" / "soup-gate.yml"
assert wf.exists()
doc = yaml.safe_load(wf.read_text(encoding="utf-8"))
assert "soup-gate" in doc["jobs"]
def test_ci_init_refuses_overwrite_without_force(self, tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
first = runner.invoke(app, ["ci", "init"])
assert first.exit_code == 0, (first.output, repr(first.exception))
second = runner.invoke(app, ["ci", "init"])
assert second.exit_code == 1
assert "exists" in second.output.lower()
forced = runner.invoke(app, ["ci", "init", "--force"])
assert forced.exit_code == 0, (forced.output, repr(forced.exception))
def test_ci_init_rejects_bad_python(self, tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["ci", "init", "--python", "3.x"])
assert result.exit_code == 1
assert not (tmp_path / ".github" / "workflows" / "soup-gate.yml").exists()
class TestCodeReviewFixes:
"""Regression pins for the v0.71.35 code-review findings."""
def test_workflow_installs_published_package_not_editable(self):
"""HIGH: `pip install -e ".[dev]"` only works inside the Soup source
tree, but `soup ci init` targets a DOWNSTREAM fine-tuning repo (no
pyproject.toml), so the workflow's first step would break for every
real user."""
from soup_cli.utils.ci_workflow import render_soup_gate_workflow
body = render_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
)
assert "pip install -e" not in body
assert "pip install soup-cli" in body
def test_usage_snippet_survives_quote_in_base_model(self):
"""LOW: a `\"` in base_model must not emit broken Python."""
from soup_cli.commands.card import build_model_card
entry = {
"id": "r", "name": "m", "base_model": 'we"ird/model',
"task": "sft", "created_at": "t", "notes": None,
"config_hash": "", "data_hash": "", "run_id": None, "tags": [],
"config_json": "{}",
}
md = build_model_card(entry, [{"kind": "adapter", "path": "a", "sha256": "x"}], [], [])
# extract the python usage snippet and compile it
snippet = md.split("```python", 1)[1].split("```", 1)[0]
compile(snippet, "<card>", "exec") # raises SyntaxError if broken
class TestSecurityReviewFixes:
"""Regression pins for the v0.71.35 security-review findings."""
def test_training_section_escapes_hostile_base(self):
"""HIGH: `base`/`scheduler` have no charset validator in SoupConfig, so
a crafted config could smuggle raw HTML (or a backtick breaking out of
the code span) into a card published to the public HF Hub."""
from soup_cli.commands.card import build_model_card
hostile = '` </code><script>alert(1)</script><code> `'
entry = {
"id": "r", "name": "m", "base_model": "safe/base", "task": "sft",
"created_at": "t", "notes": None, "config_hash": "", "data_hash": "",
"run_id": None, "tags": [],
"config_json": json.dumps({
"base": hostile, "task": "sft",
"training": {"epochs": 1, "scheduler": "<img src=x onerror=alert(1)>"},
}),
}
md = build_model_card(entry, [], [], [])
# The security property is that no HTML TAG survives — angle brackets and
# backticks are neutralised, so the payload is inert prose. (The bare
# word "onerror" surviving as plain text is harmless.)
assert "<script>" not in md
assert "<img" not in md
assert "</code>" not in md
# and the backtick cannot break out of the `code span`
training_line = next(li for li in md.splitlines() if li.startswith("- **Base model:**"))
assert training_line.count("`") == 2
def test_safe_md_cell_strips_control_bytes(self):
"""LOW: an ESC byte in registry text must not survive into the card."""
from soup_cli.commands.push import _safe_md_cell
assert "\x1b" not in _safe_md_cell("a\x1b[31mred\x1b[0m")
assert "\x00" not in _safe_md_cell("a\x00b")
def test_safe_md_cell_neutralises_backtick(self):
from soup_cli.commands.push import _safe_md_cell
assert "`" not in _safe_md_cell("br`eak")
def test_card_notes_are_truncated(self):
"""LOW/INFO: unbounded registry notes must not render an unbounded card."""
from soup_cli.commands.card import _MAX_NOTES_CHARS, build_model_card
entry = {
"id": "r", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": "A" * (_MAX_NOTES_CHARS + 5_000),
"config_hash": "", "data_hash": "", "run_id": None, "tags": [],
"config_json": "{}",
}
md = build_model_card(entry, [], [], [])
assert "[truncated]" in md
def test_card_rows_are_capped(self):
from soup_cli.commands.card import _MAX_ROWS, build_model_card
arts = [
{"kind": "gguf", "path": f"m{i}.gguf", "sha256": "x"}
for i in range(_MAX_ROWS + 50)
]
entry = {
"id": "r", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": None, "config_hash": "", "data_hash": "",
"run_id": None, "tags": [], "config_json": "{}",
}
md = build_model_card(entry, arts, [], [])
assert f"m{_MAX_ROWS + 10}.gguf" not in md # beyond the cap
assert "m0.gguf" in md
def test_ci_workflow_uses_shared_containment_helper(self):
"""MEDIUM: the hand-rolled S_ISLNK guard missed Windows junctions."""
import inspect
from soup_cli.utils import ci_workflow
src = inspect.getsource(ci_workflow)
assert "enforce_under_cwd_and_no_symlink" in src
# no second, weaker copy of the guard (a bare S_ISLNK call misses
# Windows junctions); a mention in a comment is fine.
assert "stat.S_ISLNK" not in src
def test_ci_workflow_output_outside_cwd_rejected(self, tmp_path, monkeypatch):
from soup_cli.utils.ci_workflow import write_soup_gate_workflow
work = tmp_path / "work"
work.mkdir()
monkeypatch.chdir(work)
with pytest.raises(ValueError):
write_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
output_path="../escape.yml",
)
# --------------------------------------------------------------------------- #
# tdd-review gap closures (H1 / M1-M4 / L1-L7)
# --------------------------------------------------------------------------- #
def _model_dir(tmp_path):
"""A minimal full-model dir that `soup push` accepts."""
d = tmp_path / "out"
d.mkdir(exist_ok=True)
(d / "config.json").write_text("{}", encoding="utf-8")
return d
class TestPushCardIntegration:
"""H1 — end-to-end `push --card` wiring (was only --help-tested)."""
def test_push_card_uploads_registry_card(self, tmp_path, monkeypatch):
from unittest.mock import MagicMock
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("HF_TOKEN", "t1")
db = tmp_path / "reg.db"
eid = _make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
_model_dir(tmp_path)
fake_api = MagicMock()
monkeypatch.setattr("soup_cli.utils.hf.get_hf_api", lambda **_: fake_api)
result = runner.invoke(
app, ["push", "--model", "out", "--repo", "user/m", "--card", eid]
)
assert result.exit_code == 0, (result.output, repr(result.exception))
readme_calls = [
c for c in fake_api.upload_file.call_args_list
if c.kwargs.get("path_in_repo") == "README.md"
]
assert readme_calls, fake_api.upload_file.call_args_list
body = readme_calls[0].kwargs["path_or_fileobj"].decode("utf-8")
assert "phi-model" in body # registry-driven, not the path-based card
assert "Qwen/Qwen2.5-7B-Instruct" in body
def test_push_card_error_exits_1_before_network(self, tmp_path, monkeypatch):
from unittest.mock import MagicMock
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("HF_TOKEN", "t1")
db = tmp_path / "reg.db"
_make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
_model_dir(tmp_path)
fake_api = MagicMock()
monkeypatch.setattr("soup_cli.utils.hf.get_hf_api", lambda **_: fake_api)
result = runner.invoke(
app, ["push", "--model", "out", "--repo", "user/m", "--card", "reg_nope"]
)
assert result.exit_code == 1
assert "--card" in result.output
# fails fast: no repo creation / upload attempted
assert not fake_api.create_repo.called
assert not fake_api.upload_folder.called
def test_push_card_non_hf_hub_warns_and_ignores(self, tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("HF_TOKEN", "t1")
db = tmp_path / "reg.db"
eid = _make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
_model_dir(tmp_path)
uploaded = {}
def _fake_upload_repo(hub, repo, **kwargs):
uploaded["hub"] = hub
monkeypatch.setattr("soup_cli.utils.hubs.upload_repo", _fake_upload_repo)
result = runner.invoke(
app,
["push", "--model", "out", "--repo", "user/m", "--hub", "modelscope",
"--card", eid],
)
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "HF-only" in result.output
assert uploaded.get("hub") == "modelscope"
class TestCardErrorPaths:
def test_ambiguous_ref_raises_card_error(self, tmp_path, monkeypatch):
"""M1 — every entry id shares the reg_ prefix, so a bare prefix is
ambiguous once 2 entries exist."""
from soup_cli.commands.card import CardError, build_card_for_ref
db = tmp_path / "reg.db"
monkeypatch.chdir(tmp_path)
_make_entry(db)
_make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
with pytest.raises(CardError):
build_card_for_ref("reg_")
def test_ambiguous_ref_cli_exits_1(self, tmp_path, monkeypatch):
from soup_cli.cli import app
db = tmp_path / "reg.db"
monkeypatch.chdir(tmp_path)
_make_entry(db)
_make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
result = runner.invoke(app, ["card", "reg_", "-o", "CARD.md"])
assert result.exit_code == 1
def test_card_write_oserror_is_friendly(self, tmp_path, monkeypatch):
"""M2 — `-o .` is a directory: passes containment, fails on os.replace."""
from soup_cli.cli import app
db = tmp_path / "reg.db"
eid = _make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["card", eid, "-o", "."])
assert result.exit_code == 1
assert "cannot write card" in result.output.lower()
def test_card_overwrites_existing_output(self, tmp_path, monkeypatch):
"""L7 — pin the intended behaviour: regeneration overwrites, no --force."""
from soup_cli.cli import app
db = tmp_path / "reg.db"
eid = _make_entry(db)
monkeypatch.setenv("SOUP_REGISTRY_DB_PATH", str(db))
monkeypatch.chdir(tmp_path)
(tmp_path / "CARD.md").write_text("STALE", encoding="utf-8")
result = runner.invoke(app, ["card", eid, "-o", "CARD.md"])
assert result.exit_code == 0, (result.output, repr(result.exception))
assert "STALE" not in (tmp_path / "CARD.md").read_text(encoding="utf-8")
class TestBuildModelCardEdges:
def test_empty_entry_uses_defaults(self):
"""M3 — name -> 'model', no base_model frontmatter line, no crash."""
from soup_cli.commands.card import build_model_card
md = build_model_card({}, [], [], [])
assert "# model" in md
front = md.split("---", 2)[1]
assert "base_model:" not in front
assert "Full model" in md # no adapter artifact -> full model
def test_scorecard_drops_none_fields(self):
"""M4 — rows with a None benchmark/score must be dropped, not rendered."""
from soup_cli.commands.card import build_model_card
entry = {
"id": "r", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": None, "config_hash": "", "data_hash": "",
"run_id": "run1", "tags": [], "config_json": "{}",
}
evals = [
{"benchmark": None, "score": 0.5},
{"benchmark": "gsm8k", "score": None},
{"benchmark": "mmlu", "score": 0.42},
]
md = build_model_card(entry, [], evals, [])
assert "mmlu" in md
assert "gsm8k" not in md
def test_scorecard_duplicate_benchmark_last_wins(self):
from soup_cli.commands.card import build_model_card
entry = {
"id": "r", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": None, "config_hash": "", "data_hash": "",
"run_id": "run1", "tags": [], "config_json": "{}",
}
evals = [{"benchmark": "mmlu", "score": 0.1}, {"benchmark": "mmlu", "score": 0.9}]
md = build_model_card(entry, [], evals, [])
assert "0.900" in md
assert "0.100" not in md
def test_malformed_config_json_does_not_crash(self):
from soup_cli.commands.card import build_model_card
entry = {
"id": "r", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": None, "config_hash": "", "data_hash": "",
"run_id": None, "tags": [], "config_json": "not json{{",
}
md = build_model_card(entry, [], [], [])
assert "# m" in md
def test_yaml_dq_unit(self):
"""L5 — direct unit coverage of the frontmatter escaper."""
from soup_cli.commands.card import _yaml_dq
assert _yaml_dq("plain") == '"plain"'
assert _yaml_dq('a"b') == '"a\\"b"'
assert _yaml_dq("a\\b") == '"a\\\\b"'
assert "\n" not in _yaml_dq("a\nb")
assert "\x1b" not in _yaml_dq("a\x1bb")
def test_truncate_boundary(self):
"""L6 — exactly at the limit must NOT truncate."""
from soup_cli.commands.card import _truncate
assert _truncate("abcde", 5) == "abcde"
assert "[truncated]" in _truncate("abcdef", 5)
class TestCiWorkflowEdges:
def test_write_rejects_non_bool_overwrite(self):
from soup_cli.utils.ci_workflow import write_soup_gate_workflow
with pytest.raises(TypeError):
write_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
overwrite="yes",
)
@pytest.mark.parametrize("bad_out", ["", None, True])
def test_write_rejects_bad_output_path(self, bad_out):
from soup_cli.utils.ci_workflow import write_soup_gate_workflow
with pytest.raises((ValueError, TypeError)):
write_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
output_path=bad_out,
)
def test_size_cap_fires(self, tmp_path, monkeypatch):
"""L4 — the 64 KiB cap is unreachable via normal input (path len is
capped at 4096); pin that the guard still fires when tripped."""
from soup_cli.utils import ci_workflow
monkeypatch.chdir(tmp_path)
monkeypatch.setattr(ci_workflow, "_MAX_FILE_BYTES", 10)
with pytest.raises(ValueError, match="cap"):
ci_workflow.write_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
)
@pytest.mark.skipif(os.name != "posix", reason="POSIX symlink semantics")
def test_write_rejects_symlink_output(self, tmp_path, monkeypatch):
"""L1 — live symlink coverage at ci_workflow's own call site."""
from soup_cli.utils.ci_workflow import write_soup_gate_workflow
monkeypatch.chdir(tmp_path)
target = tmp_path / "real.yml"
target.write_text("x", encoding="utf-8")
link = tmp_path / "link.yml"
link.symlink_to(target)
with pytest.raises(ValueError):
write_soup_gate_workflow(
data_path="d.jsonl", suite_path="s.yaml", evidence_path="e.json",
output_path="link.yml", overwrite=True,
)
def test_ci_init_bad_python_names_the_problem(self, tmp_path, monkeypatch):
"""L3 — assert the message, not just the exit code."""
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
result = runner.invoke(app, ["ci", "init", "--python", "3.x"])
assert result.exit_code == 1
assert "python_version" in result.output or "3.11" in result.output
class TestAdapterInference:
"""Step-6 smoke found this: a REAL LoRA run with no artifacts attached
rendered as `Type | Full model` + `library_name: transformers` — a false
claim in a provenance document and a broken HF Hub card."""
def _entry(self, config):
return {
"id": "r", "name": "m", "base_model": "b", "task": "sft",
"created_at": "t", "notes": None, "config_hash": "", "data_hash": "",
"run_id": None, "tags": [], "config_json": json.dumps(config),
}
def test_lora_config_without_artifacts_is_adapter(self):
from soup_cli.commands.card import build_model_card
md = build_model_card(
self._entry({"training": {"lora": {"r": 4, "alpha": 8}}}), [], [], []
)
assert "LoRA adapter" in md
assert "library_name: peft" in md
assert "Full model" not in md
def test_no_lora_is_full_model(self):
from soup_cli.commands.card import build_model_card
md = build_model_card(self._entry({"training": {"epochs": 1}}), [], [], [])
assert "Full model" in md
assert "library_name: transformers" in md
def test_zero_rank_lora_is_full_model(self):
from soup_cli.commands.card import build_model_card
md = build_model_card(self._entry({"training": {"lora": {"r": 0}}}), [], [], [])
assert "Full model" in md
def test_adapter_artifact_is_definitive(self):
from soup_cli.commands.card import build_model_card
md = build_model_card(
self._entry({"training": {}}),
[{"kind": "adapter", "path": "a", "sha256": "x"}], [], [],
)
assert "LoRA adapter" in md
def test_dense_export_artifact_beats_lora_config(self):
"""A merged/gguf export is a standalone model even if trained via LoRA."""
from soup_cli.commands.card import build_model_card
md = build_model_card(
self._entry({"training": {"lora": {"r": 8}}}),
[{"kind": "merged", "path": "m", "sha256": "x"}], [], [],
)
assert "Full model" in md
def test_spectrum_full_ft_is_not_adapter(self):
"""unfrozen_parameters = full FT; the dumped config still carries a
default lora block, which must NOT be read as 'adapter'."""
from soup_cli.commands.card import build_model_card
md = build_model_card(
self._entry({"training": {"unfrozen_parameters": ["q_proj"],
"lora": {"r": 8}}}), [], [], [],
)
assert "Full model" in md
def test_lisa_full_ft_is_not_adapter(self):
from soup_cli.commands.card import build_model_card
md = build_model_card(
self._entry({"training": {"lisa_enabled": True, "lora": {"r": 8}}}), [], [], [],
)
assert "Full model" in md
def test_malformed_lora_block_does_not_crash(self):
from soup_cli.commands.card import build_model_card
md = build_model_card(self._entry({"training": {"lora": "nonsense"}}), [], [], [])
assert "Full model" in md
md2 = build_model_card(self._entry({"training": {"lora": {"r": "abc"}}}), [], [], [])
assert "Full model" in md2
# --------------------------------------------------------------------------- #
# GGUF-on-Windows validation fixes (#70/#144) — both found by the live build
# --------------------------------------------------------------------------- #
class TestGgufWindowsFixes:
def test_finds_msvc_multi_config_binary(self, tmp_path):
"""MSVC/Xcode are multi-config generators: the binary lands in
build/bin/Release/, not the flat build/bin/ a Make/Ninja build makes.
Without this, `soup export --format gguf` cannot find a correctly-built
llama.cpp on Windows."""
from soup_cli.commands.export import _find_quantize_binary
llama = tmp_path / "llama.cpp"
rel = llama / "build" / "bin" / "Release"
rel.mkdir(parents=True)
binary = rel / "llama-quantize.exe"
binary.write_text("stub", encoding="utf-8")
found = _find_quantize_binary(llama)
assert found is not None
assert found.name == "llama-quantize.exe"
assert "Release" in str(found)
def test_flat_single_config_layout_still_found(self, tmp_path):
"""Make/Ninja layout must keep working."""
from soup_cli.commands.export import _find_quantize_binary
llama = tmp_path / "llama.cpp"
bindir = llama / "build" / "bin"
bindir.mkdir(parents=True)
(bindir / "llama-quantize").write_text("stub", encoding="utf-8")
assert _find_quantize_binary(llama) is not None
def test_missing_binary_returns_none(self, tmp_path, monkeypatch):
from soup_cli.commands import export as export_mod
monkeypatch.setattr(export_mod.shutil, "which", lambda _: None)
llama = tmp_path / "llama.cpp"
(llama / "build").mkdir(parents=True)
assert export_mod._find_quantize_binary(llama) is None
def test_convert_deps_never_install_llama_requirements(self):
"""llama.cpp's requirements.txt pins torch~=2.2.1 against the CPU wheel
index; installing it downgrades a user's CUDA torch to CPU-only and
breaks training (observed live: 2.5.1+cu -> 2.2.2+cpu). Soup must only
install the convert script's EXTRA deps, unpinned."""
import inspect
from soup_cli.commands import export as export_mod
src = inspect.getsource(export_mod._find_llama_cpp)
assert "requirements.txt" not in src
assert "torch" not in export_mod._CONVERT_EXTRA_DEPS
assert "transformers" not in export_mod._CONVERT_EXTRA_DEPS
assert "gguf" in export_mod._CONVERT_EXTRA_DEPS
def test_install_convert_deps_is_unpinned_and_targeted(self, monkeypatch):
from soup_cli.commands import export as export_mod
calls = {}
def _fake_run(cmd, **kwargs):
calls["cmd"] = cmd
class R:
returncode = 0
return R()
monkeypatch.setattr(export_mod.subprocess, "run", _fake_run)
export_mod._install_convert_deps()
cmd = calls["cmd"]
assert "-r" not in cmd # never a requirements file
assert all("==" not in tok and "~=" not in tok for tok in cmd) # unpinned
assert "gguf" in cmd
def test_install_convert_deps_failure_is_non_fatal(self, monkeypatch):
"""A pip failure must warn, not abort the export."""
import subprocess as sp
from soup_cli.commands import export as export_mod
def _boom(cmd, **kwargs):
raise sp.CalledProcessError(1, cmd, stderr="network down")
monkeypatch.setattr(export_mod.subprocess, "run", _boom)
export_mod._install_convert_deps() # must not raise
class TestOllamaModelfileAbsolutePath:
"""`ollama create` resolves a relative FROM against the Modelfile's own
directory, and Soup writes the Modelfile to a temp dir — so a relative GGUF
path made Ollama try to PULL it as a remote model:
"pull model manifest: file does not exist". Found live in the v0.71.35
GGUF-on-Windows validation."""
def test_modelfile_from_is_absolute(self, tmp_path, monkeypatch):
from soup_cli.utils.ollama import create_modelfile
monkeypatch.chdir(tmp_path)
gguf = tmp_path / "m.q8_0.gguf"
gguf.write_text("stub", encoding="utf-8")
body = create_modelfile("m.q8_0.gguf") # relative input
from_line = body.splitlines()[0]
assert from_line.startswith("FROM ")
emitted = from_line[len("FROM "):]
assert os.path.isabs(emitted), emitted
assert emitted.endswith("m.q8_0.gguf")
def test_absolute_input_stays_absolute(self, tmp_path):
from soup_cli.utils.ollama import create_modelfile
gguf = tmp_path / "m.gguf"
gguf.write_text("stub", encoding="utf-8")
body = create_modelfile(str(gguf))
assert os.path.isabs(body.splitlines()[0][len("FROM "):])
def test_other_directives_still_render(self, tmp_path, monkeypatch):
from soup_cli.utils.ollama import create_modelfile
monkeypatch.chdir(tmp_path)
body = create_modelfile("m.gguf", template="chatml", system_prompt="be nice")
assert "TEMPLATE" in body
assert 'SYSTEM "be nice"' in body
class TestLlamaCppHomeAnchor:
"""GGUF bug #1: SOUP_DIR is the bare name ".soup", so using it relatively
made the lookup cwd-dependent — ~/.soup/llama.cpp was never found and the
auto-clone dropped a fresh ~200 MB checkout into whatever directory the
user ran from. Must anchor to home like tracker.py / registry/store.py."""
def test_finds_llama_cpp_under_home_not_cwd(self, tmp_path, monkeypatch):
from pathlib import Path
from soup_cli.commands import export as export_mod
fake_home = tmp_path / "home"
llama = fake_home / ".soup" / "llama.cpp"
llama.mkdir(parents=True)
(llama / "convert_hf_to_gguf.py").write_text("# stub", encoding="utf-8")
workdir = tmp_path / "someproject"
workdir.mkdir()
monkeypatch.chdir(workdir)
monkeypatch.setattr(Path, "home", classmethod(lambda cls: fake_home))
monkeypatch.delenv("LLAMA_CPP_PATH", raising=False)
# must resolve to the HOME copy without cloning
def _no_clone(*a, **k):
raise AssertionError("must not clone: the home copy already exists")
monkeypatch.setattr(export_mod.subprocess, "run", _no_clone)
found = export_mod._find_llama_cpp(None)
assert Path(found).resolve() == llama.resolve()
# and no stray .soup litter in the working directory
assert not (workdir / ".soup").exists()