mirror of https://github.com/razor-ai/soup.git
feat(adapters,loop): v0.71.4 — live canary verdict + cmaes merge + PR push + pre-wired loop + can lineage + branch↔registry
Closes #172, #173, #176, #177, #220, #223. - #172 soup adapters merge --canary/--strict-verdict: live OK/MINOR/MAJOR verdict (was UNKNOWN stub) - #220 soup adapters merge --strategy cmaes: live merge→score→write-best loop (was plan-only) - #223 soup adapters pr --push owner/repo#N: post PR comment via gh api - #176 soup loop --pre-wired: real traces→DPO→eval-gate→canary stages - #177 soup loop --pack-cans / replay --extract: iterations as Soup Cans + Registry lineage DAG - #173 soup adapters branch --from-registry / --attach-to-registry Security: backdoor-scan + license gates now run for ALL merge strategies (incl cmaes); loop canary deploy restricted to loopback/RFC1918; gh child env from allowlist; canary read uses O_NOFOLLOW+fstat (TOCTOU); pack-entry failure rolls back registry entry. Tests: 12342 → 12474 (+130 in tests/test_v0714.py).
This commit is contained in:
Alpamys
parent
22d5c4f226
commit
5652215d4a
45
CHANGELOG.md
45
CHANGELOG.md
|
|
@ -12,6 +12,51 @@ reproducing 70+ versions of notes.
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.71.4] - 2026-06-02
|
||||
|
||||
### Added
|
||||
- **Live canary verdict for `soup adapters merge`** — `--canary <suite.json>`
|
||||
scores the merged adapter against the first input and classifies
|
||||
**OK / MINOR / MAJOR** using the Quant-Lobotomy taxonomy (drop <2% OK, <5%
|
||||
MINOR, else MAJOR). `--strict-verdict` exits 2 on MAJOR. Pre-scored
|
||||
`{"baseline_scores","candidate_scores"}` suites run with no model load; a
|
||||
`{"tasks":[...]}` suite uses an injectable scorer. Replaces the v0.57 `UNKNOWN`
|
||||
stub.
|
||||
- **Live evolutionary merge** — `soup adapters merge --strategy cmaes --eval
|
||||
<suite> --budget <t>` now runs the full CMA-ES loop: each candidate is merged,
|
||||
materialised, scored against the eval suite, and the best-weighted merge is
|
||||
written to `--output`. Replaces the v0.67 plan-only stub.
|
||||
- **Publish an adapter PR to GitHub** — `soup adapters pr <title> --base-sha
|
||||
<hex> --adapter <path> --push owner/repo#N` posts the rendered PR Markdown as a
|
||||
GitHub PR comment via `gh api` (argv-list, body over JSON stdin; no shell).
|
||||
Token resolves from `GITHUB_TOKEN` / `GH_TOKEN`.
|
||||
- **Pre-wired `soup loop` production stages** — `soup loop init --pre-wired` (or
|
||||
`soup loop watch --pre-wired`) swaps the v0.58 no-op stage stubs for real
|
||||
harvest (traces → preference pairs) → DPO train → eval-gate → canary-deploy
|
||||
callables. `soup loop status` now shows the `pre_wired` flag.
|
||||
- **Loop iterations as Soup Cans + Registry lineage** — `soup loop watch
|
||||
--pack-cans` packs each successful iteration as a v0.26 Soup Can and appends a
|
||||
Registry entry (tag `loop-iter`), chaining a real lineage DAG across
|
||||
iterations visible through `soup history`. `soup loop replay <id> --extract
|
||||
<dir>` unpacks a recorded iteration.
|
||||
- **Branch pointers into the Registry** — `soup adapters branch <name>
|
||||
--attach-to-registry <id>` links a branch snapshot to a Registry entry (shown
|
||||
as a `branches` node in `soup history`); `soup adapters branch <name>
|
||||
--from-registry <id>` derives a fresh snapshot's config + base from an entry.
|
||||
|
||||
### Security
|
||||
- The backdoor-scan gate (v0.71.2 #192) and license-conflict gate (v0.60 Part E)
|
||||
now run for **all** merge strategies, including `--strategy cmaes` (previously
|
||||
bypassed because cmaes returned before the gates).
|
||||
- `soup loop` canary deploy restricts `SOUP_LOOP_SERVE_ENDPOINT` to loopback /
|
||||
RFC1918-private hosts (a serve endpoint is the operator's own box/LAN), beyond
|
||||
the general webhook SSRF policy which permits any HTTPS host.
|
||||
- `soup adapters pr --push` builds the `gh` child environment from an allowlist
|
||||
so unrelated secrets (`HF_TOKEN` / `OPENAI_API_KEY` / …) never reach the
|
||||
subprocess.
|
||||
- The canary-suite JSON read uses `O_NOFOLLOW` + `os.fstat` (size cap enforced on
|
||||
the same fd) to close the symlink/size-cap TOCTOU window.
|
||||
|
||||
## [0.71.3] - 2026-06-01
|
||||
|
||||
### Added
|
||||
|
|
|
|||
|
|
@ -120,7 +120,7 @@ src/soup_cli/
|
|||
templates/ - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
|
||||
ui/ - Web UI (FastAPI + HTML/JS SPA)
|
||||
|
||||
tests/ - Test suite (273 files, 12342 tests)
|
||||
tests/ - Test suite (274 files, 12474 tests)
|
||||
examples/ - Real-world config examples and datasets
|
||||
```
|
||||
|
||||
|
|
|
|||
26
README.md
26
README.md
|
|
@ -49,19 +49,21 @@ infrastructure instead of improving models. Soup fixes that.
|
|||
|
||||
## What's New
|
||||
|
||||
**v0.71.3 — Compliance / annex / audit / energy.** The EU AI Act + carbon + audit surfaces go live:
|
||||
**v0.71.4 — Adapter lifecycle + loop wiring.** The merge, PR, and continuous-loop surfaces go live:
|
||||
|
||||
- **Energy & CO2 measurement** — `soup train --track-energy` measures the training window with an
|
||||
offline carbon tracker (no network) and reports kWh / CO2, feeding them into the Annex doc.
|
||||
Install with `pip install soup-cli[carbon]`.
|
||||
- **PDF compliance docs** — `soup train --annex-xi report.pdf` now renders a real PDF of the
|
||||
EU AI Act Annex XI/XII (a `.md` path still gives markdown). The top crawled domains of your
|
||||
training corpus are auto-listed. `pip install soup-cli[pdf]`.
|
||||
- **Per-command audit log** — every `soup` command appends one HIPAA/SOC2 record to
|
||||
`~/.soup/audit.jsonl` (secrets redacted). Opt out with `--no-audit-log` or `SOUP_NO_AUDIT_LOG=1`.
|
||||
- **Signed Soup Cans + airgap receipts** — `soup can pack --attest statement.json` embeds in-toto
|
||||
attestations into a can; `soup airgap-bundle --repro-receipt receipt.json` ships a reproducibility
|
||||
receipt inside the offline bundle.
|
||||
- **Canary verdict on merge** — `soup adapters merge … --canary suite.json` scores the merged
|
||||
adapter and reports **OK / MINOR / MAJOR**; `--strict-verdict` exits non-zero on a MAJOR
|
||||
regression. Works with no model load using a pre-scored canary suite.
|
||||
- **Evolutionary merge for real** — `soup adapters merge --strategy cmaes --eval suite --budget 1h`
|
||||
now runs the full CMA-ES search (merge → score → optimise) and writes the best blend, instead of
|
||||
just printing a plan.
|
||||
- **Publish an adapter PR** — `soup adapters pr <title> --base-sha <hex> --adapter <path> --push
|
||||
owner/repo#42` posts the rendered PR straight to a GitHub PR comment.
|
||||
- **Continuous fine-tuning loop, wired up** — `soup loop watch --pre-wired` runs the real
|
||||
traces → DPO → eval-gate → canary pipeline; `--pack-cans` snapshots every iteration as a
|
||||
shareable Soup Can with Registry lineage (`soup loop replay <id> --extract dir`).
|
||||
- **Branches ↔ Registry** — `soup adapters branch <name> --attach-to-registry <id>` /
|
||||
`--from-registry <id>` links training-env snapshots into the Registry lineage DAG.
|
||||
|
||||
Full history: [CHANGELOG.md](CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
|
||||
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ v0.57 shipped `adapters diff / merge / blame / branch`. v0.67 finishes the lifec
|
|||
|
||||
```bash
|
||||
# 1. Evolutionary merge: search the simplex of merge weights via CMA-ES.
|
||||
# v0.71.4 makes this a LIVE loop — each candidate is merged, scored against
|
||||
# --eval, and the best blend is written to --output.
|
||||
soup adapters merge \
|
||||
adapter-finance/ adapter-medical/ adapter-legal/ \
|
||||
--strategy cmaes \
|
||||
|
|
@ -43,6 +45,12 @@ soup adapters merge \
|
|||
--max-generations 20 \
|
||||
--output merged/
|
||||
|
||||
# 1b. One-shot merge with a live canary verdict (v0.71.4). Scores the merged
|
||||
# adapter vs the first input → OK / MINOR / MAJOR; --strict-verdict exits 2
|
||||
# on a MAJOR regression. A pre-scored canary suite needs no model load.
|
||||
soup adapters merge adapter-a/ adapter-b/ -o merged/ \
|
||||
--canary evals/canary.json --strict-verdict
|
||||
|
||||
# 2. Render the merge as a GitHub PR for review (eval deltas + sample diffs).
|
||||
soup adapters pr "merge: 3-domain blend" \
|
||||
--base-sha $(git rev-parse HEAD) \
|
||||
|
|
@ -52,6 +60,12 @@ soup adapters pr "merge: 3-domain blend" \
|
|||
--dataset-diff data/diff.txt \
|
||||
--format markdown -o pr.md
|
||||
|
||||
# 2b. Post it straight to a GitHub PR comment (v0.71.4). Auth via GITHUB_TOKEN
|
||||
# / GH_TOKEN; uses `gh api` under the hood (no secret leaks to the child).
|
||||
soup adapters pr "merge: 3-domain blend" \
|
||||
--base-sha $(git rev-parse HEAD) --adapter merged/ \
|
||||
--eval evals/deltas.json --push your-org/your-repo#42
|
||||
|
||||
# 3. Lock a reproducible run state. Closure = sha(base + dataset + env).
|
||||
soup env lock # v0.64 — capture env hash
|
||||
soup lock write \
|
||||
|
|
@ -93,6 +107,16 @@ soup loop status
|
|||
# Run the daemon (foreground)
|
||||
soup loop watch --poll-interval 300
|
||||
|
||||
# Pre-wired production stages (v0.71.4): real traces → DPO → eval-gate → canary,
|
||||
# instead of the v0.58 no-op stubs. Opt in once via `loop init --pre-wired` or
|
||||
# per-run via --pre-wired. Pack each iteration as a shareable Soup Can with
|
||||
# Registry lineage via --pack-cans.
|
||||
soup loop init registry://abc12 --eval evals/lock.json --baseline registry://prod --pre-wired
|
||||
soup loop watch --pre-wired --pack-cans --poll-interval 300
|
||||
|
||||
# Unpack a recorded iteration's Soup Can for what-if analysis (v0.71.4)
|
||||
soup loop replay iter-20260515T120000-abcdef01 --extract ./iter-dump
|
||||
|
||||
# Background subprocess (writes PID, no shell)
|
||||
soup loop watch --detach
|
||||
|
||||
|
|
@ -240,6 +264,14 @@ soup adapters blame ./run-v18 --dataset train.jsonl --layer q_proj.7 \
|
|||
soup adapters branch v18 --config soup.yaml --base meta-llama/Llama-3.1-8B \
|
||||
--dataset train.jsonl
|
||||
|
||||
# Link a branch into the Registry lineage DAG (v0.71.4) — shows as a
|
||||
# `branches` node under the entry in `soup history`.
|
||||
soup adapters branch v18 --config soup.yaml --base meta-llama/Llama-3.1-8B \
|
||||
--attach-to-registry reg_20260601_abc123
|
||||
|
||||
# Or derive a fresh snapshot's config + base straight from a Registry entry (v0.71.4)
|
||||
soup adapters branch v18-from-reg --from-registry reg_20260601_abc123
|
||||
|
||||
# Restore the snapshot's config (refuses if source SHA drifted)
|
||||
soup adapters checkout v18 --output soup.yaml
|
||||
|
||||
|
|
@ -264,7 +296,9 @@ soup adapters branches
|
|||
- Branch pointers live under `~/.soup/branches/` (override via `SOUP_BRANCHES_DIR`, constrained to `$HOME` / `$CWD` / `$TMPDIR`).
|
||||
- `soup adapters checkout` SHA-checks the source config — refuses to restore when the source has drifted from the snapshot, so reproducibility never silently lies.
|
||||
|
||||
**v0.66.0:** `soup adapters blame` is now LIVE — the v0.57 `NotImplementedError` stub (#171) is lifted via a DataInf-style influence-function approximation. Pass `--top-k 50` to control the reported top-influencer count; pass a real `probe_fn` (Python API) to feed real gradients, or use the default deterministic synthetic probe for offline planning. `MergeReport.verdict` remains the `UNKNOWN` stub (live canary eval in v0.57.1).
|
||||
**v0.66.0:** `soup adapters blame` is now LIVE — the v0.57 `NotImplementedError` stub (#171) is lifted via a DataInf-style influence-function approximation. Pass `--top-k 50` to control the reported top-influencer count; pass a real `probe_fn` (Python API) to feed real gradients, or use the default deterministic synthetic probe for offline planning.
|
||||
|
||||
**v0.71.4:** the merge verdict is now LIVE — `soup adapters merge … --canary suite.json` lifts the `MergeReport.verdict` `UNKNOWN` stub. A pre-scored `{"baseline_scores","candidate_scores"}` suite classifies the blend OK / MINOR / MAJOR with no model load; a `{"tasks":[...]}` suite uses an injectable scorer. `--strict-verdict` exits 2 on MAJOR. The backdoor-scan and license-conflict gates now run for **every** strategy, including `cmaes`. `soup adapters branch <name> --from-registry <id>` / `--attach-to-registry <id>` link training-env snapshots into the Registry lineage DAG (shown as a `branches` node in `soup history`).
|
||||
|
||||
|
||||
## Soup Cans (Shareable Recipes)
|
||||
|
|
|
|||
|
|
@ -109,12 +109,12 @@ soup cost --config soup.yaml --gpu H100 Estimate training cost for specific
|
|||
soup adapters list ./output/ Scan for LoRA adapters
|
||||
soup adapters info ./output/checkpoint-500/ Show adapter metadata
|
||||
soup adapters compare adapter1/ adapter2/ Compare two adapters
|
||||
soup loop init <model> --eval <s> --baseline <b> Create .soup/loop.yaml (data flywheel)
|
||||
soup loop status Counters + status (traces / pairs / runs / shipped)
|
||||
soup loop watch [--detach] [--max-iter N] Harvest → train → gate → deploy daemon
|
||||
soup loop init <model> --eval <s> --baseline <b> [--pre-wired] Create .soup/loop.yaml (data flywheel; --pre-wired = real stages)
|
||||
soup loop status Counters + status + pre_wired flag
|
||||
soup loop watch [--detach] [--max-iter N] [--pre-wired] [--pack-cans] Harvest → train → gate → deploy daemon (pre-wired stages + Soup Can packing)
|
||||
soup loop pause / soup loop resume Atomic status flip
|
||||
soup loop canary <adapter> --traffic 5% Promote canary + auto-rollback on MAJOR
|
||||
soup loop replay [<iter-id>] Replay a recorded iteration manifest
|
||||
soup loop replay [<iter-id>] [--extract <dir>] Replay / unpack a recorded iteration manifest
|
||||
soup serve --model m --adapters chat=./c code=./d Multi-adapter serving
|
||||
soup migrate --from llamafactory config.yaml Import config from LLaMA-Factory
|
||||
soup migrate --from axolotl config.yml Import config from Axolotl
|
||||
|
|
@ -188,8 +188,11 @@ soup probe interference <losses.json> [--output o.json] Pairwise N×N adapter i
|
|||
soup probe pack <base> [--output o.json] Per-base calibrated probe pack manifest (v0.66.0)
|
||||
soup probe pack --list List bundled probe-pack bases (v0.66.0)
|
||||
soup adapters blame ... --top-k 50 Live DataInf-style influence runner (v0.66.0, closes #171)
|
||||
soup adapters merge ... --strategy cmaes --eval <s> --budget 1h CMA-ES evolutionary merge (v0.67.0)
|
||||
soup adapters merge ... --strategy cmaes --eval <s> --budget 1h CMA-ES evolutionary merge — live loop (v0.67.0 schema / v0.71.4 live)
|
||||
soup adapters merge ... --canary <suite.json> [--strict-verdict] Live OK/MINOR/MAJOR canary verdict, exit 2 on MAJOR (v0.71.4)
|
||||
soup adapters pr <title> --base-sha <hex> --adapter <path> GitHub-shaped adapter PR Markdown / JSON (v0.67.0)
|
||||
soup adapters pr <title> ... --push owner/repo#N Post the PR as a GitHub comment via gh api (v0.71.4)
|
||||
soup adapters branch <name> --from-registry <id> | --attach-to-registry <id> Branch ↔ Registry lineage (v0.71.4)
|
||||
soup adapters bisect <ckpt>... --eval-command "..." Binary search over training history (v0.67.0)
|
||||
soup lock write --base-sha <h> --dataset-sha <h> --env-hash <h> Write soup.lock (v0.67.0)
|
||||
soup lock write --base-sha <h> --dataset-sha <h> --env-lock soup-env.lock Auto-derive --env-hash from soup-env.lock (v0.71.1)
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "soup-cli"
|
||||
version = "0.71.3"
|
||||
version = "0.71.4"
|
||||
description = "Fine-tune LLMs in one command. No SSH, no config hell."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
"""Soup CLI — Fine-tune LLMs in one command."""
|
||||
|
||||
__version__ = "0.71.3"
|
||||
__version__ = "0.71.4"
|
||||
|
|
|
|||
|
|
@ -382,6 +382,18 @@ def merge(
|
|||
"that cannot be scanned)."
|
||||
),
|
||||
),
|
||||
canary: Optional[str] = typer.Option(
|
||||
None, "--canary",
|
||||
help=(
|
||||
"Canary-suite JSON to compute a live OK/MINOR/MAJOR verdict for "
|
||||
"the merged adapter vs the first input (v0.71.4 #172). Shape: "
|
||||
'{"baseline_scores": [...], "candidate_scores": [...]}.'
|
||||
),
|
||||
),
|
||||
strict_verdict: bool = typer.Option(
|
||||
False, "--strict-verdict",
|
||||
help="Exit 2 when the canary verdict is MAJOR (CI gate).",
|
||||
),
|
||||
):
|
||||
"""Merge LoRA adapters via linear / ties / dare / svd (v0.57.0, v0.60.0 license gate)."""
|
||||
from soup_cli.utils.adapter_merge import SUPPORTED_STRATEGIES, merge_adapters
|
||||
|
|
@ -403,11 +415,11 @@ def merge(
|
|||
console.print("[red]Need at least 2 adapter paths to merge[/]")
|
||||
raise typer.Exit(2)
|
||||
|
||||
# v0.67.0 Part A: cmaes evolutionary search requires an eval suite +
|
||||
# budget. Render the planned strategy and emit a deferred-live advisory.
|
||||
# Operator-supplied eval_fn lives in `soup_cli.utils.cmaes_merge.run_cmaes_merge`;
|
||||
# the CLI surface here ships the plan + validation. Live `soup eval`
|
||||
# auto-wiring against a registry run is deferred to v0.67.1.
|
||||
# cmaes argument validation runs FIRST (before any file I/O / scanning):
|
||||
# a user who forgot --eval should get that error immediately, not a scan
|
||||
# error on their inputs. The cmaes *search* itself still runs after the
|
||||
# security gates below (v0.71.4 review — preserves the "reject missing
|
||||
# --eval before file-handling" contract while gating the merge).
|
||||
if strategy == "cmaes":
|
||||
if eval_suite is None:
|
||||
console.print(
|
||||
|
|
@ -415,41 +427,18 @@ def merge(
|
|||
"(path to a YAML/JSONL eval).[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
from soup_cli.utils.cmaes_merge import build_cmaes_plan
|
||||
|
||||
try:
|
||||
cmaes_plan = build_cmaes_plan(
|
||||
adapters=adapters,
|
||||
eval_suite=eval_suite,
|
||||
budget_spec=budget,
|
||||
population_size=population,
|
||||
max_generations=max_generations,
|
||||
seed=seed,
|
||||
if canary is not None or strict_verdict:
|
||||
console.print(
|
||||
"[red]--canary / --strict-verdict are not supported with "
|
||||
"--strategy cmaes; the --eval suite already drives the search.[/]"
|
||||
)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid cmaes plan: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
panel = Panel(
|
||||
f"Strategy: [bold]cmaes[/]\n"
|
||||
f"Adapters: {len(cmaes_plan.adapters)}\n"
|
||||
f"Eval suite: [bold]{escape(os.path.basename(cmaes_plan.eval_suite))}[/]\n"
|
||||
f"Budget: [bold]{cmaes_plan.budget_seconds}s[/]\n"
|
||||
f"Population: [bold]{cmaes_plan.population_size}[/]\n"
|
||||
f"Max generations: [bold]{cmaes_plan.max_generations}[/]\n"
|
||||
f"Live cmaes loop with auto-wired eval suite is deferred to "
|
||||
f"[yellow]v0.67.1[/]. The plan is validated and ready to "
|
||||
f"hand to an operator-supplied `eval_fn` via "
|
||||
f"`run_cmaes_merge(plan, eval_fn=...)`.",
|
||||
title="Adapter merge — cmaes (plan)",
|
||||
)
|
||||
console.print(panel)
|
||||
return
|
||||
raise typer.Exit(2)
|
||||
|
||||
# v0.71.2 #192 — backdoor-scan gate. Refuse to merge any input whose
|
||||
# spectral scan returns FAIL (or that cannot be scanned at all) unless the
|
||||
# operator passes --allow-unscanned. WARN is advisory-only. Runs AFTER the
|
||||
# cmaes plan-only branch (which doesn't merge weights) so it gates only the
|
||||
# strategies that actually write a merged adapter.
|
||||
# operator passes --allow-unscanned. WARN is advisory-only. Runs BEFORE the
|
||||
# strategy dispatch (incl. cmaes, which v0.71.4 #220 made weight-writing) so
|
||||
# EVERY strategy that produces a merged adapter is gated uniformly.
|
||||
if not allow_unscanned:
|
||||
from soup_cli.utils.adapter_scan import scan_adapter
|
||||
|
||||
|
|
@ -481,7 +470,8 @@ def merge(
|
|||
# per adapter (--license, repeatable) OR rely on v0.71.2 #187
|
||||
# auto-extraction from adapter_config.json / config.json / model-card
|
||||
# frontmatter. On a conflict, --license-override <reason> proceeds and the
|
||||
# decision is recorded to the audit log (#190).
|
||||
# decision is recorded to the audit log (#190). Runs before the strategy
|
||||
# dispatch so cmaes (#220, weight-writing) is gated like every other merge.
|
||||
gate_licenses: Optional[list[str]] = None
|
||||
if license_ids:
|
||||
if len(license_ids) != len(adapters):
|
||||
|
|
@ -539,6 +529,81 @@ def merge(
|
|||
f"[dim]Reason: {escape(cleared)} (recorded to audit log)[/]"
|
||||
)
|
||||
|
||||
# v0.67.0 Part A: cmaes evolutionary search requires an eval suite +
|
||||
# budget. v0.71.4 #220 lifts this from plan-only to a live loop: the eval
|
||||
# suite is auto-wired into an eval_fn that materialises + scores each
|
||||
# candidate merge, and the best-weighted merge is written to --output.
|
||||
# (Arg validation already happened above, before the security gates.)
|
||||
if strategy == "cmaes":
|
||||
assert eval_suite is not None # narrowed by the early arg-check
|
||||
from soup_cli.utils.cmaes_merge import (
|
||||
build_cmaes_eval_fn,
|
||||
build_cmaes_plan,
|
||||
run_cmaes_merge,
|
||||
)
|
||||
|
||||
try:
|
||||
cmaes_plan = build_cmaes_plan(
|
||||
adapters=adapters,
|
||||
eval_suite=eval_suite,
|
||||
budget_spec=budget,
|
||||
population_size=population,
|
||||
max_generations=max_generations,
|
||||
seed=seed,
|
||||
)
|
||||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Invalid cmaes plan: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
console.print(
|
||||
f"[cyan]Running CMA-ES merge[/] "
|
||||
f"(pop={cmaes_plan.population_size}, "
|
||||
f"max_gen={cmaes_plan.max_generations}, "
|
||||
f"budget={cmaes_plan.budget_seconds}s)..."
|
||||
)
|
||||
console.print(
|
||||
"[dim]The default scorer reloads the base model per candidate "
|
||||
"(pop x generations loads); pass a small --population for large "
|
||||
"models, or wire a cached scorer.[/]"
|
||||
)
|
||||
try:
|
||||
eval_fn = build_cmaes_eval_fn(cmaes_plan)
|
||||
result = run_cmaes_merge(cmaes_plan, eval_fn=eval_fn)
|
||||
except (FileNotFoundError, TypeError, ValueError, RuntimeError, OSError) as exc:
|
||||
console.print(f"[red]cmaes merge failed: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
# Write the best-weighted merge to --output (linear with the winning
|
||||
# simplex weights).
|
||||
try:
|
||||
report = merge_adapters(
|
||||
adapters, output, strategy="linear",
|
||||
weights=list(result.best_weights),
|
||||
)
|
||||
except (FileNotFoundError, TypeError, ValueError, RuntimeError, OSError) as exc:
|
||||
console.print(f"[red]Could not write merged adapter: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
conv_color = "green" if result.converged else "yellow"
|
||||
best_w = ", ".join(f"{w:.3f}" for w in result.best_weights)
|
||||
console.print(
|
||||
Panel(
|
||||
f"Strategy: [bold]cmaes[/]\n"
|
||||
f"Adapters: {len(cmaes_plan.adapters)}\n"
|
||||
f"Eval suite: [bold]{escape(os.path.basename(cmaes_plan.eval_suite))}[/]\n"
|
||||
f"Generations: [bold]{result.generations_run}[/] / "
|
||||
f"{cmaes_plan.max_generations}\n"
|
||||
f"Evaluations: [bold]{result.evaluations}[/]\n"
|
||||
f"Best score: [bold]{result.best_score:.4f}[/]\n"
|
||||
f"Best weights: [dim]{escape(best_w)}[/]\n"
|
||||
f"Converged: [{conv_color}]{result.converged}[/]\n"
|
||||
f"Merged layers: [bold]{report.merged_layers}[/]\n"
|
||||
f"Output: [bold]{escape(report.output_dir)}[/]",
|
||||
title="Adapter merge — cmaes",
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
parsed_weights = None
|
||||
if weights:
|
||||
try:
|
||||
|
|
@ -564,17 +629,40 @@ def merge(
|
|||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
# v0.71.4 #172 — live canary verdict. With --canary, score the merged
|
||||
# adapter vs the first input and classify OK/MINOR/MAJOR; without it the
|
||||
# verdict stays UNKNOWN (advisory).
|
||||
verdict = report.verdict
|
||||
verdict_suffix = " (pass --canary <suite.json> to compute)"
|
||||
if canary is not None:
|
||||
from soup_cli.utils.adapter_merge import predict_merged_verdict
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
try:
|
||||
enforce_under_cwd_and_no_symlink(canary, "canary")
|
||||
verdict = predict_merged_verdict(report, canary)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]Canary verdict failed: {escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
verdict_suffix = ""
|
||||
|
||||
verdict_color = {"OK": "green", "MINOR": "yellow", "MAJOR": "red"}.get(
|
||||
verdict, "yellow"
|
||||
)
|
||||
panel = Panel(
|
||||
f"Strategy: [bold]{escape(report.strategy)}[/]\n"
|
||||
f"Inputs: {len(report.adapters)}\n"
|
||||
f"Merged layers: [bold]{report.merged_layers}[/]\n"
|
||||
f"Skipped layers: {len(report.skipped_layers)}\n"
|
||||
f"Verdict: [yellow]{report.verdict}[/] (live eval in v0.57.1)\n"
|
||||
f"Verdict: [{verdict_color}]{verdict}[/]{verdict_suffix}\n"
|
||||
f"Output: [bold]{escape(report.output_dir)}[/]",
|
||||
title="Adapter merge",
|
||||
)
|
||||
console.print(panel)
|
||||
|
||||
if strict_verdict and verdict == "MAJOR":
|
||||
raise typer.Exit(2)
|
||||
|
||||
|
||||
@app.command()
|
||||
def blame(
|
||||
|
|
@ -672,16 +760,62 @@ def blame(
|
|||
@app.command()
|
||||
def branch(
|
||||
name: str = typer.Argument(..., help="Branch name (alphanumeric + ._-)"),
|
||||
config: str = typer.Option(..., "--config", "-c", help="Path to soup.yaml"),
|
||||
base: str = typer.Option(..., "--base", help="Base model id"),
|
||||
config: Optional[str] = typer.Option(
|
||||
None, "--config", "-c",
|
||||
help="Path to soup.yaml (required unless --from-registry)",
|
||||
),
|
||||
base: Optional[str] = typer.Option(
|
||||
None, "--base", help="Base model id (required unless --from-registry)",
|
||||
),
|
||||
dataset: str = typer.Option(None, "--dataset", help="Training dataset path (optional)"),
|
||||
attach_to_registry: Optional[str] = typer.Option(
|
||||
None, "--attach-to-registry",
|
||||
help=(
|
||||
"Registry entry id/ref to attach this branch pointer to as a "
|
||||
"branch_ref artifact + link via registry_entry_id (v0.71.4)."
|
||||
),
|
||||
),
|
||||
from_registry: Optional[str] = typer.Option(
|
||||
None, "--from-registry",
|
||||
help=(
|
||||
"Derive config + base_model + dataset hash from a Registry entry "
|
||||
"instead of --config/--base (v0.71.4)."
|
||||
),
|
||||
),
|
||||
):
|
||||
"""Snapshot a training environment as a comparable branch (v0.57.0)."""
|
||||
from soup_cli.utils.adapter_branch import create_branch
|
||||
"""Snapshot a training environment as a comparable branch (v0.57.0).
|
||||
|
||||
Pass ``--from-registry <id>`` to derive everything from a v0.26 Registry
|
||||
entry, or ``--attach-to-registry <id>`` to link a fresh snapshot into the
|
||||
lineage DAG (v0.71.4 #173).
|
||||
"""
|
||||
from soup_cli.utils.adapter_branch import (
|
||||
attach_branch_to_registry,
|
||||
branch_from_registry,
|
||||
create_branch,
|
||||
load_branch,
|
||||
)
|
||||
|
||||
try:
|
||||
snap = create_branch(name, config_path=config, base_model=base,
|
||||
dataset_path=dataset)
|
||||
if from_registry is not None:
|
||||
snap = branch_from_registry(name, from_registry)
|
||||
else:
|
||||
if not config:
|
||||
console.print(
|
||||
"[red]--config is required (or use --from-registry)[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
if not base:
|
||||
console.print(
|
||||
"[red]--base is required (or use --from-registry)[/]"
|
||||
)
|
||||
raise typer.Exit(2)
|
||||
snap = create_branch(
|
||||
name, config_path=config, base_model=base, dataset_path=dataset,
|
||||
)
|
||||
if attach_to_registry is not None:
|
||||
attach_branch_to_registry(name, attach_to_registry)
|
||||
snap = load_branch(name)
|
||||
except FileNotFoundError as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
|
|
@ -689,6 +823,11 @@ def branch(
|
|||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
|
||||
reg_line = (
|
||||
f"\nRegistry: [dim]{escape(snap.registry_entry_id)}[/]"
|
||||
if snap.registry_entry_id
|
||||
else ""
|
||||
)
|
||||
console.print(
|
||||
Panel(
|
||||
f"Name: [bold]{escape(snap.name)}[/]\n"
|
||||
|
|
@ -697,7 +836,7 @@ def branch(
|
|||
f"SHA: [dim]{snap.config_sha256[:16]}...[/]\n"
|
||||
f"Dataset SHA: [dim]"
|
||||
f"{snap.dataset_sha256[:16] + '...' if snap.dataset_sha256 else '—'}[/]\n"
|
||||
f"Version: {snap.soup_version}",
|
||||
f"Version: {snap.soup_version}{reg_line}",
|
||||
title="Adapter branch",
|
||||
)
|
||||
)
|
||||
|
|
@ -1020,15 +1159,25 @@ def adapter_pr(
|
|||
format_: str = typer.Option(
|
||||
"markdown", "--format", "-f", help="markdown | json",
|
||||
),
|
||||
push: Optional[str] = typer.Option(
|
||||
None, "--push",
|
||||
help=(
|
||||
"Post the rendered Markdown as a comment on a GitHub PR: "
|
||||
"owner/repo#N (e.g. MakazhanAlpamys/Soup#42). Auth via "
|
||||
"GITHUB_TOKEN / GH_TOKEN (v0.71.4)."
|
||||
),
|
||||
),
|
||||
):
|
||||
"""Render a GitHub-style PR for an adapter (v0.67.0 Part D).
|
||||
|
||||
The PR = ``{base SHA, dataset diff, adapter file, eval report}``
|
||||
rendered as a Markdown document with eval-delta tables and sample
|
||||
diffs, ready to drop into a GitHub PR description or comment.
|
||||
diffs, ready to drop into a GitHub PR description or comment. Pass
|
||||
``--push owner/repo#N`` to post it directly as a PR comment (v0.71.4).
|
||||
"""
|
||||
from soup_cli.utils.adapter_pr import (
|
||||
build_adapter_pr,
|
||||
post_pr_comment,
|
||||
render_pr_json,
|
||||
render_pr_markdown,
|
||||
write_pr_markdown,
|
||||
|
|
@ -1040,11 +1189,16 @@ def adapter_pr(
|
|||
raise typer.Exit(2)
|
||||
|
||||
# Load deltas + samples + dataset_diff lazily; each accepts None.
|
||||
pr_input_cap = 8 * 1024 * 1024 # 8 MiB per --eval/--samples/--dataset-diff
|
||||
|
||||
def _load_json_list(path: Optional[str], field: str) -> list:
|
||||
if path is None:
|
||||
return []
|
||||
enforce_under_cwd_and_no_symlink(path, field=field)
|
||||
with open(os.path.realpath(path), encoding="utf-8") as fh:
|
||||
real = os.path.realpath(path)
|
||||
if os.path.getsize(real) > pr_input_cap:
|
||||
raise ValueError(f"{field} exceeds 8 MiB cap")
|
||||
with open(real, encoding="utf-8") as fh:
|
||||
raw = json.load(fh)
|
||||
if not isinstance(raw, list):
|
||||
raise ValueError(f"{field} must contain a JSON list")
|
||||
|
|
@ -1058,9 +1212,10 @@ def adapter_pr(
|
|||
enforce_under_cwd_and_no_symlink(
|
||||
dataset_diff_path, field="dataset_diff"
|
||||
)
|
||||
with open(
|
||||
os.path.realpath(dataset_diff_path), encoding="utf-8"
|
||||
) as fh:
|
||||
real_diff = os.path.realpath(dataset_diff_path)
|
||||
if os.path.getsize(real_diff) > pr_input_cap:
|
||||
raise ValueError("dataset_diff exceeds 8 MiB cap")
|
||||
with open(real_diff, encoding="utf-8") as fh:
|
||||
dataset_diff = fh.read()
|
||||
pr = build_adapter_pr(
|
||||
title=title,
|
||||
|
|
@ -1079,6 +1234,26 @@ def adapter_pr(
|
|||
else:
|
||||
rendered = render_pr_markdown(pr)
|
||||
|
||||
# v0.71.4 #223 — publish the rendered Markdown as a GitHub PR comment.
|
||||
# Always posts Markdown (a JSON blob is not a useful PR comment), even
|
||||
# when --format json was chosen for the local render/write.
|
||||
if push is not None:
|
||||
try:
|
||||
url = post_pr_comment(push, render_pr_markdown(pr))
|
||||
except RuntimeError as exc:
|
||||
# Missing token / gh failure — user-actionable, exit 1.
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(1) from exc
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]{escape(str(exc))}[/]")
|
||||
raise typer.Exit(2) from exc
|
||||
console.print(
|
||||
f"[green]Posted PR comment to {escape(push)}[/]"
|
||||
+ (f" -> {escape(url)}" if url else "")
|
||||
)
|
||||
if output is None:
|
||||
return
|
||||
|
||||
if output is None:
|
||||
console.print(rendered)
|
||||
return
|
||||
|
|
|
|||
|
|
@ -2,6 +2,8 @@
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
|
|
@ -68,6 +70,18 @@ def history(
|
|||
f"[cyan]{escape(desc['id'])}[/] "
|
||||
f"[yellow]({escape(desc.get('relation', ''))})[/]"
|
||||
)
|
||||
# v0.71.4 #173 — surface attached branch pointers as a distinct
|
||||
# lineage edge so `soup adapters branch --attach-to-registry`
|
||||
# snapshots show up in the DAG view.
|
||||
branch_refs = [
|
||||
art for art in store.get_artifacts(entry["id"])
|
||||
if art.get("kind") == "branch_ref"
|
||||
]
|
||||
if branch_refs:
|
||||
br_node = node.add("[dim]branches[/]")
|
||||
for art in branch_refs:
|
||||
label = os.path.basename(art.get("path", "")) or "branch"
|
||||
br_node.add(f"[green]branch[/] [cyan]{escape(label)}[/]")
|
||||
|
||||
console.print(tree)
|
||||
|
||||
|
|
|
|||
|
|
@ -74,6 +74,13 @@ def init_cmd(
|
|||
max_runs_per_day: Optional[int] = typer.Option(
|
||||
None, "--max-runs-per-day", help="Cap on iteration starts per UTC day."
|
||||
),
|
||||
pre_wired: bool = typer.Option(
|
||||
False, "--pre-wired",
|
||||
help=(
|
||||
"Use the pre-wired harvest/train/gate/deploy stages "
|
||||
"(utils/loop_stages.py) on watch instead of no-op stubs (v0.71.4)."
|
||||
),
|
||||
),
|
||||
force: bool = typer.Option(False, "--force", help="Overwrite existing loop.yaml."),
|
||||
) -> None:
|
||||
"""Create the .soup/loop.yaml control file (one-time setup)."""
|
||||
|
|
@ -96,6 +103,7 @@ def init_cmd(
|
|||
baseline=baseline,
|
||||
monthly_budget_usd=budget_usd,
|
||||
max_runs_per_day=max_runs_per_day,
|
||||
pre_wired=pre_wired,
|
||||
force=force,
|
||||
)
|
||||
except (FileExistsError, FileNotFoundError, TypeError, ValueError) as exc:
|
||||
|
|
@ -123,6 +131,7 @@ def status_cmd() -> None:
|
|||
table.add_row("served_model", escape(state.served_model))
|
||||
table.add_row("eval_suite", escape(state.eval_suite))
|
||||
table.add_row("baseline", escape(state.baseline))
|
||||
table.add_row("pre_wired", "yes" if state.pre_wired else "no")
|
||||
table.add_row("traces_collected", str(state.traces_collected))
|
||||
table.add_row("pairs_distilled", str(state.pairs_distilled))
|
||||
table.add_row("runs_gated", str(state.runs_gated))
|
||||
|
|
@ -191,12 +200,20 @@ def watch_cmd(
|
|||
poll_interval: float = typer.Option(
|
||||
60.0, "--poll-interval", help="Seconds between iterations [1, 3600]."
|
||||
),
|
||||
pre_wired: bool = typer.Option(
|
||||
False, "--pre-wired",
|
||||
help="Force the pre-wired production stages even if loop.yaml didn't set it.",
|
||||
),
|
||||
pack_cans: bool = typer.Option(
|
||||
False, "--pack-cans",
|
||||
help="Pack each iteration as a v0.26 Soup Can + Registry entry (v0.71.4).",
|
||||
),
|
||||
) -> None:
|
||||
"""Run the harvest → train → gate → deploy daemon."""
|
||||
if detach and foreground:
|
||||
console.print("[red]--detach and --foreground are mutually exclusive[/]")
|
||||
raise typer.Exit(code=2)
|
||||
_ = _safe_read() # ensure state exists before forking
|
||||
state = _safe_read() # ensure state exists before forking
|
||||
if detach:
|
||||
argv = [
|
||||
sys.executable,
|
||||
|
|
@ -210,6 +227,10 @@ def watch_cmd(
|
|||
]
|
||||
if max_iterations is not None:
|
||||
argv.extend(["--max-iterations", str(max_iterations)])
|
||||
if pre_wired:
|
||||
argv.append("--pre-wired")
|
||||
if pack_cans:
|
||||
argv.append("--pack-cans")
|
||||
proc = subprocess.Popen( # noqa: S603 — argv is internal, no shell
|
||||
argv,
|
||||
stdout=subprocess.DEVNULL,
|
||||
|
|
@ -218,11 +239,31 @@ def watch_cmd(
|
|||
)
|
||||
console.print(f"[green]watch detached[/] pid={proc.pid}")
|
||||
return
|
||||
use_prewired = pre_wired or state.pre_wired
|
||||
base_model = (
|
||||
state.served_model
|
||||
if state.served_model and not state.served_model.startswith("registry://")
|
||||
else "unknown"
|
||||
)
|
||||
try:
|
||||
cfg = WatchConfig(
|
||||
poll_interval_sec=float(poll_interval),
|
||||
max_iterations=max_iterations,
|
||||
)
|
||||
if use_prewired:
|
||||
from soup_cli.utils.loop_stages import build_prewired_watch_config
|
||||
|
||||
cfg = build_prewired_watch_config(
|
||||
poll_interval_sec=float(poll_interval),
|
||||
max_iterations=max_iterations,
|
||||
pack_iterations=pack_cans,
|
||||
served_model=state.served_model,
|
||||
base_model=base_model,
|
||||
)
|
||||
else:
|
||||
cfg = WatchConfig(
|
||||
poll_interval_sec=float(poll_interval),
|
||||
max_iterations=max_iterations,
|
||||
pack_iterations=pack_cans,
|
||||
served_model=state.served_model,
|
||||
base_model=base_model,
|
||||
)
|
||||
except (TypeError, ValueError) as exc:
|
||||
console.print(f"[red]invalid watch config:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(code=2)
|
||||
|
|
@ -281,6 +322,10 @@ def replay_cmd(
|
|||
iteration_id: Optional[str] = typer.Argument(
|
||||
None, help="Iteration id (omit to list all)."
|
||||
),
|
||||
extract: Optional[str] = typer.Option(
|
||||
None, "--extract",
|
||||
help="Extract the iteration's .can to this directory for a what-if re-run (v0.71.4).",
|
||||
),
|
||||
) -> None:
|
||||
"""Replay a recorded loop iteration manifest."""
|
||||
if iteration_id is None:
|
||||
|
|
@ -295,6 +340,29 @@ def replay_cmd(
|
|||
except (FileNotFoundError, TypeError, ValueError) as exc:
|
||||
console.print(f"[red]replay failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(code=2)
|
||||
|
||||
if extract is not None:
|
||||
from soup_cli.cans.unpack import extract_can
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
can_path = os.path.join(".soup-loops", iteration_id, "iteration.can")
|
||||
if not os.path.isfile(can_path):
|
||||
console.print(
|
||||
f"[red]no .can for {escape(iteration_id)} "
|
||||
"(was it run with `watch --pack-cans`?)[/]"
|
||||
)
|
||||
raise typer.Exit(code=1)
|
||||
try:
|
||||
enforce_under_cwd_and_no_symlink(extract, "extract")
|
||||
dest = extract_can(can_path, extract)
|
||||
except (FileNotFoundError, ValueError, TypeError) as exc:
|
||||
console.print(f"[red]extract failed:[/] {escape(str(exc))}")
|
||||
raise typer.Exit(code=2)
|
||||
console.print(
|
||||
f"[green]extracted {escape(iteration_id)} -> {escape(str(dest))}[/]"
|
||||
)
|
||||
return
|
||||
|
||||
table = Table(title=f"replay {escape(record.iteration_id)}", show_header=False)
|
||||
table.add_column("field", style="bold")
|
||||
table.add_column("value")
|
||||
|
|
|
|||
|
|
@ -45,6 +45,8 @@ _VALID_KINDS = frozenset(
|
|||
"steering_vector",
|
||||
# v0.71.1 #214 — pairwise-judge calibration reports.
|
||||
"judge_calibration",
|
||||
# v0.71.4 #173 — adapter branch-pointer JSON files.
|
||||
"branch_ref",
|
||||
}
|
||||
)
|
||||
_VALID_RELATIONS = frozenset(
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ import re
|
|||
import stat
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from dataclasses import asdict, dataclass, replace
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
|
||||
|
|
@ -48,6 +48,10 @@ class Branch:
|
|||
base_model: str
|
||||
created_at: float
|
||||
soup_version: str
|
||||
# v0.71.4 #173 — optional link to a v0.26 Registry entry so a branch
|
||||
# pointer participates in the lineage DAG. ``None`` preserves back-compat
|
||||
# with v0.57.0 pointers that have no Registry edge.
|
||||
registry_entry_id: Optional[str] = None
|
||||
|
||||
|
||||
def _validate_name(name: object) -> str:
|
||||
|
|
@ -118,17 +122,30 @@ def create_branch(
|
|||
config_path: str,
|
||||
base_model: str,
|
||||
dataset_path: Optional[str] = None,
|
||||
registry_entry_id: Optional[str] = None,
|
||||
dataset_sha256: Optional[str] = None,
|
||||
) -> Branch:
|
||||
"""Snapshot a training environment as an immutable branch pointer.
|
||||
|
||||
The config file is hashed (SHA-256) and the dataset file (if provided)
|
||||
is hashed too so callers can detect "same config, different data" drift.
|
||||
|
||||
``registry_entry_id`` (v0.71.4 #173) optionally links the pointer to a
|
||||
v0.26 Registry entry. ``dataset_sha256`` lets a caller (e.g.
|
||||
``branch_from_registry``) record a known dataset hash directly instead
|
||||
of re-hashing a file — mutually exclusive with ``dataset_path``.
|
||||
"""
|
||||
from soup_cli import __version__
|
||||
|
||||
name = _validate_name(name)
|
||||
enforce_under_cwd_and_no_symlink(config_path, "config_path")
|
||||
_validate_str_field(base_model, "base_model", max_len=512)
|
||||
if registry_entry_id is not None:
|
||||
registry_entry_id = _validate_str_field(
|
||||
registry_entry_id, "registry_entry_id", max_len=128
|
||||
)
|
||||
if dataset_path is not None and dataset_sha256 is not None:
|
||||
raise ValueError("pass dataset_path OR dataset_sha256, not both")
|
||||
|
||||
config_full = Path(config_path)
|
||||
if not config_full.is_file():
|
||||
|
|
@ -147,6 +164,10 @@ def create_branch(
|
|||
if not ds_full.is_file():
|
||||
raise FileNotFoundError(f"dataset not found: {ds_full.name}")
|
||||
dataset_sha = _hash_file(ds_full)
|
||||
elif dataset_sha256 is not None:
|
||||
dataset_sha = _validate_str_field(
|
||||
dataset_sha256, "dataset_sha256", max_len=128
|
||||
)
|
||||
|
||||
branches_dir = _branches_dir()
|
||||
existing = sorted(p for p in branches_dir.glob("*.json"))
|
||||
|
|
@ -161,6 +182,7 @@ def create_branch(
|
|||
base_model=base_model,
|
||||
created_at=time.time(),
|
||||
soup_version=__version__,
|
||||
registry_entry_id=registry_entry_id,
|
||||
)
|
||||
_atomic_write_branch(branches_dir, branch)
|
||||
return branch
|
||||
|
|
@ -220,6 +242,14 @@ def load_branch(name: str) -> Branch:
|
|||
base_model=_validate_str_field(raw["base_model"], "base_model"),
|
||||
created_at=float(raw["created_at"]),
|
||||
soup_version=_validate_str_field(raw["soup_version"], "soup_version", max_len=64),
|
||||
# ``.get`` so v0.57.0 pointers (no registry_entry_id key) load cleanly.
|
||||
registry_entry_id=(
|
||||
_validate_str_field(
|
||||
raw["registry_entry_id"], "registry_entry_id", max_len=128
|
||||
)
|
||||
if raw.get("registry_entry_id") is not None
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -238,6 +268,94 @@ def delete_branch(name: str) -> bool:
|
|||
return True
|
||||
|
||||
|
||||
def branch_pointer_path(name: str) -> Path:
|
||||
"""Return the on-disk JSON pointer path for a branch (must exist)."""
|
||||
name = _validate_name(name)
|
||||
target = _branches_dir() / f"{name}.json"
|
||||
if not target.is_file():
|
||||
raise FileNotFoundError(f"branch not found: {name}")
|
||||
return target
|
||||
|
||||
|
||||
def attach_branch_to_registry(name: str, registry_ref: str) -> int:
|
||||
"""Attach a branch pointer to a v0.26 Registry entry as ``branch_ref``.
|
||||
|
||||
Resolves ``registry_ref`` (id / prefix / name:tag), attaches the branch
|
||||
JSON file as a ``branch_ref`` artifact (``enforce_cwd=False`` since the
|
||||
pointer lives in ``~/.soup/branches`` outside cwd — a library-produced
|
||||
path), then re-writes the pointer with ``registry_entry_id`` set so the
|
||||
edge is recorded on both sides. Returns the artifact rowid.
|
||||
"""
|
||||
from soup_cli.registry.store import RegistryStore
|
||||
|
||||
branch = load_branch(name)
|
||||
pointer = branch_pointer_path(name)
|
||||
branches_dir = _branches_dir()
|
||||
with RegistryStore() as store:
|
||||
resolved = store.resolve(registry_ref)
|
||||
if resolved is None:
|
||||
raise ValueError(f"registry entry not found: {registry_ref}")
|
||||
rowid = store.add_artifact(
|
||||
entry_id=resolved,
|
||||
kind="branch_ref",
|
||||
path=str(pointer),
|
||||
enforce_cwd=False,
|
||||
)
|
||||
# Re-write the pointer with the resolved id so the edge is bidirectional.
|
||||
_atomic_write_branch(branches_dir, replace(branch, registry_entry_id=resolved))
|
||||
return rowid
|
||||
|
||||
|
||||
def branch_from_registry(
|
||||
name: str,
|
||||
registry_ref: str,
|
||||
*,
|
||||
config_out: Optional[str] = None,
|
||||
) -> Branch:
|
||||
"""Create a branch by deriving config + base_model + dataset from a Registry entry.
|
||||
|
||||
Materialises the entry's stored ``config_json`` to a cwd file (default
|
||||
``<name>.from-registry.yaml``) so the snapshot points at a real config,
|
||||
derives ``base_model`` from the entry, and records the entry's
|
||||
``data_hash`` as the branch's ``dataset_sha256`` (the registry keeps the
|
||||
file hash, not a path). The branch is linked back via ``registry_entry_id``.
|
||||
"""
|
||||
import yaml
|
||||
|
||||
from soup_cli.registry.store import RegistryStore
|
||||
|
||||
name = _validate_name(name)
|
||||
with RegistryStore() as store:
|
||||
resolved = store.resolve(registry_ref)
|
||||
if resolved is None:
|
||||
raise ValueError(f"registry entry not found: {registry_ref}")
|
||||
entry = store.get(resolved)
|
||||
if entry is None:
|
||||
raise ValueError(f"registry entry not found: {registry_ref}")
|
||||
base_model = entry.get("base_model") or "unknown"
|
||||
data_hash = entry.get("data_hash")
|
||||
try:
|
||||
config = json.loads(entry.get("config_json") or "{}")
|
||||
except (TypeError, ValueError):
|
||||
config = {}
|
||||
|
||||
out_name = config_out if config_out is not None else f"{name}.from-registry.yaml"
|
||||
enforce_under_cwd_and_no_symlink(out_name, "config_out")
|
||||
out_path = Path(out_name)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
out_path.write_text(
|
||||
yaml.safe_dump(config, sort_keys=False), encoding="utf-8"
|
||||
)
|
||||
|
||||
return create_branch(
|
||||
name,
|
||||
config_path=str(out_path),
|
||||
base_model=base_model,
|
||||
registry_entry_id=resolved,
|
||||
dataset_sha256=data_hash if data_hash else None,
|
||||
)
|
||||
|
||||
|
||||
def write_checkout(branch: Branch, target_path: str) -> Path:
|
||||
"""Copy the snapshotted config into cwd so the user can re-run."""
|
||||
if not isinstance(branch, Branch):
|
||||
|
|
|
|||
|
|
@ -21,11 +21,14 @@ import stat
|
|||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, FrozenSet, Literal, Mapping, Sequence, Tuple
|
||||
from typing import Any, Callable, FrozenSet, Literal, Mapping, Optional, Sequence, Tuple
|
||||
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
MergeStrategy = Literal["linear", "ties", "dare", "svd", "cmaes"]
|
||||
# A canary scorer: ``scorer(role, tasks) -> per-prompt scores`` where ``role``
|
||||
# is "baseline" / "candidate" and ``tasks`` is the parsed canary task list.
|
||||
CanaryScorer = Callable[[str, Sequence[Mapping[str, Any]]], Sequence[float]]
|
||||
# frozenset for O(1) membership + immutability (mirrors v0.41.0 / v0.51.0 policy);
|
||||
# tuple alias preserved for caller code that iterates in canonical order.
|
||||
# v0.67.0 Part A: added "cmaes" — evolutionary search dispatched separately
|
||||
|
|
@ -275,6 +278,15 @@ def merge_adapters(
|
|||
raise ValueError(
|
||||
f"strategy must be one of {SUPPORTED_STRATEGIES}, got {strategy!r}"
|
||||
)
|
||||
if strategy == "cmaes":
|
||||
# cmaes is an evolutionary *search* over linear weights, not a
|
||||
# one-shot tensor merge — routing it through this function would
|
||||
# silently fall into the svd branch. Fail loudly with the real path.
|
||||
raise ValueError(
|
||||
"cmaes is an evolutionary strategy; use run_cmaes_merge / "
|
||||
"`soup adapters merge --strategy cmaes --eval ...`, "
|
||||
"not merge_adapters(strategy='cmaes')"
|
||||
)
|
||||
if not isinstance(adapter_paths, Sequence) or isinstance(adapter_paths, str):
|
||||
raise TypeError("adapter_paths must be a sequence of strings")
|
||||
if len(adapter_paths) < _MIN_ADAPTERS:
|
||||
|
|
@ -300,7 +312,7 @@ def merge_adapters(
|
|||
else: # svd
|
||||
merged, skipped = merge_svd(weights_list, coeffs, rank=rank)
|
||||
|
||||
_write_merged_adapter(output_dir, adapter_paths[0], merged)
|
||||
write_merged_adapter(output_dir, adapter_paths[0], merged)
|
||||
|
||||
return MergeReport(
|
||||
strategy=strategy,
|
||||
|
|
@ -341,7 +353,7 @@ def _atomic_write_bytes(target: Path, data: bytes) -> None:
|
|||
raise
|
||||
|
||||
|
||||
def _write_merged_adapter(
|
||||
def write_merged_adapter(
|
||||
output_dir: str,
|
||||
template_source: str,
|
||||
weights: Mapping[str, Any],
|
||||
|
|
@ -402,17 +414,143 @@ def _write_merged_adapter(
|
|||
)
|
||||
|
||||
|
||||
# Back-compat alias: ``_write_merged_adapter`` was private through v0.71.4;
|
||||
# promoted to the public ``write_merged_adapter`` so cmaes_merge imports a
|
||||
# public name (review MEDIUM-3). Keep the old name for any external caller.
|
||||
_write_merged_adapter = write_merged_adapter
|
||||
|
||||
|
||||
_MAX_CANARY_BYTES = 16 * 1024 * 1024 # 16 MiB cap on the canary-suite JSON
|
||||
_VERDICT_MINOR = 0.02
|
||||
_VERDICT_MAJOR = 0.05
|
||||
|
||||
|
||||
def _classify_drop(delta: float) -> str:
|
||||
"""OK / MINOR / MAJOR per the v0.26.0 Quant-Lobotomy taxonomy.
|
||||
|
||||
``delta`` is ``candidate_mean - baseline_mean``: positive (improvement)
|
||||
or a small drop is OK, a 2-5 % drop is MINOR, a >5 % drop is MAJOR.
|
||||
"""
|
||||
if delta >= 0:
|
||||
return "OK"
|
||||
drop = -delta
|
||||
if drop < _VERDICT_MINOR:
|
||||
return "OK"
|
||||
if drop < _VERDICT_MAJOR:
|
||||
return "MINOR"
|
||||
return "MAJOR"
|
||||
|
||||
|
||||
def _require_score_list(value: Any, field: str) -> list[float]:
|
||||
if not isinstance(value, list) or isinstance(value, (str, bytes)):
|
||||
raise ValueError(f"{field} must be a JSON list of numbers")
|
||||
out: list[float] = []
|
||||
for v in value:
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
||||
raise ValueError(f"{field} entries must be numbers")
|
||||
vf = float(v)
|
||||
if not math.isfinite(vf):
|
||||
raise ValueError(f"{field} entries must be finite")
|
||||
out.append(vf)
|
||||
if not out:
|
||||
raise ValueError(f"{field} must be non-empty")
|
||||
return out
|
||||
|
||||
|
||||
def _load_canary_scores(
|
||||
canary_suite: str,
|
||||
scorer: Optional[CanaryScorer],
|
||||
) -> Tuple[list[float], list[float]]:
|
||||
"""Resolve baseline + candidate per-prompt scores from a canary suite.
|
||||
|
||||
Two supported shapes:
|
||||
|
||||
- ``{"baseline_scores": [...], "candidate_scores": [...]}`` — pre-scored
|
||||
(no model load; the no-GPU workflow). Operators run ``soup eval custom``
|
||||
against the baseline and merged adapters, then assemble the two arrays.
|
||||
- ``{"tasks": [{"prompt", "expected"}, ...]}`` — requires an injectable
|
||||
``scorer(role, tasks) -> list[float]`` (the live path).
|
||||
"""
|
||||
# Defence-in-depth cwd containment + symlink rejection at the read site so
|
||||
# the public ``predict_merged_verdict`` entry point is safe even when a
|
||||
# non-CLI caller skips the CLI-boundary check (mirrors every other read
|
||||
# surface in this changeset).
|
||||
enforce_under_cwd_and_no_symlink(canary_suite, "canary_suite")
|
||||
path = Path(canary_suite)
|
||||
if not path.is_file():
|
||||
raise ValueError(f"canary suite not found: {path.name}")
|
||||
# Open ONCE with O_NOFOLLOW and enforce the size cap on the same fd
|
||||
# (os.fstat), so the symlink-rejection + 16 MiB cap cannot be defeated by a
|
||||
# local file swap between the lstat/stat and the read (TOCTOU) — matches the
|
||||
# O_NOFOLLOW+fstat read pattern used elsewhere in the codebase.
|
||||
try:
|
||||
fd = os.open(str(path), os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0))
|
||||
except OSError as exc:
|
||||
raise ValueError(f"canary suite unreadable: {type(exc).__name__}") from exc
|
||||
fh = None
|
||||
try:
|
||||
if os.fstat(fd).st_size > _MAX_CANARY_BYTES:
|
||||
raise ValueError("canary suite exceeds 16 MiB cap")
|
||||
fh = os.fdopen(fd, "r", encoding="utf-8")
|
||||
raw = fh.read()
|
||||
finally:
|
||||
if fh is not None:
|
||||
fh.close()
|
||||
else:
|
||||
os.close(fd)
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise ValueError(f"canary suite is not valid JSON: {exc}") from exc
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("canary suite root must be a JSON object")
|
||||
|
||||
if "baseline_scores" in data and "candidate_scores" in data:
|
||||
baseline = _require_score_list(data["baseline_scores"], "baseline_scores")
|
||||
candidate = _require_score_list(data["candidate_scores"], "candidate_scores")
|
||||
else:
|
||||
tasks = data.get("tasks")
|
||||
if not isinstance(tasks, list) or not tasks:
|
||||
raise ValueError(
|
||||
"canary suite must contain 'baseline_scores'+'candidate_scores' "
|
||||
"or a non-empty 'tasks' list"
|
||||
)
|
||||
if scorer is None or not callable(scorer):
|
||||
raise ValueError(
|
||||
"canary suite has 'tasks' but no scorer; supply pre-scored "
|
||||
"'baseline_scores'/'candidate_scores' arrays or pass scorer="
|
||||
)
|
||||
baseline = _require_score_list(scorer("baseline", tasks), "baseline scorer output")
|
||||
candidate = _require_score_list(scorer("candidate", tasks), "candidate scorer output")
|
||||
|
||||
if len(baseline) != len(candidate):
|
||||
raise ValueError(
|
||||
f"baseline ({len(baseline)}) and candidate ({len(candidate)}) "
|
||||
"score counts must match"
|
||||
)
|
||||
return baseline, candidate
|
||||
|
||||
|
||||
def predict_merged_verdict(
|
||||
report: MergeReport,
|
||||
canary_suite: str | None = None,
|
||||
*,
|
||||
scorer: Optional[CanaryScorer] = None,
|
||||
) -> str:
|
||||
"""Stub for v0.57.1: live canary-eval verdict via v0.55 gate.
|
||||
"""Live canary verdict for a merged adapter (v0.71.4 #172).
|
||||
|
||||
Today returns the existing report.verdict. The signature is forward-compatible
|
||||
so v0.57.1 can lift this to a real OK/MINOR/MAJOR classification.
|
||||
Returns the merged adapter's OK / MINOR / MAJOR verdict by comparing
|
||||
per-prompt canary scores against the first input adapter (the "baseline"
|
||||
by convention) using the v0.26.0 Quant-Lobotomy thresholds. When
|
||||
``canary_suite`` is ``None`` the existing ``report.verdict`` (``UNKNOWN``
|
||||
for a fresh merge) is returned unchanged — back-compat with v0.57.0.
|
||||
"""
|
||||
if not isinstance(report, MergeReport):
|
||||
raise TypeError("report must be MergeReport")
|
||||
if canary_suite is not None and not isinstance(canary_suite, str):
|
||||
raise TypeError("canary_suite must be str or None")
|
||||
return report.verdict
|
||||
if canary_suite is None:
|
||||
return report.verdict
|
||||
baseline, candidate = _load_canary_scores(canary_suite, scorer)
|
||||
delta = (sum(candidate) / len(candidate)) - (sum(baseline) / len(baseline))
|
||||
return _classify_drop(delta)
|
||||
|
|
|
|||
|
|
@ -19,9 +19,10 @@ from __future__ import annotations
|
|||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Any, Iterable, Mapping, Tuple
|
||||
from typing import Any, Callable, Iterable, Mapping, Optional, Tuple
|
||||
|
||||
from soup_cli.utils.paths import atomic_write_text
|
||||
|
||||
|
|
@ -343,3 +344,152 @@ def write_pr_markdown(pr: AdapterPR, path: str) -> str:
|
|||
raise TypeError("pr must be AdapterPR")
|
||||
text = render_pr_markdown(pr)
|
||||
return atomic_write_text(text, path, field="pr markdown path")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GitHub PR publisher (v0.71.4 #223)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# owner/repo#<number> — owner + repo are GitHub name-safe (alnum + ._-),
|
||||
# number is a positive integer.
|
||||
_PR_TARGET_RE = re.compile(
|
||||
r"^([A-Za-z0-9][A-Za-z0-9._-]*)/([A-Za-z0-9][A-Za-z0-9._-]*)#([0-9]+)$"
|
||||
)
|
||||
_MAX_PR_BODY_BYTES = 60_000 # GitHub caps issue-comment bodies at 65_536 bytes
|
||||
|
||||
# Env keys passed through to the `gh` child — everything else (HF_TOKEN /
|
||||
# OPENAI_API_KEY / ANTHROPIC_API_KEY / ...) is filtered out so a publish call
|
||||
# never leaks unrelated secrets to the subprocess (v0.71.4 review HIGH fix,
|
||||
# mirrors v0.44.0 _LLAMA_ENV_ALLOWLIST).
|
||||
_GH_ENV_ALLOWLIST = frozenset(
|
||||
{
|
||||
"PATH", "HOME", "USERPROFILE", "APPDATA", "LOCALAPPDATA",
|
||||
"SYSTEMROOT", "SystemRoot", "TEMP", "TMP", "TMPDIR",
|
||||
"GH_TOKEN", "GITHUB_TOKEN", "GH_ENTERPRISE_TOKEN", "GITHUB_API_URL",
|
||||
"GH_HOST", "GH_CONFIG_DIR",
|
||||
"XDG_CONFIG_HOME", "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY",
|
||||
"http_proxy", "https_proxy", "no_proxy",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def parse_pr_target(target: object) -> Tuple[str, str, int]:
|
||||
"""Parse ``"owner/repo#42"`` into ``(owner, repo, pr_number)``.
|
||||
|
||||
Raises ``TypeError`` for non-strings and ``ValueError`` for any string
|
||||
that does not match the ``owner/repo#<positive-int>`` shape.
|
||||
"""
|
||||
if isinstance(target, bool) or not isinstance(target, str):
|
||||
raise TypeError("target must be str")
|
||||
match = _PR_TARGET_RE.match(target.strip())
|
||||
if not match:
|
||||
raise ValueError(
|
||||
"target must be 'owner/repo#<number>' "
|
||||
"(e.g. MakazhanAlpamys/Soup#42)"
|
||||
)
|
||||
num = int(match.group(3))
|
||||
if num < 1:
|
||||
raise ValueError("PR number must be >= 1")
|
||||
return match.group(1), match.group(2), num
|
||||
|
||||
|
||||
def resolve_github_token(env: Optional[Mapping[str, str]] = None) -> str:
|
||||
"""Resolve a GitHub token from ``GITHUB_TOKEN`` / ``GH_TOKEN`` env.
|
||||
|
||||
Mirrors the v0.29.0 HF token-resolution policy: env only, first-match
|
||||
wins, blank values treated as missing. Raises ``RuntimeError`` (a
|
||||
user-actionable error the CLI renders) when neither is set.
|
||||
"""
|
||||
source = env if env is not None else os.environ
|
||||
for key in ("GITHUB_TOKEN", "GH_TOKEN"):
|
||||
val = source.get(key)
|
||||
if val and val.strip():
|
||||
return val.strip()
|
||||
raise RuntimeError(
|
||||
"no GitHub token found; set GITHUB_TOKEN (or GH_TOKEN) "
|
||||
"to publish a PR comment"
|
||||
)
|
||||
|
||||
|
||||
def post_pr_comment(
|
||||
target: str,
|
||||
body: str,
|
||||
*,
|
||||
env: Optional[Mapping[str, str]] = None,
|
||||
runner: Optional[Callable[..., Any]] = None,
|
||||
) -> str:
|
||||
"""Post ``body`` as a comment on the GitHub PR named by ``target``.
|
||||
|
||||
Uses ``gh api`` (no PyGithub dependency) with the body sent over
|
||||
JSON stdin so multiline / markdown content is never shell-interpolated.
|
||||
Auth resolves via ``GITHUB_TOKEN`` / ``GH_TOKEN`` (gh reads the same
|
||||
vars). Returns the created comment's ``html_url`` (best-effort, may be
|
||||
empty). ``runner`` is an injectable ``subprocess.run`` for testing.
|
||||
"""
|
||||
owner, repo, num = parse_pr_target(target)
|
||||
if not isinstance(body, str):
|
||||
raise TypeError("body must be str")
|
||||
if not body.strip():
|
||||
raise ValueError("body must be non-empty")
|
||||
if "\x00" in body:
|
||||
raise ValueError("body must not contain null bytes")
|
||||
if len(body.encode("utf-8")) > _MAX_PR_BODY_BYTES:
|
||||
raise ValueError(
|
||||
f"body exceeds {_MAX_PR_BODY_BYTES} byte GitHub comment cap"
|
||||
)
|
||||
# Fail fast if no token before spawning the subprocess.
|
||||
token = resolve_github_token(env)
|
||||
|
||||
import subprocess # noqa: S404 — argv list mode, no shell
|
||||
|
||||
argv = [
|
||||
"gh",
|
||||
"api",
|
||||
"--method",
|
||||
"POST",
|
||||
f"repos/{owner}/{repo}/issues/{num}/comments",
|
||||
"--input",
|
||||
"-",
|
||||
]
|
||||
stdin = json.dumps({"body": body})
|
||||
# gh resolves GH_TOKEN / GITHUB_TOKEN from its own environment; thread
|
||||
# the resolved token through. When ``env`` is supplied (tests / explicit)
|
||||
# use it verbatim; otherwise build a MINIMAL env from an allowlist so we
|
||||
# never leak HF_TOKEN / OPENAI_API_KEY / ANTHROPIC_API_KEY etc. into the
|
||||
# gh child (mirrors the v0.44.0 _LLAMA_ENV_ALLOWLIST policy).
|
||||
if env is not None:
|
||||
base_env = dict(env)
|
||||
else:
|
||||
base_env = {
|
||||
k: v for k, v in os.environ.items() if k in _GH_ENV_ALLOWLIST
|
||||
}
|
||||
base_env.setdefault("GH_TOKEN", token)
|
||||
run = runner if runner is not None else subprocess.run
|
||||
try:
|
||||
result = run(
|
||||
argv,
|
||||
input=stdin,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
check=False,
|
||||
env=base_env,
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
raise RuntimeError(
|
||||
"`gh` CLI not found; install GitHub CLI or drop --push "
|
||||
"and use --output to write the Markdown"
|
||||
) from exc
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise RuntimeError("gh api timed out posting the PR comment") from exc
|
||||
|
||||
if getattr(result, "returncode", 1) != 0:
|
||||
stderr = (getattr(result, "stderr", "") or "").strip()[:512]
|
||||
raise RuntimeError(
|
||||
f"gh api failed (rc={result.returncode}): {stderr or 'no detail'}"
|
||||
)
|
||||
try:
|
||||
data = json.loads(getattr(result, "stdout", "") or "{}")
|
||||
return str(data.get("html_url") or "")
|
||||
except (ValueError, TypeError):
|
||||
return ""
|
||||
|
|
|
|||
|
|
@ -36,6 +36,8 @@ from __future__ import annotations
|
|||
|
||||
import math
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable, Optional, Sequence, Tuple
|
||||
|
|
@ -43,6 +45,12 @@ from typing import Callable, Optional, Sequence, Tuple
|
|||
from soup_cli.utils.blame import parse_budget
|
||||
from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
|
||||
|
||||
# Test / smoke escape hatch (mirrors v0.53.1 #109 deploy_measure pattern):
|
||||
# when set, build_cmaes_eval_fn uses this scorer instead of loading a model.
|
||||
# Production callers leave it None and either pass scorer= explicitly or let
|
||||
# the default lazy model-loading scorer run.
|
||||
_CMAES_SCORER_OVERRIDE: Optional[Callable[[str, str], float]] = None
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bounds (closed, locked at module load)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -56,6 +64,7 @@ _MIN_ADAPTERS = 2
|
|||
_MAX_ADAPTERS = 16 # mirrors v0.57.0 adapter_merge cap
|
||||
_SIMPLEX_TOL = 1e-6
|
||||
_FAILED_EVAL_SENTINEL = -1.0e9 # very negative so failed candidates never win
|
||||
_MAX_ADAPTER_CONFIG_BYTES = 256 * 1024 # mirrors adapter_merge config-read cap
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -422,6 +431,146 @@ def run_cmaes_merge(
|
|||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Live eval-suite auto-wiring (v0.71.4 #220)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _load_adapter_generator(merged_dir: str) -> Callable[[str], str]:
|
||||
"""Build a ``generate_fn(prompt) -> str`` from a merged LoRA adapter.
|
||||
|
||||
Loads the base model named in the adapter's ``adapter_config.json`` and
|
||||
applies the merged adapter via PEFT. Heavy imports stay inside the
|
||||
function (project lazy-import policy + the cmaes_merge no-top-level-torch
|
||||
grep guard).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
from pathlib import Path
|
||||
|
||||
from peft import PeftModel
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
# On the wired cmaes path ``merged_dir`` is always a Soup-written mkdtemp,
|
||||
# but this is a public helper — guard the config read with a symlink
|
||||
# rejection + size cap so a caller-supplied dir can't smuggle a symlink or
|
||||
# a multi-GB JSON (defence-in-depth, parity with every other config read).
|
||||
cfg_path = Path(merged_dir) / "adapter_config.json"
|
||||
try:
|
||||
cst = os.lstat(cfg_path)
|
||||
except OSError as exc:
|
||||
raise ValueError(
|
||||
f"merged adapter_config.json unreadable: {type(exc).__name__}"
|
||||
) from exc
|
||||
if stat.S_ISLNK(cst.st_mode):
|
||||
raise ValueError("merged adapter_config.json must not be a symlink")
|
||||
if cst.st_size > _MAX_ADAPTER_CONFIG_BYTES:
|
||||
raise ValueError("merged adapter_config.json exceeds size cap")
|
||||
cfg = json.loads(cfg_path.read_text(encoding="utf-8"))
|
||||
base = cfg.get("base_model_name_or_path")
|
||||
if not base:
|
||||
raise ValueError(
|
||||
"merged adapter_config.json has no base_model_name_or_path"
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=False)
|
||||
model = AutoModelForCausalLM.from_pretrained(base, trust_remote_code=False)
|
||||
model = PeftModel.from_pretrained(model, merged_dir)
|
||||
model.eval()
|
||||
|
||||
def _gen(prompt: str) -> str:
|
||||
if not prompt:
|
||||
return ""
|
||||
inputs = tokenizer(prompt, return_tensors="pt", truncation=True)
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=64,
|
||||
do_sample=False,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
|
||||
return tokenizer.decode(new_tokens, skip_special_tokens=True)
|
||||
|
||||
return _gen
|
||||
|
||||
|
||||
def _default_cmaes_scorer(merged_dir: str, eval_suite: str) -> float:
|
||||
"""Score a merged adapter against an eval suite (mean score in [0, 1]).
|
||||
|
||||
The live default — loads the base + merged adapter and runs the
|
||||
v0.25.0 ``soup_cli.eval.custom`` scorers. Slow (one model load); use
|
||||
the ``scorer=`` injection / ``_CMAES_SCORER_OVERRIDE`` escape hatch for
|
||||
fast offline runs.
|
||||
"""
|
||||
from soup_cli.eval.custom import load_eval_tasks, score_task
|
||||
|
||||
generate_fn = _load_adapter_generator(merged_dir)
|
||||
tasks = load_eval_tasks(eval_suite)
|
||||
if not tasks:
|
||||
return 0.0
|
||||
total = 0.0
|
||||
for task in tasks:
|
||||
output = generate_fn(task.prompt)
|
||||
total += float(score_task(task, output).score)
|
||||
return total / len(tasks)
|
||||
|
||||
|
||||
def _resolve_cmaes_scorer(
|
||||
scorer: Optional[Callable[[str, str], float]],
|
||||
) -> Callable[[str, str], float]:
|
||||
if scorer is not None:
|
||||
if not callable(scorer):
|
||||
raise TypeError("scorer must be callable")
|
||||
return scorer
|
||||
if _CMAES_SCORER_OVERRIDE is not None:
|
||||
return _CMAES_SCORER_OVERRIDE
|
||||
return _default_cmaes_scorer
|
||||
|
||||
|
||||
def build_cmaes_eval_fn(
|
||||
plan: CmaesPlan,
|
||||
*,
|
||||
scorer: Optional[Callable[[str, str], float]] = None,
|
||||
) -> Callable[[Tuple[float, ...]], float]:
|
||||
"""Return an ``eval_fn(weights) -> float`` for :func:`run_cmaes_merge`.
|
||||
|
||||
The closure linearly merges ``plan.adapters`` with the candidate weights,
|
||||
materialises the merged LoRA to a temp dir, scores it via ``scorer``
|
||||
(default: live model-loading scorer), and cleans up. The *adapter* weights
|
||||
are loaded from disk once up front so per-generation cost is merge + score.
|
||||
|
||||
Perf note: the default :func:`_default_cmaes_scorer` reloads the *base
|
||||
model* into a fresh PEFT wrapper on every candidate evaluation (it is
|
||||
stateless by design). For ``population × max_generations`` candidates that
|
||||
is many base-model loads — expensive for non-tiny models. Pass a custom
|
||||
``scorer=`` that caches the base model (or set ``_CMAES_SCORER_OVERRIDE``)
|
||||
to amortise it; the injection also avoids any model load for tests / smokes.
|
||||
|
||||
``scorer(merged_dir, eval_suite) -> float`` returns a mean score in
|
||||
``[0, 1]``.
|
||||
"""
|
||||
if not isinstance(plan, CmaesPlan):
|
||||
raise TypeError("plan must be CmaesPlan")
|
||||
resolved = _resolve_cmaes_scorer(scorer)
|
||||
|
||||
from soup_cli.utils.adapter_diff import load_adapter_weights
|
||||
from soup_cli.utils.adapter_merge import merge_linear, write_merged_adapter
|
||||
|
||||
weights_list = [load_adapter_weights(p) for p in plan.adapters]
|
||||
template_source = plan.adapters[0]
|
||||
|
||||
def _eval_fn(weights: Tuple[float, ...]) -> float:
|
||||
merged, _skipped = merge_linear(weights_list, list(weights))
|
||||
tmp_dir = tempfile.mkdtemp(prefix=".soup_cmaes_")
|
||||
try:
|
||||
write_merged_adapter(tmp_dir, template_source, merged)
|
||||
return float(resolved(tmp_dir, plan.eval_suite))
|
||||
finally:
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
|
||||
return _eval_fn
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deterministic RNG (no numpy dependency at module top)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -105,6 +105,12 @@ class WatchConfig:
|
|||
deploy_fn: DeployFn = default_deploy
|
||||
cost_fn: CostFn = default_cost
|
||||
on_iteration: Optional[Callable[[IterationRecord], None]] = None
|
||||
# v0.71.4 #177 — pack each successful iteration as a v0.26 Soup Can +
|
||||
# append a Registry entry (default off so existing tests / stub watchers
|
||||
# have no registry side effects).
|
||||
pack_iterations: bool = False
|
||||
served_model: Optional[str] = None
|
||||
base_model: str = "unknown"
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
v = self.poll_interval_sec
|
||||
|
|
@ -121,6 +127,21 @@ class WatchConfig:
|
|||
raise ValueError(f"{fname} must be callable")
|
||||
if self.on_iteration is not None and not callable(self.on_iteration):
|
||||
raise ValueError("on_iteration must be callable or None")
|
||||
if not isinstance(self.pack_iterations, bool):
|
||||
raise ValueError("pack_iterations must be bool")
|
||||
# Validate the #177 registry-packing identity fields like every other
|
||||
# field, so a NUL/oversize value fails at construction rather than deep
|
||||
# inside store.push (review LOW). They flow into registry_name_from /
|
||||
# store.push(base_model=...).
|
||||
if self.served_model is not None:
|
||||
sm = self.served_model
|
||||
if not isinstance(sm, str) or "\x00" in sm or len(sm) > 512:
|
||||
raise ValueError(
|
||||
"served_model must be a NUL-free str <= 512 chars or None"
|
||||
)
|
||||
bm = self.base_model
|
||||
if not isinstance(bm, str) or not bm or "\x00" in bm or len(bm) > 512:
|
||||
raise ValueError("base_model must be a non-empty NUL-free str <= 512 chars")
|
||||
|
||||
|
||||
def run_once(
|
||||
|
|
@ -232,6 +253,9 @@ def watch(config: WatchConfig) -> "tuple[LoopState, int]":
|
|||
pass
|
||||
|
||||
iterations = 0
|
||||
# v0.71.4 #177 — chain Registry entries across iterations so the loop
|
||||
# forms a real lineage DAG (parent links to the prior iteration's entry).
|
||||
prev_registry_id: Optional[str] = None
|
||||
state = read_state(config.state_path)
|
||||
# Only promote `stopped` → `running` automatically; `paused` must
|
||||
# survive a `soup loop watch` invocation so a SIGTERM + restart
|
||||
|
|
@ -261,10 +285,16 @@ def watch(config: WatchConfig) -> "tuple[LoopState, int]":
|
|||
# iteration_count to match the manifest count (code-review
|
||||
# HIGH #3 fix). The state still records the skip in notes.
|
||||
if decision.proceed:
|
||||
wrote = False
|
||||
try:
|
||||
write_iteration(record, base_dir=config.iteration_dir)
|
||||
wrote = True
|
||||
except (OSError, ValueError) as exc:
|
||||
_LOG.warning("iteration write failed: %s", type(exc).__name__)
|
||||
if wrote and config.pack_iterations:
|
||||
prev_registry_id = _pack_iteration_safely(
|
||||
record, config, prev_registry_id
|
||||
)
|
||||
if config.on_iteration is not None:
|
||||
try:
|
||||
config.on_iteration(record)
|
||||
|
|
@ -293,6 +323,44 @@ def watch(config: WatchConfig) -> "tuple[LoopState, int]":
|
|||
return state, iterations
|
||||
|
||||
|
||||
def _pack_iteration_safely(
|
||||
record: IterationRecord,
|
||||
config: WatchConfig,
|
||||
prev_registry_id: Optional[str],
|
||||
) -> Optional[str]:
|
||||
"""Pack one iteration as a Soup Can; never crash the daemon.
|
||||
|
||||
Returns the new Registry entry id (to chain as the next iteration's
|
||||
parent) on success, or the unchanged ``prev_registry_id`` on failure.
|
||||
On failure the iteration manifest is re-written with a ``pack-failed:``
|
||||
note appended (best-effort) so the operator can see what happened
|
||||
(#177 acceptance: swallow at WARNING + record in iteration notes).
|
||||
"""
|
||||
from soup_cli.utils.loop_iteration import pack_iteration_as_can
|
||||
|
||||
try:
|
||||
_, entry_id = pack_iteration_as_can(
|
||||
record.iteration_id,
|
||||
base_dir=config.iteration_dir,
|
||||
served_model=config.served_model,
|
||||
base_model=config.base_model,
|
||||
parent_registry_id=prev_registry_id,
|
||||
)
|
||||
return entry_id
|
||||
except Exception as exc: # noqa: BLE001 — instrumentation must not crash
|
||||
_LOG.warning("iteration pack failed: %s", type(exc).__name__)
|
||||
try:
|
||||
prefix = record.notes + " | " if record.notes else ""
|
||||
note = (prefix + f"pack-failed: {type(exc).__name__}")[:4096]
|
||||
write_iteration(
|
||||
replace(record, notes=note),
|
||||
base_dir=config.iteration_dir,
|
||||
)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
return prev_registry_id
|
||||
|
||||
|
||||
def _state_with(state: LoopState, **kwargs: object) -> LoopState:
|
||||
"""Return a copy with overrides applied (escape hatch around ``replace``).
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ from __future__ import annotations
|
|||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import tempfile
|
||||
import uuid
|
||||
|
|
@ -198,6 +199,109 @@ def read_iteration(
|
|||
raise ValueError(f"manifest contents invalid: {exc}") from exc
|
||||
|
||||
|
||||
def registry_name_from(served_model: str) -> str:
|
||||
"""Sanitise a served-model id into a valid Registry entry name.
|
||||
|
||||
``served_model`` is typically ``registry://<id>`` or a model path with
|
||||
``/`` separators — neither is a valid Registry name (``^[A-Za-z0-9][...]``).
|
||||
Strips the scheme, replaces every non-``[A-Za-z0-9_.-]`` char with ``-``,
|
||||
drops leading non-alphanumerics, and caps at 128 chars. Falls back to
|
||||
``"loop"`` when nothing usable survives.
|
||||
"""
|
||||
raw = (served_model or "").replace("registry://", "")
|
||||
cleaned = re.sub(r"[^A-Za-z0-9_.\-]", "-", raw)
|
||||
cleaned = re.sub(r"^[^A-Za-z0-9]+", "", cleaned)
|
||||
if not cleaned:
|
||||
cleaned = "loop"
|
||||
return cleaned[:128]
|
||||
|
||||
|
||||
def pack_iteration_as_can(
|
||||
iteration_id: str,
|
||||
*,
|
||||
base_dir: Optional[str] = None,
|
||||
served_model: Optional[str] = None,
|
||||
base_model: str = "unknown",
|
||||
task: str = "dpo",
|
||||
parent_registry_id: Optional[str] = None,
|
||||
) -> Tuple[str, str]:
|
||||
"""Pack a loop iteration as a v0.26 Soup Can + append a Registry entry.
|
||||
|
||||
Reads ``<base_dir>/<iteration_id>/iteration.json``, pushes a Registry
|
||||
entry (name derived from ``served_model``, tag ``loop-iter``), links it
|
||||
to ``parent_registry_id`` via a ``forked_from`` lineage edge so the loop
|
||||
forms a real DAG visible through ``soup history``, and writes
|
||||
``<base_dir>/<iteration_id>/iteration.can``. Returns ``(can_path,
|
||||
registry_entry_id)``.
|
||||
"""
|
||||
from soup_cli.cans.pack import pack_entry
|
||||
from soup_cli.registry.store import RegistryStore
|
||||
|
||||
record = read_iteration(iteration_id, base_dir=base_dir)
|
||||
parent_dir = base_dir if base_dir is not None else _DEFAULT_DIR
|
||||
iter_dir = os.path.join(parent_dir, iteration_id)
|
||||
_check_dir(iter_dir)
|
||||
if not os.path.isdir(iter_dir):
|
||||
raise FileNotFoundError(f"iteration dir not found: {iteration_id}")
|
||||
|
||||
name = registry_name_from(served_model or "loop")
|
||||
config = {
|
||||
"iteration_id": iteration_id,
|
||||
"run_id": record.run_id,
|
||||
"gate_verdict": record.gate_verdict,
|
||||
"canary_verdict": record.canary_verdict,
|
||||
"shipped": record.shipped,
|
||||
}
|
||||
with RegistryStore() as store:
|
||||
entry_id = store.push(
|
||||
name=name,
|
||||
tag="loop-iter",
|
||||
base_model=base_model or "unknown",
|
||||
task=task or "dpo",
|
||||
run_id=record.run_id,
|
||||
config=config,
|
||||
notes=f"loop iteration {iteration_id}",
|
||||
)
|
||||
if parent_registry_id is not None:
|
||||
# Best-effort lineage edge — a missing parent / cycle / FK error
|
||||
# must not prevent the new entry from being created (daemon
|
||||
# resilience). ``add_lineage`` raises ``ValueError`` for the
|
||||
# documented cycle/self-ref/missing-parent cases and may surface a
|
||||
# ``sqlite3.Error`` (e.g. FK violation on a since-deleted parent).
|
||||
import sqlite3
|
||||
|
||||
try:
|
||||
store.add_lineage(
|
||||
child_id=entry_id,
|
||||
parent_id=parent_registry_id,
|
||||
relation="forked_from",
|
||||
)
|
||||
except (ValueError, sqlite3.Error):
|
||||
pass
|
||||
|
||||
can_path = os.path.join(iter_dir, "iteration.can")
|
||||
try:
|
||||
pack_entry(
|
||||
entry_id=entry_id,
|
||||
out_path=can_path,
|
||||
author="soup-loop",
|
||||
description=f"loop iteration {iteration_id}",
|
||||
)
|
||||
except Exception:
|
||||
# All-or-nothing: a failed can write must not leave an orphaned
|
||||
# Registry entry behind. Roll it back so the watch-loop lineage
|
||||
# chain (which links the *next* iteration to the *prior successful*
|
||||
# entry) stays consistent — a half-created iteration never enters
|
||||
# the DAG. Best-effort rollback; the original error still propagates.
|
||||
try:
|
||||
with RegistryStore() as rollback_store:
|
||||
rollback_store.delete(entry_id)
|
||||
except Exception: # noqa: BLE001 — rollback is best-effort
|
||||
pass
|
||||
raise
|
||||
return can_path, entry_id
|
||||
|
||||
|
||||
def list_iterations(base_dir: Optional[str] = None) -> Tuple[str, ...]:
|
||||
"""Return iteration ids sorted by name (timestamp-prefixed)."""
|
||||
parent = base_dir if base_dir is not None else _DEFAULT_DIR
|
||||
|
|
|
|||
|
|
@ -0,0 +1,418 @@
|
|||
"""Pre-wired production stages for `soup loop watch` (v0.71.4 #176).
|
||||
|
||||
v0.58.0 shipped the watch daemon with no-op default stage callbacks; this
|
||||
module supplies real harvest / train / gate / deploy callables that compose
|
||||
the existing v0.26.0 trace-to-preference, eval-gate, and v0.30.0 multi-adapter
|
||||
deploy surfaces. Operators opt in via ``soup loop init --pre-wired`` (or
|
||||
``soup loop watch --pre-wired``).
|
||||
|
||||
Each callable matches the ``HarvestFn`` / ``TrainFn`` / ``GateFn`` /
|
||||
``DeployFn`` protocols from ``loop_daemon``. Heavy deps (transformers / peft /
|
||||
trl / httpx) are lazy-imported inside the callables so importing this module
|
||||
stays cold-start cheap (project lazy-import policy).
|
||||
|
||||
Module-level escape hatches (mirrors v0.53.1 #109 deploy_measure) let tests
|
||||
and offline smokes inject behaviour without a GPU or a network:
|
||||
|
||||
- ``_TRACE_DIR_RESOLVER(state) -> Optional[str]``
|
||||
- ``_TRAIN_RUNNER(argv, ...)`` (a ``subprocess.run`` stand-in)
|
||||
- ``_GATE_GENERATE_FACTORY(adapter_dir) -> generate_fn``
|
||||
- ``_DEPLOY_POSTER(endpoint, name) -> bool``
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Callable, Mapping, Optional
|
||||
|
||||
from soup_cli.utils.loop_state import LoopState
|
||||
from soup_cli.utils.paths import atomic_write_text
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from soup_cli.utils.loop_daemon import WatchConfig
|
||||
|
||||
_LOG = logging.getLogger(__name__)
|
||||
|
||||
_PAIRS_DIR = ".soup-loops/pairs"
|
||||
_ADAPTERS_DIR = ".soup-loops/adapters"
|
||||
_DEFAULT_TRACE_DIR = ".soup-loops/traces"
|
||||
|
||||
# Test / smoke escape hatches (default None → real behaviour).
|
||||
_TRACE_DIR_RESOLVER: Optional[Callable[[LoopState], Optional[str]]] = None
|
||||
_TRAIN_RUNNER: Optional[Callable[..., object]] = None
|
||||
_GATE_GENERATE_FACTORY: Optional[Callable[[str], Callable[[str], str]]] = None
|
||||
_DEPLOY_POSTER: Optional[Callable[[str, str], bool]] = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Harvest — production traces → preference pairs (v0.26.0 trace-to-pref)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _resolve_trace_dir(state: LoopState) -> Optional[str]:
|
||||
"""Resolve the directory of ``*.jsonl`` serve logs to harvest from.
|
||||
|
||||
Resolution order: ``SOUP_LOOP_TRACE_DIR`` env → ``served_model`` if it is
|
||||
an existing local dir → ``.soup-loops/traces``. Returns ``None`` when no
|
||||
usable directory exists.
|
||||
|
||||
Note: this is the one read surface in this module that intentionally does
|
||||
NOT enforce cwd-containment — operators legitimately keep serve logs
|
||||
outside the project dir (e.g. ``/var/log/soup-serve``). The directory is
|
||||
operator-supplied config (env / state), not untrusted input, and harvested
|
||||
content is only used to build DPO pairs (never exec'd).
|
||||
"""
|
||||
if _TRACE_DIR_RESOLVER is not None:
|
||||
return _TRACE_DIR_RESOLVER(state)
|
||||
env = os.environ.get("SOUP_LOOP_TRACE_DIR")
|
||||
candidates = []
|
||||
if env:
|
||||
candidates.append(env)
|
||||
if state.served_model and not state.served_model.startswith("registry://"):
|
||||
candidates.append(state.served_model)
|
||||
candidates.append(_DEFAULT_TRACE_DIR)
|
||||
for cand in candidates:
|
||||
if cand and os.path.isdir(cand):
|
||||
return cand
|
||||
return None
|
||||
|
||||
|
||||
def harvest_from_traces(state: LoopState) -> Mapping[str, object]:
|
||||
"""Scan serve traces for thumbs/regeneration signals → preference pairs.
|
||||
|
||||
Writes the harvested pairs to ``.soup-loops/pairs/<uuid>.jsonl`` (cwd-
|
||||
contained) and returns ``{pairs_harvested, pairs_path, traces_collected}``.
|
||||
Yields zero pairs (no path) when no trace directory is configured.
|
||||
"""
|
||||
from soup_cli.data.traces.pair_builder import build_pairs
|
||||
from soup_cli.data.traces.parsers import parse_soup_serve
|
||||
|
||||
trace_dir = _resolve_trace_dir(state)
|
||||
if not trace_dir:
|
||||
return {"pairs_harvested": 0, "pairs_path": None, "traces_collected": 0}
|
||||
|
||||
traces = list(parse_soup_serve(trace_dir))
|
||||
pairs = list(build_pairs(traces, signal="thumbs_up"))
|
||||
if not pairs:
|
||||
return {
|
||||
"pairs_harvested": 0,
|
||||
"pairs_path": None,
|
||||
"traces_collected": len(traces),
|
||||
}
|
||||
|
||||
pairs_dir = Path(_PAIRS_DIR)
|
||||
pairs_dir.mkdir(parents=True, exist_ok=True)
|
||||
pairs_path = str(pairs_dir / f"pairs-{uuid.uuid4().hex[:12]}.jsonl")
|
||||
lines = "\n".join(json.dumps(p.to_jsonl_dict()) for p in pairs) + "\n"
|
||||
# Atomic + cwd-contained + symlink-rejected (mirrors the rest of the
|
||||
# changeset's write policy).
|
||||
atomic_write_text(lines, pairs_path, field="pairs_path")
|
||||
return {
|
||||
"pairs_harvested": len(pairs),
|
||||
"pairs_path": pairs_path,
|
||||
"traces_collected": len(traces),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Train — generated DPO YAML → `soup train` subprocess
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _render_dpo_yaml(state: LoopState, pairs_path: str, output_dir: str) -> str:
|
||||
"""Render a minimal DPO ``soup.yaml`` referencing the harvested pairs."""
|
||||
import yaml
|
||||
|
||||
base = (
|
||||
state.served_model
|
||||
if state.served_model and not state.served_model.startswith("registry://")
|
||||
else "hf-internal-testing/tiny-random-gpt2"
|
||||
)
|
||||
config = {
|
||||
"base": base,
|
||||
"task": "dpo",
|
||||
"data": {"train": pairs_path, "format": "dpo"},
|
||||
"training": {"epochs": 1, "batch_size": 1, "lr": 5.0e-5},
|
||||
"output": output_dir,
|
||||
}
|
||||
return yaml.safe_dump(config, sort_keys=False)
|
||||
|
||||
|
||||
def train_dpo_from_pairs(
|
||||
state: LoopState, ctx: Mapping[str, object]
|
||||
) -> Mapping[str, object]:
|
||||
"""Train a DPO adapter from harvested pairs via a ``soup train`` subprocess.
|
||||
|
||||
Returns ``{run_id, skipped, adapter_path}``. Skips (no run) when the
|
||||
harvest produced no pairs. The subprocess is argv-list (no shell);
|
||||
``_TRAIN_RUNNER`` injects a stand-in for tests.
|
||||
"""
|
||||
pairs_path = ctx.get("pairs_path")
|
||||
pairs_n = int(ctx.get("pairs_harvested", 0) or 0)
|
||||
if not pairs_path or pairs_n <= 0:
|
||||
return {"run_id": None, "skipped": True, "adapter_path": None}
|
||||
|
||||
run_id = f"loop-train-{uuid.uuid4().hex[:8]}"
|
||||
output_dir = os.path.join(_ADAPTERS_DIR, run_id)
|
||||
yaml_dir = Path(_ADAPTERS_DIR)
|
||||
yaml_dir.mkdir(parents=True, exist_ok=True)
|
||||
yaml_path = str(yaml_dir / f"{run_id}.yaml")
|
||||
atomic_write_text(
|
||||
_render_dpo_yaml(state, str(pairs_path), output_dir),
|
||||
yaml_path,
|
||||
field="train yaml",
|
||||
)
|
||||
|
||||
argv = [
|
||||
sys.executable,
|
||||
"-m",
|
||||
"soup_cli.cli",
|
||||
"train",
|
||||
"--config",
|
||||
yaml_path,
|
||||
"--yes",
|
||||
]
|
||||
runner = _TRAIN_RUNNER
|
||||
if runner is None:
|
||||
import subprocess # noqa: S404 — argv list mode, no shell
|
||||
|
||||
runner = subprocess.run
|
||||
try:
|
||||
result = runner( # noqa: S603 — internal argv, no shell
|
||||
argv, capture_output=True, timeout=24 * 3600, check=False
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 — train failure must not crash loop
|
||||
_LOG.warning("train subprocess failed: %s", type(exc).__name__)
|
||||
return {"run_id": None, "skipped": True, "adapter_path": None}
|
||||
|
||||
rc = getattr(result, "returncode", 1)
|
||||
if rc != 0:
|
||||
_LOG.warning("soup train exited rc=%s", rc)
|
||||
return {"run_id": None, "skipped": True, "adapter_path": None}
|
||||
return {"run_id": run_id, "skipped": False, "adapter_path": output_dir}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gate — eval-gate against the registered baseline (v0.26.0 Part B)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_gate_generator(adapter_dir: str) -> Callable[[str], str]:
|
||||
"""Build a ``generate_fn`` for the trained adapter (live model load)."""
|
||||
if _GATE_GENERATE_FACTORY is not None:
|
||||
return _GATE_GENERATE_FACTORY(adapter_dir)
|
||||
# Live default — load the merged/trained adapter as a full model dir.
|
||||
from soup_cli.eval.quant_check import make_model_generator
|
||||
|
||||
return make_model_generator(adapter_dir)
|
||||
|
||||
|
||||
def gate_against_baseline(
|
||||
state: LoopState, ctx: Mapping[str, object]
|
||||
) -> Mapping[str, object]:
|
||||
"""Run the v0.26.0 eval gate for the trained adapter vs the baseline.
|
||||
|
||||
Returns ``{gate_verdict}`` — ``SKIPPED`` when training was skipped or the
|
||||
adapter dir is missing, otherwise ``OK`` / ``MAJOR`` from the gate result.
|
||||
"""
|
||||
if ctx.get("skipped"):
|
||||
return {"gate_verdict": "SKIPPED"}
|
||||
adapter_path = ctx.get("adapter_path")
|
||||
if not adapter_path or not os.path.isdir(str(adapter_path)):
|
||||
return {"gate_verdict": "SKIPPED"}
|
||||
|
||||
from soup_cli.eval.gate import load_suite, resolve_baseline, run_gate
|
||||
|
||||
try:
|
||||
suite = load_suite(state.eval_suite)
|
||||
except (FileNotFoundError, ValueError, TypeError, OSError) as exc:
|
||||
_LOG.warning("gate suite load failed: %s", type(exc).__name__)
|
||||
return {"gate_verdict": "SKIPPED"}
|
||||
# An unresolvable baseline degrades to a threshold-only gate (no
|
||||
# regression check) rather than failing the whole stage — the baseline
|
||||
# may not exist yet on the first iteration.
|
||||
try:
|
||||
baseline = resolve_baseline(state.baseline)
|
||||
except (FileNotFoundError, ValueError, TypeError, OSError):
|
||||
baseline = {}
|
||||
try:
|
||||
generate_fn = _build_gate_generator(str(adapter_path))
|
||||
result = run_gate(suite, generate_fn=generate_fn, baseline=baseline)
|
||||
except (FileNotFoundError, ValueError, TypeError, OSError) as exc:
|
||||
_LOG.warning("gate run failed: %s", type(exc).__name__)
|
||||
return {"gate_verdict": "SKIPPED"}
|
||||
verdict = "OK" if result.passed else "MAJOR"
|
||||
return {"gate_verdict": verdict}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deploy — promote to a canary via /v1/adapters/activate (v0.30.0)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Adapter-name shape accepted by the v0.30.0 /v1/adapters/activate route.
|
||||
_ADAPTER_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._\-]*$")
|
||||
|
||||
_LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"})
|
||||
|
||||
|
||||
def _endpoint_is_local(endpoint: str) -> bool:
|
||||
"""True iff ``endpoint``'s host is loopback or a private/link-local IP.
|
||||
|
||||
The deploy surface must only POST to the operator's own box / LAN, never
|
||||
an arbitrary public host (the webhook validator permits any HTTPS host).
|
||||
Loopback hostnames are accepted by name; any other host must parse as a
|
||||
private / loopback / link-local IP literal. A non-IP public hostname is
|
||||
rejected (we do not resolve DNS — project policy).
|
||||
"""
|
||||
import ipaddress
|
||||
from urllib.parse import urlparse
|
||||
|
||||
host = (urlparse(endpoint).hostname or "").strip("[]").lower()
|
||||
if not host:
|
||||
return False
|
||||
if host in _LOOPBACK_HOSTS:
|
||||
return True
|
||||
try:
|
||||
ip = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
return False
|
||||
return ip.is_loopback or ip.is_private or ip.is_link_local
|
||||
|
||||
|
||||
def _post_activate(endpoint: str, name: str) -> bool:
|
||||
"""POST to ``<endpoint>/v1/adapters/activate/<name>``; True on 2xx.
|
||||
|
||||
``endpoint`` is already SSRF-validated by the caller; ``name`` is
|
||||
re-validated here against the activate-route pattern before URL
|
||||
interpolation (defence-in-depth against a crafted adapter dir name).
|
||||
"""
|
||||
if _DEPLOY_POSTER is not None:
|
||||
return _DEPLOY_POSTER(endpoint, name)
|
||||
if not _ADAPTER_NAME_RE.match(name):
|
||||
return False
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False
|
||||
url = endpoint.rstrip("/") + f"/v1/adapters/activate/{name}"
|
||||
try:
|
||||
resp = httpx.post(url, timeout=5.0)
|
||||
return 200 <= resp.status_code < 300
|
||||
except Exception: # noqa: BLE001 — deploy must never crash the loop
|
||||
return False
|
||||
|
||||
|
||||
def deploy_to_canary(
|
||||
state: LoopState, ctx: Mapping[str, object]
|
||||
) -> Mapping[str, object]:
|
||||
"""Promote the trained adapter as a canary when the gate verdict is OK.
|
||||
|
||||
Returns ``{deployed, canary_verdict}``. No-op (not deployed) unless the
|
||||
gate passed, an adapter is available, and a serve endpoint is configured
|
||||
via ``SOUP_LOOP_SERVE_ENDPOINT``. The endpoint is SSRF-validated
|
||||
(loopback-only HTTP, scheme allowlist, private-IP rejection) via the
|
||||
shared ``validate_webhook_url`` helper before any POST.
|
||||
"""
|
||||
if ctx.get("gate_verdict") != "OK":
|
||||
return {"deployed": False, "canary_verdict": None}
|
||||
adapter_path = ctx.get("adapter_path")
|
||||
if not adapter_path:
|
||||
return {"deployed": False, "canary_verdict": None}
|
||||
raw_endpoint = os.environ.get("SOUP_LOOP_SERVE_ENDPOINT")
|
||||
if not raw_endpoint:
|
||||
return {
|
||||
"deployed": False,
|
||||
"canary_verdict": None,
|
||||
"notes": "no SOUP_LOOP_SERVE_ENDPOINT configured",
|
||||
}
|
||||
from soup_cli.utils.drift_alarm import validate_webhook_url
|
||||
|
||||
try:
|
||||
endpoint = validate_webhook_url(raw_endpoint)
|
||||
except (TypeError, ValueError):
|
||||
_LOG.warning("SOUP_LOOP_SERVE_ENDPOINT rejected by SSRF guard")
|
||||
return {
|
||||
"deployed": False,
|
||||
"canary_verdict": None,
|
||||
"notes": "SOUP_LOOP_SERVE_ENDPOINT rejected (SSRF guard)",
|
||||
}
|
||||
# A serve endpoint is the operator's own box / LAN, never an arbitrary
|
||||
# remote host. `validate_webhook_url` (a *webhook* policy) permits any
|
||||
# HTTPS host; tighten further for the deploy surface to loopback + RFC1918
|
||||
# private + link-local only, so a poisoned SOUP_LOOP_SERVE_ENDPOINT cannot
|
||||
# POST adapter names to an attacker-controlled HTTPS server (v0.71.4
|
||||
# review MEDIUM-4). Non-IP hostnames can't be verified private without DNS
|
||||
# (project policy: no DNS resolution), so the safe default rejects them.
|
||||
if not _endpoint_is_local(endpoint):
|
||||
_LOG.warning("SOUP_LOOP_SERVE_ENDPOINT must be loopback/LAN (SSRF guard)")
|
||||
return {
|
||||
"deployed": False,
|
||||
"canary_verdict": None,
|
||||
"notes": "SOUP_LOOP_SERVE_ENDPOINT must be loopback/LAN (SSRF guard)",
|
||||
}
|
||||
name = os.path.basename(str(adapter_path).rstrip("/\\")) or "canary"
|
||||
ok = _post_activate(endpoint, name)
|
||||
return {
|
||||
"deployed": ok,
|
||||
"canary_verdict": "OK" if ok else None,
|
||||
"notes": "" if ok else "activate POST failed",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cost — placeholder estimate (v0.34.0 run_cost wiring is a follow-up)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def estimate_cost(state: LoopState) -> float:
|
||||
"""Per-iteration cost estimate (USD) — currently a hard ``0.0`` placeholder.
|
||||
|
||||
KNOWN LIMITATION: returning ``0.0`` means the v0.58 budget gate
|
||||
(``monthly_budget_usd`` / ``max_runs_per_day`` in ``run_once``) never trips
|
||||
on *cost* for a pre-wired loop — only the daily run-count cap is effective.
|
||||
Wiring a real estimate (v0.34.0 ``utils/run_cost.estimate_run_cost_usd``,
|
||||
which needs the trained run's GPU + duration) is a tracked follow-up; until
|
||||
then operators relying on a dollar budget should set ``max_runs_per_day``.
|
||||
"""
|
||||
return 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config builder
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_prewired_watch_config(
|
||||
*,
|
||||
max_iterations: Optional[int] = None,
|
||||
poll_interval_sec: float = 60.0,
|
||||
state_path: Optional[str] = None,
|
||||
iteration_dir: Optional[str] = None,
|
||||
pack_iterations: bool = False,
|
||||
served_model: Optional[str] = None,
|
||||
base_model: str = "unknown",
|
||||
) -> "WatchConfig":
|
||||
"""Return a ``WatchConfig`` wired with the pre-wired production stages."""
|
||||
from soup_cli.utils.loop_daemon import WatchConfig
|
||||
|
||||
return WatchConfig(
|
||||
poll_interval_sec=poll_interval_sec,
|
||||
max_iterations=max_iterations,
|
||||
state_path=state_path,
|
||||
iteration_dir=iteration_dir,
|
||||
harvest_fn=harvest_from_traces,
|
||||
train_fn=train_dpo_from_pairs,
|
||||
gate_fn=gate_against_baseline,
|
||||
deploy_fn=deploy_to_canary,
|
||||
cost_fn=estimate_cost,
|
||||
pack_iterations=pack_iterations,
|
||||
served_model=served_model,
|
||||
base_model=base_model,
|
||||
)
|
||||
|
|
@ -51,6 +51,9 @@ class LoopState:
|
|||
eval_suite: str
|
||||
baseline: str
|
||||
status: str = "stopped"
|
||||
# v0.71.4 #176 — use the pre-wired harvest/train/gate/deploy stages
|
||||
# (utils/loop_stages.py) instead of the no-op default callbacks.
|
||||
pre_wired: bool = False
|
||||
traces_collected: int = 0
|
||||
pairs_distilled: int = 0
|
||||
runs_gated: int = 0
|
||||
|
|
@ -110,6 +113,8 @@ class LoopState:
|
|||
raise ValueError("max_runs_per_day must be a positive int or None")
|
||||
if not isinstance(self.canary_autoroll_on_regress, bool):
|
||||
raise ValueError("canary_autoroll_on_regress must be bool")
|
||||
if not isinstance(self.pre_wired, bool):
|
||||
raise ValueError("pre_wired must be bool")
|
||||
if self.canary_active is not None:
|
||||
_require_str("canary_active", self.canary_active, allow_empty=False)
|
||||
if self.last_iteration_id is not None:
|
||||
|
|
@ -273,6 +278,7 @@ def init_state(
|
|||
*,
|
||||
monthly_budget_usd: Optional[float] = None,
|
||||
max_runs_per_day: Optional[int] = None,
|
||||
pre_wired: bool = False,
|
||||
path: Optional[str] = None,
|
||||
force: bool = False,
|
||||
) -> Tuple[LoopState, str]:
|
||||
|
|
@ -300,6 +306,7 @@ def init_state(
|
|||
eval_suite=eval_suite,
|
||||
baseline=baseline,
|
||||
status="stopped",
|
||||
pre_wired=pre_wired,
|
||||
monthly_budget_usd=monthly_budget_usd,
|
||||
max_runs_per_day=max_runs_per_day,
|
||||
created_at=now,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue