Commit Graph

8 Commits

Author SHA1 Message Date
Alpamys a2287dd6a5 feat(rag): RAFT span-mask trainer + RA-DIT auto-link + live steering + eval citation (v0.71.10)
Lifts the v0.62.0 RAG-family schema-only stubs to live, validated on SmolLM2-135M:

- #199 RAFT: data.format=raft trains answer-only (prompt span masked to -100,
  [doc-N] citation ids, deterministic doc shuffle by raft_shuffle_seed); rows
  whose prompt fills max_length are dropped with a warning. New utils/raft.py +
  trainer/raft.py (RaftDataCollator + weighted-CE _RaftTrainer).
- #200 soup ra-dit: one-shot two-stage orchestrator (train retriever -> record
  it as the generator's paired retriever -> train generator); a generator-stage
  `soup train` with no retriever set auto-links the latest RA-DIT retriever from
  the Registry. New utils/ra_dit_run.py + commands/ra_dit.py.
- #201 soup steer train/apply + soup serve --steer: live CAA/ITI/RepE fit from
  {positive, negative} pairs + decode-time forward hook (transformers backend).
  Lifts the steering.py apply_steering/build_steering_vector stubs.
- #202 soup eval citation + citation-span per-token loss boost + 7th `citation`
  failure mode in soup diagnose. New commands/_eval_v07110.py +
  diagnose/citation.py.

Review fixes (3 agents, all CRITICAL->LOW): markup-escaped autolink advisory;
shared enforce_under_cwd_and_no_symlink + O_NOFOLLOW on every new file read;
steering-artifact containment; honest RA-DIT docs (records pairing, no weight
fusion); public validate_ra_dit_config_path + render_raft_prompt; repe/iti
require >=2 pairs; eval citation --shuffle-seed.

Full suite: 13034 passed, 106 skipped (13142 collected). ruff clean.
2026-06-03 19:28:45 +05:00
Alpamys 06d8ea7cc5 chore: migrate to src-layout
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.

- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
  The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
  and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
  CODEOWNERS patterns, and the PR-template checklist all repointed to
  src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
  path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
  over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
  references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
  the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.

Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
2026-05-31 12:40:06 +05:00
Alpamys 47409df730 fix(tests): v0.60.0 CI green — widen version floors + strip ANSI from merge help assert
Three failures on CI run 26084542388 — all are version-pin / Rich-wrap
artefacts, not real regressions in v0.60.0 functionality:

- test_v0560 test_pyproject_version: regex-based >=0.56 floor check
  (was substring `version = "0.5`)
- test_v0590 test_version_is_0_59 -> test_version_is_at_least_0_59:
  >=0.59 floor (matches v0.51/v0.54 floor-check idiom)
- test_v0600_part_e merge_help_lists_license_flags: strip ANSI codes
  before substring check (Rich splits `--license` across `\x1b[1;36m`
  escapes in the wrapped Typer table)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 13:15:28 +05:00
Alpamys 7da82d355a feat(adapters): v0.57.0 — git for LoRA (diff, merge, blame, branch)
Ships "soup adapters {diff,merge,blame,branch,checkout,branches}" — git-shaped
UX for LoRA adapter management. Pure-numpy math (no torch); TOCTOU defence on
every path; atomic writes throughout.

Part A — adapters diff (utils/adapter_diff.py, ~270 LOC):
  Per-layer ΔW Frobenius norm + relative drift; effective-rank delta via
  SVD entropy; top-K changed projections; JSON/Markdown/table output.

Part B — adapters merge (utils/adapter_merge.py, ~280 LOC):
  Four strategies — linear (weighted avg), ties (Yadav et al. — trim/
  elect-sign/disjoint), dare (Yu et al. — drop+rescale, deterministic),
  svd (low-rank reconstruction). MergeReport.verdict='UNKNOWN' is a stub;
  live canary verdict via v0.55 eval gate ships in v0.57.1.

Part C — adapters blame (utils/blame.py, ~190 LOC):
  Leave-one-out plan emitter + budget tracker (parse_budget mirrors v0.48.0
  data_mix idiom). Per-shard work table + feasibility check. Live ablation
  runner raises NotImplementedError v0.57.1 (mirrors v0.27.0 / v0.50.0 /
  v0.56.0 stub-then-live pattern).

Part D — adapters branch / checkout / branches (utils/adapter_branch.py,
~230 LOC):
  SHA-256 snapshot pointers under ~/.soup/branches/ (SOUP_BRANCHES_DIR
  override, $HOME/$CWD/$TMPDIR-bounded). Drift detection on checkout —
  refuses restore when source SHA != snapshot SHA. CRLF/null-byte
  rejection on env override (mirrors v0.51.0 hub-endpoint policy).

5-agent review-fix wave (1 CRITICAL + 9 HIGH + 11 MEDIUM + 4 LOW):
  - TIES tied-sign defaults to +1 (np.sign(0)==0 would silently zero all
    tied parameters)
  - load_branch / delete_branch reject symlinks via os.lstat + S_ISLNK
    before read/unlink
  - merge output safetensors + adapter_config.json atomic writes with
    symlink target rejection at output path; source config size-capped
    at 256 KB
  - compute_adapter_diff weights-file path symlink-rejected via lstat
    BEFORE is_file() (defends against .safetensors -> /etc/passwd escape)
  - _count_dataset_rows opens via realpath captured at containment check
    (closes TOCTOU window)
  - diff --output write is atomic (tempfile + os.replace)
  - SOUP_BRANCHES_DIR rejects every C0 control char, not just null
  - SUPPORTED_STRATEGIES is now frozenset (matches v0.41.0+ allowlist
    policy); STRATEGY_ORDER tuple preserved for canonical iteration
  - 5× pytest.raises(Exception) tightened to FrozenInstanceError
  - 2 zero-assertion Part D tests converted to real assertions
  - Added: bool base_model rejection, top_k boundary 1/201, density=1.0
    inclusive bound, inf weight rejection, tied-sign positive default,
    bool False for num_shards/budget_seconds, POSIX symlink rejections
    for diff weights / merge output / load_branch / delete_branch,
    no-top-level-torch source-grep guards, traversal delete_branch.

Plus v0.56.0 follow-up: test_v0560.py version-floor tests widened from
exact-match to floor-check (matches v0.51.0 / v0.54.0 idiom — every
subsequent release would otherwise edit this one line).

Test count: 8849 → 8998 (+149 net in 4 new files; 4 POSIX-only symlink
tests skipped on Windows). Full suite green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 18:37:14 +05:00
Alpamys a3810823d1 refactor(train): harden diagnose-gate rank guard from PR #169
PR #169 wired LOCAL_RANK==0 guard on _run_diagnose_gate so distributed
launches only run the gate on one worker per machine. Two minor polish
items on top of the merged version:

- Wrap the int() parse in try/except ValueError. A malformed LOCAL_RANK
  (garbage value from a misconfigured launcher) would previously crash
  the post-training gate. Falling back to True is safer than silently
  skipping the gate -- over-running is recoverable, under-running hides
  failures.
- Expand the docstring to explain why we use LOCAL_RANK (per-machine)
  rather than RANK (global): the gate reads the local output_dir, so
  one gate per machine is the right granularity for typical single-
  machine multi-GPU runs. Documents the choice for future readers.
- Add a focused test (test_diagnose_gate_handles_malformed_local_rank)
  asserting the safe fallback path.
2026-05-15 17:29:26 +05:00
Yixuan Xu 4c2a578ac0
Guard diagnose gate on distributed worker ranks (#169)
Co-authored-by: mzl2233 <mzl2233@users.noreply.github.com>
2026-05-15 17:27:54 +05:00
Alpamys 7d81496c69 test(diagnose): strip ANSI before help-output substring asserts (v0.56.0 CI fix)
Rich's CliRunner output on CI carries ANSI escape codes that split long
option names like `--badge` and `--diagnose-gate` across colour-reset
boundaries (`-\x1b[0m\x1b[1;36m-badge`), breaking naive `"--badge" in
result.output` substring checks. Same fix pattern as v0.55.0 CI hotfix.

Failures: tests/test_v0560.py::TestCli::test_diagnose_help and
TestTrainDiagnoseGate::test_help_lists_flag on all 9 CI matrix cells.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 15:16:54 +05:00
Alpamys 72189baba7 feat(diagnose): soup diagnose — post-training model report card (v0.56.0)
Six failure-mode probes (forgetting / refusal / format / mode_collapse /
memorization / contamination) + FailureReport frozen dataclass + SVG
badge + soup train --diagnose-gate. Same OK/MINOR/MAJOR taxonomy as
v0.26.0 Quant-Lobotomy.

- soup diagnose <run-id> [--evidence|--output|--badge|--attach-to-registry]
- soup train --diagnose-gate <evidence.json> refuses MAJOR runs
- diagnose_report added to registry._VALID_KINDS

Review wave (4 agents): 4 HIGH + 8 MEDIUM + 2 LOW addressed —
atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), realpath
containment, evidence size cap, contamination combined-complexity cap,
ReDoS probe, extras null-byte sanitisation, extract_row_text
centralisation, tokenize delegates to _eval_text.

Test count: 8676 -> 8849 (+123 in test_v0560.py + 50 net adjustments).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 15:03:00 +05:00