mirror of https://github.com/razor-ai/soup.git
5 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Alpamys
|
a2287dd6a5 |
feat(rag): RAFT span-mask trainer + RA-DIT auto-link + live steering + eval citation (v0.71.10)
Lifts the v0.62.0 RAG-family schema-only stubs to live, validated on SmolLM2-135M: - #199 RAFT: data.format=raft trains answer-only (prompt span masked to -100, [doc-N] citation ids, deterministic doc shuffle by raft_shuffle_seed); rows whose prompt fills max_length are dropped with a warning. New utils/raft.py + trainer/raft.py (RaftDataCollator + weighted-CE _RaftTrainer). - #200 soup ra-dit: one-shot two-stage orchestrator (train retriever -> record it as the generator's paired retriever -> train generator); a generator-stage `soup train` with no retriever set auto-links the latest RA-DIT retriever from the Registry. New utils/ra_dit_run.py + commands/ra_dit.py. - #201 soup steer train/apply + soup serve --steer: live CAA/ITI/RepE fit from {positive, negative} pairs + decode-time forward hook (transformers backend). Lifts the steering.py apply_steering/build_steering_vector stubs. - #202 soup eval citation + citation-span per-token loss boost + 7th `citation` failure mode in soup diagnose. New commands/_eval_v07110.py + diagnose/citation.py. Review fixes (3 agents, all CRITICAL->LOW): markup-escaped autolink advisory; shared enforce_under_cwd_and_no_symlink + O_NOFOLLOW on every new file read; steering-artifact containment; honest RA-DIT docs (records pairing, no weight fusion); public validate_ra_dit_config_path + render_raft_prompt; repe/iti require >=2 pairs; eval citation --shuffle-seed. Full suite: 13034 passed, 106 skipped (13142 collected). ruff clean. |
|
|
Alpamys
|
96c339f184 |
feat(edit): live ROME/MEMIT/AlphaEdit + GRACE + NPO/SimNPO/RMU unlearn (v0.71.9)
Closes #193, #194, #196, #197, #203. - #194 utils/edit_kernels.py: covariance-free rank-1 ROME/MEMIT/AlphaEdit; apply_edit live (load -> optimise residual -> rank-1 update -> save); edit diff live before/after generation. - #196 EditGovernorStore SQLite persistence + cross-process lock. - #197 apply_edit consults the governor (check_can_edit before, record after). - #203 GraceCodebook + apply_grace_edit + install_grace_hook + Registry kinds. - #193 utils/unlearn_kernels.py (NPO/SimNPO/RMU) + live UnlearnTrainerWrapper + soup train --task unlearn. Validated on SmolLM2-135M: ROME 0.0016->0.96, NPO/SimNPO forget loss down. +81 tests (tests/test_v0719.py). 2 review waves, all findings fixed. |
|
|
Alpamys
|
06d8ea7cc5 |
chore: migrate to src-layout
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.
- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
CODEOWNERS patterns, and the PR-template checklist all repointed to
src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.
Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
|
|
|
Alpamys
|
40bd6251a2 |
feat(v0.63.0): Production Trace Ecosystem — soup ingest + prune-prompt + active-sample + ab + drift-alarm
5 new top-level commands close axis 7 of the roadmap. Every Part LIVE on
day one (no deferred stubs):
- soup ingest: universal trace importer (Langfuse / LangSmith / Helicone /
OpenPipe / OTel / OpenAI Stored Completions). 6 adapters
+ frozen TraceRecord with MappingProxyType-wrapped metadata.
Zero credential-handling threat surface — Soup parses the
JSONL export, never makes the SaaS network call.
- soup prune-prompt: detect + strip a shared system-prompt prefix so the
FT model internalises it (OpenPipe's signature trick,
OSS). Binary-search over up to 32 templates finds the
longest threshold-meeting prefix.
- soup data active-sample: surface top-uncertainty prod traces for human
review. Max-entropy on single rm_score or
pairwise disagreement on dual rm_scores.
- soup ab: Wald sequential SPRT for the point alternative. LLR is a
martingale under H0 so Type-I error is controlled at every
stopping time per the optional stopping theorem.
- soup drift-alarm: rolling KL on whitespace-tokenised output distribution
+ SSRF-hardened Slack/Discord webhook (full parity with
v0.51.0 validate_hub_endpoint). Exit 3 on drift for
cron-friendly automation.
Test count: 9816 -> 10035 (+219 net across 6 new test files).
Review-fix coverage (code-reviewer + tdd-guide returned actionable;
python-reviewer + security-reviewer agents context-thrashed on the large
CLAUDE.md release-notes history — matches the v0.58.0 / v0.59.0 / v0.60.0
/ v0.61.0 / v0.62.0 idiom; verified manually):
- 1 CRITICAL: mSPRT log-likelihood-ratio sign error drove Type-I error
to 1.0 as n grew. Replaced with Wald's classic point-
alternative SPRT (martingale under H0).
- 2 HIGH: detect_common_prefix early-exit on 100% match returned the
shortest qualifying prefix instead of the longest;
_MAX_SCAN_ROWS DoS cap used 'pass' instead of 'break'.
- 3 MEDIUM: TraceRecord.metadata now MappingProxyType-wrapped post-init
(frozen-dataclass mutation hazard); _AUTH_ENV table
deduplicated; drift_alarm precedence parens on SSRF gate.
- 2 LOW: pooled_se dead-branch refactor; mean_uncertainty NaN guard.
- 8 follow-up tests: msprt zero-variance, partial-majority binary-search
activation, score_uncertainty exact boundaries, rolling_kl identical
+ disjoint, validate_budget + validate_threshold exact endpoints,
_signal_from_thumbs boundaries, no-heavy-top-level-imports source-grep
guard across all 5 new util modules.
Step 6 smoke verified for all 5 commands + 6 failure-mode rejection
paths.
CRLF gotcha note for future maintainers: PowerShell wrote the smoke
fixtures with a UTF-8 BOM on Windows during Step 6 — switched to
inline Python for the fixture write. Production CLI input handling is
already BOM-tolerant (utf-8-sig in JSONL loaders via v0.40.1 Part E).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
|
Alpamys
|
0d6f95181a |
feat(v0.62.0): RAG & Activation Steering — RAFT + RA-DIT + soup steer + citation-faithful + GRACE codebook
5 Parts shipping wedge 14 of the roadmap (RAG-aware fine-tuning +
activation steering + lifelong edit codebook). Schema-only release;
live training loops + decode-hook intervention + codebook lookup all
land in v0.62.1 (mirrors v0.50.0 / v0.52.0 / v0.61.0 stub-then-live).
Part A — RAFT data format: new data.format='raft' schema +
_convert_raft validator (64 KiB per-field cap, 64-distractor cap,
null-byte rejection on every field) + raft-llama3-8b recipe.
Part B — RA-DIT two-stage: TrainingConfig.ra_dit_stage Literal
{retriever, generator} + ra_dit_retriever_model field + closed
allowlist + cross-validator enforcing stage to base-task pairing
(retriever to embedding, generator to sft) + 2 recipes.
Part C — soup steer (CAA / ITI / RepE): closed-allowlist control-vector
methods + validate_steering_method/name/strength + Typer subcommands
train/apply/list + soup serve --steer/--steer-strength flags +
steering_vector Registry artifact kind. apply_steering +
build_steering_vector deferred-live stubs raise NotImplementedError
with v0.62.1 marker after validating inputs.
Part D — Citation-faithful FT: score_citations precision/recall/F1
kernel + extract_citation_ids public API + citation_faithful /
citation_style / citation_recall_threshold schema. Cross-validator:
citation_faithful=true requires data.format='raft' AND task in
{sft, pretrain} (silent-no-op footgun rejection mirroring v0.52.0
distill / classifier task-gate policy).
Part E — GRACE codebook: GraceCodebookConfig + bounded size [1, 100k]
+ bounded dim [1, 16384]. Extends v0.61.0 SUPPORTED_EDIT_METHODS
allowlist with 'grace'; apply_edit routes grace plans to v0.62.1
marker while legacy rome/memit/alphaedit retain v0.61.1 marker
(regression-guarded via TestEditMarkerRegressionGuard).
Test count: 9571 -> 9786 (+215 net). 4 review-agent waves resolved
0 CRITICAL + 0 HIGH + 4 MEDIUM + 11 LOW (broken list_steers registry
context-manager + dict-key access; missing version bump;
citation_faithful task-gate; shared TOCTOU helper delegation; Rich
markup escape on --steer exception messages; --base length cap +
null-byte rejection; typing.Iterable -> collections.abc.Iterable
migration; except Exception -> except ImportError narrowing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|