soup/soup_cli
Alpamys 148cb0c125 fix(active-sample): variance-based diversity score for K>2 reward models (#206)
v0.63.0 `score_uncertainty` raised on K>2 and `_row_uncertainty` fell back
to a monotone-broken `max(scores) - min(scores)`. Now generalises to K<=32
via population variance scaled by 4 — adding a fresh RM score equal to the
running mean strictly decreases uncertainty (the new contribution to the
sum-of-squares is zero while the denominator grows), so consensus on
redundant evidence can never spike the score.

K=1 max-entropy and K=2 disagreement formulas preserved verbatim (existing
operator dashboards depend on the |s1 - s2| value). Cap stays at K=32 for
DoS defence. _row_uncertainty K>2 path now routes through score_uncertainty
inside an isolated try/except — bad rows return 0.0 instead of crashing the
batch.

PEP 585 modernisation: collections.abc imports + list[...] annotations
(safe because `from __future__ import annotations` is in scope).
math.fsum used for the variance accumulation to keep rounding error
sub-ULP at K=32.

Tests: +32 net (25 in new tests/test_v0631_206.py + 7 TDD review-fix
followups). Full suite 11941 -> 11973 pass.

Closes #206.
2026-05-26 17:50:30 +05:00
..
autopilot feat(v0.53.1): Quant Menu II + Export pipeline live 2026-05-13 00:16:03 +05:00
cans fix(v0.33.0): review-wave findings (CRITICAL + HIGH + MEDIUM + LOW) 2026-04-27 19:57:57 +05:00
commands feat(brain-rot): per-language low-effort + clickbait bundles for es/fr/de/ru (#234) 2026-05-26 16:43:17 +05:00
config feat(echo-trap): add tokenizer-aware repetition scoring (#242) 2026-05-25 19:46:51 +05:00
data feat(v0.65.0): Eval Depth — judge calibration + behaviour battery + capability suite + CheckList DSL + IRT subset 2026-05-21 15:35:47 +05:00
eval feat(v0.65.0): Eval Depth — judge calibration + behaviour battery + capability suite + CheckList DSL + IRT subset 2026-05-21 15:35:47 +05:00
experiment feat(eval): soup eval design — derive evals from data (v0.55.0) 2026-05-15 12:44:14 +05:00
migrate feat: v0.21.0 — migrate, recipes, NEFTune, rsLoRA 2026-04-02 14:08:36 +05:00
monitoring feat(v0.53.11): GRPO Plus finish + preference live 2026-05-14 15:40:34 +05:00
plugins feat(v0.45.0): Plugin System & Ecosystem Wins — 5 Parts, +169 tests 2026-05-10 21:21:04 +05:00
recipes feat(v0.62.0): RAG & Activation Steering — RAFT + RA-DIT + soup steer + citation-faithful + GRACE codebook 2026-05-20 20:20:30 +05:00
registry feat(v0.62.0): RAG & Activation Steering — RAFT + RA-DIT + soup steer + citation-faithful + GRACE codebook 2026-05-20 20:20:30 +05:00
templates feat(v0.44.0): Live Dashboard & UX - 21 features, +192 tests 2026-05-10 19:20:18 +05:00
trainer feat(unlearn/edit): v0.61.0 — Unlearning & Knowledge Edit (NPO/SimNPO/RMU + ROME/MEMIT/AlphaEdit) 2026-05-19 15:16:15 +05:00
ui feat(v0.53.10): Quick wins + packaging + UX wiring 2026-05-14 13:58:20 +05:00
utils fix(active-sample): variance-based diversity score for K>2 reward models (#206) 2026-05-26 17:50:30 +05:00
__init__.py feat(v0.70.0): Loop Hardening — reward-hacking + ULD + MiniLLM + RL ckpt + iterative DPO + echo-trap 2026-05-25 17:17:03 +05:00
__main__.py Add Phase 3.1: friendly errors, soup doctor, soup quickstart, UX polish (v0.3.1) 2026-03-05 19:10:36 +05:00
cli.py feat(v0.70.0): Loop Hardening — reward-hacking + ULD + MiniLLM + RL ckpt + iterative DPO + echo-trap 2026-05-25 17:17:03 +05:00
tui_app.py feat(observability): v0.34.0 — Observability & Dev UX (7 Parts) 2026-04-28 13:14:49 +05:00