Alpamys
|
823456c1a5
|
feat(probe): real probe weights, SAE auto-download, truth/harm, interference --measure, capture-activations (v0.71.8)
Closes #216, #217, #218, #219. Partial #215 (calibrated vectors upstream-gated).
- #215 probe_kernel.py: compute_contrast_probe + load_probe_weights
(.npz/.npy/.safetensors, O_NOFOLLOW, allow_pickle=False, cwd-contained);
soup probe sleeper --weights. Synthetic seed fallback retained.
- #216 hubs.snapshot_download (SSRF-hardened, home/cwd/tmp cache, TOFU gate)
+ sae_diff.download_sae (allowlist-before-network + symlink-escape guard);
soup probe sae-diff --auto-download.
- #217 truth_probe.py + harm_probe.py over probe_kernel; soup probe truth/harm;
probe pack ships truth+harm per base.
- #218 interference_live.measure_interference_losses (live PEFT multi-adapter,
add_weighted_adapter cat off-diagonal); soup probe interference --measure.
- #219 live_eval.extract_layer_activations + resolve_layer_module PEFT-fallback;
soup train --capture-activations writes <output>/activations/activations.json.
Test count 12771 -> 12917 (+146 in tests/test_v0718.py). Step-6 smoke on
SmolLM2-135M (RTX 3050) green; caught + fixed a PEFT-wrapper layer-resolution bug.
|
2026-06-03 15:00:27 +05:00 |
|
Alpamys
|
f097528ac0
|
feat(eval): live eval runners — advise/tunability/capability/behavior/diagnose (v0.71.7)
Closes #161, #162, #208, #211, #212, #165.
New utils/live_eval.py shared model-loading layer (lazy torch/transformers/peft):
load_model_and_tokenizer, make_generator/make_multi_generator, compute_eval_loss,
lora_probe, measure_logit_agreement, token_f1.
- #161 soup advise --probe-model: live zero/few-shot token-F1 + LoRA probe
- #162 base_model_proximity via held-out logit agreement
- #208 soup tunability --live: per-candidate LoRA probe
- #211 soup eval capability --live --model: lm-eval-harness per task (per-task isolation)
- #212 soup eval behavior --base-model: live pre/post battery diff
- #165 soup diagnose --base-model: utils/diagnose/live.py runs all 6 probes live
Heuristic/neutral paths preserved when no model is supplied. Both new JSONL
readers open with O_NOFOLLOW after cwd-containment (TOCTOU close). +68 tests
(12703 -> 12771). Smoked end-to-end on SmolLM2-135M (RTX 3050).
|
2026-06-03 00:05:36 +05:00 |