# v0.53.x QA log

Manual smoke-test results for paths CI cannot reach (no GPU on CI runners,
no llama.cpp build). Each entry records command + observed result + date.

## v0.53.1

### #70 — GGUF export (q4_0 / q4_k_m / q5_k_m / q8_0 / f16 / f32)

**Status:** PENDING — requires a GPU box + built llama.cpp.

Planned smoke (run on a 4 GB+ VRAM box with TinyLlama-1.1B-LoRA in `./adapter`):

```bash
soup merge -a ./adapter -o ./merged
for q in q4_0 q4_k_m q5_k_m q8_0 f16 f32; do
  soup export --model ./merged --format gguf --quant "$q" \
    --output "./out/tinyllama.${q}.gguf"
done
```

Acceptance criteria:
- All six files exist with non-zero size.
- File sizes increase monotonically: q4_0 < q4_k_m < q5_k_m < q8_0 < f16 < f32.
- `llama-cli` can load each file and emit at least one token.

When run, record sizes + first-token-time below.

### #72 — AWQ + GPTQ export (`--bits 4`)

**Status:** PENDING — requires CUDA box with `autoawq` + `auto-gptq` extras.

Planned smoke:

```bash
soup export --model ./merged --format awq  --bits 4 --output ./out/awq
soup export --model ./merged --format gptq --bits 4 --output ./out/gptq
```

Acceptance: both output directories contain a `config.json` and weight
shards; loading each via `transformers.AutoModelForCausalLM` returns a
quantized model whose `state_dict` reports the expected dtype.

### #82 — Autopilot pre-quantized detection

**Status:** PASS — automated regression coverage in `tests/test_v0531_82.py`.

Notes:
- `detect_prequantized_format("TheBloke/Llama-2-7B-Chat-GPTQ")` → `"gptq"`.
- `detect_prequantized_format("clean/name", {"quantization_config": {"quant_method": "awq"}})` → `"awq"`.
- `decide_quantization(model_params_b=7.0, vram_gb=80.0, prequantized="gptq")` → `"gptq"` (takes precedence over the VRAM heuristic).

### #109 — `soup deploy autopilot --measure`

**Status:** PASS (orchestrator + cache); LIVE-EVAL PENDING (needs GPU + model loaders).

Coverage:
- Cache key + sha-of-file: `tests/test_v0531_109.py::TestComputeCacheKey`.
- Measurement loop + cache round-trip: `tests/test_v0531_109.py::TestRunMeasure`.
- CLI plumbing with injected generators: `TestDeployAutopilotMeasureCLI`.

The injected-generator escape hatch (`_DEPLOY_MEASURE_BEFORE_GEN`,
`_DEPLOY_MEASURE_AFTER_FACTORY` module attrs on
`soup_cli.utils.deploy_measure`) is the production hook for live model
loading in v0.53.x. Real transformers / vLLM generators land alongside
v0.53.2 deferred trainer wiring.

### #139 — Advanced GGUF export via llama.cpp imatrix

**Status:** PASS (orchestrator + subprocess argv shape + UD prefix
stripping); LIVE-RUN PENDING (needs llama.cpp build).

Coverage:
- `_run_imatrix` + `_run_quantize_binary` argv lists (no shell) verified.
- UD-prefix-strip: `UD-Q4_K_XL` → llama.cpp arg `Q4_K_XL`.
- Calibration JSONL → plain-text conversion + 4096-row safety cap.
- Apple/ARM flavours (Q4_0_4_4 etc.) skip the imatrix stage; UD ladder
  + low-bit IQ family require `--calibration-data`.

Planned manual smoke when a llama.cpp build is available:

```bash
soup export --model ./merged --format gguf-ud \
  --gguf-flavour UD-Q4_K_XL \
  --calibration-data ./calib.jsonl \
  --llama-cpp ~/.soup/llama.cpp \
  --output ./out/tinyllama.UD-Q4_K_XL.gguf
```

### #142 — Merge `--save-format 4bit` + export `--format torchao`

**Status:** PASS (orchestrator + validators + path TOCTOU); LIVE-EVAL
PENDING (needs CUDA + bitsandbytes / torchao).

Coverage:
- `merge_4bit` happy path with mocked transformers + BNB.
- `export_torchao` happy path with mocked torchao + transformers.
- `--save-format weird` rejected at CLI dispatch with exit code 2.
- `--format torchao` without `--quant-config` rejected.
- `--quant-config` outside cwd rejected.
- `load_quant_config` enforces yaml.safe_load + 256 KB cap +
  extension allowlist + symlink rejection (POSIX).

Planned manual smoke when a CUDA box is available:

```bash
soup merge -a ./adapter -o ./merged_4bit --save-format 4bit
# Expect ~70-90 MB checkpoint vs ~2.2 GB fp16 for TinyLlama
cat > q.yaml <<'EOF'
scheme: Int4WeightOnly
EOF
soup export --model ./merged --format torchao --quant-config ./q.yaml \
  --output ./out/torchao
```

## v0.53.2

### #71 — ONNX export pipeline (`soup export --format onnx`)

**Status:** PARTIAL PASS — small-model smoke ✅, TinyLlama-1.1B blocked by host
RAM during `onnx.load(load_external_data=True)` post-process.

**Platform:** Windows 11, Python 3.10, torch 2.5.1+cu121, transformers 4.57.6,
optimum 2.1.0, RTX 3050 4 GB (ONNX export runs on CPU; GPU not used).

**Smoke 1 — `hf-internal-testing/tiny-random-gpt2` (~5 MB):**

```python
from optimum.exporters.onnx import main_export
main_export(
    "hf-internal-testing/tiny-random-gpt2",
    output="./out/tinygpt2_onnx",
    task="text-generation",
    trust_remote_code=False,
)
```

Result: **PASS** — completed in 25.8s, emitted 8 files, `model.onnx` = 0.62 MB,
all tokenizer/config artefacts present.

**Smoke 2 — `TinyLlama/TinyLlama-1.1B-Chat-v1.0` + LoRA(r=4,
target=`q_proj`,`v_proj`) → merge → ONNX:**

```python
# load fp16 base, attach tiny LoRA, save adapter, merge_and_unload, save merged
# main_export(merged_dir, task="text-generation-with-past", ...)
```

Result: **PARTIAL** — base loaded ✅ (3.2s), LoRA attached ✅, merge ✅ (9.6s),
ONNX trace + serialise ✅. Post-process step
`onnx.load(model.onnx, load_external_data=True)` failed with **MemoryError**
when loading the external-data tensor file back into RAM (TinyLlama-1.1B fp32
is ~4.4 GB; host had insufficient free RAM to hold the model twice — once on
disk as `model.onnx_data`, once in the `onnx.ModelProto` for post-processing).

**Verdict:** Pipeline integrity proven by tiny-gpt2. TinyLlama-1.1B full export
is host-RAM-bound (not a bug in `soup_cli` or `optimum`). Re-run on a
≥16 GB free-RAM machine to complete the size-validation pass.

**Recorded:** 2026-05-13 (Wave 3 of v0.53.2 release).

### v0.53.2 Live-trainer CPU smokes (Step 6d)

**Status:** PASS — both new trainer wrappers train end-to-end on CPU.

**Platform:** Windows 11, Python 3.10, torch 2.5.1+cu121, transformers 4.57.6,
peft 0.x — `hf-internal-testing/tiny-random-gpt2` (safetensors-only base, avoids
the torch < 2.6 `.bin` security gate).

**`ClassifierTrainerWrapper`:** 4-row sentiment dataset, num_labels=2,
batch_size=2, max_steps=2. setup() built `GPT2ForSequenceClassification` with
`problem_type=single_label_classification`. train() completed 2 steps with
finite loss 0.787 in 0.66s.

**`DistillTrainerWrapper`:** 4-row chat dataset, teacher=same tiny-gpt2 (frozen,
verified `requires_grad=False`), `distill_divergence=forward_kl`,
`distill_temperature=2.0`, batch_size=2, max_steps=2. setup() loaded student +
teacher, applied PEFT LoRA, built `_DistillTrainer` subclass. train() completed
2 steps with finite loss 3.479 in 1.79s — KL kernel exercised in compute_loss.

**Bugs surfaced + fixed in Wave 3:**
1. `DataCollatorForLanguageModeling` did not pad pre-tokenised `labels`
   (variable-length rows crashed during batching) → switched to
   `DataCollatorForSeq2Seq(label_pad_token_id=-100)`.
2. `_DistillTrainer.compute_loss` device-mismatch: HF Trainer auto-moved the
   student to CUDA on a CUDA-capable box while the teacher (`device_map="cpu"`)
   stayed on CPU; teacher forward raised on cross-device `index_select` →
   added `teacher_inputs.to(teacher_device)` + `teacher_logits.to(student_logits.device)`
   bridge. Source-level regression guards added in `test_v0532.py`
   (`TestDistillSourceLevelGuards`).

**Recorded:** 2026-05-13.


---

## v0.53.4 #74 — HF Hub push surface (manual QA)

**Status:** DEFERRED — full live execution requires a private HF repo +
`HF_TOKEN` not available in this dev environment. The four commands ship
unchanged from v0.29.0 / v0.40.2 schema gates; this entry records the
test plan + smoke verification done locally on `--help` and dry-run paths.

**Commands exercised (`--help` + plumbing):**
- `soup push --repo <user/repo> --model ./output` — flag accepted, repo_id
  validator from v0.29.0 wired (`utils/hf.validate_repo_id`); pre-flight
  containment on `--model` confirmed via `is_under_cwd`.
- `soup train --push-as <user/repo>` — flag accepted; HFPushCallback path
  unchanged.
- `soup data push --hf-dataset <user/repo>` — flag accepted; JSONL
  containment + repo_id validator confirmed.
- `soup deploy hf-space --model <user/repo> --space <user/space>` — flag
  accepted; repo_id validators run on both args.

**Live push verification:** punted to first user with HF credentials. No
behavioural change since v0.40.2 #51 (custom Space template containment)
so a regression is unlikely; the schema + validator coverage in
`test_hf_integration.py` + `test_v0402_part_a.py` exercises the security
surface.

**Recorded:** 2026-05-13.

---

## v0.53.4 #83 — LLaMA Pro live smoke (Step 6d)

**Status:** PASS — `apply_block_expansion_if_configured` + `apply_llama_pro_freeze`
exercised end-to-end on a real `transformers.LlamaForCausalLM` (CPU, random
weights, 4 layers / 32 hidden / 4 heads).

**Outcome:**
- `expand_model_blocks(model, 2)` cloned the last 2 decoder blocks, zero-inited
  `mlp.down_proj.weight` AND `self_attn.o_proj.weight` (verified
  `abs().sum() == 0.0` on the actual PyTorch tensors), appended to
  `model.model.layers`, bumped `config.num_hidden_layers` 4 → 6.
- `apply_llama_pro_freeze(model, 2)` correctly froze every parameter of the
  original 4 blocks AND every embedding / lm_head, then unfroze the 2 appended
  blocks only.
- Forward pass on `LlamaForCausalLM` produced finite logits of shape
  `(1, 8, 128)` — the identity-init property holds (zero-init residual makes
  the new block initially a no-op on the residual stream).

**Non-llama-shape failure path also verified:** `expand_model_blocks` on
`hf-internal-testing/tiny-random-gpt2` (which uses `model.transformer.h` not
`model.model.layers`) raises `ValueError` with the actionable message
"could not find decoder layers list on the model (expected
model.model.layers or model.decoder.layers)". This is the expected loud-fail
for unsupported architectures — the schema does not pre-gate `expand_layers`
by architecture, so the user gets a clear runtime error if they try LLaMA Pro
on a GPT-2-shaped base.

**Recorded:** 2026-05-13.

## v0.53.7

### #75 — Synth data + traces providers (live backends)

**Status:** PENDING — requires a contributor with local Ollama install AND
`ANTHROPIC_API_KEY` env var.

Planned smoke:

```bash
# Ollama (localhost-only — provider rejects remote URLs)
soup data generate --provider ollama --prompt "Five short Python tips" \
  --count 10 --output ./gen.jsonl

# Anthropic (env-only API key — never CLI arg)
ANTHROPIC_API_KEY=sk-ant-... \
  soup data generate --provider anthropic \
  --model claude-3-5-sonnet-latest --count 10 --output ./gen-claude.jsonl

# Augment via LLM
soup data augment --input ./gen.jsonl --output ./aug.jsonl \
  --strategy rephrase --count 2

# Trace harvester end-to-end (langchain JSONL → preference pairs)
soup data from-traces --input ./traces.jsonl --output ./pairs.jsonl \
  --judge --judge-provider ollama --min-confidence 0.7
```

**Acceptance criteria:**
- `soup data generate` writes valid JSONL with `instruction`/`output` fields
  on alpaca format; `messages` on sharegpt/chatml; loop completes within
  rate-limit budget without 429s.
- `soup data augment` produces `count` augmented variants per input row;
  rephrase preserves semantics, translate flips language, style flips tone.
- `soup data from-traces --judge` calls the judge backend exactly twice per
  pair (chosen + rejected) and writes a `_judge_confidence` field on every
  kept row. Confidence ≥ 0.7 retained; below threshold dropped.

**Schema + validator coverage** of the security surface already lives in
`tests/test_synth_data_pro.py` + `tests/test_v0403_part_c.py` (`--judge`
filter, JudgeEvaluator mocked, providers SSRF-checked). The PENDING here is
*observational* — confirming the providers actually call the backends and
return well-formed JSON when invoked against a real local Ollama + Anthropic
key.

**Recorded:** 2026-05-13 (v0.53.7 release).

### #71 — TinyLlama ONNX export full pipeline

**Status:** PARTIAL — first half PASS, second half RAM-bound.

`soup export --format onnx --model TinyLlama/TinyLlama-1.1B-Chat-v1.0` ran
`optimum.main_export` through to the trace + serialise stage; the
post-process `onnx.load(load_external_data=True)` step needs to hold the
full ~4.4 GB fp32 weight blob in RAM alongside the serialised file. Dev box
has 8 GB; ran OOM. Tiny-gpt2 ONNX export (300 MB) PASS end-to-end.

**Mitigation:** Larger ONNX exports need a 16 GB+ RAM host. Soup wrapping is
correct; this is a host-RAM ceiling, not a Soup bug. Recorded for future QA
on a beefier box.

**Recorded:** 2026-05-13 (v0.53.7 release, re-confirmed from v0.53.2 entry).