RomanNum3ral
/
soup
mirror of https://github.com/razor-ai/soup.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
soup/tests/qa/v053_qa.md

13 KiB
Raw Permalink Blame History

v0.53.x QA log

Manual smoke-test results for paths CI cannot reach (no GPU on CI runners, no llama.cpp build). Each entry records command + observed result + date.

v0.53.1

#70 — GGUF export (q4_0 / q4_k_m / q5_k_m / q8_0 / f16 / f32)

Status: PENDING — requires a GPU box + built llama.cpp.

Planned smoke (run on a 4 GB+ VRAM box with TinyLlama-1.1B-LoRA in ./adapter):

soup merge -a ./adapter -o ./merged
for q in q4_0 q4_k_m q5_k_m q8_0 f16 f32; do
  soup export --model ./merged --format gguf --quant "$q" \
    --output "./out/tinyllama.${q}.gguf"
done

Acceptance criteria:

  • All six files exist with non-zero size.
  • File sizes increase monotonically: q4_0 < q4_k_m < q5_k_m < q8_0 < f16 < f32.
  • llama-cli can load each file and emit at least one token.

When run, record sizes + first-token-time below.

#72 — AWQ + GPTQ export (--bits 4)

Status: PENDING — requires CUDA box with autoawq + auto-gptq extras.

Planned smoke:

soup export --model ./merged --format awq  --bits 4 --output ./out/awq
soup export --model ./merged --format gptq --bits 4 --output ./out/gptq

Acceptance: both output directories contain a config.json and weight shards; loading each via transformers.AutoModelForCausalLM returns a quantized model whose state_dict reports the expected dtype.

#82 — Autopilot pre-quantized detection

Status: PASS — automated regression coverage in tests/test_v0531_82.py.

Notes:

  • detect_prequantized_format("TheBloke/Llama-2-7B-Chat-GPTQ") → "gptq".
  • detect_prequantized_format("clean/name", {"quantization_config": {"quant_method": "awq"}}) → "awq".
  • decide_quantization(model_params_b=7.0, vram_gb=80.0, prequantized="gptq") → "gptq" (takes precedence over the VRAM heuristic).

#109 — soup deploy autopilot --measure

Status: PASS (orchestrator + cache); LIVE-EVAL PENDING (needs GPU + model loaders).

Coverage:

  • Cache key + sha-of-file: tests/test_v0531_109.py::TestComputeCacheKey.
  • Measurement loop + cache round-trip: tests/test_v0531_109.py::TestRunMeasure.
  • CLI plumbing with injected generators: TestDeployAutopilotMeasureCLI.

The injected-generator escape hatch (_DEPLOY_MEASURE_BEFORE_GEN, _DEPLOY_MEASURE_AFTER_FACTORY module attrs on soup_cli.utils.deploy_measure) is the production hook for live model loading in v0.53.x. Real transformers / vLLM generators land alongside v0.53.2 deferred trainer wiring.

#139 — Advanced GGUF export via llama.cpp imatrix

Status: PASS (orchestrator + subprocess argv shape + UD prefix stripping); LIVE-RUN PENDING (needs llama.cpp build).

Coverage:

  • _run_imatrix + _run_quantize_binary argv lists (no shell) verified.
  • UD-prefix-strip: UD-Q4_K_XL → llama.cpp arg Q4_K_XL.
  • Calibration JSONL → plain-text conversion + 4096-row safety cap.
  • Apple/ARM flavours (Q4_0_4_4 etc.) skip the imatrix stage; UD ladder
    • low-bit IQ family require --calibration-data.

Planned manual smoke when a llama.cpp build is available:

soup export --model ./merged --format gguf-ud \
  --gguf-flavour UD-Q4_K_XL \
  --calibration-data ./calib.jsonl \
  --llama-cpp ~/.soup/llama.cpp \
  --output ./out/tinyllama.UD-Q4_K_XL.gguf

#142 — Merge --save-format 4bit + export --format torchao

Status: PASS (orchestrator + validators + path TOCTOU); LIVE-EVAL PENDING (needs CUDA + bitsandbytes / torchao).

Coverage:

  • merge_4bit happy path with mocked transformers + BNB.
  • export_torchao happy path with mocked torchao + transformers.
  • --save-format weird rejected at CLI dispatch with exit code 2.
  • --format torchao without --quant-config rejected.
  • --quant-config outside cwd rejected.
  • load_quant_config enforces yaml.safe_load + 256 KB cap + extension allowlist + symlink rejection (POSIX).

Planned manual smoke when a CUDA box is available:

soup merge -a ./adapter -o ./merged_4bit --save-format 4bit
# Expect ~70-90 MB checkpoint vs ~2.2 GB fp16 for TinyLlama
cat > q.yaml <<'EOF'
scheme: Int4WeightOnly
EOF
soup export --model ./merged --format torchao --quant-config ./q.yaml \
  --output ./out/torchao

v0.53.2

#71 — ONNX export pipeline (soup export --format onnx)

Status: PARTIAL PASS — small-model smoke ✅, TinyLlama-1.1B blocked by host RAM during onnx.load(load_external_data=True) post-process.

Platform: Windows 11, Python 3.10, torch 2.5.1+cu121, transformers 4.57.6, optimum 2.1.0, RTX 3050 4 GB (ONNX export runs on CPU; GPU not used).

Smoke 1 — hf-internal-testing/tiny-random-gpt2 (~5 MB):

from optimum.exporters.onnx import main_export
main_export(
    "hf-internal-testing/tiny-random-gpt2",
    output="./out/tinygpt2_onnx",
    task="text-generation",
    trust_remote_code=False,
)

Result: PASS — completed in 25.8s, emitted 8 files, model.onnx = 0.62 MB, all tokenizer/config artefacts present.

Smoke 2 — TinyLlama/TinyLlama-1.1B-Chat-v1.0 + LoRA(r=4, target=q_proj,v_proj) → merge → ONNX:

# load fp16 base, attach tiny LoRA, save adapter, merge_and_unload, save merged
# main_export(merged_dir, task="text-generation-with-past", ...)

Result: PARTIAL — base loaded ✅ (3.2s), LoRA attached ✅, merge ✅ (9.6s), ONNX trace + serialise ✅. Post-process step onnx.load(model.onnx, load_external_data=True) failed with MemoryError when loading the external-data tensor file back into RAM (TinyLlama-1.1B fp32 is ~4.4 GB; host had insufficient free RAM to hold the model twice — once on disk as model.onnx_data, once in the onnx.ModelProto for post-processing).

Verdict: Pipeline integrity proven by tiny-gpt2. TinyLlama-1.1B full export is host-RAM-bound (not a bug in soup_cli or optimum). Re-run on a ≥16 GB free-RAM machine to complete the size-validation pass.

Recorded: 2026-05-13 (Wave 3 of v0.53.2 release).

v0.53.2 Live-trainer CPU smokes (Step 6d)

Status: PASS — both new trainer wrappers train end-to-end on CPU.

Platform: Windows 11, Python 3.10, torch 2.5.1+cu121, transformers 4.57.6, peft 0.x — hf-internal-testing/tiny-random-gpt2 (safetensors-only base, avoids the torch < 2.6 .bin security gate).

ClassifierTrainerWrapper: 4-row sentiment dataset, num_labels=2, batch_size=2, max_steps=2. setup() built GPT2ForSequenceClassification with problem_type=single_label_classification. train() completed 2 steps with finite loss 0.787 in 0.66s.

DistillTrainerWrapper: 4-row chat dataset, teacher=same tiny-gpt2 (frozen, verified requires_grad=False), distill_divergence=forward_kl, distill_temperature=2.0, batch_size=2, max_steps=2. setup() loaded student + teacher, applied PEFT LoRA, built _DistillTrainer subclass. train() completed 2 steps with finite loss 3.479 in 1.79s — KL kernel exercised in compute_loss.

Bugs surfaced + fixed in Wave 3:

  1. DataCollatorForLanguageModeling did not pad pre-tokenised labels (variable-length rows crashed during batching) → switched to DataCollatorForSeq2Seq(label_pad_token_id=-100).
  2. _DistillTrainer.compute_loss device-mismatch: HF Trainer auto-moved the student to CUDA on a CUDA-capable box while the teacher (device_map="cpu") stayed on CPU; teacher forward raised on cross-device index_select → added teacher_inputs.to(teacher_device) + teacher_logits.to(student_logits.device) bridge. Source-level regression guards added in test_v0532.py (TestDistillSourceLevelGuards).

Recorded: 2026-05-13.

v0.53.4 #74 — HF Hub push surface (manual QA)

Status: DEFERRED — full live execution requires a private HF repo + HF_TOKEN not available in this dev environment. The four commands ship unchanged from v0.29.0 / v0.40.2 schema gates; this entry records the test plan + smoke verification done locally on --help and dry-run paths.

Commands exercised (--help + plumbing):

  • soup push --repo <user/repo> --model ./output — flag accepted, repo_id validator from v0.29.0 wired (utils/hf.validate_repo_id); pre-flight containment on --model confirmed via is_under_cwd.
  • soup train --push-as <user/repo> — flag accepted; HFPushCallback path unchanged.
  • soup data push --hf-dataset <user/repo> — flag accepted; JSONL containment + repo_id validator confirmed.
  • soup deploy hf-space --model <user/repo> --space <user/space> — flag accepted; repo_id validators run on both args.

Live push verification: punted to first user with HF credentials. No behavioural change since v0.40.2 #51 (custom Space template containment) so a regression is unlikely; the schema + validator coverage in test_hf_integration.py + test_v0402_part_a.py exercises the security surface.

Recorded: 2026-05-13.

v0.53.4 #83 — LLaMA Pro live smoke (Step 6d)

Status: PASS — apply_block_expansion_if_configured + apply_llama_pro_freeze exercised end-to-end on a real transformers.LlamaForCausalLM (CPU, random weights, 4 layers / 32 hidden / 4 heads).

Outcome:

  • expand_model_blocks(model, 2) cloned the last 2 decoder blocks, zero-inited mlp.down_proj.weight AND self_attn.o_proj.weight (verified abs().sum() == 0.0 on the actual PyTorch tensors), appended to model.model.layers, bumped config.num_hidden_layers 4 → 6.
  • apply_llama_pro_freeze(model, 2) correctly froze every parameter of the original 4 blocks AND every embedding / lm_head, then unfroze the 2 appended blocks only.
  • Forward pass on LlamaForCausalLM produced finite logits of shape (1, 8, 128) — the identity-init property holds (zero-init residual makes the new block initially a no-op on the residual stream).

Non-llama-shape failure path also verified: expand_model_blocks on hf-internal-testing/tiny-random-gpt2 (which uses model.transformer.h not model.model.layers) raises ValueError with the actionable message "could not find decoder layers list on the model (expected model.model.layers or model.decoder.layers)". This is the expected loud-fail for unsupported architectures — the schema does not pre-gate expand_layers by architecture, so the user gets a clear runtime error if they try LLaMA Pro on a GPT-2-shaped base.

Recorded: 2026-05-13.

v0.53.7

#75 — Synth data + traces providers (live backends)

Status: PENDING — requires a contributor with local Ollama install AND ANTHROPIC_API_KEY env var.

Planned smoke:

# Ollama (localhost-only — provider rejects remote URLs)
soup data generate --provider ollama --prompt "Five short Python tips" \
  --count 10 --output ./gen.jsonl

# Anthropic (env-only API key — never CLI arg)
ANTHROPIC_API_KEY=sk-ant-... \
  soup data generate --provider anthropic \
  --model claude-3-5-sonnet-latest --count 10 --output ./gen-claude.jsonl

# Augment via LLM
soup data augment --input ./gen.jsonl --output ./aug.jsonl \
  --strategy rephrase --count 2

# Trace harvester end-to-end (langchain JSONL → preference pairs)
soup data from-traces --input ./traces.jsonl --output ./pairs.jsonl \
  --judge --judge-provider ollama --min-confidence 0.7

Acceptance criteria:

  • soup data generate writes valid JSONL with instruction/output fields on alpaca format; messages on sharegpt/chatml; loop completes within rate-limit budget without 429s.
  • soup data augment produces count augmented variants per input row; rephrase preserves semantics, translate flips language, style flips tone.
  • soup data from-traces --judge calls the judge backend exactly twice per pair (chosen + rejected) and writes a _judge_confidence field on every kept row. Confidence ≥ 0.7 retained; below threshold dropped.

Schema + validator coverage of the security surface already lives in tests/test_synth_data_pro.py + tests/test_v0403_part_c.py (--judge filter, JudgeEvaluator mocked, providers SSRF-checked). The PENDING here is observational — confirming the providers actually call the backends and return well-formed JSON when invoked against a real local Ollama + Anthropic key.

Recorded: 2026-05-13 (v0.53.7 release).

#71 — TinyLlama ONNX export full pipeline

Status: PARTIAL — first half PASS, second half RAM-bound.

soup export --format onnx --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 ran optimum.main_export through to the trace + serialise stage; the post-process onnx.load(load_external_data=True) step needs to hold the full ~4.4 GB fp32 weight blob in RAM alongside the serialised file. Dev box has 8 GB; ran OOM. Tiny-gpt2 ONNX export (300 MB) PASS end-to-end.

Mitigation: Larger ONNX exports need a 16 GB+ RAM host. Soup wrapping is correct; this is a host-RAM ceiling, not a Soup bug. Recorded for future QA on a beefier box.

Recorded: 2026-05-13 (v0.53.7 release, re-confirmed from v0.53.2 entry).