Add `soup data doctor` and `soup data lint`, killing the top *silent*
fine-tune failures before a single training step: EOS-missing-from-labels
(the #1 "model never stops generating" bug), BOS duplication, no-system-role
templates, and preference-data length bias (the #1 silent DPO degradation) —
none of which any competitor (Unsloth/Axolotl/LlamaFactory) checks for.
- utils/data_doctor.py: 8-check chat-template compat report over a
tokenizer + sampled rows, OK/MINOR/MAJOR taxonomy mirroring diagnose;
--show-mask N renders per-token trained/masked colouring through the
SAME masking dispatch (_build_row_labels) the report itself uses, so
the two can never disagree about what's actually trained.
- utils/data_lint.py: preference-data linter (dpo/orpo/simpo/ipo/bco/kto)
— length bias (Cohen's d), label imbalance, near-duplicates (MinHash),
identical chosen==rejected pairs, prompt leakage.
- commands/data_doctor.py: Typer layer for both commands; strips C0
control bytes before untrusted dataset content reaches the terminal.
- commands/diagnose.py: hardens the --evidence loader against a TOCTOU
symlink swap (O_NOFOLLOW + fstat-on-open-fd), backporting the pattern
soup ship shipped in v0.71.25 (closes v0.71.25 known-limitation (4)).
Live smoke against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer
(Windows + RTX 3050) found and fixed two genuine bugs beyond the synthetic
fixtures: the EOS check needed to span-search the whole trained region
(not just the last token), and two apply_chat_template call sites needed
a broad except Exception for jinja2.exceptions.TemplateError.
+173 tests (14788 -> 15042). 5 sequential ECC reviews, every finding fixed.
Move soup_cli/ -> src/soup_cli/ (history preserved via git mv). src-layout
forces the test suite to import the installed package instead of the
repo-root source tree, surfacing packaging bugs that flat-layout masks —
e.g. the v0.53.8 double-shipped-fixtures regression, invisible because
`pytest tests/` imports ./soup_cli directly and never from the wheel.
- pyproject: packages = ["src/soup_cli"]; artifacts globs -> src/soup_cli/...
The import name is unchanged, so the `soup` entry point, --cov=soup_cli,
and report_to/module-path strings stay `soup_cli`.
- CI / ownership: ruff lint path (ci.yml), recipe-validation `paths:` filters,
CODEOWNERS patterns, and the PR-template checklist all repointed to
src/soup_cli/.
- tests: source-grep regression tests that read package files by repo-relative
path repointed to src/soup_cli/ (64 files; 170 path literals). Lines pushed
over 100 chars by the prefix were wrapped to keep ruff E501 clean. Module
references (`import soup_cli`, `-m soup_cli`, mock.patch("soup_cli.x")) and
the `--cov=soup_cli` coverage target are deliberately unchanged.
- docs: AGENTS.md + CONTRIBUTING.md structure tree and lint commands.
Verified locally: ruff clean (src/soup_cli + tests); `import soup_cli`
resolves to src/soup_cli/__init__.py; built wheel ships
soup_cli/data/_fixtures/*.jsonl (10 files, no duplicates, no src/ prefix);
3174 tests across every touched test file pass. Packaging-only — no version bump.
Three failures on CI run 26084542388 — all are version-pin / Rich-wrap
artefacts, not real regressions in v0.60.0 functionality:
- test_v0560 test_pyproject_version: regex-based >=0.56 floor check
(was substring `version = "0.5`)
- test_v0590 test_version_is_0_59 -> test_version_is_at_least_0_59:
>=0.59 floor (matches v0.51/v0.54 floor-check idiom)
- test_v0600_part_e merge_help_lists_license_flags: strip ANSI codes
before substring check (Rich splits `--license` across `\x1b[1;36m`
escapes in the wrapped Typer table)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #169 wired LOCAL_RANK==0 guard on _run_diagnose_gate so distributed
launches only run the gate on one worker per machine. Two minor polish
items on top of the merged version:
- Wrap the int() parse in try/except ValueError. A malformed LOCAL_RANK
(garbage value from a misconfigured launcher) would previously crash
the post-training gate. Falling back to True is safer than silently
skipping the gate -- over-running is recoverable, under-running hides
failures.
- Expand the docstring to explain why we use LOCAL_RANK (per-machine)
rather than RANK (global): the gate reads the local output_dir, so
one gate per machine is the right granularity for typical single-
machine multi-GPU runs. Documents the choice for future readers.
- Add a focused test (test_diagnose_gate_handles_malformed_local_rank)
asserting the safe fallback path.
Rich's CliRunner output on CI carries ANSI escape codes that split long
option names like `--badge` and `--diagnose-gate` across colour-reset
boundaries (`-\x1b[0m\x1b[1;36m-badge`), breaking naive `"--badge" in
result.output` substring checks. Same fix pattern as v0.55.0 CI hotfix.
Failures: tests/test_v0560.py::TestCli::test_diagnose_help and
TestTrainDiagnoseGate::test_help_lists_flag on all 9 CI matrix cells.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpamys
Yixuan Xu