Schema-first surface for the data pipeline gap with Axolotl + LlamaFactory.
Ships in one release: 5 new formats (prm, pre_tokenized, input_output,
video, multimodal), remote URI allowlist (s3/gs/gcs/az/abfs/abfss/oci) +
streaming + sharding, AOT preprocess cache + `soup data preprocess` CLI,
multi-dataset interleave (concat/under/over/probs) + 8 advanced masking
fields, vocab expansion (add_new_tokens / new_special_tokens / resize_vocab)
+ custom prompt_strategy, and document ingestion (`soup data ingest` for
PDF/DOCX/MD/TXT).
Live wiring for fsspec backends, AOT tokenize loop, custom prompt-strategy
runtime, and PRM trainer integration is deferred to v0.42.1+ (stub-then-live
pattern from v0.27.0 / v0.37.0 / v0.41.0). Schema gates fire at config
load so misconfiguration fails fast.
Security: full v0.42.0 hardening matrix — `_REMOTE_SCHEMES` MappingProxyType
allowlist; bucket regex 1-63 chars per S3/GCS spec; userinfo / fragment /
query-string rejection on remote URIs (query-string forwarded to fsspec is
SSRF-adjacent); null-byte + length caps on every string-shaped input;
bool-rejected-before-int on every numeric input; frozen InterleaveSpec
dataclass; 10k caps on add_new_tokens; `is_under_cwd` containment on
video_dir + tokenized_path schema fields and on preprocess --config / both
ingest paths; `os.lstat + S_ISLNK` symlink rejection on ingest input;
PRM converter type-checks completions (str) + labels (bool, not int);
video field null-byte + 2KB cap; field-name threading on image-pixels
validator so error messages name the actual field.
5242 tests → 5389 (+147 net). 11 review findings addressed across
python-review / code-review / security-review / tdd-guide (CRITICAL→LOW).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpamys