mirror of https://github.com/razor-ai/soup.git
ci: cache the HF Hub models instead of re-downloading them every run
Run 30942028585 reddened windows/3.10 on a 429 from huggingface.co while fetching SmolLM2-135M-Instruct/config.json. Nine of ten cells were fine; the tenth lost a coin flip against the Hub's anonymous rate limit. Over the last 30 CI runs on main, 9 were red, and this class is a share of them. A warm step with retries already existed — it just warmed two repos, and the one that 429'd was not among them, so the retry loop never covered it. Two changes: 1. Warm the models the tests actually load. The list is empirical: what a populated local ~/.cache/huggingface holds after a full run, not what grepping tests/ for repo-shaped strings suggests (60+ hits there, nearly all config fixtures that never download). Per-repo file patterns keep it at 544 MB instead of the 1.6 GB a blanket snapshot_download costs, by skipping the TF/Flax copies and the pytorch_model.bin twin of a safetensors file. 2. Restore/save that cache with actions/cache, keyed on this file so editing the model list re-warms. HF_HOME is pinned to $RUNNER_TEMP/hf-cache: the default lives under a different user per runner OS, and putting it in the checkout would feed ~0.5 GB to the tests that walk cwd for containment. Only a COMPLETE warm is saved. The key is content-addressed, so a cache written while one model was 429-ing would never be replaced and every later run would restore the same hole. Verified by running the suite with HF_HUB_OFFLINE=1 against the warmed cache — if it passes with no network, the warm list is complete. That is also what caught the real bug in the first attempt: the tiny Whisper fixture and sshleifer/tiny-gpt2 ship weights ONLY as pytorch_model.bin, so filtering to safetensors left the ASR test unable to build a model. Without the offline run this would have looked green in CI (network present, missing file just downloaded) while still 429-ing on exactly those files. Final: 16935 passed, 129 skipped, 0 failed offline. The ASR test skips on this box under its own torch<2.6 .bin-load guard (CVE-2025-32434); CI installs a newer torch, where it runs and needs that .bin. Two ids worth not "fixing" later: gpt2 is unqualified because the tests ask for the bare id and it caches as models--gpt2 — warming openai-community/gpt2 fills a different directory and every test still misses. CI-only, no version bump.
This commit is contained in:
parent
b059f1ca5d
commit
195d60b8c7
|
|
@ -49,30 +49,79 @@ jobs:
|
|||
# (seen with trl.trainer.grpo_trainer on windows-latest / py3.11).
|
||||
PYTHONUTF8: "1"
|
||||
PYTHONIOENCODING: "utf-8"
|
||||
# Pin the HF cache to a deterministic path so actions/cache restores it
|
||||
# identically on ubuntu / windows / macos — the default
|
||||
# ~/.cache/huggingface sits under a different user on each runner. Kept
|
||||
# OUTSIDE the checkout on purpose: this lands ~0.5 GB on disk, and inside
|
||||
# the repo it would be walked by the tests that scan cwd for containment.
|
||||
HF_HOME: ${{ runner.temp }}/hf-cache
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
- name: Restore HF Hub cache
|
||||
id: hf-cache
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ${{ runner.temp }}/hf-cache
|
||||
# Keyed on this file, so editing the model list below re-warms.
|
||||
key: hf-${{ runner.os }}-${{ hashFiles('.github/workflows/ci.yml') }}
|
||||
restore-keys: hf-${{ runner.os }}-
|
||||
- name: Install dependencies
|
||||
run: pip install -e ".[dev]"
|
||||
- name: Warm HF Hub cache (tiny test models)
|
||||
# Pre-download the tiny models the unit tests load, with retries, so a
|
||||
- name: Warm HF Hub cache (models the tests load)
|
||||
# Pre-download the models the unit tests load, with retries, so a
|
||||
# transient HF Hub 429 (Too Many Requests) on a single matrix cell does
|
||||
# not red the whole run. Observed: ~5/7 cells download fine and ~2/7 hit
|
||||
# a 429 — a retry loop reliably populates the cache so the tests then
|
||||
# read from disk (no network). Never fails the job: if warming can't
|
||||
# complete it emits a ::warning:: and lets the test step run as before.
|
||||
# not red the whole run. Never fails the job: if warming can't complete
|
||||
# it emits a ::warning:: and lets the test step run as before.
|
||||
#
|
||||
# The list is EMPIRICAL — it is what a full local run actually leaves in
|
||||
# ~/.cache/huggingface, not what greps for repo-shaped strings in
|
||||
# tests/ suggest (most of those are config fixtures that never
|
||||
# download). The list grew because run 30942028585 reddened
|
||||
# windows/3.10 on a 429 for SmolLM2-135M-Instruct/config.json — a repo
|
||||
# this step did not warm, so the retry loop above never protected it.
|
||||
#
|
||||
# "gpt2" is deliberately unqualified: tests ask for the bare id, which
|
||||
# caches as models--gpt2. Warming "openai-community/gpt2" instead would
|
||||
# fill a different directory and every test would still miss.
|
||||
id: warm
|
||||
shell: python
|
||||
run: |
|
||||
import os
|
||||
import time
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
models = ["sshleifer/tiny-gpt2", "hf-internal-testing/tiny-random-gpt2"]
|
||||
for repo in models:
|
||||
# Per-repo file sets, also empirical: these are the files a populated
|
||||
# local cache actually holds after running the suite. Downloading
|
||||
# whole snapshots instead costs 1.6 GB per cell — mostly the TF /
|
||||
# Flax copies and the pytorch_model.bin twin of a safetensors file
|
||||
# that transformers never opens when safetensors is present.
|
||||
TEXT_ONLY = ["*.json", "*.txt", "*.model"] # config + tokenizer
|
||||
SAFETENSORS = TEXT_ONLY + ["*.safetensors"]
|
||||
# Two of the fixture repos predate safetensors and ship weights ONLY
|
||||
# as pytorch_model.bin. Both are a few MB, so pulling the .bin costs
|
||||
# nothing here — but excluding it left the ASR test unable to build a
|
||||
# model offline, which a full offline run of the suite caught.
|
||||
LEGACY_BIN = TEXT_ONLY + ["*.bin"]
|
||||
|
||||
models = [
|
||||
("sshleifer/tiny-gpt2", LEGACY_BIN),
|
||||
("hf-internal-testing/tiny-random-gpt2", SAFETENSORS),
|
||||
("hf-internal-testing/tiny-random-WhisperForConditionalGeneration", LEGACY_BIN),
|
||||
# Weights are never loaded for these two — only the config the
|
||||
# 429 above was raised on.
|
||||
("HuggingFaceTB/SmolLM2-135M", TEXT_ONLY),
|
||||
("HuggingFaceTB/SmolLM2-135M-Instruct", TEXT_ONLY),
|
||||
("gpt2", SAFETENSORS),
|
||||
]
|
||||
|
||||
complete = True
|
||||
for repo, allow in models:
|
||||
for attempt in range(1, 7):
|
||||
try:
|
||||
snapshot_download(repo)
|
||||
snapshot_download(repo, allow_patterns=allow)
|
||||
print(f"warmed {repo}")
|
||||
break
|
||||
except Exception as exc: # noqa: BLE001 — best effort, never fail the job
|
||||
|
|
@ -80,6 +129,22 @@ jobs:
|
|||
time.sleep(min(5 * attempt, 30))
|
||||
else:
|
||||
print(f"::warning title=HF cache::could not warm {repo} after 6 attempts")
|
||||
complete = False
|
||||
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as fh:
|
||||
fh.write(f"warmed={'all' if complete else 'partial'}\n")
|
||||
- name: Save HF Hub cache
|
||||
# Only a COMPLETE warm is saved. A partial one would freeze the very
|
||||
# gap this step exists to close: the key is content-addressed, so a
|
||||
# cache written while a model was 429-ing is never replaced, and every
|
||||
# later run would restore the same hole.
|
||||
# Restricted to one cell per OS — the blobs are platform-independent,
|
||||
# and 3 python versions racing to upload the same key is pure waste.
|
||||
if: always() && steps.warm.outputs.warmed == 'all' && matrix.python-version == '3.11' && steps.hf-cache.outputs.cache-hit != 'true'
|
||||
uses: actions/cache/save@v4
|
||||
with:
|
||||
path: ${{ runner.temp }}/hf-cache
|
||||
key: hf-${{ runner.os }}-${{ hashFiles('.github/workflows/ci.yml') }}
|
||||
- name: Run unit tests with coverage
|
||||
run: pytest tests/ -v --tb=short --junitxml=report.xml --cov=soup_cli --cov-report=xml:coverage.xml --cov-report=term-missing:skip-covered
|
||||
- name: Upload coverage to Codecov
|
||||
|
|
|
|||
Loading…
Reference in New Issue