ci: cache the HF Hub models instead of re-downloading them every run

Run 30942028585 reddened windows/3.10 on a 429 from huggingface.co while
fetching SmolLM2-135M-Instruct/config.json. Nine of ten cells were fine; the
tenth lost a coin flip against the Hub's anonymous rate limit. Over the last 30
CI runs on main, 9 were red, and this class is a share of them.

A warm step with retries already existed — it just warmed two repos, and the
one that 429'd was not among them, so the retry loop never covered it.

Two changes:

1. Warm the models the tests actually load. The list is empirical: what a
   populated local ~/.cache/huggingface holds after a full run, not what
   grepping tests/ for repo-shaped strings suggests (60+ hits there, nearly all
   config fixtures that never download). Per-repo file patterns keep it at
   544 MB instead of the 1.6 GB a blanket snapshot_download costs, by skipping
   the TF/Flax copies and the pytorch_model.bin twin of a safetensors file.

2. Restore/save that cache with actions/cache, keyed on this file so editing
   the model list re-warms. HF_HOME is pinned to $RUNNER_TEMP/hf-cache: the
   default lives under a different user per runner OS, and putting it in the
   checkout would feed ~0.5 GB to the tests that walk cwd for containment.

Only a COMPLETE warm is saved. The key is content-addressed, so a cache written
while one model was 429-ing would never be replaced and every later run would
restore the same hole.

Verified by running the suite with HF_HUB_OFFLINE=1 against the warmed cache —
if it passes with no network, the warm list is complete. That is also what
caught the real bug in the first attempt: the tiny Whisper fixture and
sshleifer/tiny-gpt2 ship weights ONLY as pytorch_model.bin, so filtering to
safetensors left the ASR test unable to build a model. Without the offline run
this would have looked green in CI (network present, missing file just
downloaded) while still 429-ing on exactly those files.

Final: 16935 passed, 129 skipped, 0 failed offline. The ASR test skips on this
box under its own torch<2.6 .bin-load guard (CVE-2025-32434); CI installs a
newer torch, where it runs and needs that .bin.

Two ids worth not "fixing" later: gpt2 is unqualified because the tests ask for
the bare id and it caches as models--gpt2 — warming openai-community/gpt2 fills
a different directory and every test still misses.

CI-only, no version bump.
This commit is contained in:
Alpamys 2026-08-05 01:23:09 +05:00
parent b059f1ca5d
commit 195d60b8c7
1 changed files with 74 additions and 9 deletions

View File

@ -49,30 +49,79 @@ jobs:
# (seen with trl.trainer.grpo_trainer on windows-latest / py3.11).
PYTHONUTF8: "1"
PYTHONIOENCODING: "utf-8"
# Pin the HF cache to a deterministic path so actions/cache restores it
# identically on ubuntu / windows / macos — the default
# ~/.cache/huggingface sits under a different user on each runner. Kept
# OUTSIDE the checkout on purpose: this lands ~0.5 GB on disk, and inside
# the repo it would be walked by the tests that scan cwd for containment.
HF_HOME: ${{ runner.temp }}/hf-cache
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Restore HF Hub cache
id: hf-cache
uses: actions/cache/restore@v4
with:
path: ${{ runner.temp }}/hf-cache
# Keyed on this file, so editing the model list below re-warms.
key: hf-${{ runner.os }}-${{ hashFiles('.github/workflows/ci.yml') }}
restore-keys: hf-${{ runner.os }}-
- name: Install dependencies
run: pip install -e ".[dev]"
- name: Warm HF Hub cache (tiny test models)
# Pre-download the tiny models the unit tests load, with retries, so a
- name: Warm HF Hub cache (models the tests load)
# Pre-download the models the unit tests load, with retries, so a
# transient HF Hub 429 (Too Many Requests) on a single matrix cell does
# not red the whole run. Observed: ~5/7 cells download fine and ~2/7 hit
# a 429 — a retry loop reliably populates the cache so the tests then
# read from disk (no network). Never fails the job: if warming can't
# complete it emits a ::warning:: and lets the test step run as before.
# not red the whole run. Never fails the job: if warming can't complete
# it emits a ::warning:: and lets the test step run as before.
#
# The list is EMPIRICAL — it is what a full local run actually leaves in
# ~/.cache/huggingface, not what greps for repo-shaped strings in
# tests/ suggest (most of those are config fixtures that never
# download). The list grew because run 30942028585 reddened
# windows/3.10 on a 429 for SmolLM2-135M-Instruct/config.json — a repo
# this step did not warm, so the retry loop above never protected it.
#
# "gpt2" is deliberately unqualified: tests ask for the bare id, which
# caches as models--gpt2. Warming "openai-community/gpt2" instead would
# fill a different directory and every test would still miss.
id: warm
shell: python
run: |
import os
import time
from huggingface_hub import snapshot_download
models = ["sshleifer/tiny-gpt2", "hf-internal-testing/tiny-random-gpt2"]
for repo in models:
# Per-repo file sets, also empirical: these are the files a populated
# local cache actually holds after running the suite. Downloading
# whole snapshots instead costs 1.6 GB per cell — mostly the TF /
# Flax copies and the pytorch_model.bin twin of a safetensors file
# that transformers never opens when safetensors is present.
TEXT_ONLY = ["*.json", "*.txt", "*.model"] # config + tokenizer
SAFETENSORS = TEXT_ONLY + ["*.safetensors"]
# Two of the fixture repos predate safetensors and ship weights ONLY
# as pytorch_model.bin. Both are a few MB, so pulling the .bin costs
# nothing here — but excluding it left the ASR test unable to build a
# model offline, which a full offline run of the suite caught.
LEGACY_BIN = TEXT_ONLY + ["*.bin"]
models = [
("sshleifer/tiny-gpt2", LEGACY_BIN),
("hf-internal-testing/tiny-random-gpt2", SAFETENSORS),
("hf-internal-testing/tiny-random-WhisperForConditionalGeneration", LEGACY_BIN),
# Weights are never loaded for these two — only the config the
# 429 above was raised on.
("HuggingFaceTB/SmolLM2-135M", TEXT_ONLY),
("HuggingFaceTB/SmolLM2-135M-Instruct", TEXT_ONLY),
("gpt2", SAFETENSORS),
]
complete = True
for repo, allow in models:
for attempt in range(1, 7):
try:
snapshot_download(repo)
snapshot_download(repo, allow_patterns=allow)
print(f"warmed {repo}")
break
except Exception as exc: # noqa: BLE001 — best effort, never fail the job
@ -80,6 +129,22 @@ jobs:
time.sleep(min(5 * attempt, 30))
else:
print(f"::warning title=HF cache::could not warm {repo} after 6 attempts")
complete = False
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as fh:
fh.write(f"warmed={'all' if complete else 'partial'}\n")
- name: Save HF Hub cache
# Only a COMPLETE warm is saved. A partial one would freeze the very
# gap this step exists to close: the key is content-addressed, so a
# cache written while a model was 429-ing is never replaced, and every
# later run would restore the same hole.
# Restricted to one cell per OS — the blobs are platform-independent,
# and 3 python versions racing to upload the same key is pure waste.
if: always() && steps.warm.outputs.warmed == 'all' && matrix.python-version == '3.11' && steps.hf-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v4
with:
path: ${{ runner.temp }}/hf-cache
key: hf-${{ runner.os }}-${{ hashFiles('.github/workflows/ci.yml') }}
- name: Run unit tests with coverage
run: pytest tests/ -v --tb=short --junitxml=report.xml --cov=soup_cli --cov-report=xml:coverage.xml --cov-report=term-missing:skip-covered
- name: Upload coverage to Codecov