ci: follow redirects correctly in collection URL validator

The prior --max-filesize 1 aborted on the 301 redirect page's own body
before curl could follow it to the mirror, so the validator reported HTTP
301 for every URL and failed the whole catalog (100+ URLs that are actually
valid). download.kiwix.org 301-redirects both valid and dead files to a
mirror, so the redirect must be followed to learn the real status.

Drop --max-filesize and rely on Range: bytes=0-0 (mirrors return 206 after
one byte) with the body discarded to /dev/null and --max-time as the bound.
Verified against a sample across devdocs/libretexts/stack_exchange/maps/zimit
(all 206) and a known-dead URL (404).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Sherwood 2026-06-24 13:16:54 -07:00
parent 330774312c
commit ceabc2c72c
1 changed files with 10 additions and 10 deletions

View File

@ -28,18 +28,18 @@ jobs:
CHECKED=$((CHECKED + 1))
printf "Checking: %s ... " "$url"
# Use Range: bytes=0-0 to avoid downloading the full file.
# --max-filesize 1 aborts early if the server ignores the Range header
# and returns 200 with the full body. download.kiwix.org redirects to
# mirrors, some of which ignore Range and stream the body, making curl
# exit 63 (CURLE_FILESIZE_EXCEEDED). That just means the file EXISTS, so
# it must not fail the step: `|| true` keeps the step's `set -e` from
# aborting on that exit code, while curl still writes the real
# %{http_code} (200/206) for the check below. A genuine 404 still
# returns 404 with exit 0 and is still caught.
# Range: bytes=0-0 GET, following redirects, with the response body
# discarded. download.kiwix.org 301-redirects BOTH valid and dead files
# to a mirror, so we must follow to the mirror to learn the real status
# (206/200 = exists, 404 = gone). The previous --max-filesize 1 aborted
# on the 301 redirect page's own body BEFORE following it, reporting 301
# for every URL; it is removed. Range-honoring mirrors return 206 after
# one byte; --max-time bounds the rare mirror that ignores Range.
# `|| true` keeps the step's `set -e` from aborting on a curl-level
# error (e.g. transient network failure); those surface as a non-200
# code below.
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--range 0-0 \
--max-filesize 1 \
--max-time 30 \
--location \
"$url" || true)