ci: follow redirects correctly in collection URL validator
The prior --max-filesize 1 aborted on the 301 redirect page's own body before curl could follow it to the mirror, so the validator reported HTTP 301 for every URL and failed the whole catalog (100+ URLs that are actually valid). download.kiwix.org 301-redirects both valid and dead files to a mirror, so the redirect must be followed to learn the real status. Drop --max-filesize and rely on Range: bytes=0-0 (mirrors return 206 after one byte) with the body discarded to /dev/null and --max-time as the bound. Verified against a sample across devdocs/libretexts/stack_exchange/maps/zimit (all 206) and a known-dead URL (404). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
330774312c
commit
ceabc2c72c
|
|
@ -28,18 +28,18 @@ jobs:
|
|||
CHECKED=$((CHECKED + 1))
|
||||
printf "Checking: %s ... " "$url"
|
||||
|
||||
# Use Range: bytes=0-0 to avoid downloading the full file.
|
||||
# --max-filesize 1 aborts early if the server ignores the Range header
|
||||
# and returns 200 with the full body. download.kiwix.org redirects to
|
||||
# mirrors, some of which ignore Range and stream the body, making curl
|
||||
# exit 63 (CURLE_FILESIZE_EXCEEDED). That just means the file EXISTS, so
|
||||
# it must not fail the step: `|| true` keeps the step's `set -e` from
|
||||
# aborting on that exit code, while curl still writes the real
|
||||
# %{http_code} (200/206) for the check below. A genuine 404 still
|
||||
# returns 404 with exit 0 and is still caught.
|
||||
# Range: bytes=0-0 GET, following redirects, with the response body
|
||||
# discarded. download.kiwix.org 301-redirects BOTH valid and dead files
|
||||
# to a mirror, so we must follow to the mirror to learn the real status
|
||||
# (206/200 = exists, 404 = gone). The previous --max-filesize 1 aborted
|
||||
# on the 301 redirect page's own body BEFORE following it, reporting 301
|
||||
# for every URL; it is removed. Range-honoring mirrors return 206 after
|
||||
# one byte; --max-time bounds the rare mirror that ignores Range.
|
||||
# `|| true` keeps the step's `set -e` from aborting on a curl-level
|
||||
# error (e.g. transient network failure); those surface as a non-200
|
||||
# code below.
|
||||
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||
--range 0-0 \
|
||||
--max-filesize 1 \
|
||||
--max-time 30 \
|
||||
--location \
|
||||
"$url" || true)
|
||||
|
|
|
|||
Loading…
Reference in New Issue