The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.
Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
PyPy environments explicitly disable coverage to avoid the significant
runtime overhead it adds. Without coverage data, delta has no accurate
mapping for these envs and falls back to the CPython mapping, causing:
- Unreliable test selection (CPython-mapped tests may fail on PyPy)
- Pre-existing PyPy/min-deps failures surfacing unexpectedly (exit code 1)
Affected envs: pypy3, pypy3-extra-deps, min-pypy3
This follows the same principle as the botocore/mitmproxy fix:
delta is only used where it has accurate coverage mapping.
These envs filter tests with -m requires_botocore / -m requires_mitmproxy.
When delta selects non-matching tests based on the diff, pytest finds no
tests to run for that marker and exits with code 5 (no tests collected).
Revert these three envs back to plain pytest since they are already
highly filtered and delta provides no benefit here.
- Replace all bare pytest invocations with delta run -- in tox.ini:
[min], [testenv:pypy3], [testenv:min-pypy3], [testenv:botocore],
[testenv:min-botocore], [testenv:mitmproxy]
- Add deltatest-cli to [min] and [testenv:min-pypy3] deps
- Add passenv DELTA_* GITHUB_* to [min] and [testenv:min-pypy3]
- Add Configure Delta Cloud step to tests job in tests-ubuntu.yml
(writes ~/.delta/config.toml from secrets before tox runs)
- Remove standalone delta pre-check job; delta now runs inline
in every matrix runner via tox
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.
Closes#6120
aliowka
Andrey Rakhmatullin
JSap0914
Kushal Gupta
Adrian
Syncrain
Nishita Matlani
deltatest.dev
Labib Bin Salam