mirror of https://github.com/scrapy/scrapy.git
The HTML specification states that link type keywords like 'nofollow' are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW" were incorrectly treated as follow links, causing Scrapy to crawl pages it should skip. Fix: add .lower() before the token split so all casing variants of 'nofollow' are correctly recognized. Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| test_walk_modules | ||
| __init__.py | ||
| test.egg | ||
| test_return_with_argument_inside_generator.py | ||