RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/scrapy
History
JSap0914 75cc7b8daa Fix rel_has_nofollow to be case-insensitive per HTML spec 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.
 		2026-06-17 15:18:07 +09:00
..
commands Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
contracts Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
core Fix the check for sync ITEM_PROCESSOR methods. (#7589) 2026-06-11 00:22:30 +05:00
downloadermiddlewares Implement HTTP Auth settings and request metadata keys (#7590) 2026-06-11 15:56:53 +05:00
extensions Close GCS feed temp files after upload (#7546) 2026-06-04 20:11:50 +05:00
http Support sending requests with "unsafe" URLs (#7473) 2026-06-05 11:51:03 +02:00
linkextractors chore: more ruff rules and overall minor improvements (#7386) 2026-04-03 19:00:13 +05:00
loader Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
pipelines Fix image store ACL settings for S3 and GCS (#7614) 2026-06-16 00:24:44 +05:00
selector Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
settings Implement HTTP Auth settings and request metadata keys (#7590) 2026-06-11 15:56:53 +05:00
spidermiddlewares Drop start_requests() support. (#7490) 2026-05-04 17:34:24 +05:00
spiders Update default_settings, deprecate CRAWLSPIDER_FOLLOW_LINKS. (#7592) 2026-06-11 00:22:09 +05:00
templates docs: switch `scrapy.Item` examples to dataclasses (#7513) 2026-06-08 12:40:28 +05:00
utils Fix rel_has_nofollow to be case-insensitive per HTML spec 2026-06-17 15:18:07 +09:00
VERSION Bump version: 2.15.2 → 2.16.0 2026-05-19 17:27:16 +05:00
__init__.py Remove code deprecated in 2.12.0 (#7126) 2025-10-27 17:14:38 +05:00
__main__.py adding black formatter to all the code 2022-11-29 11:30:46 -03:00
addons.py Move to mypy --strict with exceptions. (#7300) 2026-03-02 15:47:23 +05:00
cmdline.py Reactorless shell and other small shell improvements (#7395) 2026-04-08 18:24:51 +05:00
crawler.py chore: simplify some code, get rid of nested fns where it's makes sence + pylint (#7401) 2026-05-04 23:32:46 +05:00
dupefilters.py Add Jobdir documentation (#5260) 2026-02-06 22:33:02 +04:00
exceptions.py Refactor HTTP download handlers (#7238) 2026-02-03 20:13:30 +01:00
exporters.py Mark abstract base classes as such. (#6930) 2025-06-30 15:14:34 +05:00
extension.py Restore 2.14 getwithbase, add a new method for class key deduplication (#7449) 2026-04-22 19:01:59 +05:00
interfaces.py Remove useless pylint: disable lines. 2025-01-02 12:32:25 +05:00
item.py Update the outdated Item docstring. (#6427) 2024-11-04 16:17:56 +05:00
link.py Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
logformatter.py Release notes for 2.14.0 (#7035) 2026-01-05 14:51:05 +05:00
mail.py chore: more ruff rules and overall minor improvements (#7386) 2026-04-03 19:00:13 +05:00
middleware.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
mime.types
…
pqueues.py Fix DownloaderAwarePriorityQueue tie-breaking across slots (#7351) 2026-04-08 16:10:21 +02:00
py.typed Add py.typed 2023-09-21 13:06:12 +02:00
resolver.py Add flake8-raise rules to ruff. 2024-12-12 20:33:35 +05:00
responsetypes.py chore: more ruff rules and overall minor improvements (#7386) 2026-04-03 19:00:13 +05:00
robotstxt.py Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
shell.py Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
signalmanager.py Move to mypy --strict with exceptions. (#7300) 2026-03-02 15:47:23 +05:00
signals.py Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
spiderloader.py chore: more ruff rules and overall minor improvements (#7386) 2026-04-03 19:00:13 +05:00
squeues.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
statscollectors.py Move to mypy --strict with exceptions. (#7300) 2026-03-02 15:47:23 +05:00