The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.
Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes#5297, fixes#4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Remove support for sync process_spider_output().
* Slight check fix.
* Don't expect exceptions from calling process_spider_output().
* Remove dead code.
* Fix test_deprecated_mw_spider_arg() to run all methods.
* Fix typing.
* Typos.
* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective
* Fix indentation issue
* Minor rewordings
* older versions → lower versions
* Update news.rst
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Python 3.14 (alpha3) to CI.
* Disable mitmproxy on 3.14 for now.
* 3.14.0-alpha.4.
* 3.14.0-alpha.5
* 3.14.0-beta2.
* 3.14 release.
* Fix test_non_pickable_object.
* Fix handling of file:/path feed URIs.
* Better mocking of streams for TextTestResult.
* Do not use .php in test_file_path() as it's now a known extension.
* Fix the URL in TestFeedExporterSignals.
* Fix typing.
* Bump more envs to 3.14.
* Silence pylint.
* Fix another test for .php handling change.
* Remove test_install_asyncio_reactor.
* More bumps to 3.14.
* Revert docs-tests to use 3.13.
* Debug options for Windows.
* Re-enable xdist.
* Revert Windows PYTEST_ADDOPTS.
* Silence loop policy deprecation warnings.
* Restore a lost pylint suppression.
* Update asyncio_enabled_reactor_same_loop.py to new Twisted.
* Fix RobotFileParser tests for Python 3.14.5.
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).
* Hide the _setAcceptableProtocols import.
* Silence ScrapyClientContextFactory HTTP/1.0 warnings.
* Update ScrapyClientTLSOptions for unreleased Twisted.
* Add twisted-trunk to CI.
* Update for new changes in Twisted trunk.
* Xfail a test failing with newer Twisted.
* Test Twisted trunk with extra-deps, update the ALPN code.
* Cleanup.
* Fix typing.
* Update relevant type hints for Twisted trunk.
* Update test_no_context_sharing().
* Silence a weird mypy error.
* Update Twisted versions, unpin pyOpenSSL.
* Update a comment.
* Silence pylint.
* Make a factory fixture.
* Improve the _setAcceptableProtocols comment.
* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.
* Add Twisted[http2] to extra-deps-pinned.
* Run mitmproxy-based tests for every handler and improve them.
* Fixes for H2DownloadHandler.
* Revert "Fixes for H2DownloadHandler."
This reverts commit bcdbd097cd.
* Add TestHttpsProxy for HTTP11DownloadHandler.
* Use the configured context factory in ScrapyProxyAgent.
* Reword.
* Reword.
* pragma: no cover for H2DownloadHandler proxy code.
* Raise an exception for HTTPS proxies instead of using HTTP.
* Remove non-working proxy support and explicitly forbid HTTP.
* pragma: no cover
* Add a docs note about HTTPS proxies.
* Rename ScrapyProxyAgent.
Adrian
Shashank S. Khasare
Andrey Rakhmatullin
Vishal Gawade
Lions-1
JSap0914
Kushal Gupta
Syncrain
Nishita Matlani
Adnan Awan
Ayush Singh
Ethan Kuo
Fardin Alizadeh