* Fix _get_tag_name() crash for non-string elem.tag (#7686)
* test: improve non-string tag test accuracy and add direct unit test
* test: use minimal payload for non-string tag test
* test: address Adrian+syncrain PR feedback
- remove first docstring line (Adrian: unnecessary)
- replace weak isinstance assert with no-op call
- keep Cython function mention (Adrian: wording is great)
* test: replace silent call with assert results == [] per Adrian
* chore: drop accidental pyproject.toml and uv.lock changes
* Adjust CookiesT.
* Drop list of plain cookie dicts support from request_to_curl().
* Extract _decode_cookie().
* Unify logging.
* Extract _to_verbose_cookies().
* Sync and type hint _cookie_to_set_cookie_value() in tests.
* Add tests for bytes in Request.cookies.
* Type hint assertCookieValEqual().
* Fix cell-var-from-loop in _send_catch_log_deferred
Replace lambda capturing receiver by reference with a default
argument to capture it by value, fixing a potential bug where
all deferred callbacks could reference the last receiver in the
loop instead of their respective receivers.
Removes the pylint disable comment and TODO that were suppressing
this issue.
* chore: trigger CI rerun for mypy network error
* Fix mypy error: pass receiver via addBoth args instead of lambda default
* style: apply pre-commit ruff formatting
* fix: LocalCache with limit=0 incorrectly stores items
When limit=0 is passed to LocalCache (e.g. when DNSCACHE_ENABLED=False),
the condition 'if self.limit' evaluates to False due to Python's truthiness
rules, causing items to be stored despite the cache being disabled.
This leads to an unbounded memory leak during long crawls when DNS caching
is explicitly disabled.
Fix changes the condition to 'if self.limit is not None' and adds an early
return when limit=0 to correctly handle the disabled cache case.
* test: add resolver-level regression tests for DNSCACHE_ENABLED=False
Add two regression tests that verify DNS results are not stored in
dnscache when DNSCACHE_ENABLED=False (cache_size=0):
- test_caching_hostname_resolver_dnscache_disabled_rejects_storage:
verifies _CachingResolutionReceiver does not write to dnscache
when CachingHostnameResolver is initialized with cache_size=0
- test_caching_threaded_resolver_dnscache_disabled_rejects_storage:
verifies dnscache rejects storage at the LocalCache level
when limit=0
* test: drop misleading threaded resolver test
it was writing directly to dnscache, not actually going through
CachingThreadedResolver at all. the threaded resolver already has
its own if dnscache.limit: guard so the bug doesnt affect it anyway.
keeping only the hostname resolver test which covers the actual
bug path through _CachingResolutionReceiver
* test: clean up comments in resolver test
* test: remove unnecessary comment
* Fix request_to_curl() corrupting dict cookies with bytes keys/values
PR #7603 made the list-cookie branch of request_to_curl() bytes-safe via
_cookie_value_to_unicode(), but left the sibling dict-cookie branch using
raw f-string interpolation. A dict cookie with bytes keys/values (a
supported and common form, e.g. Request(url, cookies={b"k": b"v"})) was
rendered as --cookie 'b'k'=b'v'' instead of --cookie 'k=v', producing a
broken curl command.
Route the dict branch through the same _cookie_value_to_unicode() helper,
mirroring the list branch. Add a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: apply _cookie_value_to_unicode to list-branch cookie key/value
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Set TELNETCONSOLE_ENABLED=False in get_crawler().
* Make _get_console_and_portal() a context manager.
* Allow PYTHONTRACEMALLOC in tox.
* Close the event loop in test_custom_asyncio_loop_enabled_false().
* Close the handler in _uninstall_scrapy_root_handler().
* Close queues in test_squeues.py.
* Close empty queues in init_prios().
* Close PopenSpawn stdin and stdout explicitly.
* Silence the unclosed socket warning.
* Implement more methods in CustomStatsCollector.
* Link the Twisted issue.
* Close the connection on download_maxsize.
* Fix typing.
* Add a comment about CustomStatsCollector.
* Restore the coverage.
* Properly close fixture queues.
* More robust file closing in feed storages.
* Close the file in TestMarshalItemExporter.test_nonstring_types_item().
* Cleanup temporary file handling in test_exporters.py.
* Close the file in test_stats_file_failed().
* Use an explicit TextIOWrapper in XmlItemExporter.
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.
Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes#5297, fixes#4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Remove support for sync process_spider_output().
* Slight check fix.
* Don't expect exceptions from calling process_spider_output().
* Remove dead code.
* Fix test_deprecated_mw_spider_arg() to run all methods.
* Fix typing.
* Typos.
* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective
* Fix indentation issue
* Minor rewordings
* older versions → lower versions
* Update news.rst
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Python 3.14 (alpha3) to CI.
* Disable mitmproxy on 3.14 for now.
* 3.14.0-alpha.4.
* 3.14.0-alpha.5
* 3.14.0-beta2.
* 3.14 release.
* Fix test_non_pickable_object.
* Fix handling of file:/path feed URIs.
* Better mocking of streams for TextTestResult.
* Do not use .php in test_file_path() as it's now a known extension.
* Fix the URL in TestFeedExporterSignals.
* Fix typing.
* Bump more envs to 3.14.
* Silence pylint.
* Fix another test for .php handling change.
* Remove test_install_asyncio_reactor.
* More bumps to 3.14.
* Revert docs-tests to use 3.13.
* Debug options for Windows.
* Re-enable xdist.
* Revert Windows PYTEST_ADDOPTS.
* Silence loop policy deprecation warnings.
* Restore a lost pylint suppression.
* Update asyncio_enabled_reactor_same_loop.py to new Twisted.
* Fix RobotFileParser tests for Python 3.14.5.
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).
* Hide the _setAcceptableProtocols import.
* Silence ScrapyClientContextFactory HTTP/1.0 warnings.
* Update ScrapyClientTLSOptions for unreleased Twisted.
* Add twisted-trunk to CI.
* Update for new changes in Twisted trunk.
* Xfail a test failing with newer Twisted.
* Test Twisted trunk with extra-deps, update the ALPN code.
* Cleanup.
* Fix typing.
* Update relevant type hints for Twisted trunk.
* Update test_no_context_sharing().
* Silence a weird mypy error.
* Update Twisted versions, unpin pyOpenSSL.
* Update a comment.
* Silence pylint.
* Make a factory fixture.
* Improve the _setAcceptableProtocols comment.
* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.
* Add Twisted[http2] to extra-deps-pinned.
* Run mitmproxy-based tests for every handler and improve them.
* Fixes for H2DownloadHandler.
* Revert "Fixes for H2DownloadHandler."
This reverts commit bcdbd097cd.
* Add TestHttpsProxy for HTTP11DownloadHandler.
* Use the configured context factory in ScrapyProxyAgent.
* Reword.
* Reword.
* pragma: no cover for H2DownloadHandler proxy code.
* Raise an exception for HTTPS proxies instead of using HTTP.
* Remove non-working proxy support and explicitly forbid HTTP.
* pragma: no cover
* Add a docs note about HTTPS proxies.
* Rename ScrapyProxyAgent.
* Restore 2.14 getwithbase, add a new method for class key deduplication
* Use the original getwithbase code, not some equivalent
* Fix mocking
* Test key exceptions
* Popen cleanup.
* Improve the test for response headers.
* Fix default headers of Httpx and H2 handlers, add tests.
* Check Location in redirect tests.
* Simplify bindaddress tests.
* Add download_latency to HttpxDownloadHandler, add tests.
* Remove explicit method="GET".
* Don't expect a specific value in test_download_with_maxsize_very_large_file
* Use a non-existent scheme in test_unsupported_scheme.
* Remporarily rollback the test_download_latency test check.
* Better check in test_download_latency.
* Improve logic for StreamCloseReason.CANCELLED.
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
* overall prettyfication
- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting
* return `SIM300` to ignore
* apply ruff rules to all files
* another one
* remove extra space in pyproject.toml
* simplify `__getattr__` in `scrapy.utils.url`
* lazy `url_is_from_any_domain` and `url_is_from_spider`
* improve typing
* specify `arg_to_iter` with None as arg
* improve trackref
- use `NoneType` from `types` since python 3.9 is no longer supported
- use `monotonic` instead of `time` and fix flakiness of `get_oldest`
* tracing GC is no joke
* refine tests
* explain the tests
* add note for pypy in docs + ...
return empty tuple instead of list in `iter_all`
* Import some stuff from service_identity directly.
* Extract _log_tls().
* Sync validating IP addresses.
* Introduce _deps_compat.
* Deprecate getCertificateOptions().
* Deprecate ScrapyClientTLSOptions.
* Add pragma: no cover.
* verify=False and trustRoot=platformTrust() are the defaults.
* Improve docstrings for TLS classes.
* Deprecate AcceptableProtocolsContextFactory.
* Switch from DefaultOpenSSLContextFactory to CertificateOptions.
* Deprecate load_context_factory_from_settings().
* Remove the outdated warning about incompatible factories.
* Simplify _load_context_factory_from_settings().
* Make CertificateOptions and Context once per factory.
* pragma: no cover
* Make test_start_deprecated_super() more robust.
* Work around the FTPFilesStore.FTP_* initialization.
* Enable xdist on CI.
* Add PYTEST_ADDOPTS to tox passenv.
* Fix test for getwithbase() to ensure class-keyed overrides with None are handled correctly
* Test actual import paths from default_settings
* Improvements
* Remove unused logger
* Run pre-commit
* Restore and improve duplicate key handling and warning
* type → object
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
* Await some missed deferreds.
* Simplify test_engine_stop_download_*.
* Fix the header value.
* Extend TestHttpWithCrawlerBase.
* Make test_tls_logging more universal.
* Add more tests for header handling.
* Clarify certificate and ip_address integration tests.
* Make test_download_with_maxsize_very_large_file() more universal.
* Fix test_coroutine_asyncio().
* Typo.
* Foundations for the reactorless mode.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
Gaurav Yadav
Adrian
Andrey Rakhmatullin
tanishqtayade
greymoth
Shashank S. Khasare
Vishal Gawade
Lions-1
JSap0914
Kushal Gupta
Syncrain
Nishita Matlani
Adnan Awan
Ayush Singh
Ethan Kuo
Fardin Alizadeh
Albert Eduardovich N.
Adrián Chaves
Hamzah Alshawwaf
Kailesh
Julian Hamze
pierreeurope
Ryotaro
Thalison Fernandes
Laerte Pereira
Leonardo Tozzo