* Fix _get_tag_name() crash for non-string elem.tag (#7686)
* test: improve non-string tag test accuracy and add direct unit test
* test: use minimal payload for non-string tag test
* test: address Adrian+syncrain PR feedback
- remove first docstring line (Adrian: unnecessary)
- replace weak isinstance assert with no-op call
- keep Cython function mention (Adrian: wording is great)
* test: replace silent call with assert results == [] per Adrian
* chore: drop accidental pyproject.toml and uv.lock changes
* Adjust CookiesT.
* Drop list of plain cookie dicts support from request_to_curl().
* Extract _decode_cookie().
* Unify logging.
* Extract _to_verbose_cookies().
* Sync and type hint _cookie_to_set_cookie_value() in tests.
* Add tests for bytes in Request.cookies.
* Type hint assertCookieValEqual().
* Fix cell-var-from-loop in _send_catch_log_deferred
Replace lambda capturing receiver by reference with a default
argument to capture it by value, fixing a potential bug where
all deferred callbacks could reference the last receiver in the
loop instead of their respective receivers.
Removes the pylint disable comment and TODO that were suppressing
this issue.
* chore: trigger CI rerun for mypy network error
* Fix mypy error: pass receiver via addBoth args instead of lambda default
* style: apply pre-commit ruff formatting
* fix: LocalCache with limit=0 incorrectly stores items
When limit=0 is passed to LocalCache (e.g. when DNSCACHE_ENABLED=False),
the condition 'if self.limit' evaluates to False due to Python's truthiness
rules, causing items to be stored despite the cache being disabled.
This leads to an unbounded memory leak during long crawls when DNS caching
is explicitly disabled.
Fix changes the condition to 'if self.limit is not None' and adds an early
return when limit=0 to correctly handle the disabled cache case.
* test: add resolver-level regression tests for DNSCACHE_ENABLED=False
Add two regression tests that verify DNS results are not stored in
dnscache when DNSCACHE_ENABLED=False (cache_size=0):
- test_caching_hostname_resolver_dnscache_disabled_rejects_storage:
verifies _CachingResolutionReceiver does not write to dnscache
when CachingHostnameResolver is initialized with cache_size=0
- test_caching_threaded_resolver_dnscache_disabled_rejects_storage:
verifies dnscache rejects storage at the LocalCache level
when limit=0
* test: drop misleading threaded resolver test
it was writing directly to dnscache, not actually going through
CachingThreadedResolver at all. the threaded resolver already has
its own if dnscache.limit: guard so the bug doesnt affect it anyway.
keeping only the hostname resolver test which covers the actual
bug path through _CachingResolutionReceiver
* test: clean up comments in resolver test
* test: remove unnecessary comment
* Fix request_to_curl() corrupting dict cookies with bytes keys/values
PR #7603 made the list-cookie branch of request_to_curl() bytes-safe via
_cookie_value_to_unicode(), but left the sibling dict-cookie branch using
raw f-string interpolation. A dict cookie with bytes keys/values (a
supported and common form, e.g. Request(url, cookies={b"k": b"v"})) was
rendered as --cookie 'b'k'=b'v'' instead of --cookie 'k=v', producing a
broken curl command.
Route the dict branch through the same _cookie_value_to_unicode() helper,
mirroring the list branch. Add a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: apply _cookie_value_to_unicode to list-branch cookie key/value
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Set TELNETCONSOLE_ENABLED=False in get_crawler().
* Make _get_console_and_portal() a context manager.
* Allow PYTHONTRACEMALLOC in tox.
* Close the event loop in test_custom_asyncio_loop_enabled_false().
* Close the handler in _uninstall_scrapy_root_handler().
* Close queues in test_squeues.py.
* Close empty queues in init_prios().
* Close PopenSpawn stdin and stdout explicitly.
* Silence the unclosed socket warning.
* Implement more methods in CustomStatsCollector.
* Link the Twisted issue.
* Close the connection on download_maxsize.
* Fix typing.
* Add a comment about CustomStatsCollector.
* Restore the coverage.
* Properly close fixture queues.
* More robust file closing in feed storages.
* Close the file in TestMarshalItemExporter.test_nonstring_types_item().
* Cleanup temporary file handling in test_exporters.py.
* Close the file in test_stats_file_failed().
* Use an explicit TextIOWrapper in XmlItemExporter.
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.
Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
* Remove support for sync process_spider_output().
* Slight check fix.
* Don't expect exceptions from calling process_spider_output().
* Remove dead code.
* Fix test_deprecated_mw_spider_arg() to run all methods.
* Fix typing.
* Typos.
* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective
* Fix indentation issue
* Minor rewordings
* older versions → lower versions
* Update news.rst
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Python 3.14 (alpha3) to CI.
* Disable mitmproxy on 3.14 for now.
* 3.14.0-alpha.4.
* 3.14.0-alpha.5
* 3.14.0-beta2.
* 3.14 release.
* Fix test_non_pickable_object.
* Fix handling of file:/path feed URIs.
* Better mocking of streams for TextTestResult.
* Do not use .php in test_file_path() as it's now a known extension.
* Fix the URL in TestFeedExporterSignals.
* Fix typing.
* Bump more envs to 3.14.
* Silence pylint.
* Fix another test for .php handling change.
* Remove test_install_asyncio_reactor.
* More bumps to 3.14.
* Revert docs-tests to use 3.13.
* Debug options for Windows.
* Re-enable xdist.
* Revert Windows PYTEST_ADDOPTS.
* Silence loop policy deprecation warnings.
* Restore a lost pylint suppression.
* Update asyncio_enabled_reactor_same_loop.py to new Twisted.
* Fix RobotFileParser tests for Python 3.14.5.
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).
* Hide the _setAcceptableProtocols import.
* Silence ScrapyClientContextFactory HTTP/1.0 warnings.
* Update ScrapyClientTLSOptions for unreleased Twisted.
* Add twisted-trunk to CI.
* Update for new changes in Twisted trunk.
* Xfail a test failing with newer Twisted.
* Test Twisted trunk with extra-deps, update the ALPN code.
* Cleanup.
* Fix typing.
* Update relevant type hints for Twisted trunk.
* Update test_no_context_sharing().
* Silence a weird mypy error.
* Update Twisted versions, unpin pyOpenSSL.
* Update a comment.
* Silence pylint.
* Make a factory fixture.
* Improve the _setAcceptableProtocols comment.
* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.
* Add Twisted[http2] to extra-deps-pinned.
* Cover 2.15.0 in the release notes (up to 31bf7c3).
* Replace VERSION.
* Document unified download handler exceptions.
* Updates.
* Cover 2.15.0 in the release notes (up to 0c6ccf5).
* Cover 2.15.0 in the release notes (up to b68f267).
* Run linkcheck and fix issues.
* Cleanup unused link targets.
* Cleanup.
* overall prettyfication
- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting
* return `SIM300` to ignore
* apply ruff rules to all files
* another one
* remove extra space in pyproject.toml
* simplify `__getattr__` in `scrapy.utils.url`
* lazy `url_is_from_any_domain` and `url_is_from_spider`
* improve typing
* specify `arg_to_iter` with None as arg
* improve trackref
- use `NoneType` from `types` since python 3.9 is no longer supported
- use `monotonic` instead of `time` and fix flakiness of `get_oldest`
* tracing GC is no joke
* refine tests
* explain the tests
* add note for pypy in docs + ...
return empty tuple instead of list in `iter_all`
* Import some stuff from service_identity directly.
* Extract _log_tls().
* Sync validating IP addresses.
* Introduce _deps_compat.
* Deprecate getCertificateOptions().
* Deprecate ScrapyClientTLSOptions.
* Add pragma: no cover.
* verify=False and trustRoot=platformTrust() are the defaults.
* Improve docstrings for TLS classes.
* Deprecate AcceptableProtocolsContextFactory.
* Switch from DefaultOpenSSLContextFactory to CertificateOptions.
* Deprecate load_context_factory_from_settings().
* Remove the outdated warning about incompatible factories.
* Simplify _load_context_factory_from_settings().
* Make CertificateOptions and Context once per factory.
* pragma: no cover
Andrey Rakhmatullin
Gaurav Yadav
tanishqtayade
greymoth
JSap0914
Syncrain
Nishita Matlani
Adrian
Ethan Kuo
Albert Eduardovich N.
Julian Hamze