mirror of https://github.com/scrapy/scrapy.git
6591cb756c
* Fix cell-var-from-loop in _send_catch_log_deferred Replace lambda capturing receiver by reference with a default argument to capture it by value, fixing a potential bug where all deferred callbacks could reference the last receiver in the loop instead of their respective receivers. Removes the pylint disable comment and TODO that were suppressing this issue. * chore: trigger CI rerun for mypy network error * Fix mypy error: pass receiver via addBoth args instead of lambda default * style: apply pre-commit ruff formatting * fix: LocalCache with limit=0 incorrectly stores items When limit=0 is passed to LocalCache (e.g. when DNSCACHE_ENABLED=False), the condition 'if self.limit' evaluates to False due to Python's truthiness rules, causing items to be stored despite the cache being disabled. This leads to an unbounded memory leak during long crawls when DNS caching is explicitly disabled. Fix changes the condition to 'if self.limit is not None' and adds an early return when limit=0 to correctly handle the disabled cache case. * test: add resolver-level regression tests for DNSCACHE_ENABLED=False Add two regression tests that verify DNS results are not stored in dnscache when DNSCACHE_ENABLED=False (cache_size=0): - test_caching_hostname_resolver_dnscache_disabled_rejects_storage: verifies _CachingResolutionReceiver does not write to dnscache when CachingHostnameResolver is initialized with cache_size=0 - test_caching_threaded_resolver_dnscache_disabled_rejects_storage: verifies dnscache rejects storage at the LocalCache level when limit=0 * test: drop misleading threaded resolver test it was writing directly to dnscache, not actually going through CachingThreadedResolver at all. the threaded resolver already has its own if dnscache.limit: guard so the bug doesnt affect it anyway. keeping only the hostname resolver test which covers the actual bug path through _CachingResolutionReceiver * test: clean up comments in resolver test * test: remove unnecessary comment |
||
|---|---|---|
| .github | ||
| docs | ||
| extras | ||
| scrapy | ||
| sep | ||
| tests | ||
| tests_typing | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .readthedocs.yml | ||
| AUTHORS | ||
| CITATION.cff | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| INSTALL.md | ||
| LICENSE | ||
| NEWS | ||
| README.rst | ||
| SECURITY.md | ||
| codecov.yml | ||
| conftest.py | ||
| pyproject.toml | ||
| tox.ini | ||
README.rst
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
Scrapy is a web scraping framework to extract structured data from websites. It is cross-platform, and requires Python 3.10+. It is maintained by Zyte (formerly Scrapinghub) and many other contributors.
Install with:
System Message: WARNING/2 (<stdin>, line 52)
Cannot analyze code. Pygments package not found.
.. code:: bash
pip install scrapy
And follow the documentation to learn how to use it.
If you wish to contribute, see Contributing.