mirror of https://github.com/scrapy/scrapy.git
b82c04704c
When limit=0 is passed to LocalCache (e.g. when DNSCACHE_ENABLED=False), the condition 'if self.limit' evaluates to False due to Python's truthiness rules, causing items to be stored despite the cache being disabled. This leads to an unbounded memory leak during long crawls when DNS caching is explicitly disabled. Fix changes the condition to 'if self.limit is not None' and adds an early return when limit=0 to correctly handle the disabled cache case. |
||
|---|---|---|
| .github | ||
| docs | ||
| extras | ||
| scrapy | ||
| sep | ||
| tests | ||
| tests_typing | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .readthedocs.yml | ||
| AUTHORS | ||
| CITATION.cff | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| INSTALL.md | ||
| LICENSE | ||
| NEWS | ||
| README.rst | ||
| SECURITY.md | ||
| codecov.yml | ||
| conftest.py | ||
| pyproject.toml | ||
| tox.ini | ||
README.rst
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
Scrapy is a web scraping framework to extract structured data from websites. It is cross-platform, and requires Python 3.10+. It is maintained by Zyte (formerly Scrapinghub) and many other contributors.
Install with:
System Message: WARNING/2 (<stdin>, line 52)
Cannot analyze code. Pygments package not found.
.. code:: bash
pip install scrapy
And follow the documentation to learn how to use it.
If you wish to contribute, see Contributing.