f4f2b1695c
We're storing exceptions captured by Twisted on the media pipeline cache, but we're also using the defer.returnValue method with our own methods decorated with @defer.inlineCallbacks. The defer.returnValue method passes returned values forward by throwing a defer._DefGen_Return exception, which in its turn extends the BaseException class and is captured by Twisted. This way, the latest exception stored in the Failure's object may also have an HtmlResponse object in its __context__ attribute. As the Response object also keeps track of the Request object that has originated it, you could figure it out how many RAM we're wasting here. This could easily lead to a Memory Leak problem when running spiders with Media Pipeline enabled and a particular Request set that tends to raise a significant number of exceptions. Example triggers: - media requests with 404 status responses - user land exceptins coming from custom middlewares - etc. |
||
|---|---|---|
| artwork | ||
| debian | ||
| docs | ||
| extras | ||
| scrapy | ||
| sep | ||
| tests | ||
| .bumpversion.cfg | ||
| .coveragerc | ||
| .gitignore | ||
| .travis.yml | ||
| AUTHORS | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| INSTALL | ||
| LICENSE | ||
| MANIFEST.in | ||
| Makefile.buildbot | ||
| NEWS | ||
| README.rst | ||
| appveyor.yml | ||
| codecov.yml | ||
| conftest.py | ||
| pytest.ini | ||
| requirements-py2.txt | ||
| requirements-py3.txt | ||
| setup.cfg | ||
| setup.py | ||
| tox.ini | ||
README.rst
Scrapy
Overview
Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
For more information including a list of features check the Scrapy homepage at: https://scrapy.org
Requirements
- Python 2.7 or Python 3.4+
- Works on Linux, Windows, Mac OSX, BSD
Install
The quick way:
pip install scrapy
For more details see the install section in the documentation: https://docs.scrapy.org/en/latest/intro/install.html
Documentation
Documentation is available online at https://docs.scrapy.org/ and in the docs directory.
Releases
You can find release notes at https://docs.scrapy.org/en/latest/news.html
Community (blog, twitter, mail list, IRC)
Contributing
See https://docs.scrapy.org/en/master/contributing.html
Code of Conduct
Please note that this project is released with a Contributor Code of Conduct (see https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md).
By participating in this project you agree to abide by its terms. Please report unacceptable behavior to opensource@scrapinghub.com.