RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/scrapy
History
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 		2019-06-24 12:38:05 +02:00
..
commands Try it with a string 2019-04-10 13:01:01 +02:00
contracts fix: do not catch system exceptions like KeyboardInterrupt 2019-04-05 11:52:00 +02:00
core Merge pull request #3520 from whalebot-helmsman/round-robin-scheduler-tested 2019-04-07 00:06:08 +05:00
downloadermiddlewares [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
extensions Merge pull request #3735 from anubhavp28/smallfix 2019-04-13 01:52:22 +05:00
http enabled sort keys only if not provided 2019-01-25 21:26:28 +05:30
linkextractors New constructor arg *restrict_text* for FilteringLinkExtractor. 2019-02-28 17:21:17 +01:00
loader Add tests. 2019-06-11 07:36:29 +03:00
pipelines Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 2019-06-24 12:38:05 +02:00
selector DOC use autodocs for selectors; document more methods and attributes; suggest get/getall 2018-09-12 18:36:25 +05:00
settings Add 429 to RETRY_HTTP_CODES 2019-05-24 11:11:15 -03:00
spidermiddlewares PEP8 ofsite middleware 2018-08-24 15:18:16 -04:00
spiders Rule.process_request: move deprecation warnings and compiling code, update tests 2019-03-22 19:46:17 -03:00
templates fix typo (Response -> Request) 2019-04-05 14:31:15 -03:00
utils Merge pull request #3790 from andrewbaxter/master 2019-06-17 12:33:21 -03:00
xlib Wipe scrapy.xlib.tx 2016-12-02 21:22:35 +00:00
VERSION Bump version: 1.5.0 → 1.6.0 2019-01-30 18:00:40 -03:00
__init__.py drop deprecated "optional_features" set 2015-09-15 09:00:08 +00:00
__main__.py Add support for executing scrapy using -m option of python 2017-05-15 22:52:23 -03:00
_monkeypatches.py Use https links wherever possible 2017-10-28 16:24:40 +05:30
cmdline.py backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
conf.py backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
crawler.py Merge pull request #3610 from Gallaecio/spidercls-check 2019-03-15 08:25:16 +01:00
dupefilters.py Logging the request referer when DUPEFILTER_DEBUG is active 2019-01-18 11:38:59 -02:00
exceptions.py Undocument _InvalidOutput exception 2017-03-10 15:41:57 -03:00
exporters.py Fix for CSV export unnecessary blank lines problem on Windows (#3039) 2018-07-03 18:22:24 -03:00
extension.py Replace BaseSettings._getcomposite() with public .getwithbase() method 2015-11-11 17:39:57 +01:00
interfaces.py SpiderManager shim is removed 2018-07-06 03:28:01 +05:00
item.py Implement Item.deepcopy() 2019-03-22 17:07:10 +01:00
link.py PY3 raise an exception if bytes are passed as url to Link constructor 2015-08-28 23:04:02 +05:00
log.py backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
logformatter.py Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
mail.py Use a helper for to_bytes() and None input 2017-10-05 10:16:42 +02:00
middleware.py Use collections.deque instead of list to store methods 2018-10-29 12:40:20 -03:00
mime.types moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes 2011-08-07 02:49:57 -03:00
pqueues.py improve tests and fix some lint warnings (#6) 2019-03-25 16:08:40 +03:00
resolver.py Do not cache DNS responses when cache size is 0 2017-07-04 23:18:15 +02:00
responsetypes.py Merge pull request #1503 from demelziraptor/amazon-json-response 2016-09-12 13:21:16 -03:00
shell.py Populate spider variable when using shell.inspect_response 2017-07-03 11:28:04 -03:00
signalmanager.py Use https links wherever possible 2017-10-28 16:24:40 +05:30
signals.py Revert "new signal" 2019-03-25 16:08:40 +03:00
spiderloader.py Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning 2017-03-07 17:40:40 +01:00
squeues.py assorted cleanups: comments, docstrings, etc 2019-03-25 16:08:40 +03:00
statscollectors.py enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
telnet.py Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00