RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/tests
History
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 		2019-06-24 12:38:05 +02:00
..
keys revert wrong changes 2018-03-13 13:05:37 +08:00
mocks Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
sample_data Add file listing resource + redirecting resource to MockServer 2017-03-10 21:35:25 +01:00
test_cmdline Fix cmdline profiling test on Windows by using proper path composing 2018-08-15 01:35:01 -03:00
test_settings Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
test_spiderloader Do not use self.assertRaises() as context manager 2017-03-09 12:21:03 +01:00
test_utils_misc Restore alphabetic order and two lines between import and code 2019-05-03 13:55:18 +02:00
__init__.py Be consistent with domain used for links to documentation website 2019-01-31 01:28:53 -03:00
constraints.txt TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
mockserver.py fix: do not catch system exceptions like KeyboardInterrupt 2019-04-05 11:52:00 +02:00
pipelines.py add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
py3-ignores.txt Re-enable FTP tests on Python 3 2017-02-27 17:42:00 +01:00
requirements-py2.txt Use pytest-xdist 2019-04-17 09:02:34 +02:00
requirements-py3.txt Use pytest-xdist 2019-04-17 09:02:34 +02:00
spiders.py [MRG+1] Automatic port selection for servicies in unit tests (#3210) 2018-05-23 21:25:50 +03:00
test_closespider.py [MRG+1] Show elapsed time in statscollector (#3638) 2019-04-07 00:03:40 +05:00
test_command_fetch.py Let framework handle only HTTP redirects by default for fetch and shell commands 2016-12-07 17:56:13 +01:00
test_command_parse.py Fix test_command_parse under windows 2018-08-15 01:08:40 -07:00
test_command_shell.py Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
test_command_version.py Fix test 2017-09-25 12:49:27 +02:00
test_commands.py Fix presentation of template directory in startproject command 2018-08-15 01:35:01 -03:00
test_contracts.py Merge pull request #3381 from StasDeep/fix/issue-3380 2018-09-06 11:41:02 -03:00
test_crawl.py [MRG+1] Automatic port selection for servicies in unit tests (#3210) 2018-05-23 21:25:50 +03:00
test_crawler.py report warnings 2019-03-25 13:48:58 +00:00
test_dependencies.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_downloader_handlers.py backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
test_downloadermiddleware.py removes unused var 2019-06-17 00:06:44 +05:30
test_downloadermiddleware_ajaxcrawlable.py PY3 fix AjacCrawlable middleware tests 2015-09-01 03:49:52 +05:00
test_downloadermiddleware_cookies.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_downloadermiddleware_decompression.py PY3 fix test downloadermiddleware decompression 2015-07-29 13:51:01 +00:00
test_downloadermiddleware_defaultheaders.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_downloadermiddleware_downloadtimeout.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_downloadermiddleware_httpauth.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_downloadermiddleware_httpcache.py HTTP Cache: treat 308 as 301 2017-07-26 19:01:48 +02:00
test_downloadermiddleware_httpcompression.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_downloadermiddleware_httpproxy.py Check that spidercls arguments in scrapy.crawler classes are not spider objects 2019-02-12 14:46:27 +01:00
test_downloadermiddleware_redirect.py [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
test_downloadermiddleware_retry.py Improvise the clarity of test cases 2017-03-23 19:45:04 +05:30
test_downloadermiddleware_robotstxt.py Pass user-agent as native str when checking URLs against robots.txt 2016-11-15 17:38:32 +01:00
test_downloadermiddleware_stats.py PY3 fix DownloaderStats middleware tests 2015-09-01 04:09:15 +05:00
test_downloadermiddleware_useragent.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_dupefilters.py Testing stats and log messages from RFPDupeFilter 2019-01-18 11:39:35 -02:00
test_engine.py tests for new signal 2018-08-17 14:39:54 +00:00
test_exporters.py Fixed bug on XMLItemExporter with non-string fields in items 2016-01-30 10:00:06 -02:00
test_extension_telnet.py Style changes 2018-09-26 13:28:34 -03:00
test_feedexport.py Merge branch 'master' into feed-storage-s3-acl 2019-03-15 13:26:45 -03:00
test_http_cookies.py Update asserts to use more generic ones 2017-08-09 12:52:10 +02:00
test_http_headers.py normalize header values to bytes 2014-08-01 00:02:50 -03:00
test_http_request.py made jsonrequest dump into private method 2019-01-14 23:03:14 +05:30
test_http_response.py TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
test_item.py Implement Item.deepcopy() 2019-03-22 17:07:10 +01:00
test_link.py PY3 raise an exception if bytes are passed as url to Link constructor 2015-08-28 23:04:02 +05:00
test_linkextractors.py New constructor arg *restrict_text* for FilteringLinkExtractor. 2019-02-28 17:21:17 +01:00
test_linkextractors_deprecated.py TST fixed compatibility with new link extractor whitespace handling 2017-02-21 00:05:40 +05:00
test_loader.py Linting (return previous indentation). 2019-06-11 17:38:06 +03:00
test_logformatter.py [logformatter] backward compat comments 2017-03-17 08:11:20 +01:00
test_mail.py (fixes #2272) using arg_to_iter() to wrap single values and list() to avoid consuming from generators. 2016-10-18 11:06:55 -03:00
test_middleware.py Use collections.deque instead of list to store methods 2018-10-29 12:40:20 -03:00
test_pipeline_crawl.py TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
test_pipeline_files.py added test for gcs policy 2018-04-03 18:06:34 +09:00
test_pipeline_images.py TST remove workaround for old Pillow versions which don't support BytesIO 2018-06-28 01:01:27 +05:00
test_pipeline_media.py Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 2019-06-24 12:38:05 +02:00
test_proxy_connect.py [MRG+1] Automatic port selection for servicies in unit tests (#3210) 2018-05-23 21:25:50 +03:00
test_pydispatch_deprecated.py add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
test_responsetypes.py Merge pull request #1503 from demelziraptor/amazon-json-response 2016-09-12 13:21:16 -03:00
test_scheduler.py improve tests and fix some lint warnings (#6) 2019-03-25 16:08:40 +03:00
test_selector.py TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
test_selector_csstranslator.py cleanup tests for selectors and translators 2015-08-11 14:09:34 -03:00
test_spider.py Rule.process_request: move deprecation warnings and compiling code, update tests 2019-03-22 19:46:17 -03:00
test_spidermiddleware.py Update docs, add tests, remove FIXME comment 2018-08-03 18:20:25 -03:00
test_spidermiddleware_depth.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_spidermiddleware_httperror.py [MRG+1] Automatic port selection for servicies in unit tests (#3210) 2018-05-23 21:25:50 +03:00
test_spidermiddleware_offsite.py checking for subclass of URLWarning instead of checking error message text when URL in allowed_domains 2017-11-26 20:07:04 +01:00
test_spidermiddleware_output_chain.py Update docs, add tests, remove FIXME comment 2018-08-03 18:20:25 -03:00
test_spidermiddleware_referer.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_spidermiddleware_urllength.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_spiderstate.py TST remove temporary files and folders 2017-02-16 17:50:38 +05:00
test_squeues.py Update test for unpickable objects 2018-02-08 15:03:57 -03:00
test_stats.py enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
test_toplevel.py drop deprecated "optional_features" set 2015-09-15 09:00:08 +00:00
test_urlparse_monkeypatches.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_utils_conf.py backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
test_utils_console.py Add support for bpython console. 2015-08-21 01:12:58 +01:00
test_utils_datatypes.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_utils_defer.py Update deprecated test aliases 2017-08-09 12:52:10 +02:00
test_utils_deprecate.py Ignore ScrapyDeprecationWarning warnings properly. 2015-06-09 18:22:03 +03:00
test_utils_gz.py Modifying existing gzip read failure recovery mechanism to patch read for broken archives 2016-08-11 18:21:42 +05:30
test_utils_http.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_utils_httpobj.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_utils_iterators.py DOC address @stummjr's review comments 2018-09-18 05:02:17 +05:00
test_utils_log.py Fix telnet warnings in tests 2018-07-09 13:44:02 +03:00
test_utils_project.py Fix test_utils_project under Windows 2018-08-15 01:53:20 -03:00
test_utils_python.py Add tests for MutableChain 2018-10-11 11:55:13 -03:00
test_utils_reqser.py Fix mangling test 2019-06-06 04:25:19 +09:00
test_utils_request.py Do not fail representing non-http requests 2015-08-10 14:10:58 -03:00
test_utils_response.py response_status_message should not fail on non-standard HTTP codes 2016-03-12 14:16:40 +01:00
test_utils_serialize.py Add set serialization to ScrapyJSONEncoder 2016-07-22 17:20:03 -04:00
test_utils_signal.py Update asserts to use more generic ones 2017-08-09 12:52:10 +02:00
test_utils_sitemap.py handle relative sitemap urls in robots.txt 2016-12-01 09:53:40 -03:00
test_utils_spider.py Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
test_utils_template.py Test utils.template.render_templatefile() 2015-08-26 20:44:33 +03:00
test_utils_trackref.py PY3: port scrapy.utils.trackref 2015-08-10 15:13:29 -03:00
test_utils_url.py Fix strip_url() tests 2017-03-01 20:59:52 +01:00
test_webclient.py Update asserts to use more generic ones 2017-08-09 12:52:10 +02:00