RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/tests
History
tanishqtayade 6591cb756c
Fix to LocalCache with limit=0 still stores items rather than disabling the cache (#7663) 
* Fix cell-var-from-loop in _send_catch_log_deferred

Replace lambda capturing receiver by reference with a default
argument to capture it by value, fixing a potential bug where
all deferred callbacks could reference the last receiver in the
loop instead of their respective receivers.

Removes the pylint disable comment and TODO that were suppressing
this issue.

* chore: trigger CI rerun for mypy network error

* Fix mypy error: pass receiver via addBoth args instead of lambda default

* style: apply pre-commit ruff formatting

* fix: LocalCache with limit=0 incorrectly stores items

When limit=0 is passed to LocalCache (e.g. when DNSCACHE_ENABLED=False),
the condition 'if self.limit' evaluates to False due to Python's truthiness
rules, causing items to be stored despite the cache being disabled.

This leads to an unbounded memory leak during long crawls when DNS caching
is explicitly disabled.

Fix changes the condition to 'if self.limit is not None' and adds an early
return when limit=0 to correctly handle the disabled cache case.

* test: add resolver-level regression tests for DNSCACHE_ENABLED=False

Add two regression tests that verify DNS results are not stored in
dnscache when DNSCACHE_ENABLED=False (cache_size=0):

- test_caching_hostname_resolver_dnscache_disabled_rejects_storage:
  verifies _CachingResolutionReceiver does not write to dnscache
  when CachingHostnameResolver is initialized with cache_size=0

- test_caching_threaded_resolver_dnscache_disabled_rejects_storage:
  verifies dnscache rejects storage at the LocalCache level
  when limit=0

* test: drop misleading threaded resolver test

it was writing directly to dnscache, not actually going through
CachingThreadedResolver at all. the threaded resolver already has
its own if dnscache.limit: guard so the bug doesnt affect it anyway.

keeping only the hostname resolver test which covers the actual
bug path through _CachingResolutionReceiver

* test: clean up comments in resolver test

* test: remove unnecessary comment
 		2026-06-29 21:37:06 +05:00
..
AsyncCrawlerProcess Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
AsyncCrawlerRunner Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
CrawlerProcess Add Python 3.14 to CI. (#6604) 2026-05-12 23:49:37 +05:00
CrawlerRunner Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
keys Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
mocks Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
mockserver Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
sample_data Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
test_cmdline Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_cmdline_crawl_with_pipeline Fix open_spider exception handling to set exit code 1 (#7255) 2026-02-19 11:21:59 +01:00
test_settings Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
test_spiderloader Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_utils_misc Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
utils tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
__init__.py TLS code modernization (#7353) 2026-03-25 15:42:43 +05:00
ignores.txt Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
pipelines.py Deprecate spider args of middlewares and pipelines. (#7006) 2025-08-21 13:23:10 +05:00
spiders.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_addons.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_closespider.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_command_check.py Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00
test_command_crawl.py Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
test_command_fetch.py Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
test_command_genspider.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
test_command_parse.py Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
test_command_runspider.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_command_shell.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_command_startproject.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_command_version.py Speed up and refactor command tests (#7118) 2025-10-27 12:28:16 +01:00
test_commands.py Deprecate ScrapyCommand.help() (#7633) 2026-06-19 06:49:32 +02:00
test_contracts.py Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
test_core_downloader.py Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 2026-06-25 16:44:51 +05:00
test_core_scraper.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_crawl.py Do not ignore CrawlerProcess settings (#7647) 2026-06-22 20:10:10 +05:00
test_crawler.py Do not ignore CrawlerProcess settings (#7647) 2026-06-22 20:10:10 +05:00
test_crawler_subprocess.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_dependencies.py tox.ini: pinned, typing → min, mypy (#7595) 2026-06-11 12:10:48 +05:00
test_downloader_handler_httpx.py Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575) 2026-06-09 01:10:44 +05:00
test_downloader_handler_twisted_ftp.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloader_handler_twisted_http2.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloader_handler_twisted_http11.py Add settings for TLS min/max version as a replacement for the TLS method (#6546) 2026-06-08 11:54:10 +02:00
test_downloader_handlers.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloader_handlers_http_base.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_downloadermiddleware.py Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
test_downloadermiddleware_cookies.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_defaultheaders.py Deprecate the spider arg of downloader middleware methods. (#7037) 2025-09-02 21:51:47 +05:00
test_downloadermiddleware_downloadtimeout.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_httpauth.py Implement HTTP Auth settings and request metadata keys (#7590) 2026-06-11 15:56:53 +05:00
test_downloadermiddleware_httpcache.py Solve timing issues with HTTP cache tests? (#7692) 2026-06-29 17:21:20 +05:00
test_downloadermiddleware_httpcompression.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_httpproxy.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_offsite.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_redirect.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_redirect_base.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_downloadermiddleware_redirect_metarefresh.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_retry.py Allow configuring the log level of the retry give-up message (#7567) 2026-06-08 18:44:30 +05:00
test_downloadermiddleware_robotstxt.py Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
test_downloadermiddleware_stats.py Improve test coverage for downloader middlewares (#7655) 2026-06-25 01:48:28 +05:00
test_downloadermiddleware_useragent.py Deprecate spider attributes that can be replaced by settings, round 2 (#7039) 2025-12-01 01:26:17 +05:00
test_downloaderslotssettings.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_dupefilters.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_engine.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_engine_loop.py fix typos (#7564) 2026-06-02 11:04:38 +02:00
test_engine_stop_download_bytes.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_engine_stop_download_headers.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_exporters.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_extension_memusage.py tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
test_extension_periodic_log.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_extension_statsmailer.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_extension_telnet.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_extension_throttle.py Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
test_feedexport.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_feedexport_batch.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_feedexport_postprocess.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_feedexport_storages.py Close GCS feed temp files after upload (#7546) 2026-06-04 20:11:50 +05:00
test_feedexport_uri_params.py Split long test files (#7329) 2026-03-13 10:30:25 +01:00
test_http2_client_protocol.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_http_cookies.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_headers.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_request.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_request_form.py Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
test_http_request_json.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_http_response.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_http_response_text.py Improve test coverage for http/ (#7672) 2026-06-26 12:21:12 +05:00
test_item.py Add coverage for Item.__delitem__() (#7610) 2026-06-12 15:32:38 +05:00
test_link.py Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
test_linkextractors.py Complete test coverage for linkextractors (#7639) 2026-06-19 14:22:03 +05:00
test_loader.py Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
test_logformatter.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_logstats.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_mail.py Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
test_middleware.py Deprecate the spider arg of downloader middleware methods. (#7037) 2025-09-02 21:51:47 +05:00
test_pipeline_crawl.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_pipeline_files.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_pipeline_images.py Fix image store ACL settings for S3 and GCS (#7614) 2026-06-16 00:24:44 +05:00
test_pipeline_media.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_pipelines.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_poet.py Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 2025-02-14 20:40:06 +05:00
test_pqueues.py Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
test_request_attribute_binding.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_request_cb_kwargs.py Complete test coverage for linkextractors (#7639) 2026-06-19 14:22:03 +05:00
test_request_dict.py Deprecate FormRequest in favor of form2request (#6438) 2026-05-04 22:38:29 +05:00
test_request_left.py Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
test_resolver.py Fix to LocalCache with limit=0 still stores items rather than disabling the cache (#7663) 2026-06-29 21:37:06 +05:00
test_responsetypes.py Improve test coverage for responsetypes.py (#7646) 2026-06-22 20:10:38 +05:00
test_robotstxt_interface.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_scheduler.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_scheduler_base.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_scrapy__getattr__.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_selector.py Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
test_signalmanager.py Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
test_signals.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_spider.py Complete test coverage for linkextractors (#7639) 2026-06-19 14:22:03 +05:00
test_spider_crawl.py Document scrapy-lint, remove start_url check (#7627) 2026-06-22 20:12:16 +05:00
test_spider_sitemap.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_spider_start.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_spidermiddleware.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_spidermiddleware_base.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_depth.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_httperror.py Convert tests that fail with testfixtures 12.0.0 (#7545) 2026-06-03 09:16:24 +02:00
test_spidermiddleware_output_chain.py Remove support for sync process_spider_output(). (#7504) 2026-05-15 16:36:05 +05:00
test_spidermiddleware_process_start.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_spidermiddleware_referer.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_start.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_urllength.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spiderstate.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_squeues.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_squeues_request.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_stats.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_toplevel.py Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
test_utils_asyncgen.py Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
test_utils_asyncio.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_utils_conf.py Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
test_utils_console.py Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
test_utils_curl.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_utils_datatypes.py Fix to LocalCache with limit=0 still stores items rather than disabling the cache (#7663) 2026-06-29 21:37:06 +05:00
test_utils_decorators.py Add test coverage for scrapy.utils.decorators (#7645) 2026-06-22 08:50:11 +02:00
test_utils_defer.py Add more no-reactor tests (#7259) 2026-03-24 15:29:22 +05:00
test_utils_deprecate.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_utils_display.py Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
test_utils_gz.py Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
test_utils_httpobj.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_utils_iterators.py Move to mypy --strict with exceptions. (#7300) 2026-03-02 15:47:23 +05:00
test_utils_log.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_utils_project.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_utils_python.py Complete test coverage for linkextractors (#7639) 2026-06-19 14:22:03 +05:00
test_utils_reactor.py Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00
test_utils_request.py Fix request_to_curl() corrupting dict cookies with bytes keys/values (#7675) 2026-06-27 00:53:32 +05:00
test_utils_response.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_utils_serialize.py Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
test_utils_signal.py Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
test_utils_sitemap.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_utils_spider.py Bump ruff, fix some of rules (#7277) 2026-02-23 11:48:38 +01:00
test_utils_template.py Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
test_utils_trackref.py improve trackref (#7375) 2026-04-03 15:15:30 +05:00
test_utils_url.py Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
test_zz_resources.py Assorted test fixes (#7585) 2026-06-10 11:01:06 +02:00