RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/tests
History
Mohammad Hassan f7382561cf Do not fail to disable a component with an unimportable path 
`get_component_priority_dict_with_base()` normalizes every key of a
component priority dictionary by calling `load_object()` on it, including
keys whose value is `None`, i.e. components the user is disabling.

`normalize_key()` only guarded against `(NameError, TypeError, ValueError)`,
the three exceptions `load_object()` raises itself, but not against the
`ImportError` raised by the `import_module()` call it makes. A stale entry
disabling a component whose module no longer exists, e.g.
`{"scrapy.webservice.WebService": None}`, therefore aborted settings
resolution with `ModuleNotFoundError` instead of being ignored.

Add `ImportError` to the guard so that an unimportable key falls back to its
raw string form. Since its value is `None`, it is then dropped by the
existing `if v is not None` filter, and disabling by import path keeps
working. Enabled components with an unimportable path are unaffected:
`MiddlewareManager.from_crawler()` calls `load_object()` on them again and
still fails there.

Fixes #7820

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 		2026-08-09 20:08:58 -04:00
..
AsyncCrawlerProcess Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
AsyncCrawlerRunner Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
CrawlerProcess Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
CrawlerRunner More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
benchmarks Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
keys Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
mocks Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
mockserver Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
sample_data Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
test_cmdline Simplify cmdline settings test (#7853) 2026-08-09 14:55:58 +05:00
test_cmdline_crawl_with_pipeline More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_settings Do not fail to disable a component with an unimportable path 2026-08-09 20:08:58 -04:00
test_spiderloader Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
test_utils_misc Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
utils Warn only once per request about the download warn size (#7963) 2026-08-09 21:14:20 +05:00
__init__.py TLS code modernization (#7353) 2026-03-25 15:42:43 +05:00
ignores.txt Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
pipelines.py Deprecate spider args of middlewares and pipelines. (#7006) 2025-08-21 13:23:10 +05:00
spiders.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_addons.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_closespider.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_command_check.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_command_crawl.py Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
test_command_fetch.py Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
test_command_genspider.py Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
test_command_parse.py Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
test_command_runspider.py Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
test_command_shell.py Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
test_command_startproject.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_command_version.py Speed up and refactor command tests (#7118) 2025-10-27 12:28:16 +01:00
test_commands.py Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00
test_contracts.py Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
test_core_downloader.py Type tests.mockserver and other resource-defining files (#7865) 2026-08-03 23:44:28 +05:00
test_core_scraper.py Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
test_crawl.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_crawler.py Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 2026-08-03 23:46:41 +05:00
test_crawler_runners.py Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
test_crawler_subprocess.py Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
test_dependencies.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloader_handler_httpx.py Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
test_downloader_handler_twisted_ftp.py Type tests.mockserver and other resource-defining files (#7865) 2026-08-03 23:44:28 +05:00
test_downloader_handler_twisted_http2.py Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
test_downloader_handler_twisted_http11.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_downloader_handlers.py Deprecate is_secure=False for s3://. (#7738) 2026-07-20 14:59:11 +02:00
test_downloadermiddleware.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_cookies.py Fix cookie lookup for dotless hosts and IP addresses (#7900) 2026-08-09 16:52:32 +05:00
test_downloadermiddleware_defaultheaders.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_downloadtimeout.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloadermiddleware_httpauth.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_downloadermiddleware_httpcache.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_downloadermiddleware_httpcompression.py Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
test_downloadermiddleware_httpproxy.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_downloadermiddleware_offsite.py Document how to disallow subdomains of allowed domains (#7903) 2026-08-09 23:05:43 +05:00
test_downloadermiddleware_redirect.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_downloadermiddleware_redirect_metarefresh.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_downloadermiddleware_retry.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_downloadermiddleware_robotstxt.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_downloadermiddleware_stats.py Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
test_downloadermiddleware_useragent.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_downloaderslotssettings.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_dupefilters.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_engine.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_engine_close_spider.py Test restructuring (#7736) 2026-07-21 12:44:56 +02:00
test_engine_download.py Cover passing spider to a deprecated Downloader.fetch() (#7863) 2026-08-03 13:55:39 +02:00
test_engine_loop.py Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
test_engine_stop_download_bytes.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_engine_stop_download_headers.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_exporters.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_extension_debug.py Improve a flaky test_stacktracedump_dump_stacktrace(). (#7753) 2026-07-21 13:17:56 +05:00
test_extension_memdebug.py Tests for extensions.debug and extensions.memdebug. (#7749) 2026-07-20 15:21:28 +02:00
test_extension_memusage.py Improve test coverage for scrapy.extensions (#7809) 2026-07-30 20:27:27 +05:00
test_extension_periodic_log.py Improve test coverage for scrapy.extensions (#7809) 2026-07-30 20:27:27 +05:00
test_extension_statsmailer.py Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
test_extension_telnet.py Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
test_extension_throttle.py Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 2026-08-03 23:46:41 +05:00
test_feedexport.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_feedexport_batch.py Fix immediate delivery of full feed export batches (#7730) (#7733) 2026-07-29 19:30:03 +05:00
test_feedexport_postprocess.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_feedexport_storages.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_feedexport_uri_params.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_http2_client_protocol.py Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
test_http_cookies.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_http_headers.py Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
test_http_request.py Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
test_http_request_form.py Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
test_http_request_json.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_http_response.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_http_response_text.py Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
test_item.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_link.py Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
test_linkextractors.py Canonicalize each extracted link once (#7961) 2026-08-09 22:30:30 +05:00
test_loader.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_logformatter.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_logstats.py Add type hints to test_link.py and test_downloadermiddleware_stats.py (#7785) 2026-07-27 10:07:32 +02:00
test_mail.py Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
test_middleware.py Deprecate the spider arg of downloader middleware methods. (#7037) 2025-09-02 21:51:47 +05:00
test_pipeline_crawl.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_pipeline_files.py Document how to customize media pipeline file names from responses (#7909) 2026-08-09 23:08:59 +05:00
test_pipeline_images.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_pipeline_media.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_pipelines.py Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
test_poet.py Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 2025-02-14 20:40:06 +05:00
test_pqueues.py Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
test_request_attribute_binding.py Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
test_request_cb_kwargs.py Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
test_request_dict.py Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
test_request_left.py Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
test_resolver.py Fix port issue with cached DNS (#7772) 2026-07-27 21:23:56 +05:00
test_responsetypes.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_robotstxt_interface.py Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
test_scheduler.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_scheduler_base.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_scrapy__getattr__.py Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
test_selector.py Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
test_signalmanager.py Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
test_signals.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spider.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_spider_crawl.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_spider_sitemap.py Improve spiders coverage (#7768) 2026-07-28 15:24:04 +05:00
test_spider_start.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_base.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spidermiddleware_depth.py Add a depth_reset request metadata key (#7913) 2026-08-09 23:07:10 +05:00
test_spidermiddleware_httperror.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_metacopy.py Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
test_spidermiddleware_output_chain.py Migrate away from testfixtures. (#7793) 2026-07-28 09:52:52 +02:00
test_spidermiddleware_process_start.py Remove remaining cross-imports in test modules. (#7782) 2026-07-27 12:19:12 +02:00
test_spidermiddleware_referer.py Assorted test fixes, part 3. (#7741) 2026-07-21 15:54:43 +05:00
test_spidermiddleware_start.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_spidermiddleware_urllength.py Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
test_spiderstate.py Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
test_squeues.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_squeues_request.py Close various garbage collectible resources (mostly in tests) (#7644) 2026-06-26 23:05:08 +05:00
test_stats.py Assorted test fixes, part 3. (#7741) 2026-07-21 15:54:43 +05:00
test_toplevel.py Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
test_utils_asyncgen.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_asyncio.py Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
test_utils_conf.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_console.py fix(shell): Run IPython prompt in a thread under a running loop (#7816) 2026-07-31 18:05:22 +02:00
test_utils_curl.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_datatypes.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_decorators.py Don't evaluate annotations when inspecting signatures (#7818) 2026-07-30 20:18:45 +05:00
test_utils_defer.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_deprecate.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_display.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_gz.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_httpobj.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_iterators.py Remove deprecated xmliter(), deprecate re_rsearch() (#7765) 2026-07-23 10:38:56 +02:00
test_utils_log.py Do not mangle log messages when a log formatter returns empty or tuple args (#7936) 2026-08-09 22:58:35 +05:00
test_utils_project.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_python.py Don't evaluate annotations when inspecting signatures (#7818) 2026-07-30 20:18:45 +05:00
test_utils_reactor.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_reactorless.py Reactorless import hook improvements. (#7747) 2026-07-21 19:58:40 +05:00
test_utils_request.py Add Request.to_curl() (#7743) (#7802) 2026-07-29 13:47:15 +05:00
test_utils_response.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_serialize.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_signal.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_sitemap.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_spider.py More granular untyped defs checking in tests. (#7712) 2026-07-08 10:05:45 +02:00
test_utils_template.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_trackref.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_utils_url.py Next refactoring pass of test_utils_*. (#7797) 2026-07-28 14:24:34 +02:00
test_zz_resources.py Assorted test fixes (#7585) 2026-06-10 11:01:06 +02:00