`get_component_priority_dict_with_base()` normalizes every key of a
component priority dictionary by calling `load_object()` on it, including
keys whose value is `None`, i.e. components the user is disabling.
`normalize_key()` only guarded against `(NameError, TypeError, ValueError)`,
the three exceptions `load_object()` raises itself, but not against the
`ImportError` raised by the `import_module()` call it makes. A stale entry
disabling a component whose module no longer exists, e.g.
`{"scrapy.webservice.WebService": None}`, therefore aborted settings
resolution with `ModuleNotFoundError` instead of being ignored.
Add `ImportError` to the guard so that an unimportable key falls back to its
raw string form. Since its value is `None`, it is then dropped by the
existing `if v is not None` filter, and disabling by import path keeps
working. Enabled components with an unimportable path are unaffected:
`MiddlewareManager.from_crawler()` calls `load_object()` on them again and
still fails there.
Fixes#7820
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Since Python 3.14 (PEP 649) annotations are evaluated lazily, so
inspect.signature() raises NameError for callables whose annotations
reference names imported only under TYPE_CHECKING. This broke
middleware registration (via argument_is_required()) and custom stats
collectors (via _warn_spider_arg) for user code with such annotations.
Use annotation_format=Format.FORWARDREF on 3.14+: parameter names,
kinds and defaults are unchanged, and unresolvable annotations become
ForwardRef proxies instead of raising.
Resolves#7796.
* Store full feed batches before spider closes (#7730)
Start closing and storing each batch as soon as it reaches the configured item count. Track unfinished close tasks so spider shutdown still waits for all deliveries before emitting the exporter-closed signal.
Add an end-to-end regression test that verifies the first batch is stored while the crawl is still running.
* Remove the issue reference
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Skip the deprecated scrapy.mail in coverage data
* Improve CrawlSpider coverage
* Improve XMLFeedSpider coverage
* Improve SitemapSpider coverage
* Solve mypy issues
* Align new spider tests with the shared test helper structure
* Fix CaseInsensitiveDict.copy() sharing state with the original
* Address review: don't re-normalise in __copy__, keep _keys in sync in __ior__
UserDict.__ior__ writes self.data directly, bypassing __setitem__, so
_keys never learned about the new keys.
Mohammad Hassan
Adrian
Mridankan Mandal
Laerte Pereira
Janit Rajkarnikar
Fandu
Youssef Mohamed
Andrey Rakhmatullin
Shadow_Lu