RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,324 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

11324 Commits

Author SHA1 Message Date
Mohammad Hassan f7382561cf Do not fail to disable a component with an unimportable path 
`get_component_priority_dict_with_base()` normalizes every key of a
component priority dictionary by calling `load_object()` on it, including
keys whose value is `None`, i.e. components the user is disabling.

`normalize_key()` only guarded against `(NameError, TypeError, ValueError)`,
the three exceptions `load_object()` raises itself, but not against the
`ImportError` raised by the `import_module()` call it makes. A stale entry
disabling a component whose module no longer exists, e.g.
`{"scrapy.webservice.WebService": None}`, therefore aborted settings
resolution with `ModuleNotFoundError` instead of being ignored.

Add `ImportError` to the guard so that an unimportable key falls back to its
raw string form. Since its value is `None`, it is then dropped by the
existing `if v is not None` filter, and disabling by import path keeps
working. Enabled components with an unimportable path are unaffected:
`MiddlewareManager.from_crawler()` calls `load_object()` on them again and
still fails there.

Fixes #7820

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-09 20:08:58 -04:00
Adrian ae68786210
Document how to customize media pipeline file names from responses (#7909) 2026-08-09 23:08:59 +05:00
Adrian 5270f3cf99
Add a depth_reset request metadata key (#7913) 2026-08-09 23:07:10 +05:00
Adrian d41bfaec05
Document how to disallow subdomains of allowed domains (#7903) 2026-08-09 23:05:43 +05:00
Adrian ba29ce8d3b
Add a docs section on using download_async() from a downloader middleware (#7872) 2026-08-09 23:02:44 +05:00
Adrian fc082aa914
Clarify the docs about reactor settings (#7880) 2026-08-09 23:01:31 +05:00
Adrian aadfd15327
Do not mangle log messages when a log formatter returns empty or tuple args (#7936) 2026-08-09 22:58:35 +05:00
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian a5614544a2
Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
Adrian 482a02d30c
Add a cookies documentation page (#7947) 2026-08-09 22:41:32 +05:00
Adrian 786ab10494
Document how to write a custom item exporter (#7931) 2026-08-09 22:31:53 +05:00
Adrian 7d9516f332
Canonicalize each extracted link once (#7961) 2026-08-09 22:30:30 +05:00
Adrian 609f64c55d
Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
Adrian 5427080f48
Create a page on optimization (#7938) 2026-08-09 18:37:24 +02:00
Adrian 8e94df6526
Time out hanging tests in the VCS dependencies job (#7960) 
* Time out hanging tests in the VCS dependencies job

* Do not test pydispatcher’s latest commit
 2026-08-09 21:18:33 +05:00
Adrian c67bb8ac47
Document BROWSER=wslview for open_in_browser under WSL (#7965) 2026-08-09 21:15:37 +05:00
Adrian a0d1ad2001
Document that follow_all requests share meta and cb_kwargs values (#7962) 2026-08-09 21:15:27 +05:00
Adrian 95fe7acc86
Warn only once per request about the download warn size (#7963) 2026-08-09 21:14:20 +05:00
Adrian a6c017c2ca
Advise setting an identifying user agent (#7890) 2026-08-09 16:59:22 +05:00
Adrian 18ed0c0f7c
Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
Adrian 050a8cf159
Improve the docs about delaying start request iteration (#7883) 2026-08-09 16:57:08 +05:00
Adrian 59ce27afdd
Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
Adrian 7c797968a3
Docs: clarify the handling of exceptions raised in errbacks (#7898) 2026-08-09 16:53:33 +05:00
Adrian 63d5ce6272
Fix cookie lookup for dotless hosts and IP addresses (#7900) 2026-08-09 16:52:32 +05:00
Adrian a18d58d7b5
Document that process_spider_output receives a lazy result (#7939) 2026-08-09 12:26:15 +02:00
Adrian cd7f422a1e
Use client.bucket() in GCSFeedStorage so object-level GCS permissions suffice (#7945) 2026-08-09 14:57:11 +05:00
Mridankan Mandal f123c7a1cc
Simplify cmdline settings test (#7853) 
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-08-09 14:55:58 +05:00
Adrian 9e84112221
Cover update_vars() in the shell docs (#7889) 2026-08-09 14:25:14 +05:00
Adrian 0b2d220197
Improve docs for multi-spider runs (#7907) 2026-08-09 14:19:50 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian 94dff69468
Add VCS CI job, fix support for upcoming parsel version (#7924) 2026-08-09 14:15:58 +05:00
Adrian b4279e243b
Document that process_value runs before allow and deny (#7940) 2026-08-09 14:13:29 +05:00
Adrian fc9c505e79
Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 81d12c6eb8
Document the Referer caveat of DEFAULT_REQUEST_HEADERS (#7917) 2026-08-09 14:08:06 +05:00
Adrian 5b4888a0b1
Document that signal handler order is undefined (#7941) 2026-08-09 14:07:23 +05:00
Adrian 38f6e3cfd1
Remove the leftover KEEP_ALIVE setting from scrapy shell (#7928) 2026-08-09 14:06:39 +05:00
Adrian 56f4afd84e
Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
Adrian 1e92635a18
Benchmark item processing and item concurrency (#7954) 2026-08-09 14:01:06 +05:00
Adrian 63485522b6
Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
Adrian 9d2dea7a8d
Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
Adrian 1bd839b57d
llm-check: allow-list authors by org or track record (#7906) 2026-08-06 08:39:46 +02:00
Adrian 0c89e87b18
Switch tactics (#7895) 2026-08-05 20:17:25 +02:00
Adrian e0128c20c3
Extend benchmarks (#7887) 2026-08-05 11:51:22 +05:00
Adrian 91b70e4db4
Improve the docs about scrapy parse --pipelines (#7876) 2026-08-04 17:30:51 +02:00
Adrian 639fac78b3
Cover the signature change of scrape_func (#7875) 2026-08-04 20:02:52 +05:00
Adrian 0c542afa65
Upgrade the minimum queuelib to 1.6.1 (#7874) 2026-08-04 18:47:00 +05:00
Adrian 6e2081ca41
Ask custom download handlers not to use engine.download_async() (#7871) 2026-08-04 14:41:25 +05:00
Andrey Rakhmatullin 4cdde1c79c
Bump CodSpeed to 5. (#7870) 2026-08-04 09:25:52 +02:00
Adrian 388d4fcf04
Add a _load_objects() helper for object-or-path setting lists (#7857) 
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-04 11:56:46 +05:00