RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,349 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2727 Commits

Author SHA1 Message Date
Mohammad Hassan 24cafa80bc
Merge f7382561cf into ad43bf0c56 2026-08-15 11:31:49 -05:00
Adrian ad43bf0c56
Speed up the spider argument deprecation check (#7842) 2026-08-15 21:16:52 +05:00
Adrian e28e56aa61
Use the json selector type for JsonResponse (#7972) 2026-08-15 21:15:16 +05:00
Adrian 06af687662
Fail with a clear message when Spider.start() is not an asynchronous generator (#7946) 
* Fail with a clear message when Spider.start() is not an asynchronous generator

* Silence pylint
 2026-08-14 14:25:35 +05:00
Adrian fc226e6642
Speed up test_POST_large_json_x10 with a smaller JSON (#7859) 2026-08-14 14:23:00 +05:00
Adrian a22523bfd3
Add a HTTP2_MAX_FRAME_SIZE setting (#7988) 2026-08-14 14:19:06 +05:00
Adrian f975e92159
Stop the open_in_browser ReDoS tests from failing on slow interpreters (#7995) 2026-08-14 11:07:41 +05:00
Adrian ca48915dd4
Stop test_storage from expiring its cache entry on slow runners (#7993) 2026-08-13 11:42:38 +02:00
Adrian be514d8c5d
Rerun tests that need real Internet access when they fail (#7985) 2026-08-12 21:46:00 +02:00
Adrian 8bb06bf00b
Stop marking the Twisted-based HTTP/2 download handler as experimental, and privatize its API (#7986) 2026-08-12 23:33:38 +05:00
Adrian 52cc2da72d
Deprecate PeriodicLog.multiplier instead of removing it altogether (#7982) 2026-08-12 11:23:41 +05:00
Adrian cf6ef74c79
Deprecate the old import path of FileException (#7973) 2026-08-10 20:19:46 +02:00
Adrian 95c16aa9af
Deprecate interpolating log formatter messages with the returned dict (#7971) 2026-08-10 21:29:53 +05:00
Adrian 15885a8db4
Generalize the use of build_from_crawler() internally (#7808) 
* Generalize the use of build_from_crawler() internally

* Do not use build_from_crawler() for spiders, since they are guaranteed having from_crawler()
 2026-08-10 20:55:10 +05:00
Adrian f1694269d8
Add FTPS support to the FTP feed export storage (#7953) 2026-08-10 11:33:58 +02:00
Adrian fe96c1f54b
Serialize dates and times as ISO 8601 in ScrapyJSONEncoder (#7918) 2026-08-10 14:07:42 +05:00
Adrian 4cc2356637
Implement browser-like bad header handling for the default download handler (#7806) 
* Implement browser-like bad header handling for the default download handler

* Remove dead code
 2026-08-10 13:59:44 +05:00
Adrian c7ee394442
Address warnings from the VCS workflow (#7966) 2026-08-10 13:46:20 +05:00
Adrian afbd2c9320
Let trackref release classes defined at run time (#7922) 2026-08-10 13:09:57 +05:00
Adrian c285f4cb18
Support CloseSpider during spider startup (#7905) 2026-08-10 10:06:54 +02:00
Adrian 508bd7faec
Type late Crawler attributes as always set instead of None (#7882) 2026-08-10 09:07:34 +02:00
Mohammad Hassan f7382561cf Do not fail to disable a component with an unimportable path 
`get_component_priority_dict_with_base()` normalizes every key of a
component priority dictionary by calling `load_object()` on it, including
keys whose value is `None`, i.e. components the user is disabling.

`normalize_key()` only guarded against `(NameError, TypeError, ValueError)`,
the three exceptions `load_object()` raises itself, but not against the
`ImportError` raised by the `import_module()` call it makes. A stale entry
disabling a component whose module no longer exists, e.g.
`{"scrapy.webservice.WebService": None}`, therefore aborted settings
resolution with `ModuleNotFoundError` instead of being ignored.

Add `ImportError` to the guard so that an unimportable key falls back to its
raw string form. Since its value is `None`, it is then dropped by the
existing `if v is not None` filter, and disabling by import path keeps
working. Enabled components with an unimportable path are unaffected:
`MiddlewareManager.from_crawler()` calls `load_object()` on them again and
still fails there.

Fixes #7820

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-09 20:08:58 -04:00
Adrian ae68786210
Document how to customize media pipeline file names from responses (#7909) 2026-08-09 23:08:59 +05:00
Adrian 5270f3cf99
Add a depth_reset request metadata key (#7913) 2026-08-09 23:07:10 +05:00
Adrian d41bfaec05
Document how to disallow subdomains of allowed domains (#7903) 2026-08-09 23:05:43 +05:00
Adrian aadfd15327
Do not mangle log messages when a log formatter returns empty or tuple args (#7936) 2026-08-09 22:58:35 +05:00
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian a5614544a2
Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
Adrian 786ab10494
Document how to write a custom item exporter (#7931) 2026-08-09 22:31:53 +05:00
Adrian 7d9516f332
Canonicalize each extracted link once (#7961) 2026-08-09 22:30:30 +05:00
Adrian 609f64c55d
Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
Adrian 95fe7acc86
Warn only once per request about the download warn size (#7963) 2026-08-09 21:14:20 +05:00
Adrian 18ed0c0f7c
Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
Adrian 59ce27afdd
Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
Adrian 63d5ce6272
Fix cookie lookup for dotless hosts and IP addresses (#7900) 2026-08-09 16:52:32 +05:00
Adrian cd7f422a1e
Use client.bucket() in GCSFeedStorage so object-level GCS permissions suffice (#7945) 2026-08-09 14:57:11 +05:00
Mridankan Mandal f123c7a1cc
Simplify cmdline settings test (#7853) 
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-08-09 14:55:58 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian fc9c505e79
Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 56f4afd84e
Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
Adrian 1e92635a18
Benchmark item processing and item concurrency (#7954) 2026-08-09 14:01:06 +05:00
Adrian 63485522b6
Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
Adrian 9d2dea7a8d
Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
Adrian e0128c20c3
Extend benchmarks (#7887) 2026-08-05 11:51:22 +05:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a7385d6e51
Type tests.mockserver and other resource-defining files (#7865) 
* Type tests.mockserver and other resource-defining files

* Address pylint issues
 2026-08-03 23:44:28 +05:00
Adrian 298c9e610e
Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
Adrian cde7af87fa
Cover passing spider to a deprecated Downloader.fetch() (#7863) 2026-08-03 13:55:39 +02:00
Adrian 2b2e18199b
Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00