RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,346 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2725 Commits

Author SHA1 Message Date
Adrian Chaves 8e244b048a Merge remote-tracking branch 'origin/master' into improve-coverage-utils 2026-08-10 19:03:02 +02:00
Adrian 95c16aa9af
Deprecate interpolating log formatter messages with the returned dict (#7971) 2026-08-10 21:29:53 +05:00
Adrian Chaves afccf18794 Improve test coverage 2026-08-10 18:25:28 +02:00
Adrian 15885a8db4
Generalize the use of build_from_crawler() internally (#7808) 
* Generalize the use of build_from_crawler() internally

* Do not use build_from_crawler() for spiders, since they are guaranteed having from_crawler()
 2026-08-10 20:55:10 +05:00
Adrian Chaves 30c256d58e Merge remote-tracking branch 'origin/master' into improve-coverage-utils 2026-08-10 15:18:53 +02:00
Adrian f1694269d8
Add FTPS support to the FTP feed export storage (#7953) 2026-08-10 11:33:58 +02:00
Adrian fe96c1f54b
Serialize dates and times as ISO 8601 in ScrapyJSONEncoder (#7918) 2026-08-10 14:07:42 +05:00
Adrian 4cc2356637
Implement browser-like bad header handling for the default download handler (#7806) 
* Implement browser-like bad header handling for the default download handler

* Remove dead code
 2026-08-10 13:59:44 +05:00
Adrian c7ee394442
Address warnings from the VCS workflow (#7966) 2026-08-10 13:46:20 +05:00
Adrian afbd2c9320
Let trackref release classes defined at run time (#7922) 2026-08-10 13:09:57 +05:00
Adrian c285f4cb18
Support CloseSpider during spider startup (#7905) 2026-08-10 10:06:54 +02:00
Adrian 508bd7faec
Type late Crawler attributes as always set instead of None (#7882) 2026-08-10 09:07:34 +02:00
Adrian ae68786210
Document how to customize media pipeline file names from responses (#7909) 2026-08-09 23:08:59 +05:00
Adrian 5270f3cf99
Add a depth_reset request metadata key (#7913) 2026-08-09 23:07:10 +05:00
Adrian d41bfaec05
Document how to disallow subdomains of allowed domains (#7903) 2026-08-09 23:05:43 +05:00
Adrian aadfd15327
Do not mangle log messages when a log formatter returns empty or tuple args (#7936) 2026-08-09 22:58:35 +05:00
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian a5614544a2
Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
Adrian 786ab10494
Document how to write a custom item exporter (#7931) 2026-08-09 22:31:53 +05:00
Adrian 7d9516f332
Canonicalize each extracted link once (#7961) 2026-08-09 22:30:30 +05:00
Adrian 609f64c55d
Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
Adrian 95fe7acc86
Warn only once per request about the download warn size (#7963) 2026-08-09 21:14:20 +05:00
Adrian 18ed0c0f7c
Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
Adrian 59ce27afdd
Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
Adrian 63d5ce6272
Fix cookie lookup for dotless hosts and IP addresses (#7900) 2026-08-09 16:52:32 +05:00
Adrian cd7f422a1e
Use client.bucket() in GCSFeedStorage so object-level GCS permissions suffice (#7945) 2026-08-09 14:57:11 +05:00
Mridankan Mandal f123c7a1cc
Simplify cmdline settings test (#7853) 
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-08-09 14:55:58 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian fc9c505e79
Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 56f4afd84e
Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
Adrian 1e92635a18
Benchmark item processing and item concurrency (#7954) 2026-08-09 14:01:06 +05:00
Adrian 63485522b6
Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
Adrian 9d2dea7a8d
Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
Adrian e0128c20c3
Extend benchmarks (#7887) 2026-08-05 11:51:22 +05:00
Adrian Chaves 436a2143ac Merge remote-tracking branch 'origin/master' into improve-coverage-utils 2026-08-05 02:57:52 +02:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a7385d6e51
Type tests.mockserver and other resource-defining files (#7865) 
* Type tests.mockserver and other resource-defining files

* Address pylint issues
 2026-08-03 23:44:28 +05:00
Adrian 298c9e610e
Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
Adrian cde7af87fa
Cover passing spider to a deprecated Downloader.fetch() (#7863) 2026-08-03 13:55:39 +02:00
Adrian Chaves aa42482047 Merge remote-tracking branch 'origin/master' into improve-coverage-utils 2026-08-03 13:00:44 +02:00
Adrian 2b2e18199b
Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
Adrian Chaves 7a301f3754 Increase timeouts for PyPy 2026-08-03 12:26:58 +02:00
Adrian Chaves 6673458c41 Merge remote-tracking branch 'origin/master' into improve-coverage-utils 
# Conflicts:
#	pyproject.toml
#	tox.ini
 2026-07-31 18:11:07 +02:00
Adrian Chaves f200dbeb82 Give some leeway for PyPy 2026-07-31 18:08:49 +02:00
Laerte Pereira 8caaac6ecb
fix(shell): Run IPython prompt in a thread under a running loop (#7816) 2026-07-31 18:05:22 +02:00
Adrian 14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
Adrian Chaves db6c9cc32c Improve test coverage 2026-07-31 16:50:20 +02:00
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian Chaves cd7ae6b155 Merge remote-tracking branch 'origin/master' into improve-coverage-utils 2026-07-31 14:24:00 +02:00