RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,342 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2721 Commits

Author SHA1 Message Date
Adrian Chaves 8e39a76cbb Fill __all__, add a test for it 2026-08-10 17:00:55 +02:00
Adrian Chaves 15870014e5 Merge remote-tracking branch 'origin/master' into connection-limits 2026-08-10 15:30:20 +02:00
Adrian f1694269d8
Add FTPS support to the FTP feed export storage (#7953) 2026-08-10 11:33:58 +02:00
Adrian Chaves f5bf9642d8 Fail HTTP/2 requests when the connection cannot be established 2026-08-10 11:31:03 +02:00
Adrian Chaves 12d71a0e06 Merge remote-tracking branch 'origin/master' into connection-limits 2026-08-10 11:14:10 +02:00
Adrian fe96c1f54b
Serialize dates and times as ISO 8601 in ScrapyJSONEncoder (#7918) 2026-08-10 14:07:42 +05:00
Adrian 4cc2356637
Implement browser-like bad header handling for the default download handler (#7806) 
* Implement browser-like bad header handling for the default download handler

* Remove dead code
 2026-08-10 13:59:44 +05:00
Adrian c7ee394442
Address warnings from the VCS workflow (#7966) 2026-08-10 13:46:20 +05:00
Adrian afbd2c9320
Let trackref release classes defined at run time (#7922) 2026-08-10 13:09:57 +05:00
Adrian c285f4cb18
Support CloseSpider during spider startup (#7905) 2026-08-10 10:06:54 +02:00
Adrian 508bd7faec
Type late Crawler attributes as always set instead of None (#7882) 2026-08-10 09:07:34 +02:00
Adrian ae68786210
Document how to customize media pipeline file names from responses (#7909) 2026-08-09 23:08:59 +05:00
Adrian 5270f3cf99
Add a depth_reset request metadata key (#7913) 2026-08-09 23:07:10 +05:00
Adrian d41bfaec05
Document how to disallow subdomains of allowed domains (#7903) 2026-08-09 23:05:43 +05:00
Adrian aadfd15327
Do not mangle log messages when a log formatter returns empty or tuple args (#7936) 2026-08-09 22:58:35 +05:00
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian a5614544a2
Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
Adrian 786ab10494
Document how to write a custom item exporter (#7931) 2026-08-09 22:31:53 +05:00
Adrian 7d9516f332
Canonicalize each extracted link once (#7961) 2026-08-09 22:30:30 +05:00
Adrian 609f64c55d
Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
Adrian Chaves df2e03cb27 Merge remote-tracking branch 'origin/master' into connection-limits 2026-08-09 18:46:08 +02:00
Adrian 95fe7acc86
Warn only once per request about the download warn size (#7963) 2026-08-09 21:14:20 +05:00
Adrian Chaves e1321d1931 Merge remote-tracking branch 'origin/master' into connection-limits 2026-08-09 18:03:37 +02:00
Adrian 18ed0c0f7c
Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
Adrian 59ce27afdd
Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
Adrian 63d5ce6272
Fix cookie lookup for dotless hosts and IP addresses (#7900) 2026-08-09 16:52:32 +05:00
Adrian cd7f422a1e
Use client.bucket() in GCSFeedStorage so object-level GCS permissions suffice (#7945) 2026-08-09 14:57:11 +05:00
Mridankan Mandal f123c7a1cc
Simplify cmdline settings test (#7853) 
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-08-09 14:55:58 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian fc9c505e79
Add URL benchmarks (#7914) 2026-08-09 14:12:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 56f4afd84e
Fix the telnet console shutdown error after a failed start (#7910) 2026-08-09 14:03:57 +05:00
Adrian 1e92635a18
Benchmark item processing and item concurrency (#7954) 2026-08-09 14:01:06 +05:00
Adrian 63485522b6
Remove the job directory of a download slot once it drains (#7955) 2026-08-09 13:58:59 +05:00
Adrian 9d2dea7a8d
Stop skipping the IPv6 resolver tests (#7935) 2026-08-09 13:57:44 +05:00
Adrian Chaves 60e2b0ae8c Improve test coverage 2026-08-08 11:32:39 +02:00
Adrian Chaves 346032f38d Improve coverage 2026-08-07 21:06:55 +02:00
Adrian Chaves 1b937b32a2 Add CONCURRENT_CONNECTIONS_PER_HANDLER and CONNECTION_KEEPALIVE_TIMEOUT 2026-08-07 19:06:39 +02:00
Adrian e0128c20c3
Extend benchmarks (#7887) 2026-08-05 11:51:22 +05:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a7385d6e51
Type tests.mockserver and other resource-defining files (#7865) 
* Type tests.mockserver and other resource-defining files

* Address pylint issues
 2026-08-03 23:44:28 +05:00
Adrian 298c9e610e
Type tests related to requests and responses (#7864) 2026-08-03 17:16:23 +05:00
Adrian cde7af87fa
Cover passing spider to a deprecated Downloader.fetch() (#7863) 2026-08-03 13:55:39 +02:00
Adrian 2b2e18199b
Type downloader middleware tests (#7858) 2026-08-03 15:49:15 +05:00
Laerte Pereira 8caaac6ecb
fix(shell): Run IPython prompt in a thread under a running loop (#7816) 2026-07-31 18:05:22 +02:00
Adrian 14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian 6f87d3f863
Rename the current benchmark for the future (#7839) 2026-07-31 16:46:33 +05:00
Adrian aa5ded2539
Set up CodSpeed (#7831) 
* Set up CodSpeed

* CodSpeed: update permissions
 2026-07-31 13:27:14 +05:00
Adrian 3180116cd0
Improve test coverage for scrapy.commands (#7829) 2026-07-31 12:19:49 +05:00