RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,343 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

11343 Commits

Author SHA1 Message Date
Adrian a22523bfd3
Add a HTTP2_MAX_FRAME_SIZE setting (#7988) 2026-08-14 14:19:06 +05:00
Adrian f975e92159
Stop the open_in_browser ReDoS tests from failing on slow interpreters (#7995) 2026-08-14 11:07:41 +05:00
Adrian ca48915dd4
Stop test_storage from expiring its cache entry on slow runners (#7993) 2026-08-13 11:42:38 +02:00
Adrian be514d8c5d
Rerun tests that need real Internet access when they fail (#7985) 2026-08-12 21:46:00 +02:00
Adrian 8bb06bf00b
Stop marking the Twisted-based HTTP/2 download handler as experimental, and privatize its API (#7986) 2026-08-12 23:33:38 +05:00
Adrian 65b37286cc
Document the ITEM_PROCESSOR setting (#7983) 2026-08-12 20:34:25 +05:00
Adrian 52cc2da72d
Deprecate PeriodicLog.multiplier instead of removing it altogether (#7982) 2026-08-12 11:23:41 +05:00
Adrian 74f062fe3d
Upgrade twinecheck (#7979) 2026-08-11 21:23:25 +05:00
Adrian cf6ef74c79
Deprecate the old import path of FileException (#7973) 2026-08-10 20:19:46 +02:00
Adrian 8ce041e528
:issue: → :gh: (#7974) 2026-08-10 19:30:48 +02:00
Adrian 95c16aa9af
Deprecate interpolating log formatter messages with the returned dict (#7971) 2026-08-10 21:29:53 +05:00
Adrian 15885a8db4
Generalize the use of build_from_crawler() internally (#7808) 
* Generalize the use of build_from_crawler() internally

* Do not use build_from_crawler() for spiders, since they are guaranteed having from_crawler()
 2026-08-10 20:55:10 +05:00
Adrian f1694269d8
Add FTPS support to the FTP feed export storage (#7953) 2026-08-10 11:33:58 +02:00
Adrian fe96c1f54b
Serialize dates and times as ISO 8601 in ScrapyJSONEncoder (#7918) 2026-08-10 14:07:42 +05:00
Adrian 4cc2356637
Implement browser-like bad header handling for the default download handler (#7806) 
* Implement browser-like bad header handling for the default download handler

* Remove dead code
 2026-08-10 13:59:44 +05:00
Adrian c7ee394442
Address warnings from the VCS workflow (#7966) 2026-08-10 13:46:20 +05:00
Adrian afbd2c9320
Let trackref release classes defined at run time (#7922) 2026-08-10 13:09:57 +05:00
Adrian c285f4cb18
Support CloseSpider during spider startup (#7905) 2026-08-10 10:06:54 +02:00
Adrian 508bd7faec
Type late Crawler attributes as always set instead of None (#7882) 2026-08-10 09:07:34 +02:00
Adrian 09c918115b
Document response parsing memory use in the security page (#7930) 2026-08-10 08:43:57 +02:00
Adrian ae68786210
Document how to customize media pipeline file names from responses (#7909) 2026-08-09 23:08:59 +05:00
Adrian 5270f3cf99
Add a depth_reset request metadata key (#7913) 2026-08-09 23:07:10 +05:00
Adrian d41bfaec05
Document how to disallow subdomains of allowed domains (#7903) 2026-08-09 23:05:43 +05:00
Adrian ba29ce8d3b
Add a docs section on using download_async() from a downloader middleware (#7872) 2026-08-09 23:02:44 +05:00
Adrian fc082aa914
Clarify the docs about reactor settings (#7880) 2026-08-09 23:01:31 +05:00
Adrian aadfd15327
Do not mangle log messages when a log formatter returns empty or tuple args (#7936) 2026-08-09 22:58:35 +05:00
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian a5614544a2
Type tests involving items and exports (#7867) 2026-08-09 22:51:20 +05:00
Adrian 482a02d30c
Add a cookies documentation page (#7947) 2026-08-09 22:41:32 +05:00
Adrian 786ab10494
Document how to write a custom item exporter (#7931) 2026-08-09 22:31:53 +05:00
Adrian 7d9516f332
Canonicalize each extracted link once (#7961) 2026-08-09 22:30:30 +05:00
Adrian 609f64c55d
Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
Adrian 5427080f48
Create a page on optimization (#7938) 2026-08-09 18:37:24 +02:00
Adrian 8e94df6526
Time out hanging tests in the VCS dependencies job (#7960) 
* Time out hanging tests in the VCS dependencies job

* Do not test pydispatcher’s latest commit
 2026-08-09 21:18:33 +05:00
Adrian c67bb8ac47
Document BROWSER=wslview for open_in_browser under WSL (#7965) 2026-08-09 21:15:37 +05:00
Adrian a0d1ad2001
Document that follow_all requests share meta and cb_kwargs values (#7962) 2026-08-09 21:15:27 +05:00
Adrian 95fe7acc86
Warn only once per request about the download warn size (#7963) 2026-08-09 21:14:20 +05:00
Adrian a6c017c2ca
Advise setting an identifying user agent (#7890) 2026-08-09 16:59:22 +05:00
Adrian 18ed0c0f7c
Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
Adrian 050a8cf159
Improve the docs about delaying start request iteration (#7883) 2026-08-09 16:57:08 +05:00
Adrian 59ce27afdd
Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
Adrian 7c797968a3
Docs: clarify the handling of exceptions raised in errbacks (#7898) 2026-08-09 16:53:33 +05:00
Adrian 63d5ce6272
Fix cookie lookup for dotless hosts and IP addresses (#7900) 2026-08-09 16:52:32 +05:00
Adrian a18d58d7b5
Document that process_spider_output receives a lazy result (#7939) 2026-08-09 12:26:15 +02:00
Adrian cd7f422a1e
Use client.bucket() in GCSFeedStorage so object-level GCS permissions suffice (#7945) 2026-08-09 14:57:11 +05:00
Mridankan Mandal f123c7a1cc
Simplify cmdline settings test (#7853) 
Signed-off-by: Mridankan Mandal <xerontitan90@gmail.com>
 2026-08-09 14:55:58 +05:00
Adrian 9e84112221
Cover update_vars() in the shell docs (#7889) 2026-08-09 14:25:14 +05:00
Adrian 0b2d220197
Improve docs for multi-spider runs (#7907) 2026-08-09 14:19:50 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian 94dff69468
Add VCS CI job, fix support for upcoming parsel version (#7924) 2026-08-09 14:15:58 +05:00