RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,205 Commits 49 Branches 132 Tags 158 MiB
Commit Graph

11205 Commits

Author SHA1 Message Date
Adrian Chaves 8834250059 WIP 2026-06-26 22:53:11 +02:00
Adrian Chaves d23c44858a Address issues 2026-06-26 21:41:45 +02:00
Adrian Chaves fb031dee80 Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 19:01:20 +02:00
Adrian Chaves 5aae8334a5 Improve request and scope delay support 2026-06-26 19:00:06 +02:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian edc353c975
Improve test coverage for shell/ (#7680) 
* Improve test coverage for shell/

* Address test issues

* Make the shell config test more reliable on Windows
 2026-06-26 19:40:36 +05:00
Adrian Chaves ee8c5f7aa4 Fix two environment-specific test failures 2026-06-26 16:00:11 +02:00
Adrian Chaves aeb6b4763c Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 14:47:23 +02:00
Adrian Chaves 4a7a2d5b5f Improve test coverage 2026-06-26 14:39:13 +02:00
Adrian Chaves 8caa1ac4db Fix typing issues 2026-06-26 13:27:14 +02:00
Adrian 1b940a75ac
Improve the item pipeline docs (#7676) 2026-06-26 16:18:22 +05:00
Adrian Chaves c889c1057d Improve test coverage 2026-06-26 13:09:17 +02:00
Adrian Chaves 627cecc93e Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 12:17:22 +02:00
Adrian Chaves 6ce90dde3e Minimum queuelib: 1.4.2 → 1.6.1 2026-06-26 12:15:52 +02:00
Adrian 7ab404c725
Add a documentation page about security (#7678) 2026-06-26 11:58:37 +02:00
Adrian 0ccddb4f61
Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
Adrian Chaves 3e02b9e5ab Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 11:32:43 +02:00
Adrian Chaves 8190b5b703 Add a throttling-aware scheduler 2026-06-26 10:44:31 +02:00
Adrian 0007676e8d
Improve test coverage for http/ (#7672) 
* Improve test coverage for http/

* Use isinstance() instead of type()
 2026-06-26 12:21:12 +05:00
Sriniketh24 d8d7de2339
Fix FTPDownloadHandler not closing FTP connection after download (#7667) 2026-06-26 04:26:59 +02:00
Adrian Chaves ec18ddc239 Initial implementation draft 2026-06-25 16:01:47 +02:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian Chaves 4af625ac1b Merge remote-tracking branch 'origin/master' into throttling 2026-06-25 13:39:28 +02:00
Adrian Chaves 7fdee93e42 WIP 2026-06-25 13:38:55 +02:00
Adrian c690eac770
Document “logging settings” as special settings (#7668) 2026-06-25 16:34:37 +05:00
Adrian 65e8954a06
Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
Adrian dd4549e6f9
Improve test coverage for downloader middlewares (#7655) 
* Improve test coverage for downloader middlewares

* Improve coverage further
 2026-06-25 01:48:28 +05:00
Adrian Chaves c93e430ccb Complete docs 2026-06-24 13:35:07 +02:00
Adrian Chaves d18085dc3e Update docs 2026-06-24 11:49:51 +02:00
Adrian Chaves d8d519367d Fix docs/requirements.txt 2026-06-24 10:57:56 +02:00
Adrian Chaves b6bc947cce scrapy/addons.py → scrapy/addons/__init__.py 2026-06-24 10:37:42 +02:00
Adrian Chaves e796beb6fa Merge remote-tracking branch 'origin/master' into throttling 2026-06-24 10:27:28 +02:00
Adrian b78ab3d6c8
Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
smellslikeml fb3455304d
Add content-based image filtering example (#4954) 2026-06-23 09:04:19 +02:00
tanishqtayade f5a62a293f
Fix cell-var-from-loop bug in _send_catch_log_deferred (#7649) 2026-06-23 09:04:01 +02:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian 7499d17e28
Improve test coverage for responsetypes.py (#7646) 
* Improve test coverage for responsetypes.py

* Solve typing issues
 2026-06-22 20:10:38 +05:00
Adrian b6596de317
Do not ignore CrawlerProcess settings (#7647) 
* Do not ignore CrawlerProcess settings

* Update test_crawlerrunner_accepts_crawler
 2026-06-22 20:10:10 +05:00
Shashank S. Khasare 75f05d4e80
Add test coverage for scrapy.utils.decorators (#7645) 2026-06-22 08:50:11 +02:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00