RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,202 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

2634 Commits

Author SHA1 Message Date
Andrey Rakhmatullin ae810f06ce Merge branch 'master' into tests-close-resources 2026-06-26 22:48:33 +05:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian edc353c975
Improve test coverage for shell/ (#7680) 
* Improve test coverage for shell/

* Address test issues

* Make the shell config test more reliable on Windows
 2026-06-26 19:40:36 +05:00
Adrian 0ccddb4f61
Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
Adrian 0007676e8d
Improve test coverage for http/ (#7672) 
* Improve test coverage for http/

* Use isinstance() instead of type()
 2026-06-26 12:21:12 +05:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian 65e8954a06
Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
Adrian dd4549e6f9
Improve test coverage for downloader middlewares (#7655) 
* Improve test coverage for downloader middlewares

* Improve coverage further
 2026-06-25 01:48:28 +05:00
Adrian b78ab3d6c8
Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
Andrey Rakhmatullin 67b0bf3c78 Use an explicit TextIOWrapper in XmlItemExporter. 2026-06-23 00:07:28 +05:00
Andrey Rakhmatullin 0d9147ea85 Close the file in test_stats_file_failed(). 2026-06-22 23:17:59 +05:00
Andrey Rakhmatullin a7a3ef39ba Cleanup temporary file handling in test_exporters.py. 2026-06-22 22:58:38 +05:00
Andrey Rakhmatullin e4729d36d8 Close the file in TestMarshalItemExporter.test_nonstring_types_item(). 2026-06-22 22:06:52 +05:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian 7499d17e28
Improve test coverage for responsetypes.py (#7646) 
* Improve test coverage for responsetypes.py

* Solve typing issues
 2026-06-22 20:10:38 +05:00
Adrian b6596de317
Do not ignore CrawlerProcess settings (#7647) 
* Do not ignore CrawlerProcess settings

* Update test_crawlerrunner_accepts_crawler
 2026-06-22 20:10:10 +05:00
Shashank S. Khasare 75f05d4e80
Add test coverage for scrapy.utils.decorators (#7645) 2026-06-22 08:50:11 +02:00
Andrey Rakhmatullin a07444a918 More robust file closing in feed storages. 2026-06-21 18:14:52 +05:00
Andrey Rakhmatullin 5f9cdd8b75 Properly close fixture queues. 2026-06-21 02:05:01 +05:00
Andrey Rakhmatullin d7f1920116 Restore the coverage. 2026-06-21 01:40:12 +05:00
Andrey Rakhmatullin 43a794355a Add a comment about CustomStatsCollector. 2026-06-20 17:35:27 +05:00
Andrey Rakhmatullin b880825165 Fix typing. 2026-06-20 17:31:20 +05:00
Andrey Rakhmatullin 31af88a81f Implement more methods in CustomStatsCollector. 2026-06-20 16:49:08 +05:00
Andrey Rakhmatullin a54a8923ea Close PopenSpawn stdin and stdout explicitly. 2026-06-20 03:03:39 +05:00
Andrey Rakhmatullin 22be98c3af Close queues in test_squeues.py. 2026-06-20 01:16:34 +05:00
Andrey Rakhmatullin 994b4e7fc5 Close the event loop in test_custom_asyncio_loop_enabled_false(). 2026-06-20 00:17:50 +05:00
Andrey Rakhmatullin a0964e8dcf Make _get_console_and_portal() a context manager. 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin 7af7c4094a Set TELNETCONSOLE_ENABLED=False in get_crawler(). 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00
Adrian e74647572d
Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Andrey Rakhmatullin 4f241b73be
Fix deprecation warnings with pytest 9.1. (#7621) 2026-06-15 08:45:26 +02:00
Andrey Rakhmatullin 9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
Nishita Matlani f63a3aff25
Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Adrian af30cfea12
Complete coverage for addons.py (#7612) 2026-06-12 16:15:36 +02:00
Adrian 983e6c1182
Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
Adrian b08ed1cf05
Add coverage for DummySpiderLoader.find_by_request() (#7608) 2026-06-12 13:13:17 +02:00
Adrian 7afc875081
Add coverage for Item.__delitem__() (#7610) 2026-06-12 15:32:38 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00