RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,202 Commits 49 Branches 132 Tags 187 MiB
Commit Graph

11202 Commits

Author SHA1 Message Date
Andrey Rakhmatullin ae810f06ce Merge branch 'master' into tests-close-resources 2026-06-26 22:48:33 +05:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian edc353c975
Improve test coverage for shell/ (#7680) 
* Improve test coverage for shell/

* Address test issues

* Make the shell config test more reliable on Windows
 2026-06-26 19:40:36 +05:00
Adrian 1b940a75ac
Improve the item pipeline docs (#7676) 2026-06-26 16:18:22 +05:00
Adrian 7ab404c725
Add a documentation page about security (#7678) 2026-06-26 11:58:37 +02:00
Adrian 0ccddb4f61
Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
Adrian 0007676e8d
Improve test coverage for http/ (#7672) 
* Improve test coverage for http/

* Use isinstance() instead of type()
 2026-06-26 12:21:12 +05:00
Sriniketh24 d8d7de2339
Fix FTPDownloadHandler not closing FTP connection after download (#7667) 2026-06-26 04:26:59 +02:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian c690eac770
Document “logging settings” as special settings (#7668) 2026-06-25 16:34:37 +05:00
Adrian 65e8954a06
Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
Adrian dd4549e6f9
Improve test coverage for downloader middlewares (#7655) 
* Improve test coverage for downloader middlewares

* Improve coverage further
 2026-06-25 01:48:28 +05:00
Adrian b78ab3d6c8
Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
smellslikeml fb3455304d
Add content-based image filtering example (#4954) 2026-06-23 09:04:19 +02:00
tanishqtayade f5a62a293f
Fix cell-var-from-loop bug in _send_catch_log_deferred (#7649) 2026-06-23 09:04:01 +02:00
Andrey Rakhmatullin 67b0bf3c78 Use an explicit TextIOWrapper in XmlItemExporter. 2026-06-23 00:07:28 +05:00
Andrey Rakhmatullin 0d9147ea85 Close the file in test_stats_file_failed(). 2026-06-22 23:17:59 +05:00
Andrey Rakhmatullin a7a3ef39ba Cleanup temporary file handling in test_exporters.py. 2026-06-22 22:58:38 +05:00
Andrey Rakhmatullin e4729d36d8 Close the file in TestMarshalItemExporter.test_nonstring_types_item(). 2026-06-22 22:06:52 +05:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian 7499d17e28
Improve test coverage for responsetypes.py (#7646) 
* Improve test coverage for responsetypes.py

* Solve typing issues
 2026-06-22 20:10:38 +05:00
Adrian b6596de317
Do not ignore CrawlerProcess settings (#7647) 
* Do not ignore CrawlerProcess settings

* Update test_crawlerrunner_accepts_crawler
 2026-06-22 20:10:10 +05:00
Shashank S. Khasare 75f05d4e80
Add test coverage for scrapy.utils.decorators (#7645) 2026-06-22 08:50:11 +02:00
Andrey Rakhmatullin a07444a918 More robust file closing in feed storages. 2026-06-21 18:14:52 +05:00
Andrey Rakhmatullin 5f9cdd8b75 Properly close fixture queues. 2026-06-21 02:05:01 +05:00
Andrey Rakhmatullin d7f1920116 Restore the coverage. 2026-06-21 01:40:12 +05:00
Andrey Rakhmatullin 43a794355a Add a comment about CustomStatsCollector. 2026-06-20 17:35:27 +05:00
Andrey Rakhmatullin b880825165 Fix typing. 2026-06-20 17:31:20 +05:00
Andrey Rakhmatullin 8e112e3735 Close the connection on download_maxsize. 2026-06-20 17:28:27 +05:00
Andrey Rakhmatullin 9f90618533 Link the Twisted issue. 2026-06-20 17:27:54 +05:00
Andrey Rakhmatullin 31af88a81f Implement more methods in CustomStatsCollector. 2026-06-20 16:49:08 +05:00
Andrey Rakhmatullin 5531ced9fb Silence the unclosed socket warning. 2026-06-20 15:01:41 +05:00
Andrey Rakhmatullin a54a8923ea Close PopenSpawn stdin and stdout explicitly. 2026-06-20 03:03:39 +05:00
Andrey Rakhmatullin 55b457f11d Close empty queues in init_prios(). 2026-06-20 01:40:06 +05:00
Andrey Rakhmatullin 22be98c3af Close queues in test_squeues.py. 2026-06-20 01:16:34 +05:00
Andrey Rakhmatullin a171749f95 Close the handler in _uninstall_scrapy_root_handler(). 2026-06-20 00:22:20 +05:00
Andrey Rakhmatullin 994b4e7fc5 Close the event loop in test_custom_asyncio_loop_enabled_false(). 2026-06-20 00:17:50 +05:00
Andrey Rakhmatullin 8b785bbee1 Allow PYTHONTRACEMALLOC in tox. 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin a0964e8dcf Make _get_console_and_portal() a context manager. 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin 7af7c4094a Set TELNETCONSOLE_ENABLED=False in get_crawler(). 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00