RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,182 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

2620 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 0d9147ea85 Close the file in test_stats_file_failed(). 2026-06-22 23:17:59 +05:00
Andrey Rakhmatullin a7a3ef39ba Cleanup temporary file handling in test_exporters.py. 2026-06-22 22:58:38 +05:00
Andrey Rakhmatullin e4729d36d8 Close the file in TestMarshalItemExporter.test_nonstring_types_item(). 2026-06-22 22:06:52 +05:00
Andrey Rakhmatullin a07444a918 More robust file closing in feed storages. 2026-06-21 18:14:52 +05:00
Andrey Rakhmatullin 5f9cdd8b75 Properly close fixture queues. 2026-06-21 02:05:01 +05:00
Andrey Rakhmatullin d7f1920116 Restore the coverage. 2026-06-21 01:40:12 +05:00
Andrey Rakhmatullin 43a794355a Add a comment about CustomStatsCollector. 2026-06-20 17:35:27 +05:00
Andrey Rakhmatullin b880825165 Fix typing. 2026-06-20 17:31:20 +05:00
Andrey Rakhmatullin 31af88a81f Implement more methods in CustomStatsCollector. 2026-06-20 16:49:08 +05:00
Andrey Rakhmatullin a54a8923ea Close PopenSpawn stdin and stdout explicitly. 2026-06-20 03:03:39 +05:00
Andrey Rakhmatullin 22be98c3af Close queues in test_squeues.py. 2026-06-20 01:16:34 +05:00
Andrey Rakhmatullin 994b4e7fc5 Close the event loop in test_custom_asyncio_loop_enabled_false(). 2026-06-20 00:17:50 +05:00
Andrey Rakhmatullin a0964e8dcf Make _get_console_and_portal() a context manager. 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin 7af7c4094a Set TELNETCONSOLE_ENABLED=False in get_crawler(). 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00
Adrian e74647572d
Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Andrey Rakhmatullin 4f241b73be
Fix deprecation warnings with pytest 9.1. (#7621) 2026-06-15 08:45:26 +02:00
Andrey Rakhmatullin 9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
Nishita Matlani f63a3aff25
Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Adrian af30cfea12
Complete coverage for addons.py (#7612) 2026-06-12 16:15:36 +02:00
Adrian 983e6c1182
Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
Adrian b08ed1cf05
Add coverage for DummySpiderLoader.find_by_request() (#7608) 2026-06-12 13:13:17 +02:00
Adrian 7afc875081
Add coverage for Item.__delitem__() (#7610) 2026-06-12 15:32:38 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Syncrain ba28630c98
Validate reversed telnet console port ranges (#7593) 2026-06-11 12:47:00 +05:00
Adrian 2d0a898e2d
tox.ini: pinned, typing → min, mypy (#7595) 2026-06-11 12:10:48 +05:00
Andrey Rakhmatullin beb6c51c17
Update default_settings, deprecate CRAWLSPIDER_FOLLOW_LINKS. (#7592) 2026-06-11 00:22:09 +05:00
Andrey Rakhmatullin d59f9b644a
Replace _SettingsKeyT with str. (#7586) 2026-06-10 11:12:53 +02:00
Andrey Rakhmatullin ddafb37a7c
Assorted test fixes (#7585) 
* Fix the mitmproxy junitxml file name.

* Fix TestAsyncCrawlerRunnerHasSpider regression.

* Update split out file refs.

* Update the test_counter_handler() docstring.
 2026-06-10 11:01:06 +02:00
Andrey Rakhmatullin 4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575) 
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.

* Update the docs.

* Trim the tables.

* Restore lost wording.

* Handlers docs improvements and fixes.
 2026-06-09 01:10:44 +05:00
Adnan Awan d2290c35c2
Allow configuring the log level of the retry give-up message (#7567) 
* Allow configuring the log level of the retry give-up message

The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).

Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.

Fixes #5297, fixes #4622

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Address Adrian's review feedback: simplify and reorganize docs

- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information

Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key

* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature

- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com

* Apply suggestion from @AdrianAtZyte

* Minor changes

* Fix example formatting.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2026-06-08 18:44:30 +05:00
Andrey Rakhmatullin d9e2f5fbf7
Add settings for TLS min/max version as a replacement for the TLS method (#6546) 2026-06-08 11:54:10 +02:00
Adrian b2d8b06be6
Support sending requests with "unsafe" URLs (#7473) 2026-06-05 11:51:03 +02:00
Ayush Singh 13c1c1faf8
Close GCS feed temp files after upload (#7546) 2026-06-04 20:11:50 +05:00
Ethan Kuo df2f3d708e
fix open_in_browser() logic issue plus add new tests (#7506) 2026-06-04 16:24:50 +02:00
Andrey Rakhmatullin fed75a6c76
Refactor test_crawl.py and test_crawler.py. (#7566) 2026-06-03 14:48:12 +02:00
Andrey Rakhmatullin 90deebe75e
Convert tests that fail with testfixtures 12.0.0 (#7545) 2026-06-03 09:16:24 +02:00
Fardin Alizadeh 44406806f8
fix typos (#7564) 2026-06-02 11:04:38 +02:00