RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,210 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2638 Commits

Author SHA1 Message Date
Adrian Chaves 8dfb548415 WIP 2026-06-29 16:29:31 +02:00
Adrian Chaves 184166d629 Use get_* for getters 2026-06-29 10:58:18 +02:00
Adrian Chaves 7a211960c9 Merge remote-tracking branch 'origin/master' into throttling 2026-06-29 10:47:28 +02:00
Adrian Chaves 8834250059 WIP 2026-06-26 22:53:11 +02:00
greymoth 185d6b9a20
Fix request_to_curl() corrupting dict cookies with bytes keys/values (#7675) 
* Fix request_to_curl() corrupting dict cookies with bytes keys/values

PR #7603 made the list-cookie branch of request_to_curl() bytes-safe via
_cookie_value_to_unicode(), but left the sibling dict-cookie branch using
raw f-string interpolation. A dict cookie with bytes keys/values (a
supported and common form, e.g. Request(url, cookies={b"k": b"v"})) was
rendered as --cookie 'b'k'=b'v'' instead of --cookie 'k=v', producing a
broken curl command.

Route the dict branch through the same _cookie_value_to_unicode() helper,
mirroring the list branch. Add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: apply _cookie_value_to_unicode to list-branch cookie key/value

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-27 00:53:32 +05:00
Adrian Chaves d23c44858a Address issues 2026-06-26 21:41:45 +02:00
Andrey Rakhmatullin 4b40d2d06a
Close various garbage collectible resources (mostly in tests) (#7644) 
* Set TELNETCONSOLE_ENABLED=False in get_crawler().

* Make _get_console_and_portal() a context manager.

* Allow PYTHONTRACEMALLOC in tox.

* Close the event loop in test_custom_asyncio_loop_enabled_false().

* Close the handler in _uninstall_scrapy_root_handler().

* Close queues in test_squeues.py.

* Close empty queues in init_prios().

* Close PopenSpawn stdin and stdout explicitly.

* Silence the unclosed socket warning.

* Implement more methods in CustomStatsCollector.

* Link the Twisted issue.

* Close the connection on download_maxsize.

* Fix typing.

* Add a comment about CustomStatsCollector.

* Restore the coverage.

* Properly close fixture queues.

* More robust file closing in feed storages.

* Close the file in TestMarshalItemExporter.test_nonstring_types_item().

* Cleanup temporary file handling in test_exporters.py.

* Close the file in test_stats_file_failed().

* Use an explicit TextIOWrapper in XmlItemExporter.
 2026-06-26 23:05:08 +05:00
Adrian Chaves fb031dee80 Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 19:01:20 +02:00
Adrian Chaves 5aae8334a5 Improve request and scope delay support 2026-06-26 19:00:06 +02:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian edc353c975
Improve test coverage for shell/ (#7680) 
* Improve test coverage for shell/

* Address test issues

* Make the shell config test more reliable on Windows
 2026-06-26 19:40:36 +05:00
Adrian Chaves ee8c5f7aa4 Fix two environment-specific test failures 2026-06-26 16:00:11 +02:00
Adrian Chaves 4a7a2d5b5f Improve test coverage 2026-06-26 14:39:13 +02:00
Adrian Chaves 8caa1ac4db Fix typing issues 2026-06-26 13:27:14 +02:00
Adrian Chaves c889c1057d Improve test coverage 2026-06-26 13:09:17 +02:00
Adrian Chaves 627cecc93e Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 12:17:22 +02:00
Adrian Chaves 6ce90dde3e Minimum queuelib: 1.4.2 → 1.6.1 2026-06-26 12:15:52 +02:00
Adrian 0ccddb4f61
Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
Adrian Chaves 3e02b9e5ab Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 11:32:43 +02:00
Adrian Chaves 8190b5b703 Add a throttling-aware scheduler 2026-06-26 10:44:31 +02:00
Adrian 0007676e8d
Improve test coverage for http/ (#7672) 
* Improve test coverage for http/

* Use isinstance() instead of type()
 2026-06-26 12:21:12 +05:00
Adrian Chaves ec18ddc239 Initial implementation draft 2026-06-25 16:01:47 +02:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian Chaves 4af625ac1b Merge remote-tracking branch 'origin/master' into throttling 2026-06-25 13:39:28 +02:00
Adrian Chaves 7fdee93e42 WIP 2026-06-25 13:38:55 +02:00
Adrian 65e8954a06
Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
Adrian dd4549e6f9
Improve test coverage for downloader middlewares (#7655) 
* Improve test coverage for downloader middlewares

* Improve coverage further
 2026-06-25 01:48:28 +05:00
Adrian b78ab3d6c8
Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian 7499d17e28
Improve test coverage for responsetypes.py (#7646) 
* Improve test coverage for responsetypes.py

* Solve typing issues
 2026-06-22 20:10:38 +05:00
Adrian b6596de317
Do not ignore CrawlerProcess settings (#7647) 
* Do not ignore CrawlerProcess settings

* Update test_crawlerrunner_accepts_crawler
 2026-06-22 20:10:10 +05:00
Shashank S. Khasare 75f05d4e80
Add test coverage for scrapy.utils.decorators (#7645) 2026-06-22 08:50:11 +02:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00
Adrian e74647572d
Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Andrey Rakhmatullin 4f241b73be
Fix deprecation warnings with pytest 9.1. (#7621) 2026-06-15 08:45:26 +02:00
Andrey Rakhmatullin 9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
Nishita Matlani f63a3aff25
Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Adrian af30cfea12
Complete coverage for addons.py (#7612) 2026-06-12 16:15:36 +02:00