RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,210 Commits 42 Branches 132 Tags 161 MiB
Commit Graph

11210 Commits

Author SHA1 Message Date
Adrian Chaves 8dfb548415 WIP 2026-06-29 16:29:31 +02:00
Adrian Chaves 184166d629 Use get_* for getters 2026-06-29 10:58:18 +02:00
Adrian Chaves 7a211960c9 Merge remote-tracking branch 'origin/master' into throttling 2026-06-29 10:47:28 +02:00
Adrian Chaves 8834250059 WIP 2026-06-26 22:53:11 +02:00
greymoth 185d6b9a20
Fix request_to_curl() corrupting dict cookies with bytes keys/values (#7675) 
* Fix request_to_curl() corrupting dict cookies with bytes keys/values

PR #7603 made the list-cookie branch of request_to_curl() bytes-safe via
_cookie_value_to_unicode(), but left the sibling dict-cookie branch using
raw f-string interpolation. A dict cookie with bytes keys/values (a
supported and common form, e.g. Request(url, cookies={b"k": b"v"})) was
rendered as --cookie 'b'k'=b'v'' instead of --cookie 'k=v', producing a
broken curl command.

Route the dict branch through the same _cookie_value_to_unicode() helper,
mirroring the list branch. Add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: apply _cookie_value_to_unicode to list-branch cookie key/value

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-27 00:53:32 +05:00
Adrian Chaves d23c44858a Address issues 2026-06-26 21:41:45 +02:00
Andrey Rakhmatullin 4b40d2d06a
Close various garbage collectible resources (mostly in tests) (#7644) 
* Set TELNETCONSOLE_ENABLED=False in get_crawler().

* Make _get_console_and_portal() a context manager.

* Allow PYTHONTRACEMALLOC in tox.

* Close the event loop in test_custom_asyncio_loop_enabled_false().

* Close the handler in _uninstall_scrapy_root_handler().

* Close queues in test_squeues.py.

* Close empty queues in init_prios().

* Close PopenSpawn stdin and stdout explicitly.

* Silence the unclosed socket warning.

* Implement more methods in CustomStatsCollector.

* Link the Twisted issue.

* Close the connection on download_maxsize.

* Fix typing.

* Add a comment about CustomStatsCollector.

* Restore the coverage.

* Properly close fixture queues.

* More robust file closing in feed storages.

* Close the file in TestMarshalItemExporter.test_nonstring_types_item().

* Cleanup temporary file handling in test_exporters.py.

* Close the file in test_stats_file_failed().

* Use an explicit TextIOWrapper in XmlItemExporter.
 2026-06-26 23:05:08 +05:00
Adrian Chaves fb031dee80 Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 19:01:20 +02:00
Adrian Chaves 5aae8334a5 Improve request and scope delay support 2026-06-26 19:00:06 +02:00
Adrian cf5607f8bc
Undeprecated the basic FormRequest API (#7671) 2026-06-26 17:35:46 +02:00
Adrian edc353c975
Improve test coverage for shell/ (#7680) 
* Improve test coverage for shell/

* Address test issues

* Make the shell config test more reliable on Windows
 2026-06-26 19:40:36 +05:00
Adrian Chaves ee8c5f7aa4 Fix two environment-specific test failures 2026-06-26 16:00:11 +02:00
Adrian Chaves aeb6b4763c Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 14:47:23 +02:00
Adrian Chaves 4a7a2d5b5f Improve test coverage 2026-06-26 14:39:13 +02:00
Adrian Chaves 8caa1ac4db Fix typing issues 2026-06-26 13:27:14 +02:00
Adrian 1b940a75ac
Improve the item pipeline docs (#7676) 2026-06-26 16:18:22 +05:00
Adrian Chaves c889c1057d Improve test coverage 2026-06-26 13:09:17 +02:00
Adrian Chaves 627cecc93e Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 12:17:22 +02:00
Adrian Chaves 6ce90dde3e Minimum queuelib: 1.4.2 → 1.6.1 2026-06-26 12:15:52 +02:00
Adrian 7ab404c725
Add a documentation page about security (#7678) 2026-06-26 11:58:37 +02:00
Adrian 0ccddb4f61
Improve test coverage for contracts (#7677) 2026-06-26 11:47:30 +02:00
Adrian Chaves 3e02b9e5ab Merge remote-tracking branch 'origin/master' into throttling 2026-06-26 11:32:43 +02:00
Adrian Chaves 8190b5b703 Add a throttling-aware scheduler 2026-06-26 10:44:31 +02:00
Adrian 0007676e8d
Improve test coverage for http/ (#7672) 
* Improve test coverage for http/

* Use isinstance() instead of type()
 2026-06-26 12:21:12 +05:00
Sriniketh24 d8d7de2339
Fix FTPDownloadHandler not closing FTP connection after download (#7667) 2026-06-26 04:26:59 +02:00
Adrian Chaves ec18ddc239 Initial implementation draft 2026-06-25 16:01:47 +02:00
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665) 
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
 2026-06-25 16:44:51 +05:00
Adrian Chaves 4af625ac1b Merge remote-tracking branch 'origin/master' into throttling 2026-06-25 13:39:28 +02:00
Adrian Chaves 7fdee93e42 WIP 2026-06-25 13:38:55 +02:00
Adrian c690eac770
Document “logging settings” as special settings (#7668) 2026-06-25 16:34:37 +05:00
Adrian 65e8954a06
Improve test coverage for spider middlewares (#7664) 2026-06-25 15:35:25 +05:00
Adrian dd4549e6f9
Improve test coverage for downloader middlewares (#7655) 
* Improve test coverage for downloader middlewares

* Improve coverage further
 2026-06-25 01:48:28 +05:00
Adrian Chaves c93e430ccb Complete docs 2026-06-24 13:35:07 +02:00
Adrian Chaves d18085dc3e Update docs 2026-06-24 11:49:51 +02:00
Adrian Chaves d8d519367d Fix docs/requirements.txt 2026-06-24 10:57:56 +02:00
Adrian Chaves b6bc947cce scrapy/addons.py → scrapy/addons/__init__.py 2026-06-24 10:37:42 +02:00
Adrian Chaves e796beb6fa Merge remote-tracking branch 'origin/master' into throttling 2026-06-24 10:27:28 +02:00
Adrian b78ab3d6c8
Improve test coverage for settings/ (#7654) 2026-06-23 14:47:21 +05:00
smellslikeml fb3455304d
Add content-based image filtering example (#4954) 2026-06-23 09:04:19 +02:00
tanishqtayade f5a62a293f
Fix cell-var-from-loop bug in _send_catch_log_deferred (#7649) 2026-06-23 09:04:01 +02:00
Adrian f605defefc
Document scrapy-lint, remove start_url check (#7627) 
* Document scrapy-lint, remove start_url check

* Remove the offsite URL check in favor of scrapy-lint
 2026-06-22 20:12:16 +05:00
Adrian 7499d17e28
Improve test coverage for responsetypes.py (#7646) 
* Improve test coverage for responsetypes.py

* Solve typing issues
 2026-06-22 20:10:38 +05:00
Adrian b6596de317
Do not ignore CrawlerProcess settings (#7647) 
* Do not ignore CrawlerProcess settings

* Update test_crawlerrunner_accepts_crawler
 2026-06-22 20:10:10 +05:00
Shashank S. Khasare 75f05d4e80
Add test coverage for scrapy.utils.decorators (#7645) 2026-06-22 08:50:11 +02:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00