RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,182 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

11182 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 0d9147ea85 Close the file in test_stats_file_failed(). 2026-06-22 23:17:59 +05:00
Andrey Rakhmatullin a7a3ef39ba Cleanup temporary file handling in test_exporters.py. 2026-06-22 22:58:38 +05:00
Andrey Rakhmatullin e4729d36d8 Close the file in TestMarshalItemExporter.test_nonstring_types_item(). 2026-06-22 22:06:52 +05:00
Andrey Rakhmatullin a07444a918 More robust file closing in feed storages. 2026-06-21 18:14:52 +05:00
Andrey Rakhmatullin 5f9cdd8b75 Properly close fixture queues. 2026-06-21 02:05:01 +05:00
Andrey Rakhmatullin d7f1920116 Restore the coverage. 2026-06-21 01:40:12 +05:00
Andrey Rakhmatullin 43a794355a Add a comment about CustomStatsCollector. 2026-06-20 17:35:27 +05:00
Andrey Rakhmatullin b880825165 Fix typing. 2026-06-20 17:31:20 +05:00
Andrey Rakhmatullin 8e112e3735 Close the connection on download_maxsize. 2026-06-20 17:28:27 +05:00
Andrey Rakhmatullin 9f90618533 Link the Twisted issue. 2026-06-20 17:27:54 +05:00
Andrey Rakhmatullin 31af88a81f Implement more methods in CustomStatsCollector. 2026-06-20 16:49:08 +05:00
Andrey Rakhmatullin 5531ced9fb Silence the unclosed socket warning. 2026-06-20 15:01:41 +05:00
Andrey Rakhmatullin a54a8923ea Close PopenSpawn stdin and stdout explicitly. 2026-06-20 03:03:39 +05:00
Andrey Rakhmatullin 55b457f11d Close empty queues in init_prios(). 2026-06-20 01:40:06 +05:00
Andrey Rakhmatullin 22be98c3af Close queues in test_squeues.py. 2026-06-20 01:16:34 +05:00
Andrey Rakhmatullin a171749f95 Close the handler in _uninstall_scrapy_root_handler(). 2026-06-20 00:22:20 +05:00
Andrey Rakhmatullin 994b4e7fc5 Close the event loop in test_custom_asyncio_loop_enabled_false(). 2026-06-20 00:17:50 +05:00
Andrey Rakhmatullin 8b785bbee1 Allow PYTHONTRACEMALLOC in tox. 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin a0964e8dcf Make _get_console_and_portal() a context manager. 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin 7af7c4094a Set TELNETCONSOLE_ENABLED=False in get_crawler(). 2026-06-20 00:08:35 +05:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00
Adrian e74647572d
Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Andrey Rakhmatullin 4f241b73be
Fix deprecation warnings with pytest 9.1. (#7621) 2026-06-15 08:45:26 +02:00
Andrey Rakhmatullin 9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
Nishita Matlani f63a3aff25
Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Adrian af30cfea12
Complete coverage for addons.py (#7612) 2026-06-12 16:15:36 +02:00
Adrian 983e6c1182
Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
Adrian b08ed1cf05
Add coverage for DummySpiderLoader.find_by_request() (#7608) 2026-06-12 13:13:17 +02:00
Adrian 7afc875081
Add coverage for Item.__delitem__() (#7610) 2026-06-12 15:32:38 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Adrian c99f6e2209
Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600) 2026-06-11 12:10:49 +02:00
Labib Bin Salam e0a7de7213
Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573) 
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.

Closes #6120
 2026-06-11 14:40:15 +05:00
Adrian e7f229f5b2
Clarify the first-class-citizen nature of asyncio support (#7599) 2026-06-11 10:51:46 +02:00
Andrey Rakhmatullin 4cb049cb15
Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
msn ad4549673b
Remove set_asyncio_event_loop call from Shell._schedule (#7594) 
Co-authored-by: Mayuresh <mayuresh@Mayureshs-MacBook-Air.local>
 2026-06-11 10:45:25 +02:00
Syncrain ba28630c98
Validate reversed telnet console port ranges (#7593) 2026-06-11 12:47:00 +05:00
Adrian 2d0a898e2d
tox.ini: pinned, typing → min, mypy (#7595) 2026-06-11 12:10:48 +05:00
msn 93a627ba1c
Prevent scrapy shell from starting spider requests (#7557) 
* Prevent scrapy shell from starting spider requests

* Run pre-commit fixes

* Update shell architecture notes

---------

Co-authored-by: Mayuresh <mayuresh@Mayureshs-MacBook-Air.local>
 2026-06-11 00:59:03 +05:00