RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,209 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

11209 Commits

Author SHA1 Message Date
Adrian Chaves 38fe2751b3 Fix test failure during all-tests run 2026-06-19 21:27:52 +02:00
Adrian Chaves 9a3bd79481 Merge remote-tracking branch 'origin/master' into active-downloads-maxsize 2026-06-19 21:07:15 +02:00
Andrey Rakhmatullin c9f952c258
Refactor and improve catching warnings in tests. (#7643) 2026-06-19 21:04:34 +02:00
Adrian Chaves 73e7353c00 Complete test coverage 2026-06-19 21:00:20 +02:00
Adrian 6393858c7e
Improve coverage resolver (#7642) 
* Improve test coverage for resolver.py

* Make the Twisted code more readable
 2026-06-19 23:26:54 +05:00
Adrian Chaves f0e78aa677 Improve test reliability 2026-06-19 20:06:32 +02:00
Adrian Chaves 1bcb76c7cc Solve typing issues 2026-06-19 18:29:28 +02:00
Adrian Chaves 6dd52ad276 Merge remote-tracking branch 'origin/master' into active-downloads-maxsize 2026-06-19 18:03:29 +02:00
Adrian Chaves 0292d23d7a Improve deprecation messaging around SCRAPER_SLOT_MAX_ACTIVE_SIZE 2026-06-19 17:46:22 +02:00
Adrian Chaves e746df0297 Add a comment about the content of self._last_backout 2026-06-19 17:31:10 +02:00
Adrian Chaves 03081abcf6 Remove win-precise-time support, probably not worth it for this use case 2026-06-19 17:27:13 +02:00
Adrian d2842a205c
Improve coverage statscollectors (#7641) 2026-06-19 16:59:25 +02:00
Adrian Chaves 0c326fdc03 Improve documentation wording 2026-06-19 16:57:48 +02:00
Adrian Chaves 464aa5b185 Update a test that relied on the old behavior of _schedule_coro and would hang now 2026-06-19 16:27:26 +02:00
Adrian 7b3f88f8ab
Improve test coverage for pqueues.py (#7640) 2026-06-19 18:28:32 +05:00
Adrian Chaves b1ccb277da Make _schedule_coro() behavior the same in asyncio and non-asyncio 2026-06-19 12:40:20 +02:00
Adrian Chaves 208ea7a39e Let Opus do its thing 2026-06-19 11:55:21 +02:00
Adrian 699c93f6b2
Complete test coverage for linkextractors (#7639) 
* Complete test coverage for linkextractors

* pylint: disable use-implicit-booleaness-not-comparison
 2026-06-19 14:22:03 +05:00
Adrian Chaves d13cd45da2 Blind attempt number 2 2026-06-19 11:19:11 +02:00
Adrian Chaves 95084bb609 Solve test issues? 2026-06-19 10:45:14 +02:00
Adrian Chaves 37005262b6 Merge remote-tracking branch 'origin/master' into active-downloads-maxsize 2026-06-19 10:23:40 +02:00
Adrian Chaves b8d47f74c1 Address typing issues 2026-06-19 09:20:28 +02:00
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Vishal Gawade e5e48883b5
Deprecate ScrapyCommand.help() (#7633) 
Co-authored-by: Vishal Gawade <vishalgw@amazon.com>
 2026-06-19 06:49:32 +02:00
Lions-1 abbf3b95fc
tests: integration coverage for memusage extension (#7017) 2026-06-19 06:41:26 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
JSap0914 b7824db573
Fix rel_has_nofollow to be case-insensitive per HTML spec (#7632) 
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.

Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.

Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
 2026-06-17 14:48:43 +05:00
Kushal Gupta 0a4a92e843
Fix image store ACL settings for S3 and GCS (#7614) 
* Fix image store ACL settings for S3 and GCS

* Fix typing issues in ImagesPipeline ACL settings

* Fix typing issues in ImagesPipeline ACL settings

* Call parent _update_stores in ImagesPipeline
 2026-06-16 00:24:44 +05:00
Adrian Chaves d7a11e5875 Only install win-precise-time in extra-deps 2026-06-15 15:15:56 +02:00
Adrian Chaves 4d74e47cfa Address typing issues 2026-06-15 15:01:41 +02:00
Adrian Chaves d10fd92245 Set a more realistic RESPONSE_ROUGH_SIZE 2026-06-15 14:32:37 +02:00
Adrian Chaves 537a577268 Complete the implementation 2026-06-15 14:13:30 +02:00
Adrian e74647572d
Test SignalManager.disconnect_all() (#7625) 2026-06-15 14:18:29 +05:00
Adrian Chaves 242ad88d08 Merge remote-tracking branch 'origin/master' into active-downloads-maxsize 2026-06-15 09:08:27 +02:00
Adrian Chaves b103205255 Merge remote-tracking branch 'origin/master' into active-downloads-maxsize 2026-06-15 09:08:03 +02:00
Syncrain cfef12392a
Fix request_to_curl handling of verbose cookies dictionaries (#7603) 2026-06-15 08:46:30 +02:00
Andrey Rakhmatullin 4f241b73be
Fix deprecation warnings with pytest 9.1. (#7621) 2026-06-15 08:45:26 +02:00
Andrey Rakhmatullin 9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
Nishita Matlani f63a3aff25
Fix strip_url() removing default port from password. (#7605) 2026-06-15 08:41:55 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Adrian af30cfea12
Complete coverage for addons.py (#7612) 2026-06-12 16:15:36 +02:00
Adrian 983e6c1182
Link.__eq__: return NotImplemented instead of raising NotImplementedError (#7611) 2026-06-12 14:56:15 +02:00
Adrian b08ed1cf05
Add coverage for DummySpiderLoader.find_by_request() (#7608) 2026-06-12 13:13:17 +02:00
Adrian 7afc875081
Add coverage for Item.__delitem__() (#7610) 2026-06-12 15:32:38 +05:00
Adrian a8ffdcf851
Implement HTTP Auth settings and request metadata keys (#7590) 
* Implement HTTP Auth settings and request metadata keys

* Fix doc-tests

* Require the domain to be set when using settings
 2026-06-11 15:56:53 +05:00
Adrian c99f6e2209
Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600) 2026-06-11 12:10:49 +02:00
Labib Bin Salam e0a7de7213
Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573) 
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.

Closes #6120
 2026-06-11 14:40:15 +05:00
Adrian e7f229f5b2
Clarify the first-class-citizen nature of asyncio support (#7599) 2026-06-11 10:51:46 +02:00
Andrey Rakhmatullin 4cb049cb15
Small docs fixes. (#7598) 2026-06-11 10:51:03 +02:00
msn ad4549673b
Remove set_asyncio_event_loop call from Shell._schedule (#7594) 
Co-authored-by: Mayuresh <mayuresh@Mayureshs-MacBook-Air.local>
 2026-06-11 10:45:25 +02:00