RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,511 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

2261 Commits

Author SHA1 Message Date
Andrey Rakhmatullin d08f559600
Add flake8-type-checking. (#6413) 2024-06-25 10:20:59 +02:00
Andrey Rakhmatullin 3d8dbd5648 flake8-bugbear 2024-06-20 00:27:38 +05:00
Andrey Rakhmatullin 1ef9c337ca flake8-docstrings 2024-06-19 23:57:40 +05:00
Andrey Rakhmatullin 1c70d3e605 flake8-comprehensions 2024-06-19 23:36:36 +05:00
Andrey Rakhmatullin 1282ddf8f7
Add parameters to most Deferred in scrapy/core. (#6395) 2024-06-10 10:27:50 +02:00
Andrey Rakhmatullin ddc98fe91b
Deprecate scrapy.utils.defer.process_chain_both(). (#6397) 2024-06-10 10:16:26 +02:00
Andrey Rakhmatullin 5850b8f3e6 Merge branch 'master' into pyupgrade 2024-06-07 11:20:31 +05:00
Andrey Rakhmatullin feb0b8f7dc Add pyupgrade. 2024-06-05 22:57:18 +05:00
Andrey Rakhmatullin 2e214210f6 Add parameters to iterable generics, replace generators with iterables. 2024-06-05 21:44:04 +05:00
Suvan Banerjee 3f76853bd2
Handle AttributeError: 'NoneType' in contract parsing (#6388) 2024-06-05 06:34:46 +02:00
Andrey Rakhmatullin 492c3bce9d
Don't run callbacks of requests from get_media_requests(). (#6386) 2024-06-03 13:28:20 +02:00
Fabian Schneebauer 0d58af8697 Add support for multiple referer policy tokens. 2024-05-29 10:59:32 +02:00
Sanchay Kumar cadb0dd707
Fix overridable methods in MediaPipeline (#6368) 2024-05-28 10:42:58 +02:00
Laerte Pereira 2facdd4fb0 Add change reactor test to CrawlerRunner 2024-05-26 19:55:54 -03:00
Andrey Rakhmatullin 631fc65fad
Update expectations of cookies after redirects. (#6367) 2024-05-16 16:42:09 +02:00
Adrián Chaves 812fd2368f
Allow user-defined secure cookies (#6357) 2024-05-15 11:48:43 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Bagaudin Magomedov b88f22c6c5
Add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter (#6324) 2024-05-13 14:33:24 +02:00
Sanchay Kumar c9ef520936
Add Downloader.get_slot_key() without a spider parameter (#6352) 2024-05-10 14:26:45 +02:00
aisha-partha 180bc9bad7 Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default 2024-05-08 22:06:46 +05:30
Andrey Rakhmatullin a5da77d01d
Merge pull request #6221 from jxlil/fix/LxmlLinkExtractor 
fix: LxmlLinkExtractor unique_list missing key
 2024-04-19 19:57:11 +05:00
kokobhara b1fe97dc6c
Fix test expectations (#6316) 2024-04-17 13:14:57 +02:00
Víctor Ruiz 48c5a8c98f
Fix WrappedRequest.get_header raising TypeError if default is None (#6310) 2024-04-08 11:47:53 +02:00
Andrey Rakhmatullin 5d31e89262
Merge pull request #6298 from lorenzoverardo/bug-6292/robots.txt-non-UTF-8 
Handle robots.txt files not UTF-8 encoded
 2024-04-05 13:49:10 +05:00
Lorenzo Verardo 7b37dcd80d Handle robots.txt files not UTF-8 encoded 2024-04-04 12:22:50 +02:00
igeni f7bf3f726e
modified string's concat to f-strings (#6296) 2024-04-01 15:37:23 +02:00
Andrey Rakhmatullin 8d917c0b55 Run black. 2024-03-14 17:44:57 +05:00
Andrey Rakhmatullin 5bf0e1d1db
Merge branch 'master' into master 2024-03-14 17:38:30 +05:00
Andrey Rakhmatullin 7255dfd41f
Merge pull request #6231 from elramen/check_command_tests 
Added tests for check command
 2024-03-14 15:28:19 +05:00
Lucas Belo 4460d3ed96
Remove tests/requirements.txt and refactor extra deps (#6272) 
Co-authored-by: lucasbelo777 <lucas.belo@cialdnb.com>
 2024-03-13 07:22:48 +01:00
Andrey Rakhmatullin 95a70d3fa0
Merge pull request #6290 from Gallaecio/fix-skips 
Fix WindowsRunSpiderCommandTest skip outside Windows for older Twisted
 2024-03-12 14:17:10 +05:00
Adrián Chaves d7581c6b41
Allow disabling the AutoThrottle extension for a given slot (#6246) 2024-03-12 09:44:29 +01:00
Adrián Chaves e72de11f55 Add super 2024-03-12 09:29:10 +01:00
Adrián Chaves 188d9a8bb3 Remove unnecessary comment 2024-03-11 23:28:57 +01:00
Adrián Chaves ab5ea32ffd Fix WindowsRunSpiderCommandTest skip outside Windows for older Twisted. 2024-03-11 23:26:19 +01:00
pengqiseven 6e84648c07
Fix some comments (#6285) 
Signed-off-by: pengqiseven <i@sohu.com>
Co-authored-by: pengqiseven <i@sohu.com>
 2024-03-11 10:03:06 +01:00
Laerte Pereira 532cd2eabd Use brotlicffi for PyPy 2024-03-05 21:30:20 -03:00
Laerte Pereira cab1016bb6 Add brotlicffi support 2024-03-05 20:24:17 -03:00
Andrey Rakhmatullin 1311e7db05 Regenerate the expired mitmproxy-ca.pem. 2024-02-29 16:31:06 +05:00
Laerte Pereira 2169810414 fix: Proxy tests don't use custom certificate authority 2024-02-29 06:41:14 -03:00
Laerte Pereira 26a16f2c43 Fix tests 2024-02-28 16:36:19 -03:00
Andrey Rakhmatullin 4f9dd998dc Bump pylint, cleanup the ignored tags. 2024-02-28 14:31:21 +05:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Adrián Chaves 2d46b4acf5
Complete coverage for the AutoThrottle extension (#6245) 2024-02-27 09:28:02 +01:00
Jon b0ef9a89a1
Remove usage of deprecated mktemp (#5285) 2024-02-26 19:53:06 +01:00
vishesh10 e208f82076
Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
Andrey Rakhmatullin ebd7e199f0
Merge pull request #6235 from elramen/http_request_tests 
Added tests for form.py
 2024-02-22 12:17:11 +05:00
Andrey Rakhmatullin edd7ba1c06
Merge pull request #6236 from elramen/json_request_tests_main 
Improve json_request.py coverage
 2024-02-22 12:16:47 +05:00
Can Kupeli c513e7d6e5 implemented tests for branch coverage for function printSummary 2024-02-21 23:52:35 +01:00
noon 877398a3de refactor #3 Remove inner class in form test 2024-02-21 21:13:57 +01:00