RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,453 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

2251 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 492c3bce9d
Don't run callbacks of requests from get_media_requests(). (#6386) 2024-06-03 13:28:20 +02:00
Fabian Schneebauer 0d58af8697 Add support for multiple referer policy tokens. 2024-05-29 10:59:32 +02:00
Sanchay Kumar cadb0dd707
Fix overridable methods in MediaPipeline (#6368) 2024-05-28 10:42:58 +02:00
Laerte Pereira 2facdd4fb0 Add change reactor test to CrawlerRunner 2024-05-26 19:55:54 -03:00
Andrey Rakhmatullin 631fc65fad
Update expectations of cookies after redirects. (#6367) 2024-05-16 16:42:09 +02:00
Adrián Chaves 812fd2368f
Allow user-defined secure cookies (#6357) 2024-05-15 11:48:43 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Bagaudin Magomedov b88f22c6c5
Add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter (#6324) 2024-05-13 14:33:24 +02:00
Sanchay Kumar c9ef520936
Add Downloader.get_slot_key() without a spider parameter (#6352) 2024-05-10 14:26:45 +02:00
aisha-partha 180bc9bad7 Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default 2024-05-08 22:06:46 +05:30
Andrey Rakhmatullin a5da77d01d
Merge pull request #6221 from jxlil/fix/LxmlLinkExtractor 
fix: LxmlLinkExtractor unique_list missing key
 2024-04-19 19:57:11 +05:00
kokobhara b1fe97dc6c
Fix test expectations (#6316) 2024-04-17 13:14:57 +02:00
Víctor Ruiz 48c5a8c98f
Fix WrappedRequest.get_header raising TypeError if default is None (#6310) 2024-04-08 11:47:53 +02:00
Andrey Rakhmatullin 5d31e89262
Merge pull request #6298 from lorenzoverardo/bug-6292/robots.txt-non-UTF-8 
Handle robots.txt files not UTF-8 encoded
 2024-04-05 13:49:10 +05:00
Lorenzo Verardo 7b37dcd80d Handle robots.txt files not UTF-8 encoded 2024-04-04 12:22:50 +02:00
igeni f7bf3f726e
modified string's concat to f-strings (#6296) 2024-04-01 15:37:23 +02:00
Andrey Rakhmatullin 8d917c0b55 Run black. 2024-03-14 17:44:57 +05:00
Andrey Rakhmatullin 5bf0e1d1db
Merge branch 'master' into master 2024-03-14 17:38:30 +05:00
Andrey Rakhmatullin 7255dfd41f
Merge pull request #6231 from elramen/check_command_tests 
Added tests for check command
 2024-03-14 15:28:19 +05:00
Lucas Belo 4460d3ed96
Remove tests/requirements.txt and refactor extra deps (#6272) 
Co-authored-by: lucasbelo777 <lucas.belo@cialdnb.com>
 2024-03-13 07:22:48 +01:00
Andrey Rakhmatullin 95a70d3fa0
Merge pull request #6290 from Gallaecio/fix-skips 
Fix WindowsRunSpiderCommandTest skip outside Windows for older Twisted
 2024-03-12 14:17:10 +05:00
Adrián Chaves d7581c6b41
Allow disabling the AutoThrottle extension for a given slot (#6246) 2024-03-12 09:44:29 +01:00
Adrián Chaves e72de11f55 Add super 2024-03-12 09:29:10 +01:00
Adrián Chaves 188d9a8bb3 Remove unnecessary comment 2024-03-11 23:28:57 +01:00
Adrián Chaves ab5ea32ffd Fix WindowsRunSpiderCommandTest skip outside Windows for older Twisted. 2024-03-11 23:26:19 +01:00
pengqiseven 6e84648c07
Fix some comments (#6285) 
Signed-off-by: pengqiseven <i@sohu.com>
Co-authored-by: pengqiseven <i@sohu.com>
 2024-03-11 10:03:06 +01:00
Laerte Pereira 532cd2eabd Use brotlicffi for PyPy 2024-03-05 21:30:20 -03:00
Laerte Pereira cab1016bb6 Add brotlicffi support 2024-03-05 20:24:17 -03:00
Andrey Rakhmatullin 1311e7db05 Regenerate the expired mitmproxy-ca.pem. 2024-02-29 16:31:06 +05:00
Laerte Pereira 2169810414 fix: Proxy tests don't use custom certificate authority 2024-02-29 06:41:14 -03:00
Laerte Pereira 26a16f2c43 Fix tests 2024-02-28 16:36:19 -03:00
Andrey Rakhmatullin 4f9dd998dc Bump pylint, cleanup the ignored tags. 2024-02-28 14:31:21 +05:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Adrián Chaves 2d46b4acf5
Complete coverage for the AutoThrottle extension (#6245) 2024-02-27 09:28:02 +01:00
Jon b0ef9a89a1
Remove usage of deprecated mktemp (#5285) 2024-02-26 19:53:06 +01:00
vishesh10 e208f82076
Add support for multiple-compressed responses (#6063) 2024-02-22 12:16:24 +01:00
Andrey Rakhmatullin ebd7e199f0
Merge pull request #6235 from elramen/http_request_tests 
Added tests for form.py
 2024-02-22 12:17:11 +05:00
Andrey Rakhmatullin edd7ba1c06
Merge pull request #6236 from elramen/json_request_tests_main 
Improve json_request.py coverage
 2024-02-22 12:16:47 +05:00
Can Kupeli c513e7d6e5 implemented tests for branch coverage for function printSummary 2024-02-21 23:52:35 +01:00
noon 877398a3de refactor #3 Remove inner class in form test 2024-02-21 21:13:57 +01:00
noon b7a7ae7dbb refactor #22 Change comment and warning catching 2024-02-21 21:04:45 +01:00
Andrey Rakhmatullin f798118ac2
Merge pull request #6232 from elramen/linkextractor_tests 
Added tests for LxmlLinkExtractor
 2024-02-21 23:03:20 +05:00
Elias Ram f19045403a test #8 made tests cleaner 2024-02-21 16:03:51 +01:00
Adrián Chaves 6fc7827042
Do not close the underlying file from compression plugins (#6239) 2024-02-21 14:27:42 +01:00
noon bc036542a8 refactor #3 Moved tests to FormRequestTest 2024-02-21 02:35:45 +01:00
noon 12b4417c56 test #22 Improve json_request.py coverage 2024-02-21 02:30:50 +01:00
noon e2a0c85f11 doc #3 Clarified test comments 2024-02-21 02:21:24 +01:00
noon e27d320c3c test #3 Increased branch coverage for form.py 2024-02-20 23:58:39 +01:00
Elias Ram e8e6d28479 test #8 added tests for LxmlLinkExtractor 2024-02-20 20:41:18 +01:00
Elias Ram f096f17fa4 test #6 added tests for check command 2024-02-20 20:32:02 +01:00