RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,315 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

2205 Commits

Author SHA1 Message Date
Can Kupeli c513e7d6e5 implemented tests for branch coverage for function printSummary 2024-02-21 23:52:35 +01:00
Andrey Rakhmatullin f798118ac2
Merge pull request #6232 from elramen/linkextractor_tests 
Added tests for LxmlLinkExtractor
 2024-02-21 23:03:20 +05:00
Elias Ram f19045403a test #8 made tests cleaner 2024-02-21 16:03:51 +01:00
Adrián Chaves 6fc7827042
Do not close the underlying file from compression plugins (#6239) 2024-02-21 14:27:42 +01:00
Elias Ram e8e6d28479 test #8 added tests for LxmlLinkExtractor 2024-02-20 20:41:18 +01:00
Laerte Pereira ee1189512f
Replace urlparse with urlparse_cached where possible (#6229) 2024-02-20 12:47:29 +01:00
Kevin Lloyd Bernal 9bb973dc54
Refactor LogStats extension to log IPM and RPM to stats on spider_close (#4111) 2024-02-16 12:25:38 +01:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Adrián Chaves 2f1d345e74 Solve test issues 2024-02-14 18:59:01 +01:00
Adrián Chaves 479619b340 Merge branch '2.11-redos' into 2.11 2024-02-14 17:18:26 +01:00
Adrián Chaves 809bfac489 Merge branch '2.11-compression-bomb' into 2.11 2024-02-14 17:16:25 +01:00
Adrián Chaves 5bcb8fd501 Merge branch '2.11-authorization' into 2.11 2024-02-14 17:14:29 +01:00
Adrián Chaves c5dad41190 Speed up tests, remove comments without regexps 2024-02-02 14:03:16 +01:00
Laerte Pereira 7001193c80 Simplify the logic 2024-01-31 12:53:08 -03:00
Laerte Pereira 24634f1bb2 Attend PR comments 2024-01-31 12:29:43 -03:00
Laerte Pereira 019443dd57 Remove settings from default implementation test 2024-01-31 10:08:07 -03:00
Laerte Pereira 2487e3cc03 Cleanup deprecated fingerprint code in scrapy.utils.request 2024-01-31 10:05:50 -03:00
Andrey Rakhmatullin 09a7efef7c Remove a defer.returnValue call. 2024-01-15 18:19:29 +04:00
Adrián Chaves e8dadb9592
scrapy parse: fix the signature of callbacks from the CLI (#6182) 2024-01-15 13:37:03 +01:00
Adrián Chaves fa0c598096
Add component getters to Crawler (#6181) 2024-01-15 13:14:02 +01:00
Mikhail Korobov 8ad17f7476
Merge pull request #6199 from Rotzbua/fix_flake8 
fix(flake8): lint errors
 2024-01-12 22:52:22 +05:00
Andrey Rakhmatullin c2baf4d0da Remove a defer.returnValue call. 2024-01-12 18:30:41 +04:00
Andrey Rakhmatullin becff65ccb
Merge pull request #6201 from wRAR/more-mitmproxy 
Fix and re-enable newer mitmproxy usage in tests.
 2024-01-12 14:26:39 +04:00
Mikhail Korobov 32fda2dc53
Merge pull request #6196 from Rotzbua/fix_typo 
fix(typo): correct `successfully`
 2024-01-12 00:36:16 +05:00
Andrey Rakhmatullin 68fccb1d58 Fix and re-enable newer mitmproxy usage in tests. 2024-01-06 01:35:56 +04:00
Rotzbua 745b8412f6
fix(flake8): lint errors 
E226 missing whitespace around arithmetic operator
E201 whitespace after '{'
 2024-01-05 14:54:00 +01:00
Rotzbua 0d445a3224
refactor(yield): use `yield from` syntax 2024-01-05 12:30:10 +01:00
Rotzbua c7b2b097b1
fix(typo): correct `successfully` 2024-01-05 10:50:45 +01:00
Adrián Chaves 0e78acb657
MediaPipeline: log media_to_download errors before stripping them (#5068) 2023-12-21 21:01:07 +01:00
Jalil SA d25cfe5315
Add JsonResponse (#6174) 2023-12-21 10:36:21 +01:00
Swati Anshu c67f730695
create_instance → build_from_crawler, build_from_settings (#6169) 2023-12-18 11:51:02 +01:00
Adrián Chaves 27781a85e7 Fix bad closing tags in XMLFeedSpider tests 2023-12-15 13:52:12 +01:00
Adrián Chaves cd6846b763 Merge remote-tracking branch 'scrapy/2.11' into 2.11-redos 2023-12-15 11:52:35 +01:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
Adrián Chaves 9655b0b8eb Mark slow tests, with their own tox env and CI job 2023-12-15 10:19:47 +01:00
Adrián Chaves 4f72b49f97 Fix namespaces nodename support for xmliter_lxml 2023-12-15 10:06:13 +01:00
Laerte Pereira b9c4ee26d7 Remove deprecated scrapy.downloadermiddlewares.decompression 2023-12-13 13:28:30 +01:00
Adrián Chaves 1533b69032 Test and address ReDoS attack vectors for open_in_browser 2023-12-13 12:01:35 +01:00
Laerte Pereira 09ce0ef526 Remove test_download_gzip_response test 2023-11-23 10:50:46 -03:00
Adrián Chaves 03d9866518 Also use DOWNLOAD_WARNSIZE for decompressions 2023-11-23 10:26:47 +01:00
Adrián Chaves e0b66c021a Mind Spider.download_maxsize and Request.meta['download_maxsize'] 2023-11-22 19:03:24 +01:00
Adrián Chaves 3fda2fe103 Protect against gzip bomb sitemaps 2023-11-22 18:34:37 +01:00
Adrián Chaves 9cc8703877 Protect against zstandard bombs 2023-11-22 17:53:00 +01:00
Adrián Chaves fba167c5e1 Protect against brotli bombs 2023-11-22 17:32:09 +01:00
Adrián Chaves 0c4a98f8e0 Protect against deflate bombs 2023-11-22 17:12:43 +01:00
Adrián Chaves 0bf29a7b1b Update test expectations 2023-11-22 16:10:50 +01:00
Adrián Chaves 6969041c5f Protect against gzip bombs 2023-11-22 15:52:00 +01:00
Adrián Chaves 1c4e93293a Merge remote-tracking branch 'scrapy/2.11' into 2.11-authorization 2023-11-17 15:42:08 +01:00
Adrián Chaves 1045856a50 Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-11-16 19:16:52 +04:00
Andrey Rakhmatullin 5e4fb0bc5f Re-enable uvloop tests on 3.12 (#6098) 2023-11-16 19:16:31 +04:00