RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

2349 Commits

Author SHA1 Message Date
Kevin Lloyd Bernal 9bb973dc54
Refactor LogStats extension to log IPM and RPM to stats on spider_close (#4111) 2024-02-16 12:25:38 +01:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Adrián Chaves 2f1d345e74 Solve test issues 2024-02-14 18:59:01 +01:00
Adrián Chaves 479619b340 Merge branch '2.11-redos' into 2.11 2024-02-14 17:18:26 +01:00
Adrián Chaves 809bfac489 Merge branch '2.11-compression-bomb' into 2.11 2024-02-14 17:16:25 +01:00
Adrián Chaves 5bcb8fd501 Merge branch '2.11-authorization' into 2.11 2024-02-14 17:14:29 +01:00
Adrián Chaves c5dad41190 Speed up tests, remove comments without regexps 2024-02-02 14:03:16 +01:00
Laerte Pereira 7001193c80 Simplify the logic 2024-01-31 12:53:08 -03:00
Laerte Pereira 24634f1bb2 Attend PR comments 2024-01-31 12:29:43 -03:00
Laerte Pereira 019443dd57 Remove settings from default implementation test 2024-01-31 10:08:07 -03:00
Laerte Pereira 2487e3cc03 Cleanup deprecated fingerprint code in scrapy.utils.request 2024-01-31 10:05:50 -03:00
Andrey Rakhmatullin 09a7efef7c Remove a defer.returnValue call. 2024-01-15 18:19:29 +04:00
Adrián Chaves e8dadb9592
scrapy parse: fix the signature of callbacks from the CLI (#6182) 2024-01-15 13:37:03 +01:00
Adrián Chaves fa0c598096
Add component getters to Crawler (#6181) 2024-01-15 13:14:02 +01:00
Mikhail Korobov 8ad17f7476
Merge pull request #6199 from Rotzbua/fix_flake8 
fix(flake8): lint errors
 2024-01-12 22:52:22 +05:00
Andrey Rakhmatullin c2baf4d0da Remove a defer.returnValue call. 2024-01-12 18:30:41 +04:00
Andrey Rakhmatullin becff65ccb
Merge pull request #6201 from wRAR/more-mitmproxy 
Fix and re-enable newer mitmproxy usage in tests.
 2024-01-12 14:26:39 +04:00
Mikhail Korobov 32fda2dc53
Merge pull request #6196 from Rotzbua/fix_typo 
fix(typo): correct `successfully`
 2024-01-12 00:36:16 +05:00
Andrey Rakhmatullin 68fccb1d58 Fix and re-enable newer mitmproxy usage in tests. 2024-01-06 01:35:56 +04:00
Rotzbua 745b8412f6
fix(flake8): lint errors 
E226 missing whitespace around arithmetic operator
E201 whitespace after '{'
 2024-01-05 14:54:00 +01:00
Rotzbua 0d445a3224
refactor(yield): use `yield from` syntax 2024-01-05 12:30:10 +01:00
Rotzbua c7b2b097b1
fix(typo): correct `successfully` 2024-01-05 10:50:45 +01:00
Adrián Chaves 0e78acb657
MediaPipeline: log media_to_download errors before stripping them (#5068) 2023-12-21 21:01:07 +01:00
Jalil SA d25cfe5315
Add JsonResponse (#6174) 2023-12-21 10:36:21 +01:00
Swati Anshu c67f730695
create_instance → build_from_crawler, build_from_settings (#6169) 2023-12-18 11:51:02 +01:00
Adrián Chaves 27781a85e7 Fix bad closing tags in XMLFeedSpider tests 2023-12-15 13:52:12 +01:00
Adrián Chaves cd6846b763 Merge remote-tracking branch 'scrapy/2.11' into 2.11-redos 2023-12-15 11:52:35 +01:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
Adrián Chaves 9655b0b8eb Mark slow tests, with their own tox env and CI job 2023-12-15 10:19:47 +01:00
Adrián Chaves 4f72b49f97 Fix namespaces nodename support for xmliter_lxml 2023-12-15 10:06:13 +01:00
Laerte Pereira b9c4ee26d7 Remove deprecated scrapy.downloadermiddlewares.decompression 2023-12-13 13:28:30 +01:00
Adrián Chaves 1533b69032 Test and address ReDoS attack vectors for open_in_browser 2023-12-13 12:01:35 +01:00
Laerte Pereira 09ce0ef526 Remove test_download_gzip_response test 2023-11-23 10:50:46 -03:00
Adrián Chaves 03d9866518 Also use DOWNLOAD_WARNSIZE for decompressions 2023-11-23 10:26:47 +01:00
Adrián Chaves e0b66c021a Mind Spider.download_maxsize and Request.meta['download_maxsize'] 2023-11-22 19:03:24 +01:00
Adrián Chaves 3fda2fe103 Protect against gzip bomb sitemaps 2023-11-22 18:34:37 +01:00
Adrián Chaves 9cc8703877 Protect against zstandard bombs 2023-11-22 17:53:00 +01:00
Adrián Chaves fba167c5e1 Protect against brotli bombs 2023-11-22 17:32:09 +01:00
Adrián Chaves 0c4a98f8e0 Protect against deflate bombs 2023-11-22 17:12:43 +01:00
Adrián Chaves 0bf29a7b1b Update test expectations 2023-11-22 16:10:50 +01:00
Adrián Chaves 6969041c5f Protect against gzip bombs 2023-11-22 15:52:00 +01:00
Adrián Chaves 1c4e93293a Merge remote-tracking branch 'scrapy/2.11' into 2.11-authorization 2023-11-17 15:42:08 +01:00
Adrián Chaves 1045856a50 Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-11-16 19:16:52 +04:00
Andrey Rakhmatullin 5e4fb0bc5f Re-enable uvloop tests on 3.12 (#6098) 2023-11-16 19:16:31 +04:00
Andrey Rakhmatullin 538192916f Merge pull request #6064 from wRAR/signals-proper 
Refactor installing signals.
 2023-11-16 19:15:30 +04:00
Andrey Rakhmatullin 59cfdeaa5c Merge pull request #6083 from wRAR/py3.12-release 
Adapt to the Python 3.12 final release
 2023-11-16 19:14:47 +04:00
Adrián Chaves 080fecd890 Drop the Authorization header on cross-domain redirect 2023-11-15 15:39:30 +01:00
Andrey Rakhmatullin b4acf5c827
Fix and remove most of the entries from the mypy ignore list (#6137) 2023-11-07 09:34:35 +01:00
Jeesang Kim dda6feb935
Improve assert readability (#6132) 2023-11-03 09:24:25 +01:00
Andrey Rakhmatullin 01d9d28324 Full typing for scrapy/http/response/__init__.py. 2023-11-01 01:10:32 +04:00
Chenwei Niu 1f797d0fdb
Removed some deprecated functions and functionalities (#6116) 2023-10-30 09:59:11 +01:00
Adrián Chaves 1d81585612
Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-10-30 09:35:29 +01:00
Andrey Rakhmatullin 6b0c18e921 Improve signal sending in test_shutdown_forced. 2023-10-19 23:13:47 +04:00
Andrey Rakhmatullin cc9c415bf3 Disable ipython for InteractiveShellTest. 2023-10-18 19:30:57 +04:00
Laerte Pereira 38dbd43993 Remove tests not necessary anymore 2023-10-18 06:29:47 -03:00
Laerte Pereira 644ab3af48 Remove support for feed storage backends without feed_options 2023-10-17 22:39:09 -03:00
Laerte Pereira 3db438127c Remove support for HttpCompressionMiddleware subclasses without stats 2023-10-17 21:38:38 -03:00
Laerte Pereira a2b9351f04 Remove support for using HttpAuthMiddleware without http_auth_domain 2023-10-17 20:20:38 -03:00
Laerte Pereira 2de95f1fc0 Remove leftover from _make_compatible removal 2023-10-17 19:44:12 -03:00
Laerte Pereira ad4e8b64d4 fix test_spider_manager_verify_interface test 2023-10-17 18:58:27 -03:00
Laerte Pereira aa0a425826 Remove deprecated code moved to itemloaders 2023-10-17 18:20:12 -03:00
Laerte Pereira 7a5cefbcfa Remove deprecated scrapy.downloadermiddlewares.decompression 2023-10-17 17:49:22 -03:00
Andrey Rakhmatullin 2cb1e10c76 Make shutdown tests more robust. 2023-10-17 23:56:54 +04:00
Andrey Rakhmatullin 991121fa91
Re-enable uvloop tests on 3.12 (#6098) 2023-10-17 19:24:44 +02:00
Andrey Rakhmatullin 5807970a22
Add typing hints to CaselessDict and Headers. (#6097) 2023-10-17 19:07:13 +02:00
Andrey Rakhmatullin 029a56384d Increase the timeout. 2023-10-09 00:16:41 +04:00
Andrey Rakhmatullin 9910ccf3ae Merge remote-tracking branch 'origin/master' into signals-proper 2023-10-08 23:02:11 +04:00
Andrey Rakhmatullin 2f436c05e0 Use SIGBREAK in Windows tests. 2023-10-08 22:55:05 +04:00
Andrey Rakhmatullin a338873e3a Re-enable bpython on Python 3.12. 2023-10-03 16:50:52 +04:00
Andrey Rakhmatullin fb4debda04 Use path_to_file_uri to fix the test on Windows. 2023-10-03 16:47:03 +04:00
Andrey Rakhmatullin 1ed9ed4f92 Require pyftpdlib that supports Python 3.12. 2023-10-02 21:33:02 +04:00
Andrey Rakhmatullin eb5e2e79ba Use SIGINT instead of SIGTERM to support Windows. 2023-09-24 23:31:46 +04:00
Andrey Rakhmatullin 4b5fb9b5a6 Add a test for SIGTERM handling. 2023-09-24 22:52:17 +04:00
Andrey Rakhmatullin d19e315b0b Add an interactive test for `scrapy shell`. 2023-09-24 21:51:19 +04:00
Andrey Rakhmatullin 922ff57384 Improve the backwards compatibility for RetryMiddleware.EXCEPTIONS_TO_RETRY. 2023-09-14 15:51:00 +04:00
Andrey Rakhmatullin 6428356584 Use twisted.python.log instead of failure.printTraceback(). 2023-09-13 18:20:42 +04:00
Andrey Rakhmatullin be0e33af92 Don't rely on unhandled exception logging in the expected to fail scripts. 2023-09-13 16:59:54 +04:00
Andrey Rakhmatullin 3756216339 Rename methods. 2023-09-13 15:47:22 +04:00
Andrey Rakhmatullin 619140717f Fix the new addon test. 2023-09-13 15:33:51 +04:00
Andrey Rakhmatullin 028a56b9a2 Improve and simplify tests. 2023-09-13 14:17:41 +04:00
Andrey Rakhmatullin 62183dc25f Merge branch 'master' into change-init-order 2023-09-12 22:16:21 +04:00
Andrey Rakhmatullin 77f39be407 Add integration tests for modifying settings via args. 2023-09-12 20:16:26 +04:00
Andrey Rakhmatullin e26bf4f918
Pin brotli for PyPy tests (#6045) 2023-09-12 16:18:25 +02:00
Andrey Rakhmatullin bb15c93a2b Add Crawler._load_settings(). 2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin 6629a61dd9 Fix one more testcase. 2023-09-08 20:15:49 +04:00
Andrey Rakhmatullin 036d5836d0 Move more things from Crawler.__init__() to .crawl(). 2023-09-08 19:29:05 +04:00
Andrey Rakhmatullin 97b98bf181 Roll back the update_settings() changes. 2023-09-07 17:02:37 +04:00
Maria Mitropoulou 721df895f9
Fix charset handling in MailSender #5096 (#5118) 2023-09-07 07:47:42 +02:00
Andrey Rakhmatullin b4380995da Merge remote-tracking branch 'origin/master' into change-init-order 2023-09-06 20:34:27 +04:00
Andrey Rakhmatullin b39d2d4353 Add a test. 2023-09-06 19:44:22 +04:00
Andrey Rakhmatullin d3b5c9be97 Deprecate running Crawler.crawl() twice. 2023-09-06 19:29:44 +04:00
Andrey Rakhmatullin 69282c5318 Merge branch 'master' into change-init-order 2023-09-05 19:23:36 +04:00
Víctor Ruiz c1dd5493ac
Fix enabled addons not being logged (#6036) 2023-09-04 08:03:35 +02:00
Mikhail Korobov 0fcb0554d1
Merge pull request #6016 from jxlil/fix/response.json 
Fix: response.json() call makes unnecessary memory allocation
 2023-08-31 14:07:24 +05:00
Georgiy Zatserklianyi 1f03cb1419 Merge remote-tracking branch 'upstream/master' into periodic_log_2 2023-08-21 17:07:18 +03:00
Laerte Pereira df2163ce6a
Remove datetime.utcnow() usage (#6014) 2023-08-21 15:51:49 +02:00
Jalil SA fd4292b722 fix: test_cache_json_response 2023-08-17 10:24:34 -06:00
Laerte Pereira b06936f111 Handle Tuple type on getdictorlist method, bump 3.12 python version 2023-08-14 10:33:48 -03:00
Andrey Rakhmatullin 34d050cfe5
Remove deprecated CrawlerRunner.spiders. (#6010) 2023-08-11 10:41:05 +02:00
Laerte Pereira 4a090d951a
Remove deprecated PythonItemExporter.binary (#6007) 2023-08-10 07:36:42 +02:00
Andrey Rakhmatullin 644a71bfd4 Use ftp:// URLs in FTP tests. 2023-08-07 00:17:52 +04:00
Andrey Rakhmatullin 471281d29e Fixes for scrapy/utils/iterators.py typing. 2023-08-06 23:05:02 +04:00
Andrey Rakhmatullin 518e56046e Check for async callbacks in contracts. 2023-08-06 17:28:34 +04:00
Andrey Rakhmatullin d1f87e4f08 More typing for scrapy/utils/iterators.py. 2023-08-06 00:35:33 +04:00
Georgiy Zatserklianyi e9b088f1fb periodic_log: typing 2023-08-04 12:49:22 +03:00
Laerte Pereira 09c63a178b
Remove the deprecated spider parameter from the engine (#5998) 2023-08-04 07:53:04 +02:00
Laerte Pereira b9c32a0cfd
Remove functions deprecated in 2.6.0 (#5996) 2023-08-03 17:06:55 +02:00
Andrey Rakhmatullin 9f9a2292e0
Deprecate the custom attribute of build_component_list(). (#5993) 2023-08-02 14:21:06 +02:00
Andrey Rakhmatullin 06ebdee35d
Merge pull request #5950 from wRAR/addons 
Add-ons
 2023-08-01 22:18:57 +04:00
Andrey Rakhmatullin e58b8078f0 Improve NotConfigured handling when clspath is a class. 2023-08-01 20:20:15 +04:00
Andrey Rakhmatullin 41a4a163e3 Handle NotConfigured in add-ons. 2023-07-31 21:09:18 +04:00
Andrey Rakhmatullin d8c5b41559 Add more addon tests. 2023-07-31 19:09:21 +04:00
Andrey Rakhmatullin bddbbc522a
Merge pull request #5979 from BurnzZ/closespider-timeout-no-item 2023-07-28 12:56:44 +04:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Andrey Rakhmatullin 6e3e3c2172
Merge pull request #5977 from scrapy/typing-settings 
Typing for scrapy.settings.BaseSettings
 2023-07-24 21:40:52 +04:00
Kevin Lloyd Bernal 9a1bf40c2f expose AWS_REGION_NAME in S3FeedStorage 2023-07-24 16:37:52 +08:00
Laerte Pereira 7522aeed35
fix: -O/-o option does not work with absolute paths on Windows (#5971) 2023-07-24 09:53:41 +02:00
Mikhail Korobov af2aa4b421
Merge pull request #5847 from namelessGonbai/feat/FixFeedExport 
Fix FeedExporter not to export empty file
 2023-07-22 23:18:36 +05:00
Adnan Awan 5d91ea12d6
Fix JsonItemExporter puts lone comma in the output if encoder fails (#5952) 
* Partial fix for #3090 - only addresses JSON feeds.

* Adding test case for #3090 to Json Exporter

* Changing the deliberate-fail JSON example to a complex

* Further tightening JsonItemExporter behaviour to prevent corruption.

Based on Mikhail's observation that to_bytes can fail also, leading
to the same dangling comma as the failure to encode to JSON.

Added a new test case to avoid reversion.

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add initial changes from cathal's PR
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Handle exception not to add empty item.
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add comment for handling the exception
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Remove unused import
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Fix invalid json issue
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Perform CR changes
- https://github.com/scrapy/scrapy/issues/3090

---------

Co-authored-by: Cathal Garvey <cathalgarvey@cathalgarvey.me>
 2023-07-22 23:13:40 +05:00
Andrey Rakhmatullin e058a05763 Skip tests that don't work on Python 3.12. 2023-07-22 17:51:13 +04:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Andrey Rakhmatullin 5c34f34ecb Make AddonManager.add() private. 2023-07-18 19:50:08 +04:00
Kevin Lloyd Bernal 368ab29ffc improve tests by having SlowSpider 2023-07-18 22:29:15 +08:00
Kevin Lloyd Bernal 3f5bbe3a8f introduce CLOSESPIDER_TIMEOUT_NO_ITEM in CloseSpider 2023-07-18 18:30:21 +08:00
Andrey Rakhmatullin 043a24410b Disable pylint for broken code. 2023-07-17 00:13:49 +04:00
Andrey Rakhmatullin 187e8f9a2d Typing for scrapy/settings/__init__.py. 2023-07-17 00:01:36 +04:00
Andrey Rakhmatullin c92c9af075 Add create_instance support to addons. 2023-06-29 15:04:46 +04:00
Georgiy Zatserklianyi 6fd94fdcb3 periodic_log: tests updated (errors fixed) 2023-06-28 19:55:33 +03:00
Andrey Rakhmatullin 9612ae3e93 Remove more code. 2023-06-28 18:44:30 +04:00
Laerte Pereira b6196309cb
fix: Return value instead of `SettingsAttribute` object when using `pop` method (#5963) 2023-06-28 08:28:49 +02:00
Georgiy Zatserklianyi 56c38231b4 periodic_log: tests for logging stats added 2023-06-27 11:48:51 +03:00
Georgiy Zatserklianyi ebce5b4bcb periodic_log: tests for logging deltas added 2023-06-27 09:22:50 +03:00
Andrey Rakhmatullin e7124447f7 Remove unneeded code. 2023-06-26 16:57:46 +04:00
Andrey Rakhmatullin 4e21e3e59d Merge remote-tracking branch 'origin/master' into addons 2023-06-26 14:25:35 +04:00
Georgiy Zatserklianyi 2ce4856508 periodic_log: Exception handling on init updated 2023-06-25 12:16:59 +03:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Laerte Pereira a3f8912d69 chore: Rename test 2023-06-23 00:15:32 -03:00
Laerte Pereira 876feaf339 chore: Use dunder to delete item instead of del keyword to handle immutable settings 2023-06-23 00:14:31 -03:00
Laerte Pereira 080b9bd0b8 chore: Implement `pop` method on `BaseSettings` class 2023-06-22 23:58:03 -03:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Andrey Rakhmatullin 1b78e48944
Merge pull request #5146 from elacuesta/case-insensitive-dict 
CaseInsensitiveDict (deprecate CaselessDict)
 2023-06-21 23:48:40 +04:00
Andrey Rakhmatullin 39282d7bd0 Merge branch 'master' into case-insensitive-dict 2023-06-21 22:12:44 +04:00
Marc Hernández 5360ba34bc
IOError and other cleanup (#4716) 2023-06-21 20:08:53 +02:00
Laerte Pereira a93a63c208 fix: move import to inside function 2023-06-20 17:05:49 -03:00
Laerte Pereira 6afb31b82b chore: Add `packaging` to tests deps 2023-06-20 16:48:07 -03:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00
Georgiy Zatserklianyi 6e65eeb07b periodic_log: tests [wip] added 2023-06-16 14:30:50 +03:00
Andrey Rakhmatullin 79bf8b1f2e Test cleanup. 2023-06-15 19:28:14 +04:00
Andrey Rakhmatullin 2f9ebb66c3 Remove some dead code/docs. 2023-06-15 17:00:38 +04:00
Andrey Rakhmatullin 58300e066f
Merge pull request #5949 from wRAR/tests-as-installed 
Allow running tests against the installed version
 2023-06-14 23:15:04 +04:00
Andrey Rakhmatullin c7f78a8305 Revert "Remove load_module_or_object." 
This reverts commit 815af43120.
 2023-06-14 20:43:18 +04:00
Andrey Rakhmatullin 7cfdca8f9b Actually run test_set_asyncio_event_loop(). 2023-06-14 18:23:29 +04:00
Andrey Rakhmatullin 815af43120 Remove load_module_or_object. 2023-06-14 18:11:14 +04:00
Andrey Rakhmatullin 7ebb8256f0 Some cleanup. 2023-06-14 17:52:11 +04:00
Andrey Rakhmatullin fdbc141b23 Replace pkg_resources with packaging. 2023-06-14 17:23:50 +04:00
Andrey Rakhmatullin 55ac26228b Remove builtin addons. 2023-06-14 16:50:43 +04:00
Andrey Rakhmatullin 075ad6f196 Test, linter etc. fixes. 2023-06-14 16:34:18 +04:00
Andrey Rakhmatullin 898e3045a1 Merge branch 'master' into addons 2023-06-14 15:37:08 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 8a64f3e8de
Merge branch 'master' into feat/FixFeedExport 2023-06-13 18:21:06 +04:00
Andrey Rakhmatullin 9e0bfc4a3d
Merge pull request #5705 from srki24/issue5043-feed_export 
S3 CsvItemExporter read of closed file error #5043
 2023-06-13 18:14:27 +04:00
Andrey Rakhmatullin 493ea43538 Improve finding tests.test_cmdline.settings. 2023-06-12 21:50:29 +04:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Andrey Rakhmatullin 54fa04aa0a Add typing to scrapy/utils/test.py, fix a FTP test. 2023-05-07 22:39:27 +04:00
Andrey Rakhmatullin ea15ff1d32
Merge pull request #5915 from jxlil/feature/support-parsel-jmespath 
Added support for the Parsel JMESPath feature
 2023-05-03 13:18:39 +04:00
Jalil SA a038faf11c fix: tests/tes_selector.py 2023-05-02 19:40:04 -06:00
Jalil SA 4bb99fd2f3 fix: pylint 2023-05-02 19:26:20 -06:00
Jalil SA a604dfae5c update tests 2023-05-02 19:19:00 -06:00
Laerte Pereira 7b49aa1b01 chore: add tests 2023-05-02 12:53:04 -03:00
Jalil SA 578606779d update tests 2023-04-29 00:52:39 -06:00
Jalil SA 3d29f20fc2 added tests for jmespath 2023-04-28 23:54:09 -06:00
guillermo-bondonno b50c032ee9
Add feed_slot_closed and feed_exporter_closed signals (#5876) 2023-04-26 08:20:37 +02:00
Andrey Rakhmatullin abc9c1ac6f
Merge pull request #5901 from jxlil/fix/5899 
Fix: Request.from_curl() with $-prefixed string literals
 2023-04-19 13:35:01 +04:00
tstauder ef61fb5698
Fix flaky tests involving feed export batches (#5898) 
Co-authored-by: jmannoop <jmannoop@umich.edu>
 2023-04-19 08:33:32 +02:00
Jalil SA 88c58a8c9c feat: added test_post_data_raw_with_string_prefix 2023-04-18 22:28:16 -05:00
guillermo-bondonno 441ac196e4
Implement a request_to_curl function (#5892) 2023-04-13 17:46:59 +02:00
Serhii A c2a31974ff
Improve get_func_args (#5885) 2023-04-13 11:44:20 +02:00
Andrey Rakhmatullin d911837389
Merge pull request #5881 from sbartlett97/LinkExtractor-Patch 
Patched LxmlParserLinkExtractor
 2023-04-12 12:28:21 +04:00
Adrián Chaves 3f0c2fae5e Skip test_skip_bad_links based on the w3lib version 2023-04-12 09:28:28 +02:00
Kartik Kumar d47c732ae9
base64-decode GCS checksums (#5891) 2023-04-11 18:25:42 +02:00
Andrey Rakhmatullin 1e20ba0a1b
Merge branch 'master' into feat/FixFeedExport 2023-04-11 19:48:19 +04:00
Felipe Boff Nunes a0e2e36b52 adjustments 2023-04-06 14:23:19 -03:00
Felipe Boff Nunes 0ffd1667ba adjustments 2023-04-06 14:17:35 -03:00
Samuel Bartlett 618e82dbe1 Exclude test for python versionbs less than 3.8 2023-03-31 15:12:47 +00:00
Samuel Bartlett 608b7de582 Skip new test if python version less than 3.8 2023-03-31 14:38:06 +00:00
Samuel Bartlett 4043560547 remove utf-8 encoding flag from test 2023-03-31 12:29:22 +00:00
Samuel Bartlett 00d93026c8 Fix bad test case 2023-03-31 08:30:19 +00:00
Samuel Bartlett 7cb7cf1ad1 Add link extractor back to test 2023-03-31 08:09:02 +00:00
Samuel Bartlett 9cbcf7724d Add test to make sure spider doesn't crash on bad 2023-03-31 08:07:43 +00:00
jazzthief 4ebc08ef10
Switch from `is_boto3_available()` to `IS_BOTO3_AVAILABLE` var 2023-03-16 17:24:11 +01:00
kenshi kikuchi c8ed793257 Fix test_export_no_items_multiple_feeds 2023-03-16 17:16:14 +09:00
Adrián Chaves 7e7b41c6b3 Fix test expectations for boto3 2023-03-15 16:38:13 +01:00
Adrián Chaves 39dbfa1d82 Minimize test reliance on S3; do not install botocore on the default test environments 2023-03-15 15:31:39 +01:00
auxsvr d60b4edd11
Prevent an edge case that creates an extra event loop (#5832) 2023-03-14 09:31:13 +01:00
Alex 101a0c32d7
Support genspider with HTTPS (#5808) 2023-03-14 08:13:44 +01:00
Mohammadtaher Abbasi afafc2781a
Fix parse command issues with asyncio (#5824) 2023-03-10 16:20:15 +01:00
Andrey Rakhmatullin 3659a8c8d9
Merge pull request #5328 from GeorgeA92/per_slot_settings 
Per slot settings
 2023-03-09 22:31:24 +05:00
kenshi kikuchi d311779887 Fix FeedExporter 
+ Fix FeedExporter not to export empty file
+ Change default value of FEED_STORE_EMPTY
 2023-03-08 16:24:09 +09:00
Jalil SA 8aca47e25d
Add setdefault to BaseSettings (#5821) 2023-03-07 10:26:59 +01:00