RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,915 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

2050 Commits

Author SHA1 Message Date
Andrey Rakhmatullin bddbbc522a
Merge pull request #5979 from BurnzZ/closespider-timeout-no-item 2023-07-28 12:56:44 +04:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Andrey Rakhmatullin 6e3e3c2172
Merge pull request #5977 from scrapy/typing-settings 
Typing for scrapy.settings.BaseSettings
 2023-07-24 21:40:52 +04:00
Kevin Lloyd Bernal 9a1bf40c2f expose AWS_REGION_NAME in S3FeedStorage 2023-07-24 16:37:52 +08:00
Laerte Pereira 7522aeed35
fix: -O/-o option does not work with absolute paths on Windows (#5971) 2023-07-24 09:53:41 +02:00
Mikhail Korobov af2aa4b421
Merge pull request #5847 from namelessGonbai/feat/FixFeedExport 
Fix FeedExporter not to export empty file
 2023-07-22 23:18:36 +05:00
Adnan Awan 5d91ea12d6
Fix JsonItemExporter puts lone comma in the output if encoder fails (#5952) 
* Partial fix for #3090 - only addresses JSON feeds.

* Adding test case for #3090 to Json Exporter

* Changing the deliberate-fail JSON example to a complex

* Further tightening JsonItemExporter behaviour to prevent corruption.

Based on Mikhail's observation that to_bytes can fail also, leading
to the same dangling comma as the failure to encode to JSON.

Added a new test case to avoid reversion.

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add initial changes from cathal's PR
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Handle exception not to add empty item.
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add comment for handling the exception
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Remove unused import
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Fix invalid json issue
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Perform CR changes
- https://github.com/scrapy/scrapy/issues/3090

---------

Co-authored-by: Cathal Garvey <cathalgarvey@cathalgarvey.me>
 2023-07-22 23:13:40 +05:00
Andrey Rakhmatullin e058a05763 Skip tests that don't work on Python 3.12. 2023-07-22 17:51:13 +04:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Kevin Lloyd Bernal 368ab29ffc improve tests by having SlowSpider 2023-07-18 22:29:15 +08:00
Kevin Lloyd Bernal 3f5bbe3a8f introduce CLOSESPIDER_TIMEOUT_NO_ITEM in CloseSpider 2023-07-18 18:30:21 +08:00
Andrey Rakhmatullin 043a24410b Disable pylint for broken code. 2023-07-17 00:13:49 +04:00
Andrey Rakhmatullin 187e8f9a2d Typing for scrapy/settings/__init__.py. 2023-07-17 00:01:36 +04:00
Laerte Pereira b6196309cb
fix: Return value instead of `SettingsAttribute` object when using `pop` method (#5963) 2023-06-28 08:28:49 +02:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Laerte Pereira a3f8912d69 chore: Rename test 2023-06-23 00:15:32 -03:00
Laerte Pereira 876feaf339 chore: Use dunder to delete item instead of del keyword to handle immutable settings 2023-06-23 00:14:31 -03:00
Laerte Pereira 080b9bd0b8 chore: Implement `pop` method on `BaseSettings` class 2023-06-22 23:58:03 -03:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Andrey Rakhmatullin 1b78e48944
Merge pull request #5146 from elacuesta/case-insensitive-dict 
CaseInsensitiveDict (deprecate CaselessDict)
 2023-06-21 23:48:40 +04:00
Andrey Rakhmatullin 39282d7bd0 Merge branch 'master' into case-insensitive-dict 2023-06-21 22:12:44 +04:00
Marc Hernández 5360ba34bc
IOError and other cleanup (#4716) 2023-06-21 20:08:53 +02:00
Laerte Pereira a93a63c208 fix: move import to inside function 2023-06-20 17:05:49 -03:00
Laerte Pereira 6afb31b82b chore: Add `packaging` to tests deps 2023-06-20 16:48:07 -03:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00
Andrey Rakhmatullin 58300e066f
Merge pull request #5949 from wRAR/tests-as-installed 
Allow running tests against the installed version
 2023-06-14 23:15:04 +04:00
Andrey Rakhmatullin 7cfdca8f9b Actually run test_set_asyncio_event_loop(). 2023-06-14 18:23:29 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 8a64f3e8de
Merge branch 'master' into feat/FixFeedExport 2023-06-13 18:21:06 +04:00
Andrey Rakhmatullin 9e0bfc4a3d
Merge pull request #5705 from srki24/issue5043-feed_export 
S3 CsvItemExporter read of closed file error #5043
 2023-06-13 18:14:27 +04:00
Andrey Rakhmatullin 493ea43538 Improve finding tests.test_cmdline.settings. 2023-06-12 21:50:29 +04:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Andrey Rakhmatullin 54fa04aa0a Add typing to scrapy/utils/test.py, fix a FTP test. 2023-05-07 22:39:27 +04:00
Andrey Rakhmatullin ea15ff1d32
Merge pull request #5915 from jxlil/feature/support-parsel-jmespath 
Added support for the Parsel JMESPath feature
 2023-05-03 13:18:39 +04:00
Jalil SA a038faf11c fix: tests/tes_selector.py 2023-05-02 19:40:04 -06:00
Jalil SA 4bb99fd2f3 fix: pylint 2023-05-02 19:26:20 -06:00
Jalil SA a604dfae5c update tests 2023-05-02 19:19:00 -06:00
Laerte Pereira 7b49aa1b01 chore: add tests 2023-05-02 12:53:04 -03:00
Jalil SA 578606779d update tests 2023-04-29 00:52:39 -06:00
Jalil SA 3d29f20fc2 added tests for jmespath 2023-04-28 23:54:09 -06:00
guillermo-bondonno b50c032ee9
Add feed_slot_closed and feed_exporter_closed signals (#5876) 2023-04-26 08:20:37 +02:00
Andrey Rakhmatullin abc9c1ac6f
Merge pull request #5901 from jxlil/fix/5899 
Fix: Request.from_curl() with $-prefixed string literals
 2023-04-19 13:35:01 +04:00
tstauder ef61fb5698
Fix flaky tests involving feed export batches (#5898) 
Co-authored-by: jmannoop <jmannoop@umich.edu>
 2023-04-19 08:33:32 +02:00
Jalil SA 88c58a8c9c feat: added test_post_data_raw_with_string_prefix 2023-04-18 22:28:16 -05:00
guillermo-bondonno 441ac196e4
Implement a request_to_curl function (#5892) 2023-04-13 17:46:59 +02:00
Serhii A c2a31974ff
Improve get_func_args (#5885) 2023-04-13 11:44:20 +02:00
Andrey Rakhmatullin d911837389
Merge pull request #5881 from sbartlett97/LinkExtractor-Patch 
Patched LxmlParserLinkExtractor
 2023-04-12 12:28:21 +04:00
Adrián Chaves 3f0c2fae5e Skip test_skip_bad_links based on the w3lib version 2023-04-12 09:28:28 +02:00
Kartik Kumar d47c732ae9
base64-decode GCS checksums (#5891) 2023-04-11 18:25:42 +02:00