RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,915 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

9915 Commits

Author SHA1 Message Date
Andrey Rakhmatullin bddbbc522a
Merge pull request #5979 from BurnzZ/closespider-timeout-no-item 2023-07-28 12:56:44 +04:00
freddiewanah c4f0aa4fdf
Refactor test cases to improve unit test quality (#5986) 2023-07-26 13:09:03 +02:00
Andrey Rakhmatullin 6e3e3c2172
Merge pull request #5977 from scrapy/typing-settings 
Typing for scrapy.settings.BaseSettings
 2023-07-24 21:40:52 +04:00
Andrey Rakhmatullin 7a1afff655
Merge pull request #5980 from BurnzZ/s3-region 
expose AWS_REGION_NAME in S3FeedStorage
 2023-07-24 21:40:37 +04:00
Kevin Lloyd Bernal a689fe5baf move region_name param as kwargs 2023-07-24 16:37:52 +08:00
Kevin Lloyd Bernal 9a1bf40c2f expose AWS_REGION_NAME in S3FeedStorage 2023-07-24 16:37:52 +08:00
Laerte Pereira 7522aeed35
fix: -O/-o option does not work with absolute paths on Windows (#5971) 2023-07-24 09:53:41 +02:00
Andrey Rakhmatullin 5e1582491b mypy --show-error-codes is the default now. 2023-07-22 23:19:25 +04:00
Mikhail Korobov af2aa4b421
Merge pull request #5847 from namelessGonbai/feat/FixFeedExport 
Fix FeedExporter not to export empty file
 2023-07-22 23:18:36 +05:00
Adnan Awan 5d91ea12d6
Fix JsonItemExporter puts lone comma in the output if encoder fails (#5952) 
* Partial fix for #3090 - only addresses JSON feeds.

* Adding test case for #3090 to Json Exporter

* Changing the deliberate-fail JSON example to a complex

* Further tightening JsonItemExporter behaviour to prevent corruption.

Based on Mikhail's observation that to_bytes can fail also, leading
to the same dangling comma as the failure to encode to JSON.

Added a new test case to avoid reversion.

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add initial changes from cathal's PR
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Handle exception not to add empty item.
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add comment for handling the exception
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Remove unused import
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Fix invalid json issue
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Perform CR changes
- https://github.com/scrapy/scrapy/issues/3090

---------

Co-authored-by: Cathal Garvey <cathalgarvey@cathalgarvey.me>
 2023-07-22 23:13:40 +05:00
Mikhail Korobov 1a0eb60c87
Merge pull request #5984 from wRAR/ci-3.12 
Python 3.12 support
 2023-07-22 22:51:39 +05:00
Andrey Rakhmatullin 53f8570786 Add support for the new entry_points() interface. 2023-07-22 18:46:44 +04:00
Andrey Rakhmatullin 21b6dc5f9f Add 3.12 CI jobs. 2023-07-22 17:55:32 +04:00
Andrey Rakhmatullin a346732275 Skip more non-test files during discovery. 2023-07-22 17:54:55 +04:00
Andrey Rakhmatullin e058a05763 Skip tests that don't work on Python 3.12. 2023-07-22 17:51:13 +04:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Kevin Lloyd Bernal 368ab29ffc improve tests by having SlowSpider 2023-07-18 22:29:15 +08:00
Kevin Lloyd Bernal 2f787a27dc fix conditional on task_no_item 2023-07-18 20:49:33 +08:00
Kevin Lloyd Bernal 3f5bbe3a8f introduce CLOSESPIDER_TIMEOUT_NO_ITEM in CloseSpider 2023-07-18 18:30:21 +08:00
Andrey Rakhmatullin db86f91789 Unbreak isort breakage. 2023-07-17 23:17:11 +04:00
Andrey Rakhmatullin 043a24410b Disable pylint for broken code. 2023-07-17 00:13:49 +04:00
Andrey Rakhmatullin 187e8f9a2d Typing for scrapy/settings/__init__.py. 2023-07-17 00:01:36 +04:00
Andrey Rakhmatullin 93962ebefc Bump typing package versions. 2023-07-14 23:20:35 +04:00
Andrey Rakhmatullin 8055a948dc
Merge pull request #5965 from andersoncarlosfs/patch-1 
Removing hard code entries
 2023-07-05 12:51:06 +04:00
Anderson Carlos Ferreira da Silva 7ce3d8f98a
removing hard code entries 2023-07-05 16:56:34 +09:00
Laerte Pereira b6196309cb
fix: Return value instead of `SettingsAttribute` object when using `pop` method (#5963) 2023-06-28 08:28:49 +02:00
Andrey Rakhmatullin 510574216d
Merge pull request #5925 from wRAR/typing-utils 
Typing for scrapy/utils
 2023-06-24 20:08:41 +04:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Andrey Rakhmatullin 85f5122965
Merge pull request #5960 from Laerte/master 
chore: Implement `pop` method on `BaseSettings` class
 2023-06-24 18:23:17 +04:00
Laerte Pereira a3f8912d69 chore: Rename test 2023-06-23 00:15:32 -03:00
Laerte Pereira 876feaf339 chore: Use dunder to delete item instead of del keyword to handle immutable settings 2023-06-23 00:14:31 -03:00
Laerte Pereira 080b9bd0b8 chore: Implement `pop` method on `BaseSettings` class 2023-06-22 23:58:03 -03:00
Andrey Rakhmatullin e71d6d67e5
Apply suggestions from code review 2023-06-22 21:10:50 +04:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Andrey Rakhmatullin 1b78e48944
Merge pull request #5146 from elacuesta/case-insensitive-dict 
CaseInsensitiveDict (deprecate CaselessDict)
 2023-06-21 23:48:40 +04:00
Andrey Rakhmatullin 39282d7bd0 Merge branch 'master' into case-insensitive-dict 2023-06-21 22:12:44 +04:00
Marc Hernández 5360ba34bc
IOError and other cleanup (#4716) 2023-06-21 20:08:53 +02:00
Aaron Smith ee215a2970
Change redirect text from Response.request docs (#5937) 2023-06-21 20:05:39 +02:00
Andrey Rakhmatullin e9e1034af3
Merge pull request #5958 from Laerte/master 
Remove `pkg_resources` usage
 2023-06-21 12:47:01 +04:00
Laerte Pereira 82cf00bbc9 fix: default value 2023-06-20 18:27:03 -03:00
Laerte Pereira 0b1da44a05 chore: Remove deprecated code 2023-06-20 17:14:21 -03:00
Laerte Pereira a93a63c208 fix: move import to inside function 2023-06-20 17:05:49 -03:00
Laerte Pereira 6e1af20ac4 fix: add `build-system` 2023-06-20 17:00:01 -03:00
Laerte Pereira 6afb31b82b chore: Add `packaging` to tests deps 2023-06-20 16:48:07 -03:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Andrey Rakhmatullin fae5538910
Merge pull request #5953 from wRAR/drop-3.7 
Drop Python 3.7 support
 2023-06-19 13:59:13 +04:00
Andrey Rakhmatullin 075b89eab5 Bump lxml and cryptography to versions with 3.8 wheels available. 2023-06-18 19:08:41 +04:00
Andrey Rakhmatullin 1b2c9a3e0a Bump isort and flake8 versions. 2023-06-18 18:38:56 +04:00
Andrey Rakhmatullin 2122278d4b Drop Python 3.7 support. 2023-06-18 18:37:50 +04:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00