RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,891 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

9891 Commits

Author SHA1 Message Date
Adnan Awan 5d91ea12d6
Fix JsonItemExporter puts lone comma in the output if encoder fails (#5952) 
* Partial fix for #3090 - only addresses JSON feeds.

* Adding test case for #3090 to Json Exporter

* Changing the deliberate-fail JSON example to a complex

* Further tightening JsonItemExporter behaviour to prevent corruption.

Based on Mikhail's observation that to_bytes can fail also, leading
to the same dangling comma as the failure to encode to JSON.

Added a new test case to avoid reversion.

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add initial changes from cathal's PR
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Handle exception not to add empty item.
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add comment for handling the exception
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Remove unused import
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Fix invalid json issue
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Perform CR changes
- https://github.com/scrapy/scrapy/issues/3090

---------

Co-authored-by: Cathal Garvey <cathalgarvey@cathalgarvey.me>
 2023-07-22 23:13:40 +05:00
Mikhail Korobov 1a0eb60c87
Merge pull request #5984 from wRAR/ci-3.12 
Python 3.12 support
 2023-07-22 22:51:39 +05:00
Andrey Rakhmatullin 53f8570786 Add support for the new entry_points() interface. 2023-07-22 18:46:44 +04:00
Andrey Rakhmatullin 21b6dc5f9f Add 3.12 CI jobs. 2023-07-22 17:55:32 +04:00
Andrey Rakhmatullin a346732275 Skip more non-test files during discovery. 2023-07-22 17:54:55 +04:00
Andrey Rakhmatullin e058a05763 Skip tests that don't work on Python 3.12. 2023-07-22 17:51:13 +04:00
Andrey Rakhmatullin 583df9f7d0 Simplify skipping uvloop tests. 2023-07-22 17:44:37 +04:00
Andrey Rakhmatullin 8055a948dc
Merge pull request #5965 from andersoncarlosfs/patch-1 
Removing hard code entries
 2023-07-05 12:51:06 +04:00
Anderson Carlos Ferreira da Silva 7ce3d8f98a
removing hard code entries 2023-07-05 16:56:34 +09:00
Laerte Pereira b6196309cb
fix: Return value instead of `SettingsAttribute` object when using `pop` method (#5963) 2023-06-28 08:28:49 +02:00
Andrey Rakhmatullin 510574216d
Merge pull request #5925 from wRAR/typing-utils 
Typing for scrapy/utils
 2023-06-24 20:08:41 +04:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Andrey Rakhmatullin 85f5122965
Merge pull request #5960 from Laerte/master 
chore: Implement `pop` method on `BaseSettings` class
 2023-06-24 18:23:17 +04:00
Laerte Pereira a3f8912d69 chore: Rename test 2023-06-23 00:15:32 -03:00
Laerte Pereira 876feaf339 chore: Use dunder to delete item instead of del keyword to handle immutable settings 2023-06-23 00:14:31 -03:00
Laerte Pereira 080b9bd0b8 chore: Implement `pop` method on `BaseSettings` class 2023-06-22 23:58:03 -03:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Andrey Rakhmatullin 1b78e48944
Merge pull request #5146 from elacuesta/case-insensitive-dict 
CaseInsensitiveDict (deprecate CaselessDict)
 2023-06-21 23:48:40 +04:00
Andrey Rakhmatullin 39282d7bd0 Merge branch 'master' into case-insensitive-dict 2023-06-21 22:12:44 +04:00
Marc Hernández 5360ba34bc
IOError and other cleanup (#4716) 2023-06-21 20:08:53 +02:00
Aaron Smith ee215a2970
Change redirect text from Response.request docs (#5937) 2023-06-21 20:05:39 +02:00
Andrey Rakhmatullin e9e1034af3
Merge pull request #5958 from Laerte/master 
Remove `pkg_resources` usage
 2023-06-21 12:47:01 +04:00
Laerte Pereira 82cf00bbc9 fix: default value 2023-06-20 18:27:03 -03:00
Laerte Pereira 0b1da44a05 chore: Remove deprecated code 2023-06-20 17:14:21 -03:00
Laerte Pereira a93a63c208 fix: move import to inside function 2023-06-20 17:05:49 -03:00
Laerte Pereira 6e1af20ac4 fix: add `build-system` 2023-06-20 17:00:01 -03:00
Laerte Pereira 6afb31b82b chore: Add `packaging` to tests deps 2023-06-20 16:48:07 -03:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Andrey Rakhmatullin fae5538910
Merge pull request #5953 from wRAR/drop-3.7 
Drop Python 3.7 support
 2023-06-19 13:59:13 +04:00
Andrey Rakhmatullin 075b89eab5 Bump lxml and cryptography to versions with 3.8 wheels available. 2023-06-18 19:08:41 +04:00
Andrey Rakhmatullin 1b2c9a3e0a Bump isort and flake8 versions. 2023-06-18 18:38:56 +04:00
Andrey Rakhmatullin 2122278d4b Drop Python 3.7 support. 2023-06-18 18:37:50 +04:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00
Andrey Rakhmatullin 58300e066f
Merge pull request #5949 from wRAR/tests-as-installed 
Allow running tests against the installed version
 2023-06-14 23:15:04 +04:00
Andrey Rakhmatullin bc09c9e74a
Merge pull request #5951 from wRAR/fix-asyncio-loop-test 
Actually run test_set_asyncio_event_loop().
 2023-06-14 20:01:26 +04:00
Andrey Rakhmatullin 7cfdca8f9b Actually run test_set_asyncio_event_loop(). 2023-06-14 18:23:29 +04:00
Andrey Rakhmatullin d2e7f6e209
Merge pull request #5948 from scrapy/fix-extra-deps-pinned 
Fix extra-deps-pinned tests.
 2023-06-13 21:00:31 +04:00
Andrey Rakhmatullin 0adbd210ac Fix extra-deps-pinned tests. 2023-06-13 19:34:26 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 9e0bfc4a3d
Merge pull request #5705 from srki24/issue5043-feed_export 
S3 CsvItemExporter read of closed file error #5043
 2023-06-13 18:14:27 +04:00
Andrey Rakhmatullin 493ea43538 Improve finding tests.test_cmdline.settings. 2023-06-12 21:50:29 +04:00
Andrey Rakhmatullin 49839d6071 Don't rely on get_testenv() for running mockserver. 2023-06-04 19:59:58 +04:00
Andrey Rakhmatullin 33b418dc84
Merge pull request #5927 from bulatbulat48/master 
Add additional requests examples.
 2023-05-16 19:13:02 +04:00
Adrián Chaves d362699fa3 Move inline request examples to the coroutines documentation 2023-05-16 13:39:02 +02:00
Adrián Chaves e4cf8fc121
Update asyncio.rst 2023-05-15 18:51:58 +02:00
Andrey Rakhmatullin 43afd38813
Merge pull request #5931 from belaeifert/improve_overwrite_feed_option 
Improve the overwrite feed option documentation
 2023-05-12 15:46:23 +04:00
isabela_catanante 5adada5d19 Improve the overwrite feed option documentation 2023-05-12 12:55:24 +02:00
bulat 8de2064ba3 add import. 2023-05-11 13:22:33 +05:00
bulat 4878cc7ef0 Add proper imports. 2023-05-11 13:19:40 +05:00
bulat b62c1263de add import to the example. 2023-05-11 13:15:30 +05:00