RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,718 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

60 Commits

Author SHA1 Message Date
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin cec0aeca58
Bump ruff, switch from black to ruff-format (#6631) 2025-01-27 11:07:09 +01:00
Andrey Rakhmatullin c330a399dc
Merge pull request #6601 from wRAR/ruff-rules-5 
Add more Ruff rules, do some pylint cleanups
 2025-01-02 17:38:15 +04:00
Andrey Rakhmatullin 6ae5b92671
Drop the remaining unittest.main() blocks. (#6602) 2025-01-02 12:45:04 +01:00
Andrey Rakhmatullin b70443f2d0 Split ruff and pylint ignores into two categories, some pylint cleanup. 2025-01-02 01:03:42 +05:00
Andrey Rakhmatullin 838ff99f37 Enable RUF Ruff rules. 2025-01-01 21:41:08 +05:00
Laerte Pereira 26a16f2c43 Fix tests 2024-02-28 16:36:19 -03:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Andrey Rakhmatullin b4acf5c827
Fix and remove most of the entries from the mypy ignore list (#6137) 2023-11-07 09:34:35 +01:00
Laerte Pereira 4a090d951a
Remove deprecated PythonItemExporter.binary (#6007) 2023-08-10 07:36:42 +02:00
Adnan Awan 5d91ea12d6
Fix JsonItemExporter puts lone comma in the output if encoder fails (#5952) 
* Partial fix for #3090 - only addresses JSON feeds.

* Adding test case for #3090 to Json Exporter

* Changing the deliberate-fail JSON example to a complex

* Further tightening JsonItemExporter behaviour to prevent corruption.

Based on Mikhail's observation that to_bytes can fail also, leading
to the same dangling comma as the failure to encode to JSON.

Added a new test case to avoid reversion.

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add initial changes from cathal's PR
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Handle exception not to add empty item.
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Add comment for handling the exception
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Remove unused import
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Fix invalid json issue
- https://github.com/scrapy/scrapy/issues/3090

* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails

- Perform CR changes
- https://github.com/scrapy/scrapy/issues/3090

---------

Co-authored-by: Cathal Garvey <cathalgarvey@cathalgarvey.me>
 2023-07-22 23:13:40 +05:00
Andrey Rakhmatullin 9e0bfc4a3d
Merge pull request #5705 from srki24/issue5043-feed_export 
S3 CsvItemExporter read of closed file error #5043
 2023-06-13 18:14:27 +04:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Adrián Chaves 426f3ebb7b Fix typo: causes it be closed → causes it to be closed 2023-02-02 05:58:32 +01:00
Adrián Chaves 32bc8bd436 Merge remote-tracking branch 'scrapy/master' into issue5043-feed_export 2023-02-02 05:56:48 +01:00
Adrián Chaves 2f2bcb006d Test stream detaching in CsvItemExporter 2023-02-02 05:55:59 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Mikhail Korobov e248360e6e remove compatibility code from tests for the case dataclasses module is not available 
It was Python 3.6 compat code, and Python 3.6 support is dropped.
 2022-07-18 23:49:08 +05:00
Adrián Chaves 3e994bda45 Update for Python 3.7+ 2022-06-16 20:53:14 +02:00
Adrián Chaves e8503217fb Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2022-06-16 20:19:52 +02:00
Andrey Rahmatullin 8284de5e76
Fix/silence the Pylint messages added in 2.10 (#5235) 2021-08-24 12:15:29 +02:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
maranqz a2d6fa5adc Add errors parameter for CsvItemExporter with tests 2020-08-25 13:34:43 +03:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Eugenio Lacuesta 5265853937
Use ItemAdapter.field_names when writing header in CsvItemExporter (#4668) 2020-07-28 11:15:14 +02:00
Adrián Chaves f9c63384c0
Merge branch 'master' into allow-customizing-export-column-names 2020-06-18 14:09:20 +02:00
Eugenio Lacuesta 3eeecb4273
Merge branch 'master' into flake8-max-line-length 2020-05-15 11:11:20 -03:00
Eugenio Lacuesta cc23d1cb58
Flake8: Remove E124 2020-05-06 15:40:37 -03:00
Adrián Chaves a859ea0c8b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-05-06 19:48:32 +02:00
Eugenio Lacuesta 7926da4bd0
Merge branch 'master' into flake8-max-line-length 2020-04-16 16:05:09 -03:00
Eugenio Lacuesta 4270e0a0da
Fix E731: do not assign a lambda expression 2020-04-04 21:54:55 -03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
Marc Hernández 91bbc70bc1
fix E30X flake8 (#4355) 2020-02-21 06:05:31 +01:00
Adrián Chaves ab13221b0b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2019-12-17 14:06:14 +01:00
Eugenio Lacuesta d72444b9c8
Remove more six imports 2019-11-03 01:11:28 -03:00
Eugenio Lacuesta 5d8abdde59
Remove six.text_type from tests 2019-11-03 01:01:23 -03:00
Eugenio Lacuesta 5ab0f189ce
Remove six.moves occurrences from tests 2019-11-03 00:01:09 -03:00
Eugenio Lacuesta c0bfaef37a
Remove __future__ imports from tests 2019-11-02 23:27:04 -03:00
Eugenio Lacuesta 09e27d2d2e
[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s) 2019-07-13 19:26:59 -03:00
Adrián Chaves 484bd0d22a Allow customizing export column names 2019-03-29 15:16:54 +01:00
stummjr bb2cf7c0d7 Fixed bug on XMLItemExporter with non-string fields in items 2016-01-30 10:00:06 -02:00
stummjr cf2ebb0687 Include tests for exporters: JSON, JSON-Lines, Pickle and Marshal. 2016-01-28 19:43:53 -02:00
stummjr c55ff110a3 Fix CSV exporter for non string Python types. 2016-01-27 15:43:17 -02:00
stummjr f1d971a5c0 fix PythonItemExporter for non-string types 2016-01-27 14:34:46 -02:00
Daniel Graña d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00