Adds state tracking to CsvItemExporter to detect when auto-detected fields drop data on items. Injects a warning instructing users to configure FEED_EXPORT_FIELDS. Includes unit tests. Corrects the false positive logic in stalled PR #7613.
* Set TELNETCONSOLE_ENABLED=False in get_crawler().
* Make _get_console_and_portal() a context manager.
* Allow PYTHONTRACEMALLOC in tox.
* Close the event loop in test_custom_asyncio_loop_enabled_false().
* Close the handler in _uninstall_scrapy_root_handler().
* Close queues in test_squeues.py.
* Close empty queues in init_prios().
* Close PopenSpawn stdin and stdout explicitly.
* Silence the unclosed socket warning.
* Implement more methods in CustomStatsCollector.
* Link the Twisted issue.
* Close the connection on download_maxsize.
* Fix typing.
* Add a comment about CustomStatsCollector.
* Restore the coverage.
* Properly close fixture queues.
* More robust file closing in feed storages.
* Close the file in TestMarshalItemExporter.test_nonstring_types_item().
* Cleanup temporary file handling in test_exporters.py.
* Close the file in test_stats_file_failed().
* Use an explicit TextIOWrapper in XmlItemExporter.
* Switch to pytest.raises().
* Add matches= to broad pytest.raises().
* Adjust the test_nonserializable_object() regex for Python <= 3.11.
* Adjust the test_nonserializable_object() regex for PyPy.
* Adjust other test exception regexes for PyPy.
* Cleanup.
* Partial fix for #3090 - only addresses JSON feeds.
* Adding test case for #3090 to Json Exporter
* Changing the deliberate-fail JSON example to a complex
* Further tightening JsonItemExporter behaviour to prevent corruption.
Based on Mikhail's observation that to_bytes can fail also, leading
to the same dangling comma as the failure to encode to JSON.
Added a new test case to avoid reversion.
* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails
- Add initial changes from cathal's PR
- https://github.com/scrapy/scrapy/issues/3090
* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails
- Handle exception not to add empty item.
- https://github.com/scrapy/scrapy/issues/3090
* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails
- Add comment for handling the exception
- https://github.com/scrapy/scrapy/issues/3090
* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails
- Remove unused import
- https://github.com/scrapy/scrapy/issues/3090
* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails
- Fix invalid json issue
- https://github.com/scrapy/scrapy/issues/3090
* [scrapy] JsonItemExporter puts lone comma in the output if encoder fails
- Perform CR changes
- https://github.com/scrapy/scrapy/issues/3090
---------
Co-authored-by: Cathal Garvey <cathalgarvey@cathalgarvey.me>
* put flake8 options into separate file to remove pytest warnings
* remove ResourceLeaked warning in pypy
* suppress warnings from twisted
* ignore deprecation warnings here
* ignore deprecation warning in tests of deprecated methods
* ignore deprecation warnings here
* update test classes
* don`t use deprecated method call
* ignore deprecation warnings here
* proper warning class
* more selective ignoring
* Revert "don`t use deprecated method call"
This reverts commit 59216ab560.
Shaquon Kelley
Andrey Rakhmatullin
Laerte Pereira
Laerte Pereira
Adnan Awan
Adrián Chaves
Jalil SA
Emmanuel Rondan
Mark Mayo
Mikhail Korobov
Vostretsov Nikita
maranqz
Marc Hernández
Eugenio Lacuesta
Eugenio Lacuesta