Adds state tracking to CsvItemExporter to detect when auto-detected fields drop data on items. Injects a warning instructing users to configure FEED_EXPORT_FIELDS. Includes unit tests. Corrects the false positive logic in stalled PR #7613.
* Cover 2.17.0 in the release notes up to dd10cb8.
* Fix a copypasted issue number.
* Address minor issues
* Minor fixes
* Cover additional deprecations introduced while improving test coverage
* versionadded/versionchanged improvements
* Add a versionadded to the give_up_log_level docstring
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Fix _get_tag_name() crash for non-string elem.tag (#7686)
* test: improve non-string tag test accuracy and add direct unit test
* test: use minimal payload for non-string tag test
* test: address Adrian+syncrain PR feedback
- remove first docstring line (Adrian: unnecessary)
- replace weak isinstance assert with no-op call
- keep Cython function mention (Adrian: wording is great)
* test: replace silent call with assert results == [] per Adrian
* chore: drop accidental pyproject.toml and uv.lock changes
* Adjust CookiesT.
* Drop list of plain cookie dicts support from request_to_curl().
* Extract _decode_cookie().
* Unify logging.
* Extract _to_verbose_cookies().
* Sync and type hint _cookie_to_set_cookie_value() in tests.
* Add tests for bytes in Request.cookies.
* Type hint assertCookieValEqual().
* Fix cell-var-from-loop in _send_catch_log_deferred
Replace lambda capturing receiver by reference with a default
argument to capture it by value, fixing a potential bug where
all deferred callbacks could reference the last receiver in the
loop instead of their respective receivers.
Removes the pylint disable comment and TODO that were suppressing
this issue.
* chore: trigger CI rerun for mypy network error
* Fix mypy error: pass receiver via addBoth args instead of lambda default
* style: apply pre-commit ruff formatting
* fix: LocalCache with limit=0 incorrectly stores items
When limit=0 is passed to LocalCache (e.g. when DNSCACHE_ENABLED=False),
the condition 'if self.limit' evaluates to False due to Python's truthiness
rules, causing items to be stored despite the cache being disabled.
This leads to an unbounded memory leak during long crawls when DNS caching
is explicitly disabled.
Fix changes the condition to 'if self.limit is not None' and adds an early
return when limit=0 to correctly handle the disabled cache case.
* test: add resolver-level regression tests for DNSCACHE_ENABLED=False
Add two regression tests that verify DNS results are not stored in
dnscache when DNSCACHE_ENABLED=False (cache_size=0):
- test_caching_hostname_resolver_dnscache_disabled_rejects_storage:
verifies _CachingResolutionReceiver does not write to dnscache
when CachingHostnameResolver is initialized with cache_size=0
- test_caching_threaded_resolver_dnscache_disabled_rejects_storage:
verifies dnscache rejects storage at the LocalCache level
when limit=0
* test: drop misleading threaded resolver test
it was writing directly to dnscache, not actually going through
CachingThreadedResolver at all. the threaded resolver already has
its own if dnscache.limit: guard so the bug doesnt affect it anyway.
keeping only the hostname resolver test which covers the actual
bug path through _CachingResolutionReceiver
* test: clean up comments in resolver test
* test: remove unnecessary comment
* Fix request_to_curl() corrupting dict cookies with bytes keys/values
PR #7603 made the list-cookie branch of request_to_curl() bytes-safe via
_cookie_value_to_unicode(), but left the sibling dict-cookie branch using
raw f-string interpolation. A dict cookie with bytes keys/values (a
supported and common form, e.g. Request(url, cookies={b"k": b"v"})) was
rendered as --cookie 'b'k'=b'v'' instead of --cookie 'k=v', producing a
broken curl command.
Route the dict branch through the same _cookie_value_to_unicode() helper,
mirroring the list branch. Add a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: apply _cookie_value_to_unicode to list-branch cookie key/value
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Set TELNETCONSOLE_ENABLED=False in get_crawler().
* Make _get_console_and_portal() a context manager.
* Allow PYTHONTRACEMALLOC in tox.
* Close the event loop in test_custom_asyncio_loop_enabled_false().
* Close the handler in _uninstall_scrapy_root_handler().
* Close queues in test_squeues.py.
* Close empty queues in init_prios().
* Close PopenSpawn stdin and stdout explicitly.
* Silence the unclosed socket warning.
* Implement more methods in CustomStatsCollector.
* Link the Twisted issue.
* Close the connection on download_maxsize.
* Fix typing.
* Add a comment about CustomStatsCollector.
* Restore the coverage.
* Properly close fixture queues.
* More robust file closing in feed storages.
* Close the file in TestMarshalItemExporter.test_nonstring_types_item().
* Cleanup temporary file handling in test_exporters.py.
* Close the file in test_stats_file_failed().
* Use an explicit TextIOWrapper in XmlItemExporter.
Andrey Rakhmatullin
Adrian
Shaquon Kelley
Fat-Coder-CN
Gaurav Yadav
tanishqtayade
Javier
greymoth
Sriniketh24
smellslikeml
Shashank S. Khasare
Vishal Gawade
Lions-1