* Fix request_to_curl() corrupting dict cookies with bytes keys/values
PR #7603 made the list-cookie branch of request_to_curl() bytes-safe via
_cookie_value_to_unicode(), but left the sibling dict-cookie branch using
raw f-string interpolation. A dict cookie with bytes keys/values (a
supported and common form, e.g. Request(url, cookies={b"k": b"v"})) was
rendered as --cookie 'b'k'=b'v'' instead of --cookie 'k=v', producing a
broken curl command.
Route the dict branch through the same _cookie_value_to_unicode() helper,
mirroring the list branch. Add a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: apply _cookie_value_to_unicode to list-branch cookie key/value
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Set TELNETCONSOLE_ENABLED=False in get_crawler().
* Make _get_console_and_portal() a context manager.
* Allow PYTHONTRACEMALLOC in tox.
* Close the event loop in test_custom_asyncio_loop_enabled_false().
* Close the handler in _uninstall_scrapy_root_handler().
* Close queues in test_squeues.py.
* Close empty queues in init_prios().
* Close PopenSpawn stdin and stdout explicitly.
* Silence the unclosed socket warning.
* Implement more methods in CustomStatsCollector.
* Link the Twisted issue.
* Close the connection on download_maxsize.
* Fix typing.
* Add a comment about CustomStatsCollector.
* Restore the coverage.
* Properly close fixture queues.
* More robust file closing in feed storages.
* Close the file in TestMarshalItemExporter.test_nonstring_types_item().
* Cleanup temporary file handling in test_exporters.py.
* Close the file in test_stats_file_failed().
* Use an explicit TextIOWrapper in XmlItemExporter.
The HTML specification states that link type keywords like 'nofollow'
are ASCII case-insensitive. Sites using rel="NoFollow" or rel="NOFOLLOW"
were incorrectly treated as follow links, causing Scrapy to crawl pages
it should skip.
Fix: add .lower() before the token split so all casing variants of
'nofollow' are correctly recognized.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.
Closes#6120
Javier
Adrian
greymoth
Andrey Rakhmatullin
Sriniketh24
smellslikeml
tanishqtayade
Shashank S. Khasare
Vishal Gawade
Lions-1
JSap0914
Kushal Gupta
Syncrain
Nishita Matlani
Labib Bin Salam
msn