Merge branch 'master' into feat/FixFeedExport

2023-06-13 18:21:06 +04:00 · 2023-06-13 18:21:06 +04:00 · 8a64f3e8de
parent 1e20ba0a1b 9e0bfc4a3d
commit 8a64f3e8de
45 changed files with 1085 additions and 237 deletions
--- a/.bandit.yml
+++ b/.bandit.yml
@ -1,5 +1,6 @@
 skips:
 - B101
+- B113  # https://github.com/PyCQA/bandit/issues/1010
 - B105
 - B301
 - B303
--- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.8.0
+current_version = 2.9.0
 commit = True
 tag = True
 tag_name = {new_version}
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@ -1,6 +1,6 @@
 repos:
 - repo: https://github.com/PyCQA/bandit
-  rev: 1.7.4
+  rev: 1.7.5
  hooks:
  - id: bandit
    args: [-r, -c, .bandit.yml]
@ -9,7 +9,7 @@ repos:
  hooks:
  - id: flake8
 - repo: https://github.com/psf/black.git
-  rev: 23.1.0
+  rev: 23.3.0
  hooks:
  - id: black
 - repo: https://github.com/pycqa/isort
@ -21,4 +21,4 @@ repos:
  hooks:
  - id: blacken-docs
    additional_dependencies:
-    - black==23.1.0
+    - black==23.3.0
--- a/docs/faq.rst
+++ b/docs/faq.rst
@ -231,8 +231,8 @@ Can I return (Twisted) deferreds from signal handlers?
 Some signals support returning deferreds from their handlers, others don't. See
 the :ref:`topics-signals-ref` to know which ones.

-What does the response status code 999 means?
---------------------------------------------
+What does the response status code 999 mean?
+--------------------------------------------

 999 is a custom response status code used by Yahoo sites to throttle requests.
 Try slowing down the crawling speed by using a download delay of ``2`` (or
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@ -252,7 +252,7 @@ object:
 .. code-block:: pycon

    >>> response.css("title")
-    [<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
+    [<Selector query='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]

 The result of running ``response.css('title')`` is a list-like object called
 :class:`~scrapy.selector.SelectorList`, which represents a list of
@ -329,7 +329,7 @@ the :meth:`~scrapy.selector.SelectorList.re` method to extract using
    >>> response.css("title::text").re(r"(\w+) to (\w+)")
    ['Quotes', 'Scrape']

-In order to find the proper CSS selectors to use, you might find useful opening
+In order to find the proper CSS selectors to use, you might find it useful to open
 the response page from the shell in your web browser using ``view(response)``.
 You can use your browser's developer tools to inspect the HTML and come up
 with a selector (see :ref:`topics-developer-tools`).
@ -348,7 +348,7 @@ Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions:
 .. code-block:: pycon

    >>> response.xpath("//title")
-    [<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
+    [<Selector query='//title' data='<title>Quotes to Scrape</title>'>]
    >>> response.xpath("//title/text()").get()
    'Quotes to Scrape'

@ -410,8 +410,8 @@ We get a list of selectors for the quote HTML elements with:
 .. code-block:: pycon

    >>> response.css("div.quote")
-    [<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
-    <Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
+    [<Selector query="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
+    <Selector query="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
    ...]

 Each of the selectors returned by the query above allows us to run further
--- a/docs/news.rst
+++ b/docs/news.rst
@ -3,6 +3,128 @@
 Release notes
 =============

+.. _release-2.9.0:
+
+Scrapy 2.9.0 (2023-05-08)
+-------------------------
+
+Highlights:
+
+-   Per-domain download settings.
+-   Compatibility with new cryptography_ and new parsel_.
+-   JMESPath selectors from the new parsel_.
+-   Bug fixes.
+
+Deprecations
+~~~~~~~~~~~~
+
+-   :class:`scrapy.extensions.feedexport._FeedSlot` is renamed to
+    :class:`scrapy.extensions.feedexport.FeedSlot` and the old name is
+    deprecated. (:issue:`5876`)
+
+New features
+~~~~~~~~~~~~
+
+-   Settings correponding to :setting:`DOWNLOAD_DELAY`,
+    :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and
+    :setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per-domain basis
+    via the new :setting:`DOWNLOAD_SLOTS` setting. (:issue:`5328`)
+
+-   Added :meth:`TextResponse.jmespath`, a shortcut for JMESPath selectors
+    available since parsel_ 1.8.1. (:issue:`5894`, :issue:`5915`)
+
+-   Added :signal:`feed_slot_closed` and :signal:`feed_exporter_closed`
+    signals. (:issue:`5876`)
+
+-   Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a
+    curl command from a :class:`~scrapy.Request` object. (:issue:`5892`)
+
+-   Values of :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can now be
+    :class:`pathlib.Path` instances. (:issue:`5801`)
+
+Bug fixes
+~~~~~~~~~
+
+-   Fixed a warning with Parsel 1.8.1+. (:issue:`5903`, :issue:`5918`)
+
+-   Fixed an error when using feed postprocessing with S3 storage.
+    (:issue:`5500`, :issue:`5581`)
+
+-   Added the missing :meth:`scrapy.settings.BaseSettings.setdefault` method.
+    (:issue:`5811`, :issue:`5821`)
+
+-   Fixed an error when using cryptography_ 40.0.0+ and
+    :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` is enabled.
+    (:issue:`5857`, :issue:`5858`)
+
+-   The checksums returned by :class:`~scrapy.pipelines.files.FilesPipeline`
+    for files on Google Cloud Storage are no longer Base64-encoded.
+    (:issue:`5874`, :issue:`5891`)
+
+-   :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed
+    string values for the curl ``--data-raw`` argument, which are produced by
+    browsers for data that includes certain symbols. (:issue:`5899`,
+    :issue:`5901`)
+
+-   The :command:`parse` command now also works with async generator callbacks.
+    (:issue:`5819`, :issue:`5824`)
+
+-   The :command:`genspider` command now properly works with HTTPS URLs.
+    (:issue:`3553`, :issue:`5808`)
+
+-   Improved handling of asyncio loops. (:issue:`5831`, :issue:`5832`)
+
+-   :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+    now skips certain malformed URLs instead of raising an exception.
+    (:issue:`5881`)
+
+-   :func:`scrapy.utils.python.get_func_args` now supports more types of
+    callables. (:issue:`5872`, :issue:`5885`)
+
+-   Fixed an error when processing non-UTF8 values of ``Content-Type`` headers.
+    (:issue:`5914`, :issue:`5917`)
+
+-   Fixed an error breaking user handling of send failures in
+    :meth:`scrapy.mail.MailSender.send()`. (:issue:`1611`, :issue:`5880`)
+
+Documentation
+~~~~~~~~~~~~~
+
+-   Expanded contributing docs. (:issue:`5109`, :issue:`5851`)
+
+-   Added blacken-docs_ to pre-commit and reformatted the docs with it.
+    (:issue:`5813`, :issue:`5816`)
+
+-   Fixed a JS issue. (:issue:`5875`, :issue:`5877`)
+
+-   Fixed ``make htmlview``. (:issue:`5878`, :issue:`5879`)
+
+-   Fixed typos and other small errors. (:issue:`5827`, :issue:`5839`,
+    :issue:`5883`, :issue:`5890`, :issue:`5895`, :issue:`5904`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+-   Extended typing hints. (:issue:`5805`, :issue:`5889`, :issue:`5896`)
+
+-   Tests for most of the examples in the docs are now run as a part of CI,
+    found problems were fixed. (:issue:`5816`, :issue:`5826`, :issue:`5919`)
+
+-   Removed usage of deprecated Python classes. (:issue:`5849`)
+
+-   Silenced ``include-ignored`` warnings from coverage. (:issue:`5820`)
+
+-   Fixed a random failure of the ``test_feedexport.test_batch_path_differ``
+    test. (:issue:`5855`, :issue:`5898`)
+
+-   Updated docstrings to match output produced by parsel_ 1.8.1 so that they
+    don't cause test failures. (:issue:`5902`, :issue:`5919`)
+
+-   Other CI and pre-commit improvements. (:issue:`5802`, :issue:`5823`,
+    :issue:`5908`)
+
+.. _blacken-docs: https://github.com/adamchainz/blacken-docs
+
 .. _release-2.8.0:

 Scrapy 2.8.0 (2023-02-02)
@ -4207,8 +4329,6 @@ Relocations
  + Note: telnet is not enabled on Python 3
    (https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595)

-.. _parsel: https://github.com/scrapy/parsel
-

 Bugfixes
 ~~~~~~~~
@ -5638,6 +5758,7 @@ First release of Scrapy.
 .. _LevelDB: https://github.com/google/leveldb
 .. _lxml: https://lxml.de/
 .. _marshal: https://docs.python.org/2/library/marshal.html
+.. _parsel: https://github.com/scrapy/parsel
 .. _parsel.csstranslator.GenericTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.GenericTranslator
 .. _parsel.csstranslator.HTMLTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.HTMLTranslator
 .. _parsel.csstranslator.XPathExpr: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.XPathExpr
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@ -134,6 +134,63 @@ Common use cases for asynchronous code include:
 .. _aio-libs: https://github.com/aio-libs


+.. _inline-requests:
+
+Inline requests
+===============
+
+The spider below shows how to send a request and await its response all from
+within a spider callback:
+
+.. code-block:: python
+
+    from scrapy import Spider, Request
+    from scrapy.utils.defer import maybe_deferred_to_future
+
+
+    class SingleRequestSpider(Spider):
+        name = "single"
+        start_urls = ["https://example.org/product"]
+
+        async def parse(self, response, **kwargs):
+            additional_request = Request("https://example.org/price")
+            deferred = self.crawler.engine.download(additional_request)
+            additional_response = await maybe_deferred_to_future(deferred)
+            yield {
+                "h1": response.css("h1").get(),
+                "price": additional_response.css("#price").get(),
+            }
+
+You can also send multiple requests in parallel:
+
+.. code-block:: python
+
+    from scrapy import Spider, Request
+    from scrapy.utils.defer import maybe_deferred_to_future
+    from twisted.internet.defer import DeferredList
+
+
+    class MultipleRequestsSpider(Spider):
+        name = "multiple"
+        start_urls = ["https://example.com/product"]
+
+        async def parse(self, response, **kwargs):
+            additional_requests = [
+                Request("https://example.com/price"),
+                Request("https://example.com/color"),
+            ]
+            deferreds = []
+            for r in additional_requests:
+                deferred = self.crawler.engine.download(r)
+                deferreds.append(deferred)
+            responses = await maybe_deferred_to_future(DeferredList(deferreds))
+            yield {
+                "h1": response.css("h1::text").get(),
+                "price": responses[0][1].css(".price::text").get(),
+                "price2": responses[1][1].css(".color::text").get(),
+            }
+
+
 .. _sync-async-spider-middleware:

 Mixing synchronous and asynchronous spider middlewares
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@ -175,6 +175,12 @@ FTP supports two different connection modes: `active or passive
 mode by default. To use the active connection mode instead, set the
 :setting:`FEED_STORAGE_FTP_ACTIVE` setting to ``True``.

+The default value for the ``overwrite`` key in the :setting:`FEEDS` for this 
+storage backend is: ``True``.
+
+.. caution:: The value ``True`` in ``overwrite`` will cause you to lose the
+     previous version of your data.
+
 This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.


@ -209,6 +215,12 @@ You can also define a custom ACL and custom endpoint for exported feeds using th
 -   :setting:`FEED_STORAGE_S3_ACL`
 -   :setting:`AWS_ENDPOINT_URL`

+The default value for the ``overwrite`` key in the :setting:`FEEDS` for this 
+storage backend is: ``True``.
+
+.. caution:: The value ``True`` in ``overwrite`` will cause you to lose the
+     previous version of your data.
+
 This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.


@ -236,6 +248,12 @@ You can set a *Project ID* and *Access Control List (ACL)* through the following
 -   :setting:`FEED_STORAGE_GCS_ACL`
 -   :setting:`GCS_PROJECT_ID`

+The default value for the ``overwrite`` key in the :setting:`FEEDS` for this 
+storage backend is: ``True``.
+
+.. caution:: The value ``True`` in ``overwrite`` will cause you to lose the
+     previous version of your data.
+
 This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.

 .. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
@ -488,6 +506,8 @@ as a fallback value if that key is not provided for a specific feed definition:
    -   :ref:`topics-feed-storage-s3`: ``True`` (appending `is not supported
        <https://forums.aws.amazon.com/message.jspa?messageID=540395>`_)

+    -   :ref:`topics-feed-storage-gcs`: ``True`` (appending is not supported)
+
    -   :ref:`topics-feed-storage-stdout`: ``False`` (overwriting is not supported)

    .. versionadded:: 2.4.0
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@ -1281,6 +1281,12 @@ TextResponse objects
    :class:`TextResponse` objects support the following methods in addition to
    the standard :class:`Response` ones:

+    .. method:: TextResponse.jmespath(query)
+
+        A shortcut to ``TextResponse.selector.jmespath(query)``::
+
+            response.jmespath('object.[*]')
+
    .. method:: TextResponse.xpath(query)

        A shortcut to ``TextResponse.selector.xpath(query)``::
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@ -48,6 +48,8 @@ Constructing selectors

 .. highlight:: python

+.. skip: start
+
 Response objects expose a :class:`~scrapy.Selector` instance
 on ``.selector`` attribute:

@ -66,6 +68,8 @@ more shortcuts: ``response.xpath()`` and ``response.css()``:
    >>> response.css("span::text").get()
    'good'

+.. skip: end
+
 Scrapy selectors are instances of :class:`~scrapy.Selector` class
 constructed by passing either :class:`~scrapy.http.TextResponse` object or
 markup as a string (in ``text`` argument).
@ -93,7 +97,7 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of

    >>> from scrapy.selector import Selector
    >>> from scrapy.http import HtmlResponse
-    >>> response = HtmlResponse(url="http://example.com", body=body)
+    >>> response = HtmlResponse(url="http://example.com", body=body, encoding="utf-8")
    >>> Selector(response=response).xpath("//span/text()").get()
    'good'

@ -103,6 +107,13 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
 Using selectors
 ---------------

+.. invisible-code-block: python
+
+    html_response = response = load_response(
+        "https://docs.scrapy.org/en/latest/_static/selectors-sample1.html",
+        "../_static/selectors-sample1.html",
+    )
+
 To explain how to use the selectors we'll use the ``Scrapy shell`` (which
 provides interactive testing) and an example page located in the Scrapy
 documentation server:
@ -135,7 +146,7 @@ page, let's construct an XPath for selecting the text inside the title tag:
 .. code-block:: pycon

    >>> response.xpath("//title/text()")
-    [<Selector xpath='//title/text()' data='Example website'>]
+    [<Selector query='//title/text()' data='Example website'>]

 To actually extract the textual data, you must call the selector ``.get()``
 or ``.getall()`` methods, as follows:
@ -363,11 +374,11 @@ too. Here's an example:

    >>> links = response.xpath('//a[contains(@href, "image")]')
    >>> links.getall()
-    ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
-    '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
-    '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
-    '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
-    '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
+    ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg" alt="image1"></a>',
+    '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg" alt="image2"></a>',
+    '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg" alt="image3"></a>',
+    '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg" alt="image4"></a>',
+    '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg" alt="image5"></a>']

    >>> for index, link in enumerate(links):
    ...     href_xpath = link.xpath("@href").get()
@ -447,11 +458,11 @@ Here's an example used to extract image names from the :ref:`HTML code
 .. code-block:: pycon

    >>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)")
-    ['My image 1',
-    'My image 2',
-    'My image 3',
-    'My image 4',
-    'My image 5']
+    ['My image 1 ',
+    'My image 2 ',
+    'My image 3 ',
+    'My image 4 ',
+    'My image 5 ']

 There's an additional helper reciprocating ``.get()`` (and its
 alias ``.extract_first()``) for ``.re()``, named ``.re_first()``.
@ -460,7 +471,7 @@ Use it to extract just the first matching string:
 .. code-block:: pycon

    >>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)")
-    'My image 1'
+    'My image 1 '

 .. _old-extraction-api:

@ -761,6 +772,8 @@ on `XPath variables`_.
 Removing namespaces
 -------------------

+.. skip: start
+
 When dealing with scraping projects, it is often quite convenient to get rid of
 namespaces altogether and just work with element names, to write more
 simple/convenient XPaths. You can use the
@ -808,8 +821,8 @@ nodes can be accessed directly by their names:

    >>> response.selector.remove_namespaces()
    >>> response.xpath("//link")
-    [<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
-        <Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
+    [<Selector query='//link' data='<link rel="alternate" type="text/html" h'>,
+        <Selector query='//link' data='<link rel="next" type="application/atom+'>,
        ...

 If you wonder why the namespace removal procedure isn't always called by default
@ -824,6 +837,7 @@ of relevance, are:
   case some element names clash between namespaces. These cases are very rare
   though.

+.. skip: end

 Using EXSLT extensions
 ----------------------
@ -881,6 +895,8 @@ extracting text elements for example.
 Example extracting microdata (sample content taken from https://schema.org/Product)
 with groups of itemscopes and corresponding itemprops:

+.. skip: next
+
 .. code-block:: pycon

    >>> doc = """
@ -977,26 +993,35 @@ Scrapy selectors also provide a sorely missed XPath extension function
 ``has-class`` that returns ``True`` for nodes that have all of the specified
 HTML classes.

-.. highlight:: html
+For the following HTML:

-For the following HTML::
+.. code-block:: pycon

-    <p class="foo bar-baz">First</p>
-    <p class="foo">Second</p>
-    <p class="bar">Third</p>
-    <p>Fourth</p>
-
-.. highlight:: python
+    >>> from scrapy.http import HtmlResponse
+    >>> response = HtmlResponse(
+    ...     url="http://example.com",
+    ...     body="""
+    ... <html>
+    ...     <body>
+    ...         <p class="foo bar-baz">First</p>
+    ...         <p class="foo">Second</p>
+    ...         <p class="bar">Third</p>
+    ...         <p>Fourth</p>
+    ...     </body>
+    ... </html>
+    ... """,
+    ...     encoding="utf-8",
+    ... )

 You can use it like this:

 .. code-block:: pycon

    >>> response.xpath('//p[has-class("foo")]')
-    [<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
-    <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
+    [<Selector query='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
+    <Selector query='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
    >>> response.xpath('//p[has-class("foo", "bar-baz")]')
-    [<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
+    [<Selector query='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
    >>> response.xpath('//p[has-class("foo", "bar")]')
    []

@ -1132,6 +1157,8 @@ a :class:`~scrapy.http.HtmlResponse` object like this:
 Selector examples on XML response
 ---------------------------------

+.. skip: start
+
 Here are some examples to illustrate concepts for :class:`Selector` objects
 instantiated with an :class:`~scrapy.http.XmlResponse` object:

@ -1154,4 +1181,6 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object:
      sel.register_namespace("g", "http://base.google.com/ns/1.0")
      sel.xpath("//g:price").getall()

+.. skip: end
+
 .. _Google Base XML feed: https://support.google.com/merchants/answer/160589?hl=en&ref_topic=2473799
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@ -783,7 +783,7 @@ DOWNLOAD_SLOTS

 Default: ``{}``

-Allows to define concurrency/delay parameters on per slot(domain) basis:
+Allows to define concurrency/delay parameters on per slot (domain) basis:

    .. code-block:: python

--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@ -307,6 +307,33 @@ spider_error
    :param spider: the spider which raised the exception
    :type spider: :class:`~scrapy.Spider` object

+feed_slot_closed
+~~~~~~~~~~~~~~~~
+
+.. signal:: feed_slot_closed
+.. function:: feed_slot_closed(slot)
+
+    Sent when a :ref:`feed exports <topics-feed-exports>` slot is closed.
+
+    This signal supports returning deferreds from its handlers.
+
+    :param slot: the slot closed
+    :type slot: scrapy.extensions.feedexport.FeedSlot
+
+
+feed_exporter_closed
+~~~~~~~~~~~~~~~~~~~~
+
+.. signal:: feed_exporter_closed
+.. function:: feed_exporter_closed()
+
+    Sent when the :ref:`feed exports <topics-feed-exports>` extension is closed,
+    during the handling of the :signal:`spider_closed` signal by the extension,
+    after all feed exporting has been handled.
+
+    This signal supports returning deferreds from its handlers.
+
+
 Request signals
 ---------------

--- a/pytest.ini
+++ b/pytest.ini
@ -5,7 +5,6 @@ python_files=test_*.py __init__.py
 python_classes=
 addopts =
    --assert=plain
-    --doctest-modules
    --ignore=docs/_ext
    --ignore=docs/conf.py
    --ignore=docs/news.rst
--- a/scrapy/VERSION
+++ b/scrapy/VERSION
@ -1 +1 @@
-2.8.0
+2.9.0
--- a/scrapy/core/downloader/init.py
+++ b/scrapy/core/downloader/init.py
@ -2,45 +2,52 @@ import random
 from collections import deque
 from datetime import datetime
 from time import time
+from typing import TYPE_CHECKING, Any, Deque, Dict, Set, Tuple, cast

-from twisted.internet import defer, task
+from twisted.internet import task
 from twisted.internet.defer import Deferred

 from scrapy import Request, Spider, signals
 from scrapy.core.downloader.handlers import DownloadHandlers
 from scrapy.core.downloader.middleware import DownloaderMiddlewareManager
+from scrapy.http import Response
 from scrapy.resolver import dnscache
+from scrapy.settings import BaseSettings
+from scrapy.signalmanager import SignalManager
 from scrapy.utils.defer import mustbe_deferred
 from scrapy.utils.httpobj import urlparse_cached

+if TYPE_CHECKING:
+    from scrapy.crawler import Crawler
+

 class Slot:
    """Downloader slot"""

-    def __init__(self, concurrency, delay, randomize_delay):
-        self.concurrency = concurrency
-        self.delay = delay
-        self.randomize_delay = randomize_delay
+    def __init__(self, concurrency: int, delay: float, randomize_delay: bool):
+        self.concurrency: int = concurrency
+        self.delay: float = delay
+        self.randomize_delay: bool = randomize_delay

-        self.active = set()
-        self.queue = deque()
-        self.transferring = set()
-        self.lastseen = 0
+        self.active: Set[Request] = set()
+        self.queue: Deque[Tuple[Request, Deferred]] = deque()
+        self.transferring: Set[Request] = set()
+        self.lastseen: float = 0
        self.latercall = None

-    def free_transfer_slots(self):
+    def free_transfer_slots(self) -> int:
        return self.concurrency - len(self.transferring)

-    def download_delay(self):
+    def download_delay(self) -> float:
        if self.randomize_delay:
            return random.uniform(0.5 * self.delay, 1.5 * self.delay)
        return self.delay

-    def close(self):
+    def close(self) -> None:
        if self.latercall and self.latercall.active():
            self.latercall.cancel()

-    def __repr__(self):
+    def __repr__(self) -> str:
        cls_name = self.__class__.__name__
        return (
            f"{cls_name}(concurrency={self.concurrency!r}, "
@ -48,7 +55,7 @@ class Slot:
            f"randomize_delay={self.randomize_delay!r})"
        )

-    def __str__(self):
+    def __str__(self) -> str:
        return (
            f"<downloader.Slot concurrency={self.concurrency!r} "
            f"delay={self.delay:.2f} randomize_delay={self.randomize_delay!r} "
@ -58,8 +65,10 @@ class Slot:
        )


-def _get_concurrency_delay(concurrency, spider, settings):
-    delay = settings.getfloat("DOWNLOAD_DELAY")
+def _get_concurrency_delay(
+    concurrency: int, spider: Spider, settings: BaseSettings
+) -> Tuple[int, float]:
+    delay: float = settings.getfloat("DOWNLOAD_DELAY")
    if hasattr(spider, "download_delay"):
        delay = spider.download_delay

@ -72,23 +81,29 @@ def _get_concurrency_delay(concurrency, spider, settings):
 class Downloader:
    DOWNLOAD_SLOT = "download_slot"

-    def __init__(self, crawler):
-        self.settings = crawler.settings
-        self.signals = crawler.signals
-        self.slots = {}
-        self.active = set()
-        self.handlers = DownloadHandlers(crawler)
-        self.total_concurrency = self.settings.getint("CONCURRENT_REQUESTS")
-        self.domain_concurrency = self.settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN")
-        self.ip_concurrency = self.settings.getint("CONCURRENT_REQUESTS_PER_IP")
-        self.randomize_delay = self.settings.getbool("RANDOMIZE_DOWNLOAD_DELAY")
-        self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
-        self._slot_gc_loop = task.LoopingCall(self._slot_gc)
+    def __init__(self, crawler: "Crawler"):
+        self.settings: BaseSettings = crawler.settings
+        self.signals: SignalManager = crawler.signals
+        self.slots: Dict[str, Slot] = {}
+        self.active: Set[Request] = set()
+        self.handlers: DownloadHandlers = DownloadHandlers(crawler)
+        self.total_concurrency: int = self.settings.getint("CONCURRENT_REQUESTS")
+        self.domain_concurrency: int = self.settings.getint(
+            "CONCURRENT_REQUESTS_PER_DOMAIN"
+        )
+        self.ip_concurrency: int = self.settings.getint("CONCURRENT_REQUESTS_PER_IP")
+        self.randomize_delay: bool = self.settings.getbool("RANDOMIZE_DOWNLOAD_DELAY")
+        self.middleware: DownloaderMiddlewareManager = (
+            DownloaderMiddlewareManager.from_crawler(crawler)
+        )
+        self._slot_gc_loop: task.LoopingCall = task.LoopingCall(self._slot_gc)
        self._slot_gc_loop.start(60)
-        self.per_slot_settings = self.settings.getdict("DOWNLOAD_SLOTS", {})
+        self.per_slot_settings: Dict[str, Dict[str, Any]] = self.settings.getdict(
+            "DOWNLOAD_SLOTS", {}
+        )

    def fetch(self, request: Request, spider: Spider) -> Deferred:
-        def _deactivate(response):
+        def _deactivate(response: Response) -> Response:
            self.active.remove(request)
            return response

@ -99,7 +114,7 @@ class Downloader:
    def needs_backout(self) -> bool:
        return len(self.active) >= self.total_concurrency

-    def _get_slot(self, request, spider):
+    def _get_slot(self, request: Request, spider: Spider) -> Tuple[str, Slot]:
        key = self._get_slot_key(request, spider)
        if key not in self.slots:
            slot_settings = self.per_slot_settings.get(key, {})
@ -117,9 +132,9 @@ class Downloader:

        return key, self.slots[key]

-    def _get_slot_key(self, request, spider):
+    def _get_slot_key(self, request: Request, spider: Spider) -> str:
        if self.DOWNLOAD_SLOT in request.meta:
-            return request.meta[self.DOWNLOAD_SLOT]
+            return cast(str, request.meta[self.DOWNLOAD_SLOT])

        key = urlparse_cached(request).hostname or ""
        if self.ip_concurrency:
@ -127,11 +142,11 @@ class Downloader:

        return key

-    def _enqueue_request(self, request, spider):
+    def _enqueue_request(self, request: Request, spider: Spider) -> Deferred:
        key, slot = self._get_slot(request, spider)
        request.meta[self.DOWNLOAD_SLOT] = key

-        def _deactivate(response):
+        def _deactivate(response: Response) -> Response:
            slot.active.remove(request)
            return response

@ -139,12 +154,12 @@ class Downloader:
        self.signals.send_catch_log(
            signal=signals.request_reached_downloader, request=request, spider=spider
        )
-        deferred = defer.Deferred().addBoth(_deactivate)
+        deferred = Deferred().addBoth(_deactivate)
        slot.queue.append((request, deferred))
        self._process_queue(spider, slot)
        return deferred

-    def _process_queue(self, spider, slot):
+    def _process_queue(self, spider: Spider, slot: Slot) -> None:
        from twisted.internet import reactor

        if slot.latercall and slot.latercall.active():
@ -172,7 +187,7 @@ class Downloader:
                self._process_queue(spider, slot)
                break

-    def _download(self, slot, request, spider):
+    def _download(self, slot: Slot, request: Request, spider: Spider) -> Deferred:
        # The order is very important for the following deferreds. Do not change!

        # 1. Create the download deferred
@ -180,7 +195,7 @@ class Downloader:

        # 2. Notify response_downloaded listeners about the recent download
        # before querying queue for next request
-        def _downloaded(response):
+        def _downloaded(response: Response) -> Response:
            self.signals.send_catch_log(
                signal=signals.response_downloaded,
                response=response,
@ -197,7 +212,7 @@ class Downloader:
        # middleware itself)
        slot.transferring.add(request)

-        def finish_transferring(_):
+        def finish_transferring(_: Any) -> Any:
            slot.transferring.remove(request)
            self._process_queue(spider, slot)
            self.signals.send_catch_log(
--- a/scrapy/core/downloader/contextfactory.py
+++ b/scrapy/core/downloader/contextfactory.py
@ -1,4 +1,5 @@
 import warnings
+from typing import TYPE_CHECKING, Any, List, Optional

 from OpenSSL import SSL
 from twisted.internet._sslverify import _setAcceptableProtocols
@ -18,8 +19,12 @@ from scrapy.core.downloader.tls import (
    ScrapyClientTLSOptions,
    openssl_methods,
 )
+from scrapy.settings import BaseSettings
 from scrapy.utils.misc import create_instance, load_object

+if TYPE_CHECKING:
+    from twisted.internet._sslverify import ClientTLSOptions
+

@implementer(IPolicyForHTTPS)
 class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
@ -35,25 +40,34 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):

    def __init__(
        self,
-        method=SSL.SSLv23_METHOD,
-        tls_verbose_logging=False,
-        tls_ciphers=None,
-        *args,
-        **kwargs,
+        method: int = SSL.SSLv23_METHOD,
+        tls_verbose_logging: bool = False,
+        tls_ciphers: Optional[str] = None,
+        *args: Any,
+        **kwargs: Any,
    ):
        super().__init__(*args, **kwargs)
-        self._ssl_method = method
-        self.tls_verbose_logging = tls_verbose_logging
+        self._ssl_method: int = method
+        self.tls_verbose_logging: bool = tls_verbose_logging
+        self.tls_ciphers: AcceptableCiphers
        if tls_ciphers:
            self.tls_ciphers = AcceptableCiphers.fromOpenSSLCipherString(tls_ciphers)
        else:
            self.tls_ciphers = DEFAULT_CIPHERS

    @classmethod
-    def from_settings(cls, settings, method=SSL.SSLv23_METHOD, *args, **kwargs):
-        tls_verbose_logging = settings.getbool("DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING")
-        tls_ciphers = settings["DOWNLOADER_CLIENT_TLS_CIPHERS"]
-        return cls(
+    def from_settings(
+        cls,
+        settings: BaseSettings,
+        method: int = SSL.SSLv23_METHOD,
+        *args: Any,
+        **kwargs: Any,
+    ):
+        tls_verbose_logging: bool = settings.getbool(
+            "DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING"
+        )
+        tls_ciphers: Optional[str] = settings["DOWNLOADER_CLIENT_TLS_CIPHERS"]
+        return cls(  # type: ignore[misc]
            method=method,
            tls_verbose_logging=tls_verbose_logging,
            tls_ciphers=tls_ciphers,
@ -61,7 +75,7 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
            **kwargs,
        )

-    def getCertificateOptions(self):
+    def getCertificateOptions(self) -> CertificateOptions:
        # setting verify=True will require you to provide CAs
        # to verify against; in other words: it's not that simple

@ -82,12 +96,12 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):

    # kept for old-style HTTP/1.0 downloader context twisted calls,
    # e.g. connectSSL()
-    def getContext(self, hostname=None, port=None):
+    def getContext(self, hostname: Any = None, port: Any = None) -> SSL.Context:
        ctx = self.getCertificateOptions().getContext()
        ctx.set_options(0x4)  # OP_LEGACY_SERVER_CONNECT
        return ctx

-    def creatorForNetloc(self, hostname, port):
+    def creatorForNetloc(self, hostname: bytes, port: int) -> "ClientTLSOptions":
        return ScrapyClientTLSOptions(
            hostname.decode("ascii"),
            self.getContext(),
@ -114,7 +128,7 @@ class BrowserLikeContextFactory(ScrapyClientContextFactory):
    ``SSLv23_METHOD``) which allows TLS protocol negotiation.
    """

-    def creatorForNetloc(self, hostname, port):
+    def creatorForNetloc(self, hostname: bytes, port: int) -> "ClientTLSOptions":
        # trustRoot set to platformTrust() will use the platform's root CAs.
        #
        # This means that a website like https://www.cacert.org will be rejected
@ -133,13 +147,15 @@ class AcceptableProtocolsContextFactory:
    negotiation.
    """

-    def __init__(self, context_factory, acceptable_protocols):
+    def __init__(self, context_factory: Any, acceptable_protocols: List[bytes]):
        verifyObject(IPolicyForHTTPS, context_factory)
-        self._wrapped_context_factory = context_factory
-        self._acceptable_protocols = acceptable_protocols
+        self._wrapped_context_factory: Any = context_factory
+        self._acceptable_protocols: List[bytes] = acceptable_protocols

-    def creatorForNetloc(self, hostname, port):
-        options = self._wrapped_context_factory.creatorForNetloc(hostname, port)
+    def creatorForNetloc(self, hostname: bytes, port: int) -> "ClientTLSOptions":
+        options: "ClientTLSOptions" = self._wrapped_context_factory.creatorForNetloc(
+            hostname, port
+        )
        _setAcceptableProtocols(options._ctx, self._acceptable_protocols)
        return options

--- a/scrapy/core/downloader/handlers/init.py
+++ b/scrapy/core/downloader/handlers/init.py
@ -1,25 +1,32 @@
 """Download handlers for different schemes"""

 import logging
+from typing import TYPE_CHECKING, Any, Callable, Dict, Generator, Union, cast

 from twisted.internet import defer
+from twisted.internet.defer import Deferred

-from scrapy import signals
+from scrapy import Request, Spider, signals
 from scrapy.exceptions import NotConfigured, NotSupported
 from scrapy.utils.httpobj import urlparse_cached
 from scrapy.utils.misc import create_instance, load_object
 from scrapy.utils.python import without_none_values

+if TYPE_CHECKING:
+    from scrapy.crawler import Crawler
+
 logger = logging.getLogger(__name__)


 class DownloadHandlers:
-    def __init__(self, crawler):
-        self._crawler = crawler
-        self._schemes = {}  # stores acceptable schemes on instancing
-        self._handlers = {}  # stores instanced handlers for schemes
-        self._notconfigured = {}  # remembers failed handlers
-        handlers = without_none_values(
+    def __init__(self, crawler: "Crawler"):
+        self._crawler: "Crawler" = crawler
+        self._schemes: Dict[
+            str, Union[str, Callable]
+        ] = {}  # stores acceptable schemes on instancing
+        self._handlers: Dict[str, Any] = {}  # stores instanced handlers for schemes
+        self._notconfigured: Dict[str, str] = {}  # remembers failed handlers
+        handlers: Dict[str, Union[str, Callable]] = without_none_values(
            crawler.settings.getwithbase("DOWNLOAD_HANDLERS")
        )
        for scheme, clspath in handlers.items():
@ -28,7 +35,7 @@ class DownloadHandlers:

        crawler.signals.connect(self._close, signals.engine_stopped)

-    def _get_handler(self, scheme):
+    def _get_handler(self, scheme: str) -> Any:
        """Lazy-load the downloadhandler for a scheme
        only on the first request for that scheme.
        """
@ -42,7 +49,7 @@ class DownloadHandlers:

        return self._load_handler(scheme)

-    def _load_handler(self, scheme, skip_lazy=False):
+    def _load_handler(self, scheme: str, skip_lazy: bool = False) -> Any:
        path = self._schemes[scheme]
        try:
            dhcls = load_object(path)
@ -69,17 +76,17 @@ class DownloadHandlers:
            self._handlers[scheme] = dh
            return dh

-    def download_request(self, request, spider):
+    def download_request(self, request: Request, spider: Spider) -> Deferred:
        scheme = urlparse_cached(request).scheme
        handler = self._get_handler(scheme)
        if not handler:
            raise NotSupported(
                f"Unsupported URL scheme '{scheme}': {self._notconfigured[scheme]}"
            )
-        return handler.download_request(request, spider)
+        return cast(Deferred, handler.download_request(request, spider))

    @defer.inlineCallbacks
-    def _close(self, *_a, **_kw):
+    def _close(self, *_a: Any, **_kw: Any) -> Generator[Deferred, Any, None]:
        for dh in self._handlers.values():
            if hasattr(dh, "close"):
                yield dh.close()
--- a/scrapy/core/downloader/middleware.py
+++ b/scrapy/core/downloader/middleware.py
@ -3,15 +3,16 @@ Downloader Middleware manager

 See documentation in docs/topics/downloader-middleware.rst
 """
-from typing import Callable, Union, cast
+from typing import Any, Callable, Generator, List, Union, cast

-from twisted.internet import defer
+from twisted.internet.defer import Deferred, inlineCallbacks
 from twisted.python.failure import Failure

 from scrapy import Spider
 from scrapy.exceptions import _InvalidOutput
 from scrapy.http import Request, Response
 from scrapy.middleware import MiddlewareManager
+from scrapy.settings import BaseSettings
 from scrapy.utils.conf import build_component_list
 from scrapy.utils.defer import deferred_from_coro, mustbe_deferred

@ -20,10 +21,10 @@ class DownloaderMiddlewareManager(MiddlewareManager):
    component_name = "downloader middleware"

    @classmethod
-    def _get_mwlist_from_settings(cls, settings):
+    def _get_mwlist_from_settings(cls, settings: BaseSettings) -> List[Any]:
        return build_component_list(settings.getwithbase("DOWNLOADER_MIDDLEWARES"))

-    def _add_middleware(self, mw):
+    def _add_middleware(self, mw: Any) -> None:
        if hasattr(mw, "process_request"):
            self.methods["process_request"].append(mw.process_request)
        if hasattr(mw, "process_response"):
@ -31,9 +32,11 @@ class DownloaderMiddlewareManager(MiddlewareManager):
        if hasattr(mw, "process_exception"):
            self.methods["process_exception"].appendleft(mw.process_exception)

-    def download(self, download_func: Callable, request: Request, spider: Spider):
-        @defer.inlineCallbacks
-        def process_request(request: Request):
+    def download(
+        self, download_func: Callable, request: Request, spider: Spider
+    ) -> Deferred:
+        @inlineCallbacks
+        def process_request(request: Request) -> Generator[Deferred, Any, Any]:
            for method in self.methods["process_request"]:
                method = cast(Callable, method)
                response = yield deferred_from_coro(
@ -50,8 +53,10 @@ class DownloaderMiddlewareManager(MiddlewareManager):
                    return response
            return (yield download_func(request=request, spider=spider))

-        @defer.inlineCallbacks
-        def process_response(response: Union[Response, Request]):
+        @inlineCallbacks
+        def process_response(
+            response: Union[Response, Request]
+        ) -> Generator[Deferred, Any, Union[Response, Request]]:
            if response is None:
                raise TypeError("Received None in process_response")
            elif isinstance(response, Request):
@ -71,8 +76,10 @@ class DownloaderMiddlewareManager(MiddlewareManager):
                    return response
            return response

-        @defer.inlineCallbacks
-        def process_exception(failure: Failure):
+        @inlineCallbacks
+        def process_exception(
+            failure: Failure,
+        ) -> Generator[Deferred, Any, Union[Failure, Response, Request]]:
            exception = failure.value
            for method in self.methods["process_exception"]:
                method = cast(Callable, method)
--- a/scrapy/core/downloader/tls.py
+++ b/scrapy/core/downloader/tls.py
@ -1,4 +1,5 @@
 import logging
+from typing import Any, Dict

 from OpenSSL import SSL
 from service_identity.exceptions import CertificateError
@ -20,7 +21,7 @@ METHOD_TLSv11 = "TLSv1.1"
 METHOD_TLSv12 = "TLSv1.2"


-openssl_methods = {
+openssl_methods: Dict[str, int] = {
    METHOD_TLS: SSL.SSLv23_METHOD,  # protocol negotiation (recommended)
    METHOD_TLSv10: SSL.TLSv1_METHOD,  # TLS 1.0 only
    METHOD_TLSv11: SSL.TLSv1_1_METHOD,  # TLS 1.1 only
@ -39,11 +40,13 @@ class ScrapyClientTLSOptions(ClientTLSOptions):
    logging warnings. Also, HTTPS connection parameters logging is added.
    """

-    def __init__(self, hostname, ctx, verbose_logging=False):
+    def __init__(self, hostname: str, ctx: SSL.Context, verbose_logging: bool = False):
        super().__init__(hostname, ctx)
-        self.verbose_logging = verbose_logging
+        self.verbose_logging: bool = verbose_logging

-    def _identityVerifyingInfoCallback(self, connection, where, ret):
+    def _identityVerifyingInfoCallback(
+        self, connection: SSL.Connection, where: int, ret: Any
+    ) -> None:
        if where & SSL.SSL_CB_HANDSHAKE_START:
            connection.set_tlsext_host_name(self._hostnameBytes)
        elif where & SSL.SSL_CB_HANDSHAKE_DONE:
@ -55,11 +58,12 @@ class ScrapyClientTLSOptions(ClientTLSOptions):
                    connection.get_cipher_name(),
                )
                server_cert = connection.get_peer_certificate()
-                logger.debug(
-                    'SSL connection certificate: issuer "%s", subject "%s"',
-                    x509name_to_string(server_cert.get_issuer()),
-                    x509name_to_string(server_cert.get_subject()),
-                )
+                if server_cert:
+                    logger.debug(
+                        'SSL connection certificate: issuer "%s", subject "%s"',
+                        x509name_to_string(server_cert.get_issuer()),
+                        x509name_to_string(server_cert.get_subject()),
+                    )
                key_info = get_temp_key_info(connection._ssl)
                if key_info:
                    logger.debug("SSL temp key: %s", key_info)
@ -82,4 +86,6 @@ class ScrapyClientTLSOptions(ClientTLSOptions):
                )


-DEFAULT_CIPHERS = AcceptableCiphers.fromOpenSSLCipherString("DEFAULT")
+DEFAULT_CIPHERS: AcceptableCiphers = AcceptableCiphers.fromOpenSSLCipherString(
+    "DEFAULT"
+)
--- a/scrapy/core/downloader/webclient.py
+++ b/scrapy/core/downloader/webclient.py
@ -1,22 +1,25 @@
 import re
 from time import time
-from urllib.parse import urldefrag, urlparse, urlunparse
+from typing import Optional, Tuple
+from urllib.parse import ParseResult, urldefrag, urlparse, urlunparse

 from twisted.internet import defer
 from twisted.internet.protocol import ClientFactory
 from twisted.web.http import HTTPClient

+from scrapy import Request
 from scrapy.http import Headers
 from scrapy.responsetypes import responsetypes
 from scrapy.utils.httpobj import urlparse_cached
 from scrapy.utils.python import to_bytes, to_unicode


-def _parsed_url_args(parsed):
+def _parsed_url_args(parsed: ParseResult) -> Tuple[bytes, bytes, bytes, int, bytes]:
    # Assume parsed is urlparse-d from Request.url,
    # which was passed via safe_url_string and is ascii-only.
-    path = urlunparse(("", "", parsed.path or "/", parsed.params, parsed.query, ""))
-    path = to_bytes(path, encoding="ascii")
+    path_str = urlunparse(("", "", parsed.path or "/", parsed.params, parsed.query, ""))
+    path = to_bytes(path_str, encoding="ascii")
+    assert parsed.hostname is not None
    host = to_bytes(parsed.hostname, encoding="ascii")
    port = parsed.port
    scheme = to_bytes(parsed.scheme, encoding="ascii")
@ -26,7 +29,7 @@ def _parsed_url_args(parsed):
    return scheme, netloc, host, port, path


-def _parse(url):
+def _parse(url: str) -> Tuple[bytes, bytes, bytes, int, bytes]:
    """Return tuple of (scheme, netloc, host, port, path),
    all in bytes except for port which is int.
    Assume url is from Request.url, which was passed via safe_url_string
@ -132,17 +135,19 @@ class ScrapyHTTPClientFactory(ClientFactory):
            self.scheme, _, self.host, self.port, _ = _parse(proxy)
            self.path = self.url

-    def __init__(self, request, timeout=180):
-        self._url = urldefrag(request.url)[0]
+    def __init__(self, request: Request, timeout: float = 180):
+        self._url: str = urldefrag(request.url)[0]
        # converting to bytes to comply to Twisted interface
-        self.url = to_bytes(self._url, encoding="ascii")
-        self.method = to_bytes(request.method, encoding="ascii")
-        self.body = request.body or None
-        self.headers = Headers(request.headers)
-        self.response_headers = None
-        self.timeout = request.meta.get("download_timeout") or timeout
-        self.start_time = time()
-        self.deferred = defer.Deferred().addCallback(self._build_response, request)
+        self.url: bytes = to_bytes(self._url, encoding="ascii")
+        self.method: bytes = to_bytes(request.method, encoding="ascii")
+        self.body: Optional[bytes] = request.body or None
+        self.headers: Headers = Headers(request.headers)
+        self.response_headers: Optional[Headers] = None
+        self.timeout: float = request.meta.get("download_timeout") or timeout
+        self.start_time: float = time()
+        self.deferred: defer.Deferred = defer.Deferred().addCallback(
+            self._build_response, request
+        )

        # Fixes Twisted 11.1.0+ support as HTTPClientFactory is expected
        # to have _disconnectedDeferred. See Twisted r32329.
@ -150,7 +155,7 @@ class ScrapyHTTPClientFactory(ClientFactory):
        # needed to add the callback _waitForDisconnect.
        # Specifically this avoids the AttributeError exception when
        # clientConnectionFailed method is called.
-        self._disconnectedDeferred = defer.Deferred()
+        self._disconnectedDeferred: defer.Deferred = defer.Deferred()

        self._set_connection_attributes(request)

@ -166,8 +171,8 @@ class ScrapyHTTPClientFactory(ClientFactory):
        elif self.method == b"POST":
            self.headers["Content-Length"] = 0

-    def __repr__(self):
-        return f"<{self.__class__.__name__}: {self.url}>"
+    def __repr__(self) -> str:
+        return f"<{self.__class__.__name__}: {self._url}>"

    def _cancelTimeout(self, result, timeoutCall):
        if timeoutCall.active():
--- a/scrapy/exporters.py
+++ b/scrapy/exporters.py
@ -255,6 +255,9 @@ class CsvItemExporter(BaseItemExporter):
        values = list(self._build_row(x for _, x in fields))
        self.csv_writer.writerow(values)

+    def finish_exporting(self):
+        self.stream.detach()  # Avoid closing the wrapped file.
+
    def _build_row(self, values):
        for s in values:
            try:
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@ -11,10 +11,11 @@ import warnings
 from datetime import datetime
 from pathlib import Path
 from tempfile import NamedTemporaryFile
-from typing import IO, Any, Callable, Optional, Tuple, Union
+from typing import IO, Any, Callable, List, Optional, Tuple, Union
 from urllib.parse import unquote, urlparse

 from twisted.internet import defer, threads
+from twisted.internet.defer import DeferredList
 from w3lib.url import file_uri_to_path
 from zope.interface import Interface, implementer

@ -23,6 +24,8 @@ from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
 from scrapy.extensions.postprocessing import PostProcessingManager
 from scrapy.utils.boto import is_botocore_available
 from scrapy.utils.conf import feed_complete_default_values_from_settings
+from scrapy.utils.defer import maybe_deferred_to_future
+from scrapy.utils.deprecate import create_deprecated_class
 from scrapy.utils.ftp import ftp_store_file
 from scrapy.utils.log import failure_to_exc_info
 from scrapy.utils.misc import create_instance, load_object
@ -271,7 +274,7 @@ class FTPFeedStorage(BlockingFeedStorage):
        )


-class _FeedSlot:
+class FeedSlot:
    def __init__(
        self,
        storage,
@ -341,7 +344,15 @@ class _FeedSlot:
            self._exporting = False


+_FeedSlot = create_deprecated_class(
+    name="_FeedSlot",
+    new_class=FeedSlot,
+)
+
+
 class FeedExporter:
+    _pending_deferreds: List[defer.Deferred] = []
+
    @classmethod
    def from_crawler(cls, crawler):
        exporter = cls(crawler)
@ -407,13 +418,18 @@ class FeedExporter:
                )
            )

-    def close_spider(self, spider):
-        deferred_list = []
+    async def close_spider(self, spider):
        for slot in self.slots:
-            d = self._close_slot(slot, spider)
-            if d:
-                deferred_list.append(d)
-        return defer.DeferredList(deferred_list) if deferred_list else None
+            self._close_slot(slot, spider)
+
+        # Await all deferreds
+        if self._pending_deferreds:
+            await maybe_deferred_to_future(DeferredList(self._pending_deferreds))
+
+        # Send FEED_EXPORTER_CLOSED signal
+        await maybe_deferred_to_future(
+            self.crawler.signals.send_catch_log_deferred(signals.feed_exporter_closed)
+        )

    def _close_slot(self, slot, spider):
        def get_file(slot_):
@ -442,6 +458,14 @@ class FeedExporter:
        d.addErrback(
            self._handle_store_error, logmsg, spider, type(slot.storage).__name__
        )
+        self._pending_deferreds.append(d)
+        d.addCallback(
+            lambda _: self.crawler.signals.send_catch_log_deferred(
+                signals.feed_slot_closed, slot=slot
+            )
+        )
+        d.addBoth(lambda _: self._pending_deferreds.remove(d))
+
        return d

    def _handle_store_error(self, f, logmsg, spider, slot_type):
@ -468,7 +492,21 @@ class FeedExporter:
        :param uri_template: template of uri which contains %(batch_time)s or %(batch_id)d to create new uri
        """
        storage = self._get_storage(uri, feed_options)
-        slot = _FeedSlot(
+        file = storage.open(spider)
+        if "postprocessing" in feed_options:
+            file = PostProcessingManager(
+                feed_options["postprocessing"], file, feed_options
+            )
+
+        exporter = self._get_exporter(
+            file=file,
+            format=feed_options["format"],
+            fields_to_export=feed_options["fields"],
+            encoding=feed_options["encoding"],
+            indent=feed_options["indent"],
+            **feed_options["item_export_kwargs"],
+        )
+        slot = FeedSlot(
            storage=storage,
            uri=uri,
            format=feed_options["format"],
@ -596,7 +634,7 @@ class FeedExporter:
        self,
        spider: Spider,
        uri_params_function: Optional[Union[str, Callable[[dict, Spider], dict]]],
-        slot: Optional[_FeedSlot] = None,
+        slot: Optional[FeedSlot] = None,
    ) -> dict:
        params = {}
        for k in dir(spider):
--- a/scrapy/http/response/init.py
+++ b/scrapy/http/response/init.py
@ -142,6 +142,12 @@ class Response(object_ref):
        """
        raise NotSupported("Response content isn't text")

+    def jmespath(self, *a, **kw):
+        """Shortcut method implemented only by responses whose content
+        is text (subclasses of TextResponse).
+        """
+        raise NotSupported("Response content isn't text")
+
    def xpath(self, *a, **kw):
        """Shortcut method implemented only by responses whose content
        is text (subclasses of TextResponse).
--- a/scrapy/http/response/text.py
+++ b/scrapy/http/response/text.py
@ -100,11 +100,13 @@ class TextResponse(Response):
    @memoizemethod_noargs
    def _headers_encoding(self):
        content_type = self.headers.get(b"Content-Type", b"")
-        return http_content_type_encoding(to_unicode(content_type))
+        return http_content_type_encoding(to_unicode(content_type, encoding="latin-1"))

    def _body_inferred_encoding(self):
        if self._cached_benc is None:
-            content_type = to_unicode(self.headers.get(b"Content-Type", b""))
+            content_type = to_unicode(
+                self.headers.get(b"Content-Type", b""), encoding="latin-1"
+            )
            benc, ubody = html_to_unicode(
                content_type,
                self.body,
@ -139,6 +141,14 @@ class TextResponse(Response):
            self._cached_selector = Selector(self)
        return self._cached_selector

+    def jmespath(self, query, **kwargs):
+        if not hasattr(self.selector, "jmespath"):  # type: ignore[attr-defined]
+            raise AttributeError(
+                "Please install parsel >= 1.8.1 to get jmespath support"
+            )
+
+        return self.selector.jmespath(query, **kwargs)  # type: ignore[attr-defined]
+
    def xpath(self, query, **kwargs):
        return self.selector.xpath(query, **kwargs)

--- a/scrapy/linkextractors/lxmlhtml.py
+++ b/scrapy/linkextractors/lxmlhtml.py
@ -1,6 +1,7 @@
 """
 Link extractor based on lxml.html
 """
+import logging
 import operator
 from functools import partial
 from urllib.parse import urljoin, urlparse
@ -23,6 +24,8 @@ from scrapy.utils.python import unique as unique_list
 from scrapy.utils.response import get_base_url
 from scrapy.utils.url import url_has_any_extension, url_is_from_any_domain

+logger = logging.getLogger(__name__)
+
 # from lxml/src/lxml/html/__init__.py
 XHTML_NAMESPACE = "http://www.w3.org/1999/xhtml"

@ -88,7 +91,12 @@ class LxmlParserLinkExtractor:
                url = self.process_attr(attr_val)
                if url is None:
                    continue
-            url = safe_url_string(url, encoding=response_encoding)
+            try:
+                url = safe_url_string(url, encoding=response_encoding)
+            except ValueError:
+                logger.debug(f"Skipping extraction of link with bad URL {url!r}")
+                continue
+
            # to fix relative links after process_value
            url = urljoin(response_url, url)
            link = Link(
--- a/scrapy/pipelines/files.py
+++ b/scrapy/pipelines/files.py
@ -3,6 +3,7 @@ Files Pipeline

 See documentation in topics/media-pipeline.rst
 """
+import base64
 import functools
 import hashlib
 import logging
@ -228,7 +229,7 @@ class GCSFilesStore:
    def stat_file(self, path, info):
        def _onsuccess(blob):
            if blob:
-                checksum = blob.md5_hash
+                checksum = base64.b64decode(blob.md5_hash).hex()
                last_modified = time.mktime(blob.updated.timetuple())
                return {"checksum": checksum, "last_modified": last_modified}
            return {}
--- a/scrapy/responsetypes.py
+++ b/scrapy/responsetypes.py
@ -51,7 +51,9 @@ class ResponseTypes:
        header"""
        if content_encoding:
            return Response
-        mimetype = to_unicode(content_type).split(";")[0].strip().lower()
+        mimetype = (
+            to_unicode(content_type, encoding="latin-1").split(";")[0].strip().lower()
+        )
        return self.from_mimetype(mimetype)

    def from_content_disposition(self, content_disposition):
--- a/scrapy/selector/unified.py
+++ b/scrapy/selector/unified.py
@ -10,6 +10,8 @@ from scrapy.utils.trackref import object_ref

 __all__ = ["Selector", "SelectorList"]

+_NOT_SET = object()
+

 def _st(response, st):
    if st is None:
@ -63,7 +65,7 @@ class Selector(_ParselSelector, object_ref):
    __slots__ = ["response"]
    selectorlist_cls = SelectorList

-    def __init__(self, response=None, text=None, type=None, root=None, **kwargs):
+    def __init__(self, response=None, text=None, type=None, root=_NOT_SET, **kwargs):
        if response is not None and text is not None:
            raise ValueError(
                f"{self.__class__.__name__}.__init__() received "
@ -80,4 +82,8 @@ class Selector(_ParselSelector, object_ref):
            kwargs.setdefault("base_url", response.url)

        self.response = response
-        super().__init__(text=text, type=st, root=root, **kwargs)
+
+        if root is not _NOT_SET:
+            kwargs["root"] = root
+
+        super().__init__(text=text, type=st, **kwargs)
--- a/scrapy/signals.py
+++ b/scrapy/signals.py
@ -22,6 +22,8 @@ bytes_received = object()
 item_scraped = object()
 item_dropped = object()
 item_error = object()
+feed_slot_closed = object()
+feed_exporter_closed = object()

 # for backward compatibility
 stats_spider_opened = spider_opened
--- a/scrapy/utils/curl.py
+++ b/scrapy/utils/curl.py
@ -7,6 +7,14 @@ from urllib.parse import urlparse
 from w3lib.http import basic_auth_header


+class DataAction(argparse.Action):
+    def __call__(self, parser, namespace, values, option_string=None):
+        value = str(values)
+        if value.startswith("$"):
+            value = value[1:]
+        setattr(namespace, self.dest, value)
+
+
 class CurlParser(argparse.ArgumentParser):
    def error(self, message):
        error_msg = f"There was an error parsing the curl command: {message}"
@ -17,7 +25,7 @@ curl_parser = CurlParser()
 curl_parser.add_argument("url")
 curl_parser.add_argument("-H", "--header", dest="headers", action="append")
 curl_parser.add_argument("-X", "--request", dest="method")
-curl_parser.add_argument("-d", "--data", "--data-raw", dest="data")
+curl_parser.add_argument("-d", "--data", "--data-raw", dest="data", action=DataAction)
 curl_parser.add_argument("-u", "--user", dest="auth")


--- a/scrapy/utils/defer.py
+++ b/scrapy/utils/defer.py
@ -340,8 +340,9 @@ def deferred_to_future(d: Deferred) -> Future:
        class MySpider(Spider):
            ...
            async def parse(self, response):
-                d = treq.get('https://example.com/additional')
-                additional_response = await deferred_to_future(d)
+                additional_request = scrapy.Request('https://example.org/price')
+                deferred = self.crawler.engine.download(additional_request)
+                additional_response = await deferred_to_future(deferred)
    """
    return d.asFuture(_get_asyncio_event_loop())

@ -368,8 +369,9 @@ def maybe_deferred_to_future(d: Deferred) -> Union[Deferred, Future]:
        class MySpider(Spider):
            ...
            async def parse(self, response):
-                d = treq.get('https://example.com/additional')
-                extra_response = await maybe_deferred_to_future(d)
+                additional_request = scrapy.Request('https://example.org/price')
+                deferred = self.crawler.engine.download(additional_request)
+                additional_response = await maybe_deferred_to_future(deferred)
    """
    if not is_asyncio_reactor_installed():
        return d
--- a/scrapy/utils/python.py
+++ b/scrapy/utils/python.py
@ -8,7 +8,16 @@ import sys
 import weakref
 from functools import partial, wraps
 from itertools import chain
-from typing import Any, AsyncGenerator, AsyncIterable, Iterable, Union
+from typing import (
+    Any,
+    AsyncGenerator,
+    AsyncIterable,
+    Iterable,
+    Mapping,
+    Optional,
+    Union,
+    overload,
+)

 from scrapy.utils.asyncgen import as_async_generator

@ -82,7 +91,9 @@ def unique(list_, key=lambda x: x):
    return result


-def to_unicode(text, encoding=None, errors="strict"):
+def to_unicode(
+    text: Union[str, bytes], encoding: Optional[str] = None, errors: str = "strict"
+) -> str:
    """Return the unicode representation of a bytes object ``text``. If
    ``text`` is already an unicode object, return it as-is."""
    if isinstance(text, str):
@ -97,7 +108,9 @@ def to_unicode(text, encoding=None, errors="strict"):
    return text.decode(encoding, errors)


-def to_bytes(text, encoding=None, errors="strict"):
+def to_bytes(
+    text: Union[str, bytes], encoding: Optional[str] = None, errors: str = "strict"
+) -> bytes:
    """Return the binary representation of ``text``. If ``text``
    is already a bytes object, return it as-is."""
    if isinstance(text, bytes):
@ -160,11 +173,12 @@ def memoizemethod_noargs(method):
    return new_method


-_BINARYCHARS = {to_bytes(chr(i)) for i in range(32)} - {b"\0", b"\t", b"\n", b"\r"}
-_BINARYCHARS |= {ord(ch) for ch in _BINARYCHARS}
+_BINARYCHARS = {
+    i for i in range(32) if to_bytes(chr(i)) not in {b"\0", b"\t", b"\n", b"\r"}
+}


-def binary_is_text(data):
+def binary_is_text(data: bytes) -> bool:
    """Returns ``True`` if the given ``data`` argument (a ``bytes`` object)
    does not contain unprintable control characters.
    """
@ -174,33 +188,33 @@ def binary_is_text(data):


 def get_func_args(func, stripself=False):
-    """Return the argument name list of a callable"""
-    if inspect.isfunction(func):
-        spec = inspect.getfullargspec(func)
-        func_args = spec.args + spec.kwonlyargs
-    elif inspect.isclass(func):
-        return get_func_args(func.__init__, True)
-    elif inspect.ismethod(func):
-        return get_func_args(func.__func__, True)
-    elif inspect.ismethoddescriptor(func):
-        return []
-    elif isinstance(func, partial):
-        return [
-            x
-            for x in get_func_args(func.func)[len(func.args) :]
-            if not (func.keywords and x in func.keywords)
-        ]
-    elif hasattr(func, "__call__"):
-        if inspect.isroutine(func):
-            return []
-        if getattr(func, "__name__", None) == "__call__":
-            return []
-        return get_func_args(func.__call__, True)
+    """Return the argument name list of a callable object"""
+    if not callable(func):
+        raise TypeError(f"func must be callable, got '{type(func).__name__}'")
+
+    args = []
+    try:
+        sig = inspect.signature(func)
+    except ValueError:
+        return args
+
+    if isinstance(func, partial):
+        partial_args = func.args
+        partial_kw = func.keywords
+
+        for name, param in sig.parameters.items():
+            if param.name in partial_args:
+                continue
+            if partial_kw and param.name in partial_kw:
+                continue
+            args.append(name)
    else:
-        raise TypeError(f"{type(func)} is not callable")
-    if stripself:
-        func_args.pop(0)
-    return func_args
+        for name in sig.parameters.keys():
+            args.append(name)
+
+    if stripself and args and args[0] == "self":
+        args = args[1:]
+    return args


 def get_spec(func):
@ -258,6 +272,16 @@ def equal_attributes(obj1, obj2, attributes):
    return True


+@overload
+def without_none_values(iterable: Mapping) -> dict:
+    ...
+
+
+@overload
+def without_none_values(iterable: Iterable) -> Iterable:
+    ...
+
+
 def without_none_values(iterable):
    """Return a copy of ``iterable`` with all ``None`` entries removed.

--- a/scrapy/utils/request.py
+++ b/scrapy/utils/request.py
@ -327,3 +327,34 @@ def _get_method(obj, name):
        return getattr(obj, name)
    except AttributeError:
        raise ValueError(f"Method {name!r} not found in: {obj}")
+
+
+def request_to_curl(request: Request) -> str:
+    """
+    Converts a :class:`~scrapy.Request` object to a curl command.
+
+    :param :class:`~scrapy.Request`: Request object to be converted
+    :return: string containing the curl command
+    """
+    method = request.method
+
+    data = f"--data-raw '{request.body.decode('utf-8')}'" if request.body else ""
+
+    headers = " ".join(
+        f"-H '{k.decode()}: {v[0].decode()}'" for k, v in request.headers.items()
+    )
+
+    url = request.url
+    cookies = ""
+    if request.cookies:
+        if isinstance(request.cookies, dict):
+            cookie = "; ".join(f"{k}={v}" for k, v in request.cookies.items())
+            cookies = f"--cookie '{cookie}'"
+        elif isinstance(request.cookies, list):
+            cookie = "; ".join(
+                f"{list(c.keys())[0]}={list(c.values())[0]}" for c in request.cookies
+            )
+            cookies = f"--cookie '{cookie}'"
+
+    curl_cmd = f"curl -X {method} {url} {data} {headers} {cookies}".strip()
+    return " ".join(curl_cmd.split())
--- a/scrapy/utils/ssl.py
+++ b/scrapy/utils/ssl.py
@ -1,24 +1,28 @@
+from typing import Any, Optional, cast
+
 import OpenSSL._util as pyOpenSSLutil
 import OpenSSL.SSL
+import OpenSSL.version
+from OpenSSL.crypto import X509Name

 from scrapy.utils.python import to_unicode


-def ffi_buf_to_string(buf):
+def ffi_buf_to_string(buf: Any) -> str:
    return to_unicode(pyOpenSSLutil.ffi.string(buf))


-def x509name_to_string(x509name):
+def x509name_to_string(x509name: X509Name) -> str:
    # from OpenSSL.crypto.X509Name.__repr__
-    result_buffer = pyOpenSSLutil.ffi.new("char[]", 512)
+    result_buffer: Any = pyOpenSSLutil.ffi.new("char[]", 512)
    pyOpenSSLutil.lib.X509_NAME_oneline(
-        x509name._name, result_buffer, len(result_buffer)
+        x509name._name, result_buffer, len(result_buffer)  # type: ignore[attr-defined]
    )

    return ffi_buf_to_string(result_buffer)


-def get_temp_key_info(ssl_object):
+def get_temp_key_info(ssl_object: Any) -> Optional[str]:
    # adapted from OpenSSL apps/s_cb.c::ssl_print_tmp_key()
    if not hasattr(pyOpenSSLutil.lib, "SSL_get_server_tmp_key"):
        # removed in cryptography 40.0.0
@ -53,8 +57,10 @@ def get_temp_key_info(ssl_object):
    return ", ".join(key_info)


-def get_openssl_version():
-    system_openssl = OpenSSL.SSL.SSLeay_version(OpenSSL.SSL.SSLEAY_VERSION).decode(
-        "ascii", errors="replace"
+def get_openssl_version() -> str:
+    # https://github.com/python/typeshed/issues/10024
+    system_openssl_bytes = cast(
+        bytes, OpenSSL.SSL.SSLeay_version(OpenSSL.SSL.SSLEAY_VERSION)
    )
+    system_openssl = system_openssl_bytes.decode("ascii", errors="replace")
    return f"{OpenSSL.version.__version__} ({system_openssl})"
--- a/tests/test_exporters.py
+++ b/tests/test_exporters.py
@ -90,6 +90,10 @@ class BaseItemExporterTest(unittest.TestCase):
            if self.ie.__class__ is not BaseItemExporter:
                raise
        self.ie.finish_exporting()
+        # Delete the item exporter object, so that if it causes the output
+        # file handle to be closed, which should not be the case, follow-up
+        # interactions with the output file handle will surface the issue.
+        del self.ie
        self._check_output()

    def test_export_item(self):
@ -243,6 +247,7 @@ class PickleItemExporterTest(BaseItemExporterTest):
        ie.export_item(i1)
        ie.export_item(i2)
        ie.finish_exporting()
+        del ie  # See the first “del self.ie” in this file for context.
        f.seek(0)
        self.assertEqual(self.item_class(**pickle.load(f)), i1)
        self.assertEqual(self.item_class(**pickle.load(f)), i2)
@ -254,6 +259,7 @@ class PickleItemExporterTest(BaseItemExporterTest):
        ie.start_exporting()
        ie.export_item(item)
        ie.finish_exporting()
+        del ie  # See the first “del self.ie” in this file for context.
        self.assertEqual(pickle.loads(fp.getvalue()), item)


@ -279,6 +285,7 @@ class MarshalItemExporterTest(BaseItemExporterTest):
        ie.start_exporting()
        ie.export_item(item)
        ie.finish_exporting()
+        del ie  # See the first “del self.ie” in this file for context.
        fp.seek(0)
        self.assertEqual(marshal.load(fp), item)

@ -314,6 +321,7 @@ class CsvItemExporterTest(BaseItemExporterTest):
        ie.start_exporting()
        ie.export_item(item)
        ie.finish_exporting()
+        del ie  # See the first “del self.ie” in this file for context.
        self.assertCsvEqual(fp.getvalue(), expected)

    def test_header_export_all(self):
@ -345,6 +353,7 @@ class CsvItemExporterTest(BaseItemExporterTest):
            ie.export_item(item)
            ie.export_item(item)
            ie.finish_exporting()
+            del ie  # See the first “del self.ie” in this file for context.
            self.assertCsvEqual(
                output.getvalue(), b"age,name\r\n22,John\xc2\xa3\r\n22,John\xc2\xa3\r\n"
            )
@ -429,6 +438,7 @@ class XmlItemExporterTest(BaseItemExporterTest):
        ie.start_exporting()
        ie.export_item(item)
        ie.finish_exporting()
+        del ie  # See the first “del self.ie” in this file for context.
        self.assertXmlEquivalent(fp.getvalue(), expected_value)

    def _check_output(self):
@ -536,6 +546,7 @@ class JsonLinesItemExporterTest(BaseItemExporterTest):
        self.ie.start_exporting()
        self.ie.export_item(i3)
        self.ie.finish_exporting()
+        del self.ie  # See the first “del self.ie” in this file for context.
        exported = json.loads(to_unicode(self.output.getvalue()))
        self.assertEqual(exported, self._expected_nested)

@ -550,6 +561,7 @@ class JsonLinesItemExporterTest(BaseItemExporterTest):
        self.ie.start_exporting()
        self.ie.export_item(item)
        self.ie.finish_exporting()
+        del self.ie  # See the first “del self.ie” in this file for context.
        exported = json.loads(to_unicode(self.output.getvalue()))
        item["time"] = str(item["time"])
        self.assertEqual(exported, item)
@ -575,6 +587,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
        self.ie.export_item(item)
        self.ie.export_item(item)
        self.ie.finish_exporting()
+        del self.ie  # See the first “del self.ie” in this file for context.
        exported = json.loads(to_unicode(self.output.getvalue()))
        self.assertEqual(
            exported, [ItemAdapter(item).asdict(), ItemAdapter(item).asdict()]
@ -593,6 +606,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
        self.ie.start_exporting()
        self.ie.export_item(i3)
        self.ie.finish_exporting()
+        del self.ie  # See the first “del self.ie” in this file for context.
        exported = json.loads(to_unicode(self.output.getvalue()))
        expected = {
            "name": "Jesus",
@ -607,6 +621,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
        self.ie.start_exporting()
        self.ie.export_item(i3)
        self.ie.finish_exporting()
+        del self.ie  # See the first “del self.ie” in this file for context.
        exported = json.loads(to_unicode(self.output.getvalue()))
        expected = {"name": "Jesus", "age": {"name": "Maria", "age": i1}}
        self.assertEqual(exported, [expected])
@ -616,6 +631,7 @@ class JsonItemExporterTest(JsonLinesItemExporterTest):
        self.ie.start_exporting()
        self.ie.export_item(item)
        self.ie.finish_exporting()
+        del self.ie  # See the first “del self.ie” in this file for context.
        exported = json.loads(to_unicode(self.output.getvalue()))
        item["time"] = str(item["time"])
        self.assertEqual(exported, [item])
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@ -32,18 +32,19 @@ from zope.interface import implementer
 from zope.interface.verify import verifyObject

 import scrapy
+from scrapy import signals
 from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
 from scrapy.exporters import CsvItemExporter, JsonItemExporter
 from scrapy.extensions.feedexport import (
    BlockingFeedStorage,
    FeedExporter,
+    FeedSlot,
    FileFeedStorage,
    FTPFeedStorage,
    GCSFeedStorage,
    IFeedStorage,
    S3FeedStorage,
    StdoutFeedStorage,
-    _FeedSlot,
 )
 from scrapy.settings import Settings
 from scrapy.utils.python import to_unicode
@ -660,8 +661,8 @@ class FeedExportTestBase(ABC, unittest.TestCase):
        return result


-class InstrumentedFeedSlot(_FeedSlot):
-    """Instrumented _FeedSlot subclass for keeping track of calls to
+class InstrumentedFeedSlot(FeedSlot):
+    """Instrumented FeedSlot subclass for keeping track of calls to
    start_exporting and finish_exporting."""

    def start_exporting(self):
@ -964,7 +965,7 @@ class FeedExportTest(FeedExportTestBase):
        listener = IsExportingListener()
        InstrumentedFeedSlot.subscribe__listener(listener)

-        with mock.patch("scrapy.extensions.feedexport._FeedSlot", InstrumentedFeedSlot):
+        with mock.patch("scrapy.extensions.feedexport.FeedSlot", InstrumentedFeedSlot):
            _ = yield self.exported_data(items, settings)
            self.assertFalse(listener.start_without_finish)
            self.assertFalse(listener.finish_without_start)
@ -982,7 +983,7 @@ class FeedExportTest(FeedExportTestBase):
        listener = IsExportingListener()
        InstrumentedFeedSlot.subscribe__listener(listener)

-        with mock.patch("scrapy.extensions.feedexport._FeedSlot", InstrumentedFeedSlot):
+        with mock.patch("scrapy.extensions.feedexport.FeedSlot", InstrumentedFeedSlot):
            _ = yield self.exported_data(items, settings)
            self.assertFalse(listener.start_without_finish)
            self.assertFalse(listener.finish_without_start)
@ -1003,7 +1004,7 @@ class FeedExportTest(FeedExportTestBase):
        listener = IsExportingListener()
        InstrumentedFeedSlot.subscribe__listener(listener)

-        with mock.patch("scrapy.extensions.feedexport._FeedSlot", InstrumentedFeedSlot):
+        with mock.patch("scrapy.extensions.feedexport.FeedSlot", InstrumentedFeedSlot):
            _ = yield self.exported_data(items, settings)
            self.assertFalse(listener.start_without_finish)
            self.assertFalse(listener.finish_without_start)
@ -1022,7 +1023,7 @@ class FeedExportTest(FeedExportTestBase):
        listener = IsExportingListener()
        InstrumentedFeedSlot.subscribe__listener(listener)

-        with mock.patch("scrapy.extensions.feedexport._FeedSlot", InstrumentedFeedSlot):
+        with mock.patch("scrapy.extensions.feedexport.FeedSlot", InstrumentedFeedSlot):
            _ = yield self.exported_data(items, settings)
            self.assertFalse(listener.start_without_finish)
            self.assertFalse(listener.finish_without_start)
@ -2541,7 +2542,7 @@ class BatchDeliveriesTest(FeedExportTestBase):
    def test_batch_path_differ(self):
        """
        Test that the name of all batch files differ from each other.
-        So %(batch_time)s replaced with the current date.
+        So %(batch_id)d replaced with the current id.
        """
        items = [
            self.MyItem({"foo": "bar1", "egg": "spam1"}),
@ -2551,7 +2552,7 @@ class BatchDeliveriesTest(FeedExportTestBase):
        settings = {
            "FEEDS": {
                self._random_temp_filename()
-                / "%(batch_time)s": {
+                / "%(batch_id)d": {
                    "format": "json",
                },
            },
@ -2614,7 +2615,7 @@ class BatchDeliveriesTest(FeedExportTestBase):
                return super().open(*args, **kwargs)

        key = "export.csv"
-        uri = f"s3://{bucket}/{key}/%(batch_time)s.json"
+        uri = f"s3://{bucket}/{key}/%(batch_id)d.json"
        batch_item_count = 1
        settings = {
            "AWS_ACCESS_KEY_ID": "access_key",
@ -2650,6 +2651,83 @@ class BatchDeliveriesTest(FeedExportTestBase):
            stub.assert_no_pending_responses()


+# Test that the FeedExporer sends the feed_exporter_closed and feed_slot_closed signals
+class FeedExporterSignalsTest(unittest.TestCase):
+    items = [
+        {"foo": "bar1", "egg": "spam1"},
+        {"foo": "bar2", "egg": "spam2", "baz": "quux2"},
+        {"foo": "bar3", "baz": "quux3"},
+    ]
+
+    with tempfile.NamedTemporaryFile(suffix="json") as tmp:
+        settings = {
+            "FEEDS": {
+                f"file:///{tmp.name}": {
+                    "format": "json",
+                },
+            },
+        }
+
+    def feed_exporter_closed_signal_handler(self):
+        self.feed_exporter_closed_received = True
+
+    def feed_slot_closed_signal_handler(self, slot):
+        self.feed_slot_closed_received = True
+
+    def feed_exporter_closed_signal_handler_deferred(self):
+        d = defer.Deferred()
+        d.addCallback(lambda _: setattr(self, "feed_exporter_closed_received", True))
+        d.callback(None)
+        return d
+
+    def feed_slot_closed_signal_handler_deferred(self, slot):
+        d = defer.Deferred()
+        d.addCallback(lambda _: setattr(self, "feed_slot_closed_received", True))
+        d.callback(None)
+        return d
+
+    def run_signaled_feed_exporter(
+        self, feed_exporter_signal_handler, feed_slot_signal_handler
+    ):
+        crawler = get_crawler(settings_dict=self.settings)
+        feed_exporter = FeedExporter.from_crawler(crawler)
+        spider = scrapy.Spider("default")
+        spider.crawler = crawler
+        crawler.signals.connect(
+            feed_exporter_signal_handler,
+            signal=signals.feed_exporter_closed,
+        )
+        crawler.signals.connect(
+            feed_slot_signal_handler, signal=signals.feed_slot_closed
+        )
+        feed_exporter.open_spider(spider)
+        for item in self.items:
+            feed_exporter.item_scraped(item, spider)
+        defer.ensureDeferred(feed_exporter.close_spider(spider))
+
+    def test_feed_exporter_signals_sent(self):
+        self.feed_exporter_closed_received = False
+        self.feed_slot_closed_received = False
+
+        self.run_signaled_feed_exporter(
+            self.feed_exporter_closed_signal_handler,
+            self.feed_slot_closed_signal_handler,
+        )
+        self.assertTrue(self.feed_slot_closed_received)
+        self.assertTrue(self.feed_exporter_closed_received)
+
+    def test_feed_exporter_signals_sent_deferred(self):
+        self.feed_exporter_closed_received = False
+        self.feed_slot_closed_received = False
+
+        self.run_signaled_feed_exporter(
+            self.feed_exporter_closed_signal_handler_deferred,
+            self.feed_slot_closed_signal_handler_deferred,
+        )
+        self.assertTrue(self.feed_slot_closed_received)
+        self.assertTrue(self.feed_exporter_closed_received)
+
+
 class FeedExportInitTest(unittest.TestCase):
    def test_unsupported_storage(self):
        settings = {
--- a/tests/test_http_response.py
+++ b/tests/test_http_response.py
@ -32,6 +32,9 @@ class BaseResponseTest(unittest.TestCase):
            isinstance(self.response_class("http://example.com/"), self.response_class)
        )
        self.assertRaises(TypeError, self.response_class, b"http://example.com")
+        self.assertRaises(
+            TypeError, self.response_class, url="http://example.com", body={}
+        )
        # body can be str or None
        self.assertTrue(
            isinstance(
@ -192,6 +195,7 @@ class BaseResponseTest(unittest.TestCase):
            self.assertRaisesRegex(AttributeError, msg, getattr, r, "text")
            self.assertRaisesRegex(NotSupported, msg, r.css, "body")
            self.assertRaisesRegex(NotSupported, msg, r.xpath, "//body")
+            self.assertRaisesRegex(NotSupported, msg, r.jmespath, "body")
        else:
            r.text
            r.css("body")
@ -448,6 +452,13 @@ class TextResponseTest(BaseResponseTest):
            body=codecs.BOM_UTF8 + b"\xc2\xa3",
            headers={"Content-type": ["text/html; charset=cp1251"]},
        )
+        r9 = self.response_class(
+            "http://www.example.com",
+            body=b"\x80",
+            headers={
+                "Content-type": [b"application/x-download; filename=\x80dummy.txt"]
+            },
+        )

        self.assertEqual(r1._headers_encoding(), "utf-8")
        self.assertEqual(r2._headers_encoding(), None)
@ -458,9 +469,12 @@ class TextResponseTest(BaseResponseTest):
        self.assertEqual(r4._headers_encoding(), None)
        self.assertEqual(r5._headers_encoding(), None)
        self.assertEqual(r8._headers_encoding(), "cp1251")
+        self.assertEqual(r9._headers_encoding(), None)
        self.assertEqual(r8._declared_encoding(), "utf-8")
+        self.assertEqual(r9._declared_encoding(), None)
        self._assert_response_encoding(r5, "utf-8")
        self._assert_response_encoding(r8, "utf-8")
+        self._assert_response_encoding(r9, "cp1252")
        assert (
            r4._body_inferred_encoding() is not None
            and r4._body_inferred_encoding() != "ascii"
@ -470,6 +484,7 @@ class TextResponseTest(BaseResponseTest):
        self._assert_response_values(r3, "iso-8859-1", "\xa3")
        self._assert_response_values(r6, "gb18030", "\u2015")
        self._assert_response_values(r7, "gb18030", "\u2015")
+        self._assert_response_values(r9, "cp1252", "€")

        # TextResponse (and subclasses) must be passed a encoding when instantiating with unicode bodies
        self.assertRaises(
--- a/tests/test_linkextractors.py
+++ b/tests/test_linkextractors.py
@ -2,6 +2,10 @@ import pickle
 import re
 import unittest

+from packaging.version import Version
+from pytest import mark
+from w3lib import __version__ as w3lib_version
+
 from scrapy.http import HtmlResponse, XmlResponse
 from scrapy.link import Link
 from scrapy.linkextractors.lxmlhtml import LxmlLinkExtractor
@ -815,3 +819,34 @@ class LxmlLinkExtractorTestCase(Base.LinkExtractorTestCase):

    def test_restrict_xpaths_with_html_entities(self):
        super().test_restrict_xpaths_with_html_entities()
+
+    @mark.skipif(
+        Version(w3lib_version) < Version("2.0.0"),
+        reason=(
+            "Before w3lib 2.0.0, w3lib.url.safe_url_string would not complain "
+            "about an invalid port value."
+        ),
+    )
+    def test_skip_bad_links(self):
+        html = b"""
+        <a href="http://example.org:non-port">Why would you do this?</a>
+        <a href="http://example.org/item2.html">Good Link</a>
+        <a href="http://example.org/item3.html">Good Link 2</a>
+        """
+        response = HtmlResponse("http://example.org/index.html", body=html)
+        lx = self.extractor_cls()
+        self.assertEqual(
+            [link for link in lx.extract_links(response)],
+            [
+                Link(
+                    url="http://example.org/item2.html",
+                    text="Good Link",
+                    nofollow=False,
+                ),
+                Link(
+                    url="http://example.org/item3.html",
+                    text="Good Link 2",
+                    nofollow=False,
+                ),
+            ],
+        )
--- a/tests/test_pipeline_files.py
+++ b/tests/test_pipeline_files.py
@ -601,7 +601,7 @@ class TestGCSFilesStore(unittest.TestCase):
        s = yield store.stat_file(path, info=None)
        self.assertIn("last_modified", s)
        self.assertIn("checksum", s)
-        self.assertEqual(s["checksum"], "zc2oVgXkbQr2EQdSdw3OPA==")
+        self.assertEqual(s["checksum"], "cdcda85605e46d0af6110752770dce3c")
        u = urlparse(uri)
        content, acl, blob = get_gcs_content_and_delete(u.hostname, u.path[1:] + path)
        self.assertEqual(content, data)
--- a/tests/test_responsetypes.py
+++ b/tests/test_responsetypes.py
@ -42,6 +42,7 @@ class ResponseTypesTest(unittest.TestCase):
            ("application/octet-stream", Response),
            ("application/x-json; encoding=UTF8;charset=UTF-8", TextResponse),
            ("application/json-amazonui-streaming;charset=UTF-8", TextResponse),
+            (b"application/x-download; filename=\x80dummy.txt", Response),
        ]
        for source, cls in mappings:
            retcls = responsetypes.from_content_type(source)
--- a/tests/test_selector.py
+++ b/tests/test_selector.py
@ -1,10 +1,16 @@
 import weakref

+import parsel
+import pytest
+from packaging import version
 from twisted.trial import unittest

 from scrapy.http import HtmlResponse, TextResponse, XmlResponse
 from scrapy.selector import Selector

+PARSEL_VERSION = version.parse(getattr(parsel, "__version__", "0.0"))
+PARSEL_18_PLUS = PARSEL_VERSION >= version.parse("1.8.0")
+

 class SelectorTestCase(unittest.TestCase):
    def test_simple_selection(self):
@ -108,3 +114,162 @@ class SelectorTestCase(unittest.TestCase):
    def test_selector_bad_args(self):
        with self.assertRaisesRegex(ValueError, "received both response and text"):
            Selector(TextResponse(url="http://example.com", body=b""), text="")
+
+
+class JMESPathTestCase(unittest.TestCase):
+    @pytest.mark.skipif(
+        not PARSEL_18_PLUS, reason="parsel < 1.8 doesn't support jmespath"
+    )
+    def test_json_has_html(self) -> None:
+        """Sometimes the information is returned in a json wrapper"""
+
+        body = """
+        {
+            "content": [
+                {
+                    "name": "A",
+                    "value": "a"
+                },
+                {
+                    "name": {
+                        "age": 18
+                    },
+                    "value": "b"
+                },
+                {
+                    "name": "C",
+                    "value": "c"
+                },
+                {
+                    "name": "<a>D</a>",
+                    "value": "<div>d</div>"
+                }
+            ],
+            "html": "<div><a>a<br>b</a>c</div><div><a>d</a>e<b>f</b></div>"
+        }
+        """
+        resp = TextResponse(url="http://example.com", body=body, encoding="utf-8")
+        self.assertEqual(
+            resp.jmespath("html").get(),
+            "<div><a>a<br>b</a>c</div><div><a>d</a>e<b>f</b></div>",
+        )
+        self.assertEqual(
+            resp.jmespath("html").xpath("//div/a/text()").getall(),
+            ["a", "b", "d"],
+        )
+        self.assertEqual(resp.jmespath("html").css("div > b").getall(), ["<b>f</b>"])
+        self.assertEqual(resp.jmespath("content").jmespath("name.age").get(), "18")
+
+    @pytest.mark.skipif(
+        not PARSEL_18_PLUS, reason="parsel < 1.8 doesn't support jmespath"
+    )
+    def test_html_has_json(self) -> None:
+        body = """
+        <div>
+            <h1>Information</h1>
+            <content>
+            {
+              "user": [
+                        {
+                                  "name": "A",
+                                  "age": 18
+                        },
+                        {
+                                  "name": "B",
+                                  "age": 32
+                        },
+                        {
+                                  "name": "C",
+                                  "age": 22
+                        },
+                        {
+                                  "name": "D",
+                                  "age": 25
+                        }
+              ],
+              "total": 4,
+              "status": "ok"
+            }
+            </content>
+        </div>
+        """
+        resp = TextResponse(url="http://example.com", body=body, encoding="utf-8")
+        self.assertEqual(
+            resp.xpath("//div/content/text()").jmespath("user[*].name").getall(),
+            ["A", "B", "C", "D"],
+        )
+        self.assertEqual(
+            resp.xpath("//div/content").jmespath("user[*].name").getall(),
+            ["A", "B", "C", "D"],
+        )
+        self.assertEqual(resp.xpath("//div/content").jmespath("total").get(), "4")
+
+    @pytest.mark.skipif(
+        not PARSEL_18_PLUS, reason="parsel < 1.8 doesn't support jmespath"
+    )
+    def test_jmestpath_with_re(self) -> None:
+        body = """
+            <div>
+                <h1>Information</h1>
+                <content>
+                {
+                  "user": [
+                            {
+                                      "name": "A",
+                                      "age": 18
+                            },
+                            {
+                                      "name": "B",
+                                      "age": 32
+                            },
+                            {
+                                      "name": "C",
+                                      "age": 22
+                            },
+                            {
+                                      "name": "D",
+                                      "age": 25
+                            }
+                  ],
+                  "total": 4,
+                  "status": "ok"
+                }
+                </content>
+            </div>
+            """
+        resp = TextResponse(url="http://example.com", body=body, encoding="utf-8")
+        self.assertEqual(
+            resp.xpath("//div/content/text()").jmespath("user[*].name").re(r"(\w+)"),
+            ["A", "B", "C", "D"],
+        )
+        self.assertEqual(
+            resp.xpath("//div/content").jmespath("user[*].name").re(r"(\w+)"),
+            ["A", "B", "C", "D"],
+        )
+
+        self.assertEqual(
+            resp.xpath("//div/content").jmespath("unavailable").re(r"(\d+)"), []
+        )
+
+        self.assertEqual(
+            resp.xpath("//div/content").jmespath("unavailable").re_first(r"(\d+)"),
+            None,
+        )
+
+        self.assertEqual(
+            resp.xpath("//div/content")
+            .jmespath("user[*].age.to_string(@)")
+            .re(r"(\d+)"),
+            ["18", "32", "22", "25"],
+        )
+
+    @pytest.mark.skipif(PARSEL_18_PLUS, reason="parsel >= 1.8 supports jmespath")
+    def test_jmespath_not_available(my_json_page) -> None:
+        body = """
+        {
+            "website": {"name": "Example"}
+        }
+        """
+        resp = TextResponse(url="http://example.com", body=body, encoding="utf-8")
+        with pytest.raises(AttributeError):
+            resp.jmespath("website.name").get()
--- a/tests/test_utils_curl.py
+++ b/tests/test_utils_curl.py
@ -154,6 +154,15 @@ class CurlToRequestKwargsTest(unittest.TestCase):
        }
        self._test_command(curl_command, expected_result)

+    def test_post_data_raw_with_string_prefix(self):
+        curl_command = "curl 'https://www.example.org/' --data-raw $'{\"$filters\":\"Filter\u0021\"}'"
+        expected_result = {
+            "method": "POST",
+            "url": "https://www.example.org/",
+            "body": '{"$filters":"Filter!"}',
+        }
+        self._test_command(curl_command, expected_result)
+
    def test_explicit_get_with_data(self):
        curl_command = "curl httpbin.org/anything -X GET --data asdf"
        expected_result = {
--- a/tests/test_utils_python.py
+++ b/tests/test_utils_python.py
@ -235,20 +235,16 @@ class UtilsPythonTestCase(unittest.TestCase):
        self.assertEqual(get_func_args(partial_f3), ["c"])
        self.assertEqual(get_func_args(cal), ["a", "b", "c"])
        self.assertEqual(get_func_args(object), [])
+        self.assertEqual(get_func_args(str.split, stripself=True), ["sep", "maxsplit"])
+        self.assertEqual(get_func_args(" ".join, stripself=True), ["iterable"])

        if platform.python_implementation() == "CPython":
-            # TODO: how do we fix this to return the actual argument names?
-            self.assertEqual(get_func_args(str.split), [])
-            self.assertEqual(get_func_args(" ".join), [])
+            # doesn't work on CPython: https://bugs.python.org/issue42785
            self.assertEqual(get_func_args(operator.itemgetter(2)), [])
        elif platform.python_implementation() == "PyPy":
-            self.assertEqual(
-                get_func_args(str.split, stripself=True), ["sep", "maxsplit"]
-            )
            self.assertEqual(
                get_func_args(operator.itemgetter(2), stripself=True), ["obj"]
            )
-            self.assertEqual(get_func_args(" ".join, stripself=True), ["iterable"])

    def test_without_none_values(self):
        self.assertEqual(without_none_values([1, None, 3, 4]), [1, 3, 4])
--- a/tests/test_utils_request.py
+++ b/tests/test_utils_request.py
@ -1,3 +1,4 @@
+import json
 import unittest
 import warnings
 from hashlib import sha1
@ -18,6 +19,7 @@ from scrapy.utils.request import (
    request_authenticate,
    request_fingerprint,
    request_httprepr,
+    request_to_curl,
 )
 from scrapy.utils.test import get_crawler

@ -666,5 +668,67 @@ class CustomRequestFingerprinterTestCase(unittest.TestCase):
        self.assertEqual(fingerprint, settings["FINGERPRINT"])


+class RequestToCurlTest(unittest.TestCase):
+    def _test_request(self, request_object, expected_curl_command):
+        curl_command = request_to_curl(request_object)
+        self.assertEqual(curl_command, expected_curl_command)
+
+    def test_get(self):
+        request_object = Request("https://www.example.com")
+        expected_curl_command = "curl -X GET https://www.example.com"
+        self._test_request(request_object, expected_curl_command)
+
+    def test_post(self):
+        request_object = Request(
+            "https://www.httpbin.org/post",
+            method="POST",
+            body=json.dumps({"foo": "bar"}),
+        )
+        expected_curl_command = (
+            'curl -X POST https://www.httpbin.org/post --data-raw \'{"foo": "bar"}\''
+        )
+        self._test_request(request_object, expected_curl_command)
+
+    def test_headers(self):
+        request_object = Request(
+            "https://www.httpbin.org/post",
+            method="POST",
+            headers={"Content-Type": "application/json", "Accept": "application/json"},
+            body=json.dumps({"foo": "bar"}),
+        )
+        expected_curl_command = (
+            "curl -X POST https://www.httpbin.org/post"
+            ' --data-raw \'{"foo": "bar"}\''
+            " -H 'Content-Type: application/json' -H 'Accept: application/json'"
+        )
+        self._test_request(request_object, expected_curl_command)
+
+    def test_cookies_dict(self):
+        request_object = Request(
+            "https://www.httpbin.org/post",
+            method="POST",
+            cookies={"foo": "bar"},
+            body=json.dumps({"foo": "bar"}),
+        )
+        expected_curl_command = (
+            "curl -X POST https://www.httpbin.org/post"
+            " --data-raw '{\"foo\": \"bar\"}' --cookie 'foo=bar'"
+        )
+        self._test_request(request_object, expected_curl_command)
+
+    def test_cookies_list(self):
+        request_object = Request(
+            "https://www.httpbin.org/post",
+            method="POST",
+            cookies=[{"foo": "bar"}],
+            body=json.dumps({"foo": "bar"}),
+        )
+        expected_curl_command = (
+            "curl -X POST https://www.httpbin.org/post"
+            " --data-raw '{\"foo\": \"bar\"}' --cookie 'foo=bar'"
+        )
+        self._test_request(request_object, expected_curl_command)
+
+
 if __name__ == "__main__":
    unittest.main()
--- a/tox.ini
+++ b/tox.ini
@ -30,7 +30,7 @@ passenv =
 #allow tox virtualenv to upgrade pip/wheel/setuptools
 download = true
 commands =
-    pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests}
+    pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests} --doctest-modules
 install_command =
    python -I -m pip install -ctests/upper-constraints.txt {opts} {packages}

@ -58,7 +58,7 @@ commands =
 basepython = python3
 deps =
    {[testenv:extra-deps]deps}
-    pylint==2.16.0
+    pylint==2.17.2
 commands =
    pylint conftest.py docs extras scrapy setup.py tests

@ -99,14 +99,18 @@ setenv =
    _SCRAPY_PINNED=true
 install_command =
    python -I -m pip install {opts} {packages}
+commands =
+    pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 scrapy tests}

 [testenv:pinned]
+basepython = python3.7
 deps =
    {[pinned]deps}
    PyDispatcher==2.0.5
 install_command = {[pinned]install_command}
 setenv =
    {[pinned]setenv}
+commands = {[pinned]commands}

 [testenv:windows-pinned]
 basepython = python3
@ -116,6 +120,7 @@ deps =
 install_command = {[pinned]install_command}
 setenv =
    {[pinned]setenv}
+commands = {[pinned]commands}

 [testenv:extra-deps]
 basepython = python3
@ -136,7 +141,7 @@ commands =

 [testenv:asyncio-pinned]
 deps = {[testenv:pinned]deps}
-commands = {[testenv:asyncio]commands}
+commands = {[pinned]commands} --reactor=asyncio
 install_command = {[pinned]install_command}
 setenv =
    {[pinned]setenv}
@ -151,7 +156,8 @@ basepython = {[testenv:pypy3]basepython}
 deps =
    {[pinned]deps}
    PyPyDispatcher==2.1.0
-commands = {[testenv:pypy3]commands}
+commands =
+    pytest --durations=10 scrapy tests
 install_command = {[pinned]install_command}
 setenv =
    {[pinned]setenv}
 @ -1 +1 @@
 .8.0
 .9.0