Tests for extensions.debug and extensions.memdebug. (#7749)

* Tests for extensions.debug and extensions.memdebug.

* Use a unique class in test_crawl_sets_stats.

* Don't assume a single log message.

.github/workflows/tests-ubuntu.yml

@@ -79,9 +79,6 @@ jobs:
         - python-version: "3.14"
           env:
             TOXENV: botocore
-        - python-version: "3.14"
-          env:
-            TOXENV: mitmproxy
 
     steps:
     - uses: actions/checkout@v6
@@ -97,6 +94,9 @@ jobs:
         sudo apt-get update
         sudo apt-get install libxml2-dev libxslt-dev
 
+    - name: Install mitmproxy
+      run: pipx install mitmproxy
+
     - name: Run tests
       env: ${{ matrix.env }}
       run: |

.pre-commit-config.yaml

@@ -6,7 +6,7 @@ exclude: |
   )
 repos:
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.15.2
+  rev: v0.15.20
   hooks:
     - id: ruff-check
       args: [ --fix ]
@@ -16,7 +16,7 @@ repos:
   hooks:
   - id: blacken-docs
     additional_dependencies:
-    - black==25.9.0
+    - black==26.5.1
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v6.0.0
   hooks:

conftest.py

@@ -11,7 +11,7 @@ from scrapy.utils.reactor import set_asyncio_event_loop_policy
 from scrapy.utils.reactorless import install_reactor_import_hook
 from tests.keys import generate_keys
 from tests.mockserver.http import MockServer
-from tests.mockserver.mitm_proxy import MitmProxy
+from tests.mockserver.mitm_proxy import MitmProxy, mitmdump_cmd
 
 if TYPE_CHECKING:
     from collections.abc import Generator
@@ -127,7 +127,6 @@ def pytest_runtest_setup(item):
         "uvloop",
         "botocore",
         "boto3",
-        "mitmproxy",
     ]
 
     for module in optional_deps:
@@ -137,6 +136,9 @@ def pytest_runtest_setup(item):
             except ImportError:
                 pytest.skip(f"{module} is not installed")
 
+    if item.get_closest_marker("requires_mitmproxy") and mitmdump_cmd() is None:
+        pytest.skip("mitmdump is not available")
+
 
 # Generate localhost certificate files, needed by some tests
 generate_keys()

docs/README.rst

@@ -1,68 +0,0 @@
-:orphan:
-
-======================================
-Scrapy documentation quick start guide
-======================================
-
-This file provides a quick guide on how to compile the Scrapy documentation.
-
-
-Setup the environment
----------------------
-
-To compile the documentation you need Sphinx Python library. To install it
-and all its dependencies run the following command from this dir
-
-::
-
-    pip install -r requirements.txt
-
-
-Compile the documentation
--------------------------
-
-To compile the documentation (to classic HTML output) run the following command
-from this dir::
-
-    make html
-
-Documentation will be generated (in HTML format) inside the ``build/html`` dir.
-
-
-View the documentation
-----------------------
-
-To view the documentation run the following command::
-
-    make htmlview
-
-This command will fire up your default browser and open the main page of your
-(previously generated) HTML documentation.
-
-
-Start over
-----------
-
-To clean up all generated documentation files and start from scratch run::
-
-    make clean
-
-Keep in mind that this command won't touch any documentation source files.
-
-
-Recreating documentation on the fly
------------------------------------
-
-There is a way to recreate the doc automatically when you make changes, you
-need to install watchdog (``pip install watchdog``) and then use::
-
-    make watch
-
-Alternative method using tox
-----------------------------
-
-To compile the documentation to HTML run the following command::
-
-    tox -e docs
-
-Documentation will be generated inside the ``docs/_build/all`` dir.

docs/contributing.rst

@@ -323,9 +323,10 @@ deprecation removals are documented in the :ref:`release notes <news>`.
 Tests
 =====
 
-Tests are implemented using the :doc:`Twisted unit-testing framework
-<twisted:development/test-standard>`. Running tests requires
-:doc:`tox <tox:index>`.
+Tests are implemented using pytest_. Running tests requires :doc:`tox
+<tox:index>`.
+
+.. _pytest: https://pytest.org
 
 .. _running-tests:
 
@@ -371,6 +372,21 @@ To see coverage report install :doc:`coverage <coverage:index>`
 
 see output of ``coverage --help`` for more options like html or xml report.
 
+Some tests need a ``mitmdump`` executable (from mitmproxy_) to test against a
+fully featured proxy server; they are skipped when one cannot be found
+(``mitmproxy`` is intentionally not a test dependency that would be installed
+into test venvs, as that sometimes leads to various dependency conflicts).
+To run these tests, make ``mitmdump`` available in one of these ways:
+
+* install ``mitmproxy`` so that ``mitmdump`` is on your ``PATH``, e.g. with
+  pipx_ (``pipx install mitmproxy``) or uv_ (``uv tool install mitmproxy``);
+
+* have uv_ installed, in which case the tests will run
+  ``uvx --from mitmproxy mitmdump``;
+
+* set the ``MITMDUMP`` environment variable to the path of a ``mitmdump``
+  executable.
+
 Writing tests
 -------------
 
@@ -398,3 +414,6 @@ And their unit-tests are in::
 .. _pytest-xdist: https://github.com/pytest-dev/pytest-xdist
 .. _help wanted issues: https://github.com/scrapy/scrapy/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
 .. _test coverage: https://app.codecov.io/gh/scrapy/scrapy
+.. _mitmproxy: https://mitmproxy.org/
+.. _pipx: https://pipx.pypa.io/
+.. _uv: https://docs.astral.sh/uv/

docs/faq.rst

@@ -285,7 +285,8 @@ consume a lot of memory.
 In order to avoid parsing all the entire feed at once in memory, you can use
 the :func:`~scrapy.utils.iterators.xmliter_lxml` and
 :func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what
-:class:`~scrapy.spiders.XMLFeedSpider` uses.
+:class:`~scrapy.spiders.XMLFeedSpider` and
+:class:`~scrapy.spiders.CSVFeedSpider` use.
 
 .. autofunction:: scrapy.utils.iterators.xmliter_lxml
 
@@ -360,10 +361,11 @@ method for this purpose. For example:
         def process_spider_output(self, response, result):
             for item_or_request in result:
                 if isinstance(item_or_request, Request):
+                    yield item_or_request
                     continue
-                adapter = ItemAdapter(item)
+                adapter = ItemAdapter(item_or_request)
                 for _ in range(adapter["multiply_by"]):
-                    yield deepcopy(item)
+                    yield deepcopy(item_or_request)
 
 Does Scrapy support IPv6 addresses?
 -----------------------------------
@@ -382,8 +384,9 @@ How to deal with ``<class 'ValueError'>: filedescriptor out of range in select()
 ----------------------------------------------------------------------------------------------
 
 This issue `has been reported`_ to appear when running broad crawls in macOS, where the default
-Twisted reactor is :class:`twisted.internet.selectreactor.SelectReactor`. Switching to a
-different reactor is possible by using the :setting:`TWISTED_REACTOR` setting.
+Twisted reactor was :class:`twisted.internet.selectreactor.SelectReactor` at that time.
+If you have switched to this reactor using the :setting:`TWISTED_REACTOR` setting you can switch
+to a different one in the same way.
 
 
 .. _faq-stop-response-download:
@@ -409,7 +412,6 @@ How can I make a blank request?
 
     from scrapy import Request
 
-
     blank_request = Request("data:,")
 
 In this case, the URL is set to a data URI scheme. Data URLs allow you to include data

docs/index.rst

@@ -24,7 +24,7 @@ Having trouble? We'd like to help!
 * Ask or search questions in `StackOverflow using the scrapy tag`_.
 * Ask or search questions in the `Scrapy subreddit`_.
 * Search for questions on the archives of the `scrapy-users mailing list`_.
-* Ask a question in the `#scrapy IRC channel`_,
+* Ask a question in the `#scrapy IRC channel`_.
 * Report bugs with Scrapy in our `issue tracker`_.
 * Join the Discord community `Scrapy Discord`_.
 
@@ -91,15 +91,15 @@ Basic concepts
 :doc:`topics/selectors`
     Extract the data from web pages using XPath.
 
-:doc:`topics/shell`
-    Test your extraction code in an interactive environment.
-
 :doc:`topics/items`
     Define the data you want to scrape.
 
 :doc:`topics/loaders`
     Populate your items with the extracted data.
 
+:doc:`topics/shell`
+    Test your extraction code in an interactive environment.
+
 :doc:`topics/item-pipeline`
     Post-process and store your scraped data.
 

docs/intro/install.rst

@@ -230,8 +230,8 @@ Installing Scrapy with PyPy on Windows is not tested.
 You can check that Scrapy is installed correctly by running ``scrapy bench``.
 If this command gives errors such as
 ``TypeError: ... got 2 unexpected keyword arguments``, this means
-that setuptools was unable to pick up one PyPy-specific dependency.
-To fix this issue, run ``pip install 'PyPyDispatcher>=2.1.0'``.
+that the ``PyPyDispatcher`` dependency wasn't installed. To fix this issue, run
+``pip install 'PyPyDispatcher>=2.1.0'``.
 
 
 .. _intro-install-troubleshooting:

docs/intro/overview.rst

@@ -83,16 +83,17 @@ While this enables you to do very fast crawls (sending multiple concurrent
 requests at the same time, in a fault-tolerant way) Scrapy also gives you
 control over the politeness of the crawl through :ref:`a few settings
 <topics-settings-ref>`. You can do things like setting a download delay between
-each request, limiting the amount of concurrent requests per domain or per IP, and
+each request, limiting the amount of concurrent requests per domain, and
 even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries
 to figure these settings out automatically.
 
 .. note::
 
     This is using :ref:`feed exports <topics-feed-exports>` to generate the
-    JSON file, you can easily change the export format (XML or CSV, for example) or the
-    storage backend (FTP or `Amazon S3`_, for example).  You can also write an
-    :ref:`item pipeline <topics-item-pipeline>` to store the items in a database.
+    JSON Lines file, you can easily change the export format (XML or CSV, for
+    example) or the storage backend (FTP or `Amazon S3`_, for example). You can
+    also write an :ref:`item pipeline <topics-item-pipeline>` to store the
+    items in a database.
 
 
 .. _topics-whatelse:

docs/news.rst

@@ -1035,7 +1035,7 @@ Deprecations
 New features
 ~~~~~~~~~~~~
 
--   Added a new setting, :setting:`REFERER_POLICIES`, to allow customizing
+-   Added a new setting, :setting:`REFERRER_POLICIES`, to allow customizing
     supported referrer policies.
 
 Bug fixes
@@ -6046,7 +6046,7 @@ The following changes may impact custom priority queue classes:
 
     *   A new keyword parameter has been added: ``key``. It is a string
         that is always an empty string for memory queues and indicates the
-        :setting:`JOB_DIR` value for disk queues.
+        :setting:`JOBDIR` value for disk queues.
 
     *   The parameter for disk queues that contains data from the previous
         crawl, ``startprios`` or ``slot_startprios``, is now passed as a
@@ -6600,7 +6600,7 @@ New features
     ``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be
     :ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significant
     scheduling improvement on crawls targeting multiple web domains, at the
-    cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)
+    cost of no ``CONCURRENT_REQUESTS_PER_IP`` support (:issue:`3520`)
 
 *   A new :attr:`.Request.cb_kwargs` attribute
     provides a cleaner way to pass keyword arguments to callback methods
@@ -9079,7 +9079,7 @@ New features and settings
 - In request errbacks, offending requests are now received in ``failure.request`` attribute (:rev:`2738`)
 - Big downloader refactoring to support per domain/ip concurrency limits (:rev:`2732`)
    - ``CONCURRENT_REQUESTS_PER_SPIDER`` setting has been deprecated and replaced by:
-      - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, :setting:`CONCURRENT_REQUESTS_PER_IP`
+      - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, ``CONCURRENT_REQUESTS_PER_IP``
    - check the documentation for more details
 - Added builtin caching DNS resolver (:rev:`2728`)
 - Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (:rev:`2706`, :rev:`2714`)

docs/topics/addons.rst

@@ -168,7 +168,6 @@ Use a fallback component:
 
     from scrapy.utils.misc import build_from_crawler, load_object
 
-
     FALLBACK_SETTING = "MY_FALLBACK_DOWNLOAD_HANDLER"
 
 

docs/topics/asyncio.rst

@@ -5,7 +5,7 @@ asyncio
 =======
 
 Scrapy supports :mod:`asyncio` natively. New projects created with
-:command:`scrapy startproject` have asyncio enabled by default, and you can use
+:command:`startproject` have asyncio enabled by default, and you can use
 :mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine
 <coroutines>`.
 
@@ -18,7 +18,7 @@ no additional setup is needed.
 Configuring the asyncio reactor
 ===============================
 
-New projects generated with :command:`scrapy startproject` have the asyncio
+New projects generated with :command:`startproject` have the asyncio
 reactor configured by default. No manual setup is needed.
 
 The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy
@@ -105,6 +105,9 @@ Scrapy API requires passing a Deferred to it) using the following helpers:
 
 .. autofunction:: scrapy.utils.defer.deferred_from_coro
 .. autofunction:: scrapy.utils.defer.deferred_f_from_coro_f
+
+The following function helps with a reverse wrapping:
+
 .. autofunction:: scrapy.utils.defer.ensure_awaitable
 
 
@@ -190,7 +193,7 @@ in future Scrapy versions. The following features are not available:
   :class:`~scrapy.crawler.CrawlerProcess`
   (:class:`~scrapy.crawler.AsyncCrawlerProcess` and
   :class:`~scrapy.crawler.AsyncCrawlerRunner` are available)
-* Twisted-specific DNS resolvers (the :setting:`DNS_RESOLVER` setting)
+* Twisted-specific DNS resolvers (the :setting:`TWISTED_DNS_RESOLVER` setting)
 * User and 3rd-party code that requires a reactor (see :ref:`below
   <asyncio-without-reactor-migrate>` for examples)
 
@@ -315,8 +318,7 @@ implementations, :class:`~asyncio.ProactorEventLoop` (default) and
 :class:`~asyncio.SelectorEventLoop` works with Twisted.
 
 Scrapy changes the event loop class to :class:`~asyncio.SelectorEventLoop`
-automatically when you change the :setting:`TWISTED_REACTOR` setting or call
-:func:`~scrapy.utils.reactor.install_reactor`.
+automatically when installing the asyncio reactor.
 
 .. note:: Other libraries you use may require
           :class:`~asyncio.ProactorEventLoop`, e.g. because it supports

docs/topics/autothrottle.rst

@@ -75,7 +75,7 @@ AutoThrottle algorithm adjusts download delays based on the following rules:
 .. _download-latency:
 
 In Scrapy, the download latency is measured as the time elapsed between
-establishing the TCP connection and receiving the HTTP headers.
+sending the request and receiving the HTTP headers.
 
 Note that these latencies are very hard to measure accurately in a cooperative
 multitasking environment because Scrapy may be busy processing a spider

docs/topics/commands.rst

@@ -199,6 +199,7 @@ Global commands:
 * :command:`fetch`
 * :command:`view`
 * :command:`version`
+* :command:`bench`
 
 Project-only commands:
 
@@ -207,7 +208,6 @@ Project-only commands:
 * :command:`list`
 * :command:`edit`
 * :command:`parse`
-* :command:`bench`
 
 .. command:: startproject
 
@@ -309,11 +309,25 @@ Usage examples::
       * parse_item
 
     $ scrapy check
-    [FAILED] first_spider:parse_item
-    >>> 'RetailPricex' field is missing
+    F.F.
+    ======================================================================
+    FAIL: [first_spider] parse (@returns post-hook)
+    ----------------------------------------------------------------------
+    Traceback (most recent call last):
+      ...
+    scrapy.exceptions.ContractFail: Returned 92 requests, expected 0..4
 
-    [FAILED] first_spider:parse
-    >>> Returned 92 requests, expected 0..4
+    ======================================================================
+    FAIL: [first_spider] parse_item (@scrapes post-hook)
+    ----------------------------------------------------------------------
+    Traceback (most recent call last):
+      ...
+    scrapy.exceptions.ContractFail: Missing fields: RetailPricex
+
+    ----------------------------------------------------------------------
+    Ran 4 contracts in 0.174s
+
+    FAILED (failures=2)
 
 .. skip: end
 
@@ -377,7 +391,7 @@ Supported options:
 
 * ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider
 
-* ``--headers``: print the response's HTTP headers instead of the response's body
+* ``--headers``: print the request's and response's HTTP headers instead of the response's body
 
 * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them)
 
@@ -387,15 +401,19 @@ Usage examples::
     [ ... html content here ... ]
 
     $ scrapy fetch --nolog --headers http://www.example.com/
-    {'Accept-Ranges': ['bytes'],
-     'Age': ['1263   '],
-     'Connection': ['close     '],
-     'Content-Length': ['596'],
-     'Content-Type': ['text/html; charset=UTF-8'],
-     'Date': ['Wed, 18 Aug 2010 23:59:46 GMT'],
-     'Etag': ['"573c1-254-48c9c87349680"'],
-     'Last-Modified': ['Fri, 30 Jul 2010 15:30:18 GMT'],
-     'Server': ['Apache/2.2.3 (CentOS)']}
+    > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
+    > Accept-Language: en
+    > User-Agent: Scrapy/2.16.0 (+https://scrapy.org)
+    > Accept-Encoding: gzip, deflate, br
+    >
+    < Date: Wed, 08 Jul 2026 06:15:01 GMT
+    < Content-Type: text/html
+    < Server: cloudflare
+    < Last-Modified: Wed, 01 Jul 2026 17:50:18 GMT
+    < Allow: GET, HEAD
+    < Cf-Cache-Status: HIT
+    < Age: 8184
+    < Cf-Ray: a17cf3b80eddf141-DME
 
 .. command:: view
 
@@ -476,7 +494,7 @@ Supported options:
 
 * ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider
 
-* ``--a NAME=VALUE``: set spider argument (may be repeated)
+* ``-a NAME=VALUE``: set spider argument (may be repeated)
 
 * ``--callback`` or ``-c``: spider method to use as callback for parsing the
   response
@@ -605,7 +623,10 @@ shouldn't matter to the user running the command, but when the user :ref:`needs
 a non-default Twisted reactor <disable-asyncio>`, it may be important.
 
 Scrapy decides which of these two classes to use based on the value of the
-:setting:`TWISTED_REACTOR` setting. If the setting value is the default one
+:setting:`TWISTED_REACTOR` and :setting:`TWISTED_REACTOR_ENABLED` settings.
+With :setting:`TWISTED_REACTOR_ENABLED` set to ``False`` it will use
+:class:`~scrapy.crawler.AsyncCrawlerProcess`. Otherwise, if the
+:setting:`TWISTED_REACTOR` value is the default one
 (``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``),
 :class:`~scrapy.crawler.AsyncCrawlerProcess` will be used, otherwise
 :class:`~scrapy.crawler.CrawlerProcess` will be used. The :ref:`spider settings

docs/topics/coroutines.rst

@@ -16,7 +16,7 @@ Supported callables
 The following callables may be defined as coroutines using ``async def``, and
 hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
--   The :meth:`~scrapy.spiders.Spider.start` spider method, which *must* be
+-   The :meth:`~scrapy.Spider.start` spider method, which *must* be
     defined as an :term:`asynchronous generator`.
 
     .. versionadded:: 2.13
@@ -204,13 +204,15 @@ This means you can use many useful Python libraries providing such code:
 Common use cases for asynchronous code include:
 
 * requesting data from websites, databases and other services (in
-  :meth:`~scrapy.spiders.Spider.start`, callbacks, pipelines and
+  :meth:`~scrapy.Spider.start`, callbacks, pipelines and
   middlewares);
 * storing data in databases (in pipelines and middlewares);
 * delaying the spider initialization until some external event (in the
   :signal:`spider_opened` handler);
-* calling asynchronous Scrapy methods like :meth:`ExecutionEngine.download`
-  (see :ref:`the screenshot pipeline example<ScreenshotPipeline>`).
+* calling asynchronous Scrapy methods like
+  :meth:`ExecutionEngine.download_async()
+  <scrapy.core.engine.ExecutionEngine.download_async>` (see :ref:`the
+  screenshot pipeline example <ScreenshotPipeline>`).
 
 .. _aio-libs: https://github.com/aio-libs
 
@@ -268,5 +270,5 @@ You can also send multiple requests in parallel:
             yield {
                 "h1": response.css("h1::text").get(),
                 "price": responses[0].css(".price::text").get(),
-                "price2": responses[1].css(".color::text").get(),
+                "color": responses[1].css(".color::text").get(),
             }

docs/topics/download-handlers.rst

@@ -89,7 +89,7 @@ the following API:
         If ``True``, the handler will only be instantiated when the first
         request handled by it needs to be downloaded.
 
-    .. method:: download_request(request: Request) -> Response:
+    .. method:: download_request(request: Request) -> Response
         :async:
 
         Download the given request and return a response.

docs/topics/downloader-middleware.rst

@@ -505,7 +505,7 @@ Filesystem storage backend (default)
 
     *   ``response_body`` - the plain response body
 
-    *   ``response_headers`` - the request headers (in raw HTTP format)
+    *   ``response_headers`` - the response headers (in raw HTTP format)
 
     *   ``meta`` - some metadata of this cache resource in Python ``repr()``
         format (grep-friendly format)
@@ -547,7 +547,7 @@ defines the methods described below.
     .. method:: open_spider(spider)
 
       This method gets called after a spider has been opened for crawling. It handles
-      the :signal:`open_spider <spider_opened>` signal.
+      the :signal:`spider_opened` signal.
 
       :param spider: the spider which has been opened
       :type spider: :class:`~scrapy.Spider` object
@@ -555,7 +555,7 @@ defines the methods described below.
     .. method:: close_spider(spider)
 
       This method gets called after a spider has been closed. It handles
-      the :signal:`close_spider <spider_closed>` signal.
+      the :signal:`spider_closed` signal.
 
       :param spider: the spider which has been closed
       :type spider: :class:`~scrapy.Spider` object
@@ -591,8 +591,8 @@ In order to use your storage backend, set:
 HTTPCache middleware settings
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The :class:`HttpCacheMiddleware` can be configured through the following
-settings:
+:class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` can be
+configured through the following settings:
 
 .. setting:: HTTPCACHE_ENABLED
 
@@ -814,7 +814,6 @@ HttpProxyMiddleware settings
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. setting:: HTTPPROXY_ENABLED
-.. setting:: HTTPPROXY_AUTH_ENCODING
 
 HTTPPROXY_ENABLED
 ^^^^^^^^^^^^^^^^^
@@ -823,6 +822,8 @@ Default: ``True``
 
 Whether or not to enable the :class:`HttpProxyMiddleware`.
 
+.. setting:: HTTPPROXY_AUTH_ENCODING
+
 HTTPPROXY_AUTH_ENCODING
 ^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -867,9 +868,9 @@ OffsiteMiddleware
    .. reqmeta:: allow_offsite
 
    If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to
-   ``True`` or :attr:`Request.meta` has ``allow_offsite`` set to ``True``, then
-   the OffsiteMiddleware will allow the request even if its domain is not listed
-   in allowed domains.
+   ``True`` or :attr:`Request.meta <scrapy.Request.meta>` has ``allow_offsite``
+   set to ``True``, then the OffsiteMiddleware will allow the request even if
+   its domain is not listed in allowed domains.
 
 RedirectMiddleware
 ------------------
@@ -984,7 +985,7 @@ Whether the Meta Refresh middleware will be enabled.
 METAREFRESH_IGNORE_TAGS
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-Default: ``[]``
+Default: ``["noscript"]``
 
 Meta tags within these tags are ignored.
 
@@ -1014,17 +1015,6 @@ RetryMiddleware
    A middleware to retry failed requests that are potentially caused by
    temporary problems such as a connection timeout or HTTP 500 error.
 
-Failed pages are collected on the scraping process and rescheduled at the
-end, once the spider has finished crawling all regular (non failed) pages.
-
-The :class:`RetryMiddleware` can be configured through the following
-settings (see the settings documentation for more info):
-
-* :setting:`RETRY_ENABLED`
-* :setting:`RETRY_TIMES`
-* :setting:`RETRY_HTTP_CODES`
-* :setting:`RETRY_EXCEPTIONS`
-
 .. reqmeta:: dont_retry
 
 If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key
@@ -1091,7 +1081,7 @@ Default::
         'twisted.internet.error.ConnectionDone',
         'twisted.internet.error.ConnectError',
         'twisted.internet.error.ConnectionLost',
-        IOError,
+        OSError,
         'scrapy.core.downloader.handlers.http11.TunnelError',
     ]
 

docs/topics/dynamic-content.rst

@@ -133,7 +133,7 @@ data from it depends on the type of response:
 
     .. code-block:: python
 
-        selector = Selector(data["html"])
+        selector = Selector(text=data["html"])
 
 -   If the response is JavaScript, or HTML with a ``<script/>`` element
     containing the desired data, see :ref:`topics-parsing-javascript`.

docs/topics/exceptions.rst

@@ -12,7 +12,8 @@ Exceptions
 Built-in Exceptions reference
 =============================
 
-Here's a list of all exceptions included in Scrapy and their usage.
+Here's a list of all exceptions included in Scrapy and their usage, except for
+the :ref:`download handler exceptions <download-handlers-exceptions>`.
 
 
 CloseSpider
@@ -31,7 +32,7 @@ For example:
 .. code-block:: python
 
     def parse_page(self, response):
-        if "Bandwidth exceeded" in response.body:
+        if "Bandwidth exceeded" in response.text:
             raise CloseSpider("bandwidth_exceeded")
 
 DontCloseSpider
@@ -71,7 +72,8 @@ remain disabled. Those components include:
 -   Downloader middlewares
 -   Spider middlewares
 
-The exception must be raised in the component's ``__init__`` method.
+The exception must be raised in the component's ``__init__()`` or
+``from_crawler()``  method.
 
 NotSupported
 ------------

docs/topics/exporters.rst

@@ -153,7 +153,7 @@ output examples, which assume you're exporting these two items:
 BaseItemExporter
 ----------------
 
-.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding='utf-8', indent=0, dont_fail=False)
+.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding=None, indent=None, dont_fail=False)
 
    This is the (abstract) base class for all Item Exporters. It provides
    support for common features used by all (concrete) Item Exporters, such as
@@ -239,7 +239,7 @@ BaseItemExporter
 
    .. attribute:: indent
 
-      Amount of spaces used to indent the output on each level. Defaults to ``0``.
+      Amount of spaces used to indent the output on each level. Defaults to ``None``.
 
       * ``indent=None`` selects the most compact representation,
         all items in the same line with no indentation
@@ -328,7 +328,7 @@ CsvItemExporter
 
    :param join_multivalued: The char (or chars) that will be used for joining
       multi-valued fields, if found.
-   :type include_headers_line: str
+   :type join_multivalued: str
 
    :param errors: The optional string that specifies how encoding and decoding
       errors are to be handled. For more information see
@@ -342,14 +342,14 @@ CsvItemExporter
 
    A typical output of this exporter would be::
 
-      product,price
+      name,price
       Color TV,1200
       DVD player,200
 
 PickleItemExporter
 ------------------
 
-.. class:: PickleItemExporter(file, protocol=0, **kwargs)
+.. class:: PickleItemExporter(file, protocol=4, **kwargs)
 
    Exports items in pickle format to the given file-like object.
 

docs/topics/extensions.rst

@@ -341,9 +341,6 @@ closing the spider. If the spider generates more than that number of errors,
 it will be closed with the reason ``closespider_errorcount``. If zero (or non
 set), spiders won't be closed by number of errors.
 
-.. module:: scrapy.extensions.debug
-   :synopsis: Extensions for debugging Scrapy
-
 .. module:: scrapy.extensions.periodic_log
    :synopsis: Periodic stats logging
 
@@ -418,7 +415,7 @@ Example extension configuration:
     custom_settings = {
         "LOG_LEVEL": "INFO",
         "PERIODIC_LOG_STATS": {
-            "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],
+            "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count"],
         },
         "PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
         "PERIODIC_LOG_TIMING_ENABLED": True,
@@ -463,6 +460,9 @@ Default: ``False``
 Debugging extensions
 --------------------
 
+.. module:: scrapy.extensions.debug
+   :synopsis: Extensions for debugging Scrapy
+
 Stack trace dump extension
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/topics/feed-exports.rst

@@ -143,6 +143,11 @@ Here are some examples to illustrate:
 .. note:: :ref:`Spider arguments <spiderargs>` become spider attributes, hence
           they can also be used as storage URI parameters.
 
+.. note:: Only ``%(...)s`` parameters are replaced. Any other percent
+          character is kept as-is, so percent-encoded URIs (e.g. ``%20`` for a
+          space or percent-encoded FTP credentials) and :class:`pathlib.Path`
+          keys containing ``%(...)s`` parameters both work as expected.
+
 
 .. _topics-feed-storage-backends:
 
@@ -161,7 +166,7 @@ The feeds are stored in the local filesystem.
 -   Required external libraries: none
 
 Note that for the local filesystem storage (only) you can omit the scheme if
-you specify an absolute path like ``/tmp/export.csv`` (Unix systems only).
+you specify a path (e.g. ``/tmp/export.csv``).
 Alternatively you can also use a :class:`pathlib.Path` object.
 
 .. _topics-feed-storage-ftp:
@@ -615,6 +620,7 @@ Default:
         "file": "scrapy.extensions.feedexport.FileFeedStorage",
         "stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",
         "s3": "scrapy.extensions.feedexport.S3FeedStorage",
+        "gs": "scrapy.extensions.feedexport.GCSFeedStorage",
         "ftp": "scrapy.extensions.feedexport.FTPFeedStorage",
     }
 
@@ -756,8 +762,8 @@ The function signature should be as follows:
    :param spider: source spider of the feed items
    :type spider: scrapy.Spider
 
-   .. caution:: The function should return a new dictionary, modifying
-                the received ``params`` in-place is deprecated.
+   .. caution:: The function must return a new dictionary instead of modifying
+                the received ``params`` in-place.
 
 For example, to include the :attr:`name <scrapy.Spider.name>` of the
 source spider in the feed URI:

docs/topics/item-pipeline.rst

@@ -121,7 +121,7 @@ Write items to MongoDB
 
 In this example we'll write items to MongoDB_ using pymongo_.
 MongoDB address and database name are specified in Scrapy settings;
-MongoDB collection is named after item class.
+MongoDB collection is specified in a class attribute.
 
 The main point of this example is to show how to :ref:`get the crawler
 <from-crawler>` and how to clean up the resources properly.

docs/topics/items.rst

@@ -23,7 +23,8 @@ Item Types
 
 Scrapy supports the following types of items, via the `itemadapter`_ library:
 :ref:`dictionaries <dict-items>`, :ref:`Item objects <item-objects>`,
-:ref:`dataclass objects <dataclass-items>`, and :ref:`attrs objects <attrs-items>`.
+:ref:`dataclass objects <dataclass-items>`, :ref:`attrs objects <attrs-items>`
+and :ref:`Pydantic models <pydantic-items>`.
 
 .. _itemadapter: https://github.com/scrapy/itemadapter
 
@@ -61,8 +62,8 @@ its ``__init__`` method.
 :class:`Item` also allows the defining of field metadata, which can be used to
 :ref:`customize serialization <topics-exporters-field-serialization>`.
 
-:mod:`trackref` tracks :class:`Item` objects to help find memory leaks
-(see :ref:`topics-leaks-trackrefs`).
+:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory
+leaks (see :ref:`topics-leaks-trackrefs`).
 
 Example:
 
@@ -262,7 +263,7 @@ Creating items
 
     >>> product = Product(name="Desktop PC", price=1000)
     >>> print(product)
-    Product(name='Desktop PC', price=1000)
+    {'name': 'Desktop PC', 'price': 1000}
 
 
 Getting field values
@@ -376,10 +377,12 @@ Creating dicts from items:
     >>> dict(product)  # create a dict from all populated values
     {'price': 1000, 'name': 'Desktop PC'}
 
-    Creating items from dicts:
+Creating items from dicts:
+
+.. code-block:: pycon
 
     >>> Product({"name": "Laptop PC", "price": 1500})
-    Product(price=1500, name='Laptop PC')
+    {'name': 'Laptop PC', 'price': 1500}
 
     >>> Product({"name": "Laptop PC", "lala": 1500})  # warning: unknown field in dict
     Traceback (most recent call last):

docs/topics/jobs.rst

@@ -151,8 +151,8 @@ Where:
 -   :class:`~scrapy.pqueues.ScrapyPriorityQueue` creates the ``{priority}{s?}``
     directories.
 
--   :class:`scrapy.squeues.PickleLifoDiskQueue`, a subclass of
-    :class:`queuelib.LifoDiskQueue` that uses :mod:`pickle` to serialize
+-   :class:`scrapy.squeues.PickleFifoDiskQueue`, a subclass of
+    :class:`queuelib.FifoDiskQueue` that uses :mod:`pickle` to serialize
     :class:`dict` representations of :class:`scrapy.Request` objects, creates
     the ``info.json`` and ``q{00000}`` files.
 

docs/topics/leaks.rst

@@ -62,9 +62,9 @@ Debugging memory leaks with ``trackref``
 
 .. skip: start
 
-:mod:`trackref` is a module provided by Scrapy to debug the most common cases of
-memory leaks. It basically tracks the references to all live Request,
-Response, Item, Spider and Selector objects.
+:mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most
+common cases of memory leaks. It basically tracks the references to all live
+Request, Response, Item, Spider and Selector objects.
 
 You can enter the telnet console and inspect how many objects (of the classes
 mentioned above) are currently alive using the ``prefs()`` function which is an
@@ -93,7 +93,7 @@ You can get the oldest object of each class using the
 Which objects are tracked?
 --------------------------
 
-The objects tracked by ``trackrefs`` are all from these classes (and all its
+The objects tracked by ``trackref`` are all from these classes (and all its
 subclasses):
 
 * :class:`scrapy.Request`
@@ -164,7 +164,7 @@ Too many spiders?
 -----------------
 
 If your project has too many spiders executed in parallel,
-the output of :func:`prefs` can be difficult to read.
+the output of ``prefs()`` can be difficult to read.
 For this reason, that function has a ``ignore`` argument which can be used to
 ignore a particular class (and all its subclasses). For
 example, this won't show any live references to spiders:
@@ -187,7 +187,7 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
     Inherit from this class if you want to track live
     instances with the ``trackref`` module.
 
-.. function:: print_live_refs(class_name, ignore=NoneType)
+.. function:: print_live_refs(ignore=NoneType)
 
     Print a report of live references, grouped by class name.
 
@@ -203,9 +203,9 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
 
 .. function:: iter_all(class_name)
 
-    Return an iterator over all objects alive with the given class name, or
-    ``None`` if none is found. Use :func:`print_live_refs` first to get a list
-    of all tracked live objects per class name.
+    Return an iterator over all objects alive with the given class name. Use
+    :func:`print_live_refs` first to get a list of all tracked live objects
+    per class name.
 
 .. skip: end
 

docs/topics/loaders.rst

@@ -76,7 +76,7 @@ data that will be assigned to the ``name`` field later.
 
 Afterwards, similar calls are used for ``price`` and ``stock`` fields
 (the latter using a CSS selector with the :meth:`~ItemLoader.add_css` method),
-and finally the ``last_update`` field is populated directly with a literal value
+and finally the ``last_updated`` field is populated directly with a literal value
 (``today``) using a different method: :meth:`~ItemLoader.add_value`.
 
 Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is
@@ -264,7 +264,7 @@ metadata. Here is an example:
     >>> il.add_value("name", ["Welcome to my", "<strong>website</strong>"])
     >>> il.add_value("price", ["&euro;", "<span>1000</span>"])
     >>> il.load_item()
-    {'name': 'Welcome to my website', 'price': '1000'}
+    Product(name='Welcome to my website', price='1000')
 
 .. skip: end
 
@@ -273,8 +273,8 @@ The precedence order, for both input and output processors, is as follows:
 1. Item Loader field-specific attributes: ``field_in`` and ``field_out`` (most
    precedence)
 2. Field metadata (``input_processor`` and ``output_processor`` key)
-3. Item Loader defaults: :meth:`ItemLoader.default_input_processor` and
-   :meth:`ItemLoader.default_output_processor` (least precedence)
+3. Item Loader defaults: :attr:`ItemLoader.default_input_processor` and
+   :attr:`ItemLoader.default_output_processor` (least precedence)
 
 See also: :ref:`topics-loaders-extending`.
 
@@ -323,8 +323,8 @@ There are several ways to modify Item Loader context values:
       loader = ItemLoader(product, unit="cm")
 
 3. On Item Loader declaration, for those input/output processors that support
-   instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of
-   them:
+   instantiating them with an Item Loader context.
+   :class:`~itemloaders.processors.MapCompose` is one of them:
 
    .. code-block:: python
 

docs/topics/logging.rst

@@ -4,11 +4,6 @@
 Logging
 =======
 
-.. note::
-    :mod:`scrapy.log` has been deprecated alongside its functions in favor of
-    explicit calls to the Python standard logging. Keep reading to learn more
-    about the new logging system.
-
 Scrapy uses :mod:`logging` for event logging. We'll
 provide some simple examples to get you started, but for more advanced
 use-cases it's strongly suggested to read thoroughly its documentation.

docs/topics/media-pipeline.rst

@@ -41,11 +41,10 @@ this:
 2. The item is returned from the spider and goes to the item pipeline.
 
 3. When the item reaches the :class:`FilesPipeline`, the URLs in the
-   ``file_urls`` field are scheduled for download using the standard
-   Scrapy scheduler and downloader (which means the scheduler and downloader
-   middlewares are reused), but with a higher priority, processing them before other
-   pages are scraped. The item remains "locked" at that particular pipeline stage
-   until the files have finish downloading (or fail for some reason).
+   ``file_urls`` field are downloaded using the standard Scrapy downloader
+   (which means the downloader middlewares are used, but the spider middlewares
+   aren't). The item remains "locked" at that particular pipeline stage until
+   the files have finished downloading (or failed for some reason).
 
 4. When the files are downloaded, another field (``files``) will be populated
    with the results. This field will contain a list of dicts with information
@@ -371,11 +370,12 @@ For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or
 If you need something more complex and want to override the custom pipeline
 behaviour, see :ref:`topics-media-pipeline-override`.
 
-If you have multiple image pipelines inheriting from ImagePipeline and you want
-to have different settings in different pipelines you can set setting keys
-preceded with uppercase name of your pipeline class. E.g. if your pipeline is
-called MyPipeline and you want to have custom IMAGES_URLS_FIELD you define
-setting MYPIPELINE_IMAGES_URLS_FIELD and your custom settings will be used.
+If you have multiple image pipelines inheriting from :class:`ImagesPipeline`
+and you want to have different settings in different pipelines you can set
+setting keys preceded with uppercase name of your pipeline class. E.g. if your
+pipeline is called ``MyPipeline`` and you want to have custom
+:setting:`IMAGES_URLS_FIELD` you define setting
+``MYPIPELINE_IMAGES_URLS_FIELD`` and your custom settings will be used.
 
 
 Additional features
@@ -547,10 +547,9 @@ See here the methods that you can override in your custom Files Pipeline:
 
    .. method:: FilesPipeline.get_media_requests(item, info)
 
-      As seen on the workflow, the pipeline will get the URLs of the images to
-      download from the item. In order to do this, you can override the
-      :meth:`~get_media_requests` method and return a Request for each
-      file URL:
+      As seen on the workflow, the pipeline will get the requests for the files
+      to download from the item by calling this method. You can override it to
+      change what requests are returned:
 
       .. code-block:: python
 
@@ -590,8 +589,9 @@ See here the methods that you can override in your custom Files Pipeline:
           * ``downloaded`` - file was downloaded.
           * ``uptodate`` - file was not downloaded, as it was downloaded recently,
             according to the file expiration policy.
-          * ``cached`` - file was already scheduled for download, by another item
-            sharing the same file.
+          * ``cached`` - file was taken from a cache (the response has a
+            ``"cached"`` flag, e.g. from
+            :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`).
 
       The list of tuples received by :meth:`~item_completed` is
       guaranteed to retain the same order of the requests returned from the
@@ -618,9 +618,6 @@ See here the methods that you can override in your custom Files Pipeline:
               (False, Failure(...)),
           ]
 
-      By default the :meth:`get_media_requests` method returns ``None`` which
-      means there are no files to download for the item.
-
    .. method:: FilesPipeline.item_completed(results, item, info)
 
       The :meth:`FilesPipeline.item_completed` method called when all file

docs/topics/practices.rst

@@ -17,8 +17,10 @@ Run Scrapy from a script
 You can use the :ref:`API <topics-api>` to run Scrapy from a script, instead of
 the typical way of running Scrapy via ``scrapy crawl``.
 
-Remember that Scrapy is built on top of the Twisted
-asynchronous networking library, so you need to run it inside the Twisted reactor.
+Remember that Scrapy requires a Twisted reactor or (with
+:setting:`TWISTED_REACTOR_ENABLED` set to ``False``) an asyncio event loop, so
+you need to run one of those in your script for it to work (helpers described
+below can do it for you).
 
 The first utility you can use to run your spiders is
 :class:`scrapy.crawler.AsyncCrawlerProcess` or

docs/topics/request-response.rst

@@ -247,7 +247,7 @@ Request objects
        Return a new Request which is a copy of this Request. See also:
        :ref:`topics-request-response-ref-request-callback-arguments`.
 
-    .. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs])
+    .. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs, cls])
 
        Return a Request object with the same members, except for those members
        given new values by whichever keyword arguments are specified. The
@@ -717,6 +717,7 @@ Those are:
 * :reqmeta:`download_fail_on_dataloss`
 * :reqmeta:`download_latency`
 * :reqmeta:`download_maxsize`
+* :reqmeta:`download_slot`
 * :reqmeta:`download_warnsize`
 * :reqmeta:`download_timeout`
 * ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
@@ -1085,7 +1086,7 @@ Response objects
     .. attribute:: Response.flags
 
         A list that contains flags for this response. Flags are labels used for
-        tagging Responses. For example: ``'cached'``, ``'redirected``', etc. And
+        tagging Responses. For example: ``'cached'``, ``'redirected'``', etc. And
         they're shown on the string representation of the Response (``__str__()``
         method) which is used by the engine for logging.
 
@@ -1100,8 +1101,8 @@ Response objects
 
         The IP address of the server from which the Response originated.
 
-        This attribute is currently only populated by the HTTP 1.1 download
-        handler, i.e. for ``http(s)`` responses. For other handlers,
+        This attribute is currently only populated by the HTTP download
+        handlers, i.e. for ``http(s)`` responses. For other handlers,
         :attr:`ip_address` is always ``None``.
 
     .. attribute:: Response.protocol
@@ -1119,7 +1120,7 @@ Response objects
 
        Returns a new Response which is a copy of this Response.
 
-    .. method:: Response.replace([url, status, headers, body, request, flags, cls])
+    .. method:: Response.replace([url, status, headers, body, request, flags, certificate, ip_address, protocol, cls])
 
        Returns a Response object with the same members, except for those members
        given new values by whichever keyword arguments are specified. The

docs/topics/selectors.rst

@@ -308,7 +308,7 @@ Examples:
 
 * ``*::text`` selects all descendant text nodes of the current selector context:
 
-..skip: next
+.. skip: next
 .. code-block:: pycon
 
     >>> response.css("#images *::text").getall()
@@ -634,8 +634,7 @@ Example:
 .. code-block:: pycon
 
     >>> from scrapy import Selector
-    >>> sel = Selector(
-    ...     text="""
+    >>> sel = Selector(text="""
     ...     <ul class="list">
     ...         <li>1</li>
     ...         <li>2</li>
@@ -645,8 +644,8 @@ Example:
     ...         <li>4</li>
     ...         <li>5</li>
     ...         <li>6</li>
-    ...     </ul>"""
-    ... )
+    ...     </ul>""")
+    ...
     >>> xp = lambda x: sel.xpath(x).getall()
 
 This gets all first ``<li>``  elements under whatever it is its parent:
@@ -948,11 +947,9 @@ with groups of itemscopes and corresponding itemprops:
     >>> sel = Selector(text=doc, type="html")
     >>> for scope in sel.xpath("//div[@itemscope]"):
     ...     print("current scope:", scope.xpath("@itemtype").getall())
-    ...     props = scope.xpath(
-    ...         """
+    ...     props = scope.xpath("""
     ...                 set:difference(./descendant::*/@itemprop,
-    ...                                .//*[@itemscope]/*/@itemprop)"""
-    ...     )
+    ...                                .//*[@itemscope]/*/@itemprop)""")
     ...     print(f"    properties: {props.getall()}")
     ...     print("")
     ...

docs/topics/settings.rst

@@ -409,6 +409,36 @@ Default: ``{}``
 A dict containing paths to the add-ons enabled in your project and their
 priorities. For more information, see :ref:`topics-addons`.
 
+.. setting:: ASYNCIO_EVENT_LOOP
+
+ASYNCIO_EVENT_LOOP
+------------------
+
+Default: ``None``
+
+Import path of a given ``asyncio`` event loop class.
+
+If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) or when
+:ref:`running Scrapy without a reactor <asyncio-without-reactor>` this setting
+can be used to specify the
+asyncio event loop to be used with it. Set the setting to the import path of the
+desired asyncio event loop class. If the setting is set to ``None`` the default asyncio
+event loop will be used.
+
+If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`
+function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop
+class to be used.
+
+Note that the event loop class must inherit from :class:`asyncio.AbstractEventLoop`.
+
+.. caution:: Please be aware that, when using a non-default event loop
+    (either defined via :setting:`ASYNCIO_EVENT_LOOP` or installed with
+    :func:`~scrapy.utils.reactor.install_reactor`), Scrapy will call
+    :func:`asyncio.set_event_loop`, which will set the specified event loop
+    as the current loop for the current OS thread.
+
+.. note:: This is a :ref:`reactor setting <reactor-settings>`.
+
 .. setting:: AWS_ACCESS_KEY_ID
 
 AWS_ACCESS_KEY_ID
@@ -419,6 +449,24 @@ Default: ``None``
 The AWS access key used by code that requires access to `Amazon Web services`_,
 such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
 
+.. setting:: AWS_ENDPOINT_URL
+
+AWS_ENDPOINT_URL
+----------------
+
+Default: ``None``
+
+Endpoint URL used for S3-like storage, for example Minio or s3.scality.
+
+.. setting:: AWS_REGION_NAME
+
+AWS_REGION_NAME
+---------------
+
+Default: ``None``
+
+The name of the region associated with the AWS client.
+
 .. setting:: AWS_SECRET_ACCESS_KEY
 
 AWS_SECRET_ACCESS_KEY
@@ -442,15 +490,6 @@ such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`, when using
 
 .. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
 
-.. setting:: AWS_ENDPOINT_URL
-
-AWS_ENDPOINT_URL
-----------------
-
-Default: ``None``
-
-Endpoint URL used for S3-like storage, for example Minio or s3.scality.
-
 .. setting:: AWS_USE_SSL
 
 AWS_USE_SSL
@@ -471,43 +510,6 @@ Default: ``None``
 Verify SSL connection between Scrapy and S3 or S3-like storage. By default
 SSL verification will occur.
 
-.. setting:: AWS_REGION_NAME
-
-AWS_REGION_NAME
----------------
-
-Default: ``None``
-
-The name of the region associated with the AWS client.
-
-.. setting:: ASYNCIO_EVENT_LOOP
-
-ASYNCIO_EVENT_LOOP
-------------------
-
-Default: ``None``
-
-Import path of a given ``asyncio`` event loop class.
-
-If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) this setting can be used to specify the
-asyncio event loop to be used with it. Set the setting to the import path of the
-desired asyncio event loop class. If the setting is set to ``None`` the default asyncio
-event loop will be used.
-
-If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`
-function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop
-class to be used.
-
-Note that the event loop class must inherit from :class:`asyncio.AbstractEventLoop`.
-
-.. caution:: Please be aware that, when using a non-default event loop
-    (either defined via :setting:`ASYNCIO_EVENT_LOOP` or installed with
-    :func:`~scrapy.utils.reactor.install_reactor`), Scrapy will call
-    :func:`asyncio.set_event_loop`, which will set the specified event loop
-    as the current loop for the current OS thread.
-
-.. note:: This is a :ref:`reactor setting <reactor-settings>`.
-
 .. setting:: BOT_NAME
 
 BOT_NAME
@@ -554,6 +556,8 @@ performed to any single domain.
 See also: :ref:`topics-autothrottle` and its
 :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` option.
 
+It is possible to change this setting per domain by using
+:setting:`DOWNLOAD_SLOTS`.
 
 .. setting:: DEFAULT_DROPITEM_LOG_LEVEL
 
@@ -593,7 +597,7 @@ When writing an item pipeline, you can force a different log level by setting
 DEFAULT_ITEM_CLASS
 ------------------
 
-Default: ``'scrapy.Item'``
+Default: ``'scrapy.item.Item'``
 
 The default class that will be used for instantiating items in the :ref:`the
 Scrapy shell <topics-shell>`.
@@ -687,7 +691,7 @@ Whether to enable DNS in-memory cache.
     :class:`~scrapy.resolver.CachingThreadedResolver` and
     :class:`~scrapy.resolver.CachingHostnameResolver`. It has no effect when
     :setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect
-    either when :setting:`DNS_RESOLVER` is set to a different resolver.
+    either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver.
 
 .. note:: This is a :ref:`reactor setting <reactor-settings>`.
 
@@ -702,25 +706,6 @@ DNS in-memory cache size, see :setting:`DNSCACHE_ENABLED`.
 
 .. note:: This is a :ref:`reactor setting <reactor-settings>`.
 
-.. setting:: TWISTED_DNS_RESOLVER
-
-TWISTED_DNS_RESOLVER
---------------------
-
-Default: ``'scrapy.resolver.CachingThreadedResolver'``
-
-The class to be used by Twisted to resolve DNS names. The default
-``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for
-DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4
-addresses. Scrapy provides an alternative resolver,
-``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not
-take the :setting:`DNS_TIMEOUT` setting into account.
-
-.. note::
-    This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
-
-.. note:: This is a :ref:`reactor setting <reactor-settings>`.
-
 .. setting:: DNS_TIMEOUT
 
 DNS_TIMEOUT
@@ -734,7 +719,7 @@ Timeout for processing of DNS queries in seconds. Float is supported.
     This setting is only used by
     :class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when
     :setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect
-    either when :setting:`DNS_RESOLVER` is set to a different resolver.
+    either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver.
 
 .. note:: This is a :ref:`reactor setting <reactor-settings>`.
 
@@ -936,9 +921,8 @@ desired.
 
     This delay can be set per spider using :attr:`download_delay` spider attribute.
 
-It is also possible to change this setting per domain, although it requires
-non-trivial code. See the implementation of the :ref:`AutoThrottle
-<topics-autothrottle>` extension for an example.
+It is possible to change this setting per domain by using
+:setting:`DOWNLOAD_SLOTS`.
 
 .. setting:: DOWNLOAD_BIND_ADDRESS
 
@@ -1330,6 +1314,7 @@ Default:
 
     {
         "scrapy.extensions.corestats.CoreStats": 0,
+        "scrapy.extensions.logcount.LogCount": 0,
         "scrapy.extensions.telnet.TelnetConsole": 0,
         "scrapy.extensions.memusage.MemoryUsage": 0,
         "scrapy.extensions.memdebug.MemoryDebugger": 0,
@@ -1352,6 +1337,8 @@ and the :ref:`list of available extensions <topics-extensions-ref>`.
 FEED_TEMPDIR
 ------------
 
+Default: ``None``
+
 The Feed Temp dir allows you to set a custom folder to save crawler
 temporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and
 :ref:`Amazon S3 <topics-feed-storage-s3>`.
@@ -1361,6 +1348,8 @@ temporary files before uploading with :ref:`FTP feed storage <topics-feed-storag
 FEED_STORAGE_GCS_ACL
 --------------------
 
+Default: ``""``
+
 The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
 For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://docs.cloud.google.com/storage/docs/access-control/lists>`_.
 
@@ -1372,14 +1361,19 @@ FORCE_CRAWLER_PROCESS
 Default: ``False``
 
 If ``False``, :ref:`Scrapy commands that need a CrawlerProcess
-<topics-commands-crawlerprocess>` will decide between using
+<topics-commands-crawlerprocess>`, when :setting:`TWISTED_REACTOR_ENABLED`
+is set to ``True``, will decide between using
 :class:`scrapy.crawler.AsyncCrawlerProcess` and
 :class:`scrapy.crawler.CrawlerProcess` based on the value of the
 :setting:`TWISTED_REACTOR` setting, but ignoring its value in :ref:`per-spider
 settings <spider-settings>`.
 
 If ``True``, these commands will always use
-:class:`~scrapy.crawler.CrawlerProcess`.
+:class:`~scrapy.crawler.CrawlerProcess` when :setting:`TWISTED_REACTOR_ENABLED`
+is set to ``True``.
+
+When :setting:`TWISTED_REACTOR_ENABLED` is set to ``False``,
+:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used in all cases.
 
 Set this to ``True`` if you want to set :setting:`TWISTED_REACTOR` to a
 non-default value in :ref:`per-spider settings <spider-settings>`.
@@ -1629,6 +1623,8 @@ The following special items are also supported:
 
 -   ``Python``
 
+-   ``pyOpenSSL``
+
 .. setting:: LOGSTATS_INTERVAL
 
 LOGSTATS_INTERVAL
@@ -1756,7 +1752,10 @@ significant similarities in the time between their requests.
 
 The randomization policy is the same used by `wget`_ ``--random-wait`` option.
 
-If :setting:`DOWNLOAD_DELAY` is zero (default) this option has no effect.
+If :setting:`DOWNLOAD_DELAY` is zero this option has no effect.
+
+It is possible to change this setting per domain by using
+:setting:`DOWNLOAD_SLOTS`.
 
 .. _wget: https://www.gnu.org/software/wget/manual/wget.html
 
@@ -1817,7 +1816,7 @@ The parser backend to use for parsing ``robots.txt`` files. For more information
 .. setting:: ROBOTSTXT_USER_AGENT
 
 ROBOTSTXT_USER_AGENT
-^^^^^^^^^^^^^^^^^^^^
+--------------------
 
 Default: ``None``
 
@@ -1972,6 +1971,8 @@ Default:
 
     {
         "scrapy.contracts.default.UrlContract": 1,
+        "scrapy.contracts.default.CallbackKeywordArgumentsContract": 1,
+        "scrapy.contracts.default.MetadataContract": 1,
         "scrapy.contracts.default.ReturnsContract": 2,
         "scrapy.contracts.default.ScrapesContract": 3,
     }
@@ -2036,6 +2037,7 @@ Default:
 .. code-block:: python
 
     {
+        "scrapy.spidermiddlewares.start.StartSpiderMiddleware": 25,
         "scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
         "scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
         "scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
@@ -2111,6 +2113,25 @@ command.
 The project name must not conflict with the name of custom files or directories
 in the ``project`` subdirectory.
 
+.. setting:: TWISTED_DNS_RESOLVER
+
+TWISTED_DNS_RESOLVER
+--------------------
+
+Default: ``'scrapy.resolver.CachingThreadedResolver'``
+
+The class to be used by Twisted to resolve DNS names. The default
+``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for
+DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4
+addresses. Scrapy provides an alternative resolver,
+``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not
+take the :setting:`DNS_TIMEOUT` setting into account.
+
+.. note::
+    This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
+
+.. note:: This is a :ref:`reactor setting <reactor-settings>`.
+
 .. setting:: TWISTED_REACTOR_ENABLED
 
 TWISTED_REACTOR_ENABLED
@@ -2149,6 +2170,9 @@ Default: ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``
 
 Import path of a given :mod:`~twisted.internet.reactor`.
 
+.. note::
+    This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
+
 Scrapy will install this reactor if no other reactor is installed yet, such as
 when the ``scrapy`` CLI program is invoked or when using the
 :class:`~scrapy.crawler.AsyncCrawlerProcess` class or the
@@ -2252,7 +2276,7 @@ URLLENGTH_LIMIT
 
 Default: ``2083``
 
-Scope: ``spidermiddlewares.urllength``
+Scope: ``scrapy.spidermiddlewares.urllength``
 
 The maximum URL length to allow for crawled URLs.
 

docs/topics/signals.rst

@@ -60,6 +60,8 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
 .. skip: next
 .. code-block:: python
 
+    import json
+
     import scrapy
     import treq
 
@@ -452,7 +454,7 @@ bytes_received
 .. signal:: bytes_received
 .. function:: bytes_received(data, request, spider)
 
-    Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
+    Sent by some download handlers when a group of bytes is
     received for a specific request. This signal might be fired multiple
     times for the same request, with partial data each time. For instance,
     a possible scenario for a 25 kb response would be two signals fired
@@ -480,7 +482,7 @@ headers_received
 .. signal:: headers_received
 .. function:: headers_received(headers, body_length, request, spider)
 
-    Sent by the HTTP 1.1 and S3 download handlers when the response headers are
+    Sent by some download handlers when the response headers are
     available for a given request, before downloading any additional content.
 
     Handlers for this signal can stop the download of a response while it

docs/topics/spider-middleware.rst

@@ -46,7 +46,7 @@ previous (or subsequent) middleware being applied.
 If you want to disable a builtin middleware (the ones defined in
 :setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
 in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its
-value.  For example, if you want to disable the off-site middleware:
+value.  For example, if you want to disable the referer middleware:
 
 .. code-block:: python
 

docs/topics/spiders.rst

@@ -208,12 +208,6 @@ scrapy.Spider
        :param response: the response to parse
        :type response: :class:`~scrapy.http.Response`
 
-   .. method:: log(message, [level, component])
-
-       Wrapper that sends a log message through the Spider's :attr:`logger`,
-       kept for backward compatibility. For more information see
-       :ref:`topics-logging-from-spiders`.
-
    .. method:: closed(reason)
 
        Called when the spider closes. This method provides a shortcut to
@@ -335,8 +329,8 @@ The above example can also be written as follows:
 
 If you are :ref:`running Scrapy from a script <run-from-script>`, you can
 specify spider arguments when calling
-:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
-:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
+:meth:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
+:meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
 
 .. skip: next
 .. code-block:: python
@@ -593,7 +587,7 @@ Let's now take a look at an example CrawlSpider with rules:
 This spider would start crawling example.com's home page, collecting category
 links, and item links, parsing the latter with the ``parse_item`` method. For
 each item response, some data will be extracted from the HTML using XPath, and
-an :class:`~scrapy.Item` will be filled with it.
+a dictionary will be filled with it.
 
 XMLFeedSpider
 -------------
@@ -614,7 +608,7 @@ XMLFeedSpider
 
         A string which defines the iterator to use. It can be either:
 
-           - ``'iternodes'`` - a fast iterator based on regular expressions
+           - ``'iternodes'`` - a fast iterator based on ``lxml``
 
            - ``'html'`` - an iterator which uses :class:`~scrapy.Selector`.
              Keep in mind this uses DOM parsing and must load all DOM in memory
@@ -953,6 +947,7 @@ Combine SitemapSpider with other sources of urls:
 
 .. code-block:: python
 
+    from scrapy import Request
     from scrapy.spiders import SitemapSpider
 
 

docs/topics/stats.rst

@@ -10,8 +10,8 @@ Collector, and can be accessed through the :attr:`~scrapy.crawler.Crawler.stats`
 attribute of the :ref:`topics-api-crawler`, as illustrated by the examples in
 the :ref:`topics-stats-usecases` section below.
 
-However, the Stats Collector is always available, so you can always import it
-in your module and use its API (to increment or set new stat keys), regardless
+The Stats Collector API is always available, so you can always use it (to
+increment or set new stat keys), regardless
 of whether the stats collection is enabled or not. If it's disabled, the API
 will still work but it won't collect anything. This is aimed at simplifying the
 stats collector usage: you should spend no more than one line of code for
@@ -21,9 +21,6 @@ using the Stats Collector from.
 Another feature of the Stats Collector is that it's very efficient (when
 enabled) and extremely efficient (almost unnoticeable) when disabled.
 
-The Stats Collector keeps a stats table per open spider which is automatically
-opened when the spider is opened, and closed when the spider is closed.
-
 .. _topics-stats-usecases:
 
 Common Stats Collector uses
@@ -87,13 +84,13 @@ Get all stats:
 Available Stats Collectors
 ==========================
 
+.. currentmodule:: scrapy.statscollectors
+
 Besides the basic :class:`StatsCollector` there are other Stats Collectors
 available in Scrapy which extend the basic Stats Collector. You can select
 which Stats Collector to use through the :setting:`STATS_CLASS` setting. The
 default Stats Collector used is the :class:`MemoryStatsCollector`.
 
-.. currentmodule:: scrapy.statscollectors
-
 MemoryStatsCollector
 --------------------
 
@@ -102,7 +99,7 @@ MemoryStatsCollector
     A simple stats collector that keeps the stats of the last scraping run (for
     each spider) in memory, after they're closed. The stats can be accessed
     through the :attr:`spider_stats` attribute, which is a dict keyed by spider
-    domain name.
+    name.
 
     This is the default Stats Collector used in Scrapy.
 

docs/topics/telnetconsole.rst

@@ -36,9 +36,9 @@ disable it if you want. For more information about the extension itself see
 How to access the telnet console
 ================================
 
-The telnet console listens in the TCP port defined in the
-:setting:`TELNETCONSOLE_PORT` setting, which defaults to ``6023``. To access
-the console you need to type::
+The telnet console listens on the first available TCP port from the range
+defined in the :setting:`TELNETCONSOLE_PORT` setting, which defaults to
+``[6023, 6073]``. To access the console you need to type::
 
     telnet localhost 6023
     Trying localhost...
@@ -107,8 +107,8 @@ Here are some example tasks you can do with the telnet console:
 View engine status
 ------------------
 
-You can use the ``est()`` method of the Scrapy engine to quickly show its state
-using the telnet console::
+You can use the ``est()`` method provided by the console to quickly show the
+engine status::
 
     telnet localhost 6023
     >>> est()

docs/versioning.rst

@@ -39,8 +39,8 @@ API stability
 
 API stability was one of the major goals for the *1.0* release.
 
-Methods or functions that start with a single dash (``_``) are private and
-should never be relied as stable.
+Methods or functions that start with a single underscore (``_``) are private
+and should never be relied upon as stable.
 
 Also, keep in mind that stable doesn't mean complete: stable APIs could grow
 new methods or functionality but the existing methods should keep working the

pyproject.toml

@@ -94,7 +94,82 @@ untyped_calls_exclude = [
 [[tool.mypy.overrides]]
 module = "tests.*"
 allow_untyped_defs = true
-allow_incomplete_defs = true # 48 errors
+allow_incomplete_defs = true # 59 errors
+
+# TODO
+[[tool.mypy.overrides]]
+module = [
+    "tests.mockserver.*",
+    "tests.spiders",
+    "tests.test_closespider",
+    "tests.test_cmdline",
+    "tests.test_contracts",
+    "tests.test_core_downloader",
+    "tests.test_downloader_handler_twisted_ftp",
+    "tests.test_downloadermiddleware_cookies",
+    "tests.test_downloadermiddleware_httpauth",
+    "tests.test_downloadermiddleware_httpcache",
+    "tests.test_downloadermiddleware_httpcompression",
+    "tests.test_downloadermiddleware_httpproxy",
+    "tests.test_downloadermiddleware_offsite",
+    "tests.test_downloadermiddleware_redirect",
+    "tests.test_downloadermiddleware_redirect_base",
+    "tests.test_downloadermiddleware_redirect_metarefresh",
+    "tests.test_downloadermiddleware_retry",
+    "tests.test_downloadermiddleware_robotstxt",
+    "tests.test_downloadermiddleware_stats",
+    "tests.test_downloaderslotssettings",
+    "tests.test_dupefilters",
+    "tests.test_engine_loop",
+    "tests.test_exporters",
+    "tests.test_extension_statsmailer",
+    "tests.test_extension_throttle",
+    "tests.test_feedexport",
+    "tests.test_feedexport_postprocess",
+    "tests.test_feedexport_storages",
+    "tests.test_feedexport_uri_params",
+    "tests.test_http2_client_protocol",
+    "tests.test_http_headers",
+    "tests.test_http_request",
+    "tests.test_http_request_form",
+    "tests.test_http_response",
+    "tests.test_http_response_text",
+    "tests.test_item",
+    "tests.test_link",
+    "tests.test_linkextractors",
+    "tests.test_loader",
+    "tests.test_logformatter",
+    "tests.test_logstats",
+    "tests.test_mail",
+    "tests.test_pipeline_crawl",
+    "tests.test_pipeline_files",
+    "tests.test_pipeline_images",
+    "tests.test_pipeline_media",
+    "tests.test_pipelines",
+    "tests.test_pqueues",
+    "tests.test_request_attribute_binding",
+    "tests.test_request_cb_kwargs",
+    "tests.test_request_dict",
+    "tests.test_request_left",
+    "tests.test_robotstxt_interface",
+    "tests.test_scheduler_base",
+    "tests.test_settings",
+    "tests.test_spider",
+    "tests.test_spider_crawl",
+    "tests.test_spidermiddleware_output_chain",
+    "tests.test_spidermiddleware_process_start",
+    "tests.test_spider_sitemap",
+    "tests.test_squeues",
+    "tests.test_squeues_request",
+    "tests.test_stats",
+    "tests.test_utils_datatypes",
+    "tests.test_utils_decorators",
+    "tests.test_utils_defer",
+    "tests.test_utils_deprecate",
+    "tests.test_utils_misc.test_return_with_argument_inside_generator",
+    "tests.test_utils_python",
+    "tests.test_utils_request",
+]
 check_untyped_defs = false
 
 # Interface classes are hard to support
@@ -269,7 +344,7 @@ markers = [
     "requires_uvloop: marks tests as only enabled when uvloop is known to be working",
     "requires_botocore: marks tests that need botocore (but not boto3)",
     "requires_boto3: marks tests that need botocore and boto3",
-    "requires_mitmproxy: marks tests that need mitmproxy",
+    "requires_mitmproxy: marks tests that need a mitmdump executable",
     "requires_internet: marks tests that need real Internet access",
 ]
 filterwarnings = [

scrapy/core/downloader/handlers/s3.py

@@ -1,9 +1,10 @@
 from __future__ import annotations
 
+import warnings
 from typing import TYPE_CHECKING, Any, cast
 
 from scrapy.core.downloader.handlers.base import BaseDownloadHandler
-from scrapy.exceptions import NotConfigured
+from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
 from scrapy.utils.boto import is_botocore_available
 from scrapy.utils.httpobj import urlparse_cached
 from scrapy.utils.misc import build_from_crawler, load_object
@@ -49,7 +50,16 @@ class S3DownloadHandler(BaseDownloadHandler):
 
     async def download_request(self, request: Request) -> Response:
         p = urlparse_cached(request)
-        scheme = "http" if request.meta.get("is_secure") is False else "https"
+        if request.meta.get("is_secure") is False:
+            warnings.warn(
+                "Passing is_secure=False for s3:// requests is deprecated."
+                " In future Scrapy releases this flag will be ignored.",
+                ScrapyDeprecationWarning,
+                stacklevel=2,
+            )
+            scheme = "http"
+        else:
+            scheme = "https"
         bucket = p.hostname
         path = p.path + "?" + p.query if p.query else p.path
         url = f"{scheme}://{bucket}.s3.amazonaws.com{path}"

scrapy/core/scheduler.py

@@ -289,11 +289,11 @@ class Scheduler(BaseScheduler):
 
         :param dqclass: A class to be used as persistent request queue.
                         The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default.
-        :type dqclass: class
+        :type dqclass: type
 
         :param mqclass: A class to be used as non-persistent request queue.
                         The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default.
-        :type mqclass: class
+        :type mqclass: type
 
         :param logunser: A boolean that indicates whether or not unserializable requests should be logged.
                         The value for the :setting:`SCHEDULER_DEBUG` setting is used by default.
@@ -306,7 +306,7 @@ class Scheduler(BaseScheduler):
 
         :param pqclass: A class to be used as priority queue for requests.
                         The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default.
-        :type pqclass: class
+        :type pqclass: type
 
         :param crawler: The crawler object corresponding to the current crawl.
         :type crawler: :class:`scrapy.crawler.Crawler`

scrapy/exporters.py

@@ -5,6 +5,7 @@ Item Exporters are used to export/serialize items into different formats.
 from __future__ import annotations
 
 import csv
+import logging
 import marshal
 import pickle
 import pprint
@@ -24,6 +25,8 @@ from scrapy.utils.serialize import ScrapyJSONEncoder
 if TYPE_CHECKING:
     from json import JSONEncoder
 
+logger = logging.getLogger(__name__)
+
 __all__ = [
     "BaseItemExporter",
     "CsvItemExporter",
@@ -254,6 +257,8 @@ class CsvItemExporter(BaseItemExporter):
         self.csv_writer = csv.writer(self.stream, **self._kwargs)
         self._headers_not_written = True
         self._join_multivalued = join_multivalued
+        self._autodetected_fields = False
+        self._data_loss_warned = False
 
     def serialize_field(
         self, field: Mapping[str, Any] | Field, name: str, value: Any
@@ -274,6 +279,22 @@ class CsvItemExporter(BaseItemExporter):
             self._headers_not_written = False
             self._write_headers_and_set_fields_to_export(item)
 
+        if (
+            self._autodetected_fields
+            and self.fields_to_export is not None
+            and not self._data_loss_warned
+        ):
+            item_fields = ItemAdapter(item).field_names()
+            dropped_fields = set(item_fields) - set(self.fields_to_export)
+
+            if dropped_fields:
+                dropped_fields_display = sorted(dropped_fields)
+                logger.warning(
+                    f"CSVExporter dropped fields {dropped_fields_display}. "
+                    f"To avoid this, fully configure your FEED_EXPORT_FIELDS setting. "
+                    f"See: https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-export-fields",
+                )
+                self._data_loss_warned = True
         fields = self._get_serialized_fields(item, default_value="", include_empty=True)
         values = list(self._build_row(x for _, x in fields))
         self.csv_writer.writerow(values)
@@ -293,6 +314,7 @@ class CsvItemExporter(BaseItemExporter):
             if not self.fields_to_export:
                 # use declared field names, or keys if the item is a dict
                 self.fields_to_export = ItemAdapter(item).field_names()
+                self._autodetected_fields = True
             fields: Iterable[str]
             if isinstance(self.fields_to_export, Mapping):
                 fields = self.fields_to_export.values()

scrapy/extensions/feedexport.py

@@ -47,6 +47,33 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
+
+# Printf-style placeholders (e.g. %(time)s) used to build feed URIs. Any other
+# percent character in a URI (e.g. percent-encoding such as %20 or %23) must be
+# treated as a literal rather than as the start of a placeholder.
+_FEED_URI_PLACEHOLDER_RE = re.compile(
+    r"%\([^)]+\)[-+ #0]*(?:\d+|\*)?(?:\.(?:\d+|\*))?[diouxXeEfFgGcrsa]"
+)
+
+
+def apply_uri_params(uri_template: str, uri_params: dict[str, Any]) -> str:
+    """Return *uri_template* with its ``%(...)s`` placeholders replaced using
+    *uri_params*, leaving any other percent character untouched.
+
+    This allows feed URIs to contain percent-encoded characters (e.g. ``%20``
+    in a path with spaces or ``%23`` in FTP credentials) without them being
+    misinterpreted as printf-style formatting directives.
+    """
+    parts: list[str] = []
+    last = 0
+    for match in _FEED_URI_PLACEHOLDER_RE.finditer(uri_template):
+        parts.append(uri_template[last : match.start()].replace("%", "%%"))
+        parts.append(match.group(0))
+        last = match.end()
+    parts.append(uri_template[last:].replace("%", "%%"))
+    return "".join(parts) % uri_params
+
+
 UriParamsCallableT: TypeAlias = Callable[
     [dict[str, Any], Spider], dict[str, Any] | None
 ]
@@ -473,7 +500,7 @@ class FeedExporter:
             )
             uri = self.settings["FEED_URI"]
             # handle pathlib.Path objects
-            uri = str(uri) if not isinstance(uri, Path) else uri.absolute().as_uri()
+            uri = str(uri.absolute()) if isinstance(uri, Path) else str(uri)
             feed_options = {"format": self.settings["FEED_FORMAT"]}
             self.feeds[uri] = feed_complete_default_values_from_settings(
                 feed_options, self.settings
@@ -485,9 +512,9 @@ class FeedExporter:
         for settings_uri, feed_options in self.settings.getdict("FEEDS").items():
             # handle pathlib.Path objects
             uri = (
-                str(settings_uri)
-                if not isinstance(settings_uri, Path)
-                else settings_uri.absolute().as_uri()
+                str(settings_uri.absolute())
+                if isinstance(settings_uri, Path)
+                else str(settings_uri)
             )
             self.feeds[uri] = feed_complete_default_values_from_settings(
                 feed_options, self.settings
@@ -514,7 +541,7 @@ class FeedExporter:
             self.slots.append(
                 self._start_new_batch(
                     batch_id=1,
-                    uri=uri % uri_params,
+                    uri=apply_uri_params(uri, uri_params),
                     feed_options=feed_options,
                     spider=spider,
                     uri_template=uri,
@@ -639,7 +666,7 @@ class FeedExporter:
                 slots.append(
                     self._start_new_batch(
                         batch_id=slot.batch_id + 1,
-                        uri=slot.uri_template % uri_params,
+                        uri=apply_uri_params(slot.uri_template, uri_params),
                         feed_options=self.feeds[slot.uri_template],
                         spider=spider,
                         uri_template=slot.uri_template,

scrapy/settings/__init__.py

@@ -288,10 +288,10 @@ class BaseSettings(MutableMapping[str, Any]):
         -   ``['one', 'two']`` if set to ``'["one", "two"]'`` or ``'one,two'``
 
         :param name: the setting name
-        :type name: string
+        :type name: str
 
         :param default: the value to return if no setting is found
-        :type default: any
+        :type default: object
         """
         value = self.get(name, default)
         if value is None:

scrapy/settings/default_settings.py

@@ -240,7 +240,7 @@ BOT_NAME = "scrapybot"
 CLOSESPIDER_ERRORCOUNT = 0
 CLOSESPIDER_ITEMCOUNT = 0
 CLOSESPIDER_PAGECOUNT = 0
-CLOSESPIDER_TIMEOUT = 0
+CLOSESPIDER_TIMEOUT = 0.0
 CLOSESPIDER_PAGECOUNT_NO_ITEM = 0
 CLOSESPIDER_TIMEOUT_NO_ITEM = 0
 

scrapy/spiders/__init__.py

@@ -7,9 +7,11 @@ See documentation in docs/topics/spiders.rst
 from __future__ import annotations
 
 import logging
+import warnings
 from typing import TYPE_CHECKING, Any, cast
 
 from scrapy import signals
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import Request, Response
 from scrapy.utils.trackref import object_ref
 from scrapy.utils.url import url_is_from_spider
@@ -66,6 +68,11 @@ class Spider(object_ref):
         can use it directly (e.g. Spider.logger.info('msg')) or use any other
         Python logger too.
         """
+        warnings.warn(
+            "Spider.log() is deprecated, use methods of Spider.logger instead.",
+            ScrapyDeprecationWarning,
+            stacklevel=2,
+        )
         self.logger.log(level, message, **kw)
 
     @classmethod

scrapy/utils/curl.py

@@ -23,6 +23,11 @@ class DataAction(argparse.Action):
     ) -> None:
         value = str(values)
         value = value.removeprefix("$")
+        # curl merges repeated -d/--data/--data-raw options into a single body
+        # joined with "&"; mirror that instead of keeping only the last one.
+        previous = getattr(namespace, self.dest, None)
+        if previous is not None:
+            value = f"{previous}&{value}"
         setattr(namespace, self.dest, value)
 
 

tests/AsyncCrawlerProcess/asyncio_deferred_signal.py

@@ -9,7 +9,7 @@ from scrapy.utils.defer import deferred_from_coro
 
 
 class UppercasePipeline:
-    async def _open_spider(self, spider):
+    async def _open_spider(self, spider: Spider) -> None:
         spider.logger.info("async pipeline opened!")
         await asyncio.sleep(0.1)
 

tests/AsyncCrawlerProcess/caching_hostname_resolver.py

@@ -10,6 +10,7 @@ class CachingHostnameResolverSpider(scrapy.Spider):
     """
 
     name = "caching_hostname_resolver_spider"
+    url: str
 
     async def start(self):
         yield scrapy.Request(self.url)

tests/CrawlerProcess/asyncio_deferred_signal.py

@@ -9,7 +9,7 @@ from scrapy.utils.defer import deferred_from_coro
 
 
 class UppercasePipeline:
-    async def _open_spider(self, spider):
+    async def _open_spider(self, spider: Spider) -> None:
         spider.logger.info("async pipeline opened!")
         await asyncio.sleep(0.1)
 

tests/CrawlerProcess/caching_hostname_resolver.py

@@ -10,6 +10,7 @@ class CachingHostnameResolverSpider(scrapy.Spider):
     """
 
     name = "caching_hostname_resolver_spider"
+    url: str
 
     async def start(self):
         yield scrapy.Request(self.url)

tests/CrawlerProcess/sleeping.py

@@ -15,7 +15,7 @@ class SleepingSpider(scrapy.Spider):
     async def parse(self, response):
         from twisted.internet import reactor
 
-        d = Deferred()
+        d: Deferred[None] = Deferred()
         reactor.callLater(int(sys.argv[1]), d.callback, None)
         await maybe_deferred_to_future(d)
 

tests/CrawlerRunner/ip_address.py

@@ -36,6 +36,7 @@ def createResolver(servers: list[tuple[str, int]]) -> ResolverBase:
 
 class LocalhostSpider(Spider):
     name = "localhost_spider"
+    url: str
 
     async def start(self):
         yield Request(self.url)

tests/mockserver/mitm_proxy.py

@@ -1,11 +1,42 @@
 from __future__ import annotations
 
-import re
-import sys
+import contextlib
+import functools
+import os
+import shutil
+import signal
+import socket
+import time
 from pathlib import Path
-from subprocess import PIPE, Popen
+from subprocess import DEVNULL, Popen
 from urllib.parse import urlsplit, urlunsplit
 
+from .utils import _free_port
+
+
+@functools.cache
+def mitmdump_cmd() -> list[str] | None:
+    """Return the command prefix used to invoke ``mitmdump``, or ``None`` if it
+    cannot be resolved.
+
+    We don't want to install ``mitmproxy`` into the test env (it has a lot of
+    dependencies that can conflict with some of the Scrapy/test ones, and its
+    newer versions may not support older Python versions). So we expect it
+    installed externally. We look for the ``mitmdump`` binary in the following
+    sources:
+
+    1. the ``MITMDUMP`` environment variable;
+    2. a ``mitmdump`` binary on ``PATH``;
+    3. using ``uvx --from mitmproxy mitmdump`` if ``uvx`` is available.
+    """
+    if env := os.environ.get("MITMDUMP"):
+        return [env]
+    if path := shutil.which("mitmdump"):
+        return [path]
+    if uvx := shutil.which("uvx"):
+        return [uvx, "--from", "mitmproxy", "mitmdump"]
+    return None
+
 
 class MitmProxy:
     auth_user = "scrapy"
@@ -15,18 +46,22 @@ class MitmProxy:
         self.mode = mode
 
     def start(self) -> str:
-        script = """
-import sys
-from mitmproxy.tools.main import mitmdump
-sys.argv[0] = "mitmdump"
-sys.exit(mitmdump())
-        """
+        cmd = mitmdump_cmd()
+        if not cmd:
+            raise RuntimeError(
+                "mitmdump is not available. Please install mitmproxy or uv."
+            )
         cert_path = Path(__file__).parent.parent.resolve() / "keys"
+        # Choose a free port ourselves instead of reading the mitmdump output
+        # as there is no easy way to disable stdout buffering for all kinds of
+        # mitmdump installs that we support.
+        host = "127.0.0.1"
+        port = _free_port()
         args = [
             "--listen-host",
-            "127.0.0.1",
+            host,
             "--listen-port",
-            "0",
+            str(port),
             "--proxyauth",
             f"{self.auth_user}:{self.auth_pass}",
             "--set",
@@ -37,30 +72,39 @@ sys.exit(mitmdump())
         ]
         if self.mode:
             args += ["--mode", self.mode]
-        self.proc: Popen[str] = Popen(
-            [
-                sys.executable,
-                "-u",
-                "-c",
-                script,
-                *args,
-            ],
-            stdout=PIPE,
-            text=True,
+        self.proc: Popen[bytes] = Popen(
+            [*cmd, *args],
+            stdout=DEVNULL,
+            stderr=DEVNULL,
+            start_new_session=True,  # needed for killpg() to make sense
         )
-        assert self.proc.stdout is not None
         scheme = "socks5" if self.mode == "socks5" else "http"
-        line = ""
-        for line in self.proc.stdout:
-            m = re.search(r"listening at (?:\w+://)?([^:]+:\d+)", line)
-            if m:
-                host_port = m.group(1)
-                return f"{scheme}://{self.auth_user}:{self.auth_pass}@{host_port}"
+        deadline = time.monotonic() + 60
+        while True:
+            if self.proc.poll() is not None:
+                raise RuntimeError(
+                    f"mitmdump exited with code {self.proc.returncode} before it "
+                    f"started listening"
+                )
+            try:
+                with socket.create_connection((host, port), timeout=1):
+                    return f"{scheme}://{self.auth_user}:{self.auth_pass}@{host}:{port}"
+            except OSError:
+                if time.monotonic() >= deadline:
+                    break
+                time.sleep(0.05)
         self.stop()
-        raise RuntimeError(f"Failed to parse mitmdump output: {line}")
+        raise RuntimeError(f"mitmdump did not start listening on {host}:{port} in time")
 
     def stop(self) -> None:
-        self.proc.kill()
+        if os.name == "posix":
+            # SIGKILL doesn't propagate to the actual process (child of uvx)
+            # https://github.com/astral-sh/uv/issues/11817#issuecomment-2688830077
+            # https://stackoverflow.com/a/61980200/113586
+            with contextlib.suppress(ProcessLookupError):
+                os.killpg(os.getpgid(self.proc.pid), signal.SIGKILL)
+        else:
+            self.proc.kill()
         self.proc.communicate()
 
 

tests/mockserver/utils.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import socket
 from pathlib import Path
 from typing import TYPE_CHECKING, cast
 
@@ -18,6 +19,13 @@ if TYPE_CHECKING:
     from twisted.internet.interfaces import IOpenSSLContextFactory
 
 
+def _free_port() -> int:
+    # racy but should be fine for tests
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        s.bind(("127.0.0.1", 0))
+        return cast("int", s.getsockname()[1])
+
+
 def ssl_context_factory(
     keyfile: str = "keys/localhost.key",
     certfile: str = "keys/localhost.crt",

tests/test_addons.py

@@ -107,11 +107,11 @@ class TestAddonManager:
         addonlist = []
         for i in range(3):
             addon = get_addon_cls({"KEY1": i})
-            addon.number = i
+            addon.number = i  # type: ignore[attr-defined]
             addonlist.append(addon)
         # Test for every possible ordering
         for ordered_addons in itertools.permutations(addonlist):
-            expected_order = [a.number for a in ordered_addons]
+            expected_order = [a.number for a in ordered_addons]  # type: ignore[attr-defined]
             settings = {"ADDONS": {a: i for i, a in enumerate(ordered_addons)}}
             crawler = get_crawler(settings_dict=settings)
             manager = crawler.addons
@@ -177,7 +177,7 @@ class TestAddonManager:
                     )
                 settings["SCHEDULER"] = "AddonScheduler"
 
-        settings_dict = {
+        settings_dict: dict[str, Any] = {
             "ADDONS": {AddonWithFallback: 1},
         }
         crawler = get_crawler(settings_dict=settings_dict)

tests/test_cmdline_crawl_with_pipeline/__init__.py

@@ -1,21 +1,24 @@
+from __future__ import annotations
+
 import sys
 from pathlib import Path
 from subprocess import PIPE, Popen
 
 
-class TestCmdlineCrawlPipeline:
-    def _execute(self, spname):
-        args = (sys.executable, "-m", "scrapy.cmdline", "crawl", spname)
-        cwd = Path(__file__).resolve().parent
-        proc = Popen(args, stdout=PIPE, stderr=PIPE, cwd=cwd)
-        _, stderr = proc.communicate()
-        return proc.returncode, stderr
+def _execute(spname: str) -> int:
+    args = (sys.executable, "-m", "scrapy.cmdline", "crawl", spname)
+    cwd = Path(__file__).resolve().parent
+    proc = Popen(args, stdout=PIPE, stderr=PIPE, cwd=cwd)
+    proc.communicate()
+    return proc.returncode
 
-    def test_open_spider_normally_in_pipeline(self):
-        returncode, _ = self._execute("normal")
-        assert returncode == 0
 
-    def test_exception_at_open_spider_in_pipeline(self):
-        returncode, _ = self._execute("exception")
-        # An exception in pipeline's open_spider should result in a non-zero exit code
-        assert returncode == 1
+def test_open_spider_normally_in_pipeline():
+    returncode = _execute("normal")
+    assert returncode == 0
+
+
+def test_exception_at_open_spider_in_pipeline():
+    returncode = _execute("exception")
+    # An exception in pipeline's open_spider should result in a non-zero exit code
+    assert returncode == 1

tests/test_commands.py

@@ -89,6 +89,7 @@ class TestCommandSettings:
             args=["-s", f"FEEDS={feeds_json}", "spider.py"]
         )
         self.command.process_options(args, opts)
+        assert self.command.settings is not None
         assert isinstance(self.command.settings["FEEDS"], scrapy.settings.BaseSettings)
         assert dict(self.command.settings["FEEDS"]) == json.loads(feeds_json)
 

tests/test_crawl.py

@@ -356,9 +356,10 @@ with multiples lines
 
     @coroutine_test
     async def test_engine_status(self, mockserver: MockServer) -> None:
-        est = []
+        est: list[list[tuple[str, Any]]] = []
 
         def cb(response):
+            assert crawler.engine
             est.append(get_engine_status(crawler.engine))
 
         crawler = get_crawler(SingleRequestSpider)
@@ -373,9 +374,10 @@ with multiples lines
 
     @coroutine_test
     async def test_format_engine_status(self, mockserver: MockServer) -> None:
-        est = []
+        est: list[str] = []
 
         def cb(response):
+            assert crawler.engine
             est.append(format_engine_status(crawler.engine))
 
         crawler = get_crawler(SingleRequestSpider)
@@ -386,8 +388,8 @@ with multiples lines
         assert len(est) == 1, est
         est = est[0].split("\n")[2:-2]  # remove header & footer
         # convert to dict
-        est = [x.split(":") for x in est]
-        est = [x for sublist in est for x in sublist]  # flatten
+        est_split = [x.split(":") for x in est]
+        est = [x for sublist in est_split for x in sublist]  # flatten
         est = [x.lstrip().rstrip() for x in est]
         it = iter(est)
         s = dict(zip(it, it, strict=True))

tests/test_crawler.py

@@ -523,6 +523,7 @@ class TestCrawlerLogging:
             }
 
             async def start(self):
+                assert crawler.stats
                 info_count_start = crawler.stats.get_value("log_count/INFO")
                 logging.debug("debug message")  # noqa: LOG015
                 logging.info("info message")  # noqa: LOG015

tests/test_dependencies.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import os
 import re
 from configparser import ConfigParser
@@ -23,6 +25,7 @@ class TestScrapyUtils:
         config_parser.read(tox_config_file_path)
         pattern = r"Twisted==([\d.]+)"
         match = re.search(pattern, config_parser["min"]["deps"])
+        assert match
         pinned_twisted_version_string = match[1]
 
         assert twisted_version.short() == pinned_twisted_version_string

tests/test_downloader_handlers.py

@@ -16,7 +16,7 @@ from scrapy.core.downloader.handlers.datauri import DataURIDownloadHandler
 from scrapy.core.downloader.handlers.file import FileDownloadHandler
 from scrapy.core.downloader.handlers.s3 import S3DownloadHandler
 from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
-from scrapy.http import Request
+from scrapy.http import Request, TextResponse
 from scrapy.responsetypes import responsetypes
 from scrapy.utils.boto import is_botocore_available
 from scrapy.utils.misc import build_from_crawler
@@ -175,7 +175,11 @@ class TestS3Anon:
     @coroutine_test
     async def test_anon_request_insecure(self):
         req = Request("s3://aws-publicdatasets/", meta={"is_secure": False})
-        httpreq = await self.download_request(req)
+        with pytest.warns(
+            ScrapyDeprecationWarning,
+            match="Passing is_secure=False for s3:// requests is deprecated",
+        ):
+            httpreq = await self.download_request(req)
         assert httpreq.url == "http://aws-publicdatasets.s3.amazonaws.com/"
 
 
@@ -215,7 +219,11 @@ class TestS3:
     @coroutine_test
     async def test_insecure_opt_out(self):
         req = Request("s3://johnsmith/photos/puppy.jpg", meta={"is_secure": False})
-        httpreq = await self.download_request(req)
+        with pytest.warns(
+            ScrapyDeprecationWarning,
+            match="Passing is_secure=False for s3:// requests is deprecated",
+        ):
+            httpreq = await self.download_request(req)
         assert httpreq.url == "http://johnsmith.s3.amazonaws.com/photos/puppy.jpg"
 
     @coroutine_test
@@ -354,6 +362,7 @@ class TestDataURI:
         response = await self.download_request(request)
         assert response.text == "A brief note"
         assert type(response) is responsetypes.from_mimetype("text/plain")  # pylint: disable=unidiomatic-typecheck
+        assert isinstance(response, TextResponse)
         assert response.encoding == "US-ASCII"
 
     @coroutine_test
@@ -362,6 +371,7 @@ class TestDataURI:
         response = await self.download_request(request)
         assert response.text == "\u038e\u03a3\u038e"
         assert type(response) is responsetypes.from_mimetype("text/plain")  # pylint: disable=unidiomatic-typecheck
+        assert isinstance(response, TextResponse)
         assert response.encoding == "iso-8859-7"
 
     @coroutine_test
@@ -370,6 +380,7 @@ class TestDataURI:
         response = await self.download_request(request)
         assert response.text == "\u038e\u03a3\u038e"
         assert response.body == b"\xbe\xd3\xbe"
+        assert isinstance(response, TextResponse)
         assert response.encoding == "iso-8859-7"
 
     @coroutine_test
@@ -382,6 +393,7 @@ class TestDataURI:
         response = await self.download_request(request)
         assert response.text == "\u038e\u03a3\u038e"
         assert type(response) is responsetypes.from_mimetype("text/plain")  # pylint: disable=unidiomatic-typecheck
+        assert isinstance(response, TextResponse)
         assert response.encoding == "utf-8"
 
     @coroutine_test

tests/test_downloadermiddleware.py

@@ -233,7 +233,7 @@ class TestMiddlewareUsingDeferreds(TestManagerBase):
                 return result
 
             def process_request(self, request):
-                d = Deferred()
+                d: Deferred[Response] = Deferred()
                 d.addCallback(self.cb)
                 d.callback(resp)
                 return d
@@ -298,6 +298,8 @@ class TestDownloadDeprecated(TestManagerBase):
             return succeed(resp)
 
         async with self.get_mwman() as mwman:
+            assert mwman.crawler
+            assert mwman.crawler.spider
             with pytest.warns(
                 ScrapyDeprecationWarning,
                 match=r"DownloaderMiddlewareManager.download\(\) is deprecated, use download_async\(\) instead",

tests/test_downloadermiddleware_defaultheaders.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 from scrapy.downloadermiddlewares.defaultheaders import DefaultHeadersMiddleware
 from scrapy.http import Request
 from scrapy.spiders import Spider
@@ -5,28 +7,29 @@ from scrapy.utils.python import to_bytes
 from scrapy.utils.test import get_crawler
 
 
-class TestDefaultHeadersMiddleware:
-    def get_defaults_mw(self):
-        crawler = get_crawler(Spider)
-        defaults = {
-            to_bytes(k): [to_bytes(v)]
-            for k, v in crawler.settings.get("DEFAULT_REQUEST_HEADERS").items()
-        }
-        return defaults, DefaultHeadersMiddleware.from_crawler(crawler)
+def get_defaults_mw() -> tuple[dict[bytes, list[bytes]], DefaultHeadersMiddleware]:
+    crawler = get_crawler(Spider)
+    defaults = {
+        to_bytes(k): [to_bytes(v)]
+        for k, v in crawler.settings.get("DEFAULT_REQUEST_HEADERS").items()
+    }
+    return defaults, DefaultHeadersMiddleware.from_crawler(crawler)
 
-    def test_process_request(self):
-        defaults, mw = self.get_defaults_mw()
-        req = Request("http://www.scrapytest.org")
-        mw.process_request(req)
-        assert req.headers == defaults
 
-    def test_update_headers(self):
-        defaults, mw = self.get_defaults_mw()
-        headers = {"Accept-Language": ["es"], "Test-Header": ["test"]}
-        bytes_headers = {b"Accept-Language": [b"es"], b"Test-Header": [b"test"]}
-        req = Request("http://www.scrapytest.org", headers=headers)
-        assert req.headers == bytes_headers
+def test_process_request():
+    defaults, mw = get_defaults_mw()
+    req = Request("http://www.scrapytest.org")
+    mw.process_request(req)
+    assert req.headers == defaults
 
-        mw.process_request(req)
-        defaults.update(bytes_headers)
-        assert req.headers == defaults
+
+def test_update_headers():
+    defaults, mw = get_defaults_mw()
+    headers = {"Accept-Language": ["es"], "Test-Header": ["test"]}
+    bytes_headers = {b"Accept-Language": [b"es"], b"Test-Header": [b"test"]}
+    req = Request("http://www.scrapytest.org", headers=headers)
+    assert req.headers == bytes_headers
+
+    mw.process_request(req)
+    defaults.update(bytes_headers)
+    assert req.headers == defaults

tests/test_downloadermiddleware_downloadtimeout.py

@@ -1,43 +1,51 @@
+from __future__ import annotations
+
+from typing import Any
+
 from scrapy.downloadermiddlewares.downloadtimeout import DownloadTimeoutMiddleware
 from scrapy.http import Request
 from scrapy.spiders import Spider
 from scrapy.utils.test import get_crawler
 
 
-class TestDownloadTimeoutMiddleware:
-    def get_request_spider_mw(self, settings=None):
-        crawler = get_crawler(Spider, settings)
-        spider = crawler._create_spider("foo")
-        request = Request("http://scrapytest.org/")
-        return request, spider, DownloadTimeoutMiddleware.from_crawler(crawler)
+def get_request_spider_mw(settings: dict[str, Any] | None = None):
+    crawler = get_crawler(Spider, settings)
+    spider = crawler._create_spider("foo")
+    request = Request("http://scrapytest.org/")
+    return request, spider, DownloadTimeoutMiddleware.from_crawler(crawler)
 
-    def test_default_download_timeout(self):
-        req, spider, mw = self.get_request_spider_mw()
-        mw.spider_opened(spider)
-        assert mw.process_request(req) is None
-        assert req.meta.get("download_timeout") == 180
 
-    def test_string_download_timeout(self):
-        req, spider, mw = self.get_request_spider_mw({"DOWNLOAD_TIMEOUT": "20.1"})
-        mw.spider_opened(spider)
-        assert mw.process_request(req) is None
-        assert req.meta.get("download_timeout") == 20.1
+def test_default_download_timeout():
+    req, spider, mw = get_request_spider_mw()
+    mw.spider_opened(spider)
+    assert mw.process_request(req) is None
+    assert req.meta.get("download_timeout") == 180
 
-    def test_setting_has_download_timeout(self):
-        req, spider, mw = self.get_request_spider_mw({"DOWNLOAD_TIMEOUT": 2})
-        mw.spider_opened(spider)
-        assert mw.process_request(req) is None
-        assert req.meta.get("download_timeout") == 2
 
-    def test_request_has_download_timeout(self):
-        req, spider, mw = self.get_request_spider_mw({"DOWNLOAD_TIMEOUT": 2})
-        mw.spider_opened(spider)
-        req.meta["download_timeout"] = 1
-        assert mw.process_request(req) is None
-        assert req.meta.get("download_timeout") == 1
+def test_string_download_timeout():
+    req, spider, mw = get_request_spider_mw({"DOWNLOAD_TIMEOUT": "20.1"})
+    mw.spider_opened(spider)
+    assert mw.process_request(req) is None
+    assert req.meta.get("download_timeout") == 20.1
 
-    def test_zero_download_timeout(self):
-        req, spider, mw = self.get_request_spider_mw({"DOWNLOAD_TIMEOUT": 0})
-        mw.spider_opened(spider)
-        assert mw.process_request(req) is None
-        assert req.meta.get("download_timeout") is None
+
+def test_setting_has_download_timeout():
+    req, spider, mw = get_request_spider_mw({"DOWNLOAD_TIMEOUT": 2})
+    mw.spider_opened(spider)
+    assert mw.process_request(req) is None
+    assert req.meta.get("download_timeout") == 2
+
+
+def test_request_has_download_timeout():
+    req, spider, mw = get_request_spider_mw({"DOWNLOAD_TIMEOUT": 2})
+    mw.spider_opened(spider)
+    req.meta["download_timeout"] = 1
+    assert mw.process_request(req) is None
+    assert req.meta.get("download_timeout") == 1
+
+
+def test_zero_download_timeout():
+    req, spider, mw = get_request_spider_mw({"DOWNLOAD_TIMEOUT": 0})
+    mw.spider_opened(spider)
+    assert mw.process_request(req) is None
+    assert req.meta.get("download_timeout") is None

tests/test_downloadermiddleware_useragent.py

@@ -1,33 +1,37 @@
+from __future__ import annotations
+
 from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
 from scrapy.http import Request
 from scrapy.spiders import Spider
 from scrapy.utils.test import get_crawler
 
 
-class TestUserAgentMiddleware:
-    def get_spider_and_mw(self, default_useragent):
-        crawler = get_crawler(Spider, {"USER_AGENT": default_useragent})
-        spider = crawler._create_spider("foo")
-        return spider, UserAgentMiddleware.from_crawler(crawler)
+def get_spider_and_mw(
+    default_useragent: str | None,
+) -> tuple[Spider, UserAgentMiddleware]:
+    crawler = get_crawler(Spider, {"USER_AGENT": default_useragent})
+    spider = crawler._create_spider("foo")
+    return spider, UserAgentMiddleware.from_crawler(crawler)
 
-    def test_default_agent(self):
-        _, mw = self.get_spider_and_mw("default_useragent")
-        req = Request("http://scrapytest.org/")
-        assert mw.process_request(req) is None
-        assert req.headers["User-Agent"] == b"default_useragent"
 
-    def test_header_agent(self):
-        spider, mw = self.get_spider_and_mw("default_useragent")
-        mw.spider_opened(spider)
-        req = Request(
-            "http://scrapytest.org/", headers={"User-Agent": "header_useragent"}
-        )
-        assert mw.process_request(req) is None
-        assert req.headers["User-Agent"] == b"header_useragent"
+def test_default_agent():
+    _, mw = get_spider_and_mw("default_useragent")
+    req = Request("http://scrapytest.org/")
+    assert mw.process_request(req) is None
+    assert req.headers["User-Agent"] == b"default_useragent"
 
-    def test_no_agent(self):
-        spider, mw = self.get_spider_and_mw(None)
-        mw.spider_opened(spider)
-        req = Request("http://scrapytest.org/")
-        assert mw.process_request(req) is None
-        assert "User-Agent" not in req.headers
+
+def test_header_agent():
+    spider, mw = get_spider_and_mw("default_useragent")
+    mw.spider_opened(spider)
+    req = Request("http://scrapytest.org/", headers={"User-Agent": "header_useragent"})
+    assert mw.process_request(req) is None
+    assert req.headers["User-Agent"] == b"header_useragent"
+
+
+def test_no_agent():
+    spider, mw = get_spider_and_mw(None)
+    mw.spider_opened(spider)
+    req = Request("http://scrapytest.org/")
+    assert mw.process_request(req) is None
+    assert "User-Agent" not in req.headers

tests/test_engine.py

@@ -38,6 +38,8 @@ from tests import get_testdata
 from tests.utils.decorators import coroutine_test, inline_callbacks_test
 
 if TYPE_CHECKING:
+    from collections.abc import AsyncIterator
+
     from twisted.python.failure import Failure
 
     from scrapy.core.scheduler import Scheduler
@@ -608,8 +610,8 @@ class TestEngineDownload(TestEngineDownloadAsync):
 @coroutine_test
 async def test_request_scheduled_signal():
     class TestScheduler(BaseScheduler):
-        def __init__(self):
-            self.enqueued = []
+        def __init__(self) -> None:
+            self.enqueued: list[Request] = []
 
         def enqueue_request(self, request: Request) -> bool:
             self.enqueued.append(request)
@@ -621,9 +623,9 @@ async def test_request_scheduled_signal():
 
     crawler = get_crawler(MySpider)
     engine = ExecutionEngine(crawler, lambda _: None)
-    scheduler = TestScheduler()
+    scheduler = TestScheduler()  # type: ignore[abstract]
 
-    async def start():
+    async def start() -> AsyncIterator[Any]:
         return
         yield
 

tests/test_exporters.py

@@ -384,6 +384,20 @@ class TestCsvItemExporter(TestBaseItemExporter):
             errors="xmlcharrefreplace",
         )
 
+    def test_csv_dropped_fields_warning(self, caplog):
+        out = BytesIO()
+        exporter = CsvItemExporter(out)
+        exporter.start_exporting()
+
+        exporter.export_item({"name": "Apple"})
+
+        with caplog.at_level("WARNING", logger="scrapy.exporters"):
+            exporter.export_item({"name": "Banana", "price": 2.00})
+
+        assert len(caplog.records) == 1
+        assert "CSVExporter dropped fields" in caplog.text
+        assert "price" in caplog.text
+
 
 class TestCsvItemExporterDataclass(TestCsvItemExporter):
     item_class = MyDataClass

tests/test_extension_debug.py

@@ -0,0 +1,135 @@
+from __future__ import annotations
+
+import logging
+import os
+import signal
+import sys
+import threading
+from typing import TYPE_CHECKING
+from unittest import mock
+
+import pytest
+
+from scrapy.extensions.debug import Debugger, StackTraceDump
+from scrapy.spiders import Spider
+from scrapy.utils.spider import DefaultSpider
+from scrapy.utils.test import get_crawler
+from tests.utils.decorators import coroutine_test
+
+if TYPE_CHECKING:
+    from collections.abc import Generator
+
+
+@pytest.fixture(autouse=True)
+def preserve_signal_handlers() -> Generator[None]:
+    """Restore the signal handlers that the extensions replace."""
+    signums = [
+        getattr(signal, name)
+        for name in ("SIGUSR2", "SIGQUIT")
+        if hasattr(signal, name)
+    ]
+    handlers = {signum: signal.getsignal(signum) for signum in signums}
+    yield
+    for signum, handler in handlers.items():
+        signal.signal(signum, handler)
+
+
+class SignalSpider(Spider):
+    name = "signal_spider"
+    start_urls = ["data:,"]
+
+    def parse(self, response):
+        os.kill(os.getpid(), signal.SIGUSR2)
+        return []
+
+
+@pytest.mark.skipif(
+    sys.platform == "win32", reason="SIGUSR2 and SIGQUIT are POSIX-only"
+)
+def test_stacktracedump_installs_signal_handlers() -> None:
+    crawler = get_crawler()
+    ext = StackTraceDump.from_crawler(crawler)
+    assert signal.getsignal(signal.SIGUSR2) == ext.dump_stacktrace  # pylint: disable=comparison-with-callable
+    assert signal.getsignal(signal.SIGQUIT) == ext.dump_stacktrace  # pylint: disable=comparison-with-callable
+
+
+def test_stacktracedump_works_without_signal_support(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    # simulate win32 platforms, which don't support SIGUSR signals
+    monkeypatch.delattr(signal, "SIGUSR2", raising=False)
+    ext = StackTraceDump.from_crawler(get_crawler())
+    assert isinstance(ext, StackTraceDump)
+
+
+def test_stacktracedump_dump_stacktrace(caplog: pytest.LogCaptureFixture) -> None:
+    crawler = get_crawler()
+    crawler.engine = mock.Mock()
+    ext = StackTraceDump.from_crawler(crawler)
+    spider = DefaultSpider()
+    with caplog.at_level(logging.INFO, logger="scrapy.extensions.debug"):
+        ext.dump_stacktrace(0, None)
+    assert len(caplog.records) == 1
+    message = caplog.records[0].getMessage()
+    assert "Dumping stack trace and engine status" in message
+    assert "Execution engine status" in message
+    assert "Live References" in message
+    assert type(spider).__name__ in message
+    assert "# Thread: MainThread" in message
+    assert getattr(caplog.records[0], "crawler", None) is crawler
+
+
+def test_stacktracedump_thread_stacks() -> None:
+    ext = StackTraceDump.from_crawler(get_crawler())
+    stop = threading.Event()
+    thread = threading.Thread(target=stop.wait, name="dump-test-thread")
+    thread.start()
+    try:
+        stacks = ext._thread_stacks()
+    finally:
+        stop.set()
+        thread.join()
+    assert "# Thread: MainThread" in stacks
+    assert "# Thread: dump-test-thread" in stacks
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="SIGUSR2 is POSIX-only")
+@coroutine_test
+async def test_stacktracedump_dumps_on_signal(caplog: pytest.LogCaptureFixture) -> None:
+    settings = {
+        "EXTENSIONS": {"scrapy.extensions.debug.StackTraceDump": 0},
+        "LOG_LEVEL": "INFO",
+    }
+    crawler = get_crawler(spidercls=SignalSpider, settings_dict=settings)
+    with caplog.at_level(logging.INFO, logger="scrapy.extensions.debug"):
+        await crawler.crawl_async()
+    for r in caplog.records:
+        message = r.getMessage()
+        if "Dumping stack trace and engine status" in message:
+            assert "Dumping stack trace and engine status" in message
+            assert "engine.spider.name" in message
+            assert "signal_spider" in message
+            return
+    raise AssertionError("No stack trace dump log message found")
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="SIGUSR2 is POSIX-only")
+def test_debugger_installs_signal_handler() -> None:
+    ext = Debugger()
+    assert signal.getsignal(signal.SIGUSR2) == ext._enter_debugger  # pylint: disable=comparison-with-callable
+
+
+def test_debugger_works_without_signal_support(monkeypatch: pytest.MonkeyPatch) -> None:
+    # simulate win32 platforms, which don't support SIGUSR signals
+    monkeypatch.delattr(signal, "SIGUSR2", raising=False)
+    ext = Debugger()
+    assert isinstance(ext, Debugger)
+
+
+def test_debugger_enter_debugger(monkeypatch: pytest.MonkeyPatch) -> None:
+    pdb_cls = mock.Mock()
+    monkeypatch.setattr("scrapy.extensions.debug.Pdb", pdb_cls)
+    ext = Debugger()
+    frame = sys._getframe()
+    ext._enter_debugger(0, frame)
+    pdb_cls.return_value.set_trace.assert_called_once_with(frame.f_back)

tests/test_extension_memdebug.py

@@ -0,0 +1,51 @@
+from __future__ import annotations
+
+import gc
+
+import pytest
+
+from scrapy.exceptions import NotConfigured
+from scrapy.extensions.memdebug import MemoryDebugger
+from scrapy.utils.spider import DefaultSpider
+from scrapy.utils.test import get_crawler
+from scrapy.utils.trackref import object_ref
+from tests.utils.decorators import coroutine_test
+
+
+def test_disabled_by_default() -> None:
+    with pytest.raises(NotConfigured):
+        MemoryDebugger.from_crawler(get_crawler())
+
+
+def test_spider_closed_sets_stats() -> None:
+    crawler = get_crawler(settings_dict={"MEMDEBUG_ENABLED": True})
+    ext = MemoryDebugger.from_crawler(crawler)
+
+    class TrackedObject(object_ref):
+        pass
+
+    class CollectedObject(object_ref):
+        pass
+
+    tracked = [TrackedObject(), TrackedObject()]
+    CollectedObject()
+
+    ext.spider_closed(DefaultSpider(), "finished")
+
+    assert crawler.stats
+    assert crawler.stats.get_value("memdebug/gc_garbage_count") == len(gc.garbage)
+    assert crawler.stats.get_value("memdebug/live_refs/TrackedObject") == len(tracked)
+    assert crawler.stats.get_value("memdebug/live_refs/CollectedObject") is None
+
+
+@coroutine_test
+async def test_crawl_sets_stats() -> None:
+    # unique class so that other tests don't pollute live_refs
+    class MemDebugSpider(DefaultSpider):
+        pass
+
+    crawler = get_crawler(MemDebugSpider, settings_dict={"MEMDEBUG_ENABLED": True})
+    await crawler.crawl_async()
+    assert crawler.stats
+    assert crawler.stats.get_value("memdebug/gc_garbage_count") is not None
+    assert crawler.stats.get_value("memdebug/live_refs/MemDebugSpider") == 1

tests/test_extension_periodic_log.py

@@ -102,7 +102,9 @@ class TestPeriodicLog:
             ext.spider_closed(spider, reason="finished")
             return ext, a, b
 
-        def check(settings: dict[str, Any], condition: Callable) -> None:
+        def check(
+            settings: dict[str, Any], condition: Callable[[str, Any], bool]
+        ) -> None:
             ext, a, b = emulate(settings)
             assert list(a["delta"].keys()) == [
                 k for k, v in ext.stats._stats.items() if condition(k, v)
@@ -168,7 +170,9 @@ class TestPeriodicLog:
             ext.spider_closed(spider, reason="finished")
             return ext, a, b
 
-        def check(settings: dict[str, Any], condition: Callable) -> None:
+        def check(
+            settings: dict[str, Any], condition: Callable[[str, Any], bool]
+        ) -> None:
             ext, a, b = emulate(settings)
             assert list(a["stats"].keys()) == [
                 k for k, v in ext.stats._stats.items() if condition(k, v)

tests/test_feedexport.py

@@ -32,6 +32,7 @@ from scrapy.extensions.feedexport import (
     FeedSlot,
     FileFeedStorage,
     IFeedStorage,
+    apply_uri_params,
 )
 from scrapy.utils.python import to_unicode
 from scrapy.utils.test import get_crawler
@@ -526,6 +527,91 @@ class TestFeedExport(TestFeedExportBase):
         header = self.MyItem.fields.keys()
         await self.assertExported(items, header, rows)
 
+    @coroutine_test
+    async def test_pathlib_uri_with_placeholders(self):
+        feed_dir = Path(self.temp_dir, "pathlib_placeholders")
+        feed_dir.mkdir()
+        items = [self.MyItem({"foo": "bar1", "egg": "spam1"})]
+
+        class TestSpider(scrapy.Spider):
+            name = "testspider"
+
+            def parse(self, response):
+                yield from items
+
+        TestSpider.start_urls = [self.mockserver.url("/")]
+        settings = {
+            "FEEDS": {
+                feed_dir / "%(time)s.json": {"format": "json"},
+            },
+        }
+        crawler = get_crawler(TestSpider, settings)
+        await crawler.crawl_async()
+
+        files = list(feed_dir.iterdir())
+        assert len(files) == 1
+        assert "%(time)s" not in files[0].name
+        assert files[0].suffix == ".json"
+
+    @coroutine_test
+    async def test_pathlib_uri_with_spaces_and_unicode(self):
+        # A pathlib.Path key with spaces and non-ASCII characters must be kept
+        # verbatim (not percent-encoded), while %()s placeholders are still
+        # substituted. %(name)s resolves to the spider name deterministically,
+        # so the resulting file name can be asserted exactly.
+        feed_dir = Path(self.temp_dir, "pathlib_spaces_unicode")
+        feed_dir.mkdir()
+        items = [self.MyItem({"foo": "bar1", "egg": "spam1"})]
+
+        class TestSpider(scrapy.Spider):
+            name = "testspider"
+
+            def parse(self, response):
+                yield from items
+
+        TestSpider.start_urls = [self.mockserver.url("/")]
+        settings = {
+            "FEEDS": {
+                feed_dir / "out %(name)s ünïcode.json": {"format": "json"},
+            },
+        }
+        crawler = get_crawler(TestSpider, settings)
+        await crawler.crawl_async()
+
+        files = list(feed_dir.iterdir())
+        assert len(files) == 1
+        assert files[0].name == "out testspider ünïcode.json"
+
+    @coroutine_test
+    async def test_str_uri_with_percent_encoding_and_placeholder(self):
+        # A percent-encoded string URI (e.g. %20 for a space) must reach
+        # storage verbatim rather than being misinterpreted as a printf
+        # directive, while %()s placeholders are still substituted. See #6425
+        # and #5794.
+        feed_dir = Path(self.temp_dir, "dir with spaces")
+        feed_dir.mkdir()
+        items = [self.MyItem({"foo": "bar1", "egg": "spam1"})]
+
+        class TestSpider(scrapy.Spider):
+            name = "testspider"
+
+            def parse(self, response):
+                yield from items
+
+        TestSpider.start_urls = [self.mockserver.url("/")]
+        settings = {
+            "FEEDS": {
+                f"{feed_dir.as_uri()}/%(time)s.json": {"format": "json"},
+            },
+        }
+        crawler = get_crawler(TestSpider, settings)
+        await crawler.crawl_async()
+
+        files = list(feed_dir.iterdir())
+        assert len(files) == 1
+        assert "%(time)s" not in files[0].name
+        assert files[0].suffix == ".json"
+
     @coroutine_test
     async def test_export_no_items_not_store_empty(self):
         for fmt in ("json", "jsonlines", "xml", "csv"):
@@ -1408,3 +1494,34 @@ class TestFeedExportInit:
         crawler = get_crawler(settings_dict=settings)
         exporter = FeedExporter.from_crawler(crawler)
         assert isinstance(exporter, FeedExporter)
+
+
+class TestApplyUriParams:
+    params = {
+        "name": "myspider",
+        "time": "2020-01-01T00-00-00",
+        "batch_id": 2,
+        "batch_time": "2020-01-01T00-00-00",
+    }
+
+    @pytest.mark.parametrize(
+        ("uri_template", "expected"),
+        [
+            # Placeholders are substituted, including width/flags.
+            ("/data/%(name)s/%(time)s.json", "/data/myspider/2020-01-01T00-00-00.json"),
+            ("/data/%(batch_id)05d.json", "/data/00002.json"),
+            # Percent-encoding is kept verbatim (#6425, #5794).
+            (
+                "file:///path%20with%20spaces/%(name)s.json",
+                "file:///path%20with%20spaces/myspider.json",
+            ),
+            (
+                "ftp://user:2%23um25%21M%23JZ@ftp.example.com/%(name)s.csv",
+                "ftp://user:2%23um25%21M%23JZ@ftp.example.com/myspider.csv",
+            ),
+            # A lone percent character next to a placeholder stays literal.
+            ("/100%/%(name)s.json", "/100%/myspider.json"),
+        ],
+    )
+    def test_apply_uri_params(self, uri_template, expected):
+        assert apply_uri_params(uri_template, self.params) == expected

tests/test_feedexport_batch.py

@@ -378,6 +378,7 @@ class TestBatchDeliveries(TestFeedExportBase):
         }
         crawler = get_crawler(ItemSpider, settings)
         yield crawler.crawl(total=2, mockserver=self.mockserver)
+        assert crawler.stats
         assert "feedexport/success_count/FileFeedStorage" in crawler.stats.get_stats()
         assert crawler.stats.get_value("feedexport/success_count/FileFeedStorage") == 12
 
@@ -392,7 +393,9 @@ class TestBatchDeliveries(TestFeedExportBase):
         ]
 
         class CustomS3FeedStorage(S3FeedStorage):
-            stubs = []
+            from botocore.stub import Stubber  # noqa: PLC0415
+
+            stubs: list[Stubber] = []
 
             def open(self, *args, **kwargs):
                 from botocore import __version__ as botocore_version  # noqa: PLC0415
@@ -449,6 +452,7 @@ class TestBatchDeliveries(TestFeedExportBase):
         assert len(CustomS3FeedStorage.stubs) == len(items)
         for stub in CustomS3FeedStorage.stubs:
             stub.assert_no_pending_responses()
+        assert crawler.stats
         assert (
             "feedexport/success_count/CustomS3FeedStorage" in crawler.stats.get_stats()
         )

tests/test_http_cookies.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 from http.cookiejar import DefaultCookiePolicy
 
 from scrapy.http import Request, Response
@@ -46,7 +48,7 @@ class TestCookieJar:
     def test_set_policy(self):
         policy = DefaultCookiePolicy()
         self.jar.set_policy(policy)
-        assert self.jar.jar._policy is policy
+        assert self.jar.jar._policy is policy  # type: ignore[attr-defined]
 
     def test_check_expired_frequency(self):
         jar = CookieJar(check_expired_frequency=1)

tests/test_responsetypes.py

@@ -1,3 +1,5 @@
+from typing import Any
+
 from scrapy.http import (
     Headers,
     HtmlResponse,
@@ -38,13 +40,13 @@ class TestResponseTypes:
         ]
         for source, cls in mappings:
             retcls = responsetypes.from_content_disposition(source)
-            assert retcls is cls, f"{source} ==> {retcls} != {cls}"
+            assert retcls is cls, f"{source!r} ==> {retcls} != {cls}"
 
     def test_from_content_disposition_no_filename(self):
         assert responsetypes.from_content_disposition(b"attachment") is Response
 
     def test_from_content_type(self):
-        mappings = [
+        mappings: list[tuple[str | bytes, type[Response]]] = [
             ("text/html; charset=UTF-8", HtmlResponse),
             ("text/xml; charset=UTF-8", XmlResponse),
             ("application/xhtml+xml; charset=UTF-8", HtmlResponse),
@@ -58,7 +60,7 @@ class TestResponseTypes:
         ]
         for source, cls in mappings:
             retcls = responsetypes.from_content_type(source)
-            assert retcls is cls, f"{source} ==> {retcls} != {cls}"
+            assert retcls is cls, f"{source!r} ==> {retcls} != {cls}"
 
     def test_from_body(self):
         mappings = [
@@ -71,7 +73,7 @@ class TestResponseTypes:
         ]
         for source, cls in mappings:
             retcls = responsetypes.from_body(source)
-            assert retcls is cls, f"{source} ==> {retcls} != {cls}"
+            assert retcls is cls, f"{source!r} ==> {retcls} != {cls}"
 
     def test_from_headers(self):
         mappings = [
@@ -98,7 +100,7 @@ class TestResponseTypes:
 
     def test_from_args(self):
         # TODO: add more tests that check precedence between the different arguments
-        mappings = [
+        mappings: list[tuple[dict[str, Any], type[Response]]] = [
             ({"url": "http://www.example.com/data.csv"}, TextResponse),
             # headers takes precedence over url
             (

tests/test_scheduler.py

@@ -379,13 +379,14 @@ class TestIntegrationWithDownloaderAwareInMemory:
             url = mockserver.url("/status?n=200", is_secure=False)
             start_urls = [url] * 6
             yield self.crawler.crawl(start_urls)
+            assert self.crawler.stats
             assert self.crawler.stats.get_value("downloader/response_count") == len(
                 start_urls
             )
 
 
 class TestIncompatibility:
-    def _incompatible(self):
+    def _incompatible(self) -> None:
         settings = {
             "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.DownloaderAwarePriorityQueue",
             "CONCURRENT_REQUESTS_PER_IP": 1,

tests/test_signals.py

@@ -1,13 +1,20 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
 import pytest
 
 from scrapy import Request, Spider, signals
 from scrapy.utils.test import get_crawler, get_from_asyncio_queue
-from tests.mockserver.http import MockServer
-from tests.utils.decorators import coroutine_test, inline_callbacks_test
+from tests.utils.decorators import coroutine_test
+
+if TYPE_CHECKING:
+    from tests.mockserver.http import MockServer
 
 
 class ItemSpider(Spider):
     name = "itemspider"
+    mockserver: MockServer
 
     async def start(self):
         for index in range(10):
@@ -34,15 +41,6 @@ class TestMain:
 
 
 class TestMockServer:
-    @classmethod
-    def setup_class(cls):
-        cls.mockserver = MockServer()
-        cls.mockserver.__enter__()
-
-    @classmethod
-    def teardown_class(cls):
-        cls.mockserver.__exit__(None, None, None)
-
     def setup_method(self):
         self.items = []
 
@@ -51,11 +49,11 @@ class TestMockServer:
         self.items.append(item)
 
     @pytest.mark.only_asyncio
-    @inline_callbacks_test
-    def test_simple_pipeline(self):
+    @coroutine_test
+    async def test_simple_pipeline(self, mockserver: MockServer) -> None:
         crawler = get_crawler(ItemSpider)
         crawler.signals.connect(self._on_item_scraped, signals.item_scraped)
-        yield crawler.crawl(mockserver=self.mockserver)
+        await crawler.crawl_async(mockserver=mockserver)
         assert len(self.items) == 10
         for index in range(10):
             assert {"index": index} in self.items

tests/test_spider.py

@@ -8,6 +8,7 @@ from testfixtures import LogCapture
 
 from scrapy import signals
 from scrapy.crawler import Crawler
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import Response, TextResponse, XmlResponse
 from scrapy.settings import Settings
 from scrapy.spiders import CSVFeedSpider, Spider, XMLFeedSpider
@@ -116,7 +117,12 @@ class TestSpider:
 
     def test_log(self):
         spider = self.spider_class("example.com")
-        with mock.patch("scrapy.spiders.Spider.logger") as mock_logger:
+        with (
+            mock.patch("scrapy.spiders.Spider.logger") as mock_logger,
+            pytest.warns(
+                ScrapyDeprecationWarning, match=r"Spider.log\(\) is deprecated"
+            ),
+        ):
             spider.log("test log msg", "INFO")
         mock_logger.log.assert_called_once_with("INFO", "test log msg")
 

tests/test_spider_start.py

@@ -1,7 +1,7 @@
 from __future__ import annotations
 
 from asyncio import sleep
-from typing import Any
+from typing import TYPE_CHECKING, Any
 
 import pytest
 
@@ -12,6 +12,9 @@ from scrapy.utils.test import get_crawler
 from .utils import twisted_sleep
 from .utils.decorators import coroutine_test
 
+if TYPE_CHECKING:
+    from collections.abc import AsyncIterator, Callable
+
 SLEEP_SECONDS = 0.1
 
 ITEM_A = {"id": "a"}
@@ -67,7 +70,11 @@ class TestMain:
 
         await self._test_spider(TestSpider, [ITEM_A])
 
-    async def _test_start(self, start_, expected_items=None):
+    async def _test_start(
+        self,
+        start_: Callable[[Any], AsyncIterator[Any]],
+        expected_items: list[Any] | None = None,
+    ) -> None:
         class TestSpider(Spider):
             name = "test"
             start = start_

tests/test_spidermiddleware.py

@@ -186,7 +186,7 @@ class ProcessSpiderExceptionSimpleIterableMiddleware:
 class ProcessSpiderExceptionAsyncIteratorMiddleware:
     async def process_spider_exception(self, response, exception):
         yield {"foo": 1}
-        d = defer.Deferred()
+        d: defer.Deferred[None] = defer.Deferred()
         call_later(0, d.callback, None)
         await maybe_deferred_to_future(d)
         yield {"foo": 2}

tests/test_spidermiddleware_httperror.py

@@ -22,6 +22,7 @@ class _HttpErrorSpider(MockServerSpider):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
+        assert self.mockserver
         self.start_urls = [
             self.mockserver.url("/status?n=200"),
             self.mockserver.url("/status?n=404"),
@@ -36,7 +37,7 @@ class _HttpErrorSpider(MockServerSpider):
         for url in self.start_urls:
             yield Request(url, self.parse, errback=self.on_error)
 
-    def parse(self, response):
+    def parse(self, response: Response) -> None:
         self.parsed.add(response.url[-3:])
 
     def on_error(self, failure):

tests/test_spidermiddleware_referer.py

@@ -1,7 +1,7 @@
 from __future__ import annotations
 
 import warnings
-from typing import Any
+from typing import TYPE_CHECKING, Any
 from urllib.parse import urlparse
 
 import pytest
@@ -35,6 +35,9 @@ from scrapy.utils.misc import build_from_crawler
 from scrapy.utils.test import get_crawler
 from tests.utils.decorators import coroutine_test
 
+if TYPE_CHECKING:
+    from collections.abc import AsyncIterator
+
 
 class TestRefererMiddleware:
     req_meta: dict[str, Any] = {}
@@ -1031,7 +1034,7 @@ async def test_response_policy_only_supports_policy_names():
     crawler = get_crawler(settings_dict={"REFERRER_POLICY": "no-referrer"})
     mw = build_from_crawler(RefererMiddleware, crawler)
 
-    async def input_result():
+    async def input_result() -> AsyncIterator[Any]:
         yield Request("https://example.com/")
 
     response = Response(
@@ -1079,7 +1082,7 @@ async def test_referer_policies_setting():
     )
     mw = build_from_crawler(RefererMiddleware, crawler)
 
-    async def input_result():
+    async def input_result() -> AsyncIterator[Any]:
         yield Request("https://example.com/")
 
     # "no-referrer-when-downgrade": None,

tests/test_spidermiddleware_start.py

@@ -1,3 +1,7 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
 from scrapy.http import Request, Response
 from scrapy.spidermiddlewares.start import StartSpiderMiddleware
 from scrapy.spiders import Spider
@@ -5,6 +9,9 @@ from scrapy.utils.misc import build_from_crawler
 from scrapy.utils.test import get_crawler
 from tests.utils.decorators import coroutine_test
 
+if TYPE_CHECKING:
+    from collections.abc import AsyncIterator
+
 
 class TestMiddleware:
     @coroutine_test
@@ -12,7 +19,7 @@ class TestMiddleware:
         crawler = get_crawler(Spider)
         mw = build_from_crawler(StartSpiderMiddleware, crawler)
 
-        async def start():
+        async def start() -> AsyncIterator[Request]:
             yield Request("data:,1")
             yield Request("data:,2", meta={"is_start_request": True})
             yield Request("data:,2", meta={"is_start_request": False})

tests/test_utils_asyncio.py

@@ -72,7 +72,7 @@ class TestParallelAsyncio:
         for length in [20, 50, 100]:
             parallel_count = [0]
             max_parallel_count = [0]
-            results = []
+            results: list[int] = []
             ait = self.get_async_iterable(length)
             await _parallel_asyncio(
                 ait,
@@ -91,7 +91,7 @@ class TestParallelAsyncio:
         for length in [20, 50, 100]:
             parallel_count = [0]
             max_parallel_count = [0]
-            results = []
+            results: list[int] = []
             ait = self.get_async_iterable_with_delays(length)
             await _parallel_asyncio(
                 ait,

tests/test_utils_conf.py

@@ -1,3 +1,7 @@
+from __future__ import annotations
+
+from typing import Any
+
 import pytest
 
 from scrapy.exceptions import UsageError
@@ -106,7 +110,7 @@ class TestFeedExportConfig:
             )
 
     def test_feed_complete_default_values_from_settings_empty(self):
-        feed = {}
+        feed: dict[str, Any] = {}
         settings = Settings(
             {
                 "FEED_EXPORT_ENCODING": "custom encoding",

tests/test_utils_console.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import pytest
 
 from scrapy.utils.console import get_shell_embed_func
@@ -38,4 +40,5 @@ def test_get_shell_embed_func_bpython():
 def test_get_shell_embed_func_ipython():
     # default shell should be 'ipython'
     shell = get_shell_embed_func()
+    assert shell is not None
     assert shell.__name__ == "_embed_ipython_shell"

tests/test_utils_curl.py

@@ -163,6 +163,39 @@ class TestCurlToRequestKwargs:
         }
         self._test_command(curl_command, expected_result)
 
+    def test_post_data_multiple(self):
+        # curl merges repeated -d/--data/--data-raw options into a single body
+        # joined with "&"; scrapy must do the same, not keep only the last one.
+        curl_command = "curl 'https://www.example.org/' -d 'a=1' -d 'b=2' -d 'c=3'"
+        expected_result = {
+            "method": "POST",
+            "url": "https://www.example.org/",
+            "body": "a=1&b=2&c=3",
+        }
+        self._test_command(curl_command, expected_result)
+
+    def test_post_data_multiple_mixed_flag_names(self):
+        curl_command = (
+            "curl 'https://www.example.org/' --data 'a=1' --data-raw 'b=2' -d 'c=3'"
+        )
+        expected_result = {
+            "method": "POST",
+            "url": "https://www.example.org/",
+            "body": "a=1&b=2&c=3",
+        }
+        self._test_command(curl_command, expected_result)
+
+    def test_post_data_multiple_with_string_prefix(self):
+        # The leading "$" left by bash $'...' quoting is stripped per option,
+        # before the values are merged.
+        curl_command = "curl 'https://www.example.org/' -d $'a=1' -d $'b=2'"
+        expected_result = {
+            "method": "POST",
+            "url": "https://www.example.org/",
+            "body": "a=1&b=2",
+        }
+        self._test_command(curl_command, expected_result)
+
     def test_explicit_get_with_data(self):
         curl_command = "curl httpbin.org/anything -X GET --data asdf"
         expected_result = {

tests/test_utils_display.py

@@ -1,9 +1,16 @@
+from __future__ import annotations
+
 import builtins
 from io import StringIO
+from typing import TYPE_CHECKING
 from unittest import mock
 
 from scrapy.utils.display import pformat, pprint
 
+if TYPE_CHECKING:
+    from collections.abc import Mapping, Sequence
+    from types import ModuleType
+
 value = {"a": 1}
 colorized_strings = {
     (
@@ -77,14 +84,19 @@ def test_pformat_no_pygments(isatty):
 
     real_import = builtins.__import__
 
-    def mock_import(name, globals_, locals_, fromlist, level):
+    def mock_import(
+        name: str,
+        globals_: Mapping[str, object] | None = None,
+        locals_: Mapping[str, object] | None = None,
+        fromlist: Sequence[str] | None = (),
+        level: int = 0,
+    ) -> ModuleType:
         if "pygments" in name:
             raise ImportError
         return real_import(name, globals_, locals_, fromlist, level)
 
-    builtins.__import__ = mock_import
-    assert pformat(value) == plain_string
-    builtins.__import__ = real_import
+    with mock.patch("builtins.__import__", mock_import):
+        assert pformat(value) == plain_string
 
 
 def test_pprint():

tests/test_utils_iterators.py

@@ -238,7 +238,7 @@ class TestXmliterBase(ABC):
             next(my_iter)
 
     def test_xmliter_objtype_exception(self):
-        i = self.xmliter(42, "product")
+        i = self.xmliter(42, "product")  # type: ignore[arg-type]
         with pytest.raises(TypeError):
             next(i)
 
@@ -353,7 +353,7 @@ class TestLxmlXmliter(TestXmliterBase):
         assert node.xpath("f:name/text()").getall() == ["African Coffee Table"]
 
     def test_xmliter_objtype_exception(self):
-        i = self.xmliter(42, "product")
+        i = self.xmliter(42, "product")  # type: ignore[arg-type]
         with pytest.raises(TypeError):
             next(i)
 

tests/test_utils_log.py

@@ -38,7 +38,7 @@ class TestFailureToExcInfo:
         assert exc_info == failure_to_exc_info(failure)
 
     def test_non_failure(self):
-        assert failure_to_exc_info("test") is None
+        assert failure_to_exc_info("test") is None  # type: ignore[arg-type]
 
 
 class TestTopLevelFormatter:

tests/test_utils_misc/__init__.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import os
 import sys
 from pathlib import Path
@@ -37,7 +39,7 @@ class TestUtilsMisc:
         with pytest.raises(NameError):
             load_object("scrapy.utils.misc.load_object999")
         with pytest.raises(TypeError):
-            load_object({})
+            load_object({})  # type: ignore[arg-type]
 
     def test_walk_modules(self):
         mods = walk_modules_iter("tests.test_utils_misc.test_walk_modules")
@@ -114,7 +116,7 @@ class TestUtilsMisc:
         args = (True, 100.0)
         kwargs = {"key": "val"}
 
-        def _test_with_crawler(mock, crawler):
+        def _test_with_crawler(mock: mock.MagicMock, crawler: mock.MagicMock) -> None:
             build_from_crawler(mock, crawler, *args, **kwargs)
             if hasattr(mock, "from_crawler"):
                 mock.from_crawler.assert_called_once_with(crawler, *args, **kwargs)

tests/test_utils_response.py

@@ -38,7 +38,7 @@ def test_open_in_browser():
 
     resp = Response(url, body=body)
     with pytest.raises(TypeError):
-        open_in_browser(resp, debug=True)  # pylint: disable=unexpected-keyword-arg
+        open_in_browser(resp, _openfunc=browser_open)  # type: ignore[arg-type]
 
 
 def test_get_meta_refresh():
@@ -320,4 +320,4 @@ def test_open_in_browser_text_response_uses_txt_extension():
 def test_open_in_browser_raises_for_unsupported_response_type():
     response = Response("http://www.example.com", body=b"binary")
     with pytest.raises(TypeError):
-        open_in_browser(response, _openfunc=lambda _: True)
+        open_in_browser(response, _openfunc=lambda _: True)  # type: ignore[arg-type]

tests/test_utils_signal.py

@@ -1,4 +1,7 @@
+from __future__ import annotations
+
 import asyncio
+from typing import TYPE_CHECKING
 
 import pytest
 from pydispatch import dispatcher
@@ -16,6 +19,9 @@ from scrapy.utils.signal import (
 from scrapy.utils.test import get_from_asyncio_queue
 from tests.utils.decorators import inline_callbacks_test
 
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
 
 class TestSendCatchLog:
     # whether the function being tested returns exceptions or failures
@@ -24,7 +30,7 @@ class TestSendCatchLog:
     @inline_callbacks_test
     def test_send_catch_log(self):
         test_signal = object()
-        handlers_called = set()
+        handlers_called: set[Callable[..., None]] = set()
 
         dispatcher.connect(self.error_handler, signal=test_signal)
         dispatcher.connect(self.ok_handler, signal=test_signal)
@@ -74,7 +80,7 @@ class TestSendCatchLogDeferred2(TestSendCatchLogDeferred):
     def ok_handler(self, arg, handlers_called):
         handlers_called.add(self.ok_handler)
         assert arg == "test"
-        d = defer.Deferred()
+        d: defer.Deferred[str] = defer.Deferred()
         call_later(0, d.callback, "OK")
         return d
 
@@ -108,7 +114,7 @@ class TestSendCatchLogAsync2(TestSendCatchLogAsync):
     def ok_handler(self, arg, handlers_called):
         handlers_called.add(self.ok_handler)
         assert arg == "test"
-        d = defer.Deferred()
+        d: defer.Deferred[str] = defer.Deferred()
         call_later(0, d.callback, "OK")
         return d
 

tests/test_utils_spider.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 from scrapy import Spider
 from scrapy.http import Request
 from scrapy.item import Item
@@ -17,7 +19,7 @@ def test_iterate_spider_output():
     r = Request("http://scrapytest.org")
     o = object()
 
-    assert list(iterate_spider_output(i)) == [i]
+    assert list(iterate_spider_output(i)) == [i]  # type: ignore[call-overload]
     assert list(iterate_spider_output(r)) == [r]
     assert list(iterate_spider_output(o)) == [o]
     assert list(iterate_spider_output([r, i, o])) == [r, i, o]

tests/test_utils_trackref.py

@@ -79,9 +79,9 @@ def test_get_oldest():
     stale entries in `live_refs`, affecting results unless explicitly cleared.
     """
 
-    def _delete_o1():
+    def _delete_o1() -> None:
         """Delete `o1` and ensure it is actually collected on PyPy."""
-        nonlocal o1
+        nonlocal o1  # type: ignore[misc]
         del o1
 
         if _IS_PYPY:
@@ -89,7 +89,7 @@ def test_get_oldest():
             # still exist until the GC runs, so we force a collection cycle.
             garbage_collect()
 
-    def _do_asserts():
+    def _do_asserts() -> None:
         assert trackref.get_oldest("Foo") is o1
         assert trackref.get_oldest("Bar") is o2
         # Ensure the newer Foo is not incorrectly considered the oldest

tox.ini

@@ -28,7 +28,6 @@ envlist =
     no-reactor
     no-reactor-extra-deps
     botocore
-    mitmproxy
     pypy3
     pypy3-extra-deps
 minversion = 1.7.0
@@ -55,6 +54,7 @@ deps =
 passenv =
     PYTHONTRACEMALLOC
     PYTEST_ADDOPTS
+    MITMDUMP
     S3_TEST_FILE_URI
     AWS_ACCESS_KEY_ID
     AWS_SECRET_ACCESS_KEY
@@ -70,13 +70,13 @@ commands =
 basepython = python3.10
 deps =
     mypy==2.1.0
-    typing-extensions==4.15.0
-    Pillow==12.2.0
-    Protego==0.6.0
+    typing-extensions==4.16.0
+    Pillow==12.3.0
+    Protego==0.6.2
     Twisted==26.4.0
     attrs==26.1.0
-    boto3-stubs[s3]==1.43.9
-    botocore-stubs==1.42.41
+    boto3-stubs[s3]==1.43.41
+    botocore-stubs==1.43.14
     h2==4.3.0
     httpx==0.28.1
     itemadapter==0.13.1
@@ -84,12 +84,12 @@ deps =
     # newer ones require newer Python
     ipython==8.39.0
     pyOpenSSL==26.3.0
-    pytest==9.0.3
+    pytest==9.1.1
     socksio==1.0.0
-    types-Pygments==2.20.0.20260508
+    types-Pygments==2.20.0.20260518
     types-defusedxml==0.7.0.20260504
     types-lxml==2026.2.16
-    types-pexpect==4.9.0.20260508
+    types-pexpect==4.9.0.20260518
     uvloop==0.22.1
     w3lib==2.4.1
     zstandard==0.25.0
@@ -116,7 +116,7 @@ commands =
 basepython = python3
 deps =
     {[testenv:extra-deps]deps}
-    pylint==4.0.2
+    pylint==4.0.6
     pylint-per-file-ignores # https://github.com/pylint-dev/pylint/issues/3767#issuecomment-1319916278
 commands =
     pylint {posargs:conftest.py docs extras scrapy tests}
@@ -125,7 +125,7 @@ commands =
 basepython = python3
 deps =
     twine==6.2.0
-    build==1.3.0
+    build==1.5.0
 commands =
     python -m build --sdist
     twine check dist/*
@@ -319,17 +319,3 @@ setenv =
     {[min]setenv}
 commands =
     pytest {posargs:--cov-config=pyproject.toml --cov=scrapy --cov-report=xml --cov-report= tests --junitxml=min-botocore.junit.xml -o junit_family=legacy} -m requires_botocore
-
-
-# Run proxy tests that use mitmproxy in a separate env to avoid installing
-# numerous mitmproxy deps in other envs (even in extra-deps), as they can
-# conflict with other deps we want, or don't want, to have installed there.
-
-[testenv:mitmproxy]
-deps =
-    {[testenv]deps}
-    # mitmproxy does not support PyPy
-    mitmproxy; implementation_name != "pypy"
-    httpx[http2,socks]
-commands =
-    pytest {posargs:--cov-config=pyproject.toml --cov=scrapy --cov-report=xml --cov-report= tests --junitxml=mitmproxy.junit.xml -o junit_family=legacy} -m requires_mitmproxy