Release notes for 2.15.0 (#7373)

* Cover 2.15.0 in the release notes (up to 31bf7c3). * Replace VERSION. * Document unified download handler exceptions. * Updates. * Cover 2.15.0 in the release notes (up to 0c6ccf5). * Cover 2.15.0 in the release notes (up to b68f267). * Run linkcheck and fix issues. * Cleanup unused link targets. * Cleanup.
2026-04-09 16:56:30 +05:00 · 2026-04-09 16:56:30 +05:00 · 7e881ce2d7
parent b68f26726a
commit 7e881ce2d7
22 changed files with 379 additions and 60 deletions
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@ -258,7 +258,7 @@ Scrapy:

 * Don't put your name in the code you contribute; git provides enough
  metadata to identify author of the code.
-  See https://docs.github.com/en/get-started/getting-started-with-git/setting-your-username-in-git
+  See https://docs.github.com/en/get-started/git-basics/setting-your-username-in-git
  for setup instructions.

 .. _scrapy-pre-commit:
@ -390,8 +390,7 @@ And their unit-tests are in::

 .. _issue tracker: https://github.com/scrapy/scrapy/issues
 .. _scrapy-users: https://groups.google.com/forum/#!forum/scrapy-users
-.. _Scrapy subreddit: https://reddit.com/r/scrapy
-.. _AUTHORS: https://github.com/scrapy/scrapy/blob/master/AUTHORS
+.. _Scrapy subreddit: https://www.reddit.com/r/scrapy/
 .. _tests/: https://github.com/scrapy/scrapy/tree/master/tests
 .. _open issues: https://github.com/scrapy/scrapy/issues
 .. _PEP 257: https://peps.python.org/pep-0257/
--- a/docs/faq.rst
+++ b/docs/faq.rst
@ -418,4 +418,3 @@ See :issue:`2680`.
 .. _has been reported: https://github.com/scrapy/scrapy/issues/2905
 .. _Python standard library modules: https://docs.python.org/3/py-modindex.html
 .. _Python package: https://pypi.org/
-.. _user agents: https://en.wikipedia.org/wiki/User_agent
--- a/docs/intro/install.rst
+++ b/docs/intro/install.rst
@ -263,7 +263,6 @@ reinstall Twisted with the :code:`tls` extra option::
 For details, see `Issue #2473 <https://github.com/scrapy/scrapy/issues/2473>`_.

 .. _Python: https://www.python.org/
-.. _pip: https://pip.pypa.io/en/latest/installing/
 .. _lxml: https://lxml.de/index.html
 .. _parsel: https://pypi.org/project/parsel/
 .. _w3lib: https://pypi.org/project/w3lib/
@ -273,8 +272,7 @@ For details, see `Issue #2473 <https://github.com/scrapy/scrapy/issues/2473>`_.
 .. _setuptools: https://pypi.org/pypi/setuptools
 .. _homebrew: https://brew.sh/
 .. _zsh: https://www.zsh.org/
-.. _Anaconda: https://docs.anaconda.com/anaconda/
+.. _Anaconda: https://www.anaconda.com/docs/main
 .. _Miniconda: https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html
-.. _Visual Studio: https://docs.microsoft.com/en-us/visualstudio/install/install-visual-studio
 .. _Microsoft C++ Build Tools: https://visualstudio.microsoft.com/visual-cpp-build-tools/
 .. _conda-forge: https://conda-forge.org/
--- a/docs/intro/overview.rst
+++ b/docs/intro/overview.rst
@ -150,7 +150,7 @@ The next steps for you are to :ref:`install Scrapy <intro-install>`,
 a full-blown Scrapy project and `join the community`_. Thanks for your
 interest!

-.. _join the community: https://scrapy.org/community/
+.. _join the community: https://www.scrapy.org/community
 .. _web scraping: https://en.wikipedia.org/wiki/Web_scraping
 .. _Amazon Associates Web Services: https://affiliate-program.amazon.com/welcome/ecs
 .. _Amazon S3: https://aws.amazon.com/s3/
--- a/docs/news.rst
+++ b/docs/news.rst
@ -3,13 +3,28 @@
 Release notes
 =============

-Scrapy VERSION (unreleased)
---------------------------
+.. _release-2.15.0:
+
+Scrapy 2.15.0 (unreleased)
+--------------------------
+
+Highlights:
+
+-   Experimental support for running without a Twisted reactor
+
+-   Experimental ``httpx``-based download handler

 Backward-incompatible changes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   In order to fix a long-standing bug with handling of asynchronous storages
+-   The built-in HTTP :ref:`download handlers <download-handlers-ref>` now
+    raise Scrapy-specific exceptions instead of implementation-specific ones,
+    see :ref:`download-handlers-exceptions`. This can affect user code that
+    handles downloader exceptions, such as ``process_exception()`` methods of
+    custom :ref:`downloader middlewares <topics-downloader-middleware-custom>`.
+    (:issue:`7208`)
+
+-   In order to fix a long-standing bug with handling of asynchronous storages,
    the following changes were made to media pipeline classes, which can impact
    some of the user code that subclasses them or calls their methods directly:

@ -44,14 +59,308 @@ Backward-incompatible changes
        being set to ``None``. Update code that relied on ``is None`` checks or
        the previous coercion behaviour.

+    (:issue:`7036`, :issue:`7367`, :issue:`7374`)
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+-   The context factory class set as the value of the
+    ``DOWNLOADER_CLIENTCONTEXTFACTORY`` setting is now required to support the
+    ``method`` argument of ``__init__()``, recommended since Scrapy 1.2.0.
+    (:issue:`7353`)
+
+Deprecations
+~~~~~~~~~~~~
+
+-   ``scrapy.mail.MailSender`` is deprecated. Please use :mod:`smtplib`,
+    :mod:`twisted.mail.smtp` or other 3rd party email libraries.
+    (:issue:`7249`, :issue:`7263`)
+
+-   The ``scrapy.extensions.statsmailer.StatsMailer`` extension is deprecated.
+    You can instead implement your own notifications by handling the
+    :signal:`spider_closed` signal.
+    (:issue:`7249`, :issue:`7263`)
+
+-   The ``MEMUSAGE_NOTIFY_MAIL`` setting is deprecated. You can instead
+    implement your own notifications by handling the
+    :signal:`memusage_warning_reached` and :signal:`spider_closed` signals.
+    (:issue:`7249`, :issue:`7263`)
+
+-   The ``DNS_RESOLVER`` setting was renamed to :setting:`TWISTED_DNS_RESOLVER`
+    and the old name is deprecated.
+    (:issue:`7350`, :issue:`7361`)
+
+-   The ``DOWNLOADER_CLIENTCONTEXTFACTORY`` setting is deprecated. If you were
+    using it to switch to
+    ``scrapy.core.downloader.contextfactory.BrowserLikeContextFactory``, please
+    use the new :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting instead. If you
+    cannot use the default context factory for some other reason, please
+    subclass the :ref:`download handler <download-handlers-ref>` instead.
+    (:issue:`7352`, :issue:`7379`)
+
+-   ``scrapy.core.downloader.contextfactory.BrowserLikeContextFactory`` is
+    deprecated. You can set the new :setting:`DOWNLOAD_VERIFY_CERTIFICATES`
+    setting to ``True`` instead.
+    (:issue:`7379`)
+
+-   The following implementation details of the context factory handling code
+    are deprecated:
+
+    - ``scrapy.core.downloader.contextfactory.AcceptableProtocolsContextFactory``
+
+    - ``scrapy.core.downloader.contextfactory.load_context_factory_from_settings()``
+
+    - ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory``
+
+    - ``scrapy.core.downloader.tls.ScrapyClientTLSOptions``
+
+    (:issue:`7353`, :issue:`7391`)
+
+-   Passing :class:`str` instead of :class:`bytes` to
+    :class:`scrapy.utils.sitemap.Sitemap` and
+    :func:`scrapy.utils.sitemap.sitemap_urls_from_robots` is deprecated.
+    (:issue:`7007`)
+
+-   ``scrapy.utils.misc.walk_modules()`` is deprecated. You can use
+    :func:`scrapy.utils.misc.walk_modules_iter` instead.
+    (:issue:`7388`)
+
+-   ``scrapy.shell.Shell.inthread`` is deprecated. You can use
+    :attr:`scrapy.shell.Shell.fetch_available` instead to check if
+    :func:`~scrapy.shell.Shell.fetch` can be used.
+    (:issue:`7395`)
+
+-   ``scrapy.commands.ScrapyCommand.set_crawler()`` is deprecated.
+    (:issue:`7276`)
+
 New features
 ~~~~~~~~~~~~

+-   Added an *experimental* mode for running Scrapy without installing a
+    Twisted reactor: set :setting:`TWISTED_REACTOR_ENABLED` to ``False`` to
+    enable it. This mode has limitations, refer to :ref:`its documentation
+    <asyncio-without-reactor>` for details. As long as it's experimental, its
+    behavior and related features and APIs may change in future Scrapy releases
+    in a breaking way.
+    (:issue:`6219`,
+    :issue:`7185`,
+    :issue:`7186`,
+    :issue:`7187`,
+    :issue:`7188`,
+    :issue:`7190`,
+    :issue:`7197`,
+    :issue:`7199`,
+    :issue:`7209`,
+    :issue:`7228`,
+    :issue:`7355`,
+    :issue:`7366`,
+    :issue:`7385`,
+    :issue:`7395`)
+
+-   Added the :func:`scrapy.utils.reactorless.is_reactorless` function that
+    checks if there is a running asyncio event loop but no Twisted reactor.
+    (:issue:`7185`, :issue:`7199`)
+
+-   Changed :func:`scrapy.utils.asyncio.is_asyncio_available` to return
+    ``True`` if there is a running asyncio loop, even if no Twisted reactor is
+    installed.
+    (:issue:`7185`, :issue:`7199`)
+
+-   Added an *experimental* download handler that uses the httpx_ library and
+    doesn't require a Twisted reactor:
+    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. As
+    long as it's experimental, its behavior may change in future Scrapy
+    releases in a breaking way.
+    (:issue:`6805`, :issue:`7239`, :issue:`7368`, :issue:`7384`)
+
+    .. _httpx: https://www.python-httpx.org/
+
+-   Added the :setting:`DOWNLOAD_BIND_ADDRESS` setting as a global counterpart
+    to the per-request :reqmeta:`bindaddress` meta key.
+    (:issue:`7266`, :issue:`7283`)
+
+-   Added the :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting that can be set
+    to ``True`` to make Scrapy abort HTTPS requests when the server certificate
+    is invalid or doesn't match the domain.
+    (:issue:`7379`)
+
+-   The built-in HTTP :ref:`download handlers <download-handlers-ref>` now
+    raise Scrapy-specific exceptions instead of implementation-specific ones,
+    to allow unified handling of similar problems caused by different
+    implementations. The default value of the :setting:`RETRY_EXCEPTIONS`
+    setting was updated replacing Twisted-specific exceptions with these new
+    ones. The exceptions:
+
+    - :exc:`~scrapy.exceptions.CannotResolveHostError`
+
+    - :exc:`~scrapy.exceptions.DownloadCancelledError`
+
+    - :exc:`~scrapy.exceptions.DownloadConnectionRefusedError`
+
+    - :exc:`~scrapy.exceptions.DownloadFailedError`
+
+    - :exc:`~scrapy.exceptions.DownloadTimeoutError`
+
+    - :exc:`~scrapy.exceptions.ResponseDataLossError`
+
+    - :exc:`~scrapy.exceptions.UnsupportedURLSchemeError`
+
+    (:issue:`7208`)
+
+-   Added the :signal:`memusage_warning_reached` signal emitted by the
+    :class:`~scrapy.extensions.memusage.MemoryUsage` extension when the memory
+    usage reaches :setting:`MEMUSAGE_WARNING_MB`.
+    (:issue:`7249`, :issue:`7263`)
+
 -   Added
    :meth:`Headers.to_tuple_list() <scrapy.http.headers.Headers.to_tuple_list>`
    that returns headers as a list of ``(key, value)`` tuples.
    (:issue:`7239`)

+-   :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler` now uses the
+    download handler configured for the ``"https"`` scheme to make requests
+    instead of always using
+    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.
+    (:issue:`7369`, :issue:`7370`)
+
+-   Added :func:`scrapy.utils.misc.walk_modules_iter` as a replacement for
+    ``scrapy.utils.misc.walk_modules()`` that returns an iterable instead of a
+    list.
+    (:issue:`7388`)
+
+Improvements
+~~~~~~~~~~~~
+
+-   :func:`asyncio.to_thread` is now used instead of
+    :func:`twisted.internet.threads.deferToThread` in the built-in feed
+    storages, media pipeline storages and the
+    :func:`scrapy.utils.decorators.inthread` decorator when available.
+    (:issue:`7183`, :issue:`7184`, :issue:`7349`)
+
+-   Improved memory footprint of :class:`~scrapy.Request` and
+    :class:`~scrapy.http.Response` objects by adding ``__slots__`` and omitting
+    empty lists and dicts in some internal attributes.
+    (:issue:`7036`, :issue:`7367`, :issue:`7374`)
+
+-   :class:`~scrapy.core.downloader.contextfactory._ScrapyClientContextFactory`
+    no longer mutates the SSL context, to avoid the behavior that was
+    deprecated in pyOpenSSL 25.1.0.
+    (:issue:`6859`, :issue:`7353`)
+
+-   Improved memory usage of :class:`~scrapy.spiders.sitemap.SitemapSpider` and
+    :class:`scrapy.utils.sitemap.Sitemap`.
+    (:issue:`3529`, :issue:`7007`)
+
+-   Improved the scheduling behavior of
+    :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` when crawling
+    multiple domains.
+    (:issue:`7293`, :issue:`7351`)
+
+-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
+    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now handle
+    TLS verbose logging (see :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING`)
+    directly instead of relying on
+    :class:`~scrapy.core.downloader.contextfactory._ScrapyClientContextFactory`.
+    (:issue:`7387`)
+
+-   The server certificate verification code now correctly handles certificates
+    with IP addresses in ``subjectAltName``.
+    (:issue:`7353`)
+
+-   Improved reliability of :func:`scrapy.utils.trackref.get_oldest`.
+    (:issue:`1758`, :issue:`7375`)
+
+-   Other code refactoring and improvements.
+    (:issue:`7210`, :issue:`7238`, :issue:`7376`, :issue:`7386`, :issue:`7395`,
+    :issue:`7405`, :issue:`7410`)
+
+Bug fixes
+~~~~~~~~~
+
+-   :ref:`Media pipelines <topics-media-pipeline>` should now wait for uploads
+    to asynchronous storages (e.g.
+    :class:`~scrapy.pipelines.files.S3FilesStore`) to complete.
+    (:issue:`2183`, :issue:`6369`, :issue:`7182`)
+
+-   Fixed merging ``*_BASE`` settings (e.g. merging
+    :setting:`DOWNLOADER_MIDDLEWARES` with
+    :setting:`DOWNLOADER_MIDDLEWARES_BASE`) when a component is referred to by
+    a class object in one setting and by a string import path in the other one.
+    (:issue:`6912`, :issue:`6993`)
+
+-   ``scrapy runspider`` and ``scrapy crawl`` now set the exit code to 1 if an
+    exception happened early (this was broken in an earlier Scrapy version).
+    (:issue:`6820`, :issue:`7255`)
+
+-   Fixed repeated warnings about data loss (see
+    :setting:`DOWNLOAD_FAIL_ON_DATALOSS`) not being suppressed in
+    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.
+    (:issue:`7222`)
+
+-   Improved FTP connection management in
+    :class:`scrapy.pipelines.files.FTPFilesStore`.
+    (:issue:`7256`)
+
+-   Fixed the ``spider`` variable in the :ref:`shell <topics-shell>`, which
+    wasn't available since Scrapy 2.13.0.
+    (:issue:`7395`)
+
+Documentation
+~~~~~~~~~~~~~
+
+-   The ``llms.txt`` and ``llms-full.txt`` files and Markdown versions of pages
+    are now generated when the HTML documentation is built.
+    (:issue:`7380`)
+
+-   Added a "Copy as Markdown" button to the HTML documentation.
+    (:issue:`7380`)
+
+-   Added :ref:`docs for using Pydantic models as items <pydantic-items>`.
+    (:issue:`6955`, :issue:`6966`)
+
+-   Documented :ref:`job directory contents <job-dir-contents>`.
+    (:issue:`4842`, :issue:`5260`)
+
+-   Improved docs for :attr:`~scrapy.Request.dont_filter`.
+    (:issue:`6398`, :issue:`7245`)
+
+-   Clarified that settings related to :setting:`TWISTED_DNS_RESOLVER` are only
+    taken into account if the selected resolver supports them.
+    (:issue:`7385`)
+
+-   Other documentation improvements and fixes.
+    (:issue:`7248`, :issue:`7274`, :issue:`7406`, :issue:`7408`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+-   Added the ``no-reactor`` test environment that doesn't install a Twisted
+    reactor and uses ``pytest-asyncio`` instead of ``pytest-twisted`` to run
+    asynchronous test functions.
+    (:issue:`6952`, :issue:`7189`, :issue:`7233`, :issue:`7234`, :issue:`7254`,
+    :issue:`7259`)
+
+-   Fixed running tests with ``pytest-xdist``.
+    (:issue:`7216`, :issue:`7257`)
+
+-   Type hints improvements and fixes.
+    (:issue:`7300`, :issue:`7331`)
+
+-   CI and test improvements and fixes.
+    (:issue:`7060`,
+    :issue:`7223`,
+    :issue:`7232`,
+    :issue:`7241`,
+    :issue:`7250`,
+    :issue:`7256`,
+    :issue:`7276`,
+    :issue:`7277`,
+    :issue:`7279`,
+    :issue:`7329`,
+    :issue:`7363`,
+    :issue:`7381`,
+    :issue:`7402`)
+
 .. _release-2.14.2:

 Scrapy 2.14.2 (2026-03-12)
@ -4939,7 +5248,7 @@ New features
    components already supported (:issue:`4126`)

 *   :class:`scrapy.utils.python.MutableChain.__iter__` now returns ``self``,
-    `allowing it to be used as a sequence <https://lgtm.com/rules/4850080/>`_
+    allowing it to be used as a sequence.
    (:issue:`4153`)


@ -6880,8 +7189,7 @@ Keep reading for more details on other improvements and bug fixes.
 Beta Python 3 Support
 ~~~~~~~~~~~~~~~~~~~~~

-We have been `hard at work to make Scrapy run on Python 3
-<https://github.com/scrapy/scrapy/wiki/Python-3-Porting>`_. As a result, now
+We have been hard at work to make Scrapy run on Python 3. As a result, now
 you can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Some
 features are still missing (and some may never be ported).

@ -6949,7 +7257,7 @@ Additional New Features and Enhancements
 - Other refactoring, optimizations and cleanup (:issue:`1476`, :issue:`1481`,
  :issue:`1477`, :issue:`1315`, :issue:`1290`, :issue:`1750`, :issue:`1881`).

-.. _`Code of Conduct`: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
+.. _Code of Conduct: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md


 Deprecations and Removals
--- a/docs/topics/download-handlers.rst
+++ b/docs/topics/download-handlers.rst
@ -102,6 +102,32 @@ An optional base class for custom handlers is provided:
    :undoc-members:
    :member-order: bysource

+.. _download-handlers-exceptions:
+
+Exceptions raised by download handlers
+======================================
+
+.. versionadded:: 2.15.0
+
+The built-in download handlers raise Scrapy-specific exceptions instead of
+implementation-specific ones, so that code that handles these exceptions can be
+written in a generic way. We recommend custom download handlers to also use
+these exceptions.
+
+.. autoexception:: scrapy.exceptions.CannotResolveHostError
+
+.. autoexception:: scrapy.exceptions.DownloadCancelledError
+
+.. autoexception:: scrapy.exceptions.DownloadConnectionRefusedError
+
+.. autoexception:: scrapy.exceptions.DownloadFailedError
+
+.. autoexception:: scrapy.exceptions.DownloadTimeoutError
+
+.. autoexception:: scrapy.exceptions.ResponseDataLossError
+
+.. autoexception:: scrapy.exceptions.UnsupportedURLSchemeError
+
 .. _download-handlers-ref:

 Built-in download handlers reference
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@ -1103,7 +1103,7 @@ Parsers vary in several aspects:

 * Support for wildcard matching

-* Usage of `length based rule <https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#order-of-precedence-for-rules>`_:
+* Usage of `length based rule <https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec#order-of-precedence-for-rules>`_:
  in particular for ``Allow`` and ``Disallow`` directives, where the most
  specific rule based on the length of the path trumps the less specific
  (shorter) rule
@ -1121,7 +1121,7 @@ Based on `Protego <https://github.com/scrapy/protego>`_:
 * implemented in Python

 * is compliant with `Google's Robots.txt Specification
-  <https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt>`_
+  <https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec>`_

 * supports wildcard matching

--- a/docs/topics/dynamic-content.rst
+++ b/docs/topics/dynamic-content.rst
@ -274,16 +274,13 @@ However, using `playwright-python`_ directly as in the above example
 circumvents most of the Scrapy components (middlewares, dupefilter, etc).
 We recommend using `scrapy-playwright`_ for a better integration.

-.. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29
 .. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets
-.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
 .. _chompjs: https://github.com/Nykakin/chompjs
 .. _curl: https://curl.se/
 .. _headless browser: https://en.wikipedia.org/wiki/Headless_browser
 .. _js2xml: https://github.com/scrapinghub/js2xml
 .. _playwright-python: https://github.com/microsoft/playwright-python
 .. _playwright: https://github.com/microsoft/playwright
-.. _pyppeteer: https://pyppeteer.github.io/pyppeteer/
 .. _pytesseract: https://github.com/madmaze/pytesseract
 .. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright
 .. _tabula-py: https://github.com/chezou/tabula-py
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@ -246,7 +246,7 @@ The feeds are stored on `Google Cloud Storage`_.

 -   Required external libraries: `google-cloud-storage`_.

-For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication>`_.
+For more information about authentication, please refer to `Google Cloud documentation <https://docs.cloud.google.com/docs/authentication>`_.

 You can set a *Project ID* and *Access Control List (ACL)* through the following settings:

@ -261,7 +261,7 @@ storage backend is: ``True``.

 This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.

-.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
+.. _google-cloud-storage: https://docs.cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python


 .. _topics-feed-storage-stdout:
--- a/docs/topics/link-extractors.rst
+++ b/docs/topics/link-extractors.rst
@ -159,5 +159,3 @@ Link
   :synopsis: Link from link extractors

 .. autoclass:: Link
-
-.. _scrapy.linkextractors: https://github.com/scrapy/scrapy/blob/master/scrapy/linkextractors/__init__.py
--- a/docs/topics/loaders.rst
+++ b/docs/topics/loaders.rst
@ -452,4 +452,3 @@ organization of your Loaders collection - that's up to you and your project's
 needs.

 .. _itemloaders: https://itemloaders.readthedocs.io/en/latest/
-.. _processors: https://itemloaders.readthedocs.io/en/latest/built-in-processors.html
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@ -290,7 +290,7 @@ Google Cloud Storage
 :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can represent a Google Cloud Storage
 bucket. Scrapy will automatically upload the files to the bucket. (requires `google-cloud-storage`_ )

-.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
+.. _google-cloud-storage: https://docs.cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python

 For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_ID` settings:

@ -301,7 +301,7 @@ For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_I

 For information about authentication, see this `documentation`_.

-.. _documentation: https://cloud.google.com/docs/authentication
+.. _documentation: https://docs.cloud.google.com/docs/authentication

 You can modify the Access Control List (ACL) policy used for the stored files,
 which is defined by the :setting:`FILES_STORE_GCS_ACL` and
@ -316,7 +316,7 @@ policy:

 For more information, see `Predefined ACLs`_ in the Google Cloud Platform Developer Guide.

-.. _Predefined ACLs: https://cloud.google.com/storage/docs/access-control/lists#predefined-acl
+.. _Predefined ACLs: https://docs.cloud.google.com/storage/docs/access-control/lists#predefined-acl

 Usage example
 =============
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@ -417,7 +417,7 @@ If you are still unable to prevent your bot getting banned, consider contacting
 `commercial support`_.

 .. _Tor project: https://www.torproject.org/
-.. _commercial support: https://scrapy.org/support/
+.. _commercial support: https://www.scrapy.org/companies
 .. _ProxyMesh: https://proxymesh.com/
 .. _Common Crawl: https://commoncrawl.org/
 .. _testspiders: https://github.com/scrapinghub/testspiders
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@ -543,7 +543,7 @@ you may want to take a look first at this `XPath tutorial`_.
 .. note::
    Some of the tips are based on `this post from Zyte's blog`_.

-.. _`XPath tutorial`: http://www.zvon.org/comp/r/tut-XPath_1.html
+.. _XPath tutorial: http://www.zvon.org/comp/r/tut-XPath_1.html
 .. _this post from Zyte's blog: https://www.zyte.com/blog/xpath-tips-from-the-web-scraping-trenches/


@ -728,7 +728,7 @@ But using the ``.`` to mean the node, works:
    >>> sel.xpath("//a[contains(., 'Next Page')]").getall()
    ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']

-.. _`XPath string function`: https://www.w3.org/TR/xpath-10/#section-String-Functions
+.. _XPath string function: https://www.w3.org/TR/xpath-10/#section-String-Functions

 .. _topics-selectors-xpath-variables:

@ -983,9 +983,9 @@ Here we first iterate over ``itemscope`` elements, and for each one,
 we look for all ``itemprops`` elements and exclude those that are themselves
 inside another ``itemscope``.

-.. _EXSLT: http://exslt.org/
-.. _regular expressions: http://exslt.org/regexp/index.html
-.. _set manipulation: http://exslt.org/set/index.html
+.. _EXSLT: https://exslt.github.io/
+.. _regular expressions: https://exslt.github.io/regexp/index.html
+.. _set manipulation: https://exslt.github.io/set/index.html

 Other XPath extensions
 ----------------------
@ -1190,4 +1190,4 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object:

 .. skip: end

-.. _Google Base XML feed: https://support.google.com/merchants/answer/160589?hl=en&ref_topic=2473799
+.. _Google Base XML feed: https://support.google.com/merchants/answer/14987622
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@ -787,7 +787,7 @@ the TLS-related libraries.
 DOWNLOADER_MIDDLEWARES
 ----------------------

-Default:: ``{}``
+Default: ``{}``

 A dict containing the downloader middlewares enabled in your project, and their
 orders. For more info see :ref:`topics-downloader-middleware-setting`.
@ -1239,7 +1239,7 @@ command will prefer it over the default setting.
 EXTENSIONS
 ----------

-Default:: ``{}``
+Default: ``{}``

 :ref:`Component priority dictionary <component-priority-dictionaries>` of
 enabled extensions. See :ref:`topics-extensions`.
@ -1287,7 +1287,7 @@ FEED_STORAGE_GCS_ACL
 --------------------

 The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
-For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://cloud.google.com/storage/docs/access-control/lists>`_.
+For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://docs.cloud.google.com/storage/docs/access-control/lists>`_.

 .. setting:: FORCE_CRAWLER_PROCESS

@ -1861,7 +1861,7 @@ Scrapy does not process new requests.
 SPIDER_CONTRACTS
 ----------------

-Default:: ``{}``
+Default: ``{}``

 A dict containing the spider contracts enabled in your project, used for
 testing spiders. For more info see :ref:`topics-contracts`.
@ -1922,7 +1922,7 @@ warning by setting ``SPIDER_LOADER_WARN_ONLY = True``.
 SPIDER_MIDDLEWARES
 ------------------

-Default:: ``{}``
+Default: ``{}``

 A dict containing the spider middlewares enabled in your project, and their
 orders. For more info see :ref:`topics-spider-middleware-setting`.
@ -2162,7 +2162,7 @@ Use ``0`` to allow URLs of any length.
 The default value is copied from the `Microsoft Internet Explorer maximum URL
 length`_, even though this setting exists for different reasons.

-.. _Microsoft Internet Explorer maximum URL length: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
+.. _Microsoft Internet Explorer maximum URL length: https://web.archive.org/web/20250206050143/https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246

 .. setting:: USER_AGENT

@ -2201,6 +2201,4 @@ case to see how to enable and use them.
 .. settingslist::

 .. _Amazon web services: https://aws.amazon.com/
-.. _breadth-first order: https://en.wikipedia.org/wiki/Breadth-first_search
-.. _depth-first order: https://en.wikipedia.org/wiki/Depth-first_search
 .. _Google Cloud Storage: https://cloud.google.com/storage/
--- a/docs/topics/shell.rst
+++ b/docs/topics/shell.rst
@ -40,7 +40,7 @@ variable; or by defining it in your :ref:`scrapy.cfg <topics-config-settings>`::
    shell = bpython

 .. _IPython: https://ipython.org/
-.. _IPython installation guide: https://ipython.org/install.html
+.. _IPython installation guide: https://ipython.org/install/
 .. _bpython: https://bpython-interpreter.org/

 Launch the shell
@ -111,7 +111,7 @@ Available Shortcuts
    Note, however, that this will create a temporary file in your computer,
    which won't be removed automatically.

-.. _<base> tag: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/base
+.. _<base> tag: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/base

 Available Scrapy objects
 ------------------------
@ -145,7 +145,7 @@ Example of shell session
 .. skip: start

 Here's an example of a typical shell session where we start by scraping the
-https://scrapy.org page, and then proceed to scrape the https://old.reddit.com/
+https://www.scrapy.org/ page, and then proceed to scrape the https://old.reddit.com/
 page. Finally, we modify the (Reddit) request method to POST and re-fetch it
 getting an error. We end the session by typing Ctrl-D (in Unix systems) or
 Ctrl-Z in Windows.
--- a/docs/versioning.rst
+++ b/docs/versioning.rst
@ -23,7 +23,7 @@ Development releases do not follow 3-numbers version and are generally
 released as ``dev`` suffixed versions, e.g. ``1.3dev``.

 .. note::
-    With Scrapy 0.* series, Scrapy used `odd-numbered versions for development releases`_.
+    With Scrapy 0.* series, Scrapy used odd-numbered versions for development releases.
    This is not the case anymore from Scrapy 1.0 onwards.

    Starting with Scrapy 1.0, all releases should be considered production-ready.
@ -63,6 +63,3 @@ feature.

 All deprecated features removed in a Scrapy release are explicitly mentioned in
 the :ref:`release notes <news>`.
-
-
-.. _odd-numbered versions for development releases: https://en.wikipedia.org/wiki/Software_versioning#Odd-numbered_versions_for_development_releases
--- a/scrapy/core/downloader/handlers/_httpx.py
+++ b/scrapy/core/downloader/handlers/_httpx.py
@ -90,7 +90,7 @@ class HttpxDownloadHandler(BaseHttpDownloadHandler):
            )
        super().__init__(crawler)
        logger.warning(
-            "HttpxDownloadHandler is experimental and is not recommented for production use."
+            "HttpxDownloadHandler is experimental and is not recommended for production use."
        )
        bind_address = crawler.settings.get("DOWNLOAD_BIND_ADDRESS")
        bind_address = normalize_bind_address(bind_address)
--- a/scrapy/utils/asyncio.py
+++ b/scrapy/utils/asyncio.py
@ -65,7 +65,7 @@ def is_asyncio_available() -> bool:
        calling it from code such as spiders and Scrapy components, if Scrapy
        is run using one of the supported ways).

-    .. versionchanged:: VERSION
+    .. versionchanged:: 2.15.0
        This function now also returns ``True`` if there is a running asyncio
        loop, even if no Twisted reactor is installed.
    """
@ -302,7 +302,7 @@ async def run_in_thread(
    :func:`twisted.internet.threads.deferToThread`, depending on whether
    asyncio support is available.

-    .. versionadded:: VERSION
+    .. versionadded:: 2.15.0
    """
    if is_asyncio_available():
        return await asyncio.to_thread(func, *args, **kwargs)
--- a/scrapy/utils/decorators.py
+++ b/scrapy/utils/decorators.py
@ -62,7 +62,7 @@ def inthread(func: Callable[_P, _T]) -> Callable[_P, Deferred[_T]]:
    """Decorator to call a function in a thread and return a deferred with the
    result.

-    .. versionchanged:: VERSION
+    .. versionchanged:: 2.15.0
        Now uses :func:`asyncio.to_thread` if the asyncio support is available.
    """

--- a/scrapy/utils/reactorless.py
+++ b/scrapy/utils/reactorless.py
@ -25,7 +25,7 @@ def is_reactorless() -> bool:
        :func:`scrapy.utils.asyncio.is_asyncio_available()`, it has the same
        limitations for detecting a running asyncio event loop as that one.

-    .. versionadded:: VERSION
+    .. versionadded:: 2.15.0
    """
    return is_asyncio_available() and not is_reactor_installed()

--- a/scrapy/utils/sitemap.py
+++ b/scrapy/utils/sitemap.py
@ -117,6 +117,11 @@ def sitemap_urls_from_robots(
                yield urljoin(base_url or "", url)

    else:
+        warnings.warn(
+            "Passing `str` type as `robots_text` is deprecated, use `bytes`",
+            ScrapyDeprecationWarning,
+            stacklevel=2,
+        )
        yield from _sitemap_urls_from_robots_str(robots_text, base_url)


@ -124,11 +129,6 @@ def _sitemap_urls_from_robots_str(
    robots_text: str,
    base_url: str | None = None,
 ) -> Iterable[str]:
-    warnings.warn(
-        "Passing `str` type as `robots_text` is deprecated, use `bytes`",
-        ScrapyDeprecationWarning,
-        stacklevel=2,
-    )
    for line in StringIO(robots_text):
        if line.lstrip()[:8].lower() == "sitemap:":
            url = line.partition(":")[2].strip()