diff --git a/docs/news.rst b/docs/news.rst index 64a3ad2b1..d574317bb 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,38 +3,59 @@ Release notes ============= -.. _release-VERSION: +.. _release-2.13.0: -Scrapy VERSION (unreleased) ---------------------------- +Scrapy 2.13.0 (unreleased) +-------------------------- Highlights: +- The asyncio reactor is now enabled by default + - Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start` (async) and changed how it is iterated. +- Added the :reqmeta:`allow_offsite` request meta key + +- :ref:`Spider middlewares that don't support asynchronous spider output + ` are deprecated + +- Added a base class for :ref:`universal spider middlewares + ` + +Modified requirements +~~~~~~~~~~~~~~~~~~~~~ + +- Dropped support for PyPy 3.9. + (:issue:`6613`) + +- Added support for PyPy 3.11. + (:issue:`6697`) + Backward-incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- The ``from_settings()`` method of - :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is removed - without a deprecation period (this was needed because after the - introduction of the - :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and - switching built-in spider middlewares to it those middlewares need the - :class:`~scrapy.crawler.Crawler` instance at run time). Please use - ``from_crawler()`` instead. +- The default value of the :setting:`TWISTED_REACTOR` setting was changed + from ``None`` to + ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value + was used in newly generated projects since Scrapy 2.7.0 but now existing + projects that don't explicitly set this setting will also use the asyncio + reactor. You can :ref:`change this setting in your project + ` to use a different reactor. + (:issue:`6659`, :issue:`6713`) - The iteration of start requests and items no longer stops once there are requests in the scheduler, and instead runs continuously until all start requests have been scheduled. To reproduce the previous behavior, see :ref:`start-requests-lazy`. + (:issue:`6729`) - An unhandled exception from the :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.open_spider` method of a :ref:`spider middleware ` no longer stops the crawl. + (:issue:`6729`) - In ``scrapy.core.engine.ExecutionEngine``: @@ -45,23 +66,67 @@ Backward-incompatible changes - The ``slot`` attribute has been renamed to ``_slot`` and should not be used. + (:issue:`6729`) + - In ``scrapy.core.engine``, the ``Slot`` class has been renamed to ``_Slot`` and should not be used. + (:issue:`6729`) - The ``slot`` :ref:`telnet variable ` has been removed. + (:issue:`6729`) - In ``scrapy.core.spidermw.SpiderMiddlewareManager``, ``process_start_requests()`` has been replaced by ``process_start()``. + (:issue:`6729`) - The now-deprecated ``start_requests()`` method, when it returns an iterable instead of being defined as a generator, is now executed *after* the :ref:`scheduler ` instance has been created. + (:issue:`6729`) - When using :setting:`JOBDIR`, :ref:`start requests ` are now serialized into their own, ``s``-suffixed priority folders. You can set :setting:`SCHEDULER_START_DISK_QUEUE` to ``None`` or ``""`` to change that, but the side effects may be undesirable. See :setting:`SCHEDULER_START_DISK_QUEUE` for details. + (:issue:`6729`) + +- The URL length limit, set by the :setting:`URLLENGTH_LIMIT` setting, is now + also enforced for start requests. + (:issue:`6777`) + +- The ``from_settings()`` method of + :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`, + deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation + period (this was needed because after the introduction of the + :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and + switching built-in spider middlewares to it those middlewares need the + :class:`~scrapy.crawler.Crawler` instance at run time). Please use + ``from_crawler()`` instead. + (:issue:`6693`) + +- ``scrapy.utils.url.escape_ajax()`` is no longer called when a + :class:`~scrapy.Request` instance is created. It was only useful for + websites supporting the ``_escaped_fragment_`` feature which most modern + websites don't support. If you still need this you can modify the URLs + before passing them to :class:`~scrapy.Request`. + (:issue:`6523`, :issue:`6651`) + +Deprecation removals +~~~~~~~~~~~~~~~~~~~~ + +- Removed old deprecated name aliases for some signals: + + - ``stats_spider_opened`` (use ``spider_opened`` instead) + + - ``stats_spider_closing`` and ``stats_spider_closed`` (use + ``spider_closed`` instead) + + - ``item_passed`` (use ``item_scraped`` instead) + + - ``request_received`` (use ``request_scheduled`` instead) + + (:issue:`6654`, :issue:`6655`) Deprecations ~~~~~~~~~~~~ @@ -69,22 +134,132 @@ Deprecations - The ``start_requests()`` method of :class:`~scrapy.Spider` is deprecated, use :meth:`~scrapy.Spider.start` instead, or both to maintain support for lower Scrapy versions. - (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`) - The ``process_start_requests()`` method of :ref:`spider middlewares ` is deprecated, use :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead, or both to maintain support for lower Scrapy versions. - (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`) - The ``__init__`` method of priority queue classes (see :setting:`SCHEDULER_PRIORITY_QUEUE`) should now support a keyword-only ``start_queue_cls`` parameter. - (:issue:`6752`) +- :ref:`Spider middlewares that don't support asynchronous spider output + ` are deprecated. The async iterable + downgrading feature, needed for using such middlewares with asynchronous + callbacks and with other spider middlewares that produce asynchronous + iterables, is also deprecated. Please update all such middlewares to + support asynchronous spider output. + (:issue:`6664`) + +- Functions that were imported from :mod:`w3lib.url` and re-exported in + :mod:`scrapy.utils.url` are now deprecated, you should import them from + :mod:`w3lib.url` directly. They are: + + - ``scrapy.utils.url.add_or_replace_parameter()`` + + - ``scrapy.utils.url.add_or_replace_parameters()`` + + - ``scrapy.utils.url.any_to_uri()`` + + - ``scrapy.utils.url.canonicalize_url()`` + + - ``scrapy.utils.url.file_uri_to_path()`` + + - ``scrapy.utils.url.is_url()`` + + - ``scrapy.utils.url.parse_data_uri()`` + + - ``scrapy.utils.url.parse_url()`` + + - ``scrapy.utils.url.path_to_file_uri()`` + + - ``scrapy.utils.url.safe_download_url()`` + + - ``scrapy.utils.url.safe_url_string()`` + + - ``scrapy.utils.url.url_query_cleaner()`` + + - ``scrapy.utils.url.url_query_parameter()`` + + (:issue:`4577`, :issue:`6583`, :issue:`6586`) + +- HTTP/1.0 support code is deprecated. It was disabled by default and + couldn't be used together with HTTP/1.1. If you still need it, you should + write your own download handler or copy the code from Scrapy. The + deprecations include: + + - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler`` + + - ``scrapy.core.downloader.webclient.ScrapyHTTPClientFactory`` + + - ``scrapy.core.downloader.webclient.ScrapyHTTPPageGetter`` + + - Overriding + ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory.getContext()`` + + (:issue:`6634`) + +- The following modules and functions used only in tests are deprecated: + + - the ``scrapy/utils/testproc`` module + + - the ``scrapy/utils/testsite`` module + + - ``scrapy.utils.test.assert_gcs_environ()`` + + - ``scrapy.utils.test.get_ftp_content_and_delete()`` + + - ``scrapy.utils.test.get_gcs_content_and_delete()`` + + - ``scrapy.utils.test.mock_google_cloud_storage()`` + + - ``scrapy.utils.test.skip_if_no_boto()`` + + If you need to use them in your tests or code, you can copy the code from Scrapy. + (:issue:`6696`) + +- ``scrapy.utils.test.TestSpider`` is deprecated. If you need an empty spider + class you can use :class:`scrapy.utils.spider.DefaultSpider` or create your + own subclass of :class:`scrapy.Spider`. + (:issue:`6678`) + +- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` is + deprecated. It was disabled by default and isn't useful for most of the + existing websites. + (:issue:`6523`, :issue:`6651`, :issue:`6656`) + +- ``scrapy.utils.url.escape_ajax()`` is deprecated. + (:issue:`6523`, :issue:`6651`) + +- ``scrapy.spiders.init.InitSpider`` is deprecated. If you find it useful, + you can copy its code from Scrapy. + (:issue:`6708`, :issue:`6714`) + +- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use + :func:`scrapy.utils.versions.get_versions()` instead. + (:issue:`6582`) + +- ``BaseDupeFilter.log()`` is deprecated. It does nothing and shouldn't be + called. + (:issue:`4151`) + +- Passing the ``spider`` argument to the following methods of + :class:`~scrapy.core.scraper.Scraper` is deprecated: + + - ``close_spider()`` + + - ``enqueue_scrape()`` + + - ``handle_spider_error()`` + + - ``handle_spider_output()`` + + (:issue:`6764`) + New features ~~~~~~~~~~~~ @@ -97,7 +272,6 @@ New features This makes it possible to use asynchronous code to generate those start requests and items, e.g. reading them from a queue service or database using an asynchronous client, without workarounds. - (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`) - Start requests are now :ref:`scheduled ` as soon as @@ -105,24 +279,95 @@ New features As a result, their :attr:`~scrapy.Request.priority` is now taken into account as soon as :setting:`CONCURRENT_REQUESTS` is reached. - (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`) - :class:`Crawler.signals ` has a new :meth:`~scrapy.signalmanager.SignalManager.wait_for` method. + (:issue:`6729`) - Added a new :signal:`scheduler_empty` signal. + (:issue:`6729`) - Added new settings: :setting:`SCHEDULER_START_DISK_QUEUE` and :setting:`SCHEDULER_START_MEMORY_QUEUE`. + (:issue:`6729`) - Added :class:`~scrapy.spidermiddlewares.start.StartSpiderMiddleware`, which sets :reqmeta:`is_start_request` to ``True`` on :ref:`start requests `. + (:issue:`6729`) - Exposed a new method of :class:`Crawler.engine `: :meth:`~scrapy.core.engine.ExecutionEngine.needs_backout`. + (:issue:`6729`) + +- Added the :reqmeta:`allow_offsite` request meta key that can be used + instead of the more general :attr:`~scrapy.Request.dont_filter` request + attribute to skip processing of the request by + :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` (but not + by other code that checks :attr:`~scrapy.Request.dont_filter`). + (:issue:`3690`, :issue:`6151`, :issue:`6366`) + +- Added an optional base class for spider middlewares, + :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which can be + helpful for writing :ref:`universal spider middlewares + ` without boilerplate and code duplication. + The built-in spider middlewares now inherit from this class. + (:issue:`6693`, :issue:`6777`) + +- :ref:`Scrapy add-ons ` can now define a class method called + ``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings + `. + (:issue:`6544`, :issue:`6568`) + +- Added :ref:`helpers ` for modifying :ref:`component + priority dictionary ` settings. + (:issue:`6614`) + +- Responses that use an unknown/unsupported encoding now produce a warning. + If Scrapy knows that installing an additional package (such as brotli_) + will allow decoding the response, that will be mentioned in the warning. + (:issue:`4697`, :issue:`6618`) + +- Added the ``spider_exceptions/count`` stat which tracks the total count of + exceptions (tracked also by per-type ``spider_exceptions/*`` stats). + (:issue:`6739`, :issue:`6740`) + +- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the + :attr:`scrapy.exceptions.DropItem.log_level` attribute that allow + customizing the log level of the message that is logged when an item is + dropped. + (:issue:`6603`, :issue:`6608`) + +- Added support for the ``-b, --cookie`` curl argument to + :meth:`scrapy.Request.from_curl`. + (:issue:`6684`) + +- Added the :setting:`LOG_VERSIONS` setting that allows customizing the + list of software whose versions are logged when the spider starts. + (:issue:`6582`) + +- Added the :setting:`WARN_ON_GENERATOR_RETURN_VALUE` setting that allows + disabling run time analysis of callback code used to warn about incorrect + ``return`` statements in generator-based callbacks. You may need to disable + this setting if this analysis breaks on your callback code. + (:issue:`6731`, :issue:`6738`) + +Improvements +~~~~~~~~~~~~ + +- Removed or postponed some calls of :func:`itemadapter.is_item` to increase + performance. + (:issue:`6719`) + +- Improved the error message when running a ``scrapy`` command that requires + a project (such as ``scrapy crawl``) outside of a project directory. + (:issue:`2349`, :issue:`3426`) + +- Added an empty :setting:`ADDONS` setting to the ``settings.py`` template + for new projects. + (:issue:`6587`) Bug fixes ~~~~~~~~~ @@ -131,9 +376,166 @@ Bug fixes :meth:`SpiderMiddleware.process_start ` no longer delays the next iteration of starting requests and items by up to 5 seconds. - (:issue:`6729`) +- Fixed calculation of ``items_per_minute`` and ``responses_per_minute`` + stats. + (:issue:`6599`) + +- Fixed an error initializing + :class:`scrapy.extensions.feedexport.GCSFeedStorage`. + (:issue:`6617`, :issue:`6628`) + +- Fixed an error running ``scrapy bench``. + (:issue:`6632`, :issue:`6633`) + +- Fixed duplicated log messages about the reactor and the event loop. + (:issue:`6636`, :issue:`6657`) + +- Fixed resolving type annotations of ``SitemapSpider._parse_sitemap()`` at + run time, required by tools such as scrapy-poet_. + (:issue:`6665`, :issue:`6671`) + + .. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet + +- Calling :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` without + an installed reactor now raises an exception instead of installing a + reactor. + (:issue:`6732`, :issue:`6735`) + +- Restored support for the ``x-gzip`` content encoding. + (:issue:`6618`) + +Documentation +~~~~~~~~~~~~~ + +- Documented the setting values set in the default project template. + (:issue:`6762`, :issue:`6775`) + +- Improved the :ref:`docs ` about asynchronous + iterable support in spider middlewares. + (:issue:`6688`) + +- Improved the :ref:`docs ` about using + :class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based + code and included a list of such APIs. + (:issue:`6677`, :issue:`6734`, :issue:`6776`) + +- Improved the :ref:`contribution docs `. + (:issue:`6561`, :issue:`6575`) + +- Removed the ``Splash`` recommendation from the :ref:`headless browser + ` suggestion. We no longer recommend using + ``Splash`` and recommend using other headless browser solutions instead. + (:issue:`6642`, :issue:`6701`) + +- Added the dark mode to the HTML documentation. + (:issue:`6653`) + +- Other documentation improvements and fixes. + (:issue:`4151`, + :issue:`6526`, + :issue:`6620`, + :issue:`6621`, + :issue:`6622`, + :issue:`6623`, + :issue:`6624`, + :issue:`6721`, + :issue:`6723`, + :issue:`6780`) + +Packaging +~~~~~~~~~ + +- Switched from ``setup.py`` to ``pyproject.toml``. + (:issue:`6514`, :issue:`6547`) + +- Switched the build backend from setuptools_ to hatchling_. + (:issue:`6771`) + + .. _hatchling: https://pypi.org/project/hatchling/ + +Quality assurance +~~~~~~~~~~~~~~~~~ + +- Replaced most linters with ruff_. + (:issue:`6565`, + :issue:`6576`, + :issue:`6577`, + :issue:`6581`, + :issue:`6584`, + :issue:`6595`, + :issue:`6601`, + :issue:`6631`) + + .. _ruff: https://docs.astral.sh/ruff/ + +- Improved accuracy and performance of collecting test coverage. + (:issue:`6255`, :issue:`6610`) + +- Fixed an error that prevented running tests from directories other than the + top level source directory. + (:issue:`6567`) + +- Reduced the amount of ``mockserver`` calls in tests to improve the overall + test run time. + (:issue:`6637`, :issue:`6648`) + +- Fixed tests that were running the same test code more than once. + (:issue:`6646`, :issue:`6647`, :issue:`6650`) + +- Refactored tests to use more ``pytest`` features instead of ``unittest`` + ones where possible. + (:issue:`6678`, + :issue:`6680`, + :issue:`6695`, + :issue:`6699`, + :issue:`6700`, + :issue:`6702`, + :issue:`6709`, + :issue:`6710`, + :issue:`6711`, + :issue:`6712`, + :issue:`6725`) + +- Type hints improvements and fixes. + (:issue:`6578`, + :issue:`6579`, + :issue:`6593`, + :issue:`6605`, + :issue:`6694`) + +- CI and test improvements and fixes. + (:issue:`5360`, + :issue:`6271`, + :issue:`6547`, + :issue:`6560`, + :issue:`6602`, + :issue:`6607`, + :issue:`6609`, + :issue:`6613`, + :issue:`6619`, + :issue:`6626`, + :issue:`6679`, + :issue:`6703`, + :issue:`6704`, + :issue:`6716`, + :issue:`6720`, + :issue:`6722`, + :issue:`6724`, + :issue:`6741`, + :issue:`6743`, + :issue:`6766`, + :issue:`6770`, + :issue:`6772`, + :issue:`6773`) + +- Code cleanups. + (:issue:`6600`, + :issue:`6606`, + :issue:`6635`, + :issue:`6764`) + .. _release-2.12.0: @@ -737,8 +1139,6 @@ Bug fixes - Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in favor of brotli_. (:issue:`6261`) - .. _brotli: https://github.com/google/brotli - .. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli instead if you can. @@ -2322,8 +2722,6 @@ Scrapy 2.5.1 (2021-10-05) need to upgrade scrapy-splash to a greater version for it to continue to work. -.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash - .. _release-2.5.0: diff --git a/docs/topics/addons.rst b/docs/topics/addons.rst index 17e3c177a..815501e66 100644 --- a/docs/topics/addons.rst +++ b/docs/topics/addons.rst @@ -124,6 +124,8 @@ Set some basic configuration: "ITEM_PIPELINES", MyPipeline, 200 ) +.. _priority-dict-helpers: + .. tip:: When editing a :ref:`component priority dictionary ` setting, like :setting:`ITEM_PIPELINES`, consider using setting methods like diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 448bf07e7..00812ed7f 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -21,7 +21,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): - The :meth:`~scrapy.spiders.Spider.start` spider method, which *must* be defined as an :term:`asynchronous generator`. - .. versionadded: VERSION + .. versionadded: 2.13 - :class:`~scrapy.Request` callbacks. @@ -59,7 +59,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): of :ref:`spider middlewares `, which *must* be defined as an :term:`asynchronous generator`. - .. versionadded:: VERSION + .. versionadded:: 2.13 - :ref:`Signal handlers that support deferreds `. @@ -441,3 +441,9 @@ For example: feature will be removed, and all spider middlewares will be expected to define their ``process_spider_output`` method as an asynchronous generator. + +Since 2.13.0, Scrapy provides a base class, +:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements +the ``process_spider_output()`` and ``process_spider_output_async()`` methods, +so instead of duplicating the processing code you can override the +``get_processed_request()`` and/or the ``get_processed_item()`` method. diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 537e51e40..0f81a0c0a 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -710,7 +710,8 @@ connections (for ``HTTP10DownloadHandler``). .. note:: - HTTP/1.0 is rarely used nowadays so you can safely ignore this setting, + HTTP/1.0 is rarely used nowadays and its Scrapy support is deprecated, + so you can safely ignore this setting, unless you really want to use HTTP/1.0 and override :setting:`DOWNLOAD_HANDLERS` for ``http(s)`` scheme accordingly, i.e. to ``'scrapy.core.downloader.handlers.http.HTTP10DownloadHandler'``. @@ -2064,7 +2065,7 @@ current platform. ``twisted.internet.asyncioreactor.AsyncioSelectorReactor`` in the generated ``settings.py`` file. -.. versionchanged:: VERSION +.. versionchanged:: 2.13 The default value was changed from ``None`` to ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 638035e64..67178b8fd 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -86,7 +86,7 @@ one or more of these methods: You may yield the same type of objects as :meth:`~scrapy.Spider.start`. To write spider middlewares that work on Scrapy versions lower than - VERSION, define also a synchronous ``process_start_requests()`` method + 2.13, define also a synchronous ``process_start_requests()`` method that returns an iterable. For example: .. code-block:: python diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py index 4a0cd9464..310abb9b7 100644 --- a/scrapy/core/spidermw.py +++ b/scrapy/core/spidermw.py @@ -85,8 +85,8 @@ class SpiderMiddlewareManager(MiddlewareManager): "either disable or make universal 1 of those 2 sets of " "spider middlewares. Making a spider middleware universal " "means having it define both methods. See the release notes " - "of Scrapy VERSION for details: " - "https://docs.scrapy.org/en/VERSION/news.html" + "of Scrapy 2.13 for details: " + "https://docs.scrapy.org/en/2.13/news.html" ) self._use_start_requests = bool(deprecated_middlewares) @@ -103,15 +103,15 @@ class SpiderMiddlewareManager(MiddlewareManager): f"been deprecated in favor of a new method, process_start(), " f"to support asynchronous code execution. " f"process_start_requests() will stop being called in a future " - f"version of Scrapy. If you use Scrapy VERSION or higher " + f"version of Scrapy. If you use Scrapy 2.13 or higher " f"only, replace process_start_requests() with " f"process_start(); note that process_start() is a coroutine " f"(async def). If you need to maintain compatibility with " f"lower Scrapy versions, when defining " f"process_start_requests() in a spider middleware class, " f"define process_start() as well. See the release notes of " - f"Scrapy VERSION for details: " - f"https://docs.scrapy.org/en/VERSION/news.html", + f"Scrapy 2.13 for details: " + f"https://docs.scrapy.org/en/2.13/news.html", ScrapyDeprecationWarning, ) @@ -435,15 +435,15 @@ class SpiderMiddlewareManager(MiddlewareManager): f"start_requests() has been deprecated in favor of a new " f"method, start(), to support asynchronous code " f"execution. start_requests() will stop being called in a " - f"future version of Scrapy. If you use Scrapy VERSION or " + f"future version of Scrapy. If you use Scrapy 2.13 or " f"higher only, replace start_requests() with start(); " f"note that start() is a coroutine (async def). If you " f"need to maintain compatibility with lower Scrapy versions, " f"when overriding start_requests() in a spider class, " f"override start() as well; you can use super() to " f"reuse the inherited start() implementation without " - f"copy-pasting. See the release notes of Scrapy VERSION for " - f"details: https://docs.scrapy.org/en/VERSION/news.html", + f"copy-pasting. See the release notes of Scrapy 2.13 for " + f"details: https://docs.scrapy.org/en/2.13/news.html", ScrapyDeprecationWarning, ) @@ -469,8 +469,8 @@ class SpiderMiddlewareManager(MiddlewareManager): f"deprecated spider middlewares (and earlier Scrapy versions) " f"by defining a sync start_requests() method that works " f"similarly to its existing start() method. See the " - f"release notes of Scrapy VERSION for details: " - f"https://docs.scrapy.org/en/VERSION/news.html" + f"release notes of Scrapy 2.13 for details: " + f"https://docs.scrapy.org/en/2.13/news.html" ) # This method is only needed until _async compatibility methods are removed. diff --git a/scrapy/spidermiddlewares/base.py b/scrapy/spidermiddlewares/base.py index cfb50c599..196b84d0d 100644 --- a/scrapy/spidermiddlewares/base.py +++ b/scrapy/spidermiddlewares/base.py @@ -17,6 +17,8 @@ if TYPE_CHECKING: class BaseSpiderMiddleware: """Optional base class for spider middlewares. + .. versionadded:: 2.13 + This class provides helper methods for asynchronous ``process_spider_output()`` and ``process_start()`` methods. Middlewares that don't have either of these methods don't need to use this class. diff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.py index 0a1d85ae6..a722dd83b 100644 --- a/scrapy/spiders/__init__.py +++ b/scrapy/spiders/__init__.py @@ -83,7 +83,7 @@ class Spider(object_ref): async def start(self) -> AsyncIterator[Any]: """Yield the initial :class:`~scrapy.Request` objects to send. - .. versionadded:: VERSION + .. versionadded:: 2.13 For example: @@ -115,7 +115,7 @@ class Spider(object_ref): async def start(self): yield {"foo": "bar"} - To write spiders that work on Scrapy versions lower than VERSION, + To write spiders that work on Scrapy versions lower than 2.13, define also a synchronous ``start_requests()`` method that returns an iterable. For example: