Add docs about universal start request definition

This commit is contained in:
Adrián Chaves 2025-04-01 12:13:59 +02:00
parent 6c1852d313
commit 9dc9999eaa
2 changed files with 47 additions and 3 deletions

View File

@ -512,6 +512,42 @@ override the :meth:`~scrapy.Spider.start` method as follows:
.. seealso:: :class:`~scrapy.crawler.Crawler`, :ref:`topics-signals`.
Universal start request definition
----------------------------------
The :meth:`~scrapy.Spider.start` method was introduced in Scrapy VERSION. It
replaced ``start_requests()``, which could be defined either as a synchronous
:term:`generator` or as a synchronous method returning an
:class:`~collections.abc.Iterable`.
If you write spiders that must work with both Scrapy VERSION+ and lower
versions, you must define both methods. For example:
.. code-block:: python
from scrapy import Spider
class MySpider(Spider):
name = "myspider"
def start_requests(self):
yield Request("https://toscrape.com", headers={"Foo": "Bar"})
async def start(self):
for request in self.start_requests():
yield request
Spiders that define both methods will not trigger a deprecation warning about
``start_requests()``. When subclassing such a spider, if you override one of
these methods, also override the other method, or you will get a warning.
.. warning:: Do not call the ``start_requests()`` method of
:class:`~scrapy.Spider` or of other spider classes not defined by yourself
(e.g. using ``super().start_requests()``) from your
:meth:`~scrapy.Spider.start` implementation. However, you can call your own
``start_requests()`` method, as shown above.
.. _builtin-spiders:
Generic Spiders

View File

@ -150,9 +150,17 @@ class Spider(object_ref):
warnings.warn(
(
"The Spider.start_requests() method is deprecated, use "
"Spider.start() instead. If you are calling "
"super().start_requests() from a Spider.start() override, "
"iterate super().start() instead."
"Spider.start() instead.\n"
"\n"
"If you are calling super().start_requests() from a "
"Spider.start() override, iterate super().start() instead.\n"
"\n"
"If you are calling super().start_requests() from a "
"Spider.start_requests() override, either redefine your "
"override to avoid a call to super().start_requests(), or use "
"warnings.catch_warnings() with warnings.filterwarnings() to "
"silence this warning (see the Spider.start() implementation "
"for an example)."
),
ScrapyDeprecationWarning,
stacklevel=2,