diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 5e69055d1..49c0cefb5 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -80,6 +80,9 @@ scrapy.Spider allowed to crawl. Requests for URLs not belonging to the domain names specified in this list (or their subdomains) won't be followed if :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled. + + Let's say your target url is ``https://www.example.com/1.html``, + then add ``'example.com'`` to the list. .. attribute:: start_urls