mirror of https://github.com/scrapy/scrapy.git
Update spiders.rst
Added note to allowed_domain attribute with an example explaining what goes in the list
This commit is contained in:
parent
c5f74f7d1a
commit
8ecc307e8f
|
|
@ -80,8 +80,9 @@ scrapy.Spider
|
|||
allowed to crawl. Requests for URLs not belonging to the domain names
|
||||
specified in this list (or their subdomains) won't be followed if
|
||||
:class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled.
|
||||
.. note:: If you are scraping an url ``https://www.example.com/1.html``
|
||||
you should add ``'example.com'`` to allowed_domains list.
|
||||
|
||||
Let's say your target url is ``https://www.example.com/1.html``,
|
||||
then add ``'example.com'`` to the list.
|
||||
|
||||
.. attribute:: start_urls
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue