Update spiders.rst

Added note to allowed_domain attribute with an example explaining what goes in the list
This commit is contained in:
Qiwei Huang 2017-03-20 19:37:07 -07:00 committed by GitHub
parent c5f74f7d1a
commit 8ecc307e8f
1 changed files with 3 additions and 2 deletions

View File

@ -80,8 +80,9 @@ scrapy.Spider
allowed to crawl. Requests for URLs not belonging to the domain names
specified in this list (or their subdomains) won't be followed if
:class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled.
.. note:: If you are scraping an url ``https://www.example.com/1.html``
you should add ``'example.com'`` to allowed_domains list.
Let's say your target url is ``https://www.example.com/1.html``,
then add ``'example.com'`` to the list.
.. attribute:: start_urls