Update spiders.rst

Added a note to allowed_domains attribute, reminding users not to add urls into the list.
2017-03-20 18:52:33 -07:00 · 2017-03-20 18:52:33 -07:00 · c5f74f7d1a
parent ec55799d5e
commit c5f74f7d1a
1 changed files with 2 additions and 0 deletions
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@ -80,6 +80,8 @@ scrapy.Spider
       allowed to crawl. Requests for URLs not belonging to the domain names
       specified in this list (or their subdomains) won't be followed if
       :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled.
+       .. note:: If you are scraping an url ``https://www.example.com/1.html`` 
+                 you should add ``'example.com'`` to allowed_domains list.

   .. attribute:: start_urls