From c5f74f7d1a5245b65e5c7863b02a379667bee0dd Mon Sep 17 00:00:00 2001 From: Qiwei Huang Date: Mon, 20 Mar 2017 18:52:33 -0700 Subject: [PATCH 1/2] Update spiders.rst Added a note to allowed_domains attribute, reminding users not to add urls into the list. --- docs/topics/spiders.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 5e69055d1..9e27614d8 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -80,6 +80,8 @@ scrapy.Spider allowed to crawl. Requests for URLs not belonging to the domain names specified in this list (or their subdomains) won't be followed if :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled. + .. note:: If you are scraping an url ``https://www.example.com/1.html`` + you should add ``'example.com'`` to allowed_domains list. .. attribute:: start_urls From 8ecc307e8f04b058f2fc8a1a47759203ff6ebca0 Mon Sep 17 00:00:00 2001 From: Qiwei Huang Date: Mon, 20 Mar 2017 19:37:07 -0700 Subject: [PATCH 2/2] Update spiders.rst Added note to allowed_domain attribute with an example explaining what goes in the list --- docs/topics/spiders.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 9e27614d8..49c0cefb5 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -80,8 +80,9 @@ scrapy.Spider allowed to crawl. Requests for URLs not belonging to the domain names specified in this list (or their subdomains) won't be followed if :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled. - .. note:: If you are scraping an url ``https://www.example.com/1.html`` - you should add ``'example.com'`` to allowed_domains list. + + Let's say your target url is ``https://www.example.com/1.html``, + then add ``'example.com'`` to the list. .. attribute:: start_urls