From 07ff9248a5fd2eac4f53da92766dcb5a7ca48569 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Mon, 1 Apr 2019 12:31:26 -0300 Subject: [PATCH 1/4] [Docs] CrawlSpider: add note about link text --- docs/topics/spiders.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 8c4049f85..5417ef129 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -377,7 +377,10 @@ Crawling rules .. class:: Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None) ``link_extractor`` is a :ref:`Link Extractor ` object which - defines how links will be extracted from each crawled page. + defines how links will be extracted from each crawled page. Each produced link will + be used to generate a :class:`~scrapy.http.Request` object, which will contain the + link's text in its ``meta`` attribute. + The link text can be accessed from the callback method though ``response.meta['link_text']`` ``callback`` is a callable or a string (in which case a method from the spider object with that name will be used) to be called for each link extracted with @@ -438,6 +441,7 @@ Let's now take a look at an example CrawlSpider with rules:: item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)') item['name'] = response.xpath('//td[@id="item_name"]/text()').get() item['description'] = response.xpath('//td[@id="item_description"]/text()').get() + item['link_text'] = response.meta['link_text'] return item From 8ebbc731b2bdf8e2a2b5a2f0673da838369f31b5 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Mon, 1 Apr 2019 16:15:03 -0300 Subject: [PATCH 2/4] [Docs] Rephrase Rule docs --- docs/topics/spiders.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 5417ef129..7290bb844 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -380,13 +380,13 @@ Crawling rules defines how links will be extracted from each crawled page. Each produced link will be used to generate a :class:`~scrapy.http.Request` object, which will contain the link's text in its ``meta`` attribute. - The link text can be accessed from the callback method though ``response.meta['link_text']`` ``callback`` is a callable or a string (in which case a method from the spider object with that name will be used) to be called for each link extracted with - the specified link_extractor. This callback receives a response as its first - argument and must return a list containing :class:`~scrapy.item.Item` and/or - :class:`~scrapy.http.Request` objects (or any subclass of them). + the specified link extractor. This callback receives a :class:`~scrapy.http.Response` + as its first argument and must return either a single instance or an iterable of + :class:`~scrapy.item.Item`, ``dict`` and/or :class:`~scrapy.http.Request` objects + (or any subclass of them). .. warning:: When writing crawl spider rules, avoid using ``parse`` as callback, since the :class:`CrawlSpider` uses the ``parse`` method From 7a38623cecc6c60d7ffe14c75d1fe679bb04b774 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Mon, 1 Apr 2019 17:09:49 -0300 Subject: [PATCH 3/4] [Docs] Clarify comment about meta dictionary --- docs/topics/spiders.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 7290bb844..3cd051cdf 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -379,7 +379,7 @@ Crawling rules ``link_extractor`` is a :ref:`Link Extractor ` object which defines how links will be extracted from each crawled page. Each produced link will be used to generate a :class:`~scrapy.http.Request` object, which will contain the - link's text in its ``meta`` attribute. + link's text in its ``meta`` dictionary (under the ``link_text`` key). ``callback`` is a callable or a string (in which case a method from the spider object with that name will be used) to be called for each link extracted with From 9fb0f8454eed974f5d2ed1808142169364145348 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Tue, 9 Jul 2019 15:30:22 -0300 Subject: [PATCH 4/4] Extend docs about Crawling Rules --- docs/topics/spiders.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 84a3a8fbe..869a61441 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -386,7 +386,9 @@ Crawling rules the specified link extractor. This callback receives a :class:`~scrapy.http.Response` as its first argument and must return either a single instance or an iterable of :class:`~scrapy.item.Item`, ``dict`` and/or :class:`~scrapy.http.Request` objects - (or any subclass of them). + (or any subclass of them). As mentioned above, the received :class:`~scrapy.http.Response` + object will contain the text of the link that produced the :class:`~scrapy.http.Request` + in its ``meta`` dictionary (under the ``link_text`` key) .. warning:: When writing crawl spider rules, avoid using ``parse`` as callback, since the :class:`CrawlSpider` uses the ``parse`` method