1.7 KiB
Link Extractors
A link extractor is an object that extracts links from responses.
The __init__ method of :class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` takes settings that determine which links may be extracted. :class:`LxmlLinkExtractor.extract_links <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` returns a list of matching :class:`~scrapy.link.Link` objects from a :class:`~scrapy.http.Response` object.
System Message: ERROR/3 (<stdin>, line 9); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 9); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 9); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 9); backlink
Unknown interpreted text role "class".Link extractors are used in :class:`~scrapy.spiders.CrawlSpider` spiders through a set of :class:`~scrapy.spiders.Rule` objects.
System Message: ERROR/3 (<stdin>, line 16); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 16); backlink
Unknown interpreted text role "class".You can also use link extractors in regular spiders. For example, you can instantiate :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class variable in your spider, and use it from your spider callbacks:
System Message: ERROR/3 (<stdin>, line 19); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 23)
Cannot analyze code. Pygments package not found.
.. code-block:: python
def parse(self, response):
for link in self.link_extractor.extract_links(response):
yield Request(link.url, callback=self.parse)
Link extractor reference
System Message: ERROR/3 (<stdin>, line 34)
Unknown directive type "module".
.. module:: scrapy.linkextractors :synopsis: Link extractors classes
The link extractor class is :class:`scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor`. For convenience it can also be imported as scrapy.linkextractors.LinkExtractor:
System Message: ERROR/3 (<stdin>, line 37); backlink
Unknown interpreted text role "class".from scrapy.linkextractors import LinkExtractor
LxmlLinkExtractor
System Message: ERROR/3 (<stdin>, line 46)
Unknown directive type "module".
.. module:: scrapy.linkextractors.lxmlhtml :synopsis: lxml's HTMLParser-based link extractors
System Message: ERROR/3 (<stdin>, line 50)
Unknown directive type "autoclass".
.. autoclass:: LxmlLinkExtractor
.. automethod:: extract_links
Link
System Message: ERROR/3 (<stdin>, line 57)
Unknown directive type "module".
.. module:: scrapy.link :synopsis: Link from link extractors
System Message: ERROR/3 (<stdin>, line 60)
Unknown directive type "autoclass".
.. autoclass:: Link