mirror of https://github.com/scrapy/scrapy.git
61 lines
1.7 KiB
ReStructuredText
61 lines
1.7 KiB
ReStructuredText
.. _topics-link-extractors:
|
|
|
|
===============
|
|
Link Extractors
|
|
===============
|
|
|
|
A link extractor is an object that extracts links from responses.
|
|
|
|
The ``__init__`` method of
|
|
:class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` takes settings that
|
|
determine which links may be extracted. :class:`LxmlLinkExtractor.extract_links
|
|
<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` returns a
|
|
list of matching :class:`~scrapy.link.Link` objects from a
|
|
:class:`~scrapy.http.Response` object.
|
|
|
|
Link extractors are used in :class:`~scrapy.spiders.CrawlSpider` spiders
|
|
through a set of :class:`~scrapy.spiders.Rule` objects.
|
|
|
|
You can also use link extractors in regular spiders. For example, you can instantiate
|
|
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class
|
|
variable in your spider, and use it from your spider callbacks:
|
|
|
|
.. code-block:: python
|
|
|
|
def parse(self, response):
|
|
for link in self.link_extractor.extract_links(response):
|
|
yield Request(link.url, callback=self.parse)
|
|
|
|
.. _topics-link-extractors-ref:
|
|
|
|
Link extractor reference
|
|
========================
|
|
|
|
.. module:: scrapy.linkextractors
|
|
:synopsis: Link extractors classes
|
|
|
|
The link extractor class is
|
|
:class:`scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor`. For convenience it
|
|
can also be imported as ``scrapy.linkextractors.LinkExtractor``::
|
|
|
|
from scrapy.linkextractors import LinkExtractor
|
|
|
|
LxmlLinkExtractor
|
|
-----------------
|
|
|
|
.. module:: scrapy.linkextractors.lxmlhtml
|
|
:synopsis: lxml's HTMLParser-based link extractors
|
|
|
|
|
|
.. autoclass:: LxmlLinkExtractor
|
|
|
|
.. automethod:: extract_links
|
|
|
|
Link
|
|
----
|
|
|
|
.. module:: scrapy.link
|
|
:synopsis: Link from link extractors
|
|
|
|
.. autoclass:: Link
|