mirror of https://github.com/scrapy/scrapy.git
Format python console blocks using blacken-docs
Prepend '.. code-block:: pycon' to make python console blocks detectable by blacken-docs
This commit is contained in:
pankaj1707k
parent
c1bbb299d7
commit
cc9eb3fa79
|
|
@ -83,7 +83,9 @@ optionally how to follow links in the pages, and how to parse the downloaded
|
|||
page content to extract data.
|
||||
|
||||
This is the code for our first Spider. Save it in a file named
|
||||
``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project::
|
||||
``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
|
@ -95,17 +97,17 @@ This is the code for our first Spider. Save it in a file named
|
|||
|
||||
def start_requests(self):
|
||||
urls = [
|
||||
'https://quotes.toscrape.com/page/1/',
|
||||
'https://quotes.toscrape.com/page/2/',
|
||||
"https://quotes.toscrape.com/page/1/",
|
||||
"https://quotes.toscrape.com/page/2/",
|
||||
]
|
||||
for url in urls:
|
||||
yield scrapy.Request(url=url, callback=self.parse)
|
||||
|
||||
def parse(self, response):
|
||||
page = response.url.split("/")[-2]
|
||||
filename = f'quotes-{page}.html'
|
||||
filename = f"quotes-{page}.html"
|
||||
Path(filename).write_bytes(response.body)
|
||||
self.log(f'Saved file {filename}')
|
||||
self.log(f"Saved file {filename}")
|
||||
|
||||
|
||||
As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.Spider>`
|
||||
|
|
@ -247,8 +249,10 @@ object:
|
|||
|
||||
response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html')
|
||||
|
||||
>>> response.css('title')
|
||||
[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title")
|
||||
[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
|
||||
|
||||
The result of running ``response.css('title')`` is a list-like object called
|
||||
:class:`~scrapy.selector.SelectorList`, which represents a list of
|
||||
|
|
@ -258,42 +262,54 @@ data.
|
|||
|
||||
To extract the text from the title above, you can do:
|
||||
|
||||
>>> response.css('title::text').getall()
|
||||
['Quotes to Scrape']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title::text").getall()
|
||||
['Quotes to Scrape']
|
||||
|
||||
There are two things to note here: one is that we've added ``::text`` to the
|
||||
CSS query, to mean we want to select only the text elements directly inside
|
||||
``<title>`` element. If we don't specify ``::text``, we'd get the full title
|
||||
element, including its tags:
|
||||
|
||||
>>> response.css('title').getall()
|
||||
['<title>Quotes to Scrape</title>']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title").getall()
|
||||
['<title>Quotes to Scrape</title>']
|
||||
|
||||
The other thing is that the result of calling ``.getall()`` is a list: it is
|
||||
possible that a selector returns more than one result, so we extract them all.
|
||||
When you know you just want the first result, as in this case, you can do:
|
||||
|
||||
>>> response.css('title::text').get()
|
||||
'Quotes to Scrape'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title::text").get()
|
||||
'Quotes to Scrape'
|
||||
|
||||
As an alternative, you could've written:
|
||||
|
||||
>>> response.css('title::text')[0].get()
|
||||
'Quotes to Scrape'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title::text")[0].get()
|
||||
'Quotes to Scrape'
|
||||
|
||||
Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will
|
||||
raise an :exc:`IndexError` exception if there are no results::
|
||||
raise an :exc:`IndexError` exception if there are no results:
|
||||
|
||||
>>> response.css('noelement')[0].get()
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("noelement")[0].get()
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
IndexError: list index out of range
|
||||
|
||||
You might want to use ``.get()`` directly on the
|
||||
:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None``
|
||||
if there are no results::
|
||||
if there are no results:
|
||||
|
||||
>>> response.css("noelement").get()
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("noelement").get()
|
||||
|
||||
There's a lesson here: for most scraping code, you want it to be resilient to
|
||||
errors due to things not being found on a page, so that even if some parts fail
|
||||
|
|
@ -304,12 +320,14 @@ Besides the :meth:`~scrapy.selector.SelectorList.getall` and
|
|||
the :meth:`~scrapy.selector.SelectorList.re` method to extract using
|
||||
:doc:`regular expressions <library/re>`:
|
||||
|
||||
>>> response.css('title::text').re(r'Quotes.*')
|
||||
['Quotes to Scrape']
|
||||
>>> response.css('title::text').re(r'Q\w+')
|
||||
['Quotes']
|
||||
>>> response.css('title::text').re(r'(\w+) to (\w+)')
|
||||
['Quotes', 'Scrape']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title::text").re(r"Quotes.*")
|
||||
['Quotes to Scrape']
|
||||
>>> response.css("title::text").re(r"Q\w+")
|
||||
['Quotes']
|
||||
>>> response.css("title::text").re(r"(\w+) to (\w+)")
|
||||
['Quotes', 'Scrape']
|
||||
|
||||
In order to find the proper CSS selectors to use, you might find useful opening
|
||||
the response page from the shell in your web browser using ``view(response)``.
|
||||
|
|
@ -327,10 +345,12 @@ XPath: a brief intro
|
|||
|
||||
Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions:
|
||||
|
||||
>>> response.xpath('//title')
|
||||
[<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
|
||||
>>> response.xpath('//title/text()').get()
|
||||
'Quotes to Scrape'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//title")
|
||||
[<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
|
||||
>>> response.xpath("//title/text()").get()
|
||||
'Quotes to Scrape'
|
||||
|
||||
XPath expressions are very powerful, and are the foundation of Scrapy
|
||||
Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You
|
||||
|
|
@ -387,33 +407,41 @@ we want::
|
|||
|
||||
We get a list of selectors for the quote HTML elements with:
|
||||
|
||||
>>> response.css("div.quote")
|
||||
[<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
|
||||
<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
|
||||
...]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("div.quote")
|
||||
[<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
|
||||
<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
|
||||
...]
|
||||
|
||||
Each of the selectors returned by the query above allows us to run further
|
||||
queries over their sub-elements. Let's assign the first selector to a
|
||||
variable, so that we can run our CSS selectors directly on a particular quote:
|
||||
|
||||
>>> quote = response.css("div.quote")[0]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> quote = response.css("div.quote")[0]
|
||||
|
||||
Now, let's extract ``text``, ``author`` and the ``tags`` from that quote
|
||||
using the ``quote`` object we just created:
|
||||
|
||||
>>> text = quote.css("span.text::text").get()
|
||||
>>> text
|
||||
'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
|
||||
>>> author = quote.css("small.author::text").get()
|
||||
>>> author
|
||||
'Albert Einstein'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> text = quote.css("span.text::text").get()
|
||||
>>> text
|
||||
'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
|
||||
>>> author = quote.css("small.author::text").get()
|
||||
>>> author
|
||||
'Albert Einstein'
|
||||
|
||||
Given that the tags are a list of strings, we can use the ``.getall()`` method
|
||||
to get all of them:
|
||||
|
||||
>>> tags = quote.css("div.tags a.tag::text").getall()
|
||||
>>> tags
|
||||
['change', 'deep-thoughts', 'thinking', 'world']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> tags = quote.css("div.tags a.tag::text").getall()
|
||||
>>> tags
|
||||
['change', 'deep-thoughts', 'thinking', 'world']
|
||||
|
||||
.. invisible-code-block: python
|
||||
|
||||
|
|
@ -422,14 +450,17 @@ to get all of them:
|
|||
Having figured out how to extract each bit, we can now iterate over all the
|
||||
quotes elements and put them together into a Python dictionary:
|
||||
|
||||
>>> for quote in response.css("div.quote"):
|
||||
... text = quote.css("span.text::text").get()
|
||||
... author = quote.css("small.author::text").get()
|
||||
... tags = quote.css("div.tags a.tag::text").getall()
|
||||
... print(dict(text=text, author=author, tags=tags))
|
||||
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
|
||||
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}
|
||||
...
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> for quote in response.css("div.quote"):
|
||||
... text = quote.css("span.text::text").get()
|
||||
... author = quote.css("small.author::text").get()
|
||||
... tags = quote.css("div.tags a.tag::text").getall()
|
||||
... print(dict(text=text, author=author, tags=tags))
|
||||
...
|
||||
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
|
||||
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}
|
||||
...
|
||||
|
||||
Extracting data in our spider
|
||||
-----------------------------
|
||||
|
|
@ -537,14 +568,18 @@ This gets the anchor element, but we want the attribute ``href``. For that,
|
|||
Scrapy supports a CSS extension that lets you select the attribute contents,
|
||||
like this:
|
||||
|
||||
>>> response.css('li.next a::attr(href)').get()
|
||||
'/page/2/'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("li.next a::attr(href)").get()
|
||||
'/page/2/'
|
||||
|
||||
There is also an ``attrib`` property available
|
||||
(see :ref:`selecting-attributes` for more):
|
||||
|
||||
>>> response.css('li.next a').attrib['href']
|
||||
'/page/2/'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("li.next a").attrib["href"]
|
||||
'/page/2/'
|
||||
|
||||
Let's see now our spider modified to recursively follow the link to the next
|
||||
page, extracting data from it:
|
||||
|
|
|
|||
|
|
@ -2417,11 +2417,13 @@ Backward-incompatible changes
|
|||
* :class:`~scrapy.loader.ItemLoader` now turns the values of its input item
|
||||
into lists:
|
||||
|
||||
>>> item = MyItem()
|
||||
>>> item['field'] = 'value1'
|
||||
>>> loader = ItemLoader(item=item)
|
||||
>>> item['field']
|
||||
['value1']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> item = MyItem()
|
||||
>>> item["field"] = "value1"
|
||||
>>> loader = ItemLoader(item=item)
|
||||
>>> item["field"]
|
||||
['value1']
|
||||
|
||||
This is needed to allow adding values to existing fields
|
||||
(``loader.add_value('field', 'value2')``).
|
||||
|
|
|
|||
|
|
@ -134,14 +134,14 @@ Settings API
|
|||
|
||||
.. highlight:: python
|
||||
|
||||
::
|
||||
.. code-block:: python
|
||||
|
||||
SETTINGS_PRIORITIES = {
|
||||
'default': 0,
|
||||
'command': 10,
|
||||
'project': 20,
|
||||
'spider': 30,
|
||||
'cmdline': 40,
|
||||
"default": 0,
|
||||
"command": 10,
|
||||
"project": 20,
|
||||
"spider": 30,
|
||||
"cmdline": 40,
|
||||
}
|
||||
|
||||
For a detailed explanation on each settings sources, see:
|
||||
|
|
|
|||
|
|
@ -94,8 +94,10 @@ Then, back to your web browser, right-click on the ``span`` tag, select
|
|||
|
||||
response = load_response('https://quotes.toscrape.com/', 'quotes.html')
|
||||
|
||||
>>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall()
|
||||
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("/html/body/div/div[2]/div[1]/div[1]/span[1]/text()").getall()
|
||||
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
|
||||
|
||||
Adding ``text()`` at the end we are able to extract the first quote with this
|
||||
basic selector. But this XPath is not really that clever. All it does is
|
||||
|
|
@ -124,11 +126,13 @@ With this knowledge we can refine our XPath: Instead of a path to follow,
|
|||
we'll simply select all ``span`` tags with the ``class="text"`` by using
|
||||
the `has-class-extension`_:
|
||||
|
||||
>>> response.xpath('//span[has-class("text")]/text()').getall()
|
||||
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
|
||||
'“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
|
||||
'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
|
||||
...]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//span[has-class("text")]/text()').getall()
|
||||
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
|
||||
'“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
|
||||
'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
|
||||
...]
|
||||
|
||||
And with one simple, cleverer XPath we are able to extract all quotes from
|
||||
the page. We could have constructed a loop over our first XPath to increase
|
||||
|
|
|
|||
|
|
@ -183,10 +183,12 @@ data from it:
|
|||
For example, if the JavaScript code contains a separate line like
|
||||
``var data = {"field": "value"};`` you can extract that data as follows:
|
||||
|
||||
>>> pattern = r'\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n'
|
||||
>>> json_data = response.css('script::text').re_first(pattern)
|
||||
>>> json.loads(json_data)
|
||||
{'field': 'value'}
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> pattern = r"\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n"
|
||||
>>> json_data = response.css("script::text").re_first(pattern)
|
||||
>>> json.loads(json_data)
|
||||
{'field': 'value'}
|
||||
|
||||
- chompjs_ provides an API to parse JavaScript objects into a :class:`dict`.
|
||||
|
||||
|
|
@ -194,11 +196,13 @@ data from it:
|
|||
``var data = {field: "value", secondField: "second value"};``
|
||||
you can extract that data as follows:
|
||||
|
||||
>>> import chompjs
|
||||
>>> javascript = response.css('script::text').get()
|
||||
>>> data = chompjs.parse_js_object(javascript)
|
||||
>>> data
|
||||
{'field': 'value', 'secondField': 'second value'}
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> import chompjs
|
||||
>>> javascript = response.css("script::text").get()
|
||||
>>> data = chompjs.parse_js_object(javascript)
|
||||
>>> data
|
||||
{'field': 'value', 'secondField': 'second value'}
|
||||
|
||||
- Otherwise, use js2xml_ to convert the JavaScript code into an XML document
|
||||
that you can parse using :ref:`selectors <topics-selectors>`.
|
||||
|
|
@ -206,14 +210,16 @@ data from it:
|
|||
For example, if the JavaScript code contains
|
||||
``var data = {field: "value"};`` you can extract that data as follows:
|
||||
|
||||
>>> import js2xml
|
||||
>>> import lxml.etree
|
||||
>>> from parsel import Selector
|
||||
>>> javascript = response.css('script::text').get()
|
||||
>>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding='unicode')
|
||||
>>> selector = Selector(text=xml)
|
||||
>>> selector.css('var[name="data"]').get()
|
||||
'<var name="data"><object><property name="field"><string>value</string></property></object></var>'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> import js2xml
|
||||
>>> import lxml.etree
|
||||
>>> from parsel import Selector
|
||||
>>> javascript = response.css("script::text").get()
|
||||
>>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding="unicode")
|
||||
>>> selector = Selector(text=xml)
|
||||
>>> selector.css('var[name="data"]').get()
|
||||
'<var name="data"><object><property name="field"><string>value</string></property></object></var>'
|
||||
|
||||
.. _topics-javascript-rendering:
|
||||
|
||||
|
|
|
|||
|
|
@ -234,62 +234,68 @@ notice the API is very similar to the :class:`dict` API.
|
|||
Creating items
|
||||
''''''''''''''
|
||||
|
||||
>>> product = Product(name='Desktop PC', price=1000)
|
||||
>>> print(product)
|
||||
Product(name='Desktop PC', price=1000)
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> product = Product(name="Desktop PC", price=1000)
|
||||
>>> print(product)
|
||||
Product(name='Desktop PC', price=1000)
|
||||
|
||||
|
||||
Getting field values
|
||||
''''''''''''''''''''
|
||||
|
||||
>>> product['name']
|
||||
Desktop PC
|
||||
>>> product.get('name')
|
||||
Desktop PC
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> product['price']
|
||||
1000
|
||||
>>> product["name"]
|
||||
Desktop PC
|
||||
>>> product.get("name")
|
||||
Desktop PC
|
||||
|
||||
>>> product['last_updated']
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'last_updated'
|
||||
>>> product["price"]
|
||||
1000
|
||||
|
||||
>>> product.get('last_updated', 'not set')
|
||||
not set
|
||||
>>> product["last_updated"]
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'last_updated'
|
||||
|
||||
>>> product['lala'] # getting unknown field
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'lala'
|
||||
>>> product.get("last_updated", "not set")
|
||||
not set
|
||||
|
||||
>>> product.get('lala', 'unknown field')
|
||||
'unknown field'
|
||||
>>> product["lala"] # getting unknown field
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'lala'
|
||||
|
||||
>>> 'name' in product # is name field populated?
|
||||
True
|
||||
>>> product.get("lala", "unknown field")
|
||||
'unknown field'
|
||||
|
||||
>>> 'last_updated' in product # is last_updated populated?
|
||||
False
|
||||
>>> "name" in product # is name field populated?
|
||||
True
|
||||
|
||||
>>> 'last_updated' in product.fields # is last_updated a declared field?
|
||||
True
|
||||
>>> "last_updated" in product # is last_updated populated?
|
||||
False
|
||||
|
||||
>>> 'lala' in product.fields # is lala a declared field?
|
||||
False
|
||||
>>> "last_updated" in product.fields # is last_updated a declared field?
|
||||
True
|
||||
|
||||
>>> "lala" in product.fields # is lala a declared field?
|
||||
False
|
||||
|
||||
|
||||
Setting field values
|
||||
''''''''''''''''''''
|
||||
|
||||
>>> product['last_updated'] = 'today'
|
||||
>>> product['last_updated']
|
||||
today
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> product['lala'] = 'test' # setting unknown field
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'Product does not support field: lala'
|
||||
>>> product["last_updated"] = "today"
|
||||
>>> product["last_updated"]
|
||||
today
|
||||
|
||||
>>> product["lala"] = "test" # setting unknown field
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'Product does not support field: lala'
|
||||
|
||||
|
||||
Accessing all populated values
|
||||
|
|
@ -297,11 +303,13 @@ Accessing all populated values
|
|||
|
||||
To access all populated values, just use the typical :class:`dict` API:
|
||||
|
||||
>>> product.keys()
|
||||
['price', 'name']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> product.items()
|
||||
[('price', 1000), ('name', 'Desktop PC')]
|
||||
>>> product.keys()
|
||||
['price', 'name']
|
||||
|
||||
>>> product.items()
|
||||
[('price', 1000), ('name', 'Desktop PC')]
|
||||
|
||||
|
||||
.. _copying-items:
|
||||
|
|
@ -339,18 +347,20 @@ Other common tasks
|
|||
|
||||
Creating dicts from items:
|
||||
|
||||
>>> dict(product) # create a dict from all populated values
|
||||
{'price': 1000, 'name': 'Desktop PC'}
|
||||
.. code-block:: pycon
|
||||
|
||||
Creating items from dicts:
|
||||
>>> dict(product) # create a dict from all populated values
|
||||
{'price': 1000, 'name': 'Desktop PC'}
|
||||
|
||||
>>> Product({'name': 'Laptop PC', 'price': 1500})
|
||||
Product(price=1500, name='Laptop PC')
|
||||
Creating items from dicts:
|
||||
|
||||
>>> Product({'name': 'Laptop PC', 'lala': 1500}) # warning: unknown field in dict
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'Product does not support field: lala'
|
||||
>>> Product({"name": "Laptop PC", "price": 1500})
|
||||
Product(price=1500, name='Laptop PC')
|
||||
|
||||
>>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
KeyError: 'Product does not support field: lala'
|
||||
|
||||
|
||||
Extending Item subclasses
|
||||
|
|
|
|||
|
|
@ -70,13 +70,15 @@ alias to the :func:`~scrapy.utils.trackref.print_live_refs` function::
|
|||
|
||||
telnet localhost 6023
|
||||
|
||||
>>> prefs()
|
||||
Live References
|
||||
.. code-block:: pycon
|
||||
|
||||
ExampleSpider 1 oldest: 15s ago
|
||||
HtmlResponse 10 oldest: 1s ago
|
||||
Selector 2 oldest: 0s ago
|
||||
FormRequest 878 oldest: 7s ago
|
||||
>>> prefs()
|
||||
Live References
|
||||
|
||||
ExampleSpider 1 oldest: 15s ago
|
||||
HtmlResponse 10 oldest: 1s ago
|
||||
Selector 2 oldest: 0s ago
|
||||
FormRequest 878 oldest: 7s ago
|
||||
|
||||
As you can see, that report also shows the "age" of the oldest object in each
|
||||
class. If you're running multiple spiders per process chances are you can
|
||||
|
|
@ -114,7 +116,9 @@ a priori, of course) by using the ``trackref`` tool.
|
|||
|
||||
After the crawler is running for a few minutes and we notice its memory usage
|
||||
has grown a lot, we can enter its telnet console and check the live
|
||||
references::
|
||||
references:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> prefs()
|
||||
Live References
|
||||
|
|
@ -134,19 +138,23 @@ generating the leaks (passing response references inside requests).
|
|||
Sometimes extra information about live objects can be helpful.
|
||||
Let's check the oldest response:
|
||||
|
||||
>>> from scrapy.utils.trackref import get_oldest
|
||||
>>> r = get_oldest('HtmlResponse')
|
||||
>>> r.url
|
||||
'http://www.somenastyspider.com/product.php?pid=123'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy.utils.trackref import get_oldest
|
||||
>>> r = get_oldest("HtmlResponse")
|
||||
>>> r.url
|
||||
'http://www.somenastyspider.com/product.php?pid=123'
|
||||
|
||||
If you want to iterate over all objects, instead of getting the oldest one, you
|
||||
can use the :func:`scrapy.utils.trackref.iter_all` function:
|
||||
|
||||
>>> from scrapy.utils.trackref import iter_all
|
||||
>>> [r.url for r in iter_all('HtmlResponse')]
|
||||
['http://www.somenastyspider.com/product.php?pid=123',
|
||||
'http://www.somenastyspider.com/product.php?pid=584',
|
||||
...]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy.utils.trackref import iter_all
|
||||
>>> [r.url for r in iter_all("HtmlResponse")]
|
||||
['http://www.somenastyspider.com/product.php?pid=123',
|
||||
'http://www.somenastyspider.com/product.php?pid=584',
|
||||
...]
|
||||
|
||||
Too many spiders?
|
||||
-----------------
|
||||
|
|
@ -157,8 +165,10 @@ For this reason, that function has a ``ignore`` argument which can be used to
|
|||
ignore a particular class (and all its subclasses). For
|
||||
example, this won't show any live references to spiders:
|
||||
|
||||
>>> from scrapy.spiders import Spider
|
||||
>>> prefs(ignore=Spider)
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy.spiders import Spider
|
||||
>>> prefs(ignore=Spider)
|
||||
|
||||
.. module:: scrapy.utils.trackref
|
||||
:synopsis: Track references of live objects
|
||||
|
|
@ -216,30 +226,32 @@ If you use ``pip``, you can install muppy with the following command::
|
|||
Here's an example to view all Python objects available in
|
||||
the heap using muppy:
|
||||
|
||||
>>> from pympler import muppy
|
||||
>>> all_objects = muppy.get_objects()
|
||||
>>> len(all_objects)
|
||||
28667
|
||||
>>> from pympler import summary
|
||||
>>> suml = summary.summarize(all_objects)
|
||||
>>> summary.print_(suml)
|
||||
types | # objects | total size
|
||||
==================================== | =========== | ============
|
||||
<class 'str | 9822 | 1.10 MB
|
||||
<class 'dict | 1658 | 856.62 KB
|
||||
<class 'type | 436 | 443.60 KB
|
||||
<class 'code | 2974 | 419.56 KB
|
||||
<class '_io.BufferedWriter | 2 | 256.34 KB
|
||||
<class 'set | 420 | 159.88 KB
|
||||
<class '_io.BufferedReader | 1 | 128.17 KB
|
||||
<class 'wrapper_descriptor | 1130 | 88.28 KB
|
||||
<class 'tuple | 1304 | 86.57 KB
|
||||
<class 'weakref | 1013 | 79.14 KB
|
||||
<class 'builtin_function_or_method | 958 | 67.36 KB
|
||||
<class 'method_descriptor | 865 | 60.82 KB
|
||||
<class 'abc.ABCMeta | 62 | 59.96 KB
|
||||
<class 'list | 446 | 58.52 KB
|
||||
<class 'int | 1425 | 43.20 KB
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from pympler import muppy
|
||||
>>> all_objects = muppy.get_objects()
|
||||
>>> len(all_objects)
|
||||
28667
|
||||
>>> from pympler import summary
|
||||
>>> suml = summary.summarize(all_objects)
|
||||
>>> summary.print_(suml)
|
||||
types | # objects | total size
|
||||
==================================== | =========== | ============
|
||||
<class 'str | 9822 | 1.10 MB
|
||||
<class 'dict | 1658 | 856.62 KB
|
||||
<class 'type | 436 | 443.60 KB
|
||||
<class 'code | 2974 | 419.56 KB
|
||||
<class '_io.BufferedWriter | 2 | 256.34 KB
|
||||
<class 'set | 420 | 159.88 KB
|
||||
<class '_io.BufferedReader | 1 | 128.17 KB
|
||||
<class 'wrapper_descriptor | 1130 | 88.28 KB
|
||||
<class 'tuple | 1304 | 86.57 KB
|
||||
<class 'weakref | 1013 | 79.14 KB
|
||||
<class 'builtin_function_or_method | 958 | 67.36 KB
|
||||
<class 'method_descriptor | 865 | 60.82 KB
|
||||
<class 'abc.ABCMeta | 62 | 59.96 KB
|
||||
<class 'list | 446 | 58.52 KB
|
||||
<class 'int | 1425 | 43.20 KB
|
||||
|
||||
For more info about muppy, refer to the `muppy documentation`_.
|
||||
|
||||
|
|
|
|||
|
|
@ -250,12 +250,15 @@ metadata. Here is an example:
|
|||
output_processor=TakeFirst(),
|
||||
)
|
||||
|
||||
>>> from scrapy.loader import ItemLoader
|
||||
>>> il = ItemLoader(item=Product())
|
||||
>>> il.add_value('name', ['Welcome to my', '<strong>website</strong>'])
|
||||
>>> il.add_value('price', ['€', '<span>1000</span>'])
|
||||
>>> il.load_item()
|
||||
{'name': 'Welcome to my website', 'price': '1000'}
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy.loader import ItemLoader
|
||||
>>> il = ItemLoader(item=Product())
|
||||
>>> il.add_value("name", ["Welcome to my", "<strong>website</strong>"])
|
||||
>>> il.add_value("price", ["€", "<span>1000</span>"])
|
||||
>>> il.load_item()
|
||||
{'name': 'Welcome to my website', 'price': '1000'}
|
||||
|
||||
The precedence order, for both input and output processors, is as follows:
|
||||
|
||||
|
|
|
|||
|
|
@ -1243,8 +1243,10 @@ TextResponse objects
|
|||
``str(response.body)`` is not a correct way to convert the response
|
||||
body into a string:
|
||||
|
||||
>>> str(b'body')
|
||||
"b'body'"
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> str(b"body")
|
||||
"b'body'"
|
||||
|
||||
|
||||
.. attribute:: TextResponse.encoding
|
||||
|
|
|
|||
|
|
@ -51,16 +51,20 @@ Constructing selectors
|
|||
Response objects expose a :class:`~scrapy.Selector` instance
|
||||
on ``.selector`` attribute:
|
||||
|
||||
>>> response.selector.xpath('//span/text()').get()
|
||||
'good'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.selector.xpath("//span/text()").get()
|
||||
'good'
|
||||
|
||||
Querying responses using XPath and CSS is so common that responses include two
|
||||
more shortcuts: ``response.xpath()`` and ``response.css()``:
|
||||
|
||||
>>> response.xpath('//span/text()').get()
|
||||
'good'
|
||||
>>> response.css('span::text').get()
|
||||
'good'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//span/text()").get()
|
||||
'good'
|
||||
>>> response.css("span::text").get()
|
||||
'good'
|
||||
|
||||
Scrapy selectors are instances of :class:`~scrapy.Selector` class
|
||||
constructed by passing either :class:`~scrapy.http.TextResponse` object or
|
||||
|
|
@ -75,19 +79,23 @@ you can also ensure the response body is parsed only once.
|
|||
But if required, it is possible to use ``Selector`` directly.
|
||||
Constructing from text:
|
||||
|
||||
>>> from scrapy.selector import Selector
|
||||
>>> body = '<html><body><span>good</span></body></html>'
|
||||
>>> Selector(text=body).xpath('//span/text()').get()
|
||||
'good'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy.selector import Selector
|
||||
>>> body = "<html><body><span>good</span></body></html>"
|
||||
>>> Selector(text=body).xpath("//span/text()").get()
|
||||
'good'
|
||||
|
||||
Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
|
||||
:class:`~scrapy.http.TextResponse` subclasses:
|
||||
|
||||
>>> from scrapy.selector import Selector
|
||||
>>> from scrapy.http import HtmlResponse
|
||||
>>> response = HtmlResponse(url='http://example.com', body=body)
|
||||
>>> Selector(response=response).xpath('//span/text()').get()
|
||||
'good'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy.selector import Selector
|
||||
>>> from scrapy.http import HtmlResponse
|
||||
>>> response = HtmlResponse(url="http://example.com", body=body)
|
||||
>>> Selector(response=response).xpath("//span/text()").get()
|
||||
'good'
|
||||
|
||||
``Selector`` automatically chooses the best parsing rules
|
||||
(XML vs HTML) based on input type.
|
||||
|
|
@ -124,16 +132,20 @@ Since we're dealing with HTML, the selector will automatically use an HTML parse
|
|||
So, by looking at the :ref:`HTML code <topics-selectors-htmlcode>` of that
|
||||
page, let's construct an XPath for selecting the text inside the title tag:
|
||||
|
||||
>>> response.xpath('//title/text()')
|
||||
[<Selector xpath='//title/text()' data='Example website'>]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//title/text()")
|
||||
[<Selector xpath='//title/text()' data='Example website'>]
|
||||
|
||||
To actually extract the textual data, you must call the selector ``.get()``
|
||||
or ``.getall()`` methods, as follows:
|
||||
|
||||
>>> response.xpath('//title/text()').getall()
|
||||
['Example website']
|
||||
>>> response.xpath('//title/text()').get()
|
||||
'Example website'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//title/text()").getall()
|
||||
['Example website']
|
||||
>>> response.xpath("//title/text()").get()
|
||||
'Example website'
|
||||
|
||||
``.get()`` always returns a single result; if there are several matches,
|
||||
content of a first match is returned; if there are no matches, None
|
||||
|
|
@ -142,98 +154,116 @@ is returned. ``.getall()`` returns a list with all results.
|
|||
Notice that CSS selectors can select text or attribute nodes using CSS3
|
||||
pseudo-elements:
|
||||
|
||||
>>> response.css('title::text').get()
|
||||
'Example website'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title::text").get()
|
||||
'Example website'
|
||||
|
||||
As you can see, ``.xpath()`` and ``.css()`` methods return a
|
||||
:class:`~scrapy.selector.SelectorList` instance, which is a list of new
|
||||
selectors. This API can be used for quickly selecting nested data:
|
||||
|
||||
>>> response.css('img').xpath('@src').getall()
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("img").xpath("@src").getall()
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
|
||||
If you want to extract only the first matched element, you can call the
|
||||
selector ``.get()`` (or its alias ``.extract_first()`` commonly used in
|
||||
previous Scrapy versions):
|
||||
|
||||
>>> response.xpath('//div[@id="images"]/a/text()').get()
|
||||
'Name: My image 1 '
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//div[@id="images"]/a/text()').get()
|
||||
'Name: My image 1 '
|
||||
|
||||
It returns ``None`` if no element was found:
|
||||
|
||||
>>> response.xpath('//div[@id="not-exists"]/text()').get() is None
|
||||
True
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//div[@id="not-exists"]/text()').get() is None
|
||||
True
|
||||
|
||||
A default return value can be provided as an argument, to be used instead
|
||||
of ``None``:
|
||||
|
||||
>>> response.xpath('//div[@id="not-exists"]/text()').get(default='not-found')
|
||||
'not-found'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//div[@id="not-exists"]/text()').get(default="not-found")
|
||||
'not-found'
|
||||
|
||||
Instead of using e.g. ``'@src'`` XPath it is possible to query for attributes
|
||||
using ``.attrib`` property of a :class:`~scrapy.Selector`:
|
||||
|
||||
>>> [img.attrib['src'] for img in response.css('img')]
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> [img.attrib["src"] for img in response.css("img")]
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
|
||||
As a shortcut, ``.attrib`` is also available on SelectorList directly;
|
||||
it returns attributes for the first matching element:
|
||||
|
||||
>>> response.css('img').attrib['src']
|
||||
'image1_thumb.jpg'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("img").attrib["src"]
|
||||
'image1_thumb.jpg'
|
||||
|
||||
This is most useful when only a single result is expected, e.g. when selecting
|
||||
by id, or selecting unique elements on a web page:
|
||||
|
||||
>>> response.css('base').attrib['href']
|
||||
'http://example.com/'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("base").attrib["href"]
|
||||
'http://example.com/'
|
||||
|
||||
Now we're going to get the base URL and some image links:
|
||||
|
||||
>>> response.xpath('//base/@href').get()
|
||||
'http://example.com/'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css('base::attr(href)').get()
|
||||
'http://example.com/'
|
||||
>>> response.xpath("//base/@href").get()
|
||||
'http://example.com/'
|
||||
|
||||
>>> response.css('base').attrib['href']
|
||||
'http://example.com/'
|
||||
>>> response.css("base::attr(href)").get()
|
||||
'http://example.com/'
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/@href').getall()
|
||||
['image1.html',
|
||||
'image2.html',
|
||||
'image3.html',
|
||||
'image4.html',
|
||||
'image5.html']
|
||||
>>> response.css("base").attrib["href"]
|
||||
'http://example.com/'
|
||||
|
||||
>>> response.css('a[href*=image]::attr(href)').getall()
|
||||
['image1.html',
|
||||
'image2.html',
|
||||
'image3.html',
|
||||
'image4.html',
|
||||
'image5.html']
|
||||
>>> response.xpath('//a[contains(@href, "image")]/@href').getall()
|
||||
['image1.html',
|
||||
'image2.html',
|
||||
'image3.html',
|
||||
'image4.html',
|
||||
'image5.html']
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
>>> response.css("a[href*=image]::attr(href)").getall()
|
||||
['image1.html',
|
||||
'image2.html',
|
||||
'image3.html',
|
||||
'image4.html',
|
||||
'image5.html']
|
||||
|
||||
>>> response.css('a[href*=image] img::attr(src)').getall()
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
>>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
|
||||
>>> response.css("a[href*=image] img::attr(src)").getall()
|
||||
['image1_thumb.jpg',
|
||||
'image2_thumb.jpg',
|
||||
'image3_thumb.jpg',
|
||||
'image4_thumb.jpg',
|
||||
'image5_thumb.jpg']
|
||||
|
||||
.. _topics-selectors-css-extensions:
|
||||
|
||||
|
|
@ -260,45 +290,55 @@ Examples:
|
|||
|
||||
* ``title::text`` selects children text nodes of a descendant ``<title>`` element:
|
||||
|
||||
>>> response.css('title::text').get()
|
||||
'Example website'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("title::text").get()
|
||||
'Example website'
|
||||
|
||||
* ``*::text`` selects all descendant text nodes of the current selector context:
|
||||
|
||||
>>> response.css('#images *::text').getall()
|
||||
['\n ',
|
||||
'Name: My image 1 ',
|
||||
'\n ',
|
||||
'Name: My image 2 ',
|
||||
'\n ',
|
||||
'Name: My image 3 ',
|
||||
'\n ',
|
||||
'Name: My image 4 ',
|
||||
'\n ',
|
||||
'Name: My image 5 ',
|
||||
'\n ']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("#images *::text").getall()
|
||||
['\n ',
|
||||
'Name: My image 1 ',
|
||||
'\n ',
|
||||
'Name: My image 2 ',
|
||||
'\n ',
|
||||
'Name: My image 3 ',
|
||||
'\n ',
|
||||
'Name: My image 4 ',
|
||||
'\n ',
|
||||
'Name: My image 5 ',
|
||||
'\n ']
|
||||
|
||||
* ``foo::text`` returns no results if ``foo`` element exists, but contains
|
||||
no text (i.e. text is empty):
|
||||
|
||||
>>> response.css('img::text').getall()
|
||||
[]
|
||||
.. code-block:: pycon
|
||||
|
||||
This means ``.css('foo::text').get()`` could return None even if an element
|
||||
exists. Use ``default=''`` if you always want a string:
|
||||
>>> response.css("img::text").getall()
|
||||
[]
|
||||
|
||||
>>> response.css('img::text').get()
|
||||
>>> response.css('img::text').get(default='')
|
||||
''
|
||||
is means ``.css('foo::text').get()`` could return None even if an element
|
||||
ists. Use ``default=''`` if you always want a string:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("img::text").get()
|
||||
>>> response.css("img::text").get(default="")
|
||||
''
|
||||
|
||||
* ``a::attr(href)`` selects the *href* attribute value of descendant links:
|
||||
|
||||
>>> response.css('a::attr(href)').getall()
|
||||
['image1.html',
|
||||
'image2.html',
|
||||
'image3.html',
|
||||
'image4.html',
|
||||
'image5.html']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("a::attr(href)").getall()
|
||||
['image1.html',
|
||||
'image2.html',
|
||||
'image3.html',
|
||||
'image4.html',
|
||||
'image5.html']
|
||||
|
||||
.. note::
|
||||
See also: :ref:`selecting-attributes`.
|
||||
|
|
@ -319,23 +359,26 @@ The selection methods (``.xpath()`` or ``.css()``) return a list of selectors
|
|||
of the same type, so you can call the selection methods for those selectors
|
||||
too. Here's an example:
|
||||
|
||||
>>> links = response.xpath('//a[contains(@href, "image")]')
|
||||
>>> links.getall()
|
||||
['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
|
||||
'<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
|
||||
'<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
|
||||
'<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
|
||||
'<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> for index, link in enumerate(links):
|
||||
... href_xpath = link.xpath('@href').get()
|
||||
... img_xpath = link.xpath('img/@src').get()
|
||||
... print(f'Link number {index} points to url {href_xpath!r} and image {img_xpath!r}')
|
||||
Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'
|
||||
Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'
|
||||
Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'
|
||||
Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'
|
||||
Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'
|
||||
>>> links = response.xpath('//a[contains(@href, "image")]')
|
||||
>>> links.getall()
|
||||
['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
|
||||
'<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
|
||||
'<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
|
||||
'<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
|
||||
'<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
|
||||
|
||||
>>> for index, link in enumerate(links):
|
||||
... href_xpath = link.xpath("@href").get()
|
||||
... img_xpath = link.xpath("img/@src").get()
|
||||
... print(f"Link number {index} points to url {href_xpath!r} and image {img_xpath!r}")
|
||||
...
|
||||
Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'
|
||||
Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'
|
||||
Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'
|
||||
Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'
|
||||
Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'
|
||||
|
||||
.. _selecting-attributes:
|
||||
|
||||
|
|
@ -345,8 +388,10 @@ Selecting element attributes
|
|||
There are several ways to get a value of an attribute. First, one can use
|
||||
XPath syntax:
|
||||
|
||||
>>> response.xpath("//a/@href").getall()
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//a/@href").getall()
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
|
||||
XPath syntax has a few advantages: it is a standard XPath feature, and
|
||||
``@attributes`` can be used in other parts of an XPath expression - e.g.
|
||||
|
|
@ -355,30 +400,38 @@ it is possible to filter by attribute value.
|
|||
Scrapy also provides an extension to CSS selectors (``::attr(...)``)
|
||||
which allows to get attribute values:
|
||||
|
||||
>>> response.css('a::attr(href)').getall()
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("a::attr(href)").getall()
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
|
||||
In addition to that, there is a ``.attrib`` property of Selector.
|
||||
You can use it if you prefer to lookup attributes in Python
|
||||
code, without using XPaths or CSS extensions:
|
||||
|
||||
>>> [a.attrib['href'] for a in response.css('a')]
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> [a.attrib["href"] for a in response.css("a")]
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
|
||||
This property is also available on SelectorList; it returns a dictionary
|
||||
with attributes of a first matching element. It is convenient to use when
|
||||
a selector is expected to give a single result (e.g. when selecting by element
|
||||
ID, or when selecting an unique element on a page):
|
||||
|
||||
>>> response.css('base').attrib
|
||||
{'href': 'http://example.com/'}
|
||||
>>> response.css('base').attrib['href']
|
||||
'http://example.com/'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("base").attrib
|
||||
{'href': 'http://example.com/'}
|
||||
>>> response.css("base").attrib["href"]
|
||||
'http://example.com/'
|
||||
|
||||
``.attrib`` property of an empty SelectorList is empty:
|
||||
|
||||
>>> response.css('foo').attrib
|
||||
{}
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("foo").attrib
|
||||
{}
|
||||
|
||||
Using selectors with regular expressions
|
||||
----------------------------------------
|
||||
|
|
@ -391,19 +444,23 @@ can't construct nested ``.re()`` calls.
|
|||
Here's an example used to extract image names from the :ref:`HTML code
|
||||
<topics-selectors-htmlcode>` above:
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)')
|
||||
['My image 1',
|
||||
'My image 2',
|
||||
'My image 3',
|
||||
'My image 4',
|
||||
'My image 5']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)")
|
||||
['My image 1',
|
||||
'My image 2',
|
||||
'My image 3',
|
||||
'My image 4',
|
||||
'My image 5']
|
||||
|
||||
There's an additional helper reciprocating ``.get()`` (and its
|
||||
alias ``.extract_first()``) for ``.re()``, named ``.re_first()``.
|
||||
Use it to extract just the first matching string:
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r'Name:\s*(.*)')
|
||||
'My image 1'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)")
|
||||
'My image 1'
|
||||
|
||||
.. _old-extraction-api:
|
||||
|
||||
|
|
@ -423,28 +480,36 @@ The following examples show how these methods map to each other.
|
|||
|
||||
1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``:
|
||||
|
||||
>>> response.css('a::attr(href)').get()
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("a::attr(href)").get()
|
||||
'image1.html'
|
||||
>>> response.css('a::attr(href)').extract_first()
|
||||
>>> response.css("a::attr(href)").extract_first()
|
||||
'image1.html'
|
||||
|
||||
2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``:
|
||||
|
||||
>>> response.css('a::attr(href)').getall()
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("a::attr(href)").getall()
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
>>> response.css('a::attr(href)').extract()
|
||||
>>> response.css("a::attr(href)").extract()
|
||||
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
|
||||
|
||||
3. ``Selector.get()`` is the same as ``Selector.extract()``:
|
||||
|
||||
>>> response.css('a::attr(href)')[0].get()
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("a::attr(href)")[0].get()
|
||||
'image1.html'
|
||||
>>> response.css('a::attr(href)')[0].extract()
|
||||
>>> response.css("a::attr(href)")[0].extract()
|
||||
'image1.html'
|
||||
|
||||
4. For consistency, there is also ``Selector.getall()``, which returns a list:
|
||||
|
||||
>>> response.css('a::attr(href)')[0].getall()
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("a::attr(href)")[0].getall()
|
||||
['image1.html']
|
||||
|
||||
So, the main difference is that output of ``.get()`` and ``.getall()`` methods
|
||||
|
|
@ -482,24 +547,34 @@ with ``/``, that XPath will be absolute to the document and not relative to the
|
|||
For example, suppose you want to extract all ``<p>`` elements inside ``<div>``
|
||||
elements. First, you would get all ``<div>`` elements:
|
||||
|
||||
>>> divs = response.xpath('//div')
|
||||
.. code-block:: pycon
|
||||
>>> divs = response.xpath("//div")
|
||||
|
||||
At first, you may be tempted to use the following approach, which is wrong, as
|
||||
it actually extracts all ``<p>`` elements from the document, not only those
|
||||
inside ``<div>`` elements:
|
||||
|
||||
>>> for p in divs.xpath('//p'): # this is wrong - gets all <p> from the whole document
|
||||
... print(p.get())
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> for p in divs.xpath("//p"): # this is wrong - gets all <p> from the whole document
|
||||
... print(p.get())
|
||||
...
|
||||
|
||||
This is the proper way to do it (note the dot prefixing the ``.//p`` XPath):
|
||||
|
||||
>>> for p in divs.xpath('.//p'): # extracts all <p> inside
|
||||
... print(p.get())
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> for p in divs.xpath(".//p"): # extracts all <p> inside
|
||||
... print(p.get())
|
||||
...
|
||||
|
||||
Another common case would be to extract all direct ``<p>`` children:
|
||||
|
||||
>>> for p in divs.xpath('p'):
|
||||
... print(p.get())
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> for p in divs.xpath("p"):
|
||||
... print(p.get())
|
||||
...
|
||||
|
||||
For more details about relative XPaths see the `Location Paths`_ section in the
|
||||
XPath specification.
|
||||
|
|
@ -522,10 +597,14 @@ class name that shares the string ``someclass``.
|
|||
As it turns out, Scrapy selectors allow you to chain selectors, so most of the time
|
||||
you can just select by class using CSS and then switch to XPath when needed:
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> sel = Selector(text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>')
|
||||
>>> sel.css('.shout').xpath('./time/@datetime').getall()
|
||||
['2014-07-23 19:00']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> sel = Selector(
|
||||
... text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>'
|
||||
... )
|
||||
>>> sel.css(".shout").xpath("./time/@datetime").getall()
|
||||
['2014-07-23 19:00']
|
||||
|
||||
This is cleaner than using the verbose XPath trick shown above. Just remember
|
||||
to use the ``.`` in the XPath expressions that will follow.
|
||||
|
|
@ -539,39 +618,51 @@ Beware of the difference between //node[1] and (//node)[1]
|
|||
|
||||
Example:
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> sel = Selector(text="""
|
||||
....: <ul class="list">
|
||||
....: <li>1</li>
|
||||
....: <li>2</li>
|
||||
....: <li>3</li>
|
||||
....: </ul>
|
||||
....: <ul class="list">
|
||||
....: <li>4</li>
|
||||
....: <li>5</li>
|
||||
....: <li>6</li>
|
||||
....: </ul>""")
|
||||
>>> xp = lambda x: sel.xpath(x).getall()
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> sel = Selector(
|
||||
... text="""
|
||||
... <ul class="list">
|
||||
... <li>1</li>
|
||||
... <li>2</li>
|
||||
... <li>3</li>
|
||||
... </ul>
|
||||
... <ul class="list">
|
||||
... <li>4</li>
|
||||
... <li>5</li>
|
||||
... <li>6</li>
|
||||
... </ul>"""
|
||||
... )
|
||||
>>> xp = lambda x: sel.xpath(x).getall()
|
||||
|
||||
This gets all first ``<li>`` elements under whatever it is its parent:
|
||||
|
||||
>>> xp("//li[1]")
|
||||
['<li>1</li>', '<li>4</li>']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> xp("//li[1]")
|
||||
['<li>1</li>', '<li>4</li>']
|
||||
|
||||
And this gets the first ``<li>`` element in the whole document:
|
||||
|
||||
>>> xp("(//li)[1]")
|
||||
['<li>1</li>']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> xp("(//li)[1]")
|
||||
['<li>1</li>']
|
||||
|
||||
This gets all first ``<li>`` elements under an ``<ul>`` parent:
|
||||
|
||||
>>> xp("//ul/li[1]")
|
||||
['<li>1</li>', '<li>4</li>']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> xp("//ul/li[1]")
|
||||
['<li>1</li>', '<li>4</li>']
|
||||
|
||||
And this gets the first ``<li>`` element under an ``<ul>`` parent in the whole document:
|
||||
|
||||
>>> xp("(//ul/li)[1]")
|
||||
['<li>1</li>']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> xp("(//ul/li)[1]")
|
||||
['<li>1</li>']
|
||||
|
||||
Using text nodes in a condition
|
||||
-------------------------------
|
||||
|
|
@ -585,32 +676,44 @@ a string function like ``contains()`` or ``starts-with()``, it results in the te
|
|||
|
||||
Example:
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> sel = Selector(
|
||||
... text='<a href="#">Click here to go to the <strong>Next Page</strong></a>'
|
||||
... )
|
||||
|
||||
Converting a *node-set* to string:
|
||||
|
||||
>>> sel.xpath('//a//text()').getall() # take a peek at the node-set
|
||||
['Click here to go to the ', 'Next Page']
|
||||
>>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
|
||||
['Click here to go to the ']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> sel.xpath("//a//text()").getall() # take a peek at the node-set
|
||||
['Click here to go to the ', 'Next Page']
|
||||
>>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
|
||||
['Click here to go to the ']
|
||||
|
||||
A *node* converted to a string, however, puts together the text of itself plus of all its descendants:
|
||||
|
||||
>>> sel.xpath("//a[1]").getall() # select the first node
|
||||
['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
|
||||
>>> sel.xpath("string(//a[1])").getall() # convert it to string
|
||||
['Click here to go to the Next Page']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> sel.xpath("//a[1]").getall() # select the first node
|
||||
['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
|
||||
>>> sel.xpath("string(//a[1])").getall() # convert it to string
|
||||
['Click here to go to the Next Page']
|
||||
|
||||
So, using the ``.//text()`` node-set won't select anything in this case:
|
||||
|
||||
>>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
|
||||
[]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
|
||||
[]
|
||||
|
||||
But using the ``.`` to mean the node, works:
|
||||
|
||||
>>> sel.xpath("//a[contains(., 'Next Page')]").getall()
|
||||
['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> sel.xpath("//a[contains(., 'Next Page')]").getall()
|
||||
['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
|
||||
|
||||
.. _`XPath string function`: https://www.w3.org/TR/xpath/all/#section-String-Functions
|
||||
|
||||
|
|
@ -628,15 +731,19 @@ which are then substituted with values passed with the query.
|
|||
Here's an example to match an element based on its "id" attribute value,
|
||||
without hard-coding it (that was shown previously):
|
||||
|
||||
>>> # `$val` used in the expression, a `val` argument needs to be passed
|
||||
>>> response.xpath('//div[@id=$val]/a/text()', val='images').get()
|
||||
'Name: My image 1 '
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> # `$val` used in the expression, a `val` argument needs to be passed
|
||||
>>> response.xpath("//div[@id=$val]/a/text()", val="images").get()
|
||||
'Name: My image 1 '
|
||||
|
||||
Here's another example, to find the "id" attribute of a ``<div>`` tag containing
|
||||
five ``<a>`` children (here we pass the value ``5`` as an integer):
|
||||
|
||||
>>> response.xpath('//div[count(a)=$cnt]/@id', cnt=5).get()
|
||||
'images'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//div[count(a)=$cnt]/@id", cnt=5).get()
|
||||
'images'
|
||||
|
||||
All variable references must have a binding value when calling ``.xpath()``
|
||||
(otherwise you'll get a ``ValueError: XPath error:`` exception).
|
||||
|
|
@ -688,17 +795,21 @@ You can see several namespace declarations including a default
|
|||
Once in the shell we can try selecting all ``<link>`` objects and see that it
|
||||
doesn't work (because the Atom XML namespace is obfuscating those nodes):
|
||||
|
||||
>>> response.xpath("//link")
|
||||
[]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//link")
|
||||
[]
|
||||
|
||||
But once we call the :meth:`Selector.remove_namespaces` method, all
|
||||
nodes can be accessed directly by their names:
|
||||
|
||||
>>> response.selector.remove_namespaces()
|
||||
>>> response.xpath("//link")
|
||||
[<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
|
||||
<Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
|
||||
...
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.selector.remove_namespaces()
|
||||
>>> response.xpath("//link")
|
||||
[<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
|
||||
<Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
|
||||
...
|
||||
|
||||
If you wonder why the namespace removal procedure isn't always called by default
|
||||
instead of having to call it manually, this is because of two reasons, which, in order
|
||||
|
|
@ -735,23 +846,25 @@ The ``test()`` function, for example, can prove quite useful when XPath's
|
|||
|
||||
Example selecting links in list item with a "class" attribute ending with a digit:
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> doc = """
|
||||
... <div>
|
||||
... <ul>
|
||||
... <li class="item-0"><a href="link1.html">first item</a></li>
|
||||
... <li class="item-1"><a href="link2.html">second item</a></li>
|
||||
... <li class="item-inactive"><a href="link3.html">third item</a></li>
|
||||
... <li class="item-1"><a href="link4.html">fourth item</a></li>
|
||||
... <li class="item-0"><a href="link5.html">fifth item</a></li>
|
||||
... </ul>
|
||||
... </div>
|
||||
... """
|
||||
>>> sel = Selector(text=doc, type="html")
|
||||
>>> sel.xpath('//li//@href').getall()
|
||||
['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']
|
||||
>>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall()
|
||||
['link1.html', 'link2.html', 'link4.html', 'link5.html']
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> from scrapy import Selector
|
||||
>>> doc = """
|
||||
... <div>
|
||||
... <ul>
|
||||
... <li class="item-0"><a href="link1.html">first item</a></li>
|
||||
... <li class="item-1"><a href="link2.html">second item</a></li>
|
||||
... <li class="item-inactive"><a href="link3.html">third item</a></li>
|
||||
... <li class="item-1"><a href="link4.html">fourth item</a></li>
|
||||
... <li class="item-0"><a href="link5.html">fifth item</a></li>
|
||||
... </ul>
|
||||
... </div>
|
||||
... """
|
||||
>>> sel = Selector(text=doc, type="html")
|
||||
>>> sel.xpath("//li//@href").getall()
|
||||
['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']
|
||||
>>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall()
|
||||
['link1.html', 'link2.html', 'link4.html', 'link5.html']
|
||||
|
||||
.. warning:: C library ``libxslt`` doesn't natively support EXSLT regular
|
||||
expressions so `lxml`_'s implementation uses hooks to Python's ``re`` module.
|
||||
|
|
@ -765,7 +878,9 @@ These can be handy for excluding parts of a document tree before
|
|||
extracting text elements for example.
|
||||
|
||||
Example extracting microdata (sample content taken from https://schema.org/Product)
|
||||
with groups of itemscopes and corresponding itemprops::
|
||||
with groups of itemscopes and corresponding itemprops:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> doc = """
|
||||
... <div itemscope itemtype="http://schema.org/Product">
|
||||
|
|
@ -776,19 +891,15 @@ with groups of itemscopes and corresponding itemprops::
|
|||
... Rated <span itemprop="ratingValue">3.5</span>/5
|
||||
... based on <span itemprop="reviewCount">11</span> customer reviews
|
||||
... </div>
|
||||
...
|
||||
... <div itemprop="offers" itemscope itemtype="http://schema.org/Offer">
|
||||
... <span itemprop="price">$55.00</span>
|
||||
... <link itemprop="availability" href="http://schema.org/InStock" />In stock
|
||||
... </div>
|
||||
...
|
||||
... Product description:
|
||||
... <span itemprop="description">0.7 cubic feet countertop microwave.
|
||||
... Has six preset cooking categories and convenience features like
|
||||
... Add-A-Minute and Child Lock.</span>
|
||||
...
|
||||
... Customer reviews:
|
||||
...
|
||||
... <div itemprop="review" itemscope itemtype="http://schema.org/Review">
|
||||
... <span itemprop="name">Not a happy camper</span> -
|
||||
... by <span itemprop="author">Ellie</span>,
|
||||
|
|
@ -801,7 +912,6 @@ with groups of itemscopes and corresponding itemprops::
|
|||
... <span itemprop="description">The lamp burned out and now I have to replace
|
||||
... it. </span>
|
||||
... </div>
|
||||
...
|
||||
... <div itemprop="review" itemscope itemtype="http://schema.org/Review">
|
||||
... <span itemprop="name">Value purchase</span> -
|
||||
... by <span itemprop="author">Lucas</span>,
|
||||
|
|
@ -818,13 +928,16 @@ with groups of itemscopes and corresponding itemprops::
|
|||
... </div>
|
||||
... """
|
||||
>>> sel = Selector(text=doc, type="html")
|
||||
>>> for scope in sel.xpath('//div[@itemscope]'):
|
||||
... print("current scope:", scope.xpath('@itemtype').getall())
|
||||
... props = scope.xpath('''
|
||||
>>> for scope in sel.xpath("//div[@itemscope]"):
|
||||
... print("current scope:", scope.xpath("@itemtype").getall())
|
||||
... props = scope.xpath(
|
||||
... """
|
||||
... set:difference(./descendant::*/@itemprop,
|
||||
... .//*[@itemscope]/*/@itemprop)''')
|
||||
... .//*[@itemscope]/*/@itemprop)"""
|
||||
... )
|
||||
... print(f" properties: {props.getall()}")
|
||||
... print("")
|
||||
...
|
||||
|
||||
current scope: ['http://schema.org/Product']
|
||||
properties: ['name', 'aggregateRating', 'offers', 'description', 'review', 'review']
|
||||
|
|
@ -876,13 +989,15 @@ For the following HTML::
|
|||
|
||||
You can use it like this:
|
||||
|
||||
>>> response.xpath('//p[has-class("foo")]')
|
||||
[<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
|
||||
<Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
|
||||
>>> response.xpath('//p[has-class("foo", "bar-baz")]')
|
||||
[<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
|
||||
>>> response.xpath('//p[has-class("foo", "bar")]')
|
||||
[]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//p[has-class("foo")]')
|
||||
[<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
|
||||
<Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
|
||||
>>> response.xpath('//p[has-class("foo", "bar-baz")]')
|
||||
[<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
|
||||
>>> response.xpath('//p[has-class("foo", "bar")]')
|
||||
[]
|
||||
|
||||
So XPath ``//p[has-class("foo", "bar-baz")]`` is roughly equivalent to CSS
|
||||
``p.foo.bar-baz``. Please note, that it is slower in most of the cases,
|
||||
|
|
|
|||
|
|
@ -191,44 +191,46 @@ all start with the ``[s]`` prefix)::
|
|||
|
||||
After that, we can start playing with the objects:
|
||||
|
||||
>>> response.xpath('//title/text()').get()
|
||||
'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> fetch("https://old.reddit.com/")
|
||||
>>> response.xpath("//title/text()").get()
|
||||
'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'
|
||||
|
||||
>>> response.xpath('//title/text()').get()
|
||||
'reddit: the front page of the internet'
|
||||
>>> fetch("https://old.reddit.com/")
|
||||
|
||||
>>> request = request.replace(method="POST")
|
||||
>>> response.xpath("//title/text()").get()
|
||||
'reddit: the front page of the internet'
|
||||
|
||||
>>> fetch(request)
|
||||
>>> request = request.replace(method="POST")
|
||||
|
||||
>>> response.status
|
||||
404
|
||||
>>> fetch(request)
|
||||
|
||||
>>> from pprint import pprint
|
||||
>>> response.status
|
||||
404
|
||||
|
||||
>>> pprint(response.headers)
|
||||
{'Accept-Ranges': ['bytes'],
|
||||
'Cache-Control': ['max-age=0, must-revalidate'],
|
||||
'Content-Type': ['text/html; charset=UTF-8'],
|
||||
'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],
|
||||
'Server': ['snooserv'],
|
||||
'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
|
||||
'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
|
||||
'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
|
||||
'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],
|
||||
'Vary': ['accept-encoding'],
|
||||
'Via': ['1.1 varnish'],
|
||||
'X-Cache': ['MISS'],
|
||||
'X-Cache-Hits': ['0'],
|
||||
'X-Content-Type-Options': ['nosniff'],
|
||||
'X-Frame-Options': ['SAMEORIGIN'],
|
||||
'X-Moose': ['majestic'],
|
||||
'X-Served-By': ['cache-cdg8730-CDG'],
|
||||
'X-Timer': ['S1481214079.394283,VS0,VE159'],
|
||||
'X-Ua-Compatible': ['IE=edge'],
|
||||
'X-Xss-Protection': ['1; mode=block']}
|
||||
>>> from pprint import pprint
|
||||
|
||||
>>> pprint(response.headers)
|
||||
{'Accept-Ranges': ['bytes'],
|
||||
'Cache-Control': ['max-age=0, must-revalidate'],
|
||||
'Content-Type': ['text/html; charset=UTF-8'],
|
||||
'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],
|
||||
'Server': ['snooserv'],
|
||||
'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
|
||||
'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
|
||||
'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
|
||||
'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],
|
||||
'Vary': ['accept-encoding'],
|
||||
'Via': ['1.1 varnish'],
|
||||
'X-Cache': ['MISS'],
|
||||
'X-Cache-Hits': ['0'],
|
||||
'X-Content-Type-Options': ['nosniff'],
|
||||
'X-Frame-Options': ['SAMEORIGIN'],
|
||||
'X-Moose': ['majestic'],
|
||||
'X-Served-By': ['cache-cdg8730-CDG'],
|
||||
'X-Timer': ['S1481214079.394283,VS0,VE159'],
|
||||
'X-Ua-Compatible': ['IE=edge'],
|
||||
'X-Xss-Protection': ['1; mode=block']}
|
||||
|
||||
|
||||
.. _topics-shell-inspect-response:
|
||||
|
|
@ -279,14 +281,18 @@ When you run the spider, you will get something similar to this::
|
|||
|
||||
Then, you can check if the extraction code is working:
|
||||
|
||||
>>> response.xpath('//h1[@class="fn"]')
|
||||
[]
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//h1[@class="fn"]')
|
||||
[]
|
||||
|
||||
Nope, it doesn't. So you can open the response in your web browser and see if
|
||||
it's the response you were expecting:
|
||||
|
||||
>>> view(response)
|
||||
True
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> view(response)
|
||||
True
|
||||
|
||||
Finally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume the
|
||||
crawling::
|
||||
|
|
|
|||
|
|
@ -68,13 +68,17 @@ Set stat value only if lower than previous:
|
|||
|
||||
Get stat value:
|
||||
|
||||
>>> stats.get_value('custom_count')
|
||||
1
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> stats.get_value("custom_count")
|
||||
1
|
||||
|
||||
Get all stats:
|
||||
|
||||
>>> stats.get_stats()
|
||||
{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> stats.get_stats()
|
||||
{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
|
||||
|
||||
Available Stats Collectors
|
||||
==========================
|
||||
|
|
|
|||
Loading…
Reference in New Issue