diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index f5e9b372e..e565c3ebb 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -83,7 +83,9 @@ optionally how to follow links in the pages, and how to parse the downloaded page content to extract data. This is the code for our first Spider. Save it in a file named -``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project:: +``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project: + +.. code-block:: python from pathlib import Path @@ -95,17 +97,17 @@ This is the code for our first Spider. Save it in a file named def start_requests(self): urls = [ - 'https://quotes.toscrape.com/page/1/', - 'https://quotes.toscrape.com/page/2/', + "https://quotes.toscrape.com/page/1/", + "https://quotes.toscrape.com/page/2/", ] for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): page = response.url.split("/")[-2] - filename = f'quotes-{page}.html' + filename = f"quotes-{page}.html" Path(filename).write_bytes(response.body) - self.log(f'Saved file {filename}') + self.log(f"Saved file {filename}") As you can see, our Spider subclasses :class:`scrapy.Spider ` @@ -247,8 +249,10 @@ object: response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html') ->>> response.css('title') -[] +.. code-block:: pycon + + >>> response.css("title") + [] The result of running ``response.css('title')`` is a list-like object called :class:`~scrapy.selector.SelectorList`, which represents a list of @@ -258,42 +262,54 @@ data. To extract the text from the title above, you can do: ->>> response.css('title::text').getall() -['Quotes to Scrape'] +.. code-block:: pycon + + >>> response.css("title::text").getall() + ['Quotes to Scrape'] There are two things to note here: one is that we've added ``::text`` to the CSS query, to mean we want to select only the text elements directly inside ```` element. If we don't specify ``::text``, we'd get the full title element, including its tags: ->>> response.css('title').getall() -['<title>Quotes to Scrape'] +.. code-block:: pycon + + >>> response.css("title").getall() + ['Quotes to Scrape'] The other thing is that the result of calling ``.getall()`` is a list: it is possible that a selector returns more than one result, so we extract them all. When you know you just want the first result, as in this case, you can do: ->>> response.css('title::text').get() -'Quotes to Scrape' +.. code-block:: pycon + + >>> response.css("title::text").get() + 'Quotes to Scrape' As an alternative, you could've written: ->>> response.css('title::text')[0].get() -'Quotes to Scrape' +.. code-block:: pycon + + >>> response.css("title::text")[0].get() + 'Quotes to Scrape' Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will -raise an :exc:`IndexError` exception if there are no results:: +raise an :exc:`IndexError` exception if there are no results: - >>> response.css('noelement')[0].get() +.. code-block:: pycon + + >>> response.css("noelement")[0].get() Traceback (most recent call last): ... IndexError: list index out of range You might want to use ``.get()`` directly on the :class:`~scrapy.selector.SelectorList` instance instead, which returns ``None`` -if there are no results:: +if there are no results: ->>> response.css("noelement").get() +.. code-block:: pycon + + >>> response.css("noelement").get() There's a lesson here: for most scraping code, you want it to be resilient to errors due to things not being found on a page, so that even if some parts fail @@ -304,12 +320,14 @@ Besides the :meth:`~scrapy.selector.SelectorList.getall` and the :meth:`~scrapy.selector.SelectorList.re` method to extract using :doc:`regular expressions `: ->>> response.css('title::text').re(r'Quotes.*') -['Quotes to Scrape'] ->>> response.css('title::text').re(r'Q\w+') -['Quotes'] ->>> response.css('title::text').re(r'(\w+) to (\w+)') -['Quotes', 'Scrape'] +.. code-block:: pycon + + >>> response.css("title::text").re(r"Quotes.*") + ['Quotes to Scrape'] + >>> response.css("title::text").re(r"Q\w+") + ['Quotes'] + >>> response.css("title::text").re(r"(\w+) to (\w+)") + ['Quotes', 'Scrape'] In order to find the proper CSS selectors to use, you might find useful opening the response page from the shell in your web browser using ``view(response)``. @@ -327,10 +345,12 @@ XPath: a brief intro Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions: ->>> response.xpath('//title') -[] ->>> response.xpath('//title/text()').get() -'Quotes to Scrape' +.. code-block:: pycon + + >>> response.xpath("//title") + [] + >>> response.xpath("//title/text()").get() + 'Quotes to Scrape' XPath expressions are very powerful, and are the foundation of Scrapy Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You @@ -387,33 +407,41 @@ we want:: We get a list of selectors for the quote HTML elements with: ->>> response.css("div.quote") -[, - , - ...] +.. code-block:: pycon + + >>> response.css("div.quote") + [, + , + ...] Each of the selectors returned by the query above allows us to run further queries over their sub-elements. Let's assign the first selector to a variable, so that we can run our CSS selectors directly on a particular quote: ->>> quote = response.css("div.quote")[0] +.. code-block:: pycon + + >>> quote = response.css("div.quote")[0] Now, let's extract ``text``, ``author`` and the ``tags`` from that quote using the ``quote`` object we just created: ->>> text = quote.css("span.text::text").get() ->>> text -'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”' ->>> author = quote.css("small.author::text").get() ->>> author -'Albert Einstein' +.. code-block:: pycon + + >>> text = quote.css("span.text::text").get() + >>> text + '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”' + >>> author = quote.css("small.author::text").get() + >>> author + 'Albert Einstein' Given that the tags are a list of strings, we can use the ``.getall()`` method to get all of them: ->>> tags = quote.css("div.tags a.tag::text").getall() ->>> tags -['change', 'deep-thoughts', 'thinking', 'world'] +.. code-block:: pycon + + >>> tags = quote.css("div.tags a.tag::text").getall() + >>> tags + ['change', 'deep-thoughts', 'thinking', 'world'] .. invisible-code-block: python @@ -422,14 +450,17 @@ to get all of them: Having figured out how to extract each bit, we can now iterate over all the quotes elements and put them together into a Python dictionary: ->>> for quote in response.css("div.quote"): -... text = quote.css("span.text::text").get() -... author = quote.css("small.author::text").get() -... tags = quote.css("div.tags a.tag::text").getall() -... print(dict(text=text, author=author, tags=tags)) -{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']} -{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']} -... +.. code-block:: pycon + + >>> for quote in response.css("div.quote"): + ... text = quote.css("span.text::text").get() + ... author = quote.css("small.author::text").get() + ... tags = quote.css("div.tags a.tag::text").getall() + ... print(dict(text=text, author=author, tags=tags)) + ... + {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']} + {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']} + ... Extracting data in our spider ----------------------------- @@ -537,14 +568,18 @@ This gets the anchor element, but we want the attribute ``href``. For that, Scrapy supports a CSS extension that lets you select the attribute contents, like this: ->>> response.css('li.next a::attr(href)').get() -'/page/2/' +.. code-block:: pycon + + >>> response.css("li.next a::attr(href)").get() + '/page/2/' There is also an ``attrib`` property available (see :ref:`selecting-attributes` for more): ->>> response.css('li.next a').attrib['href'] -'/page/2/' +.. code-block:: pycon + + >>> response.css("li.next a").attrib["href"] + '/page/2/' Let's see now our spider modified to recursively follow the link to the next page, extracting data from it: diff --git a/docs/news.rst b/docs/news.rst index 07264827b..dd1574e26 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -2417,11 +2417,13 @@ Backward-incompatible changes * :class:`~scrapy.loader.ItemLoader` now turns the values of its input item into lists: - >>> item = MyItem() - >>> item['field'] = 'value1' - >>> loader = ItemLoader(item=item) - >>> item['field'] - ['value1'] + .. code-block:: pycon + + >>> item = MyItem() + >>> item["field"] = "value1" + >>> loader = ItemLoader(item=item) + >>> item["field"] + ['value1'] This is needed to allow adding values to existing fields (``loader.add_value('field', 'value2')``). diff --git a/docs/topics/api.rst b/docs/topics/api.rst index 60b5acd10..1eac444e6 100644 --- a/docs/topics/api.rst +++ b/docs/topics/api.rst @@ -134,14 +134,14 @@ Settings API .. highlight:: python - :: + .. code-block:: python SETTINGS_PRIORITIES = { - 'default': 0, - 'command': 10, - 'project': 20, - 'spider': 30, - 'cmdline': 40, + "default": 0, + "command": 10, + "project": 20, + "spider": 30, + "cmdline": 40, } For a detailed explanation on each settings sources, see: diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst index 39e7b7d3c..a15ee1059 100644 --- a/docs/topics/developer-tools.rst +++ b/docs/topics/developer-tools.rst @@ -94,8 +94,10 @@ Then, back to your web browser, right-click on the ``span`` tag, select response = load_response('https://quotes.toscrape.com/', 'quotes.html') ->>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall() -['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'] +.. code-block:: pycon + + >>> response.xpath("/html/body/div/div[2]/div[1]/div[1]/span[1]/text()").getall() + ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'] Adding ``text()`` at the end we are able to extract the first quote with this basic selector. But this XPath is not really that clever. All it does is @@ -124,11 +126,13 @@ With this knowledge we can refine our XPath: Instead of a path to follow, we'll simply select all ``span`` tags with the ``class="text"`` by using the `has-class-extension`_: ->>> response.xpath('//span[has-class("text")]/text()').getall() -['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', -'“It is our choices, Harry, that show what we truly are, far more than our abilities.”', -'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', -...] +.. code-block:: pycon + + >>> response.xpath('//span[has-class("text")]/text()').getall() + ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', + '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', + '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', + ...] And with one simple, cleverer XPath we are able to extract all quotes from the page. We could have constructed a loop over our first XPath to increase diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index 9be0ed058..d01e0a8d4 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -183,10 +183,12 @@ data from it: For example, if the JavaScript code contains a separate line like ``var data = {"field": "value"};`` you can extract that data as follows: - >>> pattern = r'\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n' - >>> json_data = response.css('script::text').re_first(pattern) - >>> json.loads(json_data) - {'field': 'value'} + .. code-block:: pycon + + >>> pattern = r"\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n" + >>> json_data = response.css("script::text").re_first(pattern) + >>> json.loads(json_data) + {'field': 'value'} - chompjs_ provides an API to parse JavaScript objects into a :class:`dict`. @@ -194,11 +196,13 @@ data from it: ``var data = {field: "value", secondField: "second value"};`` you can extract that data as follows: - >>> import chompjs - >>> javascript = response.css('script::text').get() - >>> data = chompjs.parse_js_object(javascript) - >>> data - {'field': 'value', 'secondField': 'second value'} + .. code-block:: pycon + + >>> import chompjs + >>> javascript = response.css("script::text").get() + >>> data = chompjs.parse_js_object(javascript) + >>> data + {'field': 'value', 'secondField': 'second value'} - Otherwise, use js2xml_ to convert the JavaScript code into an XML document that you can parse using :ref:`selectors `. @@ -206,14 +210,16 @@ data from it: For example, if the JavaScript code contains ``var data = {field: "value"};`` you can extract that data as follows: - >>> import js2xml - >>> import lxml.etree - >>> from parsel import Selector - >>> javascript = response.css('script::text').get() - >>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding='unicode') - >>> selector = Selector(text=xml) - >>> selector.css('var[name="data"]').get() - 'value' + .. code-block:: pycon + + >>> import js2xml + >>> import lxml.etree + >>> from parsel import Selector + >>> javascript = response.css("script::text").get() + >>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding="unicode") + >>> selector = Selector(text=xml) + >>> selector.css('var[name="data"]').get() + 'value' .. _topics-javascript-rendering: diff --git a/docs/topics/items.rst b/docs/topics/items.rst index 7b6dfc5c1..3c38ac2dc 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -234,62 +234,68 @@ notice the API is very similar to the :class:`dict` API. Creating items '''''''''''''' ->>> product = Product(name='Desktop PC', price=1000) ->>> print(product) -Product(name='Desktop PC', price=1000) +.. code-block:: pycon + + >>> product = Product(name="Desktop PC", price=1000) + >>> print(product) + Product(name='Desktop PC', price=1000) Getting field values '''''''''''''''''''' ->>> product['name'] -Desktop PC ->>> product.get('name') -Desktop PC +.. code-block:: pycon ->>> product['price'] -1000 + >>> product["name"] + Desktop PC + >>> product.get("name") + Desktop PC ->>> product['last_updated'] -Traceback (most recent call last): - ... -KeyError: 'last_updated' + >>> product["price"] + 1000 ->>> product.get('last_updated', 'not set') -not set + >>> product["last_updated"] + Traceback (most recent call last): + ... + KeyError: 'last_updated' ->>> product['lala'] # getting unknown field -Traceback (most recent call last): - ... -KeyError: 'lala' + >>> product.get("last_updated", "not set") + not set ->>> product.get('lala', 'unknown field') -'unknown field' + >>> product["lala"] # getting unknown field + Traceback (most recent call last): + ... + KeyError: 'lala' ->>> 'name' in product # is name field populated? -True + >>> product.get("lala", "unknown field") + 'unknown field' ->>> 'last_updated' in product # is last_updated populated? -False + >>> "name" in product # is name field populated? + True ->>> 'last_updated' in product.fields # is last_updated a declared field? -True + >>> "last_updated" in product # is last_updated populated? + False ->>> 'lala' in product.fields # is lala a declared field? -False + >>> "last_updated" in product.fields # is last_updated a declared field? + True + + >>> "lala" in product.fields # is lala a declared field? + False Setting field values '''''''''''''''''''' ->>> product['last_updated'] = 'today' ->>> product['last_updated'] -today +.. code-block:: pycon ->>> product['lala'] = 'test' # setting unknown field -Traceback (most recent call last): - ... -KeyError: 'Product does not support field: lala' + >>> product["last_updated"] = "today" + >>> product["last_updated"] + today + + >>> product["lala"] = "test" # setting unknown field + Traceback (most recent call last): + ... + KeyError: 'Product does not support field: lala' Accessing all populated values @@ -297,11 +303,13 @@ Accessing all populated values To access all populated values, just use the typical :class:`dict` API: ->>> product.keys() -['price', 'name'] +.. code-block:: pycon ->>> product.items() -[('price', 1000), ('name', 'Desktop PC')] + >>> product.keys() + ['price', 'name'] + + >>> product.items() + [('price', 1000), ('name', 'Desktop PC')] .. _copying-items: @@ -339,18 +347,20 @@ Other common tasks Creating dicts from items: ->>> dict(product) # create a dict from all populated values -{'price': 1000, 'name': 'Desktop PC'} +.. code-block:: pycon -Creating items from dicts: + >>> dict(product) # create a dict from all populated values + {'price': 1000, 'name': 'Desktop PC'} ->>> Product({'name': 'Laptop PC', 'price': 1500}) -Product(price=1500, name='Laptop PC') + Creating items from dicts: ->>> Product({'name': 'Laptop PC', 'lala': 1500}) # warning: unknown field in dict -Traceback (most recent call last): - ... -KeyError: 'Product does not support field: lala' + >>> Product({"name": "Laptop PC", "price": 1500}) + Product(price=1500, name='Laptop PC') + + >>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict + Traceback (most recent call last): + ... + KeyError: 'Product does not support field: lala' Extending Item subclasses diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index 33441838a..cd8914644 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -70,13 +70,15 @@ alias to the :func:`~scrapy.utils.trackref.print_live_refs` function:: telnet localhost 6023 - >>> prefs() - Live References + .. code-block:: pycon - ExampleSpider 1 oldest: 15s ago - HtmlResponse 10 oldest: 1s ago - Selector 2 oldest: 0s ago - FormRequest 878 oldest: 7s ago + >>> prefs() + Live References + + ExampleSpider 1 oldest: 15s ago + HtmlResponse 10 oldest: 1s ago + Selector 2 oldest: 0s ago + FormRequest 878 oldest: 7s ago As you can see, that report also shows the "age" of the oldest object in each class. If you're running multiple spiders per process chances are you can @@ -114,7 +116,9 @@ a priori, of course) by using the ``trackref`` tool. After the crawler is running for a few minutes and we notice its memory usage has grown a lot, we can enter its telnet console and check the live -references:: +references: + +.. code-block:: pycon >>> prefs() Live References @@ -134,19 +138,23 @@ generating the leaks (passing response references inside requests). Sometimes extra information about live objects can be helpful. Let's check the oldest response: ->>> from scrapy.utils.trackref import get_oldest ->>> r = get_oldest('HtmlResponse') ->>> r.url -'http://www.somenastyspider.com/product.php?pid=123' +.. code-block:: pycon + + >>> from scrapy.utils.trackref import get_oldest + >>> r = get_oldest("HtmlResponse") + >>> r.url + 'http://www.somenastyspider.com/product.php?pid=123' If you want to iterate over all objects, instead of getting the oldest one, you can use the :func:`scrapy.utils.trackref.iter_all` function: ->>> from scrapy.utils.trackref import iter_all ->>> [r.url for r in iter_all('HtmlResponse')] -['http://www.somenastyspider.com/product.php?pid=123', - 'http://www.somenastyspider.com/product.php?pid=584', -...] +.. code-block:: pycon + + >>> from scrapy.utils.trackref import iter_all + >>> [r.url for r in iter_all("HtmlResponse")] + ['http://www.somenastyspider.com/product.php?pid=123', + 'http://www.somenastyspider.com/product.php?pid=584', + ...] Too many spiders? ----------------- @@ -157,8 +165,10 @@ For this reason, that function has a ``ignore`` argument which can be used to ignore a particular class (and all its subclasses). For example, this won't show any live references to spiders: ->>> from scrapy.spiders import Spider ->>> prefs(ignore=Spider) +.. code-block:: pycon + + >>> from scrapy.spiders import Spider + >>> prefs(ignore=Spider) .. module:: scrapy.utils.trackref :synopsis: Track references of live objects @@ -216,30 +226,32 @@ If you use ``pip``, you can install muppy with the following command:: Here's an example to view all Python objects available in the heap using muppy: ->>> from pympler import muppy ->>> all_objects = muppy.get_objects() ->>> len(all_objects) -28667 ->>> from pympler import summary ->>> suml = summary.summarize(all_objects) ->>> summary.print_(suml) - types | # objects | total size -==================================== | =========== | ============ - >> from pympler import muppy + >>> all_objects = muppy.get_objects() + >>> len(all_objects) + 28667 + >>> from pympler import summary + >>> suml = summary.summarize(all_objects) + >>> summary.print_(suml) + types | # objects | total size + ==================================== | =========== | ============ + >> from scrapy.loader import ItemLoader ->>> il = ItemLoader(item=Product()) ->>> il.add_value('name', ['Welcome to my', 'website']) ->>> il.add_value('price', ['€', '1000']) ->>> il.load_item() -{'name': 'Welcome to my website', 'price': '1000'} + +.. code-block:: pycon + + >>> from scrapy.loader import ItemLoader + >>> il = ItemLoader(item=Product()) + >>> il.add_value("name", ["Welcome to my", "website"]) + >>> il.add_value("price", ["€", "1000"]) + >>> il.load_item() + {'name': 'Welcome to my website', 'price': '1000'} The precedence order, for both input and output processors, is as follows: diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 090554e6e..fd38eb504 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -1243,8 +1243,10 @@ TextResponse objects ``str(response.body)`` is not a correct way to convert the response body into a string: - >>> str(b'body') - "b'body'" + .. code-block:: pycon + + >>> str(b"body") + "b'body'" .. attribute:: TextResponse.encoding diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index ede3ca7cd..80a979650 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -51,16 +51,20 @@ Constructing selectors Response objects expose a :class:`~scrapy.Selector` instance on ``.selector`` attribute: ->>> response.selector.xpath('//span/text()').get() -'good' +.. code-block:: pycon + + >>> response.selector.xpath("//span/text()").get() + 'good' Querying responses using XPath and CSS is so common that responses include two more shortcuts: ``response.xpath()`` and ``response.css()``: ->>> response.xpath('//span/text()').get() -'good' ->>> response.css('span::text').get() -'good' +.. code-block:: pycon + + >>> response.xpath("//span/text()").get() + 'good' + >>> response.css("span::text").get() + 'good' Scrapy selectors are instances of :class:`~scrapy.Selector` class constructed by passing either :class:`~scrapy.http.TextResponse` object or @@ -75,19 +79,23 @@ you can also ensure the response body is parsed only once. But if required, it is possible to use ``Selector`` directly. Constructing from text: ->>> from scrapy.selector import Selector ->>> body = 'good' ->>> Selector(text=body).xpath('//span/text()').get() -'good' +.. code-block:: pycon + + >>> from scrapy.selector import Selector + >>> body = "good" + >>> Selector(text=body).xpath("//span/text()").get() + 'good' Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of :class:`~scrapy.http.TextResponse` subclasses: ->>> from scrapy.selector import Selector ->>> from scrapy.http import HtmlResponse ->>> response = HtmlResponse(url='http://example.com', body=body) ->>> Selector(response=response).xpath('//span/text()').get() -'good' +.. code-block:: pycon + + >>> from scrapy.selector import Selector + >>> from scrapy.http import HtmlResponse + >>> response = HtmlResponse(url="http://example.com", body=body) + >>> Selector(response=response).xpath("//span/text()").get() + 'good' ``Selector`` automatically chooses the best parsing rules (XML vs HTML) based on input type. @@ -124,16 +132,20 @@ Since we're dealing with HTML, the selector will automatically use an HTML parse So, by looking at the :ref:`HTML code ` of that page, let's construct an XPath for selecting the text inside the title tag: ->>> response.xpath('//title/text()') -[] +.. code-block:: pycon + + >>> response.xpath("//title/text()") + [] To actually extract the textual data, you must call the selector ``.get()`` or ``.getall()`` methods, as follows: ->>> response.xpath('//title/text()').getall() -['Example website'] ->>> response.xpath('//title/text()').get() -'Example website' +.. code-block:: pycon + + >>> response.xpath("//title/text()").getall() + ['Example website'] + >>> response.xpath("//title/text()").get() + 'Example website' ``.get()`` always returns a single result; if there are several matches, content of a first match is returned; if there are no matches, None @@ -142,98 +154,116 @@ is returned. ``.getall()`` returns a list with all results. Notice that CSS selectors can select text or attribute nodes using CSS3 pseudo-elements: ->>> response.css('title::text').get() -'Example website' +.. code-block:: pycon + + >>> response.css("title::text").get() + 'Example website' As you can see, ``.xpath()`` and ``.css()`` methods return a :class:`~scrapy.selector.SelectorList` instance, which is a list of new selectors. This API can be used for quickly selecting nested data: ->>> response.css('img').xpath('@src').getall() -['image1_thumb.jpg', - 'image2_thumb.jpg', - 'image3_thumb.jpg', - 'image4_thumb.jpg', - 'image5_thumb.jpg'] +.. code-block:: pycon + + >>> response.css("img").xpath("@src").getall() + ['image1_thumb.jpg', + 'image2_thumb.jpg', + 'image3_thumb.jpg', + 'image4_thumb.jpg', + 'image5_thumb.jpg'] If you want to extract only the first matched element, you can call the selector ``.get()`` (or its alias ``.extract_first()`` commonly used in previous Scrapy versions): ->>> response.xpath('//div[@id="images"]/a/text()').get() -'Name: My image 1 ' +.. code-block:: pycon + + >>> response.xpath('//div[@id="images"]/a/text()').get() + 'Name: My image 1 ' It returns ``None`` if no element was found: ->>> response.xpath('//div[@id="not-exists"]/text()').get() is None -True +.. code-block:: pycon + + >>> response.xpath('//div[@id="not-exists"]/text()').get() is None + True A default return value can be provided as an argument, to be used instead of ``None``: ->>> response.xpath('//div[@id="not-exists"]/text()').get(default='not-found') -'not-found' +.. code-block:: pycon + + >>> response.xpath('//div[@id="not-exists"]/text()').get(default="not-found") + 'not-found' Instead of using e.g. ``'@src'`` XPath it is possible to query for attributes using ``.attrib`` property of a :class:`~scrapy.Selector`: ->>> [img.attrib['src'] for img in response.css('img')] -['image1_thumb.jpg', - 'image2_thumb.jpg', - 'image3_thumb.jpg', - 'image4_thumb.jpg', - 'image5_thumb.jpg'] +.. code-block:: pycon + + >>> [img.attrib["src"] for img in response.css("img")] + ['image1_thumb.jpg', + 'image2_thumb.jpg', + 'image3_thumb.jpg', + 'image4_thumb.jpg', + 'image5_thumb.jpg'] As a shortcut, ``.attrib`` is also available on SelectorList directly; it returns attributes for the first matching element: ->>> response.css('img').attrib['src'] -'image1_thumb.jpg' +.. code-block:: pycon + + >>> response.css("img").attrib["src"] + 'image1_thumb.jpg' This is most useful when only a single result is expected, e.g. when selecting by id, or selecting unique elements on a web page: ->>> response.css('base').attrib['href'] -'http://example.com/' +.. code-block:: pycon + + >>> response.css("base").attrib["href"] + 'http://example.com/' Now we're going to get the base URL and some image links: ->>> response.xpath('//base/@href').get() -'http://example.com/' +.. code-block:: pycon ->>> response.css('base::attr(href)').get() -'http://example.com/' + >>> response.xpath("//base/@href").get() + 'http://example.com/' ->>> response.css('base').attrib['href'] -'http://example.com/' + >>> response.css("base::attr(href)").get() + 'http://example.com/' ->>> response.xpath('//a[contains(@href, "image")]/@href').getall() -['image1.html', - 'image2.html', - 'image3.html', - 'image4.html', - 'image5.html'] + >>> response.css("base").attrib["href"] + 'http://example.com/' ->>> response.css('a[href*=image]::attr(href)').getall() -['image1.html', - 'image2.html', - 'image3.html', - 'image4.html', - 'image5.html'] + >>> response.xpath('//a[contains(@href, "image")]/@href').getall() + ['image1.html', + 'image2.html', + 'image3.html', + 'image4.html', + 'image5.html'] ->>> response.xpath('//a[contains(@href, "image")]/img/@src').getall() -['image1_thumb.jpg', - 'image2_thumb.jpg', - 'image3_thumb.jpg', - 'image4_thumb.jpg', - 'image5_thumb.jpg'] + >>> response.css("a[href*=image]::attr(href)").getall() + ['image1.html', + 'image2.html', + 'image3.html', + 'image4.html', + 'image5.html'] ->>> response.css('a[href*=image] img::attr(src)').getall() -['image1_thumb.jpg', - 'image2_thumb.jpg', - 'image3_thumb.jpg', - 'image4_thumb.jpg', - 'image5_thumb.jpg'] + >>> response.xpath('//a[contains(@href, "image")]/img/@src').getall() + ['image1_thumb.jpg', + 'image2_thumb.jpg', + 'image3_thumb.jpg', + 'image4_thumb.jpg', + 'image5_thumb.jpg'] + + >>> response.css("a[href*=image] img::attr(src)").getall() + ['image1_thumb.jpg', + 'image2_thumb.jpg', + 'image3_thumb.jpg', + 'image4_thumb.jpg', + 'image5_thumb.jpg'] .. _topics-selectors-css-extensions: @@ -260,45 +290,55 @@ Examples: * ``title::text`` selects children text nodes of a descendant ```` element: ->>> response.css('title::text').get() -'Example website' +.. code-block:: pycon + + >>> response.css("title::text").get() + 'Example website' * ``*::text`` selects all descendant text nodes of the current selector context: ->>> response.css('#images *::text').getall() -['\n ', - 'Name: My image 1 ', - '\n ', - 'Name: My image 2 ', - '\n ', - 'Name: My image 3 ', - '\n ', - 'Name: My image 4 ', - '\n ', - 'Name: My image 5 ', - '\n '] +.. code-block:: pycon + + >>> response.css("#images *::text").getall() + ['\n ', + 'Name: My image 1 ', + '\n ', + 'Name: My image 2 ', + '\n ', + 'Name: My image 3 ', + '\n ', + 'Name: My image 4 ', + '\n ', + 'Name: My image 5 ', + '\n '] * ``foo::text`` returns no results if ``foo`` element exists, but contains no text (i.e. text is empty): ->>> response.css('img::text').getall() -[] +.. code-block:: pycon - This means ``.css('foo::text').get()`` could return None even if an element - exists. Use ``default=''`` if you always want a string: + >>> response.css("img::text").getall() + [] ->>> response.css('img::text').get() ->>> response.css('img::text').get(default='') -'' + is means ``.css('foo::text').get()`` could return None even if an element + ists. Use ``default=''`` if you always want a string: + +.. code-block:: pycon + + >>> response.css("img::text").get() + >>> response.css("img::text").get(default="") + '' * ``a::attr(href)`` selects the *href* attribute value of descendant links: ->>> response.css('a::attr(href)').getall() -['image1.html', - 'image2.html', - 'image3.html', - 'image4.html', - 'image5.html'] +.. code-block:: pycon + + >>> response.css("a::attr(href)").getall() + ['image1.html', + 'image2.html', + 'image3.html', + 'image4.html', + 'image5.html'] .. note:: See also: :ref:`selecting-attributes`. @@ -319,23 +359,26 @@ The selection methods (``.xpath()`` or ``.css()``) return a list of selectors of the same type, so you can call the selection methods for those selectors too. Here's an example: ->>> links = response.xpath('//a[contains(@href, "image")]') ->>> links.getall() -['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>', - '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>', - '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>', - '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>', - '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>'] +.. code-block:: pycon ->>> for index, link in enumerate(links): -... href_xpath = link.xpath('@href').get() -... img_xpath = link.xpath('img/@src').get() -... print(f'Link number {index} points to url {href_xpath!r} and image {img_xpath!r}') -Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg' -Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg' -Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg' -Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg' -Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg' + >>> links = response.xpath('//a[contains(@href, "image")]') + >>> links.getall() + ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>', + '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>', + '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>', + '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>', + '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>'] + + >>> for index, link in enumerate(links): + ... href_xpath = link.xpath("@href").get() + ... img_xpath = link.xpath("img/@src").get() + ... print(f"Link number {index} points to url {href_xpath!r} and image {img_xpath!r}") + ... + Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg' + Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg' + Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg' + Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg' + Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg' .. _selecting-attributes: @@ -345,8 +388,10 @@ Selecting element attributes There are several ways to get a value of an attribute. First, one can use XPath syntax: ->>> response.xpath("//a/@href").getall() -['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] +.. code-block:: pycon + + >>> response.xpath("//a/@href").getall() + ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] XPath syntax has a few advantages: it is a standard XPath feature, and ``@attributes`` can be used in other parts of an XPath expression - e.g. @@ -355,30 +400,38 @@ it is possible to filter by attribute value. Scrapy also provides an extension to CSS selectors (``::attr(...)``) which allows to get attribute values: ->>> response.css('a::attr(href)').getall() -['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] +.. code-block:: pycon + + >>> response.css("a::attr(href)").getall() + ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] In addition to that, there is a ``.attrib`` property of Selector. You can use it if you prefer to lookup attributes in Python code, without using XPaths or CSS extensions: ->>> [a.attrib['href'] for a in response.css('a')] -['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] +.. code-block:: pycon + + >>> [a.attrib["href"] for a in response.css("a")] + ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] This property is also available on SelectorList; it returns a dictionary with attributes of a first matching element. It is convenient to use when a selector is expected to give a single result (e.g. when selecting by element ID, or when selecting an unique element on a page): ->>> response.css('base').attrib -{'href': 'http://example.com/'} ->>> response.css('base').attrib['href'] -'http://example.com/' +.. code-block:: pycon + + >>> response.css("base").attrib + {'href': 'http://example.com/'} + >>> response.css("base").attrib["href"] + 'http://example.com/' ``.attrib`` property of an empty SelectorList is empty: ->>> response.css('foo').attrib -{} +.. code-block:: pycon + + >>> response.css("foo").attrib + {} Using selectors with regular expressions ---------------------------------------- @@ -391,19 +444,23 @@ can't construct nested ``.re()`` calls. Here's an example used to extract image names from the :ref:`HTML code <topics-selectors-htmlcode>` above: ->>> response.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)') -['My image 1', - 'My image 2', - 'My image 3', - 'My image 4', - 'My image 5'] +.. code-block:: pycon + + >>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)") + ['My image 1', + 'My image 2', + 'My image 3', + 'My image 4', + 'My image 5'] There's an additional helper reciprocating ``.get()`` (and its alias ``.extract_first()``) for ``.re()``, named ``.re_first()``. Use it to extract just the first matching string: ->>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r'Name:\s*(.*)') -'My image 1' +.. code-block:: pycon + + >>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)") + 'My image 1' .. _old-extraction-api: @@ -423,28 +480,36 @@ The following examples show how these methods map to each other. 1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``: - >>> response.css('a::attr(href)').get() +.. code-block:: pycon + + >>> response.css("a::attr(href)").get() 'image1.html' - >>> response.css('a::attr(href)').extract_first() + >>> response.css("a::attr(href)").extract_first() 'image1.html' 2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``: - >>> response.css('a::attr(href)').getall() +.. code-block:: pycon + + >>> response.css("a::attr(href)").getall() ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] - >>> response.css('a::attr(href)').extract() + >>> response.css("a::attr(href)").extract() ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] 3. ``Selector.get()`` is the same as ``Selector.extract()``: - >>> response.css('a::attr(href)')[0].get() +.. code-block:: pycon + + >>> response.css("a::attr(href)")[0].get() 'image1.html' - >>> response.css('a::attr(href)')[0].extract() + >>> response.css("a::attr(href)")[0].extract() 'image1.html' 4. For consistency, there is also ``Selector.getall()``, which returns a list: - >>> response.css('a::attr(href)')[0].getall() +.. code-block:: pycon + + >>> response.css("a::attr(href)")[0].getall() ['image1.html'] So, the main difference is that output of ``.get()`` and ``.getall()`` methods @@ -482,24 +547,34 @@ with ``/``, that XPath will be absolute to the document and not relative to the For example, suppose you want to extract all ``<p>`` elements inside ``<div>`` elements. First, you would get all ``<div>`` elements: ->>> divs = response.xpath('//div') +.. code-block:: pycon + >>> divs = response.xpath("//div") At first, you may be tempted to use the following approach, which is wrong, as it actually extracts all ``<p>`` elements from the document, not only those inside ``<div>`` elements: ->>> for p in divs.xpath('//p'): # this is wrong - gets all <p> from the whole document -... print(p.get()) +.. code-block:: pycon + + >>> for p in divs.xpath("//p"): # this is wrong - gets all <p> from the whole document + ... print(p.get()) + ... This is the proper way to do it (note the dot prefixing the ``.//p`` XPath): ->>> for p in divs.xpath('.//p'): # extracts all <p> inside -... print(p.get()) +.. code-block:: pycon + + >>> for p in divs.xpath(".//p"): # extracts all <p> inside + ... print(p.get()) + ... Another common case would be to extract all direct ``<p>`` children: ->>> for p in divs.xpath('p'): -... print(p.get()) +.. code-block:: pycon + + >>> for p in divs.xpath("p"): + ... print(p.get()) + ... For more details about relative XPaths see the `Location Paths`_ section in the XPath specification. @@ -522,10 +597,14 @@ class name that shares the string ``someclass``. As it turns out, Scrapy selectors allow you to chain selectors, so most of the time you can just select by class using CSS and then switch to XPath when needed: ->>> from scrapy import Selector ->>> sel = Selector(text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>') ->>> sel.css('.shout').xpath('./time/@datetime').getall() -['2014-07-23 19:00'] +.. code-block:: pycon + + >>> from scrapy import Selector + >>> sel = Selector( + ... text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>' + ... ) + >>> sel.css(".shout").xpath("./time/@datetime").getall() + ['2014-07-23 19:00'] This is cleaner than using the verbose XPath trick shown above. Just remember to use the ``.`` in the XPath expressions that will follow. @@ -539,39 +618,51 @@ Beware of the difference between //node[1] and (//node)[1] Example: ->>> from scrapy import Selector ->>> sel = Selector(text=""" -....: <ul class="list"> -....: <li>1</li> -....: <li>2</li> -....: <li>3</li> -....: </ul> -....: <ul class="list"> -....: <li>4</li> -....: <li>5</li> -....: <li>6</li> -....: </ul>""") ->>> xp = lambda x: sel.xpath(x).getall() +.. code-block:: pycon + + >>> from scrapy import Selector + >>> sel = Selector( + ... text=""" + ... <ul class="list"> + ... <li>1</li> + ... <li>2</li> + ... <li>3</li> + ... </ul> + ... <ul class="list"> + ... <li>4</li> + ... <li>5</li> + ... <li>6</li> + ... </ul>""" + ... ) + >>> xp = lambda x: sel.xpath(x).getall() This gets all first ``<li>`` elements under whatever it is its parent: ->>> xp("//li[1]") -['<li>1</li>', '<li>4</li>'] +.. code-block:: pycon + + >>> xp("//li[1]") + ['<li>1</li>', '<li>4</li>'] And this gets the first ``<li>`` element in the whole document: ->>> xp("(//li)[1]") -['<li>1</li>'] +.. code-block:: pycon + + >>> xp("(//li)[1]") + ['<li>1</li>'] This gets all first ``<li>`` elements under an ``<ul>`` parent: ->>> xp("//ul/li[1]") -['<li>1</li>', '<li>4</li>'] +.. code-block:: pycon + + >>> xp("//ul/li[1]") + ['<li>1</li>', '<li>4</li>'] And this gets the first ``<li>`` element under an ``<ul>`` parent in the whole document: ->>> xp("(//ul/li)[1]") -['<li>1</li>'] +.. code-block:: pycon + + >>> xp("(//ul/li)[1]") + ['<li>1</li>'] Using text nodes in a condition ------------------------------- @@ -585,32 +676,44 @@ a string function like ``contains()`` or ``starts-with()``, it results in the te Example: ->>> from scrapy import Selector ->>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>') +.. code-block:: pycon + + >>> from scrapy import Selector + >>> sel = Selector( + ... text='<a href="#">Click here to go to the <strong>Next Page</strong></a>' + ... ) Converting a *node-set* to string: ->>> sel.xpath('//a//text()').getall() # take a peek at the node-set -['Click here to go to the ', 'Next Page'] ->>> sel.xpath("string(//a[1]//text())").getall() # convert it to string -['Click here to go to the '] +.. code-block:: pycon + + >>> sel.xpath("//a//text()").getall() # take a peek at the node-set + ['Click here to go to the ', 'Next Page'] + >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string + ['Click here to go to the '] A *node* converted to a string, however, puts together the text of itself plus of all its descendants: ->>> sel.xpath("//a[1]").getall() # select the first node -['<a href="#">Click here to go to the <strong>Next Page</strong></a>'] ->>> sel.xpath("string(//a[1])").getall() # convert it to string -['Click here to go to the Next Page'] +.. code-block:: pycon + + >>> sel.xpath("//a[1]").getall() # select the first node + ['<a href="#">Click here to go to the <strong>Next Page</strong></a>'] + >>> sel.xpath("string(//a[1])").getall() # convert it to string + ['Click here to go to the Next Page'] So, using the ``.//text()`` node-set won't select anything in this case: ->>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall() -[] +.. code-block:: pycon + + >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall() + [] But using the ``.`` to mean the node, works: ->>> sel.xpath("//a[contains(., 'Next Page')]").getall() -['<a href="#">Click here to go to the <strong>Next Page</strong></a>'] +.. code-block:: pycon + + >>> sel.xpath("//a[contains(., 'Next Page')]").getall() + ['<a href="#">Click here to go to the <strong>Next Page</strong></a>'] .. _`XPath string function`: https://www.w3.org/TR/xpath/all/#section-String-Functions @@ -628,15 +731,19 @@ which are then substituted with values passed with the query. Here's an example to match an element based on its "id" attribute value, without hard-coding it (that was shown previously): ->>> # `$val` used in the expression, a `val` argument needs to be passed ->>> response.xpath('//div[@id=$val]/a/text()', val='images').get() -'Name: My image 1 ' +.. code-block:: pycon + + >>> # `$val` used in the expression, a `val` argument needs to be passed + >>> response.xpath("//div[@id=$val]/a/text()", val="images").get() + 'Name: My image 1 ' Here's another example, to find the "id" attribute of a ``<div>`` tag containing five ``<a>`` children (here we pass the value ``5`` as an integer): ->>> response.xpath('//div[count(a)=$cnt]/@id', cnt=5).get() -'images' +.. code-block:: pycon + + >>> response.xpath("//div[count(a)=$cnt]/@id", cnt=5).get() + 'images' All variable references must have a binding value when calling ``.xpath()`` (otherwise you'll get a ``ValueError: XPath error:`` exception). @@ -688,17 +795,21 @@ You can see several namespace declarations including a default Once in the shell we can try selecting all ``<link>`` objects and see that it doesn't work (because the Atom XML namespace is obfuscating those nodes): ->>> response.xpath("//link") -[] +.. code-block:: pycon + + >>> response.xpath("//link") + [] But once we call the :meth:`Selector.remove_namespaces` method, all nodes can be accessed directly by their names: ->>> response.selector.remove_namespaces() ->>> response.xpath("//link") -[<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>, - <Selector xpath='//link' data='<link rel="next" type="application/atom+'>, - ... +.. code-block:: pycon + + >>> response.selector.remove_namespaces() + >>> response.xpath("//link") + [<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>, + <Selector xpath='//link' data='<link rel="next" type="application/atom+'>, + ... If you wonder why the namespace removal procedure isn't always called by default instead of having to call it manually, this is because of two reasons, which, in order @@ -735,23 +846,25 @@ The ``test()`` function, for example, can prove quite useful when XPath's Example selecting links in list item with a "class" attribute ending with a digit: ->>> from scrapy import Selector ->>> doc = """ -... <div> -... <ul> -... <li class="item-0"><a href="link1.html">first item</a></li> -... <li class="item-1"><a href="link2.html">second item</a></li> -... <li class="item-inactive"><a href="link3.html">third item</a></li> -... <li class="item-1"><a href="link4.html">fourth item</a></li> -... <li class="item-0"><a href="link5.html">fifth item</a></li> -... </ul> -... </div> -... """ ->>> sel = Selector(text=doc, type="html") ->>> sel.xpath('//li//@href').getall() -['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] ->>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() -['link1.html', 'link2.html', 'link4.html', 'link5.html'] +.. code-block:: pycon + + >>> from scrapy import Selector + >>> doc = """ + ... <div> + ... <ul> + ... <li class="item-0"><a href="link1.html">first item</a></li> + ... <li class="item-1"><a href="link2.html">second item</a></li> + ... <li class="item-inactive"><a href="link3.html">third item</a></li> + ... <li class="item-1"><a href="link4.html">fourth item</a></li> + ... <li class="item-0"><a href="link5.html">fifth item</a></li> + ... </ul> + ... </div> + ... """ + >>> sel = Selector(text=doc, type="html") + >>> sel.xpath("//li//@href").getall() + ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html'] + >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall() + ['link1.html', 'link2.html', 'link4.html', 'link5.html'] .. warning:: C library ``libxslt`` doesn't natively support EXSLT regular expressions so `lxml`_'s implementation uses hooks to Python's ``re`` module. @@ -765,7 +878,9 @@ These can be handy for excluding parts of a document tree before extracting text elements for example. Example extracting microdata (sample content taken from https://schema.org/Product) -with groups of itemscopes and corresponding itemprops:: +with groups of itemscopes and corresponding itemprops: + +.. code-block:: pycon >>> doc = """ ... <div itemscope itemtype="http://schema.org/Product"> @@ -776,19 +891,15 @@ with groups of itemscopes and corresponding itemprops:: ... Rated <span itemprop="ratingValue">3.5</span>/5 ... based on <span itemprop="reviewCount">11</span> customer reviews ... </div> - ... ... <div itemprop="offers" itemscope itemtype="http://schema.org/Offer"> ... <span itemprop="price">$55.00</span> ... <link itemprop="availability" href="http://schema.org/InStock" />In stock ... </div> - ... ... Product description: ... <span itemprop="description">0.7 cubic feet countertop microwave. ... Has six preset cooking categories and convenience features like ... Add-A-Minute and Child Lock.</span> - ... ... Customer reviews: - ... ... <div itemprop="review" itemscope itemtype="http://schema.org/Review"> ... <span itemprop="name">Not a happy camper</span> - ... by <span itemprop="author">Ellie</span>, @@ -801,7 +912,6 @@ with groups of itemscopes and corresponding itemprops:: ... <span itemprop="description">The lamp burned out and now I have to replace ... it. </span> ... </div> - ... ... <div itemprop="review" itemscope itemtype="http://schema.org/Review"> ... <span itemprop="name">Value purchase</span> - ... by <span itemprop="author">Lucas</span>, @@ -818,13 +928,16 @@ with groups of itemscopes and corresponding itemprops:: ... </div> ... """ >>> sel = Selector(text=doc, type="html") - >>> for scope in sel.xpath('//div[@itemscope]'): - ... print("current scope:", scope.xpath('@itemtype').getall()) - ... props = scope.xpath(''' + >>> for scope in sel.xpath("//div[@itemscope]"): + ... print("current scope:", scope.xpath("@itemtype").getall()) + ... props = scope.xpath( + ... """ ... set:difference(./descendant::*/@itemprop, - ... .//*[@itemscope]/*/@itemprop)''') + ... .//*[@itemscope]/*/@itemprop)""" + ... ) ... print(f" properties: {props.getall()}") ... print("") + ... current scope: ['http://schema.org/Product'] properties: ['name', 'aggregateRating', 'offers', 'description', 'review', 'review'] @@ -876,13 +989,15 @@ For the following HTML:: You can use it like this: ->>> response.xpath('//p[has-class("foo")]') -[<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>, - <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>] ->>> response.xpath('//p[has-class("foo", "bar-baz")]') -[<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>] ->>> response.xpath('//p[has-class("foo", "bar")]') -[] +.. code-block:: pycon + + >>> response.xpath('//p[has-class("foo")]') + [<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>, + <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>] + >>> response.xpath('//p[has-class("foo", "bar-baz")]') + [<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>] + >>> response.xpath('//p[has-class("foo", "bar")]') + [] So XPath ``//p[has-class("foo", "bar-baz")]`` is roughly equivalent to CSS ``p.foo.bar-baz``. Please note, that it is slower in most of the cases, diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index 69a90f620..4898843e4 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -191,44 +191,46 @@ all start with the ``[s]`` prefix):: After that, we can start playing with the objects: ->>> response.xpath('//title/text()').get() -'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework' +.. code-block:: pycon ->>> fetch("https://old.reddit.com/") + >>> response.xpath("//title/text()").get() + 'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework' ->>> response.xpath('//title/text()').get() -'reddit: the front page of the internet' + >>> fetch("https://old.reddit.com/") ->>> request = request.replace(method="POST") + >>> response.xpath("//title/text()").get() + 'reddit: the front page of the internet' ->>> fetch(request) + >>> request = request.replace(method="POST") ->>> response.status -404 + >>> fetch(request) ->>> from pprint import pprint + >>> response.status + 404 ->>> pprint(response.headers) -{'Accept-Ranges': ['bytes'], - 'Cache-Control': ['max-age=0, must-revalidate'], - 'Content-Type': ['text/html; charset=UTF-8'], - 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'], - 'Server': ['snooserv'], - 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure', - 'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure', - 'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure', - 'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'], - 'Vary': ['accept-encoding'], - 'Via': ['1.1 varnish'], - 'X-Cache': ['MISS'], - 'X-Cache-Hits': ['0'], - 'X-Content-Type-Options': ['nosniff'], - 'X-Frame-Options': ['SAMEORIGIN'], - 'X-Moose': ['majestic'], - 'X-Served-By': ['cache-cdg8730-CDG'], - 'X-Timer': ['S1481214079.394283,VS0,VE159'], - 'X-Ua-Compatible': ['IE=edge'], - 'X-Xss-Protection': ['1; mode=block']} + >>> from pprint import pprint + + >>> pprint(response.headers) + {'Accept-Ranges': ['bytes'], + 'Cache-Control': ['max-age=0, must-revalidate'], + 'Content-Type': ['text/html; charset=UTF-8'], + 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'], + 'Server': ['snooserv'], + 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure', + 'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure', + 'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure', + 'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'], + 'Vary': ['accept-encoding'], + 'Via': ['1.1 varnish'], + 'X-Cache': ['MISS'], + 'X-Cache-Hits': ['0'], + 'X-Content-Type-Options': ['nosniff'], + 'X-Frame-Options': ['SAMEORIGIN'], + 'X-Moose': ['majestic'], + 'X-Served-By': ['cache-cdg8730-CDG'], + 'X-Timer': ['S1481214079.394283,VS0,VE159'], + 'X-Ua-Compatible': ['IE=edge'], + 'X-Xss-Protection': ['1; mode=block']} .. _topics-shell-inspect-response: @@ -279,14 +281,18 @@ When you run the spider, you will get something similar to this:: Then, you can check if the extraction code is working: ->>> response.xpath('//h1[@class="fn"]') -[] +.. code-block:: pycon + + >>> response.xpath('//h1[@class="fn"]') + [] Nope, it doesn't. So you can open the response in your web browser and see if it's the response you were expecting: ->>> view(response) -True +.. code-block:: pycon + + >>> view(response) + True Finally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume the crawling:: diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst index f8b0c4d2a..be8ecb7a5 100644 --- a/docs/topics/stats.rst +++ b/docs/topics/stats.rst @@ -68,13 +68,17 @@ Set stat value only if lower than previous: Get stat value: ->>> stats.get_value('custom_count') -1 +.. code-block:: pycon + + >>> stats.get_value("custom_count") + 1 Get all stats: ->>> stats.get_stats() -{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} +.. code-block:: pycon + + >>> stats.get_stats() + {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} Available Stats Collectors ==========================