From beea9267a17354101cf3b4611c1e595ed2a8b45c Mon Sep 17 00:00:00 2001 From: bagratte Date: Sat, 18 Apr 2015 19:48:25 +0400 Subject: [PATCH] minor corrections in documentation. --- .gitignore | 3 +++ docs/intro/tutorial.rst | 4 +-- docs/topics/commands.rst | 12 ++++----- docs/topics/loaders.rst | 12 ++++----- docs/topics/selectors.rst | 52 +++++++++++++++++++-------------------- docs/topics/spiders.rst | 10 ++++---- 6 files changed, 48 insertions(+), 45 deletions(-) diff --git a/.gitignore b/.gitignore index 97535ccaf..dc6e8bb67 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,6 @@ venv build dist .idea + +# Windows +Thumbs.db \ No newline at end of file diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index b24366eb1..a3bf7025a 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -108,7 +108,7 @@ define the three main mandatory attributes: listed here. The subsequent URLs will be generated successively from data contained in the start URLs. -* :meth:`~scrapy.spider.Spider.parse` a method of the spider, which will +* :meth:`~scrapy.spider.Spider.parse`: a method of the spider, which will be called with the downloaded :class:`~scrapy.http.Response` object of each start URL. The response is passed to the method as the first and only argument. @@ -248,7 +248,7 @@ To start a shell, you must go to the project's top level directory and run:: .. note:: - Remember to always enclose urls with quotes in running Scrapy shell from + Remember to always enclose urls in quotes when running Scrapy shell from command-line, otherwise urls containing arguments (ie. ``&`` character) will not work. diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index 478746d3c..5a223f583 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -80,8 +80,8 @@ some usage help and the available commands:: fetch Fetch a URL using the Scrapy downloader [...] -The first line will print the currently active project, if you're inside a -Scrapy project. In this, it was run from outside a project. If run from inside +The first line will print the currently active project if you're inside a +Scrapy project. In this example it was run from outside a project. If run from inside a project it would have printed something like this:: Scrapy X.Y - project: myproject @@ -135,7 +135,7 @@ Available tool commands ======================= This section contains a list of the available built-in commands with a -description and some usage examples. Remember you can always get more info +description and some usage examples. Remember, you can always get more info about each command by running:: scrapy -h @@ -196,7 +196,7 @@ genspider Create a new spider in the current project. -This is just a convenient shortcut command for creating spiders based on +This is just a convenience shortcut command for creating spiders based on pre-defined templates, but certainly not the only way to create spiders. You can just create the spider source code files yourself, instead of using this command. @@ -298,7 +298,7 @@ edit Edit the given spider using the editor defined in the :setting:`EDITOR` setting. -This command is provided only as a convenient shortcut for the most common +This command is provided only as a convenience shortcut for the most common case, the developer is of course free to choose any tool or IDE to write and debug his spiders. @@ -318,7 +318,7 @@ Downloads the given URL using the Scrapy downloader and writes the contents to standard output. The interesting thing about this command is that it fetches the page how the -spider would download it. For example, if the spider has an ``USER_AGENT`` +spider would download it. For example, if the spider has a ``USER_AGENT`` attribute which overrides the User Agent, it will use that one. So this command can be used to "see" how your spider would fetch a certain page. diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index b2c8a018d..603d28f78 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -9,7 +9,7 @@ Item Loaders Item Loaders provide a convenient mechanism for populating scraped :ref:`Items `. Even though Items can be populated using their own -dictionary-like API, the Item Loaders provide a much more convenient API for +dictionary-like API, Item Loaders provide a much more convenient API for populating them from a scraping process, by automating some common tasks like parsing the raw extracted data before assigning it. @@ -25,7 +25,7 @@ Using Item Loaders to populate items ==================================== To use an Item Loader, you must first instantiate it. You can either -instantiate it with an dict-like object (e.g. Item or dict) or without one, in +instantiate it with a dict-like object (e.g. Item or dict) or without one, in which case an Item is automatically instantiated in the Item Loader constructor using the Item class specified in the :attr:`ItemLoader.default_item_class` attribute. @@ -67,7 +67,7 @@ and finally the ``last_update`` field is populated directly with a literal value (``today``) using a different method: :meth:`~ItemLoader.add_value`. Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is -called which actually populates and returns the item populated with the data +called which actually returns the item populated with the data previously extracted and collected with the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, and :meth:`~ItemLoader.add_value` calls. @@ -565,8 +565,8 @@ Here is a list of all built-in processors: .. class:: Identity The simplest processor, which doesn't do anything. It returns the original - values unchanged. It doesn't receive any constructor arguments nor accepts - Loader contexts. + values unchanged. It doesn't receive any constructor arguments, nor does it + accept Loader contexts. Example:: @@ -579,7 +579,7 @@ Here is a list of all built-in processors: Returns the first non-null/non-empty value from the values received, so it's typically used as an output processor to single-valued fields. - It doesn't receive any constructor arguments, nor accept Loader contexts. + It doesn't receive any constructor arguments, nor does it accept Loader contexts. Example:: diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index a84ac9c99..ccac9ec0b 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -13,9 +13,9 @@ achieve this: HTML code and also deals with bad markup reasonably well, but it has one drawback: it's slow. - * `lxml`_ is a XML parsing library (which also parses HTML) with a pythonic - API based on `ElementTree`_ (which is not part of the Python standard - library). + * `lxml`_ is an XML parsing library (which also parses HTML) with a pythonic + API based on `ElementTree`_. (lxml is not part of the Python standard + library.) Scrapy comes with its own mechanism for extracting data. They're called selectors because they "select" certain parts of the HTML document specified @@ -72,7 +72,7 @@ Constructing from response:: >>> Selector(response=response).xpath('//span/text()').extract() [u'good'] -For convenience, response objects exposes a selector on `.selector` attribute, +For convenience, response objects expose a selector on `.selector` attribute, it's totally OK to use this shortcut when possible:: >>> response.selector.xpath('//span/text()').extract() @@ -114,17 +114,17 @@ page, let's construct an XPath for selecting the text inside the title tag:: >>> response.selector.xpath('//title/text()') [] -Querying responses using XPath and CSS is so common that responses includes two -convenient shortcuts: ``response.xpath()`` and ``response.css()``:: +Querying responses using XPath and CSS is so common that responses include two +convenience shortcuts: ``response.xpath()`` and ``response.css()``:: >>> response.xpath('//title/text()') [] >>> response.css('title::text') [] -As you can see, ``.xpath()`` and ``.css()`` methods returns an +As you can see, ``.xpath()`` and ``.css()`` methods return a :class:`~scrapy.selector.SelectorList` instance, which is a list of new -selectors. This API can be used quickly for selecting nested data:: +selectors. This API can be used for quickly selecting nested data:: >>> response.css('img').xpath('@src').extract() [u'image1_thumb.jpg', @@ -196,7 +196,7 @@ Now we're going to get the base URL and some image links:: Nesting selectors ----------------- -The selection methods (``.xpath()`` or ``.css()``) returns a list of selectors +The selection methods (``.xpath()`` or ``.css()``) return a list of selectors of the same type, so you can call the selection methods for those selectors too. Here's an example:: @@ -221,12 +221,12 @@ too. Here's an example:: Using selectors with regular expressions ---------------------------------------- -:class:`~scrapy.selector.Selector` also have a ``.re()`` method for extracting +:class:`~scrapy.selector.Selector` also has a ``.re()`` method for extracting data using regular expressions. However, unlike using ``.xpath()`` or -``.css()`` methods, ``.re()`` method returns a list of unicode strings. So you +``.css()`` methods, ``.re()`` returns a list of unicode strings. So you can't construct nested ``.re()`` calls. -Here's an example used to extract images names from the :ref:`HTML code +Here's an example used to extract image names from the :ref:`HTML code ` above:: >>> response.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)') @@ -295,7 +295,7 @@ set \http://exslt.org/sets `set manipulation`_ Regular expressions ~~~~~~~~~~~~~~~~~~~ -The ``test()`` function for example can prove quite useful when XPath's +The ``test()`` function, for example, can prove quite useful when XPath's ``starts-with()`` or ``contains()`` are not sufficient. Example selecting links in list item with a "class" attribute ending with a digit:: @@ -440,7 +440,7 @@ you may want to take a look first at this `XPath tutorial`_. Using text nodes in a condition ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When you need to use the text content as argument to a `XPath string function`_, +When you need to use the text content as argument to an `XPath string function`_, avoid using ``.//text()`` and use just ``.`` instead. This is because the expression ``.//text()`` yields a collection of text elements -- a *node-set*. @@ -478,7 +478,7 @@ But using the ``.`` to mean the node, works:: .. _`XPath string function`: http://www.w3.org/TR/xpath/#section-String-Functions -Beware the difference between //node[1] and (//node)[1] +Beware of the difference between //node[1] and (//node)[1] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``//node[1]`` selects all the nodes occurring first under their respective parents. @@ -559,7 +559,7 @@ Built-in Selectors reference An instance of :class:`Selector` is a wrapper over response to select certain parts of its content. - ``response`` is a :class:`~scrapy.http.HtmlResponse` or + ``response`` is an :class:`~scrapy.http.HtmlResponse` or an :class:`~scrapy.http.XmlResponse` object that will be used for selecting and extracting data. @@ -593,7 +593,7 @@ Built-in Selectors reference .. note:: - For convenience this method can be called as ``response.xpath()`` + For convenience, this method can be called as ``response.xpath()`` .. method:: css(query) @@ -644,7 +644,7 @@ SelectorList objects .. class:: SelectorList - The :class:`SelectorList` class is subclass of the builtin ``list`` + The :class:`SelectorList` class is a subclass of the builtin ``list`` class, which provides a few additional methods. .. method:: xpath(query) @@ -680,17 +680,17 @@ Selector examples on HTML response ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here's a couple of :class:`Selector` examples to illustrate several concepts. -In all cases, we assume there is already an :class:`Selector` instantiated with +In all cases, we assume there is already a :class:`Selector` instantiated with a :class:`~scrapy.http.HtmlResponse` object like this:: sel = Selector(html_response) -1. Select all ``

`` elements from a HTML response body, returning a list of +1. Select all ``

`` elements from an HTML response body, returning a list of :class:`Selector` objects (ie. a :class:`SelectorList` object):: sel.xpath("//h1") -2. Extract the text of all ``

`` elements from a HTML response body, +2. Extract the text of all ``

`` elements from an HTML response body, returning a list of unicode strings:: sel.xpath("//h1").extract() # this includes the h1 tag @@ -705,12 +705,12 @@ Selector examples on XML response ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here's a couple of examples to illustrate several concepts. In both cases we -assume there is already an :class:`Selector` instantiated with a +assume there is already a :class:`Selector` instantiated with an :class:`~scrapy.http.XmlResponse` object like this:: sel = Selector(xml_response) -1. Select all ```` elements from a XML response body, returning a list +1. Select all ```` elements from an XML response body, returning a list of :class:`Selector` objects (ie. a :class:`SelectorList` object):: sel.xpath("//product") @@ -752,12 +752,12 @@ nodes can be accessed directly by their names::