From 210a0a6fe1331ae9ca770b7b482ee2a74a8f01bd Mon Sep 17 00:00:00 2001 From: Alexey Bezhan Date: Thu, 27 Feb 2014 18:02:22 +0000 Subject: [PATCH] Fix some typos, whitespace and small errors in docs --- docs/intro/tutorial.rst | 28 ++++++++++++++-------------- docs/topics/commands.rst | 22 +++++++++++----------- docs/topics/items.rst | 8 ++++---- docs/topics/loaders.rst | 14 +++++++------- docs/topics/spiders.rst | 38 +++++++++++++++++++------------------- 5 files changed, 55 insertions(+), 55 deletions(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index 9a5f2e9b0..b48898f0b 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -65,10 +65,10 @@ Defining our Item ================= `Items` are containers that will be loaded with the scraped data; they work -like simple python dicts but provide additional protecting against populating +like simple python dicts but provide additional protection against populating undeclared fields, to prevent typos. -They are declared by creating an :class:`scrapy.item.Item` class and defining +They are declared by creating a :class:`scrapy.item.Item` class and defining its attributes as :class:`scrapy.item.Field` objects, like you will in an ORM (don't worry if you're not familiar with ORMs, you will see that this is an easy task). @@ -76,7 +76,7 @@ easy task). We begin by modeling the item that we will use to hold the sites data obtained from dmoz.org, as we want to capture the name, url and description of the sites, we define fields for each of these three attributes. To do that, we edit -items.py, found in the ``tutorial`` directory. Our Item class looks like this:: +``items.py``, found in the ``tutorial`` directory. Our Item class looks like this:: from scrapy.item import Item, Field @@ -86,7 +86,7 @@ items.py, found in the ``tutorial`` directory. Our Item class looks like this:: desc = Field() This may seem complicated at first, but defining the item allows you to use other handy -components of Scrapy that need to know how your item looks like. +components of Scrapy that need to know how your item looks. Our first Spider ================ @@ -97,8 +97,8 @@ of domains). They define an initial list of URLs to download, how to follow links, and how to parse the contents of those pages to extract :ref:`items `. -To create a Spider, you must subclass :class:`scrapy.spider.Spider`, and -define the three main, mandatory, attributes: +To create a Spider, you must subclass :class:`scrapy.spider.Spider` and +define the three main mandatory attributes: * :attr:`~scrapy.spider.Spider.name`: identifies the Spider. It must be unique, that is, you can't set the same name for different Spiders. @@ -162,7 +162,7 @@ will get an output similar to this:: Pay attention to the lines containing ``[dmoz]``, which corresponds to our spider. You can see a log line for each URL defined in ``start_urls``. Because these URLs are the starting ones, they have no referrers, which is shown at the -end of the log line, where it says ``(referer: )``. +end of the log line, where it says ``(referer: None)``. But more interesting, as our ``parse`` method instructs, two files have been created: *Books* and *Resources*, with the content of both URLs. @@ -210,15 +210,15 @@ XPath expressions are indeed much more powerful. To learn more about XPath we recommend `this XPath tutorial `_. For working with XPaths, Scrapy provides a :class:`~scrapy.selector.Selector` -class, it is instantiated with a :class:`~scrapy.http.HtmlResponse` or +class, which is instantiated with a :class:`~scrapy.http.HtmlResponse` or :class:`~scrapy.http.XmlResponse` object as first argument. You can see selectors as objects that represent nodes in the document -structure. So, the first instantiated selectors are associated to the root +structure. So, the first instantiated selectors are associated with the root node, or the entire document. Selectors have four basic methods (click on the method to see the complete API -documentation). +documentation): * :meth:`~scrapy.selector.Selector.xpath`: returns a list of selectors, each of them representing the nodes selected by the xpath expression given as @@ -275,7 +275,7 @@ After the shell loads, you will have the response fetched in a local ``response`` variable, so if you type ``response.body`` you will see the body of the response, or you can type ``response.headers`` to see its headers. -The shell also pre-instantiate a selector for this response in variable ``sel``, +The shell also pre-instantiates a selector for this response in variable ``sel``, the selector automatically chooses the best parsing rules (XML vs HTML) based on response's type. @@ -327,7 +327,7 @@ And the sites links:: sel.xpath('//ul/li/a/@href').extract() -As we said before, each ``.xpath()`` call returns a list of selectors, so we can +As we've said before, each ``.xpath()`` call returns a list of selectors, so we can concatenate further ``.xpath()`` calls to dig deeper into a node. We are going to use that property here, so:: @@ -418,7 +418,7 @@ scraped so far, the final code for our Spider would be like this:: .. note:: You can find a fully-functional variant of this spider in the dirbot_ project available at https://github.com/scrapy/dirbot -Now doing a crawl on the dmoz.org domain yields ``DmozItem``'s:: +Now doing a crawl on the dmoz.org domain yields ``DmozItem`` objects:: [dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> {'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n], @@ -445,7 +445,7 @@ However, if you want to perform more complex things with the scraped items, you can write an :ref:`Item Pipeline `. As with Items, a placeholder file for Item Pipelines has been set up for you when the project is created, in ``tutorial/pipelines.py``. Though you don't need to implement any item -pipeline if you just want to store the scraped items. +pipelines if you just want to store the scraped items. Next steps ========== diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index 933905a4d..36439fe2d 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -7,8 +7,8 @@ Command line tool .. versionadded:: 0.10 Scrapy is controlled through the ``scrapy`` command-line tool, to be referred -here as the "Scrapy tool" to differentiate it from their sub-commands which we -just call "commands", or "Scrapy commands". +here as the "Scrapy tool" to differentiate it from the sub-commands, which we +just call "commands" or "Scrapy commands". The Scrapy tool provides several commands, for multiple purposes, and each one accepts a different set of arguments and options. @@ -214,7 +214,7 @@ crawl * Syntax: ``scrapy crawl `` * Requires project: *yes* -Start crawling a spider. +Start crawling using a spider. Usage examples:: @@ -297,13 +297,13 @@ Downloads the given URL using the Scrapy downloader and writes the contents to standard output. The interesting thing about this command is that it fetches the page how the -the spider would download it. For example, if the spider has an ``USER_AGENT`` +spider would download it. For example, if the spider has an ``USER_AGENT`` attribute which overrides the User Agent, it will use that one. -So this command can be used to "see" how your spider would fetch certain page. +So this command can be used to "see" how your spider would fetch a certain page. If used outside a project, no particular per-spider behaviour would be applied -and it will just use the default Scrapy downloder settings. +and it will just use the default Scrapy downloader settings. Usage examples:: @@ -346,7 +346,7 @@ shell * Syntax: ``scrapy shell [url]`` * Requires project: *no* -Starts the Scrapy shell for the given URL (if given) or empty if not URL is +Starts the Scrapy shell for the given URL (if given) or empty if no URL is given. See :ref:`topics-shell` for more info. Usage example:: @@ -362,7 +362,7 @@ parse * Syntax: ``scrapy parse [options]`` * Requires project: *yes* -Fetches the given URL and parses with the spider that handles it, using the +Fetches the given URL and parses it with the spider that handles it, using the method passed with the ``--callback`` option, or ``parse`` if not given. Supported options: @@ -371,7 +371,7 @@ Supported options: response * ``--rules`` or ``-r``: use :class:`~scrapy.contrib.spiders.CrawlSpider` - rules to discover the callback (ie. spider method) to use for parsing the + rules to discover the callback (i.e. spider method) to use for parsing the response * ``--noitems``: don't show scraped items @@ -467,7 +467,7 @@ bench * Syntax: ``scrapy bench`` * Requires project: *no* -Run quick benchmark test. :ref:`benchmarking`. +Run a quick benchmark test. :ref:`benchmarking`. Custom project commands ======================= @@ -484,7 +484,7 @@ COMMANDS_MODULE Default: ``''`` (empty string) -A module to use for looking custom Scrapy commands. This is used to add custom +A module to use for looking up custom Scrapy commands. This is used to add custom commands for your Scrapy project. Example:: diff --git a/docs/topics/items.rst b/docs/topics/items.rst index 124d6996a..bf7ca74d1 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -47,11 +47,11 @@ Item Fields :class:`Field` objects are used to specify metadata for each field. For example, the serializer function for the ``last_updated`` field illustrated in -the example above. +the example above. You can specify any kind of metadata for each field. There is no restriction on the values accepted by :class:`Field` objects. For this same -reason, there isn't a reference list of all available metadata keys. Each key +reason, there is no reference list of all available metadata keys. Each key defined in :class:`Field` objects could be used by a different components, and only those components know about it. You can also define and use any other :class:`Field` key in your project too, for your own needs. The main goal of @@ -62,9 +62,9 @@ documentation to see which metadata keys are used by each component. It's important to note that the :class:`Field` objects used to declare the item do not stay assigned as class attributes. Instead, they can be accessed through -the :attr:`Item.fields` attribute. +the :attr:`Item.fields` attribute. -And that's all you need to know about declaring items. +And that's all you need to know about declaring items. Working with Items ================== diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index 5c55cb940..be20ea10a 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -227,7 +227,7 @@ value and extracts a length from it:: return parsed_length By accepting a ``loader_context`` argument the function is explicitly telling -the Item Loader that is able to receive an Item Loader context, so the Item +the Item Loader that it's able to receive an Item Loader context, so the Item Loader passes the currently active context when calling it, and the processor function (``parse_length`` in this case) can thus use them. @@ -245,7 +245,7 @@ There are several ways to modify Item Loader context values: loader = ItemLoader(product, unit='cm') 3. On Item Loader declaration, for those input/output processors that support - instatiating them with a Item Loader context. :class:`~processor.MapCompose` is one of + instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of them:: class ProductLoader(ItemLoader): @@ -486,7 +486,7 @@ Reusing and extending Item Loaders ================================== As your project grows bigger and acquires more and more spiders, maintenance -becomes a fundamental problem, specially when you have to deal with many +becomes a fundamental problem, especially when you have to deal with many different parsing rules for each spider, having a lot of exceptions, but also wanting to reuse the common processors. @@ -497,7 +497,7 @@ support traditional Python class inheritance for dealing with differences of specific spiders (or groups of spiders). Suppose, for example, that some particular site encloses their product names in -three dashes (ie. ``---Plasma TV---``) and you don't want to end up scraping +three dashes (e.g. ``---Plasma TV---``) and you don't want to end up scraping those dashes in the final product names. Here's how you can remove those dashes by reusing and extending the default @@ -567,7 +567,7 @@ Here is a list of all built-in processors: .. class:: TakeFirst - Return the first non-null/non-empty value from the values received, + Returns the first non-null/non-empty value from the values received, so it's typically used as an output processor to single-valued fields. It doesn't receive any constructor arguments, nor accept Loader contexts. @@ -604,8 +604,8 @@ Here is a list of all built-in processors: function, and so on, until the last function returns the output value of this processor. - By default, stop process on None value. This behaviour can be changed by - passing keyword argument stop_on_none=False. + By default, stop process on ``None`` value. This behaviour can be changed by + passing keyword argument ``stop_on_none=False``. Example:: diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index f9cf28efb..399aa7e6a 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -4,11 +4,11 @@ Spiders ======= -Spiders are classes which define how a certain site (or group of sites) will be -scraped, including how to perform the crawl (ie. follow links) and how to -extract structured data from their pages (ie. scraping items). In other words, +Spiders are classes which define how a certain site (or a group of sites) will be +scraped, including how to perform the crawl (i.e. follow links) and how to +extract structured data from their pages (i.e. scraping items). In other words, Spiders are the place where you define the custom behaviour for crawling and -parsing pages for a particular site (or, in some cases, group of sites). +parsing pages for a particular site (or, in some cases, a group of sites). For spiders, the scraping cycle goes through something like this: @@ -105,7 +105,7 @@ Spider method ``parse`` for each of the resulting responses. .. attribute:: name - + A string which defines the name for this spider. The spider name is how the spider is located (and instantiated) by Scrapy, so it must be unique. However, nothing prevents you from instantiating more than one @@ -113,7 +113,7 @@ Spider and it's required. If the spider scrapes a single domain, a common practice is to name the - spider after the domain, or without the `TLD`_. So, for example, a + spider after the domain, with or without the `TLD`_. So, for example, a spider that crawls ``mywebsite.com`` would often be called ``mywebsite``. @@ -134,8 +134,8 @@ Spider .. method:: start_requests() This method must return an iterable with the first Requests to crawl for - this spider. - + this spider. + This is the method called by Scrapy when the spider is opened for scraping when no particular URLs are specified. If particular URLs are specified, the :meth:`make_requests_from_url` is used instead to create @@ -150,7 +150,7 @@ Spider a POST request, you could do:: def start_requests(self): - return [FormRequest("http://www.example.com/login", + return [FormRequest("http://www.example.com/login", formdata={'user': 'john', 'pass': 'secret'}, callback=self.logged_in)] @@ -214,7 +214,7 @@ Let's see an example:: def parse(self, response): self.log('A response from %s just arrived!' % response.url) -Another example returning multiples Requests and Items from a single callback:: +Another example returning multiple Requests and Items from a single callback:: from scrapy.selector import Selector from scrapy.spider import Spider @@ -267,10 +267,10 @@ CrawlSpider .. method:: parse_start_url(response) This method is called for the start_urls responses. It allows to parse - the initial responses and must return either a - :class:`~scrapy.item.Item` object, a :class:`~scrapy.http.Request` + the initial responses and must return either a + :class:`~scrapy.item.Item` object, a :class:`~scrapy.http.Request` object, or an iterable containing any of them. - + Crawling rules ~~~~~~~~~~~~~~ @@ -278,7 +278,7 @@ Crawling rules ``link_extractor`` is a :ref:`Link Extractor ` object which defines how links will be extracted from each crawled page. - + ``callback`` is a callable or a string (in which case a method from the spider object with that name will be used) to be called for each link extracted with the specified link_extractor. This callback receives a response as its first @@ -291,7 +291,7 @@ Crawling rules the crawl spider will no longer work. ``cb_kwargs`` is a dict containing the keyword arguments to be passed to the - callback function + callback function. ``follow`` is a boolean which specifies if links should be followed from each response extracted with this rule. If ``callback`` is None ``follow`` defaults @@ -300,7 +300,7 @@ Crawling rules ``process_links`` is a callable, or a string (in which case a method from the spider object with that name will be used) which will be called for each list of links extracted from each response using the specified ``link_extractor``. - This is mainly used for filtering purposes. + This is mainly used for filtering purposes. ``process_request`` is a callable, or a string (in which case a method from the spider object with that name will be used) which will be called with @@ -321,7 +321,7 @@ Let's now take a look at an example CrawlSpider with rules:: name = 'example.com' allowed_domains = ['example.com'] start_urls = ['http://www.example.com'] - + rules = ( # Extract links matching 'category.php' (but not matching 'subsection.php') # and follow links from them (since no callback means follow=True by default). @@ -360,7 +360,7 @@ XMLFeedSpider iterator may be useful when parsing XML with bad markup. To set the iterator and the tag name, you must define the following class - attributes: + attributes: .. attribute:: iterator @@ -396,7 +396,7 @@ XMLFeedSpider attribute. Example:: - + class YourSpider(XMLFeedSpider): namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]