diff --git a/AUTHORS b/AUTHORS index bb82f6aef..249191f5e 100644 --- a/AUTHORS +++ b/AUTHORS @@ -23,4 +23,4 @@ Here is the list of the primary authors & contributors: * Patrick Mezard * Rolando Espinoza * Ping Yin - + * Lucian Ursu diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 4a24b7db8..569e680c5 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -4,8 +4,8 @@ Installation guide ================== -This document describes how to install Scrapy in Linux, Windows and Mac OS X -systems and it consists on the following 3 steps: +This document describes how to install Scrapy on Linux, Windows and Mac OS X +systems and it consists of the following 3 steps: * :ref:`intro-install-step1` * :ref:`intro-install-step2` @@ -16,7 +16,7 @@ systems and it consists on the following 3 steps: Requirements ============ -* `Python`_ 2.5 or 2.6 (3.x is not yet supported) +* `Python`_ 2.5, 2.6, 2.7 (3.x is not yet supported) * `Twisted`_ 2.5.0, 8.0 or above (Windows users: you'll need to install `Zope.Interface`_ and maybe `pywin32`_ because of `this Twisted bug`_) @@ -40,7 +40,7 @@ Optional: Step 1. Install Python ====================== -Scrapy works with Python 2.5 or 2.6, you can get it at http://www.python.org/download/ +Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/ .. highlight:: sh @@ -55,7 +55,7 @@ platform and operating system you use. Ubuntu/Debian ------------- -If you're running Ubuntu/Debian Linux run the following command as root:: +If you're running Ubuntu/Debian Linux, run the following command as root:: apt-get install python-twisted python-libxml2 @@ -66,7 +66,7 @@ To install optional libraries:: Arch Linux ---------- -If you are running Arch Linux run the following command as root:: +If you are running Arch Linux, run the following command as root:: pacman -S twisted libxml2 @@ -146,7 +146,7 @@ Installing an official release Download Scrapy from the `Download page`_. Scrapy is distributed in two ways: a source code tarball (for Unix and Mac OS X systems) and a Windows installer -(for Windows). If you downloaded the tarball you can install it as any Python +(for Windows). If you downloaded the tarball, you can install it as any Python package using ``setup.py``:: tar zxf scrapy-X.X.X.tar.gz diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 3f7d336f9..08c1de0b8 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -4,7 +4,7 @@ Scrapy at a glance ================== -Scrapy a is an application framework for crawling web sites and extracting +Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. @@ -49,15 +49,15 @@ If we take a look at that page content we'll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where ``NUMBER`` is an integer. We'll use that to construct the regular expression for the links to follow: ``/tor/\d+``. -For extracting data we'll use `XPath`_ to select the part of the document where -the data is to be extracted. Let's take one of those torrent pages: +To extracting data, we'll use `XPath`_ to select the part of the document where +the data is to be extracted from. Let's take one of those torrent pages: http://www.mininova.org/tor/2657665 .. _XPath: http://www.w3.org/TR/xpath And look at the page HTML source to construct the XPath to select the data we -want to extract which is: torrent name, description and size. +want which is: torrent name, description and size. .. highlight:: html @@ -144,7 +144,7 @@ Finally, here's the spider code:: return torrent -For brevity sake, we intentionally left out the import statements and the +For brevity's sake, we intentionally left out the import statements and the Torrent class definition (which is included some paragraphs above). Write a pipeline to store the items extracted diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index c62f72c92..27039abff 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -4,13 +4,13 @@ Scrapy Tutorial =============== -In this tutorial, we'll assume that Scrapy is already installed in your system. -If that's not the case see :ref:`intro-install`. +In this tutorial, we'll assume that Scrapy is already installed on your system. +If that's not the case, see :ref:`intro-install`. We are going to use `Open directory project (dmoz) `_ as our example domain to scrape. -This tutorial will walk you through through these tasks: +This tutorial will walk you through these tasks: 1. Creating a new Scrapy project 2. Defining the Items you will extract @@ -33,7 +33,7 @@ for non-programmers`_. Creating a project ================== -Before start scraping, you will have set up a new Scrapy project. Enter a +Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you'd like to store your code and then run:: scrapy startproject dmoz @@ -64,19 +64,19 @@ These are basically: Defining our Item ================= -`Items` are containers that will be loaded with the scraped data, they work +`Items` are containers that will be loaded with the scraped data; they work like simple python dicts but they offer some additional features like providing default values. They are declared by creating an :class:`scrapy.item.Item` class an defining its attributes as :class:`scrapy.item.Field` objects, like you will in an ORM -(don't worry if you're not familiar with ORM's, you will see that this is an +(don't worry if you're not familiar with ORMs, you will see that this is an easy task). We begin by modeling the item that we will use to hold the sites data obtained from dmoz.org, as we want to capture the name, url and description of the sites, we define fields for each of these three attributes. To do that, we edit -items.py, found in the dmoz directory. Our Item class looks like:: +items.py, found in the dmoz directory. Our Item class looks like this:: # Define here the models for your scraped items @@ -93,7 +93,7 @@ components of Scrapy that need to know how your item looks like. Our first Spider ================ -Spiders are user written classes to scrape information from a domain (or group +Spiders are user-written classes used to scrape information from a domain (or group of domains). They define an initial list of URLs to download, how to follow links, and how @@ -122,7 +122,7 @@ define the three main, mandatory, attributes: the response and returning scraped data (as :class:`~scrapy.item.Item` objects) and more URLs to follow (as :class:`~scrapy.http.Request` objects). -This is the code for our first Spider, save it in a file named +This is the code for our first Spider; save it in a file named ``dmoz_spider.py`` under the ``dmoz/spiders`` directory:: from scrapy.spider import BaseSpider @@ -176,7 +176,7 @@ Scrapy creates :class:`scrapy.http.Request` objects for each URL in the ``start_urls`` attribute of the Spider, and assigns them the ``parse`` method of the spider as their callback function. -These Requests are scheduled, then executed, and a +These Requests are scheduled, then executed, and :class:`scrapy.http.Response` objects are returned and then fed back to the spider, through the :meth:`~scrapy.spider.BaseSpider.parse` method. @@ -186,7 +186,7 @@ Extracting Items Introduction to Selectors ^^^^^^^^^^^^^^^^^^^^^^^^^ -There are several ways to extract data from web pages, Scrapy uses a mechanism +There are several ways to extract data from web pages. Scrapy uses a mechanism based on `XPath`_ expressions called :ref:`XPath selectors `. For more information about selectors and other extraction mechanisms see the :ref:`XPath selectors documentation `. @@ -207,7 +207,7 @@ Here are some examples of XPath expressions and their meanings: attribute ``class="mine"`` These are just a couple of simple examples of what you can do with XPath, but -XPath expression are indeed much more powerful. To learn more about XPath we +XPath expressions are indeed much more powerful. To learn more about XPath we recommend `this XPath tutorial `_. For working with XPaths, Scrapy provides a :class:`~scrapy.selector.XPathSelector` @@ -216,21 +216,21 @@ class, which comes in two flavours, :class:`~scrapy.selector.HtmlXPathSelector` order to use them you must instantiate the desired class with a :class:`~scrapy.http.Response` object. -You can see selectors as objects that represents nodes in the document +You can see selectors as objects that represent nodes in the document structure. So, the first instantiated selectors are associated to the root node, or the entire document. Selectors have three methods (click on the method to see the complete API documentation). -* :meth:`~scrapy.selector.XPathSelector.x`: returns a list of selectors, each of +* :meth:`~scrapy.selector.XPathSelector.select`: returns a list of selectors, each of them representing the nodes selected by the xpath expression given as argument. * :meth:`~scrapy.selector.XPathSelector.extract`: returns a unicode string with the data selected by the XPath selector. -* :meth:`~scrapy.selector.XPathSelector.re`: returns a list unicode strings +* :meth:`~scrapy.selector.XPathSelector.re`: returns a list of unicode strings extracted by applying the regular expression given as argument. @@ -241,7 +241,7 @@ To illustrate the use of Selectors we're going to use the built-in :ref:`Scrapy shell `, which also requires IPython (an extended Python console) installed on your system. -To start a shell you must go to the project's top level directory and run:: +To start a shell, you must go to the project's top level directory and run:: scrapy shell http://www.dmoz.org/Computers/Programming/Languages/Python/Books/ @@ -266,10 +266,10 @@ This is what the shell looks like:: After the shell loads, you will have the response fetched in a local ``response`` variable, so if you type ``response.body`` you will see the body -of the response, or you can ``response.headers`` to see its headers. +of the response, or you can type ``response.headers`` to see its headers. The shell also instantiates two selectors, one for HTML (in the ``hxs`` -variable) and one for XML (in the ``xxs`` variable)with this response. So let's +variable) and one for XML (in the ``xxs`` variable) with this response. So let's try them:: In [1]: hxs.select('/html/head/title') @@ -298,7 +298,7 @@ there could become a very tedious task. To make this an easier task, you can use some Firefox extensions like Firebug. For more information see :ref:`topics-firebug` and :ref:`topics-firefox`. -After inspecting the page source you'll find that the web sites information +After inspecting the page source, you'll find that the web sites information is inside a ``
    `` element, in fact the *second* ``
      `` element. So we can select each ``
    • `` element belonging to the sites list with this @@ -331,9 +331,9 @@ that property here, so:: .. note:: - For a more detailed description of using nested selectors see + For a more detailed description of using nested selectors, see :ref:`topics-selectors-nesting-selectors` and - :ref:`topics-selectors-relative-xpaths` in :ref:`topics-selectors` + :ref:`topics-selectors-relative-xpaths` in the :ref:`topics-selectors` documentation Let's add this code to our spider:: @@ -366,8 +366,8 @@ in your output, run:: Using our item -------------- -:class:`~scrapy.item.Item` objects are custom python dict, you can access the -values oftheir fields (attributes of the class we defined earlier) using the +:class:`~scrapy.item.Item` objects are custom python dicts; you can access the +values of their fields (attributes of the class we defined earlier) using the standard dict syntax like:: >>> item = DmozItem() @@ -422,7 +422,7 @@ validation, checking for duplicates, or storing it in a database), and then decide if the Item continues through the Pipeline or it's dropped and no longer processed. -In small projects (like the one on this tutorial) we will use only one Item +In small projects (like the one on this tutorial), we will use only one Item Pipeline that just stores our Items. As with Items, a Pipeline placeholder has been set up for you in the project diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index 7cb3d7839..cc1aa5d8f 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -4,11 +4,11 @@ Item Pipeline ============= -After an item has been scraped by a spider it is sent to the Item Pipeline +After an item has been scraped by a spider, it is sent to the Item Pipeline which process it through several components that are executed sequentially. -Item pipeline are usually implemented on each project. Typical usage for item -pipelines are: +Item pipelines are usually implemented on each project. Typical usage for item +pipelines consists of: * HTML cleansing * validation @@ -54,7 +54,7 @@ Additionally, they may also implement the following methods: Item pipeline example ===================== -Let's take a look at following hypothetic pipeline that adjusts the ``price`` +Let's take a look at the following hypothetic pipeline that adjusts the ``price`` attribute for those items that do not include VAT (``price_excludes_vat`` attribute), and drops those items which don't contain a price:: @@ -73,8 +73,8 @@ attribute), and drops those items which don't contain a price:: raise DropItem("Missing price in %s" % item) -Activating a Item Pipeline component -==================================== +Activating an Item Pipeline component +===================================== To activate an Item Pipeline component you must add its class to the :setting:`ITEM_PIPELINES` list, like in the following example:: @@ -87,10 +87,10 @@ Item pipeline example with resources per spider =============================================== Sometimes you need to keep resources about the items processed grouped per -spider, and delete those resource when a spider finish. +spider, and delete those resource when a spider finishes. An example is a filter that looks for duplicate items, and drops those items -that were already processed. Let say that our items has an unique id, but our +that were already processed. Let say that our items have an unique id, but our spider returns multiples items with the same id:: diff --git a/docs/topics/items.rst b/docs/topics/items.rst index c4e49e45d..5142f649b 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -56,7 +56,7 @@ defined in :class:`Field` objects could be used by a different components, and only those components know about it. You can also define and use any other :class:`Field` key in your project too, for your own needs. The main goal of :class:`Field` objects is to provide a way to define all field metadata in one -place. Typically, those components whose behaviour depends on each field, use +place. Typically, those components whose behaviour depends on each field use certain field keys to configure that behaviour. You must refer to their documentation to see which metadata keys are used by each component. @@ -143,7 +143,7 @@ Setting field values Accesing all populated values ----------------------------- -To access all populated values just use the typical `dict API`_:: +To access all populated values, just use the typical `dict API`_:: >>> product.keys() ['price', 'name'] diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index bdb121baf..207cc36af 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -7,12 +7,12 @@ Debugging memory leaks In Scrapy, objects such as Requests, Responses and Items have a finite lifetime: they are created, used for a while, and finally destroyed. -From all those objects the Request is probably the one with the longest +From all those objects, the Request is probably the one with the longest lifetime, as it stays waiting in the Scheduler queue until it's time to process it. For more info see :ref:`topics-architecture`. -As these Scrapy objects have a (rather long) lifetime there is always the risk -accumulated them in memory without releasing them properly and thus causing +As these Scrapy objects have a (rather long) lifetime, there is always the risk +of accumulating them in memory without releasing them properly and thus causing what is known as a "memory leak". To help debugging memory leaks, Scrapy provides a built-in mechanism for @@ -34,13 +34,13 @@ in Scrapy projects, and a quite difficult one to debug for newcomers. In big projects, the spiders are typically written by different people and some of those spiders could be "leaking" and thus affecting the rest of the other -(well-written) spiders when they get to run concurrently which, in turn, +(well-written) spiders when they get to run concurrently, which, in turn, affects the whole crawling process. -At the same time, it's hard to avoid the reasons that causes these leaks +At the same time, it's hard to avoid the reasons that cause these leaks without restricting the power of the framework, so we have decided not to restrict the functionally but provide useful tools for debugging these leaks, -which quite often consists in answer the question: *which spider is leaking?*. +which quite often consist in an answer to the question: *which spider is leaking?*. The leak could also come from a custom middleware, pipeline or extension that you have written, if you are not releasing the (previously allocated) resources @@ -57,10 +57,10 @@ memory leaks. It basically tracks the references to all live Requests, Responses, Item and Selector objects. To activate the ``trackref`` module, enable the :setting:`TRACK_REFS` setting. -It only imposes a minor performance impact so it should be OK for use it, even +It only imposes a minor performance impact, so it should be OK to use it, even in production environments. -Once you have ``trackref`` enabled you can enter the telnet console and inspect +Once you have ``trackref`` enabled, you can enter the telnet console and inspect how many objects (of the classes mentioned above) are currently alive using the ``prefs()`` function which is an alias to the :func:`~scrapy.utils.trackref.print_live_refs` function:: @@ -107,7 +107,7 @@ Suppose we have some spider with a line similar to this one:: callback=self.parse, meta={referer: response}") That line is passing a response reference inside a request which effectively -ties the response lifetime to the requests one, and that's would definitely +ties the response lifetime to the requests' one, and that would definitely cause memory leaks. Let's see how we can discover which one is the nasty spider (without knowing it @@ -203,7 +203,7 @@ Debugging memory leaks with Guppy leaks, but it only keeps track of the objects that are more likely to cause memory leaks (Requests, Responses, Items, and Selectors). However, there are other cases where the memory leaks could come from other (more or less obscure) -objects. If this is your case, and you can't find your leaks using ``trackref`` +objects. If this is your case, and you can't find your leaks using ``trackref``, you still have another resource: the `Guppy library`_. .. _Guppy library: http://pypi.python.org/pypi/guppy @@ -235,7 +235,7 @@ the heap using Guppy:: <1676 more rows. Type e.g. '_.more' to view.> You can see that most space is used by dicts. Then, if you want to see from -which attribute those dicts are referenced you could do:: +which attribute those dicts are referenced, you could do:: >>> x.bytype[0].byvia Partition of a set of 22307 objects. Total size = 16423880 bytes. @@ -252,7 +252,7 @@ which attribute those dicts are referenced you could do:: 9 27 0 155016 1 14841328 90 '[1]' <333 more rows. Type e.g. '_.more' to view.> -As you can see, the Guppy module is very powerful, but also requires some deep +As you can see, the Guppy module is very powerful but also requires some deep knowledge about Python internals. For more info about Guppy, refer to the `Guppy documentation`_. @@ -274,7 +274,7 @@ the operating system in some cases. For more information on this issue see: * `Python Memory Management Part 3 `_ The improvements proposed by Evan Jones, which are detailed in `this paper`_, -got merged in Python 2.5, but the only reduce the problem, it doesn't fixes it +got merged in Python 2.5, but this only reduces the problem, it doesn't fix it completely. To quote the paper: *Unfortunately, this patch can only free an arena if there are no more diff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rst index c2a296ef5..41160a57d 100644 --- a/docs/topics/link-extractors.rst +++ b/docs/topics/link-extractors.rst @@ -9,10 +9,10 @@ pages (:class:`scrapy.http.Response` objects) which will be eventually followed. There are two Link Extractors available in Scrapy by default, but you create -your own custom Link Extractors to suit your needs by implanting a simple +your own custom Link Extractors to suit your needs by implementing a simple interface. -The only public method that every LinkExtractor have is ``extract_links``, +The only public method that every LinkExtractor has is ``extract_links``, which receives a :class:`~scrapy.http.Response` object and returns a list of links. Link Extractors are meant to be instantiated once and their ``extract_links`` method called several times with different responses, to @@ -20,7 +20,7 @@ extract links to follow. Link extractors are used in the :class:`~scrapy.contrib.spiders.CrawlSpider` class (available in Scrapy), through a set of rules, but you can also use it in -your spiders even if you don't subclass from +your spiders, even if you don't subclass from :class:`~scrapy.contrib.spiders.CrawlSpider`, as its purpose is very simple: to extract links. @@ -61,12 +61,12 @@ SgmlLinkExtractor given (or empty) it won't exclude any links. :type allow: a regular expression (or list of) - :param allow_domains: is single value or a list of string containing + :param allow_domains: a single value or a list of string containing domains which will be considered for extracting the links :type allow: str or list - :param deny_domains: is single value or a list of strings containing - domains which which won't be considered for extracting the links + :param deny_domains: a single value or a list of strings containing + domains which won't be considered for extracting the links :type allow: str or list :param restrict_xpaths: is a XPath (or list of XPath's) which defines @@ -108,14 +108,14 @@ BaseSgmlLinkExtractor :param tag: either a string (with the name of a tag) or a function that receives a tag name and returns ``True`` if links should be extracted from - those tag, or ``False`` if they shouldn't. Defaults to ``'a'``. request - (once its downloaded) as its first parameter. For more information see + that tag, or ``False`` if they shouldn't. Defaults to ``'a'``. request + (once it's downloaded) as its first parameter. For more information, see :ref:`topics-request-response-ref-request-callback-arguments`. :type tag: str or callable :param attr: either string (with the name of a tag attribute), or a - function that receives a an attribute name and returns ``True`` if - links should be extracted from it, or ``False`` if the shouldn't. + function that receives an attribute name and returns ``True`` if + links should be extracted from it, or ``False`` if they shouldn't. Defaults to ``href``. :type attr: str or callable diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index 89dd12f01..79310b3b7 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -32,7 +32,7 @@ attribute. Then, you start collecting values into the Item Loader, typically using :ref:`XPath Selectors `. You can add more than one value to -the same item field, the Item Loader will know how to "join" those values later +the same item field; the Item Loader will know how to "join" those values later using a proper processing function. Here is a typical Item Loader usage in a :ref:`Spider `, using @@ -51,7 +51,7 @@ chapter `:: l.add_value('last_updated', 'today') # you can also use literal values return l.load_item() -By quickly looking at that code we can see the ``name`` field is being +By quickly looking at that code, we can see the ``name`` field is being extracted from two different XPath locations in the page: 1. ``//div[@class="product_name"]`` @@ -86,7 +86,7 @@ called with the data previously collected (and processed using the input processor). The result of the output processor is the final value that gets assigned to the item. -Let's see an example to illustrate how this input and output processors are +Let's see an example to illustrate how the input and output processors are called for a particular field (the same applies for any other field):: l = XPathItemLoader(Product(), some_xpath_selector) @@ -105,7 +105,7 @@ So what happens is: processor* used in (1). The result of the input processor is appended to the data collected in (1) (if any). -3. This case is similar to the previous ones, except that the values to be +3. This case is similar to the previous ones, except that the value to be collected is assigned directly, instead of being extracted from a XPath. However, the value is still passed through the input processors. In this case, since the value is not iterable it is converted to an iterable of a @@ -118,7 +118,7 @@ So what happens is: It's worth noticing that processors are just callable objects, which are called with the data to be parsed, and return a parsed value. So you can use any -function as input or output processor. They only requirement is that they must +function as input or output processor. The only requirement is that they must accept one (and only one) positional argument, which will be an iterator. .. note:: Both input and output processors must receive an iterator as their @@ -435,10 +435,10 @@ different parsing rules for each spider, having a lot of exceptions, but also wanting to reuse the common processors. Item Loaders are designed to ease the maintenance burden of parsing rules, -without loosing flexibility and, at the same time, providing a convenient +without losing flexibility and, at the same time, providing a convenient mechanism for extending and overriding them. For this reason Item Loaders support traditional Python class inheritance for dealing with differences of -specific spiders (or group of spiders). +specific spiders (or groups of spiders). Suppose, for example, that some particular site encloses their product names in three dashes (ie. ``---Plasma TV---``) and you don't want to end up scraping @@ -454,7 +454,7 @@ Product Item Loader (``ProductLoader``):: return x.strip('-') class SiteSpecificLoader(ProductLoader): - name_in = MapCompose(ProductLoader.name_in, strip_dashes) + name_in = MapCompose(strip_dashes, ProductLoader.name_in) Another case where extending Item Loaders can be very helpful is when you have multiple source formats, for example XML and HTML. In the XML version you may @@ -476,8 +476,8 @@ rule (as input processors do). See also: There are many other possible ways to extend, inherit and override your Item Loaders, and different Item Loaders hierarchies may fit better for different -projects. Scrapy only provides the mechanism, it doesn't impose any specific -organization of your Loaders collection - that's up to you and your project +projects. Scrapy only provides the mechanism; it doesn't impose any specific +organization of your Loaders collection - that's up to you and your project's needs. .. _topics-loaders-available-processors: @@ -491,7 +491,7 @@ Available built-in processors Even though you can use any callable function as input and output processors, Scrapy provides some commonly used processors, which are described below. Some of them, like the :class:`MapCompose` (which is typically used as input -processor) composes the output of several functions executed in order, to +processor) compose the output of several functions executed in order, to produce the final parsed value. Here is a list of all built-in processors: @@ -524,7 +524,7 @@ Here is a list of all built-in processors: .. class:: Join(separator=u' ') - Return the values joined with the separator given in the constructor, which + Returns the values joined with the separator given in the constructor, which defaults to ``u' '``. It doesn't accept Loader contexts. When using the default separator, this processor is equivalent to the @@ -559,7 +559,7 @@ Here is a list of all built-in processors: 'HELLO' Each function can optionally receive a ``loader_context`` parameter. For - those which does this processor will pass the currently active :ref:`Loader + those which do, this processor will pass the currently active :ref:`Loader context ` through that parameter. The keyword arguments passed in the constructor are used as the default diff --git a/docs/topics/logging.rst b/docs/topics/logging.rst index 185fed1a0..2b99ef4d4 100644 --- a/docs/topics/logging.rst +++ b/docs/topics/logging.rst @@ -5,12 +5,12 @@ Logging ======= Scrapy provides a logging facility which can be used through the -:mod:`scrapy.log` module. The current underling implementation uses `Twisted +:mod:`scrapy.log` module. The current underlying implementation uses `Twisted logging`_ but this may change in the future. .. _Twisted logging: http://twistedmatrix.com/projects/core/documentation/howto/logging.html -Logging service must be explicitly started through the :func:`scrapy.log.start` function. +The logging service must be explicitly started through the :func:`scrapy.log.start` function. .. _topics-logging-levels: @@ -55,7 +55,7 @@ scrapy.log module .. attribute:: started - A boolean which is ``True`` is logging has been started or ``False`` otherwise. + A boolean which is ``True`` if logging has been started or ``False`` otherwise. .. function:: start(logfile=None, loglevel=None, logstdout=None) diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index d7b66878c..f6c96a49e 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -15,8 +15,8 @@ across the system until they reach the Downloader, which executes the request and returns a :class:`Response` object which travels back to the spider that issued the request. -Both :class:`Request` and :class:`Response` classes have subclasses which adds -additional functionality not required in the base classes. These are described +Both :class:`Request` and :class:`Response` classes have subclasses which add +functionality not required in the base classes. These are described below in :ref:`topics-request-response-ref-request-subclasses` and :ref:`topics-request-response-ref-response-subclasses`. @@ -261,7 +261,7 @@ objects. Keep in mind that this method is implemented using `ClientForm`_ whose policy is to automatically simulate a click, by default, on any form - control that looks clickable, like a a ````. Even + control that looks clickable, like a ````. Even though this is quite convenient, and often the desired behaviour, sometimes it can cause problems which could be hard to debug. For example, when working with forms that are filled and/or submitted using @@ -284,12 +284,12 @@ objects. overridden by the one passed in this parameter. :type formdata: dict - :param clickdata: Arguments to be passed directly to ClientForm + :param clickdata: Arguments to be passed directly to the ClientForm ``click_request_data()`` method. See `ClientForm`_ homepage for more info. :type clickdata: dict - :param dont_click: If True the form data will be sumbitted without + :param dont_click: If True, the form data will be sumbitted without clicking in any element. :type dont_click: boolean @@ -302,8 +302,8 @@ Request usage examples Using FormRequest to send data via HTTP POST ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you want to simulate a HTML Form POST in your spider, and send a couple of -key-value fields you could return a :class:`FormRequest` object (from your +If you want to simulate a HTML Form POST in your spider and send a couple of +key-value fields, you can return a :class:`FormRequest` object (from your spider) like this:: return [FormRequest(url="http://www.example.com/post/action", @@ -396,7 +396,7 @@ Response objects .. attribute:: Response.request The :class:`Request` object that generated this response. This attribute is - assigned in the Scrapy engine, after the response and request has passed + assigned in the Scrapy engine, after the response and the request have passed through all :ref:`Downloader Middlewares `. In particular, this means that: @@ -404,7 +404,7 @@ Response objects redirection) to be assigned to the redirected response (with the final URL after redirection). - - Response.request.url doesn't always equals Response.url + - Response.request.url doesn't always equal Response.url - This attribute is only available in the spider code, and in the :ref:`Spider Middlewares `, but not in @@ -426,11 +426,11 @@ Response objects .. method:: Response.copy() - Return a new Response which is a copy of this Response. + Returns a new Response which is a copy of this Response. .. method:: Response.replace([url, status, headers, body, meta, flags, cls]) - Return a Response object with the same members, except for those members + Returns a Response object with the same members, except for those members given new values by whichever keyword arguments are specified. The attribute :attr:`Response.meta` is copied by default (unless a new value is given in the ``meta`` argument). @@ -452,15 +452,15 @@ TextResponse objects :class:`Response` class, which is meant to be used only for binary data, such as images, sounds or any media file. - :class:`TextResponse` objects support a new constructor arguments, in + :class:`TextResponse` objects support a new constructor argument, in addition to the base :class:`Response` objects. The remaining functionality is the same as for the :class:`Response` class and is not documented here. :param encoding: is a string which contains the encoding to use for this response. If you create a :class:`TextResponse` object with a unicode - body it will be encoded using this encoding (remember the body attribute + body, it will be encoded using this encoding (remember the body attribute is always a string). If ``encoding`` is ``None`` (default value), the - encoding will be looked up in the response headers anb body instead. + encoding will be looked up in the response headers and body instead. :type encoding: string :class:`TextResponse` objects support the following attributes in addition diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index 90149daa9..cd3796296 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -10,7 +10,7 @@ achieve this: * `BeautifulSoup`_ is a very popular screen scraping library among Python programmers which constructs a Python object based on the - structure of the HTML code and also deals with bad markup reasonable well, + structure of the HTML code and also deals with bad markup reasonably well, but it has one drawback: it's slow. * `lxml`_ is a XML parsing library (which also parses HTML) with a pythonic @@ -21,15 +21,14 @@ Scrapy comes with its own mechanism for extracting data. They're called XPath selectors (or just "selectors", for short) because they "select" certain parts of the HTML document specified by `XPath`_ expressions. -`XPath`_ is a language for selecting nodes in XML documents, which can be used -to with HTML. +`XPath`_ is a language for selecting nodes in XML documents, which can also be used with HTML. Both `lxml`_ and Scrapy Selectors are built over the `libxml2`_ library, which means they're very similar in speed and parsing accuracy. This page explains how selectors work and describes their API which is very small and simple, unlike the `lxml`_ API which is much bigger because the -`lxml`_ library can be use for many other tasks, besides selecting markup +`lxml`_ library can be used for many other tasks, besides selecting markup documents. For a complete reference of the selectors API see the :ref:`XPath selector @@ -56,7 +55,7 @@ There are two types of selectors bundled with Scrapy. Those are: .. highlight:: python Both share the same selector API, and are constructed with a Response object as -its first parameter. This is the Response they're gonna be "selecting". +their first parameter. This is the Response they're going to be "selecting". Example:: @@ -67,7 +66,7 @@ Using selectors with XPaths --------------------------- To explain how to use the selectors we'll use the `Scrapy shell` (which -provides interactive testing) and an example page located in Scrapy +provides interactive testing) and an example page located in the Scrapy documentation server: http://doc.scrapy.org/_static/selectors-sample1.html @@ -85,26 +84,26 @@ First, let's open the shell:: scrapy shell http://doc.scrapy.org/_static/selectors-sample1.html -Then, after the shell loads, you'll have some selectors already instanced and +Then, after the shell loads, you'll have some selectors already instantiated and ready to use. -Since we're dealing with HTML we'll be using the +Since we're dealing with HTML, we'll be using the :class:`~scrapy.selector.HtmlXPathSelector` object which is found, by default, in the ``hxs`` shell variable. .. highlight:: python -So, by looking at the :ref:`HTML code ` of that page +So, by looking at the :ref:`HTML code ` of that page, let's construct an XPath (using an HTML selector) for selecting the text inside the title tag:: >>> hxs.select('//title/text()') [] -As you can see, the select() method returns a XPathSelectorList, which is a list of +As you can see, the select() method returns an XPathSelectorList, which is a list of new selectors. This API can be used quickly for extracting nested data. -To actually extract the textual data you must call the selector ``extract()`` +To actually extract the textual data, you must call the selector ``extract()`` method, as follows:: >>> hxs.select('//title/text()').extract() @@ -184,7 +183,7 @@ starts with ``/``, that XPath will be absolute to the document and not relative to the ``XPathSelector`` you're calling it from. For example, suppose you want to extract all ``

      `` elements inside ``

      `` -elements. First you get would get all ``
      `` elements:: +elements. First, you would get all ``
      `` elements:: >>> divs = hxs.select('//div') @@ -235,7 +234,7 @@ XPathSelector objects ``response`` is a :class:`~scrapy.http.Response` object that will be used for selecting and extracting data - .. method:: XPathSelector.select(xpath) + .. method:: select(xpath) Apply the given XPath relative to this XPathSelector and return a list of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList`) with @@ -243,7 +242,7 @@ XPathSelector objects ``xpath`` is a string containing the XPath to apply - .. method:: XPathSelector.re(regex) + .. method:: re(regex) Apply the given regex and return a list of unicode strings with the matches. @@ -251,12 +250,12 @@ XPathSelector objects ``regex`` can be either a compiled regular expression or a string which will be compiled to a regular expression using ``re.compile(regex)`` - .. method:: XPathSelector.extract() + .. method:: extract() Return a unicode string with the content of this :class:`XPathSelector` object. - .. method:: XPathSelector.extract_unquoted() + .. method:: extract_unquoted() Return a unicode string with the content of this :class:`XPathSelector` without entities or CDATA. This method is intended to be use for text-only @@ -264,13 +263,13 @@ XPathSelector objects :class:`XPathSelector` objects which don't select a textual content (ie. if they contain tags), the output of this method is undefined. - .. method:: XPathSelector.register_namespace(prefix, uri) + .. method:: register_namespace(prefix, uri) Register the given namespace to be used in this :class:`XPathSelector`. Without registering namespaces you can't select or extract data from non-standard namespaces. See examples below. - .. method:: XPathSelector.__nonzero__() + .. method:: __nonzero__() Returns ``True`` if there is any real content selected by this :class:`XPathSelector` or ``False`` otherwise. In other words, the boolean @@ -284,15 +283,15 @@ XPathSelectorList objects The :class:`XPathSelectorList` class is subclass of the builtin ``list`` class, which provides a few additional methods. - .. method:: XPathSelectorList.select(xpath) + .. method:: select(xpath) - Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector` - objects in this list and return their results flattened, as new + Call the :meth:`XPathSelector.select` method for all :class:`XPathSelector` + objects in this list and return their results flattened, as a new :class:`XPathSelectorList`. - ``xpath`` is the same argument as the one in :meth:`XPathSelector.x` + ``xpath`` is the same argument as the one in :meth:`XPathSelector.select` - .. method:: XPathSelector.re(regex) + .. method:: re(regex) Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector` objects in this list and return their results flattened, as a list of @@ -300,13 +299,13 @@ XPathSelectorList objects ``regex`` is the same argument as the one in :meth:`XPathSelector.re` - .. method:: XPathSelector.extract() + .. method:: extract() - Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector` + Call the :meth:`XPathSelector.extract` method for all :class:`XPathSelector` objects in this list and return their results flattened, as a list of unicode strings. - .. method:: XPathSelector.extract_unquoted() + .. method:: extract_unquoted() Call the :meth:`XPathSelector.extract_unoquoted` method for all :class:`XPathSelector` objects in this list and return their results @@ -328,8 +327,8 @@ HtmlXPathSelector examples ~~~~~~~~~~~~~~~~~~~~~~~~~~ Here's a couple of :class:`HtmlXPathSelector` examples to illustrate several -concepts. In all cases we assume there is already a :class:`HtmlPathSelector` -instanced with a :class:`~scrapy.http.Response` object like this:: +concepts. In all cases, we assume there is already an :class:`HtmlPathSelector` +instantiated with a :class:`~scrapy.http.Response` object like this:: x = HtmlXPathSelector(html_response) @@ -371,7 +370,7 @@ XmlXPathSelector examples Here's a couple of :class:`XmlXPathSelector` examples to illustrate several concepts. In all cases we assume there is already a :class:`XmlPathSelector` -instanced with a :class:`~scrapy.http.Response` object like this:: +instantiated with a :class:`~scrapy.http.Response` object like this:: x = HtmlXPathSelector(xml_response) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 33f8a2f8c..7f5aba5c7 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -10,11 +10,11 @@ Settings The Scrapy settings allows you to customize the behaviour of all Scrapy components, including the core, extensions, pipelines and spiders themselves. -The infrastructure of setting provides a global namespace of key-value mappings +The infrastructure of the settings provides a global namespace of key-value mappings that the code can use to pull configuration values from. The settings can be populated through different mechanisms, which are described below. -The settings is also the mechanism for selecting the currently active Scrapy +The settings are also the mechanism for selecting the currently active Scrapy project (in case you have many). For a list of available built-in settings see: :ref:`topics-settings-ref`. @@ -46,13 +46,13 @@ precedence: 4. Default settings per-command 5. Default global settings (less precedence) -This mechanisms are described with more detail below. +These mechanisms are described in more detail below. 1. Global overrides ------------------- -Global overrides are the ones that takes most precedence, and are usually -populated by command line options. +Global overrides are the ones that take most precedence, and are usually +populated by command-line options. Example:: >>> from scrapy.conf import settings @@ -163,7 +163,7 @@ to do that you'll have to use one of the following methods: .. method:: Settings.getlist(name, default=None) Get a setting value as a list. If the setting original type is a list it - will be returned verbatim. If it's a string it will be splitted by ",". + will be returned verbatim. If it's a string it will be split by ",". For example, settings populated through environment variables set to ``'one,two'`` will return a list ['one', 'two'] when using this method. @@ -245,7 +245,7 @@ COMMANDS_MODULE Default: ``''`` (empty string) A module to use for looking for custom Scrapy commands. This is used to add -custom command for your Scrapy project. +custom commands for your Scrapy project. Example:: @@ -530,7 +530,7 @@ Default:: 'scrapy.contrib.closedomain.CloseDomain': 0, } -The list of available extensions. Keep in mind that some of them need need to +The list of available extensions. Keep in mind that some of them need to be enabled through a setting. By default, this setting contains all stable built-in extensions. @@ -553,7 +553,7 @@ GROUPSETTINGS_MODULE Default: ``''`` (empty string) -The module to use for pulling settings from, if the group settings is enabled. +The module to use for pulling settings from, if group settings are enabled. .. setting:: ITEM_PIPELINES @@ -740,7 +740,7 @@ spider. This randomization decreases the chance of the crawler being detected (and subsequently blocked) by sites which analyze requests looking for statistically -significant similarities in the time between their times. +significant similarities in the time between their requests. The randomization policy is the same used by `wget`_ ``--random-wait`` option. @@ -966,7 +966,7 @@ STATSMAILER_RCPTS Default: ``[]`` (empty list) -Send Scrapy stats after domains finish scrapy. See +Send Scrapy stats after domains finish scraping. See :class:`~scrapy.contrib.statsmailer.StatsMailer` for more info. .. setting:: TELNETCONSOLE_ENABLED @@ -1019,6 +1019,6 @@ USER_AGENT Default: ``"%s/%s" % (BOT_NAME, BOT_VERSION)`` -The default User-Agent to use when crawling, unless overrided. +The default User-Agent to use when crawling, unless overridden. .. _Amazon web services: http://aws.amazon.com/ diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index 85690a74a..87dc27d51 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -14,14 +14,14 @@ data they extract from the web pages you're trying to scrape. It allows you to interactively test your XPaths while you're writing your spider, without having to run the spider to test every change. -Once you get familiarized with the Scrapy shell you'll see that it's an +Once you get familiarized with the Scrapy shell, you'll see that it's an invaluable tool for developing and debugging your spiders. If you have `IPython`_ installed, the Scrapy shell will use it (instead of the -standard Python console). The `IPython`_ console is a much more powerful and +standard Python console). The `IPython`_ console is much more powerful and provides smart auto-completion and colorized output, among other things. -We highly recommend you to install `IPython`_, specially if you're working on +We highly recommend you install `IPython`_, specially if you're working on Unix systems (where `IPython`_ excels). See the `IPython installation guide`_ for more info. @@ -72,11 +72,11 @@ content). Those objects are: * ``spider`` - the Spider which is known to handle the URL, or a - :class:`~scrapy.spider.BaseSpider` object if there is no spider is found for + :class:`~scrapy.spider.BaseSpider` object if there is no spider found for the current URL * ``request`` - a :class:`~scrapy.http.Request` object of the last fetched - page. You can modify this request using :meth:`~scrapy.http.Request.replace` + page. You can modify this request using :meth:`~scrapy.http.Request.replace` or fetch a new request (without leaving the shell) using the ``fetch`` shortcut. @@ -125,7 +125,7 @@ all start with the ``[s]`` prefix):: >>> -After that, we can stary playing with the objects:: +After that, we can star playing with the objects:: >>> hxs.select("//h2/text()").extract()[0] u'Welcome to Scrapy' @@ -164,7 +164,7 @@ Here's an example of how you would call it from your spider:: # ... your parsing code .. -When you the spider you will get something similar to this:: +When you run the spider, you will get something similar to this:: 2009-08-27 19:15:25-0300 [example.com] DEBUG: Crawled (referer: ) 2009-08-27 19:15:26-0300 [example.com] DEBUG: Crawled (referer: ) diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 25fc100f1..ceda30023 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -137,7 +137,7 @@ spider_closed :type spider: :class:`~scrapy.spider.BaseSpider` object :param reason: a string which describes the reason why the spider was closed. If - it was closed because the spider has completed scraping, it the reason + it was closed because the spider has completed scraping, the reason is ``'finished'``. Otherwise, if the spider was manually closed by calling the ``close_spider`` engine method, then the reason is the one passed in the ``reason`` argument of that method (which defaults to diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 2889378cd..da3c9d5f9 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -7,7 +7,7 @@ Spider Middleware The spider middleware is a framework of hooks into Scrapy's spider processing mechanism where you can plug custom functionality to process the requests that are sent to :ref:`topics-spiders` for processing and to process the responses -and item that are generated from spiders. +and items that are generated from spiders. .. _topics-spider-middleware-setting: diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index fafd76788..d42c8b24d 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -22,21 +22,21 @@ For spiders, the scraping cycle goes through something like this: :attr:`~scrapy.spider.BaseSpider.parse` method as callback function for the Requests. -2. In the callback function you parse the response (web page) and return either +2. In the callback function, you parse the response (web page) and return either :class:`~scrapy.item.Item` objects, :class:`~scrapy.http.Request` objects, or an iterable of both. Those Requests will also contain a callback (maybe - the same) and will then be followed by downloaded by Scrapy and then their - response handled to the specified callback. + the same) and will then be downloaded by Scrapy and then their + response handled by the specified callback. -3. In callback functions you parse the page contants, typically using +3. In callback functions, you parse the page contents, typically using :ref:`topics-selectors` (but you can also use BeautifuSoup, lxml or whatever mechanism you prefer) and generate items with the parsed data. -4. Finally the items returned from the spider will be typically persisted in +4. Finally, the items returned from the spider will be typically persisted in some Item pipeline. -Even though this cycles applies (more or less) to any kind of spider, there are -different kind of default spiders bundled into Scrapy for different purposes. +Even though this cycle applies (more or less) to any kind of spider, there are +different kinds of default spiders bundled into Scrapy for different purposes. We will talk about those types here. @@ -45,7 +45,7 @@ We will talk about those types here. Built-in spiders reference ========================== -For the examples used in the following spiders reference we'll assume we have a +For the examples used in the following spiders reference, we'll assume we have a ``TestItem`` declared in a ``myproject.items`` module, in your project:: from scrapy.item import Item @@ -89,7 +89,7 @@ BaseSpider .. attribute:: start_urls - Is a list of URLs where the spider will begin to crawl from, when no + A list of URLs where the spider will begin to crawl from, when no particular URLs are specified. So, the first pages downloaded will be those listed here. The subsequent URLs will be generated successively from data contained in the start URLs. @@ -109,7 +109,7 @@ BaseSpider generate Requests for each url in :attr:`start_urls`. If you want to change the Requests used to start scraping a domain, this is - the method to override. For example, if you need to start by login in using + the method to override. For example, if you need to start by logging in using a POST request, you could do:: def start_requests(self): @@ -213,7 +213,7 @@ CrawlSpider provides a convenient mechanism for following links by defining a set of rules. It may not be the best suited for your particular web sites or project, but it's generic enough for several cases, so you can start from it and override it - as need more custom functionality, or just implement your own spider. + as needed for more custom functionality, or just implement your own spider. Apart from the attributes inherited from BaseSpider (that you must specify), this class supports a new attribute: @@ -222,7 +222,7 @@ CrawlSpider Which is a list of one (or more) :class:`Rule` objects. Each :class:`Rule` defines a certain behaviour for crawling the site. Rules objects are - described below . + described below. Crawling rules ~~~~~~~~~~~~~~ @@ -240,7 +240,7 @@ Crawling rules ``cb_kwargs`` is a dict containing the keyword arguments to be passed to the callback function - ``follow`` is a boolean which specified if links should be followed from each + ``follow`` is a boolean which specifies if links should be followed from each response extracted with this rule. If ``callback`` is None ``follow`` defaults to ``True``, otherwise it default to ``False``. @@ -306,7 +306,7 @@ XMLFeedSpider whole DOM at once in order to parse it. However, using ``html`` as the iterator may be useful when parsing XML with bad markup. - For setting the iterator and the tag name, you must define the following class + To set the iterator and the tag name, you must define the following class attributes: .. attribute:: iterator @@ -356,9 +356,9 @@ XMLFeedSpider .. method:: adapt_response(response) A method that receives the response as soon as it arrives from the spider - middleware and before start parsing it. It can be used used for modifying + middleware, before the spider starts parsing it. It can be used to modify the response body before parsing it. This method receives a response and - returns response (it could be the same or another one). + also returns a response (it could be the same or another one). .. method:: parse_node(response, selector) @@ -375,13 +375,13 @@ XMLFeedSpider spider, and it's intended to perform any last time processing required before returning the results to the framework core, for example setting the item IDs. It receives a list of results and the response which originated - that results. It must return a list of results (Items or Requests).""" + those results. It must return a list of results (Items or Requests). XMLFeedSpider example ~~~~~~~~~~~~~~~~~~~~~ -These spiders are pretty easy to use, let's have at one example:: +These spiders are pretty easy to use, let's have a look at one example:: from scrapy import log from scrapy.contrib.spiders import XMLFeedSpider @@ -403,7 +403,7 @@ These spiders are pretty easy to use, let's have at one example:: item['description'] = node.select('description').extract() return item -Basically what we did up there was creating a spider that downloads a feed from +Basically what we did up there was to create a spider that downloads a feed from the given ``start_urls``, and then iterates through each of its ``item`` tags, prints them out, and stores some random data in an :class:`~scrapy.item.Item`. @@ -416,22 +416,22 @@ CSVFeedSpider over rows, instead of nodes. The method that gets called in each iteration is :meth:`parse_row`. - .. attribute:: CSVFeedSpider.delimiter + .. attribute:: delimiter A string with the separator character for each field in the CSV file Defaults to ``','`` (comma). - .. attribute:: CSVFeedSpider.headers + .. attribute:: headers - A list of the rows contained in the file CSV feed which will be used for - extracting fields from it. + A list of the rows contained in the file CSV feed which will be used to + extract fields from it. - .. method:: CSVFeedSpider.parse_row(response, row) + .. method:: parse_row(response, row) Receives a response and a dict (representing each row) with a key for each provided (or detected) header of the CSV file. This spider also gives the opportunity to override ``adapt_response`` and ``process_results`` methods - for pre and post-processing purposes. + for pre- and post-processing purposes. CSVFeedSpider example ~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst index 34a3c8b7c..81021f230 100644 --- a/docs/topics/stats.rst +++ b/docs/topics/stats.rst @@ -121,7 +121,7 @@ class (which they all inherit from). Get all stats from the given spider (if spider is given) or all global stats otherwise, as a dict. If spider is not opened ``KeyError`` is - raied. + raised. .. method:: set_value(key, value, spider=None) @@ -146,7 +146,7 @@ class (which they all inherit from). Set the given value for the given key only if current value for the same key is lower than value. If there is no current value for the - given key, the value is always set. If spider is not given the global + given key, the value is always set. If spider is not given, the global stats table is used, otherwise the spider-specific stats table is used, which must be opened or a KeyError will be raised. @@ -154,7 +154,7 @@ class (which they all inherit from). Set the given value for the given key only if current value for the same key is greater than value. If there is no current value for the - given key, the value is always set. If spider is not given the global + given key, the value is always set. If spider is not given, the global stats table is used, otherwise the spider-specific stats table is used, which must be opened or a KeyError will be raised. @@ -191,7 +191,7 @@ Available Stats Collectors Besides the basic :class:`StatsCollector` there are other Stats Collectors available in Scrapy which extend the basic Stats Collector. You can select which Stats Collector to use through the :setting:`STATS_CLASS` setting. The -default Stats Collector is the :class:`MemoryStatsCollector` is used. +default Stats Collector used is the :class:`MemoryStatsCollector`. When stats are disabled (through the :setting:`STATS_ENABLED` setting) the :setting:`STATS_CLASS` setting is ignored and the :class:`DummyStatsCollector` @@ -220,7 +220,7 @@ DummyStatsCollector .. class:: DummyStatsCollector A Stats collector which does nothing but is very efficient. This is the - Stats Collector used when stats are diabled (through the + Stats Collector used when stats are disabled (through the :setting:`STATS_ENABLED` setting). SimpledbStatsCollector @@ -237,17 +237,17 @@ SimpledbStatsCollector :setting:`STATS_SDB_DOMAIN` setting. The domain will be created if it doesn't exist. - In addition to the existing stats keys the following keys are added at + In addition to the existing stats keys, the following keys are added at persitance time: * ``spider``: the spider name (so you can use it later for querying stats for that spider) - * ``timestamp``: the timestamp when the stats were persisited + * ``timestamp``: the timestamp when the stats were persisted - Both the ``spider`` and ``timestamp`` are used for generating the SimpleDB + Both the ``spider`` and ``timestamp`` are used to generate the SimpleDB item name in order to avoid overwriting stats of previous scraping runs. - As `required by SimpleDB`_, datetime's are stored in ISO 8601 format and + As `required by SimpleDB`_, datetimes are stored in ISO 8601 format and numbers are zero-padded to 16 digits. Negative numbers are not currently supported. @@ -276,7 +276,7 @@ STATS_SDB_ASYNC Default: ``False`` -If ``True`` communication with SimpleDB will be performed asynchronously. If +If ``True``, communication with SimpleDB will be performed asynchronously. If ``False`` blocking IO will be used instead. This is the default as using asynchronous communication can result in the stats not being persisted if the Scrapy engine is shut down in the middle (for example, when you run only one @@ -295,7 +295,7 @@ functionality: .. function:: stats_spider_opened(spider) Sent right after the stats spider is opened. You can use this signal to add - startup stats for spider (example: start time). + startup stats for the spider (example: start time). :param spider: the stats spider just opened :type spider: str @@ -318,7 +318,7 @@ functionality: Sent right after the stats spider is closed. You can use this signal to collect resources, but not to add any more stats as the stats spider has - already been close (use :signal:`stats_spider_closing` for that instead). + already been closed (use :signal:`stats_spider_closing` for that instead). :param spider: the stats spider just closed :type spider: str diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst index 8fd414d2d..99b6e627e 100644 --- a/docs/topics/telnetconsole.rst +++ b/docs/topics/telnetconsole.rst @@ -64,15 +64,15 @@ convenience: .. _pprint.pprint: http://docs.python.org/library/pprint.html#pprint.pprint -Some example of using the telnet console -======================================== +Telnet console usage examples +============================= Here are some example tasks you can do with the telnet console: View engine status ------------------ -You can use the ``st()`` method of the Scrapy engine to quickly show its state +You can use the ``est()`` method of the Scrapy engine to quickly show its state using the telnet console:: telnet localhost 6023 @@ -105,8 +105,8 @@ using the telnet console:: len(self._scraping[domain]) : 0 -Pause, resume and stop Scrapy engine ------------------------------------- +Pause, resume and stop the Scrapy engine +---------------------------------------- To pause::