mirror of https://github.com/scrapy/scrapy.git
Fix some typos, whitespace and small errors in docs
This commit is contained in:
Alexey Bezhan
parent
2e59d77580
commit
210a0a6fe1
|
|
@ -65,10 +65,10 @@ Defining our Item
|
|||
=================
|
||||
|
||||
`Items` are containers that will be loaded with the scraped data; they work
|
||||
like simple python dicts but provide additional protecting against populating
|
||||
like simple python dicts but provide additional protection against populating
|
||||
undeclared fields, to prevent typos.
|
||||
|
||||
They are declared by creating an :class:`scrapy.item.Item` class and defining
|
||||
They are declared by creating a :class:`scrapy.item.Item` class and defining
|
||||
its attributes as :class:`scrapy.item.Field` objects, like you will in an ORM
|
||||
(don't worry if you're not familiar with ORMs, you will see that this is an
|
||||
easy task).
|
||||
|
|
@ -76,7 +76,7 @@ easy task).
|
|||
We begin by modeling the item that we will use to hold the sites data obtained
|
||||
from dmoz.org, as we want to capture the name, url and description of the
|
||||
sites, we define fields for each of these three attributes. To do that, we edit
|
||||
items.py, found in the ``tutorial`` directory. Our Item class looks like this::
|
||||
``items.py``, found in the ``tutorial`` directory. Our Item class looks like this::
|
||||
|
||||
from scrapy.item import Item, Field
|
||||
|
||||
|
|
@ -86,7 +86,7 @@ items.py, found in the ``tutorial`` directory. Our Item class looks like this::
|
|||
desc = Field()
|
||||
|
||||
This may seem complicated at first, but defining the item allows you to use other handy
|
||||
components of Scrapy that need to know how your item looks like.
|
||||
components of Scrapy that need to know how your item looks.
|
||||
|
||||
Our first Spider
|
||||
================
|
||||
|
|
@ -97,8 +97,8 @@ of domains).
|
|||
They define an initial list of URLs to download, how to follow links, and how
|
||||
to parse the contents of those pages to extract :ref:`items <topics-items>`.
|
||||
|
||||
To create a Spider, you must subclass :class:`scrapy.spider.Spider`, and
|
||||
define the three main, mandatory, attributes:
|
||||
To create a Spider, you must subclass :class:`scrapy.spider.Spider` and
|
||||
define the three main mandatory attributes:
|
||||
|
||||
* :attr:`~scrapy.spider.Spider.name`: identifies the Spider. It must be
|
||||
unique, that is, you can't set the same name for different Spiders.
|
||||
|
|
@ -162,7 +162,7 @@ will get an output similar to this::
|
|||
Pay attention to the lines containing ``[dmoz]``, which corresponds to our
|
||||
spider. You can see a log line for each URL defined in ``start_urls``. Because
|
||||
these URLs are the starting ones, they have no referrers, which is shown at the
|
||||
end of the log line, where it says ``(referer: <None>)``.
|
||||
end of the log line, where it says ``(referer: None)``.
|
||||
|
||||
But more interesting, as our ``parse`` method instructs, two files have been
|
||||
created: *Books* and *Resources*, with the content of both URLs.
|
||||
|
|
@ -210,15 +210,15 @@ XPath expressions are indeed much more powerful. To learn more about XPath we
|
|||
recommend `this XPath tutorial <http://www.w3schools.com/XPath/default.asp>`_.
|
||||
|
||||
For working with XPaths, Scrapy provides a :class:`~scrapy.selector.Selector`
|
||||
class, it is instantiated with a :class:`~scrapy.http.HtmlResponse` or
|
||||
class, which is instantiated with a :class:`~scrapy.http.HtmlResponse` or
|
||||
:class:`~scrapy.http.XmlResponse` object as first argument.
|
||||
|
||||
You can see selectors as objects that represent nodes in the document
|
||||
structure. So, the first instantiated selectors are associated to the root
|
||||
structure. So, the first instantiated selectors are associated with the root
|
||||
node, or the entire document.
|
||||
|
||||
Selectors have four basic methods (click on the method to see the complete API
|
||||
documentation).
|
||||
documentation):
|
||||
|
||||
* :meth:`~scrapy.selector.Selector.xpath`: returns a list of selectors, each of
|
||||
them representing the nodes selected by the xpath expression given as
|
||||
|
|
@ -275,7 +275,7 @@ After the shell loads, you will have the response fetched in a local
|
|||
``response`` variable, so if you type ``response.body`` you will see the body
|
||||
of the response, or you can type ``response.headers`` to see its headers.
|
||||
|
||||
The shell also pre-instantiate a selector for this response in variable ``sel``,
|
||||
The shell also pre-instantiates a selector for this response in variable ``sel``,
|
||||
the selector automatically chooses the best parsing rules (XML vs HTML) based
|
||||
on response's type.
|
||||
|
||||
|
|
@ -327,7 +327,7 @@ And the sites links::
|
|||
|
||||
sel.xpath('//ul/li/a/@href').extract()
|
||||
|
||||
As we said before, each ``.xpath()`` call returns a list of selectors, so we can
|
||||
As we've said before, each ``.xpath()`` call returns a list of selectors, so we can
|
||||
concatenate further ``.xpath()`` calls to dig deeper into a node. We are going to use
|
||||
that property here, so::
|
||||
|
||||
|
|
@ -418,7 +418,7 @@ scraped so far, the final code for our Spider would be like this::
|
|||
.. note:: You can find a fully-functional variant of this spider in the dirbot_
|
||||
project available at https://github.com/scrapy/dirbot
|
||||
|
||||
Now doing a crawl on the dmoz.org domain yields ``DmozItem``'s::
|
||||
Now doing a crawl on the dmoz.org domain yields ``DmozItem`` objects::
|
||||
|
||||
[dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
|
||||
{'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n],
|
||||
|
|
@ -445,7 +445,7 @@ However, if you want to perform more complex things with the scraped items, you
|
|||
can write an :ref:`Item Pipeline <topics-item-pipeline>`. As with Items, a
|
||||
placeholder file for Item Pipelines has been set up for you when the project is
|
||||
created, in ``tutorial/pipelines.py``. Though you don't need to implement any item
|
||||
pipeline if you just want to store the scraped items.
|
||||
pipelines if you just want to store the scraped items.
|
||||
|
||||
Next steps
|
||||
==========
|
||||
|
|
|
|||
|
|
@ -7,8 +7,8 @@ Command line tool
|
|||
.. versionadded:: 0.10
|
||||
|
||||
Scrapy is controlled through the ``scrapy`` command-line tool, to be referred
|
||||
here as the "Scrapy tool" to differentiate it from their sub-commands which we
|
||||
just call "commands", or "Scrapy commands".
|
||||
here as the "Scrapy tool" to differentiate it from the sub-commands, which we
|
||||
just call "commands" or "Scrapy commands".
|
||||
|
||||
The Scrapy tool provides several commands, for multiple purposes, and each one
|
||||
accepts a different set of arguments and options.
|
||||
|
|
@ -214,7 +214,7 @@ crawl
|
|||
* Syntax: ``scrapy crawl <spider>``
|
||||
* Requires project: *yes*
|
||||
|
||||
Start crawling a spider.
|
||||
Start crawling using a spider.
|
||||
|
||||
Usage examples::
|
||||
|
||||
|
|
@ -297,13 +297,13 @@ Downloads the given URL using the Scrapy downloader and writes the contents to
|
|||
standard output.
|
||||
|
||||
The interesting thing about this command is that it fetches the page how the
|
||||
the spider would download it. For example, if the spider has an ``USER_AGENT``
|
||||
spider would download it. For example, if the spider has an ``USER_AGENT``
|
||||
attribute which overrides the User Agent, it will use that one.
|
||||
|
||||
So this command can be used to "see" how your spider would fetch certain page.
|
||||
So this command can be used to "see" how your spider would fetch a certain page.
|
||||
|
||||
If used outside a project, no particular per-spider behaviour would be applied
|
||||
and it will just use the default Scrapy downloder settings.
|
||||
and it will just use the default Scrapy downloader settings.
|
||||
|
||||
Usage examples::
|
||||
|
||||
|
|
@ -346,7 +346,7 @@ shell
|
|||
* Syntax: ``scrapy shell [url]``
|
||||
* Requires project: *no*
|
||||
|
||||
Starts the Scrapy shell for the given URL (if given) or empty if not URL is
|
||||
Starts the Scrapy shell for the given URL (if given) or empty if no URL is
|
||||
given. See :ref:`topics-shell` for more info.
|
||||
|
||||
Usage example::
|
||||
|
|
@ -362,7 +362,7 @@ parse
|
|||
* Syntax: ``scrapy parse <url> [options]``
|
||||
* Requires project: *yes*
|
||||
|
||||
Fetches the given URL and parses with the spider that handles it, using the
|
||||
Fetches the given URL and parses it with the spider that handles it, using the
|
||||
method passed with the ``--callback`` option, or ``parse`` if not given.
|
||||
|
||||
Supported options:
|
||||
|
|
@ -371,7 +371,7 @@ Supported options:
|
|||
response
|
||||
|
||||
* ``--rules`` or ``-r``: use :class:`~scrapy.contrib.spiders.CrawlSpider`
|
||||
rules to discover the callback (ie. spider method) to use for parsing the
|
||||
rules to discover the callback (i.e. spider method) to use for parsing the
|
||||
response
|
||||
|
||||
* ``--noitems``: don't show scraped items
|
||||
|
|
@ -467,7 +467,7 @@ bench
|
|||
* Syntax: ``scrapy bench``
|
||||
* Requires project: *no*
|
||||
|
||||
Run quick benchmark test. :ref:`benchmarking`.
|
||||
Run a quick benchmark test. :ref:`benchmarking`.
|
||||
|
||||
Custom project commands
|
||||
=======================
|
||||
|
|
@ -484,7 +484,7 @@ COMMANDS_MODULE
|
|||
|
||||
Default: ``''`` (empty string)
|
||||
|
||||
A module to use for looking custom Scrapy commands. This is used to add custom
|
||||
A module to use for looking up custom Scrapy commands. This is used to add custom
|
||||
commands for your Scrapy project.
|
||||
|
||||
Example::
|
||||
|
|
|
|||
|
|
@ -47,11 +47,11 @@ Item Fields
|
|||
|
||||
:class:`Field` objects are used to specify metadata for each field. For
|
||||
example, the serializer function for the ``last_updated`` field illustrated in
|
||||
the example above.
|
||||
the example above.
|
||||
|
||||
You can specify any kind of metadata for each field. There is no restriction on
|
||||
the values accepted by :class:`Field` objects. For this same
|
||||
reason, there isn't a reference list of all available metadata keys. Each key
|
||||
reason, there is no reference list of all available metadata keys. Each key
|
||||
defined in :class:`Field` objects could be used by a different components, and
|
||||
only those components know about it. You can also define and use any other
|
||||
:class:`Field` key in your project too, for your own needs. The main goal of
|
||||
|
|
@ -62,9 +62,9 @@ documentation to see which metadata keys are used by each component.
|
|||
|
||||
It's important to note that the :class:`Field` objects used to declare the item
|
||||
do not stay assigned as class attributes. Instead, they can be accessed through
|
||||
the :attr:`Item.fields` attribute.
|
||||
the :attr:`Item.fields` attribute.
|
||||
|
||||
And that's all you need to know about declaring items.
|
||||
And that's all you need to know about declaring items.
|
||||
|
||||
Working with Items
|
||||
==================
|
||||
|
|
|
|||
|
|
@ -227,7 +227,7 @@ value and extracts a length from it::
|
|||
return parsed_length
|
||||
|
||||
By accepting a ``loader_context`` argument the function is explicitly telling
|
||||
the Item Loader that is able to receive an Item Loader context, so the Item
|
||||
the Item Loader that it's able to receive an Item Loader context, so the Item
|
||||
Loader passes the currently active context when calling it, and the processor
|
||||
function (``parse_length`` in this case) can thus use them.
|
||||
|
||||
|
|
@ -245,7 +245,7 @@ There are several ways to modify Item Loader context values:
|
|||
loader = ItemLoader(product, unit='cm')
|
||||
|
||||
3. On Item Loader declaration, for those input/output processors that support
|
||||
instatiating them with a Item Loader context. :class:`~processor.MapCompose` is one of
|
||||
instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of
|
||||
them::
|
||||
|
||||
class ProductLoader(ItemLoader):
|
||||
|
|
@ -486,7 +486,7 @@ Reusing and extending Item Loaders
|
|||
==================================
|
||||
|
||||
As your project grows bigger and acquires more and more spiders, maintenance
|
||||
becomes a fundamental problem, specially when you have to deal with many
|
||||
becomes a fundamental problem, especially when you have to deal with many
|
||||
different parsing rules for each spider, having a lot of exceptions, but also
|
||||
wanting to reuse the common processors.
|
||||
|
||||
|
|
@ -497,7 +497,7 @@ support traditional Python class inheritance for dealing with differences of
|
|||
specific spiders (or groups of spiders).
|
||||
|
||||
Suppose, for example, that some particular site encloses their product names in
|
||||
three dashes (ie. ``---Plasma TV---``) and you don't want to end up scraping
|
||||
three dashes (e.g. ``---Plasma TV---``) and you don't want to end up scraping
|
||||
those dashes in the final product names.
|
||||
|
||||
Here's how you can remove those dashes by reusing and extending the default
|
||||
|
|
@ -567,7 +567,7 @@ Here is a list of all built-in processors:
|
|||
|
||||
.. class:: TakeFirst
|
||||
|
||||
Return the first non-null/non-empty value from the values received,
|
||||
Returns the first non-null/non-empty value from the values received,
|
||||
so it's typically used as an output processor to single-valued fields.
|
||||
It doesn't receive any constructor arguments, nor accept Loader contexts.
|
||||
|
||||
|
|
@ -604,8 +604,8 @@ Here is a list of all built-in processors:
|
|||
function, and so on, until the last function returns the output value of
|
||||
this processor.
|
||||
|
||||
By default, stop process on None value. This behaviour can be changed by
|
||||
passing keyword argument stop_on_none=False.
|
||||
By default, stop process on ``None`` value. This behaviour can be changed by
|
||||
passing keyword argument ``stop_on_none=False``.
|
||||
|
||||
Example::
|
||||
|
||||
|
|
|
|||
|
|
@ -4,11 +4,11 @@
|
|||
Spiders
|
||||
=======
|
||||
|
||||
Spiders are classes which define how a certain site (or group of sites) will be
|
||||
scraped, including how to perform the crawl (ie. follow links) and how to
|
||||
extract structured data from their pages (ie. scraping items). In other words,
|
||||
Spiders are classes which define how a certain site (or a group of sites) will be
|
||||
scraped, including how to perform the crawl (i.e. follow links) and how to
|
||||
extract structured data from their pages (i.e. scraping items). In other words,
|
||||
Spiders are the place where you define the custom behaviour for crawling and
|
||||
parsing pages for a particular site (or, in some cases, group of sites).
|
||||
parsing pages for a particular site (or, in some cases, a group of sites).
|
||||
|
||||
For spiders, the scraping cycle goes through something like this:
|
||||
|
||||
|
|
@ -105,7 +105,7 @@ Spider
|
|||
method ``parse`` for each of the resulting responses.
|
||||
|
||||
.. attribute:: name
|
||||
|
||||
|
||||
A string which defines the name for this spider. The spider name is how
|
||||
the spider is located (and instantiated) by Scrapy, so it must be
|
||||
unique. However, nothing prevents you from instantiating more than one
|
||||
|
|
@ -113,7 +113,7 @@ Spider
|
|||
and it's required.
|
||||
|
||||
If the spider scrapes a single domain, a common practice is to name the
|
||||
spider after the domain, or without the `TLD`_. So, for example, a
|
||||
spider after the domain, with or without the `TLD`_. So, for example, a
|
||||
spider that crawls ``mywebsite.com`` would often be called
|
||||
``mywebsite``.
|
||||
|
||||
|
|
@ -134,8 +134,8 @@ Spider
|
|||
.. method:: start_requests()
|
||||
|
||||
This method must return an iterable with the first Requests to crawl for
|
||||
this spider.
|
||||
|
||||
this spider.
|
||||
|
||||
This is the method called by Scrapy when the spider is opened for
|
||||
scraping when no particular URLs are specified. If particular URLs are
|
||||
specified, the :meth:`make_requests_from_url` is used instead to create
|
||||
|
|
@ -150,7 +150,7 @@ Spider
|
|||
a POST request, you could do::
|
||||
|
||||
def start_requests(self):
|
||||
return [FormRequest("http://www.example.com/login",
|
||||
return [FormRequest("http://www.example.com/login",
|
||||
formdata={'user': 'john', 'pass': 'secret'},
|
||||
callback=self.logged_in)]
|
||||
|
||||
|
|
@ -214,7 +214,7 @@ Let's see an example::
|
|||
def parse(self, response):
|
||||
self.log('A response from %s just arrived!' % response.url)
|
||||
|
||||
Another example returning multiples Requests and Items from a single callback::
|
||||
Another example returning multiple Requests and Items from a single callback::
|
||||
|
||||
from scrapy.selector import Selector
|
||||
from scrapy.spider import Spider
|
||||
|
|
@ -267,10 +267,10 @@ CrawlSpider
|
|||
.. method:: parse_start_url(response)
|
||||
|
||||
This method is called for the start_urls responses. It allows to parse
|
||||
the initial responses and must return either a
|
||||
:class:`~scrapy.item.Item` object, a :class:`~scrapy.http.Request`
|
||||
the initial responses and must return either a
|
||||
:class:`~scrapy.item.Item` object, a :class:`~scrapy.http.Request`
|
||||
object, or an iterable containing any of them.
|
||||
|
||||
|
||||
Crawling rules
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
|
|
@ -278,7 +278,7 @@ Crawling rules
|
|||
|
||||
``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
|
||||
defines how links will be extracted from each crawled page.
|
||||
|
||||
|
||||
``callback`` is a callable or a string (in which case a method from the spider
|
||||
object with that name will be used) to be called for each link extracted with
|
||||
the specified link_extractor. This callback receives a response as its first
|
||||
|
|
@ -291,7 +291,7 @@ Crawling rules
|
|||
the crawl spider will no longer work.
|
||||
|
||||
``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
|
||||
callback function
|
||||
callback function.
|
||||
|
||||
``follow`` is a boolean which specifies if links should be followed from each
|
||||
response extracted with this rule. If ``callback`` is None ``follow`` defaults
|
||||
|
|
@ -300,7 +300,7 @@ Crawling rules
|
|||
``process_links`` is a callable, or a string (in which case a method from the
|
||||
spider object with that name will be used) which will be called for each list
|
||||
of links extracted from each response using the specified ``link_extractor``.
|
||||
This is mainly used for filtering purposes.
|
||||
This is mainly used for filtering purposes.
|
||||
|
||||
``process_request`` is a callable, or a string (in which case a method from
|
||||
the spider object with that name will be used) which will be called with
|
||||
|
|
@ -321,7 +321,7 @@ Let's now take a look at an example CrawlSpider with rules::
|
|||
name = 'example.com'
|
||||
allowed_domains = ['example.com']
|
||||
start_urls = ['http://www.example.com']
|
||||
|
||||
|
||||
rules = (
|
||||
# Extract links matching 'category.php' (but not matching 'subsection.php')
|
||||
# and follow links from them (since no callback means follow=True by default).
|
||||
|
|
@ -360,7 +360,7 @@ XMLFeedSpider
|
|||
iterator may be useful when parsing XML with bad markup.
|
||||
|
||||
To set the iterator and the tag name, you must define the following class
|
||||
attributes:
|
||||
attributes:
|
||||
|
||||
.. attribute:: iterator
|
||||
|
||||
|
|
@ -396,7 +396,7 @@ XMLFeedSpider
|
|||
attribute.
|
||||
|
||||
Example::
|
||||
|
||||
|
||||
class YourSpider(XMLFeedSpider):
|
||||
|
||||
namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]
|
||||
|
|
|
|||
Loading…
Reference in New Issue