fixed doc typos

This commit is contained in:
Pablo Hoffman 2012-12-26 16:16:53 -02:00
parent 1fbb715228
commit 243be84eb8
2 changed files with 20 additions and 18 deletions

View File

@ -43,7 +43,7 @@ Increase concurrency
====================
Concurrency is the number of requests that are processed in parallel. There is
a global limit a per-domain limit.
a global limit and a per-domain limit.
The default global concurrency limit in Scrapy is not suitable for crawling
many different domains in parallel, so you will want to increase it. How much

View File

@ -4,21 +4,20 @@
Common Practices
================
The section documents sommon common practices when using Scrapy. These are
things that don't often fall into other specific sections, or cover many of
them.
This section documents common practices when using Scrapy. These are things
that cover many topics and don't often fall into any other specific section.
.. _run-from-script:
Run Scrapy from a script
========================
You can use the :ref:`API <topics-api>` to run script from a script, instead of
You can use the :ref:`API <topics-api>` to run Scrapy from a script, instead of
the typical way of running Scrapy via ``scrapy crawl``.
What follows is a working example of how to do that, using the `testspiders`_
project as example. Remember that Scrapy is asynchronous so you need run inside
the Twisted reactor.
project as example. Remember that Scrapy is built on top of the Twisted
asynchronous networking library, so you need run it inside the Twisted reactor.
::
@ -36,12 +35,14 @@ the Twisted reactor.
log.start()
reactor.run() # the script will block here
.. seealso:: `Twisted Reactor Overview`_.
Running multiple spiders in the same process
============================================
By default, Scrapy runs a single spider per process when you run ``scrapy
crawl``. However, Scrapy supports running multiple spiders per process if you
use the :ref:`internal API <topics-api>`.
crawl``. However, Scrapy supports running multiple spiders per process using
the :ref:`internal API <topics-api>`.
Here is an example, using the `testspiders`_ project:
@ -65,33 +66,33 @@ Here is an example, using the `testspiders`_ project:
log.start()
reactor.run()
See also: :ref:`run-from-script`.
.. seealso:: :ref:`run-from-script`.
.. _distributed-crawls:
Distributed crawls
==================
Scrapy doesn't provide any built-in facility to distribute crawls, however
there are some ways to distribute crawls, depending on what kind of crawling
you do.
Scrapy doesn't provide any built-in facility for running crawls in a distribute
(multi-server) manner. However, there are some ways to distribute crawls, which
vary depending on how you plan to distribute them.
If you have many spiders, the obvious way to distribute the load is to setup
many Scrapyd instances and distribute spider runs among those.
If you instead want to run a single (big) spider through many machines, what
you usually do is to partition the urls to crawl and send them to each separate
you usually do is partition the urls to crawl and send them to each separate
spider. Here is a concrete example:
First, you prepare a list of urls to crawl and put them into separate
First, you prepare the list of urls to crawl and put them into separate
files/urls::
http://somedomain.com/urls-to-crawl/spider1/part1.list
http://somedomain.com/urls-to-crawl/spider1/part2.list
http://somedomain.com/urls-to-crawl/spider1/part3.list
Then you would fire a spider run on 3 different Scrapyd servers. The spider
would receive a spider argument ``part`` with the number of the partition to
Then you fire a spider run on 3 different Scrapyd servers. The spider would
receive a (spider) argument ``part`` with the number of the partition to
crawl::
curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1
@ -115,7 +116,7 @@ Here are some tips to keep in mind when dealing with these kind of sites:
* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
cookies to spot bot behaviour
* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites
* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
directly
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
services like `ProxyMesh`_
@ -128,3 +129,4 @@ If you are still unable to prevent your bot getting banned, consider contacting
.. _ProxyMesh: http://proxymesh.com/
.. _Google cache: http://www.googleguide.com/cached_pages.html
.. _testspiders: https://github.com/scrapinghub/testspiders
.. _Twisted Reactor Overview: http://twistedmatrix.com/documents/current/core/howto/reactor-basics.html