From 51b8feb4ced04db8c1c2c477eaded3e8c25f2651 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Wed, 26 Dec 2012 16:16:53 -0200 Subject: [PATCH] fixed doc typos --- docs/topics/broad-crawls.rst | 2 +- docs/topics/practices.rst | 36 +++++++++++++++++++----------------- 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 09bd63e8a..189620a0b 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -43,7 +43,7 @@ Increase concurrency ==================== Concurrency is the number of requests that are processed in parallel. There is -a global limit a per-domain limit. +a global limit and a per-domain limit. The default global concurrency limit in Scrapy is not suitable for crawling many different domains in parallel, so you will want to increase it. How much diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index bf0b4f18b..1ed670fe5 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -4,21 +4,20 @@ Common Practices ================ -The section documents sommon common practices when using Scrapy. These are -things that don't often fall into other specific sections, or cover many of -them. +This section documents common practices when using Scrapy. These are things +that cover many topics and don't often fall into any other specific section. .. _run-from-script: Run Scrapy from a script ======================== -You can use the :ref:`API ` to run script from a script, instead of +You can use the :ref:`API ` to run Scrapy from a script, instead of the typical way of running Scrapy via ``scrapy crawl``. What follows is a working example of how to do that, using the `testspiders`_ -project as example. Remember that Scrapy is asynchronous so you need run inside -the Twisted reactor. +project as example. Remember that Scrapy is built on top of the Twisted +asynchronous networking library, so you need run it inside the Twisted reactor. :: @@ -36,12 +35,14 @@ the Twisted reactor. log.start() reactor.run() # the script will block here +.. seealso:: `Twisted Reactor Overview`_. + Running multiple spiders in the same process ============================================ By default, Scrapy runs a single spider per process when you run ``scrapy -crawl``. However, Scrapy supports running multiple spiders per process if you -use the :ref:`internal API `. +crawl``. However, Scrapy supports running multiple spiders per process using +the :ref:`internal API `. Here is an example, using the `testspiders`_ project: @@ -65,33 +66,33 @@ Here is an example, using the `testspiders`_ project: log.start() reactor.run() -See also: :ref:`run-from-script`. +.. seealso:: :ref:`run-from-script`. .. _distributed-crawls: Distributed crawls ================== -Scrapy doesn't provide any built-in facility to distribute crawls, however -there are some ways to distribute crawls, depending on what kind of crawling -you do. +Scrapy doesn't provide any built-in facility for running crawls in a distribute +(multi-server) manner. However, there are some ways to distribute crawls, which +vary depending on how you plan to distribute them. If you have many spiders, the obvious way to distribute the load is to setup many Scrapyd instances and distribute spider runs among those. If you instead want to run a single (big) spider through many machines, what -you usually do is to partition the urls to crawl and send them to each separate +you usually do is partition the urls to crawl and send them to each separate spider. Here is a concrete example: -First, you prepare a list of urls to crawl and put them into separate +First, you prepare the list of urls to crawl and put them into separate files/urls:: http://somedomain.com/urls-to-crawl/spider1/part1.list http://somedomain.com/urls-to-crawl/spider1/part2.list http://somedomain.com/urls-to-crawl/spider1/part3.list -Then you would fire a spider run on 3 different Scrapyd servers. The spider -would receive a spider argument ``part`` with the number of the partition to +Then you fire a spider run on 3 different Scrapyd servers. The spider would +receive a (spider) argument ``part`` with the number of the partition to crawl:: curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 @@ -115,7 +116,7 @@ Here are some tips to keep in mind when dealing with these kind of sites: * disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use cookies to spot bot behaviour * use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. -* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites +* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites directly * use a pool of rotating IPs. For example, the free `Tor project`_ or paid services like `ProxyMesh`_ @@ -128,3 +129,4 @@ If you are still unable to prevent your bot getting banned, consider contacting .. _ProxyMesh: http://proxymesh.com/ .. _Google cache: http://www.googleguide.com/cached_pages.html .. _testspiders: https://github.com/scrapinghub/testspiders +.. _Twisted Reactor Overview: http://twistedmatrix.com/documents/current/core/howto/reactor-basics.html