mirror of https://github.com/scrapy/scrapy.git
fixed doc typos
This commit is contained in:
parent
1fbb715228
commit
243be84eb8
|
|
@ -43,7 +43,7 @@ Increase concurrency
|
|||
====================
|
||||
|
||||
Concurrency is the number of requests that are processed in parallel. There is
|
||||
a global limit a per-domain limit.
|
||||
a global limit and a per-domain limit.
|
||||
|
||||
The default global concurrency limit in Scrapy is not suitable for crawling
|
||||
many different domains in parallel, so you will want to increase it. How much
|
||||
|
|
|
|||
|
|
@ -4,21 +4,20 @@
|
|||
Common Practices
|
||||
================
|
||||
|
||||
The section documents sommon common practices when using Scrapy. These are
|
||||
things that don't often fall into other specific sections, or cover many of
|
||||
them.
|
||||
This section documents common practices when using Scrapy. These are things
|
||||
that cover many topics and don't often fall into any other specific section.
|
||||
|
||||
.. _run-from-script:
|
||||
|
||||
Run Scrapy from a script
|
||||
========================
|
||||
|
||||
You can use the :ref:`API <topics-api>` to run script from a script, instead of
|
||||
You can use the :ref:`API <topics-api>` to run Scrapy from a script, instead of
|
||||
the typical way of running Scrapy via ``scrapy crawl``.
|
||||
|
||||
What follows is a working example of how to do that, using the `testspiders`_
|
||||
project as example. Remember that Scrapy is asynchronous so you need run inside
|
||||
the Twisted reactor.
|
||||
project as example. Remember that Scrapy is built on top of the Twisted
|
||||
asynchronous networking library, so you need run it inside the Twisted reactor.
|
||||
|
||||
::
|
||||
|
||||
|
|
@ -36,12 +35,14 @@ the Twisted reactor.
|
|||
log.start()
|
||||
reactor.run() # the script will block here
|
||||
|
||||
.. seealso:: `Twisted Reactor Overview`_.
|
||||
|
||||
Running multiple spiders in the same process
|
||||
============================================
|
||||
|
||||
By default, Scrapy runs a single spider per process when you run ``scrapy
|
||||
crawl``. However, Scrapy supports running multiple spiders per process if you
|
||||
use the :ref:`internal API <topics-api>`.
|
||||
crawl``. However, Scrapy supports running multiple spiders per process using
|
||||
the :ref:`internal API <topics-api>`.
|
||||
|
||||
Here is an example, using the `testspiders`_ project:
|
||||
|
||||
|
|
@ -65,33 +66,33 @@ Here is an example, using the `testspiders`_ project:
|
|||
log.start()
|
||||
reactor.run()
|
||||
|
||||
See also: :ref:`run-from-script`.
|
||||
.. seealso:: :ref:`run-from-script`.
|
||||
|
||||
.. _distributed-crawls:
|
||||
|
||||
Distributed crawls
|
||||
==================
|
||||
|
||||
Scrapy doesn't provide any built-in facility to distribute crawls, however
|
||||
there are some ways to distribute crawls, depending on what kind of crawling
|
||||
you do.
|
||||
Scrapy doesn't provide any built-in facility for running crawls in a distribute
|
||||
(multi-server) manner. However, there are some ways to distribute crawls, which
|
||||
vary depending on how you plan to distribute them.
|
||||
|
||||
If you have many spiders, the obvious way to distribute the load is to setup
|
||||
many Scrapyd instances and distribute spider runs among those.
|
||||
|
||||
If you instead want to run a single (big) spider through many machines, what
|
||||
you usually do is to partition the urls to crawl and send them to each separate
|
||||
you usually do is partition the urls to crawl and send them to each separate
|
||||
spider. Here is a concrete example:
|
||||
|
||||
First, you prepare a list of urls to crawl and put them into separate
|
||||
First, you prepare the list of urls to crawl and put them into separate
|
||||
files/urls::
|
||||
|
||||
http://somedomain.com/urls-to-crawl/spider1/part1.list
|
||||
http://somedomain.com/urls-to-crawl/spider1/part2.list
|
||||
http://somedomain.com/urls-to-crawl/spider1/part3.list
|
||||
|
||||
Then you would fire a spider run on 3 different Scrapyd servers. The spider
|
||||
would receive a spider argument ``part`` with the number of the partition to
|
||||
Then you fire a spider run on 3 different Scrapyd servers. The spider would
|
||||
receive a (spider) argument ``part`` with the number of the partition to
|
||||
crawl::
|
||||
|
||||
curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1
|
||||
|
|
@ -115,7 +116,7 @@ Here are some tips to keep in mind when dealing with these kind of sites:
|
|||
* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
|
||||
cookies to spot bot behaviour
|
||||
* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
|
||||
* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites
|
||||
* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
|
||||
directly
|
||||
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
|
||||
services like `ProxyMesh`_
|
||||
|
|
@ -128,3 +129,4 @@ If you are still unable to prevent your bot getting banned, consider contacting
|
|||
.. _ProxyMesh: http://proxymesh.com/
|
||||
.. _Google cache: http://www.googleguide.com/cached_pages.html
|
||||
.. _testspiders: https://github.com/scrapinghub/testspiders
|
||||
.. _Twisted Reactor Overview: http://twistedmatrix.com/documents/current/core/howto/reactor-basics.html
|
||||
|
|
|
|||
Loading…
Reference in New Issue