mirror of https://github.com/scrapy/scrapy.git
add documentation topics: Broad Crawls & Common Practies
This commit is contained in:
Pablo Hoffman
parent
dabb064293
commit
1e2ee76df2
24
docs/faq.rst
24
docs/faq.rst
|
|
@ -250,25 +250,7 @@ more info see: :exc:`~scrapy.exceptions.CloseSpider`.
|
|||
How can I prevent my Scrapy bot from getting banned?
|
||||
----------------------------------------------------
|
||||
|
||||
Some websites implement certain measures to prevent bots from crawling them,
|
||||
with varying degrees of sophistication. Getting around those measures can be
|
||||
difficult and tricky, and may sometimes require special infrastructure. Please
|
||||
consider contacting `commercial support` if in doubt.
|
||||
|
||||
Here are some tips to keep in mind when dealing with these kind of sites:
|
||||
|
||||
* rotate your user agent from a pool of well-known ones from browsers (google
|
||||
around to get a list of them)
|
||||
* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
|
||||
cookies to spot bot behaviour
|
||||
* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
|
||||
* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites
|
||||
directly
|
||||
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
|
||||
services like `ProxyMesh`_
|
||||
|
||||
If you are still unable to prevent your bot getting banned, consider contacting
|
||||
`commercial support`_.
|
||||
See :ref:`bans`.
|
||||
|
||||
Should I use spider arguments or settings to configure my spider?
|
||||
-----------------------------------------------------------------
|
||||
|
|
@ -287,10 +269,6 @@ log in would be settings, while the url of the section to scrape would be a
|
|||
spider argument.
|
||||
|
||||
.. _user agents: http://en.wikipedia.org/wiki/User_agent
|
||||
.. _Google cache: http://www.googleguide.com/cached_pages.html
|
||||
.. _Tor project: https://www.torproject.org/
|
||||
.. _commercial support: http://scrapy.org/support/
|
||||
.. _LIFO: http://en.wikipedia.org/wiki/LIFO
|
||||
.. _DFO order: http://en.wikipedia.org/wiki/Depth-first_search
|
||||
.. _BFO order: http://en.wikipedia.org/wiki/Breadth-first_search
|
||||
.. _ProxyMesh: http://proxymesh.com/
|
||||
|
|
|
|||
|
|
@ -130,6 +130,8 @@ Solving specific problems
|
|||
faq
|
||||
topics/debug
|
||||
topics/contracts
|
||||
topics/practices
|
||||
topics/broad-crawls
|
||||
topics/firefox
|
||||
topics/firebug
|
||||
topics/leaks
|
||||
|
|
@ -147,7 +149,13 @@ Solving specific problems
|
|||
Learn how to debug common problems of your scrapy spider.
|
||||
|
||||
:doc:`topics/contracts`
|
||||
Learn how to use contracts for testing your spiders.
|
||||
Learn how to use contracts for testing your spiders.
|
||||
|
||||
:doc:`topics/practices`
|
||||
Get familiar with some Scrapy common practices.
|
||||
|
||||
:doc:`topics/broad-crawls`
|
||||
Tune Scrapy for crawling a lot domains in parallel.
|
||||
|
||||
:doc:`topics/firefox`
|
||||
Learn how to scrape with Firefox and some useful add-ons.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,120 @@
|
|||
.. _topics-broad-crawls:
|
||||
|
||||
============
|
||||
Broad Crawls
|
||||
============
|
||||
|
||||
Scrapy defaults are optimized for crawling specific sites. These sites are
|
||||
often handled by a single Scrapy spider, although this is not necessary or
|
||||
required (for example, there are generic spiders that handle any given site
|
||||
thrown at them).
|
||||
|
||||
In addition to this "focused crawl", there is another common type of crawling
|
||||
which covers a large (potentially unlimited) number of domains, and is only
|
||||
limited by time or other arbitrary constraint, rather than stopping when the
|
||||
domain was crawled to completion or when there are no more requests to perform.
|
||||
These are called "broad crawls" and is the typical crawlers employed by search
|
||||
engines.
|
||||
|
||||
These are some common properties often found in broad crawls:
|
||||
|
||||
* they crawl many domains (often, unbounded) instead of a specific set of sites
|
||||
|
||||
* they don't necessarily crawl domains to completion, because it would
|
||||
impractical (or impossible) to do so, and instead limit the crawl by time or
|
||||
number of pages crawled
|
||||
|
||||
* they are simpler in logic (as opposed to very complex spiders with many
|
||||
extraction rules) because data is often post-processed in a separate stage
|
||||
|
||||
* they crawl many domains concurrently, which allows them to achieve faster
|
||||
crawl speeds by not being limited by any particular site constraint (each site
|
||||
is crawled slowly to respect politeness, but many sites are crawled in
|
||||
parallel)
|
||||
|
||||
As said above, Scrapy default settings are optimized for focused crawls, not
|
||||
broad crawls. However, due to its asynchronous architecture, Scrapy is very
|
||||
well suited for performing fast broad crawls. This page summarize some things
|
||||
you need to keep in mind when using Scrapy for doing broad crawls, along with
|
||||
concrete suggestions of Scrapy settings to tune in order to achieve an
|
||||
efficient broad crawl.
|
||||
|
||||
Increase concurrency
|
||||
====================
|
||||
|
||||
Concurrency is the number of requests that are processed in parallel. There is
|
||||
a global limit a per-domain limit.
|
||||
|
||||
The default global concurrency limit in Scrapy is not suitable for crawling
|
||||
many different domains in parallel, so you will want to increase it. How much
|
||||
to increase it will depend on how much CPU you crawler will have available. A
|
||||
good starting point is ``100``, but the best way to find out is by doing some
|
||||
trials and identifying at what concurrency your Scrapy process gets CPU
|
||||
bounded. For optimum performance, You should pick a concurrency where CPU usage
|
||||
is at 80-90%.
|
||||
|
||||
To increase the global concurrency use::
|
||||
|
||||
CONCURRENT_REQUESTS = 100
|
||||
|
||||
Reduce log level
|
||||
================
|
||||
|
||||
When doing broad crawls you are often only interested in the crawl rates you
|
||||
get and any errors found. These stats are reported by Scrapy when using the
|
||||
``INFO`` log level. In order to save CPU (and log storage requirements) you
|
||||
should not use ``DEBUG`` log level when preforming large broad crawls in
|
||||
production. Using ``DEBUG`` level when developing your (broad) crawler may fine
|
||||
though.
|
||||
|
||||
To set the log level use::
|
||||
|
||||
LOG_LEVEL = 'INFO'
|
||||
|
||||
Disable cookies
|
||||
===============
|
||||
|
||||
Disable cookies unless you *really* need. Cookies are often not needed when
|
||||
doing broad crawls (search engine crawlers ignore them), and they improve
|
||||
performance by saving some CPU cycles and reducing the memory footprint of your
|
||||
Scrapy crawler.
|
||||
|
||||
To disable cookies use::
|
||||
|
||||
COOKIES_ENABLED = False
|
||||
|
||||
Disable retries
|
||||
===============
|
||||
|
||||
Retrying failed HTTP requests can slow down the crawls substantially, specially
|
||||
when sites causes are very slow (or fail) to respond, thus causing a timeout
|
||||
error which gets retried many times, unnecessarily, preventing crawler capacity
|
||||
to be reused for other domains.
|
||||
|
||||
To disable retries use::
|
||||
|
||||
RETRY_ENABLED = False
|
||||
|
||||
Reduce download timeout
|
||||
=======================
|
||||
|
||||
Unless you are crawling from a very slow connection (which shouldn't be the
|
||||
case for broad crawls) reduce the download timeout so that stuck requests are
|
||||
discarded quickly and free up capacity to process the next ones.
|
||||
|
||||
To reduce the download timeout use::
|
||||
|
||||
DOWNLOAD_TIMEOUT = 15
|
||||
|
||||
Disable redirects
|
||||
=================
|
||||
|
||||
Consider disabling redirects, unless you are interested in following them. When
|
||||
doing broad crawls it's common to save redirects and resolve them when
|
||||
revisiting the site at a later crawl. This also help to keep the number of
|
||||
request constant per crawl batch, otherwise redirect loops may cause the
|
||||
crawler to dedicate too many resources on any specific domain.
|
||||
|
||||
To disable redirects use::
|
||||
|
||||
REDIRECT_ENABLED = False
|
||||
|
|
@ -0,0 +1,130 @@
|
|||
.. _topics-practices:
|
||||
|
||||
================
|
||||
Common Practices
|
||||
================
|
||||
|
||||
The section documents sommon common practices when using Scrapy. These are
|
||||
things that don't often fall into other specific sections, or cover many of
|
||||
them.
|
||||
|
||||
.. _run-from-script:
|
||||
|
||||
Run Scrapy from a script
|
||||
========================
|
||||
|
||||
You can use the :ref:`API <topics-api>` to run script from a script, instead of
|
||||
the typical way of running Scrapy via ``scrapy crawl``.
|
||||
|
||||
What follows is a working example of how to do that, using the `testspiders`_
|
||||
project as example. Remember that Scrapy is asynchronous so you need run inside
|
||||
the Twisted reactor.
|
||||
|
||||
::
|
||||
|
||||
from twisted.internet import reactor
|
||||
from scrapy.crawler import Crawler
|
||||
from scrapy.settings import Settings
|
||||
from scrapy import log
|
||||
from testspiders.spiders.followall import FollowAllSpider
|
||||
|
||||
spider = FollowAllSpider(domain='scrapinghub.com')
|
||||
crawler = Crawler(Settings())
|
||||
crawler.configure()
|
||||
crawler.crawl(spider)
|
||||
crawler.start()
|
||||
log.start()
|
||||
reactor.run() # the script will block here
|
||||
|
||||
Running multiple spiders in the same process
|
||||
============================================
|
||||
|
||||
By default, Scrapy runs a single spider per process when you run ``scrapy
|
||||
crawl``. However, Scrapy supports running multiple spiders per process if you
|
||||
use the :ref:`internal API <topics-api>`.
|
||||
|
||||
Here is an example, using the `testspiders`_ project:
|
||||
|
||||
::
|
||||
|
||||
from twisted.internet import reactor
|
||||
from scrapy.crawler import Crawler
|
||||
from scrapy.settings import Settings
|
||||
from scrapy import log
|
||||
from testspiders.spiders.followall import FollowAllSpider
|
||||
|
||||
def setup_crawler(domain):
|
||||
spider = FollowAllSpider(domain=domain)
|
||||
crawler = Crawler(Settings())
|
||||
crawler.configure()
|
||||
crawler.crawl(spider)
|
||||
crawler.start()
|
||||
|
||||
for domain in ['scrapinghub.com', 'insophia.com']:
|
||||
setup_crawler(domain)
|
||||
log.start()
|
||||
reactor.run()
|
||||
|
||||
See also: :ref:`run-from-script`.
|
||||
|
||||
.. _distributed-crawls:
|
||||
|
||||
Distributed crawls
|
||||
==================
|
||||
|
||||
Scrapy doesn't provide any built-in facility to distribute crawls, however
|
||||
there are some ways to distribute crawls, depending on what kind of crawling
|
||||
you do.
|
||||
|
||||
If you have many spiders, the obvious way to distribute the load is to setup
|
||||
many Scrapyd instances and distribute spider runs among those.
|
||||
|
||||
If you instead want to run a single (big) spider through many machines, what
|
||||
you usually do is to partition the urls to crawl and send them to each separate
|
||||
spider. Here is a concrete example:
|
||||
|
||||
First, you prepare a list of urls to crawl and put them into separate
|
||||
files/urls::
|
||||
|
||||
http://somedomain.com/urls-to-crawl/spider1/part1.list
|
||||
http://somedomain.com/urls-to-crawl/spider1/part2.list
|
||||
http://somedomain.com/urls-to-crawl/spider1/part3.list
|
||||
|
||||
Then you would fire a spider run on 3 different Scrapyd servers. The spider
|
||||
would receive a spider argument ``part`` with the number of the partition to
|
||||
crawl::
|
||||
|
||||
curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1
|
||||
curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
|
||||
curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
|
||||
|
||||
.. _bans:
|
||||
|
||||
Avoiding getting banned
|
||||
=======================
|
||||
|
||||
Some websites implement certain measures to prevent bots from crawling them,
|
||||
with varying degrees of sophistication. Getting around those measures can be
|
||||
difficult and tricky, and may sometimes require special infrastructure. Please
|
||||
consider contacting `commercial support`_ if in doubt.
|
||||
|
||||
Here are some tips to keep in mind when dealing with these kind of sites:
|
||||
|
||||
* rotate your user agent from a pool of well-known ones from browsers (google
|
||||
around to get a list of them)
|
||||
* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
|
||||
cookies to spot bot behaviour
|
||||
* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
|
||||
* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites
|
||||
directly
|
||||
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
|
||||
services like `ProxyMesh`_
|
||||
|
||||
If you are still unable to prevent your bot getting banned, consider contacting
|
||||
`commercial support`_.
|
||||
|
||||
.. _Tor project: https://www.torproject.org/
|
||||
.. _commercial support: http://scrapy.org/support/
|
||||
.. _ProxyMesh: http://proxymesh.com/
|
||||
.. _Google cache: http://www.googleguide.com/cached_pages.html
|
||||
.. _testspiders: https://github.com/scrapinghub/testspiders
|
||||
Loading…
Reference in New Issue