From 1e2ee76df26cd86e8a6ed1b86879aacfce0a44a8 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Wed, 26 Dec 2012 14:02:13 -0200 Subject: [PATCH] add documentation topics: Broad Crawls & Common Practies --- docs/faq.rst | 24 +------ docs/index.rst | 10 ++- docs/topics/broad-crawls.rst | 120 ++++++++++++++++++++++++++++++++ docs/topics/practices.rst | 130 +++++++++++++++++++++++++++++++++++ 4 files changed, 260 insertions(+), 24 deletions(-) create mode 100644 docs/topics/broad-crawls.rst create mode 100644 docs/topics/practices.rst diff --git a/docs/faq.rst b/docs/faq.rst index 70e04c185..490e641cd 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -250,25 +250,7 @@ more info see: :exc:`~scrapy.exceptions.CloseSpider`. How can I prevent my Scrapy bot from getting banned? ---------------------------------------------------- -Some websites implement certain measures to prevent bots from crawling them, -with varying degrees of sophistication. Getting around those measures can be -difficult and tricky, and may sometimes require special infrastructure. Please -consider contacting `commercial support` if in doubt. - -Here are some tips to keep in mind when dealing with these kind of sites: - -* rotate your user agent from a pool of well-known ones from browsers (google - around to get a list of them) -* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use - cookies to spot bot behaviour -* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. -* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites - directly -* use a pool of rotating IPs. For example, the free `Tor project`_ or paid - services like `ProxyMesh`_ - -If you are still unable to prevent your bot getting banned, consider contacting -`commercial support`_. +See :ref:`bans`. Should I use spider arguments or settings to configure my spider? ----------------------------------------------------------------- @@ -287,10 +269,6 @@ log in would be settings, while the url of the section to scrape would be a spider argument. .. _user agents: http://en.wikipedia.org/wiki/User_agent -.. _Google cache: http://www.googleguide.com/cached_pages.html -.. _Tor project: https://www.torproject.org/ -.. _commercial support: http://scrapy.org/support/ .. _LIFO: http://en.wikipedia.org/wiki/LIFO .. _DFO order: http://en.wikipedia.org/wiki/Depth-first_search .. _BFO order: http://en.wikipedia.org/wiki/Breadth-first_search -.. _ProxyMesh: http://proxymesh.com/ diff --git a/docs/index.rst b/docs/index.rst index b32c1c42d..a7503db3e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -130,6 +130,8 @@ Solving specific problems faq topics/debug topics/contracts + topics/practices + topics/broad-crawls topics/firefox topics/firebug topics/leaks @@ -147,7 +149,13 @@ Solving specific problems Learn how to debug common problems of your scrapy spider. :doc:`topics/contracts` - Learn how to use contracts for testing your spiders. + Learn how to use contracts for testing your spiders. + +:doc:`topics/practices` + Get familiar with some Scrapy common practices. + +:doc:`topics/broad-crawls` + Tune Scrapy for crawling a lot domains in parallel. :doc:`topics/firefox` Learn how to scrape with Firefox and some useful add-ons. diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst new file mode 100644 index 000000000..09bd63e8a --- /dev/null +++ b/docs/topics/broad-crawls.rst @@ -0,0 +1,120 @@ +.. _topics-broad-crawls: + +============ +Broad Crawls +============ + +Scrapy defaults are optimized for crawling specific sites. These sites are +often handled by a single Scrapy spider, although this is not necessary or +required (for example, there are generic spiders that handle any given site +thrown at them). + +In addition to this "focused crawl", there is another common type of crawling +which covers a large (potentially unlimited) number of domains, and is only +limited by time or other arbitrary constraint, rather than stopping when the +domain was crawled to completion or when there are no more requests to perform. +These are called "broad crawls" and is the typical crawlers employed by search +engines. + +These are some common properties often found in broad crawls: + +* they crawl many domains (often, unbounded) instead of a specific set of sites + +* they don't necessarily crawl domains to completion, because it would + impractical (or impossible) to do so, and instead limit the crawl by time or + number of pages crawled + +* they are simpler in logic (as opposed to very complex spiders with many + extraction rules) because data is often post-processed in a separate stage + +* they crawl many domains concurrently, which allows them to achieve faster + crawl speeds by not being limited by any particular site constraint (each site + is crawled slowly to respect politeness, but many sites are crawled in + parallel) + +As said above, Scrapy default settings are optimized for focused crawls, not +broad crawls. However, due to its asynchronous architecture, Scrapy is very +well suited for performing fast broad crawls. This page summarize some things +you need to keep in mind when using Scrapy for doing broad crawls, along with +concrete suggestions of Scrapy settings to tune in order to achieve an +efficient broad crawl. + +Increase concurrency +==================== + +Concurrency is the number of requests that are processed in parallel. There is +a global limit a per-domain limit. + +The default global concurrency limit in Scrapy is not suitable for crawling +many different domains in parallel, so you will want to increase it. How much +to increase it will depend on how much CPU you crawler will have available. A +good starting point is ``100``, but the best way to find out is by doing some +trials and identifying at what concurrency your Scrapy process gets CPU +bounded. For optimum performance, You should pick a concurrency where CPU usage +is at 80-90%. + +To increase the global concurrency use:: + + CONCURRENT_REQUESTS = 100 + +Reduce log level +================ + +When doing broad crawls you are often only interested in the crawl rates you +get and any errors found. These stats are reported by Scrapy when using the +``INFO`` log level. In order to save CPU (and log storage requirements) you +should not use ``DEBUG`` log level when preforming large broad crawls in +production. Using ``DEBUG`` level when developing your (broad) crawler may fine +though. + +To set the log level use:: + + LOG_LEVEL = 'INFO' + +Disable cookies +=============== + +Disable cookies unless you *really* need. Cookies are often not needed when +doing broad crawls (search engine crawlers ignore them), and they improve +performance by saving some CPU cycles and reducing the memory footprint of your +Scrapy crawler. + +To disable cookies use:: + + COOKIES_ENABLED = False + +Disable retries +=============== + +Retrying failed HTTP requests can slow down the crawls substantially, specially +when sites causes are very slow (or fail) to respond, thus causing a timeout +error which gets retried many times, unnecessarily, preventing crawler capacity +to be reused for other domains. + +To disable retries use:: + + RETRY_ENABLED = False + +Reduce download timeout +======================= + +Unless you are crawling from a very slow connection (which shouldn't be the +case for broad crawls) reduce the download timeout so that stuck requests are +discarded quickly and free up capacity to process the next ones. + +To reduce the download timeout use:: + + DOWNLOAD_TIMEOUT = 15 + +Disable redirects +================= + +Consider disabling redirects, unless you are interested in following them. When +doing broad crawls it's common to save redirects and resolve them when +revisiting the site at a later crawl. This also help to keep the number of +request constant per crawl batch, otherwise redirect loops may cause the +crawler to dedicate too many resources on any specific domain. + +To disable redirects use:: + + REDIRECT_ENABLED = False diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst new file mode 100644 index 000000000..bf0b4f18b --- /dev/null +++ b/docs/topics/practices.rst @@ -0,0 +1,130 @@ +.. _topics-practices: + +================ +Common Practices +================ + +The section documents sommon common practices when using Scrapy. These are +things that don't often fall into other specific sections, or cover many of +them. + +.. _run-from-script: + +Run Scrapy from a script +======================== + +You can use the :ref:`API ` to run script from a script, instead of +the typical way of running Scrapy via ``scrapy crawl``. + +What follows is a working example of how to do that, using the `testspiders`_ +project as example. Remember that Scrapy is asynchronous so you need run inside +the Twisted reactor. + +:: + + from twisted.internet import reactor + from scrapy.crawler import Crawler + from scrapy.settings import Settings + from scrapy import log + from testspiders.spiders.followall import FollowAllSpider + + spider = FollowAllSpider(domain='scrapinghub.com') + crawler = Crawler(Settings()) + crawler.configure() + crawler.crawl(spider) + crawler.start() + log.start() + reactor.run() # the script will block here + +Running multiple spiders in the same process +============================================ + +By default, Scrapy runs a single spider per process when you run ``scrapy +crawl``. However, Scrapy supports running multiple spiders per process if you +use the :ref:`internal API `. + +Here is an example, using the `testspiders`_ project: + +:: + + from twisted.internet import reactor + from scrapy.crawler import Crawler + from scrapy.settings import Settings + from scrapy import log + from testspiders.spiders.followall import FollowAllSpider + + def setup_crawler(domain): + spider = FollowAllSpider(domain=domain) + crawler = Crawler(Settings()) + crawler.configure() + crawler.crawl(spider) + crawler.start() + + for domain in ['scrapinghub.com', 'insophia.com']: + setup_crawler(domain) + log.start() + reactor.run() + +See also: :ref:`run-from-script`. + +.. _distributed-crawls: + +Distributed crawls +================== + +Scrapy doesn't provide any built-in facility to distribute crawls, however +there are some ways to distribute crawls, depending on what kind of crawling +you do. + +If you have many spiders, the obvious way to distribute the load is to setup +many Scrapyd instances and distribute spider runs among those. + +If you instead want to run a single (big) spider through many machines, what +you usually do is to partition the urls to crawl and send them to each separate +spider. Here is a concrete example: + +First, you prepare a list of urls to crawl and put them into separate +files/urls:: + + http://somedomain.com/urls-to-crawl/spider1/part1.list + http://somedomain.com/urls-to-crawl/spider1/part2.list + http://somedomain.com/urls-to-crawl/spider1/part3.list + +Then you would fire a spider run on 3 different Scrapyd servers. The spider +would receive a spider argument ``part`` with the number of the partition to +crawl:: + + curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 + curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2 + curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3 + +.. _bans: + +Avoiding getting banned +======================= + +Some websites implement certain measures to prevent bots from crawling them, +with varying degrees of sophistication. Getting around those measures can be +difficult and tricky, and may sometimes require special infrastructure. Please +consider contacting `commercial support`_ if in doubt. + +Here are some tips to keep in mind when dealing with these kind of sites: + +* rotate your user agent from a pool of well-known ones from browsers (google + around to get a list of them) +* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use + cookies to spot bot behaviour +* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. +* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites + directly +* use a pool of rotating IPs. For example, the free `Tor project`_ or paid + services like `ProxyMesh`_ + +If you are still unable to prevent your bot getting banned, consider contacting +`commercial support`_. + +.. _Tor project: https://www.torproject.org/ +.. _commercial support: http://scrapy.org/support/ +.. _ProxyMesh: http://proxymesh.com/ +.. _Google cache: http://www.googleguide.com/cached_pages.html +.. _testspiders: https://github.com/scrapinghub/testspiders