From 1e2ee76df26cd86e8a6ed1b86879aacfce0a44a8 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Wed, 26 Dec 2012 14:02:13 -0200 Subject: [PATCH 1/2] add documentation topics: Broad Crawls & Common Practies --- docs/faq.rst | 24 +------ docs/index.rst | 10 ++- docs/topics/broad-crawls.rst | 120 ++++++++++++++++++++++++++++++++ docs/topics/practices.rst | 130 +++++++++++++++++++++++++++++++++++ 4 files changed, 260 insertions(+), 24 deletions(-) create mode 100644 docs/topics/broad-crawls.rst create mode 100644 docs/topics/practices.rst diff --git a/docs/faq.rst b/docs/faq.rst index 70e04c185..490e641cd 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -250,25 +250,7 @@ more info see: :exc:`~scrapy.exceptions.CloseSpider`. How can I prevent my Scrapy bot from getting banned? ---------------------------------------------------- -Some websites implement certain measures to prevent bots from crawling them, -with varying degrees of sophistication. Getting around those measures can be -difficult and tricky, and may sometimes require special infrastructure. Please -consider contacting `commercial support` if in doubt. - -Here are some tips to keep in mind when dealing with these kind of sites: - -* rotate your user agent from a pool of well-known ones from browsers (google - around to get a list of them) -* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use - cookies to spot bot behaviour -* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. -* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites - directly -* use a pool of rotating IPs. For example, the free `Tor project`_ or paid - services like `ProxyMesh`_ - -If you are still unable to prevent your bot getting banned, consider contacting -`commercial support`_. +See :ref:`bans`. Should I use spider arguments or settings to configure my spider? ----------------------------------------------------------------- @@ -287,10 +269,6 @@ log in would be settings, while the url of the section to scrape would be a spider argument. .. _user agents: http://en.wikipedia.org/wiki/User_agent -.. _Google cache: http://www.googleguide.com/cached_pages.html -.. _Tor project: https://www.torproject.org/ -.. _commercial support: http://scrapy.org/support/ .. _LIFO: http://en.wikipedia.org/wiki/LIFO .. _DFO order: http://en.wikipedia.org/wiki/Depth-first_search .. _BFO order: http://en.wikipedia.org/wiki/Breadth-first_search -.. _ProxyMesh: http://proxymesh.com/ diff --git a/docs/index.rst b/docs/index.rst index b32c1c42d..a7503db3e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -130,6 +130,8 @@ Solving specific problems faq topics/debug topics/contracts + topics/practices + topics/broad-crawls topics/firefox topics/firebug topics/leaks @@ -147,7 +149,13 @@ Solving specific problems Learn how to debug common problems of your scrapy spider. :doc:`topics/contracts` - Learn how to use contracts for testing your spiders. + Learn how to use contracts for testing your spiders. + +:doc:`topics/practices` + Get familiar with some Scrapy common practices. + +:doc:`topics/broad-crawls` + Tune Scrapy for crawling a lot domains in parallel. :doc:`topics/firefox` Learn how to scrape with Firefox and some useful add-ons. diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst new file mode 100644 index 000000000..09bd63e8a --- /dev/null +++ b/docs/topics/broad-crawls.rst @@ -0,0 +1,120 @@ +.. _topics-broad-crawls: + +============ +Broad Crawls +============ + +Scrapy defaults are optimized for crawling specific sites. These sites are +often handled by a single Scrapy spider, although this is not necessary or +required (for example, there are generic spiders that handle any given site +thrown at them). + +In addition to this "focused crawl", there is another common type of crawling +which covers a large (potentially unlimited) number of domains, and is only +limited by time or other arbitrary constraint, rather than stopping when the +domain was crawled to completion or when there are no more requests to perform. +These are called "broad crawls" and is the typical crawlers employed by search +engines. + +These are some common properties often found in broad crawls: + +* they crawl many domains (often, unbounded) instead of a specific set of sites + +* they don't necessarily crawl domains to completion, because it would + impractical (or impossible) to do so, and instead limit the crawl by time or + number of pages crawled + +* they are simpler in logic (as opposed to very complex spiders with many + extraction rules) because data is often post-processed in a separate stage + +* they crawl many domains concurrently, which allows them to achieve faster + crawl speeds by not being limited by any particular site constraint (each site + is crawled slowly to respect politeness, but many sites are crawled in + parallel) + +As said above, Scrapy default settings are optimized for focused crawls, not +broad crawls. However, due to its asynchronous architecture, Scrapy is very +well suited for performing fast broad crawls. This page summarize some things +you need to keep in mind when using Scrapy for doing broad crawls, along with +concrete suggestions of Scrapy settings to tune in order to achieve an +efficient broad crawl. + +Increase concurrency +==================== + +Concurrency is the number of requests that are processed in parallel. There is +a global limit a per-domain limit. + +The default global concurrency limit in Scrapy is not suitable for crawling +many different domains in parallel, so you will want to increase it. How much +to increase it will depend on how much CPU you crawler will have available. A +good starting point is ``100``, but the best way to find out is by doing some +trials and identifying at what concurrency your Scrapy process gets CPU +bounded. For optimum performance, You should pick a concurrency where CPU usage +is at 80-90%. + +To increase the global concurrency use:: + + CONCURRENT_REQUESTS = 100 + +Reduce log level +================ + +When doing broad crawls you are often only interested in the crawl rates you +get and any errors found. These stats are reported by Scrapy when using the +``INFO`` log level. In order to save CPU (and log storage requirements) you +should not use ``DEBUG`` log level when preforming large broad crawls in +production. Using ``DEBUG`` level when developing your (broad) crawler may fine +though. + +To set the log level use:: + + LOG_LEVEL = 'INFO' + +Disable cookies +=============== + +Disable cookies unless you *really* need. Cookies are often not needed when +doing broad crawls (search engine crawlers ignore them), and they improve +performance by saving some CPU cycles and reducing the memory footprint of your +Scrapy crawler. + +To disable cookies use:: + + COOKIES_ENABLED = False + +Disable retries +=============== + +Retrying failed HTTP requests can slow down the crawls substantially, specially +when sites causes are very slow (or fail) to respond, thus causing a timeout +error which gets retried many times, unnecessarily, preventing crawler capacity +to be reused for other domains. + +To disable retries use:: + + RETRY_ENABLED = False + +Reduce download timeout +======================= + +Unless you are crawling from a very slow connection (which shouldn't be the +case for broad crawls) reduce the download timeout so that stuck requests are +discarded quickly and free up capacity to process the next ones. + +To reduce the download timeout use:: + + DOWNLOAD_TIMEOUT = 15 + +Disable redirects +================= + +Consider disabling redirects, unless you are interested in following them. When +doing broad crawls it's common to save redirects and resolve them when +revisiting the site at a later crawl. This also help to keep the number of +request constant per crawl batch, otherwise redirect loops may cause the +crawler to dedicate too many resources on any specific domain. + +To disable redirects use:: + + REDIRECT_ENABLED = False diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst new file mode 100644 index 000000000..bf0b4f18b --- /dev/null +++ b/docs/topics/practices.rst @@ -0,0 +1,130 @@ +.. _topics-practices: + +================ +Common Practices +================ + +The section documents sommon common practices when using Scrapy. These are +things that don't often fall into other specific sections, or cover many of +them. + +.. _run-from-script: + +Run Scrapy from a script +======================== + +You can use the :ref:`API ` to run script from a script, instead of +the typical way of running Scrapy via ``scrapy crawl``. + +What follows is a working example of how to do that, using the `testspiders`_ +project as example. Remember that Scrapy is asynchronous so you need run inside +the Twisted reactor. + +:: + + from twisted.internet import reactor + from scrapy.crawler import Crawler + from scrapy.settings import Settings + from scrapy import log + from testspiders.spiders.followall import FollowAllSpider + + spider = FollowAllSpider(domain='scrapinghub.com') + crawler = Crawler(Settings()) + crawler.configure() + crawler.crawl(spider) + crawler.start() + log.start() + reactor.run() # the script will block here + +Running multiple spiders in the same process +============================================ + +By default, Scrapy runs a single spider per process when you run ``scrapy +crawl``. However, Scrapy supports running multiple spiders per process if you +use the :ref:`internal API `. + +Here is an example, using the `testspiders`_ project: + +:: + + from twisted.internet import reactor + from scrapy.crawler import Crawler + from scrapy.settings import Settings + from scrapy import log + from testspiders.spiders.followall import FollowAllSpider + + def setup_crawler(domain): + spider = FollowAllSpider(domain=domain) + crawler = Crawler(Settings()) + crawler.configure() + crawler.crawl(spider) + crawler.start() + + for domain in ['scrapinghub.com', 'insophia.com']: + setup_crawler(domain) + log.start() + reactor.run() + +See also: :ref:`run-from-script`. + +.. _distributed-crawls: + +Distributed crawls +================== + +Scrapy doesn't provide any built-in facility to distribute crawls, however +there are some ways to distribute crawls, depending on what kind of crawling +you do. + +If you have many spiders, the obvious way to distribute the load is to setup +many Scrapyd instances and distribute spider runs among those. + +If you instead want to run a single (big) spider through many machines, what +you usually do is to partition the urls to crawl and send them to each separate +spider. Here is a concrete example: + +First, you prepare a list of urls to crawl and put them into separate +files/urls:: + + http://somedomain.com/urls-to-crawl/spider1/part1.list + http://somedomain.com/urls-to-crawl/spider1/part2.list + http://somedomain.com/urls-to-crawl/spider1/part3.list + +Then you would fire a spider run on 3 different Scrapyd servers. The spider +would receive a spider argument ``part`` with the number of the partition to +crawl:: + + curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 + curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2 + curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3 + +.. _bans: + +Avoiding getting banned +======================= + +Some websites implement certain measures to prevent bots from crawling them, +with varying degrees of sophistication. Getting around those measures can be +difficult and tricky, and may sometimes require special infrastructure. Please +consider contacting `commercial support`_ if in doubt. + +Here are some tips to keep in mind when dealing with these kind of sites: + +* rotate your user agent from a pool of well-known ones from browsers (google + around to get a list of them) +* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use + cookies to spot bot behaviour +* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. +* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites + directly +* use a pool of rotating IPs. For example, the free `Tor project`_ or paid + services like `ProxyMesh`_ + +If you are still unable to prevent your bot getting banned, consider contacting +`commercial support`_. + +.. _Tor project: https://www.torproject.org/ +.. _commercial support: http://scrapy.org/support/ +.. _ProxyMesh: http://proxymesh.com/ +.. _Google cache: http://www.googleguide.com/cached_pages.html +.. _testspiders: https://github.com/scrapinghub/testspiders From 51b8feb4ced04db8c1c2c477eaded3e8c25f2651 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Wed, 26 Dec 2012 16:16:53 -0200 Subject: [PATCH 2/2] fixed doc typos --- docs/topics/broad-crawls.rst | 2 +- docs/topics/practices.rst | 36 +++++++++++++++++++----------------- 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 09bd63e8a..189620a0b 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -43,7 +43,7 @@ Increase concurrency ==================== Concurrency is the number of requests that are processed in parallel. There is -a global limit a per-domain limit. +a global limit and a per-domain limit. The default global concurrency limit in Scrapy is not suitable for crawling many different domains in parallel, so you will want to increase it. How much diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index bf0b4f18b..1ed670fe5 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -4,21 +4,20 @@ Common Practices ================ -The section documents sommon common practices when using Scrapy. These are -things that don't often fall into other specific sections, or cover many of -them. +This section documents common practices when using Scrapy. These are things +that cover many topics and don't often fall into any other specific section. .. _run-from-script: Run Scrapy from a script ======================== -You can use the :ref:`API ` to run script from a script, instead of +You can use the :ref:`API ` to run Scrapy from a script, instead of the typical way of running Scrapy via ``scrapy crawl``. What follows is a working example of how to do that, using the `testspiders`_ -project as example. Remember that Scrapy is asynchronous so you need run inside -the Twisted reactor. +project as example. Remember that Scrapy is built on top of the Twisted +asynchronous networking library, so you need run it inside the Twisted reactor. :: @@ -36,12 +35,14 @@ the Twisted reactor. log.start() reactor.run() # the script will block here +.. seealso:: `Twisted Reactor Overview`_. + Running multiple spiders in the same process ============================================ By default, Scrapy runs a single spider per process when you run ``scrapy -crawl``. However, Scrapy supports running multiple spiders per process if you -use the :ref:`internal API `. +crawl``. However, Scrapy supports running multiple spiders per process using +the :ref:`internal API `. Here is an example, using the `testspiders`_ project: @@ -65,33 +66,33 @@ Here is an example, using the `testspiders`_ project: log.start() reactor.run() -See also: :ref:`run-from-script`. +.. seealso:: :ref:`run-from-script`. .. _distributed-crawls: Distributed crawls ================== -Scrapy doesn't provide any built-in facility to distribute crawls, however -there are some ways to distribute crawls, depending on what kind of crawling -you do. +Scrapy doesn't provide any built-in facility for running crawls in a distribute +(multi-server) manner. However, there are some ways to distribute crawls, which +vary depending on how you plan to distribute them. If you have many spiders, the obvious way to distribute the load is to setup many Scrapyd instances and distribute spider runs among those. If you instead want to run a single (big) spider through many machines, what -you usually do is to partition the urls to crawl and send them to each separate +you usually do is partition the urls to crawl and send them to each separate spider. Here is a concrete example: -First, you prepare a list of urls to crawl and put them into separate +First, you prepare the list of urls to crawl and put them into separate files/urls:: http://somedomain.com/urls-to-crawl/spider1/part1.list http://somedomain.com/urls-to-crawl/spider1/part2.list http://somedomain.com/urls-to-crawl/spider1/part3.list -Then you would fire a spider run on 3 different Scrapyd servers. The spider -would receive a spider argument ``part`` with the number of the partition to +Then you fire a spider run on 3 different Scrapyd servers. The spider would +receive a (spider) argument ``part`` with the number of the partition to crawl:: curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 @@ -115,7 +116,7 @@ Here are some tips to keep in mind when dealing with these kind of sites: * disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use cookies to spot bot behaviour * use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. -* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites +* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites directly * use a pool of rotating IPs. For example, the free `Tor project`_ or paid services like `ProxyMesh`_ @@ -128,3 +129,4 @@ If you are still unable to prevent your bot getting banned, consider contacting .. _ProxyMesh: http://proxymesh.com/ .. _Google cache: http://www.googleguide.com/cached_pages.html .. _testspiders: https://github.com/scrapinghub/testspiders +.. _Twisted Reactor Overview: http://twistedmatrix.com/documents/current/core/howto/reactor-basics.html