From 66311db23ebbc6fc7ef9fe313b403eef02499388 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Sat, 4 May 2013 18:19:45 -0300 Subject: [PATCH] mention crawlera in best practices, as a way to deal with bans --- docs/topics/practices.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 1ed670fe5..b892e921e 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -120,6 +120,9 @@ Here are some tips to keep in mind when dealing with these kind of sites: directly * use a pool of rotating IPs. For example, the free `Tor project`_ or paid services like `ProxyMesh`_ +* use a highly distributed downloader that circumvents bans internally, so you + can just focus on parsing clean pages. One example of such downloaders is + `Crawlera`_ If you are still unable to prevent your bot getting banned, consider contacting `commercial support`_. @@ -130,3 +133,4 @@ If you are still unable to prevent your bot getting banned, consider contacting .. _Google cache: http://www.googleguide.com/cached_pages.html .. _testspiders: https://github.com/scrapinghub/testspiders .. _Twisted Reactor Overview: http://twistedmatrix.com/documents/current/core/howto/reactor-basics.html +.. _Crawlera: http://crawlera.com