From e1be9c01bc6f2522395036d8724ed33db1f0b087 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Fri, 8 Jun 2012 18:33:53 -0300 Subject: [PATCH] updated FAQ about bot bans --- docs/faq.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index 16ce4f1e2..4b8c05d8e 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -252,7 +252,8 @@ How can I prevent my Scrapy bot from getting banned? Some websites implement certain measures to prevent bots from crawling them, with varying degrees of sophistication. Getting around those measures can be -difficult and tricky, and may sometimes require special infrastructure. +difficult and tricky, and may sometimes require special infrastructure. Please +consider contacting `commercial support` if in doubt. Here are some tips to keep in mind when dealing with these kind of sites: @@ -263,7 +264,8 @@ Here are some tips to keep in mind when dealing with these kind of sites: * use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. * is possible, use `Google cache`_ to fetch pages, instead of hitting the sites directly -* use a pool of rotating IPs. For example, the free `Tor project`_. +* use a pool of rotating IPs. For example, the free `Tor project`_ or paid + services like `ProxyMesh`_ or `ProxyHub`_. If you are still unable to prevent your bot getting banned, consider contacting `commercial support`_. @@ -275,3 +277,5 @@ If you are still unable to prevent your bot getting banned, consider contacting .. _LIFO: http://en.wikipedia.org/wiki/LIFO .. _DFO order: http://en.wikipedia.org/wiki/Depth-first_search .. _BFO order: http://en.wikipedia.org/wiki/Breadth-first_search +.. _ProxyMesh: http://proxymesh.com/ +.. _ProxyHub: http://scrapinghub.com/proxyhub.html