diff --git a/docs/faq.rst b/docs/faq.rst index 16ce4f1e2..4b8c05d8e 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -252,7 +252,8 @@ How can I prevent my Scrapy bot from getting banned? Some websites implement certain measures to prevent bots from crawling them, with varying degrees of sophistication. Getting around those measures can be -difficult and tricky, and may sometimes require special infrastructure. +difficult and tricky, and may sometimes require special infrastructure. Please +consider contacting `commercial support` if in doubt. Here are some tips to keep in mind when dealing with these kind of sites: @@ -263,7 +264,8 @@ Here are some tips to keep in mind when dealing with these kind of sites: * use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. * is possible, use `Google cache`_ to fetch pages, instead of hitting the sites directly -* use a pool of rotating IPs. For example, the free `Tor project`_. +* use a pool of rotating IPs. For example, the free `Tor project`_ or paid + services like `ProxyMesh`_ or `ProxyHub`_. If you are still unable to prevent your bot getting banned, consider contacting `commercial support`_. @@ -275,3 +277,5 @@ If you are still unable to prevent your bot getting banned, consider contacting .. _LIFO: http://en.wikipedia.org/wiki/LIFO .. _DFO order: http://en.wikipedia.org/wiki/Depth-first_search .. _BFO order: http://en.wikipedia.org/wiki/Breadth-first_search +.. _ProxyMesh: http://proxymesh.com/ +.. _ProxyHub: http://scrapinghub.com/proxyhub.html