diff --git a/docs/faq.rst b/docs/faq.rst index 1e969af69..7085a585b 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -245,3 +245,29 @@ How can I instruct a spider to stop itself? Raise the :exc:`~scrapy.exceptions.CloseSpider` exception from a callback. For more info see: :exc:`~scrapy.exceptions.CloseSpider`. + +How can I prevent my Scrapy bot from getting banned? +---------------------------------------------------- + +Some websites implement certain measures to prevent bots from crawling them, +with varying degrees of sophistication. Getting around those measures can be +difficult and tricky, and may sometimes require special infrastructure. + +Here are some tips to keep in mind when dealing with these kind of sites: + +* rotate your user agent from a pool of well-known ones from browsers (google + around to get a list of them) +* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use + cookies to spot bot behaviour +* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. +* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites + directly +* use a pool of rotating IPs. For example, the free `Tor project`_. + +If you are still unable to prevent your bot getting banned, consider contacting +`commercial support`_. + +.. _user agents: http://en.wikipedia.org/wiki/User_agent +.. _Google cache: http://www.googleguide.com/cached_pages.html +.. _Tor project: https://www.torproject.org/ +.. _commercial support: http://scrapy.org/support/