RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

updated FAQ about bot bans

This commit is contained in:
Pablo Hoffman 2012-06-08 18:33:53 -03:00
parent 3d6edc2f40
commit e1be9c01bc
1 changed files with 6 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/faq.rst
View File

@ -252,7 +252,8 @@ How can I prevent my Scrapy bot from getting banned?
Some websites implement certain measures to prevent bots from crawling them,
with varying degrees of sophistication. Getting around those measures can be
difficult and tricky, and may sometimes require special infrastructure.
difficult and tricky, and may sometimes require special infrastructure. Please
consider contacting `commercial support` if in doubt.
Here are some tips to keep in mind when dealing with these kind of sites:
@ -263,7 +264,8 @@ Here are some tips to keep in mind when dealing with these kind of sites:
* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
* is possible, use `Google cache`_ to fetch pages, instead of hitting the sites
directly
* use a pool of rotating IPs. For example, the free `Tor project`_.
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
services like `ProxyMesh`_ or `ProxyHub`_.
If you are still unable to prevent your bot getting banned, consider contacting
`commercial support`_.
@ -275,3 +277,5 @@ If you are still unable to prevent your bot getting banned, consider contacting
.. _LIFO: http://en.wikipedia.org/wiki/LIFO
.. _DFO order: http://en.wikipedia.org/wiki/Depth-first_search
.. _BFO order: http://en.wikipedia.org/wiki/Breadth-first_search
.. _ProxyMesh: http://proxymesh.com/
.. _ProxyHub: http://scrapinghub.com/proxyhub.html