RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

mention scrapoxy in best practices doc

This commit is contained in:
Pablo Hoffman 2016-11-16 12:19:32 -03:00
parent 1cbc57faa5
commit d62776a858
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/topics/practices.rst
View File

@ -238,7 +238,8 @@ Here are some tips to keep in mind when dealing with these kinds of sites:
* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
directly
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
services like `ProxyMesh`_
services like `ProxyMesh`_. An open source alterantive is `scrapoxy`_, a
super proxy that you can attach your own proxies to.
* use a highly distributed downloader that circumvents bans internally, so you
can just focus on parsing clean pages. One example of such downloaders is
`Crawlera`_
@ -253,3 +254,4 @@ If you are still unable to prevent your bot getting banned, consider contacting
.. _testspiders: https://github.com/scrapinghub/testspiders
.. _Twisted Reactor Overview: https://twistedmatrix.com/documents/current/core/howto/reactor-basics.html
.. _Crawlera: http://scrapinghub.com/crawlera
.. _scrapoxy: http://scrapoxy.io/