mirror of https://github.com/scrapy/scrapy.git
parent
aa59cf4241
commit
4bc0c6b0f4
|
|
@ -238,7 +238,7 @@ Here are some tips to keep in mind when dealing with these kinds of sites:
|
|||
* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
|
||||
directly
|
||||
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
|
||||
services like `ProxyMesh`_. An open source alterantive is `scrapoxy`_, a
|
||||
services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a
|
||||
super proxy that you can attach your own proxies to.
|
||||
* use a highly distributed downloader that circumvents bans internally, so you
|
||||
can just focus on parsing clean pages. One example of such downloaders is
|
||||
|
|
|
|||
Loading…
Reference in New Issue