diff --git a/docs/index.rst b/docs/index.rst index 433798aa8..7647b3781 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -155,6 +155,7 @@ Solving specific problems topics/debug topics/contracts topics/practices + topics/avoiding-bans topics/broad-crawls topics/developer-tools topics/dynamic-content @@ -179,6 +180,9 @@ Solving specific problems :doc:`topics/practices` Get familiar with some Scrapy common practices. +:doc:`topics/avoiding-bans` + Avoid getting banned from websites. + :doc:`topics/broad-crawls` Tune Scrapy for crawling a lot domains in parallel. diff --git a/docs/topics/avoiding-bans.rst b/docs/topics/avoiding-bans.rst new file mode 100644 index 000000000..59f0da191 --- /dev/null +++ b/docs/topics/avoiding-bans.rst @@ -0,0 +1,340 @@ +.. _bans: + +============= +Avoiding bans +============= + +This topic covers some of the strategies that you can follow to avoid getting +different or bad responses from a website that you are crawling due to filters +such as regional filters, web browser filters, etc. + +.. _avoiding-crawls: + +Avoiding crawls +=============== + +The best way not to be banned from a website is not to send requests to it in +the first place. + +One way to avoid crawling a website is to find the desired dataset through +other means. For example, you can use Google’s `dataset search engine`_. + +If the target website is the only or best source of the desired information, +and you only need to extract the data on a monthly basis or a lower frequency, +you may be able to crawl a public snapshot of the target website instead. +`Common Crawl`_ is an open repository of web crawl data that you can access +freely. It contains monthly snapshots of a wide variety of websites and, if you +are lucky, your target website will be among them. + +.. _Common Crawl: https://commoncrawl.org/ +.. _dataset search engine: https://datasetsearch.research.google.com/ + + +.. _being-polite: + +Being polite +============ + +To avoid being banned, you should first avoid giving a website reasons to ban +you. + +.. _identifying-yourself: + +Identifying yourself +-------------------- + +If your crawling has a noticeable negative impact on a website or you crawl +content that should not be crawled, website administrators will need to do +something. + +Set :setting:`USER_AGENT` to a value that uniquely identifies your spider and +includes contact information, so that website administrators can contact you. + + +.. _following-robotstxt: + +Following robots.txt guidelines +------------------------------- + +Some websites provide a ``robots.txt`` file at their root path (e.g. +``http://example.com/robots.txt``) that describes the guidelines that they wish +bots to follow when crawling their website. + +Before you start writing a spider for a website, read their ``robots.txt`` +file and implement your spider following its guidelines. See the `robots.txt +standard draft`_ or the `robots.txt Google specification`_ for information on +how to read ``robots.txt`` files. + +To ensure that your spider does not crawl pages restricted by ``robots.txt`` +guidelines, set :setting:`ROBOTSTXT_OBEY` to ``True`` to enable the +:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` +middleware. When you do, if your spider attempts to crawl a restricted page, +this middleware aborts that request with the following message:: + + Forbidden by robots.txt + +Also set :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and +:setting:`DOWNLOAD_DELAY` to values that comply with the ``Crawl-Delay`` or +``Request-Rate`` directives from the ``robots.txt`` guidelines. + +You may also use the :ref:`AutoThrottle extension ` on top +of that, so that when the target website experiences a high load, your spider +automatically switches to higher download delays. + +.. _robots.txt Google specification: https://developers.google.com/search/reference/robots_txt +.. _robots.txt standard draft: https://tools.ietf.org/html/draft-koster-rep-00 + + +.. _choosing-crawl-speed: + +Finding the right guidelines on your own +---------------------------------------- + +If a website does not specify a desired download delay, or does not provide a +``robots.txt`` file, you should make an effort to find out the right values for +:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY` that +will not have a noticeable negative impact on the target website. + +Use a service like `SimilarWeb`_ to find out the amount of monthly traffic that +the target website receives, and choose concurrency and delay values that will +not cause a noticeable traffic increase. + +.. _SimilarWeb: https://www.similarweb.com + + +.. _filters-and-challenges: + +Bypassing filters and solving challenges +======================================== + +Some websites implement filters and challenges that aim to deny access or alter +their content based on aspects of the visitor, such as the country where they +are or the web browsing tool they use. + +.. _regional-filter: + +Bypassing regional filters +-------------------------- + +Some websites send different or bad responses based on the region or country +associated to your `IP address`_. + +To bypass these filters, get access to a `proxy server`_ that has an outgoing +IP address from a region that gets the desired responses. + +Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` +middleware to configure your spider to use that proxy. + +.. _IP address: https://en.wikipedia.org/wiki/IP_address +.. _proxy server: https://en.wikipedia.org/wiki/Proxy_server + + +.. _web-browser-filter: + +Bypassing web browser filters +----------------------------- + +Some websites send different or bad responses if they detect that your request +does not come from a web browser. + +To bypass these filters, switch your :setting:`USER_AGENT` to a value copied +from those that popular web browsers use. In some rare cases, you may need a +user agent string from a specific web browser. + +There are multiple Scrapy plugins that can rotate your requests through popular +web browser user agent strings, such as scrapy-fake-useragent_, +scrapy-random-useragent_ or Scrapy-UserAgents_. + +For advanced web browser filters, +:ref:`pre-rendering JavaScript ` or +:ref:`using a headless browser ` may be necessary. +Use these options only as a last resort, however, because they cause a higher +load per request on the target website. + +.. _scrapy-fake-useragent: https://github.com/alecxe/scrapy-fake-useragent +.. _scrapy-random-useragent: https://github.com/cleocn/scrapy-random-useragent +.. _Scrapy-UserAgents: https://pypi.org/project/Scrapy-UserAgents/ + + +.. _request-delay-filter: + +Bypassing request delay filters +------------------------------- + +Some websites may ban your IP after they detect that your requests use a +constant download delay. + +To help bypassing these filters, the :setting:`RANDOMIZE_DOWNLOAD_DELAY` +setting is enabled by default. When that is not enough, an +:ref:`IP address rotation solution ` may be much more effective. + + +.. _isp-filter: + +Bypassing internet service provider filters +------------------------------------------- + +Some websites send different or bad responses if they detect that your request +comes from an IP address that belongs to a `data center`_, as opposed to a +residential IP address from an `internet service provider`_ or a mobile IP +address from a `mobile network`_. + +To bypass these filters, get access to a proxy server that has an outgoing IP +address that is either residential or mobile. Note that you may also get +different responses depending on whether your IP address is residential or +mobile. + +Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` +middleware to configure your spider to use that proxy. + +.. _data center: https://en.wikipedia.org/wiki/Data_center +.. _internet service provider: https://en.wikipedia.org/wiki/Internet_service_provider +.. _mobile network: https://en.wikipedia.org/wiki/Cellular_network + + +.. _captcha: + +Solving CAPTCHA challenges +-------------------------- + +Some websites require you to solve a `CAPTCHA challenge`_ to get the desired +response. + +To bypass these filters, several options exist: + +- You could have your spider present the CAPTCHA challenge to you and wait + for you to solve it manually. + +- Some CAPTCHA challenges can be solved using an `optical character + recognition`_ (OCR) solution such as pytesseract_. + +- Paid CAPTCHA solving services exist. + +Whichever solution you choose, implement it as a :ref:`downloader middleware +` that automatically detects CAPTCHA challenges +in responses and solves them, so that your spider code only receives successful +responses. + +.. _CAPTCHA challenge: https://en.wikipedia.org/wiki/CAPTCHA +.. _optical character recognition: https://en.wikipedia.org/wiki/Optical_character_recognition +.. _pytesseract: https://github.com/madmaze/pytesseract + + +.. _ip-rotation: + +IP address rotation solutions +============================= + +See below some of the different solutions there are to have your requests use +different outgoing IP addresses. + +When using this approach, remember to set :setting:`COOKIES_ENABLED` to +``False`` to disable global cookie handling. This prevents websites from +identifying two requests as coming from the same user agent even if they come +from different IP addresses and have different user-agent strings. You can +still include some cookies manually in your requests. Define them through the +``Cookies`` header of your requests. See +:class:`Request.headers `. + +.. _smart-proxy: + +Smart proxies +------------- + +An increasing number of websites use solutions that apply many of the above +filters and challenges at the same time. + +There are paid proxy services, like `Zyte Smart Proxy Manager`_, that +automatically bypass website filters and challenges, so that your spider only +gets successful responses. They also allow managing sessions to simulate user +behavior. + +For Zyte Smart Proxy Manager, installing scrapy-crawlera_ will offer advanced +integration with Scrapy. For other services, use the +:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` middleware +or implement your own :ref:`downloader middleware +`. + +.. _scrapy-crawlera: https://scrapy-crawlera.readthedocs.io/en/latest/ +.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/ + + +.. _rotating-proxy: + +Rotating proxies +---------------- + +Rotating proxy services like ProxyMesh_ send different requests through +different proxies. This can decrease the likelihood of being affected by some +filters or challenges. + +.. _ProxyMesh: https://proxymesh.com/ + + +.. _free-proxies: + +Free proxies +------------ + +You can easily find lists of free proxies in the internet, and you can use +a solution like `scrapy-rotating-proxies`_ to configure multiple proxies in +your spider and have requests rotate through them automatically. + +This approach, however, has serious drawbacks: + +- Free proxies may stop working at any moment. You need to implement a way to + refresh your list of free proxies. + +- In addition to handling occasional bad responses from websites, you + need to handle all kinds of bad responses from proxies. You may even need + to inspect the response body to determine if a response comes from the + target website or from a misbehaving proxy. + +- Advanced antibot solutions may automatically detect and filter out traffic + from free proxies. + +.. _scrapy-rotating-proxies: https://github.com/TeamHG-Memex/scrapy-rotating-proxies + + +.. _custom-rotating-proxy: + +Custom rotating proxy +--------------------- + +If you have spare servers, you can set them up as proxies and use scrapoxy_ to +build a custom proxy that rotates traffic through them. However, the initial +setup can be complex, and your requests will be vulnerable to +:ref:`internet service provider filtering `. + +.. _scrapoxy: https://scrapoxy.io/ + + +.. _tor: + +The Tor network +--------------- + +It is possible to send requests through the `Tor network`_. + +The initial setup to have Scrapy working with Tor is not straightforward. +Use a search engine to find up-to-date documentation specific to using +Scrapy and Tor together. + +The main drawback of using the Tor network is that traffic can be extremely +slow. + +.. _Tor network: https://en.wikipedia.org/wiki/Tor_(anonymity_network) + + +.. _commercial-support: + +Seeking professional help +========================= + +Avoiding bans, filters and challenges can be difficult and tricky, and may +sometimes require special infrastructure. + +If you find yourself unable to prevent your spider from getting bad responses, +consider contacting `commercial support`_. + +.. _commercial support: https://scrapy.org/support/ diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 732eba587..a7a6fd129 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -226,39 +226,5 @@ crawl:: curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2 curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3 -.. _bans: -Avoiding getting banned -======================= - -Some websites implement certain measures to prevent bots from crawling them, -with varying degrees of sophistication. Getting around those measures can be -difficult and tricky, and may sometimes require special infrastructure. Please -consider contacting `commercial support`_ if in doubt. - -Here are some tips to keep in mind when dealing with these kinds of sites: - -* rotate your user agent from a pool of well-known ones from browsers (google - around to get a list of them) -* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use - cookies to spot bot behaviour -* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. -* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites - directly -* use a pool of rotating IPs. For example, the free `Tor project`_ or paid - services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a - super proxy that you can attach your own proxies to. -* use a highly distributed downloader that circumvents bans internally, so you - can just focus on parsing clean pages. One example of such downloaders is - `Zyte Smart Proxy Manager`_ - -If you are still unable to prevent your bot getting banned, consider contacting -`commercial support`_. - -.. _Tor project: https://www.torproject.org/ -.. _commercial support: https://scrapy.org/support/ -.. _ProxyMesh: https://proxymesh.com/ -.. _Google cache: http://www.googleguide.com/cached_pages.html .. _testspiders: https://github.com/scrapinghub/testspiders -.. _scrapoxy: https://scrapoxy.io/ -.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/