mirror of https://github.com/scrapy/scrapy.git
694 B
694 B
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
robots.txt
Scrapy deals with robots.txt files using a :ref:`topics-downloader-middleware`. called RobotsTxtMiddleware.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".To make sure Scrapy respects robots.txt files make sure the following middleware is enabled:
scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware
And the :setting:`ROBOTSTXT_OBEY` setting is enabled.
System Message: ERROR/3 (<stdin>, line 15); backlink
Unknown interpreted text role "setting".Keep in mind that, if you crawl using multiple concurrent requests per domain, Scrapy could get to download some forbidden pages if they were requested to download before the robots.txt file was downloaded. This is a known limitation of the current robots.txt middleware and will be fixed in the future.