From 77c8ab2e62f25496acfd812381d84aac369fd3b8 Mon Sep 17 00:00:00 2001 From: Anubhav Patel Date: Tue, 27 Aug 2019 18:44:08 +0530 Subject: [PATCH] makes suggested changes --- docs/topics/downloader-middleware.rst | 21 ++++++--------------- docs/topics/settings.rst | 3 ++- 2 files changed, 8 insertions(+), 16 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index ae413dc84..93c91f18a 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -989,6 +989,12 @@ RobotsTxtMiddleware To make sure Scrapy respects robots.txt make sure the middleware is enabled and the :setting:`ROBOTSTXT_OBEY` setting is enabled. + The :setting:`ROBOTSTXT_USER_AGENT` setting can be used to specify the + user agent string to use for matching in the robots.txt_ file. If it + is ``None``, the User-Agent header you are sending with the request or the + :setting:`USER_AGENT` setting (in that order) will be used for determining + the user agent to use in the robots.txt_ file. + This middleware has to be combined with a robots.txt_ parser. Scrapy ships with support for the following robots.txt_ parsers: @@ -1074,21 +1080,6 @@ implementing the methods described below. .. autoclass:: RobotParser :members: -RobotsTxtMiddleware Settings -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. setting:: ROBOTSTXT_USER_AGENT - -ROBOTSTXT_USER_AGENT -^^^^^^^^^^^^^^^^^^^^ - -Default: ``None`` - -The user agent string to use for matching in the robots.txt_ file. If ``None``, -the User-Agent header you are sending with the request or the -:setting:`USER_AGENT` setting (in that order) will be used for determining -the user agent to use in the robots.txt_ file. - .. _robots.txt: http://www.robotstxt.org/ DownloaderStats diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index c4b55cc7b..d3ad777f2 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1422,7 +1422,8 @@ USER_AGENT Default: ``"Scrapy/VERSION (+https://scrapy.org)"`` The default User-Agent to use when crawling, unless overridden. This user agent is -also used in robots.txt if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and +also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` +if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and there is no overridding User-Agent header specified for the request.