From 114437c1693c85125c4275387616d398e68ec20f Mon Sep 17 00:00:00 2001 From: Pengyu CHEN Date: Wed, 4 Nov 2015 02:29:28 +0800 Subject: [PATCH 001/126] added: Doc for `scrapy.http.TextResponse.urljoin` --- docs/topics/request-response.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 75b98d3b3..ba3d697ef 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -666,6 +666,14 @@ TextResponse objects The same as :attr:`text`, but available as a method. This method is kept for backwards compatibility; please prefer ``response.text``. + .. method:: TextResponse.urljoin(url) + + Constructs an absolute url by combining the Response's base url with + a possible relative url. The base url shall be extracted from the + ```` tag, or just the Response's :attr:`url` if there is no such + tag. + + HtmlResponse objects -------------------- From 412f8526029f63fbebd6dae3bad7240dee8dc090 Mon Sep 17 00:00:00 2001 From: Arvind Prasanna <1108710+aprasanna@users.noreply.github.com> Date: Tue, 6 Mar 2018 23:58:27 -0500 Subject: [PATCH 002/126] A few typo fixes and some grammatical enhancements --- docs/news.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 1629510b2..be4cac3f3 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1888,7 +1888,7 @@ Scrapy 0.14.3 - include egg files used by testsuite in source distribution. #118 (:commit:`c897793`) - update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 (:commit:`2548dcc`) - added note to docs/topics/firebug.rst about google directory being shut down (:commit:`668e352`) -- dont discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`) +- do not discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`) - do not fail handling unicode xpaths in libxml2 backed selectors (:commit:`b830e95`) - fixed minor mistake in Request objects documentation (:commit:`bf3c9ee`) - fixed minor defect in link extractors documentation (:commit:`ba14f38`) @@ -1984,7 +1984,7 @@ Code rearranged and removed - Removed googledir project from `examples/googledir`. There's now a new example project called `dirbot` available on github: https://github.com/scrapy/dirbot - Removed support for default field values in Scrapy items (:rev:`2616`) - Removed experimental crawlspider v2 (:rev:`2632`) -- Removed scheduler middleware to simplify architecture. Duplicates filter is now done in the scheduler itself, using the same dupe fltering class as before (`DUPEFILTER_CLASS` setting) (:rev:`2640`) +- Removed scheduler middleware to simplify architecture. Duplicates filter is now done in the scheduler itself, using the same dupe filtering class as before (`DUPEFILTER_CLASS` setting) (:rev:`2640`) - Removed support for passing urls to ``scrapy crawl`` command (use ``scrapy parse`` instead) (:rev:`2704`) - Removed deprecated Execution Queue (:rev:`2704`) - Removed (undocumented) spider context extension (from scrapy.contrib.spidercontext) (:rev:`2780`) @@ -2054,7 +2054,7 @@ New features and improvements - Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195) - Support for overriding default request headers per spider (#181) - Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186) -- Splitted Debian package into two packages - the library and the service (#187) +- The Debian package has been split into two packages - library and the service (#187) - Scrapy log refactoring (#188) - New extension for keeping persistent spider contexts among different runs (#203) - Added `dont_redirect` request.meta key for avoiding redirects (#233) @@ -2075,7 +2075,7 @@ API changes - ``url`` and ``body`` attributes of Request objects are now read-only (#230) - ``Request.copy()`` and ``Request.replace()`` now also copies their ``callback`` and ``errback`` attributes (#231) - Removed ``UrlFilterMiddleware`` from ``scrapy.contrib`` (already disabled by default) -- Offsite middelware doesn't filter out any request coming from a spider that doesn't have a allowed_domains attribute (#225) +- Offsite middleware doesn't filter out any request coming from a spider that doesn't have a allowed_domains attribute (#225) - Removed Spider Manager ``load()`` method. Now spiders are loaded in the constructor itself. - Changes to Scrapy Manager (now called "Crawler"): - ``scrapy.core.manager.ScrapyManager`` class renamed to ``scrapy.crawler.Crawler`` From 6d6243afbb16cb5b7d401a0b2ea7a174b7be71b8 Mon Sep 17 00:00:00 2001 From: leobalestri <33645316+leobalestri@users.noreply.github.com> Date: Sun, 16 Feb 2020 23:45:41 -0800 Subject: [PATCH 003/126] Update install.rst Minor grammar and typo fixes --- docs/intro/install.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 51b41b4d7..a08dedbd0 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -186,7 +186,7 @@ prevents ``pip`` from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions: -* *(Recommended)* **Don't** use system python, install a new, updated version +* *(Recommended)* **Don't** use system python. Install a new, updated version that doesn't conflict with the rest of your system. Here's how to do it using the `homebrew`_ package manager: @@ -231,9 +231,9 @@ PyPy We recommend using the latest PyPy version. The version tested is 5.9.0. For PyPy3, only Linux installation was tested. -Most scrapy dependencides now have binary wheels for CPython, but not for PyPy. -This means that these dependecies will be built during installation. -On OS X, you are likely to face an issue with building Cryptography dependency, +Most scrapy dependencies now have binary wheels for CPython, but not for PyPy. +This means that these dependencies will be built during installation. +On OS X, you are likely to face an issue with building Cryptography dependency. The solution to this problem is described `here `_, that is to ``brew install openssl`` and then export the flags that this command From 231c9ddef8be9d749dcc2684f07ea9bb8bd02aa3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 28 Feb 2020 18:50:45 +0100 Subject: [PATCH 004/126] Update docs/intro/install.rst --- docs/intro/install.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index a08dedbd0..b71379e4d 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -186,7 +186,7 @@ prevents ``pip`` from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions: -* *(Recommended)* **Don't** use system python. Install a new, updated version +* *(Recommended)* **Don't** use system Python. Install a new, updated version that doesn't conflict with the rest of your system. Here's how to do it using the `homebrew`_ package manager: From ac9175964dda07da1e838ebb063fc8dd95925e0d Mon Sep 17 00:00:00 2001 From: maanijou <19888963+maanijou@users.noreply.github.com> Date: Sun, 12 Sep 2021 17:59:20 +0200 Subject: [PATCH 005/126] Improve documentation for spider middlewares --- docs/topics/spider-middleware.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index f0158dc41..f3fb0d5d7 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -122,7 +122,7 @@ object gives you access, for example, to the :ref:`settings `. method (from a previous spider middleware) raises an exception. :meth:`process_spider_exception` should return either ``None`` or an - iterable of :class:`~scrapy.Request` objects and :ref:`item object + iterable of :class:`~scrapy.Request` objects or :ref:`item object `. If it returns ``None``, Scrapy will continue processing this exception, From e5998fb8469dff1e2e965826d2b43f9bad0c17ad Mon Sep 17 00:00:00 2001 From: kamran890 Date: Wed, 22 Sep 2021 03:00:18 +0500 Subject: [PATCH 006/126] Document spider.state attribute (#5174) --- docs/topics/jobs.rst | 2 ++ docs/topics/spiders.rst | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index e49f37a2f..f16d306c7 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -39,6 +39,8 @@ a signal), and resume it later by issuing the same command:: scrapy crawl somespider -s JOBDIR=crawls/somespider-1 +.. _topics-keeping-persistent-state-between-batches: + Keeping persistent state between batches ======================================== diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 67b9e2e0e..4d3d32941 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -122,6 +122,11 @@ scrapy.Spider send log messages through it as described on :ref:`topics-logging-from-spiders`. + .. attribute:: state + + A dict you can use to persist some spider state between batches. + See :ref:`topics-keeping-persistent-state-between-batches` to know more about it. + .. method:: from_crawler(crawler, *args, **kwargs) This is the class method used by Scrapy to create your spiders. From 1829dd774ca0f056e97f7c4621575ef9126e4b57 Mon Sep 17 00:00:00 2001 From: "Reza (Milad) Maanijou" <19888963+maanijou@users.noreply.github.com> Date: Sat, 25 Sep 2021 20:22:09 +0330 Subject: [PATCH 007/126] Update spider-middleware.rst --- docs/topics/spider-middleware.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index f3fb0d5d7..08be2b03c 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -122,7 +122,7 @@ object gives you access, for example, to the :ref:`settings `. method (from a previous spider middleware) raises an exception. :meth:`process_spider_exception` should return either ``None`` or an - iterable of :class:`~scrapy.Request` objects or :ref:`item object + iterable of :class:`~scrapy.Request` objects or :ref:`item objects `. If it returns ``None``, Scrapy will continue processing this exception, From dfdb779756aa1df2059980de02d359e4e93bac55 Mon Sep 17 00:00:00 2001 From: "Reza (Milad) Maanijou" <19888963+maanijou@users.noreply.github.com> Date: Sun, 26 Sep 2021 12:45:44 +0330 Subject: [PATCH 008/126] Apply review comments --- docs/topics/spider-middleware.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 08be2b03c..f1373b9ee 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -122,7 +122,7 @@ object gives you access, for example, to the :ref:`settings `. method (from a previous spider middleware) raises an exception. :meth:`process_spider_exception` should return either ``None`` or an - iterable of :class:`~scrapy.Request` objects or :ref:`item objects + iterable of :class:`~scrapy.Request` or :ref:`item objects `. If it returns ``None``, Scrapy will continue processing this exception, From 3c57825b0f3a7bb250aec753b09f964f36500e2b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Sun, 26 Sep 2021 13:41:26 +0200 Subject: [PATCH 009/126] Update docs/topics/spider-middleware.rst --- docs/topics/spider-middleware.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index f1373b9ee..73bedf655 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -122,8 +122,8 @@ object gives you access, for example, to the :ref:`settings `. method (from a previous spider middleware) raises an exception. :meth:`process_spider_exception` should return either ``None`` or an - iterable of :class:`~scrapy.Request` or :ref:`item objects - `. + iterable of :class:`~scrapy.Request` or :ref:`item ` + objects. If it returns ``None``, Scrapy will continue processing this exception, executing any other :meth:`process_spider_exception` in the following From 74f146bbe0c41e2c21f431c00ff34d6c5d10cb63 Mon Sep 17 00:00:00 2001 From: Deepanshu <73387559+iDeepverma@users.noreply.github.com> Date: Fri, 1 Oct 2021 01:47:05 +0530 Subject: [PATCH 010/126] Document update URLLENGTH_LIMIT --- docs/topics/settings.rst | 8 ++++++-- docs/topics/spider-middleware.rst | 1 + 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 2ab2020fa..4d3ae20cc 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1645,8 +1645,12 @@ Default: ``2083`` Scope: ``spidermiddlewares.urllength`` -The maximum URL length to allow for crawled URLs. For more information about -the default value for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 +The maximum URL length to allow for crawled URLs. You can set this to ``0`` +to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working +with URLs longer than the default value. The default limit acts as a stopping condition in case of +URLs of increasing length, usually caused by a loop. +For more information about the default value +for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 .. setting:: USER_AGENT diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 73bedf655..a0a7b1fb6 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -440,4 +440,5 @@ UrlLengthMiddleware settings (see the settings documentation for more info): * :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs. + If ``0``, then :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is disabled. From 890f884de46602352de48fa844edd9959c62e473 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com> Date: Fri, 1 Oct 2021 04:50:42 -0300 Subject: [PATCH 011/126] Allow 'callback' key in keyword arguments for request callbacks (#5251) --- scrapy/core/scraper.py | 2 +- tests/test_request_cb_kwargs.py | 13 ++++++++----- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.py index d6d6f64f9..f40bccbb3 100644 --- a/scrapy/core/scraper.py +++ b/scrapy/core/scraper.py @@ -156,7 +156,7 @@ class Scraper: callback = result.request.callback or spider._parse warn_on_generator_with_return_value(spider, callback) dfd = defer_succeed(result) - dfd.addCallback(callback, **result.request.cb_kwargs) + dfd.addCallbacks(callback=callback, callbackKeywords=result.request.cb_kwargs) else: # result is a Failure result.request = request warn_on_generator_with_return_value(spider, request.errback) diff --git a/tests/test_request_cb_kwargs.py b/tests/test_request_cb_kwargs.py index 738502de8..8b96fe1a1 100644 --- a/tests/test_request_cb_kwargs.py +++ b/tests/test_request_cb_kwargs.py @@ -60,7 +60,7 @@ class KeywordArgumentsSpider(MockServerSpider): checks = [] def start_requests(self): - data = {'key': 'value', 'number': 123} + data = {'key': 'value', 'number': 123, 'callback': 'some_callback'} yield Request(self.mockserver.url('/first'), self.parse_first, cb_kwargs=data) yield Request(self.mockserver.url('/general_with'), self.parse_general, cb_kwargs=data) yield Request(self.mockserver.url('/general_without'), self.parse_general) @@ -88,7 +88,8 @@ class KeywordArgumentsSpider(MockServerSpider): if response.url.endswith('/general_with'): self.checks.append(kwargs['key'] == 'value') self.checks.append(kwargs['number'] == 123) - self.crawler.stats.inc_value('boolean_checks', 2) + self.checks.append(kwargs['callback'] == 'some_callback') + self.crawler.stats.inc_value('boolean_checks', 3) elif response.url.endswith('/general_without'): self.checks.append(kwargs == {}) self.crawler.stats.inc_value('boolean_checks') @@ -110,7 +111,7 @@ class KeywordArgumentsSpider(MockServerSpider): TypeError: parse_takes_less() got an unexpected keyword argument 'number' """ - def parse_takes_more(self, response, key, number, other): + def parse_takes_more(self, response, key, number, callback, other): """ Should raise TypeError: parse_takes_more() missing 1 required positional argument: 'other' @@ -161,11 +162,13 @@ class CallbackKeywordArgumentsTestCase(TestCase): self.assertTrue( str(exceptions['takes_less'].exc_info[1]).endswith( "parse_takes_less() got an unexpected keyword argument 'number'" - ) + ), + msg="Exception message: " + str(exceptions['takes_less'].exc_info[1]), ) self.assertEqual(exceptions['takes_more'].exc_info[0], TypeError) self.assertTrue( str(exceptions['takes_more'].exc_info[1]).endswith( "parse_takes_more() missing 1 required positional argument: 'other'" - ) + ), + msg="Exception message: " + str(exceptions['takes_more'].exc_info[1]), ) From fbb1236fd6ee283414360f5209b3a569e112c8cb Mon Sep 17 00:00:00 2001 From: Deepanshu verma <73387559+iDeepverma@users.noreply.github.com> Date: Fri, 1 Oct 2021 18:46:11 +0530 Subject: [PATCH 012/126] Update docs/topics/settings.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit added suggestion Co-authored-by: Adrián Chaves --- docs/topics/settings.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 4d3ae20cc..ed8f1f105 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1645,12 +1645,19 @@ Default: ``2083`` Scope: ``spidermiddlewares.urllength`` -The maximum URL length to allow for crawled URLs. You can set this to ``0`` -to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working -with URLs longer than the default value. The default limit acts as a stopping condition in case of -URLs of increasing length, usually caused by a loop. -For more information about the default value -for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 +The maximum URL length to allow for crawled URLs. + +This setting can act as a stopping condition in case of URLs of ever-increasing +length, which may be caused for example by a programming error either in the +target server or in your code. See also :setting:`REDIRECT_MAX_TIMES` and +:setting:`DEPTH_LIMIT`. + +Use ``0`` to allow URLs of any length. + +The default value is copied from the `Microsoft Internet Explorer maximum URL +length`_, even though this setting exists for different reasons. + +.. _Microsoft Internet Explorer maximum URL length: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 .. setting:: USER_AGENT From d91d82b5064c512e741da106b5fb3a398027d5a3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 1 Oct 2021 16:31:29 +0200 Subject: [PATCH 013/126] Make Scrapy SFW again --- docs/topics/request-response.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index d3e08efd4..42ce22158 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -300,7 +300,7 @@ errors if needed:: "http://www.httpbin.org/status/404", # Not found error "http://www.httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout expected - "http://www.httphttpbinbin.org/", # DNS error expected + "https://example.invalid/", # DNS error expected ] def start_requests(self): From de2043f9c1661208de7b73305a8e3a2395d29583 Mon Sep 17 00:00:00 2001 From: Deepanshu <73387559+iDeepverma@users.noreply.github.com> Date: Fri, 1 Oct 2021 20:20:00 +0530 Subject: [PATCH 014/126] updated docs/topics/spider-middleware.rst --- docs/topics/spider-middleware.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index a0a7b1fb6..f27bc79c0 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -440,5 +440,3 @@ UrlLengthMiddleware settings (see the settings documentation for more info): * :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs. - If ``0``, then :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is disabled. - From 47533985f4dc7e58895e1a34c3ea88502f83572a Mon Sep 17 00:00:00 2001 From: Peter Morrison Date: Fri, 1 Oct 2021 12:30:14 -0600 Subject: [PATCH 015/126] Document file expiration method in media-pipeline --- docs/topics/media-pipeline.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index 46bd2859b..10d2ac990 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -383,6 +383,9 @@ class name. E.g. given pipeline class called MyPipeline you can set setting key: and pipeline class MyPipeline will have expiration time set to 180. +The last modified time from the file is used to determine the age of the file in days, +which is then compared to the set expiration time to determine if the file is expired. + .. _topics-images-thumbnails: Thumbnail generation for images From de70b3c58b5b4b2f95468218c194b1e4b99a33c4 Mon Sep 17 00:00:00 2001 From: Deepanshu verma <73387559+iDeepverma@users.noreply.github.com> Date: Sat, 2 Oct 2021 12:12:58 +0530 Subject: [PATCH 016/126] Update spider-middleware.rst added empty line --- docs/topics/spider-middleware.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index f27bc79c0..3545e760b 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -440,3 +440,4 @@ UrlLengthMiddleware settings (see the settings documentation for more info): * :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs. + From f10880022273c005a6cb6d9e6ff9cc9a370cc375 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Sat, 2 Oct 2021 13:25:15 +0200 Subject: [PATCH 017/126] Update spider-middleware.rst --- docs/topics/spider-middleware.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 3545e760b..f27bc79c0 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -440,4 +440,3 @@ UrlLengthMiddleware settings (see the settings documentation for more info): * :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs. - From ef263042d75586e94d74290d1a41c432f7d87bec Mon Sep 17 00:00:00 2001 From: Raihan Nismara <31585789+raihan71@users.noreply.github.com> Date: Sun, 3 Oct 2021 13:26:20 +0700 Subject: [PATCH 018/126] Using Logo Scrapy in Readme.md Logo scrapy used in readme.md made looks nicer --- README.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.rst b/README.rst index 5750e2c0f..05f10bb6c 100644 --- a/README.rst +++ b/README.rst @@ -1,3 +1,6 @@ +.. image:: /artwork/scrapy-logo.jpg + :width: 400px + ====== Scrapy ====== From b9647b85d3bc9dcefc7d829ce8304bb28f7cc798 Mon Sep 17 00:00:00 2001 From: Ryan Whelchel Date: Sun, 3 Oct 2021 17:32:38 -0400 Subject: [PATCH 019/126] docs: restructed phrasing for clarity --- docs/intro/tutorial.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index 438f3d6df..ba27b18bc 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -277,9 +277,9 @@ As an alternative, you could've written: >>> response.css('title::text')[0].get() 'Quotes to Scrape' -However, using ``.get()`` directly on a :class:`~scrapy.selector.SelectorList` -instance avoids an ``IndexError`` and returns ``None`` when it doesn't -find any element matching the selection. +Directly accessing an index on a :class:`~scrapy.selector.SelectorList` +instance could potentially run into an ``IndexError``. It is recommended to use +``.get()`` directly instead as it avoids such index errors. There's a lesson here: for most scraping code, you want it to be resilient to errors due to things not being found on a page, so that even if some parts fail From 764cf0178bb7bc346f1c15ed7ab0adcbb75c43b0 Mon Sep 17 00:00:00 2001 From: Ryan Whelchel Date: Tue, 5 Oct 2021 10:22:57 -0400 Subject: [PATCH 020/126] Update docs/intro/tutorial.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adrián Chaves --- docs/intro/tutorial.rst | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index ba27b18bc..fa321a770 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -277,9 +277,20 @@ As an alternative, you could've written: >>> response.css('title::text')[0].get() 'Quotes to Scrape' -Directly accessing an index on a :class:`~scrapy.selector.SelectorList` -instance could potentially run into an ``IndexError``. It is recommended to use -``.get()`` directly instead as it avoids such index errors. +Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will +raise an :exc:`IndexError` exception if there are no results:: + + >>> response.css('noelement')[0].get() + Traceback (most recent call last): + File "", line 1, in + ... + IndexError: list index out of range + +You might want to use ``.get()`` directly on the +:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None`` +if there are no results:: + +>>> response.css("noelement").get() There's a lesson here: for most scraping code, you want it to be resilient to errors due to things not being found on a page, so that even if some parts fail From b081f18a2f232a1bfd04c829ae6894cac93a89a1 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 16 Aug 2019 14:53:42 +0500 Subject: [PATCH 021/126] Add http_auth_domain to HttpAuthMiddleware. --- docs/topics/downloader-middleware.rst | 18 ++++- scrapy/downloadermiddlewares/httpauth.py | 21 ++++- tests/test_downloadermiddleware_httpauth.py | 87 ++++++++++++++++++++- 3 files changed, 119 insertions(+), 7 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 99d57bda9..28b019c80 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -323,8 +323,21 @@ HttpAuthMiddleware This middleware authenticates all requests generated from certain spiders using `Basic access authentication`_ (aka. HTTP auth). - To enable HTTP authentication from certain spiders, set the ``http_user`` - and ``http_pass`` attributes of those spiders. + To enable HTTP authentication for a spider, set the ``http_user`` and + ``http_pass`` spider attributes to the authentication data and the + ``http_auth_domain`` spider attribute to the domain which requires this + authentication (its subdomains will be also handled in the same way). + You can set ``http_auth_domain`` to ``None`` to enable the + authentication for all requests but usually this is not needed. + + .. warning:: + In the previous Scrapy versions HttpAuthMiddleware sent the + authentication data with all requests, which is a security problem if + the spider makes requests to several different domains. Currently if + the ``http_auth_domain`` attribute is not set, the middleware will use + the domain of the first request, which will work for some spider but + not for others. In the future the middleware will produce an error + instead. Example:: @@ -334,6 +347,7 @@ HttpAuthMiddleware http_user = 'someuser' http_pass = 'somepass' + http_auth_domain = 'intranet.example.com' name = 'intranet.example.com' # .. rest of the spider code omitted ... diff --git a/scrapy/downloadermiddlewares/httpauth.py b/scrapy/downloadermiddlewares/httpauth.py index 089bf0d85..1bee3e279 100644 --- a/scrapy/downloadermiddlewares/httpauth.py +++ b/scrapy/downloadermiddlewares/httpauth.py @@ -3,10 +3,14 @@ HTTP basic auth downloader middleware See documentation in docs/topics/downloader-middleware.rst """ +import warnings from w3lib.http import basic_auth_header from scrapy import signals +from scrapy.exceptions import ScrapyDeprecationWarning +from scrapy.utils.httpobj import urlparse_cached +from scrapy.utils.url import url_is_from_any_domain class HttpAuthMiddleware: @@ -24,8 +28,23 @@ class HttpAuthMiddleware: pwd = getattr(spider, 'http_pass', '') if usr or pwd: self.auth = basic_auth_header(usr, pwd) + if not hasattr(spider, 'http_auth_domain'): + warnings.warn('Using HttpAuthMiddleware without http_auth_domain is deprecated and can cause security ' + 'problems if the spider makes requests to several different domains. http_auth_domain ' + 'will be set to the domain of the first request, please set it to the correct value ' + 'explicitly.', + category=ScrapyDeprecationWarning) + self.domain_unset = True + else: + self.domain = spider.http_auth_domain + self.domain_unset = False def process_request(self, request, spider): auth = getattr(self, 'auth', None) if auth and b'Authorization' not in request.headers: - request.headers[b'Authorization'] = auth + domain = urlparse_cached(request).hostname + if self.domain_unset: + self.domain = domain + self.domain_unset = False + if not self.domain or url_is_from_any_domain(request.url, [self.domain]): + request.headers[b'Authorization'] = auth diff --git a/tests/test_downloadermiddleware_httpauth.py b/tests/test_downloadermiddleware_httpauth.py index 3381632b0..0362e2018 100644 --- a/tests/test_downloadermiddleware_httpauth.py +++ b/tests/test_downloadermiddleware_httpauth.py @@ -1,13 +1,60 @@ import unittest +from w3lib.http import basic_auth_header + from scrapy.http import Request from scrapy.downloadermiddlewares.httpauth import HttpAuthMiddleware from scrapy.spiders import Spider +class TestSpiderLegacy(Spider): + http_user = 'foo' + http_pass = 'bar' + + class TestSpider(Spider): http_user = 'foo' http_pass = 'bar' + http_auth_domain = 'example.com' + + +class TestSpiderAny(Spider): + http_user = 'foo' + http_pass = 'bar' + http_auth_domain = None + + +class HttpAuthMiddlewareLegacyTest(unittest.TestCase): + + def setUp(self): + self.spider = TestSpiderLegacy('foo') + + def test_auth(self): + mw = HttpAuthMiddleware() + mw.spider_opened(self.spider) + + # initial request, sets the domain and sends the header + req = Request('http://example.com/') + assert mw.process_request(req, self.spider) is None + self.assertEqual(req.headers['Authorization'], basic_auth_header('foo', 'bar')) + + # subsequent request to the same domain, should send the header + req = Request('http://example.com/') + assert mw.process_request(req, self.spider) is None + self.assertEqual(req.headers['Authorization'], basic_auth_header('foo', 'bar')) + + # subsequent request to a different domain, shouldn't send the header + req = Request('http://example-noauth.com/') + assert mw.process_request(req, self.spider) is None + self.assertNotIn('Authorization', req.headers) + + def test_auth_already_set(self): + mw = HttpAuthMiddleware() + mw.spider_opened(self.spider) + req = Request('http://example.com/', + headers=dict(Authorization='Digest 123')) + assert mw.process_request(req, self.spider) is None + self.assertEqual(req.headers['Authorization'], b'Digest 123') class HttpAuthMiddlewareTest(unittest.TestCase): @@ -20,13 +67,45 @@ class HttpAuthMiddlewareTest(unittest.TestCase): def tearDown(self): del self.mw - def test_auth(self): - req = Request('http://scrapytest.org/') + def test_no_auth(self): + req = Request('http://example-noauth.com/') assert self.mw.process_request(req, self.spider) is None - self.assertEqual(req.headers['Authorization'], b'Basic Zm9vOmJhcg==') + self.assertNotIn('Authorization', req.headers) + + def test_auth_domain(self): + req = Request('http://example.com/') + assert self.mw.process_request(req, self.spider) is None + self.assertEqual(req.headers['Authorization'], basic_auth_header('foo', 'bar')) + + def test_auth_subdomain(self): + req = Request('http://foo.example.com/') + assert self.mw.process_request(req, self.spider) is None + self.assertEqual(req.headers['Authorization'], basic_auth_header('foo', 'bar')) def test_auth_already_set(self): - req = Request('http://scrapytest.org/', + req = Request('http://example.com/', + headers=dict(Authorization='Digest 123')) + assert self.mw.process_request(req, self.spider) is None + self.assertEqual(req.headers['Authorization'], b'Digest 123') + + +class HttpAuthAnyMiddlewareTest(unittest.TestCase): + + def setUp(self): + self.mw = HttpAuthMiddleware() + self.spider = TestSpiderAny('foo') + self.mw.spider_opened(self.spider) + + def tearDown(self): + del self.mw + + def test_auth(self): + req = Request('http://example.com/') + assert self.mw.process_request(req, self.spider) is None + self.assertEqual(req.headers['Authorization'], basic_auth_header('foo', 'bar')) + + def test_auth_already_set(self): + req = Request('http://example.com/', headers=dict(Authorization='Digest 123')) assert self.mw.process_request(req, self.spider) is None self.assertEqual(req.headers['Authorization'], b'Digest 123') From 7ec5f299c42d07768c02970f0a11f018ed790188 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 22 Aug 2019 20:32:56 +0500 Subject: [PATCH 022/126] Small documentation fixes. --- docs/topics/downloader-middleware.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 28b019c80..caf44a903 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -328,16 +328,16 @@ HttpAuthMiddleware ``http_auth_domain`` spider attribute to the domain which requires this authentication (its subdomains will be also handled in the same way). You can set ``http_auth_domain`` to ``None`` to enable the - authentication for all requests but usually this is not needed. + authentication for all requests but you risk leaking your authentication + credentials to unrelated domains. .. warning:: - In the previous Scrapy versions HttpAuthMiddleware sent the - authentication data with all requests, which is a security problem if - the spider makes requests to several different domains. Currently if - the ``http_auth_domain`` attribute is not set, the middleware will use - the domain of the first request, which will work for some spider but - not for others. In the future the middleware will produce an error - instead. + In previous Scrapy versions HttpAuthMiddleware sent the authentication + data with all requests, which is a security problem if the spider + makes requests to several different domains. Currently if the + ``http_auth_domain`` attribute is not set, the middleware will use the + domain of the first request, which will work for some spiders but not + for others. In the future the middleware will produce an error instead. Example:: From f0105a882df200f71088603fecaeb9d40679c387 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 5 Oct 2021 13:29:06 +0200 Subject: [PATCH 023/126] Cover 2.5.1 in the release notes --- docs/news.rst | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index 0ea412e75..4b5cbb2da 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,44 @@ Release notes ============= +.. _release-2.5.1: + +Scrapy 2.5.1 (2021-10-05) +------------------------- + +* **Security bug fix:** + + If you use + :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware` + (i.e. the ``http_user`` and ``http_pass`` spider attributes) for HTTP + authentication, any request exposes your credentials to the request target. + + To prevent unintended exposure of authentication credentials to unintended + domains, you must now additionally set a new, additional spider attribute, + ``http_auth_domain``, and point it to the specific domain to which the + authentication credentials must be sent. + + If the ``http_auth_domain`` spider attribute is not set, the domain of the + first request will be considered the HTTP authentication target, and + authentication credentials will only be sent in requests targeting that + domain. + + If you need to send the same HTTP authentication credentials to multiple + domains, you can use :func:`w3lib.http.basic_auth_header` instead to + set the value of the ``Authorization`` header of your requests. + + If you *really* want your spider to send the same HTTP authentication + credentials to any domain, set the ``http_auth_domain`` spider attribute + to ``None``. + + Finally, if you are a user of `scrapy-splash`_, know that this version of + Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will + need to upgrade scrapy-splash to a greater version for it to continue to + work. + +.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash + + .. _release-2.5.0: Scrapy 2.5.0 (2021-04-06) From 735750c254e6e82af46b4ebbb35e28b8c0a52250 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 5 Oct 2021 21:10:49 +0200 Subject: [PATCH 024/126] Cover 1.8.1 in the release notes --- docs/news.rst | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index 4b5cbb2da..5e590f027 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1492,6 +1492,44 @@ affect subclasses: (:issue:`3884`) +.. _release-1.8.1: + +Scrapy 1.8.1 (2021-10-05) +------------------------- + +* **Security bug fix:** + + If you use + :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware` + (i.e. the ``http_user`` and ``http_pass`` spider attributes) for HTTP + authentication, any request exposes your credentials to the request target. + + To prevent unintended exposure of authentication credentials to unintended + domains, you must now additionally set a new, additional spider attribute, + ``http_auth_domain``, and point it to the specific domain to which the + authentication credentials must be sent. + + If the ``http_auth_domain`` spider attribute is not set, the domain of the + first request will be considered the HTTP authentication target, and + authentication credentials will only be sent in requests targeting that + domain. + + If you need to send the same HTTP authentication credentials to multiple + domains, you can use :func:`w3lib.http.basic_auth_header` instead to + set the value of the ``Authorization`` header of your requests. + + If you *really* want your spider to send the same HTTP authentication + credentials to any domain, set the ``http_auth_domain`` spider attribute + to ``None``. + + Finally, if you are a user of `scrapy-splash`_, know that this version of + Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will + need to upgrade scrapy-splash to a greater version for it to continue to + work. + +.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash + + .. _release-1.8.0: Scrapy 1.8.0 (2019-10-28) From 6c858cec91b013853a73e6215b74c90b609bc2da Mon Sep 17 00:00:00 2001 From: Laerte <5853172+Laerte@users.noreply.github.com> Date: Wed, 6 Oct 2021 12:32:04 -0300 Subject: [PATCH 025/126] Cookies: Cast primitive types to str (#5253) * cast primitive types to str * add tests --- scrapy/downloadermiddlewares/cookies.py | 4 ++-- tests/test_downloadermiddleware_cookies.py | 21 +++++++++++++++++++++ 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/scrapy/downloadermiddlewares/cookies.py b/scrapy/downloadermiddlewares/cookies.py index d95ed3d38..0eee8d758 100644 --- a/scrapy/downloadermiddlewares/cookies.py +++ b/scrapy/downloadermiddlewares/cookies.py @@ -80,8 +80,8 @@ class CookiesMiddleware: logger.warning(msg.format(request, cookie, key)) return continue - if isinstance(cookie[key], str): - decoded[key] = cookie[key] + if isinstance(cookie[key], (bool, float, int, str)): + decoded[key] = str(cookie[key]) else: try: decoded[key] = cookie[key].decode("utf8") diff --git a/tests/test_downloadermiddleware_cookies.py b/tests/test_downloadermiddleware_cookies.py index aff8542e9..36021bfbf 100644 --- a/tests/test_downloadermiddleware_cookies.py +++ b/tests/test_downloadermiddleware_cookies.py @@ -347,3 +347,24 @@ class CookiesMiddlewareTest(TestCase): self.assertCookieValEqual(req1.headers['Cookie'], 'key=value1') self.assertCookieValEqual(req2.headers['Cookie'], 'key=value2') self.assertCookieValEqual(req3.headers['Cookie'], 'key=') + + def test_primitive_type_cookies(self): + # Boolean + req1 = Request('http://example.org', cookies={'a': True}) + assert self.mw.process_request(req1, self.spider) is None + self.assertCookieValEqual(req1.headers['Cookie'], b'a=True') + + # Float + req2 = Request('http://example.org', cookies={'a': 9.5}) + assert self.mw.process_request(req2, self.spider) is None + self.assertCookieValEqual(req2.headers['Cookie'], b'a=9.5') + + # Integer + req3 = Request('http://example.org', cookies={'a': 10}) + assert self.mw.process_request(req3, self.spider) is None + self.assertCookieValEqual(req3.headers['Cookie'], b'a=10') + + # String + req4 = Request('http://example.org', cookies={'a': 'b'}) + assert self.mw.process_request(req4, self.spider) is None + self.assertCookieValEqual(req4.headers['Cookie'], b'a=b') From b1cb007b3b8fef6f037cd1abd38fe9da7190ed26 Mon Sep 17 00:00:00 2001 From: MarvinPetzoldt <78762153+MarvinPetzoldt@users.noreply.github.com> Date: Wed, 6 Oct 2021 19:08:19 +0200 Subject: [PATCH 026/126] Fixed documentation example --- docs/topics/exporters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index 8c30122b6..923336769 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -122,7 +122,7 @@ Example:: class ProductXmlExporter(XmlItemExporter): def serialize_field(self, field, name, value): - if field == 'price': + if name == 'price': return f'$ {str(value)}' return super().serialize_field(field, name, value) From 029cab72e8a9b20ebb6b9540e9e01747f0e83dba Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com> Date: Wed, 6 Oct 2021 14:34:09 -0300 Subject: [PATCH 027/126] [CI] fix pypy test (#5264) --- tests/test_request_cb_kwargs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_request_cb_kwargs.py b/tests/test_request_cb_kwargs.py index 8b96fe1a1..473a93e69 100644 --- a/tests/test_request_cb_kwargs.py +++ b/tests/test_request_cb_kwargs.py @@ -105,7 +105,7 @@ class KeywordArgumentsSpider(MockServerSpider): self.checks.append(default == 99) self.crawler.stats.inc_value('boolean_checks', 4) - def parse_takes_less(self, response, key): + def parse_takes_less(self, response, key, callback): """ Should raise TypeError: parse_takes_less() got an unexpected keyword argument 'number' From d3f1bf79e883fe3662df827ee47cfc93a372ff02 Mon Sep 17 00:00:00 2001 From: Georgiy Zatserklianyi Date: Thu, 7 Oct 2021 17:27:20 +0300 Subject: [PATCH 028/126] Use f-strings where appropriate (#5246) --- scrapy/__init__.py | 2 +- scrapy/core/downloader/contextfactory.py | 12 +++++---- scrapy/core/downloader/tls.py | 8 +++--- scrapy/downloadermiddlewares/httpcache.py | 2 +- scrapy/extensions/feedexport.py | 33 ++++++++++------------- scrapy/extensions/httpcache.py | 4 +-- scrapy/utils/misc.py | 2 +- tests/spiders.py | 16 +++++------ tests/test_closespider.py | 2 +- tests/test_commands.py | 4 +-- tests/test_downloader_handlers_http2.py | 2 +- tests/test_http_request.py | 19 +++++++------ 12 files changed, 51 insertions(+), 55 deletions(-) diff --git a/scrapy/__init__.py b/scrapy/__init__.py index 8a8065bf2..396f98219 100644 --- a/scrapy/__init__.py +++ b/scrapy/__init__.py @@ -29,7 +29,7 @@ twisted_version = (_txv.major, _txv.minor, _txv.micro) # Check minimum required Python version if sys.version_info < (3, 6): - print("Scrapy %s requires Python 3.6+" % __version__) + print(f"Scrapy {__version__} requires Python 3.6+") sys.exit(1) diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py index 073ef16bf..b5318c7bb 100644 --- a/scrapy/core/downloader/contextfactory.py +++ b/scrapy/core/downloader/contextfactory.py @@ -135,11 +135,13 @@ def load_context_factory_from_settings(settings, crawler): settings=settings, crawler=crawler, ) - msg = """ - '%s' does not accept `method` argument (type OpenSSL.SSL method,\ - e.g. OpenSSL.SSL.SSLv23_METHOD) and/or `tls_verbose_logging` argument and/or `tls_ciphers` argument.\ - Please upgrade your context factory class to handle them or ignore them.""" % ( - settings['DOWNLOADER_CLIENTCONTEXTFACTORY'],) + msg = ( + f"{settings['DOWNLOADER_CLIENTCONTEXTFACTORY']} does not accept " + "a `method` argument (type OpenSSL.SSL method, e.g. " + "OpenSSL.SSL.SSLv23_METHOD) and/or a `tls_verbose_logging` " + "argument and/or a `tls_ciphers` argument. Please, upgrade your " + "context factory class to handle them or ignore them." + ) warnings.warn(msg) return context_factory diff --git a/scrapy/core/downloader/tls.py b/scrapy/core/downloader/tls.py index 2b8990b75..19a56d9b6 100644 --- a/scrapy/core/downloader/tls.py +++ b/scrapy/core/downloader/tls.py @@ -65,14 +65,14 @@ class ScrapyClientTLSOptions(ClientTLSOptions): verifyHostname(connection, self._hostnameASCII) except (CertificateError, VerificationError) as e: logger.warning( - 'Remote certificate is not valid for hostname "{}"; {}'.format( - self._hostnameASCII, e)) + 'Remote certificate is not valid for hostname "%s"; %s', + self._hostnameASCII, e) except ValueError as e: logger.warning( 'Ignoring error while verifying certificate ' - 'from host "{}" (exception: {})'.format( - self._hostnameASCII, repr(e))) + 'from host "%s" (exception: %r)', + self._hostnameASCII, e) DEFAULT_CIPHERS = AcceptableCiphers.fromOpenSSLCipherString('DEFAULT') diff --git a/scrapy/downloadermiddlewares/httpcache.py b/scrapy/downloadermiddlewares/httpcache.py index 62f1c3a29..80ed7ac75 100644 --- a/scrapy/downloadermiddlewares/httpcache.py +++ b/scrapy/downloadermiddlewares/httpcache.py @@ -70,7 +70,7 @@ class HttpCacheMiddleware: self.stats.inc_value('httpcache/miss', spider=spider) if self.ignore_missing: self.stats.inc_value('httpcache/ignore', spider=spider) - raise IgnoreRequest("Ignored request not in cache: %s" % request) + raise IgnoreRequest(f"Ignored request not in cache: {request}") return None # first time request # Return cached response only if not expired diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py index 0f5bf01d0..370723368 100644 --- a/scrapy/extensions/feedexport.py +++ b/scrapy/extensions/feedexport.py @@ -38,11 +38,10 @@ def build_storage(builder, uri, *args, feed_options=None, preargs=(), **kwargs): kwargs['feed_options'] = feed_options else: warnings.warn( - "{} does not support the 'feed_options' keyword argument. Add a " + f"{builder.__qualname__} does not support the 'feed_options' keyword argument. Add a " "'feed_options' parameter to its signature to remove this " "warning. This parameter will become mandatory in a future " - "version of Scrapy." - .format(builder.__qualname__), + "version of Scrapy.", category=ScrapyDeprecationWarning ) return builder(*preargs, uri, *args, **kwargs) @@ -356,32 +355,28 @@ class FeedExporter: # properly closed. return defer.maybeDeferred(slot.storage.store, slot.file) slot.finish_exporting() - logfmt = "%s %%(format)s feed (%%(itemcount)d items) in: %%(uri)s" - log_args = {'format': slot.format, - 'itemcount': slot.itemcount, - 'uri': slot.uri} + logmsg = f"{slot.format} feed ({slot.itemcount} items) in: {slot.uri}" d = defer.maybeDeferred(slot.storage.store, slot.file) - # Use `largs=log_args` to copy log_args into function's scope - # instead of using `log_args` from the outer scope d.addCallback( - self._handle_store_success, log_args, logfmt, spider, type(slot.storage).__name__ + self._handle_store_success, logmsg, spider, type(slot.storage).__name__ ) d.addErrback( - self._handle_store_error, log_args, logfmt, spider, type(slot.storage).__name__ + self._handle_store_error, logmsg, spider, type(slot.storage).__name__ ) return d - def _handle_store_error(self, f, largs, logfmt, spider, slot_type): + def _handle_store_error(self, f, logmsg, spider, slot_type): logger.error( - logfmt % "Error storing", largs, + "Error storing %s", logmsg, exc_info=failure_to_exc_info(f), extra={'spider': spider} ) self.crawler.stats.inc_value(f"feedexport/failed_count/{slot_type}") - def _handle_store_success(self, f, largs, logfmt, spider, slot_type): + def _handle_store_success(self, f, logmsg, spider, slot_type): logger.info( - logfmt % "Stored", largs, extra={'spider': spider} + "Stored %s", logmsg, + extra={'spider': spider} ) self.crawler.stats.inc_value(f"feedexport/success_count/{slot_type}") @@ -474,10 +469,10 @@ class FeedExporter: for uri_template, values in self.feeds.items(): if values['batch_item_count'] and not re.search(r'%\(batch_time\)s|%\(batch_id\)', uri_template): logger.error( - '%(batch_time)s or %(batch_id)d must be in the feed URI ({}) if FEED_EXPORT_BATCH_ITEM_COUNT ' + '%%(batch_time)s or %%(batch_id)d must be in the feed URI (%s) if FEED_EXPORT_BATCH_ITEM_COUNT ' 'setting or FEEDS.batch_item_count is specified and greater than 0. For more info see: ' - 'https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-export-batch-item-count' - ''.format(uri_template) + 'https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-export-batch-item-count', + uri_template ) return False return True @@ -526,7 +521,7 @@ class FeedExporter: instance = build_instance(feedcls) method_name = '__new__' if instance is None: - raise TypeError("%s.%s returned None" % (feedcls.__qualname__, method_name)) + raise TypeError(f"{feedcls.__qualname__}.{method_name} returned None") return instance def _get_uri_params(self, spider, uri_params, slot=None): diff --git a/scrapy/extensions/httpcache.py b/scrapy/extensions/httpcache.py index e0c04b2de..d0ae29b90 100644 --- a/scrapy/extensions/httpcache.py +++ b/scrapy/extensions/httpcache.py @@ -226,7 +226,7 @@ class DbmCacheStorage: dbpath = os.path.join(self.cachedir, f'{spider.name}.db') self.db = self.dbmodule.open(dbpath, 'c') - logger.debug("Using DBM cache storage in %(cachepath)s" % {'cachepath': dbpath}, extra={'spider': spider}) + logger.debug("Using DBM cache storage in %(cachepath)s", {'cachepath': dbpath}, extra={'spider': spider}) def close_spider(self, spider): self.db.close() @@ -280,7 +280,7 @@ class FilesystemCacheStorage: self._open = gzip.open if self.use_gzip else open def open_spider(self, spider): - logger.debug("Using filesystem cache storage in %(cachedir)s" % {'cachedir': self.cachedir}, + logger.debug("Using filesystem cache storage in %(cachedir)s", {'cachedir': self.cachedir}, extra={'spider': spider}) def close_spider(self, spider): diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py index 51cef1e91..11c4206c2 100644 --- a/scrapy/utils/misc.py +++ b/scrapy/utils/misc.py @@ -50,7 +50,7 @@ def load_object(path): return path else: raise TypeError("Unexpected argument type, expected string " - "or object, got: %s" % type(path)) + f"or object, got: {type(path)}") try: dot = path.rindex('.') diff --git a/tests/spiders.py b/tests/spiders.py index 5b45f897e..67dbbbe0f 100644 --- a/tests/spiders.py +++ b/tests/spiders.py @@ -86,7 +86,7 @@ class SimpleSpider(MetaSpider): self.start_urls = [url] def parse(self, response): - self.logger.info("Got response %d" % response.status) + self.logger.info(f"Got response {response.status}") class AsyncDefSpider(SimpleSpider): @@ -95,7 +95,7 @@ class AsyncDefSpider(SimpleSpider): async def parse(self, response): await defer.succeed(42) - self.logger.info("Got response %d" % response.status) + self.logger.info(f"Got response {response.status}") class AsyncDefAsyncioSpider(SimpleSpider): @@ -105,7 +105,7 @@ class AsyncDefAsyncioSpider(SimpleSpider): async def parse(self, response): await asyncio.sleep(0.2) status = await get_from_asyncio_queue(response.status) - self.logger.info("Got response %d" % status) + self.logger.info(f"Got response {status}") class AsyncDefAsyncioReturnSpider(SimpleSpider): @@ -115,7 +115,7 @@ class AsyncDefAsyncioReturnSpider(SimpleSpider): async def parse(self, response): await asyncio.sleep(0.2) status = await get_from_asyncio_queue(response.status) - self.logger.info("Got response %d" % status) + self.logger.info(f"Got response {status}") return [{'id': 1}, {'id': 2}] @@ -126,7 +126,7 @@ class AsyncDefAsyncioReturnSingleElementSpider(SimpleSpider): async def parse(self, response): await asyncio.sleep(0.1) status = await get_from_asyncio_queue(response.status) - self.logger.info("Got response %d" % status) + self.logger.info(f"Got response {status}") return {"foo": 42} @@ -138,7 +138,7 @@ class AsyncDefAsyncioReqsReturnSpider(SimpleSpider): await asyncio.sleep(0.2) req_id = response.meta.get('req_id', 0) status = await get_from_asyncio_queue(response.status) - self.logger.info("Got response %d, req_id %d" % (status, req_id)) + self.logger.info(f"Got response {status}, req_id {req_id}") if req_id > 0: return reqs = [] @@ -155,7 +155,7 @@ class AsyncDefAsyncioGenSpider(SimpleSpider): async def parse(self, response): await asyncio.sleep(0.2) yield {'foo': 42} - self.logger.info("Got response %d" % response.status) + self.logger.info(f"Got response {response.status}") class AsyncDefAsyncioGenLoopSpider(SimpleSpider): @@ -166,7 +166,7 @@ class AsyncDefAsyncioGenLoopSpider(SimpleSpider): for i in range(10): await asyncio.sleep(0.1) yield {'foo': i} - self.logger.info("Got response %d" % response.status) + self.logger.info(f"Got response {response.status}") class AsyncDefAsyncioGenComplexSpider(SimpleSpider): diff --git a/tests/test_closespider.py b/tests/test_closespider.py index 5ec5e2989..be8adadb3 100644 --- a/tests/test_closespider.py +++ b/tests/test_closespider.py @@ -41,7 +41,7 @@ class TestCloseSpider(TestCase): yield crawler.crawl(total=1000000, mockserver=self.mockserver) reason = crawler.spider.meta['close_reason'] self.assertEqual(reason, 'closespider_errorcount') - key = 'spider_exceptions/{name}'.format(name=crawler.spider.exception_cls.__name__) + key = f'spider_exceptions/{crawler.spider.exception_cls.__name__}' errorcount = crawler.stats.get_value(key) self.assertTrue(errorcount >= close_on) diff --git a/tests/test_commands.py b/tests/test_commands.py index 086286b3a..75098a77a 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -498,7 +498,7 @@ class GenspiderCommandTest(CommandTest): self.find_in_file(join(self.proj_mod_path, 'spiders', 'test_name.py'), r'allowed_domains\s*=\s*\[\'(.+)\'\]').group(1)) - self.assertEqual('http://%s/' % domain, + self.assertEqual(f'http://{domain}/', self.find_in_file(join(self.proj_mod_path, 'spiders', 'test_name.py'), r'start_urls\s*=\s*\[\'(.+)\'\]').group(1)) @@ -708,7 +708,7 @@ class MySpider(scrapy.Spider): ]) import asyncio loop = asyncio.new_event_loop() - self.assertIn("Using asyncio event loop: %s.%s" % (loop.__module__, loop.__class__.__name__), log) + self.assertIn(f"Using asyncio event loop: {loop.__module__}.{loop.__class__.__name__}", log) def test_output(self): spider_code = """ diff --git a/tests/test_downloader_handlers_http2.py b/tests/test_downloader_handlers_http2.py index 8c8c30597..3a9db3ee5 100644 --- a/tests/test_downloader_handlers_http2.py +++ b/tests/test_downloader_handlers_http2.py @@ -248,7 +248,7 @@ class Https2ProxyTestCase(Http11ProxyTestCase): self.assertEqual(response.url, request.url) self.assertEqual(response.body, b'/') - http_proxy = '%s?noconnect' % self.getURL('') + http_proxy = f"{self.getURL('')}?noconnect" request = Request('https://example.com', meta={'proxy': http_proxy}) with self.assertWarnsRegex( Warning, diff --git a/tests/test_http_request.py b/tests/test_http_request.py index b610087bd..579ef9fa2 100644 --- a/tests/test_http_request.py +++ b/tests/test_http_request.py @@ -1217,18 +1217,17 @@ class FormRequestTest(RequestTest): response, formcss="input[name='abc']") def test_from_response_valid_form_methods(self): - body = """
- -
""" + form_methods = [[method, method] for method in self.request_class.valid_form_methods] + form_methods.append(['UNKNOWN', 'GET']) - for method in self.request_class.valid_form_methods: - response = _buildresponse(body % method) + for method, expected in form_methods: + response = _buildresponse( + f'
' + '' + '
' + ) r = self.request_class.from_response(response) - self.assertEqual(r.method, method) - - response = _buildresponse(body % 'UNKNOWN') - r = self.request_class.from_response(response) - self.assertEqual(r.method, 'GET') + self.assertEqual(r.method, expected) def _buildresponse(body, **kwargs): From 65d60b9692dc3475b42d14c744d8a5ac0f2b38cf Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com> Date: Sun, 10 Oct 2021 05:06:36 -0300 Subject: [PATCH 029/126] [docs] add missing parameter to headers_received signal (#5270) --- docs/topics/signals.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index a67cc1879..63ad3a9ad 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -413,7 +413,7 @@ headers_received .. versionadded:: 2.5 .. signal:: headers_received -.. function:: headers_received(headers, request, spider) +.. function:: headers_received(headers, body_length, request, spider) Sent by the HTTP 1.1 and S3 download handlers when the response headers are available for a given request, before downloading any additional content. From 6fbd6f941f00037ae4718805352e0fa86a781e41 Mon Sep 17 00:00:00 2001 From: ankur19 Date: Sat, 9 Oct 2021 19:09:51 -0400 Subject: [PATCH 030/126] Fix issue#5145 Fix condition for failing tests set Selector to None on AttributeError Add test and remove unused imports Fix imports --- scrapy/loader/__init__.py | 5 ++++- tests/test_loader.py | 8 +++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/scrapy/loader/__init__.py b/scrapy/loader/__init__.py index 014951a8e..91337b949 100644 --- a/scrapy/loader/__init__.py +++ b/scrapy/loader/__init__.py @@ -83,6 +83,9 @@ class ItemLoader(itemloaders.ItemLoader): def __init__(self, item=None, selector=None, response=None, parent=None, **context): if selector is None and response is not None: - selector = self.default_selector_class(response) + try: + selector = self.default_selector_class(response) + except AttributeError: + selector = None context.update(response=response) super().__init__(item=item, selector=selector, parent=parent, **context) diff --git a/tests/test_loader.py b/tests/test_loader.py index b0bc82f4e..f7ab1f236 100644 --- a/tests/test_loader.py +++ b/tests/test_loader.py @@ -4,7 +4,7 @@ import attr from itemadapter import ItemAdapter from itemloaders.processors import Compose, Identity, MapCompose, TakeFirst -from scrapy.http import HtmlResponse +from scrapy.http import HtmlResponse, Response from scrapy.item import Item, Field from scrapy.loader import ItemLoader from scrapy.selector import Selector @@ -304,6 +304,12 @@ class SelectortemLoaderTest(unittest.TestCase): l.add_css('name', 'div::text') self.assertEqual(l.get_output_value('name'), ['Marta']) + + def test_init_method_with_base_response(self): + """Selector should be None after initialization""" + response = Response("https://scrapy.org") + l = TestItemLoader(response=response) + self.assertIs(l.selector, None) def test_init_method_with_response(self): l = TestItemLoader(response=self.response) From 3a263280bad53a26490381f293a454be6c25ea30 Mon Sep 17 00:00:00 2001 From: "Kian-Meng, Ang" Date: Mon, 11 Oct 2021 22:32:42 +0800 Subject: [PATCH 031/126] Fix typos --- docs/news.rst | 10 +++++----- docs/topics/settings.rst | 4 ++-- docs/topics/shell.rst | 2 +- docs/topics/spiders.rst | 4 ++-- docs/versioning.rst | 2 +- extras/qpsclient.py | 2 +- scrapy/downloadermiddlewares/retry.py | 2 +- scrapy/exporters.py | 2 +- scrapy/linkextractors/lxmlhtml.py | 2 +- scrapy/pipelines/files.py | 8 ++++---- scrapy/spiders/feed.py | 4 ++-- scrapy/utils/console.py | 2 +- scrapy/utils/datatypes.py | 2 +- scrapy/utils/defer.py | 2 +- scrapy/utils/request.py | 4 ++-- sep/sep-001.rst | 2 +- sep/sep-005.rst | 2 +- sep/sep-014.rst | 2 +- sep/sep-021.rst | 2 +- tests/test_http_response.py | 4 ++-- tests/test_request_attribute_binding.py | 8 ++++---- tests/test_utils_defer.py | 4 ++-- tests/test_utils_template.py | 2 +- 23 files changed, 39 insertions(+), 39 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 5e590f027..509366c17 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1830,7 +1830,7 @@ New features * A new scheduler priority queue, ``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be :ref:`enabled ` for a significant - scheduling improvement on crawls targetting multiple web domains, at the + scheduling improvement on crawls targeting multiple web domains, at the cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`) * A new :attr:`Request.cb_kwargs ` attribute @@ -2868,7 +2868,7 @@ Bug fixes - Fix for selected callbacks when using ``CrawlSpider`` with :command:`scrapy parse ` (:issue:`2225`). - Fix for invalid JSON and XML files when spider yields no items (:issue:`872`). -- Implement ``flush()`` fpr ``StreamLogger`` avoiding a warning in logs (:issue:`2125`). +- Implement ``flush()`` for ``StreamLogger`` avoiding a warning in logs (:issue:`2125`). Refactoring ~~~~~~~~~~~ @@ -3731,7 +3731,7 @@ Scrapy 0.24.3 (2014-08-09) - adding some xpath tips to selectors docs (:commit:`2d103e0`) - fix tests to account for https://github.com/scrapy/w3lib/pull/23 (:commit:`f8d366a`) - get_func_args maximum recursion fix #728 (:commit:`81344ea`) -- Updated input/ouput processor example according to #560. (:commit:`f7c4ea8`) +- Updated input/output processor example according to #560. (:commit:`f7c4ea8`) - Fixed Python syntax in tutorial. (:commit:`db59ed9`) - Add test case for tunneling proxy (:commit:`f090260`) - Bugfix for leaking Proxy-Authorization header to remote host when using tunneling (:commit:`d8793af`) @@ -4393,7 +4393,7 @@ Scrapyd changes ~~~~~~~~~~~~~~~ - Scrapyd now uses one process per spider -- It stores one log file per spider run, and rotate them keeping the lastest 5 logs per spider (by default) +- It stores one log file per spider run, and rotate them keeping the latest 5 logs per spider (by default) - A minimal web ui was added, available at http://localhost:6800 by default - There is now a ``scrapy server`` command to start a Scrapyd server of the current project @@ -4429,7 +4429,7 @@ New features and improvements - Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195) - Support for overriding default request headers per spider (#181) - Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186) -- Splitted Debian package into two packages - the library and the service (#187) +- Split Debian package into two packages - the library and the service (#187) - Scrapy log refactoring (#188) - New extension for keeping persistent spider contexts among different runs (#203) - Added ``dont_redirect`` request.meta key for avoiding redirects (#233) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 2ab2020fa..19a549a02 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1566,7 +1566,7 @@ If a reactor is already installed, :meth:`CrawlerRunner.__init__ ` raises :exc:`Exception` if the installed reactor does not match the -:setting:`TWISTED_REACTOR` setting; therfore, having top-level +:setting:`TWISTED_REACTOR` setting; therefore, having top-level :mod:`~twisted.internet.reactor` imports in project files and imported third-party libraries will make Scrapy raise :exc:`Exception` when it checks which reactor is installed. @@ -1658,7 +1658,7 @@ Default: ``"Scrapy/VERSION (+https://scrapy.org)"`` The default User-Agent to use when crawling, unless overridden. This user agent is also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and -there is no overridding User-Agent header specified for the request. +there is no overriding User-Agent header specified for the request. Settings documented elsewhere: diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index 8c90a506c..007e9fc2f 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -99,7 +99,7 @@ Available Shortcuts shortcuts - ``fetch(url[, redirect=True])`` - fetch a new response from the given URL - and update all related objects accordingly. You can optionaly ask for HTTP + and update all related objects accordingly. You can optionally ask for HTTP 3xx redirections to not be followed by passing ``redirect=False`` - ``fetch(request)`` - fetch a new response from the given request and update diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 4d3d32941..99e74233a 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -372,7 +372,7 @@ CrawlSpider described below. If multiple rules match the same link, the first one will be used, according to the order they're defined in this attribute. - This spider also exposes an overrideable method: + This spider also exposes an overridable method: .. method:: parse_start_url(response, **kwargs) @@ -534,7 +534,7 @@ XMLFeedSpider itertag = 'n:url' # ... - Apart from these new attributes, this spider has the following overrideable + Apart from these new attributes, this spider has the following overridable methods too: .. method:: adapt_response(response) diff --git a/docs/versioning.rst b/docs/versioning.rst index 57643ea9a..9d02757b0 100644 --- a/docs/versioning.rst +++ b/docs/versioning.rst @@ -13,7 +13,7 @@ There are 3 numbers in a Scrapy version: *A.B.C* large changes. * *B* is the release number. This will include many changes including features and things that possibly break backward compatibility, although we strive to - keep theses cases at a minimum. + keep these cases at a minimum. * *C* is the bugfix release number. Backward-incompatibilities are explicitly mentioned in the :ref:`release notes `, diff --git a/extras/qpsclient.py b/extras/qpsclient.py index f9fb70342..28703650d 100644 --- a/extras/qpsclient.py +++ b/extras/qpsclient.py @@ -1,5 +1,5 @@ """ -A spider that generate light requests to meassure QPS troughput +A spider that generate light requests to meassure QPS throughput usage: diff --git a/scrapy/downloadermiddlewares/retry.py b/scrapy/downloadermiddlewares/retry.py index f1fdc3858..c6cc7c56d 100644 --- a/scrapy/downloadermiddlewares/retry.py +++ b/scrapy/downloadermiddlewares/retry.py @@ -2,7 +2,7 @@ An extension to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error. -You can change the behaviour of this middleware by modifing the scraping settings: +You can change the behaviour of this middleware by modifying the scraping settings: RETRY_TIMES - how many times to retry a failed page RETRY_HTTP_CODES - which HTTP response codes to retry diff --git a/scrapy/exporters.py b/scrapy/exporters.py index fb4b565cf..36cca2d05 100644 --- a/scrapy/exporters.py +++ b/scrapy/exporters.py @@ -30,7 +30,7 @@ class BaseItemExporter: self._configure(kwargs, dont_fail=dont_fail) def _configure(self, options, dont_fail=False): - """Configure the exporter by poping options from the ``options`` dict. + """Configure the exporter by popping options from the ``options`` dict. If dont_fail is set, it won't raise an exception on unexpected options (useful for using with keyword arguments in subclasses ``__init__`` methods) """ diff --git a/scrapy/linkextractors/lxmlhtml.py b/scrapy/linkextractors/lxmlhtml.py index e941c4321..b5d2585a8 100644 --- a/scrapy/linkextractors/lxmlhtml.py +++ b/scrapy/linkextractors/lxmlhtml.py @@ -88,7 +88,7 @@ class LxmlParserLinkExtractor: def _process_links(self, links): """ Normalize and filter extracted links - The subclass should override it if neccessary + The subclass should override it if necessary """ return self._deduplicate_if_needed(links) diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py index 8766ef66f..5c52c6c28 100644 --- a/scrapy/pipelines/files.py +++ b/scrapy/pipelines/files.py @@ -85,7 +85,7 @@ class S3FilesStore: AWS_USE_SSL = None AWS_VERIFY = None - POLICY = 'private' # Overriden from settings.FILES_STORE_S3_ACL in FilesPipeline.from_settings + POLICY = 'private' # Overridden from settings.FILES_STORE_S3_ACL in FilesPipeline.from_settings HEADERS = { 'Cache-Control': 'max-age=172800', } @@ -142,7 +142,7 @@ class S3FilesStore: **extra) def _headers_to_botocore_kwargs(self, headers): - """ Convert headers to botocore keyword agruments. + """ Convert headers to botocore keyword arguments. """ # This is required while we need to support both boto and botocore. mapping = CaselessDict({ @@ -190,7 +190,7 @@ class GCSFilesStore: CACHE_CONTROL = 'max-age=172800' # The bucket's default object ACL will be applied to the object. - # Overriden from settings.FILES_STORE_GCS_ACL in FilesPipeline.from_settings. + # Overridden from settings.FILES_STORE_GCS_ACL in FilesPipeline.from_settings. POLICY = None def __init__(self, uri): @@ -291,7 +291,7 @@ class FilesPipeline(MediaPipeline): """Abstract pipeline that implement the file downloading This pipeline tries to minimize network transfers and file processing, - doing stat of the files and determining if file is new, uptodate or + doing stat of the files and determining if file is new, up-to-date or expired. ``new`` files are those that pipeline never processed and needs to be diff --git a/scrapy/spiders/feed.py b/scrapy/spiders/feed.py index 6ed17e4dd..bef2d6b24 100644 --- a/scrapy/spiders/feed.py +++ b/scrapy/spiders/feed.py @@ -43,7 +43,7 @@ class XMLFeedSpider(Spider): return response def parse_node(self, response, selector): - """This method must be overriden with your custom spider functionality""" + """This method must be overridden with your custom spider functionality""" if hasattr(self, 'parse_item'): # backward compatibility return self.parse_item(response, selector) raise NotImplementedError @@ -113,7 +113,7 @@ class CSVFeedSpider(Spider): return response def parse_row(self, response, row): - """This method must be overriden with your custom spider functionality""" + """This method must be overridden with your custom spider functionality""" raise NotImplementedError def parse_rows(self, response): diff --git a/scrapy/utils/console.py b/scrapy/utils/console.py index 133261fd7..1bc0bd45f 100644 --- a/scrapy/utils/console.py +++ b/scrapy/utils/console.py @@ -14,7 +14,7 @@ def _embed_ipython_shell(namespace={}, banner=''): @wraps(_embed_ipython_shell) def wrapper(namespace=namespace, banner=''): config = load_default_config() - # Always use .instace() to ensure _instance propagation to all parents + # Always use .instance() to ensure _instance propagation to all parents # this is needed for completion works well for new imports # and clear the instance to always have the fresh env # on repeated breaks like with inspect_response() diff --git a/scrapy/utils/datatypes.py b/scrapy/utils/datatypes.py index e31284a7f..47df8a717 100644 --- a/scrapy/utils/datatypes.py +++ b/scrapy/utils/datatypes.py @@ -41,7 +41,7 @@ class CaselessDict(dict): return key.lower() def normvalue(self, value): - """Method to normalize values prior to be setted""" + """Method to normalize values prior to be set""" return value def get(self, key, def_val=None): diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.py index b317c12a3..b02bfdccb 100644 --- a/scrapy/utils/defer.py +++ b/scrapy/utils/defer.py @@ -34,7 +34,7 @@ def defer_succeed(result) -> Deferred: """Same as twisted.internet.defer.succeed but delay calling callback until next reactor loop - It delays by 100ms so reactor has a chance to go trough readers and writers + It delays by 100ms so reactor has a chance to go through readers and writers before attending pending delayed calls, so do not set delay to zero. """ from twisted.internet import reactor diff --git a/scrapy/utils/request.py b/scrapy/utils/request.py index 57dcc5f2c..70ef3ba2b 100644 --- a/scrapy/utils/request.py +++ b/scrapy/utils/request.py @@ -48,7 +48,7 @@ def request_fingerprint( the fingerprint. For this reason, request headers are ignored by default when calculating - the fingeprint. If you want to include specific headers use the + the fingerprint. If you want to include specific headers use the include_headers argument, which is a list of Request headers to include. Also, servers usually ignore fragments in urls when handling requests, @@ -78,7 +78,7 @@ def request_fingerprint( def request_authenticate(request: Request, username: str, password: str) -> None: - """Autenticate the given request (in place) using the HTTP basic access + """Authenticate the given request (in place) using the HTTP basic access authentication mechanism (RFC 2617) and the given username and password """ request.headers['Authorization'] = basic_auth_header(username, password) diff --git a/sep/sep-001.rst b/sep/sep-001.rst index 00226283f..f704e113f 100644 --- a/sep/sep-001.rst +++ b/sep/sep-001.rst @@ -260,7 +260,7 @@ ItemForm ia['width'] = x.x('//p[@class="width"]') ia['volume'] = x.x('//p[@class="volume"]') - # another example passing parametes on instance + # another example passing parameters on instance ia = NewsForm(response, encoding='utf-8') ia['name'] = x.x('//p[@class="name"]') diff --git a/sep/sep-005.rst b/sep/sep-005.rst index e795838e4..08ed367b3 100644 --- a/sep/sep-005.rst +++ b/sep/sep-005.rst @@ -107,7 +107,7 @@ gUsing default_builder This will use default_builder as the builder for every field in the item class. -As a reducer is not set reducers will be set based on Item Field classess. +As a reducer is not set reducers will be set based on Item Field classes. gReset default_builder for a field ================================== diff --git a/sep/sep-014.rst b/sep/sep-014.rst index 8ca81824d..0859e3f7c 100644 --- a/sep/sep-014.rst +++ b/sep/sep-014.rst @@ -64,7 +64,7 @@ Request Processors takes requests objects and can perform any action to them, like filtering or modifying on the fly. The current ``LinkExtractor`` had integrated link processing, like -canonicalize. Request Processors can be reutilized and applied in serie. +canonicalize. Request Processors can be reutilized and applied in series. Request Generator ----------------- diff --git a/sep/sep-021.rst b/sep/sep-021.rst index 372429791..c1ec16f7f 100644 --- a/sep/sep-021.rst +++ b/sep/sep-021.rst @@ -22,7 +22,7 @@ Instead, the hooks are spread over: * Downloader handlers (DOWNLOADER_HANDLERS) * Item pipelines (ITEM_PIPELINES) * Feed exporters and storages (FEED_EXPORTERS, FEED_STORAGES) -* Overrideable components (DUPEFILTER_CLASS, STATS_CLASS, SCHEDULER, SPIDER_MANAGER_CLASS, ITEM_PROCESSOR, etc) +* Overridable components (DUPEFILTER_CLASS, STATS_CLASS, SCHEDULER, SPIDER_MANAGER_CLASS, ITEM_PROCESSOR, etc) * Generic extensions (EXTENSIONS) * CLI commands (COMMANDS_MODULE) diff --git a/tests/test_http_response.py b/tests/test_http_response.py index c376a46cd..0ec5257e1 100644 --- a/tests/test_http_response.py +++ b/tests/test_http_response.py @@ -19,7 +19,7 @@ class BaseResponseTest(unittest.TestCase): response_class = Response def test_init(self): - # Response requires url in the consturctor + # Response requires url in the constructor self.assertRaises(Exception, self.response_class) self.assertTrue(isinstance(self.response_class('http://example.com/'), self.response_class)) self.assertRaises(TypeError, self.response_class, b"http://example.com") @@ -392,7 +392,7 @@ class TextResponseTest(BaseResponseTest): def test_declared_encoding_invalid(self): """Check that unknown declared encodings are ignored""" r = self.response_class("http://www.example.com", - headers={"Content-type": ["text/html; charset=UKNOWN"]}, + headers={"Content-type": ["text/html; charset=UNKNOWN"]}, body=b"\xc2\xa3") self.assertEqual(r._declared_encoding(), None) self._assert_response_values(r, 'utf-8', "\xa3") diff --git a/tests/test_request_attribute_binding.py b/tests/test_request_attribute_binding.py index 00c532c41..25d9657d5 100644 --- a/tests/test_request_attribute_binding.py +++ b/tests/test_request_attribute_binding.py @@ -106,9 +106,9 @@ class CrawlTestCase(TestCase): """ Downloader middleware which returns a response with an specific 'request' attribute. - * The spider callback should receive the overriden response.request - * Handlers listening to the response_received signal should receive the overriden response.request - * The "crawled" log message should show the overriden response.request + * The spider callback should receive the overridden response.request + * Handlers listening to the response_received signal should receive the overridden response.request + * The "crawled" log message should show the overridden response.request """ signal_params = {} @@ -144,7 +144,7 @@ class CrawlTestCase(TestCase): An exception is raised but caught by the next middleware, which returns a Response with a specific 'request' attribute. - The spider callback should receive the overriden response.request + The spider callback should receive the overridden response.request """ url = self.mockserver.url("/status?n=200") runner = CrawlerRunner(settings={ diff --git a/tests/test_utils_defer.py b/tests/test_utils_defer.py index 7a5f458c7..032dbc8c5 100644 --- a/tests/test_utils_defer.py +++ b/tests/test_utils_defer.py @@ -23,7 +23,7 @@ class MustbeDeferredTest(unittest.TestCase): dfd = mustbe_deferred(_append, 1) dfd.addCallback(self.assertEqual, [1, 2]) # it is [1] with maybeDeferred - steps.append(2) # add another value, that should be catched by assertEqual + steps.append(2) # add another value, that should be caught by assertEqual return dfd def test_unfired_deferred(self): @@ -37,7 +37,7 @@ class MustbeDeferredTest(unittest.TestCase): dfd = mustbe_deferred(_append, 1) dfd.addCallback(self.assertEqual, [1, 2]) # it is [1] with maybeDeferred - steps.append(2) # add another value, that should be catched by assertEqual + steps.append(2) # add another value, that should be caught by assertEqual return dfd diff --git a/tests/test_utils_template.py b/tests/test_utils_template.py index 5ff2e41ef..1d5e63363 100644 --- a/tests/test_utils_template.py +++ b/tests/test_utils_template.py @@ -36,7 +36,7 @@ class UtilsRenderTemplateFileTestCase(unittest.TestCase): self.assertEqual(result.read().decode('utf8'), rendered) os.remove(render_path) - assert not os.path.exists(render_path) # Failure of test iself + assert not os.path.exists(render_path) # Failure of test itself if '__main__' == __name__: From d08199f631814bdaadafc441bb47cbe71ced2d8d Mon Sep 17 00:00:00 2001 From: Jake Herbst Date: Tue, 12 Oct 2021 13:20:09 -0400 Subject: [PATCH 032/126] Removing unnecessary line from docs to prevent test failure (#5274) --- docs/intro/tutorial.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index fa321a770..ca5856881 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -282,7 +282,6 @@ raise an :exc:`IndexError` exception if there are no results:: >>> response.css('noelement')[0].get() Traceback (most recent call last): - File "", line 1, in ... IndexError: list index out of range From 3243aa2cd54c8789eb5667998aa8296f6adaa9a9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=B0=AD=E4=B9=9D=E9=BC=8E?= <109224573@qq.com> Date: Thu, 14 Oct 2021 10:18:26 +0800 Subject: [PATCH 033/126] docs: fix typo --- docs/topics/loaders.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index c0f534493..0d63700c8 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -56,7 +56,7 @@ chapter `:: l.add_xpath('name', '//div[@class="product_name"]') l.add_xpath('name', '//div[@class="product_title"]') l.add_xpath('price', '//p[@id="price"]') - l.add_css('stock', 'p#stock]') + l.add_css('stock', 'p#stock') l.add_value('last_updated', 'today') # you can also use literal values return l.load_item() From ca320feb2afa5eae3990ed21df6a8df930e8cf9f Mon Sep 17 00:00:00 2001 From: Erik Kemperman Date: Fri, 15 Oct 2021 15:43:55 +0200 Subject: [PATCH 034/126] Add LOG_FILE_APPEND to settings --- docs/topics/logging.rst | 5 ++++- docs/topics/settings.rst | 10 ++++++++++ scrapy/settings/default_settings.py | 1 + scrapy/utils/log.py | 3 ++- tests/test_crawler.py | 29 ++++++++++++++++++++++++++++- 5 files changed, 45 insertions(+), 3 deletions(-) diff --git a/docs/topics/logging.rst b/docs/topics/logging.rst index dda04dc4d..d593c74c6 100644 --- a/docs/topics/logging.rst +++ b/docs/topics/logging.rst @@ -143,6 +143,7 @@ Logging settings These settings can be used to configure the logging: * :setting:`LOG_FILE` +* :setting:`LOG_FILE_APPEND` * :setting:`LOG_ENABLED` * :setting:`LOG_ENCODING` * :setting:`LOG_LEVEL` @@ -155,7 +156,9 @@ The first couple of settings define a destination for log messages. If :setting:`LOG_FILE` is set, messages sent through the root logger will be redirected to a file named :setting:`LOG_FILE` with encoding :setting:`LOG_ENCODING`. If unset and :setting:`LOG_ENABLED` is ``True``, log -messages will be displayed on the standard error. Lastly, if +messages will be displayed on the standard error. If :setting:`LOG_FILE` is set +and :setting:`LOG_FILE_APPEND` is ``False``, the file will be overwritten +(discarding the output from previous runs, if any). Lastly, if :setting:`LOG_ENABLED` is ``False``, there won't be any visible log output. :setting:`LOG_LEVEL` determines the minimum level of severity to display, those diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index e63aca312..210c1def7 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1026,6 +1026,16 @@ Default: ``None`` File name to use for logging output. If ``None``, standard error will be used. +.. setting:: LOG_FILE_APPEND + +LOG_FILE_APPEND +--------------- + +Default: ``True`` + +If ``False``, the log file specified with :setting:`LOG_FILE` will be +overwritten (discarding the output from previous runs, if any). + .. setting:: LOG_FORMAT LOG_FORMAT diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 4ef330dd2..8389a70cb 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -207,6 +207,7 @@ LOG_DATEFORMAT = '%Y-%m-%d %H:%M:%S' LOG_STDOUT = False LOG_LEVEL = 'DEBUG' LOG_FILE = None +LOG_FILE_APPEND = True LOG_SHORT_NAMES = False SCHEDULER_DEBUG = False diff --git a/scrapy/utils/log.py b/scrapy/utils/log.py index 6c456ed60..0441c0358 100644 --- a/scrapy/utils/log.py +++ b/scrapy/utils/log.py @@ -124,8 +124,9 @@ def _get_handler(settings): """ Return a log handler object according to settings """ filename = settings.get('LOG_FILE') if filename: + mode = 'a' if settings.getbool('LOG_FILE_APPEND') else 'w' encoding = settings.get('LOG_ENCODING') - handler = logging.FileHandler(filename, encoding=encoding) + handler = logging.FileHandler(filename, mode=mode, encoding=encoding) elif settings.getbool('LOG_ENABLED'): handler = logging.StreamHandler() else: diff --git a/tests/test_crawler.py b/tests/test_crawler.py index dec517bb6..a80ad4388 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -98,6 +98,8 @@ class CrawlerLoggingTestCase(unittest.TestCase): def test_spider_custom_settings_log_level(self): log_file = self.mktemp() + with open(log_file, 'wb') as fo: + fo.write('previous message\n'.encode('utf-8')) class MySpider(scrapy.Spider): name = 'spider' @@ -119,8 +121,9 @@ class CrawlerLoggingTestCase(unittest.TestCase): logging.error('error message') with open(log_file, 'rb') as fo: - logged = fo.read().decode('utf8') + logged = fo.read().decode('utf-8') + self.assertIn('previous message', logged) self.assertNotIn('debug message', logged) self.assertIn('info message', logged) self.assertIn('warning message', logged) @@ -131,6 +134,30 @@ class CrawlerLoggingTestCase(unittest.TestCase): crawler.stats.get_value('log_count/INFO') - info_count, 1) self.assertEqual(crawler.stats.get_value('log_count/DEBUG', 0), 0) + def test_spider_custom_settings_log_append(self): + log_file = self.mktemp() + with open(log_file, 'wb') as fo: + fo.write('previous message\n'.encode('utf-8')) + + class MySpider(scrapy.Spider): + name = 'spider' + custom_settings = { + 'LOG_FILE': log_file, + 'LOG_FILE_APPEND': False, + # disable telnet if not available to avoid an extra warning + 'TELNETCONSOLE_ENABLED': telnet.TWISTED_CONCH_AVAILABLE, + } + + configure_logging() + crawler = Crawler(MySpider, {}) + logging.debug('debug message') + + with open(log_file, 'rb') as fo: + logged = fo.read().decode('utf-8') + + self.assertNotIn('previous message', logged) + self.assertIn('debug message', logged) + class SpiderLoaderWithWrongInterface: From 98ee3ddffeeabdc896c3f2cffce8310ebfc97570 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 15 Oct 2021 17:18:46 +0200 Subject: [PATCH 035/126] Freeze flake8 --- tox.ini | 1 + 1 file changed, 1 insertion(+) diff --git a/tox.ini b/tox.ini index e274fc8d2..07552ba8d 100644 --- a/tox.ini +++ b/tox.ini @@ -57,6 +57,7 @@ deps = # Twisted[http2] is required to import some files Twisted[http2]>=17.9.0 pytest-flake8 + flake8==3.9.2 # https://github.com/tholo/pytest-flake8/issues/81 commands = py.test --flake8 {posargs:docs scrapy tests} From d774d6a9c46cb9697d8fe64a55da9ae321be7539 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 15 Oct 2021 17:25:22 +0200 Subject: [PATCH 036/126] Remove unused variable --- tests/test_crawler.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_crawler.py b/tests/test_crawler.py index a80ad4388..be067155e 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -149,7 +149,7 @@ class CrawlerLoggingTestCase(unittest.TestCase): } configure_logging() - crawler = Crawler(MySpider, {}) + Crawler(MySpider, {}) logging.debug('debug message') with open(log_file, 'rb') as fo: From aec7146e2f870f5e8f0f58bd596b1ec40c64b3c7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Raphael=20Tom=C3=A9=20Santana?= Date: Fri, 15 Oct 2021 20:38:53 -0300 Subject: [PATCH 037/126] Add how Scrapy is pronounced to the docs --- docs/intro/overview.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index dd80c7bd0..d75f7f636 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -4,7 +4,7 @@ Scrapy at a glance ================== -Scrapy is an application framework for crawling web sites and extracting +Scrapy (pronounced SKRAY-peye /ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. From 027ecd8686d7da74b73412e4aa38d8be36b5b9f1 Mon Sep 17 00:00:00 2001 From: raphaelts3 Date: Sat, 16 Oct 2021 10:52:54 -0300 Subject: [PATCH 038/126] Update docs/intro/overview.rst Co-authored-by: azzamsa <17734314+azzamsa@users.noreply.github.com> --- docs/intro/overview.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index d75f7f636..405bf845d 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -4,7 +4,7 @@ Scrapy at a glance ================== -Scrapy (pronounced SKRAY-peye /ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting +Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. From cfff79cee6a97528185b7d24e2b660b99c07945f Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com> Date: Mon, 18 Oct 2021 17:09:17 -0300 Subject: [PATCH 039/126] Make Python 3.10 support official (#5265) --- .github/workflows/checks.yml | 8 ++++---- .github/workflows/publish.yml | 4 ++-- .github/workflows/tests-macos.yml | 2 +- .github/workflows/tests-ubuntu.yml | 14 ++++++-------- .readthedocs.yml | 9 +++++---- setup.py | 1 + tox.ini | 1 + 7 files changed, 20 insertions(+), 19 deletions(-) diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 6bdfcb5dc..80df9469d 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -8,10 +8,10 @@ jobs: fail-fast: false matrix: include: - - python-version: 3.9 + - python-version: "3.10" env: TOXENV: security - - python-version: 3.9 + - python-version: "3.10" env: TOXENV: flake8 # Pylint requires installing reppy, which does not support Python 3.9 @@ -20,10 +20,10 @@ jobs: env: TOXENV: pylint TOX_PIP_VERSION: 20.3.3 - - python-version: 3.9 + - python-version: 3.6 env: TOXENV: typing - - python-version: 3.8 # Keep in sync with .readthedocs.yml + - python-version: "3.10" # Keep in sync with .readthedocs.yml env: TOXENV: docs diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index b48066ea4..44b682830 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -9,10 +9,10 @@ jobs: steps: - uses: actions/checkout@v2 - - name: Set up Python 3.9 + - name: Set up Python uses: actions/setup-python@v2 with: - python-version: 3.9 + python-version: "3.10" - name: Check Tag id: check-release-tag diff --git a/.github/workflows/tests-macos.yml b/.github/workflows/tests-macos.yml index 095ca1013..3aaf688c7 100644 --- a/.github/workflows/tests-macos.yml +++ b/.github/workflows/tests-macos.yml @@ -7,7 +7,7 @@ jobs: strategy: fail-fast: false matrix: - python-version: [3.6, 3.7, 3.8, 3.9] + python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"] steps: - uses: actions/checkout@v2 diff --git a/.github/workflows/tests-ubuntu.yml b/.github/workflows/tests-ubuntu.yml index ef1c8362f..5ea50e644 100644 --- a/.github/workflows/tests-ubuntu.yml +++ b/.github/workflows/tests-ubuntu.yml @@ -17,6 +17,12 @@ jobs: - python-version: 3.9 env: TOXENV: py + - python-version: "3.10" + env: + TOXENV: py + - python-version: "3.10" + env: + TOXENV: asyncio - python-version: pypy3 env: TOXENV: pypy3 @@ -42,14 +48,6 @@ jobs: TOXENV: extra-deps TOX_PIP_VERSION: 20.3.3 - # 3.10-pre - - python-version: "3.10.0-beta.4" - env: - TOXENV: py - - python-version: "3.10.0-beta.4" - env: - TOXENV: asyncio - steps: - uses: actions/checkout@v2 diff --git a/.readthedocs.yml b/.readthedocs.yml index 80a1cd036..390be3749 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -5,12 +5,13 @@ sphinx: fail_on_warning: true build: - image: latest + os: ubuntu-20.04 + tools: + # For available versions, see: + # https://docs.readthedocs.io/en/stable/config-file/v2.html#build-tools-python + python: "3.10" # Keep in sync with .github/workflows/checks.yml python: - # For available versions, see: - # https://docs.readthedocs.io/en/stable/config-file/v2.html#build-image - version: 3.8 # Keep in sync with .github/workflows/checks.yml install: - requirements: docs/requirements.txt - path: . diff --git a/setup.py b/setup.py index ed2b6e347..3a6ff2836 100644 --- a/setup.py +++ b/setup.py @@ -87,6 +87,7 @@ setup( 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', + 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', 'Topic :: Internet :: WWW/HTTP', diff --git a/tox.ini b/tox.ini index 07552ba8d..021dd9988 100644 --- a/tox.ini +++ b/tox.ini @@ -41,6 +41,7 @@ deps = types-pyOpenSSL==20.0.3 types-setuptools==57.0.0 commands = + pip install types-dataclasses # remove once py36 support is dropped mypy --show-error-codes {posargs: scrapy tests} [testenv:security] From 144d1eb8341c427fa1fae109db3e9487f255d3fe Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Fri, 22 Oct 2021 21:46:01 +0500 Subject: [PATCH 040/126] Add Deferred-to-Future helpers (#5288) --- conftest.py | 6 ++++ docs/topics/asyncio.rst | 17 ++++++++++ docs/topics/coroutines.rst | 13 +++++--- docs/topics/item-pipeline.rst | 4 ++- pytest.ini | 1 + scrapy/utils/defer.py | 63 +++++++++++++++++++++++++++++++++-- tests/test_pipelines.py | 30 ++++++++++++++++- 7 files changed, 126 insertions(+), 8 deletions(-) diff --git a/conftest.py b/conftest.py index 05b4ccdad..117087790 100644 --- a/conftest.py +++ b/conftest.py @@ -75,6 +75,12 @@ def only_asyncio(request, reactor_pytest): pytest.skip('This test is only run with --reactor=asyncio') +@pytest.fixture(autouse=True) +def only_not_asyncio(request, reactor_pytest): + if request.node.get_closest_marker('only_not_asyncio') and reactor_pytest == 'asyncio': + pytest.skip('This test is only run without --reactor=asyncio') + + def pytest_configure(config): if config.getoption("--reactor") == "asyncio": install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor") diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 82c5f271f..28241ae24 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -39,5 +39,22 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the :setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class to use it instead of the default asyncio event loop. +.. _asyncio-await-dfd: +Awaiting on Deferreds +===================== +When the asyncio reactor isn't installed, you can await on Deferreds in the +coroutines directly. When it is installed, this is not possible anymore, due to +specifics of the Scrapy coroutine integration (the coroutines are wrapped into +:class:`asyncio.Future` objects, not into +:class:`~twisted.internet.defer.Deferred` directly), and you need to wrap them into +Futures. Scrapy provides two helpers for this: + +.. autofunction:: scrapy.utils.defer.deferred_to_future +.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future +.. tip:: If you need to use these functions in code that aims to be compatible + with lower versions of Scrapy that do not provide these functions, + down to Scrapy 2.0 (earlier versions do not support + :mod:`asyncio`), you can copy the implementation of these functions + into your own code. diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 0904637b0..2aef755c7 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -75,23 +75,28 @@ coroutines, functions that return Deferreds and functions that return :term:`awaitable objects ` such as :class:`~asyncio.Future`. This means you can use many useful Python libraries providing such code:: - class MySpider(Spider): + class MySpiderDeferred(Spider): # ... - async def parse_with_deferred(self, response): + async def parse(self, response): additional_response = await treq.get('https://additional.url') additional_data = await treq.content(additional_response) # ... use response and additional_data to yield items and requests - async def parse_with_asyncio(self, response): + class MySpiderAsyncio(Spider): + # ... + async def parse(self, response): async with aiohttp.ClientSession() as session: async with session.get('https://additional.url') as additional_response: - additional_data = await r.text() + additional_data = await additional_response.text() # ... use response and additional_data to yield items and requests .. note:: Many libraries that use coroutines, such as `aio-libs`_, require the :mod:`asyncio` loop and to use them you need to :doc:`enable asyncio support in Scrapy`. +.. note:: If you want to ``await`` on Deferreds while using the asyncio reactor, + you need to :ref:`wrap them`. + Common use cases for asynchronous code include: * requesting data from websites, databases and other services (in callbacks, diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index 5351a2293..391751364 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -190,6 +190,8 @@ item. import scrapy from itemadapter import ItemAdapter + from scrapy.utils.defer import maybe_deferred_to_future + class ScreenshotPipeline: """Pipeline that uses Splash to render screenshot of @@ -202,7 +204,7 @@ item. encoded_item_url = quote(adapter["url"]) screenshot_url = self.SPLASH_URL.format(encoded_item_url) request = scrapy.Request(screenshot_url) - response = await spider.crawler.engine.download(request, spider) + response = await maybe_deferred_to_future(spider.crawler.engine.download(request, spider)) if response.status != 200: # Error happened, return item. diff --git a/pytest.ini b/pytest.ini index 6de08c78d..fa5d6b34f 100644 --- a/pytest.ini +++ b/pytest.ini @@ -20,5 +20,6 @@ addopts = --ignore=docs/utils markers = only_asyncio: marks tests as only enabled when --reactor=asyncio is passed + only_not_asyncio: marks tests as only enabled when --reactor=asyncio is not passed filterwarnings= ignore::DeprecationWarning:twisted.web.test.test_webclient diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.py index b02bfdccb..d7adc0a77 100644 --- a/scrapy/utils/defer.py +++ b/scrapy/utils/defer.py @@ -3,9 +3,16 @@ Helper functions for dealing with Twisted deferreds """ import asyncio import inspect -from collections.abc import Coroutine +from asyncio import Future from functools import wraps -from typing import Any, Callable, Generator, Iterable +from typing import ( + Any, + Callable, + Coroutine, + Generator, + Iterable, + Union +) from twisted.internet import defer from twisted.internet.defer import Deferred, DeferredList, ensureDeferred @@ -171,3 +178,55 @@ def maybeDeferred_coro(f: Callable, *args, **kw) -> Deferred: return defer.fail(result) else: return defer.succeed(result) + + +def deferred_to_future(d: Deferred) -> Future: + """ + .. versionadded:: VERSION + + Return an :class:`asyncio.Future` object that wraps *d*. + + When :ref:`using the asyncio reactor `, you cannot await + on :class:`~twisted.internet.defer.Deferred` objects from :ref:`Scrapy + callables defined as coroutines `, you can only await on + ``Future`` objects. Wrapping ``Deferred`` objects into ``Future`` objects + allows you to wait on them:: + + class MySpider(Spider): + ... + async def parse(self, response): + d = treq.get('https://example.com/additional') + additional_response = await deferred_to_future(d) + """ + return d.asFuture(asyncio.get_event_loop()) + + +def maybe_deferred_to_future(d: Deferred) -> Union[Deferred, Future]: + """ + .. versionadded:: VERSION + + Return *d* as an object that can be awaited from a :ref:`Scrapy callable + defined as a coroutine `. + + What you can await in Scrapy callables defined as coroutines depends on the + value of :setting:`TWISTED_REACTOR`: + + - When not using the asyncio reactor, you can only await on + :class:`~twisted.internet.defer.Deferred` objects. + + - When :ref:`using the asyncio reactor `, you can only + await on :class:`asyncio.Future` objects. + + If you want to write code that uses ``Deferred`` objects but works with any + reactor, use this function on all ``Deferred`` objects:: + + class MySpider(Spider): + ... + async def parse(self, response): + d = treq.get('https://example.com/additional') + extra_response = await maybe_deferred_to_future(d) + """ + if not is_asyncio_reactor_installed(): + return d + else: + return deferred_to_future(d) diff --git a/tests/test_pipelines.py b/tests/test_pipelines.py index ff3af9a74..8e432b913 100644 --- a/tests/test_pipelines.py +++ b/tests/test_pipelines.py @@ -6,6 +6,7 @@ from twisted.internet.defer import Deferred from twisted.trial import unittest from scrapy import Spider, signals, Request +from scrapy.utils.defer import maybe_deferred_to_future, deferred_to_future from scrapy.utils.test import get_crawler, get_from_asyncio_queue from tests.mockserver import MockServer @@ -31,18 +32,38 @@ class DeferredPipeline: class AsyncDefPipeline: async def process_item(self, item, spider): - await defer.succeed(42) + d = Deferred() + from twisted.internet import reactor + reactor.callLater(0, d.callback, None) + await maybe_deferred_to_future(d) item['pipeline_passed'] = True return item class AsyncDefAsyncioPipeline: async def process_item(self, item, spider): + d = Deferred() + from twisted.internet import reactor + reactor.callLater(0, d.callback, None) + await deferred_to_future(d) await asyncio.sleep(0.2) item['pipeline_passed'] = await get_from_asyncio_queue(True) return item +class AsyncDefNotAsyncioPipeline: + async def process_item(self, item, spider): + d1 = Deferred() + from twisted.internet import reactor + reactor.callLater(0, d1.callback, None) + await d1 + d2 = Deferred() + reactor.callLater(0, d2.callback, None) + await maybe_deferred_to_future(d2) + item['pipeline_passed'] = True + return item + + class ItemSpider(Spider): name = 'itemspider' @@ -99,3 +120,10 @@ class PipelineTestCase(unittest.TestCase): crawler = self._create_crawler(AsyncDefAsyncioPipeline) yield crawler.crawl(mockserver=self.mockserver) self.assertEqual(len(self.items), 1) + + @mark.only_not_asyncio() + @defer.inlineCallbacks + def test_asyncdef_not_asyncio_pipeline(self): + crawler = self._create_crawler(AsyncDefNotAsyncioPipeline) + yield crawler.crawl(mockserver=self.mockserver) + self.assertEqual(len(self.items), 1) From 51adf71b1b4ae703bb1cb561882c710eedac2359 Mon Sep 17 00:00:00 2001 From: azzamsa Date: Sun, 24 Oct 2021 10:52:56 +0700 Subject: [PATCH 041/126] refactor: use `pytest` command as the recommended entry point `pytest` is recommended command since pytest 3.0. There is a possibility for `py.test` to be deprecated or even removed. https://github.com/pytest-dev/pytest/issues/1629 --- tox.ini | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tox.ini b/tox.ini index 021dd9988..e4514f512 100644 --- a/tox.ini +++ b/tox.ini @@ -29,7 +29,7 @@ passenv = #allow tox virtualenv to upgrade pip/wheel/setuptools download = true commands = - py.test --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests} + pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests} install_command = pip install -U -ctests/upper-constraints.txt {opts} {packages} @@ -60,7 +60,7 @@ deps = pytest-flake8 flake8==3.9.2 # https://github.com/tholo/pytest-flake8/issues/81 commands = - py.test --flake8 {posargs:docs scrapy tests} + pytest --flake8 {posargs:docs scrapy tests} [testenv:pylint] basepython = python3 @@ -142,7 +142,7 @@ setenv = [testenv:pypy3] basepython = pypy3 commands = - py.test {posargs:--durations=10 docs scrapy tests} + pytest {posargs:--durations=10 docs scrapy tests} [testenv:pypy3-pinned] basepython = {[testenv:pypy3]basepython} From 67994d1dddcba4c1fe53dd7bdf7b978d1733ae1b Mon Sep 17 00:00:00 2001 From: azzamsa Date: Wed, 27 Oct 2021 21:55:05 +0700 Subject: [PATCH 042/126] fix: `CodeBlockParser` has been renamed to `PythonCodeBlockParser` --- docs/conftest.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/conftest.py b/docs/conftest.py index 8c735e838..a0636f8ac 100644 --- a/docs/conftest.py +++ b/docs/conftest.py @@ -3,7 +3,11 @@ from doctest import ELLIPSIS, NORMALIZE_WHITESPACE from scrapy.http.response.html import HtmlResponse from sybil import Sybil -from sybil.parsers.codeblock import CodeBlockParser +try: + # >2.0.1 + from sybil.parsers.codeblock import PythonCodeBlockParser +except ImportError: + from sybil.parsers.codeblock import CodeBlockParser as PythonCodeBlockParser from sybil.parsers.doctest import DocTestParser from sybil.parsers.skip import skip @@ -21,7 +25,7 @@ def setup(namespace): pytest_collect_file = Sybil( parsers=[ DocTestParser(optionflags=ELLIPSIS | NORMALIZE_WHITESPACE), - CodeBlockParser(future_imports=['print_function']), + PythonCodeBlockParser(future_imports=['print_function']), skip, ], pattern='*.rst', From 55cce25a799ab0ed9d5edd60ff0f35988318e9b9 Mon Sep 17 00:00:00 2001 From: azzamsa Date: Mon, 25 Oct 2021 21:14:11 +0700 Subject: [PATCH 043/126] test: `test_format_engine_status` --- tests/test_crawl.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/test_crawl.py b/tests/test_crawl.py index 84bac9b50..7bda3bef2 100644 --- a/tests/test_crawl.py +++ b/tests/test_crawl.py @@ -274,6 +274,28 @@ with multiples lines self.assertEqual(s['engine.spider.name'], crawler.spider.name) self.assertEqual(s['len(engine.scraper.slot.active)'], 1) + @defer.inlineCallbacks + def test_format_engine_status(self): + from scrapy.utils.engine import format_engine_status + est = [] + + def cb(response): + est.append(format_engine_status(crawler.engine)) + + crawler = self.runner.create_crawler(SingleRequestSpider) + yield crawler.crawl(seed=self.mockserver.url('/'), callback_func=cb, mockserver=self.mockserver) + self.assertEqual(len(est), 1, est) + est = est[0].split("\n")[2:-2] # remove header & footer + # convert to dict + est = [x.split(":") for x in est] + est = [x for sublist in est for x in sublist] # flatten + est = [x.lstrip().rstrip() for x in est] + it = iter(est) + s = dict(zip(it, it)) + + self.assertEqual(s['engine.spider.name'], crawler.spider.name) + self.assertEqual(s['len(engine.scraper.slot.active)'], '1') + @defer.inlineCallbacks def test_graceful_crawl_error_handling(self): """ From 28eba610e22c0d2a42e830b4e64746edf44598f9 Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Mon, 15 Nov 2021 12:24:54 +0500 Subject: [PATCH 044/126] Re-enable Windows tests for Python 3.9 and 3.10. (#5316) --- .github/workflows/tests-windows.yml | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/.github/workflows/tests-windows.yml b/.github/workflows/tests-windows.yml index 30fda33e8..6fabf5cde 100644 --- a/.github/workflows/tests-windows.yml +++ b/.github/workflows/tests-windows.yml @@ -17,10 +17,12 @@ jobs: - python-version: 3.8 env: TOXENV: py - # https://twistedmatrix.com/trac/ticket/9990 - #- python-version: 3.9 - #env: - #TOXENV: py + - python-version: 3.9 + env: + TOXENV: py + - python-version: "3.10" + env: + TOXENV: py steps: - uses: actions/checkout@v2 From f2c800c5c9f88b4b583181e9cf49eb3cd8d538f0 Mon Sep 17 00:00:00 2001 From: Samuel Marchal Date: Mon, 15 Nov 2021 11:14:54 +0100 Subject: [PATCH 045/126] Improve open_in_browser base tag injection (#5319) --- scrapy/utils/response.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/scrapy/utils/response.py b/scrapy/utils/response.py index b3ef7b463..8b109dced 100644 --- a/scrapy/utils/response.py +++ b/scrapy/utils/response.py @@ -3,8 +3,9 @@ This module provides some useful functions for working with scrapy.http.Response objects """ import os -import webbrowser +import re import tempfile +import webbrowser from typing import Any, Callable, Iterable, Optional, Tuple, Union from weakref import WeakKeyDictionary @@ -80,8 +81,9 @@ def open_in_browser( body = response.body if isinstance(response, HtmlResponse): if b'' - body = body.replace(b'', to_bytes(repl)) + repl = fr'\1' + body = re.sub(b"", b"", body, flags=re.DOTALL) + body = re.sub(rb"(|\s.*?>))", to_bytes(repl), body) ext = '.html' elif isinstance(response, TextResponse): ext = '.txt' From 75ed765476a2ac66ea8f52e7b29186864f65535c Mon Sep 17 00:00:00 2001 From: Samuel Marchal Date: Mon, 15 Nov 2021 14:31:24 +0100 Subject: [PATCH 046/126] Test coverage for open_in_browser base tag injection (#5319) --- tests/test_utils_response.py | 53 ++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/tests/test_utils_response.py b/tests/test_utils_response.py index d6f4c0bb5..0a09f6109 100644 --- a/tests/test_utils_response.py +++ b/tests/test_utils_response.py @@ -83,3 +83,56 @@ class ResponseUtilsTest(unittest.TestCase): self.assertEqual(response_status_message(200), '200 OK') self.assertEqual(response_status_message(404), '404 Not Found') self.assertEqual(response_status_message(573), "573 Unknown Status") + + def test_inject_base_url(self): + url = "http://www.example.com" + + def check_base_url(burl): + path = urlparse(burl).path + if not os.path.exists(path): + path = burl.replace('file://', '') + with open(path, "rb") as f: + bbody = f.read() + self.assertEqual(bbody.count(b''), 1) + return True + + r1 = HtmlResponse(url, body=b""" + + Dummy +

Hello world.

+ """) + r2 = HtmlResponse(url, body=b""" + + Dummy + Hello world. + """) + r3 = HtmlResponse(url, body=b""" + + Dummy + +
Hello header
+

Hello world.

+ + """) + r4 = HtmlResponse(url, body=b""" + + + Dummy +

Hello world.

+ """) + r5 = HtmlResponse(url, body=b""" + + + + Standard head + +

Hello world.

+ """) + + assert open_in_browser(r1, _openfunc=check_base_url), "Inject base url" + assert open_in_browser(r2, _openfunc=check_base_url), "Inject base url with argumented head" + assert open_in_browser(r3, _openfunc=check_base_url), "Inject unique base url with misleading tag" + assert open_in_browser(r4, _openfunc=check_base_url), "Inject unique base url with misleading comment" + assert open_in_browser(r5, _openfunc=check_base_url), "Inject unique base url with conditional comment" From c316ca45a5b1b19622c96049c9378d8c45adba60 Mon Sep 17 00:00:00 2001 From: Alex Date: Tue, 16 Nov 2021 01:20:56 -0800 Subject: [PATCH 047/126] Use augmented assignment statements (#5322) --- scrapy/core/downloader/handlers/http11.py | 2 +- scrapy/core/http2/stream.py | 4 ++-- tests/test_request_left.py | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py index 8a91d4c5e..38935667d 100644 --- a/scrapy/core/downloader/handlers/http11.py +++ b/scrapy/core/downloader/handlers/http11.py @@ -213,7 +213,7 @@ class TunnelingAgent(Agent): # proxy host and port are required for HTTP pool `key` # otherwise, same remote host connection request could reuse # a cached tunneled connection to a different proxy - key = key + self._proxyConf + key += self._proxyConf return super()._requestWithEndpoint( key=key, endpoint=endpoint, diff --git a/scrapy/core/http2/stream.py b/scrapy/core/http2/stream.py index c2a4b702f..5c393c027 100644 --- a/scrapy/core/http2/stream.py +++ b/scrapy/core/http2/stream.py @@ -285,8 +285,8 @@ class Stream: self._protocol.conn.send_data(self.stream_id, data_chunk, end_stream=False) - bytes_to_send_size = bytes_to_send_size - chunk_size - self.metadata['remaining_content_length'] = self.metadata['remaining_content_length'] - chunk_size + bytes_to_send_size -= chunk_size + self.metadata['remaining_content_length'] -= chunk_size self.metadata['remaining_content_length'] = max(0, self.metadata['remaining_content_length']) diff --git a/tests/test_request_left.py b/tests/test_request_left.py index 373b2e49c..4d4483881 100644 --- a/tests/test_request_left.py +++ b/tests/test_request_left.py @@ -22,7 +22,7 @@ class SignalCatcherSpider(Spider): return spider def on_request_left(self, request, spider): - self.caught_times = self.caught_times + 1 + self.caught_times += 1 class TestCatching(TestCase): From 6ec66c96fb962a276c2d285cd5ffa34d84925df1 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 26 Nov 2021 12:25:45 +0500 Subject: [PATCH 048/126] Fix and pin pylint. --- pylintrc | 1 + tox.ini | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/pylintrc b/pylintrc index 699686e16..2cdd6321e 100644 --- a/pylintrc +++ b/pylintrc @@ -112,6 +112,7 @@ disable=abstract-method, unused-private-member, unused-variable, unused-wildcard-import, + use-implicit-booleaness-not-comparison, used-before-assignment, useless-object-inheritance, # Required for Python 2 support useless-return, diff --git a/tox.ini b/tox.ini index e4514f512..2031a2d92 100644 --- a/tox.ini +++ b/tox.ini @@ -66,7 +66,7 @@ commands = basepython = python3 deps = {[testenv:extra-deps]deps} - pylint + pylint==2.12.1 commands = pylint conftest.py docs extras scrapy setup.py tests From 4cc039628eb4861f98f7997e90125745e30f8687 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 26 Nov 2021 19:52:03 +0500 Subject: [PATCH 049/126] Fix typing of middleware methods. --- scrapy/core/downloader/middleware.py | 5 ++++- scrapy/core/spidermw.py | 3 ++- scrapy/middleware.py | 16 ++++++++++------ 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.py index a5619d8a4..289147466 100644 --- a/scrapy/core/downloader/middleware.py +++ b/scrapy/core/downloader/middleware.py @@ -3,7 +3,7 @@ Downloader Middleware manager See documentation in docs/topics/downloader-middleware.rst """ -from typing import Callable, Union +from typing import Callable, Union, cast from twisted.internet import defer from twisted.python.failure import Failure @@ -37,6 +37,7 @@ class DownloaderMiddlewareManager(MiddlewareManager): @defer.inlineCallbacks def process_request(request: Request): for method in self.methods['process_request']: + method = cast(Callable, method) response = yield deferred_from_coro(method(request=request, spider=spider)) if response is not None and not isinstance(response, (Response, Request)): raise _InvalidOutput( @@ -55,6 +56,7 @@ class DownloaderMiddlewareManager(MiddlewareManager): return response for method in self.methods['process_response']: + method = cast(Callable, method) response = yield deferred_from_coro(method(request=request, response=response, spider=spider)) if not isinstance(response, (Response, Request)): raise _InvalidOutput( @@ -69,6 +71,7 @@ class DownloaderMiddlewareManager(MiddlewareManager): def process_exception(failure: Failure): exception = failure.value for method in self.methods['process_exception']: + method = cast(Callable, method) response = yield deferred_from_coro(method(request=request, exception=exception, spider=spider)) if response is not None and not isinstance(response, (Response, Request)): raise _InvalidOutput( diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py index 7e58521ac..7cdc28284 100644 --- a/scrapy/core/spidermw.py +++ b/scrapy/core/spidermw.py @@ -4,7 +4,7 @@ Spider Middleware manager See documentation in docs/topics/spider-middleware.rst """ from itertools import islice -from typing import Any, Callable, Generator, Iterable, Union +from typing import Any, Callable, Generator, Iterable, Union, cast from twisted.internet.defer import Deferred from twisted.python.failure import Failure @@ -47,6 +47,7 @@ class SpiderMiddlewareManager(MiddlewareManager): def _process_spider_input(self, scrape_func: ScrapeFunc, response: Response, request: Request, spider: Spider) -> Any: for method in self.methods['process_spider_input']: + method = cast(Callable, method) try: result = method(response=response, spider=spider) if result is not None: diff --git a/scrapy/middleware.py b/scrapy/middleware.py index bbec38086..e8f60287a 100644 --- a/scrapy/middleware.py +++ b/scrapy/middleware.py @@ -1,7 +1,7 @@ import logging import pprint from collections import defaultdict, deque -from typing import Callable, Deque, Dict +from typing import Callable, Deque, Dict, Optional, cast, Iterable from twisted.internet.defer import Deferred @@ -21,7 +21,8 @@ class MiddlewareManager: def __init__(self, *middlewares): self.middlewares = middlewares - self.methods: Dict[str, Deque[Callable]] = defaultdict(deque) + # Optional because process_spider_output and process_spider_exception can be None + self.methods: Dict[str, Deque[Optional[Callable]]] = defaultdict(deque) for mw in middlewares: self._add_middleware(mw) @@ -64,14 +65,17 @@ class MiddlewareManager: self.methods['close_spider'].appendleft(mw.close_spider) def _process_parallel(self, methodname: str, obj, *args) -> Deferred: - return process_parallel(self.methods[methodname], obj, *args) + methods = cast(Iterable[Callable], self.methods[methodname]) + return process_parallel(methods, obj, *args) def _process_chain(self, methodname: str, obj, *args) -> Deferred: - return process_chain(self.methods[methodname], obj, *args) + methods = cast(Iterable[Callable], self.methods[methodname]) + return process_chain(methods, obj, *args) def _process_chain_both(self, cb_methodname: str, eb_methodname: str, obj, *args) -> Deferred: - return process_chain_both(self.methods[cb_methodname], - self.methods[eb_methodname], obj, *args) + cb_methods = cast(Iterable[Callable], self.methods[cb_methodname]) + eb_methods = cast(Iterable[Callable], self.methods[eb_methodname]) + return process_chain_both(cb_methods, eb_methods, obj, *args) def open_spider(self, spider: Spider) -> Deferred: return self._process_parallel('open_spider', spider) From eb62906c3e4c1e1f8e3e6c7965a04d5e65c61907 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Wed, 1 Dec 2021 17:40:41 +0500 Subject: [PATCH 050/126] Extract utils.log.log_reactor_info(). --- scrapy/utils/log.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/scrapy/utils/log.py b/scrapy/utils/log.py index 0441c0358..9887ecc40 100644 --- a/scrapy/utils/log.py +++ b/scrapy/utils/log.py @@ -143,7 +143,7 @@ def _get_handler(settings): return handler -def log_scrapy_info(settings): +def log_scrapy_info(settings: Settings) -> None: logger.info("Scrapy %(version)s started (bot: %(bot)s)", {'version': scrapy.__version__, 'bot': settings['BOT_NAME']}) versions = [ @@ -152,6 +152,10 @@ def log_scrapy_info(settings): if name != "Scrapy" ] logger.info("Versions: %(versions)s", {'versions': ", ".join(versions)}) + log_reactor_info() + + +def log_reactor_info() -> None: from twisted.internet import reactor logger.debug("Using reactor: %s.%s", reactor.__module__, reactor.__class__.__name__) from twisted.internet import asyncioreactor From 6483dfdbe17cd66c409435b95a05850a3c94b5ee Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Wed, 1 Dec 2021 19:53:39 +0500 Subject: [PATCH 051/126] Move install_shutdown_handlers() from __init__() to start(). --- scrapy/crawler.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scrapy/crawler.py b/scrapy/crawler.py index 578016536..357f14dc0 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -278,7 +278,6 @@ class CrawlerProcess(CrawlerRunner): def __init__(self, settings=None, install_root_handler=True): super().__init__(settings) - install_shutdown_handlers(self._signal_shutdown) configure_logging(self.settings, install_root_handler) log_scrapy_info(self.settings) @@ -318,6 +317,7 @@ class CrawlerProcess(CrawlerRunner): return d.addBoth(self._stop_reactor) + install_shutdown_handlers(self._signal_shutdown) resolver_class = load_object(self.settings["DNS_RESOLVER"]) resolver = create_instance(resolver_class, self.settings, self, reactor=reactor) resolver.install_on_reactor() From d6a384b3cfdb36cd19b32942479487a9e47c244b Mon Sep 17 00:00:00 2001 From: yogender26 <95638485+yogender26@users.noreply.github.com> Date: Thu, 23 Dec 2021 04:09:05 +0530 Subject: [PATCH 052/126] corrrection of coma (#5347) --- scrapy/commands/__init__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scrapy/commands/__init__.py b/scrapy/commands/__init__.py index 6e77551c6..5f1dabd33 100644 --- a/scrapy/commands/__init__.py +++ b/scrapy/commands/__init__.py @@ -43,14 +43,14 @@ class ScrapyCommand: def long_desc(self): """A long description of the command. Return short description when not - available. It cannot contain newlines, since contents will be formatted + available. It cannot contain newlines since contents will be formatted by optparser which removes newlines and wraps text. """ return self.short_desc() def help(self): """An extensive help for the command. It will be shown when using the - "help" command. It can contain newlines, since no post-formatting will + "help" command. It can contain newlines since no post-formatting will be applied to its contents. """ return self.long_desc() From 46ef9cf771789f1db513bbf2f65243d3320ce695 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Wed, 22 Dec 2021 21:24:59 +0500 Subject: [PATCH 053/126] Don't install non-working shutdown handlers in `scrapy shell`. --- scrapy/commands/shell.py | 2 +- scrapy/crawler.py | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/scrapy/commands/shell.py b/scrapy/commands/shell.py index d1944df3d..de81986d8 100644 --- a/scrapy/commands/shell.py +++ b/scrapy/commands/shell.py @@ -75,6 +75,6 @@ class Command(ScrapyCommand): def _start_crawler_thread(self): t = Thread(target=self.crawler_process.start, - kwargs={'stop_after_crawl': False}) + kwargs={'stop_after_crawl': False, 'install_signal_handlers': False}) t.daemon = True t.start() diff --git a/scrapy/crawler.py b/scrapy/crawler.py index 357f14dc0..e54ad9750 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -297,7 +297,7 @@ class CrawlerProcess(CrawlerRunner): {'signame': signame}) reactor.callFromThread(self._stop_reactor) - def start(self, stop_after_crawl=True): + def start(self, stop_after_crawl=True, install_signal_handlers=True): """ This method starts a :mod:`~twisted.internet.reactor`, adjusts its pool size to :setting:`REACTOR_THREADPOOL_MAXSIZE`, and installs a DNS cache @@ -308,6 +308,9 @@ class CrawlerProcess(CrawlerRunner): :param bool stop_after_crawl: stop or not the reactor when all crawlers have finished + + :param bool install_signal_handlers: whether to install the shutdown + handlers (default: True) """ from twisted.internet import reactor if stop_after_crawl: @@ -317,7 +320,8 @@ class CrawlerProcess(CrawlerRunner): return d.addBoth(self._stop_reactor) - install_shutdown_handlers(self._signal_shutdown) + if install_signal_handlers: + install_shutdown_handlers(self._signal_shutdown) resolver_class = load_object(self.settings["DNS_RESOLVER"]) resolver = create_instance(resolver_class, self.settings, self, reactor=reactor) resolver.install_on_reactor() From 60c8838554a79e70c22a7c6a57baedfcaf521444 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 23 Dec 2021 16:07:18 +0500 Subject: [PATCH 054/126] Move installing the reactor from CrawlerProcess to Crawler. --- scrapy/crawler.py | 31 ++++++++++++++++++++----------- scrapy/utils/log.py | 1 - tests/test_crawler.py | 5 ++++- 3 files changed, 24 insertions(+), 13 deletions(-) diff --git a/scrapy/crawler.py b/scrapy/crawler.py index e54ad9750..95cfb1bd1 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -25,6 +25,7 @@ from scrapy.utils.log import ( configure_logging, get_scrapy_root_handler, install_scrapy_root_handler, + log_reactor_info, log_scrapy_info, LogCounterHandler, ) @@ -38,7 +39,7 @@ logger = logging.getLogger(__name__) class Crawler: - def __init__(self, spidercls, settings=None): + def __init__(self, spidercls, settings=None, init_reactor: bool = False): if isinstance(spidercls, Spider): raise ValueError('The spidercls argument must be a class, not an object') @@ -69,6 +70,19 @@ class Crawler: lf_cls = load_object(self.settings['LOG_FORMATTER']) self.logformatter = lf_cls.from_crawler(self) + + if init_reactor: + # this needs to be done after the spider settings are merged, + # but before something imports twisted.internet.reactor + if self.settings.get("TWISTED_REACTOR"): + install_reactor(self.settings["TWISTED_REACTOR"], self.settings["ASYNCIO_EVENT_LOOP"]) + else: + from twisted.internet import default + default.install() + log_reactor_info() + if self.settings.get("TWISTED_REACTOR"): + verify_installed_reactor(self.settings["TWISTED_REACTOR"]) + self.extensions = ExtensionManager.from_crawler(self) self.settings.freeze() @@ -153,7 +167,6 @@ class CrawlerRunner: self._crawlers = set() self._active = set() self.bootstrap_failed = False - self._handle_twisted_reactor() @property def spiders(self): @@ -247,10 +260,6 @@ class CrawlerRunner: while self._active: yield defer.DeferredList(self._active) - def _handle_twisted_reactor(self): - if self.settings.get("TWISTED_REACTOR"): - verify_installed_reactor(self.settings["TWISTED_REACTOR"]) - class CrawlerProcess(CrawlerRunner): """ @@ -297,6 +306,11 @@ class CrawlerProcess(CrawlerRunner): {'signame': signame}) reactor.callFromThread(self._stop_reactor) + def _create_crawler(self, spidercls): + if isinstance(spidercls, str): + spidercls = self.spider_loader.load(spidercls) + return Crawler(spidercls, self.settings, init_reactor=True) + def start(self, stop_after_crawl=True, install_signal_handlers=True): """ This method starts a :mod:`~twisted.internet.reactor`, adjusts its pool @@ -341,8 +355,3 @@ class CrawlerProcess(CrawlerRunner): reactor.stop() except RuntimeError: # raised if already stopped or in shutdown stage pass - - def _handle_twisted_reactor(self): - if self.settings.get("TWISTED_REACTOR"): - install_reactor(self.settings["TWISTED_REACTOR"], self.settings["ASYNCIO_EVENT_LOOP"]) - super()._handle_twisted_reactor() diff --git a/scrapy/utils/log.py b/scrapy/utils/log.py index 9887ecc40..78e302d19 100644 --- a/scrapy/utils/log.py +++ b/scrapy/utils/log.py @@ -152,7 +152,6 @@ def log_scrapy_info(settings: Settings) -> None: if name != "Scrapy" ] logger.info("Versions: %(versions)s", {'versions': ", ".join(versions)}) - log_reactor_info() def log_reactor_info() -> None: diff --git a/tests/test_crawler.py b/tests/test_crawler.py index be067155e..118cb631b 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -271,6 +271,7 @@ class CrawlerRunnerHasSpider(unittest.TestCase): self.assertEqual(runner.bootstrap_failed, True) + @defer.inlineCallbacks def test_crawler_runner_asyncio_enabled_true(self): if self.reactor_pytest == 'asyncio': CrawlerRunner(settings={ @@ -279,9 +280,10 @@ class CrawlerRunnerHasSpider(unittest.TestCase): else: msg = r"The installed reactor \(.*?\) does not match the requested one \(.*?\)" with self.assertRaisesRegex(Exception, msg): - CrawlerRunner(settings={ + runner = CrawlerRunner(settings={ "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", }) + yield runner.crawl(NoRequestsSpider) @defer.inlineCallbacks # https://twistedmatrix.com/trac/ticket/9766 @@ -301,6 +303,7 @@ class CrawlerRunnerHasSpider(unittest.TestCase): runner = CrawlerProcess(settings={ "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", }) + yield runner.crawl(NoRequestsSpider) @defer.inlineCallbacks def test_crawler_process_asyncio_enabled_false(self): From 041699b54cfa6cde9f886a98ff300e3276e2eaad Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 23 Dec 2021 16:14:47 +0500 Subject: [PATCH 055/126] Remove tests that want to modify the test process reactor. --- tests/test_crawler.py | 27 --------------------------- 1 file changed, 27 deletions(-) diff --git a/tests/test_crawler.py b/tests/test_crawler.py index 118cb631b..f445c181e 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -285,33 +285,6 @@ class CrawlerRunnerHasSpider(unittest.TestCase): }) yield runner.crawl(NoRequestsSpider) - @defer.inlineCallbacks - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") - def test_crawler_process_asyncio_enabled_true(self): - with LogCapture(level=logging.DEBUG) as log: - if self.reactor_pytest == 'asyncio': - runner = CrawlerProcess(settings={ - "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", - }) - yield runner.crawl(NoRequestsSpider) - self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", str(log)) - else: - msg = r"The installed reactor \(.*?\) does not match the requested one \(.*?\)" - with self.assertRaisesRegex(Exception, msg): - runner = CrawlerProcess(settings={ - "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", - }) - yield runner.crawl(NoRequestsSpider) - - @defer.inlineCallbacks - def test_crawler_process_asyncio_enabled_false(self): - runner = CrawlerProcess(settings={"TWISTED_REACTOR": None}) - with LogCapture(level=logging.DEBUG) as log: - yield runner.crawl(NoRequestsSpider) - self.assertNotIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", str(log)) - class ScriptRunnerMixin: def run_script(self, script_name, *script_args): From ebcafdf4a9e0692bf301546b6d60465b3b2c4b06 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 23 Dec 2021 16:35:26 +0500 Subject: [PATCH 056/126] Add tests for TWISTED_REACTOR in custom_settings. --- .../twisted_reactor_custom_settings.py | 14 +++++++++++ ...wisted_reactor_custom_settings_conflict.py | 22 +++++++++++++++++ .../twisted_reactor_custom_settings_same.py | 21 ++++++++++++++++ tests/test_crawler.py | 24 +++++++++++++++++++ 4 files changed, 81 insertions(+) create mode 100644 tests/CrawlerProcess/twisted_reactor_custom_settings.py create mode 100644 tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py create mode 100644 tests/CrawlerProcess/twisted_reactor_custom_settings_same.py diff --git a/tests/CrawlerProcess/twisted_reactor_custom_settings.py b/tests/CrawlerProcess/twisted_reactor_custom_settings.py new file mode 100644 index 000000000..56304bd23 --- /dev/null +++ b/tests/CrawlerProcess/twisted_reactor_custom_settings.py @@ -0,0 +1,14 @@ +import scrapy +from scrapy.crawler import CrawlerProcess + + +class AsyncioReactorSpider(scrapy.Spider): + name = 'asyncio_reactor' + custom_settings = { + "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", + } + + +process = CrawlerProcess() +process.crawl(AsyncioReactorSpider) +process.start() diff --git a/tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py b/tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py new file mode 100644 index 000000000..9a6c01d72 --- /dev/null +++ b/tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py @@ -0,0 +1,22 @@ +import scrapy +from scrapy.crawler import CrawlerProcess + + +class PollReactorSpider(scrapy.Spider): + name = 'poll_reactor' + custom_settings = { + "TWISTED_REACTOR": "twisted.internet.pollreactor.PollReactor", + } + + +class AsyncioReactorSpider(scrapy.Spider): + name = 'asyncio_reactor' + custom_settings = { + "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", + } + + +process = CrawlerProcess() +process.crawl(PollReactorSpider) +process.crawl(AsyncioReactorSpider) +process.start() diff --git a/tests/CrawlerProcess/twisted_reactor_custom_settings_same.py b/tests/CrawlerProcess/twisted_reactor_custom_settings_same.py new file mode 100644 index 000000000..1f5a44010 --- /dev/null +++ b/tests/CrawlerProcess/twisted_reactor_custom_settings_same.py @@ -0,0 +1,21 @@ +import scrapy +from scrapy.crawler import CrawlerProcess + + +class AsyncioReactorSpider1(scrapy.Spider): + name = 'asyncio_reactor1' + custom_settings = { + "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", + } + +class AsyncioReactorSpider2(scrapy.Spider): + name = 'asyncio_reactor2' + custom_settings = { + "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", + } + + +process = CrawlerProcess() +process.crawl(AsyncioReactorSpider1) +process.crawl(AsyncioReactorSpider2) +process.start() diff --git a/tests/test_crawler.py b/tests/test_crawler.py index f445c181e..6d6763aec 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -361,6 +361,30 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase): self.assertIn("Spider closed (finished)", log) self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log) + # https://twistedmatrix.com/trac/ticket/9766 + @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), + "the asyncio reactor is broken on Windows when running Python ≥ 3.8") + def test_reactor_asyncio_custom_settings(self): + log = self.run_script("twisted_reactor_custom_settings.py") + self.assertIn("Spider closed (finished)", log) + self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log) + + # https://twistedmatrix.com/trac/ticket/9766 + @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), + "the asyncio reactor is broken on Windows when running Python ≥ 3.8") + def test_reactor_asyncio_custom_settings_same(self): + log = self.run_script("twisted_reactor_custom_settings_same.py") + self.assertIn("Spider closed (finished)", log) + self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log) + + # https://twistedmatrix.com/trac/ticket/9766 + @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), + "the asyncio reactor is broken on Windows when running Python ≥ 3.8") + def test_reactor_asyncio_custom_settings_conflict(self): + log = self.run_script("twisted_reactor_custom_settings_conflict.py") + self.assertIn("Using reactor: twisted.internet.pollreactor.PollReactor", log) + self.assertIn("(twisted.internet.pollreactor.PollReactor) does not match the requested one", log) + @mark.skipif(sys.implementation.name == 'pypy', reason='uvloop does not support pypy properly') @mark.skipif(platform.system() == 'Windows', reason='uvloop does not support Windows') @mark.skipif(twisted_version == Version('twisted', 21, 2, 0), reason='https://twistedmatrix.com/trac/ticket/10106') From 002513438204eea5062b5a1d75fb4f261880da4f Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 23 Dec 2021 16:45:17 +0500 Subject: [PATCH 057/126] Completely skip WindowsRunSpiderCommandTest outside Windows. --- tests/test_commands.py | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/tests/test_commands.py b/tests/test_commands.py index 75098a77a..efe9b0531 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -765,6 +765,7 @@ class MySpider(scrapy.Spider): self.assertIn("error: Please use only one of -o/--output and -O/--overwrite-output", log) +@skipIf(platform.system() != 'Windows', "Windows required for .pyw files") class WindowsRunSpiderCommandTest(RunSpiderCommandTest): spider_filename = 'myspider.pyw' @@ -777,35 +778,27 @@ class WindowsRunSpiderCommandTest(RunSpiderCommandTest): self.assertIn("start_requests", log) self.assertIn("badspider.pyw", log) - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_run_good_spider(self): super().test_run_good_spider() - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_runspider(self): super().test_runspider() - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_runspider_dnscache_disabled(self): super().test_runspider_dnscache_disabled() - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_runspider_log_level(self): super().test_runspider_log_level() - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_runspider_log_short_names(self): super().test_runspider_log_short_names() - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_runspider_no_spider_found(self): super().test_runspider_no_spider_found() - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_output(self): super().test_output() - @skipIf(platform.system() != 'Windows', "Windows required for .pyw files") def test_overwrite_output(self): super().test_overwrite_output() From 9c4bfb48362f736fce81b71a6ca1fa0b3600231d Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 23 Dec 2021 17:17:36 +0500 Subject: [PATCH 058/126] Remove an unused import. --- tests/test_crawler.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/test_crawler.py b/tests/test_crawler.py index 6d6763aec..d68c50026 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -7,7 +7,6 @@ import warnings from unittest import skipIf from pytest import raises, mark -from testfixtures import LogCapture from twisted import version as twisted_version from twisted.internet import defer from twisted.python.versions import Version From d4565318c7061c2ccd17fa5d5eabcacef8c34826 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 23 Dec 2021 17:40:31 +0500 Subject: [PATCH 059/126] Fix a reactor test on Windows. --- .../twisted_reactor_custom_settings_conflict.py | 8 ++++---- tests/test_crawler.py | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py b/tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py index 9a6c01d72..3f219098c 100644 --- a/tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py +++ b/tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py @@ -2,10 +2,10 @@ import scrapy from scrapy.crawler import CrawlerProcess -class PollReactorSpider(scrapy.Spider): - name = 'poll_reactor' +class SelectReactorSpider(scrapy.Spider): + name = 'select_reactor' custom_settings = { - "TWISTED_REACTOR": "twisted.internet.pollreactor.PollReactor", + "TWISTED_REACTOR": "twisted.internet.selectreactor.SelectReactor", } @@ -17,6 +17,6 @@ class AsyncioReactorSpider(scrapy.Spider): process = CrawlerProcess() -process.crawl(PollReactorSpider) +process.crawl(SelectReactorSpider) process.crawl(AsyncioReactorSpider) process.start() diff --git a/tests/test_crawler.py b/tests/test_crawler.py index d68c50026..e7d5c8132 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -381,8 +381,8 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase): "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_reactor_asyncio_custom_settings_conflict(self): log = self.run_script("twisted_reactor_custom_settings_conflict.py") - self.assertIn("Using reactor: twisted.internet.pollreactor.PollReactor", log) - self.assertIn("(twisted.internet.pollreactor.PollReactor) does not match the requested one", log) + self.assertIn("Using reactor: twisted.internet.selectreactor.SelectReactor", log) + self.assertIn("(twisted.internet.selectreactor.SelectReactor) does not match the requested one", log) @mark.skipif(sys.implementation.name == 'pypy', reason='uvloop does not support pypy properly') @mark.skipif(platform.system() == 'Windows', reason='uvloop does not support Windows') From 940cc0776ff86f726e79c2ab2018f4b83a833936 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 24 Dec 2021 17:12:50 +0500 Subject: [PATCH 060/126] Add docs about TWISTED_REACTOR and other per-process settings. --- docs/topics/practices.rst | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 732eba587..bd0dd8ce0 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -102,6 +102,17 @@ reactor after ``MySpider`` has finished running. d.addBoth(lambda _: reactor.stop()) reactor.run() # the script will block here until the crawling is finished +.. note:: + .. versionchanged:: VERSION + + The Twisted reactor is now installed when + :meth:`~scrapy.crawler.CrawlerProcess.crawl` is first called, not when a + :class:`scrapy.crawler.CrawlerProcess` object is created. Because of this, + :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` are now + honored in :attr:`~scrapy.Spider.custom_settings`. In older Scrapy versions + they are silently ignored when set there and you need to set these settings + in some other way. + .. seealso:: :doc:`twisted:core/howto/reactor-basics` .. _run-multiple-spiders: @@ -193,6 +204,25 @@ Same example but running the spiders sequentially by chaining the deferreds: crawl() reactor.run() # the script will block here until the last crawl call is finished +Different spiders can set different values for the same setting, but when they +run in the same process it may be impossible, by design or because of some +limitations, to use these different values. What happens in practice is +different for different settings: + +* :setting:`SPIDER_LOADER_CLASS` and the ones used by its value + (:setting:`SPIDER_MODULES`, :setting:`SPIDER_LOADER_WARN_ONLY` for the + default one) cannot be read from the per-spider settings. These are applied + when the :class:`~scrapy.crawler.CrawlerRunner` or + :class:`~scrapy.crawler.CrawlerProcess` object is created. +* For :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` the first + available value is used, and if a spider requests a different reactor an + exception will be raised. These are applied when the reactor is installed. +* For :setting:`REACTOR_THREADPOOL_MAXSIZE`, :setting:`DNS_RESOLVER` and the + ones used by the resolver (:setting:`DNSCACHE_ENABLED`, + :setting:`DNSCACHE_SIZE`, :setting:`DNS_TIMEOUT` for ones included in Scrapy) + the first available value is used. These are applied when the reactor is + started. + .. seealso:: :ref:`run-from-script`. .. _distributed-crawls: From a986792def6df1b2bbdf1bc996308d3afd8528c4 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 24 Dec 2021 19:43:14 +0500 Subject: [PATCH 061/126] Add more docs for TWISTED_REACTOR. --- docs/topics/settings.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 210c1def7..cff6d80cb 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1638,10 +1638,18 @@ which raises :exc:`Exception`, becomes:: The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, which -means that Scrapy will not attempt to install any specific reactor, and the -default reactor defined by Twisted for the current platform will be used. This -is to maintain backward compatibility and avoid possible problems caused by -using a non-default reactor. +means that Scrapy will install the default reactor defined by Twisted for the +current platform will be used. This is to maintain backward compatibility and +avoid possible problems caused by using a non-default reactor. + +.. note:: + .. versionchanged:: VERSION + + Previously this setting had no effect in a spider + :attr:`~scrapy.Spider.custom_settings` attribute. Now it will be used, but + if you :ref:`run several spiders in one process `, + they must not have different values for this setting, because they will use + a single reactor instance. For additional information, see :doc:`core/howto/choosing-reactor`. From a9dfd85ea6e983f255afc1b5b0f295fccddcbacb Mon Sep 17 00:00:00 2001 From: Burak Can Kahraman Date: Thu, 30 Dec 2021 15:48:53 +0300 Subject: [PATCH 062/126] Document coroutines for signals. --- docs/topics/coroutines.rst | 2 ++ docs/topics/signals.rst | 20 +++++++++----------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 2aef755c7..549552bd1 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -1,3 +1,5 @@ +.. _topics-coroutines: + ========== Coroutines ========== diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 63ad3a9ad..328fb88d2 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -51,12 +51,12 @@ Deferred signal handlers ======================== Some signals support returning :class:`~twisted.internet.defer.Deferred` -objects from their handlers, allowing you to run asynchronous code that -does not block Scrapy. If a signal handler returns a -:class:`~twisted.internet.defer.Deferred`, Scrapy waits for that -:class:`~twisted.internet.defer.Deferred` to fire. +or :term:`awaitable objects ` from their handlers, allowing +you to run asynchronous code that does not block Scrapy. If a signal +handler returns one of these objects, Scrapy waits for that asynchronous +operation to finish. -Let's take an example:: +Let's take an example using :ref:`coroutines `:: class SignalSpider(scrapy.Spider): name = 'signals' @@ -68,17 +68,15 @@ Let's take an example:: crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped) return spider - def item_scraped(self, item): + async def item_scraped(self, item): # Send the scraped item to the server - d = treq.post( + response = await treq.post( 'http://example.com/post', json.dumps(item).encode('ascii'), headers={b'Content-Type': [b'application/json']} ) - # The next item will be scraped only after - # deferred (d) is fired - return d + return response def parse(self, response): for quote in response.css('div.quote'): @@ -89,7 +87,7 @@ Let's take an example:: } See the :ref:`topics-signals-ref` below to know which signals support -:class:`~twisted.internet.defer.Deferred`. +:class:`~twisted.internet.defer.Deferred` and :term:`awaitable objects `. .. _topics-signals-ref: From 7380888cad2c255e6f4a7efb6fe4cfd1240fb42c Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Thu, 30 Dec 2021 18:55:16 +0500 Subject: [PATCH 063/126] Fix a warning message. (#5359) --- scrapy/utils/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scrapy/utils/conf.py b/scrapy/utils/conf.py index 24873f75d..24a6187b9 100644 --- a/scrapy/utils/conf.py +++ b/scrapy/utils/conf.py @@ -164,7 +164,7 @@ def feed_process_params_from_cli(settings, output, output_format=None, message = ( 'The -t command line option is deprecated in favor of ' 'specifying the output format within the output URI. See the ' - 'documentation of the -o and -O options for more information.', + 'documentation of the -o and -O options for more information.' ) warnings.warn(message, ScrapyDeprecationWarning, stacklevel=2) return {output[0]: {'format': output_format}} From 64261d9e389737621caa85f320cf81ef2aef1faa Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 31 Dec 2021 15:45:59 +0500 Subject: [PATCH 064/126] Slight refactoring. --- scrapy/crawler.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scrapy/crawler.py b/scrapy/crawler.py index 95cfb1bd1..a638254f1 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -71,17 +71,18 @@ class Crawler: lf_cls = load_object(self.settings['LOG_FORMATTER']) self.logformatter = lf_cls.from_crawler(self) + reactor_class = self.settings.get("TWISTED_REACTOR") if init_reactor: # this needs to be done after the spider settings are merged, # but before something imports twisted.internet.reactor - if self.settings.get("TWISTED_REACTOR"): - install_reactor(self.settings["TWISTED_REACTOR"], self.settings["ASYNCIO_EVENT_LOOP"]) + if reactor_class: + install_reactor(reactor_class, self.settings["ASYNCIO_EVENT_LOOP"]) else: from twisted.internet import default default.install() log_reactor_info() - if self.settings.get("TWISTED_REACTOR"): - verify_installed_reactor(self.settings["TWISTED_REACTOR"]) + if reactor_class: + verify_installed_reactor(reactor_class) self.extensions = ExtensionManager.from_crawler(self) From b81938684b55fca29e48faa766f2b6f6e3ab5d6a Mon Sep 17 00:00:00 2001 From: Andrey Oskin Date: Fri, 31 Dec 2021 21:49:18 +1100 Subject: [PATCH 065/126] Docs: correct process repetition start step (#5356) The process repeats from step 3, the scheduler feeds request to the engine. Steps 1 and 2 are not parts of the loop as their incarnations steps 7 and 8 are parts of the loop. --- docs/topics/architecture.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rst index 71d027c86..0c3a7ed88 100644 --- a/docs/topics/architecture.rst +++ b/docs/topics/architecture.rst @@ -67,7 +67,7 @@ this: the :ref:`Scheduler ` and asks for possible next Requests to crawl. -9. The process repeats (from step 1) until there are no more requests from the +9. The process repeats (from step 3) until there are no more requests from the :ref:`Scheduler `. Components From e4bdd1cb958b7d89b86ea66f0af1cec2d91a6d44 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Miech?= Date: Fri, 31 Dec 2021 11:57:12 +0100 Subject: [PATCH 066/126] downloader.webclient: make reactor import local (#5357) --- scrapy/core/downloader/webclient.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scrapy/core/downloader/webclient.py b/scrapy/core/downloader/webclient.py index 915cb5fe3..06cb96489 100644 --- a/scrapy/core/downloader/webclient.py +++ b/scrapy/core/downloader/webclient.py @@ -3,7 +3,7 @@ from time import time from urllib.parse import urlparse, urlunparse, urldefrag from twisted.web.http import HTTPClient -from twisted.internet import defer, reactor +from twisted.internet import defer from twisted.internet.protocol import ClientFactory from scrapy.http import Headers @@ -170,6 +170,7 @@ class ScrapyHTTPClientFactory(ClientFactory): p.followRedirect = self.followRedirect p.afterFoundGet = self.afterFoundGet if self.timeout: + from twisted.internet import reactor timeoutCall = reactor.callLater(self.timeout, p.timeout) self.deferred.addBoth(self._cancelTimeout, timeoutCall) return p From 57dc58123b98e2026025cc87bdee474bf0656dcb Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Fri, 31 Dec 2021 17:15:08 +0500 Subject: [PATCH 067/126] Remove the experimental note about asyncio (#5332) --- docs/topics/asyncio.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 28241ae24..402352721 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -10,11 +10,6 @@ Scrapy has partial support for :mod:`asyncio`. After you :ref:`install the asyncio reactor `, you may use :mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine `. -.. warning:: :mod:`asyncio` support in Scrapy is experimental, and not yet - recommended for production environments. Future Scrapy versions - may introduce related changes without a deprecation period or - warning. - .. _install-asyncio: Installing the asyncio reactor From a2763c608d33a9254034d650457b7efa7b434ec0 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 31 Dec 2021 18:35:00 +0500 Subject: [PATCH 068/126] Remove unused MiddlewareManager._process_chain_both(). --- scrapy/middleware.py | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/scrapy/middleware.py b/scrapy/middleware.py index e8f60287a..2eb1d8609 100644 --- a/scrapy/middleware.py +++ b/scrapy/middleware.py @@ -9,7 +9,7 @@ from scrapy import Spider from scrapy.exceptions import NotConfigured from scrapy.settings import Settings from scrapy.utils.misc import create_instance, load_object -from scrapy.utils.defer import process_parallel, process_chain, process_chain_both +from scrapy.utils.defer import process_parallel, process_chain logger = logging.getLogger(__name__) @@ -72,11 +72,6 @@ class MiddlewareManager: methods = cast(Iterable[Callable], self.methods[methodname]) return process_chain(methods, obj, *args) - def _process_chain_both(self, cb_methodname: str, eb_methodname: str, obj, *args) -> Deferred: - cb_methods = cast(Iterable[Callable], self.methods[cb_methodname]) - eb_methods = cast(Iterable[Callable], self.methods[eb_methodname]) - return process_chain_both(cb_methods, eb_methods, obj, *args) - def open_spider(self, spider: Spider) -> Deferred: return self._process_parallel('open_spider', spider) From 6eaceec735d551f5b777bc641ff8d85dbb3ba98c Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 31 Dec 2021 20:14:24 +0500 Subject: [PATCH 069/126] Implement docs suggestions. --- docs/news.rst | 22 ++++++++++++++++++++++ docs/topics/practices.rst | 11 ----------- docs/topics/settings.rst | 13 ++----------- 3 files changed, 24 insertions(+), 22 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 509366c17..2afe318f6 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1,3 +1,25 @@ +.. note:: + .. versionchanged:: VERSION + + The Twisted reactor is now installed when + :meth:`~scrapy.crawler.CrawlerProcess.crawl` is first called, not when a + :class:`scrapy.crawler.CrawlerProcess` object is created. Because of this, + :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` are now + honored in :attr:`~scrapy.Spider.custom_settings`. In older Scrapy versions + they are silently ignored when set there and you need to set these settings + in some other way. + + +.. note:: + .. versionchanged:: VERSION + + Previously this setting had no effect in a spider + :attr:`~scrapy.Spider.custom_settings` attribute. Now it will be used, but + if you :ref:`run several spiders in one process `, + they must not have different values for this setting, because they will use + a single reactor instance. + + .. _news: Release notes diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index bd0dd8ce0..1a9d56143 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -102,17 +102,6 @@ reactor after ``MySpider`` has finished running. d.addBoth(lambda _: reactor.stop()) reactor.run() # the script will block here until the crawling is finished -.. note:: - .. versionchanged:: VERSION - - The Twisted reactor is now installed when - :meth:`~scrapy.crawler.CrawlerProcess.crawl` is first called, not when a - :class:`scrapy.crawler.CrawlerProcess` object is created. Because of this, - :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` are now - honored in :attr:`~scrapy.Spider.custom_settings`. In older Scrapy versions - they are silently ignored when set there and you need to set these settings - in some other way. - .. seealso:: :doc:`twisted:core/howto/reactor-basics` .. _run-multiple-spiders: diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index cff6d80cb..f6c95c502 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1639,17 +1639,8 @@ which raises :exc:`Exception`, becomes:: The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, which means that Scrapy will install the default reactor defined by Twisted for the -current platform will be used. This is to maintain backward compatibility and -avoid possible problems caused by using a non-default reactor. - -.. note:: - .. versionchanged:: VERSION - - Previously this setting had no effect in a spider - :attr:`~scrapy.Spider.custom_settings` attribute. Now it will be used, but - if you :ref:`run several spiders in one process `, - they must not have different values for this setting, because they will use - a single reactor instance. +current platform. This is to maintain backward compatibility and avoid possible +problems caused by using a non-default reactor. For additional information, see :doc:`core/howto/choosing-reactor`. From c5ab58056c29c2c35b183572b0780acfbb15dfe8 Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Sat, 1 Jan 2022 00:38:10 +0500 Subject: [PATCH 070/126] Set WindowsSelectorEventLoopPolicy on Windows (#5315) --- .github/workflows/tests-windows.yml | 3 ++ docs/topics/asyncio.rst | 28 +++++++++++++++++++ scrapy/utils/reactor.py | 5 ++++ .../CrawlerProcess/asyncio_enabled_reactor.py | 3 ++ tests/test_commands.py | 11 +++----- tests/test_crawler.py | 16 ----------- tests/test_downloader_handlers.py | 16 +++++++++++ tests/test_utils_asyncio.py | 7 +---- 8 files changed, 60 insertions(+), 29 deletions(-) diff --git a/.github/workflows/tests-windows.yml b/.github/workflows/tests-windows.yml index 6fabf5cde..ab7385118 100644 --- a/.github/workflows/tests-windows.yml +++ b/.github/workflows/tests-windows.yml @@ -23,6 +23,9 @@ jobs: - python-version: "3.10" env: TOXENV: py + - python-version: "3.10" + env: + TOXENV: asyncio steps: - uses: actions/checkout@v2 diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 402352721..8712d4268 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -36,6 +36,34 @@ use it instead of the default asyncio event loop. .. _asyncio-await-dfd: +Windows-specific notes +====================== + +The Windows implementation of :mod:`asyncio` can use two event loop +implementations: :class:`~asyncio.SelectorEventLoop` (default before Python +3.8, required when using Twisted) and :class:`~asyncio.ProactorEventLoop` +(default since Python 3.8, cannot work with Twisted). So on Python 3.8+ the +event loop class needs to be changed. Scrapy since VERSION does this +automatically when you change the :setting:`TWISTED_REACTOR` setting or call +:func:`~scrapy.utils.reactor.install_reactor`, but if you install the reactor +by other means or use an older Scrapy version you need to call the following +code before installing the reactor:: + + import asyncio + asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) + +You can put this in the same function that installs the reactor, if you do that +yourself, or in some code that runs before the reactor is installed, e.g. +``settings.py``. + +.. note:: Other libraries you use may require + :class:`~asyncio.ProactorEventLoop`, e.g. because it supports + subprocesses (this is the case with `playwright`_), so you cannot use + them together with Scrapy on Windows (but you should be able to use + them on WSL or native Linux). + +.. _playwright: https://github.com/microsoft/playwright-python + Awaiting on Deferreds ===================== diff --git a/scrapy/utils/reactor.py b/scrapy/utils/reactor.py index 6723d9b37..96395543c 100644 --- a/scrapy/utils/reactor.py +++ b/scrapy/utils/reactor.py @@ -1,4 +1,5 @@ import asyncio +import sys from contextlib import suppress from twisted.internet import asyncioreactor, error @@ -57,6 +58,10 @@ def install_reactor(reactor_path, event_loop_path=None): reactor_class = load_object(reactor_path) if reactor_class is asyncioreactor.AsyncioSelectorReactor: with suppress(error.ReactorAlreadyInstalledError): + if sys.version_info >= (3, 8) and sys.platform == "win32": + policy = asyncio.get_event_loop_policy() + if not isinstance(policy, asyncio.WindowsSelectorEventLoopPolicy): + asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) if event_loop_path is not None: event_loop_class = load_object(event_loop_path) event_loop = event_loop_class() diff --git a/tests/CrawlerProcess/asyncio_enabled_reactor.py b/tests/CrawlerProcess/asyncio_enabled_reactor.py index 8568bd8b8..f2a93074b 100644 --- a/tests/CrawlerProcess/asyncio_enabled_reactor.py +++ b/tests/CrawlerProcess/asyncio_enabled_reactor.py @@ -1,6 +1,9 @@ import asyncio +import sys from twisted.internet import asyncioreactor +if sys.version_info >= (3, 8) and sys.platform == "win32": + asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) asyncioreactor.install(asyncio.get_event_loop()) import scrapy diff --git a/tests/test_commands.py b/tests/test_commands.py index 75098a77a..81d1a1cab 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -674,9 +674,6 @@ class MySpider(scrapy.Spider): self.assertIn("start_requests", log) self.assertIn("badspider.py", log) - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_asyncio_enabled_true(self): log = self.get_log(self.debug_log_spider, args=[ '-s', 'TWISTED_REACTOR=twisted.internet.asyncioreactor.AsyncioSelectorReactor' @@ -699,15 +696,15 @@ class MySpider(scrapy.Spider): ]) self.assertIn("Using asyncio event loop: uvloop.Loop", log) - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_custom_asyncio_loop_enabled_false(self): log = self.get_log(self.debug_log_spider, args=[ '-s', 'TWISTED_REACTOR=twisted.internet.asyncioreactor.AsyncioSelectorReactor' ]) import asyncio - loop = asyncio.new_event_loop() + if sys.platform != 'win32': + loop = asyncio.new_event_loop() + else: + loop = asyncio.SelectorEventLoop() self.assertIn(f"Using asyncio event loop: {loop.__module__}.{loop.__class__.__name__}", log) def test_output(self): diff --git a/tests/test_crawler.py b/tests/test_crawler.py index be067155e..7bc4fba40 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -4,7 +4,6 @@ import platform import subprocess import sys import warnings -from unittest import skipIf from pytest import raises, mark from testfixtures import LogCapture @@ -284,9 +283,6 @@ class CrawlerRunnerHasSpider(unittest.TestCase): }) @defer.inlineCallbacks - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_crawler_process_asyncio_enabled_true(self): with LogCapture(level=logging.DEBUG) as log: if self.reactor_pytest == 'asyncio': @@ -328,17 +324,11 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase): self.assertIn('Spider closed (finished)', log) self.assertNotIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log) - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_asyncio_enabled_no_reactor(self): log = self.run_script('asyncio_enabled_no_reactor.py') self.assertIn('Spider closed (finished)', log) self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log) - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_asyncio_enabled_reactor(self): log = self.run_script('asyncio_enabled_reactor.py') self.assertIn('Spider closed (finished)', log) @@ -377,9 +367,6 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase): self.assertIn("Spider closed (finished)", log) self.assertIn("Using reactor: twisted.internet.pollreactor.PollReactor", log) - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_reactor_asyncio(self): log = self.run_script("twisted_reactor_asyncio.py") self.assertIn("Spider closed (finished)", log) @@ -404,9 +391,6 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase): self.assertIn("Using asyncio event loop: uvloop.Loop", log) self.assertIn("async pipeline opened!", log) - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_default_loop_asyncio_deferred_signal(self): log = self.run_script("asyncio_deferred_signal.py") self.assertIn("Spider closed (finished)", log) diff --git a/tests/test_downloader_handlers.py b/tests/test_downloader_handlers.py index 9c11820e5..a1ea4c679 100644 --- a/tests/test_downloader_handlers.py +++ b/tests/test_downloader_handlers.py @@ -1,6 +1,7 @@ import contextlib import os import shutil +import sys import tempfile from typing import Optional, Type from unittest import mock @@ -287,6 +288,12 @@ class HttpTestCase(unittest.TestCase): @defer.inlineCallbacks def test_timeout_download_from_spider_nodata_rcvd(self): + if self.reactor_pytest == "asyncio" and sys.platform == "win32": + # https://twistedmatrix.com/trac/ticket/10279 + raise unittest.SkipTest( + "This test produces DirtyReactorAggregateError on Windows with asyncio" + ) + # client connects but no data is received spider = Spider('foo') meta = {'download_timeout': 0.5} @@ -296,6 +303,11 @@ class HttpTestCase(unittest.TestCase): @defer.inlineCallbacks def test_timeout_download_from_spider_server_hangs(self): + if self.reactor_pytest == "asyncio" and sys.platform == "win32": + # https://twistedmatrix.com/trac/ticket/10279 + raise unittest.SkipTest( + "This test produces DirtyReactorAggregateError on Windows with asyncio" + ) # client connects, server send headers and some body bytes but hangs spider = Spider('foo') meta = {'download_timeout': 0.5} @@ -1055,6 +1067,10 @@ class BaseFTPTestCase(unittest.TestCase): class FTPTestCase(BaseFTPTestCase): def test_invalid_credentials(self): + if self.reactor_pytest == "asyncio" and sys.platform == "win32": + raise unittest.SkipTest( + "This test produces DirtyReactorAggregateError on Windows with asyncio" + ) from twisted.protocols.ftp import ConnectionLost meta = dict(self.req_meta) diff --git a/tests/test_utils_asyncio.py b/tests/test_utils_asyncio.py index a2114bd18..295323e4d 100644 --- a/tests/test_utils_asyncio.py +++ b/tests/test_utils_asyncio.py @@ -1,6 +1,4 @@ -import platform -import sys -from unittest import skipIf, TestCase +from unittest import TestCase from pytest import mark @@ -14,9 +12,6 @@ class AsyncioTest(TestCase): # the result should depend only on the pytest --reactor argument self.assertEqual(is_asyncio_reactor_installed(), self.reactor_pytest == 'asyncio') - # https://twistedmatrix.com/trac/ticket/9766 - @skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8), - "the asyncio reactor is broken on Windows when running Python ≥ 3.8") def test_install_asyncio_reactor(self): # this should do nothing install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor") From 4bdaa54af4470582845206a9fa21779f80a4b123 Mon Sep 17 00:00:00 2001 From: Georgiy Zatserklianyi Date: Fri, 28 Jan 2022 15:39:32 +0200 Subject: [PATCH 071/126] response_httprepr memory issue fixed (#4972) * response_httprepr replaced by response.body * unused import deleted * get_header_size function added * response size calculation updated * flake8 codestyle fix * added counting status code, line breaks to response size * get_status size: list to tuple, comments added * test added: comparing new response size counting method with old `len(response_httprepr)` * downloader stats : unreachable code deleted * `get_status_size` optimized * comment added * tests.test_downloadermiddleware_stats: statement formatting updated * scrapy.utils.response: `response_httprepr` -> deprecated * tests.test_downloadermiddleware_stats: flake8 fix --- scrapy/downloadermiddlewares/stats.py | 22 +++++++++++++++++++--- scrapy/utils/response.py | 2 ++ tests/test_downloadermiddleware_stats.py | 19 +++++++++++++++++++ 3 files changed, 40 insertions(+), 3 deletions(-) diff --git a/scrapy/downloadermiddlewares/stats.py b/scrapy/downloadermiddlewares/stats.py index 5479cd0e2..25fb1ed9d 100644 --- a/scrapy/downloadermiddlewares/stats.py +++ b/scrapy/downloadermiddlewares/stats.py @@ -1,7 +1,22 @@ from scrapy.exceptions import NotConfigured +from scrapy.utils.python import global_object_name, to_bytes from scrapy.utils.request import request_httprepr -from scrapy.utils.response import response_httprepr -from scrapy.utils.python import global_object_name + +from twisted.web import http + + +def get_header_size(headers): + size = 0 + for key, value in headers.items(): + if isinstance(value, (list, tuple)): + for v in value: + size += len(b": ") + len(key) + len(v) + return size + len(b'\r\n') * (len(headers.keys()) - 1) + + +def get_status_size(response_status): + return len(to_bytes(http.RESPONSES.get(response_status, b''))) + 15 + # resp.status + b"\r\n" + b"HTTP/1.1 <100-599> " class DownloaderStats: @@ -24,7 +39,8 @@ class DownloaderStats: def process_response(self, request, response, spider): self.stats.inc_value('downloader/response_count', spider=spider) self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider) - reslen = len(response_httprepr(response)) + reslen = len(response.body) + get_header_size(response.headers) + get_status_size(response.status) + 4 + # response.body + b"\r\n"+ response.header + b"\r\n" + response.status self.stats.inc_value('downloader/response_bytes', reslen, spider=spider) return response diff --git a/scrapy/utils/response.py b/scrapy/utils/response.py index 8b109dced..741dce350 100644 --- a/scrapy/utils/response.py +++ b/scrapy/utils/response.py @@ -14,6 +14,7 @@ from scrapy.http.response import Response from twisted.web import http from scrapy.utils.python import to_bytes, to_unicode +from scrapy.utils.decorators import deprecated from w3lib import html @@ -51,6 +52,7 @@ def response_status_message(status: Union[bytes, float, int, str]) -> str: return f'{status_int} {to_unicode(message)}' +@deprecated def response_httprepr(response: Response) -> bytes: """Return raw HTTP representation (as bytes) of the given response. This is provided only for reference, since it's not the exact stream of bytes diff --git a/tests/test_downloadermiddleware_stats.py b/tests/test_downloadermiddleware_stats.py index 1f2616e35..9e75f0a50 100644 --- a/tests/test_downloadermiddleware_stats.py +++ b/tests/test_downloadermiddleware_stats.py @@ -1,8 +1,10 @@ +from itertools import product from unittest import TestCase from scrapy.downloadermiddlewares.stats import DownloaderStats from scrapy.http import Request, Response from scrapy.spiders import Spider +from scrapy.utils.response import response_httprepr from scrapy.utils.test import get_crawler @@ -37,6 +39,23 @@ class TestDownloaderStats(TestCase): self.mw.process_response(self.req, self.res, self.spider) self.assertStatsEqual('downloader/response_count', 1) + def test_response_len(self): + body = (b'', b'not_empty') # empty/notempty body + headers = ({}, {'lang': 'en'}, {'lang': 'en', 'User-Agent': 'scrapy'}) # 0 headers, 1h and 2h + test_responses = [ # form test responses with all combinations of body/headers + Response( + url='scrapytest.org', + status=200, + body=r[0], + headers=r[1] + ) + for r in product(body, headers) + ] + for test_response in test_responses: + self.crawler.stats.set_value('downloader/response_bytes', 0) + self.mw.process_response(self.req, test_response, self.spider) + self.assertStatsEqual('downloader/response_bytes', len(response_httprepr(test_response))) + def test_process_exception(self): self.mw.process_exception(self.req, MyException(), self.spider) self.assertStatsEqual('downloader/exception_count', 1) From 30d5779ea94ed1e9343a4590895a3f5e65e444b9 Mon Sep 17 00:00:00 2001 From: "Sixuan (Cherie) Wu" <73203695+inspurwusixuan@users.noreply.github.com> Date: Fri, 28 Jan 2022 09:30:30 -0800 Subject: [PATCH 072/126] Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix FEED_URI_PARAMS: custom params throws KeyError closes #4962 * another try FEED_URI_PARAMS * add warning message and change default function * Add tests for FEED_URI_PARAMS * FEED_URI_PARAMS: warn if the params dict has been modified in-place * [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated * Remove whileline * Rename parameters for lambda function * Type hints for FeedExporter._get_uri_params Co-authored-by: Adrián Chaves Co-authored-by: Eugenio Lacuesta --- docs/topics/feed-exports.rst | 7 +- scrapy/extensions/feedexport.py | 26 +++-- tests/test_feedexport.py | 163 ++++++++++++++++++++++++++++++++ 3 files changed, 188 insertions(+), 8 deletions(-) diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index 116967280..7994027d2 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -322,7 +322,7 @@ Post-Processing Scrapy provides an option to activate plugins to post-process feeds before they are exported to feed storages. In addition to using :ref:`builtin plugins `, you -can create your own :ref:`plugins `. +can create your own :ref:`plugins `. These plugins can be activated through the ``postprocessing`` option of a feed. The option must be passed a list of post-processing plugins in the order you want @@ -366,7 +366,7 @@ Each plugin is a class that must implement the following methods: Close the target file object. -To pass a parameter to your plugin, use :ref:`feed options `. You +To pass a parameter to your plugin, use :ref:`feed options `. You can then access those parameters from the ``__init__`` method of your plugin. @@ -744,6 +744,9 @@ The function signature should be as follows: :param spider: source spider of the feed items :type spider: scrapy.Spider + .. caution:: The function should return a new dictionary, modifying + the received ``params`` in-place is deprecated. + For example, to include the :attr:`name ` of the source spider in the feed URI: diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py index 370723368..e7097b7a1 100644 --- a/scrapy/extensions/feedexport.py +++ b/scrapy/extensions/feedexport.py @@ -11,14 +11,14 @@ import sys import warnings from datetime import datetime from tempfile import NamedTemporaryFile -from typing import Any, Optional, Tuple +from typing import Any, Callable, Optional, Tuple, Union from urllib.parse import unquote, urlparse from twisted.internet import defer, threads from w3lib.url import file_uri_to_path from zope.interface import implementer, Interface -from scrapy import signals +from scrapy import signals, Spider from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning from scrapy.extensions.postprocessing import PostProcessingManager from scrapy.utils.boto import is_botocore_available @@ -524,7 +524,12 @@ class FeedExporter: raise TypeError(f"{feedcls.__qualname__}.{method_name} returned None") return instance - def _get_uri_params(self, spider, uri_params, slot=None): + def _get_uri_params( + self, + spider: Spider, + uri_params_function: Optional[Union[str, Callable[[dict, Spider], dict]]], + slot: Optional[_FeedSlot] = None, + ) -> dict: params = {} for k in dir(spider): params[k] = getattr(spider, k) @@ -532,9 +537,18 @@ class FeedExporter: params['time'] = utc_now.replace(microsecond=0).isoformat().replace(':', '-') params['batch_time'] = utc_now.isoformat().replace(':', '-') params['batch_id'] = slot.batch_id + 1 if slot is not None else 1 - uripar_function = load_object(uri_params) if uri_params else lambda x, y: None - uripar_function(params, spider) - return params + original_params = params.copy() + uripar_function = load_object(uri_params_function) if uri_params_function else lambda params, _: params + new_params = uripar_function(params, spider) + if new_params is None or original_params != params: + warnings.warn( + 'Modifying the params dictionary in-place in the function defined in ' + 'the FEED_URI_PARAMS setting or in the uri_params key of the FEEDS ' + 'setting is deprecated. The function must return a new dictionary ' + 'instead.', + category=ScrapyDeprecationWarning + ) + return new_params if new_params is not None else params def _load_filter(self, feed_options): # load the item filter if declared else load the default filter class diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py index 253f3119c..f0acf1941 100644 --- a/tests/test_feedexport.py +++ b/tests/test_feedexport.py @@ -2608,3 +2608,166 @@ class FTPFeedStoragePreFeedOptionsTest(unittest.TestCase): ), ) ) + + +class URIParamsTest: + + spider_name = "uri_params_spider" + + def build_settings(self, uri='file:///tmp/foobar', uri_params=None): + raise NotImplementedError + + def test_default(self): + settings = self.build_settings( + uri='file:///tmp/%(name)s', + ) + crawler = get_crawler(settings_dict=settings) + feed_exporter = FeedExporter.from_crawler(crawler) + spider = scrapy.Spider(self.spider_name) + spider.crawler = crawler + with warnings.catch_warnings(record=True) as w: + feed_exporter.open_spider(spider) + messages = tuple( + str(item.message) for item in w + if item.category is ScrapyDeprecationWarning + ) + self.assertEqual(messages, tuple()) + + self.assertEqual( + feed_exporter.slots[0].uri, + f'file:///tmp/{self.spider_name}' + ) + + def test_none(self): + def uri_params(params, spider): + pass + + settings = self.build_settings( + uri='file:///tmp/%(name)s', + uri_params=uri_params, + ) + crawler = get_crawler(settings_dict=settings) + feed_exporter = FeedExporter.from_crawler(crawler) + spider = scrapy.Spider(self.spider_name) + spider.crawler = crawler + with warnings.catch_warnings(record=True) as w: + feed_exporter.open_spider(spider) + messages = tuple( + str(item.message) for item in w + if item.category is ScrapyDeprecationWarning + ) + self.assertEqual( + messages, + ( + ( + 'Modifying the params dictionary in-place in the ' + 'function defined in the FEED_URI_PARAMS setting or ' + 'in the uri_params key of the FEEDS setting is ' + 'deprecated. The function must return a new ' + 'dictionary instead.' + ), + ) + ) + + self.assertEqual( + feed_exporter.slots[0].uri, + f'file:///tmp/{self.spider_name}' + ) + + def test_empty_dict(self): + def uri_params(params, spider): + return {} + + settings = self.build_settings( + uri='file:///tmp/%(name)s', + uri_params=uri_params, + ) + crawler = get_crawler(settings_dict=settings) + feed_exporter = FeedExporter.from_crawler(crawler) + spider = scrapy.Spider(self.spider_name) + spider.crawler = crawler + with warnings.catch_warnings(record=True) as w: + with self.assertRaises(KeyError): + feed_exporter.open_spider(spider) + messages = tuple( + str(item.message) for item in w + if item.category is ScrapyDeprecationWarning + ) + self.assertEqual(messages, tuple()) + + def test_params_as_is(self): + def uri_params(params, spider): + return params + + settings = self.build_settings( + uri='file:///tmp/%(name)s', + uri_params=uri_params, + ) + crawler = get_crawler(settings_dict=settings) + feed_exporter = FeedExporter.from_crawler(crawler) + spider = scrapy.Spider(self.spider_name) + spider.crawler = crawler + with warnings.catch_warnings(record=True) as w: + feed_exporter.open_spider(spider) + messages = tuple( + str(item.message) for item in w + if item.category is ScrapyDeprecationWarning + ) + self.assertEqual(messages, tuple()) + + self.assertEqual( + feed_exporter.slots[0].uri, + f'file:///tmp/{self.spider_name}' + ) + + def test_custom_param(self): + def uri_params(params, spider): + return {**params, 'foo': self.spider_name} + + settings = self.build_settings( + uri='file:///tmp/%(foo)s', + uri_params=uri_params, + ) + crawler = get_crawler(settings_dict=settings) + feed_exporter = FeedExporter.from_crawler(crawler) + spider = scrapy.Spider(self.spider_name) + spider.crawler = crawler + with warnings.catch_warnings(record=True) as w: + feed_exporter.open_spider(spider) + messages = tuple( + str(item.message) for item in w + if item.category is ScrapyDeprecationWarning + ) + self.assertEqual(messages, tuple()) + + self.assertEqual( + feed_exporter.slots[0].uri, + f'file:///tmp/{self.spider_name}' + ) + + +class URIParamsSettingTest(URIParamsTest, unittest.TestCase): + + def build_settings(self, uri='file:///tmp/foobar', uri_params=None): + extra_settings = {} + if uri_params: + extra_settings['FEED_URI_PARAMS'] = uri_params + return { + 'FEED_URI': uri, + **extra_settings, + } + + +class URIParamsFeedOptionTest(URIParamsTest, unittest.TestCase): + + def build_settings(self, uri='file:///tmp/foobar', uri_params=None): + options = { + 'format': 'jl', + } + if uri_params: + options['uri_params'] = uri_params + return { + 'FEEDS': { + uri: options, + }, + } From fe43411bc4d0164a0f0ecc596c23b59c99d31f17 Mon Sep 17 00:00:00 2001 From: Laerte <5853172+Laerte@users.noreply.github.com> Date: Fri, 4 Feb 2022 05:57:57 -0300 Subject: [PATCH 073/126] Fix TypeError on using pathlib.Path as key on FEEDS settings (#5384) --- scrapy/settings/__init__.py | 6 +++++- tests/test_cmdline/__init__.py | 4 ++++ tests/test_cmdline/settings.py | 9 +++++++++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/scrapy/settings/__init__.py b/scrapy/settings/__init__.py index 1fe1e6fd1..6b1ad0828 100644 --- a/scrapy/settings/__init__.py +++ b/scrapy/settings/__init__.py @@ -375,9 +375,13 @@ class BaseSettings(MutableMapping): return len(self.attributes) def _to_dict(self): - return {k: (v._to_dict() if isinstance(v, BaseSettings) else v) + return {self._get_key(k): (v._to_dict() if isinstance(v, BaseSettings) else v) for k, v in self.items()} + def _get_key(self, key_value): + return (key_value if isinstance(key_value, (bool, float, int, str, type(None))) + else str(key_value)) + def copy_to_dict(self): """ Make a copy of current settings and convert to a dict. diff --git a/tests/test_cmdline/__init__.py b/tests/test_cmdline/__init__.py index 591075a98..8233e0101 100644 --- a/tests/test_cmdline/__init__.py +++ b/tests/test_cmdline/__init__.py @@ -64,3 +64,7 @@ class CmdlineTest(unittest.TestCase): settingsdict = json.loads(settingsstr) self.assertCountEqual(settingsdict.keys(), EXTENSIONS.keys()) self.assertEqual(200, settingsdict[EXT_PATH]) + + def test_pathlib_path_as_feeds_key(self): + self.assertEqual(self._execute('settings', '--get', 'FEEDS'), + json.dumps({"items.csv": {"format": "csv", "fields": ["price", "name"]}})) diff --git a/tests/test_cmdline/settings.py b/tests/test_cmdline/settings.py index 8a719ddf2..b0ac6e98b 100644 --- a/tests/test_cmdline/settings.py +++ b/tests/test_cmdline/settings.py @@ -1,5 +1,14 @@ +from pathlib import Path + EXTENSIONS = { 'tests.test_cmdline.extensions.TestExtension': 0, } TEST1 = 'default' + +FEEDS = { + Path('items.csv'): { + 'format': 'csv', + 'fields': ['price', 'name'], + }, +} From 9be878fc09cf71bb2cb98695f5042cb344bd2e25 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 4 Feb 2022 12:27:39 +0100 Subject: [PATCH 074/126] CI: stop using tox-pip-version (#5389) --- .github/workflows/checks.yml | 4 ---- .github/workflows/tests-ubuntu.yml | 4 ---- 2 files changed, 8 deletions(-) diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 80df9469d..98fa44c7f 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -19,7 +19,6 @@ jobs: - python-version: 3.8 env: TOXENV: pylint - TOX_PIP_VERSION: 20.3.3 - python-version: 3.6 env: TOXENV: typing @@ -38,8 +37,5 @@ jobs: - name: Run check env: ${{ matrix.env }} run: | - if [[ ! -z "$TOX_PIP_VERSION" ]]; then - pip install tox-pip-version - fi pip install -U tox tox diff --git a/.github/workflows/tests-ubuntu.yml b/.github/workflows/tests-ubuntu.yml index 5ea50e644..1fc8d914b 100644 --- a/.github/workflows/tests-ubuntu.yml +++ b/.github/workflows/tests-ubuntu.yml @@ -46,7 +46,6 @@ jobs: - python-version: 3.8 env: TOXENV: extra-deps - TOX_PIP_VERSION: 20.3.3 steps: - uses: actions/checkout@v2 @@ -73,9 +72,6 @@ jobs: $PYPY_VERSION/bin/pypy3 -m venv "$HOME/virtualenvs/$PYPY_VERSION" source "$HOME/virtualenvs/$PYPY_VERSION/bin/activate" fi - if [[ ! -z "$TOX_PIP_VERSION" ]]; then - pip install tox-pip-version - fi pip install -U tox tox From 55ae2109c95e497d4a730afeb9caf71aa78a7723 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Sat, 5 Feb 2022 13:02:02 -0300 Subject: [PATCH 075/126] Remove deprecated TextResponse.body_as_unicode --- scrapy/http/response/text.py | 9 --------- tests/test_http_response.py | 18 ------------------ 2 files changed, 27 deletions(-) diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py index 27bd55c07..89516b9b6 100644 --- a/scrapy/http/response/text.py +++ b/scrapy/http/response/text.py @@ -6,7 +6,6 @@ See documentation in docs/topics/request-response.rst """ import json -import warnings from contextlib import suppress from typing import Generator, Tuple from urllib.parse import urljoin @@ -16,7 +15,6 @@ from w3lib.encoding import (html_body_declared_encoding, html_to_unicode, http_content_type_encoding, resolve_encoding) from w3lib.html import strip_html5_whitespace -from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.http import Request from scrapy.http.response import Response from scrapy.utils.python import memoizemethod_noargs, to_unicode @@ -66,13 +64,6 @@ class TextResponse(Response): or self._body_declared_encoding() ) - def body_as_unicode(self): - """Return body as unicode""" - warnings.warn('Response.body_as_unicode() is deprecated, ' - 'please use Response.text instead.', - ScrapyDeprecationWarning, stacklevel=2) - return self.text - def json(self): """ .. versionadded:: 2.2 diff --git a/tests/test_http_response.py b/tests/test_http_response.py index 0ec5257e1..2986f884f 100644 --- a/tests/test_http_response.py +++ b/tests/test_http_response.py @@ -1,10 +1,8 @@ import unittest from unittest import mock -from warnings import catch_warnings, filterwarnings from w3lib.encoding import resolve_encoding -from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.http import (Request, Response, TextResponse, HtmlResponse, XmlResponse, Headers) from scrapy.selector import Selector @@ -134,9 +132,6 @@ class BaseResponseTest(unittest.TestCase): assert isinstance(response.text, str) self._assert_response_encoding(response, encoding) self.assertEqual(response.body, body_bytes) - with catch_warnings(): - filterwarnings("ignore", category=ScrapyDeprecationWarning) - self.assertEqual(response.body_as_unicode(), body_unicode) self.assertEqual(response.text, body_unicode) def _assert_response_encoding(self, response, encoding): @@ -346,12 +341,6 @@ class TextResponseTest(BaseResponseTest): original_string = unicode_string.encode('cp1251') r1 = self.response_class('http://www.example.com', body=original_string, encoding='cp1251') - # check body_as_unicode - with catch_warnings(): - filterwarnings("ignore", category=ScrapyDeprecationWarning) - self.assertTrue(isinstance(r1.body_as_unicode(), str)) - self.assertEqual(r1.body_as_unicode(), unicode_string) - # check response.text self.assertTrue(isinstance(r1.text, str)) self.assertEqual(r1.text, unicode_string) @@ -683,13 +672,6 @@ class TextResponseTest(BaseResponseTest): with self.assertRaises(ValueError): response.follow_all(css='a[href*="example.com"]', xpath='//a[contains(@href, "example.com")]') - def test_body_as_unicode_deprecation_warning(self): - with catch_warnings(record=True) as warnings: - r1 = self.response_class("http://www.example.com", body='Hello', encoding='utf-8') - self.assertEqual(r1.body_as_unicode(), 'Hello') - self.assertEqual(len(warnings), 1) - self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - def test_json_response(self): json_body = b"""{"ip": "109.187.217.200"}""" json_response = self.response_class("http://www.example.com", body=json_body) From 38d2a154ec79767558f699ec663697ccc7f64ca8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?P=C3=A9ter=20Ferenc=20Gyarmati?= Date: Sun, 6 Feb 2022 18:52:15 +0100 Subject: [PATCH 076/126] docs: use https scheme for each quotes.toscrape.com url occurrence --- docs/intro/examples.rst | 2 +- docs/intro/overview.rst | 4 +-- docs/intro/tutorial.rst | 52 ++++++++++++++++----------------- docs/topics/developer-tools.rst | 20 ++++++------- docs/topics/logging.rst | 2 +- docs/topics/settings.rst | 4 +-- docs/topics/signals.rst | 2 +- 7 files changed, 43 insertions(+), 43 deletions(-) diff --git a/docs/intro/examples.rst b/docs/intro/examples.rst index 96363c7d5..edff894c6 100644 --- a/docs/intro/examples.rst +++ b/docs/intro/examples.rst @@ -7,7 +7,7 @@ Examples The best way to learn is with examples, and Scrapy is no exception. For this reason, there is an example Scrapy project named quotesbot_, that you can use to play and learn more about Scrapy. It contains two spiders for -http://quotes.toscrape.com, one using CSS selectors and another one using XPath +https://quotes.toscrape.com, one using CSS selectors and another one using XPath expressions. The quotesbot_ project is available at: https://github.com/scrapy/quotesbot. diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 405bf845d..f3d652621 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -20,7 +20,7 @@ In order to show you what Scrapy brings to the table, we'll walk you through an example of a Scrapy Spider using the simplest way to run a spider. Here's the code for a spider that scrapes famous quotes from website -http://quotes.toscrape.com, following the pagination:: +https://quotes.toscrape.com, following the pagination:: import scrapy @@ -28,7 +28,7 @@ http://quotes.toscrape.com, following the pagination:: class QuotesSpider(scrapy.Spider): name = 'quotes' start_urls = [ - 'http://quotes.toscrape.com/tag/humor/', + 'https://quotes.toscrape.com/tag/humor/', ] def parse(self, response): diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index ca5856881..5697b9608 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -7,7 +7,7 @@ Scrapy Tutorial In this tutorial, we'll assume that Scrapy is already installed on your system. If that's not the case, see :ref:`intro-install`. -We are going to scrape `quotes.toscrape.com `_, a website +We are going to scrape `quotes.toscrape.com `_, a website that lists quotes from famous authors. This tutorial will walk you through these tasks: @@ -93,8 +93,8 @@ This is the code for our first Spider. Save it in a file named def start_requests(self): urls = [ - 'http://quotes.toscrape.com/page/1/', - 'http://quotes.toscrape.com/page/2/', + 'https://quotes.toscrape.com/page/1/', + 'https://quotes.toscrape.com/page/2/', ] for url in urls: yield scrapy.Request(url=url, callback=self.parse) @@ -143,9 +143,9 @@ similar to this:: 2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened 2016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 - 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None) - 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) - 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) + 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None) + 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) + 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html 2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html 2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished) @@ -184,8 +184,8 @@ for your spider:: class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ - 'http://quotes.toscrape.com/page/1/', - 'http://quotes.toscrape.com/page/2/', + 'https://quotes.toscrape.com/page/1/', + 'https://quotes.toscrape.com/page/2/', ] def parse(self, response): @@ -207,7 +207,7 @@ Extracting data The best way to learn how to extract data with Scrapy is trying selectors using the :ref:`Scrapy shell `. Run:: - scrapy shell 'http://quotes.toscrape.com/page/1/' + scrapy shell 'https://quotes.toscrape.com/page/1/' .. note:: @@ -217,18 +217,18 @@ using the :ref:`Scrapy shell `. Run:: On Windows, use double quotes instead:: - scrapy shell "http://quotes.toscrape.com/page/1/" + scrapy shell "https://quotes.toscrape.com/page/1/" You will see something like:: [ ... Scrapy log here ... ] - 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) + 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) [s] Available Scrapy objects: [s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc) [s] crawler [s] item {} - [s] request - [s] response <200 http://quotes.toscrape.com/page/1/> + [s] request + [s] response <200 https://quotes.toscrape.com/page/1/> [s] settings [s] spider [s] Useful shortcuts: @@ -241,7 +241,7 @@ object: .. invisible-code-block: python - response = load_response('http://quotes.toscrape.com/page/1/', 'quotes1.html') + response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html') >>> response.css('title') [] @@ -355,7 +355,7 @@ Extracting quotes and authors Now that you know a bit about selection and extraction, let's complete our spider by writing the code to extract the quotes from the web page. -Each quote in http://quotes.toscrape.com is represented by HTML elements that look +Each quote in https://quotes.toscrape.com is represented by HTML elements that look like this: .. code-block:: html @@ -379,7 +379,7 @@ like this: Let's open up scrapy shell and play a bit to find out how to extract the data we want:: - $ scrapy shell 'http://quotes.toscrape.com' + $ scrapy shell 'https://quotes.toscrape.com' We get a list of selectors for the quote HTML elements with: @@ -444,8 +444,8 @@ in the callback, as you can see below:: class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ - 'http://quotes.toscrape.com/page/1/', - 'http://quotes.toscrape.com/page/2/', + 'https://quotes.toscrape.com/page/1/', + 'https://quotes.toscrape.com/page/2/', ] def parse(self, response): @@ -458,9 +458,9 @@ in the callback, as you can see below:: If you run this spider, it will output the extracted data with the log:: - 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/> + 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/> {'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'} - 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/> + 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/> {'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"} @@ -505,7 +505,7 @@ Following links =============== Let's say, instead of just scraping the stuff from the first two pages -from http://quotes.toscrape.com, you want quotes from all the pages in the website. +from https://quotes.toscrape.com, you want quotes from all the pages in the website. Now that you know how to extract data from pages, let's see how to follow links from them. @@ -549,7 +549,7 @@ page, extracting data from it:: class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ - 'http://quotes.toscrape.com/page/1/', + 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): @@ -600,7 +600,7 @@ As a shortcut for creating Request objects you can use class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ - 'http://quotes.toscrape.com/page/1/', + 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): @@ -654,7 +654,7 @@ this time for scraping author information:: class AuthorSpider(scrapy.Spider): name = 'author' - start_urls = ['http://quotes.toscrape.com/'] + start_urls = ['https://quotes.toscrape.com/'] def parse(self, response): author_page_links = response.css('.author + a') @@ -727,7 +727,7 @@ with a specific tag, building the URL based on the argument:: name = "quotes" def start_requests(self): - url = 'http://quotes.toscrape.com/' + url = 'https://quotes.toscrape.com/' tag = getattr(self, 'tag', None) if tag is not None: url = url + 'tag/' + tag @@ -747,7 +747,7 @@ with a specific tag, building the URL based on the argument:: If you pass the ``tag=humor`` argument to this spider, you'll notice that it will only visit URLs from the ``humor`` tag, such as -``http://quotes.toscrape.com/tag/humor``. +``https://quotes.toscrape.com/tag/humor``. You can :ref:`learn more about handling spider arguments here `. diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst index 057b1ec62..96475899f 100644 --- a/docs/topics/developer-tools.rst +++ b/docs/topics/developer-tools.rst @@ -81,18 +81,18 @@ clicking directly on the tag. If we expand the ``span`` tag with the ``class= "text"`` we will see the quote-text we clicked on. The `Inspector` lets you copy XPaths to selected elements. Let's try it out. -First open the Scrapy shell at http://quotes.toscrape.com/ in a terminal: +First open the Scrapy shell at https://quotes.toscrape.com/ in a terminal: .. code-block:: none - $ scrapy shell "http://quotes.toscrape.com/" + $ scrapy shell "https://quotes.toscrape.com/" Then, back to your web browser, right-click on the ``span`` tag, select ``Copy > XPath`` and paste it in the Scrapy shell like so: .. invisible-code-block: python - response = load_response('http://quotes.toscrape.com/', 'quotes.html') + response = load_response('https://quotes.toscrape.com/', 'quotes.html') >>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall() ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'] @@ -227,7 +227,7 @@ interests us is the one request called ``quotes?page=1`` with the type ``json``. If we click on this request, we see that the request URL is -``http://quotes.toscrape.com/api/quotes?page=1`` and the response +``https://quotes.toscrape.com/api/quotes?page=1`` and the response is a JSON-object that contains our quotes. We can also right-click on the request and open ``Open in new tab`` to get a better overview. @@ -247,7 +247,7 @@ also request each page to get every quote on the site:: name = 'quote' allowed_domains = ['quotes.toscrape.com'] page = 1 - start_urls = ['http://quotes.toscrape.com/api/quotes?page=1'] + start_urls = ['https://quotes.toscrape.com/api/quotes?page=1'] def parse(self, response): data = json.loads(response.text) @@ -255,7 +255,7 @@ also request each page to get every quote on the site:: yield {"quote": quote["text"]} if data["has_next"]: self.page += 1 - url = f"http://quotes.toscrape.com/api/quotes?page={self.page}" + url = f"https://quotes.toscrape.com/api/quotes?page={self.page}" yield scrapy.Request(url=url, callback=self.parse) This spider starts at the first page of the quotes-API. With each @@ -280,7 +280,7 @@ request:: from scrapy import Request request = Request.from_curl( - "curl 'http://quotes.toscrape.com/api/quotes?page=1' -H 'User-Agent: Mozil" + "curl 'https://quotes.toscrape.com/api/quotes?page=1' -H 'User-Agent: Mozil" "la/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0' -H 'Acce" "pt: */*' -H 'Accept-Language: ca,en-US;q=0.7,en;q=0.3' --compressed -H 'X" "-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM" @@ -304,8 +304,8 @@ daunting and pages can be very complex, but it (mostly) boils down to identifying the correct request and replicating it in your spider. .. _Developer Tools: https://en.wikipedia.org/wiki/Web_development_tools -.. _quotes.toscrape.com: http://quotes.toscrape.com -.. _quotes.toscrape.com/scroll: http://quotes.toscrape.com/scroll -.. _quotes.toscrape.com/api/quotes?page=10: http://quotes.toscrape.com/api/quotes?page=10 +.. _quotes.toscrape.com: https://quotes.toscrape.com +.. _quotes.toscrape.com/scroll: https://quotes.toscrape.com/scroll +.. _quotes.toscrape.com/api/quotes?page=10: https://quotes.toscrape.com/api/quotes?page=10 .. _has-class-extension: https://parsel.readthedocs.io/en/latest/usage.html#other-xpath-extensions diff --git a/docs/topics/logging.rst b/docs/topics/logging.rst index d593c74c6..3bf23d5f5 100644 --- a/docs/topics/logging.rst +++ b/docs/topics/logging.rst @@ -218,7 +218,7 @@ For example, let's say you're scraping a website which returns many HTTP 404 and 500 responses, and you want to hide all messages like this:: 2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring - response <500 http://quotes.toscrape.com/page/1-34/>: HTTP status code + response <500 https://quotes.toscrape.com/page/1-34/>: HTTP status code is not handled or not allowed The first thing to note is a logger name - it is in brackets: diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index f6c95c502..4e105642d 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1597,7 +1597,7 @@ In order to use the reactor installed by Scrapy:: def start_requests(self): reactor.callLater(self.timeout, self.stop) - urls = ['http://quotes.toscrape.com/page/1'] + urls = ['https://quotes.toscrape.com/page/1'] for url in urls: yield scrapy.Request(url=url, callback=self.parse) @@ -1625,7 +1625,7 @@ which raises :exc:`Exception`, becomes:: from twisted.internet import reactor reactor.callLater(self.timeout, self.stop) - urls = ['http://quotes.toscrape.com/page/1'] + urls = ['https://quotes.toscrape.com/page/1'] for url in urls: yield scrapy.Request(url=url, callback=self.parse) diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 63ad3a9ad..2fbd0b51c 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -60,7 +60,7 @@ Let's take an example:: class SignalSpider(scrapy.Spider): name = 'signals' - start_urls = ['http://quotes.toscrape.com/page/1/'] + start_urls = ['https://quotes.toscrape.com/page/1/'] @classmethod def from_crawler(cls, crawler, *args, **kwargs): From bbfa185664cef79299b48cb0ae22065439bb07fc Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Sun, 6 Feb 2022 18:12:28 -0300 Subject: [PATCH 077/126] Remove deprecated BaseItem class --- scrapy/exporters.py | 4 +-- scrapy/item.py | 34 ++------------------ scrapy/utils/misc.py | 4 +-- tests/test_item.py | 75 +------------------------------------------- 4 files changed, 8 insertions(+), 109 deletions(-) diff --git a/scrapy/exporters.py b/scrapy/exporters.py index 36cca2d05..1c26e81db 100644 --- a/scrapy/exporters.py +++ b/scrapy/exporters.py @@ -13,7 +13,7 @@ from xml.sax.saxutils import XMLGenerator from itemadapter import is_item, ItemAdapter from scrapy.exceptions import ScrapyDeprecationWarning -from scrapy.item import _BaseItem +from scrapy.item import Item from scrapy.utils.python import is_listlike, to_bytes, to_unicode from scrapy.utils.serialize import ScrapyJSONEncoder @@ -315,7 +315,7 @@ class PythonItemExporter(BaseItemExporter): return serializer(value) def _serialize_value(self, value): - if isinstance(value, _BaseItem): + if isinstance(value, Item): return self.export_item(value) elif is_item(value): return dict(self._serialize_item(value)) diff --git a/scrapy/item.py b/scrapy/item.py index 2ccd7ad18..839bee3fa 100644 --- a/scrapy/item.py +++ b/scrapy/item.py @@ -15,39 +15,11 @@ from scrapy.utils.deprecate import ScrapyDeprecationWarning from scrapy.utils.trackref import object_ref -class _BaseItem(object_ref): - """ - Temporary class used internally to avoid the deprecation - warning raised by isinstance checks using BaseItem. - """ - pass - - -class _BaseItemMeta(ABCMeta): - def __instancecheck__(cls, instance): - if cls is BaseItem: - warn('scrapy.item.BaseItem is deprecated, please use scrapy.item.Item instead', - ScrapyDeprecationWarning, stacklevel=2) - return super().__instancecheck__(instance) - - -class BaseItem(_BaseItem, metaclass=_BaseItemMeta): - """ - Deprecated, please use :class:`scrapy.item.Item` instead - """ - - def __new__(cls, *args, **kwargs): - if issubclass(cls, BaseItem) and not issubclass(cls, (Item, DictItem)): - warn('scrapy.item.BaseItem is deprecated, please use scrapy.item.Item instead', - ScrapyDeprecationWarning, stacklevel=2) - return super().__new__(cls, *args, **kwargs) - - class Field(dict): """Container of field metadata""" -class ItemMeta(_BaseItemMeta): +class ItemMeta(ABCMeta): """Metaclass_ of :class:`Item` that handles field definitions. .. _metaclass: https://realpython.com/python-metaclasses @@ -74,7 +46,7 @@ class ItemMeta(_BaseItemMeta): return super().__new__(mcs, class_name, bases, new_attrs) -class DictItem(MutableMapping, BaseItem): +class DictItem(MutableMapping, object_ref): fields: Dict[str, Field] = {} @@ -118,7 +90,7 @@ class DictItem(MutableMapping, BaseItem): def __iter__(self): return iter(self._values) - __hash__ = BaseItem.__hash__ + __hash__ = object_ref.__hash__ def keys(self): return self._values.keys() diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py index 11c4206c2..1221b39b2 100644 --- a/scrapy/utils/misc.py +++ b/scrapy/utils/misc.py @@ -14,11 +14,11 @@ from w3lib.html import replace_entities from scrapy.utils.datatypes import LocalWeakReferencedCache from scrapy.utils.python import flatten, to_unicode -from scrapy.item import _BaseItem +from scrapy.item import Item from scrapy.utils.deprecate import ScrapyDeprecationWarning -_ITERABLE_SINGLE_VALUES = dict, _BaseItem, str, bytes +_ITERABLE_SINGLE_VALUES = dict, Item, str, bytes def arg_to_iter(arg): diff --git a/tests/test_item.py b/tests/test_item.py index c94bb44af..7d82fbffe 100644 --- a/tests/test_item.py +++ b/tests/test_item.py @@ -3,7 +3,7 @@ from unittest import mock from warnings import catch_warnings, filterwarnings from scrapy.exceptions import ScrapyDeprecationWarning -from scrapy.item import ABCMeta, _BaseItem, BaseItem, DictItem, Field, Item, ItemMeta +from scrapy.item import ABCMeta, DictItem, Field, Item, ItemMeta class ItemTest(unittest.TestCase): @@ -318,79 +318,6 @@ class DictItemTest(unittest.TestCase): self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) -class BaseItemTest(unittest.TestCase): - - def test_isinstance_check(self): - - class SubclassedBaseItem(BaseItem): - pass - - class SubclassedItem(Item): - pass - - with catch_warnings(): - filterwarnings("ignore", category=ScrapyDeprecationWarning) - self.assertTrue(isinstance(BaseItem(), BaseItem)) - self.assertTrue(isinstance(SubclassedBaseItem(), BaseItem)) - self.assertTrue(isinstance(Item(), BaseItem)) - self.assertTrue(isinstance(SubclassedItem(), BaseItem)) - - # make sure internal checks using private _BaseItem class succeed - self.assertTrue(isinstance(BaseItem(), _BaseItem)) - self.assertTrue(isinstance(SubclassedBaseItem(), _BaseItem)) - self.assertTrue(isinstance(Item(), _BaseItem)) - self.assertTrue(isinstance(SubclassedItem(), _BaseItem)) - - def test_deprecation_warning(self): - """ - Make sure deprecation warnings are logged whenever BaseItem is used, - either instantiated or in an isinstance check - """ - with catch_warnings(record=True) as warnings: - BaseItem() - self.assertEqual(len(warnings), 1) - self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - - with catch_warnings(record=True) as warnings: - - class SubclassedBaseItem(BaseItem): - pass - - SubclassedBaseItem() - self.assertEqual(len(warnings), 1) - self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - - with catch_warnings(record=True) as warnings: - self.assertFalse(isinstance("foo", BaseItem)) - self.assertEqual(len(warnings), 1) - self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - - with catch_warnings(record=True) as warnings: - self.assertTrue(isinstance(BaseItem(), BaseItem)) - self.assertEqual(len(warnings), 1) - self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - - -class ItemNoDeprecationWarningTest(unittest.TestCase): - def test_no_deprecation_warning(self): - """ - Make sure deprecation warnings are NOT logged whenever BaseItem subclasses are used. - """ - class SubclassedItem(Item): - pass - - with catch_warnings(record=True) as warnings: - Item() - SubclassedItem() - _BaseItem() - self.assertFalse(isinstance("foo", _BaseItem)) - self.assertFalse(isinstance("foo", Item)) - self.assertFalse(isinstance("foo", SubclassedItem)) - self.assertTrue(isinstance(_BaseItem(), _BaseItem)) - self.assertTrue(isinstance(Item(), Item)) - self.assertTrue(isinstance(SubclassedItem(), SubclassedItem)) - self.assertEqual(len(warnings), 0) - if __name__ == "__main__": unittest.main() From c8c1edd43b04ce7a2d9b1da198af26ba271cb1d6 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Sun, 6 Feb 2022 18:27:41 -0300 Subject: [PATCH 078/126] Flake8 adjustments --- tests/test_item.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/test_item.py b/tests/test_item.py index 7d82fbffe..a12e425e0 100644 --- a/tests/test_item.py +++ b/tests/test_item.py @@ -1,6 +1,6 @@ import unittest from unittest import mock -from warnings import catch_warnings, filterwarnings +from warnings import catch_warnings from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.item import ABCMeta, DictItem, Field, Item, ItemMeta @@ -318,6 +318,5 @@ class DictItemTest(unittest.TestCase): self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - if __name__ == "__main__": unittest.main() From fca49cca929de035fb5d179d83f7f79da22fd205 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Sun, 6 Feb 2022 18:31:55 -0300 Subject: [PATCH 079/126] Remove deprecated DictItem class --- docs/conf.py | 1 - scrapy/item.py | 55 +++++++++++++++++++--------------------------- tests/test_item.py | 31 +------------------------- 3 files changed, 23 insertions(+), 64 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index 406c4d94a..d5e139e66 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -272,7 +272,6 @@ coverage_ignore_pyobjects = [ r'^scrapy\.extensions\.[a-z]\w*?\.[a-z]', # helper functions # Never documented before, and deprecated now. - r'^scrapy\.item\.DictItem$', r'^scrapy\.linkextractors\.FilteringLinkExtractor$', # Implementation detail of LxmlLinkExtractor diff --git a/scrapy/item.py b/scrapy/item.py index 839bee3fa..2521ac829 100644 --- a/scrapy/item.py +++ b/scrapy/item.py @@ -9,9 +9,7 @@ from collections.abc import MutableMapping from copy import deepcopy from pprint import pformat from typing import Dict -from warnings import warn -from scrapy.utils.deprecate import ScrapyDeprecationWarning from scrapy.utils.trackref import object_ref @@ -46,15 +44,30 @@ class ItemMeta(ABCMeta): return super().__new__(mcs, class_name, bases, new_attrs) -class DictItem(MutableMapping, object_ref): +class Item(MutableMapping, object_ref, metaclass=ItemMeta): + """ + Base class for scraped items. - fields: Dict[str, Field] = {} + In Scrapy, an object is considered an ``item`` if it is an instance of either + :class:`Item` or :class:`dict`, or any subclass. For example, when the output of a + spider callback is evaluated, only instances of :class:`Item` or + :class:`dict` are passed to :ref:`item pipelines `. - def __new__(cls, *args, **kwargs): - if issubclass(cls, DictItem) and not issubclass(cls, Item): - warn('scrapy.item.DictItem is deprecated, please use scrapy.item.Item instead', - ScrapyDeprecationWarning, stacklevel=2) - return super().__new__(cls, *args, **kwargs) + If you need instances of a custom class to be considered items by Scrapy, + you must inherit from either :class:`Item` or :class:`dict`. + + Items must declare :class:`Field` attributes, which are processed and stored + in the ``fields`` attribute. This restricts the set of allowed field names + and prevents typos, raising ``KeyError`` when referring to undefined fields. + Additionally, fields can be used to define metadata and control the way + data is processed internally. Please refer to the :ref:`documentation + about fields ` for additional information. + + Unlike instances of :class:`dict`, instances of :class:`Item` may be + :ref:`tracked ` to debug memory leaks. + """ + + fields: Dict[str, Field] def __init__(self, *args, **kwargs): self._values = {} @@ -105,27 +118,3 @@ class DictItem(MutableMapping, object_ref): """Return a :func:`~copy.deepcopy` of this item. """ return deepcopy(self) - - -class Item(DictItem, metaclass=ItemMeta): - """ - Base class for scraped items. - - In Scrapy, an object is considered an ``item`` if it is an instance of either - :class:`Item` or :class:`dict`, or any subclass. For example, when the output of a - spider callback is evaluated, only instances of :class:`Item` or - :class:`dict` are passed to :ref:`item pipelines `. - - If you need instances of a custom class to be considered items by Scrapy, - you must inherit from either :class:`Item` or :class:`dict`. - - Items must declare :class:`Field` attributes, which are processed and stored - in the ``fields`` attribute. This restricts the set of allowed field names - and prevents typos, raising ``KeyError`` when referring to undefined fields. - Additionally, fields can be used to define metadata and control the way - data is processed internally. Please refer to the :ref:`documentation - about fields ` for additional information. - - Unlike instances of :class:`dict`, instances of :class:`Item` may be - :ref:`tracked ` to debug memory leaks. - """ diff --git a/tests/test_item.py b/tests/test_item.py index a12e425e0..25f2aea0a 100644 --- a/tests/test_item.py +++ b/tests/test_item.py @@ -1,9 +1,7 @@ import unittest from unittest import mock -from warnings import catch_warnings -from scrapy.exceptions import ScrapyDeprecationWarning -from scrapy.item import ABCMeta, DictItem, Field, Item, ItemMeta +from scrapy.item import ABCMeta, Field, Item, ItemMeta class ItemTest(unittest.TestCase): @@ -254,18 +252,6 @@ class ItemTest(unittest.TestCase): item['tags'].append('tag2') assert item['tags'] != copied_item['tags'] - def test_dictitem_deprecation_warning(self): - """Make sure the DictItem deprecation warning is not issued for - Item""" - with catch_warnings(record=True) as warnings: - Item() - self.assertEqual(len(warnings), 0) - - class SubclassedItem(Item): - pass - SubclassedItem() - self.assertEqual(len(warnings), 0) - class ItemMetaTest(unittest.TestCase): @@ -303,20 +289,5 @@ class ItemMetaClassCellRegression(unittest.TestCase): super().__init__(*args, **kwargs) -class DictItemTest(unittest.TestCase): - - def test_deprecation_warning(self): - with catch_warnings(record=True) as warnings: - DictItem() - self.assertEqual(len(warnings), 1) - self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - with catch_warnings(record=True) as warnings: - class SubclassedDictItem(DictItem): - pass - SubclassedDictItem() - self.assertEqual(len(warnings), 1) - self.assertEqual(warnings[0].category, ScrapyDeprecationWarning) - - if __name__ == "__main__": unittest.main() From b282a7af012a4804eb91bdd850df3b86065b3fd6 Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Tue, 8 Feb 2022 01:25:08 +0500 Subject: [PATCH 080/126] Temporarily pin Twisted to an older version in CI (#5401) --- tox.ini | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tox.ini b/tox.ini index 2031a2d92..cf7855cf9 100644 --- a/tox.ini +++ b/tox.ini @@ -19,6 +19,8 @@ deps = mitmproxy >= 4.0.4, < 5; python_version >= '3.6' and python_version < '3.7' and platform_system != 'Windows' and implementation_name != 'pypy' # Extras botocore>=1.4.87 + # Temporary until the tests are updated + Twisted<22.1.0 passenv = S3_TEST_FILE_URI AWS_ACCESS_KEY_ID From 4bda0976b28917405f54c781afda5ea55b65b16b Mon Sep 17 00:00:00 2001 From: Laerte <5853172+Laerte@users.noreply.github.com> Date: Tue, 8 Feb 2022 10:57:19 -0300 Subject: [PATCH 081/126] Fix csviter call, add parse_rows test (#5394) --- scrapy/spiders/feed.py | 2 +- tests/test_spider.py | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/scrapy/spiders/feed.py b/scrapy/spiders/feed.py index bef2d6b24..79e12e030 100644 --- a/scrapy/spiders/feed.py +++ b/scrapy/spiders/feed.py @@ -123,7 +123,7 @@ class CSVFeedSpider(Spider): process_results methods for pre and post-processing purposes. """ - for row in csviter(response, self.delimiter, self.headers, self.quotechar): + for row in csviter(response, self.delimiter, self.headers, quotechar=self.quotechar): ret = iterate_spider_output(self.parse_row(response, row)) for result_item in self.process_results(response, ret): yield result_item diff --git a/tests/test_spider.py b/tests/test_spider.py index a7c3ee048..689349999 100644 --- a/tests/test_spider.py +++ b/tests/test_spider.py @@ -21,6 +21,7 @@ from scrapy.spiders import ( ) from scrapy.linkextractors import LinkExtractor from scrapy.utils.test import get_crawler +from tests import get_testdata class SpiderTest(unittest.TestCase): @@ -167,6 +168,23 @@ class CSVFeedSpiderTest(SpiderTest): spider_class = CSVFeedSpider + def test_parse_rows(self): + body = get_testdata('feeds', 'feed-sample6.csv') + response = Response("http://example.org/dummy.csv", body=body) + + class _CrawlSpider(self.spider_class): + name = "test" + delimiter = "," + quotechar = "'" + + def parse_row(self, response, row): + return row + + spider = _CrawlSpider() + rows = list(spider.parse_rows(response)) + assert rows[0] == {'id': '1', 'name': 'alpha', 'value': 'foobar'} + assert len(rows) == 4 + class CrawlSpiderTest(SpiderTest): From fd55f62207bbbb18d7758c8e2ef46fe9115eb2c5 Mon Sep 17 00:00:00 2001 From: Raihan Nismara <31585789+raihan71@users.noreply.github.com> Date: Tue, 8 Feb 2022 21:36:25 +0700 Subject: [PATCH 082/126] Update Logo in README.rst (#5258) --- README.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 05f10bb6c..6b563d638 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,4 @@ -.. image:: /artwork/scrapy-logo.jpg - :width: 400px +.. image:: https://scrapy.org/img/scrapylogo.png ====== Scrapy From 1e1cfc26dbdde5fc3035169884c4d1218844d0de Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Tue, 8 Feb 2022 21:01:16 +0500 Subject: [PATCH 083/126] Copy resource classes from twisted.web.test.test_webclient. --- pytest.ini | 2 - tests/mockserver.py | 74 ++++++++++++++++++++++++++++--- tests/test_downloader_handlers.py | 12 +++-- tests/test_webclient.py | 18 ++++---- 4 files changed, 86 insertions(+), 20 deletions(-) diff --git a/pytest.ini b/pytest.ini index fa5d6b34f..ae2ed2029 100644 --- a/pytest.ini +++ b/pytest.ini @@ -21,5 +21,3 @@ addopts = markers = only_asyncio: marks tests as only enabled when --reactor=asyncio is passed only_not_asyncio: marks tests as only enabled when --reactor=asyncio is not passed -filterwarnings= - ignore::DeprecationWarning:twisted.web.test.test_webclient diff --git a/tests/mockserver.py b/tests/mockserver.py index ab9aec6a6..72d7e0241 100644 --- a/tests/mockserver.py +++ b/tests/mockserver.py @@ -14,10 +14,9 @@ from twisted.internet import defer, reactor, ssl from twisted.internet.task import deferLater from twisted.names import dns, error from twisted.names.server import DNSServerFactory -from twisted.web.resource import EncodingResourceWrapper, Resource +from twisted.web import resource, server from twisted.web.server import GzipEncoderFactory, NOT_DONE_YET, Site from twisted.web.static import File -from twisted.web.test.test_webclient import PayloadResource from twisted.web.util import redirectTo from scrapy.utils.python import to_bytes, to_unicode @@ -35,7 +34,70 @@ def getarg(request, name, default=None, type=None): return default -class LeafResource(Resource): +# most of the following resources are copied from twisted.web.test.test_webclient +class ForeverTakingResource(resource.Resource): + """ + L{ForeverTakingResource} is a resource which never finishes responding + to requests. + """ + + def __init__(self, write=False): + resource.Resource.__init__(self) + self._write = write + + def render(self, request): + if self._write: + request.write(b"some bytes") + return server.NOT_DONE_YET + + +class ErrorResource(resource.Resource): + def render(self, request): + request.setResponseCode(401) + if request.args.get(b"showlength"): + request.setHeader(b"content-length", b"0") + return b"" + + +class NoLengthResource(resource.Resource): + def render(self, request): + return b"nolength" + + +class HostHeaderResource(resource.Resource): + """ + A testing resource which renders itself as the value of the host header + from the request. + """ + + def render(self, request): + return request.requestHeaders.getRawHeaders(b"host")[0] + + +class PayloadResource(resource.Resource): + """ + A testing resource which renders itself as the contents of the request body + as long as the request body is 100 bytes long, otherwise which renders + itself as C{"ERROR"}. + """ + + def render(self, request): + data = request.content.read() + contentLength = request.requestHeaders.getRawHeaders(b"content-length")[0] + if len(data) != 100 or int(contentLength) != 100: + return b"ERROR" + return data + + +class BrokenDownloadResource(resource.Resource): + def render(self, request): + # only sends 3 bytes even though it claims to send 5 + request.setHeader(b"content-length", b"5") + request.write(b"abc") + return b"" + + +class LeafResource(resource.Resource): isLeaf = True @@ -175,10 +237,10 @@ class ArbitraryLengthPayloadResource(LeafResource): return request.content.read() -class Root(Resource): +class Root(resource.Resource): def __init__(self): - Resource.__init__(self) + resource.Resource.__init__(self) self.putChild(b"status", Status()) self.putChild(b"follow", Follow()) self.putChild(b"delay", Delay()) @@ -187,7 +249,7 @@ class Root(Resource): self.putChild(b"raw", Raw()) self.putChild(b"echo", Echo()) self.putChild(b"payload", PayloadResource()) - self.putChild(b"xpayload", EncodingResourceWrapper(PayloadResource(), [GzipEncoderFactory()])) + self.putChild(b"xpayload", resource.EncodingResourceWrapper(PayloadResource(), [GzipEncoderFactory()])) self.putChild(b"alpayload", ArbitraryLengthPayloadResource()) try: from tests import tests_datadir diff --git a/tests/test_downloader_handlers.py b/tests/test_downloader_handlers.py index a1ea4c679..2bb53950d 100644 --- a/tests/test_downloader_handlers.py +++ b/tests/test_downloader_handlers.py @@ -15,8 +15,6 @@ from twisted.trial import unittest from twisted.web import resource, server, static, util from twisted.web._newclient import ResponseFailed from twisted.web.http import _DataLoss -from twisted.web.test.test_webclient import (ForeverTakingResource, HostHeaderResource, - NoLengthResource, PayloadResource) from w3lib.url import path_to_file_uri from scrapy.core.downloader.handlers import DownloadHandlers @@ -34,7 +32,15 @@ from scrapy.spiders import Spider from scrapy.utils.misc import create_instance from scrapy.utils.python import to_bytes from scrapy.utils.test import get_crawler, skip_if_no_boto -from tests.mockserver import MockServer, ssl_context_factory, Echo +from tests.mockserver import ( + Echo, + ForeverTakingResource, + HostHeaderResource, + MockServer, + NoLengthResource, + PayloadResource, + ssl_context_factory, +) from tests.spiders import SingleRequestSpider diff --git a/tests/test_webclient.py b/tests/test_webclient.py index 6e4cb9b6e..a6d55cb38 100644 --- a/tests/test_webclient.py +++ b/tests/test_webclient.py @@ -21,14 +21,6 @@ except ImportError: from twisted.python.filepath import FilePath from twisted.protocols.policies import WrappingFactory from twisted.internet.defer import inlineCallbacks -from twisted.web.test.test_webclient import ( - ForeverTakingResource, - ErrorResource, - NoLengthResource, - HostHeaderResource, - PayloadResource, - BrokenDownloadResource, -) from scrapy.core.downloader import webclient as client from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory @@ -36,7 +28,15 @@ from scrapy.http import Request, Headers from scrapy.settings import Settings from scrapy.utils.misc import create_instance from scrapy.utils.python import to_bytes, to_unicode -from tests.mockserver import ssl_context_factory +from tests.mockserver import ( + BrokenDownloadResource, + ErrorResource, + ForeverTakingResource, + HostHeaderResource, + NoLengthResource, + PayloadResource, + ssl_context_factory, +) def getPage(url, contextFactory=None, response_transform=None, *args, **kwargs): From 77547a1ab554a5b9afa7d7a343d8f90ef1d4cfe8 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Tue, 8 Feb 2022 21:06:02 +0500 Subject: [PATCH 084/126] Revert "Temporarily pin Twisted to an older version in CI (#5401)" This reverts commit b282a7af012a4804eb91bdd850df3b86065b3fd6. --- tox.ini | 2 -- 1 file changed, 2 deletions(-) diff --git a/tox.ini b/tox.ini index cf7855cf9..2031a2d92 100644 --- a/tox.ini +++ b/tox.ini @@ -19,8 +19,6 @@ deps = mitmproxy >= 4.0.4, < 5; python_version >= '3.6' and python_version < '3.7' and platform_system != 'Windows' and implementation_name != 'pypy' # Extras botocore>=1.4.87 - # Temporary until the tests are updated - Twisted<22.1.0 passenv = S3_TEST_FILE_URI AWS_ACCESS_KEY_ID From e2e2ffd0d162cfed5a2e82e9fb9472dbf233c919 Mon Sep 17 00:00:00 2001 From: Alex Date: Wed, 9 Feb 2022 11:52:07 -0800 Subject: [PATCH 085/126] Move from optparse to argparse (#5374) --- scrapy/cmdline.py | 14 +++---- scrapy/commands/__init__.py | 75 ++++++++++++++++++++++++------------ scrapy/commands/check.py | 8 ++-- scrapy/commands/fetch.py | 10 ++--- scrapy/commands/genspider.py | 20 +++++----- scrapy/commands/parse.py | 44 ++++++++++----------- scrapy/commands/settings.py | 20 +++++----- scrapy/commands/shell.py | 12 +++--- scrapy/commands/version.py | 4 +- scrapy/commands/view.py | 3 +- tests/test_command_parse.py | 19 +++++++++ tests/test_commands.py | 38 ++++++++++++++---- 12 files changed, 168 insertions(+), 99 deletions(-) diff --git a/scrapy/cmdline.py b/scrapy/cmdline.py index 91482ce01..491c4beab 100644 --- a/scrapy/cmdline.py +++ b/scrapy/cmdline.py @@ -1,13 +1,13 @@ import sys import os -import optparse +import argparse import cProfile import inspect import pkg_resources import scrapy from scrapy.crawler import CrawlerProcess -from scrapy.commands import ScrapyCommand +from scrapy.commands import ScrapyCommand, ScrapyHelpFormatter from scrapy.exceptions import UsageError from scrapy.utils.misc import walk_modules from scrapy.utils.project import inside_project, get_project_settings @@ -123,8 +123,6 @@ def execute(argv=None, settings=None): inproject = inside_project() cmds = _get_commands_dict(settings, inproject) cmdname = _pop_command_name(argv) - parser = optparse.OptionParser(formatter=optparse.TitledHelpFormatter(), - conflict_handler='resolve') if not cmdname: _print_commands(settings, inproject) sys.exit(0) @@ -133,12 +131,14 @@ def execute(argv=None, settings=None): sys.exit(2) cmd = cmds[cmdname] - parser.usage = f"scrapy {cmdname} {cmd.syntax()}" - parser.description = cmd.long_desc() + parser = argparse.ArgumentParser(formatter_class=ScrapyHelpFormatter, + usage=f"scrapy {cmdname} {cmd.syntax()}", + conflict_handler='resolve', + description=cmd.long_desc()) settings.setdict(cmd.default_settings, priority='command') cmd.settings = settings cmd.add_options(parser) - opts, args = parser.parse_args(args=argv[1:]) + opts, args = parser.parse_known_args(args=argv[1:]) _run_print_help(parser, cmd.process_options, args, opts) cmd.crawler_process = CrawlerProcess(settings) diff --git a/scrapy/commands/__init__.py b/scrapy/commands/__init__.py index 5f1dabd33..fb304b8c0 100644 --- a/scrapy/commands/__init__.py +++ b/scrapy/commands/__init__.py @@ -2,7 +2,7 @@ Base class for Scrapy commands """ import os -from optparse import OptionGroup +import argparse from typing import Any, Dict from twisted.python import failure @@ -59,22 +59,20 @@ class ScrapyCommand: """ Populate option parse with options available for this command """ - group = OptionGroup(parser, "Global Options") - group.add_option("--logfile", metavar="FILE", - help="log file. if omitted stderr will be used") - group.add_option("-L", "--loglevel", metavar="LEVEL", default=None, - help=f"log level (default: {self.settings['LOG_LEVEL']})") - group.add_option("--nolog", action="store_true", - help="disable logging completely") - group.add_option("--profile", metavar="FILE", default=None, - help="write python cProfile stats to FILE") - group.add_option("--pidfile", metavar="FILE", - help="write process ID to FILE") - group.add_option("-s", "--set", action="append", default=[], metavar="NAME=VALUE", - help="set/override setting (may be repeated)") - group.add_option("--pdb", action="store_true", help="enable pdb on failure") - - parser.add_option_group(group) + group = parser.add_argument_group(title='Global Options') + group.add_argument("--logfile", metavar="FILE", + help="log file. if omitted stderr will be used") + group.add_argument("-L", "--loglevel", metavar="LEVEL", default=None, + help=f"log level (default: {self.settings['LOG_LEVEL']})") + group.add_argument("--nolog", action="store_true", + help="disable logging completely") + group.add_argument("--profile", metavar="FILE", default=None, + help="write python cProfile stats to FILE") + group.add_argument("--pidfile", metavar="FILE", + help="write process ID to FILE") + group.add_argument("-s", "--set", action="append", default=[], metavar="NAME=VALUE", + help="set/override setting (may be repeated)") + group.add_argument("--pdb", action="store_true", help="enable pdb on failure") def process_options(self, args, opts): try: @@ -114,14 +112,14 @@ class BaseRunSpiderCommand(ScrapyCommand): """ def add_options(self, parser): ScrapyCommand.add_options(self, parser) - parser.add_option("-a", dest="spargs", action="append", default=[], metavar="NAME=VALUE", - help="set spider argument (may be repeated)") - parser.add_option("-o", "--output", metavar="FILE", action="append", - help="append scraped items to the end of FILE (use - for stdout)") - parser.add_option("-O", "--overwrite-output", metavar="FILE", action="append", - help="dump scraped items into FILE, overwriting any existing file") - parser.add_option("-t", "--output-format", metavar="FORMAT", - help="format to use for dumping items") + parser.add_argument("-a", dest="spargs", action="append", default=[], metavar="NAME=VALUE", + help="set spider argument (may be repeated)") + parser.add_argument("-o", "--output", metavar="FILE", action="append", + help="append scraped items to the end of FILE (use - for stdout)") + parser.add_argument("-O", "--overwrite-output", metavar="FILE", action="append", + help="dump scraped items into FILE, overwriting any existing file") + parser.add_argument("-t", "--output-format", metavar="FORMAT", + help="format to use for dumping items") def process_options(self, args, opts): ScrapyCommand.process_options(self, args, opts) @@ -137,3 +135,30 @@ class BaseRunSpiderCommand(ScrapyCommand): opts.overwrite_output, ) self.settings.set('FEEDS', feeds, priority='cmdline') + + +class ScrapyHelpFormatter(argparse.HelpFormatter): + """ + Help Formatter for scrapy command line help messages. + """ + def __init__(self, prog, indent_increment=2, max_help_position=24, width=None): + super().__init__(prog, indent_increment=indent_increment, + max_help_position=max_help_position, width=width) + + def _join_parts(self, part_strings): + parts = self.format_part_strings(part_strings) + return super()._join_parts(parts) + + def format_part_strings(self, part_strings): + """ + Underline and title case command line help message headers. + """ + if part_strings and part_strings[0].startswith("usage: "): + part_strings[0] = "Usage\n=====\n " + part_strings[0][len('usage: '):] + headings = [i for i in range(len(part_strings)) if part_strings[i].endswith(':\n')] + for index in headings[::-1]: + char = '-' if "Global Options" in part_strings[index] else '=' + part_strings[index] = part_strings[index][:-2].title() + underline = ''.join(["\n", (char * len(part_strings[index])), "\n"]) + part_strings.insert(index + 1, underline) + return part_strings diff --git a/scrapy/commands/check.py b/scrapy/commands/check.py index ae21d86e6..a16f4beb7 100644 --- a/scrapy/commands/check.py +++ b/scrapy/commands/check.py @@ -49,10 +49,10 @@ class Command(ScrapyCommand): def add_options(self, parser): ScrapyCommand.add_options(self, parser) - parser.add_option("-l", "--list", dest="list", action="store_true", - help="only list contracts, without checking them") - parser.add_option("-v", "--verbose", dest="verbose", default=False, action='store_true', - help="print contract tests for all spiders") + parser.add_argument("-l", "--list", dest="list", action="store_true", + help="only list contracts, without checking them") + parser.add_argument("-v", "--verbose", dest="verbose", default=False, action='store_true', + help="print contract tests for all spiders") def run(self, args, opts): # load contracts diff --git a/scrapy/commands/fetch.py b/scrapy/commands/fetch.py index 95f87e8c3..9b2ebb37f 100644 --- a/scrapy/commands/fetch.py +++ b/scrapy/commands/fetch.py @@ -26,11 +26,11 @@ class Command(ScrapyCommand): def add_options(self, parser): ScrapyCommand.add_options(self, parser) - parser.add_option("--spider", dest="spider", help="use this spider") - parser.add_option("--headers", dest="headers", action="store_true", - help="print response HTTP headers instead of body") - parser.add_option("--no-redirect", dest="no_redirect", action="store_true", default=False, - help="do not handle HTTP 3xx status codes and print response as-is") + parser.add_argument("--spider", dest="spider", help="use this spider") + parser.add_argument("--headers", dest="headers", action="store_true", + help="print response HTTP headers instead of body") + parser.add_argument("--no-redirect", dest="no_redirect", action="store_true", default=False, + help="do not handle HTTP 3xx status codes and print response as-is") def _print_headers(self, headers, prefix): for key, values in headers.items(): diff --git a/scrapy/commands/genspider.py b/scrapy/commands/genspider.py index 2082a4974..ed5f588e9 100644 --- a/scrapy/commands/genspider.py +++ b/scrapy/commands/genspider.py @@ -44,16 +44,16 @@ class Command(ScrapyCommand): def add_options(self, parser): ScrapyCommand.add_options(self, parser) - parser.add_option("-l", "--list", dest="list", action="store_true", - help="List available templates") - parser.add_option("-e", "--edit", dest="edit", action="store_true", - help="Edit spider after creating it") - parser.add_option("-d", "--dump", dest="dump", metavar="TEMPLATE", - help="Dump template to standard output") - parser.add_option("-t", "--template", dest="template", default="basic", - help="Uses a custom template.") - parser.add_option("--force", dest="force", action="store_true", - help="If the spider already exists, overwrite it with the template") + parser.add_argument("-l", "--list", dest="list", action="store_true", + help="List available templates") + parser.add_argument("-e", "--edit", dest="edit", action="store_true", + help="Edit spider after creating it") + parser.add_argument("-d", "--dump", dest="dump", metavar="TEMPLATE", + help="Dump template to standard output") + parser.add_argument("-t", "--template", dest="template", default="basic", + help="Uses a custom template.") + parser.add_argument("--force", dest="force", action="store_true", + help="If the spider already exists, overwrite it with the template") def run(self, args, opts): if opts.list: diff --git a/scrapy/commands/parse.py b/scrapy/commands/parse.py index 52118db1b..a3f6b96f4 100644 --- a/scrapy/commands/parse.py +++ b/scrapy/commands/parse.py @@ -32,28 +32,28 @@ class Command(BaseRunSpiderCommand): def add_options(self, parser): BaseRunSpiderCommand.add_options(self, parser) - parser.add_option("--spider", dest="spider", default=None, - help="use this spider without looking for one") - parser.add_option("--pipelines", action="store_true", - help="process items through pipelines") - parser.add_option("--nolinks", dest="nolinks", action="store_true", - help="don't show links to follow (extracted requests)") - parser.add_option("--noitems", dest="noitems", action="store_true", - help="don't show scraped items") - parser.add_option("--nocolour", dest="nocolour", action="store_true", - help="avoid using pygments to colorize the output") - parser.add_option("-r", "--rules", dest="rules", action="store_true", - help="use CrawlSpider rules to discover the callback") - parser.add_option("-c", "--callback", dest="callback", - help="use this callback for parsing, instead looking for a callback") - parser.add_option("-m", "--meta", dest="meta", - help="inject extra meta into the Request, it must be a valid raw json string") - parser.add_option("--cbkwargs", dest="cbkwargs", - help="inject extra callback kwargs into the Request, it must be a valid raw json string") - parser.add_option("-d", "--depth", dest="depth", type="int", default=1, - help="maximum depth for parsing requests [default: %default]") - parser.add_option("-v", "--verbose", dest="verbose", action="store_true", - help="print each depth level one by one") + parser.add_argument("--spider", dest="spider", default=None, + help="use this spider without looking for one") + parser.add_argument("--pipelines", action="store_true", + help="process items through pipelines") + parser.add_argument("--nolinks", dest="nolinks", action="store_true", + help="don't show links to follow (extracted requests)") + parser.add_argument("--noitems", dest="noitems", action="store_true", + help="don't show scraped items") + parser.add_argument("--nocolour", dest="nocolour", action="store_true", + help="avoid using pygments to colorize the output") + parser.add_argument("-r", "--rules", dest="rules", action="store_true", + help="use CrawlSpider rules to discover the callback") + parser.add_argument("-c", "--callback", dest="callback", + help="use this callback for parsing, instead looking for a callback") + parser.add_argument("-m", "--meta", dest="meta", + help="inject extra meta into the Request, it must be a valid raw json string") + parser.add_argument("--cbkwargs", dest="cbkwargs", + help="inject extra callback kwargs into the Request, it must be a valid raw json string") + parser.add_argument("-d", "--depth", dest="depth", type=int, default=1, + help="maximum depth for parsing requests [default: %default]") + parser.add_argument("-v", "--verbose", dest="verbose", action="store_true", + help="print each depth level one by one") @property def max_level(self): diff --git a/scrapy/commands/settings.py b/scrapy/commands/settings.py index 8d49e440f..1b2e2601e 100644 --- a/scrapy/commands/settings.py +++ b/scrapy/commands/settings.py @@ -18,16 +18,16 @@ class Command(ScrapyCommand): def add_options(self, parser): ScrapyCommand.add_options(self, parser) - parser.add_option("--get", dest="get", metavar="SETTING", - help="print raw setting value") - parser.add_option("--getbool", dest="getbool", metavar="SETTING", - help="print setting value, interpreted as a boolean") - parser.add_option("--getint", dest="getint", metavar="SETTING", - help="print setting value, interpreted as an integer") - parser.add_option("--getfloat", dest="getfloat", metavar="SETTING", - help="print setting value, interpreted as a float") - parser.add_option("--getlist", dest="getlist", metavar="SETTING", - help="print setting value, interpreted as a list") + parser.add_argument("--get", dest="get", metavar="SETTING", + help="print raw setting value") + parser.add_argument("--getbool", dest="getbool", metavar="SETTING", + help="print setting value, interpreted as a boolean") + parser.add_argument("--getint", dest="getint", metavar="SETTING", + help="print setting value, interpreted as an integer") + parser.add_argument("--getfloat", dest="getfloat", metavar="SETTING", + help="print setting value, interpreted as a float") + parser.add_argument("--getlist", dest="getlist", metavar="SETTING", + help="print setting value, interpreted as a list") def run(self, args, opts): settings = self.crawler_process.settings diff --git a/scrapy/commands/shell.py b/scrapy/commands/shell.py index de81986d8..f67a5886a 100644 --- a/scrapy/commands/shell.py +++ b/scrapy/commands/shell.py @@ -33,12 +33,12 @@ class Command(ScrapyCommand): def add_options(self, parser): ScrapyCommand.add_options(self, parser) - parser.add_option("-c", dest="code", - help="evaluate the code in the shell, print the result and exit") - parser.add_option("--spider", dest="spider", - help="use this spider") - parser.add_option("--no-redirect", dest="no_redirect", action="store_true", default=False, - help="do not handle HTTP 3xx status codes and print response as-is") + parser.add_argument("-c", dest="code", + help="evaluate the code in the shell, print the result and exit") + parser.add_argument("--spider", dest="spider", + help="use this spider") + parser.add_argument("--no-redirect", dest="no_redirect", action="store_true", default=False, + help="do not handle HTTP 3xx status codes and print response as-is") def update_vars(self, vars): """You can use this function to update the Scrapy objects that will be diff --git a/scrapy/commands/version.py b/scrapy/commands/version.py index 1237610cb..c6a3c273a 100644 --- a/scrapy/commands/version.py +++ b/scrapy/commands/version.py @@ -16,8 +16,8 @@ class Command(ScrapyCommand): def add_options(self, parser): ScrapyCommand.add_options(self, parser) - parser.add_option("--verbose", "-v", dest="verbose", action="store_true", - help="also display twisted/python/platform info (useful for bug reports)") + parser.add_argument("--verbose", "-v", dest="verbose", action="store_true", + help="also display twisted/python/platform info (useful for bug reports)") def run(self, args, opts): if opts.verbose: diff --git a/scrapy/commands/view.py b/scrapy/commands/view.py index c8f873334..b1f52abe2 100644 --- a/scrapy/commands/view.py +++ b/scrapy/commands/view.py @@ -1,3 +1,4 @@ +import argparse from scrapy.commands import fetch from scrapy.utils.response import open_in_browser @@ -12,7 +13,7 @@ class Command(fetch.Command): def add_options(self, parser): super().add_options(parser) - parser.remove_option("--headers") + parser.add_argument('--headers', help=argparse.SUPPRESS) def _print_response(self, response, opts): open_in_browser(response) diff --git a/tests/test_command_parse.py b/tests/test_command_parse.py index ed3848d88..f21ee971d 100644 --- a/tests/test_command_parse.py +++ b/tests/test_command_parse.py @@ -1,6 +1,9 @@ import os +import argparse from os.path import join, abspath, isfile, exists from twisted.internet import defer +from scrapy.commands import parse +from scrapy.settings import Settings from scrapy.utils.testsite import SiteTest from scrapy.utils.testproc import ProcessTest from scrapy.utils.python import to_unicode @@ -239,3 +242,19 @@ ITEM_PIPELINES = {{'{self.project_name}.pipelines.MyPipeline': 1}} content = '[\n{},\n{"foo": "bar"}\n]' with open(file_path, 'r') as f: self.assertEqual(f.read(), content) + + def test_parse_add_options(self): + command = parse.Command() + command.settings = Settings() + parser = argparse.ArgumentParser( + prog='scrapy', formatter_class=argparse.HelpFormatter, + conflict_handler='resolve', prefix_chars='-' + ) + command.add_options(parser) + namespace = parser.parse_args( + ['--verbose', '--nolinks', '-d', '2', '--spider', self.spider_name] + ) + self.assertTrue(namespace.nolinks) + self.assertEqual(namespace.depth, 2) + self.assertEqual(namespace.spider, self.spider_name) + self.assertTrue(namespace.verbose) diff --git a/tests/test_commands.py b/tests/test_commands.py index 7473b53df..7cd19b29a 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -1,6 +1,6 @@ import inspect import json -import optparse +import argparse import os import platform import re @@ -23,7 +23,7 @@ from twisted.python.versions import Version from twisted.trial import unittest import scrapy -from scrapy.commands import ScrapyCommand +from scrapy.commands import view, ScrapyCommand, ScrapyHelpFormatter from scrapy.commands.startproject import IGNORE from scrapy.settings import Settings from scrapy.utils.python import to_unicode @@ -37,19 +37,28 @@ class CommandSettings(unittest.TestCase): def setUp(self): self.command = ScrapyCommand() self.command.settings = Settings() - self.parser = optparse.OptionParser( - formatter=optparse.TitledHelpFormatter(), - conflict_handler='resolve', - ) + self.parser = argparse.ArgumentParser(formatter_class=ScrapyHelpFormatter, + conflict_handler='resolve') self.command.add_options(self.parser) def test_settings_json_string(self): feeds_json = '{"data.json": {"format": "json"}, "data.xml": {"format": "xml"}}' - opts, args = self.parser.parse_args(args=['-s', f'FEEDS={feeds_json}', 'spider.py']) + opts, args = self.parser.parse_known_args(args=['-s', f'FEEDS={feeds_json}', 'spider.py']) self.command.process_options(args, opts) self.assertIsInstance(self.command.settings['FEEDS'], scrapy.settings.BaseSettings) self.assertEqual(dict(self.command.settings['FEEDS']), json.loads(feeds_json)) + def test_help_formatter(self): + formatter = ScrapyHelpFormatter(prog='scrapy') + part_strings = ['usage: scrapy genspider [options] \n\n', + '\n', 'optional arguments:\n', '\n', 'Global Options:\n'] + self.assertEqual( + formatter._join_parts(part_strings), + ('Usage\n=====\n scrapy genspider [options] \n\n\n' + 'Optional Arguments\n==================\n\n' + 'Global Options\n--------------\n') + ) + class ProjectTest(unittest.TestCase): project_name = 'testproject' @@ -812,6 +821,21 @@ class BenchCommandTest(CommandTest): self.assertNotIn('Unhandled Error', log) +class ViewCommandTest(CommandTest): + + def test_methods(self): + command = view.Command() + command.settings = Settings() + parser = argparse.ArgumentParser(prog='scrapy', prefix_chars='-', + formatter_class=ScrapyHelpFormatter, + conflict_handler='resolve') + command.add_options(parser) + self.assertEqual(command.short_desc(), + "Open URL in browser, as seen by Scrapy") + self.assertIn("URL using the Scrapy downloader and show its", + command.long_desc()) + + class CrawlCommandTest(CommandTest): def crawl(self, code, args=()): From 5d7c0a5f861327c1a51ffcb3a39e283eb999fae7 Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Thu, 10 Feb 2022 14:50:12 +0500 Subject: [PATCH 086/126] Use toscrape.com instead of example.com in test_command_check. (#5407) --- tests/test_command_check.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_command_check.py b/tests/test_command_check.py index 34f5e59dd..c3d705194 100644 --- a/tests/test_command_check.py +++ b/tests/test_command_check.py @@ -19,11 +19,11 @@ import scrapy class CheckSpider(scrapy.Spider): name = '{self.spider_name}' - start_urls = ['http://example.com'] + start_urls = ['http://toscrape.com'] def parse(self, response, **cb_kwargs): \"\"\" - @url http://example.com + @url http://toscrape.com {contracts} \"\"\" {parse_def} From befb6df119058db8a6a340b8235ccb565a60f3ca Mon Sep 17 00:00:00 2001 From: Laerte Pereira Date: Fri, 11 Feb 2022 06:19:27 -0300 Subject: [PATCH 087/126] Remove Python 2 code from WrappedRequest --- scrapy/http/cookies.py | 6 +----- tests/test_http_cookies.py | 1 - 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/scrapy/http/cookies.py b/scrapy/http/cookies.py index bf4ae7b45..b43c383fe 100644 --- a/scrapy/http/cookies.py +++ b/scrapy/http/cookies.py @@ -142,10 +142,6 @@ class WrappedRequest: """ return self.request.meta.get('is_unverifiable', False) - def get_origin_req_host(self): - return urlparse_cached(self.request).hostname - - # python3 uses attributes instead of methods @property def full_url(self): return self.get_full_url() @@ -164,7 +160,7 @@ class WrappedRequest: @property def origin_req_host(self): - return self.get_origin_req_host() + return urlparse_cached(self.request).hostname def has_header(self, name): return name in self.request.headers diff --git a/tests/test_http_cookies.py b/tests/test_http_cookies.py index 540e27907..08420332c 100644 --- a/tests/test_http_cookies.py +++ b/tests/test_http_cookies.py @@ -34,7 +34,6 @@ class WrappedRequestTest(TestCase): self.assertTrue(self.wrapped.unverifiable) def test_get_origin_req_host(self): - self.assertEqual(self.wrapped.get_origin_req_host(), 'www.example.com') self.assertEqual(self.wrapped.origin_req_host, 'www.example.com') def test_has_header(self): From bbb693d046a1942965ea9579bf7bb8a20fc92ba3 Mon Sep 17 00:00:00 2001 From: Boris Zabolotskikh Date: Mon, 14 Feb 2022 12:07:45 +0300 Subject: [PATCH 088/126] Update downloader-middleware.rst Added a link to the method --- docs/topics/downloader-middleware.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index caf44a903..44201d0d5 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -89,7 +89,7 @@ object gives you access, for example, to the :ref:`settings `. methods of installed middleware is always called on every response. If it returns a :class:`~scrapy.Request` object, Scrapy will stop calling - process_request methods and reschedule the returned request. Once the newly returned + :meth:`process_request` methods and reschedule the returned request. Once the newly returned request is performed, the appropriate middleware chain will be called on the downloaded response. From 187b5c887602218dee2fb57ad1dba223c11ce84f Mon Sep 17 00:00:00 2001 From: Abhishek K M <67158080+Sync271@users.noreply.github.com> Date: Mon, 14 Feb 2022 23:46:53 +0530 Subject: [PATCH 089/126] Update the documentation link for robots.txt (#5415) --- docs/topics/downloader-middleware.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index caf44a903..f9208d550 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -1019,7 +1019,7 @@ Parsers vary in several aspects: (shorter) rule Performance comparison of different parsers is available at `the following link -`_. +`_. .. _protego-parser: From 3b42ccfebadd72d9b455f6526ab63835b72b1558 Mon Sep 17 00:00:00 2001 From: Gowtham Chowdary <42214663+GowthamChowdary@users.noreply.github.com> Date: Thu, 17 Feb 2022 02:03:56 +0530 Subject: [PATCH 090/126] Add a link to Discord (#5422) --- docs/index.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/index.rst b/docs/index.rst index 433798aa8..69becd4a8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -24,12 +24,14 @@ Having trouble? We'd like to help! * Search for questions on the archives of the `scrapy-users mailing list`_. * Ask a question in the `#scrapy IRC channel`_, * Report bugs with Scrapy in our `issue tracker`_. +* Join the Discord community `Scrapy Discord`_. .. _scrapy-users mailing list: https://groups.google.com/forum/#!forum/scrapy-users .. _Scrapy subreddit: https://www.reddit.com/r/scrapy/ .. _StackOverflow using the scrapy tag: https://stackoverflow.com/tags/scrapy .. _#scrapy IRC channel: irc://irc.freenode.net/scrapy .. _issue tracker: https://github.com/scrapy/scrapy/issues +.. _Scrapy Discord : https://discord.gg/mv3yErfpvq First steps From 08557e09db4bcb109eb78e9058622ab5cef77415 Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Wed, 23 Feb 2022 23:52:18 +0500 Subject: [PATCH 091/126] Pin old markupsafe when we pin old mitmproxy (#5427) --- tox.ini | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tox.ini b/tox.ini index 2031a2d92..fcd3563b2 100644 --- a/tox.ini +++ b/tox.ini @@ -17,6 +17,8 @@ deps = #mitmproxy >= 5.3.0; python_version >= '3.9' and implementation_name != 'pypy' mitmproxy >= 4.0.4; python_version >= '3.7' and python_version < '3.9' and implementation_name != 'pypy' mitmproxy >= 4.0.4, < 5; python_version >= '3.6' and python_version < '3.7' and platform_system != 'Windows' and implementation_name != 'pypy' + # newer markupsafe is incompatible with deps of old mitmproxy (which we get on Python 3.7 and lower) + markupsafe < 2.1.0; python_version >= '3.6' and python_version < '3.8' and implementation_name != 'pypy' # Extras botocore>=1.4.87 passenv = @@ -127,6 +129,9 @@ deps = robotexclusionrulesparser Pillow>=4.0.0 Twisted[http2]>=17.9.0 + # Twisted[http2] currently forces old mitmproxy because of h2 version restrictions in their deps, + # so we need to pin old markupsafe here too + markupsafe < 2.1.0 [testenv:asyncio] commands = From aa0306a167ef34b23cc2ec407a48359a4b5a8d0a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 12:16:37 +0100 Subject: [PATCH 092/126] Cover 2.6.0 in the release notes (#5399) --- docs/conf.py | 4 + docs/index.rst | 4 +- docs/news.rst | 379 +++++++++++++++++++++++++++++-- docs/topics/asyncio.rst | 33 ++- docs/topics/commands.rst | 10 +- docs/topics/feed-exports.rst | 10 +- docs/topics/media-pipeline.rst | 2 + docs/topics/request-response.rst | 4 + docs/topics/spiders.rst | 5 +- scrapy/utils/defer.py | 4 +- 10 files changed, 407 insertions(+), 48 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index d5e139e66..55aa72d5a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -303,10 +303,14 @@ intersphinx_mapping = { hoverxref_auto_ref = True hoverxref_role_types = { "class": "tooltip", + "command": "tooltip", "confval": "tooltip", "hoverxref": "tooltip", "mod": "tooltip", "ref": "tooltip", + "reqmeta": "tooltip", + "setting": "tooltip", + "signal": "tooltip", } hoverxref_roles = ['command', 'reqmeta', 'setting', 'signal'] diff --git a/docs/index.rst b/docs/index.rst index 69becd4a8..75e08f537 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -12,6 +12,8 @@ testing. .. _web crawling: https://en.wikipedia.org/wiki/Web_crawler .. _web scraping: https://en.wikipedia.org/wiki/Web_scraping +.. _getting-help: + Getting help ============ @@ -31,7 +33,7 @@ Having trouble? We'd like to help! .. _StackOverflow using the scrapy tag: https://stackoverflow.com/tags/scrapy .. _#scrapy IRC channel: irc://irc.freenode.net/scrapy .. _issue tracker: https://github.com/scrapy/scrapy/issues -.. _Scrapy Discord : https://discord.gg/mv3yErfpvq +.. _Scrapy Discord: https://discord.gg/mv3yErfpvq First steps diff --git a/docs/news.rst b/docs/news.rst index 47a808693..2128f2f0e 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1,30 +1,360 @@ -.. note:: - .. versionchanged:: VERSION - - The Twisted reactor is now installed when - :meth:`~scrapy.crawler.CrawlerProcess.crawl` is first called, not when a - :class:`scrapy.crawler.CrawlerProcess` object is created. Because of this, - :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` are now - honored in :attr:`~scrapy.Spider.custom_settings`. In older Scrapy versions - they are silently ignored when set there and you need to set these settings - in some other way. - - -.. note:: - .. versionchanged:: VERSION - - Previously this setting had no effect in a spider - :attr:`~scrapy.Spider.custom_settings` attribute. Now it will be used, but - if you :ref:`run several spiders in one process `, - they must not have different values for this setting, because they will use - a single reactor instance. - - .. _news: Release notes ============= +.. _release-2.6.0: + +Scrapy 2.6.0 (2022-02-??) +------------------------- + +Highlights: + +* Python 3.10 support + +* :ref:`asyncio support ` is no longer considered + experimental, and works out-of-the-box on Windows regardless of your Python + version + +* Feed exports now support :class:`pathlib.Path` output paths and per-feed + :ref:`item filtering ` and + :ref:`post-processing ` + +Modified requirements +~~~~~~~~~~~~~~~~~~~~~ + +- The h2_ dependency is now optional, only needed to + :ref:`enable HTTP/2 support `. (:issue:`5113`) + + .. _h2: https://pypi.org/project/h2/ + + +Backward-incompatible changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- The ``formdata`` parameter of :class:`~scrapy.FormRequest`, if specified + for a non-POST request, now overrides the URL query string, instead of + being appended to it. (:issue:`2919`, :issue:`3579`) + +- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting, now + the return value of that function, and not the ``params`` input parameter, + will determine the feed URI parameters, unless that return value is + ``None``. (:issue:`4962`, :issue:`4966`) + +- In :class:`scrapy.core.engine.ExecutionEngine`, methods + :meth:`~scrapy.core.engine.ExecutionEngine.crawl`, + :meth:`~scrapy.core.engine.ExecutionEngine.download`, + :meth:`~scrapy.core.engine.ExecutionEngine.schedule`, + and :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle` + now raise :exc:`RuntimeError` if called before + :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`. (:issue:`5090`) + + These methods used to assume that + :attr:`ExecutionEngine.slot ` had + been defined by a prior call to + :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`, so they were + raising :exc:`AttributeError` instead. + +- If the API of the configured :ref:`scheduler ` does not + meet expectations, :exc:`TypeError` is now raised at startup time. Before, + other exceptions would be raised at run time. (:issue:`3559`) + + +Deprecation removals +~~~~~~~~~~~~~~~~~~~~ + +- ``scrapy.http.TextResponse.body_as_unicode``, deprecated in Scrapy 2.2, has + now been removed. (:issue:`5393`) + +- ``scrapy.item.BaseItem``, deprecated in Scrapy 2.2, has now been removed. + (:issue:`5398`) + +- ``scrapy.item.DictItem``, deprecated in Scrapy 1.8, has now been removed. + (:issue:`5398`) + +- ``scrapy.Spider.make_requests_from_url``, deprecated in Scrapy 1.4, has now + been removed. (:issue:`4178`, :issue:`4356`) + + +Deprecations +~~~~~~~~~~~~ + +- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting, + returning ``None`` or modifying the ``params`` input parameter is now + deprecated. Return a new dictionary instead. (:issue:`4962`, :issue:`4966`) + +- :mod:`scrapy.utils.reqser` is deprecated. (:issue:`5130`) + + - Instead of :func:`~scrapy.utils.reqser.request_to_dict`, use the new + :meth:`Request.to_dict ` method. + + - Instead of :func:`~scrapy.utils.reqser.request_from_dict`, use the new + :func:`scrapy.utils.request.request_from_dict` function. + +- In :mod:`scrapy.squeues`, the following queue classes are deprecated: + :class:`~scrapy.squeues.PickleFifoDiskQueueNonRequest`, + :class:`~scrapy.squeues.PickleLifoDiskQueueNonRequest`, + :class:`~scrapy.squeues.MarshalFifoDiskQueueNonRequest`, + and :class:`~scrapy.squeues.MarshalLifoDiskQueueNonRequest`. You should + instead use: + :class:`~scrapy.squeues.PickleFifoDiskQueue`, + :class:`~scrapy.squeues.PickleLifoDiskQueue`, + :class:`~scrapy.squeues.MarshalFifoDiskQueue`, + and :class:`~scrapy.squeues.MarshalLifoDiskQueue`. (:issue:`5117`) + +- Many aspects of :class:`scrapy.core.engine.ExecutionEngine` that come from + a time when this class could handle multiple :class:`~scrapy.Spider` + objects at a time have been deprecated. (:issue:`5090`) + + - The :meth:`~scrapy.core.engine.ExecutionEngine.has_capacity` method + is deprecated. + + - The :meth:`~scrapy.core.engine.ExecutionEngine.schedule` method is + deprecated, use :meth:`~scrapy.core.engine.ExecutionEngine.crawl` or + :meth:`~scrapy.core.engine.ExecutionEngine.download` instead. + + - The :attr:`~scrapy.core.engine.ExecutionEngine.open_spiders` attribute + is deprecated, use :attr:`~scrapy.core.engine.ExecutionEngine.spider` + instead. + + - The ``spider`` parameter is deprecated for the following methods: + + - :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle` + + - :meth:`~scrapy.core.engine.ExecutionEngine.crawl` + + - :meth:`~scrapy.core.engine.ExecutionEngine.download` + + Instead, call :meth:`~scrapy.core.engine.ExecutionEngine.open_spider` + first to set the :class:`~scrapy.Spider` object. + + +New features +~~~~~~~~~~~~ + +- You can now use :ref:`item filtering ` to control which items + are exported to each output feed. (:issue:`4575`, :issue:`5178`, + :issue:`5161`, :issue:`5203`) + +- You can now apply :ref:`post-processing ` to feeds, and + :ref:`built-in post-processing plugins ` are provided for + output file compression. (:issue:`2174`, :issue:`5168`, :issue:`5190`) + +- The :setting:`FEEDS` setting now supports :class:`pathlib.Path` objects as + keys. (:issue:`5383`, :issue:`5384`) + +- Enabling :ref:`asyncio ` while using Windows and Python 3.8 + or later will automatically switch the asyncio event loop to one that + allows Scrapy to work. See :ref:`asyncio-windows`. (:issue:`4976`, + :issue:`5315`) + +- The :command:`genspider` command now supports a start URL instead of a + domain name. (:issue:`4439`) + +- :mod:`scrapy.utils.defer` gained 2 new functions, + :func:`~scrapy.utils.defer.deferred_to_future` and + :func:`~scrapy.utils.defer.maybe_deferred_to_future`, to help :ref:`await + on Deferreds when using the asyncio reactor `. + (:issue:`5288`) + +- :ref:`Amazon S3 feed export storage ` gained + support for `temporary security credentials`_ + (:setting:`AWS_SESSION_TOKEN`) and endpoint customization + (:setting:`AWS_ENDPOINT_URL`). (:issue:`4998`, :issue:`5210`) + + .. _temporary security credentials: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys + +- New :setting:`LOG_FILE_APPEND` setting to allow truncating the log file. + (:issue:`5279`) + +- :attr:`Request.cookies ` values that are + :class:`bool`, :class:`float` or :class:`int` are cast to :class:`str`. + (:issue:`5252`, :issue:`5253`) + +- You may now raise :exc:`~scrapy.exceptions.CloseSpider` from a handler of + the :signal:`spider_idle` signal to customize the reason why the spider is + stopping. (:issue:`5191`) + +- When using + :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`, the + proxy URL for non-HTTPS HTTP/1.1 requests no longer needs to include a URL + scheme. (:issue:`4505`, :issue:`4649`) + +- All built-in queues now expose a ``peek`` method that returns the next + queue object (like ``pop``) but does not remove the returned object from + the queue. (:issue:`5112`) + + If the underlying queue does not support peeking (e.g. because you are not + using ``queuelib`` 1.6.1 or later), the ``peek`` method raises + :exc:`NotImplementedError`. + +- :class:`~scrapy.http.Request` and :class:`~scrapy.http.Response` now have + an ``attributes`` attribute that makes subclassing easier. For + :class:`~scrapy.http.Request`, it also allows subclasses to work with + :func:`scrapy.utils.request.request_from_dict`. (:issue:`1877`, + :issue:`5130`, :issue:`5218`) + +- The :meth:`~scrapy.core.scheduler.BaseScheduler.open` and + :meth:`~scrapy.core.scheduler.BaseScheduler.close` methods of the + :ref:`scheduler ` are now optional. (:issue:`3559`) + +- HTTP/1.1 :exc:`~scrapy.core.downloader.handlers.http11.TunnelError` + exceptions now only truncate response bodies longer than 1000 characters, + instead of those longer than 32 characters, making it easier to debug such + errors. (:issue:`4881`, :issue:`5007`) + +- :class:`~scrapy.loader.ItemLoader` now supports non-text responses. + (:issue:`5145`, :issue:`5269`) + + +Bug fixes +~~~~~~~~~ + +- The :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` settings + are no longer ignored if defined in :attr:`~scrapy.Spider.custom_settings`. + (:issue:`4485`, :issue:`5352`) + +- Removed a module-level Twisted reactor import that could prevent + :ref:`using the asyncio reactor `. (:issue:`5357`) + +- The :command:`startproject` command works with existing folders again. + (:issue:`4665`, :issue:`4676`) + +- The :setting:`FEED_URI_PARAMS` setting now behaves as documented. + (:issue:`4962`, :issue:`4966`) + +- :attr:`Request.cb_kwargs ` once again allows the + ``callback`` keyword. (:issue:`5237`, :issue:`5251`, :issue:`5264`) + +- Made :func:`scrapy.utils.response.open_in_browser` support more complex + HTML. (:issue:`5319`, :issue:`5320`) + +- Fixed :attr:`CSVFeedSpider.quotechar + ` being interpreted as the CSV file + encoding. (:issue:`5391`, :issue:`5394`) + +- Added missing setuptools_ to the list of dependencies. (:issue:`5122`) + + .. _setuptools: https://pypi.org/project/setuptools/ + +- :class:`LinkExtractor ` + now also works as expected with links that have comma-separated ``rel`` + attribute values including ``nofollow``. (:issue:`5225`) + +- Fixed a :exc:`TypeError` that could be raised during :ref:`feed export + ` parameter parsing. (:issue:`5359`) + + +Documentation +~~~~~~~~~~~~~ + +- :ref:`asyncio support ` is no longer considered + experimental. (:issue:`5332`) + +- Included :ref:`Windows-specific help for asyncio usage `. + (:issue:`4976`, :issue:`5315`) + +- Rewrote :ref:`topics-headless-browsing` with up-to-date best practices. + (:issue:`4484`, :issue:`4613`) + +- Documented :ref:`local file naming in media pipelines + `. (:issue:`5069`, :issue:`5152`) + +- :ref:`faq` now covers spider file name collision issues. (:issue:`2680`, + :issue:`3669`) + +- Provided better context and instructions to disable the + :setting:`URLLENGTH_LIMIT` setting. (:issue:`5135`, :issue:`5250`) + +- Documented that :ref:`reppy-parser` does not support Python 3.9+. + (:issue:`5226`, :issue:`5231`) + +- Documented :ref:`the scheduler component `. + (:issue:`3537`, :issue:`3559`) + +- Documented the method used by :ref:`media pipelines + ` to :ref:`determine if a file has expired + `. (:issue:`5120`, :issue:`5254`) + +- :ref:`run-multiple-spiders` now features + :func:`scrapy.utils.project.get_project_settings` usage. (:issue:`5070`) + +- :ref:`run-multiple-spiders` now covers what happens when you define + different per-spider values for some settings that cannot differ at run + time. (:issue:`4485`, :issue:`5352`) + +- Extended the documentation of the + :class:`~scrapy.extensions.statsmailer.StatsMailer` extension. + (:issue:`5199`, :issue:`5217`) + +- Added :setting:`JOBDIR` to :ref:`topics-settings`. (:issue:`5173`, + :issue:`5224`) + +- Documented :attr:`Spider.attribute `. + (:issue:`5174`, :issue:`5244`) + +- Documented :attr:`TextResponse.urljoin `. + (:issue:`1582`) + +- Added the ``body_length`` parameter to the documented signature of the + :signal:`headers_received` signal. (:issue:`5270`) + +- Clarified :meth:`SelectorList.get ` usage + in the :ref:`tutorial `. (:issue:`5256`) + +- The documentation now features the shortest import path of classes with + multiple import paths. (:issue:`2733`, :issue:`5099`) + +- ``quotes.toscrape.com`` references now use HTTPS instead of HTTP. + (:issue:`5395`, :issue:`5396`) + +- Added a link to `our Discord server `_ + to :ref:`getting-help`. (:issue:`5421`, :issue:`5422`) + +- The pronunciation of the project name is now :ref:`officially + ` /ˈskreɪpaɪ/. (:issue:`5280`, :issue:`5281`) + +- Added the Scrapy logo to the README. (:issue:`5255`, :issue:`5258`) + +- Fixed issues and implemented minor improvements. (:issue:`3155`, + :issue:`4335`, :issue:`5074`, :issue:`5098`, :issue:`5134`, :issue:`5180`, + :issue:`5194`, :issue:`5239`, :issue:`5266`, :issue:`5271`, :issue:`5273`, + :issue:`5274`, :issue:`5276`, :issue:`5347`, :issue:`5356`, :issue:`5414`, + :issue:`5415`, :issue:`5416`, :issue:`5419`, :issue:`5420`) + + +Quality Assurance +~~~~~~~~~~~~~~~~~ + +- Added support for Python 3.10. (:issue:`5212`, :issue:`5221`, + :issue:`5265`) + +- Significantly reduced memory usage by + :func:`scrapy.utils.response.response_httprepr`, used by the + :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats` downloader + middleware, which is enabled by default. (:issue:`4964`, :issue:`4972`) + +- Removed uses of the deprecated :mod:`optparse` module. (:issue:`5366`, + :issue:`5374`) + +- Extended typing hints. (:issue:`5077`, :issue:`5090`, :issue:`5100`, + :issue:`5108`, :issue:`5171`, :issue:`5215`, :issue:`5334`) + +- Improved tests, fixed CI issues, removed unused code. (:issue:`5094`, + :issue:`5157`, :issue:`5162`, :issue:`5198`, :issue:`5207`, :issue:`5208`, + :issue:`5229`, :issue:`5298`, :issue:`5299`, :issue:`5310`, :issue:`5316`, + :issue:`5333`, :issue:`5388`, :issue:`5389`, :issue:`5400`, :issue:`5401`, + :issue:`5404`, :issue:`5405`, :issue:`5407`, :issue:`5410`, :issue:`5412`, + :issue:`5425`, :issue:`5427`) + +- Implemented improvements for contributors. (:issue:`5080`, :issue:`5082`, + :issue:`5177`, :issue:`5200`) + +- Implemented cleanups. (:issue:`5095`, :issue:`5106`, :issue:`5209`, + :issue:`5228`, :issue:`5235`, :issue:`5245`, :issue:`5246`, :issue:`5292`, + :issue:`5314`, :issue:`5322`) + + .. _release-2.5.1: Scrapy 2.5.1 (2021-10-05) @@ -1000,9 +1330,8 @@ Bug fixes * zope.interface 5.0.0 and later versions are now supported (:issue:`4447`, :issue:`4448`) -* :meth:`Spider.make_requests_from_url - `, deprecated in Scrapy - 1.4.0, now issues a warning when used (:issue:`4412`) +* ``Spider.make_requests_from_url``, deprecated in Scrapy 1.4.0, now issues a + warning when used (:issue:`4412`) Documentation diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 8712d4268..3a6941a2c 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -10,6 +10,7 @@ Scrapy has partial support for :mod:`asyncio`. After you :ref:`install the asyncio reactor `, you may use :mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine `. + .. _install-asyncio: Installing the asyncio reactor @@ -25,6 +26,7 @@ reactor manually. You can do that using install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor') + .. _using-custom-loops: Using custom asyncio loops @@ -34,20 +36,30 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the :setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class to use it instead of the default asyncio event loop. -.. _asyncio-await-dfd: + +.. _asyncio-windows: Windows-specific notes ====================== The Windows implementation of :mod:`asyncio` can use two event loop -implementations: :class:`~asyncio.SelectorEventLoop` (default before Python -3.8, required when using Twisted) and :class:`~asyncio.ProactorEventLoop` -(default since Python 3.8, cannot work with Twisted). So on Python 3.8+ the -event loop class needs to be changed. Scrapy since VERSION does this -automatically when you change the :setting:`TWISTED_REACTOR` setting or call -:func:`~scrapy.utils.reactor.install_reactor`, but if you install the reactor -by other means or use an older Scrapy version you need to call the following -code before installing the reactor:: +implementations: + +- :class:`~asyncio.SelectorEventLoop`, default before Python 3.8, required + when using Twisted. + +- :class:`~asyncio.ProactorEventLoop`, default since Python 3.8, cannot work + with Twisted. + +So on Python 3.8+ the event loop class needs to be changed. + +.. versionchanged:: 2.6.0 + The event loop class is changed automatically when you change the + :setting:`TWISTED_REACTOR` setting or call + :func:`~scrapy.utils.reactor.install_reactor`. + +To change the event loop class manually, call the following code before +installing the reactor:: import asyncio asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) @@ -64,6 +76,9 @@ yourself, or in some code that runs before the reactor is installed, e.g. .. _playwright: https://github.com/microsoft/playwright-python + +.. _asyncio-await-dfd: + Awaiting on Deferreds ===================== diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index eef6b36ff..8c0b8e55f 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -230,10 +230,16 @@ Usage example:: genspider --------- -* Syntax: ``scrapy genspider [-t template] `` +* Syntax: ``scrapy genspider [-t template] `` * Requires project: *no* -Create a new spider in the current folder or in the current project's ``spiders`` folder, if called from inside a project. The ```` parameter is set as the spider's ``name``, while ```` is used to generate the ``allowed_domains`` and ``start_urls`` spider's attributes. +.. versionadded:: 2.6.0 + The ability to pass a URL instead of a domain. + +Create a new spider in the current folder or in the current project's ``spiders`` folder, if called from inside a project. The ```` parameter is set as the spider's ``name``, while ```` is used to generate the ``allowed_domains`` and ``start_urls`` spider's attributes. + +.. note:: Even if an HTTPS URL is specified, the protocol used in + ``start_urls`` is always HTTP. This is a known issue: :issue:`3553`. Usage example:: diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index 7994027d2..9a13eb82f 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -278,7 +278,7 @@ feed URI, allowing item delivery to start way before the end of the crawl. Item filtering ============== -.. versionadded:: VERSION +.. versionadded:: 2.6.0 You can filter items that you want to allow for a particular feed by using the ``item_classes`` option in :ref:`feeds options `. Only items of @@ -318,7 +318,7 @@ ItemFilter Post-Processing =============== -.. versionadded:: VERSION +.. versionadded:: 2.6.0 Scrapy provides an option to activate plugins to post-process feeds before they are exported to feed storages. In addition to using :ref:`builtin plugins `, you @@ -457,13 +457,13 @@ as a fallback value if that key is not provided for a specific feed definition: If undefined or empty, all items are exported. - .. versionadded:: VERSION + .. versionadded:: 2.6.0 - ``item_filter``: a :ref:`filter class ` to filter items to export. :class:`~scrapy.extensions.feedexport.ItemFilter` is used be default. - .. versionadded:: VERSION + .. versionadded:: 2.6.0 - ``indent``: falls back to :setting:`FEED_EXPORT_INDENT`. @@ -499,7 +499,7 @@ as a fallback value if that key is not provided for a specific feed definition: The plugins will be used in the order of the list passed. - .. versionadded:: VERSION + .. versionadded:: 2.6.0 .. setting:: FEED_EXPORT_ENCODING diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index 10d2ac990..7dff78390 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -356,6 +356,8 @@ setting MYPIPELINE_IMAGES_URLS_FIELD and your custom settings will be used. Additional features =================== +.. _file-expiration: + File expiration --------------- diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index e0435e901..92a471faf 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -110,6 +110,10 @@ Request objects :class:`Request.cookies ` parameter. This is a known current limitation that is being worked on. + .. versionadded:: 2.6.0 + Cookie values that are :class:`bool`, :class:`float` or :class:`int` + are casted to :class:`str`. + :type cookies: dict or list :param encoding: the encoding of this request (defaults to ``'utf-8'``). diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 99e74233a..ece02ae47 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -42,15 +42,12 @@ Even though this cycle applies (more or less) to any kind of spider, there are different kinds of default spiders bundled into Scrapy for different purposes. We will talk about those types here. -.. module:: scrapy.spiders - :synopsis: Spiders base class, spider manager and spider middleware - .. _topics-spiders-ref: scrapy.Spider ============= -.. class:: scrapy.spiders.Spider() +.. class:: scrapy.spiders.Spider .. class:: scrapy.Spider() This is the simplest spider, and the one from which every other spider diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.py index d7adc0a77..7ecb8ea3f 100644 --- a/scrapy/utils/defer.py +++ b/scrapy/utils/defer.py @@ -182,7 +182,7 @@ def maybeDeferred_coro(f: Callable, *args, **kw) -> Deferred: def deferred_to_future(d: Deferred) -> Future: """ - .. versionadded:: VERSION + .. versionadded:: 2.6.0 Return an :class:`asyncio.Future` object that wraps *d*. @@ -203,7 +203,7 @@ def deferred_to_future(d: Deferred) -> Future: def maybe_deferred_to_future(d: Deferred) -> Union[Deferred, Future]: """ - .. versionadded:: VERSION + .. versionadded:: 2.6.0 Return *d* as an object that can be awaited from a :ref:`Scrapy callable defined as a coroutine `. From 8ce01b3b76d4634f55067d6cfdf632ec70ba304a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 12:26:05 +0100 Subject: [PATCH 093/126] Merge pull request from GHSA-cjvr-mfj7-j4j8 * Do not carry over cookies to a different domain on redirect * Cover the cookie-domain redirect fix in the release notes * Cover 1.8.2 in the release notes * Fix redirect Cookie handling when the cookie middleware is disabled * Update the 1.8.2 release date --- docs/news.rst | 67 ++++++++- scrapy/downloadermiddlewares/redirect.py | 31 ++++- tests/test_downloadermiddleware_cookies.py | 155 +++++++++++++++++++++ 3 files changed, 247 insertions(+), 6 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 2128f2f0e..aef12d9db 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -5,11 +5,13 @@ Release notes .. _release-2.6.0: -Scrapy 2.6.0 (2022-02-??) +Scrapy 2.6.0 (2022-03-01) ------------------------- Highlights: +* :ref:`Security fixes for cookie handling <2.6-security-fixes>` + * Python 3.10 support * :ref:`asyncio support ` is no longer considered @@ -20,6 +22,37 @@ Highlights: :ref:`item filtering ` and :ref:`post-processing ` +.. _2.6-security-fixes: + +Security bug fixes +~~~~~~~~~~~~~~~~~~ + +- When a :class:`~scrapy.http.Request` object with cookies defined gets a + redirect response causing a new :class:`~scrapy.http.Request` object to be + scheduled, the cookies defined in the original + :class:`~scrapy.http.Request` object are no longer copied into the new + :class:`~scrapy.http.Request` object. + + If you manually set the ``Cookie`` header on a + :class:`~scrapy.http.Request` object and the domain name of the redirect + URL is not an exact match for the domain of the URL of the original + :class:`~scrapy.http.Request` object, your ``Cookie`` header is now dropped + from the new :class:`~scrapy.http.Request` object. + + The old behavior could be exploited by an attacker to gain access to your + cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more + information. + + .. _cjvr-mfj7-j4j8 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cjvr-mfj7-j4j8 + + .. note:: It is still possible to enable the sharing of cookies between + different domains with a shared domain suffix (e.g. + ``example.com`` and any subdomain) by defining the shared domain + suffix (e.g. ``example.com``) as the cookie domain when defining + your cookies. See the documentation of the + :class:`~scrapy.http.Request` class for more information. + + Modified requirements ~~~~~~~~~~~~~~~~~~~~~ @@ -1842,6 +1875,38 @@ affect subclasses: (:issue:`3884`) +.. _release-1.8.2: + +Scrapy 1.8.2 (2022-03-01) +------------------------- + +**Security bug fixes:** + +- When a :class:`~scrapy.http.Request` object with cookies defined gets a + redirect response causing a new :class:`~scrapy.http.Request` object to be + scheduled, the cookies defined in the original + :class:`~scrapy.http.Request` object are no longer copied into the new + :class:`~scrapy.http.Request` object. + + If you manually set the ``Cookie`` header on a + :class:`~scrapy.http.Request` object and the domain name of the redirect + URL is not an exact match for the domain of the URL of the original + :class:`~scrapy.http.Request` object, your ``Cookie`` header is now dropped + from the new :class:`~scrapy.http.Request` object. + + The old behavior could be exploited by an attacker to gain access to your + cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more + information. + + .. _cjvr-mfj7-j4j8 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cjvr-mfj7-j4j8 + + .. note:: It is still possible to enable the sharing of cookies between + different domains with a shared domain suffix (e.g. + ``example.com`` and any subdomain) by defining the shared domain + suffix (e.g. ``example.com``) as the cookie domain when defining + your cookies. See the documentation of the + :class:`~scrapy.http.Request` class for more information. + .. _release-1.8.1: diff --git a/scrapy/downloadermiddlewares/redirect.py b/scrapy/downloadermiddlewares/redirect.py index 4053fecc5..fcd6c298b 100644 --- a/scrapy/downloadermiddlewares/redirect.py +++ b/scrapy/downloadermiddlewares/redirect.py @@ -4,6 +4,7 @@ from urllib.parse import urljoin, urlparse from w3lib.url import safe_url_string from scrapy.http import HtmlResponse +from scrapy.utils.httpobj import urlparse_cached from scrapy.utils.response import get_meta_refresh from scrapy.exceptions import IgnoreRequest, NotConfigured @@ -11,6 +12,21 @@ from scrapy.exceptions import IgnoreRequest, NotConfigured logger = logging.getLogger(__name__) +def _build_redirect_request(source_request, *, url, method=None, body=None): + redirect_request = source_request.replace( + url=url, + method=method, + body=body, + cookies=None, + ) + if 'Cookie' in redirect_request.headers: + source_request_netloc = urlparse_cached(source_request).netloc + redirect_request_netloc = urlparse_cached(redirect_request).netloc + if source_request_netloc != redirect_request_netloc: + del redirect_request.headers['Cookie'] + return redirect_request + + class BaseRedirectMiddleware: enabled_setting = 'REDIRECT_ENABLED' @@ -47,10 +63,15 @@ class BaseRedirectMiddleware: raise IgnoreRequest("max redirections reached") def _redirect_request_using_get(self, request, redirect_url): - redirected = request.replace(url=redirect_url, method='GET', body='') - redirected.headers.pop('Content-Type', None) - redirected.headers.pop('Content-Length', None) - return redirected + redirect_request = _build_redirect_request( + request, + url=redirect_url, + method='GET', + body='', + ) + redirect_request.headers.pop('Content-Type', None) + redirect_request.headers.pop('Content-Length', None) + return redirect_request class RedirectMiddleware(BaseRedirectMiddleware): @@ -80,7 +101,7 @@ class RedirectMiddleware(BaseRedirectMiddleware): redirected_url = urljoin(request.url, location) if response.status in (301, 307, 308) or request.method == 'HEAD': - redirected = request.replace(url=redirected_url) + redirected = _build_redirect_request(request, url=redirected_url) return self._redirect(redirected, request, spider, response.status) redirected = self._redirect_request_using_get(request, redirected_url) diff --git a/tests/test_downloadermiddleware_cookies.py b/tests/test_downloadermiddleware_cookies.py index 36021bfbf..1747f3b94 100644 --- a/tests/test_downloadermiddleware_cookies.py +++ b/tests/test_downloadermiddleware_cookies.py @@ -6,8 +6,10 @@ import pytest from scrapy.downloadermiddlewares.cookies import CookiesMiddleware from scrapy.downloadermiddlewares.defaultheaders import DefaultHeadersMiddleware +from scrapy.downloadermiddlewares.redirect import RedirectMiddleware from scrapy.exceptions import NotConfigured from scrapy.http import Response, Request +from scrapy.settings import Settings from scrapy.spiders import Spider from scrapy.utils.python import to_bytes from scrapy.utils.test import get_crawler @@ -23,9 +25,11 @@ class CookiesMiddlewareTest(TestCase): def setUp(self): self.spider = Spider('foo') self.mw = CookiesMiddleware() + self.redirect_middleware = RedirectMiddleware(settings=Settings()) def tearDown(self): del self.mw + del self.redirect_middleware def test_basic(self): req = Request('http://scrapytest.org/') @@ -368,3 +372,154 @@ class CookiesMiddlewareTest(TestCase): req4 = Request('http://example.org', cookies={'a': 'b'}) assert self.mw.process_request(req4, self.spider) is None self.assertCookieValEqual(req4.headers['Cookie'], b'a=b') + + def _test_cookie_redirect( + self, + source, + target, + *, + cookies1, + cookies2, + ): + input_cookies = {'a': 'b'} + + if not isinstance(source, dict): + source = {'url': source} + if not isinstance(target, dict): + target = {'url': target} + target.setdefault('status', 301) + + request1 = Request(cookies=input_cookies, **source) + self.mw.process_request(request1, self.spider) + cookies = request1.headers.get('Cookie') + self.assertEqual(cookies, b"a=b" if cookies1 else None) + + response = Response( + headers={ + 'Location': target['url'], + }, + **target, + ) + self.assertEqual( + self.mw.process_response(request1, response, self.spider), + response, + ) + + request2 = self.redirect_middleware.process_response( + request1, + response, + self.spider, + ) + self.assertIsInstance(request2, Request) + + self.mw.process_request(request2, self.spider) + cookies = request2.headers.get('Cookie') + self.assertEqual(cookies, b"a=b" if cookies2 else None) + + def test_cookie_redirect_same_domain(self): + self._test_cookie_redirect( + 'https://toscrape.com', + 'https://toscrape.com', + cookies1=True, + cookies2=True, + ) + + def test_cookie_redirect_same_domain_forcing_get(self): + self._test_cookie_redirect( + 'https://toscrape.com', + {'url': 'https://toscrape.com', 'status': 302}, + cookies1=True, + cookies2=True, + ) + + def test_cookie_redirect_different_domain(self): + self._test_cookie_redirect( + 'https://toscrape.com', + 'https://example.com', + cookies1=True, + cookies2=False, + ) + + def test_cookie_redirect_different_domain_forcing_get(self): + self._test_cookie_redirect( + 'https://toscrape.com', + {'url': 'https://example.com', 'status': 302}, + cookies1=True, + cookies2=False, + ) + + def _test_cookie_header_redirect( + self, + source, + target, + *, + cookies2, + ): + """Test the handling of a user-defined Cookie header when building a + redirect follow-up request. + + We follow RFC 6265 for cookie handling. The Cookie header can only + contain a list of key-value pairs (i.e. no additional cookie + parameters like Domain or Path). Because of that, we follow the same + rules that we would follow for the handling of the Set-Cookie response + header when the Domain is not set: the cookies must be limited to the + target URL domain (not even subdomains can receive those cookies). + + .. note:: This method tests the scenario where the cookie middleware is + disabled. Because of known issue #1992, when the cookies + middleware is enabled we do not need to be concerned about + the Cookie header getting leaked to unintended domains, + because the middleware empties the header from every request. + """ + if not isinstance(source, dict): + source = {'url': source} + if not isinstance(target, dict): + target = {'url': target} + target.setdefault('status', 301) + + request1 = Request(headers={'Cookie': b'a=b'}, **source) + + response = Response( + headers={ + 'Location': target['url'], + }, + **target, + ) + + request2 = self.redirect_middleware.process_response( + request1, + response, + self.spider, + ) + self.assertIsInstance(request2, Request) + + cookies = request2.headers.get('Cookie') + self.assertEqual(cookies, b"a=b" if cookies2 else None) + + def test_cookie_header_redirect_same_domain(self): + self._test_cookie_header_redirect( + 'https://toscrape.com', + 'https://toscrape.com', + cookies2=True, + ) + + def test_cookie_header_redirect_same_domain_forcing_get(self): + self._test_cookie_header_redirect( + 'https://toscrape.com', + {'url': 'https://toscrape.com', 'status': 302}, + cookies2=True, + ) + + def test_cookie_header_redirect_different_domain(self): + self._test_cookie_header_redirect( + 'https://toscrape.com', + 'https://example.com', + cookies2=False, + ) + + def test_cookie_header_redirect_different_domain_forcing_get(self): + self._test_cookie_header_redirect( + 'https://toscrape.com', + {'url': 'https://example.com', 'status': 302}, + cookies2=False, + ) From e865c4430e58a4faa0e0766b23830f8423d6167a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 12:38:19 +0100 Subject: [PATCH 094/126] Merge pull request from GHSA-mfjm-vh54-3f96 * Ignore cookies with a public suffix as domain unless it matches the request domain * Fix the merge of 1.8.2 release notes * Re-apply removal of tldextract restriction --- docs/news.rst | 24 +++ scrapy/downloadermiddlewares/cookies.py | 34 ++++- setup.py | 1 + tests/test_downloadermiddleware_cookies.py | 170 +++++++++++++++++++++ 4 files changed, 226 insertions(+), 3 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index aef12d9db..9590fb1c4 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -52,6 +52,18 @@ Security bug fixes your cookies. See the documentation of the :class:`~scrapy.http.Request` class for more information. +- When the domain of a cookie, either received in the ``Set-Cookie`` header + of a response or defined in a :class:`~scrapy.http.Request` object, is set + to a `public suffix `_, the cookie is now + ignored unless the cookie domain is the same as the request domain. + + The old behavior could be exploited by an attacker to inject cookies from a + controlled domain into your cookiejar that could be sent to other domains + not controlled by the attacker. Please, see the `mfjm-vh54-3f96 security + advisory`_ for more information. + + .. _mfjm-vh54-3f96 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-mfjm-vh54-3f96 + Modified requirements ~~~~~~~~~~~~~~~~~~~~~ @@ -1875,6 +1887,7 @@ affect subclasses: (:issue:`3884`) + .. _release-1.8.2: Scrapy 1.8.2 (2022-03-01) @@ -1907,6 +1920,17 @@ Scrapy 1.8.2 (2022-03-01) your cookies. See the documentation of the :class:`~scrapy.http.Request` class for more information. +- When the domain of a cookie, either received in the ``Set-Cookie`` header + of a response or defined in a :class:`~scrapy.http.Request` object, is set + to a `public suffix `_, the cookie is now + ignored unless the cookie domain is the same as the request domain. + + The old behavior could be exploited by an attacker to inject cookies into + your requests to some other domains. Please, see the `mfjm-vh54-3f96 + security advisory`_ for more information. + + .. _mfjm-vh54-3f96 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-mfjm-vh54-3f96 + .. _release-1.8.1: diff --git a/scrapy/downloadermiddlewares/cookies.py b/scrapy/downloadermiddlewares/cookies.py index 0eee8d758..3afa06077 100644 --- a/scrapy/downloadermiddlewares/cookies.py +++ b/scrapy/downloadermiddlewares/cookies.py @@ -1,15 +1,26 @@ import logging from collections import defaultdict +from tldextract import TLDExtract + from scrapy.exceptions import NotConfigured from scrapy.http import Response from scrapy.http.cookies import CookieJar +from scrapy.utils.httpobj import urlparse_cached from scrapy.utils.python import to_unicode logger = logging.getLogger(__name__) +_split_domain = TLDExtract(include_psl_private_domains=True) + + +def _is_public_domain(domain): + parts = _split_domain(domain) + return not parts.domain + + class CookiesMiddleware: """This middleware enables working with sites that need cookies""" @@ -23,14 +34,29 @@ class CookiesMiddleware: raise NotConfigured return cls(crawler.settings.getbool('COOKIES_DEBUG')) + def _process_cookies(self, cookies, *, jar, request): + for cookie in cookies: + cookie_domain = cookie.domain + if cookie_domain.startswith('.'): + cookie_domain = cookie_domain[1:] + + request_domain = urlparse_cached(request).hostname.lower() + + if cookie_domain and _is_public_domain(cookie_domain): + if cookie_domain != request_domain: + continue + cookie.domain = request_domain + + jar.set_cookie_if_ok(cookie, request) + def process_request(self, request, spider): if request.meta.get('dont_merge_cookies', False): return cookiejarkey = request.meta.get("cookiejar") jar = self.jars[cookiejarkey] - for cookie in self._get_request_cookies(jar, request): - jar.set_cookie_if_ok(cookie, request) + cookies = self._get_request_cookies(jar, request) + self._process_cookies(cookies, jar=jar, request=request) # set Cookie header request.headers.pop('Cookie', None) @@ -44,7 +70,9 @@ class CookiesMiddleware: # extract cookies from Set-Cookie and drop invalid/expired cookies cookiejarkey = request.meta.get("cookiejar") jar = self.jars[cookiejarkey] - jar.extract_cookies(response, request) + cookies = jar.make_cookies(response, request) + self._process_cookies(cookies, jar=jar, request=request) + self._debug_set_cookie(response, spider) return response diff --git a/setup.py b/setup.py index 3a6ff2836..d86c0f285 100644 --- a/setup.py +++ b/setup.py @@ -32,6 +32,7 @@ install_requires = [ 'protego>=0.1.15', 'itemadapter>=0.1.0', 'setuptools', + 'tldextract', ] extras_require = {} cpython_dependencies = [ diff --git a/tests/test_downloadermiddleware_cookies.py b/tests/test_downloadermiddleware_cookies.py index 1747f3b94..ba7453255 100644 --- a/tests/test_downloadermiddleware_cookies.py +++ b/tests/test_downloadermiddleware_cookies.py @@ -15,6 +15,48 @@ from scrapy.utils.python import to_bytes from scrapy.utils.test import get_crawler +def _cookie_to_set_cookie_value(cookie): + """Given a cookie defined as a dictionary with name and value keys, and + optional path and domain keys, return the equivalent string that can be + associated to a ``Set-Cookie`` header.""" + decoded = {} + for key in ("name", "value", "path", "domain"): + if cookie.get(key) is None: + if key in ("name", "value"): + return + continue + if isinstance(cookie[key], (bool, float, int, str)): + decoded[key] = str(cookie[key]) + else: + try: + decoded[key] = cookie[key].decode("utf8") + except UnicodeDecodeError: + decoded[key] = cookie[key].decode("latin1", errors="replace") + + cookie_str = f"{decoded.pop('name')}={decoded.pop('value')}" + for key, value in decoded.items(): # path, domain + cookie_str += f"; {key.capitalize()}={value}" + return cookie_str + + +def _cookies_to_set_cookie_list(cookies): + """Given a group of cookie defined either as a dictionary or as a list of + dictionaries (i.e. in a format supported by the cookies parameter of + Request), return the equivalen list of strings that can be associated to a + ``Set-Cookie`` header.""" + if not cookies: + return [] + if isinstance(cookies, dict): + cookies = ({"name": k, "value": v} for k, v in cookies.items()) + return filter( + None, + ( + _cookie_to_set_cookie_value(cookie) + for cookie in cookies + ) + ) + + class CookiesMiddlewareTest(TestCase): def assertCookieValEqual(self, first, second, msg=None): @@ -523,3 +565,131 @@ class CookiesMiddlewareTest(TestCase): {'url': 'https://example.com', 'status': 302}, cookies2=False, ) + + def _test_user_set_cookie_domain_followup( + self, + url1, + url2, + domain, + *, + cookies1, + cookies2, + ): + input_cookies = [ + { + 'name': 'a', + 'value': 'b', + 'domain': domain, + } + ] + + request1 = Request(url1, cookies=input_cookies) + self.mw.process_request(request1, self.spider) + cookies = request1.headers.get('Cookie') + self.assertEqual(cookies, b"a=b" if cookies1 else None) + + request2 = Request(url2) + self.mw.process_request(request2, self.spider) + cookies = request2.headers.get('Cookie') + self.assertEqual(cookies, b"a=b" if cookies2 else None) + + def test_user_set_cookie_domain_suffix_private(self): + self._test_user_set_cookie_domain_followup( + 'https://books.toscrape.com', + 'https://quotes.toscrape.com', + 'toscrape.com', + cookies1=True, + cookies2=True, + ) + + def test_user_set_cookie_domain_suffix_public_period(self): + self._test_user_set_cookie_domain_followup( + 'https://foo.co.uk', + 'https://bar.co.uk', + 'co.uk', + cookies1=False, + cookies2=False, + ) + + def test_user_set_cookie_domain_suffix_public_private(self): + self._test_user_set_cookie_domain_followup( + 'https://foo.blogspot.com', + 'https://bar.blogspot.com', + 'blogspot.com', + cookies1=False, + cookies2=False, + ) + + def test_user_set_cookie_domain_public_period(self): + self._test_user_set_cookie_domain_followup( + 'https://co.uk', + 'https://co.uk', + 'co.uk', + cookies1=True, + cookies2=True, + ) + + def _test_server_set_cookie_domain_followup( + self, + url1, + url2, + domain, + *, + cookies, + ): + request1 = Request(url1) + self.mw.process_request(request1, self.spider) + + input_cookies = [ + { + 'name': 'a', + 'value': 'b', + 'domain': domain, + } + ] + + headers = { + 'Set-Cookie': _cookies_to_set_cookie_list(input_cookies), + } + response = Response(url1, status=200, headers=headers) + self.assertEqual( + self.mw.process_response(request1, response, self.spider), + response, + ) + + request2 = Request(url2) + self.mw.process_request(request2, self.spider) + actual_cookies = request2.headers.get('Cookie') + self.assertEqual(actual_cookies, b"a=b" if cookies else None) + + def test_server_set_cookie_domain_suffix_private(self): + self._test_server_set_cookie_domain_followup( + 'https://books.toscrape.com', + 'https://quotes.toscrape.com', + 'toscrape.com', + cookies=True, + ) + + def test_server_set_cookie_domain_suffix_public_period(self): + self._test_server_set_cookie_domain_followup( + 'https://foo.co.uk', + 'https://bar.co.uk', + 'co.uk', + cookies=False, + ) + + def test_server_set_cookie_domain_suffix_public_private(self): + self._test_server_set_cookie_domain_followup( + 'https://foo.blogspot.com', + 'https://bar.blogspot.com', + 'blogspot.com', + cookies=False, + ) + + def test_server_set_cookie_domain_public_period(self): + self._test_server_set_cookie_domain_followup( + 'https://co.uk', + 'https://co.uk', + 'co.uk', + cookies=True, + ) From 6b63e7c14758fdc59f37cb6c2c9b88abebe8606f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 12:43:11 +0100 Subject: [PATCH 095/126] =?UTF-8?q?Bump=20version:=202.5.0=20=E2=86=92=202?= =?UTF-8?q?.6.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .bumpversion.cfg | 2 +- scrapy/VERSION | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index d9e4a2831..5a5b51a01 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 2.5.0 +current_version = 2.6.0 commit = True tag = True tag_name = {new_version} diff --git a/scrapy/VERSION b/scrapy/VERSION index 437459cd9..e70b4523a 100644 --- a/scrapy/VERSION +++ b/scrapy/VERSION @@ -1 +1 @@ -2.5.0 +2.6.0 From 84853c4fa6eb30bdcba0f70b4426994d731509fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 13:01:20 +0100 Subject: [PATCH 096/126] bandit: allow-list B324 for the time being --- .bandit.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.bandit.yml b/.bandit.yml index 243379b0b..41f1bb597 100644 --- a/.bandit.yml +++ b/.bandit.yml @@ -8,6 +8,7 @@ skips: - B311 - B320 - B321 +- B324 - B402 # https://github.com/scrapy/scrapy/issues/4180 - B403 - B404 From d60636d0de94c5a08c25d1d6820faed0b45506b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 13:06:58 +0100 Subject: [PATCH 097/126] Fix redirect handling regression --- scrapy/downloadermiddlewares/redirect.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/scrapy/downloadermiddlewares/redirect.py b/scrapy/downloadermiddlewares/redirect.py index fcd6c298b..c8c84ffb2 100644 --- a/scrapy/downloadermiddlewares/redirect.py +++ b/scrapy/downloadermiddlewares/redirect.py @@ -12,11 +12,10 @@ from scrapy.exceptions import IgnoreRequest, NotConfigured logger = logging.getLogger(__name__) -def _build_redirect_request(source_request, *, url, method=None, body=None): +def _build_redirect_request(source_request, *, url, **kwargs): redirect_request = source_request.replace( url=url, - method=method, - body=body, + **kwargs, cookies=None, ) if 'Cookie' in redirect_request.headers: From fab3e907297abd89106fb040c1c0c6a24b9522a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 13:41:20 +0100 Subject: [PATCH 098/126] Cover 2.6.1 in the release notes --- docs/news.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index 9590fb1c4..5d92067b5 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,15 @@ Release notes ============= +.. _release-2.6.1: + +Scrapy 2.6.1 (2022-03-01) +------------------------- + +Fixes a regression introduced in 2.6.0 that would unset the request method when +following redirects. + + .. _release-2.6.0: Scrapy 2.6.0 (2022-03-01) From 23537a0f9580bfb28ac5d8b88f37df47e838f463 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 13:48:40 +0100 Subject: [PATCH 099/126] =?UTF-8?q?Bump=20version:=202.6.0=20=E2=86=92=202?= =?UTF-8?q?.6.1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .bumpversion.cfg | 2 +- scrapy/VERSION | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index 5a5b51a01..1d9b9c02f 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 2.6.0 +current_version = 2.6.1 commit = True tag = True tag_name = {new_version} diff --git a/scrapy/VERSION b/scrapy/VERSION index e70b4523a..6a6a3d8e3 100644 --- a/scrapy/VERSION +++ b/scrapy/VERSION @@ -1 +1 @@ -2.6.0 +2.6.1 From 50c8becbe02e6e71a6e7a57af0b28bcf38d9a3c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 1 Mar 2022 17:29:08 +0100 Subject: [PATCH 100/126] Freeze and upgrade CI packages (#5429) --- tests/requirements.txt | 2 +- tox.ini | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/requirements.txt b/tests/requirements.txt index bd72c8c46..398d1d16d 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -3,7 +3,7 @@ attrs dataclasses; python_version == '3.6' pyftpdlib pytest -pytest-cov +pytest-cov==3.0.0 pytest-xdist sybil >= 1.3.0 # https://github.com/cjw296/sybil/issues/20#issuecomment-605433422 testfixtures diff --git a/tox.ini b/tox.ini index fcd3563b2..db151f215 100644 --- a/tox.ini +++ b/tox.ini @@ -49,7 +49,7 @@ commands = [testenv:security] basepython = python3 deps = - bandit + bandit==1.7.3 commands = bandit -r -c .bandit.yml {posargs:scrapy} @@ -68,7 +68,7 @@ commands = basepython = python3 deps = {[testenv:extra-deps]deps} - pylint==2.12.1 + pylint==2.12.2 commands = pylint conftest.py docs extras scrapy setup.py tests From ccdbb795ff2fef0ff89c1fa478ad2d7c52ef64be Mon Sep 17 00:00:00 2001 From: Florentin Date: Tue, 1 Mar 2022 22:01:55 +0100 Subject: [PATCH 101/126] Recommend Common Crawl instead of Google Cache --- docs/topics/practices.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 1a9d56143..d0207fd18 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -262,7 +262,7 @@ Here are some tips to keep in mind when dealing with these kinds of sites: * disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use cookies to spot bot behaviour * use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting. -* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites +* if possible, use `Common Crawl`_ to fetch pages, instead of hitting the sites directly * use a pool of rotating IPs. For example, the free `Tor project`_ or paid services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a @@ -277,7 +277,7 @@ If you are still unable to prevent your bot getting banned, consider contacting .. _Tor project: https://www.torproject.org/ .. _commercial support: https://scrapy.org/support/ .. _ProxyMesh: https://proxymesh.com/ -.. _Google cache: http://www.googleguide.com/cached_pages.html +.. _Common Crawl: https://commoncrawl.org/ .. _testspiders: https://github.com/scrapinghub/testspiders .. _scrapoxy: https://scrapoxy.io/ .. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/ From d469214fe73574d35521e8e87963629a6c12bcd8 Mon Sep 17 00:00:00 2001 From: Ali Rastegar <68519335+VolVox99@users.noreply.github.com> Date: Tue, 8 Mar 2022 01:29:22 -0800 Subject: [PATCH 102/126] Update tutorial.rst (#5442) Fixed typo --- docs/intro/tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index 5697b9608..cde1b1ef4 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -488,7 +488,7 @@ The `JSON Lines`_ format is useful because it's stream-like, you can easily append new records to it. It doesn't have the same problem of JSON when you run twice. Also, as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like `JQ`_ to help -doing that at the command-line. +do that at the command-line. In small projects (like the one in this tutorial), that should be enough. However, if you want to perform more complex things with the scraped items, you From e264cc30d1e73d53de2e4048d7d9b6cffd59f8f0 Mon Sep 17 00:00:00 2001 From: NaincyKumariKnoldus <87004609+NaincyKumariKnoldus@users.noreply.github.com> Date: Thu, 10 Mar 2022 19:24:33 +0530 Subject: [PATCH 103/126] removed the pywin32 docs section (#5370) --- docs/faq.rst | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index 8283cab11..8a9ba809b 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -94,15 +94,6 @@ How can I scrape an item with attributes in different pages? See :ref:`topics-request-response-ref-request-callback-arguments`. - -Scrapy crashes with: ImportError: No module named win32api ----------------------------------------------------------- - -You need to install `pywin32`_ because of `this Twisted bug`_. - -.. _pywin32: https://sourceforge.net/projects/pywin32/ -.. _this Twisted bug: https://twistedmatrix.com/trac/ticket/3707 - How can I simulate a user login in my spider? --------------------------------------------- From 9a28eb0bad1acf986d997905a410058f77911b7c Mon Sep 17 00:00:00 2001 From: Eugene Date: Thu, 17 Mar 2022 05:39:54 +0100 Subject: [PATCH 104/126] Suggest installing the brotli package instead of brotlipy (#4267) --- docs/topics/downloader-middleware.rst | 5 +++-- tests/requirements.txt | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index a15637ed6..912600428 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -704,14 +704,15 @@ HttpCompressionMiddleware sent/received from web sites. This middleware also supports decoding `brotli-compressed`_ as well as - `zstd-compressed`_ responses, provided that `brotlipy`_ or `zstandard`_ is + `zstd-compressed`_ responses, provided that `brotli`_ or `zstandard`_ is installed, respectively. .. _brotli-compressed: https://www.ietf.org/rfc/rfc7932.txt -.. _brotlipy: https://pypi.org/project/brotlipy/ +.. _brotli: https://pypi.org/project/Brotli/ .. _zstd-compressed: https://www.ietf.org/rfc/rfc8478.txt .. _zstandard: https://pypi.org/project/zstandard/ + HttpCompressionMiddleware Settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/tests/requirements.txt b/tests/requirements.txt index 398d1d16d..d2a8aae1b 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -12,7 +12,7 @@ uvloop; platform_system != "Windows" and python_version > '3.6' # optional for shell wrapper tests bpython -brotlipy # optional for HTTP compress downloader middleware tests +brotli # optional for HTTP compress downloader middleware tests zstandard; implementation_name != 'pypy' # optional for HTTP compress downloader middleware tests ipython pywin32; sys_platform == "win32" From 6a3f2ee6876145bd4bd9ee1ff89d94474a1e85a0 Mon Sep 17 00:00:00 2001 From: FJMonteroInformatica Date: Thu, 17 Mar 2022 20:09:56 +0100 Subject: [PATCH 105/126] HTML Conventions --- docs/_static/selectors-sample1.html | 31 +++++++------- .../link_extractor/linkextractor.html | 40 ++++++++++--------- .../link_extractor/linkextractor_latin1.html | 8 ++-- .../link_extractor/linkextractor_no_href.html | 3 +- .../link_extractor/linkextractor_noenc.html | 23 ++++++----- tests/sample_data/test_site/index.html | 31 +++++++------- tests/sample_data/test_site/item1.html | 27 ++++++------- tests/sample_data/test_site/item2.html | 29 ++++++-------- 8 files changed, 96 insertions(+), 96 deletions(-) diff --git a/docs/_static/selectors-sample1.html b/docs/_static/selectors-sample1.html index 8a79a3381..915718832 100644 --- a/docs/_static/selectors-sample1.html +++ b/docs/_static/selectors-sample1.html @@ -1,16 +1,17 @@ - - - - Example website - - - - - + + + + + Example website + + + + + \ No newline at end of file diff --git a/tests/sample_data/link_extractor/linkextractor.html b/tests/sample_data/link_extractor/linkextractor.html index 2307ea865..e3a2a4145 100644 --- a/tests/sample_data/link_extractor/linkextractor.html +++ b/tests/sample_data/link_extractor/linkextractor.html @@ -1,20 +1,22 @@ + + - - -Sample page with links for testing LinkExtractor - - - - - + + + Sample page with links for testing LinkExtractor + + + + + \ No newline at end of file diff --git a/tests/sample_data/link_extractor/linkextractor_latin1.html b/tests/sample_data/link_extractor/linkextractor_latin1.html index e7eee18de..1e05bf0f0 100644 --- a/tests/sample_data/link_extractor/linkextractor_latin1.html +++ b/tests/sample_data/link_extractor/linkextractor_latin1.html @@ -1,3 +1,5 @@ + + @@ -7,11 +9,11 @@ diff --git a/tests/sample_data/link_extractor/linkextractor_no_href.html b/tests/sample_data/link_extractor/linkextractor_no_href.html index 0b01cede8..2d67ec6ff 100644 --- a/tests/sample_data/link_extractor/linkextractor_no_href.html +++ b/tests/sample_data/link_extractor/linkextractor_no_href.html @@ -1,3 +1,5 @@ + + @@ -21,5 +23,4 @@ - \ No newline at end of file diff --git a/tests/sample_data/link_extractor/linkextractor_noenc.html b/tests/sample_data/link_extractor/linkextractor_noenc.html index f9166adbe..6fa137cd9 100644 --- a/tests/sample_data/link_extractor/linkextractor_noenc.html +++ b/tests/sample_data/link_extractor/linkextractor_noenc.html @@ -1,14 +1,17 @@ + + - - -Sample page without encoding for testing LinkExtractor - + + + Sample page without encoding for testing LinkExtractor + + -
-
- -
-sample € text -
+
+
+ sample2 +
+ sample € text +
diff --git a/tests/sample_data/test_site/index.html b/tests/sample_data/test_site/index.html index d268c846a..afe17d8e2 100644 --- a/tests/sample_data/test_site/index.html +++ b/tests/sample_data/test_site/index.html @@ -1,18 +1,15 @@ + + - - -Scrapy test site - - - - -

Scrapy test site

- - - - - + + Scrapy test site + + +

Scrapy test site

+ + + \ No newline at end of file diff --git a/tests/sample_data/test_site/item1.html b/tests/sample_data/test_site/item1.html index ceeb6dc87..ee39f16f3 100644 --- a/tests/sample_data/test_site/item1.html +++ b/tests/sample_data/test_site/item1.html @@ -1,17 +1,14 @@ + + - - -Item 1 - Scrapy test site - - - - -

Item 1 name

- -
    -
  • Price: $100
    • -
  • Stock: 12
    • -
- - + + Item 1 - Scrapy test site + + +

Item 1 name

+
    +
  • Price: $100
    • +
  • Stock: 12
    • +
+ diff --git a/tests/sample_data/test_site/item2.html b/tests/sample_data/test_site/item2.html index a64c92810..f40f70750 100644 --- a/tests/sample_data/test_site/item2.html +++ b/tests/sample_data/test_site/item2.html @@ -1,17 +1,14 @@ + + - - -Item 2 - Scrapy test site - - - - -

Item 2 name

- -
    -
  • Price: $200
    • -
  • Stock: 5
    • -
- - - + + Item 2 - Scrapy test site + + +

Item 2 name

+
    +
  • Price: $200
    • +
  • Stock: 5
    • +
+ + \ No newline at end of file From fcf3d8e0a0df447fd7cd81e98c846a16b8b42a73 Mon Sep 17 00:00:00 2001 From: D00399830 Date: Mon, 21 Mar 2022 14:09:31 -0600 Subject: [PATCH 106/126] Updated the documentation for developer tools to have JavaScript instead of Javascript, as JavaScript is the more correct way to write it --- docs/topics/developer-tools.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst index 96475899f..9bf97c628 100644 --- a/docs/topics/developer-tools.rst +++ b/docs/topics/developer-tools.rst @@ -19,14 +19,14 @@ Caveats with inspecting the live browser DOM Since Developer Tools operate on a live browser DOM, what you'll actually see when inspecting the page source is not the original HTML, but a modified one -after applying some browser clean up and executing Javascript code. Firefox, +after applying some browser clean up and executing JavaScript code. Firefox, in particular, is known for adding ```` elements to tables. Scrapy, on the other hand, does not modify the original page HTML, so you won't be able to extract any data if you use ```` in your XPath expressions. Therefore, you should keep in mind the following things: -* Disable Javascript while inspecting the DOM looking for XPaths to be +* Disable JavaScript while inspecting the DOM looking for XPaths to be used in Scrapy (in the Developer Tools settings click `Disable JavaScript`) * Never use full XPath paths, use relative and clever ones based on attributes From 2227be7af6d0504d52bd4c2e299efb1becbd992f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?V=C3=ADctor=20Ruiz?= Date: Tue, 22 Mar 2022 15:21:16 +0100 Subject: [PATCH 107/126] Fix a typo in the HTTP cache documentation (#5455) --- docs/topics/downloader-middleware.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 912600428..29e350651 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -366,7 +366,7 @@ HttpCacheMiddleware This middleware provides low-level cache to all HTTP requests and responses. It has to be combined with a cache storage backend as well as a cache policy. - Scrapy ships with three HTTP cache storage backends: + Scrapy ships with the following HTTP cache storage backends: * :ref:`httpcache-storage-fs` * :ref:`httpcache-storage-dbm` From 319e67f779163df3ad44327bfbc9733edcb34908 Mon Sep 17 00:00:00 2001 From: Yash <76577754+yash-fn@users.noreply.github.com> Date: Sat, 26 Mar 2022 18:17:03 -0500 Subject: [PATCH 108/126] documentation update for multiple spiders i noticed passing settings to configure logging function made weird output go away. checked documentation and it says first parameter is settings file. Is this correct? --- docs/topics/practices.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index d0207fd18..7313c9246 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -180,8 +180,8 @@ Same example but running the spiders sequentially by chaining the deferreds: # Your second spider definition ... - configure_logging() settings = get_project_settings() + configure_logging(settings) runner = CrawlerRunner(settings) @defer.inlineCallbacks From b2afcbfe2bf090827540d072866bef0d1ab3a3e8 Mon Sep 17 00:00:00 2001 From: AngelikiBoura <73474686+AngelikiBoura@users.noreply.github.com> Date: Thu, 5 May 2022 16:49:52 +0300 Subject: [PATCH 109/126] Fix typos in three files for Flake8 check (#5487) * Fix typos in extensions files Made some fixes in files memusage.py and statsmailer.py in order to pass the flake8 check. * Fix typos in twisted_reactor_custom_settings_same.py A small change was needed in order for flake8 check to pass. --- scrapy/extensions/memusage.py | 10 +++++----- scrapy/extensions/statsmailer.py | 1 + .../twisted_reactor_custom_settings_same.py | 1 + 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/scrapy/extensions/memusage.py b/scrapy/extensions/memusage.py index 9de119a10..f5081a7d7 100644 --- a/scrapy/extensions/memusage.py +++ b/scrapy/extensions/memusage.py @@ -33,8 +33,8 @@ class MemoryUsage: self.crawler = crawler self.warned = False self.notify_mails = crawler.settings.getlist('MEMUSAGE_NOTIFY_MAIL') - self.limit = crawler.settings.getint('MEMUSAGE_LIMIT_MB')*1024*1024 - self.warning = crawler.settings.getint('MEMUSAGE_WARNING_MB')*1024*1024 + self.limit = crawler.settings.getint('MEMUSAGE_LIMIT_MB') * 1024 * 1024 + self.warning = crawler.settings.getint('MEMUSAGE_WARNING_MB') * 1024 * 1024 self.check_interval = crawler.settings.getfloat('MEMUSAGE_CHECK_INTERVAL_SECONDS') self.mail = MailSender.from_settings(crawler.settings) crawler.signals.connect(self.engine_started, signal=signals.engine_started) @@ -77,7 +77,7 @@ class MemoryUsage: def _check_limit(self): if self.get_virtual_size() > self.limit: self.crawler.stats.set_value('memusage/limit_reached', 1) - mem = self.limit/1024/1024 + mem = self.limit / 1024 / 1024 logger.error("Memory usage exceeded %(memusage)dM. Shutting down Scrapy...", {'memusage': mem}, extra={'crawler': self.crawler}) if self.notify_mails: @@ -94,11 +94,11 @@ class MemoryUsage: self.crawler.stop() def _check_warning(self): - if self.warned: # warn only once + if self.warned: # warn only once return if self.get_virtual_size() > self.warning: self.crawler.stats.set_value('memusage/warning_reached', 1) - mem = self.warning/1024/1024 + mem = self.warning / 1024 / 1024 logger.warning("Memory usage reached %(memusage)dM", {'memusage': mem}, extra={'crawler': self.crawler}) if self.notify_mails: diff --git a/scrapy/extensions/statsmailer.py b/scrapy/extensions/statsmailer.py index bcdbaff24..739e6b958 100644 --- a/scrapy/extensions/statsmailer.py +++ b/scrapy/extensions/statsmailer.py @@ -8,6 +8,7 @@ from scrapy import signals from scrapy.mail import MailSender from scrapy.exceptions import NotConfigured + class StatsMailer: def __init__(self, stats, recipients, mail): diff --git a/tests/CrawlerProcess/twisted_reactor_custom_settings_same.py b/tests/CrawlerProcess/twisted_reactor_custom_settings_same.py index 1f5a44010..72bb986bc 100644 --- a/tests/CrawlerProcess/twisted_reactor_custom_settings_same.py +++ b/tests/CrawlerProcess/twisted_reactor_custom_settings_same.py @@ -8,6 +8,7 @@ class AsyncioReactorSpider1(scrapy.Spider): "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", } + class AsyncioReactorSpider2(scrapy.Spider): name = 'asyncio_reactor2' custom_settings = { From 83c1939281197242511931c6e9f356f2498eb623 Mon Sep 17 00:00:00 2001 From: Andreas Tziortziortziopoulos Date: Fri, 6 May 2022 03:59:30 +0300 Subject: [PATCH 110/126] Issue #3264, fix error handling when spider is not matched Changes Implementation: - Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider Testing: - Add a test case with invalid url inside test_command_parse Test proves that non-matched Spider does not throw an AttributeError --- scrapy/commands/parse.py | 3 ++- tests/test_command_parse.py | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/scrapy/commands/parse.py b/scrapy/commands/parse.py index a3f6b96f4..99fc8f955 100644 --- a/scrapy/commands/parse.py +++ b/scrapy/commands/parse.py @@ -146,7 +146,8 @@ class Command(BaseRunSpiderCommand): def _start_requests(spider): yield self.prepare_request(spider, Request(url), opts) - self.spidercls.start_requests = _start_requests + if self.spidercls: + self.spidercls.start_requests = _start_requests def start_parsing(self, url, opts): self.crawler_process.crawl(self.spidercls, **opts.spargs) diff --git a/tests/test_command_parse.py b/tests/test_command_parse.py index f21ee971d..0622074a3 100644 --- a/tests/test_command_parse.py +++ b/tests/test_command_parse.py @@ -1,6 +1,7 @@ import os import argparse from os.path import join, abspath, isfile, exists + from twisted.internet import defer from scrapy.commands import parse from scrapy.settings import Settings @@ -222,6 +223,10 @@ ITEM_PIPELINES = {{'{self.project_name}.pipelines.MyPipeline': 1}} self.assertRegex(_textmode(out), r"""# Scraped Items -+\n\[\]""") self.assertIn("""Cannot find a rule that matches""", _textmode(stderr)) + status, out, stderr = yield self.execute([self.url('/invalid_url')]) + self.assertEqual(status, 0) + self.assertIn("""""", _textmode(stderr)) + @defer.inlineCallbacks def test_output_flag(self): """Checks if a file was created successfully having From 965fde24a4798ee51f05ce8669b7a28958ad3238 Mon Sep 17 00:00:00 2001 From: Andrey Rahmatullin Date: Fri, 8 Apr 2022 14:26:23 +0500 Subject: [PATCH 111/126] Pin mitmproxy to < 8 for now (#5459) --- tox.ini | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tox.ini b/tox.ini index db151f215..d13bb7b38 100644 --- a/tox.ini +++ b/tox.ini @@ -12,10 +12,11 @@ deps = -rtests/requirements.txt # mitmproxy does not support PyPy # mitmproxy does not support Windows when running Python < 3.7 - # Python 3.9+ requires https://github.com/mitmproxy/mitmproxy/commit/8e5e43de24c9bc93092b63efc67fbec029a9e7fe + # Python 3.9+ requires mitmproxy >= 5.3.0 # mitmproxy >= 5.3.0 requires h2 >= 4.0, Twisted 21.2 requires h2 < 4.0 #mitmproxy >= 5.3.0; python_version >= '3.9' and implementation_name != 'pypy' - mitmproxy >= 4.0.4; python_version >= '3.7' and python_version < '3.9' and implementation_name != 'pypy' + # The tests hang with mitmproxy 8.0.0: https://github.com/scrapy/scrapy/issues/5454 + mitmproxy >= 4.0.4, < 8; python_version >= '3.7' and python_version < '3.9' and implementation_name != 'pypy' mitmproxy >= 4.0.4, < 5; python_version >= '3.6' and python_version < '3.7' and platform_system != 'Windows' and implementation_name != 'pypy' # newer markupsafe is incompatible with deps of old mitmproxy (which we get on Python 3.7 and lower) markupsafe < 2.1.0; python_version >= '3.6' and python_version < '3.8' and implementation_name != 'pypy' From 078622cfb0ee364acba5d91a20244f9c1ee87d30 Mon Sep 17 00:00:00 2001 From: Maxime Nannan <28675918+mnannan@users.noreply.github.com> Date: Fri, 20 May 2022 08:30:06 +0200 Subject: [PATCH 112/126] Fix file expiration issue with GCS (#5318) --- scrapy/pipelines/files.py | 10 +++++++--- tests/test_pipeline_files.py | 23 +++++++++++++++++++++++ tox.ini | 7 ++++--- 3 files changed, 34 insertions(+), 6 deletions(-) diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py index 5c52c6c28..906e7eb24 100644 --- a/scrapy/pipelines/files.py +++ b/scrapy/pipelines/files.py @@ -222,8 +222,8 @@ class GCSFilesStore: return {'checksum': checksum, 'last_modified': last_modified} else: return {} - - return threads.deferToThread(self.bucket.get_blob, path).addCallback(_onsuccess) + blob_path = self._get_blob_path(path) + return threads.deferToThread(self.bucket.get_blob, blob_path).addCallback(_onsuccess) def _get_content_type(self, headers): if headers and 'Content-Type' in headers: @@ -231,8 +231,12 @@ class GCSFilesStore: else: return 'application/octet-stream' + def _get_blob_path(self, path): + return self.prefix + path + def persist_file(self, path, buf, info, meta=None, headers=None): - blob = self.bucket.blob(self.prefix + path) + blob_path = self._get_blob_path(path) + blob = self.bucket.blob(blob_path) blob.cache_control = self.CACHE_CONTROL blob.metadata = {k: str(v) for k, v in (meta or {}).items()} return threads.deferToThread( diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py index 4e1b90787..0ff2045ed 100644 --- a/tests/test_pipeline_files.py +++ b/tests/test_pipeline_files.py @@ -525,6 +525,29 @@ class TestGCSFilesStore(unittest.TestCase): self.assertEqual(blob.content_type, 'application/octet-stream') self.assertIn(expected_policy, acl) + @defer.inlineCallbacks + def test_blob_path_consistency(self): + """Test to make sure that paths used to store files is the same as the one used to get + already uploaded files. + """ + assert_gcs_environ() + try: + import google.cloud.storage # noqa + except ModuleNotFoundError: + raise unittest.SkipTest("google-cloud-storage is not installed") + else: + with mock.patch('google.cloud.storage') as _: + with mock.patch('scrapy.pipelines.files.time') as _: + uri = 'gs://my_bucket/my_prefix/' + store = GCSFilesStore(uri) + store.bucket = mock.Mock() + path = 'full/my_data.txt' + yield store.persist_file(path, mock.Mock(), info=None, meta=None, headers=None) + yield store.stat_file(path, info=None) + expected_blob_path = store.prefix + path + store.bucket.blob.assert_called_with(expected_blob_path) + store.bucket.get_blob.assert_called_with(expected_blob_path) + class TestFTPFileStore(unittest.TestCase): @defer.inlineCallbacks diff --git a/tox.ini b/tox.ini index d13bb7b38..6951b6d16 100644 --- a/tox.ini +++ b/tox.ini @@ -126,13 +126,14 @@ setenv = deps = {[testenv]deps} boto + google-cloud-storage + # Twisted[http2] currently forces old mitmproxy because of h2 version + # restrictions in their deps, so we need to pin old markupsafe here too. + markupsafe < 2.1.0 reppy robotexclusionrulesparser Pillow>=4.0.0 Twisted[http2]>=17.9.0 - # Twisted[http2] currently forces old mitmproxy because of h2 version restrictions in their deps, - # so we need to pin old markupsafe here too - markupsafe < 2.1.0 [testenv:asyncio] commands = From b5c15d87ff5770220bca31792c89e58804b923bb Mon Sep 17 00:00:00 2001 From: Andreas Tziortziortziopoulos Date: Sun, 22 May 2022 12:19:20 +0300 Subject: [PATCH 113/126] [issue3264] Separate test for not matched spider to a url --- tests/test_command_parse.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_command_parse.py b/tests/test_command_parse.py index 0622074a3..0d992be56 100644 --- a/tests/test_command_parse.py +++ b/tests/test_command_parse.py @@ -223,9 +223,10 @@ ITEM_PIPELINES = {{'{self.project_name}.pipelines.MyPipeline': 1}} self.assertRegex(_textmode(out), r"""# Scraped Items -+\n\[\]""") self.assertIn("""Cannot find a rule that matches""", _textmode(stderr)) + @defer.inlineCallbacks + def test_crawlspider_not_exists_with_not_matched_url(self): status, out, stderr = yield self.execute([self.url('/invalid_url')]) self.assertEqual(status, 0) - self.assertIn("""""", _textmode(stderr)) @defer.inlineCallbacks def test_output_flag(self): From 86331900125dc311223cbd1ebb0e10d09e7c592d Mon Sep 17 00:00:00 2001 From: Mohammadtaher Abbasi Date: Tue, 24 May 2022 14:47:00 +0430 Subject: [PATCH 114/126] pass on item to thumb_path function as additional argument resolves #5504 --- scrapy/pipelines/images.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scrapy/pipelines/images.py b/scrapy/pipelines/images.py index 9c99dc69e..45ac03820 100644 --- a/scrapy/pipelines/images.py +++ b/scrapy/pipelines/images.py @@ -141,7 +141,7 @@ class ImagesPipeline(FilesPipeline): yield path, image, buf for thumb_id, size in self.thumbs.items(): - thumb_path = self.thumb_path(request, thumb_id, response=response, info=info) + thumb_path = self.thumb_path(request, thumb_id, response=response, info=info, item=item) thumb_image, thumb_buf = self.convert_image(image, size) yield thumb_path, thumb_image, thumb_buf @@ -179,6 +179,6 @@ class ImagesPipeline(FilesPipeline): image_guid = hashlib.sha1(to_bytes(request.url)).hexdigest() return f'full/{image_guid}.jpg' - def thumb_path(self, request, thumb_id, response=None, info=None): + def thumb_path(self, request, thumb_id, response=None, info=None, item=None): thumb_guid = hashlib.sha1(to_bytes(request.url)).hexdigest() return f'thumbs/{thumb_id}/{thumb_guid}.jpg' From f39def4492f838b2414324e22a12f3b355f5c062 Mon Sep 17 00:00:00 2001 From: Mohammadtaher Abbasi Date: Wed, 25 May 2022 23:57:38 +0430 Subject: [PATCH 115/126] add docs --- docs/topics/media-pipeline.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index 7dff78390..2513faae2 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -656,6 +656,26 @@ See here the methods that you can override in your custom Images Pipeline: .. versionadded:: 2.4 The *item* parameter. + .. method:: ImagesPipeline.thumb_path(self, request, thumb_id, response=None, info=None, *, item=None) + + This method is called for every item of :setting:`IMAGES_THUMBS` per downloaded item. It returns the + thumbnail download path of the image originating from the specified + :class:`response `. + + In addition to ``response``, this method receives the original + :class:`request `, + ``thumb_id``, + :class:`info ` and + :class:`item `. + + You can override this method to customize the thumbnail download path of each image. + You can use the ``item`` to determine the file path based on some item + property. + + By default the :meth:`thumb_path` method returns + ``thumbs//.``. + + .. method:: ImagesPipeline.get_media_requests(item, info) Works the same way as :meth:`FilesPipeline.get_media_requests` method, From 5c586d78f0e1c5b66358ed644bb6e528ad4b062b Mon Sep 17 00:00:00 2001 From: Mohammadtaher Abbasi Date: Wed, 25 May 2022 23:58:09 +0430 Subject: [PATCH 116/126] add tests --- tests/test_pipeline_images.py | 16 ++++++++++++++++ tests/test_pipeline_media.py | 28 +++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/tests/test_pipeline_images.py b/tests/test_pipeline_images.py index c69cd0e4a..dd94d296b 100644 --- a/tests/test_pipeline_images.py +++ b/tests/test_pipeline_images.py @@ -93,6 +93,22 @@ class ImagesPipelineTestCase(unittest.TestCase): info=object()), 'thumbs/50/850233df65a5b83361798f532f1fc549cd13cbe9.jpg') + def test_thumbnail_name_from_item(self): + """ + Custom thumbnail name based on item data, overriding default implementation + """ + + class CustomImagesPipeline(ImagesPipeline): + def thumb_path(self, request, thumb_id, response=None, info=None, item=None): + return f"thumb/{thumb_id}/{item.get('path')}" + + thumb_path = CustomImagesPipeline.from_settings(Settings( + {'IMAGES_STORE': self.tempdir} + )).thumb_path + item = dict(path='path-to-store-file') + request = Request("http://example.com") + self.assertEqual(thumb_path(request, 'small', item=item), 'thumb/small/path-to-store-file') + def test_convert_image(self): SIZE = (100, 100) # straigh forward case: RGB and JPEG diff --git a/tests/test_pipeline_media.py b/tests/test_pipeline_media.py index 893d43052..a802c7cf1 100644 --- a/tests/test_pipeline_media.py +++ b/tests/test_pipeline_media.py @@ -1,4 +1,5 @@ from typing import Optional +import io from testfixtures import LogCapture from twisted.trial import unittest @@ -355,9 +356,12 @@ class MockedMediaPipelineDeprecatedMethods(ImagesPipeline): def get_media_requests(self, item, info): item_url = item['image_urls'][0] + output_img = io.BytesIO() + img = Image.new('RGB', (60, 30), color='red') + img.save(output_img, format='JPEG') return Request( item_url, - meta={'response': Response(item_url, status=200, body=b'data')} + meta={'response': Response(item_url, status=200, body=output_img.getvalue())} ) def inc_stats(self, *args, **kwargs): @@ -379,9 +383,13 @@ class MockedMediaPipelineDeprecatedMethods(ImagesPipeline): self._mockcalled.append('file_path') return super(MockedMediaPipelineDeprecatedMethods, self).file_path(request, response, info) + def thumb_path(self, request, thumb_id, response=None, info=None): + self._mockcalled.append('thumb_path') + return super(MockedMediaPipelineDeprecatedMethods, self).thumb_path(request, thumb_id, response, info) + def get_images(self, response, request, info): self._mockcalled.append('get_images') - return [] + return super(MockedMediaPipelineDeprecatedMethods, self).get_images(response, request, info) def image_downloaded(self, response, request, info): self._mockcalled.append('image_downloaded') @@ -392,7 +400,11 @@ class MediaPipelineDeprecatedMethodsTestCase(unittest.TestCase): skip = skip_pillow def setUp(self): - self.pipe = MockedMediaPipelineDeprecatedMethods(store_uri='store-uri', download_func=_mocked_download_func) + self.pipe = MockedMediaPipelineDeprecatedMethods( + store_uri='store-uri', + download_func=_mocked_download_func, + settings=Settings({"IMAGES_THUMBS": {'small': (50, 50)}}) + ) self.pipe.open_spider(None) self.item = dict(image_urls=['http://picsum.photos/id/1014/200/300'], images=[]) @@ -444,6 +456,16 @@ class MediaPipelineDeprecatedMethodsTestCase(unittest.TestCase): ) self._assert_method_called_with_warnings('file_path', message, warnings) + @inlineCallbacks + def test_thumb_path_called(self): + yield self.pipe.process_item(self.item, None) + warnings = self.flushWarnings([MediaPipeline._compatible]) + message = ( + 'thumb_path(self, request, thumb_id, response=None, info=None) is deprecated, ' + 'please use thumb_path(self, request, thumb_id, response=None, info=None, *, item=None)' + ) + self._assert_method_called_with_warnings('thumb_path', message, warnings) + @inlineCallbacks def test_get_images_called(self): yield self.pipe.process_item(self.item, None) From 896f16f2def7c276350421352dddf6e7b0145519 Mon Sep 17 00:00:00 2001 From: Mohammadtaher Abbasi Date: Wed, 25 May 2022 23:59:25 +0430 Subject: [PATCH 117/126] make thumb_path method backwards compatible --- scrapy/pipelines/images.py | 2 +- scrapy/pipelines/media.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/scrapy/pipelines/images.py b/scrapy/pipelines/images.py index 45ac03820..6b97190ee 100644 --- a/scrapy/pipelines/images.py +++ b/scrapy/pipelines/images.py @@ -179,6 +179,6 @@ class ImagesPipeline(FilesPipeline): image_guid = hashlib.sha1(to_bytes(request.url)).hexdigest() return f'full/{image_guid}.jpg' - def thumb_path(self, request, thumb_id, response=None, info=None, item=None): + def thumb_path(self, request, thumb_id, response=None, info=None, *, item=None): thumb_guid = hashlib.sha1(to_bytes(request.url)).hexdigest() return f'thumbs/{thumb_id}/{thumb_guid}.jpg' diff --git a/scrapy/pipelines/media.py b/scrapy/pipelines/media.py index d1bccf323..430c37227 100644 --- a/scrapy/pipelines/media.py +++ b/scrapy/pipelines/media.py @@ -121,7 +121,7 @@ class MediaPipeline: def _make_compatible(self): """Make overridable methods of MediaPipeline and subclasses backwards compatible""" methods = [ - "file_path", "media_to_download", "media_downloaded", + "file_path", "thumb_path", "media_to_download", "media_downloaded", "file_downloaded", "image_downloaded", "get_images" ] From c5627af15bcf413c04539aeb47dd07cf8b3e4092 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 7 Jun 2022 18:44:54 +0200 Subject: [PATCH 118/126] Centralize request fingerprints (#4524) Co-authored-by: Mikhail Korobov --- docs/conf.py | 2 + docs/news.rst | 2 +- docs/topics/api.rst | 7 + docs/topics/item-pipeline.rst | 4 +- docs/topics/request-response.rst | 268 +++++++ docs/topics/settings.rst | 8 +- scrapy/crawler.py | 7 + scrapy/dupefilters.py | 49 +- scrapy/extensions/httpcache.py | 14 +- scrapy/pipelines/media.py | 4 +- scrapy/settings/default_settings.py | 3 + .../templates/project/module/settings.py.tmpl | 3 + scrapy/utils/request.py | 235 +++++- scrapy/utils/test.py | 9 +- tests/test_crawler.py | 3 +- tests/test_dupefilters.py | 80 +- tests/test_pipeline_files.py | 5 +- tests/test_pipeline_media.py | 53 +- tests/test_utils_request.py | 699 ++++++++++++++++-- 19 files changed, 1304 insertions(+), 151 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index 55aa72d5a..378b01804 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -294,7 +294,9 @@ intersphinx_mapping = { 'tox': ('https://tox.readthedocs.io/en/latest', None), 'twisted': ('https://twistedmatrix.com/documents/current', None), 'twistedapi': ('https://twistedmatrix.com/documents/current/api', None), + 'w3lib': ('https://w3lib.readthedocs.io/en/latest', None), } +intersphinx_disabled_reftypes = [] # Options for sphinx-hoverxref options diff --git a/docs/news.rst b/docs/news.rst index 5d92067b5..2d0ab485e 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1643,7 +1643,7 @@ New features :issue:`4370`) * A new ``keep_fragments`` parameter of - :func:`scrapy.utils.request.request_fingerprint` allows to generate + ``scrapy.utils.request.request_fingerprint`` allows to generate different fingerprints for requests with different fragments in their URL (:issue:`4104`) diff --git a/docs/topics/api.rst b/docs/topics/api.rst index 900b19c7a..60b5acd10 100644 --- a/docs/topics/api.rst +++ b/docs/topics/api.rst @@ -32,6 +32,13 @@ how you :ref:`configure the downloader middlewares :class:`scrapy.Spider` subclass and a :class:`scrapy.settings.Settings` object. + .. attribute:: request_fingerprinter + + The request fingerprint builder of this crawler. + + This is used from extensions and middlewares to build short, unique + identifiers for requests. See :ref:`request-fingerprints`. + .. attribute:: settings The settings manager of this crawler. diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index 391751364..882ff5661 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -60,9 +60,9 @@ Additionally, they may also implement the following methods: :param spider: the spider which was closed :type spider: :class:`~scrapy.Spider` object -.. method:: from_crawler(cls, crawler) +.. classmethod:: from_crawler(cls, crawler) - If present, this classmethod is called to create a pipeline instance + If present, this class method is called to create a pipeline instance from a :class:`~scrapy.crawler.Crawler`. It must return a new instance of the pipeline. Crawler object provides access to all Scrapy core components like settings and signals; it is a way for pipeline to diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 92a471faf..49cb69f67 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -339,6 +339,7 @@ errors if needed:: request = failure.request self.logger.error('TimeoutError on %s', request.url) + .. _errback-cb_kwargs: Accessing additional data in errback functions @@ -364,6 +365,273 @@ achieve this by using ``Failure.request.cb_kwargs``:: main_url=failure.request.cb_kwargs['main_url'], ) + +.. _request-fingerprints: + +Request fingerprints +-------------------- + +There are some aspects of scraping, such as filtering out duplicate requests +(see :setting:`DUPEFILTER_CLASS`) or caching responses (see +:setting:`HTTPCACHE_POLICY`), where you need the ability to generate a short, +unique identifier from a :class:`~scrapy.http.Request` object: a request +fingerprint. + +You often do not need to worry about request fingerprints, the default request +fingerprinter works for most projects. + +However, there is no universal way to generate a unique identifier from a +request, because different situations require comparing requests differently. +For example, sometimes you may need to compare URLs case-insensitively, include +URL fragments, exclude certain URL query parameters, include some or all +headers, etc. + +To change how request fingerprints are built for your requests, use the +:setting:`REQUEST_FINGERPRINTER_CLASS` setting. + +.. setting:: REQUEST_FINGERPRINTER_CLASS + +REQUEST_FINGERPRINTER_CLASS +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: VERSION + +Default: :class:`scrapy.utils.request.RequestFingerprinter` + +A :ref:`request fingerprinter class ` or its +import path. + +.. autoclass:: scrapy.utils.request.RequestFingerprinter + + +.. setting:: REQUEST_FINGERPRINTER_IMPLEMENTATION + +REQUEST_FINGERPRINTER_IMPLEMENTATION +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: VERSION + +Default: ``'PREVIOUS_VERSION'`` + +Determines which request fingerprinting algorithm is used by the default +request fingerprinter class (see :setting:`REQUEST_FINGERPRINTER_CLASS`). + +Possible values are: + +- ``'PREVIOUS_VERSION'`` (default) + + This implementation uses the same request fingerprinting algorithm as + Scrapy PREVIOUS_VERSION and earlier versions. + + Even though this is the default value for backward compatibility reasons, + it is a deprecated value. + +- ``'VERSION'`` + + This implementation was introduced in Scrapy VERSION to fix an issue of the + previous implementation. + + New projects should use this value. The :command:`startproject` command + sets this value in the generated ``settings.py`` file. + +If you are using the default value (``'PREVIOUS_VERSION'``) for this setting, and you are +using Scrapy components where changing the request fingerprinting algorithm +would cause undesired results, you need to carefully decide when to change the +value of this setting, or switch the :setting:`REQUEST_FINGERPRINTER_CLASS` +setting to a custom request fingerprinter class that implements the PREVIOUS_VERSION request +fingerprinting algorithm and does not log this warning ( +:ref:`PREVIOUS_VERSION-request-fingerprinter` includes an example implementation of such a +class). + +Scenarios where changing the request fingerprinting algorithm may cause +undesired results include, for example, using the HTTP cache middleware (see +:class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`). +Changing the request fingerprinting algorithm would invalidade the current +cache, requiring you to redownload all requests again. + +Otherwise, set :setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION` to ``'VERSION'`` in +your settings to switch already to the request fingerprinting implementation +that will be the only request fingerprinting implementation available in a +future version of Scrapy, and remove the deprecation warning triggered by using +the default value (``'PREVIOUS_VERSION'``). + + +.. _PREVIOUS_VERSION-request-fingerprinter: +.. _custom-request-fingerprinter: + +Writing your own request fingerprinter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A request fingerprinter is a class that must implement the following method: + +.. method:: fingerprint(self, request) + + Return a :class:`bytes` object that uniquely identifies *request*. + + See also :ref:`request-fingerprint-restrictions`. + + :param request: request to fingerprint + :type request: scrapy.http.Request + +Additionally, it may also implement the following methods: + +.. classmethod:: from_crawler(cls, crawler) + + If present, this class method is called to create a request fingerprinter + instance from a :class:`~scrapy.crawler.Crawler` object. It must return a + new instance of the request fingerprinter. + + *crawler* provides access to all Scrapy core components like settings and + signals; it is a way for the request fingerprinter to access them and hook + its functionality into Scrapy. + + :param crawler: crawler that uses this request fingerprinter + :type crawler: :class:`~scrapy.crawler.Crawler` object + +.. classmethod:: from_settings(cls, settings) + + If present, and ``from_crawler`` is not defined, this class method is called + to create a request fingerprinter instance from a + :class:`~scrapy.settings.Settings` object. It must return a new instance of + the request fingerprinter. + +The ``fingerprint`` method of the default request fingerprinter, +:class:`scrapy.utils.request.RequestFingerprinter`, uses +:func:`scrapy.utils.request.fingerprint` with its default parameters. For some +common use cases you can use :func:`~scrapy.utils.request.fingerprint` as well +in your ``fingerprint`` method implementation: + +.. autofunction:: scrapy.utils.request.fingerprint + +For example, to take the value of a request header named ``X-ID`` into +account:: + + # my_project/settings.py + REQUEST_FINGERPRINTER_CLASS = 'my_project.utils.RequestFingerprinter' + + # my_project/utils.py + from scrapy.utils.request import fingerprint + + class RequestFingerprinter: + + def fingerprint(self, request): + return fingerprint(request, include_headers=['X-ID']) + +You can also write your own fingerprinting logic from scratch. + +However, if you do not use :func:`~scrapy.utils.request.fingerprint`, make sure +you use :class:`~weakref.WeakKeyDictionary` to cache request fingerprints: + +- Caching saves CPU by ensuring that fingerprints are calculated only once + per request, and not once per Scrapy component that needs the fingerprint + of a request. + +- Using :class:`~weakref.WeakKeyDictionary` saves memory by ensuring that + request objects do not stay in memory forever just because you have + references to them in your cache dictionary. + +For example, to take into account only the URL of a request, without any prior +URL canonicalization or taking the request method or body into account:: + + from hashlib import sha1 + from weakref import WeakKeyDictionary + + from scrapy.utils.python import to_bytes + + class RequestFingerprinter: + + cache = WeakKeyDictionary() + + def fingerprint(self, request): + if request not in self.cache: + fp = sha1() + fp.update(to_bytes(request.url)) + self.cache[request] = fp.digest() + return self.cache[request] + +If you need to be able to override the request fingerprinting for arbitrary +requests from your spider callbacks, you may implement a request fingerprinter +that reads fingerprints from :attr:`request.meta ` +when available, and then falls back to +:func:`~scrapy.utils.request.fingerprint`. For example:: + + from scrapy.utils.request import fingerprint + + class RequestFingerprinter: + + def fingerprint(self, request): + if 'fingerprint' in request.meta: + return request.meta['fingerprint'] + return fingerprint(request) + +If you need to reproduce the same fingerprinting algorithm as Scrapy PREVIOUS_VERSION +without using the deprecated ``'PREVIOUS_VERSION'`` value of the +:setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION` setting, use the following +request fingerprinter:: + + from hashlib import sha1 + from weakref import WeakKeyDictionary + + from scrapy.utils.python import to_bytes + from w3lib.url import canonicalize_url + + class RequestFingerprinter: + + cache = WeakKeyDictionary() + + def fingerprint(self, request): + if request not in self.cache: + fp = sha1() + fp.update(to_bytes(request.method)) + fp.update(to_bytes(canonicalize_url(request.url))) + fp.update(request.body or b'') + self.cache[request] = fp.digest() + return self.cache[request] + + +.. _request-fingerprint-restrictions: + +Request fingerprint restrictions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Scrapy components that use request fingerprints may impose additional +restrictions on the format of the fingerprints that your :ref:`request +fingerprinter ` generates. + +The following built-in Scrapy components have such restrictions: + +- :class:`scrapy.extensions.httpcache.FilesystemCacheStorage` (default + value of :setting:`HTTPCACHE_STORAGE`) + + Request fingerprints must be at least 1 byte long. + + Path and filename length limits of the file system of + :setting:`HTTPCACHE_DIR` also apply. Inside :setting:`HTTPCACHE_DIR`, + the following directory structure is created: + + - :attr:`Spider.name ` + + - first byte of a request fingerprint as hexadecimal + + - fingerprint as hexadecimal + + - filenames up to 16 characters long + + For example, if a request fingerprint is made of 20 bytes (default), + :setting:`HTTPCACHE_DIR` is ``'/home/user/project/.scrapy/httpcache'``, + and the name of your spider is ``'my_spider'`` your file system must + support a file path like:: + + /home/user/project/.scrapy/httpcache/my_spider/01/0123456789abcdef0123456789abcdef01234567/response_headers + +- :class:`scrapy.extensions.httpcache.DbmCacheStorage` + + The underlying DBM implementation must support keys as long as twice + the number of bytes of a request fingerprint, plus 5. For example, + if a request fingerprint is made of 20 bytes (default), + 45-character-long keys must be supported. + + .. _topics-request-meta: Request.meta special keys diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 4e105642d..2046c6446 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -825,12 +825,8 @@ Default: ``'scrapy.dupefilters.RFPDupeFilter'`` The class used to detect and filter duplicate requests. -The default (``RFPDupeFilter``) filters based on request fingerprint using -the ``scrapy.utils.request.request_fingerprint`` function. In order to change -the way duplicates are checked you could subclass ``RFPDupeFilter`` and -override its ``request_fingerprint`` method. This method should accept -scrapy :class:`~scrapy.Request` object and return its fingerprint -(a string). +The default (``RFPDupeFilter``) filters based on the +:setting:`REQUEST_FINGERPRINTER_CLASS` setting. You can disable filtering of duplicate requests by setting :setting:`DUPEFILTER_CLASS` to ``'scrapy.dupefilters.BaseDupeFilter'``. diff --git a/scrapy/crawler.py b/scrapy/crawler.py index a638254f1..fdca7b335 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -51,6 +51,7 @@ class Crawler: self.spidercls.update_settings(self.settings) self.signals = SignalManager(self) + self.stats = load_object(self.settings['STATS_CLASS'])(self) handler = LogCounterHandler(self, level=self.settings.get('LOG_LEVEL')) @@ -71,6 +72,12 @@ class Crawler: lf_cls = load_object(self.settings['LOG_FORMATTER']) self.logformatter = lf_cls.from_crawler(self) + self.request_fingerprinter = create_instance( + load_object(self.settings['REQUEST_FINGERPRINTER_CLASS']), + settings=self.settings, + crawler=self, + ) + reactor_class = self.settings.get("TWISTED_REACTOR") if init_reactor: # this needs to be done after the spider settings are merged, diff --git a/scrapy/dupefilters.py b/scrapy/dupefilters.py index 292c68099..d1b0559ef 100644 --- a/scrapy/dupefilters.py +++ b/scrapy/dupefilters.py @@ -1,14 +1,16 @@ import logging import os from typing import Optional, Set, Type, TypeVar +from warnings import warn from twisted.internet.defer import Deferred from scrapy.http.request import Request from scrapy.settings import BaseSettings from scrapy.spiders import Spider +from scrapy.utils.deprecate import ScrapyDeprecationWarning from scrapy.utils.job import job_dir -from scrapy.utils.request import referer_str, request_fingerprint +from scrapy.utils.request import referer_str, RequestFingerprinter BaseDupeFilterTV = TypeVar("BaseDupeFilterTV", bound="BaseDupeFilter") @@ -39,8 +41,15 @@ RFPDupeFilterTV = TypeVar("RFPDupeFilterTV", bound="RFPDupeFilter") class RFPDupeFilter(BaseDupeFilter): """Request Fingerprint duplicates filter""" - def __init__(self, path: Optional[str] = None, debug: bool = False) -> None: + def __init__( + self, + path: Optional[str] = None, + debug: bool = False, + *, + fingerprinter=None, + ) -> None: self.file = None + self.fingerprinter = fingerprinter or RequestFingerprinter() self.fingerprints: Set[str] = set() self.logdupes = True self.debug = debug @@ -51,9 +60,39 @@ class RFPDupeFilter(BaseDupeFilter): self.fingerprints.update(x.rstrip() for x in self.file) @classmethod - def from_settings(cls: Type[RFPDupeFilterTV], settings: BaseSettings) -> RFPDupeFilterTV: + def from_settings(cls: Type[RFPDupeFilterTV], settings: BaseSettings, *, fingerprinter=None) -> RFPDupeFilterTV: debug = settings.getbool('DUPEFILTER_DEBUG') - return cls(job_dir(settings), debug) + try: + return cls(job_dir(settings), debug, fingerprinter=fingerprinter) + except TypeError: + warn( + "RFPDupeFilter subclasses must either modify their '__init__' " + "method to support a 'fingerprinter' parameter or reimplement " + "the 'from_settings' class method.", + ScrapyDeprecationWarning, + ) + result = cls(job_dir(settings), debug) + result.fingerprinter = fingerprinter + return result + + @classmethod + def from_crawler(cls, crawler): + try: + return cls.from_settings( + crawler.settings, + fingerprinter=crawler.request_fingerprinter, + ) + except TypeError: + warn( + "RFPDupeFilter subclasses must either modify their overridden " + "'__init__' method and 'from_settings' class method to " + "support a 'fingerprinter' parameter, or reimplement the " + "'from_crawler' class method.", + ScrapyDeprecationWarning, + ) + result = cls.from_settings(crawler.settings) + result.fingerprinter = crawler.request_fingerprinter + return result def request_seen(self, request: Request) -> bool: fp = self.request_fingerprint(request) @@ -65,7 +104,7 @@ class RFPDupeFilter(BaseDupeFilter): return False def request_fingerprint(self, request: Request) -> str: - return request_fingerprint(request) + return self.fingerprinter.fingerprint(request).hex() def close(self, reason: str) -> None: if self.file: diff --git a/scrapy/extensions/httpcache.py b/scrapy/extensions/httpcache.py index d0ae29b90..c71484cfa 100644 --- a/scrapy/extensions/httpcache.py +++ b/scrapy/extensions/httpcache.py @@ -14,7 +14,6 @@ from scrapy.responsetypes import responsetypes from scrapy.utils.httpobj import urlparse_cached from scrapy.utils.project import data_path from scrapy.utils.python import to_bytes, to_unicode -from scrapy.utils.request import request_fingerprint logger = logging.getLogger(__name__) @@ -228,6 +227,8 @@ class DbmCacheStorage: logger.debug("Using DBM cache storage in %(cachepath)s", {'cachepath': dbpath}, extra={'spider': spider}) + self._fingerprinter = spider.crawler.request_fingerprinter + def close_spider(self, spider): self.db.close() @@ -244,7 +245,7 @@ class DbmCacheStorage: return response def store_response(self, spider, request, response): - key = self._request_key(request) + key = self._fingerprinter.fingerprint(request).hex() data = { 'status': response.status, 'url': response.url, @@ -255,7 +256,7 @@ class DbmCacheStorage: self.db[f'{key}_time'] = str(time()) def _read_data(self, spider, request): - key = self._request_key(request) + key = self._fingerprinter.fingerprint(request).hex() db = self.db tkey = f'{key}_time' if tkey not in db: @@ -267,9 +268,6 @@ class DbmCacheStorage: return pickle.loads(db[f'{key}_data']) - def _request_key(self, request): - return request_fingerprint(request) - class FilesystemCacheStorage: @@ -283,6 +281,8 @@ class FilesystemCacheStorage: logger.debug("Using filesystem cache storage in %(cachedir)s", {'cachedir': self.cachedir}, extra={'spider': spider}) + self._fingerprinter = spider.crawler.request_fingerprinter + def close_spider(self, spider): pass @@ -329,7 +329,7 @@ class FilesystemCacheStorage: f.write(request.body) def _get_request_path(self, spider, request): - key = request_fingerprint(request) + key = self._fingerprinter.fingerprint(request).hex() return os.path.join(self.cachedir, spider.name, key[0:2], key) def _read_meta(self, spider, request): diff --git a/scrapy/pipelines/media.py b/scrapy/pipelines/media.py index 430c37227..5308a9793 100644 --- a/scrapy/pipelines/media.py +++ b/scrapy/pipelines/media.py @@ -11,7 +11,6 @@ from scrapy.settings import Settings from scrapy.utils.datatypes import SequenceExclude from scrapy.utils.defer import mustbe_deferred, defer_result from scrapy.utils.deprecate import ScrapyDeprecationWarning -from scrapy.utils.request import request_fingerprint from scrapy.utils.misc import arg_to_iter from scrapy.utils.log import failure_to_exc_info @@ -77,6 +76,7 @@ class MediaPipeline: except AttributeError: pipe = cls() pipe.crawler = crawler + pipe._fingerprinter = crawler.request_fingerprinter return pipe def open_spider(self, spider): @@ -90,7 +90,7 @@ class MediaPipeline: return dfd.addCallback(self.item_completed, item, info) def _process_request(self, request, info, item): - fp = request_fingerprint(request) + fp = self._fingerprinter.fingerprint(request) cb = request.callback or (lambda _: _) eb = request.errback request.callback = None diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 8389a70cb..f5a3efe69 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -246,6 +246,9 @@ REDIRECT_PRIORITY_ADJUST = +2 REFERER_ENABLED = True REFERRER_POLICY = 'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy' +REQUEST_FINGERPRINTER_CLASS = 'scrapy.utils.request.RequestFingerprinter' +REQUEST_FINGERPRINTER_IMPLEMENTATION = 'PREVIOUS_VERSION' + RETRY_ENABLED = True RETRY_TIMES = 2 # initial response + 2 retries = 3 requests RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429] diff --git a/scrapy/templates/project/module/settings.py.tmpl b/scrapy/templates/project/module/settings.py.tmpl index a414b5fde..5e541e2c0 100644 --- a/scrapy/templates/project/module/settings.py.tmpl +++ b/scrapy/templates/project/module/settings.py.tmpl @@ -86,3 +86,6 @@ ROBOTSTXT_OBEY = True #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' + +# Set settings whose default value is deprecated to a future-proof value +REQUEST_FINGERPRINTER_IMPLEMENTATION = 'VERSION' diff --git a/scrapy/utils/request.py b/scrapy/utils/request.py index 70ef3ba2b..cf33317ce 100644 --- a/scrapy/utils/request.py +++ b/scrapy/utils/request.py @@ -4,7 +4,9 @@ scrapy.http.Request objects """ import hashlib -from typing import Dict, Iterable, Optional, Tuple, Union +import json +import warnings +from typing import Dict, Iterable, List, Optional, Tuple, Union from urllib.parse import urlunparse from weakref import WeakKeyDictionary @@ -12,13 +14,22 @@ from w3lib.http import basic_auth_header from w3lib.url import canonicalize_url from scrapy import Request, Spider +from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.utils.httpobj import urlparse_cached from scrapy.utils.misc import load_object from scrapy.utils.python import to_bytes, to_unicode -_fingerprint_cache: "WeakKeyDictionary[Request, Dict[Tuple[Optional[Tuple[bytes, ...]], bool], str]]" -_fingerprint_cache = WeakKeyDictionary() +_deprecated_fingerprint_cache: "WeakKeyDictionary[Request, Dict[Tuple[Optional[Tuple[bytes, ...]], bool], str]]" +_deprecated_fingerprint_cache = WeakKeyDictionary() + + +def _serialize_headers(headers, request): + for header in headers: + if header in request.headers: + yield header + for value in request.headers.getlist(header): + yield value def request_fingerprint( @@ -26,6 +37,123 @@ def request_fingerprint( include_headers: Optional[Iterable[Union[bytes, str]]] = None, keep_fragments: bool = False, ) -> str: + """ + Return the request fingerprint as an hexadecimal string. + + The request fingerprint is a hash that uniquely identifies the resource the + request points to. For example, take the following two urls: + + http://www.example.com/query?id=111&cat=222 + http://www.example.com/query?cat=222&id=111 + + Even though those are two different URLs both point to the same resource + and are equivalent (i.e. they should return the same response). + + Another example are cookies used to store session ids. Suppose the + following page is only accessible to authenticated users: + + http://www.example.com/members/offers.html + + Lots of sites use a cookie to store the session id, which adds a random + component to the HTTP Request and thus should be ignored when calculating + the fingerprint. + + For this reason, request headers are ignored by default when calculating + the fingerprint. If you want to include specific headers use the + include_headers argument, which is a list of Request headers to include. + + Also, servers usually ignore fragments in urls when handling requests, + so they are also ignored by default when calculating the fingerprint. + If you want to include them, set the keep_fragments argument to True + (for instance when handling requests with a headless browser). + """ + if include_headers or keep_fragments: + message = ( + 'Call to deprecated function ' + 'scrapy.utils.request.request_fingerprint().\n' + '\n' + 'If you are using this function in a Scrapy component because you ' + 'need a non-default fingerprinting algorithm, and you are OK ' + 'with that non-default fingerprinting algorithm being used by ' + 'all Scrapy components and not just the one calling this ' + 'function, use crawler.request_fingerprinter.fingerprint() ' + 'instead in your Scrapy component (you can get the crawler ' + 'object from the \'from_crawler\' class method), and use the ' + '\'REQUEST_FINGERPRINTER_CLASS\' setting to configure your ' + 'non-default fingerprinting algorithm.\n' + '\n' + 'Otherwise, consider using the ' + 'scrapy.utils.request.fingerprint() function instead.\n' + '\n' + 'If you switch to \'fingerprint()\', or assign the ' + '\'REQUEST_FINGERPRINTER_CLASS\' setting a class that uses ' + '\'fingerprint()\', the generated fingerprints will not only be ' + 'bytes instead of a string, but they will also be different from ' + 'those generated by \'request_fingerprint()\'. Before you switch, ' + 'make sure that you understand the consequences of this (e.g. ' + 'cache invalidation) and are OK with them; otherwise, consider ' + 'implementing your own function which returns the same ' + 'fingerprints as the deprecated \'request_fingerprint()\' function.' + ) + else: + message = ( + 'Call to deprecated function ' + 'scrapy.utils.request.request_fingerprint().\n' + '\n' + 'If you are using this function in a Scrapy component, and you ' + 'are OK with users of your component changing the fingerprinting ' + 'algorithm through settings, use ' + 'crawler.request_fingerprinter.fingerprint() instead in your ' + 'Scrapy component (you can get the crawler object from the ' + '\'from_crawler\' class method).\n' + '\n' + 'Otherwise, consider using the ' + 'scrapy.utils.request.fingerprint() function instead.\n' + '\n' + 'Either way, the resulting fingerprints will be returned as ' + 'bytes, not as a string, and they will also be different from ' + 'those generated by \'request_fingerprint()\'. Before you switch, ' + 'make sure that you understand the consequences of this (e.g. ' + 'cache invalidation) and are OK with them; otherwise, consider ' + 'implementing your own function which returns the same ' + 'fingerprints as the deprecated \'request_fingerprint()\' function.' + ) + warnings.warn(message, category=ScrapyDeprecationWarning, stacklevel=2) + processed_include_headers: Optional[Tuple[bytes, ...]] = None + if include_headers: + processed_include_headers = tuple( + to_bytes(h.lower()) for h in sorted(include_headers) + ) + cache = _deprecated_fingerprint_cache.setdefault(request, {}) + cache_key = (processed_include_headers, keep_fragments) + if cache_key not in cache: + fp = hashlib.sha1() + fp.update(to_bytes(request.method)) + fp.update(to_bytes(canonicalize_url(request.url, keep_fragments=keep_fragments))) + fp.update(request.body or b'') + if processed_include_headers: + for part in _serialize_headers(processed_include_headers, request): + fp.update(part) + cache[cache_key] = fp.hexdigest() + return cache[cache_key] + + +def _request_fingerprint_as_bytes(*args, **kwargs): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + return bytes.fromhex(request_fingerprint(*args, **kwargs)) + + +_fingerprint_cache: "WeakKeyDictionary[Request, Dict[Tuple[Optional[Tuple[bytes, ...]], bool], bytes]]" +_fingerprint_cache = WeakKeyDictionary() + + +def fingerprint( + request: Request, + *, + include_headers: Optional[Iterable[Union[bytes, str]]] = None, + keep_fragments: bool = False, +) -> bytes: """ Return the request fingerprint. @@ -43,7 +171,7 @@ def request_fingerprint( http://www.example.com/members/offers.html - Lot of sites use a cookie to store the session id, which adds a random + Lots of sites use a cookie to store the session id, which adds a random component to the HTTP Request and thus should be ignored when calculating the fingerprint. @@ -55,29 +183,96 @@ def request_fingerprint( so they are also ignored by default when calculating the fingerprint. If you want to include them, set the keep_fragments argument to True (for instance when handling requests with a headless browser). - """ - headers: Optional[Tuple[bytes, ...]] = None + processed_include_headers: Optional[Tuple[bytes, ...]] = None if include_headers: - headers = tuple(to_bytes(h.lower()) for h in sorted(include_headers)) + processed_include_headers = tuple( + to_bytes(h.lower()) for h in sorted(include_headers) + ) cache = _fingerprint_cache.setdefault(request, {}) - cache_key = (headers, keep_fragments) + cache_key = (processed_include_headers, keep_fragments) if cache_key not in cache: - fp = hashlib.sha1() - fp.update(to_bytes(request.method)) - fp.update(to_bytes(canonicalize_url(request.url, keep_fragments=keep_fragments))) - fp.update(request.body or b'') - if headers: - for hdr in headers: - if hdr in request.headers: - fp.update(hdr) - for v in request.headers.getlist(hdr): - fp.update(v) - cache[cache_key] = fp.hexdigest() + # To decode bytes reliably (JSON does not support bytes), regardless of + # character encoding, we use bytes.hex() + headers: Dict[str, List[str]] = {} + if processed_include_headers: + for header in processed_include_headers: + if header in request.headers: + headers[header.hex()] = [ + header_value.hex() + for header_value in request.headers.getlist(header) + ] + fingerprint_data = { + 'method': to_unicode(request.method), + 'url': canonicalize_url(request.url, keep_fragments=keep_fragments), + 'body': (request.body or b'').hex(), + 'headers': headers, + } + fingerprint_json = json.dumps(fingerprint_data, sort_keys=True) + cache[cache_key] = hashlib.sha1(fingerprint_json.encode()).digest() return cache[cache_key] -def request_authenticate(request: Request, username: str, password: str) -> None: +class RequestFingerprinter: + """Default fingerprinter. + + It takes into account a canonical version + (:func:`w3lib.url.canonicalize_url`) of :attr:`request.url + ` and the values of :attr:`request.method + ` and :attr:`request.body + `. It then generates an `SHA1 + `_ hash. + + .. seealso:: :setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION`. + """ + + @classmethod + def from_crawler(cls, crawler): + return cls(crawler) + + def __init__(self, crawler=None): + if crawler: + implementation = crawler.settings.get( + 'REQUEST_FINGERPRINTER_IMPLEMENTATION' + ) + else: + implementation = 'PREVIOUS_VERSION' + if implementation == 'PREVIOUS_VERSION': + message = ( + '\'PREVIOUS_VERSION\' is a deprecated value for the ' + '\'REQUEST_FINGERPRINTER_IMPLEMENTATION\' setting.\n' + '\n' + 'It is also the default value. In other words, it is normal ' + 'to get this warning if you have not defined a value for the ' + '\'REQUEST_FINGERPRINTER_IMPLEMENTATION\' setting. This is so ' + 'for backward compatibility reasons, but it will change in a ' + 'future version of Scrapy.\n' + '\n' + 'See the documentation of the ' + '\'REQUEST_FINGERPRINTER_IMPLEMENTATION\' setting for ' + 'information on how to handle this deprecation.' + ) + warnings.warn(message, category=ScrapyDeprecationWarning, stacklevel=2) + self._fingerprint = _request_fingerprint_as_bytes + elif implementation == 'VERSION': + self._fingerprint = fingerprint + else: + raise ValueError( + f'Got an invalid value on setting ' + f'\'REQUEST_FINGERPRINTER_IMPLEMENTATION\': ' + f'{implementation!r}. Valid values are \'PREVIOUS_VERSION\' (deprecated) ' + f'and \'VERSION\'.' + ) + + def fingerprint(self, request): + return self._fingerprint(request) + + +def request_authenticate( + request: Request, + username: str, + password: str, +) -> None: """Authenticate the given request (in place) using the HTTP basic access authentication mechanism (RFC 2617) and the given username and password """ diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py index 24c38283a..b90ea5009 100644 --- a/scrapy/utils/test.py +++ b/scrapy/utils/test.py @@ -54,7 +54,7 @@ def get_ftp_content_and_delete( return "".join(ftp_data) -def get_crawler(spidercls=None, settings_dict=None): +def get_crawler(spidercls=None, settings_dict=None, prevent_warnings=True): """Return an unconfigured Crawler object. If settings_dict is given, it will be used to populate the crawler settings with a project level priority. @@ -62,7 +62,12 @@ def get_crawler(spidercls=None, settings_dict=None): from scrapy.crawler import CrawlerRunner from scrapy.spiders import Spider - runner = CrawlerRunner(settings_dict) + # Set by default settings that prevent deprecation warnings. + settings = {} + if prevent_warnings: + settings['REQUEST_FINGERPRINTER_IMPLEMENTATION'] = 'VERSION' + settings.update(settings_dict or {}) + runner = CrawlerRunner(settings) return runner.create_crawler(spidercls or Spider) diff --git a/tests/test_crawler.py b/tests/test_crawler.py index 8f6227109..f7aa769e4 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -104,7 +104,8 @@ class CrawlerLoggingTestCase(unittest.TestCase): custom_settings = { 'LOG_LEVEL': 'INFO', 'LOG_FILE': log_file, - # disable telnet if not available to avoid an extra warning + # settings to avoid extra warnings + 'REQUEST_FINGERPRINTER_IMPLEMENTATION': 'VERSION', 'TELNETCONSOLE_ENABLED': telnet.TWISTED_CONCH_AVAILABLE, } diff --git a/tests/test_dupefilters.py b/tests/test_dupefilters.py index 680bb6dc8..b7df2554a 100644 --- a/tests/test_dupefilters.py +++ b/tests/test_dupefilters.py @@ -15,6 +15,16 @@ from scrapy.utils.test import get_crawler from tests.spiders import SimpleSpider +def _get_dupefilter(*, crawler=None, settings=None, open=True): + if crawler is None: + crawler = get_crawler(settings_dict=settings) + scheduler = Scheduler.from_crawler(crawler) + dupefilter = scheduler.df + if open: + dupefilter.open() + return dupefilter + + class FromCrawlerRFPDupeFilter(RFPDupeFilter): @classmethod @@ -64,9 +74,7 @@ class RFPDupeFilterTest(unittest.TestCase): self.assertEqual(scheduler.df.method, 'n/a') def test_filter(self): - dupefilter = RFPDupeFilter() - dupefilter.open() - + dupefilter = _get_dupefilter() r1 = Request('http://scrapytest.org/1') r2 = Request('http://scrapytest.org/2') r3 = Request('http://scrapytest.org/2') @@ -85,7 +93,7 @@ class RFPDupeFilterTest(unittest.TestCase): path = tempfile.mkdtemp() try: - df = RFPDupeFilter(path) + df = _get_dupefilter(settings={'JOBDIR': path}, open=False) try: df.open() assert not df.request_seen(r1) @@ -93,7 +101,8 @@ class RFPDupeFilterTest(unittest.TestCase): finally: df.close('finished') - df2 = RFPDupeFilter(path) + df2 = _get_dupefilter(settings={'JOBDIR': path}, open=False) + assert df != df2 try: df2.open() assert df2.request_seen(r1) @@ -109,26 +118,24 @@ class RFPDupeFilterTest(unittest.TestCase): output of request_seen. """ + dupefilter = _get_dupefilter() r1 = Request('http://scrapytest.org/index.html') r2 = Request('http://scrapytest.org/INDEX.html') - dupefilter = RFPDupeFilter() - dupefilter.open() - assert not dupefilter.request_seen(r1) assert not dupefilter.request_seen(r2) dupefilter.close('finished') - class CaseInsensitiveRFPDupeFilter(RFPDupeFilter): + class RequestFingerprinter: - def request_fingerprint(self, request): + def fingerprint(self, request): fp = hashlib.sha1() fp.update(to_bytes(request.url.lower())) - return fp.hexdigest() + return fp.digest() - case_insensitive_dupefilter = CaseInsensitiveRFPDupeFilter() - case_insensitive_dupefilter.open() + settings = {'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter} + case_insensitive_dupefilter = _get_dupefilter(settings=settings) assert not case_insensitive_dupefilter.request_seen(r1) assert case_insensitive_dupefilter.request_seen(r2) @@ -142,8 +149,10 @@ class RFPDupeFilterTest(unittest.TestCase): r1 = Request('http://scrapytest.org/1') path = tempfile.mkdtemp() + crawler = get_crawler(settings_dict={'JOBDIR': path}) try: - df = RFPDupeFilter(path) + scheduler = Scheduler.from_crawler(crawler) + df = scheduler.df df.open() df.request_seen(r1) df.close('finished') @@ -164,11 +173,8 @@ class RFPDupeFilterTest(unittest.TestCase): settings = {'DUPEFILTER_DEBUG': False, 'DUPEFILTER_CLASS': FromCrawlerRFPDupeFilter} crawler = get_crawler(SimpleSpider, settings_dict=settings) - scheduler = Scheduler.from_crawler(crawler) spider = SimpleSpider.from_crawler(crawler) - - dupefilter = scheduler.df - dupefilter.open() + dupefilter = _get_dupefilter(crawler=crawler) r1 = Request('http://scrapytest.org/index.html') r2 = Request('http://scrapytest.org/index.html') @@ -193,11 +199,41 @@ class RFPDupeFilterTest(unittest.TestCase): settings = {'DUPEFILTER_DEBUG': True, 'DUPEFILTER_CLASS': FromCrawlerRFPDupeFilter} crawler = get_crawler(SimpleSpider, settings_dict=settings) - scheduler = Scheduler.from_crawler(crawler) spider = SimpleSpider.from_crawler(crawler) - - dupefilter = scheduler.df - dupefilter.open() + dupefilter = _get_dupefilter(crawler=crawler) + + r1 = Request('http://scrapytest.org/index.html') + r2 = Request('http://scrapytest.org/index.html', + headers={'Referer': 'http://scrapytest.org/INDEX.html'}) + + dupefilter.log(r1, spider) + dupefilter.log(r2, spider) + + assert crawler.stats.get_value('dupefilter/filtered') == 2 + log.check_present( + ( + 'scrapy.dupefilters', + 'DEBUG', + 'Filtered duplicate request: (referer: None)' + ) + ) + log.check_present( + ( + 'scrapy.dupefilters', + 'DEBUG', + 'Filtered duplicate request: ' + ' (referer: http://scrapytest.org/INDEX.html)' + ) + ) + + dupefilter.close('finished') + + def test_log_debug_default_dupefilter(self): + with LogCapture() as log: + settings = {'DUPEFILTER_DEBUG': True} + crawler = get_crawler(SimpleSpider, settings_dict=settings) + spider = SimpleSpider.from_crawler(crawler) + dupefilter = _get_dupefilter(crawler=crawler) r1 = Request('http://scrapytest.org/index.html') r2 = Request('http://scrapytest.org/index.html', diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py index 0ff2045ed..4228173ed 100644 --- a/tests/test_pipeline_files.py +++ b/tests/test_pipeline_files.py @@ -25,6 +25,7 @@ from scrapy.pipelines.files import ( from scrapy.settings import Settings from scrapy.utils.test import ( assert_gcs_environ, + get_crawler, get_ftp_content_and_delete, get_gcs_content_and_delete, skip_if_no_boto, @@ -47,7 +48,9 @@ class FilesPipelineTestCase(unittest.TestCase): def setUp(self): self.tempdir = mkdtemp() - self.pipeline = FilesPipeline.from_settings(Settings({'FILES_STORE': self.tempdir})) + settings_dict = {'FILES_STORE': self.tempdir} + crawler = get_crawler(spidercls=None, settings_dict=settings_dict) + self.pipeline = FilesPipeline.from_crawler(crawler) self.pipeline.download_func = _mocked_download_func self.pipeline.open_spider(None) diff --git a/tests/test_pipeline_media.py b/tests/test_pipeline_media.py index a802c7cf1..84e867660 100644 --- a/tests/test_pipeline_media.py +++ b/tests/test_pipeline_media.py @@ -7,17 +7,17 @@ from twisted.python.failure import Failure from twisted.internet import reactor from twisted.internet.defer import Deferred, inlineCallbacks +from scrapy import signals from scrapy.http import Request, Response from scrapy.settings import Settings from scrapy.spiders import Spider -from scrapy.utils.deprecate import ScrapyDeprecationWarning -from scrapy.utils.request import request_fingerprint +from scrapy.pipelines.files import FileException from scrapy.pipelines.images import ImagesPipeline from scrapy.pipelines.media import MediaPipeline -from scrapy.pipelines.files import FileException +from scrapy.utils.deprecate import ScrapyDeprecationWarning from scrapy.utils.log import failure_to_exc_info from scrapy.utils.signal import disconnect_all -from scrapy import signals +from scrapy.utils.test import get_crawler try: @@ -39,11 +39,14 @@ class BaseMediaPipelineTestCase(unittest.TestCase): settings = None def setUp(self): - self.spider = Spider('media.com') - self.pipe = self.pipeline_class(download_func=_mocked_download_func, - settings=Settings(self.settings)) + spider_cls = Spider + self.spider = spider_cls('media.com') + crawler = get_crawler(spider_cls, self.settings) + self.pipe = self.pipeline_class.from_crawler(crawler) + self.pipe.download_func = _mocked_download_func self.pipe.open_spider(self.spider) self.info = self.pipe.spiderinfo + self.fingerprint = crawler.request_fingerprinter.fingerprint def tearDown(self): for name, signal in vars(signals).items(): @@ -156,7 +159,7 @@ class BaseMediaPipelineTestCase(unittest.TestCase): self.assertEqual(failure.value.__context__, def_gen_return_exc) # Let's calculate the request fingerprint and fake some runtime data... - fp = request_fingerprint(request) + fp = self.fingerprint(request) info = self.pipe.spiderinfo info.downloading.add(fp) info.waiting[fp] = [] @@ -273,7 +276,7 @@ class MediaPipelineTestCase(BaseMediaPipelineTestCase): item = dict(requests=req) # pass a single item new_item = yield self.pipe.process_item(item, self.spider) assert new_item is item - assert request_fingerprint(req) in self.info.downloaded + self.assertIn(self.fingerprint(req), self.info.downloaded) # returns iterable of Requests req1 = Request('http://url1') @@ -281,8 +284,8 @@ class MediaPipelineTestCase(BaseMediaPipelineTestCase): item = dict(requests=iter([req1, req2])) new_item = yield self.pipe.process_item(item, self.spider) assert new_item is item - assert request_fingerprint(req1) in self.info.downloaded - assert request_fingerprint(req2) in self.info.downloaded + assert self.fingerprint(req1) in self.info.downloaded + assert self.fingerprint(req2) in self.info.downloaded @inlineCallbacks def test_results_are_cached_across_multiple_items(self): @@ -298,7 +301,7 @@ class MediaPipelineTestCase(BaseMediaPipelineTestCase): item = dict(requests=req2) new_item = yield self.pipe.process_item(item, self.spider) self.assertTrue(new_item is item) - self.assertEqual(request_fingerprint(req1), request_fingerprint(req2)) + self.assertEqual(self.fingerprint(req1), self.fingerprint(req2)) self.assertEqual(new_item['results'], [(True, rsp1)]) @inlineCallbacks @@ -314,7 +317,7 @@ class MediaPipelineTestCase(BaseMediaPipelineTestCase): @inlineCallbacks def test_wait_if_request_is_downloading(self): def _check_downloading(response): - fp = request_fingerprint(req1) + fp = self.fingerprint(req1) self.assertTrue(fp in self.info.downloading) self.assertTrue(fp in self.info.waiting) self.assertTrue(fp not in self.info.downloaded) @@ -351,7 +354,7 @@ class MediaPipelineTestCase(BaseMediaPipelineTestCase): class MockedMediaPipelineDeprecatedMethods(ImagesPipeline): def __init__(self, *args, **kwargs): - super(MockedMediaPipelineDeprecatedMethods, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self._mockcalled = [] def get_media_requests(self, item, info): @@ -369,19 +372,19 @@ class MockedMediaPipelineDeprecatedMethods(ImagesPipeline): def media_to_download(self, request, info): self._mockcalled.append('media_to_download') - return super(MockedMediaPipelineDeprecatedMethods, self).media_to_download(request, info) + return super().media_to_download(request, info) def media_downloaded(self, response, request, info): self._mockcalled.append('media_downloaded') - return super(MockedMediaPipelineDeprecatedMethods, self).media_downloaded(response, request, info) + return super().media_downloaded(response, request, info) def file_downloaded(self, response, request, info): self._mockcalled.append('file_downloaded') - return super(MockedMediaPipelineDeprecatedMethods, self).file_downloaded(response, request, info) + return super().file_downloaded(response, request, info) def file_path(self, request, response=None, info=None): self._mockcalled.append('file_path') - return super(MockedMediaPipelineDeprecatedMethods, self).file_path(request, response, info) + return super().file_path(request, response, info) def thumb_path(self, request, thumb_id, response=None, info=None): self._mockcalled.append('thumb_path') @@ -393,18 +396,20 @@ class MockedMediaPipelineDeprecatedMethods(ImagesPipeline): def image_downloaded(self, response, request, info): self._mockcalled.append('image_downloaded') - return super(MockedMediaPipelineDeprecatedMethods, self).image_downloaded(response, request, info) + return super().image_downloaded(response, request, info) class MediaPipelineDeprecatedMethodsTestCase(unittest.TestCase): skip = skip_pillow def setUp(self): - self.pipe = MockedMediaPipelineDeprecatedMethods( - store_uri='store-uri', - download_func=_mocked_download_func, - settings=Settings({"IMAGES_THUMBS": {'small': (50, 50)}}) - ) + settings_dict = { + 'IMAGES_STORE': 'store-uri', + 'IMAGES_THUMBS': {'small': (50, 50)}, + } + crawler = get_crawler(spidercls=None, settings_dict=settings_dict) + self.pipe = MockedMediaPipelineDeprecatedMethods.from_crawler(crawler) + self.pipe.download_func = _mocked_download_func self.pipe.open_spider(None) self.item = dict(image_urls=['http://picsum.photos/id/1014/200/300'], images=[]) diff --git a/tests/test_utils_request.py b/tests/test_utils_request.py index 7e0049b1d..e9edfee98 100644 --- a/tests/test_utils_request.py +++ b/tests/test_utils_request.py @@ -1,73 +1,29 @@ import unittest +import warnings +from hashlib import sha1 +from typing import Dict, Mapping, Optional, Tuple, Union +from weakref import WeakKeyDictionary + +import pytest +from w3lib.url import canonicalize_url + from scrapy.http import Request +from scrapy.utils.deprecate import ScrapyDeprecationWarning +from scrapy.utils.python import to_bytes from scrapy.utils.request import ( + _deprecated_fingerprint_cache, _fingerprint_cache, + _request_fingerprint_as_bytes, + fingerprint, request_authenticate, request_fingerprint, request_httprepr, ) +from scrapy.utils.test import get_crawler class UtilsRequestTest(unittest.TestCase): - def test_request_fingerprint(self): - r1 = Request("http://www.example.com/query?id=111&cat=222") - r2 = Request("http://www.example.com/query?cat=222&id=111") - self.assertEqual(request_fingerprint(r1), request_fingerprint(r1)) - self.assertEqual(request_fingerprint(r1), request_fingerprint(r2)) - - r1 = Request('http://www.example.com/hnnoticiaj1.aspx?78132,199') - r2 = Request('http://www.example.com/hnnoticiaj1.aspx?78160,199') - self.assertNotEqual(request_fingerprint(r1), request_fingerprint(r2)) - - # make sure caching is working - self.assertEqual(request_fingerprint(r1), _fingerprint_cache[r1][(None, False)]) - - r1 = Request("http://www.example.com/members/offers.html") - r2 = Request("http://www.example.com/members/offers.html") - r2.headers['SESSIONID'] = b"somehash" - self.assertEqual(request_fingerprint(r1), request_fingerprint(r2)) - - r1 = Request("http://www.example.com/") - r2 = Request("http://www.example.com/") - r2.headers['Accept-Language'] = b'en' - r3 = Request("http://www.example.com/") - r3.headers['Accept-Language'] = b'en' - r3.headers['SESSIONID'] = b"somehash" - - self.assertEqual(request_fingerprint(r1), request_fingerprint(r2), request_fingerprint(r3)) - - self.assertEqual(request_fingerprint(r1), - request_fingerprint(r1, include_headers=['Accept-Language'])) - - self.assertNotEqual( - request_fingerprint(r1), - request_fingerprint(r2, include_headers=['Accept-Language'])) - - self.assertEqual(request_fingerprint(r3, include_headers=['accept-language', 'sessionid']), - request_fingerprint(r3, include_headers=['SESSIONID', 'Accept-Language'])) - - r1 = Request("http://www.example.com/test.html") - r2 = Request("http://www.example.com/test.html#fragment") - self.assertEqual(request_fingerprint(r1), request_fingerprint(r2)) - self.assertEqual(request_fingerprint(r1), request_fingerprint(r1, keep_fragments=True)) - self.assertNotEqual(request_fingerprint(r2), request_fingerprint(r2, keep_fragments=True)) - self.assertNotEqual(request_fingerprint(r1), request_fingerprint(r2, keep_fragments=True)) - - r1 = Request("http://www.example.com") - r2 = Request("http://www.example.com", method='POST') - r3 = Request("http://www.example.com", method='POST', body=b'request body') - - self.assertNotEqual(request_fingerprint(r1), request_fingerprint(r2)) - self.assertNotEqual(request_fingerprint(r2), request_fingerprint(r3)) - - # cached fingerprint must be cleared on request copy - r1 = Request("http://www.example.com") - fp1 = request_fingerprint(r1) - r2 = r1.replace(url="http://www.example.com/other") - fp2 = request_fingerprint(r2) - self.assertNotEqual(fp1, fp2) - def test_request_authenticate(self): r = Request("http://www.example.com") request_authenticate(r, 'someuser', 'somepass') @@ -93,5 +49,632 @@ class UtilsRequestTest(unittest.TestCase): request_httprepr(Request("ftp://localhost/tmp/foo.txt")) +class FingerprintTest(unittest.TestCase): + maxDiff = None + + function = staticmethod(fingerprint) + cache: Union[ + "WeakKeyDictionary[Request, Dict[Tuple[Optional[Tuple[bytes, ...]], bool], bytes]]", + "WeakKeyDictionary[Request, Dict[Tuple[Optional[Tuple[bytes, ...]], bool], str]]", + ] = _fingerprint_cache + default_cache_key = (None, False) + known_hashes: Tuple[Tuple[Request, Union[bytes, str], Dict], ...] = ( + ( + Request("http://example.org"), + b'xs\xd7\x0c3uj\x15\xfe\xd7d\x9b\xa9\t\xe0d\xbf\x9cXD', + {}, + ), + ( + Request("https://example.org"), + b'\xc04\x85P,\xaa\x91\x06\xf8t\xb4\xbd*\xd9\xe9\x8a:m\xc3l', + {}, + ), + ( + Request("https://example.org?a"), + b'G\xad\xb8Ck\x19\x1c\xed\x838,\x01\xc4\xde;\xee\xa5\x94a\x0c', + {}, + ), + ( + Request("https://example.org?a=b"), + b'\x024MYb\x8a\xc2\x1e\xbc>\xd6\xac*\xda\x9cF\xc1r\x7f\x17', + {}, + ), + ( + Request("https://example.org?a=b&a"), + b't+\xe8*\xfb\x84\xe3v\x1a}\x88p\xc0\xccB\xd7\x9d\xfez\x96', + {}, + ), + ( + Request("https://example.org?a=b&a=c"), + b'\xda\x1ec\xd0\x9c\x08s`\xb4\x9b\xe2\xb6R\xf8k\xef\xeaQG\xef', + {}, + ), + ( + Request("https://example.org", method='POST'), + b'\x9d\xcdA\x0fT\x02:\xca\xa0}\x90\xda\x05B\xded\x8aN7\x1d', + {}, + ), + ( + Request("https://example.org", body=b'a'), + b'\xc34z>\xd8\x99\x8b\xda7\x05r\x99I\xa8\xa0x;\xa41_', + {}, + ), + ( + Request("https://example.org", method='POST', body=b'a'), + b'5`\xe2y4\xd0\x9d\xee\xe0\xbatw\x87Q\xe8O\xd78\xfc\xe7', + {}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + b'\xc04\x85P,\xaa\x91\x06\xf8t\xb4\xbd*\xd9\xe9\x8a:m\xc3l', + {}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + b']\xc7\x1f\xf2\xafG2\xbc\xa4\xfa\x99\n33\xda\x18\x94\x81U.', + {'include_headers': ['A']}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + b'<\x1a\xeb\x85y\xdeW\xfb\xdcq\x88\xee\xaf\x17\xdd\x0c\xbfH\x18\x1f', + {'keep_fragments': True}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + b'\xc1\xef~\x94\x9bS\xc1\x83\t\xdcz8\x9f\xdc{\x11\x16I.\x11', + {'include_headers': ['A'], 'keep_fragments': True}, + ), + ( + Request("https://example.org/ab"), + b'N\xe5l\xb8\x12@iw\xe2\xf3\x1bp\xea\xffp!u\xe2\x8a\xc6', + {}, + ), + ( + Request("https://example.org/a", body=b'b'), + b'_NOv\xbco$6\xfcW\x9f\xb24g\x9f\xbb\xdd\xa82\xc5', + {}, + ), + ) + + def test_query_string_key_order(self): + r1 = Request("http://www.example.com/query?id=111&cat=222") + r2 = Request("http://www.example.com/query?cat=222&id=111") + self.assertEqual(self.function(r1), self.function(r1)) + self.assertEqual(self.function(r1), self.function(r2)) + + def test_query_string_key_without_value(self): + r1 = Request('http://www.example.com/hnnoticiaj1.aspx?78132,199') + r2 = Request('http://www.example.com/hnnoticiaj1.aspx?78160,199') + self.assertNotEqual(self.function(r1), self.function(r2)) + + def test_caching(self): + r1 = Request('http://www.example.com/hnnoticiaj1.aspx?78160,199') + self.assertEqual( + self.function(r1), + self.cache[r1][self.default_cache_key] + ) + + def test_header(self): + r1 = Request("http://www.example.com/members/offers.html") + r2 = Request("http://www.example.com/members/offers.html") + r2.headers['SESSIONID'] = b"somehash" + self.assertEqual(self.function(r1), self.function(r2)) + + def test_headers(self): + r1 = Request("http://www.example.com/") + r2 = Request("http://www.example.com/") + r2.headers['Accept-Language'] = b'en' + r3 = Request("http://www.example.com/") + r3.headers['Accept-Language'] = b'en' + r3.headers['SESSIONID'] = b"somehash" + + self.assertEqual(self.function(r1), self.function(r2), self.function(r3)) + + self.assertEqual(self.function(r1), + self.function(r1, include_headers=['Accept-Language'])) + + self.assertNotEqual( + self.function(r1), + self.function(r2, include_headers=['Accept-Language'])) + + self.assertEqual(self.function(r3, include_headers=['accept-language', 'sessionid']), + self.function(r3, include_headers=['SESSIONID', 'Accept-Language'])) + + def test_fragment(self): + r1 = Request("http://www.example.com/test.html") + r2 = Request("http://www.example.com/test.html#fragment") + self.assertEqual(self.function(r1), self.function(r2)) + self.assertEqual(self.function(r1), self.function(r1, keep_fragments=True)) + self.assertNotEqual(self.function(r2), self.function(r2, keep_fragments=True)) + self.assertNotEqual(self.function(r1), self.function(r2, keep_fragments=True)) + + def test_method_and_body(self): + r1 = Request("http://www.example.com") + r2 = Request("http://www.example.com", method='POST') + r3 = Request("http://www.example.com", method='POST', body=b'request body') + + self.assertNotEqual(self.function(r1), self.function(r2)) + self.assertNotEqual(self.function(r2), self.function(r3)) + + def test_request_replace(self): + # cached fingerprint must be cleared on request copy + r1 = Request("http://www.example.com") + fp1 = self.function(r1) + r2 = r1.replace(url="http://www.example.com/other") + fp2 = self.function(r2) + self.assertNotEqual(fp1, fp2) + + def test_part_separation(self): + # An old implementation used to serialize request data in a way that + # would put the body right after the URL. + r1 = Request("http://www.example.com/foo") + fp1 = self.function(r1) + r2 = Request("http://www.example.com/f", body=b'oo') + fp2 = self.function(r2) + self.assertNotEqual(fp1, fp2) + + def test_hashes(self): + """Test hardcoded hashes, to make sure future changes to not introduce + backward incompatibilities.""" + actual = [ + self.function(request, **kwargs) + for request, _, kwargs in self.known_hashes + ] + expected = [ + _fingerprint + for _, _fingerprint, _ in self.known_hashes + ] + self.assertEqual(actual, expected) + + +class RequestFingerprintTest(FingerprintTest): + function = staticmethod(request_fingerprint) + cache = _deprecated_fingerprint_cache + known_hashes: Tuple[Tuple[Request, Union[bytes, str], Dict], ...] = ( + ( + Request("http://example.org"), + 'b2e5245ef826fd9576c93bd6e392fce3133fab62', + {}, + ), + ( + Request("https://example.org"), + 'bd10a0a89ea32cdee77917320f1309b0da87e892', + {}, + ), + ( + Request("https://example.org?a"), + '2fb7d48ae02f04b749f40caa969c0bc3c43204ce', + {}, + ), + ( + Request("https://example.org?a=b"), + '42e5fe149b147476e3f67ad0670c57b4cc57856a', + {}, + ), + ( + Request("https://example.org?a=b&a"), + 'd23a9787cb56c6375c2cae4453c5a8c634526942', + {}, + ), + ( + Request("https://example.org?a=b&a=c"), + '9a18a7a8552a9182b7f1e05d33876409e421e5c5', + {}, + ), + ( + Request("https://example.org", method='POST'), + 'ba20a80cb5c5ca460021ceefb3c2467b2bfd1bc6', + {}, + ), + ( + Request("https://example.org", body=b'a'), + '4bb136e54e715a4ea7a9dd1101831765d33f2d60', + {}, + ), + ( + Request("https://example.org", method='POST', body=b'a'), + '6c6595374a304b293be762f7b7be3f54e9947c65', + {}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + 'bd10a0a89ea32cdee77917320f1309b0da87e892', + {}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + '515b633cb3ca502a33a9d8c890e889ec1e425e65', + {'include_headers': ['A']}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + '505c96e7da675920dfef58725e8c957dfdb38f47', + {'keep_fragments': True}, + ), + ( + Request("https://example.org#a", headers={'A': b'B'}), + 'd6f673cdcb661b7970c2b9a00ee63e87d1e2e5da', + {'include_headers': ['A'], 'keep_fragments': True}, + ), + ( + Request("https://example.org/ab"), + '4e2870fee58582d6f81755e9b8fdefe3cba0c951', + {}, + ), + ( + Request("https://example.org/a", body=b'b'), + '4e2870fee58582d6f81755e9b8fdefe3cba0c951', + {}, + ), + ) + + @pytest.mark.xfail(reason='known bug kept for backward compatibility', strict=True) + def test_part_separation(self): + super().test_part_separation() + + def test_deprecation_default_parameters(self): + with pytest.warns(ScrapyDeprecationWarning) as warnings: + self.function(Request("http://www.example.com")) + messages = [str(warning.message) for warning in warnings] + self.assertTrue( + any( + 'Call to deprecated function' in message + for message in messages + ) + ) + self.assertFalse(any('non-default' in message for message in messages)) + + def test_deprecation_non_default_parameters(self): + with pytest.warns(ScrapyDeprecationWarning) as warnings: + self.function(Request("http://www.example.com"), keep_fragments=True) + messages = [str(warning.message) for warning in warnings] + self.assertTrue( + any( + 'Call to deprecated function' in message + for message in messages + ) + ) + self.assertTrue(any('non-default' in message for message in messages)) + + +class RequestFingerprintAsBytesTest(FingerprintTest): + function = staticmethod(_request_fingerprint_as_bytes) + cache = _deprecated_fingerprint_cache + known_hashes = RequestFingerprintTest.known_hashes + + def test_caching(self): + r1 = Request('http://www.example.com/hnnoticiaj1.aspx?78160,199') + self.assertEqual( + self.function(r1), + bytes.fromhex(self.cache[r1][self.default_cache_key]) + ) + + @pytest.mark.xfail(reason='known bug kept for backward compatibility', strict=True) + def test_part_separation(self): + super().test_part_separation() + + def test_hashes(self): + actual = [ + self.function(request, **kwargs) + for request, _, kwargs in self.known_hashes + ] + expected = [ + bytes.fromhex(_fingerprint) + for _, _fingerprint, _ in self.known_hashes + ] + self.assertEqual(actual, expected) + + +_fingerprint_cache_2_6: Mapping[Request, Tuple[None, bool]] = WeakKeyDictionary() + + +def request_fingerprint_2_6(request, include_headers=None, keep_fragments=False): + if include_headers: + include_headers = tuple(to_bytes(h.lower()) for h in sorted(include_headers)) + cache = _fingerprint_cache_2_6.setdefault(request, {}) + cache_key = (include_headers, keep_fragments) + if cache_key not in cache: + fp = sha1() + fp.update(to_bytes(request.method)) + fp.update(to_bytes(canonicalize_url(request.url, keep_fragments=keep_fragments))) + fp.update(request.body or b'') + if include_headers: + for hdr in include_headers: + if hdr in request.headers: + fp.update(hdr) + for v in request.headers.getlist(hdr): + fp.update(v) + cache[cache_key] = fp.hexdigest() + return cache[cache_key] + + +REQUEST_OBJECTS_TO_TEST = ( + Request("http://www.example.com/"), + Request("http://www.example.com/query?id=111&cat=222"), + Request("http://www.example.com/query?cat=222&id=111"), + Request('http://www.example.com/hnnoticiaj1.aspx?78132,199'), + Request('http://www.example.com/hnnoticiaj1.aspx?78160,199'), + Request("http://www.example.com/members/offers.html"), + Request( + "http://www.example.com/members/offers.html", + headers={'SESSIONID': b"somehash"}, + ), + Request( + "http://www.example.com/", + headers={'Accept-Language': b"en"}, + ), + Request( + "http://www.example.com/", + headers={ + 'Accept-Language': b"en", + 'SESSIONID': b"somehash", + }, + ), + Request("http://www.example.com/test.html"), + Request("http://www.example.com/test.html#fragment"), + Request("http://www.example.com", method='POST'), + Request("http://www.example.com", method='POST', body=b'request body'), +) + + +class BackwardCompatibilityTestCase(unittest.TestCase): + + def test_function_backward_compatibility(self): + include_headers_to_test = ( + None, + ['Accept-Language'], + ['accept-language', 'sessionid'], + ['SESSIONID', 'Accept-Language'], + ) + for request_object in REQUEST_OBJECTS_TO_TEST: + for include_headers in include_headers_to_test: + for keep_fragments in (False, True): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + fp = request_fingerprint( + request_object, + include_headers=include_headers, + keep_fragments=keep_fragments, + ) + old_fp = request_fingerprint_2_6( + request_object, + include_headers=include_headers, + keep_fragments=keep_fragments, + ) + self.assertEqual(fp, old_fp) + + def test_component_backward_compatibility(self): + for request_object in REQUEST_OBJECTS_TO_TEST: + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + crawler = get_crawler(prevent_warnings=False) + fp = crawler.request_fingerprinter.fingerprint(request_object) + old_fp = request_fingerprint_2_6(request_object) + self.assertEqual(fp.hex(), old_fp) + + def test_custom_component_backward_compatibility(self): + """Tests that the backward-compatible request fingerprinting class featured + in the documentation is indeed backward compatible and does not cause a + warning to be logged.""" + + class RequestFingerprinter: + + cache = WeakKeyDictionary() + + def fingerprint(self, request): + if request not in self.cache: + fp = sha1() + fp.update(to_bytes(request.method)) + fp.update(to_bytes(canonicalize_url(request.url))) + fp.update(request.body or b'') + self.cache[request] = fp.digest() + return self.cache[request] + + for request_object in REQUEST_OBJECTS_TO_TEST: + with warnings.catch_warnings() as logged_warnings: + settings = { + 'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter, + } + crawler = get_crawler(settings_dict=settings) + fp = crawler.request_fingerprinter.fingerprint(request_object) + old_fp = request_fingerprint_2_6(request_object) + self.assertEqual(fp.hex(), old_fp) + self.assertFalse(logged_warnings) + + +class RequestFingerprinterTestCase(unittest.TestCase): + + def test_default_implementation(self): + with warnings.catch_warnings(record=True) as logged_warnings: + crawler = get_crawler(prevent_warnings=False) + request = Request('https://example.com') + self.assertEqual( + crawler.request_fingerprinter.fingerprint(request), + _request_fingerprint_as_bytes(request), + ) + self.assertTrue(logged_warnings) + + def test_deprecated_implementation(self): + settings = { + 'REQUEST_FINGERPRINTER_IMPLEMENTATION': 'PREVIOUS_VERSION', + } + with warnings.catch_warnings(record=True) as logged_warnings: + crawler = get_crawler(settings_dict=settings) + request = Request('https://example.com') + self.assertEqual( + crawler.request_fingerprinter.fingerprint(request), + _request_fingerprint_as_bytes(request), + ) + self.assertTrue(logged_warnings) + + def test_recommended_implementation(self): + settings = { + 'REQUEST_FINGERPRINTER_IMPLEMENTATION': 'VERSION', + } + with warnings.catch_warnings(record=True) as logged_warnings: + crawler = get_crawler(settings_dict=settings) + request = Request('https://example.com') + self.assertEqual( + crawler.request_fingerprinter.fingerprint(request), + fingerprint(request), + ) + self.assertFalse(logged_warnings) + + def test_unknown_implementation(self): + settings = { + 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.5', + } + with self.assertRaises(ValueError): + get_crawler(settings_dict=settings) + + +class CustomRequestFingerprinterTestCase(unittest.TestCase): + + def test_include_headers(self): + + class RequestFingerprinter: + + def fingerprint(self, request): + return fingerprint(request, include_headers=['X-ID']) + + settings = { + 'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter, + } + crawler = get_crawler(settings_dict=settings) + + r1 = Request("http://www.example.com", headers={'X-ID': '1'}) + fp1 = crawler.request_fingerprinter.fingerprint(r1) + r2 = Request("http://www.example.com", headers={'X-ID': '2'}) + fp2 = crawler.request_fingerprinter.fingerprint(r2) + self.assertNotEqual(fp1, fp2) + + def test_dont_canonicalize(self): + + class RequestFingerprinter: + cache = WeakKeyDictionary() + + def fingerprint(self, request): + if request not in self.cache: + fp = sha1() + fp.update(to_bytes(request.url)) + self.cache[request] = fp.digest() + return self.cache[request] + + settings = { + 'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter, + } + crawler = get_crawler(settings_dict=settings) + + r1 = Request("http://www.example.com?a=1&a=2") + fp1 = crawler.request_fingerprinter.fingerprint(r1) + r2 = Request("http://www.example.com?a=2&a=1") + fp2 = crawler.request_fingerprinter.fingerprint(r2) + self.assertNotEqual(fp1, fp2) + + def test_meta(self): + + class RequestFingerprinter: + + def fingerprint(self, request): + if 'fingerprint' in request.meta: + return request.meta['fingerprint'] + return fingerprint(request) + + settings = { + 'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter, + } + crawler = get_crawler(settings_dict=settings) + + r1 = Request("http://www.example.com") + fp1 = crawler.request_fingerprinter.fingerprint(r1) + r2 = Request("http://www.example.com", meta={'fingerprint': 'a'}) + fp2 = crawler.request_fingerprinter.fingerprint(r2) + r3 = Request("http://www.example.com", meta={'fingerprint': 'a'}) + fp3 = crawler.request_fingerprinter.fingerprint(r3) + r4 = Request("http://www.example.com", meta={'fingerprint': 'b'}) + fp4 = crawler.request_fingerprinter.fingerprint(r4) + self.assertNotEqual(fp1, fp2) + self.assertNotEqual(fp1, fp4) + self.assertNotEqual(fp2, fp4) + self.assertEqual(fp2, fp3) + + def test_from_crawler(self): + + class RequestFingerprinter: + + @classmethod + def from_crawler(cls, crawler): + return cls(crawler) + + def __init__(self, crawler): + self._fingerprint = crawler.settings['FINGERPRINT'] + + def fingerprint(self, request): + return self._fingerprint + + settings = { + 'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter, + 'FINGERPRINT': b'fingerprint', + } + crawler = get_crawler(settings_dict=settings) + + request = Request("http://www.example.com") + fingerprint = crawler.request_fingerprinter.fingerprint(request) + self.assertEqual(fingerprint, settings['FINGERPRINT']) + + def test_from_settings(self): + + class RequestFingerprinter: + + @classmethod + def from_settings(cls, settings): + return cls(settings) + + def __init__(self, settings): + self._fingerprint = settings['FINGERPRINT'] + + def fingerprint(self, request): + return self._fingerprint + + settings = { + 'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter, + 'FINGERPRINT': b'fingerprint', + } + crawler = get_crawler(settings_dict=settings) + + request = Request("http://www.example.com") + fingerprint = crawler.request_fingerprinter.fingerprint(request) + self.assertEqual(fingerprint, settings['FINGERPRINT']) + + def test_from_crawler_and_settings(self): + + class RequestFingerprinter: + + # This method is ignored due to the presence of from_crawler + @classmethod + def from_settings(cls, settings): + return cls(settings) + + @classmethod + def from_crawler(cls, crawler): + return cls(crawler) + + def __init__(self, crawler): + self._fingerprint = crawler.settings['FINGERPRINT'] + + def fingerprint(self, request): + return self._fingerprint + + settings = { + 'REQUEST_FINGERPRINTER_CLASS': RequestFingerprinter, + 'FINGERPRINT': b'fingerprint', + } + crawler = get_crawler(settings_dict=settings) + + request = Request("http://www.example.com") + fingerprint = crawler.request_fingerprinter.fingerprint(request) + self.assertEqual(fingerprint, settings['FINGERPRINT']) + + if __name__ == "__main__": unittest.main() From 407562b38b6ab375ae650c8799bdd511025527f4 Mon Sep 17 00:00:00 2001 From: Laerte Pereira <5853172+Laerte@users.noreply.github.com> Date: Thu, 9 Jun 2022 00:25:03 -0300 Subject: [PATCH 119/126] Drop Python 3.6 support (#5514) * chore: Drop Python 3.6 support * Attend PR comments * Tweak versions * Update dependencies version * fix: Ubuntu workflow * fix windows workflow * chore: Remove comment * update `install_requires` dependencies versions * move lxml to main pinned requirements * Attend code-review comments * remove non-pinned 3.7 from windows workflow * simplify condition * lint * remove paragraph * refactor * remove leftover --- .github/workflows/checks.yml | 2 +- .github/workflows/tests-macos.yml | 2 +- .github/workflows/tests-ubuntu.yml | 11 ++++------- .github/workflows/tests-windows.yml | 5 +---- README.rst | 2 +- docs/contributing.rst | 10 +++++----- docs/intro/install.rst | 4 ++-- docs/topics/items.rst | 5 ----- docs/topics/media-pipeline.rst | 2 +- scrapy/__init__.py | 4 ++-- scrapy/utils/py36.py | 11 ----------- setup.py | 19 ++++++------------- tests/requirements.txt | 4 +--- tests/test_utils_python.py | 8 +------- tox.ini | 23 ++++++++--------------- 15 files changed, 34 insertions(+), 78 deletions(-) delete mode 100644 scrapy/utils/py36.py diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 98fa44c7f..b26f344ff 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -19,7 +19,7 @@ jobs: - python-version: 3.8 env: TOXENV: pylint - - python-version: 3.6 + - python-version: 3.7 env: TOXENV: typing - python-version: "3.10" # Keep in sync with .readthedocs.yml diff --git a/.github/workflows/tests-macos.yml b/.github/workflows/tests-macos.yml index 3aaf688c7..7819a4e12 100644 --- a/.github/workflows/tests-macos.yml +++ b/.github/workflows/tests-macos.yml @@ -7,7 +7,7 @@ jobs: strategy: fail-fast: false matrix: - python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"] + python-version: ["3.7", "3.8", "3.9", "3.10"] steps: - uses: actions/checkout@v2 diff --git a/.github/workflows/tests-ubuntu.yml b/.github/workflows/tests-ubuntu.yml index 1fc8d914b..be40c7c71 100644 --- a/.github/workflows/tests-ubuntu.yml +++ b/.github/workflows/tests-ubuntu.yml @@ -8,9 +8,6 @@ jobs: fail-fast: false matrix: include: - - python-version: 3.7 - env: - TOXENV: py - python-version: 3.8 env: TOXENV: py @@ -26,19 +23,19 @@ jobs: - python-version: pypy3 env: TOXENV: pypy3 - PYPY_VERSION: 3.6-v7.3.3 + PYPY_VERSION: 3.9-v7.3.9 # pinned deps - - python-version: 3.6.12 + - python-version: 3.7.13 env: TOXENV: pinned - - python-version: 3.6.12 + - python-version: 3.7.13 env: TOXENV: asyncio-pinned - python-version: pypy3 env: TOXENV: pypy3-pinned - PYPY_VERSION: 3.6-v7.2.0 + PYPY_VERSION: 3.7-v7.3.5 # extras # extra-deps includes reppy, which does not support Python 3.9 diff --git a/.github/workflows/tests-windows.yml b/.github/workflows/tests-windows.yml index ab7385118..955b9b449 100644 --- a/.github/workflows/tests-windows.yml +++ b/.github/workflows/tests-windows.yml @@ -8,12 +8,9 @@ jobs: fail-fast: false matrix: include: - - python-version: 3.6 - env: - TOXENV: windows-pinned - python-version: 3.7 env: - TOXENV: py + TOXENV: windows-pinned - python-version: 3.8 env: TOXENV: py diff --git a/README.rst b/README.rst index 6b563d638..b543a30f4 100644 --- a/README.rst +++ b/README.rst @@ -57,7 +57,7 @@ including a list of features. Requirements ============ -* Python 3.6+ +* Python 3.7+ * Works on Linux, Windows, macOS, BSD Install diff --git a/docs/contributing.rst b/docs/contributing.rst index 4d2580a6c..946bdc23e 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -232,15 +232,15 @@ To run a specific test (say ``tests/test_loader.py``) use: To run the tests on a specific :doc:`tox ` environment, use ``-e `` with an environment name from ``tox.ini``. For example, to run -the tests with Python 3.6 use:: +the tests with Python 3.7 use:: - tox -e py36 + tox -e py37 You can also specify a comma-separated list of environments, and use :ref:`tox’s parallel mode ` to run the tests on multiple environments in parallel:: - tox -e py36,py38 -p auto + tox -e py37,py38 -p auto To pass command-line options to :doc:`pytest `, add them after ``--`` in your call to :doc:`tox `. Using ``--`` overrides the @@ -250,9 +250,9 @@ default positional arguments (``scrapy tests``) after ``--`` as well:: tox -- scrapy tests -x # stop after first failure You can also use the `pytest-xdist`_ plugin. For example, to run all tests on -the Python 3.6 :doc:`tox ` environment using all your CPU cores:: +the Python 3.7 :doc:`tox ` environment using all your CPU cores:: - tox -e py36 -- scrapy tests -n auto + tox -e py37 -- scrapy tests -n auto To see coverage report install :doc:`coverage ` (``pip install coverage``) and run: diff --git a/docs/intro/install.rst b/docs/intro/install.rst index b8d3a16bc..1f01c068d 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -9,8 +9,8 @@ Installation guide Supported Python versions ========================= -Scrapy requires Python 3.6+, either the CPython implementation (default) or -the PyPy 7.2.0+ implementation (see :ref:`python:implementations`). +Scrapy requires Python 3.7+, either the CPython implementation (default) or +the PyPy 7.3.5+ implementation (see :ref:`python:implementations`). .. _intro-install-scrapy: diff --git a/docs/topics/items.rst b/docs/topics/items.rst index 7cd482d07..167014381 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -102,11 +102,6 @@ Additionally, ``dataclass`` items also allow to: * define custom field metadata through :func:`dataclasses.field`, which can be used to :ref:`customize serialization `. -They work natively in Python 3.7 or later, or using the `dataclasses -backport`_ in Python 3.6. - -.. _dataclasses backport: https://pypi.org/project/dataclasses/ - Example:: from dataclasses import dataclass diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index 2513faae2..0925e6bb5 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -70,7 +70,7 @@ The advantage of using the :class:`ImagesPipeline` for image files is that you can configure some extra functions like generating thumbnails and filtering the images based on their size. -The Images Pipeline requires Pillow_ 4.0.0 or greater. It is used for +The Images Pipeline requires Pillow_ 7.1.0 or greater. It is used for thumbnailing and normalizing images to JPEG/RGB format. .. _Pillow: https://github.com/python-pillow/Pillow diff --git a/scrapy/__init__.py b/scrapy/__init__.py index 396f98219..86e584396 100644 --- a/scrapy/__init__.py +++ b/scrapy/__init__.py @@ -28,8 +28,8 @@ twisted_version = (_txv.major, _txv.minor, _txv.micro) # Check minimum required Python version -if sys.version_info < (3, 6): - print(f"Scrapy {__version__} requires Python 3.6+") +if sys.version_info < (3, 7): + print(f"Scrapy {__version__} requires Python 3.7+") sys.exit(1) diff --git a/scrapy/utils/py36.py b/scrapy/utils/py36.py deleted file mode 100644 index 653e2bbbb..000000000 --- a/scrapy/utils/py36.py +++ /dev/null @@ -1,11 +0,0 @@ -import warnings - -from scrapy.exceptions import ScrapyDeprecationWarning -from scrapy.utils.asyncgen import collect_asyncgen # noqa: F401 - - -warnings.warn( - "Module `scrapy.utils.py36` is deprecated, please import from `scrapy.utils.asyncgen` instead.", - category=ScrapyDeprecationWarning, - stacklevel=2, -) diff --git a/setup.py b/setup.py index d86c0f285..ed197273f 100644 --- a/setup.py +++ b/setup.py @@ -19,35 +19,29 @@ def has_environment_marker_platform_impl_support(): install_requires = [ - 'Twisted>=17.9.0', - 'cryptography>=2.0', + 'Twisted>=18.9.0', + 'cryptography>=2.8', 'cssselect>=0.9.1', 'itemloaders>=1.0.1', 'parsel>=1.5.0', - 'pyOpenSSL>=16.2.0', + 'pyOpenSSL>=19.1.0', 'queuelib>=1.4.2', 'service_identity>=16.0.0', 'w3lib>=1.17.0', - 'zope.interface>=4.1.3', + 'zope.interface>=5.1.0', 'protego>=0.1.15', 'itemadapter>=0.1.0', 'setuptools', 'tldextract', + 'lxml>=4.3.0', ] extras_require = {} cpython_dependencies = [ - 'lxml>=3.5.0', 'PyDispatcher>=2.0.5', ] if has_environment_marker_platform_impl_support(): extras_require[':platform_python_implementation == "CPython"'] = cpython_dependencies extras_require[':platform_python_implementation == "PyPy"'] = [ - # Earlier lxml versions are affected by - # https://foss.heptapod.net/pypy/pypy/-/issues/2498, - # which was fixed in Cython 0.26, released on 2017-06-19, and used to - # generate the C headers of lxml release tarballs published since then, the - # first of which was: - 'lxml>=4.0.0', 'PyPyDispatcher>=2.1.0', ] else: @@ -84,7 +78,6 @@ setup( 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', @@ -95,7 +88,7 @@ setup( 'Topic :: Software Development :: Libraries :: Application Frameworks', 'Topic :: Software Development :: Libraries :: Python Modules', ], - python_requires='>=3.6', + python_requires='>=3.7', install_requires=install_requires, extras_require=extras_require, ) diff --git a/tests/requirements.txt b/tests/requirements.txt index d2a8aae1b..d9373dfa8 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,14 +1,12 @@ # Tests requirements attrs -dataclasses; python_version == '3.6' pyftpdlib pytest pytest-cov==3.0.0 pytest-xdist sybil >= 1.3.0 # https://github.com/cjw296/sybil/issues/20#issuecomment-605433422 testfixtures -uvloop < 0.15.0; platform_system != "Windows" and python_version == '3.6' -uvloop; platform_system != "Windows" and python_version > '3.6' +uvloop; platform_system != "Windows" # optional for shell wrapper tests bpython diff --git a/tests/test_utils_python.py b/tests/test_utils_python.py index 4b3964154..7dec5624a 100644 --- a/tests/test_utils_python.py +++ b/tests/test_utils_python.py @@ -3,7 +3,6 @@ import gc import operator import platform import unittest -from datetime import datetime from itertools import count from warnings import catch_warnings, filterwarnings @@ -224,12 +223,7 @@ class UtilsPythonTestCase(unittest.TestCase): elif platform.python_implementation() == 'PyPy': self.assertEqual(get_func_args(str.split, stripself=True), ['sep', 'maxsplit']) self.assertEqual(get_func_args(operator.itemgetter(2), stripself=True), ['obj']) - - build_date = datetime.strptime(platform.python_build()[1], '%b %d %Y') - if build_date >= datetime(2020, 4, 7): # PyPy 3.6-v7.3.1 - self.assertEqual(get_func_args(" ".join, stripself=True), ['iterable']) - else: - self.assertEqual(get_func_args(" ".join, stripself=True), ['list']) + self.assertEqual(get_func_args(" ".join, stripself=True), ['iterable']) def test_without_none_values(self): self.assertEqual(without_none_values([1, None, 3, 4]), [1, 3, 4]) diff --git a/tox.ini b/tox.ini index 6951b6d16..ab8a715c2 100644 --- a/tox.ini +++ b/tox.ini @@ -11,15 +11,13 @@ minversion = 1.7.0 deps = -rtests/requirements.txt # mitmproxy does not support PyPy - # mitmproxy does not support Windows when running Python < 3.7 # Python 3.9+ requires mitmproxy >= 5.3.0 # mitmproxy >= 5.3.0 requires h2 >= 4.0, Twisted 21.2 requires h2 < 4.0 #mitmproxy >= 5.3.0; python_version >= '3.9' and implementation_name != 'pypy' # The tests hang with mitmproxy 8.0.0: https://github.com/scrapy/scrapy/issues/5454 - mitmproxy >= 4.0.4, < 8; python_version >= '3.7' and python_version < '3.9' and implementation_name != 'pypy' - mitmproxy >= 4.0.4, < 5; python_version >= '3.6' and python_version < '3.7' and platform_system != 'Windows' and implementation_name != 'pypy' + mitmproxy >= 4.0.4, < 8; python_version < '3.9' and implementation_name != 'pypy' # newer markupsafe is incompatible with deps of old mitmproxy (which we get on Python 3.7 and lower) - markupsafe < 2.1.0; python_version >= '3.6' and python_version < '3.8' and implementation_name != 'pypy' + markupsafe < 2.1.0; python_version < '3.8' and implementation_name != 'pypy' # Extras botocore>=1.4.87 passenv = @@ -44,7 +42,6 @@ deps = types-pyOpenSSL==20.0.3 types-setuptools==57.0.0 commands = - pip install types-dataclasses # remove once py36 support is dropped mypy --show-error-codes {posargs: scrapy tests} [testenv:security] @@ -75,18 +72,19 @@ commands = [pinned] deps = - cryptography==2.0 + cryptography==2.8 cssselect==0.9.1 h2==3.0 itemadapter==0.1.0 parsel==1.5.0 Protego==0.1.15 - pyOpenSSL==16.2.0 + pyOpenSSL==19.1.0 queuelib==1.4.2 service_identity==16.0.0 - Twisted[http2]==17.9.0 + Twisted[http2]==18.9.0 w3lib==1.17.0 - zope.interface==4.1.3 + zope.interface==5.1.0 + lxml==4.3.0 -rtests/requirements.txt # mitmproxy 4.0.4+ requires upgrading some of the pinned dependencies @@ -95,7 +93,7 @@ deps = # Extras botocore==1.4.87 google-cloud-storage==1.29.0 - Pillow==4.0.0 + Pillow==7.1.0 setenv = _SCRAPY_PINNED=true install_command = @@ -104,7 +102,6 @@ install_command = [testenv:pinned] deps = {[pinned]deps} - lxml==3.5.0 PyDispatcher==2.0.5 install_command = {[pinned]install_command} setenv = @@ -114,9 +111,6 @@ setenv = basepython = python3 deps = {[pinned]deps} - # First lxml version that includes a Windows wheel for Python 3.6, so we do - # not need to build lxml from sources in a CI Windows job: - lxml==3.8.0 PyDispatcher==2.0.5 install_command = {[pinned]install_command} setenv = @@ -155,7 +149,6 @@ commands = basepython = {[testenv:pypy3]basepython} deps = {[pinned]deps} - lxml==4.0.0 PyPyDispatcher==2.1.0 commands = {[testenv:pypy3]commands} install_command = {[pinned]install_command} From 2e6721fd86e3bd00301f8cd3ceb4175b2f395017 Mon Sep 17 00:00:00 2001 From: Laerte Pereira <5853172+Laerte@users.noreply.github.com> Date: Thu, 9 Jun 2022 08:37:01 -0300 Subject: [PATCH 120/126] docs: Update minimal versions that Scrapy is tested against --- docs/intro/install.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 1f01c068d..23c3af74b 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -54,9 +54,9 @@ Scrapy is written in pure Python and depends on a few key Python packages (among The minimal versions which Scrapy is tested against are: -* Twisted 14.0 -* lxml 3.4 -* pyOpenSSL 0.14 +* Twisted 18.9.0 +* lxml 4.3.0 +* pyOpenSSL 19.1.0 Scrapy may work with older versions of these packages but it is not guaranteed it will continue working From 6770d1ec62012fcfe8a36fdebeeb89cb5157c2df Mon Sep 17 00:00:00 2001 From: Laerte Pereira Date: Thu, 9 Jun 2022 09:08:09 -0300 Subject: [PATCH 121/126] chore(tests): Remove validations for unsupported modules versions --- tests/test_downloadermiddleware.py | 20 +------------------- tests/test_utils_signal.py | 20 -------------------- tests/test_webclient.py | 5 ----- 3 files changed, 1 insertion(+), 44 deletions(-) diff --git a/tests/test_downloadermiddleware.py b/tests/test_downloadermiddleware.py index b538a0ed3..38be915f2 100644 --- a/tests/test_downloadermiddleware.py +++ b/tests/test_downloadermiddleware.py @@ -1,13 +1,11 @@ import asyncio -from unittest import mock, SkipTest +from unittest import mock from pytest import mark -from twisted import version as twisted_version from twisted.internet import defer from twisted.internet.defer import Deferred from twisted.trial.unittest import TestCase from twisted.python.failure import Failure -from twisted.python.versions import Version from scrapy.http import Request, Response from scrapy.spiders import Spider @@ -218,16 +216,6 @@ class MiddlewareUsingCoro(ManagerTestCase): """Middlewares using asyncio coroutines should work""" def test_asyncdef(self): - if ( - self.reactor_pytest == 'asyncio' - and twisted_version < Version('twisted', 18, 4, 0) - ): - raise SkipTest( - 'Due to https://twistedmatrix.com/trac/ticket/9390, this test ' - 'hangs when using AsyncIO and Twisted versions lower than ' - '18.4.0' - ) - resp = Response('http://example.com/index.html') class CoroMiddleware: @@ -248,12 +236,6 @@ class MiddlewareUsingCoro(ManagerTestCase): @mark.only_asyncio() def test_asyncdef_asyncio(self): - if twisted_version < Version('twisted', 18, 4, 0): - raise SkipTest( - 'Due to https://twistedmatrix.com/trac/ticket/9390, this test ' - 'hangs when using Twisted versions lower than 18.4.0' - ) - resp = Response('http://example.com/index.html') class CoroMiddleware: diff --git a/tests/test_utils_signal.py b/tests/test_utils_signal.py index ad7394232..a36e7bc97 100644 --- a/tests/test_utils_signal.py +++ b/tests/test_utils_signal.py @@ -1,13 +1,10 @@ import asyncio -from unittest import SkipTest from pydispatch import dispatcher from pytest import mark from testfixtures import LogCapture -from twisted import version as twisted_version from twisted.internet import defer, reactor from twisted.python.failure import Failure -from twisted.python.versions import Version from twisted.trial import unittest from scrapy.utils.signal import send_catch_log, send_catch_log_deferred @@ -81,16 +78,6 @@ class SendCatchLogDeferredAsyncDefTest(SendCatchLogDeferredTest): return "OK" def test_send_catch_log(self): - if ( - self.reactor_pytest == 'asyncio' - and twisted_version < Version('twisted', 18, 4, 0) - ): - raise SkipTest( - 'Due to https://twistedmatrix.com/trac/ticket/9390, this test ' - 'fails due to a timeout when using AsyncIO and Twisted ' - 'versions lower than 18.4.0' - ) - return super().test_send_catch_log() @@ -104,13 +91,6 @@ class SendCatchLogDeferredAsyncioTest(SendCatchLogDeferredTest): return await get_from_asyncio_queue("OK") def test_send_catch_log(self): - if twisted_version < Version('twisted', 18, 4, 0): - raise SkipTest( - 'Due to https://twistedmatrix.com/trac/ticket/9390, this test ' - 'fails due to a timeout when using Twisted versions lower ' - 'than 18.4.0' - ) - return super().test_send_catch_log() diff --git a/tests/test_webclient.py b/tests/test_webclient.py index a6d55cb38..0d5827339 100644 --- a/tests/test_webclient.py +++ b/tests/test_webclient.py @@ -4,10 +4,7 @@ Tests borrowed from the twisted.web.client tests. """ import os import shutil -import sys -from pkg_resources import parse_version -import cryptography import OpenSSL.SSL from twisted.trial import unittest from twisted.web import server, static, util, resource @@ -417,8 +414,6 @@ class WebClientCustomCiphersSSLTestCase(WebClientSSLTestCase): ).addCallback(self.assertEqual, to_bytes(s)) def testPayloadDisabledCipher(self): - if sys.implementation.name == "pypy" and parse_version(cryptography.__version__) <= parse_version("2.3.1"): - self.skipTest("This test expects a failure, but the code does work in PyPy with cryptography<=2.3.1") s = "0123456789" * 10 settings = Settings({'DOWNLOADER_CLIENT_TLS_CIPHERS': 'ECDHE-RSA-AES256-GCM-SHA384'}) client_context_factory = create_instance(ScrapyClientContextFactory, settings=settings, crawler=None) From c4c5c9f25841a783aab2c682125f3200d6c6e446 Mon Sep 17 00:00:00 2001 From: Laerte Pereira Date: Thu, 9 Jun 2022 10:00:44 -0300 Subject: [PATCH 122/126] docs: Remove minimal versions paragraphs --- docs/intro/install.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 23c3af74b..c1fd6d522 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -52,12 +52,6 @@ Scrapy is written in pure Python and depends on a few key Python packages (among * `twisted`_, an asynchronous networking framework * `cryptography`_ and `pyOpenSSL`_, to deal with various network-level security needs -The minimal versions which Scrapy is tested against are: - -* Twisted 18.9.0 -* lxml 4.3.0 -* pyOpenSSL 19.1.0 - Scrapy may work with older versions of these packages but it is not guaranteed it will continue working because it’s not being tested against them. From 197aca2c94201f9944404f30fc4a002309cad99b Mon Sep 17 00:00:00 2001 From: Laerte Pereira Date: Thu, 9 Jun 2022 10:11:49 -0300 Subject: [PATCH 123/126] docs: Remove leftover --- docs/intro/install.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/intro/install.rst b/docs/intro/install.rst index c1fd6d522..80a9c16d6 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -52,10 +52,6 @@ Scrapy is written in pure Python and depends on a few key Python packages (among * `twisted`_, an asynchronous networking framework * `cryptography`_ and `pyOpenSSL`_, to deal with various network-level security needs -Scrapy may work with older versions of these packages -but it is not guaranteed it will continue working -because it’s not being tested against them. - Some of these packages themselves depends on non-Python packages that might require additional installation steps depending on your platform. Please check :ref:`platform-specific guides below `. From 9e265a2c1f6bccb551e8292785e09462594b8402 Mon Sep 17 00:00:00 2001 From: Kromitvs <74136201+Kromitvs@users.noreply.github.com> Date: Thu, 16 Jun 2022 19:52:19 +0100 Subject: [PATCH 124/126] Mind body to choose response class in cache, FTP and HTTP/1.0 (#4873) --- scrapy/core/downloader/handlers/ftp.py | 6 +-- scrapy/core/downloader/webclient.py | 2 +- scrapy/extensions/httpcache.py | 4 +- scrapy/responsetypes.py | 10 ++-- tests/test_downloader_handlers.py | 54 ++++++++++++++++++-- tests/test_downloadermiddleware_httpcache.py | 15 ++++++ tests/test_responsetypes.py | 2 + 7 files changed, 78 insertions(+), 15 deletions(-) diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py index 3ef129587..a495874bd 100644 --- a/scrapy/core/downloader/handlers/ftp.py +++ b/scrapy/core/downloader/handlers/ftp.py @@ -102,11 +102,11 @@ class FTPDownloadHandler: def _build_response(self, result, request, protocol): self.result = result - respcls = responsetypes.from_args(url=request.url) protocol.close() - body = protocol.filename or protocol.body.read() headers = {"local filename": protocol.filename or '', "size": protocol.size} - return respcls(url=request.url, status=200, body=to_bytes(body), headers=headers) + body = to_bytes(protocol.filename or protocol.body.read()) + respcls = responsetypes.from_args(url=request.url, body=body) + return respcls(url=request.url, status=200, body=body, headers=headers) def _failed(self, result, request): message = result.getErrorMessage() diff --git a/scrapy/core/downloader/webclient.py b/scrapy/core/downloader/webclient.py index 06cb96489..7d048c1e4 100644 --- a/scrapy/core/downloader/webclient.py +++ b/scrapy/core/downloader/webclient.py @@ -112,7 +112,7 @@ class ScrapyHTTPClientFactory(ClientFactory): request.meta['download_latency'] = self.headers_time - self.start_time status = int(self.status) headers = Headers(self.response_headers) - respcls = responsetypes.from_args(headers=headers, url=self._url) + respcls = responsetypes.from_args(headers=headers, url=self._url, body=body) return respcls(url=self._url, status=status, headers=headers, body=body, protocol=to_unicode(self.version)) def _set_connection_attributes(self, request): diff --git a/scrapy/extensions/httpcache.py b/scrapy/extensions/httpcache.py index c71484cfa..843e14812 100644 --- a/scrapy/extensions/httpcache.py +++ b/scrapy/extensions/httpcache.py @@ -240,7 +240,7 @@ class DbmCacheStorage: status = data['status'] headers = Headers(data['headers']) body = data['body'] - respcls = responsetypes.from_args(headers=headers, url=url) + respcls = responsetypes.from_args(headers=headers, url=url, body=body) response = respcls(url=url, headers=headers, status=status, body=body) return response @@ -299,7 +299,7 @@ class FilesystemCacheStorage: url = metadata.get('response_url') status = metadata['status'] headers = Headers(headers_raw_to_dict(rawheaders)) - respcls = responsetypes.from_args(headers=headers, url=url) + respcls = responsetypes.from_args(headers=headers, url=url, body=body) response = respcls(url=url, headers=headers, status=status, body=body) return response diff --git a/scrapy/responsetypes.py b/scrapy/responsetypes.py index 6ed9f8b8f..3efd4d2fd 100644 --- a/scrapy/responsetypes.py +++ b/scrapy/responsetypes.py @@ -95,12 +95,14 @@ class ResponseTypes: chunk = to_bytes(chunk) if not binary_is_text(chunk): return self.from_mimetype('application/octet-stream') - elif b"" in chunk.lower(): + lowercase_chunk = chunk.lower() + if b"" in lowercase_chunk: return self.from_mimetype('text/html') - elif b"' in lowercase_chunk: + return self.from_mimetype('text/html') + return self.from_mimetype('text') def from_args(self, headers=None, url=None, filename=None, body=None): """Guess the most appropriate Response class based on diff --git a/tests/test_downloader_handlers.py b/tests/test_downloader_handlers.py index 2bb53950d..72f52121e 100644 --- a/tests/test_downloader_handlers.py +++ b/tests/test_downloader_handlers.py @@ -25,7 +25,7 @@ from scrapy.core.downloader.handlers.http10 import HTTP10DownloadHandler from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler from scrapy.core.downloader.handlers.s3 import S3DownloadHandler from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning -from scrapy.http import Headers, Request +from scrapy.http import Headers, HtmlResponse, Request from scrapy.http.response.text import TextResponse from scrapy.responsetypes import responsetypes from scrapy.spiders import Spider @@ -389,6 +389,23 @@ class HttpTestCase(unittest.TestCase): d.addCallback(self.assertEqual, b'159') return d + def _test_response_class(self, filename, body, response_class): + def _test(response): + self.assertEqual(type(response), response_class) + + request = Request(self.getURL(filename), body=body) + return self.download_request(request, Spider('foo')).addCallback(_test) + + def test_response_class_from_url(self): + return self._test_response_class('foo.html', b'', HtmlResponse) + + def test_response_class_from_body(self): + return self._test_response_class( + 'foo', + b"\n.", + HtmlResponse, + ) + class Http10TestCase(HttpTestCase): """HTTP 1.0 test case""" @@ -971,6 +988,12 @@ class BaseFTPTestCase(unittest.TestCase): password = "passwd" req_meta = {"ftp_user": username, "ftp_password": password} + test_files = ( + ('file.txt', b"I have the power!"), + ('file with spaces.txt', b"Moooooooooo power!"), + ('html-file-without-extension', b"\n."), + ) + def setUp(self): from twisted.protocols.ftp import FTPRealm, FTPFactory from scrapy.core.downloader.handlers.ftp import FTPDownloadHandler @@ -981,8 +1004,8 @@ class BaseFTPTestCase(unittest.TestCase): userdir = os.path.join(self.directory, self.username) os.mkdir(userdir) fp = FilePath(userdir) - fp.child('file.txt').setContent(b"I have the power!") - fp.child('file with spaces.txt').setContent(b"Moooooooooo power!") + for filename, content in self.test_files: + fp.child(filename).setContent(content) # setup server realm = FTPRealm(anonymousRoot=self.directory, userHome=self.directory) @@ -1069,6 +1092,27 @@ class BaseFTPTestCase(unittest.TestCase): return self._add_test_callbacks(d, _test) + def _test_response_class(self, filename, response_class): + f, local_fname = tempfile.mkstemp() + local_fname = to_bytes(local_fname) + os.close(f) + meta = {} + meta.update(self.req_meta) + request = Request(url=f"ftp://127.0.0.1:{self.portNum}/{filename}", + meta=meta) + d = self.download_handler.download_request(request, None) + + def _test(r): + self.assertEqual(type(r), response_class) + os.remove(local_fname) + return self._add_test_callbacks(d, _test) + + def test_response_class_from_url(self): + return self._test_response_class('file.txt', TextResponse) + + def test_response_class_from_body(self): + return self._test_response_class('html-file-without-extension', HtmlResponse) + class FTPTestCase(BaseFTPTestCase): @@ -1104,8 +1148,8 @@ class AnonymousFTPTestCase(BaseFTPTestCase): os.mkdir(self.directory) fp = FilePath(self.directory) - fp.child('file.txt').setContent(b"I have the power!") - fp.child('file with spaces.txt').setContent(b"Moooooooooo power!") + for filename, content in self.test_files: + fp.child(filename).setContent(content) # setup server for anonymous access realm = FTPRealm(anonymousRoot=self.directory) diff --git a/tests/test_downloadermiddleware_httpcache.py b/tests/test_downloadermiddleware_httpcache.py index 0c6dcf2aa..928c007f5 100644 --- a/tests/test_downloadermiddleware_httpcache.py +++ b/tests/test_downloadermiddleware_httpcache.py @@ -122,6 +122,21 @@ class DefaultStorageTest(_BaseTest): time.sleep(0.5) # give the chance to expire assert storage.retrieve_response(self.spider, self.request) + def test_storage_no_content_type_header(self): + """Test that the response body is used to get the right response class + even if there is no Content-Type header""" + with self._storage() as storage: + assert storage.retrieve_response(self.spider, self.request) is None + response = Response( + 'http://www.example.com', + body=b'\n.', + status=202, + ) + storage.store_response(self.spider, self.request, response) + cached_response = storage.retrieve_response(self.spider, self.request) + self.assertIsInstance(cached_response, HtmlResponse) + self.assertEqualResponse(response, cached_response) + class DbmStorageTest(DefaultStorageTest): diff --git a/tests/test_responsetypes.py b/tests/test_responsetypes.py index c07d3a99c..4b4095fb0 100644 --- a/tests/test_responsetypes.py +++ b/tests/test_responsetypes.py @@ -54,6 +54,8 @@ class ResponseTypesTest(unittest.TestCase): (b'\x03\x02\xdf\xdd\x23', Response), (b'Some plain text\ndata with tabs\t and null bytes\0', TextResponse), (b'Hello', HtmlResponse), + # https://codersblock.com/blog/the-smallest-valid-html5-page/ + (b'\n.', HtmlResponse), (b' Date: Fri, 17 Jun 2022 08:37:14 +0200 Subject: [PATCH 125/126] Support and prefer the .jsonl file extension (#4848) --- docs/intro/overview.rst | 4 ++-- docs/intro/tutorial.rst | 2 +- docs/topics/feed-exports.rst | 3 ++- docs/topics/item-pipeline.rst | 8 ++++---- scrapy/settings/default_settings.py | 1 + 5 files changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index f3d652621..cfa6bfa83 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -45,9 +45,9 @@ https://quotes.toscrape.com, following the pagination:: Put this in a text file, name it to something like ``quotes_spider.py`` and run the spider using the :command:`runspider` command:: - scrapy runspider quotes_spider.py -o quotes.jl + scrapy runspider quotes_spider.py -o quotes.jsonl -When this finishes you will have in the ``quotes.jl`` file a list of the +When this finishes you will have in the ``quotes.jsonl`` file a list of the quotes in JSON Lines format, containing text and author, looking like this:: {"author": "Jane Austen", "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"} diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index cde1b1ef4..75928077e 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -482,7 +482,7 @@ to append new content to any existing file. However, appending to a JSON file makes the file contents invalid JSON. When appending to a file, consider using a different serialization format, such as `JSON Lines`_:: - scrapy crawl quotes -o quotes.jl + scrapy crawl quotes -o quotes.jsonl The `JSON Lines`_ format is useful because it's stream-like, you can easily append new records to it. It doesn't have the same problem of JSON when you run diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index 9a13eb82f..398f80633 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -638,6 +638,7 @@ Default:: { 'json': 'scrapy.exporters.JsonItemExporter', 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter', + 'jsonl': 'scrapy.exporters.JsonLinesItemExporter', 'jl': 'scrapy.exporters.JsonLinesItemExporter', 'csv': 'scrapy.exporters.CsvItemExporter', 'xml': 'scrapy.exporters.XmlItemExporter', @@ -763,7 +764,7 @@ source spider in the feed URI: #. Use ``%(spider_name)s`` in your feed URI:: - scrapy crawl -o "%(spider_name)s.jl" + scrapy crawl -o "%(spider_name)s.jsonl" .. _URIs: https://en.wikipedia.org/wiki/Uniform_Resource_Identifier diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index 882ff5661..af294f52c 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -99,11 +99,11 @@ contain a price:: raise DropItem(f"Missing price in {item}") -Write items to a JSON file --------------------------- +Write items to a JSON lines file +-------------------------------- The following pipeline stores all scraped items (from all spiders) into a -single ``items.jl`` file, containing one item per line serialized in JSON +single ``items.jsonl`` file, containing one item per line serialized in JSON format:: import json @@ -113,7 +113,7 @@ format:: class JsonWriterPipeline: def open_spider(self, spider): - self.file = open('items.jl', 'w') + self.file = open('items.jsonl', 'w') def close_spider(self, spider): self.file.close() diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index f5a3efe69..ff86af125 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -154,6 +154,7 @@ FEED_EXPORTERS = {} FEED_EXPORTERS_BASE = { 'json': 'scrapy.exporters.JsonItemExporter', 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter', + 'jsonl': 'scrapy.exporters.JsonLinesItemExporter', 'jl': 'scrapy.exporters.JsonLinesItemExporter', 'csv': 'scrapy.exporters.CsvItemExporter', 'xml': 'scrapy.exporters.XmlItemExporter', From d8223adfacc7e0ae684e5f9463474707bfcb008d Mon Sep 17 00:00:00 2001 From: Emanuele Date: Mon, 20 Jun 2022 11:54:05 +0200 Subject: [PATCH 126/126] =?UTF-8?q?Typo:=20cleanup=20(verb)=20=E2=86=92=20?= =?UTF-8?q?clean=20up=20(#5538)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/README.rst b/docs/README.rst index 0b7afa548..36dd5aea4 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -43,7 +43,7 @@ This command will fire up your default browser and open the main page of your Start over ---------- -To cleanup all generated documentation files and start from scratch run:: +To clean up all generated documentation files and start from scratch run:: make clean