From 67bfb304cdeb19a0b72b0a09542582f718cf07d6 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 21 Apr 2023 18:52:58 +0400 Subject: [PATCH 1/4] Release notes for the current master. --- docs/news.rst | 105 ++++++++++++++++++++++++++++++++++++++- docs/topics/settings.rst | 2 +- 2 files changed, 104 insertions(+), 3 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 9b9eeac71..6cf366449 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,108 @@ Release notes ============= +.. _release-2.9.0: + +Scrapy 2.9.0 (YYYY-MM-DD) +------------------------- + +Highlights: + +- Per-domain request settings. +- Compatibility with new cryptography_ and new parsel_. +- TBD + +New features +~~~~~~~~~~~~ + +- Settings correponding to :setting:`DOWNLOAD_DELAY`, + :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and + :setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per domain basis + via the new :setting:`DOWNLOAD_SLOTS` setting. (:issue:`5328`) + +- Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a + curl command from a :class:`~scrapy.Request` object. (:issue:`5892`) + +- Values of :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can now be + :class:`pathlib.Path` instances. (:issue:`5801`) + +- :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed + string values for the curl ``--data-raw`` argument, which are produced by + browsers for data that includes certain symbols. (:issue:`5899`, + :issue:`5901`) + +- The ``scrapy parse`` command now also works with async generator callbacks. + (:issue:`5819`, :issue:`5824`) + +- The ``scrapy genspider`` command now properly works with HTTPS URLs. + (:issue:`3553`, :issue:`5808`) + +- Improved handling of asyncio loops. (:issue:`5831`, :issue:`5832`) + +- :class:`LinkExtractor ` + now skips certain malformed URLs instead of raising an exception. + (:issue:`5881`) + +- :func:`scrapy.utils.python.get_func_args` now supports more types of + callables. (:issue:`5872`, :issue:`5885`) + +Bug fixes +~~~~~~~~~ + +- Fixed an error when using feed postprocessing with S3 storage. + (:issue:`5500`, :issue:`5581`) + +- Added the missing :meth:`scrapy.settings.BaseSettings.setdefault` method. + (:issue:`5811`, :issue:`5821`) + +- Fixed an error when using cryptography_ 40.0.0+ and + :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` is enabled. + (:issue:`5857`, :issue:`5858`) + +- The checksums returned by :class:`~scrapy.pipelines.files.FilesPipeline` + for files on Google Cloud Storage are no longer Base64-encoded. + (:issue:`5874`, :issue:`5891`) + +- Fixed an error breaking user handling of send failures in + :meth:`scrapy.mail.MailSender.send()`. (:issue:`1611`, :issue:`5880`) + +Documentation +~~~~~~~~~~~~~ + +- Expanded contributing docs. (:issue:`5109`, :issue:`5851`) + +- Added blacken-docs_ to pre-commit and reformatted the docs with it. + (:issue:`5813`, :issue:`5816`) + +- Fixed a JS issue. (:issue:`5875`, :issue:`5877`) + +- Fixed ``make htmlview``. (:issue:`5878`, :issue:`5879`) + +- Fixed typos and other small errors. (:issue:`5827`, :issue:`5839`, + :issue:`5883`, :issue:`5890`, :issue:`5895`, :issue:`5904`) + +Quality assurance +~~~~~~~~~~~~~~~~~ + +- Extended typing hints. (:issue:`5805`, :issue:`5889`, :issue:`5896`) + +- Tests for most of the examples in the docs are now run as a part of CI, + found problems were fixed. (:issue:`5816`, :issue:`5826`) + +- Removed usage of deprecated Python classes. (:issue:`5849`) + +- Silenced ``include-ignored`` warnings from coverage. (:issue:`5820`) + +- Fixed a random failure of the ``test_feedexport.test_batch_path_differ`` + test. (:issue:`5855`, :issue:`5898`) + +- Updated docstrings to match output produced by parsel_ 1.8.1 so that they + don't cause test failures. (:issue:`5902`) + +- Other CI and pre-commit improvements. (:issue:`5802`, :issue:`5823`) + +.. _blacken-docs: https://github.com/adamchainz/blacken-docs + .. _release-2.8.0: Scrapy 2.8.0 (2023-02-02) @@ -4207,8 +4309,6 @@ Relocations + Note: telnet is not enabled on Python 3 (https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595) -.. _parsel: https://github.com/scrapy/parsel - Bugfixes ~~~~~~~~ @@ -5638,6 +5738,7 @@ First release of Scrapy. .. _LevelDB: https://github.com/google/leveldb .. _lxml: https://lxml.de/ .. _marshal: https://docs.python.org/2/library/marshal.html +.. _parsel: https://github.com/scrapy/parsel .. _parsel.csstranslator.GenericTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.GenericTranslator .. _parsel.csstranslator.HTMLTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.HTMLTranslator .. _parsel.csstranslator.XPathExpr: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.XPathExpr diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 4412b5c1c..3e06d84f9 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -783,7 +783,7 @@ DOWNLOAD_SLOTS Default: ``{}`` -Allows to define concurrency/delay parameters on per slot(domain) basis: +Allows to define concurrency/delay parameters on per slot (domain) basis: .. code-block:: python From d1d6465ef4ea8987efb08e2f9abfd65b36719e04 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 4 May 2023 17:19:01 +0400 Subject: [PATCH 2/4] Address feedback. --- docs/news.rst | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 6cf366449..5f189760d 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -10,7 +10,7 @@ Scrapy 2.9.0 (YYYY-MM-DD) Highlights: -- Per-domain request settings. +- Per-domain download settings. - Compatibility with new cryptography_ and new parsel_. - TBD @@ -19,7 +19,7 @@ New features - Settings correponding to :setting:`DOWNLOAD_DELAY`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and - :setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per domain basis + :setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per-domain basis via the new :setting:`DOWNLOAD_SLOTS` setting. (:issue:`5328`) - Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a @@ -28,26 +28,6 @@ New features - Values of :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can now be :class:`pathlib.Path` instances. (:issue:`5801`) -- :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed - string values for the curl ``--data-raw`` argument, which are produced by - browsers for data that includes certain symbols. (:issue:`5899`, - :issue:`5901`) - -- The ``scrapy parse`` command now also works with async generator callbacks. - (:issue:`5819`, :issue:`5824`) - -- The ``scrapy genspider`` command now properly works with HTTPS URLs. - (:issue:`3553`, :issue:`5808`) - -- Improved handling of asyncio loops. (:issue:`5831`, :issue:`5832`) - -- :class:`LinkExtractor ` - now skips certain malformed URLs instead of raising an exception. - (:issue:`5881`) - -- :func:`scrapy.utils.python.get_func_args` now supports more types of - callables. (:issue:`5872`, :issue:`5885`) - Bug fixes ~~~~~~~~~ @@ -65,6 +45,26 @@ Bug fixes for files on Google Cloud Storage are no longer Base64-encoded. (:issue:`5874`, :issue:`5891`) +- :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed + string values for the curl ``--data-raw`` argument, which are produced by + browsers for data that includes certain symbols. (:issue:`5899`, + :issue:`5901`) + +- The :command:`parse` command now also works with async generator callbacks. + (:issue:`5819`, :issue:`5824`) + +- The :command:`genspider` command now properly works with HTTPS URLs. + (:issue:`3553`, :issue:`5808`) + +- Improved handling of asyncio loops. (:issue:`5831`, :issue:`5832`) + +- :class:`LinkExtractor ` + now skips certain malformed URLs instead of raising an exception. + (:issue:`5881`) + +- :func:`scrapy.utils.python.get_func_args` now supports more types of + callables. (:issue:`5872`, :issue:`5885`) + - Fixed an error breaking user handling of send failures in :meth:`scrapy.mail.MailSender.send()`. (:issue:`1611`, :issue:`5880`) From 636559f1cc652e839ef42522e7168e3ff9d77921 Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Thu, 4 May 2023 17:55:07 +0400 Subject: [PATCH 3/4] Add newer changes. --- docs/news.rst | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 5f189760d..cbbb376e5 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -12,7 +12,8 @@ Highlights: - Per-domain download settings. - Compatibility with new cryptography_ and new parsel_. -- TBD +- JMESPath selectors from the new parsel_. +- Bug fixes. New features ~~~~~~~~~~~~ @@ -22,6 +23,12 @@ New features :setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per-domain basis via the new :setting:`DOWNLOAD_SLOTS` setting. (:issue:`5328`) +- Added :meth:`TextResponse.jmespath`, a shortcut for JMESPath selectors + available since parsel_ 1.8.1. (:issue:`5894`, :issue:`5915`) + +- Added :signal:`feed_slot_closed` and :signal:`feed_exporter_closed` + signals. (:issue:`5876`) + - Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a curl command from a :class:`~scrapy.Request` object. (:issue:`5892`) @@ -31,6 +38,8 @@ New features Bug fixes ~~~~~~~~~ +- Fixed a warning with Parsel 1.8.1+. (:issue:`5903`, :issue:`5918`) + - Fixed an error when using feed postprocessing with S3 storage. (:issue:`5500`, :issue:`5581`) @@ -65,6 +74,9 @@ Bug fixes - :func:`scrapy.utils.python.get_func_args` now supports more types of callables. (:issue:`5872`, :issue:`5885`) +- Fixed an error when processing non-UTF8 values of ``Content-Type`` headers. + (:issue:`5914`, :issue:`5917`) + - Fixed an error breaking user handling of send failures in :meth:`scrapy.mail.MailSender.send()`. (:issue:`1611`, :issue:`5880`) @@ -89,7 +101,7 @@ Quality assurance - Extended typing hints. (:issue:`5805`, :issue:`5889`, :issue:`5896`) - Tests for most of the examples in the docs are now run as a part of CI, - found problems were fixed. (:issue:`5816`, :issue:`5826`) + found problems were fixed. (:issue:`5816`, :issue:`5826`, :issue:`5919`) - Removed usage of deprecated Python classes. (:issue:`5849`) @@ -99,9 +111,10 @@ Quality assurance test. (:issue:`5855`, :issue:`5898`) - Updated docstrings to match output produced by parsel_ 1.8.1 so that they - don't cause test failures. (:issue:`5902`) + don't cause test failures. (:issue:`5902`, :issue:`5919`) -- Other CI and pre-commit improvements. (:issue:`5802`, :issue:`5823`) +- Other CI and pre-commit improvements. (:issue:`5802`, :issue:`5823`, + :issue:`5908`) .. _blacken-docs: https://github.com/adamchainz/blacken-docs From caa66fa15ae5d353434e5bb1d0c1cc2bdf857b6c Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Mon, 8 May 2023 13:27:01 +0400 Subject: [PATCH 4/4] Mention deprecating _FeedSlot. --- docs/news.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index cbbb376e5..4e198c893 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -15,6 +15,13 @@ Highlights: - JMESPath selectors from the new parsel_. - Bug fixes. +Deprecations +~~~~~~~~~~~~ + +- :class:`scrapy.extensions.feedexport._FeedSlot` is renamed to + :class:`scrapy.extensions.feedexport.FeedSlot` and the old name is + deprecated. (:issue:`5876`) + New features ~~~~~~~~~~~~