From 068af85722a41a1361e170f104e64fccfff662b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Thu, 2 Feb 2023 04:54:02 +0100 Subject: [PATCH] Add release notes for Scrapy 2.8 (#5799) --- docs/contributing.rst | 20 +-- docs/news.rst | 206 +++++++++++++++++++++++++++++++ docs/topics/debug.rst | 4 +- docs/topics/feed-exports.rst | 2 +- docs/topics/request-response.rst | 6 +- scrapy/http/request/__init__.py | 6 + 6 files changed, 230 insertions(+), 14 deletions(-) diff --git a/docs/contributing.rst b/docs/contributing.rst index edc6c2179..6b1a41339 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -49,7 +49,7 @@ guidelines when you're going to report a new bug. (use "scrapy" tag). * check the `open issues`_ to see if the issue has already been reported. If it - has, don't dismiss the report, but check the ticket history and comments. If + has, don't dismiss the report, but check the ticket history and comments. If you have additional useful information, please leave a comment, or consider :ref:`sending a pull request ` with a fix. @@ -169,7 +169,7 @@ Coding style Please follow these coding conventions when writing code for inclusion in Scrapy: -* We use `black `_ for code formatting. +* We use `black `_ for code formatting. There is a hook in the pre-commit config that will automatically format your code before every commit. You can also run black manually with ``tox -e black``. @@ -179,29 +179,31 @@ Scrapy: See https://help.github.com/en/github/using-git/setting-your-username-in-git for setup instructions. +.. _scrapy-pre-commit: + Pre-commit ========== -We use `pre-commit`_ to automatically address simple code issues before every +We use `pre-commit`_ to automatically address simple code issues before every commit. .. _pre-commit: https://pre-commit.com/ -Before you start writing a patch: +After your create a local clone of your fork of the Scrapy repository: #. `Install pre-commit `_. -#. On the root of your local clone of the Scrapy repository, run the following +#. On the root of your local clone of the Scrapy repository, run the following command: .. code-block:: bash pre-commit install -Now pre-commit will check your changes every time you create a Git commit. Upon -finding issues, pre-commit aborts your commit, and either fixes those issues -automatically, or only reports them to you. If it fixes those issues -automatically, creating your commit again should succeed. Otherwise, you may +Now pre-commit will check your changes every time you create a Git commit. Upon +finding issues, pre-commit aborts your commit, and either fixes those issues +automatically, or only reports them to you. If it fixes those issues +automatically, creating your commit again should succeed. Otherwise, you may need to address the corresponding issues manually first. .. _documentation-policies: diff --git a/docs/news.rst b/docs/news.rst index 07264827b..42166c18d 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,212 @@ Release notes ============= +.. _release-2.8.0: + +Scrapy 2.8.0 (2023-02-02) +------------------------- + +This is a maintenance release, with minor features, bug fixes, and cleanups. + +Deprecation removals +~~~~~~~~~~~~~~~~~~~~ + +- The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now + been removed. Use the :meth:`~io.BufferedIOBase.read1` method of + :class:`~gzip.GzipFile` instead. + (:issue:`5719`) + +- The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy + 2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode` + instead. + (:issue:`5719`) + +- The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy + 2.0, has now been removed. Use + :meth:`~scrapy.utils.python.MutableChain.__next__` instead. + (:issue:`5719`) + +- The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated + in Scrapy 2.0, has now been removed. Use + :class:`LinkExtractor ` + instead. + (:issue:`5720`) + +- Support for using environment variables prefixed with ``SCRAPY_`` to + override settings, deprecated in Scrapy 2.0, has now been removed. + (:issue:`5724`) + +- Support for the ``noconnect`` query string argument in proxy URLs, + deprecated in Scrapy 2.0, has now been removed. We expect proxies that used + to need it to work fine without it. + (:issue:`5731`) + +- The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy + 2.3, has now been removed. + (:issue:`5719`) + +- The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4, + has now been removed. + (:issue:`5719`) + + +Deprecations +~~~~~~~~~~~~ + +- :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated. + (:issue:`5368`, :issue:`5489`) + +- :meth:`ImagesPipeline.convert_image + ` must now accept a + ``response_body`` parameter. + (:issue:`3055`, :issue:`3689`, :issue:`4753`) + + +New features +~~~~~~~~~~~~ + +- Applied black_ coding style to files generated with the + :command:`genspider` and :command:`startproject` commands. + (:issue:`5809`, :issue:`5814`) + + .. _black: https://black.readthedocs.io/en/stable/ + +- :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the + ``settings.py`` file that the :command:`startproject` command generates. + With this value, JSON exports won’t force the use of escape sequences for + non-ASCII characters. + (:issue:`5797`, :issue:`5800`) + +- The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the + peak memory usage during checks, and the binary unit MiB is now used to + avoid confusion. + (:issue:`5717`, :issue:`5722`, :issue:`5727`) + +- The ``callback`` parameter of :class:`~scrapy.http.Request` can now be set + to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from + ``None``, as the latter indicates that the default spider callback + (:meth:`~scrapy.Spider.parse`) is to be used. + (:issue:`5798`) + + +Bug fixes +~~~~~~~~~ + +- Enabled unsafe legacy SSL renegotiation to fix access to some outdated + websites. + (:issue:`5491`, :issue:`5790`) + +- Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and + better. + (:issue:`5386`, :issue:`5406`) + +- Fixed the :meth:`finish_exporting` method of :ref:`item exporters + ` not being called for empty files. + (:issue:`5537`, :issue:`5758`) + +- Fixed HTTP/2 responses getting only the last value for a header when + multiple headers with the same name are received. + (:issue:`5777`) + +- Fixed an exception raised by the :command:`shell` command on some cases + when :ref:`using asyncio `. + (:issue:`5740`, :issue:`5742`, :issue:`5748`, :issue:`5759`, :issue:`5760`, + :issue:`5771`) + +- When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments + (``cb_kwargs``) added to a request in the ``process_request`` callback of a + :class:`~scrapy.spiders.Rule` will no longer be ignored. + (:issue:`5699`) + +- The :ref:`images pipeline ` no longer re-encodes JPEG + files. + (:issue:`3055`, :issue:`3689`, :issue:`4753`) + +- Fixed the handling of transparent WebP images by the :ref:`images pipeline + `. + (:issue:`3072`, :issue:`5766`, :issue:`5767`) + +- :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT`` + (Ctrl+C). + (:issue:`2918`) + +- :class:`LinkExtractor ` + with ``unique=False`` no longer filters out links that have identical URL + *and* text. + (:issue:`3798`, :issue:`3799`, :issue:`4695`, :issue:`5458`) + +- :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now + ignores URL protocols that do not support ``robots.txt`` (``data://``, + ``file://``). + (:issue:`5807`) + +- Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6. + (:issue:`5753`, :issue:`5754`) + +- Fixed the output of ``scrapy -h`` showing an unintended ``**commands**`` + line. + (:issue:`5709`, :issue:`5711`, :issue:`5712`) + +- Made the active project indication in the output of :ref:`commands + ` more clear. + (:issue:`5715`) + + +Documentation +~~~~~~~~~~~~~ + +- Documented how to :ref:`debug spiders from Visual Studio Code + `. + (:issue:`5721`) + +- Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency. + (:issue:`5083`, :issue:`5540`) + +- Improved consistency. + (:issue:`5761`) + +- Fixed typos. + (:issue:`5714`, :issue:`5744`, :issue:`5764`) + + +Quality assurance +~~~~~~~~~~~~~~~~~ + +- Applied :ref:`black coding style `, sorted import statements, + and introduced :ref:`pre-commit `. + (:issue:`4654`, :issue:`4658`, :issue:`5734`, :issue:`5737`, :issue:`5806`, + :issue:`5810`) + +- Switched from :mod:`os.path` to :mod:`pathlib`. + (:issue:`4916`, :issue:`4497`, :issue:`5682`) + +- Addressed many issues reported by Pylint. + (:issue:`5677`) + +- Improved code readability. + (:issue:`5736`) + +- Improved package metadata. + (:issue:`5768`) + +- Removed direct invocations of ``setup.py``. + (:issue:`5774`, :issue:`5776`) + +- Removed unnecessary :class:`~collections.OrderedDict` usages. + (:issue:`5795`) + +- Removed unnecessary ``__str__`` definitions. + (:issue:`5150`) + +- Removed obsolete code and comments. + (:issue:`5725`, :issue:`5729`, :issue:`5730`, :issue:`5732`) + +- Fixed test and CI issues. + (:issue:`5749`, :issue:`5750`, :issue:`5756`, :issue:`5762`, :issue:`5765`, + :issue:`5780`, :issue:`5781`, :issue:`5782`, :issue:`5783`, :issue:`5785`, + :issue:`5786`) + + .. _release-2.7.1: Scrapy 2.7.1 (2022-11-02) diff --git a/docs/topics/debug.rst b/docs/topics/debug.rst index edbcaf432..89e35affc 100644 --- a/docs/topics/debug.rst +++ b/docs/topics/debug.rst @@ -19,7 +19,7 @@ Consider the following Scrapy spider below:: def parse(self, response): # - # collect `item_urls` + # collect `item_urls` for item_url in item_urls: yield scrapy.Request(item_url, self.parse_item) @@ -151,6 +151,8 @@ For more information, check the :ref:`topics-logging` section. .. _base tag: https://www.w3schools.com/tags/tag_base.asp +.. _debug-vscode: + Visual Studio Code ================== diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index 8f96b1154..8775a99d0 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -515,7 +515,7 @@ which uses safe numeric encoding (``\uXXXX`` sequences) for historic reasons. Use ``utf-8`` if you want UTF-8 for JSON too. -.. versionchanged:: VERSION +.. versionchanged:: 2.8 The :command:`startproject` command now sets this setting to ``utf-8`` in the generated ``settings.py`` file. diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 766710d66..236775f93 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -39,9 +39,7 @@ Request objects - ``None`` (default), which indicates that the spider's :meth:`~scrapy.Spider.parse` method must be used. - - :py:data:`scrapy.http.request.NO_CALLBACK` - - .. autodata:: scrapy.http.request.NO_CALLBACK + - :func:`~scrapy.http.request.NO_CALLBACK` For more information, see :ref:`topics-request-response-ref-request-callback-arguments`. @@ -247,6 +245,8 @@ Request objects Other functions related to requests ----------------------------------- +.. autofunction:: scrapy.http.request.NO_CALLBACK + .. autofunction:: scrapy.utils.request.request_from_dict diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py index 7afb28db5..9ba6ddf20 100644 --- a/scrapy/http/request/__init__.py +++ b/scrapy/http/request/__init__.py @@ -25,6 +25,12 @@ def NO_CALLBACK(*args, **kwargs): :class:`~scrapy.http.Request`, it indicates that the request is not meant to have a spider callback at all. + For example: + + .. code-block:: python + + Request("https://example.com", callback=NO_CALLBACK) + This value should be used by :ref:`components ` that create and handle their own requests, e.g. through :meth:`scrapy.core.engine.ExecutionEngine.download`, so that downloader