mirror of https://github.com/scrapy/scrapy.git
Add release notes for Scrapy 2.8 (#5799)
This commit is contained in:
Adrián Chaves
GitHub
parent
8c8894f4be
commit
068af85722
|
|
@ -49,7 +49,7 @@ guidelines when you're going to report a new bug.
|
|||
(use "scrapy" tag).
|
||||
|
||||
* check the `open issues`_ to see if the issue has already been reported. If it
|
||||
has, don't dismiss the report, but check the ticket history and comments. If
|
||||
has, don't dismiss the report, but check the ticket history and comments. If
|
||||
you have additional useful information, please leave a comment, or consider
|
||||
:ref:`sending a pull request <writing-patches>` with a fix.
|
||||
|
||||
|
|
@ -169,7 +169,7 @@ Coding style
|
|||
Please follow these coding conventions when writing code for inclusion in
|
||||
Scrapy:
|
||||
|
||||
* We use `black <https://black.readthedocs.io/en/stable/>`_ for code formatting.
|
||||
* We use `black <https://black.readthedocs.io/en/stable/>`_ for code formatting.
|
||||
There is a hook in the pre-commit config
|
||||
that will automatically format your code before every commit. You can also
|
||||
run black manually with ``tox -e black``.
|
||||
|
|
@ -179,29 +179,31 @@ Scrapy:
|
|||
See https://help.github.com/en/github/using-git/setting-your-username-in-git for
|
||||
setup instructions.
|
||||
|
||||
.. _scrapy-pre-commit:
|
||||
|
||||
Pre-commit
|
||||
==========
|
||||
|
||||
We use `pre-commit`_ to automatically address simple code issues before every
|
||||
We use `pre-commit`_ to automatically address simple code issues before every
|
||||
commit.
|
||||
|
||||
.. _pre-commit: https://pre-commit.com/
|
||||
|
||||
Before you start writing a patch:
|
||||
After your create a local clone of your fork of the Scrapy repository:
|
||||
|
||||
#. `Install pre-commit <https://pre-commit.com/#installation>`_.
|
||||
|
||||
#. On the root of your local clone of the Scrapy repository, run the following
|
||||
#. On the root of your local clone of the Scrapy repository, run the following
|
||||
command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pre-commit install
|
||||
|
||||
Now pre-commit will check your changes every time you create a Git commit. Upon
|
||||
finding issues, pre-commit aborts your commit, and either fixes those issues
|
||||
automatically, or only reports them to you. If it fixes those issues
|
||||
automatically, creating your commit again should succeed. Otherwise, you may
|
||||
Now pre-commit will check your changes every time you create a Git commit. Upon
|
||||
finding issues, pre-commit aborts your commit, and either fixes those issues
|
||||
automatically, or only reports them to you. If it fixes those issues
|
||||
automatically, creating your commit again should succeed. Otherwise, you may
|
||||
need to address the corresponding issues manually first.
|
||||
|
||||
.. _documentation-policies:
|
||||
|
|
|
|||
206
docs/news.rst
206
docs/news.rst
|
|
@ -3,6 +3,212 @@
|
|||
Release notes
|
||||
=============
|
||||
|
||||
.. _release-2.8.0:
|
||||
|
||||
Scrapy 2.8.0 (2023-02-02)
|
||||
-------------------------
|
||||
|
||||
This is a maintenance release, with minor features, bug fixes, and cleanups.
|
||||
|
||||
Deprecation removals
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now
|
||||
been removed. Use the :meth:`~io.BufferedIOBase.read1` method of
|
||||
:class:`~gzip.GzipFile` instead.
|
||||
(:issue:`5719`)
|
||||
|
||||
- The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy
|
||||
2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode`
|
||||
instead.
|
||||
(:issue:`5719`)
|
||||
|
||||
- The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy
|
||||
2.0, has now been removed. Use
|
||||
:meth:`~scrapy.utils.python.MutableChain.__next__` instead.
|
||||
(:issue:`5719`)
|
||||
|
||||
- The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated
|
||||
in Scrapy 2.0, has now been removed. Use
|
||||
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
|
||||
instead.
|
||||
(:issue:`5720`)
|
||||
|
||||
- Support for using environment variables prefixed with ``SCRAPY_`` to
|
||||
override settings, deprecated in Scrapy 2.0, has now been removed.
|
||||
(:issue:`5724`)
|
||||
|
||||
- Support for the ``noconnect`` query string argument in proxy URLs,
|
||||
deprecated in Scrapy 2.0, has now been removed. We expect proxies that used
|
||||
to need it to work fine without it.
|
||||
(:issue:`5731`)
|
||||
|
||||
- The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy
|
||||
2.3, has now been removed.
|
||||
(:issue:`5719`)
|
||||
|
||||
- The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4,
|
||||
has now been removed.
|
||||
(:issue:`5719`)
|
||||
|
||||
|
||||
Deprecations
|
||||
~~~~~~~~~~~~
|
||||
|
||||
- :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated.
|
||||
(:issue:`5368`, :issue:`5489`)
|
||||
|
||||
- :meth:`ImagesPipeline.convert_image
|
||||
<scrapy.pipelines.images.ImagesPipeline.convert_image>` must now accept a
|
||||
``response_body`` parameter.
|
||||
(:issue:`3055`, :issue:`3689`, :issue:`4753`)
|
||||
|
||||
|
||||
New features
|
||||
~~~~~~~~~~~~
|
||||
|
||||
- Applied black_ coding style to files generated with the
|
||||
:command:`genspider` and :command:`startproject` commands.
|
||||
(:issue:`5809`, :issue:`5814`)
|
||||
|
||||
.. _black: https://black.readthedocs.io/en/stable/
|
||||
|
||||
- :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the
|
||||
``settings.py`` file that the :command:`startproject` command generates.
|
||||
With this value, JSON exports won’t force the use of escape sequences for
|
||||
non-ASCII characters.
|
||||
(:issue:`5797`, :issue:`5800`)
|
||||
|
||||
- The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the
|
||||
peak memory usage during checks, and the binary unit MiB is now used to
|
||||
avoid confusion.
|
||||
(:issue:`5717`, :issue:`5722`, :issue:`5727`)
|
||||
|
||||
- The ``callback`` parameter of :class:`~scrapy.http.Request` can now be set
|
||||
to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from
|
||||
``None``, as the latter indicates that the default spider callback
|
||||
(:meth:`~scrapy.Spider.parse`) is to be used.
|
||||
(:issue:`5798`)
|
||||
|
||||
|
||||
Bug fixes
|
||||
~~~~~~~~~
|
||||
|
||||
- Enabled unsafe legacy SSL renegotiation to fix access to some outdated
|
||||
websites.
|
||||
(:issue:`5491`, :issue:`5790`)
|
||||
|
||||
- Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and
|
||||
better.
|
||||
(:issue:`5386`, :issue:`5406`)
|
||||
|
||||
- Fixed the :meth:`finish_exporting` method of :ref:`item exporters
|
||||
<topics-exporters>` not being called for empty files.
|
||||
(:issue:`5537`, :issue:`5758`)
|
||||
|
||||
- Fixed HTTP/2 responses getting only the last value for a header when
|
||||
multiple headers with the same name are received.
|
||||
(:issue:`5777`)
|
||||
|
||||
- Fixed an exception raised by the :command:`shell` command on some cases
|
||||
when :ref:`using asyncio <using-asyncio>`.
|
||||
(:issue:`5740`, :issue:`5742`, :issue:`5748`, :issue:`5759`, :issue:`5760`,
|
||||
:issue:`5771`)
|
||||
|
||||
- When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments
|
||||
(``cb_kwargs``) added to a request in the ``process_request`` callback of a
|
||||
:class:`~scrapy.spiders.Rule` will no longer be ignored.
|
||||
(:issue:`5699`)
|
||||
|
||||
- The :ref:`images pipeline <images-pipeline>` no longer re-encodes JPEG
|
||||
files.
|
||||
(:issue:`3055`, :issue:`3689`, :issue:`4753`)
|
||||
|
||||
- Fixed the handling of transparent WebP images by the :ref:`images pipeline
|
||||
<images-pipeline>`.
|
||||
(:issue:`3072`, :issue:`5766`, :issue:`5767`)
|
||||
|
||||
- :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT``
|
||||
(Ctrl+C).
|
||||
(:issue:`2918`)
|
||||
|
||||
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
|
||||
with ``unique=False`` no longer filters out links that have identical URL
|
||||
*and* text.
|
||||
(:issue:`3798`, :issue:`3799`, :issue:`4695`, :issue:`5458`)
|
||||
|
||||
- :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
|
||||
ignores URL protocols that do not support ``robots.txt`` (``data://``,
|
||||
``file://``).
|
||||
(:issue:`5807`)
|
||||
|
||||
- Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6.
|
||||
(:issue:`5753`, :issue:`5754`)
|
||||
|
||||
- Fixed the output of ``scrapy -h`` showing an unintended ``**commands**``
|
||||
line.
|
||||
(:issue:`5709`, :issue:`5711`, :issue:`5712`)
|
||||
|
||||
- Made the active project indication in the output of :ref:`commands
|
||||
<topics-commands>` more clear.
|
||||
(:issue:`5715`)
|
||||
|
||||
|
||||
Documentation
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
- Documented how to :ref:`debug spiders from Visual Studio Code
|
||||
<debug-vscode>`.
|
||||
(:issue:`5721`)
|
||||
|
||||
- Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency.
|
||||
(:issue:`5083`, :issue:`5540`)
|
||||
|
||||
- Improved consistency.
|
||||
(:issue:`5761`)
|
||||
|
||||
- Fixed typos.
|
||||
(:issue:`5714`, :issue:`5744`, :issue:`5764`)
|
||||
|
||||
|
||||
Quality assurance
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Applied :ref:`black coding style <coding-style>`, sorted import statements,
|
||||
and introduced :ref:`pre-commit <scrapy-pre-commit>`.
|
||||
(:issue:`4654`, :issue:`4658`, :issue:`5734`, :issue:`5737`, :issue:`5806`,
|
||||
:issue:`5810`)
|
||||
|
||||
- Switched from :mod:`os.path` to :mod:`pathlib`.
|
||||
(:issue:`4916`, :issue:`4497`, :issue:`5682`)
|
||||
|
||||
- Addressed many issues reported by Pylint.
|
||||
(:issue:`5677`)
|
||||
|
||||
- Improved code readability.
|
||||
(:issue:`5736`)
|
||||
|
||||
- Improved package metadata.
|
||||
(:issue:`5768`)
|
||||
|
||||
- Removed direct invocations of ``setup.py``.
|
||||
(:issue:`5774`, :issue:`5776`)
|
||||
|
||||
- Removed unnecessary :class:`~collections.OrderedDict` usages.
|
||||
(:issue:`5795`)
|
||||
|
||||
- Removed unnecessary ``__str__`` definitions.
|
||||
(:issue:`5150`)
|
||||
|
||||
- Removed obsolete code and comments.
|
||||
(:issue:`5725`, :issue:`5729`, :issue:`5730`, :issue:`5732`)
|
||||
|
||||
- Fixed test and CI issues.
|
||||
(:issue:`5749`, :issue:`5750`, :issue:`5756`, :issue:`5762`, :issue:`5765`,
|
||||
:issue:`5780`, :issue:`5781`, :issue:`5782`, :issue:`5783`, :issue:`5785`,
|
||||
:issue:`5786`)
|
||||
|
||||
|
||||
.. _release-2.7.1:
|
||||
|
||||
Scrapy 2.7.1 (2022-11-02)
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ Consider the following Scrapy spider below::
|
|||
|
||||
def parse(self, response):
|
||||
# <processing code not shown>
|
||||
# collect `item_urls`
|
||||
# collect `item_urls`
|
||||
for item_url in item_urls:
|
||||
yield scrapy.Request(item_url, self.parse_item)
|
||||
|
||||
|
|
@ -151,6 +151,8 @@ For more information, check the :ref:`topics-logging` section.
|
|||
|
||||
.. _base tag: https://www.w3schools.com/tags/tag_base.asp
|
||||
|
||||
.. _debug-vscode:
|
||||
|
||||
Visual Studio Code
|
||||
==================
|
||||
|
||||
|
|
|
|||
|
|
@ -515,7 +515,7 @@ which uses safe numeric encoding (``\uXXXX`` sequences) for historic reasons.
|
|||
|
||||
Use ``utf-8`` if you want UTF-8 for JSON too.
|
||||
|
||||
.. versionchanged:: VERSION
|
||||
.. versionchanged:: 2.8
|
||||
The :command:`startproject` command now sets this setting to
|
||||
``utf-8`` in the generated ``settings.py`` file.
|
||||
|
||||
|
|
|
|||
|
|
@ -39,9 +39,7 @@ Request objects
|
|||
- ``None`` (default), which indicates that the spider's
|
||||
:meth:`~scrapy.Spider.parse` method must be used.
|
||||
|
||||
- :py:data:`scrapy.http.request.NO_CALLBACK`
|
||||
|
||||
.. autodata:: scrapy.http.request.NO_CALLBACK
|
||||
- :func:`~scrapy.http.request.NO_CALLBACK`
|
||||
|
||||
For more information, see
|
||||
:ref:`topics-request-response-ref-request-callback-arguments`.
|
||||
|
|
@ -247,6 +245,8 @@ Request objects
|
|||
Other functions related to requests
|
||||
-----------------------------------
|
||||
|
||||
.. autofunction:: scrapy.http.request.NO_CALLBACK
|
||||
|
||||
.. autofunction:: scrapy.utils.request.request_from_dict
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -25,6 +25,12 @@ def NO_CALLBACK(*args, **kwargs):
|
|||
:class:`~scrapy.http.Request`, it indicates that the request is not meant
|
||||
to have a spider callback at all.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
Request("https://example.com", callback=NO_CALLBACK)
|
||||
|
||||
This value should be used by :ref:`components <topics-components>` that
|
||||
create and handle their own requests, e.g. through
|
||||
:meth:`scrapy.core.engine.ExecutionEngine.download`, so that downloader
|
||||
|
|
|
|||
Loading…
Reference in New Issue