mirror of https://github.com/scrapy/scrapy.git
Cover the current master in the release notes (up to daf9db7).
This commit is contained in:
Andrey Rakhmatullin
parent
b1f85b5a17
commit
095140f134
223
docs/news.rst
223
docs/news.rst
|
|
@ -3,16 +3,22 @@
|
|||
Release notes
|
||||
=============
|
||||
|
||||
.. _release-VERSION:
|
||||
.. _release-2.13.0:
|
||||
|
||||
Scrapy 2.13.0 (unreleased)
|
||||
--------------------------
|
||||
|
||||
Highlights:
|
||||
|
||||
- The asyncio reactor is now enabled by default
|
||||
|
||||
- Added the :reqmeta:`allow_offsite` request meta key
|
||||
|
||||
- HTTP/1.0 support is deprecated
|
||||
- :ref:`Spider middlewares that don't support asynchronous spider output
|
||||
<sync-async-spider-middleware>` are deprecated
|
||||
|
||||
- Added a base class for :ref:`universal spider middlewares
|
||||
<universal-spider-middleware>`
|
||||
|
||||
Modified requirements
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -23,21 +29,62 @@ Modified requirements
|
|||
Backward-incompatible changes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- The default value of the :setting:`TWISTED_REACTOR` setting was changed
|
||||
from ``None`` to
|
||||
``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value
|
||||
was used in newly generated projects since Scrapy 2.7.0 but now existing
|
||||
projects that don't explicitly set this setting will also use the asyncio
|
||||
reactor. You can :ref:`change this setting in your project
|
||||
<disable-asyncio>` to use a different reactor.
|
||||
(:issue:`6659`, :issue:`6713`)
|
||||
|
||||
- The ``from_settings()`` method of
|
||||
:class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`,
|
||||
deprecated in 2.12.0, is removed earlier than the usual deprecation period
|
||||
(this was needed because after the introduction of the
|
||||
deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation
|
||||
period (this was needed because after the introduction of the
|
||||
:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and
|
||||
switching built-in spider middlewares to it those middlewares need the
|
||||
:class:`~scrapy.crawler.Crawler` instance at run time). Please use
|
||||
``from_crawler()`` instead.
|
||||
(:issue:`6693`)
|
||||
|
||||
- ``scrapy.utils.url.escape_ajax()`` is no longer called when a
|
||||
:class:`~scrapy.Request` instance is created. It was only useful for
|
||||
websites supporting the ``_escaped_fragment_`` feature which most modern
|
||||
websites don't support. If you still need this you can modify the URLs
|
||||
before passing them to :class:`~scrapy.Request`.
|
||||
(:issue:`6523`, :issue:`6651`)
|
||||
|
||||
Deprecation removals
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Removed old deprecated name aliases for some signals:
|
||||
|
||||
- ``stats_spider_opened`` (use ``spider_opened`` instead)
|
||||
|
||||
- ``stats_spider_closing`` and ``stats_spider_closed`` (use
|
||||
``spider_closed`` instead)
|
||||
|
||||
- ``item_passed`` (use ``item_scraped`` instead)
|
||||
|
||||
- ``request_received`` (use ``request_scheduled`` instead)
|
||||
|
||||
(:issue:`6654`, :issue:`6655`)
|
||||
|
||||
Deprecations
|
||||
~~~~~~~~~~~~
|
||||
|
||||
- :ref:`Spider middlewares that don't support asynchronous spider output
|
||||
<sync-async-spider-middleware>` are deprecated. The async iterable
|
||||
downgrading feature, needed for using such middlewares with asynchronous
|
||||
callbacks and with other spider middlewares that produce asynchronous
|
||||
iterables, is also deprecated. Please update all such middlewares to
|
||||
support asynchronous spider output.
|
||||
(:issue:`6664`)
|
||||
|
||||
- Functions that were imported from :mod:`w3lib.url` and re-exported in
|
||||
:mod:`scrapy.utils.url` are now deprecated, you should import them from
|
||||
``w3lib.url`` directly. They are:
|
||||
:mod:`w3lib.url` directly. They are:
|
||||
|
||||
- ``scrapy.utils.url.add_or_replace_parameter()``
|
||||
|
||||
|
|
@ -65,10 +112,6 @@ Deprecations
|
|||
|
||||
- ``scrapy.utils.url.url_query_parameter()``
|
||||
|
||||
- ``scrapy.utils.url._unquotepath()``
|
||||
|
||||
- ``scrapy.utils.url._safe_chars`` attribute
|
||||
|
||||
(:issue:`4577`, :issue:`6583`, :issue:`6586`)
|
||||
|
||||
- HTTP/1.0 support code is deprecated. It was disabled by default and
|
||||
|
|
@ -87,6 +130,37 @@ Deprecations
|
|||
|
||||
(:issue:`6634`)
|
||||
|
||||
- The following modules and functions used only in tests are deprecated:
|
||||
|
||||
- the ``scrapy/utils/testproc`` module
|
||||
|
||||
- the ``scrapy/utils/testsite`` module
|
||||
|
||||
- ``scrapy.utils.test.assert_gcs_environ()``
|
||||
|
||||
- ``scrapy.utils.test.get_ftp_content_and_delete()``
|
||||
|
||||
- ``scrapy.utils.test.get_gcs_content_and_delete()``
|
||||
|
||||
- ``scrapy.utils.test.mock_google_cloud_storage()``
|
||||
|
||||
- ``scrapy.utils.test.skip_if_no_boto()``
|
||||
|
||||
If you need to use them in your tests or code, you can copy the code from Scrapy.
|
||||
(:issue:`6696`)
|
||||
|
||||
- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` is
|
||||
deprecated. It was disabled by default and isn't useful for most of the
|
||||
existing websites.
|
||||
(:issue:`6523`, :issue:`6651`, :issue:`6656`)
|
||||
|
||||
- ``scrapy.utils.url.escape_ajax()`` is deprecated.
|
||||
(:issue:`6523`, :issue:`6651`)
|
||||
|
||||
- ``scrapy.spiders.init.InitSpider`` is deprecated. If you find it useful,
|
||||
you can copy its code from Scrapy.
|
||||
(:issue:`6708`, :issue:`6714`)
|
||||
|
||||
- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use
|
||||
:func:`scrapy.utils.versions.get_versions()` instead.
|
||||
(:issue:`6582`)
|
||||
|
|
@ -105,29 +179,63 @@ New features
|
|||
by other code that checks :attr:`~scrapy.Request.dont_filter`).
|
||||
(:issue:`3690`, :issue:`6151`, :issue:`6366`)
|
||||
|
||||
- Added an optional base class for spider middlewares,
|
||||
:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which can be
|
||||
helpful for writing :ref:`universal spider middlewares
|
||||
<universal-spider-middleware>` without boilerplate and code duplication.
|
||||
The built-in spider middlewares now inherit from this class.
|
||||
(:issue:`6693`)
|
||||
|
||||
- :ref:`Scrapy add-ons <topics-addons>` can now define a class method called
|
||||
``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings
|
||||
<pre-crawler-settings>`.
|
||||
(:issue:`6544`, :issue:`6568`)
|
||||
|
||||
- Added :ref:`helpers <priority-dict-helpers>` for modifying :ref:`component
|
||||
priority dictionary <component-priority-dictionaries>` settings.
|
||||
(:issue:`6614`)
|
||||
|
||||
- Responses that use an unknown/unsupported encoding now produce a warning.
|
||||
If Scrapy knows that installing an additional package (such as brotli_)
|
||||
will allow decoding the response, that will be mentioned in the warning.
|
||||
(:issue:`4697`, :issue:`6618`)
|
||||
|
||||
- Added the ``spider_exceptions/count`` stat which tracks the total count of
|
||||
exceptions (tracked also by per-type ``spider_exceptions/*`` stats).
|
||||
(:issue:`6739`, :issue:`6740`)
|
||||
|
||||
- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the
|
||||
:attr:`scrapy.exceptions.DropItem.log_level` attribute that allow
|
||||
customizing the log level of the message that is logged when an item is
|
||||
dropped.
|
||||
(:issue:`6603`, :issue:`6608`)
|
||||
|
||||
- Added support for the ``-b, --cookie`` curl argument to
|
||||
:meth:`scrapy.Request.from_curl`.
|
||||
(:issue:`6684`)
|
||||
|
||||
- Added the :setting:`LOG_VERSIONS` setting that allows customizing the
|
||||
list of software which versions are logged when the spider starts.
|
||||
list of software whose versions are logged when the spider starts.
|
||||
(:issue:`6582`)
|
||||
|
||||
- Added the :setting:`WARN_ON_GENERATOR_RETURN_VALUE` setting that allows
|
||||
disabling run time analysis of callback code used to warn about incorrect
|
||||
``return`` statements in generator-based callbacks. You may need to disable
|
||||
this setting if this analysis breaks on your callback code.
|
||||
(:issue:`6731`, :issue:`6738`)
|
||||
|
||||
Improvements
|
||||
~~~~~~~~~~~~
|
||||
|
||||
- Removed or postponed some calls of :func:`itemadapter.is_item` to increase
|
||||
performance.
|
||||
(:issue:`6719`)
|
||||
|
||||
- Improved the error message when running a ``scrapy`` command that requires
|
||||
a project (such as ``scrapy crawl``) outside of a project directory.
|
||||
(:issue:`2349`, :issue:`3426`)
|
||||
|
||||
- An empty :setting:`ADDONS` setting added to the ``settings.py`` template
|
||||
- Added an empty :setting:`ADDONS` setting to the ``settings.py`` template
|
||||
for new projects.
|
||||
(:issue:`6587`)
|
||||
|
||||
|
|
@ -145,12 +253,46 @@ Bug fixes
|
|||
- Fixed an error running ``scrapy bench``.
|
||||
(:issue:`6632`, :issue:`6633`)
|
||||
|
||||
- Fixed duplicated log messages about the reactor and the event loop.
|
||||
(:issue:`6636`, :issue:`6657`)
|
||||
|
||||
- Fixed resolving type annotations of ``SitemapSpider._parse_sitemap()`` at
|
||||
run time, required by tools such as scrapy-poet_.
|
||||
(:issue:`6665`, :issue:`6671`)
|
||||
|
||||
.. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet
|
||||
|
||||
- Calling :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` without
|
||||
an installed reactor now raises an exception instead of installing a
|
||||
reactor.
|
||||
(:issue:`6732`, :issue:`6735`)
|
||||
|
||||
- Restored support for the ``x-gzip`` content encoding.
|
||||
(:issue:`6618`)
|
||||
|
||||
Documentation
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
- Improved the contribution docs.
|
||||
- Improved the :ref:`docs <sync-async-spider-middleware>` about asynchronous
|
||||
iterable support in spider middlewares.
|
||||
(:issue:`6688`)
|
||||
|
||||
- Improved the :ref:`docs <coroutine-deferred-apis>` about using
|
||||
:class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based
|
||||
code.
|
||||
(:issue:`6734`)
|
||||
|
||||
- Improved the :ref:`contribution docs <topics-contributing>`.
|
||||
(:issue:`6561`, :issue:`6575`)
|
||||
|
||||
- Removed the ``Splash`` recommendation from the :ref:`headless browser
|
||||
<topics-headless-browsing>` suggestion. We no longer recommend using
|
||||
``Splash`` and recommend using other headless browser solutions instead.
|
||||
(:issue:`6642`, :issue:`6701`)
|
||||
|
||||
- Added the dark mode to the HTML documentation.
|
||||
(:issue:`6653`)
|
||||
|
||||
- Other documentation improvements and fixes.
|
||||
(:issue:`4151`,
|
||||
:issue:`6526`,
|
||||
|
|
@ -158,7 +300,9 @@ Documentation
|
|||
:issue:`6621`,
|
||||
:issue:`6622`,
|
||||
:issue:`6623`,
|
||||
:issue:`6624`)
|
||||
:issue:`6624`,
|
||||
:issue:`6721`,
|
||||
:issue:`6723`)
|
||||
|
||||
Packaging
|
||||
~~~~~~~~~
|
||||
|
|
@ -169,12 +313,20 @@ Packaging
|
|||
Quality assurance
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Replaced most linters with ``ruff``.
|
||||
(:issue:`6565`, :issue:`6576`, :issue:`6577`, :issue:`6581`, :issue:`6584`,
|
||||
:issue:`6595`, :issue:`6601`, :issue:`6631`)
|
||||
- Replaced most linters with ruff_.
|
||||
(:issue:`6565`,
|
||||
:issue:`6576`,
|
||||
:issue:`6577`,
|
||||
:issue:`6581`,
|
||||
:issue:`6584`,
|
||||
:issue:`6595`,
|
||||
:issue:`6601`,
|
||||
:issue:`6631`)
|
||||
|
||||
.. _ruff: https://docs.astral.sh/ruff/
|
||||
|
||||
- Improved accuracy and performance of collecting test coverage.
|
||||
(:issue:`6567`)
|
||||
(:issue:`6255`, :issue:`6610`)
|
||||
|
||||
- Fixed an error that prevented running tests from directories other than the
|
||||
top level source directory.
|
||||
|
|
@ -185,10 +337,28 @@ Quality assurance
|
|||
(:issue:`6637`, :issue:`6648`)
|
||||
|
||||
- Fixed tests that were running the same test code more than once.
|
||||
(:issue:`6646`)
|
||||
(:issue:`6646`, :issue:`6647`, :issue:`6650`)
|
||||
|
||||
- Refactored tests to use more ``pytest`` features instead of ``unittest``
|
||||
ones where possible.
|
||||
(:issue:`6678`,
|
||||
:issue:`6680`,
|
||||
:issue:`6695`,
|
||||
:issue:`6699`,
|
||||
:issue:`6700`,
|
||||
:issue:`6702`,
|
||||
:issue:`6709`,
|
||||
:issue:`6710`,
|
||||
:issue:`6711`,
|
||||
:issue:`6712`,
|
||||
:issue:`6725`)
|
||||
|
||||
- Type hints improvements and fixes.
|
||||
(:issue:`6578`, :issue:`6579`, :issue:`6593`, :issue:`6605`)
|
||||
(:issue:`6578`,
|
||||
:issue:`6579`,
|
||||
:issue:`6593`,
|
||||
:issue:`6605`,
|
||||
:issue:`6694`)
|
||||
|
||||
- CI and test improvements and fixes.
|
||||
(:issue:`5360`,
|
||||
|
|
@ -200,7 +370,16 @@ Quality assurance
|
|||
:issue:`6609`,
|
||||
:issue:`6613`,
|
||||
:issue:`6619`,
|
||||
:issue:`6626`)
|
||||
:issue:`6626`,
|
||||
:issue:`6679`,
|
||||
:issue:`6703`,
|
||||
:issue:`6704`,
|
||||
:issue:`6716`,
|
||||
:issue:`6720`,
|
||||
:issue:`6722`,
|
||||
:issue:`6724`,
|
||||
:issue:`6741`,
|
||||
:issue:`6743`)
|
||||
|
||||
- Code cleanups.
|
||||
(:issue:`6600`,
|
||||
|
|
@ -806,8 +985,6 @@ Bug fixes
|
|||
- Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in
|
||||
favor of brotli_. (:issue:`6261`)
|
||||
|
||||
.. _brotli: https://github.com/google/brotli
|
||||
|
||||
.. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli
|
||||
instead if you can.
|
||||
|
||||
|
|
@ -2391,8 +2568,6 @@ Scrapy 2.5.1 (2021-10-05)
|
|||
need to upgrade scrapy-splash to a greater version for it to continue to
|
||||
work.
|
||||
|
||||
.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
|
||||
|
||||
|
||||
.. _release-2.5.0:
|
||||
|
||||
|
|
|
|||
|
|
@ -124,6 +124,8 @@ Set some basic configuration:
|
|||
"ITEM_PIPELINES", MyPipeline, 200
|
||||
)
|
||||
|
||||
.. _priority-dict-helpers:
|
||||
|
||||
.. tip:: When editing a :ref:`component priority dictionary
|
||||
<component-priority-dictionaries>` setting, like :setting:`ITEM_PIPELINES`,
|
||||
consider using setting methods like
|
||||
|
|
|
|||
|
|
@ -344,3 +344,9 @@ For example:
|
|||
feature will be removed, and all spider middlewares will be expected
|
||||
to define their ``process_spider_output`` method as an asynchronous
|
||||
generator.
|
||||
|
||||
Since 2.13.0, Scrapy provides a base class,
|
||||
:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements
|
||||
the ``process_spider_output()`` and ``process_spider_output_async()`` methods,
|
||||
so instead of duplicating the processing code you can override the
|
||||
``get_processed_request()`` and/or the ``get_processed_item()`` method.
|
||||
|
|
|
|||
|
|
@ -2005,7 +2005,7 @@ current platform.
|
|||
``twisted.internet.asyncioreactor.AsyncioSelectorReactor`` in the generated
|
||||
``settings.py`` file.
|
||||
|
||||
.. versionchanged:: VERSION
|
||||
.. versionchanged:: 2.13
|
||||
The default value was changed from ``None`` to
|
||||
``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``.
|
||||
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ if TYPE_CHECKING:
|
|||
class BaseSpiderMiddleware:
|
||||
"""Optional base class for spider middlewares.
|
||||
|
||||
.. versionadded:: 2.13
|
||||
|
||||
This class provides helper methods for asynchronous ``process_spider_output``
|
||||
methods. Middlewares that don't have a ``process_spider_output`` method don't need
|
||||
to use it.
|
||||
|
|
|
|||
Loading…
Reference in New Issue