diff --git a/docs/news.rst b/docs/news.rst
index 0922d3be3..05c6cbba9 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -58,13 +58,6 @@ Backward-incompatible changes
(:issue:`6136`, :issue:`7882`)
-- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now
- uses `httpx2 `__, the successor of ``httpx``,
- which the new :ref:`httpx extra ` installs together with its HTTP/2
- and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not
- installed, but it is no longer tested.
- (:issue:`7762`)
-
- ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br``
is always included in the ``Accept-Encoding`` header of requests, and
Brotli-compressed responses are always decoded. Websites may now serve
@@ -87,6 +80,27 @@ Backward-incompatible changes
the key order of each item.
(:issue:`6662`, :issue:`7824`)
+- ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed
+ exports `, the :ref:`telnet console
+ ` and the
+ :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension, now
+ serializes :class:`~datetime.datetime`, :class:`~datetime.date` and
+ :class:`~datetime.time` objects in ISO 8601 format, e.g.
+ ``2023-08-03T23:24:57.148903+00:00`` instead of ``2023-08-03 23:24:57``,
+ keeping microseconds and time zone information.
+
+ Its ``DATE_FORMAT`` and ``TIME_FORMAT`` attributes are removed.
+
+ (:issue:`2087`, :issue:`7918`)
+
+- ``scrapy.utils.trackref.live_refs`` is now a
+ :class:`~weakref.WeakKeyDictionary` instead of a
+ :class:`collections.defaultdict`, so that classes defined at run time are
+ released once they are no longer used. Reading the entry of a class with no
+ tracked instances now raises :exc:`KeyError` instead of creating and
+ returning an empty mapping.
+ (:issue:`5995`, :issue:`7922`)
+
- The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code
reading it now gets ``None`` instead of its default value, an empty list.
(:issue:`7737`)
@@ -105,28 +119,12 @@ Backward-incompatible changes
(:issue:`5570`, :issue:`7936`)
-- ``scrapy.core.http2.agent.H2ConnectionPool``,
- ``scrapy.core.http2.protocol.H2ClientProtocol`` and
- ``scrapy.core.http2.protocol.H2ClientFactory`` now take a
- :class:`~scrapy.crawler.Crawler` object where they used to take a
- :class:`~scrapy.settings.Settings` object, and
- ``scrapy.core.http2.stream.Stream`` takes an additional, required
- ``crawler`` argument.
- (:issue:`5046`, :issue:`5055`, :issue:`7896`)
-
- :setting:`FEEDS` keys and ``FEED_URI`` values that are
:class:`pathlib.Path` objects are now used as paths, instead of being
converted into ``file://`` URIs. This makes them keep working when they
contain :ref:`URI parameters ` or characters that
URI conversion would percent-encode.
- (:issue:`5794`, :issue:`6425`, :issue:`7674`)
-
-- :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware` is
- enabled by default, so crawls that copy internal
- :attr:`~scrapy.Request.meta` keys from a response into a new request now
- log a warning. See :setting:`META_COPY_WARN_SKIP_KEYS` to silence it for
- specific keys.
- (:issue:`7588`)
+ (:issue:`5794`, :issue:`6425`, :issue:`6611`, :issue:`7674`)
- :class:`~scrapy.Selector` and :attr:`TextResponse.selector
` no longer force the ``html`` selector
@@ -138,7 +136,7 @@ Backward-incompatible changes
- :ref:`AutoThrottle ` no longer sets the
``download_delay`` attribute of the running spider to define the starting
delay of download slots.
- (:issue:`7167`, :issue:`7833`)
+ (:issue:`7167`, :issue:`7175`, :issue:`7833`)
- :class:`~scrapy.spiders.XMLFeedSpider` and
:class:`~scrapy.spiders.CSVFeedSpider` no longer raise
@@ -178,7 +176,7 @@ Deprecations
Both attributes are ignored, with a different warning, when the
corresponding setting is already set at the ``spider`` priority or higher.
- (:issue:`7167`, :issue:`7833`)
+ (:issue:`7167`, :issue:`7175`, :issue:`7833`)
- The ``Spider.log()`` method is deprecated. Use the methods of
:attr:`Spider.logger ` instead.
@@ -211,6 +209,13 @@ New features
``zstd``. For example, ``pip install scrapy[s3,images]``.
(:issue:`7596`)
+- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now
+ uses `httpx2 `__, the successor of ``httpx``,
+ which the new :ref:`httpx extra ` installs together with its HTTP/2
+ and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not
+ installed, but it is no longer tested.
+ (:issue:`7762`)
+
- Added a :signal:`robots_parsed` signal, sent by
:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after
it parses a :file:`robots.txt` file. It supports :ref:`asynchronous
@@ -224,7 +229,7 @@ New features
- Added a :meth:`Request.to_curl() ` method, the
inverse of :meth:`~scrapy.Request.from_curl`.
- (:issue:`7743`, :issue:`7802`)
+ (:issue:`7743`, :issue:`7746`, :issue:`7802`)
- Added a :reqmeta:`depth_reset` request meta key that gives a request depth
0 instead of the depth of its source response plus 1.
@@ -272,6 +277,19 @@ New features
(:issue:`3463`, :issue:`4182`, :issue:`7884`)
+- :exc:`~scrapy.exceptions.CloseSpider` can now also be raised while the
+ spider is starting, e.g. from a :signal:`spider_opened` signal handler or
+ from the ``open_spider()`` method of an :ref:`item pipeline
+ `, to close the spider before it starts crawling.
+ Every component still gets started, and stopped, before the spider is
+ closed with the given reason.
+ (:issue:`3435`, :issue:`7905`)
+
+- Added an :ref:`FTPS feed storage backend `, i.e. support
+ for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed over
+ a TLS connection, verifying the certificate of the server.
+ (:issue:`4180`, :issue:`7953`)
+
- Changes to :attr:`Spider.allowed_domains `
during a crawl are now taken into account by
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, whose
@@ -333,7 +351,7 @@ Bug fixes
- :class:`~scrapy.exporters.CsvItemExporter` now warns when the fields that
it took from the first item do not cover the fields of a later item, i.e.
when it silently drops data.
- (:issue:`4002`, :issue:`7651`)
+ (:issue:`4002`, :issue:`4053`, :issue:`7613`, :issue:`7651`)
- ``GCSFeedStorage`` no longer requires the ``storage.buckets.get``
permission.
@@ -344,6 +362,12 @@ Bug fixes
without a traceback, instead of reporting them as download errors.
(:issue:`7544`, :issue:`7673`)
+- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
+ skips response header lines that have no colon, logging them at the
+ ``DEBUG`` level, as web browsers do, instead of being unable to download
+ such a response at all.
+ (:issue:`210`, :issue:`7806`)
+
- :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now sends
domain cookies to hosts without a dot in their name and to hosts given as
an IP address.
@@ -597,7 +621,8 @@ Quality assurance
:issue:`7895`,
:issue:`7906`,
:issue:`7928`,
- :issue:`7935`)
+ :issue:`7935`,
+ :issue:`7966`)
.. _release-2.17.0: