From 4ed6b3acef40ab8121f60853409ee2e3fe427965 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 14:58:42 +0200 Subject: [PATCH] WIP --- docs/news.rst | 83 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 54 insertions(+), 29 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 0922d3be3..05c6cbba9 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -58,13 +58,6 @@ Backward-incompatible changes (:issue:`6136`, :issue:`7882`) -- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now - uses `httpx2 `__, the successor of ``httpx``, - which the new :ref:`httpx extra ` installs together with its HTTP/2 - and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not - installed, but it is no longer tested. - (:issue:`7762`) - - ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br`` is always included in the ``Accept-Encoding`` header of requests, and Brotli-compressed responses are always decoded. Websites may now serve @@ -87,6 +80,27 @@ Backward-incompatible changes the key order of each item. (:issue:`6662`, :issue:`7824`) +- ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed + exports `, the :ref:`telnet console + ` and the + :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension, now + serializes :class:`~datetime.datetime`, :class:`~datetime.date` and + :class:`~datetime.time` objects in ISO 8601 format, e.g. + ``2023-08-03T23:24:57.148903+00:00`` instead of ``2023-08-03 23:24:57``, + keeping microseconds and time zone information. + + Its ``DATE_FORMAT`` and ``TIME_FORMAT`` attributes are removed. + + (:issue:`2087`, :issue:`7918`) + +- ``scrapy.utils.trackref.live_refs`` is now a + :class:`~weakref.WeakKeyDictionary` instead of a + :class:`collections.defaultdict`, so that classes defined at run time are + released once they are no longer used. Reading the entry of a class with no + tracked instances now raises :exc:`KeyError` instead of creating and + returning an empty mapping. + (:issue:`5995`, :issue:`7922`) + - The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code reading it now gets ``None`` instead of its default value, an empty list. (:issue:`7737`) @@ -105,28 +119,12 @@ Backward-incompatible changes (:issue:`5570`, :issue:`7936`) -- ``scrapy.core.http2.agent.H2ConnectionPool``, - ``scrapy.core.http2.protocol.H2ClientProtocol`` and - ``scrapy.core.http2.protocol.H2ClientFactory`` now take a - :class:`~scrapy.crawler.Crawler` object where they used to take a - :class:`~scrapy.settings.Settings` object, and - ``scrapy.core.http2.stream.Stream`` takes an additional, required - ``crawler`` argument. - (:issue:`5046`, :issue:`5055`, :issue:`7896`) - - :setting:`FEEDS` keys and ``FEED_URI`` values that are :class:`pathlib.Path` objects are now used as paths, instead of being converted into ``file://`` URIs. This makes them keep working when they contain :ref:`URI parameters ` or characters that URI conversion would percent-encode. - (:issue:`5794`, :issue:`6425`, :issue:`7674`) - -- :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware` is - enabled by default, so crawls that copy internal - :attr:`~scrapy.Request.meta` keys from a response into a new request now - log a warning. See :setting:`META_COPY_WARN_SKIP_KEYS` to silence it for - specific keys. - (:issue:`7588`) + (:issue:`5794`, :issue:`6425`, :issue:`6611`, :issue:`7674`) - :class:`~scrapy.Selector` and :attr:`TextResponse.selector ` no longer force the ``html`` selector @@ -138,7 +136,7 @@ Backward-incompatible changes - :ref:`AutoThrottle ` no longer sets the ``download_delay`` attribute of the running spider to define the starting delay of download slots. - (:issue:`7167`, :issue:`7833`) + (:issue:`7167`, :issue:`7175`, :issue:`7833`) - :class:`~scrapy.spiders.XMLFeedSpider` and :class:`~scrapy.spiders.CSVFeedSpider` no longer raise @@ -178,7 +176,7 @@ Deprecations Both attributes are ignored, with a different warning, when the corresponding setting is already set at the ``spider`` priority or higher. - (:issue:`7167`, :issue:`7833`) + (:issue:`7167`, :issue:`7175`, :issue:`7833`) - The ``Spider.log()`` method is deprecated. Use the methods of :attr:`Spider.logger ` instead. @@ -211,6 +209,13 @@ New features ``zstd``. For example, ``pip install scrapy[s3,images]``. (:issue:`7596`) +- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now + uses `httpx2 `__, the successor of ``httpx``, + which the new :ref:`httpx extra ` installs together with its HTTP/2 + and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not + installed, but it is no longer tested. + (:issue:`7762`) + - Added a :signal:`robots_parsed` signal, sent by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after it parses a :file:`robots.txt` file. It supports :ref:`asynchronous @@ -224,7 +229,7 @@ New features - Added a :meth:`Request.to_curl() ` method, the inverse of :meth:`~scrapy.Request.from_curl`. - (:issue:`7743`, :issue:`7802`) + (:issue:`7743`, :issue:`7746`, :issue:`7802`) - Added a :reqmeta:`depth_reset` request meta key that gives a request depth 0 instead of the depth of its source response plus 1. @@ -272,6 +277,19 @@ New features (:issue:`3463`, :issue:`4182`, :issue:`7884`) +- :exc:`~scrapy.exceptions.CloseSpider` can now also be raised while the + spider is starting, e.g. from a :signal:`spider_opened` signal handler or + from the ``open_spider()`` method of an :ref:`item pipeline + `, to close the spider before it starts crawling. + Every component still gets started, and stopped, before the spider is + closed with the given reason. + (:issue:`3435`, :issue:`7905`) + +- Added an :ref:`FTPS feed storage backend `, i.e. support + for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed over + a TLS connection, verifying the certificate of the server. + (:issue:`4180`, :issue:`7953`) + - Changes to :attr:`Spider.allowed_domains ` during a crawl are now taken into account by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, whose @@ -333,7 +351,7 @@ Bug fixes - :class:`~scrapy.exporters.CsvItemExporter` now warns when the fields that it took from the first item do not cover the fields of a later item, i.e. when it silently drops data. - (:issue:`4002`, :issue:`7651`) + (:issue:`4002`, :issue:`4053`, :issue:`7613`, :issue:`7651`) - ``GCSFeedStorage`` no longer requires the ``storage.buckets.get`` permission. @@ -344,6 +362,12 @@ Bug fixes without a traceback, instead of reporting them as download errors. (:issue:`7544`, :issue:`7673`) +- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now + skips response header lines that have no colon, logging them at the + ``DEBUG`` level, as web browsers do, instead of being unable to download + such a response at all. + (:issue:`210`, :issue:`7806`) + - :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now sends domain cookies to hosts without a dot in their name and to hosts given as an IP address. @@ -597,7 +621,8 @@ Quality assurance :issue:`7895`, :issue:`7906`, :issue:`7928`, - :issue:`7935`) + :issue:`7935`, + :issue:`7966`) .. _release-2.17.0: