From e70750eb67993b5623ae3dfc74afdfda0b4fe6cb Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 11:27:58 +0200 Subject: [PATCH 01/14] WIP --- docs/news.rst | 574 ++++++++++++++++++++++++++++++++++++++++++++ docs/topics/api.rst | 2 + scrapy/crawler.py | 7 + 3 files changed, 583 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index a5cc6c723..0922d3be3 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -6,6 +6,24 @@ Release notes Scrapy VERSION (unreleased) --------------------------- +Highlights: + +- ``HttpxDownloadHandler`` now uses `httpx2 `__ + +- ``brotli`` is now a required dependency, and :ref:`optional extras + ` cover the rest of the optional features + +- Late :class:`~scrapy.crawler.Crawler` attributes, such as + :attr:`~scrapy.crawler.Crawler.stats`, now raise :exc:`RuntimeError` + instead of being ``None`` before the crawl starts + +- Item exporters now export fields in declaration order + +- New :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware` + +- New :ref:`optimization ` page and :ref:`built-in stats reference + ` + Backward-incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -25,6 +43,562 @@ Backward-incompatible changes (:issue:`6585`, :issue:`7731`) +- The :attr:`~scrapy.crawler.Crawler.engine`, + :attr:`~scrapy.crawler.Crawler.extensions`, + :attr:`~scrapy.crawler.Crawler.logformatter`, + :attr:`~scrapy.crawler.Crawler.request_fingerprinter` and + :attr:`~scrapy.crawler.Crawler.stats` attributes of + :class:`~scrapy.crawler.Crawler` raise :exc:`RuntimeError` when read before + the crawl starts, instead of being ``None`` until then. + + Code that reads them from the :signal:`spider_opened` signal handler + onwards is unaffected, and no longer needs to narrow their type. Code that + checked whether they were set, e.g. ``if crawler.stats:``, must be updated, + since reading them now raises instead of returning ``None``. + + (:issue:`6136`, :issue:`7882`) + +- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now + uses `httpx2 `__, the successor of ``httpx``, + which the new :ref:`httpx extra ` installs together with its HTTP/2 + and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not + installed, but it is no longer tested. + (:issue:`7762`) + +- ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br`` + is always included in the ``Accept-Encoding`` header of requests, and + Brotli-compressed responses are always decoded. Websites may now serve + Brotli-compressed responses to crawls that previously did not advertise + support for them. + + The minimum required versions are ``brotli`` 1.2.0 and ``brotlicffi`` + 1.2.0.0. + + (:issue:`4698`, :issue:`7929`) + +- The minimum required ``queuelib`` version is now 1.6.1. + (:issue:`7874`) + +- :ref:`Item exporters ` other than + :class:`~scrapy.exporters.CsvItemExporter` now export the fields of an item + in declaration order, i.e. the order in which they are defined in the + :ref:`item class `, instead of the order in which they were + populated. :class:`dict` items, which have no declared fields, keep using + the key order of each item. + (:issue:`6662`, :issue:`7824`) + +- The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code + reading it now gets ``None`` instead of its default value, an empty list. + (:issue:`7737`) + +- ``scrapy.utils.log.logformatter_adapter()`` no longer passes the whole + :class:`dict` returned by a :ref:`log formatter ` + method as logging arguments when that ``dict`` has no ``args`` key or its + ``args`` are empty. Such messages are now logged verbatim, so a literal + ``%`` in them no longer breaks logging, and placeholders that referenced + other keys of the returned ``dict``, such as ``%(msg)s``, are no longer + interpolated. + + An ``args`` :class:`tuple` is now expanded into one logging argument per + item, so that ``%``-style placeholders work with it as they do with a + ``dict``. + + (:issue:`5570`, :issue:`7936`) + +- ``scrapy.core.http2.agent.H2ConnectionPool``, + ``scrapy.core.http2.protocol.H2ClientProtocol`` and + ``scrapy.core.http2.protocol.H2ClientFactory`` now take a + :class:`~scrapy.crawler.Crawler` object where they used to take a + :class:`~scrapy.settings.Settings` object, and + ``scrapy.core.http2.stream.Stream`` takes an additional, required + ``crawler`` argument. + (:issue:`5046`, :issue:`5055`, :issue:`7896`) + +- :setting:`FEEDS` keys and ``FEED_URI`` values that are + :class:`pathlib.Path` objects are now used as paths, instead of being + converted into ``file://`` URIs. This makes them keep working when they + contain :ref:`URI parameters ` or characters that + URI conversion would percent-encode. + (:issue:`5794`, :issue:`6425`, :issue:`7674`) + +- :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware` is + enabled by default, so crawls that copy internal + :attr:`~scrapy.Request.meta` keys from a response into a new request now + log a warning. See :setting:`META_COPY_WARN_SKIP_KEYS` to silence it for + specific keys. + (:issue:`7588`) + +- :class:`~scrapy.Selector` and :attr:`TextResponse.selector + ` no longer force the ``html`` selector + type for responses that are neither :class:`~scrapy.http.HtmlResponse` nor + :class:`~scrapy.http.XmlResponse` objects, e.g. for a JSON response. + ``parsel`` determines the type from the body in those cases instead. + (:issue:`5291`, :issue:`6025`, :issue:`7924`) + +- :ref:`AutoThrottle ` no longer sets the + ``download_delay`` attribute of the running spider to define the starting + delay of download slots. + (:issue:`7167`, :issue:`7833`) + +- :class:`~scrapy.spiders.XMLFeedSpider` and + :class:`~scrapy.spiders.CSVFeedSpider` no longer raise + :exc:`~scrapy.exceptions.NotConfigured` when ``parse_node()`` or + ``parse_row()`` is not defined; the resulting :exc:`AttributeError` is + reported instead. + (:issue:`7768`) + +- ``scrapy.pipelines.files.FileException`` moved to + ``scrapy.pipelines.media``. It is still importable from its old location. + (:issue:`7544`, :issue:`7673`) + +- The :ref:`ipython extra ` requires IPython 8.15.0 or higher, as + support for the ``IPython.frontend`` module layout of IPython older than + 1.0 is removed. + (:issue:`5447`, :issue:`7596`, :issue:`7816`) + +Deprecation removals +~~~~~~~~~~~~~~~~~~~~ + +- ``scrapy.utils.iterators.xmliter()``, deprecated since Scrapy 2.11.1 + because it is vulnerable to ReDoS attacks, is removed. Use + :func:`~scrapy.utils.iterators.xmliter_lxml` instead. + (:issue:`7765`) + +Deprecations +~~~~~~~~~~~~ + +- The ``download_delay`` spider attribute is deprecated. Use the + :setting:`DOWNLOAD_DELAY` setting, or :setting:`DOWNLOAD_SLOTS` to set a + delay for specific domains, instead. + + The ``max_concurrent_requests`` spider attribute, deprecated since Scrapy + 2.13.0, now sets the :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` setting, + which is what it always mapped to, and warns accordingly. + + Both attributes are ignored, with a different warning, when the + corresponding setting is already set at the ``spider`` priority or higher. + + (:issue:`7167`, :issue:`7833`) + +- The ``Spider.log()`` method is deprecated. Use the methods of + :attr:`Spider.logger ` instead. + (:issue:`7739`) + +- The ``scrapy.interfaces`` module and its ``ISpiderLoader`` interface are + deprecated. Custom spider loaders only need to follow + :class:`~scrapy.spiderloader.SpiderLoaderProtocol`. + (:issue:`6585`, :issue:`7731`) + +- ``scrapy.extensions.feedexport.IFeedStorage`` is deprecated. Custom feed + storages only need to follow + ``scrapy.extensions.feedexport.FeedStorageProtocol``. + (:issue:`6585`, :issue:`7731`) + +- ``scrapy.utils.python.re_rsearch()`` is deprecated. + (:issue:`7765`) + +- Setting ``request.meta["is_secure"]`` to ``False`` to send an ``s3://`` + request over plaintext HTTP is deprecated. The flag will be ignored in a + future Scrapy version. + (:issue:`7738`) + +New features +~~~~~~~~~~~~ + +- Added :ref:`optional extras ` for every optional dependency of + Scrapy: ``bpython``, ``gcs``, ``httpx``, ``images``, ``ipython``, + ``ptpython``, ``robotparser``, ``s3``, ``twisted-http2``, ``uvloop`` and + ``zstd``. For example, ``pip install scrapy[s3,images]``. + (:issue:`7596`) + +- Added a :signal:`robots_parsed` signal, sent by + :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after + it parses a :file:`robots.txt` file. It supports :ref:`asynchronous + handlers `. + + Added a :meth:`~scrapy.robotstxt.RobotParser.crawl_delay` method to + :class:`~scrapy.robotstxt.RobotParser`, implemented by all built-in + :ref:`robots.txt parsers `. + + (:issue:`7830`) + +- Added a :meth:`Request.to_curl() ` method, the + inverse of :meth:`~scrapy.Request.from_curl`. + (:issue:`7743`, :issue:`7802`) + +- Added a :reqmeta:`depth_reset` request meta key that gives a request depth + 0 instead of the depth of its source response plus 1. + (:issue:`891`, :issue:`7913`) + +- Added + :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`, + enabled by default, which warns once per crawl when a spider yields a + request carrying internal :attr:`~scrapy.Request.meta` keys that were + likely copied from ``response.meta``, and a + :setting:`META_COPY_WARN_SKIP_KEYS` setting to exclude keys from that + check. + (:issue:`7588`) + +- Added an :setting:`AWS_MAX_POOL_CONNECTIONS` setting, which defines the + connection pool size of the AWS clients of the :ref:`S3 feed storage + backend ` and the :ref:`S3 media pipeline storage + backend `, and defaults to + :setting:`REACTOR_THREADPOOL_MAXSIZE`. It is also exposed as a + ``max_pool_connections`` parameter of ``S3FeedStorage`` and as an + ``AWS_MAX_POOL_CONNECTIONS`` attribute of ``S3FilesStore``. + (:issue:`4985`, :issue:`7794`) + +- Added a :func:`scrapy.utils.asyncio.sleep` function, which works both with + and without a Twisted reactor. + (:issue:`7843`) + +- :setting:`CONCURRENT_REQUESTS` can now be set to ``0`` for no limit. + (:issue:`7840`) + +- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now sends + the :signal:`bytes_received` and :signal:`headers_received` signals, and + supports :exc:`~scrapy.exceptions.StopDownload`. + (:issue:`5046`, :issue:`5055`, :issue:`7896`) + +- An exception raised by :meth:`Spider.start() ` is now + reported through the :signal:`spider_error` signal and the + :stat:`spider_exceptions/count` and :stat:`spider_exceptions/{exception}` + stats, and closes the spider with the new ``start_error`` + :stat:`finish_reason` instead of ``finished``. See :ref:`start-error`. + + :exc:`~scrapy.exceptions.CloseSpider` raised from :meth:`Spider.start() + ` now closes the spider with the given reason, instead + of being reported as a start error. + + (:issue:`3463`, :issue:`4182`, :issue:`7884`) + +- Changes to :attr:`Spider.allowed_domains ` + during a crawl are now taken into account by + :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, whose + :meth:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware.should_follow` + method is now documented as the way to implement a different offsite + policy. + (:issue:`3257`, :issue:`3412`, :issue:`7903`, :issue:`7912`) + +- :class:`~scrapy.settings.BaseSettings` methods that take settings, such as + :meth:`~scrapy.settings.BaseSettings.update` and the ``settings`` parameter + of crawler classes, now also accept an iterable of ``(name, value)`` + tuples. + (:issue:`7759`, :issue:`7763`) + +- The ``cookies`` parameter of :class:`~scrapy.Request` now also accepts + :class:`bool`, :class:`float` and :class:`int` values, and the ``formdata`` + parameter of :class:`~scrapy.FormRequest` now accepts any mapping or + iterable of key-value pairs. + (:issue:`7858`, :issue:`7864`) + +- Added a ``scrapy.utils.reactorless.uninstall_reactor_import_hook()`` + function, which :meth:`AsyncCrawlerProcess.start() + ` now uses to uninstall the + :mod:`twisted.internet.reactor` import hook when it exits. + (:issue:`7747`) + +- Added the :stat:`depth/request_ignored_count` and + :stat:`httpcache/retrieve_error` stats. + (:issue:`1308`, :issue:`2222`, :issue:`7805`, :issue:`7916`) + +- The :meth:`~scrapy.exporters.BaseItemExporter.get_serialized_fields` method + of :ref:`item exporters `, previously named + ``_get_serialized_fields()``, is now public and documented, for + :ref:`custom item exporters ` to use. + (:issue:`5706`, :issue:`7931`) + +Bug fixes +~~~~~~~~~ + +- :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now + logs a warning and handles the request as a cache miss when reading a cache + entry raises an exception, e.g. because the entry is corrupted, instead of + letting the exception propagate. It also counts those entries in the new + :stat:`httpcache/retrieve_error` stat. + (:issue:`2222`, :issue:`7805`) + +- :ref:`Feed URIs ` now only expand ``%(...)s`` + parameters, keeping any other percent character as is, so that + percent-encoded URIs, e.g. one with ``%20`` in a path or with + percent-encoded FTP credentials, are no longer misinterpreted as + printf-style formatting directives. + (:issue:`5794`, :issue:`6425`, :issue:`7674`) + +- :ref:`Feed exports ` now start storing a + :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` batch as soon as it is complete, + instead of waiting until the spider closes. + (:issue:`7730`, :issue:`7733`) + +- :class:`~scrapy.exporters.CsvItemExporter` now warns when the fields that + it took from the first item do not cover the fields of a later item, i.e. + when it silently drops data. + (:issue:`4002`, :issue:`7651`) + +- ``GCSFeedStorage`` no longer requires the ``storage.buckets.get`` + permission. + (:issue:`5475`, :issue:`7945`) + +- :ref:`Media pipelines ` now log media requests that + were filtered out, e.g. as offsite requests, at the ``DEBUG`` level and + without a traceback, instead of reporting them as download errors. + (:issue:`7544`, :issue:`7673`) + +- :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now sends + domain cookies to hosts without a dot in their name and to hosts given as + an IP address. + (:issue:`6410`, :issue:`7900`) + +- :meth:`TextResponse.json() ` now decodes + bodies that are not valid UTF-8, UTF-16 or UTF-32 using + :attr:`TextResponse.encoding `, instead + of raising :exc:`UnicodeDecodeError`. + (:issue:`6456`, :issue:`7897`) + +- ``scrapy.resolver.CachingHostnameResolver`` now caches addresses without a + port, and sets the requested port on cache hits, so that a cached address + no longer carries the port of the request that populated the cache. + (:issue:`6442`, :issue:`7772`) + +- :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` now removes the + directory of a download slot from the :setting:`JOBDIR` directory once that + slot is drained. + (:issue:`5275`, :issue:`7955`) + +- :class:`~scrapy.extensions.telnet.TelnetConsole` no longer raises an + exception on shutdown when it could not listen on any of the + :setting:`TELNETCONSOLE_PORT` ports. + (:issue:`2702`, :issue:`7910`) + +- The :setting:`DOWNLOAD_WARNSIZE` warning is no longer logged twice for a + response whose ``Content-Length`` header already exceeded the limit. + (:issue:`2476`, :issue:`7963`) + +- :class:`HttpCompressionMiddleware + ` + now logs a warning when it drops a response for exceeding + :setting:`DOWNLOAD_MAXSIZE` during decompression. + (:issue:`6616`, :issue:`7742`) + +- :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware` now logs only the + first request ignored for exceeding :setting:`DEPTH_LIMIT`, and counts them + all in the new :stat:`depth/request_ignored_count` stat. + (:issue:`1308`, :issue:`7916`) + +- :command:`parse` now sets the callback it uses on the request of the + response it passes to that callback. + (:issue:`3095`, :issue:`7803`) + +- The IPython :ref:`shell ` now works when an asyncio event + loop is already running in the same thread, e.g. when calling + ``scrapy.shell.inspect_response()`` from a callback while using the asyncio + reactor. + (:issue:`5447`, :issue:`7816`) + +- :meth:`Request.from_curl() ` now merges repeated + ``-d``, ``--data`` and ``--data-raw`` options into a single body joined + with ``&``, as curl does, instead of keeping only the last one. + (:issue:`7728`) + +- The ``copy()`` method and the ``|=`` operator of + ``scrapy.utils.datatypes.CaseInsensitiveDict`` no longer leave the internal + mapping of original key spellings shared or out of date. + (:issue:`7783`) + +- :meth:`ExecutionEngine.download_async() + ` no longer recurses + once per returned request, e.g. once per redirect. + (:issue:`7544`, :issue:`7673`) + +- :class:`LinkExtractor ` + now canonicalizes each extracted URL once instead of twice when + ``canonicalize`` is ``True``. + (:issue:`7961`) + +- Fixed :exc:`NameError` exceptions on Python 3.14, where :pep:`649` made + annotation evaluation lazy, when inspecting the signature of a callable + with annotations imported only for type checking. + (:issue:`7796`, :issue:`7818`) + +- ``scrapy.utils.decorators.deprecated`` can now be used both as + ``@deprecated`` and as ``@deprecated(...)`` without confusing type + checkers. + (:issue:`7797`) + +Documentation +~~~~~~~~~~~~~ + +- Added a :ref:`built-in stats reference `, covering + every stat that Scrapy sets. + (:issue:`6351`, :issue:`7814`) + +- Replaced the broad crawls page with a new :ref:`optimization ` + page, about finding the bottleneck of a crawl before changing any setting, + which covers :ref:`broad crawls ` as one of its sections. + (:issue:`4737`, :issue:`7938`) + +- Added a :ref:`cookies ` page, which gathers what used to be + spread across the request and downloader middleware pages. + (:issue:`7947`) + +- Added :ref:`callbacks ` and :ref:`errbacks ` sections + to the request and response page, covering :ref:`callback assignment + `, :ref:`how to write a callback ` + and :ref:`supported callback output `. + (:issue:`5054`, :issue:`6437`, :issue:`7821`, :issue:`7898`) + +- Documented the :ref:`optional extras ` of Scrapy, and which feature + each of them enables. + (:issue:`7596`) + +- Documented :ref:`how to write an item exporter `, + :ref:`how to test an item pipeline `, :ref:`how to + download a request from a downloader middleware `, :ref:`how + to name media files after the response `, :ref:`how + to add objects to the shell ` and :ref:`how to run + spiders inside an existing application ` or :ref:`in a + Jupyter notebook `. + (:issue:`1199`, + :issue:`2594`, + :issue:`5706`, + :issue:`6554`, + :issue:`6594`, + :issue:`7751`, + :issue:`7872`, + :issue:`7876`, + :issue:`7889`, + :issue:`7909`, + :issue:`7931`) + +- Documented the :ref:`memory use of response parsing + ` and the :ref:`parser limits + ` that Scrapy lifts, in the security page. + (:issue:`5700`, :issue:`7930`) + +- Documented that :ref:`signal handlers run in an undefined order + `, that :signal:`scheduler_empty` must only be awaited from + :meth:`~scrapy.Spider.start`, that concurrency and politeness settings + apply per crawler when :ref:`running multiple spiders in the same process + `, and that a :setting:`JOBDIR` directory cannot be + shared across Scrapy versions. + (:issue:`3191`, + :issue:`5330`, + :issue:`5522`, + :issue:`7861`, + :issue:`7883`, + :issue:`7907`, + :issue:`7941`) + +- Switched several API references to autodoc, so that they are generated from + the docstrings: contracts, download handlers, exceptions, spider loaders, + stats collectors, ``trackref``, and the depth and offsite middlewares. + (:issue:`7767`, + :issue:`7769`, + :issue:`7771`, + :issue:`7775`, + :issue:`7871`, + :issue:`7903`, + :issue:`7913`) + +- Many other corrections and improvements. + (:issue:`4589`, + :issue:`4796`, + :issue:`5532`, + :issue:`5548`, + :issue:`6053`, + :issue:`6184`, + :issue:`6627`, + :issue:`6787`, + :issue:`6943`, + :issue:`6989`, + :issue:`7710`, + :issue:`7725`, + :issue:`7737`, + :issue:`7774`, + :issue:`7777`, + :issue:`7779`, + :issue:`7780`, + :issue:`7817`, + :issue:`7832`, + :issue:`7835`, + :issue:`7862`, + :issue:`7875`, + :issue:`7880`, + :issue:`7890`, + :issue:`7917`, + :issue:`7939`, + :issue:`7940`, + :issue:`7962`, + :issue:`7965`) + +Quality assurance +~~~~~~~~~~~~~~~~~ + +- Added CPU benchmarks, tracked on CodSpeed, and a ``benchmark`` tox + environment to run them. + (:issue:`7831`, + :issue:`7839`, + :issue:`7870`, + :issue:`7887`, + :issue:`7914`, + :issue:`7954`) + +- Added a nightly job that runs the test suite against the development + branches of dependencies, and a ``vcs-deps`` tox environment for it. + (:issue:`5291`, :issue:`6025`, :issue:`7924`, :issue:`7960`) + +- Tests that need a proxy server now look for a ``mitmdump`` executable, from + the ``PATH``, from `uv `__ or from the + ``MITMDUMP`` environment variable, instead of requiring mitmproxy to be + installed in the test environment. The ``mitmproxy`` tox environment is + gone as a result. + (:issue:`7437`, :issue:`7720`) + +- Dropped the ``testfixtures`` test dependency. + (:issue:`6478`, :issue:`7793`) + +- Type hints improvements and fixes. + (:issue:`7712`, + :issue:`7785`, + :issue:`7858`, + :issue:`7864`, + :issue:`7865`, + :issue:`7867`) + +- CI and test improvements and fixes. + (:issue:`5620`, + :issue:`5837`, + :issue:`6478`, + :issue:`6794`, + :issue:`7724`, + :issue:`7727`, + :issue:`7736`, + :issue:`7741`, + :issue:`7749`, + :issue:`7753`, + :issue:`7755`, + :issue:`7768`, + :issue:`7778`, + :issue:`7782`, + :issue:`7792`, + :issue:`7795`, + :issue:`7797`, + :issue:`7798`, + :issue:`7809`, + :issue:`7829`, + :issue:`7834`, + :issue:`7836`, + :issue:`7838`, + :issue:`7844`, + :issue:`7848`, + :issue:`7853`, + :issue:`7857`, + :issue:`7863`, + :issue:`7895`, + :issue:`7906`, + :issue:`7928`, + :issue:`7935`) + .. _release-2.17.0: Scrapy 2.17.0 (2026-07-07) diff --git a/docs/topics/api.rst b/docs/topics/api.rst index 598edfeb5..1f07440b7 100644 --- a/docs/topics/api.rst +++ b/docs/topics/api.rst @@ -82,6 +82,8 @@ how you :ref:`configure the downloader middlewares For the API see :class:`~scrapy.statscollectors.StatsCollector` class. + .. autoattribute:: logformatter + .. attribute:: extensions The extension manager that keeps track of enabled extensions. diff --git a/scrapy/crawler.py b/scrapy/crawler.py index 44c4ffdcf..fa481bd42 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -102,6 +102,13 @@ class Crawler: engine: _LateAttribute[ExecutionEngine] = _LateAttribute() extensions: _LateAttribute[ExtensionManager] = _LateAttribute() logformatter: _LateAttribute[LogFormatter] = _LateAttribute() + """The log formatter of this crawler. + + This is used from extensions & middlewares to build the messages that they + log about crawling events. + + For the API see the :class:`~scrapy.logformatter.LogFormatter` class. + """ request_fingerprinter: _LateAttribute[RequestFingerprinterProtocol] = ( _LateAttribute() ) From 4ed6b3acef40ab8121f60853409ee2e3fe427965 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 14:58:42 +0200 Subject: [PATCH 02/14] WIP --- docs/news.rst | 83 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 54 insertions(+), 29 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 0922d3be3..05c6cbba9 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -58,13 +58,6 @@ Backward-incompatible changes (:issue:`6136`, :issue:`7882`) -- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now - uses `httpx2 `__, the successor of ``httpx``, - which the new :ref:`httpx extra ` installs together with its HTTP/2 - and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not - installed, but it is no longer tested. - (:issue:`7762`) - - ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br`` is always included in the ``Accept-Encoding`` header of requests, and Brotli-compressed responses are always decoded. Websites may now serve @@ -87,6 +80,27 @@ Backward-incompatible changes the key order of each item. (:issue:`6662`, :issue:`7824`) +- ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed + exports `, the :ref:`telnet console + ` and the + :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension, now + serializes :class:`~datetime.datetime`, :class:`~datetime.date` and + :class:`~datetime.time` objects in ISO 8601 format, e.g. + ``2023-08-03T23:24:57.148903+00:00`` instead of ``2023-08-03 23:24:57``, + keeping microseconds and time zone information. + + Its ``DATE_FORMAT`` and ``TIME_FORMAT`` attributes are removed. + + (:issue:`2087`, :issue:`7918`) + +- ``scrapy.utils.trackref.live_refs`` is now a + :class:`~weakref.WeakKeyDictionary` instead of a + :class:`collections.defaultdict`, so that classes defined at run time are + released once they are no longer used. Reading the entry of a class with no + tracked instances now raises :exc:`KeyError` instead of creating and + returning an empty mapping. + (:issue:`5995`, :issue:`7922`) + - The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code reading it now gets ``None`` instead of its default value, an empty list. (:issue:`7737`) @@ -105,28 +119,12 @@ Backward-incompatible changes (:issue:`5570`, :issue:`7936`) -- ``scrapy.core.http2.agent.H2ConnectionPool``, - ``scrapy.core.http2.protocol.H2ClientProtocol`` and - ``scrapy.core.http2.protocol.H2ClientFactory`` now take a - :class:`~scrapy.crawler.Crawler` object where they used to take a - :class:`~scrapy.settings.Settings` object, and - ``scrapy.core.http2.stream.Stream`` takes an additional, required - ``crawler`` argument. - (:issue:`5046`, :issue:`5055`, :issue:`7896`) - - :setting:`FEEDS` keys and ``FEED_URI`` values that are :class:`pathlib.Path` objects are now used as paths, instead of being converted into ``file://`` URIs. This makes them keep working when they contain :ref:`URI parameters ` or characters that URI conversion would percent-encode. - (:issue:`5794`, :issue:`6425`, :issue:`7674`) - -- :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware` is - enabled by default, so crawls that copy internal - :attr:`~scrapy.Request.meta` keys from a response into a new request now - log a warning. See :setting:`META_COPY_WARN_SKIP_KEYS` to silence it for - specific keys. - (:issue:`7588`) + (:issue:`5794`, :issue:`6425`, :issue:`6611`, :issue:`7674`) - :class:`~scrapy.Selector` and :attr:`TextResponse.selector ` no longer force the ``html`` selector @@ -138,7 +136,7 @@ Backward-incompatible changes - :ref:`AutoThrottle ` no longer sets the ``download_delay`` attribute of the running spider to define the starting delay of download slots. - (:issue:`7167`, :issue:`7833`) + (:issue:`7167`, :issue:`7175`, :issue:`7833`) - :class:`~scrapy.spiders.XMLFeedSpider` and :class:`~scrapy.spiders.CSVFeedSpider` no longer raise @@ -178,7 +176,7 @@ Deprecations Both attributes are ignored, with a different warning, when the corresponding setting is already set at the ``spider`` priority or higher. - (:issue:`7167`, :issue:`7833`) + (:issue:`7167`, :issue:`7175`, :issue:`7833`) - The ``Spider.log()`` method is deprecated. Use the methods of :attr:`Spider.logger ` instead. @@ -211,6 +209,13 @@ New features ``zstd``. For example, ``pip install scrapy[s3,images]``. (:issue:`7596`) +- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now + uses `httpx2 `__, the successor of ``httpx``, + which the new :ref:`httpx extra ` installs together with its HTTP/2 + and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not + installed, but it is no longer tested. + (:issue:`7762`) + - Added a :signal:`robots_parsed` signal, sent by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after it parses a :file:`robots.txt` file. It supports :ref:`asynchronous @@ -224,7 +229,7 @@ New features - Added a :meth:`Request.to_curl() ` method, the inverse of :meth:`~scrapy.Request.from_curl`. - (:issue:`7743`, :issue:`7802`) + (:issue:`7743`, :issue:`7746`, :issue:`7802`) - Added a :reqmeta:`depth_reset` request meta key that gives a request depth 0 instead of the depth of its source response plus 1. @@ -272,6 +277,19 @@ New features (:issue:`3463`, :issue:`4182`, :issue:`7884`) +- :exc:`~scrapy.exceptions.CloseSpider` can now also be raised while the + spider is starting, e.g. from a :signal:`spider_opened` signal handler or + from the ``open_spider()`` method of an :ref:`item pipeline + `, to close the spider before it starts crawling. + Every component still gets started, and stopped, before the spider is + closed with the given reason. + (:issue:`3435`, :issue:`7905`) + +- Added an :ref:`FTPS feed storage backend `, i.e. support + for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed over + a TLS connection, verifying the certificate of the server. + (:issue:`4180`, :issue:`7953`) + - Changes to :attr:`Spider.allowed_domains ` during a crawl are now taken into account by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, whose @@ -333,7 +351,7 @@ Bug fixes - :class:`~scrapy.exporters.CsvItemExporter` now warns when the fields that it took from the first item do not cover the fields of a later item, i.e. when it silently drops data. - (:issue:`4002`, :issue:`7651`) + (:issue:`4002`, :issue:`4053`, :issue:`7613`, :issue:`7651`) - ``GCSFeedStorage`` no longer requires the ``storage.buckets.get`` permission. @@ -344,6 +362,12 @@ Bug fixes without a traceback, instead of reporting them as download errors. (:issue:`7544`, :issue:`7673`) +- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now + skips response header lines that have no colon, logging them at the + ``DEBUG`` level, as web browsers do, instead of being unable to download + such a response at all. + (:issue:`210`, :issue:`7806`) + - :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now sends domain cookies to hosts without a dot in their name and to hosts given as an IP address. @@ -597,7 +621,8 @@ Quality assurance :issue:`7895`, :issue:`7906`, :issue:`7928`, - :issue:`7935`) + :issue:`7935`, + :issue:`7966`) .. _release-2.17.0: From 1b9f965b659d9391c0b31da8cbcbcce4ab9920d8 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 17:39:30 +0200 Subject: [PATCH 03/14] Clarify the user implact of the AutoThrottle change --- docs/news.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/news.rst b/docs/news.rst index 05c6cbba9..45567f324 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -135,7 +135,8 @@ Backward-incompatible changes - :ref:`AutoThrottle ` no longer sets the ``download_delay`` attribute of the running spider to define the starting - delay of download slots. + delay of download slots. The starting delay is still applied, but code + that reads that attribute at run time no longer sees it. (:issue:`7167`, :issue:`7175`, :issue:`7833`) - :class:`~scrapy.spiders.XMLFeedSpider` and From db5a243b9fae80c3378e3ede4378c07012b10641 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 18:22:14 +0200 Subject: [PATCH 04/14] Fix IPython wording --- docs/news.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 45567f324..7080892b3 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -150,9 +150,8 @@ Backward-incompatible changes ``scrapy.pipelines.media``. It is still importable from its old location. (:issue:`7544`, :issue:`7673`) -- The :ref:`ipython extra ` requires IPython 8.15.0 or higher, as - support for the ``IPython.frontend`` module layout of IPython older than - 1.0 is removed. +- The IPython :ref:`shell ` requires IPython 8.15.0 or higher. + Install the :ref:`ipython extra ` to get a compatible version. (:issue:`5447`, :issue:`7596`, :issue:`7816`) Deprecation removals From 737a1c15c7e7d731f6f3cbd6c3c9d95e8cbec2f0 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 18:58:07 +0200 Subject: [PATCH 05/14] WIP --- docs/news.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index 7080892b3..3ae9ae6e1 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -41,6 +41,10 @@ Backward-incompatible changes :class:`~scrapy.extensions.feedexport.StdoutFeedStorage` are no longer marked as implementing the ``IFeedStorage`` interface. + - :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no + longer checks that the ``DOWNLOADER_CLIENTCONTEXTFACTORY`` class + implements the ``IPolicyForHTTPS`` interface. + (:issue:`6585`, :issue:`7731`) - The :attr:`~scrapy.crawler.Crawler.engine`, @@ -326,6 +330,12 @@ New features :ref:`custom item exporters ` to use. (:issue:`5706`, :issue:`7931`) +- Log formatters (:setting:`LOG_FORMATTER`), item processors + (``ITEM_PROCESSOR``) and :ref:`robots.txt parsers ` + (:setting:`ROBOTSTXT_PARSER`) are now built as :ref:`components + `, so they no longer need a ``from_crawler()`` method. + (:issue:`7808`) + Bug fixes ~~~~~~~~~ From 5f4e08f361dfcefe808fa2b1fe501cba086584f8 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 19:33:26 +0200 Subject: [PATCH 06/14] WIP --- docs/news.rst | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 3ae9ae6e1..8bea2dbb2 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -111,11 +111,10 @@ Backward-incompatible changes - ``scrapy.utils.log.logformatter_adapter()`` no longer passes the whole :class:`dict` returned by a :ref:`log formatter ` - method as logging arguments when that ``dict`` has no ``args`` key or its - ``args`` are empty. Such messages are now logged verbatim, so a literal - ``%`` in them no longer breaks logging, and placeholders that referenced - other keys of the returned ``dict``, such as ``%(msg)s``, are no longer - interpolated. + method as logging arguments when that ``dict`` has no ``args`` key, or its + ``args`` are empty, and its ``msg`` has no ``%(name)s`` placeholders. Such + messages are now logged verbatim, so a literal ``%`` in them no longer + breaks logging. An ``args`` :class:`tuple` is now expanded into one logging argument per item, so that ``%``-style placeholders work with it as they do with a @@ -204,6 +203,13 @@ Deprecations future Scrapy version. (:issue:`7738`) +- Returning, from a :ref:`log formatter ` method, a + ``msg`` with ``%(name)s`` placeholders and no ``args`` is deprecated. Those + placeholders are still interpolated with the returned :class:`dict`, but in + a future Scrapy version the message will be logged verbatim. Return those + values under ``args`` instead. + (:issue:`5570`, :issue:`7971`) + New features ~~~~~~~~~~~~ From 6a2928f8d7d5ad69c112d597bd42ef89743e0130 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Mon, 10 Aug 2026 19:53:21 +0200 Subject: [PATCH 07/14] WIP --- docs/news.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index e797fdd5a..88f18310c 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -84,6 +84,13 @@ Backward-incompatible changes the key order of each item. (:gh:`6662`, :gh:`7824`) +- The ``_get_serialized_fields()`` method of :ref:`item exporters + ` is renamed to + :meth:`~scrapy.exporters.BaseItemExporter.get_serialized_fields`, and the + old name is gone, so :ref:`custom item exporters ` that + call or override it must be updated. + (:gh:`5706`, :gh:`7931`) + - ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed exports `, the :ref:`telnet console ` and the @@ -153,6 +160,23 @@ Backward-incompatible changes ``scrapy.pipelines.media``. It is still importable from its old location. (:gh:`7544`, :gh:`7673`) +- ``H2ConnectionPool``, ``H2ClientFactory`` and ``H2ClientProtocol``, from + ``scrapy.core.http2``, now take a :class:`~scrapy.crawler.Crawler` object + where they used to take a :class:`~scrapy.settings.Settings` object, and + ``scrapy.core.http2.stream.Stream`` takes an additional ``crawler`` + parameter, after ``protocol``. + (:gh:`7896`) + +- :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` now raises + :exc:`~scrapy.exceptions.IgnoreRequest` with a message, e.g. ``Filtered + offsite request to 'offsite.example'``, which errbacks and log messages that + report that exception now include. + (:gh:`7544`, :gh:`7673`) + +- The unused ``multiplier`` attribute of + :class:`~scrapy.extensions.periodic_log.PeriodicLog` is removed. + (:gh:`7809`) + - The IPython :ref:`shell ` requires IPython 8.15.0 or higher. Install the :ref:`ipython extra ` to get a compatible version. (:gh:`5447`, :gh:`7596`, :gh:`7816`) From acb54187af635a625f71e79180a1f774c45ffa9a Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Wed, 12 Aug 2026 05:59:45 +0200 Subject: [PATCH 08/14] Replace the VERSION placeholder with 2.18.0 --- docs/news.rst | 4 ++-- docs/topics/api.rst | 2 +- docs/topics/download-handlers.rst | 2 +- docs/topics/exporters.rst | 2 +- docs/topics/feed-exports.rst | 2 +- docs/topics/item-pipeline.rst | 2 +- docs/topics/settings.rst | 2 +- docs/topics/signals.rst | 6 +++--- docs/topics/spiders.rst | 4 ++-- scrapy/exceptions.py | 2 +- scrapy/http/request/__init__.py | 2 +- scrapy/http/response/text.py | 2 +- scrapy/robotstxt.py | 2 +- scrapy/spidermiddlewares/depth.py | 2 +- scrapy/utils/asyncio.py | 2 +- 15 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 88f18310c..6e898ac20 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,8 +3,8 @@ Release notes ============= -Scrapy VERSION (unreleased) ---------------------------- +Scrapy 2.18.0 (unreleased) +-------------------------- Highlights: diff --git a/docs/topics/api.rst b/docs/topics/api.rst index 1f07440b7..24f46f656 100644 --- a/docs/topics/api.rst +++ b/docs/topics/api.rst @@ -39,7 +39,7 @@ how you :ref:`configure the downloader middlewares :attr:`request_fingerprinter` and :attr:`stats` attributes get their value when the crawl starts, and raise :exc:`RuntimeError` when read before that. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Those attributes used to be ``None`` before getting their value. .. attribute:: request_fingerprinter diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index 433a6d139..e1b118195 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -252,7 +252,7 @@ Per-request ``bindaddress`` Yes TLS implementation ``pyOpenSSL``/``cryptography`` =========================== ================================================ -.. versionchanged:: VERSION +.. versionchanged:: 2.18.0 Bad header lines with no colon in them are now skipped, instead of making the whole response impossible to download. diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index 56b995e18..d7bc5675e 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -282,7 +282,7 @@ BaseItemExporter :class:`dict` items, which have no declared fields, the key order of each item is used instead. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Fields of non-\ :class:`dict` items used to be exported in the order in which they had been populated, except in :class:`CsvItemExporter`, which has always used declaration order. diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index 467abc989..f18d2743a 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -205,7 +205,7 @@ FTPS The feeds are stored in a FTP server, over a TLS connection, with the certificate of the server verified. -.. versionadded:: VERSION +.. versionadded:: 2.18.0 - URI scheme: ``ftps`` - Example URI: ``ftps://user:pass@ftp.example.com/path/to/export.csv`` diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index a6aac78ac..643b48aa0 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -47,7 +47,7 @@ Additionally, they may also implement the following methods: This method is called when the spider is opened. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Added support for :exc:`~scrapy.exceptions.CloseSpider`. It may raise :exc:`~scrapy.exceptions.CloseSpider` to close the spider before diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 27ef3f7ef..5667fc68a 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -487,7 +487,7 @@ Endpoint URL used for S3-like storage, for example Minio or s3.scality. AWS_MAX_POOL_CONNECTIONS ------------------------ -.. versionadded:: VERSION +.. versionadded:: 2.18.0 Default: ``None`` diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 0a85c3c05..c73edbf05 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -290,7 +290,7 @@ spider_opened reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Added support for :exc:`~scrapy.exceptions.CloseSpider`. You may raise a :exc:`~scrapy.exceptions.CloseSpider` exception to close the @@ -348,7 +348,7 @@ spider_error Sent when a spider callback or the :meth:`~scrapy.Spider.start` method of a spider generates an error (i.e. raises an exception). - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Exceptions from :meth:`~scrapy.Spider.start` are also reported, see :ref:`start-error`. @@ -542,7 +542,7 @@ robots_parsed .. signal:: robots_parsed .. function:: robots_parsed(robotparser, request) - .. versionadded:: VERSION + .. versionadded:: 2.18.0 Sent by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after it diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 95c80d5dc..f45fca35e 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -59,7 +59,7 @@ scrapy.Spider :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is enabled. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Changes to this attribute during a crawl are now taken into account. Let's say your target url is ``https://www.example.com/1.html``, @@ -422,7 +422,7 @@ sends the :signal:`spider_error` signal, and, once the already scheduled requests are done, closes the spider with the ``start_error`` :stat:`finish_reason`. -.. versionchanged:: VERSION +.. versionchanged:: 2.18.0 The close reason used to be ``finished``, and neither the :signal:`spider_error` signal nor the :stat:`spider_exceptions/count` stat reported the exception. diff --git a/scrapy/exceptions.py b/scrapy/exceptions.py index cd2560df1..439f5b9f5 100644 --- a/scrapy/exceptions.py +++ b/scrapy/exceptions.py @@ -59,7 +59,7 @@ class CloseSpider(Exception): """Raised from a :ref:`spider callback `, or while the spider is starting, to request the spider to be closed/stopped. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Added support for raising it while the spider is starting. *reason* is a string with the reason for closing. diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py index 7c53b6b48..b9399eb0f 100644 --- a/scrapy/http/request/__init__.py +++ b/scrapy/http/request/__init__.py @@ -396,7 +396,7 @@ class Request(object_ref): Inverse of :meth:`from_curl`. See also :func:`scrapy.utils.request.request_to_curl`. - .. versionadded:: VERSION + .. versionadded:: 2.18.0 """ # Imported here to avoid a circular import. from scrapy.utils.request import request_to_curl # noqa: PLC0415 diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py index a8e452f7e..525043918 100644 --- a/scrapy/http/response/text.py +++ b/scrapy/http/response/text.py @@ -86,7 +86,7 @@ class TextResponse(Response): def json(self) -> Any: """Deserialize a JSON document to a Python object. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Bodies that cannot be decoded as UTF-8, UTF-16 or UTF-32, as the JSON specification requires, are now decoded using :attr:`TextResponse.encoding` instead of raising diff --git a/scrapy/robotstxt.py b/scrapy/robotstxt.py index b54011784..a9e2f5563 100644 --- a/scrapy/robotstxt.py +++ b/scrapy/robotstxt.py @@ -72,7 +72,7 @@ class RobotParser(metaclass=ABCMeta): of seconds, or ``None`` if it is not set or the backend does not support it. - .. versionadded:: VERSION + .. versionadded:: 2.18.0 """ return None diff --git a/scrapy/spidermiddlewares/depth.py b/scrapy/spidermiddlewares/depth.py index 49683168e..83d242f82 100644 --- a/scrapy/spidermiddlewares/depth.py +++ b/scrapy/spidermiddlewares/depth.py @@ -42,7 +42,7 @@ class DepthMiddleware(BaseSpiderMiddleware): depth_reset ----------- - .. versionadded:: VERSION + .. versionadded:: 2.18.0 :attr:`~scrapy.Request.meta` key that, set to ``True``, gives a request depth 0 instead of the depth of its source response plus 1, e.g. to keep diff --git a/scrapy/utils/asyncio.py b/scrapy/utils/asyncio.py index 7c7697f56..f1317032b 100644 --- a/scrapy/utils/asyncio.py +++ b/scrapy/utils/asyncio.py @@ -296,7 +296,7 @@ class CallLaterResult: async def sleep(seconds: float) -> None: """Sleep for *seconds*. - .. versionadded:: VERSION + .. versionadded:: 2.18.0 This uses either :func:`asyncio.sleep` or :func:`~twisted.internet.task.deferLater`, depending on whether asyncio From 09627b0f0568d5cdda514e05ce8ecff5bee56e08 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Wed, 12 Aug 2026 06:38:51 +0200 Subject: [PATCH 09/14] WIP --- docs/news.rst | 45 ++++++++++++++++----------------------------- 1 file changed, 16 insertions(+), 29 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 6e898ac20..160a1e4be 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -76,21 +76,14 @@ Backward-incompatible changes - The minimum required ``queuelib`` version is now 1.6.1. (:gh:`7874`) -- :ref:`Item exporters ` other than - :class:`~scrapy.exporters.CsvItemExporter` now export the fields of an item +- :ref:`Item exporters ` now export the fields of an item in declaration order, i.e. the order in which they are defined in the :ref:`item class `, instead of the order in which they were - populated. :class:`dict` items, which have no declared fields, keep using - the key order of each item. + populated, as :class:`~scrapy.exporters.CsvItemExporter` already did. + :class:`dict` items, which have no declared fields, keep using the key + order of each item. (:gh:`6662`, :gh:`7824`) -- The ``_get_serialized_fields()`` method of :ref:`item exporters - ` is renamed to - :meth:`~scrapy.exporters.BaseItemExporter.get_serialized_fields`, and the - old name is gone, so :ref:`custom item exporters ` that - call or override it must be updated. - (:gh:`5706`, :gh:`7931`) - - ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed exports `, the :ref:`telnet console ` and the @@ -113,7 +106,8 @@ Backward-incompatible changes (:gh:`5995`, :gh:`7922`) - The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code - reading it now gets ``None`` instead of its default value, an empty list. + reading it now gets ``None`` instead of its default value, which was an + empty list. (:gh:`7737`) - ``scrapy.utils.log.logformatter_adapter()`` no longer passes the whole @@ -156,23 +150,6 @@ Backward-incompatible changes reported instead. (:gh:`7768`) -- ``scrapy.pipelines.files.FileException`` moved to - ``scrapy.pipelines.media``. It is still importable from its old location. - (:gh:`7544`, :gh:`7673`) - -- ``H2ConnectionPool``, ``H2ClientFactory`` and ``H2ClientProtocol``, from - ``scrapy.core.http2``, now take a :class:`~scrapy.crawler.Crawler` object - where they used to take a :class:`~scrapy.settings.Settings` object, and - ``scrapy.core.http2.stream.Stream`` takes an additional ``crawler`` - parameter, after ``protocol``. - (:gh:`7896`) - -- :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` now raises - :exc:`~scrapy.exceptions.IgnoreRequest` with a message, e.g. ``Filtered - offsite request to 'offsite.example'``, which errbacks and log messages that - report that exception now include. - (:gh:`7544`, :gh:`7673`) - - The unused ``multiplier`` attribute of :class:`~scrapy.extensions.periodic_log.PeriodicLog` is removed. (:gh:`7809`) @@ -222,6 +199,10 @@ Deprecations - ``scrapy.utils.python.re_rsearch()`` is deprecated. (:gh:`7765`) +- Importing ``FileException`` from ``scrapy.pipelines.files`` is deprecated. + Import it from ``scrapy.pipelines.media`` instead. + (:gh:`7544`, :gh:`7673`, :gh:`7973`) + - Setting ``request.meta["is_secure"]`` to ``False`` to send an ``s3://`` request over plaintext HTTP is deprecated. The flag will be ignored in a future Scrapy version. @@ -402,6 +383,12 @@ Bug fixes without a traceback, instead of reporting them as download errors. (:gh:`7544`, :gh:`7673`) +- :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` now raises + :exc:`~scrapy.exceptions.IgnoreRequest` with a message, e.g. ``Filtered + offsite request to 'offsite.example'``, which errbacks and log messages + that report that exception now include. + (:gh:`7544`, :gh:`7673`) + - :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now skips response header lines that have no colon, logging them at the ``DEBUG`` level, as web browsers do, instead of being unable to download From 8f03e5ca3944c0d6c62ed690007035e91308f714 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Wed, 12 Aug 2026 15:20:43 +0200 Subject: [PATCH 10/14] Address feedback --- docs/news.rst | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 160a1e4be..851e2c659 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -24,6 +24,27 @@ Highlights: - New :ref:`optimization ` page and :ref:`built-in stats reference ` +Modified requirements +~~~~~~~~~~~~~~~~~~~~~ + +- ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br`` + is always included in the ``Accept-Encoding`` header of requests, and + Brotli-compressed responses are always decoded. Websites may now serve + Brotli-compressed responses to crawls that previously did not advertise + support for them. + + The minimum required versions are ``brotli`` 1.2.0 and ``brotlicffi`` + 1.2.0.0. + + (:gh:`4698`, :gh:`7929`) + +- The minimum required ``queuelib`` version is now 1.6.1. + (:gh:`7874`) + +- The IPython :ref:`shell ` requires IPython 8.15.0 or higher. + Install the :ref:`ipython extra ` to get a compatible version. + (:gh:`5447`, :gh:`7596`, :gh:`7816`) + Backward-incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -62,20 +83,6 @@ Backward-incompatible changes (:gh:`6136`, :gh:`7882`) -- ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br`` - is always included in the ``Accept-Encoding`` header of requests, and - Brotli-compressed responses are always decoded. Websites may now serve - Brotli-compressed responses to crawls that previously did not advertise - support for them. - - The minimum required versions are ``brotli`` 1.2.0 and ``brotlicffi`` - 1.2.0.0. - - (:gh:`4698`, :gh:`7929`) - -- The minimum required ``queuelib`` version is now 1.6.1. - (:gh:`7874`) - - :ref:`Item exporters ` now export the fields of an item in declaration order, i.e. the order in which they are defined in the :ref:`item class `, instead of the order in which they were @@ -154,10 +161,6 @@ Backward-incompatible changes :class:`~scrapy.extensions.periodic_log.PeriodicLog` is removed. (:gh:`7809`) -- The IPython :ref:`shell ` requires IPython 8.15.0 or higher. - Install the :ref:`ipython extra ` to get a compatible version. - (:gh:`5447`, :gh:`7596`, :gh:`7816`) - Deprecation removals ~~~~~~~~~~~~~~~~~~~~ From 4caee732a1676d11bbdd9641d31cf1583f02cf58 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Wed, 12 Aug 2026 15:38:38 +0200 Subject: [PATCH 11/14] WIP --- docs/news.rst | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 851e2c659..0e00c674f 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -157,10 +157,6 @@ Backward-incompatible changes reported instead. (:gh:`7768`) -- The unused ``multiplier`` attribute of - :class:`~scrapy.extensions.periodic_log.PeriodicLog` is removed. - (:gh:`7809`) - Deprecation removals ~~~~~~~~~~~~~~~~~~~~ @@ -211,6 +207,10 @@ Deprecations future Scrapy version. (:gh:`7738`) +- The unused ``multiplier`` attribute of + :class:`~scrapy.extensions.periodic_log.PeriodicLog` is deprecated. + (:gh:`7809`, :gh:`7982`) + - Returning, from a :ref:`log formatter ` method, a ``msg`` with ``%(name)s`` placeholders and no ``args`` is deprecated. Those placeholders are still interpolated with the returned :class:`dict`, but in @@ -652,7 +652,9 @@ Quality assurance :gh:`7906`, :gh:`7928`, :gh:`7935`, - :gh:`7966`) + :gh:`7966`, + :gh:`7974`, + :gh:`7979`) .. _release-2.17.0: From 24bec313664338f560e69a162962f668a05caade Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Wed, 12 Aug 2026 15:55:21 +0200 Subject: [PATCH 12/14] WIP --- docs/news.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 0e00c674f..1d348f609 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -304,8 +304,8 @@ New features (:gh:`3435`, :gh:`7905`) - Added an :ref:`FTPS feed storage backend `, i.e. support - for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed over - a TLS connection, verifying the certificate of the server. + for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed + over a TLS connection, verifying the certificate of the server. (:gh:`4180`, :gh:`7953`) - Changes to :attr:`Spider.allowed_domains ` From 26584e99ae77cb6e43a925d223e62281fe22da7c Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Wed, 12 Aug 2026 19:12:39 +0200 Subject: [PATCH 13/14] WIP --- docs/news.rst | 68 +++++++++++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 38 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 1d348f609..44c2e0acc 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -345,9 +345,10 @@ New features (:gh:`5706`, :gh:`7931`) - Log formatters (:setting:`LOG_FORMATTER`), item processors - (``ITEM_PROCESSOR``) and :ref:`robots.txt parsers ` - (:setting:`ROBOTSTXT_PARSER`) are now built as :ref:`components - `, so they no longer need a ``from_crawler()`` method. + (:setting:`ITEM_PROCESSOR`) and :ref:`robots.txt parsers + ` (:setting:`ROBOTSTXT_PARSER`) are now built as + :ref:`components `, so they no longer need a + ``from_crawler()`` method. (:gh:`7808`) Bug fixes @@ -501,9 +502,10 @@ Documentation and :ref:`supported callback output `. (:gh:`5054`, :gh:`6437`, :gh:`7821`, :gh:`7898`) -- Documented the :ref:`optional extras ` of Scrapy, and which feature - each of them enables. - (:gh:`7596`) +- Documented the :setting:`ITEM_PROCESSOR` setting and the + :class:`~scrapy.pipelines.ItemProcessorProtocol` protocol that its value + must implement. + (:gh:`7983`) - Documented :ref:`how to write an item exporter `, :ref:`how to test an item pipeline `, :ref:`how to @@ -543,17 +545,6 @@ Documentation :gh:`7907`, :gh:`7941`) -- Switched several API references to autodoc, so that they are generated from - the docstrings: contracts, download handlers, exceptions, spider loaders, - stats collectors, ``trackref``, and the depth and offsite middlewares. - (:gh:`7767`, - :gh:`7769`, - :gh:`7771`, - :gh:`7775`, - :gh:`7871`, - :gh:`7903`, - :gh:`7913`) - - Many other corrections and improvements. (:gh:`4589`, :gh:`4796`, @@ -568,7 +559,11 @@ Documentation :gh:`7710`, :gh:`7725`, :gh:`7737`, + :gh:`7767`, + :gh:`7769`, + :gh:`7771`, :gh:`7774`, + :gh:`7775`, :gh:`7777`, :gh:`7779`, :gh:`7780`, @@ -576,9 +571,12 @@ Documentation :gh:`7832`, :gh:`7835`, :gh:`7862`, + :gh:`7871`, :gh:`7875`, :gh:`7880`, :gh:`7890`, + :gh:`7903`, + :gh:`7913`, :gh:`7917`, :gh:`7939`, :gh:`7940`, @@ -588,8 +586,16 @@ Documentation Quality assurance ~~~~~~~~~~~~~~~~~ -- Added CPU benchmarks, tracked on CodSpeed, and a ``benchmark`` tox - environment to run them. +- Improved and fixed type hints. + (:gh:`7712`, + :gh:`7785`, + :gh:`7858`, + :gh:`7864`, + :gh:`7865`, + :gh:`7867`) + +- Added CPU benchmarks, tracked on CodSpeed, so that performance regressions + are caught before they are merged and performance work can be measured. (:gh:`7831`, :gh:`7839`, :gh:`7870`, @@ -598,32 +604,17 @@ Quality assurance :gh:`7954`) - Added a nightly job that runs the test suite against the development - branches of dependencies, and a ``vcs-deps`` tox environment for it. + branches of dependencies, so that incompatibilities are found before those + dependencies are released. (:gh:`5291`, :gh:`6025`, :gh:`7924`, :gh:`7960`) -- Tests that need a proxy server now look for a ``mitmdump`` executable, from - the ``PATH``, from `uv `__ or from the - ``MITMDUMP`` environment variable, instead of requiring mitmproxy to be - installed in the test environment. The ``mitmproxy`` tox environment is - gone as a result. - (:gh:`7437`, :gh:`7720`) - -- Dropped the ``testfixtures`` test dependency. - (:gh:`6478`, :gh:`7793`) - -- Type hints improvements and fixes. - (:gh:`7712`, - :gh:`7785`, - :gh:`7858`, - :gh:`7864`, - :gh:`7865`, - :gh:`7867`) - - CI and test improvements and fixes. (:gh:`5620`, :gh:`5837`, :gh:`6478`, :gh:`6794`, + :gh:`7437`, + :gh:`7720`, :gh:`7724`, :gh:`7727`, :gh:`7736`, @@ -635,6 +626,7 @@ Quality assurance :gh:`7778`, :gh:`7782`, :gh:`7792`, + :gh:`7793`, :gh:`7795`, :gh:`7797`, :gh:`7798`, From fd37bf145e25f1c12b410a36a3952d7226344a89 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Thu, 13 Aug 2026 11:04:51 +0200 Subject: [PATCH 14/14] Sync with be514d8c5df4c0327e050ac6d596d7f5b702806c, add missing issues/PRs --- docs/news.rst | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 44c2e0acc..19943cbb4 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -10,6 +10,8 @@ Highlights: - ``HttpxDownloadHandler`` now uses `httpx2 `__ +- The Twisted-based HTTP/2 download handler is no longer experimental + - ``brotli`` is now a required dependency, and :ref:`optional extras ` cover the rest of the optional features @@ -89,7 +91,7 @@ Backward-incompatible changes populated, as :class:`~scrapy.exporters.CsvItemExporter` already did. :class:`dict` items, which have no declared fields, keep using the key order of each item. - (:gh:`6662`, :gh:`7824`) + (:gh:`6662`, :gh:`6854`, :gh:`7824`) - ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed exports `, the :ref:`telnet console @@ -128,7 +130,7 @@ Backward-incompatible changes item, so that ``%``-style placeholders work with it as they do with a ``dict``. - (:gh:`5570`, :gh:`7936`) + (:gh:`5570`, :gh:`5572`, :gh:`7936`) - :setting:`FEEDS` keys and ``FEED_URI`` values that are :class:`pathlib.Path` objects are now used as paths, instead of being @@ -278,10 +280,11 @@ New features - :setting:`CONCURRENT_REQUESTS` can now be set to ``0`` for no limit. (:gh:`7840`) -- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now sends - the :signal:`bytes_received` and :signal:`headers_received` signals, and - supports :exc:`~scrapy.exceptions.StopDownload`. - (:gh:`5046`, :gh:`5055`, :gh:`7896`) +- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` is no + longer experimental, and it now sends the :signal:`bytes_received` and + :signal:`headers_received` signals and supports + :exc:`~scrapy.exceptions.StopDownload`. + (:gh:`5046`, :gh:`5047`, :gh:`5055`, :gh:`7896`, :gh:`7986`) - An exception raised by :meth:`Spider.start() ` is now reported through the :signal:`spider_error` signal and the @@ -293,7 +296,7 @@ New features ` now closes the spider with the given reason, instead of being reported as a start error. - (:gh:`3463`, :gh:`4182`, :gh:`7884`) + (:gh:`3463`, :gh:`4058`, :gh:`4182`, :gh:`6148`, :gh:`7884`) - :exc:`~scrapy.exceptions.CloseSpider` can now also be raised while the spider is starting, e.g. from a :signal:`spider_opened` signal handler or @@ -442,7 +445,7 @@ Bug fixes - :command:`parse` now sets the callback it uses on the request of the response it passes to that callback. - (:gh:`3095`, :gh:`7803`) + (:gh:`3095`, :gh:`3124`, :gh:`7803`) - The IPython :ref:`shell ` now works when an asyncio event loop is already running in the same thread, e.g. when calling @@ -514,7 +517,8 @@ Documentation to add objects to the shell ` and :ref:`how to run spiders inside an existing application ` or :ref:`in a Jupyter notebook `. - (:gh:`1199`, + (:gh:`915`, + :gh:`1199`, :gh:`2594`, :gh:`5706`, :gh:`6554`, @@ -646,7 +650,8 @@ Quality assurance :gh:`7935`, :gh:`7966`, :gh:`7974`, - :gh:`7979`) + :gh:`7979`, + :gh:`7985`) .. _release-2.17.0: