diff --git a/docs/news.rst b/docs/news.rst index 38624948d..19943cbb4 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,8 +3,49 @@ Release notes ============= -Scrapy VERSION (unreleased) ---------------------------- +Scrapy 2.18.0 (unreleased) +-------------------------- + +Highlights: + +- ``HttpxDownloadHandler`` now uses `httpx2 `__ + +- The Twisted-based HTTP/2 download handler is no longer experimental + +- ``brotli`` is now a required dependency, and :ref:`optional extras + ` cover the rest of the optional features + +- Late :class:`~scrapy.crawler.Crawler` attributes, such as + :attr:`~scrapy.crawler.Crawler.stats`, now raise :exc:`RuntimeError` + instead of being ``None`` before the crawl starts + +- Item exporters now export fields in declaration order + +- New :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware` + +- New :ref:`optimization ` page and :ref:`built-in stats reference + ` + +Modified requirements +~~~~~~~~~~~~~~~~~~~~~ + +- ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br`` + is always included in the ``Accept-Encoding`` header of requests, and + Brotli-compressed responses are always decoded. Websites may now serve + Brotli-compressed responses to crawls that previously did not advertise + support for them. + + The minimum required versions are ``brotli`` 1.2.0 and ``brotlicffi`` + 1.2.0.0. + + (:gh:`4698`, :gh:`7929`) + +- The minimum required ``queuelib`` version is now 1.6.1. + (:gh:`7874`) + +- The IPython :ref:`shell ` requires IPython 8.15.0 or higher. + Install the :ref:`ipython extra ` to get a compatible version. + (:gh:`5447`, :gh:`7596`, :gh:`7816`) Backward-incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -23,8 +64,595 @@ Backward-incompatible changes :class:`~scrapy.extensions.feedexport.StdoutFeedStorage` are no longer marked as implementing the ``IFeedStorage`` interface. + - :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no + longer checks that the ``DOWNLOADER_CLIENTCONTEXTFACTORY`` class + implements the ``IPolicyForHTTPS`` interface. + (:gh:`6585`, :gh:`7731`) +- The :attr:`~scrapy.crawler.Crawler.engine`, + :attr:`~scrapy.crawler.Crawler.extensions`, + :attr:`~scrapy.crawler.Crawler.logformatter`, + :attr:`~scrapy.crawler.Crawler.request_fingerprinter` and + :attr:`~scrapy.crawler.Crawler.stats` attributes of + :class:`~scrapy.crawler.Crawler` raise :exc:`RuntimeError` when read before + the crawl starts, instead of being ``None`` until then. + + Code that reads them from the :signal:`spider_opened` signal handler + onwards is unaffected, and no longer needs to narrow their type. Code that + checked whether they were set, e.g. ``if crawler.stats:``, must be updated, + since reading them now raises instead of returning ``None``. + + (:gh:`6136`, :gh:`7882`) + +- :ref:`Item exporters ` now export the fields of an item + in declaration order, i.e. the order in which they are defined in the + :ref:`item class `, instead of the order in which they were + populated, as :class:`~scrapy.exporters.CsvItemExporter` already did. + :class:`dict` items, which have no declared fields, keep using the key + order of each item. + (:gh:`6662`, :gh:`6854`, :gh:`7824`) + +- ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed + exports `, the :ref:`telnet console + ` and the + :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension, now + serializes :class:`~datetime.datetime`, :class:`~datetime.date` and + :class:`~datetime.time` objects in ISO 8601 format, e.g. + ``2023-08-03T23:24:57.148903+00:00`` instead of ``2023-08-03 23:24:57``, + keeping microseconds and time zone information. + + Its ``DATE_FORMAT`` and ``TIME_FORMAT`` attributes are removed. + + (:gh:`2087`, :gh:`7918`) + +- ``scrapy.utils.trackref.live_refs`` is now a + :class:`~weakref.WeakKeyDictionary` instead of a + :class:`collections.defaultdict`, so that classes defined at run time are + released once they are no longer used. Reading the entry of a class with no + tracked instances now raises :exc:`KeyError` instead of creating and + returning an empty mapping. + (:gh:`5995`, :gh:`7922`) + +- The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code + reading it now gets ``None`` instead of its default value, which was an + empty list. + (:gh:`7737`) + +- ``scrapy.utils.log.logformatter_adapter()`` no longer passes the whole + :class:`dict` returned by a :ref:`log formatter ` + method as logging arguments when that ``dict`` has no ``args`` key, or its + ``args`` are empty, and its ``msg`` has no ``%(name)s`` placeholders. Such + messages are now logged verbatim, so a literal ``%`` in them no longer + breaks logging. + + An ``args`` :class:`tuple` is now expanded into one logging argument per + item, so that ``%``-style placeholders work with it as they do with a + ``dict``. + + (:gh:`5570`, :gh:`5572`, :gh:`7936`) + +- :setting:`FEEDS` keys and ``FEED_URI`` values that are + :class:`pathlib.Path` objects are now used as paths, instead of being + converted into ``file://`` URIs. This makes them keep working when they + contain :ref:`URI parameters ` or characters that + URI conversion would percent-encode. + (:gh:`5794`, :gh:`6425`, :gh:`6611`, :gh:`7674`) + +- :class:`~scrapy.Selector` and :attr:`TextResponse.selector + ` no longer force the ``html`` selector + type for responses that are neither :class:`~scrapy.http.HtmlResponse` nor + :class:`~scrapy.http.XmlResponse` objects, e.g. for a JSON response. + ``parsel`` determines the type from the body in those cases instead. + (:gh:`5291`, :gh:`6025`, :gh:`7924`) + +- :ref:`AutoThrottle ` no longer sets the + ``download_delay`` attribute of the running spider to define the starting + delay of download slots. The starting delay is still applied, but code + that reads that attribute at run time no longer sees it. + (:gh:`7167`, :gh:`7175`, :gh:`7833`) + +- :class:`~scrapy.spiders.XMLFeedSpider` and + :class:`~scrapy.spiders.CSVFeedSpider` no longer raise + :exc:`~scrapy.exceptions.NotConfigured` when ``parse_node()`` or + ``parse_row()`` is not defined; the resulting :exc:`AttributeError` is + reported instead. + (:gh:`7768`) + +Deprecation removals +~~~~~~~~~~~~~~~~~~~~ + +- ``scrapy.utils.iterators.xmliter()``, deprecated since Scrapy 2.11.1 + because it is vulnerable to ReDoS attacks, is removed. Use + :func:`~scrapy.utils.iterators.xmliter_lxml` instead. + (:gh:`7765`) + +Deprecations +~~~~~~~~~~~~ + +- The ``download_delay`` spider attribute is deprecated. Use the + :setting:`DOWNLOAD_DELAY` setting, or :setting:`DOWNLOAD_SLOTS` to set a + delay for specific domains, instead. + + The ``max_concurrent_requests`` spider attribute, deprecated since Scrapy + 2.13.0, now sets the :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` setting, + which is what it always mapped to, and warns accordingly. + + Both attributes are ignored, with a different warning, when the + corresponding setting is already set at the ``spider`` priority or higher. + + (:gh:`7167`, :gh:`7175`, :gh:`7833`) + +- The ``Spider.log()`` method is deprecated. Use the methods of + :attr:`Spider.logger ` instead. + (:gh:`7739`) + +- The ``scrapy.interfaces`` module and its ``ISpiderLoader`` interface are + deprecated. Custom spider loaders only need to follow + :class:`~scrapy.spiderloader.SpiderLoaderProtocol`. + (:gh:`6585`, :gh:`7731`) + +- ``scrapy.extensions.feedexport.IFeedStorage`` is deprecated. Custom feed + storages only need to follow + ``scrapy.extensions.feedexport.FeedStorageProtocol``. + (:gh:`6585`, :gh:`7731`) + +- ``scrapy.utils.python.re_rsearch()`` is deprecated. + (:gh:`7765`) + +- Importing ``FileException`` from ``scrapy.pipelines.files`` is deprecated. + Import it from ``scrapy.pipelines.media`` instead. + (:gh:`7544`, :gh:`7673`, :gh:`7973`) + +- Setting ``request.meta["is_secure"]`` to ``False`` to send an ``s3://`` + request over plaintext HTTP is deprecated. The flag will be ignored in a + future Scrapy version. + (:gh:`7738`) + +- The unused ``multiplier`` attribute of + :class:`~scrapy.extensions.periodic_log.PeriodicLog` is deprecated. + (:gh:`7809`, :gh:`7982`) + +- Returning, from a :ref:`log formatter ` method, a + ``msg`` with ``%(name)s`` placeholders and no ``args`` is deprecated. Those + placeholders are still interpolated with the returned :class:`dict`, but in + a future Scrapy version the message will be logged verbatim. Return those + values under ``args`` instead. + (:gh:`5570`, :gh:`7971`) + +New features +~~~~~~~~~~~~ + +- Added :ref:`optional extras ` for every optional dependency of + Scrapy: ``bpython``, ``gcs``, ``httpx``, ``images``, ``ipython``, + ``ptpython``, ``robotparser``, ``s3``, ``twisted-http2``, ``uvloop`` and + ``zstd``. For example, ``pip install scrapy[s3,images]``. + (:gh:`7596`) + +- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now + uses `httpx2 `__, the successor of ``httpx``, + which the new :ref:`httpx extra ` installs together with its HTTP/2 + and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not + installed, but it is no longer tested. + (:gh:`7762`) + +- Added a :signal:`robots_parsed` signal, sent by + :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after + it parses a :file:`robots.txt` file. It supports :ref:`asynchronous + handlers `. + + Added a :meth:`~scrapy.robotstxt.RobotParser.crawl_delay` method to + :class:`~scrapy.robotstxt.RobotParser`, implemented by all built-in + :ref:`robots.txt parsers `. + + (:gh:`7830`) + +- Added a :meth:`Request.to_curl() ` method, the + inverse of :meth:`~scrapy.Request.from_curl`. + (:gh:`7743`, :gh:`7746`, :gh:`7802`) + +- Added a :reqmeta:`depth_reset` request meta key that gives a request depth + 0 instead of the depth of its source response plus 1. + (:gh:`891`, :gh:`7913`) + +- Added + :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`, + enabled by default, which warns once per crawl when a spider yields a + request carrying internal :attr:`~scrapy.Request.meta` keys that were + likely copied from ``response.meta``, and a + :setting:`META_COPY_WARN_SKIP_KEYS` setting to exclude keys from that + check. + (:gh:`7588`) + +- Added an :setting:`AWS_MAX_POOL_CONNECTIONS` setting, which defines the + connection pool size of the AWS clients of the :ref:`S3 feed storage + backend ` and the :ref:`S3 media pipeline storage + backend `, and defaults to + :setting:`REACTOR_THREADPOOL_MAXSIZE`. It is also exposed as a + ``max_pool_connections`` parameter of ``S3FeedStorage`` and as an + ``AWS_MAX_POOL_CONNECTIONS`` attribute of ``S3FilesStore``. + (:gh:`4985`, :gh:`7794`) + +- Added a :func:`scrapy.utils.asyncio.sleep` function, which works both with + and without a Twisted reactor. + (:gh:`7843`) + +- :setting:`CONCURRENT_REQUESTS` can now be set to ``0`` for no limit. + (:gh:`7840`) + +- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` is no + longer experimental, and it now sends the :signal:`bytes_received` and + :signal:`headers_received` signals and supports + :exc:`~scrapy.exceptions.StopDownload`. + (:gh:`5046`, :gh:`5047`, :gh:`5055`, :gh:`7896`, :gh:`7986`) + +- An exception raised by :meth:`Spider.start() ` is now + reported through the :signal:`spider_error` signal and the + :stat:`spider_exceptions/count` and :stat:`spider_exceptions/{exception}` + stats, and closes the spider with the new ``start_error`` + :stat:`finish_reason` instead of ``finished``. See :ref:`start-error`. + + :exc:`~scrapy.exceptions.CloseSpider` raised from :meth:`Spider.start() + ` now closes the spider with the given reason, instead + of being reported as a start error. + + (:gh:`3463`, :gh:`4058`, :gh:`4182`, :gh:`6148`, :gh:`7884`) + +- :exc:`~scrapy.exceptions.CloseSpider` can now also be raised while the + spider is starting, e.g. from a :signal:`spider_opened` signal handler or + from the ``open_spider()`` method of an :ref:`item pipeline + `, to close the spider before it starts crawling. + Every component still gets started, and stopped, before the spider is + closed with the given reason. + (:gh:`3435`, :gh:`7905`) + +- Added an :ref:`FTPS feed storage backend `, i.e. support + for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed + over a TLS connection, verifying the certificate of the server. + (:gh:`4180`, :gh:`7953`) + +- Changes to :attr:`Spider.allowed_domains ` + during a crawl are now taken into account by + :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, whose + :meth:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware.should_follow` + method is now documented as the way to implement a different offsite + policy. + (:gh:`3257`, :gh:`3412`, :gh:`7903`, :gh:`7912`) + +- :class:`~scrapy.settings.BaseSettings` methods that take settings, such as + :meth:`~scrapy.settings.BaseSettings.update` and the ``settings`` parameter + of crawler classes, now also accept an iterable of ``(name, value)`` + tuples. + (:gh:`7759`, :gh:`7763`) + +- The ``cookies`` parameter of :class:`~scrapy.Request` now also accepts + :class:`bool`, :class:`float` and :class:`int` values, and the ``formdata`` + parameter of :class:`~scrapy.FormRequest` now accepts any mapping or + iterable of key-value pairs. + (:gh:`7858`, :gh:`7864`) + +- Added a ``scrapy.utils.reactorless.uninstall_reactor_import_hook()`` + function, which :meth:`AsyncCrawlerProcess.start() + ` now uses to uninstall the + :mod:`twisted.internet.reactor` import hook when it exits. + (:gh:`7747`) + +- Added the :stat:`depth/request_ignored_count` and + :stat:`httpcache/retrieve_error` stats. + (:gh:`1308`, :gh:`2222`, :gh:`7805`, :gh:`7916`) + +- The :meth:`~scrapy.exporters.BaseItemExporter.get_serialized_fields` method + of :ref:`item exporters `, previously named + ``_get_serialized_fields()``, is now public and documented, for + :ref:`custom item exporters ` to use. + (:gh:`5706`, :gh:`7931`) + +- Log formatters (:setting:`LOG_FORMATTER`), item processors + (:setting:`ITEM_PROCESSOR`) and :ref:`robots.txt parsers + ` (:setting:`ROBOTSTXT_PARSER`) are now built as + :ref:`components `, so they no longer need a + ``from_crawler()`` method. + (:gh:`7808`) + +Bug fixes +~~~~~~~~~ + +- :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now + logs a warning and handles the request as a cache miss when reading a cache + entry raises an exception, e.g. because the entry is corrupted, instead of + letting the exception propagate. It also counts those entries in the new + :stat:`httpcache/retrieve_error` stat. + (:gh:`2222`, :gh:`7805`) + +- :ref:`Feed URIs ` now only expand ``%(...)s`` + parameters, keeping any other percent character as is, so that + percent-encoded URIs, e.g. one with ``%20`` in a path or with + percent-encoded FTP credentials, are no longer misinterpreted as + printf-style formatting directives. + (:gh:`5794`, :gh:`6425`, :gh:`7674`) + +- :ref:`Feed exports ` now start storing a + :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` batch as soon as it is complete, + instead of waiting until the spider closes. + (:gh:`7730`, :gh:`7733`) + +- :class:`~scrapy.exporters.CsvItemExporter` now warns when the fields that + it took from the first item do not cover the fields of a later item, i.e. + when it silently drops data. + (:gh:`4002`, :gh:`4053`, :gh:`7613`, :gh:`7651`) + +- ``GCSFeedStorage`` no longer requires the ``storage.buckets.get`` + permission. + (:gh:`5475`, :gh:`7945`) + +- :ref:`Media pipelines ` now log media requests that + were filtered out, e.g. as offsite requests, at the ``DEBUG`` level and + without a traceback, instead of reporting them as download errors. + (:gh:`7544`, :gh:`7673`) + +- :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` now raises + :exc:`~scrapy.exceptions.IgnoreRequest` with a message, e.g. ``Filtered + offsite request to 'offsite.example'``, which errbacks and log messages + that report that exception now include. + (:gh:`7544`, :gh:`7673`) + +- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now + skips response header lines that have no colon, logging them at the + ``DEBUG`` level, as web browsers do, instead of being unable to download + such a response at all. + (:gh:`210`, :gh:`7806`) + +- :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now sends + domain cookies to hosts without a dot in their name and to hosts given as + an IP address. + (:gh:`6410`, :gh:`7900`) + +- :meth:`TextResponse.json() ` now decodes + bodies that are not valid UTF-8, UTF-16 or UTF-32 using + :attr:`TextResponse.encoding `, instead + of raising :exc:`UnicodeDecodeError`. + (:gh:`6456`, :gh:`7897`) + +- ``scrapy.resolver.CachingHostnameResolver`` now caches addresses without a + port, and sets the requested port on cache hits, so that a cached address + no longer carries the port of the request that populated the cache. + (:gh:`6442`, :gh:`7772`) + +- :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` now removes the + directory of a download slot from the :setting:`JOBDIR` directory once that + slot is drained. + (:gh:`5275`, :gh:`7955`) + +- :class:`~scrapy.extensions.telnet.TelnetConsole` no longer raises an + exception on shutdown when it could not listen on any of the + :setting:`TELNETCONSOLE_PORT` ports. + (:gh:`2702`, :gh:`7910`) + +- The :setting:`DOWNLOAD_WARNSIZE` warning is no longer logged twice for a + response whose ``Content-Length`` header already exceeded the limit. + (:gh:`2476`, :gh:`7963`) + +- :class:`HttpCompressionMiddleware + ` + now logs a warning when it drops a response for exceeding + :setting:`DOWNLOAD_MAXSIZE` during decompression. + (:gh:`6616`, :gh:`7742`) + +- :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware` now logs only the + first request ignored for exceeding :setting:`DEPTH_LIMIT`, and counts them + all in the new :stat:`depth/request_ignored_count` stat. + (:gh:`1308`, :gh:`7916`) + +- :command:`parse` now sets the callback it uses on the request of the + response it passes to that callback. + (:gh:`3095`, :gh:`3124`, :gh:`7803`) + +- The IPython :ref:`shell ` now works when an asyncio event + loop is already running in the same thread, e.g. when calling + ``scrapy.shell.inspect_response()`` from a callback while using the asyncio + reactor. + (:gh:`5447`, :gh:`7816`) + +- :meth:`Request.from_curl() ` now merges repeated + ``-d``, ``--data`` and ``--data-raw`` options into a single body joined + with ``&``, as curl does, instead of keeping only the last one. + (:gh:`7728`) + +- The ``copy()`` method and the ``|=`` operator of + ``scrapy.utils.datatypes.CaseInsensitiveDict`` no longer leave the internal + mapping of original key spellings shared or out of date. + (:gh:`7783`) + +- :meth:`ExecutionEngine.download_async() + ` no longer recurses + once per returned request, e.g. once per redirect. + (:gh:`7544`, :gh:`7673`) + +- :class:`LinkExtractor ` + now canonicalizes each extracted URL once instead of twice when + ``canonicalize`` is ``True``. + (:gh:`7961`) + +- Fixed :exc:`NameError` exceptions on Python 3.14, where :pep:`649` made + annotation evaluation lazy, when inspecting the signature of a callable + with annotations imported only for type checking. + (:gh:`7796`, :gh:`7818`) + +- ``scrapy.utils.decorators.deprecated`` can now be used both as + ``@deprecated`` and as ``@deprecated(...)`` without confusing type + checkers. + (:gh:`7797`) + +Documentation +~~~~~~~~~~~~~ + +- Added a :ref:`built-in stats reference `, covering + every stat that Scrapy sets. + (:gh:`6351`, :gh:`7814`) + +- Replaced the broad crawls page with a new :ref:`optimization ` + page, about finding the bottleneck of a crawl before changing any setting, + which covers :ref:`broad crawls ` as one of its sections. + (:gh:`4737`, :gh:`7938`) + +- Added a :ref:`cookies ` page, which gathers what used to be + spread across the request and downloader middleware pages. + (:gh:`7947`) + +- Added :ref:`callbacks ` and :ref:`errbacks ` sections + to the request and response page, covering :ref:`callback assignment + `, :ref:`how to write a callback ` + and :ref:`supported callback output `. + (:gh:`5054`, :gh:`6437`, :gh:`7821`, :gh:`7898`) + +- Documented the :setting:`ITEM_PROCESSOR` setting and the + :class:`~scrapy.pipelines.ItemProcessorProtocol` protocol that its value + must implement. + (:gh:`7983`) + +- Documented :ref:`how to write an item exporter `, + :ref:`how to test an item pipeline `, :ref:`how to + download a request from a downloader middleware `, :ref:`how + to name media files after the response `, :ref:`how + to add objects to the shell ` and :ref:`how to run + spiders inside an existing application ` or :ref:`in a + Jupyter notebook `. + (:gh:`915`, + :gh:`1199`, + :gh:`2594`, + :gh:`5706`, + :gh:`6554`, + :gh:`6594`, + :gh:`7751`, + :gh:`7872`, + :gh:`7876`, + :gh:`7889`, + :gh:`7909`, + :gh:`7931`) + +- Documented the :ref:`memory use of response parsing + ` and the :ref:`parser limits + ` that Scrapy lifts, in the security page. + (:gh:`5700`, :gh:`7930`) + +- Documented that :ref:`signal handlers run in an undefined order + `, that :signal:`scheduler_empty` must only be awaited from + :meth:`~scrapy.Spider.start`, that concurrency and politeness settings + apply per crawler when :ref:`running multiple spiders in the same process + `, and that a :setting:`JOBDIR` directory cannot be + shared across Scrapy versions. + (:gh:`3191`, + :gh:`5330`, + :gh:`5522`, + :gh:`7861`, + :gh:`7883`, + :gh:`7907`, + :gh:`7941`) + +- Many other corrections and improvements. + (:gh:`4589`, + :gh:`4796`, + :gh:`5532`, + :gh:`5548`, + :gh:`6053`, + :gh:`6184`, + :gh:`6627`, + :gh:`6787`, + :gh:`6943`, + :gh:`6989`, + :gh:`7710`, + :gh:`7725`, + :gh:`7737`, + :gh:`7767`, + :gh:`7769`, + :gh:`7771`, + :gh:`7774`, + :gh:`7775`, + :gh:`7777`, + :gh:`7779`, + :gh:`7780`, + :gh:`7817`, + :gh:`7832`, + :gh:`7835`, + :gh:`7862`, + :gh:`7871`, + :gh:`7875`, + :gh:`7880`, + :gh:`7890`, + :gh:`7903`, + :gh:`7913`, + :gh:`7917`, + :gh:`7939`, + :gh:`7940`, + :gh:`7962`, + :gh:`7965`) + +Quality assurance +~~~~~~~~~~~~~~~~~ + +- Improved and fixed type hints. + (:gh:`7712`, + :gh:`7785`, + :gh:`7858`, + :gh:`7864`, + :gh:`7865`, + :gh:`7867`) + +- Added CPU benchmarks, tracked on CodSpeed, so that performance regressions + are caught before they are merged and performance work can be measured. + (:gh:`7831`, + :gh:`7839`, + :gh:`7870`, + :gh:`7887`, + :gh:`7914`, + :gh:`7954`) + +- Added a nightly job that runs the test suite against the development + branches of dependencies, so that incompatibilities are found before those + dependencies are released. + (:gh:`5291`, :gh:`6025`, :gh:`7924`, :gh:`7960`) + +- CI and test improvements and fixes. + (:gh:`5620`, + :gh:`5837`, + :gh:`6478`, + :gh:`6794`, + :gh:`7437`, + :gh:`7720`, + :gh:`7724`, + :gh:`7727`, + :gh:`7736`, + :gh:`7741`, + :gh:`7749`, + :gh:`7753`, + :gh:`7755`, + :gh:`7768`, + :gh:`7778`, + :gh:`7782`, + :gh:`7792`, + :gh:`7793`, + :gh:`7795`, + :gh:`7797`, + :gh:`7798`, + :gh:`7809`, + :gh:`7829`, + :gh:`7834`, + :gh:`7836`, + :gh:`7838`, + :gh:`7844`, + :gh:`7848`, + :gh:`7853`, + :gh:`7857`, + :gh:`7863`, + :gh:`7895`, + :gh:`7906`, + :gh:`7928`, + :gh:`7935`, + :gh:`7966`, + :gh:`7974`, + :gh:`7979`, + :gh:`7985`) + .. _release-2.17.0: Scrapy 2.17.0 (2026-07-07) diff --git a/docs/topics/api.rst b/docs/topics/api.rst index 598edfeb5..24f46f656 100644 --- a/docs/topics/api.rst +++ b/docs/topics/api.rst @@ -39,7 +39,7 @@ how you :ref:`configure the downloader middlewares :attr:`request_fingerprinter` and :attr:`stats` attributes get their value when the crawl starts, and raise :exc:`RuntimeError` when read before that. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Those attributes used to be ``None`` before getting their value. .. attribute:: request_fingerprinter @@ -82,6 +82,8 @@ how you :ref:`configure the downloader middlewares For the API see :class:`~scrapy.statscollectors.StatsCollector` class. + .. autoattribute:: logformatter + .. attribute:: extensions The extension manager that keeps track of enabled extensions. diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index 2d04b621c..dc24680aa 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -242,7 +242,7 @@ Per-request ``bindaddress`` Yes TLS implementation ``pyOpenSSL``/``cryptography`` =========================== ================================================ -.. versionchanged:: VERSION +.. versionchanged:: 2.18.0 Bad header lines with no colon in them are now skipped, instead of making the whole response impossible to download. diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index 56b995e18..d7bc5675e 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -282,7 +282,7 @@ BaseItemExporter :class:`dict` items, which have no declared fields, the key order of each item is used instead. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Fields of non-\ :class:`dict` items used to be exported in the order in which they had been populated, except in :class:`CsvItemExporter`, which has always used declaration order. diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index 467abc989..f18d2743a 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -205,7 +205,7 @@ FTPS The feeds are stored in a FTP server, over a TLS connection, with the certificate of the server verified. -.. versionadded:: VERSION +.. versionadded:: 2.18.0 - URI scheme: ``ftps`` - Example URI: ``ftps://user:pass@ftp.example.com/path/to/export.csv`` diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index a6aac78ac..643b48aa0 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -47,7 +47,7 @@ Additionally, they may also implement the following methods: This method is called when the spider is opened. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Added support for :exc:`~scrapy.exceptions.CloseSpider`. It may raise :exc:`~scrapy.exceptions.CloseSpider` to close the spider before diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index e58815fad..ffa8e6075 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -487,7 +487,7 @@ Endpoint URL used for S3-like storage, for example Minio or s3.scality. AWS_MAX_POOL_CONNECTIONS ------------------------ -.. versionadded:: VERSION +.. versionadded:: 2.18.0 Default: ``None`` diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 0a85c3c05..c73edbf05 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -290,7 +290,7 @@ spider_opened reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Added support for :exc:`~scrapy.exceptions.CloseSpider`. You may raise a :exc:`~scrapy.exceptions.CloseSpider` exception to close the @@ -348,7 +348,7 @@ spider_error Sent when a spider callback or the :meth:`~scrapy.Spider.start` method of a spider generates an error (i.e. raises an exception). - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Exceptions from :meth:`~scrapy.Spider.start` are also reported, see :ref:`start-error`. @@ -542,7 +542,7 @@ robots_parsed .. signal:: robots_parsed .. function:: robots_parsed(robotparser, request) - .. versionadded:: VERSION + .. versionadded:: 2.18.0 Sent by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after it diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 95c80d5dc..f45fca35e 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -59,7 +59,7 @@ scrapy.Spider :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is enabled. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Changes to this attribute during a crawl are now taken into account. Let's say your target url is ``https://www.example.com/1.html``, @@ -422,7 +422,7 @@ sends the :signal:`spider_error` signal, and, once the already scheduled requests are done, closes the spider with the ``start_error`` :stat:`finish_reason`. -.. versionchanged:: VERSION +.. versionchanged:: 2.18.0 The close reason used to be ``finished``, and neither the :signal:`spider_error` signal nor the :stat:`spider_exceptions/count` stat reported the exception. diff --git a/scrapy/crawler.py b/scrapy/crawler.py index 444a5fb67..74800f039 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -102,6 +102,13 @@ class Crawler: engine: _LateAttribute[ExecutionEngine] = _LateAttribute() extensions: _LateAttribute[ExtensionManager] = _LateAttribute() logformatter: _LateAttribute[LogFormatter] = _LateAttribute() + """The log formatter of this crawler. + + This is used from extensions & middlewares to build the messages that they + log about crawling events. + + For the API see the :class:`~scrapy.logformatter.LogFormatter` class. + """ request_fingerprinter: _LateAttribute[RequestFingerprinterProtocol] = ( _LateAttribute() ) diff --git a/scrapy/exceptions.py b/scrapy/exceptions.py index cd2560df1..439f5b9f5 100644 --- a/scrapy/exceptions.py +++ b/scrapy/exceptions.py @@ -59,7 +59,7 @@ class CloseSpider(Exception): """Raised from a :ref:`spider callback `, or while the spider is starting, to request the spider to be closed/stopped. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Added support for raising it while the spider is starting. *reason* is a string with the reason for closing. diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py index 7c53b6b48..b9399eb0f 100644 --- a/scrapy/http/request/__init__.py +++ b/scrapy/http/request/__init__.py @@ -396,7 +396,7 @@ class Request(object_ref): Inverse of :meth:`from_curl`. See also :func:`scrapy.utils.request.request_to_curl`. - .. versionadded:: VERSION + .. versionadded:: 2.18.0 """ # Imported here to avoid a circular import. from scrapy.utils.request import request_to_curl # noqa: PLC0415 diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py index a8e452f7e..525043918 100644 --- a/scrapy/http/response/text.py +++ b/scrapy/http/response/text.py @@ -86,7 +86,7 @@ class TextResponse(Response): def json(self) -> Any: """Deserialize a JSON document to a Python object. - .. versionchanged:: VERSION + .. versionchanged:: 2.18.0 Bodies that cannot be decoded as UTF-8, UTF-16 or UTF-32, as the JSON specification requires, are now decoded using :attr:`TextResponse.encoding` instead of raising diff --git a/scrapy/robotstxt.py b/scrapy/robotstxt.py index b54011784..a9e2f5563 100644 --- a/scrapy/robotstxt.py +++ b/scrapy/robotstxt.py @@ -72,7 +72,7 @@ class RobotParser(metaclass=ABCMeta): of seconds, or ``None`` if it is not set or the backend does not support it. - .. versionadded:: VERSION + .. versionadded:: 2.18.0 """ return None diff --git a/scrapy/spidermiddlewares/depth.py b/scrapy/spidermiddlewares/depth.py index 49683168e..83d242f82 100644 --- a/scrapy/spidermiddlewares/depth.py +++ b/scrapy/spidermiddlewares/depth.py @@ -42,7 +42,7 @@ class DepthMiddleware(BaseSpiderMiddleware): depth_reset ----------- - .. versionadded:: VERSION + .. versionadded:: 2.18.0 :attr:`~scrapy.Request.meta` key that, set to ``True``, gives a request depth 0 instead of the depth of its source response plus 1, e.g. to keep diff --git a/scrapy/utils/asyncio.py b/scrapy/utils/asyncio.py index 7c7697f56..f1317032b 100644 --- a/scrapy/utils/asyncio.py +++ b/scrapy/utils/asyncio.py @@ -296,7 +296,7 @@ class CallLaterResult: async def sleep(seconds: float) -> None: """Sleep for *seconds*. - .. versionadded:: VERSION + .. versionadded:: 2.18.0 This uses either :func:`asyncio.sleep` or :func:`~twisted.internet.task.deferLater`, depending on whether asyncio