diff --git a/docs/news.rst b/docs/news.rst index 2237697c7..7e26299c7 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -10,7 +10,9 @@ Scrapy 2.11.0 (to be released) Highlights: -- +- Spiders can now modify :ref:`settings ` in their + :meth:`~scrapy.Spider.from_crawler` methods, e.g. based on :ref:`spider + arguments `. - Periodic stats logging. @@ -18,14 +20,25 @@ Highlights: Backward-incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- Most of the initialization of :class:`scrapy.crawler.Crawler` instances is + now done in :meth:`~scrapy.crawler.Crawler.crawl`, so the state of + instances before that method is called is now different compared to older + Scrapy versions. We do not recommend using the + :class:`~scrapy.crawler.Crawler` instances before + :meth:`~scrapy.crawler.Crawler.crawl` is called. (:issue:`6038`) + +- :meth:`scrapy.Spider.from_crawler` is now called before the initialization + of various components previously initialized in + :meth:`scrapy.crawler.Crawler.__init__` and before the settings are + finalized and frozen. This change was needed to allow changing the settings + in :meth:`scrapy.Spider.from_crawler`. If you want to access the final + setting values in the spider code as early as possible you can do this in + :meth:`~scrapy.Spider.start_requests`. (:issue:`6038`) + - The :meth:`TextResponse.json ` method now requires the response to be in a valid JSON encoding (UTF-8, UTF-16, or - UTF-32). - - If you need to deal with JSON documents in an invalid encoding, use - ``json.loads(response.text)`` instead. - - (:issue:`5968`) + UTF-32). If you need to deal with JSON documents in an invalid encoding, + use ``json.loads(response.text)`` instead. (:issue:`6016`) Deprecation removals ~~~~~~~~~~~~~~~~~~~~ @@ -38,19 +51,55 @@ Deprecation removals 1.0.0, use :attr:`CrawlerRunner.spider_loader ` instead. (:issue:`6010`) + .. note:: If you are using this Scrapy version on Scrapy Cloud with a stack + that includes an older Scrapy version and get a "TypeError: + Unexpected options: binary" error, you may need to add + ``scrapinghub-entrypoint-scrapy > 0.14.0`` to your project + requirements or switch to a stack that includes Scrapy 2.11. + +Deprecations +~~~~~~~~~~~~ + +- Running :meth:`~scrapy.crawler.Crawler.crawl` more than once on the same + :class:`scrapy.crawler.Crawler` instance is now deprecated. (:issue:`1587`, + :issue:`6040`) + New features ~~~~~~~~~~~~ +- Changed the :class:`scrapy.crawler.Crawler` initialization order, so that + most of the initialization that previously happened in + :meth:`~scrapy.crawler.Crawler.__init__` now happens in + :meth:`~scrapy.crawler.Crawler.crawl` after the spider instance is created. + This allows spider instances to modify settings in their + :meth:`~scrapy.Spider.from_crawler` methods, e.g. based on :ref:`spider + arguments `. (:issue:`1305`, :issue:`1580`, :issue:`2392`, + :issue:`3663`, :issue:`6038`) + - Added the :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension which can be enabled to log stats and/or their differences periodically. (:issue:`5926`) +- Optimized the memory usage in :meth:`TextResponse.json + ` by removing unnecessary body decoding. + (:issue:`5968`, :issue:`6016`) + - Links to ``.webp`` files are now ignored by :ref:`link extractors `. (:issue:`6021`) Bug fixes ~~~~~~~~~ +- Fixed logging enabled add-ons. (:issue:`6036`) + +- Fixed :class:`~scrapy.mail.MailSender` producing invalid message bodies + when the ``charset`` argument is passed to + :meth:`~scrapy.mail.MailSender.send`. (:issue:`5096`, :issue:`5118`) + +- Fixed an exception when accessing ``self.EXCEPTIONS_TO_RETRY`` from a + subclass of :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`. + (:issue:`6049`, :issue:`6050`) + - :meth:`scrapy.settings.BaseSettings.getdictorlist`, used to parse :setting:`FEED_EXPORT_FIELDS`, now handles tuple values. (:issue:`6011`, :issue:`6013`) @@ -67,9 +116,14 @@ Documentation Quality assurance ~~~~~~~~~~~~~~~~~ -- Extended typing hints. (:issue:`6003`, :issue:`6005`) +- Extended typing hints. (:issue:`6003`, :issue:`6005`, :issue:`6031`, + :issue:`6034`) -- Other CI and pre-commit improvements. (:issue:`6002`, :issue:`6013`) +- Pinned brotli_ to 1.0.9 for the PyPy tests as 1.1.0 breaks them. + (:issue:`6044`, :issue:`6045`) + +- Other CI and pre-commit improvements. (:issue:`6002`, :issue:`6013`, + :issue:`6046`) .. _release-2.10.1: