.. _news:
Release notes
=============
Scrapy VERSION (unreleased)
---------------------------
Highlights:
- ``HttpxDownloadHandler`` now uses `httpx2 `__
- ``brotli`` is now a required dependency, and :ref:`optional extras
` cover the rest of the optional features
- Late :class:`~scrapy.crawler.Crawler` attributes, such as
:attr:`~scrapy.crawler.Crawler.stats`, now raise :exc:`RuntimeError`
instead of being ``None`` before the crawl starts
- Item exporters now export fields in declaration order
- New :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`
- New :ref:`optimization ` page and :ref:`built-in stats reference
`
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The following runtime usage of zope.interface_ interfaces is removed:
- :class:`~scrapy.spiderloader.SpiderLoader` and
:class:`~scrapy.spiderloader.DummySpiderLoader` are no longer marked
as implementing the ``ISpiderLoader`` interface.
- :func:`~scrapy.spiderloader.get_spider_loader` no longer checks that the
configured spider loader implements the ``ISpiderLoader`` interface.
- :class:`~scrapy.extensions.feedexport.BlockingFeedStorage`,
:class:`~scrapy.extensions.feedexport.FileFeedStorage` and
:class:`~scrapy.extensions.feedexport.StdoutFeedStorage` are no longer
marked as implementing the ``IFeedStorage`` interface.
- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no
longer checks that the ``DOWNLOADER_CLIENTCONTEXTFACTORY`` class
implements the ``IPolicyForHTTPS`` interface.
(:gh:`6585`, :gh:`7731`)
- The :attr:`~scrapy.crawler.Crawler.engine`,
:attr:`~scrapy.crawler.Crawler.extensions`,
:attr:`~scrapy.crawler.Crawler.logformatter`,
:attr:`~scrapy.crawler.Crawler.request_fingerprinter` and
:attr:`~scrapy.crawler.Crawler.stats` attributes of
:class:`~scrapy.crawler.Crawler` raise :exc:`RuntimeError` when read before
the crawl starts, instead of being ``None`` until then.
Code that reads them from the :signal:`spider_opened` signal handler
onwards is unaffected, and no longer needs to narrow their type. Code that
checked whether they were set, e.g. ``if crawler.stats:``, must be updated,
since reading them now raises instead of returning ``None``.
(:gh:`6136`, :gh:`7882`)
- ``brotli`` (``brotlicffi`` on PyPy) is now a required dependency, so ``br``
is always included in the ``Accept-Encoding`` header of requests, and
Brotli-compressed responses are always decoded. Websites may now serve
Brotli-compressed responses to crawls that previously did not advertise
support for them.
The minimum required versions are ``brotli`` 1.2.0 and ``brotlicffi``
1.2.0.0.
(:gh:`4698`, :gh:`7929`)
- The minimum required ``queuelib`` version is now 1.6.1.
(:gh:`7874`)
- :ref:`Item exporters ` other than
:class:`~scrapy.exporters.CsvItemExporter` now export the fields of an item
in declaration order, i.e. the order in which they are defined in the
:ref:`item class `, instead of the order in which they were
populated. :class:`dict` items, which have no declared fields, keep using
the key order of each item.
(:gh:`6662`, :gh:`7824`)
- ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed
exports `, the :ref:`telnet console
` and the
:class:`~scrapy.extensions.periodic_log.PeriodicLog` extension, now
serializes :class:`~datetime.datetime`, :class:`~datetime.date` and
:class:`~datetime.time` objects in ISO 8601 format, e.g.
``2023-08-03T23:24:57.148903+00:00`` instead of ``2023-08-03 23:24:57``,
keeping microseconds and time zone information.
Its ``DATE_FORMAT`` and ``TIME_FORMAT`` attributes are removed.
(:gh:`2087`, :gh:`7918`)
- ``scrapy.utils.trackref.live_refs`` is now a
:class:`~weakref.WeakKeyDictionary` instead of a
:class:`collections.defaultdict`, so that classes defined at run time are
released once they are no longer used. Reading the entry of a class with no
tracked instances now raises :exc:`KeyError` instead of creating and
returning an empty mapping.
(:gh:`5995`, :gh:`7922`)
- The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code
reading it now gets ``None`` instead of its default value, an empty list.
(:gh:`7737`)
- ``scrapy.utils.log.logformatter_adapter()`` no longer passes the whole
:class:`dict` returned by a :ref:`log formatter `
method as logging arguments when that ``dict`` has no ``args`` key, or its
``args`` are empty, and its ``msg`` has no ``%(name)s`` placeholders. Such
messages are now logged verbatim, so a literal ``%`` in them no longer
breaks logging.
An ``args`` :class:`tuple` is now expanded into one logging argument per
item, so that ``%``-style placeholders work with it as they do with a
``dict``.
(:gh:`5570`, :gh:`7936`)
- :setting:`FEEDS` keys and ``FEED_URI`` values that are
:class:`pathlib.Path` objects are now used as paths, instead of being
converted into ``file://`` URIs. This makes them keep working when they
contain :ref:`URI parameters ` or characters that
URI conversion would percent-encode.
(:gh:`5794`, :gh:`6425`, :gh:`6611`, :gh:`7674`)
- :class:`~scrapy.Selector` and :attr:`TextResponse.selector
` no longer force the ``html`` selector
type for responses that are neither :class:`~scrapy.http.HtmlResponse` nor
:class:`~scrapy.http.XmlResponse` objects, e.g. for a JSON response.
``parsel`` determines the type from the body in those cases instead.
(:gh:`5291`, :gh:`6025`, :gh:`7924`)
- :ref:`AutoThrottle ` no longer sets the
``download_delay`` attribute of the running spider to define the starting
delay of download slots. The starting delay is still applied, but code
that reads that attribute at run time no longer sees it.
(:gh:`7167`, :gh:`7175`, :gh:`7833`)
- :class:`~scrapy.spiders.XMLFeedSpider` and
:class:`~scrapy.spiders.CSVFeedSpider` no longer raise
:exc:`~scrapy.exceptions.NotConfigured` when ``parse_node()`` or
``parse_row()`` is not defined; the resulting :exc:`AttributeError` is
reported instead.
(:gh:`7768`)
- ``scrapy.pipelines.files.FileException`` moved to
``scrapy.pipelines.media``. It is still importable from its old location.
(:gh:`7544`, :gh:`7673`)
- The IPython :ref:`shell ` requires IPython 8.15.0 or higher.
Install the :ref:`ipython extra ` to get a compatible version.
(:gh:`5447`, :gh:`7596`, :gh:`7816`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- ``scrapy.utils.iterators.xmliter()``, deprecated since Scrapy 2.11.1
because it is vulnerable to ReDoS attacks, is removed. Use
:func:`~scrapy.utils.iterators.xmliter_lxml` instead.
(:gh:`7765`)
Deprecations
~~~~~~~~~~~~
- The ``download_delay`` spider attribute is deprecated. Use the
:setting:`DOWNLOAD_DELAY` setting, or :setting:`DOWNLOAD_SLOTS` to set a
delay for specific domains, instead.
The ``max_concurrent_requests`` spider attribute, deprecated since Scrapy
2.13.0, now sets the :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` setting,
which is what it always mapped to, and warns accordingly.
Both attributes are ignored, with a different warning, when the
corresponding setting is already set at the ``spider`` priority or higher.
(:gh:`7167`, :gh:`7175`, :gh:`7833`)
- The ``Spider.log()`` method is deprecated. Use the methods of
:attr:`Spider.logger ` instead.
(:gh:`7739`)
- The ``scrapy.interfaces`` module and its ``ISpiderLoader`` interface are
deprecated. Custom spider loaders only need to follow
:class:`~scrapy.spiderloader.SpiderLoaderProtocol`.
(:gh:`6585`, :gh:`7731`)
- ``scrapy.extensions.feedexport.IFeedStorage`` is deprecated. Custom feed
storages only need to follow
``scrapy.extensions.feedexport.FeedStorageProtocol``.
(:gh:`6585`, :gh:`7731`)
- ``scrapy.utils.python.re_rsearch()`` is deprecated.
(:gh:`7765`)
- Setting ``request.meta["is_secure"]`` to ``False`` to send an ``s3://``
request over plaintext HTTP is deprecated. The flag will be ignored in a
future Scrapy version.
(:gh:`7738`)
- Returning, from a :ref:`log formatter ` method, a
``msg`` with ``%(name)s`` placeholders and no ``args`` is deprecated. Those
placeholders are still interpolated with the returned :class:`dict`, but in
a future Scrapy version the message will be logged verbatim. Return those
values under ``args`` instead.
(:gh:`5570`, :gh:`7971`)
New features
~~~~~~~~~~~~
- Added :ref:`optional extras ` for every optional dependency of
Scrapy: ``bpython``, ``gcs``, ``httpx``, ``images``, ``ipython``,
``ptpython``, ``robotparser``, ``s3``, ``twisted-http2``, ``uvloop`` and
``zstd``. For example, ``pip install scrapy[s3,images]``.
(:gh:`7596`)
- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now
uses `httpx2 `__, the successor of ``httpx``,
which the new :ref:`httpx extra ` installs together with its HTTP/2
and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not
installed, but it is no longer tested.
(:gh:`7762`)
- Added a :signal:`robots_parsed` signal, sent by
:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after
it parses a :file:`robots.txt` file. It supports :ref:`asynchronous
handlers `.
Added a :meth:`~scrapy.robotstxt.RobotParser.crawl_delay` method to
:class:`~scrapy.robotstxt.RobotParser`, implemented by all built-in
:ref:`robots.txt parsers `.
(:gh:`7830`)
- Added a :meth:`Request.to_curl() ` method, the
inverse of :meth:`~scrapy.Request.from_curl`.
(:gh:`7743`, :gh:`7746`, :gh:`7802`)
- Added a :reqmeta:`depth_reset` request meta key that gives a request depth
0 instead of the depth of its source response plus 1.
(:gh:`891`, :gh:`7913`)
- Added
:class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`,
enabled by default, which warns once per crawl when a spider yields a
request carrying internal :attr:`~scrapy.Request.meta` keys that were
likely copied from ``response.meta``, and a
:setting:`META_COPY_WARN_SKIP_KEYS` setting to exclude keys from that
check.
(:gh:`7588`)
- Added an :setting:`AWS_MAX_POOL_CONNECTIONS` setting, which defines the
connection pool size of the AWS clients of the :ref:`S3 feed storage
backend ` and the :ref:`S3 media pipeline storage
backend `, and defaults to
:setting:`REACTOR_THREADPOOL_MAXSIZE`. It is also exposed as a
``max_pool_connections`` parameter of ``S3FeedStorage`` and as an
``AWS_MAX_POOL_CONNECTIONS`` attribute of ``S3FilesStore``.
(:gh:`4985`, :gh:`7794`)
- Added a :func:`scrapy.utils.asyncio.sleep` function, which works both with
and without a Twisted reactor.
(:gh:`7843`)
- :setting:`CONCURRENT_REQUESTS` can now be set to ``0`` for no limit.
(:gh:`7840`)
- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now sends
the :signal:`bytes_received` and :signal:`headers_received` signals, and
supports :exc:`~scrapy.exceptions.StopDownload`.
(:gh:`5046`, :gh:`5055`, :gh:`7896`)
- An exception raised by :meth:`Spider.start() ` is now
reported through the :signal:`spider_error` signal and the
:stat:`spider_exceptions/count` and :stat:`spider_exceptions/{exception}`
stats, and closes the spider with the new ``start_error``
:stat:`finish_reason` instead of ``finished``. See :ref:`start-error`.
:exc:`~scrapy.exceptions.CloseSpider` raised from :meth:`Spider.start()
` now closes the spider with the given reason, instead
of being reported as a start error.
(:gh:`3463`, :gh:`4182`, :gh:`7884`)
- :exc:`~scrapy.exceptions.CloseSpider` can now also be raised while the
spider is starting, e.g. from a :signal:`spider_opened` signal handler or
from the ``open_spider()`` method of an :ref:`item pipeline
`, to close the spider before it starts crawling.
Every component still gets started, and stopped, before the spider is
closed with the given reason.
(:gh:`3435`, :gh:`7905`)
- Added an :ref:`FTPS feed storage backend `, i.e. support
for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed over
a TLS connection, verifying the certificate of the server.
(:gh:`4180`, :gh:`7953`)
- Changes to :attr:`Spider.allowed_domains `
during a crawl are now taken into account by
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, whose
:meth:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware.should_follow`
method is now documented as the way to implement a different offsite
policy.
(:gh:`3257`, :gh:`3412`, :gh:`7903`, :gh:`7912`)
- :class:`~scrapy.settings.BaseSettings` methods that take settings, such as
:meth:`~scrapy.settings.BaseSettings.update` and the ``settings`` parameter
of crawler classes, now also accept an iterable of ``(name, value)``
tuples.
(:gh:`7759`, :gh:`7763`)
- The ``cookies`` parameter of :class:`~scrapy.Request` now also accepts
:class:`bool`, :class:`float` and :class:`int` values, and the ``formdata``
parameter of :class:`~scrapy.FormRequest` now accepts any mapping or
iterable of key-value pairs.
(:gh:`7858`, :gh:`7864`)
- Added a ``scrapy.utils.reactorless.uninstall_reactor_import_hook()``
function, which :meth:`AsyncCrawlerProcess.start()
` now uses to uninstall the
:mod:`twisted.internet.reactor` import hook when it exits.
(:gh:`7747`)
- Added the :stat:`depth/request_ignored_count` and
:stat:`httpcache/retrieve_error` stats.
(:gh:`1308`, :gh:`2222`, :gh:`7805`, :gh:`7916`)
- The :meth:`~scrapy.exporters.BaseItemExporter.get_serialized_fields` method
of :ref:`item exporters `, previously named
``_get_serialized_fields()``, is now public and documented, for
:ref:`custom item exporters ` to use.
(:gh:`5706`, :gh:`7931`)
- Log formatters (:setting:`LOG_FORMATTER`), item processors
(``ITEM_PROCESSOR``) and :ref:`robots.txt parsers `
(:setting:`ROBOTSTXT_PARSER`) are now built as :ref:`components
`, so they no longer need a ``from_crawler()`` method.
(:gh:`7808`)
Bug fixes
~~~~~~~~~
- :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now
logs a warning and handles the request as a cache miss when reading a cache
entry raises an exception, e.g. because the entry is corrupted, instead of
letting the exception propagate. It also counts those entries in the new
:stat:`httpcache/retrieve_error` stat.
(:gh:`2222`, :gh:`7805`)
- :ref:`Feed URIs ` now only expand ``%(...)s``
parameters, keeping any other percent character as is, so that
percent-encoded URIs, e.g. one with ``%20`` in a path or with
percent-encoded FTP credentials, are no longer misinterpreted as
printf-style formatting directives.
(:gh:`5794`, :gh:`6425`, :gh:`7674`)
- :ref:`Feed exports ` now start storing a
:setting:`FEED_EXPORT_BATCH_ITEM_COUNT` batch as soon as it is complete,
instead of waiting until the spider closes.
(:gh:`7730`, :gh:`7733`)
- :class:`~scrapy.exporters.CsvItemExporter` now warns when the fields that
it took from the first item do not cover the fields of a later item, i.e.
when it silently drops data.
(:gh:`4002`, :gh:`4053`, :gh:`7613`, :gh:`7651`)
- ``GCSFeedStorage`` no longer requires the ``storage.buckets.get``
permission.
(:gh:`5475`, :gh:`7945`)
- :ref:`Media pipelines ` now log media requests that
were filtered out, e.g. as offsite requests, at the ``DEBUG`` level and
without a traceback, instead of reporting them as download errors.
(:gh:`7544`, :gh:`7673`)
- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
skips response header lines that have no colon, logging them at the
``DEBUG`` level, as web browsers do, instead of being unable to download
such a response at all.
(:gh:`210`, :gh:`7806`)
- :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now sends
domain cookies to hosts without a dot in their name and to hosts given as
an IP address.
(:gh:`6410`, :gh:`7900`)
- :meth:`TextResponse.json() ` now decodes
bodies that are not valid UTF-8, UTF-16 or UTF-32 using
:attr:`TextResponse.encoding `, instead
of raising :exc:`UnicodeDecodeError`.
(:gh:`6456`, :gh:`7897`)
- ``scrapy.resolver.CachingHostnameResolver`` now caches addresses without a
port, and sets the requested port on cache hits, so that a cached address
no longer carries the port of the request that populated the cache.
(:gh:`6442`, :gh:`7772`)
- :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` now removes the
directory of a download slot from the :setting:`JOBDIR` directory once that
slot is drained.
(:gh:`5275`, :gh:`7955`)
- :class:`~scrapy.extensions.telnet.TelnetConsole` no longer raises an
exception on shutdown when it could not listen on any of the
:setting:`TELNETCONSOLE_PORT` ports.
(:gh:`2702`, :gh:`7910`)
- The :setting:`DOWNLOAD_WARNSIZE` warning is no longer logged twice for a
response whose ``Content-Length`` header already exceeded the limit.
(:gh:`2476`, :gh:`7963`)
- :class:`HttpCompressionMiddleware
`
now logs a warning when it drops a response for exceeding
:setting:`DOWNLOAD_MAXSIZE` during decompression.
(:gh:`6616`, :gh:`7742`)
- :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware` now logs only the
first request ignored for exceeding :setting:`DEPTH_LIMIT`, and counts them
all in the new :stat:`depth/request_ignored_count` stat.
(:gh:`1308`, :gh:`7916`)
- :command:`parse` now sets the callback it uses on the request of the
response it passes to that callback.
(:gh:`3095`, :gh:`7803`)
- The IPython :ref:`shell ` now works when an asyncio event
loop is already running in the same thread, e.g. when calling
``scrapy.shell.inspect_response()`` from a callback while using the asyncio
reactor.
(:gh:`5447`, :gh:`7816`)
- :meth:`Request.from_curl() ` now merges repeated
``-d``, ``--data`` and ``--data-raw`` options into a single body joined
with ``&``, as curl does, instead of keeping only the last one.
(:gh:`7728`)
- The ``copy()`` method and the ``|=`` operator of
``scrapy.utils.datatypes.CaseInsensitiveDict`` no longer leave the internal
mapping of original key spellings shared or out of date.
(:gh:`7783`)
- :meth:`ExecutionEngine.download_async()
` no longer recurses
once per returned request, e.g. once per redirect.
(:gh:`7544`, :gh:`7673`)
- :class:`LinkExtractor `
now canonicalizes each extracted URL once instead of twice when
``canonicalize`` is ``True``.
(:gh:`7961`)
- Fixed :exc:`NameError` exceptions on Python 3.14, where :pep:`649` made
annotation evaluation lazy, when inspecting the signature of a callable
with annotations imported only for type checking.
(:gh:`7796`, :gh:`7818`)
- ``scrapy.utils.decorators.deprecated`` can now be used both as
``@deprecated`` and as ``@deprecated(...)`` without confusing type
checkers.
(:gh:`7797`)
Documentation
~~~~~~~~~~~~~
- Added a :ref:`built-in stats reference `, covering
every stat that Scrapy sets.
(:gh:`6351`, :gh:`7814`)
- Replaced the broad crawls page with a new :ref:`optimization `
page, about finding the bottleneck of a crawl before changing any setting,
which covers :ref:`broad crawls ` as one of its sections.
(:gh:`4737`, :gh:`7938`)
- Added a :ref:`cookies ` page, which gathers what used to be
spread across the request and downloader middleware pages.
(:gh:`7947`)
- Added :ref:`callbacks ` and :ref:`errbacks ` sections
to the request and response page, covering :ref:`callback assignment
`, :ref:`how to write a callback `
and :ref:`supported callback output `.
(:gh:`5054`, :gh:`6437`, :gh:`7821`, :gh:`7898`)
- Documented the :ref:`optional extras ` of Scrapy, and which feature
each of them enables.
(:gh:`7596`)
- Documented :ref:`how to write an item exporter `,
:ref:`how to test an item pipeline `, :ref:`how to
download a request from a downloader middleware `, :ref:`how
to name media files after the response `, :ref:`how
to add objects to the shell ` and :ref:`how to run
spiders inside an existing application ` or :ref:`in a
Jupyter notebook `.
(:gh:`1199`,
:gh:`2594`,
:gh:`5706`,
:gh:`6554`,
:gh:`6594`,
:gh:`7751`,
:gh:`7872`,
:gh:`7876`,
:gh:`7889`,
:gh:`7909`,
:gh:`7931`)
- Documented the :ref:`memory use of response parsing
` and the :ref:`parser limits
` that Scrapy lifts, in the security page.
(:gh:`5700`, :gh:`7930`)
- Documented that :ref:`signal handlers run in an undefined order
`, that :signal:`scheduler_empty` must only be awaited from
:meth:`~scrapy.Spider.start`, that concurrency and politeness settings
apply per crawler when :ref:`running multiple spiders in the same process
`, and that a :setting:`JOBDIR` directory cannot be
shared across Scrapy versions.
(:gh:`3191`,
:gh:`5330`,
:gh:`5522`,
:gh:`7861`,
:gh:`7883`,
:gh:`7907`,
:gh:`7941`)
- Switched several API references to autodoc, so that they are generated from
the docstrings: contracts, download handlers, exceptions, spider loaders,
stats collectors, ``trackref``, and the depth and offsite middlewares.
(:gh:`7767`,
:gh:`7769`,
:gh:`7771`,
:gh:`7775`,
:gh:`7871`,
:gh:`7903`,
:gh:`7913`)
- Many other corrections and improvements.
(:gh:`4589`,
:gh:`4796`,
:gh:`5532`,
:gh:`5548`,
:gh:`6053`,
:gh:`6184`,
:gh:`6627`,
:gh:`6787`,
:gh:`6943`,
:gh:`6989`,
:gh:`7710`,
:gh:`7725`,
:gh:`7737`,
:gh:`7774`,
:gh:`7777`,
:gh:`7779`,
:gh:`7780`,
:gh:`7817`,
:gh:`7832`,
:gh:`7835`,
:gh:`7862`,
:gh:`7875`,
:gh:`7880`,
:gh:`7890`,
:gh:`7917`,
:gh:`7939`,
:gh:`7940`,
:gh:`7962`,
:gh:`7965`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Added CPU benchmarks, tracked on CodSpeed, and a ``benchmark`` tox
environment to run them.
(:gh:`7831`,
:gh:`7839`,
:gh:`7870`,
:gh:`7887`,
:gh:`7914`,
:gh:`7954`)
- Added a nightly job that runs the test suite against the development
branches of dependencies, and a ``vcs-deps`` tox environment for it.
(:gh:`5291`, :gh:`6025`, :gh:`7924`, :gh:`7960`)
- Tests that need a proxy server now look for a ``mitmdump`` executable, from
the ``PATH``, from `uv `__ or from the
``MITMDUMP`` environment variable, instead of requiring mitmproxy to be
installed in the test environment. The ``mitmproxy`` tox environment is
gone as a result.
(:gh:`7437`, :gh:`7720`)
- Dropped the ``testfixtures`` test dependency.
(:gh:`6478`, :gh:`7793`)
- Type hints improvements and fixes.
(:gh:`7712`,
:gh:`7785`,
:gh:`7858`,
:gh:`7864`,
:gh:`7865`,
:gh:`7867`)
- CI and test improvements and fixes.
(:gh:`5620`,
:gh:`5837`,
:gh:`6478`,
:gh:`6794`,
:gh:`7724`,
:gh:`7727`,
:gh:`7736`,
:gh:`7741`,
:gh:`7749`,
:gh:`7753`,
:gh:`7755`,
:gh:`7768`,
:gh:`7778`,
:gh:`7782`,
:gh:`7792`,
:gh:`7795`,
:gh:`7797`,
:gh:`7798`,
:gh:`7809`,
:gh:`7829`,
:gh:`7834`,
:gh:`7836`,
:gh:`7838`,
:gh:`7844`,
:gh:`7848`,
:gh:`7853`,
:gh:`7857`,
:gh:`7863`,
:gh:`7895`,
:gh:`7906`,
:gh:`7928`,
:gh:`7935`,
:gh:`7966`)
.. _release-2.17.0:
Scrapy 2.17.0 (2026-07-07)
--------------------------
Highlights:
- Security bug fixes
- HTTP/2 and SOCKS proxy support for ``HttpxDownloadHandler``
- Improved settings for changing allowed TLS versions
Security bug fixes
~~~~~~~~~~~~~~~~~~
- ``s3://`` requests now use HTTPS by default, instead of plaintext HTTP.
Previously, :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler`
sent signed S3 requests over plaintext HTTP unless
``request.meta["is_secure"]`` was set to a true value, exposing the request
path, the AWS ``Authorization`` header, the ``X-Amz-Security-Token`` header
(when using temporary credentials), and the response contents to network
attackers, who could also tamper with responses. See the `76g3-c3x4-crvx`_
security advisory for details.
To restore the previous behavior for a given request, set
``request.meta["is_secure"]`` to ``False``.
.. _76g3-c3x4-crvx: https://github.com/scrapy/scrapy/security/advisories/GHSA-76g3-c3x4-crvx
Deprecations
~~~~~~~~~~~~
- The ``DOWNLOADER_CLIENT_TLS_METHOD`` setting is deprecated. You should use
the :setting:`DOWNLOAD_TLS_MIN_VERSION` and/or
:setting:`DOWNLOAD_TLS_MAX_VERSION` settings instead if you want to change
the TLS method selection.
(:gh:`3288`, :gh:`6546`)
- The following spider attributes are deprecated in favor of settings:
- ``http_user`` (use :setting:`HTTPAUTH_USER`)
- ``http_pass`` (use :setting:`HTTPAUTH_PASS`)
- ``http_auth_domain`` (use :setting:`HTTPAUTH_DOMAIN`)
(:gh:`7590`)
- The ``scrapy.commands.ScrapyCommand.help()`` method is deprecated. It was
never called by Scrapy.
(:gh:`7626`, :gh:`7633`)
- The following TLS-related functions and constants, intended for internal
use, are deprecated:
- ``scrapy.core.downloader.tls.METHOD_TLS``
- ``scrapy.core.downloader.tls.METHOD_TLSv10``
- ``scrapy.core.downloader.tls.METHOD_TLSv11``
- ``scrapy.core.downloader.tls.METHOD_TLSv12``
- ``scrapy.core.downloader.tls.openssl_methods``
- ``scrapy.core.downloader.tls.DEFAULT_CIPHERS``
- ``scrapy.utils.ssl.ffi_buf_to_string()``
- ``scrapy.utils.ssl.get_temp_key_info()``
- ``scrapy.utils.ssl.x509name_to_string()``
(:gh:`6546`, :gh:`7619`, :gh:`7665`)
- The ``CRAWLSPIDER_FOLLOW_LINKS`` setting is deprecated. You can set
``follow=False`` in your rules to achieve the same effect.
(:gh:`7592`)
- Instantiating
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
without a ``crawler`` argument is deprecated.
(:gh:`7655`)
- Instantiating
:class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` without a
``settings`` argument is deprecated.
(:gh:`7664`)
New features
~~~~~~~~~~~~
- Added support for HTTP/2 requests to
:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. It
requires setting the new :setting:`HTTPX_HTTP2_ENABLED` setting to
``True``.
(:gh:`7575`)
- Added support for SOCKS proxies to
:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
(:gh:`747`, :gh:`7575`)
- Added :setting:`DOWNLOAD_TLS_MIN_VERSION` and
:setting:`DOWNLOAD_TLS_MAX_VERSION` settings as replacements for the
``DOWNLOADER_CLIENT_TLS_METHOD`` setting (which is now deprecated).
Compared to the old setting, they support specifying a range of allowed
versions and support newer TLS versions.
(:gh:`4821`, :gh:`6546`)
- Added :setting:`HTTPAUTH_USER`, :setting:`HTTPAUTH_PASS` and
:setting:`HTTPAUTH_DOMAIN` settings and :reqmeta:`http_user`,
:reqmeta:`http_pass` and :reqmeta:`http_auth_domain` meta keys as more
flexible ways to set HTTP authentication data.
(:gh:`7590`)
- Added a :reqmeta:`verbatim_url` meta key that can be set to ``True`` to
skip request URL canonicalization.
(:gh:`7473`)
- Added ``deny_tags`` and ``deny_attrs`` arguments to :class:`LinkExtractor
`.
(:gh:`6321`, :gh:`7679`)
- :attr:`scrapy.Item.fields` now returns the fields in the definition order
instead of the alphabetical one.
(:gh:`7015`, :gh:`7694`)
- Added a :setting:`RETRY_GIVE_UP_LOG_LEVEL` setting, a
:reqmeta:`give_up_log_level` meta key and a ``give_up_log_level`` argument
of the
:func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function that
allow changing the log level of the message logged when the retry limit has
been reached.
(:gh:`4622`, :gh:`5297`, :gh:`7567`)
- It's now possible to set :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` to
``None`` to use the default ciphers of the underlying TLS implementation.
(:gh:`7499`, :gh:`7665`)
Improvements
~~~~~~~~~~~~
- :class:`~scrapy.FormRequest` is no longer deprecated, only its
``from_response()`` method is still deprecated.
(:gh:`7561`, :gh:`7671`)
- Switched the item definition in the default project template from a
:class:`scrapy.item.Item` to a dataclass.
(:gh:`7493`, :gh:`7513`)
- Fixed deprecation warnings with pyOpenSSL 26.3.0.
(:gh:`7619`)
- Removed the runtime warnings for :attr:`Spider.allowed_domains
` containing URLs or domains with ports
instead of just domains and for spider classes having a ``start_url``
attribute instead of :class:`~scrapy.spiders.Spider.start_urls`. Please use
:doc:`scrapy-lint ` to find mistakes in your spider code
instead.
(:gh:`4421`, :gh:`7627`)
- :func:`scrapy.utils.test.get_crawler` now disables
:setting:`TELNETCONSOLE_ENABLED` by default.
(:gh:`7644`)
- Other code refactoring and improvements.
(:gh:`7409`, :gh:`7593`, :gh:`7594`, :gh:`7611`, :gh:`7649`)
Bug fixes
~~~~~~~~~
- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no
longer ignores proxy credentials for redirected or retried requests.
(:gh:`7601`, :gh:`7630`)
- :class:`~scrapy.extensions.feedexport.GCSFeedStorage` now closes the
temporary file after the upload.
(:gh:`7546`)
- Fixed ``scrapy shell `` running a full spider crawl when there is a
spider for the requested URL. This bug was introduced in Scrapy 2.13.0.
(:gh:`7552`, :gh:`7557`)
- The :setting:`IMAGES_STORE_S3_ACL` and :setting:`IMAGES_STORE_GCS_ACL`
settings are no longer ignored. This bug was introduced in Scrapy 2.12.0.
(:gh:`7597`, :gh:`7614`)
- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now closes
the connection after making the request.
(:gh:`7602`, :gh:`7667`)
- Removed the deprecated ``spider`` argument from the pipeline defined in the
default project template.
(:gh:`7676`)
- Fixed ``scrapy genspider --edit`` not working.
(:gh:`7260`, :gh:`7683`)
- When a :class:`~scrapy.crawler.Crawler` instance is passed to
:meth:`AsyncCrawlerRunner.create_crawler()
` or
:meth:`CrawlerRunner.create_crawler()
`, settings from both classes
are now merged, previously only the settings from the
:class:`~scrapy.crawler.Crawler` instance were used.
(:gh:`1280`, :gh:`7647`)
- Fixed several issues with cookie handling in
:func:`scrapy.utils.request.request_to_curl`.
(:gh:`7603`, :gh:`7675`, :gh:`7684`)
- Fixed :class:`scrapy.resolver.CachingThreadedResolver` not disabling the
cache when :setting:`DNSCACHE_ENABLED` is set to ``False``.
(:gh:`7663`)
- Fixed :func:`scrapy.utils.response.open_in_browser` not removing comments
when looking for the ```` tag.
(:gh:`7506`)
- Fixed checking for deprecated methods in custom :setting:`ITEM_PROCESSOR`
implementations.
(:gh:`7589`)
- Fixed :func:`scrapy.utils.url.strip_url` corrupting some URLs with
credentials.
(:gh:`7604`, :gh:`7605`)
- :func:`scrapy.utils.misc.rel_has_nofollow` now ignores the case when
looking for "nofollow" strings.
(:gh:`7632`)
- Fixed an exception in :class:`scrapy.utils.sitemap.Sitemap` when parsing
some malformed sitemaps.
(:gh:`7686`, :gh:`7687`)
Documentation
~~~~~~~~~~~~~
- Mentioned :doc:`scrapy-lint ` in the docs.
(:gh:`4421`, :gh:`7627`)
- Added the docs about :ref:`security considerations `.
(:gh:`7389`, :gh:`7678`)
- Improved the :ref:`item pipeline docs `.
(:gh:`2350`, :gh:`7676`)
- Documented which stats are collected by
:class:`~scrapy.extensions.corestats.CoreStats`.
(:gh:`7421`)
- Switched documentation examples from using :class:`scrapy.item.Item` to
using dataclasses.
(:gh:`7493`, :gh:`7513`)
- Added feature comparison tables to the :ref:`download handler
` docs.
(:gh:`7575`)
- Improved the docs for :ref:`logging settings `.
(:gh:`6909`, :gh:`7668`)
- Documented a way to :ref:`improve startup time and memory usage
` by using :setting:`SPIDER_MODULES`.
(:gh:`7576`, :gh:`7600`)
- Clarified handling of the ``type`` argument of :class:`~scrapy.Selector`.
(:gh:`7704`)
- Other documentation improvements and fixes.
(:gh:`4954`,
:gh:`6120`,
:gh:`7286`,
:gh:`7564`,
:gh:`7573`,
:gh:`7598`,
:gh:`7599`,
:gh:`7698`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Fixed deprecation warnings with pytest 9.1.0.
(:gh:`7621`)
- Type hints improvements and fixes.
(:gh:`6958`, :gh:`7586`)
- CI and test improvements and fixes.
(:gh:`5954`,
:gh:`7002`,
:gh:`7017`,
:gh:`7247`,
:gh:`7508`,
:gh:`7545`,
:gh:`7566`,
:gh:`7574`,
:gh:`7585`,
:gh:`7595`,
:gh:`7608`,
:gh:`7610`,
:gh:`7612`,
:gh:`7616`,
:gh:`7625`,
:gh:`7637`,
:gh:`7639`,
:gh:`7640`,
:gh:`7641`,
:gh:`7642`,
:gh:`7643`,
:gh:`7644`,
:gh:`7645`,
:gh:`7646`,
:gh:`7654`,
:gh:`7655`,
:gh:`7664`,
:gh:`7672`,
:gh:`7677`,
:gh:`7680`,
:gh:`7682`,
:gh:`7692`)
.. _release-2.16.0:
Scrapy 2.16.0 (2026-05-19)
--------------------------
Highlights:
- Official support for Python 3.14
- Support for Twisted 26.4.0+
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- Increased the minimum versions of the following dependencies:
- service_identity_: 18.1.0 → 23.1.0
(:gh:`7347`)
- Added support for Twisted 26.4.0+.
(:gh:`7347`, :gh:`7505`, :gh:`7520`)
- Added support for Python 3.14.
(:gh:`6604`, :gh:`7460`)
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The following classes and functions, intended for internal use by
:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
and :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`, have
been made private:
- ``scrapy.core.downloader.handlers.http11.ScrapyAgent``
- ``scrapy.core.downloader.handlers.http11.ScrapyProxyAgent``
- ``scrapy.core.downloader.handlers.http11.TunnelingAgent``
- ``scrapy.core.downloader.handlers.http11.TunnelingTCP4ClientEndpoint``
- ``scrapy.core.downloader.handlers.http11.tunnel_request_data()``
- ``scrapy.core.downloader.handlers.http2.ScrapyH2Agent``
(:gh:`7496`, :gh:`7510`)
Deprecations
~~~~~~~~~~~~
- ``scrapy.FormRequest`` is deprecated. You can use the :doc:`form2request
` library instead, see :ref:`form`.
(:gh:`6438`)
- ``scrapy.utils.python.MutableChain`` is deprecated.
(:gh:`7504`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- The ``start_requests()`` method of :class:`~scrapy.Spider`, deprecated in
2.13.0, is removed and no longer called. Use :meth:`~scrapy.Spider.start`
instead, or both to maintain support for lower Scrapy versions.
(:gh:`7490`)
- Support for ``process_start_requests()`` methods of :ref:`spider middlewares
`, deprecated in 2.13.0, is removed. Use
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
or both to maintain support for lower Scrapy versions.
(:gh:`7490`)
- Support for synchronous ``process_spider_output()`` methods of spider
middlewares, deprecated in Scrapy 2.13.0, is removed. You should upgrade
the affected middlewares to have asynchronous ``process_spider_output()``
methods.
(:gh:`7504`)
- The ``spider`` arguments of the following methods of
:class:`~scrapy.core.scraper.Scraper`, deprecated in Scrapy 2.13.0, are
removed:
- ``close_spider()``
- ``enqueue_scrape()``
- ``handle_spider_error()``
- ``handle_spider_output()``
(:gh:`7487`)
- HTTP/1.0 support code, deprecated in Scrapy 2.13.0, is removed. This
includes:
- ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
- The ``scrapy.core.downloader.webclient`` module.
- The ``DOWNLOADER_HTTPCLIENTFACTORY`` setting.
(:gh:`7486`)
- The following functions, deprecated in Scrapy 2.13.0, are removed, you
should import them from :mod:`w3lib.url` directly instead:
- ``scrapy.utils.url.add_or_replace_parameter()``
- ``scrapy.utils.url.add_or_replace_parameters()``
- ``scrapy.utils.url.any_to_uri()``
- ``scrapy.utils.url.canonicalize_url()``
- ``scrapy.utils.url.file_uri_to_path()``
- ``scrapy.utils.url.is_url()``
- ``scrapy.utils.url.parse_data_uri()``
- ``scrapy.utils.url.parse_url()``
- ``scrapy.utils.url.path_to_file_uri()``
- ``scrapy.utils.url.safe_download_url()``
- ``scrapy.utils.url.safe_url_string()``
- ``scrapy.utils.url.url_query_cleaner()``
- ``scrapy.utils.url.url_query_parameter()``
(:gh:`7487`)
- The following test-related code, deprecated in Scrapy 2.13.0, is removed:
- the ``scrapy.utils.testproc`` module
- the ``scrapy.utils.testsite`` module
- ``scrapy.utils.test.assert_gcs_environ()``
- ``scrapy.utils.test.get_ftp_content_and_delete()``
- ``scrapy.utils.test.get_gcs_content_and_delete()``
- ``scrapy.utils.test.mock_google_cloud_storage()``
- ``scrapy.utils.test.skip_if_no_boto()``
- ``scrapy.utils.test.TestSpider``
(:gh:`7487`)
- ``scrapy.utils.versions.scrapy_components_versions()``, deprecated in
Scrapy 2.13.0, is removed, you can use
:func:`scrapy.utils.versions.get_versions` instead.
(:gh:`7487`)
- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` and
``scrapy.utils.url.escape_ajax()``, deprecated in Scrapy 2.13.0, are
removed.
(:gh:`7487`)
- The ``__init__()`` method of priority queue classes (see
:setting:`SCHEDULER_PRIORITY_QUEUE`) now needs to support a keyword-only
``start_queue_cls`` parameter, not supporting it was deprecated in Scrapy
2.13.0.
(:gh:`7487`)
- ``scrapy.spiders.init.InitSpider``, deprecated in Scrapy 2.13.0, is
removed.
(:gh:`7487`)
New features
~~~~~~~~~~~~
- New features and improvements for
:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`:
- Support for proxies.
- Support for the :reqmeta:`download_latency` meta key.
- Support for :attr:`Response.certificate
`.
- Default headers set by the ``httpx`` library are no longer added to
requests.
(:gh:`7441`, :gh:`7524`)
- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
skips HTTPS proxy certificate verification when the
:setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting is set to ``False``.
(:gh:`7496`)
Improvements
~~~~~~~~~~~~
- :func:`time.monotonic` is used instead of :func:`time.time` to calculate
elapsed time in various places.
(:gh:`7377`)
- Improved extraction of the file extension from the URL in
:class:`~scrapy.pipelines.files.FilesPipeline`.
(:gh:`4225`, :gh:`7414`)
- Other code refactoring and improvements.
(:gh:`7401`)
Bug fixes
~~~~~~~~~
- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
raises an exception when a request has an ``https://`` destination and an
``https://`` proxy, which is not supported by this handler. Previously it
tried to connect to the proxy via HTTP in this case.
(:gh:`7496`)
- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
raises an exception for requests with ``http://`` URLs instead of trying to
connect, which is not supported by this handler.
(:gh:`7496`)
- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no longer
adds the ``:status`` pseudo-header to :attr:`Response.headers
`.
(:gh:`7441`)
- Fixed :func:`scrapy.utils.response.open_in_browser` removing the ````
tag when adding the ```` tag.
(:gh:`7459`)
Documentation
~~~~~~~~~~~~~
- Documented that
:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
doesn't support HTTPS proxies for HTTPS destinations and that
:class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` doesn't
support proxies at all.
(:gh:`7496`)
- Added an example of using
:class:`logging.handlers.TimedRotatingFileHandler` to rotate Scrapy logs.
(:gh:`3628`, :gh:`7501`)
- Added a ``CITATION.cff`` file.
(:gh:`7502`, :gh:`7519`)
- Mentioned ``DOWNLOADER_CLIENT_TLS_METHOD`` in :ref:`bans`.
(:gh:`5232`, :gh:`7518`)
- Other documentation improvements and fixes.
(:gh:`7417`,
:gh:`7463`,
:gh:`7472`,
:gh:`7480`,
:gh:`7489`,
:gh:`7503`,
:gh:`7507`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Added tests that connect to https://books.toscrape.com/ to test the
behavior with a real website. These tests are marked with the
``requires_internet`` pytest mark and can be skipped with e.g.
``-m 'not requires_internet'`` if you cannot or don't want to run them.
(:gh:`7520`)
- Type hints improvements and fixes.
(:gh:`7492`, :gh:`7532`)
- CI and test improvements and fixes.
(:gh:`7441`, :gh:`7466`, :gh:`7491`, :gh:`7496`)
.. _release-2.15.2:
Scrapy 2.15.2 (2026-04-28)
--------------------------
Bug fixes
~~~~~~~~~
- Fixed links in https://docs.scrapy.org/llms.txt (:gh:`7467`)
.. _release-2.15.1:
Scrapy 2.15.1 (2026-04-23)
--------------------------
Bug fixes
~~~~~~~~~
- Sharing of the SSL context between multiple connections, introduced in
Scrapy 2.15.0, is reverted as it caused problems and wasn't actually
needed.
(:gh:`7445`, :gh:`7450`)
- Fixed :meth:`scrapy.settings.BaseSettings.getwithbase` failing on keys with
dots that aren't import names. It now works the way it worked before Scrapy
2.15.0, without trying to match class objects and import path. A separate
method,
:func:`~scrapy.settings.BaseSettings.get_component_priority_dict_with_base`,
was added that does that, and it is now used for :ref:`component priority
dictionaries `.
(:gh:`7426`, :gh:`7449`)
- Documentation rendering improvements.
(:gh:`7452`, :gh:`7454`)
.. _release-2.15.0:
Scrapy 2.15.0 (2026-04-09)
--------------------------
Highlights:
- Experimental support for running without a Twisted reactor
- Experimental ``httpx``-based download handler
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The built-in HTTP :ref:`download handlers ` now
raise Scrapy-specific exceptions instead of implementation-specific ones,
see :ref:`download-handlers-exceptions`. This can affect user code that
handles downloader exceptions, such as ``process_exception()`` methods of
custom :ref:`downloader middlewares `.
(:gh:`7208`)
- In order to fix a long-standing bug with handling of asynchronous storages,
the following changes were made to media pipeline classes, which can impact
some of the user code that subclasses them or calls their methods directly:
- overrides of :meth:`scrapy.pipelines.media.MediaPipeline.media_downloaded`
and :meth:`~scrapy.pipelines.files.FilesPipeline.file_downloaded` can now
return coroutines
- :meth:`~scrapy.pipelines.files.FilesPipeline.media_downloaded`,
:meth:`~scrapy.pipelines.files.FilesPipeline.file_downloaded` and
:meth:`~scrapy.pipelines.images.ImagesPipeline.image_downloaded` now
return coroutines
(:gh:`2183`, :gh:`6369`, :gh:`7182`)
- ``Request`` and ``Response`` objects: ``__slots__`` and setter changes:
- :class:`scrapy.http.Request` and :class:`scrapy.http.Response` now
define ``__slots__``. Assigning arbitrary attributes to instances (for
example, ``response.foo = 1``) will raise ``AttributeError``. Store
per-request/response data in the request/response ``meta`` mapping
instead of attaching new attributes to the objects.
- If you maintain custom ``Request`` or ``Response`` subclasses that
relied on dynamic instance attributes, either add ``'__dict__'`` to
your subclass ``__slots__`` to allow dynamic attributes, or migrate
per-instance state to ``meta`` or explicit documented attributes.
- The setters for ``headers``, ``flags`` and ``cookies`` no longer coerce
falsy values into ``None``. For example, ``request.headers = {}`` now
stores an empty :class:`scrapy.http.headers.Headers` instance (not
``None``), and ``request.flags = []`` remains an empty list instead of
being set to ``None``. Update code that relied on ``is None`` checks or
the previous coercion behaviour.
(:gh:`7036`, :gh:`7367`, :gh:`7374`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- The context factory class set as the value of the
``DOWNLOADER_CLIENTCONTEXTFACTORY`` setting is now required to support the
``method`` argument of ``__init__()``, recommended since Scrapy 1.2.0.
(:gh:`7353`)
Deprecations
~~~~~~~~~~~~
- ``scrapy.mail.MailSender`` is deprecated. Please use :mod:`smtplib`,
:mod:`twisted.mail.smtp` or other 3rd party email libraries.
(:gh:`7249`, :gh:`7263`)
- The ``scrapy.extensions.statsmailer.StatsMailer`` extension is deprecated.
You can instead implement your own notifications by handling the
:signal:`spider_closed` signal.
(:gh:`7249`, :gh:`7263`)
- The ``MEMUSAGE_NOTIFY_MAIL`` setting is deprecated. You can instead
implement your own notifications by handling the
:signal:`memusage_warning_reached` and :signal:`spider_closed` signals.
(:gh:`7249`, :gh:`7263`)
- The ``DNS_RESOLVER`` setting was renamed to :setting:`TWISTED_DNS_RESOLVER`
and the old name is deprecated.
(:gh:`7350`, :gh:`7361`)
- The ``DOWNLOADER_CLIENTCONTEXTFACTORY`` setting is deprecated. If you were
using it to switch to
``scrapy.core.downloader.contextfactory.BrowserLikeContextFactory``, please
use the new :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting instead. If you
cannot use the default context factory for some other reason, please
subclass the :ref:`download handler ` instead.
(:gh:`7352`, :gh:`7379`)
- ``scrapy.core.downloader.contextfactory.BrowserLikeContextFactory`` is
deprecated. You can set the new :setting:`DOWNLOAD_VERIFY_CERTIFICATES`
setting to ``True`` instead.
(:gh:`7379`)
- The following implementation details of the context factory handling code
are deprecated:
- ``scrapy.core.downloader.contextfactory.AcceptableProtocolsContextFactory``
- ``scrapy.core.downloader.contextfactory.load_context_factory_from_settings()``
- ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory``
- ``scrapy.core.downloader.tls.ScrapyClientTLSOptions``
(:gh:`7353`, :gh:`7391`)
- Passing :class:`str` instead of :class:`bytes` to
:class:`scrapy.utils.sitemap.Sitemap` and
:func:`scrapy.utils.sitemap.sitemap_urls_from_robots` is deprecated.
(:gh:`7007`)
- ``scrapy.utils.misc.walk_modules()`` is deprecated. You can use
:func:`scrapy.utils.misc.walk_modules_iter` instead.
(:gh:`7388`)
- ``scrapy.shell.Shell.inthread`` is deprecated. You can use
:attr:`scrapy.shell.Shell.fetch_available` instead to check if
:func:`~scrapy.shell.Shell.fetch` can be used.
(:gh:`7395`)
- ``scrapy.commands.ScrapyCommand.set_crawler()`` is deprecated.
(:gh:`7276`)
New features
~~~~~~~~~~~~
- Added an *experimental* mode for running Scrapy without installing a
Twisted reactor: set :setting:`TWISTED_REACTOR_ENABLED` to ``False`` to
enable it. This mode has limitations, refer to :ref:`its documentation
` for details. As long as it's experimental, its
behavior and related features and APIs may change in future Scrapy releases
in a breaking way.
(:gh:`6219`,
:gh:`7185`,
:gh:`7186`,
:gh:`7187`,
:gh:`7188`,
:gh:`7190`,
:gh:`7197`,
:gh:`7199`,
:gh:`7209`,
:gh:`7228`,
:gh:`7355`,
:gh:`7366`,
:gh:`7385`,
:gh:`7395`)
- Added the :func:`scrapy.utils.reactorless.is_reactorless` function that
checks if there is a running asyncio event loop but no Twisted reactor.
(:gh:`7185`, :gh:`7199`)
- Changed :func:`scrapy.utils.asyncio.is_asyncio_available` to return
``True`` if there is a running asyncio loop, even if no Twisted reactor is
installed.
(:gh:`7185`, :gh:`7199`)
- Added an *experimental* download handler that uses the httpx_ library and
doesn't require a Twisted reactor:
:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. As
long as it's experimental, its behavior may change in future Scrapy
releases in a breaking way.
(:gh:`6805`, :gh:`7239`, :gh:`7368`, :gh:`7384`)
.. _httpx: https://www.python-httpx.org/
- Added the :setting:`DOWNLOAD_BIND_ADDRESS` setting as a global counterpart
to the per-request :reqmeta:`bindaddress` meta key.
(:gh:`7266`, :gh:`7283`)
- Added the :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting that can be set
to ``True`` to make Scrapy abort HTTPS requests when the server certificate
is invalid or doesn't match the domain.
(:gh:`7379`)
- The built-in HTTP :ref:`download handlers ` now
raise Scrapy-specific exceptions instead of implementation-specific ones,
to allow unified handling of similar problems caused by different
implementations. The default value of the :setting:`RETRY_EXCEPTIONS`
setting was updated replacing Twisted-specific exceptions with these new
ones. The exceptions:
- :exc:`~scrapy.exceptions.CannotResolveHostError`
- :exc:`~scrapy.exceptions.DownloadCancelledError`
- :exc:`~scrapy.exceptions.DownloadConnectionRefusedError`
- :exc:`~scrapy.exceptions.DownloadFailedError`
- :exc:`~scrapy.exceptions.DownloadTimeoutError`
- :exc:`~scrapy.exceptions.ResponseDataLossError`
- :exc:`~scrapy.exceptions.UnsupportedURLSchemeError`
(:gh:`7208`)
- Added the :signal:`memusage_warning_reached` signal emitted by the
:class:`~scrapy.extensions.memusage.MemoryUsage` extension when the memory
usage reaches :setting:`MEMUSAGE_WARNING_MB`.
(:gh:`7249`, :gh:`7263`)
- Added
:meth:`Headers.to_tuple_list() `
that returns headers as a list of ``(key, value)`` tuples.
(:gh:`7239`)
- :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler` now uses the
download handler configured for the ``"https"`` scheme to make requests
instead of always using
:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.
(:gh:`7369`, :gh:`7370`)
- Added :func:`scrapy.utils.misc.walk_modules_iter` as a replacement for
``scrapy.utils.misc.walk_modules()`` that returns an iterable instead of a
list.
(:gh:`7388`)
Improvements
~~~~~~~~~~~~
- :func:`asyncio.to_thread` is now used instead of
:func:`twisted.internet.threads.deferToThread` in the built-in feed
storages, media pipeline storages and the
:func:`scrapy.utils.decorators.inthread` decorator when available.
(:gh:`7183`, :gh:`7184`, :gh:`7349`)
- Improved memory footprint of :class:`~scrapy.Request` and
:class:`~scrapy.http.Response` objects by adding ``__slots__`` and omitting
empty lists and dicts in some internal attributes.
(:gh:`7036`, :gh:`7367`, :gh:`7374`)
- :class:`~scrapy.core.downloader.contextfactory._ScrapyClientContextFactory`
no longer mutates the SSL context, to avoid the behavior that was
deprecated in pyOpenSSL 25.1.0.
(:gh:`6859`, :gh:`7353`)
- Improved memory usage of :class:`~scrapy.spiders.sitemap.SitemapSpider` and
:class:`scrapy.utils.sitemap.Sitemap`.
(:gh:`3529`, :gh:`7007`)
- Improved the scheduling behavior of
:class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` when crawling
multiple domains.
(:gh:`7293`, :gh:`7351`)
- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
:class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now handle
TLS verbose logging (see :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING`)
directly instead of relying on
:class:`~scrapy.core.downloader.contextfactory._ScrapyClientContextFactory`.
(:gh:`7387`)
- The server certificate verification code now correctly handles certificates
with IP addresses in ``subjectAltName``.
(:gh:`7353`)
- Improved reliability of :func:`scrapy.utils.trackref.get_oldest`.
(:gh:`1758`, :gh:`7375`)
- Other code refactoring and improvements.
(:gh:`7210`, :gh:`7238`, :gh:`7376`, :gh:`7386`, :gh:`7395`,
:gh:`7405`, :gh:`7410`)
Bug fixes
~~~~~~~~~
- :ref:`Media pipelines ` should now wait for uploads
to asynchronous storages (e.g.
:class:`~scrapy.pipelines.files.S3FilesStore`) to complete.
(:gh:`2183`, :gh:`6369`, :gh:`7182`)
- Fixed merging ``*_BASE`` settings (e.g. merging
:setting:`DOWNLOADER_MIDDLEWARES` with
:setting:`DOWNLOADER_MIDDLEWARES_BASE`) when a component is referred to by
a class object in one setting and by a string import path in the other one.
(:gh:`6912`, :gh:`6993`)
- ``scrapy runspider`` and ``scrapy crawl`` now set the exit code to 1 if an
exception happened early (this was broken since Scrapy 2.13.0).
(:gh:`6820`, :gh:`7255`)
- Fixed repeated warnings about data loss (see
:setting:`DOWNLOAD_FAIL_ON_DATALOSS`) not being suppressed in
:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.
(:gh:`7222`)
- Improved FTP connection management in
:class:`scrapy.pipelines.files.FTPFilesStore`.
(:gh:`7256`)
- Fixed the ``spider`` variable in the :ref:`shell `, which
wasn't available since Scrapy 2.13.0.
(:gh:`7395`)
Documentation
~~~~~~~~~~~~~
- The ``llms.txt`` and ``llms-full.txt`` files and Markdown versions of pages
are now generated when the HTML documentation is built.
(:gh:`7380`)
- Added a "Copy as Markdown" button to the HTML documentation.
(:gh:`7380`)
- Added :ref:`docs for using Pydantic models as items `.
(:gh:`6955`, :gh:`6966`)
- Documented :ref:`job directory contents `.
(:gh:`4842`, :gh:`5260`)
- Improved docs for :attr:`~scrapy.Request.dont_filter`.
(:gh:`6398`, :gh:`7245`)
- Clarified that settings related to :setting:`TWISTED_DNS_RESOLVER` are only
taken into account if the selected resolver supports them.
(:gh:`7385`)
- Other documentation improvements and fixes.
(:gh:`7248`, :gh:`7274`, :gh:`7406`, :gh:`7408`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Added the ``no-reactor`` test environment that doesn't install a Twisted
reactor and uses ``pytest-asyncio`` instead of ``pytest-twisted`` to run
asynchronous test functions.
(:gh:`6952`, :gh:`7189`, :gh:`7233`, :gh:`7234`, :gh:`7254`,
:gh:`7259`)
- Fixed running tests with ``pytest-xdist``.
(:gh:`7216`, :gh:`7257`)
- Type hints improvements and fixes.
(:gh:`7300`, :gh:`7331`)
- CI and test improvements and fixes.
(:gh:`7060`,
:gh:`7223`,
:gh:`7232`,
:gh:`7241`,
:gh:`7250`,
:gh:`7256`,
:gh:`7276`,
:gh:`7277`,
:gh:`7279`,
:gh:`7329`,
:gh:`7363`,
:gh:`7381`,
:gh:`7402`)
.. _release-2.14.2:
Scrapy 2.14.2 (2026-03-12)
--------------------------
Security bug fixes
~~~~~~~~~~~~~~~~~~
- Values from the ``Referrer-Policy`` header of HTTP responses are no longer
executed as Python callables. See the `cwxj-rr6w-m6w7`_ security advisory
for details.
.. _cwxj-rr6w-m6w7: https://github.com/scrapy/scrapy/security/advisories/GHSA-cwxj-rr6w-m6w7
- In line with the `standard
`__, 301 redirects of
``POST`` requests are converted into ``GET`` requests.
Converting to a ``GET`` request implies not only a method change, but also
omitting the body and ``Content-*`` headers in the redirect request. On
cross-origin redirects (for example, cross-domain redirects), this is
effectively a security bug fix for scenarios where the body contains
secrets.
Deprecations
~~~~~~~~~~~~
- Passing a response URL string as the first positional argument to
:meth:`scrapy.spidermiddlewares.referer.RefererMiddleware.policy` is
deprecated. Pass a :class:`~scrapy.http.Response` instead.
The parameter has also been renamed to ``response`` to reflect this change.
The old parameter name (``resp_or_url``) is deprecated.
New features
~~~~~~~~~~~~
- Added a new setting, :setting:`REFERRER_POLICIES`, to allow customizing
supported referrer policies.
Bug fixes
~~~~~~~~~
- Made additional redirect scenarios convert to ``GET`` in line with the
`standard `__:
- Only ``POST`` 302 redirects are converted into ``GET`` requests; other
methods are preserved.
- ``HEAD`` 303 redirects are not converted into ``GET`` requests.
- ``GET`` 303 redirects do not have their body or standard ``Content-*``
headers removed.
- Redirects where the original request body is dropped now also have their
``Content-Encoding``, ``Content-Language`` and ``Content-Location`` headers
removed, in addition to the ``Content-Type`` and ``Content-Length`` headers
that were already being removed.
- Redirects now preserve the source URL fragment if the redirect URL does not
include one. This is useful when using browser-based download handlers,
such as `scrapy-playwright`_ or `scrapy-zyte-api`_, while letting Scrapy
handle redirects.
.. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright
.. _scrapy-zyte-api: https://scrapy-zyte-api.readthedocs.io/en/latest/
- The ``Referer`` header is now removed on redirect if
:class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` is disabled.
- The handling of the ``Referer`` header on redirects now takes into account
the ``Referer-Policy`` header of the response that triggers the redirect.
.. _release-2.14.1:
Scrapy 2.14.1 (2026-01-12)
--------------------------
Deprecations
~~~~~~~~~~~~
- ``scrapy.utils.defer.maybeDeferred_coro()`` is deprecated. (:gh:`7212`)
Bug fixes
~~~~~~~~~
- Fixed custom stats collectors that require a ``spider`` argument in their
``open_spider()`` and ``close_spider()`` methods not receiving the
argument when called by the engine.
Note, however, that the ``spider`` argument is now deprecated and will stop
being passed in a future version of Scrapy.
(:gh:`7213`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Replaced deprecated ``codecov/test-results-action@v1`` GitHub Action with
``codecov/codecov-action@v5``.
(:gh:`7180`, :gh:`7215`)
.. _release-2.14.0:
Scrapy 2.14.0 (2026-01-05)
--------------------------
Highlights:
- More coroutine-based replacements for Deferred-based APIs
- The default priority queue is now ``DownloaderAwarePriorityQueue``
- Dropped support for Python 3.9 and PyPy 3.10
- Improved and documented the API for custom download handlers
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- Dropped support for Python 3.9.
(:gh:`7121`)
- Dropped support for PyPy 3.10.
(:gh:`7050`)
- Increased the minimum versions of the following dependencies:
- lxml_: 4.6.0 → 4.6.4
- Pillow_ (optional dependency): 8.0.0 → 8.3.2
- botocore_ (optional dependency): 1.4.87 → 1.13.45
- Restored support for ``brotlicffi`` dropped in Scrapy 2.13.4. Its minimum
supported version is now ``1.2.0.0``.
(:gh:`7160`)
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- If you set the :setting:`TWISTED_REACTOR` setting to a :ref:`non-asyncio
value ` at the :ref:`spider level `, you
may now need to set the :setting:`FORCE_CRAWLER_PROCESS` setting to
``True`` when running Scrapy via :ref:`its command-line tool
` to avoid a reactor mismatch exception.
(:gh:`6845`)
- The ``log_count/*`` stats no longer count some of the early messages that
they counted before. While the earliest log messages, emitted before the
counter is initialized, were never counted, the counter initialization now
happens later than in previous Scrapy versions. You may need to adjust
expected values if you retrieve and compare values of these stats in your
code.
(:gh:`7046`)
- The classes listed below are now :term:`abstract base classes `. They cannot be instantiated directly and their subclasses
need to override the abstract methods listed below to be able to be
instantiated. If you previously instantiated these classes directly, you
will now need to subclass them and provide trivial (e.g. empty)
implementations for the abstract methods.
- :class:`scrapy.commands.ScrapyCommand`
- :meth:`~scrapy.commands.ScrapyCommand.run`
- :meth:`~scrapy.commands.ScrapyCommand.short_desc`
- :class:`scrapy.exporters.BaseItemExporter`
- :meth:`~scrapy.exporters.BaseItemExporter.export_item`
- :class:`scrapy.extensions.feedexport.BlockingFeedStorage`
- :meth:`~scrapy.extensions.feedexport.BlockingFeedStorage._store_in_thread`
- :class:`scrapy.middleware.MiddlewareManager`
- :meth:`~scrapy.middleware.MiddlewareManager._get_mwlist_from_settings`
- :class:`scrapy.spidermiddlewares.referer.ReferrerPolicy`
- :meth:`~scrapy.spidermiddlewares.referer.ReferrerPolicy.referrer`
(:gh:`6930`)
- Scrapy no longer passes a ``spider`` argument to any methods of the
:setting:`stats collector `. It wasn't passed in many of the
calls even in older Scrapy versions, so we don't expect existing custom
stats collector implementations to require a ``spider`` argument. If your
implementation needs a :class:`~scrapy.Spider` instance, you can get it
from the :class:`~scrapy.crawler.Crawler` instance passed to the
constructor.
(:gh:`7011`)
- :class:`scrapy.middleware.MiddlewareManager` no longer includes code for
handling ``open_spider()`` and ``close_spider()`` component methods. As
this code was only used for pipelines it was moved into
:class:`scrapy.pipelines.ItemPipelineManager`. This change should only
affect custom subclasses of :class:`~scrapy.middleware.MiddlewareManager`.
The following code was moved:
- ``scrapy.middleware.MiddlewareManager.open_spider()``
- ``scrapy.middleware.MiddlewareManager.close_spider()``
- Code in ``scrapy.middleware.MiddlewareManager._add_middleware()`` that
processes ``open_spider()`` and ``close_spider()`` component methods.
(:gh:`7006`)
- :meth:`scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware.process_request`
now returns a coroutine, previously it returned a
:class:`~twisted.internet.defer.Deferred` object or ``None``. The
``robot_parser()`` method was also changed to return a coroutine. This
change only impacts code that subclasses
:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` or
calls its methods directly.
(:gh:`6802`)
- The built-in :ref:`download handlers ` have been
refactored, changing the signatures of their methods. This change should
only affect user code that subclasses any of these handlers or calls their
methods directly.
(:gh:`6778`, :gh:`7164`)
- :meth:`scrapy.pipelines.media.MediaPipeline.process_item` now returns a
coroutine, previously it returned a
:class:`~twisted.internet.defer.Deferred` object. This
change only impacts code that calls this method directly.
(:gh:`7177`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- The ``from_settings()`` method of the following components, deprecated in
Scrapy 2.12.0, is removed. You should use ``from_crawler()`` instead.
- :class:`scrapy.dupefilters.RFPDupeFilter`
- :class:`scrapy.mail.MailSender`
- :class:`scrapy.middleware.MiddlewareManager`
- :class:`scrapy.core.downloader.contextfactory.ScrapyClientContextFactory`
- :class:`scrapy.pipelines.files.FilesPipeline`
- :class:`scrapy.pipelines.images.ImagesPipeline`
(:gh:`7126`)
- Scrapy no longer calls ``from_settings()`` methods of 3rd-party
:ref:`components `, deprecated in Scrapy 2.12.0. You
should define a ``from_crawler()`` method instead.
(:gh:`7126`)
- The initialization flow of :class:`scrapy.pipelines.media.MediaPipeline`
and its subclasses was simplified, it now mandates ``from_crawler()``
methods and ``crawler`` arguments of ``__init__()`` methods. Not using
these was deprecated in Scrapy 2.12.0.
(:gh:`7126`)
- The ``REQUEST_FINGERPRINTER_IMPLEMENTATION`` setting, deprecated in Scrapy
2.12.0, is removed.
(:gh:`7126`)
- The ``scrapy.utils.misc.create_instance()`` function, deprecated in Scrapy
2.12.0, is removed. Use :func:`scrapy.utils.misc.build_from_crawler`
instead.
(:gh:`7126`)
- The ``scrapy.core.downloader.Downloader._get_slot_key()`` function,
deprecated in Scrapy 2.12.0, is removed. Use
:meth:`scrapy.core.downloader.Downloader.get_slot_key` instead.
(:gh:`7126`)
- The ``scrapy.twisted_version`` attribute, deprecated in Scrapy 2.12.0, is
removed. You should instead use the :attr:`twisted.version` attribute
directly.
(:gh:`7126`)
- The following utility functions, deprecated in Scrapy 2.12.0, are removed:
- ``scrapy.utils.defer.process_chain_both()``
- ``scrapy.utils.python.equal_attributes()``
- ``scrapy.utils.python.flatten()``
- ``scrapy.utils.python.iflatten()``
- ``scrapy.utils.request.request_authenticate()``
- ``scrapy.utils.test.assert_samelines()``
(:gh:`7126`)
- ``scrapy.utils.serialize.ScrapyJSONDecoder``, deprecated in Scrapy 2.12.0,
is removed.
(:gh:`7126`)
- The ``scrapy.extensions.feedexport.build_storage()`` function, deprecated
in Scrapy 2.12.0, is removed, you can instead call the builder callable
directly.
(:gh:`7126`)
- ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware``, deprecated in
Scrapy 2.11.2, is removed.
:class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` should be
used instead.
(:gh:`6926`)
Deprecations
~~~~~~~~~~~~
- The following methods that return a
:class:`~twisted.internet.defer.Deferred` are deprecated in favor of their
coroutine-based replacements:
- :class:`scrapy.core.downloader.handlers.DownloadHandlers`
- ``download_request()`` (use
:meth:`~scrapy.core.downloader.handlers.DownloadHandlers.download_request_async`)
- :class:`scrapy.core.downloader.middleware.DownloaderMiddlewareManager`
- ``download()`` (use
:meth:`~scrapy.core.downloader.middleware.DownloaderMiddlewareManager.download_async`)
- :class:`scrapy.core.engine.ExecutionEngine`
- ``start()`` (use
:meth:`~scrapy.core.engine.ExecutionEngine.start_async`)
- ``stop()`` (use
:meth:`~scrapy.core.engine.ExecutionEngine.stop_async`)
- ``close()`` (use
:meth:`~scrapy.core.engine.ExecutionEngine.close_async`)
- ``open_spider()`` (use
:meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async`)
- ``close_spider()`` (use
:meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async`)
- ``download()`` (use
:meth:`~scrapy.core.engine.ExecutionEngine.download_async`)
- :class:`scrapy.core.scraper.Scraper`
- ``open_spider()`` (use
:meth:`~scrapy.core.scraper.Scraper.open_spider_async`)
- ``call_spider()`` (use
:meth:`~scrapy.core.scraper.Scraper.call_spider_async`)
- ``close_spider()`` (use
:meth:`~scrapy.core.scraper.Scraper.close_spider_async`)
- ``handle_spider_output()`` (use
:meth:`~scrapy.core.scraper.Scraper.handle_spider_output_async`)
- ``start_itemproc()`` (use
:meth:`~scrapy.core.scraper.Scraper.start_itemproc_async`)
- :class:`scrapy.core.spidermw.SpiderMiddlewareManager`
- ``scrape_response()`` (use
:meth:`~scrapy.core.spidermw.SpiderMiddlewareManager.scrape_response_async`)
- :class:`scrapy.crawler.Crawler`
- ``stop()`` (use :meth:`~scrapy.crawler.Crawler.stop_async`)
- :class:`scrapy.pipelines.ItemPipelineManager`
- ``process_item()`` (use
:meth:`~scrapy.pipelines.ItemPipelineManager.process_item_async`)
- ``open_spider()`` (use
:meth:`~scrapy.pipelines.ItemPipelineManager.open_spider_async`)
- ``close_spider()`` (use
:meth:`~scrapy.pipelines.ItemPipelineManager.close_spider_async`)
- :class:`scrapy.signalmanager.SignalManager`
- ``send_catch_log_deferred()`` (use
:meth:`~scrapy.signalmanager.SignalManager.send_catch_log_async`)
- ``scrapy.utils.signal.send_catch_log_deferred()`` (use
:func:`scrapy.utils.signal.send_catch_log_async`)
(:gh:`6791`, :gh:`6842`, :gh:`6979`, :gh:`6997`, :gh:`6999`,
:gh:`7005`, :gh:`7043`, :gh:`7069`, :gh:`7161`, :gh:`7164`)
- The following spider attributes are deprecated in favor of settings:
- ``download_maxsize`` (use :setting:`DOWNLOAD_MAXSIZE`)
- ``download_timeout`` (use :setting:`DOWNLOAD_TIMEOUT`)
- ``download_warnsize`` (use :setting:`DOWNLOAD_WARNSIZE`)
- ``max_concurrent_requests`` (use
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`)
- ``user_agent`` (use :setting:`USER_AGENT`)
(:gh:`6988`, :gh:`6994`, :gh:`7038`, :gh:`7039`, :gh:`7117`,
:gh:`7176`)
- Returning a :class:`~twisted.internet.defer.Deferred` from the following
user-defined functions is deprecated in favor of defining them as coroutine
functions:
- spider callbacks and errbacks (which was never officially supported and
may work incorrectly)
- the ``process_request()``, ``process_response()`` and
``process_exception()`` methods of custom downloader middlewares
- the ``process_item()``, ``open_spider()`` and ``close_spider()`` methods
of custom pipelines
- signal handlers
- the ``download_request()`` and ``close()`` methods of custom download
handlers
(:gh:`6718`, :gh:`6778`, :gh:`7069`, :gh:`7147`, :gh:`7148`,
:gh:`7149`, :gh:`7150`, :gh:`7151`, :gh:`7161`, :gh:`7164`,
:gh:`7179`)
- Passing a ``spider`` argument to the following methods is deprecated:
- :meth:`scrapy.core.spidermw.SpiderMiddlewareManager.process_start`
- :meth:`scrapy.core.downloader.Downloader.fetch`
- :meth:`scrapy.core.downloader.Downloader._get_slot`
- :meth:`scrapy.core.downloader.handlers.DownloadHandlers.download_request`
- all public methods of :class:`scrapy.statscollectors.StatsCollector`
- :meth:`scrapy.spidermiddlewares.base.BaseSpiderMiddleware.process_spider_output`
- :meth:`scrapy.spidermiddlewares.base.BaseSpiderMiddleware.process_spider_output_async`
- all ``process_*()`` methods of built-in downloader middlewares
- all ``process_*()`` methods of built-in spider middlewares
- :meth:`scrapy.pipelines.media.MediaPipeline.open_spider`
- :meth:`scrapy.pipelines.media.MediaPipeline.process_item`
(:gh:`6750`, :gh:`6927`, :gh:`6984`, :gh:`7006`, :gh:`7011`,
:gh:`7033`, :gh:`7037`, :gh:`7045`, :gh:`7178`)
- Instantiating subclasses of :class:`scrapy.middleware.MiddlewareManager`
without a :class:`~scrapy.crawler.Crawler` instance is deprecated.
(:gh:`6984`)
- For the following user-defined functions and methods requiring a ``spider``
argument is deprecated, if you need a :class:`~scrapy.Spider` instance
inside them you should get it from the :class:`~scrapy.crawler.Crawler`
instance (you may need to refactor your code to save that instance in e.g.
the ``from_crawler()`` method):
- the ``process_request()``, ``process_response()`` and
``process_exception()`` methods of custom downloader middlewares
- the ``process_spider_input()``, ``process_spider_output()``,
``process_spider_output_async()`` and ``process_spider_exception()``
methods of custom spider middlewares
- the ``process_item()`` method of custom pipelines
- the ``fetch()`` method of a custom :setting:`DOWNLOADER`
(:gh:`6927`, :gh:`6984`, :gh:`7006`, :gh:`7037`)
- The following things in custom download handlers are deprecated:
- not having a ``lazy`` attribute (you should define it as ``True`` if you
want to keep the current behavior)
- returning a :class:`~twisted.internet.defer.Deferred` from the
``download_request()`` method (you should refactor it to return a
coroutine; you also need to remove the ``spider`` argument when doing
this)
- not having a ``close()`` method, having a synchronous one or one that
returns a :class:`~twisted.internet.defer.Deferred` (you should refactor
it to return a coroutine or add an empty one if you don't have it)
(:gh:`6778`, :gh:`7164`)
- Custom implementations of :setting:`ITEM_PROCESSOR` should now define
``process_item_async()``, ``open_spider_async()`` and
``close_spider_async()`` methods instead of, or in addition to,
``process_item()``, ``open_spider()`` and ``close_spider()``.
(:gh:`7005`, :gh:`7043`)
- The ``CONCURRENT_REQUESTS_PER_IP`` setting is deprecated, use
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` instead.
(:gh:`6917`, :gh:`6921`)
- The ``scrapy.core.downloader.handlers.http`` module is deprecated. You
should import
:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
directly instead of importing the
``scrapy.core.downloader.handlers.http.HTTPDownloadHandler`` alias.
(:gh:`7079`)
- The ``scrapy.utils.decorators.defers()`` decorator is deprecated, you can
use :func:`twisted.internet.defer.maybeDeferred` directly or reimplement
this decorator in your code.
(:gh:`7164`)
- ``scrapy.spiders.CrawlSpider._parse_response()`` is deprecated, use
:meth:`scrapy.spiders.CrawlSpider.parse_with_rules` instead.
(:gh:`4463`, :gh:`6804`)
- The functions that add a delay to a Deferred are deprecated, their
underlying Twisted functions can be used instead, either directly if a
delay isn't needed, or with some explicit way to add a delay if it's
needed:
- ``scrapy.utils.defer.mustbe_deferred()`` (you can use
:func:`twisted.internet.defer.maybeDeferred`)
- ``scrapy.utils.defer.defer_succeed()`` (you can use
:func:`twisted.internet.defer.succeed`)
- ``scrapy.utils.defer.defer_fail()`` (you can use
:func:`twisted.internet.defer.fail`)
- ``scrapy.utils.defer.defer_result()`` (you can use
:func:`twisted.internet.defer.succeed` and
:func:`twisted.internet.defer.fail`)
(:gh:`6937`)
New features
~~~~~~~~~~~~
- Added :class:`scrapy.crawler.AsyncCrawlerProcess` and
:class:`scrapy.crawler.AsyncCrawlerRunner` as counterparts to
:class:`~scrapy.crawler.CrawlerProcess` and
:class:`~scrapy.crawler.CrawlerRunner` that offer coroutine-based APIs.
(:gh:`6789`, :gh:`6790`, :gh:`6796`, :gh:`6817`, :gh:`6845`,
:gh:`7034`)
- Added coroutine counterparts to some of the Deferred-based APIs:
- :class:`scrapy.core.downloader.handlers.DownloadHandlers`
- :meth:`~scrapy.core.downloader.handlers.DownloadHandlers.download_request_async`
(to ``download_request()``)
- :class:`scrapy.core.downloader.middleware.DownloaderMiddlewareManager`
- :meth:`~scrapy.core.downloader.middleware.DownloaderMiddlewareManager.download_async`
(to ``download()``)
- :class:`scrapy.core.engine.ExecutionEngine`
- :meth:`~scrapy.core.engine.ExecutionEngine.start_async` (to
``start()``)
- :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` (to
``stop()``)
- :meth:`~scrapy.core.engine.ExecutionEngine.close_async` (to
``close()``)
- :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async` (to
``open_spider()``)
- :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` (to
``close_spider()``)
- :meth:`~scrapy.core.engine.ExecutionEngine.download_async` (to
``download()``)
- :class:`scrapy.core.scraper.Scraper`
- :meth:`~scrapy.core.scraper.Scraper.open_spider_async` (to
``open_spider()``)
- :meth:`~scrapy.core.scraper.Scraper.close_spider_async` (to
``close_spider()``)
- :meth:`~scrapy.core.scraper.Scraper.start_itemproc_async` (to
``start_itemproc()``)
- :class:`scrapy.crawler.Crawler`
- :meth:`~scrapy.crawler.Crawler.crawl_async` (to ``crawl()``)
- :meth:`~scrapy.crawler.Crawler.stop_async` (to ``stop()``)
- :class:`scrapy.pipelines.ItemPipelineManager`
- :meth:`~scrapy.pipelines.ItemPipelineManager.process_item_async` (to
``process_item()``)
- :meth:`~scrapy.pipelines.ItemPipelineManager.open_spider_async` (to
``open_spider()``)
- :meth:`~scrapy.pipelines.ItemPipelineManager.close_spider_async` (to
``close_spider()``)
- :class:`scrapy.signalmanager.SignalManager`
- :meth:`~scrapy.signalmanager.SignalManager.send_catch_log_async` (to
``send_catch_log_deferred()``)
(:gh:`6781`, :gh:`6791`, :gh:`6792`, :gh:`6795`, :gh:`6801`,
:gh:`6817`, :gh:`6842`, :gh:`6997`, :gh:`7005`, :gh:`7043`,
:gh:`7069`,:gh:`7164`, :gh:`7202`)
- The default value of the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is now
``'scrapy.pqueues.DownloaderAwarePriorityQueue'``.
(:gh:`6924`, :gh:`6940`)
- Added :class:`scrapy.extensions.logcount.LogCount`, an enabled-by-default
extension that is responsible for the ``log_count/*`` stats. Previously,
this code was in :class:`scrapy.crawler.Crawler` and couldn't be disabled.
(:gh:`7046`)
- Added :meth:`scrapy.spiders.CrawlSpider.parse_with_rules` as a public
replacement for ``_parse_response()``.
(:gh:`4463`, :gh:`6804`)
- Added :func:`scrapy.utils.asyncio.is_asyncio_available` as an alternative
to :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` with a
future-proof name and semantics.
(:gh:`6827`)
- The API for :ref:`download handlers `, previously
undocumented, has been modernized and documented. An optional base class,
:class:`scrapy.core.downloader.handlers.base.BaseDownloadHandler`, has been
added to simplify writing custom download handlers that conform to the
current API.
(:gh:`4944`, :gh:`6778`, :gh:`7164`)
- Added :func:`scrapy.utils.defer.ensure_awaitable`, which can be helpful to
call user-defined functions that can return coroutines, Deferreds or
values directly.
(:gh:`7005`)
- The ``requests.seen`` file, written by
:class:`~scrapy.dupefilters.RFPDupeFilter` when :ref:`job persistence
` is enabled, now uses line buffering to reduce data loss in
spider crashes.
(:gh:`6019`, :gh:`7094`)
- Images downloaded by :class:`~scrapy.pipelines.images.ImagesPipeline` are
now automatically transposed based on EXIF data.
(:gh:`6525`, :gh:`6975`)
Improvements
~~~~~~~~~~~~
- Refactored internal functions to use coroutines instead of Deferreds.
(:gh:`6795`, :gh:`6852`, :gh:`6855`, :gh:`6858`, :gh:`7159`)
- Commands that don't need a :class:`~scrapy.crawler.CrawlerProcess` instance
no longer create it.
(:gh:`6824`)
- Improved :command:`shell` help formatting when using IPython 9+.
(:gh:`6915`, :gh:`6980`)
Bug fixes
~~~~~~~~~
- Setting :setting:`FILES_STORE` or :setting:`IMAGES_STORE` to ``None`` now
correctly disables the respective pipeline.
(:gh:`6964`, :gh:`6969`)
- :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` now
uses the URL set in the ```` tag as the base URL when redirecting to
a relative URL.
(:gh:`7042`, :gh:`7047`)
- Passing ``None`` as a value of the :reqmeta:`download_slot` request meta
key is now handled in the same way as not setting this meta key at all.
(:gh:`7172`)
- Fixed parsing of the first line of ``robots.txt`` files that have a BOM.
(:gh:`6195`, :gh:`7095`)
Documentation
~~~~~~~~~~~~~
- Added :ref:`documentation ` about download
handlers, their API and built-in handlers.
(:gh:`4944`, :gh:`7164`)
- Added a section about the `scrapy-spider-metadata`_ library to the
:ref:`spider argument docs `.
(:gh:`6676`, :gh:`6957`, :gh:`7116`)
.. _scrapy-spider-metadata: https://scrapy-spider-metadata.readthedocs.io/en/latest/
- Improved :ref:`the docs ` about coroutine-based
and Deferred-based APIs.
(:gh:`6800`, :gh:`7146`)
- Other documentation improvements and fixes.
(:gh:`7058`, :gh:`7076`, :gh:`7109`, :gh:`7195`, :gh:`7198`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Switched from ``twisted.trial`` to ``pytest-twisted`` and replaced
remaining ``unittest`` and ``twisted.trial`` features with ``pytest`` ones.
(:gh:`6658`, :gh:`6873`, :gh:`6884`, :gh:`6938`)
- Enabled fancy ``pytest`` asserts.
(:gh:`6888`)
- Added `Sphinx Lint`_ to the ``pre-commit`` configuration.
(:gh:`6920`)
.. _Sphinx Lint: https://github.com/sphinx-contrib/sphinx-lint
- CI and test improvements and fixes.
(:gh:`6649`,
:gh:`6769`,
:gh:`6821`,
:gh:`6835`,
:gh:`6836`,
:gh:`6846`,
:gh:`6883`,
:gh:`6885`,
:gh:`6889`,
:gh:`6905`,
:gh:`6928`,
:gh:`6933`,
:gh:`6941`,
:gh:`6942`,
:gh:`6945`,
:gh:`6947`,
:gh:`6960`,
:gh:`6968`,
:gh:`6972`,
:gh:`6974`,
:gh:`6996`,
:gh:`7003`,
:gh:`7012`,
:gh:`7013`,
:gh:`7050`,
:gh:`7059`,
:gh:`7070`,
:gh:`7073`,
:gh:`7118`,
:gh:`7127`,
:gh:`7141`,
:gh:`7143`,
:gh:`7145`,
:gh:`7173`)
- Code cleanups.
(:gh:`6803`,
:gh:`6838`,
:gh:`6849`,
:gh:`6875`,
:gh:`6876`,
:gh:`6892`,
:gh:`6930`,
:gh:`6949`,
:gh:`6970`,
:gh:`6977`,
:gh:`6986`,
:gh:`7008`,
:gh:`7177`)
.. _release-2.13.4:
Scrapy 2.13.4 (2025-11-17)
--------------------------
Security bug fixes
~~~~~~~~~~~~~~~~~~
- Improved protection against decompression bombs in
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
for responses compressed using the ``br`` and ``deflate`` methods: if a
single compressed chunk would be larger than the response size limit (see
:setting:`DOWNLOAD_MAXSIZE`) when decompressed, decompression is no longer
carried out. This is especially important for the ``br`` (Brotli) method
that can provide a very high compression ratio. Please, see the
`CVE-2025-6176`_ and `GHSA-2qfp-q593-8484`_ security advisories for more
information.
(:gh:`7134`)
.. _CVE-2025-6176: https://nvd.nist.gov/vuln/detail/CVE-2025-6176
.. _GHSA-2qfp-q593-8484: https://github.com/advisories/GHSA-2qfp-q593-8484
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- The minimum supported version of the optional ``brotli`` package is now
``1.2.0``.
(:gh:`7134`)
- The ``brotlicffi`` and ``brotlipy`` packages can no longer be used to
decompress Brotli-compressed responses. Please install the ``brotli``
package instead.
(:gh:`7134`)
Other changes
~~~~~~~~~~~~~
- Restricted the maximum supported Twisted version to ``25.5.0``, as Scrapy
currently uses some private APIs changed in later Twisted versions.
(:gh:`7142`)
- Stopped setting the ``COVERAGE_CORE`` environment variable in tests, it
didn't have an effect but caused the ``coverage`` module to produce a
warning or an error.
(:gh:`7137`)
- Removed the documentation build dependency on the deprecated
``sphinx-hoverxref`` module.
(:gh:`6786`, :gh:`6922`)
.. _release-2.13.3:
Scrapy 2.13.3 (2025-07-02)
--------------------------
- Changed the values for :setting:`DOWNLOAD_DELAY` (from ``0`` to ``1``) and
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (from ``8`` to ``1``) in the
default project template.
(:gh:`6597`, :gh:`6918`, :gh:`6923`)
- Improved :class:`scrapy.core.engine.ExecutionEngine` logic related to
initialization and exception handling, fixing several cases where the
spider would crash, hang or log an unhandled exception.
(:gh:`6783`, :gh:`6784`, :gh:`6900`, :gh:`6908`, :gh:`6910`,
:gh:`6911`)
- Fixed a Windows issue with :ref:`feed exports ` using
:class:`scrapy.extensions.feedexport.FileFeedStorage` that caused the file
to be created on the wrong drive.
(:gh:`6894`, :gh:`6897`)
- Allowed running tests with Twisted 25.5.0+ again. Pytest 8.4.1+ is now
required for running tests in non-pinned envs as support for the new
Twisted version was added in that version.
(:gh:`6893`)
- Fixed running tests with lxml 6.0.0+.
(:gh:`6919`)
- Added a deprecation notice for
``scrapy.spidermiddlewares.offsite.OffsiteMiddleware`` to :ref:`the Scrapy
2.11.2 release notes `.
(:gh:`6926`)
- Updated :ref:`contribution docs ` to refer to ruff_
instead of black_.
(:gh:`6903`)
- Added ``.venv/`` and ``.vscode/`` to ``.gitignore``.
(:gh:`6901`, :gh:`6907`)
.. _release-2.13.2:
Scrapy 2.13.2 (2025-06-09)
--------------------------
- Fixed a bug introduced in Scrapy 2.13.0 that caused results of request
errbacks to be ignored when the errback was called because of a downloader
error.
(:gh:`6861`, :gh:`6863`)
- Added a note about the behavior change of
:func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to its docs and
to the "Backward-incompatible changes" section of :ref:`the Scrapy 2.13.0
release notes `.
(:gh:`6866`)
- Improved the message in the exception raised by
:func:`scrapy.utils.test.get_reactor_settings` when there is no reactor
installed.
(:gh:`6866`)
- Updated the :class:`scrapy.crawler.CrawlerRunner` examples in
:ref:`topics-practices` to install the reactor explicitly, to fix
reactor-related errors with Scrapy 2.13.0 and later.
(:gh:`6865`)
- Fixed ``scrapy fetch`` not working with scrapy-poet_.
(:gh:`6872`)
- Fixed an exception produced by :class:`scrapy.core.engine.ExecutionEngine`
when it's closed before being fully initialized.
(:gh:`6857`, :gh:`6867`)
- Improved the README, updated the Scrapy logo in it.
(:gh:`6831`, :gh:`6833`, :gh:`6839`)
- Restricted the Twisted version used in tests to below 25.5.0, as some tests
fail with 25.5.0.
(:gh:`6878`, :gh:`6882`)
- Updated type hints for Twisted 25.5.0 changes.
(:gh:`6882`)
- Removed the old artwork.
(:gh:`6874`)
.. _release-2.13.1:
Scrapy 2.13.1 (2025-05-28)
--------------------------
- Give callback requests precedence over start requests when priority values
are the same.
This makes changes from 2.13.0 to start request handling more intuitive and
backward compatible. For scenarios where all requests have the same
priorities, in 2.13.0 all start requests were sent before the first
callback request. In 2.13.1, same as in 2.12 and lower, start requests are
only sent when there are not enough pending callback requests to reach
concurrency limits.
(:gh:`6828`)
- Added a deepwiki_ badge to the README. (:gh:`6793`)
.. _deepwiki: https://deepwiki.com/scrapy/scrapy
- Fixed a typo in the code example of :ref:`start-requests-lazy`.
(:gh:`6812`, :gh:`6815`)
- Fixed a typo in the :ref:`coroutine-support` section of the documentation.
(:gh:`6822`)
- Made this page more prominently listed in PyPI project links.
(:gh:`6826`)
.. _release-2.13.0:
Scrapy 2.13.0 (2025-05-08)
--------------------------
Highlights:
- The asyncio reactor is now enabled by default
- Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start`
(async) and changed how it is iterated
- Added the :reqmeta:`allow_offsite` request meta key
- Spider middlewares that don't support asynchronous spider output are
deprecated
- Added a base class for :ref:`universal spider middlewares
`
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- Dropped support for PyPy 3.9.
(:gh:`6613`)
- Added support for PyPy 3.11.
(:gh:`6697`)
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The default value of the :setting:`TWISTED_REACTOR` setting was changed
from ``None`` to
``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value
was used in newly generated projects since Scrapy 2.7.0 but now existing
projects that don't explicitly set this setting will also use the asyncio
reactor. You can :ref:`change this setting in your project
` to use a different reactor.
(:gh:`6659`, :gh:`6713`)
- The iteration of start requests and items no longer stops once there are
requests in the scheduler, and instead runs continuously until all start
requests have been scheduled.
To reproduce the previous behavior, see :ref:`start-requests-lazy`.
(:gh:`6729`)
- An unhandled exception from the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.open_spider` method of a
:ref:`spider middleware ` no longer stops the
crawl.
(:gh:`6729`)
- In ``scrapy.core.engine.ExecutionEngine``:
- The second parameter of ``open_spider()``, ``start_requests``, has been
removed. The start requests are determined by the ``spider`` parameter
instead (see :meth:`~scrapy.Spider.start`).
- The ``slot`` attribute has been renamed to ``_slot`` and should not be
used.
(:gh:`6729`)
- In ``scrapy.core.engine``, the ``Slot`` class has been renamed to ``_Slot``
and should not be used.
(:gh:`6729`)
- The ``slot`` :ref:`telnet variable ` has been removed.
(:gh:`6729`)
- In ``scrapy.core.spidermw.SpiderMiddlewareManager``,
``process_start_requests()`` has been replaced by ``process_start()``.
(:gh:`6729`)
- The ``scrape_func`` callable passed to
``scrapy.core.spidermw.SpiderMiddlewareManager.scrape_response()`` is now
called with 2 parameters, ``response`` and ``request``, instead of 3, and
must return a :class:`~twisted.internet.defer.Deferred` instead of an
iterable.
(:gh:`6787`)
- The now-deprecated ``start_requests()`` method, when it returns an iterable
instead of being defined as a generator, is now executed *after* the
:ref:`scheduler ` instance has been created.
(:gh:`6729`)
- When using :setting:`JOBDIR`, :ref:`start requests ` are
now serialized into their own, ``s``-suffixed priority folders. You can set
:setting:`SCHEDULER_START_DISK_QUEUE` to ``None`` or ``""`` to change that,
but the side effects may be undesirable. See
:setting:`SCHEDULER_START_DISK_QUEUE` for details.
(:gh:`6729`)
- The URL length limit, set by the :setting:`URLLENGTH_LIMIT` setting, is now
also enforced for start requests.
(:gh:`6777`)
- Calling :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` without
an installed reactor now raises an exception instead of installing a
reactor. This shouldn't affect normal Scrapy use cases, but it may affect
3rd-party test suites that use Scrapy internals such as
:class:`~scrapy.crawler.Crawler` and don't install a reactor explicitly. If
you are affected by this change, you most likely need to install the
reactor before running Scrapy code that expects it to be installed.
(:gh:`6732`, :gh:`6735`)
- The ``from_settings()`` method of
:class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`,
deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation
period (this was needed because after the introduction of the
:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and
switching built-in spider middlewares to it those middlewares need the
:class:`~scrapy.crawler.Crawler` instance at run time). Please use
``from_crawler()`` instead.
(:gh:`6693`)
- ``scrapy.utils.url.escape_ajax()`` is no longer called when a
:class:`~scrapy.Request` instance is created. It was only useful for
websites supporting the ``_escaped_fragment_`` feature which most modern
websites don't support. If you still need this you can modify the URLs
before passing them to :class:`~scrapy.Request`.
(:gh:`6523`, :gh:`6651`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- Removed old deprecated name aliases for some signals:
- ``stats_spider_opened`` (use ``spider_opened`` instead)
- ``stats_spider_closing`` and ``stats_spider_closed`` (use
``spider_closed`` instead)
- ``item_passed`` (use ``item_scraped`` instead)
- ``request_received`` (use ``request_scheduled`` instead)
(:gh:`6654`, :gh:`6655`)
Deprecations
~~~~~~~~~~~~
- The ``start_requests()`` method of :class:`~scrapy.Spider` is deprecated,
use :meth:`~scrapy.Spider.start` instead, or both to maintain support for
lower Scrapy versions.
(:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
- The ``process_start_requests()`` method of :ref:`spider middlewares
` is deprecated, use
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
or both to maintain support for lower Scrapy versions.
(:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
- The ``__init__`` method of priority queue classes (see
:setting:`SCHEDULER_PRIORITY_QUEUE`) should now support a keyword-only
``start_queue_cls`` parameter.
(:gh:`6752`)
- Spider middlewares that don't support asynchronous spider output are
deprecated. The async iterable downgrading feature, needed for using such
middlewares with asynchronous callbacks and with other spider middlewares
that produce asynchronous iterables, is also deprecated. Please update all
such middlewares to support asynchronous spider output. (:gh:`6664`)
- Functions that were imported from :mod:`w3lib.url` and re-exported in
:mod:`scrapy.utils.url` are now deprecated, you should import them from
:mod:`w3lib.url` directly. They are:
- ``scrapy.utils.url.add_or_replace_parameter()``
- ``scrapy.utils.url.add_or_replace_parameters()``
- ``scrapy.utils.url.any_to_uri()``
- ``scrapy.utils.url.canonicalize_url()``
- ``scrapy.utils.url.file_uri_to_path()``
- ``scrapy.utils.url.is_url()``
- ``scrapy.utils.url.parse_data_uri()``
- ``scrapy.utils.url.parse_url()``
- ``scrapy.utils.url.path_to_file_uri()``
- ``scrapy.utils.url.safe_download_url()``
- ``scrapy.utils.url.safe_url_string()``
- ``scrapy.utils.url.url_query_cleaner()``
- ``scrapy.utils.url.url_query_parameter()``
(:gh:`4577`, :gh:`6583`, :gh:`6586`)
- HTTP/1.0 support code is deprecated. It was disabled by default and
couldn't be used together with HTTP/1.1. If you still need it, you should
write your own download handler or copy the code from Scrapy. The
deprecations include:
- ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
- ``scrapy.core.downloader.webclient.ScrapyHTTPClientFactory``
- ``scrapy.core.downloader.webclient.ScrapyHTTPPageGetter``
- Overriding
``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory.getContext()``
(:gh:`6634`)
- The following modules and functions used only in tests are deprecated:
- the ``scrapy.utils.testproc`` module
- the ``scrapy.utils.testsite`` module
- ``scrapy.utils.test.assert_gcs_environ()``
- ``scrapy.utils.test.get_ftp_content_and_delete()``
- ``scrapy.utils.test.get_gcs_content_and_delete()``
- ``scrapy.utils.test.mock_google_cloud_storage()``
- ``scrapy.utils.test.skip_if_no_boto()``
If you need to use them in your tests or code, you can copy the code from Scrapy.
(:gh:`6696`)
- ``scrapy.utils.test.TestSpider`` is deprecated. If you need an empty spider
class you can use :class:`scrapy.utils.spider.DefaultSpider` or create your
own subclass of :class:`scrapy.Spider`.
(:gh:`6678`)
- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` is
deprecated. It was disabled by default and isn't useful for most of the
existing websites.
(:gh:`6523`, :gh:`6651`, :gh:`6656`)
- ``scrapy.utils.url.escape_ajax()`` is deprecated.
(:gh:`6523`, :gh:`6651`)
- ``scrapy.spiders.init.InitSpider`` is deprecated. If you find it useful,
you can copy its code from Scrapy.
(:gh:`6708`, :gh:`6714`)
- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use
:func:`scrapy.utils.versions.get_versions` instead.
(:gh:`6582`)
- ``BaseDupeFilter.log()`` is deprecated. It does nothing and shouldn't be
called.
(:gh:`4151`)
- Passing the ``spider`` argument to the following methods of
:class:`~scrapy.core.scraper.Scraper` is deprecated:
- ``close_spider()``
- ``enqueue_scrape()``
- ``handle_spider_error()``
- ``handle_spider_output()``
(:gh:`6764`)
New features
~~~~~~~~~~~~
- You can now yield the start requests and items of a spider from the
:meth:`~scrapy.Spider.start` spider method and from the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` spider
middleware method, both :term:`asynchronous generators `.
This makes it possible to use asynchronous code to generate those start
requests and items, e.g. reading them from a queue service or database
using an asynchronous client, without workarounds.
(:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
- Start requests are now :ref:`scheduled ` as soon as
possible.
As a result, their :attr:`~scrapy.Request.priority` is now taken into
account as soon as :setting:`CONCURRENT_REQUESTS` is reached.
(:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
- :class:`Crawler.signals ` has a new
:meth:`~scrapy.signalmanager.SignalManager.wait_for` method.
(:gh:`6729`)
- Added a new :signal:`scheduler_empty` signal.
(:gh:`6729`)
- Added new settings: :setting:`SCHEDULER_START_DISK_QUEUE` and
:setting:`SCHEDULER_START_MEMORY_QUEUE`.
(:gh:`6729`)
- Added :class:`~scrapy.spidermiddlewares.start.StartSpiderMiddleware`, which
sets :reqmeta:`is_start_request` to ``True`` on :ref:`start requests
`.
(:gh:`6729`)
- Exposed a new method of :class:`Crawler.engine
`:
:meth:`~scrapy.core.engine.ExecutionEngine.needs_backout`.
(:gh:`6729`)
- Added the :reqmeta:`allow_offsite` request meta key that can be used
instead of the more general :attr:`~scrapy.Request.dont_filter` request
attribute to skip processing of the request by
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` (but not
by other code that checks :attr:`~scrapy.Request.dont_filter`).
(:gh:`3690`, :gh:`6151`, :gh:`6366`)
- Added an optional base class for spider middlewares,
:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which can be
helpful for writing :ref:`universal spider middlewares
` without boilerplate and code duplication.
The built-in spider middlewares now inherit from this class.
(:gh:`6693`, :gh:`6777`)
- :ref:`Scrapy add-ons ` can now define a class method called
``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings
`.
(:gh:`6544`, :gh:`6568`)
- Added :ref:`helpers ` for modifying :ref:`component
priority dictionary ` settings.
(:gh:`6614`)
- Responses that use an unknown/unsupported encoding now produce a warning.
If Scrapy knows that installing an additional package (such as brotli_)
will allow decoding the response, that will be mentioned in the warning.
(:gh:`4697`, :gh:`6618`)
- Added the ``spider_exceptions/count`` stat which tracks the total count of
exceptions (tracked also by per-type ``spider_exceptions/*`` stats).
(:gh:`6739`, :gh:`6740`)
- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the
:attr:`scrapy.exceptions.DropItem.log_level` attribute that allow
customizing the log level of the message that is logged when an item is
dropped.
(:gh:`6603`, :gh:`6608`)
- Added support for the ``-b, --cookie`` curl argument to
:meth:`scrapy.Request.from_curl`.
(:gh:`6684`)
- Added the :setting:`LOG_VERSIONS` setting that allows customizing the
list of software whose versions are logged when the spider starts.
(:gh:`6582`)
- Added the :setting:`WARN_ON_GENERATOR_RETURN_VALUE` setting that allows
disabling run time analysis of callback code used to warn about incorrect
``return`` statements in generator-based callbacks. You may need to disable
this setting if this analysis breaks on your callback code.
(:gh:`6731`, :gh:`6738`)
Improvements
~~~~~~~~~~~~
- Removed or postponed some calls of :func:`itemadapter.is_item` to increase
performance.
(:gh:`6719`)
- Improved the error message when running a ``scrapy`` command that requires
a project (such as ``scrapy crawl``) outside of a project directory.
(:gh:`2349`, :gh:`3426`)
- Added an empty :setting:`ADDONS` setting to the ``settings.py`` template
for new projects.
(:gh:`6587`)
Bug fixes
~~~~~~~~~
- Yielding an item from :meth:`Spider.start ` or from
:meth:`SpiderMiddleware.process_start
` no longer delays
the next iteration of starting requests and items by up to 5 seconds.
(:gh:`6729`)
- Fixed calculation of ``items_per_minute`` and ``responses_per_minute``
stats.
(:gh:`6599`)
- Fixed an error initializing
:class:`scrapy.extensions.feedexport.GCSFeedStorage`.
(:gh:`6617`, :gh:`6628`)
- Fixed an error running ``scrapy bench``.
(:gh:`6632`, :gh:`6633`)
- Fixed duplicated log messages about the reactor and the event loop.
(:gh:`6636`, :gh:`6657`)
- Fixed resolving type annotations of ``SitemapSpider._parse_sitemap()`` at
run time, required by tools such as scrapy-poet_.
(:gh:`6665`, :gh:`6671`)
.. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet
- Calling :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` without
an installed reactor now raises an exception instead of installing a
reactor.
(:gh:`6732`, :gh:`6735`)
- Restored support for the ``x-gzip`` content encoding.
(:gh:`6618`)
Documentation
~~~~~~~~~~~~~
- Documented the setting values set in the default project template.
(:gh:`6762`, :gh:`6775`)
- Improved the docs about asynchronous iterable support in spider
middlewares. (:gh:`6688`)
- Improved the :ref:`docs ` about using
:class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based
code and included a list of such APIs.
(:gh:`6677`, :gh:`6734`, :gh:`6776`)
- Improved the :ref:`contribution docs `.
(:gh:`6561`, :gh:`6575`)
- Removed the ``Splash`` recommendation from the :ref:`headless browser
` suggestion. We no longer recommend using
``Splash`` and recommend using other headless browser solutions instead.
(:gh:`6642`, :gh:`6701`)
- Added the dark mode to the HTML documentation.
(:gh:`6653`)
- Other documentation improvements and fixes.
(:gh:`4151`,
:gh:`6526`,
:gh:`6620`,
:gh:`6621`,
:gh:`6622`,
:gh:`6623`,
:gh:`6624`,
:gh:`6721`,
:gh:`6723`,
:gh:`6780`)
Packaging
~~~~~~~~~
- Switched from ``setup.py`` to ``pyproject.toml``.
(:gh:`6514`, :gh:`6547`)
- Switched the build backend from setuptools_ to hatchling_.
(:gh:`6771`)
.. _hatchling: https://pypi.org/project/hatchling/
Quality assurance
~~~~~~~~~~~~~~~~~
- Replaced most linters with ruff_.
(:gh:`6565`,
:gh:`6576`,
:gh:`6577`,
:gh:`6581`,
:gh:`6584`,
:gh:`6595`,
:gh:`6601`,
:gh:`6631`)
.. _ruff: https://docs.astral.sh/ruff/
- Improved accuracy and performance of collecting test coverage.
(:gh:`6255`, :gh:`6610`)
- Fixed an error that prevented running tests from directories other than the
top level source directory.
(:gh:`6567`)
- Reduced the amount of ``mockserver`` calls in tests to improve the overall
test run time.
(:gh:`6637`, :gh:`6648`)
- Fixed tests that were running the same test code more than once.
(:gh:`6646`, :gh:`6647`, :gh:`6650`)
- Refactored tests to use more ``pytest`` features instead of ``unittest``
ones where possible.
(:gh:`6678`,
:gh:`6680`,
:gh:`6695`,
:gh:`6699`,
:gh:`6700`,
:gh:`6702`,
:gh:`6709`,
:gh:`6710`,
:gh:`6711`,
:gh:`6712`,
:gh:`6725`)
- Type hints improvements and fixes.
(:gh:`6578`,
:gh:`6579`,
:gh:`6593`,
:gh:`6605`,
:gh:`6694`)
- CI and test improvements and fixes.
(:gh:`5360`,
:gh:`6271`,
:gh:`6547`,
:gh:`6560`,
:gh:`6602`,
:gh:`6607`,
:gh:`6609`,
:gh:`6613`,
:gh:`6619`,
:gh:`6626`,
:gh:`6679`,
:gh:`6703`,
:gh:`6704`,
:gh:`6716`,
:gh:`6720`,
:gh:`6722`,
:gh:`6724`,
:gh:`6741`,
:gh:`6743`,
:gh:`6766`,
:gh:`6770`,
:gh:`6772`,
:gh:`6773`)
- Code cleanups.
(:gh:`6600`,
:gh:`6606`,
:gh:`6635`,
:gh:`6764`)
.. _release-2.12.0:
Scrapy 2.12.0 (2024-11-18)
--------------------------
Highlights:
- Dropped support for Python 3.8, added support for Python 3.13
- ``scrapy.Spider.start_requests()`` can now yield items
- Added :class:`~scrapy.http.JsonResponse`
- Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM`
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- Dropped support for Python 3.8.
(:gh:`6466`, :gh:`6472`)
- Added support for Python 3.13.
(:gh:`6166`)
- Minimum versions increased for these dependencies:
- Twisted_: 18.9.0 → 21.7.0
- cryptography_: 36.0.0 → 37.0.0
- pyOpenSSL_: 21.0.0 → 22.0.0
- lxml_: 4.4.1 → 4.6.0
- Removed ``setuptools`` from the dependency list.
(:gh:`6487`)
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- User-defined cookies for HTTPS requests will have the ``secure`` flag set
to ``True`` unless it's set to ``False`` explicitly. This is important when
these cookies are reused in HTTP requests, e.g. after a redirect to an HTTP
URL.
(:gh:`6357`)
- The Reppy-based ``robots.txt`` parser,
``scrapy.robotstxt.ReppyRobotParser``, was removed, as it doesn't support
Python 3.9+.
(:gh:`5230`, :gh:`6099`, :gh:`6499`)
- The initialization API of :class:`scrapy.pipelines.media.MediaPipeline` and
its subclasses was improved and it's possible that some previously working
usage scenarios will no longer work. It can only affect you if you define
custom subclasses of ``MediaPipeline`` or create instances of these
pipelines via ``from_settings()`` or ``__init__()`` calls instead of
``from_crawler()`` calls.
Previously, ``MediaPipeline.from_crawler()`` called the ``from_settings()``
method if it existed or the ``__init__()`` method otherwise, and then did
some additional initialization using the ``crawler`` instance. If the
``from_settings()`` method existed (like in ``FilesPipeline``) it called
``__init__()`` to create the instance. It wasn't possible to override
``from_crawler()`` without calling ``MediaPipeline.from_crawler()`` from it
which, in turn, couldn't be called in some cases (including subclasses of
``FilesPipeline``).
Now, in line with the general usage of ``from_crawler()`` and
``from_settings()`` and the deprecation of the latter the recommended
initialization order is the following one:
- All ``__init__()`` methods should take a ``crawler`` argument. If they
also take a ``settings`` argument they should ignore it, using
``crawler.settings`` instead. When they call ``__init__()`` of the base
class they should pass the ``crawler`` argument to it too.
- A ``from_settings()`` method shouldn't be defined. Class-specific
initialization code should go into either an overridden ``from_crawler()``
method or into ``__init__()``.
- It's now possible to override ``from_crawler()`` and it's not necessary
to call ``MediaPipeline.from_crawler()`` in it if other recommendations
were followed.
- If pipeline instances were created with ``from_settings()`` or
``__init__()`` calls (which wasn't supported even before, as it missed
important initialization code), they should now be created with
``from_crawler()`` calls.
(:gh:`6540`)
- The ``response_body`` argument of :meth:`ImagesPipeline.convert_image
` is now
positional-only, as it was changed from optional to required.
(:gh:`6500`)
- The ``convert`` argument of :func:`scrapy.utils.conf.build_component_list`
is now positional-only, as the preceding argument (``custom``) was removed.
(:gh:`6500`)
- The ``overwrite_output`` argument of
:func:`scrapy.utils.conf.feed_process_params_from_cli` is now
positional-only, as the preceding argument (``output_format``) was removed.
(:gh:`6500`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- Removed the ``scrapy.utils.request.request_fingerprint()`` function,
deprecated in Scrapy 2.7.0.
(:gh:`6212`, :gh:`6213`)
- Removed support for value ``"2.6"`` of setting
``REQUEST_FINGERPRINTER_IMPLEMENTATION``, deprecated in Scrapy 2.7.0.
(:gh:`6212`, :gh:`6213`)
- :class:`~scrapy.dupefilters.RFPDupeFilter` subclasses now require
supporting the ``fingerprinter`` parameter in their ``__init__`` method,
introduced in Scrapy 2.7.0.
(:gh:`6102`, :gh:`6113`)
- Removed the ``scrapy.downloadermiddlewares.decompression`` module,
deprecated in Scrapy 2.7.0.
(:gh:`6100`, :gh:`6113`)
- Removed the ``scrapy.utils.response.response_httprepr()`` function,
deprecated in Scrapy 2.6.0.
(:gh:`6111`, :gh:`6116`)
- Spiders with spider-level HTTP authentication, i.e. with the ``http_user``
or ``http_pass`` attributes, must now define ``http_auth_domain`` as well,
which was introduced in Scrapy 2.5.1.
(:gh:`6103`, :gh:`6113`)
- :ref:`Media pipelines ` methods ``file_path()``,
``file_downloaded()``, ``get_images()``, ``image_downloaded()``,
``media_downloaded()``, ``media_to_download()``, and ``thumb_path()`` must
now support an ``item`` parameter, added in Scrapy 2.4.0.
(:gh:`6107`, :gh:`6113`)
- The ``__init__()`` and ``from_crawler()`` methods of :ref:`feed storage
backend classes ` must now support the keyword-only
``feed_options`` parameter, introduced in Scrapy 2.4.0.
(:gh:`6105`, :gh:`6113`)
- Removed the ``scrapy.loader.common`` and ``scrapy.loader.processors``
modules, deprecated in Scrapy 2.3.0.
(:gh:`6106`, :gh:`6113`)
- Removed the ``scrapy.utils.misc.extract_regex()`` function, deprecated in
Scrapy 2.3.0.
(:gh:`6106`, :gh:`6113`)
- Removed the ``scrapy.http.JSONRequest`` class, replaced with
``JsonRequest`` in Scrapy 1.8.0.
(:gh:`6110`, :gh:`6113`)
- ``scrapy.utils.log.logformatter_adapter`` no longer supports missing
``args``, ``level``, or ``msg`` parameters, and no longer supports a
``format`` parameter, all scenarios that were deprecated in Scrapy 1.0.0.
(:gh:`6109`, :gh:`6116`)
- A custom class assigned to the :setting:`SPIDER_LOADER_CLASS` setting that
does not implement the ``ISpiderLoader`` interface
will now raise a :exc:`zope.interface.verify.DoesNotImplement` exception at
run time. Non-compliant classes have been triggering a deprecation warning
since Scrapy 1.0.0.
(:gh:`6101`, :gh:`6113`)
- Removed the ``--output-format``/``-t`` command line option, deprecated in
Scrapy 2.1.0. ``-O :`` should be used instead.
(:gh:`6500`)
- Running :meth:`~scrapy.crawler.Crawler.crawl` more than once on the same
:class:`~scrapy.crawler.Crawler` instance, deprecated in Scrapy 2.11.0, now
raises an exception.
(:gh:`6500`)
- Subclassing
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
without support for the ``crawler`` argument in ``__init__()`` and without
a custom ``from_crawler()`` method, deprecated in Scrapy 2.5.0, is no
longer allowed.
(:gh:`6500`)
- Removed the ``EXCEPTIONS_TO_RETRY`` attribute of
:class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`, deprecated in
Scrapy 2.10.0.
(:gh:`6500`)
- Removed support for :ref:`S3 feed exports ` without
the boto3_ package installed, deprecated in Scrapy 2.10.0.
(:gh:`6500`)
- Removed the ``scrapy.extensions.feedexport._FeedSlot`` class, deprecated in
Scrapy 2.10.0.
(:gh:`6500`)
- Removed the ``scrapy.pipelines.images.NoimagesDrop`` exception, deprecated
in Scrapy 2.8.0.
(:gh:`6500`)
- The ``response_body`` argument of :meth:`ImagesPipeline.convert_image
` is now required,
not passing it was deprecated in Scrapy 2.8.0.
(:gh:`6500`)
- Removed the ``custom`` argument of
:func:`scrapy.utils.conf.build_component_list`, deprecated in Scrapy
2.10.0.
(:gh:`6500`)
- Removed the ``scrapy.utils.reactor.get_asyncio_event_loop_policy()``
function, deprecated in Scrapy 2.9.0. Use :func:`asyncio.get_event_loop`
and related standard library functions instead.
(:gh:`6500`)
Deprecations
~~~~~~~~~~~~
- The ``from_settings()`` methods of the :ref:`Scrapy components
` that have them are now deprecated. ``from_crawler()``
should now be used instead. Affected components:
- :class:`scrapy.dupefilters.RFPDupeFilter`
- :class:`scrapy.mail.MailSender`
- :class:`scrapy.middleware.MiddlewareManager`
- :class:`scrapy.core.downloader.contextfactory.ScrapyClientContextFactory`
- :class:`scrapy.pipelines.files.FilesPipeline`
- :class:`scrapy.pipelines.images.ImagesPipeline`
- :class:`scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`
(:gh:`6540`)
- It's now deprecated to have a ``from_settings()`` method but no
``from_crawler()`` method in 3rd-party :ref:`Scrapy components
`. You can define a simple ``from_crawler()`` method
that calls ``cls.from_settings(crawler.settings)`` to fix this if you don't
want to refactor the code. Note that if you have a ``from_crawler()``
method Scrapy will not call the ``from_settings()`` method so the latter
can be removed.
(:gh:`6540`)
- The initialization API of :class:`scrapy.pipelines.media.MediaPipeline` and
its subclasses was improved and some old usage scenarios are now deprecated
(see also the "Backward-incompatible changes" section). Specifically:
- It's deprecated to define an ``__init__()`` method that doesn't take a
``crawler`` argument.
- It's deprecated to call an ``__init__()`` method without passing a
``crawler`` argument. If it's passed, it's also deprecated to pass a
``settings`` argument, which will be ignored anyway.
- Calling ``from_settings()`` is deprecated, use ``from_crawler()``
instead.
- Overriding ``from_settings()`` is deprecated, override ``from_crawler()``
instead.
(:gh:`6540`)
- The ``REQUEST_FINGERPRINTER_IMPLEMENTATION`` setting is now deprecated.
(:gh:`6212`, :gh:`6213`)
- The ``scrapy.utils.misc.create_instance()`` function is now deprecated, use
:func:`scrapy.utils.misc.build_from_crawler` instead.
(:gh:`5523`, :gh:`5884`, :gh:`6162`, :gh:`6169`, :gh:`6540`)
- ``scrapy.core.downloader.Downloader._get_slot_key()`` is deprecated, use
:meth:`scrapy.core.downloader.Downloader.get_slot_key` instead.
(:gh:`6340`, :gh:`6352`)
- ``scrapy.utils.defer.process_chain_both()`` is now deprecated.
(:gh:`6397`)
- ``scrapy.twisted_version`` is now deprecated, you should instead use
:attr:`twisted.version` directly (but note that it's an
``incremental.Version`` object, not a tuple).
(:gh:`6509`, :gh:`6512`)
- ``scrapy.utils.python.flatten()`` and ``scrapy.utils.python.iflatten()``
are now deprecated.
(:gh:`6517`, :gh:`6519`)
- ``scrapy.utils.python.equal_attributes()`` is now deprecated.
(:gh:`6517`, :gh:`6519`)
- ``scrapy.utils.request.request_authenticate()`` is now deprecated, you
should instead just set the ``Authorization`` header directly.
(:gh:`6517`, :gh:`6519`)
- ``scrapy.utils.serialize.ScrapyJSONDecoder`` is now deprecated, it didn't
contain any code since Scrapy 1.0.0.
(:gh:`6517`, :gh:`6519`)
- ``scrapy.utils.test.assert_samelines()`` is now deprecated.
(:gh:`6517`, :gh:`6519`)
- ``scrapy.extensions.feedexport.build_storage()`` is now deprecated. You can
instead call the builder callable directly.
(:gh:`6540`)
New features
~~~~~~~~~~~~
- ``scrapy.Spider.start_requests()`` can now yield items.
(:gh:`5289`, :gh:`6417`)
.. note:: Some spider middlewares may need to be updated for Scrapy 2.12
support before you can use them in combination with the ability to
yield items from ``start_requests()``.
- Added a new :class:`~scrapy.http.Response` subclass,
:class:`~scrapy.http.JsonResponse`, for responses with a `JSON MIME type
`_.
(:gh:`6069`, :gh:`6171`, :gh:`6174`)
- The :class:`~scrapy.extensions.logstats.LogStats` extension now adds
``items_per_minute`` and ``responses_per_minute`` to the :ref:`stats
` when the spider closes.
(:gh:`4110`, :gh:`4111`)
- Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM` which allows closing the
spider if no items were scraped in a set amount of time.
(:gh:`6434`)
- User-defined cookies can now include the ``secure`` field.
(:gh:`6357`)
- Added component getters to :class:`~scrapy.crawler.Crawler`:
:meth:`~scrapy.crawler.Crawler.get_addon`,
:meth:`~scrapy.crawler.Crawler.get_downloader_middleware`,
:meth:`~scrapy.crawler.Crawler.get_extension`,
:meth:`~scrapy.crawler.Crawler.get_item_pipeline`,
:meth:`~scrapy.crawler.Crawler.get_spider_middleware`.
(:gh:`6181`)
- Slot delay updates by the :ref:`AutoThrottle extension
` based on response latencies can now be disabled for
specific requests via the :reqmeta:`autothrottle_dont_adjust_delay` meta
key.
(:gh:`6246`, :gh:`6527`)
- If :setting:`SPIDER_LOADER_WARN_ONLY` is set to ``True``,
:class:`~scrapy.spiderloader.SpiderLoader` does not raise
:exc:`SyntaxError` but emits a warning instead.
(:gh:`6483`, :gh:`6484`)
- Added support for multiple-compressed responses (ones with several
encodings in the ``Content-Encoding`` header).
(:gh:`5143`, :gh:`5964`, :gh:`6063`)
- Added support for multiple standard values in :setting:`REFERRER_POLICY`.
(:gh:`6381`)
- Added support for brotlicffi_ (previously named brotlipy_). brotli_ is
still recommended but only brotlicffi_ works on PyPy.
(:gh:`6263`, :gh:`6269`)
.. _brotlicffi: https://github.com/python-hyper/brotlicffi
- Added :class:`~scrapy.contracts.default.MetadataContract` that sets the
request meta.
(:gh:`6468`, :gh:`6469`)
Improvements
~~~~~~~~~~~~
- Extended the list of file extensions that
:class:`LinkExtractor `
ignores by default.
(:gh:`6074`, :gh:`6125`)
- :func:`scrapy.utils.httpobj.urlparse_cached` is now used in more places
instead of :func:`urllib.parse.urlparse`.
(:gh:`6228`, :gh:`6229`)
Bug fixes
~~~~~~~~~
- :class:`~scrapy.pipelines.media.MediaPipeline` is now an abstract class and
its methods that were expected to be overridden in subclasses are now
abstract methods.
(:gh:`6365`, :gh:`6368`)
- Fixed handling of invalid ``@``-prefixed lines in contract extraction.
(:gh:`6383`, :gh:`6388`)
- Importing ``scrapy.extensions.telnet`` no longer installs the default
reactor.
(:gh:`6432`)
- Reduced log verbosity for dropped requests that was increased in 2.11.2.
(:gh:`6433`, :gh:`6475`)
Documentation
~~~~~~~~~~~~~
- Added ``SECURITY.md`` that documents the security policy.
(:gh:`5364`, :gh:`6051`)
- Example code for :ref:`running Scrapy from a script ` no
longer imports ``twisted.internet.reactor`` at the top level, which caused
problems with non-default reactors when this code was used unmodified.
(:gh:`6361`, :gh:`6374`)
- Documented the :class:`~scrapy.extensions.spiderstate.SpiderState`
extension.
(:gh:`6278`, :gh:`6522`)
- Other documentation improvements and fixes.
(:gh:`5920`,
:gh:`6094`,
:gh:`6177`,
:gh:`6200`,
:gh:`6207`,
:gh:`6216`,
:gh:`6223`,
:gh:`6317`,
:gh:`6328`,
:gh:`6389`,
:gh:`6394`,
:gh:`6402`,
:gh:`6411`,
:gh:`6427`,
:gh:`6429`,
:gh:`6440`,
:gh:`6448`,
:gh:`6449`,
:gh:`6462`,
:gh:`6497`,
:gh:`6506`,
:gh:`6507`,
:gh:`6524`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Added ``py.typed``, in line with `PEP 561
`_.
(:gh:`6058`, :gh:`6059`)
- Fully covered the code with type hints (except for the most complicated
parts, mostly related to ``twisted.web.http`` and other Twisted parts
without type hints).
(:gh:`5989`,
:gh:`6097`,
:gh:`6127`,
:gh:`6129`,
:gh:`6130`,
:gh:`6133`,
:gh:`6143`,
:gh:`6191`,
:gh:`6268`,
:gh:`6274`,
:gh:`6275`,
:gh:`6276`,
:gh:`6279`,
:gh:`6325`,
:gh:`6326`,
:gh:`6333`,
:gh:`6335`,
:gh:`6336`,
:gh:`6337`,
:gh:`6341`,
:gh:`6353`,
:gh:`6356`,
:gh:`6370`,
:gh:`6371`,
:gh:`6384`,
:gh:`6385`,
:gh:`6387`,
:gh:`6391`,
:gh:`6395`,
:gh:`6414`,
:gh:`6422`,
:gh:`6460`,
:gh:`6466`,
:gh:`6472`,
:gh:`6494`,
:gh:`6498`,
:gh:`6516`)
- Improved Bandit_ checks.
(:gh:`6260`, :gh:`6264`, :gh:`6265`)
- Added pyupgrade_ to the ``pre-commit`` configuration.
(:gh:`6392`)
.. _pyupgrade: https://github.com/asottile/pyupgrade
- Added ``flake8-bugbear``, ``flake8-comprehensions``, ``flake8-debugger``,
``flake8-docstrings``, ``flake8-string-format`` and
``flake8-type-checking`` to the ``pre-commit`` configuration.
(:gh:`6406`, :gh:`6413`)
- CI and test improvements and fixes.
(:gh:`5285`,
:gh:`5454`,
:gh:`5997`,
:gh:`6078`,
:gh:`6084`,
:gh:`6087`,
:gh:`6132`,
:gh:`6153`,
:gh:`6154`,
:gh:`6201`,
:gh:`6231`,
:gh:`6232`,
:gh:`6235`,
:gh:`6236`,
:gh:`6242`,
:gh:`6245`,
:gh:`6253`,
:gh:`6258`,
:gh:`6259`,
:gh:`6270`,
:gh:`6272`,
:gh:`6286`,
:gh:`6290`,
:gh:`6296`
:gh:`6367`,
:gh:`6372`,
:gh:`6403`,
:gh:`6416`,
:gh:`6435`,
:gh:`6489`,
:gh:`6501`,
:gh:`6504`,
:gh:`6511`,
:gh:`6543`,
:gh:`6545`)
- Code cleanups.
(:gh:`6196`,
:gh:`6197`,
:gh:`6198`,
:gh:`6199`,
:gh:`6254`,
:gh:`6257`,
:gh:`6285`,
:gh:`6305`,
:gh:`6343`,
:gh:`6349`,
:gh:`6386`,
:gh:`6415`,
:gh:`6463`,
:gh:`6470`,
:gh:`6499`,
:gh:`6505`,
:gh:`6510`,
:gh:`6531`,
:gh:`6542`)
Other
~~~~~
- Issue tracker improvements. (:gh:`6066`)
.. _release-2.11.2:
Scrapy 2.11.2 (2024-05-14)
--------------------------
Security bug fixes
~~~~~~~~~~~~~~~~~~
- Redirects to non-HTTP protocols are no longer followed. Please, see the
`23j4-mw76-5v7h security advisory`_ for more information. (:gh:`457`)
.. _23j4-mw76-5v7h security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h
- The ``Authorization`` header is now dropped on redirects to a different
scheme (``http://`` or ``https://``) or port, even if the domain is the
same. Please, see the `4qqq-9vqf-3h3f security advisory`_ for more
information.
.. _4qqq-9vqf-3h3f security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-4qqq-9vqf-3h3f
- When using system proxy settings that are different for ``http://`` and
``https://``, redirects to a different URL scheme will now also trigger the
corresponding change in proxy settings for the redirected request. Please,
see the `jm3v-qxmh-hxwv security advisory`_ for more information.
(:gh:`767`)
.. _jm3v-qxmh-hxwv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv
- :attr:`Spider.allowed_domains ` is now
enforced for all requests, and not only requests from spider callbacks.
(:gh:`1042`, :gh:`2241`, :gh:`6358`)
- :func:`~scrapy.utils.iterators.xmliter_lxml` no longer resolves XML
entities. (:gh:`6265`)
- defusedxml_ is now used to make
:class:`scrapy.http.request.rpc.XmlRpcRequest` more secure.
(:gh:`6250`, :gh:`6251`)
.. _defusedxml: https://github.com/tiran/defusedxml
Deprecations
~~~~~~~~~~~~
- ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware`` (a spider
middleware) is now deprecated and not enabled by default. The new
downloader middleware with the same functionality,
:class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, is enabled
instead.
(:gh:`2241`, :gh:`6358`)
Bug fixes
~~~~~~~~~
- Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in
favor of brotli_. (:gh:`6261`)
.. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli
instead if you can.
- Make :setting:`METAREFRESH_IGNORE_TAGS` ``["noscript"]`` by default. This
prevents
:class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` from
following redirects that would not be followed by web browsers with
JavaScript enabled. (:gh:`6342`, :gh:`6347`)
- During :ref:`feed export `, do not close the
underlying file from :ref:`built-in post-processing plugins
`.
(:gh:`5932`, :gh:`6178`, :gh:`6239`)
- :class:`LinkExtractor `
now properly applies the ``unique`` and ``canonicalize`` parameters.
(:gh:`3273`, :gh:`6221`)
- Do not initialize the scheduler disk queue if :setting:`JOBDIR` is an empty
string. (:gh:`6121`, :gh:`6124`)
- Fix :attr:`Spider.logger ` not logging custom extra
information. (:gh:`6323`, :gh:`6324`)
- ``robots.txt`` files with a non-UTF-8 encoding no longer prevent parsing
the UTF-8-compatible (e.g. ASCII) parts of the document.
(:gh:`6292`, :gh:`6298`)
- :meth:`scrapy.http.cookies.WrappedRequest.get_header` no longer raises an
exception if ``default`` is ``None``.
(:gh:`6308`, :gh:`6310`)
- :class:`~scrapy.Selector` now uses
:func:`scrapy.utils.response.get_base_url` to determine the base URL of a
given :class:`~scrapy.http.Response`. (:gh:`6265`)
- The :meth:`media_to_download` method of :ref:`media pipelines
` now logs exceptions before stripping them.
(:gh:`5067`, :gh:`5068`)
- When passing a callback to the :command:`parse` command, build the callback
callable with the right signature.
(:gh:`6182`)
Documentation
~~~~~~~~~~~~~
- Add a FAQ entry about :ref:`creating blank requests `.
(:gh:`6203`, :gh:`6208`)
- Document that :attr:`scrapy.Selector.type` can be ``"json"``.
(:gh:`6328`, :gh:`6334`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Make builds reproducible. (:gh:`5019`, :gh:`6322`)
- Packaging and test fixes.
(:gh:`6286`, :gh:`6290`, :gh:`6312`, :gh:`6316`, :gh:`6344`)
.. _release-2.11.1:
Scrapy 2.11.1 (2024-02-14)
--------------------------
Highlights:
- Security bug fixes.
- Support for Twisted >= 23.8.0.
- Documentation improvements.
Security bug fixes
~~~~~~~~~~~~~~~~~~
- Addressed `ReDoS vulnerabilities`_:
- ``scrapy.utils.iterators.xmliter`` is now deprecated in favor of
:func:`~scrapy.utils.iterators.xmliter_lxml`, which
:class:`~scrapy.spiders.XMLFeedSpider` now uses.
To minimize the impact of this change on existing code,
:func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
the node namespace with a prefix in the node name, and big files with
highly nested trees when using libxml2 2.7+.
- Fixed regular expressions in the implementation of the
:func:`~scrapy.utils.response.open_in_browser` function.
Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.
.. _ReDoS vulnerabilities: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
.. _cc65-xxvf-f7r9 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cc65-xxvf-f7r9
- :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
advisory`_ for more information.
.. _7j7m-v7m3-jqm7 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-7j7m-v7m3-jqm7
- Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, the
deprecated ``scrapy.downloadermiddlewares.decompression`` module has been
removed.
- The ``Authorization`` header is now dropped on redirects to a different
domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
information.
.. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- The Twisted dependency is no longer restricted to < 23.8.0. (:gh:`6024`,
:gh:`6064`, :gh:`6142`)
Bug fixes
~~~~~~~~~
- The OS signal handling code was refactored to no longer use private Twisted
functions. (:gh:`6024`, :gh:`6064`, :gh:`6112`)
Documentation
~~~~~~~~~~~~~
- Improved documentation for :class:`~scrapy.crawler.Crawler` initialization
changes made in the 2.11.0 release. (:gh:`6057`, :gh:`6147`)
- Extended documentation for :attr:`.Request.meta`.
(:gh:`5565`)
- Fixed the :reqmeta:`dont_merge_cookies` documentation. (:gh:`5936`,
:gh:`6077`)
- Added a link to Zyte's export guides to the :ref:`feed exports
` documentation. (:gh:`6183`)
- Added a missing note about backward-incompatible changes in
:class:`~scrapy.exporters.PythonItemExporter` to the 2.11.0 release notes.
(:gh:`6060`, :gh:`6081`)
- Added a missing note about removing the deprecated
``scrapy.utils.boto.is_botocore()`` function to the 2.8.0 release notes.
(:gh:`6056`, :gh:`6061`)
- Other documentation improvements. (:gh:`6128`, :gh:`6144`,
:gh:`6163`, :gh:`6190`, :gh:`6192`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Added Python 3.12 to the CI configuration, re-enabled tests that were
disabled when the pre-release support was added. (:gh:`5985`,
:gh:`6083`, :gh:`6098`)
- Fixed a test issue on PyPy 7.3.14. (:gh:`6204`, :gh:`6205`)
.. _release-2.11.0:
Scrapy 2.11.0 (2023-09-18)
--------------------------
Highlights:
- Spiders can now modify :ref:`settings ` in their
:meth:`~scrapy.Spider.from_crawler` methods, e.g. based on :ref:`spider
arguments `.
- Periodic logging of stats.
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Most of the initialization of :class:`scrapy.crawler.Crawler` instances is
now done in :meth:`~scrapy.crawler.Crawler.crawl`, so the state of
instances before that method is called is now different compared to older
Scrapy versions. We do not recommend using the
:class:`~scrapy.crawler.Crawler` instances before
:meth:`~scrapy.crawler.Crawler.crawl` is called. (:gh:`6038`)
- :meth:`scrapy.Spider.from_crawler` is now called before the initialization
of various components previously initialized in
:meth:`scrapy.crawler.Crawler.__init__` and before the settings are
finalized and frozen. This change was needed to allow changing the settings
in :meth:`scrapy.Spider.from_crawler`. If you want to access the final
setting values and the initialized :class:`~scrapy.crawler.Crawler`
attributes in the spider code as early as possible you can do this in
``scrapy.Spider.start_requests()`` or in a handler of the
:signal:`engine_started` signal. (:gh:`6038`)
- The :meth:`TextResponse.json ` method now
requires the response to be in a valid JSON encoding (UTF-8, UTF-16, or
UTF-32). If you need to deal with JSON documents in an invalid encoding,
use ``json.loads(response.text)`` instead. (:gh:`6016`)
- :class:`~scrapy.exporters.PythonItemExporter` used the binary output by
default but it no longer does. (:gh:`6006`, :gh:`6007`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- Removed the binary export mode of
:class:`~scrapy.exporters.PythonItemExporter`, deprecated in Scrapy 1.1.0.
(:gh:`6006`, :gh:`6007`)
.. note:: If you are using this Scrapy version on Scrapy Cloud with a stack
that includes an older Scrapy version and get a "TypeError:
Unexpected options: binary" error, you may need to add
``scrapinghub-entrypoint-scrapy >= 0.14.1`` to your project
requirements or switch to a stack that includes Scrapy 2.11.
- Removed the ``CrawlerRunner.spiders`` attribute, deprecated in Scrapy
1.0.0, use :attr:`CrawlerRunner.spider_loader
` instead. (:gh:`6010`)
- The :func:`scrapy.utils.response.response_httprepr` function, deprecated in
Scrapy 2.6.0, has now been removed. (:gh:`6111`)
Deprecations
~~~~~~~~~~~~
- Running :meth:`~scrapy.crawler.Crawler.crawl` more than once on the same
:class:`scrapy.crawler.Crawler` instance is now deprecated. (:gh:`1587`,
:gh:`6040`)
New features
~~~~~~~~~~~~
- Spiders can now modify settings in their
:meth:`~scrapy.Spider.from_crawler` method, e.g. based on :ref:`spider
arguments `. (:gh:`1305`, :gh:`1580`, :gh:`2392`,
:gh:`3663`, :gh:`6038`)
- Added the :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension
which can be enabled to log stats and/or their differences periodically.
(:gh:`5926`)
- Optimized the memory usage in :meth:`TextResponse.json
` by removing unnecessary body decoding.
(:gh:`5968`, :gh:`6016`)
- Links to ``.webp`` files are now ignored by :ref:`link extractors
`. (:gh:`6021`)
Bug fixes
~~~~~~~~~
- Fixed logging enabled add-ons. (:gh:`6036`)
- Fixed :class:`~scrapy.mail.MailSender` producing invalid message bodies
when the ``charset`` argument is passed to
:meth:`~scrapy.mail.MailSender.send`. (:gh:`5096`, :gh:`5118`)
- Fixed an exception when accessing ``self.EXCEPTIONS_TO_RETRY`` from a
subclass of :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`.
(:gh:`6049`, :gh:`6050`)
- :meth:`scrapy.settings.BaseSettings.getdictorlist`, used to parse
:setting:`FEED_EXPORT_FIELDS`, now handles tuple values. (:gh:`6011`,
:gh:`6013`)
- Calls to ``datetime.utcnow()``, no longer recommended to be used, have been
replaced with calls to ``datetime.now()`` with a timezone. (:gh:`6014`)
Documentation
~~~~~~~~~~~~~
- Updated a deprecated function call in a pipeline example. (:gh:`6008`,
:gh:`6009`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Extended typing hints. (:gh:`6003`, :gh:`6005`, :gh:`6031`,
:gh:`6034`)
- Pinned brotli_ to 1.0.9 for the PyPy tests as 1.1.0 breaks them.
(:gh:`6044`, :gh:`6045`)
- Other CI and pre-commit improvements. (:gh:`6002`, :gh:`6013`,
:gh:`6046`)
.. _release-2.10.1:
Scrapy 2.10.1 (2023-08-30)
--------------------------
Marked ``Twisted >= 23.8.0`` as unsupported. (:gh:`6024`, :gh:`6026`)
.. _release-2.10.0:
Scrapy 2.10.0 (2023-08-04)
--------------------------
Highlights:
- Added Python 3.12 support, dropped Python 3.7 support.
- The new add-ons framework simplifies configuring 3rd-party components that
support it.
- Exceptions to retry can now be configured.
- Many fixes and improvements for feed exports.
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- Dropped support for Python 3.7. (:gh:`5953`)
- Added support for the upcoming Python 3.12. (:gh:`5984`)
- Minimum versions increased for these dependencies:
- lxml_: 4.3.0 → 4.4.1
- cryptography_: 3.4.6 → 36.0.0
- ``pkg_resources`` is no longer used. (:gh:`5956`, :gh:`5958`)
- boto3_ is now recommended instead of botocore_ for exporting to S3.
(:gh:`5833`).
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The value of the :setting:`FEED_STORE_EMPTY` setting is now ``True``
instead of ``False``. In earlier Scrapy versions empty files were created
even when this setting was ``False`` (which was a bug that is now fixed),
so the new default should keep the old behavior. (:gh:`872`,
:gh:`5847`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting,
returning ``None`` or modifying the ``params`` input parameter, deprecated
in Scrapy 2.6, is no longer supported. (:gh:`5994`, :gh:`5996`)
- The ``scrapy.utils.reqser`` module, deprecated in Scrapy 2.6, is removed.
(:gh:`5994`, :gh:`5996`)
- The ``scrapy.squeues`` classes ``PickleFifoDiskQueueNonRequest``,
``PickleLifoDiskQueueNonRequest``, ``MarshalFifoDiskQueueNonRequest``,
and ``MarshalLifoDiskQueueNonRequest``, deprecated in
Scrapy 2.6, are removed. (:gh:`5994`, :gh:`5996`)
- The property ``open_spiders`` and the methods ``has_capacity`` and
``schedule`` of :class:`scrapy.core.engine.ExecutionEngine`,
deprecated in Scrapy 2.6, are removed. (:gh:`5994`, :gh:`5998`)
- Passing a ``spider`` argument to the
:meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`,
:meth:`~scrapy.core.engine.ExecutionEngine.crawl` and
:meth:`~scrapy.core.engine.ExecutionEngine.download` methods of
:class:`scrapy.core.engine.ExecutionEngine`, deprecated in Scrapy 2.6, is
no longer supported. (:gh:`5994`, :gh:`5998`)
Deprecations
~~~~~~~~~~~~
- :class:`scrapy.utils.datatypes.CaselessDict` is deprecated, use
:class:`scrapy.utils.datatypes.CaseInsensitiveDict` instead.
(:gh:`5146`)
- Passing the ``custom`` argument to
:func:`scrapy.utils.conf.build_component_list` is deprecated, it was used
in the past to merge ``FOO`` and ``FOO_BASE`` setting values but now Scrapy
uses :func:`scrapy.settings.BaseSettings.getwithbase` to do the same.
Code that uses this argument and cannot be switched to ``getwithbase()``
can be switched to merging the values explicitly. (:gh:`5726`,
:gh:`5923`)
New features
~~~~~~~~~~~~
- Added support for :ref:`Scrapy add-ons `. (:gh:`5950`)
- Added the :setting:`RETRY_EXCEPTIONS` setting that configures which
exceptions will be retried by
:class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`.
(:gh:`2701`, :gh:`5929`)
- Added the possiiblity to close the spider if no items were produced in the
specified time, configured by :setting:`CLOSESPIDER_TIMEOUT_NO_ITEM`.
(:gh:`5979`)
- Added support for the :setting:`AWS_REGION_NAME` setting to feed exports.
(:gh:`5980`)
- Added support for using :class:`pathlib.Path` objects that refer to
absolute Windows paths in the :setting:`FEEDS` setting. (:gh:`5939`)
Bug fixes
~~~~~~~~~
- Fixed creating empty feeds even with ``FEED_STORE_EMPTY=False``.
(:gh:`872`, :gh:`5847`)
- Fixed using absolute Windows paths when specifying output files.
(:gh:`5969`, :gh:`5971`)
- Fixed problems with uploading large files to S3 by switching to multipart
uploads (requires boto3_). (:gh:`960`, :gh:`5735`, :gh:`5833`)
- Fixed the JSON exporter writing extra commas when some exceptions occur.
(:gh:`3090`, :gh:`5952`)
- Fixed the "read of closed file" error in the CSV exporter. (:gh:`5043`,
:gh:`5705`)
- Fixed an error when a component added by the class object throws
:exc:`~scrapy.exceptions.NotConfigured` with a message. (:gh:`5950`,
:gh:`5992`)
- Added the missing :meth:`scrapy.settings.BaseSettings.pop` method.
(:gh:`5959`, :gh:`5960`, :gh:`5963`)
- Added :class:`~scrapy.utils.datatypes.CaseInsensitiveDict` as a replacement
for :class:`~scrapy.utils.datatypes.CaselessDict` that fixes some API
inconsistencies. (:gh:`5146`)
Documentation
~~~~~~~~~~~~~
- Documented :meth:`scrapy.Spider.update_settings`. (:gh:`5745`,
:gh:`5846`)
- Documented possible problems with early Twisted reactor installation and
their solutions. (:gh:`5981`, :gh:`6000`)
- Added examples of making additional requests in callbacks. (:gh:`5927`)
- Improved the feed export docs. (:gh:`5579`, :gh:`5931`)
- Clarified the docs about request objects on redirection. (:gh:`5707`,
:gh:`5937`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Added support for running tests against the installed Scrapy version.
(:gh:`4914`, :gh:`5949`)
- Extended typing hints. (:gh:`5925`, :gh:`5977`)
- Fixed the ``test_utils_asyncio.AsyncioTest.test_set_asyncio_event_loop``
test. (:gh:`5951`)
- Fixed the ``test_feedexport.BatchDeliveriesTest.test_batch_path_differ``
test on Windows. (:gh:`5847`)
- Enabled CI runs for Python 3.11 on Windows. (:gh:`5999`)
- Simplified skipping tests that depend on ``uvloop``. (:gh:`5984`)
- Fixed the ``extra-deps-pinned`` tox env. (:gh:`5948`)
- Implemented cleanups. (:gh:`5965`, :gh:`5986`)
.. _release-2.9.0:
Scrapy 2.9.0 (2023-05-08)
-------------------------
Highlights:
- Per-domain download settings.
- Compatibility with new cryptography_ and new parsel_.
- JMESPath selectors from the new parsel_.
- Bug fixes.
Deprecations
~~~~~~~~~~~~
- :class:`scrapy.extensions.feedexport._FeedSlot` is renamed to
:class:`scrapy.extensions.feedexport.FeedSlot` and the old name is
deprecated. (:gh:`5876`)
New features
~~~~~~~~~~~~
- Settings corresponding to :setting:`DOWNLOAD_DELAY`,
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and
:setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per-domain basis
via the new :setting:`DOWNLOAD_SLOTS` setting. (:gh:`5328`)
- Added :meth:`.TextResponse.jmespath`, a shortcut for JMESPath selectors
available since parsel_ 1.8.1. (:gh:`5894`, :gh:`5915`)
- Added :signal:`feed_slot_closed` and :signal:`feed_exporter_closed`
signals. (:gh:`5876`)
- Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a
curl command from a :class:`~scrapy.Request` object. (:gh:`5892`)
- Values of :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can now be
:class:`pathlib.Path` instances. (:gh:`5801`)
Bug fixes
~~~~~~~~~
- Fixed a warning with Parsel 1.8.1+. (:gh:`5903`, :gh:`5918`)
- Fixed an error when using feed postprocessing with S3 storage.
(:gh:`5500`, :gh:`5581`)
- Added the missing :meth:`scrapy.settings.BaseSettings.setdefault` method.
(:gh:`5811`, :gh:`5821`)
- Fixed an error when using cryptography_ 40.0.0+ and
:setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` is enabled.
(:gh:`5857`, :gh:`5858`)
- The checksums returned by :class:`~scrapy.pipelines.files.FilesPipeline`
for files on Google Cloud Storage are no longer Base64-encoded.
(:gh:`5874`, :gh:`5891`)
- :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed
string values for the curl ``--data-raw`` argument, which are produced by
browsers for data that includes certain symbols. (:gh:`5899`,
:gh:`5901`)
- The :command:`parse` command now also works with async generator callbacks.
(:gh:`5819`, :gh:`5824`)
- The :command:`genspider` command now properly works with HTTPS URLs.
(:gh:`3553`, :gh:`5808`)
- Improved handling of asyncio loops. (:gh:`5831`, :gh:`5832`)
- :class:`LinkExtractor `
now skips certain malformed URLs instead of raising an exception.
(:gh:`5881`)
- :func:`scrapy.utils.python.get_func_args` now supports more types of
callables. (:gh:`5872`, :gh:`5885`)
- Fixed an error when processing non-UTF8 values of ``Content-Type`` headers.
(:gh:`5914`, :gh:`5917`)
- Fixed an error breaking user handling of send failures in
:meth:`scrapy.mail.MailSender.send`. (:gh:`1611`, :gh:`5880`)
Documentation
~~~~~~~~~~~~~
- Expanded contributing docs. (:gh:`5109`, :gh:`5851`)
- Added blacken-docs_ to pre-commit and reformatted the docs with it.
(:gh:`5813`, :gh:`5816`)
- Fixed a JS issue. (:gh:`5875`, :gh:`5877`)
- Fixed ``make htmlview``. (:gh:`5878`, :gh:`5879`)
- Fixed typos and other small errors. (:gh:`5827`, :gh:`5839`,
:gh:`5883`, :gh:`5890`, :gh:`5895`, :gh:`5904`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Extended typing hints. (:gh:`5805`, :gh:`5889`, :gh:`5896`)
- Tests for most of the examples in the docs are now run as a part of CI,
found problems were fixed. (:gh:`5816`, :gh:`5826`, :gh:`5919`)
- Removed usage of deprecated Python classes. (:gh:`5849`)
- Silenced ``include-ignored`` warnings from coverage. (:gh:`5820`)
- Fixed a random failure of the ``test_feedexport.test_batch_path_differ``
test. (:gh:`5855`, :gh:`5898`)
- Updated docstrings to match output produced by parsel_ 1.8.1 so that they
don't cause test failures. (:gh:`5902`, :gh:`5919`)
- Other CI and pre-commit improvements. (:gh:`5802`, :gh:`5823`,
:gh:`5908`)
.. _blacken-docs: https://github.com/adamchainz/blacken-docs
.. _release-2.8.0:
Scrapy 2.8.0 (2023-02-02)
-------------------------
This is a maintenance release, with minor features, bug fixes, and cleanups.
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now
been removed. Use the :meth:`~io.BufferedIOBase.read1` method of
:class:`~gzip.GzipFile` instead.
(:gh:`5719`)
- The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy
2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode`
instead.
(:gh:`5719`)
- The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy
2.0, has now been removed. Use
:meth:`~scrapy.utils.python.MutableChain.__next__` instead.
(:gh:`5719`)
- The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated
in Scrapy 2.0, has now been removed. Use
:class:`LinkExtractor `
instead.
(:gh:`5720`)
- Support for using environment variables prefixed with ``SCRAPY_`` to
override settings, deprecated in Scrapy 2.0, has now been removed.
(:gh:`5724`)
- Support for the ``noconnect`` query string argument in proxy URLs,
deprecated in Scrapy 2.0, has now been removed. We expect proxies that used
to need it to work fine without it.
(:gh:`5731`)
- The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy
2.3, has now been removed.
(:gh:`5719`)
- The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4,
has now been removed.
(:gh:`5719`)
- The ``scrapy.utils.boto.is_botocore()`` function, deprecated in Scrapy 2.4,
has now been removed.
(:gh:`5719`)
Deprecations
~~~~~~~~~~~~
- :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated.
(:gh:`5368`, :gh:`5489`)
- :meth:`ImagesPipeline.convert_image
` must now accept a
``response_body`` parameter.
(:gh:`3055`, :gh:`3689`, :gh:`4753`)
New features
~~~~~~~~~~~~
- Applied black_ coding style to files generated with the
:command:`genspider` and :command:`startproject` commands.
(:gh:`5809`, :gh:`5814`)
.. _black: https://black.readthedocs.io/en/stable/
- :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the
``settings.py`` file that the :command:`startproject` command generates.
With this value, JSON exports won’t force the use of escape sequences for
non-ASCII characters.
(:gh:`5797`, :gh:`5800`)
- The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the
peak memory usage during checks, and the binary unit MiB is now used to
avoid confusion.
(:gh:`5717`, :gh:`5722`, :gh:`5727`)
- The ``callback`` parameter of :class:`~scrapy.Request` can now be set
to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from
``None``, as the latter indicates that the default spider callback
(:meth:`~scrapy.Spider.parse`) is to be used.
(:gh:`5798`)
Bug fixes
~~~~~~~~~
- Enabled unsafe legacy SSL renegotiation to fix access to some outdated
websites.
(:gh:`5491`, :gh:`5790`)
- Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and
better.
(:gh:`5386`, :gh:`5406`)
- Fixed the :meth:`finish_exporting` method of :ref:`item exporters
` not being called for empty files.
(:gh:`5537`, :gh:`5758`)
- Fixed HTTP/2 responses getting only the last value for a header when
multiple headers with the same name are received.
(:gh:`5777`)
- Fixed an exception raised by the :command:`shell` command on some cases
when :ref:`using asyncio `.
(:gh:`5740`, :gh:`5742`, :gh:`5748`, :gh:`5759`, :gh:`5760`,
:gh:`5771`)
- When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments
(``cb_kwargs``) added to a request in the ``process_request`` callback of a
:class:`~scrapy.spiders.Rule` will no longer be ignored.
(:gh:`5699`)
- The :ref:`images pipeline ` no longer re-encodes JPEG
files.
(:gh:`3055`, :gh:`3689`, :gh:`4753`)
- Fixed the handling of transparent WebP images by the :ref:`images pipeline
`.
(:gh:`3072`, :gh:`5766`, :gh:`5767`)
- :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT``
(Ctrl+C).
(:gh:`2918`)
- :class:`LinkExtractor `
with ``unique=False`` no longer filters out links that have identical URL
*and* text.
(:gh:`3798`, :gh:`3799`, :gh:`4695`, :gh:`5458`)
- :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
ignores URL protocols that do not support ``robots.txt`` (``data://``,
``file://``).
(:gh:`5807`)
- Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6.
(:gh:`5753`, :gh:`5754`)
- Fixed the output of ``scrapy -h`` showing an unintended ``**commands**``
line.
(:gh:`5709`, :gh:`5711`, :gh:`5712`)
- Made the active project indication in the output of :ref:`commands
` more clear.
(:gh:`5715`)
Documentation
~~~~~~~~~~~~~
- Documented how to :ref:`debug spiders from Visual Studio Code
`.
(:gh:`5721`)
- Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency.
(:gh:`5083`, :gh:`5540`)
- Improved consistency.
(:gh:`5761`)
- Fixed typos.
(:gh:`5714`, :gh:`5744`, :gh:`5764`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Applied :ref:`black coding style `, sorted import statements,
and introduced :ref:`pre-commit `.
(:gh:`4654`, :gh:`4658`, :gh:`5734`, :gh:`5737`, :gh:`5806`,
:gh:`5810`)
- Switched from :mod:`os.path` to :mod:`pathlib`.
(:gh:`4916`, :gh:`4497`, :gh:`5682`)
- Addressed many issues reported by Pylint.
(:gh:`5677`)
- Improved code readability.
(:gh:`5736`)
- Improved package metadata.
(:gh:`5768`)
- Removed direct invocations of ``setup.py``.
(:gh:`5774`, :gh:`5776`)
- Removed unnecessary :class:`~collections.OrderedDict` usages.
(:gh:`5795`)
- Removed unnecessary ``__str__`` definitions.
(:gh:`5150`)
- Removed obsolete code and comments.
(:gh:`5725`, :gh:`5729`, :gh:`5730`, :gh:`5732`)
- Fixed test and CI issues.
(:gh:`5749`, :gh:`5750`, :gh:`5756`, :gh:`5762`, :gh:`5765`,
:gh:`5780`, :gh:`5781`, :gh:`5782`, :gh:`5783`, :gh:`5785`,
:gh:`5786`)
.. _release-2.7.1:
Scrapy 2.7.1 (2022-11-02)
-------------------------
New features
~~~~~~~~~~~~
- Relaxed the restriction introduced in 2.6.2 so that the
``Proxy-Authorization`` header can again be set explicitly, as long as the
proxy URL in the :reqmeta:`proxy` metadata has no other credentials, and
for as long as that proxy URL remains the same; this restores compatibility
with scrapy-zyte-smartproxy 2.1.0 and older (:gh:`5626`).
Bug fixes
~~~~~~~~~
- Using ``-O``/``--overwrite-output`` and ``-t``/``--output-format`` options
together now produces an error instead of ignoring the former option
(:gh:`5516`, :gh:`5605`).
- Replaced deprecated :mod:`asyncio` APIs that implicitly use the current
event loop with code that explicitly requests a loop from the event loop
policy (:gh:`5685`, :gh:`5689`).
- Fixed uses of deprecated Scrapy APIs in Scrapy itself (:gh:`5588`,
:gh:`5589`).
- Fixed uses of a deprecated Pillow API (:gh:`5684`, :gh:`5692`).
- Improved code that checks if generators return values, so that it no longer
fails on decorated methods and partial methods (:gh:`5323`,
:gh:`5592`, :gh:`5599`, :gh:`5691`).
Documentation
~~~~~~~~~~~~~
- Upgraded the Code of Conduct to Contributor Covenant v2.1 (:gh:`5698`).
- Fixed typos (:gh:`5681`, :gh:`5694`).
Quality assurance
~~~~~~~~~~~~~~~~~
- Re-enabled some erroneously disabled flake8 checks (:gh:`5688`).
- Ignored harmless deprecation warnings from :mod:`typing` in tests
(:gh:`5686`, :gh:`5697`).
- Modernized our CI configuration (:gh:`5695`, :gh:`5696`).
.. _release-2.7.0:
Scrapy 2.7.0 (2022-10-17)
-----------------------------
Highlights:
- Added Python 3.11 support, dropped Python 3.6 support
- Improved support for :ref:`asynchronous callbacks `
- :ref:`Asyncio support ` is enabled by default on new
projects
- Output names of item fields can now be arbitrary strings
- Centralized :ref:`request fingerprinting `
configuration is now possible
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
Python 3.7 or greater is now required; support for Python 3.6 has been dropped.
Support for the upcoming Python 3.11 has been added.
The minimum required version of some dependencies has changed as well:
- lxml_: 3.5.0 → 4.3.0
- Pillow_ (:ref:`images pipeline `): 4.0.0 → 7.1.0
- zope.interface_: 5.0.0 → 5.1.0
(:gh:`5512`, :gh:`5514`, :gh:`5524`, :gh:`5563`, :gh:`5664`,
:gh:`5670`, :gh:`5678`)
Deprecations
~~~~~~~~~~~~
- :meth:`ImagesPipeline.thumb_path
` must now accept an
``item`` parameter (:gh:`5504`, :gh:`5508`).
- The ``scrapy.downloadermiddlewares.decompression`` module is now
deprecated (:gh:`5546`, :gh:`5547`).
New features
~~~~~~~~~~~~
- The
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
method of :ref:`spider middlewares ` can now be
defined as an :term:`asynchronous generator` (:gh:`4978`).
- The output of :class:`~scrapy.Request` callbacks defined as
:ref:`coroutines ` is now processed asynchronously
(:gh:`4978`).
- :class:`~scrapy.spiders.crawl.CrawlSpider` now supports :ref:`asynchronous
callbacks ` (:gh:`5657`).
- New projects created with the :command:`startproject` command have
:ref:`asyncio support ` enabled by default (:gh:`5590`,
:gh:`5679`).
- The :setting:`FEED_EXPORT_FIELDS` setting can now be defined as a
dictionary to customize the output name of item fields, lifting the
restriction that required output names to be valid Python identifiers, e.g.
preventing them to have whitespace (:gh:`1008`, :gh:`3266`,
:gh:`3696`).
- You can now customize :ref:`request fingerprinting `
through the new :setting:`REQUEST_FINGERPRINTER_CLASS` setting, instead of
having to change it on every Scrapy component that relies on request
fingerprinting (:gh:`900`, :gh:`3420`, :gh:`4113`, :gh:`4762`,
:gh:`4524`).
- ``jsonl`` is now supported and encouraged as a file extension for `JSON
Lines`_ files (:gh:`4848`).
.. _JSON Lines: https://jsonlines.org/
- :meth:`ImagesPipeline.thumb_path
` now receives the
source :ref:`item ` (:gh:`5504`, :gh:`5508`).
Bug fixes
~~~~~~~~~
- When using Google Cloud Storage with a :ref:`media pipeline
`, :setting:`FILES_EXPIRES` now also works when
:setting:`FILES_STORE` does not point at the root of your Google Cloud
Storage bucket (:gh:`5317`, :gh:`5318`).
- The :command:`parse` command now supports :ref:`asynchronous callbacks
` (:gh:`5424`, :gh:`5577`).
- When using the :command:`parse` command with a URL for which there is no
available spider, an exception is no longer raised (:gh:`3264`,
:gh:`3265`, :gh:`5375`, :gh:`5376`, :gh:`5497`).
- :class:`~scrapy.http.TextResponse` now gives higher priority to the `byte
order mark`_ when determining the text encoding of the response body,
following the `HTML living standard`_ (:gh:`5601`, :gh:`5611`).
.. _byte order mark: https://en.wikipedia.org/wiki/Byte_order_mark
.. _HTML living standard: https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding
- MIME sniffing takes the response body into account in FTP and HTTP/1.0
requests, as well as in cached requests (:gh:`4873`).
- MIME sniffing now detects valid HTML 5 documents even if the ``html`` tag
is missing (:gh:`4873`).
- An exception is now raised if :setting:`ASYNCIO_EVENT_LOOP` has a value
that does not match the asyncio event loop actually installed
(:gh:`5529`).
- Fixed :meth:`Headers.getlist() `
returning only the last header (:gh:`5515`, :gh:`5526`).
- Fixed :class:`LinkExtractor
` not ignoring the
``tar.gz`` file extension by default (:gh:`1837`, :gh:`2067`,
:gh:`4066`)
Documentation
~~~~~~~~~~~~~
- Clarified the return type of :meth:`Spider.parse `
(:gh:`5602`, :gh:`5608`).
- To enable
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
to do `brotli compression`_, installing brotli_ is now recommended instead
of installing brotlipy_, as the former provides a more recent version of
brotli.
.. _brotli: https://github.com/google/brotli
.. _brotli compression: https://www.ietf.org/rfc/rfc7932.txt
- :ref:`Signal documentation ` now mentions :ref:`coroutine
support ` and uses it in code examples (:gh:`4852`,
:gh:`5358`).
- :ref:`bans` now recommends `Common Crawl`_ instead of `Google cache`_
(:gh:`3582`, :gh:`5432`).
.. _Common Crawl: https://commoncrawl.org/
.. _Google cache: https://www.googleguide.com/cached_pages.html
- The new :ref:`topics-components` topic covers enforcing requirements on
Scrapy components, like :ref:`downloader middlewares
`, :ref:`extensions `,
:ref:`item pipelines `, :ref:`spider middlewares
`, and more; :ref:`enforce-asyncio-requirement`
has also been added (:gh:`4978`).
- :ref:`topics-settings` now indicates that setting values must be
:ref:`picklable ` (:gh:`5607`, :gh:`5629`).
- Removed outdated documentation (:gh:`5446`, :gh:`5373`,
:gh:`5369`, :gh:`5370`, :gh:`5554`).
- Fixed typos (:gh:`5442`, :gh:`5455`, :gh:`5457`, :gh:`5461`,
:gh:`5538`, :gh:`5553`, :gh:`5558`, :gh:`5624`, :gh:`5631`).
- Fixed other issues (:gh:`5283`, :gh:`5284`, :gh:`5559`,
:gh:`5567`, :gh:`5648`, :gh:`5659`, :gh:`5665`).
Quality assurance
~~~~~~~~~~~~~~~~~
- Added a continuous integration job to run `twine check`_ (:gh:`5655`,
:gh:`5656`).
.. _twine check: https://twine.readthedocs.io/en/stable/#twine-check
- Addressed test issues and warnings (:gh:`5560`, :gh:`5561`,
:gh:`5612`, :gh:`5617`, :gh:`5639`, :gh:`5645`, :gh:`5662`,
:gh:`5671`, :gh:`5675`).
- Cleaned up code (:gh:`4991`, :gh:`4995`, :gh:`5451`,
:gh:`5487`, :gh:`5542`, :gh:`5667`, :gh:`5668`, :gh:`5672`).
- Applied minor code improvements (:gh:`5661`).
.. _release-2.6.3:
Scrapy 2.6.3 (2022-09-27)
-------------------------
- Added support for pyOpenSSL_ 22.1.0, removing support for SSLv3
(:gh:`5634`, :gh:`5635`, :gh:`5636`).
- Upgraded the minimum versions of the following dependencies:
- cryptography_: 2.0 → 3.3
- pyOpenSSL_: 16.2.0 → 21.0.0
- service_identity_: 16.0.0 → 18.1.0
- Twisted_: 17.9.0 → 18.9.0
- zope.interface_: 4.1.3 → 5.0.0
(:gh:`5621`, :gh:`5632`)
- Fixes test and documentation issues (:gh:`5612`, :gh:`5617`,
:gh:`5631`).
.. _release-2.6.2:
Scrapy 2.6.2 (2022-07-25)
-------------------------
**Security bug fix:**
- When :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
processes a request with :reqmeta:`proxy` metadata, and that
:reqmeta:`proxy` metadata includes proxy credentials,
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` sets
the ``Proxy-Authorization`` header, but only if that header is not already
set.
There are third-party proxy-rotation downloader middlewares that set
different :reqmeta:`proxy` metadata every time they process a request.
Because of request retries and redirects, the same request can be processed
by downloader middlewares more than once, including both
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` and
any third-party proxy-rotation downloader middleware.
These third-party proxy-rotation downloader middlewares could change the
:reqmeta:`proxy` metadata of a request to a new value, but fail to remove
the ``Proxy-Authorization`` header from the previous value of the
:reqmeta:`proxy` metadata, causing the credentials of one proxy to be sent
to a different proxy.
To prevent the unintended leaking of proxy credentials, the behavior of
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` is now
as follows when processing a request:
- If the request being processed defines :reqmeta:`proxy` metadata that
includes credentials, the ``Proxy-Authorization`` header is always
updated to feature those credentials.
- If the request being processed defines :reqmeta:`proxy` metadata
without credentials, the ``Proxy-Authorization`` header is removed
*unless* it was originally defined for the same proxy URL.
To remove proxy credentials while keeping the same proxy URL, remove
the ``Proxy-Authorization`` header.
- If the request has no :reqmeta:`proxy` metadata, or that metadata is a
falsy value (e.g. ``None``), the ``Proxy-Authorization`` header is
removed.
It is no longer possible to set a proxy URL through the
:reqmeta:`proxy` metadata but set the credentials through the
``Proxy-Authorization`` header. Set proxy credentials through the
:reqmeta:`proxy` metadata instead.
Also fixes the following regressions introduced in 2.6.0:
- :class:`~scrapy.crawler.CrawlerProcess` supports again crawling multiple
spiders (:gh:`5435`, :gh:`5436`)
- Installing a Twisted reactor before Scrapy does (e.g. importing
:mod:`twisted.internet.reactor` somewhere at the module level) no longer
prevents Scrapy from starting, as long as a different reactor is not
specified in :setting:`TWISTED_REACTOR` (:gh:`5525`, :gh:`5528`)
- Fixed an exception that was being logged after the spider finished under
certain conditions (:gh:`5437`, :gh:`5440`)
- The ``--output``/``-o`` command-line parameter supports again a value
starting with a hyphen (:gh:`5444`, :gh:`5445`)
- The ``scrapy parse -h`` command no longer throws an error (:gh:`5481`,
:gh:`5482`)
.. _release-2.6.1:
Scrapy 2.6.1 (2022-03-01)
-------------------------
Fixes a regression introduced in 2.6.0 that would unset the request method when
following redirects.
.. _release-2.6.0:
Scrapy 2.6.0 (2022-03-01)
-------------------------
Highlights:
* :ref:`Security fixes for cookie handling <2.6-security-fixes>`
* Python 3.10 support
* :ref:`asyncio support ` is no longer considered
experimental, and works out-of-the-box on Windows regardless of your Python
version
* Feed exports now support :class:`pathlib.Path` output paths and per-feed
:ref:`item filtering ` and
:ref:`post-processing `
.. _2.6-security-fixes:
Security bug fixes
~~~~~~~~~~~~~~~~~~
- When a :class:`~scrapy.Request` object with cookies defined gets a
redirect response causing a new :class:`~scrapy.Request` object to be
scheduled, the cookies defined in the original
:class:`~scrapy.Request` object are no longer copied into the new
:class:`~scrapy.Request` object.
If you manually set the ``Cookie`` header on a
:class:`~scrapy.Request` object and the domain name of the redirect
URL is not an exact match for the domain of the URL of the original
:class:`~scrapy.Request` object, your ``Cookie`` header is now dropped
from the new :class:`~scrapy.Request` object.
The old behavior could be exploited by an attacker to gain access to your
cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more
information.
.. _cjvr-mfj7-j4j8 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cjvr-mfj7-j4j8
.. note:: It is still possible to enable the sharing of cookies between
different domains with a shared domain suffix (e.g.
``example.com`` and any subdomain) by defining the shared domain
suffix (e.g. ``example.com``) as the cookie domain when defining
your cookies. See the documentation of the
:class:`~scrapy.Request` class for more information.
- When the domain of a cookie, either received in the ``Set-Cookie`` header
of a response or defined in a :class:`~scrapy.Request` object, is set
to a `public suffix `_, the cookie is now
ignored unless the cookie domain is the same as the request domain.
The old behavior could be exploited by an attacker to inject cookies from a
controlled domain into your cookiejar that could be sent to other domains
not controlled by the attacker. Please, see the `mfjm-vh54-3f96 security
advisory`_ for more information.
.. _mfjm-vh54-3f96 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-mfjm-vh54-3f96
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
- The h2_ dependency is now optional, only needed to
:ref:`enable HTTP/2 support `. (:gh:`5113`)
.. _h2: https://pypi.org/project/h2/
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The ``formdata`` parameter of :class:`~scrapy.FormRequest`, if specified
for a non-POST request, now overrides the URL query string, instead of
being appended to it. (:gh:`2919`, :gh:`3579`)
- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting, now
the return value of that function, and not the ``params`` input parameter,
will determine the feed URI parameters, unless that return value is
``None``. (:gh:`4962`, :gh:`4966`)
- In :class:`scrapy.core.engine.ExecutionEngine`, methods
:meth:`~scrapy.core.engine.ExecutionEngine.crawl`,
:meth:`~scrapy.core.engine.ExecutionEngine.download`,
:meth:`~scrapy.core.engine.ExecutionEngine.schedule`,
and :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`
now raise :exc:`RuntimeError` if called before
:meth:`~scrapy.core.engine.ExecutionEngine.open_spider`. (:gh:`5090`)
These methods used to assume that
:attr:`ExecutionEngine.slot ` had
been defined by a prior call to
:meth:`~scrapy.core.engine.ExecutionEngine.open_spider`, so they were
raising :exc:`AttributeError` instead.
- If the API of the configured :ref:`scheduler ` does not
meet expectations, :exc:`TypeError` is now raised at startup time. Before,
other exceptions would be raised at run time. (:gh:`3559`)
- The ``_encoding`` field of serialized :class:`~scrapy.Request` objects
is now named ``encoding``, in line with all other fields (:gh:`5130`)
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- ``scrapy.http.TextResponse.body_as_unicode``, deprecated in Scrapy 2.2, has
now been removed. (:gh:`5393`)
- ``scrapy.item.BaseItem``, deprecated in Scrapy 2.2, has now been removed.
(:gh:`5398`)
- ``scrapy.item.DictItem``, deprecated in Scrapy 1.8, has now been removed.
(:gh:`5398`)
- ``scrapy.Spider.make_requests_from_url``, deprecated in Scrapy 1.4, has now
been removed. (:gh:`4178`, :gh:`4356`)
Deprecations
~~~~~~~~~~~~
- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting,
returning ``None`` or modifying the ``params`` input parameter is now
deprecated. Return a new dictionary instead. (:gh:`4962`, :gh:`4966`)
- :mod:`scrapy.utils.reqser` is deprecated. (:gh:`5130`)
- Instead of :func:`~scrapy.utils.reqser.request_to_dict`, use the new
:meth:`.Request.to_dict` method.
- Instead of :func:`~scrapy.utils.reqser.request_from_dict`, use the new
:func:`scrapy.utils.request.request_from_dict` function.
- In :mod:`scrapy.squeues`, the following queue classes are deprecated:
:class:`~scrapy.squeues.PickleFifoDiskQueueNonRequest`,
:class:`~scrapy.squeues.PickleLifoDiskQueueNonRequest`,
:class:`~scrapy.squeues.MarshalFifoDiskQueueNonRequest`,
and :class:`~scrapy.squeues.MarshalLifoDiskQueueNonRequest`. You should
instead use:
:class:`~scrapy.squeues.PickleFifoDiskQueue`,
:class:`~scrapy.squeues.PickleLifoDiskQueue`,
:class:`~scrapy.squeues.MarshalFifoDiskQueue`,
and :class:`~scrapy.squeues.MarshalLifoDiskQueue`. (:gh:`5117`)
- Many aspects of :class:`scrapy.core.engine.ExecutionEngine` that come from
a time when this class could handle multiple :class:`~scrapy.Spider`
objects at a time have been deprecated. (:gh:`5090`)
- The :meth:`~scrapy.core.engine.ExecutionEngine.has_capacity` method
is deprecated.
- The :meth:`~scrapy.core.engine.ExecutionEngine.schedule` method is
deprecated, use :meth:`~scrapy.core.engine.ExecutionEngine.crawl` or
:meth:`~scrapy.core.engine.ExecutionEngine.download` instead.
- The :attr:`~scrapy.core.engine.ExecutionEngine.open_spiders` attribute
is deprecated, use :attr:`~scrapy.core.engine.ExecutionEngine.spider`
instead.
- The ``spider`` parameter is deprecated for the following methods:
- :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`
- :meth:`~scrapy.core.engine.ExecutionEngine.crawl`
- :meth:`~scrapy.core.engine.ExecutionEngine.download`
Instead, call :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`
first to set the :class:`~scrapy.Spider` object.
- :func:`scrapy.utils.response.response_httprepr` is now deprecated.
(:gh:`4972`)
New features
~~~~~~~~~~~~
- You can now use :ref:`item filtering ` to control which items
are exported to each output feed. (:gh:`4575`, :gh:`5178`,
:gh:`5161`, :gh:`5203`)
- You can now apply :ref:`post-processing ` to feeds, and
:ref:`built-in post-processing plugins ` are provided for
output file compression. (:gh:`2174`, :gh:`5168`, :gh:`5190`)
- The :setting:`FEEDS` setting now supports :class:`pathlib.Path` objects as
keys. (:gh:`5383`, :gh:`5384`)
- Enabling :ref:`asyncio ` while using Windows and Python 3.8
or later will automatically switch the asyncio event loop to one that
allows Scrapy to work. See :ref:`asyncio-windows`. (:gh:`4976`,
:gh:`5315`)
- The :command:`genspider` command now supports a start URL instead of a
domain name. (:gh:`4439`)
- :mod:`scrapy.utils.defer` gained 2 new functions,
:func:`~scrapy.utils.defer.deferred_to_future` and
:func:`~scrapy.utils.defer.maybe_deferred_to_future`, to help :ref:`await
on Deferreds when using the asyncio reactor `.
(:gh:`5288`)
- :ref:`Amazon S3 feed export storage ` gained
support for `temporary security credentials`_
(:setting:`AWS_SESSION_TOKEN`) and endpoint customization
(:setting:`AWS_ENDPOINT_URL`). (:gh:`4998`, :gh:`5210`)
.. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
- New :setting:`LOG_FILE_APPEND` setting to allow truncating the log file.
(:gh:`5279`)
- :attr:`Request.cookies ` values that are
:class:`bool`, :class:`float` or :class:`int` are cast to :class:`str`.
(:gh:`5252`, :gh:`5253`)
- You may now raise :exc:`~scrapy.exceptions.CloseSpider` from a handler of
the :signal:`spider_idle` signal to customize the reason why the spider is
stopping. (:gh:`5191`)
- When using
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`, the
proxy URL for non-HTTPS HTTP/1.1 requests no longer needs to include a URL
scheme. (:gh:`4505`, :gh:`4649`)
- All built-in queues now expose a ``peek`` method that returns the next
queue object (like ``pop``) but does not remove the returned object from
the queue. (:gh:`5112`)
If the underlying queue does not support peeking (e.g. because you are not
using ``queuelib`` 1.6.1 or later), the ``peek`` method raises
:exc:`NotImplementedError`.
- :class:`~scrapy.Request` and :class:`~scrapy.http.Response` now have
an ``attributes`` attribute that makes subclassing easier. For
:class:`~scrapy.Request`, it also allows subclasses to work with
:func:`scrapy.utils.request.request_from_dict`. (:gh:`1877`,
:gh:`5130`, :gh:`5218`)
- The :meth:`~scrapy.core.scheduler.BaseScheduler.open` and
:meth:`~scrapy.core.scheduler.BaseScheduler.close` methods of the
:ref:`scheduler ` are now optional. (:gh:`3559`)
- HTTP/1.1 :exc:`~scrapy.core.downloader.handlers.http11.TunnelError`
exceptions now only truncate response bodies longer than 1000 characters,
instead of those longer than 32 characters, making it easier to debug such
errors. (:gh:`4881`, :gh:`5007`)
- :class:`~scrapy.loader.ItemLoader` now supports non-text responses.
(:gh:`5145`, :gh:`5269`)
Bug fixes
~~~~~~~~~
- The :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` settings
are no longer ignored if defined in :attr:`~scrapy.Spider.custom_settings`.
(:gh:`4485`, :gh:`5352`)
- Removed a module-level Twisted reactor import that could prevent
:ref:`using the asyncio reactor `. (:gh:`5357`)
- The :command:`startproject` command works with existing folders again.
(:gh:`4665`, :gh:`4676`)
- The :setting:`FEED_URI_PARAMS` setting now behaves as documented.
(:gh:`4962`, :gh:`4966`)
- :attr:`Request.cb_kwargs ` once again allows the
``callback`` keyword. (:gh:`5237`, :gh:`5251`, :gh:`5264`)
- Made :func:`scrapy.utils.response.open_in_browser` support more complex
HTML. (:gh:`5319`, :gh:`5320`)
- Fixed :attr:`CSVFeedSpider.quotechar
` being interpreted as the CSV file
encoding. (:gh:`5391`, :gh:`5394`)
- Added missing setuptools_ to the list of dependencies. (:gh:`5122`)
.. _setuptools: https://pypi.org/project/setuptools/
- :class:`LinkExtractor `
now also works as expected with links that have comma-separated ``rel``
attribute values including ``nofollow``. (:gh:`5225`)
- Fixed a :exc:`TypeError` that could be raised during :ref:`feed export
` parameter parsing. (:gh:`5359`)
Documentation
~~~~~~~~~~~~~
- :ref:`asyncio support ` is no longer considered
experimental. (:gh:`5332`)
- Included :ref:`Windows-specific help for asyncio usage `.
(:gh:`4976`, :gh:`5315`)
- Rewrote :ref:`topics-headless-browsing` with up-to-date best practices.
(:gh:`4484`, :gh:`4613`)
- Documented :ref:`local file naming in media pipelines
`. (:gh:`5069`, :gh:`5152`)
- :ref:`faq` now covers spider file name collision issues. (:gh:`2680`,
:gh:`3669`)
- Provided better context and instructions to disable the
:setting:`URLLENGTH_LIMIT` setting. (:gh:`5135`, :gh:`5250`)
- Documented that Reppy parser does not support Python 3.9+.
(:gh:`5226`, :gh:`5231`)
- Documented :ref:`the scheduler component `.
(:gh:`3537`, :gh:`3559`)
- Documented the method used by :ref:`media pipelines
` to :ref:`determine if a file has expired
`. (:gh:`5120`, :gh:`5254`)
- :ref:`run-multiple-spiders` now features
:func:`scrapy.utils.project.get_project_settings` usage. (:gh:`5070`)
- :ref:`run-multiple-spiders` now covers what happens when you define
different per-spider values for some settings that cannot differ at run
time. (:gh:`4485`, :gh:`5352`)
- Extended the documentation of the
:class:`~scrapy.extensions.statsmailer.StatsMailer` extension.
(:gh:`5199`, :gh:`5217`)
- Added :setting:`JOBDIR` to :ref:`topics-settings`. (:gh:`5173`,
:gh:`5224`)
- Documented :attr:`Spider.attribute `.
(:gh:`5174`, :gh:`5244`)
- Documented :attr:`TextResponse.urljoin `.
(:gh:`1582`)
- Added the ``body_length`` parameter to the documented signature of the
:signal:`headers_received` signal. (:gh:`5270`)
- Clarified :meth:`SelectorList.get ` usage
in the :ref:`tutorial `. (:gh:`5256`)
- The documentation now features the shortest import path of classes with
multiple import paths. (:gh:`2733`, :gh:`5099`)
- ``quotes.toscrape.com`` references now use HTTPS instead of HTTP.
(:gh:`5395`, :gh:`5396`)
- Added a link to `our Discord server `_
to :ref:`getting-help`. (:gh:`5421`, :gh:`5422`)
- The pronunciation of the project name is now :ref:`officially
` /ˈskreɪpaɪ/. (:gh:`5280`, :gh:`5281`)
- Added the Scrapy logo to the README. (:gh:`5255`, :gh:`5258`)
- Fixed issues and implemented minor improvements. (:gh:`3155`,
:gh:`4335`, :gh:`5074`, :gh:`5098`, :gh:`5134`, :gh:`5180`,
:gh:`5194`, :gh:`5239`, :gh:`5266`, :gh:`5271`, :gh:`5273`,
:gh:`5274`, :gh:`5276`, :gh:`5347`, :gh:`5356`, :gh:`5414`,
:gh:`5415`, :gh:`5416`, :gh:`5419`, :gh:`5420`)
Quality Assurance
~~~~~~~~~~~~~~~~~
- Added support for Python 3.10. (:gh:`5212`, :gh:`5221`,
:gh:`5265`)
- Significantly reduced memory usage by
:func:`scrapy.utils.response.response_httprepr`, used by the
:class:`~scrapy.downloadermiddlewares.stats.DownloaderStats` downloader
middleware, which is enabled by default. (:gh:`4964`, :gh:`4972`)
- Removed uses of the deprecated :mod:`optparse` module. (:gh:`5366`,
:gh:`5374`)
- Extended typing hints. (:gh:`5077`, :gh:`5090`, :gh:`5100`,
:gh:`5108`, :gh:`5171`, :gh:`5215`, :gh:`5334`)
- Improved tests, fixed CI issues, removed unused code. (:gh:`5094`,
:gh:`5157`, :gh:`5162`, :gh:`5198`, :gh:`5207`, :gh:`5208`,
:gh:`5229`, :gh:`5298`, :gh:`5299`, :gh:`5310`, :gh:`5316`,
:gh:`5333`, :gh:`5388`, :gh:`5389`, :gh:`5400`, :gh:`5401`,
:gh:`5404`, :gh:`5405`, :gh:`5407`, :gh:`5410`, :gh:`5412`,
:gh:`5425`, :gh:`5427`)
- Implemented improvements for contributors. (:gh:`5080`, :gh:`5082`,
:gh:`5177`, :gh:`5200`)
- Implemented cleanups. (:gh:`5095`, :gh:`5106`, :gh:`5209`,
:gh:`5228`, :gh:`5235`, :gh:`5245`, :gh:`5246`, :gh:`5292`,
:gh:`5314`, :gh:`5322`)
.. _release-2.5.1:
Scrapy 2.5.1 (2021-10-05)
-------------------------
* **Security bug fix:**
If you use
:class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
(i.e. the ``http_user`` and ``http_pass`` spider attributes) for HTTP
authentication, any request exposes your credentials to the request target.
To prevent unintended exposure of authentication credentials to unintended
domains, you must now additionally set a new, additional spider attribute,
``http_auth_domain``, and point it to the specific domain to which the
authentication credentials must be sent.
If the ``http_auth_domain`` spider attribute is not set, the domain of the
first request will be considered the HTTP authentication target, and
authentication credentials will only be sent in requests targeting that
domain.
If you need to send the same HTTP authentication credentials to multiple
domains, you can use :func:`w3lib.http.basic_auth_header` instead to
set the value of the ``Authorization`` header of your requests.
If you *really* want your spider to send the same HTTP authentication
credentials to any domain, set the ``http_auth_domain`` spider attribute
to ``None``.
Finally, if you are a user of `scrapy-splash`_, know that this version of
Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will
need to upgrade scrapy-splash to a greater version for it to continue to
work.
.. _release-2.5.0:
Scrapy 2.5.0 (2021-04-06)
-------------------------
Highlights:
- Official Python 3.9 support
- Experimental :ref:`HTTP/2 support `
- New :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function
to retry requests from spider callbacks
- New :class:`~scrapy.signals.headers_received` signal that allows stopping
downloads early
- New :class:`Response.protocol ` attribute
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- Removed all code that :ref:`was deprecated in 1.7.0 <1.7-deprecations>` and
had not :ref:`already been removed in 2.4.0 <2.4-deprecation-removals>`.
(:gh:`4901`)
- Removed support for the ``SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE`` environment
variable, :ref:`deprecated in 1.8.0 <1.8-deprecations>`. (:gh:`4912`)
Deprecations
~~~~~~~~~~~~
- The :mod:`scrapy.utils.py36` module is now deprecated in favor of
:mod:`scrapy.utils.asyncgen`. (:gh:`4900`)
New features
~~~~~~~~~~~~
- Experimental :ref:`HTTP/2 support ` through a new download handler
that can be assigned to the ``https`` protocol in the
:setting:`DOWNLOAD_HANDLERS` setting.
(:gh:`1854`, :gh:`4769`, :gh:`5058`, :gh:`5059`, :gh:`5066`)
- The new :func:`scrapy.downloadermiddlewares.retry.get_retry_request`
function may be used from spider callbacks or middlewares to handle the
retrying of a request beyond the scenarios that
:class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` supports.
(:gh:`3590`, :gh:`3685`, :gh:`4902`)
- The new :class:`~scrapy.signals.headers_received` signal gives early access
to response headers and allows :ref:`stopping downloads
`.
(:gh:`1772`, :gh:`4897`)
- The new :attr:`Response.protocol `
attribute gives access to the string that identifies the protocol used to
download a response. (:gh:`4878`)
- :ref:`Stats ` now include the following entries that indicate
the number of successes and failures in storing
:ref:`feeds `::
feedexport/success_count/
feedexport/failed_count/
Where ```` is the feed storage backend class name, such as
:class:`~scrapy.extensions.feedexport.FileFeedStorage` or
:class:`~scrapy.extensions.feedexport.FTPFeedStorage`.
(:gh:`3947`, :gh:`4850`)
- The :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` spider
middleware now logs ignored URLs with ``INFO`` :ref:`logging level
` instead of ``DEBUG``, and it now includes the following entry
into :ref:`stats ` to keep track of the number of ignored
URLs::
urllength/request_ignored_count
(:gh:`5036`)
- The
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
downloader middleware now logs the number of decompressed responses and the
total count of resulting bytes::
httpcompression/response_bytes
httpcompression/response_count
(:gh:`4797`, :gh:`4799`)
Bug fixes
~~~~~~~~~
- Fixed installation on PyPy installing PyDispatcher in addition to
PyPyDispatcher, which could prevent Scrapy from working depending on which
package got imported. (:gh:`4710`, :gh:`4814`)
- When inspecting a callback to check if it is a generator that also returns
a value, an exception is no longer raised if the callback has a docstring
with lower indentation than the following code.
(:gh:`4477`, :gh:`4935`)
- The `Content-Length `_
header is no longer omitted from responses when using the default, HTTP/1.1
download handler (see :setting:`DOWNLOAD_HANDLERS`).
(:gh:`5009`, :gh:`5034`, :gh:`5045`, :gh:`5057`, :gh:`5062`)
- Setting the :reqmeta:`handle_httpstatus_all` request meta key to ``False``
now has the same effect as not setting it at all, instead of having the
same effect as setting it to ``True``.
(:gh:`3851`, :gh:`4694`)
Documentation
~~~~~~~~~~~~~
- Added instructions to :ref:`install Scrapy in Windows using pip
`.
(:gh:`4715`, :gh:`4736`)
- Logging documentation now includes :ref:`additional ways to filter logs
`.
(:gh:`4216`, :gh:`4257`, :gh:`4965`)
- Covered how to deal with long lists of allowed domains in the :ref:`FAQ
`. (:gh:`2263`, :gh:`3667`)
- Covered scrapy-bench_ in :ref:`benchmarking`.
(:gh:`4996`, :gh:`5016`)
- Clarified that one :ref:`extension ` instance is created
per crawler.
(:gh:`5014`)
- Fixed some errors in examples.
(:gh:`4829`, :gh:`4830`, :gh:`4907`, :gh:`4909`,
:gh:`5008`)
- Fixed some external links, typos, and so on.
(:gh:`4892`, :gh:`4899`, :gh:`4936`, :gh:`4942`, :gh:`5005`,
:gh:`5063`)
- The :ref:`list of Request.meta keys ` is now sorted
alphabetically.
(:gh:`5061`, :gh:`5065`)
- Updated references to Scrapinghub, which is now called Zyte.
(:gh:`4973`, :gh:`5072`)
- Added a mention to contributors in the README. (:gh:`4956`)
- Reduced the top margin of lists. (:gh:`4974`)
Quality Assurance
~~~~~~~~~~~~~~~~~
- Made Python 3.9 support official (:gh:`4757`, :gh:`4759`)
- Extended typing hints (:gh:`4895`)
- Fixed deprecated uses of the Twisted API.
(:gh:`4940`, :gh:`4950`, :gh:`5073`)
- Made our tests run with the new pip resolver.
(:gh:`4710`, :gh:`4814`)
- Added tests to ensure that :ref:`coroutine support `
is tested. (:gh:`4987`)
- Migrated from Travis CI to GitHub Actions. (:gh:`4924`)
- Fixed CI issues.
(:gh:`4986`, :gh:`5020`, :gh:`5022`, :gh:`5027`, :gh:`5052`,
:gh:`5053`)
- Implemented code refactorings, style fixes and cleanups.
(:gh:`4911`, :gh:`4982`, :gh:`5001`, :gh:`5002`, :gh:`5076`)
.. _release-2.4.1:
Scrapy 2.4.1 (2020-11-17)
-------------------------
- Fixed :ref:`feed exports ` overwrite support (:gh:`4845`, :gh:`4857`, :gh:`4859`)
- Fixed the AsyncIO event loop handling, which could make code hang
(:gh:`4855`, :gh:`4872`)
- Fixed the IPv6-capable DNS resolver
:class:`~scrapy.resolver.CachingHostnameResolver` for download handlers
that call
:meth:`reactor.resolve `
(:gh:`4802`, :gh:`4803`)
- Fixed the output of the :command:`genspider` command showing placeholders
instead of the import path of the generated spider module (:gh:`4874`)
- Migrated Windows CI from Azure Pipelines to GitHub Actions (:gh:`4869`,
:gh:`4876`)
.. _release-2.4.0:
Scrapy 2.4.0 (2020-10-11)
-------------------------
Highlights:
* Python 3.5 support has been dropped.
* The ``file_path`` method of :ref:`media pipelines `
can now access the source :ref:`item `.
This allows you to set a download file path based on item data.
* The new ``item_export_kwargs`` key of the :setting:`FEEDS` setting allows
to define keyword parameters to pass to :ref:`item exporter classes
`
* You can now choose whether :ref:`feed exports `
overwrite or append to the output file.
For example, when using the :command:`crawl` or :command:`runspider`
commands, you can use the ``-O`` option instead of ``-o`` to overwrite the
output file.
* Zstd-compressed responses are now supported if zstandard_ is installed.
* In settings, where the import path of a class is required, it is now
possible to pass a class object instead.
Modified requirements
~~~~~~~~~~~~~~~~~~~~~
* Python 3.6 or greater is now required; support for Python 3.5 has been
dropped
As a result:
- When using PyPy, PyPy 7.2.0 or greater :ref:`is now required
`
- For Amazon S3 storage support in :ref:`feed exports