diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 754bd6f9c..311df7052 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -6,7 +6,7 @@ exclude: | ) repos: - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.15.2 + rev: v0.15.20 hooks: - id: ruff-check args: [ --fix ] @@ -16,7 +16,7 @@ repos: hooks: - id: blacken-docs additional_dependencies: - - black==25.9.0 + - black==26.5.1 - repo: https://github.com/pre-commit/pre-commit-hooks rev: v6.0.0 hooks: diff --git a/SECURITY.md b/SECURITY.md index 49cafdf9e..c1a5482c6 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -4,8 +4,8 @@ | Version | Supported | | ------- | ------------------ | -| 2.16.x | :white_check_mark: | -| < 2.16.x | :x: | +| 2.17.x | :white_check_mark: | +| < 2.17.x | :x: | ## Reporting a Vulnerability diff --git a/docs/faq.rst b/docs/faq.rst index 2f5ddb51f..a396a8212 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -411,7 +411,6 @@ How can I make a blank request? from scrapy import Request - blank_request = Request("data:,") In this case, the URL is set to a data URI scheme. Data URLs allow you to include data diff --git a/docs/news.rst b/docs/news.rst index a4137996a..63f0d69e4 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,330 @@ Release notes ============= +.. _release-2.17.0: + +Scrapy 2.17.0 (2026-07-07) +-------------------------- + +Highlights: + +- Security bug fixes + +- HTTP/2 and SOCKS proxy support for ``HttpxDownloadHandler`` + +- Improved settings for changing allowed TLS versions + +Security bug fixes +~~~~~~~~~~~~~~~~~~ + +- ``s3://`` requests now use HTTPS by default, instead of plaintext HTTP. + + Previously, :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler` + sent signed S3 requests over plaintext HTTP unless + ``request.meta["is_secure"]`` was set to a true value, exposing the request + path, the AWS ``Authorization`` header, the ``X-Amz-Security-Token`` header + (when using temporary credentials), and the response contents to network + attackers, who could also tamper with responses. See the `76g3-c3x4-crvx`_ + security advisory for details. + + To restore the previous behavior for a given request, set + ``request.meta["is_secure"]`` to ``False``. + + .. _76g3-c3x4-crvx: https://github.com/scrapy/scrapy/security/advisories/GHSA-76g3-c3x4-crvx + +Deprecations +~~~~~~~~~~~~ + +- The ``DOWNLOADER_CLIENT_TLS_METHOD`` setting is deprecated. You should use + the :setting:`DOWNLOAD_TLS_MIN_VERSION` and/or + :setting:`DOWNLOAD_TLS_MAX_VERSION` settings instead if you want to change + the TLS method selection. + (:issue:`3288`, :issue:`6546`) + +- The following spider attributes are deprecated in favor of settings: + + - ``http_user`` (use :setting:`HTTPAUTH_USER`) + + - ``http_pass`` (use :setting:`HTTPAUTH_PASS`) + + - ``http_auth_domain`` (use :setting:`HTTPAUTH_DOMAIN`) + + (:issue:`7590`) + +- The ``scrapy.commands.ScrapyCommand.help()`` method is deprecated. It was + never called by Scrapy. + (:issue:`7626`, :issue:`7633`) + +- The following TLS-related functions and constants, intended for internal + use, are deprecated: + + - ``scrapy.core.downloader.tls.METHOD_TLS`` + + - ``scrapy.core.downloader.tls.METHOD_TLSv10`` + + - ``scrapy.core.downloader.tls.METHOD_TLSv11`` + + - ``scrapy.core.downloader.tls.METHOD_TLSv12`` + + - ``scrapy.core.downloader.tls.openssl_methods`` + + - ``scrapy.core.downloader.tls.DEFAULT_CIPHERS`` + + - ``scrapy.utils.ssl.ffi_buf_to_string()`` + + - ``scrapy.utils.ssl.get_temp_key_info()`` + + - ``scrapy.utils.ssl.x509name_to_string()`` + + (:issue:`6546`, :issue:`7619`, :issue:`7665`) + +- The ``CRAWLSPIDER_FOLLOW_LINKS`` setting is deprecated. You can set + ``follow=False`` in your rules to achieve the same effect. + (:issue:`7592`) + +- Instantiating + :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware` + without a ``crawler`` argument is deprecated. + (:issue:`7655`) + +- Instantiating + :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` without a + ``settings`` argument is deprecated. + (:issue:`7664`) + +New features +~~~~~~~~~~~~ + +- Added support for HTTP/2 requests to + :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. It + requires setting the new :setting:`HTTPX_HTTP2_ENABLED` setting to + ``True``. + (:issue:`7575`) + +- Added support for SOCKS proxies to + :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. + (:issue:`747`, :issue:`7575`) + +- Added :setting:`DOWNLOAD_TLS_MIN_VERSION` and + :setting:`DOWNLOAD_TLS_MAX_VERSION` settings as replacements for the + ``DOWNLOADER_CLIENT_TLS_METHOD`` setting (which is now deprecated). + Compared to the old setting, they support specifying a range of allowed + versions and support newer TLS versions. + (:issue:`4821`, :issue:`6546`) + +- Added :setting:`HTTPAUTH_USER`, :setting:`HTTPAUTH_PASS` and + :setting:`HTTPAUTH_DOMAIN` settings and :reqmeta:`http_user`, + :reqmeta:`http_pass` and :reqmeta:`http_auth_domain` meta keys as more + flexible ways to set HTTP authentication data. + (:issue:`7590`) + +- Added a :reqmeta:`verbatim_url` meta key that can be set to ``True`` to + skip request URL canonicalization. + (:issue:`7473`) + +- Added ``deny_tags`` and ``deny_attrs`` arguments to :class:`LinkExtractor + `. + (:issue:`6321`, :issue:`7679`) + +- :attr:`scrapy.Item.fields` now returns the fields in the definition order + instead of the alphabetical one. + (:issue:`7015`, :issue:`7694`) + +- Added a :setting:`RETRY_GIVE_UP_LOG_LEVEL` setting, a + :reqmeta:`give_up_log_level` meta key and a ``give_up_log_level`` argument + of the + :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function that + allow changing the log level of the message logged when the retry limit has + been reached. + (:issue:`4622`, :issue:`5297`, :issue:`7567`) + +- It's now possible to set :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` to + ``None`` to use the default ciphers of the underlying TLS implementation. + (:issue:`7499`, :issue:`7665`) + +Improvements +~~~~~~~~~~~~ + +- :class:`~scrapy.FormRequest` is no longer deprecated, only its + ``from_response()`` method is still deprecated. + (:issue:`7561`, :issue:`7671`) + +- Switched the item definition in the default project template from a + :class:`scrapy.item.Item` to a dataclass. + (:issue:`7493`, :issue:`7513`) + +- Fixed deprecation warnings with pyOpenSSL 26.3.0. + (:issue:`7619`) + +- Removed the runtime warnings for :attr:`Spider.allowed_domains + ` containing URLs or domains with ports + instead of just domains and for spider classes having a ``start_url`` + attribute instead of :class:`~scrapy.spiders.Spider.start_urls`. Please use + :doc:`scrapy-lint ` to find mistakes in your spider code + instead. + (:issue:`4421`, :issue:`7627`) + +- :func:`scrapy.utils.test.get_crawler` now disables + :setting:`TELNETCONSOLE_ENABLED` by default. + (:issue:`7644`) + +- Other code refactoring and improvements. + (:issue:`7409`, :issue:`7593`, :issue:`7594`, :issue:`7611`, :issue:`7649`) + +Bug fixes +~~~~~~~~~ + +- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no + longer ignores proxy credentials for redirected or retried requests. + (:issue:`7601`, :issue:`7630`) + +- :class:`~scrapy.extensions.feedexport.GCSFeedStorage` now closes the + temporary file after the upload. + (:issue:`7546`) + +- Fixed ``scrapy shell `` running a full spider crawl when there is a + spider for the requested URL. This bug was introduced in Scrapy 2.13.0. + (:issue:`7552`, :issue:`7557`) + +- The :setting:`IMAGES_STORE_S3_ACL` and :setting:`IMAGES_STORE_GCS_ACL` + settings are no longer ignored. This bug was introduced in Scrapy 2.12.0. + (:issue:`7597`, :issue:`7614`) + +- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now closes + the connection after making the request. + (:issue:`7602`, :issue:`7667`) + +- Removed the deprecated ``spider`` argument from the pipeline defined in the + default project template. + (:issue:`7676`) + +- Fixed ``scrapy genspider --edit`` not working. + (:issue:`7260`, :issue:`7683`) + +- When a :class:`~scrapy.crawler.Crawler` instance is passed to + :meth:`AsyncCrawlerRunner.create_crawler() + ` or + :meth:`CrawlerRunner.create_crawler() + `, settings from both classes + are now merged, previously only the settings from the + :class:`~scrapy.crawler.Crawler` instance were used. + (:issue:`1280`, :issue:`7647`) + +- Fixed several issues with cookie handling in + :func:`scrapy.utils.request.request_to_curl`. + (:issue:`7603`, :issue:`7675`, :issue:`7684`) + +- Fixed :class:`scrapy.resolver.CachingThreadedResolver` not disabling the + cache when :setting:`DNSCACHE_ENABLED` is set to ``False``. + (:issue:`7663`) + +- Fixed :func:`scrapy.utils.response.open_in_browser` not removing comments + when looking for the ```` tag. + (:issue:`7506`) + +- Fixed checking for deprecated methods in custom :setting:`ITEM_PROCESSOR` + implementations. + (:issue:`7589`) + +- Fixed :func:`scrapy.utils.url.strip_url` corrupting some URLs with + credentials. + (:issue:`7604`, :issue:`7605`) + +- :func:`scrapy.utils.misc.rel_has_nofollow` now ignores the case when + looking for "nofollow" strings. + (:issue:`7632`) + +- Fixed an exception in :class:`scrapy.utils.sitemap.Sitemap` when parsing + some malformed sitemaps. + (:issue:`7686`, :issue:`7687`) + +Documentation +~~~~~~~~~~~~~ + +- Mentioned :doc:`scrapy-lint ` in the docs. + (:issue:`4421`, :issue:`7627`) + +- Added the docs about :ref:`security considerations `. + (:issue:`7389`, :issue:`7678`) + +- Improved the :ref:`item pipeline docs `. + (:issue:`2350`, :issue:`7676`) + +- Documented which stats are collected by + :class:`~scrapy.extensions.corestats.CoreStats`. + (:issue:`7421`) + +- Switched documentation examples from using :class:`scrapy.item.Item` to + using dataclasses. + (:issue:`7493`, :issue:`7513`) + +- Added feature comparison tables to the :ref:`download handler + ` docs. + (:issue:`7575`) + +- Improved the docs for :ref:`logging settings `. + (:issue:`6909`, :issue:`7668`) + +- Documented a way to :ref:`improve startup time and memory usage + ` by using :setting:`SPIDER_MODULES`. + (:issue:`7576`, :issue:`7600`) + +- Clarified handling of the ``type`` argument of :class:`~scrapy.Selector`. + (:issue:`7704`) + +- Other documentation improvements and fixes. + (:issue:`4954`, + :issue:`6120`, + :issue:`7286`, + :issue:`7564`, + :issue:`7573`, + :issue:`7598`, + :issue:`7599`, + :issue:`7698`) + +Quality assurance +~~~~~~~~~~~~~~~~~ + +- Fixed deprecation warnings with pytest 9.1.0. + (:issue:`7621`) + +- Type hints improvements and fixes. + (:issue:`6958`, :issue:`7586`) + +- CI and test improvements and fixes. + (:issue:`5954`, + :issue:`7002`, + :issue:`7017`, + :issue:`7247`, + :issue:`7508`, + :issue:`7545`, + :issue:`7566`, + :issue:`7574`, + :issue:`7585`, + :issue:`7595`, + :issue:`7608`, + :issue:`7610`, + :issue:`7612`, + :issue:`7616`, + :issue:`7625`, + :issue:`7637`, + :issue:`7639`, + :issue:`7640`, + :issue:`7641`, + :issue:`7642`, + :issue:`7643`, + :issue:`7644`, + :issue:`7645`, + :issue:`7646`, + :issue:`7654`, + :issue:`7655`, + :issue:`7664`, + :issue:`7672`, + :issue:`7677`, + :issue:`7680`, + :issue:`7682`, + :issue:`7692`) + .. _release-2.16.0: Scrapy 2.16.0 (2026-05-19) @@ -6243,8 +6567,8 @@ Backward-incompatible changes ``429``, you must override :setting:`RETRY_HTTP_CODES` accordingly. * :class:`~scrapy.crawler.Crawler`, - :class:`CrawlerRunner.crawl ` and - :class:`CrawlerRunner.create_crawler ` + :meth:`CrawlerRunner.crawl ` and + :meth:`CrawlerRunner.create_crawler ` no longer accept a :class:`~scrapy.spiders.Spider` subclass instance, they only accept a :class:`~scrapy.spiders.Spider` subclass now. diff --git a/docs/topics/addons.rst b/docs/topics/addons.rst index 01e4bcac1..78b1ab92b 100644 --- a/docs/topics/addons.rst +++ b/docs/topics/addons.rst @@ -168,7 +168,6 @@ Use a fallback component: from scrapy.utils.misc import build_from_crawler, load_object - FALLBACK_SETTING = "MY_FALLBACK_DOWNLOAD_HANDLER" diff --git a/docs/topics/autothrottle.rst b/docs/topics/autothrottle.rst index d0321c906..787ef842a 100644 --- a/docs/topics/autothrottle.rst +++ b/docs/topics/autothrottle.rst @@ -88,6 +88,8 @@ server) is, and this extension builds on that premise. Prevent specific requests from triggering slot delay adjustments ================================================================ +.. versionadded:: 2.12.0 + AutoThrottle adjusts the delay of download slots based on the latencies of responses that belong to that download slot. The only exceptions are non-200 responses, which are only taken into account to increase that delay, but diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index c901e01e4..9bbfee9e2 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -329,13 +329,13 @@ Other limitations: HTTPX_HTTP2_ENABLED ^^^^^^^^^^^^^^^^^^^ +.. versionadded:: 2.17.0 + Default: ``False`` Whether to enable HTTP/2 support in this handler. The ``httpx[http2]`` extra needs to be installed if you want to enable this setting. -.. versionadded:: VERSION - Built-in non-HTTP download handlers reference ============================================= diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 5ab9773b0..383f331b5 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -351,6 +351,8 @@ HttpAuthMiddleware HTTPAUTH_USER ~~~~~~~~~~~~~ +.. versionadded:: 2.17.0 + Default: ``""`` The username to use for HTTP basic authentication, applied to all requests @@ -361,6 +363,8 @@ whose URL matches :setting:`HTTPAUTH_DOMAIN`. HTTPAUTH_PASS ~~~~~~~~~~~~~ +.. versionadded:: 2.17.0 + Default: ``""`` The password to use for HTTP basic authentication. @@ -370,6 +374,8 @@ The password to use for HTTP basic authentication. HTTPAUTH_DOMAIN ~~~~~~~~~~~~~~~ +.. versionadded:: 2.17.0 + Default: ``None`` The domain (and its subdomains) to which HTTP basic authentication credentials @@ -1105,6 +1111,8 @@ exception propagation, see RETRY_GIVE_UP_LOG_LEVEL ^^^^^^^^^^^^^^^^^^^^^^^ +.. versionadded:: 2.17.0 + Default: ``"ERROR"`` :ref:`Logging level ` used for the message logged when a request diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index bd5663c13..db6dd127c 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -528,10 +528,6 @@ safe numeric encoding (``\uXXXX`` sequences) for historic reasons. Use ``"utf-8"`` if you want UTF-8 for JSON too. -.. versionchanged:: 2.8 - The :command:`startproject` command now sets this setting to - ``"utf-8"`` in the generated ``settings.py`` file. - .. setting:: FEED_EXPORT_FIELDS FEED_EXPORT_FIELDS diff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rst index 5af3f93c1..3fc896507 100644 --- a/docs/topics/link-extractors.rst +++ b/docs/topics/link-extractors.rst @@ -47,108 +47,7 @@ LxmlLinkExtractor :synopsis: lxml's HTMLParser-based link extractors -.. class:: LxmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), restrict_css=(), tags=('a', 'area'), attrs=('href',), canonicalize=False, unique=True, process_value=None, strip=True, restrict_text=None) - - LxmlLinkExtractor is the recommended link extractor with handy filtering - options. It is implemented using lxml's robust HTMLParser. - - :param allow: a single regular expression (or list of regular expressions) - that the (absolute) urls must match in order to be extracted. If not - given (or empty), it will match all links. - :type allow: str or list - - :param deny: a single regular expression (or list of regular expressions) - that the (absolute) urls must match in order to be excluded (i.e. not - extracted). It has precedence over the ``allow`` parameter. If not - given (or empty) it won't exclude any links. - :type deny: str or list - - :param allow_domains: a single value or a list of string containing - domains which will be considered for extracting the links - :type allow_domains: str or list - - :param deny_domains: a single value or a list of strings containing - domains which won't be considered for extracting the links - :type deny_domains: str or list - - :param deny_extensions: a single value or list of strings containing - extensions that should be ignored when extracting links. - If not given, it will default to - :data:`scrapy.linkextractors.IGNORED_EXTENSIONS`. - - :type deny_extensions: list - - :param restrict_xpaths: is an XPath (or list of XPath's) which defines - regions inside the response where links should be extracted from. - If given, only the text selected by those XPath will be scanned for - links. - :type restrict_xpaths: str or list - - :param restrict_css: a CSS selector (or list of selectors) which defines - regions inside the response where links should be extracted from. - Has the same behaviour as ``restrict_xpaths``. - :type restrict_css: str or list - - :param restrict_text: a single regular expression (or list of regular expressions) - that the link's text must match in order to be extracted. If not - given (or empty), it will match all links. If a list of regular expressions is - given, the link will be extracted if it matches at least one. - :type restrict_text: str or list - - :param tags: a tag or a list of tags to consider when extracting links. - Defaults to ``('a', 'area')``. - :type tags: str or list - - :param attrs: an attribute or list of attributes which should be considered when looking - for links to extract (only for those tags specified in the ``tags`` - parameter). Defaults to ``('href',)`` - :type attrs: list - - :param canonicalize: canonicalize each extracted url (using - w3lib.url.canonicalize_url). Defaults to ``False``. - Note that canonicalize_url is meant for duplicate checking; - it can change the URL visible at server side, so the response can be - different for requests with canonicalized and raw URLs. If you're - using LinkExtractor to follow links it is more robust to - keep the default ``canonicalize=False``. - :type canonicalize: bool - - :param unique: whether duplicate filtering should be applied to extracted - links. - :type unique: bool - - :param process_value: a function which receives each value extracted from - the tag and attributes scanned and can modify the value and return a - new one, or return ``None`` to ignore the link altogether. If not - given, ``process_value`` defaults to ``lambda x: x``. - - .. highlight:: html - - For example, to extract links from this code:: - - Link text - - .. highlight:: python - - You can use the following function in ``process_value``: - - .. code-block:: python - - def process_value(value): - m = re.search(r"javascript:goToPage\('(.*?)'", value) - if m: - return m.group(1) - - :type process_value: collections.abc.Callable - - :param strip: whether to strip whitespaces from extracted attributes. - According to HTML5 standard, leading and trailing whitespaces - must be stripped from ``href`` attributes of ````, ```` - and many other elements, ``src`` attribute of ````, ``