RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into assorted-docs-fixes-1

This commit is contained in:
Andrey Rakhmatullin 2026-07-07 18:13:08 +05:00
parent 81cff94b26 c5ec881f1d
commit 0499f9736a
22 changed files with 593 additions and 141 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.pre-commit-config.yaml
View File

@ -6,7 +6,7 @@ exclude: |
)
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.2
rev: v0.15.20
hooks:
- id: ruff-check
args: [ --fix ]
@ -16,7 +16,7 @@ repos:
hooks:
- id: blacken-docs
additional_dependencies:
- black==25.9.0
- black==26.5.1
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:

4
SECURITY.md
View File

@ -4,8 +4,8 @@
| Version | Supported |
| ------- | ------------------ |
| 2.16.x | :white_check_mark: |
| < 2.16.x | :x: |
| 2.17.x | :white_check_mark: |
| < 2.17.x | :x: |
## Reporting a Vulnerability

1
docs/faq.rst
View File

@ -411,7 +411,6 @@ How can I make a blank request?
from scrapy import Request
blank_request = Request("data:,")
In this case, the URL is set to a data URI scheme. Data URLs allow you to include data

328
docs/news.rst
View File

@ -3,6 +3,330 @@
Release notes
=============
.. _release-2.17.0:
Scrapy 2.17.0 (2026-07-07)
--------------------------
Highlights:
- Security bug fixes
- HTTP/2 and SOCKS proxy support for ``HttpxDownloadHandler``
- Improved settings for changing allowed TLS versions
Security bug fixes
~~~~~~~~~~~~~~~~~~
- ``s3://`` requests now use HTTPS by default, instead of plaintext HTTP.
Previously, :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler`
sent signed S3 requests over plaintext HTTP unless
``request.meta["is_secure"]`` was set to a true value, exposing the request
path, the AWS ``Authorization`` header, the ``X-Amz-Security-Token`` header
(when using temporary credentials), and the response contents to network
attackers, who could also tamper with responses. See the `76g3-c3x4-crvx`_
security advisory for details.
To restore the previous behavior for a given request, set
``request.meta["is_secure"]`` to ``False``.
.. _76g3-c3x4-crvx: https://github.com/scrapy/scrapy/security/advisories/GHSA-76g3-c3x4-crvx
Deprecations
~~~~~~~~~~~~
- The ``DOWNLOADER_CLIENT_TLS_METHOD`` setting is deprecated. You should use
the :setting:`DOWNLOAD_TLS_MIN_VERSION` and/or
:setting:`DOWNLOAD_TLS_MAX_VERSION` settings instead if you want to change
the TLS method selection.
(:issue:`3288`, :issue:`6546`)
- The following spider attributes are deprecated in favor of settings:
- ``http_user`` (use :setting:`HTTPAUTH_USER`)
- ``http_pass`` (use :setting:`HTTPAUTH_PASS`)
- ``http_auth_domain`` (use :setting:`HTTPAUTH_DOMAIN`)
(:issue:`7590`)
- The ``scrapy.commands.ScrapyCommand.help()`` method is deprecated. It was
never called by Scrapy.
(:issue:`7626`, :issue:`7633`)
- The following TLS-related functions and constants, intended for internal
use, are deprecated:
- ``scrapy.core.downloader.tls.METHOD_TLS``
- ``scrapy.core.downloader.tls.METHOD_TLSv10``
- ``scrapy.core.downloader.tls.METHOD_TLSv11``
- ``scrapy.core.downloader.tls.METHOD_TLSv12``
- ``scrapy.core.downloader.tls.openssl_methods``
- ``scrapy.core.downloader.tls.DEFAULT_CIPHERS``
- ``scrapy.utils.ssl.ffi_buf_to_string()``
- ``scrapy.utils.ssl.get_temp_key_info()``
- ``scrapy.utils.ssl.x509name_to_string()``
(:issue:`6546`, :issue:`7619`, :issue:`7665`)
- The ``CRAWLSPIDER_FOLLOW_LINKS`` setting is deprecated. You can set
``follow=False`` in your rules to achieve the same effect.
(:issue:`7592`)
- Instantiating
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
without a ``crawler`` argument is deprecated.
(:issue:`7655`)
- Instantiating
:class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` without a
``settings`` argument is deprecated.
(:issue:`7664`)
New features
~~~~~~~~~~~~
- Added support for HTTP/2 requests to
:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. It
requires setting the new :setting:`HTTPX_HTTP2_ENABLED` setting to
``True``.
(:issue:`7575`)
- Added support for SOCKS proxies to
:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
(:issue:`747`, :issue:`7575`)
- Added :setting:`DOWNLOAD_TLS_MIN_VERSION` and
:setting:`DOWNLOAD_TLS_MAX_VERSION` settings as replacements for the
``DOWNLOADER_CLIENT_TLS_METHOD`` setting (which is now deprecated).
Compared to the old setting, they support specifying a range of allowed
versions and support newer TLS versions.
(:issue:`4821`, :issue:`6546`)
- Added :setting:`HTTPAUTH_USER`, :setting:`HTTPAUTH_PASS` and
:setting:`HTTPAUTH_DOMAIN` settings and :reqmeta:`http_user`,
:reqmeta:`http_pass` and :reqmeta:`http_auth_domain` meta keys as more
flexible ways to set HTTP authentication data.
(:issue:`7590`)
- Added a :reqmeta:`verbatim_url` meta key that can be set to ``True`` to
skip request URL canonicalization.
(:issue:`7473`)
- Added ``deny_tags`` and ``deny_attrs`` arguments to :class:`LinkExtractor
<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`.
(:issue:`6321`, :issue:`7679`)
- :attr:`scrapy.Item.fields` now returns the fields in the definition order
instead of the alphabetical one.
(:issue:`7015`, :issue:`7694`)
- Added a :setting:`RETRY_GIVE_UP_LOG_LEVEL` setting, a
:reqmeta:`give_up_log_level` meta key and a ``give_up_log_level`` argument
of the
:func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function that
allow changing the log level of the message logged when the retry limit has
been reached.
(:issue:`4622`, :issue:`5297`, :issue:`7567`)
- It's now possible to set :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` to
``None`` to use the default ciphers of the underlying TLS implementation.
(:issue:`7499`, :issue:`7665`)
Improvements
~~~~~~~~~~~~
- :class:`~scrapy.FormRequest` is no longer deprecated, only its
``from_response()`` method is still deprecated.
(:issue:`7561`, :issue:`7671`)
- Switched the item definition in the default project template from a
:class:`scrapy.item.Item` to a dataclass.
(:issue:`7493`, :issue:`7513`)
- Fixed deprecation warnings with pyOpenSSL 26.3.0.
(:issue:`7619`)
- Removed the runtime warnings for :attr:`Spider.allowed_domains
<scrapy.Spider.allowed_domains>` containing URLs or domains with ports
instead of just domains and for spider classes having a ``start_url``
attribute instead of :class:`~scrapy.spiders.Spider.start_urls`. Please use
:doc:`scrapy-lint <scrapy-lint:index>` to find mistakes in your spider code
instead.
(:issue:`4421`, :issue:`7627`)
- :func:`scrapy.utils.test.get_crawler` now disables
:setting:`TELNETCONSOLE_ENABLED` by default.
(:issue:`7644`)
- Other code refactoring and improvements.
(:issue:`7409`, :issue:`7593`, :issue:`7594`, :issue:`7611`, :issue:`7649`)
Bug fixes
~~~~~~~~~
- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no
longer ignores proxy credentials for redirected or retried requests.
(:issue:`7601`, :issue:`7630`)
- :class:`~scrapy.extensions.feedexport.GCSFeedStorage` now closes the
temporary file after the upload.
(:issue:`7546`)
- Fixed ``scrapy shell <URL>`` running a full spider crawl when there is a
spider for the requested URL. This bug was introduced in Scrapy 2.13.0.
(:issue:`7552`, :issue:`7557`)
- The :setting:`IMAGES_STORE_S3_ACL` and :setting:`IMAGES_STORE_GCS_ACL`
settings are no longer ignored. This bug was introduced in Scrapy 2.12.0.
(:issue:`7597`, :issue:`7614`)
- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now closes
the connection after making the request.
(:issue:`7602`, :issue:`7667`)
- Removed the deprecated ``spider`` argument from the pipeline defined in the
default project template.
(:issue:`7676`)
- Fixed ``scrapy genspider --edit`` not working.
(:issue:`7260`, :issue:`7683`)
- When a :class:`~scrapy.crawler.Crawler` instance is passed to
:meth:`AsyncCrawlerRunner.create_crawler()
<scrapy.crawler.AsyncCrawlerRunner.create_crawler>` or
:meth:`CrawlerRunner.create_crawler()
<scrapy.crawler.CrawlerRunner.create_crawler>`, settings from both classes
are now merged, previously only the settings from the
:class:`~scrapy.crawler.Crawler` instance were used.
(:issue:`1280`, :issue:`7647`)
- Fixed several issues with cookie handling in
:func:`scrapy.utils.request.request_to_curl`.
(:issue:`7603`, :issue:`7675`, :issue:`7684`)
- Fixed :class:`scrapy.resolver.CachingThreadedResolver` not disabling the
cache when :setting:`DNSCACHE_ENABLED` is set to ``False``.
(:issue:`7663`)
- Fixed :func:`scrapy.utils.response.open_in_browser` not removing comments
when looking for the ``<base>`` tag.
(:issue:`7506`)
- Fixed checking for deprecated methods in custom :setting:`ITEM_PROCESSOR`
implementations.
(:issue:`7589`)
- Fixed :func:`scrapy.utils.url.strip_url` corrupting some URLs with
credentials.
(:issue:`7604`, :issue:`7605`)
- :func:`scrapy.utils.misc.rel_has_nofollow` now ignores the case when
looking for "nofollow" strings.
(:issue:`7632`)
- Fixed an exception in :class:`scrapy.utils.sitemap.Sitemap` when parsing
some malformed sitemaps.
(:issue:`7686`, :issue:`7687`)
Documentation
~~~~~~~~~~~~~
- Mentioned :doc:`scrapy-lint <scrapy-lint:index>` in the docs.
(:issue:`4421`, :issue:`7627`)
- Added the docs about :ref:`security considerations <security>`.
(:issue:`7389`, :issue:`7678`)
- Improved the :ref:`item pipeline docs <topics-item-pipeline>`.
(:issue:`2350`, :issue:`7676`)
- Documented which stats are collected by
:class:`~scrapy.extensions.corestats.CoreStats`.
(:issue:`7421`)
- Switched documentation examples from using :class:`scrapy.item.Item` to
using dataclasses.
(:issue:`7493`, :issue:`7513`)
- Added feature comparison tables to the :ref:`download handler
<download-handlers-ref>` docs.
(:issue:`7575`)
- Improved the docs for :ref:`logging settings <logging-settings>`.
(:issue:`6909`, :issue:`7668`)
- Documented a way to :ref:`improve startup time and memory usage
<large-project-startup>` by using :setting:`SPIDER_MODULES`.
(:issue:`7576`, :issue:`7600`)
- Clarified handling of the ``type`` argument of :class:`~scrapy.Selector`.
(:issue:`7704`)
- Other documentation improvements and fixes.
(:issue:`4954`,
:issue:`6120`,
:issue:`7286`,
:issue:`7564`,
:issue:`7573`,
:issue:`7598`,
:issue:`7599`,
:issue:`7698`)
Quality assurance
~~~~~~~~~~~~~~~~~
- Fixed deprecation warnings with pytest 9.1.0.
(:issue:`7621`)
- Type hints improvements and fixes.
(:issue:`6958`, :issue:`7586`)
- CI and test improvements and fixes.
(:issue:`5954`,
:issue:`7002`,
:issue:`7017`,
:issue:`7247`,
:issue:`7508`,
:issue:`7545`,
:issue:`7566`,
:issue:`7574`,
:issue:`7585`,
:issue:`7595`,
:issue:`7608`,
:issue:`7610`,
:issue:`7612`,
:issue:`7616`,
:issue:`7625`,
:issue:`7637`,
:issue:`7639`,
:issue:`7640`,
:issue:`7641`,
:issue:`7642`,
:issue:`7643`,
:issue:`7644`,
:issue:`7645`,
:issue:`7646`,
:issue:`7654`,
:issue:`7655`,
:issue:`7664`,
:issue:`7672`,
:issue:`7677`,
:issue:`7680`,
:issue:`7682`,
:issue:`7692`)
.. _release-2.16.0:
Scrapy 2.16.0 (2026-05-19)
@ -6243,8 +6567,8 @@ Backward-incompatible changes
``429``, you must override :setting:`RETRY_HTTP_CODES` accordingly.
* :class:`~scrapy.crawler.Crawler`,
:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
:class:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
:meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
:meth:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
no longer accept a :class:`~scrapy.spiders.Spider` subclass instance, they
only accept a :class:`~scrapy.spiders.Spider` subclass now.

1
docs/topics/addons.rst
View File

@ -168,7 +168,6 @@ Use a fallback component:
from scrapy.utils.misc import build_from_crawler, load_object
FALLBACK_SETTING = "MY_FALLBACK_DOWNLOAD_HANDLER"

2
docs/topics/autothrottle.rst
View File

@ -88,6 +88,8 @@ server) is, and this extension builds on that premise.
Prevent specific requests from triggering slot delay adjustments
================================================================
.. versionadded:: 2.12.0
AutoThrottle adjusts the delay of download slots based on the latencies of
responses that belong to that download slot. The only exceptions are non-200
responses, which are only taken into account to increase that delay, but

4
docs/topics/download-handlers.rst
View File

@ -329,13 +329,13 @@ Other limitations:
HTTPX_HTTP2_ENABLED
^^^^^^^^^^^^^^^^^^^
.. versionadded:: 2.17.0
Default: ``False``
Whether to enable HTTP/2 support in this handler. The ``httpx[http2]`` extra
needs to be installed if you want to enable this setting.
.. versionadded:: VERSION
Built-in non-HTTP download handlers reference
=============================================

8
docs/topics/downloader-middleware.rst
View File

@ -351,6 +351,8 @@ HttpAuthMiddleware
HTTPAUTH_USER
~~~~~~~~~~~~~
.. versionadded:: 2.17.0
Default: ``""``
The username to use for HTTP basic authentication, applied to all requests
@ -361,6 +363,8 @@ whose URL matches :setting:`HTTPAUTH_DOMAIN`.
HTTPAUTH_PASS
~~~~~~~~~~~~~
.. versionadded:: 2.17.0
Default: ``""``
The password to use for HTTP basic authentication.
@ -370,6 +374,8 @@ The password to use for HTTP basic authentication.
HTTPAUTH_DOMAIN
~~~~~~~~~~~~~~~
.. versionadded:: 2.17.0
Default: ``None``
The domain (and its subdomains) to which HTTP basic authentication credentials
@ -1105,6 +1111,8 @@ exception propagation, see
RETRY_GIVE_UP_LOG_LEVEL
^^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 2.17.0
Default: ``"ERROR"``
:ref:`Logging level <levels>` used for the message logged when a request

4
docs/topics/feed-exports.rst
View File

@ -528,10 +528,6 @@ safe numeric encoding (``\uXXXX`` sequences) for historic reasons.
Use ``"utf-8"`` if you want UTF-8 for JSON too.
.. versionchanged:: 2.8
The :command:`startproject` command now sets this setting to
``"utf-8"`` in the generated ``settings.py`` file.
.. setting:: FEED_EXPORT_FIELDS
FEED_EXPORT_FIELDS

103
docs/topics/link-extractors.rst
View File

@ -47,108 +47,7 @@ LxmlLinkExtractor
:synopsis: lxml's HTMLParser-based link extractors
.. class:: LxmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), restrict_css=(), tags=('a', 'area'), attrs=('href',), canonicalize=False, unique=True, process_value=None, strip=True, restrict_text=None)
LxmlLinkExtractor is the recommended link extractor with handy filtering
options. It is implemented using lxml's robust HTMLParser.
:param allow: a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be extracted. If not
given (or empty), it will match all links.
:type allow: str or list
:param deny: a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be excluded (i.e. not
extracted). It has precedence over the ``allow`` parameter. If not
given (or empty) it won't exclude any links.
:type deny: str or list
:param allow_domains: a single value or a list of string containing
domains which will be considered for extracting the links
:type allow_domains: str or list
:param deny_domains: a single value or a list of strings containing
domains which won't be considered for extracting the links
:type deny_domains: str or list
:param deny_extensions: a single value or list of strings containing
extensions that should be ignored when extracting links.
If not given, it will default to
:data:`scrapy.linkextractors.IGNORED_EXTENSIONS`.
:type deny_extensions: list
:param restrict_xpaths: is an XPath (or list of XPath's) which defines
regions inside the response where links should be extracted from.
If given, only the text selected by those XPath will be scanned for
links.
:type restrict_xpaths: str or list
:param restrict_css: a CSS selector (or list of selectors) which defines
regions inside the response where links should be extracted from.
Has the same behaviour as ``restrict_xpaths``.
:type restrict_css: str or list
:param restrict_text: a single regular expression (or list of regular expressions)
that the link's text must match in order to be extracted. If not
given (or empty), it will match all links. If a list of regular expressions is
given, the link will be extracted if it matches at least one.
:type restrict_text: str or list
:param tags: a tag or a list of tags to consider when extracting links.
Defaults to ``('a', 'area')``.
:type tags: str or list
:param attrs: an attribute or list of attributes which should be considered when looking
for links to extract (only for those tags specified in the ``tags``
parameter). Defaults to ``('href',)``
:type attrs: list
:param canonicalize: canonicalize each extracted url (using
w3lib.url.canonicalize_url). Defaults to ``False``.
Note that canonicalize_url is meant for duplicate checking;
it can change the URL visible at server side, so the response can be
different for requests with canonicalized and raw URLs. If you're
using LinkExtractor to follow links it is more robust to
keep the default ``canonicalize=False``.
:type canonicalize: bool
:param unique: whether duplicate filtering should be applied to extracted
links.
:type unique: bool
:param process_value: a function which receives each value extracted from
the tag and attributes scanned and can modify the value and return a
new one, or return ``None`` to ignore the link altogether. If not
given, ``process_value`` defaults to ``lambda x: x``.
.. highlight:: html
For example, to extract links from this code::
<a href="javascript:goToPage('../other/page.html'); return false">Link text</a>
.. highlight:: python
You can use the following function in ``process_value``:
.. code-block:: python
def process_value(value):
m = re.search(r"javascript:goToPage\('(.*?)'", value)
if m:
return m.group(1)
:type process_value: collections.abc.Callable
:param strip: whether to strip whitespaces from extracted attributes.
According to HTML5 standard, leading and trailing whitespaces
must be stripped from ``href`` attributes of ``<a>``, ``<area>``
and many other elements, ``src`` attribute of ``<img>``, ``<iframe>``
elements, etc., so LinkExtractor strips space chars by default.
Set ``strip=False`` to turn it off (e.g. if you're extracting urls
from elements or attributes which allow leading/trailing whitespaces).
:type strip: bool
.. autoclass:: LxmlLinkExtractor
.. automethod:: extract_links

11
docs/topics/request-response.rst
View File

@ -807,6 +807,8 @@ Whether or not to fail on broken responses. See:
give_up_log_level
-----------------
.. versionadded:: 2.17.0
:ref:`Logging level <levels>` used for the message logged when a request
exceeds its retries. See :setting:`RETRY_GIVE_UP_LOG_LEVEL` for details.
@ -815,6 +817,8 @@ exceeds its retries. See :setting:`RETRY_GIVE_UP_LOG_LEVEL` for details.
http_auth_domain
----------------
.. versionadded:: 2.17.0
Overrides :setting:`HTTPAUTH_DOMAIN` for this request.
.. reqmeta:: http_pass
@ -822,6 +826,8 @@ Overrides :setting:`HTTPAUTH_DOMAIN` for this request.
http_pass
---------
.. versionadded:: 2.17.0
Overrides :setting:`HTTPAUTH_PASS` for this request.
.. reqmeta:: http_user
@ -829,6 +835,8 @@ Overrides :setting:`HTTPAUTH_PASS` for this request.
http_user
---------
.. versionadded:: 2.17.0
Overrides :setting:`HTTPAUTH_USER` for this request.
.. reqmeta:: max_retry_times
@ -845,6 +853,8 @@ The meta key is used set retry times per request. When set, the
verbatim_url
------------
.. versionadded:: 2.17.0
Set this key to ``True`` to keep the request URL as passed to
:class:`~scrapy.Request`, without URL percent-encoding.
@ -855,7 +865,6 @@ characters that would otherwise be canonicalized get different fingerprints.
In this mode, the ``keep_fragments`` parameter is ignored, and it is
effectively true.
.. _topics-stop-response-download:
Stopping the download of a Response

13
docs/topics/selectors.rst
View File

@ -634,8 +634,7 @@ Example:
.. code-block:: pycon
>>> from scrapy import Selector
>>> sel = Selector(
... text="""
>>> sel = Selector(text="""
... <ul class="list">
... <li>1</li>
... <li>2</li>
@ -645,8 +644,8 @@ Example:
... <li>4</li>
... <li>5</li>
... <li>6</li>
... </ul>"""
... )
... </ul>""")
...
>>> xp = lambda x: sel.xpath(x).getall()
This gets all first ``<li>`` elements under whatever it is its parent:
@ -948,11 +947,9 @@ with groups of itemscopes and corresponding itemprops:
>>> sel = Selector(text=doc, type="html")
>>> for scope in sel.xpath("//div[@itemscope]"):
... print("current scope:", scope.xpath("@itemtype").getall())
... props = scope.xpath(
... """
... props = scope.xpath("""
... set:difference(./descendant::*/@itemprop,
... .//*[@itemscope]/*/@itemprop)"""
... )
... .//*[@itemscope]/*/@itemprop)""")
... print(f" properties: {props.getall()}")
... print("")
...

7
docs/topics/settings.rst
View File

@ -747,6 +747,9 @@ specific cipher that is not included in ``DEFAULT`` if a website requires it.
Set this setting to ``None`` to use the default ciphers of the underlying TLS
implementation.
.. versionchanged:: 2.17.0
Added support for setting this to ``None``.
.. _OpenSSL cipher list format: https://docs.openssl.org/master/man1/openssl-ciphers/#cipher-list-format
.. note::
@ -762,6 +765,8 @@ implementation.
DOWNLOAD_TLS_MAX_VERSION
------------------------
.. versionadded:: 2.17.0
Default: ``None``
Use this setting to change the maximum version of the TLS protocol allowed to
@ -797,6 +802,8 @@ modern environments.
DOWNLOAD_TLS_MIN_VERSION
------------------------
.. versionadded:: 2.17.0
Default: ``None``
Use this setting to change the minimum version of the TLS protocol allowed to

2
pyproject.toml
View File

@ -137,7 +137,7 @@ module = [
ignore_missing_imports = true
[tool.bumpversion]
current_version = "2.16.0"
current_version = "2.17.0"
commit = true
tag = true
tag_name = "{new_version}"

2
scrapy/VERSION
View File

@ -1 +1 @@
2.16.0
2.17.0

2
scrapy/core/downloader/handlers/s3.py
View File

@ -49,7 +49,7 @@ class S3DownloadHandler(BaseDownloadHandler):
async def download_request(self, request: Request) -> Response:
p = urlparse_cached(request)
scheme = "https" if request.meta.get("is_secure") else "http"
scheme = "http" if request.meta.get("is_secure") is False else "https"
bucket = p.hostname
path = p.path + "?" + p.query if p.query else p.path
url = f"{scheme}://{bucket}.s3.amazonaws.com{path}"

3
scrapy/downloadermiddlewares/retry.py
View File

@ -87,6 +87,9 @@ def get_retry_request(
message logged when a request exceeds its retries. See
:setting:`RETRY_GIVE_UP_LOG_LEVEL` for details.
.. versionadded:: 2.17.0
The *give_up_log_level* parameter.
*stats_base_key* is a string to be used as the base key for the
retry-related job stats
"""

2
scrapy/item.py
View File

@ -100,7 +100,7 @@ class Item(MutableMapping[str, Any], object_ref, metaclass=ItemMeta):
#: come first, followed by fields declared in subclasses, and a field
#: redefined in a subclass keeps the position of its first definition.
#:
#: .. versionchanged:: VERSION
#: .. versionchanged:: 2.17.0
#: Fields are now returned in definition order rather than alphabetical
#: order.
fields: dict[str, Field]

142
scrapy/linkextractors/lxmlhtml.py
View File

@ -57,6 +57,18 @@ def _canonicalize_link_url(link: Link) -> str:
return canonicalize_url(link.url, keep_fragments=True)
def _name_matches(allowed: set[str], denied: set[str], name: str) -> bool:
"""Return whether a tag or attribute *name* should be considered.
A name matches when it is allowed and not denied. A name is allowed if it is
listed in *allowed*, or if *allowed* contains the ``"*"`` wildcard, which
matches every name. *denied* has precedence over *allowed*.
"""
if name in denied:
return False
return "*" in allowed or name in allowed
class LxmlParserLinkExtractor:
def __init__(
self,
@ -162,6 +174,126 @@ _RegexOrSeveral: TypeAlias = _Regex | Iterable[_Regex]
class LxmlLinkExtractor:
r"""LxmlLinkExtractor is the recommended link extractor with handy filtering
options. It is implemented using lxml's robust HTMLParser.
:param allow: a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be extracted. If not
given (or empty), it will match all links.
:type allow: str or list
:param deny: a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be excluded (i.e. not
extracted). It has precedence over the ``allow`` parameter. If not
given (or empty) it won't exclude any links.
:type deny: str or list
:param allow_domains: a single value or a list of string containing
domains which will be considered for extracting the links
:type allow_domains: str or list
:param deny_domains: a single value or a list of strings containing
domains which won't be considered for extracting the links
:type deny_domains: str or list
:param deny_extensions: a single value or list of strings containing
extensions that should be ignored when extracting links.
If not given, it will default to
:data:`scrapy.linkextractors.IGNORED_EXTENSIONS`.
:type deny_extensions: list
:param restrict_xpaths: is an XPath (or list of XPath's) which defines
regions inside the response where links should be extracted from.
If given, only the text selected by those XPath will be scanned for
links.
:type restrict_xpaths: str or list
:param restrict_css: a CSS selector (or list of selectors) which defines
regions inside the response where links should be extracted from.
Has the same behaviour as ``restrict_xpaths``.
:type restrict_css: str or list
:param restrict_text: a single regular expression (or list of regular
expressions) that the link's text must match in order to be extracted.
If not given (or empty), it will match all links. If a list of regular
expressions is given, the link will be extracted if it matches at least
one.
:type restrict_text: str or list
:param tags: a tag or a list of tags to consider when extracting links.
Defaults to ``('a', 'area')``. Use ``'*'`` to consider every tag.
:type tags: str or list
:param attrs: an attribute or list of attributes which should be considered
when looking for links to extract (only for those tags specified in the
``tags`` parameter). Defaults to ``('href',)``. Use ``'*'`` to consider
every attribute.
:type attrs: list
:param deny_tags: a tag or a list of tags that should not be considered when
extracting links. It has precedence over the ``tags`` parameter, so it
can be combined with ``tags='*'`` to consider every tag except a few.
Defaults to ``()`` (no tag is excluded).
.. versionadded:: 2.17.0
:type deny_tags: str or list
:param deny_attrs: an attribute or a list of attributes that should not be
considered when looking for links to extract. It has precedence over the
``attrs`` parameter, so it can be combined with ``attrs='*'`` to consider
every attribute except a few. Defaults to ``()`` (no attribute is
excluded).
.. versionadded:: 2.17.0
:type deny_attrs: str or list
:param canonicalize: canonicalize each extracted url (using
w3lib.url.canonicalize_url). Defaults to ``False``.
Note that canonicalize_url is meant for duplicate checking;
it can change the URL visible at server side, so the response can be
different for requests with canonicalized and raw URLs. If you're
using LinkExtractor to follow links it is more robust to
keep the default ``canonicalize=False``.
:type canonicalize: bool
:param unique: whether duplicate filtering should be applied to extracted
links.
:type unique: bool
:param process_value: a function which receives each value extracted from
the tag and attributes scanned and can modify the value and return a
new one, or return ``None`` to ignore the link altogether. If not
given, ``process_value`` defaults to ``lambda x: x``.
.. highlight:: html
For example, to extract links from this code::
<a href="javascript:goToPage('../other/page.html'); return false">Link text</a>
.. highlight:: python
You can use the following function in ``process_value``:
.. code-block:: python
def process_value(value):
m = re.search(r"javascript:goToPage\('(.*?)'", value)
if m:
return m.group(1)
:type process_value: collections.abc.Callable
:param strip: whether to strip whitespaces from extracted attributes.
According to HTML5 standard, leading and trailing whitespaces
must be stripped from ``href`` attributes of ``<a>``, ``<area>``
and many other elements, ``src`` attribute of ``<img>``, ``<iframe>``
elements, etc., so LinkExtractor strips space chars by default.
Set ``strip=False`` to turn it off (e.g. if you're extracting urls
from elements or attributes which allow leading/trailing whitespaces).
:type strip: bool
"""
_csstranslator = HTMLTranslator()
def __init__(
@ -180,11 +312,17 @@ class LxmlLinkExtractor:
restrict_css: str | Iterable[str] = (),
strip: bool = True,
restrict_text: _RegexOrSeveral | None = None,
deny_tags: str | Iterable[str] = (),
deny_attrs: str | Iterable[str] = (),
):
tags, attrs = set(arg_to_iter(tags)), set(arg_to_iter(attrs))
deny_tags, deny_attrs = (
set(arg_to_iter(deny_tags)),
set(arg_to_iter(deny_attrs)),
)
self.link_extractor = LxmlParserLinkExtractor(
tag=partial(operator.contains, tags),
attr=partial(operator.contains, attrs),
tag=partial(_name_matches, tags, deny_tags),
attr=partial(_name_matches, attrs, deny_attrs),
unique=unique,
process=process_value,
strip=strip,

18
tests/test_downloader_handlers.py
View File

@ -170,6 +170,12 @@ class TestS3Anon:
httpreq = await self.download_request(req)
assert hasattr(self.s3reqh, "anon")
assert self.s3reqh.anon
assert httpreq.url == "https://aws-publicdatasets.s3.amazonaws.com/"
@coroutine_test
async def test_anon_request_insecure(self):
req = Request("s3://aws-publicdatasets/", meta={"is_secure": False})
httpreq = await self.download_request(req)
assert httpreq.url == "http://aws-publicdatasets.s3.amazonaws.com/"
@ -200,6 +206,18 @@ class TestS3:
mock_formatdate.return_value = date
yield
@coroutine_test
async def test_secure_by_default(self):
req = Request("s3://johnsmith/photos/puppy.jpg")
httpreq = await self.download_request(req)
assert httpreq.url == "https://johnsmith.s3.amazonaws.com/photos/puppy.jpg"
@coroutine_test
async def test_insecure_opt_out(self):
req = Request("s3://johnsmith/photos/puppy.jpg", meta={"is_secure": False})
httpreq = await self.download_request(req)
assert httpreq.url == "http://johnsmith.s3.amazonaws.com/photos/puppy.jpg"
@coroutine_test
async def test_request_signing1(self):
# gets an object from the johnsmith bucket.

53
tests/test_linkextractors.py
View File

@ -499,6 +499,59 @@ class Base:
),
]
def test_tags_attrs_wildcard_and_deny(self):
html = b"""
<html><body>
<a href="a.html">a</a>
<div data-url="div.html">div</div>
<span href="span.html">span</span>
<p data-url="p.html">p</p>
</body></html>
"""
response = HtmlResponse("http://example.com/index.html", body=html)
def urls(**kwargs):
lx = self.extractor_cls(**kwargs)
return [link.url for link in lx.extract_links(response)]
# Default behavior is unchanged: only the listed tags and attributes.
assert urls() == ["http://example.com/a.html"]
# "*" as a tag matches every tag.
assert urls(tags="*") == [
"http://example.com/a.html",
"http://example.com/span.html",
]
# "*" as an attribute matches every attribute.
assert urls(tags="*", attrs="*") == [
"http://example.com/a.html",
"http://example.com/div.html",
"http://example.com/span.html",
"http://example.com/p.html",
]
# deny_tags excludes tags from the wildcard.
assert urls(tags="*", attrs="*", deny_tags="div") == [
"http://example.com/a.html",
"http://example.com/span.html",
"http://example.com/p.html",
]
# deny_attrs excludes attributes from the wildcard.
assert urls(tags="*", attrs="*", deny_attrs="data-url") == [
"http://example.com/a.html",
"http://example.com/span.html",
]
# deny_tags also applies when tags are listed explicitly.
assert urls(tags=("a", "span"), attrs="*", deny_tags="span") == [
"http://example.com/a.html",
]
# The wildcard for one parameter is independent of the other.
assert urls(tags="a", attrs="*") == ["http://example.com/a.html"]
def test_xhtml(self):
xhtml = b"""
<?xml version="1.0"?>

20
tox.ini
View File

@ -70,13 +70,13 @@ commands =
basepython = python3.10
deps =
mypy==2.1.0
typing-extensions==4.15.0
Pillow==12.2.0
Protego==0.6.0
typing-extensions==4.16.0
Pillow==12.3.0
Protego==0.6.2
Twisted==26.4.0
attrs==26.1.0
boto3-stubs[s3]==1.43.9
botocore-stubs==1.42.41
boto3-stubs[s3]==1.43.41
botocore-stubs==1.43.14
h2==4.3.0
httpx==0.28.1
itemadapter==0.13.1
@ -84,12 +84,12 @@ deps =
# newer ones require newer Python
ipython==8.39.0
pyOpenSSL==26.3.0
pytest==9.0.3
pytest==9.1.1
socksio==1.0.0
types-Pygments==2.20.0.20260508
types-Pygments==2.20.0.20260518
types-defusedxml==0.7.0.20260504
types-lxml==2026.2.16
types-pexpect==4.9.0.20260508
types-pexpect==4.9.0.20260518
uvloop==0.22.1
w3lib==2.4.1
zstandard==0.25.0
@ -116,7 +116,7 @@ commands =
basepython = python3
deps =
{[testenv:extra-deps]deps}
pylint==4.0.2
pylint==4.0.6
pylint-per-file-ignores # https://github.com/pylint-dev/pylint/issues/3767#issuecomment-1319916278
commands =
pylint {posargs:conftest.py docs extras scrapy tests}
@ -125,7 +125,7 @@ commands =
basepython = python3
deps =
twine==6.2.0
build==1.3.0
build==1.5.0
commands =
python -m build --sdist
twine check dist/*