`_ in its `Content-Type`
+ header.
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index 4a64d530b..202b0823a 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -591,7 +591,7 @@ Another common case would be to extract all direct ```` children:
For more details about relative XPaths see the `Location Paths`_ section in the
XPath specification.
-.. _Location Paths: https://www.w3.org/TR/xpath/all/#location-paths
+.. _Location Paths: https://www.w3.org/TR/xpath-10/#location-paths
When querying by class, consider using CSS
------------------------------------------
@@ -727,7 +727,7 @@ But using the ``.`` to mean the node, works:
>>> sel.xpath("//a[contains(., 'Next Page')]").getall()
['Click here to go to the Next Page']
-.. _`XPath string function`: https://www.w3.org/TR/xpath/all/#section-String-Functions
+.. _`XPath string function`: https://www.w3.org/TR/xpath-10/#section-String-Functions
.. _topics-selectors-xpath-variables:
@@ -801,8 +801,8 @@ This is how the file starts::
...
You can see several namespace declarations including a default
-"http://www.w3.org/2005/Atom" and another one using the "gd:" prefix for
-"http://schemas.google.com/g/2005".
+``"http://www.w3.org/2005/Atom"`` and another one using the ``gd:`` prefix for
+``"http://schemas.google.com/g/2005"``.
.. highlight:: python
@@ -1032,10 +1032,8 @@ whereas the CSS lookup is translated into XPath and thus runs more efficiently,
so performance-wise its uses are limited to situations that are not easily
described with CSS selectors.
-Parsel also simplifies adding your own XPath extensions.
-
-.. autofunction:: parsel.xpathfuncs.set_xpathfunc
-
+Parsel also simplifies adding your own XPath extensions with
+:func:`~parsel.xpathfuncs.set_xpathfunc`.
.. _topics-selectors-ref:
@@ -1062,6 +1060,12 @@ Selector objects
For convenience, this method can be called as ``response.css()``
+ .. automethod:: jmespath
+
+ .. note::
+
+ For convenience, this method can be called as ``response.jmespath()``
+
.. automethod:: get
See also: :ref:`old-extraction-api`
@@ -1094,6 +1098,8 @@ SelectorList objects
.. automethod:: css
+ .. automethod:: jmespath
+
.. automethod:: getall
See also: :ref:`old-extraction-api`
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 06f8481ba..76904a26e 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -288,7 +288,7 @@ The AWS security token used by code that requires access to `Amazon Web services
such as the :ref:`S3 feed storage backend `, when using
`temporary security credentials`_.
-.. _temporary security credentials: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys
+.. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
.. setting:: AWS_ENDPOINT_URL
@@ -617,7 +617,7 @@ necessary to access certain HTTPS websites: for example, you may need to use
``'DEFAULT:!DH'`` for a website with weak DH parameters or enable a
specific cipher that is not included in ``DEFAULT`` if a website requires it.
-.. _OpenSSL cipher list format: https://www.openssl.org/docs/manmaster/man1/openssl-ciphers.html#CIPHER-LIST-FORMAT
+.. _OpenSSL cipher list format: https://docs.openssl.org/master/man1/openssl-ciphers/#cipher-list-format
.. setting:: DOWNLOADER_CLIENT_TLS_METHOD
@@ -674,6 +674,7 @@ Default:
.. code-block:: python
{
+ "scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": 50,
"scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,
"scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,
"scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,
@@ -828,14 +829,14 @@ The default HTTPS handler uses HTTP/1.1. To use HTTP/2:
- No support for the :signal:`bytes_received` and
:signal:`headers_received` signals.
-.. _frame size: https://tools.ietf.org/html/rfc7540#section-4.2
+.. _frame size: https://datatracker.ietf.org/doc/html/rfc7540#section-4.2
.. _http2 faq: https://http2.github.io/faq/#does-http2-require-encryption
-.. _server pushes: https://tools.ietf.org/html/rfc7540#section-8.2
+.. _server pushes: https://datatracker.ietf.org/doc/html/rfc7540#section-8.2
.. setting:: DOWNLOAD_SLOTS
DOWNLOAD_SLOTS
-----------------
+--------------
Default: ``{}``
@@ -873,40 +874,42 @@ The amount of time (in secs) that the downloader will wait before timing out.
Request.meta key.
.. setting:: DOWNLOAD_MAXSIZE
+.. reqmeta:: download_maxsize
DOWNLOAD_MAXSIZE
----------------
-Default: ``1073741824`` (1024MB)
+Default: ``1073741824`` (1 GiB)
-The maximum response size (in bytes) that downloader will download.
+The maximum response body size (in bytes) allowed. Bigger responses are
+aborted and ignored.
-If you want to disable it set to 0.
+This applies both before and after compression. If decompressing a response
+body would exceed this limit, decompression is aborted and the response is
+ignored.
-.. reqmeta:: download_maxsize
+Use ``0`` to disable this limit.
-.. note::
-
- This size can be set per spider using :attr:`download_maxsize`
- spider attribute and per-request using :reqmeta:`download_maxsize`
- Request.meta key.
+This limit can be set per spider using the :attr:`download_maxsize` spider
+attribute and per request using the :reqmeta:`download_maxsize` Request.meta
+key.
.. setting:: DOWNLOAD_WARNSIZE
+.. reqmeta:: download_warnsize
DOWNLOAD_WARNSIZE
-----------------
-Default: ``33554432`` (32MB)
+Default: ``33554432`` (32 MiB)
-The response size (in bytes) that downloader will start to warn.
+If the size of a response exceeds this value, before or after compression, a
+warning will be logged about it.
-If you want to disable it set to 0.
+Use ``0`` to disable this limit.
-.. note::
-
- This size can be set per spider using :attr:`download_warnsize`
- spider attribute and per-request using :reqmeta:`download_warnsize`
- Request.meta key.
+This limit can be set per spider using the :attr:`download_warnsize` spider
+attribute and per request using the :reqmeta:`download_warnsize` Request.meta
+key.
.. setting:: DOWNLOAD_FAIL_ON_DATALOSS
@@ -1063,7 +1066,7 @@ in ``Request`` meta.
some FTP servers explicitly ask for the user's e-mail address
and will not allow login with the "guest" password.
-.. _RFC 1635: https://tools.ietf.org/html/rfc1635
+.. _RFC 1635: https://datatracker.ietf.org/doc/html/rfc1635
.. reqmeta:: ftp_user
.. setting:: FTP_USER
@@ -1225,6 +1228,25 @@ Default: ``False``
If ``True``, the logs will just contain the root path. If it is set to ``False``
then it displays the component responsible for the log output
+.. setting:: LOG_VERSIONS
+
+LOG_VERSIONS
+------------
+
+Default: ``["lxml", "libxml2", "cssselect", "parsel", "w3lib", "Twisted", "Python", "pyOpenSSL", "cryptography", "Platform"]``
+
+Logs the installed versions of the specified items.
+
+An item can be any installed Python package.
+
+The following special items are also supported:
+
+- ``libxml2``
+
+- ``Platform`` (:func:`platform.platform`)
+
+- ``Python``
+
.. setting:: LOGSTATS_INTERVAL
LOGSTATS_INTERVAL
@@ -1569,7 +1591,7 @@ SPIDER_LOADER_WARN_ONLY
Default: ``False``
By default, when Scrapy tries to import spider classes from :setting:`SPIDER_MODULES`,
-it will fail loudly if there is any ``ImportError`` exception.
+it will fail loudly if there is any ``ImportError`` or ``SyntaxError`` exception.
But you can choose to silence this exception and turn it into a simple
warning by setting ``SPIDER_LOADER_WARN_ONLY = True``.
@@ -1603,7 +1625,6 @@ Default:
{
"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
- "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": 500,
"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
"scrapy.spidermiddlewares.depth.DepthMiddleware": 900,
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index 9bfd1761c..b45b12540 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -159,8 +159,9 @@ item_scraped
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.Spider` object
- :param response: the response from where the item was scraped
- :type response: :class:`~scrapy.http.Response` object
+ :param response: the response from where the item was scraped, or ``None``
+ if it was yielded from :meth:`~scrapy.Spider.start_requests`.
+ :type response: :class:`~scrapy.http.Response` | ``None``
item_dropped
~~~~~~~~~~~~
@@ -179,8 +180,9 @@ item_dropped
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.Spider` object
- :param response: the response from where the item was dropped
- :type response: :class:`~scrapy.http.Response` object
+ :param response: the response from where the item was dropped, or ``None``
+ if it was yielded from :meth:`~scrapy.Spider.start_requests`.
+ :type response: :class:`~scrapy.http.Response` | ``None``
:param exception: the exception (which must be a
:exc:`~scrapy.exceptions.DropItem` subclass) which caused the item
@@ -201,8 +203,10 @@ item_error
:param item: the item that caused the error in the :ref:`topics-item-pipeline`
:type item: :ref:`item object `
- :param response: the response being processed when the exception was raised
- :type response: :class:`~scrapy.http.Response` object
+ :param response: the response being processed when the exception was
+ raised, or ``None`` if it was yielded from
+ :meth:`~scrapy.Spider.start_requests`.
+ :type response: :class:`~scrapy.http.Response` | ``None``
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.Spider` object
@@ -343,11 +347,18 @@ request_scheduled
.. signal:: request_scheduled
.. function:: request_scheduled(request, spider)
- Sent when the engine schedules a :class:`~scrapy.Request`, to be
- downloaded later.
+ Sent when the engine is asked to schedule a :class:`~scrapy.Request`, to be
+ downloaded later, before the request reaches the :ref:`scheduler
+ `.
+
+ Raise :exc:`~scrapy.exceptions.IgnoreRequest` to drop a request before it
+ reaches the scheduler.
This signal does not support returning deferreds from its handlers.
+ .. versionadded:: 2.11.2
+ Allow dropping requests with :exc:`~scrapy.exceptions.IgnoreRequest`.
+
:param request: the request that reached the scheduler
:type request: :class:`~scrapy.Request` object
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 3f16efea5..2b59cabe1 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -51,8 +51,8 @@ value. For example, if you want to disable the off-site middleware:
.. code-block:: python
SPIDER_MIDDLEWARES = {
- "myproject.middlewares.CustomSpiderMiddleware": 543,
- "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,
+ "scrapy.spidermiddlewares.referer.RefererMiddleware": None,
+ "myproject.middlewares.CustomRefererSpiderMiddleware": 700,
}
Finally, keep in mind that some middlewares may need to be enabled through a
@@ -176,7 +176,7 @@ object gives you access, for example, to the :ref:`settings `.
items).
It receives an iterable (in the ``start_requests`` parameter) and must
- return another iterable of :class:`~scrapy.Request` objects.
+ return another iterable of :class:`~scrapy.Request` objects and/or :ref:`item objects `.
.. note:: When implementing this method in your spider middleware, you
should always return an iterable (that follows the input one) and
@@ -313,42 +313,6 @@ Default: ``False``
Pass all responses, regardless of its status code.
-OffsiteMiddleware
------------------
-
-.. module:: scrapy.spidermiddlewares.offsite
- :synopsis: Offsite Spider Middleware
-
-.. class:: OffsiteMiddleware
-
- Filters out Requests for URLs outside the domains covered by the spider.
-
- This middleware filters out every request whose host names aren't in the
- spider's :attr:`~scrapy.Spider.allowed_domains` attribute.
- All subdomains of any domain in the list are also allowed.
- E.g. the rule ``www.example.org`` will also allow ``bob.www.example.org``
- but not ``www2.example.com`` nor ``example.com``.
-
- When your spider returns a request for a domain not belonging to those
- covered by the spider, this middleware will log a debug message similar to
- this one::
-
- DEBUG: Filtered offsite request to 'www.othersite.com':
-
- To avoid filling the log with too much noise, it will only print one of
- these messages for each new domain filtered. So, for example, if another
- request for ``www.othersite.com`` is filtered, no log message will be
- printed. But if a request for ``someothersite.com`` is filtered, a message
- will be printed (but only for the first request filtered).
-
- If the spider doesn't define an
- :attr:`~scrapy.Spider.allowed_domains` attribute, or the
- attribute is empty, the offsite middleware will allow all requests.
-
- If the request has the :attr:`~scrapy.Request.dont_filter` attribute
- set, the offsite middleware will allow the request even if its domain is not
- listed in allowed domains.
-
RefererMiddleware
-----------------
@@ -394,7 +358,7 @@ Acceptable values for REFERRER_POLICY
- either a path to a ``scrapy.spidermiddlewares.referer.ReferrerPolicy``
subclass — a custom policy or one of the built-in ones (see classes below),
-- or one of the standard W3C-defined string values,
+- or one or more comma-separated standard W3C-defined string values,
- or the special ``"scrapy-default"``.
======================================= ========================================================================
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 30677fe74..e1b1c5ad6 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -75,7 +75,8 @@ scrapy.Spider
An optional list of strings containing domains that this spider is
allowed to crawl. Requests for URLs not belonging to the domain names
specified in this list (or their subdomains) won't be followed if
- :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled.
+ :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is
+ enabled.
Let's say your target url is ``https://www.example.com/1.html``,
then add ``'example.com'`` to the list.
@@ -202,7 +203,8 @@ scrapy.Spider
.. method:: start_requests()
- This method must return an iterable with the first Requests to crawl for
+ This method must return an iterable with the first Requests to crawl and/or with :ref:`item objects
+ ` for
this spider. It is called by Scrapy when the spider is opened for
scraping. Scrapy calls it only once, so it is safe to implement
:meth:`start_requests` as a generator.
diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst
index 832829b75..0e4a8fa6c 100644
--- a/docs/topics/telnetconsole.rst
+++ b/docs/topics/telnetconsole.rst
@@ -172,8 +172,8 @@ TELNETCONSOLE_PORT
Default: ``[6023, 6073]``
-The port range to use for the telnet console. If set to ``None`` or ``0``, a
-dynamically assigned port is used.
+The port range to use for the telnet console. If set to ``None``, a dynamically
+assigned port is used.
.. setting:: TELNETCONSOLE_HOST
diff --git a/extras/coverage-report.sh b/extras/coverage-report.sh
index 842d0e46e..7eaa214cf 100755
--- a/extras/coverage-report.sh
+++ b/extras/coverage-report.sh
@@ -1,6 +1,6 @@
# Run tests, generate coverage report and open it on a browser
#
-# Requires: coverage 3.3 or above from https://pypi.python.org/pypi/coverage
+# Requires: coverage 3.3 or above from https://pypi.org/pypi/coverage
coverage run --branch $(which trial) --reporter=text tests
coverage html -i
diff --git a/extras/scrapy_zsh_completion b/extras/scrapy_zsh_completion
index e2f2dc82b..82eb77cc0 100644
--- a/extras/scrapy_zsh_completion
+++ b/extras/scrapy_zsh_completion
@@ -41,7 +41,6 @@ _scrapy() {
(runspider)
local options=(
{'(--output)-o','(-o)--output='}'[dump scraped items into FILE (use - for stdout)]:file:_files'
- {'(--output-format)-t','(-t)--output-format='}'[format to use for dumping items with -o]:format:(FORMAT)'
'*-a[set spider argument (may be repeated)]:value pair:(NAME=VALUE)'
'1:spider file:_files -g \*.py'
)
@@ -99,7 +98,6 @@ _scrapy() {
(crawl)
local options=(
{'(--output)-o','(-o)--output='}'[dump scraped items into FILE (use - for stdout)]:file:_files'
- {'(--output-format)-t','(-t)--output-format='}'[format to use for dumping items with -o]:format:(FORMAT)'
'*-a[set spider argument (may be repeated)]:value pair:(NAME=VALUE)'
'1:spider:_scrapy_spiders'
)
diff --git a/pylintrc b/pylintrc
deleted file mode 100644
index c8654b8d3..000000000
--- a/pylintrc
+++ /dev/null
@@ -1,99 +0,0 @@
-[MASTER]
-persistent=no
-jobs=1 # >1 hides results
-
-[MESSAGES CONTROL]
-disable=abstract-method,
- anomalous-backslash-in-string,
- arguments-differ,
- arguments-renamed,
- attribute-defined-outside-init,
- bad-classmethod-argument,
- bad-mcs-classmethod-argument,
- bare-except,
- broad-except,
- broad-exception-raised,
- c-extension-no-member,
- catching-non-exception,
- cell-var-from-loop,
- comparison-with-callable,
- consider-using-dict-items,
- consider-using-in,
- consider-using-with,
- cyclic-import,
- dangerous-default-value,
- disallowed-name,
- duplicate-code, # https://github.com/PyCQA/pylint/issues/214
- eval-used,
- expression-not-assigned,
- fixme,
- function-redefined,
- global-statement,
- implicit-str-concat,
- import-error,
- import-outside-toplevel,
- import-self,
- inconsistent-return-statements,
- inherit-non-class,
- invalid-name,
- invalid-overridden-method,
- isinstance-second-argument-not-valid-type,
- keyword-arg-before-vararg,
- line-too-long,
- logging-format-interpolation,
- logging-fstring-interpolation,
- logging-not-lazy,
- lost-exception,
- method-hidden,
- missing-docstring,
- no-else-raise,
- no-else-return,
- no-member,
- no-method-argument,
- no-name-in-module,
- no-self-argument,
- no-value-for-parameter,
- not-callable,
- pointless-exception-statement,
- pointless-statement,
- pointless-string-statement,
- protected-access,
- raise-missing-from,
- redefined-argument-from-local,
- redefined-builtin,
- redefined-outer-name,
- reimported,
- signature-differs,
- super-init-not-called,
- too-few-public-methods,
- too-many-ancestors,
- too-many-arguments,
- too-many-branches,
- too-many-format-args,
- too-many-function-args,
- too-many-instance-attributes,
- too-many-lines,
- too-many-locals,
- too-many-public-methods,
- too-many-return-statements,
- unbalanced-tuple-unpacking,
- undefined-variable,
- undefined-loop-variable,
- unexpected-special-method-signature,
- unnecessary-comprehension,
- unnecessary-dunder-call,
- unnecessary-pass,
- unreachable,
- unsubscriptable-object,
- unused-argument,
- unused-import,
- unused-private-member,
- unused-variable,
- unused-wildcard-import,
- use-dict-literal,
- used-before-assignment,
- useless-object-inheritance, # Required for Python 2 support
- useless-return,
- useless-super-delegation,
- wildcard-import,
- wrong-import-position
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 000000000..ad85e5c75
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,353 @@
+[build-system]
+requires = ["setuptools >= 61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "Scrapy"
+dynamic = ["version"]
+description = "A high-level Web Crawling and Web Scraping framework"
+dependencies = [
+ "Twisted>=21.7.0",
+ "cryptography>=37.0.0",
+ "cssselect>=0.9.1",
+ "itemloaders>=1.0.1",
+ "parsel>=1.5.0",
+ "pyOpenSSL>=22.0.0",
+ "queuelib>=1.4.2",
+ "service_identity>=18.1.0",
+ "w3lib>=1.17.0",
+ "zope.interface>=5.1.0",
+ "protego>=0.1.15",
+ "itemadapter>=0.1.0",
+ "packaging",
+ "tldextract",
+ "lxml>=4.6.0",
+ "defusedxml>=0.7.1",
+ # Platform-specific dependencies
+ 'PyDispatcher>=2.0.5; platform_python_implementation == "CPython"',
+ 'PyPyDispatcher>=2.1.0; platform_python_implementation == "PyPy"',
+]
+classifiers = [
+ "Framework :: Scrapy",
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: BSD License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Programming Language :: Python :: 3.13",
+ "Programming Language :: Python :: Implementation :: CPython",
+ "Programming Language :: Python :: Implementation :: PyPy",
+ "Topic :: Internet :: WWW/HTTP",
+ "Topic :: Software Development :: Libraries :: Application Frameworks",
+ "Topic :: Software Development :: Libraries :: Python Modules",
+]
+readme = "README.rst"
+requires-python = ">=3.9"
+authors = [{ name = "Scrapy developers", email = "pablo@pablohoffman.com" }]
+maintainers = [{ name = "Pablo Hoffman", email = "pablo@pablohoffman.com" }]
+
+[project.urls]
+Homepage = "https://scrapy.org/"
+Documentation = "https://docs.scrapy.org/"
+Source = "https://github.com/scrapy/scrapy"
+Tracker = "https://github.com/scrapy/scrapy/issues"
+Changelog = "https://github.com/scrapy/scrapy/commits/master/"
+releasenotes = "https://docs.scrapy.org/en/latest/news.html"
+
+[project.scripts]
+scrapy = "scrapy.cmdline:execute"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["scrapy", "scrapy.*",]
+
+[tool.setuptools.dynamic]
+version = {file = "./scrapy/VERSION"}
+
+[tool.mypy]
+ignore_missing_imports = true
+
+# Interface classes are hard to support
+
+[[tool.mypy.overrides]]
+module = "twisted.internet.interfaces"
+follow_imports = "skip"
+
+[[tool.mypy.overrides]]
+module = "scrapy.interfaces"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module = "twisted.internet.reactor"
+follow_imports = "skip"
+
+# FIXME: remove the following section once the issues are solved
+[[tool.mypy.overrides]]
+module = "scrapy.settings.default_settings"
+ignore_errors = true
+
+[tool.bumpversion]
+current_version = "2.12.0"
+commit = true
+tag = true
+tag_name = "{new_version}"
+
+[[tool.bumpversion.files]]
+filename = "scrapy/VERSION"
+
+[[tool.bumpversion.files]]
+filename = "SECURITY.md"
+parse = """(?P0|[1-9]\\d*)\\.(?P0|[1-9]\\d*)"""
+serialize = ["{major}.{minor}"]
+
+[tool.coverage.run]
+branch = true
+include = ["scrapy/*"]
+omit = ["tests/*"]
+disable_warnings = ["include-ignored"]
+
+[tool.coverage.report]
+# https://github.com/nedbat/coveragepy/issues/831#issuecomment-517778185
+exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]
+
+[tool.pylint.MASTER]
+persistent = "no"
+jobs = 1 # >1 hides results
+extension-pkg-allow-list=[
+ "lxml",
+]
+
+[tool.pylint."MESSAGES CONTROL"]
+disable = [
+ "abstract-method",
+ "arguments-differ",
+ "arguments-renamed",
+ "attribute-defined-outside-init",
+ "broad-exception-caught",
+ "consider-using-with",
+ "cyclic-import",
+ "dangerous-default-value",
+ "disallowed-name",
+ "duplicate-code", # https://github.com/PyCQA/pylint/issues/214
+ "eval-used",
+ "fixme",
+ "import-error",
+ "import-outside-toplevel",
+ "inherit-non-class",
+ "invalid-name",
+ "invalid-overridden-method",
+ "isinstance-second-argument-not-valid-type",
+ "keyword-arg-before-vararg",
+ "line-too-long",
+ "logging-format-interpolation",
+ "logging-fstring-interpolation",
+ "logging-not-lazy",
+ "missing-docstring",
+ "no-member",
+ "no-method-argument",
+ "no-name-in-module",
+ "no-self-argument",
+ "no-value-for-parameter", # https://github.com/pylint-dev/pylint/issues/3268
+ "not-callable",
+ "pointless-statement",
+ "pointless-string-statement",
+ "protected-access",
+ "raise-missing-from",
+ "redefined-builtin",
+ "redefined-outer-name",
+ "signature-differs",
+ "too-few-public-methods",
+ "too-many-ancestors",
+ "too-many-arguments",
+ "too-many-branches",
+ "too-many-function-args",
+ "too-many-instance-attributes",
+ "too-many-lines",
+ "too-many-locals",
+ "too-many-positional-arguments",
+ "too-many-public-methods",
+ "too-many-return-statements",
+ "unbalanced-tuple-unpacking",
+ "unnecessary-dunder-call",
+ "unused-argument",
+ "unused-import",
+ "unused-variable",
+ "used-before-assignment",
+ "useless-return",
+ "wrong-import-position",
+]
+
+[tool.pytest.ini_options]
+xfail_strict = true
+usefixtures = "chdir"
+python_files = ["test_*.py", "__init__.py"]
+python_classes = []
+addopts = [
+ "--assert=plain",
+ "--ignore=docs/_ext",
+ "--ignore=docs/conf.py",
+ "--ignore=docs/news.rst",
+ "--ignore=docs/topics/dynamic-content.rst",
+ "--ignore=docs/topics/items.rst",
+ "--ignore=docs/topics/leaks.rst",
+ "--ignore=docs/topics/loaders.rst",
+ "--ignore=docs/topics/selectors.rst",
+ "--ignore=docs/topics/shell.rst",
+ "--ignore=docs/topics/stats.rst",
+ "--ignore=docs/topics/telnetconsole.rst",
+ "--ignore=docs/utils",
+]
+markers = [
+ "only_asyncio: marks tests as only enabled when --reactor=asyncio is passed",
+ "only_not_asyncio: marks tests as only enabled when --reactor=asyncio is not passed",
+ "requires_uvloop: marks tests as only enabled when uvloop is known to be working",
+ "requires_botocore: marks tests that need botocore (but not boto3)",
+ "requires_boto3: marks tests that need botocore and boto3",
+]
+filterwarnings = []
+
+[tool.ruff.lint]
+extend-select = [
+ # flake8-bugbear
+ "B",
+ # flake8-comprehensions
+ "C4",
+ # pydocstyle
+ "D",
+ # flake8-future-annotations
+ "FA",
+ # refurb
+ "FURB",
+ # isort
+ "I",
+ # flake8-implicit-str-concat
+ "ISC",
+ # flake8-logging
+ "LOG",
+ # Perflint
+ "PERF",
+ # pygrep-hooks
+ "PGH",
+ # flake8-pie
+ "PIE",
+ # pylint
+ "PL",
+ # flake8-pyi
+ "PYI",
+ # flake8-quotes
+ "Q",
+ # flake8-return
+ "RET",
+ # flake8-raise
+ "RSE",
+ # flake8-bandit
+ "S",
+ # flake8-slots
+ "SLOT",
+ # flake8-debugger
+ "T10",
+ # flake8-type-checking
+ "TC",
+ # pyupgrade
+ "UP",
+ # pycodestyle warnings
+ "W",
+ # flake8-2020
+ "YTT",
+]
+ignore = [
+ # Assigning to `os.environ` doesn't clear the environment.
+ "B003",
+ # Do not use mutable data structures for argument defaults.
+ "B006",
+ # Loop control variable not used within the loop body.
+ "B007",
+ # Do not perform function calls in argument defaults.
+ "B008",
+ # Star-arg unpacking after a keyword argument is strongly discouraged.
+ "B026",
+ # Found useless expression.
+ "B018",
+ # No explicit stacklevel argument found.
+ "B028",
+ # Within an `except` clause, raise exceptions with `raise ... from`
+ "B904",
+ # Missing docstring in public module
+ "D100",
+ # Missing docstring in public class
+ "D101",
+ # Missing docstring in public method
+ "D102",
+ # Missing docstring in public function
+ "D103",
+ # Missing docstring in public package
+ "D104",
+ # Missing docstring in magic method
+ "D105",
+ # Missing docstring in public nested class
+ "D106",
+ # Missing docstring in __init__
+ "D107",
+ # One-line docstring should fit on one line with quotes
+ "D200",
+ # No blank lines allowed after function docstring
+ "D202",
+ # 1 blank line required between summary line and description
+ "D205",
+ # Multi-line docstring closing quotes should be on a separate line
+ "D209",
+ # First line should end with a period
+ "D400",
+ # First line should be in imperative mood; try rephrasing
+ "D401",
+ # First line should not be the function's "signature"
+ "D402",
+ # First word of the first line should be properly capitalized
+ "D403",
+ # `try`-`except` within a loop incurs performance overhead
+ "PERF203",
+ # Too many return statements
+ "PLR0911",
+ # Too many branches
+ "PLR0912",
+ # Too many arguments in function definition
+ "PLR0913",
+ # Too many statements
+ "PLR0915",
+ # Magic value used in comparison
+ "PLR2004",
+ # `for` loop variable overwritten by assignment target
+ "PLW2901",
+ # Use of `assert` detected; needed for mypy
+ "S101",
+ # FTP-related functions are being called; https://github.com/scrapy/scrapy/issues/4180
+ "S321",
+ # Argument default set to insecure SSL protocol
+ "S503",
+]
+
+[tool.ruff.lint.per-file-ignores]
+# Exclude files that are meant to provide top-level imports
+"scrapy/__init__.py" = ["E402"]
+"scrapy/core/downloader/handlers/http.py" = ["F401"]
+"scrapy/http/__init__.py" = ["F401"]
+"scrapy/linkextractors/__init__.py" = ["E402", "F401"]
+"scrapy/selector/__init__.py" = ["F401"]
+"scrapy/spiders/__init__.py" = ["E402", "F401"]
+
+# Skip bandit in tests
+"tests/**" = ["S"]
+
+# Issues pending a review:
+"docs/conf.py" = ["E402"]
+"scrapy/utils/url.py" = ["F403", "F405"]
+"tests/test_loader.py" = ["E741"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "pep257"
diff --git a/pytest.ini b/pytest.ini
deleted file mode 100644
index 16983be5e..000000000
--- a/pytest.ini
+++ /dev/null
@@ -1,28 +0,0 @@
-[pytest]
-xfail_strict = true
-usefixtures = chdir
-python_files=test_*.py __init__.py
-python_classes=
-addopts =
- --assert=plain
- --ignore=docs/_ext
- --ignore=docs/conf.py
- --ignore=docs/news.rst
- --ignore=docs/topics/dynamic-content.rst
- --ignore=docs/topics/items.rst
- --ignore=docs/topics/leaks.rst
- --ignore=docs/topics/loaders.rst
- --ignore=docs/topics/selectors.rst
- --ignore=docs/topics/shell.rst
- --ignore=docs/topics/stats.rst
- --ignore=docs/topics/telnetconsole.rst
- --ignore=docs/utils
-markers =
- only_asyncio: marks tests as only enabled when --reactor=asyncio is passed
- only_not_asyncio: marks tests as only enabled when --reactor=asyncio is not passed
- requires_uvloop: marks tests as only enabled when uvloop is known to be working
-filterwarnings =
- ignore:scrapy.downloadermiddlewares.decompression is deprecated
- ignore:Module scrapy.utils.reqser is deprecated
- ignore:typing.re is deprecated
- ignore:typing.io is deprecated
diff --git a/scrapy/VERSION b/scrapy/VERSION
index 46b81d815..d8b698973 100644
--- a/scrapy/VERSION
+++ b/scrapy/VERSION
@@ -1 +1 @@
-2.11.0
+2.12.0
diff --git a/scrapy/__init__.py b/scrapy/__init__.py
index cc0e539c4..c19710a6a 100644
--- a/scrapy/__init__.py
+++ b/scrapy/__init__.py
@@ -6,8 +6,6 @@ import pkgutil
import sys
import warnings
-from twisted import version as _txv
-
# Declare top-level shortcuts
from scrapy.http import FormRequest, Request
from scrapy.item import Field, Item
@@ -17,7 +15,6 @@ from scrapy.spiders import Spider
__all__ = [
"__version__",
"version_info",
- "twisted_version",
"Spider",
"Request",
"FormRequest",
@@ -30,13 +27,23 @@ __all__ = [
# Scrapy and Twisted versions
__version__ = (pkgutil.get_data(__package__, "VERSION") or b"").decode("ascii").strip()
version_info = tuple(int(v) if v.isdigit() else v for v in __version__.split("."))
-twisted_version = (_txv.major, _txv.minor, _txv.micro)
-# Check minimum required Python version
-if sys.version_info < (3, 8):
- print(f"Scrapy {__version__} requires Python 3.8+")
- sys.exit(1)
+def __getattr__(name: str):
+ if name == "twisted_version":
+ import warnings # pylint: disable=reimported
+
+ from twisted import version as _txv
+
+ from scrapy.exceptions import ScrapyDeprecationWarning
+
+ warnings.warn(
+ "The scrapy.twisted_version attribute is deprecated, use twisted.version instead",
+ ScrapyDeprecationWarning,
+ )
+ return _txv.major, _txv.minor, _txv.micro
+
+ raise AttributeError
# Ignore noisy twisted deprecation warnings
diff --git a/scrapy/addons.py b/scrapy/addons.py
index 9060d4f3f..7a1da3afc 100644
--- a/scrapy/addons.py
+++ b/scrapy/addons.py
@@ -1,13 +1,16 @@
+from __future__ import annotations
+
import logging
-from typing import TYPE_CHECKING, Any, List
+from typing import TYPE_CHECKING, Any
from scrapy.exceptions import NotConfigured
-from scrapy.settings import Settings
from scrapy.utils.conf import build_component_list
-from scrapy.utils.misc import create_instance, load_object
+from scrapy.utils.misc import build_from_crawler, load_object
if TYPE_CHECKING:
from scrapy.crawler import Crawler
+ from scrapy.settings import Settings
+
logger = logging.getLogger(__name__)
@@ -15,9 +18,9 @@ logger = logging.getLogger(__name__)
class AddonManager:
"""This class facilitates loading and storing :ref:`topics-addons`."""
- def __init__(self, crawler: "Crawler") -> None:
- self.crawler: "Crawler" = crawler
- self.addons: List[Any] = []
+ def __init__(self, crawler: Crawler) -> None:
+ self.crawler: Crawler = crawler
+ self.addons: list[Any] = []
def load_settings(self, settings: Settings) -> None:
"""Load add-ons and configurations from a settings object and apply them.
@@ -32,9 +35,7 @@ class AddonManager:
for clspath in build_component_list(settings["ADDONS"]):
try:
addoncls = load_object(clspath)
- addon = create_instance(
- addoncls, settings=settings, crawler=self.crawler
- )
+ addon = build_from_crawler(addoncls, self.crawler)
addon.update_settings(settings)
self.addons.append(addon)
except NotConfigured as e:
diff --git a/scrapy/cmdline.py b/scrapy/cmdline.py
index 6580ba9ce..9a24871de 100644
--- a/scrapy/cmdline.py
+++ b/scrapy/cmdline.py
@@ -1,9 +1,12 @@
+from __future__ import annotations
+
import argparse
import cProfile
import inspect
import os
import sys
from importlib.metadata import entry_points
+from typing import TYPE_CHECKING
import scrapy
from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter
@@ -13,9 +16,21 @@ from scrapy.utils.misc import walk_modules
from scrapy.utils.project import get_project_settings, inside_project
from scrapy.utils.python import garbage_collect
+if TYPE_CHECKING:
+ from collections.abc import Callable, Iterable
+
+ # typing.ParamSpec requires Python 3.10
+ from typing_extensions import ParamSpec
+
+ from scrapy.settings import BaseSettings, Settings
+
+ _P = ParamSpec("_P")
+
class ScrapyArgumentParser(argparse.ArgumentParser):
- def _parse_optional(self, arg_string):
+ def _parse_optional(
+ self, arg_string: str
+ ) -> tuple[argparse.Action | None, str, str | None] | None:
# if starts with -: it means that is a parameter not a argument
if arg_string[:2] == "-:":
return None
@@ -23,7 +38,7 @@ class ScrapyArgumentParser(argparse.ArgumentParser):
return super()._parse_optional(arg_string)
-def _iter_command_classes(module_name):
+def _iter_command_classes(module_name: str) -> Iterable[type[ScrapyCommand]]:
# TODO: add `name` attribute to commands and merge this function with
# scrapy.utils.spider.iter_spider_classes
for module in walk_modules(module_name):
@@ -37,8 +52,8 @@ def _iter_command_classes(module_name):
yield obj
-def _get_commands_from_module(module, inproject):
- d = {}
+def _get_commands_from_module(module: str, inproject: bool) -> dict[str, ScrapyCommand]:
+ d: dict[str, ScrapyCommand] = {}
for cmd in _iter_command_classes(module):
if inproject or not cmd.requires_project:
cmdname = cmd.__module__.split(".")[-1]
@@ -46,8 +61,10 @@ def _get_commands_from_module(module, inproject):
return d
-def _get_commands_from_entry_points(inproject, group="scrapy.commands"):
- cmds = {}
+def _get_commands_from_entry_points(
+ inproject: bool, group: str = "scrapy.commands"
+) -> dict[str, ScrapyCommand]:
+ cmds: dict[str, ScrapyCommand] = {}
if sys.version_info >= (3, 10):
eps = entry_points(group=group)
else:
@@ -57,11 +74,13 @@ def _get_commands_from_entry_points(inproject, group="scrapy.commands"):
if inspect.isclass(obj):
cmds[entry_point.name] = obj()
else:
- raise Exception(f"Invalid entry point {entry_point.name}")
+ raise ValueError(f"Invalid entry point {entry_point.name}")
return cmds
-def _get_commands_dict(settings, inproject):
+def _get_commands_dict(
+ settings: BaseSettings, inproject: bool
+) -> dict[str, ScrapyCommand]:
cmds = _get_commands_from_module("scrapy.commands", inproject)
cmds.update(_get_commands_from_entry_points(inproject))
cmds_module = settings["COMMANDS_MODULE"]
@@ -70,16 +89,17 @@ def _get_commands_dict(settings, inproject):
return cmds
-def _pop_command_name(argv):
+def _pop_command_name(argv: list[str]) -> str | None:
i = 0
for arg in argv[1:]:
if not arg.startswith("-"):
del argv[i]
return arg
i += 1
+ return None
-def _print_header(settings, inproject):
+def _print_header(settings: BaseSettings, inproject: bool) -> None:
version = scrapy.__version__
if inproject:
print(f"Scrapy {version} - active project: {settings['BOT_NAME']}\n")
@@ -88,7 +108,7 @@ def _print_header(settings, inproject):
print(f"Scrapy {version} - no active project\n")
-def _print_commands(settings, inproject):
+def _print_commands(settings: BaseSettings, inproject: bool) -> None:
_print_header(settings, inproject)
print("Usage:")
print(" scrapy [options] [args]\n")
@@ -103,13 +123,20 @@ def _print_commands(settings, inproject):
print('Use "scrapy -h" to see more info about a command')
-def _print_unknown_command(settings, cmdname, inproject):
+def _print_unknown_command(
+ settings: BaseSettings, cmdname: str, inproject: bool
+) -> None:
_print_header(settings, inproject)
print(f"Unknown command: {cmdname}\n")
print('Use "scrapy" to see available commands')
-def _run_print_help(parser, func, *a, **kw):
+def _run_print_help(
+ parser: argparse.ArgumentParser,
+ func: Callable[_P, None],
+ *a: _P.args,
+ **kw: _P.kwargs,
+) -> None:
try:
func(*a, **kw)
except UsageError as e:
@@ -120,7 +147,7 @@ def _run_print_help(parser, func, *a, **kw):
sys.exit(2)
-def execute(argv=None, settings=None):
+def execute(argv: list[str] | None = None, settings: Settings | None = None) -> None:
if argv is None:
argv = sys.argv
@@ -162,14 +189,16 @@ def execute(argv=None, settings=None):
sys.exit(cmd.exitcode)
-def _run_command(cmd, args, opts):
+def _run_command(cmd: ScrapyCommand, args: list[str], opts: argparse.Namespace) -> None:
if opts.profile:
_run_command_profiled(cmd, args, opts)
else:
cmd.run(args, opts)
-def _run_command_profiled(cmd, args, opts):
+def _run_command_profiled(
+ cmd: ScrapyCommand, args: list[str], opts: argparse.Namespace
+) -> None:
if opts.profile:
sys.stderr.write(f"scrapy: writing cProfile stats to {opts.profile!r}\n")
loc = locals()
diff --git a/scrapy/commands/__init__.py b/scrapy/commands/__init__.py
index 2aa569cdd..56199cc01 100644
--- a/scrapy/commands/__init__.py
+++ b/scrapy/commands/__init__.py
@@ -1,62 +1,70 @@
"""
Base class for Scrapy commands
"""
+
+from __future__ import annotations
+
import argparse
+import builtins
import os
from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import TYPE_CHECKING, Any
from twisted.python import failure
-from scrapy.crawler import CrawlerProcess
from scrapy.exceptions import UsageError
from scrapy.utils.conf import arglist_to_dict, feed_process_params_from_cli
+if TYPE_CHECKING:
+ from collections.abc import Iterable
+
+ from scrapy.crawler import Crawler, CrawlerProcess
+
class ScrapyCommand:
- requires_project = False
- crawler_process: Optional[CrawlerProcess] = None
+ requires_project: bool = False
+ crawler_process: CrawlerProcess | None = None
# default settings to be used for this command instead of global defaults
- default_settings: Dict[str, Any] = {}
+ default_settings: dict[str, Any] = {}
- exitcode = 0
+ exitcode: int = 0
def __init__(self) -> None:
self.settings: Any = None # set in scrapy.cmdline
- def set_crawler(self, crawler):
+ def set_crawler(self, crawler: Crawler) -> None:
if hasattr(self, "_crawler"):
raise RuntimeError("crawler already set")
- self._crawler = crawler
+ self._crawler: Crawler = crawler
- def syntax(self):
+ def syntax(self) -> str:
"""
Command syntax (preferably one-line). Do not include command name.
"""
return ""
- def short_desc(self):
+ def short_desc(self) -> str:
"""
A short description of the command
"""
return ""
- def long_desc(self):
+ def long_desc(self) -> str:
"""A long description of the command. Return short description when not
available. It cannot contain newlines since contents will be formatted
by optparser which removes newlines and wraps text.
"""
return self.short_desc()
- def help(self):
+ def help(self) -> str:
"""An extensive help for the command. It will be shown when using the
"help" command. It can contain newlines since no post-formatting will
be applied to its contents.
"""
return self.long_desc()
- def add_options(self, parser):
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
"""
Populate option parse with options available for this command
"""
@@ -91,7 +99,7 @@ class ScrapyCommand:
)
group.add_argument("--pdb", action="store_true", help="enable pdb on failure")
- def process_options(self, args, opts):
+ def process_options(self, args: list[str], opts: argparse.Namespace) -> None:
try:
self.settings.setdict(arglist_to_dict(opts.set), priority="cmdline")
except ValueError:
@@ -116,7 +124,7 @@ class ScrapyCommand:
if opts.pdb:
failure.startDebugMode()
- def run(self, args: List[str], opts: argparse.Namespace) -> None:
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
"""
Entry point for running commands
"""
@@ -128,8 +136,8 @@ class BaseRunSpiderCommand(ScrapyCommand):
Common class used to share functionality between the crawl, parse and runspider commands
"""
- def add_options(self, parser):
- ScrapyCommand.add_options(self, parser)
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument(
"-a",
dest="spargs",
@@ -154,15 +162,9 @@ class BaseRunSpiderCommand(ScrapyCommand):
help="dump scraped items into FILE, overwriting any existing file,"
" to define format set a colon at the end of the output URI (i.e. -O FILE:FORMAT)",
)
- parser.add_argument(
- "-t",
- "--output-format",
- metavar="FORMAT",
- help="format to use for dumping items",
- )
- def process_options(self, args, opts):
- ScrapyCommand.process_options(self, args, opts)
+ def process_options(self, args: list[str], opts: argparse.Namespace) -> None:
+ super().process_options(args, opts)
try:
opts.spargs = arglist_to_dict(opts.spargs)
except ValueError:
@@ -171,8 +173,7 @@ class BaseRunSpiderCommand(ScrapyCommand):
feeds = feed_process_params_from_cli(
self.settings,
opts.output,
- opts.output_format,
- opts.overwrite_output,
+ overwrite_output=opts.overwrite_output,
)
self.settings.set("FEEDS", feeds, priority="cmdline")
@@ -182,7 +183,13 @@ class ScrapyHelpFormatter(argparse.HelpFormatter):
Help Formatter for scrapy command line help messages.
"""
- def __init__(self, prog, indent_increment=2, max_help_position=24, width=None):
+ def __init__(
+ self,
+ prog: str,
+ indent_increment: int = 2,
+ max_help_position: int = 24,
+ width: int | None = None,
+ ):
super().__init__(
prog,
indent_increment=indent_increment,
@@ -190,11 +197,12 @@ class ScrapyHelpFormatter(argparse.HelpFormatter):
width=width,
)
- def _join_parts(self, part_strings):
- parts = self.format_part_strings(part_strings)
+ def _join_parts(self, part_strings: Iterable[str]) -> str:
+ # scrapy.commands.list shadows builtins.list
+ parts = self.format_part_strings(builtins.list(part_strings))
return super()._join_parts(parts)
- def format_part_strings(self, part_strings):
+ def format_part_strings(self, part_strings: list[str]) -> list[str]:
"""
Underline and title case command line help message headers.
"""
diff --git a/scrapy/commands/bench.py b/scrapy/commands/bench.py
index e1ccdc451..714bc38da 100644
--- a/scrapy/commands/bench.py
+++ b/scrapy/commands/bench.py
@@ -1,12 +1,22 @@
+from __future__ import annotations
+
import subprocess
import sys
import time
+from typing import TYPE_CHECKING, Any
from urllib.parse import urlencode
import scrapy
from scrapy.commands import ScrapyCommand
+from scrapy.http import Response, TextResponse
from scrapy.linkextractors import LinkExtractor
+if TYPE_CHECKING:
+ import argparse
+ from collections.abc import Iterable
+
+ from scrapy import Request
+
class Command(ScrapyCommand):
default_settings = {
@@ -15,24 +25,28 @@ class Command(ScrapyCommand):
"CLOSESPIDER_TIMEOUT": 10,
}
- def short_desc(self):
+ def short_desc(self) -> str:
return "Run quick benchmark test"
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
with _BenchServer():
+ assert self.crawler_process
self.crawler_process.crawl(_BenchSpider, total=100000)
self.crawler_process.start()
class _BenchServer:
- def __enter__(self):
+ def __enter__(self) -> None:
from scrapy.utils.test import get_testenv
pargs = [sys.executable, "-u", "-m", "scrapy.utils.benchserver"]
- self.proc = subprocess.Popen(pargs, stdout=subprocess.PIPE, env=get_testenv())
+ self.proc = subprocess.Popen( # noqa: S603
+ pargs, stdout=subprocess.PIPE, env=get_testenv()
+ )
+ assert self.proc.stdout
self.proc.stdout.readline()
- def __exit__(self, exc_type, exc_value, traceback):
+ def __exit__(self, exc_type, exc_value, traceback) -> None:
self.proc.kill()
self.proc.wait()
time.sleep(0.2)
@@ -47,11 +61,12 @@ class _BenchSpider(scrapy.Spider):
baseurl = "http://localhost:8998"
link_extractor = LinkExtractor()
- def start_requests(self):
+ def start_requests(self) -> Iterable[Request]:
qargs = {"total": self.total, "show": self.show}
url = f"{self.baseurl}?{urlencode(qargs, doseq=True)}"
return [scrapy.Request(url, dont_filter=True)]
- def parse(self, response):
+ def parse(self, response: Response) -> Any:
+ assert isinstance(Response, TextResponse)
for link in self.link_extractor.extract_links(response):
yield scrapy.Request(link.url, callback=self.parse)
diff --git a/scrapy/commands/check.py b/scrapy/commands/check.py
index de54ca4d3..1ce155da7 100644
--- a/scrapy/commands/check.py
+++ b/scrapy/commands/check.py
@@ -1,3 +1,4 @@
+import argparse
import time
from collections import defaultdict
from unittest import TextTestResult as _TextTestResult
@@ -10,7 +11,7 @@ from scrapy.utils.misc import load_object, set_environ
class TextTestResult(_TextTestResult):
- def printSummary(self, start, stop):
+ def printSummary(self, start: float, stop: float) -> None:
write = self.stream.write
writeln = self.stream.writeln
@@ -42,14 +43,14 @@ class Command(ScrapyCommand):
requires_project = True
default_settings = {"LOG_ENABLED": False}
- def syntax(self):
+ def syntax(self) -> str:
return "[options] "
- def short_desc(self):
+ def short_desc(self) -> str:
return "Check spider contracts"
- def add_options(self, parser):
- ScrapyCommand.add_options(self, parser)
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument(
"-l",
"--list",
@@ -66,7 +67,7 @@ class Command(ScrapyCommand):
help="print contract tests for all spiders",
)
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
# load contracts
contracts = build_component_list(self.settings.getwithbase("SPIDER_CONTRACTS"))
conman = ContractsManager(load_object(c) for c in contracts)
@@ -76,12 +77,13 @@ class Command(ScrapyCommand):
# contract requests
contract_reqs = defaultdict(list)
+ assert self.crawler_process
spider_loader = self.crawler_process.spider_loader
with set_environ(SCRAPY_CHECK="true"):
for spidername in args or spider_loader.list():
spidercls = spider_loader.load(spidername)
- spidercls.start_requests = lambda s: conman.from_spider(s, result)
+ spidercls.start_requests = lambda s: conman.from_spider(s, result) # type: ignore[assignment,method-assign,return-value]
tested_methods = conman.tested_methods_from_spidercls(spidercls)
if opts.list:
diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py
index 2f0f1c7b9..86d4cc41c 100644
--- a/scrapy/commands/crawl.py
+++ b/scrapy/commands/crawl.py
@@ -1,29 +1,39 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, cast
+
+from twisted.python.failure import Failure
+
from scrapy.commands import BaseRunSpiderCommand
from scrapy.exceptions import UsageError
+if TYPE_CHECKING:
+ import argparse
+
class Command(BaseRunSpiderCommand):
requires_project = True
- def syntax(self):
+ def syntax(self) -> str:
return "[options] "
- def short_desc(self):
+ def short_desc(self) -> str:
return "Run a spider"
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
if len(args) < 1:
- raise UsageError()
- elif len(args) > 1:
+ raise UsageError
+ if len(args) > 1:
raise UsageError(
"running 'scrapy crawl' with more than one spider is not supported"
)
spname = args[0]
+ assert self.crawler_process
crawl_defer = self.crawler_process.crawl(spname, **opts.spargs)
if getattr(crawl_defer, "result", None) is not None and issubclass(
- crawl_defer.result.type, Exception
+ cast(Failure, crawl_defer.result).type, Exception
):
self.exitcode = 1
else:
diff --git a/scrapy/commands/edit.py b/scrapy/commands/edit.py
index 03a8ed5c7..d153a5271 100644
--- a/scrapy/commands/edit.py
+++ b/scrapy/commands/edit.py
@@ -1,3 +1,4 @@
+import argparse
import os
import sys
@@ -9,32 +10,35 @@ class Command(ScrapyCommand):
requires_project = True
default_settings = {"LOG_ENABLED": False}
- def syntax(self):
+ def syntax(self) -> str:
return ""
- def short_desc(self):
+ def short_desc(self) -> str:
return "Edit spider"
- def long_desc(self):
+ def long_desc(self) -> str:
return (
"Edit a spider using the editor defined in the EDITOR environment"
" variable or else the EDITOR setting"
)
- def _err(self, msg):
+ def _err(self, msg: str) -> None:
sys.stderr.write(msg + os.linesep)
self.exitcode = 1
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
if len(args) != 1:
- raise UsageError()
+ raise UsageError
editor = self.settings["EDITOR"]
+ assert self.crawler_process
try:
spidercls = self.crawler_process.spider_loader.load(args[0])
except KeyError:
- return self._err(f"Spider not found: {args[0]}")
+ self._err(f"Spider not found: {args[0]}")
+ return
sfile = sys.modules[spidercls.__module__].__file__
+ assert sfile
sfile = sfile.replace(".pyc", ".py")
- self.exitcode = os.system(f'{editor} "{sfile}"')
+ self.exitcode = os.system(f'{editor} "{sfile}"') # noqa: S605
diff --git a/scrapy/commands/fetch.py b/scrapy/commands/fetch.py
index cdb7ad4ae..8a8d04ff6 100644
--- a/scrapy/commands/fetch.py
+++ b/scrapy/commands/fetch.py
@@ -1,34 +1,39 @@
+from __future__ import annotations
+
import sys
-from argparse import Namespace
-from typing import List, Type
+from typing import TYPE_CHECKING
from w3lib.url import is_url
-from scrapy import Spider
from scrapy.commands import ScrapyCommand
from scrapy.exceptions import UsageError
-from scrapy.http import Request
+from scrapy.http import Request, Response
from scrapy.utils.datatypes import SequenceExclude
from scrapy.utils.spider import DefaultSpider, spidercls_for_request
+if TYPE_CHECKING:
+ from argparse import ArgumentParser, Namespace
+
+ from scrapy import Spider
+
class Command(ScrapyCommand):
requires_project = False
- def syntax(self):
+ def syntax(self) -> str:
return "[options] "
- def short_desc(self):
+ def short_desc(self) -> str:
return "Fetch a URL using the Scrapy downloader"
- def long_desc(self):
+ def long_desc(self) -> str:
return (
"Fetch a URL using the Scrapy downloader and print its content"
" to stdout. You may want to use --nolog to disable logging"
)
- def add_options(self, parser):
- ScrapyCommand.add_options(self, parser)
+ def add_options(self, parser: ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument("--spider", dest="spider", help="use this spider")
parser.add_argument(
"--headers",
@@ -44,25 +49,26 @@ class Command(ScrapyCommand):
help="do not handle HTTP 3xx status codes and print response as-is",
)
- def _print_headers(self, headers, prefix):
+ def _print_headers(self, headers: dict[bytes, list[bytes]], prefix: bytes) -> None:
for key, values in headers.items():
for value in values:
self._print_bytes(prefix + b" " + key + b": " + value)
- def _print_response(self, response, opts):
+ def _print_response(self, response: Response, opts: Namespace) -> None:
if opts.headers:
+ assert response.request
self._print_headers(response.request.headers, b">")
print(">")
self._print_headers(response.headers, b"<")
else:
self._print_bytes(response.body)
- def _print_bytes(self, bytes_):
+ def _print_bytes(self, bytes_: bytes) -> None:
sys.stdout.buffer.write(bytes_ + b"\n")
- def run(self, args: List[str], opts: Namespace) -> None:
+ def run(self, args: list[str], opts: Namespace) -> None:
if len(args) != 1 or not is_url(args[0]):
- raise UsageError()
+ raise UsageError
request = Request(
args[0],
callback=self._print_response,
@@ -76,7 +82,7 @@ class Command(ScrapyCommand):
else:
request.meta["handle_httpstatus_all"] = True
- spidercls: Type[Spider] = DefaultSpider
+ spidercls: type[Spider] = DefaultSpider
assert self.crawler_process
spider_loader = self.crawler_process.spider_loader
if opts.spider:
diff --git a/scrapy/commands/genspider.py b/scrapy/commands/genspider.py
index 68cbe8ff6..d7dc104c2 100644
--- a/scrapy/commands/genspider.py
+++ b/scrapy/commands/genspider.py
@@ -1,9 +1,11 @@
+from __future__ import annotations
+
import os
import shutil
import string
from importlib import import_module
from pathlib import Path
-from typing import Optional, cast
+from typing import TYPE_CHECKING, Any, cast
from urllib.parse import urlparse
import scrapy
@@ -11,8 +13,11 @@ from scrapy.commands import ScrapyCommand
from scrapy.exceptions import UsageError
from scrapy.utils.template import render_templatefile, string_camelcase
+if TYPE_CHECKING:
+ import argparse
-def sanitize_module_name(module_name):
+
+def sanitize_module_name(module_name: str) -> str:
"""Sanitize the given module name, by replacing dashes and points
with underscores and prefixing it with a letter if it doesn't start
with one
@@ -23,7 +28,7 @@ def sanitize_module_name(module_name):
return module_name
-def extract_domain(url):
+def extract_domain(url: str) -> str:
"""Extract domain name from URL string"""
o = urlparse(url)
if o.scheme == "" and o.netloc == "":
@@ -31,7 +36,7 @@ def extract_domain(url):
return o.netloc
-def verify_url_scheme(url):
+def verify_url_scheme(url: str) -> str:
"""Check url for scheme and insert https if none found."""
parsed = urlparse(url)
if parsed.scheme == "" and parsed.netloc == "":
@@ -43,14 +48,14 @@ class Command(ScrapyCommand):
requires_project = False
default_settings = {"LOG_ENABLED": False}
- def syntax(self):
+ def syntax(self) -> str:
return "[options] "
- def short_desc(self):
+ def short_desc(self) -> str:
return "Generate new spider using pre-defined templates"
- def add_options(self, parser):
- ScrapyCommand.add_options(self, parser)
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument(
"-l",
"--list",
@@ -86,7 +91,7 @@ class Command(ScrapyCommand):
help="If the spider already exists, overwrite it with the template",
)
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
if opts.list:
self._list_templates()
return
@@ -96,7 +101,7 @@ class Command(ScrapyCommand):
print(template_file.read_text(encoding="utf-8"))
return
if len(args) != 2:
- raise UsageError()
+ raise UsageError
name, url = args[0:2]
url = verify_url_scheme(url)
@@ -113,23 +118,39 @@ class Command(ScrapyCommand):
if template_file:
self._genspider(module, name, url, opts.template, template_file)
if opts.edit:
- self.exitcode = os.system(f'scrapy edit "{name}"')
+ self.exitcode = os.system(f'scrapy edit "{name}"') # noqa: S605
- def _genspider(self, module, name, url, template_name, template_file):
- """Generate the spider module, based on the given template"""
+ def _generate_template_variables(
+ self,
+ module: str,
+ name: str,
+ url: str,
+ template_name: str,
+ ) -> dict[str, Any]:
capitalized_module = "".join(s.capitalize() for s in module.split("_"))
- domain = extract_domain(url)
- tvars = {
+ return {
"project_name": self.settings.get("BOT_NAME"),
"ProjectName": string_camelcase(self.settings.get("BOT_NAME")),
"module": module,
"name": name,
"url": url,
- "domain": domain,
+ "domain": extract_domain(url),
"classname": f"{capitalized_module}Spider",
}
+
+ def _genspider(
+ self,
+ module: str,
+ name: str,
+ url: str,
+ template_name: str,
+ template_file: str | os.PathLike,
+ ) -> None:
+ """Generate the spider module, based on the given template"""
+ tvars = self._generate_template_variables(module, name, url, template_name)
if self.settings.get("NEWSPIDER_MODULE"):
spiders_module = import_module(self.settings["NEWSPIDER_MODULE"])
+ assert spiders_module.__file__
spiders_dir = Path(spiders_module.__file__).parent.resolve()
else:
spiders_module = None
@@ -144,7 +165,7 @@ class Command(ScrapyCommand):
if spiders_module:
print(f"in module:\n {spiders_module.__name__}.{module}")
- def _find_template(self, template: str) -> Optional[Path]:
+ def _find_template(self, template: str) -> Path | None:
template_file = Path(self.templates_dir, f"{template}.tmpl")
if template_file.exists():
return template_file
@@ -152,7 +173,7 @@ class Command(ScrapyCommand):
print('Use "scrapy genspider --list" to see all available templates.')
return None
- def _list_templates(self):
+ def _list_templates(self) -> None:
print("Available templates:")
for file in sorted(Path(self.templates_dir).iterdir()):
if file.suffix == ".tmpl":
diff --git a/scrapy/commands/list.py b/scrapy/commands/list.py
index 2f5032360..3b2f127c2 100644
--- a/scrapy/commands/list.py
+++ b/scrapy/commands/list.py
@@ -1,13 +1,21 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
from scrapy.commands import ScrapyCommand
+if TYPE_CHECKING:
+ import argparse
+
class Command(ScrapyCommand):
requires_project = True
default_settings = {"LOG_ENABLED": False}
- def short_desc(self):
+ def short_desc(self) -> str:
return "List available spiders"
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
+ assert self.crawler_process
for s in sorted(self.crawler_process.spider_loader.list()):
print(s)
diff --git a/scrapy/commands/parse.py b/scrapy/commands/parse.py
index ac937e464..cc5c1350b 100644
--- a/scrapy/commands/parse.py
+++ b/scrapy/commands/parse.py
@@ -1,15 +1,18 @@
+from __future__ import annotations
+
+import functools
import inspect
import json
import logging
-from typing import Dict
+from typing import TYPE_CHECKING, Any, TypeVar, overload
from itemadapter import ItemAdapter, is_item
-from twisted.internet.defer import maybeDeferred
+from twisted.internet.defer import Deferred, maybeDeferred
from w3lib.url import is_url
from scrapy.commands import BaseRunSpiderCommand
from scrapy.exceptions import UsageError
-from scrapy.http import Request
+from scrapy.http import Request, Response
from scrapy.utils import display
from scrapy.utils.asyncgen import collect_asyncgen
from scrapy.utils.defer import aiter_errback, deferred_from_coro
@@ -17,26 +20,39 @@ from scrapy.utils.log import failure_to_exc_info
from scrapy.utils.misc import arg_to_iter
from scrapy.utils.spider import spidercls_for_request
+if TYPE_CHECKING:
+ import argparse
+ from collections.abc import AsyncGenerator, Coroutine, Iterable
+
+ from twisted.python.failure import Failure
+
+ from scrapy.http.request import CallbackT
+ from scrapy.spiders import Spider
+
+
logger = logging.getLogger(__name__)
+_T = TypeVar("_T")
+
class Command(BaseRunSpiderCommand):
requires_project = True
- spider = None
- items: Dict[int, list] = {}
- requests: Dict[int, list] = {}
+ spider: Spider | None = None
+ items: dict[int, list[Any]] = {}
+ requests: dict[int, list[Request]] = {}
+ spidercls: type[Spider] | None
first_response = None
- def syntax(self):
+ def syntax(self) -> str:
return "[options] "
- def short_desc(self):
+ def short_desc(self) -> str:
return "Parse URL (using its spider) and print the results"
- def add_options(self, parser):
- BaseRunSpiderCommand.add_options(self, parser)
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument(
"--spider",
dest="spider",
@@ -105,7 +121,7 @@ class Command(BaseRunSpiderCommand):
)
@property
- def max_level(self):
+ def max_level(self) -> int:
max_items, max_requests = 0, 0
if self.items:
max_items = max(self.items)
@@ -113,13 +129,21 @@ class Command(BaseRunSpiderCommand):
max_requests = max(self.requests)
return max(max_items, max_requests)
- def handle_exception(self, _failure):
+ def handle_exception(self, _failure: Failure) -> None:
logger.error(
"An error is caught while iterating the async iterable",
exc_info=failure_to_exc_info(_failure),
)
- def iterate_spider_output(self, result):
+ @overload
+ def iterate_spider_output(
+ self, result: AsyncGenerator[_T] | Coroutine[Any, Any, _T]
+ ) -> Deferred[_T]: ...
+
+ @overload
+ def iterate_spider_output(self, result: _T) -> Iterable[Any]: ...
+
+ def iterate_spider_output(self, result: Any) -> Iterable[Any] | Deferred[Any]:
if inspect.isasyncgen(result):
d = deferred_from_coro(
collect_asyncgen(aiter_errback(result, self.handle_exception))
@@ -132,15 +156,15 @@ class Command(BaseRunSpiderCommand):
return d
return arg_to_iter(deferred_from_coro(result))
- def add_items(self, lvl, new_items):
+ def add_items(self, lvl: int, new_items: list[Any]) -> None:
old_items = self.items.get(lvl, [])
self.items[lvl] = old_items + new_items
- def add_requests(self, lvl, new_reqs):
+ def add_requests(self, lvl: int, new_reqs: list[Request]) -> None:
old_reqs = self.requests.get(lvl, [])
self.requests[lvl] = old_reqs + new_reqs
- def print_items(self, lvl=None, colour=True):
+ def print_items(self, lvl: int | None = None, colour: bool = True) -> None:
if lvl is None:
items = [item for lst in self.items.values() for item in lst]
else:
@@ -149,7 +173,7 @@ class Command(BaseRunSpiderCommand):
print("# Scraped Items ", "-" * 60)
display.pprint([ItemAdapter(x).asdict() for x in items], colorize=colour)
- def print_requests(self, lvl=None, colour=True):
+ def print_requests(self, lvl: int | None = None, colour: bool = True) -> None:
if lvl is None:
if self.requests:
requests = self.requests[max(self.requests)]
@@ -161,7 +185,7 @@ class Command(BaseRunSpiderCommand):
print("# Requests ", "-" * 65)
display.pprint(requests, colorize=colour)
- def print_results(self, opts):
+ def print_results(self, opts: argparse.Namespace) -> None:
colour = not opts.nocolour
if opts.verbose:
@@ -178,7 +202,14 @@ class Command(BaseRunSpiderCommand):
if not opts.nolinks:
self.print_requests(colour=colour)
- def _get_items_and_requests(self, spider_output, opts, depth, spider, callback):
+ def _get_items_and_requests(
+ self,
+ spider_output: Iterable[Any],
+ opts: argparse.Namespace,
+ depth: int,
+ spider: Spider,
+ callback: CallbackT,
+ ) -> tuple[list[Any], list[Request], argparse.Namespace, int, Spider, CallbackT]:
items, requests = [], []
for x in spider_output:
if is_item(x):
@@ -187,14 +218,22 @@ class Command(BaseRunSpiderCommand):
requests.append(x)
return items, requests, opts, depth, spider, callback
- def run_callback(self, response, callback, cb_kwargs=None):
+ def run_callback(
+ self,
+ response: Response,
+ callback: CallbackT,
+ cb_kwargs: dict[str, Any] | None = None,
+ ) -> Deferred[Any]:
cb_kwargs = cb_kwargs or {}
- d = maybeDeferred(self.iterate_spider_output, callback(response, **cb_kwargs))
- return d
+ return maybeDeferred(
+ self.iterate_spider_output, callback(response, **cb_kwargs)
+ )
- def get_callback_from_rules(self, spider, response):
+ def get_callback_from_rules(
+ self, spider: Spider, response: Response
+ ) -> CallbackT | str | None:
if getattr(spider, "rules", None):
- for rule in spider.rules:
+ for rule in spider.rules: # type: ignore[attr-defined]
if rule.link_extractor.matches(response.url):
return rule.callback or "parse"
else:
@@ -203,8 +242,10 @@ class Command(BaseRunSpiderCommand):
"please specify a callback to use for parsing",
{"spider": spider.name},
)
+ return None
- def set_spidercls(self, url, opts):
+ def set_spidercls(self, url: str, opts: argparse.Namespace) -> None:
+ assert self.crawler_process
spider_loader = self.crawler_process.spider_loader
if opts.spider:
try:
@@ -218,13 +259,15 @@ class Command(BaseRunSpiderCommand):
if not self.spidercls:
logger.error("Unable to find spider for: %(url)s", {"url": url})
- def _start_requests(spider):
+ def _start_requests(spider: Spider) -> Iterable[Request]:
yield self.prepare_request(spider, Request(url), opts)
if self.spidercls:
- self.spidercls.start_requests = _start_requests
+ self.spidercls.start_requests = _start_requests # type: ignore[assignment,method-assign]
- def start_parsing(self, url, opts):
+ def start_parsing(self, url: str, opts: argparse.Namespace) -> None:
+ assert self.crawler_process
+ assert self.spidercls
self.crawler_process.crawl(self.spidercls, **opts.spargs)
self.pcrawler = list(self.crawler_process.crawlers)[0]
self.crawler_process.start()
@@ -232,7 +275,12 @@ class Command(BaseRunSpiderCommand):
if not self.first_response:
logger.error("No response downloaded for: %(url)s", {"url": url})
- def scraped_data(self, args):
+ def scraped_data(
+ self,
+ args: tuple[
+ list[Any], list[Request], argparse.Namespace, int, Spider, CallbackT
+ ],
+ ) -> list[Any]:
items, requests, opts, depth, spider, callback = args
if opts.pipelines:
itemproc = self.pcrawler.engine.scraper.itemproc
@@ -251,42 +299,53 @@ class Command(BaseRunSpiderCommand):
return scraped_data
- def prepare_request(self, spider, request, opts):
- def callback(response, **cb_kwargs):
+ def _get_callback(
+ self,
+ *,
+ spider: Spider,
+ opts: argparse.Namespace,
+ response: Response | None = None,
+ ) -> CallbackT:
+ cb: str | CallbackT | None = None
+ if response:
+ cb = response.meta["_callback"]
+ if not cb:
+ if opts.callback:
+ cb = opts.callback
+ elif response and opts.rules and self.first_response == response:
+ cb = self.get_callback_from_rules(spider, response)
+ if not cb:
+ raise ValueError(
+ f"Cannot find a rule that matches {response.url!r} in spider: "
+ f"{spider.name}"
+ )
+ else:
+ cb = "parse"
+
+ if not callable(cb):
+ assert cb is not None
+ cb_method = getattr(spider, cb, None)
+ if callable(cb_method):
+ cb = cb_method
+ else:
+ raise ValueError(
+ f"Cannot find callback {cb!r} in spider: {spider.name}"
+ )
+ assert callable(cb)
+ return cb
+
+ def prepare_request(
+ self, spider: Spider, request: Request, opts: argparse.Namespace
+ ) -> Request:
+ def callback(response: Response, **cb_kwargs: Any) -> Deferred[list[Any]]:
# memorize first request
if not self.first_response:
self.first_response = response
- # determine real callback
- cb = response.meta["_callback"]
- if not cb:
- if opts.callback:
- cb = opts.callback
- elif opts.rules and self.first_response == response:
- cb = self.get_callback_from_rules(spider, response)
-
- if not cb:
- logger.error(
- "Cannot find a rule that matches %(url)r in spider: %(spider)s",
- {"url": response.url, "spider": spider.name},
- )
- return
- else:
- cb = "parse"
-
- if not callable(cb):
- cb_method = getattr(spider, cb, None)
- if callable(cb_method):
- cb = cb_method
- else:
- logger.error(
- "Cannot find callback %(callback)r in spider: %(spider)s",
- {"callback": cb, "spider": spider.name},
- )
- return
+ cb = self._get_callback(spider=spider, opts=opts, response=response)
# parse items and requests
- depth = response.meta["_depth"]
+ depth: int = response.meta["_depth"]
d = self.run_callback(response, cb, cb_kwargs)
d.addCallback(self._get_items_and_requests, opts, depth, spider, callback)
@@ -303,16 +362,19 @@ class Command(BaseRunSpiderCommand):
request.meta["_depth"] = 1
request.meta["_callback"] = request.callback
+ if not request.callback and not opts.rules:
+ cb = self._get_callback(spider=spider, opts=opts)
+ functools.update_wrapper(callback, cb)
request.callback = callback
return request
- def process_options(self, args, opts):
- BaseRunSpiderCommand.process_options(self, args, opts)
+ def process_options(self, args: list[str], opts: argparse.Namespace) -> None:
+ super().process_options(args, opts)
self.process_request_meta(opts)
self.process_request_cb_kwargs(opts)
- def process_request_meta(self, opts):
+ def process_request_meta(self, opts: argparse.Namespace) -> None:
if opts.meta:
try:
opts.meta = json.loads(opts.meta)
@@ -323,7 +385,7 @@ class Command(BaseRunSpiderCommand):
print_help=False,
)
- def process_request_cb_kwargs(self, opts):
+ def process_request_cb_kwargs(self, opts: argparse.Namespace) -> None:
if opts.cbkwargs:
try:
opts.cbkwargs = json.loads(opts.cbkwargs)
@@ -334,12 +396,11 @@ class Command(BaseRunSpiderCommand):
print_help=False,
)
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
# parse arguments
if not len(args) == 1 or not is_url(args[0]):
- raise UsageError()
- else:
- url = args[0]
+ raise UsageError
+ url = args[0]
# prepare spidercls
self.set_spidercls(url, opts)
diff --git a/scrapy/commands/runspider.py b/scrapy/commands/runspider.py
index 58ed89a81..bf8e41020 100644
--- a/scrapy/commands/runspider.py
+++ b/scrapy/commands/runspider.py
@@ -1,16 +1,21 @@
+from __future__ import annotations
+
import sys
from importlib import import_module
-from os import PathLike
from pathlib import Path
-from types import ModuleType
-from typing import Union
+from typing import TYPE_CHECKING
from scrapy.commands import BaseRunSpiderCommand
from scrapy.exceptions import UsageError
from scrapy.utils.spider import iter_spider_classes
+if TYPE_CHECKING:
+ import argparse
+ from os import PathLike
+ from types import ModuleType
-def _import_file(filepath: Union[str, PathLike]) -> ModuleType:
+
+def _import_file(filepath: str | PathLike[str]) -> ModuleType:
abspath = Path(filepath).resolve()
if abspath.suffix not in (".py", ".pyw"):
raise ValueError(f"Not a Python source file: {abspath}")
@@ -27,18 +32,18 @@ class Command(BaseRunSpiderCommand):
requires_project = False
default_settings = {"SPIDER_LOADER_WARN_ONLY": True}
- def syntax(self):
+ def syntax(self) -> str:
return "[options] "
- def short_desc(self):
+ def short_desc(self) -> str:
return "Run a self-contained spider (without creating a project)"
- def long_desc(self):
+ def long_desc(self) -> str:
return "Run the spider defined in the given file"
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
if len(args) != 1:
- raise UsageError()
+ raise UsageError
filename = Path(args[0])
if not filename.exists():
raise UsageError(f"File not found: {filename}\n")
@@ -51,6 +56,7 @@ class Command(BaseRunSpiderCommand):
raise UsageError(f"No spider found in file: {filename}\n")
spidercls = spclasses.pop()
+ assert self.crawler_process
self.crawler_process.crawl(spidercls, **opts.spargs)
self.crawler_process.start()
diff --git a/scrapy/commands/settings.py b/scrapy/commands/settings.py
index 318187204..59f86b9a7 100644
--- a/scrapy/commands/settings.py
+++ b/scrapy/commands/settings.py
@@ -1,3 +1,4 @@
+import argparse
import json
from scrapy.commands import ScrapyCommand
@@ -8,14 +9,14 @@ class Command(ScrapyCommand):
requires_project = False
default_settings = {"LOG_ENABLED": False, "SPIDER_LOADER_WARN_ONLY": True}
- def syntax(self):
+ def syntax(self) -> str:
return "[options]"
- def short_desc(self):
+ def short_desc(self) -> str:
return "Get settings values"
- def add_options(self, parser):
- ScrapyCommand.add_options(self, parser)
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument(
"--get", dest="get", metavar="SETTING", help="print raw setting value"
)
@@ -44,7 +45,8 @@ class Command(ScrapyCommand):
help="print setting value, interpreted as a list",
)
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
+ assert self.crawler_process
settings = self.crawler_process.settings
if opts.get:
s = settings.get(opts.get)
diff --git a/scrapy/commands/shell.py b/scrapy/commands/shell.py
index 12e37babc..3047ae396 100644
--- a/scrapy/commands/shell.py
+++ b/scrapy/commands/shell.py
@@ -3,17 +3,23 @@ Scrapy Shell
See documentation in docs/topics/shell.rst
"""
-from argparse import Namespace
-from threading import Thread
-from typing import List, Type
-from scrapy import Spider
+from __future__ import annotations
+
+from threading import Thread
+from typing import TYPE_CHECKING, Any
+
from scrapy.commands import ScrapyCommand
from scrapy.http import Request
from scrapy.shell import Shell
from scrapy.utils.spider import DefaultSpider, spidercls_for_request
from scrapy.utils.url import guess_scheme
+if TYPE_CHECKING:
+ from argparse import ArgumentParser, Namespace
+
+ from scrapy import Spider
+
class Command(ScrapyCommand):
requires_project = False
@@ -23,20 +29,20 @@ class Command(ScrapyCommand):
"DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter",
}
- def syntax(self):
+ def syntax(self) -> str:
return "[url|file]"
- def short_desc(self):
+ def short_desc(self) -> str:
return "Interactive scraping console"
- def long_desc(self):
+ def long_desc(self) -> str:
return (
"Interactive console for scraping the given url or file. "
"Use ./file.html syntax or full path for local file."
)
- def add_options(self, parser):
- ScrapyCommand.add_options(self, parser)
+ def add_options(self, parser: ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument(
"-c",
dest="code",
@@ -51,13 +57,12 @@ class Command(ScrapyCommand):
help="do not handle HTTP 3xx status codes and print response as-is",
)
- def update_vars(self, vars):
+ def update_vars(self, vars: dict[str, Any]) -> None:
"""You can use this function to update the Scrapy objects that will be
available in the shell
"""
- pass
- def run(self, args: List[str], opts: Namespace) -> None:
+ def run(self, args: list[str], opts: Namespace) -> None:
url = args[0] if args else None
if url:
# first argument may be a local file
@@ -66,7 +71,7 @@ class Command(ScrapyCommand):
assert self.crawler_process
spider_loader = self.crawler_process.spider_loader
- spidercls: Type[Spider] = DefaultSpider
+ spidercls: type[Spider] = DefaultSpider
if opts.spider:
spidercls = spider_loader.load(opts.spider)
elif url:
@@ -87,7 +92,8 @@ class Command(ScrapyCommand):
shell = Shell(crawler, update_vars=self.update_vars, code=opts.code)
shell.start(url=url, redirect=not opts.no_redirect)
- def _start_crawler_thread(self):
+ def _start_crawler_thread(self) -> None:
+ assert self.crawler_process
t = Thread(
target=self.crawler_process.start,
kwargs={"stop_after_crawl": False, "install_signal_handlers": False},
diff --git a/scrapy/commands/startproject.py b/scrapy/commands/startproject.py
index fde609c6f..5cb73f0d2 100644
--- a/scrapy/commands/startproject.py
+++ b/scrapy/commands/startproject.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
import os
import re
import string
@@ -5,13 +7,17 @@ from importlib.util import find_spec
from pathlib import Path
from shutil import copy2, copystat, ignore_patterns, move
from stat import S_IWUSR as OWNER_WRITE_PERMISSION
+from typing import TYPE_CHECKING
import scrapy
from scrapy.commands import ScrapyCommand
from scrapy.exceptions import UsageError
from scrapy.utils.template import render_templatefile, string_camelcase
-TEMPLATES_TO_RENDER = (
+if TYPE_CHECKING:
+ import argparse
+
+TEMPLATES_TO_RENDER: tuple[tuple[str, ...], ...] = (
("scrapy.cfg",),
("${project_name}", "settings.py.tmpl"),
("${project_name}", "items.py.tmpl"),
@@ -22,7 +28,7 @@ TEMPLATES_TO_RENDER = (
IGNORE = ignore_patterns("*.pyc", "__pycache__", ".svn")
-def _make_writable(path):
+def _make_writable(path: str | os.PathLike) -> None:
current_permissions = os.stat(path).st_mode
os.chmod(path, current_permissions | OWNER_WRITE_PERMISSION)
@@ -31,14 +37,14 @@ class Command(ScrapyCommand):
requires_project = False
default_settings = {"LOG_ENABLED": False, "SPIDER_LOADER_WARN_ONLY": True}
- def syntax(self):
+ def syntax(self) -> str:
return " [project_dir]"
- def short_desc(self):
+ def short_desc(self) -> str:
return "Create new project"
- def _is_valid_name(self, project_name):
- def _module_exists(module_name):
+ def _is_valid_name(self, project_name: str) -> bool:
+ def _module_exists(module_name: str) -> bool:
spec = find_spec(module_name)
return spec is not None and spec.loader is not None
@@ -53,7 +59,7 @@ class Command(ScrapyCommand):
return True
return False
- def _copytree(self, src: Path, dst: Path):
+ def _copytree(self, src: Path, dst: Path) -> None:
"""
Since the original function always creates the directory, to resolve
the issue a new function had to be created. It's a simple copy and
@@ -84,9 +90,9 @@ class Command(ScrapyCommand):
copystat(src, dst)
_make_writable(dst)
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
if len(args) not in (1, 2):
- raise UsageError()
+ raise UsageError
project_name = args[0]
diff --git a/scrapy/commands/version.py b/scrapy/commands/version.py
index 47582866b..713a78ad9 100644
--- a/scrapy/commands/version.py
+++ b/scrapy/commands/version.py
@@ -1,19 +1,21 @@
+import argparse
+
import scrapy
from scrapy.commands import ScrapyCommand
-from scrapy.utils.versions import scrapy_components_versions
+from scrapy.utils.versions import get_versions
class Command(ScrapyCommand):
default_settings = {"LOG_ENABLED": False, "SPIDER_LOADER_WARN_ONLY": True}
- def syntax(self):
+ def syntax(self) -> str:
return "[-v]"
- def short_desc(self):
+ def short_desc(self) -> str:
return "Print Scrapy version"
- def add_options(self, parser):
- ScrapyCommand.add_options(self, parser)
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
+ super().add_options(parser)
parser.add_argument(
"--verbose",
"-v",
@@ -22,9 +24,9 @@ class Command(ScrapyCommand):
help="also display twisted/python/platform info (useful for bug reports)",
)
- def run(self, args, opts):
+ def run(self, args: list[str], opts: argparse.Namespace) -> None:
if opts.verbose:
- versions = scrapy_components_versions()
+ versions = get_versions()
width = max(len(n) for (n, _) in versions)
for name, version in versions:
print(f"{name:<{width}} : {version}")
diff --git a/scrapy/commands/view.py b/scrapy/commands/view.py
index ebdfa10a8..21679e3aa 100644
--- a/scrapy/commands/view.py
+++ b/scrapy/commands/view.py
@@ -1,21 +1,28 @@
import argparse
+import logging
from scrapy.commands import fetch
+from scrapy.http import Response, TextResponse
from scrapy.utils.response import open_in_browser
+logger = logging.getLogger(__name__)
+
class Command(fetch.Command):
- def short_desc(self):
+ def short_desc(self) -> str:
return "Open URL in browser, as seen by Scrapy"
- def long_desc(self):
+ def long_desc(self) -> str:
return (
"Fetch a URL using the Scrapy downloader and show its contents in a browser"
)
- def add_options(self, parser):
+ def add_options(self, parser: argparse.ArgumentParser) -> None:
super().add_options(parser)
parser.add_argument("--headers", help=argparse.SUPPRESS)
- def _print_response(self, response, opts):
+ def _print_response(self, response: Response, opts: argparse.Namespace) -> None:
+ if not isinstance(response, TextResponse):
+ logger.error("Cannot view a non-text response.")
+ return
open_in_browser(response)
diff --git a/scrapy/contracts/__init__.py b/scrapy/contracts/__init__.py
index d46eb7c51..3b4f932a0 100644
--- a/scrapy/contracts/__init__.py
+++ b/scrapy/contracts/__init__.py
@@ -1,32 +1,44 @@
+from __future__ import annotations
+
import re
import sys
+from collections.abc import AsyncGenerator, Iterable
from functools import wraps
from inspect import getmembers
from types import CoroutineType
-from typing import AsyncGenerator, Dict, Optional, Type
-from unittest import TestCase
+from typing import TYPE_CHECKING, Any, cast
+from unittest import TestCase, TestResult
-from scrapy.http import Request
+from scrapy.http import Request, Response
from scrapy.utils.python import get_spec
from scrapy.utils.spider import iterate_spider_output
+if TYPE_CHECKING:
+ from collections.abc import Callable
+
+ from twisted.python.failure import Failure
+
+ from scrapy import Spider
+
class Contract:
"""Abstract class for contracts"""
- request_cls: Optional[Type[Request]] = None
+ request_cls: type[Request] | None = None
+ name: str
- def __init__(self, method, *args):
+ def __init__(self, method: Callable, *args: Any):
self.testcase_pre = _create_testcase(method, f"@{self.name} pre-hook")
self.testcase_post = _create_testcase(method, f"@{self.name} post-hook")
- self.args = args
+ self.args: tuple[Any, ...] = args
- def add_pre_hook(self, request, results):
+ def add_pre_hook(self, request: Request, results: TestResult) -> Request:
if hasattr(self, "pre_process"):
cb = request.callback
+ assert cb is not None
@wraps(cb)
- def wrapper(response, **cb_kwargs):
+ def wrapper(response: Response, **cb_kwargs: Any) -> list[Any]:
try:
results.startTest(self.testcase_pre)
self.pre_process(response)
@@ -37,28 +49,26 @@ class Contract:
results.addError(self.testcase_pre, sys.exc_info())
else:
results.addSuccess(self.testcase_pre)
- finally:
- cb_result = cb(response, **cb_kwargs)
- if isinstance(cb_result, (AsyncGenerator, CoroutineType)):
- raise TypeError("Contracts don't support async callbacks")
- return list( # pylint: disable=return-in-finally
- iterate_spider_output(cb_result)
- )
+ cb_result = cb(response, **cb_kwargs)
+ if isinstance(cb_result, (AsyncGenerator, CoroutineType)):
+ raise TypeError("Contracts don't support async callbacks")
+ return list(cast(Iterable[Any], iterate_spider_output(cb_result)))
request.callback = wrapper
return request
- def add_post_hook(self, request, results):
+ def add_post_hook(self, request: Request, results: TestResult) -> Request:
if hasattr(self, "post_process"):
cb = request.callback
+ assert cb is not None
@wraps(cb)
- def wrapper(response, **cb_kwargs):
+ def wrapper(response: Response, **cb_kwargs: Any) -> list[Any]:
cb_result = cb(response, **cb_kwargs)
if isinstance(cb_result, (AsyncGenerator, CoroutineType)):
raise TypeError("Contracts don't support async callbacks")
- output = list(iterate_spider_output(cb_result))
+ output = list(cast(Iterable[Any], iterate_spider_output(cb_result)))
try:
results.startTest(self.testcase_post)
self.post_process(output)
@@ -69,25 +79,24 @@ class Contract:
results.addError(self.testcase_post, sys.exc_info())
else:
results.addSuccess(self.testcase_post)
- finally:
- return output # pylint: disable=return-in-finally
+ return output
request.callback = wrapper
return request
- def adjust_request_args(self, args):
+ def adjust_request_args(self, args: dict[str, Any]) -> dict[str, Any]:
return args
class ContractsManager:
- contracts: Dict[str, Contract] = {}
+ contracts: dict[str, type[Contract]] = {}
- def __init__(self, contracts):
+ def __init__(self, contracts: Iterable[type[Contract]]):
for contract in contracts:
self.contracts[contract.name] = contract
- def tested_methods_from_spidercls(self, spidercls):
+ def tested_methods_from_spidercls(self, spidercls: type[Spider]) -> list[str]:
is_method = re.compile(r"^\s*@", re.MULTILINE).search
methods = []
for key, value in getmembers(spidercls):
@@ -96,21 +105,25 @@ class ContractsManager:
return methods
- def extract_contracts(self, method):
- contracts = []
+ def extract_contracts(self, method: Callable) -> list[Contract]:
+ contracts: list[Contract] = []
+ assert method.__doc__ is not None
for line in method.__doc__.split("\n"):
line = line.strip()
if line.startswith("@"):
- name, args = re.match(r"@(\w+)\s*(.*)", line).groups()
+ m = re.match(r"@(\w+)\s*(.*)", line)
+ if m is None:
+ continue
+ name, args = m.groups()
args = re.split(r"\s+", args)
contracts.append(self.contracts[name](method, *args))
return contracts
- def from_spider(self, spider, results):
- requests = []
+ def from_spider(self, spider: Spider, results: TestResult) -> list[Request | None]:
+ requests: list[Request | None] = []
for method in self.tested_methods_from_spidercls(type(spider)):
bound_method = spider.__getattribute__(method)
try:
@@ -121,7 +134,7 @@ class ContractsManager:
return requests
- def from_method(self, method, results):
+ def from_method(self, method: Callable, results: TestResult) -> Request | None:
contracts = self.extract_contracts(method)
if contracts:
request_cls = Request
@@ -154,22 +167,26 @@ class ContractsManager:
self._clean_req(request, method, results)
return request
+ return None
- def _clean_req(self, request, method, results):
+ def _clean_req(
+ self, request: Request, method: Callable, results: TestResult
+ ) -> None:
"""stop the request from returning objects and records any errors"""
cb = request.callback
+ assert cb is not None
@wraps(cb)
- def cb_wrapper(response, **cb_kwargs):
+ def cb_wrapper(response: Response, **cb_kwargs: Any) -> None:
try:
output = cb(response, **cb_kwargs)
- output = list(iterate_spider_output(output))
+ output = list(cast(Iterable[Any], iterate_spider_output(output)))
except Exception:
case = _create_testcase(method, "callback")
results.addError(case, sys.exc_info())
- def eb_wrapper(failure):
+ def eb_wrapper(failure: Failure) -> None:
case = _create_testcase(method, "errback")
exc_info = failure.type, failure.value, failure.getTracebackObject()
results.addError(case, exc_info)
@@ -178,11 +195,11 @@ class ContractsManager:
request.errback = eb_wrapper
-def _create_testcase(method, desc):
- spider = method.__self__.name
+def _create_testcase(method: Callable, desc: str) -> TestCase:
+ spider = method.__self__.name # type: ignore[attr-defined]
class ContractTestCase(TestCase):
- def __str__(_self):
+ def __str__(_self) -> str:
return f"[{spider}] {method.__name__} ({desc})"
name = f"{spider}_{method.__name__}"
diff --git a/scrapy/contracts/default.py b/scrapy/contracts/default.py
index eac702cef..6f357ba20 100644
--- a/scrapy/contracts/default.py
+++ b/scrapy/contracts/default.py
@@ -1,4 +1,7 @@
+from __future__ import annotations
+
import json
+from typing import Any, Callable
from itemadapter import ItemAdapter, is_item
@@ -15,7 +18,7 @@ class UrlContract(Contract):
name = "url"
- def adjust_request_args(self, args):
+ def adjust_request_args(self, args: dict[str, Any]) -> dict[str, Any]:
args["url"] = self.args[0]
return args
@@ -29,11 +32,25 @@ class CallbackKeywordArgumentsContract(Contract):
name = "cb_kwargs"
- def adjust_request_args(self, args):
+ def adjust_request_args(self, args: dict[str, Any]) -> dict[str, Any]:
args["cb_kwargs"] = json.loads(" ".join(self.args))
return args
+class MetadataContract(Contract):
+ """Contract to set metadata arguments for the request.
+ The value should be JSON-encoded dictionary, e.g.:
+
+ @meta {"arg1": "some value"}
+ """
+
+ name = "meta"
+
+ def adjust_request_args(self, args: dict[str, Any]) -> dict[str, Any]:
+ args["meta"] = json.loads(" ".join(self.args))
+ return args
+
+
class ReturnsContract(Contract):
"""Contract to check the output of a callback
@@ -48,14 +65,14 @@ class ReturnsContract(Contract):
"""
name = "returns"
- object_type_verifiers = {
+ object_type_verifiers: dict[str | None, Callable[[Any], bool]] = {
"request": lambda x: isinstance(x, Request),
"requests": lambda x: isinstance(x, Request),
"item": is_item,
"items": is_item,
}
- def __init__(self, *args, **kwargs):
+ def __init__(self, *args: Any, **kwargs: Any):
super().__init__(*args, **kwargs)
if len(self.args) not in [1, 2, 3]:
@@ -66,16 +83,16 @@ class ReturnsContract(Contract):
self.obj_type_verifier = self.object_type_verifiers[self.obj_name]
try:
- self.min_bound = int(self.args[1])
+ self.min_bound: float = int(self.args[1])
except IndexError:
self.min_bound = 1
try:
- self.max_bound = int(self.args[2])
+ self.max_bound: float = int(self.args[2])
except IndexError:
self.max_bound = float("inf")
- def post_process(self, output):
+ def post_process(self, output: list[Any]) -> None:
occurrences = 0
for x in output:
if self.obj_type_verifier(x):
@@ -85,7 +102,7 @@ class ReturnsContract(Contract):
if not assertion:
if self.min_bound == self.max_bound:
- expected = self.min_bound
+ expected = str(self.min_bound)
else:
expected = f"{self.min_bound}..{self.max_bound}"
@@ -101,7 +118,7 @@ class ScrapesContract(Contract):
name = "scrapes"
- def post_process(self, output):
+ def post_process(self, output: list[Any]) -> None:
for x in output:
if is_item(x):
missing = [arg for arg in self.args if arg not in ItemAdapter(x)]
diff --git a/scrapy/core/downloader/__init__.py b/scrapy/core/downloader/__init__.py
index c84525160..78dc16df6 100644
--- a/scrapy/core/downloader/__init__.py
+++ b/scrapy/core/downloader/__init__.py
@@ -1,8 +1,11 @@
+from __future__ import annotations
+
import random
+import warnings
from collections import deque
from datetime import datetime
from time import time
-from typing import TYPE_CHECKING, Any, Deque, Dict, Set, Tuple, cast
+from typing import TYPE_CHECKING, Any, TypeVar, cast
from twisted.internet import task
from twisted.internet.defer import Deferred
@@ -10,28 +13,37 @@ from twisted.internet.defer import Deferred
from scrapy import Request, Spider, signals
from scrapy.core.downloader.handlers import DownloadHandlers
from scrapy.core.downloader.middleware import DownloaderMiddlewareManager
-from scrapy.http import Response
+from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.resolver import dnscache
-from scrapy.settings import BaseSettings
-from scrapy.signalmanager import SignalManager
from scrapy.utils.defer import mustbe_deferred
from scrapy.utils.httpobj import urlparse_cached
if TYPE_CHECKING:
from scrapy.crawler import Crawler
+ from scrapy.http import Response
+ from scrapy.settings import BaseSettings
+ from scrapy.signalmanager import SignalManager
+
+
+_T = TypeVar("_T")
class Slot:
"""Downloader slot"""
- def __init__(self, concurrency: int, delay: float, randomize_delay: bool):
+ def __init__(
+ self,
+ concurrency: int,
+ delay: float,
+ randomize_delay: bool,
+ ):
self.concurrency: int = concurrency
self.delay: float = delay
self.randomize_delay: bool = randomize_delay
- self.active: Set[Request] = set()
- self.queue: Deque[Tuple[Request, Deferred]] = deque()
- self.transferring: Set[Request] = set()
+ self.active: set[Request] = set()
+ self.queue: deque[tuple[Request, Deferred[Response]]] = deque()
+ self.transferring: set[Request] = set()
self.lastseen: float = 0
self.latercall = None
@@ -40,7 +52,7 @@ class Slot:
def download_delay(self) -> float:
if self.randomize_delay:
- return random.uniform(0.5 * self.delay, 1.5 * self.delay)
+ return random.uniform(0.5 * self.delay, 1.5 * self.delay) # noqa: S311
return self.delay
def close(self) -> None:
@@ -67,7 +79,7 @@ class Slot:
def _get_concurrency_delay(
concurrency: int, spider: Spider, settings: BaseSettings
-) -> Tuple[int, float]:
+) -> tuple[int, float]:
delay: float = settings.getfloat("DOWNLOAD_DELAY")
if hasattr(spider, "download_delay"):
delay = spider.download_delay
@@ -81,11 +93,11 @@ def _get_concurrency_delay(
class Downloader:
DOWNLOAD_SLOT = "download_slot"
- def __init__(self, crawler: "Crawler"):
+ def __init__(self, crawler: Crawler):
self.settings: BaseSettings = crawler.settings
self.signals: SignalManager = crawler.signals
- self.slots: Dict[str, Slot] = {}
- self.active: Set[Request] = set()
+ self.slots: dict[str, Slot] = {}
+ self.active: set[Request] = set()
self.handlers: DownloadHandlers = DownloadHandlers(crawler)
self.total_concurrency: int = self.settings.getint("CONCURRENT_REQUESTS")
self.domain_concurrency: int = self.settings.getint(
@@ -98,24 +110,26 @@ class Downloader:
)
self._slot_gc_loop: task.LoopingCall = task.LoopingCall(self._slot_gc)
self._slot_gc_loop.start(60)
- self.per_slot_settings: Dict[str, Dict[str, Any]] = self.settings.getdict(
+ self.per_slot_settings: dict[str, dict[str, Any]] = self.settings.getdict(
"DOWNLOAD_SLOTS", {}
)
- def fetch(self, request: Request, spider: Spider) -> Deferred:
- def _deactivate(response: Response) -> Response:
+ def fetch(self, request: Request, spider: Spider) -> Deferred[Response | Request]:
+ def _deactivate(response: _T) -> _T:
self.active.remove(request)
return response
self.active.add(request)
- dfd = self.middleware.download(self._enqueue_request, request, spider)
+ dfd: Deferred[Response | Request] = self.middleware.download(
+ self._enqueue_request, request, spider
+ )
return dfd.addBoth(_deactivate)
def needs_backout(self) -> bool:
return len(self.active) >= self.total_concurrency
- def _get_slot(self, request: Request, spider: Spider) -> Tuple[str, Slot]:
- key = self._get_slot_key(request, spider)
+ def _get_slot(self, request: Request, spider: Spider) -> tuple[str, Slot]:
+ key = self.get_slot_key(request)
if key not in self.slots:
slot_settings = self.per_slot_settings.get(key, {})
conc = (
@@ -132,7 +146,7 @@ class Downloader:
return key, self.slots[key]
- def _get_slot_key(self, request: Request, spider: Spider) -> str:
+ def get_slot_key(self, request: Request) -> str:
if self.DOWNLOAD_SLOT in request.meta:
return cast(str, request.meta[self.DOWNLOAD_SLOT])
@@ -142,7 +156,15 @@ class Downloader:
return key
- def _enqueue_request(self, request: Request, spider: Spider) -> Deferred:
+ def _get_slot_key(self, request: Request, spider: Spider | None) -> str:
+ warnings.warn(
+ "Use of this protected method is deprecated. Consider using its corresponding public method get_slot_key() instead.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ return self.get_slot_key(request)
+
+ def _enqueue_request(self, request: Request, spider: Spider) -> Deferred[Response]:
key, slot = self._get_slot(request, spider)
request.meta[self.DOWNLOAD_SLOT] = key
@@ -154,7 +176,7 @@ class Downloader:
self.signals.send_catch_log(
signal=signals.request_reached_downloader, request=request, spider=spider
)
- deferred: Deferred = Deferred().addBoth(_deactivate)
+ deferred: Deferred[Response] = Deferred().addBoth(_deactivate)
slot.queue.append((request, deferred))
self._process_queue(spider, slot)
return deferred
@@ -187,11 +209,15 @@ class Downloader:
self._process_queue(spider, slot)
break
- def _download(self, slot: Slot, request: Request, spider: Spider) -> Deferred:
+ def _download(
+ self, slot: Slot, request: Request, spider: Spider
+ ) -> Deferred[Response]:
# The order is very important for the following deferreds. Do not change!
# 1. Create the download deferred
- dfd = mustbe_deferred(self.handlers.download_request, request, spider)
+ dfd: Deferred[Response] = mustbe_deferred(
+ self.handlers.download_request, request, spider
+ )
# 2. Notify response_downloaded listeners about the recent download
# before querying queue for next request
@@ -212,7 +238,7 @@ class Downloader:
# middleware itself)
slot.transferring.add(request)
- def finish_transferring(_: Any) -> Any:
+ def finish_transferring(_: _T) -> _T:
slot.transferring.remove(request)
self._process_queue(spider, slot)
self.signals.send_catch_log(
diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py
index 909cc273f..d44c663bb 100644
--- a/scrapy/core/downloader/contextfactory.py
+++ b/scrapy/core/downloader/contextfactory.py
@@ -1,5 +1,7 @@
+from __future__ import annotations
+
import warnings
-from typing import TYPE_CHECKING, Any, List, Optional
+from typing import TYPE_CHECKING, Any
from OpenSSL import SSL
from twisted.internet._sslverify import _setAcceptableProtocols
@@ -19,12 +21,18 @@ from scrapy.core.downloader.tls import (
ScrapyClientTLSOptions,
openssl_methods,
)
-from scrapy.settings import BaseSettings
-from scrapy.utils.misc import create_instance, load_object
+from scrapy.exceptions import ScrapyDeprecationWarning
+from scrapy.utils.misc import build_from_crawler, load_object
if TYPE_CHECKING:
from twisted.internet._sslverify import ClientTLSOptions
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.settings import BaseSettings
+
@implementer(IPolicyForHTTPS)
class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
@@ -42,7 +50,7 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
self,
method: int = SSL.SSLv23_METHOD,
tls_verbose_logging: bool = False,
- tls_ciphers: Optional[str] = None,
+ tls_ciphers: str | None = None,
*args: Any,
**kwargs: Any,
):
@@ -62,11 +70,36 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
method: int = SSL.SSLv23_METHOD,
*args: Any,
**kwargs: Any,
- ):
+ ) -> Self:
+ warnings.warn(
+ f"{cls.__name__}.from_settings() is deprecated, use from_crawler() instead.",
+ category=ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ return cls._from_settings(settings, method, *args, **kwargs)
+
+ @classmethod
+ def from_crawler(
+ cls,
+ crawler: Crawler,
+ method: int = SSL.SSLv23_METHOD,
+ *args: Any,
+ **kwargs: Any,
+ ) -> Self:
+ return cls._from_settings(crawler.settings, method, *args, **kwargs)
+
+ @classmethod
+ def _from_settings(
+ cls,
+ settings: BaseSettings,
+ method: int = SSL.SSLv23_METHOD,
+ *args: Any,
+ **kwargs: Any,
+ ) -> Self:
tls_verbose_logging: bool = settings.getbool(
"DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING"
)
- tls_ciphers: Optional[str] = settings["DOWNLOADER_CLIENT_TLS_CIPHERS"]
+ tls_ciphers: str | None = settings["DOWNLOADER_CLIENT_TLS_CIPHERS"]
return cls( # type: ignore[misc]
method=method,
tls_verbose_logging=tls_verbose_logging,
@@ -78,18 +111,9 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
def getCertificateOptions(self) -> CertificateOptions:
# setting verify=True will require you to provide CAs
# to verify against; in other words: it's not that simple
-
- # backward-compatible SSL/TLS method:
- #
- # * this will respect `method` attribute in often recommended
- # `ScrapyClientContextFactory` subclass
- # (https://github.com/scrapy/scrapy/issues/1429#issuecomment-131782133)
- #
- # * getattr() for `_ssl_method` attribute for context factories
- # not calling super().__init__
return CertificateOptions(
verify=False,
- method=getattr(self, "method", getattr(self, "_ssl_method", None)),
+ method=self._ssl_method,
fixBrokenPeers=True,
acceptableCiphers=self.tls_ciphers,
)
@@ -97,11 +121,11 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
# kept for old-style HTTP/1.0 downloader context twisted calls,
# e.g. connectSSL()
def getContext(self, hostname: Any = None, port: Any = None) -> SSL.Context:
- ctx = self.getCertificateOptions().getContext()
+ ctx: SSL.Context = self.getCertificateOptions().getContext()
ctx.set_options(0x4) # OP_LEGACY_SERVER_CONNECT
return ctx
- def creatorForNetloc(self, hostname: bytes, port: int) -> "ClientTLSOptions":
+ def creatorForNetloc(self, hostname: bytes, port: int) -> ClientTLSOptions:
return ScrapyClientTLSOptions(
hostname.decode("ascii"),
self.getContext(),
@@ -128,7 +152,7 @@ class BrowserLikeContextFactory(ScrapyClientContextFactory):
``SSLv23_METHOD``) which allows TLS protocol negotiation.
"""
- def creatorForNetloc(self, hostname: bytes, port: int) -> "ClientTLSOptions":
+ def creatorForNetloc(self, hostname: bytes, port: int) -> ClientTLSOptions:
# trustRoot set to platformTrust() will use the platform's root CAs.
#
# This means that a website like https://www.cacert.org will be rejected
@@ -147,36 +171,36 @@ class AcceptableProtocolsContextFactory:
negotiation.
"""
- def __init__(self, context_factory: Any, acceptable_protocols: List[bytes]):
+ def __init__(self, context_factory: Any, acceptable_protocols: list[bytes]):
verifyObject(IPolicyForHTTPS, context_factory)
self._wrapped_context_factory: Any = context_factory
- self._acceptable_protocols: List[bytes] = acceptable_protocols
+ self._acceptable_protocols: list[bytes] = acceptable_protocols
- def creatorForNetloc(self, hostname: bytes, port: int) -> "ClientTLSOptions":
- options: "ClientTLSOptions" = self._wrapped_context_factory.creatorForNetloc(
+ def creatorForNetloc(self, hostname: bytes, port: int) -> ClientTLSOptions:
+ options: ClientTLSOptions = self._wrapped_context_factory.creatorForNetloc(
hostname, port
)
_setAcceptableProtocols(options._ctx, self._acceptable_protocols)
return options
-def load_context_factory_from_settings(settings, crawler):
+def load_context_factory_from_settings(
+ settings: BaseSettings, crawler: Crawler
+) -> IPolicyForHTTPS:
ssl_method = openssl_methods[settings.get("DOWNLOADER_CLIENT_TLS_METHOD")]
context_factory_cls = load_object(settings["DOWNLOADER_CLIENTCONTEXTFACTORY"])
# try method-aware context factory
try:
- context_factory = create_instance(
- objcls=context_factory_cls,
- settings=settings,
- crawler=crawler,
+ context_factory = build_from_crawler(
+ context_factory_cls,
+ crawler,
method=ssl_method,
)
except TypeError:
# use context factory defaults
- context_factory = create_instance(
- objcls=context_factory_cls,
- settings=settings,
- crawler=crawler,
+ context_factory = build_from_crawler(
+ context_factory_cls,
+ crawler,
)
msg = (
f"{settings['DOWNLOADER_CLIENTCONTEXTFACTORY']} does not accept "
diff --git a/scrapy/core/downloader/handlers/__init__.py b/scrapy/core/downloader/handlers/__init__.py
index 6a211aafa..7f3da67eb 100644
--- a/scrapy/core/downloader/handlers/__init__.py
+++ b/scrapy/core/downloader/handlers/__init__.py
@@ -1,33 +1,51 @@
"""Download handlers for different schemes"""
+from __future__ import annotations
+
import logging
-from typing import TYPE_CHECKING, Any, Callable, Dict, Generator, Union, cast
+from typing import TYPE_CHECKING, Any, Protocol, cast
from twisted.internet import defer
-from twisted.internet.defer import Deferred
from scrapy import Request, Spider, signals
from scrapy.exceptions import NotConfigured, NotSupported
from scrapy.utils.httpobj import urlparse_cached
-from scrapy.utils.misc import create_instance, load_object
+from scrapy.utils.misc import build_from_crawler, load_object
from scrapy.utils.python import without_none_values
if TYPE_CHECKING:
+ from collections.abc import Callable, Generator
+
+ from twisted.internet.defer import Deferred
+
from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
logger = logging.getLogger(__name__)
+class DownloadHandlerProtocol(Protocol):
+ def download_request(
+ self, request: Request, spider: Spider
+ ) -> Deferred[Response]: ...
+
+
class DownloadHandlers:
- def __init__(self, crawler: "Crawler"):
- self._crawler: "Crawler" = crawler
- self._schemes: Dict[
- str, Union[str, Callable]
- ] = {} # stores acceptable schemes on instancing
- self._handlers: Dict[str, Any] = {} # stores instanced handlers for schemes
- self._notconfigured: Dict[str, str] = {} # remembers failed handlers
- handlers: Dict[str, Union[str, Callable]] = without_none_values(
- crawler.settings.getwithbase("DOWNLOAD_HANDLERS")
+ def __init__(self, crawler: Crawler):
+ self._crawler: Crawler = crawler
+ self._schemes: dict[str, str | Callable[..., Any]] = (
+ {}
+ ) # stores acceptable schemes on instancing
+ self._handlers: dict[str, DownloadHandlerProtocol] = (
+ {}
+ ) # stores instanced handlers for schemes
+ self._notconfigured: dict[str, str] = {} # remembers failed handlers
+ handlers: dict[str, str | Callable[..., Any]] = without_none_values(
+ cast(
+ "dict[str, str | Callable[..., Any]]",
+ crawler.settings.getwithbase("DOWNLOAD_HANDLERS"),
+ )
)
for scheme, clspath in handlers.items():
self._schemes[scheme] = clspath
@@ -35,7 +53,7 @@ class DownloadHandlers:
crawler.signals.connect(self._close, signals.engine_stopped)
- def _get_handler(self, scheme: str) -> Any:
+ def _get_handler(self, scheme: str) -> DownloadHandlerProtocol | None:
"""Lazy-load the downloadhandler for a scheme
only on the first request for that scheme.
"""
@@ -49,16 +67,17 @@ class DownloadHandlers:
return self._load_handler(scheme)
- def _load_handler(self, scheme: str, skip_lazy: bool = False) -> Any:
+ def _load_handler(
+ self, scheme: str, skip_lazy: bool = False
+ ) -> DownloadHandlerProtocol | None:
path = self._schemes[scheme]
try:
- dhcls = load_object(path)
+ dhcls: type[DownloadHandlerProtocol] = load_object(path)
if skip_lazy and getattr(dhcls, "lazy", True):
return None
- dh = create_instance(
- objcls=dhcls,
- settings=self._crawler.settings,
- crawler=self._crawler,
+ dh = build_from_crawler(
+ dhcls,
+ self._crawler,
)
except NotConfigured as ex:
self._notconfigured[scheme] = str(ex)
@@ -72,21 +91,20 @@ class DownloadHandlers:
)
self._notconfigured[scheme] = str(ex)
return None
- else:
- self._handlers[scheme] = dh
- return dh
+ self._handlers[scheme] = dh
+ return dh
- def download_request(self, request: Request, spider: Spider) -> Deferred:
+ def download_request(self, request: Request, spider: Spider) -> Deferred[Response]:
scheme = urlparse_cached(request).scheme
handler = self._get_handler(scheme)
if not handler:
raise NotSupported(
f"Unsupported URL scheme '{scheme}': {self._notconfigured[scheme]}"
)
- return cast(Deferred, handler.download_request(request, spider))
+ return handler.download_request(request, spider)
@defer.inlineCallbacks
- def _close(self, *_a: Any, **_kw: Any) -> Generator[Deferred, Any, None]:
+ def _close(self, *_a: Any, **_kw: Any) -> Generator[Deferred[Any], Any, None]:
for dh in self._handlers.values():
if hasattr(dh, "close"):
yield dh.close()
diff --git a/scrapy/core/downloader/handlers/datauri.py b/scrapy/core/downloader/handlers/datauri.py
index a7ae56a85..b3f286d87 100644
--- a/scrapy/core/downloader/handlers/datauri.py
+++ b/scrapy/core/downloader/handlers/datauri.py
@@ -1,12 +1,16 @@
-from typing import Any, Dict
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
from w3lib.url import parse_data_uri
-from scrapy import Request, Spider
from scrapy.http import Response, TextResponse
from scrapy.responsetypes import responsetypes
from scrapy.utils.decorators import defers
+if TYPE_CHECKING:
+ from scrapy import Request, Spider
+
class DataURIDownloadHandler:
lazy = False
@@ -16,7 +20,7 @@ class DataURIDownloadHandler:
uri = parse_data_uri(request.url)
respcls = responsetypes.from_mimetype(uri.media_type)
- resp_kwargs: Dict[str, Any] = {}
+ resp_kwargs: dict[str, Any] = {}
if issubclass(respcls, TextResponse) and uri.media_type.split("/")[0] == "text":
charset = uri.media_type_parameters.get("charset")
resp_kwargs["encoding"] = charset
diff --git a/scrapy/core/downloader/handlers/file.py b/scrapy/core/downloader/handlers/file.py
index 4824167da..d55c516f0 100644
--- a/scrapy/core/downloader/handlers/file.py
+++ b/scrapy/core/downloader/handlers/file.py
@@ -1,16 +1,23 @@
+from __future__ import annotations
+
from pathlib import Path
+from typing import TYPE_CHECKING
from w3lib.url import file_uri_to_path
from scrapy.responsetypes import responsetypes
from scrapy.utils.decorators import defers
+if TYPE_CHECKING:
+ from scrapy import Request, Spider
+ from scrapy.http import Response
+
class FileDownloadHandler:
lazy = False
@defers
- def download_request(self, request, spider):
+ def download_request(self, request: Request, spider: Spider) -> Response:
filepath = file_uri_to_path(request.url)
body = Path(filepath).read_bytes()
respcls = responsetypes.from_args(filename=filepath, body=body)
diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py
index 4081545ce..598659b4d 100644
--- a/scrapy/core/downloader/handlers/ftp.py
+++ b/scrapy/core/downloader/handlers/ftp.py
@@ -28,8 +28,11 @@ In case of status 200 request, response.headers will come with two keys:
'Size' - with size of the downloaded data
"""
+from __future__ import annotations
+
import re
from io import BytesIO
+from typing import TYPE_CHECKING, Any, BinaryIO
from urllib.parse import unquote
from twisted.internet.protocol import ClientCreator, Protocol
@@ -40,23 +43,37 @@ from scrapy.responsetypes import responsetypes
from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.python import to_bytes
+if TYPE_CHECKING:
+ from twisted.internet.defer import Deferred
+ from twisted.python.failure import Failure
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.settings import BaseSettings
+
class ReceivedDataProtocol(Protocol):
- def __init__(self, filename=None):
- self.__filename = filename
- self.body = open(filename, "wb") if filename else BytesIO()
- self.size = 0
+ def __init__(self, filename: str | None = None):
+ self.__filename: str | None = filename
+ self.body: BinaryIO = open(filename, "wb") if filename else BytesIO()
+ self.size: int = 0
- def dataReceived(self, data):
+ def dataReceived(self, data: bytes) -> None:
self.body.write(data)
self.size += len(data)
@property
- def filename(self):
+ def filename(self) -> str | None:
return self.__filename
- def close(self):
- self.body.close() if self.filename else self.body.seek(0)
+ def close(self) -> None:
+ if self.filename:
+ self.body.close()
+ else:
+ self.body.seek(0)
_CODE_RE = re.compile(r"\d+")
@@ -65,21 +82,21 @@ _CODE_RE = re.compile(r"\d+")
class FTPDownloadHandler:
lazy = False
- CODE_MAPPING = {
+ CODE_MAPPING: dict[str, int] = {
"550": 404,
"default": 503,
}
- def __init__(self, settings):
+ def __init__(self, settings: BaseSettings):
self.default_user = settings["FTP_USER"]
self.default_password = settings["FTP_PASSWORD"]
self.passive_mode = settings["FTP_PASSIVE_MODE"]
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler.settings)
- def download_request(self, request, spider):
+ def download_request(self, request: Request, spider: Spider) -> Deferred[Response]:
from twisted.internet import reactor
parsed_url = urlparse_cached(request)
@@ -91,28 +108,33 @@ class FTPDownloadHandler:
creator = ClientCreator(
reactor, FTPClient, user, password, passive=passive_mode
)
- dfd = creator.connectTCP(parsed_url.hostname, parsed_url.port or 21)
+ dfd: Deferred[FTPClient] = creator.connectTCP(
+ parsed_url.hostname, parsed_url.port or 21
+ )
return dfd.addCallback(self.gotClient, request, unquote(parsed_url.path))
- def gotClient(self, client, request, filepath):
+ def gotClient(
+ self, client: FTPClient, request: Request, filepath: str
+ ) -> Deferred[Response]:
self.client = client
protocol = ReceivedDataProtocol(request.meta.get("ftp_local_filename"))
- return client.retrieveFile(filepath, protocol).addCallbacks(
- callback=self._build_response,
- callbackArgs=(request, protocol),
- errback=self._failed,
- errbackArgs=(request,),
- )
+ d = client.retrieveFile(filepath, protocol)
+ d.addCallback(self._build_response, request, protocol)
+ d.addErrback(self._failed, request)
+ return d
- def _build_response(self, result, request, protocol):
+ def _build_response(
+ self, result: Any, request: Request, protocol: ReceivedDataProtocol
+ ) -> Response:
self.result = result
protocol.close()
headers = {"local filename": protocol.filename or "", "size": protocol.size}
body = to_bytes(protocol.filename or protocol.body.read())
respcls = responsetypes.from_args(url=request.url, body=body)
- return respcls(url=request.url, status=200, body=body, headers=headers)
+ # hints for Headers-related types may need to be fixed to not use AnyStr
+ return respcls(url=request.url, status=200, body=body, headers=headers) # type: ignore[arg-type]
- def _failed(self, result, request):
+ def _failed(self, result: Failure, request: Request) -> Response:
message = result.getErrorMessage()
if result.type == CommandFailed:
m = _CODE_RE.search(message)
@@ -122,4 +144,5 @@ class FTPDownloadHandler:
return Response(
url=request.url, status=httpcode, body=to_bytes(message)
)
+ assert result.type
raise result.type(result.value)
diff --git a/scrapy/core/downloader/handlers/http10.py b/scrapy/core/downloader/handlers/http10.py
index 6c1dac4a5..58f7ad577 100644
--- a/scrapy/core/downloader/handlers/http10.py
+++ b/scrapy/core/downloader/handlers/http10.py
@@ -1,39 +1,58 @@
-"""Download handlers for http and https schemes
-"""
-from scrapy.utils.misc import create_instance, load_object
+"""Download handlers for http and https schemes"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from scrapy.utils.misc import build_from_crawler, load_object
from scrapy.utils.python import to_unicode
+if TYPE_CHECKING:
+ from twisted.internet.defer import Deferred
+ from twisted.internet.interfaces import IConnector
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy import Request, Spider
+ from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
+ from scrapy.core.downloader.webclient import ScrapyHTTPClientFactory
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+ from scrapy.settings import BaseSettings
+
class HTTP10DownloadHandler:
lazy = False
- def __init__(self, settings, crawler=None):
- self.HTTPClientFactory = load_object(settings["DOWNLOADER_HTTPCLIENTFACTORY"])
- self.ClientContextFactory = load_object(
+ def __init__(self, settings: BaseSettings, crawler: Crawler):
+ self.HTTPClientFactory: type[ScrapyHTTPClientFactory] = load_object(
+ settings["DOWNLOADER_HTTPCLIENTFACTORY"]
+ )
+ self.ClientContextFactory: type[ScrapyClientContextFactory] = load_object(
settings["DOWNLOADER_CLIENTCONTEXTFACTORY"]
)
- self._settings = settings
- self._crawler = crawler
+ self._settings: BaseSettings = settings
+ self._crawler: Crawler = crawler
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler.settings, crawler)
- def download_request(self, request, spider):
+ def download_request(self, request: Request, spider: Spider) -> Deferred[Response]:
"""Return a deferred for the HTTP download"""
factory = self.HTTPClientFactory(request)
self._connect(factory)
return factory.deferred
- def _connect(self, factory):
+ def _connect(self, factory: ScrapyHTTPClientFactory) -> IConnector:
from twisted.internet import reactor
host, port = to_unicode(factory.host), factory.port
if factory.scheme == b"https":
- client_context_factory = create_instance(
- objcls=self.ClientContextFactory,
- settings=self._settings,
- crawler=self._crawler,
+ client_context_factory = build_from_crawler(
+ self.ClientContextFactory,
+ self._crawler,
)
return reactor.connectSSL(host, port, factory, client_context_factory)
return reactor.connectTCP(host, port, factory)
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
index c3704de3d..9f65794fe 100644
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -1,16 +1,21 @@
"""Download handlers for http and https schemes"""
+from __future__ import annotations
+
import ipaddress
import logging
import re
from contextlib import suppress
from io import BytesIO
from time import time
+from typing import TYPE_CHECKING, Any, TypedDict, TypeVar
from urllib.parse import urldefrag, urlunparse
-from twisted.internet import defer, protocol, ssl
+from twisted.internet import ssl
+from twisted.internet.defer import CancelledError, Deferred, succeed
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.internet.error import TimeoutError
+from twisted.internet.protocol import Factory, Protocol, connectionDone
from twisted.python.failure import Failure
from twisted.web.client import (
URI,
@@ -19,47 +24,72 @@ from twisted.web.client import (
ResponseDone,
ResponseFailed,
)
+from twisted.web.client import Response as TxResponse
from twisted.web.http import PotentialDataLoss, _DataLoss
from twisted.web.http_headers import Headers as TxHeaders
-from twisted.web.iweb import UNKNOWN_LENGTH, IBodyProducer
+from twisted.web.iweb import UNKNOWN_LENGTH, IBodyProducer, IPolicyForHTTPS
from zope.interface import implementer
-from scrapy import signals
+from scrapy import Request, Spider, signals
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
from scrapy.core.downloader.webclient import _parse
from scrapy.exceptions import StopDownload
-from scrapy.http import Headers
+from scrapy.http import Headers, Response
from scrapy.responsetypes import responsetypes
from scrapy.utils.python import to_bytes, to_unicode
+if TYPE_CHECKING:
+ from twisted.internet.base import ReactorBase
+ from twisted.internet.interfaces import IConsumer
+
+ # typing.NotRequired and typing.Self require Python 3.11
+ from typing_extensions import NotRequired, Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.settings import BaseSettings
+
+
logger = logging.getLogger(__name__)
+_T = TypeVar("_T")
+
+
+class _ResultT(TypedDict):
+ txresponse: TxResponse
+ body: bytes
+ flags: list[str] | None
+ certificate: ssl.Certificate | None
+ ip_address: ipaddress.IPv4Address | ipaddress.IPv6Address | None
+ failure: NotRequired[Failure | None]
+
class HTTP11DownloadHandler:
lazy = False
- def __init__(self, settings, crawler=None):
+ def __init__(self, settings: BaseSettings, crawler: Crawler):
self._crawler = crawler
from twisted.internet import reactor
- self._pool = HTTPConnectionPool(reactor, persistent=True)
+ self._pool: HTTPConnectionPool = HTTPConnectionPool(reactor, persistent=True)
self._pool.maxPersistentPerHost = settings.getint(
"CONCURRENT_REQUESTS_PER_DOMAIN"
)
self._pool._factory.noisy = False
- self._contextFactory = load_context_factory_from_settings(settings, crawler)
- self._default_maxsize = settings.getint("DOWNLOAD_MAXSIZE")
- self._default_warnsize = settings.getint("DOWNLOAD_WARNSIZE")
- self._fail_on_dataloss = settings.getbool("DOWNLOAD_FAIL_ON_DATALOSS")
- self._disconnect_timeout = 1
+ self._contextFactory: IPolicyForHTTPS = load_context_factory_from_settings(
+ settings, crawler
+ )
+ self._default_maxsize: int = settings.getint("DOWNLOAD_MAXSIZE")
+ self._default_warnsize: int = settings.getint("DOWNLOAD_WARNSIZE")
+ self._fail_on_dataloss: bool = settings.getbool("DOWNLOAD_FAIL_ON_DATALOSS")
+ self._disconnect_timeout: int = 1
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler.settings, crawler)
- def download_request(self, request, spider):
+ def download_request(self, request: Request, spider: Spider) -> Deferred[Response]:
"""Return a deferred for the HTTP download"""
agent = ScrapyAgent(
contextFactory=self._contextFactory,
@@ -71,10 +101,10 @@ class HTTP11DownloadHandler:
)
return agent.download_request(request)
- def close(self):
+ def close(self) -> Deferred[None]:
from twisted.internet import reactor
- d = self._pool.closeCachedConnections()
+ d: Deferred[None] = self._pool.closeCachedConnections()
# closeCachedConnections will hang on network or server issues, so
# we'll manually timeout the deferred.
#
@@ -85,7 +115,7 @@ class HTTP11DownloadHandler:
# issue a callback after `_disconnect_timeout` seconds.
delayed_call = reactor.callLater(self._disconnect_timeout, d.callback, [])
- def cancel_delayed_call(result):
+ def cancel_delayed_call(result: _T) -> _T:
if delayed_call.active():
delayed_call.cancel()
return result
@@ -115,39 +145,41 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
def __init__(
self,
- reactor,
- host,
- port,
- proxyConf,
- contextFactory,
- timeout=30,
- bindAddress=None,
+ reactor: ReactorBase,
+ host: str,
+ port: int,
+ proxyConf: tuple[str, int, bytes | None],
+ contextFactory: IPolicyForHTTPS,
+ timeout: float = 30,
+ bindAddress: tuple[str, int] | None = None,
):
proxyHost, proxyPort, self._proxyAuthHeader = proxyConf
super().__init__(reactor, proxyHost, proxyPort, timeout, bindAddress)
- self._tunnelReadyDeferred = defer.Deferred()
- self._tunneledHost = host
- self._tunneledPort = port
- self._contextFactory = contextFactory
- self._connectBuffer = bytearray()
+ self._tunnelReadyDeferred: Deferred[Protocol] = Deferred()
+ self._tunneledHost: str = host
+ self._tunneledPort: int = port
+ self._contextFactory: IPolicyForHTTPS = contextFactory
+ self._connectBuffer: bytearray = bytearray()
- def requestTunnel(self, protocol):
+ def requestTunnel(self, protocol: Protocol) -> Protocol:
"""Asks the proxy to open a tunnel."""
+ assert protocol.transport
tunnelReq = tunnel_request_data(
self._tunneledHost, self._tunneledPort, self._proxyAuthHeader
)
protocol.transport.write(tunnelReq)
self._protocolDataReceived = protocol.dataReceived
- protocol.dataReceived = self.processProxyResponse
+ protocol.dataReceived = self.processProxyResponse # type: ignore[method-assign]
self._protocol = protocol
return protocol
- def processProxyResponse(self, rcvd_bytes):
+ def processProxyResponse(self, data: bytes) -> None:
"""Processes the response from the proxy. If the tunnel is successfully
created, notifies the client that we are ready to send requests. If not
raises a TunnelError.
"""
- self._connectBuffer += rcvd_bytes
+ assert self._protocol.transport
+ self._connectBuffer += data
# make sure that enough (all) bytes are consumed
# and that we've got all HTTP headers (ending with a blank line)
# from the proxy so that we don't send those bytes to the TLS layer
@@ -155,23 +187,24 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
# see https://github.com/scrapy/scrapy/issues/2491
if b"\r\n\r\n" not in self._connectBuffer:
return
- self._protocol.dataReceived = self._protocolDataReceived
+ self._protocol.dataReceived = self._protocolDataReceived # type: ignore[method-assign]
respm = TunnelingTCP4ClientEndpoint._responseMatcher.match(self._connectBuffer)
if respm and int(respm.group("status")) == 200:
# set proper Server Name Indication extension
- sslOptions = self._contextFactory.creatorForNetloc(
+ sslOptions = self._contextFactory.creatorForNetloc( # type: ignore[call-arg,misc]
self._tunneledHost, self._tunneledPort
)
self._protocol.transport.startTLS(sslOptions, self._protocolFactory)
self._tunnelReadyDeferred.callback(self._protocol)
else:
+ extra: Any
if respm:
extra = {
"status": int(respm.group("status")),
"reason": respm.group("reason").strip(),
}
else:
- extra = rcvd_bytes[: self._truncatedLength]
+ extra = data[: self._truncatedLength]
self._tunnelReadyDeferred.errback(
TunnelError(
"Could not open CONNECT tunnel with proxy "
@@ -179,11 +212,11 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
)
)
- def connectFailed(self, reason):
+ def connectFailed(self, reason: Failure) -> None:
"""Propagates the errback to the appropriate deferred."""
self._tunnelReadyDeferred.errback(reason)
- def connect(self, protocolFactory):
+ def connect(self, protocolFactory: Factory) -> Deferred[Protocol]:
self._protocolFactory = protocolFactory
connectDeferred = super().connect(protocolFactory)
connectDeferred.addCallback(self.requestTunnel)
@@ -191,7 +224,9 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
return self._tunnelReadyDeferred
-def tunnel_request_data(host, port, proxy_auth_header=None):
+def tunnel_request_data(
+ host: str, port: int, proxy_auth_header: bytes | None = None
+) -> bytes:
r"""
Return binary content of a CONNECT request.
@@ -222,18 +257,19 @@ class TunnelingAgent(Agent):
def __init__(
self,
- reactor,
- proxyConf,
- contextFactory=None,
- connectTimeout=None,
- bindAddress=None,
- pool=None,
+ *,
+ reactor: ReactorBase,
+ proxyConf: tuple[str, int, bytes | None],
+ contextFactory: IPolicyForHTTPS,
+ connectTimeout: float | None = None,
+ bindAddress: bytes | None = None,
+ pool: HTTPConnectionPool | None = None,
):
super().__init__(reactor, contextFactory, connectTimeout, bindAddress, pool)
- self._proxyConf = proxyConf
- self._contextFactory = contextFactory
+ self._proxyConf: tuple[str, int, bytes | None] = proxyConf
+ self._contextFactory: IPolicyForHTTPS = contextFactory
- def _getEndpoint(self, uri):
+ def _getEndpoint(self, uri: URI) -> TunnelingTCP4ClientEndpoint:
return TunnelingTCP4ClientEndpoint(
reactor=self._reactor,
host=uri.host,
@@ -245,8 +281,15 @@ class TunnelingAgent(Agent):
)
def _requestWithEndpoint(
- self, key, endpoint, method, parsedURI, headers, bodyProducer, requestPath
- ):
+ self,
+ key: Any,
+ endpoint: TCP4ClientEndpoint,
+ method: bytes,
+ parsedURI: bytes,
+ headers: TxHeaders | None,
+ bodyProducer: IBodyProducer | None,
+ requestPath: bytes,
+ ) -> Deferred[TxResponse]:
# proxy host and port are required for HTTP pool `key`
# otherwise, same remote host connection request could reuse
# a cached tunneled connection to a different proxy
@@ -264,7 +307,12 @@ class TunnelingAgent(Agent):
class ScrapyProxyAgent(Agent):
def __init__(
- self, reactor, proxyURI, connectTimeout=None, bindAddress=None, pool=None
+ self,
+ reactor: ReactorBase,
+ proxyURI: bytes,
+ connectTimeout: float | None = None,
+ bindAddress: bytes | None = None,
+ pool: HTTPConnectionPool | None = None,
):
super().__init__(
reactor=reactor,
@@ -272,9 +320,15 @@ class ScrapyProxyAgent(Agent):
bindAddress=bindAddress,
pool=pool,
)
- self._proxyURI = URI.fromBytes(proxyURI)
+ self._proxyURI: URI = URI.fromBytes(proxyURI)
- def request(self, method, uri, headers=None, bodyProducer=None):
+ def request(
+ self,
+ method: bytes,
+ uri: bytes,
+ headers: TxHeaders | None = None,
+ bodyProducer: IBodyProducer | None = None,
+ ) -> Deferred[TxResponse]:
"""
Issue a new request via the configured proxy.
"""
@@ -298,26 +352,27 @@ class ScrapyAgent:
def __init__(
self,
- contextFactory=None,
- connectTimeout=10,
- bindAddress=None,
- pool=None,
- maxsize=0,
- warnsize=0,
- fail_on_dataloss=True,
- crawler=None,
+ *,
+ contextFactory: IPolicyForHTTPS,
+ connectTimeout: float = 10,
+ bindAddress: bytes | None = None,
+ pool: HTTPConnectionPool | None = None,
+ maxsize: int = 0,
+ warnsize: int = 0,
+ fail_on_dataloss: bool = True,
+ crawler: Crawler,
):
- self._contextFactory = contextFactory
- self._connectTimeout = connectTimeout
- self._bindAddress = bindAddress
- self._pool = pool
- self._maxsize = maxsize
- self._warnsize = warnsize
- self._fail_on_dataloss = fail_on_dataloss
- self._txresponse = None
- self._crawler = crawler
+ self._contextFactory: IPolicyForHTTPS = contextFactory
+ self._connectTimeout: float = connectTimeout
+ self._bindAddress: bytes | None = bindAddress
+ self._pool: HTTPConnectionPool | None = pool
+ self._maxsize: int = maxsize
+ self._warnsize: int = warnsize
+ self._fail_on_dataloss: bool = fail_on_dataloss
+ self._txresponse: TxResponse | None = None
+ self._crawler: Crawler = crawler
- def _get_agent(self, request, timeout):
+ def _get_agent(self, request: Request, timeout: float) -> Agent:
from twisted.internet import reactor
bindaddress = request.meta.get("bindaddress") or self._bindAddress
@@ -325,10 +380,10 @@ class ScrapyAgent:
if proxy:
proxyScheme, proxyNetloc, proxyHost, proxyPort, proxyParams = _parse(proxy)
scheme = _parse(request.url)[0]
- proxyHost = to_unicode(proxyHost)
+ proxyHost_str = to_unicode(proxyHost)
if scheme == b"https":
proxyAuth = request.headers.get(b"Proxy-Authorization", None)
- proxyConf = (proxyHost, proxyPort, proxyAuth)
+ proxyConf = (proxyHost_str, proxyPort, proxyAuth)
return self._TunnelingAgent(
reactor=reactor,
proxyConf=proxyConf,
@@ -338,7 +393,9 @@ class ScrapyAgent:
pool=self._pool,
)
proxyScheme = proxyScheme or b"http"
- proxyURI = urlunparse((proxyScheme, proxyNetloc, proxyParams, "", "", ""))
+ proxyURI = urlunparse(
+ (proxyScheme, proxyNetloc, proxyParams, b"", b"", b"")
+ )
return self._ProxyAgent(
reactor=reactor,
proxyURI=to_bytes(proxyURI, encoding="ascii"),
@@ -355,7 +412,7 @@ class ScrapyAgent:
pool=self._pool,
)
- def download_request(self, request):
+ def download_request(self, request: Request) -> Deferred[Response]:
from twisted.internet import reactor
timeout = request.meta.get("download_timeout") or self._connectTimeout
@@ -372,20 +429,20 @@ class ScrapyAgent:
else:
bodyproducer = None
start_time = time()
- d = agent.request(
+ d: Deferred[TxResponse] = agent.request(
method, to_bytes(url, encoding="ascii"), headers, bodyproducer
)
# set download latency
d.addCallback(self._cb_latency, request, start_time)
# response body is ready to be consumed
- d.addCallback(self._cb_bodyready, request)
- d.addCallback(self._cb_bodydone, request, url)
+ d2: Deferred[_ResultT] = d.addCallback(self._cb_bodyready, request)
+ d3: Deferred[Response] = d2.addCallback(self._cb_bodydone, request, url)
# check download timeout
- self._timeout_cl = reactor.callLater(timeout, d.cancel)
- d.addBoth(self._cb_timeout, request, url, timeout)
- return d
+ self._timeout_cl = reactor.callLater(timeout, d3.cancel)
+ d3.addBoth(self._cb_timeout, request, url, timeout)
+ return d3
- def _cb_timeout(self, result, request, url, timeout):
+ def _cb_timeout(self, result: _T, request: Request, url: str, timeout: float) -> _T:
if self._timeout_cl.active():
self._timeout_cl.cancel()
return result
@@ -396,19 +453,21 @@ class ScrapyAgent:
raise TimeoutError(f"Getting {url} took longer than {timeout} seconds.")
- def _cb_latency(self, result, request, start_time):
+ def _cb_latency(self, result: _T, request: Request, start_time: float) -> _T:
request.meta["download_latency"] = time() - start_time
return result
@staticmethod
- def _headers_from_twisted_response(response):
+ def _headers_from_twisted_response(response: TxResponse) -> Headers:
headers = Headers()
if response.length != UNKNOWN_LENGTH:
headers[b"Content-Length"] = str(response.length).encode()
headers.update(response.headers.getAllRawHeaders())
return headers
- def _cb_bodyready(self, txresponse, request):
+ def _cb_bodyready(
+ self, txresponse: TxResponse, request: Request
+ ) -> _ResultT | Deferred[_ResultT]:
headers_received_result = self._crawler.signals.send_catch_log(
signal=signals.headers_received,
headers=self._headers_from_twisted_response(txresponse),
@@ -464,7 +523,7 @@ class ScrapyAgent:
logger.warning(warning_msg, warning_args)
txresponse._transport.loseConnection()
- raise defer.CancelledError(warning_msg % warning_args)
+ raise CancelledError(warning_msg % warning_args)
if warnsize and expected_size > warnsize:
logger.warning(
@@ -473,11 +532,11 @@ class ScrapyAgent:
{"size": expected_size, "warnsize": warnsize, "request": request},
)
- def _cancel(_):
+ def _cancel(_: Any) -> None:
# Abort connection immediately.
txresponse._transport._producer.abortConnection()
- d = defer.Deferred(_cancel)
+ d: Deferred[_ResultT] = Deferred(_cancel)
txresponse.deliverBody(
_ResponseReader(
finished=d,
@@ -495,7 +554,9 @@ class ScrapyAgent:
return d
- def _cb_bodydone(self, result, request, url):
+ def _cb_bodydone(
+ self, result: _ResultT, request: Request, url: str
+ ) -> Response | Failure:
headers = self._headers_from_twisted_response(result["txresponse"])
respcls = responsetypes.from_args(headers=headers, url=url, body=result["body"])
try:
@@ -514,6 +575,7 @@ class ScrapyAgent:
protocol=protocol,
)
if result.get("failure"):
+ assert result["failure"]
result["failure"].value.response = response
return result["failure"]
return response
@@ -521,47 +583,49 @@ class ScrapyAgent:
@implementer(IBodyProducer)
class _RequestBodyProducer:
- def __init__(self, body):
+ def __init__(self, body: bytes):
self.body = body
self.length = len(body)
- def startProducing(self, consumer):
+ def startProducing(self, consumer: IConsumer) -> Deferred[None]:
consumer.write(self.body)
- return defer.succeed(None)
+ return succeed(None)
- def pauseProducing(self):
+ def pauseProducing(self) -> None:
pass
- def stopProducing(self):
+ def stopProducing(self) -> None:
pass
-class _ResponseReader(protocol.Protocol):
+class _ResponseReader(Protocol):
def __init__(
self,
- finished,
- txresponse,
- request,
- maxsize,
- warnsize,
- fail_on_dataloss,
- crawler,
+ finished: Deferred[_ResultT],
+ txresponse: TxResponse,
+ request: Request,
+ maxsize: int,
+ warnsize: int,
+ fail_on_dataloss: bool,
+ crawler: Crawler,
):
- self._finished = finished
- self._txresponse = txresponse
- self._request = request
- self._bodybuf = BytesIO()
- self._maxsize = maxsize
- self._warnsize = warnsize
- self._fail_on_dataloss = fail_on_dataloss
- self._fail_on_dataloss_warned = False
- self._reached_warnsize = False
- self._bytes_received = 0
- self._certificate = None
- self._ip_address = None
- self._crawler = crawler
+ self._finished: Deferred[_ResultT] = finished
+ self._txresponse: TxResponse = txresponse
+ self._request: Request = request
+ self._bodybuf: BytesIO = BytesIO()
+ self._maxsize: int = maxsize
+ self._warnsize: int = warnsize
+ self._fail_on_dataloss: bool = fail_on_dataloss
+ self._fail_on_dataloss_warned: bool = False
+ self._reached_warnsize: bool = False
+ self._bytes_received: int = 0
+ self._certificate: ssl.Certificate | None = None
+ self._ip_address: ipaddress.IPv4Address | ipaddress.IPv6Address | None = None
+ self._crawler: Crawler = crawler
- def _finish_response(self, flags=None, failure=None):
+ def _finish_response(
+ self, flags: list[str] | None = None, failure: Failure | None = None
+ ) -> None:
self._finished.callback(
{
"txresponse": self._txresponse,
@@ -573,7 +637,8 @@ class _ResponseReader(protocol.Protocol):
}
)
- def connectionMade(self):
+ def connectionMade(self) -> None:
+ assert self.transport
if self._certificate is None:
with suppress(AttributeError):
self._certificate = ssl.Certificate(
@@ -585,11 +650,12 @@ class _ResponseReader(protocol.Protocol):
self.transport._producer.getPeer().host
)
- def dataReceived(self, bodyBytes):
+ def dataReceived(self, bodyBytes: bytes) -> None:
# This maybe called several times after cancel was called with buffered data.
if self._finished.called:
return
+ assert self.transport
self._bodybuf.write(bodyBytes)
self._bytes_received += len(bodyBytes)
@@ -636,7 +702,7 @@ class _ResponseReader(protocol.Protocol):
{"warnsize": self._warnsize, "request": self._request},
)
- def connectionLost(self, reason):
+ def connectionLost(self, reason: Failure = connectionDone) -> None:
if self._finished.called:
return
diff --git a/scrapy/core/downloader/handlers/http2.py b/scrapy/core/downloader/handlers/http2.py
index b2579362c..f0f9ceeb7 100644
--- a/scrapy/core/downloader/handlers/http2.py
+++ b/scrapy/core/downloader/handlers/http2.py
@@ -1,28 +1,33 @@
+from __future__ import annotations
+
from time import time
-from typing import Optional, Type, TypeVar
+from typing import TYPE_CHECKING
from urllib.parse import urldefrag
-from twisted.internet.base import DelayedCall
-from twisted.internet.defer import Deferred
from twisted.internet.error import TimeoutError
from twisted.web.client import URI
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
from scrapy.core.downloader.webclient import _parse
from scrapy.core.http2.agent import H2Agent, H2ConnectionPool, ScrapyProxyH2Agent
-from scrapy.crawler import Crawler
-from scrapy.http import Request, Response
-from scrapy.settings import Settings
-from scrapy.spiders import Spider
from scrapy.utils.python import to_bytes
-H2DownloadHandlerOrSubclass = TypeVar(
- "H2DownloadHandlerOrSubclass", bound="H2DownloadHandler"
-)
+if TYPE_CHECKING:
+ from twisted.internet.base import DelayedCall
+ from twisted.internet.defer import Deferred
+ from twisted.web.iweb import IPolicyForHTTPS
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.http import Request, Response
+ from scrapy.settings import Settings
+ from scrapy.spiders import Spider
class H2DownloadHandler:
- def __init__(self, settings: Settings, crawler: Optional[Crawler] = None):
+ def __init__(self, settings: Settings, crawler: Crawler):
self._crawler = crawler
from twisted.internet import reactor
@@ -31,12 +36,10 @@ class H2DownloadHandler:
self._context_factory = load_context_factory_from_settings(settings, crawler)
@classmethod
- def from_crawler(
- cls: Type[H2DownloadHandlerOrSubclass], crawler: Crawler
- ) -> H2DownloadHandlerOrSubclass:
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler.settings, crawler)
- def download_request(self, request: Request, spider: Spider) -> Deferred:
+ def download_request(self, request: Request, spider: Spider) -> Deferred[Response]:
agent = ScrapyH2Agent(
context_factory=self._context_factory,
pool=self._pool,
@@ -54,11 +57,11 @@ class ScrapyH2Agent:
def __init__(
self,
- context_factory,
+ context_factory: IPolicyForHTTPS,
pool: H2ConnectionPool,
connect_timeout: int = 10,
- bind_address: Optional[bytes] = None,
- crawler: Optional[Crawler] = None,
+ bind_address: bytes | None = None,
+ crawler: Crawler | None = None,
) -> None:
self._context_factory = context_factory
self._connect_timeout = connect_timeout
@@ -66,7 +69,7 @@ class ScrapyH2Agent:
self._pool = pool
self._crawler = crawler
- def _get_agent(self, request: Request, timeout: Optional[float]) -> H2Agent:
+ def _get_agent(self, request: Request, timeout: float | None) -> H2Agent:
from twisted.internet import reactor
bind_address = request.meta.get("bindaddress") or self._bind_address
@@ -97,7 +100,7 @@ class ScrapyH2Agent:
pool=self._pool,
)
- def download_request(self, request: Request, spider: Spider) -> Deferred:
+ def download_request(self, request: Request, spider: Spider) -> Deferred[Response]:
from twisted.internet import reactor
timeout = request.meta.get("download_timeout") or self._connect_timeout
diff --git a/scrapy/core/downloader/handlers/s3.py b/scrapy/core/downloader/handlers/s3.py
index 81d8e8115..870a26f04 100644
--- a/scrapy/core/downloader/handlers/s3.py
+++ b/scrapy/core/downloader/handlers/s3.py
@@ -1,21 +1,36 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
from scrapy.core.downloader.handlers.http import HTTPDownloadHandler
from scrapy.exceptions import NotConfigured
from scrapy.utils.boto import is_botocore_available
from scrapy.utils.httpobj import urlparse_cached
-from scrapy.utils.misc import create_instance
+from scrapy.utils.misc import build_from_crawler
+
+if TYPE_CHECKING:
+ from twisted.internet.defer import Deferred
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+ from scrapy.settings import BaseSettings
class S3DownloadHandler:
def __init__(
self,
- settings,
+ settings: BaseSettings,
*,
- crawler=None,
- aws_access_key_id=None,
- aws_secret_access_key=None,
- aws_session_token=None,
- httpdownloadhandler=HTTPDownloadHandler,
- **kw,
+ crawler: Crawler,
+ aws_access_key_id: str | None = None,
+ aws_secret_access_key: str | None = None,
+ aws_session_token: str | None = None,
+ httpdownloadhandler: type[HTTPDownloadHandler] = HTTPDownloadHandler,
+ **kw: Any,
):
if not is_botocore_available():
raise NotConfigured("missing botocore library")
@@ -43,25 +58,27 @@ class S3DownloadHandler:
if kw:
raise TypeError(f"Unexpected keyword arguments: {kw}")
if not self.anon:
+ assert aws_access_key_id is not None
+ assert aws_secret_access_key is not None
SignerCls = botocore.auth.AUTH_TYPE_MAPS["s3"]
- self._signer = SignerCls(
+ # botocore.auth.BaseSigner doesn't have an __init__() with args, only subclasses do
+ self._signer = SignerCls( # type: ignore[call-arg]
botocore.credentials.Credentials(
aws_access_key_id, aws_secret_access_key, aws_session_token
)
)
- _http_handler = create_instance(
- objcls=httpdownloadhandler,
- settings=settings,
- crawler=crawler,
+ _http_handler = build_from_crawler(
+ httpdownloadhandler,
+ crawler,
)
self._download_http = _http_handler.download_request
@classmethod
- def from_crawler(cls, crawler, **kwargs):
+ def from_crawler(cls, crawler: Crawler, **kwargs: Any) -> Self:
return cls(crawler.settings, crawler=crawler, **kwargs)
- def download_request(self, request, spider):
+ def download_request(self, request: Request, spider: Spider) -> Deferred[Response]:
p = urlparse_cached(request)
scheme = "https" if request.meta.get("is_secure") else "http"
bucket = p.hostname
@@ -78,6 +95,7 @@ class S3DownloadHandler:
headers=request.headers.to_unicode_dict(),
data=request.body,
)
+ assert self._signer
self._signer.add_auth(awsrequest)
request = request.replace(url=url, headers=awsrequest.headers.items())
return self._download_http(request, spider)
diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.py
index dca13c01e..db4191385 100644
--- a/scrapy/core/downloader/middleware.py
+++ b/scrapy/core/downloader/middleware.py
@@ -3,25 +3,34 @@ Downloader Middleware manager
See documentation in docs/topics/downloader-middleware.rst
"""
-from typing import Any, Callable, Generator, List, Union, cast
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from typing import TYPE_CHECKING, Any, cast
from twisted.internet.defer import Deferred, inlineCallbacks
-from twisted.python.failure import Failure
-from scrapy import Spider
from scrapy.exceptions import _InvalidOutput
from scrapy.http import Request, Response
from scrapy.middleware import MiddlewareManager
-from scrapy.settings import BaseSettings
from scrapy.utils.conf import build_component_list
from scrapy.utils.defer import deferred_from_coro, mustbe_deferred
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from twisted.python.failure import Failure
+
+ from scrapy import Spider
+ from scrapy.settings import BaseSettings
+
class DownloaderMiddlewareManager(MiddlewareManager):
component_name = "downloader middleware"
@classmethod
- def _get_mwlist_from_settings(cls, settings: BaseSettings) -> List[Any]:
+ def _get_mwlist_from_settings(cls, settings: BaseSettings) -> list[Any]:
return build_component_list(settings.getwithbase("DOWNLOADER_MIDDLEWARES"))
def _add_middleware(self, mw: Any) -> None:
@@ -33,10 +42,15 @@ class DownloaderMiddlewareManager(MiddlewareManager):
self.methods["process_exception"].appendleft(mw.process_exception)
def download(
- self, download_func: Callable, request: Request, spider: Spider
- ) -> Deferred:
+ self,
+ download_func: Callable[[Request, Spider], Deferred[Response]],
+ request: Request,
+ spider: Spider,
+ ) -> Deferred[Response | Request]:
@inlineCallbacks
- def process_request(request: Request) -> Generator[Deferred, Any, Any]:
+ def process_request(
+ request: Request,
+ ) -> Generator[Deferred[Any], Any, Response | Request]:
for method in self.methods["process_request"]:
method = cast(Callable, method)
response = yield deferred_from_coro(
@@ -51,15 +65,15 @@ class DownloaderMiddlewareManager(MiddlewareManager):
)
if response:
return response
- return (yield download_func(request=request, spider=spider))
+ return (yield download_func(request, spider))
@inlineCallbacks
def process_response(
- response: Union[Response, Request]
- ) -> Generator[Deferred, Any, Union[Response, Request]]:
+ response: Response | Request,
+ ) -> Generator[Deferred[Any], Any, Response | Request]:
if response is None:
raise TypeError("Received None in process_response")
- elif isinstance(response, Request):
+ if isinstance(response, Request):
return response
for method in self.methods["process_response"]:
@@ -79,7 +93,7 @@ class DownloaderMiddlewareManager(MiddlewareManager):
@inlineCallbacks
def process_exception(
failure: Failure,
- ) -> Generator[Deferred, Any, Union[Failure, Response, Request]]:
+ ) -> Generator[Deferred[Any], Any, Failure | Response | Request]:
exception = failure.value
for method in self.methods["process_exception"]:
method = cast(Callable, method)
@@ -97,7 +111,9 @@ class DownloaderMiddlewareManager(MiddlewareManager):
return response
return failure
- deferred = mustbe_deferred(process_request, request)
+ deferred: Deferred[Response | Request] = mustbe_deferred(
+ process_request, request
+ )
deferred.addErrback(process_exception)
deferred.addCallback(process_response)
return deferred
diff --git a/scrapy/core/downloader/tls.py b/scrapy/core/downloader/tls.py
index 33cea7263..1ae66f614 100644
--- a/scrapy/core/downloader/tls.py
+++ b/scrapy/core/downloader/tls.py
@@ -1,5 +1,5 @@
import logging
-from typing import Any, Dict
+from typing import Any
from OpenSSL import SSL
from service_identity.exceptions import CertificateError
@@ -21,7 +21,7 @@ METHOD_TLSv11 = "TLSv1.1"
METHOD_TLSv12 = "TLSv1.2"
-openssl_methods: Dict[str, int] = {
+openssl_methods: dict[str, int] = {
METHOD_TLS: SSL.SSLv23_METHOD, # protocol negotiation (recommended)
METHOD_TLSv10: SSL.TLSv1_METHOD, # TLS 1.0 only
METHOD_TLSv11: SSL.TLSv1_1_METHOD, # TLS 1.1 only
diff --git a/scrapy/core/downloader/webclient.py b/scrapy/core/downloader/webclient.py
index bb1f73805..ee10ae73b 100644
--- a/scrapy/core/downloader/webclient.py
+++ b/scrapy/core/downloader/webclient.py
@@ -1,20 +1,24 @@
+from __future__ import annotations
+
import re
from time import time
-from typing import Optional, Tuple
+from typing import TYPE_CHECKING
from urllib.parse import ParseResult, urldefrag, urlparse, urlunparse
from twisted.internet import defer
from twisted.internet.protocol import ClientFactory
from twisted.web.http import HTTPClient
-from scrapy import Request
-from scrapy.http import Headers
+from scrapy.http import Headers, Response
from scrapy.responsetypes import responsetypes
from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.python import to_bytes, to_unicode
+if TYPE_CHECKING:
+ from scrapy import Request
-def _parsed_url_args(parsed: ParseResult) -> Tuple[bytes, bytes, bytes, int, bytes]:
+
+def _parsed_url_args(parsed: ParseResult) -> tuple[bytes, bytes, bytes, int, bytes]:
# Assume parsed is urlparse-d from Request.url,
# which was passed via safe_url_string and is ascii-only.
path_str = urlunparse(("", "", parsed.path or "/", parsed.params, parsed.query, ""))
@@ -29,7 +33,7 @@ def _parsed_url_args(parsed: ParseResult) -> Tuple[bytes, bytes, bytes, int, byt
return scheme, netloc, host, port, path
-def _parse(url: str) -> Tuple[bytes, bytes, bytes, int, bytes]:
+def _parse(url: str) -> tuple[bytes, bytes, bytes, int, bytes]:
"""Return tuple of (scheme, netloc, host, port, path),
all in bytes except for port which is int.
Assume url is from Request.url, which was passed via safe_url_string
@@ -140,12 +144,12 @@ class ScrapyHTTPClientFactory(ClientFactory):
# converting to bytes to comply to Twisted interface
self.url: bytes = to_bytes(self._url, encoding="ascii")
self.method: bytes = to_bytes(request.method, encoding="ascii")
- self.body: Optional[bytes] = request.body or None
+ self.body: bytes | None = request.body or None
self.headers: Headers = Headers(request.headers)
- self.response_headers: Optional[Headers] = None
+ self.response_headers: Headers | None = None
self.timeout: float = request.meta.get("download_timeout") or timeout
self.start_time: float = time()
- self.deferred: defer.Deferred = defer.Deferred().addCallback(
+ self.deferred: defer.Deferred[Response] = defer.Deferred().addCallback(
self._build_response, request
)
@@ -155,7 +159,7 @@ class ScrapyHTTPClientFactory(ClientFactory):
# needed to add the callback _waitForDisconnect.
# Specifically this avoids the AttributeError exception when
# clientConnectionFailed method is called.
- self._disconnectedDeferred: defer.Deferred = defer.Deferred()
+ self._disconnectedDeferred: defer.Deferred[None] = defer.Deferred()
self._set_connection_attributes(request)
diff --git a/scrapy/core/engine.py b/scrapy/core/engine.py
index dd1f56f8c..5480df72c 100644
--- a/scrapy/core/engine.py
+++ b/scrapy/core/engine.py
@@ -4,60 +4,57 @@ This is the Scrapy engine which controls the Scheduler, Downloader and Spider.
For more information see docs/topics/architecture.rst
"""
+
+from __future__ import annotations
+
import logging
from time import time
-from typing import (
- TYPE_CHECKING,
- Any,
- Callable,
- Generator,
- Iterable,
- Iterator,
- Optional,
- Set,
- Type,
- Union,
- cast,
-)
+from typing import TYPE_CHECKING, Any, TypeVar, cast
+from itemadapter import is_item
from twisted.internet.defer import Deferred, inlineCallbacks, succeed
from twisted.internet.task import LoopingCall
from twisted.python.failure import Failure
from scrapy import signals
-from scrapy.core.downloader import Downloader
-from scrapy.core.scraper import Scraper
-from scrapy.exceptions import CloseSpider, DontCloseSpider
+from scrapy.core.scraper import Scraper, _HandleOutputDeferred
+from scrapy.exceptions import CloseSpider, DontCloseSpider, IgnoreRequest
from scrapy.http import Request, Response
-from scrapy.logformatter import LogFormatter
-from scrapy.settings import BaseSettings, Settings
-from scrapy.signalmanager import SignalManager
-from scrapy.spiders import Spider
from scrapy.utils.log import failure_to_exc_info, logformatter_adapter
-from scrapy.utils.misc import create_instance, load_object
+from scrapy.utils.misc import build_from_crawler, load_object
from scrapy.utils.reactor import CallLaterOnce
if TYPE_CHECKING:
+ from collections.abc import Callable, Generator, Iterable, Iterator
+
+ from scrapy.core.downloader import Downloader
from scrapy.core.scheduler import BaseScheduler
from scrapy.crawler import Crawler
+ from scrapy.logformatter import LogFormatter
+ from scrapy.settings import BaseSettings, Settings
+ from scrapy.signalmanager import SignalManager
+ from scrapy.spiders import Spider
+
logger = logging.getLogger(__name__)
+_T = TypeVar("_T")
+
class Slot:
def __init__(
self,
start_requests: Iterable[Request],
close_if_idle: bool,
- nextcall: CallLaterOnce,
- scheduler: "BaseScheduler",
+ nextcall: CallLaterOnce[None],
+ scheduler: BaseScheduler,
) -> None:
- self.closing: Optional[Deferred] = None
- self.inprogress: Set[Request] = set()
- self.start_requests: Optional[Iterator[Request]] = iter(start_requests)
+ self.closing: Deferred[None] | None = None
+ self.inprogress: set[Request] = set()
+ self.start_requests: Iterator[Request] | None = iter(start_requests)
self.close_if_idle: bool = close_if_idle
- self.nextcall: CallLaterOnce = nextcall
- self.scheduler: "BaseScheduler" = scheduler
+ self.nextcall: CallLaterOnce[None] = nextcall
+ self.scheduler: BaseScheduler = scheduler
self.heartbeat: LoopingCall = LoopingCall(nextcall.schedule)
def add_request(self, request: Request) -> None:
@@ -67,7 +64,7 @@ class Slot:
self.inprogress.remove(request)
self._maybe_fire_closing()
- def close(self) -> Deferred:
+ def close(self) -> Deferred[None]:
self.closing = Deferred()
self._maybe_fire_closing()
return self.closing
@@ -82,29 +79,35 @@ class Slot:
class ExecutionEngine:
- def __init__(self, crawler: "Crawler", spider_closed_callback: Callable) -> None:
- self.crawler: "Crawler" = crawler
+ def __init__(
+ self,
+ crawler: Crawler,
+ spider_closed_callback: Callable[[Spider], Deferred[None] | None],
+ ) -> None:
+ self.crawler: Crawler = crawler
self.settings: Settings = crawler.settings
self.signals: SignalManager = crawler.signals
assert crawler.logformatter
self.logformatter: LogFormatter = crawler.logformatter
- self.slot: Optional[Slot] = None
- self.spider: Optional[Spider] = None
+ self.slot: Slot | None = None
+ self.spider: Spider | None = None
self.running: bool = False
self.paused: bool = False
- self.scheduler_cls: Type["BaseScheduler"] = self._get_scheduler_class(
+ self.scheduler_cls: type[BaseScheduler] = self._get_scheduler_class(
crawler.settings
)
- downloader_cls: Type[Downloader] = load_object(self.settings["DOWNLOADER"])
+ downloader_cls: type[Downloader] = load_object(self.settings["DOWNLOADER"])
self.downloader: Downloader = downloader_cls(crawler)
- self.scraper = Scraper(crawler)
- self._spider_closed_callback: Callable = spider_closed_callback
- self.start_time: Optional[float] = None
+ self.scraper: Scraper = Scraper(crawler)
+ self._spider_closed_callback: Callable[[Spider], Deferred[None] | None] = (
+ spider_closed_callback
+ )
+ self.start_time: float | None = None
- def _get_scheduler_class(self, settings: BaseSettings) -> Type["BaseScheduler"]:
+ def _get_scheduler_class(self, settings: BaseSettings) -> type[BaseScheduler]:
from scrapy.core.scheduler import BaseScheduler
- scheduler_cls: Type = load_object(settings["SCHEDULER"])
+ scheduler_cls: type[BaseScheduler] = load_object(settings["SCHEDULER"])
if not issubclass(scheduler_cls, BaseScheduler):
raise TypeError(
f"The provided scheduler class ({settings['SCHEDULER']})"
@@ -113,20 +116,20 @@ class ExecutionEngine:
return scheduler_cls
@inlineCallbacks
- def start(self) -> Generator[Deferred, Any, None]:
+ def start(self) -> Generator[Deferred[Any], Any, None]:
if self.running:
raise RuntimeError("Engine already running")
self.start_time = time()
yield self.signals.send_catch_log_deferred(signal=signals.engine_started)
self.running = True
- self._closewait: Deferred = Deferred()
+ self._closewait: Deferred[None] = Deferred()
yield self._closewait
- def stop(self) -> Deferred:
+ def stop(self) -> Deferred[None]:
"""Gracefully stop the execution engine"""
@inlineCallbacks
- def _finish_stopping_engine(_: Any) -> Generator[Deferred, Any, None]:
+ def _finish_stopping_engine(_: Any) -> Generator[Deferred[Any], Any, None]:
yield self.signals.send_catch_log_deferred(signal=signals.engine_stopped)
self._closewait.callback(None)
@@ -141,7 +144,7 @@ class ExecutionEngine:
)
return dfd.addBoth(_finish_stopping_engine)
- def close(self) -> Deferred:
+ def close(self) -> Deferred[None]:
"""
Gracefully close the execution engine.
If it has already been started, stop it. In all cases, close the spider and the downloader.
@@ -168,7 +171,7 @@ class ExecutionEngine:
assert self.spider is not None # typing
if self.paused:
- return None
+ return
while (
not self._needs_backout()
@@ -178,7 +181,7 @@ class ExecutionEngine:
if self.slot.start_requests is not None and not self._needs_backout():
try:
- request = next(self.slot.start_requests)
+ request_or_item = next(self.slot.start_requests)
except StopIteration:
self.slot.start_requests = None
except Exception:
@@ -189,7 +192,16 @@ class ExecutionEngine:
extra={"spider": self.spider},
)
else:
- self.crawl(request)
+ if isinstance(request_or_item, Request):
+ self.crawl(request_or_item)
+ elif is_item(request_or_item):
+ self.scraper.start_itemproc(request_or_item, response=None)
+ else:
+ logger.error(
+ f"Got {request_or_item!r} among start requests. Only "
+ f"requests and items are supported. It will be "
+ f"ignored."
+ )
if self.spider_is_idle() and self.slot.close_if_idle:
self._spider_idle()
@@ -204,7 +216,7 @@ class ExecutionEngine:
or self.scraper.slot.needs_backout()
)
- def _next_request_from_scheduler(self) -> Optional[Deferred]:
+ def _next_request_from_scheduler(self) -> Deferred[None] | None:
assert self.slot is not None # typing
assert self.spider is not None # typing
@@ -212,7 +224,7 @@ class ExecutionEngine:
if request is None:
return None
- d = self._download(request)
+ d: Deferred[Response | Request] = self._download(request)
d.addBoth(self._handle_downloader_output, request)
d.addErrback(
lambda f: logger.info(
@@ -226,8 +238,8 @@ class ExecutionEngine:
assert self.slot
self.slot.remove_request(request)
- d.addBoth(_remove_request)
- d.addErrback(
+ d2: Deferred[None] = d.addBoth(_remove_request)
+ d2.addErrback(
lambda f: logger.info(
"Error while removing request from slot",
exc_info=failure_to_exc_info(f),
@@ -235,19 +247,19 @@ class ExecutionEngine:
)
)
slot = self.slot
- d.addBoth(lambda _: slot.nextcall.schedule())
- d.addErrback(
+ d2.addBoth(lambda _: slot.nextcall.schedule())
+ d2.addErrback(
lambda f: logger.info(
"Error while scheduling new request",
exc_info=failure_to_exc_info(f),
extra={"spider": self.spider},
)
)
- return d
+ return d2
def _handle_downloader_output(
- self, result: Union[Request, Response, Failure], request: Request
- ) -> Optional[Deferred]:
+ self, result: Request | Response | Failure, request: Request
+ ) -> _HandleOutputDeferred | None:
assert self.spider is not None # typing
if not isinstance(result, (Request, Response, Failure)):
@@ -291,33 +303,42 @@ class ExecutionEngine:
self.slot.nextcall.schedule() # type: ignore[union-attr]
def _schedule_request(self, request: Request, spider: Spider) -> None:
- self.signals.send_catch_log(
- signals.request_scheduled, request=request, spider=spider
+ request_scheduled_result = self.signals.send_catch_log(
+ signals.request_scheduled,
+ request=request,
+ spider=spider,
+ dont_log=IgnoreRequest,
)
+ for handler, result in request_scheduled_result:
+ if isinstance(result, Failure) and isinstance(result.value, IgnoreRequest):
+ return
if not self.slot.scheduler.enqueue_request(request): # type: ignore[union-attr]
self.signals.send_catch_log(
signals.request_dropped, request=request, spider=spider
)
- def download(self, request: Request) -> Deferred:
+ def download(self, request: Request) -> Deferred[Response]:
"""Return a Deferred which fires with a Response as result, only downloader middlewares are applied"""
if self.spider is None:
raise RuntimeError(f"No open spider to crawl: {request}")
- return self._download(request).addBoth(self._downloaded, request)
+ d: Deferred[Response | Request] = self._download(request)
+ # Deferred.addBoth() overloads don't seem to support a Union[_T, Deferred[_T]] return type
+ d2: Deferred[Response] = d.addBoth(self._downloaded, request) # type: ignore[call-overload]
+ return d2
def _downloaded(
- self, result: Union[Response, Request, Failure], request: Request
- ) -> Union[Deferred, Response, Failure]:
+ self, result: Response | Request | Failure, request: Request
+ ) -> Deferred[Response] | Response | Failure:
assert self.slot is not None # typing
self.slot.remove_request(request)
return self.download(result) if isinstance(result, Request) else result
- def _download(self, request: Request) -> Deferred:
+ def _download(self, request: Request) -> Deferred[Response | Request]:
assert self.slot is not None # typing
self.slot.add_request(request)
- def _on_success(result: Union[Response, Request]) -> Union[Response, Request]:
+ def _on_success(result: Response | Request) -> Response | Request:
if not isinstance(result, (Response, Request)):
raise TypeError(
f"Incorrect type: expected Response or Request, got {type(result)}: {result!r}"
@@ -339,35 +360,37 @@ class ExecutionEngine:
)
return result
- def _on_complete(_: Any) -> Any:
+ def _on_complete(_: _T) -> _T:
assert self.slot is not None
self.slot.nextcall.schedule()
return _
assert self.spider is not None
- dwld = self.downloader.fetch(request, self.spider)
- dwld.addCallbacks(_on_success)
+ dwld: Deferred[Response | Request] = self.downloader.fetch(request, self.spider)
+ dwld.addCallback(_on_success)
dwld.addBoth(_on_complete)
return dwld
@inlineCallbacks
def open_spider(
- self, spider: Spider, start_requests: Iterable = (), close_if_idle: bool = True
- ) -> Generator[Deferred, Any, None]:
+ self,
+ spider: Spider,
+ start_requests: Iterable[Request] = (),
+ close_if_idle: bool = True,
+ ) -> Generator[Deferred[Any], Any, None]:
if self.slot is not None:
raise RuntimeError(f"No free spider slot when opening {spider.name!r}")
logger.info("Spider opened", extra={"spider": spider})
nextcall = CallLaterOnce(self._next_request)
- scheduler = create_instance(
- self.scheduler_cls, settings=None, crawler=self.crawler
- )
+ scheduler = build_from_crawler(self.scheduler_cls, self.crawler)
start_requests = yield self.scraper.spidermw.process_start_requests(
start_requests, spider
)
self.slot = Slot(start_requests, close_if_idle, nextcall, scheduler)
self.spider = spider
if hasattr(scheduler, "open"):
- yield scheduler.open(spider)
+ if d := scheduler.open(spider):
+ yield d
yield self.scraper.open_spider(spider)
assert self.crawler.stats
self.crawler.stats.open_spider(spider)
@@ -394,13 +417,13 @@ class ExecutionEngine:
if isinstance(x, Failure) and isinstance(x.value, ex)
}
if DontCloseSpider in detected_ex:
- return None
+ return
if self.spider_is_idle():
ex = detected_ex.get(CloseSpider, CloseSpider(reason="finished"))
assert isinstance(ex, CloseSpider) # typing
self.close_spider(self.spider, reason=ex.reason)
- def close_spider(self, spider: Spider, reason: str = "cancelled") -> Deferred:
+ def close_spider(self, spider: Spider, reason: str = "cancelled") -> Deferred[None]:
"""Close (cancel) spider and clear all its outstanding requests"""
if self.slot is None:
raise RuntimeError("Engine slot not assigned")
@@ -414,7 +437,7 @@ class ExecutionEngine:
dfd = self.slot.close()
- def log_failure(msg: str) -> Callable:
+ def log_failure(msg: str) -> Callable[[Failure], None]:
def errback(failure: Failure) -> None:
logger.error(
msg, exc_info=failure_to_exc_info(failure), extra={"spider": spider}
diff --git a/scrapy/core/http2/agent.py b/scrapy/core/http2/agent.py
index 215ea9716..45f32daaa 100644
--- a/scrapy/core/http2/agent.py
+++ b/scrapy/core/http2/agent.py
@@ -1,10 +1,10 @@
+from __future__ import annotations
+
from collections import deque
-from typing import Deque, Dict, List, Optional, Tuple
+from typing import TYPE_CHECKING
from twisted.internet import defer
-from twisted.internet.base import ReactorBase
from twisted.internet.defer import Deferred
-from twisted.internet.endpoints import HostnameEndpoint
from twisted.python.failure import Failure
from twisted.web.client import (
URI,
@@ -16,9 +16,17 @@ from twisted.web.error import SchemeNotSupported
from scrapy.core.downloader.contextfactory import AcceptableProtocolsContextFactory
from scrapy.core.http2.protocol import H2ClientFactory, H2ClientProtocol
-from scrapy.http.request import Request
-from scrapy.settings import Settings
-from scrapy.spiders import Spider
+
+if TYPE_CHECKING:
+ from twisted.internet.base import ReactorBase
+ from twisted.internet.endpoints import HostnameEndpoint
+
+ from scrapy.http import Request, Response
+ from scrapy.settings import Settings
+ from scrapy.spiders import Spider
+
+
+ConnectionKeyT = tuple[bytes, bytes, int]
class H2ConnectionPool:
@@ -28,19 +36,21 @@ class H2ConnectionPool:
# Store a dictionary which is used to get the respective
# H2ClientProtocolInstance using the key as Tuple(scheme, hostname, port)
- self._connections: Dict[Tuple, H2ClientProtocol] = {}
+ self._connections: dict[ConnectionKeyT, H2ClientProtocol] = {}
# Save all requests that arrive before the connection is established
- self._pending_requests: Dict[Tuple, Deque[Deferred]] = {}
+ self._pending_requests: dict[
+ ConnectionKeyT, deque[Deferred[H2ClientProtocol]]
+ ] = {}
def get_connection(
- self, key: Tuple, uri: URI, endpoint: HostnameEndpoint
- ) -> Deferred:
+ self, key: ConnectionKeyT, uri: URI, endpoint: HostnameEndpoint
+ ) -> Deferred[H2ClientProtocol]:
if key in self._pending_requests:
# Received a request while connecting to remote
# Create a deferred which will fire with the H2ClientProtocol
# instance
- d: Deferred = Deferred()
+ d: Deferred[H2ClientProtocol] = Deferred()
self._pending_requests[key].append(d)
return d
@@ -54,22 +64,24 @@ class H2ConnectionPool:
return self._new_connection(key, uri, endpoint)
def _new_connection(
- self, key: Tuple, uri: URI, endpoint: HostnameEndpoint
- ) -> Deferred:
+ self, key: ConnectionKeyT, uri: URI, endpoint: HostnameEndpoint
+ ) -> Deferred[H2ClientProtocol]:
self._pending_requests[key] = deque()
- conn_lost_deferred: Deferred = Deferred()
+ conn_lost_deferred: Deferred[list[BaseException]] = Deferred()
conn_lost_deferred.addCallback(self._remove_connection, key)
factory = H2ClientFactory(uri, self.settings, conn_lost_deferred)
conn_d = endpoint.connect(factory)
conn_d.addCallback(self.put_connection, key)
- d: Deferred = Deferred()
+ d: Deferred[H2ClientProtocol] = Deferred()
self._pending_requests[key].append(d)
return d
- def put_connection(self, conn: H2ClientProtocol, key: Tuple) -> H2ClientProtocol:
+ def put_connection(
+ self, conn: H2ClientProtocol, key: ConnectionKeyT
+ ) -> H2ClientProtocol:
self._connections[key] = conn
# Now as we have established a proper HTTP/2 connection
@@ -81,7 +93,9 @@ class H2ConnectionPool:
return conn
- def _remove_connection(self, errors: List[BaseException], key: Tuple) -> None:
+ def _remove_connection(
+ self, errors: list[BaseException], key: ConnectionKeyT
+ ) -> None:
self._connections.pop(key)
# Call the errback of all the pending requests for this connection
@@ -107,8 +121,8 @@ class H2Agent:
reactor: ReactorBase,
pool: H2ConnectionPool,
context_factory: BrowserLikePolicyForHTTPS = BrowserLikePolicyForHTTPS(),
- connect_timeout: Optional[float] = None,
- bind_address: Optional[bytes] = None,
+ connect_timeout: float | None = None,
+ bind_address: bytes | None = None,
) -> None:
self._reactor = reactor
self._pool = pool
@@ -119,17 +133,17 @@ class H2Agent:
self._reactor, self._context_factory, connect_timeout, bind_address
)
- def get_endpoint(self, uri: URI):
+ def get_endpoint(self, uri: URI) -> HostnameEndpoint:
return self.endpoint_factory.endpointForURI(uri)
- def get_key(self, uri: URI) -> Tuple:
+ def get_key(self, uri: URI) -> ConnectionKeyT:
"""
Arguments:
uri - URI obtained directly from request URL
"""
return uri.scheme, uri.host, uri.port
- def request(self, request: Request, spider: Spider) -> Deferred:
+ def request(self, request: Request, spider: Spider) -> Deferred[Response]:
uri = URI.fromBytes(bytes(request.url, encoding="utf-8"))
try:
endpoint = self.get_endpoint(uri)
@@ -137,9 +151,11 @@ class H2Agent:
return defer.fail(Failure())
key = self.get_key(uri)
- d = self._pool.get_connection(key, uri, endpoint)
- d.addCallback(lambda conn: conn.request(request, spider))
- return d
+ d: Deferred[H2ClientProtocol] = self._pool.get_connection(key, uri, endpoint)
+ d2: Deferred[Response] = d.addCallback(
+ lambda conn: conn.request(request, spider)
+ )
+ return d2
class ScrapyProxyH2Agent(H2Agent):
@@ -149,8 +165,8 @@ class ScrapyProxyH2Agent(H2Agent):
proxy_uri: URI,
pool: H2ConnectionPool,
context_factory: BrowserLikePolicyForHTTPS = BrowserLikePolicyForHTTPS(),
- connect_timeout: Optional[float] = None,
- bind_address: Optional[bytes] = None,
+ connect_timeout: float | None = None,
+ bind_address: bytes | None = None,
) -> None:
super().__init__(
reactor=reactor,
@@ -161,9 +177,9 @@ class ScrapyProxyH2Agent(H2Agent):
)
self._proxy_uri = proxy_uri
- def get_endpoint(self, uri: URI):
+ def get_endpoint(self, uri: URI) -> HostnameEndpoint:
return self.endpoint_factory.endpointForURI(self._proxy_uri)
- def get_key(self, uri: URI) -> Tuple:
+ def get_key(self, uri: URI) -> ConnectionKeyT:
"""We use the proxy uri instead of uri obtained from request url"""
- return "http-proxy", self._proxy_uri.host, self._proxy_uri.port
+ return b"http-proxy", self._proxy_uri.host, self._proxy_uri.port
diff --git a/scrapy/core/http2/protocol.py b/scrapy/core/http2/protocol.py
index bc8da50d7..23335b7b2 100644
--- a/scrapy/core/http2/protocol.py
+++ b/scrapy/core/http2/protocol.py
@@ -1,9 +1,10 @@
+from __future__ import annotations
+
import ipaddress
import itertools
import logging
from collections import deque
-from ipaddress import IPv4Address, IPv6Address
-from typing import Dict, List, Optional, Union
+from typing import TYPE_CHECKING, Any
from h2.config import H2Configuration
from h2.connection import H2Connection
@@ -20,20 +21,30 @@ from h2.events import (
WindowUpdated,
)
from h2.exceptions import FrameTooLargeError, H2Error
-from twisted.internet.defer import Deferred
from twisted.internet.error import TimeoutError
-from twisted.internet.interfaces import IHandshakeListener, IProtocolNegotiationFactory
+from twisted.internet.interfaces import (
+ IAddress,
+ IHandshakeListener,
+ IProtocolNegotiationFactory,
+)
from twisted.internet.protocol import Factory, Protocol, connectionDone
from twisted.internet.ssl import Certificate
from twisted.protocols.policies import TimeoutMixin
-from twisted.python.failure import Failure
-from twisted.web.client import URI
from zope.interface import implementer
from scrapy.core.http2.stream import Stream, StreamCloseReason
-from scrapy.http import Request
-from scrapy.settings import Settings
-from scrapy.spiders import Spider
+from scrapy.http import Request, Response
+
+if TYPE_CHECKING:
+ from ipaddress import IPv4Address, IPv6Address
+
+ from twisted.internet.defer import Deferred
+ from twisted.python.failure import Failure
+ from twisted.web.client import URI
+
+ from scrapy.settings import Settings
+ from scrapy.spiders import Spider
+
logger = logging.getLogger(__name__)
@@ -52,7 +63,7 @@ class InvalidNegotiatedProtocol(H2Error):
class RemoteTerminatedConnection(H2Error):
def __init__(
self,
- remote_ip_address: Optional[Union[IPv4Address, IPv6Address]],
+ remote_ip_address: IPv4Address | IPv6Address | None,
event: ConnectionTerminated,
) -> None:
self.remote_ip_address = remote_ip_address
@@ -63,9 +74,7 @@ class RemoteTerminatedConnection(H2Error):
class MethodNotAllowed405(H2Error):
- def __init__(
- self, remote_ip_address: Optional[Union[IPv4Address, IPv6Address]]
- ) -> None:
+ def __init__(self, remote_ip_address: IPv4Address | IPv6Address | None) -> None:
self.remote_ip_address = remote_ip_address
def __str__(self) -> str:
@@ -77,7 +86,10 @@ class H2ClientProtocol(Protocol, TimeoutMixin):
IDLE_TIMEOUT = 240
def __init__(
- self, uri: URI, settings: Settings, conn_lost_deferred: Deferred
+ self,
+ uri: URI,
+ settings: Settings,
+ conn_lost_deferred: Deferred[list[BaseException]],
) -> None:
"""
Arguments:
@@ -88,7 +100,7 @@ class H2ClientProtocol(Protocol, TimeoutMixin):
conn_lost_deferred -- Deferred fires with the reason: Failure to notify
that connection was lost
"""
- self._conn_lost_deferred = conn_lost_deferred
+ self._conn_lost_deferred: Deferred[list[BaseException]] = conn_lost_deferred
config = H2Configuration(client_side=True, header_encoding="utf-8")
self.conn = H2Connection(config=config)
@@ -99,19 +111,19 @@ class H2ClientProtocol(Protocol, TimeoutMixin):
self._stream_id_generator = itertools.count(start=1, step=2)
# Streams are stored in a dictionary keyed off their stream IDs
- self.streams: Dict[int, Stream] = {}
+ self.streams: dict[int, Stream] = {}
# If requests are received before connection is made we keep
# all requests in a pool and send them as the connection is made
- self._pending_request_stream_pool: deque = deque()
+ self._pending_request_stream_pool: deque[Stream] = deque()
# Save an instance of errors raised which lead to losing the connection
# We pass these instances to the streams ResponseFailed() failure
- self._conn_lost_errors: List[BaseException] = []
+ self._conn_lost_errors: list[BaseException] = []
# Some meta data of this connection
# initialized when connection is successfully made
- self.metadata: Dict = {
+ self.metadata: dict[str, Any] = {
# Peer certificate instance
"certificate": None,
# Address of the server we are connected to which
@@ -204,14 +216,14 @@ class H2ClientProtocol(Protocol, TimeoutMixin):
data = self.conn.data_to_send()
self.transport.write(data)
- def request(self, request: Request, spider: Spider) -> Deferred:
+ def request(self, request: Request, spider: Spider) -> Deferred[Response]:
if not isinstance(request, Request):
raise TypeError(
f"Expected scrapy.http.Request, received {request.__class__.__qualname__}"
)
stream = self._new_stream(request, spider)
- d = stream.get_response()
+ d: Deferred[Response] = stream.get_response()
# Add the stream to the request pool
self._pending_request_stream_pool.append(stream)
@@ -236,7 +248,7 @@ class H2ClientProtocol(Protocol, TimeoutMixin):
self.conn.initiate_connection()
self._write_to_transport()
- def _lose_connection_with_error(self, errors: List[BaseException]) -> None:
+ def _lose_connection_with_error(self, errors: list[BaseException]) -> None:
"""Helper function to lose the connection with the error sent as a
reason"""
self._conn_lost_errors += errors
@@ -339,7 +351,7 @@ class H2ClientProtocol(Protocol, TimeoutMixin):
self._pending_request_stream_pool.clear()
self.conn.close_connection()
- def _handle_events(self, events: List[Event]) -> None:
+ def _handle_events(self, events: list[Event]) -> None:
"""Private method which acts as a bridge between the events
received from the HTTP/2 data and IH2EventsHandler
@@ -425,14 +437,17 @@ class H2ClientProtocol(Protocol, TimeoutMixin):
@implementer(IProtocolNegotiationFactory)
class H2ClientFactory(Factory):
def __init__(
- self, uri: URI, settings: Settings, conn_lost_deferred: Deferred
+ self,
+ uri: URI,
+ settings: Settings,
+ conn_lost_deferred: Deferred[list[BaseException]],
) -> None:
self.uri = uri
self.settings = settings
self.conn_lost_deferred = conn_lost_deferred
- def buildProtocol(self, addr) -> H2ClientProtocol:
+ def buildProtocol(self, addr: IAddress) -> H2ClientProtocol:
return H2ClientProtocol(self.uri, self.settings, self.conn_lost_deferred)
- def acceptableProtocols(self) -> List[bytes]:
+ def acceptableProtocols(self) -> list[bytes]:
return [PROTOCOL_NAME]
diff --git a/scrapy/core/http2/stream.py b/scrapy/core/http2/stream.py
index 39d5921f4..a4dc89c18 100644
--- a/scrapy/core/http2/stream.py
+++ b/scrapy/core/http2/stream.py
@@ -1,23 +1,26 @@
+from __future__ import annotations
+
import logging
from enum import Enum
from io import BytesIO
-from typing import TYPE_CHECKING, Dict, List, Optional, Tuple
-from urllib.parse import urlparse
+from typing import TYPE_CHECKING, Any
from h2.errors import ErrorCodes
from h2.exceptions import H2Error, ProtocolError, StreamClosedError
-from hpack import HeaderTuple
from twisted.internet.defer import CancelledError, Deferred
from twisted.internet.error import ConnectionClosed
from twisted.python.failure import Failure
from twisted.web.client import ResponseFailed
-from scrapy.http import Request
from scrapy.http.headers import Headers
from scrapy.responsetypes import responsetypes
+from scrapy.utils.httpobj import urlparse_cached
if TYPE_CHECKING:
+ from hpack import HeaderTuple
+
from scrapy.core.http2.protocol import H2ClientProtocol
+ from scrapy.http import Request, Response
logger = logging.getLogger(__name__)
@@ -87,7 +90,7 @@ class Stream:
self,
stream_id: int,
request: Request,
- protocol: "H2ClientProtocol",
+ protocol: H2ClientProtocol,
download_maxsize: int = 0,
download_warnsize: int = 0,
) -> None:
@@ -99,7 +102,7 @@ class Stream:
"""
self.stream_id: int = stream_id
self._request: Request = request
- self._protocol: "H2ClientProtocol" = protocol
+ self._protocol: H2ClientProtocol = protocol
self._download_maxsize = self._request.meta.get(
"download_maxsize", download_maxsize
@@ -110,18 +113,18 @@ class Stream:
# Metadata of an HTTP/2 connection stream
# initialized when stream is instantiated
- self.metadata: Dict = {
- "request_content_length": 0
- if self._request.body is None
- else len(self._request.body),
+ self.metadata: dict[str, Any] = {
+ "request_content_length": (
+ 0 if self._request.body is None else len(self._request.body)
+ ),
# Flag to keep track whether the stream has initiated the request
"request_sent": False,
# Flag to track whether we have logged about exceeding download warnsize
"reached_warnsize": False,
# Each time we send a data frame, we will decrease value by the amount send.
- "remaining_content_length": 0
- if self._request.body is None
- else len(self._request.body),
+ "remaining_content_length": (
+ 0 if self._request.body is None else len(self._request.body)
+ ),
# Flag to keep track whether client (self) have closed this stream
"stream_closed_local": False,
# Flag to keep track whether the server has closed the stream
@@ -131,7 +134,7 @@ class Stream:
# Private variable used to build the response
# this response is then converted to appropriate Response class
# passed to the response deferred callback
- self._response: Dict = {
+ self._response: dict[str, Any] = {
# Data received frame by frame from the server is appended
# and passed to the response Deferred when completely received.
"body": BytesIO(),
@@ -142,7 +145,7 @@ class Stream:
"headers": Headers({}),
}
- def _cancel(_) -> None:
+ def _cancel(_: Any) -> None:
# Close this stream as gracefully as possible
# If the associated request is initiated we reset this stream
# else we directly call close() method
@@ -151,7 +154,7 @@ class Stream:
else:
self.close(StreamCloseReason.CANCELLED)
- self._deferred_response: Deferred = Deferred(_cancel)
+ self._deferred_response: Deferred[Response] = Deferred(_cancel)
def __repr__(self) -> str:
return f"Stream(id={self.stream_id!r})"
@@ -177,7 +180,7 @@ class Stream:
and not self.metadata["reached_warnsize"]
)
- def get_response(self) -> Deferred:
+ def get_response(self) -> Deferred[Response]:
"""Simply return a Deferred which fires when response
from the asynchronous request is available
"""
@@ -185,7 +188,7 @@ class Stream:
def check_request_url(self) -> bool:
# Make sure that we are sending the request to the correct URL
- url = urlparse(self._request.url)
+ url = urlparse_cached(self._request)
return (
url.netloc == str(self._protocol.metadata["uri"].host, "utf-8")
or url.netloc == str(self._protocol.metadata["uri"].netloc, "utf-8")
@@ -193,8 +196,8 @@ class Stream:
== f'{self._protocol.metadata["ip_address"]}:{self._protocol.metadata["uri"].port}'
)
- def _get_request_headers(self) -> List[Tuple[str, str]]:
- url = urlparse(self._request.url)
+ def _get_request_headers(self) -> list[tuple[str, str]]:
+ url = urlparse_cached(self._request)
path = url.path
if url.query:
@@ -346,7 +349,7 @@ class Stream:
self._response["flow_controlled_size"], self.stream_id
)
- def receive_headers(self, headers: List[HeaderTuple]) -> None:
+ def receive_headers(self, headers: list[HeaderTuple]) -> None:
for name, value in headers:
self._response["headers"].appendlist(name, value)
@@ -379,7 +382,7 @@ class Stream:
def close(
self,
reason: StreamCloseReason,
- errors: Optional[List[BaseException]] = None,
+ errors: list[BaseException] | None = None,
from_protocol: bool = False,
) -> None:
"""Based on the reason sent we will handle each case."""
diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py
index 17c95f1ea..fcc94879a 100644
--- a/scrapy/core/scheduler.py
+++ b/scrapy/core/scheduler.py
@@ -4,22 +4,28 @@ import json
import logging
from abc import abstractmethod
from pathlib import Path
-from typing import TYPE_CHECKING, Any, Optional, Type, TypeVar, cast
+from typing import TYPE_CHECKING, Any, cast
-from twisted.internet.defer import Deferred
+# working around https://github.com/sphinx-doc/sphinx/issues/10400
+from twisted.internet.defer import Deferred # noqa: TC002
-from scrapy.crawler import Crawler
-from scrapy.dupefilters import BaseDupeFilter
-from scrapy.http.request import Request
-from scrapy.spiders import Spider
-from scrapy.statscollectors import StatsCollector
+from scrapy.spiders import Spider # noqa: TC001
from scrapy.utils.job import job_dir
-from scrapy.utils.misc import create_instance, load_object
+from scrapy.utils.misc import build_from_crawler, load_object
if TYPE_CHECKING:
+ # requires queuelib >= 1.6.2
+ from queuelib.queue import BaseQueue
+
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy.crawler import Crawler
+ from scrapy.dupefilters import BaseDupeFilter
+ from scrapy.http.request import Request
+ from scrapy.pqueues import ScrapyPriorityQueue
+ from scrapy.statscollectors import StatsCollector
+
logger = logging.getLogger(__name__)
@@ -67,7 +73,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
"""
return cls()
- def open(self, spider: Spider) -> Optional[Deferred]:
+ def open(self, spider: Spider) -> Deferred[None] | None:
"""
Called when the spider is opened by the engine. It receives the spider
instance as argument and it's useful to execute initialization code.
@@ -75,9 +81,8 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
:param spider: the spider object for the current crawl
:type spider: :class:`~scrapy.spiders.Spider`
"""
- pass
- def close(self, reason: str) -> Optional[Deferred]:
+ def close(self, reason: str) -> Deferred[None] | None:
"""
Called when the spider is closed by the engine. It receives the reason why the crawl
finished as argument and it's useful to execute cleaning code.
@@ -85,14 +90,13 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
:param reason: a string which describes the reason why the spider was closed
:type reason: :class:`str`
"""
- pass
@abstractmethod
def has_pending_requests(self) -> bool:
"""
``True`` if the scheduler has enqueued requests, ``False`` otherwise
"""
- raise NotImplementedError()
+ raise NotImplementedError
@abstractmethod
def enqueue_request(self, request: Request) -> bool:
@@ -106,10 +110,10 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
For reference, the default Scrapy scheduler returns ``False`` when the
request is rejected by the dupefilter.
"""
- raise NotImplementedError()
+ raise NotImplementedError
@abstractmethod
- def next_request(self) -> Optional[Request]:
+ def next_request(self) -> Request | None:
"""
Return the next :class:`~scrapy.http.Request` to be processed, or ``None``
to indicate that there are no requests to be considered ready at the moment.
@@ -118,10 +122,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
to the downloader in the current reactor cycle. The engine will continue
calling ``next_request`` until ``has_pending_requests`` is ``False``.
"""
- raise NotImplementedError()
-
-
-SchedulerTV = TypeVar("SchedulerTV", bound="Scheduler")
+ raise NotImplementedError
class Scheduler(BaseScheduler):
@@ -178,31 +179,31 @@ class Scheduler(BaseScheduler):
def __init__(
self,
dupefilter: BaseDupeFilter,
- jobdir: Optional[str] = None,
- dqclass=None,
- mqclass=None,
+ jobdir: str | None = None,
+ dqclass: type[BaseQueue] | None = None,
+ mqclass: type[BaseQueue] | None = None,
logunser: bool = False,
- stats: Optional[StatsCollector] = None,
- pqclass=None,
- crawler: Optional[Crawler] = None,
+ stats: StatsCollector | None = None,
+ pqclass: type[ScrapyPriorityQueue] | None = None,
+ crawler: Crawler | None = None,
):
self.df: BaseDupeFilter = dupefilter
- self.dqdir: Optional[str] = self._dqdir(jobdir)
- self.pqclass = pqclass
- self.dqclass = dqclass
- self.mqclass = mqclass
+ self.dqdir: str | None = self._dqdir(jobdir)
+ self.pqclass: type[ScrapyPriorityQueue] | None = pqclass
+ self.dqclass: type[BaseQueue] | None = dqclass
+ self.mqclass: type[BaseQueue] | None = mqclass
self.logunser: bool = logunser
- self.stats: Optional[StatsCollector] = stats
- self.crawler: Optional[Crawler] = crawler
+ self.stats: StatsCollector | None = stats
+ self.crawler: Crawler | None = crawler
@classmethod
- def from_crawler(cls: Type[SchedulerTV], crawler: Crawler) -> SchedulerTV:
+ def from_crawler(cls, crawler: Crawler) -> Self:
"""
Factory method, initializes the scheduler with arguments taken from the crawl settings
"""
dupefilter_cls = load_object(crawler.settings["DUPEFILTER_CLASS"])
return cls(
- dupefilter=create_instance(dupefilter_cls, crawler.settings, crawler),
+ dupefilter=build_from_crawler(dupefilter_cls, crawler),
jobdir=job_dir(crawler.settings),
dqclass=load_object(crawler.settings["SCHEDULER_DISK_QUEUE"]),
mqclass=load_object(crawler.settings["SCHEDULER_MEMORY_QUEUE"]),
@@ -215,18 +216,18 @@ class Scheduler(BaseScheduler):
def has_pending_requests(self) -> bool:
return len(self) > 0
- def open(self, spider: Spider) -> Optional[Deferred]:
+ def open(self, spider: Spider) -> Deferred[None] | None:
"""
(1) initialize the memory queue
(2) initialize the disk queue if the ``jobdir`` attribute is a valid directory
(3) return the result of the dupefilter's ``open`` method
"""
- self.spider = spider
- self.mqs = self._mq()
- self.dqs = self._dq() if self.dqdir else None
+ self.spider: Spider = spider
+ self.mqs: ScrapyPriorityQueue = self._mq()
+ self.dqs: ScrapyPriorityQueue | None = self._dq() if self.dqdir else None
return self.df.open()
- def close(self, reason: str) -> Optional[Deferred]:
+ def close(self, reason: str) -> Deferred[None] | None:
"""
(1) dump pending requests to disk if there is a disk queue
(2) return the result of the dupefilter's ``close`` method
@@ -260,7 +261,7 @@ class Scheduler(BaseScheduler):
self.stats.inc_value("scheduler/enqueued", spider=self.spider)
return True
- def next_request(self) -> Optional[Request]:
+ def next_request(self) -> Request | None:
"""
Return a :class:`~scrapy.http.Request` object from the memory queue,
falling back to the disk queue if the memory queue is empty.
@@ -269,7 +270,7 @@ class Scheduler(BaseScheduler):
Increment the appropriate stats, such as: ``scheduler/dequeued``,
``scheduler/dequeued/disk``, ``scheduler/dequeued/memory``.
"""
- request: Optional[Request] = self.mqs.pop()
+ request: Request | None = self.mqs.pop()
assert self.stats is not None
if request is not None:
self.stats.inc_value("scheduler/dequeued/memory", spider=self.spider)
@@ -309,35 +310,36 @@ class Scheduler(BaseScheduler):
assert self.stats is not None
self.stats.inc_value("scheduler/unserializable", spider=self.spider)
return False
- else:
- return True
+ return True
def _mqpush(self, request: Request) -> None:
self.mqs.push(request)
- def _dqpop(self) -> Optional[Request]:
+ def _dqpop(self) -> Request | None:
if self.dqs is not None:
return self.dqs.pop()
return None
- def _mq(self):
+ def _mq(self) -> ScrapyPriorityQueue:
"""Create a new priority queue instance, with in-memory storage"""
- return create_instance(
+ assert self.crawler
+ assert self.pqclass
+ return build_from_crawler(
self.pqclass,
- settings=None,
- crawler=self.crawler,
+ self.crawler,
downstream_queue_cls=self.mqclass,
key="",
)
- def _dq(self):
+ def _dq(self) -> ScrapyPriorityQueue:
"""Create a new priority queue instance, with disk storage"""
+ assert self.crawler
assert self.dqdir
+ assert self.pqclass
state = self._read_dqs_state(self.dqdir)
- q = create_instance(
+ q = build_from_crawler(
self.pqclass,
- settings=None,
- crawler=self.crawler,
+ self.crawler,
downstream_queue_cls=self.dqclass,
key=self.dqdir,
startprios=state,
@@ -350,7 +352,7 @@ class Scheduler(BaseScheduler):
)
return q
- def _dqdir(self, jobdir: Optional[str]) -> Optional[str]:
+ def _dqdir(self, jobdir: str | None) -> str | None:
"""Return a folder name to keep disk queue state at"""
if jobdir:
dqdir = Path(jobdir, "requests.queue")
@@ -359,13 +361,13 @@ class Scheduler(BaseScheduler):
return str(dqdir)
return None
- def _read_dqs_state(self, dqdir: str) -> list:
+ def _read_dqs_state(self, dqdir: str) -> list[int]:
path = Path(dqdir, "active.json")
if not path.exists():
return []
with path.open(encoding="utf-8") as f:
- return cast(list, json.load(f))
+ return cast(list[int], json.load(f))
- def _write_dqs_state(self, dqdir: str, state: list) -> None:
+ def _write_dqs_state(self, dqdir: str, state: list[int]) -> None:
with Path(dqdir, "active.json").open("w", encoding="utf-8") as f:
json.dump(state, f)
diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.py
index 8fb16b8a9..03301717d 100644
--- a/scrapy/core/scraper.py
+++ b/scrapy/core/scraper.py
@@ -1,23 +1,12 @@
"""This module implements the Scraper component which parses responses and
extracts information from them"""
+
from __future__ import annotations
import logging
from collections import deque
-from typing import (
- TYPE_CHECKING,
- Any,
- AsyncGenerator,
- AsyncIterable,
- Deque,
- Generator,
- Iterable,
- Optional,
- Set,
- Tuple,
- Type,
- Union,
-)
+from collections.abc import AsyncIterable, Iterator
+from typing import TYPE_CHECKING, Any, TypeVar, Union, cast
from itemadapter import is_item
from twisted.internet.defer import Deferred, inlineCallbacks
@@ -27,9 +16,6 @@ from scrapy import Spider, signals
from scrapy.core.spidermw import SpiderMiddlewareManager
from scrapy.exceptions import CloseSpider, DropItem, IgnoreRequest
from scrapy.http import Request, Response
-from scrapy.logformatter import LogFormatter
-from scrapy.pipelines import ItemPipelineManager
-from scrapy.signalmanager import SignalManager
from scrapy.utils.defer import (
aiter_errback,
defer_fail,
@@ -43,32 +29,40 @@ from scrapy.utils.misc import load_object, warn_on_generator_with_return_value
from scrapy.utils.spider import iterate_spider_output
if TYPE_CHECKING:
+ from collections.abc import Generator, Iterable
+
from scrapy.crawler import Crawler
-
-
-QueueTuple = Tuple[Union[Response, Failure], Request, Deferred]
+ from scrapy.logformatter import LogFormatter
+ from scrapy.pipelines import ItemPipelineManager
+ from scrapy.signalmanager import SignalManager
logger = logging.getLogger(__name__)
+_T = TypeVar("_T")
+_ParallelResult = list[tuple[bool, Iterator[Any]]]
+_HandleOutputDeferred = Deferred[Union[_ParallelResult, None]]
+QueueTuple = tuple[Union[Response, Failure], Request, _HandleOutputDeferred]
+
+
class Slot:
"""Scraper slot (one per running spider)"""
MIN_RESPONSE_SIZE = 1024
def __init__(self, max_active_size: int = 5000000):
- self.max_active_size = max_active_size
- self.queue: Deque[QueueTuple] = deque()
- self.active: Set[Request] = set()
+ self.max_active_size: int = max_active_size
+ self.queue: deque[QueueTuple] = deque()
+ self.active: set[Request] = set()
self.active_size: int = 0
self.itemproc_size: int = 0
- self.closing: Optional[Deferred] = None
+ self.closing: Deferred[Spider] | None = None
def add_response_request(
- self, result: Union[Response, Failure], request: Request
- ) -> Deferred:
- deferred: Deferred = Deferred()
+ self, result: Response | Failure, request: Request
+ ) -> _HandleOutputDeferred:
+ deferred: _HandleOutputDeferred = Deferred()
self.queue.append((result, request, deferred))
if isinstance(result, Response):
self.active_size += max(len(result.body), self.MIN_RESPONSE_SIZE)
@@ -81,9 +75,7 @@ class Slot:
self.active.add(request)
return response, request, deferred
- def finish_response(
- self, result: Union[Response, Failure], request: Request
- ) -> None:
+ def finish_response(self, result: Response | Failure, request: Request) -> None:
self.active.remove(request)
if isinstance(result, Response):
self.active_size -= max(len(result.body), self.MIN_RESPONSE_SIZE)
@@ -99,11 +91,11 @@ class Slot:
class Scraper:
def __init__(self, crawler: Crawler) -> None:
- self.slot: Optional[Slot] = None
+ self.slot: Slot | None = None
self.spidermw: SpiderMiddlewareManager = SpiderMiddlewareManager.from_crawler(
crawler
)
- itemproc_cls: Type[ItemPipelineManager] = load_object(
+ itemproc_cls: type[ItemPipelineManager] = load_object(
crawler.settings["ITEM_PROCESSOR"]
)
self.itemproc: ItemPipelineManager = itemproc_cls.from_crawler(crawler)
@@ -114,12 +106,12 @@ class Scraper:
self.logformatter: LogFormatter = crawler.logformatter
@inlineCallbacks
- def open_spider(self, spider: Spider) -> Generator[Deferred, Any, None]:
+ def open_spider(self, spider: Spider) -> Generator[Deferred[Any], Any, None]:
"""Open the given spider for scraping and allocate resources for it"""
self.slot = Slot(self.crawler.settings.getint("SCRAPER_SLOT_MAX_ACTIVE_SIZE"))
yield self.itemproc.open_spider(spider)
- def close_spider(self, spider: Spider) -> Deferred:
+ def close_spider(self, spider: Spider) -> Deferred[Spider]:
"""Close a spider being scraped and release its resources"""
if self.slot is None:
raise RuntimeError("Scraper slot not assigned")
@@ -138,13 +130,13 @@ class Scraper:
self.slot.closing.callback(spider)
def enqueue_scrape(
- self, result: Union[Response, Failure], request: Request, spider: Spider
- ) -> Deferred:
+ self, result: Response | Failure, request: Request, spider: Spider
+ ) -> _HandleOutputDeferred:
if self.slot is None:
raise RuntimeError("Scraper slot not assigned")
dfd = self.slot.add_response_request(result, request)
- def finish_scraping(_: Any) -> Any:
+ def finish_scraping(_: _T) -> _T:
assert self.slot is not None
self.slot.finish_response(result, request)
self._check_if_closing(spider)
@@ -170,8 +162,8 @@ class Scraper:
self._scrape(response, request, spider).chainDeferred(deferred)
def _scrape(
- self, result: Union[Response, Failure], request: Request, spider: Spider
- ) -> Deferred:
+ self, result: Response | Failure, request: Request, spider: Spider
+ ) -> _HandleOutputDeferred:
"""
Handle the downloaded response or failure through the spider callback/errback
"""
@@ -179,30 +171,35 @@ class Scraper:
raise TypeError(
f"Incorrect type: expected Response or Failure, got {type(result)}: {result!r}"
)
- dfd = self._scrape2(
+ dfd: Deferred[Iterable[Any] | AsyncIterable[Any]] = self._scrape2(
result, request, spider
) # returns spider's processed output
dfd.addErrback(self.handle_spider_error, request, result, spider)
- dfd.addCallback(self.handle_spider_output, request, result, spider)
- return dfd
+ dfd2: _HandleOutputDeferred = dfd.addCallback(
+ self.handle_spider_output, request, cast(Response, result), spider
+ )
+ return dfd2
def _scrape2(
- self, result: Union[Response, Failure], request: Request, spider: Spider
- ) -> Deferred:
+ self, result: Response | Failure, request: Request, spider: Spider
+ ) -> Deferred[Iterable[Any] | AsyncIterable[Any]]:
"""
Handle the different cases of request's result been a Response or a Failure
"""
if isinstance(result, Response):
- return self.spidermw.scrape_response(
+ # Deferreds are invariant so Mutable*Chain isn't matched to *Iterable
+ return self.spidermw.scrape_response( # type: ignore[return-value]
self.call_spider, result, request, spider
)
# else result is a Failure
dfd = self.call_spider(result, request, spider)
- return dfd.addErrback(self._log_download_errors, result, request, spider)
+ dfd.addErrback(self._log_download_errors, result, request, spider)
+ return dfd
def call_spider(
- self, result: Union[Response, Failure], request: Request, spider: Spider
- ) -> Deferred:
+ self, result: Response | Failure, request: Request, spider: Spider
+ ) -> Deferred[Iterable[Any] | AsyncIterable[Any]]:
+ dfd: Deferred[Any]
if isinstance(result, Response):
if getattr(result, "request", None) is None:
result.request = request
@@ -220,13 +217,16 @@ class Scraper:
if request.errback:
warn_on_generator_with_return_value(spider, request.errback)
dfd.addErrback(request.errback)
- return dfd.addCallback(iterate_spider_output)
+ dfd2: Deferred[Iterable[Any] | AsyncIterable[Any]] = dfd.addCallback(
+ iterate_spider_output
+ )
+ return dfd2
def handle_spider_error(
self,
_failure: Failure,
request: Request,
- response: Union[Response, Failure],
+ response: Response | Failure,
spider: Spider,
) -> None:
exc = _failure.value
@@ -253,14 +253,15 @@ class Scraper:
def handle_spider_output(
self,
- result: Union[Iterable, AsyncIterable],
+ result: Iterable[_T] | AsyncIterable[_T],
request: Request,
- response: Union[Response, Failure],
+ response: Response,
spider: Spider,
- ) -> Deferred:
+ ) -> _HandleOutputDeferred:
if not result:
return defer_succeed(None)
- it: Union[Generator, AsyncGenerator]
+ it: Iterable[_T] | AsyncIterable[_T]
+ dfd: Deferred[_ParallelResult]
if isinstance(result, AsyncIterable):
it = aiter_errback(
result, self.handle_spider_error, request, response, spider
@@ -285,23 +286,20 @@ class Scraper:
response,
spider,
)
- return dfd
+ # returning Deferred[_ParallelResult] instead of Deferred[Union[_ParallelResult, None]]
+ return dfd # type: ignore[return-value]
def _process_spidermw_output(
self, output: Any, request: Request, response: Response, spider: Spider
- ) -> Optional[Deferred]:
+ ) -> Deferred[Any] | None:
"""Process each Request/Item (given in the output parameter) returned
from the given spider
"""
- assert self.slot is not None # typing
if isinstance(output, Request):
assert self.crawler.engine is not None # typing
self.crawler.engine.crawl(request=output)
elif is_item(output):
- self.slot.itemproc_size += 1
- dfd = self.itemproc.process_item(output, spider)
- dfd.addBoth(self._itemproc_finished, output, response, spider)
- return dfd
+ return self.start_itemproc(output, response=response)
elif output is None:
pass
else:
@@ -313,13 +311,26 @@ class Scraper:
)
return None
+ def start_itemproc(self, item: Any, *, response: Response | None) -> Deferred[Any]:
+ """Send *item* to the item pipelines for processing.
+
+ *response* is the source of the item data. If the item does not come
+ from response data, e.g. it was hard-coded, set it to ``None``.
+ """
+ assert self.slot is not None # typing
+ assert self.crawler.spider is not None # typing
+ self.slot.itemproc_size += 1
+ dfd = self.itemproc.process_item(item, self.crawler.spider)
+ dfd.addBoth(self._itemproc_finished, item, response, self.crawler.spider)
+ return dfd
+
def _log_download_errors(
self,
spider_failure: Failure,
download_failure: Failure,
request: Request,
spider: Spider,
- ) -> Union[Failure, None]:
+ ) -> Failure | None:
"""Log and silence errors that come from the engine (typically download
errors that got propagated thru here).
@@ -353,8 +364,8 @@ class Scraper:
return None
def _itemproc_finished(
- self, output: Any, item: Any, response: Response, spider: Spider
- ) -> Deferred:
+ self, output: Any, item: Any, response: Response | None, spider: Spider
+ ) -> Deferred[Any]:
"""ItemProcessor finished for the given ``item`` and returned ``output``"""
assert self.slot is not None # typing
self.slot.itemproc_size -= 1
diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py
index dcf1a6dbc..a63ee40bf 100644
--- a/scrapy/core/spidermw.py
+++ b/scrapy/core/spidermw.py
@@ -3,22 +3,14 @@ Spider Middleware manager
See documentation in docs/topics/spider-middleware.rst
"""
+
+from __future__ import annotations
+
import logging
+from collections.abc import AsyncIterable, Callable, Iterable
from inspect import isasyncgenfunction, iscoroutine
from itertools import islice
-from typing import (
- Any,
- AsyncGenerator,
- AsyncIterable,
- Callable,
- Generator,
- Iterable,
- List,
- Optional,
- Tuple,
- Union,
- cast,
-)
+from typing import TYPE_CHECKING, Any, TypeVar, Union, cast
from twisted.internet.defer import Deferred, inlineCallbacks
from twisted.python.failure import Failure
@@ -27,7 +19,6 @@ from scrapy import Request, Spider
from scrapy.exceptions import _InvalidOutput
from scrapy.http import Response
from scrapy.middleware import MiddlewareManager
-from scrapy.settings import BaseSettings
from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen
from scrapy.utils.conf import build_component_list
from scrapy.utils.defer import (
@@ -38,10 +29,19 @@ from scrapy.utils.defer import (
)
from scrapy.utils.python import MutableAsyncChain, MutableChain
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from scrapy.settings import BaseSettings
+
+
logger = logging.getLogger(__name__)
-ScrapeFunc = Callable[[Union[Response, Failure], Request, Spider], Any]
+_T = TypeVar("_T")
+ScrapeFunc = Callable[
+ [Union[Response, Failure], Request, Spider], Union[Iterable[_T], AsyncIterable[_T]]
+]
def _isiterable(o: Any) -> bool:
@@ -56,7 +56,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
self.downgrade_warning_done = False
@classmethod
- def _get_mwlist_from_settings(cls, settings: BaseSettings) -> List[Any]:
+ def _get_mwlist_from_settings(cls, settings: BaseSettings) -> list[Any]:
return build_component_list(settings.getwithbase("SPIDER_MIDDLEWARES"))
def _add_middleware(self, mw: Any) -> None:
@@ -72,11 +72,11 @@ class SpiderMiddlewareManager(MiddlewareManager):
def _process_spider_input(
self,
- scrape_func: ScrapeFunc,
+ scrape_func: ScrapeFunc[_T],
response: Response,
request: Request,
spider: Spider,
- ) -> Any:
+ ) -> Iterable[_T] | AsyncIterable[_T]:
for method in self.methods["process_spider_input"]:
method = cast(Callable, method)
try:
@@ -97,32 +97,39 @@ class SpiderMiddlewareManager(MiddlewareManager):
self,
response: Response,
spider: Spider,
- iterable: Union[Iterable, AsyncIterable],
+ iterable: Iterable[_T] | AsyncIterable[_T],
exception_processor_index: int,
- recover_to: Union[MutableChain, MutableAsyncChain],
- ) -> Union[Generator, AsyncGenerator]:
- def process_sync(iterable: Iterable) -> Generator:
+ recover_to: MutableChain[_T] | MutableAsyncChain[_T],
+ ) -> Iterable[_T] | AsyncIterable[_T]:
+ def process_sync(iterable: Iterable[_T]) -> Iterable[_T]:
try:
- for r in iterable:
- yield r
+ yield from iterable
except Exception as ex:
- exception_result = self._process_spider_exception(
- response, spider, Failure(ex), exception_processor_index
+ exception_result = cast(
+ Union[Failure, MutableChain[_T]],
+ self._process_spider_exception(
+ response, spider, Failure(ex), exception_processor_index
+ ),
)
if isinstance(exception_result, Failure):
raise
+ assert isinstance(recover_to, MutableChain)
recover_to.extend(exception_result)
- async def process_async(iterable: AsyncIterable) -> AsyncGenerator:
+ async def process_async(iterable: AsyncIterable[_T]) -> AsyncIterable[_T]:
try:
async for r in iterable:
yield r
except Exception as ex:
- exception_result = self._process_spider_exception(
- response, spider, Failure(ex), exception_processor_index
+ exception_result = cast(
+ Union[Failure, MutableAsyncChain[_T]],
+ self._process_spider_exception(
+ response, spider, Failure(ex), exception_processor_index
+ ),
)
if isinstance(exception_result, Failure):
raise
+ assert isinstance(recover_to, MutableAsyncChain)
recover_to.extend(exception_result)
if isinstance(iterable, AsyncIterable):
@@ -135,7 +142,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
spider: Spider,
_failure: Failure,
start_index: int = 0,
- ) -> Union[Failure, MutableChain]:
+ ) -> Failure | MutableChain[_T] | MutableAsyncChain[_T]:
exception = _failure.value
# don't handle _InvalidOutput exception
if isinstance(exception, _InvalidOutput):
@@ -151,26 +158,29 @@ class SpiderMiddlewareManager(MiddlewareManager):
if _isiterable(result):
# stop exception handling by handing control over to the
# process_spider_output chain if an iterable has been returned
- dfd: Deferred = self._process_spider_output(
- response, spider, result, method_index + 1
+ dfd: Deferred[MutableChain[_T] | MutableAsyncChain[_T]] = (
+ self._process_spider_output(
+ response, spider, result, method_index + 1
+ )
)
# _process_spider_output() returns a Deferred only because of downgrading so this can be
# simplified when downgrading is removed.
if dfd.called:
# the result is available immediately if _process_spider_output didn't do downgrading
- return cast(MutableChain, dfd.result)
+ return cast(
+ Union[MutableChain[_T], MutableAsyncChain[_T]], dfd.result
+ )
# we forbid waiting here because otherwise we would need to return a deferred from
# _process_spider_exception too, which complicates the architecture
msg = f"Async iterable returned from {method.__qualname__} cannot be downgraded"
raise _InvalidOutput(msg)
- elif result is None:
+ if result is None:
continue
- else:
- msg = (
- f"{method.__qualname__} must return None "
- f"or an iterable, got {type(result)}"
- )
- raise _InvalidOutput(msg)
+ msg = (
+ f"{method.__qualname__} must return None "
+ f"or an iterable, got {type(result)}"
+ )
+ raise _InvalidOutput(msg)
return _failure
# This method cannot be made async def, as _process_spider_exception relies on the Deferred result
@@ -181,12 +191,12 @@ class SpiderMiddlewareManager(MiddlewareManager):
self,
response: Response,
spider: Spider,
- result: Union[Iterable, AsyncIterable],
+ result: Iterable[_T] | AsyncIterable[_T],
start_index: int = 0,
- ) -> Generator[Deferred, Any, Union[MutableChain, MutableAsyncChain]]:
+ ) -> Generator[Deferred[Any], Any, MutableChain[_T] | MutableAsyncChain[_T]]:
# items in this iterable do not need to go through the process_spider_output
# chain, they went through it already from the process_spider_exception method
- recovered: Union[MutableChain, MutableAsyncChain]
+ recovered: MutableChain[_T] | MutableAsyncChain[_T]
last_result_is_async = isinstance(result, AsyncIterable)
if last_result_is_async:
recovered = MutableAsyncChain()
@@ -237,8 +247,10 @@ class SpiderMiddlewareManager(MiddlewareManager):
# might fail directly if the output value is not a generator
result = method(response=response, result=result, spider=spider)
except Exception as ex:
- exception_result = self._process_spider_exception(
- response, spider, Failure(ex), method_index + 1
+ exception_result: Failure | MutableChain[_T] | MutableAsyncChain[_T] = (
+ self._process_spider_exception(
+ response, spider, Failure(ex), method_index + 1
+ )
)
if isinstance(exception_result, Failure):
raise
@@ -267,16 +279,22 @@ class SpiderMiddlewareManager(MiddlewareManager):
return MutableChain(result, recovered) # type: ignore[arg-type]
async def _process_callback_output(
- self, response: Response, spider: Spider, result: Union[Iterable, AsyncIterable]
- ) -> Union[MutableChain, MutableAsyncChain]:
- recovered: Union[MutableChain, MutableAsyncChain]
+ self,
+ response: Response,
+ spider: Spider,
+ result: Iterable[_T] | AsyncIterable[_T],
+ ) -> MutableChain[_T] | MutableAsyncChain[_T]:
+ recovered: MutableChain[_T] | MutableAsyncChain[_T]
if isinstance(result, AsyncIterable):
recovered = MutableAsyncChain()
else:
recovered = MutableChain()
result = self._evaluate_iterable(response, spider, result, 0, recovered)
result = await maybe_deferred_to_future(
- self._process_spider_output(response, spider, result)
+ cast(
+ "Deferred[Iterable[_T] | AsyncIterable[_T]]",
+ self._process_spider_output(response, spider, result),
+ )
)
if isinstance(result, AsyncIterable):
return MutableAsyncChain(result, recovered)
@@ -287,41 +305,43 @@ class SpiderMiddlewareManager(MiddlewareManager):
def scrape_response(
self,
- scrape_func: ScrapeFunc,
+ scrape_func: ScrapeFunc[_T],
response: Response,
request: Request,
spider: Spider,
- ) -> Deferred:
+ ) -> Deferred[MutableChain[_T] | MutableAsyncChain[_T]]:
async def process_callback_output(
- result: Union[Iterable, AsyncIterable]
- ) -> Union[MutableChain, MutableAsyncChain]:
+ result: Iterable[_T] | AsyncIterable[_T],
+ ) -> MutableChain[_T] | MutableAsyncChain[_T]:
return await self._process_callback_output(response, spider, result)
- def process_spider_exception(_failure: Failure) -> Union[Failure, MutableChain]:
+ def process_spider_exception(
+ _failure: Failure,
+ ) -> Failure | MutableChain[_T] | MutableAsyncChain[_T]:
return self._process_spider_exception(response, spider, _failure)
- dfd = mustbe_deferred(
+ dfd: Deferred[Iterable[_T] | AsyncIterable[_T]] = mustbe_deferred(
self._process_spider_input, scrape_func, response, request, spider
)
- dfd.addCallbacks(
- callback=deferred_f_from_coro_f(process_callback_output),
- errback=process_spider_exception,
+ dfd2: Deferred[MutableChain[_T] | MutableAsyncChain[_T]] = dfd.addCallback(
+ deferred_f_from_coro_f(process_callback_output)
)
- return dfd
+ dfd2.addErrback(process_spider_exception)
+ return dfd2
def process_start_requests(
self, start_requests: Iterable[Request], spider: Spider
- ) -> Deferred:
+ ) -> Deferred[Iterable[Request]]:
return self._process_chain("process_start_requests", start_requests, spider)
# This method is only needed until _async compatibility methods are removed.
@staticmethod
def _get_async_method_pair(
mw: Any, methodname: str
- ) -> Union[None, Callable, Tuple[Callable, Callable]]:
- normal_method: Optional[Callable] = getattr(mw, methodname, None)
+ ) -> Callable | tuple[Callable, Callable] | None:
+ normal_method: Callable | None = getattr(mw, methodname, None)
methodname_async = methodname + "_async"
- async_method: Optional[Callable] = getattr(mw, methodname_async, None)
+ async_method: Callable | None = getattr(mw, methodname_async, None)
if not async_method:
return normal_method
if not normal_method:
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index 1d3a11208..05af1bf8a 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -3,8 +3,7 @@ from __future__ import annotations
import logging
import pprint
import signal
-import warnings
-from typing import TYPE_CHECKING, Any, Dict, Generator, Optional, Set, Type, Union, cast
+from typing import TYPE_CHECKING, Any, TypeVar, cast
from twisted.internet.defer import (
Deferred,
@@ -12,25 +11,15 @@ from twisted.internet.defer import (
inlineCallbacks,
maybeDeferred,
)
-
-try:
- # zope >= 5.0 only supports MultipleInvalid
- from zope.interface.exceptions import MultipleInvalid
-except ImportError:
- MultipleInvalid = None
-
from zope.interface.verify import verifyClass
from scrapy import Spider, signals
from scrapy.addons import AddonManager
from scrapy.core.engine import ExecutionEngine
-from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.extension import ExtensionManager
from scrapy.interfaces import ISpiderLoader
-from scrapy.logformatter import LogFormatter
from scrapy.settings import BaseSettings, Settings, overridden_settings
from scrapy.signalmanager import SignalManager
-from scrapy.statscollectors import StatsCollector
from scrapy.utils.log import (
LogCounterHandler,
configure_logging,
@@ -39,7 +28,7 @@ from scrapy.utils.log import (
log_reactor_info,
log_scrapy_info,
)
-from scrapy.utils.misc import create_instance, load_object
+from scrapy.utils.misc import build_from_crawler, load_object
from scrapy.utils.ossignal import install_shutdown_handlers, signal_names
from scrapy.utils.reactor import (
install_reactor,
@@ -49,17 +38,24 @@ from scrapy.utils.reactor import (
)
if TYPE_CHECKING:
+ from collections.abc import Generator, Iterable
+
+ from scrapy.logformatter import LogFormatter
+ from scrapy.spiderloader import SpiderLoader
+ from scrapy.statscollectors import StatsCollector
from scrapy.utils.request import RequestFingerprinter
logger = logging.getLogger(__name__)
+_T = TypeVar("_T")
+
class Crawler:
def __init__(
self,
- spidercls: Type[Spider],
- settings: Union[None, Dict[str, Any], Settings] = None,
+ spidercls: type[Spider],
+ settings: dict[str, Any] | Settings | None = None,
init_reactor: bool = False,
):
if isinstance(spidercls, Spider):
@@ -68,7 +64,7 @@ class Crawler:
if isinstance(settings, dict) or settings is None:
settings = Settings(settings)
- self.spidercls: Type[Spider] = spidercls
+ self.spidercls: type[Spider] = spidercls
self.settings: Settings = settings.copy()
self.spidercls.update_settings(self.settings)
self._update_root_log_handler()
@@ -80,12 +76,12 @@ class Crawler:
self.crawling: bool = False
self._started: bool = False
- self.extensions: Optional[ExtensionManager] = None
- self.stats: Optional[StatsCollector] = None
- self.logformatter: Optional[LogFormatter] = None
- self.request_fingerprinter: Optional[RequestFingerprinter] = None
- self.spider: Optional[Spider] = None
- self.engine: Optional[ExecutionEngine] = None
+ self.extensions: ExtensionManager | None = None
+ self.stats: StatsCollector | None = None
+ self.logformatter: LogFormatter | None = None
+ self.request_fingerprinter: RequestFingerprinter | None = None
+ self.spider: Spider | None = None
+ self.engine: ExecutionEngine | None = None
def _update_root_log_handler(self) -> None:
if get_scrapy_root_handler() is not None:
@@ -106,13 +102,12 @@ class Crawler:
self.__remove_handler = lambda: logging.root.removeHandler(handler)
self.signals.connect(self.__remove_handler, signals.engine_stopped)
- lf_cls: Type[LogFormatter] = load_object(self.settings["LOG_FORMATTER"])
+ lf_cls: type[LogFormatter] = load_object(self.settings["LOG_FORMATTER"])
self.logformatter = lf_cls.from_crawler(self)
- self.request_fingerprinter = create_instance(
+ self.request_fingerprinter = build_from_crawler(
load_object(self.settings["REQUEST_FINGERPRINTER_CLASS"]),
- settings=self.settings,
- crawler=self,
+ self,
)
reactor_class: str = self.settings["TWISTED_REACTOR"]
@@ -130,6 +125,8 @@ class Crawler:
if is_asyncio_reactor_installed() and event_loop:
verify_installed_asyncio_event_loop(event_loop)
+ log_reactor_info()
+
self.extensions = ExtensionManager.from_crawler(self)
self.settings.freeze()
@@ -139,14 +136,12 @@ class Crawler:
)
@inlineCallbacks
- def crawl(self, *args: Any, **kwargs: Any) -> Generator[Deferred, Any, None]:
+ def crawl(self, *args: Any, **kwargs: Any) -> Generator[Deferred[Any], Any, None]:
if self.crawling:
raise RuntimeError("Crawling already taking place")
if self._started:
- warnings.warn(
- "Running Crawler.crawl() more than once is deprecated.",
- ScrapyDeprecationWarning,
- stacklevel=2,
+ raise RuntimeError(
+ "Cannot run Crawler.crawl() more than once on the same instance."
)
self.crawling = self._started = True
@@ -171,7 +166,7 @@ class Crawler:
return ExecutionEngine(self, lambda _: self.stop())
@inlineCallbacks
- def stop(self) -> Generator[Deferred, Any, None]:
+ def stop(self) -> Generator[Deferred[Any], Any, None]:
"""Starts a graceful stop of the crawler and returns a deferred that is
fired when the crawler is stopped."""
if self.crawling:
@@ -179,6 +174,92 @@ class Crawler:
assert self.engine
yield maybeDeferred(self.engine.stop)
+ @staticmethod
+ def _get_component(
+ component_class: type[_T], components: Iterable[Any]
+ ) -> _T | None:
+ for component in components:
+ if isinstance(component, component_class):
+ return component
+ return None
+
+ def get_addon(self, cls: type[_T]) -> _T | None:
+ """Return the run-time instance of an :ref:`add-on ` of
+ the specified class or a subclass, or ``None`` if none is found.
+
+ .. versionadded:: 2.12
+ """
+ return self._get_component(cls, self.addons.addons)
+
+ def get_downloader_middleware(self, cls: type[_T]) -> _T | None:
+ """Return the run-time instance of a :ref:`downloader middleware
+ ` of the specified class or a subclass,
+ or ``None`` if none is found.
+
+ .. versionadded:: 2.12
+
+ This method can only be called after the crawl engine has been created,
+ e.g. at signals :signal:`engine_started` or :signal:`spider_opened`.
+ """
+ if not self.engine:
+ raise RuntimeError(
+ "Crawler.get_downloader_middleware() can only be called after "
+ "the crawl engine has been created."
+ )
+ return self._get_component(cls, self.engine.downloader.middleware.middlewares)
+
+ def get_extension(self, cls: type[_T]) -> _T | None:
+ """Return the run-time instance of an :ref:`extension
+ ` of the specified class or a subclass,
+ or ``None`` if none is found.
+
+ .. versionadded:: 2.12
+
+ This method can only be called after the extension manager has been
+ created, e.g. at signals :signal:`engine_started` or
+ :signal:`spider_opened`.
+ """
+ if not self.extensions:
+ raise RuntimeError(
+ "Crawler.get_extension() can only be called after the "
+ "extension manager has been created."
+ )
+ return self._get_component(cls, self.extensions.middlewares)
+
+ def get_item_pipeline(self, cls: type[_T]) -> _T | None:
+ """Return the run-time instance of a :ref:`item pipeline
+ ` of the specified class or a subclass, or
+ ``None`` if none is found.
+
+ .. versionadded:: 2.12
+
+ This method can only be called after the crawl engine has been created,
+ e.g. at signals :signal:`engine_started` or :signal:`spider_opened`.
+ """
+ if not self.engine:
+ raise RuntimeError(
+ "Crawler.get_item_pipeline() can only be called after the "
+ "crawl engine has been created."
+ )
+ return self._get_component(cls, self.engine.scraper.itemproc.middlewares)
+
+ def get_spider_middleware(self, cls: type[_T]) -> _T | None:
+ """Return the run-time instance of a :ref:`spider middleware
+ ` of the specified class or a subclass, or
+ ``None`` if none is found.
+
+ .. versionadded:: 2.12
+
+ This method can only be called after the crawl engine has been created,
+ e.g. at signals :signal:`engine_started` or :signal:`spider_opened`.
+ """
+ if not self.engine:
+ raise RuntimeError(
+ "Crawler.get_spider_middleware() can only be called after the "
+ "crawl engine has been created."
+ )
+ return self._get_component(cls, self.engine.scraper.spidermw.middlewares)
+
class CrawlerRunner:
"""
@@ -200,28 +281,28 @@ class CrawlerRunner:
)
@staticmethod
- def _get_spider_loader(settings: BaseSettings):
+ def _get_spider_loader(settings: BaseSettings) -> SpiderLoader:
"""Get SpiderLoader instance from settings"""
cls_path = settings.get("SPIDER_LOADER_CLASS")
loader_cls = load_object(cls_path)
verifyClass(ISpiderLoader, loader_cls)
- return loader_cls.from_settings(settings.frozencopy())
+ return cast("SpiderLoader", loader_cls.from_settings(settings.frozencopy()))
- def __init__(self, settings: Union[Dict[str, Any], Settings, None] = None):
+ def __init__(self, settings: dict[str, Any] | Settings | None = None):
if isinstance(settings, dict) or settings is None:
settings = Settings(settings)
- self.settings = settings
- self.spider_loader = self._get_spider_loader(settings)
- self._crawlers: Set[Crawler] = set()
- self._active: Set[Deferred] = set()
+ self.settings: Settings = settings
+ self.spider_loader: SpiderLoader = self._get_spider_loader(settings)
+ self._crawlers: set[Crawler] = set()
+ self._active: set[Deferred[None]] = set()
self.bootstrap_failed = False
def crawl(
self,
- crawler_or_spidercls: Union[Type[Spider], str, Crawler],
+ crawler_or_spidercls: type[Spider] | str | Crawler,
*args: Any,
**kwargs: Any,
- ) -> Deferred:
+ ) -> Deferred[None]:
"""
Run a crawler with the provided arguments.
@@ -251,12 +332,12 @@ class CrawlerRunner:
crawler = self.create_crawler(crawler_or_spidercls)
return self._crawl(crawler, *args, **kwargs)
- def _crawl(self, crawler: Crawler, *args: Any, **kwargs: Any) -> Deferred:
+ def _crawl(self, crawler: Crawler, *args: Any, **kwargs: Any) -> Deferred[None]:
self.crawlers.add(crawler)
d = crawler.crawl(*args, **kwargs)
self._active.add(d)
- def _done(result: Any) -> Any:
+ def _done(result: _T) -> _T:
self.crawlers.discard(crawler)
self._active.discard(d)
self.bootstrap_failed |= not getattr(crawler, "spider", None)
@@ -265,7 +346,7 @@ class CrawlerRunner:
return d.addBoth(_done)
def create_crawler(
- self, crawler_or_spidercls: Union[Type[Spider], str, Crawler]
+ self, crawler_or_spidercls: type[Spider] | str | Crawler
) -> Crawler:
"""
Return a :class:`~scrapy.crawler.Crawler` object.
@@ -286,13 +367,12 @@ class CrawlerRunner:
return crawler_or_spidercls
return self._create_crawler(crawler_or_spidercls)
- def _create_crawler(self, spidercls: Union[str, Type[Spider]]) -> Crawler:
+ def _create_crawler(self, spidercls: str | type[Spider]) -> Crawler:
if isinstance(spidercls, str):
spidercls = self.spider_loader.load(spidercls)
- # temporary cast until self.spider_loader is typed
- return Crawler(cast(Type[Spider], spidercls), self.settings)
+ return Crawler(spidercls, self.settings)
- def stop(self) -> Deferred:
+ def stop(self) -> Deferred[Any]:
"""
Stops simultaneously all the crawling jobs taking place.
@@ -301,7 +381,7 @@ class CrawlerRunner:
return DeferredList([c.stop() for c in list(self.crawlers)])
@inlineCallbacks
- def join(self) -> Generator[Deferred, Any, None]:
+ def join(self) -> Generator[Deferred[Any], Any, None]:
"""
join()
@@ -338,13 +418,13 @@ class CrawlerProcess(CrawlerRunner):
def __init__(
self,
- settings: Union[Dict[str, Any], Settings, None] = None,
+ settings: dict[str, Any] | Settings | None = None,
install_root_handler: bool = True,
):
super().__init__(settings)
configure_logging(self.settings, install_root_handler)
log_scrapy_info(self.settings)
- self._initialized_reactor = False
+ self._initialized_reactor: bool = False
def _signal_shutdown(self, signum: int, _: Any) -> None:
from twisted.internet import reactor
@@ -367,15 +447,13 @@ class CrawlerProcess(CrawlerRunner):
)
reactor.callFromThread(self._stop_reactor)
- def _create_crawler(self, spidercls: Union[Type[Spider], str]) -> Crawler:
+ def _create_crawler(self, spidercls: type[Spider] | str) -> Crawler:
if isinstance(spidercls, str):
spidercls = self.spider_loader.load(spidercls)
init_reactor = not self._initialized_reactor
self._initialized_reactor = True
# temporary cast until self.spider_loader is typed
- return Crawler(
- cast(Type[Spider], spidercls), self.settings, init_reactor=init_reactor
- )
+ return Crawler(spidercls, self.settings, init_reactor=init_reactor)
def start(
self, stop_after_crawl: bool = True, install_signal_handlers: bool = True
@@ -404,7 +482,9 @@ class CrawlerProcess(CrawlerRunner):
d.addBoth(self._stop_reactor)
resolver_class = load_object(self.settings["DNS_RESOLVER"])
- resolver = create_instance(resolver_class, self.settings, self, reactor=reactor)
+ # We pass self, which is CrawlerProcess, instead of Crawler here,
+ # which works because the default resolvers only use crawler.settings.
+ resolver = build_from_crawler(resolver_class, self, reactor=reactor) # type: ignore[arg-type]
resolver.install_on_reactor()
tp = reactor.getThreadPool()
tp.adjustPoolsize(maxthreads=self.settings.getint("REACTOR_THREADPOOL_MAXSIZE"))
@@ -415,7 +495,7 @@ class CrawlerProcess(CrawlerRunner):
)
reactor.run(installSignalHandlers=install_signal_handlers) # blocking call
- def _graceful_stop_reactor(self) -> Deferred:
+ def _graceful_stop_reactor(self) -> Deferred[Any]:
d = self.stop()
d.addBoth(self._stop_reactor)
return d
diff --git a/scrapy/downloadermiddlewares/ajaxcrawl.py b/scrapy/downloadermiddlewares/ajaxcrawl.py
index 0e757e4be..166192b4f 100644
--- a/scrapy/downloadermiddlewares/ajaxcrawl.py
+++ b/scrapy/downloadermiddlewares/ajaxcrawl.py
@@ -2,27 +2,28 @@ from __future__ import annotations
import logging
import re
-from typing import TYPE_CHECKING, Union
+from typing import TYPE_CHECKING
from w3lib import html
-from scrapy import Request, Spider
-from scrapy.crawler import Crawler
from scrapy.exceptions import NotConfigured
from scrapy.http import HtmlResponse, Response
-from scrapy.settings import BaseSettings
if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.settings import BaseSettings
+
+
logger = logging.getLogger(__name__)
class AjaxCrawlMiddleware:
"""
Handle 'AJAX crawlable' pages marked as crawlable via meta tag.
- For more info see https://developers.google.com/webmasters/ajax-crawling/docs/getting-started.
"""
def __init__(self, settings: BaseSettings):
@@ -41,7 +42,7 @@ class AjaxCrawlMiddleware:
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
if not isinstance(response, HtmlResponse) or response.status != 200:
return response
@@ -68,8 +69,7 @@ class AjaxCrawlMiddleware:
def _has_ajax_crawlable_variant(self, response: Response) -> bool:
"""
- Return True if a page without hash fragment could be "AJAX crawlable"
- according to https://developers.google.com/webmasters/ajax-crawling/docs/getting-started.
+ Return True if a page without hash fragment could be "AJAX crawlable".
"""
body = response.text[: self.lookup_bytes]
return _has_ajaxcrawlable_meta(body)
diff --git a/scrapy/downloadermiddlewares/cookies.py b/scrapy/downloadermiddlewares/cookies.py
index 85781efd6..43348f632 100644
--- a/scrapy/downloadermiddlewares/cookies.py
+++ b/scrapy/downloadermiddlewares/cookies.py
@@ -2,22 +2,10 @@ from __future__ import annotations
import logging
from collections import defaultdict
-from http.cookiejar import Cookie
-from typing import (
- TYPE_CHECKING,
- Any,
- DefaultDict,
- Dict,
- Iterable,
- Optional,
- Sequence,
- Union,
-)
+from typing import TYPE_CHECKING, Any
from tldextract import TLDExtract
-from scrapy import Request, Spider
-from scrapy.crawler import Crawler
from scrapy.exceptions import NotConfigured
from scrapy.http import Response
from scrapy.http.cookies import CookieJar
@@ -25,14 +13,22 @@ from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.python import to_unicode
if TYPE_CHECKING:
+ from collections.abc import Iterable, Sequence
+ from http.cookiejar import Cookie
+
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.http.request import VerboseCookie
+
logger = logging.getLogger(__name__)
_split_domain = TLDExtract(include_psl_private_domains=True)
+_UNSET = object()
def _is_public_domain(domain: str) -> bool:
@@ -44,7 +40,7 @@ class CookiesMiddleware:
"""This middleware enables working with sites that need cookies"""
def __init__(self, debug: bool = False):
- self.jars: DefaultDict[Any, CookieJar] = defaultdict(CookieJar)
+ self.jars: defaultdict[Any, CookieJar] = defaultdict(CookieJar)
self.debug: bool = debug
@classmethod
@@ -74,7 +70,7 @@ class CookiesMiddleware:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
if request.meta.get("dont_merge_cookies", False):
return None
@@ -91,7 +87,7 @@ class CookiesMiddleware:
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
if request.meta.get("dont_merge_cookies", False):
return response
@@ -127,35 +123,44 @@ class CookiesMiddleware:
msg = f"Received cookies from: {response}\n{cookies}"
logger.debug(msg, extra={"spider": spider})
- def _format_cookie(self, cookie: Dict[str, Any], request: Request) -> Optional[str]:
+ def _format_cookie(self, cookie: VerboseCookie, request: Request) -> str | None:
"""
Given a dict consisting of cookie components, return its string representation.
Decode from bytes if necessary.
"""
decoded = {}
+ flags = set()
for key in ("name", "value", "path", "domain"):
- if cookie.get(key) is None:
+ value = cookie.get(key)
+ if value is None:
if key in ("name", "value"):
msg = f"Invalid cookie found in request {request}: {cookie} ('{key}' is missing)"
logger.warning(msg)
return None
continue
- if isinstance(cookie[key], (bool, float, int, str)):
- decoded[key] = str(cookie[key])
+ if isinstance(value, (bool, float, int, str)):
+ decoded[key] = str(value)
else:
+ assert isinstance(value, bytes)
try:
- decoded[key] = cookie[key].decode("utf8")
+ decoded[key] = value.decode("utf8")
except UnicodeDecodeError:
logger.warning(
"Non UTF-8 encoded cookie found in request %s: %s",
request,
cookie,
)
- decoded[key] = cookie[key].decode("latin1", errors="replace")
-
+ decoded[key] = value.decode("latin1", errors="replace")
+ for flag in ("secure",):
+ value = cookie.get(flag, _UNSET)
+ if value is _UNSET or not value:
+ continue
+ flags.add(flag)
cookie_str = f"{decoded.pop('name')}={decoded.pop('value')}"
for key, value in decoded.items(): # path, domain
cookie_str += f"; {key.capitalize()}={value}"
+ for flag in flags: # secure
+ cookie_str += f"; {flag.capitalize()}"
return cookie_str
def _get_request_cookies(
@@ -166,11 +171,13 @@ class CookiesMiddleware:
"""
if not request.cookies:
return []
- cookies: Iterable[Dict[str, Any]]
+ cookies: Iterable[VerboseCookie]
if isinstance(request.cookies, dict):
- cookies = ({"name": k, "value": v} for k, v in request.cookies.items())
+ cookies = tuple({"name": k, "value": v} for k, v in request.cookies.items())
else:
cookies = request.cookies
+ for cookie in cookies:
+ cookie.setdefault("secure", urlparse_cached(request).scheme == "https")
formatted = filter(None, (self._format_cookie(c, request) for c in cookies))
response = Response(request.url, headers={"Set-Cookie": formatted})
return jar.make_cookies(response, request)
diff --git a/scrapy/downloadermiddlewares/defaultheaders.py b/scrapy/downloadermiddlewares/defaultheaders.py
index 8aec37cf1..d58b4490b 100644
--- a/scrapy/downloadermiddlewares/defaultheaders.py
+++ b/scrapy/downloadermiddlewares/defaultheaders.py
@@ -3,23 +3,27 @@ DefaultHeaders downloader middleware
See documentation in docs/topics/downloader-middleware.rst
"""
+
from __future__ import annotations
-from typing import TYPE_CHECKING, Iterable, Tuple, Union
+from typing import TYPE_CHECKING
-from scrapy import Request, Spider
-from scrapy.crawler import Crawler
-from scrapy.http import Response
from scrapy.utils.python import without_none_values
if TYPE_CHECKING:
+ from collections.abc import Iterable
+
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
class DefaultHeadersMiddleware:
- def __init__(self, headers: Iterable[Tuple[str, str]]):
- self._headers: Iterable[Tuple[str, str]] = headers
+ def __init__(self, headers: Iterable[tuple[str, str]]):
+ self._headers: Iterable[tuple[str, str]] = headers
@classmethod
def from_crawler(cls, crawler: Crawler) -> Self:
@@ -28,7 +32,7 @@ class DefaultHeadersMiddleware:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
for k, v in self._headers:
request.headers.setdefault(k, v)
return None
diff --git a/scrapy/downloadermiddlewares/downloadtimeout.py b/scrapy/downloadermiddlewares/downloadtimeout.py
index 1c904c05b..28456c697 100644
--- a/scrapy/downloadermiddlewares/downloadtimeout.py
+++ b/scrapy/downloadermiddlewares/downloadtimeout.py
@@ -3,18 +3,20 @@ Download timeout middleware
See documentation in docs/topics/downloader-middleware.rst
"""
+
from __future__ import annotations
-from typing import TYPE_CHECKING, Union
+from typing import TYPE_CHECKING
from scrapy import Request, Spider, signals
-from scrapy.crawler import Crawler
-from scrapy.http import Response
if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
class DownloadTimeoutMiddleware:
def __init__(self, timeout: float = 180):
@@ -31,7 +33,7 @@ class DownloadTimeoutMiddleware:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
if self._timeout:
request.meta.setdefault("download_timeout", self._timeout)
return None
diff --git a/scrapy/downloadermiddlewares/httpauth.py b/scrapy/downloadermiddlewares/httpauth.py
index 63490a37a..b74140ee1 100644
--- a/scrapy/downloadermiddlewares/httpauth.py
+++ b/scrapy/downloadermiddlewares/httpauth.py
@@ -6,19 +6,20 @@ See documentation in docs/topics/downloader-middleware.rst
from __future__ import annotations
-from typing import TYPE_CHECKING, Union
+from typing import TYPE_CHECKING
from w3lib.http import basic_auth_header
from scrapy import Request, Spider, signals
-from scrapy.crawler import Crawler
-from scrapy.http import Response
from scrapy.utils.url import url_is_from_any_domain
if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
class HttpAuthMiddleware:
"""Set Basic HTTP Authorization header
@@ -39,7 +40,7 @@ class HttpAuthMiddleware:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
auth = getattr(self, "auth", None)
if auth and b"Authorization" not in request.headers:
if not self.domain or url_is_from_any_domain(request.url, [self.domain]):
diff --git a/scrapy/downloadermiddlewares/httpcache.py b/scrapy/downloadermiddlewares/httpcache.py
index 971473403..3892dba23 100644
--- a/scrapy/downloadermiddlewares/httpcache.py
+++ b/scrapy/downloadermiddlewares/httpcache.py
@@ -1,7 +1,7 @@
from __future__ import annotations
from email.utils import formatdate
-from typing import TYPE_CHECKING, Optional, Union
+from typing import TYPE_CHECKING
from twisted.internet import defer
from twisted.internet.error import (
@@ -16,19 +16,20 @@ from twisted.internet.error import (
from twisted.web.client import ResponseFailed
from scrapy import signals
-from scrapy.crawler import Crawler
from scrapy.exceptions import IgnoreRequest, NotConfigured
-from scrapy.http.request import Request
-from scrapy.http.response import Response
-from scrapy.settings import Settings
-from scrapy.spiders import Spider
-from scrapy.statscollectors import StatsCollector
from scrapy.utils.misc import load_object
if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy.crawler import Crawler
+ from scrapy.http.request import Request
+ from scrapy.http.response import Response
+ from scrapy.settings import Settings
+ from scrapy.spiders import Spider
+ from scrapy.statscollectors import StatsCollector
+
class HttpCacheMiddleware:
DOWNLOAD_EXCEPTIONS = (
@@ -68,7 +69,7 @@ class HttpCacheMiddleware:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
if request.meta.get("dont_cache", False):
return None
@@ -78,7 +79,7 @@ class HttpCacheMiddleware:
return None
# Look for cached response and check if expired
- cachedresponse: Optional[Response] = self.storage.retrieve_response(
+ cachedresponse: Response | None = self.storage.retrieve_response(
spider, request
)
if cachedresponse is None:
@@ -102,7 +103,7 @@ class HttpCacheMiddleware:
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
if request.meta.get("dont_cache", False):
return response
@@ -117,7 +118,7 @@ class HttpCacheMiddleware:
response.headers["Date"] = formatdate(usegmt=True)
# Do not validate first-hand responses
- cachedresponse: Optional[Response] = request.meta.pop("cached_response", None)
+ cachedresponse: Response | None = request.meta.pop("cached_response", None)
if cachedresponse is None:
self.stats.inc_value("httpcache/firsthand", spider=spider)
self._cache_response(spider, response, request, cachedresponse)
@@ -133,8 +134,8 @@ class HttpCacheMiddleware:
def process_exception(
self, request: Request, exception: Exception, spider: Spider
- ) -> Union[Request, Response, None]:
- cachedresponse: Optional[Response] = request.meta.pop("cached_response", None)
+ ) -> Request | Response | None:
+ cachedresponse: Response | None = request.meta.pop("cached_response", None)
if cachedresponse is not None and isinstance(
exception, self.DOWNLOAD_EXCEPTIONS
):
@@ -147,7 +148,7 @@ class HttpCacheMiddleware:
spider: Spider,
response: Response,
request: Request,
- cachedresponse: Optional[Response],
+ cachedresponse: Response | None,
) -> None:
if self.policy.should_cache_response(response, request):
self.stats.inc_value("httpcache/store", spider=spider)
diff --git a/scrapy/downloadermiddlewares/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.py
index 95d5dba66..f56748897 100644
--- a/scrapy/downloadermiddlewares/httpcompression.py
+++ b/scrapy/downloadermiddlewares/httpcompression.py
@@ -1,93 +1,102 @@
from __future__ import annotations
-import io
import warnings
-import zlib
-from typing import TYPE_CHECKING, List, Optional, Union
+from itertools import chain
+from logging import getLogger
+from typing import TYPE_CHECKING, Any
-from scrapy import Request, Spider
-from scrapy.crawler import Crawler
-from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
+from scrapy import Request, Spider, signals
+from scrapy.exceptions import IgnoreRequest, NotConfigured, ScrapyDeprecationWarning
from scrapy.http import Response, TextResponse
from scrapy.responsetypes import responsetypes
-from scrapy.statscollectors import StatsCollector
+from scrapy.utils._compression import (
+ _DecompressionMaxSizeExceeded,
+ _inflate,
+ _unbrotli,
+ _unzstd,
+)
from scrapy.utils.gz import gunzip
if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
-ACCEPTED_ENCODINGS: List[bytes] = [b"gzip", b"deflate"]
+ from scrapy.crawler import Crawler
+ from scrapy.statscollectors import StatsCollector
+
+
+logger = getLogger(__name__)
+
+ACCEPTED_ENCODINGS: list[bytes] = [b"gzip", b"deflate"]
try:
- import brotli
-
+ try:
+ import brotli # noqa: F401
+ except ImportError:
+ import brotlicffi # noqa: F401
+except ImportError:
+ pass
+else:
ACCEPTED_ENCODINGS.append(b"br")
-except ImportError:
- pass
try:
- import zstandard
-
- ACCEPTED_ENCODINGS.append(b"zstd")
+ import zstandard # noqa: F401
except ImportError:
pass
+else:
+ ACCEPTED_ENCODINGS.append(b"zstd")
class HttpCompressionMiddleware:
"""This middleware allows compressed (gzip, deflate) traffic to be
- sent/received from web sites"""
+ sent/received from websites"""
- def __init__(self, stats: Optional[StatsCollector] = None, settings=None):
- self.stats = stats
- if settings:
- self.keep_encoding_header = settings.getbool(
- "COMPRESSION_KEEP_ENCODING_HEADER"
- )
- if not self.keep_encoding_header:
- warnings.warn(
- "Setting COMPRESSION_KEEP_ENCODING_HEADER=False is deprecated",
- ScrapyDeprecationWarning,
- )
- else:
+ def __init__(
+ self,
+ stats: StatsCollector | None = None,
+ *,
+ crawler: Crawler | None = None,
+ ):
+ if not crawler:
+ self.stats = stats
+ self._max_size = 1073741824
+ self._warn_size = 33554432
self.keep_encoding_header = False
+ return
+ self.stats = crawler.stats
+ self._max_size = crawler.settings.getint("DOWNLOAD_MAXSIZE")
+ self._warn_size = crawler.settings.getint("DOWNLOAD_WARNSIZE")
+ self.keep_encoding_header = crawler.settings.getbool(
+ "COMPRESSION_KEEP_ENCODING_HEADER"
+ )
+ if not self.keep_encoding_header:
warnings.warn(
- "The default value of COMPRESSION_KEEP_ENCODING_HEADER, "
- "False, is deprecated, and will stop working and stop "
- "being its default value in a future version of Scrapy. "
- "Set COMPRESSION_KEEP_ENCODING_HEADER=True in your "
- "settings to remove this warning.",
+ "Setting COMPRESSION_KEEP_ENCODING_HEADER=False is deprecated",
ScrapyDeprecationWarning,
- stacklevel=2,
)
+ crawler.signals.connect(self.open_spider, signals.spider_opened)
@classmethod
def from_crawler(cls, crawler: Crawler) -> Self:
if not crawler.settings.getbool("COMPRESSION_ENABLED"):
raise NotConfigured
- try:
- return cls(stats=crawler.stats, settings=crawler.settings)
- except TypeError:
- warnings.warn(
- "HttpCompressionMiddleware subclasses must either modify "
- "their '__init__' method to support 'stats' and 'settings' parameters or "
- "reimplement the 'from_crawler' method.",
- ScrapyDeprecationWarning,
- )
- result = cls()
- result.stats = crawler.stats
- result.keep_encoding_header = False
- return result
+ return cls(crawler=crawler)
+
+ def open_spider(self, spider: Spider) -> None:
+ if hasattr(spider, "download_maxsize"):
+ self._max_size = spider.download_maxsize
+ if hasattr(spider, "download_warnsize"):
+ self._warn_size = spider.download_warnsize
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
request.headers.setdefault("Accept-Encoding", b", ".join(ACCEPTED_ENCODINGS))
return None
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
if request.method == "HEAD":
return response
if b"decoded" in response.flags:
@@ -95,9 +104,26 @@ class HttpCompressionMiddleware:
content_encoding = response.headers.getlist("Content-Encoding")
if not content_encoding:
return response
-
- encoding = content_encoding[0]
- decoded_body = self._decode(response.body, encoding.lower())
+ max_size = request.meta.get("download_maxsize", self._max_size)
+ warn_size = request.meta.get("download_warnsize", self._warn_size)
+ try:
+ decoded_body, content_encoding = self._handle_encoding(
+ response.body, content_encoding, max_size
+ )
+ except _DecompressionMaxSizeExceeded:
+ raise IgnoreRequest(
+ f"Ignored response {response} because its body "
+ f"({len(response.body)} B compressed) exceeded "
+ f"DOWNLOAD_MAXSIZE ({max_size} B) during "
+ f"decompression."
+ )
+ if len(response.body) < warn_size <= len(decoded_body):
+ logger.warning(
+ f"{response} body size after decompression "
+ f"({len(decoded_body)} B) is larger than the "
+ f"download warning size ({warn_size} B)."
+ )
+ response.headers["Content-Encoding"] = content_encoding
if self.stats:
self.stats.inc_value(
"httpcompression/response_bytes", len(decoded_body), spider=spider
@@ -106,37 +132,51 @@ class HttpCompressionMiddleware:
respcls = responsetypes.from_args(
headers=response.headers, url=response.url, body=decoded_body
)
- kwargs = dict(cls=respcls, body=decoded_body)
+ kwargs: dict[str, Any] = {"body": decoded_body}
if issubclass(respcls, TextResponse):
# force recalculating the encoding until we make sure the
# responsetypes guessing is reliable
kwargs["encoding"] = None
kwargs["flags"] = response.flags + [b"decoded"]
- response = response.replace(**kwargs)
+ response = response.replace(cls=respcls, **kwargs)
if not self.keep_encoding_header:
del response.headers["Content-Encoding"]
return response
- def _decode(self, body: bytes, encoding: bytes) -> bytes:
- if encoding == b"gzip" or encoding == b"x-gzip":
- body = gunzip(body)
+ def _handle_encoding(
+ self, body: bytes, content_encoding: list[bytes], max_size: int
+ ) -> tuple[bytes, list[bytes]]:
+ to_decode, to_keep = self._split_encodings(content_encoding)
+ for encoding in to_decode:
+ body = self._decode(body, encoding, max_size)
+ return body, to_keep
+ def _split_encodings(
+ self, content_encoding: list[bytes]
+ ) -> tuple[list[bytes], list[bytes]]:
+ to_keep: list[bytes] = [
+ encoding.strip().lower()
+ for encoding in chain.from_iterable(
+ encodings.split(b",") for encodings in content_encoding
+ )
+ ]
+ to_decode: list[bytes] = []
+ while to_keep:
+ encoding = to_keep.pop()
+ if encoding not in ACCEPTED_ENCODINGS:
+ to_keep.append(encoding)
+ return to_decode, to_keep
+ to_decode.append(encoding)
+ return to_decode, to_keep
+
+ def _decode(self, body: bytes, encoding: bytes, max_size: int) -> bytes:
+ if encoding in {b"gzip", b"x-gzip"}:
+ return gunzip(body, max_size=max_size)
if encoding == b"deflate":
- try:
- body = zlib.decompress(body)
- except zlib.error:
- # ugly hack to work with raw deflate content that may
- # be sent by microsoft servers. For more information, see:
- # http://carsten.codimi.de/gzip.yaws/
- # http://www.port80software.com/200ok/archive/2005/10/31/868.aspx
- # http://www.gzip.org/zlib/zlib_faq.html#faq38
- body = zlib.decompress(body, -15)
+ return _inflate(body, max_size=max_size)
if encoding == b"br" and b"br" in ACCEPTED_ENCODINGS:
- body = brotli.decompress(body)
+ return _unbrotli(body, max_size=max_size)
if encoding == b"zstd" and b"zstd" in ACCEPTED_ENCODINGS:
- # Using its streaming API since its simple API could handle only cases
- # where there is content size data embedded in the frame
- reader = zstandard.ZstdDecompressor().stream_reader(io.BytesIO(body))
- body = reader.read()
+ return _unzstd(body, max_size=max_size)
return body
diff --git a/scrapy/downloadermiddlewares/httpproxy.py b/scrapy/downloadermiddlewares/httpproxy.py
index 335896ac1..2f3f2db47 100644
--- a/scrapy/downloadermiddlewares/httpproxy.py
+++ b/scrapy/downloadermiddlewares/httpproxy.py
@@ -1,7 +1,7 @@
from __future__ import annotations
import base64
-from typing import TYPE_CHECKING, Dict, Optional, Tuple, Union
+from typing import TYPE_CHECKING
from urllib.parse import unquote, urlunparse
from urllib.request import ( # type: ignore[attr-defined]
_parse_proxy,
@@ -9,10 +9,7 @@ from urllib.request import ( # type: ignore[attr-defined]
proxy_bypass,
)
-from scrapy import Request, Spider
-from scrapy.crawler import Crawler
from scrapy.exceptions import NotConfigured
-from scrapy.http import Response
from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.python import to_bytes
@@ -20,11 +17,15 @@ if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
class HttpProxyMiddleware:
- def __init__(self, auth_encoding: Optional[str] = "latin-1"):
- self.auth_encoding: Optional[str] = auth_encoding
- self.proxies: Dict[str, Tuple[Optional[bytes], str]] = {}
+ def __init__(self, auth_encoding: str | None = "latin-1"):
+ self.auth_encoding: str | None = auth_encoding
+ self.proxies: dict[str, tuple[bytes | None, str]] = {}
for type_, url in getproxies().items():
try:
self.proxies[type_] = self._get_proxy(url, type_)
@@ -37,7 +38,7 @@ class HttpProxyMiddleware:
def from_crawler(cls, crawler: Crawler) -> Self:
if not crawler.settings.getbool("HTTPPROXY_ENABLED"):
raise NotConfigured
- auth_encoding: Optional[str] = crawler.settings.get("HTTPPROXY_AUTH_ENCODING")
+ auth_encoding: str | None = crawler.settings.get("HTTPPROXY_AUTH_ENCODING")
return cls(auth_encoding)
def _basic_auth_header(self, username: str, password: str) -> bytes:
@@ -46,7 +47,7 @@ class HttpProxyMiddleware:
)
return base64.b64encode(user_pass)
- def _get_proxy(self, url: str, orig_type: str) -> Tuple[Optional[bytes], str]:
+ def _get_proxy(self, url: str, orig_type: str) -> tuple[bytes | None, str]:
proxy_type, user, password, hostport = _parse_proxy(url)
proxy_url = urlunparse((proxy_type or orig_type, hostport, "", "", "", ""))
@@ -59,27 +60,34 @@ class HttpProxyMiddleware:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
- creds, proxy_url = None, None
+ ) -> Request | Response | None:
+ creds, proxy_url, scheme = None, None, None
if "proxy" in request.meta:
if request.meta["proxy"] is not None:
creds, proxy_url = self._get_proxy(request.meta["proxy"], "")
elif self.proxies:
parsed = urlparse_cached(request)
- scheme = parsed.scheme
+ _scheme = parsed.scheme
if (
# 'no_proxy' is only supported by http schemes
- scheme not in ("http", "https")
+ _scheme not in ("http", "https")
or (parsed.hostname and not proxy_bypass(parsed.hostname))
- ) and scheme in self.proxies:
+ ) and _scheme in self.proxies:
+ scheme = _scheme
creds, proxy_url = self.proxies[scheme]
- self._set_proxy_and_creds(request, proxy_url, creds)
+ self._set_proxy_and_creds(request, proxy_url, creds, scheme)
return None
def _set_proxy_and_creds(
- self, request: Request, proxy_url: Optional[str], creds: Optional[bytes]
+ self,
+ request: Request,
+ proxy_url: str | None,
+ creds: bytes | None,
+ scheme: str | None,
) -> None:
+ if scheme:
+ request.meta["_scheme_proxy"] = True
if proxy_url:
request.meta["proxy"] = proxy_url
elif request.meta.get("proxy") is not None:
diff --git a/scrapy/downloadermiddlewares/offsite.py b/scrapy/downloadermiddlewares/offsite.py
new file mode 100644
index 000000000..a69f531a7
--- /dev/null
+++ b/scrapy/downloadermiddlewares/offsite.py
@@ -0,0 +1,89 @@
+from __future__ import annotations
+
+import logging
+import re
+import warnings
+from typing import TYPE_CHECKING
+
+from scrapy import Request, Spider, signals
+from scrapy.exceptions import IgnoreRequest
+from scrapy.utils.httpobj import urlparse_cached
+
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.statscollectors import StatsCollector
+
+
+logger = logging.getLogger(__name__)
+
+
+class OffsiteMiddleware:
+ @classmethod
+ def from_crawler(cls, crawler: Crawler) -> Self:
+ assert crawler.stats
+ o = cls(crawler.stats)
+ crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
+ crawler.signals.connect(o.request_scheduled, signal=signals.request_scheduled)
+ return o
+
+ def __init__(self, stats: StatsCollector):
+ self.stats = stats
+ self.domains_seen: set[str] = set()
+
+ def spider_opened(self, spider: Spider) -> None:
+ self.host_regex: re.Pattern[str] = self.get_host_regex(spider)
+
+ def request_scheduled(self, request: Request, spider: Spider) -> None:
+ self.process_request(request, spider)
+
+ def process_request(self, request: Request, spider: Spider) -> None:
+ if request.dont_filter or self.should_follow(request, spider):
+ return
+ domain = urlparse_cached(request).hostname
+ if domain and domain not in self.domains_seen:
+ self.domains_seen.add(domain)
+ logger.debug(
+ "Filtered offsite request to %(domain)r: %(request)s",
+ {"domain": domain, "request": request},
+ extra={"spider": spider},
+ )
+ self.stats.inc_value("offsite/domains", spider=spider)
+ self.stats.inc_value("offsite/filtered", spider=spider)
+ raise IgnoreRequest
+
+ def should_follow(self, request: Request, spider: Spider) -> bool:
+ regex = self.host_regex
+ # hostname can be None for wrong urls (like javascript links)
+ host = urlparse_cached(request).hostname or ""
+ return bool(regex.search(host))
+
+ def get_host_regex(self, spider: Spider) -> re.Pattern[str]:
+ """Override this method to implement a different offsite policy"""
+ allowed_domains = getattr(spider, "allowed_domains", None)
+ if not allowed_domains:
+ return re.compile("") # allow all by default
+ url_pattern = re.compile(r"^https?://.*$")
+ port_pattern = re.compile(r":\d+$")
+ domains = []
+ for domain in allowed_domains:
+ if domain is None:
+ continue
+ if url_pattern.match(domain):
+ message = (
+ "allowed_domains accepts only domains, not URLs. "
+ f"Ignoring URL entry {domain} in allowed_domains."
+ )
+ warnings.warn(message)
+ elif port_pattern.search(domain):
+ message = (
+ "allowed_domains accepts only domains without ports. "
+ f"Ignoring entry {domain} in allowed_domains."
+ )
+ warnings.warn(message)
+ else:
+ domains.append(re.escape(domain))
+ regex = rf'^(.*\.)?({"|".join(domains)})$'
+ return re.compile(regex)
diff --git a/scrapy/downloadermiddlewares/redirect.py b/scrapy/downloadermiddlewares/redirect.py
index 814b1a561..0b883b43a 100644
--- a/scrapy/downloadermiddlewares/redirect.py
+++ b/scrapy/downloadermiddlewares/redirect.py
@@ -1,16 +1,13 @@
from __future__ import annotations
import logging
-from typing import TYPE_CHECKING, Any, List, Union, cast
-from urllib.parse import urljoin, urlparse
+from typing import TYPE_CHECKING, Any, cast
+from urllib.parse import urljoin
from w3lib.url import safe_url_string
-from scrapy import Request, Spider
-from scrapy.crawler import Crawler
from scrapy.exceptions import IgnoreRequest, NotConfigured
from scrapy.http import HtmlResponse, Response
-from scrapy.settings import BaseSettings
from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.response import get_meta_refresh
@@ -18,6 +15,11 @@ if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.settings import BaseSettings
+
+
logger = logging.getLogger(__name__)
@@ -27,13 +29,52 @@ def _build_redirect_request(
redirect_request = source_request.replace(
url=url,
**kwargs,
+ cls=None,
cookies=None,
)
- if "Cookie" in redirect_request.headers:
- source_request_netloc = urlparse_cached(source_request).netloc
- redirect_request_netloc = urlparse_cached(redirect_request).netloc
- if source_request_netloc != redirect_request_netloc:
+ if "_scheme_proxy" in redirect_request.meta:
+ source_request_scheme = urlparse_cached(source_request).scheme
+ redirect_request_scheme = urlparse_cached(redirect_request).scheme
+ if source_request_scheme != redirect_request_scheme:
+ redirect_request.meta.pop("_scheme_proxy")
+ redirect_request.meta.pop("proxy", None)
+ redirect_request.meta.pop("_auth_proxy", None)
+ redirect_request.headers.pop(b"Proxy-Authorization", None)
+ has_cookie_header = "Cookie" in redirect_request.headers
+ has_authorization_header = "Authorization" in redirect_request.headers
+ if has_cookie_header or has_authorization_header:
+ default_ports = {"http": 80, "https": 443}
+
+ parsed_source_request = urlparse_cached(source_request)
+ source_scheme, source_host, source_port = (
+ parsed_source_request.scheme,
+ parsed_source_request.hostname,
+ parsed_source_request.port
+ or default_ports.get(parsed_source_request.scheme),
+ )
+
+ parsed_redirect_request = urlparse_cached(redirect_request)
+ redirect_scheme, redirect_host, redirect_port = (
+ parsed_redirect_request.scheme,
+ parsed_redirect_request.hostname,
+ parsed_redirect_request.port
+ or default_ports.get(parsed_redirect_request.scheme),
+ )
+
+ if has_cookie_header and (
+ redirect_scheme not in {source_scheme, "https"}
+ or source_host != redirect_host
+ ):
del redirect_request.headers["Cookie"]
+
+ # https://fetch.spec.whatwg.org/#ref-for-cors-non-wildcard-request-header-name
+ if has_authorization_header and (
+ source_scheme != redirect_scheme
+ or source_host != redirect_host
+ or source_port != redirect_port
+ ):
+ del redirect_request.headers["Authorization"]
+
return redirect_request
@@ -103,7 +144,7 @@ class RedirectMiddleware(BaseRedirectMiddleware):
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
if (
request.meta.get("dont_redirect", False)
or response.status in getattr(spider, "handle_httpstatus_list", [])
@@ -119,13 +160,15 @@ class RedirectMiddleware(BaseRedirectMiddleware):
assert response.headers["Location"] is not None
location = safe_url_string(response.headers["Location"])
if response.headers["Location"].startswith(b"//"):
- request_scheme = urlparse(request.url).scheme
+ request_scheme = urlparse_cached(request).scheme
location = request_scheme + "://" + location.lstrip("/")
redirected_url = urljoin(request.url, location)
+ redirected = _build_redirect_request(request, url=redirected_url)
+ if urlparse_cached(redirected).scheme not in {"http", "https"}:
+ return response
if response.status in (301, 307, 308) or request.method == "HEAD":
- redirected = _build_redirect_request(request, url=redirected_url)
return self._redirect(redirected, request, spider, response.status)
redirected = self._redirect_request_using_get(request, redirected_url)
@@ -137,22 +180,26 @@ class MetaRefreshMiddleware(BaseRedirectMiddleware):
def __init__(self, settings: BaseSettings):
super().__init__(settings)
- self._ignore_tags: List[str] = settings.getlist("METAREFRESH_IGNORE_TAGS")
+ self._ignore_tags: list[str] = settings.getlist("METAREFRESH_IGNORE_TAGS")
self._maxdelay: int = settings.getint("METAREFRESH_MAXDELAY")
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
if (
request.meta.get("dont_redirect", False)
or request.method == "HEAD"
or not isinstance(response, HtmlResponse)
+ or urlparse_cached(request).scheme not in {"http", "https"}
):
return response
interval, url = get_meta_refresh(response, ignore_tags=self._ignore_tags)
- if url and cast(float, interval) < self._maxdelay:
- redirected = self._redirect_request_using_get(request, url)
+ if not url:
+ return response
+ redirected = self._redirect_request_using_get(request, url)
+ if urlparse_cached(redirected).scheme not in {"http", "https"}:
+ return response
+ if cast(float, interval) < self._maxdelay:
return self._redirect(redirected, request, spider, "meta refresh")
-
return response
diff --git a/scrapy/downloadermiddlewares/retry.py b/scrapy/downloadermiddlewares/retry.py
index 3c494de78..723fe5e93 100644
--- a/scrapy/downloadermiddlewares/retry.py
+++ b/scrapy/downloadermiddlewares/retry.py
@@ -7,20 +7,15 @@ RETRY_TIMES - how many times to retry a failed page
RETRY_HTTP_CODES - which HTTP response codes to retry
Failed pages are collected on the scraping process and rescheduled at the end,
-once the spider has finished crawling all regular (non failed) pages.
+once the spider has finished crawling all regular (non-failed) pages.
"""
+
from __future__ import annotations
-import warnings
from logging import Logger, getLogger
-from typing import TYPE_CHECKING, Any, Optional, Tuple, Type, Union
+from typing import TYPE_CHECKING
-from scrapy.crawler import Crawler
-from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
-from scrapy.http import Response
-from scrapy.http.request import Request
-from scrapy.settings import BaseSettings, Settings
-from scrapy.spiders import Spider
+from scrapy.exceptions import NotConfigured
from scrapy.utils.misc import load_object
from scrapy.utils.python import global_object_name
from scrapy.utils.response import response_status_message
@@ -29,40 +24,26 @@ if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+ from scrapy.http.request import Request
+ from scrapy.settings import BaseSettings
+ from scrapy.spiders import Spider
+
+
retry_logger = getLogger(__name__)
-def backwards_compatibility_getattr(self: Any, name: str) -> Tuple[Any, ...]:
- if name == "EXCEPTIONS_TO_RETRY":
- warnings.warn(
- "Attribute RetryMiddleware.EXCEPTIONS_TO_RETRY is deprecated. "
- "Use the RETRY_EXCEPTIONS setting instead.",
- ScrapyDeprecationWarning,
- stacklevel=2,
- )
- return tuple(
- load_object(x) if isinstance(x, str) else x
- for x in Settings().getlist("RETRY_EXCEPTIONS")
- )
- raise AttributeError(
- f"{self.__class__.__name__!r} object has no attribute {name!r}"
- )
-
-
-class BackwardsCompatibilityMetaclass(type):
- __getattr__ = backwards_compatibility_getattr
-
-
def get_retry_request(
request: Request,
*,
spider: Spider,
- reason: Union[str, Exception, Type[Exception]] = "unspecified",
- max_retry_times: Optional[int] = None,
- priority_adjust: Optional[int] = None,
+ reason: str | Exception | type[Exception] = "unspecified",
+ max_retry_times: int | None = None,
+ priority_adjust: int | None = None,
logger: Logger = retry_logger,
stats_base_key: str = "retry",
-) -> Optional[Request]:
+) -> Request | None:
"""
Returns a new :class:`~scrapy.Request` object to retry the specified
request, or ``None`` if retries of the specified request have been
@@ -134,31 +115,24 @@ def get_retry_request(
return new_request
stats.inc_value(f"{stats_base_key}/max_reached")
logger.error(
- "Gave up retrying %(request)s (failed %(retry_times)d times): " "%(reason)s",
+ "Gave up retrying %(request)s (failed %(retry_times)d times): %(reason)s",
{"request": request, "retry_times": retry_times, "reason": reason},
extra={"spider": spider},
)
return None
-class RetryMiddleware(metaclass=BackwardsCompatibilityMetaclass):
+class RetryMiddleware:
def __init__(self, settings: BaseSettings):
if not settings.getbool("RETRY_ENABLED"):
raise NotConfigured
self.max_retry_times = settings.getint("RETRY_TIMES")
- self.retry_http_codes = set(
- int(x) for x in settings.getlist("RETRY_HTTP_CODES")
- )
+ self.retry_http_codes = {int(x) for x in settings.getlist("RETRY_HTTP_CODES")}
self.priority_adjust = settings.getint("RETRY_PRIORITY_ADJUST")
-
- try:
- self.exceptions_to_retry = self.__getattribute__("EXCEPTIONS_TO_RETRY")
- except AttributeError:
- # If EXCEPTIONS_TO_RETRY is not "overridden"
- self.exceptions_to_retry = tuple(
- load_object(x) if isinstance(x, str) else x
- for x in settings.getlist("RETRY_EXCEPTIONS")
- )
+ self.exceptions_to_retry = tuple(
+ load_object(x) if isinstance(x, str) else x
+ for x in settings.getlist("RETRY_EXCEPTIONS")
+ )
@classmethod
def from_crawler(cls, crawler: Crawler) -> Self:
@@ -166,7 +140,7 @@ class RetryMiddleware(metaclass=BackwardsCompatibilityMetaclass):
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
if request.meta.get("dont_retry", False):
return response
if response.status in self.retry_http_codes:
@@ -176,7 +150,7 @@ class RetryMiddleware(metaclass=BackwardsCompatibilityMetaclass):
def process_exception(
self, request: Request, exception: Exception, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
if isinstance(exception, self.exceptions_to_retry) and not request.meta.get(
"dont_retry", False
):
@@ -186,9 +160,9 @@ class RetryMiddleware(metaclass=BackwardsCompatibilityMetaclass):
def _retry(
self,
request: Request,
- reason: Union[str, Exception, Type[Exception]],
+ reason: str | Exception | type[Exception],
spider: Spider,
- ) -> Optional[Request]:
+ ) -> Request | None:
max_retry_times = request.meta.get("max_retry_times", self.max_retry_times)
priority_adjust = request.meta.get("priority_adjust", self.priority_adjust)
return get_retry_request(
@@ -198,5 +172,3 @@ class RetryMiddleware(metaclass=BackwardsCompatibilityMetaclass):
max_retry_times=max_retry_times,
priority_adjust=priority_adjust,
)
-
- __getattr__ = backwards_compatibility_getattr
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
index 6a0ecb7bf..9411cff14 100644
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -7,25 +7,27 @@ enable this middleware and enable the ROBOTSTXT_OBEY setting.
from __future__ import annotations
import logging
-from typing import TYPE_CHECKING, Any, Dict, Optional, Union
+from typing import TYPE_CHECKING
from twisted.internet.defer import Deferred, maybeDeferred
-from twisted.python.failure import Failure
-from scrapy import Spider
-from scrapy.crawler import Crawler
from scrapy.exceptions import IgnoreRequest, NotConfigured
from scrapy.http import Request, Response
from scrapy.http.request import NO_CALLBACK
-from scrapy.robotstxt import RobotParser
from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.log import failure_to_exc_info
from scrapy.utils.misc import load_object
if TYPE_CHECKING:
+ from twisted.python.failure import Failure
+
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy import Spider
+ from scrapy.crawler import Crawler
+ from scrapy.robotstxt import RobotParser
+
logger = logging.getLogger(__name__)
@@ -37,11 +39,11 @@ class RobotsTxtMiddleware:
if not crawler.settings.getbool("ROBOTSTXT_OBEY"):
raise NotConfigured
self._default_useragent: str = crawler.settings.get("USER_AGENT", "Scrapy")
- self._robotstxt_useragent: Optional[str] = crawler.settings.get(
+ self._robotstxt_useragent: str | None = crawler.settings.get(
"ROBOTSTXT_USER_AGENT", None
)
self.crawler: Crawler = crawler
- self._parsers: Dict[str, Union[RobotParser, Deferred, None]] = {}
+ self._parsers: dict[str, RobotParser | Deferred[RobotParser | None] | None] = {}
self._parserimpl: RobotParser = load_object(
crawler.settings.get("ROBOTSTXT_PARSER")
)
@@ -53,22 +55,26 @@ class RobotsTxtMiddleware:
def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler)
- def process_request(self, request: Request, spider: Spider) -> Optional[Deferred]:
+ def process_request(
+ self, request: Request, spider: Spider
+ ) -> Deferred[None] | None:
if request.meta.get("dont_obey_robotstxt"):
return None
if request.url.startswith("data:") or request.url.startswith("file:"):
return None
- d: Deferred = maybeDeferred(self.robot_parser, request, spider)
- d.addCallback(self.process_request_2, request, spider)
- return d
+ d: Deferred[RobotParser | None] = maybeDeferred(
+ self.robot_parser, request, spider # type: ignore[call-overload]
+ )
+ d2: Deferred[None] = d.addCallback(self.process_request_2, request, spider)
+ return d2
def process_request_2(
- self, rp: Optional[RobotParser], request: Request, spider: Spider
+ self, rp: RobotParser | None, request: Request, spider: Spider
) -> None:
if rp is None:
return
- useragent: Union[str, bytes, None] = self._robotstxt_useragent
+ useragent: str | bytes | None = self._robotstxt_useragent
if not useragent:
useragent = request.headers.get(b"User-Agent", self._default_useragent)
assert useragent is not None
@@ -84,7 +90,7 @@ class RobotsTxtMiddleware:
def robot_parser(
self, request: Request, spider: Spider
- ) -> Union[RobotParser, Deferred, None]:
+ ) -> RobotParser | Deferred[RobotParser | None] | None:
url = urlparse_cached(request)
netloc = url.netloc
@@ -107,9 +113,9 @@ class RobotsTxtMiddleware:
parser = self._parsers[netloc]
if isinstance(parser, Deferred):
- d: Deferred = Deferred()
+ d: Deferred[RobotParser | None] = Deferred()
- def cb(result: Any) -> Any:
+ def cb(result: RobotParser | None) -> RobotParser | None:
d.callback(result)
return result
diff --git a/scrapy/downloadermiddlewares/stats.py b/scrapy/downloadermiddlewares/stats.py
index df30e8ca4..fb0f30620 100644
--- a/scrapy/downloadermiddlewares/stats.py
+++ b/scrapy/downloadermiddlewares/stats.py
@@ -1,14 +1,10 @@
from __future__ import annotations
-from typing import TYPE_CHECKING, Dict, Union
+from typing import TYPE_CHECKING
from twisted.web import http
-from scrapy import Request, Spider
-from scrapy.crawler import Crawler
from scrapy.exceptions import NotConfigured
-from scrapy.http import Response
-from scrapy.statscollectors import StatsCollector
from scrapy.utils.python import global_object_name, to_bytes
from scrapy.utils.request import request_httprepr
@@ -16,8 +12,15 @@ if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy import Request, Spider
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+ from scrapy.statscollectors import StatsCollector
-def get_header_size(headers: Dict[str, Union[list, tuple]]) -> int:
+
+def get_header_size(
+ headers: dict[str, list[str | bytes] | tuple[str | bytes, ...]]
+) -> int:
size = 0
for key, value in headers.items():
if isinstance(value, (list, tuple)):
@@ -44,7 +47,7 @@ class DownloaderStats:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
self.stats.inc_value("downloader/request_count", spider=spider)
self.stats.inc_value(
f"downloader/request_method_count/{request.method}", spider=spider
@@ -55,7 +58,7 @@ class DownloaderStats:
def process_response(
self, request: Request, response: Response, spider: Spider
- ) -> Union[Request, Response]:
+ ) -> Request | Response:
self.stats.inc_value("downloader/response_count", spider=spider)
self.stats.inc_value(
f"downloader/response_status_count/{response.status}", spider=spider
@@ -72,7 +75,7 @@ class DownloaderStats:
def process_exception(
self, request: Request, exception: Exception, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
ex_class = global_object_name(exception.__class__)
self.stats.inc_value("downloader/exception_count", spider=spider)
self.stats.inc_value(
diff --git a/scrapy/downloadermiddlewares/useragent.py b/scrapy/downloadermiddlewares/useragent.py
index 92f1ec897..ba379f862 100644
--- a/scrapy/downloadermiddlewares/useragent.py
+++ b/scrapy/downloadermiddlewares/useragent.py
@@ -2,16 +2,17 @@
from __future__ import annotations
-from typing import TYPE_CHECKING, Union
+from typing import TYPE_CHECKING
from scrapy import Request, Spider, signals
-from scrapy.crawler import Crawler
-from scrapy.http import Response
if TYPE_CHECKING:
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
class UserAgentMiddleware:
"""This middleware allows spiders to override the user_agent"""
@@ -30,7 +31,7 @@ class UserAgentMiddleware:
def process_request(
self, request: Request, spider: Spider
- ) -> Union[Request, Response, None]:
+ ) -> Request | Response | None:
if self.user_agent:
request.headers.setdefault(b"User-Agent", self.user_agent)
return None
diff --git a/scrapy/dupefilters.py b/scrapy/dupefilters.py
index dd2420e98..caf69daf4 100644
--- a/scrapy/dupefilters.py
+++ b/scrapy/dupefilters.py
@@ -1,14 +1,11 @@
from __future__ import annotations
import logging
+import warnings
from pathlib import Path
-from typing import TYPE_CHECKING, Optional, Set
+from typing import TYPE_CHECKING
-from twisted.internet.defer import Deferred
-
-from scrapy.http.request import Request
-from scrapy.settings import BaseSettings
-from scrapy.spiders import Spider
+from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.utils.job import job_dir
from scrapy.utils.request import (
RequestFingerprinter,
@@ -17,29 +14,42 @@ from scrapy.utils.request import (
)
if TYPE_CHECKING:
+ from twisted.internet.defer import Deferred
+
# typing.Self requires Python 3.11
from typing_extensions import Self
from scrapy.crawler import Crawler
+ from scrapy.http.request import Request
+ from scrapy.settings import BaseSettings
+ from scrapy.spiders import Spider
class BaseDupeFilter:
@classmethod
def from_settings(cls, settings: BaseSettings) -> Self:
+ warnings.warn(
+ f"{cls.__name__}.from_settings() is deprecated, use from_crawler() instead.",
+ category=ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ return cls()
+
+ @classmethod
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls()
def request_seen(self, request: Request) -> bool:
return False
- def open(self) -> Optional[Deferred]:
+ def open(self) -> Deferred[None] | None:
pass
- def close(self, reason: str) -> Optional[Deferred]:
+ def close(self, reason: str) -> Deferred[None] | None:
pass
def log(self, request: Request, spider: Spider) -> None:
"""Log that a request has been filtered"""
- pass
class RFPDupeFilter(BaseDupeFilter):
@@ -47,16 +57,16 @@ class RFPDupeFilter(BaseDupeFilter):
def __init__(
self,
- path: Optional[str] = None,
+ path: str | None = None,
debug: bool = False,
*,
- fingerprinter: Optional[RequestFingerprinterProtocol] = None,
+ fingerprinter: RequestFingerprinterProtocol | None = None,
) -> None:
self.file = None
self.fingerprinter: RequestFingerprinterProtocol = (
fingerprinter or RequestFingerprinter()
)
- self.fingerprints: Set[str] = set()
+ self.fingerprints: set[str] = set()
self.logdupes = True
self.debug = debug
self.logger = logging.getLogger(__name__)
@@ -70,19 +80,33 @@ class RFPDupeFilter(BaseDupeFilter):
cls,
settings: BaseSettings,
*,
- fingerprinter: Optional[RequestFingerprinterProtocol] = None,
+ fingerprinter: RequestFingerprinterProtocol | None = None,
) -> Self:
- debug = settings.getbool("DUPEFILTER_DEBUG")
- return cls(job_dir(settings), debug, fingerprinter=fingerprinter)
+ warnings.warn(
+ f"{cls.__name__}.from_settings() is deprecated, use from_crawler() instead.",
+ category=ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ return cls._from_settings(settings, fingerprinter=fingerprinter)
@classmethod
def from_crawler(cls, crawler: Crawler) -> Self:
assert crawler.request_fingerprinter
- return cls.from_settings(
+ return cls._from_settings(
crawler.settings,
fingerprinter=crawler.request_fingerprinter,
)
+ @classmethod
+ def _from_settings(
+ cls,
+ settings: BaseSettings,
+ *,
+ fingerprinter: RequestFingerprinterProtocol | None = None,
+ ) -> Self:
+ debug = settings.getbool("DUPEFILTER_DEBUG")
+ return cls(job_dir(settings), debug, fingerprinter=fingerprinter)
+
def request_seen(self, request: Request) -> bool:
fp = self.request_fingerprint(request)
if fp in self.fingerprints:
diff --git a/scrapy/exceptions.py b/scrapy/exceptions.py
index 6d188c489..96566ba86 100644
--- a/scrapy/exceptions.py
+++ b/scrapy/exceptions.py
@@ -4,6 +4,7 @@ Scrapy core exceptions
These exceptions are documented in docs/topics/exceptions.rst. Please don't add
new exceptions here without documenting them there.
"""
+
from typing import Any
# Internal
@@ -12,8 +13,6 @@ from typing import Any
class NotConfigured(Exception):
"""Indicates a missing configuration situation"""
- pass
-
class _InvalidOutput(TypeError):
"""
@@ -21,8 +20,6 @@ class _InvalidOutput(TypeError):
Internal and undocumented, it should not be raised or caught by user code.
"""
- pass
-
# HTTP and crawling
@@ -34,8 +31,6 @@ class IgnoreRequest(Exception):
class DontCloseSpider(Exception):
"""Request the spider not to be closed yet"""
- pass
-
class CloseSpider(Exception):
"""Raise this from callbacks to request the spider to be closed"""
@@ -63,14 +58,10 @@ class StopDownload(Exception):
class DropItem(Exception):
"""Drop item from the item pipeline"""
- pass
-
class NotSupported(Exception):
"""Indicates a feature or method is not supported"""
- pass
-
# Commands
@@ -88,10 +79,6 @@ class ScrapyDeprecationWarning(Warning):
DeprecationWarning is silenced on Python 2.7+
"""
- pass
-
class ContractFail(AssertionError):
"""Error raised in case of a failing contract"""
-
- pass
diff --git a/scrapy/exporters.py b/scrapy/exporters.py
index f85f1dad8..cdb7ac159 100644
--- a/scrapy/exporters.py
+++ b/scrapy/exporters.py
@@ -2,20 +2,27 @@
Item Exporters are used to export/serialize items into different formats.
"""
+from __future__ import annotations
+
import csv
-import io
import marshal
import pickle
import pprint
-from collections.abc import Mapping
+from collections.abc import Callable, Iterable, Mapping
+from io import BytesIO, TextIOWrapper
+from typing import TYPE_CHECKING, Any
from xml.sax.saxutils import XMLGenerator
+from xml.sax.xmlreader import AttributesImpl
from itemadapter import ItemAdapter, is_item
-from scrapy.item import Item
+from scrapy.item import Field, Item
from scrapy.utils.python import is_listlike, to_bytes, to_unicode
from scrapy.utils.serialize import ScrapyJSONEncoder
+if TYPE_CHECKING:
+ from json import JSONEncoder
+
__all__ = [
"BaseItemExporter",
"PprintItemExporter",
@@ -29,36 +36,42 @@ __all__ = [
class BaseItemExporter:
- def __init__(self, *, dont_fail=False, **kwargs):
- self._kwargs = kwargs
+ def __init__(self, *, dont_fail: bool = False, **kwargs: Any):
+ self._kwargs: dict[str, Any] = kwargs
self._configure(kwargs, dont_fail=dont_fail)
- def _configure(self, options, dont_fail=False):
+ def _configure(self, options: dict[str, Any], dont_fail: bool = False) -> None:
"""Configure the exporter by popping options from the ``options`` dict.
If dont_fail is set, it won't raise an exception on unexpected options
(useful for using with keyword arguments in subclasses ``__init__`` methods)
"""
- self.encoding = options.pop("encoding", None)
- self.fields_to_export = options.pop("fields_to_export", None)
- self.export_empty_fields = options.pop("export_empty_fields", False)
- self.indent = options.pop("indent", None)
+ self.encoding: str | None = options.pop("encoding", None)
+ self.fields_to_export: Mapping[str, str] | Iterable[str] | None = options.pop(
+ "fields_to_export", None
+ )
+ self.export_empty_fields: bool = options.pop("export_empty_fields", False)
+ self.indent: int | None = options.pop("indent", None)
if not dont_fail and options:
raise TypeError(f"Unexpected options: {', '.join(options.keys())}")
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
raise NotImplementedError
- def serialize_field(self, field, name, value):
- serializer = field.get("serializer", lambda x: x)
+ def serialize_field(
+ self, field: Mapping[str, Any] | Field, name: str, value: Any
+ ) -> Any:
+ serializer: Callable[[Any], Any] = field.get("serializer", lambda x: x)
return serializer(value)
- def start_exporting(self):
+ def start_exporting(self) -> None:
pass
- def finish_exporting(self):
+ def finish_exporting(self) -> None:
pass
- def _get_serialized_fields(self, item, default_value=None, include_empty=None):
+ def _get_serialized_fields(
+ self, item: Any, default_value: Any = None, include_empty: bool | None = None
+ ) -> Iterable[tuple[str, Any]]:
"""Return the fields to export as an iterable of tuples
(name, serialized_value)
"""
@@ -79,11 +92,10 @@ class BaseItemExporter:
field_iter = (
(x, y) for x, y in self.fields_to_export.items() if x in item
)
+ elif include_empty:
+ field_iter = self.fields_to_export
else:
- if include_empty:
- field_iter = self.fields_to_export
- else:
- field_iter = (x for x in self.fields_to_export if x in item)
+ field_iter = (x for x in self.fields_to_export if x in item)
for field_name in field_iter:
if isinstance(field_name, str):
@@ -100,22 +112,22 @@ class BaseItemExporter:
class JsonLinesItemExporter(BaseItemExporter):
- def __init__(self, file, **kwargs):
+ def __init__(self, file: BytesIO, **kwargs: Any):
super().__init__(dont_fail=True, **kwargs)
- self.file = file
+ self.file: BytesIO = file
self._kwargs.setdefault("ensure_ascii", not self.encoding)
- self.encoder = ScrapyJSONEncoder(**self._kwargs)
+ self.encoder: JSONEncoder = ScrapyJSONEncoder(**self._kwargs)
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
itemdict = dict(self._get_serialized_fields(item))
data = self.encoder.encode(itemdict) + "\n"
self.file.write(to_bytes(data, self.encoding))
class JsonItemExporter(BaseItemExporter):
- def __init__(self, file, **kwargs):
+ def __init__(self, file: BytesIO, **kwargs: Any):
super().__init__(dont_fail=True, **kwargs)
- self.file = file
+ self.file: BytesIO = file
# there is a small difference between the behaviour or JsonItemExporter.indent
# and ScrapyJSONEncoder.indent. ScrapyJSONEncoder.indent=None is needed to prevent
# the addition of newlines everywhere
@@ -127,26 +139,26 @@ class JsonItemExporter(BaseItemExporter):
self.encoder = ScrapyJSONEncoder(**self._kwargs)
self.first_item = True
- def _beautify_newline(self):
+ def _beautify_newline(self) -> None:
if self.indent is not None:
self.file.write(b"\n")
- def _add_comma_after_first(self):
+ def _add_comma_after_first(self) -> None:
if self.first_item:
self.first_item = False
else:
self.file.write(b",")
self._beautify_newline()
- def start_exporting(self):
+ def start_exporting(self) -> None:
self.file.write(b"[")
self._beautify_newline()
- def finish_exporting(self):
+ def finish_exporting(self) -> None:
self._beautify_newline()
self.file.write(b"]")
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
itemdict = dict(self._get_serialized_fields(item))
data = to_bytes(self.encoder.encode(itemdict), self.encoding)
self._add_comma_after_first()
@@ -154,7 +166,7 @@ class JsonItemExporter(BaseItemExporter):
class XmlItemExporter(BaseItemExporter):
- def __init__(self, file, **kwargs):
+ def __init__(self, file: BytesIO, **kwargs: Any):
self.item_element = kwargs.pop("item_element", "item")
self.root_element = kwargs.pop("root_element", "items")
super().__init__(**kwargs)
@@ -162,22 +174,22 @@ class XmlItemExporter(BaseItemExporter):
self.encoding = "utf-8"
self.xg = XMLGenerator(file, encoding=self.encoding)
- def _beautify_newline(self, new_item=False):
+ def _beautify_newline(self, new_item: bool = False) -> None:
if self.indent is not None and (self.indent > 0 or new_item):
self.xg.characters("\n")
- def _beautify_indent(self, depth=1):
+ def _beautify_indent(self, depth: int = 1) -> None:
if self.indent:
self.xg.characters(" " * self.indent * depth)
- def start_exporting(self):
+ def start_exporting(self) -> None:
self.xg.startDocument()
- self.xg.startElement(self.root_element, {})
+ self.xg.startElement(self.root_element, AttributesImpl({}))
self._beautify_newline(new_item=True)
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
self._beautify_indent(depth=1)
- self.xg.startElement(self.item_element, {})
+ self.xg.startElement(self.item_element, AttributesImpl({}))
self._beautify_newline()
for name, value in self._get_serialized_fields(item, default_value=""):
self._export_xml_field(name, value, depth=2)
@@ -185,13 +197,13 @@ class XmlItemExporter(BaseItemExporter):
self.xg.endElement(self.item_element)
self._beautify_newline(new_item=True)
- def finish_exporting(self):
+ def finish_exporting(self) -> None:
self.xg.endElement(self.root_element)
self.xg.endDocument()
- def _export_xml_field(self, name, serialized_value, depth):
+ def _export_xml_field(self, name: str, serialized_value: Any, depth: int) -> None:
self._beautify_indent(depth=depth)
- self.xg.startElement(name, {})
+ self.xg.startElement(name, AttributesImpl({}))
if hasattr(serialized_value, "items"):
self._beautify_newline()
for subname, value in serialized_value.items():
@@ -213,17 +225,17 @@ class XmlItemExporter(BaseItemExporter):
class CsvItemExporter(BaseItemExporter):
def __init__(
self,
- file,
- include_headers_line=True,
- join_multivalued=",",
- errors=None,
- **kwargs,
+ file: BytesIO,
+ include_headers_line: bool = True,
+ join_multivalued: str = ",",
+ errors: str | None = None,
+ **kwargs: Any,
):
super().__init__(dont_fail=True, **kwargs)
if not self.encoding:
self.encoding = "utf-8"
self.include_headers_line = include_headers_line
- self.stream = io.TextIOWrapper(
+ self.stream = TextIOWrapper(
file,
line_buffering=False,
write_through=True,
@@ -235,11 +247,13 @@ class CsvItemExporter(BaseItemExporter):
self._headers_not_written = True
self._join_multivalued = join_multivalued
- def serialize_field(self, field, name, value):
- serializer = field.get("serializer", self._join_if_needed)
+ def serialize_field(
+ self, field: Mapping[str, Any] | Field, name: str, value: Any
+ ) -> Any:
+ serializer: Callable[[Any], Any] = field.get("serializer", self._join_if_needed)
return serializer(value)
- def _join_if_needed(self, value):
+ def _join_if_needed(self, value: Any) -> Any:
if isinstance(value, (list, tuple)):
try:
return self._join_multivalued.join(value)
@@ -247,7 +261,7 @@ class CsvItemExporter(BaseItemExporter):
pass
return value
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
if self._headers_not_written:
self._headers_not_written = False
self._write_headers_and_set_fields_to_export(item)
@@ -256,36 +270,38 @@ class CsvItemExporter(BaseItemExporter):
values = list(self._build_row(x for _, x in fields))
self.csv_writer.writerow(values)
- def finish_exporting(self):
+ def finish_exporting(self) -> None:
self.stream.detach() # Avoid closing the wrapped file.
- def _build_row(self, values):
+ def _build_row(self, values: Iterable[Any]) -> Iterable[Any]:
for s in values:
try:
yield to_unicode(s, self.encoding)
except TypeError:
yield s
- def _write_headers_and_set_fields_to_export(self, item):
+ def _write_headers_and_set_fields_to_export(self, item: Any) -> None:
if self.include_headers_line:
if not self.fields_to_export:
# use declared field names, or keys if the item is a dict
self.fields_to_export = ItemAdapter(item).field_names()
+ fields: Iterable[str]
if isinstance(self.fields_to_export, Mapping):
fields = self.fields_to_export.values()
else:
+ assert self.fields_to_export
fields = self.fields_to_export
row = list(self._build_row(fields))
self.csv_writer.writerow(row)
class PickleItemExporter(BaseItemExporter):
- def __init__(self, file, protocol=4, **kwargs):
+ def __init__(self, file: BytesIO, protocol: int = 4, **kwargs: Any):
super().__init__(**kwargs)
- self.file = file
- self.protocol = protocol
+ self.file: BytesIO = file
+ self.protocol: int = protocol
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
d = dict(self._get_serialized_fields(item))
pickle.dump(d, self.file, self.protocol)
@@ -299,20 +315,20 @@ class MarshalItemExporter(BaseItemExporter):
opened in binary mode, a :class:`~io.BytesIO` object, etc)
"""
- def __init__(self, file, **kwargs):
+ def __init__(self, file: BytesIO, **kwargs: Any):
super().__init__(**kwargs)
- self.file = file
+ self.file: BytesIO = file
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
marshal.dump(dict(self._get_serialized_fields(item)), self.file)
class PprintItemExporter(BaseItemExporter):
- def __init__(self, file, **kwargs):
+ def __init__(self, file: BytesIO, **kwargs: Any):
super().__init__(**kwargs)
- self.file = file
+ self.file: BytesIO = file
- def export_item(self, item):
+ def export_item(self, item: Any) -> None:
itemdict = dict(self._get_serialized_fields(item))
self.file.write(to_bytes(pprint.pformat(itemdict) + "\n"))
@@ -327,16 +343,20 @@ class PythonItemExporter(BaseItemExporter):
.. _msgpack: https://pypi.org/project/msgpack/
"""
- def _configure(self, options, dont_fail=False):
+ def _configure(self, options: dict[str, Any], dont_fail: bool = False) -> None:
super()._configure(options, dont_fail)
if not self.encoding:
self.encoding = "utf-8"
- def serialize_field(self, field, name, value):
- serializer = field.get("serializer", self._serialize_value)
+ def serialize_field(
+ self, field: Mapping[str, Any] | Field, name: str, value: Any
+ ) -> Any:
+ serializer: Callable[[Any], Any] = field.get(
+ "serializer", self._serialize_value
+ )
return serializer(value)
- def _serialize_value(self, value):
+ def _serialize_value(self, value: Any) -> Any:
if isinstance(value, Item):
return self.export_item(value)
if is_item(value):
@@ -347,10 +367,10 @@ class PythonItemExporter(BaseItemExporter):
return to_unicode(value, encoding=self.encoding)
return value
- def _serialize_item(self, item):
+ def _serialize_item(self, item: Any) -> Iterable[tuple[str | bytes, Any]]:
for key, value in ItemAdapter(item).items():
yield key, self._serialize_value(value)
- def export_item(self, item):
- result = dict(self._get_serialized_fields(item))
+ def export_item(self, item: Any) -> dict[str | bytes, Any]: # type: ignore[override]
+ result: dict[str | bytes, Any] = dict(self._get_serialized_fields(item))
return result
diff --git a/scrapy/extension.py b/scrapy/extension.py
index 4e365cfa1..9f978fa32 100644
--- a/scrapy/extension.py
+++ b/scrapy/extension.py
@@ -3,13 +3,21 @@ The Extension Manager
See documentation in docs/topics/extensions.rst
"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
from scrapy.middleware import MiddlewareManager
from scrapy.utils.conf import build_component_list
+if TYPE_CHECKING:
+ from scrapy.settings import Settings
+
class ExtensionManager(MiddlewareManager):
component_name = "extension"
@classmethod
- def _get_mwlist_from_settings(cls, settings):
+ def _get_mwlist_from_settings(cls, settings: Settings) -> list[Any]:
return build_component_list(settings.getwithbase("EXTENSIONS"))
diff --git a/scrapy/extensions/closespider.py b/scrapy/extensions/closespider.py
index 4307b4170..dff8bc97e 100644
--- a/scrapy/extensions/closespider.py
+++ b/scrapy/extensions/closespider.py
@@ -4,68 +4,96 @@ conditions are met.
See documentation in docs/topics/extensions.rst
"""
+from __future__ import annotations
+
import logging
from collections import defaultdict
+from typing import TYPE_CHECKING, Any
-from scrapy import signals
+from scrapy import Request, Spider, signals
from scrapy.exceptions import NotConfigured
+if TYPE_CHECKING:
+ from twisted.python.failure import Failure
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
+
logger = logging.getLogger(__name__)
class CloseSpider:
- def __init__(self, crawler):
- self.crawler = crawler
+ def __init__(self, crawler: Crawler):
+ self.crawler: Crawler = crawler
- self.close_on = {
+ self.close_on: dict[str, Any] = {
"timeout": crawler.settings.getfloat("CLOSESPIDER_TIMEOUT"),
"itemcount": crawler.settings.getint("CLOSESPIDER_ITEMCOUNT"),
"pagecount": crawler.settings.getint("CLOSESPIDER_PAGECOUNT"),
"errorcount": crawler.settings.getint("CLOSESPIDER_ERRORCOUNT"),
"timeout_no_item": crawler.settings.getint("CLOSESPIDER_TIMEOUT_NO_ITEM"),
+ "pagecount_no_item": crawler.settings.getint(
+ "CLOSESPIDER_PAGECOUNT_NO_ITEM"
+ ),
}
if not any(self.close_on.values()):
raise NotConfigured
- self.counter = defaultdict(int)
+ self.counter: defaultdict[str, int] = defaultdict(int)
if self.close_on.get("errorcount"):
crawler.signals.connect(self.error_count, signal=signals.spider_error)
- if self.close_on.get("pagecount"):
+ if self.close_on.get("pagecount") or self.close_on.get("pagecount_no_item"):
crawler.signals.connect(self.page_count, signal=signals.response_received)
if self.close_on.get("timeout"):
crawler.signals.connect(self.spider_opened, signal=signals.spider_opened)
- if self.close_on.get("itemcount"):
+ if self.close_on.get("itemcount") or self.close_on.get("pagecount_no_item"):
crawler.signals.connect(self.item_scraped, signal=signals.item_scraped)
if self.close_on.get("timeout_no_item"):
- self.timeout_no_item = self.close_on["timeout_no_item"]
- self.items_in_period = 0
+ self.timeout_no_item: int = self.close_on["timeout_no_item"]
+ self.items_in_period: int = 0
crawler.signals.connect(
self.spider_opened_no_item, signal=signals.spider_opened
)
crawler.signals.connect(
self.item_scraped_no_item, signal=signals.item_scraped
)
+
crawler.signals.connect(self.spider_closed, signal=signals.spider_closed)
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler)
- def error_count(self, failure, response, spider):
+ def error_count(self, failure: Failure, response: Response, spider: Spider) -> None:
self.counter["errorcount"] += 1
if self.counter["errorcount"] == self.close_on["errorcount"]:
+ assert self.crawler.engine
self.crawler.engine.close_spider(spider, "closespider_errorcount")
- def page_count(self, response, request, spider):
+ def page_count(self, response: Response, request: Request, spider: Spider) -> None:
self.counter["pagecount"] += 1
+ self.counter["pagecount_since_last_item"] += 1
if self.counter["pagecount"] == self.close_on["pagecount"]:
+ assert self.crawler.engine
self.crawler.engine.close_spider(spider, "closespider_pagecount")
+ return
+ if self.close_on["pagecount_no_item"] and (
+ self.counter["pagecount_since_last_item"]
+ >= self.close_on["pagecount_no_item"]
+ ):
+ assert self.crawler.engine
+ self.crawler.engine.close_spider(spider, "closespider_pagecount_no_item")
- def spider_opened(self, spider):
+ def spider_opened(self, spider: Spider) -> None:
from twisted.internet import reactor
+ assert self.crawler.engine
self.task = reactor.callLater(
self.close_on["timeout"],
self.crawler.engine.close_spider,
@@ -73,21 +101,23 @@ class CloseSpider:
reason="closespider_timeout",
)
- def item_scraped(self, item, spider):
+ def item_scraped(self, item: Any, spider: Spider) -> None:
self.counter["itemcount"] += 1
+ self.counter["pagecount_since_last_item"] = 0
if self.counter["itemcount"] == self.close_on["itemcount"]:
+ assert self.crawler.engine
self.crawler.engine.close_spider(spider, "closespider_itemcount")
- def spider_closed(self, spider):
- task = getattr(self, "task", False)
+ def spider_closed(self, spider: Spider) -> None:
+ task = getattr(self, "task", None)
if task and task.active():
task.cancel()
- task_no_item = getattr(self, "task_no_item", False)
+ task_no_item = getattr(self, "task_no_item", None)
if task_no_item and task_no_item.running:
task_no_item.stop()
- def spider_opened_no_item(self, spider):
+ def spider_opened_no_item(self, spider: Spider) -> None:
from twisted.internet import task
self.task_no_item = task.LoopingCall(self._count_items_produced, spider)
@@ -98,10 +128,10 @@ class CloseSpider:
f"{self.timeout_no_item} seconds."
)
- def item_scraped_no_item(self, item, spider):
+ def item_scraped_no_item(self, item: Any, spider: Spider) -> None:
self.items_in_period += 1
- def _count_items_produced(self, spider):
+ def _count_items_produced(self, spider: Spider) -> None:
if self.items_in_period >= 1:
self.items_in_period = 0
else:
@@ -109,4 +139,5 @@ class CloseSpider:
f"Closing spider since no items were produced in the last "
f"{self.timeout_no_item} seconds."
)
+ assert self.crawler.engine
self.crawler.engine.close_spider(spider, "closespider_timeout_no_item")
diff --git a/scrapy/extensions/corestats.py b/scrapy/extensions/corestats.py
index 302a615f2..779cd5d1c 100644
--- a/scrapy/extensions/corestats.py
+++ b/scrapy/extensions/corestats.py
@@ -1,18 +1,30 @@
"""
Extension for collecting core stats like items scraped and start/finish times
"""
-from datetime import datetime, timezone
-from scrapy import signals
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import TYPE_CHECKING, Any
+
+from scrapy import Spider, signals
+
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.statscollectors import StatsCollector
class CoreStats:
- def __init__(self, stats):
- self.stats = stats
- self.start_time = None
+ def __init__(self, stats: StatsCollector):
+ self.stats: StatsCollector = stats
+ self.start_time: datetime | None = None
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
+ assert crawler.stats
o = cls(crawler.stats)
crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)
@@ -21,11 +33,12 @@ class CoreStats:
crawler.signals.connect(o.response_received, signal=signals.response_received)
return o
- def spider_opened(self, spider):
+ def spider_opened(self, spider: Spider) -> None:
self.start_time = datetime.now(tz=timezone.utc)
self.stats.set_value("start_time", self.start_time, spider=spider)
- def spider_closed(self, spider, reason):
+ def spider_closed(self, spider: Spider, reason: str) -> None:
+ assert self.start_time is not None
finish_time = datetime.now(tz=timezone.utc)
elapsed_time = finish_time - self.start_time
elapsed_time_seconds = elapsed_time.total_seconds()
@@ -35,13 +48,13 @@ class CoreStats:
self.stats.set_value("finish_time", finish_time, spider=spider)
self.stats.set_value("finish_reason", reason, spider=spider)
- def item_scraped(self, item, spider):
+ def item_scraped(self, item: Any, spider: Spider) -> None:
self.stats.inc_value("item_scraped_count", spider=spider)
- def response_received(self, spider):
+ def response_received(self, spider: Spider) -> None:
self.stats.inc_value("response_received_count", spider=spider)
- def item_dropped(self, item, spider, exception):
+ def item_dropped(self, item: Any, spider: Spider, exception: BaseException) -> None:
reason = exception.__class__.__name__
self.stats.inc_value("item_dropped_count", spider=spider)
self.stats.inc_value(f"item_dropped_reasons_count/{reason}", spider=spider)
diff --git a/scrapy/extensions/debug.py b/scrapy/extensions/debug.py
index 1b6c7777f..6948c394c 100644
--- a/scrapy/extensions/debug.py
+++ b/scrapy/extensions/debug.py
@@ -4,34 +4,47 @@ Extensions for debugging Scrapy
See documentation in docs/topics/extensions.rst
"""
+from __future__ import annotations
+
import logging
import signal
import sys
import threading
import traceback
from pdb import Pdb
+from typing import TYPE_CHECKING
from scrapy.utils.engine import format_engine_status
from scrapy.utils.trackref import format_live_refs
+if TYPE_CHECKING:
+ from types import FrameType
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+
+
logger = logging.getLogger(__name__)
class StackTraceDump:
- def __init__(self, crawler=None):
- self.crawler = crawler
+ def __init__(self, crawler: Crawler):
+ self.crawler: Crawler = crawler
try:
- signal.signal(signal.SIGUSR2, self.dump_stacktrace)
- signal.signal(signal.SIGQUIT, self.dump_stacktrace)
+ signal.signal(signal.SIGUSR2, self.dump_stacktrace) # type: ignore[attr-defined]
+ signal.signal(signal.SIGQUIT, self.dump_stacktrace) # type: ignore[attr-defined]
except AttributeError:
# win32 platforms don't support SIGUSR signals
pass
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler)
- def dump_stacktrace(self, signum, frame):
+ def dump_stacktrace(self, signum: int, frame: FrameType | None) -> None:
+ assert self.crawler.engine
log_args = {
"stackdumps": self._thread_stacks(),
"enginestatus": format_engine_status(self.crawler.engine),
@@ -44,8 +57,8 @@ class StackTraceDump:
extra={"crawler": self.crawler},
)
- def _thread_stacks(self):
- id2name = dict((th.ident, th.name) for th in threading.enumerate())
+ def _thread_stacks(self) -> str:
+ id2name = {th.ident: th.name for th in threading.enumerate()}
dumps = ""
for id_, frame in sys._current_frames().items():
name = id2name.get(id_, "")
@@ -55,12 +68,13 @@ class StackTraceDump:
class Debugger:
- def __init__(self):
+ def __init__(self) -> None:
try:
- signal.signal(signal.SIGUSR2, self._enter_debugger)
+ signal.signal(signal.SIGUSR2, self._enter_debugger) # type: ignore[attr-defined]
except AttributeError:
# win32 platforms don't support SIGUSR signals
pass
- def _enter_debugger(self, signum, frame):
- Pdb().set_trace(frame.f_back)
+ def _enter_debugger(self, signum: int, frame: FrameType | None) -> None:
+ assert frame
+ Pdb().set_trace(frame.f_back) # noqa: T100
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
index fadbbb582..b6e6f55a6 100644
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -4,44 +4,68 @@ Feed Exports extension
See documentation in docs/topics/feed-exports.rst
"""
+from __future__ import annotations
+
import logging
import re
import sys
import warnings
+from collections.abc import Callable
from datetime import datetime, timezone
from pathlib import Path, PureWindowsPath
from tempfile import NamedTemporaryFile
-from typing import IO, Any, Callable, Dict, List, Optional, Tuple, Union
+from typing import IO, TYPE_CHECKING, Any, Optional, Protocol, TypeVar, cast
from urllib.parse import unquote, urlparse
-from twisted.internet import defer, threads
-from twisted.internet.defer import DeferredList
+from twisted.internet.defer import Deferred, DeferredList, maybeDeferred
+from twisted.internet.threads import deferToThread
from w3lib.url import file_uri_to_path
from zope.interface import Interface, implementer
from scrapy import Spider, signals
from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
from scrapy.extensions.postprocessing import PostProcessingManager
-from scrapy.utils.boto import is_botocore_available
from scrapy.utils.conf import feed_complete_default_values_from_settings
from scrapy.utils.defer import maybe_deferred_to_future
-from scrapy.utils.deprecate import create_deprecated_class
from scrapy.utils.ftp import ftp_store_file
from scrapy.utils.log import failure_to_exc_info
-from scrapy.utils.misc import create_instance, load_object
+from scrapy.utils.misc import build_from_crawler, load_object
from scrapy.utils.python import without_none_values
+if TYPE_CHECKING:
+ from collections.abc import Iterable
+
+ from _typeshed import OpenBinaryMode
+ from twisted.python.failure import Failure
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.exporters import BaseItemExporter
+ from scrapy.settings import BaseSettings, Settings
+
+
logger = logging.getLogger(__name__)
-try:
- import boto3 # noqa: F401
+UriParamsCallableT = Callable[[dict[str, Any], Spider], Optional[dict[str, Any]]]
- IS_BOTO3_AVAILABLE = True
-except ImportError:
- IS_BOTO3_AVAILABLE = False
+_StorageT = TypeVar("_StorageT", bound="FeedStorageProtocol")
-def build_storage(builder, uri, *args, feed_options=None, preargs=(), **kwargs):
+def build_storage(
+ builder: Callable[..., _StorageT],
+ uri: str,
+ *args: Any,
+ feed_options: dict[str, Any] | None = None,
+ preargs: Iterable[Any] = (),
+ **kwargs: Any,
+) -> _StorageT:
+ warnings.warn(
+ "scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly.",
+ category=ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
kwargs["feed_options"] = feed_options
return builder(*preargs, uri, *args, **kwargs)
@@ -55,10 +79,10 @@ class ItemFilter:
:type feed_options: dict
"""
- feed_options: Optional[dict]
- item_classes: Tuple
+ feed_options: dict[str, Any] | None
+ item_classes: tuple[type, ...]
- def __init__(self, feed_options: Optional[dict]) -> None:
+ def __init__(self, feed_options: dict[str, Any] | None) -> None:
self.feed_options = feed_options
if feed_options is not None:
self.item_classes = tuple(
@@ -66,7 +90,7 @@ class ItemFilter:
for item_class in feed_options.get("item_classes") or ()
)
else:
- self.item_classes = tuple()
+ self.item_classes = ()
def accepts(self, item: Any) -> bool:
"""
@@ -85,7 +109,7 @@ class ItemFilter:
class IFeedStorage(Interface):
"""Interface that all Feed Storages must implement"""
- def __init__(uri, *, feed_options=None):
+ def __init__(uri, *, feed_options=None): # pylint: disable=super-init-not-called
"""Initialize the storage with the parameters given in the URI and the
feed-specific options (see :setting:`FEEDS`)"""
@@ -97,28 +121,49 @@ class IFeedStorage(Interface):
"""Store the given file stream"""
+class FeedStorageProtocol(Protocol):
+ """Reimplementation of ``IFeedStorage`` that can be used in type hints."""
+
+ def __init__(self, uri: str, *, feed_options: dict[str, Any] | None = None):
+ """Initialize the storage with the parameters given in the URI and the
+ feed-specific options (see :setting:`FEEDS`)"""
+
+ def open(self, spider: Spider) -> IO[bytes]:
+ """Open the storage for the given spider. It must return a file-like
+ object that will be used for the exporters"""
+
+ def store(self, file: IO[bytes]) -> Deferred[None] | None:
+ """Store the given file stream"""
+
+
@implementer(IFeedStorage)
class BlockingFeedStorage:
- def open(self, spider):
+ def open(self, spider: Spider) -> IO[bytes]:
path = spider.crawler.settings["FEED_TEMPDIR"]
if path and not Path(path).is_dir():
raise OSError("Not a Directory: " + str(path))
return NamedTemporaryFile(prefix="feed-", dir=path)
- def store(self, file):
- return threads.deferToThread(self._store_in_thread, file)
+ def store(self, file: IO[bytes]) -> Deferred[None] | None:
+ return deferToThread(self._store_in_thread, file)
- def _store_in_thread(self, file):
+ def _store_in_thread(self, file: IO[bytes]) -> None:
raise NotImplementedError
@implementer(IFeedStorage)
class StdoutFeedStorage:
- def __init__(self, uri, _stdout=None, *, feed_options=None):
+ def __init__(
+ self,
+ uri: str,
+ _stdout: IO[bytes] | None = None,
+ *,
+ feed_options: dict[str, Any] | None = None,
+ ):
if not _stdout:
_stdout = sys.stdout.buffer
- self._stdout = _stdout
+ self._stdout: IO[bytes] = _stdout
if feed_options and feed_options.get("overwrite", False) is True:
logger.warning(
"Standard output (stdout) storage does not support "
@@ -127,87 +172,70 @@ class StdoutFeedStorage:
"it to False."
)
- def open(self, spider):
+ def open(self, spider: Spider) -> IO[bytes]:
return self._stdout
- def store(self, file):
+ def store(self, file: IO[bytes]) -> Deferred[None] | None:
pass
@implementer(IFeedStorage)
class FileFeedStorage:
- def __init__(self, uri, *, feed_options=None):
- self.path = file_uri_to_path(uri)
+ def __init__(self, uri: str, *, feed_options: dict[str, Any] | None = None):
+ self.path: str = file_uri_to_path(uri)
feed_options = feed_options or {}
- self.write_mode = "wb" if feed_options.get("overwrite", False) else "ab"
+ self.write_mode: OpenBinaryMode = (
+ "wb" if feed_options.get("overwrite", False) else "ab"
+ )
- def open(self, spider) -> IO[Any]:
+ def open(self, spider: Spider) -> IO[bytes]:
dirname = Path(self.path).parent
if dirname and not dirname.exists():
dirname.mkdir(parents=True)
return Path(self.path).open(self.write_mode)
- def store(self, file):
+ def store(self, file: IO[bytes]) -> Deferred[None] | None:
file.close()
+ return None
class S3FeedStorage(BlockingFeedStorage):
def __init__(
self,
- uri,
- access_key=None,
- secret_key=None,
- acl=None,
- endpoint_url=None,
+ uri: str,
+ access_key: str | None = None,
+ secret_key: str | None = None,
+ acl: str | None = None,
+ endpoint_url: str | None = None,
*,
- feed_options=None,
- session_token=None,
- region_name=None,
+ feed_options: dict[str, Any] | None = None,
+ session_token: str | None = None,
+ region_name: str | None = None,
):
- if not is_botocore_available():
- raise NotConfigured("missing botocore library")
- u = urlparse(uri)
- self.bucketname = u.hostname
- self.access_key = u.username or access_key
- self.secret_key = u.password or secret_key
- self.session_token = session_token
- self.keyname = u.path[1:] # remove first "/"
- self.acl = acl
- self.endpoint_url = endpoint_url
- self.region_name = region_name
-
- if IS_BOTO3_AVAILABLE:
+ try:
import boto3.session
+ except ImportError:
+ raise NotConfigured("missing boto3 library")
+ u = urlparse(uri)
+ assert u.hostname
+ self.bucketname: str = u.hostname
+ self.access_key: str | None = u.username or access_key
+ self.secret_key: str | None = u.password or secret_key
+ self.session_token: str | None = session_token
+ self.keyname: str = u.path[1:] # remove first "/"
+ self.acl: str | None = acl
+ self.endpoint_url: str | None = endpoint_url
+ self.region_name: str | None = region_name
- session = boto3.session.Session()
-
- self.s3_client = session.client(
- "s3",
- aws_access_key_id=self.access_key,
- aws_secret_access_key=self.secret_key,
- aws_session_token=self.session_token,
- endpoint_url=self.endpoint_url,
- region_name=self.region_name,
- )
- else:
- warnings.warn(
- "`botocore` usage has been deprecated for S3 feed "
- "export, please use `boto3` to avoid problems",
- category=ScrapyDeprecationWarning,
- )
-
- import botocore.session
-
- session = botocore.session.get_session()
-
- self.s3_client = session.create_client(
- "s3",
- aws_access_key_id=self.access_key,
- aws_secret_access_key=self.secret_key,
- aws_session_token=self.session_token,
- endpoint_url=self.endpoint_url,
- region_name=self.region_name,
- )
+ boto3_session = boto3.session.Session()
+ self.s3_client = boto3_session.client(
+ "s3",
+ aws_access_key_id=self.access_key,
+ aws_secret_access_key=self.secret_key,
+ aws_session_token=self.session_token,
+ endpoint_url=self.endpoint_url,
+ region_name=self.region_name,
+ )
if feed_options and feed_options.get("overwrite", True) is False:
logger.warning(
@@ -217,9 +245,14 @@ class S3FeedStorage(BlockingFeedStorage):
)
@classmethod
- def from_crawler(cls, crawler, uri, *, feed_options=None):
- return build_storage(
- cls,
+ def from_crawler(
+ cls,
+ crawler: Crawler,
+ uri: str,
+ *,
+ feed_options: dict[str, Any] | None = None,
+ ) -> Self:
+ return cls(
uri,
access_key=crawler.settings["AWS_ACCESS_KEY_ID"],
secret_key=crawler.settings["AWS_SECRET_ACCESS_KEY"],
@@ -230,38 +263,33 @@ class S3FeedStorage(BlockingFeedStorage):
feed_options=feed_options,
)
- def _store_in_thread(self, file):
+ def _store_in_thread(self, file: IO[bytes]) -> None:
file.seek(0)
- if IS_BOTO3_AVAILABLE:
- kwargs = {"ExtraArgs": {"ACL": self.acl}} if self.acl else {}
- self.s3_client.upload_fileobj(
- Bucket=self.bucketname, Key=self.keyname, Fileobj=file, **kwargs
- )
- else:
- kwargs = {"ACL": self.acl} if self.acl else {}
- self.s3_client.put_object(
- Bucket=self.bucketname, Key=self.keyname, Body=file, **kwargs
- )
+ kwargs: dict[str, Any] = {"ExtraArgs": {"ACL": self.acl}} if self.acl else {}
+ self.s3_client.upload_fileobj(
+ Bucket=self.bucketname, Key=self.keyname, Fileobj=file, **kwargs
+ )
file.close()
class GCSFeedStorage(BlockingFeedStorage):
- def __init__(self, uri, project_id, acl):
- self.project_id = project_id
- self.acl = acl
+ def __init__(self, uri: str, project_id: str | None, acl: str | None):
+ self.project_id: str | None = project_id
+ self.acl: str | None = acl
u = urlparse(uri)
- self.bucket_name = u.hostname
- self.blob_name = u.path[1:] # remove first "/"
+ assert u.hostname
+ self.bucket_name: str = u.hostname
+ self.blob_name: str = u.path[1:] # remove first "/"
@classmethod
- def from_crawler(cls, crawler, uri):
+ def from_crawler(cls, crawler: Crawler, uri: str) -> Self:
return cls(
uri,
crawler.settings["GCS_PROJECT_ID"],
crawler.settings["FEED_STORAGE_GCS_ACL"] or None,
)
- def _store_in_thread(self, file):
+ def _store_in_thread(self, file: IO[bytes]) -> None:
file.seek(0)
from google.cloud.storage import Client
@@ -277,7 +305,7 @@ class FTPFeedStorage(BlockingFeedStorage):
uri: str,
use_active_mode: bool = False,
*,
- feed_options: Optional[Dict[str, Any]] = None,
+ feed_options: dict[str, Any] | None = None,
):
u = urlparse(uri)
if not u.hostname:
@@ -291,15 +319,20 @@ class FTPFeedStorage(BlockingFeedStorage):
self.overwrite: bool = not feed_options or feed_options.get("overwrite", True)
@classmethod
- def from_crawler(cls, crawler, uri, *, feed_options=None):
- return build_storage(
- cls,
+ def from_crawler(
+ cls,
+ crawler: Crawler,
+ uri: str,
+ *,
+ feed_options: dict[str, Any] | None = None,
+ ) -> Self:
+ return cls(
uri,
- crawler.settings.getbool("FEED_STORAGE_FTP_ACTIVE"),
+ use_active_mode=crawler.settings.getbool("FEED_STORAGE_FTP_ACTIVE"),
feed_options=feed_options,
)
- def _store_in_thread(self, file):
+ def _store_in_thread(self, file: IO[bytes]) -> None:
ftp_store_file(
path=self.path,
file=file,
@@ -315,46 +348,51 @@ class FTPFeedStorage(BlockingFeedStorage):
class FeedSlot:
def __init__(
self,
- storage,
- uri,
- format,
- store_empty,
- batch_id,
- uri_template,
- filter,
- feed_options,
- spider,
- exporters,
- settings,
- crawler,
+ storage: FeedStorageProtocol,
+ uri: str,
+ format: str,
+ store_empty: bool,
+ batch_id: int,
+ uri_template: str,
+ filter: ItemFilter,
+ feed_options: dict[str, Any],
+ spider: Spider,
+ exporters: dict[str, type[BaseItemExporter]],
+ settings: BaseSettings,
+ crawler: Crawler,
):
- self.file = None
- self.exporter = None
- self.storage = storage
+ self.file: IO[bytes] | None = None
+ self.exporter: BaseItemExporter | None = None
+ self.storage: FeedStorageProtocol = storage
# feed params
- self.batch_id = batch_id
- self.format = format
- self.store_empty = store_empty
- self.uri_template = uri_template
- self.uri = uri
- self.filter = filter
+ self.batch_id: int = batch_id
+ self.format: str = format
+ self.store_empty: bool = store_empty
+ self.uri_template: str = uri_template
+ self.uri: str = uri
+ self.filter: ItemFilter = filter
# exporter params
- self.feed_options = feed_options
- self.spider = spider
- self.exporters = exporters
- self.settings = settings
- self.crawler = crawler
+ self.feed_options: dict[str, Any] = feed_options
+ self.spider: Spider = spider
+ self.exporters: dict[str, type[BaseItemExporter]] = exporters
+ self.settings: BaseSettings = settings
+ self.crawler: Crawler = crawler
# flags
- self.itemcount = 0
- self._exporting = False
- self._fileloaded = False
+ self.itemcount: int = 0
+ self._exporting: bool = False
+ self._fileloaded: bool = False
- def start_exporting(self):
+ def start_exporting(self) -> None:
if not self._fileloaded:
self.file = self.storage.open(self.spider)
if "postprocessing" in self.feed_options:
- self.file = PostProcessingManager(
- self.feed_options["postprocessing"], self.file, self.feed_options
+ self.file = cast(
+ IO[bytes],
+ PostProcessingManager(
+ self.feed_options["postprocessing"],
+ self.file,
+ self.feed_options,
+ ),
)
self.exporter = self._get_exporter(
file=self.file,
@@ -367,44 +405,41 @@ class FeedSlot:
self._fileloaded = True
if not self._exporting:
+ assert self.exporter
self.exporter.start_exporting()
self._exporting = True
- def _get_instance(self, objcls, *args, **kwargs):
- return create_instance(objcls, self.settings, self.crawler, *args, **kwargs)
+ def _get_exporter(
+ self, file: IO[bytes], format: str, *args: Any, **kwargs: Any
+ ) -> BaseItemExporter:
+ return build_from_crawler(
+ self.exporters[format], self.crawler, file, *args, **kwargs
+ )
- def _get_exporter(self, file, format, *args, **kwargs):
- return self._get_instance(self.exporters[format], file, *args, **kwargs)
-
- def finish_exporting(self):
+ def finish_exporting(self) -> None:
if self._exporting:
+ assert self.exporter
self.exporter.finish_exporting()
self._exporting = False
-_FeedSlot = create_deprecated_class(
- name="_FeedSlot",
- new_class=FeedSlot,
-)
-
-
class FeedExporter:
- _pending_deferreds: List[defer.Deferred] = []
+ _pending_deferreds: list[Deferred[None]] = []
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
exporter = cls(crawler)
crawler.signals.connect(exporter.open_spider, signals.spider_opened)
crawler.signals.connect(exporter.close_spider, signals.spider_closed)
crawler.signals.connect(exporter.item_scraped, signals.item_scraped)
return exporter
- def __init__(self, crawler):
- self.crawler = crawler
- self.settings = crawler.settings
+ def __init__(self, crawler: Crawler):
+ self.crawler: Crawler = crawler
+ self.settings: Settings = crawler.settings
self.feeds = {}
- self.slots = []
- self.filters = {}
+ self.slots: list[FeedSlot] = []
+ self.filters: dict[str, ItemFilter] = {}
if not self.settings["FEEDS"] and not self.settings["FEED_URI"]:
raise NotConfigured
@@ -436,8 +471,12 @@ class FeedExporter:
)
self.filters[uri] = self._load_filter(feed_options)
- self.storages = self._load_components("FEED_STORAGES")
- self.exporters = self._load_components("FEED_EXPORTERS")
+ self.storages: dict[str, type[FeedStorageProtocol]] = self._load_components(
+ "FEED_STORAGES"
+ )
+ self.exporters: dict[str, type[BaseItemExporter]] = self._load_components(
+ "FEED_EXPORTERS"
+ )
for uri, feed_options in self.feeds.items():
if not self._storage_supported(uri, feed_options):
raise NotConfigured
@@ -446,7 +485,7 @@ class FeedExporter:
if not self._exporter_supported(feed_options["format"]):
raise NotConfigured
- def open_spider(self, spider):
+ def open_spider(self, spider: Spider) -> None:
for uri, feed_options in self.feeds.items():
uri_params = self._get_uri_params(spider, feed_options["uri_params"])
self.slots.append(
@@ -459,7 +498,7 @@ class FeedExporter:
)
)
- async def close_spider(self, spider):
+ async def close_spider(self, spider: Spider) -> None:
for slot in self.slots:
self._close_slot(slot, spider)
@@ -472,8 +511,9 @@ class FeedExporter:
self.crawler.signals.send_catch_log_deferred(signals.feed_exporter_closed)
)
- def _close_slot(self, slot, spider):
- def get_file(slot_):
+ def _close_slot(self, slot: FeedSlot, spider: Spider) -> Deferred[None] | None:
+ def get_file(slot_: FeedSlot) -> IO[bytes]:
+ assert slot_.file
if isinstance(slot_.file, PostProcessingManager):
slot_.file.close()
return slot_.file.file
@@ -491,7 +531,7 @@ class FeedExporter:
return None
logmsg = f"{slot.format} feed ({slot.itemcount} items) in: {slot.uri}"
- d = defer.maybeDeferred(slot.storage.store, get_file(slot))
+ d: Deferred[None] = maybeDeferred(slot.storage.store, get_file(slot)) # type: ignore[call-overload]
d.addCallback(
self._handle_store_success, logmsg, spider, type(slot.storage).__name__
@@ -509,20 +549,33 @@ class FeedExporter:
return d
- def _handle_store_error(self, f, logmsg, spider, slot_type):
+ def _handle_store_error(
+ self, f: Failure, logmsg: str, spider: Spider, slot_type: str
+ ) -> None:
logger.error(
"Error storing %s",
logmsg,
exc_info=failure_to_exc_info(f),
extra={"spider": spider},
)
+ assert self.crawler.stats
self.crawler.stats.inc_value(f"feedexport/failed_count/{slot_type}")
- def _handle_store_success(self, f, logmsg, spider, slot_type):
+ def _handle_store_success(
+ self, result: Any, logmsg: str, spider: Spider, slot_type: str
+ ) -> None:
logger.info("Stored %s", logmsg, extra={"spider": spider})
+ assert self.crawler.stats
self.crawler.stats.inc_value(f"feedexport/success_count/{slot_type}")
- def _start_new_batch(self, batch_id, uri, feed_options, spider, uri_template):
+ def _start_new_batch(
+ self,
+ batch_id: int,
+ uri: str,
+ feed_options: dict[str, Any],
+ spider: Spider,
+ uri_template: str,
+ ) -> FeedSlot:
"""
Redirect the output data stream to a new file.
Execute multiple times if FEED_EXPORT_BATCH_ITEM_COUNT setting or FEEDS.batch_item_count is specified
@@ -533,7 +586,7 @@ class FeedExporter:
:param uri_template: template of uri which contains %(batch_time)s or %(batch_id)d to create new uri
"""
storage = self._get_storage(uri, feed_options)
- slot = FeedSlot(
+ return FeedSlot(
storage=storage,
uri=uri,
format=feed_options["format"],
@@ -545,11 +598,10 @@ class FeedExporter:
spider=spider,
exporters=self.exporters,
settings=self.settings,
- crawler=getattr(self, "crawler", None),
+ crawler=self.crawler,
)
- return slot
- def item_scraped(self, item, spider):
+ def item_scraped(self, item: Any, spider: Spider) -> None:
slots = []
for slot in self.slots:
if not slot.filter.accepts(item):
@@ -559,6 +611,7 @@ class FeedExporter:
continue
slot.start_exporting()
+ assert slot.exporter
slot.exporter.export_item(item)
slot.itemcount += 1
# create new slot for each slot with itemcount == FEED_EXPORT_BATCH_ITEM_COUNT and close the old one
@@ -583,8 +636,10 @@ class FeedExporter:
slots.append(slot)
self.slots = slots
- def _load_components(self, setting_prefix):
- conf = without_none_values(self.settings.getwithbase(setting_prefix))
+ def _load_components(self, setting_prefix: str) -> dict[str, Any]:
+ conf = without_none_values(
+ cast(dict[str, str], self.settings.getwithbase(setting_prefix))
+ )
d = {}
for k, v in conf.items():
try:
@@ -593,12 +648,13 @@ class FeedExporter:
pass
return d
- def _exporter_supported(self, format):
+ def _exporter_supported(self, format: str) -> bool:
if format in self.exporters:
return True
logger.error("Unknown feed format: %(format)s", {"format": format})
+ return False
- def _settings_are_valid(self):
+ def _settings_are_valid(self) -> bool:
"""
If FEED_EXPORT_BATCH_ITEM_COUNT setting or FEEDS.batch_item_count is specified uri has to contain
%(batch_time)s or %(batch_id)d to distinguish different files of partial output
@@ -616,7 +672,7 @@ class FeedExporter:
return False
return True
- def _storage_supported(self, uri, feed_options):
+ def _storage_supported(self, uri: str, feed_options: dict[str, Any]) -> bool:
scheme = urlparse(uri).scheme
if scheme in self.storages or PureWindowsPath(uri).drive:
try:
@@ -624,45 +680,27 @@ class FeedExporter:
return True
except NotConfigured as e:
logger.error(
- "Disabled feed storage scheme: %(scheme)s. " "Reason: %(reason)s",
+ "Disabled feed storage scheme: %(scheme)s. Reason: %(reason)s",
{"scheme": scheme, "reason": str(e)},
)
else:
logger.error("Unknown feed storage scheme: %(scheme)s", {"scheme": scheme})
+ return False
- def _get_storage(self, uri, feed_options):
- """Fork of create_instance specific to feed storage classes
-
- It supports not passing the *feed_options* parameters to classes that
- do not support it, and issuing a deprecation warning instead.
- """
- feedcls = self.storages.get(urlparse(uri).scheme, self.storages["file"])
- crawler = getattr(self, "crawler", None)
-
- def build_instance(builder, *preargs):
- return build_storage(
- builder, uri, feed_options=feed_options, preargs=preargs
- )
-
- if crawler and hasattr(feedcls, "from_crawler"):
- instance = build_instance(feedcls.from_crawler, crawler)
- method_name = "from_crawler"
- elif hasattr(feedcls, "from_settings"):
- instance = build_instance(feedcls.from_settings, self.settings)
- method_name = "from_settings"
- else:
- instance = build_instance(feedcls)
- method_name = "__new__"
- if instance is None:
- raise TypeError(f"{feedcls.__qualname__}.{method_name} returned None")
- return instance
+ def _get_storage(
+ self, uri: str, feed_options: dict[str, Any]
+ ) -> FeedStorageProtocol:
+ """Build a storage object for the specified *uri* with the specified
+ *feed_options*."""
+ cls = self.storages.get(urlparse(uri).scheme, self.storages["file"])
+ return build_from_crawler(cls, self.crawler, uri, feed_options=feed_options)
def _get_uri_params(
self,
spider: Spider,
- uri_params_function: Optional[Union[str, Callable[[dict, Spider], dict]]],
- slot: Optional[FeedSlot] = None,
- ) -> dict:
+ uri_params_function: str | UriParamsCallableT | None,
+ slot: FeedSlot | None = None,
+ ) -> dict[str, Any]:
params = {}
for k in dir(spider):
params[k] = getattr(spider, k)
@@ -670,7 +708,7 @@ class FeedExporter:
params["time"] = utc_now.replace(microsecond=0).isoformat().replace(":", "-")
params["batch_time"] = utc_now.isoformat().replace(":", "-")
params["batch_id"] = slot.batch_id + 1 if slot is not None else 1
- uripar_function = (
+ uripar_function: UriParamsCallableT = (
load_object(uri_params_function)
if uri_params_function
else lambda params, _: params
@@ -678,7 +716,9 @@ class FeedExporter:
new_params = uripar_function(params, spider)
return new_params if new_params is not None else params
- def _load_filter(self, feed_options):
+ def _load_filter(self, feed_options: dict[str, Any]) -> ItemFilter:
# load the item filter if declared else load the default filter class
- item_filter_class = load_object(feed_options.get("item_filter", ItemFilter))
+ item_filter_class: type[ItemFilter] = load_object(
+ feed_options.get("item_filter", ItemFilter)
+ )
return item_filter_class(feed_options)
diff --git a/scrapy/extensions/httpcache.py b/scrapy/extensions/httpcache.py
index 7e4f047a8..929807de8 100644
--- a/scrapy/extensions/httpcache.py
+++ b/scrapy/extensions/httpcache.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
import gzip
import logging
import pickle
@@ -5,56 +7,76 @@ from email.utils import mktime_tz, parsedate_tz
from importlib import import_module
from pathlib import Path
from time import time
+from typing import IO, TYPE_CHECKING, Any, cast
from weakref import WeakKeyDictionary
from w3lib.http import headers_dict_to_raw, headers_raw_to_dict
from scrapy.http import Headers, Response
-from scrapy.http.request import Request
from scrapy.responsetypes import responsetypes
-from scrapy.spiders import Spider
from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.project import data_path
from scrapy.utils.python import to_bytes, to_unicode
+if TYPE_CHECKING:
+ import os
+ from collections.abc import Callable
+ from types import ModuleType
+
+ # typing.Concatenate requires Python 3.10
+ from typing_extensions import Concatenate
+
+ from scrapy.http.request import Request
+ from scrapy.settings import BaseSettings
+ from scrapy.spiders import Spider
+ from scrapy.utils.request import RequestFingerprinter
+
+
logger = logging.getLogger(__name__)
class DummyPolicy:
- def __init__(self, settings):
- self.ignore_schemes = settings.getlist("HTTPCACHE_IGNORE_SCHEMES")
- self.ignore_http_codes = [
+ def __init__(self, settings: BaseSettings):
+ self.ignore_schemes: list[str] = settings.getlist("HTTPCACHE_IGNORE_SCHEMES")
+ self.ignore_http_codes: list[int] = [
int(x) for x in settings.getlist("HTTPCACHE_IGNORE_HTTP_CODES")
]
- def should_cache_request(self, request):
+ def should_cache_request(self, request: Request) -> bool:
return urlparse_cached(request).scheme not in self.ignore_schemes
- def should_cache_response(self, response, request):
+ def should_cache_response(self, response: Response, request: Request) -> bool:
return response.status not in self.ignore_http_codes
- def is_cached_response_fresh(self, cachedresponse, request):
+ def is_cached_response_fresh(
+ self, cachedresponse: Response, request: Request
+ ) -> bool:
return True
- def is_cached_response_valid(self, cachedresponse, response, request):
+ def is_cached_response_valid(
+ self, cachedresponse: Response, response: Response, request: Request
+ ) -> bool:
return True
class RFC2616Policy:
MAXAGE = 3600 * 24 * 365 # one year
- def __init__(self, settings):
- self.always_store = settings.getbool("HTTPCACHE_ALWAYS_STORE")
- self.ignore_schemes = settings.getlist("HTTPCACHE_IGNORE_SCHEMES")
- self._cc_parsed = WeakKeyDictionary()
- self.ignore_response_cache_controls = [
+ def __init__(self, settings: BaseSettings):
+ self.always_store: bool = settings.getbool("HTTPCACHE_ALWAYS_STORE")
+ self.ignore_schemes: list[str] = settings.getlist("HTTPCACHE_IGNORE_SCHEMES")
+ self._cc_parsed: WeakKeyDictionary[
+ Request | Response, dict[bytes, bytes | None]
+ ] = WeakKeyDictionary()
+ self.ignore_response_cache_controls: list[bytes] = [
to_bytes(cc)
for cc in settings.getlist("HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS")
]
- def _parse_cachecontrol(self, r):
+ def _parse_cachecontrol(self, r: Request | Response) -> dict[bytes, bytes | None]:
if r not in self._cc_parsed:
cch = r.headers.get(b"Cache-Control", b"")
+ assert cch is not None
parsed = parse_cachecontrol(cch)
if isinstance(r, Response):
for key in self.ignore_response_cache_controls:
@@ -62,7 +84,7 @@ class RFC2616Policy:
self._cc_parsed[r] = parsed
return self._cc_parsed[r]
- def should_cache_request(self, request):
+ def should_cache_request(self, request: Request) -> bool:
if urlparse_cached(request).scheme in self.ignore_schemes:
return False
cc = self._parse_cachecontrol(request)
@@ -72,7 +94,7 @@ class RFC2616Policy:
# Any other is eligible for caching
return True
- def should_cache_response(self, response, request):
+ def should_cache_response(self, response: Response, request: Request) -> bool:
# What is cacheable - https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1
# Response cacheability - https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.4
# Status code 206 is not included because cache can not deal with partial contents
@@ -100,7 +122,9 @@ class RFC2616Policy:
# info and can not be revalidated
return False
- def is_cached_response_fresh(self, cachedresponse, request):
+ def is_cached_response_fresh(
+ self, cachedresponse: Response, request: Request
+ ) -> bool:
cc = self._parse_cachecontrol(cachedresponse)
ccreq = self._parse_cachecontrol(request)
if b"no-cache" in cc or b"no-cache" in ccreq:
@@ -141,7 +165,9 @@ class RFC2616Policy:
self._set_conditional_validators(request, cachedresponse)
return False
- def is_cached_response_valid(self, cachedresponse, response, request):
+ def is_cached_response_valid(
+ self, cachedresponse: Response, response: Response, request: Request
+ ) -> bool:
# Use the cached response if the new response is a server error,
# as long as the old response didn't specify must-revalidate.
if response.status >= 500:
@@ -152,7 +178,9 @@ class RFC2616Policy:
# Use the cached response if the server says it hasn't changed.
return response.status == 304
- def _set_conditional_validators(self, request, cachedresponse):
+ def _set_conditional_validators(
+ self, request: Request, cachedresponse: Response
+ ) -> None:
if b"Last-Modified" in cachedresponse.headers:
request.headers[b"If-Modified-Since"] = cachedresponse.headers[
b"Last-Modified"
@@ -161,13 +189,15 @@ class RFC2616Policy:
if b"ETag" in cachedresponse.headers:
request.headers[b"If-None-Match"] = cachedresponse.headers[b"ETag"]
- def _get_max_age(self, cc):
+ def _get_max_age(self, cc: dict[bytes, bytes | None]) -> int | None:
try:
- return max(0, int(cc[b"max-age"]))
+ return max(0, int(cc[b"max-age"])) # type: ignore[arg-type]
except (KeyError, ValueError):
return None
- def _compute_freshness_lifetime(self, response, request, now):
+ def _compute_freshness_lifetime(
+ self, response: Response, request: Request, now: float
+ ) -> float:
# Reference nsHttpResponseHead::ComputeFreshnessLifetime
# https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#706
cc = self._parse_cachecontrol(response)
@@ -198,10 +228,12 @@ class RFC2616Policy:
# Insufficient information to compute freshness lifetime
return 0
- def _compute_current_age(self, response, request, now):
+ def _compute_current_age(
+ self, response: Response, request: Request, now: float
+ ) -> float:
# Reference nsHttpResponseHead::ComputeCurrentAge
# https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#658
- currentage = 0
+ currentage: float = 0
# If Date header is not set we assume it is a fast connection, and
# clock is in sync with the server
date = rfc1123_to_epoch(response.headers.get(b"Date")) or now
@@ -210,7 +242,7 @@ class RFC2616Policy:
if b"Age" in response.headers:
try:
- age = int(response.headers[b"Age"])
+ age = int(response.headers[b"Age"]) # type: ignore[arg-type]
currentage = max(currentage, age)
except ValueError:
pass
@@ -219,13 +251,13 @@ class RFC2616Policy:
class DbmCacheStorage:
- def __init__(self, settings):
- self.cachedir = data_path(settings["HTTPCACHE_DIR"], createdir=True)
- self.expiration_secs = settings.getint("HTTPCACHE_EXPIRATION_SECS")
- self.dbmodule = import_module(settings["HTTPCACHE_DBM_MODULE"])
- self.db = None
+ def __init__(self, settings: BaseSettings):
+ self.cachedir: str = data_path(settings["HTTPCACHE_DIR"], createdir=True)
+ self.expiration_secs: int = settings.getint("HTTPCACHE_EXPIRATION_SECS")
+ self.dbmodule: ModuleType = import_module(settings["HTTPCACHE_DBM_MODULE"])
+ self.db: Any = None # the real type is private
- def open_spider(self, spider: Spider):
+ def open_spider(self, spider: Spider) -> None:
dbpath = Path(self.cachedir, f"{spider.name}.db")
self.db = self.dbmodule.open(str(dbpath), "c")
@@ -235,24 +267,26 @@ class DbmCacheStorage:
extra={"spider": spider},
)
- self._fingerprinter = spider.crawler.request_fingerprinter
+ assert spider.crawler.request_fingerprinter
+ self._fingerprinter: RequestFingerprinter = spider.crawler.request_fingerprinter
- def close_spider(self, spider):
+ def close_spider(self, spider: Spider) -> None:
self.db.close()
- def retrieve_response(self, spider, request):
+ def retrieve_response(self, spider: Spider, request: Request) -> Response | None:
data = self._read_data(spider, request)
if data is None:
- return # not cached
+ return None # not cached
url = data["url"]
status = data["status"]
headers = Headers(data["headers"])
body = data["body"]
respcls = responsetypes.from_args(headers=headers, url=url, body=body)
- response = respcls(url=url, headers=headers, status=status, body=body)
- return response
+ return respcls(url=url, headers=headers, status=status, body=body)
- def store_response(self, spider, request, response):
+ def store_response(
+ self, spider: Spider, request: Request, response: Response
+ ) -> None:
key = self._fingerprinter.fingerprint(request).hex()
data = {
"status": response.status,
@@ -263,28 +297,31 @@ class DbmCacheStorage:
self.db[f"{key}_data"] = pickle.dumps(data, protocol=4)
self.db[f"{key}_time"] = str(time())
- def _read_data(self, spider, request):
+ def _read_data(self, spider: Spider, request: Request) -> dict[str, Any] | None:
key = self._fingerprinter.fingerprint(request).hex()
db = self.db
tkey = f"{key}_time"
if tkey not in db:
- return # not found
+ return None # not found
ts = db[tkey]
if 0 < self.expiration_secs < time() - float(ts):
- return # expired
+ return None # expired
- return pickle.loads(db[f"{key}_data"])
+ return cast(dict[str, Any], pickle.loads(db[f"{key}_data"])) # noqa: S301
class FilesystemCacheStorage:
- def __init__(self, settings):
- self.cachedir = data_path(settings["HTTPCACHE_DIR"])
- self.expiration_secs = settings.getint("HTTPCACHE_EXPIRATION_SECS")
- self.use_gzip = settings.getbool("HTTPCACHE_GZIP")
- self._open = gzip.open if self.use_gzip else open
+ def __init__(self, settings: BaseSettings):
+ self.cachedir: str = data_path(settings["HTTPCACHE_DIR"])
+ self.expiration_secs: int = settings.getint("HTTPCACHE_EXPIRATION_SECS")
+ self.use_gzip: bool = settings.getbool("HTTPCACHE_GZIP")
+ # https://github.com/python/mypy/issues/10740
+ self._open: Callable[Concatenate[str | os.PathLike, str, ...], IO[bytes]] = (
+ gzip.open if self.use_gzip else open # type: ignore[assignment]
+ )
- def open_spider(self, spider: Spider):
+ def open_spider(self, spider: Spider) -> None:
logger.debug(
"Using filesystem cache storage in %(cachedir)s",
{"cachedir": self.cachedir},
@@ -294,27 +331,28 @@ class FilesystemCacheStorage:
assert spider.crawler.request_fingerprinter
self._fingerprinter = spider.crawler.request_fingerprinter
- def close_spider(self, spider):
+ def close_spider(self, spider: Spider) -> None:
pass
- def retrieve_response(self, spider: Spider, request: Request):
+ def retrieve_response(self, spider: Spider, request: Request) -> Response | None:
"""Return response if present in cache, or None otherwise."""
metadata = self._read_meta(spider, request)
if metadata is None:
- return # not cached
+ return None # not cached
rpath = Path(self._get_request_path(spider, request))
with self._open(rpath / "response_body", "rb") as f:
body = f.read()
with self._open(rpath / "response_headers", "rb") as f:
rawheaders = f.read()
- url = metadata.get("response_url")
+ url = metadata["response_url"]
status = metadata["status"]
headers = Headers(headers_raw_to_dict(rawheaders))
respcls = responsetypes.from_args(headers=headers, url=url, body=body)
- response = respcls(url=url, headers=headers, status=status, body=body)
- return response
+ return respcls(url=url, headers=headers, status=status, body=body)
- def store_response(self, spider: Spider, request: Request, response):
+ def store_response(
+ self, spider: Spider, request: Request, response: Response
+ ) -> None:
"""Store the given response in the cache."""
rpath = Path(self._get_request_path(spider, request))
if not rpath.exists():
@@ -343,19 +381,19 @@ class FilesystemCacheStorage:
key = self._fingerprinter.fingerprint(request).hex()
return str(Path(self.cachedir, spider.name, key[0:2], key))
- def _read_meta(self, spider: Spider, request: Request):
+ def _read_meta(self, spider: Spider, request: Request) -> dict[str, Any] | None:
rpath = Path(self._get_request_path(spider, request))
metapath = rpath / "pickled_meta"
if not metapath.exists():
- return # not found
+ return None # not found
mtime = metapath.stat().st_mtime
if 0 < self.expiration_secs < time() - mtime:
- return # expired
+ return None # expired
with self._open(metapath, "rb") as f:
- return pickle.load(f)
+ return cast(dict[str, Any], pickle.load(f)) # noqa: S301
-def parse_cachecontrol(header):
+def parse_cachecontrol(header: bytes) -> dict[bytes, bytes | None]:
"""Parse Cache-Control header
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9
@@ -375,9 +413,9 @@ def parse_cachecontrol(header):
return directives
-def rfc1123_to_epoch(date_str):
+def rfc1123_to_epoch(date_str: str | bytes | None) -> int | None:
try:
- date_str = to_unicode(date_str, encoding="ascii")
- return mktime_tz(parsedate_tz(date_str))
+ date_str = to_unicode(date_str, encoding="ascii") # type: ignore[arg-type]
+ return mktime_tz(parsedate_tz(date_str)) # type: ignore[arg-type]
except Exception:
return None
diff --git a/scrapy/extensions/logstats.py b/scrapy/extensions/logstats.py
index 78874a6db..e829d8b92 100644
--- a/scrapy/extensions/logstats.py
+++ b/scrapy/extensions/logstats.py
@@ -1,58 +1,96 @@
+from __future__ import annotations
+
import logging
+from typing import TYPE_CHECKING
from twisted.internet import task
-from scrapy import signals
+from scrapy import Spider, signals
from scrapy.exceptions import NotConfigured
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.statscollectors import StatsCollector
+
+
logger = logging.getLogger(__name__)
class LogStats:
- """Log basic scraping stats periodically"""
+ """Log basic scraping stats periodically like:
+ * RPM - Requests per Minute
+ * IPM - Items per Minute
+ """
- def __init__(self, stats, interval=60.0):
- self.stats = stats
- self.interval = interval
- self.multiplier = 60.0 / self.interval
- self.task = None
+ def __init__(self, stats: StatsCollector, interval: float = 60.0):
+ self.stats: StatsCollector = stats
+ self.interval: float = interval
+ self.multiplier: float = 60.0 / self.interval
+ self.task: task.LoopingCall | None = None
@classmethod
- def from_crawler(cls, crawler):
- interval = crawler.settings.getfloat("LOGSTATS_INTERVAL")
+ def from_crawler(cls, crawler: Crawler) -> Self:
+ interval: float = crawler.settings.getfloat("LOGSTATS_INTERVAL")
if not interval:
raise NotConfigured
+ assert crawler.stats
o = cls(crawler.stats, interval)
crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)
return o
- def spider_opened(self, spider):
- self.pagesprev = 0
- self.itemsprev = 0
+ def spider_opened(self, spider: Spider) -> None:
+ self.pagesprev: int = 0
+ self.itemsprev: int = 0
self.task = task.LoopingCall(self.log, spider)
self.task.start(self.interval)
- def log(self, spider):
- items = self.stats.get_value("item_scraped_count", 0)
- pages = self.stats.get_value("response_received_count", 0)
- irate = (items - self.itemsprev) * self.multiplier
- prate = (pages - self.pagesprev) * self.multiplier
- self.pagesprev, self.itemsprev = pages, items
+ def log(self, spider: Spider) -> None:
+ self.calculate_stats()
msg = (
"Crawled %(pages)d pages (at %(pagerate)d pages/min), "
"scraped %(items)d items (at %(itemrate)d items/min)"
)
log_args = {
- "pages": pages,
- "pagerate": prate,
- "items": items,
- "itemrate": irate,
+ "pages": self.pages,
+ "pagerate": self.prate,
+ "items": self.items,
+ "itemrate": self.irate,
}
logger.info(msg, log_args, extra={"spider": spider})
- def spider_closed(self, spider, reason):
+ def calculate_stats(self) -> None:
+ self.items: int = self.stats.get_value("item_scraped_count", 0)
+ self.pages: int = self.stats.get_value("response_received_count", 0)
+ self.irate: float = (self.items - self.itemsprev) * self.multiplier
+ self.prate: float = (self.pages - self.pagesprev) * self.multiplier
+ self.pagesprev, self.itemsprev = self.pages, self.items
+
+ def spider_closed(self, spider: Spider, reason: str) -> None:
if self.task and self.task.running:
self.task.stop()
+
+ rpm_final, ipm_final = self.calculate_final_stats(spider)
+ self.stats.set_value("responses_per_minute", rpm_final)
+ self.stats.set_value("items_per_minute", ipm_final)
+
+ def calculate_final_stats(
+ self, spider: Spider
+ ) -> tuple[None, None] | tuple[float, float]:
+ start_time = self.stats.get_value("start_time")
+ finished_time = self.stats.get_value("finished_time")
+
+ if not start_time or not finished_time:
+ return None, None
+
+ mins_elapsed = (finished_time - start_time).seconds / 60
+
+ items = self.stats.get_value("item_scraped_count", 0)
+ pages = self.stats.get_value("response_received_count", 0)
+
+ return (pages / mins_elapsed), (items / mins_elapsed)
diff --git a/scrapy/extensions/memdebug.py b/scrapy/extensions/memdebug.py
index 03ede0681..3cbbb64e5 100644
--- a/scrapy/extensions/memdebug.py
+++ b/scrapy/extensions/memdebug.py
@@ -4,26 +4,37 @@ MemoryDebugger extension
See documentation in docs/topics/extensions.rst
"""
-import gc
+from __future__ import annotations
-from scrapy import signals
+import gc
+from typing import TYPE_CHECKING
+
+from scrapy import Spider, signals
from scrapy.exceptions import NotConfigured
from scrapy.utils.trackref import live_refs
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.statscollectors import StatsCollector
+
class MemoryDebugger:
- def __init__(self, stats):
- self.stats = stats
+ def __init__(self, stats: StatsCollector):
+ self.stats: StatsCollector = stats
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
if not crawler.settings.getbool("MEMDEBUG_ENABLED"):
raise NotConfigured
+ assert crawler.stats
o = cls(crawler.stats)
crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)
return o
- def spider_closed(self, spider, reason):
+ def spider_closed(self, spider: Spider, reason: str) -> None:
gc.collect()
self.stats.set_value(
"memdebug/gc_garbage_count", len(gc.garbage), spider=spider
diff --git a/scrapy/extensions/memusage.py b/scrapy/extensions/memusage.py
index 221967bda..d7f810107 100644
--- a/scrapy/extensions/memusage.py
+++ b/scrapy/extensions/memusage.py
@@ -3,11 +3,15 @@ MemoryUsage extension
See documentation in docs/topics/extensions.rst
"""
+
+from __future__ import annotations
+
import logging
import socket
import sys
from importlib import import_module
from pprint import pformat
+from typing import TYPE_CHECKING
from twisted.internet import task
@@ -16,11 +20,18 @@ from scrapy.exceptions import NotConfigured
from scrapy.mail import MailSender
from scrapy.utils.engine import get_engine_status
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+
+
logger = logging.getLogger(__name__)
class MemoryUsage:
- def __init__(self, crawler):
+ def __init__(self, crawler: Crawler):
if not crawler.settings.getbool("MEMUSAGE_ENABLED"):
raise NotConfigured
try:
@@ -29,32 +40,33 @@ class MemoryUsage:
except ImportError:
raise NotConfigured
- self.crawler = crawler
- self.warned = False
- self.notify_mails = crawler.settings.getlist("MEMUSAGE_NOTIFY_MAIL")
- self.limit = crawler.settings.getint("MEMUSAGE_LIMIT_MB") * 1024 * 1024
- self.warning = crawler.settings.getint("MEMUSAGE_WARNING_MB") * 1024 * 1024
- self.check_interval = crawler.settings.getfloat(
+ self.crawler: Crawler = crawler
+ self.warned: bool = False
+ self.notify_mails: list[str] = crawler.settings.getlist("MEMUSAGE_NOTIFY_MAIL")
+ self.limit: int = crawler.settings.getint("MEMUSAGE_LIMIT_MB") * 1024 * 1024
+ self.warning: int = crawler.settings.getint("MEMUSAGE_WARNING_MB") * 1024 * 1024
+ self.check_interval: float = crawler.settings.getfloat(
"MEMUSAGE_CHECK_INTERVAL_SECONDS"
)
- self.mail = MailSender.from_settings(crawler.settings)
+ self.mail: MailSender = MailSender.from_crawler(crawler)
crawler.signals.connect(self.engine_started, signal=signals.engine_started)
crawler.signals.connect(self.engine_stopped, signal=signals.engine_stopped)
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler)
- def get_virtual_size(self):
- size = self.resource.getrusage(self.resource.RUSAGE_SELF).ru_maxrss
+ def get_virtual_size(self) -> int:
+ size: int = self.resource.getrusage(self.resource.RUSAGE_SELF).ru_maxrss
if sys.platform != "darwin":
# on macOS ru_maxrss is in bytes, on Linux it is in KB
size *= 1024
return size
- def engine_started(self):
+ def engine_started(self) -> None:
+ assert self.crawler.stats
self.crawler.stats.set_value("memusage/startup", self.get_virtual_size())
- self.tasks = []
+ self.tasks: list[task.LoopingCall] = []
tsk = task.LoopingCall(self.update)
self.tasks.append(tsk)
tsk.start(self.check_interval, now=True)
@@ -67,15 +79,18 @@ class MemoryUsage:
self.tasks.append(tsk)
tsk.start(self.check_interval, now=True)
- def engine_stopped(self):
+ def engine_stopped(self) -> None:
for tsk in self.tasks:
if tsk.running:
tsk.stop()
- def update(self):
+ def update(self) -> None:
+ assert self.crawler.stats
self.crawler.stats.max_value("memusage/max", self.get_virtual_size())
- def _check_limit(self):
+ def _check_limit(self) -> None:
+ assert self.crawler.engine
+ assert self.crawler.stats
peak_mem_usage = self.get_virtual_size()
if peak_mem_usage > self.limit:
self.crawler.stats.set_value("memusage/limit_reached", 1)
@@ -105,9 +120,10 @@ class MemoryUsage:
{"virtualsize": peak_mem_usage / 1024 / 1024},
)
- def _check_warning(self):
+ def _check_warning(self) -> None:
if self.warned: # warn only once
return
+ assert self.crawler.stats
if self.get_virtual_size() > self.warning:
self.crawler.stats.set_value("memusage/warning_reached", 1)
mem = self.warning / 1024 / 1024
@@ -125,12 +141,14 @@ class MemoryUsage:
self.crawler.stats.set_value("memusage/warning_notified", 1)
self.warned = True
- def _send_report(self, rcpts, subject):
+ def _send_report(self, rcpts: list[str], subject: str) -> None:
"""send notification mail with some additional useful info"""
+ assert self.crawler.engine
+ assert self.crawler.stats
stats = self.crawler.stats
- s = f"Memory usage at engine startup : {stats.get_value('memusage/startup')/1024/1024}M\r\n"
- s += f"Maximum memory usage : {stats.get_value('memusage/max')/1024/1024}M\r\n"
- s += f"Current memory usage : {self.get_virtual_size()/1024/1024}M\r\n"
+ s = f"Memory usage at engine startup : {stats.get_value('memusage/startup') / 1024 / 1024}M\r\n"
+ s += f"Maximum memory usage : {stats.get_value('memusage/max') / 1024 / 1024}M\r\n"
+ s += f"Current memory usage : {self.get_virtual_size() / 1024 / 1024}M\r\n"
s += (
"ENGINE STATUS ------------------------------------------------------- \r\n"
diff --git a/scrapy/extensions/periodic_log.py b/scrapy/extensions/periodic_log.py
index 2d557f123..7cf08a1bb 100644
--- a/scrapy/extensions/periodic_log.py
+++ b/scrapy/extensions/periodic_log.py
@@ -1,12 +1,25 @@
+from __future__ import annotations
+
import logging
from datetime import datetime, timezone
+from typing import TYPE_CHECKING, Any
from twisted.internet import task
-from scrapy import signals
+from scrapy import Spider, signals
from scrapy.exceptions import NotConfigured
from scrapy.utils.serialize import ScrapyJSONEncoder
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from json import JSONEncoder
+
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.statscollectors import StatsCollector
+
+
logger = logging.getLogger(__name__)
@@ -15,32 +28,34 @@ class PeriodicLog:
def __init__(
self,
- stats,
- interval=60.0,
- ext_stats={},
- ext_delta={},
- ext_timing_enabled=False,
+ stats: StatsCollector,
+ interval: float = 60.0,
+ ext_stats: dict[str, Any] = {},
+ ext_delta: dict[str, Any] = {},
+ ext_timing_enabled: bool = False,
):
- self.stats = stats
- self.interval = interval
- self.multiplier = 60.0 / self.interval
- self.task = None
- self.encoder = ScrapyJSONEncoder(sort_keys=True, indent=4)
- self.ext_stats_enabled = bool(ext_stats)
- self.ext_stats_include = ext_stats.get("include", [])
- self.ext_stats_exclude = ext_stats.get("exclude", [])
- self.ext_delta_enabled = bool(ext_delta)
- self.ext_delta_include = ext_delta.get("include", [])
- self.ext_delta_exclude = ext_delta.get("exclude", [])
- self.ext_timing_enabled = ext_timing_enabled
+ self.stats: StatsCollector = stats
+ self.interval: float = interval
+ self.multiplier: float = 60.0 / self.interval
+ self.task: task.LoopingCall | None = None
+ self.encoder: JSONEncoder = ScrapyJSONEncoder(sort_keys=True, indent=4)
+ self.ext_stats_enabled: bool = bool(ext_stats)
+ self.ext_stats_include: list[str] = ext_stats.get("include", [])
+ self.ext_stats_exclude: list[str] = ext_stats.get("exclude", [])
+ self.ext_delta_enabled: bool = bool(ext_delta)
+ self.ext_delta_include: list[str] = ext_delta.get("include", [])
+ self.ext_delta_exclude: list[str] = ext_delta.get("exclude", [])
+ self.ext_timing_enabled: bool = ext_timing_enabled
@classmethod
- def from_crawler(cls, crawler):
- interval = crawler.settings.getfloat("LOGSTATS_INTERVAL")
+ def from_crawler(cls, crawler: Crawler) -> Self:
+ interval: float = crawler.settings.getfloat("LOGSTATS_INTERVAL")
if not interval:
raise NotConfigured
try:
- ext_stats = crawler.settings.getdict("PERIODIC_LOG_STATS")
+ ext_stats: dict[str, Any] | None = crawler.settings.getdict(
+ "PERIODIC_LOG_STATS"
+ )
except (TypeError, ValueError):
ext_stats = (
{"enabled": True}
@@ -48,7 +63,9 @@ class PeriodicLog:
else None
)
try:
- ext_delta = crawler.settings.getdict("PERIODIC_LOG_DELTA")
+ ext_delta: dict[str, Any] | None = crawler.settings.getdict(
+ "PERIODIC_LOG_DELTA"
+ )
except (TypeError, ValueError):
ext_delta = (
{"enabled": True}
@@ -56,11 +73,14 @@ class PeriodicLog:
else None
)
- ext_timing_enabled = crawler.settings.getbool(
+ ext_timing_enabled: bool = crawler.settings.getbool(
"PERIODIC_LOG_TIMING_ENABLED", False
)
if not (ext_stats or ext_delta or ext_timing_enabled):
raise NotConfigured
+ assert crawler.stats
+ assert ext_stats is not None
+ assert ext_delta is not None
o = cls(
crawler.stats,
interval,
@@ -72,16 +92,16 @@ class PeriodicLog:
crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)
return o
- def spider_opened(self, spider):
- self.time_prev = datetime.now(tz=timezone.utc)
- self.delta_prev = {}
- self.stats_prev = {}
+ def spider_opened(self, spider: Spider) -> None:
+ self.time_prev: datetime = datetime.now(tz=timezone.utc)
+ self.delta_prev: dict[str, int | float] = {}
+ self.stats_prev: dict[str, int | float] = {}
self.task = task.LoopingCall(self.log)
self.task.start(self.interval)
- def log(self):
- data = {}
+ def log(self) -> None:
+ data: dict[str, Any] = {}
if self.ext_timing_enabled:
data.update(self.log_timing())
if self.ext_delta_enabled:
@@ -90,8 +110,8 @@ class PeriodicLog:
data.update(self.log_crawler_stats())
logger.info(self.encoder.encode(data))
- def log_delta(self):
- num_stats = {
+ def log_delta(self) -> dict[str, Any]:
+ num_stats: dict[str, int | float] = {
k: v
for k, v in self.stats._stats.items()
if isinstance(v, (int, float))
@@ -101,7 +121,7 @@ class PeriodicLog:
self.delta_prev = num_stats
return {"delta": delta}
- def log_timing(self):
+ def log_timing(self) -> dict[str, Any]:
now = datetime.now(tz=timezone.utc)
time = {
"log_interval": self.interval,
@@ -113,7 +133,7 @@ class PeriodicLog:
self.time_prev = now
return {"time": time}
- def log_crawler_stats(self):
+ def log_crawler_stats(self) -> dict[str, Any]:
stats = {
k: v
for k, v in self.stats._stats.items()
@@ -121,7 +141,9 @@ class PeriodicLog:
}
return {"stats": stats}
- def param_allowed(self, stat_name, include, exclude):
+ def param_allowed(
+ self, stat_name: str, include: list[str], exclude: list[str]
+ ) -> bool:
if not include and not exclude:
return True
for p in exclude:
@@ -134,7 +156,7 @@ class PeriodicLog:
return True
return False
- def spider_closed(self, spider, reason):
+ def spider_closed(self, spider: Spider, reason: str) -> None:
self.log()
if self.task and self.task.running:
self.task.stop()
diff --git a/scrapy/extensions/postprocessing.py b/scrapy/extensions/postprocessing.py
index 79e3b1656..b1fa160c8 100644
--- a/scrapy/extensions/postprocessing.py
+++ b/scrapy/extensions/postprocessing.py
@@ -1,11 +1,12 @@
"""
Extension for processing data before they are exported to feeds.
"""
+
from bz2 import BZ2File
from gzip import GzipFile
from io import IOBase
from lzma import LZMAFile
-from typing import Any, BinaryIO, Dict, List
+from typing import IO, Any, BinaryIO, cast
from scrapy.utils.misc import load_object
@@ -23,7 +24,7 @@ class GzipPlugin:
See :py:class:`gzip.GzipFile` for more info about parameters.
"""
- def __init__(self, file: BinaryIO, feed_options: Dict[str, Any]) -> None:
+ def __init__(self, file: BinaryIO, feed_options: dict[str, Any]) -> None:
self.file = file
self.feed_options = feed_options
compress_level = self.feed_options.get("gzip_compresslevel", 9)
@@ -42,7 +43,6 @@ class GzipPlugin:
def close(self) -> None:
self.gzipfile.close()
- self.file.close()
class Bz2Plugin:
@@ -56,7 +56,7 @@ class Bz2Plugin:
See :py:class:`bz2.BZ2File` for more info about parameters.
"""
- def __init__(self, file: BinaryIO, feed_options: Dict[str, Any]) -> None:
+ def __init__(self, file: BinaryIO, feed_options: dict[str, Any]) -> None:
self.file = file
self.feed_options = feed_options
compress_level = self.feed_options.get("bz2_compresslevel", 9)
@@ -69,7 +69,6 @@ class Bz2Plugin:
def close(self) -> None:
self.bz2file.close()
- self.file.close()
class LZMAPlugin:
@@ -89,7 +88,7 @@ class LZMAPlugin:
See :py:class:`lzma.LZMAFile` for more info about parameters.
"""
- def __init__(self, file: BinaryIO, feed_options: Dict[str, Any]) -> None:
+ def __init__(self, file: BinaryIO, feed_options: dict[str, Any]) -> None:
self.file = file
self.feed_options = feed_options
@@ -111,7 +110,6 @@ class LZMAPlugin:
def close(self) -> None:
self.lzmafile.close()
- self.file.close()
# io.IOBase is subclassed here, so that exporters can use the PostProcessingManager
@@ -128,7 +126,7 @@ class PostProcessingManager(IOBase):
"""
def __init__(
- self, plugins: List[Any], file: BinaryIO, feed_options: Dict[str, Any]
+ self, plugins: list[Any], file: IO[bytes], feed_options: dict[str, Any]
) -> None:
self.plugins = self._load_plugins(plugins)
self.file = file
@@ -144,7 +142,7 @@ class PostProcessingManager(IOBase):
:return: returns number of bytes written
:rtype: int
"""
- return self.head_plugin.write(data)
+ return cast(int, self.head_plugin.write(data))
def tell(self) -> int:
return self.file.tell()
@@ -158,9 +156,8 @@ class PostProcessingManager(IOBase):
def writable(self) -> bool:
return True
- def _load_plugins(self, plugins: List[Any]) -> List[Any]:
- plugins = [load_object(plugin) for plugin in plugins]
- return plugins
+ def _load_plugins(self, plugins: list[Any]) -> list[Any]:
+ return [load_object(plugin) for plugin in plugins]
def _get_head_plugin(self) -> Any:
prev = self.file
diff --git a/scrapy/extensions/spiderstate.py b/scrapy/extensions/spiderstate.py
index 929a3be70..7b8756572 100644
--- a/scrapy/extensions/spiderstate.py
+++ b/scrapy/extensions/spiderstate.py
@@ -1,19 +1,28 @@
+from __future__ import annotations
+
import pickle
from pathlib import Path
+from typing import TYPE_CHECKING
-from scrapy import signals
+from scrapy import Spider, signals
from scrapy.exceptions import NotConfigured
from scrapy.utils.job import job_dir
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+
class SpiderState:
"""Store and load spider state during a scraping job"""
- def __init__(self, jobdir=None):
- self.jobdir = jobdir
+ def __init__(self, jobdir: str | None = None):
+ self.jobdir: str | None = jobdir
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
jobdir = job_dir(crawler.settings)
if not jobdir:
raise NotConfigured
@@ -23,18 +32,20 @@ class SpiderState:
crawler.signals.connect(obj.spider_opened, signal=signals.spider_opened)
return obj
- def spider_closed(self, spider):
+ def spider_closed(self, spider: Spider) -> None:
if self.jobdir:
with Path(self.statefn).open("wb") as f:
+ assert hasattr(spider, "state") # set in spider_opened
pickle.dump(spider.state, f, protocol=4)
- def spider_opened(self, spider):
+ def spider_opened(self, spider: Spider) -> None:
if self.jobdir and Path(self.statefn).exists():
with Path(self.statefn).open("rb") as f:
- spider.state = pickle.load(f)
+ spider.state = pickle.load(f) # type: ignore[attr-defined] # noqa: S301
else:
- spider.state = {}
+ spider.state = {} # type: ignore[attr-defined]
@property
def statefn(self) -> str:
+ assert self.jobdir
return str(Path(self.jobdir, "spider.state"))
diff --git a/scrapy/extensions/statsmailer.py b/scrapy/extensions/statsmailer.py
index 58610c25e..221628642 100644
--- a/scrapy/extensions/statsmailer.py
+++ b/scrapy/extensions/statsmailer.py
@@ -4,28 +4,42 @@ StatsMailer extension sends an email when a spider finishes scraping.
Use STATSMAILER_RCPTS setting to enable and give the recipient mail address
"""
-from scrapy import signals
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from scrapy import Spider, signals
from scrapy.exceptions import NotConfigured
from scrapy.mail import MailSender
+if TYPE_CHECKING:
+ from twisted.internet.defer import Deferred
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+ from scrapy.statscollectors import StatsCollector
+
class StatsMailer:
- def __init__(self, stats, recipients, mail):
- self.stats = stats
- self.recipients = recipients
- self.mail = mail
+ def __init__(self, stats: StatsCollector, recipients: list[str], mail: MailSender):
+ self.stats: StatsCollector = stats
+ self.recipients: list[str] = recipients
+ self.mail: MailSender = mail
@classmethod
- def from_crawler(cls, crawler):
- recipients = crawler.settings.getlist("STATSMAILER_RCPTS")
+ def from_crawler(cls, crawler: Crawler) -> Self:
+ recipients: list[str] = crawler.settings.getlist("STATSMAILER_RCPTS")
if not recipients:
raise NotConfigured
- mail = MailSender.from_settings(crawler.settings)
+ mail: MailSender = MailSender.from_crawler(crawler)
+ assert crawler.stats
o = cls(crawler.stats, recipients, mail)
crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)
return o
- def spider_closed(self, spider):
+ def spider_closed(self, spider: Spider) -> Deferred[None] | None:
spider_stats = self.stats.get_stats(spider)
body = "Global stats\n\n"
body += "\n".join(f"{k:<50} : {v}" for k, v in self.stats.get_stats().items())
diff --git a/scrapy/extensions/telnet.py b/scrapy/extensions/telnet.py
index c92b7f5fe..ee28d86ba 100644
--- a/scrapy/extensions/telnet.py
+++ b/scrapy/extensions/telnet.py
@@ -4,23 +4,16 @@ Scrapy Telnet Console extension
See documentation in docs/topics/telnetconsole.rst
"""
+from __future__ import annotations
+
import binascii
import logging
import os
import pprint
-import traceback
+from typing import TYPE_CHECKING, Any
from twisted.internet import protocol
-try:
- from twisted.conch import manhole, telnet
- from twisted.conch.insults import insults
-
- TWISTED_CONCH_AVAILABLE = True
-except (ImportError, SyntaxError):
- _TWISTED_CONCH_TRACEBACK = traceback.format_exc()
- TWISTED_CONCH_AVAILABLE = False
-
from scrapy import signals
from scrapy.exceptions import NotConfigured
from scrapy.utils.decorators import defers
@@ -28,6 +21,16 @@ from scrapy.utils.engine import print_engine_status
from scrapy.utils.reactor import listen_tcp
from scrapy.utils.trackref import print_live_refs
+if TYPE_CHECKING:
+ from twisted.conch import telnet
+ from twisted.internet.tcp import Port
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.crawler import Crawler
+
+
logger = logging.getLogger(__name__)
# signal to update telnet variables
@@ -36,22 +39,18 @@ update_telnet_vars = object()
class TelnetConsole(protocol.ServerFactory):
- def __init__(self, crawler):
+ def __init__(self, crawler: Crawler):
if not crawler.settings.getbool("TELNETCONSOLE_ENABLED"):
raise NotConfigured
- if not TWISTED_CONCH_AVAILABLE:
- raise NotConfigured(
- "TELNETCONSOLE_ENABLED setting is True but required twisted "
- "modules failed to import:\n" + _TWISTED_CONCH_TRACEBACK
- )
- self.crawler = crawler
- self.noisy = False
- self.portrange = [
+
+ self.crawler: Crawler = crawler
+ self.noisy: bool = False
+ self.portrange: list[int] = [
int(x) for x in crawler.settings.getlist("TELNETCONSOLE_PORT")
]
- self.host = crawler.settings["TELNETCONSOLE_HOST"]
- self.username = crawler.settings["TELNETCONSOLE_USERNAME"]
- self.password = crawler.settings["TELNETCONSOLE_PASSWORD"]
+ self.host: str = crawler.settings["TELNETCONSOLE_HOST"]
+ self.username: str = crawler.settings["TELNETCONSOLE_USERNAME"]
+ self.password: str = crawler.settings["TELNETCONSOLE_PASSWORD"]
if not self.password:
self.password = binascii.hexlify(os.urandom(8)).decode("utf8")
@@ -61,11 +60,11 @@ class TelnetConsole(protocol.ServerFactory):
self.crawler.signals.connect(self.stop_listening, signals.engine_stopped)
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler)
- def start_listening(self):
- self.port = listen_tcp(self.portrange, self.host, self)
+ def start_listening(self) -> None:
+ self.port: Port = listen_tcp(self.portrange, self.host, self)
h = self.port.getHost()
logger.info(
"Telnet console listening on %(host)s:%(port)d",
@@ -73,10 +72,14 @@ class TelnetConsole(protocol.ServerFactory):
extra={"crawler": self.crawler},
)
- def stop_listening(self):
+ def stop_listening(self) -> None:
self.port.stopListening()
- def protocol(self):
+ def protocol(self) -> telnet.TelnetTransport:
+ # these import twisted.internet.reactor
+ from twisted.conch import manhole, telnet
+ from twisted.conch.insults import insults
+
class Portal:
"""An implementation of IPortal"""
@@ -95,9 +98,10 @@ class TelnetConsole(protocol.ServerFactory):
return telnet.TelnetTransport(telnet.AuthenticatingTelnetProtocol, Portal())
- def _get_telnet_vars(self):
+ def _get_telnet_vars(self) -> dict[str, Any]:
# Note: if you add entries here also update topics/telnetconsole.rst
- telnet_vars = {
+ assert self.crawler.engine
+ telnet_vars: dict[str, Any] = {
"engine": self.crawler.engine,
"spider": self.crawler.engine.spider,
"slot": self.crawler.engine.slot,
diff --git a/scrapy/extensions/throttle.py b/scrapy/extensions/throttle.py
index 396800775..cdb0671ae 100644
--- a/scrapy/extensions/throttle.py
+++ b/scrapy/extensions/throttle.py
@@ -1,51 +1,74 @@
-import logging
+from __future__ import annotations
-from scrapy import signals
+import logging
+from typing import TYPE_CHECKING
+
+from scrapy import Request, Spider, signals
from scrapy.exceptions import NotConfigured
+if TYPE_CHECKING:
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy.core.downloader import Slot
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+
+
logger = logging.getLogger(__name__)
class AutoThrottle:
- def __init__(self, crawler):
- self.crawler = crawler
+ def __init__(self, crawler: Crawler):
+ self.crawler: Crawler = crawler
if not crawler.settings.getbool("AUTOTHROTTLE_ENABLED"):
raise NotConfigured
- self.debug = crawler.settings.getbool("AUTOTHROTTLE_DEBUG")
- self.target_concurrency = crawler.settings.getfloat(
+ self.debug: bool = crawler.settings.getbool("AUTOTHROTTLE_DEBUG")
+ self.target_concurrency: float = crawler.settings.getfloat(
"AUTOTHROTTLE_TARGET_CONCURRENCY"
)
+ if self.target_concurrency <= 0.0:
+ raise NotConfigured(
+ f"AUTOTHROTTLE_TARGET_CONCURRENCY "
+ f"({self.target_concurrency!r}) must be higher than 0."
+ )
crawler.signals.connect(self._spider_opened, signal=signals.spider_opened)
crawler.signals.connect(
self._response_downloaded, signal=signals.response_downloaded
)
@classmethod
- def from_crawler(cls, crawler):
+ def from_crawler(cls, crawler: Crawler) -> Self:
return cls(crawler)
- def _spider_opened(self, spider):
+ def _spider_opened(self, spider: Spider) -> None:
self.mindelay = self._min_delay(spider)
self.maxdelay = self._max_delay(spider)
- spider.download_delay = self._start_delay(spider)
+ spider.download_delay = self._start_delay(spider) # type: ignore[attr-defined]
- def _min_delay(self, spider):
+ def _min_delay(self, spider: Spider) -> float:
s = self.crawler.settings
return getattr(spider, "download_delay", s.getfloat("DOWNLOAD_DELAY"))
- def _max_delay(self, spider):
+ def _max_delay(self, spider: Spider) -> float:
return self.crawler.settings.getfloat("AUTOTHROTTLE_MAX_DELAY")
- def _start_delay(self, spider):
+ def _start_delay(self, spider: Spider) -> float:
return max(
self.mindelay, self.crawler.settings.getfloat("AUTOTHROTTLE_START_DELAY")
)
- def _response_downloaded(self, response, request, spider):
+ def _response_downloaded(
+ self, response: Response, request: Request, spider: Spider
+ ) -> None:
key, slot = self._get_slot(request, spider)
latency = request.meta.get("download_latency")
- if latency is None or slot is None:
+ if (
+ latency is None
+ or slot is None
+ or request.meta.get("autothrottle_dont_adjust_delay", False) is True
+ ):
return
olddelay = slot.delay
@@ -69,11 +92,16 @@ class AutoThrottle:
extra={"spider": spider},
)
- def _get_slot(self, request, spider):
- key = request.meta.get("download_slot")
+ def _get_slot(
+ self, request: Request, spider: Spider
+ ) -> tuple[str | None, Slot | None]:
+ key: str | None = request.meta.get("download_slot")
+ if key is None:
+ return None, None
+ assert self.crawler.engine
return key, self.crawler.engine.downloader.slots.get(key)
- def _adjust_delay(self, slot, latency, response):
+ def _adjust_delay(self, slot: Slot, latency: float, response: Response) -> None:
"""Define delay adjustment policy"""
# If a server needs `latency` seconds to respond then
diff --git a/scrapy/http/__init__.py b/scrapy/http/__init__.py
index ac3946302..d0b726bad 100644
--- a/scrapy/http/__init__.py
+++ b/scrapy/http/__init__.py
@@ -12,5 +12,6 @@ from scrapy.http.request.json_request import JsonRequest
from scrapy.http.request.rpc import XmlRpcRequest
from scrapy.http.response import Response
from scrapy.http.response.html import HtmlResponse
+from scrapy.http.response.json import JsonResponse
from scrapy.http.response.text import TextResponse
from scrapy.http.response.xml import XmlResponse
diff --git a/scrapy/http/cookies.py b/scrapy/http/cookies.py
index 2595f328e..60322fe6e 100644
--- a/scrapy/http/cookies.py
+++ b/scrapy/http/cookies.py
@@ -1,36 +1,49 @@
+from __future__ import annotations
+
import re
import time
-from http.cookiejar import Cookie
+from http.cookiejar import Cookie, CookiePolicy, DefaultCookiePolicy
from http.cookiejar import CookieJar as _CookieJar
-from http.cookiejar import DefaultCookiePolicy
-from typing import Sequence
+from typing import TYPE_CHECKING, Any, cast
-from scrapy import Request
-from scrapy.http import Response
from scrapy.utils.httpobj import urlparse_cached
from scrapy.utils.python import to_unicode
+if TYPE_CHECKING:
+ from collections.abc import Iterator, Sequence
+
+ # typing.Self requires Python 3.11
+ from typing_extensions import Self
+
+ from scrapy import Request
+ from scrapy.http import Response
+
+
# Defined in the http.cookiejar module, but undocumented:
# https://github.com/python/cpython/blob/v3.9.0/Lib/http/cookiejar.py#L527
IPV4_RE = re.compile(r"\.\d+$", re.ASCII)
class CookieJar:
- def __init__(self, policy=None, check_expired_frequency=10000):
- self.policy = policy or DefaultCookiePolicy()
- self.jar = _CookieJar(self.policy)
- self.jar._cookies_lock = _DummyLock()
- self.check_expired_frequency = check_expired_frequency
- self.processed = 0
+ def __init__(
+ self,
+ policy: CookiePolicy | None = None,
+ check_expired_frequency: int = 10000,
+ ):
+ self.policy: CookiePolicy = policy or DefaultCookiePolicy()
+ self.jar: _CookieJar = _CookieJar(self.policy)
+ self.jar._cookies_lock = _DummyLock() # type: ignore[attr-defined]
+ self.check_expired_frequency: int = check_expired_frequency
+ self.processed: int = 0
- def extract_cookies(self, response, request):
+ def extract_cookies(self, response: Response, request: Request) -> None:
wreq = WrappedRequest(request)
wrsp = WrappedResponse(response)
- return self.jar.extract_cookies(wrsp, wreq)
+ self.jar.extract_cookies(wrsp, wreq) # type: ignore[arg-type]
def add_cookie_header(self, request: Request) -> None:
wreq = WrappedRequest(request)
- self.policy._now = self.jar._now = int(time.time())
+ self.policy._now = self.jar._now = int(time.time()) # type: ignore[attr-defined]
# the cookiejar implementation iterates through all domains
# instead we restrict to potential matches on the domain
@@ -47,10 +60,10 @@ class CookieJar:
cookies = []
for host in hosts:
- if host in self.jar._cookies:
- cookies += self.jar._cookies_for_domain(host, wreq)
+ if host in self.jar._cookies: # type: ignore[attr-defined]
+ cookies += self.jar._cookies_for_domain(host, wreq) # type: ignore[attr-defined]
- attrs = self.jar._cookie_attrs(cookies)
+ attrs = self.jar._cookie_attrs(cookies) # type: ignore[attr-defined]
if attrs:
if not wreq.has_header("Cookie"):
wreq.add_unredirected_header("Cookie", "; ".join(attrs))
@@ -61,37 +74,42 @@ class CookieJar:
self.jar.clear_expired_cookies()
@property
- def _cookies(self):
- return self.jar._cookies
+ def _cookies(self) -> dict[str, dict[str, dict[str, Cookie]]]:
+ return self.jar._cookies # type: ignore[attr-defined,no-any-return]
- def clear_session_cookies(self, *args, **kwargs):
- return self.jar.clear_session_cookies(*args, **kwargs)
+ def clear_session_cookies(self) -> None:
+ return self.jar.clear_session_cookies()
- def clear(self, domain=None, path=None, name=None):
- return self.jar.clear(domain, path, name)
+ def clear(
+ self,
+ domain: str | None = None,
+ path: str | None = None,
+ name: str | None = None,
+ ) -> None:
+ self.jar.clear(domain, path, name)
- def __iter__(self):
+ def __iter__(self) -> Iterator[Cookie]:
return iter(self.jar)
- def __len__(self):
+ def __len__(self) -> int:
return len(self.jar)
- def set_policy(self, pol):
- return self.jar.set_policy(pol)
+ def set_policy(self, pol: CookiePolicy) -> None:
+ self.jar.set_policy(pol)
def make_cookies(self, response: Response, request: Request) -> Sequence[Cookie]:
wreq = WrappedRequest(request)
wrsp = WrappedResponse(response)
- return self.jar.make_cookies(wrsp, wreq)
+ return self.jar.make_cookies(wrsp, wreq) # type: ignore[arg-type]
- def set_cookie(self, cookie):
+ def set_cookie(self, cookie: Cookie) -> None:
self.jar.set_cookie(cookie)
def set_cookie_if_ok(self, cookie: Cookie, request: Request) -> None:
- self.jar.set_cookie_if_ok(cookie, WrappedRequest(request))
+ self.jar.set_cookie_if_ok(cookie, WrappedRequest(request)) # type: ignore[arg-type]
-def potential_domain_matches(domain):
+def potential_domain_matches(domain: str) -> list[str]:
"""Potential domain matches for a cookie
>>> potential_domain_matches('www.example.com')
@@ -111,10 +129,10 @@ def potential_domain_matches(domain):
class _DummyLock:
- def acquire(self):
+ def acquire(self) -> None:
pass
- def release(self):
+ def release(self) -> None:
pass
@@ -124,19 +142,19 @@ class WrappedRequest:
see http://docs.python.org/library/urllib2.html#urllib2.Request
"""
- def __init__(self, request):
+ def __init__(self, request: Request):
self.request = request
- def get_full_url(self):
+ def get_full_url(self) -> str:
return self.request.url
- def get_host(self):
+ def get_host(self) -> str:
return urlparse_cached(self.request).netloc
- def get_type(self):
+ def get_type(self) -> str:
return urlparse_cached(self.request).scheme
- def is_unverifiable(self):
+ def is_unverifiable(self) -> bool:
"""Unverifiable should indicate whether the request is unverifiable, as defined by RFC 2965.
It defaults to False. An unverifiable request is one whose URL the user did not have the
@@ -144,35 +162,36 @@ class WrappedRequest:
HTML document, and the user had no option to approve the automatic
fetching of the image, this should be true.
"""
- return self.request.meta.get("is_unverifiable", False)
+ return cast(bool, self.request.meta.get("is_unverifiable", False))
@property
- def full_url(self):
+ def full_url(self) -> str:
return self.get_full_url()
@property
- def host(self):
+ def host(self) -> str:
return self.get_host()
@property
- def type(self):
+ def type(self) -> str:
return self.get_type()
@property
- def unverifiable(self):
+ def unverifiable(self) -> bool:
return self.is_unverifiable()
@property
- def origin_req_host(self):
- return urlparse_cached(self.request).hostname
+ def origin_req_host(self) -> str:
+ return cast(str, urlparse_cached(self.request).hostname)
- def has_header(self, name):
+ def has_header(self, name: str) -> bool:
return name in self.request.headers
- def get_header(self, name, default=None):
- return to_unicode(self.request.headers.get(name, default), errors="replace")
+ def get_header(self, name: str, default: str | None = None) -> str | None:
+ value = self.request.headers.get(name, default)
+ return to_unicode(value, errors="replace") if value is not None else None
- def header_items(self):
+ def header_items(self) -> list[tuple[str, list[str]]]:
return [
(
to_unicode(k, errors="replace"),
@@ -181,18 +200,18 @@ class WrappedRequest:
for k, v in self.request.headers.items()
]
- def add_unredirected_header(self, name, value):
+ def add_unredirected_header(self, name: str, value: str) -> None:
self.request.headers.appendlist(name, value)
class WrappedResponse:
- def __init__(self, response):
+ def __init__(self, response: Response):
self.response = response
- def info(self):
+ def info(self) -> Self:
return self
- def get_all(self, name, default=None):
+ def get_all(self, name: str, default: Any = None) -> list[str]:
return [
to_unicode(v, errors="replace") for v in self.response.headers.getlist(name)
]
diff --git a/scrapy/http/headers.py b/scrapy/http/headers.py
index 21eb9fb73..29ba9533b 100644
--- a/scrapy/http/headers.py
+++ b/scrapy/http/headers.py
@@ -1,18 +1,7 @@
from __future__ import annotations
from collections.abc import Mapping
-from typing import (
- TYPE_CHECKING,
- Any,
- AnyStr,
- Dict,
- Iterable,
- List,
- Optional,
- Tuple,
- Union,
- cast,
-)
+from typing import TYPE_CHECKING, Any, AnyStr, Union, cast
from w3lib.http import headers_dict_to_raw
@@ -20,6 +9,8 @@ from scrapy.utils.datatypes import CaseInsensitiveDict, CaselessDict
from scrapy.utils.python import to_unicode
if TYPE_CHECKING:
+ from collections.abc import Iterable
+
# typing.Self requires Python 3.11
from typing_extensions import Self
@@ -34,17 +25,17 @@ class Headers(CaselessDict):
def __init__(
self,
- seq: Union[Mapping[AnyStr, Any], Iterable[Tuple[AnyStr, Any]], None] = None,
+ seq: Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None = None,
encoding: str = "utf-8",
):
self.encoding: str = encoding
super().__init__(seq)
def update( # type: ignore[override]
- self, seq: Union[Mapping[AnyStr, Any], Iterable[Tuple[AnyStr, Any]]]
+ self, seq: Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]]
) -> None:
seq = seq.items() if isinstance(seq, Mapping) else seq
- iseq: Dict[bytes, List[bytes]] = {}
+ iseq: dict[bytes, list[bytes]] = {}
for k, v in seq:
iseq.setdefault(self.normkey(k), []).extend(self.normvalue(v))
super().update(iseq)
@@ -53,7 +44,7 @@ class Headers(CaselessDict):
"""Normalize key to bytes"""
return self._tobytes(key.title())
- def normvalue(self, value: Union[_RawValueT, Iterable[_RawValueT]]) -> List[bytes]:
+ def normvalue(self, value: _RawValueT | Iterable[_RawValueT]) -> list[bytes]:
"""Normalize values to bytes"""
_value: Iterable[_RawValueT]
if value is None:
@@ -76,21 +67,21 @@ class Headers(CaselessDict):
return str(x).encode(self.encoding)
raise TypeError(f"Unsupported value type: {type(x)}")
- def __getitem__(self, key: AnyStr) -> Optional[bytes]:
+ def __getitem__(self, key: AnyStr) -> bytes | None:
try:
- return cast(List[bytes], super().__getitem__(key))[-1]
+ return cast(list[bytes], super().__getitem__(key))[-1]
except IndexError:
return None
- def get(self, key: AnyStr, def_val: Any = None) -> Optional[bytes]:
+ def get(self, key: AnyStr, def_val: Any = None) -> bytes | None:
try:
- return cast(List[bytes], super().get(key, def_val))[-1]
+ return cast(list[bytes], super().get(key, def_val))[-1]
except IndexError:
return None
- def getlist(self, key: AnyStr, def_val: Any = None) -> List[bytes]:
+ def getlist(self, key: AnyStr, def_val: Any = None) -> list[bytes]:
try:
- return cast(List[bytes], super().__getitem__(key))
+ return cast(list[bytes], super().__getitem__(key))
except KeyError:
if def_val is not None:
return self.normvalue(def_val)
@@ -109,15 +100,16 @@ class Headers(CaselessDict):
lst.extend(self.normvalue(value))
self[key] = lst
- def items(self) -> Iterable[Tuple[bytes, List[bytes]]]: # type: ignore[override]
+ def items(self) -> Iterable[tuple[bytes, list[bytes]]]: # type: ignore[override]
return ((k, self.getlist(k)) for k in self.keys())
- def values(self) -> List[Optional[bytes]]: # type: ignore[override]
- return [self[k] for k in self.keys()]
+ def values(self) -> list[bytes | None]: # type: ignore[override]
+ return [
+ self[k] for k in self.keys() # pylint: disable=consider-using-dict-items
+ ]
def to_string(self) -> bytes:
- # cast() can be removed if the headers_dict_to_raw() hint is improved
- return cast(bytes, headers_dict_to_raw(self))
+ return headers_dict_to_raw(self)
def to_unicode_dict(self) -> CaseInsensitiveDict:
"""Return headers as a CaseInsensitiveDict with str keys
diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py
index a1c5a5e51..3d6cf4816 100644
--- a/scrapy/http/request/__init__.py
+++ b/scrapy/http/request/__init__.py
@@ -4,33 +4,56 @@ requests in Scrapy.
See documentation in docs/topics/request-response.rst
"""
+
+from __future__ import annotations
+
import inspect
from typing import (
+ TYPE_CHECKING,
Any,
AnyStr,
- Callable,
- Dict,
- Iterable,
- List,
- Mapping,
NoReturn,
- Optional,
- Tuple,
- Type,
+ TypedDict,
TypeVar,
Union,
- cast,
+ overload,
)
from w3lib.url import safe_url_string
-import scrapy
+# a workaround for the docs "more than one target found" problem
+import scrapy # noqa: TC001
from scrapy.http.headers import Headers
from scrapy.utils.curl import curl_to_request_kwargs
from scrapy.utils.python import to_bytes
from scrapy.utils.trackref import object_ref
from scrapy.utils.url import escape_ajax
+if TYPE_CHECKING:
+ from collections.abc import Callable, Iterable, Mapping
+
+ from twisted.python.failure import Failure
+
+ # typing.Concatenate requires Python 3.10
+ # typing.NotRequired and typing.Self require Python 3.11
+ from typing_extensions import Concatenate, NotRequired, Self
+
+ from scrapy.http import Response
+
+ CallbackT = Callable[Concatenate[Response, ...], Any]
+
+
+class VerboseCookie(TypedDict):
+ name: str | bytes
+ value: str | bytes | bool | float | int
+ domain: NotRequired[str | bytes]
+ path: NotRequired[str | bytes]
+ secure: NotRequired[bool]
+
+
+CookiesT = Union[dict[str, str], list[VerboseCookie]]
+
+
RequestTypeVar = TypeVar("RequestTypeVar", bound="Request")
@@ -63,7 +86,7 @@ class Request(object_ref):
executed by the Downloader, thus generating a :class:`Response`.
"""
- attributes: Tuple[str, ...] = (
+ attributes: tuple[str, ...] = (
"url",
"callback",
"method",
@@ -89,18 +112,18 @@ class Request(object_ref):
def __init__(
self,
url: str,
- callback: Optional[Callable] = None,
+ callback: CallbackT | None = None,
method: str = "GET",
- headers: Union[Mapping[AnyStr, Any], Iterable[Tuple[AnyStr, Any]], None] = None,
- body: Optional[Union[bytes, str]] = None,
- cookies: Optional[Union[dict, List[dict]]] = None,
- meta: Optional[Dict[str, Any]] = None,
+ headers: Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None = None,
+ body: bytes | str | None = None,
+ cookies: CookiesT | None = None,
+ meta: dict[str, Any] | None = None,
encoding: str = "utf-8",
priority: int = 0,
dont_filter: bool = False,
- errback: Optional[Callable] = None,
- flags: Optional[List[str]] = None,
- cb_kwargs: Optional[Dict[str, Any]] = None,
+ errback: Callable[[Failure], Any] | None = None,
+ flags: list[str] | None = None,
+ cb_kwargs: dict[str, Any] | None = None,
) -> None:
self._encoding: str = encoding # this one has to be set first
self.method: str = str(method).upper()
@@ -116,27 +139,25 @@ class Request(object_ref):
)
if not (callable(errback) or errback is None):
raise TypeError(f"errback must be a callable, got {type(errback).__name__}")
- self.callback: Optional[Callable] = callback
- self.errback: Optional[Callable] = errback
+ self.callback: CallbackT | None = callback
+ self.errback: Callable[[Failure], Any] | None = errback
- self.cookies: Union[dict, List[dict]] = cookies or {}
+ self.cookies: CookiesT = cookies or {}
self.headers: Headers = Headers(headers or {}, encoding=encoding)
self.dont_filter: bool = dont_filter
- self._meta: Optional[Dict[str, Any]] = dict(meta) if meta else None
- self._cb_kwargs: Optional[Dict[str, Any]] = (
- dict(cb_kwargs) if cb_kwargs else None
- )
- self.flags: List[str] = [] if flags is None else list(flags)
+ self._meta: dict[str, Any] | None = dict(meta) if meta else None
+ self._cb_kwargs: dict[str, Any] | None = dict(cb_kwargs) if cb_kwargs else None
+ self.flags: list[str] = [] if flags is None else list(flags)
@property
- def cb_kwargs(self) -> Dict[str, Any]:
+ def cb_kwargs(self) -> dict[str, Any]:
if self._cb_kwargs is None:
self._cb_kwargs = {}
return self._cb_kwargs
@property
- def meta(self) -> Dict[str, Any]:
+ def meta(self) -> dict[str, Any]:
if self._meta is None:
self._meta = {}
return self._meta
@@ -163,7 +184,7 @@ class Request(object_ref):
def body(self) -> bytes:
return self._body
- def _set_body(self, body: Optional[Union[str, bytes]]) -> None:
+ def _set_body(self, body: str | bytes | None) -> None:
self._body = b"" if body is None else to_bytes(body, self.encoding)
@property
@@ -173,25 +194,36 @@ class Request(object_ref):
def __repr__(self) -> str:
return f"<{self.method} {self.url}>"
- def copy(self) -> "Request":
+ def copy(self) -> Self:
return self.replace()
- def replace(self, *args: Any, **kwargs: Any) -> "Request":
+ @overload
+ def replace(
+ self, *args: Any, cls: type[RequestTypeVar], **kwargs: Any
+ ) -> RequestTypeVar: ...
+
+ @overload
+ def replace(self, *args: Any, cls: None = None, **kwargs: Any) -> Self: ...
+
+ def replace(
+ self, *args: Any, cls: type[Request] | None = None, **kwargs: Any
+ ) -> Request:
"""Create a new Request with the same attributes except for those given new values"""
for x in self.attributes:
kwargs.setdefault(x, getattr(self, x))
- cls = kwargs.pop("cls", self.__class__)
- return cast(Request, cls(*args, **kwargs))
+ if cls is None:
+ cls = self.__class__
+ return cls(*args, **kwargs)
@classmethod
def from_curl(
- cls: Type[RequestTypeVar],
+ cls,
curl_command: str,
ignore_unknown_options: bool = True,
**kwargs: Any,
- ) -> RequestTypeVar:
+ ) -> Self:
"""Create a Request object from a string containing a `cURL
- `_ command. It populates the HTTP method, the
+ `_ command. It populates the HTTP method, the
URL, the headers, the cookies and the body. It accepts the same
arguments as the :class:`Request` class, taking preference and
overriding the values of the same arguments contained in the cURL
@@ -221,7 +253,7 @@ class Request(object_ref):
request_kwargs.update(kwargs)
return cls(**request_kwargs)
- def to_dict(self, *, spider: Optional["scrapy.Spider"] = None) -> Dict[str, Any]:
+ def to_dict(self, *, spider: scrapy.Spider | None = None) -> dict[str, Any]:
"""Return a dictionary containing the Request's data.
Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object.
@@ -231,12 +263,16 @@ class Request(object_ref):
"""
d = {
"url": self.url, # urls are safe (safe_string_url)
- "callback": _find_method(spider, self.callback)
- if callable(self.callback)
- else self.callback,
- "errback": _find_method(spider, self.errback)
- if callable(self.errback)
- else self.errback,
+ "callback": (
+ _find_method(spider, self.callback)
+ if callable(self.callback)
+ else self.callback
+ ),
+ "errback": (
+ _find_method(spider, self.errback)
+ if callable(self.errback)
+ else self.errback
+ ),
"headers": dict(self.headers),
}
for attr in self.attributes:
@@ -246,7 +282,7 @@ class Request(object_ref):
return d
-def _find_method(obj: Any, func: Callable) -> str:
+def _find_method(obj: Any, func: Callable[..., Any]) -> str:
"""Helper function for Request.to_dict"""
# Only instance methods contain ``__func__``
if obj and hasattr(func, "__func__"):
diff --git a/scrapy/http/request/form.py b/scrapy/http/request/form.py
index 0f80a0ab7..de3b24de0 100644
--- a/scrapy/http/request/form.py
+++ b/scrapy/http/request/form.py
@@ -7,32 +7,33 @@ See documentation in docs/topics/request-response.rst
from __future__ import annotations
-from typing import TYPE_CHECKING, Any, Iterable, List, Optional, Tuple, Union, cast
+from collections.abc import Iterable
+from typing import TYPE_CHECKING, Any, Optional, Union, cast
from urllib.parse import urlencode, urljoin, urlsplit, urlunsplit
from lxml.html import (
FormElement,
- HTMLParser,
InputElement,
MultipleSelectOptions,
SelectElement,
TextareaElement,
)
-from parsel.selector import create_root_node
from w3lib.html import strip_html5_whitespace
from scrapy.http.request import Request
-from scrapy.http.response.text import TextResponse
from scrapy.utils.python import is_listlike, to_bytes
-from scrapy.utils.response import get_base_url
if TYPE_CHECKING:
+
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from scrapy.http.response.text import TextResponse
-FormdataKVType = Tuple[str, Union[str, Iterable[str]]]
-FormdataType = Optional[Union[dict, List[FormdataKVType]]]
+
+FormdataVType = Union[str, Iterable[str]]
+FormdataKVType = tuple[str, FormdataVType]
+FormdataType = Optional[Union[dict[str, FormdataVType], list[FormdataKVType]]]
class FormRequest(Request):
@@ -63,14 +64,14 @@ class FormRequest(Request):
def from_response(
cls,
response: TextResponse,
- formname: Optional[str] = None,
- formid: Optional[str] = None,
+ formname: str | None = None,
+ formid: str | None = None,
formnumber: int = 0,
formdata: FormdataType = None,
- clickdata: Optional[dict] = None,
+ clickdata: dict[str, str | int] | None = None,
dont_click: bool = False,
- formxpath: Optional[str] = None,
- formcss: Optional[str] = None,
+ formxpath: str | None = None,
+ formcss: str | None = None,
**kwargs: Any,
) -> Self:
kwargs.setdefault("encoding", response.encoding)
@@ -93,7 +94,7 @@ class FormRequest(Request):
return cls(url=url, method=method, formdata=formdata, **kwargs)
-def _get_form_url(form: FormElement, url: Optional[str]) -> str:
+def _get_form_url(form: FormElement, url: str | None) -> str:
assert form.base_url is not None # typing
if url is None:
action = form.get("action")
@@ -114,13 +115,13 @@ def _urlencode(seq: Iterable[FormdataKVType], enc: str) -> str:
def _get_form(
response: TextResponse,
- formname: Optional[str],
- formid: Optional[str],
+ formname: str | None,
+ formid: str | None,
formnumber: int,
- formxpath: Optional[str],
+ formxpath: str | None,
) -> FormElement:
"""Find the wanted form element within the given response."""
- root = create_root_node(response.text, HTMLParser, base_url=get_base_url(response))
+ root = response.selector.root
forms = root.xpath("//form")
if not forms:
raise ValueError(f"No