mirror of https://github.com/scrapy/scrapy.git
Merge 2.11.2 changes (#6363)
This commit is contained in:
Adrián Chaves
GitHub
parent
b88f22c6c5
commit
d2f1e00a6a
|
|
@ -1,5 +1,5 @@
|
|||
[bumpversion]
|
||||
current_version = 2.11.1
|
||||
current_version = 2.11.2
|
||||
commit = True
|
||||
tag = True
|
||||
tag_name = {new_version}
|
||||
|
|
|
|||
40
docs/faq.rst
40
docs/faq.rst
|
|
@ -138,39 +138,37 @@ See previous question.
|
|||
How can I prevent memory errors due to many allowed domains?
|
||||
------------------------------------------------------------
|
||||
|
||||
If you have a spider with a long list of
|
||||
:attr:`~scrapy.Spider.allowed_domains` (e.g. 50,000+), consider
|
||||
replacing the default
|
||||
:class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` spider middleware
|
||||
with a :ref:`custom spider middleware <custom-spider-middleware>` that requires
|
||||
less memory. For example:
|
||||
If you have a spider with a long list of :attr:`~scrapy.Spider.allowed_domains`
|
||||
(e.g. 50,000+), consider replacing the default
|
||||
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` downloader
|
||||
middleware with a :ref:`custom downloader middleware
|
||||
<topics-downloader-middleware-custom>` that requires less memory. For example:
|
||||
|
||||
- If your domain names are similar enough, use your own regular expression
|
||||
instead joining the strings in
|
||||
:attr:`~scrapy.Spider.allowed_domains` into a complex regular
|
||||
expression.
|
||||
instead joining the strings in :attr:`~scrapy.Spider.allowed_domains` into
|
||||
a complex regular expression.
|
||||
|
||||
- If you can `meet the installation requirements`_, use pyre2_ instead of
|
||||
Python’s re_ to compile your URL-filtering regular expression. See
|
||||
:issue:`1908`.
|
||||
|
||||
See also other suggestions at `StackOverflow`_.
|
||||
See also `other suggestions at StackOverflow
|
||||
<https://stackoverflow.com/q/36440681>`__.
|
||||
|
||||
.. note:: Remember to disable
|
||||
:class:`scrapy.spidermiddlewares.offsite.OffsiteMiddleware` when you enable
|
||||
your custom implementation:
|
||||
:class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` when you
|
||||
enable your custom implementation:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
SPIDER_MIDDLEWARES = {
|
||||
"scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,
|
||||
"myproject.middlewares.CustomOffsiteMiddleware": 500,
|
||||
DOWNLOADER_MIDDLEWARES = {
|
||||
"scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": None,
|
||||
"myproject.middlewares.CustomOffsiteMiddleware": 50,
|
||||
}
|
||||
|
||||
.. _meet the installation requirements: https://github.com/andreasvc/pyre2#installation
|
||||
.. _pyre2: https://github.com/andreasvc/pyre2
|
||||
.. _re: https://docs.python.org/library/re.html
|
||||
.. _StackOverflow: https://stackoverflow.com/q/36440681/939364
|
||||
|
||||
Can I use Basic HTTP Authentication in my spiders?
|
||||
--------------------------------------------------
|
||||
|
|
@ -206,12 +204,10 @@ I get "Filtered offsite request" messages. How can I fix them?
|
|||
Those messages (logged with ``DEBUG`` level) don't necessarily mean there is a
|
||||
problem, so you may not need to fix them.
|
||||
|
||||
Those messages are thrown by the Offsite Spider Middleware, which is a spider
|
||||
middleware (enabled by default) whose purpose is to filter out requests to
|
||||
domains outside the ones covered by the spider.
|
||||
|
||||
For more info see:
|
||||
:class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware`.
|
||||
Those messages are thrown by
|
||||
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, which is a
|
||||
downloader middleware (enabled by default) whose purpose is to filter out
|
||||
requests to domains outside the ones covered by the spider.
|
||||
|
||||
What is the recommended way to deploy a Scrapy crawler in production?
|
||||
---------------------------------------------------------------------
|
||||
|
|
|
|||
116
docs/news.rst
116
docs/news.rst
|
|
@ -3,7 +3,6 @@
|
|||
Release notes
|
||||
=============
|
||||
|
||||
|
||||
.. _release-VERSION:
|
||||
|
||||
Scrapy VERSION (YYYY-MM-DD)
|
||||
|
|
@ -12,11 +11,122 @@ Scrapy VERSION (YYYY-MM-DD)
|
|||
Deprecations
|
||||
~~~~~~~~~~~~
|
||||
|
||||
- :func:`scrapy.core.downloader.Downloader._get_slot_key` is now deprecated.
|
||||
Consider using its corresponding public method get_slot_key() instead.
|
||||
- :meth:`scrapy.core.downloader.Downloader._get_slot_key` is deprecated, use
|
||||
:meth:`scrapy.core.downloader.Downloader.get_slot_key` instead.
|
||||
(:issue:`6340`)
|
||||
|
||||
|
||||
.. _release-2.11.2:
|
||||
|
||||
Scrapy 2.11.2 (2024-05-14)
|
||||
--------------------------
|
||||
|
||||
Security bug fixes
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Redirects to non-HTTP protocols are no longer followed. Please, see the
|
||||
`23j4-mw76-5v7h security advisory`_ for more information. (:issue:`457`)
|
||||
|
||||
.. _23j4-mw76-5v7h security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h
|
||||
|
||||
- The ``Authorization`` header is now dropped on redirects to a different
|
||||
scheme (``http://`` or ``https://``) or port, even if the domain is the
|
||||
same. Please, see the `4qqq-9vqf-3h3f security advisory`_ for more
|
||||
information.
|
||||
|
||||
.. _4qqq-9vqf-3h3f security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-4qqq-9vqf-3h3f
|
||||
|
||||
- When using system proxy settings that are different for ``http://`` and
|
||||
``https://``, redirects to a different URL scheme will now also trigger the
|
||||
corresponding change in proxy settings for the redirected request. Please,
|
||||
see the `jm3v-qxmh-hxwv security advisory`_ for more information.
|
||||
(:issue:`767`)
|
||||
|
||||
.. _jm3v-qxmh-hxwv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv
|
||||
|
||||
- :attr:`Spider.allowed_domains <scrapy.Spider.allowed_domains>` is now
|
||||
enforced for all requests, and not only requests from spider callbacks.
|
||||
(:issue:`1042`, :issue:`2241`, :issue:`6358`)
|
||||
|
||||
- :func:`~scrapy.utils.iterators.xmliter_lxml` no longer resolves XML
|
||||
entities. (:issue:`6265`)
|
||||
|
||||
- defusedxml_ is now used to make
|
||||
:class:`scrapy.http.request.rpc.XmlRpcRequest` more secure.
|
||||
(:issue:`6250`, :issue:`6251`)
|
||||
|
||||
.. _defusedxml: https://github.com/tiran/defusedxml
|
||||
|
||||
Bug fixes
|
||||
~~~~~~~~~
|
||||
|
||||
- Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in
|
||||
favor of brotli_. (:issue:`6261`)
|
||||
|
||||
.. _brotli: https://github.com/google/brotli
|
||||
|
||||
.. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli
|
||||
instead if you can.
|
||||
|
||||
- Make :setting:`METAREFRESH_IGNORE_TAGS` ``["noscript"]`` by default. This
|
||||
prevents
|
||||
:class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` from
|
||||
following redirects that would not be followed by web browsers with
|
||||
JavaScript enabled. (:issue:`6342`, :issue:`6347`)
|
||||
|
||||
- During :ref:`feed export <topics-feed-exports>`, do not close the
|
||||
underlying file from :ref:`built-in post-processing plugins
|
||||
<builtin-plugins>`.
|
||||
(:issue:`5932`, :issue:`6178`, :issue:`6239`)
|
||||
|
||||
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
|
||||
now properly applies the ``unique`` and ``canonicalize`` parameters.
|
||||
(:issue:`3273`, :issue:`6221`)
|
||||
|
||||
- Do not initialize the scheduler disk queue if :setting:`JOBDIR` is an empty
|
||||
string. (:issue:`6121`, :issue:`6124`)
|
||||
|
||||
- Fix :attr:`Spider.logger <scrapy.Spider.logger>` not logging custom extra
|
||||
information. (:issue:`6323`, :issue:`6324`)
|
||||
|
||||
- ``robots.txt`` files with a non-UTF-8 encoding no longer prevent parsing
|
||||
the UTF-8-compatible (e.g. ASCII) parts of the document.
|
||||
(:issue:`6292`, :issue:`6298`)
|
||||
|
||||
- :meth:`scrapy.http.cookies.WrappedRequest.get_header` no longer raises an
|
||||
exception if ``default`` is ``None``.
|
||||
(:issue:`6308`, :issue:`6310`)
|
||||
|
||||
- :class:`~scrapy.selector.Selector` now uses
|
||||
:func:`scrapy.utils.response.get_base_url` to determine the base URL of a
|
||||
given :class:`~scrapy.http.Response`. (:issue:`6265`)
|
||||
|
||||
- The :meth:`media_to_download` method of :ref:`media pipelines
|
||||
<topics-media-pipeline>` now logs exceptions before stripping them.
|
||||
(:issue:`5067`, :issue:`5068`)
|
||||
|
||||
- When passing a callback to the :command:`parse` command, build the callback
|
||||
callable with the right signature.
|
||||
(:issue:`6182`)
|
||||
|
||||
Documentation
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
- Add a FAQ entry about :ref:`creating blank requests <faq-blank-request>`.
|
||||
(:issue:`6203`, :issue:`6208`)
|
||||
|
||||
- Document that :attr:`scrapy.selector.Selector.type` can be ``"json"``.
|
||||
(:issue:`6328`, :issue:`6334`)
|
||||
|
||||
Quality assurance
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Make builds reproducible. (:issue:`5019`, :issue:`6322`)
|
||||
|
||||
- Packaging and test fixes.
|
||||
(:issue:`6286`, :issue:`6290`, :issue:`6312`, :issue:`6316`, :issue:`6344`)
|
||||
|
||||
|
||||
.. _release-2.11.1:
|
||||
|
||||
Scrapy 2.11.1 (2024-02-14)
|
||||
|
|
|
|||
|
|
@ -24,7 +24,8 @@ You should see an output like this::
|
|||
'scrapy.extensions.telnet.TelnetConsole',
|
||||
'scrapy.extensions.corestats.CoreStats']
|
||||
2016-12-16 21:18:49 [scrapy.middleware] INFO: Enabled downloader middlewares:
|
||||
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
|
||||
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
|
||||
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
|
||||
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
|
||||
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
|
||||
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
|
||||
|
|
@ -37,7 +38,6 @@ You should see an output like this::
|
|||
'scrapy.downloadermiddlewares.stats.DownloaderStats']
|
||||
2016-12-16 21:18:49 [scrapy.middleware] INFO: Enabled spider middlewares:
|
||||
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
|
||||
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
|
||||
'scrapy.spidermiddlewares.referer.RefererMiddleware',
|
||||
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
|
||||
'scrapy.spidermiddlewares.depth.DepthMiddleware']
|
||||
|
|
|
|||
|
|
@ -763,6 +763,44 @@ HttpProxyMiddleware
|
|||
Keep in mind this value will take precedence over ``http_proxy``/``https_proxy``
|
||||
environment variables, and it will also ignore ``no_proxy`` environment variable.
|
||||
|
||||
OffsiteMiddleware
|
||||
-----------------
|
||||
|
||||
.. module:: scrapy.downloadermiddlewares.offsite
|
||||
:synopsis: Offsite Middleware
|
||||
|
||||
.. class:: OffsiteMiddleware
|
||||
|
||||
.. versionadded:: 2.11.2
|
||||
|
||||
Filters out Requests for URLs outside the domains covered by the spider.
|
||||
|
||||
This middleware filters out every request whose host names aren't in the
|
||||
spider's :attr:`~scrapy.Spider.allowed_domains` attribute.
|
||||
All subdomains of any domain in the list are also allowed.
|
||||
E.g. the rule ``www.example.org`` will also allow ``bob.www.example.org``
|
||||
but not ``www2.example.com`` nor ``example.com``.
|
||||
|
||||
When your spider returns a request for a domain not belonging to those
|
||||
covered by the spider, this middleware will log a debug message similar to
|
||||
this one::
|
||||
|
||||
DEBUG: Filtered offsite request to 'offsite.example': <GET http://offsite.example/some/page.html>
|
||||
|
||||
To avoid filling the log with too much noise, it will only print one of
|
||||
these messages for each new domain filtered. So, for example, if another
|
||||
request for ``offsite.example`` is filtered, no log message will be
|
||||
printed. But if a request for ``other.example`` is filtered, a message
|
||||
will be printed (but only for the first request filtered).
|
||||
|
||||
If the spider doesn't define an
|
||||
:attr:`~scrapy.Spider.allowed_domains` attribute, or the
|
||||
attribute is empty, the offsite middleware will allow all requests.
|
||||
|
||||
If the request has the :attr:`~scrapy.Request.dont_filter` attribute
|
||||
set, the offsite middleware will allow the request even if its domain is not
|
||||
listed in allowed domains.
|
||||
|
||||
RedirectMiddleware
|
||||
------------------
|
||||
|
||||
|
|
@ -882,7 +920,11 @@ Meta tags within these tags are ignored.
|
|||
|
||||
.. versionchanged:: 2.0
|
||||
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
|
||||
``['script', 'noscript']`` to ``[]``.
|
||||
``["script", "noscript"]`` to ``[]``.
|
||||
|
||||
.. versionchanged:: 2.11.2
|
||||
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
|
||||
``[]`` to ``["noscript"]``.
|
||||
|
||||
.. versionchanged:: VERSION
|
||||
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
|
||||
|
|
|
|||
|
|
@ -674,6 +674,7 @@ Default:
|
|||
.. code-block:: python
|
||||
|
||||
{
|
||||
"scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": 50,
|
||||
"scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,
|
||||
"scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,
|
||||
"scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,
|
||||
|
|
@ -1613,7 +1614,6 @@ Default:
|
|||
|
||||
{
|
||||
"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
|
||||
"scrapy.spidermiddlewares.offsite.OffsiteMiddleware": 500,
|
||||
"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
|
||||
"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
|
||||
"scrapy.spidermiddlewares.depth.DepthMiddleware": 900,
|
||||
|
|
|
|||
|
|
@ -343,11 +343,18 @@ request_scheduled
|
|||
.. signal:: request_scheduled
|
||||
.. function:: request_scheduled(request, spider)
|
||||
|
||||
Sent when the engine schedules a :class:`~scrapy.Request`, to be
|
||||
downloaded later.
|
||||
Sent when the engine is asked to schedule a :class:`~scrapy.Request`, to be
|
||||
downloaded later, before the request reaches the :ref:`scheduler
|
||||
<topics-scheduler>`.
|
||||
|
||||
Raise :exc:`~scrapy.exceptions.IgnoreRequest` to drop a request before it
|
||||
reaches the scheduler.
|
||||
|
||||
This signal does not support returning deferreds from its handlers.
|
||||
|
||||
.. versionadded:: 2.11.2
|
||||
Allow dropping requests with :exc:`~scrapy.exceptions.IgnoreRequest`.
|
||||
|
||||
:param request: the request that reached the scheduler
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
|
|
|
|||
|
|
@ -51,8 +51,8 @@ value. For example, if you want to disable the off-site middleware:
|
|||
.. code-block:: python
|
||||
|
||||
SPIDER_MIDDLEWARES = {
|
||||
"myproject.middlewares.CustomSpiderMiddleware": 543,
|
||||
"scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,
|
||||
"scrapy.spidermiddlewares.referer.RefererMiddleware": None,
|
||||
"myproject.middlewares.CustomRefererSpiderMiddleware": 700,
|
||||
}
|
||||
|
||||
Finally, keep in mind that some middlewares may need to be enabled through a
|
||||
|
|
@ -313,42 +313,6 @@ Default: ``False``
|
|||
|
||||
Pass all responses, regardless of its status code.
|
||||
|
||||
OffsiteMiddleware
|
||||
-----------------
|
||||
|
||||
.. module:: scrapy.spidermiddlewares.offsite
|
||||
:synopsis: Offsite Spider Middleware
|
||||
|
||||
.. class:: OffsiteMiddleware
|
||||
|
||||
Filters out Requests for URLs outside the domains covered by the spider.
|
||||
|
||||
This middleware filters out every request whose host names aren't in the
|
||||
spider's :attr:`~scrapy.Spider.allowed_domains` attribute.
|
||||
All subdomains of any domain in the list are also allowed.
|
||||
E.g. the rule ``www.example.org`` will also allow ``bob.www.example.org``
|
||||
but not ``www2.example.com`` nor ``example.com``.
|
||||
|
||||
When your spider returns a request for a domain not belonging to those
|
||||
covered by the spider, this middleware will log a debug message similar to
|
||||
this one::
|
||||
|
||||
DEBUG: Filtered offsite request to 'www.othersite.com': <GET http://www.othersite.com/some/page.html>
|
||||
|
||||
To avoid filling the log with too much noise, it will only print one of
|
||||
these messages for each new domain filtered. So, for example, if another
|
||||
request for ``www.othersite.com`` is filtered, no log message will be
|
||||
printed. But if a request for ``someothersite.com`` is filtered, a message
|
||||
will be printed (but only for the first request filtered).
|
||||
|
||||
If the spider doesn't define an
|
||||
:attr:`~scrapy.Spider.allowed_domains` attribute, or the
|
||||
attribute is empty, the offsite middleware will allow all requests.
|
||||
|
||||
If the request has the :attr:`~scrapy.Request.dont_filter` attribute
|
||||
set, the offsite middleware will allow the request even if its domain is not
|
||||
listed in allowed domains.
|
||||
|
||||
|
||||
RefererMiddleware
|
||||
-----------------
|
||||
|
|
|
|||
|
|
@ -75,7 +75,8 @@ scrapy.Spider
|
|||
An optional list of strings containing domains that this spider is
|
||||
allowed to crawl. Requests for URLs not belonging to the domain names
|
||||
specified in this list (or their subdomains) won't be followed if
|
||||
:class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled.
|
||||
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is
|
||||
enabled.
|
||||
|
||||
Let's say your target url is ``https://www.example.com/1.html``,
|
||||
then add ``'example.com'`` to the list.
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
2.11.1
|
||||
2.11.2
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ from twisted.python.failure import Failure
|
|||
from scrapy import signals
|
||||
from scrapy.core.downloader import Downloader
|
||||
from scrapy.core.scraper import Scraper
|
||||
from scrapy.exceptions import CloseSpider, DontCloseSpider
|
||||
from scrapy.exceptions import CloseSpider, DontCloseSpider, IgnoreRequest
|
||||
from scrapy.http import Request, Response
|
||||
from scrapy.logformatter import LogFormatter
|
||||
from scrapy.settings import BaseSettings, Settings
|
||||
|
|
@ -36,6 +36,7 @@ from scrapy.signalmanager import SignalManager
|
|||
from scrapy.spiders import Spider
|
||||
from scrapy.utils.log import failure_to_exc_info, logformatter_adapter
|
||||
from scrapy.utils.misc import build_from_crawler, load_object
|
||||
from scrapy.utils.python import global_object_name
|
||||
from scrapy.utils.reactor import CallLaterOnce
|
||||
|
||||
if TYPE_CHECKING:
|
||||
|
|
@ -292,9 +293,19 @@ class ExecutionEngine:
|
|||
self.slot.nextcall.schedule() # type: ignore[union-attr]
|
||||
|
||||
def _schedule_request(self, request: Request, spider: Spider) -> None:
|
||||
self.signals.send_catch_log(
|
||||
signals.request_scheduled, request=request, spider=spider
|
||||
request_scheduled_result = self.signals.send_catch_log(
|
||||
signals.request_scheduled,
|
||||
request=request,
|
||||
spider=spider,
|
||||
dont_log=IgnoreRequest,
|
||||
)
|
||||
for handler, result in request_scheduled_result:
|
||||
if isinstance(result, Failure) and isinstance(result.value, IgnoreRequest):
|
||||
logger.debug(
|
||||
f"Signal handler {global_object_name(handler)} dropped "
|
||||
f"request {request} before it reached the scheduler."
|
||||
)
|
||||
return
|
||||
if not self.slot.scheduler.enqueue_request(request): # type: ignore[union-attr]
|
||||
self.signals.send_catch_log(
|
||||
signals.request_dropped, request=request, spider=spider
|
||||
|
|
|
|||
|
|
@ -60,26 +60,33 @@ class HttpProxyMiddleware:
|
|||
def process_request(
|
||||
self, request: Request, spider: Spider
|
||||
) -> Union[Request, Response, None]:
|
||||
creds, proxy_url = None, None
|
||||
creds, proxy_url, scheme = None, None, None
|
||||
if "proxy" in request.meta:
|
||||
if request.meta["proxy"] is not None:
|
||||
creds, proxy_url = self._get_proxy(request.meta["proxy"], "")
|
||||
elif self.proxies:
|
||||
parsed = urlparse_cached(request)
|
||||
scheme = parsed.scheme
|
||||
_scheme = parsed.scheme
|
||||
if (
|
||||
# 'no_proxy' is only supported by http schemes
|
||||
scheme not in ("http", "https")
|
||||
_scheme not in ("http", "https")
|
||||
or (parsed.hostname and not proxy_bypass(parsed.hostname))
|
||||
) and scheme in self.proxies:
|
||||
) and _scheme in self.proxies:
|
||||
scheme = _scheme
|
||||
creds, proxy_url = self.proxies[scheme]
|
||||
|
||||
self._set_proxy_and_creds(request, proxy_url, creds)
|
||||
self._set_proxy_and_creds(request, proxy_url, creds, scheme)
|
||||
return None
|
||||
|
||||
def _set_proxy_and_creds(
|
||||
self, request: Request, proxy_url: Optional[str], creds: Optional[bytes]
|
||||
self,
|
||||
request: Request,
|
||||
proxy_url: Optional[str],
|
||||
creds: Optional[bytes],
|
||||
scheme: Optional[str],
|
||||
) -> None:
|
||||
if scheme:
|
||||
request.meta["_scheme_proxy"] = True
|
||||
if proxy_url:
|
||||
request.meta["proxy"] = proxy_url
|
||||
elif request.meta.get("proxy") is not None:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,77 @@
|
|||
import logging
|
||||
import re
|
||||
import warnings
|
||||
|
||||
from scrapy import signals
|
||||
from scrapy.exceptions import IgnoreRequest
|
||||
from scrapy.utils.httpobj import urlparse_cached
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class OffsiteMiddleware:
|
||||
@classmethod
|
||||
def from_crawler(cls, crawler):
|
||||
o = cls(crawler.stats)
|
||||
crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
|
||||
crawler.signals.connect(o.request_scheduled, signal=signals.request_scheduled)
|
||||
return o
|
||||
|
||||
def __init__(self, stats):
|
||||
self.stats = stats
|
||||
self.domains_seen = set()
|
||||
|
||||
def spider_opened(self, spider):
|
||||
self.host_regex = self.get_host_regex(spider)
|
||||
|
||||
def request_scheduled(self, request, spider):
|
||||
self.process_request(request, spider)
|
||||
|
||||
def process_request(self, request, spider):
|
||||
if request.dont_filter or self.should_follow(request, spider):
|
||||
return None
|
||||
domain = urlparse_cached(request).hostname
|
||||
if domain and domain not in self.domains_seen:
|
||||
self.domains_seen.add(domain)
|
||||
logger.debug(
|
||||
"Filtered offsite request to %(domain)r: %(request)s",
|
||||
{"domain": domain, "request": request},
|
||||
extra={"spider": spider},
|
||||
)
|
||||
self.stats.inc_value("offsite/domains", spider=spider)
|
||||
self.stats.inc_value("offsite/filtered", spider=spider)
|
||||
raise IgnoreRequest
|
||||
|
||||
def should_follow(self, request, spider):
|
||||
regex = self.host_regex
|
||||
# hostname can be None for wrong urls (like javascript links)
|
||||
host = urlparse_cached(request).hostname or ""
|
||||
return bool(regex.search(host))
|
||||
|
||||
def get_host_regex(self, spider):
|
||||
"""Override this method to implement a different offsite policy"""
|
||||
allowed_domains = getattr(spider, "allowed_domains", None)
|
||||
if not allowed_domains:
|
||||
return re.compile("") # allow all by default
|
||||
url_pattern = re.compile(r"^https?://.*$")
|
||||
port_pattern = re.compile(r":\d+$")
|
||||
domains = []
|
||||
for domain in allowed_domains:
|
||||
if domain is None:
|
||||
continue
|
||||
if url_pattern.match(domain):
|
||||
message = (
|
||||
"allowed_domains accepts only domains, not URLs. "
|
||||
f"Ignoring URL entry {domain} in allowed_domains."
|
||||
)
|
||||
warnings.warn(message)
|
||||
elif port_pattern.search(domain):
|
||||
message = (
|
||||
"allowed_domains accepts only domains without ports. "
|
||||
f"Ignoring entry {domain} in allowed_domains."
|
||||
)
|
||||
warnings.warn(message)
|
||||
else:
|
||||
domains.append(re.escape(domain))
|
||||
regex = rf'^(.*\.)?({"|".join(domains)})$'
|
||||
return re.compile(regex)
|
||||
|
|
@ -29,17 +29,49 @@ def _build_redirect_request(
|
|||
**kwargs,
|
||||
cookies=None,
|
||||
)
|
||||
if "_scheme_proxy" in redirect_request.meta:
|
||||
source_request_scheme = urlparse_cached(source_request).scheme
|
||||
redirect_request_scheme = urlparse_cached(redirect_request).scheme
|
||||
if source_request_scheme != redirect_request_scheme:
|
||||
redirect_request.meta.pop("_scheme_proxy")
|
||||
redirect_request.meta.pop("proxy", None)
|
||||
redirect_request.meta.pop("_auth_proxy", None)
|
||||
redirect_request.headers.pop(b"Proxy-Authorization", None)
|
||||
has_cookie_header = "Cookie" in redirect_request.headers
|
||||
has_authorization_header = "Authorization" in redirect_request.headers
|
||||
if has_cookie_header or has_authorization_header:
|
||||
source_request_netloc = urlparse_cached(source_request).netloc
|
||||
redirect_request_netloc = urlparse_cached(redirect_request).netloc
|
||||
if source_request_netloc != redirect_request_netloc:
|
||||
if has_cookie_header:
|
||||
del redirect_request.headers["Cookie"]
|
||||
# https://fetch.spec.whatwg.org/#ref-for-cors-non-wildcard-request-header-name
|
||||
if has_authorization_header:
|
||||
del redirect_request.headers["Authorization"]
|
||||
default_ports = {"http": 80, "https": 443}
|
||||
|
||||
parsed_source_request = urlparse_cached(source_request)
|
||||
source_scheme, source_host, source_port = (
|
||||
parsed_source_request.scheme,
|
||||
parsed_source_request.hostname,
|
||||
parsed_source_request.port
|
||||
or default_ports.get(parsed_source_request.scheme),
|
||||
)
|
||||
|
||||
parsed_redirect_request = urlparse_cached(redirect_request)
|
||||
redirect_scheme, redirect_host, redirect_port = (
|
||||
parsed_redirect_request.scheme,
|
||||
parsed_redirect_request.hostname,
|
||||
parsed_redirect_request.port
|
||||
or default_ports.get(parsed_redirect_request.scheme),
|
||||
)
|
||||
|
||||
if has_cookie_header and (
|
||||
redirect_scheme not in {source_scheme, "https"}
|
||||
or source_host != redirect_host
|
||||
):
|
||||
del redirect_request.headers["Cookie"]
|
||||
|
||||
# https://fetch.spec.whatwg.org/#ref-for-cors-non-wildcard-request-header-name
|
||||
if has_authorization_header and (
|
||||
source_scheme != redirect_scheme
|
||||
or source_host != redirect_host
|
||||
or source_port != redirect_port
|
||||
):
|
||||
del redirect_request.headers["Authorization"]
|
||||
|
||||
return redirect_request
|
||||
|
||||
|
||||
|
|
@ -129,9 +161,11 @@ class RedirectMiddleware(BaseRedirectMiddleware):
|
|||
location = request_scheme + "://" + location.lstrip("/")
|
||||
|
||||
redirected_url = urljoin(request.url, location)
|
||||
redirected = _build_redirect_request(request, url=redirected_url)
|
||||
if urlparse_cached(redirected).scheme not in {"http", "https"}:
|
||||
return response
|
||||
|
||||
if response.status in (301, 307, 308) or request.method == "HEAD":
|
||||
redirected = _build_redirect_request(request, url=redirected_url)
|
||||
return self._redirect(redirected, request, spider, response.status)
|
||||
|
||||
redirected = self._redirect_request_using_get(request, redirected_url)
|
||||
|
|
@ -153,12 +187,16 @@ class MetaRefreshMiddleware(BaseRedirectMiddleware):
|
|||
request.meta.get("dont_redirect", False)
|
||||
or request.method == "HEAD"
|
||||
or not isinstance(response, HtmlResponse)
|
||||
or urlparse_cached(request).scheme not in {"http", "https"}
|
||||
):
|
||||
return response
|
||||
|
||||
interval, url = get_meta_refresh(response, ignore_tags=self._ignore_tags)
|
||||
if url and cast(float, interval) < self._maxdelay:
|
||||
redirected = self._redirect_request_using_get(request, url)
|
||||
if not url:
|
||||
return response
|
||||
redirected = self._redirect_request_using_get(request, url)
|
||||
if urlparse_cached(redirected).scheme not in {"http", "https"}:
|
||||
return response
|
||||
if cast(float, interval) < self._maxdelay:
|
||||
return self._redirect(redirected, request, spider, "meta refresh")
|
||||
|
||||
return response
|
||||
|
|
|
|||
|
|
@ -101,6 +101,7 @@ DOWNLOADER_MIDDLEWARES = {}
|
|||
|
||||
DOWNLOADER_MIDDLEWARES_BASE = {
|
||||
# Engine side
|
||||
"scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": 50,
|
||||
"scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,
|
||||
"scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,
|
||||
"scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,
|
||||
|
|
@ -301,7 +302,6 @@ SPIDER_MIDDLEWARES = {}
|
|||
SPIDER_MIDDLEWARES_BASE = {
|
||||
# Engine side
|
||||
"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
|
||||
"scrapy.spidermiddlewares.offsite.OffsiteMiddleware": 500,
|
||||
"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
|
||||
"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
|
||||
"scrapy.spidermiddlewares.depth.DepthMiddleware": 900,
|
||||
|
|
|
|||
|
|
@ -13,15 +13,21 @@ from typing import TYPE_CHECKING, Any, AsyncIterable, Iterable, Set
|
|||
|
||||
from scrapy import Spider, signals
|
||||
from scrapy.crawler import Crawler
|
||||
from scrapy.exceptions import ScrapyDeprecationWarning
|
||||
from scrapy.http import Request, Response
|
||||
from scrapy.statscollectors import StatsCollector
|
||||
from scrapy.utils.httpobj import urlparse_cached
|
||||
|
||||
warnings.warn(
|
||||
"The scrapy.spidermiddlewares.offsite module is deprecated, use "
|
||||
"scrapy.downloadermiddlewares.offsite instead.",
|
||||
ScrapyDeprecationWarning,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# typing.Self requires Python 3.11
|
||||
from typing_extensions import Self
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -331,7 +331,7 @@ def global_object_name(obj: Any) -> str:
|
|||
>>> global_object_name(Request)
|
||||
'scrapy.http.request.Request'
|
||||
"""
|
||||
return f"{obj.__module__}.{obj.__name__}"
|
||||
return f"{obj.__module__}.{obj.__qualname__}"
|
||||
|
||||
|
||||
if hasattr(sys, "pypy_version_info"):
|
||||
|
|
|
|||
|
|
@ -22,13 +22,11 @@ class ManagerTestCase(TestCase):
|
|||
self.crawler = get_crawler(Spider, self.settings_dict)
|
||||
self.spider = self.crawler._create_spider("foo")
|
||||
self.mwman = DownloaderMiddlewareManager.from_crawler(self.crawler)
|
||||
# some mw depends on stats collector
|
||||
self.crawler.stats.open_spider(self.spider)
|
||||
return self.mwman.open_spider(self.spider)
|
||||
self.crawler.engine = self.crawler._create_engine()
|
||||
return self.crawler.engine.open_spider(self.spider, start_requests=())
|
||||
|
||||
def tearDown(self):
|
||||
self.crawler.stats.close_spider(self.spider, "")
|
||||
return self.mwman.close_spider(self.spider)
|
||||
return self.crawler.engine.close_spider(self.spider)
|
||||
|
||||
def _download(self, request, response=None):
|
||||
"""Executes downloader mw manager's download method and returns
|
||||
|
|
|
|||
|
|
@ -0,0 +1,184 @@
|
|||
import pytest
|
||||
|
||||
from scrapy import Request, Spider
|
||||
from scrapy.downloadermiddlewares.offsite import OffsiteMiddleware
|
||||
from scrapy.exceptions import IgnoreRequest
|
||||
from scrapy.utils.test import get_crawler
|
||||
|
||||
UNSET = object()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("allowed_domain", "url", "allowed"),
|
||||
(
|
||||
("example.com", "http://example.com/1", True),
|
||||
("example.com", "http://example.org/1", False),
|
||||
("example.com", "http://sub.example.com/1", True),
|
||||
("sub.example.com", "http://sub.example.com/1", True),
|
||||
("sub.example.com", "http://example.com/1", False),
|
||||
("example.com", "http://example.com:8000/1", True),
|
||||
("example.com", "http://example.org/example.com", False),
|
||||
("example.com", "http://example.org/foo.example.com", False),
|
||||
("example.com", "http://example.com.example", False),
|
||||
("a.example", "http://nota.example", False),
|
||||
("b.a.example", "http://notb.a.example", False),
|
||||
),
|
||||
)
|
||||
def test_process_request_domain_filtering(allowed_domain, url, allowed):
|
||||
crawler = get_crawler(Spider)
|
||||
spider = crawler._create_spider(name="a", allowed_domains=[allowed_domain])
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
request = Request(url)
|
||||
if allowed:
|
||||
assert mw.process_request(request, spider) is None
|
||||
else:
|
||||
with pytest.raises(IgnoreRequest):
|
||||
mw.process_request(request, spider)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("value", "filtered"),
|
||||
(
|
||||
(UNSET, True),
|
||||
(None, True),
|
||||
(False, True),
|
||||
(True, False),
|
||||
),
|
||||
)
|
||||
def test_process_request_dont_filter(value, filtered):
|
||||
crawler = get_crawler(Spider)
|
||||
spider = crawler._create_spider(name="a", allowed_domains=["a.example"])
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
kwargs = {}
|
||||
if value is not UNSET:
|
||||
kwargs["dont_filter"] = value
|
||||
request = Request("https://b.example", **kwargs)
|
||||
if filtered:
|
||||
with pytest.raises(IgnoreRequest):
|
||||
mw.process_request(request, spider)
|
||||
else:
|
||||
assert mw.process_request(request, spider) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
(
|
||||
UNSET,
|
||||
None,
|
||||
[],
|
||||
),
|
||||
)
|
||||
def test_process_request_no_allowed_domains(value):
|
||||
crawler = get_crawler(Spider)
|
||||
kwargs = {}
|
||||
if value is not UNSET:
|
||||
kwargs["allowed_domains"] = value
|
||||
spider = crawler._create_spider(name="a", **kwargs)
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
request = Request("https://example.com")
|
||||
assert mw.process_request(request, spider) is None
|
||||
|
||||
|
||||
def test_process_request_invalid_domains():
|
||||
crawler = get_crawler(Spider)
|
||||
allowed_domains = ["a.example", None, "http:////b.example", "//c.example"]
|
||||
spider = crawler._create_spider(name="a", allowed_domains=allowed_domains)
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
request = Request("https://a.example")
|
||||
assert mw.process_request(request, spider) is None
|
||||
for letter in ("b", "c"):
|
||||
request = Request(f"https://{letter}.example")
|
||||
with pytest.raises(IgnoreRequest):
|
||||
mw.process_request(request, spider)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("allowed_domain", "url", "allowed"),
|
||||
(
|
||||
("example.com", "http://example.com/1", True),
|
||||
("example.com", "http://example.org/1", False),
|
||||
("example.com", "http://sub.example.com/1", True),
|
||||
("sub.example.com", "http://sub.example.com/1", True),
|
||||
("sub.example.com", "http://example.com/1", False),
|
||||
("example.com", "http://example.com:8000/1", True),
|
||||
("example.com", "http://example.org/example.com", False),
|
||||
("example.com", "http://example.org/foo.example.com", False),
|
||||
("example.com", "http://example.com.example", False),
|
||||
("a.example", "http://nota.example", False),
|
||||
("b.a.example", "http://notb.a.example", False),
|
||||
),
|
||||
)
|
||||
def test_request_scheduled_domain_filtering(allowed_domain, url, allowed):
|
||||
crawler = get_crawler(Spider)
|
||||
spider = crawler._create_spider(name="a", allowed_domains=[allowed_domain])
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
request = Request(url)
|
||||
if allowed:
|
||||
assert mw.request_scheduled(request, spider) is None
|
||||
else:
|
||||
with pytest.raises(IgnoreRequest):
|
||||
mw.request_scheduled(request, spider)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("value", "filtered"),
|
||||
(
|
||||
(UNSET, True),
|
||||
(None, True),
|
||||
(False, True),
|
||||
(True, False),
|
||||
),
|
||||
)
|
||||
def test_request_scheduled_dont_filter(value, filtered):
|
||||
crawler = get_crawler(Spider)
|
||||
spider = crawler._create_spider(name="a", allowed_domains=["a.example"])
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
kwargs = {}
|
||||
if value is not UNSET:
|
||||
kwargs["dont_filter"] = value
|
||||
request = Request("https://b.example", **kwargs)
|
||||
if filtered:
|
||||
with pytest.raises(IgnoreRequest):
|
||||
mw.request_scheduled(request, spider)
|
||||
else:
|
||||
assert mw.request_scheduled(request, spider) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
(
|
||||
UNSET,
|
||||
None,
|
||||
[],
|
||||
),
|
||||
)
|
||||
def test_request_scheduled_no_allowed_domains(value):
|
||||
crawler = get_crawler(Spider)
|
||||
kwargs = {}
|
||||
if value is not UNSET:
|
||||
kwargs["allowed_domains"] = value
|
||||
spider = crawler._create_spider(name="a", **kwargs)
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
request = Request("https://example.com")
|
||||
assert mw.request_scheduled(request, spider) is None
|
||||
|
||||
|
||||
def test_request_scheduled_invalid_domains():
|
||||
crawler = get_crawler(Spider)
|
||||
allowed_domains = ["a.example", None, "http:////b.example", "//c.example"]
|
||||
spider = crawler._create_spider(name="a", allowed_domains=allowed_domains)
|
||||
mw = OffsiteMiddleware.from_crawler(crawler)
|
||||
mw.spider_opened(spider)
|
||||
request = Request("https://a.example")
|
||||
assert mw.request_scheduled(request, spider) is None
|
||||
for letter in ("b", "c"):
|
||||
request = Request(f"https://{letter}.example")
|
||||
with pytest.raises(IgnoreRequest):
|
||||
mw.request_scheduled(request, spider)
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -15,8 +15,10 @@ import subprocess
|
|||
import sys
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
from logging import DEBUG
|
||||
from pathlib import Path
|
||||
from threading import Timer
|
||||
from unittest.mock import Mock
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import attr
|
||||
|
|
@ -27,11 +29,13 @@ from twisted.trial import unittest
|
|||
from twisted.web import server, static, util
|
||||
|
||||
from scrapy import signals
|
||||
from scrapy.core.engine import ExecutionEngine
|
||||
from scrapy.exceptions import CloseSpider
|
||||
from scrapy.core.engine import ExecutionEngine, Slot
|
||||
from scrapy.core.scheduler import BaseScheduler
|
||||
from scrapy.exceptions import CloseSpider, IgnoreRequest
|
||||
from scrapy.http import Request
|
||||
from scrapy.item import Field, Item
|
||||
from scrapy.linkextractors import LinkExtractor
|
||||
from scrapy.signals import request_scheduled
|
||||
from scrapy.spiders import Spider
|
||||
from scrapy.utils.signal import disconnect_all
|
||||
from scrapy.utils.test import get_crawler
|
||||
|
|
@ -467,6 +471,38 @@ class EngineTest(unittest.TestCase):
|
|||
self.assertNotIn(b"Traceback", stderr)
|
||||
|
||||
|
||||
def test_request_scheduled_signal(caplog):
|
||||
class TestScheduler(BaseScheduler):
|
||||
def __init__(self):
|
||||
self.enqueued = []
|
||||
|
||||
def enqueue_request(self, request: Request) -> bool:
|
||||
self.enqueued.append(request)
|
||||
return True
|
||||
|
||||
def signal_handler(request: Request, spider: Spider) -> None:
|
||||
if "drop" in request.url:
|
||||
raise IgnoreRequest
|
||||
|
||||
spider = TestSpider()
|
||||
crawler = get_crawler(spider.__class__)
|
||||
engine = ExecutionEngine(crawler, lambda _: None)
|
||||
engine.downloader._slot_gc_loop.stop()
|
||||
scheduler = TestScheduler()
|
||||
engine.slot = Slot((), None, Mock(), scheduler)
|
||||
crawler.signals.connect(signal_handler, request_scheduled)
|
||||
keep_request = Request("https://keep.example")
|
||||
engine._schedule_request(keep_request, spider)
|
||||
drop_request = Request("https://drop.example")
|
||||
caplog.set_level(DEBUG)
|
||||
engine._schedule_request(drop_request, spider)
|
||||
assert scheduler.enqueued == [
|
||||
keep_request
|
||||
], f"{scheduler.enqueued!r} != [{keep_request!r}]"
|
||||
assert "dropped request <GET https://drop.example>" in caplog.text
|
||||
crawler.signals.disconnect(signal_handler, request_scheduled)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "runserver":
|
||||
start_test_site(debug=True)
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import unittest
|
|||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
from scrapy.utils.misc import set_environ
|
||||
from scrapy.utils.project import data_path, get_project_settings
|
||||
|
||||
|
||||
|
|
@ -38,20 +39,6 @@ class ProjectUtilsTest(unittest.TestCase):
|
|||
self.assertEqual(abspath, data_path(abspath))
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def set_env(**update):
|
||||
modified = set(update.keys()) & set(os.environ.keys())
|
||||
update_after = {k: os.environ[k] for k in modified}
|
||||
remove_after = frozenset(k for k in update if k not in os.environ)
|
||||
try:
|
||||
os.environ.update(update)
|
||||
yield
|
||||
finally:
|
||||
os.environ.update(update_after)
|
||||
for k in remove_after:
|
||||
os.environ.pop(k)
|
||||
|
||||
|
||||
class GetProjectSettingsTestCase(unittest.TestCase):
|
||||
def test_valid_envvar(self):
|
||||
value = "tests.test_cmdline.settings"
|
||||
|
|
@ -60,7 +47,7 @@ class GetProjectSettingsTestCase(unittest.TestCase):
|
|||
}
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error")
|
||||
with set_env(**envvars):
|
||||
with set_environ(**envvars):
|
||||
settings = get_project_settings()
|
||||
|
||||
assert settings.get("SETTINGS_MODULE") == value
|
||||
|
|
@ -69,7 +56,7 @@ class GetProjectSettingsTestCase(unittest.TestCase):
|
|||
envvars = {
|
||||
"SCRAPY_FOO": "bar",
|
||||
}
|
||||
with set_env(**envvars):
|
||||
with set_environ(**envvars):
|
||||
settings = get_project_settings()
|
||||
|
||||
assert settings.get("SCRAPY_FOO") is None
|
||||
|
|
@ -80,7 +67,7 @@ class GetProjectSettingsTestCase(unittest.TestCase):
|
|||
"SCRAPY_FOO": "bar",
|
||||
"SCRAPY_SETTINGS_MODULE": value,
|
||||
}
|
||||
with set_env(**envvars):
|
||||
with set_environ(**envvars):
|
||||
settings = get_project_settings()
|
||||
assert settings.get("SETTINGS_MODULE") == value
|
||||
assert settings.get("SCRAPY_FOO") is None
|
||||
|
|
|
|||
17
tox.ini
17
tox.ini
|
|
@ -26,6 +26,9 @@ deps =
|
|||
|
||||
# mitmproxy does not support PyPy
|
||||
mitmproxy; implementation_name != 'pypy'
|
||||
# https://github.com/pallets/werkzeug/pull/2768 breaks flask, required by
|
||||
# mitmproxy.
|
||||
werkzeug < 3; python_version < '3.9' and implementation_name != 'pypy'
|
||||
passenv =
|
||||
S3_TEST_FILE_URI
|
||||
AWS_ACCESS_KEY_ID
|
||||
|
|
@ -90,6 +93,7 @@ commands =
|
|||
twine check dist/*
|
||||
|
||||
[pinned]
|
||||
basepython = python3.8
|
||||
deps =
|
||||
cryptography==36.0.0
|
||||
cssselect==0.9.1
|
||||
|
|
@ -116,7 +120,7 @@ commands =
|
|||
pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 scrapy tests}
|
||||
|
||||
[testenv:pinned]
|
||||
basepython = python3.8
|
||||
basepython = {[pinned]basepython}
|
||||
deps =
|
||||
{[pinned]deps}
|
||||
PyDispatcher==2.0.5
|
||||
|
|
@ -126,7 +130,7 @@ setenv =
|
|||
commands = {[pinned]commands}
|
||||
|
||||
[testenv:windows-pinned]
|
||||
basepython = python3
|
||||
basepython = {[pinned]basepython}
|
||||
deps =
|
||||
{[pinned]deps}
|
||||
PyDispatcher==2.0.5
|
||||
|
|
@ -155,7 +159,7 @@ deps =
|
|||
ipython
|
||||
|
||||
[testenv:extra-deps-pinned]
|
||||
basepython = python3.8
|
||||
basepython = {[pinned]basepython}
|
||||
deps =
|
||||
{[pinned]deps}
|
||||
boto3==1.20.0
|
||||
|
|
@ -179,6 +183,7 @@ commands =
|
|||
{[testenv]commands} --reactor=asyncio
|
||||
|
||||
[testenv:asyncio-pinned]
|
||||
basepython = {[pinned]basepython}
|
||||
deps = {[testenv:pinned]deps}
|
||||
commands = {[pinned]commands} --reactor=asyncio
|
||||
install_command = {[pinned]install_command}
|
||||
|
|
@ -191,12 +196,12 @@ commands =
|
|||
pytest {posargs:--durations=10 docs scrapy tests}
|
||||
|
||||
[testenv:pypy3-pinned]
|
||||
basepython = {[testenv:pypy3]basepython}
|
||||
basepython = pypy3.8
|
||||
deps =
|
||||
{[pinned]deps}
|
||||
PyPyDispatcher==2.1.0
|
||||
commands =
|
||||
pytest --durations=10 scrapy tests
|
||||
pytest {posargs:--durations=10 scrapy tests}
|
||||
install_command = {[pinned]install_command}
|
||||
setenv =
|
||||
{[pinned]setenv}
|
||||
|
|
@ -244,7 +249,7 @@ commands =
|
|||
pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:tests -k s3}
|
||||
|
||||
[testenv:botocore-pinned]
|
||||
basepython = python3.8
|
||||
basepython = {[pinned]basepython}
|
||||
deps =
|
||||
{[pinned]deps}
|
||||
botocore==1.4.87
|
||||
|
|
|
|||
Loading…
Reference in New Issue