RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased

This commit is contained in:
Andrey Rakhmatullin 2022-01-10 19:37:46 +05:00
parent c105a7a85f 9b8285d98a
commit 8864407c4d
33 changed files with 332 additions and 118 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
.github/workflows/tests-windows.yml vendored
View File

@ -17,10 +17,15 @@ jobs:
- python-version: 3.8
env:
TOXENV: py
# https://twistedmatrix.com/trac/ticket/9990
#- python-version: 3.9
#env:
#TOXENV: py
- python-version: 3.9
env:
TOXENV: py
- python-version: "3.10"
env:
TOXENV: py
- python-version: "3.10"
env:
TOXENV: asyncio
steps:
- uses: actions/checkout@v2

8
docs/conftest.py
View File

@ -3,7 +3,11 @@ from doctest import ELLIPSIS, NORMALIZE_WHITESPACE
from scrapy.http.response.html import HtmlResponse
from sybil import Sybil
from sybil.parsers.codeblock import CodeBlockParser
try:
# >2.0.1
from sybil.parsers.codeblock import PythonCodeBlockParser
except ImportError:
from sybil.parsers.codeblock import CodeBlockParser as PythonCodeBlockParser
from sybil.parsers.doctest import DocTestParser
from sybil.parsers.skip import skip
@ -21,7 +25,7 @@ def setup(namespace):
pytest_collect_file = Sybil(
parsers=[
DocTestParser(optionflags=ELLIPSIS | NORMALIZE_WHITESPACE),
CodeBlockParser(future_imports=['print_function']),
PythonCodeBlockParser(future_imports=['print_function']),
skip,
],
pattern='*.rst',

22
docs/news.rst
View File

@ -1,3 +1,25 @@
.. note::
.. versionchanged:: VERSION
The Twisted reactor is now installed when
:meth:`~scrapy.crawler.CrawlerProcess.crawl` is first called, not when a
:class:`scrapy.crawler.CrawlerProcess` object is created. Because of this,
:setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` are now
honored in :attr:`~scrapy.Spider.custom_settings`. In older Scrapy versions
they are silently ignored when set there and you need to set these settings
in some other way.
.. note::
.. versionchanged:: VERSION
Previously this setting had no effect in a spider
:attr:`~scrapy.Spider.custom_settings` attribute. Now it will be used, but
if you :ref:`run several spiders in one process <run-multiple-spiders>`,
they must not have different values for this setting, because they will use
a single reactor instance.
.. _news:
Release notes

2
docs/topics/architecture.rst
View File

@ -67,7 +67,7 @@ this:
the :ref:`Scheduler <component-scheduler>` and asks for possible next Requests
to crawl.
9. The process repeats (from step 1) until there are no more requests from the
9. The process repeats (from step 3) until there are no more requests from the
:ref:`Scheduler <component-scheduler>`.
Components

33
docs/topics/asyncio.rst
View File

@ -10,11 +10,6 @@ Scrapy has partial support for :mod:`asyncio`. After you :ref:`install the
asyncio reactor <install-asyncio>`, you may use :mod:`asyncio` and
:mod:`asyncio`-powered libraries in any :doc:`coroutine <coroutines>`.
.. warning:: :mod:`asyncio` support in Scrapy is experimental, and not yet
recommended for production environments. Future Scrapy versions
may introduce related changes without a deprecation period or
warning.
.. _install-asyncio:
Installing the asyncio reactor
@ -41,6 +36,34 @@ use it instead of the default asyncio event loop.
.. _asyncio-await-dfd:
Windows-specific notes
======================
The Windows implementation of :mod:`asyncio` can use two event loop
implementations: :class:`~asyncio.SelectorEventLoop` (default before Python
3.8, required when using Twisted) and :class:`~asyncio.ProactorEventLoop`
(default since Python 3.8, cannot work with Twisted). So on Python 3.8+ the
event loop class needs to be changed. Scrapy since VERSION does this
automatically when you change the :setting:`TWISTED_REACTOR` setting or call
:func:`~scrapy.utils.reactor.install_reactor`, but if you install the reactor
by other means or use an older Scrapy version you need to call the following
code before installing the reactor::
import asyncio
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
You can put this in the same function that installs the reactor, if you do that
yourself, or in some code that runs before the reactor is installed, e.g.
``settings.py``.
.. note:: Other libraries you use may require
:class:`~asyncio.ProactorEventLoop`, e.g. because it supports
subprocesses (this is the case with `playwright`_), so you cannot use
them together with Scrapy on Windows (but you should be able to use
them on WSL or native Linux).
.. _playwright: https://github.com/microsoft/playwright-python
Awaiting on Deferreds
=====================

19
docs/topics/practices.rst
View File

@ -193,6 +193,25 @@ Same example but running the spiders sequentially by chaining the deferreds:
crawl()
reactor.run() # the script will block here until the last crawl call is finished
Different spiders can set different values for the same setting, but when they
run in the same process it may be impossible, by design or because of some
limitations, to use these different values. What happens in practice is
different for different settings:
* :setting:`SPIDER_LOADER_CLASS` and the ones used by its value
(:setting:`SPIDER_MODULES`, :setting:`SPIDER_LOADER_WARN_ONLY` for the
default one) cannot be read from the per-spider settings. These are applied
when the :class:`~scrapy.crawler.CrawlerRunner` or
:class:`~scrapy.crawler.CrawlerProcess` object is created.
* For :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` the first
available value is used, and if a spider requests a different reactor an
exception will be raised. These are applied when the reactor is installed.
* For :setting:`REACTOR_THREADPOOL_MAXSIZE`, :setting:`DNS_RESOLVER` and the
ones used by the resolver (:setting:`DNSCACHE_ENABLED`,
:setting:`DNSCACHE_SIZE`, :setting:`DNS_TIMEOUT` for ones included in Scrapy)
the first available value is used. These are applied when the reactor is
started.
.. seealso:: :ref:`run-from-script`.
.. _distributed-crawls:

7
docs/topics/settings.rst
View File

@ -1638,10 +1638,9 @@ which raises :exc:`Exception`, becomes::
The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, which
means that Scrapy will not attempt to install any specific reactor, and the
default reactor defined by Twisted for the current platform will be used. This
is to maintain backward compatibility and avoid possible problems caused by
using a non-default reactor.
means that Scrapy will install the default reactor defined by Twisted for the
current platform. This is to maintain backward compatibility and avoid possible
problems caused by using a non-default reactor.
For additional information, see :doc:`core/howto/choosing-reactor`.

1
pylintrc
View File

@ -112,6 +112,7 @@ disable=abstract-method,
unused-private-member,
unused-variable,
unused-wildcard-import,
use-implicit-booleaness-not-comparison,
used-before-assignment,
useless-object-inheritance, # Required for Python 2 support
useless-return,

4
scrapy/commands/__init__.py
View File

@ -43,14 +43,14 @@ class ScrapyCommand:
def long_desc(self):
"""A long description of the command. Return short description when not
available. It cannot contain newlines, since contents will be formatted
available. It cannot contain newlines since contents will be formatted
by optparser which removes newlines and wraps text.
"""
return self.short_desc()
def help(self):
"""An extensive help for the command. It will be shown when using the
"help" command. It can contain newlines, since no post-formatting will
"help" command. It can contain newlines since no post-formatting will
be applied to its contents.
"""
return self.long_desc()

2
scrapy/commands/shell.py
View File

@ -75,6 +75,6 @@ class Command(ScrapyCommand):
def _start_crawler_thread(self):
t = Thread(target=self.crawler_process.start,
kwargs={'stop_after_crawl': False})
kwargs={'stop_after_crawl': False, 'install_signal_handlers': False})
t.daemon = True
t.start()

2
scrapy/core/downloader/handlers/http11.py
View File

@ -213,7 +213,7 @@ class TunnelingAgent(Agent):
# proxy host and port are required for HTTP pool `key`
# otherwise, same remote host connection request could reuse
# a cached tunneled connection to a different proxy
key = key + self._proxyConf
key += self._proxyConf
return super()._requestWithEndpoint(
key=key,
endpoint=endpoint,

5
scrapy/core/downloader/middleware.py
View File

@ -3,7 +3,7 @@ Downloader Middleware manager
See documentation in docs/topics/downloader-middleware.rst
"""
from typing import Callable, Union
from typing import Callable, Union, cast
from twisted.internet import defer
from twisted.python.failure import Failure
@ -37,6 +37,7 @@ class DownloaderMiddlewareManager(MiddlewareManager):
@defer.inlineCallbacks
def process_request(request: Request):
for method in self.methods['process_request']:
method = cast(Callable, method)
response = yield deferred_from_coro(method(request=request, spider=spider))
if response is not None and not isinstance(response, (Response, Request)):
raise _InvalidOutput(
@ -55,6 +56,7 @@ class DownloaderMiddlewareManager(MiddlewareManager):
return response
for method in self.methods['process_response']:
method = cast(Callable, method)
response = yield deferred_from_coro(method(request=request, response=response, spider=spider))
if not isinstance(response, (Response, Request)):
raise _InvalidOutput(
@ -69,6 +71,7 @@ class DownloaderMiddlewareManager(MiddlewareManager):
def process_exception(failure: Failure):
exception = failure.value
for method in self.methods['process_exception']:
method = cast(Callable, method)
response = yield deferred_from_coro(method(request=request, exception=exception, spider=spider))
if response is not None and not isinstance(response, (Response, Request)):
raise _InvalidOutput(

3
scrapy/core/downloader/webclient.py
View File

@ -3,7 +3,7 @@ from time import time
from urllib.parse import urlparse, urlunparse, urldefrag
from twisted.web.http import HTTPClient
from twisted.internet import defer, reactor
from twisted.internet import defer
from twisted.internet.protocol import ClientFactory
from scrapy.http import Headers
@ -170,6 +170,7 @@ class ScrapyHTTPClientFactory(ClientFactory):
p.followRedirect = self.followRedirect
p.afterFoundGet = self.afterFoundGet
if self.timeout:
from twisted.internet import reactor
timeoutCall = reactor.callLater(self.timeout, p.timeout)
self.deferred.addBoth(self._cancelTimeout, timeoutCall)
return p

4
scrapy/core/http2/stream.py
View File

@ -285,8 +285,8 @@ class Stream:
self._protocol.conn.send_data(self.stream_id, data_chunk, end_stream=False)
bytes_to_send_size = bytes_to_send_size - chunk_size
self.metadata['remaining_content_length'] = self.metadata['remaining_content_length'] - chunk_size
bytes_to_send_size -= chunk_size
self.metadata['remaining_content_length'] -= chunk_size
self.metadata['remaining_content_length'] = max(0, self.metadata['remaining_content_length'])

3
scrapy/core/spidermw.py
View File

@ -4,7 +4,7 @@ Spider Middleware manager
See documentation in docs/topics/spider-middleware.rst
"""
from itertools import islice
from typing import Any, AsyncGenerator, AsyncIterable, Callable, Generator, Iterable, Union
from typing import Any, AsyncGenerator, AsyncIterable, Callable, Generator, Iterable, Union, cast
from twisted.internet.defer import Deferred
from twisted.python.failure import Failure
@ -48,6 +48,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
def _process_spider_input(self, scrape_func: ScrapeFunc, response: Response, request: Request,
spider: Spider) -> Any:
for method in self.methods['process_spider_input']:
method = cast(Callable, method)
try:
result = method(response=response, spider=spider)
if result is not None:

40
scrapy/crawler.py
View File

@ -25,6 +25,7 @@ from scrapy.utils.log import (
configure_logging,
get_scrapy_root_handler,
install_scrapy_root_handler,
log_reactor_info,
log_scrapy_info,
LogCounterHandler,
)
@ -38,7 +39,7 @@ logger = logging.getLogger(__name__)
class Crawler:
def __init__(self, spidercls, settings=None):
def __init__(self, spidercls, settings=None, init_reactor: bool = False):
if isinstance(spidercls, Spider):
raise ValueError('The spidercls argument must be a class, not an object')
@ -69,6 +70,20 @@ class Crawler:
lf_cls = load_object(self.settings['LOG_FORMATTER'])
self.logformatter = lf_cls.from_crawler(self)
reactor_class = self.settings.get("TWISTED_REACTOR")
if init_reactor:
# this needs to be done after the spider settings are merged,
# but before something imports twisted.internet.reactor
if reactor_class:
install_reactor(reactor_class, self.settings["ASYNCIO_EVENT_LOOP"])
else:
from twisted.internet import default
default.install()
log_reactor_info()
if reactor_class:
verify_installed_reactor(reactor_class)
self.extensions = ExtensionManager.from_crawler(self)
self.settings.freeze()
@ -153,7 +168,6 @@ class CrawlerRunner:
self._crawlers = set()
self._active = set()
self.bootstrap_failed = False
self._handle_twisted_reactor()
@property
def spiders(self):
@ -247,10 +261,6 @@ class CrawlerRunner:
while self._active:
yield defer.DeferredList(self._active)
def _handle_twisted_reactor(self):
if self.settings.get("TWISTED_REACTOR"):
verify_installed_reactor(self.settings["TWISTED_REACTOR"])
class CrawlerProcess(CrawlerRunner):
"""
@ -278,7 +288,6 @@ class CrawlerProcess(CrawlerRunner):
def __init__(self, settings=None, install_root_handler=True):
super().__init__(settings)
install_shutdown_handlers(self._signal_shutdown)
configure_logging(self.settings, install_root_handler)
log_scrapy_info(self.settings)
@ -298,7 +307,12 @@ class CrawlerProcess(CrawlerRunner):
{'signame': signame})
reactor.callFromThread(self._stop_reactor)
def start(self, stop_after_crawl=True):
def _create_crawler(self, spidercls):
if isinstance(spidercls, str):
spidercls = self.spider_loader.load(spidercls)
return Crawler(spidercls, self.settings, init_reactor=True)
def start(self, stop_after_crawl=True, install_signal_handlers=True):
"""
This method starts a :mod:`~twisted.internet.reactor`, adjusts its pool
size to :setting:`REACTOR_THREADPOOL_MAXSIZE`, and installs a DNS cache
@ -309,6 +323,9 @@ class CrawlerProcess(CrawlerRunner):
:param bool stop_after_crawl: stop or not the reactor when all
crawlers have finished
:param bool install_signal_handlers: whether to install the shutdown
handlers (default: True)
"""
from twisted.internet import reactor
if stop_after_crawl:
@ -318,6 +335,8 @@ class CrawlerProcess(CrawlerRunner):
return
d.addBoth(self._stop_reactor)
if install_signal_handlers:
install_shutdown_handlers(self._signal_shutdown)
resolver_class = load_object(self.settings["DNS_RESOLVER"])
resolver = create_instance(resolver_class, self.settings, self, reactor=reactor)
resolver.install_on_reactor()
@ -337,8 +356,3 @@ class CrawlerProcess(CrawlerRunner):
reactor.stop()
except RuntimeError: # raised if already stopped or in shutdown stage
pass
def _handle_twisted_reactor(self):
if self.settings.get("TWISTED_REACTOR"):
install_reactor(self.settings["TWISTED_REACTOR"], self.settings["ASYNCIO_EVENT_LOOP"])
super()._handle_twisted_reactor()

17
scrapy/middleware.py
View File

@ -1,7 +1,7 @@
import logging
import pprint
from collections import defaultdict, deque
from typing import Callable, Deque, Dict
from typing import Callable, Deque, Dict, Optional, cast, Iterable
from twisted.internet.defer import Deferred
@ -9,7 +9,7 @@ from scrapy import Spider
from scrapy.exceptions import NotConfigured
from scrapy.settings import Settings
from scrapy.utils.misc import create_instance, load_object
from scrapy.utils.defer import process_parallel, process_chain, process_chain_both
from scrapy.utils.defer import process_parallel, process_chain
logger = logging.getLogger(__name__)
@ -21,7 +21,8 @@ class MiddlewareManager:
def __init__(self, *middlewares):
self.middlewares = middlewares
self.methods: Dict[str, Deque[Callable]] = defaultdict(deque)
# Optional because process_spider_output and process_spider_exception can be None
self.methods: Dict[str, Deque[Optional[Callable]]] = defaultdict(deque)
for mw in middlewares:
self._add_middleware(mw)
@ -64,14 +65,12 @@ class MiddlewareManager:
self.methods['close_spider'].appendleft(mw.close_spider)
def _process_parallel(self, methodname: str, obj, *args) -> Deferred:
return process_parallel(self.methods[methodname], obj, *args)
methods = cast(Iterable[Callable], self.methods[methodname])
return process_parallel(methods, obj, *args)
def _process_chain(self, methodname: str, obj, *args) -> Deferred:
return process_chain(self.methods[methodname], obj, *args)
def _process_chain_both(self, cb_methodname: str, eb_methodname: str, obj, *args) -> Deferred:
return process_chain_both(self.methods[cb_methodname],
self.methods[eb_methodname], obj, *args)
methods = cast(Iterable[Callable], self.methods[methodname])
return process_chain(methods, obj, *args)
def open_spider(self, spider: Spider) -> Deferred:
return self._process_parallel('open_spider', spider)

2
scrapy/utils/conf.py
View File

@ -164,7 +164,7 @@ def feed_process_params_from_cli(settings, output, output_format=None,
message = (
'The -t command line option is deprecated in favor of '
'specifying the output format within the output URI. See the '
'documentation of the -o and -O options for more information.',
'documentation of the -o and -O options for more information.'
)
warnings.warn(message, ScrapyDeprecationWarning, stacklevel=2)
return {output[0]: {'format': output_format}}

5
scrapy/utils/log.py
View File

@ -143,7 +143,7 @@ def _get_handler(settings):
return handler
def log_scrapy_info(settings):
def log_scrapy_info(settings: Settings) -> None:
logger.info("Scrapy %(version)s started (bot: %(bot)s)",
{'version': scrapy.__version__, 'bot': settings['BOT_NAME']})
versions = [
@ -152,6 +152,9 @@ def log_scrapy_info(settings):
if name != "Scrapy"
]
logger.info("Versions: %(versions)s", {'versions': ", ".join(versions)})
def log_reactor_info() -> None:
from twisted.internet import reactor
logger.debug("Using reactor: %s.%s", reactor.__module__, reactor.__class__.__name__)
from twisted.internet import asyncioreactor

5
scrapy/utils/reactor.py
View File

@ -1,4 +1,5 @@
import asyncio
import sys
from contextlib import suppress
from twisted.internet import asyncioreactor, error
@ -57,6 +58,10 @@ def install_reactor(reactor_path, event_loop_path=None):
reactor_class = load_object(reactor_path)
if reactor_class is asyncioreactor.AsyncioSelectorReactor:
with suppress(error.ReactorAlreadyInstalledError):
if sys.version_info >= (3, 8) and sys.platform == "win32":
policy = asyncio.get_event_loop_policy()
if not isinstance(policy, asyncio.WindowsSelectorEventLoopPolicy):
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
if event_loop_path is not None:
event_loop_class = load_object(event_loop_path)
event_loop = event_loop_class()

8
scrapy/utils/response.py
View File

@ -3,8 +3,9 @@ This module provides some useful functions for working with
scrapy.http.Response objects
"""
import os
import webbrowser
import re
import tempfile
import webbrowser
from typing import Any, Callable, Iterable, Optional, Tuple, Union
from weakref import WeakKeyDictionary
@ -80,8 +81,9 @@ def open_in_browser(
body = response.body
if isinstance(response, HtmlResponse):
if b'<base' not in body:
repl = f'<head><base href="{response.url}">'
body = body.replace(b'<head>', to_bytes(repl))
repl = fr'\1<base href="{response.url}">'
body = re.sub(b"<!--.*?-->", b"", body, flags=re.DOTALL)
body = re.sub(rb"(<head(?:>|\s.*?>))", to_bytes(repl), body)
ext = '.html'
elif isinstance(response, TextResponse):
ext = '.txt'

3
tests/CrawlerProcess/asyncio_enabled_reactor.py
View File

@ -1,6 +1,9 @@
import asyncio
import sys
from twisted.internet import asyncioreactor
if sys.version_info >= (3, 8) and sys.platform == "win32":
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncioreactor.install(asyncio.get_event_loop())
import scrapy

14
tests/CrawlerProcess/twisted_reactor_custom_settings.py Normal file
View File

@ -0,0 +1,14 @@
import scrapy
from scrapy.crawler import CrawlerProcess
class AsyncioReactorSpider(scrapy.Spider):
name = 'asyncio_reactor'
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
}
process = CrawlerProcess()
process.crawl(AsyncioReactorSpider)
process.start()

22
tests/CrawlerProcess/twisted_reactor_custom_settings_conflict.py Normal file
View File

@ -0,0 +1,22 @@
import scrapy
from scrapy.crawler import CrawlerProcess
class SelectReactorSpider(scrapy.Spider):
name = 'select_reactor'
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.selectreactor.SelectReactor",
}
class AsyncioReactorSpider(scrapy.Spider):
name = 'asyncio_reactor'
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
}
process = CrawlerProcess()
process.crawl(SelectReactorSpider)
process.crawl(AsyncioReactorSpider)
process.start()

21
tests/CrawlerProcess/twisted_reactor_custom_settings_same.py Normal file
View File

@ -0,0 +1,21 @@
import scrapy
from scrapy.crawler import CrawlerProcess
class AsyncioReactorSpider1(scrapy.Spider):
name = 'asyncio_reactor1'
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
}
class AsyncioReactorSpider2(scrapy.Spider):
name = 'asyncio_reactor2'
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
}
process = CrawlerProcess()
process.crawl(AsyncioReactorSpider1)
process.crawl(AsyncioReactorSpider2)
process.start()

20
tests/test_commands.py
View File

@ -674,9 +674,6 @@ class MySpider(scrapy.Spider):
self.assertIn("start_requests", log)
self.assertIn("badspider.py", log)
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_asyncio_enabled_true(self):
log = self.get_log(self.debug_log_spider, args=[
'-s', 'TWISTED_REACTOR=twisted.internet.asyncioreactor.AsyncioSelectorReactor'
@ -699,15 +696,15 @@ class MySpider(scrapy.Spider):
])
self.assertIn("Using asyncio event loop: uvloop.Loop", log)
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_custom_asyncio_loop_enabled_false(self):
log = self.get_log(self.debug_log_spider, args=[
'-s', 'TWISTED_REACTOR=twisted.internet.asyncioreactor.AsyncioSelectorReactor'
])
import asyncio
loop = asyncio.new_event_loop()
if sys.platform != 'win32':
loop = asyncio.new_event_loop()
else:
loop = asyncio.SelectorEventLoop()
self.assertIn(f"Using asyncio event loop: {loop.__module__}.{loop.__class__.__name__}", log)
def test_output(self):
@ -765,6 +762,7 @@ class MySpider(scrapy.Spider):
self.assertIn("error: Please use only one of -o/--output and -O/--overwrite-output", log)
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
class WindowsRunSpiderCommandTest(RunSpiderCommandTest):
spider_filename = 'myspider.pyw'
@ -777,35 +775,27 @@ class WindowsRunSpiderCommandTest(RunSpiderCommandTest):
self.assertIn("start_requests", log)
self.assertIn("badspider.pyw", log)
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_run_good_spider(self):
super().test_run_good_spider()
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_runspider(self):
super().test_runspider()
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_runspider_dnscache_disabled(self):
super().test_runspider_dnscache_disabled()
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_runspider_log_level(self):
super().test_runspider_log_level()
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_runspider_log_short_names(self):
super().test_runspider_log_short_names()
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_runspider_no_spider_found(self):
super().test_runspider_no_spider_found()
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_output(self):
super().test_output()
@skipIf(platform.system() != 'Windows', "Windows required for .pyw files")
def test_overwrite_output(self):
super().test_overwrite_output()

22
tests/test_crawl.py
View File

@ -278,6 +278,28 @@ with multiples lines
self.assertEqual(s['engine.spider.name'], crawler.spider.name)
self.assertEqual(s['len(engine.scraper.slot.active)'], 1)
@defer.inlineCallbacks
def test_format_engine_status(self):
from scrapy.utils.engine import format_engine_status
est = []
def cb(response):
est.append(format_engine_status(crawler.engine))
crawler = self.runner.create_crawler(SingleRequestSpider)
yield crawler.crawl(seed=self.mockserver.url('/'), callback_func=cb, mockserver=self.mockserver)
self.assertEqual(len(est), 1, est)
est = est[0].split("\n")[2:-2] # remove header & footer
# convert to dict
est = [x.split(":") for x in est]
est = [x for sublist in est for x in sublist] # flatten
est = [x.lstrip().rstrip() for x in est]
it = iter(est)
s = dict(zip(it, it))
self.assertEqual(s['engine.spider.name'], crawler.spider.name)
self.assertEqual(s['len(engine.scraper.slot.active)'], '1')
@defer.inlineCallbacks
def test_graceful_crawl_error_handling(self):
"""

57
tests/test_crawler.py
View File

@ -4,10 +4,8 @@ import platform
import subprocess
import sys
import warnings
from unittest import skipIf
from pytest import raises, mark
from testfixtures import LogCapture
from twisted import version as twisted_version
from twisted.internet import defer
from twisted.python.versions import Version
@ -271,6 +269,7 @@ class CrawlerRunnerHasSpider(unittest.TestCase):
self.assertEqual(runner.bootstrap_failed, True)
@defer.inlineCallbacks
def test_crawler_runner_asyncio_enabled_true(self):
if self.reactor_pytest == 'asyncio':
CrawlerRunner(settings={
@ -279,35 +278,10 @@ class CrawlerRunnerHasSpider(unittest.TestCase):
else:
msg = r"The installed reactor \(.*?\) does not match the requested one \(.*?\)"
with self.assertRaisesRegex(Exception, msg):
CrawlerRunner(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
@defer.inlineCallbacks
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_crawler_process_asyncio_enabled_true(self):
with LogCapture(level=logging.DEBUG) as log:
if self.reactor_pytest == 'asyncio':
runner = CrawlerProcess(settings={
runner = CrawlerRunner(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
yield runner.crawl(NoRequestsSpider)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", str(log))
else:
msg = r"The installed reactor \(.*?\) does not match the requested one \(.*?\)"
with self.assertRaisesRegex(Exception, msg):
runner = CrawlerProcess(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
@defer.inlineCallbacks
def test_crawler_process_asyncio_enabled_false(self):
runner = CrawlerProcess(settings={"TWISTED_REACTOR": None})
with LogCapture(level=logging.DEBUG) as log:
yield runner.crawl(NoRequestsSpider)
self.assertNotIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", str(log))
class ScriptRunnerMixin:
@ -328,17 +302,11 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase):
self.assertIn('Spider closed (finished)', log)
self.assertNotIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_asyncio_enabled_no_reactor(self):
log = self.run_script('asyncio_enabled_no_reactor.py')
self.assertIn('Spider closed (finished)', log)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_asyncio_enabled_reactor(self):
log = self.run_script('asyncio_enabled_reactor.py')
self.assertIn('Spider closed (finished)', log)
@ -377,14 +345,26 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase):
self.assertIn("Spider closed (finished)", log)
self.assertIn("Using reactor: twisted.internet.pollreactor.PollReactor", log)
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_reactor_asyncio(self):
log = self.run_script("twisted_reactor_asyncio.py")
self.assertIn("Spider closed (finished)", log)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
def test_reactor_asyncio_custom_settings(self):
log = self.run_script("twisted_reactor_custom_settings.py")
self.assertIn("Spider closed (finished)", log)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
def test_reactor_asyncio_custom_settings_same(self):
log = self.run_script("twisted_reactor_custom_settings_same.py")
self.assertIn("Spider closed (finished)", log)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
def test_reactor_asyncio_custom_settings_conflict(self):
log = self.run_script("twisted_reactor_custom_settings_conflict.py")
self.assertIn("Using reactor: twisted.internet.selectreactor.SelectReactor", log)
self.assertIn("(twisted.internet.selectreactor.SelectReactor) does not match the requested one", log)
@mark.skipif(sys.implementation.name == 'pypy', reason='uvloop does not support pypy properly')
@mark.skipif(platform.system() == 'Windows', reason='uvloop does not support Windows')
@mark.skipif(twisted_version == Version('twisted', 21, 2, 0), reason='https://twistedmatrix.com/trac/ticket/10106')
@ -404,9 +384,6 @@ class CrawlerProcessSubprocess(ScriptRunnerMixin, unittest.TestCase):
self.assertIn("Using asyncio event loop: uvloop.Loop", log)
self.assertIn("async pipeline opened!", log)
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_default_loop_asyncio_deferred_signal(self):
log = self.run_script("asyncio_deferred_signal.py")
self.assertIn("Spider closed (finished)", log)

16
tests/test_downloader_handlers.py
View File

@ -1,6 +1,7 @@
import contextlib
import os
import shutil
import sys
import tempfile
from typing import Optional, Type
from unittest import mock
@ -287,6 +288,12 @@ class HttpTestCase(unittest.TestCase):
@defer.inlineCallbacks
def test_timeout_download_from_spider_nodata_rcvd(self):
if self.reactor_pytest == "asyncio" and sys.platform == "win32":
# https://twistedmatrix.com/trac/ticket/10279
raise unittest.SkipTest(
"This test produces DirtyReactorAggregateError on Windows with asyncio"
)
# client connects but no data is received
spider = Spider('foo')
meta = {'download_timeout': 0.5}
@ -296,6 +303,11 @@ class HttpTestCase(unittest.TestCase):
@defer.inlineCallbacks
def test_timeout_download_from_spider_server_hangs(self):
if self.reactor_pytest == "asyncio" and sys.platform == "win32":
# https://twistedmatrix.com/trac/ticket/10279
raise unittest.SkipTest(
"This test produces DirtyReactorAggregateError on Windows with asyncio"
)
# client connects, server send headers and some body bytes but hangs
spider = Spider('foo')
meta = {'download_timeout': 0.5}
@ -1055,6 +1067,10 @@ class BaseFTPTestCase(unittest.TestCase):
class FTPTestCase(BaseFTPTestCase):
def test_invalid_credentials(self):
if self.reactor_pytest == "asyncio" and sys.platform == "win32":
raise unittest.SkipTest(
"This test produces DirtyReactorAggregateError on Windows with asyncio"
)
from twisted.protocols.ftp import ConnectionLost
meta = dict(self.req_meta)

2
tests/test_request_left.py
View File

@ -22,7 +22,7 @@ class SignalCatcherSpider(Spider):
return spider
def on_request_left(self, request, spider):
self.caught_times = self.caught_times + 1
self.caught_times += 1
class TestCatching(TestCase):

7
tests/test_utils_asyncio.py
View File

@ -1,6 +1,4 @@
import platform
import sys
from unittest import skipIf, TestCase
from unittest import TestCase
from pytest import mark
@ -14,9 +12,6 @@ class AsyncioTest(TestCase):
# the result should depend only on the pytest --reactor argument
self.assertEqual(is_asyncio_reactor_installed(), self.reactor_pytest == 'asyncio')
# https://twistedmatrix.com/trac/ticket/9766
@skipIf(platform.system() == 'Windows' and sys.version_info >= (3, 8),
"the asyncio reactor is broken on Windows when running Python ≥ 3.8")
def test_install_asyncio_reactor(self):
# this should do nothing
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")

53
tests/test_utils_response.py
View File

@ -83,3 +83,56 @@ class ResponseUtilsTest(unittest.TestCase):
self.assertEqual(response_status_message(200), '200 OK')
self.assertEqual(response_status_message(404), '404 Not Found')
self.assertEqual(response_status_message(573), "573 Unknown Status")
def test_inject_base_url(self):
url = "http://www.example.com"
def check_base_url(burl):
path = urlparse(burl).path
if not os.path.exists(path):
path = burl.replace('file://', '')
with open(path, "rb") as f:
bbody = f.read()
self.assertEqual(bbody.count(b'<base href="' + to_bytes(url) + b'">'), 1)
return True
r1 = HtmlResponse(url, body=b"""
<html>
<head><title>Dummy</title></head>
<body><p>Hello world.</p></body>
</html>""")
r2 = HtmlResponse(url, body=b"""
<html>
<head id="foo"><title>Dummy</title></head>
<body>Hello world.</body>
</html>""")
r3 = HtmlResponse(url, body=b"""
<html>
<head><title>Dummy</title></head>
<body>
<header>Hello header</header>
<p>Hello world.</p>
</body>
</html>""")
r4 = HtmlResponse(url, body=b"""
<html>
<!-- <head>Dummy comment</head> -->
<head><title>Dummy</title></head>
<body><p>Hello world.</p></body>
</html>""")
r5 = HtmlResponse(url, body=b"""
<html>
<!--[if IE]>
<head><title>IE head</title></head>
<![endif]-->
<!--[if !IE]>-->
<head><title>Standard head</title></head>
<!--<![endif]-->
<body><p>Hello world.</p></body>
</html>""")
assert open_in_browser(r1, _openfunc=check_base_url), "Inject base url"
assert open_in_browser(r2, _openfunc=check_base_url), "Inject base url with argumented head"
assert open_in_browser(r3, _openfunc=check_base_url), "Inject unique base url with misleading tag"
assert open_in_browser(r4, _openfunc=check_base_url), "Inject unique base url with misleading comment"
assert open_in_browser(r5, _openfunc=check_base_url), "Inject unique base url with conditional comment"

8
tox.ini
View File

@ -29,7 +29,7 @@ passenv =
#allow tox virtualenv to upgrade pip/wheel/setuptools
download = true
commands =
py.test --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests}
pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests}
install_command =
pip install -U -ctests/upper-constraints.txt {opts} {packages}
@ -60,13 +60,13 @@ deps =
pytest-flake8
flake8==3.9.2 # https://github.com/tholo/pytest-flake8/issues/81
commands =
py.test --flake8 {posargs:docs scrapy tests}
pytest --flake8 {posargs:docs scrapy tests}
[testenv:pylint]
basepython = python3
deps =
{[testenv:extra-deps]deps}
pylint
pylint==2.12.1
commands =
pylint conftest.py docs extras scrapy setup.py tests
@ -142,7 +142,7 @@ setenv =
[testenv:pypy3]
basepython = pypy3
commands =
py.test {posargs:--durations=10 docs scrapy tests}
pytest {posargs:--durations=10 docs scrapy tests}
[testenv:pypy3-pinned]
basepython = {[testenv:pypy3]basepython}