RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into asyncio-signals

This commit is contained in:
Andrey Rakhmatullin 2020-02-07 16:58:59 +05:00
parent 489ffcda51 4f31c3ce01
commit 4a7c7340a0
28 changed files with 314 additions and 108 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/faq.rst
View File

@ -361,6 +361,17 @@ Note that by doing so, you lose the ability to set a specific timeout for DNS re
(the value of the :setting:`DNS_TIMEOUT` setting is ignored).
.. _faq-specific-reactor:
How to deal with ``<class 'ValueError'>: filedescriptor out of range in select()`` exceptions?
----------------------------------------------------------------------------------------------
This issue `has been reported`_ to appear when running broad crawls in macOS, where the default
Twisted reactor is :class:`twisted.internet.selectreactor.SelectReactor`. Switching to a
different reactor is possible by using the :setting:`TWISTED_REACTOR` setting.
.. _has been reported: https://github.com/scrapy/scrapy/issues/2905
.. _user agents: https://en.wikipedia.org/wiki/User_agent
.. _LIFO: https://en.wikipedia.org/wiki/Stack_(abstract_data_type)
.. _DFO order: https://en.wikipedia.org/wiki/Depth-first_search

7
docs/news.rst
View File

@ -288,6 +288,13 @@ Backward-incompatible changes
:class:`~scrapy.http.Request` objects instead of arbitrary Python data
structures.
* An additional ``crawler`` parameter has been added to the ``__init__`` method
of the :class:`scrapy.core.scheduler.Scheduler` class.
Custom scheduler subclasses which don't accept arbitrary parameters in
their ``__init__`` method might break because of this change.
For more information, refer to the documentation for the :setting:`SCHEDULER` setting.
See also :ref:`1.7-deprecation-removals` below.

7
docs/topics/broad-crawls.rst
View File

@ -211,3 +211,10 @@ If your broad crawl shows a high memory usage, in addition to :ref:`crawling in
BFO order <broad-crawls-bfo>` and :ref:`lowering concurrency
<broad-crawls-concurrency>` you should :ref:`debug your memory leaks
<topics-leaks>`.
Install a specific Twisted reactor
==================================
If the crawl is exceeding the system's capabilities, you might want to try
installing a specific Twisted reactor, via the :setting:`TWISTED_REACTOR` setting.

2
docs/topics/developer-tools.rst
View File

@ -142,7 +142,7 @@ a use case:
Say you want to find the ``Next`` button on the page. Type ``Next`` into the
search bar on the top right of the `Inspector`. You should get two results.
The first is a ``li`` tag with the ``class="text"``, the second the text
The first is a ``li`` tag with the ``class="next"``, the second the text
of an ``a`` tag. Right click on the ``a`` tag and select ``Scroll into View``.
If you hover over the tag, you'll see the button highlighted. From here
we could easily create a :ref:`Link Extractor <topics-link-extractors>` to

2
docs/topics/downloader-middleware.rst
View File

@ -868,7 +868,7 @@ Whether the Meta Refresh middleware will be enabled.
METAREFRESH_IGNORE_TAGS
^^^^^^^^^^^^^^^^^^^^^^^
Default: ``['script', 'noscript']``
Default: ``[]``
Meta tags within these tags are ignored.

45
docs/topics/settings.rst
View File

@ -160,27 +160,6 @@ to any particular component. In that case the module of that component will be
shown, typically an extension, middleware or pipeline. It also means that the
component must be enabled in order for the setting to have any effect.
.. setting:: ASYNCIO_REACTOR
ASYNCIO_REACTOR
---------------
Default: ``False``
Whether to install and require the Twisted reactor that uses the asyncio loop.
When this option is set to ``True``, Scrapy will require
:class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`. It will
install this reactor if no reactor is installed yet, such as when using the
``scrapy`` script or :class:`~scrapy.crawler.CrawlerProcess`. If you are using
:class:`~scrapy.crawler.CrawlerRunner`, you need to install the correct reactor
manually. If a different reactor is installed outside Scrapy, it will raise an
exception.
The default value for this option is currently ``False`` to maintain backward
compatibility and avoid possible problems caused by using a different Twisted
reactor.
.. setting:: AWS_ACCESS_KEY_ID
AWS_ACCESS_KEY_ID
@ -1463,6 +1442,30 @@ command.
The project name must not conflict with the name of custom files or directories
in the ``project`` subdirectory.
.. setting:: TWISTED_REACTOR
TWISTED_REACTOR
---------------
Default: ``None``
Import path of a given Twisted reactor, for instance:
:class:`twisted.internet.asyncioreactor.AsyncioSelectorReactor`.
Scrapy will install this reactor if no other is installed yet, such as when
the ``scrapy`` CLI program is invoked or when using the
:class:`~scrapy.crawler.CrawlerProcess` class. If you are using the
:class:`~scrapy.crawler.CrawlerRunner` class, you need to install the correct
reactor manually. An exception will be raised if the installation fails.
The default value for this option is currently ``None``, which means that Scrapy
will not attempt to install any specific reactor, and the default one defined by
Twisted for the current platform will be used. This is to maintain backward
compatibility and avoid possible problems caused by using a non-default reactor.
For additional information, please see
:doc:`core/howto/choosing-reactor`.
.. setting:: URLLENGTH_LIMIT

7
pytest.ini
View File

@ -21,6 +21,7 @@ twisted = 1
markers =
only_asyncio: marks tests as only enabled when --reactor=asyncio is passed
flake8-ignore =
W503
# Files that are only meant to provide top-level imports are expected not
# to use any of their imports:
scrapy/core/downloader/handlers/http.py F401
@ -109,7 +110,7 @@ flake8-ignore =
# scrapy/spidermiddlewares
scrapy/spidermiddlewares/httperror.py E501
scrapy/spidermiddlewares/offsite.py E501
scrapy/spidermiddlewares/referer.py E501 E129 W503 W504
scrapy/spidermiddlewares/referer.py E501 E129 W504
scrapy/spidermiddlewares/urllength.py E501
# scrapy/spiders
scrapy/spiders/__init__.py E501 E402
@ -129,13 +130,13 @@ flake8-ignore =
scrapy/utils/http.py F403 E226
scrapy/utils/httpobj.py E501
scrapy/utils/iterators.py E501 E701
scrapy/utils/log.py E128 W503
scrapy/utils/log.py E128 E501
scrapy/utils/markup.py F403
scrapy/utils/misc.py E501 E226
scrapy/utils/multipart.py F403
scrapy/utils/project.py E501
scrapy/utils/python.py E501
scrapy/utils/reactor.py E226
scrapy/utils/reactor.py E226 E501
scrapy/utils/reqser.py E501
scrapy/utils/request.py E127 E501
scrapy/utils/response.py E501 E128

19
scrapy/crawler.py
View File

@ -13,7 +13,6 @@ from scrapy.extension import ExtensionManager
from scrapy.interfaces import ISpiderLoader
from scrapy.settings import overridden_settings, Settings
from scrapy.signalmanager import SignalManager
from scrapy.utils.asyncio import install_asyncio_reactor, is_asyncio_reactor_installed
from scrapy.utils.log import (
configure_logging,
get_scrapy_root_handler,
@ -23,6 +22,7 @@ from scrapy.utils.log import (
)
from scrapy.utils.misc import create_instance, load_object
from scrapy.utils.ossignal import install_shutdown_handlers, signal_names
from scrapy.utils.reactor import install_reactor, verify_installed_reactor
logger = logging.getLogger(__name__)
@ -138,7 +138,7 @@ class CrawlerRunner:
self._crawlers = set()
self._active = set()
self.bootstrap_failed = False
self._handle_asyncio_reactor()
self._handle_twisted_reactor()
@property
def spiders(self):
@ -232,10 +232,9 @@ class CrawlerRunner:
while self._active:
yield defer.DeferredList(self._active)
def _handle_asyncio_reactor(self):
if self.settings.getbool('ASYNCIO_REACTOR') and not is_asyncio_reactor_installed():
raise Exception("ASYNCIO_REACTOR is on but the Twisted asyncio "
"reactor is not installed.")
def _handle_twisted_reactor(self):
if self.settings.get("TWISTED_REACTOR"):
verify_installed_reactor(self.settings["TWISTED_REACTOR"])
class CrawlerProcess(CrawlerRunner):
@ -324,10 +323,10 @@ class CrawlerProcess(CrawlerRunner):
except RuntimeError: # raised if already stopped or in shutdown stage
pass
def _handle_asyncio_reactor(self):
if self.settings.getbool('ASYNCIO_REACTOR'):
install_asyncio_reactor()
super()._handle_asyncio_reactor()
def _handle_twisted_reactor(self):
if self.settings.get("TWISTED_REACTOR"):
install_reactor(self.settings["TWISTED_REACTOR"])
super()._handle_twisted_reactor()
def _get_spider_loader(settings):

2
scrapy/dupefilters.py
View File

@ -49,7 +49,7 @@ class RFPDupeFilter(BaseDupeFilter):
return True
self.fingerprints.add(fp)
if self.file:
self.file.write(fp + os.linesep)
self.file.write(fp + '\n')
def request_fingerprint(self, request):
return request_fingerprint(request)

6
scrapy/settings/default_settings.py
View File

@ -19,8 +19,6 @@ from os.path import join, abspath, dirname
AJAXCRAWL_ENABLED = False
ASYNCIO_REACTOR = False
AUTOTHROTTLE_ENABLED = False
AUTOTHROTTLE_DEBUG = False
AUTOTHROTTLE_MAX_DELAY = 60.0
@ -225,7 +223,7 @@ MEMUSAGE_NOTIFY_MAIL = []
MEMUSAGE_WARNING_MB = 0
METAREFRESH_ENABLED = True
METAREFRESH_IGNORE_TAGS = ['script', 'noscript']
METAREFRESH_IGNORE_TAGS = []
METAREFRESH_MAXDELAY = 100
NEWSPIDER_MODULE = ''
@ -291,6 +289,8 @@ TELNETCONSOLE_HOST = '127.0.0.1'
TELNETCONSOLE_USERNAME = 'scrapy'
TELNETCONSOLE_PASSWORD = None
TWISTED_REACTOR = None
SPIDER_CONTRACTS = {}
SPIDER_CONTRACTS_BASE = {
'scrapy.contracts.default.UrlContract': 1,

17
scrapy/utils/asyncio.py
View File

@ -1,17 +0,0 @@
import asyncio
from contextlib import suppress
from twisted.internet import asyncioreactor
from twisted.internet.error import ReactorAlreadyInstalledError
def install_asyncio_reactor():
""" Tries to install AsyncioSelectorReactor
"""
with suppress(ReactorAlreadyInstalledError):
asyncioreactor.install(asyncio.get_event_loop())
def is_asyncio_reactor_installed():
from twisted.internet import reactor
return isinstance(reactor, asyncioreactor.AsyncioSelectorReactor)

4
scrapy/utils/defer.py
View File

@ -2,14 +2,14 @@
Helper functions for dealing with Twisted deferreds
"""
import asyncio
from functools import wraps
import inspect
from functools import wraps
from twisted.internet import defer, task
from twisted.python import failure
from scrapy.exceptions import IgnoreRequest
from scrapy.utils.asyncio import is_asyncio_reactor_installed
from scrapy.utils.reactor import is_asyncio_reactor_installed
def defer_fail(_failure):

11
scrapy/utils/log.py
View File

@ -1,17 +1,16 @@
# -*- coding: utf-8 -*-
import sys
import logging
import sys
import warnings
from logging.config import dictConfig
from twisted.python.failure import Failure
from twisted.python import log as twisted_log
from twisted.python.failure import Failure
import scrapy
from scrapy.settings import Settings
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.utils.asyncio import is_asyncio_reactor_installed
from scrapy.settings import Settings
from scrapy.utils.versions import scrapy_components_versions
@ -149,8 +148,8 @@ def log_scrapy_info(settings):
{'versions': ", ".join("%s %s" % (name, version)
for name, version in scrapy_components_versions()
if name != "Scrapy")})
if is_asyncio_reactor_installed():
logger.debug("Asyncio reactor is installed")
from twisted.internet import reactor
logger.debug("Using reactor: %s.%s", reactor.__module__, reactor.__class__.__name__)
class StreamLogger(object):

35
scrapy/utils/reactor.py
View File

@ -1,4 +1,9 @@
from twisted.internet import error
import asyncio
from contextlib import suppress
from twisted.internet import asyncioreactor, error
from scrapy.utils.misc import load_object
def listen_tcp(portrange, host, factory):
@ -42,3 +47,31 @@ class CallLaterOnce(object):
def __call__(self):
self._call = None
return self._func(*self._a, **self._kw)
def install_reactor(reactor_path):
reactor_class = load_object(reactor_path)
if reactor_class is asyncioreactor.AsyncioSelectorReactor:
with suppress(error.ReactorAlreadyInstalledError):
asyncioreactor.install(asyncio.get_event_loop())
else:
*module, _ = reactor_path.split(".")
installer_path = module + ["install"]
installer = load_object(".".join(installer_path))
with suppress(error.ReactorAlreadyInstalledError):
installer()
def verify_installed_reactor(reactor_path):
from twisted.internet import reactor
reactor_class = load_object(reactor_path)
if not isinstance(reactor, reactor_class):
msg = "The installed reactor ({}.{}) does not match the requested one ({})".format(
reactor.__module__, reactor.__class__.__name__, reactor_path
)
raise Exception(msg)
def is_asyncio_reactor_installed():
from twisted.internet import reactor
return isinstance(reactor, asyncioreactor.AsyncioSelectorReactor)

5
scrapy/utils/spider.py
View File

@ -2,14 +2,15 @@ import logging
import inspect
from scrapy.spiders import Spider
from scrapy.utils.misc import arg_to_iter
from scrapy.utils.defer import deferred_from_coro
from scrapy.utils.misc import arg_to_iter
logger = logging.getLogger(__name__)
def iterate_spider_output(result):
return arg_to_iter(result)
return arg_to_iter(deferred_from_coro(result))
def iter_spider_classes(module):

3
tests/CrawlerProcess/asyncio_enabled_no_reactor.py
View File

@ -10,8 +10,7 @@ class NoRequestsSpider(scrapy.Spider):
process = CrawlerProcess(settings={
'ASYNCIO_REACTOR': True,
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
process.crawl(NoRequestsSpider)
process.start()

3
tests/CrawlerProcess/asyncio_enabled_reactor.py
View File

@ -15,8 +15,7 @@ class NoRequestsSpider(scrapy.Spider):
process = CrawlerProcess(settings={
'ASYNCIO_REACTOR': True,
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
process.crawl(NoRequestsSpider)
process.start()

13
tests/CrawlerProcess/twisted_reactor_asyncio.py Normal file
View File

@ -0,0 +1,13 @@
import scrapy
from scrapy.crawler import CrawlerProcess
class AsyncioReactorSpider(scrapy.Spider):
name = 'asyncio_reactor'
process = CrawlerProcess(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
process.crawl(AsyncioReactorSpider)
process.start()

13
tests/CrawlerProcess/twisted_reactor_poll.py Normal file
View File

@ -0,0 +1,13 @@
import scrapy
from scrapy.crawler import CrawlerProcess
class PollReactorSpider(scrapy.Spider):
name = 'poll_reactor'
process = CrawlerProcess(settings={
"TWISTED_REACTOR": "twisted.internet.pollreactor.PollReactor",
})
process.crawl(PollReactorSpider)
process.start()

13
tests/CrawlerProcess/twisted_reactor_select.py Normal file
View File

@ -0,0 +1,13 @@
import scrapy
from scrapy.crawler import CrawlerProcess
class SelectReactorSpider(scrapy.Spider):
name = 'epoll_reactor'
process = CrawlerProcess(settings={
"TWISTED_REACTOR": "twisted.internet.selectreactor.SelectReactor",
})
process.crawl(SelectReactorSpider)
process.start()

34
tests/spiders.py
View File

@ -1,14 +1,18 @@
"""
Some spiders used for testing and benchmarking
"""
import asyncio
import time
from urllib.parse import urlencode
from twisted.internet import defer
from scrapy.http import Request
from scrapy.item import Item
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import Spider
from scrapy.spiders.crawl import CrawlSpider, Rule
from scrapy.utils.test import get_from_asyncio_queue
class MockServerSpider(Spider):
@ -83,6 +87,36 @@ class SimpleSpider(MetaSpider):
self.logger.info("Got response %d" % response.status)
class AsyncDefSpider(SimpleSpider):
name = 'asyncdef'
async def parse(self, response):
await defer.succeed(42)
self.logger.info("Got response %d" % response.status)
class AsyncDefAsyncioSpider(SimpleSpider):
name = 'asyncdef_asyncio'
async def parse(self, response):
await asyncio.sleep(0.2)
status = await get_from_asyncio_queue(response.status)
self.logger.info("Got response %d" % status)
class AsyncDefAsyncioReturnSpider(SimpleSpider):
name = 'asyncdef_asyncio_return'
async def parse(self, response):
await asyncio.sleep(0.2)
status = await get_from_asyncio_queue(response.status)
self.logger.info("Got response %d" % status)
return [{'id': 1}, {'id': 2}]
class ItemSpider(FollowAllSpider):
name = 'item'

10
tests/test_commands.py
View File

@ -296,12 +296,14 @@ class BadSpider(scrapy.Spider):
self.assertIn("badspider.py", log)
def test_asyncio_enabled_true(self):
log = self.get_log(self.debug_log_spider, args=['-s', 'ASYNCIO_REACTOR=True'])
self.assertIn("DEBUG: Asyncio reactor is installed", log)
log = self.get_log(self.debug_log_spider, args=[
'-s', 'TWISTED_REACTOR=twisted.internet.asyncioreactor.AsyncioSelectorReactor'
])
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
def test_asyncio_enabled_false(self):
log = self.get_log(self.debug_log_spider, args=['-s', 'ASYNCIO_REACTOR=False'])
self.assertNotIn("DEBUG: Asyncio reactor is installed", log)
log = self.get_log(self.debug_log_spider, args=[])
self.assertNotIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
class BenchCommandTest(CommandTest):

37
tests/test_crawl.py
View File

@ -1,16 +1,19 @@
import json
import logging
from pytest import mark
from testfixtures import LogCapture
from twisted.internet import defer
from twisted.trial.unittest import TestCase
from scrapy import signals
from scrapy.crawler import CrawlerRunner
from scrapy.http import Request
from scrapy.utils.python import to_unicode
from tests.mockserver import MockServer
from tests.spiders import (FollowAllSpider, DelaySpider, SimpleSpider, BrokenStartRequestsSpider,
SingleRequestSpider, DuplicateStartRequestsSpider, CrawlSpiderWithErrback)
SingleRequestSpider, DuplicateStartRequestsSpider, CrawlSpiderWithErrback,
AsyncDefSpider, AsyncDefAsyncioSpider, AsyncDefAsyncioReturnSpider)
class CrawlTestCase(TestCase):
@ -308,3 +311,35 @@ with multiples lines
self.assertIn("[callback] status 201", str(log))
self.assertIn("[errback] status 404", str(log))
self.assertIn("[errback] status 500", str(log))
@defer.inlineCallbacks
def test_async_def_parse(self):
self.runner.crawl(AsyncDefSpider, self.mockserver.url("/status?n=200"), mockserver=self.mockserver)
with LogCapture() as log:
yield self.runner.join()
self.assertIn("Got response 200", str(log))
@mark.only_asyncio()
@defer.inlineCallbacks
def test_async_def_asyncio_parse(self):
runner = CrawlerRunner({"ASYNCIO_REACTOR": True})
runner.crawl(AsyncDefAsyncioSpider, self.mockserver.url("/status?n=200"), mockserver=self.mockserver)
with LogCapture() as log:
yield runner.join()
self.assertIn("Got response 200", str(log))
@mark.only_asyncio()
@defer.inlineCallbacks
def test_async_def_asyncio_parse_list(self):
items = []
def _on_item_scraped(item):
items.append(item)
crawler = self.runner.create_crawler(AsyncDefAsyncioReturnSpider)
crawler.signals.connect(_on_item_scraped, signals.item_scraped)
with LogCapture() as log:
yield crawler.crawl(self.mockserver.url("/status?n=200"), mockserver=self.mockserver)
self.assertIn("Got response 200", str(log))
self.assertIn({'id': 1}, items)
self.assertIn({'id': 2}, items)

47
tests/test_crawler.py
View File

@ -254,30 +254,38 @@ class CrawlerRunnerHasSpider(unittest.TestCase):
def test_crawler_runner_asyncio_enabled_true(self):
if self.reactor_pytest == 'asyncio':
runner = CrawlerRunner(settings={'ASYNCIO_REACTOR': True})
runner = CrawlerRunner(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
else:
msg = "ASYNCIO_REACTOR is on but the Twisted asyncio reactor is not installed"
msg = r"The installed reactor \(.*?\) does not match the requested one \(.*?\)"
with self.assertRaisesRegex(Exception, msg):
runner = CrawlerRunner(settings={'ASYNCIO_REACTOR': True})
runner = CrawlerRunner(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
@defer.inlineCallbacks
def test_crawler_process_asyncio_enabled_true(self):
with LogCapture(level=logging.DEBUG) as log:
if self.reactor_pytest == 'asyncio':
runner = CrawlerProcess(settings={'ASYNCIO_REACTOR': True})
runner = CrawlerProcess(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
yield runner.crawl(NoRequestsSpider)
self.assertIn("Asyncio reactor is installed", str(log))
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", str(log))
else:
msg = "ASYNCIO_REACTOR is on but the Twisted asyncio reactor is not installed"
msg = r"The installed reactor \(.*?\) does not match the requested one \(.*?\)"
with self.assertRaisesRegex(Exception, msg):
runner = CrawlerProcess(settings={'ASYNCIO_REACTOR': True})
runner = CrawlerProcess(settings={
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
})
@defer.inlineCallbacks
def test_crawler_process_asyncio_enabled_false(self):
runner = CrawlerProcess(settings={'ASYNCIO_REACTOR': False})
runner = CrawlerProcess(settings={"TWISTED_REACTOR": None})
with LogCapture(level=logging.DEBUG) as log:
yield runner.crawl(NoRequestsSpider)
self.assertNotIn("Asyncio reactor is installed", str(log))
self.assertNotIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", str(log))
class CrawlerProcessSubprocess(unittest.TestCase):
@ -294,17 +302,17 @@ class CrawlerProcessSubprocess(unittest.TestCase):
def test_simple(self):
log = self.run_script('simple.py')
self.assertIn('Spider closed (finished)', log)
self.assertNotIn("DEBUG: Asyncio reactor is installed", log)
self.assertNotIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
def test_asyncio_enabled_no_reactor(self):
log = self.run_script('asyncio_enabled_no_reactor.py')
self.assertIn('Spider closed (finished)', log)
self.assertIn("DEBUG: Asyncio reactor is installed", log)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
def test_asyncio_enabled_reactor(self):
log = self.run_script('asyncio_enabled_reactor.py')
self.assertIn('Spider closed (finished)', log)
self.assertIn("DEBUG: Asyncio reactor is installed", log)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)
def test_ipv6_default_name_resolver(self):
log = self.run_script('default_name_resolver.py')
@ -323,3 +331,18 @@ class CrawlerProcessSubprocess(unittest.TestCase):
"'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 1," in log,
"'downloader/exception_type_count/twisted.internet.error.ConnectError': 1," in log,
]))
def test_reactor_select(self):
log = self.run_script("twisted_reactor_select.py")
self.assertIn("Spider closed (finished)", log)
self.assertIn("Using reactor: twisted.internet.selectreactor.SelectReactor", log)
def test_reactor_poll(self):
log = self.run_script("twisted_reactor_poll.py")
self.assertIn("Spider closed (finished)", log)
self.assertIn("Using reactor: twisted.internet.pollreactor.PollReactor", log)
def test_reactor_asyncio(self):
log = self.run_script("twisted_reactor_asyncio.py")
self.assertIn("Spider closed (finished)", log)
self.assertIn("Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor", log)

16
tests/test_downloadermiddleware_redirect.py
View File

@ -300,19 +300,21 @@ class MetaRefreshMiddlewareTest(unittest.TestCase):
body = ('''<noscript><meta http-equiv="refresh" '''
'''content="0;URL='http://example.org/newpage'"></noscript>''')
rsp = HtmlResponse(req.url, body=body.encode())
response = self.mw.process_response(req, rsp, self.spider)
assert isinstance(response, Response)
req2 = self.mw.process_response(req, rsp, self.spider)
assert isinstance(req2, Request)
self.assertEqual(req2.url, 'http://example.org/newpage')
def test_ignore_tags_empty_list(self):
crawler = get_crawler(Spider, {'METAREFRESH_IGNORE_TAGS': []})
def test_ignore_tags_1_x_list(self):
"""Test that Scrapy 1.x behavior remains possible"""
settings = {'METAREFRESH_IGNORE_TAGS': ['script', 'noscript']}
crawler = get_crawler(Spider, settings)
mw = MetaRefreshMiddleware.from_crawler(crawler)
req = Request(url='http://example.org')
body = ('''<noscript><meta http-equiv="refresh" '''
'''content="0;URL='http://example.org/newpage'"></noscript>''')
rsp = HtmlResponse(req.url, body=body.encode())
req2 = mw.process_response(req, rsp, self.spider)
assert isinstance(req2, Request)
self.assertEqual(req2.url, 'http://example.org/newpage')
response = mw.process_response(req, rsp, self.spider)
assert isinstance(response, Response)
if __name__ == "__main__":
unittest.main()

48
tests/test_dupefilters.py
View File

@ -2,6 +2,8 @@ import hashlib
import tempfile
import unittest
import shutil
import os
import sys
from testfixtures import LogCapture
from scrapy.dupefilters import RFPDupeFilter
@ -84,17 +86,21 @@ class RFPDupeFilterTest(unittest.TestCase):
path = tempfile.mkdtemp()
try:
df = RFPDupeFilter(path)
df.open()
assert not df.request_seen(r1)
assert df.request_seen(r1)
df.close('finished')
try:
df.open()
assert not df.request_seen(r1)
assert df.request_seen(r1)
finally:
df.close('finished')
df2 = RFPDupeFilter(path)
df2.open()
assert df2.request_seen(r1)
assert not df2.request_seen(r2)
assert df2.request_seen(r2)
df2.close('finished')
try:
df2.open()
assert df2.request_seen(r1)
assert not df2.request_seen(r2)
assert df2.request_seen(r2)
finally:
df2.close('finished')
finally:
shutil.rmtree(path)
@ -129,6 +135,30 @@ class RFPDupeFilterTest(unittest.TestCase):
case_insensitive_dupefilter.close('finished')
def test_seenreq_newlines(self):
""" Checks against adding duplicate \r to
line endings on Windows platforms. """
r1 = Request('http://scrapytest.org/1')
path = tempfile.mkdtemp()
try:
df = RFPDupeFilter(path)
df.open()
df.request_seen(r1)
df.close('finished')
with open(os.path.join(path, 'requests.seen'), 'rb') as seen_file:
line = next(seen_file).decode()
assert not line.endswith('\r\r\n')
if sys.platform == 'win32':
assert line.endswith('\r\n')
else:
assert line.endswith('\n')
finally:
shutil.rmtree(path)
def test_log(self):
with LogCapture() as l:
settings = {'DUPEFILTER_DEBUG': False,

4
tests/test_utils_asyncio.py
View File

@ -2,7 +2,7 @@ from unittest import TestCase
from pytest import mark
from scrapy.utils.asyncio import is_asyncio_reactor_installed, install_asyncio_reactor
from scrapy.utils.reactor import is_asyncio_reactor_installed, install_reactor
@mark.usefixtures('reactor_pytest')
@ -14,4 +14,4 @@ class AsyncioTest(TestCase):
def test_install_asyncio_reactor(self):
# this should do nothing
install_asyncio_reactor()
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")

1
tox.ini
View File

@ -56,7 +56,6 @@ deps =
pyOpenSSL==16.2.0
queuelib==1.4.2
service_identity==16.0.0
six==1.10.0
Twisted==17.9.0
w3lib==1.17.0
zope.interface==4.1.3