RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into json_request

This commit is contained in:
kasun Herath 2019-01-14 23:03:21 +05:30
parent 3f914f6d8c 094dde6fdb
commit f3813e376c
28 changed files with 446 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.travis.yml
View File

@ -1,5 +1,4 @@
language: python
sudo: false
branches:
only:
- master

11
MANIFEST.in
View File

@ -3,13 +3,24 @@ include AUTHORS
include INSTALL
include LICENSE
include MANIFEST.in
include NEWS
include scrapy/VERSION
include scrapy/mime.types
include codecov.yml
include conftest.py
include pytest.ini
include requirements-*.txt
include tox.ini
recursive-include scrapy/templates *
recursive-include scrapy license.txt
recursive-include docs *
prune docs/build
recursive-include extras *
recursive-include bin *
recursive-include tests *
global-exclude __pycache__ *.py[cod]

31
docs/intro/install.rst
View File

@ -30,7 +30,8 @@ dependencies depending on your operating system, so be sure to check the
We strongly recommend that you install Scrapy in :ref:`a dedicated virtualenv <intro-using-virtualenv>`,
to avoid conflicting with your system packages.
For more detailed and platform specifics instructions, read on.
For more detailed and platform specifics instructions, as well as
troubleshooting information, read on.
Things that are good to know
@ -247,6 +248,34 @@ that setuptools was unable to pick up one PyPy-specific dependency.
To fix this issue, run ``pip install 'PyPyDispatcher>=2.1.0'``.
.. _intro-install-troubleshooting:
Troubleshooting
===============
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
----------------------------------------------------------------
After you install or upgrade Scrapy, Twisted or pyOpenSSL, you may get an
exception with the following traceback::
[…]
File "[…]/site-packages/twisted/protocols/tls.py", line 63, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "[…]/site-packages/twisted/internet/_sslverify.py", line 38, in <module>
TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
The reason you get this exception is that your system or virtual environment
has a version of pyOpenSSL that your version of Twisted does not support.
To install a version of pyOpenSSL that your version of Twisted supports,
reinstall Twisted with the :code:`tls` extra option::
pip install twisted[tls]
For details, see `Issue #2473 <https://github.com/scrapy/scrapy/issues/2473>`_.
.. _Python: https://www.python.org/
.. _pip: https://pip.pypa.io/en/latest/installing/
.. _lxml: http://lxml.de/

2
docs/intro/overview.rst
View File

@ -26,7 +26,7 @@ http://quotes.toscrape.com, following the pagination::
class QuotesSpider(scrapy.Spider):
name = "quotes"
name = 'quotes'
start_urls = [
'http://quotes.toscrape.com/tag/humor/',
]

29
docs/topics/commands.rst
View File

@ -37,7 +37,7 @@ Scrapy also understands, and can be configured through, a number of environment
variables. Currently these are:
* ``SCRAPY_SETTINGS_MODULE`` (see :ref:`topics-settings-module-envvar`)
* ``SCRAPY_PROJECT``
* ``SCRAPY_PROJECT`` (see :ref:`topics-project-envvar`)
* ``SCRAPY_PYTHON_SHELL`` (see :ref:`topics-shell`)
.. _topics-project-structure:
@ -71,6 +71,33 @@ the project settings. Here is an example::
[settings]
default = myproject.settings
.. _topics-project-envvar:
Sharing the root directory between projects
===========================================
A project root directory, the one that contains the ``scrapy.cfg``, may be
shared by multiple Scrapy projects, each with its own settings module.
In that case, you must define one or more aliases for those settings modules
under ``[settings]`` in your ``scrapy.cfg`` file::
[settings]
default = myproject1.settings
project1 = myproject1.settings
project2 = myproject2.settings
By default, the ``scrapy`` command-line tool will use the ``default`` settings.
Use the ``SCRAPY_PROJECT`` environment variable to specify a different project
for ``scrapy`` to use::
$ scrapy settings --get BOT_NAME
Project 1 Bot
$ export SCRAPY_PROJECT=project2
$ scrapy settings --get BOT_NAME
Project 2 Bot
Using the ``scrapy`` tool
=========================

29
docs/topics/settings.rst
View File

@ -178,35 +178,48 @@ Default: ``None``
The AWS secret key used by code that requires access to `Amazon Web services`_,
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
.. setting:: BOT_NAME
.. setting:: AWS_ENDPOINT_URL
AWS_ENDPOINT_URL
----------------
Default: ``None``
Endpoint URL used for S3-like self-hosted storage. Storage like Minio or s3.scality.
Endpoint URL used for S3-like storage, for example Minio or s3.scality.
Only supported with ``botocore`` library.
.. setting:: AWS_ENDPOINT_URL
.. setting:: AWS_USE_SSL
AWS_USE_SSL
-----------
Default: ``None``
Use this option if you want to disable SSL connection for communication with S3 or S3-like storage.
By default SSL will be used.
Use this option if you want to disable SSL connection for communication with
S3 or S3-like storage. By default SSL will be used.
Only supported with ``botocore`` library.
.. setting:: AWS_USE_SSL
.. setting:: AWS_VERIFY
AWS_VERIFY
----------
Default: ``None``
Verify SSL connection between Scrapy and S3 or S3-like storage. By default SSL verification will occur.
Verify SSL connection between Scrapy and S3 or S3-like storage. By default
SSL verification will occur. Only supported with ``botocore`` library.
.. setting:: AWS_VERIFY
.. setting:: AWS_REGION_NAME
AWS_REGION_NAME
---------------
Default: ``None``
The name of the region associated with the AWS client.
Only supported with ``botocore`` library.
.. setting:: BOT_NAME
BOT_NAME
--------

44
docs/topics/spiders.rst
View File

@ -680,6 +680,50 @@ SitemapSpider
Default is ``sitemap_alternate_links`` disabled.
.. method:: sitemap_filter(entries)
This is a filter funtion that could be overridden to select sitemap entries
based on their attributes.
For example::
<url>
<loc>http://example.com/</loc>
<lastmod>2005-01-01</lastmod>
</url>
We can define a ``sitemap_filter`` function to filter ``entries`` by date::
from datetime import datetime
from scrapy.spiders import SitemapSpider
class FilteredSitemapSpider(SitemapSpider):
name = 'filtered_sitemap_spider'
allowed_domains = ['example.com']
sitemap_urls = ['http://example.com/sitemap.xml']
def sitemap_filter(self, entries):
for entry in entries:
date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d')
if date_time.year >= 2005:
yield entry
This would retrieve only ``entries`` modified on 2005 and the following
years.
Entries are dict objects extracted from the sitemap document.
Usually, the key is the tag name and the value is the text inside it.
It's important to notice that:
- as the loc attribute is required, entries without this tag are discarded
- alternate links are stored in a list with the key ``alternate``
(see ``sitemap_alternate_links``)
- namespaces are removed, so lxml tags named as ``{namespace}tagname`` become only ``tagname``
If you omit this method, all entries found in sitemaps will be
processed, observing other attributes and their settings.
SitemapSpider examples
~~~~~~~~~~~~~~~~~~~~~~

52
docs/topics/telnetconsole.rst
View File

@ -16,6 +16,17 @@ The telnet console is a :ref:`built-in Scrapy extension
disable it if you want. For more information about the extension itself see
:ref:`topics-extensions-ref-telnetconsole`.
.. warning::
It is not secure to use telnet console via public networks, as telnet
doesn't provide any transport-layer security. Having username/password
authentication doesn't change that.
Intended usage is connecting to a running Scrapy spider locally
(spider process and telnet client are on the same machine)
or over a secure connection (VPN, SSH tunnel).
Please avoid using telnet console over insecure connections,
or disable it completely using :setting:`TELNETCONSOLE_ENABLED` option.
.. highlight:: none
How to access the telnet console
@ -26,8 +37,26 @@ The telnet console listens in the TCP port defined in the
the console you need to type::
telnet localhost 6023
Trying localhost...
Connected to localhost.
Escape character is '^]'.
Username:
Password:
>>>
By default Username is ``scrapy`` and Password is autogenerated. The
autogenerated Password can be seen on scrapy logs like the example bellow::
2018-10-16 14:35:21 [scrapy.extensions.telnet] INFO: Telnet Password: 16f92501e8a59326
Default Username and Password can be overriden by the settings
:setting:`TELNETCONSOLE_USERNAME` and :setting:`TELNETCONSOLE_PASSWORD`.
.. warning::
Username and password provide only a limited protection, as telnet
is not using secure transport - by default traffic is not encrypted
even if username and password are set.
You need the telnet program which comes installed by default in Windows, and
most Linux distros.
@ -160,3 +189,24 @@ Default: ``'127.0.0.1'``
The interface the telnet console should listen on
.. setting:: TELNETCONSOLE_USERNAME
TELNETCONSOLE_USERNAME
----------------------
Default: ``'scrapy'``
The username used for the telnet console
.. setting:: TELNETCONSOLE_PASSWORD
TELNETCONSOLE_PASSWORD
----------------------
Default: ``None``
The password used for the telnet console, default behaviour is to have it
autogenerated

3
scrapy/commands/shell.py
View File

@ -28,7 +28,8 @@ class Command(ScrapyCommand):
return "Interactive scraping console"
def long_desc(self):
return "Interactive console for scraping the given url"
return ("Interactive console for scraping the given url or file. "
"Use ./file.html syntax or full path for local file.")
def add_options(self, parser):
ScrapyCommand.add_options(self, parser)

10
scrapy/core/downloader/handlers/__init__.py
View File

@ -24,6 +24,7 @@ class DownloadHandlers(object):
crawler.settings.getwithbase('DOWNLOAD_HANDLERS'))
for scheme, clspath in six.iteritems(handlers):
self._schemes[scheme] = clspath
self._load_handler(scheme, skip_lazy=True)
crawler.signals.connect(self._close, signals.engine_stopped)
@ -39,9 +40,14 @@ class DownloadHandlers(object):
self._notconfigured[scheme] = 'no handler available for that scheme'
return None
return self._load_handler(scheme)
def _load_handler(self, scheme, skip_lazy=False):
path = self._schemes[scheme]
try:
dhcls = load_object(path)
if skip_lazy and getattr(dhcls, 'lazy', True):
return None
dh = dhcls(self._crawler.settings)
except NotConfigured as ex:
self._notconfigured[scheme] = str(ex)
@ -49,12 +55,12 @@ class DownloadHandlers(object):
except Exception as ex:
logger.error('Loading "%(clspath)s" for scheme "%(scheme)s"',
{"clspath": path, "scheme": scheme},
exc_info=True, extra={'crawler': self._crawler})
exc_info=True, extra={'crawler': self._crawler})
self._notconfigured[scheme] = str(ex)
return None
else:
self._handlers[scheme] = dh
return self._handlers[scheme]
return dh
def download_request(self, request, spider):
scheme = urlparse_cached(request).scheme

2
scrapy/core/downloader/handlers/datauri.py
View File

@ -6,6 +6,8 @@ from scrapy.utils.decorators import defers
class DataURIDownloadHandler(object):
lazy = False
def __init__(self, settings):
super(DataURIDownloadHandler, self).__init__()

2
scrapy/core/downloader/handlers/file.py
View File

@ -2,7 +2,9 @@ from w3lib.url import file_uri_to_path
from scrapy.responsetypes import responsetypes
from scrapy.utils.decorators import defers
class FileDownloadHandler(object):
lazy = False
def __init__(self, settings):
pass

3
scrapy/core/downloader/handlers/ftp.py
View File

@ -60,7 +60,10 @@ class ReceivedDataProtocol(Protocol):
self.body.close() if self.filename else self.body.seek(0)
_CODE_RE = re.compile("\d+")
class FTPDownloadHandler(object):
lazy = False
CODE_MAPPING = {
"550": 404,

1
scrapy/core/downloader/handlers/http10.py
View File

@ -6,6 +6,7 @@ from scrapy.utils.python import to_unicode
class HTTP10DownloadHandler(object):
lazy = False
def __init__(self, settings):
self.HTTPClientFactory = load_object(settings['DOWNLOADER_HTTPCLIENTFACTORY'])

1
scrapy/core/downloader/handlers/http11.py
View File

@ -33,6 +33,7 @@ logger = logging.getLogger(__name__)
class HTTP11DownloadHandler(object):
lazy = False
def __init__(self, settings):
self._pool = HTTPConnectionPool(reactor, persistent=True)

4
scrapy/core/downloader/middleware.py
View File

@ -26,9 +26,9 @@ class DownloaderMiddlewareManager(MiddlewareManager):
if hasattr(mw, 'process_request'):
self.methods['process_request'].append(mw.process_request)
if hasattr(mw, 'process_response'):
self.methods['process_response'].insert(0, mw.process_response)
self.methods['process_response'].appendleft(mw.process_response)
if hasattr(mw, 'process_exception'):
self.methods['process_exception'].insert(0, mw.process_exception)
self.methods['process_exception'].appendleft(mw.process_exception)
def download(self, download_func, request, spider):
@defer.inlineCallbacks

6
scrapy/core/spidermw.py
View File

@ -25,11 +25,11 @@ class SpiderMiddlewareManager(MiddlewareManager):
if hasattr(mw, 'process_spider_input'):
self.methods['process_spider_input'].append(mw.process_spider_input)
if hasattr(mw, 'process_spider_output'):
self.methods['process_spider_output'].insert(0, mw.process_spider_output)
self.methods['process_spider_output'].appendleft(mw.process_spider_output)
if hasattr(mw, 'process_spider_exception'):
self.methods['process_spider_exception'].insert(0, mw.process_spider_exception)
self.methods['process_spider_exception'].appendleft(mw.process_spider_exception)
if hasattr(mw, 'process_start_requests'):
self.methods['process_start_requests'].insert(0, mw.process_start_requests)
self.methods['process_start_requests'].appendleft(mw.process_start_requests)
def scrape_response(self, scrape_func, response, request, spider):
fname = lambda f:'%s.%s' % (

2
scrapy/downloadermiddlewares/httpproxy.py
View File

@ -30,7 +30,7 @@ class HttpProxyMiddleware(object):
user_pass = to_bytes(
'%s:%s' % (unquote(username), unquote(password)),
encoding=self.auth_encoding)
return base64.b64encode(user_pass).strip()
return base64.b64encode(user_pass)
def _get_proxy(self, url, orig_type):
proxy_type, user, password, hostport = _parse_proxy(url)

36
scrapy/extensions/telnet.py
View File

@ -7,6 +7,8 @@ See documentation in docs/topics/telnetconsole.rst
import pprint
import logging
import traceback
import binascii
import os
from twisted.internet import protocol
try:
@ -22,6 +24,7 @@ from scrapy import signals
from scrapy.utils.trackref import print_live_refs
from scrapy.utils.engine import print_engine_status
from scrapy.utils.reactor import listen_tcp
from scrapy.utils.decorators import defers
try:
import guppy
@ -49,6 +52,13 @@ class TelnetConsole(protocol.ServerFactory):
self.noisy = False
self.portrange = [int(x) for x in crawler.settings.getlist('TELNETCONSOLE_PORT')]
self.host = crawler.settings['TELNETCONSOLE_HOST']
self.username = crawler.settings['TELNETCONSOLE_USERNAME']
self.password = crawler.settings['TELNETCONSOLE_PASSWORD']
if not self.password:
self.password = binascii.hexlify(os.urandom(8)).decode('utf8')
logger.info('Telnet Password: %s', self.password)
self.crawler.signals.connect(self.start_listening, signals.engine_started)
self.crawler.signals.connect(self.stop_listening, signals.engine_stopped)
@ -67,9 +77,25 @@ class TelnetConsole(protocol.ServerFactory):
self.port.stopListening()
def protocol(self):
telnet_vars = self._get_telnet_vars()
return telnet.TelnetTransport(telnet.TelnetBootstrapProtocol,
insults.ServerProtocol, manhole.Manhole, telnet_vars)
class Portal:
"""An implementation of IPortal"""
@defers
def login(self_, credentials, mind, *interfaces):
if not (credentials.username == self.username.encode('utf8') and
credentials.checkPassword(self.password.encode('utf8'))):
raise ValueError("Invalid credentials")
protocol = telnet.TelnetBootstrapProtocol(
insults.ServerProtocol,
manhole.Manhole,
self._get_telnet_vars()
)
return (interfaces[0], protocol, lambda: None)
return telnet.TelnetTransport(
telnet.AuthenticatingTelnetProtocol,
Portal()
)
def _get_telnet_vars(self):
# Note: if you add entries here also update topics/telnetconsole.rst
@ -85,8 +111,8 @@ class TelnetConsole(protocol.ServerFactory):
'p': pprint.pprint,
'prefs': print_live_refs,
'hpy': hpy,
'help': "This is Scrapy telnet console. For more info see: " \
"https://doc.scrapy.org/en/latest/topics/telnetconsole.html",
'help': "This is Scrapy telnet console. For more info see: "
"https://doc.scrapy.org/en/latest/topics/telnetconsole.html",
}
self.crawler.signals.send_catch_log(update_telnet_vars, telnet_vars=telnet_vars)
return telnet_vars

6
scrapy/middleware.py
View File

@ -1,4 +1,4 @@
from collections import defaultdict
from collections import defaultdict, deque
import logging
import pprint
@ -16,7 +16,7 @@ class MiddlewareManager(object):
def __init__(self, *middlewares):
self.middlewares = middlewares
self.methods = defaultdict(list)
self.methods = defaultdict(deque)
for mw in middlewares:
self._add_middleware(mw)
@ -56,7 +56,7 @@ class MiddlewareManager(object):
if hasattr(mw, 'open_spider'):
self.methods['open_spider'].append(mw.open_spider)
if hasattr(mw, 'close_spider'):
self.methods['close_spider'].insert(0, mw.close_spider)
self.methods['close_spider'].appendleft(mw.close_spider)
def _process_parallel(self, methodname, obj, *args):
return process_parallel(self.methods[methodname], obj, *args)

2
scrapy/settings/default_settings.py
View File

@ -277,6 +277,8 @@ USER_AGENT = 'Scrapy/%s (+https://scrapy.org)' % import_module('scrapy').__versi
TELNETCONSOLE_ENABLED = 1
TELNETCONSOLE_PORT = [6023, 6073]
TELNETCONSOLE_HOST = '127.0.0.1'
TELNETCONSOLE_USERNAME = 'scrapy'
TELNETCONSOLE_PASSWORD = None
SPIDER_CONTRACTS = {}
SPIDER_CONTRACTS_BASE = {

14
scrapy/spiders/sitemap.py
View File

@ -31,6 +31,14 @@ class SitemapSpider(Spider):
for url in self.sitemap_urls:
yield Request(url, self._parse_sitemap)
def sitemap_filter(self, entries):
"""This method can be used to filter sitemap entries by their
attributes, for example, you can filter locs with lastmod greater
than a given date (see docs).
"""
for entry in entries:
yield entry
def _parse_sitemap(self, response):
if response.url.endswith('/robots.txt'):
for url in sitemap_urls_from_robots(response.text, base_url=response.url):
@ -43,12 +51,14 @@ class SitemapSpider(Spider):
return
s = Sitemap(body)
it = self.sitemap_filter(s)
if s.type == 'sitemapindex':
for loc in iterloc(s, self.sitemap_alternate_links):
for loc in iterloc(it, self.sitemap_alternate_links):
if any(x.search(loc) for x in self._follow):
yield Request(loc, callback=self._parse_sitemap)
elif s.type == 'urlset':
for loc in iterloc(s, self.sitemap_alternate_links):
for loc in iterloc(it, self.sitemap_alternate_links):
for r, c in self._cbs:
if r.search(loc):
yield Request(loc, callback=c)

1
setup.py
View File

@ -56,6 +56,7 @@ setup(
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
'Topic :: Internet :: WWW/HTTP',

27
tests/test_downloader_handlers.py
View File

@ -41,13 +41,23 @@ from scrapy.exceptions import NotConfigured
from tests.mockserver import MockServer, ssl_context_factory, Echo
from tests.spiders import SingleRequestSpider
class DummyDH(object):
lazy = False
def __init__(self, crawler):
pass
class DummyLazyDH(object):
# Default is lazy for backwards compatibility
def __init__(self, crawler):
pass
class OffDH(object):
lazy = False
def __init__(self, crawler):
raise NotConfigured
@ -60,8 +70,6 @@ class LoadTestCase(unittest.TestCase):
crawler = get_crawler(settings_dict={'DOWNLOAD_HANDLERS': handlers})
dh = DownloadHandlers(crawler)
self.assertIn('scheme', dh._schemes)
for scheme in handlers: # force load handlers
dh._get_handler(scheme)
self.assertIn('scheme', dh._handlers)
self.assertNotIn('scheme', dh._notconfigured)
@ -70,8 +78,6 @@ class LoadTestCase(unittest.TestCase):
crawler = get_crawler(settings_dict={'DOWNLOAD_HANDLERS': handlers})
dh = DownloadHandlers(crawler)
self.assertIn('scheme', dh._schemes)
for scheme in handlers: # force load handlers
dh._get_handler(scheme)
self.assertNotIn('scheme', dh._handlers)
self.assertIn('scheme', dh._notconfigured)
@ -80,11 +86,22 @@ class LoadTestCase(unittest.TestCase):
crawler = get_crawler(settings_dict={'DOWNLOAD_HANDLERS': handlers})
dh = DownloadHandlers(crawler)
self.assertNotIn('scheme', dh._schemes)
for scheme in handlers: # force load handlers
for scheme in handlers: # force load handlers
dh._get_handler(scheme)
self.assertNotIn('scheme', dh._handlers)
self.assertIn('scheme', dh._notconfigured)
def test_lazy_handlers(self):
handlers = {'scheme': 'tests.test_downloader_handlers.DummyLazyDH'}
crawler = get_crawler(settings_dict={'DOWNLOAD_HANDLERS': handlers})
dh = DownloadHandlers(crawler)
self.assertIn('scheme', dh._schemes)
self.assertNotIn('scheme', dh._handlers)
for scheme in handlers: # force load lazy handler
dh._get_handler(scheme)
self.assertIn('scheme', dh._handlers)
self.assertNotIn('scheme', dh._notconfigured)
class FileTestCase(unittest.TestCase):

60
tests/test_extension_telnet.py Normal file
View File

@ -0,0 +1,60 @@
try:
import unittest.mock as mock
except ImportError:
import mock
from twisted.trial import unittest
from twisted.conch.telnet import ITelnetProtocol
from twisted.cred import credentials
from twisted.internet import defer
from scrapy.extensions.telnet import TelnetConsole, logger
from scrapy.utils.test import get_crawler
class TelnetExtensionTest(unittest.TestCase):
def _get_console_and_portal(self, settings=None):
crawler = get_crawler(settings_dict=settings)
console = TelnetConsole(crawler)
username = console.username
password = console.password
# This function has some side effects we don't need for this test
console._get_telnet_vars = lambda: {}
console.start_listening()
protocol = console.protocol()
portal = protocol.protocolArgs[0]
return console, portal
@defer.inlineCallbacks
def test_bad_credentials(self):
console, portal = self._get_console_and_portal()
creds = credentials.UsernamePassword(b'username', b'password')
d = portal.login(creds, None, ITelnetProtocol)
yield self.assertFailure(d, ValueError)
console.stop_listening()
@defer.inlineCallbacks
def test_good_credentials(self):
console, portal = self._get_console_and_portal()
creds = credentials.UsernamePassword(
console.username.encode('utf8'),
console.password.encode('utf8')
)
d = portal.login(creds, None, ITelnetProtocol)
yield d
console.stop_listening()
@defer.inlineCallbacks
def test_custom_credentials(self):
settings = {
'TELNETCONSOLE_USERNAME': 'user',
'TELNETCONSOLE_PASSWORD': 'pass',
}
console, portal = self._get_console_and_portal(settings=settings)
creds = credentials.UsernamePassword(b'user', b'pass')
d = portal.login(creds, None, ITelnetProtocol)
yield d
console.stop_listening()

6
tests/test_middleware.py
View File

@ -60,9 +60,9 @@ class MiddlewareManagerTest(unittest.TestCase):
def test_init(self):
m1, m2, m3 = M1(), M2(), M3()
mwman = TestMiddlewareManager(m1, m2, m3)
self.assertEqual(mwman.methods['open_spider'], [m1.open_spider, m2.open_spider])
self.assertEqual(mwman.methods['close_spider'], [m2.close_spider, m1.close_spider])
self.assertEqual(mwman.methods['process'], [m1.process, m3.process])
self.assertEqual(list(mwman.methods['open_spider']), [m1.open_spider, m2.open_spider])
self.assertEqual(list(mwman.methods['close_spider']), [m2.close_spider, m1.close_spider])
self.assertEqual(list(mwman.methods['process']), [m1.process, m3.process])
def test_methods(self):
mwman = TestMiddlewareManager(M1(), M2(), M3())

98
tests/test_spider.py
View File

@ -375,6 +375,104 @@ Sitemap: /sitemap-relative-url.xml
'http://www.example.com/schweiz-deutsch/',
'http://www.example.com/italiano/'])
def test_sitemap_filter(self):
sitemap = b"""<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
<loc>http://www.example.com/english/</loc>
<lastmod>2010-01-01</lastmod>
</url>
<url>
<loc>http://www.example.com/portuguese/</loc>
<lastmod>2005-01-01</lastmod>
</url>
</urlset>"""
class FilteredSitemapSpider(self.spider_class):
def sitemap_filter(self, entries):
from datetime import datetime
for entry in entries:
date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d')
if date_time.year > 2008:
yield entry
r = TextResponse(url="http://www.example.com/sitemap.xml", body=sitemap)
spider = self.spider_class("example.com")
self.assertEqual([req.url for req in spider._parse_sitemap(r)],
['http://www.example.com/english/',
'http://www.example.com/portuguese/'])
spider = FilteredSitemapSpider("example.com")
self.assertEqual([req.url for req in spider._parse_sitemap(r)],
['http://www.example.com/english/'])
def test_sitemap_filter_with_alternate_links(self):
sitemap = b"""<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
<loc>http://www.example.com/english/article_1/</loc>
<lastmod>2010-01-01</lastmod>
<xhtml:link rel="alternate" hreflang="de"
href="http://www.example.com/deutsch/article_1/"/>
</url>
<url>
<loc>http://www.example.com/english/article_2/</loc>
<lastmod>2015-01-01</lastmod>
</url>
</urlset>"""
class FilteredSitemapSpider(self.spider_class):
def sitemap_filter(self, entries):
for entry in entries:
alternate_links = entry.get('alternate', tuple())
for link in alternate_links:
if '/deutsch/' in link:
entry['loc'] = link
yield entry
r = TextResponse(url="http://www.example.com/sitemap.xml", body=sitemap)
spider = self.spider_class("example.com")
self.assertEqual([req.url for req in spider._parse_sitemap(r)],
['http://www.example.com/english/article_1/',
'http://www.example.com/english/article_2/'])
spider = FilteredSitemapSpider("example.com")
self.assertEqual([req.url for req in spider._parse_sitemap(r)],
['http://www.example.com/deutsch/article_1/'])
def test_sitemapindex_filter(self):
sitemap = b"""<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>http://www.example.com/sitemap1.xml</loc>
<lastmod>2004-01-01T20:00:00+00:00</lastmod>
</sitemap>
<sitemap>
<loc>http://www.example.com/sitemap2.xml</loc>
<lastmod>2005-01-01</lastmod>
</sitemap>
</sitemapindex>"""
class FilteredSitemapSpider(self.spider_class):
def sitemap_filter(self, entries):
from datetime import datetime
for entry in entries:
date_time = datetime.strptime(entry['lastmod'].split('T')[0], '%Y-%m-%d')
if date_time.year > 2004:
yield entry
r = TextResponse(url="http://www.example.com/sitemap.xml", body=sitemap)
spider = self.spider_class("example.com")
self.assertEqual([req.url for req in spider._parse_sitemap(r)],
['http://www.example.com/sitemap1.xml',
'http://www.example.com/sitemap2.xml'])
spider = FilteredSitemapSpider("example.com")
self.assertEqual([req.url for req in spider._parse_sitemap(r)],
['http://www.example.com/sitemap2.xml'])
class DeprecationTest(unittest.TestCase):

3
tox.ini
View File

@ -51,6 +51,9 @@ deps =
cssselect==0.9.1
zope.interface==4.1.1
-rtests/requirements-py2.txt
# Not used directly but allows boto GCE plugins to load.
# https://github.com/GoogleCloudPlatform/compute-image-packages/issues/262
google-compute-engine==2.8.12
[testenv:trunk]
basepython = python2.7