mirror of https://github.com/scrapy/scrapy.git
merge and fix
This commit is contained in:
Taito Horiuchi
commit
95881ee62b
|
|
@ -124,15 +124,6 @@ Scrapy:
|
|||
* Don't put your name in the code you contribute. Our policy is to keep
|
||||
the contributor's name in the `AUTHORS`_ file distributed with Scrapy.
|
||||
|
||||
Scrapy Contrib
|
||||
==============
|
||||
|
||||
Scrapy contrib shares a similar rationale as Django contrib, which is explained
|
||||
in `this post <https://jacobian.org/writing/what-is-django-contrib/>`_. If you
|
||||
are working on a new functionality, please follow that rationale to decide
|
||||
whether it should be a Scrapy contrib. If unsure, you can ask in
|
||||
`scrapy-users`_.
|
||||
|
||||
Documentation policies
|
||||
======================
|
||||
|
||||
|
|
|
|||
|
|
@ -7,14 +7,25 @@ Installation guide
|
|||
Installing Scrapy
|
||||
=================
|
||||
|
||||
Scrapy runs on Python 2.7 and Python 3.3 or above
|
||||
(except on Windows where Python 3 is not supported yet).
|
||||
Scrapy runs on Python 2.7 and Python 3.3 or above.
|
||||
|
||||
If you’re already familiar with installation of Python packages,
|
||||
If you're using `Anaconda`_ or `Miniconda`_, you can install the package from
|
||||
the `conda-forge`_ channel, which has up-to-date packages for Linux, Windows
|
||||
and OS X.
|
||||
|
||||
To install Scrapy using ``conda``, run::
|
||||
|
||||
conda install -c conda-forge scrapy
|
||||
|
||||
Alternatively, if you’re already familiar with installation of Python packages,
|
||||
you can install Scrapy and its dependencies from PyPI with::
|
||||
|
||||
pip install Scrapy
|
||||
|
||||
Note that sometimes this may require solving compilation issues for some Scrapy
|
||||
dependencies depending on your operating system, so be sure to check the
|
||||
:ref:`intro-install-platform-notes`.
|
||||
|
||||
We strongly recommend that you install Scrapy in :ref:`a dedicated virtualenv <intro-using-virtualenv>`,
|
||||
to avoid conflicting with your system packages.
|
||||
|
||||
|
|
@ -108,42 +119,14 @@ Platform specific installation notes
|
|||
Windows
|
||||
-------
|
||||
|
||||
* Install Python 2.7 from https://www.python.org/downloads/
|
||||
Though it's possible to install Scrapy on Windows using pip, we recommend you
|
||||
to install `Anaconda`_ or `Miniconda`_ and use the package from the
|
||||
`conda-forge`_ channel, which will avoid most installation issues.
|
||||
|
||||
You need to adjust ``PATH`` environment variable to include paths to
|
||||
the Python executable and additional scripts. The following paths need to be
|
||||
added to ``PATH``::
|
||||
Once you've installed `Anaconda`_ or `Miniconda`_, install Scrapy with::
|
||||
|
||||
C:\Python27\;C:\Python27\Scripts\;
|
||||
conda install -c conda-forge scrapy
|
||||
|
||||
To update the ``PATH`` open a Command prompt and run::
|
||||
|
||||
c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py
|
||||
|
||||
Close the command prompt window and reopen it so changes take effect, run the
|
||||
following command and check it shows the expected Python version::
|
||||
|
||||
python --version
|
||||
|
||||
* Install `pywin32` from http://sourceforge.net/projects/pywin32/
|
||||
|
||||
Be sure you download the architecture (win32 or amd64) that matches your system
|
||||
|
||||
* *(Only required for Python<2.7.9)* Install `pip`_ from
|
||||
https://pip.pypa.io/en/latest/installing/
|
||||
|
||||
Now open a Command prompt to check ``pip`` is installed correctly::
|
||||
|
||||
pip --version
|
||||
|
||||
* At this point Python 2.7 and ``pip`` package manager must be working, let's
|
||||
install Scrapy::
|
||||
|
||||
pip install Scrapy
|
||||
|
||||
.. note::
|
||||
Python 3 is not supported on Windows. This is because Scrapy core requirement Twisted does not support
|
||||
Python 3 on Windows.
|
||||
|
||||
Ubuntu 12.04 or above
|
||||
---------------------
|
||||
|
|
@ -234,27 +217,8 @@ After any of these workarounds you should be able to install Scrapy::
|
|||
pip install Scrapy
|
||||
|
||||
|
||||
Anaconda
|
||||
--------
|
||||
|
||||
|
||||
Using Anaconda is an alternative to using a virtualenv and installing with ``pip``.
|
||||
|
||||
.. note::
|
||||
|
||||
For Windows users, or if you have issues installing through ``pip``, this is
|
||||
the recommended way to install Scrapy.
|
||||
|
||||
If you already have `Anaconda`_ or `Miniconda`_ installed, the `conda-forge`_
|
||||
community have up-to-date packages for Linux, Windows and OS X.
|
||||
|
||||
To install Scrapy using ``conda``, run::
|
||||
|
||||
conda install -c conda-forge scrapy
|
||||
|
||||
.. _Python: https://www.python.org/
|
||||
.. _pip: https://pip.pypa.io/en/latest/installing/
|
||||
.. _Control Panel: https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sysdm_advancd_environmnt_addchange_variable.mspx
|
||||
.. _lxml: http://lxml.de/
|
||||
.. _parsel: https://pypi.python.org/pypi/parsel
|
||||
.. _w3lib: https://pypi.python.org/pypi/w3lib
|
||||
|
|
|
|||
|
|
@ -3,13 +3,25 @@
|
|||
Release notes
|
||||
=============
|
||||
|
||||
Scrapy 1.3.3 (2017-03-10)
|
||||
-------------------------
|
||||
|
||||
Bug fixes
|
||||
~~~~~~~~~
|
||||
|
||||
- Make ``SpiderLoader`` raise ``ImportError`` again by default for missing
|
||||
dependencies and wrong :setting:`SPIDER_MODULES`.
|
||||
These exceptions were silenced as warnings since 1.3.0.
|
||||
A new setting is introduced to toggle between warning or exception if needed ;
|
||||
see :setting:`SPIDER_LOADER_WARN_ONLY` for details.
|
||||
|
||||
Scrapy 1.3.2 (2017-02-13)
|
||||
-------------------------
|
||||
|
||||
Bug fixes
|
||||
~~~~~~~~~
|
||||
|
||||
- Preserve crequest class when converting to/from dicts (utils.reqser) (:issue:`2510`).
|
||||
- Preserve request class when converting to/from dicts (utils.reqser) (:issue:`2510`).
|
||||
- Use consistent selectors for author field in tutorial (:issue:`2551`).
|
||||
- Fix TLS compatibility in Twisted 17+ (:issue:`2558`)
|
||||
|
||||
|
|
@ -101,6 +113,12 @@ Dependencies & Cleanups
|
|||
downloader middlewares.
|
||||
|
||||
|
||||
Scrapy 1.2.3 (2017-03-03)
|
||||
-------------------------
|
||||
|
||||
- Packaging fix: disallow unsupported Twisted versions in setup.py
|
||||
|
||||
|
||||
Scrapy 1.2.2 (2016-12-06)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -229,6 +247,12 @@ Documentation
|
|||
- Add StackOverflow as a support channel (:issue:`2257`).
|
||||
|
||||
|
||||
Scrapy 1.1.4 (2017-03-03)
|
||||
-------------------------
|
||||
|
||||
- Packaging fix: disallow unsupported Twisted versions in setup.py
|
||||
|
||||
|
||||
Scrapy 1.1.3 (2016-09-22)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -501,6 +525,12 @@ Bugfixes
|
|||
to same remote host (:issue:`1912`).
|
||||
|
||||
|
||||
Scrapy 1.0.7 (2017-03-03)
|
||||
-------------------------
|
||||
|
||||
- Packaging fix: disallow unsupported Twisted versions in setup.py
|
||||
|
||||
|
||||
Scrapy 1.0.6 (2016-05-04)
|
||||
-------------------------
|
||||
|
||||
|
|
|
|||
|
|
@ -645,6 +645,12 @@ HttpCompressionMiddleware
|
|||
This middleware allows compressed (gzip, deflate) traffic to be
|
||||
sent/received from web sites.
|
||||
|
||||
This middleware also supports decoding `brotli-compressed`_ responses,
|
||||
provided `brotlipy`_ is installed.
|
||||
|
||||
.. _brotli-compressed: https://www.ietf.org/rfc/rfc7932.txt
|
||||
.. _brotlipy: https://pypi.python.org/pypi/brotlipy
|
||||
|
||||
HttpCompressionMiddleware Settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ LxmlLinkExtractor
|
|||
:synopsis: lxml's HTMLParser-based link extractors
|
||||
|
||||
|
||||
.. class:: LxmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), restrict_css=(), tags=('a', 'area'), attrs=('href',), canonicalize=True, unique=True, process_value=None, strip=True)
|
||||
.. class:: LxmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), restrict_css=(), tags=('a', 'area'), attrs=('href',), canonicalize=False, unique=True, process_value=None, strip=True)
|
||||
|
||||
LxmlLinkExtractor is the recommended link extractor with handy filtering
|
||||
options. It is implemented using lxml's robust HTMLParser.
|
||||
|
|
@ -103,7 +103,12 @@ LxmlLinkExtractor
|
|||
:type attrs: list
|
||||
|
||||
:param canonicalize: canonicalize each extracted url (using
|
||||
w3lib.url.canonicalize_url). Defaults to ``True``.
|
||||
w3lib.url.canonicalize_url). Defaults to ``False``.
|
||||
Note that canonicalize_url is meant for duplicate checking;
|
||||
it can change the URL visible at server side, so the response can be
|
||||
different for requests with canonicalized and raw URLs. If you're
|
||||
using LinkExtractor to follow links it is more robust to
|
||||
keep the default ``canonicalize=False``.
|
||||
:type canonicalize: boolean
|
||||
|
||||
:param unique: whether duplicate filtering should be applied to extracted
|
||||
|
|
|
|||
|
|
@ -322,6 +322,19 @@ By default, there are no size constraints, so all images are processed.
|
|||
|
||||
.. _topics-media-pipeline-override:
|
||||
|
||||
Allowing redirections
|
||||
---------------------
|
||||
|
||||
.. setting:: MEDIA_ALLOW_REDIRECTS
|
||||
|
||||
By default media pipelines ignore redirects, i.e. an HTTP redirection
|
||||
to a media file URL request will mean the media download is considered failed.
|
||||
|
||||
To handle media redirections, set this settings to ``True``:
|
||||
|
||||
MEDIA_ALLOW_REDIRECTS = True
|
||||
|
||||
|
||||
Extending the Media Pipelines
|
||||
=============================
|
||||
|
||||
|
|
|
|||
|
|
@ -303,7 +303,11 @@ Those are:
|
|||
* :reqmeta:`download_timeout`
|
||||
* :reqmeta:`download_maxsize`
|
||||
* :reqmeta:`download_latency`
|
||||
* :reqmeta:`download_fail_on_dataloss`
|
||||
* :reqmeta:`proxy`
|
||||
* ``ftp_user`` (See :setting:`FTP_USER` for more info)
|
||||
* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
|
||||
* :reqmeta:`referrer_policy`
|
||||
|
||||
.. reqmeta:: bindaddress
|
||||
|
||||
|
|
@ -330,6 +334,14 @@ started, i.e. HTTP message sent over the network. This meta key only becomes
|
|||
available when the response has been downloaded. While most other meta keys are
|
||||
used to control Scrapy behavior, this one is supposed to be read-only.
|
||||
|
||||
.. reqmeta:: download_fail_on_dataloss
|
||||
|
||||
download_fail_on_dataloss
|
||||
-------------------------
|
||||
|
||||
Whether or not to fail on broken responses. See:
|
||||
:setting:`DOWNLOAD_FAIL_ON_DATALOSS`.
|
||||
|
||||
.. _topics-request-response-ref-request-subclasses:
|
||||
|
||||
Request subclasses
|
||||
|
|
|
|||
|
|
@ -604,6 +604,32 @@ If you want to disable it set to 0.
|
|||
|
||||
This feature needs Twisted >= 11.1.
|
||||
|
||||
.. setting:: DOWNLOAD_FAIL_ON_DATALOSS
|
||||
|
||||
DOWNLOAD_FAIL_ON_DATALOSS
|
||||
-------------------------
|
||||
|
||||
Default: ``True``
|
||||
|
||||
Whether or not to fail on broken responses, that is, declared
|
||||
``Content-Length`` does not match content sent by the server or chunked
|
||||
response was not properly finish. If ``True``, these responses raise a
|
||||
``ResponseFailed([_DataLoss])`` error. If ``False``, these responses
|
||||
are passed through and the flag ``dataloss`` is added to the response, i.e.:
|
||||
``'dataloss' in response.flags`` is ``True``.
|
||||
|
||||
Optionally, this can be set per-request basis by using the
|
||||
:reqmeta:`download_fail_on_dataloss` Request.meta key to ``False``.
|
||||
|
||||
.. note::
|
||||
|
||||
A broken response, or data loss error, may happen under several
|
||||
circumstances, from server misconfiguration to network errors to data
|
||||
corruption. It is up to the user to decide if it makes sense to process
|
||||
broken responses considering they may contain partial or incomplete content.
|
||||
If setting:`RETRY_ENABLED` is ``True`` and this setting is set to ``True``,
|
||||
the ``ResponseFailed([_DataLoss])`` failure will be retried as usual.
|
||||
|
||||
.. setting:: DUPEFILTER_CLASS
|
||||
|
||||
DUPEFILTER_CLASS
|
||||
|
|
@ -1154,6 +1180,29 @@ Default: ``'scrapy.spiderloader.SpiderLoader'``
|
|||
The class that will be used for loading spiders, which must implement the
|
||||
:ref:`topics-api-spiderloader`.
|
||||
|
||||
.. setting:: SPIDER_LOADER_WARN_ONLY
|
||||
|
||||
SPIDER_LOADER_WARN_ONLY
|
||||
-----------------------
|
||||
|
||||
.. versionadded:: 1.3.3
|
||||
|
||||
Default: ``False``
|
||||
|
||||
By default, when scrapy tries to import spider classes from :setting:`SPIDER_MODULES`,
|
||||
it will fail loudly if there is any ``ImportError`` exception.
|
||||
But you can choose to silence this exception and turn it into a simple
|
||||
warning by setting ``SPIDER_LOADER_WARN_ONLY = True``.
|
||||
|
||||
.. note::
|
||||
Some :ref:`scrapy commands <topics-commands>` run with this setting to ``True``
|
||||
already (i.e. they will only issue a warning and will not fail)
|
||||
since they do not actually need to load spider classes to work:
|
||||
:command:`scrapy runspider <runspider>`,
|
||||
:command:`scrapy settings <settings>`,
|
||||
:command:`scrapy startproject <startproject>`,
|
||||
:command:`scrapy version <version>`.
|
||||
|
||||
.. setting:: SPIDER_MIDDLEWARES
|
||||
|
||||
SPIDER_MIDDLEWARES
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@ following methods:
|
|||
it has processed the response.
|
||||
|
||||
:meth:`process_spider_output` must return an iterable of
|
||||
:class:`~scrapy.http.Request`, dict or :class:`~scrapy.item.Item`
|
||||
:class:`~scrapy.http.Request`, dict or :class:`~scrapy.item.Item`
|
||||
objects.
|
||||
|
||||
:param response: the response which generated this output from the
|
||||
|
|
@ -328,6 +328,90 @@ Default: ``True``
|
|||
|
||||
Whether to enable referer middleware.
|
||||
|
||||
.. setting:: REFERRER_POLICY
|
||||
|
||||
REFERRER_POLICY
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
.. versionadded:: 1.4
|
||||
|
||||
Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``
|
||||
|
||||
.. reqmeta:: referrer_policy
|
||||
|
||||
`Referrer Policy`_ to apply when populating Request "Referer" header.
|
||||
|
||||
.. note::
|
||||
You can also set the Referrer Policy per request,
|
||||
using the special ``"referrer_policy"`` :ref:`Request.meta <topics-request-meta>` key,
|
||||
with the same acceptable values as for the ``REFERRER_POLICY`` setting.
|
||||
|
||||
Acceptable values for REFERRER_POLICY
|
||||
*************************************
|
||||
|
||||
- either a path to a ``scrapy.spidermiddlewares.referer.ReferrerPolicy``
|
||||
subclass — a custom policy or one of the built-in ones (see classes below),
|
||||
- or one of the standard W3C-defined string values,
|
||||
- or the special ``"scrapy-default"``.
|
||||
|
||||
======================================= ========================================================================
|
||||
String value Class name (as a string)
|
||||
======================================= ========================================================================
|
||||
``"scrapy-default"`` (default) :class:`scrapy.spidermiddlewares.referer.DefaultReferrerPolicy`
|
||||
`"no-referrer"`_ :class:`scrapy.spidermiddlewares.referer.NoReferrerPolicy`
|
||||
`"no-referrer-when-downgrade"`_ :class:`scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy`
|
||||
`"same-origin"`_ :class:`scrapy.spidermiddlewares.referer.SameOriginPolicy`
|
||||
`"origin"`_ :class:`scrapy.spidermiddlewares.referer.OriginPolicy`
|
||||
`"strict-origin"`_ :class:`scrapy.spidermiddlewares.referer.StrictOriginPolicy`
|
||||
`"origin-when-cross-origin"`_ :class:`scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy`
|
||||
`"strict-origin-when-cross-origin"`_ :class:`scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy`
|
||||
`"unsafe-url"`_ :class:`scrapy.spidermiddlewares.referer.UnsafeUrlPolicy`
|
||||
======================================= ========================================================================
|
||||
|
||||
.. autoclass:: DefaultReferrerPolicy
|
||||
.. warning::
|
||||
Scrapy's default referrer policy — just like `"no-referrer-when-downgrade"`_,
|
||||
the W3C-recommended value for browsers — will send a non-empty
|
||||
"Referer" header from any ``http(s)://`` to any ``https://`` URL,
|
||||
even if the domain is different.
|
||||
|
||||
`"same-origin"`_ may be a better choice if you want to remove referrer
|
||||
information for cross-domain requests.
|
||||
|
||||
.. autoclass:: NoReferrerPolicy
|
||||
|
||||
.. autoclass:: NoReferrerWhenDowngradePolicy
|
||||
.. note::
|
||||
"no-referrer-when-downgrade" policy is the W3C-recommended default,
|
||||
and is used by major web browsers.
|
||||
|
||||
However, it is NOT Scrapy's default referrer policy (see :class:`DefaultReferrerPolicy`).
|
||||
|
||||
.. autoclass:: SameOriginPolicy
|
||||
|
||||
.. autoclass:: OriginPolicy
|
||||
|
||||
.. autoclass:: StrictOriginPolicy
|
||||
|
||||
.. autoclass:: OriginWhenCrossOriginPolicy
|
||||
|
||||
.. autoclass:: StrictOriginWhenCrossOriginPolicy
|
||||
|
||||
.. autoclass:: UnsafeUrlPolicy
|
||||
.. warning::
|
||||
"unsafe-url" policy is NOT recommended.
|
||||
|
||||
.. _Referrer Policy: https://www.w3.org/TR/referrer-policy
|
||||
.. _"no-referrer": https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer
|
||||
.. _"no-referrer-when-downgrade": https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade
|
||||
.. _"same-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin
|
||||
.. _"origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-origin
|
||||
.. _"strict-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin
|
||||
.. _"origin-when-cross-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin
|
||||
.. _"strict-origin-when-cross-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin
|
||||
.. _"unsafe-url": https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url
|
||||
|
||||
|
||||
UrlLengthMiddleware
|
||||
-------------------
|
||||
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ def _import_file(filepath):
|
|||
class Command(ScrapyCommand):
|
||||
|
||||
requires_project = False
|
||||
default_settings = {'SPIDER_LOADER_WARN_ONLY': True}
|
||||
|
||||
def syntax(self):
|
||||
return "[options] <spider_file>"
|
||||
|
|
|
|||
|
|
@ -7,7 +7,8 @@ from scrapy.settings import BaseSettings
|
|||
class Command(ScrapyCommand):
|
||||
|
||||
requires_project = False
|
||||
default_settings = {'LOG_ENABLED': False}
|
||||
default_settings = {'LOG_ENABLED': False,
|
||||
'SPIDER_LOADER_WARN_ONLY': True}
|
||||
|
||||
def syntax(self):
|
||||
return "[options]"
|
||||
|
|
|
|||
|
|
@ -26,7 +26,8 @@ IGNORE = ignore_patterns('*.pyc', '.svn')
|
|||
class Command(ScrapyCommand):
|
||||
|
||||
requires_project = False
|
||||
default_settings = {'LOG_ENABLED': False}
|
||||
default_settings = {'LOG_ENABLED': False,
|
||||
'SPIDER_LOADER_WARN_ONLY': True}
|
||||
|
||||
def syntax(self):
|
||||
return "<project_name> [project_dir]"
|
||||
|
|
@ -118,4 +119,4 @@ class Command(ScrapyCommand):
|
|||
_templates_base_dir = self.settings['TEMPLATES_DIR'] or \
|
||||
join(scrapy.__path__[0], 'templates')
|
||||
return join(_templates_base_dir, 'project')
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -11,7 +11,8 @@ from scrapy.commands import ScrapyCommand
|
|||
|
||||
class Command(ScrapyCommand):
|
||||
|
||||
default_settings = {'LOG_ENABLED': False}
|
||||
default_settings = {'LOG_ENABLED': False,
|
||||
'SPIDER_LOADER_WARN_ONLY': True}
|
||||
|
||||
def syntax(self):
|
||||
return "[-v]"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,23 @@
|
|||
from w3lib.url import parse_data_uri
|
||||
|
||||
from scrapy.http import TextResponse
|
||||
from scrapy.responsetypes import responsetypes
|
||||
from scrapy.utils.decorators import defers
|
||||
|
||||
|
||||
class DataURIDownloadHandler(object):
|
||||
def __init__(self, settings):
|
||||
super(DataURIDownloadHandler, self).__init__()
|
||||
|
||||
@defers
|
||||
def download_request(self, request, spider):
|
||||
uri = parse_data_uri(request.url)
|
||||
respcls = responsetypes.from_mimetype(uri.media_type)
|
||||
|
||||
resp_kwargs = {}
|
||||
if (issubclass(respcls, TextResponse) and
|
||||
uri.media_type.split('/')[0] == 'text'):
|
||||
charset = uri.media_type_parameters.get('charset')
|
||||
resp_kwargs['encoding'] = charset
|
||||
|
||||
return respcls(url=request.url, body=uri.data, **resp_kwargs)
|
||||
|
|
@ -12,9 +12,9 @@ from twisted.internet import defer, reactor, protocol
|
|||
from twisted.web.http_headers import Headers as TxHeaders
|
||||
from twisted.web.iweb import IBodyProducer, UNKNOWN_LENGTH
|
||||
from twisted.internet.error import TimeoutError
|
||||
from twisted.web.http import PotentialDataLoss
|
||||
from twisted.web.http import _DataLoss, PotentialDataLoss
|
||||
from twisted.web.client import Agent, ProxyAgent, ResponseDone, \
|
||||
HTTPConnectionPool
|
||||
HTTPConnectionPool, ResponseFailed
|
||||
from twisted.internet.endpoints import TCP4ClientEndpoint
|
||||
|
||||
from scrapy.http import Headers
|
||||
|
|
@ -51,13 +51,15 @@ class HTTP11DownloadHandler(object):
|
|||
warnings.warn(msg)
|
||||
self._default_maxsize = settings.getint('DOWNLOAD_MAXSIZE')
|
||||
self._default_warnsize = settings.getint('DOWNLOAD_WARNSIZE')
|
||||
self._fail_on_dataloss = settings.getbool('DOWNLOAD_FAIL_ON_DATALOSS')
|
||||
self._disconnect_timeout = 1
|
||||
|
||||
def download_request(self, request, spider):
|
||||
"""Return a deferred for the HTTP download"""
|
||||
agent = ScrapyAgent(contextFactory=self._contextFactory, pool=self._pool,
|
||||
maxsize=getattr(spider, 'download_maxsize', self._default_maxsize),
|
||||
warnsize=getattr(spider, 'download_warnsize', self._default_warnsize))
|
||||
warnsize=getattr(spider, 'download_warnsize', self._default_warnsize),
|
||||
fail_on_dataloss=self._fail_on_dataloss)
|
||||
return agent.download_request(request)
|
||||
|
||||
def close(self):
|
||||
|
|
@ -233,13 +235,14 @@ class ScrapyAgent(object):
|
|||
_TunnelingAgent = TunnelingAgent
|
||||
|
||||
def __init__(self, contextFactory=None, connectTimeout=10, bindAddress=None, pool=None,
|
||||
maxsize=0, warnsize=0):
|
||||
maxsize=0, warnsize=0, fail_on_dataloss=True):
|
||||
self._contextFactory = contextFactory
|
||||
self._connectTimeout = connectTimeout
|
||||
self._bindAddress = bindAddress
|
||||
self._pool = pool
|
||||
self._maxsize = maxsize
|
||||
self._warnsize = warnsize
|
||||
self._fail_on_dataloss = fail_on_dataloss
|
||||
self._txresponse = None
|
||||
|
||||
def _get_agent(self, request, timeout):
|
||||
|
|
@ -326,6 +329,7 @@ class ScrapyAgent(object):
|
|||
maxsize = request.meta.get('download_maxsize', self._maxsize)
|
||||
warnsize = request.meta.get('download_warnsize', self._warnsize)
|
||||
expected_size = txresponse.length if txresponse.length != UNKNOWN_LENGTH else -1
|
||||
fail_on_dataloss = request.meta.get('download_fail_on_dataloss', self._fail_on_dataloss)
|
||||
|
||||
if maxsize and expected_size > maxsize:
|
||||
error_msg = ("Cancelling download of %(url)s: expected response "
|
||||
|
|
@ -345,7 +349,8 @@ class ScrapyAgent(object):
|
|||
txresponse._transport._producer.loseConnection()
|
||||
|
||||
d = defer.Deferred(_cancel)
|
||||
txresponse.deliverBody(_ResponseReader(d, txresponse, request, maxsize, warnsize))
|
||||
txresponse.deliverBody(_ResponseReader(
|
||||
d, txresponse, request, maxsize, warnsize, fail_on_dataloss))
|
||||
|
||||
# save response for timeouts
|
||||
self._txresponse = txresponse
|
||||
|
|
@ -380,13 +385,16 @@ class _RequestBodyProducer(object):
|
|||
|
||||
class _ResponseReader(protocol.Protocol):
|
||||
|
||||
def __init__(self, finished, txresponse, request, maxsize, warnsize):
|
||||
def __init__(self, finished, txresponse, request, maxsize, warnsize,
|
||||
fail_on_dataloss):
|
||||
self._finished = finished
|
||||
self._txresponse = txresponse
|
||||
self._request = request
|
||||
self._bodybuf = BytesIO()
|
||||
self._maxsize = maxsize
|
||||
self._warnsize = warnsize
|
||||
self._fail_on_dataloss = fail_on_dataloss
|
||||
self._fail_on_dataloss_warned = False
|
||||
self._reached_warnsize = False
|
||||
self._bytes_received = 0
|
||||
|
||||
|
|
@ -415,7 +423,22 @@ class _ResponseReader(protocol.Protocol):
|
|||
body = self._bodybuf.getvalue()
|
||||
if reason.check(ResponseDone):
|
||||
self._finished.callback((self._txresponse, body, None))
|
||||
elif reason.check(PotentialDataLoss):
|
||||
return
|
||||
|
||||
if reason.check(PotentialDataLoss):
|
||||
self._finished.callback((self._txresponse, body, ['partial']))
|
||||
else:
|
||||
self._finished.errback(reason)
|
||||
return
|
||||
|
||||
if reason.check(ResponseFailed) and any(r.check(_DataLoss) for r in reason.value.reasons):
|
||||
if not self._fail_on_dataloss:
|
||||
self._finished.callback((self._txresponse, body, ['dataloss']))
|
||||
return
|
||||
|
||||
elif not self._fail_on_dataloss_warned:
|
||||
logger.warn("Got data loss in %s. If you want to process broken "
|
||||
"responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False"
|
||||
" -- This message won't be shown in further requests",
|
||||
self._txresponse.request.absoluteURI.decode())
|
||||
self._fail_on_dataloss_warned = True
|
||||
|
||||
self._finished.errback(reason)
|
||||
|
|
|
|||
|
|
@ -218,10 +218,8 @@ class ExecutionEngine(object):
|
|||
request=request, spider=spider)
|
||||
|
||||
def download(self, request, spider):
|
||||
slot = self.slot
|
||||
slot.add_request(request)
|
||||
d = self._download(request, spider)
|
||||
d.addBoth(self._downloaded, slot, request, spider)
|
||||
d.addBoth(self._downloaded, self.slot, request, spider)
|
||||
return d
|
||||
|
||||
def _downloaded(self, response, slot, request, spider):
|
||||
|
|
|
|||
|
|
@ -16,7 +16,9 @@ from scrapy.signalmanager import SignalManager
|
|||
from scrapy.exceptions import ScrapyDeprecationWarning
|
||||
from scrapy.utils.ossignal import install_shutdown_handlers, signal_names
|
||||
from scrapy.utils.misc import load_object
|
||||
from scrapy.utils.log import LogCounterHandler, configure_logging, log_scrapy_info
|
||||
from scrapy.utils.log import (
|
||||
LogCounterHandler, configure_logging, log_scrapy_info,
|
||||
get_scrapy_root_handler, install_scrapy_root_handler)
|
||||
from scrapy import signals
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -35,8 +37,11 @@ class Crawler(object):
|
|||
self.signals = SignalManager(self)
|
||||
self.stats = load_object(self.settings['STATS_CLASS'])(self)
|
||||
|
||||
handler = LogCounterHandler(self, level=settings.get('LOG_LEVEL'))
|
||||
handler = LogCounterHandler(self, level=self.settings.get('LOG_LEVEL'))
|
||||
logging.root.addHandler(handler)
|
||||
if get_scrapy_root_handler() is not None:
|
||||
# scrapy root handler alread installed: update it with new settings
|
||||
install_scrapy_root_handler(self.settings)
|
||||
# lambda is assigned to Crawler attribute because this way it is not
|
||||
# garbage collected after leaving __init__ scope
|
||||
self.__remove_handler = lambda: logging.root.removeHandler(handler)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import zlib
|
||||
|
||||
from scrapy.utils.gz import gunzip, is_gzipped
|
||||
from scrapy.utils.gz import gunzip
|
||||
from scrapy.http import Response, TextResponse
|
||||
from scrapy.responsetypes import responsetypes
|
||||
from scrapy.exceptions import NotConfigured
|
||||
|
|
@ -34,11 +34,11 @@ class HttpCompressionMiddleware(object):
|
|||
return response
|
||||
if isinstance(response, Response):
|
||||
content_encoding = response.headers.getlist('Content-Encoding')
|
||||
if content_encoding and not is_gzipped(response):
|
||||
if content_encoding:
|
||||
encoding = content_encoding.pop()
|
||||
decoded_body = self._decode(response.body, encoding.lower())
|
||||
respcls = responsetypes.from_args(headers=response.headers, \
|
||||
url=response.url)
|
||||
url=response.url, body=decoded_body)
|
||||
kwargs = dict(cls=respcls, body=decoded_body)
|
||||
if issubclass(respcls, TextResponse):
|
||||
# force recalculating the encoding until we make sure the
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
from __future__ import print_function
|
||||
import os
|
||||
import gzip
|
||||
import logging
|
||||
from six.moves import cPickle as pickle
|
||||
from importlib import import_module
|
||||
from time import time
|
||||
|
|
@ -15,6 +16,9 @@ from scrapy.utils.httpobj import urlparse_cached
|
|||
from scrapy.utils.python import to_bytes, to_unicode
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DummyPolicy(object):
|
||||
|
||||
def __init__(self, settings):
|
||||
|
|
@ -220,6 +224,8 @@ class DbmCacheStorage(object):
|
|||
dbpath = os.path.join(self.cachedir, '%s.db' % spider.name)
|
||||
self.db = self.dbmodule.open(dbpath, 'c')
|
||||
|
||||
logger.debug("Using DBM cache storage in %(cachepath)s" % {'cachepath': dbpath}, extra={'spider': spider})
|
||||
|
||||
def close_spider(self, spider):
|
||||
self.db.close()
|
||||
|
||||
|
|
@ -272,7 +278,8 @@ class FilesystemCacheStorage(object):
|
|||
self._open = gzip.open if self.use_gzip else open
|
||||
|
||||
def open_spider(self, spider):
|
||||
pass
|
||||
logger.debug("Using filesystem cache storage in %(cachedir)s" % {'cachedir': self.cachedir},
|
||||
extra={'spider': spider})
|
||||
|
||||
def close_spider(self, spider):
|
||||
pass
|
||||
|
|
@ -348,6 +355,8 @@ class LeveldbCacheStorage(object):
|
|||
dbpath = os.path.join(self.cachedir, '%s.leveldb' % spider.name)
|
||||
self.db = self._leveldb.LevelDB(dbpath)
|
||||
|
||||
logger.debug("Using LevelDB cache storage in %(cachepath)s" % {'cachepath': dbpath}, extra={'spider': spider})
|
||||
|
||||
def close_spider(self, spider):
|
||||
# Do compactation each time to save space and also recreate files to
|
||||
# avoid them being removed in storages with timestamp-based autoremoval.
|
||||
|
|
|
|||
|
|
@ -101,7 +101,7 @@ class FilteringLinkExtractor(object):
|
|||
links = [x for x in links if self._link_allowed(x)]
|
||||
if self.canonicalize:
|
||||
for link in links:
|
||||
link.url = canonicalize_url(urlparse(link.url))
|
||||
link.url = canonicalize_url(link.url)
|
||||
links = self.link_extractor._process_links(links)
|
||||
return links
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ from six.moves.urllib.parse import urljoin
|
|||
|
||||
import lxml.etree as etree
|
||||
from w3lib.html import strip_html5_whitespace
|
||||
from w3lib.url import canonicalize_url
|
||||
|
||||
from scrapy.link import Link
|
||||
from scrapy.utils.misc import arg_to_iter, rel_has_nofollow
|
||||
|
|
@ -29,12 +30,17 @@ def _nons(tag):
|
|||
|
||||
class LxmlParserLinkExtractor(object):
|
||||
def __init__(self, tag="a", attr="href", process=None, unique=False,
|
||||
strip=True):
|
||||
strip=True, canonicalized=False):
|
||||
self.scan_tag = tag if callable(tag) else lambda t: t == tag
|
||||
self.scan_attr = attr if callable(attr) else lambda a: a == attr
|
||||
self.process_attr = process if callable(process) else lambda v: v
|
||||
self.unique = unique
|
||||
self.strip = strip
|
||||
if canonicalized:
|
||||
self.link_key = lambda link: link.url
|
||||
else:
|
||||
self.link_key = lambda link: canonicalize_url(link.url,
|
||||
keep_fragments=True)
|
||||
|
||||
def _iter_links(self, document):
|
||||
for el in document.iter(etree.Element):
|
||||
|
|
@ -82,21 +88,27 @@ class LxmlParserLinkExtractor(object):
|
|||
|
||||
def _deduplicate_if_needed(self, links):
|
||||
if self.unique:
|
||||
return unique_list(links, key=lambda link: link.url)
|
||||
return unique_list(links, key=self.link_key)
|
||||
return links
|
||||
|
||||
|
||||
class LxmlLinkExtractor(FilteringLinkExtractor):
|
||||
|
||||
def __init__(self, allow=(), deny=(), allow_domains=(), deny_domains=(), restrict_xpaths=(),
|
||||
tags=('a', 'area'), attrs=('href',), canonicalize=True,
|
||||
tags=('a', 'area'), attrs=('href',), canonicalize=False,
|
||||
unique=True, process_value=None, deny_extensions=None, restrict_css=(),
|
||||
strip=True):
|
||||
tags, attrs = set(arg_to_iter(tags)), set(arg_to_iter(attrs))
|
||||
tag_func = lambda x: x in tags
|
||||
attr_func = lambda x: x in attrs
|
||||
lx = LxmlParserLinkExtractor(tag=tag_func, attr=attr_func,
|
||||
unique=unique, process=process_value, strip=strip)
|
||||
lx = LxmlParserLinkExtractor(
|
||||
tag=tag_func,
|
||||
attr=attr_func,
|
||||
unique=unique,
|
||||
process=process_value,
|
||||
strip=strip,
|
||||
canonicalized=canonicalize
|
||||
)
|
||||
|
||||
super(LxmlLinkExtractor, self).__init__(lx, allow=allow, deny=deny,
|
||||
allow_domains=allow_domains, deny_domains=deny_domains,
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ from six.moves.urllib.parse import urljoin
|
|||
import warnings
|
||||
from sgmllib import SGMLParser
|
||||
|
||||
from w3lib.url import safe_url_string
|
||||
from w3lib.url import safe_url_string, canonicalize_url
|
||||
from w3lib.html import strip_html5_whitespace
|
||||
|
||||
from scrapy.link import Link
|
||||
|
|
@ -20,7 +20,7 @@ from scrapy.exceptions import ScrapyDeprecationWarning
|
|||
class BaseSgmlLinkExtractor(SGMLParser):
|
||||
|
||||
def __init__(self, tag="a", attr="href", unique=False, process_value=None,
|
||||
strip=True):
|
||||
strip=True, canonicalized=False):
|
||||
warnings.warn(
|
||||
"BaseSgmlLinkExtractor is deprecated and will be removed in future releases. "
|
||||
"Please use scrapy.linkextractors.LinkExtractor",
|
||||
|
|
@ -33,6 +33,11 @@ class BaseSgmlLinkExtractor(SGMLParser):
|
|||
self.current_link = None
|
||||
self.unique = unique
|
||||
self.strip = strip
|
||||
if canonicalized:
|
||||
self.link_key = lambda link: link.url
|
||||
else:
|
||||
self.link_key = lambda link: canonicalize_url(link.url,
|
||||
keep_fragments=True)
|
||||
|
||||
def _extract_links(self, response_text, response_url, response_encoding, base_url=None):
|
||||
""" Do the real extraction work """
|
||||
|
|
@ -61,8 +66,7 @@ class BaseSgmlLinkExtractor(SGMLParser):
|
|||
|
||||
The subclass should override it if necessary
|
||||
"""
|
||||
links = unique_list(links, key=lambda link: link.url) if self.unique else links
|
||||
return links
|
||||
return unique_list(links, key=self.link_key) if self.unique else links
|
||||
|
||||
def extract_links(self, response):
|
||||
# wrapper needed to allow to work directly with text
|
||||
|
|
@ -107,10 +111,9 @@ class BaseSgmlLinkExtractor(SGMLParser):
|
|||
class SgmlLinkExtractor(FilteringLinkExtractor):
|
||||
|
||||
def __init__(self, allow=(), deny=(), allow_domains=(), deny_domains=(), restrict_xpaths=(),
|
||||
tags=('a', 'area'), attrs=('href',), canonicalize=True, unique=True,
|
||||
tags=('a', 'area'), attrs=('href',), canonicalize=False, unique=True,
|
||||
process_value=None, deny_extensions=None, restrict_css=(),
|
||||
strip=True):
|
||||
|
||||
warnings.warn(
|
||||
"SgmlLinkExtractor is deprecated and will be removed in future releases. "
|
||||
"Please use scrapy.linkextractors.LinkExtractor",
|
||||
|
|
@ -124,7 +127,8 @@ class SgmlLinkExtractor(FilteringLinkExtractor):
|
|||
with warnings.catch_warnings():
|
||||
warnings.simplefilter('ignore', ScrapyDeprecationWarning)
|
||||
lx = BaseSgmlLinkExtractor(tag=tag_func, attr=attr_func,
|
||||
unique=unique, process_value=process_value, strip=strip)
|
||||
unique=unique, process_value=process_value, strip=strip,
|
||||
canonicalized=canonicalize)
|
||||
|
||||
super(SgmlLinkExtractor, self).__init__(lx, allow=allow, deny=deny,
|
||||
allow_domains=allow_domains, deny_domains=deny_domains,
|
||||
|
|
|
|||
|
|
@ -43,6 +43,8 @@ class LogFormatter(object):
|
|||
'request_flags' : request_flags,
|
||||
'referer': referer_str(request),
|
||||
'response_flags': response_flags,
|
||||
# backward compatibility with Scrapy logformatter below 1.4 version
|
||||
'flags': response_flags
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ class FSFilesStore(object):
|
|||
absolute_path = self._get_filesystem_path(path)
|
||||
try:
|
||||
last_modified = os.path.getmtime(absolute_path)
|
||||
except: # FIXME: catching everything!
|
||||
except os.error:
|
||||
return {}
|
||||
|
||||
with open(absolute_path, 'rb') as f:
|
||||
|
|
@ -249,7 +249,7 @@ class FilesPipeline(MediaPipeline):
|
|||
resolve('FILES_RESULT_FIELD'), self.FILES_RESULT_FIELD
|
||||
)
|
||||
|
||||
super(FilesPipeline, self).__init__(download_func=download_func)
|
||||
super(FilesPipeline, self).__init__(download_func=download_func, settings=settings)
|
||||
|
||||
@classmethod
|
||||
def from_settings(cls, settings):
|
||||
|
|
|
|||
|
|
@ -1,10 +1,13 @@
|
|||
from __future__ import print_function
|
||||
|
||||
import functools
|
||||
import logging
|
||||
from collections import defaultdict
|
||||
from twisted.internet.defer import Deferred, DeferredList
|
||||
from twisted.python.failure import Failure
|
||||
|
||||
from scrapy.settings import Settings
|
||||
from scrapy.utils.datatypes import SequenceExclude
|
||||
from scrapy.utils.defer import mustbe_deferred, defer_result
|
||||
from scrapy.utils.request import request_fingerprint
|
||||
from scrapy.utils.misc import arg_to_iter
|
||||
|
|
@ -24,9 +27,23 @@ class MediaPipeline(object):
|
|||
self.downloaded = {}
|
||||
self.waiting = defaultdict(list)
|
||||
|
||||
def __init__(self, download_func=None):
|
||||
def __init__(self, download_func=None, settings=None):
|
||||
self.download_func = download_func
|
||||
|
||||
if isinstance(settings, dict) or settings is None:
|
||||
settings = Settings(settings)
|
||||
resolve = functools.partial(self._key_for_pipe,
|
||||
base_class_name="MediaPipeline",
|
||||
settings=settings)
|
||||
self.allow_redirects = settings.getbool(
|
||||
resolve('MEDIA_ALLOW_REDIRECTS'), False
|
||||
)
|
||||
self._handle_statuses(self.allow_redirects)
|
||||
|
||||
def _handle_statuses(self, allow_redirects):
|
||||
self.handle_httpstatus_list = None
|
||||
if allow_redirects:
|
||||
self.handle_httpstatus_list = SequenceExclude(range(300, 400))
|
||||
|
||||
def _key_for_pipe(self, key, base_class_name=None,
|
||||
settings=None):
|
||||
|
|
@ -93,6 +110,12 @@ class MediaPipeline(object):
|
|||
)
|
||||
return dfd.addBoth(lambda _: wad) # it must return wad at last
|
||||
|
||||
def _modify_media_request(self, request):
|
||||
if self.handle_httpstatus_list:
|
||||
request.meta['handle_httpstatus_list'] = self.handle_httpstatus_list
|
||||
else:
|
||||
request.meta['handle_httpstatus_all'] = True
|
||||
|
||||
def _check_media_to_download(self, result, request, info):
|
||||
if result is not None:
|
||||
return result
|
||||
|
|
@ -103,7 +126,7 @@ class MediaPipeline(object):
|
|||
callback=self.media_downloaded, callbackArgs=(request, info),
|
||||
errback=self.media_failed, errbackArgs=(request, info))
|
||||
else:
|
||||
request.meta['handle_httpstatus_all'] = True
|
||||
self._modify_media_request(request)
|
||||
dfd = self.crawler.engine.download(request, info.spider)
|
||||
dfd.addCallbacks(
|
||||
callback=self.media_downloaded, callbackArgs=(request, info),
|
||||
|
|
|
|||
|
|
@ -67,6 +67,7 @@ DOWNLOAD_DELAY = 0
|
|||
|
||||
DOWNLOAD_HANDLERS = {}
|
||||
DOWNLOAD_HANDLERS_BASE = {
|
||||
'data': 'scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler',
|
||||
'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler',
|
||||
'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
|
||||
'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
|
||||
|
|
@ -79,6 +80,8 @@ DOWNLOAD_TIMEOUT = 180 # 3mins
|
|||
DOWNLOAD_MAXSIZE = 1024*1024*1024 # 1024m
|
||||
DOWNLOAD_WARNSIZE = 32*1024*1024 # 32m
|
||||
|
||||
DOWNLOAD_FAIL_ON_DATALOSS = True
|
||||
|
||||
DOWNLOADER = 'scrapy.core.downloader.Downloader'
|
||||
|
||||
DOWNLOADER_HTTPCLIENTFACTORY = 'scrapy.core.downloader.webclient.ScrapyHTTPClientFactory'
|
||||
|
|
@ -232,6 +235,7 @@ REDIRECT_MAX_TIMES = 20 # uses Firefox default setting
|
|||
REDIRECT_PRIORITY_ADJUST = +2
|
||||
|
||||
REFERER_ENABLED = True
|
||||
REFERRER_POLICY = 'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'
|
||||
|
||||
RETRY_ENABLED = True
|
||||
RETRY_TIMES = 2 # initial response + 2 retries = 3 requests
|
||||
|
|
@ -246,6 +250,7 @@ SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.LifoMemoryQueue'
|
|||
SCHEDULER_PRIORITY_QUEUE = 'queuelib.PriorityQueue'
|
||||
|
||||
SPIDER_LOADER_CLASS = 'scrapy.spiderloader.SpiderLoader'
|
||||
SPIDER_LOADER_WARN_ONLY = False
|
||||
|
||||
SPIDER_MIDDLEWARES = {}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
from __future__ import absolute_import
|
||||
from collections import defaultdict
|
||||
import traceback
|
||||
import warnings
|
||||
|
||||
|
|
@ -18,11 +19,26 @@ class SpiderLoader(object):
|
|||
"""
|
||||
def __init__(self, settings):
|
||||
self.spider_modules = settings.getlist('SPIDER_MODULES')
|
||||
self.warn_only = settings.getbool('SPIDER_LOADER_WARN_ONLY')
|
||||
self._spiders = {}
|
||||
self._found = defaultdict(list)
|
||||
self._load_all_spiders()
|
||||
|
||||
def _check_name_duplicates(self):
|
||||
dupes = ["\n".join(" {cls} named {name!r} (in {module})".format(
|
||||
module=mod, cls=cls, name=name)
|
||||
for (mod, cls) in locations)
|
||||
for name, locations in self._found.items()
|
||||
if len(locations)>1]
|
||||
if dupes:
|
||||
msg = ("There are several spiders with the same name:\n\n"
|
||||
"{}\n\n This can cause unexpected behavior.".format(
|
||||
"\n\n".join(dupes)))
|
||||
warnings.warn(msg, UserWarning)
|
||||
|
||||
def _load_spiders(self, module):
|
||||
for spcls in iter_spider_classes(module):
|
||||
self._found[spcls.name].append((module.__name__, spcls.__name__))
|
||||
self._spiders[spcls.name] = spcls
|
||||
|
||||
def _load_all_spiders(self):
|
||||
|
|
@ -31,10 +47,14 @@ class SpiderLoader(object):
|
|||
for module in walk_modules(name):
|
||||
self._load_spiders(module)
|
||||
except ImportError as e:
|
||||
msg = ("\n{tb}Could not load spiders from module '{modname}'. "
|
||||
"Check SPIDER_MODULES setting".format(
|
||||
modname=name, tb=traceback.format_exc()))
|
||||
warnings.warn(msg, RuntimeWarning)
|
||||
if self.warn_only:
|
||||
msg = ("\n{tb}Could not load spiders from module '{modname}'. "
|
||||
"See above traceback for details.".format(
|
||||
modname=name, tb=traceback.format_exc()))
|
||||
warnings.warn(msg, RuntimeWarning)
|
||||
else:
|
||||
raise
|
||||
self._check_name_duplicates()
|
||||
|
||||
@classmethod
|
||||
def from_settings(cls, settings):
|
||||
|
|
|
|||
|
|
@ -2,22 +2,359 @@
|
|||
RefererMiddleware: populates Request referer field, based on the Response which
|
||||
originated it.
|
||||
"""
|
||||
from six.moves.urllib.parse import urlparse
|
||||
import warnings
|
||||
|
||||
from scrapy.http import Request
|
||||
from w3lib.url import safe_url_string
|
||||
|
||||
from scrapy.http import Request, Response
|
||||
from scrapy.exceptions import NotConfigured
|
||||
from scrapy import signals
|
||||
from scrapy.utils.python import to_native_str
|
||||
from scrapy.utils.httpobj import urlparse_cached
|
||||
from scrapy.utils.misc import load_object
|
||||
from scrapy.utils.url import strip_url
|
||||
|
||||
|
||||
LOCAL_SCHEMES = ('about', 'blob', 'data', 'filesystem',)
|
||||
|
||||
POLICY_NO_REFERRER = "no-referrer"
|
||||
POLICY_NO_REFERRER_WHEN_DOWNGRADE = "no-referrer-when-downgrade"
|
||||
POLICY_SAME_ORIGIN = "same-origin"
|
||||
POLICY_ORIGIN = "origin"
|
||||
POLICY_STRICT_ORIGIN = "strict-origin"
|
||||
POLICY_ORIGIN_WHEN_CROSS_ORIGIN = "origin-when-cross-origin"
|
||||
POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN = "strict-origin-when-cross-origin"
|
||||
POLICY_UNSAFE_URL = "unsafe-url"
|
||||
POLICY_SCRAPY_DEFAULT = "scrapy-default"
|
||||
|
||||
|
||||
class ReferrerPolicy(object):
|
||||
|
||||
NOREFERRER_SCHEMES = LOCAL_SCHEMES
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
raise NotImplementedError()
|
||||
|
||||
def stripped_referrer(self, url):
|
||||
if urlparse(url).scheme not in self.NOREFERRER_SCHEMES:
|
||||
return self.strip_url(url)
|
||||
|
||||
def origin_referrer(self, url):
|
||||
if urlparse(url).scheme not in self.NOREFERRER_SCHEMES:
|
||||
return self.origin(url)
|
||||
|
||||
def strip_url(self, url, origin_only=False):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#strip-url
|
||||
|
||||
If url is null, return no referrer.
|
||||
If url's scheme is a local scheme, then return no referrer.
|
||||
Set url's username to the empty string.
|
||||
Set url's password to null.
|
||||
Set url's fragment to null.
|
||||
If the origin-only flag is true, then:
|
||||
Set url's path to null.
|
||||
Set url's query to null.
|
||||
Return url.
|
||||
"""
|
||||
if not url:
|
||||
return None
|
||||
return strip_url(url,
|
||||
strip_credentials=True,
|
||||
strip_fragment=True,
|
||||
strip_default_port=True,
|
||||
origin_only=origin_only)
|
||||
|
||||
def origin(self, url):
|
||||
"""Return serialized origin (scheme, host, path) for a request or response URL."""
|
||||
return self.strip_url(url, origin_only=True)
|
||||
|
||||
def potentially_trustworthy(self, url):
|
||||
# Note: this does not follow https://w3c.github.io/webappsec-secure-contexts/#is-url-trustworthy
|
||||
parsed_url = urlparse(url)
|
||||
if parsed_url.scheme in ('data',):
|
||||
return False
|
||||
return self.tls_protected(url)
|
||||
|
||||
def tls_protected(self, url):
|
||||
return urlparse(url).scheme in ('https', 'ftps')
|
||||
|
||||
|
||||
class NoReferrerPolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer
|
||||
|
||||
The simplest policy is "no-referrer", which specifies that no referrer information
|
||||
is to be sent along with requests made from a particular request client to any origin.
|
||||
The header will be omitted entirely.
|
||||
"""
|
||||
name = POLICY_NO_REFERRER
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
return None
|
||||
|
||||
|
||||
class NoReferrerWhenDowngradePolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade
|
||||
|
||||
The "no-referrer-when-downgrade" policy sends a full URL along with requests
|
||||
from a TLS-protected environment settings object to a potentially trustworthy URL,
|
||||
and requests from clients which are not TLS-protected to any origin.
|
||||
|
||||
Requests from TLS-protected clients to non-potentially trustworthy URLs,
|
||||
on the other hand, will contain no referrer information.
|
||||
A Referer HTTP header will not be sent.
|
||||
|
||||
This is a user agent's default behavior, if no policy is otherwise specified.
|
||||
"""
|
||||
name = POLICY_NO_REFERRER_WHEN_DOWNGRADE
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
if not self.tls_protected(response_url) or self.tls_protected(request_url):
|
||||
return self.stripped_referrer(response_url)
|
||||
|
||||
|
||||
class SameOriginPolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin
|
||||
|
||||
The "same-origin" policy specifies that a full URL, stripped for use as a referrer,
|
||||
is sent as referrer information when making same-origin requests from a particular request client.
|
||||
|
||||
Cross-origin requests, on the other hand, will contain no referrer information.
|
||||
A Referer HTTP header will not be sent.
|
||||
"""
|
||||
name = POLICY_SAME_ORIGIN
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
if self.origin(response_url) == self.origin(request_url):
|
||||
return self.stripped_referrer(response_url)
|
||||
|
||||
|
||||
class OriginPolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-origin
|
||||
|
||||
The "origin" policy specifies that only the ASCII serialization
|
||||
of the origin of the request client is sent as referrer information
|
||||
when making both same-origin requests and cross-origin requests
|
||||
from a particular request client.
|
||||
"""
|
||||
name = POLICY_ORIGIN
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
return self.origin_referrer(response_url)
|
||||
|
||||
|
||||
class StrictOriginPolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin
|
||||
|
||||
The "strict-origin" policy sends the ASCII serialization
|
||||
of the origin of the request client when making requests:
|
||||
- from a TLS-protected environment settings object to a potentially trustworthy URL, and
|
||||
- from non-TLS-protected environment settings objects to any origin.
|
||||
|
||||
Requests from TLS-protected request clients to non- potentially trustworthy URLs,
|
||||
on the other hand, will contain no referrer information.
|
||||
A Referer HTTP header will not be sent.
|
||||
"""
|
||||
name = POLICY_STRICT_ORIGIN
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
if ((self.tls_protected(response_url) and
|
||||
self.potentially_trustworthy(request_url))
|
||||
or not self.tls_protected(response_url)):
|
||||
return self.origin_referrer(response_url)
|
||||
|
||||
|
||||
class OriginWhenCrossOriginPolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin
|
||||
|
||||
The "origin-when-cross-origin" policy specifies that a full URL,
|
||||
stripped for use as a referrer, is sent as referrer information
|
||||
when making same-origin requests from a particular request client,
|
||||
and only the ASCII serialization of the origin of the request client
|
||||
is sent as referrer information when making cross-origin requests
|
||||
from a particular request client.
|
||||
"""
|
||||
name = POLICY_ORIGIN_WHEN_CROSS_ORIGIN
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
origin = self.origin(response_url)
|
||||
if origin == self.origin(request_url):
|
||||
return self.stripped_referrer(response_url)
|
||||
else:
|
||||
return origin
|
||||
|
||||
|
||||
class StrictOriginWhenCrossOriginPolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin
|
||||
|
||||
The "strict-origin-when-cross-origin" policy specifies that a full URL,
|
||||
stripped for use as a referrer, is sent as referrer information
|
||||
when making same-origin requests from a particular request client,
|
||||
and only the ASCII serialization of the origin of the request client
|
||||
when making cross-origin requests:
|
||||
|
||||
- from a TLS-protected environment settings object to a potentially trustworthy URL, and
|
||||
- from non-TLS-protected environment settings objects to any origin.
|
||||
|
||||
Requests from TLS-protected clients to non- potentially trustworthy URLs,
|
||||
on the other hand, will contain no referrer information.
|
||||
A Referer HTTP header will not be sent.
|
||||
"""
|
||||
name = POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
origin = self.origin(response_url)
|
||||
if origin == self.origin(request_url):
|
||||
return self.stripped_referrer(response_url)
|
||||
elif ((self.tls_protected(response_url) and
|
||||
self.potentially_trustworthy(request_url))
|
||||
or not self.tls_protected(response_url)):
|
||||
return self.origin_referrer(response_url)
|
||||
|
||||
|
||||
class UnsafeUrlPolicy(ReferrerPolicy):
|
||||
"""
|
||||
https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url
|
||||
|
||||
The "unsafe-url" policy specifies that a full URL, stripped for use as a referrer,
|
||||
is sent along with both cross-origin requests
|
||||
and same-origin requests made from a particular request client.
|
||||
|
||||
Note: The policy's name doesn't lie; it is unsafe.
|
||||
This policy will leak origins and paths from TLS-protected resources
|
||||
to insecure origins.
|
||||
Carefully consider the impact of setting such a policy for potentially sensitive documents.
|
||||
"""
|
||||
name = POLICY_UNSAFE_URL
|
||||
|
||||
def referrer(self, response_url, request_url):
|
||||
return self.stripped_referrer(response_url)
|
||||
|
||||
|
||||
class DefaultReferrerPolicy(NoReferrerWhenDowngradePolicy):
|
||||
"""
|
||||
A variant of "no-referrer-when-downgrade",
|
||||
with the addition that "Referer" is not sent if the parent request was
|
||||
using ``file://`` or ``s3://`` scheme.
|
||||
"""
|
||||
NOREFERRER_SCHEMES = LOCAL_SCHEMES + ('file', 's3')
|
||||
name = POLICY_SCRAPY_DEFAULT
|
||||
|
||||
|
||||
_policy_classes = {p.name: p for p in (
|
||||
NoReferrerPolicy,
|
||||
NoReferrerWhenDowngradePolicy,
|
||||
SameOriginPolicy,
|
||||
OriginPolicy,
|
||||
StrictOriginPolicy,
|
||||
OriginWhenCrossOriginPolicy,
|
||||
StrictOriginWhenCrossOriginPolicy,
|
||||
UnsafeUrlPolicy,
|
||||
DefaultReferrerPolicy,
|
||||
)}
|
||||
|
||||
# Reference: https://www.w3.org/TR/referrer-policy/#referrer-policy-empty-string
|
||||
_policy_classes[''] = NoReferrerWhenDowngradePolicy
|
||||
|
||||
|
||||
def _load_policy_class(policy, warning_only=False):
|
||||
"""
|
||||
Expect a string for the path to the policy class,
|
||||
otherwise try to interpret the string as a standard value
|
||||
from https://www.w3.org/TR/referrer-policy/#referrer-policies
|
||||
"""
|
||||
try:
|
||||
return load_object(policy)
|
||||
except ValueError:
|
||||
try:
|
||||
return _policy_classes[policy.lower()]
|
||||
except KeyError:
|
||||
msg = "Could not load referrer policy %r" % policy
|
||||
if not warning_only:
|
||||
raise RuntimeError(msg)
|
||||
else:
|
||||
warnings.warn(msg, RuntimeWarning)
|
||||
return None
|
||||
|
||||
|
||||
class RefererMiddleware(object):
|
||||
|
||||
def __init__(self, settings=None):
|
||||
self.default_policy = DefaultReferrerPolicy
|
||||
if settings is not None:
|
||||
self.default_policy = _load_policy_class(
|
||||
settings.get('REFERRER_POLICY'))
|
||||
|
||||
@classmethod
|
||||
def from_crawler(cls, crawler):
|
||||
if not crawler.settings.getbool('REFERER_ENABLED'):
|
||||
raise NotConfigured
|
||||
return cls()
|
||||
mw = cls(crawler.settings)
|
||||
|
||||
# Note: this hook is a bit of a hack to intercept redirections
|
||||
crawler.signals.connect(mw.request_scheduled, signal=signals.request_scheduled)
|
||||
|
||||
return mw
|
||||
|
||||
def policy(self, resp_or_url, request):
|
||||
"""
|
||||
Determine Referrer-Policy to use from a parent Response (or URL),
|
||||
and a Request to be sent.
|
||||
|
||||
- if a valid policy is set in Request meta, it is used.
|
||||
- if the policy is set in meta but is wrong (e.g. a typo error),
|
||||
the policy from settings is used
|
||||
- if the policy is not set in Request meta,
|
||||
but there is a Referrer-policy header in the parent response,
|
||||
it is used if valid
|
||||
- otherwise, the policy from settings is used.
|
||||
"""
|
||||
policy_name = request.meta.get('referrer_policy')
|
||||
if policy_name is None:
|
||||
if isinstance(resp_or_url, Response):
|
||||
policy_header = resp_or_url.headers.get('Referrer-Policy')
|
||||
if policy_header is not None:
|
||||
policy_name = to_native_str(policy_header.decode('latin1'))
|
||||
if policy_name is None:
|
||||
return self.default_policy()
|
||||
|
||||
cls = _load_policy_class(policy_name, warning_only=True)
|
||||
return cls() if cls else self.default_policy()
|
||||
|
||||
def process_spider_output(self, response, result, spider):
|
||||
def _set_referer(r):
|
||||
if isinstance(r, Request):
|
||||
r.headers.setdefault('Referer', response.url)
|
||||
referrer = self.policy(response, r).referrer(response.url, r.url)
|
||||
if referrer is not None:
|
||||
r.headers.setdefault('Referer', referrer)
|
||||
return r
|
||||
return (_set_referer(r) for r in result or ())
|
||||
|
||||
def request_scheduled(self, request, spider):
|
||||
# check redirected request to patch "Referer" header if necessary
|
||||
redirected_urls = request.meta.get('redirect_urls', [])
|
||||
if redirected_urls:
|
||||
request_referrer = request.headers.get('Referer')
|
||||
# we don't patch the referrer value if there is none
|
||||
if request_referrer is not None:
|
||||
# the request's referrer header value acts as a surrogate
|
||||
# for the parent response URL
|
||||
#
|
||||
# Note: if the 3xx response contained a Referrer-Policy header,
|
||||
# the information is not available using this hook
|
||||
parent_url = safe_url_string(request_referrer)
|
||||
policy_referrer = self.policy(parent_url, request).referrer(
|
||||
parent_url, request.url)
|
||||
if policy_referrer != request_referrer:
|
||||
if policy_referrer is None:
|
||||
request.headers.pop('Referer')
|
||||
else:
|
||||
request.headers['Referer'] = policy_referrer
|
||||
|
|
|
|||
|
|
@ -5,7 +5,8 @@ import six
|
|||
from scrapy.spiders import Spider
|
||||
from scrapy.http import Request, XmlResponse
|
||||
from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots
|
||||
from scrapy.utils.gz import gunzip, is_gzipped
|
||||
from scrapy.utils.gz import gunzip, gzip_magic_number
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -59,12 +60,19 @@ class SitemapSpider(Spider):
|
|||
"""
|
||||
if isinstance(response, XmlResponse):
|
||||
return response.body
|
||||
elif is_gzipped(response):
|
||||
elif gzip_magic_number(response):
|
||||
return gunzip(response.body)
|
||||
elif response.url.endswith('.xml'):
|
||||
# actual gzipped sitemap files are decompressed above ;
|
||||
# if we are here (response body is not gzipped)
|
||||
# and have a response for .xml.gz,
|
||||
# it usually means that it was already gunzipped
|
||||
# by HttpCompression middleware,
|
||||
# the HTTP response being sent with "Content-Encoding: gzip"
|
||||
# without actually being a .xml.gz file in the first place,
|
||||
# merely XML gzip-compressed on the fly,
|
||||
# in other word, here, we have plain XML
|
||||
elif response.url.endswith('.xml') or response.url.endswith('.xml.gz'):
|
||||
return response.body
|
||||
elif response.url.endswith('.xml.gz'):
|
||||
return gunzip(response.body)
|
||||
|
||||
|
||||
def regex(x):
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ This module must not depend on any module outside the Standard Library.
|
|||
import copy
|
||||
import six
|
||||
import warnings
|
||||
from collections import OrderedDict
|
||||
from collections import OrderedDict, Mapping
|
||||
|
||||
from scrapy.exceptions import ScrapyDeprecationWarning
|
||||
|
||||
|
|
@ -224,7 +224,7 @@ class CaselessDict(dict):
|
|||
return dict.setdefault(self, self.normkey(key), self.normvalue(def_val))
|
||||
|
||||
def update(self, seq):
|
||||
seq = seq.items() if isinstance(seq, dict) else seq
|
||||
seq = seq.items() if isinstance(seq, Mapping) else seq
|
||||
iseq = ((self.normkey(k), self.normvalue(v)) for k, v in seq)
|
||||
super(CaselessDict, self).update(iseq)
|
||||
|
||||
|
|
|
|||
|
|
@ -59,3 +59,7 @@ def is_gzipped(response):
|
|||
cenc = response.headers.get('Content-Encoding', b'').lower()
|
||||
return (_is_gzipped(ctype) or
|
||||
(_is_octetstream(ctype) and cenc in (b'gzip', b'x-gzip')))
|
||||
|
||||
|
||||
def gzip_magic_number(response):
|
||||
return response.body[:3] == b'\x1f\x8b\x08'
|
||||
|
|
|
|||
|
|
@ -103,9 +103,25 @@ def configure_logging(settings=None, install_root_handler=True):
|
|||
dictConfig(LOGGING)
|
||||
|
||||
elif install_root_handler:
|
||||
logging.root.setLevel(logging.NOTSET)
|
||||
handler = _get_handler(settings)
|
||||
logging.root.addHandler(handler)
|
||||
install_scrapy_root_handler(settings)
|
||||
|
||||
|
||||
def install_scrapy_root_handler(settings):
|
||||
global _scrapy_root_handler
|
||||
|
||||
if (_scrapy_root_handler is not None
|
||||
and _scrapy_root_handler in logging.root.handlers):
|
||||
logging.root.removeHandler(_scrapy_root_handler)
|
||||
logging.root.setLevel(logging.NOTSET)
|
||||
_scrapy_root_handler = _get_handler(settings)
|
||||
logging.root.addHandler(_scrapy_root_handler)
|
||||
|
||||
|
||||
def get_scrapy_root_handler():
|
||||
return _scrapy_root_handler
|
||||
|
||||
|
||||
_scrapy_root_handler = None
|
||||
|
||||
|
||||
def _get_handler(settings):
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ to the w3lib.url module. Always import those from there instead.
|
|||
"""
|
||||
import posixpath
|
||||
import re
|
||||
from six.moves.urllib.parse import (ParseResult, urldefrag, urlparse)
|
||||
from six.moves.urllib.parse import (ParseResult, urldefrag, urlparse, urlunparse)
|
||||
|
||||
# scrapy.utils.url was moved to w3lib.url and import * ensures this
|
||||
# move doesn't break old code
|
||||
|
|
@ -103,3 +103,34 @@ def guess_scheme(url):
|
|||
return any_to_uri(url)
|
||||
else:
|
||||
return add_http_if_no_scheme(url)
|
||||
|
||||
|
||||
def strip_url(url, strip_credentials=True, strip_default_port=True, origin_only=False, strip_fragment=True):
|
||||
|
||||
"""Strip URL string from some of its components:
|
||||
|
||||
- `strip_credentials` removes "user:password@"
|
||||
- `strip_default_port` removes ":80" (resp. ":443", ":21")
|
||||
from http:// (resp. https://, ftp://) URLs
|
||||
- `origin_only` replaces path component with "/", also dropping
|
||||
query and fragment components ; it also strips credentials
|
||||
- `strip_fragment` drops any #fragment component
|
||||
"""
|
||||
|
||||
parsed_url = urlparse(url)
|
||||
netloc = parsed_url.netloc
|
||||
if (strip_credentials or origin_only) and (parsed_url.username or parsed_url.password):
|
||||
netloc = netloc.split('@')[-1]
|
||||
if strip_default_port and parsed_url.port:
|
||||
if (parsed_url.scheme, parsed_url.port) in (('http', 80),
|
||||
('https', 443),
|
||||
('ftp', 21)):
|
||||
netloc = netloc.replace(':{p.port}'.format(p=parsed_url), '')
|
||||
return urlunparse((
|
||||
parsed_url.scheme,
|
||||
netloc,
|
||||
'/' if origin_only else parsed_url.path,
|
||||
'' if origin_only else parsed_url.params,
|
||||
'' if origin_only else parsed_url.query,
|
||||
'' if strip_fragment else parsed_url.fragment
|
||||
))
|
||||
|
|
|
|||
|
|
@ -5,13 +5,17 @@ from subprocess import Popen, PIPE
|
|||
|
||||
from twisted.web.server import Site, NOT_DONE_YET
|
||||
from twisted.web.resource import Resource
|
||||
from twisted.web.static import File
|
||||
from twisted.web.test.test_webclient import PayloadResource
|
||||
from twisted.web.server import GzipEncoderFactory
|
||||
from twisted.web.resource import EncodingResourceWrapper
|
||||
from twisted.web.util import redirectTo
|
||||
from twisted.internet import reactor, ssl
|
||||
from twisted.internet.task import deferLater
|
||||
|
||||
|
||||
from scrapy.utils.python import to_bytes, to_unicode
|
||||
from tests import tests_datadir
|
||||
|
||||
|
||||
def getarg(request, name, default=None, type=None):
|
||||
|
|
@ -120,6 +124,16 @@ class Echo(LeafResource):
|
|||
return to_bytes(json.dumps(output))
|
||||
|
||||
|
||||
class RedirectTo(LeafResource):
|
||||
|
||||
def render(self, request):
|
||||
goto = getarg(request, b'goto', b'/')
|
||||
# we force the body content, otherwise Twisted redirectTo()
|
||||
# returns HTML with <meta http-equiv="refresh"
|
||||
redirectTo(goto, request)
|
||||
return b'redirecting...'
|
||||
|
||||
|
||||
class Partial(LeafResource):
|
||||
|
||||
def render_GET(self, request):
|
||||
|
|
@ -160,6 +174,8 @@ class Root(Resource):
|
|||
self.putChild(b"echo", Echo())
|
||||
self.putChild(b"payload", PayloadResource())
|
||||
self.putChild(b"xpayload", EncodingResourceWrapper(PayloadResource(), [GzipEncoderFactory()]))
|
||||
self.putChild(b"files", File(os.path.join(tests_datadir, 'test_site/files/')))
|
||||
self.putChild(b"redirect-to", RedirectTo())
|
||||
|
||||
def getChild(self, name, request):
|
||||
return self
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@
|
|||
</div>
|
||||
<a href='http://example.com/sample3.html' title='sample 3'>sample 3 text</a>
|
||||
<a href='sample3.html'>sample 3 repetition</a>
|
||||
<a href='sample3.html#foo'>sample 3 repetition with fragment</a>
|
||||
<a href='http://www.google.com/something'></a>
|
||||
<a href='http://example.com/innertag.html'><b>inner</b> tag</a>
|
||||
<a href=' page 4.html '>href with whitespaces</a>
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.6 KiB |
|
|
@ -1,3 +1,6 @@
|
|||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import warnings
|
||||
import unittest
|
||||
|
||||
|
|
@ -5,6 +8,7 @@ import scrapy
|
|||
from scrapy.crawler import Crawler, CrawlerRunner, CrawlerProcess
|
||||
from scrapy.settings import Settings, default_settings
|
||||
from scrapy.spiderloader import SpiderLoader
|
||||
from scrapy.utils.log import configure_logging, get_scrapy_root_handler
|
||||
from scrapy.utils.spider import DefaultSpider
|
||||
from scrapy.utils.misc import load_object
|
||||
from scrapy.extensions.throttle import AutoThrottle
|
||||
|
|
@ -74,6 +78,48 @@ class SpiderSettingsTestCase(unittest.TestCase):
|
|||
self.assertIn(AutoThrottle, enabled_exts)
|
||||
|
||||
|
||||
class CrawlerLoggingTestCase(unittest.TestCase):
|
||||
def test_no_root_handler_installed(self):
|
||||
handler = get_scrapy_root_handler()
|
||||
if handler is not None:
|
||||
logging.root.removeHandler(handler)
|
||||
|
||||
class MySpider(scrapy.Spider):
|
||||
name = 'spider'
|
||||
|
||||
crawler = Crawler(MySpider, {})
|
||||
assert get_scrapy_root_handler() is None
|
||||
|
||||
def test_spider_custom_settings_log_level(self):
|
||||
with tempfile.NamedTemporaryFile() as log_file:
|
||||
class MySpider(scrapy.Spider):
|
||||
name = 'spider'
|
||||
custom_settings = {
|
||||
'LOG_LEVEL': 'INFO',
|
||||
'LOG_FILE': log_file.name,
|
||||
}
|
||||
|
||||
configure_logging()
|
||||
self.assertEqual(get_scrapy_root_handler().level, logging.DEBUG)
|
||||
crawler = Crawler(MySpider, {})
|
||||
self.assertEqual(get_scrapy_root_handler().level, logging.INFO)
|
||||
info_count = crawler.stats.get_value('log_count/INFO')
|
||||
logging.debug('debug message')
|
||||
logging.info('info message')
|
||||
logging.warning('warning message')
|
||||
logging.error('error message')
|
||||
logged = log_file.read().decode('utf8')
|
||||
self.assertNotIn('debug message', logged)
|
||||
self.assertIn('info message', logged)
|
||||
self.assertIn('warning message', logged)
|
||||
self.assertIn('error message', logged)
|
||||
self.assertEqual(crawler.stats.get_value('log_count/ERROR'), 1)
|
||||
self.assertEqual(crawler.stats.get_value('log_count/WARNING'), 1)
|
||||
self.assertEqual(
|
||||
crawler.stats.get_value('log_count/INFO') - info_count, 1)
|
||||
self.assertEqual(crawler.stats.get_value('log_count/DEBUG', 0), 0)
|
||||
|
||||
|
||||
class SpiderLoaderWithWrongInterface(object):
|
||||
|
||||
def unneeded_method(self):
|
||||
|
|
|
|||
|
|
@ -6,20 +6,22 @@ try:
|
|||
from unittest import mock
|
||||
except ImportError:
|
||||
import mock
|
||||
import shutil
|
||||
|
||||
from twisted.trial import unittest
|
||||
from twisted.protocols.policies import WrappingFactory
|
||||
from twisted.python.filepath import FilePath
|
||||
from twisted.internet import reactor, defer, error
|
||||
from twisted.web import server, static, util, resource
|
||||
from twisted.web._newclient import ResponseFailed
|
||||
from twisted.web.http import _DataLoss
|
||||
from twisted.web.test.test_webclient import ForeverTakingResource, \
|
||||
NoLengthResource, HostHeaderResource, \
|
||||
PayloadResource, BrokenDownloadResource
|
||||
PayloadResource
|
||||
from twisted.cred import portal, checkers, credentials
|
||||
from w3lib.url import path_to_file_uri
|
||||
|
||||
from scrapy.core.downloader.handlers import DownloadHandlers
|
||||
from scrapy.core.downloader.handlers.datauri import DataURIDownloadHandler
|
||||
from scrapy.core.downloader.handlers.file import FileDownloadHandler
|
||||
from scrapy.core.downloader.handlers.http import HTTPDownloadHandler, HttpDownloadHandler
|
||||
from scrapy.core.downloader.handlers.http10 import HTTP10DownloadHandler
|
||||
|
|
@ -29,6 +31,7 @@ from scrapy.core.downloader.handlers.s3 import S3DownloadHandler
|
|||
from scrapy.spiders import Spider
|
||||
from scrapy.http import Request
|
||||
from scrapy.http.response.text import TextResponse
|
||||
from scrapy.responsetypes import responsetypes
|
||||
from scrapy.settings import Settings
|
||||
from scrapy.utils.test import get_crawler, skip_if_no_boto
|
||||
from scrapy.utils.python import to_bytes
|
||||
|
|
@ -118,6 +121,52 @@ class ContentLengthHeaderResource(resource.Resource):
|
|||
return request.requestHeaders.getRawHeaders(b"content-length")[0]
|
||||
|
||||
|
||||
class ChunkedResource(resource.Resource):
|
||||
|
||||
def render(self, request):
|
||||
def response():
|
||||
request.write(b"chunked ")
|
||||
request.write(b"content\n")
|
||||
request.finish()
|
||||
reactor.callLater(0, response)
|
||||
return server.NOT_DONE_YET
|
||||
|
||||
|
||||
class BrokenChunkedResource(resource.Resource):
|
||||
|
||||
def render(self, request):
|
||||
def response():
|
||||
request.write(b"chunked ")
|
||||
request.write(b"content\n")
|
||||
# Disable terminating chunk on finish.
|
||||
request.chunked = False
|
||||
closeConnection(request)
|
||||
reactor.callLater(0, response)
|
||||
return server.NOT_DONE_YET
|
||||
|
||||
|
||||
class BrokenDownloadResource(resource.Resource):
|
||||
|
||||
def render(self, request):
|
||||
def response():
|
||||
request.setHeader(b"Content-Length", b"20")
|
||||
request.write(b"partial")
|
||||
closeConnection(request)
|
||||
|
||||
reactor.callLater(0, response)
|
||||
return server.NOT_DONE_YET
|
||||
|
||||
|
||||
def closeConnection(request):
|
||||
# We have to force a disconnection for HTTP/1.1 clients. Otherwise
|
||||
# client keeps the connection open waiting for more data.
|
||||
if hasattr(request.channel, 'loseConnection'): # twisted >=16.3.0
|
||||
request.channel.loseConnection()
|
||||
else:
|
||||
request.channel.transport.loseConnection()
|
||||
request.finish()
|
||||
|
||||
|
||||
class EmptyContentTypeHeaderResource(resource.Resource):
|
||||
"""
|
||||
A testing resource which renders itself as the value of request body
|
||||
|
|
@ -149,6 +198,8 @@ class HttpTestCase(unittest.TestCase):
|
|||
r.putChild(b"host", HostHeaderResource())
|
||||
r.putChild(b"payload", PayloadResource())
|
||||
r.putChild(b"broken", BrokenDownloadResource())
|
||||
r.putChild(b"chunked", ChunkedResource())
|
||||
r.putChild(b"broken-chunked", BrokenChunkedResource())
|
||||
r.putChild(b"contentlength", ContentLengthHeaderResource())
|
||||
r.putChild(b"nocontenttype", EmptyContentTypeHeaderResource())
|
||||
self.site = server.Site(r, timeout=None)
|
||||
|
|
@ -341,6 +392,53 @@ class Http11TestCase(HttpTestCase):
|
|||
d.addCallback(self.assertEquals, b"0123456789")
|
||||
return d
|
||||
|
||||
def test_download_chunked_content(self):
|
||||
request = Request(self.getURL('chunked'))
|
||||
d = self.download_request(request, Spider('foo'))
|
||||
d.addCallback(lambda r: r.body)
|
||||
d.addCallback(self.assertEquals, b"chunked content\n")
|
||||
return d
|
||||
|
||||
def test_download_broken_content_cause_data_loss(self, url='broken'):
|
||||
request = Request(self.getURL(url))
|
||||
d = self.download_request(request, Spider('foo'))
|
||||
|
||||
def checkDataLoss(failure):
|
||||
if failure.check(ResponseFailed):
|
||||
if any(r.check(_DataLoss) for r in failure.value.reasons):
|
||||
return None
|
||||
return failure
|
||||
|
||||
d.addCallback(lambda _: self.fail("No DataLoss exception"))
|
||||
d.addErrback(checkDataLoss)
|
||||
return d
|
||||
|
||||
def test_download_broken_chunked_content_cause_data_loss(self):
|
||||
return self.test_download_broken_content_cause_data_loss('broken-chunked')
|
||||
|
||||
def test_download_broken_content_allow_data_loss(self, url='broken'):
|
||||
request = Request(self.getURL(url), meta={'download_fail_on_dataloss': False})
|
||||
d = self.download_request(request, Spider('foo'))
|
||||
d.addCallback(lambda r: r.flags)
|
||||
d.addCallback(self.assertEqual, ['dataloss'])
|
||||
return d
|
||||
|
||||
def test_download_broken_chunked_content_allow_data_loss(self):
|
||||
return self.test_download_broken_content_allow_data_loss('broken-chunked')
|
||||
|
||||
def test_download_broken_content_allow_data_loss_via_setting(self, url='broken'):
|
||||
download_handler = self.download_handler_cls(Settings({
|
||||
'DOWNLOAD_FAIL_ON_DATALOSS': False,
|
||||
}))
|
||||
request = Request(self.getURL(url))
|
||||
d = download_handler.download_request(request, Spider('foo'))
|
||||
d.addCallback(lambda r: r.flags)
|
||||
d.addCallback(self.assertEqual, ['dataloss'])
|
||||
return d
|
||||
|
||||
def test_download_broken_chunked_content_allow_data_loss_via_setting(self):
|
||||
return self.test_download_broken_content_allow_data_loss_via_setting('broken-chunked')
|
||||
|
||||
|
||||
class Https11TestCase(Http11TestCase):
|
||||
scheme = 'https'
|
||||
|
|
@ -825,3 +923,69 @@ class AnonymousFTPTestCase(BaseFTPTestCase):
|
|||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.directory)
|
||||
|
||||
|
||||
class DataURITestCase(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
self.download_handler = DataURIDownloadHandler(Settings())
|
||||
self.download_request = self.download_handler.download_request
|
||||
self.spider = Spider('foo')
|
||||
|
||||
def test_response_attrs(self):
|
||||
uri = "data:,A%20brief%20note"
|
||||
|
||||
def _test(response):
|
||||
self.assertEquals(response.url, uri)
|
||||
self.assertFalse(response.headers)
|
||||
|
||||
request = Request(uri)
|
||||
return self.download_request(request, self.spider).addCallback(_test)
|
||||
|
||||
def test_default_mediatype_encoding(self):
|
||||
def _test(response):
|
||||
self.assertEquals(response.text, 'A brief note')
|
||||
self.assertEquals(type(response),
|
||||
responsetypes.from_mimetype("text/plain"))
|
||||
self.assertEquals(response.encoding, "US-ASCII")
|
||||
|
||||
request = Request("data:,A%20brief%20note")
|
||||
return self.download_request(request, self.spider).addCallback(_test)
|
||||
|
||||
def test_default_mediatype(self):
|
||||
def _test(response):
|
||||
self.assertEquals(response.text, u'\u038e\u03a3\u038e')
|
||||
self.assertEquals(type(response),
|
||||
responsetypes.from_mimetype("text/plain"))
|
||||
self.assertEquals(response.encoding, "iso-8859-7")
|
||||
|
||||
request = Request("data:;charset=iso-8859-7,%be%d3%be")
|
||||
return self.download_request(request, self.spider).addCallback(_test)
|
||||
|
||||
def test_text_charset(self):
|
||||
def _test(response):
|
||||
self.assertEquals(response.text, u'\u038e\u03a3\u038e')
|
||||
self.assertEquals(response.body, b'\xbe\xd3\xbe')
|
||||
self.assertEquals(response.encoding, "iso-8859-7")
|
||||
|
||||
request = Request("data:text/plain;charset=iso-8859-7,%be%d3%be")
|
||||
return self.download_request(request, self.spider).addCallback(_test)
|
||||
|
||||
def test_mediatype_parameters(self):
|
||||
def _test(response):
|
||||
self.assertEquals(response.text, u'\u038e\u03a3\u038e')
|
||||
self.assertEquals(type(response),
|
||||
responsetypes.from_mimetype("text/plain"))
|
||||
self.assertEquals(response.encoding, "utf-8")
|
||||
|
||||
request = Request('data:text/plain;foo=%22foo;bar%5C%22%22;'
|
||||
'charset=utf-8;bar=%22foo;%5C%22 foo ;/,%22'
|
||||
',%CE%8E%CE%A3%CE%8E')
|
||||
return self.download_request(request, self.spider).addCallback(_test)
|
||||
|
||||
def test_base64(self):
|
||||
def _test(response):
|
||||
self.assertEquals(response.text, 'Hello, world.')
|
||||
|
||||
request = Request('data:text/plain;base64,SGVsbG8sIHdvcmxkLg%3D%3D')
|
||||
return self.download_request(request, self.spider).addCallback(_test)
|
||||
|
|
|
|||
|
|
@ -7,6 +7,8 @@ from scrapy.spiders import Spider
|
|||
from scrapy.http import Response, Request, HtmlResponse
|
||||
from scrapy.downloadermiddlewares.httpcompression import HttpCompressionMiddleware, \
|
||||
ACCEPTED_ENCODINGS
|
||||
from scrapy.responsetypes import responsetypes
|
||||
from scrapy.utils.gz import gunzip
|
||||
from tests import tests_datadir
|
||||
from w3lib.encoding import resolve_encoding
|
||||
|
||||
|
|
@ -152,15 +154,29 @@ class HttpCompressionTest(TestCase):
|
|||
self.assertEqual(newresponse.body, plainbody)
|
||||
self.assertEqual(newresponse.encoding, resolve_encoding('gb2312'))
|
||||
|
||||
def test_process_response_no_content_type_header(self):
|
||||
headers = {
|
||||
'Content-Encoding': 'identity',
|
||||
}
|
||||
plainbody = b"""<html><head><title>Some page</title><meta http-equiv="Content-Type" content="text/html; charset=gb2312">"""
|
||||
respcls = responsetypes.from_args(url="http://www.example.com/index", headers=headers, body=plainbody)
|
||||
response = respcls("http://www.example.com/index", headers=headers, body=plainbody)
|
||||
request = Request("http://www.example.com/index")
|
||||
|
||||
newresponse = self.mw.process_response(request, response, self.spider)
|
||||
assert isinstance(newresponse, respcls)
|
||||
self.assertEqual(newresponse.body, plainbody)
|
||||
self.assertEqual(newresponse.encoding, resolve_encoding('gb2312'))
|
||||
|
||||
def test_process_response_gzipped_contenttype(self):
|
||||
response = self._getresponse('gzip')
|
||||
response.headers['Content-Type'] = 'application/gzip'
|
||||
request = response.request
|
||||
|
||||
newresponse = self.mw.process_response(request, response, self.spider)
|
||||
self.assertIs(newresponse, response)
|
||||
self.assertEqual(response.headers['Content-Encoding'], b'gzip')
|
||||
self.assertEqual(response.headers['Content-Type'], b'application/gzip')
|
||||
self.assertIsNot(newresponse, response)
|
||||
self.assertTrue(newresponse.body.startswith(b'<!DOCTYPE'))
|
||||
self.assertNotIn('Content-Encoding', newresponse.headers)
|
||||
|
||||
def test_process_response_gzip_app_octetstream_contenttype(self):
|
||||
response = self._getresponse('gzip')
|
||||
|
|
@ -168,9 +184,9 @@ class HttpCompressionTest(TestCase):
|
|||
request = response.request
|
||||
|
||||
newresponse = self.mw.process_response(request, response, self.spider)
|
||||
self.assertIs(newresponse, response)
|
||||
self.assertEqual(response.headers['Content-Encoding'], b'gzip')
|
||||
self.assertEqual(response.headers['Content-Type'], b'application/octet-stream')
|
||||
self.assertIsNot(newresponse, response)
|
||||
self.assertTrue(newresponse.body.startswith(b'<!DOCTYPE'))
|
||||
self.assertNotIn('Content-Encoding', newresponse.headers)
|
||||
|
||||
def test_process_response_gzip_binary_octetstream_contenttype(self):
|
||||
response = self._getresponse('x-gzip')
|
||||
|
|
@ -178,9 +194,51 @@ class HttpCompressionTest(TestCase):
|
|||
request = response.request
|
||||
|
||||
newresponse = self.mw.process_response(request, response, self.spider)
|
||||
self.assertIs(newresponse, response)
|
||||
self.assertEqual(response.headers['Content-Encoding'], b'gzip')
|
||||
self.assertEqual(response.headers['Content-Type'], b'binary/octet-stream')
|
||||
self.assertIsNot(newresponse, response)
|
||||
self.assertTrue(newresponse.body.startswith(b'<!DOCTYPE'))
|
||||
self.assertNotIn('Content-Encoding', newresponse.headers)
|
||||
|
||||
def test_process_response_gzipped_gzip_file(self):
|
||||
"""Test that a gzip Content-Encoded .gz file is gunzipped
|
||||
only once by the middleware, leaving gunzipping of the file
|
||||
to upper layers.
|
||||
"""
|
||||
headers = {
|
||||
'Content-Type': 'application/gzip',
|
||||
'Content-Encoding': 'gzip',
|
||||
}
|
||||
# build a gzipped file (here, a sitemap)
|
||||
f = BytesIO()
|
||||
plainbody = b"""<?xml version="1.0" encoding="UTF-8"?>
|
||||
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">
|
||||
<url>
|
||||
<loc>http://www.example.com/</loc>
|
||||
<lastmod>2009-08-16</lastmod>
|
||||
<changefreq>daily</changefreq>
|
||||
<priority>1</priority>
|
||||
</url>
|
||||
<url>
|
||||
<loc>http://www.example.com/Special-Offers.html</loc>
|
||||
<lastmod>2009-08-16</lastmod>
|
||||
<changefreq>weekly</changefreq>
|
||||
<priority>0.8</priority>
|
||||
</url>
|
||||
</urlset>"""
|
||||
gz_file = GzipFile(fileobj=f, mode='wb')
|
||||
gz_file.write(plainbody)
|
||||
gz_file.close()
|
||||
|
||||
# build a gzipped response body containing this gzipped file
|
||||
r = BytesIO()
|
||||
gz_resp = GzipFile(fileobj=r, mode='wb')
|
||||
gz_resp.write(f.getvalue())
|
||||
gz_resp.close()
|
||||
|
||||
response = Response("http;//www.example.com/", headers=headers, body=r.getvalue())
|
||||
request = Request("http://www.example.com/")
|
||||
|
||||
newresponse = self.mw.process_response(request, response, self.spider)
|
||||
self.assertEqual(gunzip(newresponse.body), plainbody)
|
||||
|
||||
def test_process_response_head_request_no_decode_required(self):
|
||||
response = self._getresponse('gzip')
|
||||
|
|
|
|||
|
|
@ -392,6 +392,7 @@ class TextResponseTest(BaseResponseTest):
|
|||
'http://example.com/sample2.html',
|
||||
'http://example.com/sample3.html',
|
||||
'http://example.com/sample3.html',
|
||||
'http://example.com/sample3.html#foo',
|
||||
'http://www.google.com/something',
|
||||
'http://example.com/innertag.html'
|
||||
]
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ from tests import get_testdata
|
|||
class Base:
|
||||
class LinkExtractorTestCase(unittest.TestCase):
|
||||
extractor_cls = None
|
||||
escapes_whitespace = False
|
||||
|
||||
def setUp(self):
|
||||
body = get_testdata('link_extractor', 'sgml_linkextractor.html')
|
||||
|
|
@ -26,13 +27,19 @@ class Base:
|
|||
|
||||
def test_extract_all_links(self):
|
||||
lx = self.extractor_cls()
|
||||
if self.escapes_whitespace:
|
||||
page4_url = 'http://example.com/page%204.html'
|
||||
else:
|
||||
page4_url = 'http://example.com/page 4.html'
|
||||
|
||||
self.assertEqual([link for link in lx.extract_links(self.response)], [
|
||||
Link(url='http://example.com/sample1.html', text=u''),
|
||||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html#foo', text='sample 3 repetition with fragment'),
|
||||
Link(url='http://www.google.com/something', text=u''),
|
||||
Link(url='http://example.com/innertag.html', text=u'inner tag'),
|
||||
Link(url='http://example.com/page%204.html', text=u'href with whitespaces'),
|
||||
Link(url=page4_url, text=u'href with whitespaces'),
|
||||
])
|
||||
|
||||
def test_extract_filter_allow(self):
|
||||
|
|
@ -41,6 +48,7 @@ class Base:
|
|||
Link(url='http://example.com/sample1.html', text=u''),
|
||||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html#foo', text='sample 3 repetition with fragment')
|
||||
])
|
||||
|
||||
def test_extract_filter_allow_with_duplicates(self):
|
||||
|
|
@ -50,6 +58,27 @@ class Base:
|
|||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 repetition'),
|
||||
Link(url='http://example.com/sample3.html#foo', text='sample 3 repetition with fragment')
|
||||
])
|
||||
|
||||
def test_extract_filter_allow_with_duplicates_canonicalize(self):
|
||||
lx = self.extractor_cls(allow=('sample', ), unique=False,
|
||||
canonicalize=True)
|
||||
self.assertEqual([link for link in lx.extract_links(self.response)], [
|
||||
Link(url='http://example.com/sample1.html', text=u''),
|
||||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 repetition'),
|
||||
Link(url='http://example.com/sample3.html', text='sample 3 repetition with fragment')
|
||||
])
|
||||
|
||||
def test_extract_filter_allow_no_duplicates_canonicalize(self):
|
||||
lx = self.extractor_cls(allow=('sample',), unique=True,
|
||||
canonicalize=True)
|
||||
self.assertEqual([link for link in lx.extract_links(self.response)], [
|
||||
Link(url='http://example.com/sample1.html', text=u''),
|
||||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
])
|
||||
|
||||
def test_extract_filter_allow_and_deny(self):
|
||||
|
|
@ -73,6 +102,8 @@ class Base:
|
|||
Link(url='http://example.com/sample1.html', text=u''),
|
||||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html#foo',
|
||||
text='sample 3 repetition with fragment')
|
||||
])
|
||||
|
||||
lx = self.extractor_cls(allow='sample', deny='3')
|
||||
|
|
@ -276,13 +307,19 @@ class Base:
|
|||
|
||||
def test_attrs(self):
|
||||
lx = self.extractor_cls(attrs="href")
|
||||
if self.escapes_whitespace:
|
||||
page4_url = 'http://example.com/page%204.html'
|
||||
else:
|
||||
page4_url = 'http://example.com/page 4.html'
|
||||
|
||||
self.assertEqual(lx.extract_links(self.response), [
|
||||
Link(url='http://example.com/sample1.html', text=u''),
|
||||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html#foo', text='sample 3 repetition with fragment'),
|
||||
Link(url='http://www.google.com/something', text=u''),
|
||||
Link(url='http://example.com/innertag.html', text=u'inner tag'),
|
||||
Link(url='http://example.com/page%204.html', text=u'href with whitespaces'),
|
||||
Link(url=page4_url, text=u'href with whitespaces'),
|
||||
])
|
||||
|
||||
lx = self.extractor_cls(attrs=("href","src"), tags=("a","area","img"), deny_extensions=())
|
||||
|
|
@ -291,9 +328,10 @@ class Base:
|
|||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample2.jpg', text=u''),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html#foo', text='sample 3 repetition with fragment'),
|
||||
Link(url='http://www.google.com/something', text=u''),
|
||||
Link(url='http://example.com/innertag.html', text=u'inner tag'),
|
||||
Link(url='http://example.com/page%204.html', text=u'href with whitespaces'),
|
||||
Link(url=page4_url, text=u'href with whitespaces'),
|
||||
])
|
||||
|
||||
lx = self.extractor_cls(attrs=None)
|
||||
|
|
|
|||
|
|
@ -121,6 +121,7 @@ class HtmlParserLinkExtractorTestCase(unittest.TestCase):
|
|||
Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 repetition'),
|
||||
Link(url='http://example.com/sample3.html#foo', text=u'sample 3 repetition with fragment'),
|
||||
Link(url='http://www.google.com/something', text=u''),
|
||||
Link(url='http://example.com/innertag.html', text=u'inner tag'),
|
||||
Link(url='http://example.com/page%204.html', text=u'href with whitespaces'),
|
||||
|
|
@ -142,6 +143,7 @@ class HtmlParserLinkExtractorTestCase(unittest.TestCase):
|
|||
|
||||
class SgmlLinkExtractorTestCase(Base.LinkExtractorTestCase):
|
||||
extractor_cls = SgmlLinkExtractor
|
||||
escapes_whitespace = True
|
||||
|
||||
def test_deny_extensions(self):
|
||||
html = """<a href="page.html">asd</a> and <a href="photo.jpg">"""
|
||||
|
|
@ -190,6 +192,7 @@ class RegexLinkExtractorTestCase(unittest.TestCase):
|
|||
self.assertEqual(lx.extract_links(self.response),
|
||||
[Link(url='http://example.com/sample2.html', text=u'sample 2'),
|
||||
Link(url='http://example.com/sample3.html', text=u'sample 3 text'),
|
||||
Link(url='http://example.com/sample3.html#foo', text=u'sample 3 repetition with fragment'),
|
||||
Link(url='http://www.google.com/something', text=u''),
|
||||
Link(url='http://example.com/innertag.html', text=u'inner tag'),])
|
||||
|
||||
|
|
|
|||
|
|
@ -64,5 +64,30 @@ class LoggingContribTest(unittest.TestCase):
|
|||
assert all(isinstance(x, six.text_type) for x in lines)
|
||||
self.assertEqual(lines, [u"Scraped from <200 http://www.example.com>", u'name: \xa3'])
|
||||
|
||||
|
||||
class LogFormatterSubclass(LogFormatter):
|
||||
def crawled(self, request, response, spider):
|
||||
kwargs = super(LogFormatterSubclass, self).crawled(
|
||||
request, response, spider)
|
||||
CRAWLEDMSG = (
|
||||
u"Crawled (%(status)s) %(request)s (referer: "
|
||||
u"%(referer)s)%(flags)s"
|
||||
)
|
||||
return {
|
||||
'level': kwargs['level'],
|
||||
'msg': CRAWLEDMSG,
|
||||
'args': kwargs['args']
|
||||
}
|
||||
|
||||
|
||||
class LogformatterSubclassTest(LoggingContribTest):
|
||||
def setUp(self):
|
||||
self.formatter = LogFormatterSubclass()
|
||||
self.spider = Spider('default')
|
||||
|
||||
def test_flags_in_request(self):
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
|
|||
|
|
@ -0,0 +1,182 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
import os
|
||||
import shutil
|
||||
|
||||
from testfixtures import LogCapture
|
||||
from twisted.internet import defer
|
||||
from twisted.trial.unittest import TestCase
|
||||
from w3lib.url import add_or_replace_parameter
|
||||
|
||||
from scrapy.crawler import CrawlerRunner
|
||||
from scrapy import signals
|
||||
from tests.mockserver import MockServer
|
||||
from tests.spiders import SimpleSpider
|
||||
|
||||
|
||||
class MediaDownloadSpider(SimpleSpider):
|
||||
name = 'mediadownload'
|
||||
|
||||
def _process_url(self, url):
|
||||
return url
|
||||
|
||||
def parse(self, response):
|
||||
self.logger.info(response.headers)
|
||||
self.logger.info(response.text)
|
||||
item = {
|
||||
self.media_key: [],
|
||||
self.media_urls_key: [
|
||||
self._process_url(response.urljoin(href))
|
||||
for href in response.xpath('''
|
||||
//table[thead/tr/th="Filename"]
|
||||
/tbody//a/@href
|
||||
''').extract()],
|
||||
}
|
||||
yield item
|
||||
|
||||
|
||||
class BrokenLinksMediaDownloadSpider(MediaDownloadSpider):
|
||||
name = 'brokenmedia'
|
||||
|
||||
def _process_url(self, url):
|
||||
return url + '.foo'
|
||||
|
||||
|
||||
class RedirectedMediaDownloadSpider(MediaDownloadSpider):
|
||||
name = 'redirectedmedia'
|
||||
|
||||
def _process_url(self, url):
|
||||
return add_or_replace_parameter(
|
||||
'http://localhost:8998/redirect-to',
|
||||
'goto', url)
|
||||
|
||||
|
||||
class FileDownloadCrawlTestCase(TestCase):
|
||||
pipeline_class = 'scrapy.pipelines.files.FilesPipeline'
|
||||
store_setting_key = 'FILES_STORE'
|
||||
media_key = 'files'
|
||||
media_urls_key = 'file_urls'
|
||||
expected_checksums = set([
|
||||
'5547178b89448faf0015a13f904c936e',
|
||||
'c2281c83670e31d8aaab7cb642b824db',
|
||||
'ed3f6538dc15d4d9179dae57319edc5f'])
|
||||
|
||||
def setUp(self):
|
||||
self.mockserver = MockServer()
|
||||
self.mockserver.__enter__()
|
||||
|
||||
# prepare a directory for storing files
|
||||
self.tmpmediastore = self.mktemp()
|
||||
os.mkdir(self.tmpmediastore)
|
||||
self.settings = {
|
||||
'ITEM_PIPELINES': {self.pipeline_class: 1},
|
||||
self.store_setting_key: self.tmpmediastore,
|
||||
}
|
||||
self.runner = CrawlerRunner(self.settings)
|
||||
self.items = []
|
||||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.tmpmediastore)
|
||||
self.items = []
|
||||
self.mockserver.__exit__(None, None, None)
|
||||
|
||||
def _on_item_scraped(self, item):
|
||||
self.items.append(item)
|
||||
|
||||
def _create_crawler(self, spider_class, **kwargs):
|
||||
crawler = self.runner.create_crawler(spider_class, **kwargs)
|
||||
crawler.signals.connect(self._on_item_scraped, signals.item_scraped)
|
||||
return crawler
|
||||
|
||||
def _assert_files_downloaded(self, items, logs):
|
||||
self.assertEqual(len(items), 1)
|
||||
self.assertIn(self.media_key, items[0])
|
||||
|
||||
# check that logs show the expected number of successful file downloads
|
||||
file_dl_success = 'File (downloaded): Downloaded file from'
|
||||
self.assertEqual(logs.count(file_dl_success), 3)
|
||||
|
||||
# check that the images/files checksums are what we know they should be
|
||||
if self.expected_checksums is not None:
|
||||
checksums = set(
|
||||
i['checksum']
|
||||
for item in items
|
||||
for i in item[self.media_key])
|
||||
self.assertEqual(checksums, self.expected_checksums)
|
||||
|
||||
# check that the image files where actually written to the media store
|
||||
for item in items:
|
||||
for i in item[self.media_key]:
|
||||
self.assertTrue(
|
||||
os.path.exists(
|
||||
os.path.join(self.tmpmediastore, i['path'])))
|
||||
|
||||
def _assert_files_download_failure(self, crawler, items, code, logs):
|
||||
|
||||
# check that the item does NOT have the "images/files" field populated
|
||||
self.assertEqual(len(items), 1)
|
||||
self.assertIn(self.media_key, items[0])
|
||||
self.assertFalse(items[0][self.media_key])
|
||||
|
||||
# check that there was 1 successful fetch and 3 other responses with non-200 code
|
||||
self.assertEqual(crawler.stats.get_value('downloader/request_method_count/GET'), 4)
|
||||
self.assertEqual(crawler.stats.get_value('downloader/response_count'), 4)
|
||||
self.assertEqual(crawler.stats.get_value('downloader/response_status_count/200'), 1)
|
||||
self.assertEqual(crawler.stats.get_value('downloader/response_status_count/%d' % code), 3)
|
||||
|
||||
# check that logs do show the failure on the file downloads
|
||||
file_dl_failure = 'File (code: %d): Error downloading file from' % code
|
||||
self.assertEqual(logs.count(file_dl_failure), 3)
|
||||
|
||||
# check that no files were written to the media store
|
||||
self.assertEqual(os.listdir(self.tmpmediastore), [])
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def test_download_media(self):
|
||||
crawler = self._create_crawler(MediaDownloadSpider)
|
||||
with LogCapture() as log:
|
||||
yield crawler.crawl("http://localhost:8998/files/images/",
|
||||
media_key=self.media_key,
|
||||
media_urls_key=self.media_urls_key)
|
||||
self._assert_files_downloaded(self.items, str(log))
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def test_download_media_wrong_urls(self):
|
||||
crawler = self._create_crawler(BrokenLinksMediaDownloadSpider)
|
||||
with LogCapture() as log:
|
||||
yield crawler.crawl("http://localhost:8998/files/images/",
|
||||
media_key=self.media_key,
|
||||
media_urls_key=self.media_urls_key)
|
||||
self._assert_files_download_failure(crawler, self.items, 404, str(log))
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def test_download_media_redirected_default_failure(self):
|
||||
crawler = self._create_crawler(RedirectedMediaDownloadSpider)
|
||||
with LogCapture() as log:
|
||||
yield crawler.crawl("http://localhost:8998/files/images/",
|
||||
media_key=self.media_key,
|
||||
media_urls_key=self.media_urls_key)
|
||||
self._assert_files_download_failure(crawler, self.items, 302, str(log))
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def test_download_media_redirected_allowed(self):
|
||||
settings = dict(self.settings)
|
||||
settings.update({'MEDIA_ALLOW_REDIRECTS': True})
|
||||
self.runner = CrawlerRunner(settings)
|
||||
|
||||
crawler = self._create_crawler(RedirectedMediaDownloadSpider)
|
||||
with LogCapture() as log:
|
||||
yield crawler.crawl("http://localhost:8998/files/images/",
|
||||
media_key=self.media_key,
|
||||
media_urls_key=self.media_urls_key)
|
||||
self._assert_files_downloaded(self.items, str(log))
|
||||
self.assertEqual(crawler.stats.get_value('downloader/response_status_count/302'), 3)
|
||||
|
||||
|
||||
class ImageDownloadCrawlTestCase(FileDownloadCrawlTestCase):
|
||||
pipeline_class = 'scrapy.pipelines.images.ImagesPipeline'
|
||||
store_setting_key = 'IMAGES_STORE'
|
||||
media_key = 'images'
|
||||
media_urls_key = 'image_urls'
|
||||
|
||||
# somehow checksums for images are different for Python 3.3
|
||||
expected_checksums = None
|
||||
|
|
@ -6,6 +6,7 @@ from twisted.internet import reactor
|
|||
from twisted.internet.defer import Deferred, inlineCallbacks
|
||||
|
||||
from scrapy.http import Request, Response
|
||||
from scrapy.settings import Settings
|
||||
from scrapy.spiders import Spider
|
||||
from scrapy.utils.request import request_fingerprint
|
||||
from scrapy.pipelines.media import MediaPipeline
|
||||
|
|
@ -22,10 +23,12 @@ def _mocked_download_func(request, info):
|
|||
class BaseMediaPipelineTestCase(unittest.TestCase):
|
||||
|
||||
pipeline_class = MediaPipeline
|
||||
settings = None
|
||||
|
||||
def setUp(self):
|
||||
self.spider = Spider('media.com')
|
||||
self.pipe = self.pipeline_class(download_func=_mocked_download_func)
|
||||
self.pipe = self.pipeline_class(download_func=_mocked_download_func,
|
||||
settings=Settings(self.settings))
|
||||
self.pipe.open_spider(self.spider)
|
||||
self.info = self.pipe.spiderinfo
|
||||
|
||||
|
|
@ -82,6 +85,11 @@ class BaseMediaPipelineTestCase(unittest.TestCase):
|
|||
new_item = yield self.pipe.process_item(item, self.spider)
|
||||
assert new_item is item
|
||||
|
||||
def test_modify_media_request(self):
|
||||
request = Request('http://url')
|
||||
self.pipe._modify_media_request(request)
|
||||
assert request.meta == {'handle_httpstatus_all': True}
|
||||
|
||||
|
||||
class MockedMediaPipeline(MediaPipeline):
|
||||
|
||||
|
|
@ -249,3 +257,61 @@ class MediaPipelineTestCase(BaseMediaPipelineTestCase):
|
|||
self.assertEqual(new_item['results'], [(True, 'ITSME')])
|
||||
self.assertEqual(self.pipe._mockcalled, \
|
||||
['get_media_requests', 'media_to_download', 'item_completed'])
|
||||
|
||||
|
||||
class MediaPipelineAllowRedirectSettingsTestCase(unittest.TestCase):
|
||||
|
||||
def _assert_request_no3xx(self, pipeline_class, settings):
|
||||
pipe = pipeline_class(settings=Settings(settings))
|
||||
request = Request('http://url')
|
||||
pipe._modify_media_request(request)
|
||||
|
||||
self.assertIn('handle_httpstatus_list', request.meta)
|
||||
for status, check in [
|
||||
(200, True),
|
||||
|
||||
# These are the status codes we want
|
||||
# the downloader to handle itself
|
||||
(301, False),
|
||||
(302, False),
|
||||
(302, False),
|
||||
(307, False),
|
||||
(308, False),
|
||||
|
||||
# we still want to get 4xx and 5xx
|
||||
(400, True),
|
||||
(404, True),
|
||||
(500, True)]:
|
||||
if check:
|
||||
self.assertIn(status, request.meta['handle_httpstatus_list'])
|
||||
else:
|
||||
self.assertNotIn(status, request.meta['handle_httpstatus_list'])
|
||||
|
||||
def test_standard_setting(self):
|
||||
self._assert_request_no3xx(
|
||||
MediaPipeline,
|
||||
{
|
||||
'MEDIA_ALLOW_REDIRECTS': True
|
||||
})
|
||||
|
||||
def test_subclass_standard_setting(self):
|
||||
|
||||
class UserDefinedPipeline(MediaPipeline):
|
||||
pass
|
||||
|
||||
self._assert_request_no3xx(
|
||||
UserDefinedPipeline,
|
||||
{
|
||||
'MEDIA_ALLOW_REDIRECTS': True
|
||||
})
|
||||
|
||||
def test_subclass_specific_setting(self):
|
||||
|
||||
class UserDefinedPipeline(MediaPipeline):
|
||||
pass
|
||||
|
||||
self._assert_request_no3xx(
|
||||
UserDefinedPipeline,
|
||||
{
|
||||
'USERDEFINEDPIPELINE_MEDIA_ALLOW_REDIRECTS': True
|
||||
})
|
||||
|
|
|
|||
|
|
@ -328,6 +328,10 @@ class SitemapSpiderTest(SpiderTest):
|
|||
r = Response(url="http://www.example.com/sitemap.xml.gz", body=self.GZBODY)
|
||||
self.assertSitemapBody(r, self.BODY)
|
||||
|
||||
# .xml.gz but body decoded by HttpCompression middleware already
|
||||
r = Response(url="http://www.example.com/sitemap.xml.gz", body=self.BODY)
|
||||
self.assertSitemapBody(r, self.BODY)
|
||||
|
||||
def test_get_sitemap_urls_from_robotstxt(self):
|
||||
robots = b"""# Sitemap files
|
||||
Sitemap: http://example.com/sitemap.xml
|
||||
|
|
|
|||
|
|
@ -91,13 +91,71 @@ class SpiderLoaderTest(unittest.TestCase):
|
|||
self.assertTrue(issubclass(crawler.spidercls, scrapy.Spider))
|
||||
self.assertEqual(crawler.spidercls.name, 'spider1')
|
||||
|
||||
def test_bad_spider_modules_exception(self):
|
||||
|
||||
module = 'tests.test_spiderloader.test_spiders.doesnotexist'
|
||||
settings = Settings({'SPIDER_MODULES': [module]})
|
||||
self.assertRaises(ImportError, SpiderLoader.from_settings, settings)
|
||||
|
||||
def test_bad_spider_modules_warning(self):
|
||||
|
||||
with warnings.catch_warnings(record=True) as w:
|
||||
module = 'tests.test_spiderloader.test_spiders.doesnotexist'
|
||||
settings = Settings({'SPIDER_MODULES': [module]})
|
||||
settings = Settings({'SPIDER_MODULES': [module],
|
||||
'SPIDER_LOADER_WARN_ONLY': True})
|
||||
spider_loader = SpiderLoader.from_settings(settings)
|
||||
self.assertIn("Could not load spiders from module", str(w[0].message))
|
||||
|
||||
spiders = spider_loader.list()
|
||||
self.assertEqual(spiders, [])
|
||||
|
||||
class DuplicateSpiderNameLoaderTest(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
orig_spiders_dir = os.path.join(module_dir, 'test_spiders')
|
||||
self.tmpdir = self.mktemp()
|
||||
os.mkdir(self.tmpdir)
|
||||
self.spiders_dir = os.path.join(self.tmpdir, 'test_spiders_xxx')
|
||||
shutil.copytree(orig_spiders_dir, self.spiders_dir)
|
||||
sys.path.append(self.tmpdir)
|
||||
self.settings = Settings({'SPIDER_MODULES': ['test_spiders_xxx']})
|
||||
|
||||
def tearDown(self):
|
||||
del sys.modules['test_spiders_xxx']
|
||||
sys.path.remove(self.tmpdir)
|
||||
|
||||
def test_dupename_warning(self):
|
||||
# copy 1 spider module so as to have duplicate spider name
|
||||
shutil.copyfile(os.path.join(self.tmpdir, 'test_spiders_xxx/spider3.py'),
|
||||
os.path.join(self.tmpdir, 'test_spiders_xxx/spider3dupe.py'))
|
||||
|
||||
with warnings.catch_warnings(record=True) as w:
|
||||
spider_loader = SpiderLoader.from_settings(self.settings)
|
||||
|
||||
self.assertEqual(len(w), 1)
|
||||
msg = str(w[0].message)
|
||||
self.assertIn("several spiders with the same name", msg)
|
||||
self.assertIn("'spider3'", msg)
|
||||
|
||||
spiders = set(spider_loader.list())
|
||||
self.assertEqual(spiders, set(['spider1', 'spider2', 'spider3', 'spider4']))
|
||||
|
||||
def test_multiple_dupename_warning(self):
|
||||
# copy 2 spider modules so as to have duplicate spider name
|
||||
# This should issue 2 warning, 1 for each duplicate spider name
|
||||
shutil.copyfile(os.path.join(self.tmpdir, 'test_spiders_xxx/spider1.py'),
|
||||
os.path.join(self.tmpdir, 'test_spiders_xxx/spider1dupe.py'))
|
||||
shutil.copyfile(os.path.join(self.tmpdir, 'test_spiders_xxx/spider2.py'),
|
||||
os.path.join(self.tmpdir, 'test_spiders_xxx/spider2dupe.py'))
|
||||
|
||||
with warnings.catch_warnings(record=True) as w:
|
||||
spider_loader = SpiderLoader.from_settings(self.settings)
|
||||
|
||||
self.assertEqual(len(w), 1)
|
||||
msg = str(w[0].message)
|
||||
self.assertIn("several spiders with the same name", msg)
|
||||
self.assertIn("'spider1'", msg)
|
||||
self.assertIn("'spider2'", msg)
|
||||
|
||||
spiders = set(spider_loader.list())
|
||||
self.assertEqual(spiders, set(['spider1', 'spider2', 'spider3', 'spider4']))
|
||||
|
|
|
|||
|
|
@ -1,21 +1,874 @@
|
|||
from six.moves.urllib.parse import urlparse
|
||||
from unittest import TestCase
|
||||
import warnings
|
||||
|
||||
from scrapy.exceptions import NotConfigured
|
||||
from scrapy.http import Response, Request
|
||||
from scrapy.settings import Settings
|
||||
from scrapy.spiders import Spider
|
||||
from scrapy.spidermiddlewares.referer import RefererMiddleware
|
||||
from scrapy.downloadermiddlewares.redirect import RedirectMiddleware
|
||||
from scrapy.spidermiddlewares.referer import RefererMiddleware, \
|
||||
POLICY_NO_REFERRER, POLICY_NO_REFERRER_WHEN_DOWNGRADE, \
|
||||
POLICY_SAME_ORIGIN, POLICY_ORIGIN, POLICY_ORIGIN_WHEN_CROSS_ORIGIN, \
|
||||
POLICY_SCRAPY_DEFAULT, POLICY_UNSAFE_URL, \
|
||||
POLICY_STRICT_ORIGIN, POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN, \
|
||||
DefaultReferrerPolicy, \
|
||||
NoReferrerPolicy, NoReferrerWhenDowngradePolicy, \
|
||||
OriginWhenCrossOriginPolicy, OriginPolicy, \
|
||||
StrictOriginWhenCrossOriginPolicy, StrictOriginPolicy, \
|
||||
SameOriginPolicy, UnsafeUrlPolicy, ReferrerPolicy
|
||||
|
||||
|
||||
class TestRefererMiddleware(TestCase):
|
||||
|
||||
req_meta = {}
|
||||
resp_headers = {}
|
||||
settings = {}
|
||||
scenarii = [
|
||||
('http://scrapytest.org', 'http://scrapytest.org/', b'http://scrapytest.org'),
|
||||
]
|
||||
|
||||
def setUp(self):
|
||||
self.spider = Spider('foo')
|
||||
self.mw = RefererMiddleware()
|
||||
settings = Settings(self.settings)
|
||||
self.mw = RefererMiddleware(settings)
|
||||
|
||||
def test_process_spider_output(self):
|
||||
res = Response('http://scrapytest.org')
|
||||
reqs = [Request('http://scrapytest.org/')]
|
||||
def get_request(self, target):
|
||||
return Request(target, meta=self.req_meta)
|
||||
|
||||
out = list(self.mw.process_spider_output(res, reqs, self.spider))
|
||||
self.assertEquals(out[0].headers.get('Referer'),
|
||||
b'http://scrapytest.org')
|
||||
def get_response(self, origin):
|
||||
return Response(origin, headers=self.resp_headers)
|
||||
|
||||
def test(self):
|
||||
|
||||
for origin, target, referrer in self.scenarii:
|
||||
response = self.get_response(origin)
|
||||
request = self.get_request(target)
|
||||
out = list(self.mw.process_spider_output(response, [request], self.spider))
|
||||
self.assertEquals(out[0].headers.get('Referer'), referrer)
|
||||
|
||||
|
||||
class MixinDefault(object):
|
||||
"""
|
||||
Based on https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade
|
||||
|
||||
with some additional filtering of s3://
|
||||
"""
|
||||
scenarii = [
|
||||
('https://example.com/', 'https://scrapy.org/', b'https://example.com/'),
|
||||
('http://example.com/', 'http://scrapy.org/', b'http://example.com/'),
|
||||
('http://example.com/', 'https://scrapy.org/', b'http://example.com/'),
|
||||
('https://example.com/', 'http://scrapy.org/', None),
|
||||
|
||||
# no credentials leak
|
||||
('http://user:password@example.com/', 'https://scrapy.org/', b'http://example.com/'),
|
||||
|
||||
# no referrer leak for local schemes
|
||||
('file:///home/path/to/somefile.html', 'https://scrapy.org/', None),
|
||||
('file:///home/path/to/somefile.html', 'http://scrapy.org/', None),
|
||||
|
||||
# no referrer leak for s3 origins
|
||||
('s3://mybucket/path/to/data.csv', 'https://scrapy.org/', None),
|
||||
('s3://mybucket/path/to/data.csv', 'http://scrapy.org/', None),
|
||||
]
|
||||
|
||||
|
||||
class MixinNoReferrer(object):
|
||||
scenarii = [
|
||||
('https://example.com/page.html', 'https://example.com/', None),
|
||||
('http://www.example.com/', 'https://scrapy.org/', None),
|
||||
('http://www.example.com/', 'http://scrapy.org/', None),
|
||||
('https://www.example.com/', 'http://scrapy.org/', None),
|
||||
('file:///home/path/to/somefile.html', 'http://scrapy.org/', None),
|
||||
]
|
||||
|
||||
|
||||
class MixinNoReferrerWhenDowngrade(object):
|
||||
scenarii = [
|
||||
# TLS to TLS: send non-empty referrer
|
||||
('https://example.com/page.html', 'https://not.example.com/', b'https://example.com/page.html'),
|
||||
('https://example.com/page.html', 'https://scrapy.org/', b'https://example.com/page.html'),
|
||||
('https://example.com:443/page.html', 'https://scrapy.org/', b'https://example.com/page.html'),
|
||||
('https://example.com:444/page.html', 'https://scrapy.org/', b'https://example.com:444/page.html'),
|
||||
('ftps://example.com/urls.zip', 'https://scrapy.org/', b'ftps://example.com/urls.zip'),
|
||||
|
||||
# TLS to non-TLS: do not send referrer
|
||||
('https://example.com/page.html', 'http://not.example.com/', None),
|
||||
('https://example.com/page.html', 'http://scrapy.org/', None),
|
||||
('ftps://example.com/urls.zip', 'http://scrapy.org/', None),
|
||||
|
||||
# non-TLS to TLS or non-TLS: send referrer
|
||||
('http://example.com/page.html', 'https://not.example.com/', b'http://example.com/page.html'),
|
||||
('http://example.com/page.html', 'https://scrapy.org/', b'http://example.com/page.html'),
|
||||
('http://example.com:8080/page.html', 'https://scrapy.org/', b'http://example.com:8080/page.html'),
|
||||
('http://example.com:80/page.html', 'http://not.example.com/', b'http://example.com/page.html'),
|
||||
('http://example.com/page.html', 'http://scrapy.org/', b'http://example.com/page.html'),
|
||||
('http://example.com:443/page.html', 'http://scrapy.org/', b'http://example.com:443/page.html'),
|
||||
('ftp://example.com/urls.zip', 'http://scrapy.org/', b'ftp://example.com/urls.zip'),
|
||||
('ftp://example.com/urls.zip', 'https://scrapy.org/', b'ftp://example.com/urls.zip'),
|
||||
|
||||
# test for user/password stripping
|
||||
('http://user:password@example.com/page.html', 'https://not.example.com/', b'http://example.com/page.html'),
|
||||
]
|
||||
|
||||
|
||||
class MixinSameOrigin(object):
|
||||
scenarii = [
|
||||
# Same origin (protocol, host, port): send referrer
|
||||
('https://example.com/page.html', 'https://example.com/not-page.html', b'https://example.com/page.html'),
|
||||
('http://example.com/page.html', 'http://example.com/not-page.html', b'http://example.com/page.html'),
|
||||
('https://example.com:443/page.html', 'https://example.com/not-page.html', b'https://example.com/page.html'),
|
||||
('http://example.com:80/page.html', 'http://example.com/not-page.html', b'http://example.com/page.html'),
|
||||
('http://example.com/page.html', 'http://example.com:80/not-page.html', b'http://example.com/page.html'),
|
||||
('http://example.com:8888/page.html', 'http://example.com:8888/not-page.html', b'http://example.com:8888/page.html'),
|
||||
|
||||
# Different host: do NOT send referrer
|
||||
('https://example.com/page.html', 'https://not.example.com/otherpage.html', None),
|
||||
('http://example.com/page.html', 'http://not.example.com/otherpage.html', None),
|
||||
('http://example.com/page.html', 'http://www.example.com/otherpage.html', None),
|
||||
|
||||
# Different port: do NOT send referrer
|
||||
('https://example.com:444/page.html', 'https://example.com/not-page.html', None),
|
||||
('http://example.com:81/page.html', 'http://example.com/not-page.html', None),
|
||||
('http://example.com/page.html', 'http://example.com:81/not-page.html', None),
|
||||
|
||||
# Different protocols: do NOT send refferer
|
||||
('https://example.com/page.html', 'http://example.com/not-page.html', None),
|
||||
('https://example.com/page.html', 'http://not.example.com/', None),
|
||||
('ftps://example.com/urls.zip', 'https://example.com/not-page.html', None),
|
||||
('ftp://example.com/urls.zip', 'http://example.com/not-page.html', None),
|
||||
('ftps://example.com/urls.zip', 'https://example.com/not-page.html', None),
|
||||
|
||||
# test for user/password stripping
|
||||
('https://user:password@example.com/page.html', 'https://example.com/not-page.html', b'https://example.com/page.html'),
|
||||
('https://user:password@example.com/page.html', 'http://example.com/not-page.html', None),
|
||||
]
|
||||
|
||||
|
||||
class MixinOrigin(object):
|
||||
scenarii = [
|
||||
# TLS or non-TLS to TLS or non-TLS: referrer origin is sent (yes, even for downgrades)
|
||||
('https://example.com/page.html', 'https://example.com/not-page.html', b'https://example.com/'),
|
||||
('https://example.com/page.html', 'https://scrapy.org', b'https://example.com/'),
|
||||
('https://example.com/page.html', 'http://scrapy.org', b'https://example.com/'),
|
||||
('http://example.com/page.html', 'http://scrapy.org', b'http://example.com/'),
|
||||
|
||||
# test for user/password stripping
|
||||
('https://user:password@example.com/page.html', 'http://scrapy.org', b'https://example.com/'),
|
||||
]
|
||||
|
||||
|
||||
class MixinStrictOrigin(object):
|
||||
scenarii = [
|
||||
# TLS or non-TLS to TLS or non-TLS: referrer origin is sent but not for downgrades
|
||||
('https://example.com/page.html', 'https://example.com/not-page.html', b'https://example.com/'),
|
||||
('https://example.com/page.html', 'https://scrapy.org', b'https://example.com/'),
|
||||
('http://example.com/page.html', 'http://scrapy.org', b'http://example.com/'),
|
||||
|
||||
# downgrade: send nothing
|
||||
('https://example.com/page.html', 'http://scrapy.org', None),
|
||||
|
||||
# upgrade: send origin
|
||||
('http://example.com/page.html', 'https://scrapy.org', b'http://example.com/'),
|
||||
|
||||
# test for user/password stripping
|
||||
('https://user:password@example.com/page.html', 'https://scrapy.org', b'https://example.com/'),
|
||||
('https://user:password@example.com/page.html', 'http://scrapy.org', None),
|
||||
]
|
||||
|
||||
|
||||
class MixinOriginWhenCrossOrigin(object):
|
||||
scenarii = [
|
||||
# Same origin (protocol, host, port): send referrer
|
||||
('https://example.com/page.html', 'https://example.com/not-page.html', b'https://example.com/page.html'),
|
||||
('http://example.com/page.html', 'http://example.com/not-page.html', b'http://example.com/page.html'),
|
||||
('https://example.com:443/page.html', 'https://example.com/not-page.html', b'https://example.com/page.html'),
|
||||
('http://example.com:80/page.html', 'http://example.com/not-page.html', b'http://example.com/page.html'),
|
||||
('http://example.com/page.html', 'http://example.com:80/not-page.html', b'http://example.com/page.html'),
|
||||
('http://example.com:8888/page.html', 'http://example.com:8888/not-page.html', b'http://example.com:8888/page.html'),
|
||||
|
||||
# Different host: send origin as referrer
|
||||
('https://example2.com/page.html', 'https://scrapy.org/otherpage.html', b'https://example2.com/'),
|
||||
('https://example2.com/page.html', 'https://not.example2.com/otherpage.html', b'https://example2.com/'),
|
||||
('http://example2.com/page.html', 'http://not.example2.com/otherpage.html', b'http://example2.com/'),
|
||||
# exact match required
|
||||
('http://example2.com/page.html', 'http://www.example2.com/otherpage.html', b'http://example2.com/'),
|
||||
|
||||
# Different port: send origin as referrer
|
||||
('https://example3.com:444/page.html', 'https://example3.com/not-page.html', b'https://example3.com:444/'),
|
||||
('http://example3.com:81/page.html', 'http://example3.com/not-page.html', b'http://example3.com:81/'),
|
||||
|
||||
# Different protocols: send origin as referrer
|
||||
('https://example4.com/page.html', 'http://example4.com/not-page.html', b'https://example4.com/'),
|
||||
('https://example4.com/page.html', 'http://not.example4.com/', b'https://example4.com/'),
|
||||
('ftps://example4.com/urls.zip', 'https://example4.com/not-page.html', b'ftps://example4.com/'),
|
||||
('ftp://example4.com/urls.zip', 'http://example4.com/not-page.html', b'ftp://example4.com/'),
|
||||
('ftps://example4.com/urls.zip', 'https://example4.com/not-page.html', b'ftps://example4.com/'),
|
||||
|
||||
# test for user/password stripping
|
||||
('https://user:password@example5.com/page.html', 'https://example5.com/not-page.html', b'https://example5.com/page.html'),
|
||||
# TLS to non-TLS downgrade: send origin
|
||||
('https://user:password@example5.com/page.html', 'http://example5.com/not-page.html', b'https://example5.com/'),
|
||||
]
|
||||
|
||||
|
||||
class MixinStrictOriginWhenCrossOrigin(object):
|
||||
scenarii = [
|
||||
# Same origin (protocol, host, port): send referrer
|
||||
('https://example.com/page.html', 'https://example.com/not-page.html', b'https://example.com/page.html'),
|
||||
('http://example.com/page.html', 'http://example.com/not-page.html', b'http://example.com/page.html'),
|
||||
('https://example.com:443/page.html', 'https://example.com/not-page.html', b'https://example.com/page.html'),
|
||||
('http://example.com:80/page.html', 'http://example.com/not-page.html', b'http://example.com/page.html'),
|
||||
('http://example.com/page.html', 'http://example.com:80/not-page.html', b'http://example.com/page.html'),
|
||||
('http://example.com:8888/page.html', 'http://example.com:8888/not-page.html', b'http://example.com:8888/page.html'),
|
||||
|
||||
# Different host: send origin as referrer
|
||||
('https://example2.com/page.html', 'https://scrapy.org/otherpage.html', b'https://example2.com/'),
|
||||
('https://example2.com/page.html', 'https://not.example2.com/otherpage.html', b'https://example2.com/'),
|
||||
('http://example2.com/page.html', 'http://not.example2.com/otherpage.html', b'http://example2.com/'),
|
||||
# exact match required
|
||||
('http://example2.com/page.html', 'http://www.example2.com/otherpage.html', b'http://example2.com/'),
|
||||
|
||||
# Different port: send origin as referrer
|
||||
('https://example3.com:444/page.html', 'https://example3.com/not-page.html', b'https://example3.com:444/'),
|
||||
('http://example3.com:81/page.html', 'http://example3.com/not-page.html', b'http://example3.com:81/'),
|
||||
|
||||
# downgrade
|
||||
('https://example4.com/page.html', 'http://example4.com/not-page.html', None),
|
||||
('https://example4.com/page.html', 'http://not.example4.com/', None),
|
||||
|
||||
# non-TLS to non-TLS
|
||||
('ftp://example4.com/urls.zip', 'http://example4.com/not-page.html', b'ftp://example4.com/'),
|
||||
|
||||
# upgrade
|
||||
('http://example4.com/page.html', 'https://example4.com/not-page.html', b'http://example4.com/'),
|
||||
('http://example4.com/page.html', 'https://not.example4.com/', b'http://example4.com/'),
|
||||
|
||||
# Different protocols: send origin as referrer
|
||||
('ftps://example4.com/urls.zip', 'https://example4.com/not-page.html', b'ftps://example4.com/'),
|
||||
('ftps://example4.com/urls.zip', 'https://example4.com/not-page.html', b'ftps://example4.com/'),
|
||||
|
||||
# test for user/password stripping
|
||||
('https://user:password@example5.com/page.html', 'https://example5.com/not-page.html', b'https://example5.com/page.html'),
|
||||
|
||||
# TLS to non-TLS downgrade: send nothing
|
||||
('https://user:password@example5.com/page.html', 'http://example5.com/not-page.html', None),
|
||||
]
|
||||
|
||||
|
||||
class MixinUnsafeUrl(object):
|
||||
scenarii = [
|
||||
# TLS to TLS: send referrer
|
||||
('https://example.com/sekrit.html', 'http://not.example.com/', b'https://example.com/sekrit.html'),
|
||||
('https://example1.com/page.html', 'https://not.example1.com/', b'https://example1.com/page.html'),
|
||||
('https://example1.com/page.html', 'https://scrapy.org/', b'https://example1.com/page.html'),
|
||||
('https://example1.com:443/page.html', 'https://scrapy.org/', b'https://example1.com/page.html'),
|
||||
('https://example1.com:444/page.html', 'https://scrapy.org/', b'https://example1.com:444/page.html'),
|
||||
('ftps://example1.com/urls.zip', 'https://scrapy.org/', b'ftps://example1.com/urls.zip'),
|
||||
|
||||
# TLS to non-TLS: send referrer (yes, it's unsafe)
|
||||
('https://example2.com/page.html', 'http://not.example2.com/', b'https://example2.com/page.html'),
|
||||
('https://example2.com/page.html', 'http://scrapy.org/', b'https://example2.com/page.html'),
|
||||
('ftps://example2.com/urls.zip', 'http://scrapy.org/', b'ftps://example2.com/urls.zip'),
|
||||
|
||||
# non-TLS to TLS or non-TLS: send referrer (yes, it's unsafe)
|
||||
('http://example3.com/page.html', 'https://not.example3.com/', b'http://example3.com/page.html'),
|
||||
('http://example3.com/page.html', 'https://scrapy.org/', b'http://example3.com/page.html'),
|
||||
('http://example3.com:8080/page.html', 'https://scrapy.org/', b'http://example3.com:8080/page.html'),
|
||||
('http://example3.com:80/page.html', 'http://not.example3.com/', b'http://example3.com/page.html'),
|
||||
('http://example3.com/page.html', 'http://scrapy.org/', b'http://example3.com/page.html'),
|
||||
('http://example3.com:443/page.html', 'http://scrapy.org/', b'http://example3.com:443/page.html'),
|
||||
('ftp://example3.com/urls.zip', 'http://scrapy.org/', b'ftp://example3.com/urls.zip'),
|
||||
('ftp://example3.com/urls.zip', 'https://scrapy.org/', b'ftp://example3.com/urls.zip'),
|
||||
|
||||
# test for user/password stripping
|
||||
('http://user:password@example4.com/page.html', 'https://not.example4.com/', b'http://example4.com/page.html'),
|
||||
('https://user:password@example4.com/page.html', 'http://scrapy.org/', b'https://example4.com/page.html'),
|
||||
]
|
||||
|
||||
|
||||
class TestRefererMiddlewareDefault(MixinDefault, TestRefererMiddleware):
|
||||
pass
|
||||
|
||||
|
||||
# --- Tests using settings to set policy using class path
|
||||
class TestSettingsNoReferrer(MixinNoReferrer, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.NoReferrerPolicy'}
|
||||
|
||||
|
||||
class TestSettingsNoReferrerWhenDowngrade(MixinNoReferrerWhenDowngrade, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy'}
|
||||
|
||||
|
||||
class TestSettingsSameOrigin(MixinSameOrigin, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.SameOriginPolicy'}
|
||||
|
||||
|
||||
class TestSettingsOrigin(MixinOrigin, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.OriginPolicy'}
|
||||
|
||||
|
||||
class TestSettingsStrictOrigin(MixinStrictOrigin, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.StrictOriginPolicy'}
|
||||
|
||||
|
||||
class TestSettingsOriginWhenCrossOrigin(MixinOriginWhenCrossOrigin, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy'}
|
||||
|
||||
|
||||
class TestSettingsStrictOriginWhenCrossOrigin(MixinStrictOriginWhenCrossOrigin, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy'}
|
||||
|
||||
|
||||
class TestSettingsUnsafeUrl(MixinUnsafeUrl, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.UnsafeUrlPolicy'}
|
||||
|
||||
|
||||
class CustomPythonOrgPolicy(ReferrerPolicy):
|
||||
"""
|
||||
A dummy policy that returns referrer as http(s)://python.org
|
||||
depending on the scheme of the target URL.
|
||||
"""
|
||||
def referrer(self, response, request):
|
||||
scheme = urlparse(request).scheme
|
||||
if scheme == 'https':
|
||||
return b'https://python.org/'
|
||||
elif scheme == 'http':
|
||||
return b'http://python.org/'
|
||||
|
||||
|
||||
class TestSettingsCustomPolicy(TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'tests.test_spidermiddleware_referer.CustomPythonOrgPolicy'}
|
||||
scenarii = [
|
||||
('https://example.com/', 'https://scrapy.org/', b'https://python.org/'),
|
||||
('http://example.com/', 'http://scrapy.org/', b'http://python.org/'),
|
||||
('http://example.com/', 'https://scrapy.org/', b'https://python.org/'),
|
||||
('https://example.com/', 'http://scrapy.org/', b'http://python.org/'),
|
||||
('file:///home/path/to/somefile.html', 'https://scrapy.org/', b'https://python.org/'),
|
||||
('file:///home/path/to/somefile.html', 'http://scrapy.org/', b'http://python.org/'),
|
||||
|
||||
]
|
||||
|
||||
# --- Tests using Request meta dict to set policy
|
||||
class TestRequestMetaDefault(MixinDefault, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_SCRAPY_DEFAULT}
|
||||
|
||||
|
||||
class TestRequestMetaNoReferrer(MixinNoReferrer, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_NO_REFERRER}
|
||||
|
||||
|
||||
class TestRequestMetaNoReferrerWhenDowngrade(MixinNoReferrerWhenDowngrade, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_NO_REFERRER_WHEN_DOWNGRADE}
|
||||
|
||||
|
||||
class TestRequestMetaSameOrigin(MixinSameOrigin, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_SAME_ORIGIN}
|
||||
|
||||
|
||||
class TestRequestMetaOrigin(MixinOrigin, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_ORIGIN}
|
||||
|
||||
|
||||
class TestRequestMetaSrictOrigin(MixinStrictOrigin, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_STRICT_ORIGIN}
|
||||
|
||||
|
||||
class TestRequestMetaOriginWhenCrossOrigin(MixinOriginWhenCrossOrigin, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_ORIGIN_WHEN_CROSS_ORIGIN}
|
||||
|
||||
|
||||
class TestRequestMetaStrictOriginWhenCrossOrigin(MixinStrictOriginWhenCrossOrigin, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN}
|
||||
|
||||
|
||||
class TestRequestMetaUnsafeUrl(MixinUnsafeUrl, TestRefererMiddleware):
|
||||
req_meta = {'referrer_policy': POLICY_UNSAFE_URL}
|
||||
|
||||
|
||||
class TestRequestMetaPredecence001(MixinUnsafeUrl, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.SameOriginPolicy'}
|
||||
req_meta = {'referrer_policy': POLICY_UNSAFE_URL}
|
||||
|
||||
|
||||
class TestRequestMetaPredecence002(MixinNoReferrer, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy'}
|
||||
req_meta = {'referrer_policy': POLICY_NO_REFERRER}
|
||||
|
||||
|
||||
class TestRequestMetaPredecence003(MixinUnsafeUrl, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy'}
|
||||
req_meta = {'referrer_policy': POLICY_UNSAFE_URL}
|
||||
|
||||
|
||||
class TestRequestMetaSettingFallback(TestCase):
|
||||
|
||||
params = [
|
||||
(
|
||||
# When an unknown policy is referenced in Request.meta
|
||||
# (here, a typo error),
|
||||
# the policy defined in settings takes precedence
|
||||
{'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy'},
|
||||
{},
|
||||
{'referrer_policy': 'ssscrapy-default'},
|
||||
OriginWhenCrossOriginPolicy,
|
||||
True
|
||||
),
|
||||
(
|
||||
# same as above but with string value for settings policy
|
||||
{'REFERRER_POLICY': 'origin-when-cross-origin'},
|
||||
{},
|
||||
{'referrer_policy': 'ssscrapy-default'},
|
||||
OriginWhenCrossOriginPolicy,
|
||||
True
|
||||
),
|
||||
(
|
||||
# request meta references a wrong policy but it is set,
|
||||
# so the Referrer-Policy header in response is not used,
|
||||
# and the settings' policy is applied
|
||||
{'REFERRER_POLICY': 'origin-when-cross-origin'},
|
||||
{'Referrer-Policy': 'unsafe-url'},
|
||||
{'referrer_policy': 'ssscrapy-default'},
|
||||
OriginWhenCrossOriginPolicy,
|
||||
True
|
||||
),
|
||||
(
|
||||
# here, request meta does not set the policy
|
||||
# so response headers take precedence
|
||||
{'REFERRER_POLICY': 'origin-when-cross-origin'},
|
||||
{'Referrer-Policy': 'unsafe-url'},
|
||||
{},
|
||||
UnsafeUrlPolicy,
|
||||
False
|
||||
),
|
||||
(
|
||||
# here, request meta does not set the policy,
|
||||
# but response headers also use an unknown policy,
|
||||
# so the settings' policy is used
|
||||
{'REFERRER_POLICY': 'origin-when-cross-origin'},
|
||||
{'Referrer-Policy': 'unknown'},
|
||||
{},
|
||||
OriginWhenCrossOriginPolicy,
|
||||
True
|
||||
)
|
||||
]
|
||||
|
||||
def test(self):
|
||||
|
||||
origin = 'http://www.scrapy.org'
|
||||
target = 'http://www.example.com'
|
||||
|
||||
for settings, response_headers, request_meta, policy_class, check_warning in self.params[3:]:
|
||||
spider = Spider('foo')
|
||||
mw = RefererMiddleware(Settings(settings))
|
||||
|
||||
response = Response(origin, headers=response_headers)
|
||||
request = Request(target, meta=request_meta)
|
||||
|
||||
with warnings.catch_warnings(record=True) as w:
|
||||
policy = mw.policy(response, request)
|
||||
self.assertIsInstance(policy, policy_class)
|
||||
|
||||
if check_warning:
|
||||
self.assertEqual(len(w), 1)
|
||||
self.assertEqual(w[0].category, RuntimeWarning, w[0].message)
|
||||
|
||||
|
||||
class TestSettingsPolicyByName(TestCase):
|
||||
|
||||
def test_valid_name(self):
|
||||
for s, p in [
|
||||
(POLICY_SCRAPY_DEFAULT, DefaultReferrerPolicy),
|
||||
(POLICY_NO_REFERRER, NoReferrerPolicy),
|
||||
(POLICY_NO_REFERRER_WHEN_DOWNGRADE, NoReferrerWhenDowngradePolicy),
|
||||
(POLICY_SAME_ORIGIN, SameOriginPolicy),
|
||||
(POLICY_ORIGIN, OriginPolicy),
|
||||
(POLICY_STRICT_ORIGIN, StrictOriginPolicy),
|
||||
(POLICY_ORIGIN_WHEN_CROSS_ORIGIN, OriginWhenCrossOriginPolicy),
|
||||
(POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN, StrictOriginWhenCrossOriginPolicy),
|
||||
(POLICY_UNSAFE_URL, UnsafeUrlPolicy),
|
||||
]:
|
||||
settings = Settings({'REFERRER_POLICY': s})
|
||||
mw = RefererMiddleware(settings)
|
||||
self.assertEquals(mw.default_policy, p)
|
||||
|
||||
def test_valid_name_casevariants(self):
|
||||
for s, p in [
|
||||
(POLICY_SCRAPY_DEFAULT, DefaultReferrerPolicy),
|
||||
(POLICY_NO_REFERRER, NoReferrerPolicy),
|
||||
(POLICY_NO_REFERRER_WHEN_DOWNGRADE, NoReferrerWhenDowngradePolicy),
|
||||
(POLICY_SAME_ORIGIN, SameOriginPolicy),
|
||||
(POLICY_ORIGIN, OriginPolicy),
|
||||
(POLICY_STRICT_ORIGIN, StrictOriginPolicy),
|
||||
(POLICY_ORIGIN_WHEN_CROSS_ORIGIN, OriginWhenCrossOriginPolicy),
|
||||
(POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN, StrictOriginWhenCrossOriginPolicy),
|
||||
(POLICY_UNSAFE_URL, UnsafeUrlPolicy),
|
||||
]:
|
||||
settings = Settings({'REFERRER_POLICY': s.upper()})
|
||||
mw = RefererMiddleware(settings)
|
||||
self.assertEquals(mw.default_policy, p)
|
||||
|
||||
def test_invalid_name(self):
|
||||
settings = Settings({'REFERRER_POLICY': 'some-custom-unknown-policy'})
|
||||
with self.assertRaises(RuntimeError):
|
||||
mw = RefererMiddleware(settings)
|
||||
|
||||
|
||||
class TestPolicyHeaderPredecence001(MixinUnsafeUrl, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.SameOriginPolicy'}
|
||||
resp_headers = {'Referrer-Policy': POLICY_UNSAFE_URL.upper()}
|
||||
|
||||
class TestPolicyHeaderPredecence002(MixinNoReferrer, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy'}
|
||||
resp_headers = {'Referrer-Policy': POLICY_NO_REFERRER.swapcase()}
|
||||
|
||||
class TestPolicyHeaderPredecence003(MixinNoReferrerWhenDowngrade, TestRefererMiddleware):
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy'}
|
||||
resp_headers = {'Referrer-Policy': POLICY_NO_REFERRER_WHEN_DOWNGRADE.title()}
|
||||
|
||||
class TestPolicyHeaderPredecence004(MixinNoReferrerWhenDowngrade, TestRefererMiddleware):
|
||||
"""
|
||||
The empty string means "no-referrer-when-downgrade"
|
||||
"""
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy'}
|
||||
resp_headers = {'Referrer-Policy': ''}
|
||||
|
||||
|
||||
class TestReferrerOnRedirect(TestRefererMiddleware):
|
||||
|
||||
settings = {'REFERRER_POLICY': 'scrapy.spidermiddlewares.referer.UnsafeUrlPolicy'}
|
||||
scenarii = [
|
||||
( 'http://scrapytest.org/1', # parent
|
||||
'http://scrapytest.org/2', # target
|
||||
(
|
||||
# redirections: code, URL
|
||||
(301, 'http://scrapytest.org/3'),
|
||||
(301, 'http://scrapytest.org/4'),
|
||||
),
|
||||
b'http://scrapytest.org/1', # expected initial referer
|
||||
b'http://scrapytest.org/1', # expected referer for the redirection request
|
||||
),
|
||||
( 'https://scrapytest.org/1',
|
||||
'https://scrapytest.org/2',
|
||||
(
|
||||
# redirecting to non-secure URL
|
||||
(301, 'http://scrapytest.org/3'),
|
||||
),
|
||||
b'https://scrapytest.org/1',
|
||||
b'https://scrapytest.org/1',
|
||||
),
|
||||
( 'https://scrapytest.org/1',
|
||||
'https://scrapytest.com/2',
|
||||
(
|
||||
# redirecting to non-secure URL: different origin
|
||||
(301, 'http://scrapytest.com/3'),
|
||||
),
|
||||
b'https://scrapytest.org/1',
|
||||
b'https://scrapytest.org/1',
|
||||
),
|
||||
]
|
||||
|
||||
def setUp(self):
|
||||
self.spider = Spider('foo')
|
||||
settings = Settings(self.settings)
|
||||
self.referrermw = RefererMiddleware(settings)
|
||||
self.redirectmw = RedirectMiddleware(settings)
|
||||
|
||||
def test(self):
|
||||
|
||||
for parent, target, redirections, init_referrer, final_referrer in self.scenarii:
|
||||
response = self.get_response(parent)
|
||||
request = self.get_request(target)
|
||||
|
||||
out = list(self.referrermw.process_spider_output(response, [request], self.spider))
|
||||
self.assertEquals(out[0].headers.get('Referer'), init_referrer)
|
||||
|
||||
for status, url in redirections:
|
||||
response = Response(request.url, headers={'Location': url}, status=status)
|
||||
request = self.redirectmw.process_response(request, response, self.spider)
|
||||
self.referrermw.request_scheduled(request, self.spider)
|
||||
|
||||
assert isinstance(request, Request)
|
||||
self.assertEquals(request.headers.get('Referer'), final_referrer)
|
||||
|
||||
|
||||
class TestReferrerOnRedirectNoReferrer(TestReferrerOnRedirect):
|
||||
"""
|
||||
No Referrer policy never sets the "Referer" header.
|
||||
HTTP redirections should not change that.
|
||||
"""
|
||||
settings = {'REFERRER_POLICY': 'no-referrer'}
|
||||
scenarii = [
|
||||
( 'http://scrapytest.org/1', # parent
|
||||
'http://scrapytest.org/2', # target
|
||||
(
|
||||
# redirections: code, URL
|
||||
(301, 'http://scrapytest.org/3'),
|
||||
(301, 'http://scrapytest.org/4'),
|
||||
),
|
||||
None, # expected initial "Referer"
|
||||
None, # expected "Referer" for the redirection request
|
||||
),
|
||||
( 'https://scrapytest.org/1',
|
||||
'https://scrapytest.org/2',
|
||||
(
|
||||
(301, 'http://scrapytest.org/3'),
|
||||
),
|
||||
None,
|
||||
None,
|
||||
),
|
||||
( 'https://scrapytest.org/1',
|
||||
'https://example.com/2', # different origin
|
||||
(
|
||||
(301, 'http://scrapytest.com/3'),
|
||||
),
|
||||
None,
|
||||
None,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
class TestReferrerOnRedirectSameOrigin(TestReferrerOnRedirect):
|
||||
"""
|
||||
Same Origin policy sends the full URL as "Referer" if the target origin
|
||||
is the same as the parent response (same protocol, same domain, same port).
|
||||
|
||||
HTTP redirections to a different domain or a lower secure level
|
||||
should have the "Referer" removed.
|
||||
"""
|
||||
settings = {'REFERRER_POLICY': 'same-origin'}
|
||||
scenarii = [
|
||||
( 'http://scrapytest.org/101', # origin
|
||||
'http://scrapytest.org/102', # target
|
||||
(
|
||||
# redirections: code, URL
|
||||
(301, 'http://scrapytest.org/103'),
|
||||
(301, 'http://scrapytest.org/104'),
|
||||
),
|
||||
b'http://scrapytest.org/101', # expected initial "Referer"
|
||||
b'http://scrapytest.org/101', # expected referer for the redirection request
|
||||
),
|
||||
( 'https://scrapytest.org/201',
|
||||
'https://scrapytest.org/202',
|
||||
(
|
||||
# redirecting from secure to non-secure URL == different origin
|
||||
(301, 'http://scrapytest.org/203'),
|
||||
),
|
||||
b'https://scrapytest.org/201',
|
||||
None,
|
||||
),
|
||||
( 'https://scrapytest.org/301',
|
||||
'https://scrapytest.org/302',
|
||||
(
|
||||
# different domain == different origin
|
||||
(301, 'http://example.com/303'),
|
||||
),
|
||||
b'https://scrapytest.org/301',
|
||||
None,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
class TestReferrerOnRedirectStrictOrigin(TestReferrerOnRedirect):
|
||||
"""
|
||||
Strict Origin policy will always send the "origin" as referrer
|
||||
(think of it as the parent URL without the path part),
|
||||
unless the security level is lower and no "Referer" is sent.
|
||||
|
||||
Redirections from secure to non-secure URLs should have the
|
||||
"Referrer" header removed if necessary.
|
||||
"""
|
||||
settings = {'REFERRER_POLICY': POLICY_STRICT_ORIGIN}
|
||||
scenarii = [
|
||||
( 'http://scrapytest.org/101',
|
||||
'http://scrapytest.org/102',
|
||||
(
|
||||
(301, 'http://scrapytest.org/103'),
|
||||
(301, 'http://scrapytest.org/104'),
|
||||
),
|
||||
b'http://scrapytest.org/', # send origin
|
||||
b'http://scrapytest.org/', # redirects to same origin: send origin
|
||||
),
|
||||
( 'https://scrapytest.org/201',
|
||||
'https://scrapytest.org/202',
|
||||
(
|
||||
# redirecting to non-secure URL: no referrer
|
||||
(301, 'http://scrapytest.org/203'),
|
||||
),
|
||||
b'https://scrapytest.org/',
|
||||
None,
|
||||
),
|
||||
( 'https://scrapytest.org/301',
|
||||
'https://scrapytest.org/302',
|
||||
(
|
||||
# redirecting to non-secure URL (different domain): no referrer
|
||||
(301, 'http://example.com/303'),
|
||||
),
|
||||
b'https://scrapytest.org/',
|
||||
None,
|
||||
),
|
||||
( 'http://scrapy.org/401',
|
||||
'http://example.com/402',
|
||||
(
|
||||
(301, 'http://scrapytest.org/403'),
|
||||
),
|
||||
b'http://scrapy.org/',
|
||||
b'http://scrapy.org/',
|
||||
),
|
||||
( 'https://scrapy.org/501',
|
||||
'https://example.com/502',
|
||||
(
|
||||
# HTTPS all along, so origin referrer is kept as-is
|
||||
(301, 'https://google.com/503'),
|
||||
(301, 'https://facebook.com/504'),
|
||||
),
|
||||
b'https://scrapy.org/',
|
||||
b'https://scrapy.org/',
|
||||
),
|
||||
( 'https://scrapytest.org/601',
|
||||
'http://scrapytest.org/602', # TLS to non-TLS: no referrer
|
||||
(
|
||||
(301, 'https://scrapytest.org/603'), # TLS URL again: (still) no referrer
|
||||
),
|
||||
None,
|
||||
None,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
class TestReferrerOnRedirectOriginWhenCrossOrigin(TestReferrerOnRedirect):
|
||||
"""
|
||||
Origin When Cross-Origin policy sends the full URL as "Referer",
|
||||
unless the target's origin is different (different domain, different protocol)
|
||||
in which case only the origin is sent.
|
||||
|
||||
Redirections to a different origin should strip the "Referer"
|
||||
to the parent origin.
|
||||
"""
|
||||
settings = {'REFERRER_POLICY': POLICY_ORIGIN_WHEN_CROSS_ORIGIN}
|
||||
scenarii = [
|
||||
( 'http://scrapytest.org/101', # origin
|
||||
'http://scrapytest.org/102', # target + redirection
|
||||
(
|
||||
# redirections: code, URL
|
||||
(301, 'http://scrapytest.org/103'),
|
||||
(301, 'http://scrapytest.org/104'),
|
||||
),
|
||||
b'http://scrapytest.org/101', # expected initial referer
|
||||
b'http://scrapytest.org/101', # expected referer for the redirection request
|
||||
),
|
||||
( 'https://scrapytest.org/201',
|
||||
'https://scrapytest.org/202',
|
||||
(
|
||||
# redirecting to non-secure URL: send origin
|
||||
(301, 'http://scrapytest.org/203'),
|
||||
),
|
||||
b'https://scrapytest.org/201',
|
||||
b'https://scrapytest.org/',
|
||||
),
|
||||
( 'https://scrapytest.org/301',
|
||||
'https://scrapytest.org/302',
|
||||
(
|
||||
# redirecting to non-secure URL (different domain): send origin
|
||||
(301, 'http://example.com/303'),
|
||||
),
|
||||
b'https://scrapytest.org/301',
|
||||
b'https://scrapytest.org/',
|
||||
),
|
||||
( 'http://scrapy.org/401',
|
||||
'http://example.com/402',
|
||||
(
|
||||
(301, 'http://scrapytest.org/403'),
|
||||
),
|
||||
b'http://scrapy.org/',
|
||||
b'http://scrapy.org/',
|
||||
),
|
||||
( 'https://scrapy.org/501',
|
||||
'https://example.com/502',
|
||||
(
|
||||
# all different domains: send origin
|
||||
(301, 'https://google.com/503'),
|
||||
(301, 'https://facebook.com/504'),
|
||||
),
|
||||
b'https://scrapy.org/',
|
||||
b'https://scrapy.org/',
|
||||
),
|
||||
( 'https://scrapytest.org/301',
|
||||
'http://scrapytest.org/302', # TLS to non-TLS: send origin
|
||||
(
|
||||
(301, 'https://scrapytest.org/303'), # TLS URL again: send origin (also)
|
||||
),
|
||||
b'https://scrapytest.org/',
|
||||
b'https://scrapytest.org/',
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
class TestReferrerOnRedirectStrictOriginWhenCrossOrigin(TestReferrerOnRedirect):
|
||||
"""
|
||||
Strict Origin When Cross-Origin policy sends the full URL as "Referer",
|
||||
unless the target's origin is different (different domain, different protocol)
|
||||
in which case only the origin is sent...
|
||||
Unless there's also a downgrade in security and then the "Referer" header
|
||||
is not sent.
|
||||
|
||||
Redirections to a different origin should strip the "Referer" to the parent origin,
|
||||
and from https:// to http:// will remove the "Referer" header.
|
||||
"""
|
||||
settings = {'REFERRER_POLICY': POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN}
|
||||
scenarii = [
|
||||
( 'http://scrapytest.org/101', # origin
|
||||
'http://scrapytest.org/102', # target + redirection
|
||||
(
|
||||
# redirections: code, URL
|
||||
(301, 'http://scrapytest.org/103'),
|
||||
(301, 'http://scrapytest.org/104'),
|
||||
),
|
||||
b'http://scrapytest.org/101', # expected initial referer
|
||||
b'http://scrapytest.org/101', # expected referer for the redirection request
|
||||
),
|
||||
( 'https://scrapytest.org/201',
|
||||
'https://scrapytest.org/202',
|
||||
(
|
||||
# redirecting to non-secure URL: do not send the "Referer" header
|
||||
(301, 'http://scrapytest.org/203'),
|
||||
),
|
||||
b'https://scrapytest.org/201',
|
||||
None,
|
||||
),
|
||||
( 'https://scrapytest.org/301',
|
||||
'https://scrapytest.org/302',
|
||||
(
|
||||
# redirecting to non-secure URL (different domain): send origin
|
||||
(301, 'http://example.com/303'),
|
||||
),
|
||||
b'https://scrapytest.org/301',
|
||||
None,
|
||||
),
|
||||
( 'http://scrapy.org/401',
|
||||
'http://example.com/402',
|
||||
(
|
||||
(301, 'http://scrapytest.org/403'),
|
||||
),
|
||||
b'http://scrapy.org/',
|
||||
b'http://scrapy.org/',
|
||||
),
|
||||
( 'https://scrapy.org/501',
|
||||
'https://example.com/502',
|
||||
(
|
||||
# all different domains: send origin
|
||||
(301, 'https://google.com/503'),
|
||||
(301, 'https://facebook.com/504'),
|
||||
),
|
||||
b'https://scrapy.org/',
|
||||
b'https://scrapy.org/',
|
||||
),
|
||||
( 'https://scrapytest.org/601',
|
||||
'http://scrapytest.org/602', # TLS to non-TLS: do not send "Referer"
|
||||
(
|
||||
(301, 'https://scrapytest.org/603'), # TLS URL again: (still) send nothing
|
||||
),
|
||||
None,
|
||||
None,
|
||||
),
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import copy
|
||||
import unittest
|
||||
from collections import Mapping, MutableMapping
|
||||
|
||||
from scrapy.utils.datatypes import CaselessDict, SequenceExclude
|
||||
|
||||
|
|
@ -7,17 +8,62 @@ __doctests__ = ['scrapy.utils.datatypes']
|
|||
|
||||
class CaselessDictTest(unittest.TestCase):
|
||||
|
||||
def test_init(self):
|
||||
def test_init_dict(self):
|
||||
seq = {'red': 1, 'black': 3}
|
||||
d = CaselessDict(seq)
|
||||
self.assertEqual(d['red'], 1)
|
||||
self.assertEqual(d['black'], 3)
|
||||
|
||||
def test_init_pair_sequence(self):
|
||||
seq = (('red', 1), ('black', 3))
|
||||
d = CaselessDict(seq)
|
||||
self.assertEqual(d['red'], 1)
|
||||
self.assertEqual(d['black'], 3)
|
||||
|
||||
def test_init_mapping(self):
|
||||
class MyMapping(Mapping):
|
||||
def __init__(self, **kwargs):
|
||||
self._d = kwargs
|
||||
|
||||
def __getitem__(self, key):
|
||||
return self._d[key]
|
||||
|
||||
def __iter__(self):
|
||||
return iter(self._d)
|
||||
|
||||
def __len__(self):
|
||||
return len(self._d)
|
||||
|
||||
seq = MyMapping(red=1, black=3)
|
||||
d = CaselessDict(seq)
|
||||
self.assertEqual(d['red'], 1)
|
||||
self.assertEqual(d['black'], 3)
|
||||
|
||||
def test_init_mutable_mapping(self):
|
||||
class MyMutableMapping(MutableMapping):
|
||||
def __init__(self, **kwargs):
|
||||
self._d = kwargs
|
||||
|
||||
def __getitem__(self, key):
|
||||
return self._d[key]
|
||||
|
||||
def __setitem__(self, key, value):
|
||||
self._d[key] = value
|
||||
|
||||
def __delitem__(self, key):
|
||||
del self._d[key]
|
||||
|
||||
def __iter__(self):
|
||||
return iter(self._d)
|
||||
|
||||
def __len__(self):
|
||||
return len(self._d)
|
||||
|
||||
seq = MyMutableMapping(red=1, black=3)
|
||||
d = CaselessDict(seq)
|
||||
self.assertEqual(d['red'], 1)
|
||||
self.assertEqual(d['black'], 3)
|
||||
|
||||
def test_caseless(self):
|
||||
d = CaselessDict()
|
||||
d['key_Lower'] = 1
|
||||
|
|
|
|||
|
|
@ -6,7 +6,8 @@ from six.moves.urllib.parse import urlparse
|
|||
|
||||
from scrapy.spiders import Spider
|
||||
from scrapy.utils.url import (url_is_from_any_domain, url_is_from_spider,
|
||||
add_http_if_no_scheme, guess_scheme, parse_url)
|
||||
add_http_if_no_scheme, guess_scheme,
|
||||
parse_url, strip_url)
|
||||
|
||||
__doctests__ = ['scrapy.utils.url']
|
||||
|
||||
|
|
@ -241,5 +242,171 @@ for k, args in enumerate ([
|
|||
setattr (GuessSchemeTest, t_method.__name__, t_method)
|
||||
|
||||
|
||||
class StripUrl(unittest.TestCase):
|
||||
|
||||
def test_noop(self):
|
||||
self.assertEqual(strip_url(
|
||||
'http://www.example.com/index.html'),
|
||||
'http://www.example.com/index.html')
|
||||
|
||||
def test_noop_query_string(self):
|
||||
self.assertEqual(strip_url(
|
||||
'http://www.example.com/index.html?somekey=somevalue'),
|
||||
'http://www.example.com/index.html?somekey=somevalue')
|
||||
|
||||
def test_fragments(self):
|
||||
self.assertEqual(strip_url(
|
||||
'http://www.example.com/index.html?somekey=somevalue#section', strip_fragment=False),
|
||||
'http://www.example.com/index.html?somekey=somevalue#section')
|
||||
|
||||
def test_path(self):
|
||||
for input_url, origin, output_url in [
|
||||
('http://www.example.com/',
|
||||
False,
|
||||
'http://www.example.com/'),
|
||||
|
||||
('http://www.example.com',
|
||||
False,
|
||||
'http://www.example.com'),
|
||||
|
||||
('http://www.example.com',
|
||||
True,
|
||||
'http://www.example.com/'),
|
||||
]:
|
||||
self.assertEqual(strip_url(input_url, origin_only=origin), output_url)
|
||||
|
||||
def test_credentials(self):
|
||||
for i, o in [
|
||||
('http://username@www.example.com/index.html?somekey=somevalue#section',
|
||||
'http://www.example.com/index.html?somekey=somevalue'),
|
||||
|
||||
('https://username:@www.example.com/index.html?somekey=somevalue#section',
|
||||
'https://www.example.com/index.html?somekey=somevalue'),
|
||||
|
||||
('ftp://username:password@www.example.com/index.html?somekey=somevalue#section',
|
||||
'ftp://www.example.com/index.html?somekey=somevalue'),
|
||||
]:
|
||||
self.assertEqual(strip_url(i, strip_credentials=True), o)
|
||||
|
||||
def test_credentials_encoded_delims(self):
|
||||
for i, o in [
|
||||
# user: "username@"
|
||||
# password: none
|
||||
('http://username%40@www.example.com/index.html?somekey=somevalue#section',
|
||||
'http://www.example.com/index.html?somekey=somevalue'),
|
||||
|
||||
# user: "username:pass"
|
||||
# password: ""
|
||||
('https://username%3Apass:@www.example.com/index.html?somekey=somevalue#section',
|
||||
'https://www.example.com/index.html?somekey=somevalue'),
|
||||
|
||||
# user: "me"
|
||||
# password: "user@domain.com"
|
||||
('ftp://me:user%40domain.com@www.example.com/index.html?somekey=somevalue#section',
|
||||
'ftp://www.example.com/index.html?somekey=somevalue'),
|
||||
]:
|
||||
self.assertEqual(strip_url(i, strip_credentials=True), o)
|
||||
|
||||
def test_default_ports_creds_off(self):
|
||||
for i, o in [
|
||||
('http://username:password@www.example.com:80/index.html?somekey=somevalue#section',
|
||||
'http://www.example.com/index.html?somekey=somevalue'),
|
||||
|
||||
('http://username:password@www.example.com:8080/index.html#section',
|
||||
'http://www.example.com:8080/index.html'),
|
||||
|
||||
('http://username:password@www.example.com:443/index.html?somekey=somevalue&someotherkey=sov#section',
|
||||
'http://www.example.com:443/index.html?somekey=somevalue&someotherkey=sov'),
|
||||
|
||||
('https://username:password@www.example.com:443/index.html',
|
||||
'https://www.example.com/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:442/index.html',
|
||||
'https://www.example.com:442/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:80/index.html',
|
||||
'https://www.example.com:80/index.html'),
|
||||
|
||||
('ftp://username:password@www.example.com:21/file.txt',
|
||||
'ftp://www.example.com/file.txt'),
|
||||
|
||||
('ftp://username:password@www.example.com:221/file.txt',
|
||||
'ftp://www.example.com:221/file.txt'),
|
||||
]:
|
||||
self.assertEqual(strip_url(i), o)
|
||||
|
||||
def test_default_ports(self):
|
||||
for i, o in [
|
||||
('http://username:password@www.example.com:80/index.html',
|
||||
'http://username:password@www.example.com/index.html'),
|
||||
|
||||
('http://username:password@www.example.com:8080/index.html',
|
||||
'http://username:password@www.example.com:8080/index.html'),
|
||||
|
||||
('http://username:password@www.example.com:443/index.html',
|
||||
'http://username:password@www.example.com:443/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:443/index.html',
|
||||
'https://username:password@www.example.com/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:442/index.html',
|
||||
'https://username:password@www.example.com:442/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:80/index.html',
|
||||
'https://username:password@www.example.com:80/index.html'),
|
||||
|
||||
('ftp://username:password@www.example.com:21/file.txt',
|
||||
'ftp://username:password@www.example.com/file.txt'),
|
||||
|
||||
('ftp://username:password@www.example.com:221/file.txt',
|
||||
'ftp://username:password@www.example.com:221/file.txt'),
|
||||
]:
|
||||
self.assertEqual(strip_url(i, strip_default_port=True, strip_credentials=False), o)
|
||||
|
||||
def test_default_ports_keep(self):
|
||||
for i, o in [
|
||||
('http://username:password@www.example.com:80/index.html?somekey=somevalue&someotherkey=sov#section',
|
||||
'http://username:password@www.example.com:80/index.html?somekey=somevalue&someotherkey=sov'),
|
||||
|
||||
('http://username:password@www.example.com:8080/index.html?somekey=somevalue&someotherkey=sov#section',
|
||||
'http://username:password@www.example.com:8080/index.html?somekey=somevalue&someotherkey=sov'),
|
||||
|
||||
('http://username:password@www.example.com:443/index.html',
|
||||
'http://username:password@www.example.com:443/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:443/index.html',
|
||||
'https://username:password@www.example.com:443/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:442/index.html',
|
||||
'https://username:password@www.example.com:442/index.html'),
|
||||
|
||||
('https://username:password@www.example.com:80/index.html',
|
||||
'https://username:password@www.example.com:80/index.html'),
|
||||
|
||||
('ftp://username:password@www.example.com:21/file.txt',
|
||||
'ftp://username:password@www.example.com:21/file.txt'),
|
||||
|
||||
('ftp://username:password@www.example.com:221/file.txt',
|
||||
'ftp://username:password@www.example.com:221/file.txt'),
|
||||
]:
|
||||
self.assertEqual(strip_url(i, strip_default_port=False, strip_credentials=False), o)
|
||||
|
||||
def test_origin_only(self):
|
||||
for i, o in [
|
||||
('http://username:password@www.example.com/index.html',
|
||||
'http://www.example.com/'),
|
||||
|
||||
('http://username:password@www.example.com:80/foo/bar?query=value#somefrag',
|
||||
'http://www.example.com/'),
|
||||
|
||||
('http://username:password@www.example.com:8008/foo/bar?query=value#somefrag',
|
||||
'http://www.example.com:8008/'),
|
||||
|
||||
('https://username:password@www.example.com:443/index.html',
|
||||
'https://www.example.com/'),
|
||||
]:
|
||||
self.assertEqual(strip_url(i, origin_only=True), o)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
|
|||
Loading…
Reference in New Issue