mirror of https://github.com/scrapy/scrapy.git
Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12
This commit is contained in:
Pablo Hoffman
commit
b12dd76bb8
1
AUTHORS
1
AUTHORS
|
|
@ -27,3 +27,4 @@ Here is the list of the primary authors & contributors:
|
|||
* Shuaib Khan
|
||||
* Didier Deshommes
|
||||
* Vikas Dhiman
|
||||
* Jochen Maes
|
||||
|
|
|
|||
2
README
2
README
|
|
@ -1,4 +1,4 @@
|
|||
This is Scrapy, an opensource screen scraping framework written in Python.
|
||||
|
||||
For more visit the project home page at http://scrapy.org
|
||||
For more info visit the project home page at http://scrapy.org
|
||||
|
||||
|
|
|
|||
|
|
@ -2,13 +2,13 @@ Source: scrapy-SUFFIX
|
|||
Section: python
|
||||
Priority: optional
|
||||
Maintainer: Insophia Team <info@insophia.com>
|
||||
Build-Depends: debhelper (>= 7.0.50), python (>=2.5), python-twisted
|
||||
Build-Depends: debhelper (>= 7.0.50), python (>=2.6), python-twisted, python-w3lib
|
||||
Standards-Version: 3.8.4
|
||||
Homepage: http://scrapy.org/
|
||||
|
||||
Package: scrapy-SUFFIX
|
||||
Architecture: all
|
||||
Depends: ${python:Depends}, python-libxml2, python-twisted, python-openssl
|
||||
Depends: ${python:Depends}, python-libxml2, python-twisted, python-openssl, python-w3lib
|
||||
Conflicts: python-scrapy, scrapy, scrapy-0.11
|
||||
Provides: python-scrapy, scrapy
|
||||
Description: Python web crawling and scraping framework
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
usr/lib/python*/*-packages/scrapy
|
||||
usr/lib/python*/*-packages/scrapy*
|
||||
usr/bin
|
||||
extras/scrapy_bash_completion etc/bash_completion.d/
|
||||
|
|
|
|||
|
|
@ -1,3 +1,2 @@
|
|||
usr/lib/python*/*-packages/scrapyd
|
||||
debian/scrapyd-files/000-default etc/scrapyd/conf.d
|
||||
extras/scrapyd.tac usr/share/scrapyd
|
||||
|
|
|
|||
18
docs/faq.rst
18
docs/faq.rst
|
|
@ -3,7 +3,7 @@
|
|||
Frequently Asked Questions
|
||||
==========================
|
||||
|
||||
How does Scrapy compare to BeautifulSoul or lxml?
|
||||
How does Scrapy compare to BeautifulSoup or lxml?
|
||||
-------------------------------------------------
|
||||
|
||||
`BeautifulSoup`_ and `lxml`_ are libraries for parsing HTML and XML. Scrapy is
|
||||
|
|
@ -29,7 +29,7 @@ comparing `jinja2`_ to `Django`_.
|
|||
What Python versions does Scrapy support?
|
||||
-----------------------------------------
|
||||
|
||||
Scrapy runs in Python 2.5, 2.6 and 2.6. But it's recommended you use Python 2.6
|
||||
Scrapy runs in Python 2.5, 2.6 and 2.7. But it's recommended you use Python 2.6
|
||||
or above, since the Python 2.5 standard library has a few bugs in their URL
|
||||
handling libraries. Some of these Python 2.5 bugs not only affect Scrapy but
|
||||
any user code, such as spiders. You can see a list of `Python 2.5 bugs that
|
||||
|
|
@ -240,3 +240,17 @@ In order to avoid parsing all the entire feed at once in memory, you can use
|
|||
the functions ``xmliter`` and ``csviter`` from ``scrapy.utils.iterators``
|
||||
module. In fact, this is what the feed spiders (see :ref:`topics-spiders`) use
|
||||
under the cover.
|
||||
|
||||
Does Scrapy manage cookies automatically?
|
||||
-----------------------------------------
|
||||
|
||||
Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them
|
||||
back on subsequent requests, like any regular web browser does.
|
||||
|
||||
For more info see :ref:`topics-request-response` and :ref:`cookies-mw`.
|
||||
|
||||
How can I see the cookies being sent and received from Scrapy?
|
||||
--------------------------------------------------------------
|
||||
|
||||
Enable the :setting:`COOKIES_DEBUG` setting.
|
||||
|
||||
|
|
|
|||
|
|
@ -158,6 +158,8 @@ middleware, see the :ref:`downloader middleware usage guide
|
|||
For a list of the components enabled by default (and their orders) see the
|
||||
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting.
|
||||
|
||||
.. _cookies-mw:
|
||||
|
||||
CookiesMiddleware
|
||||
-----------------
|
||||
|
||||
|
|
@ -166,7 +168,36 @@ CookiesMiddleware
|
|||
|
||||
.. class:: CookiesMiddleware
|
||||
|
||||
This middleware enables working with sites that need cookies.
|
||||
This middleware enables working with sites that need cookies. It keeps track
|
||||
of merging cookies sent by servers, so that they're send in future requests
|
||||
for that spider, just like a web browser would do.
|
||||
|
||||
The following settings can be used to configure the cookie middleware:
|
||||
|
||||
* :setting:`COOKIES_DEBUG`
|
||||
|
||||
.. setting:: COOKIES_DEBUG
|
||||
|
||||
COOKIES_DEBUG
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Default: ``False``
|
||||
|
||||
If enabled, Scrapy will log all cookies sent in requests (ie. ``Cookie``
|
||||
header) and all cookies received in responses (ie. ``Set-Cookie`` header).
|
||||
|
||||
Here's an example of a log with :setting:`COOKIES_DEBUG` enabled::
|
||||
|
||||
2011-04-06 14:35:10-0300 [diningcity] INFO: Spider opened
|
||||
2011-04-06 14:35:10-0300 [diningcity] DEBUG: Sending cookies to: <GET http://www.diningcity.com/netherlands/index.html>
|
||||
Cookie: clientlanguage_nl=en_EN
|
||||
2011-04-06 14:35:14-0300 [diningcity] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html>
|
||||
Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/
|
||||
Set-Cookie: ip_isocode=US
|
||||
Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/
|
||||
2011-04-06 14:49:50-0300 [diningcity] DEBUG: Crawled (200) <GET http://www.diningcity.com/netherlands/index.html> (referer: None)
|
||||
[...]
|
||||
|
||||
|
||||
DefaultHeadersMiddleware
|
||||
------------------------
|
||||
|
|
@ -233,8 +264,18 @@ HttpCacheMiddleware
|
|||
downloads every time) and for trying your spider offline, when an Internet
|
||||
connection is not available.
|
||||
|
||||
File system storage
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
Scrapy ships with two storage backends for the HTTP cache middleware:
|
||||
|
||||
* :ref:`httpcache-fs-backend`
|
||||
* :ref:`httpcache-dbm-backend`
|
||||
|
||||
You can change the storage backend with the :setting:`HTTPCACHE_STORAGE`
|
||||
setting. Or you can also implement your own backend.
|
||||
|
||||
.. _httpcache-fs-backend:
|
||||
|
||||
File system backend (default)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
By default, the :class:`HttpCacheMiddleware` uses a file system storage with the following structure:
|
||||
|
||||
|
|
@ -257,8 +298,19 @@ inefficient in many file systems). An example directory could be::
|
|||
|
||||
/path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
|
||||
|
||||
The cache storage backend can be changed with the :setting:`HTTPCACHE_STORAGE`
|
||||
setting, but no other backend is provided with Scrapy yet.
|
||||
.. _httpcache-dbm-backend:
|
||||
|
||||
DBM storage backend
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. versionadded:: 0.13
|
||||
|
||||
A DBM_ storage backend is also available for the HTTP cache middleware. To use
|
||||
it (instead of the default filesystem backend) set :setting:`HTTPCACHE_STORAGE`
|
||||
to ``scrapy.contrib.httpcache.DbmCacheStorage``.
|
||||
|
||||
By default, it uses the anydbm_ module, but you can change it with the
|
||||
:setting:`HTTPCACHE_DBM_MODULE` setting.
|
||||
|
||||
Settings
|
||||
~~~~~~~~
|
||||
|
|
@ -346,6 +398,18 @@ Default: ``'scrapy.contrib.downloadermiddleware.httpcache.FilesystemCacheStorage
|
|||
|
||||
The class which implements the cache storage backend.
|
||||
|
||||
.. setting:: HTTPCACHE_DBM_MODULE
|
||||
|
||||
HTTPCACHE_DBM_MODULE
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. versionadded:: 0.13
|
||||
|
||||
Default: ``'anydbm'``
|
||||
|
||||
The database module to use in the :ref:`DBM storage backend
|
||||
<httpcache-dbm-backend>`. This setting is specific to the DBM backend.
|
||||
|
||||
|
||||
HttpCompressionMiddleware
|
||||
-------------------------
|
||||
|
|
@ -491,3 +555,6 @@ UserAgentMiddleware
|
|||
In order for a spider to override the default user agent, its `user_agent`
|
||||
attribute must be set.
|
||||
|
||||
|
||||
.. _DBM: http://en.wikipedia.org/wiki/Dbm
|
||||
.. _anydbm: http://docs.python.org/library/anydbm.html
|
||||
|
|
|
|||
|
|
@ -264,7 +264,7 @@ XmlItemExporter
|
|||
CsvItemExporter
|
||||
---------------
|
||||
|
||||
.. class:: CsvItemExporter(file, include_headers_line=True, \**kwargs)
|
||||
.. class:: CsvItemExporter(file, include_headers_line=True, join_multivalued=',', \**kwargs)
|
||||
|
||||
Exports Items in CSV format to the given file-like object. If the
|
||||
:attr:`fields_to_export` attribute is set, it will be used to define the
|
||||
|
|
@ -278,6 +278,10 @@ CsvItemExporter
|
|||
:attr:`BaseItemExporter.fields_to_export` or the first exported item fields.
|
||||
:type include_headers_line: boolean
|
||||
|
||||
:param join_multivalued: The char (or chars) that will be used for joining
|
||||
multi-valued fields, if found.
|
||||
:type include_headers_line: str
|
||||
|
||||
The additional keyword arguments of this constructor are passed to the
|
||||
:class:`BaseItemExporter` constructor, and the leftover arguments to the
|
||||
`csv.writer`_ constructor, so you can use any `csv.writer` constructor
|
||||
|
|
|
|||
|
|
@ -68,6 +68,8 @@ Request objects
|
|||
request_with_cookies = Request(url="http://www.example.com",
|
||||
cookies={currency: 'USD', country: 'UY'},
|
||||
meta={'dont_merge_cookies': True})
|
||||
|
||||
For more info see :ref:`cookies-mw`.
|
||||
:type cookies: dict
|
||||
|
||||
:param encoding: the encoding of this request (defaults to ``'utf-8'``).
|
||||
|
|
|
|||
|
|
@ -86,7 +86,7 @@ in your Ubuntu servers.
|
|||
So, if you plan to deploy Scrapyd on a Ubuntu server, just add the Ubuntu
|
||||
repositories as described in :ref:`topics-ubuntu` and then run::
|
||||
|
||||
aptitude install scrapyd-0.12
|
||||
aptitude install scrapyd-0.13
|
||||
|
||||
This will install Scrapyd in your Ubuntu server creating a ``scrapy`` user
|
||||
which Scrapyd will run as. It will also create some directories and files that
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ latest bug fixes.
|
|||
|
||||
To use the packages, just add the following line to your
|
||||
``/etc/apt/sources.list``, and then run ``aptitude update`` and ``aptitude
|
||||
install scrapy-0.12``::
|
||||
install scrapy-0.13``::
|
||||
|
||||
deb http://archive.scrapy.org/ubuntu DISTRO main
|
||||
|
||||
|
|
|
|||
|
|
@ -2,8 +2,8 @@
|
|||
Scrapy - a screen scraping framework written in Python
|
||||
"""
|
||||
|
||||
version_info = (0, 12, 0)
|
||||
__version__ = "0.12.0"
|
||||
version_info = (0, 13, 0)
|
||||
__version__ = "0.13.0"
|
||||
|
||||
import sys, os, warnings
|
||||
|
||||
|
|
|
|||
|
|
@ -1,10 +1,9 @@
|
|||
import sys
|
||||
from w3lib.url import is_url
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.command import ScrapyCommand
|
||||
from scrapy.conf import settings
|
||||
from scrapy.http import Request
|
||||
from scrapy.utils.url import is_url
|
||||
from scrapy.utils.conf import arglist_to_dict
|
||||
from scrapy.exceptions import UsageError
|
||||
|
||||
|
|
|
|||
|
|
@ -11,10 +11,11 @@ import netrc
|
|||
from urlparse import urlparse, urljoin
|
||||
from subprocess import Popen, PIPE, check_call
|
||||
|
||||
from w3lib.form import encode_multipart
|
||||
|
||||
from scrapy.command import ScrapyCommand
|
||||
from scrapy.exceptions import UsageError
|
||||
from scrapy.utils.py26 import json
|
||||
from scrapy.utils.multipart import encode_multipart
|
||||
from scrapy.utils.http import basic_auth_header
|
||||
from scrapy.utils.conf import get_config, closest_scrapy_cfg
|
||||
|
||||
|
|
@ -57,6 +58,8 @@ class Command(ScrapyCommand):
|
|||
help="list available projects on TARGET")
|
||||
parser.add_option("--egg", metavar="FILE",
|
||||
help="use the given egg, instead of building it")
|
||||
parser.add_option("--build-egg", metavar="FILE",
|
||||
help="only build the egg, don't deploy it")
|
||||
|
||||
def run(self, args, opts):
|
||||
try:
|
||||
|
|
@ -75,18 +78,26 @@ class Command(ScrapyCommand):
|
|||
projects = json.loads(f.read())['projects']
|
||||
print os.linesep.join(projects)
|
||||
return
|
||||
target_name = _get_target_name(args)
|
||||
target = _get_target(target_name)
|
||||
project = _get_project(target, opts)
|
||||
version = _get_version(target, opts)
|
||||
|
||||
tmpdir = None
|
||||
if opts.egg:
|
||||
_log("Using egg: %s" % opts.egg)
|
||||
egg = opts.egg
|
||||
else:
|
||||
_log("Building egg of %s-%s" % (project, version))
|
||||
|
||||
if opts.build_egg: # build egg only
|
||||
egg, tmpdir = _build_egg()
|
||||
_upload_egg(target, egg, project, version)
|
||||
_log("Writing egg to %s" % opts.build_egg)
|
||||
shutil.copyfile(egg, opts.build_egg)
|
||||
else: # buld egg and deploy
|
||||
target_name = _get_target_name(args)
|
||||
target = _get_target(target_name)
|
||||
project = _get_project(target, opts)
|
||||
version = _get_version(target, opts)
|
||||
if opts.egg:
|
||||
_log("Using egg: %s" % opts.egg)
|
||||
egg = opts.egg
|
||||
else:
|
||||
_log("Building egg of %s-%s" % (project, version))
|
||||
egg, tmpdir = _build_egg()
|
||||
_upload_egg(target, egg, project, version)
|
||||
|
||||
if tmpdir:
|
||||
shutil.rmtree(tmpdir)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
import pprint
|
||||
|
||||
from w3lib.url import is_url
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.command import ScrapyCommand
|
||||
from scrapy.http import Request
|
||||
from scrapy.spider import BaseSpider
|
||||
from scrapy.utils.url import is_url
|
||||
from scrapy.exceptions import UsageError
|
||||
|
||||
class Command(ScrapyCommand):
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
from w3lib.url import is_url
|
||||
from scrapy.command import ScrapyCommand
|
||||
from scrapy.http import Request
|
||||
from scrapy.item import BaseItem
|
||||
from scrapy.utils import display
|
||||
from scrapy.utils.spider import iterate_spider_output, create_spider_for_request
|
||||
from scrapy.utils.url import is_url
|
||||
from scrapy.exceptions import UsageError
|
||||
from scrapy import log
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import os
|
||||
from collections import defaultdict
|
||||
from scrapy.xlib.pydispatch import dispatcher
|
||||
|
||||
|
|
@ -28,7 +29,7 @@ class CookiesMiddleware(object):
|
|||
# set Cookie header
|
||||
request.headers.pop('Cookie', None)
|
||||
jar.add_cookie_header(request)
|
||||
self._debug_cookie(request)
|
||||
self._debug_cookie(request, spider)
|
||||
|
||||
def process_response(self, request, response, spider):
|
||||
if 'dont_merge_cookies' in request.meta:
|
||||
|
|
@ -37,31 +38,28 @@ class CookiesMiddleware(object):
|
|||
# extract cookies from Set-Cookie and drop invalid/expired cookies
|
||||
jar = self.jars[spider]
|
||||
jar.extract_cookies(response, request)
|
||||
self._debug_set_cookie(response)
|
||||
self._debug_set_cookie(response, spider)
|
||||
|
||||
return response
|
||||
|
||||
def spider_closed(self, spider):
|
||||
self.jars.pop(spider, None)
|
||||
|
||||
def _debug_cookie(self, request):
|
||||
"""log Cookie header for request"""
|
||||
def _debug_cookie(self, request, spider):
|
||||
if self.debug:
|
||||
c = request.headers.get('Cookie')
|
||||
c = c and [p.split('=')[0] for p in c.split(';')]
|
||||
log.msg('Cookie: %s for %s' % (c, request.url), level=log.DEBUG)
|
||||
cl = request.headers.getlist('Cookie')
|
||||
if cl:
|
||||
msg = "Sending cookies to: %s" % request + os.linesep
|
||||
msg += os.linesep.join("Cookie: %s" % c for c in cl)
|
||||
log.msg(msg, spider=spider, level=log.DEBUG)
|
||||
|
||||
def _debug_set_cookie(self, response):
|
||||
"""log Set-Cookies headers but exclude cookie values"""
|
||||
def _debug_set_cookie(self, response, spider):
|
||||
if self.debug:
|
||||
cl = response.headers.getlist('Set-Cookie')
|
||||
res = []
|
||||
for c in cl:
|
||||
kv, tail = c.split(';', 1)
|
||||
k = kv.split('=', 1)[0]
|
||||
res.append('%s %s' % (k, tail))
|
||||
log.msg('Set-Cookie: %s from %s' % (res, response.url))
|
||||
|
||||
if cl:
|
||||
msg = "Received cookies from: %s" % response + os.linesep
|
||||
msg += os.linesep.join("Set-Cookie: %s" % c for c in cl)
|
||||
log.msg(msg, spider=spider, level=log.DEBUG)
|
||||
|
||||
def _get_request_cookies(self, jar, request):
|
||||
headers = {'Set-Cookie': ['%s=%s;' % (k, v) for k, v in request.cookies.iteritems()]}
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ HTTP basic auth downloader middleware
|
|||
See documentation in docs/topics/downloader-middleware.rst
|
||||
"""
|
||||
|
||||
from scrapy.utils.http import basic_auth_header
|
||||
from w3lib.http import basic_auth_header
|
||||
from scrapy.utils.python import WeakKeyCache
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -5,13 +5,14 @@ from os.path import join, exists
|
|||
from time import time
|
||||
import cPickle as pickle
|
||||
|
||||
from w3lib.http import headers_dict_to_raw, headers_raw_to_dict
|
||||
|
||||
from scrapy.xlib.pydispatch import dispatcher
|
||||
from scrapy import signals
|
||||
from scrapy.http import Headers
|
||||
from scrapy.exceptions import NotConfigured, IgnoreRequest
|
||||
from scrapy.core.downloader.responsetypes import responsetypes
|
||||
from scrapy.utils.request import request_fingerprint
|
||||
from scrapy.utils.http import headers_dict_to_raw, headers_raw_to_dict
|
||||
from scrapy.utils.httpobj import urlparse_cached
|
||||
from scrapy.utils.misc import load_object
|
||||
from scrapy.utils.project import data_path
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
from w3lib.url import urljoin_rfc
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.http import HtmlResponse
|
||||
from scrapy.utils.url import urljoin_rfc
|
||||
from scrapy.utils.response import get_meta_refresh
|
||||
from scrapy.exceptions import IgnoreRequest
|
||||
from scrapy.conf import settings
|
||||
|
|
|
|||
|
|
@ -30,9 +30,11 @@ from scrapy.conf import settings
|
|||
|
||||
class RetryMiddleware(object):
|
||||
|
||||
# IOError is raised by the HttpCompression middleware when trying to
|
||||
# decompress an empty response
|
||||
EXCEPTIONS_TO_RETRY = (ServerTimeoutError, UserTimeoutError, DNSLookupError,
|
||||
ConnectionRefusedError, ConnectionDone, ConnectError,
|
||||
ConnectionLost, PartialDownloadError)
|
||||
ConnectionLost, PartialDownloadError, IOError)
|
||||
|
||||
def __init__(self):
|
||||
self.max_retry_times = settings.getint('RETRY_TIMES')
|
||||
|
|
|
|||
|
|
@ -143,11 +143,20 @@ class XmlItemExporter(BaseItemExporter):
|
|||
|
||||
class CsvItemExporter(BaseItemExporter):
|
||||
|
||||
def __init__(self, file, include_headers_line=True, **kwargs):
|
||||
def __init__(self, file, include_headers_line=True, join_multivalued=',', **kwargs):
|
||||
self._configure(kwargs, dont_fail=True)
|
||||
self.include_headers_line = include_headers_line
|
||||
self.csv_writer = csv.writer(file, **kwargs)
|
||||
self._headers_not_written = True
|
||||
self._join_multivalued = join_multivalued
|
||||
|
||||
def _to_str_if_unicode(self, value):
|
||||
if isinstance(value, (list, tuple)):
|
||||
try:
|
||||
value = self._join_multivalued.join(value)
|
||||
except TypeError: # list in value may not contain strings
|
||||
pass
|
||||
return super(CsvItemExporter, self)._to_str_if_unicode(value)
|
||||
|
||||
def export_item(self, item):
|
||||
if self._headers_not_written:
|
||||
|
|
|
|||
|
|
@ -12,14 +12,14 @@ from ftplib import FTP
|
|||
from shutil import copyfileobj
|
||||
|
||||
from zope.interface import Interface, implements
|
||||
|
||||
from twisted.internet import defer, threads
|
||||
from w3lib.url import file_uri_to_path
|
||||
|
||||
from scrapy import log, signals
|
||||
from scrapy.xlib.pydispatch import dispatcher
|
||||
from scrapy.utils.ftp import ftp_makedirs_cwd
|
||||
from scrapy.exceptions import NotConfigured
|
||||
from scrapy.utils.misc import load_object
|
||||
from scrapy.utils.url import file_uri_to_path
|
||||
from scrapy.conf import settings
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,64 @@
|
|||
from __future__ import with_statement
|
||||
|
||||
import os
|
||||
from time import time
|
||||
import cPickle as pickle
|
||||
|
||||
from scrapy.http import Headers
|
||||
from scrapy.core.downloader.responsetypes import responsetypes
|
||||
from scrapy.utils.request import request_fingerprint
|
||||
from scrapy.utils.project import data_path
|
||||
from scrapy import conf
|
||||
|
||||
|
||||
class DbmCacheStorage(object):
|
||||
|
||||
def __init__(self, settings=conf.settings):
|
||||
self.cachedir = data_path(settings['HTTPCACHE_DIR'])
|
||||
self.expiration_secs = settings.getint('HTTPCACHE_EXPIRATION_SECS')
|
||||
self.dbmodule = __import__(settings['HTTPCACHE_DBM_MODULE'])
|
||||
self.dbs = {}
|
||||
|
||||
def open_spider(self, spider):
|
||||
dbpath = os.path.join(self.cachedir, '%s.db' % spider.name)
|
||||
self.dbs[spider] = self.dbmodule.open(dbpath, 'c')
|
||||
|
||||
def close_spider(self, spider):
|
||||
self.dbs[spider].close()
|
||||
|
||||
def retrieve_response(self, spider, request):
|
||||
data = self._read_data(spider, request)
|
||||
if data is None:
|
||||
return # not cached
|
||||
url = data['url']
|
||||
status = data['status']
|
||||
headers = Headers(data['headers'])
|
||||
body = data['body']
|
||||
respcls = responsetypes.from_args(headers=headers, url=url)
|
||||
response = respcls(url=url, headers=headers, status=status, body=body)
|
||||
return response
|
||||
|
||||
def store_response(self, spider, request, response):
|
||||
key = self._request_key(request)
|
||||
data = {
|
||||
'status': response.status,
|
||||
'url': response.url,
|
||||
'headers': dict(response.headers),
|
||||
'body': response.body,
|
||||
}
|
||||
self.dbs[spider]['%s_data' % key] = pickle.dumps(data, protocol=2)
|
||||
self.dbs[spider]['%s_time' % key] = str(time())
|
||||
|
||||
def _read_data(self, spider, request):
|
||||
key = self._request_key(request)
|
||||
db = self.dbs[spider]
|
||||
tkey = '%s_time' % key
|
||||
if not db.has_key(tkey):
|
||||
return # not found
|
||||
ts = db[tkey]
|
||||
if 0 < self.expiration_secs < time() - float(ts):
|
||||
return # expired
|
||||
return pickle.loads(db['%s_data' % key])
|
||||
|
||||
def _request_key(self, request):
|
||||
return request_fingerprint(request)
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
"""
|
||||
This contrib implements an automatic extraction library based on an Instance
|
||||
Based Learning (IBL) algorithm, as described in the following papers:
|
||||
|
||||
A hierarchical approach to wrapper induction
|
||||
http://portal.acm.org/citation.cfm?id=301191
|
||||
|
||||
Extracting web data using instance based learning
|
||||
http://portal.acm.org/citation.cfm?id=1265174
|
||||
|
||||
This code requires the numpy library.
|
||||
"""
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
"""
|
||||
Extended types for IBL extraction
|
||||
"""
|
||||
from itertools import chain
|
||||
|
||||
from scrapy.contrib.ibl.extractors import text
|
||||
|
||||
class FieldDescriptor(object):
|
||||
"""description of a scraped attribute"""
|
||||
__slots__ = ('name', 'description', 'extractor', 'required', 'allow_markup')
|
||||
|
||||
def __init__(self, name, description, extractor=text, required=False,
|
||||
allow_markup=False):
|
||||
self.name = name
|
||||
self.description = description
|
||||
self.extractor = extractor
|
||||
self.required = required
|
||||
self.allow_markup = allow_markup
|
||||
|
||||
@classmethod
|
||||
def from_field(cls, name, field):
|
||||
return cls(name, field.get('description'), \
|
||||
field.get('ibl_extractor', text), field.get('required', False), \
|
||||
field.get('allow_markup', False))
|
||||
|
||||
def __str__(self):
|
||||
return "FieldDescriptor(%s)" % self.name
|
||||
|
||||
class ItemDescriptor(object):
|
||||
"""Simple auto scraping item descriptor.
|
||||
|
||||
This used to describe type-specific operations and may be overridden where
|
||||
necessary.
|
||||
"""
|
||||
|
||||
def __init__(self, name, description, attribute_descriptors):
|
||||
self.name = name
|
||||
self.attribute_map = dict((d.name, d) for d in attribute_descriptors)
|
||||
self._required_attributes = [d.name for d in attribute_descriptors \
|
||||
if d.required]
|
||||
|
||||
@classmethod
|
||||
def from_item(cls, name, description, item):
|
||||
a = [FieldDescriptor.from_field(n, f) for n, f in item.fields.items()]
|
||||
return cls(name, description, a)
|
||||
|
||||
def validated(self, data):
|
||||
"""Only return the items in the data that are valid"""
|
||||
return [d for d in data if self._item_validates(d)]
|
||||
|
||||
def _item_validates(self, item):
|
||||
"""simply checks that all mandatory attributes are present"""
|
||||
variant_attrs = set(chain(*
|
||||
[v.keys() for v in item.get('variants', [])]))
|
||||
return all([(name in item or name in variant_attrs) \
|
||||
for name in self._required_attributes])
|
||||
|
||||
def get_required_attributes(self):
|
||||
return self._required_attributes
|
||||
|
||||
def __str__(self):
|
||||
return "ItemDescriptor(%s)" % self.name
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
"""
|
||||
IBL module
|
||||
|
||||
This contains an extraction algorithm based on the paper Extracting Web Data
|
||||
Using Instance-Based Learning by Yanhong Zhai and Bing Liu.
|
||||
|
||||
It defines the InstanceBasedLearningExtractor class, which implements this
|
||||
extraction algorithm.
|
||||
|
||||
Main departures from the original algorithm:
|
||||
* there is no limit in prefix or suffix size
|
||||
* we have "attribute adaptors" that allow generic post processing and may
|
||||
affect the extraction process. For example, a price field may require a
|
||||
numeric value to be present.
|
||||
* tags can be inserted to extract regions not wrapped by html tags. These
|
||||
regions are then identified using the longest unique character prefix and
|
||||
suffix.
|
||||
"""
|
||||
from operator import itemgetter
|
||||
from .regionextract import build_extraction_tree
|
||||
from .pageparsing import parse_template, parse_extraction_page
|
||||
from .pageobjects import TokenDict
|
||||
|
||||
class InstanceBasedLearningExtractor(object):
|
||||
"""Implementation of the instance based learning algorithm to
|
||||
extract data from web pages.
|
||||
"""
|
||||
|
||||
def __init__(self, templates, type_descriptor=None, trace=False):
|
||||
"""Initialise this extractor
|
||||
|
||||
templates should contain a sequence of strings, each containing
|
||||
annotated html that will be used as templates for extraction.
|
||||
|
||||
Tags surrounding areas to be extracted must contain a
|
||||
'data-scrapy-annotate' attribute and the value must be the name
|
||||
of the attribute. If the tag was inserted and was not present in the
|
||||
original page, the data-scrapy-generated attribute must be present.
|
||||
|
||||
type_descriptor may contain a type descriptor describing the item
|
||||
to be extracted.
|
||||
|
||||
if trace is true, the returned extracted data will have a 'trace'
|
||||
property that contains a trace of the extraction execution.
|
||||
"""
|
||||
self.token_dict = TokenDict()
|
||||
parsed_plus_templates = [(parse_template(self.token_dict, t), t) for t in templates]
|
||||
parsed_plus_epages = [(p, parse_extraction_page(self.token_dict, t)) for p, t \
|
||||
in parsed_plus_templates if _annotation_count(p)]
|
||||
parsed_templates = map(itemgetter(0), parsed_plus_epages)
|
||||
|
||||
# templates with more attributes are considered first
|
||||
sorted_templates = sorted(parsed_templates, key=_annotation_count, reverse=True)
|
||||
self.extraction_trees = [build_extraction_tree(t, type_descriptor,
|
||||
trace) for t in sorted_templates]
|
||||
self.validated = type_descriptor.validated if type_descriptor else \
|
||||
self._filter_not_none
|
||||
|
||||
def extract(self, html, pref_template_id=None, useone=False):
|
||||
"""extract data from an html page
|
||||
|
||||
If pref_template_url is specified, the template with that url will be
|
||||
used first.
|
||||
if useone is True and no data was extracted, no additional template will
|
||||
be tried. If False and no data was extracted, try with rest of item templates
|
||||
"""
|
||||
extraction_page = parse_extraction_page(self.token_dict, html)
|
||||
if pref_template_id is not None:
|
||||
if useone:
|
||||
extraction_trees = [x for x in self.extraction_trees if x.template.id == pref_template_id]
|
||||
else:
|
||||
extraction_trees = sorted(self.extraction_trees,
|
||||
key=lambda x: x.template.id != pref_template_id)
|
||||
else:
|
||||
extraction_trees = self.extraction_trees
|
||||
|
||||
for extraction_tree in extraction_trees:
|
||||
extracted = extraction_tree.extract(extraction_page)
|
||||
correctly_extracted = self.validated(extracted)
|
||||
extra_required = extraction_tree.template.extra_required_attrs
|
||||
correctly_extracted = [c for c in correctly_extracted if \
|
||||
extra_required.intersection(c.keys()) == extra_required ]
|
||||
if len(correctly_extracted) > 0:
|
||||
return correctly_extracted, extraction_tree.template
|
||||
return None, None
|
||||
|
||||
def __str__(self):
|
||||
return "InstanceBasedLearningExtractor[\n%s\n]" % \
|
||||
(',\n'.join(map(str, self.extraction_trees)))
|
||||
|
||||
@staticmethod
|
||||
def _filter_not_none(items):
|
||||
return [d for d in items if d is not None]
|
||||
|
||||
def _annotation_count(template):
|
||||
return len(template.annotations)
|
||||
|
|
@ -1,228 +0,0 @@
|
|||
"""
|
||||
Page objects
|
||||
|
||||
This module contains objects representing pages and parts of pages (e.g. tokens
|
||||
and annotations) used in the instance based learning algorithm.
|
||||
"""
|
||||
from numpy import array, ndarray
|
||||
|
||||
from scrapy.contrib.ibl.htmlpage import HtmlTagType
|
||||
|
||||
class TokenType(object):
|
||||
"""constants for token types"""
|
||||
WORD = 0
|
||||
OPEN_TAG = HtmlTagType.OPEN_TAG
|
||||
CLOSE_TAG = HtmlTagType.CLOSE_TAG
|
||||
NON_PAIRED_TAG = HtmlTagType.UNPAIRED_TAG
|
||||
|
||||
class TokenDict(object):
|
||||
"""Mapping from parse tokens to integers
|
||||
|
||||
>>> d = TokenDict()
|
||||
>>> d.tokenid('i')
|
||||
0
|
||||
>>> d.tokenid('b')
|
||||
1
|
||||
>>> d.tokenid('i')
|
||||
0
|
||||
|
||||
Tokens can be searched for by id
|
||||
>>> d.find_token(1)
|
||||
'b'
|
||||
|
||||
The lower 24 bits store the token reference and the higher bits the type.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.token_ids = {}
|
||||
|
||||
def tokenid(self, token, token_type=TokenType.WORD):
|
||||
"""create an integer id from the token and token type passed"""
|
||||
tid = self.token_ids.setdefault(token, len(self.token_ids))
|
||||
return tid | (token_type << 24)
|
||||
|
||||
@staticmethod
|
||||
def token_type(token):
|
||||
"""extract the token type from the token id passed"""
|
||||
return token >> 24
|
||||
|
||||
def find_token(self, tid):
|
||||
"""Search for a tag with the given ID
|
||||
|
||||
This is O(N) and is only intended for debugging
|
||||
"""
|
||||
tid &= 0xFFFFFF
|
||||
if tid >= len(self.token_ids) or tid < 0:
|
||||
raise ValueError("tag id %s out of range" % tid)
|
||||
|
||||
for (token, token_id) in self.token_ids.items():
|
||||
if token_id == tid:
|
||||
return token
|
||||
assert False, "token dictionary is corrupt"
|
||||
|
||||
def token_string(self, tid):
|
||||
"""create a string representation of a token
|
||||
|
||||
This is O(N).
|
||||
"""
|
||||
templates = ["%s", "<%s>", "</%s>", "<%s/>"]
|
||||
return templates[tid >> 24] % self.find_token(tid)
|
||||
|
||||
class Page(object):
|
||||
"""Basic representation of a page. This consists of a reference to a
|
||||
dictionary of tokens and an array of raw token ids
|
||||
"""
|
||||
|
||||
__slots__ = ('token_dict', 'page_tokens')
|
||||
|
||||
def __init__(self, token_dict, page_tokens):
|
||||
self.token_dict = token_dict
|
||||
# use a numpy array becuase we can index/slice easily and efficiently
|
||||
if not isinstance(page_tokens, ndarray):
|
||||
page_tokens = array(page_tokens)
|
||||
self.page_tokens = page_tokens
|
||||
|
||||
class TemplatePage(Page):
|
||||
__slots__ = ('annotations', 'id', 'ignored_regions', 'extra_required_attrs')
|
||||
|
||||
def __init__(self, token_dict, page_tokens, annotations, template_id=None, \
|
||||
ignored_regions=None, extra_required=None):
|
||||
Page.__init__(self, token_dict, page_tokens)
|
||||
# ensure order is the same as start tag order in the original page
|
||||
annotations = sorted(annotations, key=lambda x: x.end_index, reverse=True)
|
||||
self.annotations = sorted(annotations, key=lambda x: x.start_index)
|
||||
self.id = template_id
|
||||
self.ignored_regions = ignored_regions or []
|
||||
self.extra_required_attrs = set(extra_required or [])
|
||||
|
||||
def __str__(self):
|
||||
summary = []
|
||||
for index, token in enumerate(self.page_tokens):
|
||||
text = "%s: %s" % (index, self.token_dict.find_token(token))
|
||||
summary.append(text)
|
||||
return "TemplatePage\n============\nTokens: (index, token)\n%s\nAnnotations: %s\n" % \
|
||||
('\n'.join(summary), '\n'.join(map(str, self.annotations)))
|
||||
|
||||
class ExtractionPage(Page):
|
||||
"""Parsed data belonging to a web page upon which we wish to perform
|
||||
extraction.
|
||||
"""
|
||||
__slots__ = ('text',
|
||||
'token_start_indexes', # index in text of the start of a token
|
||||
'token_follow_indexes', # index in text of data following token
|
||||
'tag_attributes' # a map from token index to tag attributes
|
||||
)
|
||||
|
||||
def __init__(self, text, token_dict, page_tokens, token_start_indexes,
|
||||
token_follow_indexes, tag_attributes):
|
||||
Page.__init__(self, token_dict, page_tokens)
|
||||
self.text = text
|
||||
self.token_start_indexes = token_start_indexes
|
||||
self.token_follow_indexes = token_follow_indexes
|
||||
self.tag_attributes = tag_attributes
|
||||
|
||||
def token_html(self, token_index):
|
||||
"""The raw html for a page token at the given index in the page_tokens
|
||||
list
|
||||
"""
|
||||
text_start = self.token_start_indexes[token_index]
|
||||
text_end = self.token_follow_indexes[token_index]
|
||||
return self.text[text_start:text_end]
|
||||
|
||||
def html_between_tokens(self, start_token_index, end_token_index):
|
||||
"""The raw html between the tokens at the specified indexes in the
|
||||
page_tokens list
|
||||
|
||||
This assumes start_token_index <= end_token_index
|
||||
"""
|
||||
text_start = self.token_follow_indexes[start_token_index]
|
||||
text_end = self.token_start_indexes[end_token_index]
|
||||
return self.text[text_start:text_end]
|
||||
|
||||
def text_between_tokens(self, start_token_index, end_token_index,
|
||||
tag_replacement=u' '):
|
||||
"""The text between the the tokens at the specified indexes in the
|
||||
page_tokens list. Tags are replaced by tag_replacement (default one space
|
||||
character)
|
||||
"""
|
||||
return tag_replacement.join([self.text[
|
||||
self.token_follow_indexes[i]:self.token_start_indexes[i+1]] \
|
||||
for i in xrange(start_token_index, end_token_index)])
|
||||
|
||||
def tag_attribute(self, token_index, attribute):
|
||||
"""The value of a tag attribute. The tag is identified by its
|
||||
corresponding token index.
|
||||
|
||||
If the tag or attribute is not present, None is returned
|
||||
"""
|
||||
return self.tag_attributes.get(token_index, {}).get(attribute)
|
||||
|
||||
def __str__(self):
|
||||
summary = []
|
||||
for (token, start, follow) in zip(self.page_tokens, self.token_start_indexes,
|
||||
self.token_follow_indexes):
|
||||
text = "%s %s-%s (%s)" % (self.token_dict.find_token(token), start, follow,
|
||||
self.text[start:follow])
|
||||
summary.append(text)
|
||||
return "ExtractionPage\n==============\nTokens: %s\n\nRaw text: %s\n\n" \
|
||||
"Tag attributes: %s\n" % ('\n'.join(summary), self.text,
|
||||
self.tag_attributes)
|
||||
|
||||
class AnnotationText(object):
|
||||
__slots__ = ('start_text', 'follow_text')
|
||||
|
||||
def __init__(self, start_text=None, follow_text=None):
|
||||
self.start_text = start_text
|
||||
self.follow_text = follow_text
|
||||
|
||||
def __str__(self):
|
||||
return "AnnotationText(%s..%s)" % \
|
||||
(repr(self.start_text), repr(self.follow_text))
|
||||
|
||||
class AnnotationTag(object):
|
||||
"""A tag that annotates part of the document
|
||||
|
||||
It has the following properties:
|
||||
start_index - index of the token for the opening tag
|
||||
end_index - index of the token for the closing tag
|
||||
surrounds_attribute - the attribute name surrounded by this tag
|
||||
tag_attributes - list of (tag attribute, extracted attribute) tuples
|
||||
for each item to be extracted from a tag attribute
|
||||
annotation_text - text prefix and suffix for the attribute to be extracted
|
||||
metadata - dict with annotation data not used by IBL extractor
|
||||
"""
|
||||
__slots__ = ('surrounds_attribute', 'start_index', 'end_index',
|
||||
'tag_attributes', 'annotation_text', 'variant_id',
|
||||
'metadata')
|
||||
|
||||
def __init__(self, start_index, end_index, surrounds_attribute=None,
|
||||
annotation_text=None, tag_attributes=None, variant_id=None):
|
||||
self.start_index = start_index
|
||||
self.end_index = end_index
|
||||
self.surrounds_attribute = surrounds_attribute
|
||||
self.annotation_text = annotation_text
|
||||
self.tag_attributes = tag_attributes or []
|
||||
self.variant_id = variant_id
|
||||
self.metadata = {}
|
||||
|
||||
def __str__(self):
|
||||
return "AnnotationTag(%s)" % ", ".join(
|
||||
["%s=%s" % (s, getattr(self, s)) \
|
||||
for s in self.__slots__ if getattr(self, s)])
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
class LabelledRegion(object):
|
||||
__slots__ = ('start_index', 'end_index')
|
||||
|
||||
def __init__(self, start, end):
|
||||
self.start_index = start
|
||||
self.end_index = end
|
||||
|
||||
def __str__(self):
|
||||
return "LabelledRegion (%s, %s)" % (self.start_index, self.end_index)
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
|
|
@ -1,319 +0,0 @@
|
|||
"""
|
||||
Page parsing
|
||||
|
||||
Parsing of web pages for extraction task.
|
||||
"""
|
||||
from collections import defaultdict
|
||||
from numpy import array
|
||||
|
||||
from scrapy.utils.py26 import json
|
||||
|
||||
from scrapy.contrib.ibl.htmlpage import HtmlTagType, HtmlTag, HtmlPage
|
||||
from scrapy.contrib.ibl.extraction.pageobjects import (AnnotationTag,
|
||||
TemplatePage, ExtractionPage, AnnotationText, TokenDict)
|
||||
|
||||
def parse_strings(template_html, extraction_html):
|
||||
"""Create a template and extraction page from raw strings
|
||||
|
||||
this is useful for testing purposes
|
||||
"""
|
||||
t = TokenDict()
|
||||
template_page = HtmlPage(body=template_html)
|
||||
extraction_page = HtmlPage(body=extraction_html)
|
||||
return (parse_template(t, template_page),
|
||||
parse_extraction_page(t, extraction_page))
|
||||
|
||||
def parse_template(token_dict, template_html):
|
||||
"""Create an TemplatePage object by parsing the annotated html"""
|
||||
parser = TemplatePageParser(token_dict)
|
||||
parser.feed(template_html)
|
||||
return parser.to_template()
|
||||
|
||||
def parse_extraction_page(token_dict, page_html):
|
||||
"""Create an ExtractionPage object by parsing the html"""
|
||||
parser = ExtractionPageParser(token_dict)
|
||||
parser.feed(page_html)
|
||||
return parser.to_extraction_page()
|
||||
|
||||
class InstanceLearningParser(object):
|
||||
"""Base parser for instance based learning algorithm
|
||||
|
||||
This does not require correct HTML and the parsing method should not alter
|
||||
the original tag order. It is important that parsing results do not vary.
|
||||
"""
|
||||
def __init__(self, token_dict):
|
||||
self.token_dict = token_dict
|
||||
self.token_list = []
|
||||
|
||||
def _add_token(self, token, token_type, start, end):
|
||||
tid = self.token_dict.tokenid(token, token_type)
|
||||
self.token_list.append(tid)
|
||||
|
||||
def feed(self, html_page):
|
||||
self.html_page = html_page
|
||||
self.previous_element_class = None
|
||||
for data in html_page.parsed_body:
|
||||
if isinstance(data, HtmlTag):
|
||||
self._add_token(data.tag, data.tag_type, data.start, data.end)
|
||||
self.handle_tag(data)
|
||||
else:
|
||||
self.handle_data(data)
|
||||
self.previous_element_class = data.__class__
|
||||
|
||||
def handle_data(self, html_data_fragment):
|
||||
pass
|
||||
|
||||
def handle_tag(self, html_tag):
|
||||
pass
|
||||
|
||||
_END_UNPAIREDTAG_TAGS = ["form", "div", "p", "table", "tr", "td"]
|
||||
|
||||
class TemplatePageParser(InstanceLearningParser):
|
||||
"""Template parsing for instance based learning algorithm"""
|
||||
|
||||
def __init__(self, token_dict):
|
||||
InstanceLearningParser.__init__(self, token_dict)
|
||||
self.annotations = []
|
||||
self.ignored_regions = []
|
||||
self.extra_required_attrs = []
|
||||
self.ignored_tag_stacks = defaultdict(list)
|
||||
# tag names that have not been completed
|
||||
self.labelled_tag_stacks = defaultdict(list)
|
||||
self.replacement_stacks = defaultdict(list)
|
||||
self.unpairedtag_stack = []
|
||||
self.variant_stack = []
|
||||
self.prev_data = None
|
||||
self.last_text_region = None
|
||||
self.next_tag_index = 0
|
||||
|
||||
def handle_tag(self, html_tag):
|
||||
if self.last_text_region:
|
||||
self._process_text('')
|
||||
|
||||
if html_tag.tag_type == HtmlTagType.OPEN_TAG:
|
||||
self._handle_open_tag(html_tag)
|
||||
elif html_tag.tag_type == HtmlTagType.CLOSE_TAG:
|
||||
self._handle_close_tag(html_tag)
|
||||
else:
|
||||
# the tag is not paired, it can contain only attribute annotations
|
||||
self._handle_unpaired_tag(html_tag)
|
||||
|
||||
@staticmethod
|
||||
def _read_template_annotation(html_tag):
|
||||
template_attr = html_tag.attributes.get('data-scrapy-annotate')
|
||||
if template_attr is None:
|
||||
return None
|
||||
unescaped = template_attr.replace('"', '"')
|
||||
return json.loads(unescaped)
|
||||
|
||||
@staticmethod
|
||||
def _read_bool_template_attribute(html_tag, attribute):
|
||||
return html_tag.attributes.get("data-scrapy-" + attribute) == "true"
|
||||
|
||||
def _close_unpaired_tag(self):
|
||||
self.unpairedtag_stack[0].end_index = self.next_tag_index
|
||||
self.unpairedtag_stack = []
|
||||
|
||||
def _handle_unpaired_tag(self, html_tag):
|
||||
if self._read_bool_template_attribute(html_tag, "ignore") and html_tag.tag == "img":
|
||||
self.ignored_regions.append((self.next_tag_index, self.next_tag_index + 1))
|
||||
elif self._read_bool_template_attribute(html_tag, "ignore-beneath"):
|
||||
self.ignored_regions.append((self.next_tag_index, None))
|
||||
jannotation = self._read_template_annotation(html_tag)
|
||||
if jannotation:
|
||||
if self.unpairedtag_stack:
|
||||
self._close_unpaired_tag()
|
||||
|
||||
annotation = AnnotationTag(self.next_tag_index, self.next_tag_index + 1)
|
||||
attribute_annotations = jannotation.pop('annotations', {}).items()
|
||||
for extract_attribute, tag_value in attribute_annotations:
|
||||
if extract_attribute == 'content':
|
||||
annotation.surrounds_attribute = tag_value
|
||||
self.unpairedtag_stack.append(annotation)
|
||||
else:
|
||||
annotation.tag_attributes.append((extract_attribute, tag_value))
|
||||
self.annotations.append(annotation)
|
||||
|
||||
self.extra_required_attrs.extend(jannotation.pop('required', []))
|
||||
annotation.metadata = jannotation
|
||||
|
||||
self.next_tag_index += 1
|
||||
|
||||
def _handle_open_tag(self, html_tag):
|
||||
if self._read_bool_template_attribute(html_tag, "ignore"):
|
||||
if html_tag.tag == "img":
|
||||
self.ignored_regions.append((self.next_tag_index, self.next_tag_index + 1))
|
||||
else:
|
||||
self.ignored_regions.append((self.next_tag_index, None))
|
||||
self.ignored_tag_stacks[html_tag.tag].append(html_tag)
|
||||
|
||||
elif self.ignored_tag_stacks.get(html_tag.tag):
|
||||
self.ignored_tag_stacks[html_tag.tag].append(None)
|
||||
if self._read_bool_template_attribute(html_tag, "ignore-beneath"):
|
||||
self.ignored_regions.append((self.next_tag_index, None))
|
||||
|
||||
replacement = html_tag.attributes.pop("data-scrapy-replacement", None)
|
||||
if replacement:
|
||||
self.token_list.pop()
|
||||
self._add_token(replacement, html_tag.tag_type, html_tag.start, html_tag.end)
|
||||
self.replacement_stacks[html_tag.tag].append(replacement)
|
||||
elif html_tag.tag in self.replacement_stacks:
|
||||
self.replacement_stacks[html_tag.tag].append(None)
|
||||
|
||||
if self.unpairedtag_stack:
|
||||
if html_tag.tag in _END_UNPAIREDTAG_TAGS:
|
||||
self._close_unpaired_tag()
|
||||
else:
|
||||
self.unpairedtag_stack.append(html_tag.tag)
|
||||
|
||||
# can't be a p inside another p. Also, an open p element closes
|
||||
# a previous open p element.
|
||||
if html_tag.tag == "p" and html_tag.tag in self.labelled_tag_stacks:
|
||||
annotation = self.labelled_tag_stacks.pop(html_tag.tag)[0]
|
||||
annotation.end_index = self.next_tag_index
|
||||
self.annotations.append(annotation)
|
||||
|
||||
jannotation = self._read_template_annotation(html_tag)
|
||||
if not jannotation:
|
||||
if html_tag.tag in self.labelled_tag_stacks:
|
||||
# add this tag to the stack to match correct end tag
|
||||
self.labelled_tag_stacks[html_tag.tag].append(None)
|
||||
self.next_tag_index += 1
|
||||
return
|
||||
|
||||
annotation = AnnotationTag(self.next_tag_index, None)
|
||||
if jannotation.pop('generated', False):
|
||||
self.token_list.pop()
|
||||
annotation.start_index -= 1
|
||||
if self.previous_element_class == HtmlTag:
|
||||
annotation.annotation_text = AnnotationText('')
|
||||
else:
|
||||
annotation.annotation_text = AnnotationText(self.prev_data)
|
||||
if self._read_bool_template_attribute(html_tag, "ignore") \
|
||||
or self._read_bool_template_attribute(html_tag, "ignore-beneath"):
|
||||
ignored = self.ignored_regions.pop()
|
||||
self.ignored_regions.append((ignored[0]-1, ignored[1]))
|
||||
|
||||
self.extra_required_attrs.extend(jannotation.pop('required', []))
|
||||
|
||||
attribute_annotations = jannotation.pop('annotations', {}).items()
|
||||
for extract_attribute, tag_value in attribute_annotations:
|
||||
if extract_attribute == 'content':
|
||||
annotation.surrounds_attribute = tag_value
|
||||
else:
|
||||
annotation.tag_attributes.append((extract_attribute, tag_value))
|
||||
|
||||
variant_id = jannotation.pop('variant', 0)
|
||||
if variant_id > 0:
|
||||
if annotation.surrounds_attribute is not None:
|
||||
self.variant_stack.append(variant_id)
|
||||
else:
|
||||
annotation.variant_id = variant_id
|
||||
|
||||
annotation.metadata = jannotation
|
||||
|
||||
if annotation.annotation_text is None:
|
||||
self.next_tag_index += 1
|
||||
if self.variant_stack and annotation.variant_id is None:
|
||||
variant_id = self.variant_stack[-1]
|
||||
if variant_id == '0':
|
||||
variant_id = None
|
||||
annotation.variant_id = variant_id
|
||||
|
||||
# look for a closing tag if the content is important
|
||||
if annotation.surrounds_attribute:
|
||||
self.labelled_tag_stacks[html_tag.tag].append(annotation)
|
||||
else:
|
||||
annotation.end_index = annotation.start_index + 1
|
||||
self.annotations.append(annotation)
|
||||
|
||||
def _handle_close_tag(self, html_tag):
|
||||
|
||||
if self.unpairedtag_stack:
|
||||
if html_tag.tag == self.unpairedtag_stack[-1]:
|
||||
self.unpairedtag_stack.pop()
|
||||
else:
|
||||
self._close_unpaired_tag()
|
||||
ignored_tags = self.ignored_tag_stacks.get(html_tag.tag)
|
||||
if ignored_tags is not None:
|
||||
tag = ignored_tags.pop()
|
||||
if isinstance(tag, HtmlTag):
|
||||
for i in range(-1, -len(self.ignored_regions) - 1, -1):
|
||||
if self.ignored_regions[i][1] is None:
|
||||
self.ignored_regions[i] = (self.ignored_regions[i][0], self.next_tag_index)
|
||||
break
|
||||
if len(ignored_tags) == 0:
|
||||
del self.ignored_tag_stacks[html_tag.tag]
|
||||
|
||||
if html_tag.tag in self.replacement_stacks:
|
||||
replacement = self.replacement_stacks[html_tag.tag].pop()
|
||||
if replacement:
|
||||
self.token_list.pop()
|
||||
self._add_token(replacement, html_tag.tag_type, html_tag.start, html_tag.end)
|
||||
if len(self.replacement_stacks[html_tag.tag]) == 0:
|
||||
del self.replacement_stacks[html_tag.tag]
|
||||
|
||||
labelled_tags = self.labelled_tag_stacks.get(html_tag.tag)
|
||||
if labelled_tags is None:
|
||||
self.next_tag_index += 1
|
||||
return
|
||||
annotation = labelled_tags.pop()
|
||||
if annotation is None:
|
||||
self.next_tag_index += 1
|
||||
else:
|
||||
annotation.end_index = self.next_tag_index
|
||||
self.annotations.append(annotation)
|
||||
if annotation.annotation_text is not None:
|
||||
self.token_list.pop()
|
||||
self.last_text_region = annotation
|
||||
else:
|
||||
self.next_tag_index += 1
|
||||
if len(labelled_tags) == 0:
|
||||
del self.labelled_tag_stacks[html_tag.tag]
|
||||
if annotation.variant_id and self.variant_stack:
|
||||
prev = self.variant_stack.pop()
|
||||
if prev != annotation.variant_id:
|
||||
raise ValueError("unbalanced variant annotation tags")
|
||||
|
||||
def handle_data(self, html_data_fragment):
|
||||
fragment_text = self.html_page.fragment_data(html_data_fragment)
|
||||
self._process_text(fragment_text)
|
||||
|
||||
def _process_text(self, text):
|
||||
if self.last_text_region is not None:
|
||||
self.last_text_region.annotation_text.follow_text = text
|
||||
self.last_text_region = None
|
||||
self.prev_data = text
|
||||
|
||||
def to_template(self):
|
||||
"""create a TemplatePage from the data fed to this parser"""
|
||||
return TemplatePage(self.token_dict, self.token_list, self.annotations,
|
||||
self.html_page.page_id, self.ignored_regions, self.extra_required_attrs)
|
||||
|
||||
class ExtractionPageParser(InstanceLearningParser):
|
||||
"""Parse an HTML page for extraction using the instance based learning
|
||||
algorithm
|
||||
|
||||
This needs to extract the tokens in a similar way to LabelledPageParser,
|
||||
it needs to also maintain a mapping from token index to the original content
|
||||
so that once regions are identified, the original content can be extracted.
|
||||
"""
|
||||
def __init__(self, token_dict):
|
||||
InstanceLearningParser.__init__(self, token_dict)
|
||||
self.page_data = []
|
||||
self.token_start_index = []
|
||||
self.token_follow_index = []
|
||||
self.tag_attrs = {}
|
||||
|
||||
def _add_token(self, token, token_type, start, end):
|
||||
InstanceLearningParser._add_token(self, token, token_type, start, end)
|
||||
self.token_start_index.append(start)
|
||||
self.token_follow_index.append(end)
|
||||
|
||||
def handle_tag(self, html_tag):
|
||||
if html_tag.attributes:
|
||||
self.tag_attrs[len(self.token_list) - 1] = html_tag.attributes
|
||||
|
||||
def to_extraction_page(self):
|
||||
return ExtractionPage(self.html_page.body, self.token_dict, array(self.token_list),
|
||||
self.token_start_index, self.token_follow_index, self.tag_attrs)
|
||||
|
|
@ -1,627 +0,0 @@
|
|||
"""
|
||||
Region Extract
|
||||
|
||||
Custom extraction for regions in a document
|
||||
"""
|
||||
import re
|
||||
import operator
|
||||
import copy
|
||||
import pprint
|
||||
import cStringIO
|
||||
from itertools import groupby
|
||||
|
||||
from numpy import array
|
||||
|
||||
from scrapy.contrib.ibl.descriptor import FieldDescriptor
|
||||
from scrapy.contrib.ibl.extraction.similarity import (similar_region,
|
||||
longest_unique_subsequence, common_prefix)
|
||||
from scrapy.contrib.ibl.extraction.pageobjects import AnnotationTag, LabelledRegion
|
||||
|
||||
def build_extraction_tree(template, type_descriptor, trace=True):
|
||||
"""Build a tree of region extractors corresponding to the
|
||||
template
|
||||
"""
|
||||
attribute_map = type_descriptor.attribute_map if type_descriptor else None
|
||||
extractors = BasicTypeExtractor.create(template.annotations, attribute_map)
|
||||
if trace:
|
||||
extractors = TraceExtractor.apply(template, extractors)
|
||||
for cls in (AdjacentVariantExtractor, RepeatedDataExtractor, AdjacentVariantExtractor, RepeatedDataExtractor,
|
||||
RecordExtractor):
|
||||
extractors = cls.apply(template, extractors)
|
||||
if trace:
|
||||
extractors = TraceExtractor.apply(template, extractors)
|
||||
|
||||
return TemplatePageExtractor(template, extractors)
|
||||
|
||||
_ID = lambda x: x
|
||||
_DEFAULT_DESCRIPTOR = FieldDescriptor('none', None)
|
||||
|
||||
def _labelled(obj):
|
||||
"""
|
||||
Returns labelled element of the object (extractor or labelled region)
|
||||
"""
|
||||
if hasattr(obj, "annotation"):
|
||||
return obj.annotation
|
||||
return obj
|
||||
|
||||
def _compose(f, g):
|
||||
"""given unary functions f and g, return a function that computes f(g(x))
|
||||
"""
|
||||
def _exec(x):
|
||||
ret = g(x)
|
||||
return f(ret) if ret is not None else None
|
||||
return _exec
|
||||
|
||||
class BasicTypeExtractor(object):
|
||||
"""The BasicTypeExtractor extracts single attributes corresponding to
|
||||
annotations.
|
||||
|
||||
For example:
|
||||
>>> from scrapy.contrib.ibl.extraction.pageparsing import parse_strings
|
||||
>>> template, page = parse_strings( \
|
||||
u'<h1 data-scrapy-annotate="{"annotations": {"content": "name"}}">x</h1>', u'<h1> a name</h1>')
|
||||
>>> ex = BasicTypeExtractor(template.annotations[0])
|
||||
>>> ex.extract(page, 0, 1, None)
|
||||
[(u'name', u' a name')]
|
||||
|
||||
It supports attribute descriptors
|
||||
>>> descriptor = FieldDescriptor('name', None, lambda x: x.strip())
|
||||
>>> ex = BasicTypeExtractor(template.annotations[0], {'name': descriptor})
|
||||
>>> ex.extract(page, 0, 1, None)
|
||||
[(u'name', u'a name')]
|
||||
|
||||
It supports ignoring regions
|
||||
>>> template, page = parse_strings(\
|
||||
u'<div data-scrapy-annotate="{"annotations": {"content": "name"}}">x<b> xx</b></div>',\
|
||||
u'<div>a name<b> id-9</b></div>')
|
||||
>>> ex = BasicTypeExtractor(template.annotations[0])
|
||||
>>> ex.extract(page, 0, 3, [LabelledRegion(*(1,2))])
|
||||
[(u'name', u'a name')]
|
||||
"""
|
||||
|
||||
def __init__(self, annotation, attribute_descriptors=None):
|
||||
self.annotation = annotation
|
||||
if attribute_descriptors is None:
|
||||
attribute_descriptors = {}
|
||||
|
||||
if annotation.surrounds_attribute:
|
||||
descriptor = attribute_descriptors.get(annotation.surrounds_attribute)
|
||||
if descriptor:
|
||||
self.content_validate = descriptor.extractor
|
||||
self.allow_markup = descriptor.allow_markup
|
||||
else:
|
||||
self.content_validate = _ID
|
||||
self.allow_markup = False
|
||||
self.extract = self._extract_content
|
||||
|
||||
if annotation.tag_attributes:
|
||||
self.tag_data = []
|
||||
for (tag_attr, extraction_attr) in annotation.tag_attributes:
|
||||
descriptor = attribute_descriptors.get(extraction_attr)
|
||||
extractf = descriptor.extractor if descriptor else _ID
|
||||
self.tag_data.append((extractf, tag_attr, extraction_attr))
|
||||
|
||||
self.extract = self._extract_both if \
|
||||
annotation.surrounds_attribute else self._extract_attribute
|
||||
|
||||
def _extract_both(self, page, start_index, end_index, ignored_regions=None):
|
||||
return self._extract_content(page, start_index, end_index, ignored_regions) + \
|
||||
self._extract_attribute(page, start_index, end_index, ignored_regions)
|
||||
|
||||
def _extract_content(self, extraction_page, start_index, end_index, ignored_regions=None):
|
||||
# we might want to add opening/closing ul/ol/table if we have the
|
||||
# middle of a region. This would require support in the scrapy
|
||||
# cleansing.
|
||||
complete_data = ""
|
||||
start = start_index
|
||||
end = ignored_regions[0].start_index if ignored_regions else end_index
|
||||
while start is not None:
|
||||
if self.allow_markup:
|
||||
data = extraction_page.html_between_tokens(start, end)
|
||||
else:
|
||||
data = extraction_page.text_between_tokens(start, end)
|
||||
complete_data += data
|
||||
if ignored_regions:
|
||||
start = ignored_regions[0].end_index
|
||||
ignored_regions.pop(0)
|
||||
end = ignored_regions[0].start_index if ignored_regions else end_index
|
||||
else:
|
||||
start = None
|
||||
complete_data = self.content_validate(complete_data)
|
||||
return [(self.annotation.surrounds_attribute, complete_data)] if complete_data else []
|
||||
|
||||
def _extract_attribute(self, extraction_page, start_index, end_index, ignored_regions=None):
|
||||
data = []
|
||||
for (f, ta, ea) in self.tag_data:
|
||||
tag_value = extraction_page.tag_attribute(start_index, ta)
|
||||
if tag_value:
|
||||
extracted = f(tag_value)
|
||||
if extracted is not None:
|
||||
data.append((ea, extracted))
|
||||
return data
|
||||
|
||||
@classmethod
|
||||
def create(cls, annotations, attribute_descriptors=None):
|
||||
"""Create a list of basic extractors from the given annotations
|
||||
and attribute descriptors
|
||||
"""
|
||||
if attribute_descriptors is None:
|
||||
attribute_descriptors = {}
|
||||
return [cls._create_basic_extractor(annotation, attribute_descriptors) \
|
||||
for annotation in annotations \
|
||||
if annotation.surrounds_attribute or annotation.tag_attributes]
|
||||
|
||||
@staticmethod
|
||||
def _create_basic_extractor(annotation, attribute_descriptors):
|
||||
"""Create a basic type extractor for the annotation"""
|
||||
text_region = annotation.annotation_text
|
||||
if text_region is not None:
|
||||
region_extract = TextRegionDataExtractor(text_region.start_text,
|
||||
text_region.follow_text).extract
|
||||
# copy attribute_descriptors and add the text extractor
|
||||
descriptor_copy = dict(attribute_descriptors)
|
||||
attr_descr = descriptor_copy.get(annotation.surrounds_attribute,
|
||||
_DEFAULT_DESCRIPTOR)
|
||||
attr_descr = copy.copy(attr_descr)
|
||||
attr_descr.extractor = _compose(attr_descr.extractor, region_extract)
|
||||
descriptor_copy[annotation.surrounds_attribute] = attr_descr
|
||||
attribute_descriptors = descriptor_copy
|
||||
return BasicTypeExtractor(annotation, attribute_descriptors)
|
||||
|
||||
def extracted_item(self):
|
||||
"""key used to identify the item extracted"""
|
||||
return (self.annotation.surrounds_attribute, self.annotation.tag_attributes)
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
def __str__(self):
|
||||
messages = ['BasicTypeExtractor(']
|
||||
if self.annotation.surrounds_attribute:
|
||||
messages += [self.annotation.surrounds_attribute, ': ',
|
||||
'html content' if self.allow_markup else 'text content',
|
||||
]
|
||||
if self.content_validate != _ID:
|
||||
messages += [', extracted with \'',
|
||||
self.content_validate.__name__, '\'']
|
||||
|
||||
if self.annotation.tag_attributes:
|
||||
if self.annotation.surrounds_attribute:
|
||||
messages.append(';')
|
||||
for (f, ta, ea) in self.tag_data:
|
||||
messages += [ea, ': tag attribute "', ta, '"']
|
||||
if f != _ID:
|
||||
messages += [', validated by ', str(f)]
|
||||
messages.append(", template[%s:%s])" % \
|
||||
(self.annotation.start_index, self.annotation.end_index))
|
||||
return ''.join(messages)
|
||||
|
||||
class RepeatedDataExtractor(object):
|
||||
"""Data extractor for handling repeated data"""
|
||||
|
||||
def __init__(self, prefix, suffix, extractors):
|
||||
self.prefix = array(prefix)
|
||||
self.suffix = array(suffix)
|
||||
self.extractor = copy.copy(extractors[0])
|
||||
self.annotation = copy.copy(self.extractor.annotation)
|
||||
self.annotation.end_index = extractors[-1].annotation.end_index
|
||||
|
||||
def extract(self, page, start_index, end_index, ignored_regions):
|
||||
"""repeatedly find regions bounded by the repeated
|
||||
prefix and suffix and extract them
|
||||
"""
|
||||
prefixlen = len(self.prefix)
|
||||
suffixlen = len(self.suffix)
|
||||
index = max(0, start_index - prefixlen)
|
||||
max_index = min(len(page.page_tokens) - suffixlen, end_index + len(self.suffix))
|
||||
max_start_index = max_index - prefixlen
|
||||
extracted = []
|
||||
while index <= max_start_index:
|
||||
prefix_end = index + prefixlen
|
||||
if (page.page_tokens[index:prefix_end] == self.prefix).all():
|
||||
for peek in xrange(prefix_end, max_index):
|
||||
if (page.page_tokens[peek:peek + suffixlen] \
|
||||
== self.suffix).all():
|
||||
extracted += self.extractor.extract(page,
|
||||
prefix_end - 1, peek, ignored_regions)
|
||||
index = max(peek, index + 1)
|
||||
break
|
||||
else:
|
||||
break
|
||||
else:
|
||||
index += 1
|
||||
return extracted
|
||||
|
||||
@staticmethod
|
||||
def apply(template, extractors):
|
||||
tokens = template.page_tokens
|
||||
output_extractors = []
|
||||
group_key = lambda x: x.extracted_item()
|
||||
for extr_key, extraction_group in groupby(extractors, group_key):
|
||||
extraction_group = list(extraction_group)
|
||||
if extr_key is None or len(extraction_group) == 1:
|
||||
output_extractors += extraction_group
|
||||
continue
|
||||
|
||||
separating_tokens = [ \
|
||||
tokens[x.annotation.end_index:y.annotation.start_index+1] \
|
||||
for (x, y) in zip(extraction_group[:-1], extraction_group[1:])]
|
||||
|
||||
# calculate the common prefix
|
||||
group_start = extraction_group[0].annotation.start_index
|
||||
prefix_start = max(0, group_start - len(separating_tokens[0]))
|
||||
first_prefix = tokens[prefix_start:group_start+1]
|
||||
prefixes = [first_prefix] + separating_tokens
|
||||
prefix_pattern = list(reversed(
|
||||
common_prefix(*map(reversed, prefixes))))
|
||||
|
||||
# calculate the common suffix
|
||||
group_end = extraction_group[-1].annotation.end_index
|
||||
last_suffix = tokens[group_end:group_end + \
|
||||
len(separating_tokens[-1])]
|
||||
suffixes = separating_tokens + [last_suffix]
|
||||
suffix_pattern = common_prefix(*suffixes)
|
||||
|
||||
# create a repeated data extractor, if there is a suitable
|
||||
# prefix and suffix. (TODO: tune this heuristic)
|
||||
matchlen = len(prefix_pattern) + len(suffix_pattern)
|
||||
if matchlen >= len(separating_tokens):
|
||||
group_extractor = RepeatedDataExtractor(prefix_pattern,
|
||||
suffix_pattern, extraction_group)
|
||||
output_extractors.append(group_extractor)
|
||||
else:
|
||||
output_extractors += extraction_group
|
||||
return output_extractors
|
||||
|
||||
def extracted_item(self):
|
||||
"""key used to identify the item extracted"""
|
||||
return self.extractor.extracted_item()
|
||||
|
||||
def __repr__(self):
|
||||
return "Repeat(%r)" % self.extractor
|
||||
|
||||
def __str__(self):
|
||||
return "Repeat(%s)" % self.extractor
|
||||
|
||||
class TransposedDataExtractor(object):
|
||||
""" """
|
||||
pass
|
||||
|
||||
_namef = operator.itemgetter(0)
|
||||
_valuef = operator.itemgetter(1)
|
||||
def _attrs2dict(attributes):
|
||||
"""convert a list of attributes (name, value) tuples
|
||||
into a dict of lists.
|
||||
|
||||
For example:
|
||||
>>> l = [('name', 'sofa'), ('colour', 'red'), ('colour', 'green')]
|
||||
>>> _attrs2dict(l) == {'name': ['sofa'], 'colour': ['red', 'green']}
|
||||
True
|
||||
"""
|
||||
grouped_data = groupby(sorted(attributes, key=_namef), _namef)
|
||||
return dict((name, map(_valuef, data)) for (name, data) in grouped_data)
|
||||
|
||||
class RecordExtractor(object):
|
||||
"""The RecordExtractor will extract records given annotations.
|
||||
|
||||
It looks for a similar region in the target document, using the ibl
|
||||
similarity algorithm. The annotations are partitioned by the first similar
|
||||
region found and searched recursively.
|
||||
|
||||
Records are represented as dicts mapping attribute names to lists
|
||||
containing their values.
|
||||
|
||||
For example:
|
||||
>>> from scrapy.contrib.ibl.extraction.pageparsing import parse_strings
|
||||
>>> template, page = parse_strings( \
|
||||
u'<h1 data-scrapy-annotate="{"annotations": {"content": "name"}}">x</h1>' + \
|
||||
u'<p data-scrapy-annotate="{"annotations": {"content": "description"}}">y</p>', \
|
||||
u'<h1>name</h1> <p>description</p>')
|
||||
>>> basic_extractors = map(BasicTypeExtractor, template.annotations)
|
||||
>>> ex = RecordExtractor.apply(template, basic_extractors)[0]
|
||||
>>> ex.extract(page)
|
||||
[{u'description': [u'description'], u'name': [u'name']}]
|
||||
"""
|
||||
|
||||
def __init__(self, extractors, template_tokens):
|
||||
"""Construct a RecordExtractor for the given annotations and their
|
||||
corresponding region extractors
|
||||
"""
|
||||
self.extractors = extractors
|
||||
self.template_tokens = template_tokens
|
||||
self.template_ignored_regions = []
|
||||
start_index = min(e.annotation.start_index for e in extractors)
|
||||
end_index = max(e.annotation.end_index for e in extractors)
|
||||
self.annotation = AnnotationTag(start_index, end_index)
|
||||
|
||||
def extract(self, page, start_index=0, end_index=None, ignored_regions=None):
|
||||
"""extract data from an extraction page
|
||||
|
||||
The region in the page to be extracted from may be specified using
|
||||
start_index and end_index
|
||||
"""
|
||||
ignored_regions = [i if isinstance(i, LabelledRegion) else LabelledRegion(*i) for i in (ignored_regions or [])]
|
||||
region_elements = sorted(self.extractors + ignored_regions, key=lambda x: _labelled(x).start_index)
|
||||
_, _, attributes = self._doextract(page, region_elements, start_index,
|
||||
end_index)
|
||||
# collect variant data, maintaining the order of variants
|
||||
variant_ids = []; variants = {}; items = []
|
||||
for k, v in attributes:
|
||||
if isinstance(k, int):
|
||||
if k in variants:
|
||||
variants[k] += v
|
||||
else:
|
||||
variant_ids.append(k)
|
||||
variants[k] = v
|
||||
else:
|
||||
items.append((k, v))
|
||||
|
||||
variant_records = [('variants', _attrs2dict(variants[vid])) \
|
||||
for vid in variant_ids]
|
||||
items += variant_records
|
||||
return [_attrs2dict(items)]
|
||||
|
||||
def _doextract(self, page, region_elements, start_index, end_index, nested_regions=None, ignored_regions=None):
|
||||
"""Carry out extraction of records using the given annotations
|
||||
in the page tokens bounded by start_index and end_index
|
||||
"""
|
||||
# reorder extractors leaving nested ones for the end and separating
|
||||
# ignore regions
|
||||
nested_regions = nested_regions or []
|
||||
ignored_regions = ignored_regions or []
|
||||
first_region, following_regions = region_elements[0], region_elements[1:]
|
||||
while following_regions and _labelled(following_regions[0]).start_index \
|
||||
< _labelled(first_region).end_index:
|
||||
region = following_regions.pop(0)
|
||||
labelled = _labelled(region)
|
||||
if isinstance(labelled, AnnotationTag) or (nested_regions and \
|
||||
_labelled(nested_regions[-1]).start_index < labelled.start_index \
|
||||
< _labelled(nested_regions[-1]).end_index):
|
||||
nested_regions.append(region)
|
||||
else:
|
||||
ignored_regions.append(region)
|
||||
extracted_data = []
|
||||
# end_index is inclusive, but similar_region treats it as exclusive
|
||||
end_region = None if end_index is None else end_index + 1
|
||||
labelled = _labelled(first_region)
|
||||
score, pindex, sindex = \
|
||||
similar_region(page.page_tokens, self.template_tokens,
|
||||
labelled, start_index, end_region)
|
||||
if score > 0:
|
||||
if isinstance(labelled, AnnotationTag):
|
||||
similar_ignored_regions = []
|
||||
start = pindex
|
||||
for i in ignored_regions:
|
||||
s, p, e = similar_region(page.page_tokens, self.template_tokens, \
|
||||
i, start, sindex)
|
||||
if s > 0:
|
||||
similar_ignored_regions.append(LabelledRegion(*(p, e)))
|
||||
start = e or start
|
||||
extracted_data = first_region.extract(page, pindex, sindex, similar_ignored_regions)
|
||||
if extracted_data:
|
||||
if first_region.annotation.variant_id:
|
||||
extracted_data = [(first_region.annotation.variant_id, extracted_data)]
|
||||
|
||||
if nested_regions:
|
||||
_, _, nested_data = self._doextract(page, nested_regions, pindex, sindex)
|
||||
extracted_data += nested_data
|
||||
if following_regions:
|
||||
_, _, following_data = self._doextract(page, following_regions, sindex or start_index, end_index)
|
||||
extracted_data += following_data
|
||||
|
||||
elif following_regions:
|
||||
end_index, _, following_data = self._doextract(page, following_regions, start_index, end_index)
|
||||
if end_index is not None:
|
||||
pindex, sindex, extracted_data = self._doextract(page, [first_region], start_index, end_index - 1, nested_regions, ignored_regions)
|
||||
extracted_data += following_data
|
||||
elif nested_regions:
|
||||
_, _, nested_data = self._doextract(page, nested_regions, start_index, end_index)
|
||||
extracted_data += nested_data
|
||||
return pindex, sindex, extracted_data
|
||||
|
||||
@classmethod
|
||||
def apply(cls, template, extractors):
|
||||
return [cls(extractors, template.page_tokens)]
|
||||
|
||||
def extracted_item(self):
|
||||
return [self.__class__.__name__] + \
|
||||
sorted(e.extracted_item() for e in self.extractors)
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
def __str__(self):
|
||||
stream = cStringIO.StringIO()
|
||||
pprint.pprint(self.extractors, stream)
|
||||
stream.seek(0)
|
||||
template_data = stream.read()
|
||||
if template_data:
|
||||
return "%s[\n%s\n]" % (self.__class__.__name__, template_data)
|
||||
return "%s[none]" % (self.__class__.__name__)
|
||||
|
||||
class AdjacentVariantExtractor(RecordExtractor):
|
||||
"""Extractor for variants
|
||||
|
||||
This simply extends the RecordExtractor to output data in a "variants"
|
||||
attribute.
|
||||
|
||||
The "apply" method will only apply to variants whose items are all adjacent and
|
||||
it will appear as one record so that it can be handled by the RepeatedDataExtractor.
|
||||
"""
|
||||
|
||||
def extract(self, page, start_index=0, end_index=None, ignored_regions=None):
|
||||
records = RecordExtractor.extract(self, page, start_index, end_index, ignored_regions)
|
||||
return [('variants', r['variants'][0]) for r in records if r]
|
||||
|
||||
@classmethod
|
||||
def apply(cls, template, extractors):
|
||||
adjacent_variants = set([])
|
||||
variantf = lambda x: x.annotation.variant_id
|
||||
for vid, egroup in groupby(extractors, variantf):
|
||||
if not vid:
|
||||
continue
|
||||
if vid in adjacent_variants:
|
||||
adjacent_variants.remove(vid)
|
||||
else:
|
||||
adjacent_variants.add(vid)
|
||||
new_extractors = []
|
||||
for variant, group_seq in groupby(extractors, variantf):
|
||||
group_seq = list(group_seq)
|
||||
if variant in adjacent_variants:
|
||||
record_extractor = AdjacentVariantExtractor(group_seq, template.page_tokens)
|
||||
new_extractors.append(record_extractor)
|
||||
else:
|
||||
new_extractors += group_seq
|
||||
return new_extractors
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
class TraceExtractor(object):
|
||||
"""Extractor that wraps other extractors and prints an execution
|
||||
trace of the extraction process to aid debugging
|
||||
"""
|
||||
|
||||
def __init__(self, traced, template):
|
||||
self.traced = traced
|
||||
self.annotation = traced.annotation
|
||||
tstart = traced.annotation.start_index
|
||||
tend = traced.annotation.end_index
|
||||
self.tprefix = " ".join([template.token_dict.token_string(t)
|
||||
for t in template.page_tokens[tstart-4:tstart+1]])
|
||||
self.tsuffix = " ".join([template.token_dict.token_string(t)
|
||||
for t in template.page_tokens[tend:tend+5]])
|
||||
|
||||
def summarize_trace(self, page, start, end, ret):
|
||||
text_start = page.token_follow_indexes[start]
|
||||
text_end = page.token_start_indexes[end or -1]
|
||||
page_snippet = "(...%s)%s(%s...)" % (
|
||||
page.text[text_start-50:text_start].replace('\n', ' '),
|
||||
page.text[text_start:text_end],
|
||||
page.text[text_end:text_end+50].replace('\n', ' '))
|
||||
pre_summary = "\nstart %s page[%s:%s]\n" % (self.traced.__class__.__name__, start, end)
|
||||
post_summary = """
|
||||
%s page[%s:%s]
|
||||
|
||||
html
|
||||
%s
|
||||
|
||||
annotation
|
||||
...%s
|
||||
%s
|
||||
%s...
|
||||
|
||||
extracted
|
||||
%s
|
||||
""" % (self.traced.__class__.__name__, start, end, page_snippet,
|
||||
self.tprefix, self.annotation, self.tsuffix, [r for r in ret if 'trace' not in r])
|
||||
return pre_summary, post_summary
|
||||
|
||||
def extract(self, page, start, end, ignored_regions):
|
||||
ret = self.traced.extract(page, start, end, ignored_regions)
|
||||
if not ret:
|
||||
return []
|
||||
|
||||
# handle records by inserting a trace and combining with variant traces
|
||||
if len(ret) == 1 and isinstance(ret[0], dict):
|
||||
item = ret[0]
|
||||
trace = item.pop('trace', [])
|
||||
variants = item.get('variants', ())
|
||||
for variant in variants:
|
||||
trace += variant.pop('trace', [])
|
||||
pre_summary, post_summary = self.summarize_trace(page, start, end, ret)
|
||||
item['trace'] = [pre_summary] + trace + [post_summary]
|
||||
return ret
|
||||
|
||||
pre_summary, post_summary = self.summarize_trace(page, start, end, ret)
|
||||
return [('trace', pre_summary)] + ret + [('trace', post_summary)]
|
||||
|
||||
@staticmethod
|
||||
def apply(template, extractors):
|
||||
output = []
|
||||
for extractor in extractors:
|
||||
if not isinstance(extractor, TraceExtractor):
|
||||
extractor = TraceExtractor(extractor, template)
|
||||
output.append(extractor)
|
||||
return output
|
||||
|
||||
def extracted_item(self):
|
||||
return self.traced.extracted_item()
|
||||
|
||||
def __repr__(self):
|
||||
return "Trace(%s)" % repr(self.traced)
|
||||
|
||||
class TemplatePageExtractor(object):
|
||||
"""Top level extractor for a template page"""
|
||||
|
||||
def __init__(self, template, extractors):
|
||||
# fixme: handle multiple items per page
|
||||
self.extractor = extractors[0]
|
||||
self.template = template
|
||||
|
||||
def extract(self, page, start_index=0, end_index=None):
|
||||
return self.extractor.extract(page, start_index, end_index, self.template.ignored_regions)
|
||||
|
||||
def __repr__(self):
|
||||
return repr(self.extractor)
|
||||
|
||||
def __str__(self):
|
||||
return str(self.extractor)
|
||||
|
||||
# Based on nltk's WordPunctTokenizer
|
||||
_tokenize = re.compile(r'\w+|[^\w\s]+', re.UNICODE | re.MULTILINE | re.DOTALL).findall
|
||||
|
||||
class TextRegionDataExtractor(object):
|
||||
"""Data Extractor for extracting text fragments from within a larger
|
||||
body of text. It extracts based on the longest unique prefix and suffix.
|
||||
|
||||
for example:
|
||||
>>> extractor = TextRegionDataExtractor('designed by ', '.')
|
||||
>>> extractor.extract("by Marc Newson.")
|
||||
'Marc Newson'
|
||||
|
||||
Both prefix and suffix are optional:
|
||||
>>> extractor = TextRegionDataExtractor('designed by ')
|
||||
>>> extractor.extract("by Marc Newson.")
|
||||
'Marc Newson.'
|
||||
>>> extractor = TextRegionDataExtractor(suffix='.')
|
||||
>>> extractor.extract("by Marc Newson.")
|
||||
'by Marc Newson'
|
||||
|
||||
It requires a minimum match of at least one word or punctuation character:
|
||||
>>> extractor = TextRegionDataExtractor('designed by')
|
||||
>>> extractor.extract("y Marc Newson.") is None
|
||||
True
|
||||
"""
|
||||
def __init__(self, prefix=None, suffix=None):
|
||||
self.prefix = (prefix or '')[::-1]
|
||||
self.suffix = suffix or ''
|
||||
self.minprefix = self.minmatch(self.prefix)
|
||||
self.minsuffix = self.minmatch(self.suffix)
|
||||
|
||||
@staticmethod
|
||||
def minmatch(matchstring):
|
||||
"""the minimum number of characters that should match in order
|
||||
to consider it a match for that string.
|
||||
|
||||
This uses the last word of punctuation character
|
||||
"""
|
||||
tokens = _tokenize(matchstring or '')
|
||||
return len(tokens[0]) if tokens else 0
|
||||
|
||||
def extract(self, text):
|
||||
"""attempt to extract a substring from the text"""
|
||||
pref_index = 0
|
||||
if self.minprefix > 0:
|
||||
rev_idx, plen = longest_unique_subsequence(text[::-1], self.prefix)
|
||||
if plen < self.minprefix:
|
||||
return None
|
||||
pref_index = -rev_idx
|
||||
if self.minsuffix == 0:
|
||||
return text[pref_index:]
|
||||
sidx, slen = longest_unique_subsequence(text[pref_index:], self.suffix)
|
||||
if slen < self.minsuffix:
|
||||
return None
|
||||
return text[pref_index:pref_index + sidx]
|
||||
|
||||
|
||||
|
|
@ -1,136 +0,0 @@
|
|||
"""
|
||||
Similarity calculation for Instance based extraction algorithm.
|
||||
"""
|
||||
from itertools import izip, count
|
||||
from operator import itemgetter
|
||||
from heapq import nlargest
|
||||
|
||||
def common_prefix_length(a, b):
|
||||
"""Calculate the length of the common prefix in both sequences passed.
|
||||
|
||||
For example, the common prefix in this example is [1, 3]
|
||||
>>> common_prefix_length([1, 3, 4], [1, 3, 5, 1])
|
||||
2
|
||||
|
||||
If there is no common prefix, 0 is returned
|
||||
>>> common_prefix_length([1], [])
|
||||
0
|
||||
"""
|
||||
i = -1
|
||||
for i, x, y in izip(count(), a, b):
|
||||
if x != y:
|
||||
return i
|
||||
return i + 1
|
||||
|
||||
def common_prefix(*sequences):
|
||||
"""determine the common prefix of all sequences passed
|
||||
|
||||
For example:
|
||||
>>> common_prefix('abcdef', 'abc', 'abac')
|
||||
['a', 'b']
|
||||
"""
|
||||
prefix = []
|
||||
for sample in izip(*sequences):
|
||||
first = sample[0]
|
||||
if all(x == first for x in sample[1:]):
|
||||
prefix.append(first)
|
||||
else:
|
||||
break
|
||||
return prefix
|
||||
|
||||
def longest_unique_subsequence(to_search, subsequence, range_start=0,
|
||||
range_end=None):
|
||||
"""Find the longest unique subsequence of items in a list or array. This
|
||||
searches the to_search list or array looking for the longest overlapping
|
||||
match with subsequence. If the largest match is unique (there is no other
|
||||
match of equivalent length), the index and length of match is returned. If
|
||||
there is no match, (None, None) is returned.
|
||||
|
||||
Please see section 3.2 of Extracting Web Data Using Instance-Based
|
||||
Learning by Yanhong Zhai and Bing Liu
|
||||
|
||||
For example, the longest match occurs at index 2 and has length 3
|
||||
>>> to_search = [6, 3, 2, 4, 3, 2, 5]
|
||||
>>> longest_unique_subsequence(to_search, [2, 4, 3])
|
||||
(2, 3)
|
||||
|
||||
When there are two equally long subsequences, it does not generate a match
|
||||
>>> longest_unique_subsequence(to_search, [3, 2])
|
||||
(None, None)
|
||||
|
||||
range_start and range_end specify a range in which the match must begin
|
||||
>>> longest_unique_subsequence(to_search, [3, 2], 3)
|
||||
(4, 2)
|
||||
>>> longest_unique_subsequence(to_search, [3, 2], 0, 2)
|
||||
(1, 2)
|
||||
"""
|
||||
startval = subsequence[0]
|
||||
if range_end is None:
|
||||
range_end = len(to_search)
|
||||
|
||||
# the comparison to startval ensures only matches of length >= 1 and
|
||||
# reduces the number of calls to the common_length function
|
||||
matches = ((i, common_prefix_length(to_search[i:], subsequence)) \
|
||||
for i in xrange(range_start, range_end) if startval == to_search[i])
|
||||
best2 = nlargest(2, matches, key=itemgetter(1))
|
||||
# if there is a single unique best match, return that
|
||||
if len(best2) == 1 or len(best2) == 2 and best2[0][1] != best2[1][1]:
|
||||
return best2[0]
|
||||
return None, None
|
||||
|
||||
def similar_region(extracted_tokens, template_tokens, labelled_region,
|
||||
range_start=0, range_end=None):
|
||||
"""Given a labelled section in a template, identify a similar region
|
||||
in the extracted tokens.
|
||||
|
||||
The start and end index of the similar region in the extracted tokens
|
||||
is returned.
|
||||
|
||||
This will return a tuple containing:
|
||||
(match score, start index, end index)
|
||||
where match score is the sum of the length of the matching prefix and
|
||||
suffix. If there is no unique match, (0, None, None) will be returned.
|
||||
|
||||
start_index and end_index specify a range in which the match must begin
|
||||
"""
|
||||
data_length = len(extracted_tokens)
|
||||
if range_end is None:
|
||||
range_end = data_length
|
||||
# calculate the prefix score by finding a longest subsequence in
|
||||
# reverse order
|
||||
reverse_prefix = template_tokens[labelled_region.start_index::-1]
|
||||
reverse_tokens = extracted_tokens[::-1]
|
||||
(rpi, pscore) = longest_unique_subsequence(reverse_tokens, reverse_prefix,
|
||||
data_length - range_end, data_length - range_start)
|
||||
|
||||
# None means nothing exracted. Index 0 means there cannot be a suffix.
|
||||
if not rpi:
|
||||
return 0, None, None
|
||||
|
||||
# convert to an index from the start instead of in reverse
|
||||
prefix_index = len(extracted_tokens) - rpi - 1
|
||||
|
||||
if labelled_region.end_index is None:
|
||||
return pscore, prefix_index, None
|
||||
|
||||
suffix = template_tokens[labelled_region.end_index:]
|
||||
|
||||
# if it's not a paired tag, use the best match between prefix & suffix
|
||||
if labelled_region.start_index == labelled_region.end_index:
|
||||
(match_index, sscore) = longest_unique_subsequence(extracted_tokens,
|
||||
suffix, prefix_index, range_end)
|
||||
if match_index == prefix_index:
|
||||
return (pscore + sscore, prefix_index, match_index)
|
||||
elif pscore > sscore:
|
||||
return pscore, prefix_index, prefix_index
|
||||
elif sscore > pscore:
|
||||
return sscore, match_index, match_index
|
||||
return 0, None, None
|
||||
|
||||
# calculate the suffix match on the tokens following the prefix. We could
|
||||
# consider the whole page and require a good match.
|
||||
(match_index, sscore) = longest_unique_subsequence(extracted_tokens,
|
||||
suffix, prefix_index + 1, range_end)
|
||||
if match_index is None:
|
||||
return 0, None, None
|
||||
return (pscore + sscore, prefix_index, match_index)
|
||||
|
|
@ -1,156 +0,0 @@
|
|||
"""
|
||||
Extractors for attributes
|
||||
"""
|
||||
import re
|
||||
import urlparse
|
||||
|
||||
from scrapy.utils.markup import remove_entities
|
||||
from scrapy.utils.url import safe_url_string
|
||||
|
||||
#FIXME: the use of "." needs to be localized
|
||||
_NUMERIC_ENTITIES = re.compile("&#([0-9]+)(?:;|\s)", re.U)
|
||||
_PRICE_NUMBER_RE = re.compile('(?:^|[^a-zA-Z0-9])(\d+(?:\.\d+)?)(?:$|[^a-zA-Z0-9])')
|
||||
_NUMBER_RE = re.compile('(\d+(?:\.\d+)?)')
|
||||
|
||||
_IMAGES = (
|
||||
'mng', 'pct', 'bmp', 'gif', 'jpg', 'jpeg', 'png', 'pst', 'psp', 'tif',
|
||||
'tiff', 'ai', 'drw', 'dxf', 'eps', 'ps', 'svg',
|
||||
)
|
||||
|
||||
_IMAGES_TYPES = '|'.join(_IMAGES)
|
||||
_CSS_IMAGERE = re.compile("background(?:-image)?\s*:\s*url\((.*?)\)", re.I)
|
||||
_BASE_PATH_RE = "/?(?:[^/]+/)*(?:.+%s)"
|
||||
_IMAGE_PATH_RE = re.compile(_BASE_PATH_RE % '\.(?:%s)' % _IMAGES_TYPES, re.I)
|
||||
_GENERIC_PATH_RE = re.compile(_BASE_PATH_RE % '', re.I)
|
||||
|
||||
def text(txt):
|
||||
stripped = txt.strip() if txt else None
|
||||
if stripped:
|
||||
return stripped
|
||||
|
||||
def contains_any_numbers(txt):
|
||||
"""text that must contain at least one number
|
||||
>>> contains_any_numbers('foo')
|
||||
>>> contains_any_numbers('$67 at 15% discount')
|
||||
'$67 at 15% discount'
|
||||
"""
|
||||
if _NUMBER_RE.search(txt) is not None:
|
||||
return txt
|
||||
|
||||
def contains_prices(txt):
|
||||
"""text must contain a number that is not joined to text"""
|
||||
if _PRICE_NUMBER_RE.findall(txt) is not None:
|
||||
return txt
|
||||
|
||||
def contains_numbers(txt, count=1):
|
||||
"""Must contain a certain amount of numbers
|
||||
|
||||
>>> contains_numbers('foo', 2)
|
||||
>>> contains_numbers('this 1 has 2 numbers', 2)
|
||||
'this 1 has 2 numbers'
|
||||
"""
|
||||
numbers = _NUMBER_RE.findall(txt)
|
||||
if len(numbers) == count:
|
||||
return txt
|
||||
|
||||
def extract_number(txt):
|
||||
"""Extract a numeric value.
|
||||
|
||||
This will fail if more than one numeric value is present.
|
||||
|
||||
>>> extract_number(' 45.3')
|
||||
'45.3'
|
||||
>>> extract_number(' 45.3, 7')
|
||||
|
||||
It will handle unescaped entities:
|
||||
>>> extract_number(u'£129.99')
|
||||
u'129.99'
|
||||
"""
|
||||
txt = _NUMERIC_ENTITIES.sub(lambda m: unichr(int(m.groups()[0])), txt)
|
||||
numbers = _NUMBER_RE.findall(txt)
|
||||
if len(numbers) == 1:
|
||||
return numbers[0]
|
||||
|
||||
def url(txt):
|
||||
"""convert text to a url
|
||||
|
||||
this is quite conservative, since relative urls are supported
|
||||
"""
|
||||
txt = txt.strip("\t\r\n '\"")
|
||||
if txt:
|
||||
return txt
|
||||
|
||||
def image_url(txt):
|
||||
"""convert text to a url
|
||||
|
||||
this is quite conservative, since relative urls are supported
|
||||
Example:
|
||||
|
||||
>>> image_url('')
|
||||
|
||||
>>> image_url(' ')
|
||||
|
||||
>>> image_url(' \\n\\n ')
|
||||
|
||||
>>> image_url('foo-bar.jpg')
|
||||
['foo-bar.jpg']
|
||||
>>> image_url('/images/main_logo12.gif')
|
||||
['/images/main_logo12.gif']
|
||||
>>> image_url("http://www.image.com/image.jpg")
|
||||
['http://www.image.com/image.jpg']
|
||||
>>> image_url("http://www.domain.com/path1/path2/path3/image.jpg")
|
||||
['http://www.domain.com/path1/path2/path3/image.jpg']
|
||||
>>> image_url("/path1/path2/path3/image.jpg")
|
||||
['/path1/path2/path3/image.jpg']
|
||||
>>> image_url("path1/path2/image.jpg")
|
||||
['path1/path2/image.jpg']
|
||||
>>> image_url("background-image : url(http://www.site.com/path1/path2/image.jpg)")
|
||||
['http://www.site.com/path1/path2/image.jpg']
|
||||
>>> image_url("background-image : url('http://www.site.com/path1/path2/image.jpg')")
|
||||
['http://www.site.com/path1/path2/image.jpg']
|
||||
>>> image_url('background-image : url("http://www.site.com/path1/path2/image.jpg")')
|
||||
['http://www.site.com/path1/path2/image.jpg']
|
||||
>>> image_url("background : url(http://www.site.com/path1/path2/image.jpg)")
|
||||
['http://www.site.com/path1/path2/image.jpg']
|
||||
>>> image_url("background : url('http://www.site.com/path1/path2/image.jpg')")
|
||||
['http://www.site.com/path1/path2/image.jpg']
|
||||
>>> image_url('background : url("http://www.site.com/path1/path2/image.jpg")')
|
||||
['http://www.site.com/path1/path2/image.jpg']
|
||||
>>> image_url('/getimage.php?image=totalgardens/outbbq2_400.jpg&type=prod&resizeto=350')
|
||||
['/getimage.php?image=totalgardens/outbbq2_400.jpg&type=prod&resizeto=350']
|
||||
>>> image_url('http://www.site.com/getimage.php?image=totalgardens/outbbq2_400.jpg&type=prod&resizeto=350')
|
||||
['http://www.site.com/getimage.php?image=totalgardens/outbbq2_400.jpg&type=prod&resizeto=350']
|
||||
>>> image_url('http://s7d4.scene7.com/is/image/Kohler/jaa03267?hei=425&wid=457&op_usm=2,1,2,1&qlt=80')
|
||||
['http://s7d4.scene7.com/is/image/Kohler/jaa03267?hei=425&wid=457&op_usm=2,1,2,1&qlt=80']
|
||||
>>> image_url('../image.aspx?thumb=true&boxSize=175&img=Unknoportrait[1].jpg')
|
||||
['../image.aspx?thumb=true&boxSize=175&img=Unknoportrait%5B1%5D.jpg']
|
||||
>>> image_url('http://www.sundancecatalog.com/mgen/catalog/test.ms?args=%2245932|MERIDIAN+PENDANT|.jpg%22&is=336,336,0xffffff')
|
||||
['http://www.sundancecatalog.com/mgen/catalog/test.ms?args=%2245932|MERIDIAN+PENDANT|.jpg%22&is=336,336,0xffffff']
|
||||
>>> image_url('http://www.site.com/image.php')
|
||||
['http://www.site.com/image.php']
|
||||
>>> image_url('background-image:URL(http://s7d5.scene7.com/is/image/wasserstrom/165133?wid=227&hei=227&defaultImage=noimage_wasserstrom)')
|
||||
['http://s7d5.scene7.com/is/image/wasserstrom/165133?wid=227&hei=227&defaultImage=noimage_wasserstrom']
|
||||
|
||||
"""
|
||||
txt = url(txt)
|
||||
imgurl = None
|
||||
if txt:
|
||||
# check if the text is style content
|
||||
m = _CSS_IMAGERE.search(txt)
|
||||
txt = m.groups()[0] if m else txt
|
||||
parsed = urlparse.urlparse(txt)
|
||||
path = None
|
||||
m = _IMAGE_PATH_RE.search(parsed.path)
|
||||
if m:
|
||||
path = m.group()
|
||||
elif parsed.query:
|
||||
m = _GENERIC_PATH_RE.search(parsed.path)
|
||||
if m:
|
||||
path = m.group()
|
||||
if path is not None:
|
||||
parsed = list(parsed)
|
||||
parsed[2] = path
|
||||
imgurl = urlparse.urlunparse(parsed)
|
||||
if not imgurl:
|
||||
imgurl = txt
|
||||
return [safe_url_string(remove_entities(url(imgurl)))] if imgurl else None
|
||||
|
|
@ -1,167 +0,0 @@
|
|||
"""
|
||||
htmlpage
|
||||
|
||||
Container object for representing html pages in the IBL system. This
|
||||
encapsulates page related information and prevents parsing multiple times.
|
||||
"""
|
||||
import re
|
||||
import hashlib
|
||||
|
||||
from scrapy.utils.python import str_to_unicode
|
||||
|
||||
def create_page_from_jsonpage(jsonpage, body_key):
|
||||
"""Create an HtmlPage object from a dict object conforming to the schema
|
||||
for a page
|
||||
|
||||
`body_key` is the key where the body is stored and can be either 'body'
|
||||
(original page with annotations - if any) or 'original_body' (original
|
||||
page, always). Classification typically uses 'original_body' to avoid
|
||||
confusing the classifier with annotated pages, while extraction uses 'body'
|
||||
to pass the annotated pages.
|
||||
"""
|
||||
url = jsonpage['url']
|
||||
headers = jsonpage.get('headers')
|
||||
body = str_to_unicode(jsonpage[body_key])
|
||||
page_id = jsonpage.get('page_id')
|
||||
return HtmlPage(url, headers, body, page_id)
|
||||
|
||||
class HtmlPage(object):
|
||||
def __init__(self, url=None, headers=None, body=None, page_id=None):
|
||||
assert isinstance(body, unicode), "unicode expected, got: %s" % type(body).__name__
|
||||
self.headers = headers or {}
|
||||
self.body = body
|
||||
self.url = url or u''
|
||||
if page_id is None and url:
|
||||
self.page_id = hashlib.sha1(url).hexdigest()
|
||||
else:
|
||||
self.page_id = page_id
|
||||
|
||||
|
||||
def _set_body(self, body):
|
||||
self._body = body
|
||||
self.parsed_body = list(parse_html(body))
|
||||
|
||||
body = property(lambda x: x._body, _set_body)
|
||||
|
||||
def fragment_data(self, data_fragment):
|
||||
return self.body[data_fragment.start:data_fragment.end]
|
||||
|
||||
class HtmlTagType(object):
|
||||
OPEN_TAG = 1
|
||||
CLOSE_TAG = 2
|
||||
UNPAIRED_TAG = 3
|
||||
|
||||
class HtmlDataFragment(object):
|
||||
__slots__ = ('start', 'end')
|
||||
|
||||
def __init__(self, start, end):
|
||||
self.start = start
|
||||
self.end = end
|
||||
|
||||
def __str__(self):
|
||||
return "<HtmlDataFragment [%s:%s]>" % (self.start, self.end)
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
class HtmlTag(HtmlDataFragment):
|
||||
__slots__ = ('tag_type', 'tag', 'attributes')
|
||||
|
||||
def __init__(self, tag_type, tag, attributes, start, end):
|
||||
HtmlDataFragment.__init__(self, start, end)
|
||||
self.tag_type = tag_type
|
||||
self.tag = tag
|
||||
self.attributes = attributes
|
||||
|
||||
def __str__(self):
|
||||
return "<HtmlTag tag='%s' attributes={%s} [%s:%s]>" % (self.tag, ', '.join(sorted\
|
||||
(["%s: %s" % (k, repr(v)) for k, v in self.attributes.items()])), self.start, self.end)
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
_ATTR = "((?:[^=/>\s]|/(?!>))+)(?:\s*=(?:\s*\"(.*?)\"|\s*'(.*?)'|([^>\s]+))?)?"
|
||||
_TAG = "<(\/?)(\w+(?::\w+)?)((?:\s+" + _ATTR + ")+\s*|\s*)(\/?)>"
|
||||
_DOCTYPE = r"<!DOCTYPE.*?>"
|
||||
_SCRIPT = "(<script.*?>)(.*?)(</script.*?>)"
|
||||
_COMMENT = "(<!--.*?-->)"
|
||||
|
||||
_ATTR_REGEXP = re.compile(_ATTR, re.I | re.DOTALL)
|
||||
_HTML_REGEXP = re.compile("%s|%s|%s" % (_COMMENT, _SCRIPT, _TAG), re.I | re.DOTALL)
|
||||
_DOCTYPE_REGEXP = re.compile("(?:%s)" % _DOCTYPE)
|
||||
_COMMENT_REGEXP = re.compile(_COMMENT, re.DOTALL)
|
||||
|
||||
def parse_html(text):
|
||||
"""Higher level html parser. Calls lower level parsers and joins sucesive
|
||||
HtmlDataFragment elements in a single one.
|
||||
"""
|
||||
# If have doctype remove it.
|
||||
start_pos = 0
|
||||
match = _DOCTYPE_REGEXP.match(text)
|
||||
if match:
|
||||
start_pos = match.end()
|
||||
prev_end = start_pos
|
||||
for match in _HTML_REGEXP.finditer(text, start_pos):
|
||||
start = match.start()
|
||||
end = match.end()
|
||||
|
||||
if start > prev_end:
|
||||
yield HtmlDataFragment(prev_end, start)
|
||||
|
||||
if match.groups()[0] is not None: # comment
|
||||
yield HtmlDataFragment(start, end)
|
||||
elif match.groups()[1] is not None: # <script>...</script>
|
||||
for e in _parse_script(match):
|
||||
yield e
|
||||
else: # tag
|
||||
yield _parse_tag(match)
|
||||
prev_end = end
|
||||
textlen = len(text)
|
||||
if prev_end < textlen:
|
||||
yield HtmlDataFragment(prev_end, textlen)
|
||||
|
||||
def _parse_script(match):
|
||||
"""parse a <script>...</script> region matched by _HTML_REGEXP"""
|
||||
open_text, content, close_text = match.groups()[1:4]
|
||||
|
||||
open_tag = _parse_tag(_HTML_REGEXP.match(open_text))
|
||||
open_tag.start = match.start()
|
||||
open_tag.end = match.start() + len(open_text)
|
||||
|
||||
close_tag = _parse_tag(_HTML_REGEXP.match(close_text))
|
||||
close_tag.start = match.end() - len(close_text)
|
||||
close_tag.end = match.end()
|
||||
|
||||
yield open_tag
|
||||
if open_tag.end < close_tag.start:
|
||||
start_pos = 0
|
||||
for m in _COMMENT_REGEXP.finditer(content):
|
||||
if m.start() > start_pos:
|
||||
yield HtmlDataFragment(open_tag.end + start_pos, open_tag.end + m.start())
|
||||
yield HtmlDataFragment(open_tag.end + m.start(), open_tag.end + m.end())
|
||||
start_pos = m.end()
|
||||
if open_tag.end + start_pos < close_tag.start:
|
||||
yield HtmlDataFragment(open_tag.end + start_pos, close_tag.start)
|
||||
yield close_tag
|
||||
|
||||
def _parse_tag(match):
|
||||
"""
|
||||
parse a tag matched by _HTML_REGEXP
|
||||
"""
|
||||
data = match.groups()
|
||||
closing, tag, attr_text = data[4:7]
|
||||
# if tag is None then the match is a comment
|
||||
if tag is not None:
|
||||
unpaired = data[-1]
|
||||
if closing:
|
||||
tag_type = HtmlTagType.CLOSE_TAG
|
||||
elif unpaired:
|
||||
tag_type = HtmlTagType.UNPAIRED_TAG
|
||||
else:
|
||||
tag_type = HtmlTagType.OPEN_TAG
|
||||
attributes = []
|
||||
for attr_match in _ATTR_REGEXP.findall(attr_text):
|
||||
name = attr_match[0].lower()
|
||||
values = [v for v in attr_match[1:] if v]
|
||||
attributes.append((name, values[0] if values else None))
|
||||
return HtmlTag(tag_type, tag.lower(), dict(attributes), match.start(), match.end())
|
||||
|
|
@ -4,9 +4,10 @@ HTMLParser-based link extractor
|
|||
|
||||
from HTMLParser import HTMLParser
|
||||
|
||||
from w3lib.url import safe_url_string, urljoin_rfc
|
||||
|
||||
from scrapy.link import Link
|
||||
from scrapy.utils.python import unique as unique_list
|
||||
from scrapy.utils.url import safe_url_string, urljoin_rfc
|
||||
|
||||
class HtmlParserLinkExtractor(HTMLParser):
|
||||
|
||||
|
|
|
|||
|
|
@ -3,9 +3,9 @@ This module implements the HtmlImageLinkExtractor for extracting
|
|||
image links only.
|
||||
"""
|
||||
|
||||
|
||||
from w3lib.url import urljoin_rfc
|
||||
from scrapy.link import Link
|
||||
from scrapy.utils.url import canonicalize_url, urljoin_rfc
|
||||
from scrapy.utils.url import canonicalize_url
|
||||
from scrapy.utils.python import unicode_to_str, flatten
|
||||
from scrapy.selector.libxml2sel import XPathSelectorList, HtmlXPathSelector
|
||||
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ because it collides with the lxml library module.
|
|||
|
||||
from lxml import etree
|
||||
import lxml.html
|
||||
from w3lib.url import safe_url_string, urljoin_rfc
|
||||
|
||||
from scrapy.link import Link
|
||||
from scrapy.utils.python import unique as unique_list, str_to_unicode
|
||||
from scrapy.utils.url import safe_url_string, urljoin_rfc
|
||||
|
||||
class LxmlLinkExtractor(object):
|
||||
def __init__(self, tag="a", attr="href", process=None, unique=False):
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import re
|
||||
|
||||
from scrapy.utils.url import urljoin_rfc
|
||||
from scrapy.utils.markup import remove_tags, remove_entities, replace_escape_chars
|
||||
from w3lib.url import urljoin_rfc
|
||||
from w3lib.html import remove_tags, remove_entities, replace_escape_chars
|
||||
|
||||
from scrapy.link import Link
|
||||
from .sgml import SgmlLinkExtractor
|
||||
|
|
|
|||
|
|
@ -4,11 +4,13 @@ SGMLParser-based Link extractors
|
|||
|
||||
import re
|
||||
|
||||
from w3lib.url import safe_url_string, urljoin_rfc
|
||||
|
||||
from scrapy.selector import HtmlXPathSelector
|
||||
from scrapy.link import Link
|
||||
from scrapy.utils.misc import arg_to_iter
|
||||
from scrapy.utils.python import FixedSGMLParser, unique as unique_list, str_to_unicode
|
||||
from scrapy.utils.url import safe_url_string, urljoin_rfc, canonicalize_url, url_is_from_any_domain
|
||||
from scrapy.utils.url import canonicalize_url, url_is_from_any_domain
|
||||
from scrapy.utils.response import get_base_url
|
||||
|
||||
class BaseSgmlLinkExtractor(FixedSGMLParser):
|
||||
|
|
|
|||
|
|
@ -1,9 +1,10 @@
|
|||
"""Request Extractors"""
|
||||
from w3lib.url import safe_url_string, urljoin_rfc
|
||||
|
||||
from scrapy.http import Request
|
||||
from scrapy.selector import HtmlXPathSelector
|
||||
from scrapy.utils.misc import arg_to_iter
|
||||
from scrapy.utils.python import FixedSGMLParser, str_to_unicode
|
||||
from scrapy.utils.url import safe_url_string, urljoin_rfc
|
||||
|
||||
from itertools import ifilter
|
||||
|
||||
|
|
|
|||
|
|
@ -2,14 +2,19 @@ from scrapy.http import Response
|
|||
from scrapy.selector import XmlXPathSelector
|
||||
|
||||
|
||||
def xmliter_lxml(obj, nodename):
|
||||
def xmliter_lxml(obj, nodename, namespace=None):
|
||||
from lxml import etree
|
||||
reader = _StreamReader(obj)
|
||||
iterable = etree.iterparse(reader, tag=nodename, encoding=reader.encoding)
|
||||
tag = '{%s}%s' % (namespace, nodename) if namespace else nodename
|
||||
iterable = etree.iterparse(reader, tag=tag, encoding=reader.encoding)
|
||||
selxpath = '//' + ('x:%s' % nodename if namespace else nodename)
|
||||
for _, node in iterable:
|
||||
nodetext = etree.tostring(node)
|
||||
node.clear()
|
||||
yield XmlXPathSelector(text=nodetext).select('//' + nodename)[0]
|
||||
xs = XmlXPathSelector(text=nodetext)
|
||||
if namespace:
|
||||
xs.register_namespace('x', namespace)
|
||||
yield xs.select(selxpath)[0]
|
||||
|
||||
|
||||
class _StreamReader(object):
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
from w3lib.url import file_uri_to_path
|
||||
from scrapy.core.downloader.responsetypes import responsetypes
|
||||
from scrapy.utils.url import file_uri_to_path
|
||||
from scrapy.utils.decorator import defers
|
||||
|
||||
class FileDownloadHandler(object):
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
from w3lib.http import headers_dict_to_raw
|
||||
from scrapy.utils.datatypes import CaselessDict
|
||||
from scrapy.utils.http import headers_dict_to_raw
|
||||
|
||||
|
||||
class Headers(CaselessDict):
|
||||
|
|
|
|||
|
|
@ -7,8 +7,9 @@ See documentation in docs/topics/request-response.rst
|
|||
|
||||
import copy
|
||||
|
||||
from w3lib.url import safe_url_string
|
||||
|
||||
from scrapy.http.headers import Headers
|
||||
from scrapy.utils.url import safe_url_string
|
||||
from scrapy.utils.trackref import object_ref
|
||||
from scrapy.utils.decorator import deprecated
|
||||
from scrapy.http.common import deprecated_setter
|
||||
|
|
|
|||
|
|
@ -10,15 +10,19 @@ class Link(object):
|
|||
At the moment, it contains just the url and link text.
|
||||
"""
|
||||
|
||||
__slots__ = ['url', 'text']
|
||||
__slots__ = ['url', 'text', 'nofollow']
|
||||
|
||||
def __init__(self, url, text=''):
|
||||
def __init__(self, url, text='', nofollow=False):
|
||||
self.url = url
|
||||
self.text = text
|
||||
self.nofollow = nofollow
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.url == other.url and self.text == other.text
|
||||
return self.url == other.url and self.text == other.text and self.nofollow == other.nofollow
|
||||
|
||||
def __hash__(self):
|
||||
return hash(self.url) ^ hash(self.text) ^ hash(self.nofollow)
|
||||
|
||||
def __repr__(self):
|
||||
return '<Link url=%r text=%r >' % (self.url, self.text)
|
||||
return 'Link(url=%r, text=%r, nofollow=%r)' % (self.url, self.text, self.nofollow)
|
||||
|
||||
|
|
|
|||
|
|
@ -155,6 +155,7 @@ HTTPCACHE_STORAGE = 'scrapy.contrib.downloadermiddleware.httpcache.FilesystemCac
|
|||
HTTPCACHE_EXPIRATION_SECS = 0
|
||||
HTTPCACHE_IGNORE_HTTP_CODES = []
|
||||
HTTPCACHE_IGNORE_SCHEMES = ['file']
|
||||
HTTPCACHE_DBM_MODULE = 'anydbm'
|
||||
|
||||
ITEM_PROCESSOR = 'scrapy.contrib.pipeline.ItemPipelineManager'
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ See documentation in docs/topics/shell.rst
|
|||
import signal
|
||||
|
||||
from twisted.internet import reactor, threads
|
||||
from w3lib.url import any_to_uri
|
||||
|
||||
from scrapy.item import BaseItem
|
||||
from scrapy.spider import BaseSpider
|
||||
|
|
@ -14,7 +15,6 @@ from scrapy.selector import XPathSelector, XmlXPathSelector, HtmlXPathSelector
|
|||
from scrapy.utils.spider import create_spider_for_request
|
||||
from scrapy.utils.misc import load_object
|
||||
from scrapy.utils.response import open_in_browser
|
||||
from scrapy.utils.url import any_to_uri
|
||||
from scrapy.utils.console import start_python_console
|
||||
from scrapy.settings import Settings
|
||||
from scrapy.http import Request, Response, HtmlResponse, XmlResponse
|
||||
|
|
|
|||
|
|
@ -127,6 +127,18 @@ class CsvItemExporterTest(BaseItemExporterTest):
|
|||
ie.finish_exporting()
|
||||
self.assertEqual(output.getvalue(), '22,John\xc2\xa3\r\n')
|
||||
|
||||
def test_join_multivalue(self):
|
||||
class TestItem2(Item):
|
||||
name = Field()
|
||||
friends = Field()
|
||||
|
||||
i = TestItem2(name='John', friends=['Mary', 'Paul'])
|
||||
output = StringIO()
|
||||
ie = CsvItemExporter(output, include_headers_line=False)
|
||||
ie.start_exporting()
|
||||
ie.export_item(i)
|
||||
ie.finish_exporting()
|
||||
self.assertEqual(output.getvalue(), '"Mary,Paul",John\r\n')
|
||||
|
||||
class XmlItemExporterTest(BaseItemExporterTest):
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
import os, urlparse
|
||||
from cStringIO import StringIO
|
||||
|
||||
from zope.interface.verify import verifyObject
|
||||
from twisted.trial import unittest
|
||||
from twisted.internet import defer
|
||||
from cStringIO import StringIO
|
||||
from w3lib.url import path_to_file_uri
|
||||
|
||||
from scrapy.spider import BaseSpider
|
||||
from scrapy.contrib.feedexport import IFeedStorage, FileFeedStorage, FTPFeedStorage, S3FeedStorage, StdoutFeedStorage
|
||||
from scrapy.utils.url import path_to_file_uri
|
||||
from scrapy.utils.test import assert_aws_environ
|
||||
|
||||
class FeedStorageTest(unittest.TestCase):
|
||||
|
|
|
|||
|
|
@ -1,2 +0,0 @@
|
|||
import sys
|
||||
path = sys.modules[__name__].__path__[0]
|
||||
|
|
@ -1,190 +0,0 @@
|
|||
<head>
|
||||
|
||||
|
||||
|
||||
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
|
||||
<meta name="Copyright" content="Site Layout, Design & Content Copyright 2005 - retrosixty.co.uk">
|
||||
<meta http-equiv="content-language" content="EN">
|
||||
<meta name="Designer" content="Max Williams">
|
||||
<meta name="Keywords" content="retrosixty, retro sixty, retro, furniture, retro furniture, lighting ,retro lighting, art, retro art, ceramics, retro ceramics, technology, retro technology, fifties, sixties, seventies, 20th century design, post-war, post-war decorative, retro accessories">
|
||||
<meta name="Title" content="retrosixty - retrosixty.co.uk">
|
||||
<meta name="revisit-after" content="7">
|
||||
<meta name="Robots" content="index,follow">
|
||||
<meta name="Description" content="Dealers of retro furniture, post-war decorative and fine arts.">
|
||||
<meta http-equiv="Cache-Control" content="no-cache">
|
||||
<meta http-equiv="Expires" content="0">
|
||||
<meta name="Author" content="Nick Waters">
|
||||
|
||||
<title>retrosixty - Charlotte Perriand Infraphil lamp, c1960s for Philips, Netherlands</title>
|
||||
|
||||
<script language="JavaScript">
|
||||
<!--
|
||||
function FP_swapImg() {//v1.0
|
||||
var doc=document,args=arguments,elm,n; doc.$imgSwaps=new Array(); for(n=2; n<args.length;
|
||||
n+=2) { c=o.layers; if(elm) { doc.$imgSwaps[doc.$imgSwaps.length]=elm;
|
||||
elm.$src=elm.src; elm.src=args[n+1]; } }
|
||||
}
|
||||
|
||||
function FP_preloadImgs() {//v1.0
|
||||
var c=o.childNodes; if(!d.FP_imgs) d.FP_imgs=new Array();
|
||||
for(var d=document,a=arguments; i<a.length; i++) { d.FP_imgs[i]=new Image; d.FP_imgs[i].src=a[i]; }
|
||||
}
|
||||
|
||||
function FP_getObjectByID(id,o) {//v1.0
|
||||
var c,el,els,f,m,n; if(!o)o=document; if(o.getElementById) el=o.getElementById(id);
|
||||
else if(o.layers) el=o.all[id]; else if(o.all) el=FP_getObjectByID(id,c[n]); if(el) return el;
|
||||
if(o.id==id || o.name==id) return o; if(o.childNodes) el=FP_getObjectByID(id,els[n]); if(c)
|
||||
for(n=0; n<c.length; n++) { elm=FP_getObjectByID(args[n]); if(el) return el; }
|
||||
els=f[n].elements; if(f) for(n=0; n<f.length; n++) { f=o.forms;
|
||||
for(m=0; m<els.length; m++){ i=0; if(el) return el; } }
|
||||
return null;
|
||||
}
|
||||
// -->
|
||||
</script>
|
||||
|
||||
<style fprolloverstyle="">A:hover {color: #999999}
|
||||
span.patitre
|
||||
{}
|
||||
span.auctionblock
|
||||
{}
|
||||
</style>
|
||||
|
||||
<style id="mydeco-style" type="text/css">@import url(http://localhost:8000/as/site_media/clean.css);
|
||||
</style></head><body bottommargin="0" leftmargin="0" onload="" rightmargin="0" topmargin="0" alink="#000000" bgcolor="#c0c0c0" vlink="#000000" link="#000000">
|
||||
|
||||
<div class="mydeco-selected" align="center">
|
||||
<table id="table1" bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" width="765" height="100%">
|
||||
<tbody><tr>
|
||||
<td colspan="3" style="border-left: 1px solid rgb(0, 0, 0); border-right: 1px solid rgb(0, 0, 0);" align="center" height="120">
|
||||
<p align="center">
|
||||
<img alt="retrosixty" src="../images/logo.jpg" border="0" width="745" height="102"></p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-left: 1px solid rgb(0, 0, 0);" width="177" height="20">
|
||||
<p style="margin-left: 10px;">
|
||||
<img alt="retrosixty" src="../images/top.gif" border="0" width="160" height="20"></p></td>
|
||||
<td colspan="2" style="border-right: 1px solid rgb(0, 0, 0);" width="586" height="20">
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-left: 1px solid rgb(0, 0, 0);" background="../images/bg.gif" valign="top" width="180">
|
||||
<p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../index.html">
|
||||
<img alt="Home" fp-style="fp-btn: Linked Column 9; fp-font-style: Bold; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Home" id="img31" onmouseout="" onmouseover="" src="../buttons/button3.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../about.html">
|
||||
<img alt="About Us" fp-style="fp-btn: Linked Column 9; fp-font-style: Bold; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="About Us" id="img42" onmouseout="" onmouseover="" src="../buttons/button32.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../shipping.html">
|
||||
<img alt="Shipping" fp-style="fp-btn: Linked Column 9; fp-font-style: Bold; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Shipping" id="img43" onmouseout="" onmouseover="" src="../buttons/button34.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../links.html">
|
||||
<img alt="Links" fp-style="fp-btn: Linked Column 9; fp-font-style: Bold; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0; fp-orig: 0" fp-title="Links" id="img45" onmouseout="" onmouseover="" src="../buttons/button1.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../contact.php">
|
||||
<img alt="Contact" fp-style="fp-btn: Linked Column 9; fp-font-style: Bold; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Contact" id="img44" onmouseout="" onmouseover="" src="../buttons/button36.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
</p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../furniture.html">
|
||||
<img alt="Furniture" fp-style="fp-btn: Linked Column 9; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Furniture" id="img33" onmouseout="" onmouseover="" src="../buttons/buttonB.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../lighting.html">
|
||||
<img alt="Lighting" fp-style="fp-btn: Linked Column 9; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Lighting" id="img34" onmouseout="" onmouseover="" src="../buttons/buttonD.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../tech.html">
|
||||
<img alt="Technology" fp-style="fp-btn: Linked Column 9; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Technology" id="img35" onmouseout="" onmouseover="" src="../buttons/buttonF.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../ceramics.html">
|
||||
<img alt="Ceramics" fp-style="fp-btn: Linked Column 9; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Ceramics" id="img36" onmouseout="" onmouseover="" src="../buttons/button11.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../art.html">
|
||||
<img alt="Art" fp-style="fp-btn: Linked Column 9; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Art" id="img37" onmouseout="" onmouseover="" src="../buttons/button13.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../misc.html">
|
||||
<img alt="Misc. Items" fp-style="fp-btn: Linked Column 9; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0" fp-title="Misc. Items" id="img38" onmouseout="" onmouseover="" src="../buttons/button15.jpg" border="0" width="125" height="31"></a></p><p style="margin-top: 0pt; margin-bottom: 0pt;" align="center">
|
||||
<a href="../contemp.html">
|
||||
<img alt="Contemporary" fp-style="fp-btn: Linked Column 9; fp-img-press: 0; fp-bgcolor: #7B7B7B; fp-proportional: 0; fp-orig: 0" fp-title="Contemporary" id="img46" onmouseout="" onmouseover="" src="../buttons/button17.jpg" border="0" width="125" height="31"></a></p></td>
|
||||
<td class="" valign="top" width="433">
|
||||
<p style="margin-left: 10px; margin-right: 20px;">
|
||||
<span style="font-weight: 700;"><font id="anonymous_element_1" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "name"}}" size="5" face="Tahoma">
|
||||
Lighting..</font></span></p>
|
||||
<p style="margin-left: 10px; margin-right: 20px; margin-bottom: 15px;" align="justify">
|
||||
<font class="" size="2" face="Tahoma">Please click the thumbnails for larger
|
||||
images and the back button to return to the Lighting index.</font></p><div class="" align="center">
|
||||
<table id="table2" border="0" cellpadding="0" cellspacing="0" width="400" height="309">
|
||||
<tbody><tr>
|
||||
<td style="border-top: 1px solid rgb(123, 123, 123); border-bottom: 1px solid rgb(123, 123, 123);" width="130" height="309">
|
||||
<p align="center">
|
||||
</p><p class="" align="center">
|
||||
<a href="../photos/0642-01.JPG" target="_blank">
|
||||
<img id="anonymous_element_2" data-scrapy-annotate="{"variant": 0, "annotations": {"src": "image_urls"}}" src="../photos/0642-01_small.jpg" border="1"></a>
|
||||
</p><p align="center">
|
||||
<a href="../photos/0642-02.JPG" target="_blank">
|
||||
<img src="../photos/0642-02_small.jpg" border="1"></a></p><p align="center">
|
||||
<a href="../photos/0642-03.JPG" target="_blank">
|
||||
<img src="../photos/0642-03_small.jpg" border="1"></a></p><p align="center">
|
||||
<a href="../photos/0642-04.JPG" target="_blank">
|
||||
<img src="../photos/0642-04_small.jpg" border="1"></a></p><p align="center">
|
||||
</p><p align="center">
|
||||
</p><p align="center">
|
||||
</p></td>
|
||||
<td class="" style="border-top: 1px solid rgb(123, 123, 123); border-bottom: 1px solid rgb(123, 123, 123);" align="left" valign="top" height="309">
|
||||
<p style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
<b><font size="2" face="Tahoma">Designer</font></b><b><font size="2" face="Tahoma">:
|
||||
</font>
|
||||
</b>
|
||||
<font size="2" face="Tahoma,sans-serif">Charlotte Perriand</font><span style="font-size: 10pt; font-family: Tahoma,sans-serif;"> </span></p><p style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
<font size="2" face="Tahoma"><b>Manufacturer: </b></font>
|
||||
<font size="2"><span style="font-family: Tahoma,sans-serif;">
|
||||
Philips, Netherlands</span></font><span style="font-size: 10pt; font-family: Tahoma,sans-serif;">
|
||||
</span></p><p style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;"><font size="2" face="Tahoma"><b>
|
||||
Description:
|
||||
|
||||
</b></font>
|
||||
<span style="font-size: 10pt; font-family: Tahoma,sans-serif;">
|
||||
A Perriand designed 'infraphil' infrared heat lamp
|
||||
designed in c1960s. This example is in good vintage
|
||||
condition with some minor wear as one would expect.
|
||||
Original Philips sticker intact, although it has some
|
||||
wear as pictured. </span>
|
||||
</p><p class="" style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
<span class="" style="font-size: 10pt; font-family: Tahoma,sans-serif;">
|
||||
As with all electrical items we always
|
||||
recommend having them tested by a professional prior to
|
||||
use although it is in full working order. The lamp can
|
||||
be used as a table lamp, or mounted on the wall - full
|
||||
adjustable</span><font size="2" face="Tahoma">...</font></p><p class="" style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
<font id="anonymous_element_3" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}" size="2" face="Tahoma"><b>Price:</b> £60</font></p><p style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
<font size="2" face="Tahoma"><b>Size:</b> </font>
|
||||
<font size="2"><span style="font-family: Tahoma,sans-serif;">
|
||||
N/A</span></font><span style="font-size: 10pt; font-family: Tahoma,sans-serif;">
|
||||
<span class="auctionblock"> </span>
|
||||
<span class="auctionblock"> </span> </span></p><p style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
<font size="2" face="Tahoma"><b>Shipping:</b> </font>
|
||||
<span style="font-size: 10pt; font-family: Tahoma,sans-serif;">
|
||||
£7 to mainland UK</span><font size="2" face="Tahoma">.
|
||||
Please enquire for other locations.</font></p><p style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
<font size="2" face="Tahoma"><b>Ref #:</b> 0642</font></p><p style="margin-left: 15px; margin-top: 25px; margin-bottom: -10px;">
|
||||
</p></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
|
||||
|
||||
<p style="margin-left: 25px; margin-top: 25px;">
|
||||
<font size="2" face="Tahoma">
|
||||
<a href="about:blank" style="text-decoration: none;"><b>
|
||||
<< </b>BACK</a></font></p></div>
|
||||
</td>
|
||||
<td class="" style="border-right: 1px solid rgb(0, 0, 0);" valign="top" width="153">
|
||||
<p style="margin-right: 20px;" align="left">
|
||||
<img class="" alt="retrosixty" src="../images/icon1.jpg" border="0" width="133" height="133"></p>
|
||||
<p style="margin-right: 20px;">
|
||||
<img class="" alt="retrosixty" src="../images/icon2.jpg" border="0" width="133" height="133"></p>
|
||||
<p style="margin-right: 20px;">
|
||||
<img class="" alt="retrosixty" src="../images/icon3.jpg" border="0" width="133" height="133"></p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="border-left: 1px solid rgb(0, 0, 0);" width="177" height="25">
|
||||
<p style="margin-left: 10px; margin-bottom: 10px;">
|
||||
<img alt="retrosixty" src="../images/bottom.gif" border="0" width="160" height="25"></p></td>
|
||||
<td colspan="2" style="border-right: 1px solid rgb(0, 0, 0);" width="586" height="25">
|
||||
<p style="margin-right: 15px;" align="right">
|
||||
<font style="font-size: 8pt;" face="Tahoma">Site Layout, Design &
|
||||
Content Copyright 2006-09 - retrosixty.co.uk</font></p></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
</div>
|
||||
|
||||
</body>
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
|
|
@ -1,632 +0,0 @@
|
|||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||||
<html dir="ltr" lang="en" xmlns="http://www.w3.org/1999/xhtml">
|
||||
<head>
|
||||
<title>TEMPUR Deluxe-HD™ Mattress | Tempur</title>
|
||||
<meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type" />
|
||||
<meta content="TEMPUR Deluxe-HD™ Mattress Mattresses Pillows Small Products " name="keywords" />
|
||||
<meta content="Tempur TEMPUR Deluxe-HD™ Mattress - TEMPUR Deluxe-HD™ Mattress Product Overview The TEMPUR Deluxe-HD™ Mattress combines the unique pressure relieving qualities of TEMPUR, with extra TEMPUR-HD™ soft-touch quilted into the cover, for a luxurious feel that is unparalleled in the bedroom. It not only looks luxurious, but also offers " name="description" />
|
||||
<meta content="no" http-equiv="imagetoolbar" />
|
||||
<meta content="-1" http-equiv="Expires" />
|
||||
<meta content="webmaster@tempur.co.uk" http-equiv="reply-to" />
|
||||
<meta content="document" name="resource-type" />
|
||||
<meta content="30" name="revisit-after" />
|
||||
<meta content="TRUE" name="MSSmartTagsPreventParsing" />
|
||||
<meta content="Consumer Products/Furnishings;Consumer Products/Health" name="classification" />
|
||||
<meta content="INDEX,FOLLOW" name="ROBOTS" />
|
||||
<meta content="Global" name="distribution" />
|
||||
<meta content="Safe For Kids" name="rating" />
|
||||
<meta content="2008 Tempur-Pedic, Inc." name="copyright" />
|
||||
<meta content="Tempur UK" name="author" />
|
||||
<meta content="English" name="language" />
|
||||
<meta content="Web Page" name="doc-type" />
|
||||
<meta content="Completed" name="doc-class" />
|
||||
<meta content="Copywritten Work" name="doc-rights" />
|
||||
<meta content="6QdsKgcJMwvOOi7d0aPp99A9efMsYWnWtiD9+wwrrW4=" name="verify-v1" />
|
||||
<link href="/tempurUK/includes/css/inc.css.site_styles.css" rel="stylesheet" type="text/css" />
|
||||
|
||||
<script src="/tempurUK/includes/js/milonic/milonic_src.js" type="text/javascript"></script>
|
||||
<style>.milonic{width:1px;visibility:hidden;position:absolute}</style>
|
||||
<script type="text/javascript">
|
||||
<!--
|
||||
if (ns4) {
|
||||
_d.write("<script language=JavaScript src='/tempurUK/includes/js/milonic/mmenuns4.js'>\/script>");
|
||||
} else {
|
||||
_d.write("<script language=JavaScript src='/tempurUK/includes/js/milonic/mmenudom.js'>\/script>");
|
||||
}
|
||||
-->
|
||||
</script>
|
||||
<script language="JavaScript" src="/tempurUK/includes/js/milonic/mmenudom.js"></script>
|
||||
<script>
|
||||
<!--
|
||||
function $9(ap) {return _f}
|
||||
// --></script>
|
||||
|
||||
<base href="http://www.tempur.co.uk/" />
|
||||
|
||||
<link href="includes/templates/tempur/css/stylesheet.css" rel="stylesheet" type="text/css" />
|
||||
<script src="includes/templates/template_default/jscript/jscript_popup_ezpage.js" type="text/javascript"></script>
|
||||
<script src="includes/modules/pages/product_info/jscript_textarea_counter.js" type="text/javascript"></script>
|
||||
<script language="JavaScript" type="text/javascript">
|
||||
/*
|
||||
Milonic DHTML Menu - JavaScript Website Navigation System.
|
||||
Copyright 2004 (c) Milonic Solutions Limited. All Rights Reserved.
|
||||
Version 5+ Data File structure is the property of Milonic Solutions Ltd and must only be used in Milonic DHTML Products
|
||||
This is a commercial software product, please visit http://www.milonic.com/ for more information.
|
||||
See http://www.milonic.com/license.php for Commercial License Agreement
|
||||
All Copyright statements must always remain in place in all files at all times
|
||||
******* PLEASE NOTE: THIS IS NOT FREE SOFTWARE, IT MUST BE LICENSED FOR ALL USE *******
|
||||
|
||||
Configured by GDL & Associates on 20041212
|
||||
|
||||
*/
|
||||
|
||||
_menuCloseDelay=500 // The time delay for menus to remain visible on mouse out
|
||||
_menuOpenDelay=150 // The time delay before menus open on mouse over
|
||||
_subOffsetTop=10 // Sub menu top offset
|
||||
_subOffsetLeft=-10 // Sub menu left offset
|
||||
|
||||
var linkfront= "/tempuruk"
|
||||
var slinkfront="/tempuruk"
|
||||
var secureLink = "https://secure.tempurpedic.com/tempuruk"
|
||||
|
||||
// main-menu styles
|
||||
with(mainStyle=new mm_style()){
|
||||
borderwidth=0;
|
||||
}
|
||||
// sub-menu styles
|
||||
with(subStyle=new mm_style()){
|
||||
onbgcolor="#7e97a3";
|
||||
oncolor="#f2edd1";
|
||||
offbgcolor="#F2EDD1";
|
||||
offcolor="#7e97a3";
|
||||
bordercolor="#E2DFB7";
|
||||
borderstyle="solid";
|
||||
borderwidth=1;
|
||||
separatorcolor="#FFFFFF";
|
||||
separatorsize="1";
|
||||
padding=4;
|
||||
fontsize="90%";
|
||||
fontstyle="normal";
|
||||
fontfamily=" Arial, Helvetica, sans-serif";
|
||||
pagecolor="B85212";
|
||||
pagebgcolor="#EEEFE7";
|
||||
headercolor="#756C5A";
|
||||
headerbgcolor="#ffffff";
|
||||
subimage="/tempurUK/images/milonic_arrow.gif";
|
||||
subimagepadding="3";
|
||||
//overfilter="Fade(duration=0.2);Alpha(opacity=90);Shadow(color='#777777', Direction=135, Strength=5)";
|
||||
//outfilter="randomdissolve(duration=0.2)";
|
||||
itemheight=15;
|
||||
}
|
||||
|
||||
|
||||
// Note: Main menu is defined in each document's <body>, rather than here.
|
||||
// This is done in order to use relative positioning, as the home page
|
||||
// in particular, will need to place the menu in a different position than
|
||||
// the rest of the pages on the site. So, instead, the menu is positioned
|
||||
// inside of an HTML table. -rory
|
||||
|
||||
// Company Menu
|
||||
with(milonic=new menuname("Company")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=NASA Space Technology;showmenu=NASA;url=" + linkfront + "/company/nasa/;");
|
||||
aI("text=The TEMPUR History;url=" + linkfront + "/company/history/;");
|
||||
aI("text=Endorsements;url=" + linkfront + "/company/endorsements/;");
|
||||
aI("text=Guarantee;url=" + linkfront + "/warranty/guarantee/;");
|
||||
aI("text=TEMPUR Med;url=http://www.tempurmed.co.uk/page3736.asp;target=windowname;targetfeatures=width=900,height=500");
|
||||
aI("text=Hotels;url=" + linkfront + "/hotels/;");
|
||||
aI("text=FAQs;url=" + linkfront + "/faq/;");
|
||||
aI("text=Contact Us;url=" + linkfront + "/company/contactus/;");
|
||||
aI("text=Press Room;url=" + linkfront + "/company/pressroom/;");
|
||||
}
|
||||
|
||||
// NASA Menu
|
||||
with(milonic=new menuname("NASA")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=Recognised By NASA;url=" + linkfront + "/company/nasa/recognition/;");
|
||||
aI("text=Certificate of Achievement;url=" + linkfront + "/company/nasa/certificate/;");
|
||||
}
|
||||
|
||||
// Endorsements Menu
|
||||
with(milonic=new menuname("Endorsements")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=Consumer Endorsement;url=" + linkfront + "/company/endorsements/ConsumerEndorsement/;");
|
||||
aI("text=Consumer Surveys;url=" + linkfront + "/company/endorsements/ConsumerSurveys/;");
|
||||
aI("text=Medical Endorsements;url=" + linkfront + "/company/endorsements/MedicalEndorsements/;");
|
||||
}
|
||||
|
||||
// Contact Us Menu
|
||||
with(milonic=new menuname("Contact_us")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=Terms & Conditions Online Sale;url=" + linkfront + "/material/TermsConditions/;");
|
||||
aI("text=Terms & Conditions Web Site;url=" + linkfront + "/material/TermsConditions/websitetermsconditions/;");
|
||||
aI("text=Terms & Conditions 60-Night Trial;url=" + linkfront + "/company/contactus/60NightTrial/;");
|
||||
}
|
||||
|
||||
// Material Menu
|
||||
with(milonic=new menuname("Material")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=60 Night Trial;url=" + linkfront + "/60night/;");
|
||||
aI("text=Terms & Conditions;url=" + linkfront + "/material/TermsConditions/;");
|
||||
aI("text=Free Information Pack;url=" + linkfront + "/freeinfo/;");
|
||||
aI("text=Developed for Space;url=" + linkfront + "/material/nasa/;");
|
||||
aI("text=A Comfort Revolution;url=" + linkfront + "/material/comfortrevolution/;");
|
||||
aI("text=TEMPUR Improves Sleep Quality;url=" + linkfront + "/material/sleepquality/;");
|
||||
aI("text=Relieves & Improves Back Pain;url=" + linkfront + "/material/backpain/;");
|
||||
aI("text=Used In Healthcare;url=" + linkfront + "/material/healthcare/;");
|
||||
}
|
||||
|
||||
// Mattresses Menu
|
||||
with(milonic=new menuname("Mattresses")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=TEMPUR Combi Mattress;url=http://www.tempur.co.uk/tempuruk/mattresses/combi/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Combi-HD™ Mattress - 20cm Depth (8 Inch);url=http://www.tempur.co.uk/tempuruk/mattresses/combihd/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Combi-HD™ Mattress - 25cm Depth (10 Inch);url=http://www.tempur.co.uk/tempuruk/mattresses/combihd/25cm/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Deluxe-HD™ Mattress;url=http://www.tempur.co.uk/tempuruk/mattresses/deluxe/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Overlay Mattress;url=http://www.tempur.co.uk/tempuruk/mattresses/overlay/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
}
|
||||
|
||||
// Pillows Menu
|
||||
with(milonic=new menuname("Pillows")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=TEMPUR Original Pillow;url=http://www.tempur.co.uk/tempuruk/pillows/original/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Classic Pillow;url=http://www.tempur.co.uk/tempuruk/pillows/classicpillow/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Traditional Pillow;url=http://www.tempur.co.uk/tempuruk/pillows/traditional/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Millennium Pillow;url=http://www.tempur.co.uk/tempuruk/pillows/millenniumpillow/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR DeLuxe Pillow;url=http://www.tempur.co.uk/tempuruk/pillows/deluxepillow/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
}
|
||||
|
||||
// Small Products Menu
|
||||
with(milonic=new menuname("SmallProducts")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=Small Products;showmenu=SmallProductsSmallProducts;url=http://www.tempur.co.uk/tempuruk/smallproducts/smallproducts/?cPath=4_5&zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=Travel Products;showmenu=SmallProductsTravelProducts;url=http://www.tempur.co.uk/tempuruk/comfort/travel/?cPath=4_6&zenid=ac101e1c434adca39237334777e19b88");
|
||||
}
|
||||
|
||||
// Small Products Menu
|
||||
with(milonic=new menuname("SmallProductsSmallProducts")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=TEMPUR Seat Cushion;url=http://www.tempur.co.uk/tempuruk/comfort/comfortcushion/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Lumbar Support;url=http://www.tempur.co.uk/tempuruk/comfort/lumbarsupport/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR PC Seat Wedge;url=http://www.tempur.co.uk/tempuruk/comfort/seatwedge/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
}
|
||||
|
||||
// Travel Products Menu
|
||||
with(milonic=new menuname("SmallProductsTravelProducts")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=TEMPUR Travel Set;url=http://www.tempur.co.uk/tempuruk/comfort/travelset/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Travel Pillow;url=http://www.tempur.co.uk/tempuruk/comfort/travelneckpillow/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Transit Lumbar Support;url=http://www.tempur.co.uk/tempuruk/comfort/transitlumbar/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Transit Pillow;url=http://www.tempur.co.uk/tempuruk/comfort/transitpillow/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("text=TEMPUR Traditional Travel Pillow;url=http://www.tempur.co.uk/tempuruk/comfort/traditionaltravelpillow/?zenid=ac101e1c434adca39237334777e19b88");
|
||||
}
|
||||
|
||||
|
||||
// Beds Menu
|
||||
with(milonic=new menuname("Beds")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=Milano;url=" + linkfront + "/beds/milano/;");
|
||||
aI("text=Toscana;url=" + linkfront + "/beds/toscana/;");
|
||||
aI("text=Verona;url=" + linkfront + "/beds/verona/;");
|
||||
aI("text=Accessories;showmenu=BedAccessories;");
|
||||
}
|
||||
|
||||
// Bed Accessories Menu
|
||||
with(milonic=new menuname("BedAccessories")){
|
||||
style=subStyle;
|
||||
overflow="scroll";
|
||||
aI("text=TEMPUR Headboard Collection;url=" + linkfront + "/beds/accessories/headboards/;");
|
||||
aI("text=Remote Controls;url=" + linkfront + "/beds/accessories/remotecontrol/;");
|
||||
}
|
||||
|
||||
</script>
|
||||
<script language="javascript" type="text/javascript"><!--
|
||||
function popupWindow(url) {
|
||||
window.open(url,'popupWindow','toolbar=no,location=no,directories=no,status=no,menubar=no,scrollbars=no,resizable=yes,copyhistory=no,width=100,height=100,screenX=150,screenY=150,top=150,left=150')
|
||||
}
|
||||
function popupWindowPrice(url) {
|
||||
window.open(url,'popupWindow','toolbar=no,location=no,directories=no,status=no,menubar=no,scrollbars=yes,resizable=yes,copyhistory=no,width=600,height=400,screenX=150,screenY=150,top=150,left=150')
|
||||
}
|
||||
//--></script>
|
||||
</head>
|
||||
<body id="productinfoBody">
|
||||
<!-- ClickTale Top part -->
|
||||
<script type="text/javascript">
|
||||
var WRInitTime=(new Date()).getTime();
|
||||
</script>
|
||||
<!-- ClickTale end of Top part -->
|
||||
|
||||
|
||||
|
||||
<!--<div id="mainWrapper">-->
|
||||
|
||||
|
||||
<!--bof-header logo and navigation display-->
|
||||
|
||||
<table align="center" bgcolor="#f2edd1" border="1" bordercolor="#ac9d6a" cellpadding="0" cellspacing="0" width="840">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td valign="top">
|
||||
<!-- Begin Top Bar -->
|
||||
<script>
|
||||
drawMenus();
|
||||
</script>
|
||||
|
||||
<table align="center" bgcolor="#f2edd1" border="1" bordercolor="#ac9d6a" cellpadding="0" cellspacing="0" width="840">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<table align="center" bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" height="170" width="840">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="/tempuruk/mattresses/"><img alt="Order online now or free phone 08000 111 083" border="0" height="111" src="/tempurUK/images/frontpage/top_header.gif" width="560"></a></td>
|
||||
<td align="right" valign="center"><img src="/tempurUK/images/top/top.gif"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td colspan="2">
|
||||
<script language="javascript">
|
||||
<!-- // Main Menu
|
||||
with (milonic=new menuname("Main Menu")) {
|
||||
style=mainStyle;
|
||||
top="offset=50";
|
||||
alwaysvisible=1;
|
||||
orientation="horizontal";
|
||||
position="relative";
|
||||
|
||||
aI("image=/tempurUK/images/top/home.gif;overimage=/tempurUK/images/top/home.gif;url=/");
|
||||
aI("image=/tempurUK/images/top/ourcompany.gif;overimage=/tempurUK/images/top/ourcompany.gif;showmenu=Company;url=/tempuruk/company/");
|
||||
aI("image=/tempurUK/images/top/material.gif;overimage=/tempurUK/images/top/material.gif;showmenu=Material;url=/tempuruk/material/");
|
||||
aI("image=/tempurUK/images/top/mattresses.gif;overimage=/tempurUK/images/top/mattresses.gif;showmenu=Mattresses;url=/tempuruk/mattresses/");
|
||||
aI("image=/tempurUK/images/top/pillows.gif;overimage=/tempurUK/images/top/pillows.gif;showmenu=Pillows;url=/tempuruk/pillows/");
|
||||
aI("image=/tempurUK/images/top/beds.gif;overimage=/tempurUK/images/top/beds.gif;showmenu=Beds;url=/tempuruk/beds/");
|
||||
aI("image=/tempurUK/images/top/small.gif;overimage=/tempurUK/images/top/small.gif;showmenu=SmallProducts;url=/tempuruk/smallproducts/");
|
||||
aI("image=/tempurUK/images/top/clearance.gif;overimage=/tempurUK/images/top/clearance.gif;url=/tempuruk/clearance/");
|
||||
aI("image=/tempurUK/images/top/myaccount.gif;overimage=/tempurUK/images/top/myaccount.gif;url=https://www.tempur.co.uk/index.php?main_page=login&zenid=ac101e1c434adca39237334777e19b88");
|
||||
aI("image=/tempurUK/images/top/end.gif;overimage=/tempurUK/images/top/end.gif");
|
||||
}
|
||||
|
||||
drawMenus();
|
||||
// -->
|
||||
</script>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<table align="center" border="0" cellpadding="0" cellspacing="0" width="840">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>
|
||||
|
||||
<table border="0" cellpadding="0" cellspacing="0" id="contentMainWrapper" width="100%">
|
||||
<tr>
|
||||
|
||||
<td class="columnLeft" id="navColumnOne" style="width: 131px">
|
||||
<div id="navColumnOneWrapper" style="width: 131px">
|
||||
<div style="margin-bottom: 1em"><a href="/tempuruk/mattresses/"><img border="0" src="/tempurUK/images/frontpage/content_offer_gif.gif"></a></div>
|
||||
|
||||
<h3 class="leftBoxHeading" id="informationHeading">Free Information</h3>
|
||||
<div class="sideBoxContent">To get your Free Information Pack <a href="/tempuruk/freeinfo/">click here</a></div>
|
||||
|
||||
|
||||
<h3 class="leftBoxHeading TestimonialBoxHeading" id="testimonials_heading">Testimonials</h3>
|
||||
<div class="sideBoxContent TestimonialBoxContent">My husband and I just spent our first night on our new TEMPUR Mattress, and we are thrilled at the terrific night's rest we both had.</div>
|
||||
|
||||
<div style="margin-top: 0.8em; margin-bottom: 2em;"><a href="/tempuruk/material/"><img border="0" height="117" src="/tempurUK/images/AuthenticTempurMaterial.gif" width="117"></a></div>
|
||||
</div></td>
|
||||
<td valign="top">
|
||||
<!-- bof breadcrumb -->
|
||||
<div id="navBreadCrumb"> <a href="http://www.tempur.co.uk/">Home</a> <span>></span>
|
||||
<a href="http://www.tempur.co.uk/tempuruk/mattresses/?zenid=ac101e1c434adca39237334777e19b88">Mattresses</a> <span>></span>
|
||||
TEMPUR Deluxe-HD™ Mattress
|
||||
</div>
|
||||
<!-- eof breadcrumb -->
|
||||
|
||||
|
||||
<!-- bof upload alerts -->
|
||||
<!-- eof upload alerts -->
|
||||
|
||||
<div class="centerColumn" id="productGeneral">
|
||||
|
||||
<!--bof Form start-->
|
||||
<form action="http://www.tempur.co.uk/tempuruk/mattresses/deluxe/?&number_of_uploads=0&action=add_product&zenid=ac101e1c434adca39237334777e19b88" enctype="multipart/form-data" method="post" name="cart_quantity">
|
||||
<!--eof Form start-->
|
||||
|
||||
|
||||
<div class="productGeneral biggerText" id="productDescription">
|
||||
|
||||
<table align="center" border="0" width="100%">
|
||||
<tbody>
|
||||
|
||||
<tr>
|
||||
<td>
|
||||
<p align="left" class="bodyText"><span class="titleText" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "name"}}" >TEMPUR Deluxe-HD™ Mattress </span><br></p>
|
||||
<table border="0" width="100%">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td width="221">
|
||||
<div align="left"><img border="1" data-scrapy-annotate="{"variant": 0, "annotations": {"src": "image_urls"}}" src="/tempurUK/images/NR/rdonlyres/1CB9C3E7-3FE9-4158-B66C-A6494B845213/391/YR2Y4795_181W.jpg"></div></td>
|
||||
<td width="222">
|
||||
<div align="left">
|
||||
<div align="left"><a href="javascript:popupWindow('index.php?main_page=popup_image_path&path=tempurUK/images/popupwindow/mattresses/DeLuxeMattress_464W.jpg&text=DeLuxe');"><img border="0" data-scrapy-annotate="{"variant": 0, "annotations": {"src": "image_urls"}}" src="/tempurUK/images/NR/rdonlyres/1CB9C3E7-3FE9-4158-B66C-A6494B845213/392/deluxe_clickto_enlarge_181W.jpg">
|
||||
<div></div></a></div></div></td></tr></tbody></table>
|
||||
<table border="0" width="100%">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}" valign="top" width="100%">
|
||||
|
||||
<p align="left" class="bodyText"> </p>
|
||||
<p align="left" class="subTitleText">Product Overview </p>
|
||||
<p align="left" class="bodyText">The TEMPUR Deluxe-HD™ Mattress combines the unique pressure relieving qualities of TEMPUR, with extra TEMPUR-HD™ soft-touch quilted into the cover, for a luxurious feel that is unparalleled in the bedroom. It not only looks luxurious, but also offers enhanced comfort. </p>
|
||||
<p align="left" class="bodyText">This 22cm mattress is constructed differently to the TEMPUR Combi-HD™ Mattress. The TEMPUR Deluxe-HD™ Mattress has a quilted velour cover with a 2cm high density "soft-touch" TEMPUR Material embedded within it. Underneath the quilted cover lies a 9cm layer of TEMPUR Material, on top of 11cm of conventional polyurethane foam.</p>
|
||||
<p align="left" class="bodyText">The Deluxe-HD™ Mattress features the new TEMPUR-Tex™ Cover with in-built humidity control. The TEMPUR-Tex™ <span class="bodyText">material allows any moisture to evaporate faster from the surface of the mattress, thus providing the consumer with a drier sleeping experience. </span></p>
|
||||
<p align="left" class="subTitleText">Product Specification</p>
|
||||
<table border="0" width="100%">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td height="92" valign="top" width="39"> <img border="0" src="/tempurUK/images/NR/rdonlyres/1CB9C3E7-3FE9-4158-B66C-A6494B845213/390/deluxe_breakdown_160W2.jpg"></td>
|
||||
|
||||
<td class="bodyText" width="100%">
|
||||
<ul>
|
||||
<li>A. Quilted Cover with 2cm of HD "soft-<div align="left"> touch" TEMPUR embedded within it. </div>
|
||||
</li><li>B. 9cm TEMPUR visco-elastic temperature
|
||||
<div align="left"> sensitive material</div>
|
||||
</li><li>C. 11cm high resilient polyurethane foam
|
||||
</li><li>15 year limited guarantee
|
||||
</li><li class="bodyText">Works in perfect partnership with the TEMPUR bed range</li></ul></td></tr></tbody></table>
|
||||
<p align="left" class="bodyText" style="margin: 0cm 0cm 0pt;" style1="">
|
||||
</p><p class="subTitleText">When you purchase a TEMPUR Mattress online you will automatically receive the 60-night trial. Please note only one mattress can be trialled per household.</p></td></tr>
|
||||
|
||||
<tr>
|
||||
<td> </td></tr>
|
||||
<tr>
|
||||
<td class="bodyText">
|
||||
<p>Please refer to our most <a class="boldBodyText" href="javascript:PopUpEZPageWindow('/tempuruk/warranty/60NightFAQ?ezpopup=1', 600, 500);"><u>Frequently Asked Questions</u></a> to ensure that you know all the facts about our 60-night trial offer.</p>
|
||||
<p class="boldBodyText"><a href="javascript:PopUpEZPageWindow('/tempuruk/warranty/genuinetempur?ezpopup=1', 400, 450);"><u>Looking to purchase TEMPUR elsewhere?</u></a></p></td></tr></tbody></table></td></tr></tbody></table>
|
||||
|
||||
<table border="0" cellpadding="0" cellspacing="0">
|
||||
<tbody onload="MM_preloadImages('/tempurUK/images/addtobag_on.gif')">
|
||||
<tr>
|
||||
<td background="/tempurUK/images/RTB/readytobuy_left.gif" height="25" valign="top"><img src="/tempurUK/images/RTB/readytobuy_top_left.gif"></td>
|
||||
<td align="right" height="25" style="background: #F9F6EF url(/tempurUK/images/RTB/readytobuy_top_1.gif) top left repeat-x;" valign="top" width="100%">
|
||||
<img src="/tempurUK/images/RTB/readytobuy_top_middle.gif">
|
||||
</td>
|
||||
<td background="/tempurUK/images/RTB/readytobuy_right.gif" height="25" valign="top" width="28"><img src="/tempurUK/images/RTB/readytobuy_top_right.gif"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td background="/tempurUK/images/RTB/readytobuy_left.gif"><img src="/tempurUK/images/RTB/readytobuy_left.gif"></td>
|
||||
<td align="center" style="background: #F9F6EF">
|
||||
<table border="0" cellpadding="0" cellspacing="0" width="100%">
|
||||
<tbody><tr>
|
||||
<td align="center" class="StartingAtOnly" height="25" valign="middle"><span data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}" id="Addtocart1_lpriceLBL">From £1,049.00</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" class="DropDownText" height="25" id="ready_to_buy_options" valign="middle"><select id="attrib-4" name="id[4]">
|
||||
<option selected="selected" value="22">Tempur Deluxe-HD Mattress 2'6" x 6'6" (75 x 200 x 22 cm) ( £1,049.00 )</option>
|
||||
<option value="23">Tempur Deluxe-HD Mattress 3' x 6'3" (90 x 190 x 22 cm) ( £1,149.00 )</option>
|
||||
<option value="24">Tempur Deluxe-HD Mattress 3' x 6'6" (90 x 200 x 22 cm) ( £1,249.00 )</option>
|
||||
<option value="25">Tempur Deluxe-HD Mattress 4'6" x 6'3" (135 x 190 x 22 cm) ( £1,898.99 )</option>
|
||||
<option value="26">Tempur Deluxe-HD Mattress 5' x 6'6" (150 x 200 x 22 cm) ( £2,099.00 )</option>
|
||||
<option value="27">Tempur Deluxe-HD Mattress 5'3 x 6'6" (160 x 200 x 22 cm) ( £2,149.00 )</option>
|
||||
<option value="28">Tempur Deluxe-HD Mattress 6' x 6'6" (180 x 200 x 22 cm) ( £2,199.00 )</option>
|
||||
</select>
|
||||
|
||||
</td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
</td>
|
||||
<td background="/tempurUK/images/RTB/readytobuy_right.gif"><img src="/tempurUK/images/RTB/readytobuy_right.gif"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td background="/tempurUK/images/RTB/readytobuy_left.gif" height="45" valign="bottom"><img src="/tempurUK/images/RTB/readytobuy_btm_left.gif"></td>
|
||||
<td align="right" style="background: #F9F6EF url(/tempurUK/images/RTB/readytobuy_btm_1.gif) bottom left repeat-x; padding-bottom: 19px" valign="bottom" width="100%"><input name="cart_quantity" type="hidden" value="1" /><input name="products_id" type="hidden" value="4" /><input alt="Add to Cart" src="includes/templates/tempur/buttons/english/button_in_cart.gif" title=" Add to Cart " type="image" /></td>
|
||||
<td background="/tempurUK/images/RTB/readytobuy_right.gif" valign="bottom" width="28"><img src="/tempurUK/images/RTB/readytobuy_btm_right.gif"></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<table cellspacing="0" width="100%">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><span class="boldBodyText"><a href="/tempuruk/checkout/vatexemption/" style="text-decoration:underline">You may be eligible for VAT relief</a></span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" width="100%">
|
||||
<table border="0" id="Dimensions1_tblDim">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td align="left"><p class="subTitleText" id="Dimensions1_Label2">Dimensions</p></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" class="bodytext" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "features"}}" >
|
||||
<table border="1" bordercolor="#7e97a3" cellspacing="0" class="DimensionsTable" id="Dimensions1_dgDimensions" rules="all" width="100%">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="subTitleText" width="30%">Size (Inches)</td><td class="subTitleText" width="40%">Size (Centimetres*)</td><td class="subTitleText" width="30%">Size</td>
|
||||
</tr><tr>
|
||||
<td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">3' x 6'3"</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">90 x 190 x 22 cm</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">Single (Standard)</font></td>
|
||||
</tr><tr>
|
||||
|
||||
<td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">3' x 6'6"</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">90 x 200 x 22 cm</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">Single (Long)</font></td>
|
||||
</tr><tr>
|
||||
<td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">4'6" x 6'3"</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">135 x 190 x 22 cm</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">Double</font></td>
|
||||
</tr><tr>
|
||||
<td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">5' x 6'6"</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">150 x 200 x 22 cm</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">King</font></td>
|
||||
|
||||
</tr><tr>
|
||||
<td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">5'3 x 6'6"</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">160 x 200 x 22 cm</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">Euro King</font></td>
|
||||
</tr><tr>
|
||||
<td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">6' x 6'6"</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">180 x 200 x 22 cm</font></td><td class="bodytext"><font face="Verdana,Arial,Helvetica,sans-serif">Super King</font></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center"><span id="Dimensions1_lblText"><p class="bodyText">*Please Note: Mattress sizes are approximate. Please allow for a 2cm tolerance.</p><p class="subTitleText">Can't find the size you are looking for?</p><p class="bodyText">Special Size Mattresses are available on request, Please contact our Direct Sales Team on <span class="titleText">08000 111 083</span> for further details.</p></span></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<!--bof Form close-->
|
||||
</form>
|
||||
<!--bof Form close-->
|
||||
</div>
|
||||
</td>
|
||||
|
||||
<td class="columnRight" id="navColumnTwo" style="width: 131px">
|
||||
<div id="navColumnTwoWrapper" style="width: 131px">
|
||||
<div class="rightBoxContainer" style="width: 131px">
|
||||
<h3 class="rightBoxHeading SectionMenuBoxHeading">Mattresses</h3>
|
||||
<div class="sideBoxContent SectionMenuBoxContent">
|
||||
<p class="menuTextOver">
|
||||
<a class="menuTextOver" href="http://www.tempur.co.uk/tempuruk/mattresses/combi/?zenid=ac101e1c434adca39237334777e19b88">
|
||||
TEMPUR Combi Mattress
|
||||
</a>
|
||||
</p>
|
||||
<p class="menuTextOver">
|
||||
<a class="menuTextOver" href="http://www.tempur.co.uk/tempuruk/mattresses/combihd/?zenid=ac101e1c434adca39237334777e19b88">
|
||||
TEMPUR Combi-HD™ Mattress - 20cm Depth (8 Inch)
|
||||
</a>
|
||||
</p>
|
||||
<p class="menuTextOver">
|
||||
<a class="menuTextOver" href="http://www.tempur.co.uk/tempuruk/mattresses/combihd/25cm/?zenid=ac101e1c434adca39237334777e19b88">
|
||||
TEMPUR Combi-HD™ Mattress - 25cm Depth (10 Inch)
|
||||
</a>
|
||||
</p>
|
||||
<p class="menuTextOver">
|
||||
<a class="menuTextOver" href="http://www.tempur.co.uk/tempuruk/mattresses/deluxe/?zenid=ac101e1c434adca39237334777e19b88">
|
||||
TEMPUR Deluxe-HD™ Mattress
|
||||
</a>
|
||||
</p>
|
||||
<p class="menuTextOver">
|
||||
<a class="menuTextOver" href="http://www.tempur.co.uk/tempuruk/mattresses/overlay/?zenid=ac101e1c434adca39237334777e19b88">
|
||||
TEMPUR Overlay Mattress
|
||||
</a>
|
||||
</p>
|
||||
</div>
|
||||
</div><!--// bof: shoppingcart //-->
|
||||
<div class="rightBoxContainer" id="shoppingcart" style="width: 131px">
|
||||
<h3 class="rightBoxHeading" id="shoppingcartHeading"><a href="http://www.tempur.co.uk/index.php?main_page=shopping_cart&zenid=ac101e1c434adca39237334777e19b88">Shopping Cart [more]</a></h3>
|
||||
<div class="sideBoxContent" id="shoppingcartContent"><div id="cartBoxEmpty">Your cart is empty.</div></div></div>
|
||||
<!--// eof: shoppingcart //-->
|
||||
|
||||
<!--// bof: protxdirectcardsaccepted //-->
|
||||
<div class="rightBoxContainer" id="protxdirectcardsaccepted" style="width: 131px">
|
||||
<h3 class="rightBoxHeading" id="protxdirectcardsacceptedHeading">Cards Accepted</h3>
|
||||
<div class="sideBoxContent centeredContent" id="protxdirectcardsacceptedContent">
|
||||
<img alt="Visa" class="ProtxDirectCardsAcceptedSideboxCardIcon" height="25" src="includes/templates/template_default/images/card_icons/visa.png" title=" Visa " width="65" /><img alt="MasterCard" class="ProtxDirectCardsAcceptedSideboxCardIcon" height="25" src="includes/templates/template_default/images/card_icons/mc.png" title=" MasterCard " width="40" /><img alt="Visa Debit" class="ProtxDirectCardsAcceptedSideboxCardIcon" height="25" src="includes/templates/template_default/images/card_icons/visa_debit.png" title=" Visa Debit " width="40" /><img alt="Solo" class="ProtxDirectCardsAcceptedSideboxCardIcon" height="25" src="includes/templates/template_default/images/card_icons/solo.png" title=" Solo " width="20" /><img alt="Maestro" class="ProtxDirectCardsAcceptedSideboxCardIcon" height="25" src="includes/templates/template_default/images/card_icons/maestro.png" title=" Maestro " width="40" /><img alt="Visa Electron (UKE)" class="ProtxDirectCardsAcceptedSideboxCardIcon" height="25" src="includes/templates/template_default/images/card_icons/visa_electron.png" title=" Visa Electron (UKE) " width="40" /><div style="clear: left;"> </div>
|
||||
<img alt="Verified By Visa" class="ProtxDirectCardsAcceptedSidebox3DSecureIcon" height="34" src="includes/templates/template_default/images/card_icons/verified_by_visa_small.png" title=" Verified By Visa " width="60" />
|
||||
<img alt="MasterCard SecureCode" class="ProtxDirectCardsAcceptedSidebox3DSecureIcon" height="34" src="includes/templates/template_default/images/card_icons/mastercard_securecode_small.png" title=" MasterCard SecureCode " width="57" />
|
||||
<div style="clear: left;"> </div>
|
||||
<img alt="Secured by Protx" class="ProtxDirectCardsAcceptedSideboxProtxIcon" height="43" src="includes/templates/template_default/images/card_icons/protx_secured.png" title=" Secured by Protx " width="118" />
|
||||
|
||||
</div></div>
|
||||
<!--// eof: protxdirectcardsaccepted //-->
|
||||
|
||||
|
||||
<div style="margin-top: 1em;"><a href="/tempuruk/company/nasa"><img border="0" height="138" src="/tempurUK/images/nasa.gif" width="133"></a></div>
|
||||
|
||||
</div></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td style="background: url(/tempurUK/images/bottom.gif) bottom left repeat-x; padding-top: 4em;" />
|
||||
<table align="center" border="0" cellpadding="5" height="76" width="90%">
|
||||
<tbody>
|
||||
<tr class="navi">
|
||||
<td>
|
||||
<div align="center"><a href="/tempuruk/freeinfo/">FREE INFO PACK</a></div>
|
||||
</td>
|
||||
<td>
|
||||
<div align="center"> </div>
|
||||
</td>
|
||||
<td>
|
||||
<div align="center"><a href="/tempuruk/company/contactus/">CONTACT US</a></div>
|
||||
</td>
|
||||
<td>
|
||||
<div align="center"><a href="/tempuruk/material/">NIGHT NIGHT BACK PAIN</a></div>
|
||||
</td>
|
||||
<td>
|
||||
<div align="center"><a href="/tempuruk/material/sleepquality/">THE BEST NIGHT'S SLEEP</a></div>
|
||||
</td>
|
||||
<td>
|
||||
<div align="center"><a href="/tempuruk/mattresses/">MATTRESSES</a></div>
|
||||
</td>
|
||||
<td>
|
||||
<div align="center"><a href="/tempuruk/pillows/">PILLOWS</a></div>
|
||||
</td>
|
||||
<td>
|
||||
<img alt="" height="76" src="/tempurUK/images/spacer.gif" width="1" />
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" style="padding-bottom: 1em;">
|
||||
<span class="BinNavigation"><a href="/tempuruk/material/TermsConditions/">*Term & Conditions</a></span>
|
||||
<span class="BinNavigation"><a href="/tempuruk/faq/faq60night/">*FAQ 60Night</a></span>
|
||||
<span class="bodyTextSmall">© 2008 TEMPUR UK Ltd. All Rights Reserved</span>
|
||||
|
||||
<span class="bodyTitle">. </span><span class="BinNavigation"><a href="/tempuruk/privacy/">PRIVACY POLICY</a></span>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
<!--</div>-->
|
||||
<!--bof- parse time display -->
|
||||
<!--eof- parse time display -->
|
||||
<!--bof- banner #6 display -->
|
||||
<!--eof- banner #6 display -->
|
||||
<!-- Siteimprove: Start //-->
|
||||
<script language="JavaScript" src="//ssl.siteimprove.com/js/siteanalyze.js" type="text/javascript"></script>
|
||||
<!-- Siteimprove: End //-->
|
||||
<script type="text/javascript">
|
||||
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
|
||||
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
|
||||
</script>
|
||||
<script type="text/javascript">
|
||||
var pageTracker = _gat._getTracker("UA-5947656-1");
|
||||
pageTracker._trackPageview();
|
||||
</script>
|
||||
<!-- ClickTale Bottom part -->
|
||||
<div id="ClickTaleDiv" style="display: none;"></div>
|
||||
<script src="/WRb.js" type="text/javascript"></script>
|
||||
<script type="text/javascript">
|
||||
var ClickTaleSSL=1;
|
||||
if(typeof ClickTale=='function') ClickTale(28035,1);
|
||||
</script>
|
||||
<!-- ClickTale end of Bottom part -->
|
||||
|
||||
|
||||
</body></html>
|
||||
|
|
@ -1,78 +0,0 @@
|
|||
[
|
||||
{
|
||||
"surrounds_attribute": "name",
|
||||
"annotation_text": null,
|
||||
"match_common_prefix": false,
|
||||
"surrounds_variant": null,
|
||||
"variant_id": null,
|
||||
"tag_attributes": [],
|
||||
"end_index": 133,
|
||||
"start_index": 132,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"surrounds_attribute": null,
|
||||
"annotation_text": null,
|
||||
"match_common_prefix": false,
|
||||
"surrounds_variant": null,
|
||||
"variant_id": null,
|
||||
"tag_attributes": [
|
||||
[
|
||||
"src",
|
||||
"image_urls"
|
||||
]
|
||||
],
|
||||
"end_index": 142,
|
||||
"start_index": 141,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"surrounds_attribute": null,
|
||||
"annotation_text": null,
|
||||
"match_common_prefix": false,
|
||||
"surrounds_variant": null,
|
||||
"variant_id": null,
|
||||
"tag_attributes": [
|
||||
[
|
||||
"src",
|
||||
"image_urls"
|
||||
]
|
||||
],
|
||||
"end_index": 149,
|
||||
"start_index": 148,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"surrounds_attribute": "description",
|
||||
"annotation_text": null,
|
||||
"match_common_prefix": false,
|
||||
"surrounds_variant": null,
|
||||
"variant_id": null,
|
||||
"tag_attributes": [],
|
||||
"end_index": 207,
|
||||
"start_index": 161,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"surrounds_attribute": "price",
|
||||
"annotation_text": null,
|
||||
"match_common_prefix": false,
|
||||
"surrounds_variant": null,
|
||||
"variant_id": null,
|
||||
"tag_attributes": [],
|
||||
"end_index": 258,
|
||||
"start_index": 257,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"surrounds_attribute": "features",
|
||||
"annotation_text": null,
|
||||
"match_common_prefix": false,
|
||||
"surrounds_variant": null,
|
||||
"variant_id": null,
|
||||
"tag_attributes": [],
|
||||
"end_index": 421,
|
||||
"start_index": 324,
|
||||
"metadata": {}
|
||||
}
|
||||
]
|
||||
|
|
@ -1,955 +0,0 @@
|
|||
"""
|
||||
tests for page parsing
|
||||
|
||||
Page parsing effectiveness is measured through the evaluation system. These
|
||||
tests should focus on specific bits of functionality work correctly.
|
||||
"""
|
||||
from twisted.trial.unittest import TestCase, SkipTest
|
||||
from scrapy.contrib.ibl.htmlpage import HtmlPage
|
||||
from scrapy.contrib.ibl.descriptor import (FieldDescriptor as A,
|
||||
ItemDescriptor)
|
||||
from scrapy.contrib.ibl.extractors import (contains_any_numbers,
|
||||
image_url)
|
||||
|
||||
try:
|
||||
import numpy
|
||||
__doctests__ = ['scrapy.contrib.ibl.extraction.%s' % x for x in \
|
||||
['regionextract', 'similarity', 'pageobjects']]
|
||||
except ImportError:
|
||||
numpy = None
|
||||
|
||||
if numpy:
|
||||
from scrapy.contrib.ibl.extraction import InstanceBasedLearningExtractor
|
||||
|
||||
# simple page with all features
|
||||
|
||||
ANNOTATED_PAGE1 = u"""
|
||||
<html>
|
||||
<h1>COMPANY - <ins
|
||||
data-scrapy-annotate="{"variant": 0, "generated": true,
|
||||
"annotations": {"content": "title"}}"
|
||||
>Item Title</ins></h1>
|
||||
<p>introduction</p>
|
||||
<div>
|
||||
<img data-scrapy-annotate="{"variant": 0,
|
||||
"annotations": {"src": "image_url"}}"
|
||||
src="img.jpg"/>
|
||||
<p data-scrapy-annotate="{"variant": 0,
|
||||
"annotations": {"content": "description"}}">
|
||||
This is such a nice item<br/> Everybody likes it.
|
||||
</p>
|
||||
<br/>
|
||||
</div>
|
||||
<p>click here for other items</p>
|
||||
</html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE1 = u"""
|
||||
<html>
|
||||
<h1>Scrapy - Nice Product</h1>
|
||||
<p>introduction</p>
|
||||
<div>
|
||||
<img src="nice_product.jpg" alt="a nice product image"/>
|
||||
<p>wonderful product</p>
|
||||
<br/>
|
||||
</div>
|
||||
</html>
|
||||
"""
|
||||
|
||||
# single tag with multiple items extracted
|
||||
ANNOTATED_PAGE2 = u"""
|
||||
<a href="http://example.com/xxx" title="xxx"
|
||||
data-scrapy-annotate="{"variant": 0,
|
||||
"annotations": {"content": "description",
|
||||
"href": "image_url", "title": "name"}}"
|
||||
>xx</a>
|
||||
xxx
|
||||
</a>
|
||||
"""
|
||||
EXTRACT_PAGE2 = u"""<a href='http://example.com/product1.jpg'
|
||||
title="product 1">product 1 is great</a>"""
|
||||
|
||||
# matching must match the second attribute in order to find the first
|
||||
ANNOTATED_PAGE3 = u"""
|
||||
<p data-scrapy-annotate="{"variant": 0,
|
||||
"annotations": {"content": "description"}}">xx</p>
|
||||
<div data-scrapy-annotate="{"variant": 0,
|
||||
"annotations": {"content": "delivery"}}">xx</div>
|
||||
"""
|
||||
EXTRACT_PAGE3 = u"""
|
||||
<p>description</p>
|
||||
<div>delivery</div>
|
||||
<p>this is not the description</p>
|
||||
"""
|
||||
|
||||
# test inferring repeated elements
|
||||
ANNOTATED_PAGE4 = u"""
|
||||
<ul>
|
||||
<li data-scrapy-annotate="{"variant": 0,
|
||||
"annotations": {"content": "features"}}">feature1</li>
|
||||
<li data-scrapy-annotate="{"variant": 0,
|
||||
"annotations": {"content": "features"}}">feature2</li>
|
||||
</ul>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE4 = u"""
|
||||
<ul>
|
||||
<li>feature1</li> ignore this
|
||||
<li>feature2</li>
|
||||
<li>feature3</li>
|
||||
</ul>
|
||||
"""
|
||||
|
||||
# test variant handling with identical repeated variant
|
||||
ANNOTATED_PAGE5 = u"""
|
||||
<p data-scrapy-annotate="{"annotations":
|
||||
{"content": "description"}}">description</p>
|
||||
<table>
|
||||
<tr>
|
||||
<td data-scrapy-annotate="{"variant": 1, "annotations":
|
||||
{"content": "colour"}}" >colour 1</td>
|
||||
<td data-scrapy-annotate="{"variant": 1, "annotations":
|
||||
{"content": "price"}}" >price 1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td data-scrapy-annotate="{"variant": 2, "annotations":
|
||||
{"content": "colour"}}" >colour 2</td>
|
||||
<td data-scrapy-annotate="{"variant": 2, "annotations":
|
||||
{"content": "price"}}" >price 2</td>
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE5 = u"""
|
||||
<p>description</p>
|
||||
<table>
|
||||
<tr>
|
||||
<td>colour 1</td>
|
||||
<td>price 1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>colour 2</td>
|
||||
<td>price 2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>colour 3</td>
|
||||
<td>price 3</td>
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
|
||||
# test variant handling with irregular structure and some non-variant
|
||||
# attributes
|
||||
ANNOTATED_PAGE6 = u"""
|
||||
<p data-scrapy-annotate="{"annotations":
|
||||
{"content": "description"}}">description</p>
|
||||
<p data-scrapy-annotate="{"variant": 1, "annotations":
|
||||
{"content": "name"}}">name 1</p>
|
||||
<div data-scrapy-annotate="{"variant": 3, "annotations":
|
||||
{"content": "name"}}" >name 3</div>
|
||||
<p data-scrapy-annotate="{"variant": 2, "annotations":
|
||||
{"content": "name"}}" >name 2</p>
|
||||
"""
|
||||
EXTRACT_PAGE6 = u"""
|
||||
<p>description</p>
|
||||
<p>name 1</p>
|
||||
<div>name 3</div>
|
||||
<p>name 2</p>
|
||||
"""
|
||||
|
||||
# test repeating variants at the table column level
|
||||
ANNOTATED_PAGE7 = u"""
|
||||
<table>
|
||||
<tr>
|
||||
<td data-scrapy-annotate="{"variant": 1, "annotations":
|
||||
{"content": "colour"}}" >colour 1</td>
|
||||
<td data-scrapy-annotate="{"variant": 2, "annotations":
|
||||
{"content": "colour"}}" >colour 2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td data-scrapy-annotate="{"variant": 2, "annotations":
|
||||
{"content": "price"}}" >price 1</td>
|
||||
<td data-scrapy-annotate="{"variant": 2, "annotations":
|
||||
{"content": "price"}}" >price 2</td>
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE7 = u"""
|
||||
<table>
|
||||
<tr>
|
||||
<td>colour 1</td>
|
||||
<td>colour 2</td>
|
||||
<td>colour 3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>price 1</td>
|
||||
<td>price 2</td>
|
||||
<td>price 3</td>
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE8 = u"""
|
||||
<html><body>
|
||||
<h1>A product</h1>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<p>XXXX XXXX xxxxx</p>
|
||||
<div data-scrapy-ignore="true">
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
</div>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">
|
||||
10.00<p data-scrapy-ignore="true"> 13</p>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE8 = u"""
|
||||
<html><body>
|
||||
<h1>A product</h1>
|
||||
<div>
|
||||
<p>A very nice product for all intelligent people</p>
|
||||
<div>
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
12.00<p> ID 15</p>
|
||||
(VAT exc.)</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE9 = ANNOTATED_PAGE8
|
||||
|
||||
EXTRACT_PAGE9 = u"""
|
||||
<html><body>
|
||||
<img src="logo.jpg" />
|
||||
<h1>A product</h1>
|
||||
<div>
|
||||
<p>A very nice product for all intelligent people</p>
|
||||
<div>
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
12.00<p> ID 16</p>
|
||||
(VAT exc.)</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE11 = u"""
|
||||
<html><body>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<ins data-scrapy-annotate="{"variant": 0, "generated": true,
|
||||
"annotations": {"content": "name"}}">
|
||||
SL342
|
||||
</ins>
|
||||
<br/>
|
||||
Nice product for ladies
|
||||
<br/><ins data-scrapy-annotate="{"variant": 0, "generated": true,
|
||||
"annotations": {"content": "price"}}">
|
||||
£s;85.00
|
||||
</ins>
|
||||
</p>
|
||||
<ins data-scrapy-annotate="{"variant": 0, "generated": true,
|
||||
"annotations": {"content": "price_before_discount"}}">
|
||||
£s;100.00
|
||||
</ins>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE11 = u"""
|
||||
<html><body>
|
||||
<p>
|
||||
SL342
|
||||
<br/>
|
||||
Nice product for ladies
|
||||
<br/>
|
||||
£s;85.00
|
||||
</p>
|
||||
£s;100.00
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE12 = u"""
|
||||
<html><body>
|
||||
<h1 data-scrapy-ignore-beneath="true">A product</h1>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<p>XXXX XXXX xxxxx</p>
|
||||
<div data-scrapy-ignore-beneath="true">
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
<div>
|
||||
10.00<p> 13</p>
|
||||
</div>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE12a = u"""
|
||||
<html><body>
|
||||
<h1>A product</h1>
|
||||
<div>
|
||||
<p>A very nice product for all intelligent people</p>
|
||||
<div>
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
<div>
|
||||
12.00<p> ID 15</p>
|
||||
(VAT exc.)
|
||||
</div></div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE12b = u"""
|
||||
<html><body>
|
||||
<h1>A product</h1>
|
||||
<div>
|
||||
<p>A very nice product for all intelligent people</p>
|
||||
<div>
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
<div>
|
||||
12.00<p> ID 15</p>
|
||||
(VAT exc.)
|
||||
</div>
|
||||
<ul>
|
||||
Features
|
||||
<li>Feature A</li>
|
||||
<li>Feature B</li>
|
||||
</ul>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
# Ex1: nested annotation with token sequence replica outside exterior annotation
|
||||
# and a possible sequence pattern can be extracted only with
|
||||
# correct handling of nested annotations
|
||||
ANNOTATED_PAGE13a = u"""
|
||||
<html><body>
|
||||
<span>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<hr/>
|
||||
<h3 data-scrapy-annotate="{"variant": 0, "annotations": {"content": "name"}}">A product</h3>
|
||||
<b data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">$50.00</b>
|
||||
This product is excelent. Buy it!
|
||||
</p>
|
||||
</span>
|
||||
<span>
|
||||
<p>
|
||||
<h3>See other products:</h3>
|
||||
<b>Product b</b>
|
||||
</p>
|
||||
</span>
|
||||
<hr/>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE13a = u"""
|
||||
<html><body>
|
||||
<span>
|
||||
<p>
|
||||
<h3>A product</h3>
|
||||
<b>$50.00</b>
|
||||
This product is excelent. Buy it!
|
||||
<hr/>
|
||||
</p>
|
||||
</span>
|
||||
<span>
|
||||
<p>
|
||||
<h3>See other products:</h3>
|
||||
<b>Product B</b>
|
||||
</p>
|
||||
</span>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
# Ex2: annotation with token sequence replica inside a previous nested annotation
|
||||
# and a possible sequence pattern can be extracted only with
|
||||
# correct handling of nested annotations
|
||||
ANNOTATED_PAGE13b = u"""
|
||||
<html><body>
|
||||
<span>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<h3 data-scrapy-annotate="{"variant": 0, "annotations": {"content": "name"}}">A product</h3>
|
||||
<b>Previous price: $50.00</b>
|
||||
This product is excelent. Buy it!
|
||||
</p>
|
||||
</span>
|
||||
<span>
|
||||
<p>
|
||||
<h3>Save 10%!!</h3>
|
||||
<b data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">$45.00</b>
|
||||
</p>
|
||||
</span>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE13b = u"""
|
||||
<html><body>
|
||||
<span>
|
||||
<p>
|
||||
<h3>A product</h3>
|
||||
<b>$50.00</b>
|
||||
This product is excelent. Buy it!
|
||||
</p>
|
||||
</span>
|
||||
<span>
|
||||
<hr/>
|
||||
<p>
|
||||
<h3>Save 10%!!</h3>
|
||||
<b>$45.00</b>
|
||||
</p>
|
||||
</span>
|
||||
<hr/>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE14 = u"""
|
||||
<html><body>
|
||||
<b data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}"></b>
|
||||
<p data-scrapy-ignore="true"></p>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE14 = u"""
|
||||
<html><body>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE15 = u"""
|
||||
<html><body>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "short_description"}}">Short
|
||||
<div data-scrapy-ignore="true" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "site_id"}}">892342</div>
|
||||
</div>
|
||||
<hr/>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">Description
|
||||
<b data-scrapy-ignore="true" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">90.00</b>
|
||||
</p>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE15 = u"""
|
||||
<html><body>
|
||||
<hr/>
|
||||
<p>Description
|
||||
<b>80.00</b>
|
||||
</p>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE16 = u"""
|
||||
<html><body>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
Description
|
||||
<p data-scrapy-ignore="true" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "name"}}">
|
||||
name</p>
|
||||
<p data-scrapy-ignore="true" data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">
|
||||
80.00</p>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE16 = u"""
|
||||
<html><body>
|
||||
<p>product name</p>
|
||||
<p>90.00</p>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE17 = u"""
|
||||
<html><body>
|
||||
<span>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
This product is excelent. Buy it!
|
||||
</p>
|
||||
</span>
|
||||
<table></table>
|
||||
<img src="line.jpg" data-scrapy-ignore-beneath="true"/>
|
||||
<span>
|
||||
<h3>See other products:</h3>
|
||||
<p>Product b
|
||||
</p>
|
||||
</span>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE17 = u"""
|
||||
<html><body>
|
||||
<span>
|
||||
<p>
|
||||
This product is excelent. Buy it!
|
||||
</p>
|
||||
</span>
|
||||
<img src="line.jpg"/>
|
||||
<span>
|
||||
<h3>See other products:</h3>
|
||||
<p>Product B
|
||||
</p>
|
||||
</span>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE18 = u"""
|
||||
<html><body>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<ins data-scrapy-ignore="true" data-scrapy-annotate="{"variant": 0, "generated": true, "annotations": {"content": "site_id"}}">Item Id</ins>
|
||||
<br>
|
||||
Description
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE18 = u"""
|
||||
<html><body>
|
||||
<div>
|
||||
Item Id
|
||||
<br>
|
||||
Description
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE19 = u"""
|
||||
<html><body>
|
||||
<div>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "name"}}">Product name</p>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">60.00</p>
|
||||
<img data-scrapy-annotate="{"variant": 0, "annotations": {"src": "image_urls"}}"src="image.jpg" />
|
||||
<p data-scrapy-annotate="{"variant": 0, "required": ["description"], "annotations": {"content": "description"}}">description</p>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE19a = u"""
|
||||
<html><body>
|
||||
<div>
|
||||
<p>Product name</p>
|
||||
<p>60.00</p>
|
||||
<img src="http://example.com/image.jpg" />
|
||||
<p>description</p>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE19b = u"""
|
||||
<html><body>
|
||||
<div>
|
||||
<p>Range</p>
|
||||
<p>from 20.00</p>
|
||||
<img src="http://example.com/image1.jpg" />
|
||||
<p>
|
||||
<br/>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE20 = u"""
|
||||
<html><body>
|
||||
<h1>Product Name</h1>
|
||||
<img src="product.jpg">
|
||||
<br/>
|
||||
<span><ins data-scrapy-annotate="{"variant": 1, "generated": true,
|
||||
"annotations": {"content": "name"}}">Twin</ins>:</span> $<ins data-scrapy-annotate="{"variant": 1, "generated": true,
|
||||
"annotations": {"content": "price"}}">270</ins> - November 2010<br/>
|
||||
<span><ins data-scrapy-annotate="{"variant": 2, "generated": true,
|
||||
"annotations": {"content": "name"}}">Queen</ins>:</span> $<ins data-scrapy-annotate="{"variant": 2, "generated": true,
|
||||
"annotations": {"content": "price"}}">330</ins> - In stock<br/>
|
||||
<br/>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE20 = u"""
|
||||
<html><body>
|
||||
<h1>Product Name</h1>
|
||||
<img src="product.jpg">
|
||||
<br/>
|
||||
<span>Twin:</span> $270 - November 2010<br/>
|
||||
<span>Queen:</span> $330 - Movember 2010<br/>
|
||||
<br/>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE21 = u"""
|
||||
<html><body>
|
||||
<img src="image.jpg" data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"src": "image_urls"}}">
|
||||
<p>
|
||||
<table>
|
||||
|
||||
<tr><td><img src="swatch1.jpg" data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"src": "swatches"}}"></td></tr>
|
||||
|
||||
<tr><td><img src="swatch2.jpg"></td></tr>
|
||||
|
||||
<tr><td><img src="swatch3.jpg"></td></tr>
|
||||
|
||||
<tr><td><img src="swatch4.jpg" data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"src": "swatches"}}"></td></tr>
|
||||
|
||||
</table>
|
||||
|
||||
<div data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"content": "category"}}">tables</div>
|
||||
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE21 = u"""
|
||||
<html><body>
|
||||
<img src="image.jpg">
|
||||
<p>
|
||||
<table>
|
||||
|
||||
<tr><td><img src="swatch1.jpg"></td></tr>
|
||||
|
||||
<tr><td><img src="swatch2.jpg"></td></tr>
|
||||
|
||||
<tr><td><img src="swatch3.jpg"></td></tr>
|
||||
|
||||
<tr><td><img src="swatch4.jpg"></td></tr>
|
||||
|
||||
</table>
|
||||
|
||||
<div>chairs</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE22 = u"""
|
||||
<html><body>
|
||||
<img src="image.jpg" data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"src": "image_urls"}}">
|
||||
<p>
|
||||
<table>
|
||||
|
||||
<tr><td>
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "name"}}">product 1</p>
|
||||
<b data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "price"}}">$67</b>
|
||||
<img src="swatch1.jpg" data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"src": "swatches"}}">
|
||||
</td></tr>
|
||||
|
||||
<tr><td>
|
||||
<p>product 2</p>
|
||||
<b>$70</b>
|
||||
<img src="swatch2.jpg">
|
||||
</td></tr>
|
||||
|
||||
<tr><td>
|
||||
<p>product 3</p>
|
||||
<b>$73</b>
|
||||
<img src="swatch3.jpg">
|
||||
</td></tr>
|
||||
|
||||
<tr><td>
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "name"}}">product 4</p>
|
||||
<b data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "price"}}">$80</b>
|
||||
<img src="swatch4.jpg" data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"src": "swatches"}}">
|
||||
</td></tr>
|
||||
|
||||
</table>
|
||||
|
||||
<div data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"content": "category"}}">tables</div>
|
||||
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE22 = u"""
|
||||
<html><body>
|
||||
<img src="image.jpg">
|
||||
<p>
|
||||
<table>
|
||||
|
||||
<tr><td>
|
||||
<p>product 1</p>
|
||||
<b>$70</b>
|
||||
<img src="swatch1.jpg">
|
||||
</td></tr>
|
||||
|
||||
<tr><td>
|
||||
<p>product 2</p>
|
||||
<b>$80</b>
|
||||
<img src="swatch2.jpg">
|
||||
</td></tr>
|
||||
|
||||
<tr><td>
|
||||
<p>product 3</p>
|
||||
<b>$90</b>
|
||||
<img src="swatch3.jpg">
|
||||
</td></tr>
|
||||
|
||||
<tr><td>
|
||||
<p>product 4</p>
|
||||
<b>$100</b>
|
||||
<img src="swatch4.jpg">
|
||||
</td></tr>
|
||||
|
||||
</table>
|
||||
|
||||
<div>chairs</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
ANNOTATED_PAGE23 = u"""
|
||||
<html><body>
|
||||
<h4>Product</h4>
|
||||
<table>
|
||||
<tr><td>
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "name"}}">Variant 1<b data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "price"}}" data-scrapy-ignore="true">560</b></p>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<p>Variant 2<b>570</b></p>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "name"}}">Variant 3<b data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "price"}}" data-scrapy-ignore="true">580</b></p>
|
||||
</td></tr>
|
||||
</table>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
EXTRACT_PAGE23 = u"""
|
||||
<html><body>
|
||||
<h4>Product</h4>
|
||||
<table>
|
||||
<tr><td>
|
||||
<p>Variant 1<b>300</b></p>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<p>Variant 2<b>320</b></p>
|
||||
</td></tr>
|
||||
<tr><td>
|
||||
<p>Variant 3<b>340</b></p>
|
||||
</td></tr>
|
||||
</table>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
SAMPLE_DESCRIPTOR1 = ItemDescriptor('test', 'product test', [
|
||||
A('name', "Product name", required=True),
|
||||
A('price', "Product price, including any discounts and tax or vat",
|
||||
contains_any_numbers, True),
|
||||
A('image_urls', "URLs for one or more images", image_url, True),
|
||||
A('description', "The full description of the product", allow_markup=True),
|
||||
]
|
||||
)
|
||||
|
||||
# A list of (test name, [templates], page, extractors, expected_result)
|
||||
TEST_DATA = [
|
||||
# extract from a similar page
|
||||
('similar page extraction', [ANNOTATED_PAGE1], EXTRACT_PAGE1, None,
|
||||
{u'title': [u'Nice Product'], u'description': [u'wonderful product'],
|
||||
u'image_url': [u'nice_product.jpg']}
|
||||
),
|
||||
# strip the first 5 characters from the title
|
||||
('extractor test', [ANNOTATED_PAGE1], EXTRACT_PAGE1,
|
||||
ItemDescriptor('test', 'product test',
|
||||
[A('title', "something about a title", lambda x: x[5:])]),
|
||||
{u'title': [u'Product'], u'description': [u'wonderful product'],
|
||||
u'image_url': [u'nice_product.jpg']}
|
||||
),
|
||||
# compilicated tag (multiple attributes and annotation)
|
||||
('multiple attributes and annotation', [ANNOTATED_PAGE2], EXTRACT_PAGE2, None,
|
||||
{'name': [u'product 1'], 'image_url': [u'http://example.com/product1.jpg'],
|
||||
'description': [u'product 1 is great']}
|
||||
),
|
||||
# can only work out correct placement by matching the second attribute first
|
||||
('ambiguous description', [ANNOTATED_PAGE3], EXTRACT_PAGE3, None,
|
||||
{'description': [u'description'], 'delivery': [u'delivery']}
|
||||
),
|
||||
# infer a repeated structure
|
||||
('repeated elements', [ANNOTATED_PAGE4], EXTRACT_PAGE4, None,
|
||||
{'features': [u'feature1', u'feature2', u'feature3']}
|
||||
),
|
||||
# identical variants with a repeated structure
|
||||
('repeated identical variants', [ANNOTATED_PAGE5], EXTRACT_PAGE5, None,
|
||||
{
|
||||
'description': [u'description'],
|
||||
'variants': [
|
||||
{u'colour': [u'colour 1'], u'price': [u'price 1']},
|
||||
{u'colour': [u'colour 2'], u'price': [u'price 2']},
|
||||
{u'colour': [u'colour 3'], u'price': [u'price 3']}
|
||||
]
|
||||
}
|
||||
),
|
||||
# variants with an irregular structure
|
||||
('irregular variants', [ANNOTATED_PAGE6], EXTRACT_PAGE6, None,
|
||||
{
|
||||
'description': [u'description'],
|
||||
'variants': [
|
||||
{u'name': [u'name 1']},
|
||||
{u'name': [u'name 3']},
|
||||
{u'name': [u'name 2']}
|
||||
]
|
||||
}
|
||||
),
|
||||
|
||||
# discovering repeated variants in table columns
|
||||
# ('variants in table columns', [ANNOTATED_PAGE7], EXTRACT_PAGE7, None,
|
||||
# {'variants': [
|
||||
# {u'colour': [u'colour 1'], u'price': [u'price 1']},
|
||||
# {u'colour': [u'colour 2'], u'price': [u'price 2']},
|
||||
# {u'colour': [u'colour 3'], u'price': [u'price 3']}
|
||||
# ]}
|
||||
# ),
|
||||
|
||||
|
||||
# ignored regions
|
||||
(
|
||||
'ignored_regions', [ANNOTATED_PAGE8], EXTRACT_PAGE8, None,
|
||||
{
|
||||
'description': [u'\n A very nice product for all intelligent people \n\n'],
|
||||
'price': [u'\n12.00\n(VAT exc.)'],
|
||||
}
|
||||
),
|
||||
# shifted ignored regions (detected by region similarity)
|
||||
(
|
||||
'shifted_ignored_regions', [ANNOTATED_PAGE9], EXTRACT_PAGE9, None,
|
||||
{
|
||||
'description': [u'\n A very nice product for all intelligent people \n\n'],
|
||||
'price': [u'\n12.00\n(VAT exc.)'],
|
||||
}
|
||||
),
|
||||
(# special case with partial annotations
|
||||
'special_partial_annotation', [ANNOTATED_PAGE11], EXTRACT_PAGE11, None,
|
||||
{
|
||||
'name': [u'SL342'],
|
||||
'description': [u'\nSL342\n \nNice product for ladies\n \n£s;85.00\n'],
|
||||
'price': [u'£s;85.00'],
|
||||
'price_before_discount': [u'£s;100.00'],
|
||||
}
|
||||
),
|
||||
(# with ignore-beneath feature
|
||||
'ignore-beneath', [ANNOTATED_PAGE12], EXTRACT_PAGE12a, None,
|
||||
{
|
||||
'description': [u'\n A very nice product for all intelligent people \n'],
|
||||
}
|
||||
),
|
||||
(# ignore-beneath with extra tags
|
||||
'ignore-beneath with extra tags', [ANNOTATED_PAGE12], EXTRACT_PAGE12b, None,
|
||||
{
|
||||
'description': [u'\n A very nice product for all intelligent people \n'],
|
||||
}
|
||||
),
|
||||
('nested annotation with replica outside', [ANNOTATED_PAGE13a], EXTRACT_PAGE13a, None,
|
||||
{'description': [u'\n A product \n $50.00 \nThis product is excelent. Buy it!\n \n'],
|
||||
'price': ["$50.00"],
|
||||
'name': [u'A product']}
|
||||
),
|
||||
('outside annotation with nested replica', [ANNOTATED_PAGE13b], EXTRACT_PAGE13b, None,
|
||||
{'description': [u'\n A product \n $50.00 \nThis product is excelent. Buy it!\n'],
|
||||
'price': ["$45.00"],
|
||||
'name': [u'A product']}
|
||||
),
|
||||
('consistency check', [ANNOTATED_PAGE14], EXTRACT_PAGE14, None,
|
||||
{},
|
||||
),
|
||||
('consecutive nesting', [ANNOTATED_PAGE15], EXTRACT_PAGE15, None,
|
||||
{'description': [u'Description\n\n'],
|
||||
'price': [u'80.00']},
|
||||
),
|
||||
('nested inside not found', [ANNOTATED_PAGE16], EXTRACT_PAGE16, None,
|
||||
{'price': [u'90.00'],
|
||||
'name': [u'product name']},
|
||||
),
|
||||
('ignored region helps to find attributes', [ANNOTATED_PAGE17], EXTRACT_PAGE17, None,
|
||||
{'description': [u'\nThis product is excelent. Buy it!\n']},
|
||||
),
|
||||
('ignored region in partial annotation', [ANNOTATED_PAGE18], EXTRACT_PAGE18, None,
|
||||
{u'site_id': [u'Item Id'],
|
||||
u'description': [u'\nDescription\n']},
|
||||
),
|
||||
('extra required attribute product', [ANNOTATED_PAGE19], EXTRACT_PAGE19a,
|
||||
SAMPLE_DESCRIPTOR1,
|
||||
{u'price': [u'60.00'],
|
||||
u'description': [u'description'],
|
||||
u'image_urls': [['http://example.com/image.jpg']],
|
||||
u'name': [u'Product name']},
|
||||
),
|
||||
('extra required attribute no product', [ANNOTATED_PAGE19], EXTRACT_PAGE19b,
|
||||
SAMPLE_DESCRIPTOR1,
|
||||
None,
|
||||
),
|
||||
('repeated partial annotations with variants', [ANNOTATED_PAGE20], EXTRACT_PAGE20, None,
|
||||
{u'variants': [
|
||||
{'price': ['270'], 'name': ['Twin']},
|
||||
{'price': ['330'], 'name': ['Queen']},
|
||||
]},
|
||||
),
|
||||
('variants with swatches', [ANNOTATED_PAGE21], EXTRACT_PAGE21, None,
|
||||
{u'category': [u'chairs'],
|
||||
u'image_urls': [u'image.jpg'],
|
||||
u'variants': [
|
||||
{'swatches': ['swatch1.jpg']},
|
||||
{'swatches': ['swatch2.jpg']},
|
||||
{'swatches': ['swatch3.jpg']},
|
||||
{'swatches': ['swatch4.jpg']},
|
||||
]
|
||||
},
|
||||
),
|
||||
('variants with swatches complete', [ANNOTATED_PAGE22], EXTRACT_PAGE22, None,
|
||||
{u'category': [u'chairs'],
|
||||
u'variants': [
|
||||
{u'swatches': [u'swatch1.jpg'],
|
||||
u'price': [u'$70'],
|
||||
u'name': [u'product 1']},
|
||||
{u'swatches': [u'swatch2.jpg'],\
|
||||
u'price': [u'$80'],
|
||||
u'name': [u'product 2']},
|
||||
{u'swatches': [u'swatch3.jpg'],
|
||||
u'price': [u'$90'],
|
||||
u'name': [u'product 3']},
|
||||
{u'swatches': [u'swatch4.jpg'],
|
||||
u'price': [u'$100'],
|
||||
u'name': [u'product 4']}
|
||||
],
|
||||
u'image_urls': [u'image.jpg']},
|
||||
),
|
||||
('repeated (variants) with ignore annotations', [ANNOTATED_PAGE23], EXTRACT_PAGE23, None,
|
||||
{'variants': [
|
||||
{u'price': [u'300'], u'name': [u'Variant 1']},
|
||||
{u'price': [u'320'], u'name': [u'Variant 2']},
|
||||
{u'price': [u'340'], u'name': [u'Variant 3']}
|
||||
]},
|
||||
),
|
||||
]
|
||||
|
||||
class TestExtraction(TestCase):
|
||||
|
||||
def setUp(self):
|
||||
if not numpy:
|
||||
raise SkipTest("numpy not available")
|
||||
|
||||
def _run_extraction(self, name, templates, page, extractors, expected_output):
|
||||
self.trace = None
|
||||
template_pages = [HtmlPage(None, {}, t) for t in templates]
|
||||
extractor = InstanceBasedLearningExtractor(template_pages, extractors, True)
|
||||
actual_output, _ = extractor.extract(HtmlPage(None, {}, page))
|
||||
if not actual_output:
|
||||
if expected_output is None:
|
||||
return
|
||||
assert False, "failed to extract data for test '%s'" % name
|
||||
actual_output = actual_output[0]
|
||||
self.trace = ["Extractor:\n%s" % extractor] + actual_output.pop('trace', [])
|
||||
expected_names = set(expected_output.keys())
|
||||
actual_names = set(actual_output.keys())
|
||||
|
||||
missing_in_output = filter(None, expected_names - actual_names)
|
||||
error = "attributes '%s' were expected but were not present in test '%s'" % \
|
||||
("', '".join(missing_in_output), name)
|
||||
assert len(missing_in_output) == 0, error
|
||||
|
||||
unexpected = actual_names - expected_names
|
||||
error = "unexpected attributes %s in test '%s'" % \
|
||||
(', '.join(unexpected), name)
|
||||
assert len(unexpected) == 0, error
|
||||
|
||||
for k, v in expected_output.items():
|
||||
extracted = actual_output[k]
|
||||
assert v == extracted, "in test '%s' for attribute '%s', " \
|
||||
"expected value '%s' but got '%s'" % (name, k, v, extracted)
|
||||
|
||||
def test_expected_outputs(self):
|
||||
try:
|
||||
for data in TEST_DATA:
|
||||
self._run_extraction(*data)
|
||||
except AssertionError:
|
||||
if self.trace:
|
||||
print "Trace:"
|
||||
for line in self.trace:
|
||||
print "\n---\n%s" % line
|
||||
raise
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
try:
|
||||
import numpy
|
||||
__doctests__ = ['scrapy.contrib.ibl.extractors']
|
||||
except ImportError:
|
||||
pass
|
||||
|
|
@ -1,139 +0,0 @@
|
|||
"""
|
||||
htmlpage.py tests
|
||||
"""
|
||||
import os
|
||||
from unittest import TestCase
|
||||
|
||||
from scrapy.utils.py26 import json
|
||||
from scrapy.tests.test_contrib_ibl import path
|
||||
from scrapy.contrib.ibl.htmlpage import parse_html, HtmlTag, HtmlDataFragment
|
||||
from scrapy.tests.test_contrib_ibl.test_htmlpage_data import *
|
||||
from scrapy.utils.python import unicode_to_str, str_to_unicode
|
||||
|
||||
SAMPLES_FILE_PREFIX = os.path.join(path, "samples/samples_htmlpage")
|
||||
|
||||
def _encode_element(el):
|
||||
"""
|
||||
jsonize parse element
|
||||
"""
|
||||
if isinstance(el, HtmlTag):
|
||||
return {"tag": el.tag, "attributes": el.attributes,
|
||||
"start": el.start, "end": el.end, "tag_type": el.tag_type}
|
||||
if isinstance(el, HtmlDataFragment):
|
||||
return {"start": el.start, "end": el.end}
|
||||
raise TypeError
|
||||
|
||||
def _decode_element(dct):
|
||||
"""
|
||||
dejsonize parse element
|
||||
"""
|
||||
if "tag" in dct:
|
||||
return HtmlTag(dct["tag_type"], dct["tag"], \
|
||||
dct["attributes"], dct["start"], dct["end"])
|
||||
if "start" in dct:
|
||||
return HtmlDataFragment(dct["start"], dct["end"])
|
||||
return dct
|
||||
|
||||
def add_sample(source):
|
||||
"""
|
||||
Method for adding samples to test samples file
|
||||
(use from console)
|
||||
"""
|
||||
count = 0
|
||||
while os.path.exists("%s_%d.json" % (SAMPLES_FILE_PREFIX, count)):
|
||||
count += 1
|
||||
|
||||
open("%s_%d.html" % (SAMPLES_FILE_PREFIX, count), "wb").write(unicode_to_str(source))
|
||||
parsed = list(parse_html(source))
|
||||
open("%s_%d.json" % (SAMPLES_FILE_PREFIX, count), "wb")\
|
||||
.write(json.dumps(parsed, default=_encode_element, indent=8))
|
||||
|
||||
class TestParseHtml(TestCase):
|
||||
"""Test for parse_html"""
|
||||
def _test_sample(self, source, expected_parsed, samplecount=None):
|
||||
parsed = parse_html(source)
|
||||
count_element = 0
|
||||
count_expected = 0
|
||||
for element in parsed:
|
||||
if type(element) == HtmlTag:
|
||||
count_element += 1
|
||||
expected = expected_parsed.pop(0)
|
||||
if type(expected) == HtmlTag:
|
||||
count_expected += 1
|
||||
element_text = source[element.start:element.end]
|
||||
expected_text = source[expected.start:expected.end]
|
||||
if element.start != expected.start or element.end != expected.end:
|
||||
errstring = "[%s,%s] %s != [%s,%s] %s" % (element.start, \
|
||||
element.end, element_text, expected.start, \
|
||||
expected.end, expected_text)
|
||||
if samplecount is not None:
|
||||
errstring += " (sample %d)" % samplecount
|
||||
assert False, errstring
|
||||
if type(element) != type(expected):
|
||||
errstring = "(%s) %s != (%s) %s for text\n%s" % (count_element, \
|
||||
repr(type(element)), count_expected, repr(type(expected)), element_text)
|
||||
if samplecount is not None:
|
||||
errstring += " (sample %d)" % samplecount
|
||||
assert False, errstring
|
||||
if type(element) == HtmlTag:
|
||||
self.assertEqual(element.tag, expected.tag)
|
||||
self.assertEqual(element.attributes, expected.attributes)
|
||||
self.assertEqual(element.tag_type, expected.tag_type)
|
||||
if expected_parsed:
|
||||
errstring = "Expected %s" % repr(expected_parsed)
|
||||
if samplecount is not None:
|
||||
errstring += " (sample %d)" % samplecount
|
||||
assert False, errstring
|
||||
|
||||
def test_parse(self):
|
||||
"""simple parse_html test"""
|
||||
parsed = [_decode_element(d) for d in PARSED]
|
||||
sample = {"source": PAGE, "parsed": parsed}
|
||||
self._test_sample(PAGE, parsed)
|
||||
|
||||
def test_site_samples(self):
|
||||
"""test parse_html from real cases"""
|
||||
count = 0
|
||||
fname = "%s_%d.json" % (SAMPLES_FILE_PREFIX, count)
|
||||
while os.path.exists(fname):
|
||||
source = str_to_unicode(open("%s_%d.html" % (SAMPLES_FILE_PREFIX, count), "rb").read())
|
||||
parsed = json.loads(str_to_unicode(open(fname, "rb").read()),\
|
||||
object_hook=_decode_element)
|
||||
self._test_sample(source, parsed, count)
|
||||
count += 1
|
||||
fname = "%s_%d.json" % (SAMPLES_FILE_PREFIX, count)
|
||||
|
||||
def test_bad(self):
|
||||
"""test parsing of bad html layout"""
|
||||
parsed = [_decode_element(d) for d in PARSED2]
|
||||
self._test_sample(PAGE2, parsed)
|
||||
|
||||
def test_comments(self):
|
||||
"""test parsing of tags inside comments"""
|
||||
parsed = [_decode_element(d) for d in PARSED3]
|
||||
self._test_sample(PAGE3, parsed)
|
||||
|
||||
def test_script_text(self):
|
||||
"""test parsing of tags inside scripts"""
|
||||
parsed = [_decode_element(d) for d in PARSED4]
|
||||
self._test_sample(PAGE4, parsed)
|
||||
|
||||
def test_sucessive(self):
|
||||
"""test parsing of sucesive cleaned elements"""
|
||||
parsed = [_decode_element(d) for d in PARSED5]
|
||||
self._test_sample(PAGE5, parsed)
|
||||
|
||||
def test_sucessive2(self):
|
||||
"""test parsing of sucesive cleaned elements (variant 2)"""
|
||||
parsed = [_decode_element(d) for d in PARSED6]
|
||||
self._test_sample(PAGE6, parsed)
|
||||
|
||||
def test_special_cases(self):
|
||||
"""some special cases tests"""
|
||||
parsed = list(parse_html("<meta http-equiv='Pragma' content='no-cache' />"))
|
||||
self.assertEqual(parsed[0].attributes, {'content': 'no-cache', 'http-equiv': 'Pragma'})
|
||||
parsed = list(parse_html("<html xmlns='http://www.w3.org/1999/xhtml' xml:lang='en' lang='en'>"))
|
||||
self.assertEqual(parsed[0].attributes, {'xmlns': 'http://www.w3.org/1999/xhtml', 'xml:lang': 'en', 'lang': 'en'})
|
||||
parsed = list(parse_html("<IMG SRC='http://images.play.com/banners/SAM550a.jpg' align='left' / hspace=5>"))
|
||||
self.assertEqual(parsed[0].attributes, {'src': 'http://images.play.com/banners/SAM550a.jpg', \
|
||||
'align': 'left', 'hspace': '5', '/': None})
|
||||
|
|
@ -1,248 +0,0 @@
|
|||
PAGE = u"""
|
||||
<style id="scrapy-style" type="text/css">@import url(http://localhost:8000/as/site_media/clean.css);
|
||||
</style>
|
||||
<body>
|
||||
<div class="scrapy-selected" id="header">
|
||||
<img src="company_logo.jpg" style="margin-left: 68px; padding-top:5px;" alt="Logo" width="530" height="105">
|
||||
<div id="vertrule">
|
||||
<h1>COMPANY - <ins data-scrapy-annotate="{"variant": "0", "generated": true, "annotations": {"content": "title"}}">Item Title</ins></h1>
|
||||
<p>introduction</p>
|
||||
<div>
|
||||
<img src="/upload/img.jpg" classid=""
|
||||
data-scrapy-annotate="{"variant": "0", "annotations": {"image_url": "src"}}"
|
||||
>
|
||||
<p classid="" data-scrapy-annotate="{"variant": "0", "annotations": {"content": "description"}}"
|
||||
>
|
||||
This is such a nice item<br/> Everybody likes it.
|
||||
</p>
|
||||
<br></br>
|
||||
</div>
|
||||
<p data-scrapy-annotate="{"variant": "0", "annotations": {"content": "features"}}"
|
||||
class="" >Power: 50W</p>
|
||||
<!-- A comment --!>
|
||||
<ul data-scrapy-replacement='select' class='product'>
|
||||
<li data-scrapy-replacement='option'>Small</li>
|
||||
<li data-scrapy-replacement='option'>Big</li>
|
||||
</ul>
|
||||
<p>click here for other items</p>
|
||||
<h3>Louis Chair</h3>
|
||||
<table class="rulet" width="420" cellpadding="0" cellspacing="0"><tbody>
|
||||
<tr><td>Height</td>
|
||||
<td><ins data-scrapy-annotate="{"variant": "0", "generated": true, "annotations": {"content": "price"}}">32.00</ins></td>
|
||||
</tr><tbody></table>
|
||||
<p onmouseover='xxx' class= style="my style">
|
||||
"""
|
||||
|
||||
PARSED = [
|
||||
{'start': 0, 'end': 1},
|
||||
{'attributes': {'type': 'text/css', 'id': 'scrapy-style'}, 'tag': 'style', 'end': 42, 'start': 1, 'tag_type': 1},
|
||||
{'start': 42, 'end': 129},
|
||||
{'attributes': {}, 'tag': 'style', 'end': 137, 'start': 129, 'tag_type': 2},
|
||||
{'start': 137, 'end': 138},
|
||||
{'attributes': {}, 'tag': 'body', 'end': 144, 'start': 138, 'tag_type': 1},
|
||||
{'start': 144, 'end': 145},
|
||||
{'attributes': {'class': 'scrapy-selected', 'id': 'header'}, 'tag': 'div', 'end': 186, 'start': 145, 'tag_type': 1},
|
||||
{'start': 186, 'end': 187},
|
||||
{'attributes': {'src': 'company_logo.jpg', 'style': 'margin-left: 68px; padding-top:5px;', 'width': '530', 'alt': 'Logo', 'height': '105'}, 'tag': 'img', 'end': 295, 'start': 187, 'tag_type': 1},
|
||||
{'start': 295, 'end': 296},
|
||||
{'attributes': {'id': 'vertrule'}, 'tag': 'div', 'end': 315, 'start': 296, 'tag_type': 1},
|
||||
{'start': 315, 'end': 316},
|
||||
{'attributes': {}, 'tag': 'h1', 'end': 320, 'start': 316, 'tag_type': 1},
|
||||
{'start': 320, 'end': 330},
|
||||
{'attributes': {'data-scrapy-annotate': '{"variant": "0", "generated": true, "annotations": {"content": "title"}}'}, 'tag': 'ins', 'end': 491, 'start': 330, 'tag_type': 1},
|
||||
{'start': 491, 'end': 501},
|
||||
{'attributes': {}, 'tag': 'ins', 'end': 507, 'start': 501, 'tag_type': 2},
|
||||
{'attributes': {}, 'tag': 'h1', 'end': 512, 'start': 507, 'tag_type': 2},
|
||||
{'start': 512, 'end': 513},
|
||||
{'attributes': {}, 'tag': 'p', 'end': 516, 'start': 513, 'tag_type': 1},
|
||||
{'start': 516, 'end': 528},
|
||||
{'attributes': {}, 'tag': 'p', 'end': 532, 'start': 528, 'tag_type': 2},
|
||||
{'start': 532, 'end': 533},
|
||||
{'attributes': {}, 'tag': 'div', 'end': 538, 'start': 533, 'tag_type': 1},
|
||||
{'start': 538, 'end': 539},
|
||||
{'attributes': {'classid': None, 'src': '/upload/img.jpg', 'data-scrapy-annotate': '{"variant": "0", "annotations": {"image_url": "src"}}'}, 'tag': 'img', 'end': 709, 'start': 539, 'tag_type': 1},
|
||||
{'start': 709, 'end': 710},
|
||||
{'attributes': {'classid': None, 'data-scrapy-annotate': '{"variant": "0", "annotations": {"content": "description"}}'}, 'tag': 'p', 'end': 858, 'start': 710, 'tag_type': 1},
|
||||
{'start': 858, 'end': 883},
|
||||
{'attributes': {}, 'tag': 'br', 'end': 888, 'start': 883, 'tag_type': 3},
|
||||
{'start': 888, 'end': 909},
|
||||
{'attributes': {}, 'tag': 'p', 'end': 913, 'start': 909, 'tag_type': 2},
|
||||
{'start': 913, 'end': 914},
|
||||
{'attributes': {}, 'tag': 'br', 'end': 918, 'start': 914, 'tag_type': 1},
|
||||
{'attributes': {}, 'tag': 'br', 'end': 923, 'start': 918, 'tag_type': 2},
|
||||
{'start': 923, 'end': 924},
|
||||
{'attributes': {}, 'tag': 'div', 'end': 930, 'start': 924, 'tag_type': 2},
|
||||
{'start': 930, 'end': 931},
|
||||
{'attributes': {'data-scrapy-annotate': '{"variant": "0", "annotations": {"content": "features"}}', 'class': None}, 'tag': 'p', 'end': 1074, 'start': 931, 'tag_type': 1},
|
||||
{'start': 1074, 'end': 1084},
|
||||
{'attributes': {}, 'tag': 'p', 'end': 1088, 'start': 1084, 'tag_type': 2},
|
||||
{'start': 1088, 'end': 1109},
|
||||
{'attributes': {'data-scrapy-replacement': 'select', 'class': 'product'}, 'tag': 'ul', 'end': 1162, 'start': 1109, 'tag_type': 1},
|
||||
{'start': 1162, 'end': 1163},
|
||||
{'attributes': {'data-scrapy-replacement': 'option'}, 'tag': 'li', 'end': 1200, 'start': 1163, 'tag_type': 1},
|
||||
{'start': 1200, 'end': 1205},
|
||||
{'attributes': {}, 'tag': 'li', 'end': 1210, 'start': 1205, 'tag_type': 2},
|
||||
{'start': 1210, 'end': 1211},
|
||||
{'attributes': {'data-scrapy-replacement': 'option'}, 'tag': 'li', 'end': 1248, 'start': 1211, 'tag_type': 1},
|
||||
{'start': 1248, 'end': 1251},
|
||||
{'attributes': {}, 'tag': 'li', 'end': 1256, 'start': 1251, 'tag_type': 2},
|
||||
{'start': 1256, 'end': 1257},
|
||||
{'attributes': {}, 'tag': 'ul', 'end': 1262, 'start': 1257, 'tag_type': 2},
|
||||
{'start': 1262, 'end': 1263},
|
||||
{'attributes': {}, 'tag': 'p', 'end': 1266, 'start': 1263, 'tag_type': 1},
|
||||
{'start': 1266, 'end': 1292},
|
||||
{'attributes': {}, 'tag': 'p', 'end': 1296, 'start': 1292, 'tag_type': 2},
|
||||
{'start': 1296, 'end': 1297},
|
||||
{'attributes': {}, 'tag': 'h3', 'end': 1301, 'start': 1297, 'tag_type': 1},
|
||||
{'start': 1301, 'end': 1312},
|
||||
{'attributes': {}, 'tag': 'h3', 'end': 1317, 'start': 1312, 'tag_type': 2},
|
||||
{'start': 1317, 'end': 1318},
|
||||
{'attributes': {'cellpadding': '0', 'width': '420', 'cellspacing': '0', 'class': 'rulet'}, 'tag': 'table', 'end': 1383, 'start': 1318, 'tag_type': 1},
|
||||
{'attributes': {}, 'tag': 'tbody', 'end': 1390, 'start': 1383, 'tag_type': 1},
|
||||
{'start': 1390, 'end': 1391},
|
||||
{'attributes': {}, 'tag': 'tr', 'end': 1395, 'start': 1391, 'tag_type': 1},
|
||||
{'attributes': {}, 'tag': 'td', 'end': 1399, 'start': 1395, 'tag_type': 1},
|
||||
{'start': 1399, 'end': 1405},
|
||||
{'attributes': {}, 'tag': 'td', 'end': 1410, 'start': 1405, 'tag_type': 2},
|
||||
{'start': 1410, 'end': 1411},
|
||||
{'attributes': {}, 'tag': 'td', 'end': 1415, 'start': 1411, 'tag_type': 1},
|
||||
{'attributes': {'data-scrapy-annotate': '{"variant": "0", "generated": true, "annotations": {"content": "price"}}'}, 'tag': 'ins', 'end': 1576, 'start': 1415, 'tag_type': 1},
|
||||
{'start': 1576, 'end': 1581},
|
||||
{'attributes': {}, 'tag': 'ins', 'end': 1587, 'start': 1581, 'tag_type': 2},
|
||||
{'attributes': {}, 'tag': 'td', 'end': 1592, 'start': 1587, 'tag_type': 2},
|
||||
{'start': 1592, 'end': 1593},
|
||||
{'attributes': {}, 'tag': 'tr', 'end': 1598, 'start': 1593, 'tag_type': 2},
|
||||
{'attributes': {}, 'tag': 'tbody', 'end': 1605, 'start': 1598, 'tag_type': 1},
|
||||
{'attributes': {}, 'tag': 'table', 'end': 1613, 'start': 1605, 'tag_type': 2},
|
||||
{'start': 1613, 'end': 1614},
|
||||
{'attributes': {'style': 'my style', 'onmouseover': 'xxx', 'class': None}, 'tag': 'p', 'end': 1659, 'start': 1614, 'tag_type': 1},
|
||||
{'start': 1659, 'end': 1660},
|
||||
]
|
||||
|
||||
# for testing parsing of some invalid html code (but still managed by browsers)
|
||||
PAGE2 = u"""
|
||||
<html>
|
||||
<body>
|
||||
<p class="MsoNormal" style="margin: 0cm 0cm 0pt"><span lang="EN-GB">
|
||||
Hello world!
|
||||
</span>
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
PARSED2 = [
|
||||
{'end': 1, 'start': 0},
|
||||
{'attributes': {}, 'end': 7, 'start': 1, 'tag': u'html', 'tag_type': 1},
|
||||
{'end': 8, 'start': 7},
|
||||
{'attributes': {}, 'end': 14, 'start': 8, 'tag': u'body', 'tag_type': 1},
|
||||
{'end': 15, 'start': 14},
|
||||
{'attributes': {u'style': u'"margin:', u'0pt"': None, u'class': u'"MsoNormal"', u'0cm': None}, 'end': 80, 'start': 15, 'tag': u'p', 'tag_type': 1},
|
||||
{'attributes': {u'lang': u'"EN-GB"'}, 'end': 107, 'start': 80, 'tag': u'span', 'tag_type': 1},
|
||||
{'end': 121, 'start': 107},
|
||||
{'attributes': {}, 'end': 128, 'start': 121, 'tag': u'span', 'tag_type': 2},
|
||||
{'end': 129, 'start': 128},
|
||||
{'attributes': {}, 'end': 133, 'start': 129, 'tag': u'p', 'tag_type': 2},
|
||||
{'end': 134, 'start': 133},
|
||||
{'attributes': {}, 'end': 141, 'start': 134, 'tag': u'body', 'tag_type': 2},
|
||||
{'end': 142, 'start': 141},
|
||||
{'attributes': {}, 'end': 149, 'start': 142, 'tag': u'html', 'tag_type': 2},
|
||||
{'end': 150, 'start': 149},
|
||||
]
|
||||
|
||||
# for testing tags inside comments
|
||||
PAGE3 = u"""<html><body><h1>Helloooo!!</h1><p>Did i say hello??</p><!--<p>
|
||||
</p>--><script type="text/javascript">bla<!--comment-->blabla</script></body></html>"""
|
||||
|
||||
PARSED3 = [
|
||||
{'attributes': {}, 'end': 6, 'start': 0, 'tag': u'html', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 12, 'start': 6, 'tag': u'body', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 16, 'start': 12, 'tag': u'h1', 'tag_type': 1},
|
||||
{'end': 26, 'start': 16},
|
||||
{'attributes': {}, 'end': 31, 'start': 26, 'tag': u'h1', 'tag_type': 2},
|
||||
{'attributes': {}, 'end': 34, 'start': 31, 'tag': u'p', 'tag_type': 1},
|
||||
{'end': 51, 'start': 34},
|
||||
{'attributes': {}, 'end': 55, 'start': 51, 'tag': u'p', 'tag_type': 2},
|
||||
{'end': 70, 'start': 55},
|
||||
{'attributes': {u'type': u'text/javascript'}, 'end': 101, 'start': 70, 'tag': u'script', 'tag_type': 1},
|
||||
{'end': 104, 'start': 101},
|
||||
{'end': 118, 'start': 104},
|
||||
{'end': 124, 'start': 118},
|
||||
{'attributes': {}, 'end': 133, 'start': 124, 'tag': u'script', 'tag_type': 2},
|
||||
{'attributes': {}, 'end': 140, 'start': 133, 'tag': u'body', 'tag_type': 2},
|
||||
{'attributes': {}, 'end': 147, 'start': 140, 'tag': u'html', 'tag_type': 2}
|
||||
]
|
||||
|
||||
# for testing tags inside scripts
|
||||
PAGE4 = u"""<html><body><h1>Konnichiwa!!</h1>hello<script type="text/javascript">\
|
||||
doc.write("<img src=" + base + "product/" + productid + ">");\
|
||||
</script>hello again</body></html>"""
|
||||
|
||||
PARSED4 = [
|
||||
{'attributes': {}, 'end': 6, 'start': 0, 'tag': u'html', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 12, 'start': 6, 'tag': u'body', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 16, 'start': 12, 'tag': u'h1', 'tag_type': 1},
|
||||
{'end': 28,'start': 16},
|
||||
{'attributes': {}, 'end': 33, 'start': 28, 'tag': u'h1', 'tag_type': 2},
|
||||
{'end': 38, 'start': 33},
|
||||
{'attributes': {u'type': u'text/javascript'}, 'end': 69, 'start': 38, 'tag': u'script', 'tag_type': 1},
|
||||
{'end': 130, 'start': 69},
|
||||
{'attributes': {}, 'end': 139, 'start': 130, 'tag': u'script', 'tag_type': 2},
|
||||
{'end': 150, 'start': 139},
|
||||
{'attributes': {}, 'end': 157, 'start': 150, 'tag': u'body', 'tag_type': 2},
|
||||
{'attributes': {}, 'end': 164, 'start': 157, 'tag': u'html', 'tag_type': 2},
|
||||
]
|
||||
|
||||
# Test sucessive cleaning elements
|
||||
PAGE5 = u"""<html><body><script>hello</script><script>brb</script></body><!--commentA--><!--commentB--></html>"""
|
||||
|
||||
PARSED5 = [
|
||||
{'attributes': {}, 'end': 6, 'start': 0, 'tag': u'html', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 12, 'start': 6, 'tag': u'body', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 20, 'start': 12, 'tag': u'script', 'tag_type': 1},
|
||||
{'end': 25, 'start': 20},
|
||||
{'attributes': {}, 'end': 34, 'start': 25, 'tag': u'script', 'tag_type': 2},
|
||||
{'attributes': {}, 'end': 42, 'start': 34, 'tag': u'script', 'tag_type': 1},
|
||||
{'end': 45, 'start': 42},
|
||||
{'attributes': {}, 'end': 54, 'start': 45, 'tag': u'script', 'tag_type': 2},
|
||||
{'attributes': {}, 'end': 61, 'start': 54, 'tag': u'body', 'tag_type': 2},
|
||||
{'end': 76, 'start': 61},
|
||||
{'end': 91, 'start': 76},
|
||||
{'attributes': {}, 'end': 98, 'start': 91, 'tag': u'html', 'tag_type': 2},
|
||||
]
|
||||
|
||||
# Test sucessive cleaning elements variant 2
|
||||
PAGE6 = u"""<html><body><script>pss<!--comment-->pss</script>all<script>brb</script>\n\n</body></html>"""
|
||||
|
||||
PARSED6 = [
|
||||
{'attributes': {}, 'end': 6, 'start': 0, 'tag': u'html', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 12, 'start': 6, 'tag': u'body', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 20, 'start': 12, 'tag': u'script', 'tag_type': 1},
|
||||
{'end': 23, 'start': 20},
|
||||
{'end': 37, 'start': 23},
|
||||
{'end': 40, 'start': 37},
|
||||
{'attributes': {}, 'end': 49, 'start': 40, 'tag': u'script', 'tag_type': 2},
|
||||
{'end': 52, 'start': 49},
|
||||
{'attributes': {}, 'end': 60, 'start': 52, 'tag': u'script', 'tag_type': 1},
|
||||
{'end': 63, 'start': 60},
|
||||
{'attributes': {}, 'end': 72, 'start': 63, 'tag': u'script', 'tag_type': 2},
|
||||
{'end': 74, 'start': 72},
|
||||
{'attributes': {}, 'end': 81, 'start': 74, 'tag': u'body', 'tag_type': 2},
|
||||
{'attributes': {}, 'end': 88, 'start': 81, 'tag': u'html', 'tag_type': 2},
|
||||
]
|
||||
|
||||
# Test source without ending body nor html
|
||||
PAGE7 = u"""<html><body><p>veris in temporibus sub aprilis idibus</p><script>script code</script><!--comment-->"""
|
||||
|
||||
PARSED7 = [
|
||||
{'attributes' : {}, 'end': 6, 'start': 0, 'tag': u'html', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 12, 'start': 6, 'tag': u'body', 'tag_type': 1},
|
||||
{'attributes': {}, 'end': 15, 'start': 12, 'tag': u'p', 'tag_type': 1},
|
||||
{'end': 53, 'start': 15},
|
||||
{'attributes': {}, 'end': 57, 'start': 53, 'tag': u'p', 'tag_type': 2},
|
||||
{'attributes' : {}, 'end': 65, 'start': 57, 'tag': u'script', 'tag_type': 1},
|
||||
{'end': 76, 'start': 65},
|
||||
{'attributes' : {}, 'end': 85, 'start': 76, 'tag': u'script', 'tag_type': 2},
|
||||
{'end': 99, 'start': 85},
|
||||
]
|
||||
|
||||
|
|
@ -1,330 +0,0 @@
|
|||
"""
|
||||
Unit tests for pageparsing
|
||||
"""
|
||||
import os
|
||||
from cStringIO import StringIO
|
||||
|
||||
from twisted.trial.unittest import TestCase, SkipTest
|
||||
from scrapy.utils.python import str_to_unicode
|
||||
from scrapy.utils.py26 import json
|
||||
|
||||
from scrapy.contrib.ibl.htmlpage import HtmlPage
|
||||
from scrapy.tests.test_contrib_ibl import path
|
||||
|
||||
try:
|
||||
import numpy
|
||||
except ImportError:
|
||||
numpy = None
|
||||
|
||||
if numpy:
|
||||
from scrapy.contrib.ibl.extraction.pageparsing import (
|
||||
InstanceLearningParser, TemplatePageParser, ExtractionPageParser)
|
||||
from scrapy.contrib.ibl.extraction.pageobjects import TokenDict, TokenType
|
||||
|
||||
|
||||
SIMPLE_PAGE = u"""
|
||||
<html> <p some-attr="foo">this is a test</p> </html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE1 = u"""
|
||||
<html>
|
||||
<h1 data-scrapy-annotate="{"variant": 0, "annotations": {"content": "name"}}">Some Product</h1>
|
||||
<p> some stuff</p>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
This is such a nice item<br/>
|
||||
Everybody likes it.
|
||||
</p>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}"/>
|
||||
\xa310.00
|
||||
<br/>
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "short_description"}}">
|
||||
Old fashioned product
|
||||
<p data-scrapy-annotate="{"variant": 0, "annotations": {"content": "short_description"}}">
|
||||
For exigent individuals
|
||||
<p>click here for other items</p>
|
||||
</html>
|
||||
"""
|
||||
|
||||
BROKEN_PAGE = u"""
|
||||
<html> <p class="ruleb"align="center">html parser cannot parse this</p></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE2 = u"""
|
||||
<html><body>
|
||||
<h1>A product</h1>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<p>A very nice product for all intelligent people</p>
|
||||
<div data-scrapy-ignore="true">
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
</div>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">
|
||||
\xa310.00<p data-scrapy-ignore="true"> 13 <br></p>
|
||||
</div>
|
||||
<table data-scrapy-ignore="true">
|
||||
<tr><td data-scrapy-ignore="true"></td></tr>
|
||||
<tr></tr>
|
||||
</table>
|
||||
<img data-scrapy-ignore="true" src="image2.jpg">
|
||||
<img data-scrapy-ignore="true" src="image3.jpg" />
|
||||
<img data-scrapy-ignore-beneath="true" src="image2.jpg">
|
||||
<img data-scrapy-ignore-beneath="true" src="image3.jpg" />
|
||||
</body></html>
|
||||
|
||||
"""
|
||||
|
||||
LABELLED_PAGE3 = u"""
|
||||
<html><body>
|
||||
<h1>A product</h1>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<p>A very nice product for all intelligent people</p>
|
||||
<div data-scrapy-ignore="true">
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
</div>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
\xa310.00<p data-scrapy-ignore="true"> 13 <br></p>
|
||||
<table><tr>
|
||||
<td>Description 1</td>
|
||||
<td data-scrapy-ignore-beneath="true">Description 2</td>
|
||||
<td>Description 3</td>
|
||||
<td>Description 4</td>
|
||||
</tr></table>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE4 = u"""
|
||||
<html><body>
|
||||
<h1>A product</h1>
|
||||
<div>
|
||||
<p>A very nice product for all intelligent people</p>
|
||||
<div>
|
||||
<img scr="image.jpg" /><br/><a link="back.html">Click here to go back</a>
|
||||
</div>
|
||||
</div>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
\xa310.00<p data-scrapy-ignore="true"> 13 <br></p>
|
||||
<table><tr>
|
||||
<td>Description 1</td>
|
||||
<td data-scrapy-ignore-beneath="true">Description 2</td>
|
||||
<td>Description 3</td>
|
||||
<td data-scrapy-annotate="{"variant": 0, "annotations": {"content": "price"}}">
|
||||
Price \xa310.00</td>
|
||||
</tr></table>
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE5 = u"""
|
||||
<html><body>
|
||||
<ul data-scrapy-replacement='select'>
|
||||
<li data-scrapy-replacement='option'>Option A</li>
|
||||
<li>Option I</li>
|
||||
<li data-scrapy-replacement='option'>Option B</li>
|
||||
</ul>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE6 = u"""
|
||||
<html><body>
|
||||
Text A
|
||||
<p><ins data-scrapy-annotate="{"variant": 0, "generated": true, "annotations": {"content": "price"}}">
|
||||
65.00</ins>pounds</p>
|
||||
<p>Description: <ins data-scrapy-annotate="{"variant": 0, "generated": true, "annotations": {"content": "description"}}">
|
||||
Text B</ins></p>
|
||||
Text C
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE7 = u"""
|
||||
<html><body>
|
||||
<div data-scrapy-annotate="{"variant": 0, "annotations": {"content": "description"}}">
|
||||
<ins data-scrapy-ignore="true" data-scrapy-annotate="{"variant": 0, "generated": true, "annotations": {"content": "site_id"}}">Item Id</ins>
|
||||
Description
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE8 = u"""
|
||||
<html><body>
|
||||
<div data-scrapy-annotate="{"required": ["description"], "variant": 0, "annotations": {"content": "description"}}">
|
||||
<ins data-scrapy-ignore="true" data-scrapy-annotate="{"variant": 0, "generated": true, "annotations": {"content": "site_id"}}">Item Id</ins>
|
||||
Description
|
||||
</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE9 = u"""
|
||||
<html><body>
|
||||
<img src="image.jpg" data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"src": "image_urls"}}">
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "name"}}">product 1</p>
|
||||
<b data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "price"}}">$67</b>
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "name"}}">product 2</p>
|
||||
<b data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "price"}}">$70</b>
|
||||
<div data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"content": "category"}}">tables</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
LABELLED_PAGE10 = u"""
|
||||
<html><body>
|
||||
<img src="image.jpg" data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"src": "image_urls"}}">
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "name"}}">product 1</p>
|
||||
<b data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"content": "price"}}">$67</b>
|
||||
<img src="swatch1.jpg" data-scrapy-annotate="{"required": [], "variant": 1, "annotations": {"src": "swatches"}}">
|
||||
|
||||
<p data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "name"}}">product 2</p>
|
||||
<b data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"content": "price"}}">$70</b>
|
||||
<img src="swatch2.jpg" data-scrapy-annotate="{"required": [], "variant": 2, "annotations": {"src": "swatches"}}">
|
||||
|
||||
<div data-scrapy-annotate="{"required": [], "variant": 0, "annotations": {"content": "category"}}">tables</div>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
def _parse_page(parser_class, pagetext):
|
||||
htmlpage = HtmlPage(None, {}, pagetext)
|
||||
parser = parser_class(TokenDict())
|
||||
parser.feed(htmlpage)
|
||||
return parser
|
||||
|
||||
def _tags(pp, predicate):
|
||||
return [pp.token_dict.token_string(s) for s in pp.token_list \
|
||||
if predicate(s)]
|
||||
|
||||
class TestPageParsing(TestCase):
|
||||
|
||||
def setUp(self):
|
||||
if not numpy:
|
||||
raise SkipTest("numpy not available")
|
||||
|
||||
def test_instance_parsing(self):
|
||||
pp = _parse_page(InstanceLearningParser, SIMPLE_PAGE)
|
||||
# all tags
|
||||
self.assertEqual(_tags(pp, bool), ['<html>', '<p>', '</p>', '</html>'])
|
||||
|
||||
# open/closing tag handling
|
||||
openp = lambda x: pp.token_dict.token_type(x) == TokenType.OPEN_TAG
|
||||
self.assertEqual(_tags(pp, openp), ['<html>', '<p>'])
|
||||
closep = lambda x: pp.token_dict.token_type(x) == TokenType.CLOSE_TAG
|
||||
self.assertEqual(_tags(pp, closep), ['</p>', '</html>'])
|
||||
|
||||
def _validate_annotation(self, parser, lable_region, name, start_tag, end_tag):
|
||||
assert lable_region.surrounds_attribute == name
|
||||
start_token = parser.token_list[lable_region.start_index]
|
||||
assert parser.token_dict.token_string(start_token) == start_tag
|
||||
end_token = parser.token_list[lable_region.end_index]
|
||||
assert parser.token_dict.token_string(end_token) == end_tag
|
||||
|
||||
def test_template_parsing(self):
|
||||
lp = _parse_page(TemplatePageParser, LABELLED_PAGE1)
|
||||
self.assertEqual(len(lp.annotations), 5)
|
||||
self._validate_annotation(lp, lp.annotations[0],
|
||||
'name', '<h1>', '</h1>')
|
||||
|
||||
# all tags were closed
|
||||
self.assertEqual(len(lp.labelled_tag_stacks), 0)
|
||||
|
||||
def test_extraction_page_parsing(self):
|
||||
epp = _parse_page(ExtractionPageParser, SIMPLE_PAGE)
|
||||
ep = epp.to_extraction_page()
|
||||
assert len(ep.page_tokens) == 4
|
||||
assert ep.token_html(0) == '<html>'
|
||||
assert ep.token_html(1) == '<p some-attr="foo">'
|
||||
|
||||
assert ep.html_between_tokens(1, 2) == 'this is a test'
|
||||
assert ep.html_between_tokens(1, 3) == 'this is a test</p> '
|
||||
|
||||
def test_invalid_html(self):
|
||||
p = _parse_page(InstanceLearningParser, BROKEN_PAGE)
|
||||
assert p
|
||||
|
||||
def test_ignore_region(self):
|
||||
"""Test ignored regions"""
|
||||
p = _parse_page(TemplatePageParser, LABELLED_PAGE2)
|
||||
self.assertEqual(p.ignored_regions, [(7,12),(15,17),(19,26),(21,22),(27,28),(28,29),(29,None),(30,None)])
|
||||
self.assertEqual(len(p.ignored_tag_stacks), 0)
|
||||
|
||||
def test_ignore_regions2(self):
|
||||
"""Test ignore-beneath regions"""
|
||||
p = _parse_page(TemplatePageParser, LABELLED_PAGE3)
|
||||
self.assertEqual(p.ignored_regions, [(7,12),(15,17),(22,None)])
|
||||
self.assertEqual(len(p.ignored_tag_stacks), 0)
|
||||
|
||||
def test_ignore_regions3(self):
|
||||
"""Test ignore-beneath with annotation inside region"""
|
||||
p = _parse_page(TemplatePageParser, LABELLED_PAGE4)
|
||||
self.assertEqual(p.ignored_regions, [(15,17),(22,None)])
|
||||
self.assertEqual(len(p.ignored_tag_stacks), 0)
|
||||
|
||||
def test_replacement(self):
|
||||
"""Test parsing of replacement tags"""
|
||||
p = _parse_page(TemplatePageParser, LABELLED_PAGE5)
|
||||
self.assertEqual(_tags(p, bool), ['<html>', '<body>', '<select>', '<option>',
|
||||
'</option>', '<li>', '</li>', '<option>', '</option>', '</select>', '</body>', '</html>'])
|
||||
|
||||
def test_partial(self):
|
||||
"""Test partial annotation parsing"""
|
||||
p = _parse_page(TemplatePageParser, LABELLED_PAGE6)
|
||||
text = p.annotations[0].annotation_text
|
||||
self.assertEqual(text.start_text, '')
|
||||
self.assertEqual(text.follow_text, 'pounds')
|
||||
text = p.annotations[1].annotation_text
|
||||
self.assertEqual(text.start_text, "Description: ")
|
||||
self.assertEqual(text.follow_text, '')
|
||||
|
||||
def test_ignored_partial(self):
|
||||
"""Test ignored region declared on partial annotation"""
|
||||
p = _parse_page(TemplatePageParser, LABELLED_PAGE7)
|
||||
self.assertEqual(p.ignored_regions, [(2, 3)])
|
||||
|
||||
def test_extra_required(self):
|
||||
"""Test parsing of extra required attributes"""
|
||||
p = _parse_page(TemplatePageParser, LABELLED_PAGE8)
|
||||
self.assertEqual(p.extra_required_attrs, ["description"])
|
||||
|
||||
def test_variants(self):
|
||||
"""Test parsing of variant annotations"""
|
||||
annotations = _parse_page(TemplatePageParser, LABELLED_PAGE9).annotations
|
||||
self.assertEqual(annotations[0].variant_id, None)
|
||||
self.assertEqual(annotations[1].variant_id, 1)
|
||||
self.assertEqual(annotations[2].variant_id, 1)
|
||||
self.assertEqual(annotations[3].variant_id, 2)
|
||||
self.assertEqual(annotations[4].variant_id, 2)
|
||||
self.assertEqual(annotations[5].variant_id, None)
|
||||
|
||||
def test_variants_in_attributes(self):
|
||||
"""Test parsing of variant annotations in attributes"""
|
||||
annotations = _parse_page(TemplatePageParser, LABELLED_PAGE10).annotations
|
||||
self.assertEqual(annotations[0].variant_id, None)
|
||||
self.assertEqual(annotations[1].variant_id, 1)
|
||||
self.assertEqual(annotations[2].variant_id, 1)
|
||||
self.assertEqual(annotations[3].variant_id, 1)
|
||||
self.assertEqual(annotations[4].variant_id, 2)
|
||||
self.assertEqual(annotations[5].variant_id, 2)
|
||||
self.assertEqual(annotations[6].variant_id, 2)
|
||||
self.assertEqual(annotations[7].variant_id, None)
|
||||
|
||||
def test_site_pages(self):
|
||||
"""
|
||||
Tests from real pages. More reliable and easy to build for more complicated structures
|
||||
"""
|
||||
SAMPLES_FILE_PREFIX = os.path.join(path, "samples/samples_pageparsing")
|
||||
count = 0
|
||||
fname = "%s_%d.json" % (SAMPLES_FILE_PREFIX, count)
|
||||
while os.path.exists(fname):
|
||||
source = str_to_unicode(open("%s_%d.html" % (SAMPLES_FILE_PREFIX, count), "rb").read())
|
||||
annotations = json.loads(str_to_unicode(open(fname, "rb").read()))
|
||||
template = HtmlPage(body=source)
|
||||
parser = TemplatePageParser(TokenDict())
|
||||
parser.feed(template)
|
||||
for annotation in parser.annotations:
|
||||
test_annotation = annotations.pop(0)
|
||||
for s in annotation.__slots__:
|
||||
if s == "tag_attributes":
|
||||
for pair in getattr(annotation, s):
|
||||
self.assertEqual(list(pair), test_annotation[s].pop(0))
|
||||
else:
|
||||
self.assertEqual(getattr(annotation, s), test_annotation[s])
|
||||
self.assertEqual(annotations, [])
|
||||
count += 1
|
||||
fname = "%s_%d.json" % (SAMPLES_FILE_PREFIX, count)
|
||||
|
|
@ -8,6 +8,7 @@ from twisted.web import server, static, util, resource
|
|||
from twisted.web.test.test_webclient import ForeverTakingResource, \
|
||||
NoLengthResource, HostHeaderResource, \
|
||||
PayloadResource, BrokenDownloadResource
|
||||
from w3lib.url import path_to_file_uri
|
||||
|
||||
from scrapy.core.downloader.webclient import PartialDownloadError
|
||||
from scrapy.core.downloader.handlers.file import FileDownloadHandler
|
||||
|
|
@ -15,7 +16,6 @@ from scrapy.core.downloader.handlers.http import HttpDownloadHandler
|
|||
from scrapy.core.downloader.handlers.s3 import S3DownloadHandler
|
||||
from scrapy.spider import BaseSpider
|
||||
from scrapy.http import Request
|
||||
from scrapy.utils.url import path_to_file_uri
|
||||
from scrapy import optional_features
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,28 @@
|
|||
import unittest
|
||||
|
||||
from scrapy.link import Link
|
||||
|
||||
class LinkTest(unittest.TestCase):
|
||||
|
||||
def test_eq_and_hash(self):
|
||||
l1 = Link("http://www.example.com")
|
||||
l2 = Link("http://www.example.com/other")
|
||||
l3 = Link("http://www.example.com")
|
||||
|
||||
self.assertEqual(l1, l1)
|
||||
self.assertEqual(hash(l1), hash(l1))
|
||||
self.assertNotEqual(l1, l2)
|
||||
self.assertNotEqual(hash(l1), hash(l2))
|
||||
self.assertEqual(l1, l3)
|
||||
self.assertEqual(hash(l1), hash(l3))
|
||||
|
||||
l4 = Link("http://www.example.com", text="test")
|
||||
l5 = Link("http://www.example.com", text="test2")
|
||||
l6 = Link("http://www.example.com", text="test")
|
||||
|
||||
self.assertEqual(l4, l4)
|
||||
self.assertEqual(hash(l4), hash(l4))
|
||||
self.assertNotEqual(l4, l5)
|
||||
self.assertNotEqual(hash(l4), hash(l5))
|
||||
self.assertEqual(l4, l6)
|
||||
self.assertEqual(hash(l4), hash(l6))
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
import unittest
|
||||
from scrapy.utils.http import basic_auth_header
|
||||
|
||||
__doctests__ = ['scrapy.utils.http']
|
||||
|
||||
class UtilsHttpTestCase(unittest.TestCase):
|
||||
|
||||
def test_basic_auth_header(self):
|
||||
self.assertEqual('Basic c29tZXVzZXI6c29tZXBhc3M=',
|
||||
basic_auth_header('someuser', 'somepass'))
|
||||
# Check url unsafe encoded header
|
||||
self.assertEqual('Basic c29tZXVzZXI6QDx5dTk-Jm8_UQ==',
|
||||
basic_auth_header('someuser', '@<yu9>&o?Q'))
|
||||
|
|
@ -29,12 +29,12 @@ class XmliterTestCase(unittest.TestCase):
|
|||
for x in self.xmliter(response, 'product'):
|
||||
attrs.append((x.select("@id").extract(), x.select("name/text()").extract(), x.select("./type/text()").extract()))
|
||||
|
||||
self.assertEqual(attrs,
|
||||
self.assertEqual(attrs,
|
||||
[(['001'], ['Name 1'], ['Type 1']), (['002'], ['Name 2'], ['Type 2'])])
|
||||
|
||||
def test_xmliter_text(self):
|
||||
body = u"""<?xml version="1.0" encoding="UTF-8"?><products><product>one</product><product>two</product></products>"""
|
||||
|
||||
|
||||
self.assertEqual([x.select("text()").extract() for x in self.xmliter(body, 'product')],
|
||||
[[u'one'], [u'two']])
|
||||
|
||||
|
|
@ -74,7 +74,7 @@ class XmliterTestCase(unittest.TestCase):
|
|||
|
||||
def test_xmliter_exception(self):
|
||||
body = u"""<?xml version="1.0" encoding="UTF-8"?><products><product>one</product><product>two</product></products>"""
|
||||
|
||||
|
||||
iter = self.xmliter(body, 'product')
|
||||
iter.next()
|
||||
iter.next()
|
||||
|
|
@ -97,6 +97,35 @@ class LxmlXmliterTestCase(XmliterTestCase):
|
|||
except ImportError:
|
||||
skip = "lxml not available"
|
||||
|
||||
def test_xmliter_iterate_namespace(self):
|
||||
body = """\
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0" xmlns="http://base.google.com/ns/1.0">
|
||||
<channel>
|
||||
<title>My Dummy Company</title>
|
||||
<link>http://www.mydummycompany.com</link>
|
||||
<description>This is a dummy company. We do nothing.</description>
|
||||
<item>
|
||||
<title>Item 1</title>
|
||||
<description>This is item 1</description>
|
||||
<link>http://www.mydummycompany.com/items/1</link>
|
||||
<image_link>http://www.mydummycompany.com/images/item1.jpg</image_link>
|
||||
<image_link>http://www.mydummycompany.com/images/item2.jpg</image_link>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
"""
|
||||
response = XmlResponse(url='http://mydummycompany.com', body=body)
|
||||
|
||||
no_namespace_iter = self.xmliter(response, 'image_link')
|
||||
self.assertEqual(len(list(no_namespace_iter)), 0)
|
||||
|
||||
namespace_iter = self.xmliter(response, 'image_link', 'http://base.google.com/ns/1.0')
|
||||
node = namespace_iter.next()
|
||||
self.assertEqual(node.select('text()').extract(), ['http://www.mydummycompany.com/images/item1.jpg'])
|
||||
node = namespace_iter.next()
|
||||
self.assertEqual(node.select('text()').extract(), ['http://www.mydummycompany.com/images/item2.jpg'])
|
||||
|
||||
|
||||
class UtilsCsvTestCase(unittest.TestCase):
|
||||
sample_feeds_dir = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'sample_data', 'feeds')
|
||||
|
|
|
|||
|
|
@ -1,156 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
import unittest
|
||||
|
||||
from scrapy.utils.markup import remove_entities, replace_tags, remove_comments
|
||||
from scrapy.utils.markup import remove_tags_with_content, replace_escape_chars, remove_tags
|
||||
from scrapy.utils.markup import unquote_markup
|
||||
|
||||
class UtilsMarkupTest(unittest.TestCase):
|
||||
|
||||
def test_remove_entities(self):
|
||||
# make sure it always return uncode
|
||||
assert isinstance(remove_entities('no entities'), unicode)
|
||||
assert isinstance(remove_entities('Price: £100!'), unicode)
|
||||
|
||||
# regular conversions
|
||||
self.assertEqual(remove_entities(u'As low as £100!'),
|
||||
u'As low as \xa3100!')
|
||||
self.assertEqual(remove_entities('As low as £100!'),
|
||||
u'As low as \xa3100!')
|
||||
self.assertEqual(remove_entities('redirectTo=search&searchtext=MR0221Y&aff=buyat&affsrc=d_data&cm_mmc=buyat-_-ELECTRICAL & SEASONAL-_-MR0221Y-_-9-carat gold ½oz solid crucifix pendant'),
|
||||
u'redirectTo=search&searchtext=MR0221Y&aff=buyat&affsrc=d_data&cm_mmc=buyat-_-ELECTRICAL & SEASONAL-_-MR0221Y-_-9-carat gold \xbdoz solid crucifix pendant')
|
||||
# keep some entities
|
||||
self.assertEqual(remove_entities('<b>Low < High & Medium £ six</b>', keep=['lt', 'amp']),
|
||||
u'<b>Low < High & Medium \xa3 six</b>')
|
||||
|
||||
# illegal entities
|
||||
self.assertEqual(remove_entities('a < b &illegal; c � six', remove_illegal=False),
|
||||
u'a < b &illegal; c � six')
|
||||
self.assertEqual(remove_entities('a < b &illegal; c � six', remove_illegal=True),
|
||||
u'a < b c six')
|
||||
self.assertEqual(remove_entities('x≤y'), u'x\u2264y')
|
||||
|
||||
# check browser hack for numeric character references in the 80-9F range
|
||||
self.assertEqual(remove_entities('x™y', encoding='cp1252'), u'x\u2122y')
|
||||
|
||||
# encoding
|
||||
self.assertEqual(remove_entities('x\x99™™y', encoding='cp1252'), \
|
||||
u'x\u2122\u2122\u2122y')
|
||||
|
||||
def test_replace_tags(self):
|
||||
# make sure it always return uncode
|
||||
assert isinstance(replace_tags('no entities'), unicode)
|
||||
|
||||
self.assertEqual(replace_tags(u'This text contains <a>some tag</a>'),
|
||||
u'This text contains some tag')
|
||||
|
||||
self.assertEqual(replace_tags('This text is very im<b>port</b>ant', ' '),
|
||||
u'This text is very im port ant')
|
||||
|
||||
# multiline tags
|
||||
self.assertEqual(replace_tags('Click <a class="one"\r\n href="url">here</a>'),
|
||||
u'Click here')
|
||||
|
||||
def test_remove_comments(self):
|
||||
# make sure it always return unicode
|
||||
assert isinstance(remove_comments('without comments'), unicode)
|
||||
assert isinstance(remove_comments('<!-- with comments -->'), unicode)
|
||||
|
||||
# text without comments
|
||||
self.assertEqual(remove_comments(u'text without comments'), u'text without comments')
|
||||
|
||||
# text with comments
|
||||
self.assertEqual(remove_comments(u'<!--text with comments-->'), u'')
|
||||
self.assertEqual(remove_comments(u'Hello<!--World-->'),u'Hello')
|
||||
|
||||
def test_remove_tags(self):
|
||||
# make sure it always return unicode
|
||||
assert isinstance(remove_tags('no tags'), unicode)
|
||||
assert isinstance(remove_tags('no tags', which_ones=('p',)), unicode)
|
||||
assert isinstance(remove_tags('<p>one tag</p>'), unicode)
|
||||
assert isinstance(remove_tags('<p>one tag</p>', which_ones=('p')), unicode)
|
||||
assert isinstance(remove_tags('<a>link</a>', which_ones=('b',)), unicode)
|
||||
|
||||
# text without tags
|
||||
self.assertEqual(remove_tags(u'no tags'), u'no tags')
|
||||
self.assertEqual(remove_tags(u'no tags', which_ones=('p','b',)), u'no tags')
|
||||
|
||||
# text with tags
|
||||
self.assertEqual(remove_tags(u'<p>one p tag</p>'), u'one p tag')
|
||||
self.assertEqual(remove_tags(u'<p>one p tag</p>', which_ones=('b',)), u'<p>one p tag</p>')
|
||||
|
||||
self.assertEqual(remove_tags(u'<b>not will removed</b><i>i will removed</i>', which_ones=('i',)),
|
||||
u'<b>not will removed</b>i will removed')
|
||||
|
||||
# text with tags and attributes
|
||||
self.assertEqual(remove_tags(u'<p align="center" class="one">texty</p>'), u'texty')
|
||||
self.assertEqual(remove_tags(u'<p align="center" class="one">texty</p>', which_ones=('b',)),
|
||||
u'<p align="center" class="one">texty</p>')
|
||||
|
||||
# text with empty tags
|
||||
self.assertEqual(remove_tags(u'a<br />b<br/>c'), u'abc')
|
||||
self.assertEqual(remove_tags(u'a<br />b<br/>c', which_ones=('br',)), u'abc')
|
||||
|
||||
# test keep arg
|
||||
self.assertEqual(remove_tags(u'<p>a<br />b<br/>c</p>', keep=('br',)), u'a<br />b<br/>c')
|
||||
self.assertEqual(remove_tags(u'<p>a<br />b<br/>c</p>', keep=('p',)), u'<p>abc</p>')
|
||||
self.assertEqual(remove_tags(u'<p>a<br />b<br/>c</p>', keep=('p','br','div')), u'<p>a<br />b<br/>c</p>')
|
||||
|
||||
def test_remove_tags_with_content(self):
|
||||
# make sure it always return unicode
|
||||
assert isinstance(remove_tags_with_content('no tags'), unicode)
|
||||
assert isinstance(remove_tags_with_content('no tags', which_ones=('p',)), unicode)
|
||||
assert isinstance(remove_tags_with_content('<p>one tag</p>', which_ones=('p',)), unicode)
|
||||
assert isinstance(remove_tags_with_content('<a>link</a>', which_ones=('b',)), unicode)
|
||||
|
||||
# text without tags
|
||||
self.assertEqual(remove_tags_with_content(u'no tags'), u'no tags')
|
||||
self.assertEqual(remove_tags_with_content(u'no tags', which_ones=('p','b',)), u'no tags')
|
||||
|
||||
# text with tags
|
||||
self.assertEqual(remove_tags_with_content(u'<p>one p tag</p>'), u'<p>one p tag</p>')
|
||||
self.assertEqual(remove_tags_with_content(u'<p>one p tag</p>', which_ones=('p',)), u'')
|
||||
|
||||
self.assertEqual(remove_tags_with_content(u'<b>not will removed</b><i>i will removed</i>', which_ones=('i',)),
|
||||
u'<b>not will removed</b>')
|
||||
|
||||
# text with empty tags
|
||||
self.assertEqual(remove_tags_with_content(u'<br/>a<br />', which_ones=('br',)), u'a')
|
||||
|
||||
def test_replace_escape_chars(self):
|
||||
# make sure it always return unicode
|
||||
assert isinstance(replace_escape_chars('no ec'), unicode)
|
||||
assert isinstance(replace_escape_chars('no ec', replace_by='str'), unicode)
|
||||
assert isinstance(replace_escape_chars('no ec', which_ones=('\n','\t',)), unicode)
|
||||
|
||||
# text without escape chars
|
||||
self.assertEqual(replace_escape_chars(u'no ec'), u'no ec')
|
||||
self.assertEqual(replace_escape_chars(u'no ec', which_ones=('\n',)), u'no ec')
|
||||
|
||||
# text with escape chars
|
||||
self.assertEqual(replace_escape_chars(u'escape\n\n'), u'escape')
|
||||
self.assertEqual(replace_escape_chars(u'escape\n', which_ones=('\t',)), u'escape\n')
|
||||
self.assertEqual(replace_escape_chars(u'escape\tchars\n', which_ones=('\t')), 'escapechars\n')
|
||||
self.assertEqual(replace_escape_chars(u'escape\tchars\n', replace_by=' '), 'escape chars ')
|
||||
self.assertEqual(replace_escape_chars(u'escape\tchars\n', replace_by=u'\xa3'), u'escape\xa3chars\xa3')
|
||||
self.assertEqual(replace_escape_chars(u'escape\tchars\n', replace_by='\xc2\xa3'), u'escape\xa3chars\xa3')
|
||||
|
||||
def test_unquote_markup(self):
|
||||
sample_txt1 = u"""<node1>hi, this is sample text with entities: & ©
|
||||
<![CDATA[although this is inside a cdata! & "]]></node1>"""
|
||||
sample_txt2 = u'<node2>blah&blah<![CDATA[blahblahblah!£]]>moreblah<></node2>'
|
||||
sample_txt3 = u'something£&more<node3><![CDATA[things, stuff, and such]]>what"ever</node3><node4'
|
||||
|
||||
# make sure it always return unicode
|
||||
assert isinstance(unquote_markup(sample_txt1.encode('latin-1')), unicode)
|
||||
assert isinstance(unquote_markup(sample_txt2), unicode)
|
||||
|
||||
self.assertEqual(unquote_markup(sample_txt1), u"""<node1>hi, this is sample text with entities: & \xa9
|
||||
although this is inside a cdata! & "</node1>""")
|
||||
|
||||
self.assertEqual(unquote_markup(sample_txt2), u'<node2>blah&blahblahblahblah!£moreblah<></node2>')
|
||||
|
||||
self.assertEqual(unquote_markup(sample_txt1 + sample_txt2), u"""<node1>hi, this is sample text with entities: & \xa9
|
||||
although this is inside a cdata! & "</node1><node2>blah&blahblahblahblah!£moreblah<></node2>""")
|
||||
|
||||
self.assertEqual(unquote_markup(sample_txt3), u'something\xa3&more<node3>things, stuff, and suchwhat"ever</node3><node4')
|
||||
|
|
@ -31,108 +31,6 @@ class ResponseUtilsTest(unittest.TestCase):
|
|||
self.assertTrue(isinstance(body_or_str(u'text', unicode=False), str))
|
||||
self.assertTrue(isinstance(body_or_str(u'text', unicode=True), unicode))
|
||||
|
||||
def test_get_base_url(self):
|
||||
response = HtmlResponse(url='https://example.org', body="""\
|
||||
<html>\
|
||||
<head><title>Dummy</title><base href='http://example.org/something' /></head>\
|
||||
<body>blahablsdfsal&</body>\
|
||||
</html>""")
|
||||
self.assertEqual(get_base_url(response), 'http://example.org/something')
|
||||
|
||||
# relative url with absolute path
|
||||
response = HtmlResponse(url='https://example.org', body="""\
|
||||
<html>\
|
||||
<head><title>Dummy</title><base href='/absolutepath' /></head>\
|
||||
<body>blahablsdfsal&</body>\
|
||||
</html>""")
|
||||
self.assertEqual(get_base_url(response), 'https://example.org/absolutepath')
|
||||
|
||||
# no scheme url
|
||||
response = HtmlResponse(url='https://example.org', body="""\
|
||||
<html>\
|
||||
<head><title>Dummy</title><base href='//noscheme.com/path' /></head>\
|
||||
<body>blahablsdfsal&</body>\
|
||||
</html>""")
|
||||
self.assertEqual(get_base_url(response), 'https://noscheme.com/path')
|
||||
|
||||
def test_get_meta_refresh(self):
|
||||
body = """
|
||||
<html>
|
||||
<head><title>Dummy</title><meta http-equiv="refresh" content="5;url=http://example.org/newpage" /></head>
|
||||
<body>blahablsdfsal&</body>
|
||||
</html>"""
|
||||
response = TextResponse(url='http://example.org', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (5, 'http://example.org/newpage'))
|
||||
|
||||
# refresh without url should return (None, None)
|
||||
body = """<meta http-equiv="refresh" content="5" />"""
|
||||
response = TextResponse(url='http://example.org', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (None, None))
|
||||
|
||||
body = """<meta http-equiv="refresh" content="5;
|
||||
url=http://example.org/newpage" /></head>"""
|
||||
response = TextResponse(url='http://example.org', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (5, 'http://example.org/newpage'))
|
||||
|
||||
# meta refresh in multiple lines
|
||||
body = """<html><head>
|
||||
<META
|
||||
HTTP-EQUIV="Refresh"
|
||||
CONTENT="1; URL=http://example.org/newpage">"""
|
||||
response = TextResponse(url='http://example.org', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (1, 'http://example.org/newpage'))
|
||||
|
||||
# entities in the redirect url
|
||||
body = """<meta http-equiv="refresh" content="3; url='http://www.example.com/other'">"""
|
||||
response = TextResponse(url='http://example.com', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (3, 'http://www.example.com/other'))
|
||||
|
||||
# relative redirects
|
||||
body = """<meta http-equiv="refresh" content="3; url=other.html">"""
|
||||
response = TextResponse(url='http://example.com/page/this.html', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (3, 'http://example.com/page/other.html'))
|
||||
|
||||
# non-standard encodings (utf-16)
|
||||
body = """<meta http-equiv="refresh" content="3; url=http://example.com/redirect">"""
|
||||
body = body.decode('ascii').encode('utf-16')
|
||||
response = TextResponse(url='http://example.com', body=body, encoding='utf-16')
|
||||
self.assertEqual(get_meta_refresh(response), (3, 'http://example.com/redirect'))
|
||||
|
||||
# non-ascii chars in the url (utf8)
|
||||
body = """<meta http-equiv="refresh" content="3; url=http://example.com/to\xc2\xa3">"""
|
||||
response = TextResponse(url='http://example.com', body=body, encoding='utf-8')
|
||||
self.assertEqual(get_meta_refresh(response), (3, 'http://example.com/to%C2%A3'))
|
||||
|
||||
# non-ascii chars in the url (latin1)
|
||||
body = """<meta http-equiv="refresh" content="3; url=http://example.com/to\xa3">"""
|
||||
response = TextResponse(url='http://example.com', body=body, encoding='latin1')
|
||||
self.assertEqual(get_meta_refresh(response), (3, 'http://example.com/to%C2%A3'))
|
||||
|
||||
# responses without refresh tag should return None None
|
||||
response = TextResponse(url='http://example.org')
|
||||
self.assertEqual(get_meta_refresh(response), (None, None))
|
||||
response = TextResponse(url='http://example.org')
|
||||
self.assertEqual(get_meta_refresh(response), (None, None))
|
||||
|
||||
# html commented meta refresh header must not directed
|
||||
body = """<!--<meta http-equiv="refresh" content="3; url=http://example.com/">-->"""
|
||||
response = TextResponse(url='http://example.com', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (None, None))
|
||||
|
||||
# html comments must not interfere with uncommented meta refresh header
|
||||
body = """<!-- commented --><meta http-equiv="refresh" content="3; url=http://example.com/">-->"""
|
||||
response = TextResponse(url='http://example.com', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (3, 'http://example.com/'))
|
||||
|
||||
# float refresh intervals
|
||||
body = """<meta http-equiv="refresh" content=".1;URL=index.html" />"""
|
||||
response = TextResponse(url='http://example.com', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (0.1, 'http://example.com/index.html'))
|
||||
|
||||
body = """<meta http-equiv="refresh" content="3.1;URL=index.html" />"""
|
||||
response = TextResponse(url='http://example.com', body=body)
|
||||
self.assertEqual(get_meta_refresh(response), (3.1, 'http://example.com/index.html'))
|
||||
|
||||
def test_response_httprepr(self):
|
||||
r1 = Response("http://www.example.com")
|
||||
self.assertEqual(response_httprepr(r1), 'HTTP/1.1 200 OK\r\n\r\n')
|
||||
|
|
|
|||
|
|
@ -1,9 +1,6 @@
|
|||
import os
|
||||
import unittest
|
||||
from scrapy.spider import BaseSpider
|
||||
from scrapy.utils.url import url_is_from_any_domain, safe_url_string, safe_download_url, \
|
||||
url_query_parameter, add_or_replace_parameter, url_query_cleaner, canonicalize_url, \
|
||||
urljoin_rfc, url_is_from_spider, file_uri_to_path, path_to_file_uri, any_to_uri
|
||||
from scrapy.utils.url import url_is_from_any_domain, url_is_from_spider, canonicalize_url
|
||||
|
||||
class UrlUtilsTest(unittest.TestCase):
|
||||
|
||||
|
|
@ -55,144 +52,6 @@ class UrlUtilsTest(unittest.TestCase):
|
|||
self.assertTrue(url_is_from_spider('http://www.example.net/some/page.html', MySpider))
|
||||
self.assertFalse(url_is_from_spider('http://www.example.us/some/page.html', MySpider))
|
||||
|
||||
def test_urljoin_rfc(self):
|
||||
self.assertEqual(urljoin_rfc('http://example.com/some/path', 'newpath/test'),
|
||||
'http://example.com/some/newpath/test')
|
||||
self.assertEqual(urljoin_rfc('http://example.com/some/path/a.jpg', '../key/other'),
|
||||
'http://example.com/some/key/other')
|
||||
u = urljoin_rfc(u'http://example.com/lolo/\xa3/lele', u'lala/\xa3')
|
||||
self.assertEqual(u, 'http://example.com/lolo/\xc2\xa3/lala/\xc2\xa3')
|
||||
assert isinstance(u, str)
|
||||
u = urljoin_rfc(u'http://example.com/lolo/\xa3/lele', 'lala/\xa3', encoding='latin-1')
|
||||
self.assertEqual(u, 'http://example.com/lolo/\xa3/lala/\xa3')
|
||||
assert isinstance(u, str)
|
||||
u = urljoin_rfc('http://example.com/lolo/\xa3/lele', 'lala/\xa3')
|
||||
self.assertEqual(u, 'http://example.com/lolo/\xa3/lala/\xa3')
|
||||
assert isinstance(u, str)
|
||||
|
||||
def test_safe_url_string(self):
|
||||
# Motoko Kusanagi (Cyborg from Ghost in the Shell)
|
||||
motoko = u'\u8349\u8599 \u7d20\u5b50'
|
||||
self.assertEqual(safe_url_string(motoko), # note the %20 for space
|
||||
'%E8%8D%89%E8%96%99%20%E7%B4%A0%E5%AD%90')
|
||||
self.assertEqual(safe_url_string(motoko),
|
||||
safe_url_string(safe_url_string(motoko)))
|
||||
self.assertEqual(safe_url_string(u'\xa9'), # copyright symbol
|
||||
'%C2%A9')
|
||||
self.assertEqual(safe_url_string(u'\xa9', 'iso-8859-1'),
|
||||
'%A9')
|
||||
self.assertEqual(safe_url_string("http://www.scrapy.org/"),
|
||||
'http://www.scrapy.org/')
|
||||
|
||||
alessi = u'/ecommerce/oggetto/Te \xf2/tea-strainer/1273'
|
||||
|
||||
self.assertEqual(safe_url_string(alessi),
|
||||
'/ecommerce/oggetto/Te%20%C3%B2/tea-strainer/1273')
|
||||
|
||||
self.assertEqual(safe_url_string("http://www.example.com/test?p(29)url(http://www.another.net/page)"),
|
||||
"http://www.example.com/test?p(29)url(http://www.another.net/page)")
|
||||
self.assertEqual(safe_url_string("http://www.example.com/Brochures_&_Paint_Cards&PageSize=200"),
|
||||
"http://www.example.com/Brochures_&_Paint_Cards&PageSize=200")
|
||||
|
||||
safeurl = safe_url_string(u"http://www.example.com/\xa3", encoding='latin-1')
|
||||
self.assert_(isinstance(safeurl, str))
|
||||
self.assertEqual(safeurl, "http://www.example.com/%A3")
|
||||
|
||||
safeurl = safe_url_string(u"http://www.example.com/\xa3", encoding='utf-8')
|
||||
self.assert_(isinstance(safeurl, str))
|
||||
self.assertEqual(safeurl, "http://www.example.com/%C2%A3")
|
||||
|
||||
def test_safe_download_url(self):
|
||||
self.assertEqual(safe_download_url('http://www.scrapy.org/../'),
|
||||
'http://www.scrapy.org/')
|
||||
self.assertEqual(safe_download_url('http://www.scrapy.org/../../images/../image'),
|
||||
'http://www.scrapy.org/image')
|
||||
self.assertEqual(safe_download_url('http://www.scrapy.org/dir/'),
|
||||
'http://www.scrapy.org/dir/')
|
||||
|
||||
def test_url_query_parameter(self):
|
||||
self.assertEqual(url_query_parameter("product.html?id=200&foo=bar", "id"),
|
||||
'200')
|
||||
self.assertEqual(url_query_parameter("product.html?id=200&foo=bar", "notthere", "mydefault"),
|
||||
'mydefault')
|
||||
self.assertEqual(url_query_parameter("product.html?id=", "id"),
|
||||
None)
|
||||
self.assertEqual(url_query_parameter("product.html?id=", "id", keep_blank_values=1),
|
||||
'')
|
||||
|
||||
def test_url_query_parameter_2(self):
|
||||
"""
|
||||
This problem was seen several times in the feeds. Sometime affiliate URLs contains
|
||||
nested encoded affiliate URL with direct URL as parameters. For example:
|
||||
aff_url1 = 'http://www.tkqlhce.com/click-2590032-10294381?url=http%3A%2F%2Fwww.argos.co.uk%2Fwebapp%2Fwcs%2Fstores%2Fservlet%2FArgosCreateReferral%3FstoreId%3D10001%26langId%3D-1%26referrer%3DCOJUN%26params%3Dadref%253DGarden+and+DIY-%3EGarden+furniture-%3EChildren%26%2339%3Bs+garden+furniture%26referredURL%3Dhttp%3A%2F%2Fwww.argos.co.uk%2Fwebapp%2Fwcs%2Fstores%2Fservlet%2FProductDisplay%253FstoreId%253D10001%2526catalogId%253D1500001501%2526productId%253D1500357023%2526langId%253D-1'
|
||||
the typical code to extract needed URL from it is:
|
||||
aff_url2 = url_query_parameter(aff_url1, 'url')
|
||||
after this aff2_url is:
|
||||
'http://www.argos.co.uk/webapp/wcs/stores/servlet/ArgosCreateReferral?storeId=10001&langId=-1&referrer=COJUN¶ms=adref%3DGarden and DIY->Garden furniture->Children's gardenfurniture&referredURL=http://www.argos.co.uk/webapp/wcs/stores/servlet/ProductDisplay%3FstoreId%3D10001%26catalogId%3D1500001501%26productId%3D1500357023%26langId%3D-1'
|
||||
the direct URL extraction is
|
||||
url = url_query_parameter(aff_url2, 'referredURL')
|
||||
but this will not work, because aff_url2 contains ' (comma sign encoded in the feed)
|
||||
and the URL extraction will fail, current workaround was made in the spider,
|
||||
just a replace for ' to %27
|
||||
"""
|
||||
return # FIXME: this test should pass but currently doesnt
|
||||
# correct case
|
||||
aff_url1 = "http://www.anrdoezrs.net/click-2590032-10294381?url=http%3A%2F%2Fwww.argos.co.uk%2Fwebapp%2Fwcs%2Fstores%2Fservlet%2FArgosCreateReferral%3FstoreId%3D10001%26langId%3D-1%26referrer%3DCOJUN%26params%3Dadref%253DGarden+and+DIY-%3EGarden+furniture-%3EGarden+table+and+chair+sets%26referredURL%3Dhttp%3A%2F%2Fwww.argos.co.uk%2Fwebapp%2Fwcs%2Fstores%2Fservlet%2FProductDisplay%253FstoreId%253D10001%2526catalogId%253D1500001501%2526productId%253D1500357199%2526langId%253D-1"
|
||||
aff_url2 = url_query_parameter(aff_url1, 'url')
|
||||
self.assertEqual(aff_url2, "http://www.argos.co.uk/webapp/wcs/stores/servlet/ArgosCreateReferral?storeId=10001&langId=-1&referrer=COJUN¶ms=adref%3DGarden and DIY->Garden furniture->Garden table and chair sets&referredURL=http://www.argos.co.uk/webapp/wcs/stores/servlet/ProductDisplay%3FstoreId%3D10001%26catalogId%3D1500001501%26productId%3D1500357199%26langId%3D-1")
|
||||
prod_url = url_query_parameter(aff_url2, 'referredURL')
|
||||
self.assertEqual(prod_url, "http://www.argos.co.uk/webapp/wcs/stores/servlet/ProductDisplay?storeId=10001&catalogId=1500001501&productId=1500357199&langId=-1")
|
||||
# weird case
|
||||
aff_url1 = "http://www.tkqlhce.com/click-2590032-10294381?url=http%3A%2F%2Fwww.argos.co.uk%2Fwebapp%2Fwcs%2Fstores%2Fservlet%2FArgosCreateReferral%3FstoreId%3D10001%26langId%3D-1%26referrer%3DCOJUN%26params%3Dadref%253DGarden+and+DIY-%3EGarden+furniture-%3EChildren%26%2339%3Bs+garden+furniture%26referredURL%3Dhttp%3A%2F%2Fwww.argos.co.uk%2Fwebapp%2Fwcs%2Fstores%2Fservlet%2FProductDisplay%253FstoreId%253D10001%2526catalogId%253D1500001501%2526productId%253D1500357023%2526langId%253D-1"
|
||||
aff_url2 = url_query_parameter(aff_url1, 'url')
|
||||
self.assertEqual(aff_url2, "http://www.argos.co.uk/webapp/wcs/stores/servlet/ArgosCreateReferral?storeId=10001&langId=-1&referrer=COJUN¶ms=adref%3DGarden and DIY->Garden furniture->Children's garden furniture&referredURL=http://www.argos.co.uk/webapp/wcs/stores/servlet/ProductDisplay%3FstoreId%3D10001%26catalogId%3D1500001501%26productId%3D1500357023%26langId%3D-1")
|
||||
prod_url = url_query_parameter(aff_url2, 'referredURL')
|
||||
# fails, prod_url is None now
|
||||
self.assertEqual(prod_url, "http://www.argos.co.uk/webapp/wcs/stores/servlet/ProductDisplay?storeId=10001&catalogId=1500001501&productId=1500357023&langId=-1")
|
||||
|
||||
def test_add_or_replace_parameter(self):
|
||||
url = 'http://domain/test'
|
||||
self.assertEqual(add_or_replace_parameter(url, 'arg', 'v'),
|
||||
'http://domain/test?arg=v')
|
||||
url = 'http://domain/test?arg1=v1&arg2=v2&arg3=v3'
|
||||
self.assertEqual(add_or_replace_parameter(url, 'arg4', 'v4'),
|
||||
'http://domain/test?arg1=v1&arg2=v2&arg3=v3&arg4=v4')
|
||||
self.assertEqual(add_or_replace_parameter(url, 'arg3', 'nv3'),
|
||||
'http://domain/test?arg1=v1&arg2=v2&arg3=nv3')
|
||||
url = 'http://domain/test?arg1=v1'
|
||||
self.assertEqual(add_or_replace_parameter(url, 'arg2', 'v2', sep=';'),
|
||||
'http://domain/test?arg1=v1;arg2=v2')
|
||||
self.assertEqual(add_or_replace_parameter("http://domain/moreInfo.asp?prodID=", 'prodID', '20'),
|
||||
'http://domain/moreInfo.asp?prodID=20')
|
||||
url = 'http://rmc-offers.co.uk/productlist.asp?BCat=2%2C60&CatID=60'
|
||||
self.assertEqual(add_or_replace_parameter(url, 'BCat', 'newvalue', url_is_quoted=True),
|
||||
'http://rmc-offers.co.uk/productlist.asp?BCat=newvalue&CatID=60')
|
||||
url = 'http://rmc-offers.co.uk/productlist.asp?BCat=2,60&CatID=60'
|
||||
self.assertEqual(add_or_replace_parameter(url, 'BCat', 'newvalue'),
|
||||
'http://rmc-offers.co.uk/productlist.asp?BCat=newvalue&CatID=60')
|
||||
|
||||
def test_url_query_cleaner(self):
|
||||
self.assertEqual('product.html?id=200',
|
||||
url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id']))
|
||||
self.assertEqual('product.html?id=200',
|
||||
url_query_cleaner("product.html?&id=200&&foo=bar&name=wired", ['id']))
|
||||
self.assertEqual('product.html',
|
||||
url_query_cleaner("product.html?foo=bar&name=wired", ['id']))
|
||||
self.assertEqual('product.html?id=200&name=wired',
|
||||
url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id', 'name']))
|
||||
self.assertEqual('product.html?id',
|
||||
url_query_cleaner("product.html?id&other=3&novalue=", ['id']))
|
||||
self.assertEqual('product.html?d=1&d=2&d=3',
|
||||
url_query_cleaner("product.html?d=1&e=b&d=2&d=3&other=other", ['d'], unique=False))
|
||||
self.assertEqual('product.html?id=200&foo=bar',
|
||||
url_query_cleaner("product.html?id=200&foo=bar&name=wired#id20", ['id', 'foo']))
|
||||
self.assertEqual('product.html?foo=bar&name=wired',
|
||||
url_query_cleaner("product.html?id=200&foo=bar&name=wired", ['id'], remove=True))
|
||||
self.assertEqual('product.html?name=wired',
|
||||
url_query_cleaner("product.html?id=2&foo=bar&name=wired", ['id', 'foo'], remove=True))
|
||||
self.assertEqual('product.html?foo=bar&name=wired',
|
||||
url_query_cleaner("product.html?id=2&foo=bar&name=wired", ['id', 'footo'], remove=True))
|
||||
|
||||
def test_canonicalize_url(self):
|
||||
# simplest case
|
||||
self.assertEqual(canonicalize_url("http://www.example.com"),
|
||||
|
|
@ -283,50 +142,6 @@ class UrlUtilsTest(unittest.TestCase):
|
|||
self.assertEqual(canonicalize_url("http://www.EXAMPLE.com"),
|
||||
"http://www.example.com")
|
||||
|
||||
def test_path_to_file_uri(self):
|
||||
if os.name == 'nt':
|
||||
self.assertEqual(path_to_file_uri("C:\\windows\clock.avi"),
|
||||
"file:///C:/windows/clock.avi")
|
||||
else:
|
||||
self.assertEqual(path_to_file_uri("/some/path.txt"),
|
||||
"file:///some/path.txt")
|
||||
|
||||
fn = "test.txt"
|
||||
x = path_to_file_uri(fn)
|
||||
self.assert_(x.startswith('file:///'))
|
||||
self.assertEqual(file_uri_to_path(x).lower(), os.path.abspath(fn).lower())
|
||||
|
||||
def test_file_uri_to_path(self):
|
||||
if os.name == 'nt':
|
||||
self.assertEqual(file_uri_to_path("file:///C:/windows/clock.avi"),
|
||||
"C:\\windows\clock.avi")
|
||||
uri = "file:///C:/windows/clock.avi"
|
||||
uri2 = path_to_file_uri(file_uri_to_path(uri))
|
||||
self.assertEqual(uri, uri2)
|
||||
else:
|
||||
self.assertEqual(file_uri_to_path("file:///path/to/test.txt"),
|
||||
"/path/to/test.txt")
|
||||
self.assertEqual(file_uri_to_path("/path/to/test.txt"),
|
||||
"/path/to/test.txt")
|
||||
uri = "file:///path/to/test.txt"
|
||||
uri2 = path_to_file_uri(file_uri_to_path(uri))
|
||||
self.assertEqual(uri, uri2)
|
||||
|
||||
self.assertEqual(file_uri_to_path("test.txt"),
|
||||
"test.txt")
|
||||
|
||||
def test_any_to_uri(self):
|
||||
if os.name == 'nt':
|
||||
self.assertEqual(any_to_uri("C:\\windows\clock.avi"),
|
||||
"file:///C:/windows/clock.avi")
|
||||
else:
|
||||
self.assertEqual(any_to_uri("/some/path.txt"),
|
||||
"file:///some/path.txt")
|
||||
self.assertEqual(any_to_uri("file:///some/path.txt"),
|
||||
"file:///some/path.txt")
|
||||
self.assertEqual(any_to_uri("http://www.example.com/some/path.txt"),
|
||||
"http://www.example.com/some/path.txt")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
|
|||
|
|
@ -1,61 +1,7 @@
|
|||
from base64 import urlsafe_b64encode
|
||||
"""
|
||||
Transitional module for moving to the w3lib library.
|
||||
|
||||
def headers_raw_to_dict(headers_raw):
|
||||
"""
|
||||
Convert raw headers (single multi-line string)
|
||||
to the dictionary.
|
||||
For new code, always import from w3lib.http instead of this module
|
||||
"""
|
||||
|
||||
For example:
|
||||
>>> headers_raw_to_dict("Content-type: text/html\\n\\rAccept: gzip\\n\\n")
|
||||
{'Content-type': ['text/html'], 'Accept': ['gzip']}
|
||||
|
||||
Incorrect input:
|
||||
>>> headers_raw_to_dict("Content-typt gzip\\n\\n")
|
||||
{}
|
||||
|
||||
Argument is None:
|
||||
>>> headers_raw_to_dict(None)
|
||||
"""
|
||||
if headers_raw is None:
|
||||
return None
|
||||
return dict([
|
||||
(header_item[0].strip(), [header_item[1].strip()])
|
||||
for header_item
|
||||
in [
|
||||
header.split(':', 1)
|
||||
for header
|
||||
in headers_raw.splitlines()]
|
||||
if len(header_item) == 2])
|
||||
|
||||
|
||||
def headers_dict_to_raw(headers_dict):
|
||||
"""
|
||||
Returns a raw HTTP headers representation of headers
|
||||
|
||||
For example:
|
||||
>>> headers_dict_to_raw({'Content-type': 'text/html', 'Accept': 'gzip'})
|
||||
'Content-type: text/html\\r\\nAccept: gzip'
|
||||
>>> from twisted.python.util import InsensitiveDict
|
||||
>>> td = InsensitiveDict({'Content-type': ['text/html'], 'Accept': ['gzip']})
|
||||
>>> headers_dict_to_raw(td)
|
||||
'Content-type: text/html\\r\\nAccept: gzip'
|
||||
|
||||
Argument is None:
|
||||
>>> headers_dict_to_raw(None)
|
||||
|
||||
"""
|
||||
if headers_dict is None:
|
||||
return None
|
||||
raw_lines = []
|
||||
for key, value in headers_dict.items():
|
||||
if isinstance(value, (str, unicode)):
|
||||
raw_lines.append("%s: %s" % (key, value))
|
||||
elif isinstance(value, (list, tuple)):
|
||||
for v in value:
|
||||
raw_lines.append("%s: %s" % (key, v))
|
||||
return '\r\n'.join(raw_lines)
|
||||
|
||||
|
||||
def basic_auth_header(username, password):
|
||||
"""Return `Authorization` header for HTTP Basic Access Authentication (RFC 2617)"""
|
||||
return 'Basic ' + urlsafe_b64encode("%s:%s" % (username, password))
|
||||
from w3lib.http import *
|
||||
|
|
|
|||
|
|
@ -1,165 +1,7 @@
|
|||
"""
|
||||
Functions for dealing with markup text
|
||||
Transitional module for moving to the w3lib library.
|
||||
|
||||
For new code, always import from w3lib.html instead of this module
|
||||
"""
|
||||
|
||||
import re
|
||||
from htmlentitydefs import name2codepoint
|
||||
|
||||
from scrapy.utils.python import str_to_unicode
|
||||
|
||||
_ent_re = re.compile(r'&(#?(x?))([^&;\s]+);')
|
||||
_tag_re = re.compile(r'<[a-zA-Z\/!].*?>', re.DOTALL)
|
||||
|
||||
def remove_entities(text, keep=(), remove_illegal=True, encoding='utf-8'):
|
||||
"""Remove entities from the given text.
|
||||
|
||||
'text' can be a unicode string or a regular string encoded in the given
|
||||
`encoding` (which defaults to 'utf-8').
|
||||
|
||||
If 'keep' is passed (with a list of entity names) those entities will
|
||||
be kept (they won't be removed).
|
||||
|
||||
It supports both numeric (&#nnnn; and &#hhhh;) and named ( >)
|
||||
entities.
|
||||
|
||||
If remove_illegal is True, entities that can't be converted are removed.
|
||||
If remove_illegal is False, entities that can't be converted are kept "as
|
||||
is". For more information see the tests.
|
||||
|
||||
Always returns a unicode string (with the entities removed).
|
||||
"""
|
||||
|
||||
def convert_entity(m):
|
||||
entity_body = m.group(3)
|
||||
if m.group(1):
|
||||
try:
|
||||
if m.group(2):
|
||||
number = int(entity_body, 16)
|
||||
else:
|
||||
number = int(entity_body, 10)
|
||||
# Numeric character references in the 80-9F range are typically
|
||||
# interpreted by browsers as representing the characters mapped
|
||||
# to bytes 80-9F in the Windows-1252 encoding. For more info
|
||||
# see: http://en.wikipedia.org/wiki/Character_encodings_in_HTML
|
||||
if 0x80 <= number <= 0x9f:
|
||||
return chr(number).decode('cp1252')
|
||||
except ValueError:
|
||||
number = None
|
||||
else:
|
||||
if entity_body in keep:
|
||||
return m.group(0)
|
||||
else:
|
||||
number = name2codepoint.get(entity_body)
|
||||
if number is not None:
|
||||
try:
|
||||
return unichr(number)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
return u'' if remove_illegal else m.group(0)
|
||||
|
||||
return _ent_re.sub(convert_entity, str_to_unicode(text, encoding))
|
||||
|
||||
def has_entities(text, encoding=None):
|
||||
return bool(_ent_re.search(str_to_unicode(text, encoding)))
|
||||
|
||||
def replace_tags(text, token='', encoding=None):
|
||||
"""Replace all markup tags found in the given text by the given token. By
|
||||
default token is a null string so it just remove all tags.
|
||||
|
||||
'text' can be a unicode string or a regular string encoded as 'utf-8'
|
||||
|
||||
Always returns a unicode string.
|
||||
"""
|
||||
return _tag_re.sub(token, str_to_unicode(text, encoding))
|
||||
|
||||
|
||||
def remove_comments(text, encoding=None):
|
||||
""" Remove HTML Comments. """
|
||||
return re.sub('<!--.*?-->', u'', str_to_unicode(text, encoding), re.DOTALL)
|
||||
|
||||
def remove_tags(text, which_ones=(), keep=(), encoding=None):
|
||||
""" Remove HTML Tags only.
|
||||
|
||||
which_ones and keep are both tuples, there are four cases:
|
||||
|
||||
which_ones, keep (1 - not empty, 0 - empty)
|
||||
1, 0 - remove all tags in which_ones
|
||||
0, 1 - remove all tags except the ones in keep
|
||||
0, 0 - remove all tags
|
||||
1, 1 - not allowd
|
||||
"""
|
||||
|
||||
assert not (which_ones and keep), 'which_ones and keep can not be given at the same time'
|
||||
|
||||
def will_remove(tag):
|
||||
if which_ones:
|
||||
return tag in which_ones
|
||||
else:
|
||||
return tag not in keep
|
||||
|
||||
def remove_tag(m):
|
||||
tag = m.group(1)
|
||||
return u'' if will_remove(tag) else m.group(0)
|
||||
|
||||
regex = '</?([^ >/]+).*?>'
|
||||
retags = re.compile(regex, re.DOTALL | re.IGNORECASE)
|
||||
|
||||
return retags.sub(remove_tag, str_to_unicode(text, encoding))
|
||||
|
||||
def remove_tags_with_content(text, which_ones=(), encoding=None):
|
||||
""" Remove tags and its content.
|
||||
|
||||
which_ones -- is a tuple of which tags with its content we want to remove.
|
||||
if is empty do nothing.
|
||||
"""
|
||||
text = str_to_unicode(text, encoding)
|
||||
if which_ones:
|
||||
tags = '|'.join([r'<%s.*?</%s>|<%s\s*/>' % (tag, tag, tag) for tag in which_ones])
|
||||
retags = re.compile(tags, re.DOTALL | re.IGNORECASE)
|
||||
text = retags.sub(u'', text)
|
||||
return text
|
||||
|
||||
|
||||
def replace_escape_chars(text, which_ones=('\n', '\t', '\r'), replace_by=u'', \
|
||||
encoding=None):
|
||||
""" Remove escape chars. Default : \\n, \\t, \\r
|
||||
|
||||
which_ones -- is a tuple of which escape chars we want to remove.
|
||||
By default removes \n, \t, \r.
|
||||
|
||||
replace_by -- text to replace the escape chars for.
|
||||
It defaults to '', so the escape chars are removed.
|
||||
"""
|
||||
for ec in which_ones:
|
||||
text = text.replace(ec, str_to_unicode(replace_by, encoding))
|
||||
return str_to_unicode(text, encoding)
|
||||
|
||||
def unquote_markup(text, keep=(), remove_illegal=True, encoding=None):
|
||||
"""
|
||||
This function receives markup as a text (always a unicode string or a utf-8 encoded string) and does the following:
|
||||
- removes entities (except the ones in 'keep') from any part of it that it's not inside a CDATA
|
||||
- searches for CDATAs and extracts their text (if any) without modifying it.
|
||||
- removes the found CDATAs
|
||||
"""
|
||||
_cdata_re = re.compile(r'((?P<cdata_s><!\[CDATA\[)(?P<cdata_d>.*?)(?P<cdata_e>\]\]>))', re.DOTALL)
|
||||
|
||||
def _get_fragments(txt, pattern):
|
||||
offset = 0
|
||||
for match in pattern.finditer(txt):
|
||||
match_s, match_e = match.span(1)
|
||||
yield txt[offset:match_s]
|
||||
yield match
|
||||
offset = match_e
|
||||
yield txt[offset:]
|
||||
|
||||
text = str_to_unicode(text, encoding)
|
||||
ret_text = u''
|
||||
for fragment in _get_fragments(text, _cdata_re):
|
||||
if isinstance(fragment, basestring):
|
||||
# it's not a CDATA (so we try to remove its entities)
|
||||
ret_text += remove_entities(fragment, keep=keep, remove_illegal=remove_illegal)
|
||||
else:
|
||||
# it's a CDATA (so we just extract its content)
|
||||
ret_text += fragment.group('cdata_d')
|
||||
return ret_text
|
||||
from w3lib.html import *
|
||||
|
|
|
|||
|
|
@ -4,8 +4,8 @@ import re
|
|||
import hashlib
|
||||
from pkgutil import iter_modules
|
||||
|
||||
from w3lib.html import remove_entities
|
||||
from scrapy.utils.python import flatten
|
||||
from scrapy.utils.markup import remove_entities
|
||||
|
||||
def arg_to_iter(arg):
|
||||
"""Convert an argument to an iterable. The argument can be a None, single
|
||||
|
|
|
|||
|
|
@ -1,34 +1,7 @@
|
|||
from cStringIO import StringIO
|
||||
"""
|
||||
Transitional module for moving to the w3lib library.
|
||||
|
||||
def encode_multipart(data):
|
||||
"""Encode the given data to be used in a multipart HTTP POST. Data is a
|
||||
where keys are the field name, and values are either strings or tuples
|
||||
(filename, content) for file uploads.
|
||||
For new code, always import from w3lib.form instead of this module
|
||||
"""
|
||||
|
||||
This code is based on distutils.command.upload
|
||||
"""
|
||||
|
||||
# Build up the MIME payload for the POST data
|
||||
boundary = '--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'
|
||||
sep_boundary = '\r\n--' + boundary
|
||||
end_boundary = sep_boundary + '--'
|
||||
body = StringIO()
|
||||
for key, value in data.items():
|
||||
# handle multiple entries for the same name
|
||||
if type(value) != type([]):
|
||||
value = [value]
|
||||
for value in value:
|
||||
if type(value) is tuple:
|
||||
fn = '; filename="%s"' % value[0]
|
||||
value = value[1]
|
||||
else:
|
||||
fn = ""
|
||||
|
||||
body.write(sep_boundary)
|
||||
body.write('\r\nContent-Disposition: form-data; name="%s"' % key)
|
||||
body.write(fn)
|
||||
body.write("\r\n\r\n")
|
||||
body.write(value)
|
||||
body.write(end_boundary)
|
||||
body.write("\r\n")
|
||||
return body.getvalue(), boundary
|
||||
from w3lib.form import *
|
||||
|
|
|
|||
|
|
@ -8,9 +8,10 @@ import weakref
|
|||
from base64 import urlsafe_b64encode
|
||||
from urlparse import urlunparse
|
||||
|
||||
from w3lib.http import basic_auth_header
|
||||
|
||||
from scrapy.utils.url import canonicalize_url
|
||||
from scrapy.utils.httpobj import urlparse_cached
|
||||
from scrapy.utils.http import basic_auth_header
|
||||
|
||||
|
||||
_fingerprint_cache = weakref.WeakKeyDictionary()
|
||||
|
|
@ -64,13 +65,6 @@ def request_authenticate(request, username, password):
|
|||
"""
|
||||
request.headers['Authorization'] = basic_auth_header(username, password)
|
||||
|
||||
def request_info(request):
|
||||
"""Return a short string with request info including method, url and
|
||||
fingeprint. Mainly used for debugging
|
||||
"""
|
||||
fp = request_fingerprint(request)
|
||||
return "<Request: %s %s (%s..)>" % (request.method, request.url, fp[:8])
|
||||
|
||||
def request_httprepr(request):
|
||||
"""Return the raw HTTP representation (as string) of the given request.
|
||||
This is provided only for reference since it's not the actual stream of
|
||||
|
|
|
|||
|
|
@ -4,16 +4,14 @@ scrapy.http.Response objects
|
|||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import weakref
|
||||
import webbrowser
|
||||
import tempfile
|
||||
|
||||
from twisted.web import http
|
||||
from twisted.web.http import RESPONSES
|
||||
from w3lib import html
|
||||
|
||||
from scrapy.utils.markup import remove_entities, remove_comments
|
||||
from scrapy.utils.url import safe_url_string, urljoin_rfc
|
||||
from scrapy.xlib.BeautifulSoup import BeautifulSoup
|
||||
from scrapy.http import Response, HtmlResponse
|
||||
|
||||
|
|
@ -27,37 +25,22 @@ def body_or_str(obj, unicode=True):
|
|||
else:
|
||||
return obj if unicode else obj.encode('utf-8')
|
||||
|
||||
BASEURL_RE = re.compile(ur'<base\s+href\s*=\s*[\"\']\s*([^\"\'\s]+)\s*[\"\']', re.I)
|
||||
_baseurl_cache = weakref.WeakKeyDictionary()
|
||||
def get_base_url(response):
|
||||
""" Return the base url of the given response used to resolve relative links. """
|
||||
"""Return the base url of the given response, joined with the response url"""
|
||||
if response not in _baseurl_cache:
|
||||
match = BASEURL_RE.search(response.body_as_unicode()[0:4096])
|
||||
_baseurl_cache[response] = urljoin_rfc(response.url, match.group(1)) if match else response.url
|
||||
text = response.body_as_unicode()[0:4096]
|
||||
_baseurl_cache[response] = html.get_base_url(text, response.url, \
|
||||
response.encoding)
|
||||
return _baseurl_cache[response]
|
||||
|
||||
META_REFRESH_RE = re.compile(ur'<meta[^>]*http-equiv[^>]*refresh[^>]*content\s*=\s*(?P<quote>["\'])(?P<int>(\d*\.)?\d+)\s*;\s*url=(?P<url>.*?)(?P=quote)', \
|
||||
re.DOTALL | re.IGNORECASE)
|
||||
_metaref_cache = weakref.WeakKeyDictionary()
|
||||
def get_meta_refresh(response):
|
||||
"""Parse the http-equiv parameter of the HTML meta element from the given
|
||||
response and return a tuple (interval, url) where interval is an integer
|
||||
containing the delay in seconds (or zero if not present) and url is a
|
||||
string with the absolute url to redirect.
|
||||
|
||||
If no meta redirect is found, (None, None) is returned.
|
||||
"""
|
||||
"""Parse the http-equiv refrsh parameter from the given response"""
|
||||
if response not in _metaref_cache:
|
||||
body_chunk = remove_comments(remove_entities(response.body_as_unicode()[0:4096]))
|
||||
match = META_REFRESH_RE.search(body_chunk)
|
||||
if match:
|
||||
interval = float(match.group('int'))
|
||||
url = safe_url_string(match.group('url').strip(' "\''))
|
||||
url = urljoin_rfc(response.url, url)
|
||||
_metaref_cache[response] = (interval, url)
|
||||
else:
|
||||
_metaref_cache[response] = (None, None)
|
||||
#_metaref_cache[response] = match.groups() if match else (None, None)
|
||||
text = response.body_as_unicode()[0:4096]
|
||||
_metaref_cache[response] = html.get_meta_refresh(text, response.url, \
|
||||
response.encoding)
|
||||
return _metaref_cache[response]
|
||||
|
||||
_beautifulsoup_cache = weakref.WeakKeyDictionary()
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
"""
|
||||
This module contains general purpose URL functions not found in the standard
|
||||
library.
|
||||
|
||||
Some of the functions that used to be imported from this module have been moved
|
||||
to the w3lib.url module. Always import those from there instead.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -10,6 +13,7 @@ import urllib
|
|||
import posixpath
|
||||
import cgi
|
||||
|
||||
from w3lib.url import *
|
||||
from scrapy.utils.python import unicode_to_str
|
||||
|
||||
def url_is_from_any_domain(url, domains):
|
||||
|
|
@ -26,109 +30,6 @@ def url_is_from_spider(url, spider):
|
|||
return url_is_from_any_domain(url, [spider.name] + \
|
||||
getattr(spider, 'allowed_domains', []))
|
||||
|
||||
def urljoin_rfc(base, ref, encoding='utf-8'):
|
||||
"""Same as urlparse.urljoin but supports unicode values in base and ref
|
||||
parameters (in which case they will be converted to str using the given
|
||||
encoding).
|
||||
|
||||
Always returns a str.
|
||||
"""
|
||||
return urlparse.urljoin(unicode_to_str(base, encoding), \
|
||||
unicode_to_str(ref, encoding))
|
||||
|
||||
_reserved = ';/?:@&=+$|,#' # RFC 3986 (Generic Syntax)
|
||||
_unreserved_marks = "-_.!~*'()" # RFC 3986 sec 2.3
|
||||
_safe_chars = urllib.always_safe + '%' + _reserved + _unreserved_marks
|
||||
|
||||
def safe_url_string(url, encoding='utf8'):
|
||||
"""Convert the given url into a legal URL by escaping unsafe characters
|
||||
according to RFC-3986.
|
||||
|
||||
If a unicode url is given, it is first converted to str using the given
|
||||
encoding (which defaults to 'utf-8'). When passing a encoding, you should
|
||||
use the encoding of the original page (the page from which the url was
|
||||
extracted from).
|
||||
|
||||
Calling this function on an already "safe" url will return the url
|
||||
unmodified.
|
||||
|
||||
Always returns a str.
|
||||
"""
|
||||
s = unicode_to_str(url, encoding)
|
||||
return urllib.quote(s, _safe_chars)
|
||||
|
||||
|
||||
_parent_dirs = re.compile(r'/?(\.\./)+')
|
||||
|
||||
def safe_download_url(url):
|
||||
""" Make a url for download. This will call safe_url_string
|
||||
and then strip the fragment, if one exists. The path will
|
||||
be normalised.
|
||||
|
||||
If the path is outside the document root, it will be changed
|
||||
to be within the document root.
|
||||
"""
|
||||
safe_url = safe_url_string(url)
|
||||
scheme, netloc, path, query, _ = urlparse.urlsplit(safe_url)
|
||||
if path:
|
||||
path = _parent_dirs.sub('', posixpath.normpath(path))
|
||||
if url.endswith('/') and not path.endswith('/'):
|
||||
path += '/'
|
||||
else:
|
||||
path = '/'
|
||||
return urlparse.urlunsplit((scheme, netloc, path, query, ''))
|
||||
|
||||
def is_url(text):
|
||||
return text.partition("://")[0] in ('file', 'http', 'https')
|
||||
|
||||
def url_query_parameter(url, parameter, default=None, keep_blank_values=0):
|
||||
"""Return the value of a url parameter, given the url and parameter name"""
|
||||
queryparams = cgi.parse_qs(urlparse.urlsplit(str(url))[3], \
|
||||
keep_blank_values=keep_blank_values)
|
||||
return queryparams.get(parameter, [default])[0]
|
||||
|
||||
def url_query_cleaner(url, parameterlist=(), sep='&', kvsep='=', remove=False, unique=True):
|
||||
"""Clean url arguments leaving only those passed in the parameterlist keeping order
|
||||
|
||||
If remove is True, leave only those not in parameterlist.
|
||||
If unique is False, do not remove duplicated keys
|
||||
"""
|
||||
url = urlparse.urldefrag(url)[0]
|
||||
base, _, query = url.partition('?')
|
||||
seen = set()
|
||||
querylist = []
|
||||
for ksv in query.split(sep):
|
||||
k, _, _ = ksv.partition(kvsep)
|
||||
if unique and k in seen:
|
||||
continue
|
||||
elif remove and k in parameterlist:
|
||||
continue
|
||||
elif not remove and k not in parameterlist:
|
||||
continue
|
||||
else:
|
||||
querylist.append(ksv)
|
||||
seen.add(k)
|
||||
return '?'.join([base, sep.join(querylist)]) if querylist else base
|
||||
|
||||
def add_or_replace_parameter(url, name, new_value, sep='&', url_is_quoted=False):
|
||||
"""Add or remove a parameter to a given url"""
|
||||
def has_querystring(url):
|
||||
_, _, _, query, _ = urlparse.urlsplit(url)
|
||||
return bool(query)
|
||||
|
||||
parameter = url_query_parameter(url, name, keep_blank_values=1)
|
||||
if url_is_quoted:
|
||||
parameter = urllib.quote(parameter)
|
||||
if parameter is None:
|
||||
if has_querystring(url):
|
||||
next_url = url + sep + name + '=' + new_value
|
||||
else:
|
||||
next_url = url + '?' + name + '=' + new_value
|
||||
else:
|
||||
next_url = url.replace(name+'='+parameter,
|
||||
name+'='+new_value)
|
||||
return next_url
|
||||
|
||||
def canonicalize_url(url, keep_blank_values=True, keep_fragments=False, \
|
||||
encoding=None):
|
||||
"""Canonicalize the given url by applying the following procedures:
|
||||
|
|
@ -155,27 +56,3 @@ def canonicalize_url(url, keep_blank_values=True, keep_fragments=False, \
|
|||
path = safe_url_string(urllib.unquote(path))
|
||||
fragment = '' if not keep_fragments else fragment
|
||||
return urlparse.urlunparse((scheme, netloc.lower(), path, params, query, fragment))
|
||||
|
||||
def path_to_file_uri(path):
|
||||
"""Convert local filesystem path to legal File URIs as described in:
|
||||
http://en.wikipedia.org/wiki/File_URI_scheme
|
||||
"""
|
||||
x = urllib.pathname2url(os.path.abspath(path))
|
||||
if os.name == 'nt':
|
||||
x = x.replace('|', ':') # http://bugs.python.org/issue5861
|
||||
return 'file:///%s' % x.lstrip('/')
|
||||
|
||||
def file_uri_to_path(uri):
|
||||
"""Convert File URI to local filesystem path according to:
|
||||
http://en.wikipedia.org/wiki/File_URI_scheme
|
||||
"""
|
||||
return urllib.url2pathname(urlparse.urlparse(uri).path)
|
||||
|
||||
def any_to_uri(uri_or_path):
|
||||
"""If given a path name, return its File URI, otherwise return it
|
||||
unmodified
|
||||
"""
|
||||
if os.path.splitdrive(uri_or_path)[0]:
|
||||
return path_to_file_uri(uri_or_path)
|
||||
u = urlparse.urlparse(uri_or_path)
|
||||
return uri_or_path if u.scheme else path_to_file_uri(uri_or_path)
|
||||
|
|
|
|||
|
|
@ -87,3 +87,12 @@ class DeleteVersion(DeleteProject):
|
|||
self._delete_version(project, version)
|
||||
return {"status": "ok"}
|
||||
|
||||
class ListJobs(WsResource):
|
||||
def render_POST(self, txrequest):
|
||||
project = txrequest.args['project'][0]
|
||||
spiders = self.root.launcher.processes.values()
|
||||
jlist = list()
|
||||
for s in spiders:
|
||||
if project == s.project:
|
||||
jlist.append({"job": {"id":s.job, "spider": s.spider}})
|
||||
return {"status":"ok", "jobs": jlist}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ class Root(resource.Resource):
|
|||
self.putChild('listspiders.json', webservice.ListSpiders(self))
|
||||
self.putChild('delproject.json', webservice.DeleteProject(self))
|
||||
self.putChild('delversion.json', webservice.DeleteVersion(self))
|
||||
self.putChild('listjobs.json', webservice.ListJobs(self))
|
||||
self.putChild('logs', static.File(logsdir, 'text/plain'))
|
||||
self.putChild('procmon', ProcessMonitor(self))
|
||||
self.update_projects()
|
||||
|
|
|
|||
2
setup.py
2
setup.py
|
|
@ -120,7 +120,7 @@ setup_args = {
|
|||
|
||||
try:
|
||||
from setuptools import setup
|
||||
setup_args['install_requires'] = ['Twisted>=2.5', 'lxml']
|
||||
setup_args['install_requires'] = ['Twisted>=2.5', 'lxml', 'w3lib']
|
||||
if sys.version_info < (2, 6):
|
||||
setup_args['install_requires'] += ['simplejson']
|
||||
except ImportError:
|
||||
|
|
|
|||
Loading…
Reference in New Issue