RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Automated merge with ssh://hg.scrapy.org:2222/scrapy-0.12

This commit is contained in:
Pablo Hoffman 2011-04-14 12:36:27 -03:00
parent ecb4f44cbc 6f262a198c
commit ad496eb3b6
35 changed files with 787 additions and 269 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
AUTHORS
View File

@ -27,3 +27,4 @@ Here is the list of the primary authors & contributors:
* Shuaib Khan
* Didier Deshommes
* Vikas Dhiman
* Jochen Maes

2
README
View File

@ -1,4 +1,4 @@
This is Scrapy, an opensource screen scraping framework written in Python.
For more visit the project home page at http://scrapy.org
For more info visit the project home page at http://scrapy.org

2
debian/scrapy.install vendored
View File

@ -1,3 +1,3 @@
usr/lib/python*/*-packages/scrapy
usr/lib/python*/*-packages/scrapy*
usr/bin
extras/scrapy_bash_completion etc/bash_completion.d/

1
debian/scrapyd.install vendored
View File

@ -1,3 +1,2 @@
usr/lib/python*/*-packages/scrapyd
debian/scrapyd-files/000-default etc/scrapyd/conf.d
extras/scrapyd.tac usr/share/scrapyd

18
docs/faq.rst
View File

@ -3,7 +3,7 @@
Frequently Asked Questions
==========================
How does Scrapy compare to BeautifulSoul or lxml?
How does Scrapy compare to BeautifulSoup or lxml?
-------------------------------------------------
`BeautifulSoup`_ and `lxml`_ are libraries for parsing HTML and XML. Scrapy is
@ -29,7 +29,7 @@ comparing `jinja2`_ to `Django`_.
What Python versions does Scrapy support?
-----------------------------------------
Scrapy runs in Python 2.5, 2.6 and 2.6. But it's recommended you use Python 2.6
Scrapy runs in Python 2.5, 2.6 and 2.7. But it's recommended you use Python 2.6
or above, since the Python 2.5 standard library has a few bugs in their URL
handling libraries. Some of these Python 2.5 bugs not only affect Scrapy but
any user code, such as spiders. You can see a list of `Python 2.5 bugs that
@ -240,3 +240,17 @@ In order to avoid parsing all the entire feed at once in memory, you can use
the functions ``xmliter`` and ``csviter`` from ``scrapy.utils.iterators``
module. In fact, this is what the feed spiders (see :ref:`topics-spiders`) use
under the cover.
Does Scrapy manage cookies automatically?
-----------------------------------------
Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them
back on subsequent requests, like any regular web browser does.
For more info see :ref:`topics-request-response` and :ref:`cookies-mw`.
How can I see the cookies being sent and received from Scrapy?
--------------------------------------------------------------
Enable the :setting:`COOKIES_DEBUG` setting.

77
docs/topics/downloader-middleware.rst
View File

@ -158,6 +158,8 @@ middleware, see the :ref:`downloader middleware usage guide
For a list of the components enabled by default (and their orders) see the
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting.
.. _cookies-mw:
CookiesMiddleware
-----------------
@ -166,7 +168,36 @@ CookiesMiddleware
.. class:: CookiesMiddleware
This middleware enables working with sites that need cookies.
This middleware enables working with sites that need cookies. It keeps track
of merging cookies sent by servers, so that they're send in future requests
for that spider, just like a web browser would do.
The following settings can be used to configure the cookie middleware:
* :setting:`COOKIES_DEBUG`
.. setting:: COOKIES_DEBUG
COOKIES_DEBUG
~~~~~~~~~~~~~
Default: ``False``
If enabled, Scrapy will log all cookies sent in requests (ie. ``Cookie``
header) and all cookies received in responses (ie. ``Set-Cookie`` header).
Here's an example of a log with :setting:`COOKIES_DEBUG` enabled::
2011-04-06 14:35:10-0300 [diningcity] INFO: Spider opened
2011-04-06 14:35:10-0300 [diningcity] DEBUG: Sending cookies to: <GET http://www.diningcity.com/netherlands/index.html>
Cookie: clientlanguage_nl=en_EN
2011-04-06 14:35:14-0300 [diningcity] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html>
Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/
Set-Cookie: ip_isocode=US
Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/
2011-04-06 14:49:50-0300 [diningcity] DEBUG: Crawled (200) <GET http://www.diningcity.com/netherlands/index.html> (referer: None)
[...]
DefaultHeadersMiddleware
------------------------
@ -233,8 +264,18 @@ HttpCacheMiddleware
downloads every time) and for trying your spider offline, when an Internet
connection is not available.
File system storage
~~~~~~~~~~~~~~~~~~~
Scrapy ships with two storage backends for the HTTP cache middleware:
* :ref:`httpcache-fs-backend`
* :ref:`httpcache-dbm-backend`
You can change the storage backend with the :setting:`HTTPCACHE_STORAGE`
setting. Or you can also implement your own backend.
.. _httpcache-fs-backend:
File system backend (default)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By default, the :class:`HttpCacheMiddleware` uses a file system storage with the following structure:
@ -257,8 +298,19 @@ inefficient in many file systems). An example directory could be::
/path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
The cache storage backend can be changed with the :setting:`HTTPCACHE_STORAGE`
setting, but no other backend is provided with Scrapy yet.
.. _httpcache-dbm-backend:
DBM storage backend
~~~~~~~~~~~~~~~~~~~
.. versionadded:: 0.13
A DBM_ storage backend is also available for the HTTP cache middleware. To use
it (instead of the default filesystem backend) set :setting:`HTTPCACHE_STORAGE`
to ``scrapy.contrib.httpcache.DbmCacheStorage``.
By default, it uses the anydbm_ module, but you can change it with the
:setting:`HTTPCACHE_DBM_MODULE` setting.
Settings
~~~~~~~~
@ -346,6 +398,18 @@ Default: ``'scrapy.contrib.downloadermiddleware.httpcache.FilesystemCacheStorage
The class which implements the cache storage backend.
.. setting:: HTTPCACHE_DBM_MODULE
HTTPCACHE_DBM_MODULE
^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 0.13
Default: ``'anydbm'``
The database module to use in the :ref:`DBM storage backend
<httpcache-dbm-backend>`. This setting is specific to the DBM backend.
HttpCompressionMiddleware
-------------------------
@ -491,3 +555,6 @@ UserAgentMiddleware
In order for a spider to override the default user agent, its `user_agent`
attribute must be set.
.. _DBM: http://en.wikipedia.org/wiki/Dbm
.. _anydbm: http://docs.python.org/library/anydbm.html

6
docs/topics/exporters.rst
View File

@ -264,7 +264,7 @@ XmlItemExporter
CsvItemExporter
---------------
.. class:: CsvItemExporter(file, include_headers_line=True, \**kwargs)
.. class:: CsvItemExporter(file, include_headers_line=True, join_multivalued=',', \**kwargs)
Exports Items in CSV format to the given file-like object. If the
:attr:`fields_to_export` attribute is set, it will be used to define the
@ -278,6 +278,10 @@ CsvItemExporter
:attr:`BaseItemExporter.fields_to_export` or the first exported item fields.
:type include_headers_line: boolean
:param join_multivalued: The char (or chars) that will be used for joining
multi-valued fields, if found.
:type include_headers_line: str
The additional keyword arguments of this constructor are passed to the
:class:`BaseItemExporter` constructor, and the leftover arguments to the
`csv.writer`_ constructor, so you can use any `csv.writer` constructor

2
docs/topics/request-response.rst
View File

@ -68,6 +68,8 @@ Request objects
request_with_cookies = Request(url="http://www.example.com",
cookies={currency: 'USD', country: 'UY'},
meta={'dont_merge_cookies': True})
For more info see :ref:`cookies-mw`.
:type cookies: dict
:param encoding: the encoding of this request (defaults to ``'utf-8'``).

2
docs/topics/scrapyd.rst
View File

@ -86,7 +86,7 @@ in your Ubuntu servers.
So, if you plan to deploy Scrapyd on a Ubuntu server, just add the Ubuntu
repositories as described in :ref:`topics-ubuntu` and then run::
aptitude install scrapyd-0.12
aptitude install scrapyd-0.13
This will install Scrapyd in your Ubuntu server creating a ``scrapy`` user
which Scrapyd will run as. It will also create some directories and files that

2
docs/topics/ubuntu.rst
View File

@ -13,7 +13,7 @@ latest bug fixes.
To use the packages, just add the following line to your
``/etc/apt/sources.list``, and then run ``aptitude update`` and ``aptitude
install scrapy-0.12``::
install scrapy-0.13``::
deb http://archive.scrapy.org/ubuntu DISTRO main

4
scrapy/__init__.py
View File

@ -2,8 +2,8 @@
Scrapy - a screen scraping framework written in Python
"""
version_info = (0, 12, 0)
__version__ = "0.12.0"
version_info = (0, 13, 0)
__version__ = "0.13.0"
import sys, os, warnings

30
scrapy/commands/deploy.py
View File

@ -57,6 +57,8 @@ class Command(ScrapyCommand):
help="list available projects on TARGET")
parser.add_option("--egg", metavar="FILE",
help="use the given egg, instead of building it")
parser.add_option("--build-egg", metavar="FILE",
help="only build the egg, don't deploy it")
def run(self, args, opts):
try:
@ -75,18 +77,26 @@ class Command(ScrapyCommand):
projects = json.loads(f.read())['projects']
print os.linesep.join(projects)
return
target_name = _get_target_name(args)
target = _get_target(target_name)
project = _get_project(target, opts)
version = _get_version(target, opts)
tmpdir = None
if opts.egg:
_log("Using egg: %s" % opts.egg)
egg = opts.egg
else:
_log("Building egg of %s-%s" % (project, version))
if opts.build_egg: # build egg only
egg, tmpdir = _build_egg()
_upload_egg(target, egg, project, version)
_log("Writing egg to %s" % opts.build_egg)
shutil.copyfile(egg, opts.build_egg)
else: # buld egg and deploy
target_name = _get_target_name(args)
target = _get_target(target_name)
project = _get_project(target, opts)
version = _get_version(target, opts)
if opts.egg:
_log("Using egg: %s" % opts.egg)
egg = opts.egg
else:
_log("Building egg of %s-%s" % (project, version))
egg, tmpdir = _build_egg()
_upload_egg(target, egg, project, version)
if tmpdir:
shutil.rmtree(tmpdir)

30
scrapy/contrib/downloadermiddleware/cookies.py
View File

@ -1,3 +1,4 @@
import os
from collections import defaultdict
from scrapy.xlib.pydispatch import dispatcher
@ -28,7 +29,7 @@ class CookiesMiddleware(object):
# set Cookie header
request.headers.pop('Cookie', None)
jar.add_cookie_header(request)
self._debug_cookie(request)
self._debug_cookie(request, spider)
def process_response(self, request, response, spider):
if 'dont_merge_cookies' in request.meta:
@ -37,31 +38,28 @@ class CookiesMiddleware(object):
# extract cookies from Set-Cookie and drop invalid/expired cookies
jar = self.jars[spider]
jar.extract_cookies(response, request)
self._debug_set_cookie(response)
self._debug_set_cookie(response, spider)
return response
def spider_closed(self, spider):
self.jars.pop(spider, None)
def _debug_cookie(self, request):
"""log Cookie header for request"""
def _debug_cookie(self, request, spider):
if self.debug:
c = request.headers.get('Cookie')
c = c and [p.split('=')[0] for p in c.split(';')]
log.msg('Cookie: %s for %s' % (c, request.url), level=log.DEBUG)
cl = request.headers.getlist('Cookie')
if cl:
msg = "Sending cookies to: %s" % request + os.linesep
msg += os.linesep.join("Cookie: %s" % c for c in cl)
log.msg(msg, spider=spider, level=log.DEBUG)
def _debug_set_cookie(self, response):
"""log Set-Cookies headers but exclude cookie values"""
def _debug_set_cookie(self, response, spider):
if self.debug:
cl = response.headers.getlist('Set-Cookie')
res = []
for c in cl:
kv, tail = c.split(';', 1)
k = kv.split('=', 1)[0]
res.append('%s %s' % (k, tail))
log.msg('Set-Cookie: %s from %s' % (res, response.url))
if cl:
msg = "Received cookies from: %s" % response + os.linesep
msg += os.linesep.join("Set-Cookie: %s" % c for c in cl)
log.msg(msg, spider=spider, level=log.DEBUG)
def _get_request_cookies(self, jar, request):
headers = {'Set-Cookie': ['%s=%s;' % (k, v) for k, v in request.cookies.iteritems()]}

4
scrapy/contrib/downloadermiddleware/retry.py
View File

@ -30,9 +30,11 @@ from scrapy.conf import settings
class RetryMiddleware(object):
# IOError is raised by the HttpCompression middleware when trying to
# decompress an empty response
EXCEPTIONS_TO_RETRY = (ServerTimeoutError, UserTimeoutError, DNSLookupError,
ConnectionRefusedError, ConnectionDone, ConnectError,
ConnectionLost, PartialDownloadError)
ConnectionLost, PartialDownloadError, IOError)
def __init__(self):
self.max_retry_times = settings.getint('RETRY_TIMES')

11
scrapy/contrib/exporter/__init__.py
View File

@ -143,11 +143,20 @@ class XmlItemExporter(BaseItemExporter):
class CsvItemExporter(BaseItemExporter):
def __init__(self, file, include_headers_line=True, **kwargs):
def __init__(self, file, include_headers_line=True, join_multivalued=',', **kwargs):
self._configure(kwargs, dont_fail=True)
self.include_headers_line = include_headers_line
self.csv_writer = csv.writer(file, **kwargs)
self._headers_not_written = True
self._join_multivalued = join_multivalued
def _to_str_if_unicode(self, value):
if isinstance(value, (list, tuple)):
try:
value = self._join_multivalued.join(value)
except TypeError: # list in value may not contain strings
pass
return super(CsvItemExporter, self)._to_str_if_unicode(value)
def export_item(self, item):
if self._headers_not_written:

64
scrapy/contrib/httpcache.py Normal file
View File

@ -0,0 +1,64 @@
from __future__ import with_statement
import os
from time import time
import cPickle as pickle
from scrapy.http import Headers
from scrapy.core.downloader.responsetypes import responsetypes
from scrapy.utils.request import request_fingerprint
from scrapy.utils.project import data_path
from scrapy import conf
class DbmCacheStorage(object):
def __init__(self, settings=conf.settings):
self.cachedir = data_path(settings['HTTPCACHE_DIR'])
self.expiration_secs = settings.getint('HTTPCACHE_EXPIRATION_SECS')
self.dbmodule = __import__(settings['HTTPCACHE_DBM_MODULE'])
self.dbs = {}
def open_spider(self, spider):
dbpath = os.path.join(self.cachedir, '%s.db' % spider.name)
self.dbs[spider] = self.dbmodule.open(dbpath, 'c')
def close_spider(self, spider):
self.dbs[spider].close()
def retrieve_response(self, spider, request):
data = self._read_data(spider, request)
if data is None:
return # not cached
url = data['url']
status = data['status']
headers = Headers(data['headers'])
body = data['body']
respcls = responsetypes.from_args(headers=headers, url=url)
response = respcls(url=url, headers=headers, status=status, body=body)
return response
def store_response(self, spider, request, response):
key = self._request_key(request)
data = {
'status': response.status,
'url': response.url,
'headers': dict(response.headers),
'body': response.body,
}
self.dbs[spider]['%s_data' % key] = pickle.dumps(data, protocol=2)
self.dbs[spider]['%s_time' % key] = str(time())
def _read_data(self, spider, request):
key = self._request_key(request)
db = self.dbs[spider]
tkey = '%s_time' % key
if not db.has_key(tkey):
return # not found
ts = db[tkey]
if 0 < self.expiration_secs < time() - float(ts):
return # expired
return pickle.loads(db['%s_data' % key])
def _request_key(self, request):
return request_fingerprint(request)

11
scrapy/contrib/ibl/descriptor.py
View File

@ -3,25 +3,22 @@ Extended types for IBL extraction
"""
from itertools import chain
from scrapy.contrib.ibl.extractors import text
from scrapy.contrib.ibl.extractors import text, html
class FieldDescriptor(object):
"""description of a scraped attribute"""
__slots__ = ('name', 'description', 'extractor', 'required', 'allow_markup')
__slots__ = ('name', 'description', 'extractor', 'required')
def __init__(self, name, description, extractor=text, required=False,
allow_markup=False):
def __init__(self, name, description, extractor=text, required=False):
self.name = name
self.description = description
self.extractor = extractor
self.required = required
self.allow_markup = allow_markup
@classmethod
def from_field(cls, name, field):
return cls(name, field.get('description'), \
field.get('ibl_extractor', text), field.get('required', False), \
field.get('allow_markup', False))
field.get('ibl_extractor', text), field.get('required', False))
def __str__(self):
return "FieldDescriptor(%s)" % self.name

147
scrapy/contrib/ibl/extraction/pageobjects.py
View File

@ -4,16 +4,14 @@ Page objects
This module contains objects representing pages and parts of pages (e.g. tokens
and annotations) used in the instance based learning algorithm.
"""
from itertools import chain
from numpy import array, ndarray
from scrapy.contrib.ibl.htmlpage import HtmlTagType
from scrapy.contrib.ibl.htmlpage import HtmlTagType, HtmlPageRegion
class TokenType(object):
class TokenType(HtmlTagType):
"""constants for token types"""
WORD = 0
OPEN_TAG = HtmlTagType.OPEN_TAG
CLOSE_TAG = HtmlTagType.CLOSE_TAG
NON_PAIRED_TAG = HtmlTagType.UNPAIRED_TAG
class TokenDict(object):
"""Mapping from parse tokens to integers
@ -68,6 +66,36 @@ class TokenDict(object):
templates = ["%s", "<%s>", "</%s>", "<%s/>"]
return templates[tid >> 24] % self.find_token(tid)
class PageRegion(object):
"""A region in a page, defined by a start and end index"""
__slots__ = ('start_index', 'end_index')
def __init__(self, start, end):
self.start_index = start
self.end_index = end
def __str__(self):
return "%s(%s, %s)" % (self.__class__.__name__, self.start_index,
self.end_index)
def __repr__(self):
return str(self)
class FragmentedHtmlPageRegion(HtmlPageRegion):
"""An HtmlPageRegion consisting of possibly non-contiguous sub-regions"""
def __new__(cls, htmlpage, regions):
text = u''.join(regions)
return HtmlPageRegion.__new__(cls, htmlpage, text)
def __init__(self, htmlpage, regions):
self.htmlpage = htmlpage
self.regions = regions
@property
def parsed_fragments(self):
return chain(*(r.parsed_fragments for r in self.regions))
class Page(object):
"""Basic representation of a page. This consists of a reference to a
dictionary of tokens and an array of raw token ids
@ -92,7 +120,8 @@ class TemplatePage(Page):
annotations = sorted(annotations, key=lambda x: x.end_index, reverse=True)
self.annotations = sorted(annotations, key=lambda x: x.start_index)
self.id = template_id
self.ignored_regions = ignored_regions or []
self.ignored_regions = [i if isinstance(i, PageRegion) else PageRegion(*i) \
for i in (ignored_regions or [])]
self.extra_required_attrs = set(extra_required or [])
def __str__(self):
@ -107,66 +136,53 @@ class ExtractionPage(Page):
"""Parsed data belonging to a web page upon which we wish to perform
extraction.
"""
__slots__ = ('text',
'token_start_indexes', # index in text of the start of a token
'token_follow_indexes', # index in text of data following token
'tag_attributes' # a map from token index to tag attributes
)
__slots__ = ('htmlpage', 'token_page_indexes')
def __init__(self, text, token_dict, page_tokens, token_start_indexes,
token_follow_indexes, tag_attributes):
def __init__(self, htmlpage, token_dict, page_tokens, token_page_indexes):
"""Construct a new ExtractionPage
Arguments:
`htmlpage`: The source HtmlPage
`token_dict`: Token Dictionary used for tokenization
`page_tokens': array of page tokens for matching
`token_page_indexes`: indexes of each token in the parsed htmlpage
"""
Page.__init__(self, token_dict, page_tokens)
self.text = text
self.token_start_indexes = token_start_indexes
self.token_follow_indexes = token_follow_indexes
self.tag_attributes = tag_attributes
def token_html(self, token_index):
"""The raw html for a page token at the given index in the page_tokens
list
"""
text_start = self.token_start_indexes[token_index]
text_end = self.token_follow_indexes[token_index]
return self.text[text_start:text_end]
def html_between_tokens(self, start_token_index, end_token_index):
"""The raw html between the tokens at the specified indexes in the
page_tokens list
This assumes start_token_index <= end_token_index
"""
text_start = self.token_follow_indexes[start_token_index]
text_end = self.token_start_indexes[end_token_index]
return self.text[text_start:text_end]
self.htmlpage = htmlpage
self.token_page_indexes = token_page_indexes
def text_between_tokens(self, start_token_index, end_token_index,
tag_replacement=u' '):
"""The text between the the tokens at the specified indexes in the
page_tokens list. Tags are replaced by tag_replacement (default one space
character)
"""
return tag_replacement.join([self.text[
self.token_follow_indexes[i]:self.token_start_indexes[i+1]] \
for i in xrange(start_token_index, end_token_index)])
def htmlpage_region(self, start_token_index, end_token_index):
"""The region in the HtmlPage corresonding to the area defined by
the start_token_index and the end_token_index
def tag_attribute(self, token_index, attribute):
"""The value of a tag attribute. The tag is identified by its
corresponding token index.
If the tag or attribute is not present, None is returned
This includes the tokens at the specified indexes
"""
return self.tag_attributes.get(token_index, {}).get(attribute)
start = self.token_page_indexes[start_token_index]
end = self.token_page_indexes[end_token_index]
return self.htmlpage.subregion(start, end)
def htmlpage_region_inside(self, start_token_index, end_token_index):
"""The region in the HtmlPage corresonding to the area between
the start_token_index and the end_token_index.
This excludes the tokens at the specified indexes
"""
start = self.token_page_indexes[start_token_index] + 1
end = self.token_page_indexes[end_token_index] - 1
return self.htmlpage.subregion(start, end)
def htmlpage_tag(self, token_index):
"""The HtmlPage tag at corresponding to the token at token_index"""
return self.htmlpage.parsed_body[self.token_page_indexes[token_index]]
def __str__(self):
summary = []
for (token, start, follow) in zip(self.page_tokens, self.token_start_indexes,
self.token_follow_indexes):
text = "%s %s-%s (%s)" % (self.token_dict.find_token(token), start, follow,
self.text[start:follow])
for token, tindex in zip(self.page_tokens, self.token_page_indexes):
text = "%s page[%s]: %s" % (self.token_dict.find_token(token),
tindex, self.htmlpage.parsed_body[tindex])
summary.append(text)
return "ExtractionPage\n==============\nTokens: %s\n\nRaw text: %s\n\n" \
"Tag attributes: %s\n" % ('\n'.join(summary), self.text,
self.tag_attributes)
% ('\n'.join(summary), self.htmlpage.body)
class AnnotationText(object):
__slots__ = ('start_text', 'follow_text')
@ -179,7 +195,8 @@ class AnnotationText(object):
return "AnnotationText(%s..%s)" % \
(repr(self.start_text), repr(self.follow_text))
class AnnotationTag(object):
class AnnotationTag(PageRegion):
"""A tag that annotates part of the document
It has the following properties:
@ -197,8 +214,7 @@ class AnnotationTag(object):
def __init__(self, start_index, end_index, surrounds_attribute=None,
annotation_text=None, tag_attributes=None, variant_id=None):
self.start_index = start_index
self.end_index = end_index
PageRegion.__init__(self, start_index, end_index)
self.surrounds_attribute = surrounds_attribute
self.annotation_text = annotation_text
self.tag_attributes = tag_attributes or []
@ -213,16 +229,3 @@ class AnnotationTag(object):
def __repr__(self):
return str(self)
class LabelledRegion(object):
__slots__ = ('start_index', 'end_index')
def __init__(self, start, end):
self.start_index = start
self.end_index = end
def __str__(self):
return "LabelledRegion (%s, %s)" % (self.start_index, self.end_index)
def __repr__(self):
return str(self)

37
scrapy/contrib/ibl/extraction/pageparsing.py
View File

@ -9,8 +9,8 @@ from numpy import array
from scrapy.utils.py26 import json
from scrapy.contrib.ibl.htmlpage import HtmlTagType, HtmlTag, HtmlPage
from scrapy.contrib.ibl.extraction.pageobjects import (AnnotationTag,
TemplatePage, ExtractionPage, AnnotationText, TokenDict)
from scrapy.contrib.ibl.extraction.pageobjects import (AnnotationTag,
TemplatePage, ExtractionPage, AnnotationText, TokenDict, FragmentedHtmlPageRegion)
def parse_strings(template_html, extraction_html):
"""Create a template and extraction page from raw strings
@ -52,18 +52,18 @@ class InstanceLearningParser(object):
def feed(self, html_page):
self.html_page = html_page
self.previous_element_class = None
for data in html_page.parsed_body:
for index, data in enumerate(html_page.parsed_body):
if isinstance(data, HtmlTag):
self._add_token(data.tag, data.tag_type, data.start, data.end)
self.handle_tag(data)
self.handle_tag(data, index)
else:
self.handle_data(data)
self.handle_data(data, index)
self.previous_element_class = data.__class__
def handle_data(self, html_data_fragment):
def handle_data(self, html_data_fragment, index):
pass
def handle_tag(self, html_tag):
def handle_tag(self, html_tag, index):
pass
_END_UNPAIREDTAG_TAGS = ["form", "div", "p", "table", "tr", "td"]
@ -86,7 +86,7 @@ class TemplatePageParser(InstanceLearningParser):
self.last_text_region = None
self.next_tag_index = 0
def handle_tag(self, html_tag):
def handle_tag(self, html_tag, index):
if self.last_text_region:
self._process_text('')
@ -275,7 +275,7 @@ class TemplatePageParser(InstanceLearningParser):
if prev != annotation.variant_id:
raise ValueError("unbalanced variant annotation tags")
def handle_data(self, html_data_fragment):
def handle_data(self, html_data_fragment, index):
fragment_text = self.html_page.fragment_data(html_data_fragment)
self._process_text(fragment_text)
@ -300,20 +300,11 @@ class ExtractionPageParser(InstanceLearningParser):
"""
def __init__(self, token_dict):
InstanceLearningParser.__init__(self, token_dict)
self.page_data = []
self.token_start_index = []
self.token_follow_index = []
self.tag_attrs = {}
self._page_token_indexes = []
def _add_token(self, token, token_type, start, end):
InstanceLearningParser._add_token(self, token, token_type, start, end)
self.token_start_index.append(start)
self.token_follow_index.append(end)
def handle_tag(self, html_tag):
if html_tag.attributes:
self.tag_attrs[len(self.token_list) - 1] = html_tag.attributes
def handle_tag(self, html_tag, index):
self._page_token_indexes.append(index)
def to_extraction_page(self):
return ExtractionPage(self.html_page.body, self.token_dict, array(self.token_list),
self.token_start_index, self.token_follow_index, self.tag_attrs)
return ExtractionPage(self.html_page, self.token_dict, array(self.token_list),
self._page_token_indexes)

110
scrapy/contrib/ibl/extraction/regionextract.py
View File

@ -8,14 +8,16 @@ import operator
import copy
import pprint
import cStringIO
from itertools import groupby
from itertools import groupby, izip, starmap
from numpy import array
from scrapy.contrib.ibl.descriptor import FieldDescriptor
from scrapy.contrib.ibl.htmlpage import HtmlPageRegion
from scrapy.contrib.ibl.extraction.similarity import (similar_region,
longest_unique_subsequence, common_prefix)
from scrapy.contrib.ibl.extraction.pageobjects import AnnotationTag, LabelledRegion
from scrapy.contrib.ibl.extraction.pageobjects import (AnnotationTag,
PageRegion, FragmentedHtmlPageRegion)
def build_extraction_tree(template, type_descriptor, trace=True):
"""Build a tree of region extractors corresponding to the
@ -33,16 +35,14 @@ def build_extraction_tree(template, type_descriptor, trace=True):
return TemplatePageExtractor(template, extractors)
_ID = lambda x: x
_EXTRACT_HTML = lambda x: x
_DEFAULT_DESCRIPTOR = FieldDescriptor('none', None)
def _labelled(obj):
"""
Returns labelled element of the object (extractor or labelled region)
"""
if hasattr(obj, "annotation"):
return obj.annotation
return obj
return getattr(obj, 'annotation', obj)
def _compose(f, g):
"""given unary functions f and g, return a function that computes f(g(x))
@ -75,7 +75,7 @@ class BasicTypeExtractor(object):
u'<div data-scrapy-annotate="{&quot;annotations&quot;: {&quot;content&quot;: &quot;name&quot;}}">x<b> xx</b></div>',\
u'<div>a name<b> id-9</b></div>')
>>> ex = BasicTypeExtractor(template.annotations[0])
>>> ex.extract(page, 0, 3, [LabelledRegion(*(1,2))])
>>> ex.extract(page, 0, 3, [PageRegion(1, 2)])
[(u'name', u'a name')]
"""
@ -88,17 +88,15 @@ class BasicTypeExtractor(object):
descriptor = attribute_descriptors.get(annotation.surrounds_attribute)
if descriptor:
self.content_validate = descriptor.extractor
self.allow_markup = descriptor.allow_markup
else:
self.content_validate = _ID
self.allow_markup = False
self.content_validate = _EXTRACT_HTML
self.extract = self._extract_content
if annotation.tag_attributes:
self.tag_data = []
for (tag_attr, extraction_attr) in annotation.tag_attributes:
descriptor = attribute_descriptors.get(extraction_attr)
extractf = descriptor.extractor if descriptor else _ID
extractf = descriptor.extractor if descriptor else _EXTRACT_HTML
self.tag_data.append((extractf, tag_attr, extraction_attr))
self.extract = self._extract_both if \
@ -109,33 +107,32 @@ class BasicTypeExtractor(object):
self._extract_attribute(page, start_index, end_index, ignored_regions)
def _extract_content(self, extraction_page, start_index, end_index, ignored_regions=None):
# we might want to add opening/closing ul/ol/table if we have the
# middle of a region. This would require support in the scrapy
# cleansing.
complete_data = ""
start = start_index
end = ignored_regions[0].start_index if ignored_regions else end_index
while start is not None:
if self.allow_markup:
data = extraction_page.html_between_tokens(start, end)
else:
data = extraction_page.text_between_tokens(start, end)
complete_data += data
if ignored_regions:
start = ignored_regions[0].end_index
ignored_regions.pop(0)
end = ignored_regions[0].start_index if ignored_regions else end_index
else:
start = None
complete_data = self.content_validate(complete_data)
return [(self.annotation.surrounds_attribute, complete_data)] if complete_data else []
# extract content between annotation indexes
if not ignored_regions:
region = extraction_page.htmlpage_region_inside(start_index, end_index)
else:
# assumes ignored_regions are completely contained within start and end index
assert (start_index <= ignored_regions[0].start_index and
end_index >= ignored_regions[-1].end_index)
starts = [start_index] + [i.end_index for i in ignored_regions]
ends = [i.start_index for i in ignored_regions]
if starts[-1] is not None:
ends.append(end_index)
included_regions = izip(starts, ends)
if ends[0] is None:
included_regions.next()
regions = starmap(extraction_page.htmlpage_region_inside, included_regions)
region = FragmentedHtmlPageRegion(extraction_page.htmlpage, list(regions))
validated = self.content_validate(region)
return [(self.annotation.surrounds_attribute, validated)] if validated else []
def _extract_attribute(self, extraction_page, start_index, end_index, ignored_regions=None):
data = []
for (f, ta, ea) in self.tag_data:
tag_value = extraction_page.tag_attribute(start_index, ta)
tag_value = extraction_page.htmlpage_tag(start_index).attributes.get(ta)
if tag_value:
extracted = f(tag_value)
region = HtmlPageRegion(extraction_page.htmlpage, tag_value)
extracted = f(region)
if extracted is not None:
data.append((ea, extracted))
return data
@ -178,10 +175,8 @@ class BasicTypeExtractor(object):
def __str__(self):
messages = ['BasicTypeExtractor(']
if self.annotation.surrounds_attribute:
messages += [self.annotation.surrounds_attribute, ': ',
'html content' if self.allow_markup else 'text content',
]
if self.content_validate != _ID:
messages.append(self.annotation.surrounds_attribute)
if self.content_validate != _EXTRACT_HTML:
messages += [', extracted with \'',
self.content_validate.__name__, '\'']
@ -190,7 +185,7 @@ class BasicTypeExtractor(object):
messages.append(';')
for (f, ta, ea) in self.tag_data:
messages += [ea, ': tag attribute "', ta, '"']
if f != _ID:
if f != _EXTRACT_HTML:
messages += [', validated by ', str(f)]
messages.append(", template[%s:%s])" % \
(self.annotation.start_index, self.annotation.end_index))
@ -340,7 +335,8 @@ class RecordExtractor(object):
The region in the page to be extracted from may be specified using
start_index and end_index
"""
ignored_regions = [i if isinstance(i, LabelledRegion) else LabelledRegion(*i) for i in (ignored_regions or [])]
if ignored_regions is None:
ignored_regions = []
region_elements = sorted(self.extractors + ignored_regions, key=lambda x: _labelled(x).start_index)
_, _, attributes = self._doextract(page, region_elements, start_index,
end_index)
@ -395,7 +391,7 @@ class RecordExtractor(object):
s, p, e = similar_region(page.page_tokens, self.template_tokens, \
i, start, sindex)
if s > 0:
similar_ignored_regions.append(LabelledRegion(*(p, e)))
similar_ignored_regions.append(PageRegion(p, e))
start = e or start
extracted_data = first_region.extract(page, pindex, sindex, similar_ignored_regions)
if extracted_data:
@ -493,12 +489,12 @@ class TraceExtractor(object):
for t in template.page_tokens[tend:tend+5]])
def summarize_trace(self, page, start, end, ret):
text_start = page.token_follow_indexes[start]
text_end = page.token_start_indexes[end or -1]
text_start = page.htmlpage.parsed_body[page.token_page_indexes[start]].start
text_end = page.htmlpage.parsed_body[page.token_page_indexes[end or -1]].end
page_snippet = "(...%s)%s(%s...)" % (
page.text[text_start-50:text_start].replace('\n', ' '),
page.text[text_start:text_end],
page.text[text_end:text_end+50].replace('\n', ' '))
page.htmlpage.body[text_start-50:text_start].replace('\n', ' '),
page.htmlpage.body[text_start:text_end],
page.htmlpage.body[text_end:text_end+50].replace('\n', ' '))
pre_summary = "\nstart %s page[%s:%s]\n" % (self.traced.__class__.__name__, start, end)
post_summary = """
%s page[%s:%s]
@ -572,25 +568,26 @@ class TemplatePageExtractor(object):
_tokenize = re.compile(r'\w+|[^\w\s]+', re.UNICODE | re.MULTILINE | re.DOTALL).findall
class TextRegionDataExtractor(object):
"""Data Extractor for extracting text fragments from within a larger
body of text. It extracts based on the longest unique prefix and suffix.
"""Data Extractor for extracting text fragments from an annotation page
fragment or string. It extracts based on the longest unique prefix and
suffix.
for example:
>>> extractor = TextRegionDataExtractor('designed by ', '.')
>>> extractor.extract("by Marc Newson.")
>>> extractor.extract_text("by Marc Newson.")
'Marc Newson'
Both prefix and suffix are optional:
>>> extractor = TextRegionDataExtractor('designed by ')
>>> extractor.extract("by Marc Newson.")
>>> extractor.extract_text("by Marc Newson.")
'Marc Newson.'
>>> extractor = TextRegionDataExtractor(suffix='.')
>>> extractor.extract("by Marc Newson.")
>>> extractor.extract_text("by Marc Newson.")
'by Marc Newson'
It requires a minimum match of at least one word or punctuation character:
>>> extractor = TextRegionDataExtractor('designed by')
>>> extractor.extract("y Marc Newson.") is None
>>> extractor.extract_text("y Marc Newson.") is None
True
"""
def __init__(self, prefix=None, suffix=None):
@ -609,8 +606,13 @@ class TextRegionDataExtractor(object):
tokens = _tokenize(matchstring or '')
return len(tokens[0]) if tokens else 0
def extract(self, text):
"""attempt to extract a substring from the text"""
def extract(self, region):
"""Extract a region from the region passed"""
text = self.extract_text(region)
return HtmlPageRegion(region.htmlpage, text) if text else None
def extract_text(self, text):
"""Extract a substring from the text"""
pref_index = 0
if self.minprefix > 0:
rev_idx, plen = longest_unique_subsequence(text[::-1], self.prefix)
@ -623,5 +625,3 @@ class TextRegionDataExtractor(object):
if slen < self.minsuffix:
return None
return text[pref_index:pref_index + sidx]

181
scrapy/contrib/ibl/extractors.py
View File

@ -4,8 +4,9 @@ Extractors for attributes
import re
import urlparse
from scrapy.utils.markup import remove_entities
from scrapy.utils.markup import remove_entities, remove_comments
from scrapy.utils.url import safe_url_string
from scrapy.contrib.ibl.htmlpage import HtmlTag, HtmlTagType
#FIXME: the use of "." needs to be localized
_NUMERIC_ENTITIES = re.compile("&#([0-9]+)(?:;|\s)", re.U)
@ -22,11 +23,175 @@ _CSS_IMAGERE = re.compile("background(?:-image)?\s*:\s*url\((.*?)\)", re.I)
_BASE_PATH_RE = "/?(?:[^/]+/)*(?:.+%s)"
_IMAGE_PATH_RE = re.compile(_BASE_PATH_RE % '\.(?:%s)' % _IMAGES_TYPES, re.I)
_GENERIC_PATH_RE = re.compile(_BASE_PATH_RE % '', re.I)
_WS = re.compile("\s+", re.U)
def text(txt):
stripped = txt.strip() if txt else None
if stripped:
return stripped
# tags to keep (only for attributes with markup)
_TAGS_TO_KEEP = frozenset(['br', 'p', 'big', 'em', 'small', 'strong', 'sub',
'sup', 'ins', 'del', 'code', 'kbd', 'samp', 'tt', 'var', 'pre', 'listing',
'plaintext', 'abbr', 'acronym', 'address', 'bdo', 'blockquote', 'q',
'cite', 'dfn', 'table', 'tr', 'th', 'td', 'tbody', 'ul', 'ol', 'li', 'dl',
'dd', 'dt'])
# tag names to be replaced by other tag names (overrides tags_to_keep)
_TAGS_TO_REPLACE = {
'h1': 'strong',
'h2': 'strong',
'h3': 'strong',
'h4': 'strong',
'h5': 'strong',
'h6': 'strong',
'b' : 'strong',
'i' : 'em',
}
# tags whoose content will be completely removed (recursively)
# (overrides tags_to_keep and tags_to_replace)
_TAGS_TO_PURGE = ('script', 'img', 'input')
def htmlregion(text):
"""convenience function to make an html region from text.
This is useful for testing
"""
from scrapy.contrib.ibl.htmlpage import HtmlPage
return HtmlPage(body=text).subregion()
def notags(region, tag_replace=u' '):
"""Removes all html tags"""
fragments = getattr(region, 'parsed_fragments', None)
if fragments is None:
return region
page = region.htmlpage
data = [page.fragment_data(f) for f in fragments if not isinstance(f, HtmlTag)]
return tag_replace.join(data)
def text(region):
"""Converts HTML to text. There is no attempt at formatting other than
removing excessive whitespace,
For example:
>>> t = lambda s: text(htmlregion(s))
>>> t(u'<h1>test</h1>')
u'test'
Leading and trailing whitespace are removed
>>> t(u'<h1> test</h1> ')
u'test'
Comments are removed
>>> t(u'test <!-- this is a comment --> me')
u'test me'
Text between script tags is ignored
>>> t(u"scripts are<script>n't</script> ignored")
u'scripts are ignored'
HTML entities are converted to text
>>> t(u"only &pound;42")
u'only \\xa342'
"""
chunks = _process_markup(region,
lambda text: remove_entities(text, encoding=region.htmlpage.encoding),
lambda tag: u' '
)
text = u''.join(chunks)
return _WS.sub(u' ', text).strip()
def safehtml(region, allowed_tags=_TAGS_TO_KEEP, replace_tags=_TAGS_TO_REPLACE):
"""Creates an HTML subset, using a whitelist of HTML tags.
The HTML generated is safe for display on a website,without escaping and
should not cause formatting problems.
Allowed_tags is a set of tags that are allowed and replace_tags is a mapping of
tags to alternative tags to substitute.
For example:
>>> t = lambda s: safehtml(htmlregion(s))
>>> t(u'<strong>test <blink>test</blink></strong>')
u'<strong>test test</strong>'
Some tags, like script, are completely removed
>>> t(u'<script>test </script>test')
u'test'
replace_tags define tags that are converted. By default all headers, bold and indenting
are converted to strong and em.
>>> t(u'<h2>header</h2> test <b>bold</b> <i>indent</i>')
u'<strong>header</strong> test <strong>bold</strong> <em>indent</em>'
Comments are stripped, but entities are not converted
>>> t(u'<!-- comment --> only &pound;42')
u'only &pound;42'
Paired tags are closed
>>> t(u'<p>test')
u'<p>test</p>'
>>> t(u'<p>test <i><br/><b>test</p>')
u'<p>test <em><br/><strong>test</strong></em></p>'
"""
tagstack = []
def _process_tag(tag):
tagstr = replace_tags.get(tag.tag, tag.tag)
if tagstr not in allowed_tags:
return
if tag.tag_type == HtmlTagType.OPEN_TAG:
tagstack.append(tagstr)
return u"<%s>" % tagstr
elif tag.tag_type == HtmlTagType.CLOSE_TAG:
try:
last = tagstack.pop()
# common case of matching tag
if last == tagstr:
return u"</%s>" % last
# output all preceeding tags (if present)
revtags = tagstack[::-1]
tindex = revtags.index(tagstr)
del tagstack[-tindex-1:]
return u"</%s></%s>" % (last, u"></".join(revtags[:tindex+1]))
except (ValueError, IndexError):
# popped from empty stack or failed to find the tag
pass
else:
assert tag.tag_type == HtmlTagType.UNPAIRED_TAG, "unrecognised tag type"
return u"<%s/>" % tag.tag
chunks = list(_process_markup(region, lambda text: text, _process_tag)) + \
["</%s>" % t for t in reversed(tagstack)]
return u''.join(chunks).strip()
def _process_markup(region, textf, tagf):
fragments = getattr(region, 'parsed_fragments', None)
if fragments is None:
yield textf(region)
return
fiter = iter(fragments)
for fragment in fiter:
if isinstance(fragment, HtmlTag):
# skip forward to closing script tags
tag = fragment.tag
if tag in _TAGS_TO_PURGE:
# if opening, keep going until closed
if fragment.tag_type == HtmlTagType.OPEN_TAG:
for probe in fiter:
if isinstance(probe, HtmlTag) and \
probe.tag == tag and \
probe.tag_type == HtmlTagType.CLOSE_TAG:
break
else:
output = tagf(fragment)
if output:
yield output
else:
text = region.htmlpage.fragment_data(fragment)
text = remove_comments(text)
text = textf(text)
if text:
yield text
def html(pageregion):
"""A page region is already html, so this is the identity function"""
return pageregion
def contains_any_numbers(txt):
"""text that must contain at least one number
@ -132,6 +297,10 @@ def image_url(txt):
['http://s7d5.scene7.com/is/image/wasserstrom/165133?wid=227&hei=227&defaultImage=noimage_wasserstrom']
"""
imgurl = extract_image_url(txt)
return [safe_url_string(remove_entities(url(imgurl)))] if imgurl else None
def extract_image_url(txt):
txt = url(txt)
imgurl = None
if txt:
@ -153,4 +322,4 @@ def image_url(txt):
imgurl = urlparse.urlunparse(parsed)
if not imgurl:
imgurl = txt
return [safe_url_string(remove_entities(url(imgurl)))] if imgurl else None
return imgurl

69
scrapy/contrib/ibl/htmlpage.py
View File

@ -1,8 +1,9 @@
"""
htmlpage
Container object for representing html pages in the IBL system. This
encapsulates page related information and prevents parsing multiple times.
Container objects for representing html pages and their parts in the IBL
system. This encapsulates page related information and prevents parsing
multiple times.
"""
import re
import hashlib
@ -26,26 +27,82 @@ def create_page_from_jsonpage(jsonpage, body_key):
return HtmlPage(url, headers, body, page_id)
class HtmlPage(object):
def __init__(self, url=None, headers=None, body=None, page_id=None):
"""HtmlPage
This is a parsed HTML page. It contains the page headers, url, raw body and parsed
body.
The parsed body is a list of HtmlDataFragment objects.
"""
def __init__(self, url=None, headers=None, body=None, page_id=None, encoding='utf-8'):
assert isinstance(body, unicode), "unicode expected, got: %s" % type(body).__name__
self.headers = headers or {}
self.body = body
self.url = url or u''
self.encoding = encoding
if page_id is None and url:
self.page_id = hashlib.sha1(url).hexdigest()
else:
self.page_id = page_id
@classmethod
def from_response(cls, response, page_id=None):
"""Create an HtmlPage from a scrapy response"""
return HtmlPage(response.url, response.headers,
response.body_as_unicode(), page_id, response.encoding)
def _set_body(self, body):
self._body = body
self.parsed_body = list(parse_html(body))
body = property(lambda x: x._body, _set_body)
body = property(lambda x: x._body, _set_body, doc="raw html for the page")
def subregion(self, start=0, end=None):
"""HtmlPageRegion constructed from the start and end index (inclusive)
into the parsed page
"""
return HtmlPageParsedRegion(self, start, end)
def fragment_data(self, data_fragment):
"""portion of the body corresponding to the HtmlDataFragment"""
return self.body[data_fragment.start:data_fragment.end]
class HtmlPageRegion(unicode):
"""A Region of an HtmlPage that has been extracted
"""
def __new__(cls, htmlpage, data):
return unicode.__new__(cls, data)
def __init__(self, htmlpage, data):
"""Construct a new HtmlPageRegion object.
htmlpage is the original page and data is the raw html
"""
self.htmlpage = htmlpage
class HtmlPageParsedRegion(HtmlPageRegion):
"""A region of an HtmlPage that has been extracted
This has a parsed_fragments property that contains the parsed html
fragments contained within this region
"""
def __new__(cls, htmlpage, start_index, end_index):
text_start = htmlpage.parsed_body[start_index].start
text_end = htmlpage.parsed_body[end_index or -1].end
text = htmlpage.body[text_start:text_end]
return HtmlPageRegion.__new__(cls, htmlpage, text)
def __init__(self, htmlpage, start_index, end_index):
self.htmlpage = htmlpage
self.start_index = start_index
self.end_index = end_index
@property
def parsed_fragments(self):
"""HtmlDataFragment or HtmlTag objects for this parsed region"""
end = self.end_index + 1 if self.end_index is not None else None
return self.htmlpage.parsed_body[self.start_index:end]
class HtmlTagType(object):
OPEN_TAG = 1
CLOSE_TAG = 2
@ -80,8 +137,8 @@ class HtmlTag(HtmlDataFragment):
def __repr__(self):
return str(self)
_ATTR = "((?:[^=/>\s]|/(?!>))+)(?:\s*=(?:\s*\"(.*?)\"|\s*'(.*?)'|([^>\s]+))?)?"
_TAG = "<(\/?)(\w+(?::\w+)?)((?:\s+" + _ATTR + ")+\s*|\s*)(\/?)>"
_ATTR = "((?:[^=/<>\s]|/(?!>))+)(?:\s*=(?:\s*\"(.*?)\"|\s*'(.*?)'|([^>\s]+))?)?"
_TAG = "<(\/?)(\w+(?::\w+)?)((?:\s*" + _ATTR + ")+\s*|\s*)(\/?)>?"
_DOCTYPE = r"<!DOCTYPE.*?>"
_SCRIPT = "(<script.*?>)(.*?)(</script.*?>)"
_COMMENT = "(<!--.*?-->)"

11
scrapy/contrib_exp/iterators.py
View File

@ -2,14 +2,19 @@ from scrapy.http import Response
from scrapy.selector import XmlXPathSelector
def xmliter_lxml(obj, nodename):
def xmliter_lxml(obj, nodename, namespace=None):
from lxml import etree
reader = _StreamReader(obj)
iterable = etree.iterparse(reader, tag=nodename, encoding=reader.encoding)
tag = '{%s}%s' % (namespace, nodename) if namespace else nodename
iterable = etree.iterparse(reader, tag=tag, encoding=reader.encoding)
selxpath = '//' + ('x:%s' % nodename if namespace else nodename)
for _, node in iterable:
nodetext = etree.tostring(node)
node.clear()
yield XmlXPathSelector(text=nodetext).select('//' + nodename)[0]
xs = XmlXPathSelector(text=nodetext)
if namespace:
xs.register_namespace('x', namespace)
yield xs.select(selxpath)[0]
class _StreamReader(object):

12
scrapy/link.py
View File

@ -10,15 +10,19 @@ class Link(object):
At the moment, it contains just the url and link text.
"""
__slots__ = ['url', 'text']
__slots__ = ['url', 'text', 'nofollow']
def __init__(self, url, text=''):
def __init__(self, url, text='', nofollow=False):
self.url = url
self.text = text
self.nofollow = nofollow
def __eq__(self, other):
return self.url == other.url and self.text == other.text
return self.url == other.url and self.text == other.text and self.nofollow == other.nofollow
def __hash__(self):
return hash(self.url) ^ hash(self.text) ^ hash(self.nofollow)
def __repr__(self):
return '<Link url=%r text=%r >' % (self.url, self.text)
return 'Link(url=%r, text=%r, nofollow=%r)' % (self.url, self.text, self.nofollow)

1
scrapy/settings/default_settings.py
View File

@ -155,6 +155,7 @@ HTTPCACHE_STORAGE = 'scrapy.contrib.downloadermiddleware.httpcache.FilesystemCac
HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_IGNORE_SCHEMES = ['file']
HTTPCACHE_DBM_MODULE = 'anydbm'
ITEM_PROCESSOR = 'scrapy.contrib.pipeline.ItemPipelineManager'

12
scrapy/tests/test_contrib_exporter.py
View File

@ -127,6 +127,18 @@ class CsvItemExporterTest(BaseItemExporterTest):
ie.finish_exporting()
self.assertEqual(output.getvalue(), '22,John\xc2\xa3\r\n')
def test_join_multivalue(self):
class TestItem2(Item):
name = Field()
friends = Field()
i = TestItem2(name='John', friends=['Mary', 'Paul'])
output = StringIO()
ie = CsvItemExporter(output, include_headers_line=False)
ie.start_exporting()
ie.export_item(i)
ie.finish_exporting()
self.assertEqual(output.getvalue(), '"Mary,Paul",John\r\n')
class XmlItemExporterTest(BaseItemExporterTest):

76
scrapy/tests/test_contrib_ibl/test_extraction.py
View File

@ -9,7 +9,7 @@ from scrapy.contrib.ibl.htmlpage import HtmlPage
from scrapy.contrib.ibl.descriptor import (FieldDescriptor as A,
ItemDescriptor)
from scrapy.contrib.ibl.extractors import (contains_any_numbers,
image_url)
image_url, html, notags)
try:
import numpy
@ -249,12 +249,12 @@ SL342
Nice product for ladies
<br/><ins data-scrapy-annotate="{&quot;variant&quot;: 0, &quot;generated&quot;: true,
&quot;annotations&quot;: {&quot;content&quot;: &quot;price&quot;}}">
&pounds;85.00
&pound;85.00
</ins>
</p>
<ins data-scrapy-annotate="{&quot;variant&quot;: 0, &quot;generated&quot;: true,
&quot;annotations&quot;: {&quot;content&quot;: &quot;price_before_discount&quot;}}">
&pounds;100.00
&pound;100.00
</ins>
</body></html>
"""
@ -266,9 +266,9 @@ SL342
<br/>
Nice product for ladies
<br/>
&pounds;85.00
&pound;85.00
</p>
&pounds;100.00
&pound;100.00
</body></html>
"""
@ -516,7 +516,7 @@ ANNOTATED_PAGE19 = u"""
<div>
<p data-scrapy-annotate="{&quot;variant&quot;: 0, &quot;annotations&quot;: {&quot;content&quot;: &quot;name&quot;}}">Product name</p>
<p data-scrapy-annotate="{&quot;variant&quot;: 0, &quot;annotations&quot;: {&quot;content&quot;: &quot;price&quot;}}">60.00</p>
<img data-scrapy-annotate="{&quot;variant&quot;: 0, &quot;annotations&quot;: {&quot;src&quot;: &quot;image_urls&quot;}}"src="image.jpg" />
<img data-scrapy-annotate="{&quot;variant&quot;: 0, &quot;annotations&quot;: {&quot;src&quot;: &quot;image_urls&quot;}}" src="image.jpg" />
<p data-scrapy-annotate="{&quot;variant&quot;: 0, &quot;required&quot;: [&quot;description&quot;], &quot;annotations&quot;: {&quot;content&quot;: &quot;description&quot;}}">description</p>
</div>
</body></html>
@ -719,19 +719,23 @@ EXTRACT_PAGE23 = u"""
</body></html>
"""
DEFAULT_DESCRIPTOR = ItemDescriptor('test',
'item test, removes tags from description attribute',
[A('description', 'description field without tags', notags)])
SAMPLE_DESCRIPTOR1 = ItemDescriptor('test', 'product test', [
A('name', "Product name", required=True),
A('price', "Product price, including any discounts and tax or vat",
contains_any_numbers, True),
A('image_urls', "URLs for one or more images", image_url, True),
A('description', "The full description of the product", allow_markup=True),
A('description', "The full description of the product", html),
]
)
# A list of (test name, [templates], page, extractors, expected_result)
TEST_DATA = [
# extract from a similar page
('similar page extraction', [ANNOTATED_PAGE1], EXTRACT_PAGE1, None,
('similar page extraction', [ANNOTATED_PAGE1], EXTRACT_PAGE1, DEFAULT_DESCRIPTOR,
{u'title': [u'Nice Product'], u'description': [u'wonderful product'],
u'image_url': [u'nice_product.jpg']}
),
@ -743,20 +747,20 @@ TEST_DATA = [
u'image_url': [u'nice_product.jpg']}
),
# compilicated tag (multiple attributes and annotation)
('multiple attributes and annotation', [ANNOTATED_PAGE2], EXTRACT_PAGE2, None,
('multiple attributes and annotation', [ANNOTATED_PAGE2], EXTRACT_PAGE2, DEFAULT_DESCRIPTOR,
{'name': [u'product 1'], 'image_url': [u'http://example.com/product1.jpg'],
'description': [u'product 1 is great']}
),
# can only work out correct placement by matching the second attribute first
('ambiguous description', [ANNOTATED_PAGE3], EXTRACT_PAGE3, None,
('ambiguous description', [ANNOTATED_PAGE3], EXTRACT_PAGE3, DEFAULT_DESCRIPTOR,
{'description': [u'description'], 'delivery': [u'delivery']}
),
# infer a repeated structure
('repeated elements', [ANNOTATED_PAGE4], EXTRACT_PAGE4, None,
('repeated elements', [ANNOTATED_PAGE4], EXTRACT_PAGE4, DEFAULT_DESCRIPTOR,
{'features': [u'feature1', u'feature2', u'feature3']}
),
# identical variants with a repeated structure
('repeated identical variants', [ANNOTATED_PAGE5], EXTRACT_PAGE5, None,
('repeated identical variants', [ANNOTATED_PAGE5], EXTRACT_PAGE5, DEFAULT_DESCRIPTOR,
{
'description': [u'description'],
'variants': [
@ -767,7 +771,7 @@ TEST_DATA = [
}
),
# variants with an irregular structure
('irregular variants', [ANNOTATED_PAGE6], EXTRACT_PAGE6, None,
('irregular variants', [ANNOTATED_PAGE6], EXTRACT_PAGE6, DEFAULT_DESCRIPTOR,
{
'description': [u'description'],
'variants': [
@ -779,7 +783,7 @@ TEST_DATA = [
),
# discovering repeated variants in table columns
# ('variants in table columns', [ANNOTATED_PAGE7], EXTRACT_PAGE7, None,
# ('variants in table columns', [ANNOTATED_PAGE7], EXTRACT_PAGE7, DEFAULT_DESCRIPTOR,
# {'variants': [
# {u'colour': [u'colour 1'], u'price': [u'price 1']},
# {u'colour': [u'colour 2'], u'price': [u'price 2']},
@ -790,66 +794,66 @@ TEST_DATA = [
# ignored regions
(
'ignored_regions', [ANNOTATED_PAGE8], EXTRACT_PAGE8, None,
'ignored_regions', [ANNOTATED_PAGE8], EXTRACT_PAGE8, DEFAULT_DESCRIPTOR,
{
'description': [u'\n A very nice product for all intelligent people \n\n'],
'description': [u'\n A very nice product for all intelligent people \n \n'],
'price': [u'\n12.00\n(VAT exc.)'],
}
),
# shifted ignored regions (detected by region similarity)
(
'shifted_ignored_regions', [ANNOTATED_PAGE9], EXTRACT_PAGE9, None,
'shifted_ignored_regions', [ANNOTATED_PAGE9], EXTRACT_PAGE9, DEFAULT_DESCRIPTOR,
{
'description': [u'\n A very nice product for all intelligent people \n\n'],
'description': [u'\n A very nice product for all intelligent people \n \n'],
'price': [u'\n12.00\n(VAT exc.)'],
}
),
(# special case with partial annotations
'special_partial_annotation', [ANNOTATED_PAGE11], EXTRACT_PAGE11, None,
'special_partial_annotation', [ANNOTATED_PAGE11], EXTRACT_PAGE11, DEFAULT_DESCRIPTOR,
{
'name': [u'SL342'],
'description': [u'\nSL342\n \nNice product for ladies\n \n&pounds;85.00\n'],
'price': [u'&pounds;85.00'],
'price_before_discount': [u'&pounds;100.00'],
'description': ['\nSL342\n \nNice product for ladies\n \n&pound;85.00\n'],
'price': [u'\xa385.00'],
'price_before_discount': [u'\xa3100.00'],
}
),
(# with ignore-beneath feature
'ignore-beneath', [ANNOTATED_PAGE12], EXTRACT_PAGE12a, None,
'ignore-beneath', [ANNOTATED_PAGE12], EXTRACT_PAGE12a, DEFAULT_DESCRIPTOR,
{
'description': [u'\n A very nice product for all intelligent people \n'],
}
),
(# ignore-beneath with extra tags
'ignore-beneath with extra tags', [ANNOTATED_PAGE12], EXTRACT_PAGE12b, None,
'ignore-beneath with extra tags', [ANNOTATED_PAGE12], EXTRACT_PAGE12b, DEFAULT_DESCRIPTOR,
{
'description': [u'\n A very nice product for all intelligent people \n'],
}
),
('nested annotation with replica outside', [ANNOTATED_PAGE13a], EXTRACT_PAGE13a, None,
('nested annotation with replica outside', [ANNOTATED_PAGE13a], EXTRACT_PAGE13a, DEFAULT_DESCRIPTOR,
{'description': [u'\n A product \n $50.00 \nThis product is excelent. Buy it!\n \n'],
'price': ["$50.00"],
'name': [u'A product']}
),
('outside annotation with nested replica', [ANNOTATED_PAGE13b], EXTRACT_PAGE13b, None,
('outside annotation with nested replica', [ANNOTATED_PAGE13b], EXTRACT_PAGE13b, DEFAULT_DESCRIPTOR,
{'description': [u'\n A product \n $50.00 \nThis product is excelent. Buy it!\n'],
'price': ["$45.00"],
'name': [u'A product']}
),
('consistency check', [ANNOTATED_PAGE14], EXTRACT_PAGE14, None,
('consistency check', [ANNOTATED_PAGE14], EXTRACT_PAGE14, DEFAULT_DESCRIPTOR,
{},
),
('consecutive nesting', [ANNOTATED_PAGE15], EXTRACT_PAGE15, None,
{'description': [u'Description\n\n'],
('consecutive nesting', [ANNOTATED_PAGE15], EXTRACT_PAGE15, DEFAULT_DESCRIPTOR,
{'description': [u'Description\n \n'],
'price': [u'80.00']},
),
('nested inside not found', [ANNOTATED_PAGE16], EXTRACT_PAGE16, None,
('nested inside not found', [ANNOTATED_PAGE16], EXTRACT_PAGE16, DEFAULT_DESCRIPTOR,
{'price': [u'90.00'],
'name': [u'product name']},
),
('ignored region helps to find attributes', [ANNOTATED_PAGE17], EXTRACT_PAGE17, None,
('ignored region helps to find attributes', [ANNOTATED_PAGE17], EXTRACT_PAGE17, DEFAULT_DESCRIPTOR,
{'description': [u'\nThis product is excelent. Buy it!\n']},
),
('ignored region in partial annotation', [ANNOTATED_PAGE18], EXTRACT_PAGE18, None,
('ignored region in partial annotation', [ANNOTATED_PAGE18], EXTRACT_PAGE18, DEFAULT_DESCRIPTOR,
{u'site_id': [u'Item Id'],
u'description': [u'\nDescription\n']},
),
@ -864,13 +868,13 @@ TEST_DATA = [
SAMPLE_DESCRIPTOR1,
None,
),
('repeated partial annotations with variants', [ANNOTATED_PAGE20], EXTRACT_PAGE20, None,
('repeated partial annotations with variants', [ANNOTATED_PAGE20], EXTRACT_PAGE20, DEFAULT_DESCRIPTOR,
{u'variants': [
{'price': ['270'], 'name': ['Twin']},
{'price': ['330'], 'name': ['Queen']},
]},
),
('variants with swatches', [ANNOTATED_PAGE21], EXTRACT_PAGE21, None,
('variants with swatches', [ANNOTATED_PAGE21], EXTRACT_PAGE21, DEFAULT_DESCRIPTOR,
{u'category': [u'chairs'],
u'image_urls': [u'image.jpg'],
u'variants': [
@ -881,7 +885,7 @@ TEST_DATA = [
]
},
),
('variants with swatches complete', [ANNOTATED_PAGE22], EXTRACT_PAGE22, None,
('variants with swatches complete', [ANNOTATED_PAGE22], EXTRACT_PAGE22, DEFAULT_DESCRIPTOR,
{u'category': [u'chairs'],
u'variants': [
{u'swatches': [u'swatch1.jpg'],
@ -899,7 +903,7 @@ TEST_DATA = [
],
u'image_urls': [u'image.jpg']},
),
('repeated (variants) with ignore annotations', [ANNOTATED_PAGE23], EXTRACT_PAGE23, None,
('repeated (variants) with ignore annotations', [ANNOTATED_PAGE23], EXTRACT_PAGE23, DEFAULT_DESCRIPTOR,
{'variants': [
{u'price': [u'300'], u'name': [u'Variant 1']},
{u'price': [u'320'], u'name': [u'Variant 2']},

16
scrapy/tests/test_contrib_ibl/test_htmlpage.py
View File

@ -137,3 +137,19 @@ class TestParseHtml(TestCase):
parsed = list(parse_html("<IMG SRC='http://images.play.com/banners/SAM550a.jpg' align='left' / hspace=5>"))
self.assertEqual(parsed[0].attributes, {'src': 'http://images.play.com/banners/SAM550a.jpg', \
'align': 'left', 'hspace': '5', '/': None})
def test_no_ending_body(self):
"""Test case when no ending body nor html elements are present"""
parsed = [_decode_element(d) for d in PARSED7]
self._test_sample(PAGE7, parsed)
def test_malformed(self):
"""Test parsing of some malformed cases"""
parsed = [_decode_element(d) for d in PARSED8]
self._test_sample(PAGE8, parsed)
def test_malformed2(self):
"""Test case when attributes are not separated by space (still recognizable because of quotes)"""
parsed = [_decode_element(d) for d in PARSED9]
self._test_sample(PAGE9, parsed)

29
scrapy/tests/test_contrib_ibl/test_htmlpage_data.py
View File

@ -246,3 +246,32 @@ PARSED7 = [
{'end': 99, 'start': 85},
]
PAGE8 = u"""<a href="/overview.asp?id=277"><img border="0" src="/img/5200814311.jpg" title=\'Vinyl Cornice\'</a></td><table width=\'5\'>"""
PARSED8 = [
{'attributes' : {u'href' : u"/overview.asp?id=277"}, 'end': 31, 'start': 0, 'tag': u'a', 'tag_type': 1},
{'attributes' : {u'src' : u"/img/5200814311.jpg", u'border' : u"0", u'title': u'Vinyl Cornice'}, 'end': 94, 'start': 31, 'tag': u'img', 'tag_type': 1},
{'attributes' : {}, 'end': 98, 'start': 94, 'tag': u'a', 'tag_type': 2},
{'attributes' : {}, 'end': 103, 'start': 98, 'tag': u'td', 'tag_type': 2},
{'attributes' : {u'width': u'5'}, 'end': 120, 'start': 103, 'tag': u'table', 'tag_type': 1}
]
PAGE9 = u"""\
<html>\
<body>\
<img width='230' height='150'src='/images/9589.jpg' >\
<a href="/product/9589">Click here</a>\
</body>\
</html>\
"""
PARSED9 = [
{'attributes' : {}, 'end': 6, 'start': 0, 'tag': 'html', 'tag_type': 1},
{'attributes' : {}, 'end': 12, 'start': 6, 'tag': 'body', 'tag_type': 1},
{'attributes' : {'width': '230', 'height': '150', 'src': '/images/9589.jpg'}, 'end': 65, 'start': 12, 'tag': 'img', 'tag_type': 1},
{'attributes' : {'href': '/product/9589'}, 'end': 89, 'start': 65, 'tag': 'a', 'tag_type': 1},
{'end': 99, 'start': 89},
{'attributes' : {}, 'end': 103, 'start': 99, 'tag': 'a', 'tag_type': 2},
{'attributes' : {}, 'end': 110, 'start': 103, 'tag': 'body', 'tag_type': 2},
{'attributes' : {}, 'end': 117, 'start': 110, 'tag': 'html', 'tag_type': 2},
]

8
scrapy/tests/test_contrib_ibl/test_pageparsing.py
View File

@ -228,11 +228,11 @@ class TestPageParsing(TestCase):
epp = _parse_page(ExtractionPageParser, SIMPLE_PAGE)
ep = epp.to_extraction_page()
assert len(ep.page_tokens) == 4
assert ep.token_html(0) == '<html>'
assert ep.token_html(1) == '<p some-attr="foo">'
assert ep.htmlpage.fragment_data(ep.htmlpage_tag(0)) == '<html>'
assert ep.htmlpage.fragment_data(ep.htmlpage_tag(1)) == '<p some-attr="foo">'
assert ep.html_between_tokens(1, 2) == 'this is a test'
assert ep.html_between_tokens(1, 3) == 'this is a test</p> '
assert ep.htmlpage_region_inside(1, 2) == 'this is a test'
assert ep.htmlpage_region_inside(1, 3) == 'this is a test</p> '
def test_invalid_html(self):
p = _parse_page(InstanceLearningParser, BROKEN_PAGE)

28
scrapy/tests/test_link.py Normal file
View File

@ -0,0 +1,28 @@
import unittest
from scrapy.link import Link
class LinkTest(unittest.TestCase):
def test_eq_and_hash(self):
l1 = Link("http://www.example.com")
l2 = Link("http://www.example.com/other")
l3 = Link("http://www.example.com")
self.assertEqual(l1, l1)
self.assertEqual(hash(l1), hash(l1))
self.assertNotEqual(l1, l2)
self.assertNotEqual(hash(l1), hash(l2))
self.assertEqual(l1, l3)
self.assertEqual(hash(l1), hash(l3))
l4 = Link("http://www.example.com", text="test")
l5 = Link("http://www.example.com", text="test2")
l6 = Link("http://www.example.com", text="test")
self.assertEqual(l4, l4)
self.assertEqual(hash(l4), hash(l4))
self.assertNotEqual(l4, l5)
self.assertNotEqual(hash(l4), hash(l5))
self.assertEqual(l4, l6)
self.assertEqual(hash(l4), hash(l6))

35
scrapy/tests/test_utils_iterators.py
View File

@ -29,12 +29,12 @@ class XmliterTestCase(unittest.TestCase):
for x in self.xmliter(response, 'product'):
attrs.append((x.select("@id").extract(), x.select("name/text()").extract(), x.select("./type/text()").extract()))
self.assertEqual(attrs,
self.assertEqual(attrs,
[(['001'], ['Name 1'], ['Type 1']), (['002'], ['Name 2'], ['Type 2'])])
def test_xmliter_text(self):
body = u"""<?xml version="1.0" encoding="UTF-8"?><products><product>one</product><product>two</product></products>"""
self.assertEqual([x.select("text()").extract() for x in self.xmliter(body, 'product')],
[[u'one'], [u'two']])
@ -74,7 +74,7 @@ class XmliterTestCase(unittest.TestCase):
def test_xmliter_exception(self):
body = u"""<?xml version="1.0" encoding="UTF-8"?><products><product>one</product><product>two</product></products>"""
iter = self.xmliter(body, 'product')
iter.next()
iter.next()
@ -97,6 +97,35 @@ class LxmlXmliterTestCase(XmliterTestCase):
except ImportError:
skip = "lxml not available"
def test_xmliter_iterate_namespace(self):
body = """\
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns="http://base.google.com/ns/1.0">
<channel>
<title>My Dummy Company</title>
<link>http://www.mydummycompany.com</link>
<description>This is a dummy company. We do nothing.</description>
<item>
<title>Item 1</title>
<description>This is item 1</description>
<link>http://www.mydummycompany.com/items/1</link>
<image_link>http://www.mydummycompany.com/images/item1.jpg</image_link>
<image_link>http://www.mydummycompany.com/images/item2.jpg</image_link>
</item>
</channel>
</rss>
"""
response = XmlResponse(url='http://mydummycompany.com', body=body)
no_namespace_iter = self.xmliter(response, 'image_link')
self.assertEqual(len(list(no_namespace_iter)), 0)
namespace_iter = self.xmliter(response, 'image_link', 'http://base.google.com/ns/1.0')
node = namespace_iter.next()
self.assertEqual(node.select('text()').extract(), ['http://www.mydummycompany.com/images/item1.jpg'])
node = namespace_iter.next()
self.assertEqual(node.select('text()').extract(), ['http://www.mydummycompany.com/images/item2.jpg'])
class UtilsCsvTestCase(unittest.TestCase):
sample_feeds_dir = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'sample_data', 'feeds')

7
scrapy/utils/request.py
View File

@ -64,13 +64,6 @@ def request_authenticate(request, username, password):
"""
request.headers['Authorization'] = basic_auth_header(username, password)
def request_info(request):
"""Return a short string with request info including method, url and
fingeprint. Mainly used for debugging
"""
fp = request_fingerprint(request)
return "<Request: %s %s (%s..)>" % (request.method, request.url, fp[:8])
def request_httprepr(request):
"""Return the raw HTTP representation (as string) of the given request.
This is provided only for reference since it's not the actual stream of

9
scrapyd/webservice.py
View File

@ -87,3 +87,12 @@ class DeleteVersion(DeleteProject):
self._delete_version(project, version)
return {"status": "ok"}
class ListJobs(WsResource):
def render_POST(self, txrequest):
project = txrequest.args['project'][0]
spiders = self.root.launcher.processes.values()
jlist = list()
for s in spiders:
if project == s.project:
jlist.append({"job": {"id":s.job, "spider": s.spider}})
return {"status":"ok", "jobs": jlist}

1
scrapyd/website.py
View File

@ -22,6 +22,7 @@ class Root(resource.Resource):
self.putChild('listspiders.json', webservice.ListSpiders(self))
self.putChild('delproject.json', webservice.DeleteProject(self))
self.putChild('delversion.json', webservice.DeleteVersion(self))
self.putChild('listjobs.json', webservice.ListJobs(self))
self.putChild('logs', static.File(logsdir, 'text/plain'))
self.putChild('procmon', ProcessMonitor(self))
self.update_projects()