diff --git a/AUTHORS b/AUTHORS index 49182f68a..c34f78d0a 100644 --- a/AUTHORS +++ b/AUTHORS @@ -27,3 +27,4 @@ Here is the list of the primary authors & contributors: * Shuaib Khan * Didier Deshommes * Vikas Dhiman + * Jochen Maes diff --git a/README b/README index 694fd7e6f..d79e46bc2 100644 --- a/README +++ b/README @@ -1,4 +1,4 @@ This is Scrapy, an opensource screen scraping framework written in Python. -For more visit the project home page at http://scrapy.org +For more info visit the project home page at http://scrapy.org diff --git a/debian/scrapy.install b/debian/scrapy.install index 8aa6bc81d..2ccb5ce19 100644 --- a/debian/scrapy.install +++ b/debian/scrapy.install @@ -1,3 +1,3 @@ -usr/lib/python*/*-packages/scrapy +usr/lib/python*/*-packages/scrapy* usr/bin extras/scrapy_bash_completion etc/bash_completion.d/ diff --git a/debian/scrapyd.install b/debian/scrapyd.install index d83e70b48..f46d4e3bd 100644 --- a/debian/scrapyd.install +++ b/debian/scrapyd.install @@ -1,3 +1,2 @@ -usr/lib/python*/*-packages/scrapyd debian/scrapyd-files/000-default etc/scrapyd/conf.d extras/scrapyd.tac usr/share/scrapyd diff --git a/docs/faq.rst b/docs/faq.rst index 2444a3c05..65159a6c2 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -3,7 +3,7 @@ Frequently Asked Questions ========================== -How does Scrapy compare to BeautifulSoul or lxml? +How does Scrapy compare to BeautifulSoup or lxml? ------------------------------------------------- `BeautifulSoup`_ and `lxml`_ are libraries for parsing HTML and XML. Scrapy is @@ -29,7 +29,7 @@ comparing `jinja2`_ to `Django`_. What Python versions does Scrapy support? ----------------------------------------- -Scrapy runs in Python 2.5, 2.6 and 2.6. But it's recommended you use Python 2.6 +Scrapy runs in Python 2.5, 2.6 and 2.7. But it's recommended you use Python 2.6 or above, since the Python 2.5 standard library has a few bugs in their URL handling libraries. Some of these Python 2.5 bugs not only affect Scrapy but any user code, such as spiders. You can see a list of `Python 2.5 bugs that @@ -240,3 +240,17 @@ In order to avoid parsing all the entire feed at once in memory, you can use the functions ``xmliter`` and ``csviter`` from ``scrapy.utils.iterators`` module. In fact, this is what the feed spiders (see :ref:`topics-spiders`) use under the cover. + +Does Scrapy manage cookies automatically? +----------------------------------------- + +Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them +back on subsequent requests, like any regular web browser does. + +For more info see :ref:`topics-request-response` and :ref:`cookies-mw`. + +How can I see the cookies being sent and received from Scrapy? +-------------------------------------------------------------- + +Enable the :setting:`COOKIES_DEBUG` setting. + diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 242523829..8693c13ae 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -158,6 +158,8 @@ middleware, see the :ref:`downloader middleware usage guide For a list of the components enabled by default (and their orders) see the :setting:`DOWNLOADER_MIDDLEWARES_BASE` setting. +.. _cookies-mw: + CookiesMiddleware ----------------- @@ -166,7 +168,36 @@ CookiesMiddleware .. class:: CookiesMiddleware - This middleware enables working with sites that need cookies. + This middleware enables working with sites that need cookies. It keeps track + of merging cookies sent by servers, so that they're send in future requests + for that spider, just like a web browser would do. + +The following settings can be used to configure the cookie middleware: + +* :setting:`COOKIES_DEBUG` + +.. setting:: COOKIES_DEBUG + +COOKIES_DEBUG +~~~~~~~~~~~~~ + +Default: ``False`` + +If enabled, Scrapy will log all cookies sent in requests (ie. ``Cookie`` +header) and all cookies received in responses (ie. ``Set-Cookie`` header). + +Here's an example of a log with :setting:`COOKIES_DEBUG` enabled:: + + 2011-04-06 14:35:10-0300 [diningcity] INFO: Spider opened + 2011-04-06 14:35:10-0300 [diningcity] DEBUG: Sending cookies to: + Cookie: clientlanguage_nl=en_EN + 2011-04-06 14:35:14-0300 [diningcity] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html> + Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/ + Set-Cookie: ip_isocode=US + Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/ + 2011-04-06 14:49:50-0300 [diningcity] DEBUG: Crawled (200) (referer: None) + [...] + DefaultHeadersMiddleware ------------------------ @@ -233,8 +264,18 @@ HttpCacheMiddleware downloads every time) and for trying your spider offline, when an Internet connection is not available. -File system storage -~~~~~~~~~~~~~~~~~~~ + Scrapy ships with two storage backends for the HTTP cache middleware: + + * :ref:`httpcache-fs-backend` + * :ref:`httpcache-dbm-backend` + + You can change the storage backend with the :setting:`HTTPCACHE_STORAGE` + setting. Or you can also implement your own backend. + +.. _httpcache-fs-backend: + +File system backend (default) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default, the :class:`HttpCacheMiddleware` uses a file system storage with the following structure: @@ -257,8 +298,19 @@ inefficient in many file systems). An example directory could be:: /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7 -The cache storage backend can be changed with the :setting:`HTTPCACHE_STORAGE` -setting, but no other backend is provided with Scrapy yet. +.. _httpcache-dbm-backend: + +DBM storage backend +~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: 0.13 + +A DBM_ storage backend is also available for the HTTP cache middleware. To use +it (instead of the default filesystem backend) set :setting:`HTTPCACHE_STORAGE` +to ``scrapy.contrib.httpcache.DbmCacheStorage``. + +By default, it uses the anydbm_ module, but you can change it with the +:setting:`HTTPCACHE_DBM_MODULE` setting. Settings ~~~~~~~~ @@ -346,6 +398,18 @@ Default: ``'scrapy.contrib.downloadermiddleware.httpcache.FilesystemCacheStorage The class which implements the cache storage backend. +.. setting:: HTTPCACHE_DBM_MODULE + +HTTPCACHE_DBM_MODULE +^^^^^^^^^^^^^^^^^^^^ + +.. versionadded:: 0.13 + +Default: ``'anydbm'`` + +The database module to use in the :ref:`DBM storage backend +`. This setting is specific to the DBM backend. + HttpCompressionMiddleware ------------------------- @@ -491,3 +555,6 @@ UserAgentMiddleware In order for a spider to override the default user agent, its `user_agent` attribute must be set. + +.. _DBM: http://en.wikipedia.org/wiki/Dbm +.. _anydbm: http://docs.python.org/library/anydbm.html diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index 9c6d292b6..4090b4ed8 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -264,7 +264,7 @@ XmlItemExporter CsvItemExporter --------------- -.. class:: CsvItemExporter(file, include_headers_line=True, \**kwargs) +.. class:: CsvItemExporter(file, include_headers_line=True, join_multivalued=',', \**kwargs) Exports Items in CSV format to the given file-like object. If the :attr:`fields_to_export` attribute is set, it will be used to define the @@ -278,6 +278,10 @@ CsvItemExporter :attr:`BaseItemExporter.fields_to_export` or the first exported item fields. :type include_headers_line: boolean + :param join_multivalued: The char (or chars) that will be used for joining + multi-valued fields, if found. + :type include_headers_line: str + The additional keyword arguments of this constructor are passed to the :class:`BaseItemExporter` constructor, and the leftover arguments to the `csv.writer`_ constructor, so you can use any `csv.writer` constructor diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index a1092b8e6..baeec8701 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -68,6 +68,8 @@ Request objects request_with_cookies = Request(url="http://www.example.com", cookies={currency: 'USD', country: 'UY'}, meta={'dont_merge_cookies': True}) + + For more info see :ref:`cookies-mw`. :type cookies: dict :param encoding: the encoding of this request (defaults to ``'utf-8'``). diff --git a/docs/topics/scrapyd.rst b/docs/topics/scrapyd.rst index b44b0b47b..4e00ea978 100644 --- a/docs/topics/scrapyd.rst +++ b/docs/topics/scrapyd.rst @@ -86,7 +86,7 @@ in your Ubuntu servers. So, if you plan to deploy Scrapyd on a Ubuntu server, just add the Ubuntu repositories as described in :ref:`topics-ubuntu` and then run:: - aptitude install scrapyd-0.12 + aptitude install scrapyd-0.13 This will install Scrapyd in your Ubuntu server creating a ``scrapy`` user which Scrapyd will run as. It will also create some directories and files that diff --git a/docs/topics/ubuntu.rst b/docs/topics/ubuntu.rst index 13bdd4b7b..6cd164f7b 100644 --- a/docs/topics/ubuntu.rst +++ b/docs/topics/ubuntu.rst @@ -13,7 +13,7 @@ latest bug fixes. To use the packages, just add the following line to your ``/etc/apt/sources.list``, and then run ``aptitude update`` and ``aptitude -install scrapy-0.12``:: +install scrapy-0.13``:: deb http://archive.scrapy.org/ubuntu DISTRO main diff --git a/scrapy/__init__.py b/scrapy/__init__.py index 3d8a7a4ae..c73dae498 100644 --- a/scrapy/__init__.py +++ b/scrapy/__init__.py @@ -2,8 +2,8 @@ Scrapy - a screen scraping framework written in Python """ -version_info = (0, 12, 0) -__version__ = "0.12.0" +version_info = (0, 13, 0) +__version__ = "0.13.0" import sys, os, warnings diff --git a/scrapy/commands/deploy.py b/scrapy/commands/deploy.py index 0dcf5037f..c22b9c710 100644 --- a/scrapy/commands/deploy.py +++ b/scrapy/commands/deploy.py @@ -57,6 +57,8 @@ class Command(ScrapyCommand): help="list available projects on TARGET") parser.add_option("--egg", metavar="FILE", help="use the given egg, instead of building it") + parser.add_option("--build-egg", metavar="FILE", + help="only build the egg, don't deploy it") def run(self, args, opts): try: @@ -75,18 +77,26 @@ class Command(ScrapyCommand): projects = json.loads(f.read())['projects'] print os.linesep.join(projects) return - target_name = _get_target_name(args) - target = _get_target(target_name) - project = _get_project(target, opts) - version = _get_version(target, opts) + tmpdir = None - if opts.egg: - _log("Using egg: %s" % opts.egg) - egg = opts.egg - else: - _log("Building egg of %s-%s" % (project, version)) + + if opts.build_egg: # build egg only egg, tmpdir = _build_egg() - _upload_egg(target, egg, project, version) + _log("Writing egg to %s" % opts.build_egg) + shutil.copyfile(egg, opts.build_egg) + else: # buld egg and deploy + target_name = _get_target_name(args) + target = _get_target(target_name) + project = _get_project(target, opts) + version = _get_version(target, opts) + if opts.egg: + _log("Using egg: %s" % opts.egg) + egg = opts.egg + else: + _log("Building egg of %s-%s" % (project, version)) + egg, tmpdir = _build_egg() + _upload_egg(target, egg, project, version) + if tmpdir: shutil.rmtree(tmpdir) diff --git a/scrapy/contrib/downloadermiddleware/cookies.py b/scrapy/contrib/downloadermiddleware/cookies.py index b99c4b338..1d3fbde22 100644 --- a/scrapy/contrib/downloadermiddleware/cookies.py +++ b/scrapy/contrib/downloadermiddleware/cookies.py @@ -1,3 +1,4 @@ +import os from collections import defaultdict from scrapy.xlib.pydispatch import dispatcher @@ -28,7 +29,7 @@ class CookiesMiddleware(object): # set Cookie header request.headers.pop('Cookie', None) jar.add_cookie_header(request) - self._debug_cookie(request) + self._debug_cookie(request, spider) def process_response(self, request, response, spider): if 'dont_merge_cookies' in request.meta: @@ -37,31 +38,28 @@ class CookiesMiddleware(object): # extract cookies from Set-Cookie and drop invalid/expired cookies jar = self.jars[spider] jar.extract_cookies(response, request) - self._debug_set_cookie(response) + self._debug_set_cookie(response, spider) return response def spider_closed(self, spider): self.jars.pop(spider, None) - def _debug_cookie(self, request): - """log Cookie header for request""" + def _debug_cookie(self, request, spider): if self.debug: - c = request.headers.get('Cookie') - c = c and [p.split('=')[0] for p in c.split(';')] - log.msg('Cookie: %s for %s' % (c, request.url), level=log.DEBUG) + cl = request.headers.getlist('Cookie') + if cl: + msg = "Sending cookies to: %s" % request + os.linesep + msg += os.linesep.join("Cookie: %s" % c for c in cl) + log.msg(msg, spider=spider, level=log.DEBUG) - def _debug_set_cookie(self, response): - """log Set-Cookies headers but exclude cookie values""" + def _debug_set_cookie(self, response, spider): if self.debug: cl = response.headers.getlist('Set-Cookie') - res = [] - for c in cl: - kv, tail = c.split(';', 1) - k = kv.split('=', 1)[0] - res.append('%s %s' % (k, tail)) - log.msg('Set-Cookie: %s from %s' % (res, response.url)) - + if cl: + msg = "Received cookies from: %s" % response + os.linesep + msg += os.linesep.join("Set-Cookie: %s" % c for c in cl) + log.msg(msg, spider=spider, level=log.DEBUG) def _get_request_cookies(self, jar, request): headers = {'Set-Cookie': ['%s=%s;' % (k, v) for k, v in request.cookies.iteritems()]} diff --git a/scrapy/contrib/downloadermiddleware/retry.py b/scrapy/contrib/downloadermiddleware/retry.py index c8d44dffe..f1ab5ab10 100644 --- a/scrapy/contrib/downloadermiddleware/retry.py +++ b/scrapy/contrib/downloadermiddleware/retry.py @@ -30,9 +30,11 @@ from scrapy.conf import settings class RetryMiddleware(object): + # IOError is raised by the HttpCompression middleware when trying to + # decompress an empty response EXCEPTIONS_TO_RETRY = (ServerTimeoutError, UserTimeoutError, DNSLookupError, ConnectionRefusedError, ConnectionDone, ConnectError, - ConnectionLost, PartialDownloadError) + ConnectionLost, PartialDownloadError, IOError) def __init__(self): self.max_retry_times = settings.getint('RETRY_TIMES') diff --git a/scrapy/contrib/exporter/__init__.py b/scrapy/contrib/exporter/__init__.py index b77c747cc..1d695eb9d 100644 --- a/scrapy/contrib/exporter/__init__.py +++ b/scrapy/contrib/exporter/__init__.py @@ -143,11 +143,20 @@ class XmlItemExporter(BaseItemExporter): class CsvItemExporter(BaseItemExporter): - def __init__(self, file, include_headers_line=True, **kwargs): + def __init__(self, file, include_headers_line=True, join_multivalued=',', **kwargs): self._configure(kwargs, dont_fail=True) self.include_headers_line = include_headers_line self.csv_writer = csv.writer(file, **kwargs) self._headers_not_written = True + self._join_multivalued = join_multivalued + + def _to_str_if_unicode(self, value): + if isinstance(value, (list, tuple)): + try: + value = self._join_multivalued.join(value) + except TypeError: # list in value may not contain strings + pass + return super(CsvItemExporter, self)._to_str_if_unicode(value) def export_item(self, item): if self._headers_not_written: diff --git a/scrapy/contrib/httpcache.py b/scrapy/contrib/httpcache.py new file mode 100644 index 000000000..dfd43a1db --- /dev/null +++ b/scrapy/contrib/httpcache.py @@ -0,0 +1,64 @@ +from __future__ import with_statement + +import os +from time import time +import cPickle as pickle + +from scrapy.http import Headers +from scrapy.core.downloader.responsetypes import responsetypes +from scrapy.utils.request import request_fingerprint +from scrapy.utils.project import data_path +from scrapy import conf + + +class DbmCacheStorage(object): + + def __init__(self, settings=conf.settings): + self.cachedir = data_path(settings['HTTPCACHE_DIR']) + self.expiration_secs = settings.getint('HTTPCACHE_EXPIRATION_SECS') + self.dbmodule = __import__(settings['HTTPCACHE_DBM_MODULE']) + self.dbs = {} + + def open_spider(self, spider): + dbpath = os.path.join(self.cachedir, '%s.db' % spider.name) + self.dbs[spider] = self.dbmodule.open(dbpath, 'c') + + def close_spider(self, spider): + self.dbs[spider].close() + + def retrieve_response(self, spider, request): + data = self._read_data(spider, request) + if data is None: + return # not cached + url = data['url'] + status = data['status'] + headers = Headers(data['headers']) + body = data['body'] + respcls = responsetypes.from_args(headers=headers, url=url) + response = respcls(url=url, headers=headers, status=status, body=body) + return response + + def store_response(self, spider, request, response): + key = self._request_key(request) + data = { + 'status': response.status, + 'url': response.url, + 'headers': dict(response.headers), + 'body': response.body, + } + self.dbs[spider]['%s_data' % key] = pickle.dumps(data, protocol=2) + self.dbs[spider]['%s_time' % key] = str(time()) + + def _read_data(self, spider, request): + key = self._request_key(request) + db = self.dbs[spider] + tkey = '%s_time' % key + if not db.has_key(tkey): + return # not found + ts = db[tkey] + if 0 < self.expiration_secs < time() - float(ts): + return # expired + return pickle.loads(db['%s_data' % key]) + + def _request_key(self, request): + return request_fingerprint(request) diff --git a/scrapy/contrib/ibl/descriptor.py b/scrapy/contrib/ibl/descriptor.py index 5b4ae99ce..075920d87 100644 --- a/scrapy/contrib/ibl/descriptor.py +++ b/scrapy/contrib/ibl/descriptor.py @@ -3,25 +3,22 @@ Extended types for IBL extraction """ from itertools import chain -from scrapy.contrib.ibl.extractors import text +from scrapy.contrib.ibl.extractors import text, html class FieldDescriptor(object): """description of a scraped attribute""" - __slots__ = ('name', 'description', 'extractor', 'required', 'allow_markup') + __slots__ = ('name', 'description', 'extractor', 'required') - def __init__(self, name, description, extractor=text, required=False, - allow_markup=False): + def __init__(self, name, description, extractor=text, required=False): self.name = name self.description = description self.extractor = extractor self.required = required - self.allow_markup = allow_markup @classmethod def from_field(cls, name, field): return cls(name, field.get('description'), \ - field.get('ibl_extractor', text), field.get('required', False), \ - field.get('allow_markup', False)) + field.get('ibl_extractor', text), field.get('required', False)) def __str__(self): return "FieldDescriptor(%s)" % self.name diff --git a/scrapy/contrib/ibl/extraction/pageobjects.py b/scrapy/contrib/ibl/extraction/pageobjects.py index fa290d3c7..f7a42417f 100644 --- a/scrapy/contrib/ibl/extraction/pageobjects.py +++ b/scrapy/contrib/ibl/extraction/pageobjects.py @@ -4,16 +4,14 @@ Page objects This module contains objects representing pages and parts of pages (e.g. tokens and annotations) used in the instance based learning algorithm. """ +from itertools import chain from numpy import array, ndarray -from scrapy.contrib.ibl.htmlpage import HtmlTagType +from scrapy.contrib.ibl.htmlpage import HtmlTagType, HtmlPageRegion -class TokenType(object): +class TokenType(HtmlTagType): """constants for token types""" WORD = 0 - OPEN_TAG = HtmlTagType.OPEN_TAG - CLOSE_TAG = HtmlTagType.CLOSE_TAG - NON_PAIRED_TAG = HtmlTagType.UNPAIRED_TAG class TokenDict(object): """Mapping from parse tokens to integers @@ -68,6 +66,36 @@ class TokenDict(object): templates = ["%s", "<%s>", "", "<%s/>"] return templates[tid >> 24] % self.find_token(tid) +class PageRegion(object): + """A region in a page, defined by a start and end index""" + + __slots__ = ('start_index', 'end_index') + + def __init__(self, start, end): + self.start_index = start + self.end_index = end + + def __str__(self): + return "%s(%s, %s)" % (self.__class__.__name__, self.start_index, + self.end_index) + + def __repr__(self): + return str(self) + +class FragmentedHtmlPageRegion(HtmlPageRegion): + """An HtmlPageRegion consisting of possibly non-contiguous sub-regions""" + def __new__(cls, htmlpage, regions): + text = u''.join(regions) + return HtmlPageRegion.__new__(cls, htmlpage, text) + + def __init__(self, htmlpage, regions): + self.htmlpage = htmlpage + self.regions = regions + + @property + def parsed_fragments(self): + return chain(*(r.parsed_fragments for r in self.regions)) + class Page(object): """Basic representation of a page. This consists of a reference to a dictionary of tokens and an array of raw token ids @@ -92,7 +120,8 @@ class TemplatePage(Page): annotations = sorted(annotations, key=lambda x: x.end_index, reverse=True) self.annotations = sorted(annotations, key=lambda x: x.start_index) self.id = template_id - self.ignored_regions = ignored_regions or [] + self.ignored_regions = [i if isinstance(i, PageRegion) else PageRegion(*i) \ + for i in (ignored_regions or [])] self.extra_required_attrs = set(extra_required or []) def __str__(self): @@ -107,66 +136,53 @@ class ExtractionPage(Page): """Parsed data belonging to a web page upon which we wish to perform extraction. """ - __slots__ = ('text', - 'token_start_indexes', # index in text of the start of a token - 'token_follow_indexes', # index in text of data following token - 'tag_attributes' # a map from token index to tag attributes - ) + __slots__ = ('htmlpage', 'token_page_indexes') - def __init__(self, text, token_dict, page_tokens, token_start_indexes, - token_follow_indexes, tag_attributes): + def __init__(self, htmlpage, token_dict, page_tokens, token_page_indexes): + """Construct a new ExtractionPage + + Arguments: + `htmlpage`: The source HtmlPage + `token_dict`: Token Dictionary used for tokenization + `page_tokens': array of page tokens for matching + `token_page_indexes`: indexes of each token in the parsed htmlpage + """ Page.__init__(self, token_dict, page_tokens) - self.text = text - self.token_start_indexes = token_start_indexes - self.token_follow_indexes = token_follow_indexes - self.tag_attributes = tag_attributes - - def token_html(self, token_index): - """The raw html for a page token at the given index in the page_tokens - list - """ - text_start = self.token_start_indexes[token_index] - text_end = self.token_follow_indexes[token_index] - return self.text[text_start:text_end] - - def html_between_tokens(self, start_token_index, end_token_index): - """The raw html between the tokens at the specified indexes in the - page_tokens list - - This assumes start_token_index <= end_token_index - """ - text_start = self.token_follow_indexes[start_token_index] - text_end = self.token_start_indexes[end_token_index] - return self.text[text_start:text_end] + self.htmlpage = htmlpage + self.token_page_indexes = token_page_indexes - def text_between_tokens(self, start_token_index, end_token_index, - tag_replacement=u' '): - """The text between the the tokens at the specified indexes in the - page_tokens list. Tags are replaced by tag_replacement (default one space - character) - """ - return tag_replacement.join([self.text[ - self.token_follow_indexes[i]:self.token_start_indexes[i+1]] \ - for i in xrange(start_token_index, end_token_index)]) + def htmlpage_region(self, start_token_index, end_token_index): + """The region in the HtmlPage corresonding to the area defined by + the start_token_index and the end_token_index - def tag_attribute(self, token_index, attribute): - """The value of a tag attribute. The tag is identified by its - corresponding token index. - - If the tag or attribute is not present, None is returned + This includes the tokens at the specified indexes """ - return self.tag_attributes.get(token_index, {}).get(attribute) + start = self.token_page_indexes[start_token_index] + end = self.token_page_indexes[end_token_index] + return self.htmlpage.subregion(start, end) + + def htmlpage_region_inside(self, start_token_index, end_token_index): + """The region in the HtmlPage corresonding to the area between + the start_token_index and the end_token_index. + + This excludes the tokens at the specified indexes + """ + start = self.token_page_indexes[start_token_index] + 1 + end = self.token_page_indexes[end_token_index] - 1 + return self.htmlpage.subregion(start, end) + def htmlpage_tag(self, token_index): + """The HtmlPage tag at corresponding to the token at token_index""" + return self.htmlpage.parsed_body[self.token_page_indexes[token_index]] + def __str__(self): summary = [] - for (token, start, follow) in zip(self.page_tokens, self.token_start_indexes, - self.token_follow_indexes): - text = "%s %s-%s (%s)" % (self.token_dict.find_token(token), start, follow, - self.text[start:follow]) + for token, tindex in zip(self.page_tokens, self.token_page_indexes): + text = "%s page[%s]: %s" % (self.token_dict.find_token(token), + tindex, self.htmlpage.parsed_body[tindex]) summary.append(text) return "ExtractionPage\n==============\nTokens: %s\n\nRaw text: %s\n\n" \ - "Tag attributes: %s\n" % ('\n'.join(summary), self.text, - self.tag_attributes) + % ('\n'.join(summary), self.htmlpage.body) class AnnotationText(object): __slots__ = ('start_text', 'follow_text') @@ -179,7 +195,8 @@ class AnnotationText(object): return "AnnotationText(%s..%s)" % \ (repr(self.start_text), repr(self.follow_text)) -class AnnotationTag(object): + +class AnnotationTag(PageRegion): """A tag that annotates part of the document It has the following properties: @@ -197,8 +214,7 @@ class AnnotationTag(object): def __init__(self, start_index, end_index, surrounds_attribute=None, annotation_text=None, tag_attributes=None, variant_id=None): - self.start_index = start_index - self.end_index = end_index + PageRegion.__init__(self, start_index, end_index) self.surrounds_attribute = surrounds_attribute self.annotation_text = annotation_text self.tag_attributes = tag_attributes or [] @@ -213,16 +229,3 @@ class AnnotationTag(object): def __repr__(self): return str(self) -class LabelledRegion(object): - __slots__ = ('start_index', 'end_index') - - def __init__(self, start, end): - self.start_index = start - self.end_index = end - - def __str__(self): - return "LabelledRegion (%s, %s)" % (self.start_index, self.end_index) - - def __repr__(self): - return str(self) - diff --git a/scrapy/contrib/ibl/extraction/pageparsing.py b/scrapy/contrib/ibl/extraction/pageparsing.py index 2df63267f..1ce6633db 100644 --- a/scrapy/contrib/ibl/extraction/pageparsing.py +++ b/scrapy/contrib/ibl/extraction/pageparsing.py @@ -9,8 +9,8 @@ from numpy import array from scrapy.utils.py26 import json from scrapy.contrib.ibl.htmlpage import HtmlTagType, HtmlTag, HtmlPage -from scrapy.contrib.ibl.extraction.pageobjects import (AnnotationTag, - TemplatePage, ExtractionPage, AnnotationText, TokenDict) +from scrapy.contrib.ibl.extraction.pageobjects import (AnnotationTag, + TemplatePage, ExtractionPage, AnnotationText, TokenDict, FragmentedHtmlPageRegion) def parse_strings(template_html, extraction_html): """Create a template and extraction page from raw strings @@ -52,18 +52,18 @@ class InstanceLearningParser(object): def feed(self, html_page): self.html_page = html_page self.previous_element_class = None - for data in html_page.parsed_body: + for index, data in enumerate(html_page.parsed_body): if isinstance(data, HtmlTag): self._add_token(data.tag, data.tag_type, data.start, data.end) - self.handle_tag(data) + self.handle_tag(data, index) else: - self.handle_data(data) + self.handle_data(data, index) self.previous_element_class = data.__class__ - def handle_data(self, html_data_fragment): + def handle_data(self, html_data_fragment, index): pass - def handle_tag(self, html_tag): + def handle_tag(self, html_tag, index): pass _END_UNPAIREDTAG_TAGS = ["form", "div", "p", "table", "tr", "td"] @@ -86,7 +86,7 @@ class TemplatePageParser(InstanceLearningParser): self.last_text_region = None self.next_tag_index = 0 - def handle_tag(self, html_tag): + def handle_tag(self, html_tag, index): if self.last_text_region: self._process_text('') @@ -275,7 +275,7 @@ class TemplatePageParser(InstanceLearningParser): if prev != annotation.variant_id: raise ValueError("unbalanced variant annotation tags") - def handle_data(self, html_data_fragment): + def handle_data(self, html_data_fragment, index): fragment_text = self.html_page.fragment_data(html_data_fragment) self._process_text(fragment_text) @@ -300,20 +300,11 @@ class ExtractionPageParser(InstanceLearningParser): """ def __init__(self, token_dict): InstanceLearningParser.__init__(self, token_dict) - self.page_data = [] - self.token_start_index = [] - self.token_follow_index = [] - self.tag_attrs = {} + self._page_token_indexes = [] - def _add_token(self, token, token_type, start, end): - InstanceLearningParser._add_token(self, token, token_type, start, end) - self.token_start_index.append(start) - self.token_follow_index.append(end) - - def handle_tag(self, html_tag): - if html_tag.attributes: - self.tag_attrs[len(self.token_list) - 1] = html_tag.attributes + def handle_tag(self, html_tag, index): + self._page_token_indexes.append(index) def to_extraction_page(self): - return ExtractionPage(self.html_page.body, self.token_dict, array(self.token_list), - self.token_start_index, self.token_follow_index, self.tag_attrs) + return ExtractionPage(self.html_page, self.token_dict, array(self.token_list), + self._page_token_indexes) diff --git a/scrapy/contrib/ibl/extraction/regionextract.py b/scrapy/contrib/ibl/extraction/regionextract.py index db5eb2a38..9c1a72674 100644 --- a/scrapy/contrib/ibl/extraction/regionextract.py +++ b/scrapy/contrib/ibl/extraction/regionextract.py @@ -8,14 +8,16 @@ import operator import copy import pprint import cStringIO -from itertools import groupby +from itertools import groupby, izip, starmap from numpy import array from scrapy.contrib.ibl.descriptor import FieldDescriptor +from scrapy.contrib.ibl.htmlpage import HtmlPageRegion from scrapy.contrib.ibl.extraction.similarity import (similar_region, longest_unique_subsequence, common_prefix) -from scrapy.contrib.ibl.extraction.pageobjects import AnnotationTag, LabelledRegion +from scrapy.contrib.ibl.extraction.pageobjects import (AnnotationTag, + PageRegion, FragmentedHtmlPageRegion) def build_extraction_tree(template, type_descriptor, trace=True): """Build a tree of region extractors corresponding to the @@ -33,16 +35,14 @@ def build_extraction_tree(template, type_descriptor, trace=True): return TemplatePageExtractor(template, extractors) -_ID = lambda x: x +_EXTRACT_HTML = lambda x: x _DEFAULT_DESCRIPTOR = FieldDescriptor('none', None) def _labelled(obj): """ Returns labelled element of the object (extractor or labelled region) """ - if hasattr(obj, "annotation"): - return obj.annotation - return obj + return getattr(obj, 'annotation', obj) def _compose(f, g): """given unary functions f and g, return a function that computes f(g(x)) @@ -75,7 +75,7 @@ class BasicTypeExtractor(object): u'
x xx
',\ u'
a name id-9
') >>> ex = BasicTypeExtractor(template.annotations[0]) - >>> ex.extract(page, 0, 3, [LabelledRegion(*(1,2))]) + >>> ex.extract(page, 0, 3, [PageRegion(1, 2)]) [(u'name', u'a name')] """ @@ -88,17 +88,15 @@ class BasicTypeExtractor(object): descriptor = attribute_descriptors.get(annotation.surrounds_attribute) if descriptor: self.content_validate = descriptor.extractor - self.allow_markup = descriptor.allow_markup else: - self.content_validate = _ID - self.allow_markup = False + self.content_validate = _EXTRACT_HTML self.extract = self._extract_content if annotation.tag_attributes: self.tag_data = [] for (tag_attr, extraction_attr) in annotation.tag_attributes: descriptor = attribute_descriptors.get(extraction_attr) - extractf = descriptor.extractor if descriptor else _ID + extractf = descriptor.extractor if descriptor else _EXTRACT_HTML self.tag_data.append((extractf, tag_attr, extraction_attr)) self.extract = self._extract_both if \ @@ -109,33 +107,32 @@ class BasicTypeExtractor(object): self._extract_attribute(page, start_index, end_index, ignored_regions) def _extract_content(self, extraction_page, start_index, end_index, ignored_regions=None): - # we might want to add opening/closing ul/ol/table if we have the - # middle of a region. This would require support in the scrapy - # cleansing. - complete_data = "" - start = start_index - end = ignored_regions[0].start_index if ignored_regions else end_index - while start is not None: - if self.allow_markup: - data = extraction_page.html_between_tokens(start, end) - else: - data = extraction_page.text_between_tokens(start, end) - complete_data += data - if ignored_regions: - start = ignored_regions[0].end_index - ignored_regions.pop(0) - end = ignored_regions[0].start_index if ignored_regions else end_index - else: - start = None - complete_data = self.content_validate(complete_data) - return [(self.annotation.surrounds_attribute, complete_data)] if complete_data else [] + # extract content between annotation indexes + if not ignored_regions: + region = extraction_page.htmlpage_region_inside(start_index, end_index) + else: + # assumes ignored_regions are completely contained within start and end index + assert (start_index <= ignored_regions[0].start_index and + end_index >= ignored_regions[-1].end_index) + starts = [start_index] + [i.end_index for i in ignored_regions] + ends = [i.start_index for i in ignored_regions] + if starts[-1] is not None: + ends.append(end_index) + included_regions = izip(starts, ends) + if ends[0] is None: + included_regions.next() + regions = starmap(extraction_page.htmlpage_region_inside, included_regions) + region = FragmentedHtmlPageRegion(extraction_page.htmlpage, list(regions)) + validated = self.content_validate(region) + return [(self.annotation.surrounds_attribute, validated)] if validated else [] def _extract_attribute(self, extraction_page, start_index, end_index, ignored_regions=None): data = [] for (f, ta, ea) in self.tag_data: - tag_value = extraction_page.tag_attribute(start_index, ta) + tag_value = extraction_page.htmlpage_tag(start_index).attributes.get(ta) if tag_value: - extracted = f(tag_value) + region = HtmlPageRegion(extraction_page.htmlpage, tag_value) + extracted = f(region) if extracted is not None: data.append((ea, extracted)) return data @@ -178,10 +175,8 @@ class BasicTypeExtractor(object): def __str__(self): messages = ['BasicTypeExtractor('] if self.annotation.surrounds_attribute: - messages += [self.annotation.surrounds_attribute, ': ', - 'html content' if self.allow_markup else 'text content', - ] - if self.content_validate != _ID: + messages.append(self.annotation.surrounds_attribute) + if self.content_validate != _EXTRACT_HTML: messages += [', extracted with \'', self.content_validate.__name__, '\''] @@ -190,7 +185,7 @@ class BasicTypeExtractor(object): messages.append(';') for (f, ta, ea) in self.tag_data: messages += [ea, ': tag attribute "', ta, '"'] - if f != _ID: + if f != _EXTRACT_HTML: messages += [', validated by ', str(f)] messages.append(", template[%s:%s])" % \ (self.annotation.start_index, self.annotation.end_index)) @@ -340,7 +335,8 @@ class RecordExtractor(object): The region in the page to be extracted from may be specified using start_index and end_index """ - ignored_regions = [i if isinstance(i, LabelledRegion) else LabelledRegion(*i) for i in (ignored_regions or [])] + if ignored_regions is None: + ignored_regions = [] region_elements = sorted(self.extractors + ignored_regions, key=lambda x: _labelled(x).start_index) _, _, attributes = self._doextract(page, region_elements, start_index, end_index) @@ -395,7 +391,7 @@ class RecordExtractor(object): s, p, e = similar_region(page.page_tokens, self.template_tokens, \ i, start, sindex) if s > 0: - similar_ignored_regions.append(LabelledRegion(*(p, e))) + similar_ignored_regions.append(PageRegion(p, e)) start = e or start extracted_data = first_region.extract(page, pindex, sindex, similar_ignored_regions) if extracted_data: @@ -493,12 +489,12 @@ class TraceExtractor(object): for t in template.page_tokens[tend:tend+5]]) def summarize_trace(self, page, start, end, ret): - text_start = page.token_follow_indexes[start] - text_end = page.token_start_indexes[end or -1] + text_start = page.htmlpage.parsed_body[page.token_page_indexes[start]].start + text_end = page.htmlpage.parsed_body[page.token_page_indexes[end or -1]].end page_snippet = "(...%s)%s(%s...)" % ( - page.text[text_start-50:text_start].replace('\n', ' '), - page.text[text_start:text_end], - page.text[text_end:text_end+50].replace('\n', ' ')) + page.htmlpage.body[text_start-50:text_start].replace('\n', ' '), + page.htmlpage.body[text_start:text_end], + page.htmlpage.body[text_end:text_end+50].replace('\n', ' ')) pre_summary = "\nstart %s page[%s:%s]\n" % (self.traced.__class__.__name__, start, end) post_summary = """ %s page[%s:%s] @@ -572,25 +568,26 @@ class TemplatePageExtractor(object): _tokenize = re.compile(r'\w+|[^\w\s]+', re.UNICODE | re.MULTILINE | re.DOTALL).findall class TextRegionDataExtractor(object): - """Data Extractor for extracting text fragments from within a larger - body of text. It extracts based on the longest unique prefix and suffix. + """Data Extractor for extracting text fragments from an annotation page + fragment or string. It extracts based on the longest unique prefix and + suffix. for example: >>> extractor = TextRegionDataExtractor('designed by ', '.') - >>> extractor.extract("by Marc Newson.") + >>> extractor.extract_text("by Marc Newson.") 'Marc Newson' Both prefix and suffix are optional: >>> extractor = TextRegionDataExtractor('designed by ') - >>> extractor.extract("by Marc Newson.") + >>> extractor.extract_text("by Marc Newson.") 'Marc Newson.' >>> extractor = TextRegionDataExtractor(suffix='.') - >>> extractor.extract("by Marc Newson.") + >>> extractor.extract_text("by Marc Newson.") 'by Marc Newson' It requires a minimum match of at least one word or punctuation character: >>> extractor = TextRegionDataExtractor('designed by') - >>> extractor.extract("y Marc Newson.") is None + >>> extractor.extract_text("y Marc Newson.") is None True """ def __init__(self, prefix=None, suffix=None): @@ -609,8 +606,13 @@ class TextRegionDataExtractor(object): tokens = _tokenize(matchstring or '') return len(tokens[0]) if tokens else 0 - def extract(self, text): - """attempt to extract a substring from the text""" + def extract(self, region): + """Extract a region from the region passed""" + text = self.extract_text(region) + return HtmlPageRegion(region.htmlpage, text) if text else None + + def extract_text(self, text): + """Extract a substring from the text""" pref_index = 0 if self.minprefix > 0: rev_idx, plen = longest_unique_subsequence(text[::-1], self.prefix) @@ -623,5 +625,3 @@ class TextRegionDataExtractor(object): if slen < self.minsuffix: return None return text[pref_index:pref_index + sidx] - - diff --git a/scrapy/contrib/ibl/extractors.py b/scrapy/contrib/ibl/extractors.py index 9c460df10..6e2f2c266 100644 --- a/scrapy/contrib/ibl/extractors.py +++ b/scrapy/contrib/ibl/extractors.py @@ -4,8 +4,9 @@ Extractors for attributes import re import urlparse -from scrapy.utils.markup import remove_entities +from scrapy.utils.markup import remove_entities, remove_comments from scrapy.utils.url import safe_url_string +from scrapy.contrib.ibl.htmlpage import HtmlTag, HtmlTagType #FIXME: the use of "." needs to be localized _NUMERIC_ENTITIES = re.compile("&#([0-9]+)(?:;|\s)", re.U) @@ -22,11 +23,175 @@ _CSS_IMAGERE = re.compile("background(?:-image)?\s*:\s*url\((.*?)\)", re.I) _BASE_PATH_RE = "/?(?:[^/]+/)*(?:.+%s)" _IMAGE_PATH_RE = re.compile(_BASE_PATH_RE % '\.(?:%s)' % _IMAGES_TYPES, re.I) _GENERIC_PATH_RE = re.compile(_BASE_PATH_RE % '', re.I) +_WS = re.compile("\s+", re.U) -def text(txt): - stripped = txt.strip() if txt else None - if stripped: - return stripped +# tags to keep (only for attributes with markup) +_TAGS_TO_KEEP = frozenset(['br', 'p', 'big', 'em', 'small', 'strong', 'sub', + 'sup', 'ins', 'del', 'code', 'kbd', 'samp', 'tt', 'var', 'pre', 'listing', + 'plaintext', 'abbr', 'acronym', 'address', 'bdo', 'blockquote', 'q', + 'cite', 'dfn', 'table', 'tr', 'th', 'td', 'tbody', 'ul', 'ol', 'li', 'dl', + 'dd', 'dt']) + +# tag names to be replaced by other tag names (overrides tags_to_keep) +_TAGS_TO_REPLACE = { + 'h1': 'strong', + 'h2': 'strong', + 'h3': 'strong', + 'h4': 'strong', + 'h5': 'strong', + 'h6': 'strong', + 'b' : 'strong', + 'i' : 'em', +} + +# tags whoose content will be completely removed (recursively) +# (overrides tags_to_keep and tags_to_replace) +_TAGS_TO_PURGE = ('script', 'img', 'input') + +def htmlregion(text): + """convenience function to make an html region from text. + This is useful for testing + """ + from scrapy.contrib.ibl.htmlpage import HtmlPage + return HtmlPage(body=text).subregion() + +def notags(region, tag_replace=u' '): + """Removes all html tags""" + fragments = getattr(region, 'parsed_fragments', None) + if fragments is None: + return region + page = region.htmlpage + data = [page.fragment_data(f) for f in fragments if not isinstance(f, HtmlTag)] + return tag_replace.join(data) + +def text(region): + """Converts HTML to text. There is no attempt at formatting other than + removing excessive whitespace, + + For example: + >>> t = lambda s: text(htmlregion(s)) + >>> t(u'

test

') + u'test' + + Leading and trailing whitespace are removed + >>> t(u'

test

') + u'test' + + Comments are removed + >>> t(u'test me') + u'test me' + + Text between script tags is ignored + >>> t(u"scripts are ignored") + u'scripts are ignored' + + HTML entities are converted to text + >>> t(u"only £42") + u'only \\xa342' + """ + chunks = _process_markup(region, + lambda text: remove_entities(text, encoding=region.htmlpage.encoding), + lambda tag: u' ' + ) + text = u''.join(chunks) + return _WS.sub(u' ', text).strip() + +def safehtml(region, allowed_tags=_TAGS_TO_KEEP, replace_tags=_TAGS_TO_REPLACE): + """Creates an HTML subset, using a whitelist of HTML tags. + + The HTML generated is safe for display on a website,without escaping and + should not cause formatting problems. + + Allowed_tags is a set of tags that are allowed and replace_tags is a mapping of + tags to alternative tags to substitute. + + For example: + >>> t = lambda s: safehtml(htmlregion(s)) + >>> t(u'test test') + u'test test' + + Some tags, like script, are completely removed + >>> t(u'test') + u'test' + + replace_tags define tags that are converted. By default all headers, bold and indenting + are converted to strong and em. + >>> t(u'

header

test bold indent') + u'header test bold indent' + + Comments are stripped, but entities are not converted + >>> t(u' only £42') + u'only £42' + + Paired tags are closed + >>> t(u'

test') + u'

test

' + + >>> t(u'

test
test

') + u'

test
test

' + + """ + tagstack = [] + def _process_tag(tag): + tagstr = replace_tags.get(tag.tag, tag.tag) + if tagstr not in allowed_tags: + return + if tag.tag_type == HtmlTagType.OPEN_TAG: + tagstack.append(tagstr) + return u"<%s>" % tagstr + elif tag.tag_type == HtmlTagType.CLOSE_TAG: + try: + last = tagstack.pop() + # common case of matching tag + if last == tagstr: + return u"" % last + # output all preceeding tags (if present) + revtags = tagstack[::-1] + tindex = revtags.index(tagstr) + del tagstack[-tindex-1:] + return u"" % (last, u">" % tag.tag + chunks = list(_process_markup(region, lambda text: text, _process_tag)) + \ + ["" % t for t in reversed(tagstack)] + return u''.join(chunks).strip() + +def _process_markup(region, textf, tagf): + fragments = getattr(region, 'parsed_fragments', None) + if fragments is None: + yield textf(region) + return + fiter = iter(fragments) + for fragment in fiter: + if isinstance(fragment, HtmlTag): + # skip forward to closing script tags + tag = fragment.tag + if tag in _TAGS_TO_PURGE: + # if opening, keep going until closed + if fragment.tag_type == HtmlTagType.OPEN_TAG: + for probe in fiter: + if isinstance(probe, HtmlTag) and \ + probe.tag == tag and \ + probe.tag_type == HtmlTagType.CLOSE_TAG: + break + else: + output = tagf(fragment) + if output: + yield output + else: + text = region.htmlpage.fragment_data(fragment) + text = remove_comments(text) + text = textf(text) + if text: + yield text + +def html(pageregion): + """A page region is already html, so this is the identity function""" + return pageregion def contains_any_numbers(txt): """text that must contain at least one number @@ -132,6 +297,10 @@ def image_url(txt): ['http://s7d5.scene7.com/is/image/wasserstrom/165133?wid=227&hei=227&defaultImage=noimage_wasserstrom'] """ + imgurl = extract_image_url(txt) + return [safe_url_string(remove_entities(url(imgurl)))] if imgurl else None + +def extract_image_url(txt): txt = url(txt) imgurl = None if txt: @@ -153,4 +322,4 @@ def image_url(txt): imgurl = urlparse.urlunparse(parsed) if not imgurl: imgurl = txt - return [safe_url_string(remove_entities(url(imgurl)))] if imgurl else None + return imgurl diff --git a/scrapy/contrib/ibl/htmlpage.py b/scrapy/contrib/ibl/htmlpage.py index 023030bdc..e16ac2c7a 100644 --- a/scrapy/contrib/ibl/htmlpage.py +++ b/scrapy/contrib/ibl/htmlpage.py @@ -1,8 +1,9 @@ """ htmlpage -Container object for representing html pages in the IBL system. This -encapsulates page related information and prevents parsing multiple times. +Container objects for representing html pages and their parts in the IBL +system. This encapsulates page related information and prevents parsing +multiple times. """ import re import hashlib @@ -26,26 +27,82 @@ def create_page_from_jsonpage(jsonpage, body_key): return HtmlPage(url, headers, body, page_id) class HtmlPage(object): - def __init__(self, url=None, headers=None, body=None, page_id=None): + """HtmlPage + + This is a parsed HTML page. It contains the page headers, url, raw body and parsed + body. + + The parsed body is a list of HtmlDataFragment objects. + """ + def __init__(self, url=None, headers=None, body=None, page_id=None, encoding='utf-8'): assert isinstance(body, unicode), "unicode expected, got: %s" % type(body).__name__ self.headers = headers or {} self.body = body self.url = url or u'' + self.encoding = encoding if page_id is None and url: self.page_id = hashlib.sha1(url).hexdigest() else: self.page_id = page_id + @classmethod + def from_response(cls, response, page_id=None): + """Create an HtmlPage from a scrapy response""" + return HtmlPage(response.url, response.headers, + response.body_as_unicode(), page_id, response.encoding) def _set_body(self, body): self._body = body self.parsed_body = list(parse_html(body)) - body = property(lambda x: x._body, _set_body) + body = property(lambda x: x._body, _set_body, doc="raw html for the page") + def subregion(self, start=0, end=None): + """HtmlPageRegion constructed from the start and end index (inclusive) + into the parsed page + """ + return HtmlPageParsedRegion(self, start, end) + def fragment_data(self, data_fragment): + """portion of the body corresponding to the HtmlDataFragment""" return self.body[data_fragment.start:data_fragment.end] + +class HtmlPageRegion(unicode): + """A Region of an HtmlPage that has been extracted + """ + def __new__(cls, htmlpage, data): + return unicode.__new__(cls, data) + + def __init__(self, htmlpage, data): + """Construct a new HtmlPageRegion object. + + htmlpage is the original page and data is the raw html + """ + self.htmlpage = htmlpage +class HtmlPageParsedRegion(HtmlPageRegion): + """A region of an HtmlPage that has been extracted + + This has a parsed_fragments property that contains the parsed html + fragments contained within this region + """ + def __new__(cls, htmlpage, start_index, end_index): + text_start = htmlpage.parsed_body[start_index].start + text_end = htmlpage.parsed_body[end_index or -1].end + text = htmlpage.body[text_start:text_end] + return HtmlPageRegion.__new__(cls, htmlpage, text) + + def __init__(self, htmlpage, start_index, end_index): + self.htmlpage = htmlpage + self.start_index = start_index + self.end_index = end_index + + @property + def parsed_fragments(self): + """HtmlDataFragment or HtmlTag objects for this parsed region""" + end = self.end_index + 1 if self.end_index is not None else None + return self.htmlpage.parsed_body[self.start_index:end] + class HtmlTagType(object): OPEN_TAG = 1 CLOSE_TAG = 2 @@ -80,8 +137,8 @@ class HtmlTag(HtmlDataFragment): def __repr__(self): return str(self) -_ATTR = "((?:[^=/>\s]|/(?!>))+)(?:\s*=(?:\s*\"(.*?)\"|\s*'(.*?)'|([^>\s]+))?)?" -_TAG = "<(\/?)(\w+(?::\w+)?)((?:\s+" + _ATTR + ")+\s*|\s*)(\/?)>" +_ATTR = "((?:[^=/<>\s]|/(?!>))+)(?:\s*=(?:\s*\"(.*?)\"|\s*'(.*?)'|([^>\s]+))?)?" +_TAG = "<(\/?)(\w+(?::\w+)?)((?:\s*" + _ATTR + ")+\s*|\s*)(\/?)>?" _DOCTYPE = r"" _SCRIPT = "()(.*?)()" _COMMENT = "()" diff --git a/scrapy/contrib_exp/iterators.py b/scrapy/contrib_exp/iterators.py index 0fc73e194..0f3a8c694 100644 --- a/scrapy/contrib_exp/iterators.py +++ b/scrapy/contrib_exp/iterators.py @@ -2,14 +2,19 @@ from scrapy.http import Response from scrapy.selector import XmlXPathSelector -def xmliter_lxml(obj, nodename): +def xmliter_lxml(obj, nodename, namespace=None): from lxml import etree reader = _StreamReader(obj) - iterable = etree.iterparse(reader, tag=nodename, encoding=reader.encoding) + tag = '{%s}%s' % (namespace, nodename) if namespace else nodename + iterable = etree.iterparse(reader, tag=tag, encoding=reader.encoding) + selxpath = '//' + ('x:%s' % nodename if namespace else nodename) for _, node in iterable: nodetext = etree.tostring(node) node.clear() - yield XmlXPathSelector(text=nodetext).select('//' + nodename)[0] + xs = XmlXPathSelector(text=nodetext) + if namespace: + xs.register_namespace('x', namespace) + yield xs.select(selxpath)[0] class _StreamReader(object): diff --git a/scrapy/link.py b/scrapy/link.py index ddae07823..e4a25784a 100644 --- a/scrapy/link.py +++ b/scrapy/link.py @@ -10,15 +10,19 @@ class Link(object): At the moment, it contains just the url and link text. """ - __slots__ = ['url', 'text'] + __slots__ = ['url', 'text', 'nofollow'] - def __init__(self, url, text=''): + def __init__(self, url, text='', nofollow=False): self.url = url self.text = text + self.nofollow = nofollow def __eq__(self, other): - return self.url == other.url and self.text == other.text + return self.url == other.url and self.text == other.text and self.nofollow == other.nofollow + + def __hash__(self): + return hash(self.url) ^ hash(self.text) ^ hash(self.nofollow) def __repr__(self): - return '' % (self.url, self.text) + return 'Link(url=%r, text=%r, nofollow=%r)' % (self.url, self.text, self.nofollow) diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 015b1d911..6a072b063 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -155,6 +155,7 @@ HTTPCACHE_STORAGE = 'scrapy.contrib.downloadermiddleware.httpcache.FilesystemCac HTTPCACHE_EXPIRATION_SECS = 0 HTTPCACHE_IGNORE_HTTP_CODES = [] HTTPCACHE_IGNORE_SCHEMES = ['file'] +HTTPCACHE_DBM_MODULE = 'anydbm' ITEM_PROCESSOR = 'scrapy.contrib.pipeline.ItemPipelineManager' diff --git a/scrapy/tests/test_contrib_exporter.py b/scrapy/tests/test_contrib_exporter.py index 8b776f587..974174c42 100644 --- a/scrapy/tests/test_contrib_exporter.py +++ b/scrapy/tests/test_contrib_exporter.py @@ -127,6 +127,18 @@ class CsvItemExporterTest(BaseItemExporterTest): ie.finish_exporting() self.assertEqual(output.getvalue(), '22,John\xc2\xa3\r\n') + def test_join_multivalue(self): + class TestItem2(Item): + name = Field() + friends = Field() + + i = TestItem2(name='John', friends=['Mary', 'Paul']) + output = StringIO() + ie = CsvItemExporter(output, include_headers_line=False) + ie.start_exporting() + ie.export_item(i) + ie.finish_exporting() + self.assertEqual(output.getvalue(), '"Mary,Paul",John\r\n') class XmlItemExporterTest(BaseItemExporterTest): diff --git a/scrapy/tests/test_contrib_ibl/test_extraction.py b/scrapy/tests/test_contrib_ibl/test_extraction.py index f79c9dc78..7b82f7b33 100644 --- a/scrapy/tests/test_contrib_ibl/test_extraction.py +++ b/scrapy/tests/test_contrib_ibl/test_extraction.py @@ -9,7 +9,7 @@ from scrapy.contrib.ibl.htmlpage import HtmlPage from scrapy.contrib.ibl.descriptor import (FieldDescriptor as A, ItemDescriptor) from scrapy.contrib.ibl.extractors import (contains_any_numbers, - image_url) + image_url, html, notags) try: import numpy @@ -249,12 +249,12 @@ SL342 Nice product for ladies
-£s;85.00 +£85.00

-£s;100.00 +£100.00 """ @@ -266,9 +266,9 @@ SL342
Nice product for ladies
-£s;85.00 +£85.00

-£s;100.00 +£100.00 """ @@ -516,7 +516,7 @@ ANNOTATED_PAGE19 = u"""

Product name

60.00

- +

description

@@ -719,19 +719,23 @@ EXTRACT_PAGE23 = u""" """ +DEFAULT_DESCRIPTOR = ItemDescriptor('test', + 'item test, removes tags from description attribute', + [A('description', 'description field without tags', notags)]) + SAMPLE_DESCRIPTOR1 = ItemDescriptor('test', 'product test', [ A('name', "Product name", required=True), A('price', "Product price, including any discounts and tax or vat", contains_any_numbers, True), A('image_urls', "URLs for one or more images", image_url, True), - A('description', "The full description of the product", allow_markup=True), + A('description', "The full description of the product", html), ] ) # A list of (test name, [templates], page, extractors, expected_result) TEST_DATA = [ # extract from a similar page - ('similar page extraction', [ANNOTATED_PAGE1], EXTRACT_PAGE1, None, + ('similar page extraction', [ANNOTATED_PAGE1], EXTRACT_PAGE1, DEFAULT_DESCRIPTOR, {u'title': [u'Nice Product'], u'description': [u'wonderful product'], u'image_url': [u'nice_product.jpg']} ), @@ -743,20 +747,20 @@ TEST_DATA = [ u'image_url': [u'nice_product.jpg']} ), # compilicated tag (multiple attributes and annotation) - ('multiple attributes and annotation', [ANNOTATED_PAGE2], EXTRACT_PAGE2, None, + ('multiple attributes and annotation', [ANNOTATED_PAGE2], EXTRACT_PAGE2, DEFAULT_DESCRIPTOR, {'name': [u'product 1'], 'image_url': [u'http://example.com/product1.jpg'], 'description': [u'product 1 is great']} ), # can only work out correct placement by matching the second attribute first - ('ambiguous description', [ANNOTATED_PAGE3], EXTRACT_PAGE3, None, + ('ambiguous description', [ANNOTATED_PAGE3], EXTRACT_PAGE3, DEFAULT_DESCRIPTOR, {'description': [u'description'], 'delivery': [u'delivery']} ), # infer a repeated structure - ('repeated elements', [ANNOTATED_PAGE4], EXTRACT_PAGE4, None, + ('repeated elements', [ANNOTATED_PAGE4], EXTRACT_PAGE4, DEFAULT_DESCRIPTOR, {'features': [u'feature1', u'feature2', u'feature3']} ), # identical variants with a repeated structure - ('repeated identical variants', [ANNOTATED_PAGE5], EXTRACT_PAGE5, None, + ('repeated identical variants', [ANNOTATED_PAGE5], EXTRACT_PAGE5, DEFAULT_DESCRIPTOR, { 'description': [u'description'], 'variants': [ @@ -767,7 +771,7 @@ TEST_DATA = [ } ), # variants with an irregular structure - ('irregular variants', [ANNOTATED_PAGE6], EXTRACT_PAGE6, None, + ('irregular variants', [ANNOTATED_PAGE6], EXTRACT_PAGE6, DEFAULT_DESCRIPTOR, { 'description': [u'description'], 'variants': [ @@ -779,7 +783,7 @@ TEST_DATA = [ ), # discovering repeated variants in table columns -# ('variants in table columns', [ANNOTATED_PAGE7], EXTRACT_PAGE7, None, +# ('variants in table columns', [ANNOTATED_PAGE7], EXTRACT_PAGE7, DEFAULT_DESCRIPTOR, # {'variants': [ # {u'colour': [u'colour 1'], u'price': [u'price 1']}, # {u'colour': [u'colour 2'], u'price': [u'price 2']}, @@ -790,66 +794,66 @@ TEST_DATA = [ # ignored regions ( - 'ignored_regions', [ANNOTATED_PAGE8], EXTRACT_PAGE8, None, + 'ignored_regions', [ANNOTATED_PAGE8], EXTRACT_PAGE8, DEFAULT_DESCRIPTOR, { - 'description': [u'\n A very nice product for all intelligent people \n\n'], + 'description': [u'\n A very nice product for all intelligent people \n \n'], 'price': [u'\n12.00\n(VAT exc.)'], } ), # shifted ignored regions (detected by region similarity) ( - 'shifted_ignored_regions', [ANNOTATED_PAGE9], EXTRACT_PAGE9, None, + 'shifted_ignored_regions', [ANNOTATED_PAGE9], EXTRACT_PAGE9, DEFAULT_DESCRIPTOR, { - 'description': [u'\n A very nice product for all intelligent people \n\n'], + 'description': [u'\n A very nice product for all intelligent people \n \n'], 'price': [u'\n12.00\n(VAT exc.)'], } ), (# special case with partial annotations - 'special_partial_annotation', [ANNOTATED_PAGE11], EXTRACT_PAGE11, None, + 'special_partial_annotation', [ANNOTATED_PAGE11], EXTRACT_PAGE11, DEFAULT_DESCRIPTOR, { 'name': [u'SL342'], - 'description': [u'\nSL342\n \nNice product for ladies\n \n£s;85.00\n'], - 'price': [u'£s;85.00'], - 'price_before_discount': [u'£s;100.00'], + 'description': ['\nSL342\n \nNice product for ladies\n \n£85.00\n'], + 'price': [u'\xa385.00'], + 'price_before_discount': [u'\xa3100.00'], } ), (# with ignore-beneath feature - 'ignore-beneath', [ANNOTATED_PAGE12], EXTRACT_PAGE12a, None, + 'ignore-beneath', [ANNOTATED_PAGE12], EXTRACT_PAGE12a, DEFAULT_DESCRIPTOR, { 'description': [u'\n A very nice product for all intelligent people \n'], } ), (# ignore-beneath with extra tags - 'ignore-beneath with extra tags', [ANNOTATED_PAGE12], EXTRACT_PAGE12b, None, + 'ignore-beneath with extra tags', [ANNOTATED_PAGE12], EXTRACT_PAGE12b, DEFAULT_DESCRIPTOR, { 'description': [u'\n A very nice product for all intelligent people \n'], } ), - ('nested annotation with replica outside', [ANNOTATED_PAGE13a], EXTRACT_PAGE13a, None, + ('nested annotation with replica outside', [ANNOTATED_PAGE13a], EXTRACT_PAGE13a, DEFAULT_DESCRIPTOR, {'description': [u'\n A product \n $50.00 \nThis product is excelent. Buy it!\n \n'], 'price': ["$50.00"], 'name': [u'A product']} ), - ('outside annotation with nested replica', [ANNOTATED_PAGE13b], EXTRACT_PAGE13b, None, + ('outside annotation with nested replica', [ANNOTATED_PAGE13b], EXTRACT_PAGE13b, DEFAULT_DESCRIPTOR, {'description': [u'\n A product \n $50.00 \nThis product is excelent. Buy it!\n'], 'price': ["$45.00"], 'name': [u'A product']} ), - ('consistency check', [ANNOTATED_PAGE14], EXTRACT_PAGE14, None, + ('consistency check', [ANNOTATED_PAGE14], EXTRACT_PAGE14, DEFAULT_DESCRIPTOR, {}, ), - ('consecutive nesting', [ANNOTATED_PAGE15], EXTRACT_PAGE15, None, - {'description': [u'Description\n\n'], + ('consecutive nesting', [ANNOTATED_PAGE15], EXTRACT_PAGE15, DEFAULT_DESCRIPTOR, + {'description': [u'Description\n \n'], 'price': [u'80.00']}, ), - ('nested inside not found', [ANNOTATED_PAGE16], EXTRACT_PAGE16, None, + ('nested inside not found', [ANNOTATED_PAGE16], EXTRACT_PAGE16, DEFAULT_DESCRIPTOR, {'price': [u'90.00'], 'name': [u'product name']}, ), - ('ignored region helps to find attributes', [ANNOTATED_PAGE17], EXTRACT_PAGE17, None, + ('ignored region helps to find attributes', [ANNOTATED_PAGE17], EXTRACT_PAGE17, DEFAULT_DESCRIPTOR, {'description': [u'\nThis product is excelent. Buy it!\n']}, ), - ('ignored region in partial annotation', [ANNOTATED_PAGE18], EXTRACT_PAGE18, None, + ('ignored region in partial annotation', [ANNOTATED_PAGE18], EXTRACT_PAGE18, DEFAULT_DESCRIPTOR, {u'site_id': [u'Item Id'], u'description': [u'\nDescription\n']}, ), @@ -864,13 +868,13 @@ TEST_DATA = [ SAMPLE_DESCRIPTOR1, None, ), - ('repeated partial annotations with variants', [ANNOTATED_PAGE20], EXTRACT_PAGE20, None, + ('repeated partial annotations with variants', [ANNOTATED_PAGE20], EXTRACT_PAGE20, DEFAULT_DESCRIPTOR, {u'variants': [ {'price': ['270'], 'name': ['Twin']}, {'price': ['330'], 'name': ['Queen']}, ]}, ), - ('variants with swatches', [ANNOTATED_PAGE21], EXTRACT_PAGE21, None, + ('variants with swatches', [ANNOTATED_PAGE21], EXTRACT_PAGE21, DEFAULT_DESCRIPTOR, {u'category': [u'chairs'], u'image_urls': [u'image.jpg'], u'variants': [ @@ -881,7 +885,7 @@ TEST_DATA = [ ] }, ), - ('variants with swatches complete', [ANNOTATED_PAGE22], EXTRACT_PAGE22, None, + ('variants with swatches complete', [ANNOTATED_PAGE22], EXTRACT_PAGE22, DEFAULT_DESCRIPTOR, {u'category': [u'chairs'], u'variants': [ {u'swatches': [u'swatch1.jpg'], @@ -899,7 +903,7 @@ TEST_DATA = [ ], u'image_urls': [u'image.jpg']}, ), - ('repeated (variants) with ignore annotations', [ANNOTATED_PAGE23], EXTRACT_PAGE23, None, + ('repeated (variants) with ignore annotations', [ANNOTATED_PAGE23], EXTRACT_PAGE23, DEFAULT_DESCRIPTOR, {'variants': [ {u'price': [u'300'], u'name': [u'Variant 1']}, {u'price': [u'320'], u'name': [u'Variant 2']}, diff --git a/scrapy/tests/test_contrib_ibl/test_htmlpage.py b/scrapy/tests/test_contrib_ibl/test_htmlpage.py index cdba56853..d85ae1afb 100644 --- a/scrapy/tests/test_contrib_ibl/test_htmlpage.py +++ b/scrapy/tests/test_contrib_ibl/test_htmlpage.py @@ -137,3 +137,19 @@ class TestParseHtml(TestCase): parsed = list(parse_html("")) self.assertEqual(parsed[0].attributes, {'src': 'http://images.play.com/banners/SAM550a.jpg', \ 'align': 'left', 'hspace': '5', '/': None}) + + def test_no_ending_body(self): + """Test case when no ending body nor html elements are present""" + parsed = [_decode_element(d) for d in PARSED7] + self._test_sample(PAGE7, parsed) + + def test_malformed(self): + """Test parsing of some malformed cases""" + parsed = [_decode_element(d) for d in PARSED8] + self._test_sample(PAGE8, parsed) + + def test_malformed2(self): + """Test case when attributes are not separated by space (still recognizable because of quotes)""" + parsed = [_decode_element(d) for d in PARSED9] + self._test_sample(PAGE9, parsed) + diff --git a/scrapy/tests/test_contrib_ibl/test_htmlpage_data.py b/scrapy/tests/test_contrib_ibl/test_htmlpage_data.py index 62cd6b526..f54dc9f8c 100644 --- a/scrapy/tests/test_contrib_ibl/test_htmlpage_data.py +++ b/scrapy/tests/test_contrib_ibl/test_htmlpage_data.py @@ -246,3 +246,32 @@ PARSED7 = [ {'end': 99, 'start': 85}, ] +PAGE8 = u"""""" + +PARSED8 = [ + {'attributes' : {u'href' : u"/overview.asp?id=277"}, 'end': 31, 'start': 0, 'tag': u'a', 'tag_type': 1}, + {'attributes' : {u'src' : u"/img/5200814311.jpg", u'border' : u"0", u'title': u'Vinyl Cornice'}, 'end': 94, 'start': 31, 'tag': u'img', 'tag_type': 1}, + {'attributes' : {}, 'end': 98, 'start': 94, 'tag': u'a', 'tag_type': 2}, + {'attributes' : {}, 'end': 103, 'start': 98, 'tag': u'td', 'tag_type': 2}, + {'attributes' : {u'width': u'5'}, 'end': 120, 'start': 103, 'tag': u'table', 'tag_type': 1} +] + +PAGE9 = u"""\ +\ +\ +\ +Click here\ +\ +\ +""" + +PARSED9 = [ + {'attributes' : {}, 'end': 6, 'start': 0, 'tag': 'html', 'tag_type': 1}, + {'attributes' : {}, 'end': 12, 'start': 6, 'tag': 'body', 'tag_type': 1}, + {'attributes' : {'width': '230', 'height': '150', 'src': '/images/9589.jpg'}, 'end': 65, 'start': 12, 'tag': 'img', 'tag_type': 1}, + {'attributes' : {'href': '/product/9589'}, 'end': 89, 'start': 65, 'tag': 'a', 'tag_type': 1}, + {'end': 99, 'start': 89}, + {'attributes' : {}, 'end': 103, 'start': 99, 'tag': 'a', 'tag_type': 2}, + {'attributes' : {}, 'end': 110, 'start': 103, 'tag': 'body', 'tag_type': 2}, + {'attributes' : {}, 'end': 117, 'start': 110, 'tag': 'html', 'tag_type': 2}, +] diff --git a/scrapy/tests/test_contrib_ibl/test_pageparsing.py b/scrapy/tests/test_contrib_ibl/test_pageparsing.py index 1e3d610bb..d694d08a1 100644 --- a/scrapy/tests/test_contrib_ibl/test_pageparsing.py +++ b/scrapy/tests/test_contrib_ibl/test_pageparsing.py @@ -228,11 +228,11 @@ class TestPageParsing(TestCase): epp = _parse_page(ExtractionPageParser, SIMPLE_PAGE) ep = epp.to_extraction_page() assert len(ep.page_tokens) == 4 - assert ep.token_html(0) == '' - assert ep.token_html(1) == '

' + assert ep.htmlpage.fragment_data(ep.htmlpage_tag(0)) == '' + assert ep.htmlpage.fragment_data(ep.htmlpage_tag(1)) == '

' - assert ep.html_between_tokens(1, 2) == 'this is a test' - assert ep.html_between_tokens(1, 3) == 'this is a test

' + assert ep.htmlpage_region_inside(1, 2) == 'this is a test' + assert ep.htmlpage_region_inside(1, 3) == 'this is a test

' def test_invalid_html(self): p = _parse_page(InstanceLearningParser, BROKEN_PAGE) diff --git a/scrapy/tests/test_link.py b/scrapy/tests/test_link.py new file mode 100644 index 000000000..32e0095e6 --- /dev/null +++ b/scrapy/tests/test_link.py @@ -0,0 +1,28 @@ +import unittest + +from scrapy.link import Link + +class LinkTest(unittest.TestCase): + + def test_eq_and_hash(self): + l1 = Link("http://www.example.com") + l2 = Link("http://www.example.com/other") + l3 = Link("http://www.example.com") + + self.assertEqual(l1, l1) + self.assertEqual(hash(l1), hash(l1)) + self.assertNotEqual(l1, l2) + self.assertNotEqual(hash(l1), hash(l2)) + self.assertEqual(l1, l3) + self.assertEqual(hash(l1), hash(l3)) + + l4 = Link("http://www.example.com", text="test") + l5 = Link("http://www.example.com", text="test2") + l6 = Link("http://www.example.com", text="test") + + self.assertEqual(l4, l4) + self.assertEqual(hash(l4), hash(l4)) + self.assertNotEqual(l4, l5) + self.assertNotEqual(hash(l4), hash(l5)) + self.assertEqual(l4, l6) + self.assertEqual(hash(l4), hash(l6)) diff --git a/scrapy/tests/test_utils_iterators.py b/scrapy/tests/test_utils_iterators.py index d32d1e658..06fb4effe 100644 --- a/scrapy/tests/test_utils_iterators.py +++ b/scrapy/tests/test_utils_iterators.py @@ -29,12 +29,12 @@ class XmliterTestCase(unittest.TestCase): for x in self.xmliter(response, 'product'): attrs.append((x.select("@id").extract(), x.select("name/text()").extract(), x.select("./type/text()").extract())) - self.assertEqual(attrs, + self.assertEqual(attrs, [(['001'], ['Name 1'], ['Type 1']), (['002'], ['Name 2'], ['Type 2'])]) def test_xmliter_text(self): body = u"""onetwo""" - + self.assertEqual([x.select("text()").extract() for x in self.xmliter(body, 'product')], [[u'one'], [u'two']]) @@ -74,7 +74,7 @@ class XmliterTestCase(unittest.TestCase): def test_xmliter_exception(self): body = u"""onetwo""" - + iter = self.xmliter(body, 'product') iter.next() iter.next() @@ -97,6 +97,35 @@ class LxmlXmliterTestCase(XmliterTestCase): except ImportError: skip = "lxml not available" + def test_xmliter_iterate_namespace(self): + body = """\ + + + + My Dummy Company + http://www.mydummycompany.com + This is a dummy company. We do nothing. + + Item 1 + This is item 1 + http://www.mydummycompany.com/items/1 + http://www.mydummycompany.com/images/item1.jpg + http://www.mydummycompany.com/images/item2.jpg + + + + """ + response = XmlResponse(url='http://mydummycompany.com', body=body) + + no_namespace_iter = self.xmliter(response, 'image_link') + self.assertEqual(len(list(no_namespace_iter)), 0) + + namespace_iter = self.xmliter(response, 'image_link', 'http://base.google.com/ns/1.0') + node = namespace_iter.next() + self.assertEqual(node.select('text()').extract(), ['http://www.mydummycompany.com/images/item1.jpg']) + node = namespace_iter.next() + self.assertEqual(node.select('text()').extract(), ['http://www.mydummycompany.com/images/item2.jpg']) + class UtilsCsvTestCase(unittest.TestCase): sample_feeds_dir = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'sample_data', 'feeds') diff --git a/scrapy/utils/request.py b/scrapy/utils/request.py index 6ebb3d33f..3e3c04742 100644 --- a/scrapy/utils/request.py +++ b/scrapy/utils/request.py @@ -64,13 +64,6 @@ def request_authenticate(request, username, password): """ request.headers['Authorization'] = basic_auth_header(username, password) -def request_info(request): - """Return a short string with request info including method, url and - fingeprint. Mainly used for debugging - """ - fp = request_fingerprint(request) - return "" % (request.method, request.url, fp[:8]) - def request_httprepr(request): """Return the raw HTTP representation (as string) of the given request. This is provided only for reference since it's not the actual stream of diff --git a/scrapyd/webservice.py b/scrapyd/webservice.py index 73af9f15d..637f807d7 100644 --- a/scrapyd/webservice.py +++ b/scrapyd/webservice.py @@ -87,3 +87,12 @@ class DeleteVersion(DeleteProject): self._delete_version(project, version) return {"status": "ok"} +class ListJobs(WsResource): + def render_POST(self, txrequest): + project = txrequest.args['project'][0] + spiders = self.root.launcher.processes.values() + jlist = list() + for s in spiders: + if project == s.project: + jlist.append({"job": {"id":s.job, "spider": s.spider}}) + return {"status":"ok", "jobs": jlist} diff --git a/scrapyd/website.py b/scrapyd/website.py index 4889ad1ca..db23de51f 100644 --- a/scrapyd/website.py +++ b/scrapyd/website.py @@ -22,6 +22,7 @@ class Root(resource.Resource): self.putChild('listspiders.json', webservice.ListSpiders(self)) self.putChild('delproject.json', webservice.DeleteProject(self)) self.putChild('delversion.json', webservice.DeleteVersion(self)) + self.putChild('listjobs.json', webservice.ListJobs(self)) self.putChild('logs', static.File(logsdir, 'text/plain')) self.putChild('procmon', ProcessMonitor(self)) self.update_projects()