diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index 4b362f84d..bf52186cd 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -192,9 +192,9 @@ These are just a couple of simple examples of what you can do with XPath, but XPath expression are indeed much more powerful. To learn more about XPath we recommend `this XPath tutorial `_. -For working with XPaths, Scrapy provides a :class:`~scrapy.xpath.XPathSelector` -class, which comes in two flavours, :class:`~scrapy.xpath.HtmlXPatSelector` -(for HTML data) and :class:`~scrapy.xpath.XmlXPathSelector` (for XML data). In +For working with XPaths, Scrapy provides a :class:`~scrapy.selector.XPathSelector` +class, which comes in two flavours, :class:`~scrapy.selector.HtmlXPatSelector` +(for HTML data) and :class:`~scrapy.selector.XmlXPathSelector` (for XML data). In order to use them you must instantiate the desired class with a :class:`~scrapy.http.Response` object. @@ -205,14 +205,14 @@ node, or the entire document. Selectors have three methods (click on the method to see the complete API documentation). -* :meth:`~scrapy.xpath.XPathSelector.x`: returns a list of selectors, each of +* :meth:`~scrapy.selector.XPathSelector.x`: returns a list of selectors, each of them representing the nodes selected by the xpath expression given as argument. -* :meth:`~scrapy.xpath.XPathSelector.extract`: returns a unicode string with +* :meth:`~scrapy.selector.XPathSelector.extract`: returns a unicode string with the data selected by the XPath selector. -* :meth:`~scrapy.xpath.XPathSelector.re`: returns a list unicode strings +* :meth:`~scrapy.selector.XPathSelector.re`: returns a list unicode strings extracted by applying the regular expression given as argument. @@ -235,10 +235,10 @@ This is what the shell looks like:: ------------------------------------------------------------------------------ Available Scrapy variables: - xxs: + xxs: url: http://www.dmoz.org/Computers/Programming/Languages/Python/Books/ spider: - hxs: + hxs: item: response: Available commands: @@ -331,7 +331,7 @@ that property here, so:: Let's add this code to our spider:: from scrapy.spider import BaseSpider - from scrapy.xpath.selector import HtmlXPathSelector + from scrapy.selector import HtmlXPathSelector class DmozSpider(BaseSpider): domain_name = "dmoz.org" @@ -362,7 +362,7 @@ Spiders are expected to return their scraped data inside scraped so far, the code for our Spider should be like this:: from scrapy.spider import BaseSpider - from scrapy.xpath.selector import HtmlXPathSelector + from scrapy.selector import HtmlXPathSelector from dmoz.items import DmozItem diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index 8fd1f1c89..bfaa29c32 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -333,7 +333,7 @@ ItemLoader objects :param selector: The selector to extract data from, when using the :meth:`add_xpath` or :meth:`replace_xpath` method. - :type selector: :class:`~scrapy.xpath.XPathSelector` object + :type selector: :class:`~scrapy.selector.XPathSelector` object :param response: The response used to construct the selector using the :attr:`default_selector_class`, unless the selector argument is given, @@ -346,7 +346,7 @@ ItemLoader objects value, which is used to extract a list of unicode strings from the selector associated with this :class:`XPathItemLoader`. If the ``re`` argument is given, it's used for extrating data from the selector using - the :meth:`~scrapy.xpath.XPathSelector.re` method. + the :meth:`~scrapy.selector.XPathSelector.re` method. :param xpath: the XPath to extract data from :type xpath: str @@ -376,7 +376,7 @@ ItemLoader objects .. attribute:: selector - The :class:`~scrapy.xpath.XPathSelector` object to extract data from. + The :class:`~scrapy.selector.XPathSelector` object to extract data from. It's either the selector given in the constructor or one created from the response given in the constructor using the :attr:`default_selector_class`. This attribute is meant to be @@ -547,7 +547,7 @@ Here is a list of all built-in processors: work with single values (instead of iterables). For this reason the :class:`MapCompose` processor is typically used as input processor, since data is often extracted using the - :meth:`~scrapy.xpath.XPathSelector.extract` method of :ref:`selectors + :meth:`~scrapy.selector.XPathSelector.extract` method of :ref:`selectors `, which returns a list of unicode strings. The example below should clarify how it works:: diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index cc6e01b6f..29b46f573 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -49,9 +49,9 @@ Constructing selectors There are two types of selectors bundled with Scrapy. Those are: - * :class:`~scrapy.xpath.HtmlXPathSelector` - for working with HTML documents + * :class:`~scrapy.selector.HtmlXPathSelector` - for working with HTML documents - * :class:`~scrapy.xpath.XmlXPathSelector` - for working with XML documents + * :class:`~scrapy.selector.XmlXPathSelector` - for working with XML documents .. highlight:: python @@ -89,7 +89,7 @@ Then, after the shell loads, you'll have some selectors already instanced and ready to use. Since we're dealing with HTML we'll be using the -:class:`~scrapy.xpath.HtmlXPathSelector` object which is found, by default, in +:class:`~scrapy.selector.HtmlXPathSelector` object which is found, by default, in the ``hxs`` shell variable. .. highlight:: python @@ -135,7 +135,7 @@ Using selectors with regular expressions Selectors also have a ``re()`` method for extracting data using regular expressions. However, unlike using the ``select()`` method, the ``re()`` method -does not return a list of :class:`~scrapy.xpath.XPathSelector` objects, so you +does not return a list of :class:`~scrapy.selector.XPathSelector` objects, so you can't construct nested ``.re()`` calls. Here's an example used to extract images names from the :ref:`HTML code @@ -216,7 +216,7 @@ XPath specification. Built-in XPath Selectors reference ================================== -.. module:: scrapy.xpath +.. module:: scrapy.selector :synopsis: XPath selectors classes There are two types of selectors bundled with Scrapy: diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index 5badbcd8b..3dea2bbe0 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -66,7 +66,7 @@ Custom Shell Objects The console automatically creates some useful Scrapy objects for the downloaded page, like the :class:`~scrapy.http.Response` object and the -:class:`~scrapy.xpath.XPathSelector` objects (for both HTML and XML content). +:class:`~scrapy.selector.XPathSelector` objects (for both HTML and XML content). Those objects are: @@ -83,10 +83,10 @@ Those objects are: * ``response`` - a :class:`~scrapy.http.Response` object of the last fetched page - * ``hxs`` - a :class:`~scrapy.xpath.HtmlXPathSelector` object for the Response + * ``hxs`` - a :class:`~scrapy.selector.HtmlXPathSelector` object for the Response of the downloaded page - * ``xxs`` - a :class:`~scrapy.xpath.XmlXPathSelector` object for the Response + * ``xxs`` - a :class:`~scrapy.selector.XmlXPathSelector` object for the Response of the downloaded page * ``get ``- download a new response from the given URL and update all diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index b8b8ab8c8..d8fa39f6d 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -219,7 +219,7 @@ Let's now take a look at an example CrawlSpider with rules:: from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor - from scrapy.xpath.selector import HtmlXPathSelector + from scrapy.selector import HtmlXPathSelector from scrapy.item import Item class MySpider(CrawlSpider): @@ -297,7 +297,7 @@ XMLFeedSpider available in that document that will be processed with this spider. The ``prefix`` and ``uri`` will be used to automatically register namespaces using the - :meth:`~scrapy.xpath.XPathSelector.register_namespace` method. + :meth:`~scrapy.selector.XPathSelector.register_namespace` method. You can then specify nodes with namespaces in the :attr:`itertag` attribute. diff --git a/scrapy/command/commands/shell.py b/scrapy/command/commands/shell.py index 6aa4ea05d..56e47384d 100644 --- a/scrapy/command/commands/shell.py +++ b/scrapy/command/commands/shell.py @@ -12,7 +12,7 @@ from twisted.internet import reactor, threads from scrapy.command import ScrapyCommand from scrapy.spider import spiders -from scrapy.xpath import XmlXPathSelector, HtmlXPathSelector +from scrapy.selector import XmlXPathSelector, HtmlXPathSelector from scrapy.utils.misc import load_object from scrapy.conf import settings from scrapy.core.manager import scrapymanager diff --git a/scrapy/contrib/linkextractors/image.py b/scrapy/contrib/linkextractors/image.py index da34cd800..711623b81 100644 --- a/scrapy/contrib/linkextractors/image.py +++ b/scrapy/contrib/linkextractors/image.py @@ -8,7 +8,7 @@ import urlparse from scrapy.link import Link from scrapy.utils.url import canonicalize_url, urljoin_rfc from scrapy.utils.python import unicode_to_str, flatten -from scrapy.xpath.selector import XPathSelectorList, HtmlXPathSelector +from scrapy.selector import XPathSelectorList, HtmlXPathSelector class HTMLImageLinkExtractor(object): '''HTMLImageLinkExtractor objects are intended to extract image links from HTML pages diff --git a/scrapy/contrib/linkextractors/sgml.py b/scrapy/contrib/linkextractors/sgml.py index 710684e72..d548626ab 100644 --- a/scrapy/contrib/linkextractors/sgml.py +++ b/scrapy/contrib/linkextractors/sgml.py @@ -4,7 +4,7 @@ SGMLParser-based Link extractors import re -from scrapy.xpath import HtmlXPathSelector +from scrapy.selector import HtmlXPathSelector from scrapy.link import Link from scrapy.utils.misc import arg_to_iter from scrapy.utils.python import FixedSGMLParser, unique as unique_list, str_to_unicode diff --git a/scrapy/contrib/loader/__init__.py b/scrapy/contrib/loader/__init__.py index b91516696..8de50923c 100644 --- a/scrapy/contrib/loader/__init__.py +++ b/scrapy/contrib/loader/__init__.py @@ -7,7 +7,7 @@ See documentation in docs/topics/loaders.rst from collections import defaultdict from scrapy.item import Item -from scrapy.xpath import HtmlXPathSelector +from scrapy.selector import HtmlXPathSelector from scrapy.utils.misc import arg_to_iter from .common import wrap_loader_context from .processor import Identity diff --git a/scrapy/contrib/spiders/feed.py b/scrapy/contrib/spiders/feed.py index 142e8d916..2991fd557 100644 --- a/scrapy/contrib/spiders/feed.py +++ b/scrapy/contrib/spiders/feed.py @@ -9,7 +9,7 @@ from scrapy.contrib.spiders.init import InitSpider from scrapy.item import BaseItem from scrapy.http import Request from scrapy.utils.iterators import xmliter, csviter -from scrapy.xpath.selector import XmlXPathSelector, HtmlXPathSelector +from scrapy.selector import XmlXPathSelector, HtmlXPathSelector from scrapy.core.exceptions import NotConfigured, NotSupported class XMLFeedSpider(InitSpider): diff --git a/scrapy/selector/__init__.py b/scrapy/selector/__init__.py new file mode 100644 index 000000000..345b3dc51 --- /dev/null +++ b/scrapy/selector/__init__.py @@ -0,0 +1,153 @@ +""" +XPath selectors + +See documentation in docs/ref/selectors.rst +""" + +import libxml2 + +from scrapy.http import TextResponse +from scrapy.utils.python import flatten, unicode_to_str +from scrapy.utils.misc import extract_regex +from scrapy.utils.trackref import object_ref +from scrapy.utils.decorator import deprecated +from .factories import xmlDoc_from_html, xmlDoc_from_xml +from .document import Libxml2Document + +__all__ = ['HtmlXPathSelector', 'XmlXPathSelector', 'XPathSelector', \ + 'XPathSelectorList'] + +class XPathSelector(object_ref): + + __slots__ = ['doc', 'xmlNode', 'expr', '__weakref__'] + + def __init__(self, response=None, text=None, node=None, parent=None, expr=None): + if parent: + self.doc = parent.doc + self.xmlNode = node + elif response: + self.doc = Libxml2Document(response, factory=self._get_libxml2_doc) + self.xmlNode = self.doc.xmlDoc + elif text: + response = TextResponse(url=None, body=unicode_to_str(text), \ + encoding='utf-8') + self.doc = Libxml2Document(response, factory=self._get_libxml2_doc) + self.xmlNode = self.doc.xmlDoc + self.expr = expr + + def select(self, xpath): + """Perform the given XPath query on the current XPathSelector and + return a XPathSelectorList of the result""" + if hasattr(self.xmlNode, 'xpathEval'): + self.doc.xpathContext.setContextNode(self.xmlNode) + try: + xpath_result = self.doc.xpathContext.xpathEval(xpath) + except libxml2.xpathError: + raise ValueError("Invalid XPath: %s" % xpath) + if hasattr(xpath_result, '__iter__'): + return XPathSelectorList([self.__class__(node=node, parent=self, \ + expr=xpath) for node in xpath_result]) + else: + return XPathSelectorList([self.__class__(node=xpath_result, \ + parent=self, expr=xpath)]) + else: + return XPathSelectorList([]) + + def re(self, regex): + """Return a list of unicode strings by applying the regex over all + current XPath selections, and flattening the results""" + return extract_regex(regex, self.extract(), 'utf-8') + + def extract(self): + """Return a unicode string of the content referenced by the XPathSelector""" + if isinstance(self.xmlNode, basestring): + text = unicode(self.xmlNode, 'utf-8', errors='ignore') + elif hasattr(self.xmlNode, 'serialize'): + if isinstance(self.xmlNode, libxml2.xmlDoc): + data = self.xmlNode.getRootElement().serialize('utf-8') + text = unicode(data, 'utf-8', errors='ignore') if data else u'' + elif isinstance(self.xmlNode, libxml2.xmlAttr): + # serialization doesn't work sometimes for xmlAttr types + text = unicode(self.xmlNode.content, 'utf-8', errors='ignore') + else: + data = self.xmlNode.serialize('utf-8') + text = unicode(data, 'utf-8', errors='ignore') if data else u'' + else: + try: + text = unicode(self.xmlNode, 'utf-8', errors='ignore') + except TypeError: # catched when self.xmlNode is a float - see tests + text = unicode(self.xmlNode) + return text + + def extract_unquoted(self): + """Get unescaped contents from the text node (no entities, no CDATA)""" + if self.select('self::text()'): + return unicode(self.xmlNode.getContent(), 'utf-8', errors='ignore') + else: + return u'' + + def register_namespace(self, prefix, uri): + """Register namespace so that it can be used in XPath queries""" + self.doc.xpathContext.xpathRegisterNs(prefix, uri) + + def _get_libxml2_doc(self, response): + """Return libxml2 document (xmlDoc) from response""" + return xmlDoc_from_html(response) + + def __nonzero__(self): + return bool(self.extract()) + + def __str__(self): + return "<%s (%s) xpath=%s>" % (type(self).__name__, getattr(self.xmlNode, \ + 'name', type(self.xmlNode).__name__), self.expr) + + __repr__ = __str__ + + @deprecated(use_instead='XPathSelector.select') + def __call__(self, xpath): + return self.select(xpath) + + @deprecated(use_instead='XPathSelector.select') + def x(self, xpath): + return self.select(xpath) + + +class XPathSelectorList(list): + """List of XPathSelector objects""" + + def __getslice__(self, i, j): + return XPathSelectorList(list.__getslice__(self, i, j)) + + def select(self, xpath): + """Perform the given XPath query on each XPathSelector of the list and + return a new (flattened) XPathSelectorList of the results""" + return XPathSelectorList(flatten([x.select(xpath) for x in self])) + + def re(self, regex): + """Perform the re() method on each XPathSelector of the list, and + return the result as a flattened list of unicode strings""" + return flatten([x.re(regex) for x in self]) + + def extract(self): + """Return a list of unicode strings with the content referenced by each + XPathSelector of the list""" + return [x.extract() if isinstance(x, XPathSelector) else x for x in self] + + def extract_unquoted(self): + return [x.extract_unquoted() if isinstance(x, XPathSelector) else x for x in self] + + @deprecated(use_instead='XPathSelectorList.select') + def x(self, xpath): + return self.select(xpath) + + +class XmlXPathSelector(XPathSelector): + """XPathSelector for XML content""" + __slots__ = () + _get_libxml2_doc = staticmethod(xmlDoc_from_xml) + + +class HtmlXPathSelector(XPathSelector): + """XPathSelector for HTML content""" + __slots__ = () + _get_libxml2_doc = staticmethod(xmlDoc_from_html) diff --git a/scrapy/xpath/document.py b/scrapy/selector/document.py similarity index 96% rename from scrapy/xpath/document.py rename to scrapy/selector/document.py index e03610637..bc4bb4615 100644 --- a/scrapy/xpath/document.py +++ b/scrapy/selector/document.py @@ -5,7 +5,7 @@ garbage collection to libxml2 documents (xmlDoc). import weakref -from scrapy.xpath.factories import xmlDoc_from_html +from .factories import xmlDoc_from_html class Libxml2Document(object): diff --git a/scrapy/xpath/factories.py b/scrapy/selector/factories.py similarity index 100% rename from scrapy/xpath/factories.py rename to scrapy/selector/factories.py diff --git a/scrapy/tests/test_contrib_loader.py b/scrapy/tests/test_contrib_loader.py index 89d736298..f05cb438a 100644 --- a/scrapy/tests/test_contrib_loader.py +++ b/scrapy/tests/test_contrib_loader.py @@ -4,7 +4,7 @@ from scrapy.contrib.loader import ItemLoader, XPathItemLoader from scrapy.contrib.loader.processor import Join, Identity, TakeFirst, \ Compose, MapCompose from scrapy.item import Item, Field -from scrapy.xpath import HtmlXPathSelector +from scrapy.selector import HtmlXPathSelector from scrapy.http import HtmlResponse # test items diff --git a/scrapy/tests/test_xpath.py b/scrapy/tests/test_xpath.py index 582932132..a802da704 100644 --- a/scrapy/tests/test_xpath.py +++ b/scrapy/tests/test_xpath.py @@ -5,9 +5,9 @@ import weakref import libxml2 from scrapy.http import TextResponse, HtmlResponse, XmlResponse -from scrapy.xpath.selector import XmlXPathSelector, HtmlXPathSelector, \ +from scrapy.selector import XmlXPathSelector, HtmlXPathSelector, \ XPathSelector -from scrapy.xpath.document import Libxml2Document +from scrapy.selector.document import Libxml2Document from scrapy.utils.test import libxml2debug class XPathSelectorTestCase(unittest.TestCase): diff --git a/scrapy/utils/iterators.py b/scrapy/utils/iterators.py index 429d920b6..021bc43db 100644 --- a/scrapy/utils/iterators.py +++ b/scrapy/utils/iterators.py @@ -1,7 +1,7 @@ import re, csv from scrapy.http import Response -from scrapy.xpath import XmlXPathSelector +from scrapy.selector import XmlXPathSelector from scrapy import log from scrapy.utils.python import re_rsearch, str_to_unicode from scrapy.utils.response import body_or_str diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py index 4cfb187ff..31f2a2f8b 100644 --- a/scrapy/utils/test.py +++ b/scrapy/utils/test.py @@ -6,7 +6,7 @@ import os import libxml2 -from scrapy.xpath.document import Libxml2Document +from scrapy.selector.document import Libxml2Document def libxml2debug(testfunction): """Decorator for debugging libxml2 memory leaks inside a function. diff --git a/scrapy/xpath/__init__.py b/scrapy/xpath/__init__.py index 611b8f97e..cef3c9869 100644 --- a/scrapy/xpath/__init__.py +++ b/scrapy/xpath/__init__.py @@ -1,9 +1,5 @@ -""" -The scrapy.xpath module provides useful classes for selecting HTML and XML -documents using XPath. It requires libxml2 and its python bindings. +from scrapy.selector import * -This parent module exports the classes most commonly used when building -spiders, for convenience. -""" - -from scrapy.xpath.selector import XPathSelector, XmlXPathSelector, HtmlXPathSelector +import warnings +warnings.warn("scrapy.xpath module is deprecated, use scrapy.selector instead", + DeprecationWarning, stacklevel=2) diff --git a/scrapy/xpath/selector.py b/scrapy/xpath/selector.py index 4cdf55200..66f9834ad 100644 --- a/scrapy/xpath/selector.py +++ b/scrapy/xpath/selector.py @@ -1,149 +1,5 @@ -""" -XPath selectors +from scrapy.selector import * -See documentation in docs/ref/selectors.rst -""" - -import libxml2 - -from scrapy.http import TextResponse -from scrapy.xpath.factories import xmlDoc_from_html, xmlDoc_from_xml -from scrapy.xpath.document import Libxml2Document -from scrapy.utils.python import flatten, unicode_to_str -from scrapy.utils.misc import extract_regex -from scrapy.utils.trackref import object_ref -from scrapy.utils.decorator import deprecated - -class XPathSelector(object_ref): - - __slots__ = ['doc', 'xmlNode', 'expr', '__weakref__'] - - def __init__(self, response=None, text=None, node=None, parent=None, expr=None): - if parent: - self.doc = parent.doc - self.xmlNode = node - elif response: - self.doc = Libxml2Document(response, factory=self._get_libxml2_doc) - self.xmlNode = self.doc.xmlDoc - elif text: - response = TextResponse(url=None, body=unicode_to_str(text), \ - encoding='utf-8') - self.doc = Libxml2Document(response, factory=self._get_libxml2_doc) - self.xmlNode = self.doc.xmlDoc - self.expr = expr - - def select(self, xpath): - """Perform the given XPath query on the current XPathSelector and - return a XPathSelectorList of the result""" - if hasattr(self.xmlNode, 'xpathEval'): - self.doc.xpathContext.setContextNode(self.xmlNode) - try: - xpath_result = self.doc.xpathContext.xpathEval(xpath) - except libxml2.xpathError: - raise ValueError("Invalid XPath: %s" % xpath) - if hasattr(xpath_result, '__iter__'): - return XPathSelectorList([self.__class__(node=node, parent=self, \ - expr=xpath) for node in xpath_result]) - else: - return XPathSelectorList([self.__class__(node=xpath_result, \ - parent=self, expr=xpath)]) - else: - return XPathSelectorList([]) - - def re(self, regex): - """Return a list of unicode strings by applying the regex over all - current XPath selections, and flattening the results""" - return extract_regex(regex, self.extract(), 'utf-8') - - def extract(self): - """Return a unicode string of the content referenced by the XPathSelector""" - if isinstance(self.xmlNode, basestring): - text = unicode(self.xmlNode, 'utf-8', errors='ignore') - elif hasattr(self.xmlNode, 'serialize'): - if isinstance(self.xmlNode, libxml2.xmlDoc): - data = self.xmlNode.getRootElement().serialize('utf-8') - text = unicode(data, 'utf-8', errors='ignore') if data else u'' - elif isinstance(self.xmlNode, libxml2.xmlAttr): - # serialization doesn't work sometimes for xmlAttr types - text = unicode(self.xmlNode.content, 'utf-8', errors='ignore') - else: - data = self.xmlNode.serialize('utf-8') - text = unicode(data, 'utf-8', errors='ignore') if data else u'' - else: - try: - text = unicode(self.xmlNode, 'utf-8', errors='ignore') - except TypeError: # catched when self.xmlNode is a float - see tests - text = unicode(self.xmlNode) - return text - - def extract_unquoted(self): - """Get unescaped contents from the text node (no entities, no CDATA)""" - if self.select('self::text()'): - return unicode(self.xmlNode.getContent(), 'utf-8', errors='ignore') - else: - return u'' - - def register_namespace(self, prefix, uri): - """Register namespace so that it can be used in XPath queries""" - self.doc.xpathContext.xpathRegisterNs(prefix, uri) - - def _get_libxml2_doc(self, response): - """Return libxml2 document (xmlDoc) from response""" - return xmlDoc_from_html(response) - - def __nonzero__(self): - return bool(self.extract()) - - def __str__(self): - return "<%s (%s) xpath=%s>" % (type(self).__name__, getattr(self.xmlNode, \ - 'name', type(self.xmlNode).__name__), self.expr) - - __repr__ = __str__ - - @deprecated(use_instead='XPathSelector.select') - def __call__(self, xpath): - return self.select(xpath) - - @deprecated(use_instead='XPathSelector.select') - def x(self, xpath): - return self.select(xpath) - - -class XPathSelectorList(list): - """List of XPathSelector objects""" - - def __getslice__(self, i, j): - return XPathSelectorList(list.__getslice__(self, i, j)) - - def select(self, xpath): - """Perform the given XPath query on each XPathSelector of the list and - return a new (flattened) XPathSelectorList of the results""" - return XPathSelectorList(flatten([x.select(xpath) for x in self])) - - def re(self, regex): - """Perform the re() method on each XPathSelector of the list, and - return the result as a flattened list of unicode strings""" - return flatten([x.re(regex) for x in self]) - - def extract(self): - """Return a list of unicode strings with the content referenced by each - XPathSelector of the list""" - return [x.extract() if isinstance(x, XPathSelector) else x for x in self] - - def extract_unquoted(self): - return [x.extract_unquoted() if isinstance(x, XPathSelector) else x for x in self] - - @deprecated(use_instead='XPathSelectorList.select') - def x(self, xpath): - return self.select(xpath) - - -class XmlXPathSelector(XPathSelector): - """XPathSelector for XML content""" - __slots__ = () - _get_libxml2_doc = staticmethod(xmlDoc_from_xml) - -class HtmlXPathSelector(XPathSelector): - """XPathSelector for HTML content""" - __slots__ = () - _get_libxml2_doc = staticmethod(xmlDoc_from_html) +import warnings +warnings.warn("scrapy.xpath.selector module is deprecated, use scrapy.selector instead", + DeprecationWarning, stacklevel=2)