diff --git a/docs/faq.rst b/docs/faq.rst index 712f3b585..af6103828 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -26,15 +26,17 @@ comparing `jinja2`_ to `Django`_. .. _jinja2: http://jinja.pocoo.org/ .. _Django: https://www.djangoproject.com/ -How can I use Scrapy with BeautifulSoup? ----------------------------------------- +Can I use Scrapy with BeautifulSoup? +------------------------------------ -As mentioned :ref:`above `, BeautifulSoup can be used +Yes, you can. +As mentioned :ref:`above `, `BeautifulSoup`_ can be used for parsing HTML responses in Scrapy callbacks. You just have to feed the response's body into a ``BeautifulSoup`` object and extract whatever data you need from it. -Here's an example spider using ``lxml`` parser with BeautifulSoup API:: +Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser +(so you get the same parsing speed as with scrapy/parsel selectors):: from bs4 import BeautifulSoup @@ -49,12 +51,19 @@ Here's an example spider using ``lxml`` parser with BeautifulSoup API:: ) def parse(self, response): + # use lxml to get decent HTML parsing speed soup = BeautifulSoup(response.text, 'lxml') yield { "url": response.url, "title": soup.h1.string } +.. note:: + + ``BeautifulSoup`` supports several HTML/XML parsers. + See `BeautifulSoup's official documentation`_ on which ones are available. + +.. _BeautifulSoup's official documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#specifying-the-parser-to-use .. _faq-python-versions: