Update headless browser docs

This commit is contained in:
Eugenio Lacuesta 2020-06-06 16:32:43 -03:00
parent 63929e75e3
commit bdabc500aa
No known key found for this signature in database
GPG Key ID: DA3EF2D0913E9810
1 changed files with 31 additions and 5 deletions

View File

@ -248,8 +248,34 @@ Using a headless browser
A `headless browser`_ is a special web browser that provides an API for
automation.
The easiest way to use a headless browser with Scrapy is to use Selenium_,
along with `scrapy-selenium`_ for seamless integration.
Since version 2.0, it is possible to integrate libraries that use the
``async/await`` syntax. One such library is `pyppeteer`_ (an unnoficial
Python port of `puppeteer`_), which uses headless Chrome to download and
render pages.
The following is a simple snippet to illustrate its usage within Scrapy::
import pyppeteer
import scrapy
class PyppeteerSpider(scrapy.Spider):
name = "pyppeteer"
start_urls = ["data:,"] # avoid making an actual upstream request
async def parse(self, response):
browser = await pyppeteer.launch()
page = await browser.newPage()
await page.goto("https:/example.org")
title = await page.title()
yield {"title": title}
Keep in mind that this is just a proof of concept, since it circumvents
most of the Scrapy components (middlewares, dupefilter, etc).
There are some 3rd party projects which provider better integration:
* https://github.com/elacuesta/scrapy-pyppeteer
* https://github.com/lopuhin/scrapy-pyppeteer
* https://github.com/clemfromspace/scrapy-puppeteer
.. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29
@ -259,11 +285,11 @@ along with `scrapy-selenium`_ for seamless integration.
.. _headless browser: https://en.wikipedia.org/wiki/Headless_browser
.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
.. _js2xml: https://github.com/scrapinghub/js2xml
.. _puppeteer: https://pptr.dev/
.. _pyppeteer: https://pyppeteer.github.io/pyppeteer/
.. _pytesseract: https://github.com/madmaze/pytesseract
.. _scrapy-selenium: https://github.com/clemfromspace/scrapy-selenium
.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
.. _Selenium: https://www.selenium.dev/
.. _Splash: https://github.com/scrapinghub/splash
.. _tabula-py: https://github.com/chezou/tabula-py
.. _wget: https://www.gnu.org/software/wget/
.. _wgrep: https://github.com/stav/wgrep
.. _wgrep: https://github.com/stav/wgrep