RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update headless browser docs to mention playwright

This commit is contained in:
Eugenio Lacuesta 2021-07-28 15:00:24 -03:00
parent 9aef9b78eb
commit 4b62ac6c3a
No known key found for this signature in database
GPG Key ID: DA3EF2D0913E9810
1 changed files with 23 additions and 32 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
docs/topics/dynamic-content.rst
View File

@ -246,55 +246,46 @@ Using a headless browser
========================
A `headless browser`_ is a special web browser that provides an API for
automation.
automation. By installing the :ref:`asyncio reactor <install-asyncio>`,
it is possible to integrate ``asyncio``-based libraries which handle headless browsers.
Since version 2.0, it is possible to integrate libraries that use the
``async/await`` syntax. One such library is `pyppeteer`_ (an unnoficial
Python port of `puppeteer`_), which uses headless Chrome to download and
render pages.
One such library is `playwright-python`_ (an official Python port of `playwright`_).
The following is a simple snippet to illustrate its usage within a Scrapy spider::
The following is a simple snippet to illustrate its usage within Scrapy::
import pyppeteer
import scrapy
from playwright.async_api import async_playwright
class PyppeteerSpider(scrapy.Spider):
name = "pyppeteer"
start_urls = ["data:,"] # avoid making an actual upstream request
class PlaywrightSpider(scrapy.Spider):
name = "playwright"
start_urls = ["data:,"] # avoid using the default Scrapy downloader
async def parse(self, response):
browser = await pyppeteer.launch()
page = await browser.newPage()
await page.goto("https:/example.org")
title = await page.title()
await page.close()
yield {"title": title}
async with async_playwright() as pw:
browser = await pw.chromium.launch()
page = await browser.new_page()
await page.goto("https:/example.org")
title = await page.title()
return {"title": title}
For this example to work, Scrapy needs to be running on top of the
:ref:`asyncio reactor <install-asyncio>`.
Using pypeteer_ directly circumvents most of the
Scrapy components (middlewares, dupefilter, etc). Use
one of the following Scrapy plugins for better integration
with Scrapy:
* https://github.com/elacuesta/scrapy-pyppeteer
* https://github.com/lopuhin/scrapy-pyppeteer
* https://github.com/clemfromspace/scrapy-puppeteer
However, using `playwright-python`_ directly as in the above example
circumvents most of the Scrapy components (middlewares, dupefilter, etc).
We recommend using `scrapy-playwright`_ for a better integration.
.. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29
.. _chompjs: https://github.com/Nykakin/chompjs
.. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets
.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
.. _Splash: https://github.com/scrapinghub/splash
.. _chompjs: https://github.com/Nykakin/chompjs
.. _curl: https://curl.haxx.se/
.. _headless browser: https://en.wikipedia.org/wiki/Headless_browser
.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript
.. _js2xml: https://github.com/scrapinghub/js2xml
.. _puppeteer: https://pptr.dev/
.. _playwright-python: https://github.com/microsoft/playwright-python
.. _playwright: https://github.com/microsoft/playwright
.. _pyppeteer: https://pyppeteer.github.io/pyppeteer/
.. _pytesseract: https://github.com/madmaze/pytesseract
.. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright
.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
.. _Splash: https://github.com/scrapinghub/splash
.. _tabula-py: https://github.com/chezou/tabula-py
.. _wget: https://www.gnu.org/software/wget/
.. _wgrep: https://github.com/stav/wgrep