From bdabc500aaa37026dfceddf94b41e80f9ce2ce6b Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Sat, 6 Jun 2020 16:32:43 -0300 Subject: [PATCH 1/6] Update headless browser docs --- docs/topics/dynamic-content.rst | 36 ++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index 495111b56..7450de4a2 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -248,8 +248,34 @@ Using a headless browser A `headless browser`_ is a special web browser that provides an API for automation. -The easiest way to use a headless browser with Scrapy is to use Selenium_, -along with `scrapy-selenium`_ for seamless integration. +Since version 2.0, it is possible to integrate libraries that use the +``async/await`` syntax. One such library is `pyppeteer`_ (an unnoficial +Python port of `puppeteer`_), which uses headless Chrome to download and +render pages. +The following is a simple snippet to illustrate its usage within Scrapy:: + + import pyppeteer + import scrapy + + class PyppeteerSpider(scrapy.Spider): + name = "pyppeteer" + start_urls = ["data:,"] # avoid making an actual upstream request + + async def parse(self, response): + browser = await pyppeteer.launch() + page = await browser.newPage() + await page.goto("https:/example.org") + title = await page.title() + yield {"title": title} + +Keep in mind that this is just a proof of concept, since it circumvents +most of the Scrapy components (middlewares, dupefilter, etc). + +There are some 3rd party projects which provider better integration: + +* https://github.com/elacuesta/scrapy-pyppeteer +* https://github.com/lopuhin/scrapy-pyppeteer +* https://github.com/clemfromspace/scrapy-puppeteer .. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29 @@ -259,11 +285,11 @@ along with `scrapy-selenium`_ for seamless integration. .. _headless browser: https://en.wikipedia.org/wiki/Headless_browser .. _JavaScript: https://en.wikipedia.org/wiki/JavaScript .. _js2xml: https://github.com/scrapinghub/js2xml +.. _puppeteer: https://pptr.dev/ +.. _pyppeteer: https://pyppeteer.github.io/pyppeteer/ .. _pytesseract: https://github.com/madmaze/pytesseract -.. _scrapy-selenium: https://github.com/clemfromspace/scrapy-selenium .. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash -.. _Selenium: https://www.selenium.dev/ .. _Splash: https://github.com/scrapinghub/splash .. _tabula-py: https://github.com/chezou/tabula-py .. _wget: https://www.gnu.org/software/wget/ -.. _wgrep: https://github.com/stav/wgrep \ No newline at end of file +.. _wgrep: https://github.com/stav/wgrep From 78aa1b2bfc3eceb09f2fa3dea59811375d6ed1f8 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com> Date: Mon, 8 Jun 2020 11:19:15 -0300 Subject: [PATCH 2/6] Fix typo --- docs/topics/dynamic-content.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index 7450de4a2..e244eb7ff 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -271,7 +271,7 @@ The following is a simple snippet to illustrate its usage within Scrapy:: Keep in mind that this is just a proof of concept, since it circumvents most of the Scrapy components (middlewares, dupefilter, etc). -There are some 3rd party projects which provider better integration: +There are some 3rd party projects which provide better integration: * https://github.com/elacuesta/scrapy-pyppeteer * https://github.com/lopuhin/scrapy-pyppeteer From b6c5289fb900beb552a1ab608f572d99c180b4fb Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Wed, 10 Jun 2020 12:11:49 -0300 Subject: [PATCH 3/6] Close page in pyppeteer example, mention asyncio reactor --- docs/topics/dynamic-content.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index e244eb7ff..56c8b6ae9 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -266,12 +266,16 @@ The following is a simple snippet to illustrate its usage within Scrapy:: page = await browser.newPage() await page.goto("https:/example.org") title = await page.title() + await page.close() yield {"title": title} +For this example to work, Scrapy needs to be running on top of the +:ref:`asyncio reactor `. + Keep in mind that this is just a proof of concept, since it circumvents most of the Scrapy components (middlewares, dupefilter, etc). -There are some 3rd party projects which provide better integration: +The following is a list of 3rd party projects which provide better integration: * https://github.com/elacuesta/scrapy-pyppeteer * https://github.com/lopuhin/scrapy-pyppeteer From 494e0ad8ffc34e7d8079db6dd24fdc8265e81800 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com> Date: Wed, 28 Jul 2021 14:29:50 -0300 Subject: [PATCH 4/6] Update docs/topics/dynamic-content.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adrián Chaves --- docs/topics/dynamic-content.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index 56c8b6ae9..9706f43fe 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -272,10 +272,10 @@ The following is a simple snippet to illustrate its usage within Scrapy:: For this example to work, Scrapy needs to be running on top of the :ref:`asyncio reactor `. -Keep in mind that this is just a proof of concept, since it circumvents -most of the Scrapy components (middlewares, dupefilter, etc). - -The following is a list of 3rd party projects which provide better integration: +Using pypeteer_ directly circumvents most of the +Scrapy components (middlewares, dupefilter, etc). Use +one of the following Scrapy plugins for better integration +with Scrapy: * https://github.com/elacuesta/scrapy-pyppeteer * https://github.com/lopuhin/scrapy-pyppeteer From 0e3d50dd186666362ab8358c9f89036917242c81 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com> Date: Wed, 28 Jul 2021 14:30:16 -0300 Subject: [PATCH 5/6] Update docs/topics/dynamic-content.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adrián Chaves --- docs/topics/dynamic-content.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index 9706f43fe..e918bc006 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -252,6 +252,7 @@ Since version 2.0, it is possible to integrate libraries that use the ``async/await`` syntax. One such library is `pyppeteer`_ (an unnoficial Python port of `puppeteer`_), which uses headless Chrome to download and render pages. + The following is a simple snippet to illustrate its usage within Scrapy:: import pyppeteer From 4b62ac6c3ace093d16cf97e737689ac7942d52f9 Mon Sep 17 00:00:00 2001 From: Eugenio Lacuesta Date: Wed, 28 Jul 2021 15:00:24 -0300 Subject: [PATCH 6/6] Update headless browser docs to mention playwright --- docs/topics/dynamic-content.rst | 55 ++++++++++++++------------------- 1 file changed, 23 insertions(+), 32 deletions(-) diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index f96be0bbc..ea5d06210 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -246,55 +246,46 @@ Using a headless browser ======================== A `headless browser`_ is a special web browser that provides an API for -automation. +automation. By installing the :ref:`asyncio reactor `, +it is possible to integrate ``asyncio``-based libraries which handle headless browsers. -Since version 2.0, it is possible to integrate libraries that use the -``async/await`` syntax. One such library is `pyppeteer`_ (an unnoficial -Python port of `puppeteer`_), which uses headless Chrome to download and -render pages. +One such library is `playwright-python`_ (an official Python port of `playwright`_). +The following is a simple snippet to illustrate its usage within a Scrapy spider:: -The following is a simple snippet to illustrate its usage within Scrapy:: - - import pyppeteer import scrapy + from playwright.async_api import async_playwright - class PyppeteerSpider(scrapy.Spider): - name = "pyppeteer" - start_urls = ["data:,"] # avoid making an actual upstream request + class PlaywrightSpider(scrapy.Spider): + name = "playwright" + start_urls = ["data:,"] # avoid using the default Scrapy downloader async def parse(self, response): - browser = await pyppeteer.launch() - page = await browser.newPage() - await page.goto("https:/example.org") - title = await page.title() - await page.close() - yield {"title": title} + async with async_playwright() as pw: + browser = await pw.chromium.launch() + page = await browser.new_page() + await page.goto("https:/example.org") + title = await page.title() + return {"title": title} -For this example to work, Scrapy needs to be running on top of the -:ref:`asyncio reactor `. - -Using pypeteer_ directly circumvents most of the -Scrapy components (middlewares, dupefilter, etc). Use -one of the following Scrapy plugins for better integration -with Scrapy: - -* https://github.com/elacuesta/scrapy-pyppeteer -* https://github.com/lopuhin/scrapy-pyppeteer -* https://github.com/clemfromspace/scrapy-puppeteer +However, using `playwright-python`_ directly as in the above example +circumvents most of the Scrapy components (middlewares, dupefilter, etc). +We recommend using `scrapy-playwright`_ for a better integration. .. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29 -.. _chompjs: https://github.com/Nykakin/chompjs .. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets +.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript +.. _Splash: https://github.com/scrapinghub/splash +.. _chompjs: https://github.com/Nykakin/chompjs .. _curl: https://curl.haxx.se/ .. _headless browser: https://en.wikipedia.org/wiki/Headless_browser -.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript .. _js2xml: https://github.com/scrapinghub/js2xml -.. _puppeteer: https://pptr.dev/ +.. _playwright-python: https://github.com/microsoft/playwright-python +.. _playwright: https://github.com/microsoft/playwright .. _pyppeteer: https://pyppeteer.github.io/pyppeteer/ .. _pytesseract: https://github.com/madmaze/pytesseract +.. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright .. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash -.. _Splash: https://github.com/scrapinghub/splash .. _tabula-py: https://github.com/chezou/tabula-py .. _wget: https://www.gnu.org/software/wget/ .. _wgrep: https://github.com/stav/wgrep