From 7a92dae4c8638e275a6e85d667b21e771ebb6e91 Mon Sep 17 00:00:00 2001 From: Julia Medina Date: Thu, 19 Mar 2015 15:02:54 -0300 Subject: [PATCH] Change Scrapy log output through docs --- docs/intro/tutorial.rst | 17 +++++++-------- docs/topics/benchmarking.rst | 30 +++++++++++++-------------- docs/topics/downloader-middleware.rst | 8 +++---- docs/topics/shell.rst | 6 +++--- 4 files changed, 30 insertions(+), 31 deletions(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index ab2302154..479b17f99 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -154,15 +154,14 @@ similar to this:: 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled spider middlewares: ... 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled item pipelines: ... - 2014-01-23 18:13:07-0400 [dmoz] INFO: Spider opened - 2014-01-23 18:13:08-0400 [dmoz] DEBUG: Crawled (200) (referer: None) - 2014-01-23 18:13:09-0400 [dmoz] DEBUG: Crawled (200) (referer: None) - 2014-01-23 18:13:09-0400 [dmoz] INFO: Closing spider (finished) + 2014-01-23 18:13:07-0400 [scrapy] INFO: Spider opened + 2014-01-23 18:13:08-0400 [scrapy] DEBUG: Crawled (200) (referer: None) + 2014-01-23 18:13:09-0400 [scrapy] DEBUG: Crawled (200) (referer: None) + 2014-01-23 18:13:09-0400 [scrapy] INFO: Closing spider (finished) .. note:: - The lines containing ``[dmoz]`` correspond to the log messages for - our spider. You can see a log line for each URL defined in ``start_urls``. + At the end you can see a log line for each URL defined in ``start_urls``. Because these URLs are the starting ones, they have no referrers, which is shown at the end of the log line, where it says ``(referer: None)``. @@ -268,7 +267,7 @@ This is what the shell looks like:: [ ... Scrapy log here ... ] - 2014-01-23 17:11:42-0400 [default] DEBUG: Crawled (200) (referer: None) + 2014-01-23 17:11:42-0400 [scrapy] DEBUG: Crawled (200) (referer: None) [s] Available Scrapy objects: [s] crawler [s] item {} @@ -424,11 +423,11 @@ Spider would be like this:: Now crawling dmoz.org yields ``DmozItem`` objects:: - [dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> + [scrapy] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> {'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n], 'link': [u'http://gnosis.cx/TPiP/'], 'title': [u'Text Processing in Python']} - [dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> + [scrapy] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> {'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'title': [u'XML Processing with Python']} diff --git a/docs/topics/benchmarking.rst b/docs/topics/benchmarking.rst index 9ae6dd2cb..632190067 100644 --- a/docs/topics/benchmarking.rst +++ b/docs/topics/benchmarking.rst @@ -19,20 +19,20 @@ To run it use:: You should see an output like this:: 2013-05-16 13:08:46-0300 [scrapy] INFO: Scrapy 0.17.0 started (bot: scrapybot) - 2013-05-16 13:08:47-0300 [follow] INFO: Spider opened - 2013-05-16 13:08:47-0300 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:48-0300 [follow] INFO: Crawled 74 pages (at 4440 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:49-0300 [follow] INFO: Crawled 143 pages (at 4140 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:50-0300 [follow] INFO: Crawled 210 pages (at 4020 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:51-0300 [follow] INFO: Crawled 274 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:52-0300 [follow] INFO: Crawled 343 pages (at 4140 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:53-0300 [follow] INFO: Crawled 410 pages (at 4020 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:54-0300 [follow] INFO: Crawled 474 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:55-0300 [follow] INFO: Crawled 538 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:56-0300 [follow] INFO: Crawled 602 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:57-0300 [follow] INFO: Closing spider (closespider_timeout) - 2013-05-16 13:08:57-0300 [follow] INFO: Crawled 666 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) - 2013-05-16 13:08:57-0300 [follow] INFO: Dumping Scrapy stats: + 2013-05-16 13:08:47-0300 [scrapy] INFO: Spider opened + 2013-05-16 13:08:47-0300 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:48-0300 [scrapy] INFO: Crawled 74 pages (at 4440 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:49-0300 [scrapy] INFO: Crawled 143 pages (at 4140 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:50-0300 [scrapy] INFO: Crawled 210 pages (at 4020 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:51-0300 [scrapy] INFO: Crawled 274 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:52-0300 [scrapy] INFO: Crawled 343 pages (at 4140 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:53-0300 [scrapy] INFO: Crawled 410 pages (at 4020 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:54-0300 [scrapy] INFO: Crawled 474 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:55-0300 [scrapy] INFO: Crawled 538 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:56-0300 [scrapy] INFO: Crawled 602 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:57-0300 [scrapy] INFO: Closing spider (closespider_timeout) + 2013-05-16 13:08:57-0300 [scrapy] INFO: Crawled 666 pages (at 3840 pages/min), scraped 0 items (at 0 items/min) + 2013-05-16 13:08:57-0300 [scrapy] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 231508, 'downloader/request_count': 682, 'downloader/request_method_count/GET': 682, @@ -49,7 +49,7 @@ You should see an output like this:: 'scheduler/enqueued': 12767, 'scheduler/enqueued/memory': 12767, 'start_time': datetime.datetime(2013, 5, 16, 16, 8, 47, 676539)} - 2013-05-16 13:08:57-0300 [follow] INFO: Spider closed (closespider_timeout) + 2013-05-16 13:08:57-0300 [scrapy] INFO: Spider closed (closespider_timeout) That tells you that Scrapy is able to crawl about 3900 pages per minute in the hardware where you run it. Note that this is a very simple spider intended to diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index df679daea..3d24b29ad 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -232,14 +232,14 @@ header) and all cookies received in responses (ie. ``Set-Cookie`` header). Here's an example of a log with :setting:`COOKIES_DEBUG` enabled:: - 2011-04-06 14:35:10-0300 [diningcity] INFO: Spider opened - 2011-04-06 14:35:10-0300 [diningcity] DEBUG: Sending cookies to: + 2011-04-06 14:35:10-0300 [scrapy] INFO: Spider opened + 2011-04-06 14:35:10-0300 [scrapy] DEBUG: Sending cookies to: Cookie: clientlanguage_nl=en_EN - 2011-04-06 14:35:14-0300 [diningcity] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html> + 2011-04-06 14:35:14-0300 [scrapy] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html> Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/ Set-Cookie: ip_isocode=US Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/ - 2011-04-06 14:49:50-0300 [diningcity] DEBUG: Crawled (200) (referer: None) + 2011-04-06 14:49:50-0300 [scrapy] DEBUG: Crawled (200) (referer: None) [...] diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index 8fce0cea1..3b875fec5 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -187,8 +187,8 @@ Here's an example of how you would call it from your spider:: When you run the spider, you will get something similar to this:: - 2014-01-23 17:48:31-0400 [myspider] DEBUG: Crawled (200) (referer: None) - 2014-01-23 17:48:31-0400 [myspider] DEBUG: Crawled (200) (referer: None) + 2014-01-23 17:48:31-0400 [scrapy] DEBUG: Crawled (200) (referer: None) + 2014-01-23 17:48:31-0400 [scrapy] DEBUG: Crawled (200) (referer: None) [s] Available Scrapy objects: [s] crawler ... @@ -211,7 +211,7 @@ Finally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume the crawling:: >>> ^D - 2014-01-23 17:50:03-0400 [myspider] DEBUG: Crawled (200) (referer: None) + 2014-01-23 17:50:03-0400 [scrapy] DEBUG: Crawled (200) (referer: None) ... Note that you can't use the ``fetch`` shortcut here since the Scrapy engine is