From 495d3226912bb84651b5c86183a4adad58c509cc Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Fri, 26 Aug 2016 20:16:22 +0500 Subject: [PATCH 001/180] DOC move Data Flow below the picture; add links to components --- docs/topics/architecture.rst | 98 +++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 35 deletions(-) diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rst index ba0e2c61c..39e54ee99 100644 --- a/docs/topics/architecture.rst +++ b/docs/topics/architecture.rst @@ -16,20 +16,71 @@ components and an outline of the data flow that takes place inside the system below with links for more detailed information about them. The data flow is also described below. +.. _data-flow: + +Data flow +========= + .. image:: _images/scrapy_architecture_02.png :width: 700 :height: 470 :alt: Scrapy architecture +The data flow in Scrapy is controlled by the execution engine, and goes like +this: + +1. The :ref:`Engine ` gets the first URLs to crawl from the + :ref:`Spider `. + +2. The :ref:`Engine ` schedules the URLs in the + :ref:`Scheduler ` as Requests and asks for the + next URLs to crawl. + +3. The :ref:`Scheduler ` returns the next URLs to crawl + to the :ref:`Engine `. + +4. The :ref:`Engine ` sends the URLs to the + :ref:`Downloader `, passing through the + :ref:`Downloader Middleware ` + (request direction). + +5. Once the page finishes downloading the + :ref:`Downloader ` generates a Response (with + that page) and sends it to the Engine, passing through the + :ref:`Downloader Middleware ` + (response direction). + +6. The :ref:`Engine ` receives the Response from the + :ref:`Downloader ` and sends it to the + :ref:`Spider ` for processing, passing + through the :ref:`Spider Middleware ` + (input direction). + +7. The :ref:`Spider ` processes the Response and returns + scraped items and new Requests (to follow) to the + :ref:`Engine `, passing through the + :ref:`Spider Middleware ` (output direction). + +8. The :ref:`Engine ` sends processed items to + :ref:`Item Pipelines ` and processed Requests to + the :ref:`Scheduler `. + +9. The process repeats (from step 1) until there are no more requests from the + :ref:`Scheduler `. + Components ========== +.. _component-engine: + Scrapy Engine ------------- The engine is responsible for controlling the data flow between all components -of the system, and triggering events when certain actions occur. See the Data -Flow section below for more details. +of the system, and triggering events when certain actions occur. See the +:ref:`Data Flow ` section above for more details. + +.. _component-scheduler: Scheduler --------- @@ -37,12 +88,16 @@ Scheduler The Scheduler receives requests from the engine and enqueues them for feeding them later (also to the engine) when the engine requests them. +.. _component-downloader: + Downloader ---------- The Downloader is responsible for fetching web pages and feeding them to the engine which, in turn, feeds them to the spiders. +.. _component-spiders: + Spiders ------- @@ -50,6 +105,8 @@ Spiders are custom classes written by Scrapy users to parse responses and extract items (aka scraped items) from them or additional URLs (requests) to follow. For more information see :ref:`topics-spiders`. +.. _component-pipelines: + Item Pipeline ------------- @@ -58,6 +115,8 @@ extracted (or scraped) by the spiders. Typical tasks include cleansing, validation and persistence (like storing the item in a database). For more information see :ref:`topics-item-pipeline`. +.. _component-downloader-middleware: + Downloader middlewares ---------------------- @@ -76,6 +135,8 @@ Use a Downloader middleware if you need to do one of the following: For more information see :ref:`topics-downloader-middleware`. +.. _component-spider-middleware: + Spider middlewares ------------------ @@ -93,39 +154,6 @@ Use a Spider middleware if you need to For more information see :ref:`topics-spider-middleware`. -Data flow -========= - -The data flow in Scrapy is controlled by the execution engine, and goes like -this: - -1. The Engine gets the first URLs to crawl from the Spider. - -2. The Engine schedules the URLs in the Scheduler as Requests and asks for the - next URLs to crawl. - -3. The Scheduler returns the next URLs to crawl to the Engine. - -4. The Engine sends the URLs to the Downloader, passing through the - Downloader Middleware (request direction). - -5. Once the page finishes downloading the Downloader generates a Response (with - that page) and sends it to the Engine, passing through the Downloader - Middleware (response direction). - -6. The Engine receives the Response from the Downloader and sends it to the - Spider for processing, passing through the Spider Middleware (input direction). - -7. The Spider processes the Response and returns scraped items and new Requests - (to follow) to the Engine, passing through the Spider Middleware - (output direction). - -8. The Engine sends processed items to Item Pipelines and processed Requests to - the Scheduler. - -9. The process repeats (from step 1) until there are no more requests from the - Scheduler. - Event-driven networking ======================= From 22e870e955995bf789290504292f123f760f5f1b Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Thu, 1 Sep 2016 10:19:49 +0200 Subject: [PATCH 002/180] Add Debian Jessie test env --- tox.ini | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tox.ini b/tox.ini index f6de64b27..812302b4c 100644 --- a/tox.ini +++ b/tox.ini @@ -33,6 +33,20 @@ deps = zope.interface==3.6.1 -rtests/requirements.txt +[testenv:jessie] +# https://packages.debian.org/en/jessie/python/ +# https://packages.debian.org/en/jessie/zope/ +basepython = python2.7 +deps = + pyOpenSSL==0.14 + lxml==3.4.0 + Twisted==14.0.2 + boto==2.34.0 + Pillow==2.6.1 + cssselect==0.9.1 + zope.interface==4.1.1 + -rtests/requirements.txt + [testenv:trunk] basepython = python2.7 commands = From 2b2bfcea88a3b5c98a47adbd5c8d3979e6a5626d Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Thu, 1 Sep 2016 10:20:49 +0200 Subject: [PATCH 003/180] Add "jessie" build to Travis-CI config --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index c58ab39a5..59657b82e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -9,6 +9,7 @@ branches: env: - TOXENV=py27 - TOXENV=precise + - TOXENV=jessie - TOXENV=py33 - TOXENV=py35 - TOXENV=docs From 58cd7bf895321c39493cfd77feb4c07b7e614259 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Thu, 1 Sep 2016 11:17:53 +0200 Subject: [PATCH 004/180] Remove "precise" test env from Travis-CI config --- .travis.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 59657b82e..506f3779b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -8,7 +8,6 @@ branches: - /^\d\.\d+\.\d+(rc\d+|dev\d+)?$/ env: - TOXENV=py27 - - TOXENV=precise - TOXENV=jessie - TOXENV=py33 - TOXENV=py35 From b188f61b95d92390193bacfecda141cbf89197c9 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Thu, 1 Sep 2016 17:38:38 +0200 Subject: [PATCH 005/180] Update release notes for upcoming 1.2.0 version --- docs/news.rst | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index 5395db8e3..e514d9a5a 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,49 @@ Release notes ============= +1.2.0 (YYYY-MM-DD) +------------------ + +New Features +~~~~~~~~~~~~ + +- New :setting:`FEED_EXPORT_ENCODING` setting to customize the encoding + used when writing items to a file. This is useful for those wanting + something else than UTF-8 for XML or CSV output (:issue:`2034`). +- ``startproject`` command now supports an optional destination directory + to override the default one based on the project name (:issue:`2005`). +- New :setting:`SCHEDULER_DEBUG` setting to log requests serialization + failures (:issue:`1610`). +- JSON encoder now supports serialization of ``set`` instances (:issue:`2058`). + +Bug fixes +~~~~~~~~~ + +- DefaultRequestHeaders middleware now runs before UserAgent middleware + (:issue:`2088`). **Warning: this is technically backwards incompatible**, + though we consider this a bug fix. +- ``Selector`` does not allow passing both ``response`` and ``text`` anymore + (:issue:`2153`). +- Fixed logging of wrong callback name with ``scrapy parse`` (:issue:`2169`). +- Fix for an odd gzip decompression bug (:issue:`1606`). + +Refactoring +~~~~~~~~~~~ + +- ``canonicalize_url`` has been moved to `w3lib.url`_ (:issue:`2168`). + +.. _w3lib.url: http://w3lib.readthedocs.io/en/latest/w3lib.html#w3lib.url.canonicalize_url + +Documentation +~~~~~~~~~~~~~ + +- Grammar fixes: :issue:`2128`, :issue:`1566`. +- Download stats badge removed from README (:issue:`2160`). +- New scrapy :ref:`architecture diagram ` (:issue:`2165`) +- Updated ``Response`` parameters documentation (:issue:`2197`). +- Reworded misleading :setting:`RANDOMIZE_DOWNLOAD_DELAY` description (:issue:`2197`). + + 1.1.2 (2016-08-18) ------------------ From 743a0aa422ae2f515cbea69153e60a02c39da98d Mon Sep 17 00:00:00 2001 From: Joakim Uddholm Date: Thu, 8 Sep 2016 21:52:14 +0200 Subject: [PATCH 006/180] Two fixes for when using the parse command and the '-r' flag (rules). 1. Use default "parse" as callback when the matching rule has no callback. 2. Log error and return when no rule matches the parsed url. --- scrapy/commands/parse.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/scrapy/commands/parse.py b/scrapy/commands/parse.py index 6a8978415..5264982b6 100644 --- a/scrapy/commands/parse.py +++ b/scrapy/commands/parse.py @@ -121,8 +121,8 @@ class Command(ScrapyCommand): def get_callback_from_rules(self, spider, response): if getattr(spider, 'rules', None): for rule in spider.rules: - if rule.link_extractor.matches(response.url) and rule.callback: - return rule.callback + if rule.link_extractor.matches(response.url): + return rule.callback or "parse" else: logger.error('No CrawlSpider rules found in spider %(spider)r, ' 'please specify a callback to use for parsing', @@ -166,6 +166,11 @@ class Command(ScrapyCommand): if not cb: if opts.rules and self.first_response == response: cb = self.get_callback_from_rules(spider, response) + + if not cb: + logger.error('Cannot find a rule that matches %(url)r in spider: %(spider)s', + {'url': response.url, 'spider': spider.name}) + return else: cb = 'parse' From 80260824c65512edc3f60aa7eb8fead9818b1a90 Mon Sep 17 00:00:00 2001 From: Andrew Hlynskyi Date: Mon, 12 Sep 2016 00:43:58 +0300 Subject: [PATCH 007/180] Fix completion in `scrapy shell` for new imports --- scrapy/utils/console.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scrapy/utils/console.py b/scrapy/utils/console.py index a712df30e..567fd51bc 100644 --- a/scrapy/utils/console.py +++ b/scrapy/utils/console.py @@ -13,7 +13,9 @@ def _embed_ipython_shell(namespace={}, banner=''): @wraps(_embed_ipython_shell) def wrapper(namespace=namespace, banner=''): config = load_default_config() - shell = InteractiveShellEmbed( + # Always use .instace() to ensure _instance propagation to all parents + # this is needed for completion works well for new imports + shell = InteractiveShellEmbed.instance( banner1=banner, user_ns=namespace, config=config) shell() return wrapper From fbb555929977f91eabfa5dc28f3ae3d68972371a Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Mon, 12 Sep 2016 13:35:14 +0200 Subject: [PATCH 008/180] Add tests for crawl command non-default cases --- tests/test_commands.py | 82 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/tests/test_commands.py b/tests/test_commands.py index d25045cb9..d13024922 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -257,6 +257,9 @@ class ParseCommandTest(ProcessTest, SiteTest, CommandTest): with open(fname, 'w') as f: f.write(""" import scrapy +from scrapy.linkextractors import LinkExtractor +from scrapy.spiders import CrawlSpider, Rule + class MySpider(scrapy.Spider): name = '{0}' @@ -265,6 +268,33 @@ class MySpider(scrapy.Spider): if getattr(self, 'test_arg', None): self.logger.debug('It Works!') return [scrapy.Item(), dict(foo='bar')] + + +class MyGoodCrawlSpider(CrawlSpider): + name = 'goodcrawl{0}' + + rules = ( + Rule(LinkExtractor(allow=r'/html'), callback='parse_item', follow=True), + Rule(LinkExtractor(allow=r'/text'), follow=True), + ) + + def parse_item(self, response): + return [scrapy.Item(), dict(foo='bar')] + + def parse(self, response): + return [scrapy.Item(), dict(nomatch='default')] + + +class MyBadCrawlSpider(CrawlSpider): + '''Spider which doesn't define a parse_item callback while using it in a rule.''' + name = 'badcrawl{0}' + + rules = ( + Rule(LinkExtractor(allow=r'/html'), callback='parse_item', follow=True), + ) + + def parse(self, response): + return [scrapy.Item(), dict(foo='bar')] """.format(self.spider_name)) fname = abspath(join(self.proj_mod_path, 'pipelines.py')) @@ -309,6 +339,58 @@ ITEM_PIPELINES = {'%s.pipelines.MyPipeline': 1} ) self.assertIn("""[{}, {'foo': 'bar'}]""", to_native_str(out)) + @defer.inlineCallbacks + def test_parse_items_no_callback_passed(self): + status, out, stderr = yield self.execute( + ['--spider', self.spider_name, self.url('/html')] + ) + self.assertIn("""[{}, {'foo': 'bar'}]""", to_native_str(out)) + + @defer.inlineCallbacks + def test_wrong_callback_passed(self): + status, out, stderr = yield self.execute( + ['--spider', self.spider_name, '-c', 'dummy', self.url('/html')] + ) + self.assertRegexpMatches(to_native_str(out), """# Scraped Items -+\n\[\]""") + + @defer.inlineCallbacks + def test_crawlspider_matching_rule_callback_set(self): + """If a rule matches the URL, use it's defined callback.""" + status, out, stderr = yield self.execute( + ['--spider', 'goodcrawl'+self.spider_name, '-r', self.url('/html')] + ) + self.assertIn("""[{}, {'foo': 'bar'}]""", to_native_str(out)) + + @defer.inlineCallbacks + def test_crawlspider_matching_rule_default_callback(self): + """If a rule match but it has no callback set, use the 'parse' callback.""" + status, out, stderr = yield self.execute( + ['--spider', 'goodcrawl'+self.spider_name, '-r', self.url('/text')] + ) + self.assertIn("""[{}, {'nomatch': 'default'}]""", to_native_str(out)) + + @defer.inlineCallbacks + def test_spider_with_no_rules_attribute(self): + """Using -r with a spider with no rule should not produce items.""" + status, out, stderr = yield self.execute( + ['--spider', self.spider_name, '-r', self.url('/html')] + ) + self.assertRegexpMatches(to_native_str(out), """# Scraped Items -+\n\[\]""") + + @defer.inlineCallbacks + def test_crawlspider_missing_callback(self): + status, out, stderr = yield self.execute( + ['--spider', 'badcrawl'+self.spider_name, '-r', self.url('/html')] + ) + self.assertRegexpMatches(to_native_str(out), """# Scraped Items -+\n\[\]""") + + @defer.inlineCallbacks + def test_crawlspider_no_matching_rule(self): + """The requested URL has no matching rule, so no items should be scraped""" + status, out, stderr = yield self.execute( + ['--spider', 'badcrawl'+self.spider_name, '-r', self.url('/enc-gb18030')] + ) + self.assertRegexpMatches(to_native_str(out), """# Scraped Items -+\n\[\]""") class BenchCommandTest(CommandTest): From 10f8c52f5d3023a9f60fb440b7b1fca3274c29e2 Mon Sep 17 00:00:00 2001 From: Valdir Stumm Junior Date: Fri, 9 Sep 2016 10:35:28 -0300 Subject: [PATCH 009/180] changed tutorial examples from dmoz to quotes.toscrape.com --- docs/intro/tutorial.rst | 301 +++++++++++++++++++--------------------- 1 file changed, 141 insertions(+), 160 deletions(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index 6ecd637c3..262d6b3a4 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -7,7 +7,7 @@ Scrapy Tutorial In this tutorial, we'll assume that Scrapy is already installed on your system. If that's not the case, see :ref:`intro-install`. -We are going to use `Open directory project (dmoz) `_ as +We are going to use `quotes.toscrape.com `_ as our example domain to scrape. This tutorial will walk you through these tasks: @@ -16,8 +16,7 @@ This tutorial will walk you through these tasks: 2. Defining the Items you will extract 3. Writing a :ref:`spider ` to crawl a site and extract :ref:`Items ` -4. Writing an :ref:`Item Pipeline ` to store the - extracted Items +4. Exporting the scraped data using command line Scrapy is written in Python_. If you're new to the language you might want to start by getting an idea of what the language is like, to get the most out of @@ -54,7 +53,6 @@ This will create a ``tutorial`` directory with the following contents:: spiders/ # a directory where you'll later put your spiders __init__.py - ... Defining our Item @@ -72,16 +70,15 @@ its attributes as :class:`scrapy.Field ` objects, much like i easy task). We begin by modeling the item that we will use to hold the site's data obtained -from dmoz.org. As we want to capture the name, url and description of the -sites, we define fields for each of these three attributes. To do that, we edit +from quotes.toscrape.com. As we want to capture the text and author from each of +the quotes listed there, we define fields for each of these three attributes. To do that, we edit ``items.py``, found in the ``tutorial`` directory. Our Item class looks like this:: import scrapy - class DmozItem(scrapy.Item): - title = scrapy.Field() - link = scrapy.Field() - desc = scrapy.Field() + class QuoteItem(scrapy.Item): + text = scrapy.Field() + author = scrapy.Field() This may seem complicated at first, but defining an item class allows you to use other handy components and helpers within Scrapy. @@ -99,10 +96,11 @@ To create a Spider, you must subclass :class:`scrapy.Spider ` and define some attributes: * :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be - unique, that is, you can't set the same name for different Spiders. + unique within a project, that is, you can't set the same name for different + Spiders. * :attr:`~scrapy.spiders.Spider.start_urls`: a list of URLs where the - Spider will begin to crawl from. The first pages downloaded will be those + Spider will begin to crawl from. The first pages downloaded will be those listed here. The subsequent URLs will be generated successively from data contained in the start URLs. @@ -119,20 +117,20 @@ To create a Spider, you must subclass :class:`scrapy.Spider objects) and more URLs to follow (as :class:`~scrapy.http.Request` objects). This is the code for our first Spider; save it in a file named -``dmoz_spider.py`` under the ``tutorial/spiders`` directory:: +``quotes_spider.py`` under the ``tutorial/spiders`` directory:: import scrapy - class DmozSpider(scrapy.Spider): - name = "dmoz" - allowed_domains = ["dmoz.org"] + + class QuotesSpider(scrapy.Spider): + name = "quotes" start_urls = [ - "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", - "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" + 'http://quotes.toscrape.com/page/1/', + 'http://quotes.toscrape.com/page/2/', ] def parse(self, response): - filename = response.url.split("/")[-2] + '.html' + filename = 'quotes-' + response.url.split("/")[-2] + '.html' with open(filename, 'wb') as f: f.write(response.body) @@ -141,24 +139,25 @@ Crawling To put our spider to work, go to the project's top level directory and run:: - scrapy crawl dmoz + scrapy crawl quotes -This command runs the spider with name ``dmoz`` that we've just added, that -will send some requests for the ``dmoz.org`` domain. You will get an output +This command runs the spider with name ``quotes`` that we've just added, that +will send some requests for the ``quotes.toscrape.com`` domain. You will get an output similar to this:: - 2014-01-23 18:13:07-0400 [scrapy] INFO: Scrapy started (bot: tutorial) - 2014-01-23 18:13:07-0400 [scrapy] INFO: Optional features available: ... - 2014-01-23 18:13:07-0400 [scrapy] INFO: Overridden settings: {} - 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled extensions: ... - 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ... - 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled spider middlewares: ... - 2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled item pipelines: ... - 2014-01-23 18:13:07-0400 [scrapy] INFO: Spider opened - 2014-01-23 18:13:08-0400 [scrapy] DEBUG: Crawled (200) (referer: None) - 2014-01-23 18:13:09-0400 [scrapy] DEBUG: Crawled (200) (referer: None) - 2014-01-23 18:13:09-0400 [scrapy] INFO: Closing spider (finished) + 2016-09-01 16:51:27 [scrapy] INFO: Scrapy started (bot: tutorial) + 2016-09-01 16:51:27 [scrapy] INFO: Overridden settings: {...} + 2016-09-01 16:51:27 [scrapy] INFO: Enabled extensions: ... + 2016-09-01 16:51:27 [scrapy] INFO: Enabled downloader middlewares: ... + 2016-09-01 16:51:27 [scrapy] INFO: Enabled spider middlewares: ... + 2016-09-01 16:51:27 [scrapy] INFO: Enabled item pipelines: ... + 2016-09-01 16:51:27 [scrapy] INFO: Spider opened + 2016-09-01 16:51:27 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) + 2016-09-01 16:51:28 [scrapy] DEBUG: Crawled (404) (referer: None) + 2016-09-01 16:51:28 [scrapy] DEBUG: Crawled (200) (referer: None) + 2016-09-01 16:51:29 [scrapy] DEBUG: Crawled (200) (referer: None) + 2016-09-01 16:51:29 [scrapy] INFO: Closing spider (finished) .. note:: At the end you can see a log line for each URL defined in ``start_urls``. @@ -166,7 +165,7 @@ similar to this:: shown at the end of the log line, where it says ``(referer: None)``. Now, check the files in the current directory. You should notice two new files -have been created: *Books.html* and *Resources.html*, with the content for the respective +have been created: *quotes-1.html* and *quotes-2.html*, with the content for the respective URLs, as our ``parse`` method instructs. What just happened under the hood? @@ -197,15 +196,16 @@ mechanisms see the :ref:`Selectors documentation `. Here are some examples of XPath expressions and their meanings: * ``/html/head/title``: selects the ```` element, inside the ``<head>`` - element of an HTML document + element of an HTML document. Equivalent CSS selector: ``html > head > title``. * ``/html/head/title/text()``: selects the text inside the aforementioned - ``<title>`` element. + ``<title>`` element. Equivalent CSS selector: ``html > head > title ::text``. -* ``//td``: selects all the ``<td>`` elements +* ``//td``: selects all the ``<td>`` elements from the whole document. + Equivalent CSS selector: ``td``. * ``//div[@class="mine"]``: selects all ``div`` elements which contain an - attribute ``class="mine"`` + attribute ``class="mine"``. Equivalent CSS selector: ``div.mine``. These are just a couple of simple examples of what you can do with XPath, but XPath expressions are indeed much more powerful. To learn more about XPath, we @@ -220,7 +220,7 @@ to think in XPath" <http://plasmasturm.org/log/xpath101/>`_. Because of this, we encourage you to learn about XPath even if you already know how to construct CSS selectors. -For working with CSS and XPath expressions, Scrapy provides +For working with CSS and XPath expressions, Scrapy provides the :class:`~scrapy.selector.Selector` class and convenient shortcuts to avoid instantiating selectors yourself every time you need to select something from a response. @@ -255,7 +255,7 @@ installed on your system. To start a shell, you must go to the project's top level directory and run:: - scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/" + scrapy shell "http://quotes.toscrape.com" .. note:: @@ -267,20 +267,20 @@ This is what the shell looks like:: [ ... Scrapy log here ... ] - 2014-01-23 17:11:42-0400 [scrapy] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None) + 2016-09-01 18:14:39 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com> (referer: None) [s] Available Scrapy objects: - [s] crawler <scrapy.crawler.Crawler object at 0x3636b50> + [s] crawler <scrapy.crawler.Crawler object at 0x109001c90> [s] item {} - [s] request <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> - [s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> - [s] settings <scrapy.settings.Settings object at 0x3fadc50> - [s] spider <Spider 'default' at 0x3cebf50> + [s] request <GET http://quotes.toscrape.com> + [s] response <200 http://quotes.toscrape.com> + [s] settings <scrapy.settings.Settings object at 0x109001610> + [s] spider <DefaultSpider 'default' at 0x1092808d0> [s] Useful shortcuts: [s] shelp() Shell help (print this help) [s] fetch(req_or_url) Fetch request (or URL) and update local objects [s] view(response) View response in a browser - - In [1]: + + >>> After the shell loads, you will have the response fetched in a local ``response`` variable, so if you type ``response.body`` you will see the body @@ -297,19 +297,19 @@ or ``response.css()`` which map directly to ``response.selector.xpath()`` and So let's try it:: In [1]: response.xpath('//title') - Out[1]: [<Selector xpath='//title' data=u'<title>Open Directory - Computers: Progr'>] - + Out[1]: [<Selector xpath='//title' data=u'<title>Quotes to Scrape'>] + In [2]: response.xpath('//title').extract() - Out[2]: [u'Open Directory - Computers: Programming: Languages: Python: Books'] - + Out[2]: [u'Quotes to Scrape'] + In [3]: response.xpath('//title/text()') - Out[3]: [] - + Out[3]: [] + In [4]: response.xpath('//title/text()').extract() - Out[4]: [u'Open Directory - Computers: Programming: Languages: Python: Books'] - - In [5]: response.xpath('//title/text()').re('(\w+):') - Out[5]: [u'Computers', u'Programming', u'Languages', u'Python'] + Out[4]: [u'Quotes to Scrape'] + + In [11]: response.xpath('//title/text()').re('(\w+)') + Out[11]: [u'Quotes', u'to', u'Scrape'] Extracting the data ^^^^^^^^^^^^^^^^^^^ @@ -322,35 +322,42 @@ there could become a very tedious task. To make it easier, you can use Firefox Developer Tools or some Firefox extensions like Firebug. For more information see :ref:`topics-firebug` and :ref:`topics-firefox`. -After inspecting the page source, you'll find that the web site's information -is inside a ``