RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs: use https scheme for each quotes.toscrape.com url occurrence

This commit is contained in:
Péter Ferenc Gyarmati 2022-02-06 18:52:15 +01:00
parent 3533717488
commit 38d2a154ec
7 changed files with 43 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/intro/examples.rst
View File

@ -7,7 +7,7 @@ Examples
The best way to learn is with examples, and Scrapy is no exception. For this The best way to learn is with examples, and Scrapy is no exception. For this
reason, there is an example Scrapy project named quotesbot_, that you can use to reason, there is an example Scrapy project named quotesbot_, that you can use to
play and learn more about Scrapy. It contains two spiders for play and learn more about Scrapy. It contains two spiders for
http://quotes.toscrape.com, one using CSS selectors and another one using XPath https://quotes.toscrape.com, one using CSS selectors and another one using XPath
expressions. expressions.
The quotesbot_ project is available at: https://github.com/scrapy/quotesbot. The quotesbot_ project is available at: https://github.com/scrapy/quotesbot.

4
docs/intro/overview.rst
View File

@ -20,7 +20,7 @@ In order to show you what Scrapy brings to the table, we'll walk you through an
example of a Scrapy Spider using the simplest way to run a spider. example of a Scrapy Spider using the simplest way to run a spider.
Here's the code for a spider that scrapes famous quotes from website Here's the code for a spider that scrapes famous quotes from website
http://quotes.toscrape.com, following the pagination:: https://quotes.toscrape.com, following the pagination::
import scrapy import scrapy
@ -28,7 +28,7 @@ http://quotes.toscrape.com, following the pagination::
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = 'quotes' name = 'quotes'
start_urls = [ start_urls = [
'http://quotes.toscrape.com/tag/humor/', 'https://quotes.toscrape.com/tag/humor/',
] ]
def parse(self, response): def parse(self, response):

52
docs/intro/tutorial.rst
View File

@ -7,7 +7,7 @@ Scrapy Tutorial
In this tutorial, we'll assume that Scrapy is already installed on your system. In this tutorial, we'll assume that Scrapy is already installed on your system.
If that's not the case, see :ref:`intro-install`. If that's not the case, see :ref:`intro-install`.
We are going to scrape `quotes.toscrape.com <http://quotes.toscrape.com/>`_, a website We are going to scrape `quotes.toscrape.com <https://quotes.toscrape.com/>`_, a website
that lists quotes from famous authors. that lists quotes from famous authors.
This tutorial will walk you through these tasks: This tutorial will walk you through these tasks:
@ -93,8 +93,8 @@ This is the code for our first Spider. Save it in a file named
def start_requests(self): def start_requests(self):
urls = [ urls = [
'http://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/1/',
'http://quotes.toscrape.com/page/2/', 'https://quotes.toscrape.com/page/2/',
] ]
for url in urls: for url in urls:
yield scrapy.Request(url=url, callback=self.parse) yield scrapy.Request(url=url, callback=self.parse)
@ -143,9 +143,9 @@ similar to this::
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened 2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened
2016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None) 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://quotes.toscrape.com/robots.txt> (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None) 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/page/1/> (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None) 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/page/2/> (referer: None)
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html 2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html 2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished) 2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished)
@ -184,8 +184,8 @@ for your spider::
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'http://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/1/',
'http://quotes.toscrape.com/page/2/', 'https://quotes.toscrape.com/page/2/',
] ]
def parse(self, response): def parse(self, response):
@ -207,7 +207,7 @@ Extracting data
The best way to learn how to extract data with Scrapy is trying selectors The best way to learn how to extract data with Scrapy is trying selectors
using the :ref:`Scrapy shell <topics-shell>`. Run:: using the :ref:`Scrapy shell <topics-shell>`. Run::
scrapy shell 'http://quotes.toscrape.com/page/1/' scrapy shell 'https://quotes.toscrape.com/page/1/'
.. note:: .. note::
@ -217,18 +217,18 @@ using the :ref:`Scrapy shell <topics-shell>`. Run::
On Windows, use double quotes instead:: On Windows, use double quotes instead::
scrapy shell "http://quotes.toscrape.com/page/1/" scrapy shell "https://quotes.toscrape.com/page/1/"
You will see something like:: You will see something like::
[ ... Scrapy log here ... ] [ ... Scrapy log here ... ]
2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None) 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/page/1/> (referer: None)
[s] Available Scrapy objects: [s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc) [s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x7fa91d888c90> [s] crawler <scrapy.crawler.Crawler object at 0x7fa91d888c90>
[s] item {} [s] item {}
[s] request <GET http://quotes.toscrape.com/page/1/> [s] request <GET https://quotes.toscrape.com/page/1/>
[s] response <200 http://quotes.toscrape.com/page/1/> [s] response <200 https://quotes.toscrape.com/page/1/>
[s] settings <scrapy.settings.Settings object at 0x7fa91d888c10> [s] settings <scrapy.settings.Settings object at 0x7fa91d888c10>
[s] spider <DefaultSpider 'default' at 0x7fa91c8af990> [s] spider <DefaultSpider 'default' at 0x7fa91c8af990>
[s] Useful shortcuts: [s] Useful shortcuts:
@ -241,7 +241,7 @@ object:
.. invisible-code-block: python .. invisible-code-block: python
response = load_response('http://quotes.toscrape.com/page/1/', 'quotes1.html') response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html')
>>> response.css('title') >>> response.css('title')
[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>] [<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
@ -355,7 +355,7 @@ Extracting quotes and authors
Now that you know a bit about selection and extraction, let's complete our Now that you know a bit about selection and extraction, let's complete our
spider by writing the code to extract the quotes from the web page. spider by writing the code to extract the quotes from the web page.
Each quote in http://quotes.toscrape.com is represented by HTML elements that look Each quote in https://quotes.toscrape.com is represented by HTML elements that look
like this: like this:
.. code-block:: html .. code-block:: html
@ -379,7 +379,7 @@ like this:
Let's open up scrapy shell and play a bit to find out how to extract the data Let's open up scrapy shell and play a bit to find out how to extract the data
we want:: we want::
$ scrapy shell 'http://quotes.toscrape.com' $ scrapy shell 'https://quotes.toscrape.com'
We get a list of selectors for the quote HTML elements with: We get a list of selectors for the quote HTML elements with:
@ -444,8 +444,8 @@ in the callback, as you can see below::
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'http://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/1/',
'http://quotes.toscrape.com/page/2/', 'https://quotes.toscrape.com/page/2/',
] ]
def parse(self, response): def parse(self, response):
@ -458,9 +458,9 @@ in the callback, as you can see below::
If you run this spider, it will output the extracted data with the log:: If you run this spider, it will output the extracted data with the log::
2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/> 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>
{'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'} {'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'}
2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/> 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>
{'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"} {'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}
@ -505,7 +505,7 @@ Following links
=============== ===============
Let's say, instead of just scraping the stuff from the first two pages Let's say, instead of just scraping the stuff from the first two pages
from http://quotes.toscrape.com, you want quotes from all the pages in the website. from https://quotes.toscrape.com, you want quotes from all the pages in the website.
Now that you know how to extract data from pages, let's see how to follow links Now that you know how to extract data from pages, let's see how to follow links
from them. from them.
@ -549,7 +549,7 @@ page, extracting data from it::
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'http://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/1/',
] ]
def parse(self, response): def parse(self, response):
@ -600,7 +600,7 @@ As a shortcut for creating Request objects you can use
class QuotesSpider(scrapy.Spider): class QuotesSpider(scrapy.Spider):
name = "quotes" name = "quotes"
start_urls = [ start_urls = [
'http://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/1/',
] ]
def parse(self, response): def parse(self, response):
@ -654,7 +654,7 @@ this time for scraping author information::
class AuthorSpider(scrapy.Spider): class AuthorSpider(scrapy.Spider):
name = 'author' name = 'author'
start_urls = ['http://quotes.toscrape.com/'] start_urls = ['https://quotes.toscrape.com/']
def parse(self, response): def parse(self, response):
author_page_links = response.css('.author + a') author_page_links = response.css('.author + a')
@ -727,7 +727,7 @@ with a specific tag, building the URL based on the argument::
name = "quotes" name = "quotes"
def start_requests(self): def start_requests(self):
url = 'http://quotes.toscrape.com/' url = 'https://quotes.toscrape.com/'
tag = getattr(self, 'tag', None) tag = getattr(self, 'tag', None)
if tag is not None: if tag is not None:
url = url + 'tag/' + tag url = url + 'tag/' + tag
@ -747,7 +747,7 @@ with a specific tag, building the URL based on the argument::
If you pass the ``tag=humor`` argument to this spider, you'll notice that it If you pass the ``tag=humor`` argument to this spider, you'll notice that it
will only visit URLs from the ``humor`` tag, such as will only visit URLs from the ``humor`` tag, such as
``http://quotes.toscrape.com/tag/humor``. ``https://quotes.toscrape.com/tag/humor``.
You can :ref:`learn more about handling spider arguments here <spiderargs>`. You can :ref:`learn more about handling spider arguments here <spiderargs>`.

20
docs/topics/developer-tools.rst
View File

@ -81,18 +81,18 @@ clicking directly on the tag. If we expand the ``span`` tag with the ``class=
"text"`` we will see the quote-text we clicked on. The `Inspector` lets you "text"`` we will see the quote-text we clicked on. The `Inspector` lets you
copy XPaths to selected elements. Let's try it out. copy XPaths to selected elements. Let's try it out.
First open the Scrapy shell at http://quotes.toscrape.com/ in a terminal: First open the Scrapy shell at https://quotes.toscrape.com/ in a terminal:
.. code-block:: none .. code-block:: none
$ scrapy shell "http://quotes.toscrape.com/" $ scrapy shell "https://quotes.toscrape.com/"
Then, back to your web browser, right-click on the ``span`` tag, select Then, back to your web browser, right-click on the ``span`` tag, select
``Copy > XPath`` and paste it in the Scrapy shell like so: ``Copy > XPath`` and paste it in the Scrapy shell like so:
.. invisible-code-block: python .. invisible-code-block: python
response = load_response('http://quotes.toscrape.com/', 'quotes.html') response = load_response('https://quotes.toscrape.com/', 'quotes.html')
>>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall() >>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall()
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'] ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
@ -227,7 +227,7 @@ interests us is the one request called ``quotes?page=1`` with the
type ``json``. type ``json``.
If we click on this request, we see that the request URL is If we click on this request, we see that the request URL is
``http://quotes.toscrape.com/api/quotes?page=1`` and the response ``https://quotes.toscrape.com/api/quotes?page=1`` and the response
is a JSON-object that contains our quotes. We can also right-click is a JSON-object that contains our quotes. We can also right-click
on the request and open ``Open in new tab`` to get a better overview. on the request and open ``Open in new tab`` to get a better overview.
@ -247,7 +247,7 @@ also request each page to get every quote on the site::
name = 'quote' name = 'quote'
allowed_domains = ['quotes.toscrape.com'] allowed_domains = ['quotes.toscrape.com']
page = 1 page = 1
start_urls = ['http://quotes.toscrape.com/api/quotes?page=1'] start_urls = ['https://quotes.toscrape.com/api/quotes?page=1']
def parse(self, response): def parse(self, response):
data = json.loads(response.text) data = json.loads(response.text)
@ -255,7 +255,7 @@ also request each page to get every quote on the site::
yield {"quote": quote["text"]} yield {"quote": quote["text"]}
if data["has_next"]: if data["has_next"]:
self.page += 1 self.page += 1
url = f"http://quotes.toscrape.com/api/quotes?page={self.page}" url = f"https://quotes.toscrape.com/api/quotes?page={self.page}"
yield scrapy.Request(url=url, callback=self.parse) yield scrapy.Request(url=url, callback=self.parse)
This spider starts at the first page of the quotes-API. With each This spider starts at the first page of the quotes-API. With each
@ -280,7 +280,7 @@ request::
from scrapy import Request from scrapy import Request
request = Request.from_curl( request = Request.from_curl(
"curl 'http://quotes.toscrape.com/api/quotes?page=1' -H 'User-Agent: Mozil" "curl 'https://quotes.toscrape.com/api/quotes?page=1' -H 'User-Agent: Mozil"
"la/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0' -H 'Acce" "la/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0' -H 'Acce"
"pt: */*' -H 'Accept-Language: ca,en-US;q=0.7,en;q=0.3' --compressed -H 'X" "pt: */*' -H 'Accept-Language: ca,en-US;q=0.7,en;q=0.3' --compressed -H 'X"
"-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM" "-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM"
@ -304,8 +304,8 @@ daunting and pages can be very complex, but it (mostly) boils down
to identifying the correct request and replicating it in your spider. to identifying the correct request and replicating it in your spider.
.. _Developer Tools: https://en.wikipedia.org/wiki/Web_development_tools .. _Developer Tools: https://en.wikipedia.org/wiki/Web_development_tools
.. _quotes.toscrape.com: http://quotes.toscrape.com .. _quotes.toscrape.com: https://quotes.toscrape.com
.. _quotes.toscrape.com/scroll: http://quotes.toscrape.com/scroll .. _quotes.toscrape.com/scroll: https://quotes.toscrape.com/scroll
.. _quotes.toscrape.com/api/quotes?page=10: http://quotes.toscrape.com/api/quotes?page=10 .. _quotes.toscrape.com/api/quotes?page=10: https://quotes.toscrape.com/api/quotes?page=10
.. _has-class-extension: https://parsel.readthedocs.io/en/latest/usage.html#other-xpath-extensions .. _has-class-extension: https://parsel.readthedocs.io/en/latest/usage.html#other-xpath-extensions

2
docs/topics/logging.rst
View File

@ -218,7 +218,7 @@ For example, let's say you're scraping a website which returns many
HTTP 404 and 500 responses, and you want to hide all messages like this:: HTTP 404 and 500 responses, and you want to hide all messages like this::
2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring 2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring
response <500 http://quotes.toscrape.com/page/1-34/>: HTTP status code response <500 https://quotes.toscrape.com/page/1-34/>: HTTP status code
is not handled or not allowed is not handled or not allowed
The first thing to note is a logger name - it is in brackets: The first thing to note is a logger name - it is in brackets:

4
docs/topics/settings.rst
View File

@ -1597,7 +1597,7 @@ In order to use the reactor installed by Scrapy::
def start_requests(self): def start_requests(self):
reactor.callLater(self.timeout, self.stop) reactor.callLater(self.timeout, self.stop)
urls = ['http://quotes.toscrape.com/page/1'] urls = ['https://quotes.toscrape.com/page/1']
for url in urls: for url in urls:
yield scrapy.Request(url=url, callback=self.parse) yield scrapy.Request(url=url, callback=self.parse)
@ -1625,7 +1625,7 @@ which raises :exc:`Exception`, becomes::
from twisted.internet import reactor from twisted.internet import reactor
reactor.callLater(self.timeout, self.stop) reactor.callLater(self.timeout, self.stop)
urls = ['http://quotes.toscrape.com/page/1'] urls = ['https://quotes.toscrape.com/page/1']
for url in urls: for url in urls:
yield scrapy.Request(url=url, callback=self.parse) yield scrapy.Request(url=url, callback=self.parse)

2
docs/topics/signals.rst
View File

@ -60,7 +60,7 @@ Let's take an example::
class SignalSpider(scrapy.Spider): class SignalSpider(scrapy.Spider):
name = 'signals' name = 'signals'
start_urls = ['http://quotes.toscrape.com/page/1/'] start_urls = ['https://quotes.toscrape.com/page/1/']
@classmethod @classmethod
def from_crawler(cls, crawler, *args, **kwargs): def from_crawler(cls, crawler, *args, **kwargs):