diff --git a/README.rst b/README.rst index 27fab8e29..da63f2b93 100644 --- a/README.rst +++ b/README.rst @@ -7,7 +7,7 @@ Scrapy :alt: PyPI Version .. image:: https://img.shields.io/travis/scrapy/scrapy/master.svg - :target: http://travis-ci.org/scrapy/scrapy + :target: https://travis-ci.org/scrapy/scrapy :alt: Build Status .. image:: https://img.shields.io/badge/wheel-yes-brightgreen.svg @@ -15,7 +15,7 @@ Scrapy :alt: Wheel Status .. image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg - :target: http://codecov.io/github/scrapy/scrapy?branch=master + :target: https://codecov.io/github/scrapy/scrapy?branch=master :alt: Coverage report .. image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg diff --git a/artwork/README.rst b/artwork/README.rst index 016462f2c..92f6ecb7e 100644 --- a/artwork/README.rst +++ b/artwork/README.rst @@ -10,10 +10,10 @@ scrapy-logo.jpg Main Scrapy logo, in JPEG format. -qlassik.zip +qlassik.zip ----------- -Font used for Scrapy logo. Homepage: http://www.dafont.com/qlassik.font +Font used for Scrapy logo. Homepage: https://www.dafont.com/qlassik.font scrapy-blog.logo.xcf -------------------- diff --git a/docs/contributing.rst b/docs/contributing.rst index c969bd842..291a1054e 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -188,7 +188,7 @@ And their unit-tests are in:: .. _issue tracker: https://github.com/scrapy/scrapy/issues .. _scrapy-users: https://groups.google.com/forum/#!forum/scrapy-users -.. _Scrapy subreddit: http://reddit.com/r/scrapy +.. _Scrapy subreddit: https://reddit.com/r/scrapy .. _Twisted unit-testing framework: https://twistedmatrix.com/documents/current/core/development/policy/test-standard.html .. _AUTHORS: https://github.com/scrapy/scrapy/blob/master/AUTHORS .. _tests/: https://github.com/scrapy/scrapy/tree/master/tests diff --git a/docs/faq.rst b/docs/faq.rst index f0ee20b5e..42c3abbfa 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -21,7 +21,7 @@ Python code. In other words, comparing `BeautifulSoup`_ (or `lxml`_) to Scrapy is like comparing `jinja2`_ to `Django`_. -.. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ +.. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/ .. _lxml: http://lxml.de/ .. _jinja2: http://jinja.pocoo.org/ .. _Django: https://www.djangoproject.com/ diff --git a/docs/intro/install.rst b/docs/intro/install.rst index deb0118d4..12d489612 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -188,7 +188,7 @@ solutions: that doesn't conflict with the rest of your system. Here's how to do it using the `homebrew`_ package manager: - * Install `homebrew`_ following the instructions in http://brew.sh/ + * Install `homebrew`_ following the instructions in https://brew.sh/ * Update your ``PATH`` variable to state that homebrew packages should be used before system packages (Change ``.bashrc`` to ``.zshrc`` accordantly @@ -233,9 +233,9 @@ After any of these workarounds you should be able to install Scrapy:: .. _pyOpenSSL: https://pypi.python.org/pypi/pyOpenSSL .. _setuptools: https://pypi.python.org/pypi/setuptools .. _AUR Scrapy package: https://aur.archlinux.org/packages/scrapy/ -.. _homebrew: http://brew.sh/ -.. _zsh: http://www.zsh.org/ +.. _homebrew: https://brew.sh/ +.. _zsh: https://www.zsh.org/ .. _Scrapinghub: https://scrapinghub.com -.. _Anaconda: http://docs.continuum.io/anaconda/index -.. _Miniconda: http://conda.pydata.org/docs/install/quick.html -.. _conda-forge: https://conda-forge.github.io/ +.. _Anaconda: https://docs.anaconda.com/anaconda/ +.. _Miniconda: https://conda.io/docs/user-guide/install/index.html +.. _conda-forge: https://conda-forge.org/ diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 1da1a4059..d0ce07a8e 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -164,4 +164,4 @@ interest! .. _web scraping: https://en.wikipedia.org/wiki/Web_scraping .. _Amazon Associates Web Services: https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html .. _Amazon S3: https://aws.amazon.com/s3/ -.. _Sitemaps: http://www.sitemaps.org +.. _Sitemaps: https://www.sitemaps.org/index.html diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index a02c759bb..20538e90f 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -34,7 +34,7 @@ list of Python resources for non-programmers`_. .. _this list of Python resources for non-programmers: https://wiki.python.org/moin/BeginnersGuide/NonProgrammers .. _Dive Into Python 3: http://www.diveintopython3.net .. _Python Tutorial: https://docs.python.org/3/tutorial -.. _Learn Python The Hard Way: http://learnpythonthehardway.org/book/ +.. _Learn Python The Hard Way: https://learnpythonthehardway.org/book/ Creating a project diff --git a/docs/topics/debug.rst b/docs/topics/debug.rst index a3e72097c..d1991c02f 100644 --- a/docs/topics/debug.rst +++ b/docs/topics/debug.rst @@ -142,4 +142,4 @@ available in all future runs should they be necessary again:: For more information, check the :ref:`topics-logging` section. -.. _base tag: http://www.w3schools.com/tags/tag_base.asp +.. _base tag: https://www.w3schools.com/tags/tag_base.asp diff --git a/docs/topics/deploy.rst b/docs/topics/deploy.rst index f2e11fe8f..361914a29 100644 --- a/docs/topics/deploy.rst +++ b/docs/topics/deploy.rst @@ -55,5 +55,5 @@ just like ``scrapyd-deploy``. .. _scrapyd-client: https://github.com/scrapy/scrapyd-client .. _shub: https://doc.scrapinghub.com/shub.html .. _scrapyd-deploy documentation: https://scrapyd.readthedocs.io/en/latest/deploy.html -.. _Scrapy Cloud documentation: http://doc.scrapinghub.com/scrapy-cloud.html +.. _Scrapy Cloud documentation: https://doc.scrapinghub.com/scrapy-cloud.html .. _Scrapinghub: https://scrapinghub.com/ diff --git a/docs/topics/firebug.rst b/docs/topics/firebug.rst index 8f0a5767b..4ea8d3bd0 100644 --- a/docs/topics/firebug.rst +++ b/docs/topics/firebug.rst @@ -23,7 +23,7 @@ In this example, we'll show how to use `Firebug`_ to scrape data from the Project`_ used in the :ref:`tutorial ` but with a different face. -.. _Firebug: http://getfirebug.com +.. _Firebug: https://getfirebug.com/ .. _Google Directory: http://directory.google.com/ .. _Open Directory Project: http://www.dmoz.org diff --git a/docs/topics/firefox.rst b/docs/topics/firefox.rst index 0cf45861a..2c85848be 100644 --- a/docs/topics/firefox.rst +++ b/docs/topics/firefox.rst @@ -17,7 +17,7 @@ when inspecting the page source is not the original HTML, but a modified one after applying some browser clean up and executing Javascript code. Firefox, in particular, is known for adding ```` elements to tables. Scrapy, on the other hand, does not modify the original page HTML, so you won't be able to -extract any data if you use ```` in your XPath expressions. +extract any data if you use ```` in your XPath expressions. Therefore, you should keep in mind the following things when working with Firefox and XPath: @@ -71,11 +71,11 @@ Firecookie `Firecookie`_ makes it easier to view and manage cookies. You can use this extension to create a new cookie, delete existing cookies, see a list of cookies -for the current site, manage cookies permissions and a lot more. +for the current site, manage cookies permissions and a lot more. -.. _Firebug: http://getfirebug.com +.. _Firebug: https://getfirebug.com/ .. _Inspect Element: https://www.youtube.com/watch?v=-pT_pDe54aA -.. _XPather: https://addons.mozilla.org/en-US/firefox/addon/xpather/ +.. _XPather: https://addons.mozilla.org/en-US/firefox/addon/xpather/ .. _XPath Checker: https://addons.mozilla.org/en-US/firefox/addon/xpath-checker/ .. _Tamper Data: https://addons.mozilla.org/en-US/firefox/addon/tamper-data/ .. _Firecookie: https://addons.mozilla.org/en-US/firefox/addon/firecookie/ diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index 4f9e38086..06c7fff3d 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -100,4 +100,4 @@ If you wish to log the requests that couldn't be serialized, you can set the :setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page. It is ``False`` by default. -.. _pickle: http://docs.python.org/library/pickle.html +.. _pickle: https://docs.python.org/library/pickle.html diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index ad86dba63..0849090b4 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -518,8 +518,8 @@ a footer of a page that looks something like: Example:: diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index 9580a15d9..4c634ace5 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -171,7 +171,7 @@ policy:: For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide. -.. _canned ACLs: http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl +.. _canned ACLs: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl Google Cloud Storage --------------------- diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 21aa4a0a7..e0dd4000f 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -249,9 +249,9 @@ If you are still unable to prevent your bot getting banned, consider contacting .. _Tor project: https://www.torproject.org/ .. _commercial support: http://scrapy.org/support/ -.. _ProxyMesh: http://proxymesh.com/ +.. _ProxyMesh: https://proxymesh.com/ .. _Google cache: http://www.googleguide.com/cached_pages.html .. _testspiders: https://github.com/scrapinghub/testspiders .. _Twisted Reactor Overview: https://twistedmatrix.com/documents/current/core/howto/reactor-basics.html .. _Crawlera: https://scrapinghub.com/crawlera -.. _scrapoxy: http://scrapoxy.io/ +.. _scrapoxy: https://scrapoxy.io/ diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 92aae1ad0..121abe6b5 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -734,7 +734,7 @@ HtmlResponse objects which adds encoding auto-discovering support by looking into the HTML `meta http-equiv`_ attribute. See :attr:`TextResponse.encoding`. -.. _meta http-equiv: http://www.w3schools.com/TAGS/att_meta_http_equiv.asp +.. _meta http-equiv: https://www.w3schools.com/TAGS/att_meta_http_equiv.asp XmlResponse objects ------------------- diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index 9bae53f45..cb4c25391 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -36,7 +36,7 @@ documents. For a complete reference of the selectors API see :ref:`Selector reference ` -.. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ +.. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/ .. _lxml: http://lxml.de/ .. _ElementTree: https://docs.python.org/2/library/xml.etree.elementtree.html .. _cssselect: https://pypi.python.org/pypi/cssselect/ diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 37e3828a4..4b15cb607 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1002,7 +1002,7 @@ The randomization policy is the same used by `wget`_ ``--random-wait`` option. If :setting:`DOWNLOAD_DELAY` is zero (default) this option has no effect. -.. _wget: http://www.gnu.org/software/wget/manual/wget.html +.. _wget: https://www.gnu.org/software/wget/manual/wget.html .. setting:: REACTOR_THREADPOOL_MAXSIZE @@ -1317,7 +1317,7 @@ Default: ``2083`` Scope: ``spidermiddlewares.urllength`` The maximum URL length to allow for crawled URLs. For more information about -the default value for this setting see: http://www.boutell.com/newfaq/misc/urllength.html +the default value for this setting see: https://boutell.com/newfaq/misc/urllength.html .. setting:: USER_AGENT diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index ef6aeeed3..527116418 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -39,9 +39,9 @@ variable; or by defining it in your :ref:`scrapy.cfg `:: [settings] shell = bpython -.. _IPython: http://ipython.org/ -.. _IPython installation guide: http://ipython.org/install.html -.. _bpython: http://www.bpython-interpreter.org/ +.. _IPython: https://ipython.org/ +.. _IPython installation guide: https://ipython.org/install.html +.. _bpython: https://www.bpython-interpreter.org/ Launch the shell ================ diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index bf1532d1b..c2c271245 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -751,8 +751,8 @@ Combine SitemapSpider with other sources of urls:: def parse_other(self, response): pass # ... scrape other here ... -.. _Sitemaps: http://www.sitemaps.org -.. _Sitemap index files: http://www.sitemaps.org/protocol.html#index +.. _Sitemaps: https://www.sitemaps.org/index.html +.. _Sitemap index files: https://www.sitemaps.org/protocol.html#index .. _robots.txt: http://www.robotstxt.org/ .. _TLD: https://en.wikipedia.org/wiki/Top-level_domain .. _Scrapyd documentation: https://scrapyd.readthedocs.io/en/latest/ diff --git a/extras/coverage-report.sh b/extras/coverage-report.sh index dc20e16e4..842d0e46e 100755 --- a/extras/coverage-report.sh +++ b/extras/coverage-report.sh @@ -1,6 +1,6 @@ # Run tests, generate coverage report and open it on a browser # -# Requires: coverage 3.3 or above from http://pypi.python.org/pypi/coverage +# Requires: coverage 3.3 or above from https://pypi.python.org/pypi/coverage coverage run --branch $(which trial) --reporter=text tests coverage html -i