mirror of https://github.com/scrapy/scrapy.git
Merge 12353887d6 into ad43bf0c56
This commit is contained in:
Adrián Chaves
GitHub
commit
499179fa7f
|
|
@ -12,7 +12,7 @@ The GitHub issue tracker's purpose is to deal with bug reports and feature reque
|
|||
|
||||
Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
|
||||
|
||||
The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs
|
||||
The following is a suggested template to structure your issue, you can find more guidelines at https://docs.scrapy.org/en/latest/contributing.html#reporting-bugs
|
||||
|
||||
-->
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ The GitHub issue tracker's purpose is to deal with bug reports and feature reque
|
|||
|
||||
Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
|
||||
|
||||
The following is a suggested template to structure your pull request, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#writing-patches and https://doc.scrapy.org/en/latest/contributing.html#submitting-patches
|
||||
The following is a suggested template to structure your pull request, you can find more guidelines at https://docs.scrapy.org/en/latest/contributing.html#writing-patches and https://docs.scrapy.org/en/latest/contributing.html#submitting-patches
|
||||
|
||||
-->
|
||||
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ Contributing to Scrapy
|
|||
|
||||
.. important::
|
||||
|
||||
Double check that you are reading the most recent version of this document
|
||||
Double-check that you are reading the most recent version of this document
|
||||
at https://docs.scrapy.org/en/master/contributing.html
|
||||
|
||||
By participating in this project you agree to abide by the terms of our
|
||||
|
|
@ -216,14 +216,13 @@ has been validated and proven useful. Alternatively, you can start a
|
|||
conversation in the `Scrapy subreddit`_ to discuss your idea first.
|
||||
|
||||
Sometimes there is an existing pull request for the problem you'd like to
|
||||
solve, which is stalled for some reason. Often the pull request is in a
|
||||
right direction, but changes are requested by Scrapy maintainers, and the
|
||||
original pull request author hasn't had time to address them.
|
||||
In this case consider picking up this pull request: open
|
||||
a new pull request with all commits from the original pull request, as well as
|
||||
additional changes to address the raised issues. Doing so helps a lot; it is
|
||||
not considered rude as long as the original author is acknowledged by keeping
|
||||
his/her commits.
|
||||
solve, which is stalled for some reason. Often the pull request is in the right
|
||||
direction, but changes are requested by Scrapy maintainers, and the original
|
||||
pull request author hasn't had time to address them. In this case consider
|
||||
picking up this pull request: open a new pull request with all commits from the
|
||||
original pull request, as well as additional changes to address the raised
|
||||
issues. Doing so helps a lot; it is not considered rude as long as the original
|
||||
author is acknowledged by keeping their commits.
|
||||
|
||||
You can pull an existing pull request to a local branch
|
||||
by running ``git fetch upstream pull/$PR_NUMBER/head:$BRANCH_NAME_TO_CREATE``
|
||||
|
|
@ -233,9 +232,9 @@ with a name of the branch you want to create locally).
|
|||
See also: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally#modifying-an-inactive-pull-request-locally.
|
||||
|
||||
When writing GitHub pull requests, try to keep titles short but descriptive.
|
||||
E.g. For bug #411: "Scrapy hangs if an exception raises in start_requests"
|
||||
prefer "Fix hanging when exception occurs in start_requests (#411)"
|
||||
instead of "Fix for #411". Complete titles make it easy to skim through
|
||||
For example, for bug #411: "Scrapy hangs if an exception raises in
|
||||
start_requests" prefer "Fix hanging when exception occurs in start_requests
|
||||
(#411)" instead of "Fix for #411". Complete titles make it easy to skim through
|
||||
the issue tracker.
|
||||
|
||||
Finally, try to keep aesthetic changes (:pep:`8` compliance, unused imports
|
||||
|
|
@ -271,7 +270,7 @@ commit.
|
|||
|
||||
.. _pre-commit: https://pre-commit.com/
|
||||
|
||||
After your create a local clone of your fork of the Scrapy repository:
|
||||
After you create a local clone of your fork of the Scrapy repository:
|
||||
|
||||
#. `Install pre-commit <https://pre-commit.com/#installation>`_.
|
||||
|
||||
|
|
|
|||
14
docs/faq.rst
14
docs/faq.rst
|
|
@ -118,7 +118,7 @@ My Scrapy crawler has memory leaks. What can I do?
|
|||
|
||||
See :ref:`topics-leaks`.
|
||||
|
||||
Also, Python has a builtin memory leak issue which is described in
|
||||
Also, Python has a built-in memory leak issue which is described in
|
||||
:ref:`topics-leaks-without-leaks`.
|
||||
|
||||
How can I make Scrapy consume less memory?
|
||||
|
|
@ -136,8 +136,8 @@ middleware with a :ref:`custom downloader middleware
|
|||
<topics-downloader-middleware-custom>` that requires less memory. For example:
|
||||
|
||||
- If your domain names are similar enough, use your own regular expression
|
||||
instead of joining the strings in :attr:`~scrapy.Spider.allowed_domains` into
|
||||
a complex regular expression.
|
||||
instead of joining the strings in :attr:`~scrapy.Spider.allowed_domains`
|
||||
into a complex regular expression.
|
||||
|
||||
- If you can meet the installation requirements, use pyre2_ instead of
|
||||
Python’s re_ to compile your URL-filtering regular expression. See
|
||||
|
|
@ -276,8 +276,8 @@ Parsing big feeds with XPath selectors can be problematic since they need to
|
|||
build the DOM of the entire feed in memory, and this can be quite slow and
|
||||
consume a lot of memory.
|
||||
|
||||
In order to avoid parsing all the entire feed at once in memory, you can use
|
||||
the :func:`~scrapy.utils.iterators.xmliter_lxml` and
|
||||
In order to avoid parsing the entire feed at once in memory, you can use the
|
||||
:func:`~scrapy.utils.iterators.xmliter_lxml` and
|
||||
:func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what
|
||||
:class:`~scrapy.spiders.XMLFeedSpider` and
|
||||
:class:`~scrapy.spiders.CSVFeedSpider` use.
|
||||
|
|
@ -314,8 +314,8 @@ Should I use spider arguments or settings to configure my spider?
|
|||
-----------------------------------------------------------------
|
||||
|
||||
Both :ref:`spider arguments <spiderargs>` and :ref:`settings <topics-settings>`
|
||||
can be used to configure your spider. There is no strict rule that mandates to
|
||||
use one or the other, but settings are more suited for parameters that, once
|
||||
can be used to configure your spider. There is no strict rule that mandates
|
||||
using one or the other, but settings are more suited for parameters that, once
|
||||
set, don't change much, while spider arguments are meant to change more often,
|
||||
even on each spider run and sometimes are required for the spider to run at all
|
||||
(for example, to set the start url of a spider).
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ Having trouble? We'd like to help!
|
|||
|
||||
* Try the :doc:`FAQ <faq>` -- it's got answers to some common questions.
|
||||
* Looking for specific information? Try the :ref:`genindex` or :ref:`modindex`.
|
||||
* Ask or search questions in `StackOverflow using the scrapy tag`_.
|
||||
* Ask or search questions on `Stack Overflow using the scrapy tag`_.
|
||||
* Ask or search questions in the `Scrapy subreddit`_.
|
||||
* Search for questions on the archives of the `scrapy-users mailing list`_.
|
||||
* Ask a question in the `#scrapy IRC channel`_.
|
||||
|
|
@ -30,7 +30,7 @@ Having trouble? We'd like to help!
|
|||
|
||||
.. _scrapy-users mailing list: https://groups.google.com/forum/#!forum/scrapy-users
|
||||
.. _Scrapy subreddit: https://www.reddit.com/r/scrapy/
|
||||
.. _StackOverflow using the scrapy tag: https://stackoverflow.com/tags/scrapy
|
||||
.. _Stack Overflow using the scrapy tag: https://stackoverflow.com/tags/scrapy
|
||||
.. _#scrapy IRC channel: irc://irc.freenode.net/scrapy
|
||||
.. _issue tracker: https://github.com/scrapy/scrapy/issues
|
||||
.. _Scrapy Discord: https://discord.com/invite/mv3yErfpvq
|
||||
|
|
@ -135,7 +135,7 @@ Built-in services
|
|||
topics/telnetconsole
|
||||
|
||||
:doc:`topics/logging`
|
||||
Learn how to use Python's built-in logging on Scrapy.
|
||||
Learn how to use Python's built-in logging in Scrapy.
|
||||
|
||||
:doc:`topics/stats`
|
||||
Collect statistics about your scraping crawler.
|
||||
|
|
@ -200,7 +200,7 @@ Solving specific problems
|
|||
Download files and/or images associated with your scraped items.
|
||||
|
||||
:doc:`topics/deploy`
|
||||
Deploying your Scrapy spiders and run them in a remote server.
|
||||
Deploy your Scrapy spiders and run them on a remote server.
|
||||
|
||||
:doc:`topics/autothrottle`
|
||||
Adjust crawl rate dynamically based on load.
|
||||
|
|
@ -252,7 +252,7 @@ Extending Scrapy
|
|||
Customize the input and output of your spiders.
|
||||
|
||||
:doc:`topics/extensions`
|
||||
Extend Scrapy with your custom functionality
|
||||
Extend Scrapy with your custom functionality.
|
||||
|
||||
:doc:`topics/signals`
|
||||
See all available signals and how to work with them.
|
||||
|
|
@ -261,7 +261,7 @@ Extending Scrapy
|
|||
Understand the scheduler component.
|
||||
|
||||
:doc:`topics/exporters`
|
||||
Quickly export your scraped items to a file (XML, CSV, etc).
|
||||
Quickly export your scraped items to a file (XML, CSV, etc.).
|
||||
|
||||
:doc:`topics/download-handlers`
|
||||
Customize how requests are downloaded or add support for new URL schemes.
|
||||
|
|
@ -271,7 +271,7 @@ Extending Scrapy
|
|||
components.
|
||||
|
||||
:doc:`topics/api`
|
||||
Use it on extensions and middlewares to extend Scrapy functionality.
|
||||
Use it in extensions and middlewares to extend Scrapy functionality.
|
||||
|
||||
|
||||
All the rest
|
||||
|
|
|
|||
|
|
@ -5,16 +5,16 @@ Examples
|
|||
========
|
||||
|
||||
The best way to learn is with examples, and Scrapy is no exception. For this
|
||||
reason, there is an example Scrapy project named quotesbot_, that you can use to
|
||||
reason, there is an example Scrapy project named quotesbot_ that you can use to
|
||||
play and learn more about Scrapy. It contains two spiders for
|
||||
https://quotes.toscrape.com, one using CSS selectors and another one using XPath
|
||||
expressions.
|
||||
https://quotes.toscrape.com, one using CSS selectors and another one using
|
||||
XPath expressions.
|
||||
|
||||
The quotesbot_ project is available at: https://github.com/scrapy/quotesbot.
|
||||
You can find more information about it in the project's README.
|
||||
|
||||
If you're familiar with git, you can checkout the code. Otherwise you can
|
||||
download the project as a zip file by clicking
|
||||
`here <https://github.com/scrapy/quotesbot/archive/master.zip>`_.
|
||||
If you're familiar with git, you can check out the code. Otherwise, you can
|
||||
download the project as a zip file by clicking `here
|
||||
<https://github.com/scrapy/quotesbot/archive/master.zip>`_.
|
||||
|
||||
.. _quotesbot: https://github.com/scrapy/quotesbot
|
||||
|
|
|
|||
|
|
@ -25,15 +25,15 @@ To install Scrapy using ``conda``, run::
|
|||
|
||||
conda install -c conda-forge scrapy
|
||||
|
||||
Alternatively, if you’re already familiar with installation of Python packages,
|
||||
you can install Scrapy and its dependencies from PyPI with::
|
||||
Alternatively, if you’re already familiar with installing Python packages, you
|
||||
can install Scrapy and its dependencies from PyPI with::
|
||||
|
||||
pip install Scrapy
|
||||
|
||||
We strongly recommend that you install Scrapy in :ref:`a dedicated virtualenv <intro-using-virtualenv>`,
|
||||
to avoid conflicting with your system packages.
|
||||
|
||||
Note that sometimes this may require solving compilation issues for some Scrapy
|
||||
Note that this may sometimes require solving compilation issues for some Scrapy
|
||||
dependencies depending on your operating system, so be sure to check the
|
||||
:ref:`intro-install-platform-notes`.
|
||||
|
||||
|
|
@ -47,7 +47,7 @@ Things that are good to know
|
|||
Scrapy is written in pure Python and depends on a few key Python packages (among others):
|
||||
|
||||
* `lxml`_, an efficient XML and HTML parser
|
||||
* `parsel`_, an HTML/XML data extraction library written on top of lxml,
|
||||
* `parsel`_, an HTML/XML data extraction library written on top of lxml
|
||||
* `w3lib`_, a multi-purpose helper for dealing with URLs and web page encodings
|
||||
* `twisted`_, an asynchronous networking framework
|
||||
* `cryptography`_ and `pyOpenSSL`_, to deal with various network-level security needs
|
||||
|
|
@ -73,14 +73,14 @@ Using a virtual environment (recommended)
|
|||
TL;DR: We recommend installing Scrapy inside a virtual environment
|
||||
on all platforms.
|
||||
|
||||
Python packages can be installed either globally (a.k.a system wide),
|
||||
or in user-space. We do not recommend installing Scrapy system wide.
|
||||
Python packages can be installed either globally (a.k.a. system-wide), or in
|
||||
user-space. We do not recommend installing Scrapy system-wide.
|
||||
|
||||
Instead, we recommend that you install Scrapy within a so-called
|
||||
"virtual environment" (:mod:`venv`).
|
||||
Virtual environments allow you to not conflict with already-installed Python
|
||||
system packages (which could break some of your system tools and scripts),
|
||||
and still install packages normally with ``pip`` (without ``sudo`` and the likes).
|
||||
Instead, we recommend that you install Scrapy within a so-called "virtual
|
||||
environment" (:mod:`venv`). Virtual environments allow you to avoid conflicts
|
||||
with already-installed Python system packages (which could break some of your
|
||||
system tools and scripts), and still install packages normally with ``pip``
|
||||
(without ``sudo`` or similar tools).
|
||||
|
||||
See :ref:`tut-venv` on how to create your virtual environment.
|
||||
|
||||
|
|
@ -140,7 +140,7 @@ The following extras are available:
|
|||
|
||||
.. _intro-install-platform-notes:
|
||||
|
||||
Platform specific installation notes
|
||||
Platform-specific installation notes
|
||||
====================================
|
||||
|
||||
.. _intro-install-windows:
|
||||
|
|
@ -168,11 +168,12 @@ To install Scrapy on Windows using ``pip``:
|
|||
|
||||
#. Under the Workloads section, select **C++ build tools**.
|
||||
|
||||
#. Check the installation details and make sure following packages are selected as optional components:
|
||||
#. Check the installation details and make sure the following packages are
|
||||
selected as optional components:
|
||||
|
||||
* **MSVC** (e.g MSVC v142 - VS 2019 C++ x64/x86 build tools (v14.23) )
|
||||
* **MSVC** (e.g. MSVC v142 - VS 2019 C++ x64/x86 build tools (v14.23) )
|
||||
|
||||
* **Windows SDK** (e.g Windows 10 SDK (10.0.18362.0))
|
||||
* **Windows SDK** (e.g. Windows 10 SDK (10.0.18362.0))
|
||||
|
||||
#. Install the Visual Studio Build Tools.
|
||||
|
||||
|
|
@ -188,8 +189,8 @@ twisted and pyOpenSSL, and is compatible with recent Ubuntu distributions.
|
|||
But it should support older versions of Ubuntu too, like Ubuntu 14.04,
|
||||
albeit with potential issues with TLS connections.
|
||||
|
||||
**Don't** use the ``python-scrapy`` package provided by Ubuntu, they are
|
||||
typically too old and slow to catch up with the latest Scrapy release.
|
||||
**Don't** use the ``python-scrapy`` package provided by Ubuntu; it is typically
|
||||
too old and slow to catch up with the latest Scrapy release.
|
||||
|
||||
|
||||
To install Scrapy on Ubuntu (or Ubuntu-based) systems, you need to install
|
||||
|
|
@ -235,8 +236,8 @@ solutions:
|
|||
* Install `homebrew`_ following the instructions in https://brew.sh/
|
||||
|
||||
* Update your ``PATH`` variable to state that homebrew packages should be
|
||||
used before system packages (Change ``.bashrc`` to ``.zshrc`` accordingly
|
||||
if you're using `zsh`_ as default shell)::
|
||||
used before system packages (change ``.bashrc`` to ``.zshrc`` accordingly
|
||||
if you're using `zsh`_ as the default shell)::
|
||||
|
||||
echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc
|
||||
|
||||
|
|
@ -244,7 +245,7 @@ solutions:
|
|||
|
||||
source ~/.bashrc
|
||||
|
||||
* Install python::
|
||||
* Install Python::
|
||||
|
||||
brew install python
|
||||
|
||||
|
|
@ -254,7 +255,7 @@ solutions:
|
|||
This method is a workaround for the above macOS issue, but it's an overall
|
||||
good practice for managing dependencies and can complement the first method.
|
||||
|
||||
After any of these workarounds you should be able to install Scrapy::
|
||||
After any of these workarounds, you should be able to install Scrapy::
|
||||
|
||||
pip install Scrapy
|
||||
|
||||
|
|
@ -266,14 +267,14 @@ We recommend using the latest PyPy version.
|
|||
For PyPy3, only Linux installation was tested.
|
||||
|
||||
Most Scrapy dependencies now have binary wheels for CPython, but not for PyPy.
|
||||
This means that these dependencies will be built during installation.
|
||||
On macOS, you are likely to face an issue with building the Cryptography
|
||||
dependency. The solution to this problem is described
|
||||
`here <https://github.com/pyca/cryptography/issues/2692#issuecomment-272773481>`_,
|
||||
This means that these dependencies will be built during installation. On macOS,
|
||||
you are likely to face an issue with building the Cryptography dependency. The
|
||||
solution to this problem is described `here
|
||||
<https://github.com/pyca/cryptography/issues/2692#issuecomment-272773481>`_,
|
||||
that is to ``brew install openssl`` and then export the flags that this command
|
||||
recommends (only needed when installing Scrapy). Installing on Linux has no special
|
||||
issues besides installing build dependencies.
|
||||
Installing Scrapy with PyPy on Windows is not tested.
|
||||
recommends (only needed when installing Scrapy). Installing Scrapy on Linux has
|
||||
no special issues beyond installing the build dependencies. Installing Scrapy
|
||||
with PyPy on Windows has not been tested.
|
||||
|
||||
You can check that Scrapy is installed correctly by running ``scrapy bench``.
|
||||
If this command gives errors such as
|
||||
|
|
|
|||
|
|
@ -4,13 +4,13 @@
|
|||
Scrapy at a glance
|
||||
==================
|
||||
|
||||
Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting
|
||||
structured data which can be used for a wide range of useful applications, like
|
||||
data mining, information processing or historical archival.
|
||||
Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling websites and
|
||||
extracting structured data which can be used for a wide range of useful
|
||||
applications, like data mining, information processing or historical archival.
|
||||
|
||||
Even though Scrapy was originally designed for `web scraping`_, it can also be
|
||||
used to extract data using APIs (such as `Amazon Associates Web Services`_) or
|
||||
as a general purpose web crawler.
|
||||
as a general-purpose web crawler.
|
||||
|
||||
|
||||
Walk-through of an example spider
|
||||
|
|
@ -19,7 +19,7 @@ Walk-through of an example spider
|
|||
In order to show you what Scrapy brings to the table, we'll walk you through an
|
||||
example of a Scrapy Spider using the simplest way to run a spider.
|
||||
|
||||
Here's the code for a spider that scrapes famous quotes from website
|
||||
Here's the code for a spider that scrapes famous quotes from the website
|
||||
https://quotes.toscrape.com, following the pagination:
|
||||
|
||||
.. code-block:: python
|
||||
|
|
@ -49,8 +49,9 @@ and run the spider using the :command:`runspider` command::
|
|||
|
||||
scrapy runspider quotes_spider.py -o quotes.jsonl
|
||||
|
||||
When this finishes you will have in the ``quotes.jsonl`` file a list of the
|
||||
quotes in JSON Lines format, containing the text and author, which will look like this::
|
||||
When this finishes, you will have a list of the quotes in JSON Lines format in
|
||||
the ``quotes.jsonl`` file, containing the text and author, which will look like
|
||||
this::
|
||||
|
||||
{"author": "Jane Austen", "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"}
|
||||
{"author": "Steve Martin", "text": "\u201cA day without sunshine is like, you know, night.\u201d"}
|
||||
|
|
@ -73,19 +74,19 @@ look for a link to the next page and schedule another request using the same
|
|||
``parse`` method as callback.
|
||||
|
||||
Here you will notice one of the main advantages of Scrapy: requests are
|
||||
:ref:`scheduled and processed asynchronously <topics-architecture>`. This
|
||||
means that Scrapy doesn't need to wait for a request to be finished and
|
||||
processed, it can send another request or do other things in the meantime. This
|
||||
also means that other requests can keep going even if a request fails or an
|
||||
error happens while handling it.
|
||||
:ref:`scheduled and processed asynchronously <topics-architecture>`. This means
|
||||
that Scrapy doesn't need to wait for a request to be finished and processed; it
|
||||
can send another request or do other things in the meantime. This also means
|
||||
that other requests can keep going even if a request fails or an error happens
|
||||
while handling it.
|
||||
|
||||
While this enables you to do very fast crawls (sending multiple concurrent
|
||||
requests at the same time, in a fault-tolerant way) Scrapy also gives you
|
||||
control over the politeness of the crawl through :ref:`a few settings
|
||||
<topics-settings-ref>`. You can do things like setting a download delay between
|
||||
each request, limiting the amount of concurrent requests per domain, and
|
||||
even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries
|
||||
to figure these settings out automatically.
|
||||
each request, limiting the number of concurrent requests per domain, and even
|
||||
:ref:`using an auto-throttling extension <topics-autothrottle>` that tries to
|
||||
figure these settings out automatically.
|
||||
|
||||
.. note::
|
||||
|
||||
|
|
@ -117,7 +118,7 @@ scraping easy and efficient, such as:
|
|||
multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP,
|
||||
S3, local filesystem)
|
||||
|
||||
* Robust encoding support and auto-detection, for dealing with foreign,
|
||||
* Robust encoding support and auto-detection for dealing with foreign,
|
||||
non-standard and broken encoding declarations.
|
||||
|
||||
* :ref:`Strong extensibility support <extending-scrapy>`, allowing you to plug
|
||||
|
|
@ -138,10 +139,10 @@ scraping easy and efficient, such as:
|
|||
console running inside your Scrapy process, to introspect and debug your
|
||||
crawler
|
||||
|
||||
* Plus other goodies like reusable spiders to crawl sites from `Sitemaps`_ and
|
||||
XML/CSV feeds, a media pipeline for :ref:`automatically downloading images
|
||||
<topics-media-pipeline>` (or any other media) associated with the scraped
|
||||
items, a caching DNS resolver, and much more!
|
||||
* Plus other goodies such as reusable spiders to crawl sites from `Sitemaps`_
|
||||
and XML/CSV feeds, a media pipeline for :ref:`automatically downloading
|
||||
images <topics-media-pipeline>` (or any other media) associated with the
|
||||
scraped items, a caching DNS resolver, and much more!
|
||||
|
||||
What's next?
|
||||
============
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ This tutorial will walk you through these tasks:
|
|||
1. Creating a new Scrapy project
|
||||
2. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract data
|
||||
3. Exporting the scraped data using the command line
|
||||
4. Changing spider to recursively follow links
|
||||
4. Changing the spider to recursively follow links
|
||||
5. Using spider arguments
|
||||
|
||||
Scrapy is written in Python_. The more you learn about Python, the more you
|
||||
|
|
@ -81,10 +81,11 @@ ask you to adjust it, rather than block it.
|
|||
Our first Spider
|
||||
================
|
||||
|
||||
Spiders are classes that you define and that Scrapy uses to scrape information from a website
|
||||
(or a group of websites). They must subclass :class:`~scrapy.Spider` and define the initial
|
||||
requests to be made, and optionally, how to follow links in pages and parse the downloaded
|
||||
page content to extract data.
|
||||
Spiders are classes that you define and that Scrapy uses to scrape information
|
||||
from a website (or a group of websites). They must subclass
|
||||
:class:`~scrapy.Spider` and define the initial requests to be made and,
|
||||
optionally, how to follow links in pages and parse the downloaded page content
|
||||
to extract data.
|
||||
|
||||
This is the code for our first Spider. Save it in a file named
|
||||
``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project:
|
||||
|
|
@ -142,9 +143,9 @@ To put our spider to work, go to the project's top level directory and run::
|
|||
|
||||
scrapy crawl quotes
|
||||
|
||||
This command runs the spider named ``quotes`` that we've just added, that
|
||||
will send some requests for the ``quotes.toscrape.com`` domain. You will get an output
|
||||
similar to this::
|
||||
This command runs the spider named ``quotes`` that we've just added, which will
|
||||
send some requests for the ``quotes.toscrape.com`` domain. You will get an
|
||||
output similar to this::
|
||||
|
||||
... (omitted for brevity)
|
||||
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened
|
||||
|
|
@ -405,8 +406,8 @@ like this:
|
|||
</div>
|
||||
</div>
|
||||
|
||||
Let's open up scrapy shell and play a bit to find out how to extract the data
|
||||
we want::
|
||||
Let's open up the Scrapy shell and play a bit to find out how to extract the
|
||||
data we want::
|
||||
|
||||
scrapy shell 'https://quotes.toscrape.com'
|
||||
|
||||
|
|
@ -673,14 +674,14 @@ As a shortcut for creating Request objects you can use
|
|||
if next_page is not None:
|
||||
yield response.follow(next_page, callback=self.parse)
|
||||
|
||||
Unlike scrapy.Request, ``response.follow`` supports relative URLs directly - no
|
||||
need to call urljoin. Note that ``response.follow`` just returns a Request
|
||||
instance; you still have to yield this Request.
|
||||
Unlike :class:`scrapy.Request`, ``response.follow`` supports relative URLs
|
||||
directly - no need to call urljoin. Note that ``response.follow`` just returns
|
||||
a Request instance; you still have to yield this Request.
|
||||
|
||||
.. skip: start
|
||||
|
||||
You can also pass a selector to ``response.follow`` instead of a string;
|
||||
this selector should extract necessary attributes:
|
||||
You can also pass a selector to ``response.follow`` instead of a string; this
|
||||
selector should extract the necessary attributes:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -746,7 +747,7 @@ this time for scraping author information:
|
|||
}
|
||||
|
||||
This spider will start from the main page, it will follow all the links to the
|
||||
authors pages calling the ``parse_author`` callback for each of them, and also
|
||||
author pages calling the ``parse_author`` callback for each of them, and also
|
||||
the pagination links with the ``parse`` callback as we saw before.
|
||||
|
||||
Here we're passing callbacks to
|
||||
|
|
@ -754,8 +755,8 @@ Here we're passing callbacks to
|
|||
arguments to make the code shorter; it also works for
|
||||
:class:`~scrapy.Request`.
|
||||
|
||||
The ``parse_author`` callback defines a helper function to extract and cleanup the
|
||||
data from a CSS query and yields the Python dict with the author data.
|
||||
The ``parse_author`` callback defines a helper function to extract and clean up
|
||||
the data from a CSS query and yields the Python dict with the author data.
|
||||
|
||||
Another interesting thing this spider demonstrates is that, even if there are
|
||||
many quotes from the same author, we don't need to worry about visiting the
|
||||
|
|
|
|||
|
|
@ -1921,7 +1921,7 @@ Scrapy 2.13.2 (2025-06-09)
|
|||
when it's closed before being fully initialized.
|
||||
(:gh:`6857`, :gh:`6867`)
|
||||
|
||||
- Improved the README, updated the Scrapy logo in it.
|
||||
- Improved the README and updated the Scrapy logo in it.
|
||||
(:gh:`6831`, :gh:`6833`, :gh:`6839`)
|
||||
|
||||
- Restricted the Twisted version used in tests to below 25.5.0, as some tests
|
||||
|
|
@ -1940,14 +1940,14 @@ Scrapy 2.13.2 (2025-06-09)
|
|||
Scrapy 2.13.1 (2025-05-28)
|
||||
--------------------------
|
||||
|
||||
- Give callback requests precedence over start requests when priority values
|
||||
- Gave callback requests precedence over start requests when priority values
|
||||
are the same.
|
||||
|
||||
This makes changes from 2.13.0 to start request handling more intuitive and
|
||||
backward compatible. For scenarios where all requests have the same
|
||||
This makes the changes from 2.13.0 to start request handling more intuitive
|
||||
and backward compatible. For scenarios where all requests have the same
|
||||
priorities, in 2.13.0 all start requests were sent before the first
|
||||
callback request. In 2.13.1, same as in 2.12 and lower, start requests are
|
||||
only sent when there are not enough pending callback requests to reach
|
||||
callback request. In 2.13.1, the same as in 2.12 and lower, start requests
|
||||
are only sent when there are not enough pending callback requests to reach
|
||||
concurrency limits.
|
||||
|
||||
(:gh:`6828`)
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
Add-ons
|
||||
=======
|
||||
|
||||
Scrapy's add-on system is a framework which unifies managing and configuring
|
||||
Scrapy's add-on system is a framework that unifies managing and configuring
|
||||
components that extend Scrapy's core functionality, such as middlewares,
|
||||
extensions, or pipelines. It provides users with a plug-and-play experience in
|
||||
Scrapy extension management, and grants extensive configuration control to
|
||||
|
|
@ -77,7 +77,7 @@ usually best to leave its priority unchanged. For example, when editing a
|
|||
|
||||
If the ``update_settings`` method raises
|
||||
:exc:`scrapy.exceptions.NotConfigured`, the add-on will be skipped. This makes
|
||||
it easy to enable an add-on only when some conditions are met.
|
||||
it easy to enable an add-on only when certain conditions are met.
|
||||
|
||||
Fallbacks
|
||||
---------
|
||||
|
|
|
|||
|
|
@ -22,9 +22,9 @@ functionality into Scrapy.
|
|||
:synopsis: The Scrapy crawler
|
||||
|
||||
The Extension Manager is responsible for loading and keeping track of installed
|
||||
extensions and it's configured through the :setting:`EXTENSIONS` setting which
|
||||
contains a dictionary of all available extensions and their order similar to
|
||||
how you :ref:`configure the downloader middlewares
|
||||
extensions, and it's configured through the :setting:`EXTENSIONS` setting,
|
||||
which contains a dictionary of all available extensions and their order similar
|
||||
to how you :ref:`configure the downloader middlewares
|
||||
<topics-downloader-middleware-setting>`.
|
||||
|
||||
.. autoclass:: Crawler
|
||||
|
|
@ -96,7 +96,7 @@ how you :ref:`configure the downloader middlewares
|
|||
The execution engine, which coordinates the core crawling logic
|
||||
between the scheduler, downloader and spiders.
|
||||
|
||||
Some extension may want to access the Scrapy engine, to inspect or
|
||||
Some extensions may want to access the Scrapy engine to inspect or
|
||||
modify the downloader and scheduler behaviour, although this is an
|
||||
advanced use and this API is not yet stable.
|
||||
|
||||
|
|
|
|||
|
|
@ -62,9 +62,9 @@ this:
|
|||
:ref:`Spider Middleware <component-spider-middleware>` (see
|
||||
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`).
|
||||
|
||||
8. The :ref:`Engine <component-engine>` sends processed items to
|
||||
:ref:`Item Pipelines <component-pipelines>`, then sends processed Requests to
|
||||
the :ref:`Scheduler <component-scheduler>` and asks for possible next Requests
|
||||
8. The :ref:`Engine <component-engine>` sends processed items to :ref:`Item
|
||||
Pipelines <component-pipelines>`, then sends processed Requests to the
|
||||
:ref:`Scheduler <component-scheduler>` and asks for possible next Requests
|
||||
to crawl.
|
||||
|
||||
9. The process repeats (from step 3) until there are no more requests from the
|
||||
|
|
|
|||
|
|
@ -78,8 +78,8 @@ You should see an output like this::
|
|||
That tells you that Scrapy is able to crawl about 3000 pages per minute in the
|
||||
hardware where you run it. Note that this is a very simple spider intended to
|
||||
follow links, any custom spider you write will probably do more stuff which
|
||||
results in slower crawl rates. How slower depends on how much your spider does
|
||||
and how well it's written.
|
||||
results in slower crawl rates. How much slower depends on how much your spider
|
||||
does and how well it's written.
|
||||
|
||||
Use scrapy-bench_ for more complex benchmarking.
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ Scrapy is controlled through the ``scrapy`` command-line tool, to be referred to
|
|||
here as the "Scrapy tool" to differentiate it from the sub-commands, which we
|
||||
just call "commands" or "Scrapy commands".
|
||||
|
||||
The Scrapy tool provides several commands, for multiple purposes, and each one
|
||||
The Scrapy tool provides several commands for multiple purposes, and each one
|
||||
accepts a different set of arguments and options.
|
||||
|
||||
(The ``scrapy deploy`` command has been removed in 1.0 in favor of the
|
||||
|
|
@ -65,7 +65,7 @@ structure by default, similar to this::
|
|||
...
|
||||
|
||||
The directory where the ``scrapy.cfg`` file resides is known as the *project
|
||||
root directory*. That file contains the name of the python module that defines
|
||||
root directory*. That file contains the name of the Python module that defines
|
||||
the project settings. Here is an example:
|
||||
|
||||
.. code-block:: ini
|
||||
|
|
@ -105,7 +105,7 @@ for ``scrapy`` to use::
|
|||
Using the ``scrapy`` tool
|
||||
=========================
|
||||
|
||||
You can start by running the Scrapy tool with no arguments and it will print
|
||||
You can start by running the Scrapy tool with no arguments, and it will print
|
||||
some usage help and the available commands::
|
||||
|
||||
Scrapy X.Y - no active project
|
||||
|
|
@ -119,8 +119,8 @@ some usage help and the available commands::
|
|||
[...]
|
||||
|
||||
The first line will print the currently active project if you're inside a
|
||||
Scrapy project. In this example it was run from outside a project. If run from inside
|
||||
a project it would have printed something like this::
|
||||
Scrapy project. In this example, it was run from outside a project. If run from
|
||||
inside a project, it would have printed something like this::
|
||||
|
||||
Scrapy X.Y - project: myproject
|
||||
|
||||
|
|
@ -137,8 +137,8 @@ project::
|
|||
|
||||
scrapy startproject myproject [project_dir]
|
||||
|
||||
That will create a Scrapy project under the ``project_dir`` directory.
|
||||
If ``project_dir`` wasn't specified, ``project_dir`` will be the same as ``myproject``.
|
||||
That will create a Scrapy project under the ``project_dir`` directory. If
|
||||
``project_dir`` isn't specified, it defaults to ``myproject``.
|
||||
|
||||
Next, you go inside the new project directory::
|
||||
|
||||
|
|
@ -159,7 +159,8 @@ For example, to create a new spider::
|
|||
|
||||
Some Scrapy commands (like :command:`crawl`) must be run from inside a Scrapy
|
||||
project. See the :ref:`commands reference <topics-commands-ref>` below for more
|
||||
information on which commands must be run from inside projects, and which not.
|
||||
information on which commands must be run from inside projects and which ones
|
||||
do not.
|
||||
|
||||
Also keep in mind that some commands may have slightly different behaviours
|
||||
when running them from inside projects. For example, the fetch command will use
|
||||
|
|
@ -186,7 +187,7 @@ And you can see all available commands with::
|
|||
There are two kinds of commands, those that only work from inside a Scrapy
|
||||
project (Project-specific commands) and those that also work without an active
|
||||
Scrapy project (Global commands), though they may behave slightly differently
|
||||
when run from inside a project (as they would use the project overridden
|
||||
when run from inside a project (as they would use the project-overridden
|
||||
settings).
|
||||
|
||||
Global commands:
|
||||
|
|
@ -218,8 +219,7 @@ startproject
|
|||
* Requires project: *no*
|
||||
|
||||
Creates a new Scrapy project named ``project_name``, under the ``project_dir``
|
||||
directory.
|
||||
If ``project_dir`` wasn't specified, ``project_dir`` will be the same as ``project_name``.
|
||||
directory. If ``project_dir`` isn't specified, it defaults to ``project_name``.
|
||||
|
||||
Usage example::
|
||||
|
||||
|
|
@ -233,7 +233,10 @@ genspider
|
|||
* Syntax: ``scrapy genspider [-t template] <name> <domain or URL>``
|
||||
* Requires project: *no*
|
||||
|
||||
Creates a new spider in the current folder or in the current project's ``spiders`` folder, if called from inside a project. The ``<name>`` parameter is set as the spider's ``name``, while ``<domain or URL>`` is used to generate the ``allowed_domains`` and ``start_urls`` spider's attributes.
|
||||
Creates a new spider in the current folder or in the current project's
|
||||
``spiders`` folder, if called from inside a project. The ``<name>`` parameter
|
||||
becomes the spider's ``name``, while ``<domain or URL>`` is used to generate
|
||||
the spider's ``allowed_domains`` and ``start_urls`` attributes.
|
||||
|
||||
Usage example::
|
||||
|
||||
|
|
@ -250,9 +253,9 @@ Usage example::
|
|||
$ scrapy genspider -t crawl scrapyorg scrapy.org
|
||||
Created spider 'scrapyorg' using template 'crawl'
|
||||
|
||||
This is just a convenient shortcut command for creating spiders based on
|
||||
pre-defined templates, but certainly not the only way to create spiders. You
|
||||
can just create the spider source code files yourself, instead of using this
|
||||
This command is just a convenient shortcut for creating spiders based on
|
||||
pre-defined templates, but it's certainly not the only way to create spiders;
|
||||
you can create the spider source code files yourself instead of using this
|
||||
command.
|
||||
|
||||
.. command:: crawl
|
||||
|
|
@ -273,9 +276,13 @@ Supported options:
|
|||
|
||||
* ``-a NAME=VALUE``: set a spider argument (may be repeated)
|
||||
|
||||
* ``--output FILE`` or ``-o FILE``: append scraped items to the end of FILE (use - for stdout). To define the output format, set a colon at the end of the output URI (i.e. ``-o FILE:FORMAT``)
|
||||
* ``--output FILE`` or ``-o FILE``: append scraped items to the end of FILE
|
||||
(use ``-`` for stdout). To define the output format, add a colon at the end
|
||||
of the output URI (for example, ``-o FILE:FORMAT``)
|
||||
|
||||
* ``--overwrite-output FILE`` or ``-O FILE``: dump scraped items into FILE, overwriting any existing file. To define the output format, set a colon at the end of the output URI (i.e. ``-O FILE:FORMAT``)
|
||||
* ``--overwrite-output FILE`` or ``-O FILE``: dump scraped items into FILE,
|
||||
overwriting any existing file. To define the output format, add a colon at
|
||||
the end of the output URI (for example, ``-O FILE:FORMAT``)
|
||||
|
||||
Usage examples::
|
||||
|
||||
|
|
@ -283,10 +290,10 @@ Usage examples::
|
|||
[ ... myspider starts crawling ... ]
|
||||
|
||||
$ scrapy crawl -o myfile:csv myspider
|
||||
[ ... myspider starts crawling and appends the result to the file myfile in csv format ... ]
|
||||
[ ... myspider starts crawling and appends the result to the file myfile in CSV format ... ]
|
||||
|
||||
$ scrapy crawl -O myfile:json myspider
|
||||
[ ... myspider starts crawling and saves the result in myfile in json format overwriting the original content... ]
|
||||
[ ... myspider starts crawling and saves the result in myfile in JSON format, overwriting the original content ... ]
|
||||
|
||||
.. command:: check
|
||||
|
||||
|
|
@ -362,7 +369,7 @@ Edit the given spider using the editor defined in the ``EDITOR`` environment
|
|||
variable or (if unset) the :setting:`EDITOR` setting.
|
||||
|
||||
This command is provided only as a convenient shortcut for the most common
|
||||
case, the developer is of course free to choose any tool or IDE to write and
|
||||
case; the developer is of course free to choose any tool or IDE to write and
|
||||
debug spiders.
|
||||
|
||||
Usage example::
|
||||
|
|
@ -386,7 +393,7 @@ attribute which overrides the User Agent, it will use that one.
|
|||
|
||||
So this command can be used to "see" how your spider would fetch a certain page.
|
||||
|
||||
If used outside a project, no particular per-spider behaviour would be applied
|
||||
If used outside a project, no particular per-spider behaviour will be applied,
|
||||
and it will just use the default Scrapy downloader settings.
|
||||
|
||||
Supported options:
|
||||
|
|
@ -448,10 +455,10 @@ shell
|
|||
* Syntax: ``scrapy shell [url]``
|
||||
* Requires project: *no*
|
||||
|
||||
Starts the Scrapy shell for the given URL (if given) or empty if no URL is
|
||||
given. Also supports UNIX-style local file paths, either relative with
|
||||
``./`` or ``../`` prefixes or absolute file paths.
|
||||
See :ref:`topics-shell` for more info.
|
||||
Starts the Scrapy shell for the given URL (if provided) or leaves it empty if
|
||||
no URL is given. It also supports UNIX-style local file paths, either relative
|
||||
with ``./`` or ``../`` prefixes or absolute file paths. See :ref:`topics-shell`
|
||||
for more info.
|
||||
|
||||
Supported options:
|
||||
|
||||
|
|
@ -501,11 +508,13 @@ Supported options:
|
|||
* ``--callback`` or ``-c``: spider method to use as callback for parsing the
|
||||
response
|
||||
|
||||
* ``--meta`` or ``-m``: additional request meta that will be passed to the callback
|
||||
request. This must be a valid json string. Example: --meta='{"foo" : "bar"}'
|
||||
* ``--meta`` or ``-m``: additional request meta that will be passed to the
|
||||
callback request. This must be a valid JSON string. Example:
|
||||
``--meta='{"foo": "bar"}'``
|
||||
|
||||
* ``--cbkwargs``: additional keyword arguments that will be passed to the callback.
|
||||
This must be a valid json string. Example: --cbkwargs='{"foo" : "bar"}'
|
||||
* ``--cbkwargs``: additional keyword arguments that will be passed to the
|
||||
callback. This must be a valid JSON string. Example: ``--cbkwargs='{"foo":
|
||||
"bar"}'``
|
||||
|
||||
* ``--pipelines``: :ref:`process items through pipelines <test-item-pipeline>`
|
||||
|
||||
|
|
@ -555,8 +564,8 @@ settings
|
|||
|
||||
Get the value of a Scrapy setting.
|
||||
|
||||
If used inside a project it'll show the project setting value, otherwise it'll
|
||||
show the default Scrapy value for that setting.
|
||||
If used inside a project, it'll show the project setting value; otherwise,
|
||||
it'll show the default Scrapy value for that setting.
|
||||
|
||||
Example usage::
|
||||
|
||||
|
|
@ -621,7 +630,7 @@ spider or a special internal one:
|
|||
* :command:`view`
|
||||
|
||||
They use an internal instance of :class:`scrapy.crawler.AsyncCrawlerProcess` or
|
||||
:class:`scrapy.crawler.CrawlerProcess` for this. In most cases this detail
|
||||
:class:`scrapy.crawler.CrawlerProcess` for this. In most cases, this detail
|
||||
shouldn't matter to the user running the command, but when the user :ref:`needs
|
||||
a non-default Twisted reactor <disable-asyncio>`, it may be important.
|
||||
|
||||
|
|
@ -640,9 +649,9 @@ project-level setting is set to :ref:`the asyncio reactor <install-asyncio>`
|
|||
<default-settings>`) and :ref:`the setting of the spider being run
|
||||
<spider-settings>` is set to :ref:`a different one <disable-asyncio>`, because
|
||||
:class:`~scrapy.crawler.AsyncCrawlerProcess` only supports the asyncio reactor.
|
||||
In this case you should set the :setting:`FORCE_CRAWLER_PROCESS` setting to
|
||||
In this case, you should set the :setting:`FORCE_CRAWLER_PROCESS` setting to
|
||||
``True`` (at the project level or via the command line) so that Scrapy uses
|
||||
:class:`~scrapy.crawler.CrawlerProcess` which supports all reactors.
|
||||
:class:`~scrapy.crawler.CrawlerProcess`, which supports all reactors.
|
||||
|
||||
Custom project commands
|
||||
=======================
|
||||
|
|
@ -679,7 +688,7 @@ You can also add Scrapy commands from an external library by adding a
|
|||
``scrapy.commands`` section in the entry points of the library ``setup.py``
|
||||
file.
|
||||
|
||||
The following example adds ``my_command`` command:
|
||||
The following example adds the ``my_command`` command:
|
||||
|
||||
.. skip: next
|
||||
|
||||
|
|
|
|||
|
|
@ -80,8 +80,8 @@ For example:
|
|||
print("log is enabled!")
|
||||
|
||||
Components do not need to declare their custom settings programmatically.
|
||||
However, they should document them, so that users know they exist and how to
|
||||
use them.
|
||||
However, they should document them so that users know they exist and how to use
|
||||
them.
|
||||
|
||||
It is a good practice to prefix custom settings with the name of the component,
|
||||
to avoid collisions with custom settings of other existing (or future)
|
||||
|
|
@ -97,8 +97,8 @@ initialization, and if ``False``, raise
|
|||
:exc:`~scrapy.exceptions.NotConfigured`.
|
||||
|
||||
When choosing a name for a custom setting, it is also a good idea to have a
|
||||
look at the names of :ref:`built-in settings <topics-settings-ref>`, to try to
|
||||
maintain consistency with them.
|
||||
look at the names of :ref:`built-in settings <topics-settings-ref>` to maintain
|
||||
consistency with them.
|
||||
|
||||
.. _enforce-component-requirements:
|
||||
|
||||
|
|
@ -118,7 +118,7 @@ In the case of :ref:`downloader middlewares <topics-downloader-middleware>`,
|
|||
<topics-item-pipeline>`, and :ref:`spider middlewares
|
||||
<topics-spider-middleware>`, you should raise
|
||||
:exc:`~scrapy.exceptions.NotConfigured`, passing a description of the issue as
|
||||
a parameter to the exception so that it is printed in the logs, for the user to
|
||||
a parameter to the exception so that it is printed in the logs for the user to
|
||||
see. For other components, feel free to raise whatever other exception feels
|
||||
right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy
|
||||
version mismatch, while :exc:`ValueError` may be better if the issue is the
|
||||
|
|
@ -139,7 +139,7 @@ If your requirement is a minimum Scrapy version, you may use
|
|||
if parse_version(scrapy.__version__) < parse_version("2.7"):
|
||||
raise RuntimeError(
|
||||
f"{MyComponent.__qualname__} requires Scrapy 2.7 or "
|
||||
f"later, which allow defining the process_spider_output "
|
||||
f"later, which allows defining the process_spider_output "
|
||||
f"method of spider middlewares as an asynchronous "
|
||||
f"generator."
|
||||
)
|
||||
|
|
@ -151,7 +151,7 @@ The following function can be used to create an instance of a component class:
|
|||
|
||||
.. autofunction:: scrapy.utils.misc.build_from_crawler
|
||||
|
||||
The following function can also be useful when implementing a component, to
|
||||
The following function can also be useful when implementing a component to
|
||||
report the import path of the component class, e.g. when reporting problems:
|
||||
|
||||
.. autofunction:: scrapy.utils.python.global_object_name
|
||||
|
|
|
|||
|
|
@ -1,16 +1,16 @@
|
|||
.. _topics-contracts:
|
||||
|
||||
=================
|
||||
Spiders Contracts
|
||||
=================
|
||||
================
|
||||
Spider Contracts
|
||||
================
|
||||
|
||||
Testing spiders can get particularly annoying and while nothing prevents you
|
||||
from writing unit tests the task gets cumbersome quickly. Scrapy offers an
|
||||
Testing spiders can get particularly annoying, and while nothing prevents you
|
||||
from writing unit tests, the task gets cumbersome quickly. Scrapy offers an
|
||||
integrated way of testing your spiders by the means of contracts.
|
||||
|
||||
This allows you to test each callback of your spider by hardcoding a sample url
|
||||
and check various constraints for how the callback processes the response. Each
|
||||
contract is prefixed with an ``@`` and included in the docstring. See the
|
||||
This allows you to test each callback of your spider by hardcoding a sample URL
|
||||
and checking various constraints for how the callback processes the response.
|
||||
Each contract is prefixed with an ``@`` and included in the docstring. See the
|
||||
following example:
|
||||
|
||||
.. code-block:: python
|
||||
|
|
@ -45,7 +45,7 @@ Use the :command:`check` command to run the contract checks.
|
|||
Custom Contracts
|
||||
================
|
||||
|
||||
If you find you need more power than the built-in Scrapy contracts you can
|
||||
If you find you need more power than the built-in Scrapy contracts, you can
|
||||
create and load your own contracts in the project by using the
|
||||
:setting:`SPIDER_CONTRACTS` setting:
|
||||
|
||||
|
|
@ -68,7 +68,7 @@ override three methods:
|
|||
.. method:: pre_process(response)
|
||||
|
||||
This allows hooking in various checks on the response received from the
|
||||
sample request, before it's being passed to the callback.
|
||||
sample request, before it is passed to the callback.
|
||||
|
||||
.. method:: post_process(output)
|
||||
|
||||
|
|
@ -92,10 +92,8 @@ response received:
|
|||
|
||||
|
||||
class HasHeaderContract(Contract):
|
||||
"""
|
||||
Demo contract which checks the presence of a custom header
|
||||
@has_header X-CustomHeader
|
||||
"""
|
||||
"""Demo contract that checks the presence of a custom header:
|
||||
@has_header X-CustomHeader"""
|
||||
|
||||
name = "has_header"
|
||||
|
||||
|
|
@ -110,8 +108,8 @@ Detecting check runs
|
|||
====================
|
||||
|
||||
When ``scrapy check`` is running, the ``SCRAPY_CHECK`` environment variable is
|
||||
set to the ``true`` string. You can use :data:`os.environ` to perform any change to
|
||||
your spiders or your settings when ``scrapy check`` is used:
|
||||
set to the ``true`` string. You can use :data:`os.environ` to make any changes
|
||||
to your spiders or your settings when ``scrapy check`` is used:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -60,7 +60,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
|||
Using Deferred-based APIs
|
||||
=========================
|
||||
|
||||
In addition to native coroutine APIs Scrapy has some APIs that return a
|
||||
In addition to native coroutine APIs, Scrapy has some APIs that return a
|
||||
:class:`~twisted.internet.defer.Deferred` object or take a user-supplied
|
||||
function that returns a :class:`~twisted.internet.defer.Deferred` object. These
|
||||
APIs are also asynchronous but don't yet support native ``async def`` syntax.
|
||||
|
|
@ -121,7 +121,7 @@ return coroutines are listed in :ref:`coroutine-support`):
|
|||
|
||||
- ``stat_file()``
|
||||
|
||||
In most cases you can use these APIs in code that otherwise uses coroutines, by
|
||||
In most cases, you can use these APIs in code that otherwise uses coroutines by
|
||||
wrapping a :class:`~twisted.internet.defer.Deferred` object into a
|
||||
:class:`~asyncio.Future` object or vice versa. See :ref:`asyncio-await-dfd` for
|
||||
more information about this.
|
||||
|
|
@ -174,8 +174,8 @@ becomes:
|
|||
|
||||
Coroutines may be used to call asynchronous code. This includes other
|
||||
coroutines, functions that return Deferreds and functions that return
|
||||
:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`.
|
||||
This means you can use many useful Python libraries providing such code:
|
||||
:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`. This
|
||||
means you can use many useful Python libraries that provide such code:
|
||||
|
||||
.. skip: next
|
||||
.. code-block:: python
|
||||
|
|
@ -197,8 +197,8 @@ This means you can use many useful Python libraries providing such code:
|
|||
# ... use response and additional_data to yield items and requests
|
||||
|
||||
.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
|
||||
:mod:`asyncio` loop and to use them you need to
|
||||
:doc:`enable asyncio support in Scrapy<asyncio>`.
|
||||
:mod:`asyncio` loop, and to use them you need to :doc:`enable asyncio
|
||||
support in Scrapy<asyncio>`.
|
||||
|
||||
.. note:: If you want to ``await`` on Deferreds while using the asyncio reactor,
|
||||
you need to :ref:`wrap them<asyncio-await-dfd>`.
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ Debugging Spiders
|
|||
=================
|
||||
|
||||
This document explains the most common techniques for debugging spiders.
|
||||
Consider the following Scrapy spider below:
|
||||
Consider the following Scrapy spider:
|
||||
|
||||
.. skip: next
|
||||
.. code-block:: python
|
||||
|
|
@ -40,19 +40,19 @@ Consider the following Scrapy spider below:
|
|||
# populate more `item` fields
|
||||
return item
|
||||
|
||||
Basically this is a simple spider which parses two pages of items (the
|
||||
start_urls). Items also have a details page with additional information, so we
|
||||
use the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a
|
||||
partially populated item.
|
||||
Basically, this is a simple spider that parses two pages of items (the start
|
||||
URLs). Items also have a details page with additional information, so we use
|
||||
the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a partially
|
||||
populated item.
|
||||
|
||||
|
||||
Parse Command
|
||||
=============
|
||||
|
||||
The most basic way of checking the output of your spider is to use the
|
||||
:command:`parse` command. It allows to check the behaviour of different parts
|
||||
of the spider at the method level. It has the advantage of being flexible and
|
||||
simple to use, but does not allow debugging code inside a method.
|
||||
:command:`parse` command. It allows you to check the behaviour of different
|
||||
parts of the spider at the method level. It has the advantage of being flexible
|
||||
and simple to use, but it does not allow debugging code inside a method.
|
||||
|
||||
.. highlight:: none
|
||||
|
||||
|
|
@ -90,7 +90,7 @@ Using the ``--verbose`` or ``-v`` option we can see the status at each depth lev
|
|||
# Requests -----------------------------------------------------------------
|
||||
[]
|
||||
|
||||
Checking items scraped from a single start_url, can also be easily achieved
|
||||
Checking items scraped from a single start URL can also be easily achieved
|
||||
using::
|
||||
|
||||
$ scrapy parse --spider=myspider -d 3 'http://example.com/page1'
|
||||
|
|
@ -101,10 +101,10 @@ using::
|
|||
Scrapy Shell
|
||||
============
|
||||
|
||||
While the :command:`parse` command is very useful for checking behaviour of a
|
||||
spider, it is of little help to check what happens inside a callback, besides
|
||||
showing the response received and the output. How to debug the situation when
|
||||
``parse_details`` sometimes receives no item?
|
||||
While the :command:`parse` command is very useful for checking the behaviour of
|
||||
a spider, it is of little help when checking what happens inside a callback
|
||||
besides showing the response received and the output. How do you debug the
|
||||
situation when ``parse_details`` sometimes receives no item?
|
||||
|
||||
.. highlight:: python
|
||||
|
||||
|
|
@ -129,7 +129,7 @@ See also: :ref:`topics-shell-inspect-response`.
|
|||
Open in browser
|
||||
===============
|
||||
|
||||
Sometimes you just want to see how a certain response looks in a browser, you
|
||||
Sometimes you just want to see how a certain response looks in a browser; you
|
||||
can use the :func:`~scrapy.utils.response.open_in_browser` function for that:
|
||||
|
||||
.. autofunction:: scrapy.utils.response.open_in_browser
|
||||
|
|
@ -140,7 +140,7 @@ Logging
|
|||
|
||||
Logging is another useful option for getting information about your spider run.
|
||||
Although not as convenient, it comes with the advantage that the logs will be
|
||||
available in all future runs should they be necessary again:
|
||||
available in all future runs should you need them again:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -24,8 +24,8 @@ Deploying to a Scrapyd Server
|
|||
`Scrapyd`_ is an open source application to run Scrapy spiders. It provides
|
||||
a server with HTTP API, capable of running and monitoring Scrapy spiders.
|
||||
|
||||
To deploy spiders to Scrapyd, you can use the scrapyd-deploy tool provided by
|
||||
the `scrapyd-client`_ package. Please refer to the `scrapyd-deploy
|
||||
To deploy spiders to Scrapyd, you can use the ``scrapyd-deploy`` tool provided
|
||||
by the `scrapyd-client`_ package. Please refer to the `scrapyd-deploy
|
||||
documentation`_ for more information.
|
||||
|
||||
Scrapyd is maintained by some of the Scrapy developers.
|
||||
|
|
@ -38,12 +38,12 @@ Deploying to Zyte Scrapy Cloud
|
|||
`Zyte Scrapy Cloud`_ is a hosted, cloud-based service by Zyte_, the company
|
||||
behind Scrapy.
|
||||
|
||||
Zyte Scrapy Cloud removes the need to setup and monitor servers and provides a
|
||||
Zyte Scrapy Cloud removes the need to set up and monitor servers and provides a
|
||||
nice UI to manage spiders and review scraped items, logs and stats.
|
||||
|
||||
To deploy spiders to Zyte Scrapy Cloud you can use the `shub`_ command line
|
||||
tool.
|
||||
Please refer to the `Zyte Scrapy Cloud documentation`_ for more information.
|
||||
To deploy spiders to Zyte Scrapy Cloud, you can use the ``shub`` command-line
|
||||
tool. Please refer to the `Zyte Scrapy Cloud documentation`_ for more
|
||||
information.
|
||||
|
||||
Zyte Scrapy Cloud is compatible with Scrapyd and one can switch between
|
||||
them as needed - the configuration is read from the ``scrapy.cfg`` file
|
||||
|
|
|
|||
|
|
@ -4,10 +4,10 @@
|
|||
Using your browser's Developer Tools for scraping
|
||||
=================================================
|
||||
|
||||
Here is a general guide on how to use your browser's Developer Tools
|
||||
to ease the scraping process. Today almost all browsers come with
|
||||
built in `Developer Tools`_ and although we will use Firefox in this
|
||||
guide, the concepts are applicable to any other browser.
|
||||
Here is a general guide on how to use your browser's Developer Tools to ease
|
||||
the scraping process. Today almost all browsers come with built-in `Developer
|
||||
Tools`_, and although we will use Firefox in this guide, the concepts are
|
||||
applicable to any other browser.
|
||||
|
||||
In this guide we'll introduce the basic tools to use from a browser's
|
||||
Developer Tools by scraping `quotes.toscrape.com`_.
|
||||
|
|
@ -19,9 +19,9 @@ Caveats with inspecting the live browser DOM
|
|||
|
||||
Since Developer Tools operate on a live browser DOM, what you'll actually see
|
||||
when inspecting the page source is not the original HTML, but a modified one
|
||||
after applying some browser clean up and executing JavaScript code. Firefox,
|
||||
in particular, is known for adding ``<tbody>`` elements to tables. Scrapy, on
|
||||
the other hand, does not modify the original page HTML, so you won't be able to
|
||||
after applying some browser clean up and executing JavaScript code. Firefox, in
|
||||
particular, is known for adding ``<tbody>`` elements to tables. Scrapy, on the
|
||||
other hand, does not modify the original page HTML, so you won't be able to
|
||||
extract any data if you use ``<tbody>`` in your XPath expressions.
|
||||
|
||||
Therefore, you should keep in mind the following things:
|
||||
|
|
@ -42,17 +42,16 @@ Inspecting a website
|
|||
====================
|
||||
|
||||
By far the most handy feature of the Developer Tools is the `Inspector`
|
||||
feature, which allows you to inspect the underlying HTML code of
|
||||
any webpage. To demonstrate the Inspector, let's look at the
|
||||
`quotes.toscrape.com`_-site.
|
||||
feature, which allows you to inspect the underlying HTML code of any webpage.
|
||||
To demonstrate the Inspector, let's look at the `quotes.toscrape.com`_ site.
|
||||
|
||||
On the site we have a total of ten quotes from various authors with specific
|
||||
tags, as well as the Top Ten Tags. Let's say we want to extract all the quotes
|
||||
on this page, without any meta-information about authors, tags, etc.
|
||||
|
||||
Instead of viewing the whole source code for the page, we can simply right click
|
||||
on a quote and select ``Inspect Element (Q)``, which opens up the `Inspector`.
|
||||
In it you should see something like this:
|
||||
Instead of viewing the whole source code for the page, we can simply
|
||||
right-click on a quote and select ``Inspect Element (Q)``, which opens up the
|
||||
`Inspector`. In it you should see something like this:
|
||||
|
||||
.. image:: _images/inspector_01.png
|
||||
:width: 777
|
||||
|
|
@ -76,10 +75,10 @@ anywhere.
|
|||
|
||||
The advantage of the `Inspector` is that it automatically expands and collapses
|
||||
sections and tags of a webpage, which greatly improves readability. You can
|
||||
expand and collapse a tag by clicking on the arrow in front of it or by double
|
||||
clicking directly on the tag. If we expand the ``span`` tag with the ``class=
|
||||
"text"`` we will see the quote-text we clicked on. The `Inspector` lets you
|
||||
copy XPaths to selected elements. Let's try it out.
|
||||
expand and collapse a tag by clicking on the arrow in front of it or by
|
||||
double-clicking directly on the tag. If we expand the ``span`` tag with the
|
||||
``class="text"`` attribute, we see the quote text we clicked on. The
|
||||
`Inspector` lets you copy XPaths to selected elements. Let's try it out.
|
||||
|
||||
First open the Scrapy shell at https://quotes.toscrape.com/ in a terminal:
|
||||
|
||||
|
|
@ -99,17 +98,17 @@ Then, back to your web browser, right-click on the ``span`` tag, select
|
|||
>>> response.xpath("/html/body/div/div[2]/div[1]/div[1]/span[1]/text()").getall()
|
||||
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
|
||||
|
||||
Adding ``text()`` at the end we are able to extract the first quote with this
|
||||
basic selector. But this XPath is not really that clever. All it does is
|
||||
go down a desired path in the source code starting from ``html``. So let's
|
||||
see if we can refine our XPath a bit:
|
||||
Adding ``text()`` at the end, we are able to extract the first quote with this
|
||||
basic selector. But this XPath is not really that clever. All it does is go
|
||||
down a desired path in the source code starting from ``html``. So let's see if
|
||||
we can refine our XPath a bit:
|
||||
|
||||
If we check the `Inspector` again we'll see that directly beneath our
|
||||
expanded ``div`` tag we have nine identical ``div`` tags, each with the
|
||||
same attributes as our first. If we expand any of them, we'll see the same
|
||||
structure as with our first quote: Two ``span`` tags and one ``div`` tag. We can
|
||||
expand each ``span`` tag with the ``class="text"`` inside our ``div`` tags and
|
||||
see each quote:
|
||||
If we check the `Inspector` again we'll see that directly beneath our expanded
|
||||
``div`` tag we have nine identical ``div`` tags, each with the same attributes
|
||||
as our first. If we expand any of them, we'll see the same structure as with
|
||||
our first quote: two ``span`` tags and one ``div`` tag. We can expand each
|
||||
``span`` tag with the ``class="text"`` inside our ``div`` tags and see each
|
||||
quote:
|
||||
|
||||
.. code-block:: html
|
||||
|
||||
|
|
@ -145,14 +144,14 @@ source code or directly scrolling to an element you selected. Let's demonstrate
|
|||
a use case:
|
||||
|
||||
Say you want to find the ``Next`` button on the page. Type ``Next`` into the
|
||||
search bar on the top right of the `Inspector`. You should get two results.
|
||||
The first is a ``li`` tag with the ``class="next"``, the second the text
|
||||
of an ``a`` tag. Right click on the ``a`` tag and select ``Scroll into View``.
|
||||
If you hover over the tag, you'll see the button highlighted. From here
|
||||
we could easily create a :ref:`Link Extractor <topics-link-extractors>` to
|
||||
follow the pagination. On a simple site such as this, there may not be
|
||||
the need to find an element visually but the ``Scroll into View`` function
|
||||
can be quite useful on complex sites.
|
||||
search bar on the top-right of the `Inspector`. You should get two results. The
|
||||
first is a ``li`` tag with the ``class="next"``, the second the text of an
|
||||
``a`` tag. Right-click on the ``a`` tag and select ``Scroll into View``. If you
|
||||
hover over the tag, you'll see the button highlighted. From here we could
|
||||
easily create a :ref:`Link Extractor <topics-link-extractors>` to follow the
|
||||
pagination. On a simple site such as this, there may not be the need to find an
|
||||
element visually but the ``Scroll into View`` function can be quite useful on
|
||||
complex sites.
|
||||
|
||||
Note that the search bar can also be used to search for and test CSS
|
||||
selectors. For example, you could search for ``span.text`` to find
|
||||
|
|
@ -161,19 +160,19 @@ exactly the ``span`` tag with the ``class="text"`` in the page.
|
|||
|
||||
.. _topics-network-tool:
|
||||
|
||||
The Network-tool
|
||||
The Network tool
|
||||
================
|
||||
While scraping you may come across dynamic webpages where some parts
|
||||
of the page are loaded dynamically through multiple requests. While
|
||||
this can be quite tricky, the `Network`-tool in the Developer Tools
|
||||
greatly facilitates this task. To demonstrate the Network-tool, let's
|
||||
take a look at the page `quotes.toscrape.com/scroll`_.
|
||||
While scraping you may come across dynamic webpages where some parts of the
|
||||
page are loaded dynamically through multiple requests. While this can be quite
|
||||
tricky, the `Network` tool in the Developer Tools greatly facilitates this
|
||||
task. To demonstrate the Network tool, let's take a look at the page
|
||||
`quotes.toscrape.com/scroll`_.
|
||||
|
||||
The page is quite similar to the basic `quotes.toscrape.com`_-page,
|
||||
but instead of the above-mentioned ``Next`` button, the page
|
||||
automatically loads new quotes when you scroll to the bottom. We
|
||||
could go ahead and try out different XPaths directly, but instead
|
||||
we'll check another quite useful command from the Scrapy shell:
|
||||
The page is quite similar to the basic `quotes.toscrape.com`_ page, but instead
|
||||
of the above-mentioned ``Next`` button, the page automatically loads new quotes
|
||||
when you scroll to the bottom. We could go ahead and try out different XPaths
|
||||
directly, but instead we'll check another quite useful command from the Scrapy
|
||||
shell:
|
||||
|
||||
.. skip: next
|
||||
|
||||
|
|
@ -192,12 +191,11 @@ bar with the word ``Loading...``.
|
|||
:height: 296
|
||||
:alt: Response from quotes.toscrape.com/scroll
|
||||
|
||||
The ``view(response)`` command let's us view the response our
|
||||
shell or later our spider receives from the server. Here we see
|
||||
that some basic template is loaded which includes the title,
|
||||
the login-button and the footer, but the quotes are missing. This
|
||||
tells us that the quotes are being loaded from a different request
|
||||
than ``quotes.toscrape/scroll``.
|
||||
The ``view(response)`` command lets us view the response our shell or later our
|
||||
spider receives from the server. Here we see that some basic template is loaded
|
||||
which includes the title, the login button and the footer, but the quotes are
|
||||
missing. This tells us that the quotes are being loaded from a different
|
||||
request than ``quotes.toscrape.com/scroll``.
|
||||
|
||||
If you click on the ``Network`` tab, you will probably only see
|
||||
two entries. The first thing we do is enable persistent logs by
|
||||
|
|
@ -218,30 +216,29 @@ Here we see every request that has been made when reloading the page
|
|||
and can inspect each request and its response. So let's find out
|
||||
where our quotes are coming from:
|
||||
|
||||
First click on the request with the name ``scroll``. On the right
|
||||
you can now inspect the request. In ``Headers`` you'll find details
|
||||
about the request headers, such as the URL, the method, the IP-address,
|
||||
and so on. We'll ignore the other tabs and click directly on ``Response``.
|
||||
First click on the request with the name ``scroll``. On the right you can now
|
||||
inspect the request. In ``Headers`` you'll find details about the request
|
||||
headers, such as the URL, the method, the IP address, and so on. We'll ignore
|
||||
the other tabs and click directly on ``Response``.
|
||||
|
||||
What you should see in the ``Preview`` pane is the rendered HTML-code,
|
||||
that is exactly what we saw when we called ``view(response)`` in the
|
||||
shell. Accordingly the ``type`` of the request in the log is ``html``.
|
||||
The other requests have types like ``css`` or ``js``, but what
|
||||
interests us is the one request called ``quotes?page=1`` with the
|
||||
type ``json``.
|
||||
What you should see in the ``Preview`` pane is the rendered HTML code, exactly
|
||||
what we saw when we called ``view(response)`` in the shell. Accordingly, the
|
||||
``type`` of the request in the log is ``html``. The other requests have types
|
||||
like ``css`` or ``js``, but what interests us is the one request called
|
||||
``quotes?page=1`` with the type ``json``.
|
||||
|
||||
If we click on this request, we see that the request URL is
|
||||
``https://quotes.toscrape.com/api/quotes?page=1`` and the response
|
||||
is a JSON-object that contains our quotes. We can also right-click
|
||||
on the request and open ``Open in new tab`` to get a better overview.
|
||||
``https://quotes.toscrape.com/api/quotes?page=1`` and the response is a JSON
|
||||
object that contains our quotes. We can also right-click on the request and
|
||||
open ``Open in new tab`` to get a better overview.
|
||||
|
||||
.. image:: _images/network_03.png
|
||||
:width: 777
|
||||
:height: 375
|
||||
:alt: JSON-object returned from the quotes.toscrape API
|
||||
:alt: JSON object returned from the quotes.toscrape API
|
||||
|
||||
With this response we can now easily parse the JSON-object and
|
||||
also request each page to get every quote on the site:
|
||||
With this response we can now easily parse the JSON object and also request
|
||||
each page to get every quote on the site:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -263,15 +260,14 @@ also request each page to get every quote on the site:
|
|||
url = f"https://quotes.toscrape.com/api/quotes?page={self.page}"
|
||||
yield scrapy.Request(url=url, callback=self.parse)
|
||||
|
||||
This spider starts at the first page of the quotes-API. With each
|
||||
response, we parse the ``response.text`` and assign it to ``data``.
|
||||
This lets us operate on the JSON-object like on a Python dictionary.
|
||||
We iterate through the ``quotes`` and print out the ``quote["text"]``.
|
||||
If the handy ``has_next`` element is ``true`` (try loading
|
||||
`quotes.toscrape.com/api/quotes?page=10`_ in your browser or a
|
||||
page-number greater than 10), we increment the ``page`` attribute
|
||||
and ``yield`` a new request, inserting the incremented page-number
|
||||
into our ``url``.
|
||||
This spider starts at the first page of the quotes-API. With each response, we
|
||||
parse the ``response.text`` and assign it to ``data``. This lets us operate on
|
||||
the JSON object like on a Python dictionary. We iterate through the ``quotes``
|
||||
and print out the ``quote["text"]``. If the handy ``has_next`` element is
|
||||
``true`` (try loading `quotes.toscrape.com/api/quotes?page=10`_ in your browser
|
||||
or a page number greater than 10), we increment the ``page`` attribute and
|
||||
``yield`` a new request, inserting the incremented page number into our
|
||||
``url``.
|
||||
|
||||
.. _requests-from-curl:
|
||||
|
||||
|
|
@ -305,11 +301,11 @@ function to get a dictionary with the equivalent arguments:
|
|||
Note that to translate a cURL command into a Scrapy request,
|
||||
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
||||
|
||||
As you can see, with a few inspections in the `Network`-tool we
|
||||
were able to easily replicate the dynamic requests of the scrolling
|
||||
functionality of the page. Crawling dynamic pages can be quite
|
||||
daunting and pages can be very complex, but it (mostly) boils down
|
||||
to identifying the correct request and replicating it in your spider.
|
||||
As you can see, with a few inspections in the `Network` tool we were able to
|
||||
easily replicate the dynamic requests of the scrolling functionality of the
|
||||
page. Crawling dynamic pages can be quite daunting and pages can be very
|
||||
complex, but it (mostly) boils down to identifying the correct request and
|
||||
replicating it in your spider.
|
||||
|
||||
.. _Developer Tools: https://en.wikipedia.org/wiki/Web_development_tools
|
||||
.. _quotes.toscrape.com: https://quotes.toscrape.com
|
||||
|
|
|
|||
|
|
@ -85,16 +85,16 @@ It might be enough to yield a :class:`~scrapy.Request` with the same HTTP
|
|||
method and URL. However, you may also need to reproduce the body, headers and
|
||||
form parameters (see :ref:`form`) of that request.
|
||||
|
||||
As all major browsers allow to export the requests in curl_ format, Scrapy
|
||||
incorporates the method :meth:`~scrapy.Request.from_curl` to generate an equivalent
|
||||
:class:`~scrapy.Request` from a cURL command. To get more information
|
||||
visit :ref:`request from curl <requests-from-curl>` inside the network
|
||||
tool section.
|
||||
As all major browsers allow you to export requests in curl_ format, Scrapy
|
||||
incorporates the method :meth:`~scrapy.Request.from_curl` to generate an
|
||||
equivalent :class:`~scrapy.Request` from a cURL command. To get more
|
||||
information visit :ref:`request from curl <requests-from-curl>` inside the
|
||||
network tool section.
|
||||
|
||||
Once you get the expected response, you can :ref:`extract the desired data from
|
||||
it <topics-handling-response-formats>`.
|
||||
|
||||
You can reproduce any request with Scrapy. However, some times reproducing all
|
||||
You can reproduce any request with Scrapy. However, sometimes reproducing all
|
||||
necessary requests may not seem efficient in developer time. If that is your
|
||||
case, and crawling speed is not a major concern for you, you can alternatively
|
||||
consider :ref:`using a headless browser <topics-headless-browsing>`.
|
||||
|
|
@ -271,8 +271,8 @@ The following is a simple snippet to illustrate its usage within a Scrapy spider
|
|||
|
||||
|
||||
However, using `playwright-python`_ directly as in the above example
|
||||
circumvents most of the Scrapy components (middlewares, dupefilter, etc).
|
||||
We recommend using `scrapy-playwright`_ for a better integration.
|
||||
circumvents most of the Scrapy components (middlewares, dupefilter, etc.). We
|
||||
recommend using `scrapy-playwright`_ for a better integration.
|
||||
|
||||
.. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets
|
||||
.. _chompjs: https://github.com/Nykakin/chompjs
|
||||
|
|
|
|||
|
|
@ -11,34 +11,34 @@ Once you have scraped your items, you often want to persist or export those
|
|||
items, to use the data in some other application. That is, after all, the whole
|
||||
purpose of the scraping process.
|
||||
|
||||
For this purpose Scrapy provides a collection of Item Exporters for different
|
||||
For this purpose, Scrapy provides a collection of Item Exporters for different
|
||||
output formats, such as XML, CSV or JSON.
|
||||
|
||||
Using Item Exporters
|
||||
====================
|
||||
|
||||
If you are in a hurry, and just want to use an Item Exporter to output scraped
|
||||
data see the :ref:`topics-feed-exports`. Otherwise, if you want to know how
|
||||
data, see the :ref:`topics-feed-exports`. Otherwise, if you want to know how
|
||||
Item Exporters work or need more custom functionality (not covered by the
|
||||
default exports), continue reading below.
|
||||
|
||||
In order to use an Item Exporter, you must instantiate it with its required
|
||||
args. Each Item Exporter requires different arguments, so check each exporter
|
||||
documentation to be sure, in :ref:`topics-exporters-reference`. After you have
|
||||
In order to use an Item Exporter, you must instantiate it with its required
|
||||
args. Each Item Exporter requires different arguments, so check each exporter's
|
||||
documentation in :ref:`topics-exporters-reference` to be sure. After you have
|
||||
instantiated your exporter, you have to:
|
||||
|
||||
1. call the method :meth:`~BaseItemExporter.start_exporting` in order to
|
||||
signal the beginning of the exporting process
|
||||
1. Call the method :meth:`~BaseItemExporter.start_exporting` in order to
|
||||
signal the beginning of the exporting process
|
||||
|
||||
2. call the :meth:`~BaseItemExporter.export_item` method for each item you want
|
||||
to export
|
||||
2. Call the :meth:`~BaseItemExporter.export_item` method for each item you want
|
||||
to export
|
||||
|
||||
3. and finally call the :meth:`~BaseItemExporter.finish_exporting` to signal
|
||||
the end of the exporting process
|
||||
3. Finally, call the :meth:`~BaseItemExporter.finish_exporting` method to
|
||||
signal the end of the exporting process
|
||||
|
||||
Here you can see an :doc:`Item Pipeline <item-pipeline>` which uses multiple
|
||||
Item Exporters to group scraped items to different files according to the
|
||||
value of one of their fields:
|
||||
Here you can see an :doc:`Item Pipeline <item-pipeline>` that uses multiple
|
||||
Item Exporters to distribute scraped items into different files according to
|
||||
the value of one of their fields:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -224,9 +224,10 @@ BaseItemExporter
|
|||
defining what fields to export, whether to export empty fields, or which
|
||||
encoding to use.
|
||||
|
||||
These features can be configured through the ``__init__`` method arguments which
|
||||
populate their respective instance attributes: :attr:`fields_to_export`,
|
||||
:attr:`export_empty_fields`, :attr:`encoding`, :attr:`indent`.
|
||||
These features can be configured through the ``__init__`` method arguments,
|
||||
which populate their respective instance attributes:
|
||||
:attr:`fields_to_export`, :attr:`export_empty_fields`, :attr:`encoding`,
|
||||
:attr:`indent`.
|
||||
|
||||
.. method:: export_item(item)
|
||||
|
||||
|
|
@ -341,8 +342,9 @@ XmlItemExporter
|
|||
|
||||
Exports items in XML format to the specified file object.
|
||||
|
||||
:param file: the file-like object to use for exporting the data. Its ``write`` method should
|
||||
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
|
||||
:param file: the file-like object to use for exporting the data. Its
|
||||
``write`` method should accept ``bytes`` (a disk file opened in binary
|
||||
mode, an ``io.BytesIO`` object, etc)
|
||||
|
||||
:param root_element: The name of root element in the exported XML.
|
||||
:type root_element: str
|
||||
|
|
@ -381,7 +383,7 @@ XmlItemExporter
|
|||
|
||||
Item(name=["John", "Doe"], age="23")
|
||||
|
||||
Would be serialized as:
|
||||
It would be serialized as:
|
||||
|
||||
.. code-block:: xml
|
||||
|
||||
|
|
@ -406,8 +408,9 @@ CsvItemExporter
|
|||
CSV columns, their order and their column names. The
|
||||
:attr:`export_empty_fields` attribute has no effect on this exporter.
|
||||
|
||||
:param file: the file-like object to use for exporting the data. Its ``write`` method should
|
||||
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
|
||||
:param file: the file-like object to use for exporting the data. Its
|
||||
``write`` method should accept ``bytes`` (a disk file opened in binary
|
||||
mode, an ``io.BytesIO`` object, etc)
|
||||
|
||||
:param include_headers_line: If enabled, makes the exporter output a header
|
||||
line with the field names taken from
|
||||
|
|
@ -441,8 +444,9 @@ PickleItemExporter
|
|||
|
||||
Exports items in pickle format to the given file-like object.
|
||||
|
||||
:param file: the file-like object to use for exporting the data. Its ``write`` method should
|
||||
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
|
||||
:param file: the file-like object to use for exporting the data. Its
|
||||
``write`` method should accept ``bytes`` (a disk file opened in binary
|
||||
mode, an ``io.BytesIO`` object, etc)
|
||||
|
||||
:param protocol: The pickle protocol to use.
|
||||
:type protocol: int
|
||||
|
|
@ -452,7 +456,7 @@ PickleItemExporter
|
|||
The additional keyword arguments of this ``__init__`` method are passed to the
|
||||
:class:`BaseItemExporter` ``__init__`` method.
|
||||
|
||||
Pickle isn't a human readable format, so no output examples are provided.
|
||||
Pickle isn't a human-readable format, so no output examples are provided.
|
||||
|
||||
PprintItemExporter
|
||||
------------------
|
||||
|
|
@ -461,8 +465,9 @@ PprintItemExporter
|
|||
|
||||
Exports items in pretty print format to the specified file object.
|
||||
|
||||
:param file: the file-like object to use for exporting the data. Its ``write`` method should
|
||||
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
|
||||
:param file: the file-like object to use for exporting the data. Its
|
||||
``write`` method should accept ``bytes`` (a disk file opened in binary
|
||||
mode, an ``io.BytesIO`` object, etc)
|
||||
|
||||
The additional keyword arguments of this ``__init__`` method are passed to the
|
||||
:class:`BaseItemExporter` ``__init__`` method.
|
||||
|
|
@ -487,8 +492,9 @@ JsonItemExporter
|
|||
arguments to the :class:`~json.JSONEncoder` ``__init__`` method, so you can use any
|
||||
:class:`~json.JSONEncoder` ``__init__`` method argument to customize this exporter.
|
||||
|
||||
:param file: the file-like object to use for exporting the data. Its ``write`` method should
|
||||
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
|
||||
:param file: the file-like object to use for exporting the data. Its
|
||||
``write`` method should accept ``bytes`` (a disk file opened in binary
|
||||
mode, an ``io.BytesIO`` object, etc)
|
||||
|
||||
A typical output of this exporter would be:
|
||||
|
||||
|
|
@ -499,10 +505,10 @@ JsonItemExporter
|
|||
|
||||
.. _json-with-large-data:
|
||||
|
||||
.. warning:: JSON is very simple and flexible serialization format, but it
|
||||
doesn't scale well for large amounts of data since incremental (aka.
|
||||
.. warning:: JSON is a very simple and flexible serialization format, but it
|
||||
doesn't scale well for large amounts of data because incremental (aka
|
||||
stream-mode) parsing is not well supported (if at all) among JSON parsers
|
||||
(on any language), and most of them just parse the entire object in
|
||||
in any language, and most of them just parse the entire object in
|
||||
memory. If you want the power and simplicity of JSON with a more
|
||||
stream-friendly format, consider using :class:`JsonLinesItemExporter`
|
||||
instead, or splitting the output in multiple chunks.
|
||||
|
|
@ -518,8 +524,9 @@ JsonLinesItemExporter
|
|||
the :class:`~json.JSONEncoder` ``__init__`` method, so you can use any
|
||||
:class:`~json.JSONEncoder` ``__init__`` method argument to customize this exporter.
|
||||
|
||||
:param file: the file-like object to use for exporting the data. Its ``write`` method should
|
||||
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
|
||||
:param file: the file-like object to use for exporting the data. Its
|
||||
``write`` method should accept ``bytes`` (a disk file opened in binary
|
||||
mode, an ``io.BytesIO`` object, etc)
|
||||
|
||||
A typical output of this exporter would be:
|
||||
|
||||
|
|
|
|||
|
|
@ -48,8 +48,8 @@ tasks triggered by them.
|
|||
Sample extension
|
||||
----------------
|
||||
|
||||
Here we will implement a simple extension to illustrate the concepts described
|
||||
in the previous section. This extension will log a message every time:
|
||||
Here we implement a simple extension to illustrate the concepts described in
|
||||
the previous section. This extension will log a message every time:
|
||||
|
||||
* a spider is opened
|
||||
* a spider is closed
|
||||
|
|
@ -58,7 +58,7 @@ in the previous section. This extension will log a message every time:
|
|||
The extension will be enabled through the ``MYEXT_ENABLED`` setting and the
|
||||
number of items will be specified through the ``MYEXT_ITEMCOUNT`` setting.
|
||||
|
||||
Here is the code of such extension:
|
||||
Here is the code for that extension:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -163,7 +163,7 @@ Provides a telnet console for getting into a Python interpreter inside the
|
|||
currently running Scrapy process, which can be very useful for debugging.
|
||||
|
||||
The telnet console must be enabled by the :setting:`TELNETCONSOLE_ENABLED`
|
||||
setting, and the server will listen in the port specified in
|
||||
setting, and the server will listen on the port specified in
|
||||
:setting:`TELNETCONSOLE_PORT`.
|
||||
|
||||
.. _topics-extensions-ref-memusage:
|
||||
|
|
@ -203,7 +203,8 @@ Memory debugger extension
|
|||
An extension for debugging memory usage. It collects information about:
|
||||
|
||||
* objects uncollected by the Python garbage collector
|
||||
* objects left alive that shouldn't. For more info, see :ref:`topics-leaks-trackrefs`
|
||||
* objects left alive that shouldn't be. For more info, see
|
||||
:ref:`topics-leaks-trackrefs`
|
||||
|
||||
To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The
|
||||
info will be stored in the :stat:`memdebug/gc_garbage_count` and
|
||||
|
|
@ -258,9 +259,9 @@ settings:
|
|||
|
||||
.. note::
|
||||
|
||||
When a certain closing condition is met, requests which are
|
||||
currently in the downloader queue (up to :setting:`CONCURRENT_REQUESTS`
|
||||
requests) are still processed.
|
||||
When a certain closing condition is met, requests that are currently in the
|
||||
downloader queue (up to :setting:`CONCURRENT_REQUESTS`
|
||||
requests) are still processed.
|
||||
|
||||
.. setting:: CLOSESPIDER_TIMEOUT
|
||||
|
||||
|
|
@ -270,7 +271,7 @@ CLOSESPIDER_TIMEOUT
|
|||
Default: ``0.0``
|
||||
|
||||
If the spider remains open for more than this number of seconds, it will be
|
||||
automatically closed with the reason ``closespider_timeout``. If zero (or non
|
||||
automatically closed with the reason ``closespider_timeout``. If zero (or not
|
||||
set), spiders won't be closed by timeout.
|
||||
|
||||
.. setting:: CLOSESPIDER_TIMEOUT_NO_ITEM
|
||||
|
|
@ -282,8 +283,8 @@ Default: ``0``
|
|||
|
||||
An integer which specifies a number of seconds. If the spider has not produced
|
||||
any items in the last number of seconds, it will be closed with the reason
|
||||
``closespider_timeout_no_item``. If zero (or non set), spiders won't be closed
|
||||
regardless if it hasn't produced any items.
|
||||
``closespider_timeout_no_item``. If zero (or not set), spiders won't be closed
|
||||
even if they have not produced any items.
|
||||
|
||||
.. setting:: CLOSESPIDER_ITEMCOUNT
|
||||
|
||||
|
|
@ -293,9 +294,9 @@ CLOSESPIDER_ITEMCOUNT
|
|||
Default: ``0``
|
||||
|
||||
An integer which specifies a number of items. If the spider scrapes more than
|
||||
that amount and those items are passed by the item pipeline, the
|
||||
spider will be closed with the reason ``closespider_itemcount``.
|
||||
If zero (or non set), spiders won't be closed by number of passed items.
|
||||
that amount and those items are passed by the item pipeline, the spider will be
|
||||
closed with the reason ``closespider_itemcount``. If zero (or not set), spiders
|
||||
won't be closed by the number of passed items.
|
||||
|
||||
.. setting:: CLOSESPIDER_PAGECOUNT
|
||||
|
||||
|
|
@ -306,7 +307,7 @@ Default: ``0``
|
|||
|
||||
An integer which specifies the maximum number of responses to crawl. If the spider
|
||||
crawls more than that, the spider will be closed with the reason
|
||||
``closespider_pagecount``. If zero (or non set), spiders won't be closed by
|
||||
``closespider_pagecount``. If zero (or not set), spiders won't be closed by the
|
||||
number of crawled responses.
|
||||
|
||||
.. setting:: CLOSESPIDER_PAGECOUNT_NO_ITEM
|
||||
|
|
@ -320,7 +321,7 @@ An integer which specifies the maximum number of consecutive responses to crawl
|
|||
without items scraped. If the spider crawls more consecutive responses than that
|
||||
and no items are scraped in the meantime, the spider will be closed with the
|
||||
reason ``closespider_pagecount_no_item``. If zero (or not set), spiders won't be
|
||||
closed by number of crawled responses with no items.
|
||||
closed by the number of crawled responses with no items.
|
||||
|
||||
.. setting:: CLOSESPIDER_ERRORCOUNT
|
||||
|
||||
|
|
@ -331,8 +332,8 @@ Default: ``0``
|
|||
|
||||
An integer which specifies the maximum number of errors to receive before
|
||||
closing the spider. If the spider generates more than that number of errors,
|
||||
it will be closed with the reason ``closespider_errorcount``. If zero (or non
|
||||
set), spiders won't be closed by number of errors.
|
||||
it will be closed with the reason ``closespider_errorcount``. If zero (or not
|
||||
set), spiders won't be closed by the number of errors.
|
||||
|
||||
.. module:: scrapy.extensions.periodic_log
|
||||
:synopsis: Periodic stats logging
|
||||
|
|
|
|||
|
|
@ -5,9 +5,9 @@ Feed exports
|
|||
============
|
||||
|
||||
One of the most frequently required features when implementing scrapers is
|
||||
being able to store the scraped data properly and, quite often, that means
|
||||
generating an "export file" with the scraped data (commonly called "export
|
||||
feed") to be consumed by other systems.
|
||||
properly storing the scraped data and, quite often, that means generating an
|
||||
"export file" with the scraped data (commonly called an "export feed") to be
|
||||
consumed by other systems.
|
||||
|
||||
Scrapy provides this functionality out of the box with the Feed Exports, which
|
||||
allows you to generate feeds with the scraped items, using multiple
|
||||
|
|
@ -31,7 +31,7 @@ For serializing the scraped data, the feed exports use the :ref:`Item exporters
|
|||
- :ref:`topics-feed-format-csv`
|
||||
- :ref:`topics-feed-format-xml`
|
||||
|
||||
But you can also extend the supported format through the
|
||||
But you can also extend the supported formats through the
|
||||
:setting:`FEED_EXPORTERS` setting.
|
||||
|
||||
.. _topics-feed-format-json:
|
||||
|
|
@ -97,9 +97,9 @@ Marshal
|
|||
Storages
|
||||
========
|
||||
|
||||
When using the feed exports you define where to store the feed using one or multiple URIs_
|
||||
(through the :setting:`FEEDS` setting). The feed exports supports multiple
|
||||
storage backend types which are defined by the URI scheme.
|
||||
When using the feed exports you define where to store the feed using one or
|
||||
multiple URIs_ (through the :setting:`FEEDS` setting). The feed exports support
|
||||
multiple storage backend types that are defined by the URI scheme.
|
||||
|
||||
The storages backends supported out of the box are:
|
||||
|
||||
|
|
@ -125,8 +125,8 @@ being created. These parameters are:
|
|||
- ``%(time)s`` - gets replaced by a timestamp when the feed is being created
|
||||
- ``%(name)s`` - gets replaced by the spider name
|
||||
|
||||
Any other named parameter gets replaced by the spider attribute of the same
|
||||
name. For example, ``%(site_id)s`` would get replaced by the ``spider.site_id``
|
||||
Any other named parameter is replaced by the spider attribute of the same name.
|
||||
For example, ``%(site_id)s`` would be replaced by the ``spider.site_id``
|
||||
attribute the moment the feed is being created.
|
||||
|
||||
Here are some examples to illustrate:
|
||||
|
|
@ -164,9 +164,9 @@ The feeds are stored in the local filesystem.
|
|||
- Example URI: ``file:///tmp/export.csv``
|
||||
- Required external libraries: none
|
||||
|
||||
Note that for the local filesystem storage (only) you can omit the scheme if
|
||||
you specify a path (e.g. ``/tmp/export.csv``).
|
||||
Alternatively you can also use a :class:`pathlib.Path` object.
|
||||
Note that for the local filesystem storage you can omit the scheme if you
|
||||
specify a path (e.g. ``/tmp/export.csv``). Alternatively, you can use a
|
||||
:class:`pathlib.Path` object.
|
||||
|
||||
.. _topics-feed-storage-ftp:
|
||||
.. _feed-storage-ftp:
|
||||
|
|
@ -174,7 +174,7 @@ Alternatively you can also use a :class:`pathlib.Path` object.
|
|||
FTP
|
||||
---
|
||||
|
||||
The feeds are stored in a FTP server.
|
||||
The feeds are stored on an FTP server.
|
||||
|
||||
- URI scheme: ``ftp``
|
||||
- Example URI: ``ftp://user:pass@ftp.example.com/path/to/export.csv``
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ Item Pipeline
|
|||
After an item has been scraped by a spider, it is sent to the Item Pipeline
|
||||
which processes it through several components that are executed sequentially.
|
||||
|
||||
Each item pipeline component (sometimes referred as just "Item Pipeline") is a
|
||||
Python class that implements a simple method. They receive an item and perform
|
||||
an action over it, also deciding if the item should continue through the
|
||||
pipeline or be dropped and no longer processed.
|
||||
Each item pipeline component (sometimes referred to as just "Item Pipeline") is
|
||||
a Python class that implements a simple method. Each component receives an item
|
||||
and performs an action on it while deciding whether the item should continue
|
||||
through the pipeline or be dropped and no longer processed.
|
||||
|
||||
Typical uses of item pipelines are:
|
||||
|
||||
|
|
@ -28,7 +28,7 @@ implement the following method:
|
|||
|
||||
.. method:: process_item(self, item)
|
||||
|
||||
This method is called for every item pipeline component.
|
||||
Scrapy calls this method for every item processed by the pipeline component.
|
||||
|
||||
`item` is an :ref:`item object <item-types>`, see
|
||||
:ref:`supporting-item-types`.
|
||||
|
|
@ -41,7 +41,7 @@ implement the following method:
|
|||
:param item: the scraped item
|
||||
:type item: :ref:`item object <item-types>`
|
||||
|
||||
Additionally, they may also implement the following methods:
|
||||
Additionally, a component may implement the following methods:
|
||||
|
||||
.. method:: open_spider(self)
|
||||
|
||||
|
|
@ -72,8 +72,8 @@ Price validation and dropping items with no prices
|
|||
|
||||
Let's take a look at the following hypothetical pipeline that adjusts the
|
||||
``price`` attribute for those items that do not include VAT
|
||||
(``price_excludes_vat`` attribute), and drops those items which don't
|
||||
contain a price:
|
||||
(``price_excludes_vat`` attribute), and drops those items that don't contain a
|
||||
price:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -120,16 +120,16 @@ format:
|
|||
self.file.write(line)
|
||||
return item
|
||||
|
||||
.. note:: The purpose of JsonWriterPipeline is just to introduce how to write
|
||||
item pipelines. If you really want to store all scraped items into a JSON
|
||||
file you should use the :ref:`Feed exports <topics-feed-exports>`.
|
||||
.. note:: The JsonWriterPipeline example simply introduces how to write item
|
||||
pipelines. If you really want to store all scraped items in a JSON file,
|
||||
you should use the :ref:`Feed exports <topics-feed-exports>`.
|
||||
|
||||
Write items to MongoDB
|
||||
----------------------
|
||||
|
||||
In this example we'll write items to MongoDB_ using pymongo_.
|
||||
MongoDB address and database name are specified in Scrapy settings;
|
||||
MongoDB collection is specified in a class attribute.
|
||||
In this example, we'll write items to MongoDB_ using pymongo_. The MongoDB
|
||||
address and database name are specified in the Scrapy settings; the MongoDB
|
||||
collection is specified in a class attribute.
|
||||
|
||||
The main point of this example is to show how to :ref:`get the crawler
|
||||
<from-crawler>` and how to clean up the resources properly.
|
||||
|
|
@ -172,13 +172,13 @@ The main point of this example is to show how to :ref:`get the crawler
|
|||
|
||||
.. _ScreenshotPipeline:
|
||||
|
||||
Take screenshot of item
|
||||
-----------------------
|
||||
Take a screenshot of an item
|
||||
----------------------------
|
||||
|
||||
This example demonstrates how to use :doc:`coroutine syntax <coroutines>` in
|
||||
the :meth:`process_item` method.
|
||||
|
||||
This item pipeline makes a request to a locally-running instance of Splash_ to
|
||||
This item pipeline makes a request to a locally running instance of Splash_ to
|
||||
render a screenshot of the item URL. After the request response is downloaded,
|
||||
the item pipeline saves the screenshot to a file and adds the filename to the
|
||||
item.
|
||||
|
|
@ -195,8 +195,8 @@ item.
|
|||
|
||||
|
||||
class ScreenshotPipeline:
|
||||
"""Pipeline that uses Splash to render screenshot of
|
||||
every Scrapy item."""
|
||||
"""Pipeline that uses Splash to render a screenshot of every Scrapy
|
||||
item."""
|
||||
|
||||
SPLASH_URL = "http://localhost:8050/render.png?url={}"
|
||||
|
||||
|
|
@ -215,16 +215,17 @@ item.
|
|||
response = await self.crawler.engine.download_async(request)
|
||||
|
||||
if response.status != 200:
|
||||
# Error happened, return item.
|
||||
# An error occurred, so return the item.
|
||||
return item
|
||||
|
||||
# Save screenshot to file, filename will be hash of url.
|
||||
# Save the screenshot to a file; the filename is the hash of the
|
||||
# URL.
|
||||
url = adapter["url"]
|
||||
url_hash = hashlib.md5(url.encode("utf8")).hexdigest()
|
||||
filename = f"{url_hash}.png"
|
||||
Path(filename).write_bytes(response.body)
|
||||
|
||||
# Store filename in item.
|
||||
# Store the filename in the item.
|
||||
adapter["screenshot_filename"] = filename
|
||||
return item
|
||||
|
||||
|
|
@ -233,9 +234,9 @@ item.
|
|||
Duplicates filter
|
||||
-----------------
|
||||
|
||||
A filter that looks for duplicate items, and drops those items that were
|
||||
already processed. Let's say that our items have a unique id, but our spider
|
||||
returns multiples items with the same id:
|
||||
This filter looks for duplicate items and drops those that were already
|
||||
processed. Let's say that our items have a unique id, but our spider returns
|
||||
multiple items with the same id:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ Items
|
|||
:synopsis: Item and Field classes
|
||||
|
||||
The main goal in scraping is to extract structured data from unstructured
|
||||
sources, typically, web pages. :ref:`Spiders <topics-spiders>` may return the
|
||||
sources, typically web pages. :ref:`Spiders <topics-spiders>` may return the
|
||||
extracted data as `items`, Python objects that define key-value pairs.
|
||||
|
||||
Scrapy supports :ref:`multiple types of items <item-types>`. When you create an
|
||||
|
|
@ -21,10 +21,10 @@ receives an item, your code should :ref:`work for any item type
|
|||
Item Types
|
||||
==========
|
||||
|
||||
Scrapy supports the following types of items, via the `itemadapter`_ library:
|
||||
Scrapy supports the following types of items via the `itemadapter`_ library:
|
||||
:ref:`dictionaries <dict-items>`, :ref:`Item objects <item-objects>`,
|
||||
:ref:`dataclass objects <dataclass-items>`, :ref:`attrs objects <attrs-items>`
|
||||
and :ref:`Pydantic models <pydantic-items>`.
|
||||
:ref:`dataclass objects <dataclass-items>`, :ref:`attrs objects
|
||||
<attrs-items>`, and :ref:`Pydantic models <pydantic-items>`.
|
||||
|
||||
.. _itemadapter: https://github.com/scrapy/itemadapter
|
||||
|
||||
|
|
@ -50,16 +50,16 @@ make it the most feature-complete item type:
|
|||
:class:`Item` objects replicate the standard :class:`dict` API, including
|
||||
its ``__init__`` method.
|
||||
|
||||
:class:`Item` allows the defining of field names, so that:
|
||||
:class:`Item` allows you to define field names, so that:
|
||||
|
||||
- :class:`KeyError` is raised when using undefined field names (i.e.
|
||||
prevents typos going unnoticed)
|
||||
- :class:`KeyError` is raised when using undefined field names (i.e. prevents
|
||||
typos from going unnoticed)
|
||||
|
||||
- :ref:`Item exporters <topics-exporters>` can export all fields by
|
||||
default even if the first scraped object does not have values for all
|
||||
of them
|
||||
|
||||
:class:`Item` also allows the defining of field metadata, which can be used to
|
||||
:class:`Item` also lets you define field metadata, which can be used to
|
||||
:ref:`customize serialization <topics-exporters-field-serialization>`.
|
||||
|
||||
:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory
|
||||
|
|
@ -81,11 +81,12 @@ Example:
|
|||
Dataclass objects
|
||||
-----------------
|
||||
|
||||
:func:`~dataclasses.dataclass` allows the defining of item classes with field names,
|
||||
so that :ref:`item exporters <topics-exporters>` can export all fields by
|
||||
default even if the first scraped object does not have values for all of them.
|
||||
:func:`~dataclasses.dataclass` allows you to define item classes with field
|
||||
names, so that :ref:`item exporters <topics-exporters>` can export all fields
|
||||
by default even if the first scraped object does not have values for all of
|
||||
them.
|
||||
|
||||
Additionally, ``dataclass`` items also allow you to:
|
||||
Additionally, ``dataclass`` items allow you to:
|
||||
|
||||
* define the type and default value of each defined field.
|
||||
|
||||
|
|
@ -111,18 +112,18 @@ Example:
|
|||
attr.s objects
|
||||
--------------
|
||||
|
||||
:func:`attr.s` allows the defining of item classes with field names,
|
||||
so that :ref:`item exporters <topics-exporters>` can export all fields by
|
||||
default even if the first scraped object does not have values for all of them.
|
||||
:func:`attr.s` allows you to define item classes with field names, so that
|
||||
:ref:`item exporters <topics-exporters>` can export all fields by default even
|
||||
if the first scraped object does not have values for all of them.
|
||||
|
||||
Additionally, ``attr.s`` items also allow to:
|
||||
Additionally, ``attr.s`` items allow you to:
|
||||
|
||||
* define the type and default value of each defined field.
|
||||
|
||||
* define custom field :ref:`metadata <attrs:metadata>`, which can be used to
|
||||
:ref:`customize serialization <topics-exporters-field-serialization>`.
|
||||
|
||||
In order to use this type, the :doc:`attrs package <attrs:index>` needs to be installed.
|
||||
To use this type, the :doc:`attrs package <attrs:index>` needs to be installed.
|
||||
|
||||
Example:
|
||||
|
||||
|
|
@ -199,9 +200,9 @@ Item subclasses are declared using a simple class definition syntax and
|
|||
tags = scrapy.Field()
|
||||
last_updated = scrapy.Field(serializer=str)
|
||||
|
||||
.. note:: Those familiar with `Django`_ will notice that Scrapy Items are
|
||||
declared similar to `Django Models`_, except that Scrapy Items are much
|
||||
simpler as there is no concept of different field types.
|
||||
.. note:: Those familiar with `Django`_ will notice that Scrapy items are
|
||||
declared similarly to `Django Models`_, except that Scrapy items are much
|
||||
simpler as there is no concept of different field types.
|
||||
|
||||
.. _Django: https://www.djangoproject.com/
|
||||
.. _Django Models: https://docs.djangoproject.com/en/dev/topics/db/models/
|
||||
|
|
@ -213,19 +214,19 @@ Declaring fields
|
|||
----------------
|
||||
|
||||
:class:`Field` objects are used to specify metadata for each field. For
|
||||
example, the serializer function for the ``last_updated`` field illustrated in
|
||||
the example above.
|
||||
example, they can store the serializer function for the ``last_updated`` field
|
||||
illustrated above.
|
||||
|
||||
You can specify any kind of metadata for each field. There is no restriction on
|
||||
the values accepted by :class:`Field` objects. For this same
|
||||
reason, there is no reference list of all available metadata keys. Each key
|
||||
defined in :class:`Field` objects could be used by a different component, and
|
||||
only those components know about it. You can also define and use any other
|
||||
:class:`Field` key in your project too, for your own needs. The main goal of
|
||||
:class:`Field` objects is to provide a way to define all field metadata in one
|
||||
place. Typically, those components whose behaviour depends on each field use
|
||||
certain field keys to configure that behaviour. You must refer to their
|
||||
documentation to see which metadata keys are used by each component.
|
||||
the values accepted by :class:`Field` objects. For this same reason, there is
|
||||
no reference list of all available metadata keys. Each key defined in
|
||||
:class:`Field` objects could be used by a different component, and only those
|
||||
components know about it. You can also define and use any other :class:`Field`
|
||||
key in your project too, for your own needs. The main goal of :class:`Field`
|
||||
objects is to provide a way to define all field metadata in one place.
|
||||
Typically, those components whose behavior depends on each field use certain
|
||||
field keys to configure that behavior. You must refer to their documentation to
|
||||
see which metadata keys are used by each component.
|
||||
|
||||
It's important to note that the :class:`Field` objects used to declare the item
|
||||
do not stay assigned as class attributes. Instead, they can be accessed through
|
||||
|
|
@ -253,7 +254,7 @@ Working with Item objects
|
|||
.. skip: start
|
||||
|
||||
Here are some examples of common tasks performed with items, using the
|
||||
``Product`` item :ref:`declared above <topics-items-declaring>`. You will
|
||||
``Product`` item :ref:`declared above <topics-items-declaring>`. You will
|
||||
notice the API is very similar to the :class:`dict` API.
|
||||
|
||||
Creating items
|
||||
|
|
@ -393,8 +394,8 @@ Creating items from dicts:
|
|||
Extending Item subclasses
|
||||
-------------------------
|
||||
|
||||
You can extend Items (to add more fields or to change some metadata for some
|
||||
fields) by declaring a subclass of your original Item.
|
||||
You can extend items (to add more fields or to change some metadata for some
|
||||
fields) by declaring a subclass of your original item.
|
||||
|
||||
For example:
|
||||
|
||||
|
|
@ -425,7 +426,7 @@ Supporting All Item Types
|
|||
|
||||
In code that receives an item, such as methods of :ref:`item pipelines
|
||||
<topics-item-pipeline>` or :ref:`spider middlewares
|
||||
<topics-spider-middleware>`, it is a good practice to use the
|
||||
<topics-spider-middleware>`, it is good practice to use the
|
||||
:class:`~itemadapter.ItemAdapter` class to write code that works for any
|
||||
supported item type.
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ Jobs: pausing and resuming crawls
|
|||
Sometimes, for big sites, it's desirable to pause crawls and be able to resume
|
||||
them later.
|
||||
|
||||
Scrapy supports this functionality out of the box by providing the following
|
||||
Scrapy supports this functionality out of the box through the following
|
||||
facilities:
|
||||
|
||||
* a scheduler that persists scheduled requests on disk
|
||||
|
|
@ -44,7 +44,7 @@ To start a spider with persistence support enabled, run it like this::
|
|||
|
||||
scrapy crawl somespider -s JOBDIR=crawls/somespider-1
|
||||
|
||||
Then, you can stop the spider safely at any time (by pressing Ctrl-C or sending
|
||||
Then you can stop the spider safely at any time (by pressing Ctrl-C or sending
|
||||
a signal), and resume it later by issuing the same command::
|
||||
|
||||
scrapy crawl somespider -s JOBDIR=crawls/somespider-1
|
||||
|
|
@ -54,11 +54,11 @@ a signal), and resume it later by issuing the same command::
|
|||
Keeping persistent state between batches
|
||||
========================================
|
||||
|
||||
Sometimes you'll want to keep some persistent spider state between pause/resume
|
||||
batches. You can use the ``spider.state`` attribute for that, which should be a
|
||||
dict. There's :ref:`a built-in extension <topics-extensions-ref-spiderstate>`
|
||||
that takes care of serializing, storing and loading that attribute from the job
|
||||
directory, when the spider starts and stops.
|
||||
Sometimes you'll want to keep persistent spider state between pause/resume
|
||||
batches. Use the ``spider.state`` attribute for that. It should be a dict.
|
||||
There's :ref:`a built-in extension <topics-extensions-ref-spiderstate>` that
|
||||
takes care of serializing, storing and loading that attribute from the job
|
||||
directory when the spider starts and stops.
|
||||
|
||||
Here's an example of a callback that uses the spider state (other spider code
|
||||
is omitted for brevity):
|
||||
|
|
@ -72,8 +72,8 @@ is omitted for brevity):
|
|||
Persistence gotchas
|
||||
===================
|
||||
|
||||
There are a few things to keep in mind if you want to be able to use the Scrapy
|
||||
persistence support:
|
||||
There are a few things to keep in mind if you want to use Scrapy's persistence
|
||||
support:
|
||||
|
||||
Pause limitations
|
||||
-----------------
|
||||
|
|
@ -91,11 +91,11 @@ version that wrote them. A job must be resumed with the same Scrapy version
|
|||
that paused it; after upgrading or downgrading Scrapy, start a new job with a
|
||||
new job directory.
|
||||
|
||||
Cookies expiration
|
||||
------------------
|
||||
Cookie expiration
|
||||
-----------------
|
||||
|
||||
Cookies may expire. So, if you don't resume your spider quickly the requests
|
||||
scheduled may no longer work. This won't be an issue if your spider doesn't rely
|
||||
Cookies may expire, so if you don't resume your spider quickly, the scheduled
|
||||
requests may no longer work. This won't be an issue if your spider doesn't rely
|
||||
on cookies.
|
||||
|
||||
.. _request-serialization:
|
||||
|
|
|
|||
|
|
@ -5,55 +5,53 @@ Debugging memory leaks
|
|||
======================
|
||||
|
||||
In Scrapy, objects such as requests, responses and items have a finite
|
||||
lifetime: they are created, used for a while, and finally destroyed.
|
||||
lifetime: they are created, used for a while and finally destroyed.
|
||||
|
||||
From all those objects, the Request is probably the one with the longest
|
||||
lifetime, as it stays waiting in the Scheduler queue until it's time to process
|
||||
it. For more info see :ref:`topics-architecture`.
|
||||
Of all those objects, the Request usually has the longest lifetime because it
|
||||
waits in the Scheduler queue until it's time to process it. For more
|
||||
information, see :ref:`topics-architecture`.
|
||||
|
||||
As these Scrapy objects have a (rather long) lifetime, there is always the risk
|
||||
of accumulating them in memory without releasing them properly and thus causing
|
||||
what is known as a "memory leak".
|
||||
Because these Scrapy objects have a comparatively long lifetime, there is
|
||||
always the risk of accumulating them in memory without releasing them properly
|
||||
and thus causing what is known as a "memory leak".
|
||||
|
||||
To help debugging memory leaks, Scrapy provides a built-in mechanism for
|
||||
tracking objects references called :ref:`trackref <topics-leaks-trackrefs>`,
|
||||
and you can also use a third-party library called :ref:`muppy
|
||||
<topics-leaks-muppy>` for more advanced memory debugging (see below for more
|
||||
info). Both mechanisms must be used from the :ref:`Telnet Console
|
||||
<topics-telnetconsole>`.
|
||||
To help debug memory leaks, Scrapy provides a built-in mechanism for tracking
|
||||
object references called :ref:`trackref <topics-leaks-trackrefs>`, and you can
|
||||
also use a third-party library called :ref:`muppy <topics-leaks-muppy>` for
|
||||
more advanced memory debugging (see below for more information). Both
|
||||
mechanisms must be used from the :ref:`Telnet Console <topics-telnetconsole>`.
|
||||
|
||||
Common causes of memory leaks
|
||||
=============================
|
||||
|
||||
It happens quite often (sometimes by accident, sometimes on purpose) that the
|
||||
Scrapy developer passes objects referenced in Requests (for example, using the
|
||||
:attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta`
|
||||
attributes or the request callback function) and that effectively bounds the
|
||||
lifetime of those referenced objects to the lifetime of the Request. This is,
|
||||
by far, the most common cause of memory leaks in Scrapy projects, and a quite
|
||||
difficult one to debug for newcomers.
|
||||
Scrapy developer passes objects referenced in requests—for example, through the
|
||||
:attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta` attributes or
|
||||
the request callback function—and that effectively ties the lifetime of those
|
||||
referenced objects to the lifetime of the request. This is, by far, the most
|
||||
common cause of memory leaks in Scrapy projects, and it can be quite difficult
|
||||
for newcomers to debug.
|
||||
|
||||
In big projects, the spiders are typically written by different people and some
|
||||
of those spiders could be "leaking" and thus affecting the rest of the other
|
||||
(well-written) spiders when they get to run concurrently, which, in turn,
|
||||
affects the whole crawling process.
|
||||
In big projects, spiders are typically written by different people, and some of
|
||||
those spiders could be "leaking" and thus affecting the rest of the
|
||||
(well-written) spiders when they run concurrently, which, in turn, affects the
|
||||
whole crawling process.
|
||||
|
||||
The leak could also come from a custom middleware, pipeline or extension that
|
||||
you have written, if you are not releasing the (previously allocated) resources
|
||||
properly. For example, allocating resources on :signal:`spider_opened`
|
||||
but not releasing them on :signal:`spider_closed` may cause problems if
|
||||
you're running :ref:`multiple spiders per process <run-multiple-spiders>`.
|
||||
The leak could also come from a custom middleware, pipeline, or extension that
|
||||
you wrote if you are not releasing the previously allocated resources properly.
|
||||
For example, allocating resources on :signal:`spider_opened` but not releasing
|
||||
them on :signal:`spider_closed` may cause problems if you're running
|
||||
:ref:`multiple spiders per process <run-multiple-spiders>`.
|
||||
|
||||
Too Many Requests?
|
||||
------------------
|
||||
|
||||
By default Scrapy keeps the request queue in memory; it includes
|
||||
:class:`~scrapy.Request` objects and all objects
|
||||
referenced in Request attributes (e.g. in :attr:`~scrapy.Request.cb_kwargs`
|
||||
and :attr:`~scrapy.Request.meta`).
|
||||
While not necessarily a leak, this can take a lot of memory. Enabling
|
||||
:ref:`persistent job queue <topics-jobs>` could help keeping memory usage
|
||||
in control.
|
||||
:class:`~scrapy.Request` objects and all objects referenced in request
|
||||
attributes (for example, :attr:`~scrapy.Request.cb_kwargs` and
|
||||
:attr:`~scrapy.Request.meta`). While this is not necessarily a leak, it can
|
||||
consume a lot of memory. Enabling the :ref:`persistent job queue <topics-jobs>`
|
||||
can help keep memory usage under control.
|
||||
|
||||
.. _topics-leaks-trackrefs:
|
||||
|
||||
|
|
@ -63,8 +61,8 @@ Debugging memory leaks with ``trackref``
|
|||
.. skip: start
|
||||
|
||||
:mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most
|
||||
common cases of memory leaks. It basically tracks the references to all live
|
||||
Request, Response, Item, Spider and Selector objects.
|
||||
common cases of memory leaks. It tracks the references to all live Request,
|
||||
Response, Item, Spider and Selector objects.
|
||||
|
||||
You can enter the telnet console and inspect how many objects (of the classes
|
||||
mentioned above) are currently alive using the ``prefs()`` function which is an
|
||||
|
|
@ -85,16 +83,15 @@ alias to the :func:`~scrapy.utils.trackref.print_live_refs` function:
|
|||
Request 878 oldest: 7s ago
|
||||
|
||||
As you can see, that report also shows the "age" of the oldest object in each
|
||||
class. If you're running multiple spiders per process chances are you can
|
||||
figure out which spider is leaking by looking at the oldest request or response.
|
||||
You can get the oldest object of each class using the
|
||||
class. If you're running multiple spiders per process, chances are you can
|
||||
figure out which spider is leaking by looking at the oldest request or
|
||||
response. You can get the oldest object of each class using the
|
||||
:func:`~scrapy.utils.trackref.get_oldest` function (from the telnet console).
|
||||
|
||||
Which objects are tracked?
|
||||
--------------------------
|
||||
|
||||
The objects tracked by ``trackref`` are all from these classes (and all its
|
||||
subclasses):
|
||||
``trackref`` tracks objects from these classes (and all their subclasses):
|
||||
|
||||
* :class:`scrapy.Request`
|
||||
* :class:`scrapy.http.Response`
|
||||
|
|
@ -116,9 +113,9 @@ Suppose we have some spider with a line similar to this one:
|
|||
cb_kwargs={"referer": response},
|
||||
)
|
||||
|
||||
That line is passing a response reference inside a request which effectively
|
||||
ties the response lifetime to the requests' one, and that would definitely
|
||||
cause memory leaks.
|
||||
That line passes a response reference inside a request, which effectively ties
|
||||
the response lifetime to the request's lifetime, and that will definitely cause
|
||||
memory leaks.
|
||||
|
||||
Let's see how we can discover the cause (without knowing it
|
||||
a priori, of course) by using the ``trackref`` tool.
|
||||
|
|
@ -139,10 +136,10 @@ references:
|
|||
|
||||
The fact that there are so many live responses (and that they're so old) is
|
||||
definitely suspicious, as responses should have a relatively short lifetime
|
||||
compared to Requests. The number of responses is similar to the number
|
||||
of requests, so it looks like they are tied in a some way. We can now go
|
||||
and check the code of the spider to discover the nasty line that is
|
||||
generating the leaks (passing response references inside requests).
|
||||
compared to requests. The number of responses is similar to the number of
|
||||
requests, so it looks like they are tied in some way. We can now check the code
|
||||
of the spider to discover the line that is generating the leaks (passing
|
||||
response references inside requests).
|
||||
|
||||
Sometimes extra information about live objects can be helpful.
|
||||
Let's check the oldest response:
|
||||
|
|
@ -154,8 +151,8 @@ Let's check the oldest response:
|
|||
>>> r.url
|
||||
'http://www.somenastyspider.com/product.php?pid=123'
|
||||
|
||||
If you want to iterate over all objects, instead of getting the oldest one, you
|
||||
can use the :func:`scrapy.utils.trackref.iter_all` function:
|
||||
If you want to iterate over all objects instead of getting only the oldest one,
|
||||
you can use the :func:`scrapy.utils.trackref.iter_all` function:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
|
|
@ -168,11 +165,10 @@ can use the :func:`scrapy.utils.trackref.iter_all` function:
|
|||
Too many spiders?
|
||||
-----------------
|
||||
|
||||
If your project has too many spiders executed in parallel,
|
||||
the output of ``prefs()`` can be difficult to read.
|
||||
For this reason, that function has a ``ignore`` argument which can be used to
|
||||
ignore a particular class (and all its subclasses). For
|
||||
example, this won't show any live references to spiders:
|
||||
If your project has too many spiders executed in parallel, the output of
|
||||
``prefs()`` can be difficult to read. For this reason, that function has an
|
||||
``ignore`` argument that you can use to omit a particular class (and all its
|
||||
subclasses). For example, this won't show any live references to spiders:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
|
|
@ -202,14 +198,13 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
|
|||
Debugging memory leaks with muppy
|
||||
=================================
|
||||
|
||||
``trackref`` provides a very convenient mechanism for tracking down memory
|
||||
leaks, but it only keeps track of the objects that are more likely to cause
|
||||
memory leaks. However, there are other cases where the memory leaks could come
|
||||
from other (more or less obscure) objects. If this is your case, and you can't
|
||||
find your leaks using ``trackref``, you still have another resource: the muppy
|
||||
library.
|
||||
``trackref`` provides a convenient mechanism for tracking down memory leaks,
|
||||
but it only keeps track of the objects that are more likely to cause them.
|
||||
However, sometimes leaks come from other (more or less obscure) objects. If
|
||||
that happens and you can't find your leaks using ``trackref``, you still have
|
||||
another resource: the muppy library.
|
||||
|
||||
You can use muppy from `Pympler`_.
|
||||
muppy is available as part of `Pympler`_.
|
||||
|
||||
.. _Pympler: https://pypi.org/project/Pympler/
|
||||
|
||||
|
|
@ -217,8 +212,8 @@ If you use ``pip``, you can install muppy with the following command::
|
|||
|
||||
pip install Pympler
|
||||
|
||||
Here's an example to view all Python objects available in
|
||||
the heap using muppy:
|
||||
Here's an example that shows all Python objects available in the heap using
|
||||
muppy:
|
||||
|
||||
.. skip: start
|
||||
.. code-block:: pycon
|
||||
|
|
@ -250,7 +245,7 @@ the heap using muppy:
|
|||
|
||||
.. skip: end
|
||||
|
||||
For more info about muppy, refer to the `muppy documentation`_.
|
||||
For more information about muppy, refer to the `muppy documentation`_.
|
||||
|
||||
.. _muppy documentation: https://pythonhosted.org/Pympler/muppy.html
|
||||
|
||||
|
|
@ -259,10 +254,10 @@ For more info about muppy, refer to the `muppy documentation`_.
|
|||
Leaks without leaks
|
||||
===================
|
||||
|
||||
Sometimes, you may notice that the memory usage of your Scrapy process will
|
||||
only increase, but never decrease. Unfortunately, this could happen even
|
||||
though neither Scrapy nor your project are leaking memory. This is due to a
|
||||
(not so well) known problem of Python, which may not return released memory to
|
||||
Sometimes you may notice that the memory usage of your Scrapy process only
|
||||
increases and never decreases. Unfortunately, this could happen even though
|
||||
neither Scrapy nor your project are leaking memory. This is due to a
|
||||
not-so-well-known problem in Python, which may not return released memory to
|
||||
the operating system in some cases. For more information on this issue see:
|
||||
|
||||
* `Python Memory Management <https://www.evanjones.ca/python-memory.html>`_
|
||||
|
|
@ -283,6 +278,6 @@ completely. To quote the paper:
|
|||
|
||||
.. _this paper: https://www.evanjones.ca/memoryallocator/
|
||||
|
||||
To keep memory consumption reasonable you can split the job into several
|
||||
smaller jobs or enable :ref:`persistent job queue <topics-jobs>`
|
||||
and stop/start spider from time to time.
|
||||
To keep memory consumption reasonable, you can split the job into several
|
||||
smaller jobs or enable the :ref:`persistent job queue <topics-jobs>` and stop
|
||||
and start the spider from time to time.
|
||||
|
|
|
|||
|
|
@ -16,9 +16,10 @@ list of matching :class:`~scrapy.link.Link` objects from a
|
|||
Link extractors are used in :class:`~scrapy.spiders.CrawlSpider` spiders
|
||||
through a set of :class:`~scrapy.spiders.Rule` objects.
|
||||
|
||||
You can also use link extractors in regular spiders. For example, you can instantiate
|
||||
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class
|
||||
variable in your spider, and use it from your spider callbacks:
|
||||
You can also use link extractors in regular spiders. For example, you can
|
||||
instantiate :class:`LinkExtractor
|
||||
<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` as a class variable in your
|
||||
spider and use it from your spider callbacks:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -8,17 +8,18 @@ Item Loaders
|
|||
:synopsis: Item Loader class
|
||||
|
||||
Item Loaders provide a convenient mechanism for populating scraped :ref:`items
|
||||
<topics-items>`. Even though items can be populated directly, Item Loaders provide a
|
||||
much more convenient API for populating them from a scraping process, by automating
|
||||
some common tasks like parsing the raw extracted data before assigning it.
|
||||
<topics-items>`. Even though items can be populated directly, Item Loaders
|
||||
provide a more convenient API for populating them from a scraping process by
|
||||
automating common tasks like parsing the raw extracted data before assigning
|
||||
it.
|
||||
|
||||
In other words, :ref:`items <topics-items>` provide the *container* of
|
||||
scraped data, while Item Loaders provide the mechanism for *populating* that
|
||||
container.
|
||||
|
||||
Item Loaders are designed to provide a flexible, efficient and easy mechanism
|
||||
for extending and overriding different field parsing rules, either by spider,
|
||||
or by source format (HTML, XML, etc) without becoming a nightmare to maintain.
|
||||
for extending and overriding different field parsing rules, either by spider or
|
||||
by source format (HTML, XML, etc.) without becoming a nightmare to maintain.
|
||||
|
||||
.. note:: Item Loaders are an extension of the itemloaders_ library that make it
|
||||
easier to work with Scrapy by adding support for
|
||||
|
|
@ -80,9 +81,9 @@ and finally the ``last_updated`` field is populated directly with a literal valu
|
|||
(``today``) using a different method: :meth:`~ItemLoader.add_value`.
|
||||
|
||||
Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is
|
||||
called which actually returns the item populated with the data
|
||||
previously extracted and collected with the :meth:`~ItemLoader.add_xpath`,
|
||||
:meth:`~ItemLoader.add_css`, and :meth:`~ItemLoader.add_value` calls.
|
||||
called, which returns the item populated with the data previously extracted and
|
||||
collected with the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`,
|
||||
and :meth:`~ItemLoader.add_value` calls.
|
||||
|
||||
|
||||
.. _topics-loaders-dataclass:
|
||||
|
|
@ -117,15 +118,15 @@ Input and Output processors
|
|||
===========================
|
||||
|
||||
An Item Loader contains one input processor and one output processor for each
|
||||
(item) field. The input processor processes the extracted data as soon as it's
|
||||
received (through the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css` or
|
||||
:meth:`~ItemLoader.add_value` methods) and the result of the input processor is
|
||||
collected and kept inside the ItemLoader. After collecting all data, the
|
||||
:meth:`ItemLoader.load_item` method is called to populate and get the populated
|
||||
:ref:`item object <topics-items>`. That's when the output processor is
|
||||
called with the data previously collected (and processed using the input
|
||||
processor). The result of the output processor is the final value that gets
|
||||
assigned to the item.
|
||||
field. The input processor processes the extracted data as soon as it's
|
||||
received (through the :meth:`~ItemLoader.add_xpath`,
|
||||
:meth:`~ItemLoader.add_css`, or :meth:`~ItemLoader.add_value` methods), and the
|
||||
result of the input processor is collected and kept inside the Item Loader.
|
||||
After collecting all data, :meth:`ItemLoader.load_item` is called to populate
|
||||
and return the populated :ref:`item object <topics-items>`. That's when the
|
||||
output processor is called with the data previously collected and processed by
|
||||
the input processor. The result of the output processor is the final value that
|
||||
gets assigned to the item.
|
||||
|
||||
Let's see an example to illustrate how the input and output processors are
|
||||
called for a particular field (the same applies for any other field):
|
||||
|
|
@ -151,33 +152,32 @@ So what happens is:
|
|||
data collected in (1) (if any).
|
||||
|
||||
3. This case is similar to the previous ones, except that the data is extracted
|
||||
from the ``css`` CSS selector, and passed through the same *input
|
||||
processor* used in (1) and (2). The result of the input processor is appended to the
|
||||
from the ``css`` CSS selector and passed through the same *input processor*
|
||||
used in (1) and (2). The result of the input processor is appended to the
|
||||
data collected in (1) and (2) (if any).
|
||||
|
||||
4. This case is also similar to the previous ones, except that the value to be
|
||||
collected is assigned directly, instead of being extracted from a XPath
|
||||
expression or a CSS selector.
|
||||
However, the value is still passed through the input processors. In this
|
||||
case, since the value is not iterable it is converted to an iterable of a
|
||||
single element before passing it to the input processor, because input
|
||||
processor always receive iterables.
|
||||
expression or a CSS selector. However, the value is still passed through the
|
||||
input processors. In this case, since the value is not iterable it is
|
||||
converted to an iterable of a single element before passing it to the input
|
||||
processor because input processors always receive iterables.
|
||||
|
||||
5. The data collected in steps (1), (2), (3) and (4) is passed through
|
||||
the *output processor* of the ``name`` field.
|
||||
The result of the output processor is the value assigned to the ``name``
|
||||
field in the item.
|
||||
|
||||
It's worth noticing that processors are just callable objects, which are called
|
||||
with the data to be parsed, and return a parsed value. So you can use any
|
||||
It's worth noting that processors are just callable objects that are called
|
||||
with the data to be parsed and return a parsed value. So you can use any
|
||||
function as input or output processor. The only requirement is that they must
|
||||
accept one (and only one) positional argument, which will be an iterable.
|
||||
|
||||
.. note:: Both input and output processors must receive an iterable as their
|
||||
first argument. The output of those functions can be anything. The result of
|
||||
input processors will be appended to an internal list (in the Loader)
|
||||
containing the collected values (for that field). The result of the output
|
||||
processors is the value that will be finally assigned to the item.
|
||||
input processors will be appended to an internal list (in the loader)
|
||||
containing the collected values for that field. The result of the output
|
||||
processors is the value that will be assigned to the item.
|
||||
|
||||
The other thing you need to keep in mind is that the values returned by input
|
||||
processors are collected internally (in lists) and then passed to output
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@
|
|||
Logging
|
||||
=======
|
||||
|
||||
Scrapy uses :mod:`logging` for event logging. We'll
|
||||
provide some simple examples to get you started, but for more advanced
|
||||
use-cases it's strongly suggested to read thoroughly its documentation.
|
||||
Scrapy uses :mod:`logging` for event logging. We'll provide some simple
|
||||
examples to get you started, but for more advanced use-cases it's strongly
|
||||
suggested to read its documentation thoroughly.
|
||||
|
||||
Logging works out of the box, and can be configured to some extent with the
|
||||
Scrapy settings listed in :ref:`topics-logging-settings`.
|
||||
|
|
@ -21,8 +21,8 @@ Scrapy from scripts as described in :ref:`run-from-script`.
|
|||
Log levels
|
||||
==========
|
||||
|
||||
Python's builtin logging defines 5 different levels to indicate the severity of a
|
||||
given log message. Here are the standard ones, listed in decreasing order:
|
||||
Python's built-in logging defines 5 different levels to indicate the severity
|
||||
of a given log message. Here are the standard ones, listed in decreasing order:
|
||||
|
||||
1. ``logging.CRITICAL`` - for critical errors (highest severity)
|
||||
2. ``logging.ERROR`` - for regular errors
|
||||
|
|
@ -57,10 +57,10 @@ example, a common practice is to create different loggers for every module).
|
|||
These loggers can be configured independently, and they allow hierarchical
|
||||
constructions.
|
||||
|
||||
The previous examples use the root logger behind the scenes, which is a top level
|
||||
logger where all messages are propagated to (unless otherwise specified). Using
|
||||
``logging`` helpers is merely a shortcut for getting the root logger
|
||||
explicitly, so this is also an equivalent of the last snippets:
|
||||
The previous examples use the root logger behind the scenes, which is a
|
||||
top-level logger where all messages are propagated to (unless otherwise
|
||||
specified). Using ``logging`` helpers is merely a shortcut for getting the root
|
||||
logger explicitly, so this is also an equivalent of the last snippets:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -80,8 +80,8 @@ You can use a different logger just by getting its name with the
|
|||
logger.warning("This is a warning")
|
||||
|
||||
Finally, you can ensure having a custom logger for any module you're working on
|
||||
by using the ``__name__`` variable, which is populated with current module's
|
||||
path:
|
||||
by using the ``__name__`` variable, which is populated with the current
|
||||
module's path:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -321,9 +321,8 @@ filter out unwanted messages:
|
|||
if match:
|
||||
return False
|
||||
|
||||
A project-level filter may be attached to the root
|
||||
handler created by Scrapy, this is a wieldy way to
|
||||
filter all loggers in different parts of the project
|
||||
A project-level filter may be attached to the root handler created by Scrapy.
|
||||
This is a useful way to filter all loggers in different parts of the project
|
||||
(middlewares, spider, etc.):
|
||||
|
||||
.. code-block:: python
|
||||
|
|
|
|||
|
|
@ -14,15 +14,15 @@ typically you'll either use the Files Pipeline or the Images Pipeline.
|
|||
|
||||
Both pipelines implement these features:
|
||||
|
||||
* Avoid re-downloading media that was downloaded recently
|
||||
* Specifying where to store the media (filesystem directory, FTP server, Amazon S3 bucket,
|
||||
Google Cloud Storage bucket)
|
||||
* Avoid re-downloading media that have been downloaded recently
|
||||
* Specify where to store the media (filesystem directory, FTP server, Amazon S3
|
||||
bucket, Google Cloud Storage bucket)
|
||||
|
||||
The Images Pipeline has a few extra functions for processing images:
|
||||
|
||||
* Convert all downloaded images to a common format (JPG) and mode (RGB)
|
||||
* Convert all downloaded images to a common format (JPEG) and mode (RGB)
|
||||
* Thumbnail generation
|
||||
* Check images width/height to make sure they meet a minimum constraint
|
||||
* Check images' width/height to make sure they meet a minimum constraint
|
||||
|
||||
The pipelines also keep an internal queue of those media URLs which are currently
|
||||
being scheduled for download, and connect those responses that arrive containing
|
||||
|
|
@ -49,9 +49,9 @@ this:
|
|||
4. When the files are downloaded, another field (``files``) will be populated
|
||||
with the results. This field will contain a list of dicts with information
|
||||
about the downloaded files, such as the downloaded path, the original
|
||||
scraped url (taken from the ``file_urls`` field), the file checksum and the file status.
|
||||
The files in the list of the ``files`` field will retain the same order of
|
||||
the original ``file_urls`` field. If some file failed downloading, an
|
||||
scraped URL (taken from the ``file_urls`` field), the file checksum and the
|
||||
file status. The files in the ``files`` field will retain the same order as
|
||||
in the original ``file_urls`` field. If some file failed downloading, an
|
||||
error will be logged and the file won't be present in the ``files`` field.
|
||||
|
||||
|
||||
|
|
@ -285,8 +285,8 @@ policy:
|
|||
For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide.
|
||||
|
||||
You can also use other S3-like storages. Storages like self-hosted `Minio`_ or
|
||||
`Zenko CloudServer`_. All you need to do is set endpoint option in you Scrapy
|
||||
settings:
|
||||
`Zenko CloudServer`_. All you need to do is set the endpoint option in your
|
||||
Scrapy settings:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -434,13 +434,14 @@ specifies the delay in number of days:
|
|||
|
||||
The default value for both settings is 90 days.
|
||||
|
||||
If you have pipeline that subclasses FilesPipeline and you'd like to have
|
||||
different setting for it you can set setting keys preceded by uppercase
|
||||
class name. E.g. given pipeline class called MyPipeline you can set setting key:
|
||||
If you have a pipeline that subclasses FilesPipeline and you'd like to have a
|
||||
different setting for it, you can set setting keys preceded by the uppercase
|
||||
class name. For example, given a pipeline class called MyPipeline you can set
|
||||
the setting key:
|
||||
|
||||
MYPIPELINE_FILES_EXPIRES = 180
|
||||
|
||||
and pipeline class MyPipeline will have expiration time set to 180.
|
||||
and the pipeline class MyPipeline will have an expiration time set to 180.
|
||||
|
||||
The last modified time from the file is used to determine the age of the file in days,
|
||||
which is then compared to the set expiration time to determine if the file is expired.
|
||||
|
|
|
|||
|
|
@ -7,8 +7,8 @@ Requests and Responses
|
|||
.. module:: scrapy.http
|
||||
:synopsis: Request and Response classes
|
||||
|
||||
Scrapy uses :class:`~scrapy.Request` and :class:`Response` objects for crawling web
|
||||
sites.
|
||||
Scrapy uses :class:`~scrapy.Request` and :class:`Response` objects for crawling
|
||||
websites.
|
||||
|
||||
Typically, :class:`~scrapy.Request` objects are generated in the spiders and pass
|
||||
across the system until they reach the Downloader, which executes the request
|
||||
|
|
@ -713,7 +713,7 @@ Errbacks
|
|||
========
|
||||
|
||||
The errback of a request is a function that will be called when an exception
|
||||
is raise while processing it.
|
||||
is raised while processing it.
|
||||
|
||||
It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can
|
||||
be used to track connection establishment timeouts, DNS errors etc.
|
||||
|
|
|
|||
|
|
@ -6,10 +6,10 @@ Scheduler
|
|||
|
||||
.. module:: scrapy.core.scheduler
|
||||
|
||||
The scheduler component receives requests from the :ref:`engine <component-engine>`
|
||||
and stores them into persistent and/or non-persistent data structures.
|
||||
It also gets those requests and feeds them back to the engine when it
|
||||
asks for a next request to be downloaded.
|
||||
The scheduler component receives requests from the :ref:`engine
|
||||
<component-engine>` and stores them into persistent and/or non-persistent data
|
||||
structures. It also gets those requests and feeds them back to the engine when
|
||||
it asks for the next request to be downloaded.
|
||||
|
||||
|
||||
Overriding the default scheduler
|
||||
|
|
|
|||
|
|
@ -26,13 +26,14 @@ used with HTML. `CSS`_ is a language for applying styles to HTML documents. It
|
|||
defines selectors to associate those styles with specific HTML elements.
|
||||
|
||||
.. note::
|
||||
Scrapy Selectors is a thin wrapper around `parsel`_ library; the purpose of
|
||||
this wrapper is to provide better integration with Scrapy Response objects.
|
||||
Scrapy selectors are a thin wrapper around the `parsel`_ library; the
|
||||
purpose of this wrapper is to provide better integration with Scrapy
|
||||
Response objects.
|
||||
|
||||
`parsel`_ is a stand-alone web scraping library which can be used without
|
||||
Scrapy. It uses `lxml`_ library under the hood, and implements an
|
||||
easy API on top of lxml API. It means Scrapy selectors are very similar
|
||||
in speed and parsing accuracy to lxml.
|
||||
Scrapy. It uses the `lxml`_ library under the hood and implements an easy
|
||||
API on top of the lxml API. This means Scrapy selectors have similar speed
|
||||
and parsing accuracy to lxml.
|
||||
|
||||
.. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/
|
||||
.. _lxml: https://lxml.de/
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
Settings
|
||||
========
|
||||
|
||||
The Scrapy settings allows you to customize the behaviour of all Scrapy
|
||||
The Scrapy settings allow you to customize the behaviour of all Scrapy
|
||||
components, including the core, extensions, pipelines and spiders themselves.
|
||||
|
||||
The infrastructure of the settings provides a global namespace of key-value mappings
|
||||
|
|
@ -1192,7 +1192,7 @@ response was not properly finished. If ``True``, these responses raise a
|
|||
responses are passed through and the flag ``dataloss`` is added to the
|
||||
response, i.e.: ``'dataloss' in response.flags`` is ``True``.
|
||||
|
||||
Optionally, this can be set per-request basis by using the
|
||||
Optionally, this can be set on a per-request basis by using the
|
||||
:reqmeta:`download_fail_on_dataloss` Request.meta key to ``False``.
|
||||
|
||||
.. note::
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ and what data they extract from the web pages you're trying to scrape. It
|
|||
allows you to interactively test your expressions while you're writing your
|
||||
spider, without having to run the spider to test every change.
|
||||
|
||||
Once you get familiarized with the Scrapy shell, you'll see that it's an
|
||||
Once you become familiar with the Scrapy shell, you'll see that it's an
|
||||
invaluable tool for developing and debugging your spiders.
|
||||
|
||||
.. _shell-config:
|
||||
|
|
@ -33,11 +33,10 @@ output, and more.
|
|||
Scrapy also has support for `bpython`_ via the :ref:`bpython <extras>` extra,
|
||||
and will try to use it where neither ptpython nor IPython is available.
|
||||
|
||||
Through Scrapy's settings you can configure it to use any one of
|
||||
``ptpython``, ``ipython``, ``bpython`` or the standard ``python`` shell,
|
||||
regardless of which are installed. This is done by setting the
|
||||
``SCRAPY_PYTHON_SHELL`` environment variable; or by defining it in your
|
||||
:ref:`scrapy.cfg <topics-config-settings>`:
|
||||
Through Scrapy's settings you can configure it to use any one of ``ptpython``,
|
||||
``ipython``, ``bpython`` or the standard ``python`` shell, regardless of which
|
||||
are installed. This is done by setting the ``SCRAPY_PYTHON_SHELL`` environment
|
||||
variable or by defining it in your :ref:`scrapy.cfg <topics-config-settings>`:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
|
|
|
|||
|
|
@ -154,7 +154,7 @@ scheduler_empty
|
|||
Sent whenever the engine asks for a pending request from the
|
||||
:ref:`scheduler <topics-scheduler>` (i.e. calls its
|
||||
:meth:`~scrapy.core.scheduler.BaseScheduler.next_request` method) and the
|
||||
scheduler returns none.
|
||||
scheduler returns None.
|
||||
|
||||
See :ref:`start-requests-lazy` for an example.
|
||||
|
||||
|
|
@ -174,12 +174,11 @@ Item signals
|
|||
------------
|
||||
|
||||
.. note::
|
||||
As at max :setting:`CONCURRENT_ITEMS` items are processed in
|
||||
parallel, many deferreds are fired together using
|
||||
:class:`~twisted.internet.defer.DeferredList`. Hence the next
|
||||
batch waits for the :class:`~twisted.internet.defer.DeferredList`
|
||||
to fire and then runs the respective item signal handler for
|
||||
the next batch of scraped items.
|
||||
At most :setting:`CONCURRENT_ITEMS` items are processed in parallel, many
|
||||
deferreds are fired together using
|
||||
:class:`~twisted.internet.defer.DeferredList`. Hence the next batch waits
|
||||
for the :class:`~twisted.internet.defer.DeferredList` to fire and then runs
|
||||
the respective item signal handler for the next batch of scraped items.
|
||||
|
||||
item_scraped
|
||||
~~~~~~~~~~~~
|
||||
|
|
@ -271,12 +270,12 @@ spider_closed
|
|||
:param spider: the spider which has been closed
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
:param reason: a string which describes the reason why the spider was closed. If
|
||||
it was closed because the spider has completed scraping, the reason
|
||||
is ``'finished'``. Otherwise, if the spider was manually closed by
|
||||
calling the ``close_spider`` engine method, then the reason is the one
|
||||
passed in the ``reason`` argument of that method (which defaults to
|
||||
``'cancelled'``). If the engine was shutdown (for example, by hitting
|
||||
:param reason: a string which describes the reason why the spider was
|
||||
closed. If it was closed because the spider has completed scraping, the
|
||||
reason is ``'finished'``. Otherwise, if the spider was manually closed
|
||||
by calling the ``close_spider`` engine method, then the reason is the
|
||||
one passed in the ``reason`` argument of that method (which defaults to
|
||||
``'cancelled'``). If the engine was shut down (for example, by hitting
|
||||
Ctrl-C to stop it) the reason will be ``'shutdown'``.
|
||||
:type reason: str
|
||||
|
||||
|
|
@ -453,11 +452,11 @@ request_reached_downloader
|
|||
.. signal:: request_reached_downloader
|
||||
.. function:: request_reached_downloader(request, spider)
|
||||
|
||||
Sent when a :class:`~scrapy.Request` reached downloader.
|
||||
Sent when a :class:`~scrapy.Request` reached the downloader.
|
||||
|
||||
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
|
||||
|
||||
:param request: the request that reached downloader
|
||||
:param request: the request that reached the downloader
|
||||
:type request: :class:`~scrapy.Request` object
|
||||
|
||||
:param spider: the spider that yielded the request
|
||||
|
|
@ -486,11 +485,11 @@ bytes_received
|
|||
.. signal:: bytes_received
|
||||
.. function:: bytes_received(data, request, spider)
|
||||
|
||||
Sent by some download handlers when a group of bytes is
|
||||
received for a specific request. This signal might be fired multiple
|
||||
times for the same request, with partial data each time. For instance,
|
||||
a possible scenario for a 25 kb response would be two signals fired
|
||||
with 10 kb of data, and a final one with 5 kb of data.
|
||||
Sent by some download handlers when a group of bytes is received for a
|
||||
specific request. This signal might be fired multiple times for the same
|
||||
request, with partial data each time. For instance, a possible scenario for
|
||||
a 25 KB response would be two signals fired with 10 KB of data, and a final
|
||||
one with 5 KB of data.
|
||||
|
||||
Handlers for this signal can stop the download of a response while it
|
||||
is in progress by raising the :exc:`~scrapy.exceptions.StopDownload`
|
||||
|
|
|
|||
|
|
@ -4,10 +4,10 @@
|
|||
Spider Middleware
|
||||
=================
|
||||
|
||||
The spider middleware is a framework of hooks into Scrapy's spider processing
|
||||
mechanism where you can plug custom functionality to process the responses that
|
||||
are sent to :ref:`topics-spiders` for processing and to process the requests
|
||||
and items that are generated from spiders.
|
||||
Spider middleware is a framework of hooks into Scrapy's spider processing
|
||||
mechanism where you can plug in custom functionality to process the responses
|
||||
that are sent to :ref:`topics-spiders` for processing, and to process the
|
||||
requests and items that are generated by spiders.
|
||||
|
||||
.. _topics-spider-middleware-setting:
|
||||
|
||||
|
|
@ -16,7 +16,7 @@ Activating a spider middleware
|
|||
|
||||
To activate a spider middleware component, add it to the
|
||||
:setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the
|
||||
middleware class path and their values are the middleware orders.
|
||||
middleware class paths and whose values are the middleware orders.
|
||||
|
||||
Here's an example:
|
||||
|
||||
|
|
@ -43,10 +43,10 @@ you want to insert the middleware. The order does matter because each
|
|||
middleware performs a different action and your middleware could depend on some
|
||||
previous (or subsequent) middleware being applied.
|
||||
|
||||
If you want to disable a builtin middleware (the ones defined in
|
||||
:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
|
||||
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its
|
||||
value. For example, if you want to disable the referer middleware:
|
||||
If you want to disable a built-in middleware (the ones defined in
|
||||
:setting:`SPIDER_MIDDLEWARES_BASE` and enabled by default) you must define it
|
||||
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as
|
||||
its value. For example, if you want to disable the referer middleware:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ scrapy.Spider
|
|||
.. versionchanged:: VERSION
|
||||
Changes to this attribute during a crawl are now taken into account.
|
||||
|
||||
Let's say your target url is ``https://www.example.com/1.html``,
|
||||
Let's say your target URL is ``https://www.example.com/1.html``,
|
||||
then add ``'example.com'`` to the list.
|
||||
|
||||
You may modify this attribute while the spider runs, e.g. to allow
|
||||
|
|
@ -87,8 +87,8 @@ scrapy.Spider
|
|||
:class:`~scrapy.crawler.Crawler` object to which this spider instance is
|
||||
bound.
|
||||
|
||||
Crawlers encapsulate a lot of components in the project for their single
|
||||
entry access (such as extensions, middlewares, signals managers, etc).
|
||||
Crawlers encapsulate a lot of components in the project for single-entry
|
||||
access (such as extensions, middlewares, signal managers, etc).
|
||||
See :ref:`topics-api-crawler` to know more about them.
|
||||
|
||||
.. attribute:: settings
|
||||
|
|
@ -292,9 +292,8 @@ Spiders can access arguments in their `__init__` methods:
|
|||
self.start_urls = [f"http://www.example.com/categories/{category}"]
|
||||
# ...
|
||||
|
||||
The default `__init__` method will take any spider arguments
|
||||
and copy them to the spider as attributes.
|
||||
The above example can also be written as follows:
|
||||
The default ``__init__`` method will take any spider arguments and copy them to
|
||||
the spider as attributes. The above example can also be written as follows:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -318,15 +317,13 @@ specify spider arguments when calling
|
|||
process = CrawlerProcess()
|
||||
process.crawl(MySpider, category="electronics")
|
||||
|
||||
Keep in mind that spider arguments are only strings.
|
||||
The spider will not do any parsing on its own.
|
||||
If you were to set the ``start_urls`` attribute from the command line,
|
||||
you would have to parse it on your own into a list
|
||||
using something like :func:`ast.literal_eval` or :func:`json.loads`
|
||||
and then set it as an attribute.
|
||||
Otherwise, you would cause iteration over a ``start_urls`` string
|
||||
(a very common python pitfall)
|
||||
resulting in each character being seen as a separate url.
|
||||
Keep in mind that spider arguments are only strings. The spider will not do any
|
||||
parsing on its own. If you were to set the ``start_urls`` attribute from the
|
||||
command line, you would have to parse it on your own into a list using
|
||||
something like :func:`ast.literal_eval` or :func:`json.loads` and then set it
|
||||
as an attribute. Otherwise, you would cause iteration over a ``start_urls``
|
||||
string (a very common Python pitfall) resulting in each character being seen as
|
||||
a separate url.
|
||||
|
||||
Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
|
||||
See `Scrapyd documentation`_.
|
||||
|
|
|
|||
|
|
@ -5,10 +5,11 @@ Stats Collection
|
|||
================
|
||||
|
||||
Scrapy provides a convenient facility for collecting stats in the form of
|
||||
key/values, where values are often counters. The facility is called the Stats
|
||||
Collector, and can be accessed through the :attr:`~scrapy.crawler.Crawler.stats`
|
||||
attribute of the :ref:`topics-api-crawler`, as illustrated by the examples in
|
||||
the :ref:`topics-stats-usecases` section below.
|
||||
key/value pairs, where values are often counters. The facility is called the
|
||||
Stats Collector, and can be accessed through the
|
||||
:attr:`~scrapy.crawler.Crawler.stats` attribute of the
|
||||
:ref:`topics-api-crawler`, as illustrated by the examples in the
|
||||
:ref:`topics-stats-usecases` section below.
|
||||
|
||||
The Stats Collector API is always available, so you can always use it (to
|
||||
increment or set new stat keys), regardless
|
||||
|
|
@ -19,7 +20,7 @@ collecting stats in your spider, Scrapy extension, or whatever code you're
|
|||
using the Stats Collector from.
|
||||
|
||||
Another feature of the Stats Collector is that it's very efficient (when
|
||||
enabled) and extremely efficient (almost unnoticeable) when disabled.
|
||||
enabled) and almost unnoticeable when disabled.
|
||||
|
||||
See :ref:`topics-stats-reference` below for the stats that Scrapy sets.
|
||||
|
||||
|
|
@ -29,7 +30,7 @@ Common Stats Collector uses
|
|||
===========================
|
||||
|
||||
Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats`
|
||||
attribute. Here is an example of an extension that access stats:
|
||||
attribute. Here is an example of an extension that accesses stats:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ Telnet Console
|
|||
==============
|
||||
|
||||
Scrapy comes with a built-in telnet console for inspecting and controlling a
|
||||
Scrapy running process. The telnet console is just a regular python shell
|
||||
Scrapy running process. The telnet console is just a regular Python shell
|
||||
running inside the Scrapy process, so you can do literally anything from it.
|
||||
|
||||
The telnet console is a :ref:`built-in Scrapy extension
|
||||
|
|
|
|||
|
|
@ -12,8 +12,8 @@ There are 3 numbers in a Scrapy version: *A.B.C*
|
|||
* *A* is the major version. This will rarely change and will signify very
|
||||
large changes.
|
||||
* *B* is the release number. This will include many changes including features
|
||||
and things that possibly break backward compatibility, although we strive to
|
||||
keep these cases at a minimum.
|
||||
and things that may break backward compatibility, although we strive to keep
|
||||
these cases to a minimum.
|
||||
* *C* is the bugfix release number.
|
||||
|
||||
Backward-incompatibilities are explicitly mentioned in the :ref:`release notes <news>`,
|
||||
|
|
|
|||
|
|
@ -91,7 +91,7 @@ class StopDownload(Exception):
|
|||
exception, which is in turn stored as the ``value`` attribute of the
|
||||
received :class:`~twisted.python.failure.Failure` object. This means that
|
||||
in an errback defined as ``def errback(self, failure)``, the response can
|
||||
be accessed though ``failure.value.response``.
|
||||
be accessed through ``failure.value.response``.
|
||||
|
||||
* If ``fail=False``, the request callback is called instead.
|
||||
|
||||
|
|
|
|||
|
|
@ -178,22 +178,22 @@ class LxmlLinkExtractor:
|
|||
options. It is implemented using lxml's robust HTMLParser.
|
||||
|
||||
:param allow: a single regular expression (or list of regular expressions)
|
||||
that the (absolute) urls must match in order to be extracted. If not
|
||||
that the (absolute) URLs must match in order to be extracted. If not
|
||||
given (or empty), it will match all links.
|
||||
:type allow: str or list
|
||||
|
||||
:param deny: a single regular expression (or list of regular expressions)
|
||||
that the (absolute) urls must match in order to be excluded (i.e. not
|
||||
that the (absolute) URLs must match in order to be excluded (i.e. not
|
||||
extracted). It has precedence over the ``allow`` parameter. If not
|
||||
given (or empty) it won't exclude any links.
|
||||
given (or empty), it won't exclude any links.
|
||||
:type deny: str or list
|
||||
|
||||
:param allow_domains: a single value or a list of string containing
|
||||
domains which will be considered for extracting the links
|
||||
:param allow_domains: a single value or a list of strings containing
|
||||
domains that will be considered for extracting the links.
|
||||
:type allow_domains: str or list
|
||||
|
||||
:param deny_domains: a single value or a list of strings containing
|
||||
domains which won't be considered for extracting the links
|
||||
domains that won't be considered for extracting the links.
|
||||
:type deny_domains: str or list
|
||||
|
||||
:param deny_extensions: a single value or list of strings containing
|
||||
|
|
@ -202,15 +202,14 @@ class LxmlLinkExtractor:
|
|||
:data:`scrapy.linkextractors.IGNORED_EXTENSIONS`.
|
||||
:type deny_extensions: list
|
||||
|
||||
:param restrict_xpaths: is an XPath (or list of XPath's) which defines
|
||||
regions inside the response where links should be extracted from.
|
||||
If given, only the text selected by those XPath will be scanned for
|
||||
links.
|
||||
:param restrict_xpaths: an XPath (or list of XPaths) that defines regions
|
||||
inside the response where links should be extracted. If given, only the
|
||||
text selected by those XPaths will be scanned for links.
|
||||
:type restrict_xpaths: str or list
|
||||
|
||||
:param restrict_css: a CSS selector (or list of selectors) which defines
|
||||
regions inside the response where links should be extracted from.
|
||||
Has the same behaviour as ``restrict_xpaths``.
|
||||
:param restrict_css: a CSS selector (or list of selectors) that defines
|
||||
regions inside the response where links should be extracted. It has the
|
||||
same behaviour as ``restrict_xpaths``.
|
||||
:type restrict_css: str or list
|
||||
|
||||
:param restrict_text: a single regular expression (or list of regular
|
||||
|
|
@ -224,7 +223,7 @@ class LxmlLinkExtractor:
|
|||
Defaults to ``('a', 'area')``. Use ``'*'`` to consider every tag.
|
||||
:type tags: str or list
|
||||
|
||||
:param attrs: an attribute or list of attributes which should be considered
|
||||
:param attrs: an attribute or list of attributes that should be considered
|
||||
when looking for links to extract (only for those tags specified in the
|
||||
``tags`` parameter). Defaults to ``('href',)``. Use ``'*'`` to consider
|
||||
every attribute.
|
||||
|
|
@ -247,20 +246,20 @@ class LxmlLinkExtractor:
|
|||
.. versionadded:: 2.17.0
|
||||
:type deny_attrs: str or list
|
||||
|
||||
:param canonicalize: canonicalize each extracted url (using
|
||||
w3lib.url.canonicalize_url). Defaults to ``False``.
|
||||
Note that canonicalize_url is meant for duplicate checking;
|
||||
it can change the URL visible at server side, so the response can be
|
||||
different for requests with canonicalized and raw URLs. If you're
|
||||
using LinkExtractor to follow links it is more robust to
|
||||
keep the default ``canonicalize=False``.
|
||||
:param canonicalize: canonicalize each extracted URL (using
|
||||
w3lib.url.canonicalize_url). Defaults to ``False``. Note that
|
||||
canonicalize_url is meant for duplicate checking; it can change the URL
|
||||
visible at the server side, so the response can be different for
|
||||
requests with canonicalized and raw URLs. If you're using LinkExtractor
|
||||
to follow links, it is more robust to keep the default
|
||||
``canonicalize=False``.
|
||||
:type canonicalize: bool
|
||||
|
||||
:param unique: whether duplicate filtering should be applied to extracted
|
||||
links.
|
||||
:type unique: bool
|
||||
|
||||
:param process_value: a function which receives each value extracted from
|
||||
:param process_value: a function that receives each value extracted from
|
||||
the tag and attributes scanned and can modify the value and return a
|
||||
new one, or return ``None`` to ignore the link altogether. If not
|
||||
given, ``process_value`` defaults to ``lambda x: x``.
|
||||
|
|
@ -291,12 +290,13 @@ class LxmlLinkExtractor:
|
|||
:type process_value: collections.abc.Callable
|
||||
|
||||
:param strip: whether to strip whitespaces from extracted attributes.
|
||||
According to HTML5 standard, leading and trailing whitespaces
|
||||
must be stripped from ``href`` attributes of ``<a>``, ``<area>``
|
||||
and many other elements, ``src`` attribute of ``<img>``, ``<iframe>``
|
||||
elements, etc., so LinkExtractor strips space chars by default.
|
||||
Set ``strip=False`` to turn it off (e.g. if you're extracting urls
|
||||
from elements or attributes which allow leading/trailing whitespaces).
|
||||
According to the HTML5 standard, leading and trailing whitespaces must
|
||||
be stripped from ``href`` attributes of ``<a>``, ``<area>`` and many
|
||||
other elements, the ``src`` attribute of ``<img>`` and ``<iframe>``
|
||||
elements, etc., so LinkExtractor strips space characters by default.
|
||||
Set ``strip=False`` to turn it off (for example, if you're extracting
|
||||
URLs from elements or attributes that allow leading or trailing
|
||||
whitespaces).
|
||||
:type strip: bool
|
||||
"""
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue